--- title: "A Primer on Search Relevance" book: primer-on-search-relevance edition: 2026.10.1 verified: 2026-10-18 edition_code: 1QS9-YKDZ canonical: https://rollingedition.com/primer-on-search-relevance license: CC BY 4.0 attribution: "A Primer on Search Relevance, Edition 2026.10.1, examples passed Oct 18, 2026. By Rolling Edition, https://rollingedition.com/primer-on-search-relevance. Text CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Code MIT, copyright Rolling Edition and contributors. Written with AI. Verified by people." --- Edition 2026.10.1. Examples passed Oct 18, 2026. 132 of 132 examples pass. Written with AI. Verified by people. To cite a section, name Edition 2026.10.1 and link to the section's permanent link: https://rollingedition.com/primer-on-search-relevance/s/{section_id} # Search, end to end By the end of this chapter you can build a rough but complete search pipeline for a product catalog, from a shopper's query to an ordered list of products, and measure with one number whether a change to that pipeline helps. Each later chapter takes one part of this pipeline and makes it solid. This book follows one case study throughout: an online furniture and home-goods store, which we call the store. Shoppers type queries such as `clear shower curtain` or `48 inch kitchen hood` into its search box, and the store wants to know how good its search is and whether a change makes it better. You'll face the same two questions with any search system, so the book tries every method on the store, where you can see what it does to real queries. The store has what it needs to answer them: its catalog, a set of real queries, and people's judgments of which products match which query. In this chapter you search the catalog three ways (by matching words, by matching meaning, and by combining the two), reorder the best results with a slower model, and score each version against those judgments. Not every stage improves the score, and that is the chapter's main lesson: measure every change. ## Load the store's data {#store-data} Before you can improve a search system, you need three things: the items it searches, the queries people type, and judgments that say which items match which query. Search engines call the items documents. In this book they are products. A **judgment** is a person's label for one pair of a query and a product, and a **grade** is the number that stands for the label. Without judgments, you can look at results and form opinions, but you can't tell a fix from a regression. The store's data, which the book's examples search and score, is real. It is a sample of WANDS, a public product-search dataset that Wayfair published with a paper at ECIR 2022 ([Chen et al., 2022](https://easychair.org/publications/preprint/wm4R)), under the MIT licence ([licence](https://github.com/wayfair/WANDS/blob/main/LICENSE)). The full dataset has 42,994 products, 480 queries and 233,448 judgments ([repository](https://github.com/wayfair/WANDS)). The queries are a stratified sample of real shopper queries from a quarter's search log in 2021, and each label is the majority vote of up to three trained annotators (Chen et al., sections 4.1 and 5). The script `prep/wands_sample.py` draws the book's sample, and the sample's data card, `examples/data/wands/README.md`, says how. Setup code that many examples need lives in the book's shared modules, short files in `examples/lib/` that you can open and read. The examples load the sample through one of them, `lib.store`: Example `load-the-store` (Python 3.12.15 · numpy 2.5.4 · pandas 3.0.6), passing in Edition 2026.10.1: ```python import pandas as pd from lib.store import DATA, load_labels, load_products, load_queries products = load_products() queries = load_queries() labels = load_labels() # one row per judged (query, product) pair conflicts = pd.read_csv(DATA / "label_conflicts.csv") # pairs WANDS labelled two ways print(f"products: {len(products):,}") print("queries:", len(queries)) print("train queries:", (queries["split"] == "train").sum()) print("test queries:", (queries["split"] == "test").sum()) print(f"judged pairs: {len(labels):,}") print("pairs with conflicting labels:", len(conflicts)) for label in ["exact", "partial", "irrelevant"]: print(f"{label} pairs: {(labels['label'] == label).sum():,}") print(f"products judged for at least one query: {labels['product_id'].nunique():,}") print() print(queries.head(4).to_string()) ``` Output: ```text products: 12,000 queries: 80 train queries: 50 test queries: 30 judged pairs: 13,077 pairs with conflicting labels: 7 exact pairs: 3,252 partial pairs: 7,972 irrelevant pairs: 1,853 products judged for at least one query: 11,917 query query_class split query_id 6 acrylic clear chair Dining Chairs train 10 king poster bed Beds train 20 foutains with brick look Wallpaper train 36 laundry basket with wheels Hampers & Baskets train ``` The sample has 12,000 products and 80 queries. It splits the queries once, before any ranker runs: 50 train queries for tuning and 30 test queries for reporting. Every score in this book is reported on the test queries, and nothing is tuned on them. A score on queries you tuned on measures how well you fit those queries, not how well search works. WANDS judged some pairs more than once. The sample keeps one row per pair, 13,077 in all. When a pair's labels differ, it keeps the lower grade and lists the pair in `label_conflicts.csv`, which holds 7 pairs. Code that scores a ranking reads the judgments through `lib.store.qrels`, which returns them as `{query_id: {product_id: grade}}`. [Chapter 2](#what-relevance-means) reads the judgments closely. Each judgment is one of three labels, defined by the dataset's authors. Exact: "The surfaced product fully matches the search query." Partial: "It only matches the target entity of the query, but does not satisfy the modifiers for the query." Irrelevant: "The product is not relevant to the query." (Chen et al., section 3.2). The book's grades are 2, 1 and 0. Of the judged pairs, 3,252 are exact, 7,972 partial and 1,853 irrelevant. In this book, **relevant** means exact unless a sentence says otherwise. Three warnings come with the store's data. First, each query's `query_class` is WANDS's predicted class for it, not a judgment (Chen et al., Table 1): `foutains with brick look` is classed as `Wallpaper`. Treat it as a hint. Second, nobody judged every product for every query. That would have taken about 60 million judgments, so WANDS judged candidates gathered from products shoppers clicked or added to their carts and from several search systems, and stopped when few new exact matches turned up (Chen et al., sections 4.2 and 4.3). A product without a judgment for a query is **unjudged**: unknown, not irrelevant. Third, nearly every product in the sample, 11,917 of 12,000, is judged for at least one of its queries, so the sample holds fewer distractors than a full catalog would: products that compete for a query's top places without matching it. **What to do:** split your queries into train and test before you change anything. Read test results to understand a ranker, but choose and tune every change on train queries, and report on test. Treat unjudged products as unknown. If you have no judgments yet, collect some before you change the ranking. ## Rank products with BM25 {#rank-with-bm25} Searching the store's catalog takes two jobs. **Retrieval** finds the products that might match a query, out of the whole catalog. **Ranking** puts them in order, best first. The lexical approach does both by comparing the words of the query with the words of each product's text, and scoring the overlap. BM25 is a lexical scoring function with three parts. A product scores higher when it contains the query's words, and more so for rare words, because a word that appears in few products says more about them. Repeating a word adds less and less, up to a cap, which the BM25 literature calls saturation. And a long text is discounted against a short one, so it can't win by containing more words; the parameter `b` sets how much, from 0 (no discount) to 1 (full) ([Robertson and Zaragoza, 2009](https://www.staff.city.ac.uk/~sbrp622/papers/foundations_bm25_review.pdf)). [Chapter 5](#bm25-from-scratch) builds the formula by hand. The examples use bm25s 0.3.13, a Python library for BM25. By default, it scores with the variant of BM25 used by the Lucene search library, with `k1=1.5` (the saturation setting) and `b=0.75` ([bm25s source](https://raw.githubusercontent.com/xhluca/bm25s/0.3.13/bm25s/__init__.py)). Its scores differ a little from the textbook formula, but a reproducibility study on three newswire collections found no significant differences in effectiveness between BM25's variants ([Kamphuis et al., 2020](https://cs.uwaterloo.ca/~jimmylin/publications/Kamphuis_etal_ECIR2020_preprint.pdf)). Before scoring, bm25s splits text into tokens: it lowercases, keeps words of two or more characters, removes English stopwords such as `and`, and doesn't reduce words to a common stem ([bm25s tokenizer](https://raw.githubusercontent.com/xhluca/bm25s/0.3.13/bm25s/tokenization.py)). The book's BM25 ranker, `lib.lexical.LexicalSearch`, indexes each product's name, class, category and description, joined into one text. In the store, take the test query `clear shower curtain`. A shopper who types it wants a transparent curtain, and the catalog holds many curtains that aren't clear, plus accessories that share the words. Here are BM25's top 10 products: Example `first-bm25-search` (Python 3.12.15 · bm25s 0.3.13 · numpy 2.5.4 · pandas 3.0.6 · scipy 1.18.1), passing in Edition 2026.10.1: ```python from lib.lexical import LexicalSearch from lib.store import load_products, load_queries products = load_products() query = load_queries().at[246, "query"] # a test query lexical = LexicalSearch(products) # BM25 over name, class, category and description scores = lexical.search(query, k=10) # the 10 highest scores, best first print("query:", query) print() for rank, (product_id, score) in enumerate(scores.items(), start=1): print(f"{rank:>2} {product_id:>5} {score:5.2f} {products.at[product_id, 'name'][:55]}") ``` Output: ```text query: clear shower curtain 1 9752 7.11 rodas vinyl shower curtain 2 20428 6.97 woolley vinyl single shower curtain 3 20429 6.75 woolley vinyl solid color single shower curtain 4 20425 6.65 siniard single shower curtain 5 20430 6.48 simmon vinyl single shower curtain liner 6 3564 6.10 71 '' x71 '' 4pcs shower curtain sets beach theme showe 7 20132 5.95 newburg check 3d vinyl single shower curtain + hooks 8 33346 5.76 gilded pineapple shower curtain hooks 9 3197 5.72 double shower curtain hooks 10 1050 5.68 stainless steel shower curtain hooks ``` Most of the ten are shower curtains or liners, but the last few are shower curtain hooks. BM25 found the words "shower curtain" in the hooks' names and scored them almost as high as the curtains, because it matches words, not things: it has no idea that a hook isn't a curtain. Whether the curtains are clear, the output can't tell you. For that you need the store's judgments. **What to do:** start with BM25 at the library's defaults as your baseline. It needs no training data and no model, and when it goes wrong you can see which words caused it. Leave its parameters alone until you can measure a change on [train queries](#store-data). When shoppers and products use different words for the same thing, add semantic retrieval. ## Check the results against the judgments {#check-against-judgments} A ranking means little until you know what the shopper wanted. Judgments supply that, one pair at a time. They are better evidence than clicks for this purpose: WANDS's authors note that "User behavior log data suffer from positional biases and would favor the rankings similar to the production system" (Chen et al., section 3.1). A shopper clicks what is shown first, so clicks mostly confirm the ranking you already have. For the store's query `clear shower curtain`, put each of BM25's top 10 products next to its judgment. A product nobody judged for this query shows as `unjudged`: Example `judge-first-results` (Python 3.12.15 · bm25s 0.3.13 · numpy 2.5.4 · pandas 3.0.6 · scipy 1.18.1), passing in Edition 2026.10.1: ```python from lib.lexical import LexicalSearch from lib.store import load_products, load_queries, qrels LABELS = {2: "exact", 1: "partial", 0: "irrelevant"} products = load_products() query_id = 246 query = load_queries().at[query_id, "query"] judgments = qrels()[query_id] # {product_id: grade} for this query top = LexicalSearch(products).search(query, k=10).index results = [LABELS.get(judgments.get(product_id), "unjudged") for product_id in top] print("query:", query) print() for rank, (product_id, label) in enumerate(zip(top, results), start=1): print(f"{rank:>2} {label:<10} {products.at[product_id, 'name'][:55]}") print() for label in ["exact", "partial", "irrelevant", "unjudged"]: print(f"{label} in the top 10: {results.count(label)}") print("products judged for this query:", len(judgments)) print("exact products judged for this query:", sum(grade == 2 for grade in judgments.values())) ``` Output: ```text query: clear shower curtain 1 partial rodas vinyl shower curtain 2 exact woolley vinyl single shower curtain 3 exact woolley vinyl solid color single shower curtain 4 exact siniard single shower curtain 5 exact simmon vinyl single shower curtain liner 6 partial 71 '' x71 '' 4pcs shower curtain sets beach theme showe 7 partial newburg check 3d vinyl single shower curtain + hooks 8 irrelevant gilded pineapple shower curtain hooks 9 irrelevant double shower curtain hooks 10 irrelevant stainless steel shower curtain hooks exact in the top 10: 4 partial in the top 10: 3 irrelevant in the top 10: 3 unjudged in the top 10: 0 products judged for this query: 253 exact products judged for this query: 12 ``` The top 10 holds 4 exact products, 3 partial ones, 3 irrelevant ones (the hooks) and 0 unjudged ones. The first result is partial: by [the dataset's definition](#store-data), it matches the thing the shopper asked for, a shower curtain, but not the modifier, clear. The judgments also tell you what the best possible top 10 would hold. The judges labelled 253 products for this query, 12 of them exact. So an ideal ranking could fill all ten places with exact products, and BM25 filled 4. That gap is what a metric has to measure. Reading results this way shows you the kind of mistake a ranker makes. Here, BM25 matched the words of an accessory and missed a modifier. An average over many queries hides that, and the mistakes you see are the ones you can fix. **What to do:** before you compute any metric, read the top results of five or ten queries next to their judgments. Look for patterns: accessories, missed modifiers, wrong sizes. When several top results are unjudged, have them judged before you trust a score for that query. ## Score one ranking with nDCG@10 {#score-one-ranking} To compare rankings, you need one number per ranking. It should reward matching products, and exact ones more than partial ones, and count a product near the top more than one near the bottom, because shoppers read from the top. **Normalized discounted cumulative gain at 10**, nDCG@10, does all three. In words: each of the top 10 results earns a gain equal to its [grade](#store-data): 2 for exact, 1 for partial, 0 for irrelevant. The gain is divided by a discount that grows with position: log2 of the position plus one, so the first result counts in full and the third counts half. Adding up the discounted gains gives the discounted cumulative gain, DCG. Dividing by the DCG of the best possible top 10 for that query, the ideal DCG, puts the score between 0 and 1, where 1 means no better top 10 exists. An unjudged product earns 0. ```text DCG@10 = sum over positions i from 1 to 10 of grade(i) / log2(i + 1) nDCG@10 = DCG@10 / ideal DCG@10 ``` The WANDS paper scores search with nDCG@10 too, with gains of 1.0, 0.5 and 0 for exact, partial and irrelevant (Chen et al., section 6). The book's grades are twice those gains, which doubles both DCG and ideal DCG, so nDCG doesn't change. Take [BM25's ranking of `clear shower curtain`](#check-against-judgments): a partial product first, then exact and partial curtains, then three irrelevant hooks. Compute DCG, the ideal DCG and their ratio by hand, then compare with `lib.metrics`, the shared module that holds the book's metrics: Example `ndcg-for-one-query` (Python 3.12.15 · bm25s 0.3.13 · numpy 2.5.4 · pandas 3.0.6 · scipy 1.18.1), passing in Edition 2026.10.1: ```python import math from lib.lexical import LexicalSearch from lib.metrics import ndcg_at_k from lib.store import load_products, load_queries, qrels products = load_products() query = load_queries().at[246, "query"] # "clear shower curtain" judgments = qrels("test")[246] # {product_id: grade} ranking = list(LexicalSearch(products).search(query, k=10).index) grades = [judgments.get(product_id, 0) for product_id in ranking] # unjudged counts as 0 dcg = sum(grade / math.log2(position + 1) for position, grade in enumerate(grades, start=1)) best = sorted(judgments.values(), reverse=True)[:10] # the best possible top 10 ideal = sum(grade / math.log2(position + 1) for position, grade in enumerate(best, start=1)) print("grades:", grades) print("best grades:", best) print(f"dcg@10: {dcg:.3f}") print(f"ideal dcg@10: {ideal:.3f}") print(f"ndcg@10: {dcg / ideal:.3f}") print(f"ndcg@10 from lib.metrics: {ndcg_at_k(ranking, judgments, 10):.3f}") ``` Output: ```text grades: [1, 2, 2, 2, 2, 1, 1, 0, 0, 0] best grades: [2, 2, 2, 2, 2, 2, 2, 2, 2, 2] dcg@10: 5.586 ideal dcg@10: 9.087 ndcg@10: 0.615 ndcg@10 from lib.metrics: 0.615 ``` The ranking earns a DCG of 5.586. The best possible top 10 holds only exact products and earns 9.087. Their ratio, 0.615, matches `ndcg_at_k` from `lib.metrics`, which gives 0.615. The partial product at position 1 costs the most of any single result, because it holds the place with no discount. Each hook costs less, because it sits lower, but together the hooks cost more. That one number sums up what the judgments showed: a ranking that found the right kind of product and made two kinds of mistake. You can compute it for any ranking of any judged query, and compare. **What to do:** we recommend nDCG@10 as your headline metric when your judgments are graded and the first page of results matters most, as it does in a store's search. When you have only yes-or-no labels, or when shoppers look at one result only, another metric fits better. [Chapter 3](#choose-a-ranking-metric) builds the alternatives and says when to choose each. ## Score every test query {#score-every-test-query} One query is an anecdote. To judge a ranker, score it on a fixed set of queries and average: in this book, the test queries. The rankings for every query in a set form a **run**, written as `{query_id: [product_id, ...]}`, best first. `lib.metrics.evaluate` scores a run against the judgments and returns one row per query. Next to nDCG@10, `evaluate` reports **judged@10**: the share of the top 10 that has any judgment for the query. Because unjudged products count as grade 0, a low judged@10 means the score is partly a guess. It can't tell an irrelevant product from a relevant one that nobody judged. In the store, run [BM25](#rank-with-bm25) on every test query, keeping the top 100 products for each, and score the run: Example `bm25-test-scores` (Python 3.12.15 · bm25s 0.3.13 · numpy 2.5.4 · pandas 3.0.6 · scipy 1.18.1), passing in Edition 2026.10.1: ```python from lib.lexical import LexicalSearch from lib.metrics import evaluate from lib.store import load_products, load_queries, qrels LABELS = {2: "exact", 1: "partial", 0: "irrelevant"} products, queries, judgments = load_products(), load_queries("test"), qrels("test") run = LexicalSearch(products).run(queries, k=100) # {query_id: [product_id, ...]} scores = evaluate(run, judgments) # one row per test query scores["query"] = queries["query"] print("test queries:", len(scores)) print(f"mean ndcg@10: {scores['ndcg@10'].mean():.3f}") print(f"mean judged@10: {scores['judged@10'].mean():.3f}") print(f"lowest ndcg@10: {scores['ndcg@10'].min():.3f}") print(f"median ndcg@10: {scores['ndcg@10'].median():.3f}") print(f"highest ndcg@10: {scores['ndcg@10'].max():.3f}") print("queries with ndcg@10 of 1:", (scores["ndcg@10"].round(3) == 1).sum()) print() lowest = scores.nsmallest(3, "ndcg@10") print(lowest[["query", "ndcg@10", "judged@10"]].round(3).to_string()) print("lowest query:", lowest["query"].iloc[0]) print(f"judged@10 of the lowest query: {lowest['judged@10'].iloc[0]:.3f}") print() print(f"judged@10 for {queries.at[133, 'query']}: {scores.at[133, 'judged@10']:.3f}") print("bm25 top 5 for", queries.at[133, "query"]) for product_id in run[133][:5]: label = LABELS.get(judgments[133].get(product_id), "unjudged") print(f" {label:<10} {products.at[product_id, 'name'][:60]}") ``` Output: ```text test queries: 30 mean ndcg@10: 0.725 mean judged@10: 0.927 lowest ndcg@10: 0.212 median ndcg@10: 0.687 highest ndcg@10: 1.000 queries with ndcg@10 of 1: 4 query ndcg@10 judged@10 query_id 46 beach blue headboard 0.212 0.5 133 outdoor light fixtures 0.286 0.2 292 decorative white pillow 0.467 1.0 lowest query: beach blue headboard judged@10 of the lowest query: 0.500 judged@10 for outdoor light fixtures: 0.200 bm25 top 5 for outdoor light fixtures exact malikah 1 -bulb outdoor bulkhead light unjudged 60 watt equivalent e12 led candle light bulb unjudged 5w ( 40 watt equivalent ) , b11 led , dimmable light bulb e1 unjudged 60 watt equivalent , a19 led , non-dimmable light bulb , war unjudged 40 watt equivalent e12 dimmable led edison candle light bulb ``` Over 30 test queries, BM25 averages an nDCG@10 of 0.725, with a mean judged@10 of 0.927. That average is the store's first measured result, and every later ranker is compared with it. The average hides a wide spread. Scores run from 0.212 to 1.000, with a median of 0.687, and 4 queries score a perfect 1. The lowest-scoring query, `beach blue headboard`, has a judged@10 of 0.500. Another low scorer, `outdoor light fixtures`, has a judged@10 of 0.200: BM25's top 5 holds an exact outdoor light, then light bulbs that share the query's words and that nobody judged for this query. A bulb isn't a light fixture, so here the grade of 0 is probably fair. But the metric can't know that, and for another ranker or another query, unjudged products may be good ones. **What to do:** report the mean nDCG@10 over the test queries with mean judged@10 beside it, and read the lowest-scoring queries every time you score a run. When one ranker's judged@10 is lower than another's, find out why before you compare their nDCG: the gap may come from missing judgments, not worse results. ## Search by meaning with embeddings {#search-by-meaning} Lexical retrieval fails when the shopper's words and the product's words differ, or when shared words mean different things, as with [the hooks that BM25 ranks](#rank-with-bm25) for `clear shower curtain`. **Semantic retrieval** compares meanings instead. It rests on embeddings. An **embedding** is a list of numbers, a vector, that a neural network computes for a piece of text. The network is trained so that texts with similar meanings get vectors pointing in similar directions. You embed every product once, ahead of time. At search time, you embed the query and return the products whose vectors are closest to it. With every vector scaled to length 1, closeness is the dot product of two vectors, which equals the cosine of the angle between them. Embedding the products ahead of time is what makes this fast. In the paper that introduced Sentence-BERT, finding the most similar pair among 10,000 sentences took about 65 hours on a V100 GPU when BERT read every pair. Embedding each sentence once and comparing the embeddings took about 5 seconds, almost all of it spent computing the embeddings: the comparison itself took about 0.01 seconds ([Reimers and Gurevych, 2019](https://arxiv.org/abs/1908.10084)). The store's embeddings come from the open model `Snowflake/snowflake-arctic-embed-m-v1.5`. Examples never call a model, so that every example runs offline and gives the same output each time. Instead, a **recording**, here the script `recordings/embeddings.py`, runs the model once and saves its vectors, and the examples read them offline through `lib.recorded`. We chose this model because it is open and under the Apache 2.0 licence, so anyone can download the pinned version and rerun the recording. The recording adds to each query the instruction that the model's card asks for ("Represent this sentence for searching relevant passages:") and embeds products as they are ([model card](https://huggingface.co/Snowflake/snowflake-arctic-embed-m-v1.5)). The book's semantic ranker, `lib.semantic.SemanticSearch`, multiplies every product vector by the query vector and sorts the results. [Chapter 6](#exact-nearest-neighbours-numpy) builds it step by step. For `clear shower curtain`, semantic search returns: Example `first-semantic-search` (Python 3.12.15 · numpy 2.5.4 · pandas 3.0.6), passing in Edition 2026.10.1: ```python from lib.metrics import ndcg_at_k from lib.recorded import manifest from lib.semantic import SemanticSearch from lib.store import load_products, load_queries, qrels LABELS = {2: "exact", 1: "partial", 0: "irrelevant"} products = load_products() query_id = 246 judgments = qrels()[query_id] semantic = SemanticSearch() # recorded embeddings of every product and query print("embedding model:", manifest("embeddings")["model"]) print("dimensions:", semantic.products.shape[1]) print("query:", load_queries().at[query_id, "query"]) print() similarity = semantic.search(query_id, k=10) # the query's embedding against every product's for rank, (product_id, score) in enumerate(similarity.items(), start=1): label = LABELS.get(judgments.get(product_id), "unjudged") print(f"{rank:>2} {score:.3f} {label:<10} {products.at[product_id, 'name'][:50]}") print() labels = [LABELS.get(judgments.get(product_id), "unjudged") for product_id in similarity.index] for label in ["exact", "partial", "irrelevant", "unjudged"]: print(f"{label} in the top 10: {labels.count(label)}") print(f"similarity spread in the top 10: {similarity.max() - similarity.min():.3f}") print(f"ndcg@10: {ndcg_at_k(list(similarity.index), judgments, 10):.3f}") ``` Output: ```text embedding model: Snowflake/snowflake-arctic-embed-m-v1.5 dimensions: 768 query: clear shower curtain 1 0.577 exact woolley vinyl solid color single shower curtain 2 0.572 exact woolley vinyl single shower curtain 3 0.569 partial newburg check 3d vinyl single shower curtain + hoo 4 0.556 partial rodas vinyl shower curtain 5 0.554 exact siniard single shower curtain 6 0.552 partial 71 '' x71 '' 4pcs shower curtain sets beach theme 7 0.550 exact almazan lightweight single shower curtain 8 0.549 exact simmon vinyl single shower curtain liner 9 0.546 partial vinyl striped shower curtain set 10 0.542 partial cindy window vinyl single shower curtain exact in the top 10: 5 partial in the top 10: 5 irrelevant in the top 10: 0 unjudged in the top 10: 0 similarity spread in the top 10: 0.035 ndcg@10: 0.793 ``` Each product is a vector of 768 numbers. The top 10 holds 5 exact products, against [BM25's 4](#check-against-judgments), and 0 irrelevant ones. Its nDCG@10 is 0.793, against [BM25's 0.615](#score-one-ranking). Notice how close the similarities are: the first and the tenth differ by 0.035. Semantic search always returns something, ranked by small differences. One query can mislead, so compare the two rankers on every test query: Example `compare-lexical-semantic` (Python 3.12.15 · bm25s 0.3.13 · numpy 2.5.4 · pandas 3.0.6 · scipy 1.18.1), passing in Edition 2026.10.1: ```python from lib.lexical import LexicalSearch from lib.metrics import evaluate from lib.semantic import SemanticSearch from lib.store import load_products, load_queries, qrels products, queries, judgments = load_products(), load_queries("test"), qrels("test") bm25 = evaluate(LexicalSearch(products).run(queries, k=100), judgments) semantic = evaluate(SemanticSearch().run(queries, k=100), judgments) for name, scores in [("bm25", bm25), ("semantic", semantic)]: print(f"{name} ndcg@10: {scores['ndcg@10'].mean():.3f}") print(f"{name} judged@10: {scores['judged@10'].mean():.3f}") difference = (semantic["ndcg@10"] - bm25["ndcg@10"]).round(3) # per test query print("semantic better:", (difference > 0).sum()) print("bm25 better:", (difference < 0).sum()) print("tied:", (difference == 0).sum()) sized = queries["query"].str.contains(r"\d+\s*(?:inch|in\b|\"|'')") # a size in inches print("test queries with a size in inches:", sized.sum()) print("of those, bm25 better:", (difference[sized] < 0).sum()) print() print("queries where bm25 is better:") print(queries.loc[difference[difference < 0].index, "query"].to_string()) ``` Output: ```text bm25 ndcg@10: 0.725 bm25 judged@10: 0.927 semantic ndcg@10: 0.830 semantic judged@10: 0.990 semantic better: 21 bm25 better: 7 tied: 2 test queries with a size in inches: 2 of those, bm25 better: 2 queries where bm25 is better: query_id 63 monthly calendar 72 fold in half table 94 kohler whitehaven farmhouse kitchen sink 285 48" sliding single track , barn door for laundry 292 decorative white pillow 293 wood floor protector mat 482 48 inch kitchen hood ``` Semantic retrieval averages 0.830 against BM25's 0.725. Its judged@10 of 0.990 is higher than BM25's 0.927, so part of BM25's deficit could come from products nobody judged. For [`outdoor light fixtures`](#score-every-test-query), at least, those products are light bulbs, and a score of 0 is fair. Semantic retrieval is better on 21 queries, BM25 on 7, and 2 are tied. The test set has only 30 queries, so [chapter 3](#compare-runs-in-a-harness) tests whether a gap like this one is larger than chance. BM25 is better on 2 of the 2 test queries that give a size in inches. Exact tokens such as sizes and model numbers may favour matching words, but a count this small can't show a pattern. On the train queries that contain a digit, BM25 is better on 1 of 5 ([Find the queries each retriever wins](#queries-each-retriever-wins)), and the embedding model matches digits too, as text ([Check queries that contain digits](#embeddings-match-digits)). A theoretical result shows a limit of single vectors: Weller et al. prove that an embedding's dimension limits which combinations of top results a single vector can return. On LIMIT, a dataset they built to test that limit, BM25 finds 93.6% of the relevant items in its top 100, and the best single-vector model 18.9%. On a variant of LIMIT with words replaced by synonyms, BM25's score drops by more than 89% ([Weller et al., 2025](https://arxiv.org/abs/2508.21038)). Lexical and semantic retrieval fail in different places. **What to do:** add semantic retrieval when shoppers describe what they want in their own words. Don't assume lexical retrieval wins on exact terms such as sizes, brands and model numbers: before you rely on it there, count which retriever wins the train queries that contain them. On this data, semantic retrieval alone scores highest so far, so it is the ranker to beat. ## Fuse two rankings with reciprocal rank fusion {#fuse-with-rrf} Because lexical and semantic retrieval fail in different places, **hybrid retrieval** combines a ranking from each, here BM25's and semantic retrieval's, hoping to keep the strengths of both. Their scores can't be added directly: a BM25 score has no fixed range and depends on the query's words, while a cosine similarity lies between -1 and 1, so a sum would mostly follow whichever is larger. Reciprocal rank fusion (RRF) avoids the problem by using ranks only. Each product scores, for every ranking that contains it, 1 divided by the sum of a constant and its rank, and the scores add up: ```text RRF(product) = sum over rankings r of 1 / (k + rank of product in r), with k = 60 ``` The constant k flattens the top: with k = 60, rank 1 earns 1/61 and rank 10 earns 1/70. So a product near the top of both rankings beats one at the very top of only one. The authors fixed k = 60 in a pilot study and found that "the choice was not critical" ([Cormack et al., 2009](https://plg.uwaterloo.ca/~gvcormac/cormacksigir09-rrf.pdf)). They fused two kinds of runs: submissions to TREC, the Text REtrieval Conference's shared evaluations, and 30 configurations of one search system. In those experiments, RRF beat the best individual run by 4% to 5% on average in mean average precision (MAP), which averages, over queries, the precision at the position of each relevant result. That's a different setting from fusing one lexical and one semantic ranking, and a later study of hybrid search found that a tuned weighted sum of normalized scores beats RRF ([Bruch et al., 2023](https://arxiv.org/abs/2210.11934)). [Chapter 7](#tune-blend-on-train) tunes both on train queries. Here, RRF runs untuned. RRF is short enough to write by hand, as two functions: `rrf_scores` scores one query's products, and `reciprocal_rank_fusion` orders them for every query, putting products with equal scores in `product_id` order so that neither ranking wins ties by being passed first. The example fuses BM25's and semantic retrieval's top 100 for every test query, checks the scores of the two products that `clear shower curtain` puts first, and scores the hybrid run. Before you read the output, predict the hybrid's mean nDCG@10. [BM25 scores 0.725 and semantic retrieval 0.830](#search-by-meaning). Will their fusion score above semantic retrieval, between the two, or below BM25? Example `hybrid-rrf` (Python 3.12.15 · bm25s 0.3.13 · numpy 2.5.4 · pandas 3.0.6 · scipy 1.18.1), passing in Edition 2026.10.1: ```python from lib.lexical import LexicalSearch from lib.metrics import evaluate from lib.semantic import SemanticSearch from lib.store import load_products, load_queries, qrels def rrf_scores(rankings, k=60): """Each product scores the sum of 1 / (k + rank) over the rankings that contain it.""" scores = {} for ranking in rankings: for rank, product_id in enumerate(ranking, start=1): scores[product_id] = scores.get(product_id, 0) + 1 / (k + rank) return scores def reciprocal_rank_fusion(runs, k=60, depth=100): """For each query, products in order of their RRF scores, best first; equal scores by product_id.""" fused = {} for query_id in runs[0]: scores = rrf_scores([run[query_id] for run in runs], k) fused[query_id] = sorted(scores, key=lambda product_id: (-scores[product_id], product_id))[:depth] return fused products, queries, judgments = load_products(), load_queries("test"), qrels("test") bm25, semantic = LexicalSearch(products).run(queries, k=100), SemanticSearch().run(queries, k=100) hybrid = reciprocal_rank_fusion([bm25, semantic]) for place, product_id in zip(["first", "second"], hybrid[246]): # query 246, "clear shower curtain" bm25_rank, semantic_rank = bm25[246].index(product_id) + 1, semantic[246].index(product_id) + 1 print(f"{place} product: {products.at[product_id, 'name']}") print(f"{place} product's bm25 rank: {bm25_rank}") print(f"{place} product's semantic rank: {semantic_rank}") print(f"{place} product's score by hand: {1 / (60 + bm25_rank) + 1 / (60 + semantic_rank):.5f}") print(f"{place} product's score from rrf_scores: {rrf_scores([bm25[246], semantic[246]])[product_id]:.5f}") hybrid_scores, semantic_scores = evaluate(hybrid, judgments), evaluate(semantic, judgments) print(f"hybrid ndcg@10: {hybrid_scores['ndcg@10'].mean():.3f}") print(f"hybrid judged@10: {hybrid_scores['judged@10'].mean():.3f}") difference = (hybrid_scores["ndcg@10"] - semantic_scores["ndcg@10"]).round(3) print("better than semantic:", (difference > 0).sum()) print("worse than semantic:", (difference < 0).sum()) print("tied with semantic:", (difference == 0).sum()) ``` Output: ```text first product: woolley vinyl solid color single shower curtain first product's bm25 rank: 3 first product's semantic rank: 1 first product's score by hand: 0.03227 first product's score from rrf_scores: 0.03227 second product: woolley vinyl single shower curtain second product's bm25 rank: 2 second product's semantic rank: 2 second product's score by hand: 0.03226 second product's score from rrf_scores: 0.03226 hybrid ndcg@10: 0.812 hybrid judged@10: 0.980 better than semantic: 9 worse than semantic: 14 tied with semantic: 7 ``` The first fused product, `woolley vinyl solid color single shower curtain`, is at rank 3 in BM25's ranking and rank 1 in semantic retrieval's, so by hand its score is 1/(60 + 3) + 1/(60 + 1) = 0.03227, the same as `rrf_scores` gives. The second, `woolley vinyl single shower curtain`, is at rank 2 in BM25's ranking and rank 2 in semantic retrieval's, and scores 0.03226, so it comes second. The hybrid run scores 0.812: above BM25, and below semantic retrieval alone. It beats semantic retrieval on 9 queries, loses on 14 and ties on 7, and its judged@10 is 0.980. On 30 test queries, a gap this small may be noise, and [chapter 3](#compare-runs-in-a-harness) tests whether it is. So the finding is that fusion didn't improve on semantic retrieval here, not that it made search worse. Why it didn't help is a question for chapter 7, which [checks each retriever before fusing](#check-each-retriever-before-fusing) and [tunes the blend on train queries](#tune-blend-on-train). If you predicted "above semantic retrieval", notice that combining two rankings doesn't guarantee a gain. Fusion is a change like any other, and it has to be measured. **What to do:** measure each retriever alone before you fuse them. Use RRF with k = 60 as an untuned default, and keep the fused ranking only if it beats the best single retriever on held-out queries. When one retriever is much weaker, try weighting the rankings or tuning k on train queries instead of accepting equal votes. Chapter 7 tries both on the store's data, and none of the tuned blends it reports on the test queries beats semantic retrieval ([Report the tuned blends on test queries](#report-blends-on-test)). ## Rerank the top 50 with a cross-encoder {#rerank-with-cross-encoder} Every stage so far has been retrieval, which has to score every product in the catalog for every query and so must be cheap. **Reranking** reorders a short list from retrieval with a slower model, one that can afford to look at the query and each product together because it only sees a few dozen products. One kind of reranking model is a **cross-encoder**. It reads the query and the product text together, as one input, and outputs one relevance score for the pair, scoring each pair independently ([Nogueira and Cho, 2019](https://arxiv.org/abs/1901.04085)). An embedding model reads the product alone, so its vectors can be computed ahead of time. A cross-encoder needs the query, so it can't compute anything ahead of time, and running it on every pair costs time: the slow side of [the Sentence-BERT comparison](#search-by-meaning) was BERT reading every pair of sentences as a cross-encoder. So a cross-encoder reranks a shortlist that a cheaper first stage retrieves (Nogueira and Cho, section 2; [Retrieve & Re-Rank](https://sbert.net/examples/sentence_transformer/applications/retrieve_rerank/README.html)). In return for that cost, it can relate every word of the query to every word of the product through attention, the part of the model that lets each word draw on the others in its input. In the words of [Reimers and Gurevych (2019, section 4.3)](https://arxiv.org/abs/1908.10084), "BERT is able to use attention to compare directly both sentences". In 2019, on the MS MARCO passage benchmark, reranking BM25's top 1,000 passages this way improved MRR@10, a measure of how high the first relevant passage lands, by 27% relative to the best earlier result (Nogueira and Cho). A cross-encoder is a language model, but not a **large language model (LLM)**. An LLM is a model trained on large amounts of text to generate text in response to a prompt. Later chapters use LLMs to judge, rewrite and rerank. A cross-encoder generates nothing: it outputs one number. The store's cross-encoder is the open model `Alibaba-NLP/gte-reranker-modernbert-base`, chosen for the same reason as [the embedding model](#search-by-meaning): it is open and under the Apache 2.0 licence, so anyone can rerun the recording. Its recording, `recordings/cross_encoder.py`, saves the model's scores once, for every product in the union of BM25's and semantic retrieval's top 100 for each query, so every product in the hybrid's top 50 has a score. The example reorders the top 50 of [the hybrid run](#fuse-with-rrf), BM25 and semantic retrieval fused with RRF, by those scores, keeps the rest in place, and scores the result: Example `cross-encoder-rerank` (Python 3.12.15 · bm25s 0.3.13 · numpy 2.5.4 · pandas 3.0.6 · scipy 1.18.1), passing in Edition 2026.10.1: ```python from lib.fusion import reciprocal_rank_fusion from lib.lexical import LexicalSearch from lib.metrics import evaluate from lib.recorded import cross_encoder_scores, manifest from lib.semantic import SemanticSearch from lib.store import load_products, load_queries, qrels products, queries, judgments = load_products(), load_queries("test"), qrels("test") bm25 = LexicalSearch(products).run(queries, k=100) semantic = SemanticSearch().run(queries, k=100) hybrid = reciprocal_rank_fusion([bm25, semantic]) # the same function as in the hybrid section recorded = cross_encoder_scores() # one score per (query, product) pair score = {(query_id, product_id): value for query_id, product_id, value in recorded.itertuples(index=False)} print("cross-encoder model:", manifest("cross-encoder")["model"]) depth = 50 # rerank the top 50, keep the rest in place reranked = {} for query_id, ranking in hybrid.items(): top = sorted(ranking[:depth], key=lambda product_id: score[(query_id, product_id)], reverse=True) reranked[query_id] = top + ranking[depth:] before, after = evaluate(hybrid, judgments), evaluate(reranked, judgments) print(f"reranked ndcg@10: {after['ndcg@10'].mean():.3f}") print(f"reranked judged@10: {after['judged@10'].mean():.3f}") difference = (after["ndcg@10"] - before["ndcg@10"]).round(3) print("better than hybrid:", (difference > 0).sum()) print("worse than hybrid:", (difference < 0).sum()) ``` Output: ```text cross-encoder model: Alibaba-NLP/gte-reranker-modernbert-base reranked ndcg@10: 0.798 reranked judged@10: 0.980 better than hybrid: 11 worse than hybrid: 12 ``` Reranking scores 0.798, against 0.812 for the hybrid run it started from. It improves 11 queries and hurts 12. As with fusion, a gap this small on 30 test queries may be noise, which [chapter 3](#compare-runs-in-a-harness) tests: the finding is that this reranker didn't improve the ranking here. Across the chapter, on the test queries: | Ranker | nDCG@10 | judged@10 | | --- | --- | --- | | BM25 | 0.725 | 0.927 | | Semantic | 0.830 | 0.990 | | Hybrid (RRF, k = 60) | 0.812 | 0.980 | | Hybrid, top 50 reranked | 0.798 | 0.980 | To see what reranking does to a single query, look at one where it lowered the score, `refrigerator with ice an water in door`: Example `rerank-one-query` (Python 3.12.15 · bm25s 0.3.13 · numpy 2.5.4 · pandas 3.0.6 · scipy 1.18.1), passing in Edition 2026.10.1: ```python import re from lib.fusion import reciprocal_rank_fusion from lib.lexical import LexicalSearch from lib.metrics import ndcg_at_k from lib.recorded import cross_encoder_scores from lib.semantic import SemanticSearch from lib.store import load_products, load_queries, product_text, qrels LABELS = {2: "exact", 1: "partial", 0: "irrelevant"} products = load_products() query_id = 406 query = load_queries().at[query_id, "query"] judgments = qrels()[query_id] recorded = cross_encoder_scores() score = recorded[recorded["query_id"] == query_id].set_index("product_id")["score"] bm25 = {query_id: list(LexicalSearch(products).search(query, k=100).index)} semantic = {query_id: list(SemanticSearch().search(query_id, k=100).index)} hybrid = reciprocal_rank_fusion([bm25, semantic])[query_id] reranked = sorted(hybrid[:50], key=score.get, reverse=True) + hybrid[50:] print("query:", query) for name, ranking in [("hybrid", hybrid), ("reranked", reranked)]: first_exact = next(rank for rank, product_id in enumerate(ranking, start=1) if judgments.get(product_id) == 2) print() print(f"{name} ndcg@10: {ndcg_at_k(ranking, judgments, 10):.3f}") print(f"{name} rank of the first exact product: {first_exact}") for product_id in ranking[:5]: label = LABELS.get(judgments.get(product_id), "unjudged") print(f" {score[product_id]:.2f} {label:<10} {products.at[product_id, 'name']}") exact = next(product_id for product_id in hybrid if judgments.get(product_id) == 2) words = set(re.findall(r"\w+", product_text(products).at[exact].lower())) # name, class, category, description print() print("hybrid's first exact product:", products.at[exact, "name"]) print("its text says ice or water:", "yes" if {"ice", "water"} & words else "no") ``` Output: ```text query: refrigerator with ice an water in door hybrid ndcg@10: 0.660 hybrid rank of the first exact product: 3 1.95 partial apec water icemaker-kit-38-14-ro refrigerator/icemaker installation accessory 2.06 partial 36 '' french door 26.1 cu . ft. smart energy star refrigerator with interior ice maker 1.44 exact 36 '' french door refrigerator 27 cu . ft. energy star refrigerator 2.19 partial 36 '' energy star counter-depth french door 22.6 cu . ft. refrigerator with ice maker 2.19 partial 800 series 21 cu . ft. smart energy star counter depth french door refrigerator with internal water dispenser reranked ndcg@10: 0.533 reranked rank of the first exact product: 9 2.20 partial smeg classic 36 '' french door 22 cu . ft. refrigerator with ice maker 2.19 partial 36 '' energy star counter-depth french door 22.6 cu . ft. refrigerator with ice maker 2.19 partial 800 series 21 cu . ft. smart energy star counter depth french door refrigerator with internal water dispenser 2.10 partial 28 '' energy star counter depth top freezer 12.56 cu . ft. refrigerator with auto ice maker 2.06 partial 36 '' french door 26.1 cu . ft. smart energy star refrigerator with interior ice maker hybrid's first exact product: 36 '' french door refrigerator 27 cu . ft. energy star refrigerator its text says ice or water: no ``` For this query, the reranked top 5 are refrigerators whose names say `ice maker` or `water dispenser`, and the judges marked them partial: they match the thing asked for, a refrigerator, but not the modifier, ice and water in the door. The first exact product in the hybrid run, a French-door refrigerator, gets the lowest cross-encoder score of the hybrid's top 5. After reranking, the first exact product sits at rank 9 instead of 3, and nDCG@10 for this query drops from 0.660 to 0.533. The judges marked that refrigerator exact, so by the dataset's definition it fully matches the query. Yet its text says nothing about ice or water, so no model that reads only the text could rank it on that modifier. WANDS's judges saw each product's image and web page as well as its name ([annotation guidelines](https://github.com/wayfair/WANDS/raw/main/Product%20Search%20Relevance%20Annotation%20Guidelines.pdf)), which probably let them see what the text doesn't say. A general-purpose reranker has never seen this store's judgments. [Chapter 9](#train-lambdamart) trains rankers on the train queries' judgments. **What to do:** rerank a shortlist, here 50 products, because the cost grows with every pair. Adopt a reranker only when it beats the best retrieval alone on held-out queries. This one didn't improve on the hybrid run or on semantic retrieval, so the store's best pipeline so far is semantic retrieval alone. Before you give up on reranking, try a different depth, or a ranker trained on your own judgments. Chapter 9 tries both on the store's data, and neither shows an improvement over semantic retrieval on the test queries ([Set the rerank depth from a budget](#rerank-depth-from-budget), [Report LambdaMART on the test queries](#lambdamart-on-test-queries)). ## What goes wrong in production {#search-end-to-end-in-production} **The tokenizer drops words without telling you.** bm25s's default tokenizer keeps only tokens of two or more characters and doesn't stem words ([bm25s tokenizer](https://raw.githubusercontent.com/xhluca/bm25s/0.3.13/bm25s/tokenization.py)). Run it on a train query and a test query, and count the products it can't match: Example `tokenizer-drops-terms` (Python 3.12.15 · bm25s 0.3.13 · numpy 2.5.4 · pandas 3.0.6 · scipy 1.18.1), passing in Edition 2026.10.1: ```python import bm25s from lib.store import load_products, product_text def tokenize(texts): """The tokenizer lib.lexical uses for products and queries, with bm25s's default settings.""" return bm25s.tokenize(texts, stopwords="en", return_ids=False, show_progress=False) for query in ["ge top loading washer 4.5", "bistro table and chairs"]: print(f"{query!r} -> {tokenize([query])[0]}") products = tokenize(product_text(load_products()).tolist()) print() print("products with the token chairs:", sum("chairs" in tokens for tokens in products)) print("products with chair but not chairs:", sum("chair" in tokens and "chairs" not in tokens for tokens in products)) ``` Output: ```text 'ge top loading washer 4.5' -> ['ge', 'top', 'loading', 'washer'] 'bistro table and chairs' -> ['bistro', 'table', 'chairs'] products with the token chairs: 895 products with chair but not chairs: 359 ``` The `4.5` in `ge top loading washer 4.5` disappears, because `4` and `5` are one character each, so that number plays no part in the ranking. And `chairs` matches only products that say `chairs`: 895 products do, while 359 say `chair` and never `chairs`, and none of those can match that word of `bistro table and chairs`. Neither case raises an error. [Chapter 5](#split-text-into-terms) shows both problems and [adds stemming](#stem-word-forms). **Unjudged results look like failures.** A ranker that surfaces products nobody judged gets lower scores than it may deserve, and a ranker that surfaces the judged ones gets an advantage. Report [judged@10](#score-every-test-query) with every score, and have new products judged when it drops. **Scores from a sample are probably optimistic.** The sample keeps 12,000 products, nearly all judged for some query. The full WANDS catalog has 42,994 ([repository](https://github.com/wayfair/WANDS)), and a store's catalog may be far larger, with more near misses competing for the top 10. Nothing in this chapter measures that, but expect lower scores on a full catalog. Don't compare the scores here with the WANDS paper's either: it ran the BM25 of Solr, a search server, over product titles and descriptions (Chen et al.). **Model outputs belong to one model version.** Product and query embeddings must come from the same model at the same version, or their similarities aren't comparable. Change the model and you re-embed the whole catalog. Each folder in `examples/data/recorded/` has a `manifest.json` that pins the model and its version. **Reranking costs time on every query.** Product embeddings are computed once. A cross-encoder runs on every pair, for every query, while the shopper waits. In this chapter, a recording script computes the scores ahead of time on a GPU, which hides that cost. Chapter 9 counts what each rerank depth costs ([Set the rerank depth from a budget](#rerank-depth-from-budget)). **What to do:** check the tokens your queries become, track judged@10, and record the model version next to every vector you store. ## Decisions this chapter made {#search-end-to-end-decisions} Each choice below has a reason, and each is a default that a later chapter tests on the train queries before changing it. - **Report on test, tune on train.** Every score in this chapter is on the 30 test queries. This chapter tunes nothing; later chapters tune on the 50 train queries only. - **Headline metric: nDCG@10, with judged@10 beside it.** The grades serve as gains, unjudged products count as grade 0, and judged@10 shows when that assumption carries weight. - **Baseline: BM25 at bm25s's defaults,** `k1=1.5` and `b=0.75`, over each product's name, class, category and description. Tuning waits until it can be measured on train queries. - **Semantic retrieval from recorded embeddings** of one open model, compared by dot product of unit-length vectors, so every example runs offline and gives the same output each time. Choosing between models is a later question. - **Hybrid retrieval: RRF with k = 60 and equal weights, untuned,** as its authors used it. - **Rerank depth: the hybrid's top 50,** scored by a recorded cross-encoder, so every reranked product has a recorded score. In a live store, depth is a cost decision as much as a quality one. - **The ranker to beat is semantic retrieval alone,** because on this data neither fusion nor this reranker improved on it. [Chapter 3](#compare-runs-in-a-harness) tests whether gaps like these are larger than chance, in an evaluation harness. Every later change is tested in that harness against the ranker it changes, and adopted only if it beats the ranker to beat. # What relevance means By the end of this chapter you can read a set of relevance judgments critically: say what each grade claims about a product, find where the judgments came from and where they have holes, and decide how far to trust a score computed from them. In [chapter 1](#search-end-to-end), the store this book follows, [an online furniture and home-goods retailer](#store-data), scores its rankers against 13,077 judgments and treats every grade as fact: on the [test queries](#store-data), [BM25](#rank-with-bm25) averages an nDCG@10 ([Score one ranking](#score-one-ranking)) of 0.725, and [semantic retrieval](#search-by-meaning) 0.830. Each of those numbers rests on choices the judgments make without saying so: whose idea of relevance they record, how much a near miss is worth, and what happens to the products nobody judged. This chapter reads the store's judgments closely. It changes no ranker, so it reads [train and test queries](#store-data) alike: reading judgments isn't tuning, and every fix in this book is still chosen on train queries only. The few tests it runs on the test queries are exploratory: they describe the judgments, and none establishes a difference ([the book's rule](#when-a-difference-is-established)). [Chapter 3](#compare-runs-in-a-harness) tests the gap between BM25 and semantic retrieval as a planned comparison. ## Judge relevance against the shopper's need {#relevance-is-a-judgment} A search system returns products that match a query in the way the system defines a match: shared words for lexical retrieval, nearby vectors for semantic retrieval. Information science calls that *system* relevance and separates it from *topical* relevance, whether a product is about the subject of the query, and *situational* relevance, whether it helps with the task at hand ([Saracevic, 2007](https://tefkos.rutgers-sci.domains/Saracevic%20relevance%20pt%20II%20JASIST%20%2707.pdf)). To the argument that system relevance is always perfect, because the system retrieved what the query asked for, Saracevic answers: "Not so." A **judgment** is a person's label for one pair of a query and a product, their verdict on it, and a **grade** is the number that stands for the label. The store's judgments come from WANDS, a public product-search dataset, through a sample of its queries and products ([Load the store's data](#store-data)). WANDS's judges label each pair exact, partial or irrelevant, which the book grades 2, 1 and 0. Exact means the product is the kind of thing the query asks for and has every attribute the query names; partial means it is the right kind of thing but misses an attribute; irrelevant means it isn't the kind of thing asked for ([Read intent as a product and its modifiers](#intent-in-product-search)). The judge doesn't grade the product against the query's words but against the **information need** behind them, the thing the shopper wanted: "A document is relevant if it addresses the stated information need", and not because it happens to contain the query's words ([Manning et al., 2008, chapter 8](https://nlp.stanford.edu/IR-book/html/htmledition/information-retrieval-system-evaluation-1.html)). Evaluation then makes a bargain. It fixes a set of documents, a set of need statements and their judgments, and abstracts away the real users ([Voorhees, 2007](https://cs.uwaterloo.ca/~brecht/courses/854-Experimental-Performance-Evaluation-2017/readings/trec-cacm-2007.pdf)). In the store, the judgments stand in for shoppers, and every score in this book inherits that bargain. If matching words were the same as relevance, a product whose text holds every word of the query would be exact. Test that on every judged pair in the store. The example checks whether each product's text, its name, class, category and description, contains every word of its query. The sample cuts descriptions to about 400 characters. Before you run it, predict: of the judged pairs whose text holds every query word, what share did the judges call exact? Example `word-match-versus-judgment` (Python 3.12.15 · numpy 2.5.4 · pandas 3.0.6), passing in Edition 2026.10.1: ```python import re from lib.store import load_labels, load_products, load_queries, product_text def words(text): """Lowercase word tokens: a rough stand-in for what lexical search matches.""" return set(re.findall(r"\w+", text.lower())) products, queries, labels = load_products(), load_queries(), load_labels() text_words = {product_id: words(text) for product_id, text in product_text(products).items()} attribute_words = {product_id: words(text) for product_id, text in products["attributes"].items()} query_words = {query_id: words(query) for query_id, query in queries["query"].items()} pairs = list(zip(labels["query_id"], labels["product_id"])) labels["in text"] = [query_words[query_id] <= text_words[product_id] for query_id, product_id in pairs] labels["in text or attributes"] = [query_words[query_id] <= text_words[product_id] | attribute_words[product_id] for query_id, product_id in pairs] matched = labels[labels["in text"]] print(f"exact share of all judged pairs: {(labels['label'] == 'exact').mean():.3f}") print(f"judged pairs whose text has every query word: {len(matched):,}") for label in ["exact", "partial", "irrelevant"]: print(f"of them {label}: {(matched['label'] == label).mean():.3f}") exact = labels[labels["label"] == "exact"] print(f"exact pairs: {len(exact):,}") print(f"exact pairs whose text misses a query word: {(~exact['in text']).sum():,}") print(f"of them, still missing one with attributes added: {(~exact['in text or attributes']).sum():,}") curtains = list(exact.loc[exact["query_id"] == 246, "product_id"]) # clear shower curtain unsaid = [product_id for product_id in curtains if "clear" not in text_words[product_id]] print("exact products for clear shower curtain:", len(curtains)) print("of them without the word clear in their text:", len(unsaid)) print("of those, with clear in their attributes:", sum("clear" in attribute_words[product_id] for product_id in unsaid)) ``` Output: ```text exact share of all judged pairs: 0.249 judged pairs whose text has every query word: 1,177 of them exact: 0.777 of them partial: 0.211 of them irrelevant: 0.012 exact pairs: 3,252 exact pairs whose text misses a query word: 2,337 of them, still missing one with attributes added: 2,251 exact products for clear shower curtain: 12 of them without the word clear in their text: 8 of those, with clear in their attributes: 4 ``` Of the 1,177 pairs with a full word match, 0.777 are exact, 0.211 partial and 0.012 irrelevant. A full match is a strong signal: over all judged pairs, the exact share is 0.249. But it doesn't settle the question: the words can be present without the product being what the shopper meant. If you predicted a share close to 1, you treated matching words as relevance. The judges didn't. The other direction is more striking. Of the 3,252 exact pairs, 2,337 have a product text that misses at least one word of the query. Some misses are word forms and spelling, such as `chairs` against `chair` ([chapter 1](#search-end-to-end-in-production)), or the query `ligth bulb`; [chapter 5](#stem-word-forms) and [chapter 8](#correct-spelling-from-catalog) take those up. Others are attributes the text doesn't state. WANDS records attributes such as color, material and size apart from the text ([Chen et al., 2022, Table 1](https://easychair.org/publications/preprint/wm4R)), and the store keeps a few of them in an `attributes` column, for example `color: clear; material: vinyl` ([data card](../examples/data/wands/README.md)). The text that the store's search reads, `lib.store.product_text`, leaves that column out. Adding the attributes to the text leaves 2,251 of those exact pairs still missing a query word, so attributes explain few misses overall. For some queries, though, the attributes hold the answer. For `clear shower curtain`, 8 of the 12 exact products never say `clear` in their text, and 4 of those carry `clear` in their attributes, where the store's text search never looks. Chapter 5 tests adding them to lexical search ([Index product attributes as a field](#index-attributes-as-a-field)). For the other exact curtains, neither the text nor the attributes say `clear`. The judges had more to go on: WANDS's annotation guidelines show them each product's name, image and web page ([guidelines](https://github.com/wayfair/WANDS/raw/3b74dcf4ba29ab8ff3e6a50b5b09fc627cb882b5/Product%20Search%20Relevance%20Annotation%20Guidelines.pdf)). The photo or the page may be what told them, but that is our guess: the data doesn't record which evidence a judge used. **What to do:** treat a judgment as a person's verdict about a need, not a property of the product. Before you blame a ranker for missing an exact product, check whether any field the ranker reads says what the judges saw, and look at the fields it doesn't read, such as attributes. When your judges see more than your ranker does, give the ranker the same evidence where you can, or accept that some exact products will rank low for reasons no change to the ranking can fix. ## Expect judges to disagree on some pairs {#judges-disagree-on-some-pairs} Because a judgment is a verdict, two people can reach different ones. Each WANDS label is already the majority vote of up to three annotators ([Chen et al., 2022, section 5](https://easychair.org/publications/preprint/wm4R)), and the annotators started with moderate agreement and converged with training. Cohen's kappa, a measure of agreement that discounts the agreement expected by chance, rose from 0.467 in their first two months to 0.664 in the third and 0.826 in the fourth (Table 2). [Chapter 4](#agreement-beyond-chance) computes kappa. WANDS's file lists some of the store's pairs more than once, each row a majority vote, and for a few the votes differ. The store's sample keeps one row per pair, graded 2 for exact, 1 for partial and 0 for irrelevant, and where the rows differ it keeps the lower grade ([Load the store's data](#store-data)). List the pairs whose rows differ: Example `conflicting-judgments` (Python 3.12.15 · numpy 2.5.4 · pandas 3.0.6), passing in Edition 2026.10.1: ```python import pandas as pd from lib.store import DATA, load_labels, load_products, load_queries GRADES = {"exact": 2, "partial": 1, "irrelevant": 0} products, queries, labels = load_products(), load_queries(), load_labels() conflicts = pd.read_csv(DATA / "label_conflicts.csv") # pairs WANDS lists more than once, with different labels kept = labels.set_index(["query_id", "product_id"])["label"] for row in conflicts.itertuples(): print(f"{queries.at[row.query_id, 'query']:<32} {products.at[row.product_id, 'name'][:40]:<41} " f"{row.labels:<19} kept: {kept[(row.query_id, row.product_id)]}") pairs = conflicts["labels"].str.split("|") spread = pairs.apply(lambda both: max(GRADES[label] for label in both) - min(GRADES[label] for label in both)) print() print("pairs with conflicting labels:", len(conflicts)) print("exact against partial:", (pairs.apply(frozenset) == frozenset(["exact", "partial"])).sum()) print("partial against irrelevant:", (pairs.apply(frozenset) == frozenset(["partial", "irrelevant"])).sum()) print(f"conflicts between neighbouring grades: {(spread == 1).sum()} of {len(conflicts)}") ``` Output: ```text whimsical solar lights solar powered led pathway light pack exact|partial kept: partial indoor multi tiered plant stand taranto multi-tiered plant stand exact|partial kept: partial clear shower curtain shower curtain hooks irrelevant|partial kept: irrelevant breakfast bar table counter height dining table exact|partial kept: partial wood floor protector mat hard floor straight rectangular chair ma exact|partial kept: partial malachi sled sled coffee table irrelevant|partial kept: irrelevant kohler purist brushed bronze purist widespread bathroom faucet with d exact|partial kept: partial pairs with conflicting labels: 7 exact against partial: 5 partial against irrelevant: 2 conflicts between neighbouring grades: 7 of 7 ``` Of the judged pairs, 7 conflict: 5 are exact against partial and 2 partial against irrelevant. Of those conflicts, 7 of 7 sit between neighbouring grades. Whether a `counter height dining table` fully matches `breakfast bar table` is a fair question about a modifier, not a sign that one vote misread the task. Conflicts like these show where the line between grades is soft. **What to do:** keep one grade per pair and write down the rule that chose it. We recommend the lower grade, as the sample does: it errs against the ranker that ranks the product, not in its favour. When judges disagree across two grades, exact against irrelevant, reread the guidelines with them, because they don't share a definition. ## Read intent as a product and its modifiers {#intent-in-product-search} Every judgment starts from a reading of what the query asks for. A shopper's **intent** is what they want to do. A widely cited taxonomy of web search splits intent into navigational (reach a particular site), informational (find information) and transactional (do something on the web). Its author cautions that "there is no assumption here that this intent can be inferred with any certitude from the query" ([Broder, 2002](https://sigir.org/files/forum/F2002/broder.pdf)). Product search reshapes these kinds. In product search, Tsagkias et al. write, navigational queries are "product serial numbers or names", and informational queries "typically map to a leaf in the inventory taxonomy or seek information about specific product attributes". They add that "there is no single taxonomy for eCommerce customer intent", because it depends on the store's business and inventory ([Tsagkias et al., 2020, section 2](https://sigir.org/wp-content/uploads/2020/06/p04.pdf)). WANDS writes intent into its labels. A query names a *target entity*, the kind of product, plus *modifiers*, the attributes the shopper asked for. Exact satisfies both. Partial "only matches the target entity of the query, but does not satisfy the modifiers" ([Chen et al., 2022, section 3.2](https://easychair.org/publications/preprint/wm4R)). The guidelines' worked example is a query for a kids' chair: "The product is a chair but not for kids", so it is partial ([guidelines](https://github.com/wayfair/WANDS/raw/3b74dcf4ba29ab8ff3e6a50b5b09fc627cb882b5/Product%20Search%20Relevance%20Annotation%20Guidelines.pdf)). Even with that rule, a broad query can be hard to judge: the authors' example is a shopper who wants an induction cooktop and types `cooktop` (Chen et al., section 1). The store's queries cover the range. `ines shag rug w003096809` and `kohler whitehaven farmhouse kitchen sink` name a product line or a model, which makes them navigational. `outdoor light fixtures` names a category. `clear shower curtain` and `48 inch kitchen hood` add a modifier to an entity. The store's queries also carry `query_class`, WANDS's *predicted* class for each query (Chen et al., Table 1). It is a guess at the entity, says nothing about modifiers, and isn't a judgment. Check how often it agrees with the products the judges called exact: Example `predicted-class-check` (Python 3.12.15 · numpy 2.5.4 · pandas 3.0.6), passing in Edition 2026.10.1: ```python import pandas as pd from lib.store import load_labels, load_products, load_queries products, queries, labels = load_products(), load_queries(), load_labels() exact = labels[labels["label"] == "exact"].join(products["product_class"], on="product_id") rows = [] for query_id, group in exact.groupby("query_id"): predicted = queries.at[query_id, "query_class"] classes = group["product_class"].str.split("|") # a product can have several classes rows.append({"query": queries.at[query_id, "query"], "query_class": predicted, "share of exact products in it": classes.apply(lambda names: predicted in names).mean(), "commonest class of exact products": group["product_class"].value_counts().index[0] or "(none)"}) check = pd.DataFrame(rows) print("queries:", len(check)) print("queries without a query_class:", check["query_class"].isna().sum()) print("queries where most exact products have the query_class:", (check["share of exact products in it"] > 0.5).sum()) missed = check[(check["share of exact products in it"] == 0) & check["query_class"].notna()] print("queries where no exact product has the query_class:", len(missed)) print() print(missed.drop(columns="share of exact products in it").to_string(index=False)) ``` Output: ```text queries: 80 queries without a query_class: 1 queries where most exact products have the query_class: 63 queries where no exact product has the query_class: 6 query query_class commonest class of exact products foutains with brick look Wallpaper Fountains 70s inspired furniture Wall Art (none) 7qt slow cooker Crock Pots & Slow Cookers Slow Cookers sheffield home bath set Bath Rugs & Mats Vanities luau string lights String Lights Holiday Lighting rustic twig Faux Plants and Trees Wall & Accent Mirrors ``` For 63 of the 80 queries, most exact products carry the predicted class. For 6, none does, and 1 query has no class at all. Read the queries the example lists before you count them as errors. `foutains with brick look` is classed as `Wallpaper`, while its exact products are fountains: the classifier went wrong. `7qt slow cooker` is a naming mismatch instead: the query classes and the product classes use different names for the same thing. **What to do:** read each query as an entity plus modifiers, and check that your ranker gets the entity right before you tune it for modifiers. Use `query_class` as a hint, for a feature or a filter you check against the judgments, and never as a label. When intent is ambiguous, as with `cooktop`, judge against the guidelines' reading and write that reading down. ## Grade relevance instead of yes or no {#graded-relevance} A binary label says relevant or not. A graded label, such as WANDS's exact, partial or irrelevant, says how relevant. The case for grades is that results differ in how useful they are: "Since all documents are not of equal relevance to their users, highly relevant documents should be identified and ranked first for presentation" ([Järvelin and Kekäläinen, 2002](https://www.fi.muni.cz/~xkohout7/Research/clanky_cizi/evaluation/p422-jarvelin.pdf)). In a store, a near miss is often a fair second choice. A shower curtain that isn't clear, which WANDS grades partial because it is the right kind of product without the attribute asked for ([Read intent as a product and its modifiers](#intent-in-product-search)), is closer to what the shopper wanted than a shower curtain hook. Binary labels force one line through the grades: count only exact as relevant, and the near miss is worth as little as the hook; count exact and partial, and it is worth as much as the clear curtain. Show both failures on the store's query `clear shower curtain`. The example builds three top-10 rankings from the query's judgments: ten exact products, ten shower curtains judged partial, and ten shower curtain hooks judged irrelevant. It scores each with nDCG@10 ([Score one ranking](#score-one-ranking)) under three labelings: binary with exact as relevant, binary with exact or partial as relevant, and the book's grades, 2 for exact, 1 for partial and 0 for irrelevant, used as gains. A **gain** is what a result at a given position adds to the score before the position discount. Example `binary-labels-hide-partial` (Python 3.12.15 · numpy 2.5.4 · pandas 3.0.6), passing in Edition 2026.10.1: ```python from lib.metrics import ndcg_at_k from lib.store import load_products, qrels classes = load_products()["product_class"] judgments = qrels()[246] # clear shower curtain: {product_id: grade} def first_ten(grade, product_class=None): """Ten products judged with this grade for the query, in product_id order.""" return [product_id for product_id in sorted(judgments) if judgments[product_id] == grade and product_class in (None, classes[product_id])][:10] rankings = {"clear curtains": first_ten(2), # exact "other curtains": first_ten(1, "Shower Curtains"), # partial: not clear "hooks": first_ten(0, "Shower Curtain Hooks and Accessories")} # irrelevant labelings = {"exact only": {2: 1, 1: 0, 0: 0}, # binary: relevant means exact "exact or partial": {2: 1, 1: 1, 0: 0}, # binary: relevant means exact or partial "grades": {2: 2, 1: 1, 0: 0}} # graded: the book's grades as gains print("exact products for this query:", sum(grade == 2 for grade in judgments.values())) print("exact products in the ideal top 10:", sorted(judgments.values(), reverse=True)[:10].count(2)) for labeling, gain in labelings.items(): gains = {product_id: gain[grade] for product_id, grade in judgments.items()} for name, ranking in rankings.items(): print(f"{name}, {labeling}: {ndcg_at_k(ranking, gains, 10):.3f}") ``` Output: ```text exact products for this query: 12 exact products in the ideal top 10: 10 clear curtains, exact only: 1.000 other curtains, exact only: 0.000 hooks, exact only: 0.000 clear curtains, exact or partial: 1.000 other curtains, exact or partial: 1.000 hooks, exact or partial: 0.000 clear curtains, grades: 1.000 other curtains, grades: 0.500 hooks, grades: 0.000 ``` With exact-only labels, the other curtains and the hooks score the same, 0.000 and 0.000. With exact-or-partial labels, the other curtains score 1.000, the same as the clear ones at 1.000. Only the grades separate all three: 1.000, 0.500 and 0.000. When the ideal top 10 is all exact, as it is here with 10 exact products in it, a ranking of ten partial products scores the ratio of the partial gain to the exact gain. **What to do:** we recommend graded judgments for product search, so a metric can credit a near miss without mistaking it for a match. Collect grades even if you report a binary metric too: you can always collapse grades to yes or no, but not the reverse. Use binary labels when a near miss is worthless to the user, as when the query is a model number and only that model will do. ### Choose the gains on purpose {#gains-are-a-choice} Grades are labels. Gains are what each label is worth in a metric, and that is a choice. This book uses each grade as its gain, so partial is worth half of exact. WANDS's own paper weights exact, partial and irrelevant as 1.0, 0.5 and 0 ([Chen et al., 2022, Table 3](https://easychair.org/publications/preprint/wm4R)), the same ratio and so the same nDCG. The Shopping Queries Dataset, known as ESCI, with "130 thousand unique queries and 2.6 million manually labeled (query,product) relevance judgements", grades differently. It separates a *substitute*, an item that "can be used as a functional substitute (e.g., fleece for a 'sweater' query)", from a *complement*, an item that "could be used in combination with an exact item", and sets "gain values of 1.0, 0.1, 0.01, and 0.0" for exact, substitute, complement and irrelevant ([Reddy et al., 2022, section 3.1](https://arxiv.org/abs/2206.06588)). WANDS's partial isn't the same as ESCI's substitute, but it is the nearest equivalent. Score the store's two test runs from chapter 1, [BM25](#rank-with-bm25)'s and semantic retrieval's rankings for every test query, under four schemes: the two binary labelings from [Grade relevance instead of yes or no](#graded-relevance), the book's grades, and a partial gain of 0.1 in the spirit of ESCI. The example also runs [chapter 3's permutation test](#permutation-test-on-queries) and [paired t-test](#paired-t-test-on-queries) on semantic retrieval's lead under each scheme, and prints both p-values. They are exploratory, so they describe each lead and establish none ([the book's rule](#when-a-difference-is-established)). The semantic run reads the recorded embeddings of the open model `Snowflake/snowflake-arctic-embed-m-v1.5` ([Search by meaning](#search-by-meaning)). Example `gain-choice-changes-ndcg` (Python 3.12.15 · bm25s 0.3.13 · numpy 2.5.4 · pandas 3.0.6 · scipy 1.18.1), passing in Edition 2026.10.1: ```python from lib.lexical import LexicalSearch from lib.metrics import evaluate from lib.semantic import SemanticSearch from lib.significance import compare from lib.store import load_products, load_queries, qrels queries, judgments = load_queries("test"), qrels("test") runs = {"bm25": LexicalSearch(load_products()).run(queries, k=100), "semantic": SemanticSearch().run(queries, k=100)} for name, run in runs.items(): grades = [judgments[query_id].get(product_id) for query_id, ranking in run.items() for product_id in ranking[:10]] print(f"partial share of the {name} top 10: {grades.count(1) / len(grades):.3f}") schemes = {"exact only": {2: 1, 1: 0, 0: 0}, "exact or partial": {2: 1, 1: 1, 0: 0}, "grades 2, 1, 0": {2: 2, 1: 1, 0: 0}, # the book's choice; WANDS's 1, 0.5, 0 gives the same nDCG "partial gain 0.1": {2: 1, 1: 0.1, 0: 0}} # close to the gains ESCI chose for its substitutes leads = 0 for scheme, gain in schemes.items(): gains = {query_id: {product_id: gain[grade] for product_id, grade in per_query.items()} for query_id, per_query in judgments.items()} scores = {name: evaluate(run, gains)["ndcg@10"] for name, run in runs.items()} test = compare(scores["semantic"], scores["bm25"]) # exploratory paired tests (chapter 3): no planned comparisons print(f"bm25, {scheme}: {scores['bm25'].mean():.3f}") print(f"semantic, {scheme}: {scores['semantic'].mean():.3f}") print(f"semantic minus bm25, {scheme}: {test['mean difference']:.3f}") print(f"permutation p, {scheme}: {test['permutation p']:.3f}") print(f"t-test p, {scheme}: {test['t-test p']:.3f}") print(f"tied queries, {scheme}: {test['ties']}") leads += test["mean difference"] > 0 print(f"schemes where semantic leads: {leads} of {len(schemes)}") ``` Output: ```text partial share of the bm25 top 10: 0.403 partial share of the semantic top 10: 0.380 bm25, exact only: 0.519 semantic, exact only: 0.648 semantic minus bm25, exact only: 0.130 permutation p, exact only: 0.000 t-test p, exact only: 0.002 tied queries, exact only: 3 bm25, exact or partial: 0.903 semantic, exact or partial: 0.983 semantic minus bm25, exact or partial: 0.080 permutation p, exact or partial: 0.010 t-test p, exact or partial: 0.026 tied queries, exact or partial: 19 bm25, grades 2, 1, 0: 0.725 semantic, grades 2, 1, 0: 0.830 semantic minus bm25, grades 2, 1, 0: 0.105 permutation p, grades 2, 1, 0: 0.001 t-test p, grades 2, 1, 0: 0.003 tied queries, grades 2, 1, 0: 2 bm25, partial gain 0.1: 0.564 semantic, partial gain 0.1: 0.688 semantic minus bm25, partial gain 0.1: 0.125 permutation p, partial gain 0.1: 0.000 t-test p, partial gain 0.1: 0.002 tied queries, partial gain 0.1: 2 schemes where semantic leads: 4 of 4 ``` Partial products fill 0.403 of BM25's top-10 places and 0.380 of semantic retrieval's, so the partial gain moves both scores a long way. BM25 scores 0.519 with exact-only labels and 0.903 with exact-or-partial labels. With the ESCI-like partial gain of 0.1, BM25 scores 0.564 and semantic retrieval 0.688. Semantic retrieval leads under 4 of 4 schemes, but the size of the lead changes with the choice: 0.130 with exact-only labels, 0.125 with ESCI-like gains, 0.105 with the grades and 0.080 with exact-or-partial labels. The example doesn't test the differences between those leads, so on 30 test queries treat them as possibly noise. Exact-or-partial labels also saturate, so they separate the rankers on fewer queries: 19 test queries tie under them, against 2 under the grades. **What to do:** we recommend linear gains, the grades 2, 1 and 0, as a default for product search. Choose a lower partial gain when a near miss is worth little to your shoppers. Whatever you choose, fix it before you compare rankers, report it with every score, and never compare nDCG computed with different gains. ## Read the label mix of each query {#label-mix-per-query} Gains set what each grade is worth, but not how many grades of each kind a query has. A judgment set isn't one population: each query has its own pool of judged products and its own mix of grades, and the mix decides what a perfect ranking looks like for that query. A broad query, one that names a category many products satisfy, has many exact products. A specific query, one with several modifiers, may have only a handful. An average over queries mixes these different problems, so read the mix before you average anything. The store's queries aren't a random draw from WANDS. Each was chosen to have 60 to 300 judged products, at least 3 exact and 20 partial judgments, and a predicted class no other chosen query has ([data card](../examples/data/wands/README.md)). Those limits set the floor and the ceiling of the counts that follow, which describe the sample, not WANDS as a whole: queries outside the limits never reach the store. Count the labels for each of the 80 queries, and group the queries by their share of exact products: Example `count-labels-per-query` (Python 3.12.15 · numpy 2.5.4 · pandas 3.0.6), passing in Edition 2026.10.1: ```python import pandas as pd from lib.store import load_labels, load_queries queries, labels = load_queries(), load_labels() counts = pd.crosstab(labels["query_id"], labels["label"])[["exact", "partial", "irrelevant"]] shares = counts.div(counts.sum(axis=1), axis=0).join(queries[["query", "split"]]) print(f"partial share of all judgments: {(labels['label'] == 'partial').mean():.3f}") print(f"median exact share: {shares['exact'].median():.3f}") print(f"median irrelevant share: {shares['irrelevant'].median():.3f}") for label, which in [("exact", "lowest"), ("exact", "highest"), ("irrelevant", "highest")]: query_id = shares[label].idxmin() if which == "lowest" else shares[label].idxmax() print(f"{which} {label} share: {shares.at[query_id, label]:.3f}") print(f"query with the {which} {label} share: {shares.at[query_id, 'query']}") print("queries with no irrelevant judgment:", (counts["irrelevant"] == 0).sum()) print("fewest exact products for a query:", counts["exact"].min()) print("most exact products for a query:", counts["exact"].max()) few = counts["exact"] < 10 print("queries with fewer than 10 exact products:", few.sum()) print("test queries with fewer than 10 exact products:", (few & (shares["split"] == "test")).sum()) print("queries with an exact share of 0.1 or less:", (shares["exact"] <= 0.1).sum()) print() print("exact share queries") bins = pd.cut(shares["exact"], [tenth / 10 for tenth in range(11)]) # no share is 0: each query has 3 or more exact for interval, number in bins.value_counts(sort=False).items(): print(f"{interval.left:.1f} to {interval.right:.1f} {number:>3} {'#' * number}".rstrip()) ``` Output: ```text partial share of all judgments: 0.610 median exact share: 0.207 median irrelevant share: 0.060 lowest exact share: 0.014 query with the lowest exact share: 48 in entry table with side by side drawer highest exact share: 0.749 query with the highest exact share: outdoor light fixtures highest irrelevant share: 0.776 query with the highest irrelevant share: podium with locking cabinet queries with no irrelevant judgment: 9 fewest exact products for a query: 3 most exact products for a query: 193 queries with fewer than 10 exact products: 23 test queries with fewer than 10 exact products: 7 queries with an exact share of 0.1 or less: 29 exact share queries 0.0 to 0.1 29 ############################# 0.1 to 0.2 10 ########## 0.2 to 0.3 12 ############ 0.3 to 0.4 11 ########### 0.4 to 0.5 9 ######### 0.5 to 0.6 1 # 0.6 to 0.7 7 ####### 0.7 to 0.8 1 # 0.8 to 0.9 0 0.9 to 1.0 0 ``` Partial is the commonest label: 0.610 of all judgments. The likely reason is how WANDS gathered its candidates, which [Know where the judgments came from](#pooled-judgments) covers. The exact share varies widely from query to query. The output's histogram is tallest at the low end: 29 queries have an exact share of 0.1 or less. The share runs from 0.014 for `48 in entry table with side by side drawer` to 0.749 for `outdoor light fixtures`, with a median of 0.207. The irrelevant share has a median of 0.060 but reaches 0.776 for `podium with locking cabinet`, and 9 queries have no irrelevant judgment at all. The number of exact products per query runs from 3, the sampling rule's floor, to 193. That spread changes what nDCG@10 ([Score one ranking](#score-one-ranking)) measures. Of the queries, 23 have fewer than 10 exact products, 7 of them test queries. For those, the ideal top 10 includes partial products, so a ranking can reach a perfect 1 with partial products in its top 10. For a broad query such as `outdoor light fixtures`, a perfect score needs ten exact products. The same nDCG@10 means a different thing for each. **What to do:** print the label mix per query before you average any metric, and look at both ends. Read a low score on a query with few exact products as a different problem from a low score on a broad query. Keep per-query scores next to every mean; [chapter 3](#read-per-query-differences) shows what averages hide. ## Treat clicks as biased evidence {#clicks-are-not-judgments} A store logs every click, and clicks cost nothing to collect, so they look like free judgments. They aren't. A click says that a shopper chose to look at a product. It doesn't say the product met the need, and whether the shopper clicks at all depends on where and how the product was shown. The evidence on how clicks mislead comes from web search. In a two-phase study, Joachims et al. tracked the eyes and clicks of Cornell undergraduates answering ten set questions with Google: 34 in the first phase, and 22 in the second, where some saw the top two results swapped or the whole list reversed. They concluded that "clicks are informative but biased" ([Joachims et al., 2005](https://www.cs.cornell.edu/people/tj/publications/joachims_etal_05a.pdf)). The first bias is **position bias**: people click higher-ranked results more, partly because they trust the ranking. The authors call it "a 'trust bias' which leads to more clicks on links ranked highly by Google, even if those abstracts are less relevant than other abstracts the user viewed". They also found that relative preferences, such as a clicked result over a skipped one above it, "are reasonably accurate on average". In a large experiment that reordered results on a web search engine, Craswell et al. found that a **cascade model** best explained position bias in the top ranks: users "view results from top to bottom" and leave at the first worthwhile document ([Craswell et al., 2008](https://www.microsoft.com/en-us/research/?p=156688)). **Presentation bias** goes beyond position. On a portion of Google's web search traffic, results with more attractive titles, meaning more of the query's words in bold, drew more clicks, even after controlling for position and for relevance as judged by people ([Yue et al., 2010](https://archives.iw3c2.org/www2010/proceedings/www/p1011.pdf)). In a store, images and prices are the nearest analogue to titles. That is our inference, not a measured result. WANDS has no click data, so the store can't measure these biases. You can still see what a click log would record. Take the store's judgments for `clear shower curtain`, and apply the cascade model in its strictest form: a shopper reads from the top, clicks the first exact product and leaves. This is a thought experiment with one fixed rule, not a model of real shoppers; [chapter 11](#simulate-interleaving-with-clicks) simulates clicks with probabilities. Example `cascade-clicks` (Python 3.12.15 · bm25s 0.3.13 · numpy 2.5.4 · pandas 3.0.6 · scipy 1.18.1), passing in Edition 2026.10.1: ```python from lib.lexical import LexicalSearch from lib.semantic import SemanticSearch from lib.store import load_products, qrels products = load_products() judgments = qrels()[246] # clear shower curtain: {product_id: grade} rankings = {"bm25": list(LexicalSearch(products).search("clear shower curtain", k=10).index), "semantic": list(SemanticSearch().search(246, k=10).index)} for name, ranking in rankings.items(): grades = [judgments.get(product_id, 0) for product_id in ranking] rank = grades.index(2) + 1 # cascade: read from the top, click the first exact product, stop print(f"{name} grades, ranks 1 to 10: {' '.join(map(str, grades))}") print(f"{name} clicked rank: {rank}") print(f"{name} clicked product: {products.at[ranking[rank - 1], 'name']}") print(f"{name} exact products shown below the click: {grades[rank:].count(2)}") ``` Output: ```text bm25 grades, ranks 1 to 10: 1 2 2 2 2 1 1 0 0 0 bm25 clicked rank: 2 bm25 clicked product: woolley vinyl single shower curtain bm25 exact products shown below the click: 3 semantic grades, ranks 1 to 10: 2 2 1 1 2 1 2 2 1 1 semantic clicked rank: 1 semantic clicked product: woolley vinyl solid color single shower curtain semantic exact products shown below the click: 4 ``` Under [BM25](#rank-with-bm25)'s ranking, the shopper clicks rank 2, `woolley vinyl single shower curtain`, and never reaches the 3 exact products below it. In the click log, those exact products look the same as BM25's ranks 8 to 10, shower curtain hooks judged irrelevant ([Rank products with BM25](#rank-with-bm25)), which nobody clicked either. Under [semantic retrieval](#search-by-meaning)'s ranking, the click lands at rank 1, on `woolley vinyl solid color single shower curtain`. Two rankings from the same catalog produce two click logs that credit different products. A click log records what the ranker showed first as much as what the shopper wanted. WANDS's authors give the same reason, positional bias, for not building their dataset from logs ([Check the results against the judgments](#check-against-judgments)). Clicks still carry information. In WANDS, 62.47% of the candidates taken from click logs were judged exact, against 30.95% of those taken from open-source ranking systems ([Chen et al., 2022, Table 4](https://easychair.org/publications/preprint/wm4R)). Correcting clicks for bias is a research field of its own, unbiased learning to rank. On a large search dataset released by Baidu, those methods improved click prediction but "struggle to consistently improve ranking performance" as judged by experts ([Hager et al., 2024](https://arxiv.org/abs/2404.02543)). **What to do:** use judgments, not clicks, as the reference for offline evaluation. Use clicks as relative evidence, a clicked product over a skipped one ranked above it, and never read a missing click as a sign that a product is irrelevant. If you train on clicks, check the result against judgments. ## Know where the judgments came from {#pooled-judgments} Nobody judges every product for every query. Test collections judge a **pool** instead. In TREC, the Text REtrieval Conference's shared evaluations, the pool for a topic is "the union of the set of X top-retrieved documents for that topic by each participant (where X is usually set at 100)", and every document outside it counts as not relevant. Voorhees argues that "retrieval systems can be compared fairly by assuming that all unjudged documents are not relevant", because pools hold enough of the relevant documents, and notes that pooling "cannot produce reusable test collections for arbitrarily large document sets" ([Voorhees, 2007](https://cs.uwaterloo.ca/~brecht/courses/854-Experimental-Performance-Evaluation-2017/readings/trec-cacm-2007.pdf)). Zobel tested the fairness claim: with pooled judgments, "the measured relative performance of systems appears to be reliable, but that recall is overestimated: it is likely that many relevant documents have not been found" ([Zobel, 1998](https://www.cs.cornell.edu/courses/cs574/2002fa/assignments.org/zobel-sigir98.pdf)). WANDS built its pool in two steps ([Chen et al., 2022, sections 4.2 and 4.3](https://easychair.org/publications/preprint/wm4R)). The first took the products shoppers had clicked or added to their carts, plus the results of an open-source lexical search engine, Solr, and of a neural retrieval system. The second mined the whole catalog again with the same two systems, and added every product in the class that a query classification model predicted for the query. The second step produced most of WANDS's labels, and mostly near misses: 4.12% of its labels were exact and 63.99% partial (Table 5). Partial is the commonest label in the store's sample too ([Read the label mix](#label-mix-per-query)). The authors argue that completeness matters for product search, because "incompleteness in Product Search dataset does negatively impact the discriminative power of the evaluation" (section 3.1). A pool belongs to a query. A product without a judgment for a query is **unjudged** for it, and a product judged for one query can be unjudged for a similar one. The store has two such queries: the train query `trundle daybed` and the test query `full bed with trundle`. Count the products judged exact for the first that have no judgment for the second, and see how the judges graded the ones that do: Example `trundle-daybeds-unjudged` (Python 3.12.15 · numpy 2.5.4 · pandas 3.0.6), passing in Edition 2026.10.1: ```python from collections import Counter from lib.store import load_products, load_queries, qrels products, queries, judgments = load_products(), load_queries(), qrels() LABELS = {2: "exact", 1: "partial", 0: "irrelevant"} daybeds = sorted(product_id for product_id, grade in judgments[433].items() if grade == 2) full_bed = judgments[436] # {product_id: grade} for "full bed with trundle" print(f"433: {queries.at[433, 'query']} ({queries.at[433, 'split']})") print(f"436: {queries.at[436, 'query']} ({queries.at[436, 'split']})") print("exact for trundle daybed:", len(daybeds)) unjudged = [product_id for product_id in daybeds if product_id not in full_bed] print("of them unjudged for full bed with trundle:", len(unjudged)) print("of them judged for full bed with trundle:", len(daybeds) - len(unjudged)) judged = Counter(LABELS[full_bed[product_id]] for product_id in daybeds if product_id in full_bed) for label in LABELS.values(): print(f"judged {label}: {judged[label]}") print("first unjudged daybed:", products.at[unjudged[0], "name"]) print("second unjudged daybed:", products.at[unjudged[1], "name"]) ``` Output: ```text 433: trundle daybed (train) 436: full bed with trundle (test) exact for trundle daybed: 113 of them unjudged for full bed with trundle: 77 of them judged for full bed with trundle: 36 judged exact: 5 judged partial: 31 judged irrelevant: 0 first unjudged daybed: daybed with trundle second unjudged daybed: pattonsburg daybed with trundle ``` Of the 113 products judged exact for `trundle daybed`, 77 have no judgment for `full bed with trundle`, among them `daybed with trundle` and `pattonsburg daybed with trundle`. Of the 36 that do, the judges called 5 exact, 31 partial and 0 irrelevant. By our reading of the guidelines' rule, a daybed with a trundle is often the right kind of bed without the size the query asks for. If the unjudged daybeds resemble the judged ones, most of them would be partial too, yet a metric gives each one grade 0. Zobel warns that pooled judgments can "underestimate the performance of systems that did not contribute to the pool" (Zobel, 1998). That makes judgments fairest to rankers like the systems that built the pool: for WANDS, shoppers' clicks, a lexical engine, a neural retriever and a query classifier. It is a tendency, not a law. In the store, BM25, the ranker most like WANDS's lexical engine, has the lower judged@10 of the two rankers ([Count unjudged products](#unjudged-results)). Our guess is that the store's BM25 reads different fields than the pool's engine did, so it ranks different products high. **What to do:** find out where your pool came from before you trust a comparison. When you evaluate a ranker unlike the systems that built it, expect it to surface products nobody judged, and plan to judge them. ## Count unjudged products before you compare scores {#unjudged-results} A product nobody judged for a query is [unjudged](#pooled-judgments): its grade is unknown, not 0. Standard metrics can't tell the difference. They "make no distinction in pooled collections between documents that are explicitly judged as nonrelevant and documents that are assumed to be nonrelevant because they are unjudged", and they become unreliable when judgments are very incomplete ([Buckley and Voorhees, 2004](https://tsapps.nist.gov/publication/get_pdf.cfm?pub_id=150469)). Two diagnostics are standard in evaluation tools ([ir-measures](https://ir-measur.es/en/latest/measures.html)). **Judged@k** is the share of the top k that has a judgment of any grade: ```text judged@k = (number of the top k products that have a judgment for the query) / k ``` The second drops the unjudged products before scoring, so the judged ones move up: ir-measures offers it as the `judged_only` option, and we call the result a **condensed** list. Buckley and Voorhees's bpref measure works in the same spirit: it uses judged documents only, and they found it more robust to incomplete judgments. In [chapter 1](#score-every-test-query), [BM25](#rank-with-bm25)'s mean judged@10 on the test queries is 0.927, against 0.990 for [semantic retrieval](#search-by-meaning). One property of the store's data keeps those numbers high: its catalog is almost all judged products, 11,917 of 12,000 judged for at least one query ([Load the store's data](#store-data)). Expect a full catalog to put more unjudged products in the top 10s. Build judged@10 by hand, check it against `lib.metrics`, and list where BM25's unjudged products are: Example `unjudged-in-bm25-top-10` (Python 3.12.15 · bm25s 0.3.13 · numpy 2.5.4 · pandas 3.0.6 · scipy 1.18.1), passing in Edition 2026.10.1: ```python import pandas as pd from lib.lexical import LexicalSearch from lib.metrics import evaluate from lib.store import load_products, load_queries, qrels products, queries, judgments = load_products(), load_queries("test"), qrels("test") run = LexicalSearch(products).run(queries, k=100) def judged_at_k(ranking, judged, k): """The share of the top k that has a judgment of any grade.""" return sum(product_id in judged for product_id in ranking[:k]) / k rows = [] for query_id, ranking in run.items(): unjudged = [product_id for product_id in ranking[:10] if product_id not in judgments[query_id]] rows.append({"query": queries.at[query_id, "query"], "judged@10": judged_at_k(ranking, judgments[query_id], 10), "unjudged": len(unjudged), "first unjudged product": products.at[unjudged[0], "name"][:45] if unjudged else ""}) table = pd.DataFrame(rows) print(f"mean judged@10: {table['judged@10'].mean():.3f}") library = evaluate(run, judgments)["judged@10"] agree = sum(abs(row["judged@10"] - library[query_id]) < 1e-12 for query_id, row in zip(run, rows)) print(f"queries where the hand-built judged@10 equals lib.metrics: {agree} of {len(rows)}") print("unjudged products in the top 10s:", table["unjudged"].sum()) print("test queries with an unjudged product in the top 10:", (table["judged@10"] < 1).sum()) sleeper = list(judgments[244].values()) # pull out sleeper loveseat print("judgments for pull out sleeper loveseat:", len(sleeper)) print("irrelevant judgments for pull out sleeper loveseat:", sleeper.count(0)) print() print(table[table["judged@10"] < 1].sort_values("judged@10").to_string(index=False)) ``` Output: ```text mean judged@10: 0.927 queries where the hand-built judged@10 equals lib.metrics: 30 of 30 unjudged products in the top 10s: 22 test queries with an unjudged product in the top 10: 7 judgments for pull out sleeper loveseat: 143 irrelevant judgments for pull out sleeper loveseat: 0 query judged@10 unjudged first unjudged product outdoor light fixtures 0.2 8 60 watt equivalent e12 led candle light bulb beach blue headboard 0.5 5 lemon frosting indoor/outdoor , removable cov pull out sleeper loveseat 0.6 4 country oxford 87.75 '' rolled arm sleeper full bed with trundle 0.8 2 abel staircase twin over full bunk bed with t whimsical solar lights 0.9 1 luciole 0.12-watt led solar power dusk to daw bistro table and chairs 0.9 1 drayton patio 5 piece bistro set wooden chair outdoor 0.9 1 osterley 31.5 '' w writing desk and chair set ``` The hand-built judged@10 equals `lib.metrics` on 30 of 30 test queries, with a mean of 0.927. Unjudged products fill 22 of BM25's top-10 places, spread over 7 queries. By our reading, some are plainly irrelevant: light bulbs for `outdoor light fixtures`, an outdoor pillow for `beach blue headboard`. Others look like near misses or matches: a bunk bed with a trundle for `full bed with trundle`, a bistro set for `bistro table and chairs`, a sleeper sofa for `pull out sleeper loveseat`, which by the entity-and-modifier rule is probably a near miss. For that last query, 0 of the 143 judged products are irrelevant. Now score both rankers with condensed lists: Example `condensed-ndcg` (Python 3.12.15 · bm25s 0.3.13 · numpy 2.5.4 · pandas 3.0.6 · scipy 1.18.1), passing in Edition 2026.10.1: ```python from lib.lexical import LexicalSearch from lib.metrics import evaluate from lib.semantic import SemanticSearch from lib.significance import compare from lib.store import load_products, load_queries, qrels queries, judgments = load_queries("test"), qrels("test") runs = {"bm25": LexicalSearch(load_products()).run(queries, k=100), "semantic": SemanticSearch().run(queries, k=100)} standard, condensed = {}, {} for name, run in runs.items(): # condensed: drop every unjudged product, so the judged ones move up into the top 10 judged_only = {query_id: [product_id for product_id in ranking if product_id in judgments[query_id]] for query_id, ranking in run.items()} standard[name] = evaluate(run, judgments)["ndcg@10"] condensed[name] = evaluate(judged_only, judgments)["ndcg@10"] print(f"{name} ndcg@10: {standard[name].mean():.3f}") print(f"{name} condensed ndcg@10: {condensed[name].mean():.3f}") gain = condensed["bm25"] - standard["bm25"] # bm25's change, per test query print(f"bm25 ndcg@10 for outdoor light fixtures: {standard['bm25'][133]:.3f}") print(f"bm25 condensed ndcg@10 for outdoor light fixtures: {condensed['bm25'][133]:.3f}") print(f"share of bm25's gain from outdoor light fixtures and beach blue headboard: {gain[[133, 46]].sum() / gain.sum():.3f}") lead, condensed_lead = standard["semantic"] - standard["bm25"], condensed["semantic"] - condensed["bm25"] change = compare(condensed_lead, lead) # is the change in the lead more than noise? (exploratory) print(f"semantic minus bm25: {lead.mean():.3f}") print(f"semantic minus bm25, condensed: {condensed_lead.mean():.3f}") print(f"change in the lead: {change['mean difference']:.3f}") print(f"95% interval for the change: {change['ci low']:.3f} to {change['ci high']:.3f}") print(f"t-test p for the change: {change['t-test p']:.3f}") print(f"permutation p for the change: {change['permutation p']:.3f}") condensed_test = compare(condensed["semantic"], condensed["bm25"]) # the condensed lead itself (exploratory) print(f"t-test p for the condensed lead: {condensed_test['t-test p']:.3f}") print(f"permutation p for the condensed lead: {condensed_test['permutation p']:.3f}") ``` Output: ```text bm25 ndcg@10: 0.725 bm25 condensed ndcg@10: 0.778 semantic ndcg@10: 0.830 semantic condensed ndcg@10: 0.834 bm25 ndcg@10 for outdoor light fixtures: 0.286 bm25 condensed ndcg@10 for outdoor light fixtures: 1.000 share of bm25's gain from outdoor light fixtures and beach blue headboard: 0.612 semantic minus bm25: 0.105 semantic minus bm25, condensed: 0.056 change in the lead: -0.049 95% interval for the change: -0.138 to -0.015 t-test p for the change: 0.073 permutation p for the change: 0.036 t-test p for the condensed lead: 0.011 permutation p for the condensed lead: 0.009 ``` Condensed, BM25 goes from 0.725 to 0.778 and semantic retrieval from 0.830 to 0.834, so semantic retrieval's lead goes from 0.105 to 0.056. Of BM25's gain, a share of 0.612 comes from the two queries whose unjudged products are plainly irrelevant, `outdoor light fixtures` and `beach blue headboard`. For `outdoor light fixtures`, BM25 goes from 0.286 to 1.000: once the bulbs drop out, the judged lights below them move up. So much of BM25's gain comes from forgiving bad results, not from undoing a penalty on good ones. Whether the change in the lead is more than noise is unsettled, and both checks here are exploratory, so neither establishes anything ([the book's rule](#when-a-difference-is-established)). For the change, [chapter 3's paired t-test](#paired-t-test-on-queries) gives p = 0.073, its [permutation test](#permutation-test-on-queries) gives 0.036, and the 95% [bootstrap interval](#report-the-size-of-a-gap) runs from -0.138 to -0.015. With the change carried by a few queries, treat it as possibly noise. For the condensed lead itself, the permutation test gives p = 0.009 and the t-test 0.011. Neither score is the truth. The standard score punishes unjudged products that may be good, and the condensed score hides unjudged products that are bad. **What to do:** report judged@10 next to every nDCG@10, and compute condensed scores as a diagnostic, never as the headline. When two rankers' judged@10 differ, have the unjudged products in their top 10s judged before you compare them. That is the dependable cure, and on this sample it is a small job: 22 judgments for BM25's test run. On a full catalog, expect more. ## What goes wrong in production {#what-relevance-means-in-production} **Judgments go stale.** A store's catalog and shoppers keep changing, and relevance depends on more than the query: Tsagkias et al. name user, time, query and context, and note that "The query jacket has a different set of relevant documents in winter than summer" ([Tsagkias et al., 2020, section 3.1.1](https://sigir.org/wp-content/uploads/2020/06/p04.pdf)). Products enter the catalog unjudged, and products that leave it stay in the judgment set. Track [judged@10](#unjudged-results) for your production ranker over time, and judge again when it falls. **Click labels confirm the ranking you have.** A ranker trained on clicks logged under the current ranking learns that ranking's order along with the shoppers' preferences. Keep human judgments as the reference that clicks are checked against ([Treat clicks as biased evidence](#clicks-are-not-judgments)). **Guidelines shape the judgments.** WANDS's annotators needed four months to reach near-perfect agreement ([Chen et al., 2022, Table 2](https://easychair.org/publications/preprint/wm4R)). Small choices in the guidelines shape the whole set. WANDS told annotators to "ignore the mis-spellings and proceed to annotate the example as-is" ([guidelines](https://github.com/wayfair/WANDS/raw/3b74dcf4ba29ab8ff3e6a50b5b09fc627cb882b5/Product%20Search%20Relevance%20Annotation%20Guidelines.pdf)), so its judgments assume a reader who understands `ligth bulb`. A ranker that can't will score low on such queries, which is fair to the shopper. [Chapter 8](#correct-spelling-from-catalog) builds a spelling corrector and shows what it still gets wrong. **Your ranker sees less than your judges.** The judges saw images and product pages. The store's rankers read text without the attributes ([chapter 5 tests adding them](#index-attributes-as-a-field)); any large language model (LLM) asked to judge from the same text sees the same. Some disagreement between such a system and the human judgments comes from that gap, not from the system. Check which fields hold the attributes your judges rewarded before you blame the ranker. **Filling holes with a model still needs human judgments.** Gienapp et al. fine-tune a small classifier for each query on that query's human judgments and use it to judge the unjudged products. On TREC Robust04 and the TREC Deep Learning collections of 2019 and 2020, they report that "As little as 128 initial human judgments per topic suffice to improve the comparability of models", with more reliability than prompted LLM judges ([Gienapp et al., 2025](https://arxiv.org/abs/2510.04633)). Soboroff's advice on LLMs is blunter: "don't use LLMs to create relevance judgments for TREC-style evaluations" ([Soboroff, 2025](https://arxiv.org/abs/2409.15133)). [Chapter 4](#read-llm-error-direction) measures LLM judgments against the store's human ones before trusting them. **What to do:** date your judgments, track judged@10 in production, and budget for judging again. ## Decisions this chapter made {#what-relevance-means-decisions} Each choice below has a reason, and each is a default that later chapters keep unless a measurement on train queries says otherwise. - **Human judgments are the reference.** Every headline score in this book is computed on them, including scores for rankers built with LLMs. Clicks are evidence about shoppers, biased by position and presentation, and never serve as labels. - **Grades 2, 1 and 0 are the gains,** so partial is worth half of exact, the same ratio as WANDS's own 1.0, 0.5 and 0. Binary metrics count exact only, so **relevant** means exact unless a sentence says otherwise. - **One grade per pair.** When WANDS lists a pair more than once with different labels, the sample keeps the lower grade, which errs against the ranker. - **`query_class` is a hint,** never a label and never a measure of intent. Some of its values are wrong. - **Unjudged counts as grade 0, with judged@10 beside every nDCG@10.** Condensed scores are a diagnostic only. The cure for unjudged products in a top 10 is to have them judged. - **Rankers read the product text, not the attributes.** [Chapter 5](#index-attributes-as-a-field) tests adding them as a field and keeps them out of the BM25 text. The judges also saw images and product pages, so some exact products give a text-only ranker little to match. - **Read any judgment, tune on train.** Reading test judgments changes no ranker; choosing a fix from them would. # Measuring search By the end of this chapter you can compute the standard ranking metrics from scratch, check them against scikit-learn, choose one before you compare rankers, and tell whether the gap between two rankers on your test queries is established or could be chance. [Chapter 1](#score-every-test-query) left the store, [the furniture and home-goods retailer](#store-data) whose search the book works on, with four numbers. On its test queries, BM25 scores an nDCG@10 of 0.725, semantic retrieval 0.830, the hybrid 0.812 and the reranked hybrid 0.798. Semantic retrieval ranks by recorded embeddings from the open model `Snowflake/snowflake-arctic-embed-m-v1.5`, and the reranker is the recorded cross-encoder `Alibaba-NLP/gte-reranker-modernbert-base`. Chapter 1's decision rests on the order of the four scores: semantic retrieval alone is the ranker to beat. Before you trust it, two questions need answers. Do the metrics measure what matters to a shopper, and are they computed correctly? And are the gaps real, or what 30 queries produce by chance? This chapter answers both. It ends with the rule the book uses from here on to call a difference established, and an evaluation harness that applies it. ## Count the relevant products in the top k {#count-relevant-in-top-k} Most ranking metrics start from one question: how many of the results a shopper sees are relevant? **Precision at k**, written precision@k, is the share of the top k results that are relevant. **Recall at k**, recall@k, is the share of all relevant products that appear in the top k. In the textbook's words, precision "is the fraction of retrieved documents that are relevant" and recall "is the fraction of relevant documents that are retrieved", and raising one usually costs the other ([Manning et al., 2008, section 8.3](https://nlp.stanford.edu/IR-book/html/htmledition/evaluation-of-unranked-retrieval-sets-1.html)). Both need a yes-or-no idea of relevant. In this book a product is relevant to a query when its [judgment](#store-data) is exact, grade 2, and a product nobody judged for the query counts as not relevant. [BM25](#rank-with-bm25)'s top 10 for the store's test query `clear shower curtain` mixes clear curtains, curtains that aren't clear, and [shower curtain hooks](#check-against-judgments). Count the exact products by hand, compare with `lib.metrics`, then work out the best score each test query could reach: Example `precision-recall-by-hand` (Python 3.12.15 · bm25s 0.3.13 · numpy 2.5.4 · pandas 3.0.6 · scipy 1.18.1), passing in Edition 2026.10.1: ```python import pandas as pd from lib.lexical import LexicalSearch from lib.metrics import precision_at_k, recall_at_k from lib.store import load_products, load_queries, qrels queries, judgments = load_queries("test"), qrels("test") ranking = list(LexicalSearch(load_products()).search(queries.at[246, "query"], k=10).index) # clear shower curtain exact = {product_id for product_id, grade in judgments[246].items() if grade == 2} found = sum(product_id in exact for product_id in ranking[:10]) print("exact in the top 10:", found) print("exact products judged:", len(exact)) print(f"precision@10 by hand: {found / 10:.3f}") print(f"recall@10 by hand: {found / len(exact):.3f}") print(f"precision@10 from lib.metrics: {precision_at_k(ranking, judgments[246], 10):.3f}") print(f"recall@10 from lib.metrics: {recall_at_k(ranking, judgments[246], 10):.3f}") # the best any ranking can do: a top 10 that holds as many exact products as the query has exact_counts = pd.Series({query_id: sum(grade == 2 for grade in per_query.values()) for query_id, per_query in judgments.items()}) best_precision = exact_counts.clip(upper=10) / 10 best_recall = exact_counts.clip(upper=10) / exact_counts print() print("queries where precision@10 can't reach 1:", (best_precision < 1).sum()) print("fewest exact products for a query:", exact_counts.min()) print("queries with that few:", (exact_counts == exact_counts.min()).sum()) print(f"best precision@10 for those queries: {best_precision.min():.3f}") print("queries where recall@10 can't reach 1:", (best_recall < 1).sum()) most = exact_counts.idxmax() print("query with the most exact products:", queries.at[most, "query"]) print("its exact products:", exact_counts[most]) print(f"its best recall@10: {best_recall[most]:.3f}") ``` Output: ```text exact in the top 10: 4 exact products judged: 12 precision@10 by hand: 0.400 recall@10 by hand: 0.333 precision@10 from lib.metrics: 0.400 recall@10 from lib.metrics: 0.333 queries where precision@10 can't reach 1: 7 fewest exact products for a query: 4 queries with that few: 4 best precision@10 for those queries: 0.400 queries where recall@10 can't reach 1: 22 query with the most exact products: outdoor light fixtures its exact products: 170 its best recall@10: 0.059 ``` 4 of the top 10 are exact, so precision@10 is 0.400. The judges marked 12 products exact for this query, so recall@10 is 0.333. `precision_at_k` and `recall_at_k` give the same values. Neither metric has the same ceiling for every query. A query with fewer than 10 exact products can't fill its top 10 with them: 7 test queries are like that, and the 4 with only 4 exact products can reach a precision@10 of 0.400 at best. A query with more than 10 exact products can't reach full recall in 10 results: 22 test queries are like that. The query `outdoor light fixtures` has 170 exact products, so its recall@10 tops out at 0.059. A mean over the test queries mixes queries that can score 1 with queries that can't. Recall has a second problem: its denominator counts only the exact products the judges found. WANDS judged candidates gathered for each query, not the whole catalog ([chapter 1](#store-data)), so some exact products were never judged. Zobel found that judgments gathered this way overestimate recall ([Know where the judgments came from](#pooled-judgments)). **What to do:** use precision@10 when you care how much of the first page is right, and read it next to each query's ceiling. Don't make recall@10 a headline in product search, where a query can have more than a hundred relevant products, as `outdoor light fixtures` has. We recommend recall at a deep cutoff, such as the top 100, for one job only: checking that retrieval passes enough relevant products to a reranker. Even then, read it as an overestimate. ## Score the first relevant result with reciprocal rank {#score-the-first-relevant-result} Precision counts a whole page, but some shoppers want one product: a replacement part, or a model they already know. For them, what matters is how far down the first good result sits. **Reciprocal rank** (RR) scores a query as 1 divided by the position of the first relevant result, or 0 if there is none, and its mean over queries is the **mean reciprocal rank** (MRR). TREC, the Text REtrieval Conference, is a series of shared evaluations of search systems. MRR was the official measure of its question-answering track at TREC-8, where each question "received a score equal to the reciprocal of the rank at which the first correct response was returned, or 0 if none of the five responses contained a correct answer". The track chose it partly because it "is bounded between 0 and 1, inclusive, and averages well" ([Voorhees, 1999](https://trec.nist.gov/pubs/trec8/papers/qa_report.ps)). Whether you may average RR is disputed. Fuhr points out that in RR "the difference between ranks 1 and 2 is the same as that between ranks 2 and ∞". So RR isn't an interval scale, a scale on which equal steps mean equal differences, and in his view its mean shouldn't be computed ([Fuhr, 2017, section 2.1](https://sigir.org/wp-content/uploads/2018/01/p032.pdf)). Sakai sees no reason to forbid it ([Sakai, 2020](https://sigir.org/wp-content/uploads/2020/06/p14.pdf)), and Moffat argues that such metrics "are more meaningful in their current form than in the proposed 'intervalized' versions" ([Moffat, 2022](https://arxiv.org/abs/2207.03103)). We use MRR with that caveat: it is a coarse summary. Write RR by hand, score [BM25](#rank-with-bm25)'s ranking for the store's query `clear shower curtain` with two thresholds for relevant, then compute MRR for BM25 and [semantic retrieval](#search-by-meaning) over the test queries: Example `reciprocal-rank-by-hand` (Python 3.12.15 · bm25s 0.3.13 · numpy 2.5.4 · pandas 3.0.6 · scipy 1.18.1), passing in Edition 2026.10.1: ```python import pandas as pd from lib.lexical import LexicalSearch from lib.metrics import evaluate from lib.semantic import SemanticSearch from lib.store import load_products, load_queries, qrels def reciprocal_rank(ranking, judgments, relevant=2): """1 / the position of the first relevant product, or 0 if the ranking has none.""" for position, product_id in enumerate(ranking, start=1): if judgments.get(product_id, 0) >= relevant: return 1 / position return 0.0 queries, judgments = load_queries("test"), qrels("test") runs = {"bm25": LexicalSearch(load_products()).run(queries, k=100), "semantic": SemanticSearch().run(queries, k=100)} ranking = runs["bm25"][246] # clear shower curtain print(f"rr, exact counts as relevant: {reciprocal_rank(ranking, judgments[246]):.3f}") print(f"rr, partial or better counts: {reciprocal_rank(ranking, judgments[246], relevant=1):.3f}") for name, run in runs.items(): rr = pd.Series({query_id: reciprocal_rank(run[query_id], judgments[query_id]) for query_id in judgments}) print(f"{name} mrr: {rr.mean():.3f}") print(f"{name} mrr from lib.metrics: {evaluate(run, judgments)['rr'].mean():.3f}") print(f"{name} queries with rr of 1:", (rr == 1).sum()) ``` Output: ```text rr, exact counts as relevant: 0.500 rr, partial or better counts: 1.000 bm25 mrr: 0.650 bm25 mrr from lib.metrics: 0.650 bm25 queries with rr of 1: 16 semantic mrr: 0.787 semantic mrr from lib.metrics: 0.787 semantic queries with rr of 1: 22 ``` BM25's first result for `clear shower curtain` is a partial match and its second is exact, so RR is 0.500 when only exact products count, and 1.000 when partial ones count too. Whenever you quote RR, say which grades count. Over the test queries, BM25's MRR is 0.650 and semantic retrieval's is 0.787, matching the `rr` column of `lib.metrics.evaluate`. That column looks for the first exact product anywhere in the [run](#score-every-test-query), not only in the top 10. RR is 1 for 22 test queries under semantic retrieval, and for 16 under BM25. On those queries RR sees nothing below the first result: a ranker that puts one exact product first and nine irrelevant ones after it scores the same as a perfect top 10. **What to do:** report MRR as the headline only when shoppers need one right product, as for a model number or a named product such as `kohler whitehaven farmhouse kitchen sink`. For queries where shoppers browse a page of options, such as `wooden chair outdoor`, use nDCG@10 as the headline and report MRR beside it. ## Build nDCG from discounted gains {#build-ndcg-from-dcg} [Chapter 1](#score-one-ranking) scored one ranking with nDCG@10, which, unlike precision and RR, uses the grades. Each result's grade is its gain, a discount that grows with position shrinks it, and the sum is the **discounted cumulative gain** (DCG). **Normalized DCG** (nDCG) divides that by the DCG of the ideal ranking. Two of its parts come in several versions that change the number. The first is the ideal ranking. Järvelin and Kekäläinen, who introduced DCG, base it on "the recall base of the search topic rather than on the result of some IR technique" ([Järvelin and Kekäläinen, 2002](https://doi.org/10.1145/582415.582418)): the best order of every product judged for the query, not only of the products a ranker returned. The second is the discount. They proposed that "a simple way of discounting with this requirement is to divide the document score by the log of its rank", and versions differ in which log, from which rank. The formula in `lib.metrics`, where grade(i) is the grade of the product at position i: ```text DCG@k = sum over positions i from 1 to k of grade(i) / log2(i + 1) nDCG@k = DCG@k / DCG@k of the ideal ranking ``` This isn't the only version in use. Järvelin and Kekäläinen left the ranks below the log's base undiscounted, so with base 2 the first two positions count in full. Burges et al. used an exponential gain, 2^grade − 1, with a log(1 + rank) discount ([Burges et al., 2005](https://icml.cc/Conferences/2005/proceedings/papers/012_LearningToRank_BurgesEtAl.pdf)). Evaluation tools differ as well: the ir-measures documentation lists different nDCG options for pytrec_eval and for gdeval ([ir-measures](https://ir-measur.es/en/latest/measures.html)). So whenever you quote an nDCG, name the variant. Write DCG and nDCG from scratch and check them against `lib.metrics` on the store's BM25, semantic and [hybrid](#fuse-with-rrf) runs from chapter 1. Then look at one test query with a perfect score: Example `ndcg-from-scratch` (Python 3.12.15 · bm25s 0.3.13 · numpy 2.5.4 · pandas 3.0.6 · scipy 1.18.1), passing in Edition 2026.10.1: ```python import math from lib.fusion import reciprocal_rank_fusion from lib.lexical import LexicalSearch from lib.metrics import ndcg_at_k from lib.semantic import SemanticSearch from lib.store import load_products, load_queries, qrels def dcg(grades, k): """Each grade divided by log2 of its position plus one, added up over the top k.""" return sum(grade / math.log2(position + 1) for position, grade in enumerate(grades[:k], start=1)) def ndcg(ranking, judgments, k=10): """DCG of the ranking over DCG of the best order of every judged product (unjudged counts as 0).""" ideal = dcg(sorted(judgments.values(), reverse=True), k) return dcg([judgments.get(product_id, 0) for product_id in ranking], k) / ideal queries, judgments = load_queries("test"), qrels("test") bm25 = LexicalSearch(load_products()).run(queries, k=100) semantic = SemanticSearch().run(queries, k=100) runs = {"bm25": bm25, "semantic": semantic, "hybrid": reciprocal_rank_fusion([bm25, semantic])} gaps = [abs(ndcg(run[query_id], judgments[query_id]) - ndcg_at_k(run[query_id], judgments[query_id], 10)) for run in runs.values() for query_id in judgments] print("rankings checked:", len(gaps)) print(f"largest difference from lib.metrics: {max(gaps):.3f}") ranking, per_query = bm25[461], judgments[461] # luau string lights print() print("query:", queries.at[461, "query"]) print("grades in the top 10:", [per_query.get(product_id, 0) for product_id in ranking[:10]]) print("best possible grades:", sorted(per_query.values(), reverse=True)[:10]) print("exact products judged:", sum(grade == 2 for grade in per_query.values())) print("exact in the top 10:", sum(per_query.get(product_id, 0) == 2 for product_id in ranking[:10])) print(f"ndcg@10: {ndcg(ranking, per_query):.3f}") ``` Output: ```text rankings checked: 90 largest difference from lib.metrics: 0.000 query: luau string lights grades in the top 10: [2, 2, 2, 2, 1, 1, 1, 1, 1, 1] best possible grades: [2, 2, 2, 2, 1, 1, 1, 1, 1, 1] exact products judged: 4 exact in the top 10: 4 ndcg@10: 1.000 ``` All 90 rankings agree with `lib.metrics` to three decimals: the largest difference is 0.000. For `luau string lights`, BM25's top 10 holds 4 exact products and scores 1.000. That's not a mistake. The query has only 4 exact products judged, so the best possible top 10 holds those and then partial ones, and BM25 found that order. An nDCG@10 of 1 means "as good as the best order of the judged products", not "ten exact products". **What to do:** use the linear-gain, log2(i + 1) form at a cutoff of 10, with the ideal ranking built from every judged product, and state that variant whenever you quote a number. Compare nDCG only between runs scored on the same judgments: new judgments change the ideal ranking, and with it every score. ## Match scikit-learn's `ndcg_score` {#match-scikit-learn-ndcg} Once you've built a metric, check it against a library that computes it independently. At scikit-learn 1.9.1, `ndcg_score` uses the same variant as `lib.metrics`: the grades are the gains, each is divided by log2(rank + 1), and the normalizing DCG is that of the same grades sorted into the best order, cut at `k` ([scikit-learn source](https://raw.githubusercontent.com/scikit-learn/scikit-learn/1.9.1/sklearn/metrics/_ranking.py)). Its documentation cites Järvelin and Kekäläinen, though the code implements the later rank + 1 form ([scikit-learn documentation](https://scikit-learn.org/1.9/modules/generated/sklearn.metrics.ndcg_score.html)). Three details make a naive call disagree with your own: - It scores only the products you pass, so it builds the ideal ranking from those alone. - By default it averages the gains of products whose scores tie (scikit-learn documentation). That equals the average over every order of the tied products ([McSherry and Najork, 2008](https://www.microsoft.com/en-us/research/?p=144984)). - It takes one row per query, every row the same length, and returns the mean over the rows (scikit-learn source). In the store, first pass `ndcg_score` the grades of BM25's top 10 for `clear shower curtain`, with scores that put them in BM25's order. Then build each query's row the way `lib.metrics` sees the query, and compare all three runs: Example `ndcg-matches-scikit-learn` (Python 3.12.15 · bm25s 0.3.13 · numpy 2.5.4 · pandas 3.0.6 · scikit-learn 1.9.1 · scipy 1.18.1), passing in Edition 2026.10.1: ```python from sklearn.metrics import ndcg_score from lib.fusion import reciprocal_rank_fusion from lib.lexical import LexicalSearch from lib.metrics import ndcg_at_k from lib.semantic import SemanticSearch from lib.store import load_products, load_queries, qrels def sklearn_ndcg(ranking, judgments, k=10): """One row for one query: every judged product, plus any retrieved product nobody judged.""" products = list(dict.fromkeys([*judgments, *ranking])) position = {product_id: i for i, product_id in enumerate(ranking)} grades = [[judgments.get(product_id, 0) for product_id in products]] scores = [[-position.get(product_id, len(ranking)) for product_id in products]] # not retrieved: below all return ndcg_score(grades, scores, k=k) queries, judgments = load_queries("test"), qrels("test") bm25 = LexicalSearch(load_products()).run(queries, k=100) semantic = SemanticSearch().run(queries, k=100) runs = {"bm25": bm25, "semantic": semantic, "hybrid": reciprocal_rank_fusion([bm25, semantic])} top = bm25[246][:10] # clear shower curtain naive = ndcg_score([[judgments[246].get(product_id, 0) for product_id in top]], [list(range(10, 0, -1))], k=10) print(f"ndcg@10, only the top 10 passed: {naive:.3f}") print(f"ndcg@10 from lib.metrics: {ndcg_at_k(top, judgments[246], 10):.3f}") print(f"ndcg@10, every judged product passed: {sklearn_ndcg(top, judgments[246]):.3f}") gaps = [abs(sklearn_ndcg(run[query_id], judgments[query_id]) - ndcg_at_k(run[query_id], judgments[query_id], 10)) for run in runs.values() for query_id in judgments] print("rankings compared:", len(gaps)) print(f"largest difference: {max(gaps):.3f}") short = bm25[246][:5] # a ranking shorter than k print(f"5-product ranking, lib.metrics: {ndcg_at_k(short, judgments[246], 10):.3f}") print(f"5-product ranking, scikit-learn: {sklearn_ndcg(short, judgments[246]):.3f}") ``` Output: ```text ndcg@10, only the top 10 passed: 0.901 ndcg@10 from lib.metrics: 0.615 ndcg@10, every judged product passed: 0.615 rankings compared: 90 largest difference: 0.000 5-product ranking, lib.metrics: 0.539 5-product ranking, scikit-learn: 0.629 ``` Passed only the top 10, `ndcg_score` returns 0.901, against 0.615 from `lib.metrics`. It divides by the best order of those ten products, not of the [253 products judged](#check-against-judgments) for the query, so it can't see the exact products BM25 missed. The example's `sklearn_ndcg` fixes that with a recipe. Each row holds every judged product, plus any retrieved product nobody judged (grade 0). The recipe scores retrieved products by their rank, and gives every other product one score below all of them. With that recipe the two agree on all 90 rankings (largest difference 0.000). Call `ndcg_score` once per query and average the results yourself, because each query's row has a different length. The recipe has one limit. When a run holds fewer than k products, the products it didn't retrieve tie for the empty places inside the top k, and `ndcg_score` credits them with their average gain. `lib.metrics` gives an empty place nothing, because the shopper sees nothing there. For BM25's ranking cut to 5 products, `lib.metrics` gives 0.539 and `ndcg_score` gives 0.629. Short runs happen: `LexicalSearch` returns only products with a score above 0. **What to do:** report your own nDCG, and use `ndcg_score` as an independent check, with every judged product in each query's row and scores taken from ranks. Before you compare the two, check that every run holds at least k products. If you pass a ranker's raw scores instead, break ties in a fixed order first, such as by `product_id`, or the two will disagree wherever scores tie. ## Choose a metric before you compare {#choose-a-ranking-metric} Every metric is a model of a shopper. Precision@10 assumes the shopper looks at the whole first page and counts the good products on it. RR assumes the shopper stops at the first good product. nDCG@10 assumes attention fades down the page and that an exact product is worth more than a partial one: with the book's grades, twice as much. Exponential gains, 2^grade − 1 as in [Burges et al. (2005)](https://icml.cc/Conferences/2005/proceedings/papers/012_LearningToRank_BurgesEtAl.pdf), turn the grades into gains of 0, 1 and 3, so an exact product is worth three partial ones. As a rule of thumb, these models agree when two rankers are far apart. When the rankers are close, the metric can pick the winner, which is why you choose it before you look at results. Fuhr asks for hypotheses stated before the experiment, in the form "We want to compare methods A and B using metric X" ([Fuhr, 2017, section 2.6](https://sigir.org/wp-content/uploads/2018/01/p032.pdf)). In the store, take semantic retrieval, the ranker to beat, and the hybrid, which [chapter 1](#fuse-with-rrf) couldn't separate from it. Choosing a metric is part of tuning the evaluation, so make the choice on the 50 train queries and keep the test queries for reporting. Score both rankers four ways: Example `metric-choice-on-train` (Python 3.12.15 · bm25s 0.3.13 · numpy 2.5.4 · pandas 3.0.6 · scipy 1.18.1), passing in Edition 2026.10.1: ```python from lib.fusion import reciprocal_rank_fusion from lib.lexical import LexicalSearch from lib.metrics import evaluate from lib.semantic import SemanticSearch from lib.store import load_products, load_queries, qrels queries, judgments = load_queries("train"), qrels("train") # choosing a metric is tuning: use train bm25 = LexicalSearch(load_products()).run(queries, k=100) semantic = SemanticSearch().run(queries, k=100) runs = {"semantic": semantic, "hybrid": reciprocal_rank_fusion([bm25, semantic])} exponential = {query_id: {product_id: 2 ** grade - 1 for product_id, grade in per_query.items()} for query_id, per_query in judgments.items()} # gains 0, 1, 3 instead of 0, 1, 2 means = {} for name, run in runs.items(): scores = evaluate(run, judgments) means[name] = {"precision@10": scores["precision@10"].mean(), "mrr": scores["rr"].mean(), "ndcg@10": scores["ndcg@10"].mean(), "ndcg@10 with exponential gains": evaluate(run, exponential)["ndcg@10"].mean()} print("train queries:", len(judgments)) winners = [] for metric in means["semantic"]: for name in runs: print(f"{name} {metric}: {means[name][metric]:.3f}") winners.append(max(runs, key=lambda name: means[name][metric])) print(f"higher {metric}: {winners[-1]}") for name in runs: print(f"metrics won by {name}:", winners.count(name)) print(f"largest gap: {max(abs(means['semantic'][m] - means['hybrid'][m]) for m in means['semantic']):.3f}") ``` Output: ```text train queries: 50 semantic precision@10: 0.502 hybrid precision@10: 0.488 higher precision@10: semantic semantic mrr: 0.673 hybrid mrr: 0.710 higher mrr: hybrid semantic ndcg@10: 0.772 hybrid ndcg@10: 0.774 higher ndcg@10: hybrid semantic ndcg@10 with exponential gains: 0.709 hybrid ndcg@10 with exponential gains: 0.708 higher ndcg@10 with exponential gains: semantic metrics won by semantic: 2 metrics won by hybrid: 2 largest gap: 0.037 ``` Semantic retrieval scores higher on 2 of the four metrics and the hybrid on 2. Precision@10 puts `semantic` ahead (0.502 for semantic retrieval, 0.488 for the hybrid), and MRR puts `hybrid` ahead (0.673 and 0.710, in the same order). nDCG@10 favours `hybrid` (0.772 and 0.774), and exponential gains favour `semantic` (0.709 and 0.708). The largest of the four gaps is 0.037. These are train queries, so the gaps say nothing about which ranker is better. They show that when two rankers are this close, the metric decides which way the gap falls. So the choice should follow the shopper, not the result. The store shows a page of products and its judgments are graded, so nDCG@10 with linear gains stays the headline, as [chapter 1](#score-one-ranking) chose, with precision@10 and MRR beside it. **What to do:** choose one headline metric, its gains and its cutoff before you compare rankers, and write them down. We recommend nDCG@10 with linear gains when judgments are graded and shoppers browse a page. Switch to exponential gains when an exact match is worth far more than a near one, as for a spare part that must fit, and to MRR when most queries have one right answer. Report the other metrics beside the headline, and never change the headline after you've seen which ranker it favours. ## Read per-query differences, not only the mean {#read-per-query-differences} A mean of the headline metric over queries is a summary for the store. A shopper sees one query's results. The TREC Robust track was built on that point: "an individual user does not see the average performance of the system, but only the effectiveness of the system on his or her requests". It found that optimizing the average "further improves the effectiveness of the already-effective topics, sometimes at the expense of the poor performers" ([Voorhees, 2004](https://trec.nist.gov/pubs/trec13/papers/ROBUST.OVERVIEW.pdf)). So before you trust a change in the mean, look at the **per-query differences**: one number per query, the new ranker's score minus the old one's. They are also the evidence that the [significance tests](#permutation-test-on-queries) use. The store's hybrid run from [chapter 1](#fuse-with-rrf) scores well above BM25 on average. Compare the two query by query on nDCG@10: Example `per-query-hybrid-versus-bm25` (Python 3.12.15 · bm25s 0.3.13 · numpy 2.5.4 · pandas 3.0.6 · scipy 1.18.1), passing in Edition 2026.10.1: ```python from lib.fusion import reciprocal_rank_fusion from lib.lexical import LexicalSearch from lib.metrics import evaluate from lib.semantic import SemanticSearch from lib.store import load_products, load_queries, qrels queries, judgments = load_queries("test"), qrels("test") bm25 = LexicalSearch(load_products()).run(queries, k=100) hybrid = reciprocal_rank_fusion([bm25, SemanticSearch().run(queries, k=100)]) before, after = evaluate(bm25, judgments), evaluate(hybrid, judgments) difference = (after["ndcg@10"] - before["ndcg@10"]).round(3).sort_values() # one value per test query print(f"mean difference: {difference.mean():.3f}") print(f"median difference: {difference.median():.3f}") print("hybrid better:", (difference > 0).sum()) print("bm25 better:", (difference < 0).sum()) print("tied:", (difference == 0).sum()) print("\nhybrid's three largest losses:") print(queries.loc[difference.index[:3], "query"].to_frame().assign(difference=difference[:3]).to_string()) largest = difference.index[-1] print("\nlargest gain:", queries.at[largest, "query"]) print(f"its gain: {difference[largest]:.3f}") print(f"its bm25 judged@10: {before.at[largest, 'judged@10']:.3f}") print(f"share of the mean gain from this query: {difference[largest] / difference.sum():.3f}") relative = after["ndcg@10"] / before["ndcg@10"] - 1 # per-query relative gains print(f"\nrelative gain of the means: {after['ndcg@10'].mean() / before['ndcg@10'].mean() - 1:.3f}") print(f"mean of per-query relative gains: {relative.mean():.3f}") print(f"its bm25 ndcg@10: {before.at[largest, 'ndcg@10']:.3f}") print(f"its relative gain: {relative[largest]:.3f}") print(f"mean of per-query relative gains without it: {relative.drop(largest).mean():.3f}") ``` Output: ```text mean difference: 0.087 median difference: 0.057 hybrid better: 17 bm25 better: 7 tied: 6 hybrid's three largest losses: query difference query_id 293 wood floor protector mat -0.072 292 decorative white pillow -0.069 249 bistro table and chairs -0.066 largest gain: outdoor light fixtures its gain: 0.714 its bm25 judged@10: 0.200 share of the mean gain from this query: 0.274 relative gain of the means: 0.120 mean of per-query relative gains: 0.218 its bm25 ndcg@10: 0.286 its relative gain: 2.492 mean of per-query relative gains without it: 0.140 ``` The mean gain is 0.087, but the median is 0.057. The hybrid is better on 17 queries, worse on 7 and tied on 6. One query does much of the work: `outdoor light fixtures` gains 0.714, a share of 0.274 of the total gain, and BM25's judged@10 on it is 0.200. [Judged@10](#score-every-test-query) is the share of the top 10 that has a judgment for the query. For this query BM25's top 10 is mostly light bulbs nobody judged, and [chapter 1](#score-every-test-query) argues that their grade of 0 is fair, so the gain holds. For another query, the unjudged products a ranker drops might be good ones. The output also lists the hybrid's three largest losses. Those are the queries where a shopper would see search get worse, and the first ones to read. Report the difference as an absolute number. The relative gain of the means is 0.120, but the mean of the per-query relative gains is 0.218, because a mean of per-query ratios gives most weight to the queries where BM25 scores low. `outdoor light fixtures`, where BM25 scores 0.286, contributes a relative gain of 2.492 on its own; without it, the mean of relative gains is 0.140. Fuhr shows that the relative change in a mean says nothing reliable about the change per query, and that "for arithmetic mean, only absolute improvements are meaningful" ([Fuhr, 2017, section 2.4](https://sigir.org/wp-content/uploads/2018/01/p032.pdf)). **What to do:** for every comparison, print the mean and median difference, the counts of wins, losses and ties, and the queries with the largest losses, then read those queries. Check judged@10 on the queries that move most before you credit the ranker. Report absolute differences of means, not percentages. ## Test a gap with a permutation test {#permutation-test-on-queries} A gap between two means on a few dozen queries can come from chance alone: with a different sample of queries, the gap would be different. Shallow metrics make this worse. Buckley and Voorhees found that researchers using a measure such as precision at 10 "will need to use many more than 50 queries" to tell systems apart reliably ([Buckley and Voorhees, 2000](https://nist.gov/publications/evaluating-evaluation-measure-stability)). A **significance test** asks how surprising the gap would be if the two rankers were equally good, an assumption called the **null hypothesis**. Its answer is the **p-value**: the probability, under the null hypothesis, of a gap at least as large as the one you measured. By convention, a p-value below 0.05 counts as evidence of a real difference, and [the book's rule](#when-a-difference-is-established) tightens that when a chapter makes several comparisons. The tests in this book are **paired**: both rankers are scored on the same queries, so each test works on the [per-query differences](#read-per-query-differences), and a hard query's low scores cancel out. A **permutation test** builds the gaps you'd see under the null hypothesis directly. If the two rankers were interchangeable, each query's difference would be as likely to have the opposite sign. So flip each sign at random, recompute the mean, repeat thousands of times, and count how often the flipped mean lands at least as far from 0 as the real one. For n queries there are 2^n sign patterns, 1,073,741,824 for the test queries, so the test samples 9,999 of them, and its p-value is an estimate. In the store, the hybrid trails semantic retrieval on the test queries, and [chapter 1](#fuse-with-rrf) left open whether that gap is real. Against semantic retrieval, the hybrid's wins, losses and ties over the test queries are 9 / 14 / 7. Before you read the output, predict: will the test find the hybrid's deficit larger than chance? Example `permutation-test-by-hand` (Python 3.12.15 · bm25s 0.3.13 · numpy 2.5.4 · pandas 3.0.6 · scipy 1.18.1), passing in Edition 2026.10.1: ```python import numpy as np from scipy.stats import permutation_test from lib.fusion import reciprocal_rank_fusion from lib.lexical import LexicalSearch from lib.metrics import evaluate from lib.semantic import SemanticSearch from lib.store import load_products, load_queries, qrels queries, judgments = load_queries("test"), qrels("test") semantic = SemanticSearch().run(queries, k=100) hybrid = reciprocal_rank_fusion([LexicalSearch(load_products()).run(queries, k=100), semantic]) after = evaluate(hybrid, judgments)["ndcg@10"].to_numpy() before = evaluate(semantic, judgments)["ndcg@10"].to_numpy() difference = after - before # hybrid minus semantic, per test query print("test queries:", len(difference)) print(f"sign patterns for the test queries: {2 ** len(difference):,}") print(f"observed mean difference: {difference.mean():.3f}") # if the two rankers were interchangeable, each query's difference would be as likely to have the other sign rng = np.random.default_rng(2026) signs = rng.choice([-1, 1], size=(9999, len(difference))) flipped = (signs * difference).mean(axis=1) # 9,999 mean differences under that assumption print(f"95% of flipped means are smaller than: {np.quantile(np.abs(flipped), 0.95):.3f}") print(f"p-value by hand: {np.mean(np.abs(flipped) >= abs(difference.mean())):.3f}") result = permutation_test((after, before), lambda after_scores, before_scores, axis: np.mean(after_scores - before_scores, axis=axis), permutation_type="samples", rng=np.random.default_rng(2026)) print(f"p-value from scipy: {result.pvalue:.3f}") print("difference at the 0.05 level:", "larger than chance" if result.pvalue < 0.05 else "within chance") ``` Output: ```text test queries: 30 sign patterns for the test queries: 1,073,741,824 observed mean difference: -0.018 95% of flipped means are smaller than: 0.022 p-value by hand: 0.117 p-value from scipy: 0.125 difference at the 0.05 level: within chance ``` The observed mean difference is -0.018. Of the 9,999 flipped means, 95% are smaller than 0.022 in size, and the share at least as large as the observed one, the p-value, is 0.117. SciPy's `permutation_test` with `permutation_type="samples"`, the setting its documentation gives for paired tests, returns 0.125 ([SciPy documentation](https://docs.scipy.org/doc/scipy-1.18.0/reference/generated/scipy.stats.permutation_test.html)). The two estimates differ a little because each draws its own random sign patterns. At the 0.05 level, the difference is within chance. If you predicted a real deficit, you trusted the mean over the spread: a few queries swinging the other way would close the gap. SciPy's resampling functions take their random generator through `rng`, which replaces the `random_state` argument. At 1.18.1 both work, but the documentation says that "only the rng keyword should be used in new code", because calls that use `random_state` will emit warnings after a transition period (SciPy documentation). Every example in this book passes `rng` with a fixed seed. **What to do:** test every gap you intend to act on with a paired permutation test on the per-query differences. The book decides on this test, as [its rule](#when-a-difference-is-established) sets out. Read a p-value above the threshold as "no evidence of a difference", never as "no difference": with this few queries, a test can miss a real gap, and McKechnie et al. argue for measuring such misses alongside false positives ([McKechnie et al., 2025](https://arxiv.org/abs/2507.07924)). Never explain a gap the test can't tell from chance. ## Get the p-value from a paired t-test {#paired-t-test-on-queries} The paired t-test answers the same question with a formula instead of resampling. It divides the mean of the [per-query differences](#read-per-query-differences) by its **standard error**: the standard deviation of the differences divided by the square root of the number of queries, which estimates how far the mean would move from one sample of queries to another. If the rankers are equally good and the differences are roughly normal, that ratio, t, follows a **t distribution**: a bell-shaped curve with heavier tails than the normal, whose shape depends on its degrees of freedom, here one fewer than the number of queries. The p-value is the probability of a t at least that far from 0 in either direction ([Sakai, 2014, section 3.1.1](https://sigir.org/files/forum/2014J/2014J_sigirforum_Article_TetsuyaSakai.pdf)): ```text t = mean(d) / (sd(d) / sqrt(n)) d: the per-query differences, n: the number of queries ``` Per-query nDCG differences are bounded between −1 and 1 and have outliers, such as the [gain on `outdoor light fixtures`](#read-per-query-differences), so they aren't normal. In practice the t-test holds up. On TREC runs scored with mean average precision (MAP, another ranking metric) on 50 topics, Smucker et al. found "little practical difference between the randomization, bootstrap, and t tests", where the randomization test is the [permutation test](#permutation-test-on-queries) ([Smucker et al., 2007](https://ciir-publications.cs.umass.edu/getpdf.php?id=744)). In a simulation built from TREC data, with sets of 25, 50 and 100 topics, Urbano et al. found that "both the t-test and the permutation test behave almost identically" ([Urbano et al., 2019](https://arxiv.org/abs/1905.11096)). Both studies advise against the Wilcoxon signed-rank and sign tests for this job. SciPy's `wilcoxon` also drops every query whose difference is exactly 0 by default, with `zero_method='wilcox'` ([SciPy documentation](https://docs.scipy.org/doc/scipy-1.18.0/reference/generated/scipy.stats.wilcoxon.html)), and many queries tie when two runs differ a little. Compute t by hand for two of the store's gaps from chapter 1, [semantic retrieval against BM25](#search-by-meaning) and [the hybrid against semantic retrieval](#fuse-with-rrf), and compare with SciPy's `ttest_rel`: Example `paired-t-test-by-hand` (Python 3.12.15 · bm25s 0.3.13 · numpy 2.5.4 · pandas 3.0.6 · scipy 1.18.1), passing in Edition 2026.10.1: ```python import math from scipy.stats import t, ttest_rel from lib.fusion import reciprocal_rank_fusion from lib.lexical import LexicalSearch from lib.metrics import evaluate from lib.semantic import SemanticSearch from lib.store import load_products, load_queries, qrels queries, judgments = load_queries("test"), qrels("test") bm25 = LexicalSearch(load_products()).run(queries, k=100) semantic = SemanticSearch().run(queries, k=100) runs = {"bm25": bm25, "semantic": semantic, "hybrid": reciprocal_rank_fusion([bm25, semantic])} ndcg = {name: evaluate(run, judgments)["ndcg@10"] for name, run in runs.items()} # per test query for after, before in [("semantic", "bm25"), ("hybrid", "semantic")]: difference = ndcg[after] - ndcg[before] n = len(difference) standard_error = difference.std(ddof=1) / math.sqrt(n) # how much the mean would move between samples statistic = difference.mean() / standard_error p_value = 2 * t.sf(abs(statistic), df=n - 1) # both tails of the t distribution print(f"{after} minus {before}: {difference.mean():.3f}") print(f"{after} minus {before} t by hand: {statistic:.3f}") print(f"{after} minus {before} p by hand: {p_value:.3f}") print(f"{after} minus {before} p from ttest_rel: {ttest_rel(ndcg[after], ndcg[before]).pvalue:.3f}") ``` Output: ```text semantic minus bm25: 0.105 semantic minus bm25 t by hand: 3.294 semantic minus bm25 p by hand: 0.003 semantic minus bm25 p from ttest_rel: 0.003 hybrid minus semantic: -0.018 hybrid minus semantic t by hand: -1.590 hybrid minus semantic p by hand: 0.123 hybrid minus semantic p from ttest_rel: 0.123 ``` For semantic retrieval against BM25, the mean difference of 0.105 gives t = 3.294 and a p-value of 0.003, the same as `ttest_rel` gives: a gap that chance would rarely produce. For the hybrid against semantic retrieval, t = -1.590 and the p-value is 0.123, close to the permutation test's 0.125 for the [same gap](#permutation-test-on-queries). **What to do:** report the paired t-test, `scipy.stats.ttest_rel`, beside the permutation test, as a check that needs no resampling. When the two fall on different sides of the threshold, say so, and let the permutation test decide. Don't use the Wilcoxon or sign test to compare mean scores. ## Report how big the gap is {#report-the-size-of-a-gap} A p-value says how surprising a gap is, not how big it is. On a large sample, "almost any modification will result in a significant, but tiny difference" ([Fuhr, 2017, section 2.8](https://sigir.org/wp-content/uploads/2018/01/p032.pdf)), and Sakai recommends reporting an effect size and a confidence interval with every p-value ([Sakai, 2014](https://sigir.org/files/forum/2014J/2014J_sigirforum_Article_TetsuyaSakai.pdf)). A **confidence interval** is a range of plausible values for the true mean difference, meaning the mean over all queries like these, scored with these judgments, not over the store's search traffic. A 95% interval comes from a procedure meant to cover that value 95% of the time. The **bootstrap** builds one by resampling. Draw as many queries as you have from the queries you have, with replacement, so that some appear twice and others not at all. Compute the mean difference, repeat thousands of times, and take the middle 95% of those means: the **percentile interval**. The **effect size** of a paired comparison is the mean difference divided by the standard deviation of the differences (Sakai, 2014, section 3.1.2): how large the gap is next to how much it varies from query to query. It isn't Cohen's d, the effect size for unpaired samples, and Sakai warns that the two "should not be compared directly". In the store, build the percentile interval by hand for semantic retrieval against BM25 and the hybrid against semantic retrieval. Check it with SciPy's `bootstrap` set to `method="percentile"`, then run SciPy's default method: Example `bootstrap-interval-by-hand` (Python 3.12.15 · bm25s 0.3.13 · numpy 2.5.4 · pandas 3.0.6 · scipy 1.18.1), passing in Edition 2026.10.1: ```python import numpy as np from scipy.stats import bootstrap from lib.fusion import reciprocal_rank_fusion from lib.lexical import LexicalSearch from lib.metrics import evaluate from lib.semantic import SemanticSearch from lib.store import load_products, load_queries, qrels queries, judgments = load_queries("test"), qrels("test") bm25 = LexicalSearch(load_products()).run(queries, k=100) semantic = SemanticSearch().run(queries, k=100) runs = {"bm25": bm25, "semantic": semantic, "hybrid": reciprocal_rank_fusion([bm25, semantic])} ndcg = {name: evaluate(run, judgments)["ndcg@10"] for name, run in runs.items()} # per test query for after, before in [("semantic", "bm25"), ("hybrid", "semantic")]: difference, name = (ndcg[after] - ndcg[before]).to_numpy(), f"{after} minus {before}" resampled = np.random.default_rng(2026).choice(difference, size=(9999, len(difference))) # 9,999 sets of queries low, high = np.percentile(resampled.mean(axis=1), [2.5, 97.5]) # the middle 95% of their means print(f"{name}: {difference.mean():.3f}") print(f"{name} interval by hand: {low:.3f} to {high:.3f}") percentile = bootstrap((difference,), np.mean, method="percentile", rng=np.random.default_rng(2026)) print(f"{name} percentile interval from scipy: " f"{percentile.confidence_interval.low:.3f} to {percentile.confidence_interval.high:.3f}") bca = bootstrap((difference,), np.mean, rng=np.random.default_rng(2026)).confidence_interval # BCa, the default print(f"{name} bca low: {bca.low:.3f}\n{name} bca high: {bca.high:.3f}") print(f"{name} bca interval and 0: {'includes 0' if bca.low <= 0 <= bca.high else 'excludes 0'}") print(f"{name} effect size: {difference.mean() / difference.std(ddof=1):.3f}") ``` Output: ```text semantic minus bm25: 0.105 semantic minus bm25 interval by hand: 0.049 to 0.172 semantic minus bm25 percentile interval from scipy: 0.049 to 0.172 semantic minus bm25 bca low: 0.055 semantic minus bm25 bca high: 0.185 semantic minus bm25 bca interval and 0: excludes 0 semantic minus bm25 effect size: 0.601 hybrid minus semantic: -0.018 hybrid minus semantic interval by hand: -0.041 to 0.003 hybrid minus semantic percentile interval from scipy: -0.041 to 0.003 hybrid minus semantic bca low: -0.042 hybrid minus semantic bca high: 0.002 hybrid minus semantic bca interval and 0: includes 0 hybrid minus semantic effect size: -0.290 ``` SciPy's percentile interval matches the one built by hand: 0.049 to 0.172 and -0.041 to 0.003. Its default method, BCa (bias-corrected and accelerated), reads different percentiles off the same resampled means. It shifts them by how far the resampled means sit off-centre from the observed mean, and stretches them by the skew it estimates by leaving out one query at a time ([SciPy source](https://github.com/scipy/scipy/blob/v1.18.1/scipy/stats/_resampling.py)). The book's [harness](#compare-runs-in-a-harness) reports BCa, which is designed for skewed resampled means like these. For semantic retrieval against BM25, BCa runs from 0.055 to 0.185, so it excludes 0, and the effect size is 0.601. For the hybrid against semantic retrieval, it runs from -0.042 to 0.002, so it includes 0, and the effect size is -0.290. That interval allows anything from a loss of a few hundredths to a gain of at most 0.002: no gain worth having. A bootstrap interval only approximates its level, and on a few dozen queries it falls short. To check, treat the hybrid's 30 differences against semantic retrieval as if they were every query there is, so the true gap is known. Then draw 1000 new sets of the same size from them, build an interval on each, and count how often it covers the true gap: Example `interval-coverage-check` (Python 3.12.15 · bm25s 0.3.13 · numpy 2.5.4 · pandas 3.0.6 · scipy 1.18.1), passing in Edition 2026.10.1: ```python import numpy as np from scipy.stats import bootstrap, ttest_1samp from lib.fusion import reciprocal_rank_fusion from lib.lexical import LexicalSearch from lib.metrics import evaluate from lib.semantic import SemanticSearch from lib.store import load_products, load_queries, qrels queries, judgments = load_queries("test"), qrels("test") semantic = SemanticSearch().run(queries, k=100) hybrid = reciprocal_rank_fusion([LexicalSearch(load_products()).run(queries, k=100), semantic]) population = (evaluate(hybrid, judgments)["ndcg@10"] - evaluate(semantic, judgments)["ndcg@10"]).to_numpy() true_gap = population.mean() # known, because these differences stand in for every query there is rng, trials = np.random.default_rng(2026), 1000 print("trials:", trials) print(f"standard error of a coverage near 0.95: {(0.95 * 0.05 / trials) ** 0.5:.3f}") for level in [0.95, round(1 - 0.05 / 3, 4)]: # the usual level, and the level for three comparisons covered = {"bca": 0, "t": 0} for _ in range(trials): sample = rng.choice(population, size=len(population)) # another set of 30 queries like these bca = bootstrap((sample,), np.mean, confidence_level=level, n_resamples=1999, rng=rng).confidence_interval t = ttest_1samp(sample, 0).confidence_interval(level) # the interval that matches the t-test covered["bca"] += bca.low <= true_gap <= bca.high covered["t"] += t.low <= true_gap <= t.high for method, count in covered.items(): print(f"{method} coverage at {level}: {count / trials:.3f}") ``` Output: ```text trials: 1000 standard error of a coverage near 0.95: 0.007 bca coverage at 0.95: 0.927 t coverage at 0.95: 0.945 bca coverage at 0.9833: 0.955 t coverage at 0.9833: 0.975 ``` At a nominal 95%, BCa covers the true gap 0.927 of the time and the t-test's interval 0.945. At 0.9833, the level [the harness](#compare-runs-in-a-harness) uses, BCa covers 0.955 and the t interval 0.975. Each figure is uncertain by about 0.007 either way. BCa falls short of its level at both, and the t interval comes closer to its own. So a bootstrap interval that barely excludes 0 deserves care. In a simulation, a test built on the same resampling, the bootstrap-shift test, also "shows a systematic bias towards small p-values" ([Urbano et al., 2019](https://arxiv.org/abs/1905.11096)). Whether a gap matters is a judgment no test makes for you. For precision@10, Fuhr notes, "a difference of 0.02 translates into one more relevant document per five queries" (Fuhr, 2017, section 2.8). For nDCG@10, decide before you test what gap would justify the cost of a change. **What to do:** report every comparison as the mean difference, with its bootstrap interval, its effect size and its p-value. If the interval includes 0, the comparison shows no improvement. If it excludes 0 but lies below the smallest gap worth the change's cost, the difference may be real and still not worth shipping. Use the interval to describe a gap's size, not to decide whether it exists. ## Decide in advance what counts as established {#when-a-difference-is-established} Every test at 0.05 carries a 5% chance of a false positive when the gap is noise, and a chapter that runs several tests runs that risk several times. Comparing every pair of 10 systems this way gives a probability of about 0.90 of at least one false "significant" result, if every gap is noise and the tests are independent ([Sakai, 2014, section 3](https://sigir.org/files/forum/2014J/2014J_sigirforum_Article_TetsuyaSakai.pdf)). The **Bonferroni correction** is a simple, conservative fix: when you test m hypotheses, test each at 0.05 divided by m ([Fuhr, 2017, section 2.7](https://sigir.org/wp-content/uploads/2018/01/p032.pdf)). It keeps the chance of any false positive at or below 0.05 whether or not the tests are independent. It only works if the m comparisons are fixed before you see the results. Fuhr names the usual way to break that: "it would be incorrect if an author first determines which of the competing methods performs best on his data set, and then compares his own approach only to this one" (Fuhr, 2017, section 2.6). So the book uses one rule, in every chapter: 1. **List the planned comparisons on the test queries** before reporting any of them. 2. **A difference is established** when its paired [permutation test](#permutation-test-on-queries) gives a p-value below the chapter's Bonferroni threshold: 0.05 divided by the number of planned comparisons. 3. **Report the paired t-test and a bootstrap interval beside it**, the interval at the matching level, 1 minus the threshold. When either disagrees with the permutation test, say so. 4. **A comparison chosen after looking at test results is exploratory.** Label it so, and never call it established. 5. **Comparisons on train queries are for tuning,** not evidence. We let the permutation test decide because it assumes nothing about the shape of the differences, only that under the null hypothesis each query's sign is a coin flip. Urbano et al. found that it and the t-test "behave almost identically" ([Urbano et al., 2019](https://arxiv.org/abs/1905.11096)), which makes the t-test a good check. Bootstrap intervals on a few dozen queries cover less often than their level ([coverage check](#report-the-size-of-a-gap)), which is why an interval describes a gap but doesn't decide it. Chapter 1 builds the store's pipeline one stage at a time, and that order alone fixes its planned comparisons: each stage against the one before it. That gives semantic retrieval against BM25, the hybrid against semantic retrieval, and the reranked hybrid against the hybrid, 3 in all. The threshold is 0.0167, and the intervals use a level of 0.9833. Comparing the reranked hybrid with semantic retrieval is exploratory, because semantic retrieval became the ranker to beat by scoring highest on these same test queries. **What to do:** write the list of planned comparisons before you score anything on the test queries, and keep it short, since each one raises the bar for the others. When you need to compare many rankers at once, Bonferroni gets very strict. Sakai recommends an analysis of variance, which tests whether any of the rankers differ at all, followed by the randomised Tukey HSD test, which compares every pair while holding the chance of any false positive at the level (Sakai, 2014). ## Compare runs in an evaluation harness {#compare-runs-in-a-harness} The book applies [that rule](#when-a-difference-is-established) in an **evaluation harness**, which holds everything fixed except the ranker: the same queries, judgments, metrics and tests, so that two runs differ only in their rankings. `lib.metrics.evaluate` returns one row per query, which is the shape the paired tests need. `compare` in `lib.significance`, a short module that collects this chapter's code, runs the tests on one metric's column from two runs. It returns the mean difference, the bootstrap interval, the permutation and t-test p-values, the effect size, and the wins, losses and ties. Its `verdict` function applies [the book's rule](#when-a-difference-is-established) to a result. Keep the per-query frames, and average only at the end. In the store, run chapter 1's four rankers through the harness, then the 3 planned comparisons and the exploratory one: Example `compare-chapter-one-rankers` (Python 3.12.15 · bm25s 0.3.13 · numpy 2.5.4 · pandas 3.0.6 · scipy 1.18.1), passing in Edition 2026.10.1: ```python from lib.fusion import reciprocal_rank_fusion from lib.lexical import LexicalSearch from lib.metrics import evaluate from lib.recorded import cross_encoder_scores from lib.semantic import SemanticSearch from lib.significance import compare, verdict from lib.store import load_products, load_queries, qrels queries, judgments = load_queries("test"), qrels("test") bm25 = LexicalSearch(load_products()).run(queries, k=100) semantic = SemanticSearch().run(queries, k=100) hybrid = reciprocal_rank_fusion([bm25, semantic]) score = {(query_id, product_id): value for query_id, product_id, value in cross_encoder_scores().itertuples(index=False)} reranked = {query_id: sorted(ranking[:50], key=lambda product_id: score[(query_id, product_id)], reverse=True) + ranking[50:] for query_id, ranking in hybrid.items()} # chapter 1's rerank of the top 50 runs = {"bm25": bm25, "semantic": semantic, "hybrid": hybrid, "reranked": reranked} scores = {name: evaluate(run, judgments) for name, run in runs.items()} # one row per test query for name, frame in scores.items(): print(f"{name} ndcg@10: {frame['ndcg@10'].mean():.3f}\n{name} judged@10: {frame['judged@10'].mean():.3f}") planned = [("semantic", "bm25"), ("hybrid", "semantic"), ("reranked", "hybrid")] # each stage against the last exploratory = [("reranked", "semantic")] # chosen after chapter 1 saw the test means threshold = 0.05 / len(planned) # Bonferroni, over the planned comparisons only print(f"\nplanned comparisons: {len(planned)}\np-value threshold: {threshold:.4f}\ninterval level: {1 - threshold:.4f}") print(f"chance of a false positive, all gaps noise, tests independent: {1 - 0.95 ** len(planned):.3f}") established, disagreements = 0, 0 for after, before in planned + exploratory: result, name = compare(scores[after]["ndcg@10"], scores[before]["ndcg@10"], confidence=1 - threshold), f"{after} minus {before}" checks, kind = verdict(result, threshold), "planned" if (after, before) in planned else "exploratory" print(f"\n{name}: {result['mean difference']:.3f}\n{name} kind: {kind}") print(f"{name} interval: {result['ci low']:.3f} to {result['ci high']:.3f}") print(f"{name} permutation p: {result['permutation p']:.3f}\n{name} t-test p: {result['t-test p']:.3f}") print(f"{name} effect size: {result['effect size']:.3f}\n{name} wins, losses, ties: {result['wins']} / {result['losses']} / {result['ties']}") print(f"{name} established: {'yes' if checks['established'] and kind == 'planned' else 'no'}") for method in ["t-test", "interval"]: print(f"{name} {method} agrees: {'yes' if checks[method + ' agrees'] else 'no'}") if kind == "planned": established += checks["established"] disagreements += not (checks["t-test agrees"] and checks["interval agrees"]) print(f"\nplanned comparisons established: {established}\nplanned comparisons with a disagreement: {disagreements}") ``` Output: ```text bm25 ndcg@10: 0.725 bm25 judged@10: 0.927 semantic ndcg@10: 0.830 semantic judged@10: 0.990 hybrid ndcg@10: 0.812 hybrid judged@10: 0.980 reranked ndcg@10: 0.798 reranked judged@10: 0.980 planned comparisons: 3 p-value threshold: 0.0167 interval level: 0.9833 chance of a false positive, all gaps noise, tests independent: 0.143 semantic minus bm25: 0.105 semantic minus bm25 kind: planned semantic minus bm25 interval: 0.047 to 0.205 semantic minus bm25 permutation p: 0.001 semantic minus bm25 t-test p: 0.003 semantic minus bm25 effect size: 0.601 semantic minus bm25 wins, losses, ties: 21 / 7 / 2 semantic minus bm25 established: yes semantic minus bm25 t-test agrees: yes semantic minus bm25 interval agrees: yes hybrid minus semantic: -0.018 hybrid minus semantic kind: planned hybrid minus semantic interval: -0.048 to 0.006 hybrid minus semantic permutation p: 0.125 hybrid minus semantic t-test p: 0.123 hybrid minus semantic effect size: -0.290 hybrid minus semantic wins, losses, ties: 9 / 14 / 7 hybrid minus semantic established: no hybrid minus semantic t-test agrees: yes hybrid minus semantic interval agrees: yes reranked minus hybrid: -0.014 reranked minus hybrid kind: planned reranked minus hybrid interval: -0.053 to 0.020 reranked minus hybrid permutation p: 0.382 reranked minus hybrid t-test p: 0.375 reranked minus hybrid effect size: -0.165 reranked minus hybrid wins, losses, ties: 11 / 12 / 7 reranked minus hybrid established: no reranked minus hybrid t-test agrees: yes reranked minus hybrid interval agrees: yes reranked minus semantic: -0.032 reranked minus semantic kind: exploratory reranked minus semantic interval: -0.078 to -0.003 reranked minus semantic permutation p: 0.048 reranked minus semantic t-test p: 0.048 reranked minus semantic effect size: -0.377 reranked minus semantic wins, losses, ties: 9 / 16 / 5 reranked minus semantic established: no reranked minus semantic t-test agrees: yes reranked minus semantic interval agrees: no planned comparisons established: 1 planned comparisons with a disagreement: 0 ``` BM25's [judged@10](#score-every-test-query) is 0.927, against 0.990 for semantic retrieval, 0.980 for the hybrid and 0.980 for the reranked hybrid, so unjudged products weigh most on BM25. With 3 tests at 0.05 each, the chance of at least one false positive would be 0.143 if every gap were noise and the tests were independent. Bonferroni keeps it at or below 0.05 either way. On nDCG@10 over the test queries, with intervals at the 0.9833 level: | Measure | Semantic minus BM25 | Hybrid minus semantic | Reranked minus hybrid | Reranked minus semantic | | --- | --- | --- | --- | --- | | Kind | planned | planned | planned | exploratory | | Mean difference | 0.105 | -0.018 | -0.014 | -0.032 | | Interval | 0.047 to 0.205 | -0.048 to 0.006 | -0.053 to 0.020 | -0.078 to -0.003 | | Permutation p | 0.001 | 0.125 | 0.382 | 0.048 | | t-test p | 0.003 | 0.123 | 0.375 | 0.048 | | Effect size | 0.601 | -0.290 | -0.165 | -0.377 | | Wins / losses / ties | 21 / 7 / 2 | 9 / 14 / 7 | 11 / 12 / 7 | 9 / 16 / 5 | | Established | yes | no | no | no | Of the 3 planned comparisons, 1 is established: semantic retrieval's lead over BM25, with a permutation p-value of 0.001. The hybrid's deficit against semantic retrieval and the reranker's against the hybrid are not established: on these queries neither change shows an improvement, and neither is shown to be worse. The harness also checks the t-test and the interval against the permutation test, and counts 0 planned comparisons where either one disagrees. The exploratory column is where the methods part. Its permutation p-value is 0.048, to set against 0.05 without a correction and 0.0167 with one, and its interval runs from -0.078 to -0.003. Asked whether each check agrees with the permutation test, the harness answers yes for the t-test and no for the interval: an interval that excludes 0 beside a p-value above the threshold is the kind of disagreement the [coverage check](#report-the-size-of-a-gap) warns about. Whatever its numbers, this comparison can't be established, because it was chosen after seeing the test means. At most it raises a question for new queries: does this reranker trail semantic retrieval? Chapter 1's decision stands, by a rule that doesn't depend on which ranker scored highest: the ranker to beat is the simplest pipeline that no planned comparison has shown to be beaten. Semantic retrieval beat BM25, and neither later stage beat the stage before it. **What to do:** for every change, report the mean nDCG@10 and judged@10 of each run, then a table like this one, with a column for each comparison and the exploratory ones labelled. A change ships only when it is established against the ranker to beat. ## What goes wrong in production {#measuring-search-in-production} **One sample of queries can mislead.** Results on held-out queries depend on which queries landed there. Fuhr calls the simple holdout method unreliable, "since the results may be heavily dependent on the split", and asks for cross-validation instead ([Fuhr, 2017, section 2.5](https://sigir.org/wp-content/uploads/2018/01/p032.pdf)). Sakai agrees that "different training/validation/test splits may yield different results" ([Sakai, 2020](https://sigir.org/wp-content/uploads/2020/06/p14.pdf)). The book keeps one fixed split so that every chapter reports on the same test queries. To see how far a gap can move, run the exploratory comparison from [the harness](#compare-runs-in-a-harness) on the train queries too. This is an illustration, not evidence, because train queries are for tuning: Example `split-changes-the-verdict` (Python 3.12.15 · bm25s 0.3.13 · numpy 2.5.4 · pandas 3.0.6 · scipy 1.18.1), passing in Edition 2026.10.1: ```python from lib.fusion import reciprocal_rank_fusion from lib.lexical import LexicalSearch from lib.metrics import evaluate from lib.recorded import cross_encoder_scores from lib.semantic import SemanticSearch from lib.significance import compare from lib.store import load_products, load_queries, qrels lexical, embeddings = LexicalSearch(load_products()), SemanticSearch() score = {(query_id, product_id): value for query_id, product_id, value in cross_encoder_scores().itertuples(index=False)} for split in ["test", "train"]: # the same comparison on another sample of the store's queries queries, judgments = load_queries(split), qrels(split) semantic = embeddings.run(queries, k=100) hybrid = reciprocal_rank_fusion([lexical.run(queries, k=100), semantic]) reranked = {query_id: sorted(ranking[:50], key=lambda product_id: score[(query_id, product_id)], reverse=True) + ranking[50:] for query_id, ranking in hybrid.items()} result = compare(evaluate(reranked, judgments)["ndcg@10"], evaluate(semantic, judgments)["ndcg@10"]) gap = result["mean difference"] print(f"{split} queries:", len(judgments)) print(f"{split} reranked minus semantic: {gap:.3f}") print(f"{split} higher-scoring run:", "reranked" if gap > 0 else "semantic" if gap < 0 else "neither") if split == "test": print(f"test permutation p: {result['permutation p']:.3f}") ``` Output: ```text test queries: 30 test reranked minus semantic: -0.032 test higher-scoring run: semantic test permutation p: 0.048 train queries: 50 train reranked minus semantic: 0.014 train higher-scoring run: reranked ``` On the 30 test queries, the reranked hybrid minus semantic retrieval is -0.032 (permutation p = 0.048). On the 50 train queries it is 0.014. The higher-scoring run is `semantic` on the test queries and `reranked` on the train queries. A p-value this close to 0.05 on one sample is a weak thing to build on. [Chapter 11](#keep-test-queries-held-out) covers keeping test queries out of every choice. **Reusing the test queries is testing many hypotheses.** Every change you test on the same queries adds a comparison, even across months. Fuhr notes that "collection reuse is also a case of multiple hypothesis testing" ([Fuhr, 2017, section 2.7](https://sigir.org/wp-content/uploads/2018/01/p032.pdf)). Keep a log of the comparisons run on your test queries, and count them when you set the threshold. **Tied scores make runs non-repeatable.** Two products with the same score can come back in either order, and "the interaction between multi-threaded indexing and score ties during retrieval may yield non-deterministic rankings" ([Lin and Yang, 2019](https://arxiv.org/abs/1807.05798)). Even a repeatable order can be arbitrary: a tie falls to whatever order a library's top-k selection, or the input to a stable sort, happens to leave. So the book's modules break ties by `product_id`. `LexicalSearch` and `lib.fusion` sort equal scores by it, and `SemanticSearch` sorts stably over vectors stored in `product_id` order. With every version pinned, `re-check examples` reruns each example against its recorded output, so a change in any of these orders would show. In your own system, break ties in a fixed order that carries no information, such as by `product_id`, before you score. **Unjudged products lower a ranker's score.** A ranker that surfaces products nobody judged loses nDCG for them, even when they're good. Check [judged@10](#score-every-test-query) before you test, and have new products judged when it falls. **What to do:** log every comparison you run on the test queries, treat a borderline or exploratory result as a question for new queries, break score ties in a fixed order, and report judged@10 next to every nDCG@10. ## Decisions this chapter made {#measuring-search-decisions} Each choice below is the book's default from here on. A later chapter changes one only with a reason it states. - **Headline metric: nDCG@10 with linear gains** (the grades 2, 1 and 0), a log2(i + 1) discount, and an ideal ranking from every judged product. Report precision@10 and MRR beside it. Relevant means exact. - **Our own metrics are the reference.** The book reports `lib.metrics`, and checks it with scikit-learn's `ndcg_score`, with every judged product in each row. - **Every comparison shows per-query differences:** mean and median, wins, losses and ties, the largest losses, and judged@10 on the queries that move most. - **The rule for an established difference:** list the planned comparisons on the test queries first. A difference is established when its permutation test, with 9,999 random sign flips, gives a p-value below 0.05 divided by the number of planned comparisons. Report the paired t-test and a BCa bootstrap interval at the matching level beside it, and say when they disagree. Label comparisons chosen after seeing test results as exploratory, and never call them established. Use train queries for tuning only. - **Report a gap that isn't established as "no improvement shown",** never as a loss, and never explain it. - **All resampling uses `rng` with a fixed seed,** through `lib.significance`. - **Of chapter 1's planned comparisons, semantic retrieval's lead over BM25 is established,** and the other two are not. Semantic retrieval stays the ranker to beat. Every later change is tested in the harness against the ranker it changes, and adopted only if it beats the ranker to beat. # Judgments from people and LLMs By the end of this chapter you can write judging guidelines, measure how well two judges agree with kappa and weighted kappa, read an LLM judge's errors from a confusion matrix, and decide where LLM judgments can stand in for people's and where they can't. Every score in this book rests on human judgments for [the store](#store-data), the online furniture and home-goods retailer whose search this book studies: WANDS's labels, each the majority vote of up to three trained annotators. Those judgments have holes: on the [test queries](#store-data), BM25's top 10s hold 22 products nobody judged ([Count unjudged products](#unjudged-results)). They also take people's time, and the agreement among WANDS's annotators rose over four months of calibration ([Expect judges to disagree](#judges-disagree-on-some-pairs)). A **large language model** (LLM), a model trained on large amounts of text to generate text in response to a prompt, can grade any query and product it is shown. Before the store lets one judge, it needs to know how often the LLM agrees with the store's judges, and whether LLM grades would lead to the same decisions about its rankers. The LLM's grades come from a **recording**: the script `recordings/llm_judgments.py` ran the open model `gemma4:12b-it-q4_K_M` once, locally, at temperature 0 with a fixed seed, and saved its answers, so the examples read the grades through `lib.recorded.llm_judgments()` without calling the model. We chose the model because it is open, under the Apache 2.0 licence, and runs locally, so anyone can rerun the recording. A JSON schema limits every answer to one of the three labels. The recording graded each pair with two prompts, `short` and `guidelines` ([Write judging guidelines](#write-judging-guidelines)), and took its pairs from two sets, which this chapter calls pools. The **train pool**, for measuring agreement, holds pairs that people judged for the train queries, drawn evenly by label: up to 4 of each label per query, 550 pairs in all. The **test pool** holds the pairs an LLM would grade to score the store's rankers: every product in the top 10 of [chapter 1's four rankers](#search-end-to-end-decisions) for each test query, judged by people or not, 590 pairs. The four are BM25, semantic retrieval on embeddings from `Snowflake/snowflake-arctic-embed-m-v1.5`, the hybrid of the two, and the hybrid reranked by the cross-encoder `Alibaba-NLP/gte-reranker-modernbert-base`. A measurement of the LLM judge holds for the pool it is taken on, so each one in this chapter names its pool. The scores that decide between rankers stay on human grades. The chapter's planned comparisons on the test queries are [chapter 3's three](#when-a-difference-is-established), rescored with each prompt's LLM grades at chapter 3's threshold of 0.0167: semantic retrieval against BM25, the hybrid against semantic retrieval, and the reranked hybrid against the hybrid. ## Write judging guidelines {#write-judging-guidelines} A **judgment** is a judge's label for one pair of a query and a product, and a **grade** is the number that stands for it. [Chapter 1](#store-data) defines a judgment as a person's label; this chapter widens it to an LLM's label too. Judgments depend on the instructions behind them. **Judging guidelines** say what each grade means, how to choose between grades, and what to do with the cases that fall between them. WANDS published the guidelines its annotators used, eight pages under the MIT licence ([annotation guidelines](https://github.com/wayfair/WANDS/raw/3b74dcf4ba29ab8ff3e6a50b5b09fc627cb882b5/Product%20Search%20Relevance%20Annotation%20Guidelines.pdf)). They list what a judge sees for each pair: the query, and the product's ID, name, image and web page. They define the three grades and add two notes. One asks the judge to understand the query first: "Strongly recommend google it first". The other covers misspellings: "please ignore the mis-spellings and proceed to annotate the example as-is". Then they teach by example, with a query, a product and a reason for each grade. A wire basket without dividers is partial for `wire basket with dividers`, one of the store's own train queries. A salon anti-fatigue mat is irrelevant to `salon chair`: "the customer looks for a salon chair, not a mat. This product is irrelevant to the query." Guidelines also change as judges use them. WANDS's authors report that their annotators' agreement rose over four months, and among their reasons they list a "Daily routine to discuss the conflicting annotations" and this one: "With the input from annotators, we refine and fine-tune annotation guidelines" ([Chen et al., 2022, section 5](https://easychair.org/publications/preprint/wm4R)). One web search engine's rater guidelines do the same at 182 pages, in their version of September 11, 2025 ([Search Quality Rater Guidelines](https://guidelines.raterhub.com/searchqualityevaluatorguidelines.pdf)). In the store, the two prompts of [the recording](#judgments-from-people-and-llms) are guidelines for the LLM. `short` gives WANDS's three definitions and nothing else. `guidelines` adds five rules. Print it as the recording sent it: Example `print-judging-guidelines` (Python 3.12.15 · numpy 2.5.4 · pandas 3.0.6), passing in Edition 2026.10.1: ```python from lib.recorded import manifest recording = manifest("llm-judgments") # the recording's model, settings and both prompts, verbatim short, guidelines = recording["prompts"]["short"], recording["prompts"]["guidelines"] print("model:", recording["model"]) print("what the model saw:", recording["sees"]) print("words in the short prompt:", len(short.split())) print("words in the guidelines prompt:", len(guidelines.split())) print() print(guidelines) ``` Output: ```text model: gemma4:12b-it-q4_K_M what the model saw: lib.store text plus attributes; no images words in the short prompt: 59 words in the guidelines prompt: 206 You judge product search results for an online furniture and home-goods store. A shopper typed the query and was shown the product. Decide how well the product matches what the shopper asked for. Exact: the product fully matches the search query. Partial: the product only matches the target entity of the query, but does not satisfy the modifiers for the query. Irrelevant: the product is not relevant to the query. Guidelines: - First find the target entity: the kind of product the query asks for (a sofa, a lamp, a rug). - Then find the modifiers: everything else the query asks for, such as colour, size, material, style, brand or a feature. - Exact needs the right kind of product and every modifier. Partial is the right kind of product with at least one modifier missing or contradicted. Irrelevant is the wrong kind of product, even when it shares words with the query, such as an accessory for the product asked for. - Read misspelled queries as the shopper meant them. - Judge only from the product's text and attributes. If a modifier isn't mentioned, decide from what is written, and don't assume it is met. Query: {query} Product: {product} Answer with the label in JSON. ``` The `guidelines` prompt runs to 206 words, against 59 for `short`. Finding the entity first, then the modifiers, spells out WANDS's definition of partial. The rule on accessories settles a case on which WANDS's own labels split: for `clear shower curtain`, WANDS lists one set of shower curtain hooks twice, once partial and once irrelevant ([Expect judges to disagree](#judges-disagree-on-some-pairs)). The rule on misspellings follows WANDS's note. The last rule has no counterpart in WANDS's guidelines: the store's data has no images, so the recording shows the LLM text only. Neither prompt has worked examples, which WANDS's guidelines rely on. **What to do:** write guidelines as definitions, a procedure, a worked example with a reason for each grade, and a rule for each case your judges have disagreed on. Log disagreements and revise the guidelines when one kind keeps coming back. Give an LLM judge the same guidelines as people, plus a rule for what it can't see, and then measure what it does, because a rule you write isn't a rule the judge follows. ## Measure agreement beyond chance with kappa {#agreement-beyond-chance} Before you trust a second judge, measure how often it agrees with the first. The share of pairs on which two judges give the same grade, the **observed agreement**, is the obvious number, and it flatters. Two judges who guessed independently would agree on some pairs by chance, and on more of them when both use one label often. **Cohen's kappa** corrects for that. It compares the observed agreement with the agreement expected if each judge kept its own label shares but chose independently of the other. A kappa of 1 means complete agreement, and 0 means "no agreement beyond what would be expected by chance" ([scikit-learn documentation](https://scikit-learn.org/1.9/modules/generated/sklearn.metrics.cohen_kappa_score.html)): ```text kappa = (observed - expected) / (1 - expected) expected = sum over grades g of (share of pairs judge A gives g) × (share of pairs judge B gives g) ``` Measure how often the LLM of [the recording](#judgments-from-people-and-llms) grades a pair as the store's judges did. Use the train pool, pairs people judged for train queries and drawn evenly by label, and the [`short` prompt](#write-judging-guidelines), WANDS's three definitions alone. Build the 3×3 table of counts by hand, compute kappa, and check it against scikit-learn's `cohen_kappa_score`: Example `kappa-by-hand` (Python 3.12.15 · bm25s 0.3.13 · numpy 2.5.4 · pandas 3.0.6 · scikit-learn 1.9.1 · scipy 1.18.1), passing in Edition 2026.10.1: ```python import numpy as np from sklearn.metrics import cohen_kappa_score from lib.agreement import graded_pairs pairs = graded_pairs("train") # up to 4 human-judged pairs per label per train query human, llm = pairs["human"].astype(int), pairs["short"].astype(int) grades = [2, 1, 0] # exact, partial, irrelevant counts = np.array([[((human == row) & (llm == column)).sum() for column in grades] for row in grades]) print("pairs:", counts.sum()) print("rows are the human grades 2, 1, 0 and columns the LLM's, short prompt") print(counts) observed = np.trace(counts) / counts.sum() # the share of pairs on the diagonal human_share, llm_share = counts.sum(axis=1) / counts.sum(), counts.sum(axis=0) / counts.sum() expected = (human_share * llm_share).sum() # agreement if the two judged independently kappa = (observed - expected) / (1 - expected) print(f"observed agreement, short prompt: {observed:.3f}") print(f"agreement expected by chance, short prompt: {expected:.3f}") print(f"kappa by hand, short prompt: {kappa:.3f}") print(f"kappa from scikit-learn, short prompt: {cohen_kappa_score(human, llm):.3f}") other = pairs["guidelines"].astype(int) print(f"observed agreement, guidelines prompt: {(human == other).mean():.3f}") print(f"kappa from scikit-learn, guidelines prompt: {cohen_kappa_score(human, other):.3f}") ``` Output: ```text pairs: 550 rows are the human grades 2, 1, 0 and columns the LLM's, short prompt [[102 85 9] [ 22 132 46] [ 3 42 109]] observed agreement, short prompt: 0.624 agreement expected by chance, short prompt: 0.337 kappa by hand, short prompt: 0.432 kappa from scikit-learn, short prompt: 0.432 observed agreement, guidelines prompt: 0.633 kappa from scikit-learn, guidelines prompt: 0.454 ``` The LLM and the store's judges give the same grade to a share of 0.624 of the pairs. Chance alone would produce 0.337, so kappa is 0.432, the same as scikit-learn's 0.432. The `guidelines` prompt agrees on 0.633 of the pairs, a kappa of 0.454. Is that good? Scales that call a kappa "moderate" or "substantial" are conventions from particular fields, and Artstein and Poesio's survey concludes that "deciding what counts as an adequate level of agreement for a specific purpose is still little more than a black art" ([Artstein and Poesio, 2008](https://aclanthology.org/J08-4004/)). Published figures give context, not a bar: WANDS's annotators agreed with each other at a kappa of 0.467 in their first two months and 0.826 in their fourth ([Chen et al., 2022, Table 2](https://easychair.org/publications/preprint/wm4R)). On a stratified sample of pairs from TREC, the Text REtrieval Conference's shared evaluations, Thomas et al. found that GPT-4's kappa against the assessors varied "from as low as 0.20 ... to 0.64" with the prompt, while "two groups of trained human assessors" reached 0.52 on TREC ad hoc documents judged a second time (Cormack et al., 1998, as reported by [Thomas et al., 2024](https://arxiv.org/abs/2309.10621)). Each figure comes from a different sample, and [kappa depends on the sample's label mix](#kappa-depends-on-label-mix). **What to do:** report kappa, with observed agreement beside it, never observed agreement alone. Measure it on pairs people judged for train queries, and label any threshold you use as a rule of thumb. With more than two judges or missing labels, use Krippendorff's alpha, which compares observed with chance disagreement and handles "Any number of observers, not just two" and "Incomplete or missing data" ([Krippendorff, 2011](https://www.asc.upenn.edu/sites/default/files/2021-03/Computing%20Krippendorff%27s%20Alpha-Reliability.pdf)). ## Give near misses partial credit with weighted kappa {#weighted-kappa-for-grades} [Kappa](#agreement-beyond-chance) counts every disagreement the same. The store's grades are ordered, exact above partial above irrelevant, and calling an exact product partial is a smaller error than calling it irrelevant. In nDCG@10 ([Score one ranking](#score-one-ranking)) it costs half as much gain: one grade instead of two. **Weighted kappa** gives each disagreement a weight by its distance and compares the weighted disagreement with the weighted disagreement expected by chance ([Artstein and Poesio, 2008, section 2.6](https://aclanthology.org/J08-4004/)). With **linear** weights, a disagreement one grade apart weighs 0.5 and one two grades apart weighs 1: ```text weighted kappa = 1 - (sum of weight × observed count) / (sum of weight × count expected by chance) ``` With every disagreement weighted 1, the formula gives plain kappa. scikit-learn's `cohen_kappa_score` computes all three kinds this way: weight 1 by default, the distance between grades with `weights="linear"`, and its square with `weights="quadratic"` ([scikit-learn source](https://raw.githubusercontent.com/scikit-learn/scikit-learn/1.9.1/sklearn/metrics/_classification.py)). Scaling the weights doesn't change the result. In the store, compute linear weighted kappa by hand for both [prompts](#write-judging-guidelines) on the train pool, the evenly drawn pairs that [Measure agreement beyond chance](#agreement-beyond-chance) uses, and check it against scikit-learn: Example `weighted-kappa-by-hand` (Python 3.12.15 · bm25s 0.3.13 · numpy 2.5.4 · pandas 3.0.6 · scikit-learn 1.9.1 · scipy 1.18.1), passing in Edition 2026.10.1: ```python import numpy as np from sklearn.metrics import cohen_kappa_score, confusion_matrix from lib.agreement import PROMPTS, graded_pairs pairs = graded_pairs("train") # up to 4 human-judged pairs per label per train query grades = [2, 1, 0] # exact, partial, irrelevant cost = np.abs(np.subtract.outer(grades, grades)) / 2 # 0 on the diagonal, 0.5 one grade apart, 1 two apart print("disagreement weights, rows and columns 2, 1, 0") print(cost) for prompt in PROMPTS: human, llm = pairs["human"].astype(int), pairs[prompt].astype(int) counts = confusion_matrix(human, llm, labels=grades) # rows human, columns LLM expected = np.outer(counts.sum(axis=1), counts.sum(axis=0)) / counts.sum() # counts if independent weighted = 1 - (cost * counts).sum() / (cost * expected).sum() near = cost == 0.5 # cells one grade apart print() print(f"pairs two grades apart, {prompt} prompt: {counts[0, 2] + counts[2, 0]}") print(f"disagreements one grade apart, share, {prompt} prompt: {counts[near].sum() / counts[cost > 0].sum():.3f}") print(f"one grade apart, share expected by chance, {prompt} prompt: {expected[near].sum() / expected[cost > 0].sum():.3f}") print(f"kappa, {prompt} prompt: {cohen_kappa_score(human, llm):.3f}") print(f"linear weighted kappa by hand, {prompt} prompt: {weighted:.3f}") print(f"linear weighted kappa from scikit-learn, {prompt} prompt: " f"{cohen_kappa_score(human, llm, weights='linear'):.3f}") ``` Output: ```text disagreement weights, rows and columns 2, 1, 0 [[0. 0.5 1. ] [0.5 0. 0.5] [1. 0.5 0. ]] pairs two grades apart, short prompt: 12 disagreements one grade apart, share, short prompt: 0.942 one grade apart, share expected by chance, short prompt: 0.742 kappa, short prompt: 0.432 linear weighted kappa by hand, short prompt: 0.523 linear weighted kappa from scikit-learn, short prompt: 0.523 pairs two grades apart, guidelines prompt: 30 disagreements one grade apart, share, guidelines prompt: 0.851 one grade apart, share expected by chance, guidelines prompt: 0.664 kappa, guidelines prompt: 0.454 linear weighted kappa by hand, guidelines prompt: 0.531 linear weighted kappa from scikit-learn, guidelines prompt: 0.531 ``` For `short`, weighting raises kappa from 0.432 to 0.523, and for `guidelines` from 0.454 to 0.531. Both match scikit-learn. Weighting raises both because more of the disagreements sit one grade apart than chance would put there: a share of 0.942 against 0.742 for `short`, and 0.851 against 0.664 for `guidelines`. The prompts differ in their far misses: 12 pairs for `short` and 30 for `guidelines`. Whether either prompt agrees better is a question for [an interval](#prompt-sensitivity). **What to do:** for ordered grades, report linear weighted kappa next to plain kappa, and count the far misses separately. Weight each disagreement by the gap in gain it costs. With the book's linear gains, that's linear weights. With exponential gains of 0, 1 and 3 ([Choose a metric](#choose-a-ranking-metric)), the gaps are 1, 2 and 3, which neither of scikit-learn's options gives, so build that matrix as the example builds the linear one. ## Read kappa against the label mix {#kappa-depends-on-label-mix} [Kappa's](#agreement-beyond-chance) correction depends on how often each judge uses each label, so the same judge gets a different kappa on a sample with a different mix of labels. When one label dominates, observed agreement can be high while kappa stays low. Artstein and Poesio call this the prevalence problem: "coders may agree on a high proportion of items while producing annotations that are indeed correct to a high degree, yet the reliability coefficients remain low" ([Artstein and Poesio, 2008](https://aclanthology.org/J08-4004/)). [The recording](#judgments-from-people-and-llms) drew the store's train pool evenly, up to 4 pairs of each label per train query, so it holds more exact and irrelevant pairs than the store's judgments do. Weight each pair so that the pool has the label mix of all the train queries' judgments, and measure again: Example `kappa-under-two-label-mixes` (Python 3.12.15 · bm25s 0.3.13 · numpy 2.5.4 · pandas 3.0.6 · scikit-learn 1.9.1 · scipy 1.18.1), passing in Edition 2026.10.1: ```python import numpy as np from sklearn.metrics import cohen_kappa_score from lib.agreement import LABELS, PROMPTS, graded_pairs from lib.store import load_labels, load_queries pairs = graded_pairs("train") # drawn evenly: up to 4 pairs per label per train query drawn = pairs["human"].value_counts(normalize=True) labels = load_labels() natural = labels[labels["query_id"].isin(load_queries("train").index)]["grade"].value_counts(normalize=True) top = graded_pairs("test")["human"].dropna().value_counts(normalize=True) # the rankers' top 10s, judged pairs for grade, label in LABELS.items(): print(f"{label} share, as drawn: {drawn[grade]:.3f}") print(f"{label} share, in the train judgments: {natural[grade]:.3f}") print(f"{label} share, judged test-pool pairs: {top[grade]:.3f}") weight = pairs["human"].map(natural / drawn).astype(float) # count each pair as if drawn in that mix kappas = {} for prompt in PROMPTS: human, llm = pairs["human"].astype(int), pairs[prompt].astype(int) print() print(f"observed agreement, {prompt} prompt, as drawn: {(human == llm).mean():.3f}") print(f"kappa, {prompt} prompt, as drawn: {cohen_kappa_score(human, llm):.3f}") print(f"observed agreement, {prompt} prompt, train judgments' mix: {np.average(human == llm, weights=weight):.3f}") kappas[prompt] = cohen_kappa_score(human, llm), cohen_kappa_score(human, llm, sample_weight=weight) print(f"kappa, {prompt} prompt, train judgments' mix: {kappas[prompt][1]:.3f}") print() for position, mix in enumerate(["as drawn", "train judgments' mix"]): print(f"higher kappa, {mix}: {max(PROMPTS, key=lambda prompt: kappas[prompt][position])}") ``` Output: ```text exact share, as drawn: 0.356 exact share, in the train judgments: 0.234 exact share, judged test-pool pairs: 0.543 partial share, as drawn: 0.364 partial share, in the train judgments: 0.613 partial share, judged test-pool pairs: 0.427 irrelevant share, as drawn: 0.280 irrelevant share, in the train judgments: 0.153 irrelevant share, judged test-pool pairs: 0.030 observed agreement, short prompt, as drawn: 0.624 kappa, short prompt, as drawn: 0.432 observed agreement, short prompt, train judgments' mix: 0.635 kappa, short prompt, train judgments' mix: 0.370 observed agreement, guidelines prompt, as drawn: 0.633 kappa, guidelines prompt, as drawn: 0.454 observed agreement, guidelines prompt, train judgments' mix: 0.573 kappa, guidelines prompt, train judgments' mix: 0.345 higher kappa, as drawn: guidelines higher kappa, train judgments' mix: short ``` Partial makes up 0.364 of the pool as drawn, against 0.613 of the train judgments. In the judgments' mix, the `short` prompt's observed agreement goes from 0.624 to 0.635, while its kappa goes from 0.432 to 0.370. For `guidelines`, observed agreement goes from 0.633 to 0.573, and kappa from 0.454 to 0.345. The prompt with the higher kappa is `guidelines` as drawn and `short` in the judgments' mix. Published numbers carry their mix with them. The one study of LLM judges on WANDS we found "sampled 5000 Q-P pairs evenly across the three class labels" and reports accuracy from 0.726 for the best open model, with 70 billion parameters, to 0.837 for the best closed one, each in its best setting, with 16 worked examples retrieved for every pair ([Sachdev et al., 2025, Table 3](https://arxiv.org/html/2502.15990)). The store's prompts have no worked examples. That accuracy is observed agreement on a balanced sample, as the train pool's is, and neither number transfers to the label mix of the store's judgments. The pairs the store will ask an LLM to judge are the products its rankers put in their top 10s, the test pool. Of its judged pairs, a share of 0.543 are exact, 0.427 partial and 0.030 irrelevant. That is a third mix, and a different kind of pair, which [Read the direction of the LLM's errors](#read-llm-error-direction) measures. **What to do:** report the label mix with every kappa, and never compare kappas from samples with different mixes, your own or published ones. We recommend drawing the agreement sample the way you'll use the judge: if it will grade your rankers' top 10s, sample pairs from top 10s on train queries. ## Read the direction of the LLM's errors {#read-llm-error-direction} [Kappa](#agreement-beyond-chance) says how much two judges agree, not how they differ. A **confusion matrix** does: a table of counts with one row for each of the store's grades and one column for each of the LLM's, both in the order exact, partial, irrelevant. Above the diagonal, the LLM gave a lower grade than the store's judges; below it, a higher one. A judge whose errors fall mostly below is **lenient**, and mostly above, **strict**. Several studies find LLM judges lenient. At TREC 2024's RAG track, which evaluates retrieval-augmented generation (answers written from retrieved text), "Overall, human assessors appear to be stricter than UMBRELA in applying relevance criteria", UMBRELA being an open-source LLM judge ([Upadhyay et al., 2024a](https://arxiv.org/abs/2411.08275)). Arabzadeh and Clarke report that UMBRELA, built on GPT-4o ([Upadhyay et al., 2024b](https://arxiv.org/abs/2406.06519)), labelled "over 26% more documents as perfectly relevant (3,063 vs. 2,429)" than people did across TREC's Deep Learning tracks of 2019 to 2021 ([Arabzadeh and Clarke, 2025a](https://arxiv.org/abs/2504.12558)), and Yu et al. report LLM judges that "consistently assign inflated relevance scores" ([Yu et al., 2026](https://arxiv.org/abs/2602.17170)). Not every study agrees. With one of their 32 prompts, on a sample stratified by label, Thomas et al. found that "the LLM is more likely to say ‘not relevant’ than were TREC assessors (44% vs 33%)" ([Thomas et al., 2024](https://arxiv.org/abs/2309.10621)). In the store, build the matrix for both prompts on both [pools](#judgments-from-people-and-llms): the train pool, drawn evenly by label, and the judged pairs of the test pool, the top 10s of [chapter 1's four rankers](#search-end-to-end-decisions) that an LLM would grade to score them. The test-pool matrices are a report on test queries, not a choice of prompt: Example `llm-confusion-matrix` (Python 3.12.15 · bm25s 0.3.13 · numpy 2.5.4 · pandas 3.0.6 · scikit-learn 1.9.1 · scipy 1.18.1), passing in Edition 2026.10.1: ```python import pandas as pd from sklearn.metrics import cohen_kappa_score from lib.agreement import LABELS, PROMPTS, graded_pairs order = list(LABELS.values()) # rows: the store's grades; columns: the LLM's; both exact first for pool in ["train", "test"]: pairs = graded_pairs(pool).dropna(subset=["human"]) # in the test pool, the pairs people judged human = pairs["human"].astype(int) print(f"\njudged pairs, {pool} pool: {len(pairs)}\nmean human grade, {pool} pool: {human.mean():.3f}") for prompt in PROMPTS: llm, name = pairs[prompt].astype(int), f"{pool} pool, {prompt} prompt" table = pd.crosstab(human.map(LABELS).rename("human"), llm.map(LABELS).rename(f"llm, {name}")) table = table.reindex(index=order, columns=order, fill_value=0) errors = {f"{row} called {column}": table.at[row, column] for row in order for column in order if row != column} largest = max(errors, key=errors.get) print(table) print(f"kappa, {name}: {cohen_kappa_score(human, llm):.3f}\nmean llm grade, {name}: {llm.mean():.3f}") print(f"llm grade higher than human, {name}: {(llm > human).sum()}") print(f"llm grade lower than human, {name}: {(llm < human).sum()}") print(f"stricter than the store's judges, {name}: {'yes' if (llm < human).sum() > (llm > human).sum() else 'no'}") print(f"largest error, {name}: {largest} ({errors[largest]})") ``` Output: ```text judged pairs, train pool: 550 mean human grade, train pool: 1.076 llm, train pool, short prompt exact partial irrelevant human exact 102 85 9 partial 22 132 46 irrelevant 3 42 109 kappa, train pool, short prompt: 0.432 mean llm grade, train pool, short prompt: 0.933 llm grade higher than human, train pool, short prompt: 67 llm grade lower than human, train pool, short prompt: 140 stricter than the store's judges, train pool, short prompt: yes largest error, train pool, short prompt: exact called partial (85) llm, train pool, guidelines prompt exact partial irrelevant human exact 118 53 25 partial 30 98 72 irrelevant 5 17 132 kappa, train pool, guidelines prompt: 0.454 mean llm grade, train pool, guidelines prompt: 0.862 llm grade higher than human, train pool, guidelines prompt: 52 llm grade lower than human, train pool, guidelines prompt: 150 stricter than the store's judges, train pool, guidelines prompt: yes largest error, train pool, guidelines prompt: partial called irrelevant (72) judged pairs, test pool: 564 mean human grade, test pool: 1.512 llm, test pool, short prompt exact partial irrelevant human exact 252 53 1 partial 57 177 7 irrelevant 2 6 9 kappa, test pool, short prompt: 0.571 mean llm grade, test pool, short prompt: 1.521 llm grade higher than human, test pool, short prompt: 65 llm grade lower than human, test pool, short prompt: 61 stricter than the store's judges, test pool, short prompt: no largest error, test pool, short prompt: partial called exact (57) llm, test pool, guidelines prompt exact partial irrelevant human exact 261 41 4 partial 76 133 32 irrelevant 3 4 10 kappa, test pool, guidelines prompt: 0.470 mean llm grade, test pool, guidelines prompt: 1.521 llm grade higher than human, test pool, guidelines prompt: 83 llm grade lower than human, test pool, guidelines prompt: 77 stricter than the store's judges, test pool, guidelines prompt: no largest error, test pool, guidelines prompt: partial called exact (76) ``` On the train pool, is the LLM stricter than the store's judges? With `short`: yes. With `guidelines`: yes. With `short`, its grade is lower than the judges' on 140 pairs and higher on 67; with `guidelines`, lower on 150 and higher on 52. The largest error is exact called partial (85) with `short`, and partial called irrelevant (72) with `guidelines`. On the test pool, the answer is no with `short` and no with `guidelines`. With `short`, the LLM's grade is higher on 65 pairs and lower on 61; with `guidelines`, higher on 83 and lower on 77. Its mean grade is 1.521 with `short` and 1.521 with `guidelines`, against 1.512 for the judges. The largest error is partial called exact (57) with `short` and partial called exact (76) with `guidelines`. Kappa is 0.571 and 0.470, against 0.432 and 0.454 on the train pool. The direction, the largest error and kappa all change with the pool, as [the label mix](#kappa-depends-on-label-mix) warns. [Compare what the LLM saw](#what-the-llm-saw) finds one pattern that fits both pools. Nor does the direction of the errors say what happens to a ranker's score. A lenient judge counts more products as relevant, which raises precision@10 ([Count the relevant products](#count-relevant-in-top-k)). nDCG@10 divides by an ideal top 10 built from the same grades ([Score one ranking](#score-one-ranking)), so raising grades can raise the ideal too, and the net effect can go either way and differ by ranker. [The check of the rankers' order](#system-order-human-versus-llm) measures it on the test pool. **What to do:** read the confusion matrix before you read kappa, on pairs drawn the way you'll use the judge, and name the direction and the largest error for that pool. Don't carry a direction from one pool to another, and don't predict a ranker's nDCG@10 from it. Compare rankers with each other under the same grades, and never compare a score under LLM grades with a score under human grades. ## Compare what the LLM saw with what the judges saw {#what-the-llm-saw} A judge can grade only the evidence it is shown. WANDS's judges saw, for each pair, the query and the product's ID, name, image and web page, and the guidelines told them to understand the query first: "Strongly recommend google it first" ([annotation guidelines](https://github.com/wayfair/WANDS/raw/3b74dcf4ba29ab8ff3e6a50b5b09fc627cb882b5/Product%20Search%20Relevance%20Annotation%20Guidelines.pdf)). The recording showed the LLM each product's name, class, category, description and attributes, as `lib.agreement.as_the_llm_saw` writes them out. The store's sample cuts descriptions to about 400 characters ([data card](../examples/data/wands/README.md)) and holds no images, so the recording sent the LLM text only. Some disagreement may come from this gap rather than from how well the LLM judges; that is our inference. One consequence can be tested: if the LLM grades what is written, it should give a lower grade to exact products whose text doesn't say what the query asks for, and a higher grade to partial products whose text does. Take one of the store's train pairs, then the test query `clear shower curtain`, then every judged exact and partial pair in both pools ([the recording](#judgments-from-people-and-llms)). The example looks for each query word in what the LLM saw, split into lowercase words as [chapter 2's check](#relevance-is-a-judgment) does: Example `llm-grades-follow-written-words` (Python 3.12.15 · bm25s 0.3.13 · numpy 2.5.4 · pandas 3.0.6 · scipy 1.18.1), passing in Edition 2026.10.1: ```python import re from lib.agreement import PROMPTS, as_the_llm_saw, graded_pairs from lib.store import load_products, load_queries def words(text): """Lowercase word tokens, as chapter 2 matches query words against product text.""" return set(re.findall(r"\w+", text.lower())) products, queries = load_products(), load_queries() print("query:", queries.at[163, "query"]) # a train query print(as_the_llm_saw(products.loc[20718])) print("grades from people, short, guidelines:", *graded_pairs("train").set_index(["query_id", "product_id"]).loc[(163, 20718)]) curtains = graded_pairs("test").query("query_id == 246") # clear shower curtain: every product in a top 10 says = curtains["product_id"].map(lambda product_id: "clear" in words(as_the_llm_saw(products.loc[product_id]))) for kind, group in [("say clear", curtains[says]), ("never say clear", curtains[~says])]: print(f"\nclear shower curtain products that {kind}: {len(group)}") for column, judge in [("human", "people"), ("short", "the short prompt"), ("guidelines", "the guidelines prompt")]: print(f"products that {kind}, graded exact by {judge}: {(group[column] == 2).sum()}") for pool in ["train", "test"]: pairs = graded_pairs(pool).dropna(subset=["human"]) # in the test pool, the pairs people judged pairs["every word"] = [words(queries.at[query_id, "query"]) <= words(as_the_llm_saw(products.loc[product_id])) for query_id, product_id in zip(pairs["query_id"], pairs["product_id"])] for label, grade, every_word, direction in [("exact", 2, False, "lower"), ("exact", 2, True, "lower"), ("partial", 1, True, "higher"), ("partial", 1, False, "higher")]: group = pairs[(pairs["human"] == grade) & (pairs["every word"] == every_word)] kind = f"{label} {pool} pairs {'with every query word' if every_word else 'missing a query word'}" print(f"\n{kind}: {len(group)}") for prompt in PROMPTS: moved = group[prompt] < grade if direction == "lower" else group[prompt] > grade print(f"share graded {direction}, {prompt} prompt, {kind}: {moved.mean():.3f}") ``` Output: ```text query: dual monitor stand Name: abramowski monitor stand Class: Desktop Organizers & Desk Pads Category: Description: the monitor stand riser can help ! Attributes: color: white; material: wood; style: modern; width: 23.5; height: 4.64; depth: 9.2 grades from people, short, guidelines: 2 1 1 clear shower curtain products that say clear: 9 products that say clear, graded exact by people: 5 products that say clear, graded exact by the short prompt: 7 products that say clear, graded exact by the guidelines prompt: 7 clear shower curtain products that never say clear: 9 products that never say clear, graded exact by people: 1 products that never say clear, graded exact by the short prompt: 0 products that never say clear, graded exact by the guidelines prompt: 0 exact train pairs missing a query word: 149 share graded lower, short prompt, exact train pairs missing a query word: 0.584 share graded lower, guidelines prompt, exact train pairs missing a query word: 0.483 exact train pairs with every query word: 47 share graded lower, short prompt, exact train pairs with every query word: 0.149 share graded lower, guidelines prompt, exact train pairs with every query word: 0.128 partial train pairs with every query word: 8 share graded higher, short prompt, partial train pairs with every query word: 0.875 share graded higher, guidelines prompt, partial train pairs with every query word: 0.875 partial train pairs missing a query word: 192 share graded higher, short prompt, partial train pairs missing a query word: 0.078 share graded higher, guidelines prompt, partial train pairs missing a query word: 0.120 exact test pairs missing a query word: 128 share graded lower, short prompt, exact test pairs missing a query word: 0.305 share graded lower, guidelines prompt, exact test pairs missing a query word: 0.281 exact test pairs with every query word: 178 share graded lower, short prompt, exact test pairs with every query word: 0.084 share graded lower, guidelines prompt, exact test pairs with every query word: 0.051 partial test pairs with every query word: 42 share graded higher, short prompt, partial test pairs with every query word: 0.548 share graded higher, guidelines prompt, partial test pairs with every query word: 0.667 partial test pairs missing a query word: 199 share graded higher, short prompt, partial test pairs missing a query word: 0.171 share graded higher, guidelines prompt, partial test pairs missing a query word: 0.241 ``` For `dual monitor stand`, a train query, the store's judges called `abramowski monitor stand` exact. Its grades from the judges, `short` and `guidelines`, in that order, are 2 1 1, and nothing the LLM saw says the stand holds two monitors. For `clear shower curtain`, of the 9 test-pool products that never say `clear` anywhere the LLM could read, the store's judges called 1 exact, `short` 0 and `guidelines` 0. Of the 9 that say it, the judges called 5 exact, `short` 7 and `guidelines` 7. For this query, the LLM's exact grades follow the word. Counted over the pools, the pattern holds. With `short`, exact pairs whose text misses a query word get a lower grade from the LLM at a share of 0.584 on the train pool and 0.305 on the test pool, against 0.149 and 0.084 for exact pairs whose text has every word. On the test pool, partial pairs whose text has every query word get a higher grade at a share of 0.548, against 0.171 for those that miss a word. On the test pool, `guidelines` splits the same way: 0.281 against 0.051 for exact pairs, and 0.667 against 0.241 for partial ones. The pools differ in which case is common. The train pool holds 149 exact pairs that miss a word and 8 partial pairs that have every word. The test pool holds 128 and 42. By our reading, that is why [the error direction](#read-llm-error-direction) differs between the pools. The check is rough: a plural or a misspelling counts as a missing word too. The `guidelines` prompt asks for the demotions: "If a modifier isn't mentioned, decide from what is written, and don't assume it is met." The `short` prompt has no such rule, and demotes as well. **What to do:** give an LLM judge every field your judges relied on that you can turn into text, such as attributes and the full description, and write down what it can't see. Before you count a disagreement against the LLM, check whether the evidence was in what it saw. When your judges' evidence is visual, measure whether the LLM grades exact products lower, as this one does on both pools, or give it the images if your model accepts them. ## Measure how much the prompt changes the labels {#prompt-sensitivity} An LLM judge is a model and a prompt together. Across 42 paraphrases of one prompt, Thomas et al. found "a large range, from mean κ = 0.50 (moderate agreement) to mean κ = 0.72", and warn that one prompt's result "should be taken as a single sample from a wider range of potential performance" ([Thomas et al., 2024](https://arxiv.org/abs/2309.10621)). Across three LLMs and 72 prompts on TREC's Deep Learning collections, Arabzadeh and Clarke found that "Binary and pairwise methods exhibit the least sensitivity to input prompts", while "graded relevance judgments are highly sensitive to prompt variations", the store's kind. Prompts varied between LLMs, "with no single prompt consistently excelling across all models", though "for graded assessments, UMBRELA consistently demonstrated high performance across different LLMs" ([Arabzadeh and Clarke, 2025b](https://arxiv.org/abs/2504.12408)). In the store, the two prompts share their definitions, and `guidelines` adds five rules ([Write judging guidelines](#write-judging-guidelines)). Count their disagreements on each pool ([the recording](#judgments-from-people-and-llms)), and ask whether the train pool can tell which prompt agrees better with the store's judges. Pairs from the same query aren't independent, so the example's interval resamples whole train queries, as [chapter 3's bootstrap](#report-the-size-of-a-gap) resamples queries: Example `prompt-disagreement` (Python 3.12.15 · bm25s 0.3.13 · numpy 2.5.4 · pandas 3.0.6 · scikit-learn 1.9.1 · scipy 1.18.1), passing in Edition 2026.10.1: ```python import numpy as np import pandas as pd from sklearn.metrics import cohen_kappa_score from lib.agreement import LABELS, graded_pairs from lib.store import load_products, load_queries def kappa_gap(sample): """Kappa with people under the guidelines prompt, minus kappa under the short prompt.""" human = sample["human"].astype(int) return cohen_kappa_score(human, sample["guidelines"].astype(int)) - cohen_kappa_score(human, sample["short"].astype(int)) order = list(LABELS.values()) # exact, partial, irrelevant for pool in ["train", "test"]: pairs = graded_pairs(pool) print(f"\n{pool} pairs where the prompts disagree: {(pairs['short'] != pairs['guidelines']).sum()} of {len(pairs)}\n" f"kappa between the prompts, {pool} pool: {cohen_kappa_score(pairs['short'], pairs['guidelines']):.3f}") table = pd.crosstab(pairs["short"].map(LABELS).rename(f"short, {pool}"), pairs["guidelines"].map(LABELS).rename("guidelines")) table = table.reindex(index=order, columns=order, fill_value=0) cells = {f"{row} under short, {column} under guidelines": table.at[row, column] for row in order for column in order if row != column} print(f"{table}\nlargest disagreement, {pool} pool: {max(cells, key=cells.get)} ({max(cells.values())})") pairs, products, queries = graded_pairs("train"), load_products(), load_queries() moved = pairs[(pairs["short"] == 1) & (pairs["guidelines"] == 0)] # partial under one prompt, irrelevant under the other print("\n".join(f"moved train pairs judged {label} by people: {(moved['human'] == grade).sum()}" for grade, label in LABELS.items())) for query_id, product_id in moved[["query_id", "product_id"]].drop_duplicates("query_id").head(5).itertuples(index=False): print(f" {queries.at[query_id, 'query']:<28} {products.at[product_id, 'name'][:50]}") rng, train_queries = np.random.default_rng(2026), pairs["query_id"].unique() by_query = {query_id: group for query_id, group in pairs.groupby("query_id")} gaps = [kappa_gap(pd.concat([by_query[query_id] for query_id in rng.choice(train_queries, len(train_queries))])) for _ in range(2000)] # resample whole train queries low, high = np.percentile(gaps, [2.5, 97.5]) print(f"kappa with people, guidelines minus short: {kappa_gap(pairs):.3f}\n95% interval over resampled train queries: {low:.3f} to {high:.3f}") ``` Output: ```text train pairs where the prompts disagree: 122 of 550 kappa between the prompts, train pool: 0.668 guidelines exact partial irrelevant short, train exact 122 4 1 partial 31 153 75 irrelevant 0 11 153 largest disagreement, train pool: partial under short, irrelevant under guidelines (75) test pairs where the prompts disagree: 93 of 590 kappa between the prompts, test pool: 0.712 guidelines exact partial irrelevant short, test exact 303 11 1 partial 42 174 37 irrelevant 0 2 20 largest disagreement, test pool: partial under short, exact under guidelines (42) moved train pairs judged exact by people: 18 moved train pairs judged partial by people: 31 moved train pairs judged irrelevant by people: 26 toddler couch fold out eulah 79 '' cotton rolled arm sofa bed 70s inspired furniture angels moroccan window curtains , oriental wavy cu 7qt slow cooker berghoff international eclipse 7 qt non-stick cove gnome fairy garden fairy house tea pot fairy garden wall sconce with usb port 2 piece wall sconce set kappa with people, guidelines minus short: 0.022 95% interval over resampled train queries: -0.042 to 0.086 ``` The prompts disagree on 122 of 550 train pairs and on 93 of 590 test pairs. Their kappa with each other is 0.668 on the train pool and 0.712 on the test pool: a second prompt is a second judge. The largest disagreement differs by pool. On the train pool it is partial under short, irrelevant under guidelines (75); on the test pool, the rankers' top 10s, it is partial under short, exact under guidelines (42). Of the train pairs that `short` calls partial and `guidelines` irrelevant, the store's judges called 26 irrelevant, siding with `guidelines`, 31 partial, siding with `short`, and 18 exact. Among the products the example lists are a sofa bed for `toddler couch fold out` and window curtains for `70s inspired furniture`. The recording keeps labels, not reasons, so from names like these we infer that the move comes from the `guidelines` rule that calls the wrong kind of product irrelevant. On kappa with the store's judges, `guidelines` leads `short` by 0.022, and the 95% interval over resampled train queries, from -0.042 to 0.086, includes 0. The train pool can't tell the prompts apart, so the store keeps both. **What to do:** treat the prompt as part of the judge. Record every prompt word for word with the model version, as the recording's manifest does, and the settings and answer schema too, which this recording keeps in its script. Treat a change of prompt as a different judge whose agreement you measure again. Run at least two prompts, and trust a conclusion from LLM grades only when both give it. Choose one prompt over another on train pairs, and only when the interval for their difference excludes 0. ## Fill unjudged pairs with LLM grades {#fill-unjudged-with-llm-grades} We consider filling the holes in the human judgments the least risky use of an LLM judge. Keep every human grade, and use the LLM only for pairs nobody judged, which a metric would otherwise count as grade 0 ([Count unjudged products](#unjudged-results)). The LLM then decides only what was a guess before. The store's test pool, every product in the top 10 of [chapter 1's four rankers](#search-end-to-end-decisions) for each test query, holds 590 pairs, of which 26 are unjudged. BM25's top 10s hold 22 of them, against 3 for semantic retrieval and 6 for the hybrid. Fill them under each [prompt](#write-judging-guidelines), rescore the four rankers on nDCG@10 ([Score one ranking](#score-one-ranking)), and rerun [chapter 3's planned comparisons](#when-a-difference-is-established) on the filled grades: Example `fill-unjudged-pairs` (Python 3.12.15 · bm25s 0.3.13 · numpy 2.5.4 · pandas 3.0.6 · scipy 1.18.1), passing in Edition 2026.10.1: ```python from lib.agreement import LABELS, PLANNED, PROMPTS, chapter_one_runs, fill_unjudged, graded_pairs, llm_qrels, planned_verdicts from lib.metrics import evaluate from lib.store import load_products, load_queries, qrels def order(scores): """Rankers by mean nDCG@10 over the test queries, best first.""" return sorted(scores, key=lambda name: scores[name].mean(), reverse=True) products, queries, people = load_products(), load_queries("test"), qrels("test") runs = chapter_one_runs(products, queries) pairs = graded_pairs("test") # every product in the top 10 of the four runs, per test query unjudged = pairs[pairs["human"].isna()] print(f"pairs in the test pool: {len(pairs)}\nof them, unjudged: {len(unjudged)}") for name, run in runs.items(): missing = sum(product_id not in people[query_id] for query_id, ranking in run.items() for product_id in ranking[:10]) print(f"unjudged products in the top 10s, {name}: {missing}") lights = unjudged[unjudged["query_id"] == 133] # outdoor light fixtures print(f"unjudged products for outdoor light fixtures: {len(lights)}\n" f"of them, with bulb in the name: {lights['product_id'].map(products['name']).str.contains('bulb').sum()}") threshold = 0.05 / len(PLANNED) # chapter 3's rule human = {name: evaluate(run, people)["ndcg@10"] for name, run in runs.items()} # per-query scores print(f"order, people's grades only: {' > '.join(order(human))}") for prompt in PROMPTS: counts = ", ".join(str((unjudged[prompt] == grade).sum()) for grade in LABELS) # grades 2, 1 and 0 print(f"\nunjudged pairs graded exact, partial, irrelevant, {prompt} prompt: {counts}") print(f"unjudged bulbs graded irrelevant, {prompt} prompt: {(lights[prompt] == 0).sum()}") filled = {name: evaluate(run, fill_unjudged(people, llm_qrels(prompt)))["ndcg@10"] for name, run in runs.items()} print("\n".join(f"{name} ndcg@10, filled by the {prompt} prompt: {filled[name].mean():.3f}" for name in runs)) print(f"order, filled by the {prompt} prompt: {' > '.join(order(filled))}") moved = [name for name, place in zip(order(filled), order(human)) if name != place] print(f"rankers that change place when filled, {prompt} prompt: {', '.join(moved) or 'none'}") changed = [pair for pair, established in planned_verdicts(filled, threshold).items() if established != planned_verdicts(human, threshold)[pair]] print(f"verdicts that change when filled, {prompt} prompt: {'; '.join(' minus '.join(pair) for pair in changed) or 'none'}") ``` Output: ```text pairs in the test pool: 590 of them, unjudged: 26 unjudged products in the top 10s, bm25: 22 unjudged products in the top 10s, semantic: 3 unjudged products in the top 10s, hybrid: 6 unjudged products in the top 10s, reranked: 6 unjudged products for outdoor light fixtures: 8 of them, with bulb in the name: 8 order, people's grades only: semantic > hybrid > reranked > bm25 unjudged pairs graded exact, partial, irrelevant, short prompt: 4, 17, 5 unjudged bulbs graded irrelevant, short prompt: 0 bm25 ndcg@10, filled by the short prompt: 0.754 semantic ndcg@10, filled by the short prompt: 0.833 hybrid ndcg@10, filled by the short prompt: 0.821 reranked ndcg@10, filled by the short prompt: 0.813 order, filled by the short prompt: semantic > hybrid > reranked > bm25 rankers that change place when filled, short prompt: none verdicts that change when filled, short prompt: none unjudged pairs graded exact, partial, irrelevant, guidelines prompt: 5, 9, 12 unjudged bulbs graded irrelevant, guidelines prompt: 7 bm25 ndcg@10, filled by the guidelines prompt: 0.745 semantic ndcg@10, filled by the guidelines prompt: 0.834 hybrid ndcg@10, filled by the guidelines prompt: 0.821 reranked ndcg@10, filled by the guidelines prompt: 0.813 order, filled by the guidelines prompt: semantic > hybrid > reranked > bm25 rankers that change place when filled, guidelines prompt: none verdicts that change when filled, guidelines prompt: none ``` The prompts disagree about the unjudged pairs. Counted as exact, partial and irrelevant, `short` grades them 4, 17, 5, and `guidelines` 5, 9, 12. They part on the unjudged products for `outdoor light fixtures` in BM25's top 10 ([Score every test query](#score-every-test-query)). Of those 8, 8 have `bulb` in their names, and `guidelines` calls 7 of them irrelevant and `short` 0. Chapter 1 argued that a bulb isn't a light fixture. Which reading the store's judges would take, nobody knows until they judge the bulbs. With the holes filled, BM25 scores 0.754 with `short` and 0.745 with `guidelines`, against 0.725 on human grades alone, and semantic retrieval scores 0.833 and 0.834, against 0.830. The order of the four rankers on human grades alone is `semantic > hybrid > reranked > bm25`. The rankers that change place when `short` fills the holes are none, and when `guidelines` does, none. The verdicts of the planned comparisons that change are none with `short` and none with `guidelines`. Under human grades, the rankers' mean judged@10, the share of each top 10 that people judged, runs from 0.927 for BM25 to 0.990 for semantic retrieval ([Compare runs in a harness](#compare-runs-in-a-harness)); filled, every product in a top 10 has a grade. How far can the filled grades be trusted? The judged pairs of the same pool are the closest evidence: there, the LLM's kappa with the store's judges is 0.571 for `short` and 0.470 for `guidelines`. Its grade is lower than the judges' on 61 pairs and higher on 65 with `short`, and lower on 77 and higher on 83 with `guidelines` ([Read the direction of the LLM's errors](#read-llm-error-direction)). **What to do:** fill the unjudged pairs in your rankers' top 10s with LLM grades as a diagnostic, under every prompt you run, and report the filled score beside the human-only one with the number of pairs filled. If filling changes the order of two rankers or a verdict, have people judge those pairs. Never let a filled grade replace a human one. ## Check whether LLM grades order the rankers the same way {#system-order-human-versus-llm} Two sets of judgments can disagree on many labels and still rank systems alike. Voorhees scored TREC runs with different sets of relevance judgments and found that "Very high correlations were found among the rankings of systems produced using different relevance judgment sets" ([Voorhees, 1998](https://people.eng.unimelb.edu.au/ammoffat/sigir98/abstracts/voorhees.html)). LLM judges show the same split. Across 42 sets of LLM labels for TREC's 2023 Deep Learning track, kappa ran from about 0.06 to 0.29 while the correlation of system rankings stayed between about 0.83 and 0.95 ([Rahmani et al., 2025, Table 3](https://arxiv.org/abs/2502.13908)). That correlation is **Kendall's tau**. For every pair of systems, ask whether two orderings put them the same way round: **concordant** if they do, **discordant** if not. Tau is the difference between the two counts divided by the number of pairs, from 1 for the same order to −1 for the reverse. SciPy's `kendalltau` computes τ_b, a form that also handles ties ([SciPy documentation](https://docs.scipy.org/doc/scipy-1.18.0/reference/generated/scipy.stats.kendalltau.html)). Rankings with a tau of 0.9 or more "are often considered ‘effectively equivalent’ ... at least empirically", in Yilmaz et al.'s words ([Yilmaz et al., 2008](https://www.ccs.neu.edu/home/jaa/tmp/RankCorrelation.pdf)), and Sanderson and Soboroff found that "basing decisions on such thresholds is not as reliable as has been assumed" ([Sanderson and Soboroff, 2007](https://eprints.whiterose.ac.uk/id/eprint/4589/)). In the store, score [chapter 1's four rankers](#search-end-to-end-decisions) on nDCG@10 under human grades and under each [prompt's](#write-judging-guidelines) LLM grades. The LLM judged only the test pool, the products in those rankers' top 10s, so as a control, the example also scores them with human grades for the pool's pairs only. With four rankers there are 6 pairs, so one swapped pair moves tau by 0.333, and tau can take only seven values. That is crude. Before you run it, predict: will LLM grades put the four rankers in the same order as human grades? Example `kendall-tau-four-rankers` (Python 3.12.15 · bm25s 0.3.13 · numpy 2.5.4 · pandas 3.0.6 · scipy 1.18.1), passing in Edition 2026.10.1: ```python from itertools import combinations from scipy.stats import kendalltau from lib.agreement import chapter_one_runs, llm_qrels from lib.metrics import evaluate from lib.store import load_products, load_queries, qrels products, queries, people = load_products(), load_queries("test"), qrels("test") runs = chapter_one_runs(products, queries) short, guidelines = llm_qrels("short"), llm_qrels("guidelines") pooled = {query_id: {product_id: grade for product_id, grade in people[query_id].items() if product_id in short[query_id]} for query_id in people} # people's grades for the pairs the LLM judged, and no others judgment_sets = {"people": people, "people on the LLM's pairs": pooled, "the short prompt": short, "the guidelines prompt": guidelines} means = {name: {ranker: evaluate(run, judgments)["ndcg@10"].mean() for ranker, run in runs.items()} for name, judgments in judgment_sets.items()} for name, scores in means.items(): print(f"order under {name}: {' > '.join(sorted(scores, key=scores.get, reverse=True))}") for ranker, score in scores.items(): print(f"{ranker} ndcg@10 under {name}: {score:.3f}") orders = [sorted(scores, key=scores.get, reverse=True) for scores in means.values()] steady = [ranker for position, ranker in enumerate(orders[0]) if all(order[position] == ranker for order in orders)] print(f"rankers in the same place under every set of grades: {', '.join(steady) or 'none'}") control = means["people on the LLM's pairs"] # the same pool, human grades for name in ["the short prompt", "the guidelines prompt"]: print("\n".join(f"{ranker}, {name} minus people on the LLM's pairs: {means[name][ranker] - control[ranker]:+.3f}" for ranker in runs)) pairs, reference = list(combinations(runs, 2)), means["people"] print(f"\npairs of rankers: {len(pairs)}\nchange in tau from one swapped pair: {2 / len(pairs):.3f}") for name, scores in list(means.items())[1:]: swapped = [(first, second) for first, second in pairs # discordant: opposite signs if (reference[first] - reference[second]) * (scores[first] - scores[second]) < 0] print(f"swapped under {name}: {'; '.join(f'{first} and {second}' for first, second in swapped) or 'none'}") print(f"tau by hand, people against {name}: {(len(pairs) - 2 * len(swapped)) / len(pairs):.3f}") print(f"tau from scipy, people against {name}: {kendalltau(list(reference.values()), list(scores.values())).statistic:.3f}") ``` Output: ```text order under people: semantic > hybrid > reranked > bm25 bm25 ndcg@10 under people: 0.725 semantic ndcg@10 under people: 0.830 hybrid ndcg@10 under people: 0.812 reranked ndcg@10 under people: 0.798 order under people on the LLM's pairs: semantic > hybrid > reranked > bm25 bm25 ndcg@10 under people on the LLM's pairs: 0.795 semantic ndcg@10 under people on the LLM's pairs: 0.904 hybrid ndcg@10 under people on the LLM's pairs: 0.885 reranked ndcg@10 under people on the LLM's pairs: 0.869 order under the short prompt: reranked > semantic > hybrid > bm25 bm25 ndcg@10 under the short prompt: 0.832 semantic ndcg@10 under the short prompt: 0.903 hybrid ndcg@10 under the short prompt: 0.900 reranked ndcg@10 under the short prompt: 0.906 order under the guidelines prompt: reranked > semantic > hybrid > bm25 bm25 ndcg@10 under the guidelines prompt: 0.805 semantic ndcg@10 under the guidelines prompt: 0.878 hybrid ndcg@10 under the guidelines prompt: 0.876 reranked ndcg@10 under the guidelines prompt: 0.886 rankers in the same place under every set of grades: bm25 bm25, the short prompt minus people on the LLM's pairs: +0.037 semantic, the short prompt minus people on the LLM's pairs: -0.001 hybrid, the short prompt minus people on the LLM's pairs: +0.016 reranked, the short prompt minus people on the LLM's pairs: +0.037 bm25, the guidelines prompt minus people on the LLM's pairs: +0.010 semantic, the guidelines prompt minus people on the LLM's pairs: -0.026 hybrid, the guidelines prompt minus people on the LLM's pairs: -0.008 reranked, the guidelines prompt minus people on the LLM's pairs: +0.017 pairs of rankers: 6 change in tau from one swapped pair: 0.333 swapped under people on the LLM's pairs: none tau by hand, people against people on the LLM's pairs: 1.000 tau from scipy, people against people on the LLM's pairs: 1.000 swapped under the short prompt: semantic and reranked; hybrid and reranked tau by hand, people against the short prompt: 0.333 tau from scipy, people against the short prompt: 0.333 swapped under the guidelines prompt: semantic and reranked; hybrid and reranked tau by hand, people against the guidelines prompt: 0.333 tau from scipy, people against the guidelines prompt: 0.333 ``` Human grades give `semantic > hybrid > reranked > bm25`. The control keeps that order, with a tau of 1.000, so limiting the grades to the pool moves no ranker. It raises every score, BM25's from 0.725 to 0.795, because the ideal top 10 can only hold pooled products. `short` gives `reranked > semantic > hybrid > bm25`, and `guidelines` gives `reranked > semantic > hybrid > bm25`. These pairs swap under `short`: semantic and reranked; hybrid and reranked. Tau for `short` is 0.333 by hand and 0.333 from SciPy, and for `guidelines`, 0.333. Against the control, LLM grades don't move the four rankers by the same amount, or in the same direction. With `short`, BM25 changes by +0.037, semantic retrieval by -0.001, the hybrid by +0.016 and the reranked hybrid by +0.037. With `guidelines`, the changes are +0.010, -0.026, -0.008 and +0.017. The judge raises some rankers and lowers others, and that moves them past each other. Under `short`, semantic retrieval, the hybrid and the reranked hybrid nearly tie, at 0.903, 0.900 and 0.906. Tau ignores how far apart two rankers score, so a swap between near-ties counts as much as one between the best ranker and the worst. It also weighs a swap at the top the same as one at the bottom: "Kendall’s τ equally penalizes errors that occur at any part of the list" ([Yilmaz et al., 2008](https://www.ccs.neu.edu/home/jaa/tmp/RankCorrelation.pdf)). Yilmaz et al. note of the TREC runs they compared that the systems "have very similar performance. Hence, the Kendall’s τ statistic may be affected by these systems" ([Yilmaz et al., 2012](https://static.aminer.org/pdf/20170130/pdfs/sigir/jcmbhoqtt46bhqr5xegf9zy7snars82a.pdf)). These rankers keep their place under every set of grades: `bm25`. If you predicted the same order, notice where the orders differ: among the three rankers that human grades score 0.830, 0.812 and 0.798, gaps that no planned comparison establishes ([Rerun the planned comparisons](#llm-graded-verdicts)). **What to do:** report tau with the list of swapped pairs, never alone. With a handful of rankers, where one swap moves tau by 0.333, don't read it against 0.9. Then test whether the swapped pairs differ at all, with [the planned comparisons under each set of grades](#llm-graded-verdicts). ## Rerun the planned comparisons with LLM grades {#llm-graded-verdicts} Decisions come from verdicts, not from the [orderings that tau compares](#system-order-human-versus-llm). A difference is established when its paired [permutation test](#permutation-test-on-queries) p-value falls below the chapter's Bonferroni threshold, with the t-test and a bootstrap interval beside it ([Decide in advance what counts as established](#when-a-difference-is-established)). Would LLM grades have given the store the same verdicts? Expect trouble near the top: Otero et al. found that "LLM-based judgements are unfair at ranking top-performing systems", with "an exceedingly high rate of false positives regarding statistical differences" ([Otero et al., 2025](https://arxiv.org/abs/2411.13212)). The store's planned comparisons are chapter 3's three, at 0.0167, with intervals at the 0.9833 level. The example runs each under human grades and under each [prompt's](#write-judging-guidelines) grades on the test pool, and adds chapter 3's exploratory comparison of the reranked hybrid with semantic retrieval: Example `planned-comparisons-llm-grades` (Python 3.12.15 · bm25s 0.3.13 · numpy 2.5.4 · pandas 3.0.6 · scipy 1.18.1), passing in Edition 2026.10.1: ```python from lib.agreement import PLANNED, PROMPTS, chapter_one_runs, llm_qrels # PLANNED: chapter 3's, each stage against the last from lib.metrics import evaluate from lib.significance import compare, verdict from lib.store import load_products, load_queries, qrels EXPLORATORY = [("reranked", "semantic")] # chosen after seeing test means threshold = 0.05 / len(PLANNED) # chapter 3's threshold, so that a verdict means the same either way print(f"planned comparisons: {len(PLANNED)}\np-value threshold: {threshold:.4f}\ninterval level: {1 - threshold:.4f}") runs = chapter_one_runs(load_products(), load_queries("test")) judgment_sets = {"people": qrels("test"), **{f"{prompt} prompt": llm_qrels(prompt) for prompt in PROMPTS}} established, disagreements = {}, {judges: 0 for judges in judgment_sets} for judges, judgments in judgment_sets.items(): ndcg = {ranker: evaluate(run, judgments)["ndcg@10"] for ranker, run in runs.items()} for after, before in PLANNED + EXPLORATORY: result, name = compare(ndcg[after], ndcg[before], confidence=1 - threshold), f"{after} minus {before}, {judges}" print(f"\n{name}: {result['mean difference']:.3f}\n{name}, permutation p: {result['permutation p']:.3f}") if (after, before) in PLANNED: print(f"{name}, t-test p: {result['t-test p']:.3f}\n{name}, interval: {result['ci low']:.3f} to {result['ci high']:.3f}") checks = verdict(result, threshold) established[(judges, after, before)] = checks["established"] disagree = [check for check in ["t-test", "interval"] if not checks[f"{check} agrees"]] disagreements[judges] += bool(disagree) print(f"{name}, established: {'yes' if checks['established'] else 'no'}\n" f"{name}, disagrees with the permutation test: {' and '.join(disagree) or 'neither'}") print() for judges, count in disagreements.items(): print(f"planned comparisons where the t-test or interval disagrees, {judges}: {count}") for prompt in PROMPTS: differ = [pair for pair in PLANNED if established[(f"{prompt} prompt", *pair)] != established[("people", *pair)]] print(f"verdicts that differ from people's, {prompt} prompt: {'; '.join(f'{after} minus {before}' for after, before in differ) or 'none'}") ``` Output: ```text planned comparisons: 3 p-value threshold: 0.0167 interval level: 0.9833 semantic minus bm25, people: 0.105 semantic minus bm25, people, permutation p: 0.001 semantic minus bm25, people, t-test p: 0.003 semantic minus bm25, people, interval: 0.047 to 0.205 semantic minus bm25, people, established: yes semantic minus bm25, people, disagrees with the permutation test: neither hybrid minus semantic, people: -0.018 hybrid minus semantic, people, permutation p: 0.125 hybrid minus semantic, people, t-test p: 0.123 hybrid minus semantic, people, interval: -0.048 to 0.006 hybrid minus semantic, people, established: no hybrid minus semantic, people, disagrees with the permutation test: neither reranked minus hybrid, people: -0.014 reranked minus hybrid, people, permutation p: 0.382 reranked minus hybrid, people, t-test p: 0.375 reranked minus hybrid, people, interval: -0.053 to 0.020 reranked minus hybrid, people, established: no reranked minus hybrid, people, disagrees with the permutation test: neither reranked minus semantic, people: -0.032 reranked minus semantic, people, permutation p: 0.048 semantic minus bm25, short prompt: 0.070 semantic minus bm25, short prompt, permutation p: 0.010 semantic minus bm25, short prompt, t-test p: 0.014 semantic minus bm25, short prompt, interval: 0.016 to 0.146 semantic minus bm25, short prompt, established: yes semantic minus bm25, short prompt, disagrees with the permutation test: neither hybrid minus semantic, short prompt: -0.002 hybrid minus semantic, short prompt, permutation p: 0.851 hybrid minus semantic, short prompt, t-test p: 0.861 hybrid minus semantic, short prompt, interval: -0.029 to 0.036 hybrid minus semantic, short prompt, established: no hybrid minus semantic, short prompt, disagrees with the permutation test: neither reranked minus hybrid, short prompt: 0.006 reranked minus hybrid, short prompt, permutation p: 0.727 reranked minus hybrid, short prompt, t-test p: 0.714 reranked minus hybrid, short prompt, interval: -0.029 to 0.042 reranked minus hybrid, short prompt, established: no reranked minus hybrid, short prompt, disagrees with the permutation test: neither reranked minus semantic, short prompt: 0.003 reranked minus semantic, short prompt, permutation p: 0.847 semantic minus bm25, guidelines prompt: 0.073 semantic minus bm25, guidelines prompt, permutation p: 0.045 semantic minus bm25, guidelines prompt, t-test p: 0.057 semantic minus bm25, guidelines prompt, interval: 0.006 to 0.183 semantic minus bm25, guidelines prompt, established: no semantic minus bm25, guidelines prompt, disagrees with the permutation test: interval hybrid minus semantic, guidelines prompt: -0.001 hybrid minus semantic, guidelines prompt, permutation p: 0.929 hybrid minus semantic, guidelines prompt, t-test p: 0.945 hybrid minus semantic, guidelines prompt, interval: -0.045 to 0.037 hybrid minus semantic, guidelines prompt, established: no hybrid minus semantic, guidelines prompt, disagrees with the permutation test: neither reranked minus hybrid, guidelines prompt: 0.010 reranked minus hybrid, guidelines prompt, permutation p: 0.657 reranked minus hybrid, guidelines prompt, t-test p: 0.609 reranked minus hybrid, guidelines prompt, interval: -0.024 to 0.076 reranked minus hybrid, guidelines prompt, established: no reranked minus hybrid, guidelines prompt, disagrees with the permutation test: neither reranked minus semantic, guidelines prompt: 0.009 reranked minus semantic, guidelines prompt, permutation p: 0.581 planned comparisons where the t-test or interval disagrees, people: 0 planned comparisons where the t-test or interval disagrees, short prompt: 0 planned comparisons where the t-test or interval disagrees, guidelines prompt: 1 verdicts that differ from people's, short prompt: none verdicts that differ from people's, guidelines prompt: semantic minus bm25 ``` | Measure | Grades | Semantic minus BM25 | Hybrid minus semantic | Reranked minus hybrid | | --- | --- | --- | --- | --- | | Mean difference | Human | 0.105 | -0.018 | -0.014 | | Mean difference | `short` | 0.070 | -0.002 | 0.006 | | Mean difference | `guidelines` | 0.073 | -0.001 | 0.010 | | Interval | Human | 0.047 to 0.205 | -0.048 to 0.006 | -0.053 to 0.020 | | Interval | `short` | 0.016 to 0.146 | -0.029 to 0.036 | -0.029 to 0.042 | | Interval | `guidelines` | 0.006 to 0.183 | -0.045 to 0.037 | -0.024 to 0.076 | | Permutation p | Human | 0.001 | 0.125 | 0.382 | | Permutation p | `short` | 0.010 | 0.851 | 0.727 | | Permutation p | `guidelines` | 0.045 | 0.929 | 0.657 | | t-test p | Human | 0.003 | 0.123 | 0.375 | | t-test p | `short` | 0.014 | 0.861 | 0.714 | | t-test p | `guidelines` | 0.057 | 0.945 | 0.609 | | Established | Human | yes | no | no | | Established | `short` | yes | no | no | | Established | `guidelines` | no | no | no | Under `short`, the verdicts that differ from the human ones are none; under `guidelines`, `semantic minus bm25`, the one result chapter 3 established. The planned comparisons where the t-test or the interval disagrees with the permutation test number 0 under human grades, 0 under `short` and 1 under `guidelines`; for semantic retrieval minus BM25 under `guidelines`, the check that disagrees is the interval. So whether the store's one established result survives LLM grades depends on the prompt. [Comparing the rankers' order](#system-order-human-versus-llm) found two swapped pairs. One, the reranked hybrid against the hybrid, is a planned comparison, and the table shows it established under no set of grades. The other, the reranked hybrid against semantic retrieval, is the exploratory comparison. Under human grades, the reranked hybrid minus semantic retrieval is -0.032 (permutation p 0.048). Under LLM grades it is 0.003 with `short` and 0.009 with `guidelines`. Chapter 3 chose it after seeing the test means, so it establishes nothing either way. LLM judges can also favour rankers that use LLMs: Balog et al. report "the first empirical evidence of LLM judges exhibiting significant bias towards LLM-based rankers" ([Balog et al., 2025](https://arxiv.org/abs/2503.19092)). Chapter 1's rankers use no LLM, so this chapter can't measure that bias. The book scores the LLM-based rankers of [chapter 8](#rewrite-queries-with-llm) and [chapter 9](#llm-pointwise-reranking) on human grades. **What to do:** never ship a change on a verdict from LLM grades alone. Use LLM grades to decide which candidates are worth judging, and decide on human grades. When two prompts give different verdicts, the LLM can't settle the comparison, and people must. ## Correct LLM scores with a human sample {#calibrate-on-a-human-sample} A middle path uses both kinds of grades: grade every query with the LLM, have people judge a random sample of queries, and use the sample to correct the LLM's error. **Prediction-powered inference** makes this rigorous, with confidence intervals that hold "without making any assumptions on the machine-learning algorithm that supplies the predictions" ([Angelopoulos et al., 2023](https://arxiv.org/abs/2301.09633)). A simple form of the idea, for a mean, takes the LLM's mean over every query and adds the human sample's average of human minus LLM: ```text corrected = mean over all queries of the LLM score + mean over the human sample of (human score − LLM score) ``` In search evaluation, Oosterhuis et al. build metric intervals this way, warning that "generated relevance annotations are not immune to (systematic) errors" ([Oosterhuis et al., 2024](https://arxiv.org/abs/2407.02464)); Takehi et al. let the LLM pick which pairs people judge under a budget, then calibrate it on them ([Takehi et al., 2025](https://arxiv.org/abs/2411.06877)). Suppose the store could afford human judgments for 10 of 30 test queries. The example draws 2,000 random sets of that size and, for each, corrects the LLM's estimate, under each [prompt](#write-judging-guidelines), of two gaps between [chapter 1's rankers](#search-end-to-end-decisions): semantic retrieval minus BM25, and the hybrid minus semantic retrieval. It compares the corrected estimate with the human sample alone. **Spread** is an estimate's standard deviation over the draws. The simulation chooses nothing: Example `correct-llm-scores-with-humans` (Python 3.12.15 · bm25s 0.3.13 · numpy 2.5.4 · pandas 3.0.6 · scipy 1.18.1), passing in Edition 2026.10.1: ```python import numpy as np import pandas as pd from lib.agreement import PROMPTS, chapter_one_runs, llm_qrels from lib.metrics import evaluate from lib.store import load_products, load_queries, qrels JUDGED, DRAWS = 10, 2000 # test queries people judge in each draw, and the number of draws runs = chapter_one_runs(load_products(), load_queries("test")) people = pd.DataFrame({ranker: evaluate(run, qrels("test"))["ndcg@10"] for ranker, run in runs.items()}) rng = np.random.default_rng(2026) draws = [rng.choice(people.index, JUDGED, replace=False) for _ in range(DRAWS)] # which queries people judge print(f"queries judged by people in each draw: {JUDGED} of {len(people)}\ndraws: {DRAWS:,}") for after, before in [("semantic", "bm25"), ("hybrid", "semantic")]: gap_people = people[after] - people[before] # per-query gaps under people's grades print(f"\n{after} minus {before}, people, all queries: {gap_people.mean():.3f}") alone = np.std([gap_people[judged].mean() for judged in draws]) # the human sample used alone print(f"{after} minus {before}, human sample alone, spread: {alone:.3f}") for prompt in PROMPTS: llm = {ranker: evaluate(run, llm_qrels(prompt))["ndcg@10"] for ranker, run in runs.items()} gap_llm, name = llm[after] - llm[before], f"{after} minus {before}, {prompt} prompt" corrected = [gap_llm.mean() + (gap_people[judged] - gap_llm[judged]).mean() for judged in draws] print(f"{name}, LLM only: {gap_llm.mean():.3f}") print(f"{name}, corrected, mean over draws: {np.mean(corrected):.3f}") print(f"{name}, corrected, spread: {np.std(corrected):.3f}") print(f"{name}, corrected spread smaller than the human sample's: {'yes' if np.std(corrected) < alone else 'no'}") print(f"{name}, correlation of per-query gaps with people's: {np.corrcoef(gap_llm, gap_people)[0, 1]:.3f}") ``` Output: ```text queries judged by people in each draw: 10 of 30 draws: 2,000 semantic minus bm25, people, all queries: 0.105 semantic minus bm25, human sample alone, spread: 0.045 semantic minus bm25, short prompt, LLM only: 0.070 semantic minus bm25, short prompt, corrected, mean over draws: 0.104 semantic minus bm25, short prompt, corrected, spread: 0.036 semantic minus bm25, short prompt, corrected spread smaller than the human sample's: yes semantic minus bm25, short prompt, correlation of per-query gaps with people's: 0.641 semantic minus bm25, guidelines prompt, LLM only: 0.073 semantic minus bm25, guidelines prompt, corrected, mean over draws: 0.104 semantic minus bm25, guidelines prompt, corrected, spread: 0.039 semantic minus bm25, guidelines prompt, corrected spread smaller than the human sample's: yes semantic minus bm25, guidelines prompt, correlation of per-query gaps with people's: 0.692 hybrid minus semantic, people, all queries: -0.018 hybrid minus semantic, human sample alone, spread: 0.016 hybrid minus semantic, short prompt, LLM only: -0.002 hybrid minus semantic, short prompt, corrected, mean over draws: -0.017 hybrid minus semantic, short prompt, corrected, spread: 0.023 hybrid minus semantic, short prompt, corrected spread smaller than the human sample's: no hybrid minus semantic, short prompt, correlation of per-query gaps with people's: 0.168 hybrid minus semantic, guidelines prompt, LLM only: -0.001 hybrid minus semantic, guidelines prompt, corrected, mean over draws: -0.017 hybrid minus semantic, guidelines prompt, corrected, spread: 0.024 hybrid minus semantic, guidelines prompt, corrected spread smaller than the human sample's: no hybrid minus semantic, guidelines prompt, correlation of per-query gaps with people's: 0.342 ``` For semantic retrieval minus BM25, the LLM alone estimates 0.070 with `short`, against 0.105 on human grades. Corrected, the estimate averages 0.104 over the draws, so the correction removes the LLM's bias. Its spread is 0.036, against 0.045 for the human sample alone: the LLM's grades for the other queries add information, because its per-query gaps correlate with the human ones at 0.641. For the hybrid minus semantic retrieval, the corrected spread is 0.023, against 0.016 for the human sample alone, with a correlation of 0.168. Is the corrected spread smaller than the human sample's? For this gap: no with `short` and no with `guidelines`. For semantic retrieval minus BM25: yes and yes. The correction fixes the level of a score, but narrows the uncertainty only where the LLM's per-query scores track the human ones, and here they track them least on the closer of the two gaps. **What to do:** whenever you report a score from LLM grades, have people judge a random sample of queries and report the corrected estimate. On that sample, check how well the LLM's per-query gaps correlate with the human ones. When they correlate weakly, spend the budget on judging more queries. ## What goes wrong in production {#judgments-from-people-and-llms-in-production} **The judge changes without notice.** A store that judges with an LLM runs it again and again, and a different model version, a different quantization (the precision at which the model's weights are stored) or a reworded prompt is a different judge. The recording's `manifest.json` pins the model's digest and both prompts word for word. The answer schema and the context length live in `recordings/llm.py` and `recordings/llm_judgments.py`, so record those too. Rerun the agreement check on the same human sample whenever any of them changes. **Systems tuned to the judge score well on it.** Clarke and Dietz built a run that exploits an LLM judge, and it ranked "significantly higher under automatic assessment (5th) than under manual assessment (28th)" ([Clarke and Dietz, 2024](https://arxiv.org/abs/2412.17156)). A ranker tuned on LLM grades, or built on the same LLM, needs its headline score on human grades. **Product text can steer the judge.** Alaofi et al. found that LLM judges can be misled by query words planted in an irrelevant text, and by instructions written into it ([Alaofi et al., 2024](https://arxiv.org/abs/2501.17969)). Product descriptions may come from outside your team. Keep a human sample in every evaluation, so that a judge led astray shows up as a falling agreement. **Calibrating on test is tuning on test.** Rewording a prompt until it matches the human grades on test queries makes agreement there meaningless. Choose prompts on train queries, and report agreement on test. **Human grades aren't the truth either.** On the TREC Podcast track, where an LLM and a TREC assessor disagreed most, Mansour et al. found that experts re-judging the pairs more often sided with the LLM ([Mansour et al., 2026](https://arxiv.org/abs/2601.05603)). Treat human grades as the reference you chose, and read disagreements before you blame either side. **What to do:** pin the judge, keep a human sample in every evaluation, rerun agreement when anything changes, and never score a ranker with the judge you tuned it on. ## Decisions this chapter made {#judgments-from-people-and-llms-decisions} Each choice below is a default. Later chapters keep it unless they state a reason. - **Human grades for every headline score.** LLM grades are a diagnostic: filling holes, screening candidates and checking verdicts. The LLM-based rankers of [chapter 8](#rewrite-queries-with-llm) and [chapter 9](#llm-pointwise-reranking) are scored on human grades. - **Every finding about the judge names its pool.** Kappa and linear weighted kappa, with observed agreement, the far misses and the label mix beside them, are measured on train pairs people judged, and reported for the test pool's judged pairs too. The direction of errors and the largest error are read on each pool, and only the rankers' top 10s speak to ranker scores. - **Two prompts, recorded word for word.** The train pool can't tell them apart, so the store keeps both, and a conclusion from LLM grades counts only when both prompts give it. - **Unjudged pairs filled with LLM grades as a diagnostic,** reported beside the human-only score. On the test queries, filling changes no ranker's place and no verdict. - **Compare scores only under the same grades.** On the same pool, LLM grades don't move every ranker's nDCG@10 by the same amount or in the same direction, so a score under LLM grades never meets one under human grades. - **Tau with its swapped pairs, then the planned comparisons at chapter 3's threshold.** With four rankers tau is crude, and the verdicts decide. - **A score from LLM grades comes with a human sample** of queries that corrects it, and the correlation that says whether the LLM helped. # Lexical retrieval By the end of this chapter you can build lexical search from scratch: turn product text into terms, index the terms, and score products with TF-IDF and BM25, matching the bm25s library score for score. You can also decide which changes to the text processing and to BM25's parameters are worth keeping, by choosing on train queries and checking on test queries. **Lexical retrieval** finds and ranks products by the words they share with the query. It is what [the store](#store-data), the online furniture and home-goods retailer that serves as this book's case study, runs in its search box: BM25, through the bm25s library. [Chapter 1](#rank-with-bm25) uses it at bm25s's defaults, for a mean [nDCG@10](#score-one-ranking) of 0.725 on the test queries, but as a black box: when a shopper types `7qt slow cooker` or `pool floats`, you can see that the results are weak, but not why. This chapter opens the box. You rebuild each step by hand on the store's 12,000 products, check it against bm25s, then change one step at a time and measure each change against the store's [judgments](#store-data). Each change is chosen on the 50 [train queries](#store-data) and checked on the 30 test queries. Before reading any test result, the chapter fixes its planned comparisons on the test queries. There are five, each against the same search without the change: 1. [stemming](#check-stemming-on-test) 2. the best `k1` and `b` on train, against bm25s's defaults ([tuning](#tune-bm25-parameters)) 3. the best weight on train for the product name ([fields](#weight-the-product-name)) 4. product attributes as a field ([attributes](#index-attributes-as-a-field)) 5. the best phrase bonus on train ([phrases](#match-phrases-with-positions)) By the book's rule ([chapter 3](#when-a-difference-is-established)), a difference is established when its paired permutation test gives a p-value below 0.05 divided by 5, which is 0.01. Each comparison also reports the paired t-test and a 99% bootstrap interval, and says whether each agrees with the permutation test. Comparisons on train queries choose a setting. They are never evidence. ## Split product text into terms {#split-text-into-terms} Search never compares raw strings. The query and every product's text first go through the same steps. Splitting the text into pieces gives **tokens**. Lowercasing makes `Chair` and `chair` one token. Removing **stop words**, very common words such as `the` and `with`, leaves the **terms** that search compares, and all the terms in the catalog form its **vocabulary**. A product can only match a query on terms they share, so these steps decide what can match at all, before any scoring starts. bm25s, which runs the store's BM25, makes three choices here. It lowercases. It keeps every run of two or more word characters, which are letters, digits or underscores, with the regular expression `(?u)\b\w\w+\b`, which its docstring describes as the method of scikit-learn's `CountVectorizer`. And it removes 33 English stop words, among them `no`, `not`, `with` and `for`. It doesn't stem words unless you pass it a stemmer ([bm25s tokenizer](https://raw.githubusercontent.com/xhluca/bm25s/0.3.13/bm25s/tokenization.py), [stop words](https://raw.githubusercontent.com/xhluca/bm25s/0.3.13/bm25s/stopwords.py)). The example writes the same steps by hand, checks them against `bm25s.tokenize` on every product, then looks at what the train query `7qt slow cooker` becomes: Example `tokenize-product-text` (Python 3.12.15 · bm25s 0.3.13 · numpy 2.5.4 · pandas 3.0.6 · scipy 1.18.1), passing in Edition 2026.10.1: ```python import re import bm25s from bm25s.stopwords import STOPWORDS_EN from lib.lexical import LexicalSearch from lib.store import load_products, load_queries, product_text, qrels TOKEN = re.compile(r"(?u)\b\w\w+\b") # two or more word characters: letters, digits or underscores STOPWORDS = set(STOPWORDS_EN) # the 33 English stop words bm25s removes def tokenize(text, pattern=TOKEN): """Lowercase, split into tokens, and drop stop words.""" return [token for token in pattern.findall(text.lower()) if token not in STOPWORDS] products, queries = load_products(), load_queries() texts = product_text(products).tolist() by_hand = [tokenize(text) for text in texts] with_bm25s = bm25s.tokenize(texts, stopwords="en", return_ids=False, show_progress=False) print("products tokenized differently:", sum(mine != theirs for mine, theirs in zip(by_hand, with_bm25s))) print() query = queries.at[62, "query"] # a train query print("query 62:", query, "->", tokenize(query)) print("product 1:", products.at[1, "name"], "->", tokenize(products.at[1, "name"])) print("products with the term 7qt:", sum("7qt" in terms for terms in by_hand)) print("products whose text says 7 qt:", sum(re.search(r"(? 7, qt changed = queries["query"].map(lambda text: tokenize(text) != tokenize(text, digits_apart)) for split in ["train", "test"]: print(f"{split} queries whose terms change with digits apart:", changed[queries["split"] == split].sum()) print() print("probe:", "rug not wool", "->", tokenize("rug not wool")) # written for this chapter print("queries with not or no:", queries["query"].str.contains(r"\b(?:not|no)\b").sum()) ``` Output: ```text products tokenized differently: 0 query 62: 7qt slow cooker -> ['7qt', 'slow', 'cooker'] product 1: all-clad 7 qt . slow cooker -> ['all', 'clad', 'qt', 'slow', 'cooker'] products with the term 7qt: 0 products whose text says 7 qt: 8 exact products judged for query 62: 9 exact products in bm25's top 10: 1 train queries whose terms change with digits apart: 4 test queries whose terms change with digits apart: 0 probe: rug not wool -> ['rug', 'wool'] queries with not or no: 0 ``` The number of products that the hand-written tokenizer and bm25s's split differently is 0. The trouble is in the rules they share. The shopper wants a 7-quart slow cooker, and `7qt` survives as one token. But the number of products that contain the term `7qt` is 0, because product text writes the size apart: the text of 8 products says `7 qt`, and the tokenizer drops their `7`, which is one character long. The size plays no part in the ranking, on the query's side or the products'. The judges marked 9 products [exact](#store-data) for this query, and BM25's top 10 holds 1 of them. Without the size, the terms can't tell a 7-quart cooker from any other slow cooker. A pattern that keeps single digits and splits numbers from their units, `(?u)\d+|[^\W\d_]+`, changes the terms of 4 train queries and 0 test queries, so the store's judgments can say little about it. The stop list fails the same way, and bm25s raises no error. Take `rug not wool`, a **probe query**: one written for this chapter to test a behaviour, not one of the store's queries. It becomes `rug wool`, so a wool rug matches both of its terms. The judgments can measure this harm only on queries with a negation, and 0 of the store's queries contain `not` or `no`. Search systems have moved from long stop lists to short ones or none, in part because stop words carry meaning in some queries, and in part because keeping them has become cheap ([Manning et al., 2008](https://nlp.stanford.edu/IR-book/html/htmledition/dropping-common-terms-stop-words-1.html)). **What to do:** print the terms that your queries and products become before you tune any scoring, and run both through the same tokenizer. Start from bm25s's defaults, as the store does. When shoppers type sizes, quantities or model numbers, use a token pattern that keeps single digits and splits a number from its unit, such as the one above. When they type negations, turn the stop list off. Each is a change to search, so measure it on the [train queries](#store-data) and check it on test queries before you keep it. ## Build an inverted index {#build-an-inverted-index} To score a query made of [those terms](#split-text-into-terms), search needs the products that contain each one. Reading every product's text for every query works for a small sample, but not for a catalog of millions. An **inverted index** turns the texts around: for each term, it keeps the list of products that contain it, called the term's **postings** ([Manning et al., 2008](https://nlp.stanford.edu/IR-book/html/htmledition/a-first-take-at-building-an-inverted-index-1.html)). With each product, the postings store the **term frequency**, the number of times the term occurs in that product's text. The number of products in a term's postings is its **document frequency**: search engines call the items they search documents, and in this book they are products. A query reads only the postings of its own terms. The store's index maps each term to a dictionary of product IDs and term frequencies. It also keeps each product's length in terms, which [BM25](#bm25-from-scratch) needs. The example builds it from the text of every product, with the tokenizer from [the previous section](#split-text-into-terms), then looks up the terms of the train query `7qt slow cooker`: Example `inverted-index-by-hand` (Python 3.12.15 · bm25s 0.3.13 · numpy 2.5.4 · pandas 3.0.6 · pystemmer 3.1.0 · scipy 1.18.1), passing in Edition 2026.10.1: ```python from collections import Counter from lib.lexical_by_hand import tokenize from lib.store import load_products, load_queries, product_text postings = {} # {term: {product_id: how many times the term occurs in that product}} lengths = {} # {product_id: number of terms} for product_id, text in product_text(load_products()).items(): terms = tokenize(text) lengths[product_id] = len(terms) for term, count in Counter(terms).items(): postings.setdefault(term, {})[product_id] = count print(f"vocabulary: {len(postings):,} terms") print(f"average product length: {sum(lengths.values()) / len(lengths):.1f} terms") print("postings for cooker, first 3:", list(postings["cooker"].items())[:3]) print() query = load_queries().at[62, "query"] for term in tokenize(query): print(f"products with {term}:", len(postings.get(term, {}))) found = [set(postings.get(term, {})) for term in tokenize(query)] print("products with any query term:", len(set.union(*found))) print("products with every query term:", len(set.intersection(*found))) print("products with w003096809:", len(postings.get("w003096809", {}))) # the code in query 205 ``` Output: ```text vocabulary: 20,393 terms average product length: 50.3 terms postings for cooker, first 3: [(1, 3), (2, 2), (115, 1)] products with 7qt: 0 products with slow: 157 products with cooker: 141 products with any query term: 158 products with every query term: 0 products with w003096809: 0 ``` The store's 12,000 products use 20,393 terms, and a product's text averages 50.3 terms. Each pair in the postings for `cooker` is a product ID and a term frequency. The first is product 1, the 7 qt slow cooker from [the tokenizer section](#split-text-into-terms). For `7qt slow cooker`, the postings list 157 products for `slow`, 141 for `cooker` and 0 for `7qt`. Together they cover 158 products, and those are all that scoring this query has to read. The number that contain every term is 0, because no product contains `7qt`. A Boolean AND query, which returns only the products that contain every query term, returns nothing for this query. A search that scores every product containing any of the terms, as BM25 does, still returns slow cookers, and leaves it to the scores to decide which come first. **What to do:** when a query's results look wrong, look up its terms in the index before you change the scoring. A term with no postings, such as `7qt`, can't help any product whatever the scorer does, so fix the tokenizer or the query instead. For queries that shoppers type, match products that contain any of the terms rather than all of them, because one unmatched word would otherwise empty the results. Serve from a library's index, and keep a small one like this to check what a term matches. ## Weight terms with TF-IDF {#weight-terms-with-tf-idf} Not every match is worth the same. Two ideas weight them. A term that occurs often in a product's text is more likely to be what the product is about: that is its term frequency, tf. And a match on a rare term says more than a match on a common one. Karen Spärck Jones proposed weighting terms by how many documents contain them, so that "matches on less frequent, more specific, terms are of greater value than matches on frequent terms" ([Spärck Jones, 1972](https://doi.org/10.1108/eb026526); [free reprint](http://www.staff.city.ac.uk/~sbrp622/idfpapers/ksj_orig.pdf)). That weight became the **inverse document frequency**, IDF: the logarithm of the number of products divided by the term's document frequency, high for a rare term and low for a common one ([Manning et al., 2008](https://nlp.stanford.edu/IR-book/html/htmledition/inverse-document-frequency-1.html)). TF-IDF multiplies the two for each query term and adds them up ([Manning et al., 2008](https://nlp.stanford.edu/IR-book/html/htmledition/tf-idf-weighting-1.html)): ```text idf(t) = ln(N / df(t)) N products, df(t) of them contain t tf-idf(q, d) = sum over query terms t of tf(t, d) × idf(t) ``` In the store, `furniture` is a common term. The example computes TF-IDF over the [inverted index](#build-an-inverted-index), shows the IDF of `furniture` and of each term of the train query `laundry basket with wheels`, then TF-IDF's top 3 for that query, with each product's score and term frequencies. Last, it scores TF-IDF on all 50 train queries against [BM25 as chapter 1 runs it](#rank-with-bm25): Example `tf-idf-by-hand` (Python 3.12.15 · bm25s 0.3.13 · numpy 2.5.4 · pandas 3.0.6 · pystemmer 3.1.0 · scipy 1.18.1), passing in Edition 2026.10.1: ```python import math import pandas as pd from lib.lexical import LexicalSearch from lib.lexical_by_hand import InvertedIndex, tokenize from lib.metrics import evaluate from lib.store import load_products, load_queries, product_text, qrels products, train, judgments = load_products(), load_queries("train"), qrels("train") index = InvertedIndex(product_text(products)) n = len(index.lengths) # 12,000 products def tf_idf(query): """TF-IDF scores of the products that contain a query term, best first.""" scores = {} for term in tokenize(query): postings = index.postings.get(term, {}) idf = math.log(n / len(postings)) if postings else 0.0 for product_id, tf in postings.items(): scores[product_id] = scores.get(product_id, 0.0) + tf * idf return pd.Series(scores, dtype=float).sort_values(ascending=False, kind="stable") query, labels = train.at[36, "query"], {2: "exact", 1: "partial", 0: "irrelevant"} for term in ["furniture"] + tokenize(query): print(f"idf of {term}: {math.log(n / len(index.postings[term])):.3f}") print(f"\ntf-idf top 3 for {query!r}:") for rank, (product_id, score) in enumerate(tf_idf(query).head(3).items(), start=1): counts = ", ".join(f"{term} {index.postings[term].get(product_id, 0)}" for term in tokenize(query)) print(f" {rank} {score:4.1f} {labels[judgments[36][product_id]]:<8} {counts:<29} {products.at[product_id, 'name'][:22]}") print() runs = {"tf-idf": {query_id: list(tf_idf(text).index[:100]) for query_id, text in train["query"].items()}, "bm25": LexicalSearch(products).run(train)} results = {name: evaluate(run, judgments) for name, run in runs.items()} # one row per train query for name, frame in results.items(): print(f"{name} train ndcg@10: {frame['ndcg@10'].mean():.3f}") print(f"{name} train judged@10: {frame['judged@10'].mean():.3f}") change = (results["bm25"]["ndcg@10"] - results["tf-idf"]["ndcg@10"]).round(3) print(f"train queries where bm25 scores higher, lower: {(change > 0).sum()}, {(change < 0).sum()}") ``` Output: ```text idf of furniture: 0.919 idf of laundry: 3.985 idf of basket: 3.925 idf of wheels: 4.839 tf-idf top 3 for 'laundry basket with wheels': 1 51.3 partial laundry 4, basket 9, wheels 0 dirty clothes basket , 2 45.4 exact laundry 6, basket 3, wheels 2 laundry sorter basket 3 39.7 partial laundry 7, basket 3, wheels 0 weave laundry basket tf-idf train ndcg@10: 0.579 tf-idf train judged@10: 0.836 bm25 train ndcg@10: 0.695 bm25 train judged@10: 0.930 train queries where bm25 scores higher, lower: 36, 10 ``` `furniture` gets an IDF of 0.919. The query's terms get 3.985 for `laundry`, 3.925 for `basket` and 4.839 for `wheels`: rarer, and close to one another. TF-IDF's first product is [partial](#store-data), a basket that never says `wheels` but repeats `basket` more than either product below it. With IDFs this close, the repeats outweigh the missing `wheels`, as the scores show. The exact product, the only one of the three that says `wheels`, comes second. Term frequency has no ceiling in TF-IDF, so a product can outscore another by repeating a word. Over the train queries, TF-IDF averages an [nDCG@10](#score-one-ranking) of 0.579 against BM25's 0.695, and BM25 scores higher and lower on 36, 10 queries respectively. TF-IDF's [judged@10](#score-every-test-query) is 0.836 against BM25's 0.930; when a run's judged@10 is lower, part of its gap may be products that nobody judged rather than worse ones. This comparison on train queries tells you which scorer to build on, and the book keeps its evidence for the test queries. **What to do:** weight terms by IDF in any lexical scorer, because a match on a rare term is better evidence. Don't rank products by raw TF-IDF: cap what a repeated term can add, and correct for long texts, which is what BM25 does. Keep TF-IDF for explaining a ranking, where you can read off each term's tf and idf. ## Rebuild BM25 by hand to match bm25s {#bm25-from-scratch} BM25 keeps IDF and changes how term frequency counts, in two ways. With **saturation**, each repeat of a term adds less than the one before, and no number of repeats can push the term's contribution past a ceiling ([Robertson and Zaragoza, 2009](https://www.staff.city.ac.uk/~sbrp622/papers/foundations_bm25_review.pdf)). The parameter `k1` sets how quickly a term approaches it. With **length normalization**, a long text, which has more chances to contain any term, gets its term frequencies discounted. The parameter `b` sets how much: "b = 1 will perform full document-length normalisation, while b = 0 will switch normalisation off" (Robertson and Zaragoza). In the form bm25s computes by default ([bm25s scoring](https://raw.githubusercontent.com/xhluca/bm25s/0.3.13/bm25s/scoring.py)): ```text idf(t) = ln(1 + (N − df(t) + 0.5) / (df(t) + 0.5)) bm25(q, d) = sum over query terms t of idf(t) × tf(t, d) / (tf(t, d) + k1 × (1 − b + b × length(d) / average length)) ``` Two details differ between textbooks and libraries. A common variant multiplies the fraction by `k1` + 1, which grows every score by the same factor and so leaves the ranking unchanged (Robertson and Zaragoza, section 3.5.1). And Robertson's original IDF, without the 1 inside the logarithm, is negative for a term in more than half of the documents, so Lucene, an open-source search library, adds the 1 ([Kamphuis et al., 2020](https://cs.uwaterloo.ca/~jimmylin/publications/Kamphuis_etal_ECIR2020_preprint.pdf)). bm25s's default, `method="lucene"`, is Lucene's formula computed with exact document lengths ([bm25s README](https://github.com/xhluca/bm25s/blob/0.3.13/README.md)), where Lucene itself approximates them (Kamphuis et al.). "BM25" names a family of such variants, and a study of eight of them found no significant differences in effectiveness on three newswire collections (Kamphuis et al.). The example writes the formula over the store's [inverted index](#build-an-inverted-index) at bm25s's defaults, `k1=1.5` and `b=0.75`. It compares every product's score for each of the 80 queries with the score from bm25s 0.3.13, then compares the two rankings, by their top 100s and by [nDCG@10](#score-one-ranking): Example `bm25-by-hand` (Python 3.12.15 · bm25s 0.3.13 · numpy 2.5.4 · pandas 3.0.6 · pystemmer 3.1.0 · scipy 1.18.1), passing in Edition 2026.10.1: ```python import math import pandas as pd from lib.lexical import LexicalSearch from lib.lexical_by_hand import InvertedIndex, tokenize from lib.metrics import evaluate from lib.store import load_products, load_queries, product_text, qrels products, queries = load_products(), load_queries() index, library = InvertedIndex(product_text(products)), LexicalSearch(products) # by hand, and bm25s as chapter 1 uses it n, k1, b = len(index.lengths), 1.5, 0.75 def bm25(query): scores = {} for term in tokenize(query): df = len(index.postings.get(term, {})) idf = math.log(1 + (n - df + 0.5) / (df + 0.5)) for product_id, tf in index.postings.get(term, {}).items(): length = 1 - b + b * index.lengths[product_id] / index.average_length scores[product_id] = scores.get(product_id, 0.0) + idf * tf / (tf + k1 * length) return pd.Series(scores, dtype=float) df = len(index.postings["your"]) print(f"products with your: {df:,}") print(f"robertson idf of your: {math.log((n - df + 0.5) / (df + 0.5)):.3f}") print(f"lucene idf of your: {math.log(1 + (n - df + 0.5) / (df + 0.5)):.3f}") for tf in [1, 2, 4, 9]: # one term, in a product of average length print(f"term weight at tf {tf}: {tf / (tf + k1):.3f}") gaps = [(bm25(query).reindex(products.index, fill_value=0) - library.retriever.get_scores(tokenize(query))).abs().max() for query in queries["query"]] print(f"\nlargest score difference from bm25s: {max(gaps):.0e}") print("bm25s score type:", library.retriever.get_scores(["chair"]).dtype) ranked = {query_id: list(bm25(query).sort_index().sort_values(ascending=False, kind="stable").index[:100]) for query_id, query in queries["query"].items()} # equal scores in product_id order library_ranked = library.run(queries) # lib.lexical also orders equal scores by product_id print("same top 100 for every query:", "yes" if ranked == library_ranked else "no") for split in ["train", "test"]: for name, run in [("by hand", ranked), ("with bm25s", library_ranked)]: print(f"{split} ndcg@10 {name}: {evaluate(run, qrels(split))['ndcg@10'].mean():.3f}") ``` Output: ```text products with your: 6,135 robertson idf of your: -0.045 lucene idf of your: 0.671 term weight at tf 1: 0.400 term weight at tf 2: 0.571 term weight at tf 4: 0.727 term weight at tf 9: 0.857 largest score difference from bm25s: 1e-06 bm25s score type: float32 same top 100 for every query: yes train ndcg@10 by hand: 0.695 train ndcg@10 with bm25s: 0.695 test ndcg@10 by hand: 0.725 test ndcg@10 with bm25s: 0.725 ``` The word `your` occurs in 6,135 of the 12,000 products. Robertson's original IDF gives it -0.045, and Lucene's gives it 0.671. A negative IDF would make a match on a word lower a product's score. The term weights show saturation: in a product of average length, one occurrence of a term earns 0.400 of its IDF, and nine earn 0.857. The scores match. The largest difference over every product and query is 1e-06, at the precision of the `float32` numbers bm25s returns. Matching scores don't settle a ranking on their own, though: where scores are equal, an implementation still has to choose which tied product comes first. Ours lists tied products by product ID. So does the book's shared BM25, `lib.lexical` from [chapter 1](#rank-with-bm25), which takes every product's score from bm25s and then orders equal scores by product ID. Asked whether both give the same top 100 for every query, the example answers yes. On the train queries, the mean nDCG@10 is 0.695 by hand and 0.695 with bm25s; on the test queries, 0.725 and 0.725. **What to do:** serve BM25 from a library such as bm25s, and keep a small version you wrote yourself to check it, because yours shows where every part of a score comes from. Compare two implementations by their scores first, then by their rankings. Before you compare rankings, break ties the same way in both, by product ID as the book's BM25 does, or tied products alone can make two correct implementations rank differently. When two BM25 systems rank differently, compare their tokenizers before their formulas. The study of variants held tokenization and stop words fixed, and notes that more mundane settings, such as the choice of stop words, often make larger differences than the formula (Kamphuis et al.). ## Stem terms so word forms match {#stem-word-forms} A shopper types `pool floats`, and a product says `float` or `floating`. To the tokenizer, those are different terms, so they don't match. **Stemming** cuts common endings off words, so that the forms of a word share one term, its **stem**. Martin Porter designed his suffix-stripping algorithm for retrieval, and in his test it reduced a vocabulary of 10,000 words to 6,370 stems, "by about one third" ([Porter, 1980](https://tartarus.org/martin/PorterStemmer/def.txt)). His page recommends its successor, the Snowball English or Porter2 stemmer ([Porter](https://tartarus.org/martin/PorterStemmer/)), which stems "slightly over 5% of words" differently ([Snowball](https://snowballstem.org/algorithms/english/stemmer.html)). Stemming trades precision for recall: it "increases recall while harming precision", and "while it helps a lot for some queries, it equally hurts performance a lot for others" ([Manning et al., 2008](https://nlp.stanford.edu/IR-book/html/htmledition/stemming-and-lemmatization-1.html)). A crude stemmer takes a few lines. The example writes one by hand that cuts `-ing`, `-es`, `-ed` or `-s` when at least three letters remain, and compares it with the Snowball English stemmer from PyStemmer, which bm25s's tokenizer recommends ([bm25s tokenizer](https://raw.githubusercontent.com/xhluca/bm25s/0.3.13/bm25s/tokenization.py)) and the book pins at 3.1.0 ([PyStemmer](https://pypi.org/project/PyStemmer/)). Then it counts the products whose text contains a word, and those whose text contains its stem: Example `stemmer-merges-word-forms` (Python 3.12.15 · bm25s 0.3.13 · numpy 2.5.4 · pandas 3.0.6 · pystemmer 3.1.0 · scipy 1.18.1), passing in Edition 2026.10.1: ```python import Stemmer from lib.lexical_by_hand import InvertedIndex from lib.store import load_products, product_text def strip_suffix(word): """A stemmer by hand: cut the first ending that matches, if at least three letters are left.""" for ending in ["ing", "es", "ed", "s"]: if word.endswith(ending) and len(word) - len(ending) >= 3: return word[: -len(ending)] return word stemmer = Stemmer.Stemmer("english") # Snowball English, also called Porter2 words = ["floats", "floating", "molding", "molded", "moldings", "shelves", "shelving", "shelf"] for word, stem in zip(words, stemmer.stemWords(words)): print(f"{word}: snowball {stem}, by hand {strip_suffix(word)}") products = load_products() text = product_text(products) plain, stemmed = InvertedIndex(text), InvertedIndex(text, stem=True) print() for word in ["floats", "molding"]: print(f"products with {word}:", len(plain.postings[word])) print(f"products with the stem of {word}:", len(stemmed.postings[stemmer.stemWord(word)])) float_products = list(stemmed.postings["float"]) print("products with the stem of floats that say floating:", len(plain.postings["floating"].keys() & set(float_products))) for product_class, count in products.loc[float_products, "product_class"].value_counts().head(2).items(): print(f"products with the stem of floats in {product_class}: {count}") print(f"vocabulary without stemming: {len(plain.postings):,} terms") print(f"vocabulary with stemming: {len(stemmed.postings):,} terms") print(f"vocabulary reduction: {1 - len(stemmed.postings) / len(plain.postings):.0%}") ``` Output: ```text floats: snowball float, by hand float floating: snowball float, by hand float molding: snowball mold, by hand mold molded: snowball mold, by hand mold moldings: snowball mold, by hand molding shelves: snowball shelv, by hand shelv shelving: snowball shelv, by hand shelv shelf: snowball shelf, by hand shelf products with floats: 11 products with the stem of floats: 288 products with molding: 123 products with the stem of molding: 262 products with the stem of floats that say floating: 244 products with the stem of floats in Wall Mounted Shelves: 57 products with the stem of floats in Desks: 36 vocabulary without stemming: 20,393 terms vocabulary with stemming: 15,258 terms vocabulary reduction: 25% ``` The hand-written stemmer agrees with Snowball on every word but `moldings`, where it stops after cutting `-s`, and Snowball's rules go on to `mold`. Both turn `shelves` into `shelv` and leave `shelf` alone: rules that cut endings don't know that shelves is the plural of shelf. The counts show both sides of the trade. `floats` occurs in 11 products and its stem in 288, but 244 of those say `floating`. The two most common product classes among them are wall-mounted shelves, with 57, and desks, with 36. So the stem reaches floating shelves as well as pool floats, and only the judgments can say which effect wins ([next section](#check-stemming-on-test)). `molding` occurs in 123 and its stem in 262, because `mold` also stands for `molded`, `moldings` and `mold` itself, a word with more than one meaning. The vocabulary shrinks by 25%, from 20,393 terms to 15,258 terms, on different text and with a later algorithm than Porter's test. To serve stemmed BM25 from bm25s, pass the same stemmer to `bm25s.tokenize` for the products and for every query. The next example checks that bm25s then scores like the hand-built index with stemming, `InvertedIndex(text, stem=True)` from `lib.lexical_by_hand`: Example `bm25s-with-a-stemmer` (Python 3.12.15 · bm25s 0.3.13 · numpy 2.5.4 · pandas 3.0.6 · pystemmer 3.1.0 · scipy 1.18.1), passing in Edition 2026.10.1: ```python import bm25s import Stemmer from lib.lexical_by_hand import InvertedIndex from lib.store import load_products, load_queries, product_text products, queries = load_products(), load_queries() texts = product_text(products) stemmer = Stemmer.Stemmer("english") library = bm25s.BM25() # k1=1.5, b=0.75, as in chapter 1 library.index(bm25s.tokenize(texts.tolist(), stopwords="en", stemmer=stemmer, show_progress=False), show_progress=False) by_hand = InvertedIndex(texts, stem=True) gaps = [] for query in queries["query"]: tokens = bm25s.tokenize([query], stopwords="en", stemmer=stemmer, return_ids=False, show_progress=False)[0] scores = library.get_scores(tokens) # every product, in catalog order gaps.append(abs(by_hand.search(query, k=len(products)).reindex(products.index, fill_value=0) - scores).max()) print("query terms for pool floats:", bm25s.tokenize(["pool floats"], stopwords="en", stemmer=stemmer, return_ids=False, show_progress=False)[0]) print(f"largest score difference from bm25s with a stemmer: {max(gaps):.0e}") ``` Output: ```text query terms for pool floats: ['pool', 'float'] largest score difference from bm25s with a stemmer: 2e-06 ``` `pool floats` becomes `pool` and `float`, and the largest score difference over all products and queries is 2e-06, the same rounding as [without stemming](#bm25-from-scratch). **What to do:** stem English product text and queries with the Snowball English stemmer, at indexing time and at query time alike: in bm25s, pass `stemmer=` to `bm25s.tokenize` on both sides. Read the products that a stem adds, as the `floats` counts do, before you trust a gain. When a stem merges words that your shoppers keep apart, such as `molding` and `mold`, keep a list of words you don't stem. For text in another language, use that language's stemmer. ## Check stemming on the test queries {#check-stemming-on-test} [Stemming](#stem-word-forms) trades precision for recall, so only the judgments can say whether it pays for the store. The choice to stem comes from the 50 [train queries](#store-data). The check is the first of the chapter's [five planned comparisons](#lexical-retrieval) on the 30 test queries: the gain is established if the paired [permutation test](#permutation-test-on-queries) gives a p-value below 0.01, with the paired [t-test](#paired-t-test-on-queries) and a 99% [bootstrap interval](#report-the-size-of-a-gap) beside it. The example compares the [stemmed index](#stem-word-forms) with the unstemmed one from [the BM25 section](#bm25-from-scratch): Example `measure-stemming` (Python 3.12.15 · bm25s 0.3.13 · numpy 2.5.4 · pandas 3.0.6 · pystemmer 3.1.0 · scipy 1.18.1), passing in Edition 2026.10.1: ```python from lib.lexical_by_hand import THRESHOLD, InvertedIndex, report_test_comparison from lib.metrics import evaluate from lib.store import load_products, load_queries, product_text, qrels text = product_text(load_products()) indexes = {"without stemming": InvertedIndex(text), "with stemming": InvertedIndex(text, stem=True)} scores = {split: {name: evaluate(index.run(load_queries(split)), qrels(split)) for name, index in indexes.items()} for split in ["train", "test"]} for split in ["train", "test"]: for name, frame in scores[split].items(): print(f"{split} ndcg@10 {name}: {frame['ndcg@10'].mean():.3f}") print(f"{split} judged@10 {name}: {frame['judged@10'].mean():.3f}") change = (scores["train"]["with stemming"]["ndcg@10"] - scores["train"]["without stemming"]["ndcg@10"]).round(3) print(f"train queries better, worse: {(change > 0).sum()}, {(change < 0).sum()}") # for choosing, not evidence for query_id in [410, 359]: # pool floats and french molding, train queries for name, frame in scores["train"].items(): print(f"query {query_id} {name}: {frame.at[query_id, 'ndcg@10']:.3f}") print(f"\nthe planned test comparison, threshold {THRESHOLD:.2f}:") after, before = scores["test"]["with stemming"]["ndcg@10"], scores["test"]["without stemming"]["ndcg@10"] report_test_comparison(after, before) gain = after - before print("test query with the largest gain:", gain.idxmax()) print(f"its gain: {gain.max():.3f}") print(f"mean test gain without it: {gain.drop(gain.idxmax()).mean():.3f}") ``` Output: ```text train ndcg@10 without stemming: 0.695 train judged@10 without stemming: 0.930 train ndcg@10 with stemming: 0.725 train judged@10 with stemming: 0.950 test ndcg@10 without stemming: 0.725 test judged@10 without stemming: 0.927 test ndcg@10 with stemming: 0.774 test judged@10 with stemming: 0.950 train queries better, worse: 24, 14 query 410 without stemming: 0.360 query 410 with stemming: 0.921 query 359 without stemming: 0.961 query 359 with stemming: 0.809 the planned test comparison, threshold 0.01: mean difference: 0.050 wins, losses, ties: 16, 6, 8 permutation p: 0.005 established at 0.01: yes t-test p: 0.037 t-test agrees: no 99% interval: 0.016 to 0.162 interval agrees: yes test query with the largest gain: 133 its gain: 0.635 mean test gain without it: 0.029 ``` On the train queries, stemming takes the mean [nDCG@10](#score-one-ranking) from 0.695 to 0.725, and [judged@10](#score-every-test-query) from 0.930 to 0.950. The train queries that get better and worse number 24, 14. `pool floats` goes from 0.360 to 0.921, and `french molding` from 0.961 to 0.809. These train numbers choose stemming. They aren't evidence that it helps. On the test queries, judged@10 goes from 0.927 to 0.950. When the run without stemming has more unjudged products in its top 10s, part of the difference may come from them. Mean nDCG@10 goes from 0.725 to 0.774, a mean difference of 0.050. The permutation test establishes the gain at 0.01, and the t-test disagrees ([the planned comparisons](#lexical-retrieval-decisions)). One query explains why the two tests split. Query 133, `outdoor light fixtures`, gains 0.635, and without it the mean gain is 0.029. The t-test assumes the differences are roughly normal, so one large gain widens their spread and lowers t; the permutation test assumes nothing about their shape ([chapter 3](#when-a-difference-is-established)). That query is also the one whose unstemmed top 10 [chapter 1 found full of unjudged light bulbs](#score-every-test-query). **What to do:** choose a text-processing change on train queries, and keep it only if its planned test comparison is established. When the permutation test and the t-test disagree, read the largest per-query differences, and report the gain without the largest one beside the mean. Stemming is established, so stem when you build lexical search. The book's shared BM25 stays unstemmed, so that later chapters' results stay comparable ([decisions](#lexical-retrieval-decisions)). Treat the size of the gain as fragile. ## Tune BM25's parameters on train queries {#tune-bm25-parameters} BM25's two parameters have no single right value, and libraries ship different defaults. bm25s uses `k1=1.5` and `b=0.75` ([bm25s source](https://raw.githubusercontent.com/xhluca/bm25s/0.3.13/bm25s/__init__.py)), and [Lucene](#bm25-from-scratch) uses `k1=1.2` and `b=0.75` ([Lucene `BM25Similarity`](https://lucene.apache.org/core/10_3_0/core/org/apache/lucene/search/similarities/BM25Similarity.html)). [Kamphuis et al. (2020)](https://cs.uwaterloo.ca/~jimmylin/publications/Kamphuis_etal_ECIR2020_preprint.pdf) report `k1=0.9` and `b=0.4` as the defaults of Anserini, a research toolkit. [Robertson and Zaragoza (2009)](https://www.staff.city.ac.uk/~sbrp622/papers/foundations_bm25_review.pdf) give a rule of thumb, not a measurement: "values such as 0.5 < b < 0.8 and 1.2 < k1 < 2 are reasonably good in many circumstances", but the best values depend on "the type of documents or queries". So you tune them on your own train queries, and check the result on test queries. The [inverted index](#build-an-inverted-index) stores term frequencies and lengths, not scores, so `k1` and `b` can change from one search to the next without rebuilding it. bm25s computes every score when it builds its index ([Lù, 2024](https://arxiv.org/abs/2407.03618)), which makes its searches fast, but each new `k1` or `b` needs a new index. The example sweeps a grid of 25 settings over the [stemmed index](#stem-word-forms) on the 50 [train queries](#store-data), and picks the best. Then it compares the best with the defaults on the 30 test queries: the second of the chapter's [five planned comparisons](#lexical-retrieval), at the threshold of 0.01. Before you run it, predict: at the defaults, stemmed BM25 scores 0.774 on the test queries. How much will the best setting on train add? Example `sweep-k1-and-b` (Python 3.12.15 · bm25s 0.3.13 · numpy 2.5.4 · pandas 3.0.6 · pystemmer 3.1.0 · scipy 1.18.1), passing in Edition 2026.10.1: ```python import pandas as pd from lib.lexical_by_hand import InvertedIndex, report_test_comparison from lib.metrics import evaluate from lib.store import load_products, load_queries, product_text, qrels index = InvertedIndex(product_text(load_products()), stem=True) # stemmed terms, k1 and b set per search train, test = load_queries("train"), load_queries("test") k1_values, b_values = [0.5, 0.9, 1.2, 1.5, 2.0], [0.0, 0.25, 0.5, 0.75, 1.0] def scores(queries, split, k1, b): """One row per query, with BM25 at these parameters.""" return evaluate(index.run(queries, k1=k1, b=b), qrels(split)) grid = pd.DataFrame({b: {k1: scores(train, "train", k1, b)["ndcg@10"].mean() for k1 in k1_values} for b in b_values}) print("train ndcg@10, k1 down and b across:") print(grid.round(3).to_string()) best_k1, best_b = grid.stack().idxmax() print("settings in the grid:", grid.size) print(f"best k1 on train: {best_k1}") print(f"best b on train: {best_b}") edge = best_k1 in (k1_values[0], k1_values[-1]) or best_b in (b_values[0], b_values[-1]) print("best setting on the grid's edge:", "yes" if edge else "no") print(f"train ndcg@10 at the train best: {grid.at[best_k1, best_b]:.3f}") print(f"train ndcg@10 at k1 1.5, b 0.75: {grid.at[1.5, 0.75]:.3f}") print(f"train gain of the best over the defaults: {grid.at[best_k1, best_b] - grid.at[1.5, 0.75]:.3f}") print(f"lowest train ndcg@10 in the grid: {grid.min().min():.3f}") print() runs = {"at k1 1.5, b 0.75": scores(test, "test", 1.5, 0.75), "at the train best": scores(test, "test", best_k1, best_b)} for name, frame in runs.items(): print(f"test ndcg@10 {name}: {frame['ndcg@10'].mean():.3f}") print(f"test judged@10 {name}: {frame['judged@10'].mean():.3f}") report_test_comparison(runs["at the train best"]["ndcg@10"], runs["at k1 1.5, b 0.75"]["ndcg@10"]) ``` Output: ```text train ndcg@10, k1 down and b across: 0.00 0.25 0.50 0.75 1.00 0.5 0.707 0.710 0.716 0.720 0.720 0.9 0.711 0.711 0.716 0.722 0.720 1.2 0.709 0.711 0.715 0.723 0.720 1.5 0.709 0.714 0.719 0.725 0.715 2.0 0.714 0.720 0.726 0.729 0.713 settings in the grid: 25 best k1 on train: 2.0 best b on train: 0.75 best setting on the grid's edge: yes train ndcg@10 at the train best: 0.729 train ndcg@10 at k1 1.5, b 0.75: 0.725 train gain of the best over the defaults: 0.004 lowest train ndcg@10 in the grid: 0.707 test ndcg@10 at k1 1.5, b 0.75: 0.774 test judged@10 at k1 1.5, b 0.75: 0.950 test ndcg@10 at the train best: 0.775 test judged@10 at the train best: 0.953 mean difference: 0.001 wins, losses, ties: 7, 6, 17 permutation p: 0.757 established at 0.01: no t-test p: 0.754 t-test agrees: yes 99% interval: -0.008 to 0.008 interval agrees: yes ``` On train, the whole grid lies between 0.707 and 0.729. The best setting, `k1` = 2.0 and `b` = 0.75, scores 0.729, against 0.725 at the defaults. The example reports whether it lies on the grid's edge: yes. A wider grid could find a higher train score, but with the whole grid this close, it would change little. On test, the best setting scores 0.775 against 0.774, with [judged@10](#score-every-test-query) of 0.953 and 0.950. The difference isn't established ([the planned comparisons](#lexical-retrieval-decisions)). If you predicted a gain of a few hundredths, the train grid makes it unlikely. On train, the best setting beats the defaults by 0.004, and even that is an optimistic estimate: of all the settings, it is the one that happened to fit these queries best, so on other queries you should expect less. **What to do:** start from the library's defaults. Tune `k1` and `b` on train queries when you change what you index, such as fields or stemming, and keep a tuned setting only if its planned test comparison is established. The store's tuned setting isn't, so the store keeps `k1=1.5` and `b=0.75`. ## Weight the product name above the description {#weight-the-product-name} A product's text has parts, called **fields**: the store's text fields are name, class, category and description. They aren't equal evidence. A query word in the name says more about what the product is than the same word somewhere in a description. There are two ways to weight fields. The first that comes to mind scores each field with BM25 and adds the scores. [Robertson and Zaragoza (2009, section 3.6)](https://www.staff.city.ac.uk/~sbrp622/papers/foundations_bm25_review.pdf) argue for the opposite order, "first streams, then terms": add up each term's weighted frequencies across the fields first, then apply saturation "to the total evidence for each term". That is BM25F. Their rule implies that adding per-field scores lets a term that matches in two fields escape saturation, because each field saturates on its own. With whole-number weights, simple BM25F gives the same result as plain BM25 on a text in which each field is repeated as many times as its weight (Robertson and Zaragoza, section 3.6.4), so you can try it with any BM25 library. The store's search indexes one field: name, class, category and description, joined. The first example tries the way that comes to mind first, on the 50 [train queries](#store-data) with [stemming](#stem-word-forms): the name and the other three fields indexed apart, scored apart and added. Indexed apart, each field also gets its own IDF and its own average length, so the example changes more than saturation. By Robertson and Zaragoza's argument, we expect it to do worse. On train queries, this informs a choice and proves nothing: Example `fields-scored-apart` (Python 3.12.15 · bm25s 0.3.13 · numpy 2.5.4 · pandas 3.0.6 · pystemmer 3.1.0 · scipy 1.18.1), passing in Edition 2026.10.1: ```python from collections import Counter from lib.lexical_by_hand import InvertedIndex from lib.metrics import evaluate from lib.store import load_products, load_queries, product_text, qrels products, train = load_products(), load_queries("train") others = products["product_class"] + ". " + products["category"] + ". " + products["description"] one_field = InvertedIndex(product_text(products), stem=True) # name, class, category, description name, rest = InvertedIndex(products["name"], stem=True), InvertedIndex(others, stem=True) # own IDF and lengths summed = {} for query_id, query in train["query"].items(): scores = Counter(name.bm25(query)) + Counter(rest.bm25(query)) # each field saturates on its own summed[query_id] = [product_id for product_id, _ in scores.most_common(100)] runs = {"one field": one_field.run(train), "fields scored apart": summed} scores = {label: evaluate(run, qrels("train")) for label, run in runs.items()} for label, frame in scores.items(): print(f"{label}, train ndcg@10: {frame['ndcg@10'].mean():.3f}") print(f"{label}, train judged@10: {frame['judged@10'].mean():.3f}") change = (scores["fields scored apart"]["ndcg@10"] - scores["one field"]["ndcg@10"]).round(3) print(f"train queries better, worse with fields scored apart: {(change > 0).sum()}, {(change < 0).sum()}") ``` Output: ```text one field, train ndcg@10: 0.725 one field, train judged@10: 0.950 fields scored apart, train ndcg@10: 0.686 fields scored apart, train judged@10: 0.970 train queries better, worse with fields scored apart: 15, 24 ``` Scored apart, the fields give a train [nDCG@10](#score-one-ranking) of 0.686 against 0.725 for one field, with [judged@10](#score-every-test-query) of 0.970 and 0.950. The train queries that get better and worse number 15, 24. That is enough to choose one field over this design, and not enough to say why: saturation, IDF and length normalization all changed at once. The second example tries BM25F with whole-number weights: the name counted 1, 2, 3 or 5 times in the indexed text. It picks the weight with the best train score, then compares it with one field on the 30 test queries: the third of the chapter's [five planned comparisons](#lexical-retrieval). Example `repeat-the-name-field` (Python 3.12.15 · bm25s 0.3.13 · numpy 2.5.4 · pandas 3.0.6 · pystemmer 3.1.0 · scipy 1.18.1), passing in Edition 2026.10.1: ```python from lib.lexical_by_hand import InvertedIndex, report_test_comparison from lib.metrics import evaluate from lib.store import load_products, load_queries, product_text, qrels products, train, test = load_products(), load_queries("train"), load_queries("test") indexes = {weight: InvertedIndex((products["name"] + ". ") * (weight - 1) + product_text(products), stem=True) for weight in [1, 2, 3, 5]} # BM25F with whole-number weights: the name counted weight times train_scores = {weight: evaluate(index.run(train), qrels("train")) for weight, index in indexes.items()} for weight, frame in train_scores.items(): print(f"name weight {weight}, train ndcg@10: {frame['ndcg@10'].mean():.3f}") print(f"name weight {weight}, train judged@10: {frame['judged@10'].mean():.3f}") best = max(train_scores, key=lambda weight: train_scores[weight]["ndcg@10"].mean()) print("best name weight on train:", best) print() runs = {"one field": indexes[1].run(test), "best name weight": indexes[best].run(test)} scores = {label: evaluate(run, qrels("test")) for label, run in runs.items()} for label, frame in scores.items(): print(f"{label}, test ndcg@10: {frame['ndcg@10'].mean():.3f}") print(f"{label}, test judged@10: {frame['judged@10'].mean():.3f}") report_test_comparison(scores["best name weight"]["ndcg@10"], scores["one field"]["ndcg@10"]) ``` Output: ```text name weight 1, train ndcg@10: 0.725 name weight 1, train judged@10: 0.950 name weight 2, train ndcg@10: 0.734 name weight 2, train judged@10: 0.968 name weight 3, train ndcg@10: 0.729 name weight 3, train judged@10: 0.970 name weight 5, train ndcg@10: 0.722 name weight 5, train judged@10: 0.972 best name weight on train: 2 one field, test ndcg@10: 0.774 one field, test judged@10: 0.950 best name weight, test ndcg@10: 0.786 best name weight, test judged@10: 0.970 mean difference: 0.012 wins, losses, ties: 15, 9, 6 permutation p: 0.149 established at 0.01: no t-test p: 0.153 t-test agrees: yes 99% interval: -0.009 to 0.032 interval agrees: yes ``` On train, the best name weight is 2, which scores 0.734 against 0.725 for one field, with judged@10 of 0.968 and 0.950. On test, it scores 0.786 against 0.774, with judged@10 of 0.970 and 0.950. The gain isn't established; neither the t-test nor the interval disagrees ([the planned comparisons](#lexical-retrieval-decisions)). **What to do:** don't add up BM25 scores from separate fields. To weight a field, combine term frequencies before saturation, with BM25F or by repeating the field. Tune the weight on train queries, and keep it only if its planned test comparison is established. For the store, one combined field stays. ## Index product attributes as a field {#index-attributes-as-a-field} Shoppers often add a modifier, a word that narrows what they want: a color, a material, a size. The WANDS judges mark a product partial when it matches the thing asked for but not its [modifiers](#intent-in-product-search). Product names and descriptions don't always state a modifier, but a catalog often keeps such facts as structured **attributes**, such as `color: clear` or `material: glass`. The store's products carry a few from WANDS: color, material, style, finish, shape, size, width, height and depth (`examples/data/wands/README.md`). Indexed as one more field, attributes give a modifier something to match. They also add terms to every product that has them, so whether they help is a question for the judgments. In [chapter 1](#rank-with-bm25), BM25's results for `clear shower curtain` couldn't show which curtains were clear. The example adds each product's attributes, names and values, to its indexed text, as a field with weight 1, combined before saturation as [the previous section](#weight-the-product-name) recommends. It counts the exact products for that test query that contain `clear`, with and without attributes. Then it measures this one candidate, fixed in advance, on the 50 [train queries](#store-data), and checks it on the 30 test queries: the fourth of the chapter's [five planned comparisons](#lexical-retrieval). Being planned, it goes to test whatever train shows. Example `add-attributes-field` (Python 3.12.15 · bm25s 0.3.13 · numpy 2.5.4 · pandas 3.0.6 · pystemmer 3.1.0 · scipy 1.18.1), passing in Edition 2026.10.1: ```python from lib.lexical_by_hand import InvertedIndex, report_test_comparison, tokenize from lib.metrics import evaluate from lib.store import load_products, load_queries, product_text, qrels products = load_products() text = product_text(products) with_attributes = text + ". " + products["attributes"] # names and values, one more field, weight 1 exact = [product_id for product_id, grade in qrels("test")[246].items() if grade == 2] # clear shower curtain print("exact products for query 246:", len(exact)) print("exact, with clear in their text:", sum("clear" in tokenize(text[product_id]) for product_id in exact)) print("exact, with clear in text or attributes:", sum("clear" in tokenize(with_attributes[product_id]) for product_id in exact)) found = next(product_id for product_id in exact if "clear" in tokenize(products.at[product_id, "attributes"]) and "clear" not in tokenize(text[product_id])) print(f"for example: {products.at[found, 'name']} -> {products.at[found, 'attributes']}") names = {"color", "material", "style", "finish", "shape", "size", "width", "height", "depth"} # the attribute names print("queries that contain an attribute name:", load_queries()["query"].map(lambda query: bool(names & set(tokenize(query)))).sum()) print("products whose attributes name size:", sum("size" in tokenize(value) for value in products["attributes"])) indexes = {"one field": InvertedIndex(text, stem=True), "with attributes": InvertedIndex(with_attributes, stem=True)} scores = {split: {label: evaluate(index.run(load_queries(split)), qrels(split)) for label, index in indexes.items()} for split in ["train", "test"]} # one candidate, fixed in advance: tuned on nothing for split in ["train", "test"]: print() for label, frame in scores[split].items(): print(f"{split} ndcg@10, {label}: {frame['ndcg@10'].mean():.3f}") print(f"{split} judged@10, {label}: {frame['judged@10'].mean():.3f}") print() for query_id in [246, 58]: # clear shower curtain, medium size chandelier: test queries for label, frame in scores["test"].items(): print(f"query {query_id} ndcg@10, {label}: {frame.at[query_id, 'ndcg@10']:.3f}") report_test_comparison(scores["test"]["with attributes"]["ndcg@10"], scores["test"]["one field"]["ndcg@10"]) ``` Output: ```text exact products for query 246: 12 exact, with clear in their text: 4 exact, with clear in text or attributes: 8 for example: wynsum vinyl single shower curtain -> color: clear; width: 72; height: 84 queries that contain an attribute name: 1 products whose attributes name size: 950 train ndcg@10, one field: 0.725 train judged@10, one field: 0.950 train ndcg@10, with attributes: 0.717 train judged@10, with attributes: 0.948 test ndcg@10, one field: 0.774 test judged@10, one field: 0.950 test ndcg@10, with attributes: 0.787 test judged@10, with attributes: 0.963 query 246 ndcg@10, one field: 0.714 query 246 ndcg@10, with attributes: 0.844 query 58 ndcg@10, one field: 0.731 query 58 ndcg@10, with attributes: 0.667 mean difference: 0.013 wins, losses, ties: 8, 10, 12 permutation p: 0.303 established at 0.01: no t-test p: 0.281 t-test agrees: yes 99% interval: -0.009 to 0.053 interval agrees: yes ``` Of the 12 exact products for `clear shower curtain`, 4 say `clear` in their text, and 8 do once attributes count, such as the vinyl curtain whose attributes say `color: clear`. For this query, nDCG@10 goes from 0.714 to 0.844. Indexing the attribute names has a cost: `size` now matches the 950 products whose attributes give a size. The store has 1 query that contains an attribute name, the test query `medium size chandelier`, and it goes from 0.731 to 0.667. On train, judged@10 goes from 0.950 to 0.948, and nDCG@10 from 0.725 to 0.717. On test, judged@10 goes from 0.950 to 0.963, and nDCG@10 from 0.774 to 0.787. The permutation test doesn't establish the gain ([the planned comparisons](#lexical-retrieval-decisions)). **What to do:** keep structured attributes, and read the queries they help, but don't add them to the BM25 text on the strength of a few queries. Index the values without the names, so that a query word such as `size` doesn't match every product with a size, and keep the field only if its planned test comparison is established. For the store, the attributes stay out of the BM25 text. When a modifier names an attribute value exactly, such as a color, a filter or a ranking feature can use it without adding its words to every product's text. ## Match phrases with term positions {#match-phrases-with-positions} BM25 scores a bag of words: it counts which terms a product contains and how often, never their order. To BM25, `barn door` and `door barn` are the same query. A **phrase** match needs the terms next to each other and in order, and an index of counts can't check that. A **positional index** stores, for each term and product, the positions where the term occurs, so that a phrase becomes a check on positions ([Manning et al., 2008](https://nlp.stanford.edu/IR-book/html/htmledition/positional-indexes-1.html)). bm25s precomputes one score for each pair of product and term ([Lù, 2024](https://arxiv.org/abs/2407.03618)), and its index keeps no positions ([bm25s source](https://raw.githubusercontent.com/xhluca/bm25s/0.3.13/bm25s/__init__.py)), so phrases need an index of their own. The first example builds a positional index over the store's product names, with stemmed terms as in [the stemming section](#stem-word-forms). Positions count terms after stop words are removed, so `door for laundry` holds the adjacent terms `door` and `laundri`: a phrase can match across a removed stop word. The example checks that BM25 scores `barn door`, two words from one of the store's queries, the same in either order, and counts the names that contain both words against those that contain the phrase: Example `phrase-positions-by-hand` (Python 3.12.15 · bm25s 0.3.13 · numpy 2.5.4 · pandas 3.0.6 · pystemmer 3.1.0 · scipy 1.18.1), passing in Edition 2026.10.1: ```python from lib.lexical_by_hand import InvertedIndex, tokenize from lib.store import load_products, product_text products = load_products() positions = {} # {term: {product_id: [where the term occurs in the name, counted after stop words go]}} for product_id, name in products["name"].items(): for place, term in enumerate(tokenize(name, stem=True)): positions.setdefault(term, {}).setdefault(product_id, []).append(place) def has_phrase(product_id, terms): """True if the terms occur next to each other, in this order, in the product's name.""" def places(term): return positions.get(term, {}).get(product_id, []) return any(all(start + offset in places(term) for offset, term in enumerate(terms)) for start in places(terms[0])) index = InvertedIndex(product_text(products), stem=True) print("same bm25 scores for barn door and door barn:", index.bm25("barn door") == index.bm25("door barn")) both = set(positions["barn"]) & set(positions["door"]) print("names with barn and door:", len(both)) print("names with the phrase barn door:", sum(has_phrase(product_id, ["barn", "door"]) for product_id in both)) for product_id in sorted(product_id for product_id in both if not has_phrase(product_id, ["barn", "door"])): print(" ", products.at[product_id, "name"]) ``` Output: ```text same bm25 scores for barn door and door barn: True names with barn and door: 71 names with the phrase barn door: 66 low ceiling single bypass double door barn hardware kit artemas barn owl welcome non-slip outdoor door mat vintage rustic wooden door of old barn duvet cover set barnes entrance scraper non-slip outdoor door mat glass barn 60 '' w x 76 '' h single sliding frameless shower door ``` BM25 gives both word orders the same scores (`True`). Of the 71 names that contain both `barn` and `door`, 66 contain the phrase. The names without it show what a phrase check catches and what it costs. Two door mats and a duvet cover match both words but aren't barn doors, and one of the mats matches only because `barnes` stems to `barn`. The bypass hardware kit, on the other hand, is hardware for barn doors, and the glass sliding shower door is a barn-style door. A strict phrase filter would remove both. A bonus is gentler than a filter: a fixed amount added to a product's BM25 score for each pair of adjacent query terms that appears, in order, in its name. The second example uses `PositionalIndex` from `lib.lexical_by_hand`, which holds the same positions. It picks the bonus on the 50 [train queries](#store-data), then checks it on the 30 test queries: the last of the chapter's [five planned comparisons](#lexical-retrieval). Example `phrase-bonus-train-and-test` (Python 3.12.15 · bm25s 0.3.13 · numpy 2.5.4 · pandas 3.0.6 · pystemmer 3.1.0 · scipy 1.18.1), passing in Edition 2026.10.1: ```python from lib.lexical_by_hand import InvertedIndex, PositionalIndex, report_test_comparison, tokenize from lib.metrics import evaluate from lib.store import load_products, load_queries, product_text, qrels products = load_products() index = InvertedIndex(product_text(products), stem=True) names = PositionalIndex(products["name"], stem=True) # the positions phrase-positions-by-hand builds def with_bonus(split, bonus): """BM25's top 100, plus the bonus for each pair of adjacent query terms found, in order, in the name.""" run = {} for query_id, query in load_queries(split)["query"].items(): terms, scores = tokenize(query, stem=True), index.search(query) pairs = list(zip(terms, terms[1:])) scores += [bonus * sum(names.has_phrase(product_id, list(pair)) for pair in pairs) for product_id in scores.index] run[query_id] = list(scores.sort_values(ascending=False, kind="stable").index) return evaluate(run, qrels(split)) train_scores = {bonus: with_bonus("train", bonus) for bonus in [0, 0.5, 1, 2]} for bonus, frame in train_scores.items(): print(f"bonus {bonus}, train ndcg@10: {frame['ndcg@10'].mean():.3f}") print(f"bonus {bonus}, train judged@10: {frame['judged@10'].mean():.3f}") best = max(train_scores, key=lambda bonus: train_scores[bonus]["ndcg@10"].mean()) print("best bonus on train:", best) print() test_scores = {bonus: with_bonus("test", bonus) for bonus in [0, best]} for bonus, frame in test_scores.items(): print(f"bonus {bonus}, test ndcg@10: {frame['ndcg@10'].mean():.3f}") print(f"bonus {bonus}, test judged@10: {frame['judged@10'].mean():.3f}") report_test_comparison(test_scores[best]["ndcg@10"], test_scores[0]["ndcg@10"]) ``` Output: ```text bonus 0, train ndcg@10: 0.725 bonus 0, train judged@10: 0.950 bonus 0.5, train ndcg@10: 0.727 bonus 0.5, train judged@10: 0.956 bonus 1, train ndcg@10: 0.725 bonus 1, train judged@10: 0.964 bonus 2, train ndcg@10: 0.716 bonus 2, train judged@10: 0.968 best bonus on train: 0.5 bonus 0, test ndcg@10: 0.774 bonus 0, test judged@10: 0.950 bonus 0.5, test ndcg@10: 0.776 bonus 0.5, test judged@10: 0.950 mean difference: 0.002 wins, losses, ties: 6, 7, 17 permutation p: 0.758 established at 0.01: no t-test p: 0.769 t-test agrees: yes 99% interval: -0.012 to 0.015 interval agrees: yes ``` On train, the best bonus is 0.5, which scores 0.727 against 0.725 without one, with [judged@10](#score-every-test-query) of 0.956 and 0.950. On test, it scores 0.776 against 0.774, with judged@10 of 0.950 and 0.950. No gain is established ([the planned comparisons](#lexical-retrieval-decisions)). **What to do:** don't add a phrase bonus to every query by default, and keep one only if its planned test comparison is established. Build positions when shoppers quote phrases, or when word order changes what they want, as between `table lamp` and `lamp table`. Prefer a bonus that you measure on train queries to a filter, because a filter removes products like the hardware kit. ## What goes wrong in production {#lexical-retrieval-in-production} **The index and the queries drift apart.** Every step from [the tokenizer](#split-text-into-terms) to [the stemmer](#stem-word-forms) must run the same way on products and on queries. Stem the queries but not the products, and a query term such as `shelv` finds no postings. Nothing warns you: bm25s leaves out query tokens that aren't in its vocabulary ([bm25s source](https://raw.githubusercontent.com/xhluca/bm25s/0.3.13/bm25s/__init__.py)). Change text processing for both sides in one release, and rebuild the whole index. **Sizes, units and codes don't survive tokenizing.** `7qt` and `7 qt` never meet, as [the tokenizer section](#split-text-into-terms) shows. Train query 205 ends in a product code, `w003096809`, that appears in no product's text in the sample: its postings hold 0 products, so no lexical scorer can match it. A misspelled term such as the `ligth` in `ligth bulb` can only match products with the same misspelling, and [chapter 8](#correct-spelling-from-catalog) builds a corrector and shows what it still gets wrong. **IDF depends on the catalog.** Every IDF in this chapter comes from the sample, which holds every product judged for the 80 queries plus randomly chosen unjudged ones (`examples/data/wands/README.md`). On the full catalog, document frequencies differ, and so do IDFs and scores. Recompute them whenever you rebuild the index. **A BM25 score has no fixed scale.** Each query term adds at most its own IDF, so the highest possible score depends on which terms the query holds, and a score of 6 can be high for one query and low for another. Don't keep results above a fixed score threshold; cut the list by rank. **Stop words and stemming change meaning on a few queries.** The store's judged queries have no negations, but shoppers type them, and the default stop list removes `not` and `no`. Stemming hurts some queries badly, as `french molding` shows in [the stemming check](#check-stemming-on-test). The judgments can't warn you about queries they don't include. **The test queries wear out.** This chapter makes [five planned comparisons](#lexical-retrieval) on the same test queries, and the Bonferroni threshold of 0.01 keeps the chance of any false positive among them at 5% or less when none of the changes does anything. It can't count what nobody planned: every extra setting tried on test, and every look at a test query to explain a result, is another comparison. Confirm stemming's gain on queries and judgments that none of these choices has seen. **What to do:** keep a set of probe queries for negations, sizes and codes, and read their results after every change to text processing. Store the tokenizer's and stemmer's settings with the index they built, log the terms that each query becomes, and watch for queries whose terms have no postings. ## Decisions this chapter made {#lexical-retrieval-decisions} Each choice below is made on the train queries. Before reading any test result, the chapter fixed [five planned comparisons](#lexical-retrieval) on the test queries, read with [chapter 3's harness](#compare-runs-in-a-harness): stemming and the attributes field as single candidates fixed in advance, and the best setting on train for `k1` and `b`, the name weight and the phrase bonus. The attributes field went to test whatever train showed, because it was planned. Only an established difference changes a decision. Each column compares nDCG@10 on the test queries with the change against the same search without it, at the threshold of 0.01: | Measure | [Stemming](#check-stemming-on-test) | [Best `k1` and `b`](#tune-bm25-parameters) | [Best name weight](#weight-the-product-name) | [Attributes field](#index-attributes-as-a-field) | [Best phrase bonus](#match-phrases-with-positions) | | --- | --- | --- | --- | --- | --- | | Mean difference | 0.050 | 0.001 | 0.012 | 0.013 | 0.002 | | 99% interval | 0.016 to 0.162 | -0.008 to 0.008 | -0.009 to 0.032 | -0.009 to 0.053 | -0.012 to 0.015 | | Permutation p | 0.005 | 0.757 | 0.149 | 0.303 | 0.758 | | t-test p | 0.037 | 0.754 | 0.153 | 0.281 | 0.769 | | Wins, losses, ties | 16, 6, 8 | 7, 6, 17 | 15, 9, 6 | 8, 10, 12 | 6, 7, 17 | | Established | yes | no | no | no | no | | t-test agrees | no | yes | yes | yes | yes | | Interval agrees | yes | yes | yes | yes | yes | - **Tokenizer: bm25s's defaults.** Lowercasing, tokens of two or more word characters (letters, digits or underscores), and its 33 English stop words, rebuilt by hand and matched on every product. They lose single-digit sizes and negations. Too few judged queries could measure either: a size-aware pattern changes 4 train queries and 0 test queries, and 0 queries hold a negation. The defaults stay until probe queries or new judgments can. - **Stemming: Snowball English from PyStemmer, on products and queries.** Its test gain, from 0.725 to 0.774, is established by the permutation test, though the t-test disagrees and one query carries much of it. The book's shared baseline, `lib.lexical` from chapter 1, stays at bm25s's defaults without stemming, so that every chapter's results stay comparable. This chapter's stemmed BM25 is `InvertedIndex(text, stem=True)` in `lib.lexical_by_hand`, or bm25s with `stemmer=`. - **Scoring: BM25 as bm25s computes it by default,** Lucene's formula with exact lengths, `k1=1.5` and `b=0.75`, with equal scores ordered by product ID. The best setting on train isn't established on test, so the defaults stay. - **Fields: one field** that joins name, class, category and description. Per-field scores, added, score lower on train. The best name weight on train and the attributes field aren't established on test. - **Phrases: no phrase bonus.** Positions over product names are cheap to build, but the best bonus on train isn't established on test. - **Implementation: a by-hand index for experiments, a library for serving.** The by-hand index lets `k1` and `b` change without rebuilding, and bm25s precomputes scores so that searches are fast. On the same test queries, semantic retrieval with the recorded `Snowflake/snowflake-arctic-embed-m-v1.5` embeddings scores 0.830 ([chapter 1](#search-by-meaning)). # Semantic retrieval By the end of this chapter you can build semantic retrieval for a catalog from stored embeddings: compare vectors by cosine similarity, find a query's nearest products exactly with NumPy, switch to an approximate index when a catalog outgrows exact search and measure what the index misses, and find the queries where embeddings go wrong. You can also choose a vector size by testing it on your own judgments, and compare models the same way. For [the store](#store-data), the online retailer of furniture and home goods whose search this book improves chapter by chapter, [semantic retrieval](#search-by-meaning) became the ranker to beat in the first pipeline, which [chapter 1](#search-end-to-end) builds. On the [test queries](#store-data) it scored a mean [nDCG@10](#score-one-ranking) of 0.830, against [BM25](#rank-with-bm25)'s 0.725, a lead that [chapter 3 establishes](#compare-runs-in-a-harness). It ran as one class, `lib.semantic.SemanticSearch`, over vectors that one model computed once, ahead of time. Before the store depends on it, you need to know what those vectors hold, what searching them would cost for a far larger catalog, and which queries they get wrong. This chapter builds exact search by hand, measures an approximate index against it, and tests both on the store's queries and judgments. On the test queries, the chapter plans 2 comparisons, both in [the section on testing the index](#hnsw-on-test-queries). Each sets exact search, which compares a query with every product, against HNSW, an approximate index that compares it with only a fraction of them: once at faiss's default setting, and once at the setting chosen on the train queries. Under [the book's rule](#when-a-difference-is-established), a difference is established when its paired permutation p-value is below 0.025. Every other look at a test query in this chapter is an exploratory diagnosis. ## Turn text into embeddings {#turn-text-into-embeddings} An **embedding** is a list of numbers, a vector, that a trained model computes for a piece of text. The model is an **encoder**, trained so that texts with similar meanings get vectors that point in similar directions, whether or not they share words. Every text gets the same count of numbers, the embedding's **dimension**, whether it's a three-word query or a product's whole description. One number means nothing on its own; only comparisons between whole vectors do. Search uses the encoder as a **bi-encoder**: it encodes each product by itself, once, ahead of time, and each query by itself when the shopper types it. Comparing a query with a product is then arithmetic on two vectors rather than a run of the model, which is what makes it cheap enough to compare one query with a whole catalog. [Chapter 1](#search-by-meaning) gives the size of the difference from the Sentence-BERT paper: about 65 hours for BERT to read every pair of 10,000 sentences, against about 5 seconds to embed each sentence once and compare. A model that reads a query and a product together is a cross-encoder, and [chapter 1](#rerank-with-cross-encoder) uses one to rerank a shortlist. The store's encoder is `Snowflake/snowflake-arctic-embed-m-v1.5`, an open model with 109 million parameters under the Apache-2.0 licence ([model card](https://huggingface.co/Snowflake/snowflake-arctic-embed-m-v1.5)). Its family's training kept only documents "primarily identified as English" ([Merrick et al., 2024](https://arxiv.org/abs/2405.05374)). We chose it because anyone may use it commercially, it's small, and it was trained so that its vectors can be shortened, which [the section on dimensions](#truncate-dimensions-test-loss) tests. The examples never run the model. They read a **recording**: vectors that the script `recordings/embeddings.py` computed once, at a pinned revision, and saved at unit length, so every example sees the same numbers. The model card asks you to "use the query prefix below (just on the query)", so the script adds that prefix to each query and embeds products as they are. Examples read the vectors through `lib.recorded.embeddings()`: Example `inspect-recorded-embeddings` (Python 3.12.15 · numpy 2.5.4 · pandas 3.0.6), passing in Edition 2026.10.1: ```python import re import numpy as np from lib.recorded import RECORDED, embeddings, manifest from lib.store import load_queries product_ids, product_vectors, query_ids, query_vectors = embeddings() recording = manifest("embeddings") print("embedding model:", recording["model"]) print("query prompt:", recording["prompts"]["query"]) print(f"products embedded: {len(product_ids):,}") print("dimensions:", product_vectors.shape[1]) print("stored as:", recording["dtype"]) print("longest product text, tokens:", recording["longest_product_tokens"]) print("model's input limit, tokens:", recording["max_seq_length"]) print(f"product {product_ids[0]}, first four numbers:", ", ".join(f"{x:.3f}" for x in product_vectors[0, :4])) stored = np.load(RECORDED / "embeddings" / "products.npy").astype(np.float32) # as saved, before lib.recorded stored_lengths, lengths = np.linalg.norm(stored, axis=1), np.linalg.norm(product_vectors, axis=1) print(f"stored vector lengths: {stored_lengths.min():.5f} to {stored_lengths.max():.5f}") print(f"lengths after lib.recorded rescales: {lengths.min():.5f} to {lengths.max():.5f}") # How close are two shopper queries? Compare every query with every other one. queries = load_queries().loc[query_ids, "query"].tolist() similarity = query_vectors @ query_vectors.T np.fill_diagonal(similarity, np.nan) # leave out each query's similarity to itself print(f"median similarity of two queries: {np.nanmedian(similarity):.3f}") for query in ["whimsical solar lights", "wall design shelf"]: row = queries.index(query) nearest = int(np.nanargmax(similarity[row])) shared = set(re.findall(r"\w+", query)) & set(re.findall(r"\w+", queries[nearest])) print() print(f"{query}, nearest other query:", queries[nearest]) print(f"{query}, similarity: {similarity[row, nearest]:.3f}") print(f"{query}, words in common:", len(shared)) ``` Output: ```text embedding model: Snowflake/snowflake-arctic-embed-m-v1.5 query prompt: Represent this sentence for searching relevant passages: products embedded: 12,000 dimensions: 768 stored as: float16 longest product text, tokens: 201 model's input limit, tokens: 512 product 1, first four numbers: -0.033, 0.051, 0.111, -0.012 stored vector lengths: 0.99994 to 1.00006 lengths after lib.recorded rescales: 1.00000 to 1.00000 median similarity of two queries: 0.291 whimsical solar lights, nearest other query: outdoor light fixtures whimsical solar lights, similarity: 0.637 whimsical solar lights, words in common: 0 wall design shelf, nearest other query: kitchen storage shelves wall design shelf, similarity: 0.679 wall design shelf, words in common: 0 ``` Each of the 12,000 products is a vector of 768 numbers. Product 1's begins -0.033, 0.051, 0.111, -0.012, and none of those numbers means anything alone. The recording stores each number as a `float16`, in 2 bytes, and `lib.recorded` widens them to 32-bit floats when it loads them. That rounding moves the stored lengths slightly off 1: they run from 0.99994 to 1.00006. So `lib.recorded` also scales each vector back to length 1, and the lengths it returns run from 1.00000 to 1.00000. The model reads text as **tokens**, word pieces from its own vocabulary, not the terms of [chapter 5's tokenizer](#split-text-into-terms). The longest product text is 201 tokens, inside the model's limit of 512, so the model read every product in full. The comparison of queries at the end of the output shows what the vectors capture. The median similarity between two of the store's queries is 0.291, on a scale where 1 means two vectors point the same way ([cosine similarity](#compare-with-cosine-similarity)). The nearest query to `whimsical solar lights` is `outdoor light fixtures`, at 0.637, though they share 0 words: `lights` and `light` are different strings. The nearest to `wall design shelf` is `kitchen storage shelves`, at 0.679, with 0 words in common. The store's baseline BM25 doesn't stem words, so it sees nothing shared in either pair. [Chapter 5's stemming](#stem-word-forms) would match `lights` with `light`, but not `shelves` with `shelf`. **What to do:** embed products ahead of time and queries at search time, with the same model at the same revision, and follow the model card on prompts: here, a prefix on queries only. Record the model, revision and prompts next to the vectors, as each `manifest.json` in `examples/data/recorded/` does. Check that your longest product fits the model's input limit before you trust its embedding. ## Compare embeddings with cosine similarity {#compare-with-cosine-similarity} To rank products for a query, you need one number per pair that says how alike two vectors are. **Cosine similarity** is the cosine of the angle between them: 1 when they point the same way, 0 at right angles and −1 when they point in opposite directions. It ignores how long the vectors are. The **dot product** multiplies two vectors number by number and adds up the results, so it equals the cosine times both lengths. Scaling every vector to length 1 is called **normalizing**, and after it the two are the same number: "Dot product on normalized embeddings is equivalent to cosine similarity" ([Sentence Transformers](https://sbert.net/docs/sentence_transformer/usage/semantic_textual_similarity.html)). The dot product is then the cheaper choice, because it skips computing the lengths. ```text dot(q, d) = q1 × d1 + q2 × d2 + ... + qn × dn length(q) = √(q1² + q2² + ... + qn²) cosine(q, d) = dot(q, d) / (length(q) × length(d)) ``` In the store, every query is compared with every product this way. For the test query `clear shower curtain`, compute the dot product and the cosine by hand for its most similar product, and check the cosine with scikit-learn 1.9.1. Then see what happens when vectors aren't unit length. Keep the first 256 of each vector's 768 numbers, a shortcut that [the section on dimensions](#truncate-dimensions-test-loss) tests, but skip the rescaling, and rank every query's products both ways: Example `cosine-by-hand` (Python 3.12.15 · numpy 2.5.4 · pandas 3.0.6 · scikit-learn 1.9.1 · scipy 1.18.1), passing in Edition 2026.10.1: ```python import numpy as np from sklearn.metrics.pairwise import cosine_similarity from lib.recorded import embeddings from lib.store import load_products products = load_products() product_ids, product_vectors, query_ids, query_vectors = embeddings() query = query_vectors[list(query_ids).index(246)] # clear shower curtain best = int(np.argmax(product_vectors @ query)) product = product_vectors[best] print("product:", products.at[product_ids[best], "name"]) # cosine = dot product / (length of q * length of d) dot = query @ product cosine = dot / (np.linalg.norm(query) * np.linalg.norm(product)) print(f"dot product: {dot:.3f}") print(f"cosine by hand: {cosine:.3f}") print(f"cosine from scikit-learn: {cosine_similarity([query], [product])[0, 0]:.3f}") # Keep the first 256 numbers of each vector and don't rescale: the lengths are no longer 1. short_products, short_queries = product_vectors[:, :256], query_vectors[:, :256] lengths = np.linalg.norm(short_products, axis=1) print(f"shortened product lengths: {lengths.min():.3f} to {lengths.max():.3f}") by_dot = np.argsort(-(short_queries @ short_products.T), axis=1, kind="stable")[:, :10] by_cosine = np.argsort(-(short_queries @ (short_products / lengths[:, None]).T), axis=1, kind="stable")[:, :10] changed = sum(set(dot_top) != set(cosine_top) for dot_top, cosine_top in zip(by_dot, by_cosine)) print("queries:", len(query_ids)) print("queries whose top 10 changes:", changed) ``` Output: ```text product: woolley vinyl solid color single shower curtain dot product: 0.577 cosine by hand: 0.577 cosine from scikit-learn: 0.577 shortened product lengths: 0.706 to 0.778 queries: 80 queries whose top 10 changes: 61 ``` For the store's unit-length vectors, the dot product is 0.577, the cosine by hand is 0.577 and scikit-learn's `cosine_similarity` gives 0.577. The shortened vectors have lengths from 0.706 to 0.778, and now the two measures disagree. For 61 of the 80 queries, ranking by dot product gives a different top 10 from ranking by cosine. A query's own length scales all of its scores by the same factor, so it can't change their order. A product's length can: it multiplies that product's score for every query, a boost that has nothing to do with what the shopper asked for. Some libraries report a distance instead of a similarity. faiss's L2 indexes return the squared Euclidean distance, and its inner product "is not by itself cosine similarity, unless the vectors are normalized" ([faiss wiki](https://github.com/facebookresearch/faiss/wiki/MetricType-and-distances)). For unit vectors, the squared distance is 2 − 2 × cosine, so it ranks products in the same order as cosine, smallest distance first. **What to do:** normalize every vector once, when you store it, and rank by dot product. Normalize again after anything that changes a vector, such as shortening it or averaging several. If a model's documentation says it was trained for the dot product of unnormalized vectors, follow the documentation instead, and never mix the two in one index. ## Find the exact nearest neighbours with NumPy {#exact-nearest-neighbours-numpy} Retrieval by embeddings is a **nearest-neighbour search**: find the k products whose vectors are most similar to the query's. **Exact** search compares the query with every product and keeps the best k, so it never misses one. With [unit-length vectors](#compare-with-cosine-similarity), comparing one query with every product is one matrix–vector product, and comparing many queries is one matrix product, with a row per query and a column per product. What's left is to find the k highest scores in each row. Sorting every row does more work than needed. NumPy's `argpartition` moves the k highest to one side without ordering either side ("The order of all elements in the partitions is undefined", [NumPy documentation](https://numpy.org/doc/stable/reference/generated/numpy.argpartition.html)), so you sort only those k. In the store, [chapter 1's semantic search](#search-by-meaning) took its rankings from `lib.semantic.SemanticSearch`. Build its core by hand for the 30 test queries with NumPy 2.5.4, then check it twice: against `lib.semantic`, and against faiss's `IndexFlatIP`, which computes the inner product with every vector. With `IndexFlatL2`, it is one of the two faiss indexes that "can guarantee exact results" ([faiss wiki](https://github.com/facebookresearch/faiss/wiki/Guidelines-to-choose-an-index)): Example `exact-search-numpy` (Python 3.12.15 · faiss-cpu 1.15.1 · numpy 2.5.4 · pandas 3.0.6), passing in Edition 2026.10.1: ```python import faiss import numpy as np from lib.metrics import evaluate from lib.recorded import embeddings from lib.semantic import SemanticSearch from lib.store import load_queries, qrels product_ids, product_vectors, query_ids, query_vectors = embeddings() test = load_queries("test").index rows = np.isin(query_ids, test) def top_k(scores, k): """Column indices of the k highest scores in each row, best first.""" candidates = np.argpartition(-scores, k - 1, axis=1)[:, :k] # the top k, in no order order = np.argsort(-np.take_along_axis(scores, candidates, axis=1), axis=1, kind="stable") return np.take_along_axis(candidates, order, axis=1) scores = query_vectors[rows] @ product_vectors.T # one row per test query, one column per product top = top_k(scores, 10) print("similarities computed:", f"{scores.size:,}") print("similarities per query:", f"{scores.shape[1]:,}") semantic = SemanticSearch() same = sum(list(product_ids[found]) == list(semantic.search(query_id, 10).index) for query_id, found in zip(query_ids[rows], top)) print("same top 10 as lib.semantic:", same, "of", rows.sum()) flat = faiss.IndexFlatIP(product_vectors.shape[1]) # faiss's exact index: inner product with every product flat.add(product_vectors) _, found = flat.search(query_vectors[rows], 10) print("same top 10 as faiss IndexFlatIP:", (found == top).all(axis=1).sum(), "of", rows.sum()) run = {query_id: list(product_ids[row]) for query_id, row in zip(query_ids[rows], top)} print(f"ndcg@10 of the exact run: {evaluate(run, qrels('test'))['ndcg@10'].mean():.3f}") ``` Output: ```text similarities computed: 360,000 similarities per query: 12,000 same top 10 as lib.semantic: 30 of 30 same top 10 as faiss IndexFlatIP: 30 of 30 ndcg@10 of the exact run: 0.830 ``` One matrix product computes 360,000 similarities, 12,000 per query. The hand-built search returns the same top 10 as `lib.semantic` for 30 of 30 test queries, and the same as faiss for 30 of 30, so its run scores the same mean nDCG@10, 0.830. The book's shared module for vectors, `lib.vectors`, keeps this `top_k` for later examples and wraps it in `exact_search`. What decides whether exact search is enough is its cost. Its work grows in step with the catalog: one similarity per product, for every query. Its memory is the vectors themselves: Example `exact-search-memory` (Python 3.12.15 · numpy 2.5.4 · pandas 3.0.6), passing in Edition 2026.10.1: ```python from lib.recorded import embeddings # Exact search keeps every product vector in memory, as 32-bit floats. product_ids, product_vectors, _, _ = embeddings() print(f"products: {len(product_ids):,}") print(f"float32 bytes per product: {product_vectors[0].nbytes:,}") print(f"product vectors in memory, MB: {product_vectors.nbytes / 1e6:.1f}") print(f"the same for 1,000,000 products, GB: {1_000_000 * product_vectors[0].nbytes / 1e9:.1f}") ``` Output: ```text products: 12,000 float32 bytes per product: 3,072 product vectors in memory, MB: 36.9 the same for 1,000,000 products, GB: 3.1 ``` Each product takes 3,072 bytes as 32-bit floats, the store's catalog 36.9 MB, and a million products would take 3.1 GB. The faiss authors advise deciding first "whether indexing is needed at all: indeed, in some cases a direct brute force search is the best option" ([Douze et al., 2024](https://arxiv.org/abs/2401.08281)). In our judgement, the store's catalog is such a case. **What to do:** we recommend exact search until it no longer fits in memory or meets your latency budget, measured with your own catalog on your own hardware. It has no settings to tune and never misses a neighbour, so it is also the ground truth for any approximate index. Store the normalized vectors as one 32-bit matrix, take the top k with `argpartition`, and sort only those. ## Trade recall for speed with HNSW {#hnsw-recall-for-speed} When [exact search](#exact-nearest-neighbours-numpy) costs too much, an **approximate nearest-neighbour** (ANN) index skips most of its comparisons, and in return may miss some of the true nearest neighbours. Its quality is its **recall@k against exact search**: the share of exact search's top k that it also returns. That isn't the [recall@k of chapter 3](#count-relevant-in-top-k), which compares a ranking with judgments. An index with perfect recall against exact search returns what exact search returns, good or bad. **HNSW**, short for hierarchical navigable small world, is a graph index. Each product is a node linked to some of its nearest neighbours, in a stack of layers: the bottom layer holds every product, and each layer above holds fewer. A search starts at the top, moves to whichever linked node is nearest the query, steps down a layer when it gets no closer, and at the bottom explores a list of candidates before it returns the best k ([Malkov and Yashunin, 2018](https://arxiv.org/abs/1603.09320)). In faiss, `M` sets the links per node, and more links find more true neighbours but use more memory ([faiss wiki](https://github.com/facebookresearch/faiss/wiki/Guidelines-to-choose-an-index)). `efConstruction`, the "expansion factor at construction time", sets how many candidates the build considers, and `efSearch`, the "expansion factor at search time", how many a search considers. faiss 1.15.1 defaults them to 40 and 16, and its graph build is deterministic: "This is the only graph build that add() uses" ([`HNSW.h`](https://raw.githubusercontent.com/facebookresearch/faiss/v1.15.1/faiss/impl/HNSW.h); [changelog](https://raw.githubusercontent.com/facebookresearch/faiss/v1.15.1/CHANGELOG.md)). So the outputs here repeat exactly. Timings differ from machine to machine, so the examples count work instead: faiss records the "number of distances computed" in each search (`HNSW.h`), which for this index are similarities. The count leaves out following links and reading memory, so treat it as a guide to speed, not a measure of it. The store doesn't need an index, but a store with millions of products would, and a catalog small enough to check every answer is the place to learn the trade-off. Build HNSW over the store's 12,000 products with `hnsw_index(product_vectors, links=32)`, `lib.vectors`'s name for `M = 32`. Then sweep `efSearch` from 16 to 256 on the [train queries](#store-data): Example `hnsw-efsearch-sweep` (Python 3.12.15 · faiss-cpu 1.15.1 · numpy 2.5.4 · pandas 3.0.6), passing in Edition 2026.10.1: ```python import faiss import numpy as np from lib.recorded import embeddings from lib.store import load_products, load_queries from lib.vectors import EF_SEARCH_GRID, WORST_RECALL, exact_search, hnsw_index, hnsw_search, recall_against_exact product_ids, product_vectors, query_ids, query_vectors = embeddings() names = load_products().loc[product_ids, "name"].str[:36].to_numpy() # by row number, cut to 36 characters rows = np.isin(query_ids, load_queries("train").index) # tune on train queries only queries = load_queries().loc[query_ids[rows], "query"].to_numpy() index = hnsw_index(product_vectors, links=32) # M = 32, and faiss's default efConstruction of 40 levels = faiss.vector_to_array(index.hnsw.levels) # how many layers each product reaches for layer in range(1, levels.max() + 1): print(f"products in layer {layer}: {(levels >= layer).sum():,}") exact = exact_search(query_vectors[rows], product_vectors, 10) print("\nefSearch mean recall@10 worst queries at 0 similarities per query") sweep = {} for ef_search in EF_SEARCH_GRID: found, similarities = hnsw_search(index, query_vectors[rows], 10, ef_search) recall = recall_against_exact(found, exact) sweep[ef_search] = (recall, similarities) print(f"{ef_search:>8} {recall.mean():>14.3f} {recall.min():>5.1f} {(recall == 0).sum():>12} {similarities:>22.0f}") recall, similarities = sweep[16] # faiss's default print(f"\nefSearch 16, recall@10 against exact: {recall.mean():.3f}") print(f"efSearch 16, similarities computed per query: {similarities:.0f}") print(f"efSearch 16, train queries with recall@10 of 0: {(recall == 0).sum()}") print("efSearch 16, those queries:", ", ".join(queries[recall == 0])) print("smallest efSearch with no query at 0:", min(ef for ef, (r, _) in sweep.items() if r.min() > 0)) chosen = min(ef for ef, (r, _) in sweep.items() if r.min() >= WORST_RECALL) print(f"smallest efSearch with every query at {WORST_RECALL} or more:", chosen) print(f"chosen efSearch, recall@10 against exact: {sweep[chosen][0].mean():.3f}") print(f"chosen efSearch, similarities computed per query: {sweep[chosen][1]:.0f}\n") found, _ = hnsw_search(index, query_vectors[rows], 3, 16) row = list(queries).index("foutains with brick look") for rank, (approximate, nearest) in enumerate(zip(found[row], exact[row]), start=1): print(f"{rank} hnsw: {names[approximate]:<36} exact: {names[nearest]}") ``` Output: ```text products in layer 1: 12,000 products in layer 2: 369 products in layer 3: 10 efSearch mean recall@10 worst queries at 0 similarities per query 16 0.934 0.0 2 282 32 0.936 0.0 2 346 48 0.940 0.0 2 404 64 0.940 0.0 2 463 80 0.956 0.0 1 530 96 0.978 0.3 0 611 112 0.980 0.3 0 682 128 0.994 0.8 0 767 144 0.998 0.9 0 835 160 0.998 0.9 0 924 176 0.998 0.9 0 1020 192 0.998 0.9 0 1115 208 0.998 0.9 0 1191 224 1.000 1.0 0 1271 240 1.000 1.0 0 1349 256 1.000 1.0 0 1441 efSearch 16, recall@10 against exact: 0.934 efSearch 16, similarities computed per query: 282 efSearch 16, train queries with recall@10 of 0: 2 efSearch 16, those queries: foutains with brick look, sheffield home bath set smallest efSearch with no query at 0: 96 smallest efSearch with every query at 0.9 or more: 144 chosen efSearch, recall@10 against exact: 0.998 chosen efSearch, similarities computed per query: 835 1 hnsw: dayse outdoor square cushion with fi exact: tribeca 2 '' x 10 '' porcelain brick 2 hnsw: 3 piece wall decor set exact: barbados 5 '' x 10 '' ceramic brick 3 hnsw: rustic autumn ivy branch exact: kimary 0.75 '' x 1.5 '' glass brick ``` The layers hold 12,000, 369 and 10 products. At the default `efSearch` of 16, a query computes 282 similarities instead of 12,000, and mean recall@10 against exact search is 0.934. The mean hides a failure: 2 train queries get none of their true top 10. They are foutains with brick look, sheffield home bath set. For `foutains with brick look`, HNSW returns a cushion, a wall decor set and an ivy branch, while exact search's top 3 are brick-look tiles: for this query, the search ended in a part of the graph far from its true neighbours. The smallest `efSearch` in the sweep at which no train query is at 0 is 96, but the example's table shows that its worst query still misses most of its true top 10. So choose by a stricter rule: the smallest `efSearch` at which every train query finds at least 9 of its true top 10. That is 144, with a mean recall of 0.998 and 835 similarities per query. **What to do:** use an ANN index only when exact search is too slow or too large. Tune it on train queries by its recall against exact search, at the k your pipeline uses, and choose by the worst query, not the mean. We recommend the smallest `efSearch` at which every query keeps at least 9 of its true top 10, with `M = 32`. This chapter tunes only `efSearch`; `M` and `efConstruction` change the graph itself, and deserve the same sweep on a large catalog. Very different ANN methods "yield comparable quality-performance trade-offs" ([Aumüller et al., 2018](https://arxiv.org/abs/1807.05614)), so measure on your own data rather than choosing a method by reputation. ## Check the index on the test queries {#hnsw-on-test-queries} An index tuned on train queries still has to be checked on queries it hasn't seen, by its [recall against exact search](#hnsw-recall-for-speed) and by its effect on the judged scores. In the store, these are the chapter's 2 planned test comparisons: the HNSW index at faiss's default `efSearch` of 16, and at the setting chosen on the train queries, each against exact search. A difference is established when its paired [permutation test](#permutation-test-on-queries) gives p below 0.025, with the [t-test and an interval](#when-a-difference-is-established) at the matching level beside it. The example also asks for `k = 100`, because [hybrid retrieval](#fuse-with-rrf) fuses each retriever's top 100: Example `hnsw-test-recall` (Python 3.12.15 · faiss-cpu 1.15.1 · numpy 2.5.4 · pandas 3.0.6 · scipy 1.18.1), passing in Edition 2026.10.1: ```python import numpy as np from lib.metrics import evaluate from lib.recorded import embeddings from lib.significance import compare, verdict from lib.store import load_queries, qrels from lib.vectors import as_run, choose_ef_search, exact_search, hnsw_index, hnsw_search, recall_against_exact # Planned before scoring: HNSW at faiss's default efSearch, and at the setting chosen on train, each against exact. PLANNED = 2 threshold = 0.05 / PLANNED product_ids, product_vectors, query_ids, query_vectors = embeddings() rows = np.isin(query_ids, load_queries("test").index) index = hnsw_index(product_vectors, links=32) chosen = choose_ef_search(index, query_vectors[np.isin(query_ids, load_queries("train").index)], product_vectors) exact = exact_search(query_vectors[rows], product_vectors, 100) exact_ndcg = evaluate(as_run(exact, query_ids[rows], product_ids), qrels("test"))["ndcg@10"] print("planned test comparisons:", PLANNED) print(f"p-value threshold: {threshold:.3f}") print("efSearch chosen on train queries:", chosen) print(f"exact ndcg@10: {exact_ndcg.mean():.3f}") for name, ef_search in [("default", 16), ("chosen", chosen)]: found, _ = hnsw_search(index, query_vectors[rows], 100, ef_search) recall = recall_against_exact(found[:, :10], exact[:, :10]) ndcg = evaluate(as_run(found, query_ids[rows], product_ids), qrels("test"))["ndcg@10"] result = compare(ndcg, exact_ndcg, confidence=1 - threshold) print(f"\n{name}, recall@10 against exact: {recall.mean():.3f}") print(f"{name}, queries with recall@10 of 0: {(recall == 0).sum()}") print(f"{name}, those queries:", ", ".join(load_queries().loc[query_ids[rows][recall == 0], "query"]) or "none") print(f"{name}, recall@100 against exact: {recall_against_exact(found, exact).mean():.3f}") print(f"{name}, hnsw ndcg@10: {ndcg.mean():.3f}") print(f"{name}, queries whose ndcg@10 changes: {result['wins'] + result['losses']}") print(f"{name}, permutation p: {result['permutation p']:.3f}") print(f"{name}, t-test p: {result['t-test p']:.3f}") print(f"{name}, {1 - threshold:.1%} interval: {result['ci low']:.4f} to {result['ci high']:.4f}") for check, passed in verdict(result, threshold).items(): # established, t-test agrees, interval agrees print(f"{name}, {check}: {'yes' if passed else 'no'}") ``` Output: ```text planned test comparisons: 2 p-value threshold: 0.025 efSearch chosen on train queries: 144 exact ndcg@10: 0.830 default, recall@10 against exact: 0.930 default, queries with recall@10 of 0: 2 default, those queries: liberty hardware francisco, 48" sliding single track , barn door for laundry default, recall@100 against exact: 0.849 default, hnsw ndcg@10: 0.791 default, queries whose ndcg@10 changes: 3 default, permutation p: 0.256 default, t-test p: 0.154 default, 97.5% interval: -0.1544 to -0.0006 default, established: no default, t-test agrees: yes default, interval agrees: no chosen, recall@10 against exact: 1.000 chosen, queries with recall@10 of 0: 0 chosen, those queries: none chosen, recall@100 against exact: 0.994 chosen, hnsw ndcg@10: 0.830 chosen, queries whose ndcg@10 changes: 0 chosen, permutation p: 1.000 chosen, t-test p: 1.000 chosen, 97.5% interval: 0.0000 to 0.0000 chosen, established: no chosen, t-test agrees: yes chosen, interval agrees: yes ``` At the chosen `efSearch` of 144, recall@10 against exact search is 1.000, the nDCG@10 of 0 queries changes, and the mean nDCG@10 is 0.830, as for exact search (0.830). At the default, 2 test queries get none of their true top 10: liberty hardware francisco, 48" sliding single track , barn door for laundry. The mean nDCG@10 falls to 0.791. Only 3 queries change score, and the fall isn't established: the permutation test gives p = 0.256, and the t-test agrees with p = 0.154. The 97.5% interval, from -0.1544 to -0.0006, excludes 0, so it disagrees with the tests. With so few queries carrying the difference, the finding is the recall lost against exact search, not a measured loss of quality. Asking for more results doesn't make up for a small `efSearch`: at the default, recall@100 is 0.849, against 0.994 at the chosen setting. The graph also costs memory, on top of the vectors: Example `hnsw-index-memory` (Python 3.12.15 · faiss-cpu 1.15.1 · numpy 2.5.4 · pandas 3.0.6), passing in Edition 2026.10.1: ```python import faiss from lib.recorded import embeddings from lib.vectors import hnsw_index product_ids, product_vectors, _, _ = embeddings() dimensions = product_vectors.shape[1] flat = faiss.IndexFlatIP(dimensions) # exact search: the vectors only flat.add(product_vectors) index = hnsw_index(product_vectors, links=32) # the vectors, plus the graph's links print("links per product, upper layers:", index.hnsw.nb_neighbors(1)) print("links per product, bottom layer:", index.hnsw.nb_neighbors(0)) print(f"faiss's estimate, 4d + 8M bytes per product: {4 * dimensions + 8 * 32:,}") for name, built in [("exact index", flat), ("hnsw index", index)]: print(f"bytes per product, {name}: {faiss.serialize_index(built).nbytes / len(product_ids):,.0f}") ``` Output: ```text links per product, upper layers: 32 links per product, bottom layer: 64 faiss's estimate, 4d + 8M bytes per product: 3,328 bytes per product, exact index: 3,072 bytes per product, hnsw index: 3,344 ``` Each product has 32 links on the upper layers and 64 on the bottom one, twice `M`. faiss estimates 4d + 8M bytes per vector ([faiss wiki](https://github.com/facebookresearch/faiss/wiki/Guidelines-to-choose-an-index)): 4 bytes for each of the d numbers, and 4 bytes for each of the 2M links on the bottom layer, 3,328 here. The saved index takes 3,344 bytes per product, against 3,072 for the exact index. **What to do:** check a tuned index on held-out queries by its recall against exact search, and read the queries that lose their whole top k before the mean score. Treat a fall in nDCG@10 that rests on a handful of queries as unproven until a paired test establishes it. Budget memory for the links as well as the vectors. ## Check queries that contain digits {#embeddings-match-digits} [An index](#hnsw-on-test-queries) can only return what the embeddings put near a query, so the next question is which queries the embeddings themselves get wrong. An encoder squeezes a whole query into one vector, so a short token that changes the meaning, such as a size, a model number or a "not", may move that vector very little. A study of 13 embedding models measured this for numbers, on synthetic financial text: their accuracy at telling texts apart by a number "is merely 0.54, slightly above random guessing (0.5)", and four-digit numbers were at the level of guessing ([Deng et al., 2026](https://aclanthology.org/2026.findings-eacl.289/)). Products aren't financial reports, so check the store's own queries. In the store, few queries hold a digit. Compare BM25 and semantic retrieval on each of them, from both splits. This is an exploratory diagnosis: nothing is tuned on it, and a handful of queries can't establish anything: Example `digits-in-queries` (Python 3.12.15 · bm25s 0.3.13 · numpy 2.5.4 · pandas 3.0.6 · scipy 1.18.1), passing in Edition 2026.10.1: ```python import pandas as pd from lib.lexical import LexicalSearch from lib.metrics import evaluate from lib.semantic import SemanticSearch from lib.store import load_products, load_queries, product_text, qrels products, queries = load_products(), load_queries() with_digit = queries[queries["query"].str.contains(r"\d")] # both splits: a diagnosis, nothing is tuned judgments = {query_id: grades for query_id, grades in qrels().items() if query_id in with_digit.index} scores = pd.DataFrame({ "split": with_digit["split"], "bm25": evaluate(LexicalSearch(products).run(with_digit, k=100), judgments)["ndcg@10"], "semantic": evaluate(SemanticSearch().run(with_digit, k=100), judgments)["ndcg@10"], "query": with_digit["query"], }).round(3) print("queries with a digit:", len(scores)) print(scores.to_string()) print("semantic better:", (scores["semantic"] > scores["bm25"]).sum()) print("bm25 better:", (scores["bm25"] > scores["semantic"]).sum()) print("gaps under 0.03:", ((scores["semantic"] - scores["bm25"]).abs() < 0.03).sum()) text = product_text(products).str.lower() print("products whose text has the word 7qt:", text.str.contains(r"\b7qt\b").sum()) print("products whose text has 7 qt:", text.str.contains(r"(?2} {label:<10} {width[product_id]:>5g} {products.at[product_id, 'name'][:50]}") print() for name, top in [("semantic", semantic), ("bm25", bm25)]: print(f"{name}, width 48 in the top 10:", (width[top] == 48).sum()) print(f"{name}, exact in the top 10:", sum(judgments.get(product_id) == 2 for product_id in top)) ``` Output: ```text query: 48 inch kitchen hood judgment width name 1 exact 48 48 '' 1200 cfm ducted wall mount range hood in bru 2 exact 48 48 '' 400 cfm ducted wall mount range hood in brus 3 exact 48 48 '' 400 cfm convertible wall mount range hood 4 partial 35.75 36 '' ws-48 series 500 cfm ducted under cabinet ra 5 exact 48 48 '' hand crafted designer wood island hood serie 6 exact 48 48 '' island range hood 7 exact 48 48 inch wall mount chimney range hood with 600 cfm 8 exact 48 48 '' ducted island range hood 9 exact 48 48 '' professional 900cfm ducted island range hood 10 partial 29.8 30 '' 480 cfm convertible under cabinet range hood semantic, width 48 in the top 10: 8 semantic, exact in the top 10: 8 bm25, width 48 in the top 10: 10 bm25, exact in the top 10: 10 ``` In semantic retrieval's top 10, 8 hoods have a width of 48. The others are a 36-inch hood from a series named `ws-48` and a 30-inch hood rated at 480 cfm, cubic feet per minute, the airflow of its fan. Both contain the digits 48, neither is 48 wide, and the judges marked both [partial](#store-data). Of BM25's top 10, 10 have a width of 48. For this query, the embedding matched the digits as text, not the width they stand for. The example reads each hood's width from the `attributes` column, curated fields from WANDS's product features such as width, shape and material. `product_text`, the text the model embedded, leaves that column out, so the embeddings never saw it, and a size the shopper types can be checked against it exactly. One query can't show a pattern. To see how the model handles sizes in general, use **probe queries**: pairs of queries written for the book that differ only in a size, a code or a negation, which `recordings/probes.py` embeds with the same model. One of them repeats the test query `48 inch kitchen hood` word for word. Probes have no judgments, so the example counts what their top 10s hold and never scores them. The right size comes from the width in `attributes` for the hoods, and from the name for the rest: Example `probe-pair-overlap` (Python 3.12.15 · numpy 2.5.4 · pandas 3.0.6), passing in Edition 2026.10.1: ```python from lib.recorded import probes from lib.semantic import SemanticSearch from lib.store import load_products # Probe queries written for the book, embedded by the same model. They have no judgments, # so this example counts what each top 10 holds; it doesn't score anything. probe_queries, probe_vectors = probes() products = load_products() names = products["name"] width = products["attributes"].str.extract(r"\bwidth: ([\d.]+)", expand=False).astype(float) semantic = SemanticSearch() def states(size, unit): """Names that state this size, such as "7 qt" or "7-quart", but not "3.7 qt".""" return names.str.contains(rf"(? 0 else 'no'}") print(f"{dims} vs 768, effect size: {result['effect size']:.3f}") print() ``` Output: ```text 256 vs 768, mean difference in ndcg@10: -0.010 256 vs 768, queries worse: 24 256 vs 768, permutation test p: 0.076 256 vs 768, paired t-test p: 0.080 256 vs 768, 95% bootstrap interval: -0.0211 to -0.0002 256 vs 768, interval excludes 0: yes 256 vs 768, effect size: -0.253 128 vs 768, mean difference in ndcg@10: -0.092 128 vs 768, queries worse: 33 128 vs 768, permutation test p: below 0.001 128 vs 768, paired t-test p: below 0.001 128 vs 768, 95% bootstrap interval: -0.1569 to -0.0565 128 vs 768, interval excludes 0: yes 128 vs 768, effect size: -0.547 ``` For 128 dimensions, the mean difference is -0.092, 33 train queries get worse, the permutation test gives p below 0.001, and the 95% interval, from -0.1569 to -0.0565, excludes 0: a clear loss. For 256, the mean difference is -0.010, the permutation test gives p = 0.076 and the t-test p = 0.080, yet the interval, from -0.0211 to -0.0002, excludes 0 by a hair. The tests and the interval disagree, so the evidence of a loss at 256 is borderline. The [effect size](#report-the-size-of-a-gap), -0.253 against -0.547 at 128, says any loss is the smaller of the two. For the store, the decision follows from its size, not from the test. Its full vectors take [36.9 MB](#exact-nearest-neighbours-numpy), so there is little to save, and it keeps all 768 dimensions; its test score doesn't change. A catalog of hundreds of millions of products would weigh a saving of two-thirds against a loss that this many train queries can't pin down, and would judge more queries before it decided. **What to do:** shortlist models on licence, language, size and domain, and treat benchmark ranks as a hint. Decide between models, or between vector sizes, on your own train queries with a paired test, and read the interval as well as the p-value. Shorten vectors only for a model trained for it, normalize after shortening, and only when memory or search time requires it. ## What goes wrong in production {#semantic-retrieval-in-production} **A similarity is not a confidence.** It's tempting to show nothing when the best similarity is low. But similarities aren't on one scale across queries: Example `similarity-scale` (Python 3.12.15 · numpy 2.5.4 · pandas 3.0.6), passing in Edition 2026.10.1: ```python import numpy as np from lib.recorded import embeddings from lib.semantic import SemanticSearch from lib.store import load_queries, qrels product_ids, product_vectors, query_ids, query_vectors = embeddings() test = load_queries("test") rows = np.isin(query_ids, test.index) similarity = np.sort(query_vectors[rows] @ product_vectors.T, axis=1)[:, ::-1] # best first, per query queries = test.loc[query_ids[rows], "query"].to_numpy() weakest = similarity[:, 0].argmin() # the query whose best match is least similar strongest = similarity[:, 9].argmax() # the query whose 10th match is most similar print("first query, lowest best similarity:", queries[weakest]) print(f"its best similarity: {similarity[weakest, 0]:.3f}") judgments = qrels()[query_ids[rows][weakest]] top = SemanticSearch().search(query_ids[rows][weakest], k=10).index print("exact products in its top 10:", sum(judgments.get(product_id) == 2 for product_id in top)) print("second query, highest 10th similarity:", queries[strongest]) print(f"its 10th similarity: {similarity[strongest, 9]:.3f}") cutoff = 0.5 print() print(f"products above {cutoff} for the first query:", (similarity[weakest] > cutoff).sum()) print(f"products above {cutoff} for the second query:", (similarity[strongest] > cutoff).sum()) print(f"test queries with no product above {cutoff}:", (similarity[:, 0] <= cutoff).sum()) ``` Output: ```text first query, lowest best similarity: liberty hardware francisco its best similarity: 0.436 exact products in its top 10: 5 second query, highest 10th similarity: kohler whitehaven farmhouse kitchen sink its 10th similarity: 0.647 products above 0.5 for the first query: 0 products above 0.5 for the second query: 103 test queries with no product above 0.5: 2 ``` On the test queries, `liberty hardware francisco` has the weakest best match, at 0.436, yet its top 10 holds 5 exact products. `kohler whitehaven farmhouse kitchen sink` has the most similar 10th match, at 0.647. A cutoff of 0.5 would keep 0 products for the first query and 103 for the second, and nothing for 2 test queries. Nor does semantic search ever come back empty: a [code that no product holds](#codes-carry-no-weight) still gets a full top 10. Decide that a query has no match some other way, for example when lexical retrieval finds none of its rare terms. **Mismatched vectors fail silently.** Query and product vectors must come from the same model, revision and prompts. Vectors from two models are numbers in different spaces, and when the two have the same dimension, comparing them raises no error. Sentence Transformers adds a model's query prompt when you call `encode_query`, which uses "a predefined “query” prompt, if available" ([API reference](https://sbert.net/docs/package_reference/sentence_transformer/SentenceTransformer.html)). Make sure the code that embeds live queries does what the indexing code did. **An index doesn't follow the catalog by itself.** faiss's HNSW "does not support removing vectors from the index" ([faiss wiki](https://github.com/facebookresearch/faiss/wiki/Guidelines-to-choose-an-index)), so a discontinued product stays in the graph until you filter it out of results or rebuild. "Faiss is not a database": it has no concurrent writes, sharding or transactions ([Douze et al., 2024](https://arxiv.org/abs/2401.08281)). Recall against exact search belongs to one catalog and one set of settings, so measure it again after every rebuild. **Long texts are cut short.** The model reads at most 512 tokens, and Sentence Transformers says "Longer inputs will be truncated" ([API reference](https://sbert.net/docs/package_reference/sentence_transformer/SentenceTransformer.html)). The store's longest product is 201 tokens, but in a catalog with long descriptions the model never reads what falls past the limit. Count tokens before you embed, as `recordings/embeddings.py` does. **Typos move vectors.** For BERT-based retrievers not trained on misspellings, "typos can lead to a significant drop in retrieval and ranking effectiveness" ([Zhuang and Zuccon, 2021](https://arxiv.org/abs/2108.12139)). 4 of the store's queries contain a typo, and the probes include each one spelled correctly. A corrected probe asks for the same thing as its query, so the example scores it with that query's judgments: Example `spelling-probes` (Python 3.12.15 · bm25s 0.3.13 · numpy 2.5.4 · pandas 3.0.6 · scipy 1.18.1), passing in Edition 2026.10.1: ```python from lib.lexical import LexicalSearch from lib.metrics import ndcg_at_k from lib.recorded import probes from lib.semantic import SemanticSearch from lib.store import load_products, load_queries, qrels # Each spelling probe is the correctly spelled text of one store query (of_query_id), embedded by the same # model. It asks for the same thing, so it is scored with that query's judgments. A diagnosis, not a test. probe_queries, probe_vectors = probes() products, queries, judgments = load_products(), load_queries(), qrels() semantic, bm25 = SemanticSearch(), LexicalSearch(products) spelling = probe_queries[probe_queries["kind"] == "spelling"] for row, probe in spelling.iterrows(): query_id, pair = int(probe["of_query_id"]), probe["pair"] typed, corrected = semantic.search(query_id, k=10).index, semantic.search_vector(probe_vectors[row], k=10).index print(f"{pair}: {queries.at[query_id, 'query']} | {probe['query']} ({queries.at[query_id, 'split']} query)") print(f"{pair}, similarity of the two spellings: {semantic.query_vectors[query_id] @ probe_vectors[row]:.3f}") print(f"{pair}, products in both semantic top 10s:", len(set(typed) & set(corrected))) print(f"{pair}, semantic ndcg@10 as typed: {ndcg_at_k(list(typed), judgments[query_id], 10):.3f}") print(f"{pair}, semantic ndcg@10 corrected: {ndcg_at_k(list(corrected), judgments[query_id], 10):.3f}") for name, text in [("as typed", queries.at[query_id, "query"]), ("corrected", probe["query"])]: print(f"{pair}, bm25 ndcg@10 {name}: {ndcg_at_k(list(bm25.search(text, k=10).index), judgments[query_id], 10):.3f}") print() print("spelling probes:", len(spelling)) print(f"smallest two-sided permutation p-value possible with {len(spelling)} queries: {2 / 2 ** len(spelling):.3f}") ``` Output: ```text ligth: ligth bulb | light bulb (train query) ligth, similarity of the two spellings: 0.710 ligth, products in both semantic top 10s: 5 ligth, semantic ndcg@10 as typed: 1.000 ligth, semantic ndcg@10 corrected: 1.000 ligth, bm25 ndcg@10 as typed: 0.945 ligth, bm25 ndcg@10 corrected: 0.945 foutains: foutains with brick look | fountains with brick look (train query) foutains, similarity of the two spellings: 0.668 foutains, products in both semantic top 10s: 0 foutains, semantic ndcg@10 as typed: 0.353 foutains, semantic ndcg@10 corrected: 0.645 foutains, bm25 ndcg@10 as typed: 0.356 foutains, bm25 ndcg@10 corrected: 0.507 desins: twin over full bunk beds cool desins | twin over full bunk beds cool designs (train query) desins, similarity of the two spellings: 0.829 desins, products in both semantic top 10s: 6 desins, semantic ndcg@10 as typed: 0.565 desins, semantic ndcg@10 corrected: 0.616 desins, bm25 ndcg@10 as typed: 0.488 desins, bm25 ndcg@10 corrected: 0.488 ice-an: refrigerator with ice an water in door | refrigerator with ice and water in door (test query) ice-an, similarity of the two spellings: 0.975 ice-an, products in both semantic top 10s: 9 ice-an, semantic ndcg@10 as typed: 0.807 ice-an, semantic ndcg@10 corrected: 0.829 ice-an, bm25 ndcg@10 as typed: 0.501 ice-an, bm25 ndcg@10 corrected: 0.501 spelling probes: 4 smallest two-sided permutation p-value possible with 4 queries: 0.125 ``` For the train query `foutains with brick look`, the corrected spelling changes the whole semantic top 10, with 0 products shared, and nDCG@10 goes from 0.353 to 0.645; BM25 goes from 0.356 to 0.507. For the others, the scores move less. The one test query among them, `refrigerator with ice an water in door`, goes from 0.807 to 0.829, an exploratory look at one query. These queries can't establish a pattern: with 4 of them, the smallest two-sided permutation p-value possible is 0.125. **What to do:** don't filter on a fixed similarity. Record the model, revision and prompts with every vector, and check them where queries are embedded. Rebuild and re-measure the index as the catalog changes, count tokens before you embed, and correct spelling before you embed a query when your log shows typos, with a corrector that checks each word's neighbours ([chapter 8](#correct-spelling-from-catalog)). ## Decisions this chapter made {#semantic-retrieval-decisions} Each choice below is a default, measured on the train queries where it could be. - **Model:** `Snowflake/snowflake-arctic-embed-m-v1.5`, which `recordings/embeddings.py` records once at a pinned revision. It is open, under Apache-2.0, small, and trained for shorter vectors. Queries get the prefix its card asks for; products don't. - **Similarity:** the dot product of unit-length vectors, which is cosine similarity. Vectors are normalized once, and again after any change. - **Search:** exact, with NumPy, through `lib.vectors.exact_search`, because the store's vectors take 36.9 MB. It is also the ground truth for any index. - **If the catalog outgrows exact search:** faiss HNSW with `M = 32` and an `efSearch` of 144, the smallest setting in the train sweep at which every query keeps at least 9 of its true top 10. On the test queries, its recall@10 against exact search is 1.000. - **An index's recall is measured against exact search,** on train queries, at the k the pipeline uses, reading the worst queries as well as the mean. - **Dimensions:** all 768. Shortening to 256 would save two-thirds of the memory at a cost the train queries can't pin down, and the store has little to save. - **Probe queries test what the judgments can't,** such as negation, codes and spelling. They are compared by their top 10s and kept apart from the judged queries; only a corrected spelling, which asks what its query asks, is scored with that query's judgments. - **Known weak spots go to other stages:** codes to lexical retrieval; exact sizes to attribute fields, or to lexical retrieval once the tokenizer keeps digits ([chapter 5](#split-text-into-terms), [chapter 8](#normalize-units)); for both, a filter to test rather than a blend ([chapter 7](#when-hybrid-helps)); negation to attribute filters and reranking. # Hybrid retrieval By the end of this chapter you can combine a lexical and a semantic ranking into one, either by adding normalized scores or by reciprocal rank fusion, tune how much each ranking counts on train queries, and decide with a paired test on test queries whether the combination beats the better of its two inputs. The store, the online furniture and home-goods retailer this book follows as its case study ([chapter 1](#store-data)), has 50 train queries for tuning, 30 test queries for reporting, and two retrievers. [BM25](#rank-with-bm25) matches the query's words; it is the book's baseline, `lib.lexical` at bm25s's defaults, without stemming. [Semantic retrieval](#search-by-meaning) compares the embeddings of the query and of each product, which the open model `Snowflake/snowflake-arctic-embed-m-v1.5` computed in advance. Industry search systems commonly pair the two. One semantic product-search model was built "to complement a lexical search engine in ad hoc product retrieval" ([Nigam et al., 2019](https://arxiv.org/abs/1907.00937)), and a social network's search engine scores documents with "both embedding-based retrieval and term matching based retrieval" ([Huang et al., 2020](https://arxiv.org/abs/2006.11632)). When chapter 1 [fused the two rankings into one](#fuse-with-rrf), the result didn't beat semantic retrieval alone: it scored 0.812 [nDCG@10](#score-one-ranking) on the test queries against 0.830, a gap that [chapter 3's tests](#compare-runs-in-a-harness) don't establish either way. So the store needs to know whether a better mix of the two can beat semantic retrieval, its best ranker so far and so the one to beat. This chapter finds the queries each retriever wins and checks each retriever alone. It then builds two **blends**, ways to fuse the rankings with a weight that sets how much each one counts, and tunes that weight. Every analysis and every tuning step runs on the train queries. Comparisons there guide the tuning; they aren't evidence. The chapter plans two comparisons on the test queries, made once, at the end: each tuned blend, the [weighted sum](#normalize-scores-before-adding) and [weighted reciprocal rank fusion](#tune-blend-on-train), against semantic retrieval. Following [chapter 3](#compare-runs-in-a-harness), a difference counts as established only when its paired permutation test gives a p-value below 0.05 divided by the number of planned comparisons, here 0.025. ## Find the queries each retriever wins {#queries-each-retriever-wins} **Hybrid retrieval** combines the rankings of a lexical and a semantic retriever into one ranking. Combining rankings is called **fusion**. Fusion can only help if the two retrievers fail on different queries: if both put poor products first for the same query, no mix of their rankings fixes it. Published studies find that they do. Embeddings miss rare entities and selective keywords that BM25 matches ([chapter 6](#codes-carry-no-weight)), and on a dataset built to test the limits of one embedding per text, BM25 finds what embeddings miss but loses most of its score when words are replaced with synonyms ([chapter 1](#search-by-meaning)). So the first check is per query, not on average. Score both of the store's retrievers by [nDCG@10](#score-one-ranking) on every [train query](#store-data), keeping each one's top 100 products. Example `bm25-semantic-wins-by-query` (Python 3.12.15 · bm25s 0.3.13 · numpy 2.5.4 · pandas 3.0.6 · scipy 1.18.1), passing in Edition 2026.10.1: ```python import numpy as np from lib.lexical import LexicalSearch from lib.metrics import evaluate from lib.semantic import SemanticSearch from lib.store import load_products, load_queries, qrels products, queries, judgments = load_products(), load_queries("train"), qrels("train") bm25 = LexicalSearch(products).run(queries, k=100) semantic = SemanticSearch().run(queries, k=100) bm25_scores, semantic_scores = evaluate(bm25, judgments)["ndcg@10"], evaluate(semantic, judgments)["ndcg@10"] print("train queries:", len(queries)) print(f"bm25 ndcg@10: {bm25_scores.mean():.3f}") print(f"semantic ndcg@10: {semantic_scores.mean():.3f}") lead = (bm25_scores - semantic_scores).round(3) # per train query; above 0 means BM25 wins print("bm25 better:", (lead > 0).sum()) print("semantic better:", (lead < 0).sum()) print("tied:", (lead == 0).sum()) print(f"better of the two, chosen per query: {np.maximum(bm25_scores, semantic_scores).mean():.3f}") has_digit = queries["query"].str.contains(r"\d") print("queries with a digit:", has_digit.sum()) print("queries with a digit, bm25 better:", (lead[has_digit] > 0).sum()) print() print("queries where bm25 leads by 0.1 or more:", (lead >= 0.1).sum()) print(queries.loc[lead[lead >= 0.1].index, "query"].to_string()) ``` Output: ```text train queries: 50 bm25 ndcg@10: 0.695 semantic ndcg@10: 0.772 bm25 better: 15 semantic better: 31 tied: 4 better of the two, chosen per query: 0.802 queries with a digit: 5 queries with a digit, bm25 better: 1 queries where bm25 leads by 0.1 or more: 6 query_id 44 chrome shower head and valve 146 laundry vanity cabinet 277 barstool patio sets 313 malachi sled 359 french molding 432 butcher block island ``` On average, semantic retrieval scores 0.772 and BM25 0.695. Per query, semantic retrieval is better on 31 queries, BM25 on 15, and 4 are tied. If you could choose the better of the two rankings for each query, the mean would be 0.802. That isn't a ceiling for fusion, which mixes the lists rather than choosing one, but it measures what the two rankings' differences could be worth. Next, see whether the two lists hold different products, and what share of each query's **exact** products, the ones the judges graded as fully matching it, each list finds: Example `overlap-and-union-recall` (Python 3.12.15 · bm25s 0.3.13 · numpy 2.5.4 · pandas 3.0.6 · scipy 1.18.1), passing in Edition 2026.10.1: ```python import numpy as np from lib.lexical import LexicalSearch from lib.metrics import recall_at_k from lib.semantic import SemanticSearch from lib.store import load_products, load_queries, qrels products, queries, judgments = load_products(), load_queries("train"), qrels("train") bm25 = LexicalSearch(products).run(queries, k=100) semantic = SemanticSearch().run(queries, k=100) shared = [len(set(bm25[query_id][:10]) & set(semantic[query_id][:10])) for query_id in queries.index] print(f"products in both top 10s, mean: {np.mean(shared):.1f}") def share_retrieved(run): """The share of each query's exact products that the run's list holds, averaged over queries.""" shares = [recall_at_k(ranking, judgments[query_id], k=len(ranking)) for query_id, ranking in run.items()] return np.mean(shares) for depth in [100, 20]: lists = {"bm25": {query_id: ranking[:depth] for query_id, ranking in bm25.items()}, "semantic": {query_id: ranking[:depth] for query_id, ranking in semantic.items()}} lists["union"] = {query_id: list(dict.fromkeys(lists["bm25"][query_id] + lists["semantic"][query_id])) for query_id in queries.index} for name, run in lists.items(): print(f"{name} top {depth}, share of exact products retrieved: {share_retrieved(run):.3f}") ``` Output: ```text products in both top 10s, mean: 3.7 bm25 top 100, share of exact products retrieved: 0.672 semantic top 100, share of exact products retrieved: 0.759 union top 100, share of exact products retrieved: 0.810 bm25 top 20, share of exact products retrieved: 0.297 semantic top 20, share of exact products retrieved: 0.353 union top 20, share of exact products retrieved: 0.445 ``` The two top 10s share 3.7 products on average. BM25's top 100 holds 0.672 of each query's exact products, semantic retrieval's 0.759, and the union of the two lists 0.810. Fusion can't rank a product that neither list holds, so how deep each list goes caps how many exact products fusion can reach: cut each list to its top 20, and the union holds 0.445. Two train queries show the kinds of failure: Example `two-queries-two-winners` (Python 3.12.15 · bm25s 0.3.13 · numpy 2.5.4 · pandas 3.0.6 · scipy 1.18.1), passing in Edition 2026.10.1: ```python from lib.lexical import LexicalSearch from lib.semantic import SemanticSearch from lib.store import load_products, load_queries, product_text, qrels LABELS = {2: "exact", 1: "partial", 0: "irrelevant"} products, queries, judgments = load_products(), load_queries("train"), qrels("train") bm25 = LexicalSearch(products).run(queries, k=10) semantic = SemanticSearch().run(queries, k=10) text = product_text(products) print("products containing 7qt:", text.str.contains(r"\b7qt\b", case=False).sum()) print("products containing 7 qt:", text.str.contains(r"\b7 qt\b", case=False).sum()) for query_id in [432, 62]: # butcher block island, 7qt slow cooker query = queries.at[query_id, "query"] for name, run in [("bm25", bm25), ("semantic", semantic)]: grades = [judgments[query_id].get(product_id) for product_id in run[query_id]] print() print(f"exact in {name} top 10 for {query}: {grades.count(2)}") for product_id, grade in list(zip(run[query_id], grades))[:5]: print(f" {LABELS.get(grade, 'unjudged'):<10} {products.at[product_id, 'name'][:55]}") ``` Output: ```text products containing 7qt: 0 products containing 7 qt: 9 exact in bm25 top 10 for butcher block island: 9 exact carol kitchen island with butcher block top exact brunton kitchen island with butcher block top partial brunton kitchen cart butcher block top exact janigian kitchen island with butcher block exact avelar kitchen island with butcher block exact in semantic top 10 for butcher block island: 3 partial boosblock® prep table with butcher block top partial boosblock® butcher block partial prep table partial pro chef kitchen cart with butcher block top exact brunton kitchen island with butcher block top exact in bm25 top 10 for 7qt slow cooker: 1 partial cuisinart 6.5 quart programmable slow cooker partial midea 6.5 qt . slow cooker partial delonghi cks1660d livenza programmable 6 qt . slow cook partial nesco 1.5 qt . analog slow cooker partial 360 cookware gourmet slow cooker exact in semantic top 10 for 7qt slow cooker: 7 exact 7-qt . stay or go slow cooker exact crock-pot 7 qt . slow cooker exact zing 7 qt . digital slow cooker exact black + decker 7 qt . digital slow cooker exact crock-pot 7 qt . slow cooker ``` For `butcher block island`, BM25's top 10 holds 9 exact products and semantic retrieval's 3. BM25 rewards products whose text contains all three words: kitchen islands with butcher-block tops. Semantic retrieval ranks prep tables and carts with butcher-block tops near the top too, and the judges graded those partial: the right material, but not the island the query asks for. For `7qt slow cooker` it's the other way round: 1 against 7. The catalog has 0 products whose text contains `7qt` and 9 that contain `7 qt`. So BM25 matches only `slow cooker` and ranks cookers of every size, while semantic retrieval puts 7-quart cookers first. Those are single queries: they show a kind of failure, not how often it happens. [The per-query count](#bm25-semantic-wins-by-query) also lists the 6 train queries where BM25 leads by 0.1 nDCG@10 or more. Most name a specific kind of product in two or more words that all have to match, and one, `malachi sled`, names a product line. [Chapter 1](#search-by-meaning) found BM25 ahead on 2 of the 2 test queries that give a size in inches, yet of the 5 train queries that contain a digit, BM25 wins 1. **What to do:** before you fuse, count per query where each retriever wins on your train queries, and check how many relevant products the union of their lists holds that the better one misses. Consider fusing when each retriever wins a real share of queries and the union holds more relevant products than either list. That is necessary, not sufficient: next, [check each retriever alone](#check-each-retriever-before-fusing), and [test the tuned blend on held-out queries](#report-blends-on-test) before you adopt it. ## Check each retriever before you fuse {#check-each-retriever-before-fusing} Knowing [where each retriever wins](#queries-each-retriever-wins) isn't enough, because fusion gives each input a say in the final ranking, a weak input included. A study of hybrid search over 11 datasets reports "A “weakest link” phenomenon, where a weak path can substantially degrade overall accuracy, highlighting the need for path-wise quality assessment before fusion" ([Wang et al., PVLDB 2026](https://arxiv.org/abs/2508.01405)). So compare the retrievers' mean nDCG@10, with [judged@10](#unjudged-results), the share of the top 10 that has a judgment, beside it. Then compare them query by query with a **paired test**, which asks how likely a mean difference this large would be if the two were equally good. The book's test is the [permutation test](#permutation-test-on-queries), with the [paired t-test](#paired-t-test-on-queries) beside it. On train queries, these checks shape how you combine the retrievers; they aren't evidence about the store. In the store, fuse BM25 and semantic retrieval as [chapter 1](#fuse-with-rrf) did, with reciprocal rank fusion (RRF): each product scores 1 / (60 + its rank) in each ranking, summed over the two, so each ranking gets an equal vote. Then split the train queries by which retriever wins, and compare the fused ranking with semantic retrieval alone: Example `fusion-gain-by-winning-retriever` (Python 3.12.15 · bm25s 0.3.13 · numpy 2.5.4 · pandas 3.0.6 · scipy 1.18.1), passing in Edition 2026.10.1: ```python from lib.fusion import reciprocal_rank_fusion from lib.lexical import LexicalSearch from lib.metrics import evaluate from lib.semantic import SemanticSearch from lib.significance import compare from lib.store import load_products, load_queries, qrels products, queries, judgments = load_products(), load_queries("train"), qrels("train") bm25 = LexicalSearch(products).run(queries, k=100) semantic = SemanticSearch().run(queries, k=100) hybrid = reciprocal_rank_fusion([bm25, semantic]) # k = 60 and equal votes, as in chapter 1 bm25_scores, semantic_scores, hybrid_scores = (evaluate(run, judgments) for run in (bm25, semantic, hybrid)) print(f"bm25 judged@10: {bm25_scores['judged@10'].mean():.3f}") print(f"semantic judged@10: {semantic_scores['judged@10'].mean():.3f}") for name, after, before in [("semantic minus bm25", semantic_scores, bm25_scores), ("hybrid minus semantic", hybrid_scores, semantic_scores)]: result = compare(after["ndcg@10"], before["ndcg@10"]) # paired by query print(f"{name}, mean ndcg@10: {result['mean difference']:.3f}") print(f"{name}, permutation p: {result['permutation p']:.3f}") print(f"{name}, t-test p: {result['t-test p']:.3f}") print() change = (hybrid_scores["ndcg@10"] - semantic_scores["ndcg@10"]).round(3) # fusion against semantic alone lead = (bm25_scores["ndcg@10"] - semantic_scores["ndcg@10"]).round(3) # above 0 means BM25 wins for name, group in [("bm25 wins", lead > 0), ("semantic wins", lead < 0)]: print(f"{name}, queries: {group.sum()}") print(f"{name}, fusion better than semantic: {(change[group] > 0).sum()}") print(f"{name}, fusion worse than semantic: {(change[group] < 0).sum()}") print(f"{name}, mean change in ndcg@10: {change[group].mean():.3f}") ``` Output: ```text bm25 judged@10: 0.930 semantic judged@10: 0.964 semantic minus bm25, mean ndcg@10: 0.077 semantic minus bm25, permutation p: 0.003 semantic minus bm25, t-test p: 0.004 hybrid minus semantic, mean ndcg@10: 0.002 hybrid minus semantic, permutation p: 0.863 hybrid minus semantic, t-test p: 0.872 bm25 wins, queries: 15 bm25 wins, fusion better than semantic: 12 bm25 wins, fusion worse than semantic: 2 bm25 wins, mean change in ndcg@10: 0.101 semantic wins, queries: 31 semantic wins, fusion better than semantic: 5 semantic wins, fusion worse than semantic: 25 semantic wins, mean change in ndcg@10: -0.045 ``` BM25's judged@10 is 0.930 against 0.964 for semantic retrieval, so BM25's top 10s hold a few more products nobody judged, and [chapter 2](#unjudged-results) shows how those can lower a score. Semantic retrieval leads by 0.077 nDCG@10, with a permutation p-value of 0.003 and a t-test p-value of 0.004. On the test queries, [chapter 3](#compare-runs-in-a-harness) establishes the same lead. So the store has a strong retriever and a weaker one. On the 15 train queries where BM25 wins, the fused ranking beats semantic retrieval on 12 and loses on 2, with a mean change of 0.101. On the 31 where semantic retrieval wins, the fused ranking is worse on 25 and better on 5, with a mean change of -0.045. The split is built to show this pattern: grouping queries by the winner means almost any mix of the two helps where BM25 wins and hurts where it loses. What matters is the net change over all the train queries. The fused ranking differs from semantic retrieval by 0.002 on average, with a permutation p-value of 0.863: on these queries, equal votes gain nothing measurable. The split uses the judgments to decide which retriever wins, so it is a diagnosis: you can't use it to choose a retriever for a live query, which has no judgments. **What to do:** check each retriever alone on train queries, with judged@10 and a paired test, before you fuse. When one is weaker and wins fewer queries, don't give it an equal vote: weight it and [tune the weight](#tune-blend-on-train), or leave it out. Equal votes suit two retrievers of about the same quality. ## Normalize scores before you add them {#normalize-scores-before-adding} The most direct way to give the weaker retriever, BM25 in the store, [a smaller say](#check-each-retriever-before-fusing) is to add the two retrievers' scores with a weight: α times the semantic score plus (1 − α) times the BM25 score. A sum whose weights add up to 1 is a **convex combination**, or weighted sum. It only works if the scores are on comparable scales, and here they aren't: a [BM25](#rank-with-bm25) score has no fixed range and grows with the number and rarity of the query's words, while semantic retrieval's score, a cosine similarity, lies between −1 and 1. **Score normalization** rescales each retriever's scores for each query before adding them. Per-query min-max scaling maps the lowest score among the products ranked for the query to 0 and the highest to 1. A study of fusion functions calls it "the de facto method in the literature" and tests two alternatives ([Bruch et al., 2023](https://arxiv.org/abs/2210.11934)). **Theoretical min-max** uses each score's lowest possible value instead of the lowest one seen: 0 for BM25 and −1 for cosine. The **z-score** subtracts the mean and divides by the standard deviation. The examples' shared module `lib/blend.py` writes each in a few lines. ```text min-max(s) = (s − lowest score among the query's candidates) / (highest − lowest) blend = α · min-max(similarity) + (1 − α) · min-max(BM25) ``` A sum needs both scores for every product it ranks. The **candidate set** for a query is the union of both retrievers' top 100, and `lib.blend.candidates` looks up both scores for every product in it: the BM25 score, which is 0 when the product shares no word with the query, and the cosine similarity. Start with the store's raw scores, added with α = 0.5 on the train queries. Then do the same after min-max scaling, written out in the example and checked against `lib.blend.weighted_sum`: Example `raw-scores-versus-min-max` (Python 3.12.15 · bm25s 0.3.13 · numpy 2.5.4 · pandas 3.0.6 · scipy 1.18.1), passing in Edition 2026.10.1: ```python from lib.blend import candidates, weighted_sum from lib.lexical import LexicalSearch from lib.metrics import evaluate from lib.semantic import SemanticSearch from lib.significance import compare from lib.store import load_products, load_queries, qrels products, queries, judgments = load_products(), load_queries("train"), qrels("train") lexical, semantic = LexicalSearch(products), SemanticSearch() tables = candidates(lexical, semantic, queries) # both scores for each query's candidates table = tables[36] # laundry basket with wheels print("candidates for laundry basket with wheels:", len(table)) print(f"bm25 range: {table['bm25'].min():.3f} to {table['bm25'].max():.3f}") print(f"similarity range: {table['similarity'].min():.3f} to {table['similarity'].max():.3f}") def min_max(scores): return (scores - scores.min()) / (scores.max() - scores.min()) def blend(normalize, alpha=0.5): run = {} for query_id, table in tables.items(): score = alpha * normalize(table["similarity"]) + (1 - alpha) * normalize(table["bm25"]) run[query_id] = list(score.sort_values(ascending=False, kind="stable").index[:100]) return run top_10s = {"bm25": lexical.run(queries, k=10), "semantic": semantic.run(queries, k=10)} per_query = {} for name, normalize in [("raw", lambda scores: scores), ("min-max", min_max)]: run = blend(normalize) per_query[name] = evaluate(run, judgments)["ndcg@10"] print(f"{name} sum ndcg@10: {per_query[name].mean():.3f}") for other, top in top_10s.items(): shared = [len(set(run[query_id][:10]) & set(top[query_id])) for query_id in queries.index] print(f"{name} sum, products shared with the {other} top 10: {sum(shared) / len(shared):.1f}") by_hand, library = blend(min_max), weighted_sum(tables, 0.5) matches = sum(by_hand[query_id] == library[query_id] for query_id in tables) print(f"train queries ranked as lib.blend.weighted_sum ranks them: {matches} of {len(tables)}") p_value = compare(per_query["min-max"], per_query["raw"])["permutation p"] # a tuning check on train print("min-max minus raw, permutation p:", "below 0.001" if p_value < 0.001 else f"{p_value:.3f}") ``` Output: ```text candidates for laundry basket with wheels: 125 bm25 range: 2.059 to 8.229 similarity range: 0.336 to 0.597 raw sum ndcg@10: 0.706 raw sum, products shared with the bm25 top 10: 9.6 raw sum, products shared with the semantic top 10: 3.9 min-max sum ndcg@10: 0.771 min-max sum, products shared with the bm25 top 10: 6.5 min-max sum, products shared with the semantic top 10: 6.2 train queries ranked as lib.blend.weighted_sum ranks them: 50 of 50 min-max minus raw, permutation p: below 0.001 ``` For `laundry basket with wheels`, the 125 candidates have BM25 scores from 2.059 to 8.229 and similarities from 0.336 to 0.597. BM25's spread is many times larger, so it decides the order of the raw sum. Over the train queries, the raw sum's top 10 shares 9.6 products with BM25's top 10 on average and 3.9 with semantic retrieval's. Its nDCG@10 of 0.706 is close to [BM25's 0.695](#queries-each-retriever-wins). The weight says "half each", but the ranking is mostly BM25's. After min-max scaling, the top 10 shares 6.5 products with BM25's and 6.2 with semantic retrieval's, and nDCG@10 is 0.771, with a permutation p-value against the raw sum below 0.001. Both retrievers now have a say. The by-hand version ranks 50 of 50 train queries exactly as `lib.blend.weighted_sum` does. Raw scores can work with a tuned weight. The authors of DPR, an embedding-based retriever, ranked candidates by BM25 plus 1.1 times DPR's similarity, with the 1.1 chosen on a development set ([Karpukhin et al., 2020](https://arxiv.org/abs/2004.04906)). Normalization gives α the same meaning for every query; without it, BM25's share of the say changes with its range from query to query. Min-max isn't an absolute scale, though. It is relative to the query's candidates, so the best candidate always gets 1, however poor it is. **What to do:** never add raw BM25 and cosine scores with a guessed weight. We recommend per-query min-max as the default, because it needs no knowledge of either scoring function. Use theoretical min-max when you know each score's lowest possible value and want a floor that doesn't move with the candidates. In either case, tune α on train queries. ## Test reciprocal rank fusion's settings on train queries {#rrf-settings-on-train} Ranks, unlike [scores](#normalize-scores-before-adding), share one scale, so fusing by rank needs no normalization. **Reciprocal rank fusion** (RRF) scores each product by adding, over the rankings that contain it, 1 divided by the sum of a constant k and its rank, as [chapter 1's formula](#fuse-with-rrf) writes it ([Cormack et al., 2009](http://cormack.uwaterloo.ca/cormack/cormacksigir09-rrf.pdf)). The price is information: "because RRF is a function of ranks, it disregards the distribution of scores and, as such, discards useful information" ([Bruch et al., 2023](https://arxiv.org/abs/2210.11934)). A product a hair behind the leader and one far behind it get scores set by their ranks alone. The constant k sets how much the top ranks dominate. RRF's authors fixed k = 60 in a pilot study, and their tests "indicated that k = 60 was near-optimal, but that the choice was not critical". Those pilots fused "30 configurations of Wumpus Search", one search engine, on collections from TREC, a series of shared retrieval evaluations (Cormack et al., 2009). Bruch et al. fused one lexical and one semantic retriever, gave each its own constant, η, and found the results sensitive to the balance between the two constants, which acts as a weight: "where the semantic model already performs strongly, for example, discounting the lexical system by increasing η_Lex leads to better performance". One k shared by both rankings, as here, sets no balance between them. The depth of each list matters too. A product in one retriever's top 20 but not in the other's earns nothing from the second ranking, although it has a rank there, only a low one. As Bruch et al. put it, "ranks cannot be computed exactly unless the entire collection D is ranked by retrieval system o for every query." In the store, the union of two top-20 lists holds far fewer exact products than the union of two top-100 lists, as [the count of exact products in each list](#queries-each-retriever-wins) shows. Try six values of k on the store's train queries, fusing BM25 and semantic retrieval from lists 100 deep. One check is planned before running: does the best k on train beat the default, k = 60? Example `rrf-k-on-train` (Python 3.12.15 · bm25s 0.3.13 · numpy 2.5.4 · pandas 3.0.6 · scipy 1.18.1), passing in Edition 2026.10.1: ```python from lib.fusion import reciprocal_rank_fusion from lib.lexical import LexicalSearch from lib.metrics import evaluate from lib.semantic import SemanticSearch from lib.significance import compare from lib.store import load_products, load_queries, qrels products, queries, judgments = load_products(), load_queries("train"), qrels("train") bm25 = LexicalSearch(products).run(queries, k=100) semantic = SemanticSearch().run(queries, k=100) for k in [1, 60]: print(f"rank 1 over rank 10 at k={k}: {(1 / (k + 1)) / (1 / (k + 10)):.3f}") scores = {k: evaluate(reciprocal_rank_fusion([bm25, semantic], k=k), judgments)["ndcg@10"] for k in [1, 10, 30, 60, 100, 300]} for k, per_query in scores.items(): print(f"k={k} ndcg@10: {per_query.mean():.3f}") best = max(scores, key=lambda k: scores[k].mean()) # ties go to the smaller k print("best k on train:", best) result = compare(scores[best], scores[60]) # the planned check: best k against the default print(f"best k minus k=60: {result['mean difference']:.3f}") print(f"best k minus k=60, permutation p: {result['permutation p']:.3f}") print(f"best k minus k=60, t-test p: {result['t-test p']:.3f}") print(f"best k minus k=60, 95% interval: {result['ci low']:.3f} to {result['ci high']:.3f}") print(f"best k minus k=60, wins, losses, ties: {result['wins']}, {result['losses']}, {result['ties']}") ``` Output: ```text rank 1 over rank 10 at k=1: 5.500 rank 1 over rank 10 at k=60: 1.148 k=1 ndcg@10: 0.753 k=10 ndcg@10: 0.770 k=30 ndcg@10: 0.771 k=60 ndcg@10: 0.774 k=100 ndcg@10: 0.778 k=300 ndcg@10: 0.777 best k on train: 100 best k minus k=60: 0.004 best k minus k=60, permutation p: 0.093 best k minus k=60, t-test p: 0.103 best k minus k=60, 95% interval: 0.001 to 0.011 best k minus k=60, wins, losses, ties: 6, 3, 41 ``` At k = 1, rank 1 earns 5.500 times what rank 10 earns. At k = 60, it earns 1.148 times as much, so a product near the top of both lists beats one at the top of only one. From k = 1 to k = 300, the six settings score 0.753, 0.770, 0.771, 0.774, 0.778 and 0.777. The best, k = 100, leads k = 60 by 0.004, with a permutation p-value of 0.093 and a t-test p-value of 0.103. Its 95% bootstrap interval, 0.001 to 0.011, excludes 0, so it disagrees with both tests. The two settings' rankings differ on only a handful of queries (wins, losses and ties: 6, 3, 41), and the book takes its verdict from the permutation test, which can't tell the gain from chance at 0.05. Which k is best is fragile too: [with BM25's tied products listed the other way](#tuned-weight-stability), k = 300 wins instead. This fits Cormack et al.'s finding that the choice of k is not critical. **What to do:** keep k = 60 as the default, and fuse lists at least 100 deep, because a deeper list can only add candidates. Move off k = 60 only when the best k on train beats it by a margin the permutation test can tell from chance, with p below 0.05. When one retriever is [the stronger one](#check-each-retriever-before-fusing), as in the store, k isn't the setting to change: RRF's equal vote is. Give the rankings weights and [tune them](#tune-blend-on-train). ## Tune the blend's weight on train queries {#tune-blend-on-train} The chapter's two [blends](#hybrid-retrieval) each take one weight, α, the share given to semantic retrieval. In the [weighted sum](#normalize-scores-before-adding), α multiplies the normalized similarity and 1 − α the normalized BM25 score. In **weighted RRF**, a product scores (1 − α) / (k + its BM25 rank) plus α / (k + its semantic rank). For both, α = 0 is BM25 alone and α = 1 is semantic retrieval alone. For RRF, α = 0.5 is [chapter 1's unweighted fusion](#fuse-with-rrf) with every score halved. Here is the arithmetic for the top product of one train query, with an illustrative α = 0.75: Example `weighted-rrf-one-product` (Python 3.12.15 · bm25s 0.3.13 · numpy 2.5.4 · pandas 3.0.6 · scipy 1.18.1), passing in Edition 2026.10.1: ```python from lib.blend import weighted_rrf from lib.lexical import LexicalSearch from lib.semantic import SemanticSearch from lib.store import load_products, load_queries alpha, k = 0.75, 60 # an illustrative weight; tuning picks one on train products, queries = load_products(), load_queries("train") bm25 = LexicalSearch(products).run(queries, k=100) semantic = SemanticSearch().run(queries, k=100) query_id = 36 # laundry basket with wheels def part(ranking, weight, product_id): """weight / (k + rank) from one ranking, or 0 when the product isn't in it.""" return weight / (k + ranking.index(product_id) + 1) if product_id in ranking else 0.0 candidates = dict.fromkeys(bm25[query_id] + semantic[query_id]) # the union, in the order first seen score = {product_id: part(bm25[query_id], 1 - alpha, product_id) + part(semantic[query_id], alpha, product_id) for product_id in candidates} ranking = sorted(score, key=lambda product_id: (-score[product_id], product_id)) # ties by product_id first = ranking[0] print("first product:", products.at[first, "name"]) print("its bm25 rank:", bm25[query_id].index(first) + 1) print("its semantic rank:", semantic[query_id].index(first) + 1) print(f"its weighted rrf score: {score[first]:.5f}") library = weighted_rrf(bm25, semantic, alpha)[query_id] # the top 100 matches = sum(mine == theirs for mine, theirs in zip(ranking, library)) print(f"places that match lib.blend.weighted_rrf: {matches} of {len(library)}") ``` Output: ```text first product: laundry sorter basket on wheels its bm25 rank: 1 its semantic rank: 2 its weighted rrf score: 0.01620 places that match lib.blend.weighted_rrf: 100 of 100 ``` For `laundry basket with wheels`, laundry sorter basket on wheels sits at rank 1 in BM25's ranking and rank 2 in semantic retrieval's, so it scores 0.25 / (60 + 1) + 0.75 / (60 + 2) = 0.01620. The example scores every candidate this way, and its ranking matches `lib.blend.weighted_rrf` in 100 of 100 places. Tuning means trying a grid of α values on train queries and keeping the one with the best mean nDCG@10. Bruch et al. (2023) report three findings about tuning a [convex combination](#normalize-scores-before-adding), CC: "the learning of a CC fusion is generally agnostic to the choice of score normalization; that CC outperforms RRF in in-domain and out-of-domain settings; and finally, that CC is sample efficient." Sample efficient meant that α converged "with less than 5% of the training data", a fraction of training sets much larger than the store's. In the store, tune α over 0, 0.1, … 1 for the weighted sum under each of [the three normalizations](#normalize-scores-before-adding), and for weighted RRF with k = 60. Then test, as tuning checks, each normalization against min-max, and the two blends that go to the test queries against semantic retrieval: Example `tune-blend-weights-on-train` (Python 3.12.15 · bm25s 0.3.13 · numpy 2.5.4 · pandas 3.0.6 · scipy 1.18.1), passing in Edition 2026.10.1: ```python from lib.blend import ALPHAS, NORMALIZATIONS, best_alpha, candidates, scores_by_alpha, weighted_rrf, weighted_sum from lib.lexical import LexicalSearch from lib.semantic import SemanticSearch from lib.significance import compare from lib.store import load_products, load_queries, qrels products, queries, judgments = load_products(), load_queries("train"), qrels("train") lexical, semantic = LexicalSearch(products), SemanticSearch() tables = candidates(lexical, semantic, queries) bm25, semantic_run = lexical.run(queries, k=100), semantic.run(queries, k=100) per_query = {} # nDCG@10 per query (rows) and alpha (columns) for name in NORMALIZATIONS: per_query[name] = scores_by_alpha(lambda alpha: weighted_sum(tables, alpha, name), judgments) per_query["weighted rrf"] = scores_by_alpha(lambda alpha: weighted_rrf(bm25, semantic_run, alpha), judgments) semantic_alone = per_query["min-max"][1.0] # alpha = 1 is semantic retrieval in either blend print("alphas tried:", len(ALPHAS)) print("min-max ndcg@10 for alpha 0 to 1:", " ".join(f"{mean:.3f}" for mean in per_query["min-max"].mean())) print(f"semantic alone ndcg@10: {semantic_alone.mean():.3f}") print(f"unweighted rrf (alpha 0.5) ndcg@10: {per_query['weighted rrf'][0.5].mean():.3f}") best = {name: frame[best_alpha(frame)] for name, frame in per_query.items()} for name, frame in per_query.items(): print(f"{name} best alpha: {best_alpha(frame)}") print(f"{name} best ndcg@10: {best[name].mean():.3f}") for name in ["theoretical min-max", "z-score", "weighted rrf"]: # each tuned, against tuned min-max print(f"{name} against min-max, permutation p: {compare(best[name], best['min-max'])['permutation p']:.3f}") for name in ["min-max", "weighted rrf"]: # the two blends that go to test gain = compare(best[name], semantic_alone) print(f"{name} gain over semantic: {gain['mean difference']:.3f}") print(f"{name} gain over semantic, permutation p: {gain['permutation p']:.3f}") print(f"{name} gain over semantic, t-test p: {gain['t-test p']:.3f}") ``` Output: ```text alphas tried: 11 min-max ndcg@10 for alpha 0 to 1: 0.695 0.709 0.736 0.746 0.758 0.771 0.781 0.784 0.777 0.778 0.772 semantic alone ndcg@10: 0.772 unweighted rrf (alpha 0.5) ndcg@10: 0.774 min-max best alpha: 0.7 min-max best ndcg@10: 0.784 theoretical min-max best alpha: 0.9 theoretical min-max best ndcg@10: 0.789 z-score best alpha: 0.7 z-score best ndcg@10: 0.787 weighted rrf best alpha: 0.6 weighted rrf best ndcg@10: 0.787 theoretical min-max against min-max, permutation p: 0.164 z-score against min-max, permutation p: 0.349 weighted rrf against min-max, permutation p: 0.776 min-max gain over semantic: 0.013 min-max gain over semantic, permutation p: 0.177 min-max gain over semantic, t-test p: 0.161 weighted rrf gain over semantic: 0.015 weighted rrf gain over semantic, permutation p: 0.215 weighted rrf gain over semantic, t-test p: 0.205 ``` The output's second line traces the min-max blend from α = 0 to α = 1. It peaks at α = 0.7 with 0.784, against semantic retrieval's 0.772. Theoretical min-max peaks at α = 0.9 with 0.789, and the z-score at α = 0.7 with 0.787. Against tuned min-max, their permutation p-values are 0.164 and 0.349: at 0.05, the test on train queries can't tell either normalization from min-max, which fits Bruch et al.'s finding that the normalization hardly matters. Weighted RRF peaks at α = 0.6 with 0.787, and against tuned min-max its p-value is 0.776. Bruch et al.'s second finding, that the weighted sum beats RRF, doesn't show here, and these queries can't settle it. Every tuned blend scores at least as well as semantic retrieval on the train queries, and that is guaranteed: α = 1 is in the grid. Yet even on the queries that chose them, the permutation test can't tell either blend's gain from chance at 0.05. The weighted sum gains 0.013, with a permutation p-value of 0.177, and weighted RRF gains 0.015, with 0.215. Each gain is also the best of 11 tries on the same queries, which flatters it further. **What to do:** tune one weight on train queries over a coarse grid, and keep the normalization at min-max unless another one beats it by a margin the test on train queries can tell from chance, with p below 0.05. Treat the train gain over the best single retriever as optimistic, and expect it to shrink on test queries, a rule of thumb for any setting picked as the best of several. Before you trust the setting, [check that it's stable](#tuned-weight-stability), then [report it on test queries](#report-blends-on-test). ## Check that the tuned weight is stable {#tuned-weight-stability} The [tuned weights](#tune-blend-on-train) come from a few dozen queries, so they can depend on which queries you happened to have. The **bootstrap** shows how much: draw as many queries from the train set as it holds, with replacement, so some appear twice and others not at all, tune α on that sample, and repeat many times. If the chosen α jumps around, the tuned value reflects the sample more than the store. [Chapter 3](#report-the-size-of-a-gap) uses the same resampling to put intervals around scores. In the store, resample the train queries 1000 times with a fixed seed, and count how often each α wins for both tuned blends, the weighted sum of min-max scores and weighted RRF, on the same resamples: Example `blend-weight-bootstrap` (Python 3.12.15 · bm25s 0.3.13 · numpy 2.5.4 · pandas 3.0.6 · scipy 1.18.1), passing in Edition 2026.10.1: ```python import numpy as np import pandas as pd from lib.blend import best_alpha, candidates, scores_by_alpha, weighted_rrf, weighted_sum from lib.lexical import LexicalSearch from lib.semantic import SemanticSearch from lib.store import load_products, load_queries, qrels products, queries, judgments = load_products(), load_queries("train"), qrels("train") lexical, semantic = LexicalSearch(products), SemanticSearch() tables = candidates(lexical, semantic, queries) bm25, semantic_run = lexical.run(queries, k=100), semantic.run(queries, k=100) blends = {"weighted sum": scores_by_alpha(lambda alpha: weighted_sum(tables, alpha), judgments), "weighted rrf": scores_by_alpha(lambda alpha: weighted_rrf(bm25, semantic_run, alpha), judgments)} resamples = 1000 print("resamples:", resamples) rng = np.random.default_rng(2026) draws = [rng.integers(0, len(queries), len(queries)) for _ in range(resamples)] # the same draws for both for name, per_query in blends.items(): # per_query: train queries x alphas print(f"{name} alpha tuned on all train queries: {best_alpha(per_query)}") chosen = pd.Series([per_query.iloc[draw].mean().idxmax() for draw in draws]) # best alpha per resample for alpha, count in chosen.value_counts().sort_index().items(): print(f"{name} resamples choosing alpha {alpha}: {count}") ``` Output: ```text resamples: 1000 weighted sum alpha tuned on all train queries: 0.7 weighted sum resamples choosing alpha 0.5: 13 weighted sum resamples choosing alpha 0.6: 171 weighted sum resamples choosing alpha 0.7: 686 weighted sum resamples choosing alpha 0.8: 11 weighted sum resamples choosing alpha 0.9: 74 weighted sum resamples choosing alpha 1.0: 45 weighted rrf alpha tuned on all train queries: 0.6 weighted rrf resamples choosing alpha 0.5: 2 weighted rrf resamples choosing alpha 0.6: 513 weighted rrf resamples choosing alpha 0.7: 76 weighted rrf resamples choosing alpha 0.8: 365 weighted rrf resamples choosing alpha 0.9: 26 weighted rrf resamples choosing alpha 1.0: 18 ``` For the weighted sum, tuned at α = 0.7 on all the train queries, α = 0.7 wins 686 resamples and α = 0.6 wins 171. Semantic retrieval alone, α = 1, wins 45. Weighted RRF, tuned at α = 0.6, is less settled: α = 0.6 wins 513 resamples, α = 0.8 wins 365 and α = 0.7 wins 76. Both weights stay on the side of the grid that favours semantic retrieval, but RRF's moves between 0.6 and 0.8. Resampling isn't the only check. Products with equal BM25 scores have to be listed in some order, and `lib.lexical` lists them by `product_id`, an order that carries no information. Descending `product_id` order carries none either. The weighted sum gives tied products the same BM25 term, so their order hardly matters to it, but RRF gives them consecutive ranks, and so different scores. List BM25's ties by descending `product_id`, and tune the [best RRF k](#rrf-settings-on-train) and both weights again on the train queries: Example `tuning-with-ties-reversed` (Python 3.12.15 · bm25s 0.3.13 · numpy 2.5.4 · pandas 3.0.6 · scipy 1.18.1), passing in Edition 2026.10.1: ```python import numpy as np from lib.blend import best_alpha, candidates, scores_by_alpha, weighted_rrf, weighted_sum from lib.fusion import reciprocal_rank_fusion from lib.lexical import LexicalSearch from lib.metrics import evaluate from lib.semantic import SemanticSearch from lib.store import load_products, load_queries, qrels class TiesReversed(LexicalSearch): """BM25 with equal scores in descending product_id order, an order as arbitrary as ascending.""" def search(self, query, k=100): scores = super().search(query, k=len(self.ids)) # every match, so the cut at k can move too order = np.lexsort((-scores.index.to_numpy(), -scores.to_numpy())) return scores.iloc[order[:k]] products, queries, judgments = load_products(), load_queries("train"), qrels("train") semantic = SemanticSearch() semantic_run = semantic.run(queries, k=100) for ties, lexical in [("ties by product_id", LexicalSearch(products)), ("ties reversed", TiesReversed(products))]: bm25 = lexical.run(queries, k=100) by_k = {k: evaluate(reciprocal_rank_fusion([bm25, semantic_run], k=k), judgments)["ndcg@10"].mean() for k in [1, 10, 30, 60, 100, 300]} print(f"best rrf k, {ties}: {max(by_k, key=by_k.get)}") # ties go to the smaller k rrf = scores_by_alpha(lambda alpha: weighted_rrf(bm25, semantic_run, alpha), judgments) print(f"weighted rrf alpha, {ties}: {best_alpha(rrf)}") tables = candidates(lexical, semantic, queries) total = scores_by_alpha(lambda alpha: weighted_sum(tables, alpha), judgments) print(f"weighted sum alpha, {ties}: {best_alpha(total)}") ``` Output: ```text best rrf k, ties by product_id: 100 weighted rrf alpha, ties by product_id: 0.6 weighted sum alpha, ties by product_id: 0.7 best rrf k, ties reversed: 300 weighted rrf alpha, ties reversed: 0.8 weighted sum alpha, ties reversed: 0.7 ``` With the ties reversed, the best k is 300 instead of 100, and weighted RRF's α is 0.8 instead of 0.6. The weighted sum keeps α = 0.7. When an order that carries no information changes a setting, the train queries can't tell the settings apart: RRF's k and weight are fragile, and the weighted sum's weight isn't. The store has no more train queries to settle weighted RRF's weight, and its test comparison was planned with the weight tuned with ties by `product_id`, so α = 0.6 goes to test. Stability says the train queries agree on a weight, not that the weight improves search on queries it wasn't tuned on: the resamples reuse the same train queries, so they share whatever those queries have in common. **What to do:** bootstrap the train queries and tune on each sample, then tune once more with ties listed in another order that carries no information. If the chosen weight scatters across the grid, use a coarser grid or more train queries, or prefer the simpler setting. If the tie order alone moves it, say so wherever you report it. If it holds steady under both checks, take it to the test queries. ## Report the tuned blends on test queries {#report-blends-on-test} Only held-out queries show whether a tuned setting helps beyond the queries that chose it. Run each tuned blend once on the test queries, and compare it with semantic retrieval, the best single retriever, query by query, using the harness that [chapter 3](#compare-runs-in-a-harness) builds. The chapter fixed these two comparisons before looking. With the Bonferroni correction, a difference is established when its permutation p-value is below 0.025. The t-test and a bootstrap interval at the matching confidence level, 0.975, stand beside it. On the store's train queries, the [tuned blends](#tune-blend-on-train) gained 0.013 and 0.015 over semantic retrieval, neither of them a gain the permutation test could tell from chance. Before you read the output, predict: on the test queries, will the weighted sum with α = 0.7 and weighted RRF with α = 0.6 score above semantic retrieval, level with it, or below it? Example `tuned-blends-on-test` (Python 3.12.15 · bm25s 0.3.13 · numpy 2.5.4 · pandas 3.0.6 · scipy 1.18.1), passing in Edition 2026.10.1: ```python from lib.blend import best_alpha, candidates, scores_by_alpha, weighted_rrf, weighted_sum from lib.lexical import LexicalSearch from lib.metrics import evaluate from lib.semantic import SemanticSearch from lib.significance import compare from lib.store import load_products, load_queries, qrels lexical, semantic = LexicalSearch(load_products()), SemanticSearch() def blends(queries): tables, bm25, semantic_run = candidates(lexical, semantic, queries), lexical.run(queries), semantic.run(queries) return {"weighted sum": lambda alpha: weighted_sum(tables, alpha), # each blend's run at a given alpha "weighted rrf": lambda alpha: weighted_rrf(bm25, semantic_run, alpha)} train, test = blends(load_queries("train")), blends(load_queries("test")) threshold = 0.05 / len(test) # two planned comparisons, Bonferroni-corrected semantic_scores = evaluate(semantic.run(load_queries("test")), qrels("test")) print("test queries:", len(semantic_scores)) print(f"semantic ndcg@10: {semantic_scores['ndcg@10'].mean():.3f}") unweighted = evaluate(test["weighted rrf"](0.5), qrels("test")) # alpha 0.5 is chapter 1's fusion print(f"unweighted rrf ndcg@10: {unweighted['ndcg@10'].mean():.3f}") print(f"p-value threshold for each comparison: {threshold:.3f}") print(f"confidence level of each interval: {1 - threshold:.3f}") judged = [semantic_scores["judged@10"].mean()] for name, blend in test.items(): alpha = best_alpha(scores_by_alpha(train[name], qrels("train"))) # train only scores = evaluate(blend(alpha), qrels("test")) judged.append(scores["judged@10"].mean()) result = compare(scores["ndcg@10"], semantic_scores["ndcg@10"], confidence=1 - threshold) print(f"{name} alpha chosen on train: {alpha}") print(f"{name} ndcg@10: {scores['ndcg@10'].mean():.3f}") print(f"{name} minus semantic: {result['mean difference']:.3f}") print(f"{name} permutation p: {result['permutation p']:.3f}") print(f"{name} t-test p: {result['t-test p']:.3f}") print(f"{name} interval: {result['ci low']:.3f} to {result['ci high']:.3f}") print(f"{name} established: {result['permutation p'] < threshold}") print(f"lowest judged@10: {min(judged):.3f}") print(f"highest judged@10: {max(judged):.3f}") ``` Output: ```text test queries: 30 semantic ndcg@10: 0.830 unweighted rrf ndcg@10: 0.812 p-value threshold for each comparison: 0.025 confidence level of each interval: 0.975 weighted sum alpha chosen on train: 0.7 weighted sum ndcg@10: 0.816 weighted sum minus semantic: -0.014 weighted sum permutation p: 0.229 weighted sum t-test p: 0.221 weighted sum interval: -0.042 to 0.008 weighted sum established: False weighted rrf alpha chosen on train: 0.6 weighted rrf ndcg@10: 0.825 weighted rrf minus semantic: -0.005 weighted rrf permutation p: 0.650 weighted rrf t-test p: 0.643 weighted rrf interval: -0.029 to 0.016 weighted rrf established: False lowest judged@10: 0.980 highest judged@10: 0.990 ``` Semantic retrieval scores 0.830, the weighted sum of min-max scores 0.816, and weighted RRF 0.825. The three [runs](#score-every-test-query)' judged@10 values lie between 0.980 and 0.990, so missing judgments weigh about equally on all of them. Neither blend beats semantic retrieval. The weighted sum differs from it by -0.014, with a permutation p-value of 0.229, a t-test p-value of 0.221 and an interval from -0.042 to 0.008. Weighted RRF differs by -0.005, with p-values of 0.650 and 0.643 and an interval from -0.029 to 0.016. Both intervals contain 0, and the tests agree with them. The example prints its verdict for each comparison, whether the permutation p-value falls below the threshold: False for the weighted sum and False for weighted RRF. So the finding is "no improvement", not "worse". For weighted RRF, it is the finding for α = 0.6, a weight that [the order of BM25's ties can move](#tuned-weight-stability). If you predicted "above", you trusted a train gain that the permutation test couldn't tell from chance even on the queries that chose it, and that was the best of several tries. **What to do:** adopt a blend only when it beats the best single retriever on held-out queries, with a permutation p-value below a threshold corrected for the number of planned comparisons. When it doesn't, keep the simpler system. For the store, that leaves semantic retrieval alone as the ranker to beat. Keep the tuned weights and the code, and test the blends again when a retriever or the catalog changes. ## Learn sparse term weights with SPLADE {#learned-sparse-splade} Instead of fusing two retrievers, you can change one so that it misses less. BM25 only matches words that appear in a product's text. **Learned sparse retrieval** keeps BM25's machinery, an [inverted index](#build-an-inverted-index) of terms, but has a neural model choose each product's terms and their weights. SPLADE is one such model ([Formal et al., 2021](https://arxiv.org/abs/2107.05720)). A language model of the BERT family, a transformer that reads text as subword tokens, reads the product's text and, at each token i, gives every term j in its vocabulary a raw score w_ij. ReLU sets the negative scores to 0, and the original paper sums a logarithm of what's left over the text's tokens: ```text weight of term j = sum over the text's tokens i of log(1 + ReLU(w_ij)) ``` The logarithm is a saturation, like BM25's: in the authors' words, it "prevents some terms to dominate and naturally ensures sparsity in representations". SPLADE v2 replaced the sum with the maximum over the tokens, and the authors made max pooling "the default configuration for SPLADE" ([Formal et al., 2021, SPLADE v2](https://arxiv.org/abs/2109.10086)). A SPLADE vector has one dimension per vocabulary entry, 30,522 for a BERT vocabulary, and typically more than 99% of them are zero ([Sentence Transformers](https://sbert.net/docs/sparse_encoder/usage/usage.html)). The nonzero terms include **expansion terms**: words the text doesn't contain but the model weights anyway, so a product can match a query word it never uses. Expansion terms target one of BM25's failures in the store. Count, on the train queries, how many [exact](#queries-each-retriever-wins) products lack at least one of the query's words, as BM25's tokenizer sees them: Example `exact-products-missing-query-words` (Python 3.12.15 · bm25s 0.3.13 · numpy 2.5.4 · pandas 3.0.6 · scipy 1.18.1), passing in Edition 2026.10.1: ```python from collections import Counter import bm25s from lib.store import load_products, load_queries, product_text, qrels products, queries, judgments = load_products(), load_queries("train"), qrels("train") def words(texts): """The tokens BM25 matches on: bm25s's default tokenizer with English stopwords, as in lib.lexical.""" tokenized = bm25s.tokenize(texts, stopwords="en", return_ids=False, show_progress=False) return [set(tokens) for tokens in tokenized] product_words = dict(zip(products.index, words(product_text(products).tolist()))) query_words = dict(zip(queries.index, words(queries["query"].tolist()))) exact_pairs = [(query_id, product_id) for query_id in queries.index for product_id, grade in judgments[query_id].items() if grade == 2] missed = Counter(word for query_id, product_id in exact_pairs for word in query_words[query_id] - product_words[product_id]) missing = sum(not query_words[query_id] <= product_words[product_id] for query_id, product_id in exact_pairs) print("exact train pairs:", len(exact_pairs)) print("exact pairs missing a query word:", missing) print(f"share missing a query word: {missing / len(exact_pairs):.3f}") print("most often missing:", ", ".join(f"{word} ({count})" for word, count in missed.most_common(8))) exact = [product_id for query_id, product_id in exact_pairs if query_id == 453] # midcentury tv unit print("exact products for midcentury tv unit:", len(exact)) for word in ["midcentury", "unit", "stand"]: print(f"exact products containing {word}:", sum(word in product_words[product_id] for product_id in exact)) mid_century = product_text(products)[exact].str.contains("mid-century") print("exact products whose text says mid-century:", mid_century.sum()) ``` Output: ```text exact train pairs: 1937 exact pairs missing a query word: 1454 share missing a query word: 0.751 most often missing: ligth (193), decorative (134), modern (102), freestanding (78), floats (58), queen (57), dining (56), infant (43) exact products for midcentury tv unit: 25 exact products containing midcentury: 0 exact products containing unit: 0 exact products containing stand: 25 exact products whose text says mid-century: 20 ``` Of 1937 exact (query, product) pairs, 1454 miss at least one query word, a share of 0.751. The output lists the words missed most often. Some are typos, which [chapter 8's corrector](#correct-spelling-from-catalog) targets, and plural forms, which [chapter 5 stems](#stem-word-forms); others are descriptive words the product text leaves out, or different words for one thing. For `midcentury tv unit`, of the 25 exact products, 0 contain `midcentury` and 0 contain `unit`. 20 say `mid-century` instead, and 25 say `stand`. Expansion terms are aimed at gaps like these, though whether a given model learns that a TV stand is a TV unit is something you'd measure. The SPLADE authors' own technical report gives SPLADE-v3 an MRR@10 above 0.40 on the MS MARCO development set, a passage-ranking benchmark where [MRR@10](#score-the-first-relevant-result) measures how high the first relevant passage lands, and describes it as "comparing well to cross-encoder re-rankers" ([Lassance et al., 2024](https://arxiv.org/abs/2403.06789)). The book doesn't run SPLADE. The reference checkpoints, [`naver/splade-cocondenser-ensembledistil`](https://huggingface.co/naver/splade-cocondenser-ensembledistil) and [`naver/splade-v3`](https://huggingface.co/naver/splade-v3), are licensed CC-BY-NC-SA-4.0, which rules out commercial use. An Apache-2.0 model, [`opensearch-neural-sparse-encoding-doc-v3-distill`](https://huggingface.co/opensearch-project/opensearch-neural-sparse-encoding-doc-v3-distill), runs its neural model on products only, and encodes queries with a tokenizer and a table of weights. **What to do:** consider learned sparse retrieval when BM25's misses are mostly vocabulary misses and you want to keep an inverted index. Check the model's licence first, and measure it like any retriever: alone, per query, on train queries, then on test. ## Match token by token with ColBERT {#late-interaction-colbert} Where [SPLADE](#learned-sparse-splade) changes lexical retrieval, late interaction changes semantic retrieval. The store's semantic retrieval is **single-vector**: one embedding per product and one per query, compared once. **Late interaction** keeps one vector per token instead, so it is also called **multi-vector** retrieval. ColBERT encodes the query and the product separately, as an embedding model does, and compares them only at the end. For each query token it finds the most similar product token, and it sums those maxima, an operation called MaxSim ([Khattab and Zaharia, 2020](https://arxiv.org/abs/2004.12832)): ```text score(query, product) = sum over query tokens q of max over product tokens d of cosine(q, d) ``` Because products are encoded without the query, their token vectors are computed once, ahead of time, which a [cross-encoder](#rerank-with-cross-encoder) can't do: it reads both texts as one input. The ColBERT paper reports effectiveness "competitive with existing BERT-based models" that read the query and passage together, at speeds "two orders-of-magnitude faster". The price is storage: one vector per token rather than per product. ColBERTv2 compresses them, "reducing the space footprint of late interaction models by 6–10×" ([Santhanam et al., 2022](https://arxiv.org/abs/2112.01488)). In the store, take `midcentury tv unit` again, the query whose exact products [say `stand`, never `unit`](#learned-sparse-splade). A single vector has to fold the whole query into one direction, and the whole product text into another. With late interaction, the query's `unit` is compared with each of the product's tokens, `stand` among them, and its best match counts on its own. Token-level matching helps on some hard cases. On NevIR, a negation benchmark where random choice scores 25%, ColBERTv1 scored 19.7% and ColBERTv2 13.0%, above every single-vector model tested (12% or less), though still below chance ([Weller et al., 2024](https://arxiv.org/abs/2305.07614)). In BEIR, a benchmark that tests models zero-shot, on datasets they weren't trained for, rerankers and late-interaction models "on average achieve the best zero-shot performances", at a high computational cost ([Thakur et al., 2021](https://arxiv.org/abs/2104.08663)). Open tooling exists, such as the PyLate library and the Apache-2.0 model [`lightonai/GTE-ModernColBERT-v1`](https://huggingface.co/lightonai/GTE-ModernColBERT-v1) ([Chaffin and Sourty, 2025](https://arxiv.org/abs/2508.03555)). The book doesn't run ColBERT either. **What to do:** consider late interaction when single-vector retrieval misses details that hang on one word, and you can afford a vector per token. For a catalog the size of the store's, we recommend testing a reranker on a shortlist first ([chapter 9](#rerank-semantic-with-cross-encoder)): it reads the same details at query time and needs no new index, though none of chapter 9's rerankers shows an established gain over the store's semantic retrieval. ## Decide when hybrid retrieval pays {#when-hybrid-helps} Whether fusion helps depends on the retrievers you fuse, and published studies point both ways. Lexical and dense retrieval, dense meaning embedding-based, can complement each other even when the dense model is the weaker one: out of domain, a zero-shot hybrid gained "an average of 20.4% relative gain over the deep retrieval model" across three datasets ([T. Chen et al., 2022](https://arxiv.org/abs/2201.10582)). The lexical baseline matters. The authors of DPR, a dense retriever, found little gain from a hybrid, and a replication with a stronger BM25 found "statistically significant gains" ([Ma et al., 2021](https://arxiv.org/abs/2104.05740)). The strength of the semantic retriever matters too. On BRIGHT, a benchmark of reasoning-heavy queries, fusing BM25 by a weighted sum of min-max scores at α = 0.5 added 3.5 to 4.9 points of nDCG@10 to mid-tier dense retrievers and cost the two strongest 2.0 to 2.9. "However, fusion can harm already-strong retrievers", the authors write, because "the fused list promotes lexically similar but semantically irrelevant documents" ([Abdallah et al., 2026](https://arxiv.org/abs/2604.03676)). Production reports usually compare a hybrid with lexical search alone. A retailer's preprint reports that hybrid lexical and vector search beat lexical search alone in an online A/B test, raising click-through by 0.97% and conversion by 0.98% ([Sonagara et al., 2026](https://arxiv.org/abs/2609.31498)). That is evidence for adding semantic retrieval to a lexical system, not for adding lexical retrieval to a strong semantic one, which is the store's question. The store sits in the strong-retriever case: on the test queries, [chapter 3](#compare-runs-in-a-harness) establishes semantic retrieval's lead over BM25, and [neither tuned blend beats it](#report-blends-on-test). The store's BM25 is the book's unstemmed baseline. [Chapter 5](#stem-word-forms) builds a stronger one, and Ma et al.'s replication is a reason to test the blends again with it. Fusing scores isn't the only kind of hybrid. A search engine can also use lexical matches as a filter on semantic retrieval: one engine adds an embedding operator to its query language, so that "hybrid retrieval expressions, with arbitrary combinations of embeddings and terms" run as one query (Huang et al., 2020). For the store, that could mean requiring a size such as `48` to match exactly, then ranking the matches by embedding. That is a hypothesis to test, not a result, and a naive filter fails on `7qt slow cooker`, whose spelling [no product uses](#queries-each-retriever-wins). Deciding when a query carries such a constraint is query understanding ([chapter 8](#classify-queries-from-products)), and chapter 8 [tries such a filter](#standalone-follow-ups) on a shopper's follow-up messages that name a color, shape, size or finish. Choosing a weight per query is query understanding too, and this book doesn't build it. Picking the better ranking for each train query would score 0.802, and a per-query weight, which mixes the lists, isn't bound by that figure. One per-query method asks an LLM, a large language model, to judge each retriever's top result and sets α from that ([Hsu and Tzeng, 2025](https://arxiv.org/abs/2503.23013)). The BRIGHT study found a score-based variant of it unstable (Abdallah et al., 2026). **What to do:** we recommend trying a hybrid when your two retrievers are close in quality, when your embedding model works outside the domain it was trained for, or when many queries depend on exact terms your embeddings miss. Start with weighted RRF: it needs ranks only, and on the store's train queries the permutation test couldn't tell it from the weighted sum. Move to a weighted sum of normalized scores when both engines give you scores for every candidate: two studies found it ahead of RRF on their data (Bruch et al., 2023; Abdallah et al., 2026), and on the store's train queries its weight was [the more stable](#tuned-weight-stability). When one retriever is established as the stronger on held-out queries and no tuned blend beats it there, use it alone, keep the other as a baseline, and test a lexical filter for queries with exact constraints. That is the store's case. ## What goes wrong in production {#hybrid-retrieval-in-production} **Two retrievers cost two of everything.** The examples here score a sample of the catalog offline, but a live hybrid search keeps two indexes up to date and runs two searches on every query. They can run in parallel, but the slower one sets the response time, and a failure in either degrades the result. Count that cost against the gain you measured. **Short lists drop candidates.** Fusion can't rank a product neither retriever returned. On the store's train queries, the union of two top-100 lists holds 0.810 of each query's exact products, and the union of two top-20 lists 0.445. A depth limit in one engine's configuration can cut candidates without anyone noticing. **Missing scores need a rule.** When each retriever returns only its top k, a product in one list but not the other has no score from the second. The DPR replication assigns it "the minimum dense retrieval score among the retrieved documents", and the reverse for BM25 ([Ma et al., 2021](https://arxiv.org/abs/2104.05740)). Bruch et al. compute the missing score exactly instead, as `lib.blend.candidates` does, which a production engine may not let you do ([Bruch et al., 2023](https://arxiv.org/abs/2210.11934)). **Min-max is relative.** The best candidate for a query always gets 1, even when every candidate is poor. A query that shares no word with any product in the catalog gives every candidate a BM25 score of 0. Plain min-max then divides by zero, and `lib.blend` returns zeros, so the blend silently becomes semantic retrieval alone. No store query does this; a live search box will. **A fused score isn't a confidence.** Retrieval scores are poorly calibrated: "Confidence calibration is consistently weak across model families" ([Abdallah et al., 2026](https://arxiv.org/abs/2604.03676)). Don't set a threshold on a fused score to decide that a query has no good results. **A tuned weight belongs to one pair of retrievers.** The store's α fits these BM25 settings, this embedding model and this catalog. Change any of them, and re-tune on train queries before you trust the old weight. A tuned RRF "generalizes poorly to out-of-domain datasets" ([Bruch et al., 2023](https://arxiv.org/abs/2210.11934)). **What to do:** log both retrievers' ranks next to the fused result, check the list depth in every engine's configuration, and re-tune α whenever a retriever changes. ## Decisions this chapter made {#hybrid-retrieval-decisions} Each choice below is a default for the store, and each has a reason a later change would need to beat. - **Tune on train, report on test.** Every analysis and every tuned setting uses the 50 train queries, and comparisons there are tuning checks, not evidence. The chapter reads the test queries once, to report. - **Two planned test comparisons,** each tuned blend against semantic retrieval, decided by the permutation test at 0.05 divided by 2, with the t-test and a bootstrap interval beside it. - **Check each retriever before fusing,** with judged@10 and a paired test, and count per query where each one wins. - **Candidates: the union of both retrievers' top 100,** with both scores looked up for every candidate, so no product's score is missing. - **Normalization: per-query min-max.** The permutation test on train can't tell the other two normalizations from it at 0.05, and min-max needs no knowledge of either scoring function. - **RRF: k = 60, with lists 100 deep.** The best k on train gains nothing over 60 that the permutation test can tell from chance at 0.05, and it changes with the order of BM25's ties. A deeper list can only add candidates. - **One weight, α, tuned over 0 to 1 in steps of 0.1** for both blends, chosen by mean nDCG@10 on train, then bootstrapped and tuned again with BM25's ties reversed. The weighted sum's α holds under both checks; weighted RRF's doesn't. - **Adopt a blend only if it beats the best single retriever on test.** Neither tuned blend does, so the store's ranker to beat stays semantic retrieval alone, with BM25, unstemmed, kept as the baseline. - **SPLADE and ColBERT are described, not recorded,** because neither is needed to answer the store's question. # Understanding queries By the end of this chapter you can change a shopper's query before retrieval: correct its spelling, write its units one way, expand it with synonyms, predict its product class, rewrite it with a large language model, search with a product that model imagines, and turn a follow-up into a standalone query. You can also decide, from human judgments, whether each change helps the retriever it feeds. The store whose search this book follows, an online seller of furniture and home goods ([chapter 1](#store-data)), gets short queries, and some of them can't match the products the shopper wants. `ligth bulb` misspells a word. `7qt slow cooker` writes its size as a term that 0 products contain. For `midcentury tv unit`, [chapter 7 counts](#learned-sparse-splade) 25 [exact](#store-data) products, of which 0 contain `midcentury` and 0 contain `unit`. A shopper who types `in white` after a search for a desk means a white desk, but the words alone don't say so. **Query understanding** is everything a search system does to a query before retrieval to close gaps like these: it corrects, normalizes, expands, classifies and rewrites. Chapters 5 to 7 changed the retrievers. This chapter changes the query and keeps the retrievers fixed: [BM25](#rank-with-bm25), the book's unstemmed baseline in `lib.lexical`, and [semantic retrieval](#search-by-meaning), which compares the query's embedding with each product's, both computed in advance by `Snowflake/snowflake-arctic-embed-m-v1.5`. Three of the techniques use a **large language model** (LLM), a model trained on large amounts of text to generate text in response to a prompt. The store's LLM is `gemma4:12b-it-q4_K_M`, an open model that Google publishes under the Apache 2.0 licence ([model card](https://ai.google.dev/gemma/docs/core/model_card_4)). The examples never call it. Two scripts, `recordings/llm_rewrites.py` and `recordings/llm_conversations.py`, ran it once, at temperature 0 with a fixed seed, and saved its outputs as **recordings**, which the examples read through `lib.recorded`. Each recording's `manifest.json` holds the full prompts. We chose the model because anyone can download it and run it on their own machine, so anyone can rerun the recordings. The LLM sees only the shopper's words, never a product. Every score in the chapter uses the human judgments. Every setting is chosen on the 50 [train queries](#store-data), and the 30 test queries are kept for reporting. Before reading any test result, the chapter fixes 5 planned comparisons on the test queries, each against the same retriever on the query as typed: 1. BM25 with a [synonym list](#expand-with-synonyms) 2. semantic retrieval with a [boost for the predicted class](#boost-predicted-class) 3. BM25 on the [LLM's rewrite](#rewrite-queries-with-llm) 4. semantic retrieval on the LLM's rewrite 5. semantic retrieval with [HyDE](#hyde-hypothetical-products), which averages the query's embedding with that of a product the LLM imagines By [the book's rule](#when-a-difference-is-established), a difference is established when its paired [permutation test](#permutation-test-on-queries) gives a p-value below 0.01, 0.05 divided by the number of planned comparisons. The [paired t-test](#paired-t-test-on-queries) and a 99% [bootstrap interval](#report-the-size-of-a-gap) stand beside it. Spelling correction, unit normalization and follow-up rewriting each touch too few queries to test, so their sections are diagnoses, not tests. ## Correct spelling against the catalog's words {#correct-spelling-from-catalog} The first gap is a typo, as in `ligth bulb`. In [BM25](#rank-with-bm25), a misspelled word matches only products with the same misspelling, and it can move the query's embedding away from the products it means, as [chapter 6 shows](#semantic-retrieval-in-production). Search queries hold many such errors: one study of web search reports that "Roughly 10-15% of the queries sent to search engines contain errors" ([Cucerzan and Brill, 2004](https://aclanthology.org/W04-3238/)). The classic corrector is the **noisy channel** model: for a typed word t, choose the correction c that maximizes P(c) × P(t | c) ([Kernighan et al., 1990](https://aclanthology.org/C90-2036/)). P(c) says how common the candidate word is. P(t | c), the error model, says how likely a shopper who meant c is to type t. Kernighan et al. take as candidates the words one edit from the typo: a deletion, an insertion, a replacement or a swap of two neighbouring letters. The number of insertions, deletions and replacements that turn one string into another is their **edit distance** ([Manning et al., 2008](https://nlp.stanford.edu/IR-book/html/htmledition/edit-distance-1.html)), and the corrector here also counts a swap as one edit. A textbook states the two principles in plain terms: of the correct spellings, choose the nearest, and between near ties, choose the more common ([Manning et al., 2008](https://nlp.stanford.edu/IR-book/html/htmledition/implementing-spelling-correction-1.html)). Norvig writes such a corrector in a few lines ([Norvig](https://norvig.com/spell-correct.html)). ```text correction(t) = argmax over candidates c of P(c) × P(t | c) ``` The **lexicon**, the list of words that count as correct, decides what gets corrected. Web search engines take it from their query logs, choosing "the correction that is most common among queries typed in by other users" (Manning et al.). The store has no log, so its lexicon comes from the catalog's text. Where the lexicon comes from matters for brand names: in product search, a corrector "must distinguish genuine misspellings from novel brand names that use unconventional spelling" ([Guo et al., 2024](https://arxiv.org/abs/2410.11655)). In the store, build the corrector with the crudest error model: a word in the lexicon stays, one edit beats two, and P(c), how common each candidate is, breaks ties between candidates the same number of edits away. Run it on all 80 queries: Example `catalog-spelling-corrector` (Python 3.12.15 · numpy 2.5.4 · pandas 3.0.6 · scikit-learn 1.9.1 · scipy 1.18.1), passing in Edition 2026.10.1: ```python import re from collections import Counter from lib.queries import correct_query from lib.store import load_products, load_queries, product_text texts = product_text(load_products()).str.lower() counts = Counter(re.findall(r"[a-z]+", " ".join(texts))) # the lexicon, and how common each word is def edits1(word): """Every string one edit away: a deletion, a swap of two neighbouring letters, a replacement or an insertion.""" splits = [(word[:cut], word[cut:]) for cut in range(len(word) + 1)] letters = "abcdefghijklmnopqrstuvwxyz" return ({left + right[1:] for left, right in splits if right} | {left + right[1] + right[0] + right[2:] for left, right in splits if len(right) > 1} | {left + letter + right[1:] for left, right in splits if right for letter in letters} | {left + letter + right for left, right in splits for letter in letters}) def correct(word): """Keep a word the catalog uses. Otherwise take the commonest catalog word one edit away, else two.""" if not word.isalpha() or word in counts: return word candidates = {edit for edit in edits1(word) if edit in counts} or { two for edit in edits1(word) for two in edits1(edit) if two in counts} return max(sorted(candidates), key=counts.get) if candidates else word queries = load_queries() corrected = queries["query"].map(lambda query: " ".join(correct(word) for word in query.lower().split())) changed = queries[corrected != queries["query"].str.lower()] print(f"catalog words: {len(counts):,}") print("queries changed:", len(changed)) print("of them, train queries:", (changed["split"] == "train").sum()) for query_id, query in changed["query"].items(): print(f" {query} ({changed.at[query_id, 'split']}), corrected: {corrected[query_id]}") same = corrected.equals(queries["query"].map(lambda query: correct_query(query, counts))) print("same as lib.queries.correct_query:", same) for word in ["tye", "tie", "the", "kohler", "whitehaven", "malachi", "luau"]: # P(c), as counts print(f"catalog count of {word}:", counts[word]) ``` Output: ```text catalog words: 19,680 queries changed: 4 of them, train queries: 3 foutains with brick look (train), corrected: fountains with brick look twin over full bunk beds cool desins (train), corrected: twin over full bunk beds cool designs ligth bulb (train), corrected: light bulb tye dye duvet cover (test), corrected: the dye duvet cover same as lib.queries.correct_query: True catalog count of tye: 0 catalog count of tie: 45 catalog count of the: 20389 catalog count of kohler: 71 catalog count of whitehaven: 60 catalog count of malachi: 5 catalog count of luau: 4 ``` The catalog uses 19,680 distinct words. The corrector changes 4 queries, and the shared function `lib.queries.correct_query` makes the same corrections. Of those, 3 are [train queries](#store-data), and the corrector spells each one as [chapter 6's spelling probes](#semantic-retrieval-in-production) do: the correctly spelled copies of the typo queries that chapter 6 wrote for the book. The corrector leaves the brands alone, because the catalog uses them: `kohler` 71 times and `whitehaven` 60. Some of the words it keeps are rare, such as `malachi`, used 5 times, and `luau`, used 4. A lexicon that left out rare words would treat both as typos. The corrector's failures are in test queries. `tye dye duvet cover` becomes `the dye duvet cover`: `tye` is one edit from `tie` (45 times in the catalog) and from `the` (20389), and P(c) picks the common word. And `refrigerator with ice an water in door` keeps its `an`. That is a **real-word error**: every word is spelled right, as in "flew form Heathrow" ([Manning et al., 2008](https://nlp.stanford.edu/IR-book/html/htmledition/context-sensitive-spelling-correction-1.html)). Both need context. For `tye` and for `an`, count how often each catalog word within one edit of it comes right before the next word of the query, `dye` or `water`: Example `spelling-context-counts` (Python 3.12.15 · numpy 2.5.4 · pandas 3.0.6 · scikit-learn 1.9.1 · scipy 1.18.1), passing in Edition 2026.10.1: ```python import re from collections import Counter from lib.queries import correct_word, edits1, word_counts from lib.store import load_products, product_text texts = product_text(load_products()).str.lower() counts = word_counts(texts) # the corrector's lexicon words_per_product = [re.findall(r"[a-z]+", text) for text in texts] pairs = Counter(pair for words in words_per_product for pair in zip(words, words[1:])) # neighbouring words # tye in "tye dye duvet cover", an in "refrigerator with ice an water in door": each word, the word after it for word, after in [("tye", "dye"), ("an", "water")]: options = sorted(((pairs[(edit, after)], edit) for edit in edits1(word) | {word} if edit in counts), reverse=True) for count, edit in options[:3]: print(f"catalog count of {edit} {after}:", count) choice = correct_word(word, counts) # what the corrector without context picks print(f"the corrector's choice for {word}: {choice}") print(f"catalog count of {choice} {after}, the corrector's choice:", pairs[(choice, after)]) ``` Output: ```text catalog count of tie dye: 18 catalog count of type dye: 0 catalog count of tyke dye: 0 the corrector's choice for tye: the catalog count of the dye, the corrector's choice: 0 catalog count of and water: 59 catalog count of a water: 20 catalog count of in water: 9 the corrector's choice for an: an catalog count of an water, the corrector's choice: 0 ``` Before `dye`, the catalog's text has `tie` 18 times, and the corrector's choice, `the`, 0. Before `water`, it has `and` 59 times, ahead of `a` at 20, and `an` 0. For these two queries, a corrector that scored each candidate with its neighbouring word would pick `tie` and `and`. Now measure each corrected query on its own judgments: Example `measure-spelling-corrections` (Python 3.12.15 · bm25s 0.3.13 · numpy 2.5.4 · pandas 3.0.6 · scikit-learn 1.9.1 · scipy 1.18.1), passing in Edition 2026.10.1: ```python from lib.lexical import LexicalSearch from lib.metrics import ndcg_at_k from lib.queries import correct_query, word_counts from lib.recorded import probes from lib.semantic import SemanticSearch from lib.store import load_products, load_queries, product_text, qrels products, queries, judgments = load_products(), load_queries(), qrels() counts = word_counts(product_text(products)) bm25, semantic = LexicalSearch(products), SemanticSearch() probe_queries, probe_vectors = probes() # recorded embeddings of the four typo queries, spelled correctly spelling = probe_queries[probe_queries["kind"] == "spelling"] spelled = {row["query"]: probe_vectors[index] for index, row in spelling.iterrows()} typo_queries = spelling["of_query_id"].astype(int).tolist() print("typo queries:", len(typo_queries)) print("of them, test queries:", (queries.loc[typo_queries, "split"] == "test").sum()) for query_id in typo_queries + [459]: # the typo queries, then the one variant spelling, tye dye typed = queries.at[query_id, "query"] corrected = correct_query(typed, counts) label = f"{query_id} ({queries.at[query_id, 'split']})" print(f"{label}: {typed} -> {corrected}") for name, text in [("typed", typed), ("corrected", corrected)]: ranking = list(bm25.search(text, k=10).index) print(f"{label}, bm25 ndcg@10 {name}: {ndcg_at_k(ranking, judgments[query_id], 10):.3f}") if corrected == typed.lower(): print(f"{label}: the corrector changes nothing") elif corrected in spelled: for name, vector in [("typed", semantic.query_vectors[query_id]), ("corrected", spelled[corrected])]: ranking = list(semantic.search_vector(vector, k=10).index) print(f"{label}, semantic ndcg@10 {name}: {ndcg_at_k(ranking, judgments[query_id], 10):.3f}") else: print(f"{label}: no recorded embedding of the corrected text") ``` Output: ```text typo queries: 4 of them, test queries: 1 305 (train): ligth bulb -> light bulb 305 (train), bm25 ndcg@10 typed: 0.945 305 (train), bm25 ndcg@10 corrected: 0.945 305 (train), semantic ndcg@10 typed: 1.000 305 (train), semantic ndcg@10 corrected: 1.000 20 (train): foutains with brick look -> fountains with brick look 20 (train), bm25 ndcg@10 typed: 0.356 20 (train), bm25 ndcg@10 corrected: 0.507 20 (train), semantic ndcg@10 typed: 0.353 20 (train), semantic ndcg@10 corrected: 0.645 224 (train): twin over full bunk beds cool desins -> twin over full bunk beds cool designs 224 (train), bm25 ndcg@10 typed: 0.488 224 (train), bm25 ndcg@10 corrected: 0.488 224 (train), semantic ndcg@10 typed: 0.565 224 (train), semantic ndcg@10 corrected: 0.616 406 (test): refrigerator with ice an water in door -> refrigerator with ice an water in door 406 (test), bm25 ndcg@10 typed: 0.501 406 (test), bm25 ndcg@10 corrected: 0.501 406 (test): the corrector changes nothing 459 (test): tye dye duvet cover -> the dye duvet cover 459 (test), bm25 ndcg@10 typed: 0.953 459 (test), bm25 ndcg@10 corrected: 0.953 459 (test): no recorded embedding of the corrected text ``` For the train query `foutains with brick look`, BM25's nDCG@10 goes from 0.356 to 0.507 and semantic retrieval from 0.353 to 0.645. For `twin over full bunk beds cool desins`, only semantic retrieval moves, from 0.565 to 0.616, and for `ligth bulb` neither does. For `refrigerator with ice an water in door`, the corrector changes nothing, and a correct fix couldn't help BM25 either. Of the typed `an` and the correct `and`, bm25s drops `an, and` as stop words, and chapter 6's copy of the query spelled with `and` scores 0.501, the same as the query as typed. For `tye dye duvet cover`, BM25 scores 0.953 either way, because `the` is a stop word and `tye` matches no product, so both versions search the same terms. Semantic retrieval can't be scored on the corrected text, because the recordings hold no embedding of it. The store has 4 typo queries, 1 of them a test query, and the smallest two-sided permutation p-value that so few could give is 0.125. **What to do:** build the lexicon from your catalog, and from your query log once you have one, and correct only words it doesn't hold, so that brand names survive. Before you correct a word, check its neighbours: a corrector without context turns variant spellings into common words and misses real-word errors. Score each correction on the typo queries' own judgments, and collect many more typo queries than the store has before you test a corrector. ## Normalize units the way the catalog writes them {#normalize-units} A typo has one right spelling; a size, as in `7qt slow cooker`, often has several. **Normalization** maps the different ways of writing one thing to a single form, on the products and on the query alike. In the store, sizes need it. [Chapter 5](#split-text-into-terms) found that `7qt` and `7 qt` never match each other, because the tokenizer drops the one-character `7`. A normalizer that joins each number to its unit, written one way, turns both into one term, `7qt`. It has to run on the product text when you build the index, and on every query. It also has to produce terms the tokenizer keeps whole. The tokenizer splits words at a decimal point, so `3.7qt` would become `7qt`. To prevent that, the shared function `lib.queries.normalize_units` writes the point as an underscore, which the tokenizer keeps inside a word. It also leaves `in` alone when a number or `one` follows, as in `3 in 1`. In the store, a rule written from the queries would cover `qt`, `in`, `inch` and `"`. Before you trust it, count how the catalog writes inches. The example counts them, prints the terms that the tokenizer makes of a few normalized texts, then builds BM25 three times: over the text as written, normalized by that first rule, and normalized by `normalize_units`, which also handles the catalog's two apostrophes, `''`, for inches. It scores every query, train or test, that gives a number with a unit: Example `normalize-units-both-sides` (Python 3.12.15 · bm25s 0.3.13 · numpy 2.5.4 · pandas 3.0.6 · pystemmer 3.1.0 · scikit-learn 1.9.1 · scipy 1.18.1), passing in Edition 2026.10.1: ```python from lib.lexical import LexicalSearch from lib.lexical_by_hand import tokenize from lib.metrics import ndcg_at_k from lib.queries import INCH_SPELLINGS, normalize_units, unit_rules from lib.store import load_products, load_queries, product_text, qrels products, queries, judgments = load_products(), load_queries(), qrels() text = product_text(products).str.lower() for name, unit in [("''", r"''"), ('"', r"\""), ("inch", r"-?inch")]: print(f"products that write a number then {name}:", text.str.contains(rf"\d\s*{unit}").sum()) first_try = unit_rules([spelling for spelling in INCH_SPELLINGS if spelling != "''"]) # written from queries rules = {"as written": None, "first try": first_try, "normalized": unit_rules()} checks = [("3.7-quart slow cooker", "normalized"), ("3 in 1 sofa bed", "normalized"), ("1 in twin", "normalized"), ("48 '' range hood", "first try"), ("48 '' range hood", "normalized")] for example, name in checks: # what the tokenizer reads print(f"terms of {example}, {name}:", " ".join(tokenize(normalize_units(example, rules[name])))) searches = {} for name, rule in rules.items(): rewritten = products.copy() if rule is not None: for column in ["name", "product_class", "category", "description"]: # the columns product_text joins rewritten[column] = rewritten[column].map(lambda value: normalize_units(value, rule)) searches[name] = LexicalSearch(rewritten) with_units = queries[queries["query"].str.contains(r"\d\s*(?:qt|in\b|inch|\")")] print("queries with a unit:", len(with_units)) print("of them, test queries:", (with_units["split"] == "test").sum()) for query_id, query in with_units["query"].items(): label = f"{query_id} ({queries.at[query_id, 'split']})" print(f"{label}: {query} -> {normalize_units(query)}") for name, rule in rules.items(): typed = query if rule is None else normalize_units(query, rule) ranking = list(searches[name].search(typed, k=10).index) print(f"{label}, bm25 ndcg@10 {name}: {ndcg_at_k(ranking, judgments[query_id], 10):.3f}") ``` Output: ```text products that write a number then '': 3014 products that write a number then ": 65 products that write a number then inch: 215 terms of 3.7-quart slow cooker, normalized: 3_7qt slow cooker terms of 3 in 1 sofa bed, normalized: sofa bed terms of 1 in twin, normalized: 1in twin terms of 48 '' range hood, first try: 48 range hood terms of 48 '' range hood, normalized: 48in range hood queries with a unit: 4 of them, test queries: 2 62 (train): 7qt slow cooker -> 7qt slow cooker 62 (train), bm25 ndcg@10 as written: 0.554 62 (train), bm25 ndcg@10 first try: 0.930 62 (train), bm25 ndcg@10 normalized: 0.930 101 (train): 48 in entry table with side by side drawer -> 48in entry table with side by side drawer 101 (train), bm25 ndcg@10 as written: 0.569 101 (train), bm25 ndcg@10 first try: 0.569 101 (train), bm25 ndcg@10 normalized: 0.569 285 (test): 48" sliding single track , barn door for laundry -> 48in sliding single track , barn door for laundry 285 (test), bm25 ndcg@10 as written: 0.592 285 (test), bm25 ndcg@10 first try: 0.592 285 (test), bm25 ndcg@10 normalized: 0.592 482 (test): 48 inch kitchen hood -> 48in kitchen hood 482 (test), bm25 ndcg@10 as written: 1.000 482 (test), bm25 ndcg@10 first try: 0.824 482 (test), bm25 ndcg@10 normalized: 1.000 ``` The catalog writes a number followed by two apostrophes in 3014 products, by `"` in 65 and by `inch` in 215. The tokenizer reads `3.7-quart slow cooker`, normalized, as `3_7qt slow cooker`, and `3 in 1 sofa bed` as `sofa bed`. `normalize_units` still misreads `1 in twin` as `1in twin`, which is why the terms are worth printing. For the train query `7qt slow cooker`, normalizing takes BM25's nDCG@10 from 0.554 to 0.930, and the first rule gives 0.930. The test query `48 inch kitchen hood` shows what the first rule loses by missing the two apostrophes: it scores 1.000 as written and 0.824 under the first rule. The query becomes `48in`, while the first rule leaves a hood written `48 ''` as `48 range hood`, and the full rule makes it `48in range hood`. With the full rule, the hood query scores 1.000. The other train query, `48 in entry table with side by side drawer`, scores 0.569 as written and 0.569 normalized, and the test query about a barn door 0.592 and 0.592. That makes 4 queries with a unit, 2 of them [test queries](#store-data): too few for a test, so this is a diagnosis. **What to do:** write normalization rules from the catalog's text, not from the queries, by counting every way it writes each unit. Print the terms the tokenizer makes of the normalized text, because a rule that looks right can produce terms that match the wrong products. Apply the same rules to products and queries in one release, and rebuild the index. When the catalog keeps sizes in fields, such as the width in `attributes`, match the size against the field instead, as [chapter 6 suggests](#codes-carry-no-weight). ## Expand queries with a synonym list {#expand-with-synonyms} Some gaps are neither typos nor units but different words for one thing, as when the exact products for `midcentury tv unit` say `stand`. **Query expansion** adds terms to a query so that it can match products that use other words for the same thing. "Use of query expansion generally increases recall" ([Manning et al., 2008](https://nlp.stanford.edu/IR-book/html/htmledition/query-expansion-1.html)), the share of the relevant products that a search finds. It can cost precision, the share of found products that are relevant: "query expansion may also significantly decrease precision, particularly when the query contains ambiguous terms" ([Manning et al., 2008](https://nlp.stanford.edu/IR-book/html/htmledition/automatic-thesaurus-generation-1.html)). An added word can match products the shopper didn't ask for, and move the query away from its meaning, which is called **query drift**. Expansion terms can come from a hand-built list of synonyms, or from the results. **Pseudo-relevance feedback** assumes the top results are relevant and adds their terms to the query, which makes drift its known risk ([Manning et al., 2008](https://nlp.stanford.edu/IR-book/html/htmledition/pseudo-relevance-feedback-1.html)). In the store, `lib.queries.SYNONYMS` holds 14 groups of interchangeable words, such as `stand` and `unit`, written by reading the train queries whose exact products miss a query word. When a query contains a word or phrase from a group, `expand` appends the group's other members, and BM25 searches the result. Measure it on the train queries, then make the first of the chapter's [five planned comparisons](#understanding-queries) on the test queries, which counts as established if p is below 0.01: Example `synonym-expansion-train-and-test` (Python 3.12.15 · bm25s 0.3.13 · numpy 2.5.4 · pandas 3.0.6 · scikit-learn 1.9.1 · scipy 1.18.1), passing in Edition 2026.10.1: ```python from lib.lexical import LexicalSearch from lib.metrics import evaluate from lib.queries import PLANNED_TEST_COMPARISONS, SYNONYMS, THRESHOLD, expand, report_planned from lib.store import load_products, load_queries, qrels products, judgments = load_products(), qrels() bm25 = LexicalSearch(products) print("synonym groups:", len(SYNONYMS)) scores = {} for split in ["train", "test"]: queries = load_queries(split) expanded = queries.assign(query=queries["query"].map(expand)) changed = queries.index[expanded["query"] != queries["query"]] print(f"{split} queries the list changes:", len(changed)) for name, frame in [("typed", queries), ("expanded", expanded)]: scores[split, name] = evaluate(bm25.run(frame), qrels(split)) print(f"{split} ndcg@10 {name}: {scores[split, name]['ndcg@10'].mean():.3f}") print(f"{split} judged@10 {name}: {scores[split, name]['judged@10'].mean():.3f}") for query in queries.loc[changed, "query"]: # the test queries the list changes print(f"test query changed: {query}") print(f"words the list adds to it: {expand(query)[len(query) + 1:]}") change = (scores["train", "expanded"]["ndcg@10"] - scores["train", "typed"]["ndcg@10"]).round(3) print(f"train queries better, worse: {(change > 0).sum()}, {(change < 0).sum()}") # for choosing, not evidence print(f"share of the net train gain from the largest gain: {change.max() / change.sum():.3f}") for kind, query_id in [("gain", change.idxmax()), ("loss", change.idxmin())]: query = load_queries().at[query_id, "query"] added = expand(query)[len(query) + 1:] print(f"largest train {kind}: {query}") print(f"largest train {kind}, words added: {added}") print(f"largest train {kind}, change: {change[query_id]:+.3f}") named = [product_id for product_id in bm25.search(expand(query), k=10).index if added in products.at[product_id, "name"]] print(f"largest train loss, top 10 names with {added}:", len(named)) # query and added: the largest loss's print("largest train loss, of those, unjudged:", sum(product_id not in judgments[query_id] for product_id in named)) print("\nplanned test comparisons:", len(PLANNED_TEST_COMPARISONS)) print(f"p-value threshold: {THRESHOLD:.2f}") report_planned("synonyms, bm25", scores["test", "expanded"]["ndcg@10"], scores["test", "typed"]["ndcg@10"]) ``` Output: ```text synonym groups: 14 train queries the list changes: 14 train ndcg@10 typed: 0.695 train judged@10 typed: 0.930 train ndcg@10 expanded: 0.722 train judged@10 expanded: 0.944 test queries the list changes: 1 test ndcg@10 typed: 0.725 test judged@10 typed: 0.927 test ndcg@10 expanded: 0.725 test judged@10 expanded: 0.927 test query changed: indoor multi tiered plant stand words the list adds to it: unit train queries better, worse: 6, 5 share of the net train gain from the largest gain: 0.473 largest train gain: midcentury tv unit largest train gain, words added: stand mid century largest train gain, change: +0.632 largest train loss: wire basket with dividers largest train loss, words added: hamper largest train loss, change: -0.176 largest train loss, top 10 names with hamper: 4 largest train loss, of those, unjudged: 4 planned test comparisons: 5 p-value threshold: 0.01 synonyms, bm25, mean difference: 0.000 synonyms, bm25, wins, losses, ties: 0, 0, 30 synonyms, bm25, permutation p: 1.000 synonyms, bm25, established at 0.01: no synonyms, bm25, t-test p: 1.000 synonyms, bm25, t-test agrees: yes synonyms, bm25, 99% interval: 0.000 to 0.000 synonyms, bm25, interval agrees: yes ``` The list changes 14 train queries. Mean [nDCG@10](#score-one-ranking) goes from 0.695 to 0.722, and [judged@10](#score-every-test-query), the share of the top 10 with a judgment, from 0.930 to 0.944. The numbers of changed queries that score better and worse are 6, 5, and one query carries a share of 0.473 of the net gain: `midcentury tv unit`, which gains +0.632 with `stand mid century` added. The list was written from these queries, so even these numbers flatter it. The largest loss, -0.176, shows drift: `wire basket with dividers` gets `hamper`, and 4 products whose names contain `hamper` enter its top 10. 4 of them are unjudged, so the metric counts them as irrelevant, which for hampers is a fair guess. On the test queries, the list changes 1 query, `indoor multi tiered plant stand`, by adding `unit`. Judged@10 stays at 0.927, and the difference isn't established ([planned comparisons](#understanding-queries-decisions)). On the queries it was written from, the list helped about as many queries as it hurt, and queries it wasn't written for rarely contain its words. **What to do:** keep a synonym list for vocabulary gaps you have seen, write each entry from a failing query, and read the products each entry adds before you keep it. Never report a list's gain on the queries you wrote it from, and keep an entry only if a test on held-out queries shows a gain. Test any expansion of semantic retrieval first: the LLM's rewrites, which add synonyms and also broaden and drop words, [hurt it](#rewrite-queries-with-llm). When the gaps are too many to list, try [stemming](#stem-word-forms) or [learned sparse retrieval](#learned-sparse-splade) instead. ## Train a query classifier on product text {#classify-queries-from-products} Instead of changing a query's words, a system can predict what kind of product the query asks for. **Query classification** assigns a query to a class in the store's taxonomy, such as `Range Hoods`. In product search, "mapping a query to a pre-defined product taxonomy is an important step in e-commerce query understanding pipeline" ([Skinner and Kallumadi, 2019](https://ceur-ws.org/Vol-2410/paper37.pdf)): a predicted class can steer retrieval toward the right part of the catalog. Training a classifier usually takes labelled queries. Skinner and Kallumadi report that "an existing product-taxonomy mapping can improve query classification, and reduce the need for labeled data": they pretrain on product titles, whose classes are known, then fine-tune on fewer labelled queries. The store goes further and uses no labelled queries, because it has none. WANDS gives each query a `query_class`, "category to which the query falls under" ([WANDS](https://github.com/wayfair/WANDS)), but [chapter 2](#intent-in-product-search) shows that it is a prediction, and some values are wrong. And the book's sample of WANDS holds at most one query per class (`examples/data/wands/README.md`), so no class has two queries to learn from. So train on the products, which carry `product_class`, and apply the classifier to the queries. A **naive Bayes** classifier can learn classes from product text in a few lines. In words: a class is likely for a query when the class is common among products, and when the query's words are common in that class's product text. It treats words as independent, which is the "naive" part. Each count gets 1 added, so that a word never seen in a class doesn't rule the class out: ```text predicted class = argmax over classes c of log P(c) + sum over the query's words w of log P(w | c) P(w | c) = (count of w in class c's text + 1) / (number of words in class c's text + vocabulary size) ``` In the store, build it by hand from word counts of the product text, check it against scikit-learn's `MultinomialNB` at 1.9.1, then compare its predictions with `query_class` and with the classes of each query's exact products: Example `query-classes-by-hand` (Python 3.12.15 · numpy 2.5.4 · pandas 3.0.6 · scikit-learn 1.9.1 · scipy 1.18.1), passing in Edition 2026.10.1: ```python import numpy as np from lib.queries import class_rows, train_classifier from lib.store import load_labels, load_products, load_queries, product_text products, queries, labels = load_products(), load_queries(), load_labels() rows = class_rows(products) # one row per (product, class) for products with a class vectorizer, model = train_classifier(products) # the same, by scikit-learn's MultinomialNB words = vectorizer.transform(product_text(rows)) # word counts: one row per (product, class), one column per word classes = np.array(sorted(rows["label"].unique())) print("product classes:", len(classes)) print("products without a class:", (products["product_class"] == "").sum()) print("products with several classes:", products["product_class"].str.contains("|", regex=False).sum()) missing = queries["query_class"].dropna()[lambda names: ~names.isin(classes)] print("query classes that no product has:", len(missing)) print("query classes that no product has, names:", "; ".join(missing)) # Naive Bayes: log P(class) + sum over the query's words of log P(word | class), each word count smoothed by 1 word_counts = np.vstack([np.asarray(words[(rows["label"] == name).to_numpy()].sum(axis=0)) for name in classes]) log_prior = np.log(rows["label"].value_counts()[classes].to_numpy() / len(rows)) log_likelihood = np.log((word_counts + 1) / (word_counts.sum(axis=1, keepdims=True) + word_counts.shape[1])) query_words = vectorizer.transform(queries["query"]) predicted = classes[np.asarray(query_words @ log_likelihood.T + log_prior).argmax(axis=1)] print("same predictions as MultinomialNB:", (predicted == model.predict(query_words)).sum(), "of", len(queries)) queries = queries.assign(predicted=predicted) print("predicted class equals query_class:", (queries["predicted"] == queries["query_class"]).sum()) exact = labels[labels["grade"] == 2].join(products["product_class"].str.split("|"), on="product_id") for column, name in [("predicted", "predicted class"), ("query_class", "query_class")]: share = [exact.loc[exact["query_id"] == query_id, "product_class"].apply(lambda names: label in names).mean() for query_id, label in queries[column].items()] print(f"queries where most exact products have the {name}:", (np.array(share) > 0.5).sum()) for query_id in [133, 255, 410]: query, split, label = queries.loc[query_id, ["query", "split", "predicted"]] print(f"{query_id} ({split}): {query} -> {label}" f" (query_class: {queries.at[query_id, 'query_class']})") print("predicted class for 7qt slow cooker:", queries.at[62, "predicted"]) # its query_class has no products ``` Output: ```text product classes: 410 products without a class: 800 products with several classes: 620 query classes that no product has: 1 query classes that no product has, names: Crock Pots & Slow Cookers same predictions as MultinomialNB: 80 of 80 predicted class equals query_class: 60 queries where most exact products have the predicted class: 63 queries where most exact products have the query_class: 63 133 (test): outdoor light fixtures -> Light Bulbs (query_class: Outdoor Wall Lights) 255 (test): desk and chair set -> Desks (query_class: Office Chairs) 410 (train): pool floats -> Deck Boxes (query_class: Lawn Games) predicted class for 7qt slow cooker: Slow Cookers ``` The products use 410 classes. 800 products have none and are left out, and 620 have several, so each of those is one training row per class. The hand-built classifier makes the same prediction as `MultinomialNB` for 80 of 80 queries. Its prediction equals `query_class` for 60, but `query_class` is a guess too. Against the judgments, the two guesses tie: for 63 queries most exact products carry the predicted class, and for 63 they carry `query_class`. Three of its predictions show how it reads a query by its words alone: `outdoor light fixtures` becomes `Light Bulbs`, `desk and chair set` becomes `Desks`, and `pool floats` becomes `Deck Boxes`. The sample's design limits what this check can show. Each class is tested by at most one query, so no per-class accuracy can be measured, and 80 queries cover only part of the 410 classes, so the counts say nothing about the rest. A query classifier can't be trained on these queries at all, and two classifiers can only be compared on whether each gets these few queries right. And 1 query class, `Crock Pots & Slow Cookers`, names no product class at all, so no classifier trained on products can predict it. **What to do:** when you have no labelled queries, train a query classifier on your products' classes and check it against the classes of judged exact products, not against another classifier's guesses. Once your logs give you labelled queries, train on those. Treat the predicted class as a hint for ranking, never as a label. ## Boost the predicted class instead of filtering {#boost-predicted-class} In the store, the predicted class comes from [the naive Bayes classifier](#classify-queries-from-products) trained on product text. A predicted class can act on retrieval in two ways. A **filter** keeps only products in the class. A **boost** adds a fixed amount to their scores, so that they move up without anything else being removed. A filter is only as good as the classifier: every relevant product outside the predicted class disappears. The store's train judgments show how much a filter would cost: for the train queries, 501 of 1937 exact products lie outside their query's predicted class. The example adds a boost to the similarities of semantic retrieval, the store's best ranker and so its [ranker to beat](#compare-runs-in-a-harness), and tries five boost weights and a filter on the train queries. It takes the best boost to the test queries for the second of the chapter's [five planned comparisons](#understanding-queries), which counts as established if p is below 0.01: Example `class-boost-train-and-test` (Python 3.12.15 · numpy 2.5.4 · pandas 3.0.6 · scikit-learn 1.9.1 · scipy 1.18.1), passing in Edition 2026.10.1: ```python from lib.metrics import evaluate from lib.queries import THRESHOLD, class_boost_run, in_class, report_planned, train_classifier from lib.semantic import SemanticSearch from lib.store import load_products, load_queries, qrels products, queries = load_products(), load_queries() vectorizer, model = train_classifier(products) # trained on product text, never on queries predicted = dict(zip(queries.index, model.predict(vectorizer.transform(queries["query"])))) semantic = SemanticSearch() train, train_judgments = load_queries("train"), qrels("train") exact, outside = 0, 0 for query_id, judged in train_judgments.items(): member = in_class(products, predicted[query_id]) exact += sum(grade == 2 for grade in judged.values()) outside += sum(grade == 2 and not member[product_id] for product_id, grade in judged.items()) print(f"exact train products outside the predicted class: {outside} of {exact}") base = evaluate(semantic.run(train), train_judgments) print(f"train ndcg@10 semantic: {base['ndcg@10'].mean():.3f}") print(f"train judged@10 semantic: {base['judged@10'].mean():.3f}") train_scores = {} for weight in [None, 0.01, 0.02, 0.05, 0.1, 0.2]: # None: keep only the predicted class train_scores[weight] = evaluate(class_boost_run(semantic, train.index, predicted, products, weight), train_judgments) name = "filter" if weight is None else f"boost {weight}" print(f"train ndcg@10 {name}: {train_scores[weight]['ndcg@10'].mean():.3f}") boosts = [weight for weight in train_scores if weight is not None] best = max(boosts, key=lambda weight: train_scores[weight]["ndcg@10"].mean()) for name, weight in [("filter", None), ("best boost", best)]: print(f"train judged@10 {name}: {train_scores[weight]['judged@10'].mean():.3f}") print("best boost on train:", best) print(f"\nthe planned test comparison, threshold {THRESHOLD:.2f}:") test, test_judgments = load_queries("test"), qrels("test") boosted = evaluate(class_boost_run(semantic, test.index, predicted, products, best), test_judgments) plain = evaluate(semantic.run(test), test_judgments) print(f"test judged@10 semantic: {plain['judged@10'].mean():.3f}") print(f"test judged@10 boosted: {boosted['judged@10'].mean():.3f}") report_planned("class boost, semantic", boosted["ndcg@10"], plain["ndcg@10"]) ``` Output: ```text exact train products outside the predicted class: 501 of 1937 train ndcg@10 semantic: 0.772 train judged@10 semantic: 0.964 train ndcg@10 filter: 0.675 train ndcg@10 boost 0.01: 0.773 train ndcg@10 boost 0.02: 0.774 train ndcg@10 boost 0.05: 0.768 train ndcg@10 boost 0.1: 0.742 train ndcg@10 boost 0.2: 0.692 train judged@10 filter: 0.866 train judged@10 best boost: 0.966 best boost on train: 0.02 the planned test comparison, threshold 0.01: test judged@10 semantic: 0.990 test judged@10 boosted: 0.993 class boost, semantic, mean difference: 0.005 class boost, semantic, wins, losses, ties: 7, 4, 19 class boost, semantic, permutation p: 0.174 class boost, semantic, established at 0.01: no class boost, semantic, t-test p: 0.164 class boost, semantic, t-test agrees: yes class boost, semantic, 99% interval: -0.002 to 0.017 class boost, semantic, interval agrees: yes ``` On the train queries, the filter takes [nDCG@10](#score-one-ranking) from 0.772 to 0.675, and [judged@10](#score-every-test-query) from 0.964 to 0.866, so its top 10s also fill with products nobody judged. The boosts score from 0.773 at 0.01 and 0.774 at 0.02 down to 0.692 at 0.2, and the output lists the rest. The best boost on train is 0.02, with judged@10 of 0.966. On the test queries, judged@10 is 0.990 without the boost and 0.993 with it, and the boost's gain isn't established ([planned comparisons](#understanding-queries-decisions)). **What to do:** don't filter on a predicted class. Boost it with a weight tuned on train queries, and keep the boost only if its planned test comparison is established. The store's isn't, so its semantic retrieval stays unboosted. We recommend a class filter only when the shopper chose the class, for example by clicking a category. ## Rewrite queries with an LLM {#rewrite-queries-with-llm} **Query rewriting** replaces the query with a text written to retrieve better. Where the corrector, the unit rules and the synonym list each fix one kind of gap, an LLM can fix spelling, expand abbreviations and add synonyms in one step, with no list to maintain. In rewrite-retrieve-read, the rewritten query goes to the retriever in place of the original ([Ma et al., 2023](https://arxiv.org/abs/2305.14283)). Studies report gains. With query2doc, which appends an LLM-written passage, a **pseudo-document**, to the query, BM25 improved "by 3% to 15% on ad-hoc IR datasets", with text-davinci-003 writing the passages for MS MARCO and TREC DL queries ([Wang et al., 2023](https://arxiv.org/abs/2303.07678)). Other studies report failures. A comparison of 11 expansion methods, 12 datasets and 24 retrievers found that "expansion improves scores for weaker models, but generally harms stronger models" ([Weller et al., 2024](https://arxiv.org/abs/2309.08541)). Another study found that LLM-based expansion "can significantly degrade the retrieval effectiveness when knowledge in the LLM is insufficient or query ambiguity is high" ([Abe et al., 2025](https://arxiv.org/abs/2505.12694)). The rewrites that the examples read were recorded with the store's LLM, the open model named in [the chapter's opening](#understanding-queries), run locally on each query alone. The prompt's instruction is: "Rewrite this search query from an online furniture and home-goods store so that a search engine finds what the shopper wants. Fix spelling, expand abbreviations and add common synonyms for the product type, but keep every requirement the shopper stated and add none of your own. Return a single line." The rest of the prompt gives the query and asks for JSON, and the recording's manifest holds it in full. The recording script embedded each rewrite as a query, with the same embedding model as the products. BM25 searches the rewrite's text, and semantic retrieval its embedding. Look at the train queries first: Example `llm-rewrites-on-train` (Python 3.12.15 · bm25s 0.3.13 · numpy 2.5.4 · pandas 3.0.6 · scipy 1.18.1), passing in Edition 2026.10.1: ```python from lib.lexical import LexicalSearch from lib.metrics import evaluate from lib.recorded import llm_rewrites, manifest from lib.semantic import SemanticSearch from lib.store import load_products, load_queries, qrels recording = manifest("llm-rewrites") print("model:", recording["model"]) print("embedding model:", recording["embedding_model"]) print("rewrite prompt, first line:", recording["prompts"]["rewrite"].split("\n")[0]) table, rewrite_vectors, _ = llm_rewrites() rewrites = table.set_index("query_id")["rewrite"] row = {query_id: position for position, query_id in enumerate(table["query_id"])} for query_id in [20, 205, 313]: print(f"rewrite of {load_queries().at[query_id, 'query']}: {rewrites[query_id]}") queries, judgments = load_queries("train"), qrels("train") products = load_products() exact = [product_id for product_id, grade in judgments[313].items() if grade == 2] print("exact products for malachi sled:", "; ".join(products.loc[exact, "name"])) bm25, semantic = LexicalSearch(products), SemanticSearch() runs = {"bm25, typed": bm25.run(queries), "bm25, rewrite": bm25.run(queries.assign(query=rewrites[queries.index].to_numpy())), "semantic, typed": semantic.run(queries), "semantic, rewrite": {query_id: list(semantic.search_vector(rewrite_vectors[row[query_id]]).index) for query_id in queries.index}} scores = {name: evaluate(run, judgments) for name, run in runs.items()} for name, frame in scores.items(): print(f"train ndcg@10 {name}: {frame['ndcg@10'].mean():.3f}") print(f"train judged@10 {name}: {frame['judged@10'].mean():.3f}") for retriever in ["bm25", "semantic"]: # for choosing, not evidence change = (scores[f"{retriever}, rewrite"]["ndcg@10"] - scores[f"{retriever}, typed"]["ndcg@10"]).round(3) print(f"train queries better, worse, {retriever}: {(change > 0).sum()}, {(change < 0).sum()}") print(f"largest train loss, {retriever}: {queries.at[change.idxmin(), 'query']}") print(f"largest train loss, {retriever}, change: {change.min():+.3f}") ``` Output: ```text model: gemma4:12b-it-q4_K_M embedding model: Snowflake/snowflake-arctic-embed-m-v1.5 rewrite prompt, first line: Rewrite this search query from an online furniture and home-goods store so that a search engine finds what the shopper wants. Fix spelling, expand abbreviations and add common synonyms for the product type, but keep every requirement the shopper stated and add none of your own. Return a single line. rewrite of foutains with brick look: fountains with brick look, brick style water features, masonry fountains, faux brick fountains rewrite of ines shag rug w003096809: Ines shag rug w003096809 area rug carpet rewrite of malachi sled: Malachi sled sofa couch settee loveseat exact products for malachi sled: malachi sled end table; malachi 44 '' console table; malachi sled coffee table train ndcg@10 bm25, typed: 0.695 train judged@10 bm25, typed: 0.930 train ndcg@10 bm25, rewrite: 0.646 train judged@10 bm25, rewrite: 0.868 train ndcg@10 semantic, typed: 0.772 train judged@10 semantic, typed: 0.964 train ndcg@10 semantic, rewrite: 0.759 train judged@10 semantic, rewrite: 0.924 train queries better, worse, bm25: 23, 24 largest train loss, bm25: malachi sled largest train loss, bm25, change: -0.644 train queries better, worse, semantic: 18, 25 largest train loss, semantic: wall design shelf largest train loss, semantic, change: -0.653 ``` The rewrites fix spelling, `fountains with brick look, brick style water features, masonry fountains, faux brick fountains`, and keep codes, `Ines shag rug w003096809 area rug carpet`. For `malachi sled`, a product line, the LLM guessed a sofa: it wrote `Malachi sled sofa couch settee loveseat`, while the exact products are `malachi sled end table; malachi 44 '' console table; malachi sled coffee table`. On the train queries, BM25's nDCG@10 goes from 0.695 to 0.646, and semantic retrieval's from 0.772 to 0.759. [Judged@10](#score-every-test-query) falls for both, from 0.930 to 0.868 and from 0.964 to 0.924, so the rewrites surface products nobody judged, and part of each fall in nDCG@10 may come from them. The numbers of queries that score better and worse are 23, 24 for BM25 and 18, 25 for semantic retrieval. BM25's largest loss is `malachi sled`, at -0.644, and semantic retrieval's is `wall design shelf`, at -0.653. These train numbers shape expectations and prove nothing. The third and fourth of the chapter's [five planned comparisons](#understanding-queries) were fixed in advance, so they go to the test queries whatever the train queries show, and each counts as established if p is below 0.01. Before you run them, predict: on the test queries, will the rewrite help BM25, the weaker retriever, hurt semantic retrieval, the stronger one, or neither? Example `llm-rewrites-on-test` (Python 3.12.15 · bm25s 0.3.13 · numpy 2.5.4 · pandas 3.0.6 · scikit-learn 1.9.1 · scipy 1.18.1), passing in Edition 2026.10.1: ```python from lib.lexical import LexicalSearch from lib.metrics import evaluate from lib.queries import THRESHOLD, condensed, report_planned, rewrite_runs from lib.semantic import SemanticSearch from lib.significance import compare from lib.store import load_products, load_queries, qrels queries, judgments = load_queries("test"), qrels("test") # scored on the human judgments runs = rewrite_runs(queries, LexicalSearch(load_products()), SemanticSearch()) print(f"the planned test comparisons, threshold {THRESHOLD:.2f}:") for retriever, (rewritten, typed) in runs.items(): name = f"llm rewrite, {retriever}" after, before = evaluate(rewritten, judgments), evaluate(typed, judgments) for form, frame in [("typed", before), ("rewrite", after)]: print(f"{name}, test ndcg@10 {form}: {frame['ndcg@10'].mean():.3f}") print(f"{name}, test judged@10 {form}: {frame['judged@10'].mean():.3f}") report_planned(name, after["ndcg@10"], before["ndcg@10"]) # Exploratory, never established: the same comparison on condensed runs, which drop unjudged products after, before = (evaluate(condensed(run, judgments), judgments)["ndcg@10"] for run in (rewritten, typed)) result = compare(after, before, confidence=1 - THRESHOLD) print(f"{name}, condensed ndcg@10 typed: {before.mean():.3f}") print(f"{name}, condensed ndcg@10 rewrite: {after.mean():.3f}") print(f"{name}, condensed, exploratory mean difference: {result['mean difference']:.3f}") print(f"{name}, condensed, exploratory permutation p: {result['permutation p']:.3f}") low, high = result["ci low"], result["ci high"] print(f"{name}, condensed, exploratory {1 - THRESHOLD:.0%} interval: {low:.3f} to {high:.3f}") ``` Output: ```text the planned test comparisons, threshold 0.01: llm rewrite, bm25, test ndcg@10 typed: 0.725 llm rewrite, bm25, test judged@10 typed: 0.927 llm rewrite, bm25, test ndcg@10 rewrite: 0.669 llm rewrite, bm25, test judged@10 rewrite: 0.903 llm rewrite, bm25, mean difference: -0.056 llm rewrite, bm25, wins, losses, ties: 13, 15, 2 llm rewrite, bm25, permutation p: 0.060 llm rewrite, bm25, established at 0.01: no llm rewrite, bm25, t-test p: 0.060 llm rewrite, bm25, t-test agrees: yes llm rewrite, bm25, 99% interval: -0.143 to 0.005 llm rewrite, bm25, interval agrees: yes llm rewrite, bm25, condensed ndcg@10 typed: 0.778 llm rewrite, bm25, condensed ndcg@10 rewrite: 0.725 llm rewrite, bm25, condensed, exploratory mean difference: -0.053 llm rewrite, bm25, condensed, exploratory permutation p: 0.044 llm rewrite, bm25, condensed, exploratory 99% interval: -0.150 to -0.002 llm rewrite, semantic, test ndcg@10 typed: 0.830 llm rewrite, semantic, test judged@10 typed: 0.990 llm rewrite, semantic, test ndcg@10 rewrite: 0.749 llm rewrite, semantic, test judged@10 rewrite: 0.947 llm rewrite, semantic, mean difference: -0.081 llm rewrite, semantic, wins, losses, ties: 7, 19, 4 llm rewrite, semantic, permutation p: 0.000 llm rewrite, semantic, established at 0.01: yes llm rewrite, semantic, t-test p: 0.000 llm rewrite, semantic, t-test agrees: yes llm rewrite, semantic, 99% interval: -0.132 to -0.035 llm rewrite, semantic, interval agrees: yes llm rewrite, semantic, condensed ndcg@10 typed: 0.834 llm rewrite, semantic, condensed ndcg@10 rewrite: 0.791 llm rewrite, semantic, condensed, exploratory mean difference: -0.043 llm rewrite, semantic, condensed, exploratory permutation p: 0.002 llm rewrite, semantic, condensed, exploratory 99% interval: -0.079 to -0.014 ``` For BM25, judged@10 falls from 0.927 to 0.903, and [nDCG@10](#score-one-ranking) from 0.725 to 0.669, a difference that isn't established ([planned comparisons](#understanding-queries-decisions)). For semantic retrieval, judged@10 falls from 0.990 to 0.947, and nDCG@10 from 0.830 to 0.749, an established loss, with a permutation p-value that rounds to 0.000. Because judged@10 fell, the example repeats both comparisons on [condensed](#unjudged-results) lists, which drop unjudged products before scoring. That check was chosen after seeing the drop, so it is exploratory and can't establish anything. For semantic retrieval, condensed nDCG@10 goes from 0.834 to 0.791, a difference of -0.043 with a permutation p of 0.002 and a 99% interval from -0.079 to -0.014. So the loss doesn't rest on unjudged products alone. For BM25, condensed nDCG@10 goes from 0.778 to 0.725, with a permutation p of 0.044, a question for new queries rather than a finding. If you predicted that the rewrite would help BM25, the test doesn't show it, on these queries with this LLM and prompt: BM25 moved the other way from what Weller et al. and query2doc report, though not by an established amount. If you predicted harm to semantic retrieval, you predicted what Weller et al. found for strong retrievers, here with this model, this prompt and these queries. The queries that move most show what a rewrite does. Read them with the number of unjudged products in each rewritten top 10, which the metric counts as irrelevant: Example `rewrite-largest-changes` (Python 3.12.15 · bm25s 0.3.13 · numpy 2.5.4 · pandas 3.0.6 · scikit-learn 1.9.1 · scipy 1.18.1), passing in Edition 2026.10.1: ```python from bm25s.stopwords import STOPWORDS_EN from lib.lexical import LexicalSearch from lib.metrics import evaluate, ndcg_at_k from lib.queries import rewrite_runs from lib.recorded import llm_rewrites, probes from lib.semantic import SemanticSearch from lib.store import load_products, load_queries, qrels queries, judgments = load_queries("test"), qrels("test") bm25 = LexicalSearch(load_products()) rewrites = llm_rewrites()[0].set_index("query_id")["rewrite"] largest = {} for retriever, (rewritten, typed) in rewrite_runs(queries, bm25, SemanticSearch()).items(): change = evaluate(rewritten, judgments)["ndcg@10"] - evaluate(typed, judgments)["ndcg@10"] for kind, query_id in [("loss", change.idxmin()), ("gain", change.idxmax())]: label, largest[(retriever, kind)] = f"{retriever}, largest {kind}", query_id print(f"{label}: {queries.at[query_id, 'query']}") print(f"{label}, change: {change[query_id]:+.3f}") print(f"{label}, rewrite: {rewrites[query_id]}") unjudged = sum(product_id not in judgments[query_id] for product_id in rewritten[query_id][:10]) print(f"{label}, unjudged products in the rewrite's top 10: {unjudged}") # BM25's largest gain: score the query with only its spelling fixed (chapter 6's probe), apart from the added words query_id = largest[("bm25", "gain")] spelled = probes()[0].dropna(subset=["of_query_id"]).set_index("of_query_id")["query"] if query_id in spelled.index: for name, text in [("typed", queries.at[query_id, "query"]), ("spelling fixed only", spelled[query_id])]: ranking = list(bm25.search(text, k=10).index) print(f"bm25, largest gain, ndcg@10 {name}: {ndcg_at_k(ranking, judgments[query_id], 10):.3f}") print("bm25 stop words among an and and:", ", ".join(word for word in ["an", "and"] if word in STOPWORDS_EN)) ``` Output: ```text bm25, largest loss: fold in half table bm25, largest loss, change: -0.471 bm25, largest loss, rewrite: folding table, collapsible table, drop leaf table, gateleg table bm25, largest loss, unjudged products in the rewrite's top 10: 3 bm25, largest gain: refrigerator with ice an water in door bm25, largest gain, change: +0.159 bm25, largest gain, rewrite: refrigerator with ice and water in door, fridge with door dispenser, refrigerator with external water and ice dispenser bm25, largest gain, unjudged products in the rewrite's top 10: 1 semantic, largest loss: pull out sleeper loveseat semantic, largest loss, change: -0.304 semantic, largest loss, rewrite: pull out sleeper loveseat sofa couch convertible sofa sofa bed semantic, largest loss, unjudged products in the rewrite's top 10: 4 semantic, largest gain: nautical platters semantic, largest gain, change: +0.055 semantic, largest gain, rewrite: nautical platters serving trays seafood platters maritime serving boards semantic, largest gain, unjudged products in the rewrite's top 10: 0 bm25, largest gain, ndcg@10 typed: 0.501 bm25, largest gain, ndcg@10 spelling fixed only: 0.501 bm25 stop words among an and and: an, and ``` BM25's largest loss, -0.471 for `fold in half table`, comes from `folding table, collapsible table, drop leaf table, gateleg table`, which drops `in half` and adds other kinds of table; 3 of its top 10 are unjudged. Semantic retrieval's largest loss, -0.304 for `pull out sleeper loveseat`, comes from `pull out sleeper loveseat sofa couch convertible sofa sofa bed`, which widens a loveseat to any sofa; 4 of its top 10 are unjudged. BM25's largest gain, +0.159 for `refrigerator with ice an water in door`, comes from `refrigerator with ice and water in door, fridge with door dispenser, refrigerator with external water and ice dispenser`. The rewrite fixes `an`, but with only that fix BM25 scores 0.501, the same as the query as typed, 0.501, because both spellings are stop words. The gain comes from the words the rewrite adds. Semantic retrieval's largest gain is +0.055, for `nautical platters`, rewritten as `nautical platters serving trays seafood platters maritime serving boards`. **What to do:** don't put an LLM rewrite in front of a strong retriever without a planned test on held-out queries. Read the per-query losses for dropped requirements and broadened products, which the prompt's "keep every requirement" doesn't prevent. For the store, semantic retrieval, the ranker to beat, keeps searching the query as typed. ## Search with a hypothetical product (HyDE) {#hyde-hypothetical-products} Where a [rewrite](#rewrite-queries-with-llm) asks the LLM for a better query, HyDE, short for hypothetical document embeddings, asks it for the document it imagines would answer the query. An encoder embeds that **hypothetical document**, and "This vector identifies a neighborhood in the corpus embedding space" where real documents lie ([Gao et al., 2023](https://aclanthology.org/2023.acl-long.99/)). The made-up details needn't be true: the authors argue that the encoder's "dense bottleneck" filters out "the hallucinations", which is their argument, not a measurement. Their main results use InstructGPT (text-davinci-003), sampling at temperature 0.7. On TREC DL19, a passage-ranking benchmark, HyDE raised the nDCG@10 of Contriever, an embedding model trained without relevance labels, from 44.5 to 61.3 on a 0 to 100 scale (Gao et al., Table 1). With a Contriever fine-tuned on labels, the result depended on the LLM. The smaller ones lowered nDCG@10 slightly, since "less powerful instruction LMs can negatively impact the overall performance of the fine-tuned retriever", while InstructGPT raised it, most on DL19, from 62.1 to 67.4 (Table 4). Several expansion methods keep the original query in the mix. HyDE averages the embeddings of the hypothetical documents together with the query's own (Gao et al., section 3.2). **Relevance feedback** improves a query from judgments of its first results, and Rocchio's version moves the query vector toward the results judged relevant while keeping the original vector in its sum ([Manning et al., 2008](https://nlp.stanford.edu/IR-book/html/htmledition/the-rocchio71-algorithm-1.html)). For BM25, query2doc repeats the query five times before the generated passage, so that the original terms keep their weight ([Wang et al., 2023](https://arxiv.org/abs/2303.07678)). For the store, the HyDE recording holds one hypothetical product per query, which the store's LLM wrote from the query alone: a name, a class and a description, embedded as a product. The example's `hyde` function averages it with the query's embedding and scales the result back to length 1, as [chapter 6 advises after any averaging](#compare-with-cosine-similarity). The two embeddings differ in one way: the query's carries the embedding model's query prompt, and the product's has none, as for every product. ```text hyde(query) = (embedding(hypothetical product) + embedding(query)) / 2, then scaled to length 1 ``` On the train queries, the example also tries the hypothetical product's embedding alone, and, for BM25, the hypothetical product's text alone and with the query repeated five times before it, as query2doc weights it. Then it makes the last of the chapter's [five planned comparisons](#understanding-queries), which counts as established if p is below 0.01: Example `hyde-train-and-test` (Python 3.12.15 · bm25s 0.3.13 · numpy 2.5.4 · pandas 3.0.6 · scikit-learn 1.9.1 · scipy 1.18.1), passing in Edition 2026.10.1: ```python import numpy as np from lib.lexical import LexicalSearch from lib.metrics import evaluate from lib.queries import THRESHOLD, report_planned from lib.recorded import llm_rewrites from lib.semantic import SemanticSearch from lib.store import load_products, load_queries, qrels table, _, hypothetical_vectors = llm_rewrites() hypothetical, vectors = table.set_index("query_id")["hypothetical"], dict(zip(table["query_id"], hypothetical_vectors)) bm25, semantic = LexicalSearch(load_products()), SemanticSearch() def hyde(query_id): """Gao et al.'s average of the hypothetical product's embedding and the query's, scaled back to length 1.""" average = (vectors[query_id] + semantic.query_vectors[query_id]) / 2 return average / np.linalg.norm(average) def runs(queries): ids, typed = queries.index, queries["query"] return {"semantic": semantic.run(queries), "hypothetical alone": {query_id: list(semantic.search_vector(vectors[query_id]).index) for query_id in ids}, "hyde": {query_id: list(semantic.search_vector(hyde(query_id)).index) for query_id in ids}, "bm25": bm25.run(queries), "bm25, hypothetical alone": bm25.run(queries.assign(query=ids.map(hypothetical))), "bm25, query five times plus hypothetical": bm25.run(queries.assign( # query2doc's weighting query=[" ".join([typed[query_id]] * 5 + [hypothetical[query_id]]) for query_id in ids]))} train = {name: evaluate(run, qrels("train")) for name, run in runs(load_queries("train")).items()} for name, frame in train.items(): print(f"train ndcg@10 {name}: {frame['ndcg@10'].mean():.3f}") print(f"train judged@10 {name}: {frame['judged@10'].mean():.3f}") change = (train["hypothetical alone"]["ndcg@10"] - train["semantic"]["ndcg@10"]).round(3) # for choosing, not evidence print(f"train queries better, worse, hypothetical alone: {(change > 0).sum()}, {(change < 0).sum()}") print(f"\nthe planned test comparison, threshold {THRESHOLD:.2f}:") test = {name: evaluate(run, qrels("test")) for name, run in runs(load_queries("test")).items() if name in ["semantic", "hyde"]} for name, frame in test.items(): print(f"test judged@10 {name}: {frame['judged@10'].mean():.3f}") report_planned("hyde, semantic", test["hyde"]["ndcg@10"], test["semantic"]["ndcg@10"]) ``` Output: ```text train ndcg@10 semantic: 0.772 train judged@10 semantic: 0.964 train ndcg@10 hypothetical alone: 0.704 train judged@10 hypothetical alone: 0.890 train ndcg@10 hyde: 0.769 train judged@10 hyde: 0.946 train ndcg@10 bm25: 0.695 train judged@10 bm25: 0.930 train ndcg@10 bm25, hypothetical alone: 0.644 train judged@10 bm25, hypothetical alone: 0.854 train ndcg@10 bm25, query five times plus hypothetical: 0.694 train judged@10 bm25, query five times plus hypothetical: 0.912 train queries better, worse, hypothetical alone: 13, 29 the planned test comparison, threshold 0.01: test judged@10 semantic: 0.990 test judged@10 hyde: 0.970 hyde, semantic, mean difference: -0.031 hyde, semantic, wins, losses, ties: 8, 14, 8 hyde, semantic, permutation p: 0.108 hyde, semantic, established at 0.01: no hyde, semantic, t-test p: 0.121 hyde, semantic, t-test agrees: yes hyde, semantic, 99% interval: -0.115 to 0.001 hyde, semantic, interval agrees: yes ``` On the train queries, the hypothetical product alone takes semantic retrieval's nDCG@10 from 0.772 to 0.704, with [judged@10](#score-every-test-query) from 0.964 to 0.890, and the numbers of queries that score better and worse are 13, 29. Averaged with the query, it scores 0.769, with judged@10 of 0.946. BM25 behaves the same way: 0.695 on the query, 0.644 on the hypothetical text alone, and 0.694 with the query repeated five times before it, with judged@10 of 0.930, 0.854 and 0.912. Keeping the query recovers what the hypothetical product loses, and on the train queries, for either retriever, the combination adds nothing over the query alone. Train numbers guide the design; only the planned test comparison is evidence. On the test queries, judged@10 goes from 0.990 to 0.970, and HyDE shows no established difference from semantic retrieval ([planned comparisons](#understanding-queries-decisions)). This result belongs to one hypothetical product per query, from a 12-billion-parameter model run at temperature 0. Gao et al. sampled from a far larger LLM, and only that one raised their fine-tuned retriever. **What to do:** when you try HyDE, always average the query's own embedding in. Expect the largest gains with an encoder trained without labels; with a fine-tuned encoder, Gao et al. saw gains only from their largest LLM, and the store, with its model, prompt and encoder, saw none. On an encoder like the store's, which [already beats BM25](#compare-runs-in-a-harness) on these queries, test HyDE before you pay for an LLM call per query. The store doesn't use it. ## Turn a follow-up into a standalone query {#standalone-follow-ups} The last gap from [the chapter's opening](#understanding-queries) comes from conversation: a shopper's next message often depends on the last one, as `in white` refines a search for a desk. Retrieval sees one text, so the follow-up has to become a **standalone query**, one that keeps what the shopper still wants. Conversational query rewriting is the task of turning "a context-dependent into a self-contained question with the same answer", and the CANARD dataset holds 40,527 such rewrites ([Elgohary et al., 2019](https://aclanthology.org/D19-1605/)). In TREC CAsT 2019, a conversational search evaluation, people's rewrites beat the best automatic system, "a 35% relative improvement of manual rewrites over the best automatic system" ([Dalton et al., 2020](https://arxiv.org/abs/2003.13624)). Later work with prompted LLMs found that "informative query rewrites can yield substantially improved retrieval performance compared to human rewrites, especially with sparse retrievers" ([Ye et al., 2023](https://arxiv.org/abs/2310.09716)), and in TREC iKAT 2025, CAsT's successor, "Most of the runs leveraged LLMs in their pipelines, for single or multiple query rewriting" ([TREC iKAT 2025 overview](https://trec.nist.gov/pubs/trec34/papers/Overview_ikat.pdf)). WANDS has no conversations, so we wrote the store's conversations for the book: 12 follow-ups, each refining a test query by one attribute, a color, shape, size or finish. The store's LLM, which saw only the two turns, rewrote each to stand alone. For example, after `coat rack for bedroom` comes `something in black`, rewritten as `black coat rack for bedroom`. The judgments grade products for the first query, not for the refined one, so nDCG can't score the follow-up. Instead, count three kinds of product in each top 10: those that keep the refinement, because their `attributes` or text contain the value, such as `black` (the example also prints the count from `attributes` alone); those of the right kind, which the judges graded [exact or partial](#store-data) for the first query; and those that are both. The example compares the follow-up alone, the two turns joined and the standalone query, each embedded as a query, with the first query as a baseline. It adds a filter on the standalone query's top 100 that keeps products stating the value, taking the value from the recorded conversation; a live system would have to extract it from the follow-up. Example `follow-up-refinements` (Python 3.12.15 · numpy 2.5.4 · pandas 3.0.6 · scikit-learn 1.9.1 · scipy 1.18.1), passing in Edition 2026.10.1: ```python import numpy as np from lib.queries import mentions from lib.recorded import llm_conversations from lib.semantic import SemanticSearch from lib.store import load_products, load_queries, product_text, qrels products, judgments, semantic = load_products(), qrels("test"), SemanticSearch() attributes, text = products["attributes"], product_text(products) # the embeddings read only the text conversations, vectors = llm_conversations() # follow-ups we wrote for the book, not shoppers' print("conversations:", len(conversations)) example = conversations.iloc[4] print(f"first query: {load_queries().at[example['query_id'], 'query']}") print(f"follow-up: {example['follow_up']}") print(f"standalone: {example['standalone']}") shares, filtered_lengths = {}, [] # shares: (top 10, measure) -> one share per conversation for row, (first, value) in enumerate(zip(conversations["query_id"], conversations["value"])): standalone = semantic.search_vector(vectors["standalone"][row], k=100).index stated = mentions(standalone, value, attributes) | mentions(standalone, value, text) tops = {"first query": semantic.search(first, k=10).index, "follow-up alone": semantic.search_vector(vectors["follow_up"][row], k=10).index, "turns joined": semantic.search_vector(vectors["joined"][row], k=10).index, "standalone": standalone[:10], "standalone, filtered": standalone[stated][:10]} # given the value, which a live system extracts filtered_lengths.append(len(tops["standalone, filtered"])) for name, top in tops.items(): in_attributes = mentions(top, value, attributes) in_either = in_attributes | mentions(top, value, text) right_kind = np.array([judgments[first].get(product_id, 0) >= 1 for product_id in top], dtype=bool) for measure, hits in [("refinement in attributes", in_attributes), ("refinement in text or attributes", in_either), ("right kind", right_kind), ("right kind with the refinement", in_either & right_kind)]: shares.setdefault((name, measure), []).append(hits.sum() / 10) # empty places count as misses for (name, measure), values in shares.items(): print(f"{name}, {measure}: {np.mean(values):.3f}") kept = np.subtract(shares["standalone", "refinement in text or attributes"], shares["turns joined", "refinement in text or attributes"]) print(f"standalone against turns joined, more, fewer, same: {(kept > 0).sum()}, {(kept < 0).sum()}, {(kept == 0).sum()}") print("conversations the filter leaves with fewer than 10 products:", sum(length < 10 for length in filtered_lengths)) ``` Output: ```text conversations: 12 first query: coat rack for bedroom follow-up: something in black standalone: black coat rack for bedroom first query, refinement in attributes: 0.125 first query, refinement in text or attributes: 0.200 first query, right kind: 0.958 first query, right kind with the refinement: 0.175 follow-up alone, refinement in attributes: 0.433 follow-up alone, refinement in text or attributes: 0.950 follow-up alone, right kind: 0.058 follow-up alone, right kind with the refinement: 0.058 turns joined, refinement in attributes: 0.208 turns joined, refinement in text or attributes: 0.383 turns joined, right kind: 0.933 turns joined, right kind with the refinement: 0.325 standalone, refinement in attributes: 0.200 standalone, refinement in text or attributes: 0.392 standalone, right kind: 0.958 standalone, right kind with the refinement: 0.350 standalone, filtered, refinement in attributes: 0.583 standalone, filtered, refinement in text or attributes: 0.900 standalone, filtered, right kind: 0.675 standalone, filtered, right kind with the refinement: 0.675 standalone against turns joined, more, fewer, same: 5, 4, 3 conversations the filter leaves with fewer than 10 products: 3 ``` The follow-up alone keeps the refinement in 0.950 of its top 10, but only 0.058 are the right kind of product. Any form that carries the first query finds the right kind of product: 0.933 for the joined turns and 0.958 for the standalone query. On these conversations, the joined turns and the standalone query both keep the refinement more often than the first query, which never asked for it: 0.383 and 0.392, against 0.200. Counted in `attributes` alone, the shares are lower, 0.208, 0.200 and 0.125, because many products state the value only in their text, which is all the embeddings read. The conclusions below rest on the text-or-attributes count, which can overcount, since a product that mentions `round legs` counts as round. The LLM's rewrite does no better than joining the turns: against the joined turns, the standalone query keeps more, fewer or the same number of refined products in 5, 4, 3 conversations, and products of the right kind that also keep the refinement make up 0.350 of the standalone query's top 10s, against 0.325 for the joined turns. The filter raises that share to 0.675, but it leaves 3 conversations with fewer than 10 products. These 12 conversations are ours, not shoppers', so the counts are a diagnosis, not a test. **What to do:** never search a follow-up alone. Join it to the earlier turns at least, or rewrite it to stand alone. When the refinement names a value the catalog states, extract it and boost or filter on it, and fall back to the unfiltered list when too few products remain. Measure on conversations from your own logs before you trust any of it. ## What goes wrong in production {#understanding-queries-in-production} **Every LLM rewrite is a model call on the query's path.** The examples read rewrites recorded once, offline, but a live search box calls the model on every query while the shopper waits. query2doc's authors note that it "requires running inference with LLMs which can be considerably slower" ([Wang et al., 2023](https://arxiv.org/abs/2303.07678)), and the longer query also slows the BM25 search. A rule of thumb, not a measurement: rewrite frequent queries offline and cache the results, and call the model online only for queries where a test shows a gain. **A rewrite belongs to one model and one prompt.** Another model version or a reworded prompt can change any rewrite. The manifests of the store's recordings hold the model's digest and each prompt verbatim, and a change to either means measuring again. **The lexicon must follow the catalog.** A corrector built from last month's catalog can treat this month's brand names as typos. Rebuild the lexicon with the index, and log every query the corrector changes next to the original, so that you can find over-corrections such as `tye` → `the`. In machine-translated queries, misspellings change which products a search returns ([Hanneman et al., 2024](https://aclanthology.org/2024.acl-long.750/)), so expect an error the corrector misses to show up as unstable results for one query. **A classifier follows its taxonomy.** When product classes are renamed, split or merged, retrain the classifier. A query class that names no product class, such as `Crock Pots & Slow Cookers`, can never be predicted; for `7qt slow cooker` the classifier predicts `Slow Cookers` instead. **Rewritten queries surface unjudged products.** Every LLM-based change in this chapter that was scored with nDCG@10 lowered [judged@10](#score-every-test-query). Before you adopt a rewrite, have the unjudged products in its top 10s judged, or check condensed scores as [chapter 2](#unjudged-results) does. **What to do:** log the query as typed and as rewritten, cache LLM rewrites, version the model and prompt with the outputs, and rebuild the lexicon and the classifier whenever the catalog changes. ## Decisions this chapter made {#understanding-queries-decisions} Every setting below was chosen on the train queries. The table holds the chapter's 5 planned comparisons on the test queries, one per column: a retriever with one change, against the same retriever on the query as typed. A difference is established when its permutation p-value is below 0.01. | Measure | [BM25, synonyms](#expand-with-synonyms) | [Semantic, class boost](#boost-predicted-class) | [BM25, LLM rewrite](#rewrite-queries-with-llm) | [Semantic, LLM rewrite](#rewrite-queries-with-llm) | [Semantic, HyDE](#hyde-hypothetical-products) | | --- | --- | --- | --- | --- | --- | | Mean difference | 0.000 | 0.005 | -0.056 | -0.081 | -0.031 | | Wins, losses, ties | 0, 0, 30 | 7, 4, 19 | 13, 15, 2 | 7, 19, 4 | 8, 14, 8 | | Permutation p | 1.000 | 0.174 | 0.060 | 0.000 | 0.108 | | Established | no | no | no | yes | no | | t-test p | 1.000 | 0.164 | 0.060 | 0.000 | 0.121 | | 99% interval | 0.000 to 0.000 | -0.002 to 0.017 | -0.143 to 0.005 | -0.132 to -0.035 | -0.115 to 0.001 | | t-test agrees | yes | yes | yes | yes | yes | | Interval agrees | yes | yes | yes | yes | yes | - **Spelling: correct only words the catalog doesn't use.** The corrector fixes the store's train typos, but the typo queries are too few to test it, and it needs context before it handles `tye dye` or `ice an water`. The store doesn't run it yet. - **Units: one written form for products and queries, from the catalog's own spellings,** with decimals kept whole. The size queries are too few to test it, so the baseline BM25 stays as it is. - **Synonyms: a list for gaps seen in train queries,** for BM25 only. It changed 1 test query and showed no established difference, so the baseline doesn't expand. - **Query classes: naive Bayes trained on product text,** because the sample has at most one query per class. The predicted class is a hint. A filter loses on train, and the best boost isn't established on test, so semantic retrieval stays unboosted. - **LLM rewrites: not used.** The loss for semantic retrieval is established, and BM25's change isn't. - **HyDE: not used.** Averaged with the query, it shows no established difference, and the hypothetical product alone loses on train. - **Follow-ups: never searched alone.** Joining the turns does about as well as the LLM's standalone query. A filter on the refined value is the next step, and it needs an extracted value and a fallback. - **The ranker to beat stays semantic retrieval on the query as typed,** with BM25, unstemmed, as the baseline. # Ranking By the end of this chapter you can describe each product that retrieval returns with features, train a learning-to-rank model with LightGBM on your own judgments, rerank the top of the list with a cross-encoder or an LLM, measure an LLM reranker's positional bias, set the rerank depth by counting its costs, and distil a slow reranker into a fast one. You can also decide, with a planned test on held-out queries, whether any of them beats your retrieval. The store, the online furniture and home-goods retailer this book follows as its case study ([chapter 1](#store-data)), has one ranker to beat: [semantic retrieval](#search-by-meaning) alone. It ranks products by embeddings from `Snowflake/snowflake-arctic-embed-m-v1.5`, and scores 0.830 [nDCG@10](#score-one-ranking) on the 30 [test queries](#store-data). No change in chapters 5 to 8 has beaten it by an established margin. This chapter asks whether a second stage can: a model trained on the store's own judgments, a reranker that reads the query and the product together, or an LLM. [Chapter 1's cross-encoder](#rerank-with-cross-encoder), a reranker of the second kind, reranked the hybrid run, not semantic retrieval, and had never seen the store's judgments. **Reranking** reorders the top of retrieval's list with a slower model. The store's rerankers are **recorded**: each ran once and saved its outputs, which the examples read. The cross-encoder is the open model `Alibaba-NLP/gte-reranker-modernbert-base`, whose **manifest**, the file saved with its recording, gives its licence as Apache-2.0. `recordings/cross_encoder.py` scored the union of BM25's and semantic retrieval's top 100 for every query (for two queries, its semantic list differs from `lib.semantic`'s by a near-tie at the 100th place). A **large language model** (LLM) is a model trained on large amounts of text to generate text in response to a prompt. The store's LLM is the open model `gemma4:12b-it-q4_K_M`, licensed Apache-2.0. It ran locally with a fixed seed and three settings: temperature 0, so it takes its likeliest next word every time; thinking off, so it answers without writing out reasoning; and a JSON schema, a machine-readable description of the answer's shape. We chose both models because anyone can download them and rerun the recordings. `recordings/llm_rerank.py` reranked semantic retrieval's top 20 for the test queries only, so nothing about the LLM rerankers can be tuned. Every score uses the human judgments, because in one study LLM judges ranked LLM rerankers too high ([What goes wrong in production](#ranking-in-production)). Every finding about the LLM holds for this model, its two prompts and these test queries, not for LLMs in general. Before reading any test result, the chapter fixes 5 planned comparisons on the test queries, each against semantic retrieval's run on nDCG@10: 1. [LambdaMART](#lambdamart-on-test-queries), a learned ranker trained on the train queries' judgments, ordering each query's candidate products 2. the [cross-encoder](#rerank-semantic-with-cross-encoder) reranking semantic retrieval's top 20 3. the LLM scoring each of the top 20 [on its own](#llm-pointwise-reranking) 4. the LLM ordering the top 20 [in one prompt](#llm-listwise-reranking) 5. a small [distilled ranker](#distill-the-cross-encoder) trained on the cross-encoder's scores By [the book's rule](#when-a-difference-is-established), a difference is established when its paired [permutation test](#permutation-test-on-queries) gives a p-value below 0.01, 0.05 divided by the number of planned comparisons. The [paired t-test](#paired-t-test-on-queries) and a 99% [bootstrap interval](#report-the-size-of-a-gap) stand beside it. Numbers on the 50 train queries guide design choices; they are not evidence. ## Describe each candidate with features {#ranking-features} A learned ranker, the first of the chapter's [planned comparisons](#ranking), doesn't read text. It scores each pair of a query and a product from a list of numbers, its **features**, and learns from judgments how to weigh them. A **query-dependent** feature describes the pair, such as a BM25 score. A **query-independent** feature describes the product alone, such as how many shoppers rated it. Feature sets for web search mix both: LETOR 4.0, a benchmark for learning to rank, has 46 features, among them "BM25 of title", "PageRank" and "Number of slash in URL" ([Qin and Liu, 2013](https://arxiv.org/abs/1306.2597)). A ranker scores only **candidates**, the products a cheaper retrieval stage returns, so its features can be expensive. In the store, a query's candidates are the products the cross-encoder's recording scored: the union of BM25's top 100 and [semantic retrieval](#search-by-meaning)'s top 100. `lib.ranking.feature_table` gives each candidate 8 features: - `bm25`: the [BM25](#rank-with-bm25) score from `lib.lexical`, the book's unstemmed baseline, and 0 for a product that shares no term with the query - `similarity`: the cosine similarity of the query's and the product's embeddings - `cross_encoder`: the recorded cross-encoder score - `name_match` and `attribute_match`: the share of the query's terms, [tokenized as chapter 5 does](#split-text-into-terms), found in the product's name and in its [attributes](#index-attributes-as-a-field) - `rating_count`, `average_rating` and `review_count`, which the store's data gives each product 2632 of the catalog's products have no rating, and their three rating features stay missing. The table also holds each candidate's human grade, or a gap when nobody judged it. Build the table for the [train queries](#store-data), look at one query, then compare each feature across the grades: Example `ranking-features-by-grade` (Python 3.12.15 · bm25s 0.3.13 · lightgbm 4.7.0 · numpy 2.5.4 · pandas 3.0.6 · pystemmer 3.1.0 · scikit-learn 1.9.1 · scipy 1.18.1), passing in Edition 2026.10.1: ```python import pandas as pd from lib.ranking import FEATURES, feature_table from lib.store import load_products, load_queries products, queries = load_products(), load_queries("train") # train queries: features are design work table = feature_table(products, queries) print(f"features: {len(FEATURES)}") print(f"train candidates: {len(table)}") print(f"train candidates per query, median: {table.groupby('query_id').size().median():.0f}") print(f"share of train candidates nobody judged: {table['grade'].isna().mean():.3f}") query_id = queries.index[queries["query"] == "laundry basket with wheels"][0] one = table[table["query_id"] == query_id].sort_values("cross_encoder", ascending=False) one = one.assign(name=products.loc[one["product_id"], "name"].str[:38].to_numpy()) pd.set_option("display.width", 200) print(f"\ncandidates for laundry basket with wheels: {len(one)}") print(one[["name", "bm25", "similarity", "cross_encoder", "name_match", "attribute_match", "rating_count", "grade"]] .head(6).round(3).to_string(index=False)) print(f"top 6 by cross-encoder, exact: {(one['grade'].head(6) == 2).sum()}") label = table["grade"].map({2: "exact", 1: "partial", 0: "irrelevant"}).fillna("unjudged") means = table[FEATURES].groupby(label).mean().loc[["exact", "partial", "irrelevant", "unjudged"]] print("\nmean of each feature by grade, train candidates:") print(means.round(3).to_string()) for grade in means.index: print(f"mean cross_encoder, {grade}: {means.at[grade, 'cross_encoder']:.3f}") relevance = ["bm25", "similarity", "cross_encoder", "name_match"] nearer = [name for name in relevance if abs(means.at["unjudged", name] - means.at["irrelevant", name]) < abs(means.at["unjudged", name] - means.at["partial", name])] print(f"relevance features where unjudged sits nearer irrelevant than partial: {len(nearer)} of {len(relevance)}") medians = table[["rating_count", "review_count"]].groupby(label).median().loc[means.index] # counts are skewed for grade in ["exact", "partial", "irrelevant"]: print(f"rating_count, {grade}, mean: {means.at[grade, 'rating_count']:.3f}") print(f"rating_count, {grade}, median: {medians.at[grade, 'rating_count']:.1f}") print(f"products with no rating: {products['rating_count'].isna().sum()}") ``` Output: ```text features: 8 train candidates: 6896 train candidates per query, median: 131 share of train candidates nobody judged: 0.286 candidates for laundry basket with wheels: 125 name bm25 similarity cross_encoder name_match attribute_match rating_count grade laundry sorter basket on wheels 8.229 0.597 3.433 1.000 0.0 NaN 2.0 vintage laundry basket 6.270 0.572 2.995 0.667 0.0 638.0 2.0 rolling wicker laundry basket 5.494 0.588 2.974 0.667 0.0 110.0 2.0 aluminum frame laundry basket can be f 4.739 0.524 2.941 0.667 0.0 14.0 2.0 rebrilliant rolling laundry hamper 5.537 0.558 2.804 0.333 0.0 17.0 2.0 4-bag laundry hamper sorter cart baske 6.012 0.543 2.739 0.667 0.0 NaN 2.0 top 6 by cross-encoder, exact: 6 mean of each feature by grade, train candidates: bm25 similarity cross_encoder name_match attribute_match rating_count average_rating review_count grade exact 4.317 0.517 2.191 0.562 0.024 466.879 4.537 333.589 partial 3.936 0.454 1.513 0.461 0.030 385.454 4.539 288.779 irrelevant 2.760 0.360 1.159 0.404 0.006 246.781 4.552 159.052 unjudged 2.323 0.369 1.050 0.140 0.018 339.904 4.520 237.027 mean cross_encoder, exact: 2.191 mean cross_encoder, partial: 1.513 mean cross_encoder, irrelevant: 1.159 mean cross_encoder, unjudged: 1.050 relevance features where unjudged sits nearer irrelevant than partial: 4 of 4 rating_count, exact, mean: 466.879 rating_count, exact, median: 43.0 rating_count, partial, mean: 385.454 rating_count, partial, median: 31.0 rating_count, irrelevant, mean: 246.781 rating_count, irrelevant, median: 35.5 products with no rating: 2632 ``` The train queries have 6896 candidates, a median of 131 per query, and nobody judged a share of 0.286 of them. `laundry basket with wheels` has 125 candidates. The six that the cross-encoder scores highest include 6 [exact](#store-data) products. Their `attribute_match` is 0, because attributes hold colours, materials and sizes, not words such as `wheels`. Across the train candidates, the mean cross-encoder score falls from 2.191 for exact products to 1.513 for partial and 1.159 for irrelevant ones, and `bm25`, `similarity` and `name_match` fall the same way. `average_rating` and `attribute_match` barely move. The mean `rating_count` falls from 466.879 to 385.454 and 246.781, but the medians are 43.0, 31.0 and 35.5: a few heavily rated products set the means. Unjudged candidates average a cross-encoder score of 1.050, and on 4 of 4 features that fall with the grade, their mean sits nearer the irrelevant products' than the partial ones'. That matters for what an unjudged candidate should teach a model. **What to do:** start with the features you already compute: each retriever's score, a reranker's score if you have one, matches on fields such as the name, and popularity signals. Compare each feature by grade on train queries before you train, with means for scores and medians for skewed counts. A feature that doesn't move with the grade has little to teach the model. Compute features with the same code when you train and when you serve. ## Learn a ranking from pairs of products {#learn-from-pairs} **Learning to rank** trains a model on judged queries to order candidates by their [features](#ranking-features). Its methods fall into three families: "the pointwise, pairwise, and listwise approaches" ([Liu, 2009](https://doi.org/10.1561/1500000016)). A **pointwise** model predicts each product's grade on its own, as a regression. A **pairwise** model learns which of two products should rank higher. A **listwise** model scores a whole ranking. On the LETOR benchmark, Liu's comparison "seems to suggest that the listwise approach be the most effective one", a result from 2009. The obstacle for all three is the metric: nDCG changes only when two products swap places, so ranking metrics "are everywhere either discontinuous or flat, so gradient descent appears to be problematic" ([Burges, 2010](https://www.microsoft.com/en-us/research/publication/from-ranknet-to-lambdarank-to-lambdamart-an-overview/)). RankNet, a pairwise method, works around it with a smooth loss on pairs. For two products where i should rank above j, its gradient is large when the model scores them the wrong way round or nearly level, and small when it already has them well apart. LambdaRank keeps that gradient and multiplies it by how much nDCG would change if the two products swapped places (Burges, 2010, section 4.1). Burges writes the gradient with a minus sign, as the slope of a cost; here a positive λ pushes i up: ```text λ_ij = σ / (1 + exp(σ · (s_i − s_j))) · |ΔnDCG_ij| s: the model's scores, σ = 1 ΔnDCG_ij = (g_i − g_j) · (1 / log2(1 + r_i) − 1 / log2(1 + r_j)) / ideal DCG g: grades, r: positions ``` A product's total is its λ summed over the pairs where it should rank higher, minus those where it should rank lower. "The λ's can also be interpreted as forces" (Burges, 2010), pushing products up and down the list. **LambdaMART** uses these λ as the gradients for **gradient-boosted trees**, which add small decision trees one at a time, each fitted to the gradients of the trees before it: "LambdaMART is the boosted tree version of LambdaRank, which is based on RankNet". An ensemble of LambdaMART rankers won Track 1 of the Learning to Rank Challenge that Yahoo ran in 2010 (Burges, 2010). In the store, take the train query `outdoor waterproof chest`, and let semantic similarity be the model's scores so far. Compute every pair's λ in its top 10 by hand, check each |ΔnDCG| by swapping the pair and recomputing nDCG with `lib.metrics`, and add up the forces: Example `lambdas-for-one-query` (Python 3.12.15 · numpy 2.5.4 · pandas 3.0.6), passing in Edition 2026.10.1: ```python import math from itertools import combinations from lib.metrics import dcg_at_k, ndcg_at_k from lib.semantic import SemanticSearch from lib.store import load_queries, qrels queries, judgments = load_queries("train"), qrels("train") query_id = queries.index[queries["query"] == "outdoor waterproof chest"][0] judged = judgments[query_id] scores = SemanticSearch().search(query_id, k=10) # the model so far: semantic similarity ranking, similarity = list(scores.index), scores.to_numpy() grades = [judged.get(product_id, 0) for product_id in ranking] ideal, sigma = dcg_at_k(sorted(judged.values(), reverse=True), 10), 1.0 print("grades from the top:", " ".join(map(str, grades))) force, checks, pulls = [0.0] * len(ranking), [], [] for i, j in combinations(range(len(ranking)), 2): # i is above j if grades[i] == grades[j]: continue better, worse = (i, j) if grades[i] > grades[j] else (j, i) ranknet = 1 / (1 + math.exp(sigma * (similarity[better] - similarity[worse]))) # RankNet's pull delta = abs((grades[i] - grades[j]) * (1 / math.log2(i + 2) - 1 / math.log2(j + 2))) / ideal swapped = ranking.copy() swapped[i], swapped[j] = swapped[j], swapped[i] checks.append(math.isclose(delta, abs(ndcg_at_k(swapped, judged, 10) - ndcg_at_k(ranking, judged, 10)))) pulls.append(ranknet) pair_lambda = sigma * ranknet * delta # LambdaRank: scaled by |delta nDCG| force[better], force[worse] = force[better] + pair_lambda, force[worse] - pair_lambda print(f"pairs with different grades: {len(checks)}") print(f"pairs where delta ndcg by formula matches swapping the pair: {sum(checks)}") for position, push in enumerate(force, start=1): print(f"force at position {position}: {push:+.3f}") print(f"strongest push up, position: {force.index(max(force)) + 1}") print(f"strongest push down, position: {force.index(min(force)) + 1}") print(f"ranknet's pull, lowest: {min(pulls):.3f}\nranknet's pull, highest: {max(pulls):.3f}") ``` Output: ```text grades from the top: 1 2 2 0 1 2 0 2 0 0 pairs with different grades: 32 pairs where delta ndcg by formula matches swapping the pair: 32 force at position 1: +0.013 force at position 2: +0.159 force at position 3: +0.104 force at position 4: -0.083 force at position 5: -0.010 force at position 6: +0.063 force at position 7: -0.092 force at position 8: +0.063 force at position 9: -0.105 force at position 10: -0.111 strongest push up, position: 2 strongest push down, position: 10 ranknet's pull, lowest: 0.472 ranknet's pull, highest: 0.525 ``` The top 10's grades, from the top, are 1 2 2 0 1 2 0 2 0 0; an unjudged product counts as 0, as it does in the metric. The top 10 holds 32 pairs with different grades, and for 32 of them the formula's |ΔnDCG| matches the change from swapping the pair. RankNet's factor in λ stays between 0.472 and 0.525, because the similarities lie close together, so |ΔnDCG| decides almost everything. The strongest push up goes to position 2, an exact product, with +0.159. The strongest push down goes to position 10, an irrelevant product, with -0.111. The partial product at position 1 gets +0.013: the irrelevant products below push it up, the exact ones push it down. That example simplifies: it takes the ideal DCG from every judged product, as `lib.metrics` does, and looks at ten products. LightGBM takes the ideal from the query's training rows, and at 4.7.0 two defaults change the λ. `lambdarank_truncation_level` is 30, so it "controls the number of top-results to focus on during training": only pairs with a product in the current top 30 get a λ. `lambdarank_norm` is true, to "normalize the lambdas for different queries"; set it to false "to enforce the original lambdarank algorithm" ([Parameters](https://lightgbm.readthedocs.io/en/v4.7.0/Parameters.html)). In the source, normalizing divides each pair's |ΔnDCG| by the gap between its scores plus 0.01, then rescales each query's λ ([`rank_objective.hpp`](https://github.com/microsoft/LightGBM/blob/v4.7.0/src/objective/rank_objective.hpp)). To check the formula against the library, write it as a custom LightGBM objective, grow five small trees on this query's candidates, and grow them again with LightGBM's lambdarank, with and without those defaults: Example `lambdas-match-lightgbm` (Python 3.12.15 · bm25s 0.3.13 · lightgbm 4.7.0 · numpy 2.5.4 · pandas 3.0.6 · pystemmer 3.1.0 · scikit-learn 1.9.1 · scipy 1.18.1), passing in Edition 2026.10.1: ```python import numpy as np from lightgbm import LGBMRanker from lib.ranking import DETERMINISTIC, FEATURES, feature_table from lib.store import load_products, load_queries queries = load_queries("train") one = queries[queries["query"] == "outdoor waterproof chest"] rows = feature_table(load_products(), one) # one query's candidates, in product_id order grades = rows["grade"].fillna(0).astype(int).to_numpy() SIGMA = 1.0 # LightGBM's default sigmoid def lambdas(y_true, y_pred, weight, group): """The chapter's lambda for every pair with different grades, as gradients and second derivatives.""" rank = np.empty(len(y_pred), dtype=int) rank[np.argsort(-y_pred, kind="stable")] = np.arange(len(y_pred)) # LightGBM's stable sort on score discount, ideal = 1 / np.log2(rank + 2), np.sum(np.sort(y_true)[::-1] / np.log2(np.arange(len(y_true)) + 2)) gradient, hessian = np.zeros(len(y_pred)), np.zeros(len(y_pred)) for i in range(len(y_true)): for j in np.flatnonzero(y_true < y_true[i]): # i should rank above j delta = (y_true[i] - y_true[j]) * abs(discount[i] - discount[j]) / ideal pull = 1 / (1 + np.exp(SIGMA * (y_pred[i] - y_pred[j]))) gradient[i], gradient[j] = gradient[i] - SIGMA * pull * delta, gradient[j] + SIGMA * pull * delta hessian[[i, j]] += SIGMA ** 2 * pull * (1 - pull) * delta return gradient, hessian tree = {"n_estimators": 5, "num_leaves": 3, "min_child_samples": 5, **DETERMINISTIC} plain = {"lambdarank_norm": False, "lambdarank_truncation_level": len(rows)} # LightGBM's lambdas, untouched models = {"by hand": LGBMRanker(objective=lambdas, **tree), "lightgbm, no normalization or truncation": LGBMRanker(objective="lambdarank", label_gain=[0, 1, 2], **plain, **tree), "lightgbm at its defaults": LGBMRanker(objective="lambdarank", label_gain=[0, 1, 2], **tree)} scores = {name: model.fit(rows[FEATURES], grades, group=[len(rows)]).predict(rows[FEATURES], raw_score=True) for name, model in models.items()} print(f"candidates: {len(rows)}") for name in list(models)[1:]: difference = np.abs(scores[name] - scores["by hand"]).max() print(f"{name}, largest difference from the lambdas by hand after 5 trees: {difference:.6f}") ``` Output: ```text candidates: 180 lightgbm, no normalization or truncation, largest difference from the lambdas by hand after 5 trees: 0.000006 lightgbm at its defaults, largest difference from the lambdas by hand after 5 trees: 0.595667 ``` Over the query's 180 candidates, LightGBM without normalization or truncation scores every product within 0.000006 of the custom objective's scores. The gap is rounding, since LightGBM reads its sigmoid from a precomputed table (`rank_objective.hpp`). At its defaults, scores differ by up to 0.595667. **What to do:** train rankers with a gradient that weighs each pair by the metric you report, as LambdaMART does, rather than regressing on grades. Know which defaults of your library change that gradient, and check on train queries that it beats a pointwise model on your data, because the families' order in a benchmark doesn't transfer by itself. ## Train LambdaMART with LightGBM {#train-lambdamart} LightGBM trains [LambdaMART](#learn-from-pairs) with `LGBMRanker` and `objective="lambdarank"`. Four settings need care, and each is documented for 4.7.0: - **Rows are grouped by query.** "data should be ordered by the query", and `group` holds the number of rows of each query in that order, not query IDs: "sum(group) = n_samples" ([Parameters](https://lightgbm.readthedocs.io/en/v4.7.0/Parameters.html); [`LGBMRanker`](https://lightgbm.readthedocs.io/en/v4.7.0/pythonapi/lightgbm.LGBMRanker.html)). - **Gains are exponential by default.** `label_gain` defaults to `0,1,3,7,15,31,63,...`, so "the gain of label `2` is `3`", and every label must be smaller than the number of gains (Parameters). That is Burges's DCG, with gain 2^grade − 1. The book's nDCG uses [linear gains](#gains-are-a-choice), so set `label_gain=[0, 1, 2]`. - **nDCG is evaluated at positions 1 to 5 by default.** Set `eval_at=[10]` to stop on nDCG@10, and report the final score with `lib.metrics` anyway. - **Validation data goes in `eval_X` and `eval_y`.** In 4.7.0, `eval_set` is "Deprecated since version 4.7.0", with the note "Use eval_X and eval_y instead" (`LGBMRanker`; [release notes](https://github.com/microsoft/LightGBM/releases/tag/v4.7.0)). The examples pass `eval_X`, `eval_y` and `eval_group`. **Early stopping** scores held-out train queries after every tree, and keeps the trees up to the best score once 30 more bring no gain. So that training repeats exactly, the examples use one thread, a fixed seed and `deterministic=True`, which "should ensure the stable results when using the same data" (Parameters), with `force_row_wise=True`, which the same entry asks for "to avoid potential instability due to numerical issues". `lib.ranking` holds these settings, and its `fit` applies the grouping and early stopping. Row order matters too. Before the first tree every score is 0, so LightGBM takes the rows' order as the starting ranking for the first λ, and equal scores need a tie-break. The cross-encoder's recording lists each query's [candidates](#ranking-features) best first, which would hand the model the cross-encoder's ranking before it learned anything. So `feature_table` sorts candidates by `product_id`, and `lib.ranking.ranked` breaks ties by `product_id`. Choose the tree size by **cross-validation** on the train queries: split them into five folds, rank each fold with a model fitted on three others and stopped early on the fifth, and score every query once. Try eight settings, from 3 to 31 leaves at two learning rates, counting unjudged candidates as grade 0 as the metric does: Example `lambdamart-tree-size-on-train` (Python 3.12.15 · bm25s 0.3.13 · lightgbm 4.7.0 · numpy 2.5.4 · pandas 3.0.6 · pystemmer 3.1.0 · scikit-learn 1.9.1 · scipy 1.18.1), passing in Edition 2026.10.1: ```python from lightgbm import LGBMRanker from lib.metrics import evaluate from lib.ranking import DETERMINISTIC, FEATURES, LAMBDARANK, TREE_GRID, TREES, cross_validate, feature_table from lib.semantic import SemanticSearch from lib.store import load_products, load_queries, qrels queries, judgments = load_queries("train"), qrels("train") # tuning: train queries only candidates = feature_table(load_products(), queries) candidates["grade or 0"] = candidates["grade"].fillna(0).astype(int) # unjudged counts as irrelevant, as in nDCG print(f"train queries: {len(queries)}") print(f"semantic, train ndcg@10: {evaluate(SemanticSearch().run(queries, k=100), judgments)['ndcg@10'].mean():.3f}") results = {} for setting in TREE_GRID: # LightGBM's default is 31 leaves at 0.1 name = f"{setting['num_leaves']} leaves, learning rate {setting['learning_rate']}" model = lambda: LGBMRanker(**LAMBDARANK, n_estimators=500, **setting, **DETERMINISTIC) results[name] = cross_validate(model, candidates, candidates, "grade or 0", FEATURES, judgments) print(f"{name}, cross-validated ndcg@10: {results[name].mean():.3f}") print(f"{name}, trees kept per fold: {', '.join(map(str, results[name].attrs['trees']))}") best = max(results, key=lambda name: results[name].mean()) print(f"settings tried: {len(results)}") print(f"best setting: {best}") print(f"lib.ranking.TREES holds it: {'yes' if best == f'{TREES['num_leaves']} leaves, learning rate {TREES['learning_rate']}' else 'no'}") ``` Output: ```text train queries: 50 semantic, train ndcg@10: 0.772 3 leaves, learning rate 0.05, cross-validated ndcg@10: 0.780 3 leaves, learning rate 0.05, trees kept per fold: 18, 27, 52, 20, 85 3 leaves, learning rate 0.1, cross-validated ndcg@10: 0.798 3 leaves, learning rate 0.1, trees kept per fold: 99, 20, 8, 43, 38 7 leaves, learning rate 0.05, cross-validated ndcg@10: 0.744 7 leaves, learning rate 0.05, trees kept per fold: 2, 28, 30, 39, 55 7 leaves, learning rate 0.1, cross-validated ndcg@10: 0.772 7 leaves, learning rate 0.1, trees kept per fold: 80, 12, 13, 7, 20 15 leaves, learning rate 0.05, cross-validated ndcg@10: 0.720 15 leaves, learning rate 0.05, trees kept per fold: 4, 37, 175, 44, 120 15 leaves, learning rate 0.1, cross-validated ndcg@10: 0.728 15 leaves, learning rate 0.1, trees kept per fold: 19, 68, 136, 48, 53 31 leaves, learning rate 0.05, cross-validated ndcg@10: 0.679 31 leaves, learning rate 0.05, trees kept per fold: 15, 8, 91, 8, 22 31 leaves, learning rate 0.1, cross-validated ndcg@10: 0.737 31 leaves, learning rate 0.1, trees kept per fold: 11, 75, 166, 96, 58 settings tried: 8 best setting: 3 leaves, learning rate 0.1 lib.ranking.TREES holds it: yes ``` Of 8 settings, the best is 3 leaves, learning rate 0.1, at 0.798; LightGBM's default of 31 leaves at 0.1 (`LGBMRanker`) scores 0.737. The best of eight tries flatters itself, so expect less on held-out queries. One choice depends on the store's judgments. Nobody judged a share of 0.286 of the train candidates. A model can count them as grade 0, or learn from judged candidates only; [unjudged isn't irrelevant](#unjudged-results), but the model will rank them at query time either way. Compare both by cross-validation at the chosen tree size, with a pointwise regression on the grades beside them: Example `lambdamart-choices-on-train` (Python 3.12.15 · bm25s 0.3.13 · lightgbm 4.7.0 · numpy 2.5.4 · pandas 3.0.6 · pystemmer 3.1.0 · scikit-learn 1.9.1 · scipy 1.18.1), passing in Edition 2026.10.1: ```python from lightgbm import LGBMRanker, LGBMRegressor from lib.metrics import evaluate from lib.ranking import DETERMINISTIC, FEATURES, LAMBDARANK, TREES, cross_validate, feature_table from lib.semantic import SemanticSearch from lib.significance import compare from lib.store import load_products, load_queries, qrels queries, judgments = load_queries("train"), qrels("train") # tuning: train queries only candidates = feature_table(load_products(), queries) candidates["grade or 0"] = candidates["grade"].fillna(0).astype(int) # unjudged counts as irrelevant judged_only = candidates.dropna(subset=["grade"]).astype({"grade": int}) setups = { # (model, rows it learns from, label) "lambdarank, unjudged as 0": (lambda: LGBMRanker(**LAMBDARANK, **TREES, **DETERMINISTIC), candidates, "grade or 0"), "lambdarank, unjudged left out": (lambda: LGBMRanker(**LAMBDARANK, **TREES, **DETERMINISTIC), judged_only, "grade"), "pointwise regression, unjudged as 0": (lambda: LGBMRegressor(**TREES, **DETERMINISTIC), candidates, "grade or 0"), } semantic = evaluate(SemanticSearch().run(queries, k=100), judgments)["ndcg@10"] print(f"semantic, train ndcg@10: {semantic.mean():.3f}") scores = {name: cross_validate(model, rows, candidates, label, FEATURES, judgments) for name, (model, rows, label) in setups.items()} chosen = scores["lambdarank, unjudged as 0"] for name, per_query in scores.items(): print(f"{name}, cross-validated ndcg@10: {per_query.mean():.3f}") if name != "lambdarank, unjudged as 0": result = compare(per_query, chosen) print(f"{name}, minus the first: {result['mean difference']:.3f}") print(f"{name}, minus the first, permutation p: {result['permutation p']:.3f}") result = compare(chosen, semantic) print(f"first setup minus semantic: {result['mean difference']:.3f}") print(f"first setup minus semantic, permutation p: {result['permutation p']:.3f}") ``` Output: ```text semantic, train ndcg@10: 0.772 lambdarank, unjudged as 0, cross-validated ndcg@10: 0.798 lambdarank, unjudged left out, cross-validated ndcg@10: 0.733 lambdarank, unjudged left out, minus the first: -0.064 lambdarank, unjudged left out, minus the first, permutation p: 0.005 pointwise regression, unjudged as 0, cross-validated ndcg@10: 0.769 pointwise regression, unjudged as 0, minus the first: -0.029 pointwise regression, unjudged as 0, minus the first, permutation p: 0.007 first setup minus semantic: 0.026 first setup minus semantic, permutation p: 0.242 ``` Counting unjudged candidates as 0 scores 0.798. Leaving them out scores 0.733, a difference of -0.064 with a permutation p-value of 0.005. A model that never sees an unjudged candidate in training still has to rank them, and [the features](#ranking-features) say they look like irrelevant products. Pointwise regression scores 0.769, a difference of -0.029 (p = 0.007), so on the store's train queries the [check against a pointwise model](#learn-from-pairs) favours LambdaMART. These are tuning checks on train queries, not evidence. Against semantic retrieval's 0.772, the chosen setup differs by 0.026, with a permutation p-value of 0.242: even on the queries that chose it, it shows no gain the train queries can tell from chance. **What to do:** sort rows by query and pass `group`, set `label_gain` to the gains your metric uses, stop early on held-out train queries through `eval_X` and `eval_y`, and fix the seed and thread count. Put each query's candidates in an order that carries no information, such as by product ID, and break score ties the same way. Count unjudged candidates as grade 0 unless cross-validation on train queries says otherwise. With few judged queries, try small trees. ## Report LambdaMART on the test queries {#lambdamart-on-test-queries} A model tuned on train queries is reported once on the test queries, by a comparison planned before looking. **Feature importance** shows what it learned: LightGBM's `"gain"` importance is the "total gains of splits which use the feature" ([`Booster`](https://lightgbm.readthedocs.io/en/v4.7.0/pythonapi/lightgbm.Booster.html)), so a feature with no gain plays no part in the score. In the store, fit the setup [chosen on train](#train-lambdamart): [LambdaMART](#learn-from-pairs) on all the [features](#ranking-features), with 3 leaves at a learning rate of 0.1, unjudged candidates as grade 0, and candidates in `product_id` order. Fit on 40 train queries, and stop early on 10 others. Rank each test query's candidates with it, and make the first of the chapter's [planned comparisons](#ranking), established below p = 0.01: Example `lambdamart-on-test` (Python 3.12.15 · bm25s 0.3.13 · lightgbm 4.7.0 · numpy 2.5.4 · pandas 3.0.6 · pystemmer 3.1.0 · scikit-learn 1.9.1 · scipy 1.18.1), passing in Edition 2026.10.1: ```python from lightgbm import LGBMRanker from lib.metrics import evaluate from lib.ranking import (DETERMINISTIC, FEATURES, LAMBDARANK, PLANNED_TEST_COMPARISONS, THRESHOLD, TREES, feature_table, fit, ranked, report_planned, stopping_split) from lib.semantic import SemanticSearch from lib.store import load_products, load_queries, qrels products, train, test = load_products(), load_queries("train"), load_queries("test") print(f"planned test comparisons: {len(PLANNED_TEST_COMPARISONS)}") print(f"p-value threshold: {THRESHOLD:.2f}") table = feature_table(products, train) table["grade or 0"] = table["grade"].fillna(0).astype(int) # the setup chosen on train fit_ids, stop_ids = stopping_split(train.index) model = fit(LGBMRanker(**LAMBDARANK, **TREES, **DETERMINISTIC), table, "grade or 0", FEATURES, fit_ids, stop_ids) print(f"queries fitted: {len(fit_ids)}\nqueries held out for stopping: {len(stop_ids)}") curve = model.evals_result_["valid_0"]["ndcg@10"] # LightGBM's nDCG@10, one value per tree print(f"trees tried: {len(curve)}\ntrees kept by early stopping: {model.best_iteration_}") print(f"stopping queries' ndcg@10, first tree: {curve[0]:.3f}") print(f"stopping queries' ndcg@10, trees kept: {curve[model.best_iteration_ - 1]:.3f}") gain = model.booster_.feature_importance(importance_type="gain") # total gain of the splits on each feature print(f"features the trees use: {(gain > 0).sum()} of {len(FEATURES)}") for name, share in sorted(zip(FEATURES, gain / gain.sum()), key=lambda pair: -pair[1]): print(f"share of gain, {name}: {share:.3f}") candidates, judgments = feature_table(products, test), qrels("test") # test queries: read once, to report after = evaluate(ranked(candidates, model.predict(candidates[FEATURES])), judgments) before = evaluate(SemanticSearch().run(test, k=100), judgments) for name, frame in [("semantic", before), ("lambdamart", after)]: print(f"{name}, test ndcg@10: {frame['ndcg@10'].mean():.3f}") print(f"{name}, test judged@10: {frame['judged@10'].mean():.3f}") print(f"{name}, unjudged places in the test top 10s: {round((1 - frame['judged@10']).sum() * 10)}") report_planned("lambdamart", after["ndcg@10"], before["ndcg@10"]) ``` Output: ```text planned test comparisons: 5 p-value threshold: 0.01 queries fitted: 40 queries held out for stopping: 10 trees tried: 243 trees kept by early stopping: 213 stopping queries' ndcg@10, first tree: 0.568 stopping queries' ndcg@10, trees kept: 0.794 features the trees use: 8 of 8 share of gain, cross_encoder: 0.641 share of gain, similarity: 0.172 share of gain, name_match: 0.097 share of gain, bm25: 0.043 share of gain, review_count: 0.021 share of gain, rating_count: 0.021 share of gain, attribute_match: 0.002 share of gain, average_rating: 0.001 semantic, test ndcg@10: 0.830 semantic, test judged@10: 0.990 semantic, unjudged places in the test top 10s: 3 lambdamart, test ndcg@10: 0.817 lambdamart, test judged@10: 0.990 lambdamart, unjudged places in the test top 10s: 3 lambdamart, mean difference: -0.012 lambdamart, wins, losses, ties: 14, 12, 4 lambdamart, permutation p: 0.260 lambdamart, established at 0.01: no lambdamart, t-test p: 0.278 lambdamart, t-test agrees: yes lambdamart, 99% interval: -0.044 to 0.011 lambdamart, interval agrees: yes ``` Of 243 trees tried, early stopping keeps 213. On the queries held out for stopping, LightGBM's own nDCG@10, whose ideal comes from their candidates alone, rises from 0.568 after the first tree to 0.794 at the trees kept. The trees use 8 of 8 features. The cross-encoder takes a share of 0.641 of the gain, `similarity` 0.172, `name_match` 0.097 and `bm25` 0.043. The model leans on the cross-encoder, so serving it means a cross-encoder call for every candidate. On test, [judged@10](#score-every-test-query), the share of the top 10 that has a judgment, is 0.990 for LambdaMART and 0.990 for semantic retrieval. LambdaMART scores 0.817 against semantic retrieval's 0.830. No improvement shown, with a permutation p-value of 0.260 ([all five comparisons](#ranking-decisions)). **What to do:** adopt a learned ranker only when its planned test comparison is established. Read its gain importance before you trust it, because the features it leans on are the ones you'll pay for at query time. Early stopping on a few held-out queries keeps very different numbers of trees from one split to the next, from 99, 20, 8, 43, 38 across the cross-validation folds to 213 here. A rule of thumb, not a measurement: with this few queries, judge more queries before you tune further, and treat any one stopping point as noise. ## Rerank semantic retrieval with a cross-encoder {#rerank-semantic-with-cross-encoder} [Semantic retrieval](#search-by-meaning) uses an embedding model, also called a **bi-encoder**, which reads the query and each product separately, so product embeddings are computed ahead of time. A **cross-encoder** reads the query and one product together and outputs one score ([Nogueira and Cho, 2019](https://arxiv.org/abs/1901.04085)). It can relate every query word to every product word, and it pays for that: it is "Often slower than a Sentence Transformer model, as it requires computation for each pair rather than each text", and so "Cross Encoders are often used to re-rank the top-k results" ([Sentence Transformers](https://sbert.net/docs/cross_encoder/usage/usage.html)). The store's cross-encoder, named in [the chapter's opening](#ranking), is small for a language model, with 149M parameters ([model card](https://huggingface.co/Alibaba-NLP/gte-reranker-modernbert-base)). [Chapter 1](#rerank-with-cross-encoder) reranked the hybrid run. Here the cross-encoder reranks the ranker to beat, semantic retrieval's top 20: the same products the [LLM rerankers](#llm-pointwise-reranking) read, so the chapter's three rerankers start from identical lists. The chapter fixed that depth before reading any test result, and doesn't tune it. Reorder the top 20 by the recorded scores, keep the rest in place, and make the second of the chapter's [planned comparisons](#ranking), established below p = 0.01: Example `cross-encoder-on-semantic` (Python 3.12.15 · bm25s 0.3.13 · lightgbm 4.7.0 · numpy 2.5.4 · pandas 3.0.6 · pystemmer 3.1.0 · scikit-learn 1.9.1 · scipy 1.18.1), passing in Edition 2026.10.1: ```python from lib.metrics import evaluate from lib.ranking import DEPTH, rerank, report_planned from lib.recorded import cross_encoder_scores, llm_listwise, manifest from lib.semantic import SemanticSearch from lib.store import load_queries, qrels queries, judgments = load_queries("test"), qrels("test") # scored on the human judgments semantic = SemanticSearch().run(queries, k=100) recorded = cross_encoder_scores() score = dict(zip(zip(recorded["query_id"], recorded["product_id"]), recorded["score"])) print("cross-encoder model:", manifest("cross-encoder")["model"]) print("licence in the manifest:", manifest("cross-encoder")["model_licence"]) shown = llm_listwise().groupby("query_id")["product_id"].apply(set) # what the LLM rerankers saw same = sum(set(semantic[query_id][:DEPTH]) == shown[query_id] for query_id in queries.index) print(f"rerank depth: {DEPTH}") print(f"queries whose top {DEPTH} holds the products the llm rerankers saw: {same} of {len(queries)}") reranked = rerank(semantic, score, DEPTH) # top 20 reordered, the rest in place before, after = evaluate(semantic, judgments), evaluate(reranked, judgments) for name, frame in [("semantic", before), ("cross-encoder", after)]: print(f"{name}, test ndcg@10: {frame['ndcg@10'].mean():.3f}") print(f"{name}, test judged@10: {frame['judged@10'].mean():.3f}") print(f"{name}, unjudged places in the test top 10s: {round((1 - frame['judged@10']).sum() * 10)}") moved = sum(reranked[query_id][0] != semantic[query_id][0] for query_id in queries.index) print(f"queries whose first product changes: {moved}") report_planned("cross-encoder", after["ndcg@10"], before["ndcg@10"]) ``` Output: ```text cross-encoder model: Alibaba-NLP/gte-reranker-modernbert-base licence in the manifest: Apache-2.0 rerank depth: 20 queries whose top 20 holds the products the llm rerankers saw: 30 of 30 semantic, test ndcg@10: 0.830 semantic, test judged@10: 0.990 semantic, unjudged places in the test top 10s: 3 cross-encoder, test ndcg@10: 0.813 cross-encoder, test judged@10: 0.983 cross-encoder, unjudged places in the test top 10s: 5 queries whose first product changes: 23 cross-encoder, mean difference: -0.016 cross-encoder, wins, losses, ties: 11, 12, 7 cross-encoder, permutation p: 0.178 cross-encoder, established at 0.01: no cross-encoder, t-test p: 0.171 cross-encoder, t-test agrees: yes cross-encoder, 99% interval: -0.058 to 0.006 cross-encoder, interval agrees: yes ``` The top 20 holds the products the LLM rerankers saw for 30 of 30 test queries. The reranker changes the first product for 23 test queries. [Judged@10](#score-every-test-query) falls from 0.990 to 0.983: the unjudged places across the test queries' top 10s go from 3 to 5. The rerank scores 0.813 against 0.830. No improvement shown, with a permutation p-value of 0.178 ([all five comparisons](#ranking-decisions)). On the train queries, the same rerank scores 0.802 against semantic retrieval's 0.772 ([depth sweep](#rerank-depth-from-budget)). That is another sample, not evidence, and it repeats what [chapter 3](#split-changes-the-verdict) found: two close rankers' order can flip between sets of queries. **What to do:** rerank your strongest retrieval, at a depth fixed before you look at test results, and adopt the reranker only on an established gain on held-out queries. For the store, the general-purpose cross-encoder stays out of the pipeline. ## Score each product with an LLM {#llm-pointwise-reranking} An LLM can rerank too, in three ways. **Pointwise**, it judges each product alone and returns a label or a score. **Pairwise**, it says which of two products fits the query better. **Listwise**, it orders a whole list in one answer ([Qin et al., 2024](https://arxiv.org/abs/2306.17563)). In one study, on eight datasets from BEIR, a collection of retrieval benchmarks used to test models on data they weren't trained for, pointwise prompts did better with graded answers than with yes or no, because "the lack of intermediate relevance label options may cause the LLM to provide noisy or biased answers" ([Zhuang et al., 2024a](https://arxiv.org/abs/2310.14122)). The store's LLM, the open model named in [the chapter's opening](#ranking), saw the query and one product at a time: name, class, category, description and attributes ([what the LLM saw](#what-the-llm-saw)). The prompt asks for a label, `exact`, `partial` or `irrelevant`, and "a score from 0 to 100 for how well it matches". Reorder [semantic retrieval](#search-by-meaning)'s top 20 by that score; when two products tie, they keep semantic retrieval's order, a rule fixed with the recording. This is the third of the chapter's [planned comparisons](#ranking), established below p = 0.01: Example `llm-pointwise-rerank` (Python 3.12.15 · bm25s 0.3.13 · lightgbm 4.7.0 · numpy 2.5.4 · pandas 3.0.6 · pystemmer 3.1.0 · scikit-learn 1.9.1 · scipy 1.18.1), passing in Edition 2026.10.1: ```python from lib.metrics import evaluate from lib.ranking import DEPTH, rerank, report_planned from lib.recorded import llm_pointwise, manifest from lib.semantic import SemanticSearch from lib.store import load_queries, qrels queries, judgments = load_queries("test"), qrels("test") # scored on the human judgments, not the LLM's semantic = SemanticSearch().run(queries, k=100) answers = llm_pointwise() # one call per product: a label and a score print("llm:", manifest("llm-rerank")["model"]) print("licence in the manifest:", manifest("llm-rerank")["model_licence"]) print(f"calls per query: {len(answers) // answers['query_id'].nunique()}") print(f"answers without a score: {answers['score'].isna().sum()}") print(f"distinct scores per query, median: {answers.groupby('query_id')['score'].nunique().median():.0f}") print(f"share of products scored 100: {(answers['score'] == 100).mean():.3f}") people = answers.apply(lambda row: judgments[row["query_id"]].get(row["product_id"]), axis=1) for label, grade in [("exact", 2), ("partial", 1), ("irrelevant", 0)]: print(f"products the llm labels {label}: {(answers['label'] == label).sum()}") print(f"products people judged {label}: {(people == grade).sum()}") print(f"products nobody judged: {people.isna().sum()}") score = dict(zip(zip(answers["query_id"], answers["product_id"]), answers["score"])) reranked = rerank(semantic, score, DEPTH) # equal scores keep semantic retrieval's order counts = answers.groupby("query_id")["score"].value_counts() # products per score, for each query shared = {query_id: sum(counts[(query_id, score[(query_id, product_id)])] > 1 for product_id in reranked[query_id][:10]) for query_id in queries.index} # top-10 products that share their score print(f"queries where most of the reranked top 10 shares its score: {sum(n > 5 for n in shared.values())}") before, after = evaluate(semantic, judgments), evaluate(reranked, judgments) for name, frame in [("semantic", before), ("llm pointwise", after)]: print(f"{name}, test ndcg@10: {frame['ndcg@10'].mean():.3f}") print(f"{name}, test judged@10: {frame['judged@10'].mean():.3f}") print(f"{name}, unjudged places in the test top 10s: {round((1 - frame['judged@10']).sum() * 10)}") report_planned("llm pointwise", after["ndcg@10"], before["ndcg@10"]) ``` Output: ```text llm: gemma4:12b-it-q4_K_M licence in the manifest: Apache-2.0 calls per query: 20 answers without a score: 0 distinct scores per query, median: 4 share of products scored 100: 0.390 products the llm labels exact: 251 products people judged exact: 310 products the llm labels partial: 345 products people judged partial: 265 products the llm labels irrelevant: 4 products people judged irrelevant: 7 products nobody judged: 18 queries where most of the reranked top 10 shares its score: 30 semantic, test ndcg@10: 0.830 semantic, test judged@10: 0.990 semantic, unjudged places in the test top 10s: 3 llm pointwise, test ndcg@10: 0.837 llm pointwise, test judged@10: 0.983 llm pointwise, unjudged places in the test top 10s: 5 llm pointwise, mean difference: 0.008 llm pointwise, wins, losses, ties: 11, 8, 11 llm pointwise, permutation p: 0.241 llm pointwise, established at 0.01: no llm pointwise, t-test p: 0.244 llm pointwise, t-test agrees: yes llm pointwise, 99% interval: -0.007 to 0.026 llm pointwise, interval agrees: yes ``` The recording made 20 calls per query, and the number of answers without a score is 0. The scores are coarse: the median query gets 4 distinct scores for its 20 products, and a share of 0.390 of all products score 100. In 30 queries, most of the reranked top 10 shares its score with another product, so the tie-break, semantic retrieval's order, decides much of the list. The LLM labels 251 products exact, 345 partial and 4 irrelevant. People judged 310 of the same products exact, 265 partial and 7 irrelevant, and nobody judged 18. So on these lists, with this prompt, the LLM calls fewer products exact than people did. [Chapter 4](#read-llm-error-direction) shows that the direction of an LLM's disagreement with people changes with the pairs judged and the prompt, so don't carry this one elsewhere. On test, [judged@10](#score-every-test-query) falls from 0.990 to 0.983, with unjudged places in the top 10s going from 3 to 5. The pointwise rerank scores 0.837 against 0.830. No improvement shown with this LLM, prompt and these queries, at a permutation p-value of 0.241 ([all five comparisons](#ranking-decisions)). **What to do:** if you rerank pointwise with an LLM, ask for a graded label or a score, count the distinct scores it gives per query, and decide the tie-break in advance: retrieval's order is a safe default. Score the result with human judgments, never with the same LLM's grades. ## Order the 20 products with one LLM prompt {#llm-listwise-reranking} The third way, a [listwise](#llm-pointwise-reranking) prompt, numbers the candidates and asks the LLM to return the numbers in order of relevance. RankGPT introduced this as permutation generation ([Sun et al., 2023](https://arxiv.org/abs/2304.09542)). A prompt can hold only so many products, so a long list is reranked in a **sliding window**: RankGPT reranks BM25's top 100 with "a window size of 20 and step size of 10", from the bottom of the list up (sections 3.2 and 6.1). The answer has to be a valid ordering, and that can fail: with LLMs of 2023, such as the 20-billion-parameter Flan-UL2, "listwise prompting almost always fails on moderate LLMs", through missing, repeated or refused rankings ([Qin et al., 2024](https://arxiv.org/abs/2306.17563)). The store's LLM, the open model named in [the chapter's opening](#ranking), saw 20 products per prompt, one window. The prompt shows each product as a short line: name, class, attributes and the first 160 characters of the description. Its last instruction reads: "Order all 20 products from the best match for the query to the worst. Return the product numbers, each exactly once, in JSON." The JSON schema requires exactly 20 numbers, but not 20 different ones, so the recording repairs any answer with a repeated or missing number and flags it. Reorder [semantic retrieval](#search-by-meaning)'s top 20 by the answer the LLM gave when shown them in semantic order, and make the fourth of the chapter's [planned comparisons](#ranking), established below p = 0.01: Example `llm-listwise-rerank` (Python 3.12.15 · bm25s 0.3.13 · lightgbm 4.7.0 · numpy 2.5.4 · pandas 3.0.6 · pystemmer 3.1.0 · scikit-learn 1.9.1 · scipy 1.18.1), passing in Edition 2026.10.1: ```python from lib.metrics import evaluate from lib.ranking import DEPTH, report_planned from lib.recorded import llm_listwise, manifest from lib.semantic import SemanticSearch from lib.store import load_queries, qrels queries, judgments = load_queries("test"), qrels("test") # scored on the human judgments, not the LLM's semantic = SemanticSearch().run(queries, k=100) answers = llm_listwise() # one call per query and order shown calls = answers.drop_duplicates(["query_id", "order"]) print(f"listwise calls, both orders: {len(calls)}") print(f"answers that needed repair: {(~calls['valid']).sum()}") print(f"products per call: {answers.groupby(['query_id', 'order']).size().max()}") print("prompt asks for:", manifest("llm-rerank")["prompts"]["listwise"].splitlines()[-1].replace("{n}", str(DEPTH))) shown_in_semantic_order = answers[answers["order"] == "semantic"].sort_values(["query_id", "position"]) order = shown_in_semantic_order.groupby("query_id")["product_id"].apply(list) reranked = {query_id: order[query_id] + ranking[DEPTH:] for query_id, ranking in semantic.items()} before, after = evaluate(semantic, judgments), evaluate(reranked, judgments) for name, frame in [("semantic", before), ("llm listwise", after)]: print(f"{name}, test ndcg@10: {frame['ndcg@10'].mean():.3f}") print(f"{name}, test judged@10: {frame['judged@10'].mean():.3f}") print(f"{name}, unjudged places in the test top 10s: {round((1 - frame['judged@10']).sum() * 10)}") moved = sum(reranked[query_id][0] != semantic[query_id][0] for query_id in queries.index) print(f"queries whose first product changes: {moved}") report_planned("llm listwise", after["ndcg@10"], before["ndcg@10"]) ``` Output: ```text listwise calls, both orders: 60 answers that needed repair: 0 products per call: 20 prompt asks for: Order all 20 products from the best match for the query to the worst. Return the product numbers, each exactly once, in JSON. semantic, test ndcg@10: 0.830 semantic, test judged@10: 0.990 semantic, unjudged places in the test top 10s: 3 llm listwise, test ndcg@10: 0.819 llm listwise, test judged@10: 0.983 llm listwise, unjudged places in the test top 10s: 5 queries whose first product changes: 19 llm listwise, mean difference: -0.010 llm listwise, wins, losses, ties: 12, 12, 6 llm listwise, permutation p: 0.580 llm listwise, established at 0.01: no llm listwise, t-test p: 0.498 llm listwise, t-test agrees: yes llm listwise, 99% interval: -0.070 to 0.016 llm listwise, interval agrees: yes ``` The recording made 60 listwise calls, two per test query, and the number of answers that needed repair is 0. The rerank changes the first product for 19 queries. [Judged@10](#score-every-test-query) falls from 0.990 to 0.983, with unjudged places in the top 10s going from 3 to 5. The rerank scores 0.819 against 0.830. No improvement shown with this LLM, prompt and these queries, at a permutation p-value of 0.580 ([all five comparisons](#ranking-decisions)). **What to do:** check every listwise answer before you use it: count missing, repeated and unknown numbers, and log how many answers needed repair. Constrain the answer's shape with a schema when your runtime supports one. Fix the window, the step and the order you show the products in before you measure, because the order changes the answer, as the [next check](#llm-position-bias) shows. ## Reverse the input to measure positional bias {#llm-position-bias} A [listwise answer](#llm-listwise-reranking) can suffer from **positional bias**: an LLM's tendency to let a product's place in the prompt affect its rank, not the click [position bias](#clicks-are-not-judgments) of shoppers. "Large language models (LLMs) exhibit positional bias in how they use context, which especially complicates listwise ranking" ([Tang et al., 2024](https://arxiv.org/abs/2310.07712)). On TREC DL19, the 2019 queries of TREC's Deep Learning track, a passage-ranking benchmark, gpt-3.5-turbo reranked BM25's top 100 to an nDCG@10 of 65.80 when shown BM25's order, 32.77 when shown it reversed and 25.17 when shown a random order ([Sun et al., 2023, Table 5](https://arxiv.org/abs/2304.09542)). The direct check is to show the same products in two orders and compare the answers. **Kendall's tau** measures how alike two orderings are, from 1 for the same order to −1 for the reverse, by counting the pairs they put the same way round ([chapter 4](#system-order-human-versus-llm)). The recording ran the store's LLM, the open model named in [the chapter's opening](#ranking), on each test query's listwise prompt twice: with [semantic retrieval](#search-by-meaning)'s top 20 in semantic order, as the [listwise rerank](#llm-listwise-reranking) used, and reversed. Before you run the comparison, predict: for how many of the 30 test queries will the LLM put the same product first both times? Example `listwise-position-bias` (Python 3.12.15 · bm25s 0.3.13 · lightgbm 4.7.0 · numpy 2.5.4 · pandas 3.0.6 · pystemmer 3.1.0 · scikit-learn 1.9.1 · scipy 1.18.1), passing in Edition 2026.10.1: ```python import numpy as np from scipy.stats import kendalltau from lib.metrics import evaluate from lib.ranking import DEPTH from lib.recorded import llm_listwise from lib.semantic import SemanticSearch from lib.store import load_queries, qrels queries, judgments = load_queries("test"), qrels("test") semantic = SemanticSearch().run(queries, k=DEPTH) # the order the first prompt showed answers = llm_listwise().sort_values(["query_id", "order", "position"]) answer = {key: list(group["product_id"]) for key, group in answers.groupby(["query_id", "order"])} def tau(first: list, second: list) -> float: """Kendall's tau between two orderings of the same products.""" place = {product_id: rank for rank, product_id in enumerate(second)} return kendalltau(range(len(first)), [place[product_id] for product_id in first]).statistic between, shown_semantic, shown_reversed, overlap, same_first = [], [], [], [], 0 for query_id, shown in semantic.items(): forward, backward = answer[(query_id, "semantic")], answer[(query_id, "reversed")] between.append(tau(forward, backward)) shown_semantic.append(tau(forward, shown)) shown_reversed.append(tau(backward, shown[::-1])) overlap.append(len(set(forward[:10]) & set(backward[:10]))) same_first += forward[0] == backward[0] print(f"test queries: {len(semantic)}") print(f"tau between the two answers, mean: {np.mean(between):.3f}") print(f"tau between the two answers, lowest: {min(between):.3f}\ntau between the two answers, highest: {max(between):.3f}") print(f"top-10 products in common, mean: {np.mean(overlap):.3f}") print(f"top-10 products in common, fewest: {min(overlap)}") print(f"queries with the same first product: {same_first}") print(f"share of queries with the same first product: {same_first / len(semantic):.3f}") print(f"tau between the order shown and the answer, semantic order: {np.mean(shown_semantic):.3f}") print(f"tau between the order shown and the answer, reversed order: {np.mean(shown_reversed):.3f}") print(f"the same, if the order shown made no difference: {-np.mean(shown_semantic):.3f}") reversed_run = {query_id: answer[(query_id, "reversed")] for query_id in semantic} print(f"reversed answer, test ndcg@10, not a planned comparison: {evaluate(reversed_run, judgments)['ndcg@10'].mean():.3f}") ``` Output: ```text test queries: 30 tau between the two answers, mean: 0.311 tau between the two answers, lowest: -0.484 tau between the two answers, highest: 0.811 top-10 products in common, mean: 6.967 top-10 products in common, fewest: 3 queries with the same first product: 13 share of queries with the same first product: 0.433 tau between the order shown and the answer, semantic order: 0.414 tau between the order shown and the answer, reversed order: 0.045 the same, if the order shown made no difference: -0.414 reversed answer, test ndcg@10, not a planned comparison: 0.826 ``` The same product comes first for 13 queries. The two answers agree with a mean tau of 0.311, and per query it runs from -0.484 to 0.811. Their top 10s share 6.967 products on average, and as few as 3. The answers also lean toward the order shown. Each answer's tau against its own input order is 0.414 for the semantic order and 0.045 for the reversed one. An LLM that ignored the order would give the same answer both times, so its reversed answer would have a tau of -0.414 against the reversed input. The reversed answer sits far nearer its own input order than that: positional bias, for this model, prompt and these queries. The reversed answer scores 0.826 nDCG@10; it is not a planned comparison, so the chapter doesn't test it. If you predicted that a capable model reads the products and finds the same best match whichever way they come, notice that it did so for a share of 0.433 of the queries. Three remedies exist. Pairwise ranking prompting asks about each pair twice: "for each pair of documents, we will inquire the LLM twice by swapping their order", and it counts a disagreement as a tie ([Qin et al., 2024](https://arxiv.org/abs/2306.17563)). Permutation self-consistency reranks several shuffled orders and aggregates the answers, which improved scores "by up to 7-18% for GPT-3.5 and 8-16% for LLaMA v2 (70B)" (Tang et al., 2024). And rerankers fine-tuned for listwise ranking, such as the open RankZephyr, are reported to be "resilient against variations in initial document ordering" ([Pradeep et al., 2023](https://arxiv.org/abs/2312.02724)); the store didn't record one. **What to do:** before you trust a listwise LLM reranker, run each query in at least two orders and report how much the answers agree. When agreement is low, aggregate several orders or ask each pair in both orders, and count those calls against [your budget](#rerank-depth-from-budget). In production, show the products in retrieval's order every time, so that the same list at least gets the same answer. ## Set the rerank depth from a budget {#rerank-depth-from-budget} A search pipeline runs in stages, each slower and given fewer candidates than the last, and every reranker in this chapter adds one. Multi-stage rankers offer "the ability to trade off quality against latency by controlling the admission of candidates into each pipeline stage" ([Nogueira et al., 2019](https://arxiv.org/abs/1910.14424)). So the **rerank depth**, how many products the reranker reads per query, is a budget decision. Timings "depend on hardware and running-time choices" ([Peng et al., 2025](https://arxiv.org/abs/2507.06223)), so this section counts what drives them instead: model calls and the text they read, per query, multiplied by the model's size. The store's cross-encoder has 149M parameters ([model card](https://huggingface.co/Alibaba-NLP/gte-reranker-modernbert-base)), and its LLM about 12 billion ([model card](https://ai.google.dev/gemma/docs/core/model_card_4)), some 80 times as many. In the store, count calls and characters per test query for each reranker at depth 20, from the products' text and the recorded prompts, plus the calls two designs the store didn't record would need: Example `reranker-prompt-sizes` (Python 3.12.15 · bm25s 0.3.13 · lightgbm 4.7.0 · numpy 2.5.4 · pandas 3.0.6 · pystemmer 3.1.0 · scikit-learn 1.9.1 · scipy 1.18.1), passing in Edition 2026.10.1: ```python import numpy as np from lib.agreement import as_the_llm_saw from lib.ranking import DEPTH, listwise_line from lib.recorded import manifest from lib.semantic import SemanticSearch from lib.store import load_products, load_queries, product_text products, queries = load_products(), load_queries("test") semantic = SemanticSearch().run(queries, k=DEPTH) # the 20 products each reranker reads prompts, text = manifest("llm-rerank")["prompts"], product_text(products) cross_encoder, pointwise, listwise = [], [], [] # characters read per query for query_id, query in queries["query"].items(): shown = products.loc[semantic[query_id]] cross_encoder.append(sum(len(query) + len(text[product_id]) for product_id in semantic[query_id])) pointwise.append(sum(len(prompts["pointwise"].format(query=query, product=as_the_llm_saw(product))) for _, product in shown.iterrows())) lines = "\n".join(f"[{number}] {listwise_line(product)}" for number, (_, product) in enumerate(shown.iterrows(), 1)) listwise.append(len(prompts["listwise"].format(query=query, products=lines, n=DEPTH))) for name, calls, characters in [("cross-encoder", DEPTH, cross_encoder), ("llm pointwise", DEPTH, pointwise), ("llm listwise", 1, listwise)]: print(f"{name}, model calls per query: {calls}") print(f"{name}, characters read per query, mean: {np.mean(characters):,.0f}") print(f"llm pairwise, every pair in both orders, calls per query: {DEPTH * (DEPTH - 1)}") window, step, depth = 20, 10, 100 # a sliding window over a deeper list print(f"llm listwise over the top {depth}, window {window}, step {step}, calls per query: {(depth - window) // step + 1}") ``` Output: ```text cross-encoder, model calls per query: 20 cross-encoder, characters read per query, mean: 9,911 llm pointwise, model calls per query: 20 llm pointwise, characters read per query, mean: 17,864 llm listwise, model calls per query: 1 llm listwise, characters read per query, mean: 5,973 llm pairwise, every pair in both orders, calls per query: 380 llm listwise over the top 100, window 20, step 10, calls per query: 9 ``` | Reranker, top 20 | Model calls per query | Characters read per query | | --- | --- | --- | | Cross-encoder | 20 | 9,911 | | LLM, pointwise | 20 | 17,864 | | LLM, listwise | 1 | 5,973 | Pointwise repeats the instructions and shows each product in full, so it reads the most text, in independent calls. Listwise reads the least, in one call, but must write out 20 numbers in order. Pairwise prompting over all pairs in both orders would take 380 calls per query, and RankGPT's sliding window over a top 100 would take 9. In one published measurement on [TREC DL19](#llm-position-bias), reranking BM25's top 100 with Flan-T5-XL on one RTX A6000 GPU, pointwise yes-or-no prompting scored 0.650 nDCG@10 at 1.5 seconds per query, and pairwise prompting over all pairs 0.713 at 254.9 seconds, with 9,900 LLM calls ([Zhuang et al., 2024b](https://arxiv.org/abs/2310.09497)). [LambdaMART](#lambdamart-on-test-queries) with the cross-encoder as a feature needs a cross-encoder score for every candidate, a median of 131 pairs per query. For the cross-encoder, cost grows in step with depth, so the question is where quality stops growing. Sweep the depth on the train queries: Example `rerank-depth-on-train` (Python 3.12.15 · bm25s 0.3.13 · lightgbm 4.7.0 · numpy 2.5.4 · pandas 3.0.6 · pystemmer 3.1.0 · scikit-learn 1.9.1 · scipy 1.18.1), passing in Edition 2026.10.1: ```python from lib.metrics import evaluate from lib.ranking import rerank from lib.recorded import cross_encoder_scores from lib.semantic import SemanticSearch from lib.store import load_queries, qrels queries, judgments = load_queries("train"), qrels("train") # choosing a depth is tuning: train queries semantic = SemanticSearch().run(queries, k=100) recorded = cross_encoder_scores() score = dict(zip(zip(recorded["query_id"], recorded["product_id"]), recorded["score"])) train_pairs = recorded[recorded["query_id"].isin(queries.index)] print(f"depth 0 (semantic alone), train ndcg@10: {evaluate(semantic, judgments)['ndcg@10'].mean():.3f}") previous = round(evaluate(semantic, judgments)["ndcg@10"].mean(), 3) # changes between the printed values for depth in [5, 10, 20, 30, 50, 75]: mean = round(evaluate(rerank(semantic, score, depth), judgments)["ndcg@10"].mean(), 3) print(f"depth {depth}, train ndcg@10: {mean:.3f}") print(f"depth {depth}, change from the depth before: {mean - previous:+.3f}") previous = mean print(f"pairs per query to score every candidate, median: {train_pairs.groupby('query_id').size().median():.0f}") ``` Output: ```text depth 0 (semantic alone), train ndcg@10: 0.772 depth 5, train ndcg@10: 0.782 depth 5, change from the depth before: +0.010 depth 10, train ndcg@10: 0.784 depth 10, change from the depth before: +0.002 depth 20, train ndcg@10: 0.802 depth 20, change from the depth before: +0.018 depth 30, train ndcg@10: 0.804 depth 30, change from the depth before: +0.002 depth 50, train ndcg@10: 0.793 depth 50, change from the depth before: -0.011 depth 75, train ndcg@10: 0.788 depth 75, change from the depth before: -0.005 pairs per query to score every candidate, median: 131 ``` From semantic retrieval's 0.772, reranking the top 5 gives 0.782 and the top 20 0.802. The top 30 adds +0.002 for half as many pairs again. Deeper, it falls: 0.793 at 50 and 0.788 at 75. The curve flattens between 20 and 30, near the depth fixed to match the LLM recording. On test, [the depth-20 rerank](#rerank-semantic-with-cross-encoder) showed no improvement, so for the store the cheapest reranker, none, is also the one to keep. **What to do:** for each reranker you consider, count calls and characters per query at each depth, and set the depth on train queries. A rule of thumb, not a measurement: rerank only as deep as the depth where train nDCG@10 stops improving, because every product past it costs the same and adds nothing. Then time that configuration on your own hardware and traffic, and keep the stage only if it fits the latency budget. ## Distill the cross-encoder into a small ranker {#distill-the-cross-encoder} A reranker too slow for [the budget](#rerank-depth-from-budget) can teach a fast one. **Distillation** trains a small model, the **student**, to imitate a large one, the **teacher**. Hinton et al. call it "distilling the knowledge in an ensemble of models into a single model" ([Hinton et al., 2015](https://arxiv.org/abs/1503.02531)). In ranking, the teacher's scores become the student's labels. When the two models' scores have different scales, because "different ranking architectures tend to produce output scores in different magnitudes", the student can learn the margin between two products' scores instead, with a loss called Margin-MSE ([Hofstätter et al., 2020](https://arxiv.org/abs/2010.02666)). Distillation also runs from LLMs down: in RankGPT, "a distilled 440M model outperforms a 3B supervised model on the BEIR benchmark" ([Sun et al., 2023](https://arxiv.org/abs/2304.09542); [BEIR](#llm-pointwise-reranking)), and in one product-search study of queries with superlatives, where "direct LLM-based reranking faces prohibitive latency", lightweight models took over the LLM's interpretations ([Zhu et al., 2025](https://arxiv.org/abs/2511.13994)). The teacher needs no human labels and labels every candidate. In the store, the teacher is the cross-encoder. The student is a LightGBM regression on the 7 [features](#ranking-features) that need no model call beyond the one retrieval already makes: everything but `cross_encoder`. Its label is the teacher's raw score, which a tree regression can match on the teacher's own scale. It ranks the same [candidates](#ranking-features), in `product_id` order, with trees the size [chosen for LambdaMART](#train-lambdamart). First, compare the student with a LambdaMART model trained on human grades with the same features, by cross-validation on the train queries: Example `distillation-on-train` (Python 3.12.15 · bm25s 0.3.13 · lightgbm 4.7.0 · numpy 2.5.4 · pandas 3.0.6 · pystemmer 3.1.0 · scikit-learn 1.9.1 · scipy 1.18.1), passing in Edition 2026.10.1: ```python from lightgbm import LGBMRanker, LGBMRegressor from lib.metrics import evaluate from lib.ranking import CHEAP_FEATURES, DETERMINISTIC, LAMBDARANK, TREES, cross_validate, feature_table, ranked from lib.semantic import SemanticSearch from lib.significance import compare from lib.store import load_products, load_queries, qrels queries, judgments = load_queries("train"), qrels("train") # design work: train queries only candidates = feature_table(load_products(), queries) candidates["grade or 0"] = candidates["grade"].fillna(0).astype(int) student = cross_validate(lambda: LGBMRegressor(**TREES, **DETERMINISTIC), candidates, candidates, "cross_encoder", CHEAP_FEATURES, judgments) # labels from the teacher human = cross_validate(lambda: LGBMRanker(**LAMBDARANK, **TREES, **DETERMINISTIC), candidates, candidates, "grade or 0", CHEAP_FEATURES, judgments) # labels from people teacher = evaluate(ranked(candidates, candidates["cross_encoder"]), judgments)["ndcg@10"] semantic = evaluate(SemanticSearch().run(queries, k=100), judgments)["ndcg@10"] for name, scores in [("semantic", semantic), ("teacher", teacher), ("student, teacher's labels", student), ("lambdarank, human grades, same features", human)]: print(f"{name}, train ndcg@10: {scores.mean():.3f}") result = compare(student, human) print(f"student minus the human-graded ranker: {result['mean difference']:.3f}") print(f"student minus the human-graded ranker, permutation p: {result['permutation p']:.3f}") ``` Output: ```text semantic, train ndcg@10: 0.772 teacher, train ndcg@10: 0.785 student, teacher's labels, train ndcg@10: 0.791 lambdarank, human grades, same features, train ndcg@10: 0.747 student minus the human-graded ranker: 0.044 student minus the human-graded ranker, permutation p: 0.002 ``` The student scores 0.791, and the teacher's own order of the candidates 0.785. The model trained on human grades scores 0.747, below semantic retrieval's 0.772, and the student minus it is 0.044, with a permutation p-value of 0.002. That is a tuning check on train queries, not evidence. On these queries, with these features, the teacher's labels on every candidate trained a better ranker than people's grades with unjudged candidates counted as 0. Then fit the student on the train queries, with early stopping on the same held-out train queries as [LambdaMART](#lambdamart-on-test-queries), and make the last of the chapter's [planned comparisons](#ranking), established below p = 0.01: Example `distilled-ranker-on-test` (Python 3.12.15 · bm25s 0.3.13 · lightgbm 4.7.0 · numpy 2.5.4 · pandas 3.0.6 · pystemmer 3.1.0 · scikit-learn 1.9.1 · scipy 1.18.1), passing in Edition 2026.10.1: ```python import numpy as np from lightgbm import LGBMRegressor from scipy.stats import kendalltau from lib.metrics import evaluate from lib.ranking import CHEAP_FEATURES, DETERMINISTIC, TREES, feature_table, fit, ranked, report_planned, stopping_split from lib.semantic import SemanticSearch from lib.store import load_products, load_queries, qrels products, train, test = load_products(), load_queries("train"), load_queries("test") table = feature_table(products, train) # the teacher's score is the label: no human grades print(f"train candidates the teacher labels: {len(table)}") print(f"of them, judged by nobody: {table['grade'].isna().sum()}") print(f"student features: {len(CHEAP_FEATURES)}") student = fit(LGBMRegressor(**TREES, **DETERMINISTIC), table, "cross_encoder", CHEAP_FEATURES, *stopping_split(train.index)) print(f"trees kept by early stopping: {student.best_iteration_}") candidates, judgments = feature_table(products, test), qrels("test") # test queries: read once, to report predicted = student.predict(candidates[CHEAP_FEATURES]) gain = student.booster_.feature_importance(importance_type="gain") for name, share in sorted(zip(CHEAP_FEATURES, gain / gain.sum()), key=lambda pair: -pair[1])[:3]: print(f"student's share of gain, {name}: {share:.3f}") for name in ["cross_encoder", "similarity"]: # whose order does the student follow? tau = [kendalltau(group[name], predicted[group.index]).statistic for _, group in candidates.groupby("query_id")] print(f"tau between student and {name}, mean over test queries: {np.mean(tau):.3f}") teacher = evaluate(ranked(candidates, candidates["cross_encoder"]), judgments) print(f"teacher's own order of the candidates, test ndcg@10, not a planned comparison: {teacher['ndcg@10'].mean():.3f}") after = evaluate(ranked(candidates, predicted), judgments) before = evaluate(SemanticSearch().run(test, k=100), judgments) for name, frame in [("semantic", before), ("distilled", after)]: print(f"{name}, test ndcg@10: {frame['ndcg@10'].mean():.3f}") print(f"{name}, test judged@10: {frame['judged@10'].mean():.3f}") print(f"{name}, unjudged places in the test top 10s: {round((1 - frame['judged@10']).sum() * 10)}") report_planned("distilled", after["ndcg@10"], before["ndcg@10"]) ``` Output: ```text train candidates the teacher labels: 6896 of them, judged by nobody: 1975 student features: 7 trees kept by early stopping: 202 student's share of gain, similarity: 0.812 student's share of gain, name_match: 0.126 student's share of gain, bm25: 0.055 tau between student and cross_encoder, mean over test queries: 0.466 tau between student and similarity, mean over test queries: 0.753 teacher's own order of the candidates, test ndcg@10, not a planned comparison: 0.782 semantic, test ndcg@10: 0.830 semantic, test judged@10: 0.990 semantic, unjudged places in the test top 10s: 3 distilled, test ndcg@10: 0.826 distilled, test judged@10: 0.987 distilled, unjudged places in the test top 10s: 4 distilled, mean difference: -0.004 distilled, wins, losses, ties: 10, 13, 7 distilled, permutation p: 0.720 distilled, established at 0.01: no distilled, t-test p: 0.694 distilled, t-test agrees: yes distilled, 99% interval: -0.034 to 0.017 distilled, interval agrees: yes ``` The teacher labels all 6896 train candidates, 1975 of them judged by nobody. Early stopping keeps 202 trees. The student leans on `similarity`, which takes a share of 0.812 of the gain, ahead of `name_match` at 0.126 and `bm25` at 0.055. On the test candidates, its scores agree with semantic similarity at a mean tau of 0.753 per query, and with the teacher's at 0.466. So the student learned mostly semantic retrieval's order, nudged toward the teacher's by name and BM25 matches. On test, [judged@10](#score-every-test-query) is 0.987 for the student and 0.990 for semantic retrieval, with 4 and 3 unjudged places in the top 10s. The distilled ranker scores 0.826 against semantic retrieval's 0.830. No improvement shown, with a permutation p-value of 0.720 ([all five comparisons](#ranking-decisions)). The teacher's own order of the test candidates scores 0.782, which the chapter didn't plan to test. **What to do:** distil when a reranker earns an established gain but costs too much per query: label every candidate of your train queries with the teacher, train the student on features you can compute without the teacher, and test the student against your retrieval on held-out queries. Read the student's gain importance: when retrieval's own score dominates it, the student is mostly your retrieval. Use margins between pairs when the student can't match the teacher's scale. The store has no teacher that beats semantic retrieval yet, so it keeps the code and doesn't ship the student. ## What goes wrong in production {#ranking-in-production} A live ranker computes features and calls models while the shopper waits, with code and model versions that can drift from those it was trained and tested with. **Features drift between training and serving.** If the live system tokenizes differently, rescales BM25 or upgrades the embedding model, the learned weights meet different numbers, and nothing raises an error. Compute features with one shared code path, and log them with each served ranking. **Row order leaks into training.** LightGBM's lambdarank starts from the order of its input rows, and a ranking breaks ties somehow. A file sorted by a reranker's score hands the model that reranker's ranking. Sort candidates by an ID, or shuffle them with a fixed seed, before every fit and every scoring. **Judgments come from a pool.** WANDS judged the products that shoppers' clicks, a lexical engine, a neural retriever and a query classifier put forward ([chapter 2](#pooled-judgments)). Counting unjudged candidates as irrelevant teaches a model to demote whatever those systems didn't find. Check [judged@10](#score-every-test-query) for every ranker you add, and have its unjudged products judged before you trust a gain. **LLM judges favour LLM rerankers.** In one study, "the system rankings obtained by LLM assessors overestimate LLM-based re-rankers by 9 to 17 positions on average" ([Fröbe et al., 2025](https://eprints.gla.ac.uk/352747)), and using LLMs both to rank and to judge is circular, "illustrating how reliance on LLM-based assessments can distort system rankings" ([Clarke and Dietz, 2024](https://arxiv.org/abs/2412.17156)). Score LLM rerankers with human judgments, as this chapter does. **An LLM reranker belongs to one model version and one prompt.** A different digest (the hash that identifies the exact weights), quantization (the precision the weights are stored at) or prompt wording can change every answer, and with [positional bias](#llm-position-bias), so can a change in retrieval's order. The LLM recording's manifest records the digest and both prompts verbatim; re-measure whenever either changes. **Unfamiliar queries can do worse.** A comparison of 22 reranking methods found that LLM rerankers do better on familiar queries, while "their generalization ability to novel queries varies" ([Abdallah et al., 2025](https://arxiv.org/abs/2508.16757)). Watch queries about products and styles newer than the model's training data. **Every reranker adds a call on the shopper's path.** Count calls and text per query, as in [the budget](#rerank-depth-from-budget), set a timeout, and fall back to retrieval's order when the reranker doesn't answer in time. **What to do:** log features and model versions with every ranking, put candidates in an order that carries no information, score on human judgments, check judged@10, and give each reranking stage a time limit and a fallback. ## Decisions this chapter made {#ranking-decisions} The chapter made each choice below on the train queries, and only an established planned comparison on the test queries could change the store's search. - **5 planned test comparisons** against semantic retrieval, on human judgments, established below 0.01. - **Candidates: the union of BM25's and semantic retrieval's top 100,** in `product_id` order, with score ties broken by `product_id`. - **LambdaMART: `label_gain=[0, 1, 2]`, unjudged candidates as grade 0, 3 leaves at a learning rate of 0.1,** chosen by cross-validation on train, with early stopping on held-out train queries through `eval_X` and `eval_y`. No improvement shown on test, so it isn't used. - **Rerank depth: semantic retrieval's top 20,** fixed to match the LLM recording; on train, the cross-encoder's curve flattens between 20 and 30. No reranker shows an improvement on test, so none is used. - **LLM rerankers are scored on human judgments only,** and the listwise one is checked in two input orders. Its answer depends on the order it is shown. - **Costs are counted, not timed:** calls and characters per query. - **Distillation: a LightGBM student on the cross-encoder's scores,** with no model call beyond retrieval's. It learns mostly semantic retrieval's order, shows no improvement, and isn't used. - **The ranker to beat stays semantic retrieval alone,** with BM25, unstemmed, as the baseline. The 5 planned comparisons, each on nDCG@10 against semantic retrieval's 0.830 on the test queries, established below 0.01: | Measure | [LambdaMART](#lambdamart-on-test-queries) | [Cross-encoder](#rerank-semantic-with-cross-encoder) | [LLM pointwise](#llm-pointwise-reranking) | [LLM listwise](#llm-listwise-reranking) | [Distilled](#distill-the-cross-encoder) | | --- | --- | --- | --- | --- | --- | | nDCG@10 | 0.817 | 0.813 | 0.837 | 0.819 | 0.826 | | Mean difference | -0.012 | -0.016 | 0.008 | -0.010 | -0.004 | | Wins / losses / ties | 14, 12, 4 | 11, 12, 7 | 11, 8, 11 | 12, 12, 6 | 10, 13, 7 | | Permutation p | 0.260 | 0.178 | 0.241 | 0.580 | 0.720 | | Established | no | no | no | no | no | | t-test p | 0.278 | 0.171 | 0.244 | 0.498 | 0.694 | | 99% interval | -0.044 to 0.011 | -0.058 to 0.006 | -0.007 to 0.026 | -0.070 to 0.016 | -0.034 to 0.017 | | t-test agrees | yes | yes | yes | yes | yes | | Interval agrees | yes | yes | yes | yes | yes | # Retrieval for LLM applications By the end of this chapter you can measure the retrieval behind an LLM that answers from it: score an answer by the products it cites, check each citation against what retrieval returned, measure the products the LLM reads, choose what to index as one retrievable unit, and read an agent's searches from the record of its tool calls. The store, an online furniture and home-goods retailer and this book's case study ([chapter 1](#store-data)), shows shoppers a ranked list. A shopping assistant answers in words instead: a shopper writes what they want, and a **large language model** (LLM), a model trained on large amounts of text to generate text in response to a prompt, recommends a few products and cites each one by its ID. The prompt tells the LLM to recommend only the products that search puts in front of it, so the assistant can be no better than its retrieval. The same holds for an **agent**, an LLM that runs the searches itself and decides how many to run. This chapter measures both with the store's human judgments. The store's LLM is the open model `gemma4:12b-it-q4_K_M`. We chose it because it is open and runs locally, so anyone can rerun the **recordings**: `recordings/rag_answers.py` and `recordings/agent_trace.py` made each model call once, at temperature 0 with a fixed seed and thinking off ([chapter 9](#ranking)), and saved the replies, which the examples read through `lib.recorded`. Each recording's **manifest**, the file that describes it, gives the model's licence as Apache-2.0. For an answer, the LLM read each product's name, class, category, description and attributes as text, and the agent's search tool shows it less. The human judges also saw each product's image and web page ([what the LLM saw](#what-the-llm-saw)). [Semantic retrieval](#search-by-meaning) ranks by the recorded embeddings of `Snowflake/snowflake-arctic-embed-m-v1.5`, and [BM25](#rank-with-bm25) is the book's unstemmed baseline, `lib.lexical`. Every finding about the LLM holds for this model, these prompts and the 30 [test queries](#store-data), not for LLMs in general. Before reading any test result, the chapter fixes 2 planned comparisons on the test queries, both of semantic retrieval against BM25: 1. behind the recorded answers, on the share of each answer's cited products that people judged exact ([Score each answer](#retrieval-bounds-the-answer)) 2. behind the agent's search tool, on precision@3 of the products it picks ([Compare the retrievers](#retriever-behind-agent-tool)) By [the book's rule](#when-a-difference-is-established), a difference is established when its paired [permutation test](#permutation-test-on-queries) gives a p-value below 0.025, 0.05 divided by the number of planned comparisons. The [paired t-test](#paired-t-test-on-queries) and a [bootstrap interval](#report-the-size-of-a-gap) at the 97.5% level stand beside it. Anything else on the test queries is a description, or labelled exploratory. ## Put retrieved products in the prompt {#put-retrieved-products-in-the-prompt} **Retrieval-augmented generation** (RAG), the design behind a shopping assistant, pairs an LLM with a retriever. For each request, the retriever finds a few texts in an index, and the LLM writes its answer from them. In Lewis et al.'s RAG models, the "non-parametric memory is a dense vector index of Wikipedia, accessed with a pre-trained neural retriever". The authors note two open problems of language models on their own: "providing provenance for their decisions and updating their world knowledge remain open research problems" ([Lewis et al., 2020](https://arxiv.org/abs/2005.11401)). A store has both problems: its catalog changes, and an answer should say which products it recommends. The retrieved text goes into the **prompt**, the input the model reads, and that text is the **context**. A model reads at most a fixed number of [tokens](#turn-text-into-embeddings), the word pieces of its own vocabulary. That limit is its **context window**. Why not put the whole catalog in the prompt and skip retrieval? It has to fit, and even when it fits, models don't read a long prompt evenly. Liu et al. found that "performance is often highest when relevant information occurs at the beginning or end of the input context", and that it "significantly degrades when models must access relevant information in the middle of long contexts" ([Liu et al., 2024](https://aclanthology.org/2024.tacl-1.9/)). A later study found that this lost-in-the-middle effect, "the LiM effect", "is strongest when inputs occupy up to 50% of a model's context window", and that near the limit it gives way to a bias toward the end ([Veseli et al., 2025](https://arxiv.org/abs/2508.07479)). More passages aren't always better either. As Jin et al. gave long-context LLMs more retrieved passages, "the quality of generated output initially improves first, but then subsequently declines", and they identify retrieved hard negatives, passages that look relevant but aren't, as a key contributor to the decline ([Jin et al., 2024](https://arxiv.org/abs/2410.05983)). One vendor's engineering post suggests the opposite for small collections: "If your knowledge base is smaller than 200,000 tokens (about 500 pages of material)," put all of it in the prompt ([Anthropic, 2024](https://www.anthropic.com/news/contextual-retrieval)). That is a vendor's rule of thumb, not a measurement. In the store, the recording of answers ([Score each answer](#retrieval-bounds-the-answer)) put five products in each prompt, after these instructions: "You help shoppers at an online furniture and home-goods store. Using only the products listed below, answer the shopper. Recommend up to three products that fit, cite each one by its ID, and say plainly if none of them fit. Don't mention products that aren't listed." Count the words of the whole catalog as the LLM would read it, and compare: Example `catalog-against-the-context` (Python 3.12.15 · bm25s 0.3.13 · numpy 2.5.4 · pandas 3.0.6 · scipy 1.18.1), passing in Edition 2026.10.1: ```python from statistics import median from lib.agreement import as_the_llm_saw from lib.recorded import manifest, rag_answers from lib.store import load_products recording = manifest("rag-answers") window = recording["options"]["num_ctx"] print("model:", recording["model"]) print("licence in the manifest:", recording["model_licence"]) print("instructions:", recording["prompts"].split("\n")[0]) print("context window the recording set, in tokens:", window) products = load_products() words = {product_id: len(f"ID: {product_id}\n{as_the_llm_saw(product)}".split()) # each product as the LLM read it for product_id, product in products.iterrows()} instructions = len(recording["prompts"].split()) # the prompt's own words, without the query prompts = [instructions + sum(words[p] for p in ids) for ids in rag_answers()["context_ids"]] catalog = sum(words.values()) print(f"words in the catalog as the LLM reads it: {catalog:,}") print(f"words in one prompt with five products, median: {median(prompts):,.0f}") print(f"the catalog against one prompt: {catalog / median(prompts):,.0f} times") print(f"the catalog against the recording's window, at one token per word: {catalog / window:,.0f} times") print(f"the catalog against 200,000 tokens, at one token per word: {catalog / 200_000:.1f} times") ``` Output: ```text model: gemma4:12b-it-q4_K_M licence in the manifest: Apache-2.0 instructions: You help shoppers at an online furniture and home-goods store. Using only the products listed below, answer the shopper. Recommend up to three products that fit, cite each one by its ID, and say plainly if none of them fit. Don't mention products that aren't listed. context window the recording set, in tokens: 8192 words in the catalog as the LLM reads it: 1,091,544 words in one prompt with five products, median: 536 the catalog against one prompt: 2,035 times the catalog against the recording's window, at one token per word: 133 times the catalog against 200,000 tokens, at one token per word: 5.5 times ``` Even with the book's sample cutting descriptions to about 400 characters ([data card](../examples/data/wands/README.md)), the catalog holds 1,091,544 words, and a prompt with five products holds a median of 536: the catalog is 2,035 times larger. The examples have no tokenizer for this model, so they count words. The recording set a context window of 8192 tokens. If every word were a single token, the catalog would fill that window 133 times over, and the vendor's 200,000 tokens 5.5 times over. The store has to retrieve. **What to do:** put a few retrieved items in the prompt, not the collection, and choose how many by measuring answers on train queries at several sizes, because more context can help or hurt. The store's recording answered only the test queries, five products each, so five is a fixed setting here, not a measured one. Consider sending the whole collection only when it fits well inside the window, and then test it against retrieval on the same queries. ## Score each answer by the products it cites {#retrieval-bounds-the-answer} An LLM told to answer from a [context](#put-retrieved-products-in-the-prompt) should recommend only what the context holds, and [the ID check](#check-citations-against-context) tests that it does. If no exact product is in the context, no prompt makes a grounded answer recommend one. Retrieval sets a ceiling on the answer, and what else the context holds matters too. Cuconasu et al. found that the retriever's "highest-scoring documents that are not directly relevant to the query (e.g., do not contain the answer)" hurt answers, while, "Even more surprising, we discovered that adding random documents in the prompt improves the LLM accuracy by up to 35%" ([Cuconasu et al., 2024](https://arxiv.org/abs/2401.14887)). Products that look right but aren't are **distractors**, the store's version of those passages. In question answering, Joren et al. found a related pattern: when the context lacked what was needed, the larger models they tested, Gemini 1.5 Pro, GPT 4o and Claude 3.5, "often output incorrect answers instead of abstaining", while smaller ones, Mistral 3 and Gemma 2 among them, often hallucinated or abstained even when the context was sufficient ([Joren et al., 2024](https://arxiv.org/abs/2411.06037)). A **citation** here is the ID an answer gives for a product it recommends. With judgments, you can score an answer without reading it: look up each cited product's grade. The **exact share** of an answer is the share of its cited products that people judged [exact](#store-data). An answer that cites nothing scores 0, because it gives the shopper nothing that fits, whether or not that was the right call. A separate count of answers that recommend nothing covers that case. ```text exact share = (cited products judged exact) / (products cited), 0 when the answer cites nothing ``` The store's recording answered each test query three times, as "I'm looking for" followed by the query. Each answer came from a different context of five products: semantic retrieval's top 5, BM25's top 5, or five products drawn at random with a fixed seed. The random context is a floor, a context no retriever chose. The model, the prompt and the query stay the same, so the answers differ only because their contexts do. The examples read each context as the recording stored it, in the order the LLM read it. The BM25 contexts keep tied products in bm25s's own order, not by `product_id` as `lib.lexical` does, so 7 of the 30 differ from what `lib.lexical` returns, 6 only in order. The recording asked for JSON with the answer and the cited IDs in separate fields. `score_answers` in `lib.generation` scores each answer on the human grades. The example then makes the first of the chapter's [two planned comparisons](#retrieval-for-llm-applications), established if its permutation p-value is below 0.025. Before you run it, predict what the LLM does with the random context, where the prompt tells it to "say plainly if none of them fit". Of the 30 answers, how many will recommend nothing? Example `rag-answers-by-context` (Python 3.12.15 · bm25s 0.3.13 · numpy 2.5.4 · pandas 3.0.6 · scipy 1.18.1), passing in Edition 2026.10.1: ```python from lib.generation import PLANNED_TEST_COMPARISONS, THRESHOLD, report_planned, score_answers from lib.lexical import LexicalSearch from lib.recorded import rag_answers from lib.store import load_products, load_queries, qrels answers = rag_answers() stored = answers[answers["context"] == "bm25"].set_index("query_id")["context_ids"] # as the LLM read them rerun = LexicalSearch(load_products()).run(load_queries("test"), k=5) # ties by product_id differ = [q for q in stored.index if stored[q] != rerun[q]] print("bm25 contexts that differ from lib.lexical:", len(differ)) print("bm25 contexts that differ from lib.lexical only in order:", sum(set(stored[q]) == set(rerun[q]) for q in differ)) scores = score_answers(answers, qrels("test")) # one row per test query and context, on human grades for context, rows in scores.groupby("context", sort=False): held, nothing = rows["exact in context"] > 0, rows["cited"] == 0 print(f"{context}, unjudged products in the contexts: {rows['unjudged in context'].sum()}") print(f"{context}, contexts with an exact product: {held.sum()}") print(f"{context}, answers that cite an exact product: {(rows['exact cited'] > 0).sum()}") print(f"{context}, products cited: {rows['cited'].sum()}") print(f"{context}, cited products that are exact: {rows['exact cited'].sum()}") print(f"{context}, mean exact share: {rows['exact share'].mean():.3f}") print(f"{context}, answers that recommend nothing: {nothing.sum()}") print(f"{context}, recommend nothing, with an exact product in the context: {(nothing & held).sum()}") print(f"{context}, recommend something, with no exact product in the context: {(~nothing & ~held).sum()}") near = rows[~held] # answers whose context held no exact product print(f"{context}, products cited with no exact product in the context: {near['cited'].sum()}") print(f"{context}, of those, partial: {near['partial cited'].sum()}") print(f"{context}, unjudged products cited: {rows['unjudged cited'].sum()}") print("answers that cite more exact products than their context holds:", (scores["exact cited"] > scores["exact in context"]).sum()) print("planned test comparisons:", len(PLANNED_TEST_COMPARISONS)) print(f"p-value threshold: {THRESHOLD:.3f}") share = scores.pivot(index="query_id", columns="context", values="exact share") report_planned("answers, semantic over bm25", share["semantic"], share["bm25"]) ``` Output: ```text bm25 contexts that differ from lib.lexical: 7 bm25 contexts that differ from lib.lexical only in order: 6 semantic, unjudged products in the contexts: 0 semantic, contexts with an exact product: 25 semantic, answers that cite an exact product: 22 semantic, products cited: 80 semantic, cited products that are exact: 60 semantic, mean exact share: 0.658 semantic, answers that recommend nothing: 2 semantic, recommend nothing, with an exact product in the context: 1 semantic, recommend something, with no exact product in the context: 4 semantic, products cited with no exact product in the context: 11 semantic, of those, partial: 11 semantic, unjudged products cited: 0 bm25, unjudged products in the contexts: 10 bm25, contexts with an exact product: 24 bm25, answers that cite an exact product: 23 bm25, products cited: 73 bm25, cited products that are exact: 52 bm25, mean exact share: 0.639 bm25, answers that recommend nothing: 3 bm25, recommend nothing, with an exact product in the context: 0 bm25, recommend something, with no exact product in the context: 3 bm25, products cited with no exact product in the context: 7 bm25, of those, partial: 7 bm25, unjudged products cited: 1 random, unjudged products in the contexts: 147 random, contexts with an exact product: 1 random, answers that cite an exact product: 1 random, products cited: 1 random, cited products that are exact: 1 random, mean exact share: 0.033 random, answers that recommend nothing: 29 random, recommend nothing, with an exact product in the context: 0 random, recommend something, with no exact product in the context: 0 random, products cited with no exact product in the context: 0 random, of those, partial: 0 random, unjudged products cited: 0 answers that cite more exact products than their context holds: 0 planned test comparisons: 2 p-value threshold: 0.025 answers, semantic over bm25, mean difference: 0.019 answers, semantic over bm25, wins, losses, ties: 4, 5, 21 answers, semantic over bm25, permutation p: 0.765 answers, semantic over bm25, established at 0.025: no answers, semantic over bm25, t-test p: 0.715 answers, semantic over bm25, t-test agrees: yes answers, semantic over bm25, 97.5% interval: -0.078 to 0.161 answers, semantic over bm25, interval agrees: yes ``` | Measure | Semantic | BM25 | Random | | --- | --- | --- | --- | | Contexts with an exact product | 25 | 24 | 1 | | Answers that cite an exact product | 22 | 23 | 1 | | Products cited | 80 | 73 | 1 | | Cited products that are exact | 60 | 52 | 1 | | Mean exact share | 0.658 | 0.639 | 0.033 | | Answers that recommend nothing | 2 | 3 | 29 | Given random products, the LLM recommends nothing in 29 of the 30 answers. Its only citation is an exact product that the random draw happened to include. The random contexts hold 147 products nobody judged, which count as not exact. We expect few of them to fit, but no judgment says so. If you predicted that the LLM would force a recommendation, this model and prompt didn't, for random products. The ceiling holds: the number of answers that cite more exact products than their context holds is 0. Near misses are another matter. Where the context held no exact product, the answer still recommended something 4 times with semantic retrieval and 3 times with BM25, rather than say that no product fits exactly. Every product those answers cite is partial: 11 of 11 with semantic retrieval and 7 of 7 with BM25, the right kind of product with a modifier wrong. Whether that is a failure depends on whether the shopper wants near matches. The reverse failure is rarer: the answers that recommend nothing although the context held an exact product number 1 for semantic retrieval and 0 for BM25. BM25's contexts hold 10 unjudged products and semantic retrieval's 0. The score reads only cited products, and BM25's answers cite 1 unjudged product, which counts as not exact. Semantic retrieval's lead on [nDCG@10](#score-one-ranking) is established ([chapter 3](#compare-runs-in-a-harness)), but on the first planned comparison its difference from BM25 of 0.019 in mean exact share isn't, at a permutation p-value of 0.765 ([both comparisons](#retrieval-for-llm-applications-decisions)). **What to do:** score every answer by its cited products' human grades, and report beside it how many answers recommend nothing, split by whether the context held an exact product. Keep a random context as a floor whenever you change the model or the prompt: an answer that recommends something from random products has invented a fit. Test a retriever's gain on the answers too, because a gain in ranking needn't carry through. ## Check every citation against the context {#check-citations-against-context} **Grounding** means that an answer rests on the context it was given, as [the exact share](#retrieval-bounds-the-answer) assumes, not on what the model remembers or invents. Citations make grounding checkable, and **attribution** asks whether a cited source supports the statement it is attached to. Audits find that citations often don't. In a 2023 audit of generative search engines, "a mere 51.5% of generated sentences are fully supported by citations" and "only 74.5% of citations support their associated sentence" ([Liu et al., 2023](https://arxiv.org/abs/2304.09848)). On ELI5, the ALCE benchmark's long-form questions from the Reddit forum Explain Like I'm Five, with 2023's models, "even the best models lack complete citation support 50% of the time" ([Gao et al., 2023](https://arxiv.org/abs/2305.14627)). Check citations at two levels, cheapest first. The first is exact and needs no model: every cited ID must be one the context held. An ID from outside the context names a product the model remembered or made up, and either way the answer isn't grounded. The second level asks whether the cited product's text supports what the answer says about it, which needs a reader ([Check what the cited product supports](#check-citation-support)). In the store, the answer recording's schema asked for the cited IDs as a list of integers in a field of their own (`schemas` in the manifest). The schema describes the field's form; it says nothing about which IDs go in it, or how many. The example checks each of the [recorded answers](#retrieval-bounds-the-answer) against its context, and compares the field with the context IDs written in the answer's text. The text can't reveal an ID from outside the context, which would look like any other number an answer writes, such as a size. That is why the IDs belong in a field: Example `citations-inside-the-context` (Python 3.12.15 · bm25s 0.3.13 · numpy 2.5.4 · pandas 3.0.6 · scipy 1.18.1), passing in Edition 2026.10.1: ```python from lib.generation import ids_in_text from lib.recorded import rag_answers answers = rag_answers() outside = answers.apply(lambda a: [p for p in a["cited"] if p not in a["context_ids"]], axis=1) repeated = answers["cited"].map(lambda cited: len(cited) != len(set(cited))) written = answers.apply(lambda a: ids_in_text(a["answer"], a["context_ids"]), axis=1) # IDs the text writes out print("answers:", len(answers)) print("answers that cite an ID outside their context:", (outside.map(len) > 0).sum()) print("answers that cite one ID twice:", repeated.sum()) print("answers that cite nothing:", (answers["cited"].map(len) == 0).sum()) print("answers that cite more than three products:", (answers["cited"].map(len) > 3).sum()) print("answers whose text and cited IDs differ:", (written != answers["cited"].map(set)).sum()) ``` Output: ```text answers: 90 answers that cite an ID outside their context: 0 answers that cite one ID twice: 0 answers that cite nothing: 34 answers that cite more than three products: 3 answers whose text and cited IDs differ: 0 ``` Of the 90 answers, the check counts 0 that cite an ID outside their context, 0 that cite one ID twice, and 0 whose text and field disagree. The answers that cite nothing number 34. The instruction the answers do break is how many to recommend: 3 cite more than three products, though the prompt says "Recommend up to three". On this recording, the ID check finds no ungrounded citation. That is the result you want, and no reason to drop the check: it costs a set lookup per answer, and a different model, prompt or context length can fail it. Passing it says only that each recommended product was in front of the model, not that it fits. **What to do:** run the ID check on every answer, offline and before a shopper sees it, and treat an ID outside the context as a failed answer: drop it or retry, and never show it. Ask for citations in a field of their own so that the check doesn't parse prose, and check that the text and the field agree. Count the instructions the answers break, such as the number of recommendations, and enforce in code the ones that matter. ## Check what the cited product supports {#check-citation-support} The [second level of checking](#check-citations-against-context) asks whether the cited text supports the claim. **Faithfulness** is the share of an answer's claims that the context supports. In the Ragas documentation's words (its pages name no version; the ones opened were dated 2025-12-09), "A response is considered **faithful** if all its claims can be supported by the retrieved context" ([Ragas documentation](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/faithfulness/)). Checking it takes a reader. ALCE uses a model for natural language inference (NLI), the task of deciding whether one text entails another: "a T5-11B … model fine-tuned on a collection of natural language inference (NLI) datasets". A statement's citation recall is 1 only when its citations together entail it ([Gao et al., 2023, sections 3.2 and 3.3](https://arxiv.org/html/2305.14627)). An LLM can judge support too, with limits. In the 2024 RAG track of TREC, the Text REtrieval Conference's shared evaluations, Thakur et al. found that "human and GPT-4o predictions match perfectly (on a three-level scale)" on 56% of judgments made from scratch, "increasing to 72% in the manual with post-editing condition". They also found that "an independent human judge correlates better with GPT-4o than a human judge", so part of the disagreement is between people ([Thakur et al., 2025](https://arxiv.org/abs/2504.15205)). Support isn't relevance. A citation can be faithful to the product's text and still recommend the wrong product: the text says what the answer claims, and the product isn't what the shopper asked for. Only the judgments catch that. Read one of the store's answers by hand first: the one for the test query `medium size chandelier`, from the semantic context. Then count the human grades of every product cited in the [recorded answers](#retrieval-bounds-the-answer) from the semantic and BM25 contexts: Example `faithful-but-partial` (Python 3.12.15 · bm25s 0.3.13 · numpy 2.5.4 · pandas 3.0.6 · scipy 1.18.1), passing in Edition 2026.10.1: ```python from collections import Counter from lib.agreement import LABELS from lib.recorded import rag_answers from lib.store import load_products, load_queries, qrels products, queries, judgments = load_products(), load_queries("test"), qrels("test") answers = rag_answers().set_index(["query_id", "context"]) chandelier, claim = 58, "medium-sized" # one answer, read by hand answer, judged = answers.loc[(chandelier, "semantic")], judgments[chandelier] cited = answer["cited"][0] print("query:", queries.at[chandelier, "query"]) print("answer:", answer["answer"]) print("cited product:", products.at[cited, "name"]) print(f"the cited product's text says {claim}:", "yes" if claim in products.at[cited, "description"] else "no") print("the judges' label:", LABELS[judged[cited]]) print("exact products in this answer's context:", sum(judged.get(p, 0) == 2 for p in answer["context_ids"])) retrieved = answers.drop("random", level="context") # every citation from the two retrievers' contexts labels = Counter(LABELS.get(judgments[query_id].get(p), "unjudged") for (query_id, _), cited_ids in retrieved["cited"].items() for p in cited_ids) print("cited products, semantic and bm25 contexts:", sum(labels.values())) for label in ["exact", "partial", "irrelevant", "unjudged"]: print(f"cited products, {label}:", labels[label]) ``` Output: ```text query: medium size chandelier answer: I recommend the sybil 12 - light unique / statement classic / traditional chandelier with antler accents (ID: 8171), as it is specifically described as a perfect medium-sized lighting fixture. cited product: sybil 12 - light unique / statement classic / traditional chandelier with antler accents the cited product's text says medium-sized: yes the judges' label: partial exact products in this answer's context: 2 cited products, semantic and bm25 contexts: 153 cited products, exact: 112 cited products, partial: 40 cited products, irrelevant: 0 cited products, unjudged: 1 ``` The answer recommends one chandelier, saying it "is specifically described as a perfect medium-sized lighting fixture". The cited product's description does say `medium-sized` (yes), so a support check would pass the claim. The store's judges called the product partial: by [WANDS's definition](#store-data), the right kind of product, missing a modifier the shopper asked for. They saw its image and product page, which the LLM didn't. Its context held 2 exact products, and the answer recommended neither. For this answer, the citation is faithful, and the recommendation still misses. Across both retrievers' answers, the 153 cited products break down as 112 exact, 40 partial, 0 irrelevant and 1 unjudged. Every cited product that isn't exact is partial, apart from the 1 nobody judged: the kind of product the shopper asked for, with a modifier wrong by the judges' reading. **What to do:** score answers on your judgments first, because they say whether a recommendation fits. Add a support check for the claims an answer makes about each product, by an NLI model or an LLM judge. Calibrate that judge on a human sample, as [chapter 4 does](#calibrate-on-a-human-sample) and as ARES does: "To mitigate potential prediction errors," it uses "a small set of human-annotated datapoints for prediction-powered inference (PPI)", a method that corrects the judge's estimate with the human labels ([Saad-Falcon et al., 2024](https://arxiv.org/abs/2311.09476)). For queries you have no judgments for, a support check is all you have, so say so when you report it. ## Measure the context, then the answer {#measure-the-context} A RAG system can fail at retrieval or at generation, so measure the context as well as [the answers](#retrieval-bounds-the-answer). Ragas, a framework for evaluating RAG, scores several measures without reference answers, among them "the ability of the retrieval system to identify relevant and focused context passages", faithfulness and the answer's relevance; its context recall needs a reference ([Es et al., 2024](https://aclanthology.org/2024.eacl-demo.16/)). Its context measures carry [chapter 3's metrics](#count-relevant-in-top-k) over, with an LLM as the judge. **Context precision** is rank-aware: "It is calculated as the mean of the precision@k for each chunk in the context", over the positions that hold a relevant chunk ([Ragas documentation](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/context_precision/)). **Context recall** is the share of a reference answer's claims that the context supports, and an ID-based variant compares the retrieved IDs with reference IDs ([Ragas documentation](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/context_recall/)). With judgments in hand, the reference is the set of exact products, and the ID-based context recall is [recall@k](#count-relevant-in-top-k). ```text context recall@k = (exact products in the top k) / (exact products judged for the query) context precision@k = mean of precision@i over the positions i ≤ k that hold an exact product ``` Which measure tracks the answers? Salemi and Zamani found that relevance labels show "a small correlation with the RAG system's downstream performance", and they score each retrieved passage by the answer the LLM gives from it instead ([Salemi and Zamani, 2024](https://arxiv.org/abs/2404.13781)). **Spearman's rank correlation** answers the question for one recording: it is the correlation between two measures' ranks rather than their values, 1 when they order the cases the same way and near 0 when the orders are unrelated. In the store, take every context that a retriever built for the [recorded answers](#retrieval-bounds-the-answer), two per test query. The first example builds context recall and context precision by hand, checks them against `lib.metrics.recall_at_k` and the copies in `lib.generation`, and works through one context: Example `context-measures-by-hand` (Python 3.12.15 · bm25s 0.3.13 · numpy 2.5.4 · pandas 3.0.6 · scipy 1.18.1), passing in Edition 2026.10.1: ```python from lib import generation from lib.generation import EXACT from lib.metrics import precision_at_k, recall_at_k from lib.recorded import rag_answers from lib.store import load_queries, qrels def context_precision(context, judged, k=5): """Ragas's context precision, with the judges as the reference: mean precision@i over exact positions i.""" hits = [i for i, p in enumerate(context[:k], start=1) if judged.get(p, 0) == EXACT] return sum(precision_at_k(context, judged, i) for i in hits) / len(hits) if hits else 0.0 def context_recall(context, judged, k=5): """Ragas's ID-based context recall: the share of the reference IDs that the context holds.""" reference = {p for p, grade in judged.items() if grade == EXACT} return len(reference & set(context[:k])) / len(reference) judgments, queries = qrels("test"), load_queries("test") answers = rag_answers().query("context != 'random'") # the two retrievers' contexts for each test query contexts = [(answer.context_ids, judgments[answer.query_id]) for answer in answers.itertuples()] print("contexts:", len(contexts)) same = {"recall_at_k": all(context_recall(c, j) == recall_at_k(c, j, 5) for c, j in contexts), "lib.generation": all(context_precision(c, j) == generation.context_precision(c, j) for c, j in contexts)} print("context recall equals recall_at_k every time:", "yes" if same["recall_at_k"] else "no") print("context precision equals lib.generation's every time:", "yes" if same["lib.generation"] else "no") first = answers.iloc[0] # one context, worked through context, judged = first["context_ids"], judgments[first["query_id"]] print(f"grades in order, {queries.at[first['query_id'], 'query']}, {first['context']}:", *[judged.get(p, 0) for p in context]) print(f"context precision@5: {context_precision(context, judged):.3f}") print(f"precision@5: {precision_at_k(context, judged, 5):.3f}") exact = sorted(sum(grade == EXACT for grade in grades.values()) for grades in judgments.values()) print(f"exact products judged per test query: {exact[0]} to {exact[-1]}") print(f"highest recall@5 possible for the query with the most: {5 / exact[-1]:.3f}") ``` Output: ```text contexts: 60 context recall equals recall_at_k every time: yes context precision equals lib.generation's every time: yes grades in order, door jewelry organizer, semantic: 1 1 2 2 2 context precision@5: 0.478 precision@5: 0.600 exact products judged per test query: 4 to 170 highest recall@5 possible for the query with the most: 0.029 ``` Over the 60 contexts, context recall equals `recall_at_k` every time (yes), and context precision equals `lib.generation`'s (yes). For the semantic context of `door jewelry organizer`, the grades in order are 1 1 2 2 2. Precision@5 is 0.600, and context precision is 0.478, lower because the exact products come last. The second example correlates each measure with the answer's [exact share](#retrieval-bounds-the-answer), the share of its cited products judged exact. It computes Spearman's correlation by hand, as Pearson's correlation of the ranks, and checks it against SciPy's `spearmanr`: Example `context-measures-against-answers` (Python 3.12.15 · bm25s 0.3.13 · numpy 2.5.4 · pandas 3.0.6 · scipy 1.18.1), passing in Edition 2026.10.1: ```python import pandas as pd from scipy import stats from lib.generation import context_precision, context_recall, exact_share from lib.metrics import precision_at_k from lib.recorded import rag_answers from lib.store import qrels judgments = qrels("test") answers = rag_answers().query("context != 'random'") # the two retrievers' contexts for each test query rows = pd.DataFrame([{"recall@5": context_recall(a.context_ids, judgments[a.query_id]), "precision@5": precision_at_k(a.context_ids, judgments[a.query_id], 5), "context precision@5": context_precision(a.context_ids, judgments[a.query_id]), "exact share": exact_share(a.cited, judgments[a.query_id])} for a in answers.itertuples()]) print("contexts:", len(rows)) ranks, spearman = rows.rank(), {} # Spearman: Pearson's correlation of the ranks for measure in ["recall@5", "precision@5", "context precision@5"]: spearman[measure] = ranks[measure].corr(ranks["exact share"]) print(f"spearman with the exact share, {measure}: {spearman[measure]:.3f}") matches = all(round(value, 9) == round(stats.spearmanr(rows[measure], rows["exact share"]).statistic, 9) for measure, value in spearman.items()) print("matches scipy's spearmanr:", "yes" if matches else "no") print("weakest with the exact share:", min(spearman, key=spearman.get)) ``` Output: ```text contexts: 60 spearman with the exact share, recall@5: 0.540 spearman with the exact share, precision@5: 0.797 spearman with the exact share, context precision@5: 0.808 matches scipy's spearmanr: yes weakest with the exact share: recall@5 ``` The hand-built Spearman matches SciPy's (yes). With the exact share, recall@5 correlates at 0.540, precision@5 at 0.797 and context precision at 0.808. The weakest is recall@5, and its denominator shows why. The test queries have 4 to 170 exact products judged, so for the query with the most, a context of five exact products scores 0.029. The answer needs only three to recommend, whatever the number judged. Precision@5 counts how many of the five fit, and context precision also rewards putting them first. Read these correlations as a description of one recording, not a test. Both sides use the same judgments, and an answer can cite only what its context holds, so some correlation is built in. The contexts also come in pairs, semantic and BM25, from the same queries. **What to do:** for a context of fixed size, report the context's precision@k or context precision from your judgments, with recall@k beside it. Use recall when an answer must cover every relevant item, such as every part that fits one appliance. Without judgments, Ragas's LLM-judged versions estimate similar quantities; calibrate them on a human sample before you trust them. Choose k by measuring answers on train queries at several values of k, which takes answers recorded for those queries. ## Choose the retrievable unit {#choose-the-retrievable-unit} A **retrievable unit** is the piece of text that retrieval scores and returns, and that a [context](#measure-the-context) holds: a whole document, a passage or a sentence. A long document gets cut into **chunks**, and **chunking** is the rule that cuts it, such as a fixed number of tokens, sentences, paragraphs, or breaks chosen by meaning. Chen et al. note that "an often-overlooked design choice is the retrieval unit in which the corpus is indexed", and in their study, indexing propositions, short self-contained statements, beat indexing passages ([Chen et al., 2023](https://arxiv.org/abs/2312.06648)). Beyond that, the studies disagree. One finds that the best size depends on the questions and the embedding model: "smaller chunks (64-128 tokens) are optimal for datasets with concise, fact-based answers", while 512 to 1,024 tokens suit questions that need broader context ([Bhat et al., 2025](https://arxiv.org/abs/2505.21700)). Another finds that "the computational costs associated with semantic chunking are not justified by consistent performance gains" ([Qu et al., 2024](https://arxiv.org/abs/2410.13070)). A comparison of 36 methods ranks chunking by groups of paragraphs first, and reports that "simple fixed-size character chunking as baselines performed poorly" ([Shaukat et al., 2026](https://arxiv.org/abs/2603.06976)). A small unit can also lose what it is about. A vendor's engineering post reports that adding chunk-specific context to each chunk before embedding it "reduced the top-20-chunk retrieval failure rate by 35%", a vendor's claim about its own method ([Anthropic, 2024](https://www.anthropic.com/news/contextual-retrieval)). The store's products are short, partly because the book's sample cuts every description to about 400 characters ([data card](../examples/data/wands/README.md)). Of the descriptions, 5,933 end in that cut, and 1,568 products have none. Non-empty descriptions run to a median of 68 words, and the whole text that search reads runs to at most 140 words per product. The store's full product pages would be longer. Even so, each product is a short text about one thing, so cutting it into chunks isn't the store's question. Its question is which part of each product to index. A sentence from a description shows the risk: the last sentence of the chandelier's description from [the support check](#check-citation-support) reads `the antler 's durability and beauty will last a lifetime .`, which never says what the product is. The recorded embeddings cover each product's whole text, so the example tries other units with BM25 only. Choosing a unit is tuning, so it uses the 50 train queries. It indexes five units: the whole product (the text `lib.lexical` indexes), the name alone, the description alone, each of the 35,061 description sentences on its own, and each sentence with the product's name, class and category in front, a simple form of adding context to each chunk. The last sentence of a cut description is a fragment. The sentences with the other fields keep every field the whole product has. A product with no description gets one unit of those fields, the same as its whole text, so the two units differ only in cutting the description. `UnitSearch` in `lib.generation` ranks each product by its best-scoring unit, so the run still lists products. The example reports the precision@5 and recall@5 that a five-product context would have, with judged@5, the share of the top 5 with a judgment. `UnitSearch` orders equal scores by `product_id`, so the example also reverses that rule and counts the train queries whose top 5 changes: Example `retrievable-unit-on-train` (Python 3.12.15 · bm25s 0.3.13 · numpy 2.5.4 · pandas 3.0.6 · scipy 1.18.1), passing in Edition 2026.10.1: ```python import numpy as np from lib.generation import UnitSearch, sentences from lib.metrics import judged_at_k, precision_at_k, recall_at_k from lib.store import load_products, load_queries, product_text, qrels products = load_products() lengths = products["description"].str.split().str.len() print(f"products with no description: {(lengths == 0).sum():,}") print(f"descriptions the sample cut short: {products['description'].str.endswith(' ...').sum():,}") print(f"description words, median of the others: {lengths[lengths > 0].median():.0f}") print(f"product text words per product, most: {product_text(products).str.split().str.len().max()}") header = products["name"] + ". " + products["product_class"] + ". " + products["category"] + ". " # the other fields pieces = [(p, piece) for p, text in products["description"].items() for piece in sentences(text) or [""]] print(f"description sentences: {sum(piece != '' for _, piece in pieces):,}") print("last sentence of product 8171:", sentences(products.at[8171, "description"])[-1]) units = { # unit name: (texts to index, the product each belongs to) "whole product": (product_text(products), products.index), "name only": (products["name"], products.index), "description only": (products["description"], products.index), "sentences": ([piece for _, piece in pieces], [p for p, _ in pieces]), "sentences with the other fields": ([header[p] + piece for p, piece in pieces], [p for p, _ in pieces]), } train, judgments = load_queries("train"), qrels("train") # choosing a unit is tuning: train queries only scores = {} for name, (texts, owners) in units.items(): run = UnitSearch(list(texts), list(owners)).run(train, k=5) # equal scores by product_id for metric in [precision_at_k, recall_at_k, judged_at_k]: scores[name, metric] = np.mean([metric(run[q], judgments[q], 5) for q in train.index]) print(f"{name}, train {metric.__name__.replace('_at_k', '')}@5: {scores[name, metric]:.3f}") flipped = UnitSearch(list(texts), [-p for p in owners]).run(train, k=5) # negated owners: ties go to the highest ID changed = sum(set(run[q]) != {-p for p in flipped[q]} for q in train.index) print(f"{name}, train queries whose top 5 changes with ties reversed: {changed}") print("highest train precision@5:", max(units, key=lambda name: scores[name, precision_at_k])) for metric in [precision_at_k, recall_at_k]: gap = scores["whole product", metric] - scores["sentences with the other fields", metric] print(f"whole product minus sentences with the other fields, train {metric.__name__.replace('_at_k', '')}@5: {gap:.3f}") ``` Output: ```text products with no description: 1,568 descriptions the sample cut short: 5,933 description words, median of the others: 68 product text words per product, most: 140 description sentences: 35,061 last sentence of product 8171: the antler 's durability and beauty will last a lifetime . whole product, train precision@5: 0.464 whole product, train recall@5: 0.124 whole product, train judged@5: 0.940 whole product, train queries whose top 5 changes with ties reversed: 4 name only, train precision@5: 0.368 name only, train recall@5: 0.091 name only, train judged@5: 0.960 name only, train queries whose top 5 changes with ties reversed: 28 description only, train precision@5: 0.304 description only, train recall@5: 0.049 description only, train judged@5: 0.828 description only, train queries whose top 5 changes with ties reversed: 4 sentences, train precision@5: 0.272 sentences, train recall@5: 0.034 sentences, train judged@5: 0.732 sentences, train queries whose top 5 changes with ties reversed: 18 sentences with the other fields, train precision@5: 0.428 sentences with the other fields, train recall@5: 0.106 sentences with the other fields, train judged@5: 0.944 sentences with the other fields, train queries whose top 5 changes with ties reversed: 8 highest train precision@5: whole product whole product minus sentences with the other fields, train precision@5: 0.036 whole product minus sentences with the other fields, train recall@5: 0.018 ``` | Unit | precision@5 | recall@5 | judged@5 | Top 5 changes with ties reversed | | --- | --- | --- | --- | --- | | Whole product | 0.464 | 0.124 | 0.940 | 4 | | Name only | 0.368 | 0.091 | 0.960 | 28 | | Description only | 0.304 | 0.049 | 0.828 | 4 | | Sentences | 0.272 | 0.034 | 0.732 | 18 | | Sentences with the other fields | 0.428 | 0.106 | 0.944 | 8 | The highest train precision@5 goes to the whole product. The fair comparison is with the sentences that carry the other fields: the same fields, the same products within reach, and judged@5 of 0.944 against 0.940. Cutting the description into sentences costs 0.036 in train precision@5 and 0.018 in recall@5. Ties decide part of those gaps: where products tie across the fifth place, the tie rule picks which reach the context, and reversing it changes the top 5 for 8 train queries with the sentences with the other fields and 4 with the whole product. Read a gap of a few hundredths here as rough. The other units also drop fields. The description alone can never return the products that have no description, and bare sentences surface many products nobody judged, so their precision is partly a guess. These are train numbers and untested: they inform a choice and prove nothing. **What to do:** when your items are short, index each whole item as one unit, as the store does. When they are long, chunk them, put the item's title in front of every chunk, and choose the chunk size on train queries by the context measures of [the previous section](#measure-the-context). Pass the LLM the whole item, or the chunk with its title, so it knows what it is reading. Before you trust a small gap, count the queries whose top k changes when ties break the other way. ## Read an agent's searches from its trace {#read-agent-searches} Unlike the [recorded answers](#retrieval-bounds-the-answer), an agent searches for itself: it is an LLM that runs in a loop and acts. A **tool** is a function the program offers the model, described by a name, a description and its parameters. A **tool call** is the model's request to run a tool with arguments. The program runs it, sends back the result, and the model decides what to do next. That is the **agent loop**, and in Ollama, which ran the store's model, "An agent loop allows the model to decide when to invoke tools and incorporate their results into its replies" ([Ollama documentation](https://docs.ollama.com/capabilities/tool-calling)). `recordings/agent_trace.py` is such a loop. ReAct agents "generate both reasoning traces and task-specific actions in an interleaved manner", and on question answering (HotpotQA) and fact verification (Fever), with a Wikipedia API as the tool, ReAct "overcomes issues of hallucination and error propagation prevalent in chain-of-thought reasoning" ([Yao et al., 2023](https://arxiv.org/abs/2210.03629)). The **trace** is the record of every call and its result. An agent's searches are queries, and they need not look like a shopper's. In a log of 14 million agent search requests, "over 90% of multi-turn sessions contain at most ten steps", and "On average, 54% of newly introduced query terms appear in the accumulated evidence context" ([Ning et al., 2026](https://arxiv.org/abs/2601.17617)): the agent picks up words from what it has read. In the store, the agent's recording gave it each test query as the shopper's request, and one tool, `search`, described as "Search the store's catalog. Returns the top 5 products: ID, name, class and attributes." It may search up to 3 times, with any wording, then reply with three product IDs. The tool returns no description. The recording ran twice, with the same model and prompt: once with BM25 behind the tool, and once with semantic retrieval, which embedded the agent's search text with the same embedding model as the products. As with [the answers' contexts](#retrieval-bounds-the-answer), the traces store each BM25 search's results in bm25s's own tie order, so 7 of the 32 searches get results that differ from what `lib.lexical` returns, 6 only in order. The example reads the traces: Example `agent-searches-in-traces` (Python 3.12.15 · bm25s 0.3.13 · numpy 2.5.4 · pandas 3.0.6 · scipy 1.18.1), passing in Edition 2026.10.1: ```python from lib.lexical import LexicalSearch from lib.recorded import agent_traces, manifest, rag_answers from lib.store import load_products, load_queries recording, queries = manifest("agent-trace"), load_queries("test") print("tool description:", recording["prompts"]["tool"]["function"]["description"]) print("searches allowed:", recording["max_searches"]) traces = {(t["retriever"], t["query_id"]): t for t in agent_traces()} print("traces:", len(traces)) first = {key: t["searches"][0]["query"] for key, t in traces.items()} print("first searches the same under both tools:", sum(first[("bm25", q)] == first[("semantic", q)] for q in queries.index)) print("first searches that keep the query as typed:", sum(first[("bm25", q)] == queries.at[q, "query"] for q in queries.index)) for q in queries.index: if first[("bm25", q)] != queries.at[q, "query"]: print(f"first search for {queries.at[q, 'query']}: {first[('bm25', q)]}") context = {(a.context, a.query_id): a.context_ids for a in rag_answers().itertuples()} print("traces whose first search returns the answer recording's context:", sum(t["searches"][0]["results"] == context[key] for key, t in traces.items())) for tool in ["bm25", "semantic"]: mine = [t for (retriever, _), t in traces.items() if retriever == tool] print(f"{tool}, searches: {sum(len(t['searches']) for t in mine)}") print(f"{tool}, traces with one search: {sum(len(t['searches']) == 1 for t in mine)}") lexical, shown = LexicalSearch(load_products()), recording["results_shown"] stored = [s for (tool, _), t in traces.items() if tool == "bm25" for s in t["searches"]] # what the agent saw rerun = [lexical.search(s["query"], shown).index.tolist() for s in stored] # ties by product_id differ = [(s["results"], r) for s, r in zip(stored, rerun) if s["results"] != r] print("bm25 searches whose results differ from lib.lexical:", len(differ)) print("bm25 searches whose results differ from lib.lexical only in order:", sum(set(a) == set(b) for a, b in differ)) ``` Output: ```text tool description: Search the store's catalog. Returns the top 5 products: ID, name, class and attributes. searches allowed: 3 traces: 60 first searches the same under both tools: 30 first searches that keep the query as typed: 26 first search for 48" sliding single track , barn door for laundry: 48" sliding single track barn door laundry first search for counter top one cup hot water dispenser: counter top hot water dispenser one cup first search for refrigerator with ice an water in door: refrigerator with ice and water in door first search for tye dye duvet cover: tie dye duvet cover traces whose first search returns the answer recording's context: 55 bm25, searches: 32 bm25, traces with one search: 29 semantic, searches: 35 semantic, traces with one search: 27 bm25 searches whose results differ from lib.lexical: 7 bm25 searches whose results differ from lib.lexical only in order: 6 ``` The first search is the same under both tools for 30 queries, as expected: the agent writes it before the tool has returned anything, from the same prompt and tool description, at temperature 0. It keeps the query as typed for 26. Of the others, one fixes a misspelling, `tye dye duvet cover` becoming `tie dye duvet cover`, and one the real-word error that [chapter 8's corrector](#correct-spelling-from-catalog) misses, in `refrigerator with ice and water in door`. Both `an` and `and` are stop words for BM25 ([chapter 5](#split-text-into-terms)), so only semantic retrieval sees that change. The other two become `48" sliding single track barn door laundry` and `counter top hot water dispenser one cup`. In 55 of the 60 traces, the first search returns exactly the five products that the [answer recording](#retrieval-bounds-the-answer) put in its context. The agent rarely looks again. With BM25 it makes 32 searches in all, and 29 traces stop after one. With semantic retrieval it makes 35, and 27 stop after one. Where do the words of a later search come from? The second example compares each later search with the words already searched, and with the names, classes and attributes the tool had shown: Example `agent-later-searches` (Python 3.12.15 · bm25s 0.3.13 · numpy 2.5.4 · pandas 3.0.6 · scipy 1.18.1), passing in Edition 2026.10.1: ```python from lib.generation import shown, words from lib.recorded import agent_traces from lib.store import load_products, load_queries products, queries = load_products(), load_queries("test") for tool in ["bm25", "semantic"]: later = new = from_results = 0 for trace in (t for t in agent_traces() if t["retriever"] == tool): asked, seen = words(queries.at[trace["query_id"], "query"]), set() for number, search in enumerate(trace["searches"]): if number > 0: # a later search: the words it adds, and where from added = words(search["query"]) - asked later, new, from_results = later + 1, new + len(added), from_results + len(added & seen) asked |= words(search["query"]) seen |= words(shown(search["results"], products)) # name, class and attributes, as the tool showed them print(f"{tool}, later searches: {later}") print(f"{tool}, words that later searches add: {new}") print(f"{tool}, of those, words the tool had shown: {from_results}") ``` Output: ```text bm25, later searches: 2 bm25, words that later searches add: 2 bm25, of those, words the tool had shown: 1 semantic, later searches: 5 semantic, words that later searches add: 4 semantic, of those, words the tool had shown: 4 ``` Under BM25, 2 later searches add 2 words, 1 of them shown by the tool before. Under semantic retrieval, 5 later searches add 4, 4 of them shown before. Counts this small describe these traces; they are not a rate to compare with Ning et al.'s. **What to do:** read the traces before you score an agent. Count its searches per task, compare each search with the shopper's words, and find where added words come from. Add the agent's searches to the queries you test your retriever on, scored with the judgments for the shopper's request or judged afresh, because a retriever tuned on people's queries hasn't been tested on an agent's. Expect an agent that mostly passes the shopper's words through to be bounded by the same retrieval as a RAG answer. ## Compare the retrievers behind the agent's tool {#retriever-behind-agent-tool} An agent that can search again might seem to make the retriever matter less. One benchmark measured otherwise. BrowseComp-Plus fixes the collection that agents search, so that retrievers can be swapped inside one agent. With BM25, "the GPT-5 achieves 55.9%", and "Integrating the GPT-5 with the Qwen3-Embedding-8B retriever further enhances its accuracy to 70.1%", with fewer search calls on average, 21.74 against 23.23 ([Chen et al., 2025, Table 1](https://arxiv.org/abs/2508.06600)). Shopping is a classic task for agents. WebShop holds "1.18 million real-world products", and its authors' best model, in 2022, "achieves a task success rate of 29%", against 59% for human experts ([Yao et al., 2022](https://arxiv.org/abs/2207.01206)). On WebMall, four simulated shops, its authors report "best-performing agents achieving task completion rates below 65%" on searches for the cheapest product and for vaguely described ones ([Peeters et al., 2025](https://arxiv.org/abs/2508.13024)). An agent told to pick from its search results should pick only what they return, and its trace shows whether it did. Its picks are then bounded by retrieval, as a RAG answer is. The store's [recorded agent](#read-agent-searches) searches the catalog up to 3 times with one tool, then picks three products. The second of the chapter's [two planned comparisons](#retrieval-for-llm-applications), established if its permutation p-value is below 0.025, scores those picks under each tool by [precision@3](#count-relevant-in-top-k): exact picks divided by 3, so a missing pick counts as a miss. The example counts, for each tool, the traces with fewer than three picks, picks the tool never returned, picks nobody judged, traces whose searches never returned an exact product, and traces that saw an exact product and picked none. As an exploratory check, it compares the picks with the tool's own top 3 for the first search: Example `agent-picks-by-tool` (Python 3.12.15 · bm25s 0.3.13 · numpy 2.5.4 · pandas 3.0.6 · scipy 1.18.1), passing in Edition 2026.10.1: ```python import pandas as pd from lib.generation import EXACT, report_planned from lib.metrics import precision_at_k from lib.recorded import agent_traces from lib.significance import compare from lib.store import qrels judgments, picks, top_three, below = qrels("test"), {}, {}, 0 for tool in ["bm25", "semantic"]: traces = [t for t in agent_traces() if t["retriever"] == tool] returned = [{p for search in t["searches"] for p in search["results"]} for t in traces] seen = [sum(judgments[t["query_id"]].get(p, 0) == EXACT for p in r) for t, r in zip(traces, returned)] picked = [sum(judgments[t["query_id"]].get(p, 0) == EXACT for p in t["picked"]) for t in traces] print(f"{tool}, picks the tool never returned: {sum(len(set(t['picked']) - r) for t, r in zip(traces, returned))}") print(f"{tool}, picks nobody judged: {sum(p not in judgments[t['query_id']] for t in traces for p in t['picked'])}") print(f"{tool}, traces with fewer than three picks: {sum(len(t['picked']) < 3 for t in traces)}") print(f"{tool}, traces whose searches returned no exact product: {sum(s == 0 for s in seen)}") print(f"{tool}, traces that saw an exact product and picked none: {sum(s > 0 and k == 0 for s, k in zip(seen, picked))}") picks[tool] = pd.Series({t["query_id"]: precision_at_k(t["picked"], judgments[t["query_id"]], 3) for t in traces}) top_three[tool] = pd.Series({t["query_id"]: precision_at_k(t["searches"][0]["results"], judgments[t["query_id"]], 3) for t in traces}) # the tool's own top 3 for the first search print(f"{tool}, precision@3 of the picks: {picks[tool].mean():.3f}") print(f"{tool}, precision@3 of the first search's top 3: {top_three[tool].mean():.3f}") p_value = compare(picks[tool], top_three[tool])["permutation p"] print(f"{tool}, picks against the top 3, exploratory permutation p: {p_value:.3f}") below += p_value < 0.05 print(f"picks against the top 3, p-values below 0.05 uncorrected: {below} of 2") report_planned("agent picks, semantic over bm25", picks["semantic"], picks["bm25"]) ``` Output: ```text bm25, picks the tool never returned: 0 bm25, picks nobody judged: 3 bm25, traces with fewer than three picks: 2 bm25, traces whose searches returned no exact product: 5 bm25, traces that saw an exact product and picked none: 6 bm25, precision@3 of the picks: 0.567 bm25, precision@3 of the first search's top 3: 0.500 bm25, picks against the top 3, exploratory permutation p: 0.152 semantic, picks the tool never returned: 0 semantic, picks nobody judged: 0 semantic, traces with fewer than three picks: 0 semantic, traces whose searches returned no exact product: 5 semantic, traces that saw an exact product and picked none: 2 semantic, precision@3 of the picks: 0.622 semantic, precision@3 of the first search's top 3: 0.633 semantic, picks against the top 3, exploratory permutation p: 1.000 picks against the top 3, p-values below 0.05 uncorrected: 0 of 2 agent picks, semantic over bm25, mean difference: 0.056 agent picks, semantic over bm25, wins, losses, ties: 5, 4, 21 agent picks, semantic over bm25, permutation p: 0.422 agent picks, semantic over bm25, established at 0.025: no agent picks, semantic over bm25, t-test p: 0.305 agent picks, semantic over bm25, t-test agrees: yes agent picks, semantic over bm25, 97.5% interval: -0.033 to 0.209 agent picks, semantic over bm25, interval agrees: yes ``` Picks the tool never returned number 0 under BM25 and 0 under semantic retrieval, so the bound holds. In 5 traces under BM25 and 5 under semantic retrieval, no search returned an exact product, and no pick could be exact. The agent saw an exact product and picked none in 6 traces under BM25 and 2 under semantic retrieval. Under BM25, 3 picks are unjudged and count as misses, against 0 under semantic retrieval. Traces with fewer than three picks number 2 under BM25 and 0 under semantic retrieval, and their missing picks count as misses too. The picks score a precision@3 of 0.622 with semantic retrieval and 0.567 with BM25. With this model, prompt and these test queries, the two tools show no established difference in the picks, with a permutation p-value of 0.422 ([both comparisons](#retrieval-for-llm-applications-decisions)). That isn't evidence that the retriever doesn't matter, only that these queries don't show a difference. The exploratory check reads the same way. The tool's own top 3 for the first search scores 0.500 under BM25 and 0.633 under semantic retrieval, and the permutation p-values for picks against that top 3 are 0.152 and 1.000: 0 of 2 fall below an uncorrected 0.05. We chose the check after seeing test results, so it is never established. **What to do:** test the retriever behind an agent's tool as you test it for people, on the agent's own searches and on its final picks, scored with human judgments. Check that every pick came from a tool result. Count the tasks where the tool never returned a relevant product, because no agent can fix those and only retrieval can. ## What goes wrong in production {#retrieval-for-llm-applications-in-production} A live assistant answers every shopper with a fresh model call, on a context that retrieval builds for that request. **The model, the prompt and the context are one system.** A different model version, a reworded prompt, another context size or a different order of products can change every answer. The store's manifests pin the model's digest, the prompts, the answer and tool schemas and the context window. Change any of them and score the answers again, against the same judgments and [the random floor](#retrieval-bounds-the-answer). Keep the raw replies and each context, as the recordings do. **The citation check belongs on the answer's path.** [The ID check](#check-citations-against-context) is cheap, so run it on every answer before the shopper sees it, and log each failure with the answer and its context. **Abstentions move with retrieval.** Track the share of answers that recommend nothing. On the store's random contexts, 29 of the 30 answers recommend nothing, so a rise in that share can mean that retrieval or the catalog changed. Read those answers before you assume the model did. **Product text reaches the prompt.** Product descriptions may come from outside your team, and the LLM reads them as part of its input. [Chapter 4](#judgments-from-people-and-llms-in-production) cites a study in which words and instructions planted in a text misled LLM judges. Test the assistant on products whose text contains instructions before you trust it with a catalog you don't write. **Longer contexts change the answer.** Raising k lengthens the context and brings in more distractors, and both can hurt ([Liu et al., 2024](https://aclanthology.org/2024.tacl-1.9/); [Jin et al., 2024](https://arxiv.org/abs/2410.05983)). Measure answers at each setting on train queries. **Agent traffic is a different workload.** In the logs [Ning et al.](#read-agent-searches) studied, agents send many machine-written queries per task, while the store's agent mostly sent one. One study warns that with agent traffic, "caching may lose effectiveness, query pre-processing may add overhead without improving results" ([Pezzuti et al., 2026](https://arxiv.org/abs/2602.17518)). Log agents' searches apart from people's, and test the retriever on both. **What to do:** version the model, prompt and context settings with the answers, check citations on every answer, track abstentions, and keep agents' searches in a test set of their own. ## Decisions this chapter made {#retrieval-for-llm-applications-decisions} Only an established planned comparison on the test queries could change the store's retriever. - **The context: five whole products from semantic retrieval,** the [ranker to beat](#compare-runs-in-a-harness). Five is the recording's fixed, untested setting. The answers show no established difference between semantic retrieval and BM25, so the retriever rests on chapter 3's established lead in nDCG@10. - **Answers scored on human grades:** the exact share of what each answer cites, with the answers that recommend nothing counted beside it and a random context as a floor. - **Citations checked by ID on every answer.** The recording passes the check, and it stays. Support checks, by NLI model or LLM judge, count only when calibrated on a human sample. - **The context measured by precision@5 and context precision from the judgments,** with recall@5 beside them, because the recall@5 of a five-product context can't reach 1 when a query has more than five exact products. - **The retrievable unit: the whole product.** It has the highest train precision@5, and cutting descriptions into sentences, even with the other fields in front, costs 0.036. - **The agent's tool: the same retrievers, tested on the agent's searches and picks.** The picks show no established difference between the tools, and every pick came from what the tool returned. - **Scope: one model, two prompts, the test queries.** Every finding about the LLM is about `gemma4:12b-it-q4_K_M` with the recorded prompts. A different model or prompt is a change to score again, as a change of retriever is. The chapter's 2 planned comparisons, semantic retrieval minus BM25 on the test queries, established below 0.025: | Measure | [Answers](#retrieval-bounds-the-answer), mean exact share | [Agent picks](#retriever-behind-agent-tool), precision@3 | | --- | --- | --- | | Semantic | 0.658 | 0.622 | | BM25 | 0.639 | 0.567 | | Mean difference | 0.019 | 0.056 | | Wins / losses / ties | 4, 5, 21 | 5, 4, 21 | | Permutation p | 0.765 | 0.422 | | Established | no | no | | t-test p | 0.715 | 0.305 | | 97.5% interval | -0.078 to 0.161 | -0.033 to 0.209 | | t-test agrees | yes | yes | | Interval agrees | yes | yes | # Testing changes By the end of this chapter you can decide whether a change to search should ship: diff two rankers' results query by query, keep your test queries from flattering a change, size an A/B test, and compare two rankers by interleaving their results. The store, the online furniture and home-goods retailer that is this book's running case ([chapter 1](#store-data)), still runs [BM25](#rank-with-bm25) in its search box, as its baseline. The offline work of the earlier chapters made [semantic retrieval](#search-by-meaning) the ranker to beat, with an established lead ([chapter 3](#compare-runs-in-a-harness)). Switching is a change, and so is every tweak that follows: a cheaper embedding, a blend weight, a reranker. Each raises the same four questions. Did the average go up, by more than noise? Which queries got worse? Can the judgments score the new results at all? Will shoppers behave the way the judgments predict? This chapter answers them in that order. The store's 30 test queries become its regression set, the fixed queries every change is tested on, and every choice is made on its 50 train queries. The store has no click logs, so the online sections simulate shoppers, with each simulation's assumptions stated beside it. The chapter plans one comparison on the test queries: semantic retrieval against BM25. Following [chapter 3](#compare-runs-in-a-harness), a difference counts as established when its paired permutation test gives a p-value below 0.05 divided by the number of planned comparisons, so below 0.05 here. Every other score on the test queries in this chapter is a demonstration, and says so. ## Diff two runs query by query {#diff-runs-per-query} The first two questions, whether the average went up and which queries got worse, are answered on a **regression set**: a fixed set of queries, with judgments, that every change runs against before it ships. The ranker the store runs today is the **baseline**, and the change is the **candidate** ranker. Each produces a [run](#score-every-test-query): its rankings for every query in the set. Comparing the runs' means isn't enough, because a change can raise the average and still make some queries worse. TREC, the Text REtrieval Conference's shared evaluations of search systems, built a whole task around that risk in its 2013 Web track, scoring each system's wins and losses per query against a baseline ([Collins-Thompson et al., 2014](https://trec.nist.gov/pubs/trec22/papers/WEB.OVERVIEW.pdf)). So diff the two runs: one row per query, with the baseline's score, the candidate's score and the change. Then count the wins, losses and ties, and read the losses. In the store, the baseline is BM25 and the candidate is semantic retrieval, both scored with [nDCG@10](#score-one-ranking) on the test queries. This is the chapter's one planned comparison. The example builds the diff with pandas, and takes the tests from `lib.significance`, which [chapter 3](#compare-runs-in-a-harness) builds by hand: the [permutation test](#permutation-test-on-queries), the [paired t-test](#paired-t-test-on-queries) and the [bootstrap interval](#report-the-size-of-a-gap), at the confidence level that matches the threshold. The bootstrap and the permutation test resample at random, so `lib.significance` seeds SciPy with `rng` ([chapter 3](#permutation-test-on-queries)). Example `diff-semantic-against-bm25` (Python 3.12.15 · bm25s 0.3.13 · numpy 2.5.4 · pandas 3.0.6 · scipy 1.18.1), passing in Edition 2026.10.1: ```python import pandas as pd from lib.lexical import LexicalSearch from lib.metrics import evaluate from lib.semantic import SemanticSearch from lib.significance import compare from lib.store import load_products, load_queries, qrels PLANNED_COMPARISONS = 1 # this chapter's tests on the test queries threshold = 0.05 / PLANNED_COMPARISONS # Bonferroni: one comparison keeps 0.05 products, queries, judgments = load_products(), load_queries("test"), qrels("test") baseline = evaluate(LexicalSearch(products).run(queries, k=100), judgments) # what the store runs candidate = evaluate(SemanticSearch().run(queries, k=100), judgments) # the change diff = pd.DataFrame({"query": queries["query"], "bm25": baseline["ndcg@10"], "semantic": candidate["ndcg@10"]}) diff["change"] = (diff["semantic"] - diff["bm25"]).round(3) # one row per test query print(f"bm25 ndcg@10: {diff['bm25'].mean():.3f}") print(f"semantic ndcg@10: {diff['semantic'].mean():.3f}") result = compare(candidate["ndcg@10"], baseline["ndcg@10"], confidence=1 - threshold) # chapter 3's comparison print(f"mean change: {result['mean difference']:.3f}") print(f"threshold: {threshold:g}") print(f"permutation test p: {result['permutation p']:.3f}") print(f"paired t-test p: {result['t-test p']:.3f}") print(f"{(1 - threshold) * 100:g}% interval: {result['ci low']:.3f} to {result['ci high']:.3f}") print("established:", "yes" if result["permutation p"] < threshold else "no") print("wins:", (diff["change"] > 0).sum()) print("losses:", (diff["change"] < 0).sum()) print("ties:", (diff["change"] == 0).sum()) print(f"largest loss: {diff['change'].min():.3f}") print() print(diff[diff["change"] < 0].sort_values("change").round(3).to_string()) ``` Output: ```text bm25 ndcg@10: 0.725 semantic ndcg@10: 0.830 mean change: 0.105 threshold: 0.05 permutation test p: 0.001 paired t-test p: 0.003 95% interval: 0.055 to 0.185 established: yes wins: 21 losses: 7 ties: 2 largest loss: -0.092 query bm25 semantic change query_id 292 decorative white pillow 0.467 0.375 -0.092 293 wood floor protector mat 0.653 0.568 -0.084 482 48 inch kitchen hood 1.000 0.921 -0.079 63 monthly calendar 1.000 0.931 -0.069 285 48" sliding single track , barn door for laundry 0.592 0.555 -0.037 72 fold in half table 1.000 0.965 -0.035 94 kohler whitehaven farmhouse kitchen sink 0.694 0.690 -0.004 ``` The mean rises from 0.725 to 0.830, a change of 0.105. Against the threshold of 0.05, the permutation test gives 0.001, which settles whether the change is established: yes. Beside it, the t-test gives 0.003 and the 95% interval runs from 0.055 to 0.185. Semantic retrieval wins on 21 queries, loses on 7 and ties on 2. The losses are where you learn something. The largest is -0.092, and two of them give a size in inches, `48 inch kitchen hood` and `48" sliding single track , barn door for laundry`: two queries, not a pattern. [Chapter 6](#embeddings-match-digits) tests the pattern on every query with a digit. **What to do:** keep a regression set of judged queries that nothing is ever tuned on, and run every candidate against the baseline on it. List your planned comparisons first, and report each with its permutation p-value, its t-test and its interval. Count wins, losses and ties, and read every loss before you ship. When the losses seem to share a pattern, check it on train queries before you act on it, then fix that kind of query or send it to the baseline. ## Weigh losses more than wins with URisk {#weigh-losses-with-urisk} [The per-query diff](#diff-runs-per-query) counts the losses, but the mean change still treats a query that got worse by 0.1 and one that got better by 0.1 as cancelling out. A store may not see it that way: a shopper who found a product yesterday and can't find it today has met a regression, and a slightly better ranking elsewhere doesn't repay them. Risk-sensitive evaluation counts losses more heavily than wins. The risk-sensitive measure of the [TREC 2013 Web track](https://trec.nist.gov/pubs/trec22/papers/WEB.OVERVIEW.pdf) is URisk. For each query q, let Δ(q) be the candidate's score minus the baseline's. Add up the gains on the queries that improved, subtract the losses on the queries that got worse, counted α + 1 times, and divide by the number of queries N: ```text URisk = (1 / N) × [ sum of Δ(q) over improved queries − (α + 1) × sum of |Δ(q)| over hurt queries ] ``` The track describes this as "weighting losses α + 1 times as heavily as successes", and α = 0 gives the plain average change (Collins-Thompson et al., section 4.3). The track's overview prints the loss term with Δ(q) itself, which is negative on a hurt query. We use its size, |Δ(q)|, the only reading under which α = 0 gives the average, as the overview says. α is a choice, not a fact. We use α = 1, so each loss counts double. Compare two of the store's candidates against BM25. The first is full semantic retrieval, with all 768 numbers of each [embedding](#search-by-meaning) from `Snowflake/snowflake-arctic-embed-m-v1.5`. The second keeps only the first 128 numbers, a sixth of the memory. That is below the 256 dimensions the model's card documents, and [chapter 6](#truncate-dimensions-test-loss) rejected 128 on the train queries. We use it because on the test queries it trades gains on some queries for losses on others, which is what URisk is for. Chosen for how it behaves on test, it is a demonstration, not a planned comparison. The example writes URisk as a short function: Example `urisk-of-two-candidates` (Python 3.12.15 · bm25s 0.3.13 · numpy 2.5.4 · pandas 3.0.6 · scipy 1.18.1), passing in Edition 2026.10.1: ```python from lib.lexical import LexicalSearch from lib.metrics import evaluate from lib.semantic import SemanticSearch from lib.significance import compare from lib.store import load_products, load_queries, qrels ALPHA = 1 # each loss counts alpha + 1 times: here, double def urisk(change, alpha): """Mean change per query, with every loss counted alpha + 1 times (TREC 2013 Web track).""" wins, losses = change.clip(lower=0).sum(), -change.clip(upper=0).sum() return (wins - (alpha + 1) * losses) / len(change) products, queries, judgments = load_products(), load_queries("test"), qrels("test") baseline = evaluate(LexicalSearch(products).run(queries, k=100), judgments) candidates = {"semantic": SemanticSearch(), "semantic 128": SemanticSearch(dims=128)} # all 768 numbers, or the first 128 print(f"bm25 judged@10: {baseline['judged@10'].mean():.3f}") evaluated = {name: evaluate(search.run(queries, k=100), judgments) for name, search in candidates.items()} for name, scores in evaluated.items(): change = (scores["ndcg@10"] - baseline["ndcg@10"]).round(3) print(f"{name} judged@10: {scores['judged@10'].mean():.3f}") print(f"{name} mean change: {change.mean():.3f}") print(f"{name} wins: {(change > 0).sum()}") print(f"{name} losses: {(change < 0).sum()}") print(f"{name} largest loss: {change.min():.3f}") print(f"{name} urisk with alpha {ALPHA}: {urisk(change, ALPHA):.3f}") # a demonstration, chosen for how it behaves on test: its tests describe it, and it is never established result = compare(evaluated["semantic 128"]["ndcg@10"], baseline["ndcg@10"], confidence=0.95) print(f"semantic 128 permutation test p: {result['permutation p']:.3f}") print(f"semantic 128 paired t-test p: {result['t-test p']:.3f}") print(f"semantic 128 interval: {result['ci low']:.3f} to {result['ci high']:.3f}") ``` Output: ```text bm25 judged@10: 0.927 semantic judged@10: 0.990 semantic mean change: 0.105 semantic wins: 21 semantic losses: 7 semantic largest loss: -0.092 semantic urisk with alpha 1: 0.092 semantic 128 judged@10: 0.920 semantic 128 mean change: 0.017 semantic 128 wins: 15 semantic 128 losses: 13 semantic 128 largest loss: -0.253 semantic 128 urisk with alpha 1: -0.033 semantic 128 permutation test p: 0.592 semantic 128 paired t-test p: 0.597 semantic 128 interval: -0.036 to 0.088 ``` Full semantic retrieval's mean change is 0.105, and its URisk is 0.092. Its 7 losses are small, the largest -0.092, so doubling them costs little. The cheaper ranker's mean change is 0.017, with a permutation p-value of 0.592, a t-test p-value of 0.597 and a 95% interval from -0.036 to 0.088, which describe it and establish nothing. It wins 15 queries and loses 13, and its largest loss is -0.253. With losses counted double, its URisk is -0.033. Its mean hides the trade it makes: better results on some queries, paid for with worse results on others. Its [judged@10](#score-every-test-query) is 0.920, against 0.927 for BM25 and 0.990 for full semantic retrieval, so some of its losses may be products nobody judged. [The next section](#judge-new-results) checks. URisk describes risk without testing it. Dinçer et al. define a related measure, TRisk, that "is a linear transformation of the t statistic", so it can be tested like the [paired t-test](#paired-t-test-on-queries) ([Dinçer et al., 2014](https://eprints.gla.ac.uk/95215)). **What to do:** report URisk with α = 1 next to the mean change for every candidate. Don't ship a candidate whose URisk is negative, whatever its mean, and read its losses to find out why. Raise α when failures are expensive, and report URisk separately on the store's most frequent queries. Use α = 0 only when a loss on one query is no worse than a gain on another. ## Judge the products a new ranker surfaces {#judge-new-results} A new ranker's losses, such as the cheaper ranker's in [the URisk demonstration](#weigh-losses-with-urisk), may come from gaps in the judgments, which cover only the products that earlier systems found. WANDS, the source of the store's judgments, chose what to judge from products shoppers clicked or added to their carts, from a lexical engine, from a neural retriever and from a query classifier ([chapter 2](#pooled-judgments)). A ranker that differs from those surfaces products nobody judged, and the metrics give each one [grade](#store-data) 0, so its score may be too low, or exactly right. You can't tell which until someone judges them. The WANDS authors argue that "incompleteness in Product Search dataset does negatively impact the discriminative power of the evaluation" ([Chen et al., 2022](https://easychair.org/publications/preprint/wm4R)). [Chapter 2](#unjudged-results) counts unjudged products with judged@10, the share of the top 10 that has a judgment. For testing a change, one more step helps: a sensitivity check. Rescore both runs under two scenarios, one favouring each side. In one, every unjudged product in the baseline's top 10s is exact; in the other, every unjudged product in the candidate's. Both runs are scored with the same judgments in each scenario, because nDCG's ideal ranking is built from every judgment for the query: a product judged exact raises the ideal, and so lowers the score of every run that doesn't show it. The scenarios aren't bounds, since the true judgments could mix the two. If the leader changes between them, the judgments can't decide yet. The cheaper semantic ranker from [the URisk demonstration](#weigh-losses-with-urisk), which keeps 128 of each embedding's 768 numbers, lost 13 of the store's test queries against BM25. Before you read anything into those losses, run the check: Example `unjudged-score-range` (Python 3.12.15 · bm25s 0.3.13 · numpy 2.5.4 · pandas 3.0.6 · scipy 1.18.1), passing in Edition 2026.10.1: ```python import numpy as np from lib.lexical import LexicalSearch from lib.metrics import ndcg_at_k from lib.semantic import SemanticSearch from lib.store import load_products, load_queries, qrels products, queries, judgments = load_products(), load_queries("test"), qrels("test") runs = {"bm25": LexicalSearch(products).run(queries, k=100), "semantic 128": SemanticSearch(dims=128).run(queries, k=100), "semantic": SemanticSearch().run(queries, k=100)} def unjudged(run): """(query_id, product_id) pairs in a run's top 10 that nobody judged.""" return {(query_id, product_id) for query_id in judgments for product_id in run[query_id][:10] if product_id not in judgments[query_id]} def score(run, exact=()): """Mean nDCG@10 with the given unjudged pairs judged exact, for every run alike: the ideal moves too.""" scenario = {query_id: dict(grades) for query_id, grades in judgments.items()} for query_id, product_id in exact: scenario[query_id][product_id] = 2 return np.mean([ndcg_at_k(run[query_id], scenario[query_id], 10) for query_id in scenario]) for name in ["bm25", "semantic 128"]: print(f"unjudged pairs in the top 10, {name}: {len(unjudged(runs[name]))}") scenarios = {"as judged": set(), **{f"{name}'s unjudged exact": unjudged(runs[name]) for name in runs}} scores = {(scenario, name): score(run, exact) for scenario, exact in scenarios.items() for name, run in runs.items()} for scenario, name in [("as judged", "bm25"), ("as judged", "semantic 128"), ("bm25's unjudged exact", "bm25"), ("bm25's unjudged exact", "semantic 128"), ("bm25's unjudged exact", "semantic"), ("semantic 128's unjudged exact", "bm25"), ("semantic 128's unjudged exact", "semantic 128")]: print(f"{scenario}, {name}: {scores[scenario, name]:.3f}") for candidate in ["semantic 128", "semantic"]: leaders = {scores[s, candidate] > scores[s, "bm25"] for s in scenarios} print(f"leader changes, {candidate} against bm25:", "yes" if len(leaders) > 1 else "no") print("pairs to judge, semantic 128 against bm25:", len(unjudged(runs["bm25"]) | unjudged(runs["semantic 128"]))) ``` Output: ```text unjudged pairs in the top 10, bm25: 22 unjudged pairs in the top 10, semantic 128: 24 as judged, bm25: 0.725 as judged, semantic 128: 0.742 bm25's unjudged exact, bm25: 0.788 bm25's unjudged exact, semantic 128: 0.738 bm25's unjudged exact, semantic: 0.829 semantic 128's unjudged exact, bm25: 0.718 semantic 128's unjudged exact, semantic 128: 0.807 leader changes, semantic 128 against bm25: yes leader changes, semantic against bm25: no pairs to judge, semantic 128 against bm25: 46 ``` BM25 has 22 unjudged products in its top 10s, and the cheaper ranker 24. As judged, the cheaper ranker leads, 0.742 to 0.725. If BM25's unjudged products were exact, BM25 would lead, 0.788 to 0.738. If the cheaper ranker's were, it would lead, 0.807 to 0.718. The leader changes: yes. The comparison waits on 46 judgments, a small bill next to the cost of shipping the wrong ranker. Don't guess what the judges will say. On an old TREC collection, Voorhees et al. judged new pools of documents from neural and BM25 runs, and found that "the vast majority of those documents were not relevant" ([Voorhees et al., 2022](https://arxiv.org/abs/2201.11086)). But that collection was pooled deeply from many systems, and WANDS was built differently. Human judgments are the reference. Judgments from a large language model (LLM) are cheaper. [Chapter 4](#fill-unjudged-with-llm-grades) fills the store's unjudged pairs with them as a diagnostic, and has people judge the pairs whenever the filled grades change a verdict or the order of two rankers. Thomas et al. add a warning for testing changes: "LLM-generated labels may also show bias towards rankers that themselves incorporate LLMs" ([Thomas et al., 2024](https://arxiv.org/abs/2309.10621)). **What to do:** for every candidate, compute judged@10 for both runs and rescore them under the two scenarios. When the leader changes, judge the unjudged pairs before you compare, and add the new judgments to the regression set for good, so that it covers every ranker you have tested. When the leader holds in both scenarios, the unjudged products are unlikely to reverse the comparison. ## Keep the test queries out of every choice {#keep-test-queries-held-out} The test queries in [the regression set](#diff-runs-per-query) are held out, and held-out queries give an honest score only if nothing was chosen by looking at them. Training on test queries is the obvious mistake. Choosing on them is the subtle one: try several settings, keep the one that scores best on the test queries, and report that score. The winner is partly the best setting and partly the luckiest on those particular queries, so its score is too high. Dwork et al. state the risk: "Reusing a holdout set adaptively multiple times can easily lead to overfitting to the holdout set itself" ([Dwork et al., 2015](https://arxiv.org/abs/1506.02629)). The damage can be smaller than feared. When ImageNet's test set was rebuilt, classifiers lost accuracy, but Recht et al.'s results suggest that "the accuracy drops are not caused by adaptivity": the new images were harder ([Recht et al., 2019](https://arxiv.org/abs/1902.10811)). In our judgement, a set of 30 queries leaves little room, and every look counts. In the store, the change is a blend of BM25 and semantic retrieval with one weight, the share given to semantic retrieval. [Chapter 7](#tune-blend-on-train) calls it α, a different α from URisk's, and tunes it on the train queries with weighted reciprocal rank fusion, over 0, 0.1, … 1. Here, choose the semantic weight both ways, the right way on train queries and the wrong way on test queries, and report each choice on the test queries. These test scores are a demonstration of the bias, not a planned comparison. Then measure how much choosing on the reported queries inflates their score, without touching the test queries again: split the train queries at random, 1,000 times, into 30 to choose on and 20 to report on, and compare the 20's score when the weight is chosen on the 30 with their score when it is chosen on the 20 themselves. Example `tuning-on-test-inflates` (Python 3.12.15 · bm25s 0.3.13 · numpy 2.5.4 · pandas 3.0.6 · scipy 1.18.1), passing in Edition 2026.10.1: ```python import numpy as np from lib.blend import best_alpha, scores_by_alpha, weighted_rrf from lib.lexical import LexicalSearch from lib.semantic import SemanticSearch from lib.store import load_products, load_queries, qrels queries, judgments = load_queries(), qrels() # all 80 queries, train and test bm25 = LexicalSearch(load_products()).run(queries, k=100) semantic = SemanticSearch().run(queries, k=100) per_query = scores_by_alpha(lambda weight: weighted_rrf(bm25, semantic, weight), judgments) # nDCG@10, a column per weight train = per_query.loc[load_queries("train").index] test = per_query.loc[load_queries("test").index] chosen_on_train, chosen_on_test = best_alpha(train), best_alpha(test) print("settings tried:", per_query.shape[1]) print(f"semantic alone, test ndcg@10: {test[1.0].mean():.3f}") print("semantic weight chosen on train:", chosen_on_train) print(f"test ndcg@10, weight chosen on train: {test[chosen_on_train].mean():.3f}") print("semantic weight chosen on test:", chosen_on_test) print(f"test ndcg@10, weight chosen on test: {test[chosen_on_test].mean():.3f}") # How much does choosing on the queries you report inflate the score? Split the train queries # at random, 30 to choose on and 20 to report on, and compare the two ways of choosing. rng = np.random.default_rng(2026) scores = train.to_numpy() inflation = [] for _ in range(1000): order = rng.permutation(len(scores)) choose, report = scores[order[:30]].mean(axis=0), scores[order[30:]].mean(axis=0) inflation.append(report.max() - report[choose.argmax()]) print(f"average inflation, 20 reported queries: {np.mean(inflation):.3f}") ``` Output: ```text settings tried: 11 semantic alone, test ndcg@10: 0.830 semantic weight chosen on train: 0.6 test ndcg@10, weight chosen on train: 0.825 semantic weight chosen on test: 0.8 test ndcg@10, weight chosen on test: 0.835 average inflation, 20 reported queries: 0.007 ``` Semantic retrieval alone scores 0.830. Chosen on train, the semantic weight is 0.6, and the blend scores 0.825 on the test queries. Chosen on test, the weight is 0.8, and the blend scores 0.835. Only the first is a fair estimate of how the blend does on queries it wasn't chosen on. [Chapter 7](#report-blends-on-test) tests the train-chosen blend against semantic retrieval and finds no improvement. The random splits put a number on the bias. On report sets of 20 queries, choosing on the queries you report adds 0.007 to nDCG@10 on average, with 11 settings to choose from. That gain appears in the report and nowhere else. Each extra setting you try is another chance for a lucky winner. **What to do:** choose every setting on train queries, including which candidate to take forward. Look at the test queries once per decision, and log each look: what you compared and what you chose. When a test set has guided many decisions, move it to the train side and judge a fresh set of test queries. ## Run an A/B test {#run-an-ab-test} Offline tests measure rankings against judgments, but the store earns from shoppers. An **A/B test**, or controlled experiment, assigns users at random to the current system, the **control**, or to the change, the **treatment**, and compares a metric chosen before the test starts ([Kohavi et al., 2009](https://exp-platform.com/Documents/controlledExperimentDMKD.pdf)). Three checks make the result trustworthy. First, an **A/A test**, in which both groups get the same system: "the Null hypothesis should be rejected about 5% of the time when a 95% confidence level is used" (Kohavi et al., 2009). Far more than 5% points to a broken setup. Second, read the result once, at the size you planned. Stopping a test the moment it looks significant, called **peeking**, "significantly inflates the type-I error rates" ([Kohavi et al., 2022](https://exp-platform.com/abtestingintuitionbusters/)). Third, check that the groups came out the size you planned. A **sample ratio mismatch** (SRM) means users went missing from one group or were added to it unevenly, and "Ignoring the SRM without knowing the root cause may result in a bad product modification appearing to be good" ([Fabijan et al., 2019](https://www.microsoft.com/en-us/research/?p=651990)). For a conversion metric, the share of shoppers who buy, the test compares two proportions. With rates p_A and p_B over n shoppers in each group, and p̄ their average: ```text z = (p_B − p_A) / sqrt( p̄ × (1 − p̄) × 2 / n ), p-value = 2 × P(Z > |z|) ``` In the store, simulate 2,000 A/A tests under stated assumptions: 2,000 shoppers a day in each group, 5% of them buy, and each test is planned for 14 days. Test each one at the end, and also every day, stopping at the first p-value below 0.05. Then check one split for a mismatch. Example `aa-tests-with-peeking` (Python 3.12.15 · numpy 2.5.4 · scipy 1.18.1), passing in Edition 2026.10.1: ```python import numpy as np from scipy import stats # Assumptions for the simulation, not the store's data: 2,000 shoppers a day in each group, # 5% of them buy, and the test is planned to run 14 days. Both groups get the same search (an A/A test). TESTS, DAYS, SHOPPERS_PER_DAY, CONVERSION = 2000, 14, 2000, 0.05 rng = np.random.default_rng(2026) buyers = rng.binomial(SHOPPERS_PER_DAY, CONVERSION, size=(TESTS, DAYS, 2)).cumsum(axis=1) # running totals shoppers = SHOPPERS_PER_DAY * np.arange(1, DAYS + 1)[None, :] # per group, by day rate_a, rate_b = buyers[..., 0] / shoppers, buyers[..., 1] / shoppers pooled = (rate_a + rate_b) / 2 z = (rate_b - rate_a) / np.sqrt(pooled * (1 - pooled) * 2 / shoppers) # two-proportion z-test, every day p_value = 2 * stats.norm.sf(np.abs(z)) print(f"simulated A/A tests: {TESTS:,}") print(f"significant at the planned end: {(p_value[:, -1] < 0.05).mean():.3f}") print(f"significant on some day, checked daily: {(p_value < 0.05).any(axis=1).mean():.3f}") # A sample ratio mismatch: the split was planned 50/50, but the groups came out unequal. control, treatment = 28_000, 27_100 p = stats.binomtest(treatment, control + treatment).pvalue print(f"sample ratio mismatch p: {p:.3f}" if p >= 0.001 else "sample ratio mismatch p: below 0.001") ``` Output: ```text simulated A/A tests: 2,000 significant at the planned end: 0.053 significant on some day, checked daily: 0.231 sample ratio mismatch p: below 0.001 ``` Read at the planned end, the share of A/A tests that come out significant is 0.053, close to the 0.05 the threshold promises. Checked daily, the share that look significant on some day rises to 0.231, though neither group's search changed. For a split planned 50/50 that comes out 28,000 to 27,100, a binomial test of 27,100 out of 55,100 shoppers against one half gives a p-value below 0.001: chance almost never produces a gap that size. A p-value is often misread. A p-value of 0.05 doesn't mean a 5% chance that the result is a false positive; Kohavi et al. (2022) call that "a very common misunderstanding". It is the chance of data at least this extreme if the change did nothing. **What to do:** before the test, choose the primary metric and the guardrails that must not get worse, and size the test ([with power](#size-tests-with-power)). Run an A/A test on the same setup first. Check the sample ratio before you read any result, and read the result once, at the planned size. If you must watch a test as it runs, use a sequential method built for continuous monitoring, such as always-valid p-values ([Johari et al., 2015](https://arxiv.org/abs/1512.04922)). ## Size the test with power {#size-tests-with-power} Reading [an A/B test](#run-an-ab-test) once, at its planned size, means planning that size first, from the test's **power**: the chance that it detects a real change of a given size. A test with too little power misses real changes, and when it does come out significant, it tends to exaggerate: "a statistically significant finding with low power is likely to highly exaggerate the size of the effect" ([Kohavi et al., 2022](https://exp-platform.com/abtestingintuitionbusters/)). So decide the size before the test starts, from two numbers: how much the metric varies between users, and the smallest change worth detecting. More variance needs more users. A smaller change needs many more, because the change enters squared. For two groups, 95% confidence and 80% power, Kohavi et al. give a rule of thumb, with σ² the metric's variance and Δ the change to detect ([Kohavi et al., 2009](https://exp-platform.com/Documents/controlledExperimentDMKD.pdf)): ```text users per group ≈ 16 × σ² / Δ² ``` The 16 comes from the normal distribution: 2 × (z for 97.5% + z for 80%)², about 2 × (1.96 + 0.84)². For 90% power, they replace 16 with 21. The example computes the coefficient with SciPy, then checks the rule against their two worked examples: revenue per user with a mean of $3.75 and a standard deviation of $30, and a conversion rate of 5%, each with a 5% relative change to detect. The store's data, from WANDS, has no sales figures, so for the store the example assumes a 5% conversion rate too. Example `sample-size-from-power` (Python 3.12.15 · scipy 1.18.1), passing in Edition 2026.10.1: ```python from scipy import stats SIGNIFICANCE, POWER = 0.05, 0.80 z = stats.norm.ppf(1 - SIGNIFICANCE / 2) + stats.norm.ppf(POWER) # 1.96 + 0.84 print(f"coefficient for two groups: {2 * z ** 2:.1f}") # rounds to the 16 of the rule of thumb def users_per_group(variance, change): """Kohavi et al.'s rule of thumb: about 16 * variance / change**2 users in each group.""" return 16 * variance / change ** 2 # Their worked examples: revenue per user (mean $3.75, standard deviation $30), and a 5% conversion rate, # each with a 5% relative change to detect. print(f"users per group, revenue: {users_per_group(30 ** 2, 0.05 * 3.75):,.0f}") print(f"users per group, conversion: {users_per_group(0.05 * 0.95, 0.05 * 0.05):,.0f}") # The store, assuming a 5% conversion rate like theirs: smaller changes need far more users. for relative in [0.02, 0.01]: print(f"users per group for a {relative:.0%} change: {users_per_group(0.05 * 0.95, 0.05 * relative):,.0f}") ``` Output: ```text coefficient for two groups: 15.7 users per group, revenue: 409,600 users per group, conversion: 121,600 users per group for a 2% change: 760,000 users per group for a 1% change: 3,040,000 ``` The coefficient is 15.7. The revenue example needs 409,600 users per group, and the conversion example 121,600, matching the paper's "over 409,000" and "under 122,000". For the store, detecting a 2% change in conversion takes 760,000 users per group, and a 1% change 3,040,000: halve the change and you need four times the users. The same arithmetic says what the [regression set](#diff-runs-per-query) can detect. Both rankers score the same queries, so the test works on one sample of per-query differences instead of two groups, and the coefficient halves, to 7.8. Take σ as the spread of those differences between two close rankers: semantic retrieval and the [hybrid run](#fuse-with-rrf) that fuses it with BM25. Example `queries-needed-for-power` (Python 3.12.15 · bm25s 0.3.13 · numpy 2.5.4 · pandas 3.0.6 · scipy 1.18.1), passing in Edition 2026.10.1: ```python import numpy as np from scipy import stats from lib.fusion import reciprocal_rank_fusion from lib.lexical import LexicalSearch from lib.metrics import evaluate from lib.semantic import SemanticSearch from lib.store import load_products, load_queries, qrels SIGNIFICANCE, POWER = 0.05, 0.80 queries, judgments = load_queries("test"), qrels("test") bm25 = LexicalSearch(load_products()).run(queries, k=100) semantic = SemanticSearch().run(queries, k=100) hybrid = reciprocal_rank_fusion([bm25, semantic]) change = evaluate(hybrid, judgments)["ndcg@10"] - evaluate(semantic, judgments)["ndcg@10"] spread = change.std() # of the per-query differences z = stats.norm.ppf(1 - SIGNIFICANCE / 2) + stats.norm.ppf(POWER) print(f"spread of per-query changes: {spread:.3f}") print(f"coefficient for paired queries: {z ** 2:.1f}") # one sample of differences, so half of 16 detectable = z * spread / np.sqrt(len(change)) print(f"smallest change 30 queries detect: {detectable:.3f}") print(f"queries needed for a change of 0.01: {z ** 2 * spread ** 2 / 0.01 ** 2:.0f}") # Check the arithmetic by simulation: draw 30 normally distributed changes with that mean and spread, # 10,000 times, and test each. The real differences aren't normal, so this checks the formula, not the data. rng = np.random.default_rng(2026) power = stats.power(stats.ttest_1samp, lambda size: rng.normal(detectable, spread, size=size), (len(change),), significance=SIGNIFICANCE, kwargs={"popmean": 0}) print(f"simulated power at that change: {power.power.item():.3f}") print("simulated power below the planned 0.8:", "yes" if power.power.item() < POWER else "no") ``` Output: ```text spread of per-query changes: 0.062 coefficient for paired queries: 7.8 smallest change 30 queries detect: 0.032 queries needed for a change of 0.01: 300 simulated power at that change: 0.779 simulated power below the planned 0.8: yes ``` The spread is 0.062, so 30 test queries detect a change of 0.032 in nDCG@10 with 80% power. A change of 0.01 would need 300 queries. `scipy.stats.power` checks the arithmetic by simulation, drawing normally distributed differences: at that change, the paired t-test has a simulated power of 0.779. Below the planned 0.8: yes, because the formula uses the normal distribution where the t-test, on 30 queries, uses the wider t-distribution. The real differences aren't normal, so the simulation checks the formula, not the data. When you add queries, add many and judge each shallowly: Webber et al. found that "greater statistical power is achieved for the same relevance assessment effort by evaluating a large number of topics shallowly than a small number deeply", where topics are what information retrieval research calls queries ([Webber et al., 2008](https://people.eng.unimelb.edu.au/ammoffat/abstracts/wmz08cikm.html)). **What to do:** size every A/B test before it starts. If it needs more users than the store sees in a few weeks, test a bigger change, use a metric that varies less, or adjust the metric with each user's data from before the test. On Bing's experiments, Deng et al. report that this "can reduce variance by about 50%" ([Deng et al., 2013](https://exp-platform.com/cuped/)); how much it saves depends on how well a user's past behaviour predicts the metric. Offline, compute the smallest change your regression set can detect, and add queries when your changes are smaller. Fix the number in advance: adding queries until a power estimate says you have enough biases you toward finding significance (Webber et al.). ## Interleave two rankings with team draft {#team-draft-interleaving} An [A/B test](#run-an-ab-test) shows each group a different ranking, so differences between shoppers add noise. **Interleaving** shows each shopper one list that mixes the two rankers' results. Joachims designed it so that "the combined ranking gives (almost) equal presentation bias to both search engines" ([Joachims, 2003](https://cs.cornell.edu/~tj/publications/joachims_02b.pdf)): neither ranker's products sit systematically higher. The ranker whose products draw more clicks wins that search. Interleaving answers a narrower question than an A/B test: which of two rankings users prefer, not an absolute number for either ([Hofmann et al., 2016](https://www.microsoft.com/en-us/research/publication/online-evaluation-information-retrieval/)). **Team draft** mixes the lists like picking teams for a game ([Radlinski et al., 2008](https://cs.cornell.edu/~tj/publications/radlinski_etal_08b.pdf), Algorithm 2). The team with fewer picks goes next, and a coin flip decides when both have the same number. Each pick is the team's highest-ranked product that isn't shown yet. A click counts for the team that picked the product. Radlinski et al. show that under team draft, "a randomly clicking user will always produce equally many preferences for A as for B in expectation", which an earlier method didn't guarantee. Interleave the store's semantic retrieval and BM25 for the test query `clear shower curtain`. The example writes team draft in a few lines, with a seeded random generator for the coin flips, and checks that `lib.experiments` gives the same list: Example `team-draft-one-query` (Python 3.12.15 · bm25s 0.3.13 · numpy 2.5.4 · pandas 3.0.6 · scipy 1.18.1), passing in Edition 2026.10.1: ```python import numpy as np from lib import experiments from lib.lexical import LexicalSearch from lib.semantic import SemanticSearch from lib.store import load_products, load_queries, qrels LABELS = {2: "exact", 1: "partial", 0: "irrelevant"} def team_draft(ranking_a, ranking_b, rng, length=10): """Like picking teams: the team with fewer picks goes next, a coin flip decides when they're level, and each team adds its best product that isn't shown yet.""" shown, teams = [], [] while len(shown) < length: picks_a, picks_b = teams.count("a"), teams.count("b") a_picks = picks_a < picks_b or (picks_a == picks_b and rng.random() < 0.5) ranking, team = (ranking_a, "a") if a_picks else (ranking_b, "b") shown.append(next(p for p in ranking if p not in shown)) teams.append(team) return shown, teams products, queries, judgments = load_products(), load_queries("test"), qrels("test")[246] # clear shower curtain semantic = list(SemanticSearch().search(246, k=100).index) bm25 = list(LexicalSearch(products).search(queries.at[246, "query"], k=100).index) shown, teams = team_draft(semantic, bm25, np.random.default_rng(7)) for position, (product_id, team) in enumerate(zip(shown, teams), start=1): name = {"a": "semantic", "b": "bm25"}[team] print(f"{position:>2} {name:<8} {LABELS.get(judgments.get(product_id), 'unjudged'):<10} {products.at[product_id, 'name'][:45]}") print("same list from lib.experiments:", "yes" if experiments.team_draft(semantic, bm25, np.random.default_rng(7)) == (shown, teams) else "no") ``` Output: ```text 1 bm25 partial rodas vinyl shower curtain 2 semantic exact woolley vinyl solid color single shower curta 3 bm25 exact woolley vinyl single shower curtain 4 semantic partial newburg check 3d vinyl single shower curtain 5 bm25 exact siniard single shower curtain 6 semantic partial 71 '' x71 '' 4pcs shower curtain sets beach t 7 semantic exact almazan lightweight single shower curtain 8 bm25 exact simmon vinyl single shower curtain liner 9 semantic partial vinyl striped shower curtain set 10 bm25 irrelevant gilded pineapple shower curtain hooks same list from lib.experiments: yes ``` Team draft always gives each ranker half of a ten-place list, 5 places each; the coin flips decide which places. A product that both rankers place high appears once, credited to whichever team took it first, so neither ranker gets credit twice for the same product. For this query, one of the shower curtain hooks that BM25 ranks ([chapter 1](#rank-with-bm25)) enters, at the bottom, as BM25's last pick. A shopper's clicks on semantic retrieval's picks count for semantic retrieval, clicks on BM25's picks count for BM25, and the team with more clicks wins the search. **What to do:** use team-draft interleaving to compare two rankers online when you need to know which one users prefer, with fewer searches than an A/B test. Use an A/B test instead when you need an absolute number, such as revenue, or when the change isn't in the ranking. By our reasoning, an interleaved list can't show the effect of latency, page layout or a different number of results. ## Rehearse interleaving with simulated shoppers {#simulate-interleaving-with-clicks} WANDS, the source of the store's data, has no clicks, so the store can't run [team-draft interleaving](#team-draft-interleaving) on real shoppers, but it can rehearse with simulated ones. A **click model** is a probabilistic model of how users scan and click a result list, used both to build metrics and to simulate users ([Chuklin et al., 2015](https://staff.fnwi.uva.nl/m.derijke/wp-content/papercite-data/pdf/chuklin-introduction-2015.pdf)). The **cascade model** says users read from the top and "leave as soon as they see a worthwhile document" ([Craswell et al., 2008](https://www.microsoft.com/en-us/research/?p=156688); see [chapter 2](#clicks-are-not-judgments)). The standard simulation follows it: draw a query uniformly at random, walk down its list, click each product with a probability that depends on its grade, and after a click, leave with another probability that depends on the grade ([Oosterhuis and de Rijke, 2018](https://arxiv.org/abs/1809.08415)). Research on interleaving tests its methods this way, with clicks simulated from labelled data ([Hofmann et al., 2011](https://dare.uva.nl/id/628c8aaf-9dec-4c98-a3d8-b6bbf3962581)). The probabilities are assumptions. Oosterhuis and de Rijke's Table 2 gives a "navigational" user for five grades, one looking for a single highly relevant result. We map irrelevant, partial and exact to that user's lowest, middle and highest grades, count unjudged products as irrelevant, as the metrics do, and draw every query equally often. The values live in one place, `lib.experiments`: the shopper clicks an exact product with probability 0.95 and then leaves with probability 0.9; a partial product gets 0.5 and 0.5, and an irrelevant one 0.05 and 0.2. Send one such shopper down the team-draft list for `clear shower curtain`, by hand: Example `one-simulated-shopper` (Python 3.12.15 · bm25s 0.3.13 · numpy 2.5.4 · pandas 3.0.6 · scipy 1.18.1), passing in Edition 2026.10.1: ```python import numpy as np from lib.experiments import NAVIGATIONAL, cascade_clicks, team_draft from lib.lexical import LexicalSearch from lib.semantic import SemanticSearch from lib.store import load_products, load_queries, qrels LABELS = {2: "exact", 1: "partial", 0: "irrelevant"} products, queries, judgments = load_products(), load_queries("test"), qrels("test")[246] # clear shower curtain semantic = list(SemanticSearch().search(246, k=100).index) bm25 = list(LexicalSearch(products).search(queries.at[246, "query"], k=100).index) shown, teams = team_draft(semantic, bm25, np.random.default_rng(7)) # the team-draft example's list team_names = {"a": "semantic", "b": "bm25"} rng = np.random.default_rng(2026) clicks = [] for position, product_id in enumerate(shown): grade = judgments.get(product_id, 0) # unjudged counts as 0 clicked = rng.random() < NAVIGATIONAL["click"][grade] leaves = clicked and rng.random() < NAVIGATIONAL["stop"][grade] # only a click can end the visit label = LABELS.get(judgments.get(product_id), "unjudged") print(f"{position + 1:>2} {team_names[teams[position]]:<8} {label:<10} {'clicks' if clicked else '':<6} {'leaves' if leaves else ''}") if clicked: clicks.append(position) if leaves: break credited = [team_names[teams[position]] for position in clicks] # each click counts for the team that picked it for_semantic, for_bm25 = credited.count("semantic"), credited.count("bm25") print("clicks for semantic:", for_semantic) print("clicks for bm25:", for_bm25) print("this search goes to:", "semantic" if for_semantic > for_bm25 else "bm25" if for_bm25 > for_semantic else "neither, a tie") print("same clicks from lib.experiments:", "yes" if cascade_clicks(shown, judgments, np.random.default_rng(2026)) == clicks else "no") ``` Output: ```text 1 bm25 partial clicks 2 semantic exact clicks leaves clicks for semantic: 1 clicks for bm25: 1 this search goes to: neither, a tie same clicks from lib.experiments: yes ``` The shopper's clicks count 1 for semantic retrieval and 1 for BM25, so this search goes to neither, a tie. Now simulate 10,000 interleaved searches, with a fixed seed, for two pairs: semantic retrieval against BM25, and the [hybrid run](#fuse-with-rrf) against semantic retrieval. A sign test asks whether wins outnumber losses by more than chance; [Radlinski et al.](https://cs.cornell.edu/~tj/publications/radlinski_etal_08b.pdf) use "a binomial sign test", leaving ties out. Read the verdict at 100, 1,000 and 10,000 searches: Example `simulated-interleaving-verdict` (Python 3.12.15 · bm25s 0.3.13 · numpy 2.5.4 · pandas 3.0.6 · scipy 1.18.1), passing in Edition 2026.10.1: ```python import numpy as np from scipy import stats from lib.experiments import NAVIGATIONAL, simulate_interleaving from lib.fusion import reciprocal_rank_fusion from lib.lexical import LexicalSearch from lib.semantic import SemanticSearch from lib.store import load_products, load_queries, qrels for grade, label in [(2, "exact"), (1, "partial"), (0, "irrelevant")]: # the assumptions, from lib.experiments print(f"{label} click probability: {NAVIGATIONAL['click'][grade]}") print(f"{label} stop probability: {NAVIGATIONAL['stop'][grade]}") def ties_reversed(scores): # BM25's ranking, with equal scores in descending product_id order instead return scores.index[np.lexsort((-scores.index.to_numpy(), -scores.to_numpy()))].tolist() queries, judgments = load_queries("test"), qrels("test") lexical = LexicalSearch(load_products()) bm25 = lexical.run(queries, k=100) # equal scores by product_id semantic = SemanticSearch().run(queries, k=100) hybrid = reciprocal_rank_fusion([bm25, semantic]) bm25_reversed = {q: ties_reversed(lexical.search(query, k=100)) for q, query in queries["query"].items()} pairs = {"semantic against bm25": (semantic, bm25), "hybrid against semantic": (hybrid, semantic), "hybrid with bm25 ties reversed against semantic": (reciprocal_rank_fusion([bm25_reversed, semantic]), semantic)} below = {} for name, (run_a, run_b) in pairs.items(): outcomes = simulate_interleaving(run_a, run_b, judgments, 10_000, np.random.default_rng(2026)) for searches in [100, 1_000, 10_000]: wins, losses = (outcomes[:searches] == 1).sum(), (outcomes[:searches] == -1).sum() p = stats.binomtest(wins, wins + losses).pvalue # sign test; ties left out below[name, searches] = p < 0.05 shown = f"{p:.3f}" if p >= 0.001 else "below 0.001" print(f"{name}, {searches:,} searches: {wins} wins, {losses} losses, p {shown}") changes = below["hybrid against semantic", 100] != below["hybrid with bm25 ties reversed against semantic", 100] print("hybrid against semantic, verdict at 100 searches changes with bm25 ties reversed:", "yes" if changes else "no") ``` Output: ```text exact click probability: 0.95 exact stop probability: 0.9 partial click probability: 0.5 partial stop probability: 0.5 irrelevant click probability: 0.05 irrelevant stop probability: 0.2 semantic against bm25, 100 searches: 51 wins, 36 losses, p 0.133 semantic against bm25, 1,000 searches: 519 wins, 347 losses, p below 0.001 semantic against bm25, 10,000 searches: 5222 wins, 3552 losses, p below 0.001 hybrid against semantic, 100 searches: 32 wins, 51 losses, p 0.048 hybrid against semantic, 1,000 searches: 413 wins, 447 losses, p 0.260 hybrid against semantic, 10,000 searches: 4180 wins, 4593 losses, p below 0.001 hybrid with bm25 ties reversed against semantic, 100 searches: 35 wins, 48 losses, p 0.187 hybrid with bm25 ties reversed against semantic, 1,000 searches: 410 wins, 463 losses, p 0.078 hybrid with bm25 ties reversed against semantic, 10,000 searches: 4174 wins, 4560 losses, p below 0.001 hybrid against semantic, verdict at 100 searches changes with bm25 ties reversed: yes ``` For semantic retrieval against BM25, a large gap offline, the verdict reads 51 wins, 36 losses, p 0.133 at 100 searches, and 519 wins, 347 losses, p below 0.001 at 1,000. For the hybrid against semantic retrieval, it reads 32 wins, 51 losses, p 0.048 at 100 searches and 413 wins, 447 losses, p 0.260 at 1,000: small samples swing, which is why you fix the number of searches before the test starts. At 10,000 it reads 4180 wins, 4593 losses, p below 0.001. That doesn't overturn [chapter 3](#compare-runs-in-a-harness)'s finding of no established difference: with ever more simulated clicks on the same 30 queries, the sign test measures only click noise, so any fixed gap becomes significant. The hybrid's verdict at 100 searches also rests on a detail that carries no information. The hybrid fuses BM25, and `lib.lexical` orders equal BM25 scores by `product_id`. Order them the other way, an equally arbitrary rule, and the same seed reads 35 wins, 48 losses, p 0.187 at 100 searches. The verdict at 0.05 changes: yes. Reversing BM25's ties changes some of the hybrid's lists, so the simulated shoppers see other products, and every random draw after the first changed list can shift. A verdict that a tie rule can flip is luck, not a finding about the rankers. Three limits keep this a rehearsal. The clicks come from the same grades nDCG@10 uses, so interleaving tends to agree with nDCG@10; it adds no evidence about the rankers. Every search comes from the same 30 queries, so the sign test speaks for these queries only. And real shoppers aren't this model. Deffayet et al. studied what happens when the ranker that produced a click log changes, and found that "click models are sensitive to it" ([Deffayet et al., 2023](https://dare.uva.nl/id/4f8740ca-2677-4a9a-bb04-9475ceafb007)). On real traffic at three search engines, interleaving agreed with expert judgments whenever both were significant, and needed "approximately 1–2 orders of magnitude less data than absolute metrics" ([Chapelle et al., 2012](https://www.cs.cornell.edu/people/tj/publications/chapelle_etal_12a.pdf)). By our reasoning, the simulation can't show that advantage, because its shoppers all behave alike. **What to do:** rehearse an interleaving test in simulation to check your code and your sign test. Treat any count of searches it suggests as only as good as the assumed shopper, and rerun a simulated verdict with ties broken the other way before you read anything into it. Never report a simulated verdict as evidence about a ranker. On real traffic, fix the number of searches in advance and decide with a sign test. ## Find the cause when offline and online disagree {#offline-online-disagree} [Simulated shoppers](#simulate-interleaving-with-clicks) click by the same grades nDCG@10 uses, so they tend to agree with it; real shoppers need not. Offline metrics sometimes predict online results well. In one e-commerce study of 36 offline metrics, the best agreed with online results under a business metric about which of two ranking models was better "up to 97% of times" ([Wang et al., 2023](https://www.amazon.science/publications/how-well-do-offline-metrics-predict-online-performance-of-product-ranking-models)). Another e-commerce team found that offline ranking metrics "can show significant inconsistency with online performance, and can even be misleading" ([Huzhang et al., 2020](https://arxiv.org/abs/2003.11941)). Both are industry studies on their own traffic, which nobody outside can reproduce. When two systems perform about equally, "disagreements between different metrics are common" ([Hofmann et al., 2016](https://www.microsoft.com/en-us/research/publication/online-evaluation-information-retrieval/)). Disagreements have causes you can check. The judgments may miss products the online winner shows, which [the sensitivity check](#judge-new-results) looks for. The change may act outside the ranking, on speed or layout. Or the online metric may measure something other than success. On an academic search engine, [Radlinski et al. (2008)](https://cs.cornell.edu/~tj/publications/radlinski_etal_08b.pdf) found that clicks per query, like seven other absolute usage metrics, didn't reliably reflect retrieval quality at the sample sizes they studied. In the store, run a simulated A/B test with the navigational shopper from [the interleaving rehearsal](#simulate-interleaving-with-clicks): 5,000 searches on BM25 as the control and 5,000 on semantic retrieval as the treatment, all on test queries. Measure clicks per search, and the share of searches with a click on an exact product. Then repeat with Table 2's "perfect" user, mapped the same way, who reads all ten results, clicks every exact product and never leaves early. Before you run it, predict: which ranker gets more clicks per search? Example `clicks-per-search-misleads` (Python 3.12.15 · bm25s 0.3.13 · numpy 2.5.4 · pandas 3.0.6 · scipy 1.18.1), passing in Edition 2026.10.1: ```python import numpy as np from scipy import stats from lib.experiments import SHOPPERS, simulate_searches from lib.lexical import LexicalSearch from lib.semantic import SemanticSearch from lib.store import load_products, load_queries, qrels queries, judgments = load_queries("test"), qrels("test") arms = {"bm25": LexicalSearch(load_products()).run(queries, k=100), # control "semantic": SemanticSearch().run(queries, k=100)} # treatment metrics = {"clicks per search": "clicks", "share of searches with an exact click": "exact_click"} significant_leaders = {label: set() for label in metrics} # rankers ahead at p below 0.05 for shopper_name, shopper in SHOPPERS.items(): # navigational, then perfect rng = np.random.default_rng(2026) searches = {name: simulate_searches(run, judgments, 5_000, rng, shopper) for name, run in arms.items()} for label, column in metrics.items(): means = {name: table[column].mean() for name, table in searches.items()} for name, mean in means.items(): print(f"{shopper_name} shopper, {name} {label}: {mean:.3f}") p = stats.ttest_ind(searches["semantic"][column].astype(float), searches["bm25"][column].astype(float)).pvalue print(f"{shopper_name} shopper, {label} p: " + (f"{p:.3f}" if p >= 0.001 else "below 0.001")) if p < 0.05: significant_leaders[label].add(max(means, key=means.get)) for label in metrics: print(f"{label} reverses between shoppers:", "yes" if len(significant_leaders[label]) > 1 else "no") ``` Output: ```text navigational shopper, bm25 clicks per search: 1.362 navigational shopper, semantic clicks per search: 1.269 navigational shopper, clicks per search p: below 0.001 navigational shopper, bm25 share of searches with an exact click: 0.695 navigational shopper, semantic share of searches with an exact click: 0.799 navigational shopper, share of searches with an exact click p: below 0.001 perfect shopper, bm25 clicks per search: 6.443 perfect shopper, semantic clicks per search: 7.451 perfect shopper, clicks per search p: below 0.001 perfect shopper, bm25 share of searches with an exact click: 0.898 perfect shopper, semantic share of searches with an exact click: 0.895 perfect shopper, share of searches with an exact click p: 0.646 clicks per search reverses between shoppers: yes share of searches with an exact click reverses between shoppers: no ``` For the navigational shopper, BM25 gets 1.362 clicks per search and semantic retrieval 1.269, with a p-value below 0.001. A shopper who finds an exact product near the top clicks it and usually leaves, while one shown partial products clicks several, still looking. For the perfect shopper, semantic retrieval gets 7.451 against BM25's 6.443. Clicks per search reverses between shoppers: yes. Its sign depends on how shoppers behave, so on its own it can't measure success. The share of searches with an exact click favours semantic retrieval for the navigational shopper, 0.799 against 0.695. It reverses between shoppers: no. For the perfect shopper it shows no clear difference, with a p-value of 0.646: that shopper clicks an exact product in 0.898 of BM25's searches and 0.895 of semantic retrieval's, whenever the top 10 holds one. If you predicted one answer for every shopper, you treated clicks as a measure of quality. They measure behaviour, and behaviour varies. **What to do:** when offline and online results disagree, find the cause before you trust either. Check that the online metric measures success, such as a purchase, and not activity. Check judged@10 on the online winner's results, and judge what is missing. Then decide on the online result of the metric you chose in advance, and add queries or judgments to the regression set so that it predicts better next time. ## Decide whether to ship {#decide-whether-to-ship} A decision to ship combines the offline and online checks. Write the checklist down before you see any result, so that no single number can talk you into a change. We recommend these items, each with the section that shows it: 1. **An established offline gain on the regression set,** by the planned-comparison rule, with its t-test and interval ([per-query diff](#diff-runs-per-query)), against the strongest system you have. Armstrong et al. found that over a decade of published gains on TREC collections, "Baselines are generally weak, often being below the median original TREC system" ([Armstrong et al., 2009](https://people.eng.unimelb.edu.au/ammoffat/abstracts/amwz09cikm.html)). 2. **Losses read,** and URisk with α = 1 not negative ([URisk](#weigh-losses-with-urisk)). 3. **Judgments that cover both runs:** a leader that holds when either run's unjudged products count as exact, or new judgments where it doesn't ([unjudged products](#judge-new-results)). 4. **Human grades for the headline scores** of LLM-based rankers ([chapter 4](#llm-graded-verdicts)). 5. **Latency and cost within budget** ([chapter 9](#rerank-depth-from-budget)). 6. **Every setting chosen on train queries,** with each look at the test queries logged ([held-out queries](#keep-test-queries-held-out)). 7. **An online plan written in advance:** the primary metric and guardrails, the size from a [power calculation](#size-tests-with-power), an A/A test, a sample ratio check and no peeking ([A/B test](#run-an-ab-test)). Interleaving can screen candidates first ([team draft](#team-draft-interleaving)). For the store's switch from BM25 to semantic retrieval, here are the first three items. Item 1, an established gain: yes, with a mean change of 0.105 and a 95% interval of 0.055 to 0.185. Item 2, URisk with α = 1: 0.092. Item 3, judgments that cover both runs: if BM25's unjudged products were all exact, BM25 would score 0.788 and semantic retrieval 0.829, and the leader changes: no. The online items wait for traffic. **What to do:** ship when every offline item passes and the online test, sized in advance, shows a significant gain on the primary metric with no guardrail worse. When an offline item fails, fix it before going online, because online tests spend shoppers' time. When the online test is inconclusive at its planned size, keep the baseline. ## What goes wrong in production {#testing-changes-in-production} A store tests change after change on live traffic, with real shoppers in place of simulated ones and the same regression set at every step. **The dashboard invites peeking.** An experiment platform shows results every day, and a team that stops the first time a result looks significant gets far more than 5% false positives, as the [A/A simulation](#run-an-ab-test) shows. Fix the size in advance, or use a sequential method. **The groups don't match the plan.** [Fabijan et al.](https://www.microsoft.com/en-us/research/?p=651990) collected the causes of sample ratio mismatches from several companies into a taxonomy. Check the split before you read any result, and don't trust the result until you know why the split is off. **The regression set wears out.** Every decision made by looking at the test queries spends some of their independence ([Dwork et al., 2015](https://arxiv.org/abs/1506.02629)). Log each look, and refresh the test set with new queries and judgments when the log grows long. **An LLM judge becomes the gate.** LLM judgments are cheap enough to score every candidate. Otero et al. found that significance tests on LLM judgments showed "an exceedingly high rate of false positives regarding statistical differences" ([Otero et al., 2025](https://arxiv.org/abs/2411.13212)). Before you ship on an LLM-judged gain, confirm it on human judgments ([chapter 4](#llm-graded-verdicts)). **Ties move results.** Products with equal scores come back in whatever order a library's selection leaves them ([chapter 3](#measuring-search-in-production)), so a change that touches no score can still move a result. Reversed BM25 ties change the store's [100-search interleaving verdict](#simulate-interleaving-with-clicks): yes. Break ties by an ID in every ranker, and rerun a result near its threshold with ties reversed before you act on it. **Interleaving can't see the whole page.** By our reasoning, a ranker that is slower, or that changes how many results appear, can win an interleaved test and still lose shoppers. Run an A/B test for changes like these. **What to do:** automate the A/A test, the sample ratio check and the fixed test size in the experiment platform, keep a log of every look at the test queries, break ties by an ID, and confirm LLM-judged gains on human judgments. ## Decisions this chapter made {#testing-changes-decisions} Each choice below is a default, with its reason in the section it links to. - **The regression set is the 30 test queries with their judgments,** and every choice uses the 50 train queries ([held-out queries](#keep-test-queries-held-out)). - **One planned comparison on the test queries,** semantic retrieval against BM25, established only by a permutation p-value below 0.05, with the t-test and a 95% interval beside it ([per-query diff](#diff-runs-per-query)). The 128-dimension ranker is a demonstration, chosen for how it behaves on test ([URisk](#weigh-losses-with-urisk)). - **Every candidate gets a per-query diff,** and its losses are read by hand. - **URisk uses α = 1,** so each loss counts double. It is a judgement about how much a regression costs, and a store with expensive failures should raise it ([URisk](#weigh-losses-with-urisk)). - **When the leader changes between the two unjudged scenarios, judge before you compare,** and keep the new judgments in the regression set ([unjudged products](#judge-new-results)). - **Test queries are looked at once per decision,** and each look is logged ([held-out queries](#keep-test-queries-held-out)). - **A/B tests are sized with 16σ²/Δ² before they start,** read once at that size, and preceded by an A/A test and a sample ratio check ([power](#size-tests-with-power)). - **Interleaving uses team draft and a sign test** ([team draft](#team-draft-interleaving)). - **Simulated shoppers follow a navigational cascade model mapped onto three grades,** with uniform query frequency, probabilities stated as assumptions in `lib.experiments` and a fixed seed for every simulation. Their verdicts rehearse a method and never count as evidence ([rehearsal](#simulate-interleaving-with-clicks)). - **Clicks per search is not a success metric on its own.** Use a measure of success, such as a purchase ([offline and online](#offline-online-disagree)).