---
title: "A Primer on Search Relevance"
chapter: "What relevance means"
chapter_id: what-relevance-means
book: primer-on-search-relevance
edition: 2026.11.10
verified: 2026-11-19
edition_code: ANXY-CR00
canonical: https://rollingedition.com/primer-on-search-relevance/what-relevance-means
license: CC BY-NC 4.0
more_permissions: https://rollingedition.com/policies/reuse
attribution: "“What relevance means,” chapter 2 of A Primer on Search Relevance, Edition 2026.11.10, examples passed Nov 19, 2026. By Rolling Edition, https://rollingedition.com/primer-on-search-relevance/what-relevance-means. Text CC BY-NC 4.0 (https://creativecommons.org/licenses/by-nc/4.0/), with added permissions (https://rollingedition.com/policies/reuse). Code MIT, copyright Rolling Edition and contributors. Written with AI. Verified by people."
---
# What relevance means

By the end of this chapter you can read a set of relevance judgments critically: say what each grade claims about a product, find where the judgments came from and where they have holes, and decide how far to trust a score computed from them.

Every score so far has taken the store's judgments as fact. [Chapter 1](#search-end-to-end) scored two rankers against 13,077 judgments, the grades people gave pairs of a query typed at [the store](#store-data) and a product it sells. On the [test queries](#store-data), [BM25](#rank-with-bm25) averages an [nDCG@10](#score-one-ranking) of 0.725. [Semantic retrieval](#search-by-meaning) scores 0.830. Each of those numbers rests on choices the judgments make silently: whose idea of relevance they record, how much a near miss is worth, and what happens to the products nobody judged.

This chapter reads the store's judgments closely. It changes no ranker, so it reads [train and test queries](#store-data) alike: reading judgments isn't tuning, and every fix in this book is still chosen on train queries only. The few tests it runs on the test queries are exploratory, so they describe the judgments and establish no difference ([the book's rule](#when-a-difference-is-established)). [Chapter 3](#compare-runs-in-a-harness) tests the gap between BM25 and semantic retrieval as a planned comparison.

## Judge relevance against the shopper's need {#relevance-is-a-judgment}

A search system returns products that match a query in the way the system defines a match: shared words for lexical retrieval, nearby vectors for semantic retrieval. Information science calls that *system* relevance. It separates that from *topical* relevance, whether a product is about the query's subject, and *situational* relevance, whether it helps with the task at hand ([Saracevic, 2007](https://tefkos.rutgers-sci.domains/Saracevic%20relevance%20pt%20II%20JASIST%20%2707.pdf)). Some argue that system relevance is always perfect, because the system retrieved what the query asked for. Saracevic's answer is "Not so."

A judgment is a verdict, and this chapter asks what each grade claims. The store's judgments come from WANDS, a public product-search dataset, through a sample of its queries and products ([Load the store's data](#store-data)). WANDS's judges label each pair exact, partial or irrelevant, which the book grades 2, 1 and 0. Exact means the product is the kind of thing the query asks for and has every attribute the query names. Partial means it is the right kind of thing but misses an attribute. Irrelevant means it isn't the kind of thing asked for ([Read intent as a product and its modifiers](#intent-in-product-search)). The judge grades the product against the **information need** behind the query's words, the thing the shopper wanted, not against the words. Manning et al. define a document as relevant when it addresses the stated information need, not merely because it contains the query's words ([Manning et al., 2008, chapter 8](https://nlp.stanford.edu/IR-book/html/htmledition/information-retrieval-system-evaluation-1.html)). Evaluation then makes a bargain. It fixes a set of documents, a set of need statements and their judgments, and abstracts away the real users ([Voorhees, 2007](https://cs.uwaterloo.ca/~brecht/courses/854-Experimental-Performance-Evaluation-2017/readings/trec-cacm-2007.pdf)). The judgments stand in for shoppers here, and every score in this book inherits that bargain.

If matching words were the same as relevance, a product whose text holds every word of the query would be exact. You can test that on every judged pair in the store: for each pair, ask whether the product's name, class, category and description contain every word of the query, with the sample's descriptions cut to about 400 characters.

Of the judged pairs whose text holds every query word, what share did the judges call exact? Hold a number in mind.

Example `word-match-versus-judgment` (Python 3.12.15 · numpy 2.5.5 · pandas 3.1.1), passing in Edition 2026.11.10:

```python
"""Checks whether a full word match makes a judged pair exact, and how many exact pairs have a text that misses a
query word, with and without attributes. The prose's point rests on the `of them exact` line and the
`of them, still missing one with attributes added` line."""
import re

from lib.store import load_labels, load_products, load_queries, product_text


def words(text):
    """Lowercase word tokens: a rough stand-in for what lexical search matches."""
    return set(re.findall(r"\w+", text.lower()))


products, queries, labels = load_products(), load_queries(), load_labels()
text_words = {product_id: words(text) for product_id, text in product_text(products).items()}
attribute_words = {product_id: words(text) for product_id, text in products["attributes"].items()}
query_words = {query_id: words(query) for query_id, query in queries["query"].items()}
pairs = list(zip(labels["query_id"], labels["product_id"]))
# on sets, <= asks whether every query word is in the product's text
labels["in text"] = [query_words[query_id] <= text_words[product_id] for query_id, product_id in pairs]
# | is set union: the product's text words with its attribute words added
labels["in text or attributes"] = [query_words[query_id] <= text_words[product_id] | attribute_words[product_id]
                                   for query_id, product_id in pairs]

matched = labels[labels["in text"]]
print(f"exact share of all judged pairs: {(labels['label'] == 'exact').mean():.3f}")
print(f"judged pairs whose text has every query word: {len(matched):,}")
for label in ["exact", "partial", "irrelevant"]:
    print(f"of them {label}: {(matched['label'] == label).mean():.3f}")

exact = labels[labels["label"] == "exact"]
print(f"exact pairs: {len(exact):,}")
print(f"exact pairs whose text misses a query word: {(~exact['in text']).sum():,}")
print(f"of them, still missing one with attributes added: {(~exact['in text or attributes']).sum():,}")

curtains = list(exact.loc[exact["query_id"] == 246, "product_id"])                 # query 246: clear shower curtain
unsaid = [product_id for product_id in curtains if "clear" not in text_words[product_id]]
print("exact products for clear shower curtain:", len(curtains))
print("of them without the word clear in their text:", len(unsaid))
print("of those, with clear in their attributes:", sum("clear" in attribute_words[product_id] for product_id in unsaid))
```

Output:

```text
exact share of all judged pairs: 0.249
judged pairs whose text has every query word: 1,177
of them exact: 0.777
of them partial: 0.211
of them irrelevant: 0.012
exact pairs: 3,252
exact pairs whose text misses a query word: 2,337
of them, still missing one with attributes added: 2,251
exact products for clear shower curtain: 12
of them without the word clear in their text: 8
of those, with clear in their attributes: 4
```

A full word match is a strong signal, but it doesn't settle the question. Of the 1,177 judged pairs whose text holds every query word, 0.777 are exact, 0.211 partial and 0.012 irrelevant, against an exact share of 0.249 over all judged pairs. The words can be present without the product being what the shopper meant, so the judges decide, not the words. A share near 1 would mean matching words is relevance, and the judges say it isn't.

The other direction is more striking. Most exact pairs, 2,337 of the 3,252, have a product text that misses at least one query word. Adding the attributes still leaves 2,251 of those missing a word, so attributes explain few misses overall.

Some misses are word forms and spelling, such as `chairs` against `chair` ([chapter 1](#search-end-to-end-in-production)) or the query `ligth bulb`. [Chapter 5](#stem-word-forms) and [chapter 8](#correct-spelling-from-catalog) take those up. Others are attributes the text doesn't state. WANDS records attributes such as color, material and size apart from the text ([Chen et al., 2022, Table 1](https://easychair.org/publications/preprint/wm4R)), and the store keeps a few of them in an `attributes` column, for example `color: clear; material: vinyl` ([data card](https://github.com/rollingedition/book-primer-on-search-relevance/blob/67880c1fa437fcb912ca77d75451f3169c432ea6/examples/data/wands/README.md)). The text the store's search reads, `lib.store.product_text`, leaves that column out.

For some queries, though, the attributes hold the answer. For `clear shower curtain`, 8 of the 12 exact products never say `clear` in their text, and 4 of those carry `clear` in their attributes, where the store's text search never looks. [Chapter 5](#index-attributes-as-a-field) tests adding attributes to lexical search. The rest of those curtains say `clear` nowhere, in their text or their attributes.

The judges had more to go on. WANDS's annotation guidelines show them each product's name, image and web page ([guidelines](https://github.com/wayfair/WANDS/raw/3b74dcf4ba29ab8ff3e6a50b5b09fc627cb882b5/Product%20Search%20Relevance%20Annotation%20Guidelines.pdf)). The photo or the page may be what told them, but that is our guess, because the data doesn't record which evidence a judge used.

**Working rule:** Treat a judgment as a person's verdict about a need, not a property of the product. Before you blame a ranker for missing an exact product, check whether any field the ranker reads says what the judges saw, and look at the fields it doesn't read, such as attributes. When your judges see more than your ranker does, give the ranker the same evidence where you can, or accept that some exact products will rank low for reasons no change to the ranking can fix.

## Expect judges to disagree on some pairs {#judges-disagree-on-some-pairs}

Two people can reach different verdicts. Each WANDS label is already the majority vote of up to three annotators ([Chen et al., 2022, section 5](https://easychair.org/publications/preprint/wm4R)), and the annotators, who began in moderate agreement, converged with training. Cohen's kappa, which discounts chance agreement, rose from 0.467 in their first two months to 0.664 in the third and 0.826 in the fourth ([Table 2](https://easychair.org/publications/preprint/wm4R)). [Chapter 4](#agreement-beyond-chance) computes kappa.

WANDS's file lists some of the store's pairs more than once, each row a majority vote, and for a few the votes differ. The store's sample keeps one row per pair, graded 2 for exact, 1 for partial and 0 for irrelevant, and where the rows differ it keeps the lower grade ([Load the store's data](#store-data)). Here are the pairs whose rows differ, with the grade the sample kept:

Example `conflicting-judgments` (Python 3.12.15 · numpy 2.5.5 · pandas 3.1.1), passing in Edition 2026.11.10:

```python
"""Lists the pairs WANDS labelled two ways, with the label the sample kept, and counts the conflicts by kind. The
prose's point rests on the `conflicts between neighbouring grades` line."""
import pandas as pd

from lib.store import DATA, load_labels, load_products, load_queries

GRADES = {"exact": 2, "partial": 1, "irrelevant": 0}

products, queries, labels = load_products(), load_queries(), load_labels()
conflicts = pd.read_csv(DATA / "label_conflicts.csv")       # pairs WANDS lists more than once, with different labels
# the sample's kept label for each pair, which is the lower grade wherever WANDS's rows differ
kept = labels.set_index(["query_id", "product_id"])["label"]

for row in conflicts.itertuples():
    print(f"{queries.at[row.query_id, 'query']:<32} {products.at[row.product_id, 'name'][:40]:<41} "
          f"{row.labels:<19} kept: {kept[(row.query_id, row.product_id)]}")

# each conflicting pair's two labels, from the column where "|" separates them
pairs = conflicts["labels"].str.split("|")
# a spread of 1 means neighbouring grades: exact and partial, or partial and irrelevant
spread = pairs.apply(lambda both: max(GRADES[label] for label in both) - min(GRADES[label] for label in both))
print()
print("pairs with conflicting labels:", len(conflicts))
# frozensets ignore which of the two labels comes first
print("exact against partial:", (pairs.apply(frozenset) == frozenset(["exact", "partial"])).sum())
print("partial against irrelevant:", (pairs.apply(frozenset) == frozenset(["partial", "irrelevant"])).sum())
print(f"conflicts between neighbouring grades: {(spread == 1).sum()} of {len(conflicts)}")
```

Output:

```text
whimsical solar lights           solar powered led pathway light pack      exact|partial       kept: partial
indoor multi tiered plant stand  taranto multi-tiered plant stand          exact|partial       kept: partial
clear shower curtain             shower curtain hooks                      irrelevant|partial  kept: irrelevant
breakfast bar table              counter height dining table               exact|partial       kept: partial
wood floor protector mat         hard floor straight rectangular chair ma  exact|partial       kept: partial
malachi sled                     sled coffee table                         irrelevant|partial  kept: irrelevant
kohler purist brushed bronze     purist widespread bathroom faucet with d  exact|partial       kept: partial

pairs with conflicting labels: 7
exact against partial: 5
partial against irrelevant: 2
conflicts between neighbouring grades: 7 of 7
```

Of the judged pairs, 7 conflict: 5 are exact against partial and 2 partial against irrelevant. Every conflict sits between neighbouring grades, 7 of 7, so the soft spot is the line between two grades, not the gap between exact and irrelevant. The `breakfast bar table` row is one such case: whether a counter height dining table fully matches that query is a fair question about a modifier, not a sign that one vote misread the task.

**Working rule:** Keep one grade per pair and write down the rule that chose it. We recommend the lower grade, as the sample does: it errs against the ranker that ranks the product, not in its favour. When judges disagree across two grades, exact against irrelevant, reread the guidelines with them, because they don't share a definition.

## Read intent as a product and its modifiers {#intent-in-product-search}

Every judgment starts from a reading of what the query asks for. A shopper's **intent** is what they want to do. A widely cited taxonomy of web search sorts intent into navigational (reach a particular site), informational (find information) and transactional (do something on the web), and its author makes no assumption that intent can be inferred from the query with any certainty ([Broder, 2002](https://sigir.org/files/forum/F2002/broder.pdf)). Product search reshapes those kinds. Tsagkias et al. describe navigational product queries as ones that name a serial number or a product, and informational ones as usually mapping to a leaf in the inventory taxonomy or asking about specific attributes. They add that "there is no single taxonomy for eCommerce customer intent", because it depends on the store's business and inventory ([Tsagkias et al., 2020, section 2](https://sigir.org/wp-content/uploads/2020/06/p04.pdf)).

WANDS writes intent into its labels. A query names a **target entity**, the kind of product, and the query's **modifiers** are the attributes the shopper asked for. Exact satisfies both. Partial matches only the target entity and fails the modifiers ([Chen et al., 2022, section 3.2](https://easychair.org/publications/preprint/wm4R)). The guidelines' worked example is a query for a kids' chair: a product that is a chair but not for kids is partial ([guidelines](https://github.com/wayfair/WANDS/raw/3b74dcf4ba29ab8ff3e6a50b5b09fc627cb882b5/Product%20Search%20Relevance%20Annotation%20Guidelines.pdf)). Even with that rule, a broad query can be hard to judge. The authors' own example is a shopper who wants an induction cooktop and types `cooktop` ([section 1](https://easychair.org/publications/preprint/wm4R)).

The store's queries cover the range. `ines shag rug w003096809` and `kohler whitehaven farmhouse kitchen sink` name a product line or a model, which makes them navigational. `outdoor light fixtures` names a category. `clear shower curtain` and `48 inch kitchen hood` add a modifier to an entity. The store's queries also carry `query_class`, WANDS's *predicted* class for each query ([Table 1](https://easychair.org/publications/preprint/wm4R)). It is a guess at the entity, says nothing about modifiers, and isn't a judgment. Check how often it agrees with the products the judges called exact:

Example `predicted-class-check` (Python 3.12.15 · numpy 2.5.5 · pandas 3.1.1), passing in Edition 2026.11.10:

```python
"""For each query, measures how many of its exact products carry the predicted class, and lists the queries where none
does. The prose's point rests on the `queries where most exact products have the query_class` line and the
`queries where no exact product has the query_class` line."""
import pandas as pd

from lib.store import load_labels, load_products, load_queries

products, queries, labels = load_products(), load_queries(), load_labels()
exact = labels[labels["label"] == "exact"].join(products["product_class"], on="product_id")

rows = []
for query_id, group in exact.groupby("query_id"):
    predicted = queries.at[query_id, "query_class"]
    classes = group["product_class"].str.split("|")                 # a product can have several classes
    # the commonest product_class value, which can join several classes, comes first in value_counts(); a missing class is an empty string, which prints as (none)
    rows.append({"query": queries.at[query_id, "query"], "query_class": predicted,
                 "share of exact products in it": classes.apply(lambda names: predicted in names).mean(),
                 "commonest class of exact products": group["product_class"].value_counts().index[0] or "(none)"})
check = pd.DataFrame(rows)

print("queries:", len(check))
print("queries without a query_class:", check["query_class"].isna().sum())
# most means more than half of a query's exact products carry the predicted class
print("queries where most exact products have the query_class:",
      (check["share of exact products in it"] > 0.5).sum())
missed = check[(check["share of exact products in it"] == 0) & check["query_class"].notna()]
print("queries where no exact product has the query_class:", len(missed))
print()
print(missed.drop(columns="share of exact products in it").to_string(index=False))
```

Output:

```text
queries: 80
queries without a query_class: 1
queries where most exact products have the query_class: 63
queries where no exact product has the query_class: 6

                   query               query_class commonest class of exact products
foutains with brick look                 Wallpaper                         Fountains
  70s inspired furniture                  Wall Art                            (none)
         7qt slow cooker Crock Pots & Slow Cookers                      Slow Cookers
 sheffield home bath set          Bath Rugs & Mats                          Vanities
      luau string lights             String Lights                  Holiday Lighting
             rustic twig     Faux Plants and Trees             Wall & Accent Mirrors
```

The predicted class usually agrees with the exact products. For 63 of the 80 queries, most exact products carry it, and for 6 queries, none does, and 1 query has no class at all. Read each row the example prints before counting it as an error. `foutains with brick look` is classed as `Wallpaper`, while its exact products are fountains, so the classifier went wrong. `7qt slow cooker` is a naming mismatch instead: the query classes and the product classes use different names for the same thing.

**Working rule:** Read each query as an entity plus modifiers, and check that your ranker gets the entity right before you tune it for modifiers. Use `query_class` as a hint, for a feature or a filter you check against the judgments, and never as a label. When intent is ambiguous, as with `cooktop`, judge against the guidelines' reading and write that reading down.

## Grade relevance instead of yes or no {#graded-relevance}

A binary label says relevant or not. A graded label, such as WANDS's exact, partial or irrelevant, says how relevant. Results differ in how useful they are, so the most relevant ones should be found and ranked first. [Järvelin and Kekäläinen](https://www.fi.muni.cz/~xkohout7/Research/clanky_cizi/evaluation/p422-jarvelin.pdf) made the case for grades in 2002. In a store, a near miss is often a fair second choice. A shower curtain that isn't clear, which WANDS grades partial because it is the right kind of product without the attribute asked for ([Read intent as a product and its modifiers](#intent-in-product-search)), is closer to what the shopper wanted than a shower curtain hook. Binary labels force one line through the grades. Count only exact as relevant, and the near miss is worth as little as the hook. Count exact and partial, and it is worth as much as the clear curtain.

The store's query `clear shower curtain` shows both failures. Build three top-10 rankings from its judgments: ten exact products, ten shower curtains judged partial, and ten shower curtain hooks judged irrelevant. Then score each with nDCG@10 ([Score one ranking](#score-one-ranking)) under three labelings: binary with exact as relevant, binary with exact or partial as relevant, and the book's grades, 2 for exact, 1 for partial and 0 for irrelevant, used as gains. A **gain** is what a result at a given position adds to the score before the position discount.

Example `binary-labels-hide-partial` (Python 3.12.15 · numpy 2.5.5 · pandas 3.1.1), passing in Edition 2026.11.10:

```python
"""Scores three rankings of one query's judged products under three labelings, to show what each labeling counts as
relevant. The prose's point rests on the `grades` lines, since only the grades separate the three rankings."""
from lib.metrics import ndcg_at_k
from lib.store import load_products, qrels

classes = load_products()["product_class"]
judgments = qrels()[246]                                     # clear shower curtain: {product_id: grade}


def first_ten(grade, product_class=None):
    """Ten products judged with this grade for the query, in product_id order."""
    # sorted(judgments) takes product_id order as the tie-break; None for product_class means any class
    return [product_id for product_id in sorted(judgments) if judgments[product_id] == grade
            and product_class in (None, classes[product_id])][:10]


rankings = {"clear curtains": first_ten(2),                                   # exact
            "other curtains": first_ten(1, "Shower Curtains"),               # partial: not clear
            "hooks": first_ten(0, "Shower Curtain Hooks and Accessories")}   # irrelevant
labelings = {"exact only": {2: 1, 1: 0, 0: 0},              # binary: relevant means exact
             "exact or partial": {2: 1, 1: 1, 0: 0},        # binary: relevant means exact or partial
             "grades": {2: 2, 1: 1, 0: 0}}                  # graded: the book's grades as gains

print("exact products for this query:", sum(grade == 2 for grade in judgments.values()))
# the ideal top 10 holds the ten highest grades, so counting the 2s counts the exact products in it
print("exact products in the ideal top 10:", sorted(judgments.values(), reverse=True)[:10].count(2))
for labeling, gain in labelings.items():
    gains = {product_id: gain[grade] for product_id, grade in judgments.items()}
    for name, ranking in rankings.items():
        print(f"{name}, {labeling}: {ndcg_at_k(ranking, gains, 10):.3f}")
```

Output:

```text
exact products for this query: 12
exact products in the ideal top 10: 10
clear curtains, exact only: 1.000
other curtains, exact only: 0.000
hooks, exact only: 0.000
clear curtains, exact or partial: 1.000
other curtains, exact or partial: 1.000
hooks, exact or partial: 0.000
clear curtains, grades: 1.000
other curtains, grades: 0.500
hooks, grades: 0.000
```

Only the grades separate the three rankings. With exact-only labels, the other curtains score the same as the hooks, 0.000 and 0.000. With exact-or-partial labels they score 1.000, the same as the clear curtains at 1.000. With grades, the clear curtains score 1.000, the other curtains 0.500 and the hooks 0.000. When the ideal top 10 is all exact, as it is here with 10 exact products in it, a ranking of ten partial products scores the ratio of the partial gain to the exact gain.

**Working rule:** We recommend graded judgments for product search, so a metric can credit a near miss without mistaking it for a match. Collect grades even if you report a binary metric too: you can always collapse grades to yes or no, but not the reverse. Use binary labels when a near miss is worthless to the user, as when the query is a model number and only that model will do.

### Choose the gains on purpose {#gains-are-a-choice}

Grades are labels. Gains are what each label is worth in a metric, and that is a choice. This book uses each grade as its gain, so partial is worth half of exact. WANDS's own paper weights exact, partial and irrelevant as 1.0, 0.5 and 0 ([Chen et al., 2022, Table 3](https://easychair.org/publications/preprint/wm4R)), the same ratio and so the same nDCG. The Shopping Queries Dataset, known as ESCI ([Reddy et al., 2022](https://arxiv.org/abs/2206.06588)), has 2.6 million manually labeled query-product judgments over 130 thousand unique queries, and grades differently. It separates a *substitute*, an item that works as a functional substitute, such as fleece for a sweater query, from a *complement*, an item that can be used with an exact one. Its gains are 1.0 for exact, 0.1 for substitute, 0.01 for complement and 0.0 for irrelevant ([Reddy et al., 2022, section 3.1](https://arxiv.org/abs/2206.06588)). WANDS's partial isn't the same as ESCI's substitute, but it is the nearest equivalent.

Score the store's two test runs from chapter 1, [BM25](#rank-with-bm25)'s and semantic retrieval's rankings for every test query, under each of these schemes: the two binary labelings from [Grade relevance instead of yes or no](#graded-relevance), the book's grades, and a partial gain of 0.1 in the spirit of ESCI. The example also runs chapter 3's [permutation test](#permutation-test-on-queries) and [paired t-test](#paired-t-test-on-queries) on semantic retrieval's lead under each scheme, and prints both p-values. Both leads are [descriptive](#when-a-difference-is-established), like every test here. The semantic run reads the recorded embeddings of the open model `Snowflake/snowflake-arctic-embed-m-v1.5` ([Search by meaning](#search-by-meaning)).

Example `gain-choice-changes-ndcg` (Python 3.12.15 · bm25s 0.3.16 · numpy 2.5.5 · pandas 3.1.1 · scipy 1.18.1), passing in Edition 2026.11.10:

```python
"""Scores BM25 and semantic retrieval on the test queries under four gain schemes, with exploratory paired tests on
semantic retrieval's lead. The prose's point rests on the `semantic minus bm25` lines, which keep one sign across the
schemes while their size changes."""
from lib.lexical import LexicalSearch
from lib.metrics import evaluate
from lib.semantic import SemanticSearch
from lib.significance import compare
from lib.store import load_products, load_queries, qrels

queries, judgments = load_queries("test"), qrels("test")
# each run keeps the top 100 products per query; the scores below read only the top 10 of each
runs = {"bm25": LexicalSearch(load_products()).run(queries, k=100), "semantic": SemanticSearch().run(queries, k=100)}

for name, run in runs.items():
    # an unjudged product gives None here, which the count of 1 does not include
    grades = [judgments[query_id].get(product_id) for query_id, ranking in run.items() for product_id in ranking[:10]]
    print(f"partial share of the {name} top 10: {grades.count(1) / len(grades):.3f}")

# each scheme turns a grade into the gain it is worth
schemes = {"exact only": {2: 1, 1: 0, 0: 0},
           "exact or partial": {2: 1, 1: 1, 0: 0},
           "grades 2, 1, 0": {2: 2, 1: 1, 0: 0},       # the book's choice; WANDS's 1, 0.5, 0 gives the same nDCG
           "partial gain 0.1": {2: 1, 1: 0.1, 0: 0}}   # close to the gains ESCI chose for its substitutes
leads = 0
for scheme, gain in schemes.items():
    gains = {query_id: {product_id: gain[grade] for product_id, grade in per_query.items()}
             for query_id, per_query in judgments.items()}
    scores = {name: evaluate(run, gains)["ndcg@10"] for name, run in runs.items()}
    test = compare(scores["semantic"], scores["bm25"])  # exploratory paired tests (chapter 3): no planned comparisons
    print(f"bm25, {scheme}: {scores['bm25'].mean():.3f}")
    print(f"semantic, {scheme}: {scores['semantic'].mean():.3f}")
    print(f"semantic minus bm25, {scheme}: {test['mean difference']:.3f}")
    print(f"permutation p, {scheme}: {test['permutation p']:.3f}")
    print(f"t-test p, {scheme}: {test['t-test p']:.3f}")
    print(f"tied queries, {scheme}: {test['ties']}")
    # a lead is a positive mean difference, semantic minus BM25
    leads += test["mean difference"] > 0
print(f"schemes where semantic leads: {leads} of {len(schemes)}")
```

Output:

```text
partial share of the bm25 top 10: 0.403
partial share of the semantic top 10: 0.380
bm25, exact only: 0.519
semantic, exact only: 0.648
semantic minus bm25, exact only: 0.130
permutation p, exact only: 0.000
t-test p, exact only: 0.002
tied queries, exact only: 3
bm25, exact or partial: 0.903
semantic, exact or partial: 0.983
semantic minus bm25, exact or partial: 0.080
permutation p, exact or partial: 0.010
t-test p, exact or partial: 0.026
tied queries, exact or partial: 19
bm25, grades 2, 1, 0: 0.725
semantic, grades 2, 1, 0: 0.830
semantic minus bm25, grades 2, 1, 0: 0.105
permutation p, grades 2, 1, 0: 0.001
t-test p, grades 2, 1, 0: 0.003
tied queries, grades 2, 1, 0: 2
bm25, partial gain 0.1: 0.564
semantic, partial gain 0.1: 0.688
semantic minus bm25, partial gain 0.1: 0.125
permutation p, partial gain 0.1: 0.000
t-test p, partial gain 0.1: 0.002
tied queries, partial gain 0.1: 2
schemes where semantic leads: 4 of 4
```

Semantic retrieval leads under 4 of 4 schemes, but the size of its lead depends on the choice. Partial products fill 0.403 of BM25's top-10 places and 0.380 of semantic retrieval's, so the partial gain moves both scores a long way. BM25 scores 0.519 with exact-only labels and 0.903 with exact-or-partial labels. With the ESCI-like partial gain of 0.1, BM25 scores 0.564 and semantic retrieval 0.688. The lead is 0.130 with exact-only labels, 0.125 with ESCI-like gains, 0.105 with the grades and 0.080 with exact-or-partial labels. The example doesn't test whether those leads differ, so on 30 test queries, treat the differences between them as possibly noise.

Exact-or-partial labels also saturate, separating rankers on fewer queries. 19 test queries tie under them, against 2 under the grades.

**Working rule:** Use linear gains, the grades 2, 1 and 0, as a default for product search. Choose a lower partial gain when a near miss is worth little to your shoppers. Whatever you choose, fix it before you compare rankers, report it with every score, and never compare nDCG computed with different gains.

## Read the label mix of each query {#label-mix-per-query}

Gains set what each grade is worth, but not how many grades of each kind a query has. A judgment set isn't one population: each query has its own pool of judged products and its own mix of grades, and the mix decides what a perfect ranking looks like for that query. A broad query, one that names a category many products satisfy, has many exact products. A specific query, one with several modifiers, may have only a handful. An average over queries mixes these different problems, so read the mix before you average anything.

The store's queries aren't a random draw from WANDS. Each was chosen to have 60 to 300 judged products, at least 3 exact and 20 partial judgments, and a predicted class no other chosen query has ([data card](https://github.com/rollingedition/book-primer-on-search-relevance/blob/67880c1fa437fcb912ca77d75451f3169c432ea6/examples/data/wands/README.md)). The counts describe the sample, not WANDS as a whole, because queries outside the limits never reach the store. Count the labels for each of the 80 queries, and group the queries by their share of exact products:

Example `count-labels-per-query` (Python 3.12.15 · numpy 2.5.5 · pandas 3.1.1), passing in Edition 2026.11.10:

```python
"""Counts each query's judgments by label, and groups the queries by their share of exact products. The prose's point
rests on the `queries with an exact share of 0.1 or less` line and on the histogram below it."""
import pandas as pd

from lib.store import load_labels, load_queries

queries, labels = load_queries(), load_labels()
counts = pd.crosstab(labels["query_id"], labels["label"])[["exact", "partial", "irrelevant"]]
shares = counts.div(counts.sum(axis=1), axis=0).join(queries[["query", "split"]])

print(f"partial share of all judgments: {(labels['label'] == 'partial').mean():.3f}")
print(f"median exact share: {shares['exact'].median():.3f}")
print(f"median irrelevant share: {shares['irrelevant'].median():.3f}")
for label, which in [("exact", "lowest"), ("exact", "highest"), ("irrelevant", "highest")]:
    query_id = shares[label].idxmin() if which == "lowest" else shares[label].idxmax()
    print(f"{which} {label} share: {shares.at[query_id, label]:.3f}")
    print(f"query with the {which} {label} share: {shares.at[query_id, 'query']}")
print("queries with no irrelevant judgment:", (counts["irrelevant"] == 0).sum())
print("fewest exact products for a query:", counts["exact"].min())
print("most exact products for a query:", counts["exact"].max())
# 10 is the size of a top 10: a query with fewer exact products can't fill a perfect top 10 with exact ones
few = counts["exact"] < 10
print("queries with fewer than 10 exact products:", few.sum())
print("test queries with fewer than 10 exact products:", (few & (shares["split"] == "test")).sum())

print("queries with an exact share of 0.1 or less:", (shares["exact"] <= 0.1).sum())
print()
print("exact share  queries")
bins = pd.cut(shares["exact"], [tenth / 10 for tenth in range(11)])     # no share is 0: each query has 3 or more exact
for interval, number in bins.value_counts(sort=False).items():
    print(f"{interval.left:.1f} to {interval.right:.1f}  {number:>3}  {'#' * number}".rstrip())
```

Output:

```text
partial share of all judgments: 0.610
median exact share: 0.207
median irrelevant share: 0.060
lowest exact share: 0.014
query with the lowest exact share: 48 in entry table with side by side drawer
highest exact share: 0.749
query with the highest exact share: outdoor light fixtures
highest irrelevant share: 0.776
query with the highest irrelevant share: podium with locking cabinet
queries with no irrelevant judgment: 9
fewest exact products for a query: 3
most exact products for a query: 193
queries with fewer than 10 exact products: 23
test queries with fewer than 10 exact products: 7
queries with an exact share of 0.1 or less: 29

exact share  queries
0.0 to 0.1   29  #############################
0.1 to 0.2   10  ##########
0.2 to 0.3   12  ############
0.3 to 0.4   11  ###########
0.4 to 0.5    9  #########
0.5 to 0.6    1  #
0.6 to 0.7    7  #######
0.7 to 0.8    1  #
0.8 to 0.9    0
0.9 to 1.0    0
```

Partial is the commonest label, at 0.610 of all judgments. The likely reason is how WANDS gathered its candidates, which [Know where the judgments came from](#pooled-judgments) covers. The exact share varies widely from query to query, from 0.014 for `48 in entry table with side by side drawer` to 0.749 for `outdoor light fixtures`, with a median of 0.207. The histogram the example prints is tallest at the low end: 29 queries have an exact share of 0.1 or less.

Irrelevant judgments are rare for most queries, not all: the irrelevant share has a median of 0.060, yet it reaches 0.776 for `podium with locking cabinet`, and 9 queries have no irrelevant judgment at all.

The spread changes what [nDCG@10](#score-one-ranking) measures. Exact products per query run from 3, the sampling rule's floor, to 193, and 23 queries have fewer than 10 exact products, 7 of them test queries. For those, the ideal top 10 includes partial products, so a ranking can score perfectly with partial products in its top 10. For a broad query such as `outdoor light fixtures`, a perfect score needs ten exact products. The same nDCG@10 means a different thing for each.

**Working rule:** Print the label mix per query before you average any metric, and look at both ends. Read a low score on a query with few exact products as a different problem from a low score on a broad query. Keep per-query scores next to every mean; [chapter 3](#read-per-query-differences) shows what averages hide.

## Treat clicks as biased evidence {#clicks-are-not-judgments}

A store logs every click, and clicks cost nothing to collect, so they look like free judgments. They aren't. A click says that a shopper chose to look at a product. It doesn't say the product met the need, and whether a shopper clicks at all depends on where and how the product was shown.

The evidence on how clicks mislead comes from web search. Joachims et al. tracked the eyes and clicks of Cornell undergraduates answering ten set questions with Google, in two phases, with 34 students in the first phase and 22 in the second. In the second phase some saw the top two results swapped, or the whole list reversed ([Joachims et al., 2005](https://www.cs.cornell.edu/people/tj/publications/joachims_etal_05a.pdf)). They concluded that clicks are "informative but biased."

The first bias is **position bias**: people click higher-ranked results more, partly because they trust the ranking. Joachims et al. call the effect a "trust bias," which draws clicks to highly ranked links even when other abstracts the user viewed are more relevant. The authors also found that relative preferences, such as a clicked result over a skipped one above it, are reasonably accurate on average.

In a large experiment on a web search engine, Craswell et al. found that a **cascade model** explained position bias best in the top ranks. In the cascade model, users read results from top to bottom and leave at the first worthwhile document ([Craswell et al., 2008](https://www.microsoft.com/en-us/research/?p=156688)).

**Presentation bias** goes beyond position: clicks depend on how a result is shown, not only on where. On a portion of Google's web search traffic, results with more attractive titles, meaning more of the query's words in bold, drew more clicks, even after controlling for position and for relevance as people judged it ([Yue et al., 2010](https://archives.iw3c2.org/www2010/proceedings/www/p1011.pdf)). In a store, images and prices are the nearest analogue to titles. That is our inference, not a measured result.

WANDS has no click data, so the store can't measure these biases. You can still see what a click log would record. Take the store's judgments for `clear shower curtain`, and apply the cascade model in its strictest form: a shopper reads from the top, clicks the first exact product and leaves. This thought experiment uses one fixed rule, not a model of real shoppers. [Chapter 11](#simulate-interleaving-with-clicks) simulates clicks with probabilities. Run the thought experiment on BM25's ranking and on semantic retrieval's, and read the two click logs side by side:

Example `cascade-clicks` (Python 3.12.15 · bm25s 0.3.16 · numpy 2.5.5 · pandas 3.1.1 · scipy 1.18.1), passing in Edition 2026.11.10:

```python
"""Reads the top 10 of BM25 and of semantic retrieval for `clear shower curtain` as a cascade of clicks, which stops at
the first exact product. The prose's point rests on the `clicked rank` lines and on the `exact products shown below the
click` lines."""
from lib.lexical import LexicalSearch
from lib.semantic import SemanticSearch
from lib.store import load_products, qrels

products = load_products()
judgments = qrels()[246]                                     # clear shower curtain: {product_id: grade}
# search returns scores indexed by product_id, so .index is the ranking; semantic search takes the query's id, 246
rankings = {"bm25": list(LexicalSearch(products).search("clear shower curtain", k=10).index),
            "semantic": list(SemanticSearch().search(246, k=10).index)}

for name, ranking in rankings.items():
    grades = [judgments.get(product_id, 0) for product_id in ranking]     # an unjudged product counts as grade 0
    rank = grades.index(2) + 1             # cascade: read from the top, click the first exact product, stop
    print(f"{name} grades, ranks 1 to 10: {' '.join(map(str, grades))}")
    print(f"{name} clicked rank: {rank}")
    # rank counts from 1 and the list from 0, so the clicked product sits at index rank - 1
    print(f"{name} clicked product: {products.at[ranking[rank - 1], 'name']}")
    # grades[rank:] starts just after the click, since rank is the clicked product's position counted from 1
    print(f"{name} exact products shown below the click: {grades[rank:].count(2)}")
```

Output:

```text
bm25 grades, ranks 1 to 10: 1 2 2 2 2 1 1 0 0 0
bm25 clicked rank: 2
bm25 clicked product: woolley vinyl single shower curtain
bm25 exact products shown below the click: 3
semantic grades, ranks 1 to 10: 2 2 1 1 2 1 2 2 1 1
semantic clicked rank: 1
semantic clicked product: woolley vinyl solid color single shower curtain
semantic exact products shown below the click: 4
```

A click log records what the ranker showed first as much as what the shopper wanted. Under [BM25](#rank-with-bm25)'s ranking, the shopper clicks rank 2, `woolley vinyl single shower curtain`, and never reaches the 3 exact products below it. In the click log, those exact products look the same as BM25's ranks 8 to 10, shower curtain hooks judged irrelevant ([Rank products with BM25](#rank-with-bm25)), which nobody clicked either.

Under [semantic retrieval](#search-by-meaning)'s ranking, the click lands at rank 1, on `woolley vinyl solid color single shower curtain`. Two rankings from the same catalog produce two click logs that credit different products.

WANDS's authors give the same reason, position bias, for not building their dataset from logs ([Check the results against the judgments](#check-against-judgments)). Clicks still carry information, though. In WANDS, 62.47% of the candidates taken from click logs were judged exact, against 30.95% of those taken from open-source ranking systems ([Chen et al., 2022, Table 4](https://easychair.org/publications/preprint/wm4R)).

Correcting clicks for bias is a field of its own, unbiased learning to rank. On a large search dataset released by Baidu, those methods improved click prediction but "struggle to consistently improve ranking performance" as judged by experts ([Hager et al., 2024](https://arxiv.org/abs/2404.02543)).

**Working rule:** Use judgments, not clicks, as the reference for offline evaluation. Use clicks as relative evidence, a clicked product over a skipped one ranked above it, and never read a missing click as a sign that a product is irrelevant. If you train on clicks, check the result against judgments.

## Know where the judgments came from {#pooled-judgments}

Nobody judges every product for every query, so test collections judge a **pool** instead, a set of candidates gathered for each query. In TREC, the Text REtrieval Conference's shared evaluations, a topic's pool is the union of the top-ranked documents each participant retrieved, usually the top 100, and every document outside the pool counts as not relevant. Voorhees argues that systems can still be compared fairly when every unjudged document counts as not relevant, because pools hold enough of the relevant documents. The same article notes that pooling can't produce reusable test collections for arbitrarily large document sets ([Voorhees, 2007](https://cs.uwaterloo.ca/~brecht/courses/854-Experimental-Performance-Evaluation-2017/readings/trec-cacm-2007.pdf)). Zobel tested the fairness claim. With pooled judgments, the relative performance of systems looks reliable, but recall is overestimated, since many relevant documents have probably never been found ([Zobel, 1998](https://www.cs.cornell.edu/courses/cs574/2002fa/assignments.org/zobel-sigir98.pdf)).

WANDS built its pool in two steps ([Chen et al., 2022, sections 4.2 and 4.3](https://easychair.org/publications/preprint/wm4R)). The first took the products shoppers had clicked or added to their carts, plus the results of an open-source lexical search engine, Solr, and of a neural retrieval system. The second mined the whole catalog again with the same two systems, and added every product in the class that a query classification model predicted for the query. That second step produced most of WANDS's labels, and mostly near misses: 4.12% of its labels were exact and 63.99% partial ([Table 5](https://easychair.org/publications/preprint/wm4R)). Partial is the commonest label in the store's sample too ([Read the label mix of each query](#label-mix-per-query)). The authors argue that completeness matters for product search, because "incompleteness in Product Search dataset does negatively impact the discriminative power of the evaluation" ([section 3.1](https://easychair.org/publications/preprint/wm4R)).

A pool belongs to a query. A product without a judgment for a query is **unjudged** for it, and a product judged for one query can be unjudged for a similar one. Two of the store's queries show this: the train query `trundle daybed` and the test query `full bed with trundle`. Count the products judged exact for the first that have no judgment for the second, and see how the judges graded the ones that do:

Example `trundle-daybeds-unjudged` (Python 3.12.15 · numpy 2.5.5 · pandas 3.1.1), passing in Edition 2026.11.10:

```python
"""Counts the products judged exact for the train query `trundle daybed` that have no judgment for the test query `full
bed with trundle`, and grades the ones that do. The prose's point rests on the `of them unjudged for full bed with
trundle` line and the `judged partial` line."""
from collections import Counter

from lib.store import load_products, load_queries, qrels

products, queries, judgments = load_products(), load_queries(), qrels()
LABELS = {2: "exact", 1: "partial", 0: "irrelevant"}

# 433 is the train query trundle daybed, and its exact products come out in product_id order
daybeds = sorted(product_id for product_id, grade in judgments[433].items() if grade == 2)
full_bed = judgments[436]                                    # {product_id: grade} for "full bed with trundle"
print(f"433: {queries.at[433, 'query']} ({queries.at[433, 'split']})")
print(f"436: {queries.at[436, 'query']} ({queries.at[436, 'split']})")
print("exact for trundle daybed:", len(daybeds))

unjudged = [product_id for product_id in daybeds if product_id not in full_bed]
print("of them unjudged for full bed with trundle:", len(unjudged))
print("of them judged for full bed with trundle:", len(daybeds) - len(unjudged))
judged = Counter(LABELS[full_bed[product_id]] for product_id in daybeds if product_id in full_bed)
for label in LABELS.values():
    print(f"judged {label}: {judged[label]}")

# daybeds is in product_id order, so the first two unjudged are the two lowest product_ids among them
print("first unjudged daybed:", products.at[unjudged[0], "name"])
print("second unjudged daybed:", products.at[unjudged[1], "name"])
```

Output:

```text
433: trundle daybed (train)
436: full bed with trundle (test)
exact for trundle daybed: 113
of them unjudged for full bed with trundle: 77
of them judged for full bed with trundle: 36
judged exact: 5
judged partial: 31
judged irrelevant: 0
first unjudged daybed: daybed with trundle
second unjudged daybed: pattonsburg daybed with trundle
```

A metric that scores an unjudged product as 0 is guessing about it, and here the guess matters. By our reading of the guidelines' rule, a daybed with a trundle is often the right kind of bed without the size the query asks for. Of the 113 products judged exact for `trundle daybed`, 77 have no judgment for `full bed with trundle`, among them the two with the lowest product numbers, `daybed with trundle` and `pattonsburg daybed with trundle`. Of the 36 that do, the judges called 5 exact, 31 partial and 0 irrelevant. If the unjudged daybeds resemble the judged ones, most of them would be partial too, yet a metric gives each one grade 0.

Zobel warns that pooled judgments can underestimate systems that didn't contribute to the pool ([Zobel, 1998](https://www.cs.cornell.edu/courses/cs574/2002fa/assignments.org/zobel-sigir98.pdf)). That makes judgments fairest to rankers like the systems that built the pool: for WANDS, shoppers' clicks, a lexical engine, a neural retriever and a query classifier. It is a tendency, not a law. Here BM25, the ranker most like WANDS's lexical engine, has the lower judged@10 of the two rankers ([Count unjudged products](#unjudged-results)). Our guess is that the store's BM25 reads different fields than the pool's engine did, so it ranks different products high.

**Working rule:** Find out where your pool came from before you trust a comparison. When you evaluate a ranker unlike the systems that built it, expect it to surface products nobody judged, and plan to judge them.

## Count unjudged products before you compare scores {#unjudged-results}

A product nobody judged for a query is [unjudged](#pooled-judgments): its grade is unknown, not 0. Standard metrics can't tell the difference. They score an unjudged product exactly as they score one judged irrelevant, and Buckley and Voorhees warn that such metrics become unreliable when judgments are very incomplete ([Buckley and Voorhees, 2004](https://tsapps.nist.gov/publication/get_pdf.cfm?pub_id=150469)). Two diagnostics are standard in evaluation tools ([ir-measures](https://ir-measur.es/en/latest/measures.html)). **Judged@k** is the share of the top k that has a judgment of any grade:

```text
judged@k = (number of the top k products that have a judgment for the query) / k
```

The second diagnostic is the **condensed** list, a ranking with its unjudged products dropped before scoring, so the judged ones move up. ir-measures offers it as the `judged_only` option. Buckley and Voorhees's bpref measure works in the same spirit: it uses judged documents only, and they found it more robust to incomplete judgments.

In [chapter 1](#score-every-test-query), [BM25](#rank-with-bm25)'s mean judged@10 on the test queries is 0.927. The mean for [semantic retrieval](#search-by-meaning) is 0.990. One property of the store's data keeps those numbers high. Its catalog is almost all judged: 11,917 of 12,000 products have a judgment for at least one query ([Load the store's data](#store-data)). Expect a full catalog to put more unjudged products in the top 10s.

Build judged@10 by hand, check it against `lib.metrics`, and list where BM25's unjudged products are:

Example `unjudged-in-bm25-top-10` (Python 3.12.15 · bm25s 0.3.16 · numpy 2.5.5 · pandas 3.1.1 · scipy 1.18.1), passing in Edition 2026.11.10:

```python
"""Builds judged@10 by hand for BM25's test rankings, checks it against lib.metrics, and lists where BM25's unjudged
products sit in the top 10s. The prose's point rests on the `unjudged products in the top 10s` line and on the
`irrelevant judgments` line for pull out sleeper loveseat."""
import pandas as pd

from lib.lexical import LexicalSearch
from lib.metrics import evaluate
from lib.store import load_products, load_queries, qrels

products, queries, judgments = load_products(), load_queries("test"), qrels("test")
# 100 products per query are kept; this example reads only the top 10 of each
run = LexicalSearch(products).run(queries, k=100)


def judged_at_k(ranking, judged, k):
    """The share of the top k that has a judgment of any grade."""
    return sum(product_id in judged for product_id in ranking[:k]) / k


rows = []
for query_id, ranking in run.items():
    unjudged = [product_id for product_id in ranking[:10] if product_id not in judgments[query_id]]
    # the name is cut to 45 characters so that the table fits on the page
    rows.append({"query": queries.at[query_id, "query"], "judged@10": judged_at_k(ranking, judgments[query_id], 10),
                 "unjudged": len(unjudged), "first unjudged product": products.at[unjudged[0], "name"][:45] if unjudged else ""})
table = pd.DataFrame(rows)

print(f"mean judged@10: {table['judged@10'].mean():.3f}")
# lib.metrics' judged@10 per query, indexed by query_id
library = evaluate(run, judgments)["judged@10"]
# 1e-12 allows for floating-point rounding when the two judged@10 values are compared
agree = sum(abs(row["judged@10"] - library[query_id]) < 1e-12 for query_id, row in zip(run, rows))
print(f"queries where the hand-built judged@10 equals lib.metrics: {agree} of {len(rows)}")
print("unjudged products in the top 10s:", table["unjudged"].sum())
print("test queries with an unjudged product in the top 10:", (table["judged@10"] < 1).sum())
sleeper = list(judgments[244].values())                      # pull out sleeper loveseat
print("judgments for pull out sleeper loveseat:", len(sleeper))
print("irrelevant judgments for pull out sleeper loveseat:", sleeper.count(0))  # grade 0 is irrelevant
print()
# judged@10 below 1 means the query has at least one unjudged product in its top 10
print(table[table["judged@10"] < 1].sort_values("judged@10").to_string(index=False))
```

Output:

```text
mean judged@10: 0.927
queries where the hand-built judged@10 equals lib.metrics: 30 of 30
unjudged products in the top 10s: 22
test queries with an unjudged product in the top 10: 7
judgments for pull out sleeper loveseat: 143
irrelevant judgments for pull out sleeper loveseat: 0

                    query  judged@10  unjudged                        first unjudged product
   outdoor light fixtures        0.2         8  60 watt equivalent e12 led candle light bulb
     beach blue headboard        0.5         5 lemon frosting indoor/outdoor , removable cov
pull out sleeper loveseat        0.6         4    country oxford 87.75 '' rolled arm sleeper
    full bed with trundle        0.8         2 abel staircase twin over full bunk bed with t
   whimsical solar lights        0.9         1 luciole 0.12-watt led solar power dusk to daw
  bistro table and chairs        0.9         1              drayton patio 5 piece bistro set
     wooden chair outdoor        0.9         1 osterley 31.5 '' w writing desk and chair set
```

The hand-built judged@10 equals `lib.metrics` on 30 of 30 test queries, with a mean of 0.927. Unjudged products fill 22 of BM25's top-10 places, spread over 7 queries. By our reading, some are plainly irrelevant: light bulbs for `outdoor light fixtures`, an outdoor pillow for `beach blue headboard`. Others look like near misses or matches: a bunk bed with a trundle for `full bed with trundle`, a bistro set for `bistro table and chairs`, a sleeper sofa for `pull out sleeper loveseat`, which by the entity-and-modifier rule is probably a near miss: the judges called 0 of that query's 143 judged products irrelevant.

Score both rankers with condensed lists, which drop the unjudged products, to see how far the scores move:

Example `condensed-ndcg` (Python 3.12.15 · bm25s 0.3.16 · numpy 2.5.5 · pandas 3.1.1 · scipy 1.18.1), passing in Edition 2026.11.10:

```python
"""Scores BM25 and semantic retrieval on the test queries with standard and condensed nDCG@10, and runs exploratory
tests on the change in the lead. The prose's point rests on the `semantic minus bm25` lines and the `share of bm25's
gain` line."""
from lib.lexical import LexicalSearch
from lib.metrics import evaluate
from lib.semantic import SemanticSearch
from lib.significance import compare
from lib.store import load_products, load_queries, qrels

queries, judgments = load_queries("test"), qrels("test")
runs = {"bm25": LexicalSearch(load_products()).run(queries, k=100), "semantic": SemanticSearch().run(queries, k=100)}

standard, condensed = {}, {}
for name, run in runs.items():
    # condensed: drop every unjudged product, so the judged ones move up into the top 10
    judged_only = {query_id: [product_id for product_id in ranking if product_id in judgments[query_id]]
                   for query_id, ranking in run.items()}
    standard[name] = evaluate(run, judgments)["ndcg@10"]
    condensed[name] = evaluate(judged_only, judgments)["ndcg@10"]
    print(f"{name} ndcg@10: {standard[name].mean():.3f}")
    print(f"{name} condensed ndcg@10: {condensed[name].mean():.3f}")

gain = condensed["bm25"] - standard["bm25"]                   # bm25's change, per test query
# the scores are indexed by query_id: 133 is outdoor light fixtures, and 46 is beach blue headboard
print(f"bm25 ndcg@10 for outdoor light fixtures: {standard['bm25'][133]:.3f}")
print(f"bm25 condensed ndcg@10 for outdoor light fixtures: {condensed['bm25'][133]:.3f}")
print(f"share of bm25's gain from outdoor light fixtures and beach blue headboard: {gain[[133, 46]].sum() / gain.sum():.3f}")

lead, condensed_lead = standard["semantic"] - standard["bm25"], condensed["semantic"] - condensed["bm25"]
change = compare(condensed_lead, lead)                       # is the change in the lead more than noise? (exploratory)
print(f"semantic minus bm25: {lead.mean():.3f}")
print(f"semantic minus bm25, condensed: {condensed_lead.mean():.3f}")
print(f"change in the lead: {change['mean difference']:.3f}")
print(f"95% interval for the change: {change['ci low']:.3f} to {change['ci high']:.3f}")
print(f"t-test p for the change: {change['t-test p']:.3f}")
print(f"permutation p for the change: {change['permutation p']:.3f}")
condensed_test = compare(condensed["semantic"], condensed["bm25"])   # the condensed lead itself (exploratory)
print(f"t-test p for the condensed lead: {condensed_test['t-test p']:.3f}")
print(f"permutation p for the condensed lead: {condensed_test['permutation p']:.3f}")
```

Output:

```text
bm25 ndcg@10: 0.725
bm25 condensed ndcg@10: 0.778
semantic ndcg@10: 0.830
semantic condensed ndcg@10: 0.834
bm25 ndcg@10 for outdoor light fixtures: 0.286
bm25 condensed ndcg@10 for outdoor light fixtures: 1.000
share of bm25's gain from outdoor light fixtures and beach blue headboard: 0.612
semantic minus bm25: 0.105
semantic minus bm25, condensed: 0.056
change in the lead: -0.049
95% interval for the change: -0.138 to -0.015
t-test p for the change: 0.073
permutation p for the change: 0.036
t-test p for the condensed lead: 0.011
permutation p for the condensed lead: 0.009
```

Condensing lifts BM25 mostly by forgiving bad results, not by undoing a penalty on good ones. BM25 goes from 0.725 to 0.778, and semantic retrieval from 0.830 to 0.834. Of BM25's gain, a share of 0.612 comes from `outdoor light fixtures` and `beach blue headboard`, the two queries whose unjudged products are plainly irrelevant. For `outdoor light fixtures`, BM25 goes from 0.286 to 1.000: once the bulbs drop out, the judged lights below them move up.

Semantic retrieval's lead falls from 0.105 to 0.056 once the lists are condensed. Whether that change is more than noise is unsettled. Chapter 3's [paired t-test](#paired-t-test-on-queries) gives p = 0.073 for the change, and its [permutation test](#permutation-test-on-queries) gives 0.036. The 95% [bootstrap interval](#report-the-size-of-a-gap) runs from -0.138 to -0.015. Both checks are [exploratory](#when-a-difference-is-established). With the change carried by a few queries, treat it as possibly noise.

The condensed lead itself gets p = 0.009 from the permutation test and 0.011 from the t-test. Neither the standard score nor the condensed one is the truth. The standard score punishes unjudged products that may be good, and the condensed score hides unjudged products that are bad.

**Working rule:** Report judged@10 next to every nDCG@10, and compute condensed scores as a diagnostic, never as the headline. When two rankers' judged@10 differ, have the unjudged products in their top 10s judged before you compare them. That is the dependable cure, and on this sample it is a small job: 22 judgments for BM25's test run. On a full catalog, expect more.

## What goes wrong in production {#what-relevance-means-in-production}

**Judgments go stale.** A store's catalog and its shoppers keep changing, and relevance depends on more than the query. Tsagkias et al. name the user, time, query and context, and give an example: a query for jackets has different relevant products in winter than in summer ([Tsagkias et al., 2020, section 3.1.1](https://sigir.org/wp-content/uploads/2020/06/p04.pdf)). Products enter the catalog unjudged, and products that leave it stay in the judgment set. Track [judged@10](#unjudged-results) for your production ranker over time, and judge again when it falls.

**Click labels confirm the ranking you have.** A ranker trained on clicks logged under the current ranking learns that ranking's order along with the shoppers' preferences. Keep human judgments as the reference that clicks are checked against ([Treat clicks as biased evidence](#clicks-are-not-judgments)).

**Guidelines shape the judgments.** WANDS's annotators needed four months to reach near-perfect agreement ([Chen et al., 2022, Table 2](https://easychair.org/publications/preprint/wm4R)). Small choices in the guidelines shape the whole set. WANDS told annotators to ignore misspellings and judge the example as written ([guidelines](https://github.com/wayfair/WANDS/raw/3b74dcf4ba29ab8ff3e6a50b5b09fc627cb882b5/Product%20Search%20Relevance%20Annotation%20Guidelines.pdf)), so its judgments assume a reader who understands `ligth bulb`. A ranker that can't will score low on such queries, which is fair to the shopper. [Chapter 8](#correct-spelling-from-catalog) builds a spelling corrector and shows what it still gets wrong.

**Your ranker sees less than your judges.** The judges saw images and product pages, while the store's rankers read text without the attributes ([chapter 5 tests adding them](#index-attributes-as-a-field)). Any large language model (LLM) asked to judge from the same text sees what the ranker sees. Some disagreement between such a system and the human judgments comes from that gap, not from the system. Check which fields hold the attributes your judges rewarded before you blame the ranker.

**Filling holes with a model still needs human judgments.** Gienapp et al. fine-tune a small classifier for each query on that query's human judgments, and use it to judge the unjudged products. On TREC Robust04 and the TREC Deep Learning collections of 2019 and 2020, they report that as few as 128 initial human judgments per topic improve the comparability of models, and that the approach is more reliable than prompted LLM judges ([Gienapp et al., 2025](https://arxiv.org/abs/2510.04633)). Soboroff's advice on LLMs is blunter: "don't use LLMs to create relevance judgments for TREC-style evaluations" ([Soboroff, 2025](https://arxiv.org/abs/2409.15133)). [Chapter 4](#read-llm-error-direction) measures LLM judgments against the store's human ones before trusting them.

**Working rule:** Date your judgments, track judged@10 in production, and budget for judging again.

## Decisions this chapter made {#what-relevance-means-decisions}

Later chapters keep these unless a measurement on train queries says otherwise.

- **Human judgments are the reference.** Every headline score in this book is computed on them, including scores for rankers built with LLMs. Clicks are evidence about shoppers, biased by position and presentation, and never serve as labels.
- **Grades 2, 1 and 0 are the gains,** so partial is worth half of exact, the same ratio as WANDS's own 1.0, 0.5 and 0. Binary metrics count exact only.
- **One grade per pair.** When WANDS lists a pair more than once with different labels, the sample keeps the lower grade, which errs against the ranker.
- **`query_class` is a hint,** never a label and never a measure of intent. Some of its values are wrong.
- **Unjudged counts as grade 0, with judged@10 beside every nDCG@10.** Condensed scores are a diagnostic only. The cure for unjudged products in a top 10 is to have them judged.
- **Rankers read the product text, not the attributes.** [Chapter 5](#index-attributes-as-a-field) tests adding them as a field and keeps them out of the BM25 text. The judges also saw images and product pages, so some exact products give a text-only ranker little to match.
- **Read any judgment, tune on train.** Reading test judgments changes no ranker; choosing a fix from them would.
- **What the chapter showed:** a score is only as good as its judgments.
