---
title: "Data Governance as Code"
book: data-governance-as-code
edition: 2026.11.31
verified: 2026-11-19
edition_code: VJ1S-X2E7
canonical: https://rollingedition.com/data-governance-as-code
license: CC BY-NC 4.0
more_permissions: https://rollingedition.com/policies/reuse
attribution: "Data Governance as Code, Edition 2026.11.31, examples passed Nov 19, 2026. By Rolling Edition, https://rollingedition.com/data-governance-as-code. Text CC BY-NC 4.0 (https://creativecommons.org/licenses/by-nc/4.0/), with added permissions (https://rollingedition.com/policies/reuse). Code MIT, copyright Rolling Edition and contributors. Written with AI. Verified by people."
---
Edition 2026.11.31. Examples passed Nov 19, 2026. 108 of 108 examples pass. Written with AI. Verified by people.

To cite a section, name Edition 2026.11.31 and link to the section's permanent link: https://rollingedition.com/data-governance-as-code/s/{section_id}

# Governance end to end

By the end of this chapter you can govern one table of a data platform with code, end to end. You'll catalog it with an owning team and a classification for every column, capture the lineage of a pipeline that copies it, decide who may read which of its rows and enforce that decision on a query, and see a pull request fail because it added a column nobody classified. Each piece is small and runs in seconds, and five later chapters each make one of them solid.

This book is for data, analytics and platform engineers who already build pipelines and warehouses and have been asked to make them governed. You know SQL, Python and Parquet or a lakehouse table, and you don't need governance theory: you need governance that runs, is tested, and is reviewed in pull requests like the rest of your code. By the end of the book, you can govern a data platform with code that runs in CI:

- a catalog where every table has an owner and every column a classification;
- personal data found automatically, with what the detector misses measured;
- lineage captured from pipelines;
- access policies as tested code, enforced as row filters and column masks;
- retention and erasure that remove data;
- an audit trail;
- checks that fail a pull request;
- the same rules applied to data that trains or grounds AI.

This book follows one case throughout: an online learning platform that sells professional courses to adult learners, which we call the platform. Its warehouse is a catalog of Iceberg tables, each under the namespace of the team that runs it: learners and instructors under `crm`, courses, enrollments, learning events and course reviews under `learning`, payments under `billing`, and support tickets under `support`. SQL pipelines build reporting tables from them. Support agents, instructors, marketing, finance, data scientists and the platform's course assistant, its AI, each need something different from those tables, and that's what makes the platform worth governing.

The platform's data, which every example in this book reads, is generated. A script makes it once from fixed seeds with Faker, a library for fake data, and the files are committed with a data card that says how they were made. No real person is in the data. Its email addresses are on domains reserved for examples, so they can't reach anyone, and its IP addresses are in ranges reserved for documentation, so they point at no one. Its phone numbers, card numbers and IBANs are random in a valid format, so one may match a number or an account in service: never call, message, charge or pay them. The data card also lists what is planted for the examples to find, such as full card numbers in a legacy column and personal data in free text.

Every example runs offline, in one Python 3.12.15 process, on pinned libraries: PyIceberg 0.12.0 for the Iceberg catalog and its tables, DuckDB 2.0.0 for queries, sqlglot 30.33.0 to parse SQL, openlineage-python 1.55.0 for lineage, and regopy 1.5.2 for policies written in Rego, the policy language of Open Policy Agent (OPA). Six services that a production platform runs don't run here, and each has a stand-in:

- **Trino**, the SQL engine tools query through: DuckDB behind the book's [enforcement layer](#enforcing-policies-on-queries), which rewrites each query before running it.
- **OPA's server**, which Trino asks for access decisions ([Trino 484, OPA access control](https://trino.io/docs/484/security/opa-access-control.html)): regopy, evaluating the same Rego inside the Python process.
- **Spark**, or whatever engine runs the pipelines at scale: DuckDB, running each pipeline's SQL file.
- **A catalog server or a cloud catalog**, serving table metadata through an API such as the Iceberg REST catalog's ([REST catalog spec, 1.12.0](https://github.com/apache/iceberg/blob/apache-iceberg-1.12.0/open-api/rest-catalog-open-api.yaml)): PyIceberg's SQL catalog, backed by a SQLite file that records where each table's metadata is, with the metadata beside the data on local disk.
- **A lineage server**: a file of OpenLineage events, which the book's code reads back into a graph.
- **The CI service and git**: two folders, one for the main branch and one for a pull request's branch.

What you learn here is the open standards those services work with: Iceberg's table metadata, OpenLineage and Rego. What you build maps onto them.

The platform's governance starts with a spreadsheet. Someone listed the columns of `crm.learners`, which holds every learner's name, email, phone number and address, with a classification beside each, and that's all there is. No team is named as the table's owner. A nightly pipeline copies learners' names and emails into a table for marketing, and nothing records that it does. Support agents each serve one region, but a query on the warehouse returns every region's learners. And the CRM team is about to add a column. This chapter answers each of those with the code that the later chapters build, at the smallest scale that works: one table, one pipeline, one rule and one check. Only one of the four is a policy engine; the others are metadata, lineage events and a test, kept in the same repository and reviewed the same way.

[Chapter 2](#a-catalog-you-can-test) keeps the catalog in step with schema changes and measures a local model's drafted metadata. [Chapter 4](#lineage-from-pipelines) uses lineage for impact analysis and to propose classifications. [Chapter 5](#access-policies-as-code) writes the whole policy, with its tests, and [chapter 6](#enforcing-policies-on-queries) enforces it on every query. [Chapter 10](#governance-in-ci) runs the checks of chapters 2 to 7 in CI, with owners' approvals and waivers.

Four chapters add what this one leaves out. [Chapter 3](#finding-personal-data) finds the personal data the metadata misses. [Chapter 7](#retention-consent-erasure) covers retention, consent and erasure. [Chapter 8](#audit-and-evidence) keeps an audit trail, and [chapter 9](#governing-data-for-ai) applies the same rules to data that trains or grounds AI.

The shared modules, files in `examples/lib/` that hold the code many examples need, are yours to open and read, and each section names the ones it uses.

## Catalog a table with its owner and classifications {#catalog-a-table}

The platform's first problem is its spreadsheet, a list of columns that nothing checks. A catalog lists a platform's tables and their schemas, and governance needs two facts the data doesn't carry: who answers for each table, and how each column must be protected. Kept in a spreadsheet or a catalog server's web page, those facts change without review, and no test can say whether they're complete. Kept in a file in the repository, beside the pipelines, every change is a diff someone approves, and a test can compare the file with the catalog.

The platform's file is `governance/metadata.toml`. It gives each table an [**owner**](#keep-metadata-in-a-reviewed-file), a **domain**, the part of the business the table belongs to, and a description, and each column a description and a **classification**. Catalog tools mean different things by an owner, as [chapter 2](#keep-metadata-in-a-reviewed-file) compares, and we use one meaning throughout: the team accountable for the table, never a person, so the answer survives someone leaving. A classification is one of four values, from least to most protected:

- `public`: may be shown to anyone outside the platform;
- `internal`: for the platform's staff, and doesn't identify a person by itself;
- `personal`: identifies a person, or tells something about one;
- `sensitive`: data whose leak would directly harm someone, such as a card number.

Columns are keyed by their Iceberg [field ID](#key-column-metadata-by-field-id), which survives a rename, with the name beside it for people:

```toml
[tables."crm.learners"]
owner = "team-crm"
domain = "crm"
description = "Every learner who has signed up, with their contact details and consents. One row per learner."

[tables."crm.learners".columns]
3 = { name = "email", classification = "personal", description = "The email the learner signs in with." }
```

Three of the book's shared modules do the work. `lib.catalog.open_catalog` builds a fresh Iceberg catalog of the platform's tables in a temporary folder, which it removes when the example exits. `lib.metadata` loads the file, writes it into the catalog with `apply`, and checks the catalog against it with `problems`, which reports any table with no metadata or no owner, and any column with no classification.

Loading the file and cataloging `crm.learners` with it takes a few lines, and the check runs on what it builds:

Example `catalog-the-learners` (Python 3.12.15 · pyarrow 26.0.1 · pyiceberg 0.12.0), passing in Edition 2026.11.31:

```python
"""Apply governance/metadata.toml to crm.learners: write its owner, domain and descriptions into the catalog, then run the
check on the catalog. The point rests on the last line, the problems the check finds once the file is applied."""
from collections import Counter

from lib import metadata
from lib.catalog import open_catalog

platform = open_catalog(["crm.learners"])            # a fresh Iceberg catalog holding the learners table
described = metadata.load()
learners = described["crm.learners"]
print("owner:", learners.owner)
print("domain:", learners.domain)
counts = Counter(column.classification for column in learners.columns.values())
print("columns classified personal:", counts["personal"])
print("personal columns:", ", ".join(c.name for c in learners.columns.values() if c.classification == "personal"))
print("columns classified internal:", counts["internal"])

# Write the descriptions into the table's schema, where readers see them, and the owner and domain into its properties.
metadata.apply(platform, described)
table = platform.load_table("crm.learners")
print("table properties:", ", ".join(f"{key}={value}" for key, value in sorted(table.properties.items())))
print("email's description, in the Iceberg schema:", table.schema().find_field("email").doc)

# The check: every table in the catalog has an owner, and every column one of the four classifications.
print("problems the check finds:", len(metadata.problems(platform, described)))
```

Output:

```text
owner: team-crm
domain: crm
columns classified personal: 7
personal columns: full_name, email, phone, city, street_address, date_of_birth, notes
columns classified internal: 8
table properties: domain=crm, owner=team-crm
email's description, in the Iceberg schema: The email the learner signs in with.
problems the check finds: 0
```

The file gives `crm.learners` to `team-crm`, in the domain `crm`, and classifies 7 of its columns as personal: `full_name, email, phone, city, street_address, date_of_birth, notes`. `notes` is among them although its name says nothing, because support agents type whatever they like into it. The remaining columns are `internal`: the learner's ID, two dates, their country, their two consent flags, their account's status and their support region.

Applying the file also puts the owner and the descriptions where every reader sees them. The table's properties read `domain=crm, owner=team-crm`, and `email`'s description sits in the Iceberg schema, where any reader of the table finds it: "The email the learner signs in with." The classifications stay in the file, because that is where the access policy and the checks read them. After `apply`, the check finds 0 problems.

A spreadsheet can't give you that zero. The check walks the catalog, so a column the file doesn't describe would show up as a problem, as `work_email` does in [the section on spreadsheets](#a-spreadsheet-nobody-updates).

**Working rule:** Keep owners, domains, descriptions and classifications in one reviewed file in the repository, with a team as every table's owner and one of a closed list of classifications on every column. Write the descriptions into the catalog, so every reader of a table sees them, and test the catalog against the file by walking the catalog, never the file. If edits happen in a catalog's web page, pick one side as the source, and generate the other from it ([chapter 2](#a-catalog-you-can-test)).

## Capture one pipeline's column lineage {#capture-one-pipelines-lineage}

[The catalog](#catalog-a-table) says what a table holds. [Lineage](#lineage-from-pipelines) says where its data came from and where it went, and governance needs both: a table built from `crm.learners` may or may not copy its emails. OpenLineage, an open specification for lineage, describes a [**job**](#send-run-events), such as a pipeline, its **runs**, and the **datasets** each run reads and writes. A run reports itself in events: a START when it begins, then one that ends it, such as COMPLETE or FAIL. Events carry [**facets**](#send-run-events), OpenLineage's name for blocks of metadata attached to a run, a job or a dataset.

The [column lineage facet](#trace-output-columns), on the output of a COMPLETE event, lists for each output column the input columns it came from, each marked DIRECT or INDIRECT. [DIRECT](#add-indirect-inputs) means the output is derived from the input. [INDIRECT](#add-indirect-inputs) means the input shapes the output without being copied: the output column value is "impacted by the value of inputField column, but it's not derived from it" ([column lineage facet](https://openlineage.io/docs/spec/facets/dataset-facets/column_lineage_facet)). The columns of a join or a filter condition are INDIRECT inputs, since they decide which rows exist, and so are those of a grouping, sort, window or condition.

The client library computes no lineage of its own, as [chapter 4](#send-run-events) finds in its source: it sends what you give it. So the book's module `lib.lineage` does the work. It parses each pipeline's SQL with sqlglot and adds the INDIRECT inputs that sqlglot's own lineage leaves out. It also wraps `lib.pipelines.run`, which runs one of the platform's SQL pipelines on DuckDB and writes its output to the catalog: START before the run, COMPLETE with the lineage after it. The events go through OpenLineage's file transport to one file, with a fixed time and a run ID derived from that time and the job's name, so the example writes the same bytes every time.

The platform's `marketing_audience` pipeline is one `create table ... as select ...` statement that writes `marketing.audience`, the list of learners marketing may email. It selects each learner's ID, name, email and country from `crm.learners`, joins each learner to their favourite course category, ranked from their enrollments, and keeps the learners for whom `l.marketing_consent and l.status = 'active'` holds.

Run the pipeline with events on, read its lineage back from the events alone, and count the rows through `lib.warehouse.connect`, which gives DuckDB a view of each catalog table:

Example `audience-pipeline-lineage` (Python 3.12.15 · duckdb 2.0.0 · openlineage-python 1.55.0 · pyarrow 26.0.1 · pyiceberg 0.12.0 · pyyaml 6.0.3 · sqlglot 30.33.0), passing in Edition 2026.11.31:

```python
"""Run the marketing_audience pipeline with OpenLineage events, then read its column lineage back from the events alone.
The point rests on the two INDIRECT lines: the filter and the joins decide which rows the audience table holds."""

from lib import lineage, metadata, warehouse
from lib.catalog import open_catalog
from lib.platform import scratch

platform = open_catalog(["crm.learners", "learning.courses", "learning.enrollments"])
events_file = scratch() / "events.jsonl"                 # the file that stands in for a lineage server: one event per line
lineage.run(platform, "marketing_audience", lineage.client(events_file))
events = lineage.read_events(events_file)

print("events:", ", ".join(f"{e['eventType']} of {e['job']['name']}" for e in events))
print("the run read:", ", ".join(dataset["name"] for dataset in events[0]["inputs"]))  # the START event lists the inputs
print("the run wrote:", events[-1]["outputs"][0]["name"])                             # the last event, COMPLETE, names the output

graph = lineage.graph(events)                            # built from the COMPLETE events' column lineage, not from the SQL
for column in ("full_name", "email"):
    inputs = [e for e in graph.edges if e.target == f"marketing.audience.{column}"]
    print(f"marketing.audience.{column} comes from:", ", ".join(f"{e.source} ({e.type} {e.subtype})" for e in inputs))
# The INDIRECT inputs of each kind: the inputs that decide which rows exist, not the ones copied into them.
for subtype in ("FILTER", "JOIN"):
    shaping = [e.source for e in graph.edges if e.target == "marketing.audience" and e.subtype == subtype]
    print(f"its rows are shaped by, INDIRECT {subtype}:", ", ".join(shaping))
duckdb = warehouse.connect(platform)                     # DuckDB on the catalog's tables, to count what the join dropped
# The learners the filter keeps, counted on crm.learners before any join drops one.
consenting = "select count(*) from crm.learners where marketing_consent and status = 'active'"
print("consenting, active learners in crm.learners:", f"{duckdb.sql(consenting).fetchall()[0][0]:,}")  # one row, one column
print("rows in marketing.audience:", f"{duckdb.sql('select count(*) from marketing.audience').fetchall()[0][0]:,}")  # one row, one column
# Every column a change to email can reach, at any depth of the graph.
reached = lineage.columns_reached(graph, lineage.downstream(graph, "crm.learners.email"))
print("a change to crm.learners.email reaches:", ", ".join(reached))

# The catalog check, run on the catalog the pipeline wrote to.
for problem in metadata.problems(platform, metadata.load()):
    print("the catalog check finds:", problem)
```

Output:

```text
events: START of marketing_audience, COMPLETE of marketing_audience
the run read: crm.learners, learning.courses, learning.enrollments
the run wrote: marketing.audience
marketing.audience.full_name comes from: crm.learners.full_name (DIRECT IDENTITY)
marketing.audience.email comes from: crm.learners.email (DIRECT IDENTITY)
its rows are shaped by, INDIRECT FILTER: crm.learners.marketing_consent, crm.learners.status
its rows are shaped by, INDIRECT JOIN: crm.learners.learner_id, learning.courses.category, learning.courses.course_id, learning.enrollments.course_id, learning.enrollments.learner_id
consenting, active learners in crm.learners: 1,861
rows in marketing.audience: 1,829
a change to crm.learners.email reaches: marketing.audience.email
the catalog check finds: marketing.audience: no metadata
```

Read from the events alone, the lineage shows what the table's name leaves out. `marketing.audience.email` comes from `crm.learners.email (DIRECT IDENTITY)`, and `full_name` from `crm.learners.full_name (DIRECT IDENTITY)`, both unchanged copies of personal columns.

The filter and the joins decide which rows the table holds. The filter's inputs are `crm.learners.marketing_consent, crm.learners.status`, so every row says its learner consented. The joins' inputs are `crm.learners.learner_id, learning.courses.category, learning.courses.course_id, learning.enrollments.course_id, learning.enrollments.learner_id`. A learner with no enrollment has no favourite category, so the join to it drops them. `crm.learners` holds 1,861 consenting, active learners, and `marketing.audience` holds 1,829.

A change to `crm.learners.email` reaches only `marketing.audience.email` in this pipeline's table, so renaming the column, or changing its format, touches that column alone. With all four of the platform's pipelines sending lineage, [chapter 4](#impact-of-a-column-change) finds that it reaches no other.

The catalog check, from [the section before](#catalog-a-table) and run on the same catalog, finds `marketing.audience: no metadata`. The pipeline created a table nobody has described, and lineage changes what that gap means: from a table the platform knows nothing about to a copy of two personal columns, chosen by consent. Until someone describes it, the platform's access policy treats every column it can't find a classification for as `sensitive`. So anyone who reads the copy through the platform's enforcement layer, which [a later section](#enforce-one-access-decision) introduces, gets no value from it, as [chapter 6's who-sees-what test](#test-who-sees-what) shows.

**Working rule:** Emit lineage from the code that runs each job, never after it ([chapter 4](#send-run-events)): a START event, then a COMPLETE or FAIL event, with the output's column lineage in the COMPLETE event, parsed from the job's SQL, and the inputs that join and filter rows as well as those that supply values. Read lineage back from the events, never from the SQL, so the graph shows what ran. Where a job isn't SQL that a parser can read, such as a Python script, write its lineage in the same code, and review the two together.

## Enforce one access decision on a query {#enforce-one-access-decision}

With the platform's tables [cataloged](#catalog-a-table), the next question is who may read what. Policy as code starts from OPA's premise that policy can be decoupled from the software it governs, so the people responsible for it can write, analyze, version and distribute it separately ([OPA philosophy](https://www.openpolicyagent.org/docs/philosophy)). A [**policy**](#access-policies-as-code) answers a request with a [**decision**](#access-policies-as-code). The platform's policy answers with a document rather than a yes or no: whether to allow the read, the [**row filters**](#access-policies-as-code) that keep only the rows the user may see, and the [**column masks**](#access-policies-as-code) that change how a column is shown. The filters and masks are data, never SQL. A decision does nothing on its own. Something between the user and the engine has to apply it to every table a query reads.

Support agents on the platform each serve one region, `americas`, `emea` or `apac`, and need learners' contact details to answer tickets. The policy, `governance/policies/access.rego`, denies every request by default. It allows each group, the role a user holds in `governance/users.toml`, the tables its work needs, and reads the agent's region from the same file. This rule keeps an agent to their region:

```rego
row_filters contains {"column": "support_region", "equals": user.region} if {
	"support" in groups
	table in {"crm.learners", "support.tickets"}
}
```

`lib.policy.decide` evaluates the policy with regopy. `lib.enforce` holds the [**enforcement layer**](#enforcing-policies-on-queries): its `Enforcer` asks the policy about every table a query reads, replaces each with a subquery that applies the decision, and runs the rewrite on DuckDB. `lib.enforce.answer` is the one function a service calls, and it returns rows, never the engine's connection.

`agent-a05`, a support agent in EMEA, asks for three columns of `crm.learners` and counts learners by region through the layer. Next, the emails are checked against the generated files that `lib.platform.load` reads:

Example `support-region-enforced` (Python 3.12.15 · duckdb 2.0.0 · pyarrow 26.0.1 · pyiceberg 0.12.0 · regopy 1.5.2 · sqlglot 30.33.0), passing in Edition 2026.11.31:

```python
"""One access decision from the platform's Rego policy, and the same agent's query enforced in the layer. The point rests on
the emea line: the query names no region, yet only EMEA learners come back."""
import json

from lib import policy
from lib.catalog import open_catalog
from lib.enforce import Enforcer, answer
from lib.platform import load

# agent-a05, a support agent in EMEA, asks to read three columns of crm.learners.
request = policy.request("agent-a05", "crm.learners", ["learner_id", "email", "support_region"])  # the shape Trino's OPA plugin sends
decision = policy.decide(request)
assert decision["allow"] is True           # support may read crm.learners; a policy that denies it fails this run
print("allow:", decision["allow"])
print("row filters:", json.dumps(decision["row_filters"]))
print("column masks:", json.dumps(decision["column_masks"]))
print("reasons:", "; ".join(decision["reasons"]))

# The enforcement layer asks the policy about every table a query reads, then rewrites the query to apply the answer.
layer = Enforcer(open_catalog(["crm.learners"]))
query = "select support_region, count(*) as learners from crm.learners group by support_region order by support_region"
for region, learners in answer(layer, query, "agent-a05"):   # the layer adds the agent's region to the query before the engine runs it
    print(f"learners agent-a05 sees in {region}:", f"{learners:,}")

# Compare the emails the agent gets with the platform's data, without printing any.
stored = load("crm.learners")                                  # the generated data, not the catalog's copy
emails = dict(zip(stored.column("learner_id").to_pylist(), stored.column("email").to_pylist()))
seen = answer(layer, "select learner_id, email from crm.learners", "agent-a05")   # the agent's own query, with no region in it
print("emails agent-a05 sees, as stored:", f"{sum(email == emails[learner] for learner, email in seen):,} of {len(seen):,}")
print("learners in the platform's data:", f"{stored.num_rows:,}")
```

Output:

```text
allow: True
row filters: [{"column": "support_region", "equals": "emea"}]
column masks: {}
reasons: support may read crm.learners
learners agent-a05 sees in emea: 1,581
emails agent-a05 sees, as stored: 1,581 of 1,581
learners in the platform's data: 5,000
```

The layer applied the region inside the query, so the agent's SQL didn't have to, and couldn't avoid it. The decision allows the read, with the row filter `[{"column": "support_region", "equals": "emea"}]` on the agent's own region and the column masks `{}`, since support is the group that needs contact details. Its reason is `support may read crm.learners`. The agent's query has no filter on region, yet it returns 1,581 learners, all in EMEA, of the 5,000 the platform holds. Their emails come back as stored, 1,581 of 1,581, because the decision masks nothing.

regopy is the Python binding of rego-cpp, Microsoft's C++ implementation of Rego, which supports Rego v1.18.1 as OPA defines it ([rego-cpp README at v1.5.2](https://github.com/microsoft/rego-cpp/blob/v1.5.2/README.md)). regopy differs from OPA in ways [chapter 5](#where-regopy-differs-from-opa) lists, so OPA stays the reference for anything in production. [Chapter 6](#column-masks-leak) shows why the layer replaces whole tables rather than masking columns where a query names them.

**Working rule:** Write access rules as one policy in the repository that denies by default and returns its decision as data: allow, row filters and column masks. Apply that decision in one layer that rewrites every query before the engine runs it, and give users that layer's answers, never the engine's connection. When an engine can ask the policy itself, as Trino can ask OPA, let the engine be that layer.

## Fail the pull request that adds an unclassified column {#fail-the-pull-request}

[The catalog check](#catalog-a-table) proves the catalog complete on the day it runs, and the platform changes every week. A check that waits for someone to remember it gets run rarely and read less. Run on every pull request, where its exit code can fail it, the same check becomes a [gate](#run-every-rule-as-one-job), and its findings appear where engineers already work. The authors of Tricorder, Google's platform for code analysis, found time and again that when developers have to open a dashboard or run a separate command line tool, their use of analysis drops off ([Sadowski et al., 2015](https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/43322.pdf)). That is one company's experience with code analysis, not data, but governance checks meet the same people. A gate also fails a pull request only for what it changes, so a finding already on main is reported with its owner and fails no pull request that leaves it as it is.

The book's module for this is `lib.ci`, the platform's governance job. A schema change on the platform is a migration, an `alter table` statement in a file in the branch's `migrations/` folder. Each side of a pull request is a checkout in its own folder, one of main and one of the branch. For each side, `lib.ci.run` builds a [scratch platform](#governance-in-ci), a fresh catalog made the way the platform runs. It holds the source tables, the side's migrations, its metadata written into the catalog, and its pipelines. It runs the checks it's given on the branch's platform, runs them on main's alone to learn what is already there, and compares the findings by ID.

This section runs one check, `coverage`. The coverage check is [the catalog check](#catalog-a-table), `lib.metadata.problems`, run on the branch's platform, and for each column nobody classified it adds what a rule on column names suggests, from the rules [chapter 3](#classify-columns-from-the-schema) writes.

The CRM team's pull request adds a column for learners' work email in one migration and forgets the column's metadata, so the coverage check has something to find:

Example `unclassified-column-fails` (Python 3.12.15 · duckdb 2.0.0 · openlineage-python 1.55.0 · phonenumbers 9.0.43 · presidio-analyzer 2.2.366 · pyarrow 26.0.1 · pyiceberg 0.12.0 · pyyaml 6.0.3 · regopy 1.5.2 · sqlglot 30.33.0 · tldextract 5.4.0), passing in Edition 2026.11.31:

```python
"""The CRM team's pull request adds a column to crm.learners and no metadata for it; the coverage check runs on its branch and
on main. The point rests on the exit code line: the pull request's own finding fails the job, and the findings on main don't."""
from lib import ci

main = ci.checkout()                      # the main branch: the governance files and pipelines, and no migrations yet
branch = ci.checkout(main)                # the pull request's branch, starting from main
# The pull request's only change: a migration that adds the column, with no metadata for it.
(branch / "migrations" / "0001-add-work-email.sql").write_text("alter table crm.learners add column work_email varchar;\n")
print("files the pull request changes:", ", ".join(ci.changed(main, branch)))

# Build a scratch platform from each side, run the coverage check on both, and compare their findings by ID.
report = ci.run(main, branch, checks=["coverage"])
for line in ci.lines(report):
    if not line.startswith("change "):    # schema changes are chapter 10's report
        print(line)
for line in ci.totals(report):
    if line.startswith(("findings the pull request introduces", "findings already on main", "exit code")):
        print(line)
```

Output:

```text
files the pull request changes: migrations/0001-add-work-email.sql
coverage: failed, findings 3, new 1, on main 2, waived 0, 12 tables checked
coverage/ai.review_corpus: no metadata; on main, owner team-platform
coverage/crm.learners.work_email: not classified; its name suggests personal
coverage/marketing.audience: no metadata; on main, owner team-platform
job: failed
findings the pull request introduces: 1
findings already on main: 2
exit code: 1
```

The pull request fails on 1 finding of its own, the column nobody classified. The file it changes is `migrations/0001-add-work-email.sql`, and the coverage check reports `failed, findings 3, new 1, on main 2, waived 0, 12 tables checked`, which counts the platform's eight source tables and the four its pipelines write. The new finding is `crm.learners.work_email`, reported as `not classified; its name suggests personal`. The name rule matches `email`, which suggests the classification, and its ID, `coverage/crm.learners.work_email`, stays the same from run to run, so it can be discussed and counted.

The findings already on main were there before the pull request: `coverage/ai.review_corpus`, and `coverage/marketing.audience`, the table [the lineage section](#capture-one-pipelines-lineage) found undescribed. Nobody owns either table, so the job names the platform team, the team answerable for tables without an owner. The job exits with code `1`, so the pull request fails until the column has a classification, and the findings on main fail no pull request that leaves them as they are. [Chapter 10's job](#run-every-rule-as-one-job) runs the checks of chapters 2 to 7 on the same pull request, 11 of them.

**Working rule:** Run every governance check as one job on every pull request, required by [branch protection](#gate-the-pull-request), and fail it by exit code on any finding the pull request introduces. Report the findings already on the main branch with their owners, without failing the pull request for them. Put every finding in the pull request itself, under an ID that stays the same from run to run, with what to do about it. Run the same checks on a schedule against the production catalog too, for what changes outside pull requests.

## What goes wrong in production {#end-to-end-in-production}

Governance fails without an error in two ways that the pieces above answer: [a spreadsheet nobody updates](#a-spreadsheet-nobody-updates), and [a policy that another tool bypasses](#a-policy-another-tool-bypasses). The first is why the metadata is code, and the second why enforcement sits in one place.

### A governance spreadsheet nobody updates {#a-spreadsheet-nobody-updates}

The spreadsheet this chapter [started from](#governance-end-to-end) lists tables and columns, with a classification beside each, and a spreadsheet describes the schema on the day it was written. Schemas don't hold still. A study of ten open-source database applications, with more than 160,000 revisions between them, lists first among its major findings that database schemas evolve frequently over an application's life ([Qiu, Li and Su, 2013](https://www.cs.ucdavis.edu/~su/publications/fse13-db-study.pdf)). The study covers application databases, not warehouses, but every change that adds a column adds one its spreadsheet doesn't list.

Take the platform's spreadsheet, an inventory of `crm.learners` exported as CSV, one row per column. Then the CRM team's migration from [the pull request above](#fail-the-pull-request) adds `work_email`. Compare it with the catalog after the migration to find what it misses:

Example `spreadsheet-misses-new-column` (Python 3.12.15 · duckdb 2.0.0 · openlineage-python 1.55.0 · phonenumbers 9.0.43 · presidio-analyzer 2.2.366 · pyarrow 26.0.1 · pyiceberg 0.12.0 · pyyaml 6.0.3 · regopy 1.5.2 · sqlglot 30.33.0 · tldextract 5.4.0), passing in Edition 2026.11.31:

```python
"""A governance inventory kept as a spreadsheet, exported before a schema change, then compared with the catalog after it.
The point rests on the two work_email lines: the spreadsheet doesn't list the column, and the check finds it unclassified."""
import csv
import io

from lib import ci, metadata
from lib.catalog import open_catalog

platform = open_catalog(["crm.learners"])
described = metadata.load()

# The inventory: one row per column of crm.learners, exported on the day someone was asked for one.
inventory = io.StringIO()
writer = csv.writer(inventory)
writer.writerow(["table", "column", "classification"])
for field in platform.load_table("crm.learners").schema().fields:
    # Metadata is keyed by Iceberg field ID, not by name, so the lookup survives a rename.
    writer.writerow(["crm.learners", field.name, described["crm.learners"].columns[field.field_id].classification])

# Later, the CRM team's migration adds a column for learners' work email, applied as an Iceberg schema change.
ci.migrate(platform, "alter table crm.learners add column work_email varchar;")

rows = list(csv.DictReader(io.StringIO(inventory.getvalue())))
print("columns the spreadsheet lists:", len(rows))
print("of them without a classification, by the spreadsheet:", sum(not row["classification"] for row in rows))  # a blank classification cell: a column the spreadsheet never classified
live = [field.name for field in platform.load_table("crm.learners").schema().fields]
print("columns crm.learners has:", len(live))
print("columns the spreadsheet doesn't list:", ", ".join(sorted(set(live) - {row["column"] for row in rows})))
for problem in metadata.problems(platform, described):
    print("the check against the catalog finds:", problem)
```

Output:

```text
columns the spreadsheet lists: 15
of them without a classification, by the spreadsheet: 0
columns crm.learners has: 16
columns the spreadsheet doesn't list: work_email
the check against the catalog finds: crm.learners.work_email: not classified
```

The spreadsheet looks complete and misses a column. By its own account it lists 15 columns, and 0 of them lack a classification. The table has 16 columns, and the column it doesn't list is `work_email`. Nothing in the spreadsheet can show that, because nothing compares it with the catalog. `lib.metadata.problems`, [the catalog check](#catalog-a-table), walks the catalog and finds `crm.learners.work_email: not classified`, the same finding that fails [the CRM team's pull request](#fail-the-pull-request). In our experience, a copy of the metadata kept by hand, whether a spreadsheet, a wiki page or a data dictionary, goes stale on its first schema change, and nothing says so.

**Working rule:** Never keep governance metadata in a document that nothing checks. Keep it in a reviewed file that a test compares with the live catalog, on every pull request and on a schedule. If people need a spreadsheet, generate it from the file, and say on it when it was generated.

### A policy one tool enforces and another bypasses {#a-policy-another-tool-bypasses}

A policy protects only the reads that pass through whatever enforces it. Trino applies row filters only when it's told where to fetch them, so without its `opa.policy.row-filters-uri` setting it applies no row filtering at all ([Trino 484, OPA access control](https://trino.io/docs/484/security/opa-access-control.html)). Iceberg's REST catalog specification, at 1.12.0, lets a catalog return read restrictions with a table, and states that "A reader that supports read restrictions must fail any read against the loaded table that cannot apply a returned restriction in full" ([REST catalog spec, 1.12.0](https://github.com/apache/iceberg/blob/apache-iceberg-1.12.0/open-api/rest-catalog-open-api.yaml)). The spec gives a reader a signal for delegated access, the `X-Iceberg-Access-Delegation` header, but none for read restrictions. So a catalog can't tell which readers will apply its restrictions. PyIceberg 0.12.0's REST client reads none ([source](https://github.com/apache/iceberg-python/blob/pyiceberg-0.12.0/pyiceberg/catalog/rest/__init__.py)). We recommend that a catalog that returns read restrictions give storage access only to engines that apply them.

The [enforcement layer](#enforce-one-access-decision) keeps `agent-a05` to EMEA. The layer has routes around it, though, because anything with the catalog's access can read the table's files. Read the table around the layer twice: with DuckDB through `lib.warehouse.connect`, which the layer itself uses, as a notebook would, and with PyIceberg's own scan, as any reader holding the catalog's access would:

Example `read-around-the-layer` (Python 3.12.15 · duckdb 2.0.0 · pyarrow 26.0.1 · pyiceberg 0.12.0 · regopy 1.5.2 · sqlglot 30.33.0), passing in Edition 2026.11.31:

```python
"""The same table, read three ways: through the enforcement layer, and around it with DuckDB and with PyIceberg. The point
rests on the two lines around the layer, which return every region where the layer returns only EMEA's."""
from collections import Counter

from lib import warehouse
from lib.catalog import open_catalog
from lib.enforce import Denied, Enforcer, answer

platform = open_catalog(["crm.learners"])
query = "select support_region, count(*) from crm.learners group by support_region order by support_region"


def show(rows):
    return ", ".join(f"{region} {learners:,}" for region, learners in rows)


layer = Enforcer(platform)
print("through the enforcement layer, as agent-a05:", show(answer(layer, query, "agent-a05")))

# A notebook with DuckDB on the same catalog: the same SQL, and no policy in the way.
print("DuckDB on the same catalog:", show(warehouse.connect(platform).sql(query).fetchall()))

# Any reader that holds the catalog's access, here PyIceberg's own scan: no SQL to rewrite at all.
scan = platform.load_table("crm.learners").scan(selected_fields=("support_region", "email")).to_arrow()
print("PyIceberg's scan of the table:", show(sorted(Counter(scan.column("support_region").to_pylist()).items())))
print("emails PyIceberg's scan returns:", f"{scan.num_rows - scan.column('email').null_count:,}")  # rows whose email isn't null

# lib.warehouse builds each table's view on a scan it registers under a name of its own. Through the layer, it's denied.
try:
    answer(layer, "select count(*) from scan_crm_learners", "agent-a05")
except Denied as reason:
    print("the layer, asked for the raw scan:", reason)
```

Output:

```text
through the enforcement layer, as agent-a05: emea 1,581
DuckDB on the same catalog: americas 2,276, apac 1,143, emea 1,581
PyIceberg's scan of the table: americas 2,276, apac 1,143, emea 1,581
emails PyIceberg's scan returns: 5,000
the layer, asked for the raw scan: not a table in the catalog: scan_crm_learners
```

A reader outside the layer gets every region, because the policy was never asked. Through the layer, the agent's query returns `emea 1,581`. The same SQL on DuckDB with no layer in front returns `americas 2,276, apac 1,143, emea 1,581`. PyIceberg's scan, which has no SQL to rewrite, returns `americas 2,276, apac 1,143, emea 1,581`, with 5,000 emails. The same bypass happens in production whenever a second engine, a Spark job or a script with storage credentials reads the files the governed engine reads.

`lib.warehouse` builds each table's view on a scan that it registers under a name of its own, `scan_crm_learners` for this table, so the layer's connection holds the raw scan too. The last line of output shows the layer's answer when a query names that scan: `not a table in the catalog: scan_crm_learners`. That's the design [chapter 6](#enforcing-policies-on-queries) follows, which enforces in one place and denies everything else that reads data, [whatever isn't a catalog table](#allow-only-catalog-sources). In production, that place is the engine that tools query through, as [chapter 6](#engine-enforcement) argues. And since [storage permissions are part of enforcement](#enforcement-in-production), only that engine and the jobs that write the tables should be able to read the files.

**Working rule:** Choose one place that enforces the policy, the engine every tool queries through, and give storage credentials only to it and to the jobs that write the tables, never to people or notebooks. Deny every query in that place that reads data other than through a catalog table. Before you count a table as protected, list everything that can read its files, and treat any reader that doesn't ask the policy as a bypass, unless it's a job whose outputs are cataloged and classified like the tables it reads.

## What this chapter decided {#end-to-end-decisions}

- **Governance is four kinds of code in one repository:** a metadata file, the code that sends each job's lineage events, an access policy in Rego, and checks that run on every pull request. Only one of them is a policy engine.
- **Every table has an owning team,** never a person, and every column one of four classifications: `public`, `internal`, `personal` or `sensitive`. They live in `governance/metadata.toml`, keyed by Iceberg field ID, and the catalog is tested against the file by walking the catalog.
- **Every pipeline run sends OpenLineage events,** with column lineage parsed from its SQL, including the columns that join and filter rows.
- **Access decisions are data:** allow, row filters and column masks, from one policy that denies by default. An enforcement layer applies them to every query, and denies every read that isn't of a catalog table.
- **Missing metadata fails closed:** the access policy treats a column nobody has classified as `sensitive`, and a table nobody has described stays a finding in every report until someone describes it.
- **The governance job runs on every pull request,** fails on the findings the pull request introduces, and reports the findings already on main with their owners.
- **Everything runs in one process, offline,** on the stand-ins [listed at the start of this chapter](#governance-end-to-end). Governance done this way is a set of small rules, each checked where the data changes, so a gap in the metadata shows up as a failed check rather than as a surprise in a report. The chapters that follow test each rule in turn, on the platform's own tables.

# A catalog you can test

By the end of this chapter you can keep a catalog's metadata as code. Every table gets an owner and a domain, and every column a description and a classification, in a file your team reviews. The file is keyed so a rename can't lose anything, written into the Iceberg catalog, and checked by a test that fails when anything is missing. You'll also have a local model draft descriptions and classifications, and measure how often its drafts are wrong before anyone accepts them.

The platform's catalog has three problems. Its warehouse, an Iceberg catalog kept by an online learning platform whose pipelines keep adding tables, holds eight source tables in namespaces named after the teams that run them, such as `crm.learners` and `support.tickets`. Its pipelines build reporting tables from those and keep creating tables, and nothing says who owns `marketing.audience`, the list of learners marketing may email, or whether `ai.review_corpus`, the text the platform's course assistant reads, holds personal data. A column renamed in a schema change can lose its classification without anyone noticing. And every table a pipeline adds needs a description and a classification for each of its columns, written by hand, which is slow enough that teams skip it.

[Chapter 1](#governance-end-to-end) built a rough version of the whole governed platform. This chapter makes its catalog solid.

The chapter's examples use five of the book's shared modules. `lib.catalog.open_catalog` builds a fresh catalog of the platform's tables in a temporary folder, on PyIceberg's SQL catalog over SQLite, which PyIceberg's docs recommend only for exploration and development ([PyIceberg configuration](https://github.com/apache/iceberg-python/blob/pyiceberg-0.12.0/mkdocs/docs/configuration.md)). `lib.pipelines.run` runs one of the platform's pipelines and writes its output to the catalog. `lib.warehouse.connect` opens DuckDB on the catalog's tables, so a query can name them as the platform does, and `lib.policy` evaluates the platform's access policies, which [chapter 5](#access-policies-as-code) writes. And `lib.metadata`, which this chapter builds up, loads the platform's metadata, writes it into the catalog and checks the catalog against it.

## Keep the metadata in a reviewed file {#keep-metadata-in-a-reviewed-file}

A catalog's **metadata** is what you know about each table that the data doesn't say: who owns it, which part of the business it belongs to, what it holds, and how each column must be protected. Catalog servers keep this in their own database and edit it in a web page. That works for browsing, but a change made there is invisible to code review, and no test can say whether the catalog is complete. Keeping the metadata in a file in the repository, beside the pipelines that create the tables, makes every change a diff that someone approves, and gives a test something to compare the catalog with.

The platform's file is `governance/metadata.toml`. For each table it records:

- an **owner**, the team answerable for the table, never a person;
- a **domain**, the part of the business the table belongs to;
- a description of the table;
- for each column, its name, a description and a **classification**.

A classification is one of four values, ordered from least to most protected:

| Classification | Meaning | On the platform |
| --- | --- | --- |
| `public` | May be shown to anyone outside the platform | A course's title |
| `internal` | For the platform's staff, and doesn't identify a person by itself | When an enrollment started |
| `personal` | Identifies a person or tells something about one | A learner's email, or text they wrote |
| `sensitive` | Data whose leak would directly harm someone | A card number, an instructor's bank account |

The order matters: when a column is built from several others, as a pipeline's output columns often are, the stricter classification wins, and `lib.metadata.stricter` returns it. Loaded and counted, the file shows what it covers:

Example `metadata-model` (Python 3.12.15 · pyarrow 26.0.1 · pyiceberg 0.12.0), passing in Edition 2026.11.31:

```python
"""Loads the metadata file and counts what it describes, then prints four of the learners table's columns, looked up by
field ID. The prose's point rests on the `country` and `city` lines, whose descriptions give the reason for each classification."""
from collections import Counter

from lib import metadata

described = metadata.load()                      # governance/metadata.toml
columns = [column for table in described.values() for column in table.columns.values()]
print("tables described:", len(described))
print("columns described:", len(columns))
counts = Counter(column.classification for column in columns)
for level in metadata.LEVELS:
    print(f"columns classified {level}:", counts[level])
owners = sorted({table.owner for table in described.values()})
print("teams that own tables:", len(owners))
print("owners:", ", ".join(owners))

learners = described["crm.learners"]
print("crm.learners owner:", learners.owner)
print("crm.learners domain:", learners.domain)
for field_id in (1, 3, 5, 6):                    # field IDs, which the file keys columns by, not positions in the table
    column = learners.columns[field_id]
    print(f"crm.learners field {field_id}:", column.name, "|", column.classification, "|", column.description)
```

Output:

```text
tables described: 10
columns described: 77
columns classified public: 6
columns classified internal: 57
columns classified personal: 12
columns classified sensitive: 2
teams that own tables: 5
owners: team-analytics, team-billing, team-crm, team-learning, team-support
crm.learners owner: team-crm
crm.learners domain: crm
crm.learners field 1: learner_id | internal | The learner's ID, which the platform's other tables use for the learner. Identifies a person through this table's personal columns.
crm.learners field 3: email | personal | The email the learner signs in with.
crm.learners field 5: country | internal | The learner's country, as an ISO 3166 two-letter code.
crm.learners field 6: city | personal | The learner's city. Identifying together with date of birth.
```

Most columns are `internal`, because most of a warehouse is IDs, dates and counts: 57 of its 77 columns are. The protected columns are few. Of the file's columns, 12 are `personal`, and `sensitive` holds the card number in `billing.payments` and the payout account in `crm.instructors`. The file covers the eight source tables and the two reporting tables the platform's analysts rely on, and every table belongs to one of 5 owning teams. The point is that each description carries the reason for its classification. `country` is `internal` because a country alone doesn't single anyone out, while `city` is `personal` because, as its description says, it identifies a learner together with their date of birth.

The four values and the one-owner rule are ours, not a standard's. The Open Data Contract Standard (ODCS) records a domain, an owner as a free-text role on a team, and a free-text classification ([ODCS v3.2.0](https://github.com/bitol-io/open-data-contract-standard/tree/v3.2.0/docs)), and its JSON schema accepts any string as a classification ([ODCS JSON schema](https://github.com/bitol-io/open-data-contract-standard/blob/v3.2.0/schema/odcs-json-schema-v3.2.0.json)). The catalog servers differ too. DataHub types its owners ([ownership types](https://docs.datahub.com/docs/ownership/ownership-types)), and it lets an asset belong to only one domain at a time ([domains](https://docs.datahub.com/docs/domains)). It classifies columns with glossary terms ([business glossary](https://docs.datahub.com/docs/glossary/business-glossary)).

OpenMetadata lets an owner be either a team or several users, a rule enforced at the instance level and on by default ([data ownership](https://docs.open-metadata.org/v2.0.x/how-to-guides/guide-for-data-users/data-ownership)). Its classifications are a controlled vocabulary of tags ([classification](https://docs.open-metadata.org/v2.0.x/how-to-guides/data-governance/classification)), such as `PII.Sensitive` ([PII tags](https://docs.open-metadata.org/v2.0.x/how-to-guides/data-governance/classification/sample-data-using-pii-tag)). In Databricks' Unity Catalog every object has an owner, who holds every privilege on it ([privileges](https://docs.databricks.com/aws/en/data-governance/unity-catalog/manage-privileges)). Tables and columns carry tags ([tags](https://docs.databricks.com/aws/en/database-objects/tags)), and a governed tag takes values "only from a predefined set of allowed values" ([governed tags](https://docs.databricks.com/aws/en/admin/governed-tags/)). A file like the platform's maps onto each of them. None of these documents describes the check this chapter's test makes: a team, never a person, as the owner of every table, and one of the four classifications on every column.

**Working rule:** Keep owners, domains, descriptions and classifications in a file in the repository, reviewed like the pipelines. Use a closed list of classifications, ordered by how much protection each needs, and one owning team per table. If your organization edits metadata in a catalog server's web page, make one side the source and generate the other from it; two sources that both accept edits drift apart.

## Key column metadata by field ID {#key-column-metadata-by-field-id}

[The platform's metadata file](#keep-metadata-in-a-reviewed-file) keys each column by a number, not by its name, because names change. Iceberg gives every column a **field ID**, an integer unique within the table's schema, and the spec requires that a rename keep it: "Renaming an existing field must change the name, but not the field ID" ([Iceberg table spec](https://iceberg.apache.org/spec/)). Readers find values by that ID too, since Iceberg selects a data file's columns by field ID, so a renamed column still reads the files written before the rename ([Iceberg table spec](https://iceberg.apache.org/spec/)). Metadata keyed by name has nothing to hold on to: after a rename, the old name points at nothing and the new name has no entry.

The platform's CRM team wants `crm.learners.email` called `contact_email`, to make room for a work email. Staged in a scratch catalog, the rename shows what a name-keyed copy loses. The column is looked up in the metadata file, keyed by field ID, and in a copy keyed by name, as a spreadsheet of columns usually is:

Example `rename-orphans-name-keys` (Python 3.12.15 · pyarrow 26.0.1 · pyiceberg 0.12.0 · regopy 1.5.2), passing in Edition 2026.11.31:

```python
"""Renames crm.learners.email in a scratch catalog, then looks the column up by field ID and by name. The prose's point
rests on the two classification lines, which disagree, and on the check's `renamed from email` line."""
from lib import catalog, metadata, policy

platform = catalog.open_catalog(["crm.learners"])
described = metadata.load()
by_id = described["crm.learners"].columns                                    # the book's metadata: keyed by field ID
by_name = {column.name: column.classification for column in by_id.values()}  # a spreadsheet's: keyed by name

learners = platform.load_table("crm.learners")
with learners.update_schema() as update:
    update.rename_column("email", "contact_email")                           # renames in place: the field ID stays
field = learners.schema().find_field("contact_email")
print("contact_email field ID:", field.field_id)
values = learners.scan(selected_fields=("contact_email",)).to_arrow().column("contact_email")
print("values read under the new name:", f"{len(values) - values.null_count:,} of {len(values):,}")

print("classification of contact_email, looked up by name:", by_name.get("contact_email"))
print("classification of contact_email, looked up by field ID:", by_id[field.field_id].classification)
names = {f.name for f in learners.schema().fields}
print("name-keyed entries that match no column:", ", ".join(sorted(set(by_name) - names)))
for problem in metadata.problems(platform, described):
    print("check:", problem)
# The policies read classifications by name, generated from the file, which still says email.
decision = policy.decide(policy.request("agent-a01", "crm.learners", ["contact_email"]))
print("a support agent reading contact_email sees it as:", decision["column_masks"].get("contact_email", "shown"))
```

Output:

```text
contact_email field ID: 3
values read under the new name: 5,000 of 5,000
classification of contact_email, looked up by name: None
classification of contact_email, looked up by field ID: personal
name-keyed entries that match no column: email
check: crm.learners.contact_email: renamed from email since it was reviewed
a support agent reading contact_email sees it as: redact
```

The two lookups disagree, and only the one keyed by field ID knows the column is the same. The rename kept field ID 3, and 5,000 of 5,000 values came through under the new name. Keyed by name, `contact_email` has no classification, and the entry for `email` matches no column. Keyed by field ID, the column is still `personal`. A check built on the name-keyed copy would report an unclassified column and miss that a classified one had gone.

The ID-keyed file does notice the rename, and `lib.metadata.problems`, the check this chapter builds on, reports it: `contact_email: renamed from email since it was reviewed`. The data surviving a rename doesn't make the rename harmless: SQL that names `email` breaks, and so does any system that knows columns only by name. The platform's access policies, which [chapter 5](#access-policies-as-code) writes, receive the columns a query reads by name, so they read classifications from a name-keyed view that `lib.policy` generates from this file. When the check fails, someone updates the name in the file, rereads the description, and the generated view follows. Until then, the view has no entry for `contact_email`, and the policies treat a column they can't find as `sensitive`. A support agent, who sees every other personal column of `crm.learners`, gets `contact_email` as `redact`: the rename fails closed, and support notices first.

**Working rule:** Key column metadata by field ID, with the name beside it for people. Treat a name that no longer matches as a reason to review the column's description, not as a new column. Where a system knows columns only by name, generate its name-keyed view from the ID-keyed file, never maintain it by hand.

## Write descriptions into the table's schema {#write-descriptions-into-the-schema}

[The metadata file](#keep-metadata-in-a-reviewed-file) is where the platform's metadata is reviewed. People and tools that read a table, though, look in the catalog, so some of the metadata has to be copied there. The question is where each fact belongs in the catalog, and Iceberg answers part of it: each column can carry a `doc`, an optional comment that travels with the schema ([Iceberg table spec](https://iceberg.apache.org/spec/)). A table also has properties, a map of strings, but the spec says that map "is used to control settings that affect reading and writing and is not intended to be used for arbitrary metadata."

So we follow the spec as far as we can. Column descriptions go into `doc`, and classifications stay in the file, where the policies and checks read them. The owner and the domain go into two table properties, as a pointer for tools that read only the table, so anyone with a table and no repository can find the team to ask. That last choice goes against the spec's stated intent, and we keep it to those two short values.

The Iceberg project's own answer is a separate channel. The REST catalog spec of Iceberg 1.12.0 adds `labels`, ownership or classification that a catalog returns with a table, per table and per field ID, kept out of properties on purpose ([REST catalog spec](https://github.com/apache/iceberg/blob/apache-iceberg-1.12.0/open-api/rest-catalog-open-api.yaml)). The change came in with [pull request 15750](https://github.com/apache/iceberg/pull/15750). PyIceberg 0.12.0's REST client doesn't read them ([source](https://github.com/apache/iceberg-python/blob/pyiceberg-0.12.0/pyiceberg/catalog/rest/__init__.py)).

`lib.metadata.apply` does the copying. For every table the file describes and the catalog holds, it writes each description into its column's `doc`, and the owner and domain into the properties, changing only what differs. Against a fresh catalog of the platform's eight source tables, `apply` shows what it changes:

Example `apply-metadata` (Python 3.12.15 · pyarrow 26.0.1 · pyiceberg 0.12.0), passing in Edition 2026.11.31:

```python
"""Writes the metadata file into a fresh catalog of the eight source tables, then applies it again. The prose's point rests
on the second `tables changed` line, which is zero: applying a file the catalog already matches writes nothing."""
from lib import catalog, metadata

platform = catalog.open_catalog()                # all eight source tables, with no metadata yet
described = metadata.load()
print("crm.learners schema ID before:", platform.load_table("crm.learners").schema().schema_id)
print("tables changed:", len(metadata.apply(platform, described)))

learners = platform.load_table("crm.learners")
print("crm.learners schema ID after:", learners.schema().schema_id)
print("email's doc, in the Iceberg schema:", learners.schema().find_field("email").doc)
scanned = learners.scan(selected_fields=("email",), limit=1).to_arrow()   # one row is enough: the schema comes back with it
# Arrow keeps an Iceberg column's doc in the field's metadata, under the bytes key b"doc".
print("email's doc, in a scan's Arrow schema:", scanned.schema.field("email").metadata[b"doc"].decode())
print("crm.learners properties:", ", ".join(f"{k}={v}" for k, v in sorted(learners.properties.items())))
print("tables changed by applying again:", len(metadata.apply(platform, described)))
```

Output:

```text
crm.learners schema ID before: 0
tables changed: 8
crm.learners schema ID after: 1
email's doc, in the Iceberg schema: The email the learner signs in with.
email's doc, in a scan's Arrow schema: The email the learner signs in with.
crm.learners properties: domain=crm, owner=team-crm
tables changed by applying again: 0
```

Applying the file writes the metadata into the catalog, and applying it again writes nothing, so `apply` can run on every merge. The first run changed 8 tables. The description of `email` is now in the Iceberg schema and in the Arrow schema a scan returns, so code that reads the table through PyIceberg gets it with the data, without knowing the file exists. The properties hold `owner=team-crm` and `domain=crm`. A second run changed 0 tables. The schema ID of `crm.learners` went from 0 to 1, although no column's ID, name or type changed, which is the subject of [the next subsection](#a-description-edit-is-a-schema-change).

### A description edit is a schema change {#a-description-edit-is-a-schema-change}

Iceberg's spec gives every schema change a new schema with its own ID, which becomes the table's current schema and joins the list of schemas the table keeps ([Iceberg table spec](https://iceberg.apache.org/spec/)). In PyIceberg 0.12.0, editing nothing but a column's `doc` counts as such a change. Change nothing but the description of `learning.reviews.body`, after [`lib.metadata.apply`](#write-descriptions-into-the-schema) has written the platform's descriptions, and watch the schema ID:

Example `doc-edit-schema-id` (Python 3.12.15 · pyarrow 26.0.1 · pyiceberg 0.12.0), passing in Edition 2026.11.31:

```python
"""Changes one column's description after the metadata is applied, then compares schema IDs with field IDs, names and
types. The prose's point rests on the schema ID changing while the shape stays the same, which the assert checks."""
from lib import catalog, metadata

platform = catalog.open_catalog(["learning.reviews"])
metadata.apply(platform, metadata.load())    # the platform's descriptions go in first, so the edit below is the only change
reviews = platform.load_table("learning.reviews")
before = reviews.schema()
print("schema ID before the edit:", before.schema_id)

with reviews.update_schema() as update:                                        # change one description, nothing else
    update.update_column("body", doc="The review's text, as the learner wrote it. May hold names and emails.")
after = reviews.schema()
assert metadata.shape(before) == metadata.shape(after), "a doc edit changed a field ID, name or type"
print("schema ID after the edit:", after.schema_id)
print("schemas the table keeps:", len(reviews.metadata.schemas))
print("schema ID changed:", before.schema_id != after.schema_id)
print("field IDs, names or types changed:", metadata.shape(before) != metadata.shape(after))
```

Output:

```text
schema ID before the edit: 1
schema ID after the edit: 2
schemas the table keeps: 3
schema ID changed: True
field IDs, names or types changed: False
```

A description edit changes the schema ID and nothing a review should compare. The ID went from 1 to 2, and the table keeps 3 schemas: the original, the one `apply` wrote, and this one. The field IDs, names and types are the same, which `lib.metadata.shape` compares. A rule that asks for a new classification whenever the schema ID changes would fire on every description edit, and teams learn to ignore a check that fires for nothing.

**Working rule:** Write column descriptions into Iceberg's `doc`, so every reader of the table gets them, and keep classifications in the reviewed file. Put no more than an owner and a domain in table properties, and say in your code why. To decide whether a schema change needs a review, compare field IDs, names and types, never schema IDs.

## Test the catalog against the metadata {#test-the-catalog-against-the-metadata}

With [the metadata applied to the catalog](#write-descriptions-into-the-schema), the remaining question is whether the file covers everything the catalog holds. A test answers it, and it has to start from the catalog: a check that walks the file finds problems only in tables someone already described. `lib.metadata.problems` lists every table in every namespace of the catalog, and reports a table with no entry or no owner, a column whose field ID has no classification, a classification that isn't one of the four, and a column renamed since it was reviewed.

On the platform, the gaps come from pipelines. Each pipeline is one `create table ... as select ...` in `pipelines/`, and running it creates a table. Run all four pipelines on a fresh catalog and check it, and the gaps show:

Example `metadata-gaps` (Python 3.12.15 · duckdb 2.0.0 · pyarrow 26.0.1 · pyiceberg 0.12.0 · sqlglot 30.33.0), passing in Edition 2026.11.31:

```python
"""Runs the platform's four pipelines on a fresh catalog, then checks every table in it against the metadata file. The
prose's point rests on the `problems found` line and the two `no metadata` lines that follow it."""
from lib import catalog, metadata, pipelines

platform = catalog.open_catalog()
for name in ("course_performance", "learner_activity", "marketing_audience", "review_corpus"):   # each one a file in pipelines/
    print("pipeline output:", pipelines.run(platform, name))

# The eight source tables plus the four outputs the pipelines just created.
tables = [f"{namespace}.{name}" for (namespace,) in platform.list_namespaces() for _, name in platform.list_tables(namespace)]
print("tables in the catalog:", len(tables))
found = metadata.problems(platform, metadata.load())
print("problems found:", len(found))
for problem in found:
    print("problem:", problem)
```

Output:

```text
pipeline output: analytics.course_performance
pipeline output: analytics.learner_activity
pipeline output: marketing.audience
pipeline output: ai.review_corpus
tables in the catalog: 12
problems found: 2
problem: ai.review_corpus: no metadata
problem: marketing.audience: no metadata
```

The two tables that fail are the ones that matter most. The catalog holds 12 tables, and the check finds 2 problems: `ai.review_corpus: no metadata` and `marketing.audience: no metadata`. The two analytics tables pass, because the file describes them, with field IDs that match the order of their `select` lists.

`marketing.audience` copies learners' names and emails out of `crm.learners`, and `ai.review_corpus` copies the free text of reviews, so both hold personal data, and neither has an owner who would answer for it. Until someone classifies them, the platform's access policy, `governance/policies/access.rego`, treats every column it can't find a classification for as `sensitive`, so the gap fails closed rather than open. [Chapter 4](#classifications-from-lineage) proposes their classifications from lineage, from the columns each output column was built from.

The test is then one line, `assert not metadata.problems(catalog, metadata.load())`, run wherever the catalog changes: in the pull request that adds a pipeline, which [chapter 10](#run-every-rule-as-one-job) sets up, and on a schedule against the production catalog, which catches tables created outside any pull request.

**Working rule:** Walk the catalog, not the metadata file, so a table nobody described shows up. Fail the test on any table without an owner and any column without one of the four classifications, and never default a missing classification to anything but the most protected value. A pipeline that creates a table ships with that table's metadata, in the same pull request.

## Catch schema changes before the metadata goes stale {#catch-schema-changes}

[The test against the catalog](#test-the-catalog-against-the-metadata) catches new tables. Tables also change underneath their metadata, and each kind of change needs its own response. A rename keeps the field ID, so the entry survives, but its description may now be wrong. An added column gets a field ID nobody has reviewed. A dropped column leaves an entry that describes nothing. And dropping a column and adding one with the same name doesn't bring the old column back: the spec gives every added field a new ID ([Iceberg table spec](https://iceberg.apache.org/spec/)), so the re-added column is a different column.

Schema changes are frequent enough for this to matter. In a study of ten open-source applications, "80% of the schema changes happened in 20-30% of the tables" ([Qiu, Li and Su, 2013](https://www.cs.ucdavis.edu/~su/publications/fse13-db-study.pdf)). That study covers application databases, not warehouses. If a warehouse's changes concentrate the same way, its busiest tables are the ones whose metadata goes stale first.

Two changes staged on `crm.learners` show both checks working: one adds `preferred_name`, with a description but no classification, and the other drops `country` and adds it back. `problems` and `lib.metadata.dropped`: the second reports entries whose field ID the table no longer has, and reads the dropped column's values the way Iceberg's time travel does, by naming a snapshot:

Example `schema-changes-checked` (Python 3.12.15 · pyarrow 26.0.1 · pyiceberg 0.12.0), passing in Edition 2026.11.31:

```python
"""Adds an unreviewed column, drops a column and adds it back, then runs both checks and reads the dropped values by time
travel. The prose's point rests on the two field IDs, and on the time-travel line, which still returns every country."""
from pyiceberg.types import StringType

from lib import catalog, metadata

platform = catalog.open_catalog(["crm.learners"])
described = metadata.load()
learners = platform.load_table("crm.learners")
written = learners.current_snapshot()                                        # the snapshot the rows were written in
print("country field ID, before it was dropped:", learners.schema().find_field("country").field_id)

with learners.update_schema() as update:                                     # a column nobody has reviewed
    update.add_column("preferred_name", StringType(), doc="What the learner asks to be called.")
with learners.update_schema() as update:                                     # drop a column, then add it back
    update.delete_column("country")
with learners.update_schema() as update:
    update.add_column("country", StringType())                               # an added field gets a new ID, never the old one

country = learners.schema().find_field("country")
print("country field ID, after it was added back:", country.field_id)
values = learners.scan(selected_fields=("country",)).to_arrow().column("country")
print("country values after it was added back:", f"{values.null_count:,} nulls of {len(values):,}")
# Minus the one snapshot that wrote the rows: the schema changes wrote none of their own.
print("snapshots the schema changes wrote:", len(learners.metadata.snapshots) - 1)
old = learners.scan(snapshot_id=written.snapshot_id, selected_fields=("country",)).to_arrow().column("country")
print("country values, read by time travel to that snapshot:", f"{len(old) - old.null_count:,} of {len(old):,}")
for problem in metadata.problems(platform, described):
    print("problems():", problem)
for problem in metadata.dropped(platform, described):
    print("dropped():", problem)
```

Output:

```text
country field ID, before it was dropped: 5
country field ID, after it was added back: 17
country values after it was added back: 5,000 nulls of 5,000
snapshots the schema changes wrote: 0
country values, read by time travel to that snapshot: 5,000 of 5,000
problems(): crm.learners.preferred_name: not classified
problems(): crm.learners.country: not classified
dropped(): crm.learners.country: field 5 is described but no longer in the table
```

A re-added column is a new column: `country` came back as field 17, not 5, and a plain scan, which reads through the current schema where the old field no longer exists, returns 5,000 nulls of 5,000 for it. Both added columns are unclassified, and the description given to `preferred_name` doesn't count: a description is a doc in the schema, and a classification is a reviewed line in the file.

The learners' countries haven't gone anywhere. A schema change writes no snapshot (0 here), so the table's current snapshot is still the one its rows were written in. A snapshot records the ID of the table's current schema when it was created ([Iceberg table spec](https://iceberg.apache.org/spec/)), and in PyIceberg 0.12.0, a scan that names a snapshot reads with that schema. So time travel to that snapshot returns 5,000 of 5,000 countries. Dropping a column deletes none of its values: [chapter 7](#retention-consent-erasure) is where the platform erases data for good. `dropped` reports that the reviewed entry for the old field now describes nothing. The rename from [the field ID section](#key-column-metadata-by-field-id) is the third kind of change, which `problems` reports as a column to review.

**Working rule:** Run both checks on every schema change: `problems` for columns nobody has classified or renamed ones nobody has re-read, and `dropped` for entries that describe nothing. Remove a dropped column's entry in the same pull request that drops it. Dropping a personal column doesn't remove its values, which time travel reads until the snapshots that hold them expire and their files are deleted. Never drop and re-add a column to rename it: rename it in place, so it keeps its field ID and its data.

## Classify pseudonymous IDs for joining {#classify-pseudonymous-ids}

[The metadata file](#keep-metadata-in-a-reviewed-file) classifies some columns that point at people as `internal`, and the most common of them is `learner_id`. It's a number with no meaning of its own, and the platform's analysts and data scientists need it to join enrollments, events, payments and tickets to each other. An ID with no meaning of its own, such as `learner_id`, is a **pseudonymous identifier**: it doesn't name anyone, but it leads to someone through another table. One ID reaches further than its classification suggests. The code counts the tables that carry it, joins enrollments to learners, and asks the access policy who can make that join and see names:

Example `pseudonymous-learner-id` (Python 3.12.15 · duckdb 2.0.0 · pyarrow 26.0.1 · pyiceberg 0.12.0 · regopy 1.5.2), passing in Edition 2026.11.31:

```python
"""Counts the tables that carry learner_id and how many enrollments one join resolves to an email, then asks the access
policy who sees learners' names and also reads another table's learner_id. The prose's point rests on the enrollments
line and the last line's list of users."""
from lib import catalog, metadata, policy, warehouse

described = metadata.load()
carrying = {name: table.column("learner_id") for name, table in described.items() if table.column("learner_id")}   # column() gives None when absent
print("tables with a learner_id column:", len(carrying))
print("learner_id classified as:", ", ".join(sorted({column.classification for column in carrying.values()})))
personal = [c.name for c in described["crm.learners"].columns.values() if c.classification == "personal"]
print("personal columns in crm.learners:", ", ".join(personal))

con = warehouse.connect(catalog.open_catalog(["crm.learners", "learning.enrollments"]))
found, total = con.execute("""
    -- the left join keeps every enrollment; count(l.email) counts only those whose learner has an email
    select count(l.email), count(*)
    from learning.enrollments e left join crm.learners l on l.learner_id = e.learner_id
""").fetchall()[0]                              # fetchall returns a list of rows, and this query has exactly one
print("enrollments whose learner_id leads to an email:", f"{found:,} of {total:,}")

# Who can put a name on another table's learner_id: users who read that table and see crm.learners' names unmasked.
rego, seeing, naming = policy.interpreter(), [], []
for user in policy.data_document()["platform"]["users"]:
    names = policy.decide(policy.request(user, "crm.learners", ["full_name"]), rego)
    if names["allow"] and "full_name" not in names["column_masks"]:          # allowed, and not masked: the name shows in clear
        seeing.append(user)
        tables = [t for t in carrying if t != "crm.learners"                 # the other tables with learner_id this user may read
                  and policy.decide(policy.request(user, t, ["learner_id"]), rego)["allow"]]
        naming += [f"{user} ({', '.join(tables)})"] if tables else []
print("users who see learners' names in crm.learners:", ", ".join(seeing))
print("of them, users who read learner_id in another table:", ", ".join(naming))
```

Output:

```text
tables with a learner_id column: 7
learner_id classified as: internal
personal columns in crm.learners: full_name, email, phone, city, street_address, date_of_birth, notes
enrollments whose learner_id leads to an email: 20,000 of 20,000
users who see learners' names in crm.learners: agent-a01, agent-a05, agent-a09, marketing-1
of them, users who read learner_id in another table: agent-a01 (support.tickets), agent-a05 (support.tickets), agent-a09 (support.tickets)
```

One join reaches a person: 20,000 of 20,000 enrollments lead to an email, and with the email come the learner's name, phone number, address and date of birth, which are among the table's `personal` columns. Yet `learner_id` is `internal` in all 7 tables that carry it.

The EU's General Data Protection Regulation (GDPR) addresses data like this in Recital 26: "Personal data which have undergone pseudonymisation, which could be attributed to a natural person by the use of additional information should be considered to be information on an identifiable natural person" ([GDPR, Recital 26](https://eur-lex.europa.eu/eli/reg/2016/679/oj)). What the recital means for a given platform is a question for its lawyers, and we don't answer it.

What we do decide is how the book's code treats the ID. `internal` describes what the column reveals in the table that holds it, and each `learner_id`'s description in the metadata file says the rest, naming `crm.learners` as the table that resolves it, as `learning.enrollments`' does: "The learner who enrolled. Identifies a person when joined with crm.learners." The protection sits on the table that resolves the ID. [Chapter 5](#return-the-decision-as-one-document) decides who may read the personal columns of `crm.learners`, and the example's last lines ask its policy who sees learners' names there: agent-a01, agent-a05, agent-a09, marketing-1. Of them, only the support agents also read a `learner_id` in another table: agent-a01 (support.tickets), agent-a05 (support.tickets), agent-a09 (support.tickets). They can put a name on every ticket of their region, and that's their job. Marketing sees names but reads no other table that holds the ID.

So the policy, not the ID's classification, decides who can make the join. Classifying `support.tickets.learner_id` as `personal` would change nothing for support, the only group that reads the table, because the policy already shows support every personal column. Erasure, in [chapter 7](#plan-erasure-from-lineage), has to reach every table that holds a learner's `learner_id`, whatever its classification. We call such IDs pseudonymized, never anonymized.

**Working rule:** Classify a pseudonymous ID by what its own table reveals, so the joins your analysts need keep working, and say in its description what it resolves through. Protect the table that resolves it, and include the ID in every erasure. Who can resolve the ID is an access decision: list the users who may read both the ID and the resolving table's personal columns unmasked, and grant that pair only to those whose job is linking rows to people.

## Draft metadata with a local model {#draft-descriptions-with-a-local-model}

[The test against the catalog](#test-the-catalog-against-the-metadata) tells you which columns nobody has described, but it doesn't write the descriptions. Writing a description and choosing a classification for every column is the slowest part of keeping a catalog, and the work comes back with every table a pipeline adds. A language model can draft both from what a catalog tool can show it: a column's name, its type and a few of its values. Research supports both the help and its limits. On one benchmark, given instructions and a two-step pipeline, ChatGPT labelled the semantic types of table columns with "F1 scores of over 85% in zero- and one-shot setups" ([Korini and Bizer, 2023](https://arxiv.org/abs/2306.00745)). A study of generated column descriptions found that models struggle with columns whose meaning is ambiguous, and that those columns need manual expert input ([Wretblad et al., 2024](https://arxiv.org/abs/2408.04691)). So a model's draft is a proposal that a person accepts, corrects or rejects in the pull request. The first job is to measure how wrong the drafts are, so the review knows where to look.

The book's drafts, which the examples call the metadata suggestions, were recorded once by `recordings/metadata_suggestions.py`, and the examples read that recording, so none of them calls a model. The script runs Qwen3-1.7B at a pinned revision. The model is open, under the Apache-2.0 licence, so anyone can download the same weights. At 1.7 billion parameters it runs on a CPU, with no GPU, which keeps the sample values on the machine that runs it. And it's instruction-tuned, with a switch that turns off its "thinking", the reasoning it otherwise writes before it answers. The script decodes greedily, taking the most likely next token, although the model's card suggests sampling in this mode: "Temperature=0.7, TopP=0.8, TopK=20, and MinP=0" ([Qwen3-1.7B](https://huggingface.co/Qwen/Qwen3-1.7B)). A recording has to come out the same when someone records it again, and a sampled answer is one draw among many.

For each column of the platform's eight source tables, the model saw the table's name, the column's name and Iceberg type, five of the column's values chosen with a fixed seed, and the four classifications with the meanings from [the table of classifications](#keep-metadata-in-a-reviewed-file). Each meaning comes with two examples, and none of them is a column the platform has. For `sensitive`, they are "a government ID number or a health condition". The model was asked for a one-sentence description and one classification, as JSON. The recording's manifest names the model, its revision, a hash of the prompts and settings, and the date, so anyone can record again and compare.

Those five values are personal data whenever the column is: for `support.tickets.body`, the tickets the model read held learners' names, street addresses, a phone number, a date of birth and a bank account number. A model running on your own machine keeps them there. Sending the same prompt to a hosted model sends the platform's personal data to another company, which needs the same decision as any other transfer of it.

Set against the reviewed file, each draft's classification shows whether it agrees:

Example `suggestion-agreement` (Python 3.12.15 · pyarrow 26.0.1 · pyiceberg 0.12.0), passing in Edition 2026.11.31:

```python
"""Compares the recorded model's classification of each source column with the reviewed one, and counts how many agree.
The prose's point rests on the agreement line and on the two lines counting personal and sensitive columns."""
from collections import Counter

from lib import metadata

reviewed = metadata.load()                                   # governance/metadata.toml, reviewed by people
suggested = metadata.suggestions()                           # the recorded answers under the book's prompt: no model runs here
assert all(reviewed[s.table].columns[s.field_id].name == s.column for s in suggested)   # each answer names the file's column at its field ID
print("columns the model was asked about:", len(suggested))
print("answers that parse as the JSON asked for:", sum(s.description is not None for s in suggested))   # None when the answer isn't that JSON
print("answers with one of the four classifications:", sum(s.classification in metadata.LEVELS for s in suggested))

pairs = Counter((reviewed[s.table].columns[s.field_id].classification, s.classification) for s in suggested)
agree = sum(n for (was, said), n in pairs.items() if was == said)
print("classifications that agree with the review:", f"{agree} of {len(suggested)} ({agree / len(suggested):.0%})")
for level in metadata.LEVELS:
    said = sorted(((s, n) for (was, s), n in pairs.items() if was == level), key=lambda p: -p[1])
    print(f"reviewed {level}, the model said:", ", ".join(f"{s} {n}" for s, n in said))
for level in ("personal", "sensitive"):
    reviewed_as = sum(n for (was, _), n in pairs.items() if was == level)
    print(f"{level} columns the model called {level}:", f"{pairs[level, level]} of {reviewed_as}")
```

Output:

```text
columns the model was asked about: 64
answers that parse as the JSON asked for: 64
answers with one of the four classifications: 64
classifications that agree with the review: 40 of 64 (62%)
reviewed public, the model said: internal 3, public 3
reviewed internal, the model said: internal 37, public 5, sensitive 2
reviewed personal, the model said: internal 9, public 3
reviewed sensitive, the model said: internal 2
personal columns the model called personal: 0 of 12
sensitive columns the model called sensitive: 0 of 2
```

All 64 answers are valid JSON with one of the four classifications, which shows the format worked and says nothing about whether the answers are right. They agree with the review on 40 of 64 (62%) columns, and that single number hides where the misses are. Most of the agreement is `internal` columns called `internal`. Of the platform's personal columns, the model called 0 of 12 `personal`, and of its `sensitive` ones, the card number and the payout account, 0 of 2: it called both `internal`.

### Examples in a prompt can give the answer away {#prompt-examples-give-answers-away}

A prompt's examples are part of the measurement. The script also records the same model on the same columns under a second prompt, identical but for its examples. That prompt gives examples for `personal` and `sensitive` only, and they're the platform's own columns. For `personal`, they are "a name, contact details, an address or text they wrote", and for `sensitive`, a card number or a bank account. Those two are `billing.payments.card_number` and `crm.instructors.payout_iban`, the only sensitive columns. Side by side, the answers show what the examples change:

Example `suggestions-with-platform-examples` (Python 3.12.15 · pyarrow 26.0.1 · pyiceberg 0.12.0), passing in Edition 2026.11.31:

```python
"""Scores the recorded answers under two prompts against the reviewed classifications: the book's prompt, whose examples
come from outside the platform, and one whose examples are the platform's own columns. The prose's point rests on
the sensitive-column lines and the agreement counts."""
from lib import metadata

reviewed = metadata.load()
prompts = {"the book's prompt": metadata.SUGGESTIONS,
           "the prompt with the platform's columns as examples": metadata.SUGGESTIONS_WITH_PLATFORM_EXAMPLES}
for name, path in prompts.items():
    # Each recorded answer, paired with the classification the file reviewed for its column.
    suggested = [(s, reviewed[s.table].columns[s.field_id].classification) for s in metadata.suggestions(path)]
    agree = sum(s.classification == was for s, was in suggested)
    print(f"{name}, classifications that agree with the review:", f"{agree} of {len(suggested)}")
    for level in ("personal", "sensitive"):
        reviewed_as = [s for s, was in suggested if was == level]
        print(f"{name}, {level} columns called {level}:",
              f"{sum(s.classification == level for s in reviewed_as)} of {len(reviewed_as)}")
    for s, was in suggested:
        if was == "sensitive":
            print(f"{name}, {s.table}.{s.column}:", s.classification)
```

Output:

```text
the book's prompt, classifications that agree with the review: 40 of 64
the book's prompt, personal columns called personal: 0 of 12
the book's prompt, sensitive columns called sensitive: 0 of 2
the book's prompt, crm.instructors.payout_iban: internal
the book's prompt, billing.payments.card_number: internal
the prompt with the platform's columns as examples, classifications that agree with the review: 41 of 64
the prompt with the platform's columns as examples, personal columns called personal: 0 of 12
the prompt with the platform's columns as examples, sensitive columns called sensitive: 2 of 2
the prompt with the platform's columns as examples, crm.instructors.payout_iban: sensitive
the prompt with the platform's columns as examples, billing.payments.card_number: sensitive
```

A prompt that names the platform's own columns gets the sensitive ones right, and the book's prompt does not. With the platform's columns as examples, the model gets 2 of 2 sensitive columns right. The book's prompt gets 0 of 2, and calls both `internal`.

Overall agreement hardly moves, 41 of 64 against 40 of 64, so one number would never show the difference. Personal columns gained nothing from their examples: 0 of 12 came out `personal` under that prompt too.

"A card number" names `card_number` almost word for word, while "a name" leaves the model to decide that `full_name` is one, and it didn't. A prompt whose examples match the catalog makes a model look better than it is, on the columns that need protection most, so the book's prompt takes its examples from outside the platform's data.

**Working rule:** Draft metadata with a model only from inputs you're allowed to send it, preferably to a model on your own machines. Record each answer with the model, its revision, the prompt and the settings, so a reviewer can see what produced each draft, and parse and validate every answer before anything reads it. Take the prompt's examples from outside the tables you measure it on. Report agreement with the reviewed metadata per classification, never as one number.

## Measure where the model's drafts are wrong {#measure-where-the-drafts-are-wrong}

[The recorded drafts](#draft-descriptions-with-a-local-model), a local model's descriptions and classifications for the platform's source tables, agree with the reviewed classifications on most columns. For governance, which way they miss matters more. A classification more protected than it should be costs a reviewer a minute and an analyst a masked column. One less protected than it should be leaves personal data where the policies treat it as staff data, or as data anyone may see. Splitting the misses by direction shows which way the model fails:

Example `suggestion-mistakes` (Python 3.12.15 · pyarrow 26.0.1 · pyiceberg 0.12.0), passing in Edition 2026.11.31:

```python
"""Counts the recorded classifications that protect a column less than the review does, and those that protect it more,
then lists each kind of miss. The prose's point rests on the two counts and on the `personal columns the model called
internal` line."""
from lib import metadata

reviewed = metadata.load()
rank = metadata.LEVELS.index                                   # a column's position in LEVELS: a larger rank is more protected
weaker, stronger = [], []
for s in metadata.suggestions():
    if s.classification not in metadata.LEVELS:                    # an answer nobody can act on goes to a person
        continue
    column = reviewed[s.table].columns[s.field_id]
    if rank(s.classification) < rank(column.classification):
        weaker.append((s, column.classification))
    elif rank(s.classification) > rank(column.classification):
        stronger.append((s, column.classification))
print("columns the model protects less than the review:", len(weaker))
print("columns the model protects more than the review:", len(stronger))

# Each kind of miss, the most protected review first.
for misses in (weaker, stronger):
    for was, said in sorted({(was, s.classification) for s, was in misses}, key=lambda p: (-rank(p[0]), -rank(p[1]))):
        names = [f"{s.table}.{s.column}" for s, w in misses if (w, s.classification) == (was, said)]
        print(f"{was} columns the model called {said}:", len(names))
        print(f"which, {was} called {said}:", ", ".join(names))
```

Output:

```text
columns the model protects less than the review: 19
columns the model protects more than the review: 5
sensitive columns the model called internal: 2
which, sensitive called internal: crm.instructors.payout_iban, billing.payments.card_number
personal columns the model called internal: 9
which, personal called internal: crm.learners.full_name, crm.learners.email, crm.learners.phone, crm.learners.date_of_birth, crm.learners.notes, crm.instructors.full_name, crm.instructors.email, learning.events.ip_address, support.tickets.body
personal columns the model called public: 3
which, personal called public: crm.learners.city, crm.learners.street_address, learning.reviews.body
internal columns the model called public: 5
which, internal called public: learning.events.event_type, learning.events.occurred_at, learning.events.user_agent, learning.reviews.posted_at, support.tickets.opened_at
internal columns the model called sensitive: 2
which, internal called sensitive: crm.learners.marketing_consent, crm.learners.ai_training_consent
public columns the model called internal: 3
which, public called internal: learning.courses.course_id, learning.courses.title, learning.courses.price_usd
```

The model's misses fall on the columns that need protection most. It protected 19 columns less than the review did, and 5 more. Every personal and sensitive column is among the less protected. It called 9 personal columns `internal`, among them the names and emails in both `crm.learners` and `crm.instructors`, the learners' phone numbers, dates of birth and notes, the IP addresses of events and the text of support tickets. It called 3 of them `public`: learners' cities and street addresses, and the text of their reviews. The prompt's own meaning for `personal` begins "identifies a person or tells something about one", and a name or an email does both.

The other less protected columns are timestamps, an event's type and its user agent, which the model called `public` where the review chose `internal`: a smaller mistake, but one a reviewer still has to catch. The misses the other way cost less. The model called the 2 consent flags `sensitive`, and it called the course catalog's public ID, title and price `internal`.

Descriptions fail in another way, and four of the model's show it beside the reviewed ones:

Example `suggestion-descriptions` (Python 3.12.15 · pyarrow 26.0.1 · pyiceberg 0.12.0), passing in Edition 2026.11.31:

```python
"""Prints the recorded description beside the reviewed one for four columns, where the review knows what a name, a type
and five values can't say. The prose's point rests on the pairs themselves, read one after another."""
from lib import metadata

reviewed = metadata.load()
suggested = {(s.table, s.column): s for s in metadata.suggestions()}   # keyed by name, so the four columns below can be named

# Four columns where the platform knows more than a name, a type and five values say.
for table, column in [("crm.learners", "ai_training_consent"), ("crm.learners", "support_region"),
                      ("crm.learners", "notes"), ("billing.payments", "card_number")]:
    print(f"{table}.{column}, the model:", suggested[table, column].description)
    print(f"{table}.{column}, the review:", reviewed[table].column(column).description)   # looked up by name: the file keys columns by field ID, but `column()` finds one by name
```

Output:

```text
crm.learners.ai_training_consent, the model: This column indicates whether a learner has consented to receive AI training.
crm.learners.ai_training_consent, the review: Whether the learner agreed to their text training the platform's AI. Unknown for sign-ups before the question was asked.
crm.learners.support_region, the model: The 'support_region' column indicates the regional support area for learners in the online learning platform.
crm.learners.support_region, the review: The support team that serves the learner: americas, emea or apac.
crm.learners.notes, the model: Stores notes from learners about their move, billing address, or preferences.
crm.learners.notes, the review: Free text from support agents. May hold anything, so it is treated as personal.
billing.payments.card_number, the model: Stores the card numbers of customers for payment processing.
billing.payments.card_number, the review: The full card number, from a legacy export. Should not be stored at all.
```

Every description is fluent and plausible, and three of the four miss something that only someone who knows how the platform fills the column could write. `ai_training_consent` records whether the learner's text may train the platform's AI, not whether the learner wants training, and a null means the question was never asked. `notes` are written by support agents about learners, as the values hint, with a note about invoices for the learner's employer, and not by learners. And `card_number` is a legacy column that shouldn't exist at all, which is the most useful thing its description can say. The fourth is right in substance: the platform gives each learner the `support_region` of the support team that serves their country, and the model calls it their "regional support area". Nothing in a fluent description says whether it's right or missing something, so a reviewer reads every one. The authors of one method for generating column descriptions name the same failure in single-prompt models, which write hallucinated or incomplete descriptions ([Cai et al., 2026](https://arxiv.org/abs/2606.21685)).

These are one small model's answers, decoded greedily, to two prompts. The recording can't tell whether the model misses personal columns because of its size, or because it drafts with only a name, a type and five values to go on. A larger model or a better prompt may get more columns right, and the way to find out is to measure it the same way, against the reviewed file, before trusting it with tables nobody has reviewed.

**Working rule:** Let a model raise a column's classification, never lower it: only a reviewer lowers one, in the pull request, with the reason in the column's description. Treat the model's descriptions as first drafts for someone who knows the column to rewrite. Before a model drafts metadata for tables nobody has reviewed, measure it against your reviewed metadata, per classification and per direction of error, and measure again whenever the model or the prompt changes.

## What goes wrong in production {#catalog-in-production}

Nothing errors when metadata goes stale, and each failure below looks complete until someone relies on it.

- **Owners go stale.** Teams are renamed, merged and dissolved, and the owner in the file outlives them. A team, unlike a person, survives one departure, but the file still needs a check that every owner is a team that exists, run against your organization's list of teams. If you keep ownership in ODCS, its team members carry a `dateOut` and a `replacedByUsername` ([ODCS v3.2.0](https://github.com/bitol-io/open-data-contract-standard/tree/v3.2.0/docs)), which a check can read to find a team with no current member in the `owner` role.
- **Descriptions describe last year's schema.** A column's meaning changes while its name and type stay the same: `amount` starts holding refunds as negative numbers, or `status` gains a value. No schema check sees it. Code comments fail the same way. Research on them found that a detector catches inconsistencies more accurately when it sees a change before it's committed than when it sees only the code that results ([Panthaplackel et al., 2021](https://arxiv.org/abs/2010.01625)). For columns, the comparison holds only where a pull request shows the change. A pipeline's code changes in a diff, but a source system can change what it writes with no diff anywhere. Where there is a diff, make the pull request that changes a pipeline's logic show the descriptions of the columns it writes, so the reviewer reads both.
- **Metadata keyed by name goes stale on a rename.** A spreadsheet of columns, a dashboard's data dictionary, or a policy written against column names loses a column on every rename, as [the field ID section](#key-column-metadata-by-field-id) shows, and reports nothing. Generate every name-keyed view from the ID-keyed file, and run `problems` on every schema change.
- **Two sources of metadata overwrite each other.** A catalog server's web page and a file in a repository both accept edits, and a sync in either direction loses the other's. DataHub keeps edits made in its UI apart from ingested metadata, so that ingestion pipelines don't overwrite what users entered ([DataHub dataset entity](https://docs.datahub.com/docs/generated/metamodel/entities/dataset)). Choose one source, and make the other read-only or generated.
- **Description edits trigger reviews nobody needs.** In PyIceberg every doc edit gives the schema a new ID, as [the doc edit above](#a-description-edit-is-a-schema-change) shows, so a review keyed on schema IDs fires for nothing. Compare field IDs, names and types instead.
- **Drafted classifications are approved in bulk.** A pull request that adds forty model-drafted columns gets its descriptions read and its classifications skimmed, and [the recorded drafts](#measure-where-the-drafts-are-wrong) show where that leads. List each drafted classification separately from the descriptions, least protected first, so the reviewer sees what each draft would leave unprotected.
- **A table created outside the pipelines escapes the test.** A notebook or a one-off backfill creates a table nobody will add to the file. Run [`problems`](#test-the-catalog-against-the-metadata) on a schedule against the production catalog, not only in pull requests.

## What this chapter decided {#catalog-decisions}

- **Metadata is a reviewed file:** owners, domains, descriptions and classifications live in `governance/metadata.toml`, changed only by pull request.
- **Four classifications, ordered:** `public`, `internal`, `personal` and `sensitive`, with the stricter one winning when a column is built from several. A column without one is treated as `sensitive`.
- **One owning team per table,** never a person, and a domain beside it.
- **Columns are keyed by Iceberg field ID,** with the name kept beside it so a rename shows up for review.
- **Descriptions go into Iceberg's `doc`,** written by `lib.metadata.apply`. The owner and domain go into table properties as a pointer, and classifications stay in the file.
- **The catalog is tested against the file:** `problems` walks every table in the catalog, and `dropped` finds entries for columns that are gone. Schema changes are compared by field IDs, names and types, not by schema IDs.
- **`learner_id` is `internal`** in every table, with a description that says it resolves through `crm.learners`. The protection sits on that table, and the access policy decides who may join the two with names in view.
- **`apply` and `problems` run on every change:** `apply` writes the file into the catalog on every merge, and `problems` runs in every pull request and on a schedule against the production catalog.
- **The model that drafts metadata runs on the platform's own machines,** because the sample values it's shown are personal data whenever the column is.
- **Model-drafted metadata is a proposal.** Drafts are recorded with their model, measured against the reviewed file, and rewritten by a person. A model may raise a classification, never lower it. A prompt that names the catalog's own columns flatters the model, so the examples in a prompt come from outside the platform's data. A stale description becomes a failed check.

# Finding personal data

By the end of this chapter you can find personal data on a data platform automatically, and say how much of it you miss. You'll propose a classification for every column from its name, its type and a reproducible sample of its values; find personal data inside free text, span by span; and measure each detector against labelled data before anyone relies on it. Along the way you'll run Presidio's pattern recognizers offline, with thresholds chosen by measurement, compare them with the recorded detections of a model trained to find personal data, and classify again only the columns a schema change adds.

Every column on the platform already has a classification. [Chapter 2](#a-catalog-you-can-test) put one in `governance/metadata.toml`, a file people review, and a reviewed file records what its reviewers believe a column holds. On the platform, the online learning platform whose warehouse this book governs, three of those beliefs have never been checked. `billing.payments.card_number` is a legacy column whose description says it holds full card numbers, and nobody has looked. `crm.learners.notes` is free text that support agents type, so its name says nothing about what's in it. Learners write whatever they like in support tickets and course reviews. No classification of a whole column can protect that text. Before it can be masked, the platform has to find each name, email and phone number in it, span by span, as [chapter 9](#replace-personal-data) does for reviews before the course assistant learns from them.

This chapter builds the book's shared module for the job, `lib.pii`: rules on column names, seeded samples of a column's values, a Presidio detector configured to run offline, and code that scores any detector against labelled data. Every detector in this chapter is measured against one file, which we call the truth: `data/platform/pii_spans.parquet`. The platform's data generator writes it beside the tickets and reviews, with every piece of personal data it placed in their text and its exact start and end. Invoice numbers, order numbers and the date a problem happened stay unlabelled, because the platform doesn't treat them as personal data.

Phone numbers, card numbers and IBANs in this book are random, as [the data card](#governance-end-to-end) says, and no example prints one whole: it masks all but the last four digits.

## Classify columns from the schema {#classify-columns-from-the-schema}

The cheapest evidence about a column is its schema: its name and its type. A rule on names reads no data, so it runs over a whole catalog in a pull request, and it catches the columns whose names say what they hold, such as `email` or `date_of_birth`. Catalog tools start the same way: OpenMetadata's auto-classification begins with a column name scanner that checks names against regular expressions ([OpenMetadata docs](https://docs.open-metadata.org/v2.0.x/how-to-guides/data-governance/classification/auto-classification/auto-pii-tagging)). The type decides what comes next. A text column's values can be sampled and scanned, but a timestamp can't hold an email, and scanning one with a date pattern would flag every row. So numbers, dates, timestamps and booleans are classified by name alone.

The platform keeps its rules in `governance/pii.toml`, which holds every setting of its detector and is reviewed like the metadata. The rules are 8 regular expressions on lowercased column names, each naming the entity it suggests, and the first match wins. The same file maps each entity to a classification: a phone number to `personal`, a card number or an IBAN to `sensitive`. `lib.pii.by_name` applies the rules. The rules go over every column of the catalog, and the results sit beside the reviewed metadata:

Example `column-name-rules` (Python 3.12.15 · duckdb 2.0.0 · phonenumbers 9.0.43 · presidio-analyzer 2.2.366 · pyarrow 26.0.1 · pyiceberg 0.12.0 · pyyaml 6.0.3 · tldextract 5.4.0), passing in Edition 2026.11.31:

```python
"""Runs the name rules over every column of the catalog and compares each column they mark with the reviewed metadata.
The prose rests on the count of marked columns that agree, and on the personal columns no rule marks."""
from pyiceberg.types import StringType

from lib import metadata, pii
from lib.catalog import open_catalog

catalog = open_catalog()
described = metadata.load()                   # the reviewed metadata, keyed by table and then by field ID
# every column as (table, field): the field carries the ID the metadata is keyed by, so a renamed column still matches
columns = [(table, f) for (namespace,) in sorted(catalog.list_namespaces())
           for _, name in sorted(catalog.list_tables(namespace))
           for table in [f"{namespace}.{name}"] for f in catalog.load_table(table).schema().fields]
print("columns in the catalog:", len(columns))
print("name rules:", len(pii.NAME_RULES))

named = [(table, f, pii.by_name(f.name)) for table, f in columns if pii.by_name(f.name)]
print("columns a name rule marks:", len(named))
for table, f, entity in named:
    print(f"  {table}.{f.name}: {entity}, so {pii.CLASSIFICATION[entity]}")
agree = sum(pii.CLASSIFICATION[entity] == described[table].columns[f.field_id].classification
            for table, f, entity in named)
print("of them, agreeing with the metadata:", agree)

# Only text is sampled: numbers, dates, timestamps and booleans are classified by name alone.
text = [(table, f) for table, f in columns if isinstance(f.field_type, StringType)]
print("columns of text, whose values can be sampled:", len(text))

# Columns the metadata, reviewed by people, calls personal or sensitive, that no name rule marks.
missed = [f"{table}.{f.name}" for table, f in columns if not pii.by_name(f.name)
          and described[table].columns[f.field_id].classification in ("personal", "sensitive")]
print("personal columns no name rule marks:", ", ".join(missed))
```

Output:

```text
columns in the catalog: 64
name rules: 8
columns a name rule marks: 11
  billing.payments.card_number: CREDIT_CARD, so sensitive
  crm.instructors.full_name: PERSON, so personal
  crm.instructors.email: EMAIL_ADDRESS, so personal
  crm.instructors.payout_iban: IBAN_CODE, so sensitive
  crm.learners.full_name: PERSON, so personal
  crm.learners.email: EMAIL_ADDRESS, so personal
  crm.learners.phone: PHONE_NUMBER, so personal
  crm.learners.city: LOCATION, so personal
  crm.learners.street_address: LOCATION, so personal
  crm.learners.date_of_birth: DATE_TIME, so personal
  learning.events.ip_address: IP_ADDRESS, so personal
of them, agreeing with the metadata: 11
columns of text, whose values can be sampled: 29
personal columns no name rule marks: crm.learners.notes, learning.reviews.body, support.tickets.body
```

The rules agree with the metadata wherever they fire, and they miss the free text. Of the catalog's 64 columns, the rules mark 11, and all 11 agree with the classification people gave them. They miss three columns the reviewers classified `personal`: `crm.learners.notes`, `learning.reviews.body` and `support.tickets.body`. All three are free text, where a name describes the container, not its contents. The marked column the platform worries about most, `billing.payments.card_number`, gets `sensitive` from its name without anyone learning what it holds. Whole card numbers, the last four digits, or a payment provider's token would all match `card_?(number|num|no)`.

Name rules are brittle in a known way. The authors of Sherlock, a model for detecting column types, report that name matching isn't robust to dirty data and detects only a few types ([Hulsebos et al., 2019](https://arxiv.org/abs/1905.10688)). So the platform reads values too, in the 29 text columns its catalog holds.

**Working rule:** Run name rules over every column first, because they're free, and let them raise a classification, never lower one. Keep the rules short, in code, and reviewed like any other rule. Treat any free-text column, whatever it's called, as one that needs its values sampled, and classify columns of other types by name alone.

## Run Presidio offline {#run-presidio-offline}

To read values, the platform uses Presidio, an open-source library for finding and masking personal data. Presidio's analyzer runs **recognizers**: regular expressions with checksums, phone-number parsing, and a named-entity model reached through an NLP engine. The NLP engine does two jobs, named-entity recognition and feature extraction for rules, such as the context words that raise a score. Its default model is spaCy's `en_core_web_lg` ([Presidio docs](https://github.com/data-privacy-stack/presidio/blob/2.2.366/docs/analyzer/customizing_nlp_models.md)). At 2.2.366, `AnalyzerEngine()` with no arguments downloads that model when it isn't installed ([source](https://github.com/data-privacy-stack/presidio/blob/2.2.366/presidio-analyzer/presidio_analyzer/nlp_engine/spacy_nlp_engine.py)). So a pipeline without network access fails on its first line, and one with access installs a model of about 400 MB as a side effect ([en_core_web_lg 3.8.0](https://github.com/explosion/spacy-models/releases/tag/en_core_web_lg-3.8.0)).

`lib.pii.detector` configures the engine itself, with `NoOpNlpEngine`, which loads no language model ([source](https://github.com/data-privacy-stack/presidio/blob/2.2.366/presidio-analyzer/presidio_analyzer/nlp_engine/no_op_nlp_engine.py)). What that costs is context. With a model, Presidio raises a pattern's score when a word such as "phone" appears near the match, by comparing the lemma of each nearby word with each recognizer's context terms ([Presidio's tutorial on context](https://github.com/data-privacy-stack/presidio/blob/2.2.366/docs/tutorial/06_context.md)). With no model there are no lemmas, so the engine can't use the text's context words at all. The engine has one more way onto the network. Its [email recognizer](https://github.com/data-privacy-stack/presidio/blob/2.2.366/presidio-analyzer/presidio_analyzer/predefined_recognizers/generic/email_recognizer.py) checks domains with tldextract, which fetches the Public Suffix List the first time it's used ([tldextract 5.4.0's README](https://github.com/john-kurkowski/tldextract/blob/5.4.0/README.md)). Importing Presidio imports tldextract, whose README says an empty `TLDEXTRACT_PUBLIC_SUFFIX_LIST_URLS` turns the fetch off if it's set before the import. So `lib/__init__.py`, which Python runs before any module of the book's `lib`, empties the variable. If an example imported Presidio first, the same file replaces tldextract's [shared extractor](https://github.com/john-kurkowski/tldextract/blob/5.4.0/tldextract/tldextract.py) with one that has no URLs. `lib.pii` refuses to load if any remain, and tldextract uses the copy of the list it ships with, so the same text gives the same answer on every machine.

The defaults are tested on real text: load every predefined recognizer on one of the platform's support tickets, which holds a name, an email, a phone number and a date:

Example `presidio-offline-engine` (Python 3.12.15 · duckdb 2.0.0 · phonenumbers 9.0.43 · presidio-analyzer 2.2.366 · pyarrow 26.0.1 · pyiceberg 0.12.0 · pyyaml 6.0.3 · spacy 3.8.18 · tldextract 5.4.0), passing in Edition 2026.11.31:

```python
"""Loads Presidio's predefined recognizers on an engine that loads no language model, and runs them on one support
ticket. The prose's point rests on the detections printed under the ticket, and on the assert that the model isn't
installed."""
import spacy

from lib import pii
from lib.platform import load

# AnalyzerEngine() with no arguments loads spaCy's en_core_web_lg, and downloads it when it isn't installed.
installed = spacy.util.is_package("en_core_web_lg")
assert installed is False       # the pipeline assumes no model: a machine that has one fails here, not silently
print("en_core_web_lg installed:", installed)
print("URLs tldextract would fetch the Public Suffix List from:", len(pii.suffix_list_urls()))

detector = pii.detector()              # every predefined recognizer, on an NLP engine that loads no model
names = sorted(r.name for r in detector.registry.recognizers)
print("recognizers loaded:", len(names))
print("recognizers:", ", ".join(names))
print("default score threshold:", detector.default_score_threshold)

# The ticket holds a phone number, which may be in service: masked() prints it with only its last four digits.
ticket = pii.texts("tickets")[1421]         # one ticket with a name, an email, a phone number and a date of the problem
found = sorted(pii.detect(detector, ticket))
print("ticket 1421:", pii.masked(ticket))
for start, end, entity, score in found:
    print(f"  {entity} on {pii.masked(ticket[start:end])}: {score:.2f}")

# Where is the phone number valid? The recognizer reads it in each of its regions.
phone = next(ticket[start:end] for start, end, entity, _ in found if entity == "PHONE_NUMBER")
[learner] = [row["learner_id"] for row in load("support.tickets").to_pylist() if row["ticket_id"] == 1421]
[country] = [row["country"] for row in load("crm.learners").to_pylist() if row["learner_id"] == learner]
print("the learner's country:", country)
# The recognizer's built-in regions, not the learner's country.
print("default regions where it's valid:",
      ", ".join(r for r in pii.PhoneRecognizer.DEFAULT_SUPPORTED_REGIONS if pii.valid_phone(phone, r)))
```

Output:

```text
en_core_web_lg installed: False
URLs tldextract would fetch the Public Suffix List from: 0
recognizers loaded: 17
recognizers: CreditCardRecognizer, CryptoRecognizer, DateRecognizer, EmailRecognizer, IbanRecognizer, IpRecognizer, MacAddressRecognizer, MedicalLicenseRecognizer, NhsRecognizer, PhoneRecognizer, UrlRecognizer, UsBankRecognizer, UsItinRecognizer, UsLicenseRecognizer, UsPassportRecognizer, UsSsnRecognizer, UuidRecognizer
default score threshold: 0
ticket 1421: I can't open the videos in Introduction to Design Systems, they stop after a few seconds. It happened on 12/03 around 9pm. My colleague Jennifer Abbott (jennifer.abbott@example.net) wants to enrol as well. You can reach me on •••••••0440 after 6pm.
  DATE_TIME on 12/03: 0.10
  EMAIL_ADDRESS on jennifer.abbott@example.net: 1.00
  URL on example.net: 0.50
  PHONE_NUMBER on •••••••0440: 0.40
  US_BANK_NUMBER on •••••••0440: 0.05
  US_DRIVER_LICENSE on •••••••0440: 0.01
the learner's country: US
default regions where it's valid: BR
```

Presidio's defaults on this engine see the email and the phone number, and miss the colleague's name. `en_core_web_lg` isn't installed, so a bare `AnalyzerEngine()` would try to download it, and tldextract has 0 URLs to fetch its list from. The no-model engine loads 17 recognizers, and none of them looks for names, so Jennifer Abbott, the colleague in the ticket, goes unnoticed. Her email is found with a score of 1.00, and its domain is found again as a URL, at 0.50. The learner's phone number, an 11-digit number typed without separators, is found three times: as a phone number at 0.40, a US bank number at 0.05 and a US driver's licence at 0.01. Even the phone number is found by chance. The learner's country is US, but of Presidio's default regions, the US among them, the digits are valid only in BR. The date of the problem, `12/03`, is read as a date too, which says nothing about anyone.

Those scores aren't probabilities. Each pattern carries a constant score, and weak patterns get low ones, as Presidio's tutorial explains for a five-digit zip code ([Presidio docs](https://github.com/data-privacy-stack/presidio/blob/2.2.366/docs/tutorial/06_context.md)). A US bank number is any run of 8 to 17 digits ([supported entities](https://github.com/data-privacy-stack/presidio/blob/2.2.366/docs/supported_entities.md)), and the FAQ admits that a driver's licence "could be any 9-digit number" ([FAQ](https://github.com/data-privacy-stack/presidio/blob/2.2.366/docs/faq.md)). The analyzer's default threshold is 0, so every weak match comes back.

**Working rule:** Configure Presidio's engine explicitly: `NoOpNlpEngine` when you run patterns only, or a pinned model you install yourself when you need its names. Never call `AnalyzerEngine()` bare in a pipeline, and pin anything a recognizer would download, such as the Public Suffix List. Read each score as a constant set for its pattern, never as a probability.

## Sample a column's values the same way every run {#sample-column-values}

Scanning every value of every text column is too slow for any warehouse, so tools sample. OpenMetadata's workflow samples rows and checks them with an entity recognition engine ([OpenMetadata docs](https://docs.open-metadata.org/v2.0.x/how-to-guides/data-governance/classification/auto-classification/auto-pii-tagging)). A sample needs three decisions. **Which rows:** a random sample, not the first rows or a block from a random start. Google's Sensitive Data Protection offers a random-start mode, `RANDOM_START`, which begins at a random row and reads on ([Google Cloud docs](https://cloud.google.com/sensitive-data-protection/docs/inspecting-storage)). On a table ordered by sign-up date, that block is one cohort of learners. **The same rows every run**, so a classification can be reproduced and a change in it means the data changed. And **how many**, from how rare the data you must catch is.

DuckDB's reservoir sample takes a seed, but a seed gives a reproducible sample only when DuckDB runs on one thread, as the [DuckDB 2.0.0 docs](https://duckdb.org/docs/2.0/sql/samples) warn. `lib.pii.sample` sets one thread, filters out NULLs and then draws a seeded reservoir sample.

A rare value needs a bigger sample, and the arithmetic says how much bigger. Call the share of values that hold personal data p. The chance that n sampled values include at least one is

chance = 1 − (1 − p)ⁿ

and the n that gives a 99% chance is ln(0.01) / ln(1 − p). This is a rule of thumb: it treats the draws as independent, which a sample without replacement from thousands of rows nearly is.

Put the formula to work: draw 20 notes from `crm.learners` twice, then compare the chances over every row with those over the notes alone.

Example `seeded-column-sample` (Python 3.12.15 · duckdb 2.0.0 · phonenumbers 9.0.43 · presidio-analyzer 2.2.366 · pyarrow 26.0.1 · pyiceberg 0.12.0 · pyyaml 6.0.3 · tldextract 5.4.0), passing in Edition 2026.11.31:

```python
"""Draws the same twenty notes twice from a seeded reservoir sample, then counts the notes that hold personal data, to see
how many sampled values a rare value needs. The prose rests on the assert that the draws match, the two chances and the sample sizes."""
import math

from lib import pii
from lib.catalog import open_catalog
from lib.warehouse import connect

con = connect(open_catalog(["crm.learners"]))
first = pii.sample(con, "crm.learners", "notes", rows=20)
second = pii.sample(con, "crm.learners", "notes", rows=20)     # the same call: the seed and one thread (set in lib.pii.sample) make the draws match
assert second == first
print("second draw, same notes as the first:", "yes" if second == first else "no")
# Notes can hold a phone number, which may be in service: masked() prints it with only its last four digits.
print("first sampled notes:", " | ".join(pii.masked(note) for note in first[:3]))

# How rare is personal data in the column? Run the detector over every note, once, to know.
detector = pii.platform_detector()
[(rows, notes)] = con.sql("select count(*), count(notes) from crm.learners").fetchall()   # count(notes) skips NULLs
every_note = [note for (note,) in con.sql("select notes from crm.learners where notes is not null").fetchall()]
holding = sum(bool(pii.detect(detector, note)) for note in every_note)
print("learners:", f"{rows:,}")
print("learners with notes:", notes)
print("notes holding personal data the detector finds:", holding)

# The chance that n sampled values hold at least one of them is 1 - (1 - p)^n, for a share p of the values.
for unit, p in (("rows", holding / rows), ("notes", holding / notes)):   # p counted over all learners, then over the notes
    print(f"chance 20 sampled {unit} catch one:", f"{1 - (1 - p) ** 20:.1%}")
    print(f"sampled {unit} needed for a 99% chance:", math.ceil(math.log(0.01) / math.log(1 - p)))   # solves for n
print("values the platform samples per text column:", pii.ROWS)
# The rarest share p that a sample of ROWS values still catches 99% of the time, printed as 1 in a whole number.
print("rarest share that sample catches 99% of the time:", f"1 in {math.floor(1 / (1 - 0.01 ** (1 / pii.ROWS)))}")
```

Output:

```text
second draw, same notes as the first: yes
first sampled notes: VIP: bought a team licence. | Asked about invoices for their employer. | Called about a refund, call back on (•••)•••-1366x•••.
learners: 5,000
learners with notes: 527
notes holding personal data the detector finds: 138
chance 20 sampled rows catch one: 42.9%
sampled rows needed for a 99% chance: 165
chance 20 sampled notes catch one: 99.8%
sampled notes needed for a 99% chance: 16
values the platform samples per text column: 200
rarest share that sample catches 99% of the time: 1 in 43
```

The draws repeat, so a sample can be audited. With the same seed, the second draw returns the same 20 notes, and the third note already holds a phone number, which the output masks.

Of the 5,000 learners, 527 have notes, and 138 of those hold personal data that the platform's detector, [tuned later in this chapter](#tune-the-recognizers), finds. The NULLs dilute the sample, since a NULL can't hold personal data. Twenty sampled rows catch personal data 42.9% of the time, and a 99% chance takes 165 rows. Counted over the notes that exist, twenty catch it 99.8% of the time, and 16 are enough for 99%.

**Working rule:** Sample non-null values, with a fixed seed, on one thread, and keep the seed with the result. Size the sample from the rarest personal data you must catch, with n = ln(0.01) / ln(1 − p) for 99%. The platform samples 200 values per text column, set in `governance/pii.toml`, which catches personal data held by 1 in 43 values or more 99% of the time. For anything rarer, scan the whole column, or classify it by rule.

## Classify columns from their values {#classify-columns-from-their-values}

[The name rules](#classify-columns-from-the-schema) and [a reproducible sample](#sample-column-values) combine into one proposal per column. `lib.pii.classify` takes the strictest classification that either supports: the entity the name suggests, and every entity found in any sampled value. A column with no evidence gets `internal`, because a detector that found nothing has shown nothing, and only a person decides that a column is `public`.

Strictest is a choice. Presidio's own tool for tables, Presidio structured, gives each column one entity, and its default, Most Common, picks the entity that occurs most often ([Presidio docs](https://github.com/data-privacy-stack/presidio/blob/2.2.366/docs/structured/index.md)). That suits choosing how to mask a column. For classifying it, a column that holds one card number in 200 values still holds card numbers.

Proposals for every column of the catalog, from the platform's detector, whose recognizers [a later section](#tune-the-recognizers) chooses, show where each differs from the reviewed metadata:

Example `classify-sampled-columns` (Python 3.12.15 · duckdb 2.0.0 · phonenumbers 9.0.43 · presidio-analyzer 2.2.366 · pyarrow 26.0.1 · pyiceberg 0.12.0 · pyyaml 6.0.3 · tldextract 5.4.0), passing in Edition 2026.11.31:

```python
"""Proposes a classification for every column, the strictest of what its name and its sampled values support, and compares
each proposal with the reviewed metadata. The prose's table rests on the proposed and found lines of each column, and
its counts on the totals printed after them."""
from lib import metadata, pii
from lib.catalog import open_catalog
from lib.warehouse import connect

catalog = open_catalog()
findings = pii.classify(catalog, connect(catalog), pii.platform_detector())    # the platform's tuned detector, not the defaults
print("columns classified:", len(findings))
print("columns sampled:", sum(f.sampled > 0 for f in findings))

holding = [f for f in findings if f.found]          # columns where sampled values hold personal data
print("columns whose sampled values hold personal data:", len(holding))
# A column counts when its values hold an entity other than the one its name suggests (named is None when no rule matches).
print("of them, holding something their name doesn't suggest:", sum(set(f.found) != {f.named} for f in holding))
for f in holding:
    print(f"{f.table}.{f.column}, proposed: {f.classification}")
    print(f"{f.table}.{f.column}, found: " + ", ".join(f"{e} in {n} of {f.sampled}" for e, n in f.found.most_common()))
[cards] = [f for f in findings if f.column == "card_number"]     # the one card_number column in the catalog
# The card recognizer's count: values that match its pattern and pass its Luhn check, out of the values sampled.
print("sampled card numbers the card recognizer accepts:", f"{cards.found['CREDIT_CARD']} of {cards.sampled}")

# Classifications the values raise above what the name alone gives: the floor is used for a name no rule matches.
named_only = [f for f in findings if pii.CLASSIFICATION.get(f.named, pii.FLOOR) != f.classification]
print("columns the values classify more strictly than the name:", ", ".join(f"{f.table}.{f.column}" for f in named_only))

# Compare with the metadata people reviewed. A column the detector found nothing in proposes only the floor, internal.
described = metadata.load()
evidenced = [f for f in findings if f.named or f.found]
print("columns with evidence:", len(evidenced))
print("of them, equal to the metadata:",
      sum(f.classification == described[f.table].columns[f.field_id].classification for f in evidenced))
for f in evidenced:
    reviewed = described[f.table].columns[f.field_id].classification
    if metadata.stricter(f.classification, reviewed) != reviewed:     # the stricter of the two: a lower proposal prints nothing
        print(f"stricter than the metadata, {f.table}.{f.column}: {f.classification}, reviewed as {reviewed}")
```

Output:

```text
columns classified: 64
columns sampled: 29
columns whose sampled values hold personal data: 9
of them, holding something their name doesn't suggest: 5
billing.payments.card_number, proposed: sensitive
billing.payments.card_number, found: CREDIT_CARD in 151 of 200, PHONE_NUMBER in 3 of 200
crm.instructors.email, proposed: personal
crm.instructors.email, found: EMAIL_ADDRESS in 50 of 50
crm.instructors.payout_iban, proposed: sensitive
crm.instructors.payout_iban, found: IBAN_CODE in 50 of 50
crm.learners.email, proposed: personal
crm.learners.email, found: EMAIL_ADDRESS in 200 of 200
crm.learners.phone, proposed: personal
crm.learners.phone, found: PHONE_NUMBER in 175 of 200
crm.learners.notes, proposed: personal
crm.learners.notes, found: PHONE_NUMBER in 29 of 200, EMAIL_ADDRESS in 18 of 200
learning.events.ip_address, proposed: personal
learning.events.ip_address, found: IP_ADDRESS in 200 of 200, PHONE_NUMBER in 39 of 200
learning.reviews.body, proposed: personal
learning.reviews.body, found: EMAIL_ADDRESS in 4 of 200
support.tickets.body, proposed: sensitive
support.tickets.body, found: EMAIL_ADDRESS in 47 of 200, PHONE_NUMBER in 37 of 200, DATE_TIME in 29 of 200, IBAN_CODE in 21 of 200, CREDIT_CARD in 12 of 200
sampled card numbers the card recognizer accepts: 151 of 200
columns the values classify more strictly than the name: crm.learners.notes, learning.reviews.body, support.tickets.body
columns with evidence: 14
of them, equal to the metadata: 13
stricter than the metadata, support.tickets.body: sensitive, reviewed as personal
```

The values settle what the names leave open. Some of the 9 columns whose samples hold personal data hold only what their name says, such as `crm.learners.email`. The other 5 hold something their name doesn't suggest, or have a name that suggests nothing:

| Column | Proposed | Found in the sample |
| --- | --- | --- |
| `billing.payments.card_number` | sensitive | CREDIT_CARD in 151 of 200, PHONE_NUMBER in 3 of 200 |
| `crm.learners.notes` | personal | PHONE_NUMBER in 29 of 200, EMAIL_ADDRESS in 18 of 200 |
| `support.tickets.body` | sensitive | EMAIL_ADDRESS in 47 of 200, PHONE_NUMBER in 37 of 200, DATE_TIME in 29 of 200, IBAN_CODE in 21 of 200, CREDIT_CARD in 12 of 200 |
| `learning.reviews.body` | personal | EMAIL_ADDRESS in 4 of 200 |
| `learning.events.ip_address` | personal | IP_ADDRESS in 200 of 200, PHONE_NUMBER in 39 of 200 |

The legacy card column's description says it "Should not be stored at all", and the sample shows it holds full card numbers, which `team-billing`, the table's owner, can act on. Of the sampled card numbers, 151 of 200 match Presidio's card pattern and pass the Luhn checksum it checks ([source](https://github.com/data-privacy-stack/presidio/blob/2.2.366/presidio-analyzer/presidio_analyzer/predefined_recognizers/generic/credit_card_recognizer.py)), and the rest are formats the pattern doesn't cover ([see where patterns fail](#where-patterns-fail)). The notes are `personal` on their values alone. The ticket text is `sensitive`, because learners paste card numbers and IBANs into tickets, making it the one proposal stricter than the metadata, where people reviewed it as `personal`. Of the 14 columns with any evidence, 13 match the reviewed classification.

Read the evidence, not only the verdict. Some IP addresses are also read as phone numbers. That changes nothing here, because an IP address is `personal` anyway, but a detector's label says what a value looks like, not what it is.

**Working rule:** Propose the strictest classification that the name or any sampled value supports, with `internal` as the floor, and show the evidence with the proposal. A proposal stricter than the reviewed metadata is a finding for the table's owner, who raises the classification or records why not; never let a detector lower one. OpenMetadata skips any column that already carries a PII tag; we recommend sampling every text column on every run instead, because the reviewed tag is the thing being checked.

## Measure a detector against labelled spans {#measure-against-labelled-spans}

A column-level answer is enough to classify `support.tickets.body`, but not to mask it: masking replaces each span of personal data in each text, and a span the detector misses stays readable. To know what a detector misses, compare its spans with [the truth](#finding-personal-data), the labelled spans of every ticket and review. Two matching rules are standard, and `lib.pii` applies both.

**Exact** matching, from the CoNLL-2003 shared task, counts an entity "correct only if it is an exact match of the corresponding entity in the data file" ([Tjong Kim Sang and De Meulder, 2003](https://aclanthology.org/W03-0419.pdf)). **Overlap** matching asks only for one shared character, with a label of its type, which the SemEval-2013 task on drug interactions calls "Type matching" ([Segura-Bedmar et al., 2013](https://aclanthology.org/S13-2056.pdf)). Neither credits a span hidden under another label, so `lib.pii` adds a third measure, **coverage**: whether every character of a labelled span is inside some detection, whatever its label. Exact asks whether the detector labelled the whole span right, overlap whether it noticed the span as what it is, and coverage whether masking its detections hides the span.

`lib.pii.score` computes exact and overlap matching as **recall**, the share of labelled spans of each type that a detector finds with a label of that type, and **precision**, the share of each label's detections that match a labelled span. `lib.pii.coverage` gives the share of each type's spans covered. `score` also counts the texts in which every labelled character falls inside some detection, the texts that masking would leave with nothing personal. The Text Anonymization Benchmark protects a person only when all of their mentions are masked ([Pilán et al., 2022](https://aclanthology.org/2022.cl-4.19.pdf)), and the platform counts whole texts, which is stricter. Recall comes first: Presidio's evaluation guide says it often matters more than precision, because missing personal data is the error to avoid ([Presidio docs](https://github.com/data-privacy-stack/presidio/blob/2.2.366/docs/evaluation/index.md)).

Presidio's defaults, scored against the labelled spans of every ticket and review, give the baseline the rest of the chapter improves on:

Example `measure-pattern-recognizers` (Python 3.12.15 · duckdb 2.0.0 · phonenumbers 9.0.43 · presidio-analyzer 2.2.366 · pyarrow 26.0.1 · pyiceberg 0.12.0 · pyyaml 6.0.3 · tldextract 5.4.0), passing in Edition 2026.11.31:

```python
"""Runs Presidio with its defaults over every ticket and review, and scores its detections against the truth's labelled
spans. The prose rests on the recall and precision lines, the count of texts left with nothing personal, and the two
date lines that test the truth's limits."""
import re

from lib import pii

labelled = pii.truth()
print("texts:", f"{sum(len(pii.texts(source)) for source in pii.SOURCES):,}")
print("labelled spans:", f"{len(labelled):,}")

detector = pii.detector()                 # Presidio's defaults: every recognizer, threshold 0
found = pii.find(detector)
# scoring needs a label of the span's type; coverage doesn't
scores, covered = pii.score(found, labelled), pii.coverage(found, labelled)
print("detections:", f"{len(found):,}")

for entity, (n, exact, overlap) in scores.recall.items():
    print(f"{entity}, labelled: {n:,}")
    print(f"{entity}, recall, exact: {exact / n:.2f}")
    print(f"{entity}, recall, overlap: {overlap / n:.2f}")
    print(f"{entity}, covered, any label: {covered[entity][1] / n:.2f}")    # covered[entity] is (spans, covered), so n is the same count

for label, (n, exact, overlap) in scores.precision.items():     # precision is per label: what each detection was called
    print(f"{label}, detections: {n:,}")
    print(f"{label}, precision, exact: {exact / n:.2f}")

# every bank number and driver's licence detection is a false alarm: none matches a labelled span
assert all(scores.precision[entity][1] == 0 for entity in ("US_BANK_NUMBER", "US_DRIVER_LICENSE"))
print("precision of all detections, exact:", f"{pii.precision_overall(scores):.2f}")
print("texts holding personal data:", f"{scores.texts:,}")
print("texts with every labelled span masked:", f"{scores.masked:,}")

# The truth's limits. The generator builds every text from a few sentence templates, and writes every date of birth
# one way. Presidio's date patterns don't read every way of writing one.
print("sentence templates the texts are built from:", len(pii.templates()))
births = [pii.text_of(s) for s in labelled if s.entity == "DATE_TIME"]    # the truth's dates are the dates of birth: dates of a problem are unlabelled
written_ymd = sum(re.fullmatch(r"\d{4}-\d\d-\d\d", d) is not None for d in births)   # year first, the generator's one form
assert written_ymd == len(births)           # so every date of birth is written this way
print("dates of birth written like 1985-03-12:", written_ymd)
for written in ("1985-03-12", "12 March 1985"):
    text = f"My date of birth on the account is wrong, it should be {written}."     # a sentence in the generator's shape
    print(f"dates found in a date of birth written {written}:", sum(e == "DATE_TIME" for _, _, e, _ in pii.detect(detector, text)))
```

Output:

```text
texts: 5,000
labelled spans: 4,292
detections: 3,902
CREDIT_CARD, labelled: 213
CREDIT_CARD, recall, exact: 0.71
CREDIT_CARD, recall, overlap: 0.71
CREDIT_CARD, covered, any label: 0.89
DATE_TIME, labelled: 196
DATE_TIME, recall, exact: 1.00
DATE_TIME, recall, overlap: 1.00
DATE_TIME, covered, any label: 1.00
EMAIL_ADDRESS, labelled: 543
EMAIL_ADDRESS, recall, exact: 1.00
EMAIL_ADDRESS, recall, overlap: 1.00
EMAIL_ADDRESS, covered, any label: 1.00
IBAN_CODE, labelled: 236
IBAN_CODE, recall, exact: 1.00
IBAN_CODE, recall, overlap: 1.00
IBAN_CODE, covered, any label: 1.00
LOCATION, labelled: 210
LOCATION, recall, exact: 0.00
LOCATION, recall, overlap: 0.00
LOCATION, covered, any label: 0.00
PERSON, labelled: 2,455
PERSON, recall, exact: 0.00
PERSON, recall, overlap: 0.00
PERSON, covered, any label: 0.00
PHONE_NUMBER, labelled: 439
PHONE_NUMBER, recall, exact: 0.78
PHONE_NUMBER, recall, overlap: 0.79
PHONE_NUMBER, covered, any label: 0.91
CREDIT_CARD, detections: 151
CREDIT_CARD, precision, exact: 1.00
DATE_TIME, detections: 421
DATE_TIME, precision, exact: 0.47
EMAIL_ADDRESS, detections: 543
EMAIL_ADDRESS, precision, exact: 1.00
IBAN_CODE, detections: 236
IBAN_CODE, precision, exact: 1.00
MEDICAL_LICENSE, detections: 18
MEDICAL_LICENSE, precision, exact: 0.00
PHONE_NUMBER, detections: 384
PHONE_NUMBER, precision, exact: 0.89
UK_NHS, detections: 8
UK_NHS, precision, exact: 0.00
URL, detections: 751
URL, precision, exact: 0.00
US_BANK_NUMBER, detections: 689
US_BANK_NUMBER, precision, exact: 0.00
US_DRIVER_LICENSE, detections: 683
US_DRIVER_LICENSE, precision, exact: 0.00
US_ITIN, detections: 2
US_ITIN, precision, exact: 0.00
US_PASSPORT, detections: 8
US_PASSPORT, precision, exact: 0.00
US_SSN, detections: 8
US_SSN, precision, exact: 0.00
precision of all detections, exact: 0.38
texts holding personal data: 2,586
texts with every labelled span masked: 425
sentence templates the texts are built from: 30
dates of birth written like 1985-03-12: 196
dates found in a date of birth written 1985-03-12: 1
dates found in a date of birth written 12 March 1985: 0
```

| Entity type | Labelled | Recall, exact | Recall, overlap | Covered |
| --- | --- | --- | --- | --- |
| `PERSON` | 2,455 | 0.00 | 0.00 | 0.00 |
| `EMAIL_ADDRESS` | 543 | 1.00 | 1.00 | 1.00 |
| `PHONE_NUMBER` | 439 | 0.78 | 0.79 | 0.91 |
| `IBAN_CODE` | 236 | 1.00 | 1.00 | 1.00 |
| `CREDIT_CARD` | 213 | 0.71 | 0.71 | 0.89 |
| `LOCATION` | 210 | 0.00 | 0.00 | 0.00 |
| `DATE_TIME` | 196 | 1.00 | 1.00 | 1.00 |

The defaults find every email, IBAN and date of birth, most phone and card numbers, and no names or street addresses at all, which is most of what the tickets hold. Most of what they flag is wrong: 0.38 of all detections match a labelled span. The 689 US bank numbers, 683 driver's licences and 751 URLs are all wrong. Phone, card and invoice numbers, all runs of digits, come out as bank numbers and licences, and pieces of emails and a few addresses come out as URLs. A wrong label still hides the digits, so phone and card numbers are covered more often than they're found.

Masking would leave personal data in most texts that hold it. Only 425 of the 2,586 texts would come out with nothing personal left.

The truth has its own limits, because the text is generated. Every ticket and review is built from 30 sentence templates in `prep/generate.py`, so each [hard case](#where-patterns-fail) is one sentence frame, and all 196 dates of birth are written like `1985-03-12`. The date recall of 1.00 therefore measures one format: in the generator's own sentence, Presidio finds 1 date when the date of birth is written `1985-03-12`, and 0 when it's written `12 March 1985`. A measure on this text tells you how a detector behaves on these shapes, not how it will do on yours.

**Working rule:** Measure every detector against labelled spans before it masks anything: recall per entity type by exact and by overlap matching, both with a label of the right type, coverage whatever the label, precision per label, and the share of texts with nothing personal left. Weigh recall first. Before you trust a detector on your text, label a few hundred real records by hand and measure on those too.

## Tune the recognizers against the truth {#tune-the-recognizers}

[Presidio's defaults](#measure-against-labelled-spans) return every weak match. The FAQ names two remedies, changing the acceptance threshold and removing the recognizers a dataset doesn't need, and it notes a trade-off between false positives and false negatives ([FAQ](https://github.com/data-privacy-stack/presidio/blob/2.2.366/docs/faq.md)). Presidio 2.2.364 added per-recognizer and per-entity score thresholds, with the analyzer's default as the fallback ([changelog](https://github.com/data-privacy-stack/presidio/blob/2.2.366/CHANGELOG.md), [release notes](https://github.com/data-privacy-stack/presidio/releases/tag/2.2.364)). The phone recognizer has one more setting that matters on a platform whose learners live in many countries: the regions whose numbering plans it reads ([source](https://github.com/data-privacy-stack/presidio/blob/2.2.366/presidio-analyzer/presidio_analyzer/predefined_recognizers/generic/phone_recognizer.py)).

`governance/pii.toml` keeps 6 recognizers, for the identifiers the platform's text can hold, each with its own threshold. The phone regions aren't a setting: `lib.pii.countries` reads the countries the learners live in from `crm.learners`, so the regions follow the data. Four settings are scored against the same truth:

Example `tune-pattern-recognizers` (Python 3.12.15 · duckdb 2.0.0 · phonenumbers 9.0.43 · presidio-analyzer 2.2.366 · pyarrow 26.0.1 · pyiceberg 0.12.0 · pyyaml 6.0.3 · tldextract 5.4.0), passing in Edition 2026.11.31:

```python
"""Scores four detector settings against the same truth: Presidio's defaults, one threshold for every recognizer, the kept
recognizers with their own thresholds, and those with phone numbers read in the learners' countries. The prose rests on
the precision, phone recall, coverage and fully masked texts of each setting."""
from lib import pii

labelled = pii.truth()
print("recognizers kept:", len(pii.THRESHOLDS))                  # read from governance/pii.toml
print("recognizers kept, with their thresholds:", ", ".join(f"{name} {t}" for name, t in pii.THRESHOLDS.items()))
print("phone regions, Presidio's default:", ", ".join(pii.PhoneRecognizer.DEFAULT_SUPPORTED_REGIONS))   # Presidio's own list
print("phone regions, the learners' countries:", ", ".join(pii.countries()))   # read from crm.learners

# In the order the prose builds them: the defaults, one threshold, the kept recognizers, then their regions.
settings = {
    "Presidio's defaults": pii.detector(),
    "one threshold of 0.5": pii.detector(default_threshold=0.5),
    "kept recognizers, own thresholds": pii.detector(pii.THRESHOLDS),
    "kept recognizers, learners' phone regions": pii.platform_detector(),
}
for name, detector in settings.items():
    found = pii.find(detector)
    scores, covered = pii.score(found, labelled), pii.coverage(found, labelled)
    n, exact, _ = scores.recall["PHONE_NUMBER"]         # the table reads exact recall, so the overlap count is left out
    print(f"{name}, detections: {len(found):,}")
    print(f"{name}, precision, exact: {pii.precision_overall(scores):.2f}")
    print(f"{name}, phone recall, exact: {exact / n:.2f}")
    # coverage takes a detection under any label
    for entity, short in (("PHONE_NUMBER", "phones"), ("CREDIT_CARD", "cards")):
        print(f"{name}, {short} covered, any label: {covered[entity][1] / covered[entity][0]:.2f}")
    print(f"{name}, texts with every labelled span masked: {scores.masked:,}")
```

Output:

```text
recognizers kept: 6
recognizers kept, with their thresholds: EmailRecognizer 0.5, PhoneRecognizer 0.4, CreditCardRecognizer 0.5, IbanRecognizer 0.5, DateRecognizer 0.5, IpRecognizer 0.5
phone regions, Presidio's default: US, GB, DE, FR, IL, IN, CA, BR
phone regions, the learners' countries: AU, BR, CA, DE, ES, FR, GB, IN, JP, MX, PL, US
Presidio's defaults, detections: 3,902
Presidio's defaults, precision, exact: 0.38
Presidio's defaults, phone recall, exact: 0.78
Presidio's defaults, phones covered, any label: 0.91
Presidio's defaults, cards covered, any label: 0.89
Presidio's defaults, texts with every labelled span masked: 425
one threshold of 0.5, detections: 1,903
one threshold of 0.5, precision, exact: 0.59
one threshold of 0.5, phone recall, exact: 0.00
one threshold of 0.5, phones covered, any label: 0.01
one threshold of 0.5, cards covered, any label: 0.71
one threshold of 0.5, texts with every labelled span masked: 284
kept recognizers, own thresholds, detections: 1,510
kept recognizers, own thresholds, precision, exact: 0.97
kept recognizers, own thresholds, phone recall, exact: 0.78
kept recognizers, own thresholds, phones covered, any label: 0.78
kept recognizers, own thresholds, cards covered, any label: 0.72
kept recognizers, own thresholds, texts with every labelled span masked: 394
kept recognizers, learners' phone regions, detections: 1,539
kept recognizers, learners' phone regions, precision, exact: 0.97
kept recognizers, learners' phone regions, phone recall, exact: 0.84
kept recognizers, learners' phone regions, phones covered, any label: 0.84
kept recognizers, learners' phone regions, cards covered, any label: 0.72
kept recognizers, learners' phone regions, texts with every labelled span masked: 405
```

| Measure | Defaults | One threshold, 0.5 | Kept recognizers | And learners' regions |
| --- | --- | --- | --- | --- |
| Detections | 3,902 | 1,903 | 1,510 | 1,539 |
| Precision, exact | 0.38 | 0.59 | 0.97 | 0.97 |
| Phone recall, exact | 0.78 | 0.00 | 0.78 | 0.84 |
| Phones covered | 0.91 | 0.01 | 0.78 | 0.84 |
| Cards covered | 0.89 | 0.71 | 0.72 | 0.72 |
| Texts fully masked | 425 | 284 | 394 | 405 |

The obvious fix fails. One threshold of 0.5 removes the weak matches and every phone number with them, because a phone number scores 0.40 when no context word can raise it. Keeping only the 6 recognizers, each with its own threshold, and setting the phone threshold at the score a phone number gets, raises precision to 0.97 and leaves phone recall where it was. Reading phone numbers in the learners' regions, rather than Presidio's default regions, raises phone recall to 0.84.

One number goes the wrong way: fewer texts come out fully masked than with the defaults. The dropped recognizers had masked some phone and card numbers under the wrong label, which still hides the digits. Without them, coverage falls from 0.91 to 0.78 for phone numbers, and from 0.89 to 0.72 for card numbers. We think precision is worth the price, because detections are the evidence a reviewer reads, and with the defaults the tickets and reviews hold 689 bank numbers and 683 driver's licences, none of them real. `governance/pii.toml` maps neither entity to a classification, so they change no proposal, but each is a finding someone has to read and dismiss. Dropping the US recognizers is also a claim about the platform's text: the truth holds no US bank or licence numbers. If the platform ever asks for one, that claim, and the measurement, must change.

**Working rule:** Never run with Presidio's default threshold of 0, or with one threshold for every recognizer. Keep only the recognizers for identifiers your text can hold, set each one's threshold from its own scores, and read phone numbers in the regions your users live in, derived from your data. Put all of it in code, and measure again whenever it changes.

## See where patterns fail {#where-patterns-fail}

[The tuned recognizers](#tune-the-recognizers) score well on totals, and totals hide where they fail. A pattern detector fails in three ways: there's no pattern for the thing at all, its pattern or its validation rejects the value, or it matches something that only looks personal. The platform's tickets plant cases of each kind, as its data card says: names typed in lowercase, initials, first names that are ordinary words, phone numbers typed without separators, and invoice numbers, order numbers and dates of a problem, which [the truth](#finding-personal-data) leaves unlabelled. Count each case with the platform's detector. A case counts as caught when some detection overlaps it, whatever its label:

Example `pattern-hard-cases` (Python 3.12.15 · duckdb 2.0.0 · phonenumbers 9.0.43 · presidio-analyzer 2.2.366 · pyarrow 26.0.1 · pyiceberg 0.12.0 · pyyaml 6.0.3 · tldextract 5.4.0), passing in Edition 2026.11.31:

```python
"""Counts each hard case the tickets and reviews plant, and each look-alike number or date the platform's detector flags,
then checks the card and phone numbers it misses. The prose rests on the caught counts for names and invoice numbers,
and on the misses' lengths, first digits and validity."""
from collections import Counter

from lib import pii
from lib.platform import load

labelled = pii.truth()
found = pii.find(pii.platform_detector())

cases = pii.hard_cases(labelled)               # a case is caught when some detection overlaps it, whatever its label
for case, spans in cases.items():
    print(f"{case}, labelled: {len(spans)}")
    print(f"{case}, caught: {pii.caught(spans, found)}")
print("hard cases of a name, caught:", pii.caught([s for spans in cases.values() for s in spans if s.entity == "PERSON"], found))

# Look-alikes are numbers and dates that aren't personal data, so any detection on one is a false alarm.
for kind, spans in pii.lookalikes().items():
    print(f"{kind}, in the tickets: {len(spans)}")
    print(f"{kind}, flagged as personal data: {pii.caught(spans, found)}")

# Card numbers the card pattern misses: too short or long for its 13 to 17 digits, or a first digit it doesn't accept.
# Out of range, the length is the reason; in range, the first digit is, since the pattern accepts only some of them.
missed = pii.missed([s for s in labelled if s.entity == "CREDIT_CARD"], found, {"CREDIT_CARD"})
why = lambda digits: f"{len(digits)} digits" if not 13 <= len(digits) <= 17 else f"{len(digits)} digits starting with {digits[0]}"
print("CREDIT_CARD, missed:", len(missed))
for reason, n in Counter(why(pii.text_of(s)) for s in missed).most_common():
    print(f"CREDIT_CARD, missed, {reason}: {n}")
print("IBAN_CODE, missed:", len(pii.missed([s for s in labelled if s.entity == "IBAN_CODE"], found, {"IBAN_CODE"})))

# Are the phone numbers the patterns miss valid where the ticket's learner lives? Phones appear only in tickets,
# so each phone's learner comes from its ticket.
learner_of = {row["ticket_id"]: row["learner_id"] for row in load("support.tickets").to_pylist()}
country_of = {row["learner_id"]: row["country"] for row in load("crm.learners").to_pylist()}
phones = pii.missed([s for s in labelled if s.entity == "PHONE_NUMBER"], found, {"PHONE_NUMBER"})
print("phone numbers missed:", len(phones))
print("of them, invalid in the learner's country:",
      sum(not pii.valid_phone(pii.text_of(s), country_of[learner_of[s.row_id]]) for s in phones))
```

Output:

```text
names in lowercase, labelled: 211
names in lowercase, caught: 0
initial and surname, labelled: 163
initial and surname, caught: 0
first names that are words, labelled: 187
first names that are words, caught: 0
instructors named in reviews, labelled: 798
instructors named in reviews, caught: 0
phones typed as digits only, labelled: 257
phones typed as digits only, caught: 220
hard cases of a name, caught: 0
invoice numbers, in the tickets: 224
invoice numbers, flagged as personal data: 31
order numbers, in the tickets: 212
order numbers, flagged as personal data: 0
dates of a problem, in the tickets: 225
dates of a problem, flagged as personal data: 0
CREDIT_CARD, missed: 62
CREDIT_CARD, missed, 19 digits: 23
CREDIT_CARD, missed, 12 digits: 22
CREDIT_CARD, missed, 16 digits starting with 2: 10
CREDIT_CARD, missed, 15 digits starting with 2: 7
IBAN_CODE, missed: 0
phone numbers missed: 65
of them, invalid in the learner's country: 63
```

Names are the gap, since no pattern matches a name. Across the names in lowercase, the initials with a surname, the first names that are ordinary words and the instructors named in reviews, the patterns catch 0 of them. Phones typed as digits only do well, 220 of 257. The price is the look-alikes: 31 of the 224 invoice numbers are flagged as personal data.

The platform's detector flags 0 of the 212 order numbers and 0 of the 225 dates of a problem. With [Presidio's defaults](#run-presidio-offline), `12/03` was a date at 0.10, and the date recognizer returned 421 dates where the truth labels 196. The threshold in `governance/pii.toml` keeps such weak matches out.

The card pattern's shape causes the card misses. It accepts 13 to 17 digits starting with 1, 3, 4, 50 to 55 or 6, but no 13-digit number starting with 1 ([source](https://github.com/data-privacy-stack/presidio/blob/2.2.366/presidio-analyzer/presidio_analyzer/predefined_recognizers/generic/credit_card_recognizer.py)), and every card number in the data passes the Luhn checksum, so validation rejects none. Of the 62 card numbers it misses, 23 have 19 digits and 22 have 12, and the rest start with 2, which the pattern doesn't accept.

Validation causes the other misses. The IBAN recognizer checks each country's format, length and check digits ([source](https://github.com/data-privacy-stack/presidio/blob/2.2.366/presidio-analyzer/presidio_analyzer/predefined_recognizers/generic/iban_recognizer.py)). Every labelled IBAN passes, so it misses 0, but a learner who mistypes one digit writes a value it rejects. The phone recognizer accepts only numbers that python-phonenumbers calls valid ([source](https://github.com/daviddrysdale/python-phonenumbers/blob/v9.0.41/python/phonenumbers/phonenumbermatcher.py)), and the generator doesn't always write those. Of the 65 phone numbers the detector misses, 63 aren't valid in the country the ticket's learner lives in, so most of the phone misses are the generator's, not the detector's.

**Working rule:** Read a detector's misses by case, not by total, and keep a list of the cases your text holds. Use patterns for what has a checkable shape, such as emails, IBANs, card numbers, phone numbers and dates, and never for names or addresses. Keep the checksums and numbering plans, because they're what makes the patterns precise. Check the card pattern against the card numbers your payment provider issues, and the date patterns against the ways your users write dates.

## Compare a PII model {#compare-a-named-entity-model}

[The patterns](#where-patterns-fail) have nothing for names or addresses, and those are most of the personal data in the platform's text: 2,455 names and 210 addresses among the 4,292 labelled spans. Finding them takes a model that reads context, not a general named-entity model. The authors of the SPY dataset define PII as the entities that can identify, contact or locate a specific individual, and found that dedicated named-entity models have limits in PII contexts ([Savkin et al., 2025](https://aclanthology.org/2025.naacl-srw.23.pdf)). We use a PII model, one trained for personally identifiable information: OpenAI's Privacy Filter, an Apache-2.0 model with eight fixed labels, such as `private_person`, `private_address` and `account_number` ([model card](https://huggingface.co/openai/privacy-filter/blob/7ffa9a043d54d1be65afb281eddf0ffbe629385b/README.md)).

The labels are the model's own. OpenAI's card says its label policies can't be set while the model runs ([OpenAI, 2026](https://cdn.openai.com/pdf/c66281ed-b638-456a-8ce1-97e9f5264a90/OpenAI-Privacy-Filter-Model-Card.pdf), section 4.2). We chose it over GLiNER2-PII, the runner-up among open PII models, for those fixed labels. GLiNER2-PII takes the labels to look for at inference time, along with a threshold ([model card](https://huggingface.co/fastino/gliner2-privacy-filter-PII-multi/blob/1cb4166094dc58fa8d836429f060d6c95f62b495/README.md)), so a recording of it would have to pin both. Privacy Filter has 1.5 billion parameters, 50 million of them active, and runs on a CPU.

The book's examples never run a model, so they read a recording of its output. The recording, `data/recorded/pii-entities/`, which this chapter and [chapter 9](#replace-personal-data) use, holds every span the model found in every ticket and review, with its offsets, its text and the model's label. `recordings/pii_entities.py` made it on a CPU with the model's reference decoder, `opf` ([code](https://github.com/openai/privacy-filter/tree/f7f00ca7fb869683eb732c010299d901457f19c3)), which chooses the best sequence of labels for a whole text, so most spans come back in one piece rather than token by token. It used the default operating point, with all six transition biases at 0 ([calibration file](https://huggingface.co/openai/privacy-filter/blob/7ffa9a043d54d1be65afb281eddf0ffbe629385b/original/viterbi_calibration.json)); the model can also be set "for improved recall" (section 2.3.2). The manifest pins the model's revision and the SHA-256 of its weights, so anyone can record it again and compare, and `lib.pii.recorded` refuses spans that don't match the text at their offsets. The recording keeps the model's labels, and the example maps them to the truth's types, so a different mapping never needs a new recording. Score the model, the platform's patterns and both together, masking whatever either finds:

Example `privacy-filter-spans` (Python 3.12.15 · duckdb 2.0.0 · phonenumbers 9.0.43 · presidio-analyzer 2.2.366 · pyarrow 26.0.1 · pyiceberg 0.12.0 · pyyaml 6.0.3 · tldextract 5.4.0), passing in Edition 2026.11.31:

```python
"""Scores the recorded spans of OpenAI's Privacy Filter against the truth, alone and beside the platform's patterns. The
prose rests on the recall lines for names, addresses, phones and cards, the precision lines, and the texts left with
nothing personal."""
from lib import pii

# Privacy Filter's eight labels, mapped to the truth's entity types. A label the truth has no type for matches nothing.
MATCHES = {"private_person": {"PERSON"}, "private_email": {"EMAIL_ADDRESS"}, "private_phone": {"PHONE_NUMBER"},
           "private_address": {"LOCATION"}, "account_number": {"CREDIT_CARD", "IBAN_CODE"},
           "private_date": {"DATE_TIME"}, "private_url": set(), "secret": set()}

model, manifest = pii.recorded("pii-entities")      # the model's spans, from a recording: no model runs here
print("model:", manifest["model"], "decoded with", manifest["settings"]["decode_mode"])
labelled = pii.truth()
detectors = {"patterns": pii.find(pii.platform_detector()), "model": model}
# both lists joined: a span either one finds is masked, and a span both find is simply counted twice
detectors["both"] = detectors["patterns"] + model

for name, found in detectors.items():
    scores = pii.score(found, labelled, MATCHES)           # Presidio's labels aren't in MATCHES, so match themselves
    print(f"{name}, detections: {len(found):,}")
    for entity, (n, exact, overlap) in scores.recall.items():
        print(f"{name}, {entity}, recall, exact: {exact / n:.2f}")
        print(f"{name}, {entity}, recall, overlap: {overlap / n:.2f}")
    print(f"{name}, precision, exact: {pii.precision_overall(scores):.2f}")
    print(f"{name}, texts with every labelled span masked: {scores.masked:,} of {scores.texts:,}")

# Precision by the model's own label: a label that's right for one truth type can still be wrong for another
for label, (n, exact, overlap) in pii.score(model, labelled, MATCHES).precision.items():
    print(f"model, {label}, detections: {n:,}")
    print(f"model, {label}, precision, exact: {exact / n:.2f}")
    print(f"model, {label}, precision, overlap: {overlap / n:.2f}")
```

Output:

```text
model: openai/privacy-filter decoded with viterbi
patterns, detections: 1,539
patterns, CREDIT_CARD, recall, exact: 0.71
patterns, CREDIT_CARD, recall, overlap: 0.71
patterns, DATE_TIME, recall, exact: 1.00
patterns, DATE_TIME, recall, overlap: 1.00
patterns, EMAIL_ADDRESS, recall, exact: 1.00
patterns, EMAIL_ADDRESS, recall, overlap: 1.00
patterns, IBAN_CODE, recall, exact: 1.00
patterns, IBAN_CODE, recall, overlap: 1.00
patterns, LOCATION, recall, exact: 0.00
patterns, LOCATION, recall, overlap: 0.00
patterns, PERSON, recall, exact: 0.00
patterns, PERSON, recall, overlap: 0.00
patterns, PHONE_NUMBER, recall, exact: 0.84
patterns, PHONE_NUMBER, recall, overlap: 0.85
patterns, precision, exact: 0.97
patterns, texts with every labelled span masked: 405 of 2,586
model, detections: 4,424
model, CREDIT_CARD, recall, exact: 0.69
model, CREDIT_CARD, recall, overlap: 0.69
model, DATE_TIME, recall, exact: 0.97
model, DATE_TIME, recall, overlap: 0.97
model, EMAIL_ADDRESS, recall, exact: 1.00
model, EMAIL_ADDRESS, recall, overlap: 1.00
model, IBAN_CODE, recall, exact: 0.99
model, IBAN_CODE, recall, overlap: 1.00
model, LOCATION, recall, exact: 0.67
model, LOCATION, recall, overlap: 0.79
model, PERSON, recall, exact: 0.84
model, PERSON, recall, overlap: 0.84
model, PHONE_NUMBER, recall, exact: 0.94
model, PHONE_NUMBER, recall, overlap: 0.94
model, precision, exact: 0.84
model, texts with every labelled span masked: 2,095 of 2,586
both, detections: 5,963
both, CREDIT_CARD, recall, exact: 0.92
both, CREDIT_CARD, recall, overlap: 0.92
both, DATE_TIME, recall, exact: 1.00
both, DATE_TIME, recall, overlap: 1.00
both, EMAIL_ADDRESS, recall, exact: 1.00
both, EMAIL_ADDRESS, recall, overlap: 1.00
both, IBAN_CODE, recall, exact: 1.00
both, IBAN_CODE, recall, overlap: 1.00
both, LOCATION, recall, exact: 0.67
both, LOCATION, recall, overlap: 0.79
both, PERSON, recall, exact: 0.84
both, PERSON, recall, overlap: 0.84
both, PHONE_NUMBER, recall, exact: 0.99
both, PHONE_NUMBER, recall, overlap: 0.99
both, precision, exact: 0.88
both, texts with every labelled span masked: 2,120 of 2,586
model, account_number, detections: 746
model, account_number, precision, exact: 0.51
model, account_number, precision, overlap: 0.51
model, private_address, detections: 177
model, private_address, precision, exact: 0.79
model, private_address, precision, overlap: 1.00
model, private_date, detections: 384
model, private_date, precision, exact: 0.50
model, private_date, precision, overlap: 0.50
model, private_email, detections: 543
model, private_email, precision, exact: 1.00
model, private_email, precision, overlap: 1.00
model, private_person, detections: 2,110
model, private_person, precision, exact: 0.98
model, private_person, precision, overlap: 0.98
model, private_phone, detections: 464
model, private_phone, precision, exact: 0.89
model, private_phone, precision, overlap: 0.89
```

| Measure | Patterns | Model | Both |
| --- | --- | --- | --- |
| Recall, exact, `PERSON` | 0.00 | 0.84 | 0.84 |
| Recall, exact, `LOCATION` | 0.00 | 0.67 | 0.67 |
| Recall, exact, `PHONE_NUMBER` | 0.84 | 0.94 | 0.99 |
| Recall, exact, `CREDIT_CARD` | 0.71 | 0.69 | 0.92 |
| Recall, exact, `IBAN_CODE` | 1.00 | 0.99 | 1.00 |
| Precision, exact | 0.97 | 0.84 | 0.88 |
| Texts fully masked | 405 of 2,586 | 2,095 of 2,586 | 2,120 of 2,586 |

The model finds most of the names and addresses the patterns can't, and the two detectors complement each other on card numbers. The model and the patterns find about the same share of cards, 0.69 and 0.71, but not the same cards, so together they find 0.92. [A later section](#what-a-wrong-label-costs) shows why the model's share is low.

The model pays in precision, 0.84 overall, and only 0.51 for `account_number` and 0.50 for `private_date`. That partly measures a difference in label policy. The truth leaves invoice numbers and dates of a problem unlabelled, and OpenAI's card says appropriate label policies can differ (section 4.2).

Together, the two detectors leave 2,120 of 2,586 texts with nothing personal in them, against 405 of 2,586 for the patterns alone.

### Read the model's misses {#read-the-models-misses}

OpenAI's card says where the model fails: it misses rare names, names from other naming traditions, initials, names wrapped in titles and domain-specific identifiers, and it over-masks public entities, locations and common nouns when the local context is unclear ([OpenAI, 2026](https://cdn.openai.com/pdf/c66281ed-b638-456a-8ce1-97e9f5264a90/OpenAI-Privacy-Filter-Model-Card.pdf), section 4.3). Check that list against [the hard cases](#where-patterns-fail) the tickets plant:

Example `privacy-filter-hard-cases` (Python 3.12.15 · duckdb 2.0.0 · phonenumbers 9.0.43 · presidio-analyzer 2.2.366 · pyarrow 26.0.1 · pyiceberg 0.12.0 · pyyaml 6.0.3 · tldextract 5.4.0), passing in Edition 2026.11.31:

```python
"""Counts how many of each hard case the model's recorded spans catch, and reads the names it misses by case and length.
The prose rests on the missed-name counts and on the first lowercase ticket that both detectors miss."""
from collections import Counter

from lib import pii

model, _ = pii.recorded("pii-entities")
labelled = pii.truth()
cases = pii.hard_cases(labelled)

for case, spans in cases.items():
    print(f"{case}, labelled: {len(spans)}")
    print(f"{case}, caught by the model: {pii.caught(spans, model)}")
for kind, spans in pii.lookalikes().items():
    print(f"{kind}, flagged by the model: {pii.caught(spans, model)}")

# The names the model misses, by case: read them before trusting a recall figure.
missed = pii.missed([s for s in labelled if s.entity == "PERSON"], model, {"private_person"})
print("names the model misses:", len(missed))
for case in ("first names that are words", "instructors named in reviews"):
    often = Counter(pii.text_of(s) for s in pii.missed(cases[case], model, {"private_person"})).most_common(3)
    print(f"{case}, missed most often:", ", ".join(name for name, _ in often))

# A first name on its own says less than a full name: how many of the misses are one word?
print("missed names that are one word:", sum(len(pii.text_of(s).split()) == 1 for s in missed))
print("missed names of two words or more:", sum(len(pii.text_of(s).split()) > 1 for s in missed))
# islower() counts only names typed entirely in lowercase, so a name with a capital letter, such as "Jane", doesn't.
print("of them, typed in lowercase:", sum(len(pii.text_of(s).split()) > 1 and pii.text_of(s).islower() for s in missed))

# A lowercase name that neither the model nor the platform's patterns cover any part of, in the first ticket with one.
# The model's spans and the patterns' together: a name is missed only when both miss it. The lowest ticket ID is taken,
# so the same ticket is printed on every run.
first = min(pii.missed(cases["names in lowercase"], model + pii.find(pii.platform_detector())), key=lambda s: s.row_id)
print("first ticket with a lowercase name both detectors miss:", first.row_id)
print("the name, as typed:", pii.text_of(first))
```

Output:

```text
names in lowercase, labelled: 211
names in lowercase, caught by the model: 167
initial and surname, labelled: 163
initial and surname, caught by the model: 162
first names that are words, labelled: 187
first names that are words, caught by the model: 128
instructors named in reviews, labelled: 798
instructors named in reviews, caught by the model: 535
phones typed as digits only, labelled: 257
phones typed as digits only, caught by the model: 256
invoice numbers, flagged by the model: 224
order numbers, flagged by the model: 125
dates of a problem, flagged by the model: 188
names the model misses: 387
first names that are words, missed most often: Summer, Will, Hope
instructors named in reviews, missed most often: Marie, Lori, Augustin
missed names that are one word: 337
missed names of two words or more: 50
of them, typed in lowercase: 43
first ticket with a lowercase name both detectors miss: 19
the name, as typed: juan fonseca
```

The misses are first names that are also words, and an instructor's first name alone in a review. The model catches 128 of the 187 first names that are words, and 535 of the 798 instructors' names.

Initials aren't the problem: the model catches 162 of 163 initials with a surname, and lowercase names are mostly caught, 167 of 211.

Most often it misses the words Summer, Will, Hope and the instructors' names Marie, Lori, Augustin. The model also flags far more look-alikes than the tuned patterns: 224 invoice numbers, 125 order numbers and 188 dates of a problem, against 31, 0 and 0.

How much the misses matter depends on their length. Of the 387 names the model misses, 337 are a single word, a first name on its own, and 50 are longer, 43 of those typed in lowercase. The authors of the Text Anonymization Benchmark draw the same distinction: missing a direct identifier, such as a full name, is much more harmful than missing a quasi-identifier ([Pilán et al., 2022](https://aclanthology.org/2022.cl-4.19.pdf)). A first name alone still needs masking, since a review carries its learner's ID, and an instructor's first name in a review of their own course points at one person.

The card recommends testing the model in your own domain against local policy references before production use (section 4.5). It also lists education among the settings that need extra caution (section 4.4), and calls the model "a redaction and data minimization aid, not an anonymization, compliance, or a safety guarantee" (section 4.1).

**Working rule:** Mask free text with a PII model and the tuned patterns together, whatever either finds: the model for names and addresses, the patterns' checksums for card numbers and IBANs. Record the model's output with its revision, weights and operating point pinned, keep its own labels, and map them in code. Measure it on your own text, by hard case, before use and after every model change; if it misses too much, try an operating point that leans toward recall and record again. Treat masked text as reduced, never as anonymous.

## Count what a wrong label costs {#what-a-wrong-label-costs}

[The comparison with the patterns](#compare-a-named-entity-model) scored the model by exact and overlap matching, which both need a label of the span's type. Masking doesn't read the label: a card number the model labels `private_phone` is still hidden. Classification does, because `governance/pii.toml` maps each entity type to a classification: a phone number to `personal`, a card number to `sensitive`. Measure the model's coverage beside its recall, and the labels it gives the platform's card numbers, phone numbers and addresses:

Example `privacy-filter-labels` (Python 3.12.15 · duckdb 2.0.0 · phonenumbers 9.0.43 · presidio-analyzer 2.2.366 · pyarrow 26.0.1 · pyiceberg 0.12.0 · pyyaml 6.0.3 · tldextract 5.4.0), passing in Edition 2026.11.31:

```python
"""Measures the model's coverage of card numbers, phone numbers and addresses whatever their label, and counts the labels
that would misclassify them. The prose rests on the coverage lines, the mislabelled counts and the uncovered addresses."""
from lib import pii

# What the model's labels cost. A span is covered when every character of it is inside some detection, whatever its
# label: that's what masking needs. Exact and overlap matching also need a label of the span's type.
model, _ = pii.recorded("pii-entities")
labelled = pii.truth()
covered = pii.coverage(model, labelled)
for entity in ("CREDIT_CARD", "PHONE_NUMBER", "LOCATION"):
    n, fully = covered[entity]
    print(f"{entity}, covered by the model, any label: {fully / n:.2f}")

spans = {entity: [s for s in labelled if s.entity == entity] for entity in ("CREDIT_CARD", "PHONE_NUMBER", "LOCATION")}
print("card numbers labelled private_phone:", pii.caught(spans["CREDIT_CARD"], model, {"private_phone"}))
print("phone numbers labelled account_number:", pii.caught(spans["PHONE_NUMBER"], model, {"account_number"}))

# For each address, the labels of the detections that overlap it, and the characters no detection covers.
on = pii.labels_on(spans["LOCATION"], model)
print("addresses with no detection at all:", sum(not labels for labels in on))
print("addresses read, at least in part, as a person:", sum("private_person" in labels for labels in on))
# A person label and no address label: the address was read as a name, which overlap matching can't credit.
print("of them, with no address label:", sum({"private_person"} <= labels and "private_address" not in labels for labels in on))
left = [pii.uncovered(address, model) for address in spans["LOCATION"]]
print("addresses not fully covered:", sum(bool(chars) for chars in left))
print("of them, with only spaces and punctuation left:", sum(bool(chars) and not any(c.isalnum() for c in chars) for chars in left))
```

Output:

```text
CREDIT_CARD, covered by the model, any label: 0.87
PHONE_NUMBER, covered by the model, any label: 1.00
LOCATION, covered by the model, any label: 0.67
card numbers labelled private_phone: 40
phone numbers labelled account_number: 27
addresses with no detection at all: 24
addresses read, at least in part, as a person: 28
of them, with no address label: 20
addresses not fully covered: 70
of them, with only spaces and punctuation left: 21
```

A wrong label costs classification more than it costs masking. The model covers 0.87 of card numbers, but labels 40 of the 213 as phone numbers, so its recall for cards is only 0.69. If the platform classified columns from the model's labels, a column of those card numbers would be proposed `personal`, not `sensitive`. That's what the patterns add: their Luhn check keeps card numbers labelled as cards.

The mistake also runs the other way, toward the stricter side. The model labels 27 phone numbers `account_number`, which maps to card numbers and IBANs, so its phone coverage is 1.00 and its recall 0.94.

Addresses show what each measure counts. The model's exact recall for them is 0.67 and its overlap recall 0.79, and its coverage is 0.67. The model finds nothing at all in 24 addresses.

The model reads part of 28 addresses as a person, since Faker builds many street names from surnames, as the platform's data card notes. It gives 20 of those no address label at all, so overlap matching misses them too. Other addresses come back in pieces with the right label, such as a street and a town split at a comma. Overlap matching credits those, and coverage doesn't. Of the 70 addresses not fully covered, 21 have every letter and digit masked and only a comma or a space left. Coverage is the measure masking needs, and it's strict.

**Working rule:** Report coverage, whatever the label, beside recall with the right label: masking needs the first, classification the second. Read which labels a detector gives each type of span, and keep a recognizer with a checksum for every identifier whose label decides a classification, such as card numbers.

## Classify again when the schema changes {#reclassify-on-schema-change}

A classification describes a column at one moment, and [chapter 2](#catch-schema-changes) showed how schemas move under it. So the question is when to classify again, not on every schema ID: in PyIceberg, editing a column's description gives the schema a new ID with no new column. What needs classifying is a [field ID](#key-column-metadata-by-field-id), the number Iceberg gives each column, that the metadata has never seen. Iceberg gives every added column a new one, and a renamed column keeps its ID and so its reviewed classification. Values can also change without a schema change, so the samples of text columns need a rerun on a schedule too. Vendors promise this as a service: Databricks says its classification scans incrementally and classifies new data with no manual setup ([Databricks docs](https://docs.databricks.com/aws/en/data-governance/unity-catalog/data-classification)), a claim we don't test.

The support team wants a number to call learners back on for tickets opened by phone, so it adds a column called `callback_number` to `support.tickets`. The change comes in four steps: edit a description, add the column, fill it from the learners' phone numbers, then classify only the field IDs that `governance/metadata.toml` lacks:

Example `reclassify-new-columns` (Python 3.12.15 · duckdb 2.0.0 · phonenumbers 9.0.43 · presidio-analyzer 2.2.366 · pyarrow 26.0.1 · pyiceberg 0.12.0 · pyyaml 6.0.3 · tldextract 5.4.0), passing in Edition 2026.11.31:

```python
"""Edits the tickets table twice, first a column's description and then a new column, and after each change counts the field
IDs the metadata lacks. The prose rests on the schema IDs, the field counts and the proposal for the new column."""
from pyiceberg.types import StringType

from lib import metadata, pii
from lib.catalog import fill_column, open_catalog
from lib.platform import load
from lib.warehouse import connect

catalog = open_catalog(["support.tickets"])
tickets = catalog.load_table("support.tickets")
reviewed = metadata.load()["support.tickets"].columns           # field ID -> the column as people reviewed it
# A field ID the reviewed metadata doesn't have: Iceberg gives each added column a new one.
unclassified = lambda: {f.field_id for f in tickets.schema().fields if f.field_id not in reviewed}
print("schema ID at the start:", tickets.schema().schema_id)

with tickets.update_schema() as update:                         # a doc edit: a new schema, no new column
    update.update_column("body", doc="What the learner wrote, as they wrote it.")
print("schema ID after a doc edit:", tickets.schema().schema_id)
print("fields to classify after the doc edit:", len(unclassified()))

# The support team starts storing the number to call back for tickets opened by phone: the learner's phone number.
with tickets.update_schema() as update:
    update.add_column("callback_number", StringType(), doc="Where to call the learner back.")
phone_of = {row["learner_id"]: row["phone"] for row in load("crm.learners").to_pylist()}
# tickets from other channels have no callback number, so their value is None
fill_column(tickets, "callback_number", lambda row: phone_of[row["learner_id"]] if row["channel"] == "phone" else None)
print("schema ID after adding a column:", tickets.schema().schema_id)
print("fields to classify after adding a column:", len(unclassified()))

# only= keeps the proposals to the field IDs the metadata lacks.
for f in pii.classify(catalog, connect(catalog), pii.platform_detector(), only={"support.tickets": unclassified()}):
    found = ", ".join(f"{entity} in {n} of {f.sampled}" for entity, n in f.found.items())
    print(f"{f.table}.{f.column}, field ID: {f.field_id}")
    print(f"{f.table}.{f.column}, entity its name suggests: {f.named}")
    print(f"{f.table}.{f.column}, found in its sampled values: {found}")
    print(f"{f.table}.{f.column}, proposed classification: {f.classification}")
```

Output:

```text
schema ID at the start: 0
schema ID after a doc edit: 1
fields to classify after the doc edit: 0
schema ID after adding a column: 2
fields to classify after adding a column: 1
support.tickets.callback_number, field ID: 9
support.tickets.callback_number, entity its name suggests: None
support.tickets.callback_number, found in its sampled values: PHONE_NUMBER in 154 of 173
support.tickets.callback_number, proposed classification: personal
```

A doc edit moves the schema to a new ID and leaves 0 fields to classify, since it adds no column. Adding the column moves the schema to 2, with 1 field to classify: field 9.

No name rule matches `callback_number`, since it contains none of the words the phone rule looks for, which is how a name rule misses a column nobody wrote it for. Its values settle it. The sample finds PHONE_NUMBER in 154 of 173, and the proposal is personal.

The pull request that adds the column is where its classification belongs, beside the metadata entry it needs: [chapter 10](#run-every-rule-as-one-job) fails that pull request until the column has one. A proposal from values needs a column that holds values, so this one comes from the scheduled sample.

**Working rule:** Classify every field ID the metadata doesn't have, in the pull request that adds it, and don't let it merge without a reviewed classification. Key the trigger on field IDs, never on schema IDs. Sample every text column again on a schedule too, because values change without a schema change.

## What goes wrong in production {#personal-data-in-production}

A detector's worst failure is a miss, and its output always looks like a list of findings: what it misses never appears in it:

- **Phone numbers read as bank numbers.** With Presidio's defaults, every run of 8 to 17 digits is a US bank number, and so is every phone number typed without separators. The evidence for the ticket table lists 689 bank numbers that aren't there, a report of "bank numbers in tickets" sends someone to the wrong team, and mapping that entity to a classification would raise columns for the wrong reason. Keep only the recognizers your text needs, each with its own threshold, as [the tuning section](#tune-the-recognizers) does, and count detections per label: a label with hundreds of hits and no labelled spans is a false positive.
- **Names in free text missed.** Patterns find no names, and [the model](#read-the-models-misses) misses some lowercase names, first names that are also words, and first names standing alone. Ticket 19 still says "this is juan fonseca, i still can't log in" after both detectors have run, and passes every check that counts detections. You notice only by measuring recall against labelled text, per hard case, and by reading a sample of masked output.
- **Personal data in a column called `notes`.** No name rule marks it, and a reviewer who never read it classifies it `internal`. Sample every text column's values, and treat free-text columns as personal until a sample says otherwise.
- **A column more sensitive than its metadata.** Learners paste card numbers into tickets, so `support.tickets.body` holds `sensitive` data under a `personal` classification. Compare every proposal with the reviewed metadata, and route a stricter one to the table's owner.
- **A detector that reaches the network.** `AnalyzerEngine()` downloads a model, and tldextract fetches the Public Suffix List. Offline, the pipeline fails; online, the same text can be classified differently on two machines. A setting read at import holds only if it's made before anything imports the library. Configure the engine, pin what it would fetch, check that the setting held, and run the detector's tests with the network off.
- **Truth shaped by its generator.** The platform's text comes from 30 sentence templates filled by Faker, which writes every date of birth one way, phone numbers that aren't valid where the learner lives, and street names built from people's names. Read the misses before trusting a recall figure, and measure again on labelled samples of your own text.

## What this chapter decided {#personal-data-decisions}

- **Columns are classified from their name, their type and their values.** Name rules run on every column; text columns are sampled; the strictest evidence wins, with `internal` as the floor, and only a person sets `public`.
- **Samples are reproducible:** 200 non-null values per text column, by a seeded reservoir sample on one DuckDB thread.
- **Presidio runs offline,** on `NoOpNlpEngine`, with 6 recognizers, each with its own threshold, phone regions read from the learners' countries, and tldextract's own copy of the Public Suffix List, checked when `lib.pii` loads.
- **Detectors are measured against labelled spans:** recall per entity type by exact and overlap matching with a label of the right type, coverage whatever the label, precision per label and texts left with nothing personal, with recall weighed first.
- **A proposal stricter than the metadata is a finding:** `support.tickets.body` goes to `team-support` as `sensitive`, and `billing.payments.card_number` to `team-billing`, with the evidence that it holds full card numbers.
- **Free text is masked with two detectors:** OpenAI's Privacy Filter, recorded at a pinned revision and operating point, for names and addresses, and the tuned patterns for card numbers, IBANs and phone numbers, whose checksums keep card numbers labelled as cards. Masking hides whatever either finds; neither is trusted to make text anonymous.
- **No example prints a whole phone number, card number or IBAN:** `lib.pii.masked` keeps the last four digits.
- **Reclassification follows field IDs:** every field the metadata lacks is classified in the pull request that adds it, and text columns are sampled again on a schedule. A detector is only as good as the measurement behind it, and what it misses never appears in its output. The measurement is what tells a team which of its classifications it can rely on, and which it has to check by hand.

# Lineage from pipelines

By the end of this chapter you can capture lineage from your pipelines as OpenLineage run events, with column lineage parsed from their SQL, including the columns that only join, filter and group. You'll use the graph those events describe to say what a column change breaks downstream, and to propose classifications for tables nobody has classified. You'll also have checks for the three ways lineage fails in production: events that never arrive, a job that sends none, and a copy of personal data classified as less than personal.

The online learning platform from [chapter 1](#governance-end-to-end) builds its reporting tables with four SQL pipelines over eight source tables. Each pipeline is one `create table ... as select ...` in `examples/pipelines/`, and the book's shared module `lib.pipelines` runs it on DuckDB and writes the result back to the catalog:

- `course_performance` writes `analytics.course_performance`: enrollments, completions, ratings and revenue per course.
- `learner_activity` writes `analytics.learner_activity`: each learner's events and last activity.
- `marketing_audience` writes `marketing.audience`: the learners marketing may email, with their name and email.
- `review_corpus` writes `ai.review_corpus`: the review text the platform's course assistant is grounded on.

[Chapter 2](#a-catalog-you-can-test)'s metadata check found that two of those outputs, `marketing.audience` and `ai.review_corpus`, have no metadata at all: no owner, and no classification for any column. Yet one holds learners' names and emails, and the other holds the free text of reviews, where [chapter 3](#classify-columns-from-their-values) found names and emails too. Two questions follow, and the catalog can't answer either. Which columns of those tables are personal data? And if the CRM team renames or reformats `crm.learners.email`, what breaks? Both are lineage questions. Lineage is where each column's data came from, and where it goes.

This chapter adds the book's shared module for lineage, `lib.lineage`. It runs each pipeline, tells OpenLineage what the run read and wrote, column by column, and reads the result back as a graph that answers both questions.

## Send a run event from every pipeline {#send-run-events}

OpenLineage, an open specification for lineage, describes it with three objects ([object model](https://openlineage.io/docs/spec/object-model)). A **job** is "a process that consumes or produces Datasets", such as a pipeline. A **run** is one occurrence of a job, identified by a UUID the client generates, and the spec recommends UUIDv7. A **dataset** is a table or file, named by a namespace and a name that together identify it. A run reports itself through **run events**: a START when it begins, then one terminal event, COMPLETE, FAIL or ABORT, and a server treats the events for one run as accumulating ([run cycle](https://openlineage.io/docs/spec/run-cycle)). Lineage comes mostly from the work itself: a backend learns about datasets mainly from information about jobs.

The Python client, openlineage-python 1.55.0, builds events from `RunEvent`, `Run`, `Job`, `InputDataset` and `OutputDataset` in `openlineage.client.event_v2`, and sends them through a transport ([client docs](https://openlineage.io/docs/client/python)). It ships HTTP and Kafka transports for servers, and a file transport "designed mainly for integration testing", which with `append` set writes one JSON event per line to a single file ([configuration](https://openlineage.io/docs/client/python/configuration)). The client computes no lineage. It serializes what the caller passes in and adds only metadata of its own: a run facet with its version, the job's source location when it runs in a Git checkout, and any environment variables you configure it to collect (`emit` in `client/client.py`, [openlineage-python](https://pypi.org/project/openlineage-python/1.54.0/)). This means the lineage that reaches a server is the lineage your code put in the event.

`lib.lineage.run` wraps `lib.pipelines.run`, which runs one of the platform's four SQL pipelines and writes its table to the catalog. It sends START with the tables the SQL reads as inputs, runs the pipeline, and then sends COMPLETE with the output dataset and two facets on it. A **facet** is OpenLineage's name for a block of metadata attached to a run, a job or a dataset, and these two hold the output's schema and its column lineage, which the [next section](#trace-output-columns) builds. If the pipeline raises, it sends FAIL instead, and `lib.lineage.run_all` runs all four pipelines this way, into one events file. The datasets need a namespace, and OpenLineage's [naming conventions](https://openlineage.io/docs/spec/naming) have no entry for an Iceberg table in a local SQL catalog. The nearest is the local file system, named by its path, which would put a folder on your machine into every event. Hence `lib.lineage` uses the placeholder `iceberg://platform`, with the catalog's identifiers, such as `crm.learners`, as names.

The examples need the same bytes every time, yet an event's time and random run ID change by default. Here `lib.lineage` stamps every event with the nominal start of the nightly run on the book's "today", and derives the run ID with the client's `generate_static_uuid`, which, its docstring says, "always produces the same result" for the same time and data: here, the job's name. Run the four pipelines with lineage twice, into two files, and print what the first file's events say, not the raw JSON:

Example `pipeline-run-events` (Python 3.12.15 · duckdb 2.0.0 · openlineage-python 1.55.0 · pyarrow 26.0.1 · pyiceberg 0.12.0 · pyyaml 6.0.3 · sqlglot 30.33.0), passing in Edition 2026.11.31:

```python
"""Runs the four pipelines with lineage twice, into two files, and reads back what the first file's events say. The
prose rests on the counts of events and run IDs, on the run facet the client adds, and on the same-bytes check, which
the example asserts."""

from openlineage.client.uuid import generate_new_uuid, generate_static_uuid

from lib import lineage
from lib.catalog import open_catalog
from lib.platform import scratch

folder = scratch()
catalog = open_catalog()
lineage.run_all(catalog, folder / "first.jsonl")
lineage.run_all(catalog, folder / "second.jsonl")          # the same nightly run, sent again
# lib.lineage fixes the event time and derives the run IDs
assert (folder / "first.jsonl").read_bytes() == (folder / "second.jsonl").read_bytes()
events = lineage.read_events(folder / "first.jsonl")

print("events:", len(events))
for event in events:
    job, state = event["job"]["name"], event["eventType"]
    if state == "START":
        print(f"{job} START, reads:", ", ".join(dataset["name"] for dataset in event["inputs"]))
    else:
        output = event["outputs"][0]
        lineage_fields = output["facets"]["columnLineage"]["fields"]
        print(f"{job} {state}, writes:", output["name"], f"({len(lineage_fields)} columns with lineage)")

print("event times:", ", ".join(sorted({event["eventTime"] for event in events})))
print("distinct run IDs:", len({event["run"]["runId"] for event in events}))
print("run facets the client added:", ", ".join(events[0]["run"]["facets"]))
print("same bytes when sent again:", (folder / "first.jsonl").read_bytes() == (folder / "second.jsonl").read_bytes())
# generate_new_uuid draws a random UUID each time; generate_static_uuid derives one from its arguments.
print("two random run IDs equal:", generate_new_uuid() == generate_new_uuid())
print("two static run IDs equal:", generate_static_uuid(lineage.NIGHTLY, b"marketing_audience")
      == generate_static_uuid(lineage.NIGHTLY, b"marketing_audience"))
```

Output:

```text
events: 8
course_performance START, reads: billing.payments, learning.courses, learning.enrollments, learning.reviews
course_performance COMPLETE, writes: analytics.course_performance (7 columns with lineage)
learner_activity START, reads: crm.learners, learning.events
learner_activity COMPLETE, writes: analytics.learner_activity (6 columns with lineage)
marketing_audience START, reads: crm.learners, learning.courses, learning.enrollments
marketing_audience COMPLETE, writes: marketing.audience (5 columns with lineage)
review_corpus START, reads: learning.courses, learning.reviews
review_corpus COMPLETE, writes: ai.review_corpus (5 columns with lineage)
event times: 2026-10-01T02:00:00+00:00
distinct run IDs: 4
run facets the client added: tags
same bytes when sent again: True
two random run IDs equal: False
two static run IDs equal: True
```

Sending the same nominal run again wrote the same bytes, as random run IDs never would, and the example asserts it. The first file holds 8 events, a START and a COMPLETE for each of 4 runs, all stamped `2026-10-01T02:00:00+00:00`. The `marketing_audience` job's START names the tables its SQL reads, `crm.learners`, `learning.courses` and `learning.enrollments`, and its COMPLETE names `marketing.audience`, with lineage for each column. Yet the events hold something you didn't put there: a run facet named `tags`, carrying the client's own version ([changelog, 1.38.0](https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md)), and every facet's `_producer` URL names that version too (`DEFAULT_PRODUCER` in `client/constants.py`). So the bytes change with each client release, though the lineage doesn't, and a test should compare what events say, as this example's summary does.

A production emitter stamps COMPLETE with the time the run finished. A derived run ID is still worth keeping there, made from the scheduled time, the job's name and the attempt number: an event sent twice then lands in the same run instead of creating a second one, and a retried job gets a run of its own. Production events should also carry the output's Iceberg snapshot ID, in the `version` dataset facet the [object model](https://openlineage.io/docs/spec/object-model) names for it. The code here leaves it out, because PyIceberg 0.12.0 draws each snapshot ID from a random UUID (`_generate_snapshot_id` in its `pyiceberg/table/metadata.py`).

**Working rule:** Send a START and then a COMPLETE or FAIL event from the code that runs each job, so no run can happen without them, and name datasets by catalog identifiers under a namespace you choose, never a path. In tests, use the file transport with a fixed event time and static run IDs, and compare what events say, never their bytes. In production, send through a transport to your lineage server, and derive each run ID from the scheduled time, the job and the attempt, so a resent event never creates a duplicate run.

## Trace each output column to its sources {#trace-output-columns}

A [run event](#send-run-events) says which tables a job read and wrote. Column lineage says which input columns each output column came from, and that is what governance needs: a table that reads `crm.learners` may or may not copy its emails. OpenLineage puts column lineage in a dataset facet, `columnLineage`, on the output: `fields` maps each output column to its `inputFields`, and an optional `dataset` list holds inputs that affect the whole dataset ([facet schema](https://openlineage.io/spec/facets/1-2-0/ColumnLineageDatasetFacet.json)). Since the client computes none of it, something has to parse each job's SQL.

sqlglot, a SQL parser, does it with `sqlglot.lineage.lineage`. Given a statement and the schema of the tables it reads, it returns, for each output column, a tree whose leaves are source columns, and with `column=None` it returns every output column at once ([sqlglot lineage](https://sqlglot.com/sqlglot/lineage.html)). It needs the schema. Without one, `SELECT *` comes back as a single output named `*`, and an unqualified column in a join as a placeholder with no table. `lib.lineage.schema` builds the schema from the catalog's Iceberg metadata, so the parser sees the columns the warehouse has, at sqlglot 30.33.0.

Start with `marketing_audience`, which feeds the list marketing may email. In words, it picks learners who agreed to marketing email and whose account is active, and adds each one's favourite course category, which a CTE computes by ranking categories by the learner's enrollments. Ask sqlglot for the lineage of the pipeline's outputs and of two `course_performance` columns, to see what it leaves out:

Example `sqlglot-misses-row-inputs` (Python 3.12.15 · duckdb 2.0.0 · openlineage-python 1.55.0 · pyarrow 26.0.1 · pyiceberg 0.12.0 · pyyaml 6.0.3 · sqlglot 30.33.0), passing in Edition 2026.11.31:

```python
"""What sqlglot.lineage says the pipeline outputs come from, and what it leaves out. The prose rests on the CTE-traced
source of favourite_category, on the sources of the two course performance columns, and on the assertion that neither
column the WHERE reads reaches any output."""
import sqlglot
from sqlglot import exp

from lib import lineage, pipelines
from lib.catalog import open_catalog

# sqlglot resolves each name against the catalog's columns, so lineage.schema builds them from the table metadata.
schema = lineage.schema(open_catalog(["crm.learners", "learning.courses", "learning.enrollments", "learning.reviews",
                                      "billing.payments"]))
statement = pipelines.sql("marketing_audience")
found = lineage.sqlglot_sources(statement, schema)
for column, sources in found.items():
    print(f"marketing.audience.{column} comes from:", ", ".join(sources))

parsed = sqlglot.parse_one(statement, read="duckdb")
# the WHERE names columns by alias, as in l.status, so each alias maps to its table
tables = {table.alias_or_name: f"{table.db}.{table.name}" for table in parsed.find_all(exp.Table)}
where = parsed.find(exp.Where)
print("the pipeline's WHERE:", where.this.sql(dialect="duckdb"))
in_lineage = {source for sources in found.values() for source in sources}     # every source column of every output
for column in where.find_all(exp.Column):
    source = f"{tables[column.table]}.{column.name}"
    assert source not in in_lineage        # every column the WHERE reads, checked against the lineage above
    print(f"{source} in any column's lineage:", source in in_lineage)

course_performance = lineage.sqlglot_sources(pipelines.sql("course_performance"), schema)
for column in ("completions", "revenue_usd"):    # the two aggregates that carry a FILTER inside them
    print(f"analytics.course_performance.{column} comes from:", ", ".join(course_performance[column]))
```

Output:

```text
marketing.audience.learner_id comes from: crm.learners.learner_id
marketing.audience.full_name comes from: crm.learners.full_name
marketing.audience.email comes from: crm.learners.email
marketing.audience.country comes from: crm.learners.country
marketing.audience.favourite_category comes from: learning.courses.category
the pipeline's WHERE: l.marketing_consent AND l.status = 'active'
crm.learners.marketing_consent in any column's lineage: False
crm.learners.status in any column's lineage: False
analytics.course_performance.completions comes from: learning.enrollments.completed_at, learning.enrollments.enrollment_id
analytics.course_performance.revenue_usd comes from: billing.payments.amount, billing.payments.status
```

The values are right, but what decides which learners are in the table is missing. `email` comes from `crm.learners.email`, and `favourite_category` is traced through the CTE to `learning.courses.category`. The pipeline's `WHERE` is `l.marketing_consent AND l.status = 'active'`, and neither of its columns, `marketing_consent` or `status`, is in any output's lineage. The example asserts both. The consent filter is the reason marketing may hold this table at all, and the lineage doesn't mention it. Nor does it mention the join keys from enrollments and courses.

The second problem is the reverse: an input that only filters is listed as if it were a value. `course_performance` sums only succeeded payments, with `filter (where p.status = 'succeeded')`, and sqlglot says `revenue_usd` comes from `billing.payments.amount, billing.payments.status`: the column summed and the column that only decides which payments count, listed alike. `completions` counts enrollments with `filter (where e.completed_at is not null)`, and comes back the same way, from `learning.enrollments.completed_at, learning.enrollments.enrollment_id`. Both problems follow from how sqlglot builds the tree. It collects the columns in each output's expression, so it sees a filter inside an expression and nothing outside it, and its nodes carry no type to say how an input was used.

**Working rule:** Use sqlglot's `lineage` for the columns each output's value is computed from, and always pass the catalog's schema, generated from the table metadata, never typed. Don't read its answer as everything a column depends on: it misses the columns that join, filter and group, and it doesn't say which of its inputs are values.

## Add the inputs that decide which rows exist {#add-indirect-inputs}

[sqlglot's lineage](#trace-output-columns) lists the columns a value is computed from, and OpenLineage has a place for the rest. Each input field carries a list of transformations, each with a type, a subtype and a `masking` flag ([column lineage facet](https://openlineage.io/docs/spec/facets/dataset-facets/column_lineage_facet)). DIRECT means the output value is derived from the input value, through IDENTITY, TRANSFORMATION or AGGREGATION. INDIRECT means the output value is impacted by an input without being derived from it, through JOIN, GROUP_BY, FILTER, SORT, WINDOW or CONDITIONAL. Inputs that shape every row, such as a `WHERE`, belong in the facet's `dataset` list.

For governance, the INDIRECT side matters, since information flows through conditions as well as values. Denning calls the information that flows through conditions "implicit flow": a statement that assigns only when a condition holds still tells you the condition's value ([Denning, 1976](https://doi.org/10.1145/360051.360056)). A row in `marketing.audience` tells you that learner consented.

`lib.lineage.column_lineage` adds two things sqlglot's answer lacks: a type for each value input, and the INDIRECT inputs. It types each value input by where the column sits in the output's expression: a bare column is IDENTITY, a column inside an aggregate is AGGREGATION, and anything else is TRANSFORMATION. A column inside an aggregate's `FILTER`, a window's `PARTITION BY` or `ORDER BY`, or a `CASE`'s condition or operand is INDIRECT, and a hash on the way sets `masking`. `COALESCE`'s arguments count as values: OpenLineage lists `COALESCE` under CONDITIONAL, but any of its arguments can become the output's value. Then it walks every `SELECT` in the statement, CTEs included, and collects the columns of its row clauses, following CTE columns back to the tables they come from:

```python
ROW_CLAUSES = {"joins": "JOIN", "where": "FILTER", "group": "GROUP_BY", "having": "FILTER", "qualify": "FILTER",
               "order": "SORT"}
```

Run it on the same pipeline and the same two columns:

Example `indirect-column-lineage` (Python 3.12.15 · duckdb 2.0.0 · openlineage-python 1.55.0 · pyarrow 26.0.1 · pyiceberg 0.12.0 · pyyaml 6.0.3 · sqlglot 30.33.0), passing in Edition 2026.11.31:

```python
"""The book's column lineage for the marketing audience pipeline and two course performance columns: each input,
with its OpenLineage type and subtype, then the inputs that shape every row. The prose rests on the source counts."""
from lib import lineage, pipelines
from lib.catalog import open_catalog

schema = lineage.schema(open_catalog(["crm.learners", "learning.courses", "learning.enrollments", "learning.reviews",
                                      "billing.payments"]))
# rows holds the inputs that shape every row, which OpenLineage keeps in the dataset list
fields, rows = lineage.column_lineage(pipelines.sql("marketing_audience"), schema)
for column, inputs in fields.items():
    print(f"marketing.audience.{column}:", ", ".join(f"{dependency.source} {dependency.type} {dependency.subtype}"
                                                    for dependency in inputs))
print("inputs that shape every row of marketing.audience:")
for source in sorted({dependency.source for dependency in rows}):
    print(f"  {source}:", "INDIRECT",
          ", ".join(dependency.subtype for dependency in rows if dependency.source == source))

# the source columns that the output values are computed from
values = {dependency.source for inputs in fields.values() for dependency in inputs}
print("source columns, values only:", len(values))
# A column can both give a value and shape the rows, so the union counts it once.
print("source columns, with the rows' inputs:", len(values | {dependency.source for dependency in rows}))

fields, _ = lineage.column_lineage(pipelines.sql("course_performance"), schema)   # only these two columns are printed
for column in ("completions", "revenue_usd"):
    print(f"analytics.course_performance.{column}:",
          ", ".join(f"{dependency.source} {dependency.type} {dependency.subtype}" for dependency in fields[column]))
```

Output:

```text
marketing.audience.learner_id: crm.learners.learner_id DIRECT IDENTITY
marketing.audience.full_name: crm.learners.full_name DIRECT IDENTITY
marketing.audience.email: crm.learners.email DIRECT IDENTITY
marketing.audience.country: crm.learners.country DIRECT IDENTITY
marketing.audience.favourite_category: learning.courses.category DIRECT IDENTITY
inputs that shape every row of marketing.audience:
  crm.learners.learner_id: INDIRECT JOIN
  crm.learners.marketing_consent: INDIRECT FILTER
  crm.learners.status: INDIRECT FILTER
  learning.courses.category: INDIRECT GROUP_BY, JOIN
  learning.courses.course_id: INDIRECT JOIN
  learning.enrollments.course_id: INDIRECT JOIN
  learning.enrollments.learner_id: INDIRECT GROUP_BY, JOIN
source columns, values only: 5
source columns, with the rows' inputs: 10
analytics.course_performance.completions: learning.enrollments.completed_at INDIRECT FILTER, learning.enrollments.enrollment_id DIRECT AGGREGATION
analytics.course_performance.revenue_usd: billing.payments.amount DIRECT AGGREGATION, billing.payments.status INDIRECT FILTER
```

The values match [sqlglot's](#trace-output-columns), each an IDENTITY copy, and the rows' inputs are the new part. Adding them grows what `marketing.audience` depends on from 5 source columns to 10. Some row inputs are columns no value uses: `marketing_consent` and `status` as FILTER, the two course IDs as JOIN, and the enrollments' `learner_id` as GROUP_BY and JOIN. `crm.learners.learner_id` and `learning.courses.category`, which values copy too, also shape rows: the ID as JOIN, the category as GROUP_BY and JOIN. For `revenue_usd`, the two inputs are now told apart: `amount` is a DIRECT AGGREGATION, and `status` an INDIRECT FILTER. So are the two inputs of `completions`: `enrollment_id` is counted, and `completed_at` only filters.

One more detail: the per-field `transformationType` and `transformationDescription` are deprecated, replaced by the per-input `transformations` that `lib.lineage` writes ([changelog, 1.17.1](https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md)). And OpenLineage 1.53.0 added a Lineage dataset facet for declared lineage, such as a view's, sent on a `DatasetEvent`, which consumers should prefer where both describe the same relationship ([lineage facet](https://openlineage.io/docs/spec/facets/dataset-facets/lineage)). Run events from jobs carry `columnLineage`, as the platform's do.

**Working rule:** Record INDIRECT inputs with every job's column lineage: the join keys, filters and grouping columns of every `SELECT`, CTEs included, in the facet's `dataset` list, and type each value input with OpenLineage's subtypes. Expect more edges, many of them join keys. Keep them anyway: a filter on a consent column decides who may be emailed, and it reaches the table only as an INDIRECT input.

## Fail on statements the parser can't read {#fail-on-unreadable-lineage}

The platform's [column lineage](#add-indirect-inputs) is only as good as the parse behind it, and a parse can be wrong without failing. sqlglot's `lineage` accepts more than a `SELECT`: it finds the query inside a `CREATE TABLE ... AS`, an `INSERT ... SELECT` or a `CREATE VIEW`, and it reads a `MERGE` too ([sqlglot](https://pypi.org/project/sqlglot/30.24.0/), `optimizer/scope.py`). Three cases come back looking fine. An `INSERT`'s outputs are named after its `SELECT`, never after the table's columns. A `MERGE` comes back as the last query sqlglot reaches in it, as if that query were what the `MERGE` writes. When `USING` names a table or a CTE and no subquery follows, sqlglot returns no lineage and no error. When `USING` is a subquery, or a subquery sits in a `WHEN` condition or a `SET`, it returns the columns of the last such subquery, whether the `MERGE` writes them or not. And a misspelt column gets through when it's unqualified in a query on one table: sqlglot assigns it to that table whether the table has the column or not.

All four of the platform's pipelines are `CREATE TABLE ... AS` statements. As tables grow, pipelines like these often become incremental: an `INSERT` that appends a day's revenue to `analytics.daily_revenue`, or a `MERGE` that updates `analytics.learner_activity` in place. Ask sqlglot and `lib.lineage.column_lineage` about each, the `INSERT` and the `MERGE` in two forms, and compare the answers:

Example `insert-and-merge-lineage` (Python 3.12.15 · duckdb 2.0.0 · openlineage-python 1.55.0 · pyarrow 26.0.1 · pyiceberg 0.12.0 · pyyaml 6.0.3 · sqlglot 30.33.0), passing in Edition 2026.11.31:

```python
"""sqlglot.lineage's answers for an INSERT and for two MERGE statements, beside the book's lineage: sqlglot names the
INSERT's outputs after its SELECT, and returns nothing or a subquery's columns for a MERGE, which the book refuses. The
prose rests on the sqlglot outputs and on the refusals at the end."""
import pyarrow as pa
from sqlglot.lineage import lineage as sqlglot_lineage

from lib import lineage
from lib.catalog import open_catalog

catalog = open_catalog(["billing.payments", "crm.learners"])
catalog.create_namespace("analytics")             # the incremental table, as it exists before the day's INSERT
catalog.create_table("analytics.daily_revenue", schema=pa.schema([("day", pa.date32()), ("revenue_usd", pa.float64())]))
schema = lineage.schema(catalog)
insert = """insert into analytics.daily_revenue
select cast(p.paid_at as date), sum(p.amount) from billing.payments p where p.status = 'succeeded' group by 1"""
merges = {"USING a table": """merge into analytics.learner_activity a using crm.learners l
on a.learner_id = l.learner_id when matched then update set country = l.country""",
          "USING a subquery": """merge into analytics.learner_activity a
using (select l.learner_id, l.country, l.email from crm.learners l) s on a.learner_id = s.learner_id
when not matched then insert (learner_id, country) values (s.learner_id, s.country)"""}

print("analytics.daily_revenue columns:", ", ".join(schema["analytics"]["daily_revenue"]))
# The SELECT's expressions have no aliases, so sqlglot names each output after the SELECT, not the table. None asks for
# every output column at once.
print("sqlglot, INSERT outputs:", ", ".join(sqlglot_lineage(None, insert, schema, dialect="duckdb")))
fields, rows = lineage.column_lineage(insert, schema)
for column, inputs in fields.items():
    print(f"book, INSERT {column}:", ", ".join(f"{dependency.source} {dependency.type} {dependency.subtype}"
                                               for dependency in inputs))
print("book, INSERT rows shaped by:", ", ".join(f"{dependency.source} {dependency.subtype}" for dependency in rows))
for label, merge in merges.items():
    outputs = sqlglot_lineage(None, merge, schema, dialect="duckdb")
    print(f"sqlglot, MERGE {label}, outputs:", ", ".join(outputs) or "none")

# One value too many for daily_revenue's two columns, caught after sqlglot parses it; each MERGE is refused before sqlglot is asked.
refused = {"INSERT with one value too many": insert.replace("sum(p.amount)", "sum(p.amount), count(*)"),
           **{f"MERGE {label}": merge for label, merge in merges.items()}}
for label, statement in refused.items():
    try:
        lineage.column_lineage(statement, schema)
    except lineage.LineageError as error:
        print(f"book, {label}:", error)
```

Output:

```text
analytics.daily_revenue columns: day, revenue_usd
sqlglot, INSERT outputs: paid_at, _col_1
book, INSERT day: billing.payments.paid_at DIRECT TRANSFORMATION
book, INSERT revenue_usd: billing.payments.amount DIRECT AGGREGATION
book, INSERT rows shaped by: billing.payments.paid_at GROUP_BY, billing.payments.status FILTER
sqlglot, MERGE USING a table, outputs: none
sqlglot, MERGE USING a subquery, outputs: learner_id, country, email
book, INSERT with one value too many: analytics.daily_revenue: the INSERT writes 2 columns, and its SELECT has 3 values
book, MERGE USING a table: analytics.learner_activity: lineage is read only from CREATE ... AS and INSERT ... SELECT, not from MERGE
book, MERGE USING a subquery: analytics.learner_activity: lineage is read only from CREATE ... AS and INSERT ... SELECT, not from MERGE
```

The table `analytics.daily_revenue` has the columns `day, revenue_usd`, yet sqlglot names the `INSERT`'s outputs `paid_at, _col_1`: the first after the column inside the `cast`, the second a generated name for an expression with no alias.

An `INSERT` fills columns by position, so `lib.lineage` matches by position too, taking the names from the catalog: the columns the `INSERT` lists, or else all of the table's. So `day` comes from `billing.payments.paid_at DIRECT TRANSFORMATION`, and the `INSERT`'s grouping and its filter on `status` shape every row: `billing.payments.paid_at GROUP_BY, billing.payments.status FILTER`. When the `SELECT` has more or fewer values than the columns written, `lib.lineage` raises: `analytics.daily_revenue: the INSERT writes 2 columns, and its SELECT has 3 values`.

For the `MERGE` whose `USING` names a table, sqlglot returns none, and no error. For the one whose `USING` is a subquery, it returns `learner_id, country, email`, though that `MERGE` writes only `learner_id` and `country`. A check for empty lineage catches the first and passes the second, so `lib.lineage` doesn't judge what sqlglot returns for a `MERGE`. Before it calls sqlglot, it refuses every statement that isn't a `CREATE ... AS` or an `INSERT ... SELECT`: `analytics.learner_activity: lineage is read only from CREATE ... AS and INSERT ... SELECT, not from MERGE`.

A misspelt column, the third case, depends on how the column is written. The platform's pipelines qualify every column with its table's alias, as in `l.email`, and sqlglot checks a qualified column against the schema itself:

Example `misspelt-column-lineage` (Python 3.12.15 · duckdb 2.0.0 · openlineage-python 1.55.0 · pyarrow 26.0.1 · pyiceberg 0.12.0 · pyyaml 6.0.3 · sqlglot 30.33.0), passing in Edition 2026.11.31:

```python
"""A misspelt column, emial for email, in a query on one table: sqlglot raises when the column is qualified by its
alias, and traces it to the table when it isn't. The prose rests on the sqlglot answer for each form, and on the book's
check against the schema."""
from sqlglot.errors import OptimizeError

from lib import lineage
from lib.catalog import open_catalog

schema = lineage.schema(open_catalog(["crm.learners"]))    # the schema lets sqlglot check each column name
# the alias l is right, and only the column name is misspelt
qualified = "create table marketing.audience as select l.learner_id, l.emial from crm.learners l"
# no alias, so nothing names the table the column comes from
unqualified = "create table marketing.audience as select learner_id, emial from crm.learners"

try:
    lineage.sqlglot_sources(qualified, schema)
except OptimizeError as error:
    print("sqlglot, qualified l.emial:", f"{type(error).__name__}: {error}")
print("sqlglot, unqualified emial comes from:", ", ".join(lineage.sqlglot_sources(unqualified, schema)["emial"]))
try:
    lineage.column_lineage(unqualified, schema)
except lineage.LineageError as error:
    print("book, unqualified emial:", error)
```

Output:

```text
sqlglot, qualified l.emial: OptimizeError: Unknown column: emial
sqlglot, unqualified emial comes from: crm.learners.emial
book, unqualified emial: crm.learners has no column emial
```

A qualified typo stops sqlglot with `OptimizeError: Unknown column: emial`. An unqualified one, in a query on one table, comes back as `crm.learners.emial`: a confident answer about a column that doesn't exist. In a join, a misspelt unqualified column belongs to none of the tables, and sqlglot returns a placeholder with no table ([sqlglot](https://pypi.org/project/sqlglot/30.24.0/), `lineage.py`), which `lib.lineage` rejects as a source it can't place. For the single-table case, `lib.lineage` checks every source against the schema and raises `crm.learners has no column emial`.

`lib.lineage.record`, which `run` uses, reads the lineage before the job runs, so a pipeline whose lineage can't be read fails before it writes anything. That is strict, and deliberate: a job with empty lineage would vanish from every impact report and classification check in this chapter, and one with wrong lineage would mislead them. Run the same check in CI on every pull request that changes a pipeline, so the failure shows up in review rather than at night.

**Working rule:** Treat a statement your parser can't read correctly as an error, never as empty or partial lineage. Refuse every kind of statement it gets wrong, such as `MERGE`, before you ask it, and fail on empty lineage, a source the parser can't place and a column the schema doesn't have. Match an `INSERT`'s columns by position against the table's columns in the catalog, never the `SELECT`'s names, and fail when the counts differ. For a `MERGE`, write a `SELECT` of the columns it writes, and record the job's lineage from that, as an `INSERT ... SELECT` into the `MERGE`'s table, until your parser handles `MERGE` itself.

## Find what a column change breaks {#impact-of-a-column-change}

With [column lineage](#add-indirect-inputs) in every COMPLETE event, the platform has a graph: columns are nodes, and each input field is an edge to the column or table it feeds. **Impact analysis** asks what a change to one column can break. It walks that graph from the column, following every edge out of it and continuing from wherever each lands, at any depth. No special method is needed; it's graph reachability. One rule is specific to lineage. An INDIRECT edge into a table means every row of it can change, so the walk continues from all of that table's columns.

`lib.lineage.graph` builds the graph from the events file alone, as a lineage server would, not from the SQL. Thus it includes every job that sends events, whatever it's written in, and nothing that doesn't. `lib.lineage.downstream` does the walk.

Take three changes the platform's teams might make. The CRM team lowercases every email, or renames `crm.learners.email`. A new sign-up flow changes what `crm.learners.marketing_consent` means for older accounts. The learning team renames a category in `learning.courses`. Make a guess first: how many columns can the consent change reach?

Example `column-impact` (Python 3.12.15 · duckdb 2.0.0 · openlineage-python 1.55.0 · pyarrow 26.0.1 · pyiceberg 0.12.0 · pyyaml 6.0.3 · sqlglot 30.33.0), passing in Edition 2026.11.31:

```python
"""What a change to one source column can break, walked over the column graph that the nightly run's events describe.
The prose rests on the reach counts printed for each column: through values, and in all."""

from lib import lineage
from lib.catalog import open_catalog
from lib.platform import scratch

events_file = scratch() / "events.jsonl"
lineage.run_all(open_catalog(), events_file)
graph = lineage.graph(lineage.read_events(events_file))    # built from the events alone, as a lineage server would
print("edges in the column graph:", len(graph.edges))

for column in ("crm.learners.email", "crm.learners.marketing_consent", "learning.courses.category"):
    reached = lineage.downstream(graph, column)
    print(f"{column} reaches:")
    # dict.fromkeys drops repeated targets and keeps the order they were found in
    for target in dict.fromkeys(edge.target for edge in reached):
        how = ", ".join(f"{edge.type} {edge.subtype}" for edge in reached if edge.target == target)
        # a table's name has one dot and a column's two, so a table target means every row of it
        print(f"  {target}{', every row' if target.count('.') == 1 else ''}:", how)
    values = [edge for edge in reached if edge.type == "DIRECT"]    # only DIRECT edges copy values
    print(f"{column}, columns reached through values:", len(lineage.columns_reached(graph, values)))
    print(f"{column}, columns reached in all:", len(lineage.columns_reached(graph, reached)))
```

Output:

```text
edges in the column graph: 51
crm.learners.email reaches:
  marketing.audience.email: DIRECT IDENTITY
crm.learners.email, columns reached through values: 1
crm.learners.email, columns reached in all: 1
crm.learners.marketing_consent reaches:
  marketing.audience, every row: INDIRECT FILTER
crm.learners.marketing_consent, columns reached through values: 0
crm.learners.marketing_consent, columns reached in all: 5
learning.courses.category reaches:
  analytics.course_performance, every row: INDIRECT GROUP_BY
  analytics.course_performance.category: DIRECT IDENTITY
  marketing.audience, every row: INDIRECT GROUP_BY, INDIRECT JOIN
  marketing.audience.favourite_category: DIRECT IDENTITY
learning.courses.category, columns reached through values: 2
learning.courses.category, columns reached in all: 12
```

Consent is the surprise. A change to it reaches 0 columns through values, yet through its filter it reaches all 5 columns of `marketing.audience`, because it changes who gets emailed. A values-only lineage tool would report a change to consent as having no impact.

A change to `email` reaches 1 column, `marketing.audience.email`, so renaming it breaks the marketing pipeline and no other job that sends lineage. A category rename reaches 2 columns through values, and 12 in all, because categories also group `analytics.course_performance` and rank each learner's favourite.

The same walk serves two later chapters: [chapter 7](#plan-erasure-from-lineage) follows a learner's columns downstream to find every derived table an erasure must reach, and [chapter 10](#report-what-a-schema-change-reaches) writes an impact report for every pull request that changes a schema.

**Working rule:** Run impact analysis on the graph built from events, INDIRECT edges included, and report two lists: columns whose values can change, and tables whose rows can change. Treat a change to a column used only in filters and joins as a change to every column of the tables it filters, and send the report to the owners of each table it reaches.

## Propose classifications from lineage {#classifications-from-lineage}

A derived column holds data from its inputs, so its classification can be derived from theirs. The classic rule comes from Denning's lattice model of information flow: the class of a value computed from several inputs is the least upper bound, the "join", of their classes ([Denning, 1976](https://doi.org/10.1145/360051.360056)). The platform's classifications form a line, `public`, `internal`, `personal` and `sensitive`, so the join is the stricter of them, which `stricter` in `lib.metadata`, the book's metadata module, returns. Which inputs count is a judgement. We count INDIRECT inputs too, following Denning's implicit flows, because a table filtered on consent reveals each learner's consent. That choice lives in `governance/lineage.toml`, so a team can change it in one reviewed line.

Two more choices follow from the same caution. OpenLineage's `masking` flag says only that a value was obfuscated, such as by a hash ([column lineage facet](https://openlineage.io/docs/spec/facets/dataset-facets/column_lineage_facet)). Whether a hashed identifier is still personal data is a legal question: [chapter 2](#classify-pseudonymous-ids) quotes the GDPR recital on pseudonymized data and leaves the answer to lawyers. So the book's rule never lowers a classification because of masking. Catalog servers leave the choice to configuration too. Apache Atlas propagates classifications along lineage, and can block that on an edge, for example where a view masks a PII column ([Atlas](https://atlas.apache.org/1.0.0/ClassificationPropagation.html)). DataHub propagates glossary terms along column lineage, a feature its 1.7.0 documentation calls a public beta in DataHub Cloud ([DataHub](https://docs.datahub.com/docs/automations/glossary-term-propagation)). A proposal is a floor, not a verdict: like the drafts a local model writes in [chapter 2](#draft-descriptions-with-a-local-model), it's reviewed before it goes into `governance/metadata.toml`, the platform's file of owners and classifications.

`lib.lineage.propose` proposes a classification for every column a job wrote, and `lib.metadata.problems` lists the tables that need one. Check the rule on the two tables people did classify, `analytics.course_performance` and `analytics.learner_activity`, before you trust it on `marketing.audience` and `ai.review_corpus`, the two with no metadata:

Example `propagate-classifications` (Python 3.12.15 · duckdb 2.0.0 · openlineage-python 1.55.0 · pyarrow 26.0.1 · pyiceberg 0.12.0 · pyyaml 6.0.3 · sqlglot 30.33.0), passing in Edition 2026.11.31:

```python
"""Proposes classifications from lineage for the columns with no declared one, then checks the rule against the columns
people did classify. The prose rests on the personal count, on the reproduced counts, and on the check that no declared class is lower."""

from lib import lineage, metadata
from lib.catalog import open_catalog
from lib.platform import scratch

catalog = open_catalog()
events_file = scratch() / "events.jsonl"
lineage.run_all(catalog, events_file)
for problem in metadata.problems(catalog, metadata.load()):
    print("metadata problem:", problem)

graph, known = lineage.graph(lineage.read_events(events_file)), lineage.declared()   # known: governance/metadata.toml
print("propagated through:", " and ".join(lineage.rules()["propagate_through"]))      # from governance/lineage.toml
proposed = lineage.propose(graph, known)
for column, (classification, decided_by) in proposed.items():
    if column not in known:
        first = decided_by[0]      # lib sorts the deciding edges DIRECT first, so this is a value when one decides
        print(f"{column}: {classification}, from {first.source} ({first.type} {first.subtype})")
print("personal columns proposed:", sum(1 for column, (classification, _) in proposed.items()
                                        if column not in known and classification == "personal"))

classified = [column for column in proposed if column in known]
print("declared classifications checked:", len(classified))
print("classes declared for them:", ", ".join(sorted({known[column] for column in classified})))
for through in (["DIRECT", "INDIRECT"], ["DIRECT"]):
    proposals = lineage.propose(graph, known, through)
    agree = sum(1 for column in classified if proposals[column][0] == known[column])
    print(f"declared classifications reproduced, through {' and '.join(through)}:", f"{agree} of {len(classified)}")
# proposals is the DIRECT-only run from the loop above
missed = [column for column in classified if proposals[column][0] != known[column]]
print("missed with DIRECT inputs only:", len(missed))
for column in missed:
    print(f"  {column}: declared {known[column]}, DIRECT inputs propose {proposals[column][0]}")
print("declared less protected than lineage proposes:", len(lineage.understated(proposed, known)))
assert not lineage.understated(proposed, known)
```

Output:

```text
metadata problem: ai.review_corpus: no metadata
metadata problem: marketing.audience: no metadata
propagated through: DIRECT and INDIRECT
marketing.audience.learner_id: internal, from crm.learners.learner_id (DIRECT IDENTITY)
marketing.audience.full_name: personal, from crm.learners.full_name (DIRECT IDENTITY)
marketing.audience.email: personal, from crm.learners.email (DIRECT IDENTITY)
marketing.audience.country: internal, from crm.learners.country (DIRECT IDENTITY)
marketing.audience.favourite_category: internal, from crm.learners.learner_id (INDIRECT JOIN)
ai.review_corpus.review_id: internal, from learning.reviews.review_id (DIRECT IDENTITY)
ai.review_corpus.course_id: internal, from learning.reviews.course_id (DIRECT IDENTITY)
ai.review_corpus.course_title: internal, from learning.reviews.course_id (INDIRECT JOIN)
ai.review_corpus.rating: internal, from learning.reviews.rating (DIRECT IDENTITY)
ai.review_corpus.body: personal, from learning.reviews.body (DIRECT IDENTITY)
personal columns proposed: 3
declared classifications checked: 13
classes declared for them: internal
declared classifications reproduced, through DIRECT and INDIRECT: 13 of 13
declared classifications reproduced, through DIRECT: 10 of 13
missed with DIRECT inputs only: 3
  analytics.course_performance.course_id: declared internal, DIRECT inputs propose public
  analytics.course_performance.title: declared internal, DIRECT inputs propose public
  analytics.course_performance.category: declared internal, DIRECT inputs propose public
declared less protected than lineage proposes: 0
```

The metadata check from [chapter 2](#test-the-catalog-against-the-metadata) still reports the two tables with no metadata, and lineage proposes a classification for each of their columns. It proposes 3 personal columns: `full_name` and `email` in `marketing.audience`, each an IDENTITY copy of a personal column in `crm.learners`, and `body` in `ai.review_corpus`, a copy of the reviews' free text. The rest come out internal, including `favourite_category`, for which the example prints `internal, from crm.learners.learner_id (INDIRECT JOIN)`. Its values are public course-catalog category names, but the rows they sit in are chosen by a join on learner IDs and a consent filter, so those are internal.

Counting INDIRECT inputs reproduces the declared classifications on the columns people did classify, 13 of 13, and all of them are `internal`.

DIRECT inputs alone reproduce 10 of 13, and miss 3: `course_id`, `title` and `category` of `analytics.course_performance`. A DIRECT-only rule calls them public, as in the course catalog, while the analytics team chose internal. So the rule proposes no less than people chose here, and lineage finds no declared classification lower than its proposal. That is weak evidence: a rule that proposed internal for every column would score as well. And it can't show what the rule does with a personal filter, since none of the platform's filters or joins reads a personal column: even `marketing_consent` is classified internal.

So build a table whose filter does. For a birthday offer, marketing wants the learners who consented and were born this month, with their ID and country, and nothing else:

Example `personal-filter-propagates` (Python 3.12.15 · duckdb 2.0.0 · openlineage-python 1.55.0 · pyarrow 26.0.1 · pyiceberg 0.12.0 · pyyaml 6.0.3 · sqlglot 30.33.0), passing in Edition 2026.11.31:

```python
"""A birthday-offer table whose only personal input is a filter on date of birth. The prose rests on the learner count,
and on the proposals counting DIRECT inputs alone and counting DIRECT and INDIRECT inputs."""

from lib import lineage
from lib.catalog import open_catalog
from lib.platform import scratch
from lib.warehouse import connect

catalog = open_catalog(["crm.learners"])
# The consenting learners born in the month of the nightly run, on the book's "today".
select = ("select l.learner_id, l.country from crm.learners l "
          f"where l.marketing_consent and month(l.date_of_birth) = {lineage.NIGHTLY.month}")


def offers() -> None:
    rows = connect(catalog).sql(select).to_arrow_table()
    catalog.create_namespace_if_not_exists("marketing")
    catalog.create_table("marketing.birthday_offers", schema=rows.schema).append(rows)


events_file = scratch() / "events.jsonl"
# record derives the lineage from this statement, and reads it before offers() runs
lineage.record(lineage.client(events_file), "birthday_offers", f"create table marketing.birthday_offers as {select}",
               catalog, offers)
offers_table = catalog.load_table("marketing.birthday_offers").scan().to_arrow()
print("learners in marketing.birthday_offers:", f"{len(offers_table):,}")

graph, known = lineage.graph(lineage.read_events(events_file)), lineage.declared()
print("crm.learners.date_of_birth declared:", known["crm.learners.date_of_birth"])
for through in (["DIRECT", "INDIRECT"], ["DIRECT"]):
    for column, (classification, decided_by) in lineage.propose(graph, known, through).items():
        label = f"{column}, through {' and '.join(through)}"
        print(f"{label}:", classification)
        print(f"{label}, decided by:", f"{decided_by[0].source} ({decided_by[0].type} {decided_by[0].subtype})")
```

Output:

```text
learners in marketing.birthday_offers: 165
crm.learners.date_of_birth declared: personal
marketing.birthday_offers.learner_id, through DIRECT and INDIRECT: personal
marketing.birthday_offers.learner_id, through DIRECT and INDIRECT, decided by: crm.learners.date_of_birth (INDIRECT FILTER)
marketing.birthday_offers.country, through DIRECT and INDIRECT: personal
marketing.birthday_offers.country, through DIRECT and INDIRECT, decided by: crm.learners.date_of_birth (INDIRECT FILTER)
marketing.birthday_offers.learner_id, through DIRECT: internal
marketing.birthday_offers.learner_id, through DIRECT, decided by: crm.learners.learner_id (DIRECT IDENTITY)
marketing.birthday_offers.country, through DIRECT: internal
marketing.birthday_offers.country, through DIRECT, decided by: crm.learners.country (DIRECT IDENTITY)
```

Neither column copies anything personal, but every row says its learner was born in the offer's month, part of a date of birth. The table holds 165 learners, and `governance/metadata.toml` classifies the date of birth `personal`. Counting DIRECT inputs alone proposes `internal` for `learner_id` and `internal` for `country`.

Add INDIRECT inputs, and it proposes `personal` for `learner_id` and `personal` for `country`, both decided by `crm.learners.date_of_birth (INDIRECT FILTER)`. That is the case INDIRECT inputs are for, and it shows the cost too: one personal filter makes every column of a table personal.

OpenLineage can carry classifications in the events themselves, with the Tags dataset facet, whose `field` key "identifies the field in a dataset if a tag applies to one" ([tags facet](https://openlineage.io/docs/spec/facets/dataset-facets/tag-facet), [schema](https://openlineage.io/spec/facets/1-0-0/TagsDatasetFacet.json)). We keep them in `governance/metadata.toml` instead, the one source that the platform's access policies also read, through `lib.policy`, the book's policy module ([chapter 5](#generate-the-data-the-policy-reads)).

**Working rule:** Propose each derived column's classification as the stricter of its inputs', counting INDIRECT inputs, and send the proposals to the table's owner for review before they become metadata. Never let lineage lower a classification someone declared, and never treat masking as declassification. Count DIRECT inputs only if your pipelines often join or filter on personal columns, such as matching on email, and every output would otherwise come out personal; then review INDIRECT personal inputs by hand.

## What goes wrong in production {#lineage-in-production}

Lineage fails quietly: a missing edge looks exactly like a column with no dependencies, and every check built on the graph, from [impact analysis](#impact-of-a-column-change) to [proposed classifications](#classifications-from-lineage), then passes. Three failures are common, and each has a check.

### Events that never arrive {#events-that-never-arrive}

The client reads `OPENLINEAGE_DISABLED` from the environment, and when it's `true`, events are dropped ([configuration](https://openlineage.io/docs/client/python/configuration)). In the client's source, that check comes first ([openlineage-python](https://pypi.org/project/openlineage-python/1.54.0/), `client/client.py`): before it looks at the transport you passed explicitly, the client picks a transport that does nothing. Its only notice is an INFO log line: "OpenLineage is disabled. No events will be emitted." Python's root logger is set to `WARNING` by default, so the line never shows ([logging HOWTO](https://docs.python.org/3.12/howto/logging.html)). A variable left in a shared environment file after debugging is enough. Set the variable, run `review_corpus` with an explicit file transport, and check what reaches the file:

Example `lineage-disabled-silently` (Python 3.12.15 · duckdb 2.0.0 · openlineage-python 1.55.0 · pyarrow 26.0.1 · pyiceberg 0.12.0 · pyyaml 6.0.3 · sqlglot 30.33.0), passing in Edition 2026.11.31:

```python
"""With OPENLINEAGE_DISABLED=true in the environment, an explicit file transport writes nothing and raises nothing. The
prose rests on the transport the client chose and on the job the check finds without a COMPLETE event."""
import os

from lib import lineage
from lib.catalog import open_catalog
from lib.platform import scratch

os.environ["OPENLINEAGE_DISABLED"] = "true"        # say, left in a shared environment file after a debugging session

events_file = scratch() / "events.jsonl"
client = lineage.client(events_file)                # a file transport, passed in explicitly
catalog = open_catalog(["learning.courses", "learning.reviews"])
output = lineage.run(catalog, "review_corpus", client)

print("transport the client chose:", client.transport.kind)      # OPENLINEAGE_DISABLED wins over the transport passed in
print("pipeline wrote:", output)
print("rows written:", f"{len(catalog.load_table(output).scan().to_arrow()):,}")
assert not events_file.exists()         # the noop transport never creates the file
print("events file exists:", events_file.exists())
missing = lineage.missing_runs(["review_corpus"], lineage.read_events(events_file))   # ran, but sent no COMPLETE
print("jobs that ran with no COMPLETE event:", ", ".join(missing))
```

Output:

```text
transport the client chose: noop
pipeline wrote: ai.review_corpus
rows written: 3,000
events file exists: False
jobs that ran with no COMPLETE event: review_corpus
```

With the variable set, the pipeline still runs, but nothing reaches the file. The client chose the `noop` transport over the file transport the code passed in, and the pipeline wrote 3,000 rows to `ai.review_corpus` without an error and without an events file. The check that catches it compares the jobs the scheduler ran with the jobs that sent a COMPLETE event: `lib.lineage.missing_runs` lists `review_corpus` as having run without one.

**Working rule:** After every scheduled run, compare the jobs that ran with the jobs whose COMPLETE events arrived, and alert on any difference. Don't rely on the emitter to report its own silence.

### A job nobody instrumented {#job-nobody-instrumented}

Lineage covers only the jobs that send it. The platform's marketing team runs a weekly Python script that copies `marketing.audience` into `marketing.campaign_list`, a table its email tool reads. Nobody wired the script to OpenLineage:

Example `uninstrumented-job` (Python 3.12.15 · duckdb 2.0.0 · openlineage-python 1.55.0 · pyarrow 26.0.1 · pyiceberg 0.12.0 · pyyaml 6.0.3 · sqlglot 30.33.0), passing in Edition 2026.11.31:

```python
"""A Python step copies marketing.audience for the email tool and sends no lineage. The prose rests on the walk from the
email column, which misses the copy, and on the catalog check against the lineage, which finds it."""

from lib import lineage
from lib.catalog import open_catalog
from lib.platform import scratch
from lib.warehouse import connect

catalog = open_catalog()
events_file = scratch() / "events.jsonl"
lineage.run_all(catalog, events_file)

# The marketing team's weekly export, written straight to the catalog: no lineage event.
export = connect(catalog).sql("select learner_id, full_name, email from marketing.audience").to_arrow_table()
catalog.create_table("marketing.campaign_list", schema=export.schema).append(export)
print("rows exported:", f"{len(export):,}")

events = lineage.read_events(events_file)
reached = lineage.downstream(lineage.graph(events), "crm.learners.email")
print("crm.learners.email reaches:", ", ".join(edge.target for edge in reached))
# the catalog's tables, less the platform's sources and the outputs the events name
print("tables no job's lineage accounts for:", ", ".join(lineage.unaccounted(catalog, events)))
```

Output:

```text
rows exported: 1,829
crm.learners.email reaches: marketing.audience.email
tables no job's lineage accounts for: marketing.campaign_list
```

An export that sends no events is invisible to impact analysis. The script copied 1,829 learners' names and emails, yet `crm.learners.email` still reaches only `marketing.audience.email`, so an erasure guided by this lineage would miss the copy. The catalog knows the table exists, though, so `lib.lineage.unaccounted` compares the catalog's tables with the lineage: every table must be one of the platform's sources or the output of a job that lineage knows. It finds `marketing.campaign_list`.

**Working rule:** Check the catalog against the lineage on a schedule, and treat a table no job's lineage accounts for as an incident: find the job, and make it send events.

### Personal data in a table classified internal {#personal-data-tagged-internal}

Instrumenting [the marketing team's export](#job-nobody-instrumented) fixes the graph, and exposes a second problem. When the export's owner created `marketing.campaign_list`, they classified every column internal, because only the marketing team reads it. That's a statement about who reads a table, not what it holds. Wrap the script's work in `lib.lineage.record`, with the `SELECT` it runs as its lineage, and compare the declared classifications with lineage's proposal:

Example `personal-copy-tagged-internal` (Python 3.12.15 · duckdb 2.0.0 · openlineage-python 1.55.0 · pyarrow 26.0.1 · pyiceberg 0.12.0 · pyyaml 6.0.3 · sqlglot 30.33.0), passing in Edition 2026.11.31:

```python
"""The same export, now recorded with lineage, into a table its owner tagged internal. The prose rests on the walk to
the copy and on the check for understated classifications."""

from lib import lineage
from lib.catalog import open_catalog
from lib.platform import scratch
from lib.warehouse import connect

catalog = open_catalog()
events_file = scratch() / "events.jsonl"
lineage.run_all(catalog, events_file)

select = "select a.learner_id, a.full_name, a.email from marketing.audience a"


def export() -> None:
    rows = connect(catalog).sql(select).to_arrow_table()
    catalog.create_table("marketing.campaign_list", schema=rows.schema).append(rows)


# record reads this statement's lineage before export() runs
statement = f"create table marketing.campaign_list as {select}"
lineage.record(lineage.client(events_file), "campaign_export", statement, catalog, export)

# Its owner tagged every column internal: "only the marketing team reads it".
known = lineage.declared() | {f"marketing.campaign_list.{c}": "internal" for c in ("learner_id", "full_name", "email")}
graph = lineage.graph(lineage.read_events(events_file))
print("crm.learners.email reaches:", ", ".join(edge.target for edge in lineage.downstream(graph, "crm.learners.email")))
for problem in lineage.understated(lineage.propose(graph, known), known):
    print("understated:", problem)
```

Output:

```text
crm.learners.email reaches: marketing.audience.email, marketing.campaign_list.email
understated: marketing.campaign_list.full_name: declared internal, lineage proposes personal
understated: marketing.campaign_list.email: declared internal, lineage proposes personal
```

Once the export sends lineage, the walk reaches its copy: `crm.learners.email` reaches `marketing.audience.email, marketing.campaign_list.email`, two hops, the second through the export. And `lib.lineage.understated` reports `full_name` and `email` in `marketing.campaign_list` as declared internal while lineage proposes personal: copies of personal data that any policy reading the declared classification would treat as internal.

**Working rule:** Run `lib.lineage.understated`, or your own check like it, on every lineage update, and fail when a declared classification is lower than lineage proposes. Classify by what a table holds, never by who reads it, and fix such a finding by raising the classification, not by dropping the edge.

## What this chapter decided {#lineage-decisions}

- **Every pipeline run sends OpenLineage run events,** START and then COMPLETE or FAIL, from the code that runs it (`lib.lineage.run`), with the output's schema and column lineage. Datasets are named by catalog identifiers under the placeholder namespace `iceberg://platform`. Examples fix event times and derive run IDs from the nominal run, and compare what events say, not their bytes.
- **Column lineage is sqlglot's value lineage, typed as OpenLineage subtypes, plus INDIRECT inputs:** the join, filter, grouping and sort columns of every `SELECT`, CTEs included, in the facet's `dataset` list.
- **Lineage that can't be read correctly is an error,** never empty or partial. `lib.lineage` reads only `CREATE ... AS` and `INSERT ... SELECT`, refuses every other statement, `MERGE` included, whatever its `USING` names. Empty lineage, an unplaceable source or an unknown column stops the job before it runs. An `INSERT`'s columns come from the table's schema in the catalog, matched by position, and a count that differs is an error.
- **Impact analysis walks the graph built from events,** INDIRECT edges included, and reports columns whose values can change and tables whose rows can change.
- **Classifications propagate as the stricter of all inputs',** DIRECT and INDIRECT, as set in `governance/lineage.toml`, so a table chosen by a personal filter is proposed personal. Lineage never lowers a declared classification, and masking never declassifies. The proposals for the two unclassified tables, `full_name`, `email` and `body` personal and every other column internal, go to their owners for review before they enter `governance/metadata.toml`.
- **Three checks guard the lineage itself:** jobs that ran with no COMPLETE event, tables no job's lineage accounts for, and declared classifications lower than lineage proposes. A lineage graph is only as complete as its events, and a missing edge looks exactly like a column with no dependencies.

# Access policies as code

By the end of this chapter you can write a data platform's access rules as one policy that denies by default and returns row filters and column masks as data. The policy allows by role and narrows by attribute, and an engine can apply its whole decision. You'll write it in Rego, the policy language of Open Policy Agent (OPA), test it the way `opa test`, OPA's test runner, does, check it against a who-sees-what table that the data's owners can read, know where the in-process engine this book uses differs from OPA, and see the same rules in Cedar, a narrower policy language.

Each group that reads the platform's warehouse, from support agents to its course assistant, needs something different. Support agents each serve one region, `americas`, `emea` or `apac`, and need learners' contact details to answer tickets. Instructors see the learners and reviews of their own courses, without contact details. Marketing may reach only learners who agreed to marketing email. Finance reconciles payments, and the legacy `card_number` column of `billing.payments` must never show more than its last four digits. Data scientists see `learner_id` as stored, to join tables on it, with personal columns pseudonymized and free text redacted. The course assistant reads only the reviews whose authors agreed to AI training. [Chapter 1](#governance-end-to-end) showed one rule, support agents see only their own region, and enforced it on one query. Rules for every group need to live in one place, stated exactly, and tested before anyone relies on them.

This chapter's terms are the ones the rest of the book uses. A **policy** is code that answers whether a request may go ahead. A **decision** is a policy's answer. The **principal** is who asks, a **role** is a group of principals with the same job, and an **attribute** is any other fact a rule reads, about the principal, the data or the request. A **row filter** limits which rows a query returns, and a **column mask** changes how a column's values are shown, such as **pseudonymization**, which replaces an identifier with a consistent stand-in.

The platform's policy is `governance/policies/access.rego`, and its tests are in `governance/policies/access_test.rego`. The book's shared module `lib.policy` loads both into regopy 1.5.2, the Python binding of rego-cpp, Microsoft's C++ implementation of Rego, which evaluates policies inside the Python process with no server to run.

## Shape the request like Trino's {#shape-the-request-like-trinos}

A policy engine answers questions about requests, so the shape of a request decides which systems can ask it. OPA's premise is that policy is taken out of the service that enforces it: OPA "lets you decouple policy from that software service", so that it can be read, versioned and tested on its own ([OPA philosophy](https://www.openpolicyagent.org/docs/philosophy)). The service sends a JSON document describing the request, and the policy reads it as `input`. Trino, an open-source SQL engine, does exactly this: for each access check a query needs, its OPA plugin sends a document with `context.identity`, the user and their groups, and `action`, the operation and the resource, down to the catalog, schema, table and columns ([Trino 484, OPA access control](https://trino.io/docs/484/security/opa-access-control.html)).

The platform's policy reads the same fields, so it decides Trino's request to read columns, `SelectFromColumns`, as Trino sends it. Trino needs more than that one decision. It wants each row filter and column mask as SQL, a row filter as `{"expression": "clause"}`, where the policy returns a column and a value ([Trino 484, OPA access control](https://trino.io/docs/484/security/opa-access-control.html)). It also asks about other checks, such as `ExecuteQuery` and `AccessCatalog`, which this policy's `allow` denies ([`OpaAccessControl.java` at Trino 484](https://github.com/trinodb/trino/blob/484/plugin/trino-opa/src/main/java/io/trino/plugin/opa/OpaAccessControl.java)). [Chapter 6](#engine-enforcement) shows what a module beside the policy has to return for them. `lib.policy.request` builds the platform's requests in Trino's shape, and the next example uses it for an `emea` support agent, `agent-a05`, reading three columns of `support.tickets`:

Example `trino-shaped-request` (Python 3.12.15 · pyarrow 26.0.1 · pyiceberg 0.12.0 · regopy 1.5.2), passing in Edition 2026.11.31:

```python
"""The request a support agent's read sends the policy, shaped as Trino's OPA plugin sends it. The point rests on the printed `action`: the operation SelectFromColumns, and the table with the columns the query reads."""
import json

from lib import policy

# A support agent's query reads three columns of the tickets table: the request Trino's OPA plugin would send for it.
# no groups here, because the policy reads them from users.toml; no purpose either, since request adds one only when given
request = policy.request("agent-a05", "support.tickets", ["ticket_id", "opened_at", "body"])
print(json.dumps(request, indent=2))
```

Output:

```text
{
  "context": {
    "identity": {
      "user": "agent-a05"
    }
  },
  "action": {
    "operation": "SelectFromColumns",
    "resource": {
      "table": {
        "catalogName": "platform",
        "schemaName": "support",
        "tableName": "tickets",
        "columns": [
          "ticket_id",
          "opened_at",
          "body"
        ]
      }
    }
  }
}
```

The operation is `SelectFromColumns`, Trino's name for a read, and the table sits under `action.resource.table` with its catalog, schema, name and the columns the query touches. The platform's request differs from Trino's in two places that matter. It leaves out `context.identity.groups`, because the policy looks a user's groups up in the same file as their attributes, `governance/users.toml`, so the tests and the policy read one source of truth. And it may carry a top-level `purpose`, such as `ai-training`, which Trino doesn't send. Behind Trino, we recommend deriving the purpose from the identity, for instance by giving the training pipeline a service user of its own, rather than trusting a value the client sets.

**Working rule:** Shape the policy's input like the requests of the engine that will enforce it, field for field, and add only what that engine can't send, at the top level where a reviewer sees it. If your engine isn't Trino, start from its request format instead, and keep the shape in one function, as `lib.policy.request` does, so tests and callers can't drift apart.

## Generate the data the policy reads {#generate-the-data-the-policy-reads}

The [request](#shape-the-request-like-trinos) says who asks for what. Everything else a rule needs besides the request, such as each user's groups and each column's classification, comes from a second document, which OPA calls `data`. A fact in that document is only as current as its source, and every fact the platform's policy reads already has an owner. Users and their attributes live in `governance/users.toml`. Each column's classification lives in `governance/metadata.toml`, the reviewed metadata file that [chapter 2](#key-column-metadata-by-field-id) keys by Iceberg field ID. And which instructor teaches which course is in the `learning.courses` table itself. Typing any of these into the policy, or copying them into a second file, makes a copy that someone has to remember to update.

`lib.policy.data_document` builds the document from those three sources each time a policy is loaded, and never from anything typed:

Example `generated-policy-data` (Python 3.12.15 · pyarrow 26.0.1 · pyiceberg 0.12.0 · regopy 1.5.2), passing in Edition 2026.11.31:

```python
"""The data the policy reads besides the request, built from the files that own each fact. The point rests on the last two lines: instructor 17's courses come from learning.courses, not from anything typed."""
from lib import policy

# What the policy reads besides the request, built from the files that own each fact.
# the document's one top-level key, which the policy reads as data.platform
data = policy.data_document()["platform"]

print("users, from governance/users.toml:", len(data["users"]))
print("agent-a05:", data["users"]["agent-a05"])
print("instructor-17:", data["users"]["instructor-17"])

classifications = data["classifications"]
print("tables, from governance/metadata.toml:", len(classifications))
print("columns classified:", sum(len(columns) for columns in classifications.values()))
print("crm.learners.email:", classifications["crm.learners"]["email"])
print("billing.payments.card_number:", classifications["billing.payments"]["card_number"])

courses = data["instructor_courses"]
print("instructors with courses, from learning.courses:", len(courses))
# instructor IDs are string keys in the document, so the lookup is "17"
print("instructor 17's courses:", courses["17"])
```

Output:

```text
users, from governance/users.toml: 8
agent-a05: {'groups': ['support'], 'region': 'emea'}
instructor-17: {'groups': ['instructor'], 'instructor_id': 17}
tables, from governance/metadata.toml: 10
columns classified: 77
crm.learners.email: personal
billing.payments.card_number: sensitive
instructors with courses, from learning.courses: 48
instructor 17's courses: [2, 75, 80, 87, 103, 134]
```

Each fact in the document comes from the file or table that owns it, and the output labels each count with its source: the users in `governance/users.toml`, the classified columns across the tables in `governance/metadata.toml`, and the instructors in `learning.courses`. `agent-a05` has a group and a region, and `instructor-17` a group and an instructor ID, but nothing in `users.toml` says which courses instructor 17 teaches. The document finds [2, 75, 80, 87, 103, 134] in `learning.courses`. When the platform publishes another of their courses, the instructor's row filter includes it the next time the policy's data is generated, with no edit anywhere. A classification works the same way: changing `card_number` from `sensitive` would take one reviewed change to `metadata.toml`, and the policy would follow.

**Working rule:** Generate the policy's data from the file or table that owns each fact, every time you load the policy, and never copy a fact into the policy or into a second file. When a fact has no owner yet, such as a support agent's region, give it one file, reviewed like code, as the platform does with `users.toml`.

## Deny by default, then allow by role {#deny-by-default-then-allow-by-role}

With the [request](#shape-the-request-like-trinos) and its [data](#generate-the-data-the-policy-reads) in place, the first thing a policy must settle is what happens to a request no rule anticipated. Saltzer and Schroeder's principle of fail-safe defaults is to "Base access decisions on permission rather than exclusion" ([Saltzer and Schroeder, 1975](https://web.mit.edu/Saltzer/www/publications/protection/Basic.html)). In Rego, that is a default value. A rule whose body never evaluates to true has no value, which Rego calls **undefined**, and OPA uses the default when every rule sharing a name is undefined ([OPA policy language](https://www.openpolicyagent.org/docs/policy-language)). With `default allow := false`, a request that no `allow` rule matches is denied, whoever sends it and whatever it asks for.

Role-based access control, first formalized by [Ferraiolo and Kuhn in 1992](https://csrc.nist.gov/files/pubs/conference/1992/10/13/rolebased-access-controls/final/docs/ferraiolo-kuhn-92.pdf), grants access by the job a principal holds rather than by who they are. On the platform, a role is a group in `users.toml`, and the policy's `readable` object maps each group to the tables it may read. A request is allowed when it's a read, some group of the user may read the table, and nothing the group's row filters depend on is missing:

```rego
default allow := false

allow if {
	input.action.operation == "SelectFromColumns"
	readable_by_a_group
	not missing_attribute
}
```

The last line guards against a failure that [the section on production](#policies-in-production) shows by removing it. Print the map and decide four requests to see the default at work:

Example `allow-by-role` (Python 3.12.15 · pyarrow 26.0.1 · pyiceberg 0.12.0 · regopy 1.5.2), passing in Edition 2026.11.31:

```python
"""Which groups may read which tables, then four requests decided by the default-deny policy. The points rest on the four decisions and their reasons, printed last."""
from lib import policy

rego = policy.interpreter()

# The groups the policy knows, and the tables each may read: the role part of the policy.
readable = policy.evaluate(rego, "data.platform.access.readable", {})
for group, tables in readable.items():
    print(f"{group} may read:", ", ".join(tables))

# Who may read what. Anything no rule allows is denied, including a user or an operation the policy has never seen.
requests = [
    policy.request("finance-1", "billing.payments", ["amount"]),
    policy.request("marketing-1", "billing.payments", ["amount"]),
    policy.request("someone-else", "crm.learners", ["learner_id"]),
]
delete = policy.request("finance-1", "billing.payments", ["amount"])
# the same request as finance's read, asking to delete instead
delete["action"]["operation"] = "DeleteFromTable"
for request in requests + [delete]:
    decision = policy.decide(request, rego)
    who, table = request["context"]["identity"]["user"], request["action"]["resource"]["table"]
    print(f"{who}, {request['action']['operation']} on {table['schemaName']}.{table['tableName']}:",
          "allowed" if decision["allow"] else "denied", "|", "; ".join(decision["reasons"]))
```

Output:

```text
ai may read: learning.courses, learning.reviews
data-science may read: analytics.course_performance, analytics.learner_activity, billing.payments, crm.learners, learning.courses, learning.enrollments, learning.events, learning.reviews
finance may read: billing.payments, learning.courses, learning.enrollments
instructor may read: learning.courses, learning.enrollments, learning.reviews
marketing may read: crm.learners, learning.courses, marketing.audience
support may read: crm.learners, support.tickets
finance-1, SelectFromColumns on billing.payments: allowed | finance may read billing.payments
marketing-1, SelectFromColumns on billing.payments: denied | no group of this user may read this table
someone-else, SelectFromColumns on crm.learners: denied | no group of this user may read this table
finance-1, DeleteFromTable on billing.payments: denied | DeleteFromTable is not an operation this policy allows
```

Only what a rule grants is allowed. Finance may read `billing.payments`, and the reason names the group that allowed it. Marketing may not, and neither may `someone-else`, a user the policy has never heard of, because nothing grants either of them anything. The fourth request is finance again, asking to delete rather than read. No rule mentions `DeleteFromTable`, so it's denied too, and the reason says why. A policy that tried to list what is forbidden would have needed a rule for that operation; this one needs none. Every decision carries its reasons, so a denial can be explained to the person who got it, and logged.

**Working rule:** Start every policy with `default allow := false`, and allow only through rules that name the role and the resource. Keep the map from roles to tables in one place, where a reviewer can read all of it in one sitting, and give every decision a reason. When a request is denied that should have been allowed, add a rule that allows it; never weaken the default.

## Narrow by attribute: region, consent and purpose {#narrow-by-attribute}

[Roles](#deny-by-default-then-allow-by-role) say which tables a group may read; they can't say which rows. NIST's guide to attribute-based access control defines the general approach, in which access is decided by attributes of the subject, the object, the requested operations and, sometimes, the environment. It treats a role as one such attribute ([NIST SP 800-162](https://doi.org/10.6028/NIST.SP.800-162)). The alternative, a role for every combination, is what NIST's guide calls "role explosion".

The platform shows why. Support by region would need three support roles, and adding marketing's consent and a purpose for AI training would multiply the roles again. Instead, the policy keeps one role per job and reads three attributes. The **region** is an attribute of the principal, from `users.toml`. [**Consent**](#retention-consent-erasure) is an attribute of the data: the learner's `marketing_consent` and `ai_training_consent` columns in `crm.learners`. The **purpose** is an attribute of the request. Each one becomes a row filter, a structured condition on a column that the engine adds to the query. The support agent's filter reads the agent's region:

```rego
row_filters contains {"column": "support_region", "equals": user.region} if {
	"support" in groups
	table in {"crm.learners", "support.tickets"}
}
```

Reads by support agents, an instructor, marketing and a data scientist show what each attribute does, with the data scientist reading reviews twice, once with no purpose and once for `ai-training`:

Example `attributes-as-row-filters` (Python 3.12.15 · pyarrow 26.0.1 · pyiceberg 0.12.0 · regopy 1.5.2), passing in Edition 2026.11.31:

```python
"""Row filters the policy builds for seven reads, from three attributes: the agent's region, the learner's consent and the request's purpose. The points rest on the analyst's two review reads, with and without a purpose."""
import json

from lib import policy

rego = policy.interpreter()

# Region is the agent's attribute, consent the learner's, and purpose the request's. Each becomes a row filter.
reads = [
    ("agent-a01", "support.tickets", ["ticket_id"], None),
    ("agent-a05", "support.tickets", ["ticket_id"], None),
    ("agent-a09", "support.tickets", ["ticket_id"], None),
    ("instructor-17", "learning.reviews", ["rating"], None),
    ("marketing-1", "crm.learners", ["email"], None),
    ("analyst-1", "learning.reviews", ["rating", "body"], None),
    # the same read with a purpose, the one attribute the requester supplies
    ("analyst-1", "learning.reviews", ["rating", "body"], "ai-training"),
]
for user, table, columns, purpose in reads:
    decision = policy.decide(policy.request(user, table, columns, purpose), rego)
    who = user + (f" for {purpose}" if purpose else "")
    # json shows the consent filter with its learner_consented key, not a column
    print(f"{who} reads {table}, row filters:", json.dumps(decision["row_filters"]))
    if "body" in columns:
        print(f"{who} reads {table}, body shown as:", decision["column_masks"]["body"])
```

Output:

```text
agent-a01 reads support.tickets, row filters: [{"column": "support_region", "equals": "americas"}]
agent-a05 reads support.tickets, row filters: [{"column": "support_region", "equals": "emea"}]
agent-a09 reads support.tickets, row filters: [{"column": "support_region", "equals": "apac"}]
instructor-17 reads learning.reviews, row filters: [{"column": "course_id", "in": [2, 75, 80, 87, 103, 134]}]
marketing-1 reads crm.learners, row filters: [{"column": "marketing_consent", "equals": true}, {"column": "status", "equals": "active"}]
analyst-1 reads learning.reviews, row filters: []
analyst-1 reads learning.reviews, body shown as: redact
analyst-1 for ai-training reads learning.reviews, row filters: [{"learner_consented": "ai_training_consent"}]
analyst-1 for ai-training reads learning.reviews, body shown as: redact_contacts
```

The purpose is what changes a data scientist's view of review text. Without one, every review is visible, but the text, `body`, is shown as `redact`. For `ai-training`, only the reviews of learners who agreed to AI training remain, and the text comes back as `redact_contacts`, with contact details taken out. The three agents get the same rule with one value per region, from one role, and instructor 17's filter lists their courses, generated from `learning.courses`. Marketing gets two filters, consent given and account active, and an engine applies both. The AI-training filter, `learner_consented`, tests the review's author, not the review, so enforcing it takes a join to `crm.learners`. A learner who signed up before the platform asked about AI training has no answer in `ai_training_consent`, according to the platform's data card, so enforcing the filter has to count an unknown answer as a no.

The purpose is the one attribute the requester supplies, so it reveals more only under a narrower condition: stating `ai-training` shows review text, but only for consenting learners and without contact details. Leaving it out, or giving any other purpose, gets what the role allows anyway, with the text hidden.

**Working rule:** Model each condition as an attribute of the principal, the data or the request, read by one rule per role, rather than as another role. When an attribute comes from the requester, as a purpose does, let it unlock access only under a narrower condition, so that leaving it out, or getting it wrong, falls back to what the role allows anyway.

## Return the decision as one document {#return-the-decision-as-one-document}

A yes or no can't carry the [row filters](#narrow-by-attribute) that attributes produce. A Rego decision needn't be yes or no, because OPA policies can represent decisions as structured data, from booleans to maps of lists of maps ([OPA philosophy](https://www.openpolicyagent.org/docs/philosophy)). The platform's policy uses structured data to return its whole answer as one document, `data.platform.access.decision`, with four keys: `allow`, the `row_filters`, the `column_masks`, and the `reasons`. `lib.policy.decide` asks for it with one query and returns it as a Python dictionary.

The filters and masks are data, never SQL. A filter names a column and a value or a list of values, or, for consent, the learner's consent column. A mask is a name, one of `redact`, `last4`, `pseudonymize` and `redact_contacts`. Whatever enforces the decision turns each into SQL with its own code, so nothing a request or a user record contains, such as a region typed into `users.toml`, can reach a query as SQL text. Masks follow each column's classification. Sensitive columns are hidden from everyone, except the last four digits of a card number, which finance sees. Personal columns are shown to support, which needs contact details, and masked for everyone else as their job requires. And a column the metadata doesn't classify counts as sensitive:

```rego
classification(column) := level if {
	level := data.platform.classifications[table][column]
} else := "sensitive"
```

Take marketing reading four columns of `crm.learners`, three of them classified in the metadata. The fourth, `linkedin_url`, stands for a column a pipeline has added and nobody has classified yet:

Example `one-decision-document` (Python 3.12.15 · pyarrow 26.0.1 · pyiceberg 0.12.0 · regopy 1.5.2), passing in Edition 2026.11.31:

```python
"""The whole decision for one read: marketing's four columns of crm.learners, one of them unclassified. The point rests on the masks and the reasons, which give one line per mask and per group that allowed the read."""
import json

from lib import policy

# Marketing reads four columns of crm.learners. `linkedin_url` stands for a column added to the table and not yet
# classified in governance/metadata.toml.
request = policy.request("marketing-1", "crm.learners", ["full_name", "email", "phone", "linkedin_url"])
# with no interpreter given, decide loads the platform's policy itself
decision = policy.decide(request)
print(json.dumps(decision, indent=2))
print("keys:", ", ".join(sorted(decision)))
```

Output:

```text
{
  "allow": true,
  "column_masks": {
    "linkedin_url": "redact",
    "phone": "redact"
  },
  "reasons": [
    "linkedin_url is sensitive, shown as redact",
    "marketing may read crm.learners",
    "phone is personal, shown as redact"
  ],
  "row_filters": [
    {
      "column": "marketing_consent",
      "equals": true
    },
    {
      "column": "status",
      "equals": "active"
    }
  ]
}
keys: allow, column_masks, reasons, row_filters
```

Marketing gets the read, with its two row filters, and two of the four columns are masked. `phone` is personal, and marketing doesn't need it for email. `linkedin_url` is unclassified, so it's treated as sensitive until someone reviews it. `full_name` and `email` have no mask, and a column without a mask is shown as stored, which marketing needs to send an email. The reasons give one line per group that allowed the read and one per mask, so the document explains itself. A denial carries empty filters and masks, because the policy builds the two outcomes with two separate rules, so nothing downstream can mistake a denial for a read with some columns masked.

**Working rule:** Return one decision document per request, holding the allow, the row filters, the column masks and the reasons, and read it as a whole. Keep filters and masks as names and values that your enforcement code translates, never as SQL that the policy writes. If an engine needs SQL from the policy, as Trino does for its row filters and column masks ([Trino 484, OPA access control](https://trino.io/docs/484/security/opa-access-control.html)), generate it in one tested function between the two.

## Unit-test the policy in Rego {#unit-test-the-policy-in-rego}

The [decision document](#return-the-decision-as-one-document) is what a test can compare, which matters because a policy decides who sees personal data, and a change to it can widen access without anyone seeing it in a diff. OPA's answer is tests written in Rego: a test is a rule whose name is prefixed with `test_`, and "If the test rule is undefined or generates a non-true value the test result is reported as FAIL" ([OPA policy testing](https://www.openpolicyagent.org/docs/policy-testing)). A test sets up its request with `with`, which replaces part of `input` or `data` for one expression.

The book's engine, regopy, has no `opa test`, so `lib.policy.run_tests` plays its part. It finds every `test_` rule in the folder's `_test.rego` files, evaluates each one, and reports PASS when the rule is true, FAIL when it's false or undefined, and ERROR when evaluation fails. Each of the platform's tests sets the user and the action as two separate paths, each to a literal value, for reasons [the section on regopy](#where-regopy-differs-from-opa) shows:

```rego
test_finance_sees_only_the_last_four_digits if {
	decision := access.decision with input.context.identity.user as "finance-1"
		with input.action as {"operation": "SelectFromColumns", "resource": {"table": {"schemaName": "billing", "tableName": "payments", "columns": ["amount", "card_number"]}}}
	decision.allow
	decision.row_filters == []
	decision.column_masks == {"card_number": "last4"}
}
```

The tests cover each group's allow, its filters and its masks, and the ways a request is denied: a table the group isn't given, an operation other than a read, a user the policy doesn't know, and a support agent with no region.

Example `policy-unit-tests` (Python 3.12.15 · pyarrow 26.0.1 · pyiceberg 0.12.0 · regopy 1.5.2), passing in Edition 2026.11.31:

```python
"""Runs every test in the platform's Rego test file and counts the passes. The point rests on the last line, the tests passed out of all of them."""
from collections import Counter

from lib import policy

# Every `test_` rule in governance/policies/access_test.rego, run by name: PASS, FAIL or ERROR, as `opa test` reports.
results = policy.run_tests()
for test, result in results.items():
    print(f"{result}: {test}")
# len(results) counts every test, so one that fails or errors still shows in the total
print("tests passed:", f"{Counter(results.values())['PASS']} of {len(results)}")
```

Output:

```text
PASS: test_support_reads_only_its_own_region
PASS: test_support_without_a_region_is_denied
PASS: test_marketing_sees_only_consenting_active_learners
PASS: test_a_group_only_reads_its_own_tables
PASS: test_finance_sees_only_the_last_four_digits
PASS: test_data_science_never_sees_card_numbers
PASS: test_data_science_sees_pseudonyms_not_people
PASS: test_an_unclassified_column_is_hidden
PASS: test_an_unknown_user_is_denied
PASS: test_an_instructor_sees_only_their_courses
PASS: test_the_assistant_reads_only_consented_reviews
PASS: test_training_reads_only_consented_reviews_without_contacts
tests passed: 12 of 12
```

A test that checks only `decision.allow` would miss most mistakes, so every test of an allowed read compares the row filters and the column masks whole. In finance's test, `decision.row_filters == []` fails if the policy adds any filter to finance's reads, and `decision.column_masks == {"card_number": "last4"}` fails if it masks one column more or one fewer, or shows the card number another way. The run passes 12 of 12 tests.

**Working rule:** Write a test for every rule that grants or narrows access, and one for every way a request can be denied, and compare whole filters and masks, not single keys. Run the tests on every change to the policy and to the data the policy reads.

## Check who sees what from a decision table {#check-who-sees-what}

[Unit tests](#unit-test-the-policy-in-rego) are written in Rego by whoever writes the policy. The people who own the data, such as the support lead or the finance team, should be able to check the policy too, without reading Rego. A decision table is one row per request, saying who reads which table and what they must get back, in the terms of their job.

The platform's table is `lib.policy.WHO_SEES_WHAT`, a list of rows in Python. Each gives a user, a table, the columns read and an optional purpose, then whether the read is allowed, which columns the row filters test and which columns are masked, and how. Every row goes through `lib.policy.decide`, which returns the [decision document](#return-the-decision-as-one-document), and the example prints one line per row, in the form to show the data's owners:

Example `who-sees-what` (Python 3.12.15 · pyarrow 26.0.1 · pyiceberg 0.12.0 · regopy 1.5.2), passing in Edition 2026.11.31:

```python
"""Each row of the who-sees-what table run through the policy and printed as one line, with the count of rows the policy matches. The point rests on the last line, the count of rows whose decision matches the outcome its row expects."""
from lib import policy

rego = policy.interpreter()

# Each row of lib.policy.WHO_SEES_WHAT is a request and the decision a reviewer expects for it.
matching = 0
for case in policy.WHO_SEES_WHAT:
    decision = policy.decide(policy.request(case.user, case.table, case.columns, case.purpose), rego)
    # matches returns True or False, which add as 1 or 0
    matching += policy.matches(case, decision)
    who = case.user + (f" ({case.purpose})" if case.purpose else "")
    shown = "allowed" if decision["allow"] else "denied"
    # a consent filter has no column key, so filtered_on reads its learner_consented key
    filters = ", ".join(policy.filtered_on(decision)) or "-"
    masks = ", ".join(f"{column}: {mask}" for column, mask in decision["column_masks"].items()) or "-"
    print(f"{who:26} {case.table:17} {shown:8} rows: {filters:29} masks: {masks}")
print("rows of the table the policy matches:", f"{matching} of {len(policy.WHO_SEES_WHAT)}")
```

Output:

```text
agent-a05                  crm.learners      allowed  rows: support_region                masks: -
agent-a05                  billing.payments  denied   rows: -                             masks: -
instructor-17              learning.reviews  allowed  rows: course_id                     masks: body: redact_contacts
instructor-17              crm.learners      denied   rows: -                             masks: -
marketing-1                crm.learners      allowed  rows: marketing_consent, status     masks: phone: redact
finance-1                  billing.payments  allowed  rows: -                             masks: card_number: last4
analyst-1                  billing.payments  allowed  rows: -                             masks: card_number: redact
analyst-1                  crm.learners      allowed  rows: -                             masks: email: pseudonymize, notes: redact
analyst-1                  learning.reviews  allowed  rows: -                             masks: body: redact
analyst-1 (ai-training)    learning.reviews  allowed  rows: ai_training_consent           masks: body: redact_contacts
assistant-1                learning.reviews  allowed  rows: ai_training_consent           masks: body: redact_contacts
someone-else               crm.learners      denied   rows: -                             masks: -
rows of the table the policy matches: 12 of 12
```

Read down the table and each role's rules show up as outcomes. The policy matches 12 of 12 rows: the support agent filtered by region and denied payments; the instructor filtered by course and denied the learners table; finance seeing `card_number` as `last4` where the data scientist sees `redact`; and the data scientist's two reads of reviews, with and without a purpose. A reviewer who thinks a row is wrong can say so in a pull request without knowing which rule produced it.

**Working rule:** Keep a who-sees-what table in the repository with the policy, run it with the unit tests, and show its printed form to the data's owners, in terms they can check. When an owner disagrees with a row, change the row first, and let the failing row drive the change to the policy.

## Know where regopy differs from OPA {#where-regopy-differs-from-opa}

The platform's [tests](#unit-test-the-policy-in-rego) run in regopy because the book's examples must run offline, in seconds, inside Python, and OPA is a separate program. regopy 1.5.2 is close to OPA. Its engine states that it supports Rego v1.18.1 as OPA defines it, and that it passes all the non-built-in test suites OPA publishes. Its README lists the built-ins it doesn't support, such as `net.*` and `http.send` ([rego-cpp README at v1.5.2](https://github.com/microsoft/rego-cpp/blob/v1.5.2/README.md)), but regopy also differs from OPA in ways that README doesn't list. The platform's policy and tests avoid each of the cases that follow, and OPA itself is the reference for anything in production.

The first group runs without any error:

Example `regopy-silent-differences` (Python 3.12.15 · regopy 1.5.2), passing in Edition 2026.11.31:

```python
"""Small policies that regopy answers differently from OPA, each printed with the answer regopy gives. The prose's points rest on the printed values, and the bound and literal-mock lines are the controls where regopy agrees with OPA."""
import json

from regopy import Input, Interpreter


def ask(rules: str, query: str, request: dict) -> str:
    """One query's answer from regopy: the value bound to x, or "undefined"."""
    rego = Interpreter()
    rego.add_module("example", "package example\n" + rules)
    rego.set_input(Input(request))
    output = str(rego.query(query))
    # regopy writes undefined as a bare word; otherwise the binding for x is the answer
    return output if output == "undefined" else json.dumps(json.loads(output)["bindings"]["x"])


admins_only = 'default allow := false\nallow if input.user == "admin"\n'
# no variable in the query
print("a false decision, queried on its own:", ask(admins_only, "data.example.allow", {"user": "guest"}))
# bound to x, the same false decision comes back as a value
print("the same decision, bound to a variable:", ask(admins_only, "x := data.example.allow", {"user": "guest"}))

pre_1_0 = 'allow { input.user == "admin" }\n'                      # no `if`: OPA 1.x refuses to load this
print("a rule in pre-1.0 syntax:", ask(pre_1_0, "x := data.example.allow", {"user": "admin"}))

# regoin is never defined, so deny is undefined and not deny is true for any request
typo = 'region := "emea"\ndefault allow := false\nallow if not deny\ndeny if input.region != regoin\n'
print("a deny with a misspelt, unbound variable, for apac:", ask(typo, "x := data.example.allow", {"region": "apac"}))

mocks = """region := input.user.region
emea := {"region": "emea"}
with_a_literal := seen if { seen := region with input.user as {"region": "emea"} }
with_a_reference := seen if { seen := region with input.user as emea }
"""
print("region under a mock written as a literal:", ask(mocks, "x := data.example.with_a_literal", {"user": {"region": "apac"}}))
# the mock refers to the rule emea, which regopy ignores, so the real input's region comes back
print("region under a mock that refers to a rule:", ask(mocks, "x := data.example.with_a_reference", {"user": {"region": "apac"}}))
```

Output:

```text
a false decision, queried on its own: undefined
the same decision, bound to a variable: false
a rule in pre-1.0 syntax: true
a deny with a misspelt, unbound variable, for apac: true
region under a mock written as a literal: "emea"
region under a mock that refers to a rule: "apac"
```

- **A false decision comes back undefined.** Asked for `data.example.allow` on its own, regopy answers `undefined` where the value is `false`, so "denied" can't be told from "no decision". Bound to a variable, the same query gives `false`. `lib.policy` binds every decision it reads.
- **Pre-1.0 syntax runs.** Since OPA 1.0, `if` is required for every rule and `contains` for multi-value rules ([OPA v1.0.0 release notes](https://github.com/open-policy-agent/opa/releases/tag/v1.0.0)), and rego-cpp's changelog records that it has dropped support for v0 Rego ([rego-cpp changelog](https://github.com/microsoft/rego-cpp/blob/v1.5.2/CHANGELOG)). Yet regopy evaluates `allow { ... }` and answers `true`, so a policy can pass here and fail to load in OPA, and behind Trino as well.
- **An unbound variable switches a rule off.** OPA requires a variable in a comparison to be bound by another expression in the same rule, and rejects the policy otherwise ([OPA policy language](https://www.openpolicyagent.org/docs/policy-language)). regopy runs it anyway. A `deny` with `regoin` misspelt for `region` is silently undefined, so `allow if not deny` allows an `apac` request, as `true` shows.
- **A mock that refers to another rule is ignored.** In a policy whose `region` rule reads `input.user.region`, a mock written as a literal, `with input.user as {"region": "emea"}`, gives `"emea"`. Written as a reference to a rule holding the same object, it gives the real input's `"apac"`.

The second group fails, and one failure takes Python down with it. One case adds a test to a copy of the platform's tests; the other formats the same string from nine sources:

Example `regopy-failed-outputs` (Python 3.12.15 · pyarrow 26.0.1 · pyiceberg 0.12.0 · regopy 1.5.2), passing in Edition 2026.11.31:

```python
"""One test that replaces the whole input, and what that does to the interpreter: every query fails, and printing a failed output ends Python. The points rest on the printed error count, the assert that true does not succeed, and the signal the crash ends with."""
import signal
import subprocess
import sys
import tempfile
from collections import Counter
from pathlib import Path

from lib import policy

# The platform's tests, plus one that replaces the whole input at once rather than one path at a time.
whole_input = """
test_finance_reads_payments if {
	access.decision.allow with input as {"context": {"identity": {"user": "finance-1"}}, "action": {"operation": "SelectFromColumns", "resource": {"table": {"schemaName": "billing", "tableName": "payments", "columns": ["amount"]}}}}
}
"""
with tempfile.TemporaryDirectory() as name:
    folder = Path(name)
    for path in policy.POLICIES.glob("*.rego"):
        (folder / path.name).write_text(path.read_text(encoding="utf-8"), encoding="utf-8")
    with open(folder / "access_test.rego", "a", encoding="utf-8") as tests:
        tests.write(whole_input)
    results = policy.run_tests(folder)
    print("tests that end in ERROR:", f"{Counter(results.values())['ERROR']} of {len(results)}")
    rego = policy.interpreter(folder, tests=True)
    succeeded = rego.query("true").ok()
    # no query succeeds here, not even true: the interpreter loads the file with the broken test
    assert not succeeded
    print("the query true succeeds:", succeeded)

    # lib.policy checks each output before reading it, so a decision asked of this interpreter raises an error.
    try:
        policy.decide(policy.request("finance-1", "billing.payments", ["amount"]), rego)
    except RuntimeError as error:
        print("lib.policy.decide raises:", error)

    # Printing one of those failed outputs, in a separate Python process.
    crash = "import sys; from pathlib import Path; from lib import policy; print(policy.interpreter(Path(sys.argv[1]), tests=True).query('true'))"
    ended = subprocess.run([sys.executable, "-c", crash, str(folder)], capture_output=True)
    # a negative return code names the signal that ended the process
    print("printing the failed output ends Python with:", signal.Signals(-ended.returncode).name)
```

Output:

```text
tests that end in ERROR: 13 of 13
the query true succeeds: False
lib.policy.decide raises: regopy couldn't evaluate data.platform.access.decision, so there is no decision to read
printing the failed output ends Python with: SIGSEGV
```

Example `regopy-sprintf-quotes` (Python 3.12.15 · regopy 1.5.2), passing in Edition 2026.11.31:

```python
"""One label formatted from nine sources, the schema taken from each in turn. The point rests on the printed lines where regopy's JSON is invalid: a source that keeps its quotes breaks the output."""
import json

from regopy import Input, Interpreter

# The same label, sprintf("%s.%s", [schema, input.table]), with the schema "crm" taken from nine places.
# crm is a rule and schema_of a function, two of the nine sources
helpers = 'crm := "crm"\nschema_of(x) := "crm" if x == 1\n'
cases = [
    ("from input.schema", "input.schema", "true"),
    ("from a rule", "crm", "true"),
    ("from a function", "schema_of(1)", "true"),
    ("from iterating over a set", "s", 'some s in {"crm"}'),
    ("from object.get, the key found", 'object.get(input, "schema", "public")', "true"),
    # namespace is not in the input, so object.get returns its default, crm
    ("from object.get, its default", 'object.get(input, "namespace", "crm")', "true"),
    ("from lower(input.schema)", "lower(input.schema)", "true"),
    ("from concat", 'concat("", ["c", "rm"])', "true"),
    ("from a local variable set to a literal", "v", 'v := "crm"'),
]
for name, schema, body in cases:
    rego = Interpreter()
    rego.add_module("example", f'package example\n{helpers}label := sprintf("%s.%s", [{schema}, input.table]) if {{ {body} }}\n')
    rego.set_input(Input({"schema": "crm", "table": "learners"}))
    try:
        print(f"{name}:", json.loads(str(rego.query("x := data.example.label")))["bindings"]["x"])
    except json.JSONDecodeError as error:
        print(f"{name}, regopy's invalid JSON:", error.doc)
```

Output:

```text
from input.schema: crm.learners
from a rule: crm.learners
from a function: crm.learners
from iterating over a set: crm.learners
from object.get, the key found: crm.learners
from object.get, its default, regopy's invalid JSON: {"expressions":[true], "bindings":{"x":""crm".learners"}}
from lower(input.schema), regopy's invalid JSON: {"expressions":[true], "bindings":{"x":""crm".learners"}}
from concat, regopy's invalid JSON: {"expressions":[true], "bindings":{"x":""crm".learners"}}
from a local variable set to a literal, regopy's invalid JSON: {"expressions":[true], "bindings":{"x":""crm".learners"}}
```

- **Replacing the whole input breaks every query.** One extra test that writes its request as `with input as {...}`, and evaluates rules that call built-ins such as `sprintf`, makes every query in that interpreter fail. Every test then ends in ERROR, 13 of 13, and so does the first query asked of a fresh interpreter loaded with the same files, before any test runs: even the query `true` fails. Replacing one path at a time, as the platform's tests do, works.
- **Printing a failed output crashes Python.** Converting one of those failed outputs to a string ends the process with `SIGSEGV`. So `lib.policy` checks `output.ok()` before it reads anything, and a decision asked of that interpreter raises an error instead: `regopy couldn't evaluate data.platform.access.decision, so there is no decision to read`.
- **`sprintf` keeps stray quotes.** Formatting a string returned by some built-ins, such as `object.get`'s default, `lower` or `concat`, or a local variable set to a literal, keeps its quotes. regopy's own output is then invalid JSON, which its Python wrapper fails to parse, as `{"expressions":[true], "bindings":{"x":""crm".learners"}}` shows. Strings from `input`, a rule, a function, iteration and `object.get`'s found value format cleanly, and the platform's policy formats only strings like those.

**Working rule:** Treat regopy as a fast local evaluator and OPA as the reference. Write Rego v1 only, bind every decision to a variable, mock input one path at a time with literal values, and check that an output succeeded before reading it. Before a policy guards production data, check and test it with OPA itself, with `opa check --strict` and `opa test`.

## Write the same rules in Cedar {#the-same-rules-in-cedar}

Rego isn't the only language for policies like the platform's. Cedar's designers present it as a language for expressive, fast, safe and analyzable authorization ([Cutler et al., 2024](https://arxiv.org/abs/2403.04651)), and it takes a narrower view of a decision. A Cedar request names a principal, an action and a resource, and the answer follows three rules: default deny, forbid overrides permit, and skip on error, which the documentation defines as "if a policy's evaluation returns error, the policy does not factor into the authorization response; it is skipped" ([Cedar, authorization](https://docs.cedarpolicy.com/auth/authorization.html)). A schema declares every type of entity, the kinds of principal and resource a request can name, with its attributes, and attributes are required unless the schema marks them optional ([Cedar, schema](https://docs.cedarpolicy.com/schema/schema.html)). Validation checks policies against a schema before any request, and a set that validates avoids most of the errors that evaluation could raise, for requests that match the schema ([Cedar, validation](https://docs.cedarpolicy.com/policies/validation.html)). The book's examples run Cedar with cedarpy 4.12.2, which wraps the Rust Cedar Policy library for Python ([cedarpy](https://pypi.org/project/cedarpy/4.12.2/)).

The platform's rules in Cedar are in `governance/policies/access.cedar`, with the schema in `access.cedarschema`. A `permit` per group repeats the Rego policy's [`readable` map](#deny-by-default-then-allow-by-role). A `forbid` keeps support agents to learners in their own region, and another hides sensitive columns from everyone except finance's view of card numbers. `lib.policy.cedar_entities` generates the users, groups, tables and columns from the same [data document](#generate-the-data-the-policy-reads) the Rego policy reads, and the code adds the first learner of `crm.learners` in each region:

Example `cedar-same-rules` (Python 3.12.15 · cedarpy 4.12.2 · pyarrow 26.0.1 · pyiceberg 0.12.0 · regopy 1.5.2), passing in Edition 2026.11.31:

```python
"""The platform's Cedar policies checked against their schema, then asked the questions the Rego policy answers, over entities generated from its data. The points rest on the printed decisions, and on the misspelt-region error the schema gives before any request."""
import cedarpy

from lib import policy
from lib.platform import load

policies = policy.CEDAR_POLICIES.read_text(encoding="utf-8")
schema = policy.CEDAR_SCHEMA.read_text(encoding="utf-8")
print("policies valid against the schema:", cedarpy.validate_policies(policies, schema).validation_passed)
# misspells region in the policy's text, so the schema reports the attribute that User lacks
typo = cedarpy.validate_policies(policies.replace("principal.region", "principal.regoin"), schema)
# the validator's first error
print("with region misspelt:", typo.errors[0].error)

# Entities generated from the files the Rego policy reads, plus the first learner of crm.learners in each region.
first = {}
for learner in load("crm.learners").select(["learner_id", "support_region"]).to_pylist():
    # setdefault keeps the first learner seen in each region
    first.setdefault(learner["support_region"], learner)
entities = policy.cedar_entities(learners=first.values())


def ask(user: str, resource: str) -> cedarpy.AuthzResult:
    request = {"principal": f'User::"{user}"', "action": 'Action::"read"', "resource": resource, "context": {}}
    return cedarpy.is_authorized(request, policies, entities, schema)


for user, kind, name in [("finance-1", "Column", "billing.payments.card_number"),
                         ("analyst-1", "Column", "billing.payments.card_number"),
                         ("analyst-1", "Column", "billing.payments.amount"),
                         ("marketing-1", "Table", "billing.payments")]:
    print(f"{user} reads {name}:", ask(user, f'{kind}::"{name}"').decision.value)
for region, learner in sorted(first.items()):
    print(f"agent-a05 (emea) reads a learner in {region}:", ask("agent-a05", f'Learner::"{learner["learner_id"]}"').decision.value)
answer = ask("finance-1", 'Column::"billing.payments.card_number"')
# the policies that decided the answer, and those skipped on an error, come from its diagnostics
print("everything Cedar returns for finance-1's card numbers:", answer.decision.value, list(answer.diagnostics.reasons), list(answer.diagnostics.errors))
```

Output:

```text
policies valid against the schema: True
with region misspelt: for policy `policy6`, attribute `regoin` on entity type `User` not found
finance-1 reads billing.payments.card_number: Allow
analyst-1 reads billing.payments.card_number: Deny
analyst-1 reads billing.payments.amount: Allow
marketing-1 reads billing.payments: Deny
agent-a05 (emea) reads a learner in americas: Deny
agent-a05 (emea) reads a learner in apac: Deny
agent-a05 (emea) reads a learner in emea: Allow
everything Cedar returns for finance-1's card numbers: Allow ['policy3'] []
```

The decisions agree with the Rego policy wherever Cedar can express them. Finance may read the `card_number` column, and the data scientist may not, where Rego allows the read and redacts the column. The data scientist may read `amount`, and marketing may not read payments. The `emea` agent may read the `emea` learner and neither of the others. The policies validate against the schema too, and misspelling `region` gives a precise error before anything runs: for policy `policy6`, attribute `regoin` on entity type `User` not found.

Rego has no such check by default. The same typo in an attribute, `user.regoin` in the support agent's row filter, is undefined in OPA and regopy alike, and `opa check --strict` passes it, because OPA checks references against a schema only when you give it one ([OPA policy language, schemas](https://www.openpolicyagent.org/docs/policy-language#using-schemas-to-enhance-the-rego-type-checker)), and the platform gives none. The agent's row filter disappears, and only a test that compares the filters, such as `test_support_reads_only_its_own_region`, catches it.

Cedar returns a decision and little else. The last line of the output is everything it gives back for finance's card numbers: the decision, and the ID of the policy that made it. cedarpy names policies by position, `policy3` for the fourth, not by their `@id` annotations such as `finance-tables`. Cedar's documentation says `@id` means nothing to the language itself: its CLI uses it to set policy IDs, and other interfaces set them another way ([Cedar, syntax](https://docs.cedarpolicy.com/policies/syntax-policy.html)). Nothing in the answer says `last4`, the [mask](#return-the-decision-as-one-document) the Rego policy returns for finance's card numbers, because Cedar can permit a column but can't say how to show it. That rule would move into the enforcement code, outside the policy and its tests. Rows are the same. Cedar decides one learner at a time, so filtering the platform's learners means one request per learner, or Cedar's experimental partial evaluation, which returns residual policies for the parts it can't yet decide, and whose result is not a final authorization decision ([cedarpy](https://pypi.org/project/cedarpy/4.12.2/)). Cedar's paper makes the trade-off from the other side: "Rego's added expressiveness is a blessing but also a curse". It adds that Rego provides no built-in authorization concepts such as `allow` or `deny`. The same paper reports Cedar 42.8 to 80.8 times faster than Rego at the median, on three workloads with transitive relations, with Cedar 3.0.1 against OPA's Rego 0.61.0. Those are its designers' measurements, not this book's, and they say nothing about regopy.

**Working rule:** Choose Cedar when every question is "may this principal do this to this resource?", as in an application's permission checks, and you want a schema to check every policy before it runs. Choose Rego when the answer has to carry data, as a warehouse's row filters and column masks do, or when the engine already asks OPA, as Trino does. For the platform's warehouse, we recommend Rego.

## What goes wrong in production {#policies-in-production}

We consider the most damaging failure a rule that silently doesn't apply because an attribute is missing, and it happens in both languages. In Rego, "Expressions that refer to undefined values are also undefined" ([OPA policy language](https://www.openpolicyagent.org/docs/policy-language)). The support agent's [row filter](#narrow-by-attribute) takes its value from `user.region`, so for a user without a region the filter doesn't exist at all. A rule that allows by group still allows, and no filter means every row.

Take a copy of the platform's policy and tests, in a temporary folder, and remove the one line, `not missing_attribute`, from [`allow`](#deny-by-default-then-allow-by-role). Then add a support agent with their group and no region, as happens when someone is given a role before anyone sets their region, and ask both policies for the agent's tickets:

Example `missing-region-guard` (Python 3.12.15 · pyarrow 26.0.1 · pyiceberg 0.12.0 · regopy 1.5.2), passing in Edition 2026.11.31:

```python
"""The support policy with and without its guard, asked for a support agent's tickets. The prose's points rest on the printed reasons, row filters and regions seen with each version, and on the test the copy without the guard fails."""
import tempfile
from pathlib import Path

from lib import policy
from lib.platform import load

# A support agent added to users.toml with their group, before anyone has set their region.
data = policy.data_document()
data["platform"]["users"]["agent-new"] = {"groups": ["support"]}
request = policy.request("agent-new", "support.tickets", ["ticket_id", "body"])
# every region with a ticket, the candidates for what the agent may see
regions = sorted(set(load("support.tickets").column("support_region").to_pylist()))

# A copy of the platform's policy and tests, with one line taken out of `allow`: the missing-attribute guard.
with tempfile.TemporaryDirectory() as name:
    folder = Path(name)
    for path in policy.POLICIES.glob("*.rego"):
        # the guard's line in allow is indented with a tab, and this removes it
        (folder / path.name).write_text(path.read_text(encoding="utf-8").replace("\tnot missing_attribute\n", ""), encoding="utf-8")

    for label, policies in [("with the guard", policy.POLICIES), ("without the guard", folder)]:
        decision = policy.decide(request, policy.interpreter(policies, data=data))
        # a region counts when the decision allows the read and every row filter names it, so no filters means every region
        seen = [r for r in regions if decision["allow"] and all(f["equals"] == r for f in decision["row_filters"])]
        print(f"{label}, allowed:", decision["allow"])
        print(f"{label}, reasons:", "; ".join(decision["reasons"]))
        print(f"{label}, row filters:", decision["row_filters"])
        print(f"{label}, regions whose tickets the agent sees:", ", ".join(seen) or "none")

    failing = [test for test, result in policy.run_tests(folder).items() if result != "PASS"]
print("tests that fail without the guard:", ", ".join(failing))
```

Output:

```text
with the guard, allowed: False
with the guard, reasons: support's row filter needs the user's region, and there is none
with the guard, row filters: []
with the guard, regions whose tickets the agent sees: none
without the guard, allowed: True
without the guard, reasons: support may read support.tickets; support's row filter needs the user's region, and there is none
without the guard, row filters: []
without the guard, regions whose tickets the agent sees: americas, apac, emea
tests that fail without the guard: test_support_without_a_region_is_denied
```

Without the guard, the agent is allowed, with row filters `[]`, and sees the tickets of `americas, apac, emea`, including every learner's message and contact details. The reasons already say the region is missing; nothing in `allow` asked. With the guard, the agent is denied, and the reason names what is missing: support's row filter needs the user's region, and there is none. `default allow := false` didn't help either, because a rule did allow. One test catches it, `test_support_without_a_region_is_denied`, which is why the policy's tests include the case of the missing attribute.

Cedar fails the same way, through [skip on error](#the-same-rules-in-cedar). Strip the `has region` check from the support agents' `forbid` in the platform's Cedar policies, and it reads `principal.region` on an agent who has none:

Example `cedar-missing-region` (Python 3.12.15 · cedarpy 4.12.2 · pyarrow 26.0.1 · pyiceberg 0.12.0 · regopy 1.5.2), passing in Edition 2026.11.31:

```python
"""Cedar's support forbid with and without its `has region` check, asked about a support agent with no region. The points rest on the printed decisions with each version, and on the validation error the unguarded policy gets."""
import cedarpy

from lib import policy
from lib.platform import load

guarded = policy.CEDAR_POLICIES.read_text(encoding="utf-8")
unguarded = guarded.replace("principal has region && ", "")            # the same forbid, without its `has` check
schema = policy.CEDAR_SCHEMA.read_text(encoding="utf-8")

# A support agent with no region yet, as when a role is given before anyone sets one, and the first learner of crm.learners in each region.
data = policy.data_document()
data["platform"]["users"]["agent-new"] = {"groups": ["support"]}
first = {}
for learner in load("crm.learners").select(["learner_id", "support_region"]).to_pylist():
    first.setdefault(learner["support_region"], learner)
entities = policy.cedar_entities(learners=first.values(), data=data)

for name, policies in [("with `has region`", guarded), ("without it", unguarded)]:
    for region, learner in sorted(first.items()):
        request = {"principal": 'User::"agent-new"', "action": 'Action::"read"',
                   "resource": f'Learner::"{learner["learner_id"]}"', "context": {}}
        answer = cedarpy.is_authorized(request, policies, entities, schema)
        print(f"{name}, a learner in {region}:", answer.decision.value)
    # answer is the last learner's request, in emea, the last region in sorted order
    print(f"{name}, policies skipped on an error:", list(answer.diagnostics.errors))
# the first error the validator returns for the unguarded policies
print("validation without it:", cedarpy.validate_policies(unguarded, schema).errors[0].error)
```

Output:

```text
with `has region`, a learner in americas: Deny
with `has region`, a learner in apac: Deny
with `has region`, a learner in emea: Deny
with `has region`, policies skipped on an error: []
without it, a learner in americas: Allow
without it, a learner in apac: Allow
without it, a learner in emea: Allow
without it, policies skipped on an error: ['error while evaluating policy `policy6`: `User::"agent-new"` does not have the attribute `region`']
validation without it: for policy `policy6`, unable to guarantee safety of access to optional attribute `region` on entity type `User`
```

Without the check, Cedar lets a support agent with no region read the learners of all three regions. Evaluating the `forbid` raises an error, Cedar skips that policy, and the support `permit` still applies. With the check, the same agent is denied every learner. Passing the schema to `is_authorized` didn't stop it, because the schema declares `region` optional, since users outside support have none. Validation did, before any request: for policy `policy6`, unable to guarantee safety of access to optional attribute `region` on entity type `User`.

Other failures to watch for:

- **A misspelt attribute is undefined, not an error.** `user.regoin` for `user.region` passes `opa check --strict`, and the support filter disappears as it does for a missing region. Compare whole filters in the tests, and give OPA a schema if you want the check before any test runs.
- **A deny rule that reads a missing attribute fails open.** The guard fixes `allow`; a `deny` or `forbid` needs the same care. Deny explicitly when a required attribute is absent, and write the test for it before the rule.
- **Policies nobody can test.** Rules kept in an engine's settings or in hand-made views per role can't be unit-tested or diffed, so they drift. Keep the policy, its tests and its decision table in the repository, and run all three on every change.
- **regopy passes and OPA refuses.** Pre-1.0 syntax and unbound variables run in regopy. Check every policy with `opa check --strict` before deploying it to OPA.
- **Data that changed under the policy.** A support agent moved to another region is one edit to `users.toml`, and the decision follows only when the data document is rebuilt. Rebuild it whenever its sources change, and deploy it with the policy.

## What this chapter decided {#policies-decisions}

- **One policy, in Rego,** `governance/policies/access.rego`, denies by default, allows by group, and narrows by three attributes: the agent's region, the learner's consent and the request's purpose.
- **Requests are shaped like Trino's OPA requests,** built by `lib.policy.request`, plus an optional top-level `purpose`; groups come from `users.toml`, not from the request.
- **The policy's data is generated** by `lib.policy.data_document` from `users.toml`, `metadata.toml` and `learning.courses`, and never typed.
- **Each decision is one document** with `allow`, `row_filters`, `column_masks` and `reasons`. Filters and masks are data, never SQL, and the masks are `redact`, `last4`, `pseudonymize` and `redact_contacts`. A denial carries no filters or masks.
- **Missing facts fail closed:** a column the metadata doesn't classify is treated as sensitive, and a user whose row filter needs a missing attribute is denied.
- **The one purpose the policy knows is `ai-training`,** which limits reviews to learners who agreed to AI training and shows their text with contact details removed. A read without a purpose gets what the role allows, with review text hidden.
- **Two kinds of test run on every change:** the Rego unit tests through `lib.policy.run_tests`, and the who-sees-what table, `lib.policy.WHO_SEES_WHAT`.
- **regopy for the examples, OPA as the reference:** Rego v1 only, every decision bound to a variable, input mocked one path at a time with literals, and every output checked before it's read.
- **Cedar is for comparison** (`governance/policies/access.cedar`). It suits allow-or-deny checks on single resources; the platform's warehouse needs filters and masks as data, so it uses Rego. An access policy is only as clear as its tests are thorough. Writing the rules as code means a change to who sees what is a diff a reviewer can read, and a test can run before anyone relies on it.

# Enforcing policies on queries

By the end of this chapter you can enforce access decisions on every query the platform runs. You'll rewrite each query so that every table it reads is replaced by a subquery that keeps only the rows the user may see and masks the columns they may not, deny every query that reads anything else, test who gets what by running real queries, and say where a rewrite in front of DuckDB stops protecting anything.

A policy protects the online learning platform's warehouse only if it holds for every query that reaches it, and that warehouse keeps learners' contact details, their payments and the text they write. [Chapter 5](#access-policies-as-code) wrote the platform's access policy in Rego. Given a user, a table and the columns they want, and optionally why they want them, the policy returns a [**decision**](#access-policies-as-code): whether to `allow` the read, which **row filters** keep the rows the user may see, which **column masks** change how a column is shown (`redact`, `last4`, `pseudonymize` or `redact_contacts`), and the `reasons`. A decision does nothing on its own. A support agent's SQL reaches DuckDB as text, so something between the two must apply the decision to every table that text reads, however the query is written.

The book's shared module `lib.enforce` enforces decisions; we call it the **enforcement layer**. Its `Enforcer` holds a DuckDB connection with a view for each catalog table (`lib.warehouse.connect`), the tables' schema, the policy and a key for [pseudonyms](#consistent-pseudonyms). Users never get the connection. `Enforcer.sql(query, user)` rewrites a user's query and runs the rewrite. A service exposes only `lib.enforce.answer`, which calls it and returns the rows: never the relation, a handle on the connection, or `Enforcer.run`, which executes SQL the layer wrote with the key attached. The code runs sqlglot 30.33.0 and DuckDB 2.0.0.

## Find every table and column a query reads {#find-what-a-query-reads}

A policy can only decide about what it's asked, so the layer has to know every table a query reads and the columns it reads from each. Reading that off SQL text is a parsing problem with traps: `*`, aliases, joins with `USING`, common table expressions (CTEs) and identifiers in any case. sqlglot parses a query into a tree, and its `qualify` step resolves every column against a schema, qualifies it with its table, and expands `*` into the columns it stands for when it has a schema to expand against ([`qualify` at v30.24.0](https://github.com/tobymao/sqlglot/blob/v30.24.0/sqlglot/optimizer/qualify.py)). The layer builds that schema when it starts, listing the catalog's tables and reading each one's columns and types from the view `lib.warehouse.connect` created for it, so a view DuckDB holds for anything else never enters the schema.

Four queries, each with a trap, show what `Enforcer.reads` returns: the columns the policy is asked about, which must match what each query names.

Example `tables-a-query-reads` (Python 3.12.15 · duckdb 2.0.0 · pyarrow 26.0.1 · pyiceberg 0.12.0 · regopy 1.5.2 · sqlglot 30.33.0), passing in Edition 2026.11.31:

```python
"""The columns four queries read from each table, and how each dialect normalizes the quoted name "LEARNERS". The
prose's point rests on the query with a CTE and on the normalization lines at the end."""
import sqlglot
from sqlglot import exp
from sqlglot.optimizer.normalize_identifiers import normalize_identifiers

from lib.catalog import open_catalog
from lib.enforce import Enforcer

# only these two tables are loaded, so the layer's schema holds nothing else
platform = Enforcer(open_catalog(["crm.learners", "support.tickets"]))

# What each query reads, found by sqlglot's qualify against the catalog's schema, with `*` expanded.
queries = {
    "every column": "select * from crm.learners",
    "a join": """select l.country, count(*) as tickets
                 from crm.learners as l join support.tickets as t using (learner_id) group by all""",
    # exclude drops notes from the CTE, so the reads name every other column of crm.learners
    "a CTE": """with emea as (select * exclude (notes) from crm.learners where support_region = 'emea')
                select email from emea""",
    "names in capitals": 'select EMAIL from "CRM"."LEARNERS"',
}
for name, sql in queries.items():
    for table, columns in platform.reads(sql).items():
        print(f"{name}, {table}: {', '.join(columns)}")

# DuckDB resolves quoted capitals to the lowercase table. Whether sqlglot lowercases them depends on the dialect.
capitals = 'select count(*) from "CRM"."LEARNERS"'
print('DuckDB, "CRM"."LEARNERS":', f"{platform.connection.sql(capitals).fetchall()[0][0]:,} rows")
for dialect in ["duckdb", "trino", "postgres"]:
    # the name the dialect's parser resolves the quoted identifier to
    table = normalize_identifiers(sqlglot.parse_one('select * from "LEARNERS"', read=dialect), dialect=dialect).find(exp.Table)
    print(f'{dialect} normalizes "LEARNERS" to:', table.name)
```

Output:

```text
every column, crm.learners: ai_training_consent, city, closed_at, country, date_of_birth, email, full_name, learner_id, marketing_consent, notes, phone, signup_at, status, street_address, support_region
a join, crm.learners: country, learner_id
a join, support.tickets: learner_id
a CTE, crm.learners: ai_training_consent, city, closed_at, country, date_of_birth, email, full_name, learner_id, marketing_consent, phone, signup_at, status, street_address, support_region
names in capitals, crm.learners: email
DuckDB, "CRM"."LEARNERS": 5,000 rows
duckdb normalizes "LEARNERS" to: learners
trino normalizes "LEARNERS" to: learners
postgres normalizes "LEARNERS" to: LEARNERS
```

The reads follow what a query names, not what it returns. `SELECT *` reads every column of `crm.learners`, and the join reads `learner_id` from both tables, because `USING` compares them, though neither appears in the output. The CTE returns only `email`, but it names every column of `crm.learners` except `notes`, so the layer asks about all of those. A query's reads can be more than the query returns, and never fewer. `"CRM"."LEARNERS"` resolves to `crm.learners` as well, because the layer normalizes identifiers the way DuckDB resolves them: DuckDB returns 5,000 rows of `crm.learners` for that name. The dialect matters. sqlglot lowercases `"LEARNERS"` for DuckDB and Trino, and keeps it as LEARNERS for PostgreSQL, so the layer normalizes with the dialect of the engine that runs the query.

`qualify` has one gap that decides the layer's design: it fails on an unknown column of a known table, but it passes over a table that isn't in the schema at all, and leaves that table's `*` unexpanded. Run it on a join with `marketing.audience`, which stands for a table a pipeline wrote after the schema was read, then send that query and five other reads through the layer:

Example `unknown-tables-denied` (Python 3.12.15 · duckdb 2.0.0 · pyarrow 26.0.1 · pyiceberg 0.12.0 · regopy 1.5.2 · sqlglot 30.33.0), passing in Edition 2026.11.31:

```python
"""Runs sqlglot's qualify on a join with a table the schema lacks, then sends that join and five other reads through
the enforcement layer. The prose's point rests on the qualify line, where the star stays, and on the denial lines."""
import sqlglot
from sqlglot.optimizer.qualify import qualify

from lib.catalog import open_catalog
from lib.enforce import Denied, Enforcer

platform = Enforcer(open_catalog(["crm.learners"]))

# marketing.audience stands for a table a pipeline wrote after the schema was read. qualify alone lets it through.
query = "select * from crm.learners as l join marketing.audience as a on a.learner_id = l.learner_id"
print("qualify on its own:", qualify(sqlglot.parse_one(query, read="duckdb"), schema=platform.schema, dialect="duckdb").sql("duckdb"))

# The enforcement layer denies anything that isn't a catalog table, before it asks the policy.
attempts = {
    "a table missing from the catalog": query,
    "a column the table doesn't have": "select shoe_size from crm.learners",
    # scan_crm_learners is the scan lib.warehouse registers for crm.learners, and DuckDB resolves it like a table
    "the scan behind a view": "select count(*) from scan_crm_learners",
    "a Parquet file": "select count(*) from read_parquet('data/platform/learners.parquet')",
    "a table function": "select count(*) from query_table('crm.learners')",
    "a statement that isn't a query": "summarize crm.learners",
}
for name, sql in attempts.items():
    try:
        platform.rewrite(sql, "analyst-1")
        print(f"{name}: allowed")
    except Denied as reason:
        print(f"{name}: denied, {reason}")

# What those reads would do on the layer's own connection, without the rewrite in front of it.
rows = lambda sql: f"{platform.connection.sql(sql).fetchall()[0][0]:,} rows"
print("the scan, run directly:", rows("select count(*) from scan_crm_learners"))
print("the table function, run directly:", rows("select count(*) from query_table('crm.learners')"))
try:
    rows("select count(*) from read_parquet('data/platform/learners.parquet')")
except Exception as error:
    print("the file, run directly:", str(error).split(":")[0])
# the layer turned external access off, and DuckDB refuses to turn it back on while the database runs
try:
    platform.connection.execute("set enable_external_access = true")
except Exception as error:
    print("turning external access back on:", str(error).split(": ", 1)[1])
```

Output:

```text
qualify on its own: SELECT * FROM "crm"."learners" AS "l" JOIN "marketing"."audience" AS "a" ON "a"."learner_id" = "l"."learner_id"
a table missing from the catalog: denied, not a table in the catalog: marketing.audience AS a
a column the table doesn't have: denied, Column 'shoe_size' could not be resolved. Line: 1, Col: 16
the scan behind a view: denied, not a table in the catalog: scan_crm_learners
a Parquet file: denied, not a table in the catalog: READ_PARQUET('data/platform/learners.parquet')
a table function: denied, not a table in the catalog: QUERY_TABLE('crm.learners')
a statement that isn't a query: denied, only a single query is allowed
the scan, run directly: 5,000 rows
the table function, run directly: 5,000 rows
the file, run directly: Permission Error
turning external access back on: Cannot enable external access while database is running
```

`qualify` on its own returns the query with `SELECT *` intact and no error, so a layer that trusted it would ask the policy about `crm.learners` and pass `marketing.audience` through unchecked. The layer denies every table outside the catalog before it asks the policy anything, and `qualify` itself denies a column the table doesn't have. The other four reads show why that rule is broad. `lib.warehouse` registers each table's scan under a name of its own, such as `scan_crm_learners`. Run directly on the layer's connection, that scan returns 5,000 rows of raw learners. `query_table` reads whatever table its string names, and returns 5,000 rows when run directly. `read_parquet` reads a file, and `SUMMARIZE` isn't a query the rewrite can analyse. The first three aren't catalog tables and the last isn't a query, so all four are denied. The layer's connection also has DuckDB's `enable_external_access` turned off, so a file read that got past the rewrite would still fail, with a `Permission Error` ([Securing DuckDB](https://duckdb.org/docs/2.0/operations_manual/securing_duckdb/overview)).

**Working rule:** Ask the policy about every table and column a query names, with `*` expanded against the catalog's schema, and deny whatever that schema doesn't hold: unknown tables, files, table functions and statements other than a single query. A denial costs one question to the platform team; a pass-through costs whatever the table holds.

## Allow only catalog tables, CTEs and subqueries {#allow-only-catalog-sources}

[Denying names the catalog doesn't hold](#find-what-a-query-reads) isn't enough on its own, because a query can read data through sources that aren't table names. A table function can sit under `LATERAL`, or in a subquery in the select list. Outside a CTE's scope, or before the CTE is defined, DuckDB resolves its name to whatever else the connection holds under that name. A check that inspects only table names, and accepts any name a CTE uses, passes both.

So the layer checks sources against an allowlist. `Enforcer.check_sources` walks the whole query and requires every `FROM` and `JOIN` source, at every level, to be a catalog table, a CTE in scope or a subquery, and a `LATERAL` to be over a subquery. Everything else is denied, including table functions wherever they appear, `VALUES`, `UNNEST` and joins in parentheses. A CTE is in scope in its own query, below it and in later CTEs, and a recursive one also in its `UNION`'s recursive branch. An allowlist fails closed on a kind of source nobody thought of.

A query through the layer must change nothing. DuckDB's scalar `write_log` writes into the engine's [query log](#cross-check-the-query-log). `duckdb_functions()` marks such functions: `has_side_effects` is "`true` if this function changes the database state" ([DuckDB, `duckdb_functions`](https://duckdb.org/docs/2.0/sql/meta/duckdb_table_functions)). At 2.0.0 it also marks functions that only vary, such as `random` and `uuid`. The layer denies every flagged function, and every macro that calls one, such as `pg_sleep`, reading the list from `duckdb_functions()` when it starts. A list sorted by hand into harmful and harmless can go stale on an upgrade without anyone noticing, and a governed read seldom needs a random number.

`lib.enforce` keeps the cases as a regression test, `source_cases`, which [chapter 10's CI](#run-every-rule-as-one-job) runs along with [the order test](#gate-columns-on-row-filters) below. Running them shows each denial and whether anything was logged:

Example `catalog-sources-only` (Python 3.12.15 · duckdb 2.0.0 · pyarrow 26.0.1 · pyiceberg 0.12.0 · regopy 1.5.2 · sqlglot 30.33.0), passing in Edition 2026.11.31:

```python
"""Runs the allowlist's regression cases through the layer and prints each denial with its reason; a case passes when it's
denied exactly when it must be and its parse logs nothing. The prose's points rest on the count of cases that pass, the functions-denied line and the allowed-query line at the end."""
from lib.catalog import open_catalog
from lib.enforce import MUST_ALLOW, MUST_DENY, Enforcer, source_cases

platform = Enforcer(open_catalog(["crm.learners", "support.tickets"]))

# The allowlist's regression test, lib.enforce.source_cases, which chapter 10's CI runs too: queries that read data
# other than through a catalog table, a call to each function DuckDB marks as having side effects, and a query the
# layer must still allow. A case passes when the layer denies it exactly when it must, and its parse logs nothing.
cases = source_cases(platform)
for case in cases:
    if case["case"] in MUST_DENY:
        print(f"{case['case']}:", f"denied, {case['reason']}" if case["denied"] else "ALLOWED")
functions = [case for case in cases if case["must"] == "deny" and case["case"] not in MUST_DENY]
print("functions with side effects:", ", ".join(case["case"].removeprefix("the function ") for case in functions))
print("functions with side effects, denied:", f"{sum(case['denied'] for case in functions)} of {len(functions)}")
print("cases whose parse sqlglot logged, quoting the query:", f"{sum(case['logged'] for case in cases)} of {len(cases)}")
passing = [case for case in cases if case["denied"] == (case["must"] == "deny") and not case["logged"]]
print("cases that pass:", f"{len(passing)} of {len(cases)}")

# What two of the cases do on the layer's own connection, without the layer in front of it.
# the name is used before its CTE is defined, so DuckDB resolves it to the scan of the same name
before_the_cte = platform.connection.sql(MUST_DENY["a CTE's name before the CTE"]).fetchall()[0][0]
print("a CTE's name before the CTE, run directly:", f"{before_the_cte:,} rows")
# turns on DuckDB's query log, the one write_log writes into, so the entries below are what the call left
platform.connection.execute("CALL enable_logging('QueryLog')")
platform.connection.sql(MUST_DENY["a function that writes DuckDB's log"]).fetchall()
logged = [message for (message,) in platform.connection.sql("select message from duckdb_logs where type = 'QueryLog'").fetchall()]
print("the log function, run directly, query-log entries it leaves:", logged)

# the query the allowlist must admit: a CTE, a join and a scalar subquery over catalog tables
for name, query in MUST_ALLOW.items():
    print(f"{name}, allowed:", f"{platform.sql(query, 'agent-a05').fetchall()[0][0]:,} rows")
```

Output:

```text
a table function in FROM: denied, not a table in the catalog: QUERY_TABLE('crm.learners')
a table function in a scalar subquery: denied, not a table in the catalog: QUERY_TABLE('crm.learners')
a table function under LATERAL: denied, a LATERAL that isn't a subquery: LATERAL QUERY_TABLE('crm.learners')
a catalog-qualified name: denied, not a table in the catalog: memory.crm.learners
an information_schema view: denied, not a table in the catalog: information_schema.tables
a CTE's name outside its scope: denied, not a table in the catalog: other
a CTE's name before the CTE: denied, not a table in the catalog: scan_crm_learners
a function that writes DuckDB's log: denied, a function DuckDB marks as having side effects: write_log
a statement sqlglot keeps as text: denied, only a single query is allowed
functions with side effects: current_connection_id, current_query, current_query_id, current_transaction_id, currval, error, gen_random_uuid, nextval, pg_sleep, random, setseed, setval, sleep_ms, stats, uuid, uuidv4, uuidv7, vector_type, write_log
functions with side effects, denied: 19 of 19
cases whose parse sqlglot logged, quoting the query: 0 of 29
cases that pass: 29 of 29
a CTE's name before the CTE, run directly: 5,000 rows
the log function, run directly, query-log entries it leaves: ["select write_log('an entry', log_type := 'QueryLog')", 'an entry']
a CTE, a join and a scalar subquery, allowed: 61 rows
```

Each case the layer must deny is refused before any policy is asked, with a reason that names the source. The parse has to leave no trace of its own. sqlglot logs any statement it keeps as text, such as a `CALL`, values included ([sqlglot source, v30.24.0](https://github.com/tobymao/sqlglot/blob/v30.24.0/sqlglot/parser.py)), so the layer silences the parser's warnings, and every case checks that nothing was logged. 29 of 29 cases pass.

On the layer's own connection, two cases show what their denials prevent. The name used before its CTE reads the scan of the same name, 5,000 rows of raw learners, and `write_log` leaves `["select write_log('an entry', log_type := 'QueryLog')", 'an entry']` in the query log: its own statement, and an entry no query made. Every function with side effects is denied, 19 of 19.

The allowlist also has to let ordinary queries through. Run as `agent-a05`, the EMEA support agent, a CTE, a join and a scalar subquery over catalog tables return 61 rows.

**Working rule:** Check every source against an allowlist of catalog tables, CTEs in scope and subqueries, deny the rest, and deny every function the engine marks as having side effects, read from its own list. Keep the cases as a regression test for every layer change and engine upgrade, and add one for each new kind of source or function.

## Masking column references leaks {#column-masks-leak}

With the columns known, the obvious enforcement rewrites them: wherever the query selects `email` and the decision masks it, select the mask instead. It needs only the column list from [the first section](#find-what-a-query-reads), and it looks right on the queries people test with. It fails because a query can read a column without selecting it, and DuckDB can return a column without naming it.

`Enforcer.mask_select_list` does the column rewrite, kept only to show its failure. We test it with five reads aimed at learner 2's email, run as `analyst-1`, a data scientist who may read `crm.learners` with `email` and the other personal columns pseudonymized and free text such as `notes` redacted. The first print is the rewrite of a plain read, with any stored email replaced by a placeholder, since the examples never print one. Predict how many of the five the rewrite stops:

Example `select-list-masks-leak` (Python 3.12.15 · duckdb 2.0.0 · pyarrow 26.0.1 · pyiceberg 0.12.0 · regopy 1.5.2 · sqlglot 30.33.0), passing in Edition 2026.11.31:

```python
"""Masks the email in the select list only, then runs five reads aimed at one learner's email through that rewrite and
prints what each returns. The prose's point rests on the five attack lines, each of which returns learner 2's ID, the
stored email, or an error that holds it."""
import re

import duckdb

from lib.catalog import open_catalog
from lib.enforce import EMAIL, Enforcer

platform = Enforcer(open_catalog(["crm.learners"]))
# analyst-1, a data scientist, may read crm.learners with `email` pseudonymized. The attacks aim at learner 2.
email = platform.connection.sql("select email from crm.learners where learner_id = 2").fetchall()[0][0]
print("the column rewrite:", platform.mask_select_list("select learner_id, email from crm.learners", "analyst-1"))

attacks = {
    "a WHERE on the raw column": f"select learner_id from crm.learners where email = '{email}'",
    "the whole row as a struct": "select struct_extract(l, 'email') from crm.learners as l where learner_id = 2",
    "COLUMNS(...)": "select columns('^email$') from crm.learners where learner_id = 2",
    "to_json": "select to_json(l) ->> 'email' from crm.learners as l where learner_id = 2",
    "a cast error": "select learner_id from crm.learners where cast(email as integer) = 0",
}
for name, sql in attacks.items():
    try:
        # the column rewrite changes only the select list, so every other clause reads the stored email
        answer = str(platform.run(platform.mask_select_list(sql, "analyst-1")).fetchall())
    except duckdb.Error as error:
        answer = str(error).splitlines()[0]
    print(f"{name}:", re.sub(EMAIL, "<a stored email>", answer))      # the book never prints a stored email
```

Output:

```text
the column rewrite: SELECT "learners"."learner_id" AS "learner_id", SHA256($pseudonym_key || CAST("learners"."email" AS TEXT)) AS "email" FROM "crm"."learners" AS "learners"
a WHERE on the raw column: [(2,)]
the whole row as a struct: [('<a stored email>',)]
COLUMNS(...): [('<a stored email>',)]
to_json: [('<a stored email>',)]
a cast error: Conversion Error: Could not convert string '<a stored email>' to INT32 when casting from source column email
```

The column rewrite stops none of the five. Its rewrite of the plain read is correct: `email` comes back as a hash. The five reads get past it in three ways:

- **A `WHERE` on the raw column** never passes through the select list, so it compares the stored email. The analyst guesses an email and gets back learner 2's ID, `[(2,)]`, which links that person to every row they can join on `learner_id`.
- **The whole row as a struct, `COLUMNS(...)` and `to_json`** return the stored email, `[('<a stored email>',)]`, without any column reference named `email` for the rewrite to find: DuckDB expands the row, or matches column names against a pattern, itself.
- **A cast error** prints a stored email in its message. DuckDB's cast errors include the value, and PostgreSQL's documentation warns that "most casting functions include their input values in the error messages they produce" ([PostgreSQL 18, Rules and Privileges](https://www.postgresql.org/docs/18/rules-privileges.html)).

Masking the column wherever the query names it, in the `WHERE` clause too, would stop the first and the last read. It can't stop the middle three, because they never name the column.

**Working rule:** Don't enforce masks by rewriting column references. However many reference shapes you cover, the engine has more ways to read a column than the rewrite can recognize. Mask where the data comes from instead, so every read sees the masked value, as the [next section](#replace-each-table) does.

## Replace each table with a filtered, masked subquery {#replace-each-table}

[Rewriting column references](#column-masks-leak) leaks because the raw table stays in the query. The fix replaces the table. `Enforcer.rewrite` replaces each catalog table in the query, under the query's own alias for it, with a subquery built from the user's decision:

- every column of the table, in the table's order, so `*`, a struct or `COLUMNS(...)` sees the shape it expects;
- each column the query reads, as stored or masked as the decision says, inside a `CASE` on the row filters that [a later section](#gate-columns-on-row-filters) explains;
- each column it doesn't read, as a typed `NULL`, since the policy wasn't asked about it;
- a `WHERE` clause with every row filter in the decision.

Everything the user's query does, from its `WHERE` to its struct reads, then happens on that subquery. It's the model Trino's OPA plugin follows, where a row filter essentially behaves like an additional `WHERE` clause, as the [section on engines](#engine-enforcement) shows.

Take `instructor-17`, who reads the low ratings among their courses' reviews. The policy keeps the reviews of their own courses and removes contact details from `body`. The printed rewrite shows how:

Example `table-rewrite` (Python 3.12.15 · duckdb 2.0.0 · pyarrow 26.0.1 · pyiceberg 0.12.0 · regopy 1.5.2 · sqlglot 30.33.0), passing in Edition 2026.11.31:

```python
"""Rewrites instructor-17's query over the reviews, prints the rewrite and runs it. The prose's point rests on the two
result lines: how many reviews come back, and how many of them still hold an email."""
import sqlglot

from lib.catalog import open_catalog
from lib.enforce import Enforcer

platform = Enforcer(open_catalog(["learning.reviews"]))

# instructor-17 reads the low ratings of their courses' reviews. The policy filters reviews to their six courses and
# shows `body` with contact details removed.
query = "select course_id, rating, body from learning.reviews where rating <= 2"
# the query as text, with the reviews table replaced by a subquery built from the decision
rewritten = platform.rewrite(query, "instructor-17")
print(sqlglot.transpile(rewritten, read="duckdb", write="duckdb", pretty=True)[0])
print("reviews returned:", len(platform.run(rewritten).fetchall()))
print("of them holding an email:", platform.run(f"select count(*) from ({rewritten}) where body like '%@%'").fetchall()[0][0])
```

Output:

```text
SELECT
  "reviews"."course_id" AS "course_id",
  "reviews"."rating" AS "rating",
  "reviews"."body" AS "body"
FROM (
  SELECT
    CAST(NULL AS BIGINT) AS review_id,
    CAST(NULL AS BIGINT) AS learner_id,
    CASE WHEN source.course_id IN (2, 75, 80, 87, 103, 134) THEN source.course_id END AS course_id,
    CASE WHEN source.course_id IN (2, 75, 80, 87, 103, 134) THEN source.rating END AS rating,
    CASE
      WHEN source.course_id IN (2, 75, 80, 87, 103, 134)
      THEN REGEXP_REPLACE(
        REGEXP_REPLACE(source.body, '[A-Za-z0-9._%+-]+@[A-Za-z0-9-]+(\.[A-Za-z0-9-]+)+', '[email]', 'g'),
        '\+?\(?[0-9][0-9 ().-]{6,}[0-9]',
        '[phone]',
        'g'
      )
    END AS body,
    CAST(NULL AS TIMESTAMPTZ) AS posted_at
  FROM learning.reviews AS source
  WHERE
    source.course_id IN (2, 75, 80, 87, 103, 134)
) AS "reviews"
WHERE
  "reviews"."rating" <= 2
reviews returned: 9
of them holding an email: 0
```

The query returns 9 reviews, and 0 of them hold an email. The instructor's query stays on the outside, qualified but otherwise unchanged. Inside, `course_id` and `rating` come through as stored, and `body` as the `redact_contacts` mask, each in a `CASE` on the row filter, a literal list of the instructor's courses, and the columns the query doesn't read come through as typed `NULL`s.

The five reads that defeated the column rewrite, run by `analyst-1` through the table rewrite, and through `lib.enforce.answer`, the function a service calls at its boundary:

Example `table-rewrite-holds` (Python 3.12.15 · duckdb 2.0.0 · pyarrow 26.0.1 · pyiceberg 0.12.0 · regopy 1.5.2 · sqlglot 30.33.0), passing in Edition 2026.11.31:

```python
"""Sends the five reads from the column rewrite through the table rewrite, and through `answer`, the service's boundary.
It then shows what the analyst receives and what the platform team's log holds. The prose's point rests on the five attack
lines, the operator's log line and the relation line."""
import logging
import re

from lib.catalog import open_catalog
from lib.enforce import EMAIL, Enforcer, QueryFailed, answer

platform = Enforcer(open_catalog(["crm.learners"]))
operator_log = []                     # the platform team's log, kept in a list here so the example can print it
logging.getLogger("lib.enforce").addHandler(handler := logging.Handler())
handler.emit = lambda record: operator_log.append(record.getMessage().splitlines()[0])

# The five attacks on the column rewrite, run by analyst-1 again: through the table rewrite this time, and through
# `answer`, the function a service calls at its boundary.
email = platform.connection.sql("select email from crm.learners where learner_id = 2").fetchall()[0][0]
attacks = {
    "a WHERE on the raw column": f"select learner_id from crm.learners where email = '{email}'",
    "the whole row as a struct": "select struct_extract(l, 'email') from crm.learners as l where learner_id = 2",
    "COLUMNS(...)": "select columns('^email$') from crm.learners where learner_id = 2",
    "to_json": "select to_json(l) ->> 'email' from crm.learners as l where learner_id = 2",
    "a cast error": "select learner_id from crm.learners where cast(email as integer) = 0",
}
# a pseudonym is 64 hex digits, shortened here; the example never prints a stored email
shorten = lambda text: re.sub(r"\b([0-9a-f]{8})[0-9a-f]{56}\b", r"\1…", re.sub(EMAIL, "<a stored email>", text))
for name, sql in attacks.items():
    try:
        # answer returns the rows, or raises QueryFailed with a generic message, as a service sees it
        got = str(answer(platform, sql, "analyst-1"))
    except QueryFailed as error:
        got = f"{type(error).__name__}, {error}"
    print(f"{name}:", shorten(got))
for line in operator_log:
    print("in the operator's log:", shorten(line))      # a pseudonym, 64 hex digits, shortened

# Why `answer` returns rows: the relation Enforcer.sql returns is a handle on the layer's connection, and runs any SQL.
# the relation is a handle on the layer's connection, so its .query runs SQL of its own there
relation = platform.sql("select learner_id from crm.learners", "analyst-1")
print("the relation, running SQL of its own:", shorten(str(relation.query("r", "select email from crm.learners where learner_id = 2").fetchall())))
print("what answer returns:", type(answer(platform, "select learner_id from crm.learners", "analyst-1")).__name__)
```

Output:

```text
a WHERE on the raw column: []
the whole row as a struct: [(None,)]
COLUMNS(...): [(None,)]
to_json: [(None,)]
a cast error: QueryFailed, the query failed: the platform team can see why
in the operator's log: a query by analyst-1 failed in the engine: Conversion Error: Could not convert string 'e4f62f1a…' to INT32 when casting from source column email
the relation, running SQL of its own: [('<a stored email>',)]
what answer returns: list
```

The rewrite stops all five. The `WHERE` returns `[]`, because inside the subquery `email` is already the pseudonym. The struct, `COLUMNS(...)` and `to_json` reads return `NULL`, not the pseudonym, because those queries name no column but `learner_id`. The policy was asked about no other column, and the typed `NULL` gives the rest no value. A read the rewrite can't see fails closed. The cast error now holds a pseudonym instead of an email, and `answer` hides even that: it fetches the rows inside a `try`, turns any engine error into `QueryFailed`, and writes DuckDB's message to the platform team's log, which the example prints after the five reads. The analyst gets `QueryFailed, the query failed: the platform team can see why`.

`answer` returns rows, not a relation, because a DuckDB relation runs any SQL on its connection. The analyst's relation, asked for an email, returns `[('<a stored email>',)]`.

Generating the subquery per query, rather than writing a view per role, is what keeps it in step with the policy. The platform's decisions depend on region, courses, consent and purpose, and a hand-made view for every combination is the "role explosion" NIST describes, where conditions become "numerous roles that are ad hoc and limited in membership" ([NIST SP 800-162](https://doi.org/10.6028/NIST.SP.800-162)). Such views would drift from the policy the first time someone edited only one.

**Working rule:** Replace every protected table with a subquery generated from the decision, under the query's own alias, and leave the user's query otherwise untouched. Give unread columns a typed `NULL`. If users need whole-row reads to show values, ask the policy about every column of each table instead, so the masks cover all of them, at the cost of requests and audit records that name columns the query never used.

## Write each mask as SQL in one place {#write-masks-as-sql}

The policy names a mask; the layer decides what SQL it becomes. Keeping that mapping in one place, the `MASKS` table in `lib.enforce` and the typed `NULL` beside it, means a fix to a mask reaches every table that uses it, and the policy never carries SQL, which would be code arriving as data. The [table rewrite](#replace-each-table) calls `lib.enforce.masked` for every masked column, and prints the SQL for the four masks the policy names:

Example `masks-as-sql` (Python 3.12.15 · duckdb 2.0.0 · pyarrow 26.0.1 · pyiceberg 0.12.0 · regopy 1.5.2 · sqlglot 30.33.0), passing in Edition 2026.11.31:

```python
"""Prints the SQL the enforcement layer writes for each of the four masks the policy names, and counts DuckDB's functions
named like HMAC. The prose's points rest on the pseudonymize line and on the HMAC count."""
import duckdb
from sqlglot import exp

from lib.enforce import masked

# The four masks the policy names, as the SQL the enforcement layer writes for them.
# the column's type, which the redact mask casts its NULL to
for kind, column, type_ in [("redact", "date_of_birth", "DATE"), ("last4", "card_number", "VARCHAR"),
                            ("pseudonymize", "email", "VARCHAR"), ("redact_contacts", "body", "VARCHAR")]:
    # redact is a typed NULL; the other masks' SQL comes from lib.enforce's MASKS table
    print(f"{kind}:", masked(kind, exp.column(column), type_).sql("duckdb"))

# The pseudonymize mask hashes the key followed by the value, because DuckDB has no HMAC function to call.
# DuckDB's function catalog, searched for any name containing hmac, whatever its case
hmac = duckdb.sql("select count(*) from duckdb_functions() where function_name ilike '%hmac%'").fetchall()[0][0]
print("DuckDB functions named like HMAC:", hmac)
```

Output:

```text
redact: CAST(NULL AS DATE)
last4: RIGHT(REGEXP_REPLACE(card_number, '[^0-9]', '', 'g'), 4)
pseudonymize: SHA256($pseudonym_key || CAST(email AS TEXT))
redact_contacts: REGEXP_REPLACE(REGEXP_REPLACE(body, '[A-Za-z0-9._%+-]+@[A-Za-z0-9-]+(\.[A-Za-z0-9-]+)+', '[email]', 'g'), '\+?\(?[0-9][0-9 ().-]{6,}[0-9]', '[phone]', 'g')
DuckDB functions named like HMAC: 0
```

Each mask does one job:

- **`redact`** is a `NULL` cast to the column's type, so the column keeps its type for anything the query does with it, such as a union or a date function.
- **`last4`** keeps the last four digits of a card number, after removing anything that isn't a digit.
- **`pseudonymize`** is a keyed hash, SHA-256 over the key followed by the value, because DuckDB 2.0.0 has no [HMAC](#consistent-pseudonyms) function: `duckdb_functions()` lists 0 named like it. Where your engine has one, prefer HMAC ([RFC 2104](https://www.rfc-editor.org/rfc/rfc2104)).
- **`redact_contacts`** replaces emails and phone numbers in free text with `[email]` and `[phone]`.

Picture someone holding a list of the platform's emails, who hashes each one and looks for it among the pseudonyms `analyst-1` sees in `crm.learners`:

Example `pseudonym-key` (Python 3.12.15 · duckdb 2.0.0 · pyarrow 26.0.1 · pyiceberg 0.12.0 · regopy 1.5.2 · sqlglot 30.33.0), passing in Edition 2026.11.31:

```python
"""Hashes a list of emails the way the layer hashes pseudonyms, with and without its key, then rotates the key and counts
the pseudonyms that change. The prose's points rest on the three count lines: the matches without the key and with it, and
the pseudonyms that change on rotation."""
from lib.catalog import open_catalog
from lib.enforce import Enforcer

# The pseudonyms analyst-1, the platform's data scientist, sees in crm.learners. Someone holding a list of the
# platform's emails hashes each one and looks for it among them, first without the key and then with it.
# an example key, not a secret: production keeps the real one where only the layer can read it
platform = Enforcer(open_catalog(["crm.learners"]), key="example-key-not-a-secret")
seen = platform.sql("select learner_id, email from crm.learners", "analyst-1").to_arrow_table()
platform.connection.register("seen", seen)
# a guess is an email's hash; the second also puts the key first, as the layer hashes a pseudonym
for name, guess in [("without the key", "sha256(email)"), ("with the key", "sha256($pseudonym_key || email)")]:
    matched = platform.run(f"select count(*) from crm.learners where {guess} in (select email from seen)").fetchall()[0][0]
    print(f"pseudonyms matched from a list of emails, {name}: {matched:,} of {seen.num_rows:,}")

# A new key gives every learner a new pseudonym, so data pseudonymized under the old key no longer joins.
platform.key = "example-key-rotated"
rotated = platform.sql("select learner_id, email from crm.learners", "analyst-1").to_arrow_table()
# pairs each learner's old and new pseudonym, row by row
changed = sum(a != b for a, b in zip(seen["email"].to_pylist(), rotated["email"].to_pylist()))
print("pseudonyms that change when the key is rotated:", f"{changed:,} of {seen.num_rows:,}")
```

Output:

```text
pseudonyms matched from a list of emails, without the key: 0 of 5,000
pseudonyms matched from a list of emails, with the key: 5,000 of 5,000
pseudonyms that change when the key is rotated: 5,000 of 5,000
```

The key is what makes a pseudonym a pseudonym. Without it, the hashes match 0 of 5,000 pseudonyms; with it, they match 5,000 of 5,000, so whoever holds the key and a list of emails can name every pseudonym on it. The key belongs to the layer alone. It stays out of the policy, which is code read in pull requests, and out of the query, where a user who chose the key could compute pseudonyms of guessed emails. The layer passes the key to DuckDB as a query parameter, so it stays out of the query text that DuckDB's log records, as [chapter 8](#keep-values-out-of-the-record) shows, and the layer denies any user query that carries a parameter, since one named `$pseudonym_key` would receive the key. In production, keep the key in a secret manager that only the layer can read. Rotate it when it may have leaked, and plan for what rotation does: 5,000 of 5,000 pseudonyms change, so data pseudonymized under the old key no longer joins with data under the new one.

`redact_contacts` is two regular expressions, and it's worth measuring what they remove. The platform's tickets and reviews come with every piece of personal data the generator placed in them, labelled by construction, which [chapter 3](#finding-personal-data) calls [the truth](#measure-against-labelled-spans). Each text goes through the mask here, and the labels count what survives:

Example `redact-contacts-recall` (Python 3.12.15 · duckdb 2.0.0 · pyarrow 26.0.1 · pyiceberg 0.12.0 · regopy 1.5.2 · sqlglot 30.33.0), passing in Edition 2026.11.31:

```python
"""Applies the redact_contacts mask to every ticket and review, then counts the labelled personal data still left in the
masked text, and the phone replacements made in the tickets. The prose's points rest on the 'left in the text' lines, the
replacement count, and the assert on the tickets' card, IBAN and date lines."""
import duckdb
from sqlglot import exp

from lib.enforce import masked
from lib.platform import load

# The redact_contacts mask, run over every ticket and review, against the personal data the generator placed in them
# (pii_spans, labelled by construction).
con = duckdb.connect()
# each table registers under its short name, which the queries below use
for name in ["support.tickets", "learning.reviews", "pii_spans"]:
    con.register(name.split(".")[-1], load(name))
mask = masked("redact_contacts", exp.column("body"), "VARCHAR").sql("duckdb")
# a span counts as left when its exact text is still in the masked body, which the row_id join finds
counts = con.sql(f"""
    with masked as (select 'tickets' as source, ticket_id as row_id, {mask} as body from tickets
                    union all select 'reviews', review_id, {mask} from reviews)
    select source, entity, count(*), count(*) filter (where contains(masked.body, pii_spans.text))
    from pii_spans join masked using (source, row_id) group by all order by all""").fetchall()
# the phone pattern takes every card number, IBAN and date in the tickets, so none is left in the masked text
left_in_tickets = {entity: left for source, entity, _, left in counts if source == "tickets"}
assert all(left_in_tickets[entity] == 0 for entity in ("CREDIT_CARD", "IBAN_CODE", "DATE_TIME"))
for source, entity, placed, left in counts:
    print(f"{source}, {entity}, left in the text: {left:,} of {placed:,}")
# The phone pattern also takes card numbers, IBANs and dates, which are digits too.
replaced = con.sql(f"select sum(len(regexp_extract_all({mask}, '\\[phone\\]'))) from tickets").fetchall()[0][0]
print("[phone] replacements in tickets:", f"{replaced:,}")
```

Output:

```text
reviews, EMAIL_ADDRESS, left in the text: 0 of 120
reviews, PERSON, left in the text: 798 of 798
tickets, CREDIT_CARD, left in the text: 0 of 213
tickets, DATE_TIME, left in the text: 0 of 196
tickets, EMAIL_ADDRESS, left in the text: 0 of 423
tickets, IBAN_CODE, left in the text: 0 of 236
tickets, LOCATION, left in the text: 210 of 210
tickets, PERSON, left in the text: 1,657 of 1,657
tickets, PHONE_NUMBER, left in the text: 0 of 439
[phone] replacements in tickets: 1,520
```

The mask removes contact details and leaves names alone. In the reviews, which instructors and the assistant read through this mask, 0 of 120 emails are left, and 798 of 798 names are.

In the tickets, 0 of 439 phone numbers are left, but the phone pattern makes 1,520 replacements. Card numbers, IBANs and dates are digits too, so the pattern takes them as well, and none of them is left. Names and street addresses stay, so the mask doesn't make text anonymous.

**Working rule:** Keep every mask's SQL in one table in the enforcement layer, and let the policy name masks, never write them. Key every pseudonym with a secret only the layer holds, passed as a parameter. Measure each text mask against labelled data before a policy relies on it, and name masks for what they remove, so nobody reads `redact_contacts` as "anonymized".

## Build row filters from the decision's data {#build-row-filters-from-data}

Masks change how a value is shown; row filters decide which rows a user gets. They arrive in a decision as data, in three shapes: a column equal to a value, a column in a list of values, and a consent check on the learner a row belongs to. `lib.enforce.row_filter` turns each into a SQL condition, with the values written as SQL literals by sqlglot, never pasted into SQL text. It denies the query for a shape it doesn't know or a column the table doesn't have, and the layer checks a consent filter's column against `crm.learners` before it builds the check. [Chapter 5](#return-the-decision-as-one-document) made the policy return filters as data for exactly this reason: nothing a request carries can reach a query as code.

The consent shape is the one that needs care. The course assistant, `assistant-1`, may read reviews only by learners who agreed to their text training the platform's AI, and that consent lives in `crm.learners`, which the assistant may not read. The layer turns it into a semi-join, a test that the review's `learner_id` is among the learners who consented, and runs it with the layer's own access to `crm.learners`:

Example `row-filters-from-data` (Python 3.12.15 · duckdb 2.0.0 · pyarrow 26.0.1 · pyiceberg 0.12.0 · regopy 1.5.2 · sqlglot 30.33.0), passing in Edition 2026.11.31:

```python
"""Prints the condition the layer writes for each row filter in three decisions, then the reviews the assistant reads
through its consent filter, and the denial of its read of the learners table. The prose's points rest on the read counts
and the consent counts, and on the denied read at the end."""
from lib.catalog import open_catalog
from lib.enforce import Enforcer, row_filter

platform = Enforcer(open_catalog(["crm.learners", "learning.reviews", "support.tickets"]))

# Each row filter in a decision, as the condition the enforcement layer writes for it.
for user, table in [("agent-a05", "support.tickets"), ("instructor-17", "learning.reviews"), ("assistant-1", "learning.reviews")]:
    # the decision for this read; its row filters are data, written as SQL in the loop below
    decision = platform.decide(user, {table: set()})[table]
    for rule in decision["row_filters"]:
        print(f"{user}, {table}: {row_filter(rule, platform.columns(table)).sql('duckdb')}")

# A filter's values come from data, so a value that looks like SQL stays a value.
odd = {"column": "support_region", "equals": "emea' or '1' = '1"}
print("a region that looks like SQL:", row_filter(odd, platform.columns("support.tickets")).sql("duckdb"))

# The assistant's consent filter reads crm.learners, which the assistant itself may not read.
total = platform.connection.sql("select count(*) from learning.reviews").fetchall()[0][0]
read = platform.sql("select count(*) from learning.reviews", "assistant-1").fetchall()[0][0]
print("reviews the assistant reads:", f"{read:,} of {total:,}")
# a null consent means no answer was recorded, so the label reads 'never asked'
by_consent = platform.connection.sql("""select coalesce(l.ai_training_consent::varchar, 'never asked'), count(*)
    from learning.reviews r join crm.learners l using (learner_id) group by all order by all""").fetchall()
for consent, reviews in by_consent:
    print(f"reviews whose author's AI-training consent is {consent}: {reviews:,}")
try:
    platform.sql("select count(*) from crm.learners", "assistant-1")
except Exception as reason:
    print("the assistant reading crm.learners:", reason)
```

Output:

```text
agent-a05, support.tickets: source.support_region = 'emea'
instructor-17, learning.reviews: source.course_id IN (2, 75, 80, 87, 103, 134)
assistant-1, learning.reviews: source.learner_id IN (SELECT consent.learner_id FROM crm.learners AS consent WHERE consent.ai_training_consent IS TRUE)
a region that looks like SQL: source.support_region = 'emea'' or ''1'' = ''1'
reviews the assistant reads: 436 of 3,000
reviews whose author's AI-training consent is false: 955
reviews whose author's AI-training consent is never asked: 1,609
reviews whose author's AI-training consent is true: 436
the assistant reading crm.learners: assistant-1 may not read crm.learners: no group of this user may read this table
```

Each filter's values become SQL literals, so a value stays a value, and an unanswered consent counts as no. The agent's region and the instructor's courses become an equality and an `IN` list, and a region written to look like SQL stays one string, its quotes doubled. The consent filter tests `IS TRUE`, so the authors of 1,609 reviews, who signed up before the platform asked the question and have no answer recorded, count as not consenting, along with those who said no.

The assistant reads 436 of 3,000 reviews, which are the ones whose authors said yes. The assistant's own request for `crm.learners` is denied, with the policy's reason: the semi-join's read of that table belongs to the layer, not to the user.

**Working rule:** Build every row filter from the decision's values with a SQL builder, and deny any filter shape or column the layer doesn't recognize. Treat an unknown consent as no consent. When a filter needs a table the user may not read, read it with the layer's own access inside the subquery, and never by widening the user's access.

## Gate every column on the row filters {#gate-columns-on-row-filters}

[The row filters](#build-row-filters-from-data) sit in the subquery's `WHERE`, and the user's query wraps the subquery, so it's tempting to assume DuckDB applies the filters first. SQL promises no order, and an optimizer moves conditions wherever they're cheapest. A user's condition that can fail, such as a cast, can then run on rows the filters drop, and a generic error doesn't help: that the query fails says something about a row the user may not see.

PostgreSQL's documentation names this leak: a function that might fail depending on its arguments can reveal significant information about unseen rows if it runs before the view's row filters. In PostgreSQL, a view marked `security_barrier` keeps "maliciously-chosen functions and operators from being passed values from rows until after the view has done its work", and only functions marked `LEAKPROOF` may still run first ([PostgreSQL 18, Rules and Privileges](https://www.postgresql.org/docs/18/rules-privileges.html)). DuckDB has neither.

So the layer makes the order irrelevant. Every column the query reads leaves the subquery as `CASE WHEN <row filters> THEN <value> END`, as in [the instructor's rewrite](#replace-each-table). The user's conditions then depend on the filters' result: whatever DuckDB evaluates first, on a row the filters drop they see `NULL` in every column.

`lib.enforce.order_cases` tests each filter shape the layer writes: the support agent's equality, the instructor's `IN` list and the course assistant's consent semi-join. Each case copies the filtered table, with a `body` that holds digits in the rows the filter keeps and text in the rows it drops, then runs a query that casts `body` to a number. That cast fails only if it sees a dropped row's value:

Example `row-filter-order` (Python 3.12.15 · duckdb 2.0.0 · pyarrow 26.0.1 · pyiceberg 0.12.0 · regopy 1.5.2 · sqlglot 30.33.0), passing in Edition 2026.11.31:

```python
"""Runs the evaluation-order test for each filter shape the layer writes, with the filter in a WHERE alone and as the layer
writes it. The prose's point rests on the consent semi-join's two lines, and on the count at the end."""
from lib.enforce import order_cases

# The evaluation-order regression test, lib.enforce.order_cases, which chapter 10's CI runs on every DuckDB upgrade.
# For each shape of row filter the layer writes, it takes a real user's filter and a copy of the filtered table whose
# `body` holds digits in the rows the filter keeps and text in the rows it drops. A query that casts `body` to a number
# fails only if the cast sees a dropped row's value. It runs over the filter in a WHERE alone, and over the layer's
# subquery.
cases = order_cases()
for case in cases:
    for form, safe in [("with the filter in WHERE only", case["safe_with_where_only"]),
                       ("as the layer writes it", case["safe_as_written"])]:
        print(f"{case['shape']}, {form}: the cast {'never saw' if safe else 'saw'} a dropped row's value")
print("filter shapes the layer's subquery keeps safe:", f"{sum(case['safe_as_written'] for case in cases)} of {len(cases)}")
```

Output:

```text
equality, with the filter in WHERE only: the cast never saw a dropped row's value
equality, as the layer writes it: the cast never saw a dropped row's value
IN list, with the filter in WHERE only: the cast never saw a dropped row's value
IN list, as the layer writes it: the cast never saw a dropped row's value
consent semi-join, with the filter in WHERE only: the cast saw a dropped row's value
consent semi-join, as the layer writes it: the cast never saw a dropped row's value
filter shapes the layer's subquery keeps safe: 3 of 3
```

Gating each column is what keeps the consent semi-join safe. With the filter in the `WHERE` alone, the equality and the `IN` list were safe, and the consent semi-join wasn't: DuckDB ran the cast on reviews whose authors didn't consent. As the layer writes it, 3 of 3 filter shapes keep the cast away from dropped rows. The first two were safe by the optimizer's choice at 2.0.0, not by any promise, so the test runs on every upgrade. The gate costs evaluating the filters again in each column the query reads.

**Working rule:** Never rely on the engine's evaluation order to keep a user's conditions off rows a filter drops. Make every value a user's query can read depend on the row filters, and test each filter shape on every engine upgrade. Where an engine has a barrier of its own, such as PostgreSQL's `security_barrier` views, use it as well.

## Test who sees what by running queries {#test-who-sees-what}

[Chapter 5](#check-who-sees-what) checks the policy with a who-sees-what table, `lib.policy.WHO_SEES_WHAT`: a list of requests, each with the decision a reviewer expects for it. That tests the decisions, not what users get, which also depends on the rewrite, the masks' SQL and the data. `lib.enforce.observe` runs each case through the layer as the case's user, and compares what comes back with what's stored: how many rows, and which columns differ. A case agrees with its decision when the user is denied exactly when the case says so, gets fewer rows than stored exactly when the case expects a row filter, and gets changed values in exactly the columns the case expects masked:

Example `who-sees-what-in-queries` (Python 3.12.15 · duckdb 2.0.0 · pyarrow 26.0.1 · pyiceberg 0.12.0 · regopy 1.5.2 · sqlglot 30.33.0), passing in Edition 2026.11.31:

```python
"""Runs chapter 5's who-sees-what cases as queries and compares what each user gets with what is stored, then reads
marketing.audience, which has no metadata. The prose's points rest on the agreement count and on the two marketing lines."""
from lib import pipelines, policy
from lib.catalog import open_catalog
from lib.enforce import Enforcer, agrees, observe

catalog = open_catalog(["crm.learners", "learning.reviews", "billing.payments", "learning.enrollments", "learning.courses"])
pipelines.run(catalog, "marketing_audience")          # a copy of learners' names and emails, with no metadata yet
platform = Enforcer(catalog)

# Chapter 5's who-sees-what table, run as queries: what each user gets, against what's stored. A column is "changed"
# when at least one value the user gets differs from the stored one.
agreeing = 0
for case in policy.WHO_SEES_WHAT:
    got = observe(platform, case.user, case.table, case.columns, case.purpose)
    who = case.user + (f" for {case.purpose}" if case.purpose else "")
    if got is None:
        print(f"{who}, {case.table}: denied")
    else:
        print(f"{who}, {case.table}: {got['rows']:,} of {got['stored']:,} rows, changed: {', '.join(got['changed']) or 'none'}")
    agreeing += agrees(case, got)
print("cases where what the user gets agrees with the decision:", f"{agreeing} of {len(policy.WHO_SEES_WHAT)}")

# marketing.audience is a table marketing may read, but nobody has classified its columns, so the policy treats each
# one as sensitive: marketing gets every row, and no value in any column.
rows = platform.sql("select * from marketing.audience", "marketing-1").fetchall()
# a column counts as shown when one of its values isn't None, which is what a redact mask returns
shown = sum(any(value is not None for value in column) for column in zip(*rows))
print("marketing-1, marketing.audience, which has no metadata, rows:", f"{len(rows):,}")
print("marketing-1, marketing.audience, which has no metadata, columns with a value:", f"{shown} of {len(rows[0])}")
```

Output:

```text
agent-a05, crm.learners: 1,581 of 5,000 rows, changed: none
agent-a05, billing.payments: denied
instructor-17, learning.reviews: 69 of 3,000 rows, changed: body
instructor-17, crm.learners: denied
marketing-1, crm.learners: 1,861 of 5,000 rows, changed: phone
finance-1, billing.payments: 17,562 of 17,562 rows, changed: card_number
analyst-1, billing.payments: 17,562 of 17,562 rows, changed: card_number
analyst-1, crm.learners: 5,000 of 5,000 rows, changed: email, notes
analyst-1, learning.reviews: 3,000 of 3,000 rows, changed: body
analyst-1 for ai-training, learning.reviews: 436 of 3,000 rows, changed: body
assistant-1, learning.reviews: 436 of 3,000 rows, changed: body
someone-else, crm.learners: denied
cases where what the user gets agrees with the decision: 12 of 12
marketing-1, marketing.audience, which has no metadata, rows: 1,829
marketing-1, marketing.audience, which has no metadata, columns with a value: 0 of 5
```

What users get agrees with the decision in 12 of 12 cases. The EMEA support agent gets their region's learners as stored, marketing the consenting, active learners with `phone` changed, and the analyst every learner with `email` and `notes` changed. The last two lines cover `marketing.audience`, which a pipeline writes and nobody has classified yet. Marketing may read it, and gets 1,829 rows with 0 of 5 columns holding a value, because the policy treats an unclassified column as sensitive.

The test earns its place on a change. Suppose a pull request adds `analyst-1` to the support group for EMEA, so they can help with tickets. It's a two-line change to `governance/users.toml`, a group and the region support's row filter needs. The code below makes that change in memory and runs the same cases against it:

Example `group-change-caught` (Python 3.12.15 · duckdb 2.0.0 · pyarrow 26.0.1 · pyiceberg 0.12.0 · regopy 1.5.2 · sqlglot 30.33.0), passing in Edition 2026.11.31:

```python
"""Makes one change to the users file in memory, runs the who-sees-what cases again, and runs chapter 5's decision check
on the same change. The prose's points rest on the lines that say which cases fail, and on the decision count."""
from lib import policy
from lib.catalog import open_catalog
from lib.enforce import Enforcer, agrees, observe

# A pull request changes one user in governance/users.toml, in two lines: analyst-1 joins support, with the region
# support's row filter needs, EMEA, to help with tickets.
# the users file as a document, changed here in memory and not on disk
data = policy.data_document()
data["platform"]["users"]["analyst-1"] = {"groups": ["data-science", "support"], "region": "emea"}
platform = Enforcer(open_catalog(["crm.learners", "learning.reviews", "billing.payments"]),
                    rego=policy.interpreter(data=data))

# The who-sees-what cases, run as queries with the change in place. The cases are unchanged: they say what each
# user should get, and the change was meant to touch support tickets only.
failing = 0
for case in policy.WHO_SEES_WHAT:
    got = observe(platform, case.user, case.table, case.columns, case.purpose)
    # got is None when the user is denied, which agrees only with a case that expects a denial
    if not agrees(case, got):
        failing += 1
        who = case.user + (f" for {case.purpose}" if case.purpose else "")
        print(f"fails, {who}, {case.table}, rows: {got['rows']:,} of {got['stored']:,}")
        print(f"fails, {who}, {case.table}, columns changed: {', '.join(got['changed']) or 'none'}, "
              f"expected: {', '.join(case.masked)}")
print("cases that fail:", f"{failing} of {len(policy.WHO_SEES_WHAT)}")

# Chapter 5's table, which checks the decisions themselves, on the same change.
decisions = [policy.decide(policy.request(c.user, c.table, c.columns, c.purpose), platform.rego) for c in policy.WHO_SEES_WHAT]
print("cases that fail as decisions:", f"{sum(not policy.matches(c, d) for c, d in zip(policy.WHO_SEES_WHAT, decisions))} of {len(decisions)}")
```

Output:

```text
fails, analyst-1, crm.learners, rows: 1,581 of 5,000
fails, analyst-1, crm.learners, columns changed: none, expected: email, notes
fails, analyst-1, learning.reviews, rows: 3,000 of 3,000
fails, analyst-1, learning.reviews, columns changed: none, expected: body
fails, analyst-1 for ai-training, learning.reviews, rows: 436 of 3,000
fails, analyst-1 for ai-training, learning.reviews, columns changed: none, expected: body
cases that fail: 3 of 12
cases that fail as decisions: 3 of 12
```

The change fails 3 of 12 cases, and none of them is about tickets. All three are the analyst's reads of learners and reviews. On `crm.learners`, the analyst now gets 1,581 of 5,000 rows with no column changed: support's region filter applies to them, and the policy's personal-column masks apply only to users outside support, so every email and note comes through as stored. Review text, which a data scientist otherwise can't read at all, or reads with contacts removed for AI training, comes through whole. [Chapter 5's decision-level table](#check-who-sees-what) fails on the same cases, 3 of 12; this one shows what the analyst gets.

**Working rule:** Run the who-sees-what cases as queries through the layer on every change to the policies, the users file, the metadata or the layer, and fail the build when any case disagrees. Fix a failing case by changing the policy or the users file. Change a case only when the data's owners agree that what the user should get has changed.

## Hold back small groups in reports {#hold-back-small-groups}

Row filters and masks decide which rows and values a user sees. Aggregates are a second channel: a count or an average over a few people describes those people. The usual defence is a **minimum group size**, below which a reported group is held back. The number is a rule of thumb, not a measurement. One public data-use agreement from the US Centers for Medicare & Medicaid Services sets 11, and adds that "no use of percentages or other mathematical formulas may be used if they result in the display of a cell less than 11" ([CMS-R-0235L](https://www.cms.gov/medicare/cms-forms/cms-forms/downloads/cms-r-0235l.pdf)). The platform uses the same minimum, 11, and keeps it in `governance/enforcement.toml`, where a change is reviewed like a policy change, and `lib.enforce.minimum_group_size` reads it.

Instructors get a daily course report of learners and average progress per course and learner country, built by a job that reads countries instructors may not read. Instructors may read their own courses' enrollments with each one's progress. Combining those two gives `instructor-17`'s report:

Example `small-groups-difference` (Python 3.12.15 · duckdb 2.0.0 · pyarrow 26.0.1 · pyiceberg 0.12.0 · regopy 1.5.2 · sqlglot 30.33.0), passing in Edition 2026.11.31:

```python
"""Builds instructor-17's daily course report, counts its cells under the minimum and its cells of one learner, and matches
each cell of one against the enrollments the instructor may read. The prose's points rest on the cell and lookup counts, and
on the last line, which shows the differencing."""
from lib.catalog import open_catalog
from lib.enforce import Enforcer, minimum_group_size

platform = Enforcer(open_catalog(["crm.learners", "learning.courses", "learning.enrollments"]))
minimum = minimum_group_size()
print("minimum group size:", minimum)
count = lambda sql: platform.connection.sql(f"select count(*) from ({sql})").fetchall()[0][0]

# The platform's daily course report for instructor-17: learners and average progress, per course and learner country.
# The report job reads learners' countries, which instructors may not read.
platform.connection.execute("""create view mine as
    select e.course_id, e.progress_pct, e.enrolled_at::date as day, l.country
    from learning.enrollments e join crm.learners l using (learner_id)
    where e.course_id in (select course_id from learning.courses where instructor_id = 17)""")
platform.connection.execute("""create view report as
    select course_id, country, count(*) as learners, avg(progress_pct) as progress from mine group by all""")
print("cells in the report:", count("from report"))
# the minimum is the threshold: a cell with fewer learners than it is held back
print("cells under the minimum:", count(f"from report where learners < {minimum}"))

# instructor-17 may read their courses' enrollments, with each one's progress. A cell of one learner, matched on its
# progress, names that learner's country.
# what instructor-17 may read: enrollments with each one's progress, and no country
seen = platform.sql("select course_id, progress_pct from learning.enrollments", "instructor-17").to_arrow_table()
platform.connection.register("seen", seen)
print("cells of one learner:", count("from report where learners = 1"))
print("learners a cell of one names:", count("""from report as cell where learners = 1 and (select count(*) from seen
    where seen.course_id = cell.course_id and seen.progress_pct = cell.progress) = 1"""))
print("cells shown with the minimum:", count(f"from report where learners >= {minimum}"))

# The minimum doesn't stop differencing. An enrollment that arrives alone in its course on its day, in a cell already
# shown, raises that cell by one in the next day's report, and the instructor sees which enrollment arrived.
# an enrollment alone in its course that day, whose country already has the minimum on earlier days
revealed = count(f"""from mine as m
    where (select count(*) from mine as o where o.course_id = m.course_id and o.day = m.day) = 1
      and (select count(*) from mine as o where o.course_id = m.course_id and o.country = m.country and o.day < m.day) >= {minimum}""")
print("learners whose country the daily report reveals anyway:", f"{revealed} of {count('from mine')}")
```

Output:

```text
minimum group size: 11
cells in the report: 63
cells under the minimum: 51
cells of one learner: 6
learners a cell of one names: 3
cells shown with the minimum: 12
learners whose country the daily report reveals anyway: 92 of 482
```

A cell of one names its learner's country to anyone who can find that learner, and the instructor can: matching the cell's average progress against their own enrollments names 3 learners exactly. The report has 63 cells, of which 51 are under the minimum and 6 hold one learner each.

With the minimum, 12 cells are shown, and none of those lookups works. The minimum doesn't stop differencing. An enrollment that arrives alone in its course on its day, into a cell that's already shown, raises that cell by one in the next day's report, and the instructor sees which enrollment arrived. That reveals the country of 92 of 482 learners.

Differencing, comparing consecutive reports, is the attack Denning, Denning and Schwartz called a tracker. They showed that "the compromise of small query sets can in fact almost always be accomplished with the help of characteristic formulas called trackers" ([Denning et al., 1979](https://doi.org/10.1145/320064.320069)).

Snowflake, which enforces a minimum group size in its engine, says its policies "do not guarantee a malicious actor could not use deliberate queries to obtain potentially sensitive data" ([Snowflake aggregation policies](https://docs.snowflake.com/en/user-guide/aggregation-policies)). Against differencing, we recommend noise, as in differential privacy ([Dwork and Roth, 2014](https://doi.org/10.1561/0400000042)), rather than higher thresholds, and *Privacy Engineering for Data Teams* covers it.

**Working rule:** Hold back every reported group below the minimum in `governance/enforcement.toml`, and treat the minimum as protection against looking one person up, never against differencing. Don't break a report down by an attribute its reader may not read, sent to a reader who can see the rows behind it: the instructor's report by country is a lookup table for countries. Drop the breakdown, or report it only after it has been made noisy.

## Hand the same policies to an engine that enforces them {#engine-enforcement}

In production, enforcement belongs in the engine every tool queries through, because Saltzer and Schroeder's principle of complete mediation holds that "every access to every object must be checked for authority" ([Saltzer and Schroeder, 1975](https://web.mit.edu/Saltzer/www/publications/protection/Basic.html)), and a policy enforced in one tool and not another fails it. Trino, an open-source distributed SQL engine, can take its access control from Open Policy Agent (OPA) through a plugin, and [chapter 5](#shape-the-request-like-trinos) shaped the platform's requests like the plugin's so the same policy could serve both. Trino 484 offers file-based, OPA and Ranger access control, and none for Cedar ([Trino 484, Security](https://trino.io/docs/484/security.html)).

The plugin asks OPA separate questions at separate URIs ([Trino 484, OPA access control](https://trino.io/docs/484/security/opa-access-control.html)):

- **Whether to allow** an operation, such as `SelectFromColumns`, answered with a boolean `allow`.
- **Row filters**, `GetRowFilters`, for a table. OPA answers with a list of `{"expression": ...}`, and each expression "essentially behaves like an additional WHERE clause". The URI is optional, and Trino applies no row filtering when it isn't set.
- **Column masks**, `GetColumnMask`, for one column, or a batch of columns at `opa.policy.batch-column-masking-uri`. "Only a single column mask may be returned for a given column."

The operation names and where each request puts its table or column are in the plugin's source at tag 484 ([`OpaAccessControl.java`](https://github.com/trinodb/trino/blob/484/plugin/trino-opa/src/main/java/io/trino/plugin/opa/OpaAccessControl.java), [`OpaHighLevelClient.java`](https://github.com/trinodb/trino/blob/484/plugin/trino-opa/src/main/java/io/trino/plugin/opa/OpaHighLevelClient.java)). The documentation's batch-mask policy reads `filterResources[i].catalogName` where its own request example nests those fields under `.column`, so as printed it wouldn't match. Requests built in those shapes go to its policy:

Example `trino-requests` (Python 3.12.15 · duckdb 2.0.0 · pyarrow 26.0.1 · pyiceberg 0.12.0 · regopy 1.5.2 · sqlglot 30.33.0), passing in Edition 2026.11.31:

```python
"""Builds requests in the shape Trino 484's OPA plugin sends, and runs them through the platform's policy. The prose's
points rest on the allow lines, which the example asserts are False, and on the GetRowFilters and GetColumnMask lines."""
import json

from lib import metadata, policy
from lib.enforce import row_filter

# The platform's policy, with a support agent added before anyone set their region (chapter 5's missing-region case).
data = policy.data_document()
data["platform"]["users"]["agent-new"] = {"groups": ["support"]}
rego = policy.interpreter(data=data)
described = metadata.load()


def trino_request(user: str, operation: str, resource: dict | None = None) -> dict:
    """A request in the shape Trino 484's OPA plugin sends (OpaAccessControl, OpaHighLevelClient)."""
    action = {"operation": operation} | ({"resource": resource} if resource else {})
    return {"context": {"identity": {"user": user, "groups": []}, "softwareStack": {"trinoVersion": "484"}}, "action": action}


# Trino asks `allow` about every operation: ExecuteQuery before any query runs, AccessCatalog for each catalog it names.
for operation, resource in [("ExecuteQuery", None), ("AccessCatalog", {"catalog": {"name": "platform"}})]:
    allowed = policy.decide(trino_request("agent-a05", operation, resource), rego)["allow"]
    # allow admits only SelectFromColumns, so Trino's checks before any query are denied
    assert allowed is False
    print(f"allow, {operation}, agent-a05:", allowed)

# GetRowFilters names a table. The policy's row_filters rule on its own skips `allow` and its missing-attribute guard,
# so for agent-new it returns no filter, which Trino reads as no filtering. The answer comes from the decision instead,
# for the same read asked as SelectFromColumns: its filters in Trino's SQL, or a filter that keeps no rows.
for user, schema, table in [("agent-a05", "support", "tickets"), ("instructor-17", "learning", "reviews"),
                            ("agent-new", "support", "tickets")]:
    request = trino_request(user, "GetRowFilters", {"table": {"catalogName": "platform", "schemaName": schema, "tableName": table}})
    print(f"GetRowFilters, {user}, {schema}.{table}, the rule alone:",
          json.dumps(policy.evaluate(rego, "data.platform.access.row_filters", request)))
    decision = policy.decide(policy.request(user, f"{schema}.{table}", []), rego)
    columns = {c.name: "" for c in described[f"{schema}.{table}"].columns.values()}
    # no table alias: Trino applies the filter to the table itself, as a WHERE clause on it
    filters = [{"expression": row_filter(rule, columns, table=None).sql("trino")} for rule in decision["row_filters"]]
    # a denied read gets a filter that keeps no rows
    print(f"GetRowFilters, {user}, {schema}.{table}, answered:", json.dumps(filters if decision["allow"] else [{"expression": "false"}]))

# GetColumnMask names one column, under action.resource.column. The policy reads the columns of a SelectFromColumns
# request instead, so it masks nothing here until the request is translated.
column = {"catalogName": "platform", "schemaName": "crm", "tableName": "learners", "columnName": "email", "columnType": "varchar"}
decision = policy.decide(trino_request("analyst-1", "GetColumnMask", {"column": column}), rego)
print("GetColumnMask as sent, analyst-1, crm.learners.email:", json.dumps(decision["column_masks"]), decision["reasons"][0])
translated = policy.request("analyst-1", "crm.learners", [column["columnName"]])
print("GetColumnMask translated, analyst-1, crm.learners.email:", json.dumps(policy.decide(translated, rego)["column_masks"]))
```

Output:

```text
allow, ExecuteQuery, agent-a05: False
allow, AccessCatalog, agent-a05: False
GetRowFilters, agent-a05, support.tickets, the rule alone: [{"column": "support_region", "equals": "emea"}]
GetRowFilters, agent-a05, support.tickets, answered: [{"expression": "support_region = 'emea'"}]
GetRowFilters, instructor-17, learning.reviews, the rule alone: [{"column": "course_id", "in": [2, 75, 80, 87, 103, 134]}]
GetRowFilters, instructor-17, learning.reviews, answered: [{"expression": "course_id IN (2, 75, 80, 87, 103, 134)"}]
GetRowFilters, agent-new, support.tickets, the rule alone: []
GetRowFilters, agent-new, support.tickets, answered: [{"expression": "false"}]
GetColumnMask as sent, analyst-1, crm.learners.email: {} GetColumnMask is not an operation this policy allows
GetColumnMask translated, analyst-1, crm.learners.email: {"email": "pseudonymize"}
```

Pointed at Trino unchanged, the policy would deny the checks Trino makes first, because its `allow` admits only `SelectFromColumns`. `allow` is false for `ExecuteQuery`, which Trino asks before any query runs, and for `AccessCatalog`.

The `row_filters` rule doesn't test the operation, so it answers `GetRowFilters`, but on its own it skips `allow` and [chapter 5](#deny-by-default-then-allow-by-role)'s missing-attribute guard. For `agent-new`, a support agent with no region yet, it returns `[]`, which Trino reads as no filtering. So the answer has to come from the decision for the same read, asked as `SelectFromColumns`: its filters, written in Trino's SQL by the same [`row_filter`](#build-row-filters-from-data) the rewrite uses, or, when it denies the read, `[{"expression": "false"}]`, which keeps no rows.

`GetColumnMask` names the column under `action.resource.column`, while the policy reads a `SelectFromColumns` request's columns, so as sent it masks nothing and the policy rejects the operation; translated, it gets `pseudonymize`. In production, those translations, the `allow` check and the Trino SQL belong in a Rego module beside the policy, so OPA can answer Trino itself. No OPA server runs here, so the code shows only what that module must return. One more piece maps onto Trino directly: the consent filter reads `crm.learners`, which the assistant may not, and Trino lets a filter carry an `identity` to run as, "such that a filter can target a column the requesting user cannot see".

Since 1.9.0, OPA's Compile API can also write SQL `WHERE` clauses from a policy by partial evaluation ([OPA v1.9.0](https://github.com/open-policy-agent/opa/releases/tag/v1.9.0), [Data filtering with OPA](https://www.openpolicyagent.org/docs/filtering)), but it needs an OPA server, which the examples don't run.

**Working rule:** Put enforcement in the engine every tool queries through, with one policy for it and for any rewrite like this chapter's. With Trino, answer every operation the plugin asks about, set both the row-filter and column-mask URIs (without them it applies no filters or masks), and return row filters only from a decision that allows the read. Build tests from the request shapes in the plugin's source, not from its documentation's example policies.

## Query on someone's behalf {#query-on-someones-behalf}

An assistant that answers questions from the warehouse is one more user of the layer, with a choice to make: whose identity does its query carry? The course assistant, `assistant-1`, has its own access, for reviews by consenting learners. When it answers a person, it must query as that person, so the answer holds exactly what they could read themselves. OWASP's guidance for language-model applications says the same: "Implement authorization in downstream systems rather than relying on an LLM to decide if an action is allowed or not" ([OWASP LLM06:2025](https://genai.owasp.org/llmrisk/llm062025-excessive-agency/)). The assistant itself belongs to *Data Assistants in Practice*; this section is about its queries.

`instructor-17` asks the assistant how their courses' reviews look, and the assistant writes a query. The tool takes the user from the signed-in session, never from the model or the conversation, so no prompt can change who the query runs as:

Example `assistant-on-behalf` (Python 3.12.15 · duckdb 2.0.0 · pyarrow 26.0.1 · pyiceberg 0.12.0 · regopy 1.5.2 · sqlglot 30.33.0), passing in Edition 2026.11.31:

```python
"""Runs the course assistant's review query as instructor-17 and as assistant-1, through a tool that takes the user from
the signed-in session. The prose's point rests on the reviews-behind-it lines for each user."""
from lib import policy
from lib.catalog import open_catalog
from lib.enforce import Enforcer

platform = Enforcer(open_catalog(["crm.learners", "learning.reviews"]))
courses = set(policy.data_document()["platform"]["instructor_courses"]["17"])     # instructor-17's own courses


def query_tool(session_user: str):
    """The tool the course assistant calls to query the warehouse. The model writes the SQL; the user it runs as comes
    from the signed-in session, never from the model or the conversation."""
    # the user is the session's; the SQL is all the model contributes
    return lambda sql: platform.sql(sql, user=session_user).fetchall()


# instructor-17 asks the assistant how their courses' reviews look. The assistant writes this query.
question = "select course_id, count(*) as reviews, round(avg(rating), 2) as rating from learning.reviews group by all"
for runs_as in ["instructor-17", "assistant-1"]:
    answer = query_tool(runs_as)(question)
    print(f"run as {runs_as}, courses in the answer:", len(answer))
    print(f"run as {runs_as}, reviews behind it:", sum(reviews for _, reviews, _ in answer))
    # each answer row is (course, reviews, rating), so row[0] is the course
    mine = [row for row in answer if row[0] in courses]
    print(f"run as {runs_as}, instructor-17's reviews behind it:", sum(reviews for _, reviews, _ in mine))
```

Output:

```text
run as instructor-17, courses in the answer: 6
run as instructor-17, reviews behind it: 69
run as instructor-17, instructor-17's reviews behind it: 69
run as assistant-1, courses in the answer: 130
run as assistant-1, reviews behind it: 436
run as assistant-1, instructor-17's reviews behind it: 7
```

Whose identity the query carries decides what the answer covers. The assistant's own identity is wrong both ways: it shows the instructor what they may not see, and hides what they may. Run as `instructor-17`, the answer covers their 6 courses and 69 reviews. Run as the assistant, it covers 130 courses, most of them other instructors', and only 7 of the instructor's own reviews, the ones whose authors consented to AI training. The same holds for retrieval: an assistant grounded on governed text needs "permission-aware vector and embedding stores" ([OWASP LLM08:2025](https://genai.owasp.org/llmrisk/llm082025-vector-and-embedding-weaknesses/)).

**Working rule:** Run an assistant's queries as the person it answers, with that identity taken from their authenticated session, and let the layer apply that person's decision. Use the assistant's own identity only for work it does for no one in particular, such as a nightly summary of consented reviews for the course pages, and record both identities in the [audit log](#audit-and-evidence). Whoever builds the assistant's training set queries as themselves, with the purpose `ai-training`, as [chapter 9](#admit-consented-reviews) does.

## What goes wrong in production {#enforcement-in-production}

An enforcement layer fails by letting through a read nobody tested, and most of these failures look like correct answers:

- **`SELECT *` slips past on a table missing from the schema.** `qualify` leaves the star unexpanded and raises nothing. Deny every table the catalog doesn't hold, and rebuild the schema whenever the catalog changes.
- **Aggregates leak small groups.** A cell of one names its member, and differences between reports name members of cells that pass any minimum, as [the report example](#hold-back-small-groups) showed. Set the minimum, and remove breakdowns by attributes the reader may not read.
- **Joins and copies bring masked columns back.** A pipeline with full access, such as the one that writes `marketing.audience`, copies `email` into a table protected only by its own classification. Unclassified, the copy is hidden from everyone, as [the who-sees-what test](#test-who-sees-what) showed; classified `internal` by mistake, it's shown to anyone who may read the table. [Chapter 4](#classifications-from-lineage) propagates classifications along lineage to catch that. Unmasked columns re-identify through joins too: Sweeney found 87% of the US population likely unique on ZIP code, gender and date of birth in 1990 census data ([Sweeney, 2002](https://dataprivacylab.org/dataprivacy/projects/kanonymity/kanonymity.pdf)), and Golle, counting again, found 61% on the same 1990 data and 63% on 2000's ([Golle, 2006](https://doi.org/10.1145/1179601.1179615)). Rocher and colleagues estimate, from a model fitted to census data, that 15 demographic attributes would make 99.98% of people in Massachusetts unique ([Rocher et al., 2019](https://doi.org/10.1038/s41467-019-10933-3)).
- **Direct file reads bypass the rewrite.** `read_parquet`, a path in `FROM` and table functions read data without naming a table. Deny them in the rewrite, and turn off `enable_external_access`, which DuckDB then won't turn back on: "Cannot enable external access while database is running". `lock_configuration` keeps every other setting as the layer left it; set it once the layer's connection is fully configured ([Securing DuckDB](https://duckdb.org/docs/2.0/operations_manual/securing_duckdb/overview)), not when it's created, since [chapter 8](#cross-check-the-query-log) starts DuckDB's query log after that. Anyone who can read the warehouse's files in storage needs no SQL, so storage permissions are part of enforcement.
- **Identifier case decides what a name matches.** A layer that matched names without normalizing them would let `"LEARNERS"`, which DuckDB resolves to `learners`, reach the raw table, as [the first section](#find-what-a-query-reads) showed. Normalize with the dialect of the engine that runs the query before matching any name.
- **Error messages print values.** DuckDB's cast errors include the value. The table rewrite makes it a masked one, and [the gate on each column](#gate-columns-on-row-filters) keeps dropped rows out of reach, but the message still reaches the user unless something stops it. Return a generic error at the service boundary, as `lib.enforce.answer` does, and log the engine's message where only the platform team reads it, apart from the audit log, which [chapter 8](#keep-values-out-of-the-record) keeps free of values. PostgreSQL warns that plans and timings remain covert channels even behind a security barrier ([PostgreSQL 18, Rules and Privileges](https://www.postgresql.org/docs/18/rules-privileges.html)).
- **DuckDB is not a security boundary.** "DuckDB executes SQL with the full privileges of the user running it, much like a shell or scripting interpreter" ([Securing DuckDB](https://duckdb.org/docs/2.0/operations_manual/securing_duckdb/overview)). A rewrite inside the same process protects data only while users can't reach DuckDB by any other route. Treat this chapter's layer as a tested model of what an engine such as Trino enforces, and enforce there in production.

## What this chapter decided {#enforcement-decisions}

- **Enforcement replaces tables, not columns.** Each catalog table becomes a subquery generated from the decision; rewriting column references is kept only to show that it leaks.
- **The layer fails closed.** Its sources are an allowlist of catalog tables, CTEs in scope and subqueries. It denies functions with side effects, parameters and anything but a single query. Columns a query doesn't read are typed `NULL`s, and the ones it reads are gated on the row filters.
- **Users get one generic error** from `answer`, and DuckDB's message goes to the platform team.
- **Masks are SQL in one place.** The policy names masks; `lib.enforce` writes their SQL. Pseudonyms are keyed with a secret only the layer holds, passed as a parameter.
- **Row filters are built from data,** with values as literals. Consent is a semi-join the layer runs with its own access, and an unknown consent counts as no.
- **Who sees what is tested as queries:** chapter 5's cases run through the layer on every change.
- **Reports hold back groups under the minimum group size,** 11 in `governance/enforcement.toml`, and don't break down by attributes their readers may not read.
- **Assistants query as the person they answer,** with the identity taken from the session.
- **In production, the engine enforces:** Trino with OPA, from the same policy, with both URIs set and row filters answered only from an allowed decision. A layer is only as safe as its tests.

# Retention, consent and erasure

By the end of this chapter you can keep retention rules as reviewed config, make a withdrawal of consent change the next query, and erase one person from every table their data reached. You'll apply the rules against a day you pass in, find those tables from lineage, and keep only what your legal team has recorded an exception for. You'll also remove what a row delete leaves behind in an Iceberg table, old snapshots, unreferenced files and column statistics, and finish with a scan that reads every file left and shows that none of them holds the person.

The online learning platform this book governs has kept everything since its first sign-up in January 2024 in one Iceberg catalog: every support ticket, and every learning event with the IP address it came from. Nothing says how long either should stay. Learner 3590 withdraws their consent to marketing email, and some weeks later asks the platform to erase their data. The first should change what marketing sees from the next query on. The second should leave no copy of the learner anywhere: not in the tables built from theirs, and not in the old versions of each table that Iceberg keeps on purpose.

Three terms name the chapter's subjects. **Retention** is how long the platform keeps each kind of data. **Consent** is a person's agreement to one use of their data, such as marketing email, which they can withdraw. **Erasure** is removing a person's data on request. The chapter adds two things to the book's shared code: the platform's retention rules, in `governance/retention.toml`, and the module `lib.retention`, which applies them, plans and carries out an erasure, cleans up and checks the files afterwards, and keeps the register of erasures. It uses `lib.lineage` from [chapter 4](#lineage-from-pipelines) to find where a learner's data went, and the classifications in `lib.metadata` from [chapter 2](#a-catalog-you-can-test). Every example runs PyIceberg 0.12.0 on a local catalog.

## Keep retention rules in a reviewed file {#retention-rules-in-a-file}

A retention rule says how long one kind of data is kept, from which date the time counts, and which rows it applies to. The EU's General Data Protection Regulation (GDPR) states the principle as storage limitation: personal data are to be "kept in a form which permits identification of data subjects for no longer than is necessary for the purposes for which the personal data are processed" (Article 5(1)(e)). Its Recital 39 asks the controller to set time limits for erasure or for a periodic review ([GDPR](https://eur-lex.europa.eu/eli/reg/2016/679/oj)). NIST's catalog of security and privacy controls describes the same job, asking that information be disposed of, destroyed or erased once its retention period ends (control SI-12(3)) ([NIST SP 800-53 Rev. 5](https://csrc.nist.gov/pubs/sp/800/53/r5/upd1/final)). The platform's owners and lawyers decide how long is necessary; the code applies what they decide, every time.

That argues for rules as config, reviewed like the access policies, and for code that applies them to a day it is given, not the clock. A job that reads the clock deletes different rows depending on when it runs: a test can't pin its result, and a rerun after a failure can delete more than the run it replaces.

The platform's rules are in `governance/retention.toml`. The periods are the platform's own choices for this book's case, not legal advice:

```toml
[tables."support.tickets"]
keep_days = 730
counted_from = "opened_at"        # tickets record when they were opened, not when they were closed
only_when = { column = "status", equals = "closed" }

[tables."learning.events"]
keep_days = 365
counted_from = "occurred_at"

[tables."billing.payments"]
keep_days = 3650                  # a placeholder for the period the platform's legal team sets for payment records
counted_from = "paid_at"
```

Closed tickets are kept two years, events one year, and payments 3,650 days, a placeholder that [the erasure plan](#plan-erasure-from-lineage) relies on. The tickets rule should count from when a ticket was closed, but the table records only `opened_at`, so a ticket opened 25 months ago and closed last week goes at once. The platform lives with that because adding a `closed_at` column is the support team's change, not the platform team's; until they make it, the rule errs toward deleting early. `lib.retention.due` turns a rule and a day into an Iceberg filter. Counted on the book's "today", Oct 1, 2026, and 90 days later, the rows each rule selects come out like this:

Example `retention-rules-due` (Python 3.12.15 · duckdb 2.0.0 · openlineage-python 1.55.0 · pyarrow 26.0.1 · pyiceberg 0.12.0 · pyyaml 6.0.3 · sqlglot 30.33.0), passing in Edition 2026.11.31:

```python
"""The rows each retention rule makes due on the book's today and 90 days later, counted against the day the job is given.
The `rows due` lines are the counts the prose gives."""
from datetime import timedelta

from lib import retention
from lib.catalog import open_catalog
from lib.platform import TODAY

rules = retention.rules()["tables"]
catalog = open_catalog(sorted(rules))
for name, rule in sorted(rules.items()):
    state = rule.get("only_when")
    print(f"{name}: kept {rule['keep_days']} days from {rule['counted_from']}"
          + (f", once {state['column']} is {state['equals']}" if state else ""))

# the book's today, then 90 days later: the day is passed in, never read from the clock
for day in (TODAY, TODAY + timedelta(days=90)):
    for name, rule in sorted(rules.items()):
        table = catalog.load_table(name)
        # the earliest date kept: keep_days before the day
        print(f"{name} on {day}, kept from:", retention.cutoff(rule, day))
        print(f"{name} on {day}, rows due:", f"{retention.count(table, retention.due(rule, day)):,}")
        print(f"{name} on {day}, rows in the table:", f"{retention.count(table):,}")
```

Output:

```text
billing.payments: kept 3650 days from paid_at
learning.events: kept 365 days from occurred_at
support.tickets: kept 730 days from opened_at, once status is closed
billing.payments on 2026-10-01, kept from: 2016-10-03
billing.payments on 2026-10-01, rows due: 0
billing.payments on 2026-10-01, rows in the table: 17,562
learning.events on 2026-10-01, kept from: 2025-10-01
learning.events on 2026-10-01, rows due: 9,060
learning.events on 2026-10-01, rows in the table: 100,000
support.tickets on 2026-10-01, kept from: 2024-10-01
support.tickets on 2026-10-01, rows due: 58
support.tickets on 2026-10-01, rows in the table: 2,000
billing.payments on 2026-12-30, kept from: 2017-01-01
billing.payments on 2026-12-30, rows due: 0
billing.payments on 2026-12-30, rows in the table: 17,562
learning.events on 2026-12-30, kept from: 2025-12-30
learning.events on 2026-12-30, rows due: 15,545
learning.events on 2026-12-30, rows in the table: 100,000
support.tickets on 2026-12-30, kept from: 2024-12-30
support.tickets on 2026-12-30, rows due: 111
support.tickets on 2026-12-30, rows in the table: 2,000
```

The rows due depend on the day passed in and the data, and on nothing else. On Oct 1, 2026, the events rule keeps events from 2025-10-01 on, so 9,060 of the 100,000 events are due, and on Dec 30, 90 days later, 15,545 are. The tickets rule makes 58 of the 2,000 tickets due on Oct 1, all closed and opened over two years ago. On Dec 30 it makes 111 due. The payments rule makes 0 of the 17,562 payments due on Oct 1 and 0 on Dec 30.

**Working rule:** Keep each table's retention period, the column it counts from and the state a row must be in, in a reviewed file, and apply it against a day the scheduler passes in, never the clock. Count from the date the rule means. When the table doesn't record that date, as the platform's tickets don't record when they closed, ask the table's owner to add the column rather than let the rule count from another date for good, and say in the rule's file which way it errs until then.

## Delete expired rows with row deletes {#delete-expired-rows}

With [the rules in a file](#retention-rules-in-a-file), the retention job deletes what they say. Iceberg writers delete rows in one of two ways, set by the table property `write.delete.mode`, which defaults to `copy-on-write` ([Iceberg configuration](https://iceberg.apache.org/docs/latest/configuration/)). Copy-on-write rewrites each affected data file without the deleted rows. Merge-on-read leaves the data file alone and writes a delete file beside it. The spec has two kinds of delete file: position deletes, which mark a row by its data file and its position (a deletion vector, in tables of version 3 or above), and equality deletes, which mark it by the values of one or more columns ([Iceberg table spec](https://iceberg.apache.org/spec/)). PyIceberg 0.12.0's `Table.delete` always copies on write: with `merge-on-read` set, it warns that merge-on-read isn't supported yet, and rewrites the files anyway ([source](https://github.com/apache/iceberg-python/blob/pyiceberg-0.12.0/pyiceberg/table/__init__.py)). Either way, a delete is a commit, a new snapshot, and the snapshot before it is still there.

`lib.retention.apply` deletes, for each rule, the rows `due` returns for the day it's given, and records the rule's table and the day in the commit's snapshot summary, so the table's own history says why the rows went. Two runs for Oct 1, 2026, read now and by time travel, show what a delete leaves behind, and expiring the old snapshots by age comes last:

Example `apply-retention` (Python 3.12.15 · duckdb 2.0.0 · openlineage-python 1.55.0 · pyarrow 26.0.1 · pyiceberg 0.12.0 · pyyaml 6.0.3 · sqlglot 30.33.0), passing in Edition 2026.11.31:

```python
"""Applies the retention rules for the book's today as Iceberg row deletes, which leave the old rows in older snapshots.
The rerun line is zero, and the time-travel lines show the rows the snapshot before each delete still returns."""
from datetime import datetime, timedelta, timezone

from lib import retention
from lib.catalog import open_catalog
from lib.platform import TODAY

rules = retention.rules()["tables"]
catalog = open_catalog(sorted(rules))
# each table's snapshot from before the deletes, which the time-travel lines read
before = {name: catalog.load_table(name).current_snapshot().snapshot_id for name in rules}

deleted = retention.apply(catalog, TODAY)
for name, rows in deleted.items():
    print(f"{name}, rows deleted:", f"{rows:,}")
print("rows deleted by running again for the same day:", sum(retention.apply(catalog, TODAY).values()))

for name in (name for name, rows in deleted.items() if rows):
    rule, table = rules[name], catalog.load_table(name)
    summary = table.current_snapshot().summary
    print(f"{name}, snapshots:", " then ".join(s.summary.operation.value for s in table.snapshots()))
    # copy-on-write writes the rows it keeps again as new records, which is what this line counts
    print(f"{name}, rows written again to delete the others:", f"{int(summary['added-records']):,}")
    print(f"{name}, the commit records:", summary["retention-rule"], "on", summary["retention-day"])
    rows_due = retention.due(rule, TODAY)
    print(f"{name}, rows due, read now:", f"{retention.count(table, rows_due):,}")
    print(f"{name}, rows due, read by time travel:", f"{retention.count(table, rows_due, before[name]):,}")
    # Snapshots carry real commit times, so their age is measured on the clock, by the table's own setting.
    print(f"{name}, snapshots expired by age today:", retention.expire_by_age(table))
    # six days on, the snapshot from before the delete is older than the table's five-day setting
    later = datetime.now(timezone.utc) + timedelta(days=6)
    print(f"{name}, snapshots expired by age six days later:", retention.expire_by_age(table, later))
```

Output:

```text
billing.payments, rows deleted: 0
learning.events, rows deleted: 9,060
support.tickets, rows deleted: 58
rows deleted by running again for the same day: 0
learning.events, snapshots: append then overwrite
learning.events, rows written again to delete the others: 90,940
learning.events, the commit records: learning.events on 2026-10-01
learning.events, rows due, read now: 0
learning.events, rows due, read by time travel: 9,060
learning.events, snapshots expired by age today: 0
learning.events, snapshots expired by age six days later: 1
support.tickets, snapshots: append then overwrite
support.tickets, rows written again to delete the others: 1,942
support.tickets, the commit records: support.tickets on 2026-10-01
support.tickets, rows due, read now: 0
support.tickets, rows due, read by time travel: 58
support.tickets, snapshots expired by age today: 0
support.tickets, snapshots expired by age six days later: 1
```

A retried run is harmless: a second run for the same day deleted 0 rows and made no commit. The first run deleted 9,060 events, 58 tickets and 0 payments. Each table it changed went from an `append` snapshot to an `overwrite`, and the summary shows what copy-on-write costs: to delete the old events, PyIceberg wrote the other 90,940 again, because the table is one unpartitioned file. Partitioned by month of `occurred_at`, the table would let the same delete drop whole files instead, which `Table.delete` does when every row of a file matches. In PyIceberg 0.12.0, writing a table partitioned by month needs the `pyiceberg-core` extra, which this book's examples don't install.

A delete doesn't remove the rows from the table's earlier snapshots. Read as it is now, `learning.events` holds 0 rows past their retention, but read by time travel, the snapshot before the delete still returns all 9,060, IP addresses included, and the Parquet file that holds them is still on disk. `lib.retention.expire_by_age` expires snapshots older than the table's own `history.expire.max-snapshot-age-ms`, five days here, measured on the clock. It expires 0 snapshots of `learning.events` straight after the delete, and 1 six days later. The retention job isn't finished until the old snapshots are [expired](#expire-snapshots) and [the files no snapshot references are removed](#remove-unreferenced-files): the same clean-up an erasure needs.

**Working rule:** Apply retention as Iceberg row deletes, one rule at a time, and record the rule and the day in each commit's snapshot summary. Make the job idempotent for a given day. Partition large tables by the column a rule counts from, so its deletes drop whole files instead of rewriting them; in PyIceberg, install its `pyiceberg-core` extra to write them. Count the job done only once the old snapshots are expired and their files removed.

## Read consent where the query runs {#consent-at-query-time}

[Retention](#retention-rules-in-a-file) removes rows by age. Consent changes who may use them, and withdrawing it isn't the same event as asking for erasure, so the platform's code treats them as two paths. Article 7(3) of the EU's General Data Protection Regulation (GDPR) says: "The data subject shall have the right to withdraw his or her consent at any time. The withdrawal of consent shall not affect the lawfulness of processing based on consent before its withdrawal." ([GDPR](https://eur-lex.europa.eu/eli/reg/2016/679/oj)). Article 17(1)(b) of the same regulation makes a withdrawal one ground for erasure, but only where no other legal ground for the processing remains. In the code, a withdrawal changes one attribute; an erasure deletes rows.

On the platform, consent is an attribute of the data that the access policy reads, as [chapter 5](#narrow-by-attribute) set it up: marketing's row filter keeps only learners whose `marketing_consent` is true, and the course assistant's keeps only the reviews whose authors' `ai_training_consent` is true. So a withdrawal needs no change to the policy. The CRM, the application where learners manage their accounts, syncs the new value into `crm.learners`, and the next query marketing runs, through the enforcement layer that [chapter 6](#enforcing-policies-on-queries) builds to rewrite every query, returns one learner fewer.

Withdraw learner 3590's marketing consent, then read what marketing sees and what the tables hold before the withdrawal, after it, and once the marketing pipeline has run again.

Example `consent-withdrawal` (Python 3.12.15 · duckdb 2.0.0 · openlineage-python 1.55.0 · pyarrow 26.0.1 · pyiceberg 0.12.0 · pyyaml 6.0.3 · regopy 1.5.2 · sqlglot 30.33.0), passing in Edition 2026.11.31:

```python
"""A learner withdraws marketing consent: one attribute changes, so marketing's next query returns one learner fewer and
no row is deleted. The table built with the consent filter keeps the learner until its pipeline runs again, as the audience lines show."""
import json

import pyarrow as pa
from pyiceberg.expressions import EqualTo

from lib import pipelines, policy, retention
from lib.catalog import open_catalog
from lib.enforce import Enforcer

LEARNER = 3590
catalog = open_catalog(["crm.learners", "learning.enrollments", "learning.courses"])
# the audience is built with the consent filter applied at build time, as the nightly job builds it
pipelines.run(catalog, "marketing_audience")
decision = policy.decide(policy.request("marketing-1", "crm.learners", ["learner_id"]))
print("marketing's row filters on crm.learners:", json.dumps(decision["row_filters"]))


def report(when: str) -> list[int]:
    # marketing's query runs through the enforcement layer, which applies its row filters
    seen = [row[0] for row in Enforcer(catalog).sql("select learner_id from crm.learners", "marketing-1").fetchall()]
    print(f"{when}, learners marketing sees:", f"{len(seen):,}")
    print(f"{when}, marketing sees learner {LEARNER}:", LEARNER in seen)
    print(f"{when}, rows stored in crm.learners:", f"{retention.count(catalog.load_table('crm.learners')):,}")
    audience = retention.count(catalog.load_table("marketing.audience"), EqualTo("learner_id", LEARNER))
    print(f"{when}, learner {LEARNER}'s rows in marketing.audience:", audience)
    return seen


before = report("before")
assert LEARNER in before                                   # marketing's query returns the learner before the withdrawal
learners = catalog.load_table("crm.learners")
row = learners.scan(row_filter=EqualTo("learner_id", LEARNER)).to_arrow()
row = row.set_column(row.schema.get_field_index("marketing_consent"), "marketing_consent", pa.array([False]))
learners.overwrite(row, overwrite_filter=EqualTo("learner_id", LEARNER))     # what the CRM's sync writes
after = report("after the withdrawal")
assert LEARNER not in after                                # the next query, through the consent filter, drops the learner
pipelines.run(catalog, "marketing_audience")
report("after the audience is built again")
```

Output:

```text
marketing's row filters on crm.learners: [{"column": "marketing_consent", "equals": true}, {"column": "status", "equals": "active"}]
before, learners marketing sees: 1,861
before, marketing sees learner 3590: True
before, rows stored in crm.learners: 5,000
before, learner 3590's rows in marketing.audience: 1
after the withdrawal, learners marketing sees: 1,860
after the withdrawal, marketing sees learner 3590: False
after the withdrawal, rows stored in crm.learners: 5,000
after the withdrawal, learner 3590's rows in marketing.audience: 1
after the audience is built again, learners marketing sees: 1,860
after the audience is built again, marketing sees learner 3590: False
after the audience is built again, rows stored in crm.learners: 5,000
after the audience is built again, learner 3590's rows in marketing.audience: 0
```

The withdrawal took effect at the next query and removed nothing: marketing sees 1,861 learners before it, learner 3590 among them, and 1,860 after it, without learner 3590. Marketing's row filters stay `[{"column": "marketing_consent", "equals": true}, {"column": "status", "equals": "active"}]`, and `crm.learners` still stores 5,000 rows. Nothing was deleted; the learner's account, enrollments and history are as they were.

The copy is the problem. `marketing.audience`, the list a marketing pipeline builds each night, applies the consent filter when it's built, and the policy gives marketing that table with no consent filter of its own. After the withdrawal it still holds 1 row for learner 3590. Until someone classifies the table's columns, the policy redacts them for every reader; once someone does, as [chapter 4](#classifications-from-lineage) proposes, a campaign sent from the table before its next build reaches someone who said no. Only once the pipeline runs again does the count fall to 0. A training corpus is built with a consent filter too, and [chapter 9](#withdrawn-after-the-build) checks each build against today's consent before training on it.

A true-or-false column also says too little: `ai_training_consent` is null for everyone who signed up before the platform asked, and false can mean refused or withdrawn. The Data Privacy Vocabulary, a W3C Community Group report rather than a standard, names the states a consent can be in, given, refused, withdrawn, expired and unknown among them ([DPV 2.3](https://w3c-cg.github.io/dpv/2.3/dpv/)).

**Working rule:** Keep consent where the query runs: in one table that the system the learner uses keeps current, read by the policy as a row filter, so a withdrawal changes the next query and deletes nothing. When the platform's lawyers decide that a withdrawal is also a ground for erasure, send it through [the erasure path](#plan-erasure-from-lineage) as well. Rebuild every table built with a consent filter whenever consent changes, or give its readers the same filter. Record each consent as a state and the time it changed, and derive the true-or-false attribute the policy reads from it, counting unknown as not given.

## Plan an erasure from lineage {#plan-erasure-from-lineage}

[A withdrawal](#consent-at-query-time) sets one attribute; an erasure has to find the person everywhere. Article 17(1) of the GDPR gives a person the right to have their data erased without undue delay, on the grounds it lists ([GDPR](https://eur-lex.europa.eu/eli/reg/2016/679/oj)). The platform treats a request as reaching every copy, so its first job, an engineering question, is a list in two halves of where the person's data is. The source tables that hold a learner's ID come from the catalog: [chapter 2](#classify-pseudonymous-ids) decided that every table with a `learner_id` is part of every erasure, whatever its classification. The tables built from them come from lineage: [chapter 4](#impact-of-a-column-change)'s walk of the column graph, run from each source table, reaches every table their rows shaped.

Some data stays. Article 17(3) of the same regulation sets out where erasure doesn't apply, to the extent that processing is necessary: among its five points, for compliance with a legal obligation under Union or Member State law to which the controller is subject ([GDPR](https://eur-lex.europa.eu/eli/reg/2016/679/oj)). Whether an exception applies, to which table, and for how long is the legal team's decision, never the code's. The code gives that decision one place, `governance/retention.toml`, where the legal team writes each exception with its reason. The book's reason is a placeholder, because we decide no legal question:

```toml
[erasure.exceptions."billing.payments"]
reason = "PLACEHOLDER: a legal obligation to keep payment records, such as tax records. The legal team replaces this with the obligation and the law it comes from."
decided_by = "legal team"         # a role, never a person's name
erase_columns = ["card_number"]   # what the obligation doesn't need; an empty list keeps the whole row
```

An erasure keeps an excepted table's rows until the table's [retention rule](#retention-rules-in-a-file) deletes them, and erases the columns `erase_columns` lists from them at once: here the full card number, which the platform's metadata says "Should not be stored at all". `lib.retention.exceptions` refuses an exception with no reason, or one for a table no rule ends.

Finding a derived table isn't the same as finding the learner's rows in it. `lib.retention.plan` looks at the lineage edges into each table the walk reaches, for a column copied unchanged, an IDENTITY edge in OpenLineage's terms, from the learner's ID or from the ID of one of their rows, such as a review. If there is one, the plan deletes by that column. If there's none, the learner's rows only shaped the table, and the plan runs the table's job again once the sources are erased. The plan reads the platform as the request finds it, weeks after the withdrawal: `crm.learners` holds learner 3590's withdrawn consent, and the four pipelines of [chapter 4](#lineage-from-pipelines), which build the analytics, marketing and AI tables, have run since, with their lineage recorded.

The plan for learner 3590 prints one line per step, each table with what it does there:

Example `erasure-plan` (Python 3.12.15 · duckdb 2.0.0 · openlineage-python 1.55.0 · phonenumbers 9.0.43 · presidio-analyzer 2.2.366 · pyarrow 26.0.1 · pyiceberg 0.12.0 · pyyaml 6.0.3 · sqlglot 30.33.0 · tldextract 5.4.0), passing in Edition 2026.11.31:

```python
"""The steps that erase one learner: the tables with their learner_id column, then the tables lineage says their rows reach,
with the exception the legal team recorded. The counts show how many steps each route found; the last two lines are the text the scan searches for and the rows that name the learner."""
from lib import pii, retention

LEARNER = 3590
catalog, graph = retention.open_platform()           # weeks after the learner withdrew marketing consent
steps = retention.plan(catalog, graph, LEARNER)
for step in steps:
    # no column is copied from the learner here: the table's own job runs again once its sources are erased
    if step.column is None:
        print(f"{step.table}: run {retention.job_for(step.table)} again")
    elif step.kept:
        print(f"{step.table}: keep, under an exception, and erase {', '.join(step.cleared)}")
    else:
        print(f"{step.table}: delete where {step.column} in {list(step.values)}"
              + (f", a copy of {step.via}" if step.via else ""))

print("tables in the plan:", len(steps))
# a step with a column and no lineage edge came from the table's own learner_id column
print("tables found by their learner_id column:", sum(1 for step in steps if step.column and not step.via))
# found by lineage: a copy of another table's ID, or a rebuild, because the learner's rows only shaped the table
print("tables found by lineage:", sum(1 for step in steps if step.via or step.column is None))
for step in (step for step in steps if step.kept):
    print(f"the reason recorded for {step.table}:", step.kept)
theirs = retention.rows_of([LEARNER])
for name in ("billing.payments", "marketing.audience"):
    print(f"the learner's rows in {name}:", retention.count(catalog.load_table(name), theirs))
# the values the scan will search for, noted before any row is deleted
found = retention.traces(catalog, steps, LEARNER)
# The phone number is printed through lib.pii.masked, which keeps its shape and its last four digits.
print("the learner's identifying values, as the scan searches for them:", pii.masked(", ".join(found.texts)))
# rows the plan keeps, in any string column, whose text holds one of those values
named = retention.mentions(catalog, steps, found)
print("rows the plan doesn't delete whose text names the learner:", sum(named.values()))
```

Output:

```text
billing.payments: keep, under an exception, and erase card_number
crm.learners: delete where learner_id in [3590]
learning.enrollments: delete where learner_id in [3590]
learning.events: delete where learner_id in [3590]
learning.reviews: delete where learner_id in [3590]
support.tickets: delete where learner_id in [3590]
analytics.course_performance: run course_performance again
analytics.learner_activity: delete where learner_id in [3590], a copy of crm.learners.learner_id
marketing.audience: delete where learner_id in [3590], a copy of crm.learners.learner_id
ai.review_corpus: delete where review_id in [1199], a copy of learning.reviews.review_id
tables in the plan: 10
tables found by their learner_id column: 6
tables found by lineage: 4
the reason recorded for billing.payments: PLACEHOLDER: a legal obligation to keep payment records, such as tax records. The legal team replaces this with the obligation and the law it comes from.
the learner's rows in billing.payments: 2
the learner's rows in marketing.audience: 0
the learner's identifying values, as the scan searches for them: •••-•••-5880 x•, 87199 jacob rad, alexa reed, alexa590@post.e
rows the plan doesn't delete whose text names the learner: 0
```

The plan visits 10 tables: 6 source tables with a `learner_id` column, and 4 that lineage adds. It keeps the learner's 2 payments under the exception. The steps are decided by columns and lineage, not by rows, so the plan lists two tables that hold none of the learner's rows: `support.tickets`, since learner 3590 never opened a ticket, and `marketing.audience`, which holds 0 rows because the pipeline rebuilt it after the withdrawal. `ai.review_corpus` has no `learner_id` at all, but lineage says its `review_id` is an IDENTITY copy of `learning.reviews.review_id`, so the plan deletes the learner's review by that ID. `analytics.course_performance` holds only counts and averages per course, so the plan rebuilds it from the erased sources. Rebuilding costs one pipeline run, so the code doesn't need to decide whether an average that once counted the learner is still their data.

Text can name a person in rows the plan doesn't delete, such as another learner's ticket. The example searches every string column for the identifying values `governance/retention.toml` lists, and finds 0 such rows. The search matches the first 15 characters of each value, lowercased, anywhere in the text: here `•••-•••-5880 x•, 87199 jacob rad, alexa reed, alexa590@post.e`, printed with the phone number masked, cut to 15 for the reason [the final scan](#verify-the-erasure) explains. A name typed differently or a phone number without separators needs the detectors of [chapter 3](#finding-personal-data).

**Working rule:** Plan every erasure from the catalog and the lineage graph, not from a list someone maintains: every table with the person's ID column, then every table lineage says their rows reach. Delete from a derived table by a column that lineage shows is an unchanged copy of an ID, and rebuild the derived tables the person's rows only shaped. Keep exceptions in config, written by your legal team with their reasons, and let the plan read and show them; never let the code decide one. Run [chapter 4's check for tables no job's lineage accounts for](#job-nobody-instrumented) first, because the plan can't reach a copy the lineage doesn't show.

## Erase the rows the plan lists {#erase-the-rows}

[The plan](#plan-erasure-from-lineage) says where learner 3590's data is, and `lib.retention.erase` carries it out in the plan's order. It deletes each step's rows with `Table.delete`, sources first, writes the rows an exception keeps again with the exception's columns empty, and runs the job of each table the plan rebuilds once the sources are done. Every commit records the request's ID, `erasure-0001`, in its snapshot summary. Iceberg keeps a summary in the table's metadata files after its snapshot expires, in all of them by default, or only the last `write.metadata.previous-versions-max` (100 by default) once `write.metadata.delete-after-commit.enabled` is set ([Iceberg maintenance](https://iceberg.apache.org/docs/latest/maintenance/), [Iceberg configuration](https://iceberg.apache.org/docs/latest/configuration/)). So the summary holds only the request's ID, which [the erasure register](#erased-learner-comes-back) pairs with a keyed hash of the learner's ID, and never the learner's email.

The erasure runs on learner 3590's plan under one request ID; reading `crm.learners` and `learning.events` now and by time travel shows what it leaves:

Example `erase-learner` (Python 3.12.15 · duckdb 2.0.0 · openlineage-python 1.55.0 · pyarrow 26.0.1 · pyiceberg 0.12.0 · pyyaml 6.0.3 · sqlglot 30.33.0), passing in Edition 2026.11.31:

```python
"""Carries out the erasure plan with Iceberg row deletes. The current snapshots no longer hold the learner, but the snapshots
before them still do, and the `read by time travel` lines show it."""
from lib import retention

LEARNER = 3590
catalog, graph = retention.open_platform()
steps = retention.plan(catalog, graph, LEARNER)
# each table's snapshot from before the erasure, which the time-travel lines read
before = {step.table: catalog.load_table(step.table).current_snapshot().snapshot_id for step in steps}

done = retention.erase(catalog, steps, "erasure-0001")       # the request's ID goes into every commit's summary
for table, rows in done.items():
    # a rebuilt table reports 'rebuilt' rather than a count of rows
    print(f"{table}, rows deleted:" if rows != "rebuilt" else f"{table}:", rows)
print("tables one row was deleted from:", ", ".join(table for table, rows in done.items() if rows == 1))
theirs = retention.rows_of([LEARNER])
payments = catalog.load_table("billing.payments").scan(row_filter=theirs).to_arrow()
print("billing.payments, the learner's rows kept:", payments.num_rows)
# the exception erases card_number to null, so the values that are not null are the card numbers left
print("billing.payments, card numbers left in them:", payments.num_rows - payments["card_number"].null_count)
for name in ("crm.learners", "learning.events"):
    table = catalog.load_table(name)
    print(f"the learner's rows in {name}, read now:", retention.count(table, theirs))
    print(f"the learner's rows in {name}, read by time travel:", retention.count(table, theirs, before[name]))

summary = catalog.load_table("crm.learners").current_snapshot().summary
# copy-on-write writes the rows it keeps again as new records, so this counts every other row of the file
print("crm.learners, rows written again to delete one:", f"{int(summary['added-records']):,}")
print("crm.learners, the commit records:", summary["erasure-request"])
# snapshots of tables the plan deleted from (not rebuilt, not kept) that still return a row the erasure deleted
holding = [(table, snapshot) for step in steps if step.column is not None
           for table in [catalog.load_table(step.table)] for snapshot in table.snapshots()
           if not step.kept and retention.count(table, retention.rows(step), snapshot.snapshot_id)]
print("snapshots that still hold the rows the erasure deleted:", len(holding))
print("of them current:", sum(snapshot == table.current_snapshot() for table, snapshot in holding))
```

Output:

```text
billing.payments, rows deleted: 0
crm.learners, rows deleted: 1
learning.enrollments, rows deleted: 2
learning.events, rows deleted: 19
learning.reviews, rows deleted: 1
support.tickets, rows deleted: 0
analytics.course_performance: rebuilt
analytics.learner_activity, rows deleted: 1
marketing.audience, rows deleted: 0
ai.review_corpus, rows deleted: 1
tables one row was deleted from: crm.learners, learning.reviews, analytics.learner_activity, ai.review_corpus
billing.payments, the learner's rows kept: 2
billing.payments, card numbers left in them: 0
the learner's rows in crm.learners, read now: 0
the learner's rows in crm.learners, read by time travel: 1
the learner's rows in learning.events, read now: 0
the learner's rows in learning.events, read by time travel: 19
crm.learners, rows written again to delete one: 4,999
crm.learners, the commit records: erasure-0001
snapshots that still hold the rows the erasure deleted: 6
of them current: 0
```

The erasure took the learner out of every table in the plan holding any. It deleted 19 of the learner's events, 2 enrollments, and one row each from `crm.learners, learning.reviews, analytics.learner_activity, ai.review_corpus`. It deleted nothing from `support.tickets` and `marketing.audience`, which held none, and it rebuilt `analytics.course_performance`.

`billing.payments` keeps the learner's 2 payments, with 0 card numbers left in them. Read now, `crm.learners` holds 0 rows for the learner. Read by time travel to the snapshot before the erasure, it returns 1, with name, email, phone and address, and `learning.events` returns all 19 of the learner's events the same way.

To delete one row of `crm.learners`, PyIceberg wrote the other 4,999 into a new file, as copy-on-write does, and the old file, with the learner in it, belongs to the snapshot before.

Across the plan's tables, 6 snapshots still hold the rows the erasure deleted, and 0 of them are current. That is what snapshots are for: a reader can go back to any of them. For an erasure, it means the delete took the learner out of the table but not out of storage.

**Working rule:** Carry out an erasure as row deletes in the plan's order, and record only a request ID in each commit, never a value of the person's. Treat the delete as the first of three steps: the snapshots before it still hold the person, and anyone allowed to time travel can read them until they're [expired](#expire-snapshots).

## Expire snapshots by the table's own settings {#expire-snapshots}

[After the erasure's deletes](#erase-the-rows), learner 3590's rows live on in the snapshots before them. The Iceberg spec says snapshots expire so that the data files they removed or replaced can be physically deleted ([Iceberg table spec](https://iceberg.apache.org/spec/)). Each table sets how in its properties ([Iceberg configuration](https://iceberg.apache.org/docs/latest/configuration/)): `gc.enabled` allows garbage collection, such as expiring snapshots and removing orphan files; `history.expire.min-snapshots-to-keep` is how many snapshots to keep however old they are; and `history.expire.max-snapshot-age-ms` is the age at which the rest may go. In Iceberg's Java library, regularly expiring snapshots deletes unused data files ([Iceberg maintenance](https://iceberg.apache.org/docs/latest/maintenance/)).

PyIceberg 0.12.0 does less. Its `table.maintenance.expire_snapshots()` commits one change, which removes the snapshots from the table's metadata and deletes no file ([source](https://github.com/apache/iceberg-python/blob/pyiceberg-0.12.0/pyiceberg/table/update/snapshot.py)), though its documentation offers it to "clean up table metadata and reduce storage costs" ([PyIceberg API](https://py.iceberg.apache.org/api/)). Nothing in it reads `gc.enabled`, and only its command-line tool reads the `history.expire` settings. An issue asks for expiry to delete files too ([issue 2604](https://github.com/apache/iceberg-python/issues/2604)). Two copies of `crm.learners` each have 3 snapshots, the load and two deletes. One copy has `gc.enabled=false`, the other `history.expire.min-snapshots-to-keep=3`, and each is expired two ways: by age with PyIceberg, `older_than` the current time, and with `lib.retention.expire`, which reads the two settings first.

Example `expiry-ignores-table-settings` (Python 3.12.15 · duckdb 2.0.0 · openlineage-python 1.55.0 · pyarrow 26.0.1 · pyiceberg 0.12.0 · pyyaml 6.0.3 · sqlglot 30.33.0), passing in Edition 2026.11.31:

```python
"""PyIceberg's snapshot expiry ignores two of the table's own settings, gc.enabled and min-snapshots-to-keep, so it expires
snapshots they would keep. The book's expiry reads both first, and the `left by PyIceberg's expiry` lines show the difference."""
from datetime import datetime, timezone

from pyiceberg.expressions import EqualTo

from lib import retention
from lib.catalog import open_catalog


def learners_with(settings: dict):
    """crm.learners with three snapshots: the load, then two deletes."""
    table = open_catalog(["crm.learners"]).load_table("crm.learners")
    with table.transaction() as transaction:
        transaction.set_properties(settings)
    # two deletes, so there are snapshots from before them for the expiry to consider
    for learner in (3590, 3591):
        table.delete(EqualTo("learner_id", learner))
    return table


for settings in ({"gc.enabled": "false"}, {"history.expire.min-snapshots-to-keep": "3"}):
    setting = ", ".join(f"{key}={value}" for key, value in settings.items())
    table = learners_with(settings)
    print(f"{setting}, snapshots:", len(table.snapshots()))
    try:
        print(f"{setting}, the book's expiry expires:", retention.expire(table))
    except retention.RetentionError as error:
        print(f"{setting}, the book's expiry refuses:", error)
    # PyIceberg's expiry by age, as of now: it reads neither setting
    table.maintenance.expire_snapshots().older_than(datetime.now(timezone.utc)).commit()
    print(f"{setting}, snapshots left by PyIceberg's expiry:", len(table.snapshots()))
```

Output:

```text
gc.enabled=false, snapshots: 3
gc.enabled=false, the book's expiry refuses: crm.learners has gc.enabled=false, which turns off garbage collection for the table
gc.enabled=false, snapshots left by PyIceberg's expiry: 1
history.expire.min-snapshots-to-keep=3, snapshots: 3
history.expire.min-snapshots-to-keep=3, the book's expiry expires: 0
history.expire.min-snapshots-to-keep=3, snapshots left by PyIceberg's expiry: 1
```

PyIceberg's expiry ignores both settings. It left 1 of the first table's 3 snapshots, and 1 of the second's. `older_than` expires every snapshot older than the time it's given except those a branch or tag points at, here only the current one.

The book's expiry refused the first table, `crm.learners has gc.enabled=false, which turns off garbage collection for the table`, and expired 0 snapshots of the second. An owner sets these properties for reasons the expiring code can't see, and in PyIceberg they are only advice.

For an erasure, both settings mean the job can't finish: a table that keeps a snapshot from before the delete still holds the person. A job that stopped at that table would already have deleted rows from the tables before it and left the rest unexpired, so the check belongs before the first delete. `lib.retention.erase_everywhere` runs `lib.retention.blocked` on every table in the plan first: garbage collection off, more than one snapshot to keep, or a branch or tag besides `main`. If any table fails, the job refuses to start and names every such table.

Here `crm.learners` has garbage collection off, and `learning.events` keeps three snapshots. Then the job and the scan run:

Example `erasure-refuses-to-start` (Python 3.12.15 · duckdb 2.0.0 · openlineage-python 1.55.0 · pyarrow 26.0.1 · pyiceberg 0.12.0 · pyyaml 6.0.3 · sqlglot 30.33.0), passing in Edition 2026.11.31:

```python
"""The erasure job checks every table in its plan before it changes any, and refuses to start when a table's own settings
would keep the learner. The scan still runs, from the traces the refusal carries, and finds the learner in the files."""
from datetime import timedelta

from lib import retention

LEARNER = 3590
catalog, graph = retention.open_platform()
# each setting stops expiry from removing the snapshot from before the delete: garbage collection off, or three snapshots kept
for name, settings in (("crm.learners", {"gc.enabled": "false"}),
                       ("learning.events", {"history.expire.min-snapshots-to-keep": "3"})):
    with catalog.load_table(name).transaction() as transaction:
        transaction.set_properties(settings)
# every table's current snapshot, to count the tables the job changes
before = {name: catalog.load_table(name).current_snapshot().snapshot_id for name in retention.tables(catalog)}

try:
    # min_age of zero: one writer and nothing in flight, so no file needs a minimum age to be safe to sweep
    steps, found, done = retention.erase_everywhere(catalog, graph, LEARNER, "erasure-0001", min_age=timedelta(0))
except retention.ErasureRefused as refusal:
    for name, why in refusal.tables.items():
        print(f"the job refuses, because of {name}:", why)
    found = refusal.found
# a table the job changed has a new current snapshot
print("tables the job changed:", sum(catalog.load_table(name).current_snapshot().snapshot_id != snapshot
                                     for name, snapshot in before.items()))
report = retention.scan(catalog, found)
print("tables whose files hold the learner:", sum(1 for _, holding in report.values() if holding))
print("files holding the learner:", sum(holding for _, holding in report.values()))
```

Output:

```text
the job refuses, because of crm.learners: gc.enabled=false turns off garbage collection for the table
the job refuses, because of learning.events: history.expire.min-snapshots-to-keep=3 keeps the snapshot from before the delete
tables the job changed: 0
tables whose files hold the learner: 7
files holding the learner: 8
```

The job refuses. It names both tables, `crm.learners` because `gc.enabled=false turns off garbage collection for the table`, and `learning.events` because `history.expire.min-snapshots-to-keep=3 keeps the snapshot from before the delete`, and 0 tables changed.

The scan runs either way, from the traces the refusal carries, and finds the learner in 8 files across 7 tables. A person then decides with the tables' owners, rather than code overriding an owner's setting.

**Working rule:** Expire snapshots through one function of your own that reads `gc.enabled` and the `history.expire` settings before it expires anything, and refuses when garbage collection is off, because PyIceberg's expiry reads neither; check again at every PyIceberg release. Before an erasure deletes anything, check every table in its plan the same way, and refuse the whole erasure when one would keep the person. For an erasure, expire every snapshot older than the delete. For retention, expire by age, as `history.expire.max-snapshot-age-ms` says, measured on the clock, since snapshots carry real commit times.

## Remove the files no snapshot references {#remove-unreferenced-files}

Once [the old snapshots are expired](#expire-snapshots), the files only they referenced are unreferenced. An unreferenced file, which Iceberg calls an orphan, is under the table's location but in no snapshot's metadata. Its Java library removes them in a separate action, and warns that removing orphan files with a retention interval shorter than the time expected for any write to complete is dangerous; its default interval is 3 days ([Iceberg maintenance](https://iceberg.apache.org/docs/latest/maintenance/)). PyIceberg 0.12.0 has no such action. Its release notes list none ([release notes](https://github.com/apache/iceberg-python/releases/tag/pyiceberg-0.12.0)). The issue that asked for one was closed by the project's stale bot, marked not planned ([issue 1200](https://github.com/apache/iceberg-python/issues/1200)). Two pull requests that added one were closed without being merged ([1958](https://github.com/apache/iceberg-python/pull/1958), [3358](https://github.com/apache/iceberg-python/pull/3358)). Its only call that deletes data files is `purge_table`, which deletes the whole table ([source](https://github.com/apache/iceberg-python/blob/pyiceberg-0.12.0/pyiceberg/catalog/__init__.py)).

So `lib.retention` removes them itself, from what the metadata tables PyIceberg exposes as `table.inspect` say a table references. `lib.retention.referenced` collects the metadata files, current and logged, every snapshot's manifest list, every manifest (a manifest lists data files, with each file's column statistics), every data and delete file live in any snapshot, and any statistics files. Every other file under the table's location is unreferenced, and `lib.retention.sweep` deletes the ones older than the minimum age its caller passes. `governance/retention.toml` sets that age to 72 hours, Java's default, and `lib.retention.min_file_age` reads it. After the erasure, the plan's tables are expired, swept with that age and then with none, and every file is checked for the learner:

Example `sweep-unreferenced-files` (Python 3.12.15 · duckdb 2.0.0 · openlineage-python 1.55.0 · pyarrow 26.0.1 · pyiceberg 0.12.0 · pyyaml 6.0.3 · sqlglot 30.33.0), passing in Edition 2026.11.31:

```python
"""After the erasure, expire the old snapshots, then remove the files no snapshot references. PyIceberg's expiry deletes no
file and has no orphan-file removal, so lib.retention.sweep does it with the inspect tables; the last lines show the manifest entries still holding the learner's email."""
from datetime import timedelta

from lib import retention

LEARNER = 3590
catalog, graph = retention.open_platform()
steps = retention.plan(catalog, graph, LEARNER)
found = retention.traces(catalog, steps, LEARNER)
retention.erase(catalog, steps, "erasure-0001")
tables = [catalog.load_table(step.table) for step in steps]
print("snapshots expired:", sum(retention.expire(table) for table in tables))
print("files under the plan's tables:", sum(len(retention.files_under(table.location())) for table in tables))
print("files no metadata references:", sum(len(retention.unreferenced(table)) for table in tables))
print("files holding the learner, after expiry:", sum(held for _, held in retention.scan(catalog, found).values()))
# the configured minimum age, 72 hours, from governance/retention.toml
print("files swept with the configured minimum age:", sum(retention.sweep(t, retention.min_file_age()) for t in tables))
# One writer, nothing in flight: no file is too young to remove, so the examples sweep with no minimum age.
print("files swept with no minimum age:", sum(retention.sweep(table, timedelta(0)) for table in tables))
print("files holding the learner, after the sweep:", sum(held for _, held in retention.scan(catalog, found).values()))

for table in tables:
    # every manifest entry, including those of deleted files, with the bounds it keeps for the email column
    entries = table.inspect.entries()
    for status, metrics in zip(entries["status"].to_pylist(), entries["readable_metrics"].to_pylist()):
        bounds = metrics.get("email") or {}
        # bounds are compared lowercased: the learner's email is typed in capitals in the manifest
        kept = (bounds.get("lower_bound"), bounds.get("upper_bound"))
        if any(text in bound.lower() for bound in kept if bound for text in found.texts):
            print(f"{'.'.join(table.name()[-2:])}, manifest entry with status {status}, email bounds:",
                  bounds["lower_bound"], "to", bounds["upper_bound"])
```

Output:

```text
snapshots expired: 8
files under the plan's tables: 100
files no metadata references: 28
files holding the learner, after expiry: 9
files swept with the configured minimum age: 0
files swept with no minimum age: 28
files holding the learner, after the sweep: 1
crm.learners, manifest entry with status 2, email bounds: ALEXA590@POST.EX to zofia.haase@exan
```

Expiry removes snapshots, not files. Of the files under the plan's tables, 28 are referenced by nothing, and 9 hold the learner. The sweep at the configured 72 hours deletes 0 unreferenced files, since every file was written seconds before. The examples pass an age of zero because one process writes their catalog. The sweep with no minimum age deletes all the unreferenced files, and 1 file still holds the learner: a manifest that the current snapshot of `crm.learners` references.

The spec says why: when a file is replaced or deleted, its manifest entry stores the snapshot in which the file was deleted and status 2 (deleted) ([Iceberg table spec](https://iceberg.apache.org/spec/)), and that entry keeps the deleted file's column statistics. By default, under the metrics mode `truncate(16)` ([Iceberg configuration](https://iceberg.apache.org/docs/latest/configuration/)), those include each string column's smallest value, cut to 16 characters, and its largest, cut to 16 with the last character raised by one, so that it still bounds every value in the file ([source](https://github.com/apache/iceberg-python/blob/pyiceberg-0.12.0/pyiceberg/utils/truncate.py)). The deleted entry in `crm.learners` keeps the email bounds `ALEXA590@POST.EX to zofia.haase@exan`. The first is learner 3590's email, typed in capitals, which sorts before every other email on the platform. The second is learner 4998's, the platform's largest, whose 16th character, an `m`, became an `n`. No learner is safe from this: each data file's entry keeps the smallest and largest value of every column, so a table of many small files names many learners.

**Working rule:** Remove unreferenced files after every expiry, with a sweep of your own built on the inspect tables, since PyIceberg has none, and keep a minimum age of days in production so that a write in progress keeps its files. Use an age of zero only where a single process writes. Don't count a person erased because no data file holds them: the manifests may still hold their values.

## Keep personal values out of manifests {#keep-values-out-of-manifests}

[The sweep](#remove-unreferenced-files) left the learner's email in a manifest, as a column bound. Iceberg lets a table choose, per column, what its manifests keep, with the property `write.metadata.metrics.column.<name>`: `none` keeps nothing, `counts` keeps value and null counts, `truncate(n)` adds bounds cut to n characters, and `full` adds whole bounds ([Iceberg configuration](https://iceberg.apache.org/docs/latest/configuration/)). PyIceberg 0.12.0 looks the property up by the column's name when it writes a file ([source](https://github.com/apache/iceberg-python/blob/pyiceberg-0.12.0/pyiceberg/io/pyarrow.py)).

Typed by hand, those properties would drift from the classifications. `lib.retention.properties` generates them from `governance/metadata.toml` and the `[statistics]` section of `governance/retention.toml`: `none` for every column classified `personal` or `sensitive`, and for every column nobody has classified, the same fail-closed default the access policy uses. It adds the `history.expire` settings from the rules' `[snapshots]` section. `lib.retention.apply_properties` sets them on every table and rewrites each table's data once, because a metrics mode applies only to the files written after it's set; expiry and a sweep then remove the files from before. Learner 4998, whose email is the largest bound in `crm.learners`' manifest, is the test case, erased on two platforms, one with default statistics and one with the generated properties:

Example `statistics-off-for-personal-columns` (Python 3.12.15 · duckdb 2.0.0 · openlineage-python 1.55.0 · pyarrow 26.0.1 · pyiceberg 0.12.0 · pyyaml 6.0.3 · sqlglot 30.33.0), passing in Edition 2026.11.31:

```python
"""Generates each table's properties from the classifications, so manifests keep no bounds on personal columns, then erases
the learner whose email is the platform's largest, on a platform with the properties and on one without. The `files holding` lines are the proof."""
from datetime import timedelta

from lib import retention

LEARNER = 4998
for statistics_off in (False, True):
    catalog, graph = retention.open_platform(statistics_off)
    label = "statistics off for protected columns" if statistics_off else "default statistics"
    if statistics_off:
        generated = retention.properties(catalog)
        # the per-column metrics settings; the history.expire settings are not counted here
        turned_off = [key for wanted in generated.values() for key in wanted if key.startswith(retention.METRICS)]
        print("columns with statistics turned off:", len(turned_off))
        # tables with no entry in governance/metadata.toml, which the generated properties still cover
        unclassified = [name for name in generated if name not in retention.metadata.load()]
        print("of them in tables nobody has classified:",
              sum(key.startswith(retention.METRICS) for name in unclassified for key in generated[name]))
        print("crm.learners, properties set:", len(generated["crm.learners"]))
        for key, value in sorted(generated["crm.learners"].items()):
            print(f"  {key} = {value}")
    # crm.learners has one data file, so its one manifest entry keeps the table's email bounds
    entry = catalog.load_table("crm.learners").inspect.entries()["readable_metrics"].to_pylist()[0]
    print(f"{label}, crm.learners email bounds:", entry["email"]["lower_bound"], "to", entry["email"]["upper_bound"])
    columns = retention.metadata.load()["crm.learners"].columns.values()
    personal = [column.name for column in columns if column.classification == "personal"]
    print(f"{label}, crm.learners personal columns:", len(personal))
    with_bounds = sum(entry[name]["lower_bound"] is not None for name in personal)
    print(f"{label}, of them with bounds in the manifest:", with_bounds)
    steps, found, _ = retention.erase_everywhere(catalog, graph, LEARNER, "erasure-0002", min_age=timedelta(0))
    print(f"{label}, files holding learner {LEARNER} after erasure, expiry and sweep:",
          sum(held for _, held in retention.scan(catalog, found).values()))
```

Output:

```text
default statistics, crm.learners email bounds: ALEXA590@POST.EX to zofia.haase@exan
default statistics, crm.learners personal columns: 7
default statistics, of them with bounds in the manifest: 7
default statistics, files holding learner 4998 after erasure, expiry and sweep: 1
columns with statistics turned off: 24
of them in tables nobody has classified: 10
crm.learners, properties set: 9
  history.expire.max-snapshot-age-ms = 432000000
  history.expire.min-snapshots-to-keep = 1
  write.metadata.metrics.column.city = none
  write.metadata.metrics.column.date_of_birth = none
  write.metadata.metrics.column.email = none
  write.metadata.metrics.column.full_name = none
  write.metadata.metrics.column.notes = none
  write.metadata.metrics.column.phone = none
  write.metadata.metrics.column.street_address = none
statistics off for protected columns, crm.learners email bounds: None to None
statistics off for protected columns, crm.learners personal columns: 7
statistics off for protected columns, of them with bounds in the manifest: 0
statistics off for protected columns, files holding learner 4998 after erasure, expiry and sweep: 0
```

With default statistics, 1 file still holds learner 4998 after the erasure, expiry and sweep, through a raised upper bound. The manifest keeps bounds for 7 of the table's personal columns, and its email bounds are `ALEXA590@POST.EX to zofia.haase@exan`.

Turning statistics off for personal columns is what keeps learner 4998 out of the files. The email bounds are `None to None`, the manifest keeps bounds for 0 of the table's 7 personal columns, and 0 files hold the learner after the erasure, expiry and sweep.

The generated properties turn statistics off for 24 columns, 10 of them in `marketing.audience` and `ai.review_corpus`, the two tables that [chapter 2](#test-the-catalog-against-the-metadata) found with no metadata. `crm.learners` gets 9 properties: the two expiry settings, and `none` for each personal column.

The cost is pruning. Iceberg uses the bounds in manifests to skip files a query's filter can't match ([Iceberg table spec](https://iceberg.apache.org/spec/)), so without them, a query that filters on `email` reads every file. The platform filters and erases by `learner_id`, which is `internal` and keeps its statistics. `counts` is the other choice: it keeps the counts that help a planner, and no values. And because PyIceberg 0.12.0 keeps the property under the column's old name on a rename, leaving the new name without it, the properties are generated again on every schema change, as [chapter 2](#key-column-metadata-by-field-id) asks of every view of the metadata keyed by name.

**Working rule:** Generate per-column metrics properties from your classifications, `none` or `counts` for every personal, sensitive or unclassified column, and apply them when a table is created or, for an existing table, with one rewrite followed by expiry and a sweep. Keep statistics on the ID columns you filter and erase by. Generate the properties again after every schema change.

## Prove the erasure by reading every file {#verify-the-erasure}

Each step so far removed something the one before left behind, and left something of its own: the [delete](#erase-the-rows), the [expiry](#expire-snapshots), the [sweep](#remove-unreferenced-files) and the [statistics settings](#keep-values-out-of-manifests). That is the case for checking the result rather than trusting the steps. The European Data Protection Board's report on how controllers handle erasure, adopted in February 2026, found backups, and anonymisation used in place of deletion, among the weak points, and recommends that controllers "Verify that erasure has been carried out and be able to demonstrate such erasure" ([EDPB, 2026](https://www.edpb.europa.eu/system/files/2026-02/edpb_cef-report_2025_right-to-erasure_en.pdf)).

`lib.retention.scan` reads every file under the catalog's warehouse, each with its own format's reader, and looks for the learner's traces. `lib.retention.traces` notes the traces before the erasure: the IDs of every row the plan deletes, by column, and the first 15 characters, lowercased, of each identifying value. Fifteen, because a manifest keeps a largest value's first 15 characters unchanged and raises the 16th by one, so a search for 16 misses a person whose value is a file's largest, as learner 4998's email is. A Parquet file is read as rows. A manifest or manifest list, both Avro, is read record by record with PyIceberg's Avro reader: every string in it, bounds included, and the bounds of each ID column, decoded by the schema the manifest carries. A metadata file is read as text. In a table an exception keeps, the learner's IDs are expected, so there the scan looks only for values, including those of the columns the exception erased. `lib.retention.erase_everywhere` runs the whole job: plan, note the traces, check every table, erase, expire, sweep. On the platform with the generated properties, the whole job runs and ends with the scan:

Example `verify-erasure` (Python 3.12.15 · duckdb 2.0.0 · openlineage-python 1.55.0 · pyarrow 26.0.1 · pyiceberg 0.12.0 · pyyaml 6.0.3 · sqlglot 30.33.0), passing in Edition 2026.11.31:

```python
"""Runs the whole erasure job on the platform with statistics off for protected columns, then scans every file under the
warehouse. The `files holding the learner` line is the proof; the rest is evidence of IDs and counts, never the learner's values."""
from datetime import timedelta

from lib import retention

LEARNER, REQUEST = 3590, "erasure-0001"
catalog, graph = retention.open_platform(statistics_off=True)
# min_age of zero: one writer, so nothing is in flight that a sweep could take
steps, found, done = retention.erase_everywhere(catalog, graph, LEARNER, REQUEST, min_age=timedelta(0))

report = retention.scan(catalog, found)
for table, (files, holding) in report.items():
    print(f"{table}: {files} files read, {holding} holding the learner")

print("request:", REQUEST)
print("tables visited:", len(steps))
# each table the job deleted from has a count of rows; the one it reran has 'rebuilt' instead
print("rows deleted:", sum(rows for rows in done.values() if rows != "rebuilt"))
print("tables rebuilt:", ", ".join(table for table, rows in done.items() if rows == "rebuilt"))
print("tables kept under an exception, columns erased:",
      "; ".join(f"{step.table}, {', '.join(step.cleared)}" for step in steps if step.kept))
# every snapshot but the current one: the ones a reader can still time travel to
print("snapshots a reader can still time travel to:",
      sum(len(catalog.load_table(step.table).snapshots()) - 1 for step in steps))
print("files no metadata references:",
      sum(len(retention.unreferenced(catalog.load_table(name))) for name in retention.tables(catalog)))
print("tables missing their generated properties:", ", ".join(retention.missing_properties(catalog)))
print("files read:", sum(files for files, _ in report.values()))
print("files holding the learner:", sum(holding for _, holding in report.values()))
```

Output:

```text
ai.review_corpus: 11 files read, 0 holding the learner
analytics.course_performance: 5 files read, 0 holding the learner
analytics.learner_activity: 11 files read, 0 holding the learner
billing.payments: 12 files read, 0 holding the learner
crm.instructors: 8 files read, 0 holding the learner
crm.learners: 11 files read, 0 holding the learner
learning.courses: 8 files read, 0 holding the learner
learning.enrollments: 11 files read, 0 holding the learner
learning.events: 11 files read, 0 holding the learner
learning.reviews: 11 files read, 0 holding the learner
marketing.audience: 8 files read, 0 holding the learner
support.tickets: 8 files read, 0 holding the learner
request: erasure-0001
tables visited: 10
rows deleted: 25
tables rebuilt: analytics.course_performance
tables kept under an exception, columns erased: billing.payments, card_number
snapshots a reader can still time travel to: 0
files no metadata references: 0
tables missing their generated properties: analytics.course_performance
files read: 115
files holding the learner: 0
```

The scan finds the learner in 0 of the 115 files it reads, and no reader can time travel to the learner, since 0 older snapshots remain, and 0 files are unreferenced. The lines after the per-table counts are the evidence: the request's ID, the tables visited, the rows deleted, the table the job rebuilt, the table an exception keeps and the column erased from it, `billing.payments, card_number`, with the reason in `governance/retention.toml`, and the scan's counts. They name no value of the learner's, so keeping them puts none back, which [chapter 8](#produce-evidence-as-data) relies on when it writes them to the audit log.

One line is a warning: `analytics.course_performance` lacks its generated properties, because `lib.pipelines.run` dropped the table and created it again without them. That table has no personal columns. The next table a pipeline rebuilds might have them, and the scan would then report its manifests' bounds as a failed erasure, so the check belongs after every pipeline run.

IDs can stay in a manifest too, since the ID columns keep their statistics. An erased ID that was a file's smallest or largest, such as learner 5000's, stays in the deleted entry's bounds, and the scan reports it. One more commit to the table, followed by an expiry and a sweep, removes it, because PyIceberg's next append leaves out a manifest that holds only deleted files ([source](https://github.com/apache/iceberg-python/blob/pyiceberg-0.12.0/pyiceberg/table/update/snapshot.py)).

The scan covers the warehouse only. Exports, backups, the source systems and logs outside it can hold the learner too, and the same traces can be searched for in each the platform can read.

**Working rule:** End every erasure with a scan of every file the platform stores, for the person's IDs and identifying values, read with each format's own reader, and keep its result, counts and IDs only, as the evidence. Search for as much of each value as every bound keeps unchanged, which is one character less than the truncation length. Treat any file that still holds the person as a failed erasure, not a warning. Search the stores outside the warehouse that you control, and list in the record the ones you can't search.

## What goes wrong in production {#retention-in-production}

A delete that succeeds looks the same whether the data is gone or still in storage, and that is how erasure fails. The first failure here undoes a finished erasure.

### An erased learner comes back {#erased-learner-comes-back}

Besides the CRM's sync, the warehouse reloads all of `crm.learners` nightly from the application's export, and the erasure ran in the warehouse only. Erase learner 3590 everywhere [the plan](#plan-erasure-from-lineage) says, then let the nightly reload and the pipelines that read the table run:

Example `erased-learner-returns` (Python 3.12.15 · duckdb 2.0.0 · openlineage-python 1.55.0 · pyarrow 26.0.1 · pyiceberg 0.12.0 · pyyaml 6.0.3 · sqlglot 30.33.0), passing in Edition 2026.11.31:

```python
"""An erased learner comes back: the nightly reload of crm.learners from the application's export brings the rows back, and
the pipelines rebuild from them. A check against the erasure register catches the rows."""
from datetime import timedelta

from lib import pipelines, retention

LEARNER, REQUEST = 3590, "erasure-0001"
catalog, graph = retention.open_platform()
retention.erase_everywhere(catalog, graph, LEARNER, REQUEST, min_age=timedelta(0))
register = [retention.register_entry(REQUEST, LEARNER)]      # the request's ID and a keyed hash of the learner's ID
print("the erasure register's entry:", register[0]["request"], register[0]["learner"])
learners = catalog.load_table("crm.learners")
print("the learner's rows in crm.learners after the erasure:", retention.count(learners, retention.rows_of([LEARNER])))

# the nightly reload: the application's export still holds the learner, since nobody erased it there
learners.overwrite(retention.crm_export())
# the pipelines that read crm.learners rebuild from the reloaded table
for name in ("learner_activity", "marketing_audience"):
    pipelines.run(catalog, name)
for table in ("crm.learners", "analytics.learner_activity", "marketing.audience"):
    print(f"the learner's rows in {table} after the nightly run:",
          retention.count(catalog.load_table(table), retention.rows_of([LEARNER])))

# hashes each learner_id in every table and looks it up in the register, so no learner's ID is printed
for table, rows in retention.erased_rows(catalog, register).items():
    print(f"the check against the register, rows of erased learners in {table}:", rows)
```

Output:

```text
the erasure register's entry: erasure-0001 c25d8a8b12426596f0c0b1ebfb890246b42ea535791f07256efd16fc6c2cb84b
the learner's rows in crm.learners after the erasure: 0
the learner's rows in crm.learners after the nightly run: 1
the learner's rows in analytics.learner_activity after the nightly run: 1
the learner's rows in marketing.audience after the nightly run: 0
the check against the register, rows of erased learners in analytics.learner_activity: 1
the check against the register, rows of erased learners in crm.learners: 1
```

An erasure done only in the warehouse doesn't last. After it, `crm.learners` holds 0 rows for the learner; after the nightly run it holds 1 again, and `analytics.learner_activity`, rebuilt from it, holds 1. `marketing.audience` stays clear only because the learner's consent comes back withdrawn and their enrollments, which it joins, were erased.

The check that catches this keeps an erasure register: for each request, its ID and a keyed hash (HMAC-SHA256) of the erased learner's ID, and nothing else. The key is the erasure job's secret, kept like the enforcement layer's pseudonymization key in [chapter 6](#write-masks-as-sql) but a different one, because whoever holds it can hash every learner ID and find whom the register names. After every load, restore and rebuild, `lib.retention.erased_rows` hashes each `learner_id` in every table with that column and looks it up, so a restore or a backfill is checked without the register holding anyone's ID. It finds the learner in `analytics.learner_activity` and `crm.learners`, and leaves out `billing.payments`, whose rows the exception keeps. The audit log's record of the erasure, in [chapter 8](#produce-evidence-as-data), holds the same request ID and counts, never the learner, so the register is the only place a request meets a learner, as a hash. We treat a keyed hash of an ID as pseudonymized data, never anonymized.

**Working rule:** Erase in the source system first, or in the same request, and keep a register of keyed hashes of erased IDs that every load filters against and every run checks, with the key held as a secret apart from the data. Treat a row with an erased ID as an incident, as you would a leak.

### Other failures {#other-retention-failures}

- **Old snapshots and files keep "deleted" data.** A retention or erasure job that stops after its delete leaves everything readable by time travel, and in PyIceberg the files stay after expiry too. Run [the expiry](#expire-snapshots) and [the sweep](#remove-unreferenced-files) in the same job, then the scan.
- **Another engine deletes by merge-on-read.** PyIceberg reads the deletion vectors other engines write ([release notes](https://github.com/apache/iceberg-python/releases/tag/pyiceberg-0.12.0)). Rows deleted that way stay in their data files until compaction rewrites them, and a delete file may keep their values, since both kinds of delete file can encode the values of the rows they delete ([Iceberg table spec](https://iceberg.apache.org/spec/)). Compact the affected files before [the expiry](#expire-snapshots) and [the sweep](#remove-unreferenced-files), and scan the delete files too.
- **Backups keep everything the sweep deletes.** They are one of the weak points [the EDPB's report](#verify-the-erasure) names. Decide with the people who answer for the backups how they're erased or aged out, and record it with each request.
- **A derived table nobody erased.** A job that sends no lineage writes a copy the plan can't see, as [chapter 4's uninstrumented export](#job-nobody-instrumented) does. Run `lib.lineage.unaccounted` before every erasure.
- **A rebuilt table loses its properties.** A pipeline that drops and creates its table loses the generated statistics settings, and the next erasure leaves values in its manifests. Apply the properties after every run, or create tables with them.
- **A renamed column loses its setting.** In PyIceberg, the metrics property stays under the column's old name, so after a rename the new name has none. Generate the properties again on every schema change.
- **Retention counted from the wrong date.** A rule that means "after closing" but counts from opening deletes early, and one that counts from a load date deletes late. Store the date the rule means.

## What this chapter decided {#retention-decisions}

- **Retention rules are config,** applied against a day the job is given, never the clock. Each period is the platform's own choice, kept in [the rules](#retention-rules-in-a-file), and closed tickets count from opening, because the table has no `closed_at`.
- **Consent is an attribute the policy reads** at query time; a withdrawal deletes nothing, and tables built with a consent filter are rebuilt.
- **An erasure is planned from the catalog and the lineage:** every table with a `learner_id` column, then every table lineage reaches.
- **Exceptions to erasure are the legal team's,** recorded in `governance/retention.toml` with their reasons; the code keeps what they say until the table's retention rule ends it.
- **Commits record request IDs and rule names,** never a person's values.
- **The book's expiry reads each table's settings,** which PyIceberg's doesn't, and an erasure checks every table in its plan before it deletes anything.
- **The book's sweep removes unreferenced files,** with a minimum age of 72 hours in production.
- **Manifests keep no values of protected columns:** metrics mode `none`, generated from the classifications.
- **Every erasure ends with a scan** of every file, for IDs and 15 characters of each identifying value; its counts are the evidence.
- **The erasure register keeps a keyed hash of each erased ID,** checked after every load, restore and rebuild. Retention and erasure are checked by what remains: a scan after each erasure, a register of hashes, and rules that count from the day the job is given. A scan can pass while data remains somewhere it didn't look, so the register is checked again after every load, restore and rebuild.

# Audit and evidence

By the end of this chapter you can keep an audit log of every access decision and query, with no values in it, chained so an edit shows. You'll write it where the decisions are made, and write down its head, the number and hash of its last record, where the log's writer can't change it, so that an edit to, or a removal of, any record up to that head shows. You'll also explain an old decision after the policy has changed, cross-check the log against the query engine's own record of what it ran, and generate access reviews and evidence reports from the log and the policies, as code that anyone can run again.

When the support lead of the online learning platform this book governs asks why an analyst could read review text last week, or an auditor asks it to show its access rules work, nobody can answer from records. [Chapter 5](#access-policies-as-code) wrote the platform's access policy in Rego, and [chapter 6](#enforcing-policies-on-queries)'s enforcement layer, `lib.enforce.Enforcer`, asks the policy for a decision about every table a query reads and rewrites the query so that DuckDB returns only what the decision allows. The layer forgets each decision once applied. The obvious fix, logging the text of every query, makes a second copy of what the policy protects: a support agent's query can carry a learner's email in its `WHERE` clause, and [chapter 7](#retention-consent-erasure) has to erase a learner from everywhere their data went.

The chapter's terms, as the book uses them: an **audit log** is a record of who did what, when, and with what outcome, kept so that someone can check it later. The platform's audit log holds one **decision record** per query. A **policy revision** is a fingerprint of everything a decision depended on besides the request. A log is **tamper-evident** when a change to it can be detected, though not prevented. An **access review** compares what each person may read with what they use. **Evidence** is a report that shows a control works, generated from records rather than captured from a screen.

The code is the book's shared module `lib.audit`. Its `Audited` class wraps [chapter 6](#enforcing-policies-on-queries)'s `Enforcer` without changing it: it writes a record for every query, and then lets the layer run it. Most examples replay one day of the platform's queries, `lib.audit.DAY`: 13 queries on Oct 1, 2026, from a support agent, finance, an instructor through the course assistant, marketing, a data scientist, the assistant on its own account, and one user the platform doesn't know. The examples run DuckDB 2.0.0, sqlglot 30.33.0 and regopy 1.5.2.

## Record every decision where it's made {#record-every-decision}

An audit record is only as complete as what its writer knows. The query engine knows the SQL it ran, but not who asked or why. The policy knows its decision, but not the query. Only the enforcement layer holds all of it at once: the [principal](#access-policies-as-code), meaning the user or program that asks, the tables and columns the query reads, each table's decision, and the SQL it hands the engine. That is where Open Policy Agent (OPA) writes its own record. OPA's decision logs hold events that describe policy queries, with the policy queried, its input, the bundle metadata, and other information that makes auditing and offline debugging possible ([OPA, decision logs](https://www.openpolicyagent.org/docs/management-decision-logs)). Each event carries a `decision_id`, the `path` of the rule queried, the `input`, the `result`, a `timestamp`, and `bundles[_].revision`, the bundle's revision at evaluation. regopy, which runs the policy inside Python, returns the value of the query and nothing more, so the layer has to build every other field itself.

NIST's catalog of security controls gives a checklist for those fields. Control AU-3 asks that audit records establish six things: (a) what type of event occurred, (b) when it occurred, (c) where, (d) its source, (e) its outcome, and (f) the identity of the individuals, subjects or objects associated with it ([NIST SP 800-53 Rev. 5](https://csrc.nist.gov/pubs/sp/800/53/r5/upd1/final)).

`lib.audit.Audited` takes a catalog and a `Log`, the audit log's file. Its `sql` method takes the same arguments as `Enforcer.sql`, plus `via`, the assistant that sent it, when one did. It plans the query, asks the policy about every table the query reads, writes one record, and only then hands the rewritten query to DuckDB. So no query reaches DuckDB through the layer without a record, and when the record can't be written, the query doesn't run. [Chapter 6](#query-on-someones-behalf) left this chapter one case: when the course assistant answers `instructor-17`, the query runs as the instructor, and both identities belong in the log. The assistant's query for `instructor-17`, and its attempt to put names to the ratings, each leave a record, showing what a record holds when two identities are involved:

Example `decision-record` (Python 3.12.15 · duckdb 2.0.0 · pyarrow 26.0.1 · pyiceberg 0.12.0 · regopy 1.5.2 · sqlglot 30.33.0), passing in Edition 2026.11.31:

```python
"""The course assistant's two queries, sent as instructor-17: one allowed, one denied, each recorded with both identities.
The point rests on the `user` and `via` lines of both records and on the denial in record 2."""
from lib import audit
from lib.catalog import open_catalog
from lib.enforce import Denied
from lib.platform import scratch

# Chapter 6's enforcement layer, wrapped so that every query it's given is recorded before DuckDB runs anything.
log = audit.Log(scratch() / "audit.jsonl")
platform = audit.Audited(open_catalog(["crm.learners", "learning.reviews"]), log)

# instructor-17 asks the course assistant how their reviews look. The assistant writes the SQL; the query runs as the
# instructor, with the assistant recorded as the one that sent it.
question = "select course_id, count(*) as reviews, round(avg(rating), 2) as rating from learning.reviews group by all"
platform.sql(question, user="instructor-17", via="assistant-1").fetchall()

# Then the assistant tries to put names to the ratings, which the instructor may not see. The denial is recorded before
# it's raised, so the script only needs to go on.
try:
    platform.sql("select l.full_name, r.rating from learning.reviews as r join crm.learners as l using (learner_id)",
                 user="instructor-17", via="assistant-1").fetchall()
except Denied:
    pass

for record in log.records():
    print(f"record {record['seq']}:")
    print(audit.show(record))
```

Output:

```text
record 1:
  seq: 1
  event: query
  time: 2026-10-01T09:00:00Z
  catalog: platform
  user: instructor-17
  via: assistant-1
  purpose: null
  query: SELECT course_id, COUNT(*) AS reviews, ROUND(AVG(rating), ?) AS rating FROM learning.reviews GROUP BY ALL
  outcome: allowed
  denial: null
  decision: {"table": "learning.reviews", "columns": ["course_id", "rating"], "allow": true, "row_filters": [{"column": "course_id", "in": [2, 75, 80, 87, 103, 134]}], "column_masks": {}, "reasons": ["instructor may read learning.reviews"]}
  policy_revision: f761e4ad08d4...
  rewritten_digest: 385c5cdabeae...
  prev_hash: 000000000000...
record 2:
  seq: 2
  event: query
  time: 2026-10-01T09:01:00Z
  catalog: platform
  user: instructor-17
  via: assistant-1
  purpose: null
  query: SELECT alias_1.full_name, alias_2.rating FROM learning.reviews AS alias_2 JOIN crm.learners AS alias_1 USING (learner_id)
  outcome: denied
  denial: instructor-17 may not read crm.learners: no group of this user may read this table
  decision: {"table": "crm.learners", "columns": ["full_name", "learner_id"], "allow": false, "row_filters": [], "column_masks": {}, "reasons": ["no group of this user may read this table"]}
  decision: {"table": "learning.reviews", "columns": ["learner_id", "rating"], "allow": true, "row_filters": [{"column": "course_id", "in": [2, 75, 80, 87, 103, 134]}], "column_masks": {}, "reasons": ["instructor may read learning.reviews"]}
  policy_revision: f761e4ad08d4...
  rewritten_digest: null
  prev_hash: 0d3e61eaadeb...
```

Record 1 covers AU-3's six items. `event` is the type, and `time` says when, in UTC with a `Z`, since control AU-8 asks for time stamps that "use Coordinated Universal Time" or state their offset; the examples take it from a fixed clock. `catalog` says where. `user` and `via` are the source: the query ran as `instructor-17`, and `assistant-1` sent it. With the decision's `table`, they are the identities AU-3's item f asks for. `query` and the decision's `columns` say what was read, and `outcome` and the decision's `allow`, `row_filters`, `column_masks` and `reasons` say how it ended, down to the course IDs the row filter kept. The next sections explain `policy_revision`, `rewritten_digest` and `prev_hash`.

Record 2 is a denial, and it holds both tables' decisions: the join may read `learning.reviews` but not `crm.learners`, and one denied table denies the query. Without `via`, the log would say that the instructor tried to read learners' names. With it, a reviewer knows the assistant wrote that query, and looks at the assistant's behaviour as well as the instructor's.

**Working rule:** Write the audit record in the enforcement layer, where the principal, the query and the decision meet, and write it before the query runs, so that a failed write stops the query. Record every identity involved, the person and any assistant acting for them, and every table's decision with its filters, masks and reasons. When an engine enforces the policy itself, as the SQL engine Trino does through OPA, turn on OPA's decision log instead, and make sure its input names every identity involved: Trino's `opa.identity.extra-credentials-keys`, for example, forwards the credentials it lists "to OPA as extraCredentials in the identity context" ([Trino 484, OPA access control](https://trino.io/docs/484/security/opa-access-control.html)).

## Keep values out of the record {#keep-values-out-of-the-record}

The [decision record](#record-every-decision) holds IDs, names of tables and columns, decisions and digests, and no value from the data its query read, because everything an audit log holds becomes data to protect. NIST's discussion of AU-3 warns that the audit trail can reveal personally identifiable information, especially when it records inputs, and its enhancement AU-3(3) asks systems to limit that information in audit records to the elements the privacy risk assessment names ([NIST SP 800-53 Rev. 5](https://csrc.nist.gov/pubs/sp/800/53/r5/upd1/final)). Limit, not remove: AU-3's item f asks for the principal, so every record names one, such as `instructor-17`, one instructor's account. The GDPR defines personal data as "any information relating to an identified or identifiable natural person" ([GDPR, Article 4(1)](https://eur-lex.europa.eu/eli/reg/2016/679/oj)), and a person is identifiable directly or indirectly, in particular through an identifier such as a name or an online identifier. An account ID like this one can be such an identifier. The platform has a sharper reason to keep learners out: [chapter 7](#retention-consent-erasure) erases a learner from every table their data reached, and erasing one record from the [chain](#chain-the-records) this chapter adds can't be told from tampering.

The risk is the query text. A support agent who looks a learner up by the email in a ticket writes that email into the query. `lib.audit.normalized` keeps the query's shape and drops its values: sqlglot parses the query, and every literal, in any of DuckDB's syntaxes, becomes `?`. So does every name the catalog doesn't hold, such as a value typed in double quotes, except that an alias or a CTE's name the query defines becomes `alias_1`, `alias_2` and so on. Comments, which can hold anything, are left out, and a denial's reason is recorded with the same values taken out. The SQL the layer hands DuckDB is kept only as `rewritten_digest`, an HMAC-SHA256 under a key only the layer holds, which the [cross-check](#cross-check-the-query-log) matches against DuckDB's own log. OPA starts from the whole input instead, and can be told to drop what a policy names: "queries the data.system.log.mask path prior to encoding and uploading decision logs" ([OPA, decision logs](https://www.openpolicyagent.org/docs/management-decision-logs)). The platform's layer never writes the values at all.

Here `agent-a05` looks up an EMEA learner by email, then by ID, while `analyst-1` reads learners' emails, which the policy pseudonymizes. Neither value is printed; the lines below say where each is kept:

Example `query-without-literals` (Python 3.12.15 · duckdb 2.0.0 · pyarrow 26.0.1 · pyiceberg 0.12.0 · regopy 1.5.2 · sqlglot 30.33.0), passing in Edition 2026.11.31:

```python
"""A record keeps a query's shape and none of its values, while DuckDB's own query log keeps the email as it was typed.
The printed lines the prose rests on are the asserts about the email and the four count lines at the end."""
from lib import audit
from lib.catalog import open_catalog
from lib.platform import load, scratch

log = audit.Log(scratch() / "audit.jsonl")
platform = audit.Audited(open_catalog(["crm.learners"]), log)

# agent-a05 looks up an EMEA learner by the email in a ticket, then by their ID. Neither value is ever printed.
learners = load("crm.learners").to_pylist()
learner = next(row for row in learners if row["support_region"] == "emea")      # any EMEA learner will do, since the example needs only one
email, learner_id = learner["email"], learner["learner_id"]
platform.sql(f"select learner_id, full_name from crm.learners where email = '{email}'", "agent-a05").fetchall()
platform.sql(f"select email, phone from crm.learners where learner_id = {learner_id}", "agent-a05").fetchall()
platform.sql("select learner_id, email from crm.learners", "analyst-1").fetchall()      # pseudonymized with the key

for record in log.records():
    print("query in the record:", record["query"])
engine = audit.engine_queries(platform.enforcer.connection)      # DuckDB's own query log, literals and all
print("the email, in the audit log:", email in log.path.read_text())
assert email not in log.path.read_text()        # the record keeps `?` where the email was, so nothing matches
print("the email, in DuckDB's query log:", any(email in query for query in engine))
assert any(email in query for query in engine)  # DuckDB logs the SQL text it runs, literals included
print("the pseudonymization key, in DuckDB's query log:", any(platform.enforcer.key in query for query in engine))
assert not any(platform.enforcer.key in query for query in engine)
print("its placeholder, in DuckDB's query log:", any("$pseudonym_key" in query for query in engine))
assert any("$pseudonym_key" in query for query in engine)

# Guessing which learner the agent looked up, from a list of the platform's emails: rebuild the text DuckDB ran with
# each email, digest it, and compare. First with a plain SHA-256 of the query, as a log might keep, then with the
# record's keyed digest, by someone without the layer's key and by someone who holds it.
ran = next(query for query in engine if email in query)
guesses = [ran.replace(email, row["email"]) for row in learners]    # the same text, with each learner's email in its place
logged = log.records()[0]["rewritten_digest"]                    # record 1 is the agent's lookup by email
print("emails tried:", f"{len(guesses):,}")
print("emails a plain SHA-256 of the query confirms:", sum(audit.sha256(g) == audit.sha256(ran) for g in guesses))
print("emails the keyed digest confirms, without the layer's key:",
      sum(audit.digest(g, "a-guessed-key") == logged for g in guesses))     # a wrong key, the only kind a guesser has
print("emails the keyed digest confirms, with the layer's key:", sum(audit.digest(g, platform.key) == logged for g in guesses))
```

Output:

```text
query in the record: SELECT learner_id, full_name FROM crm.learners WHERE email = ?
query in the record: SELECT email, phone FROM crm.learners WHERE learner_id = ?
query in the record: SELECT learner_id, email FROM crm.learners
the email, in the audit log: False
the email, in DuckDB's query log: True
the pseudonymization key, in DuckDB's query log: False
its placeholder, in DuckDB's query log: True
emails tried: 5,000
emails a plain SHA-256 of the query confirms: 1
emails the keyed digest confirms, without the layer's key: 0
emails the keyed digest confirms, with the layer's key: 1
```

The records keep `WHERE email = ?` and `WHERE learner_id = ?`, and a search of the log for the email finds nothing. DuckDB's query log does hold the email, because it records the SQL text DuckDB runs, literals included. The pseudonymization key isn't there: [chapter 6](#write-masks-as-sql) passes the key as a parameter, so DuckDB's log shows the placeholder `$pseudonym_key` where the key would be.

A keyed digest of a query that held an email is a [pseudonym](#consistent-pseudonyms) of that email, and the key is what keeps it one, as [chapter 6](#write-masks-as-sql) showed for the platform's pseudonyms. The rewrite is code in the repository, so anyone with a list of emails, from an old export or a breach, can rebuild the text DuckDB ran for each and test it. Of 5,000 emails tried, a plain SHA-256 of that text confirms 1: the learner the agent looked up. The record's keyed digest confirms 0 without the layer's key, and 1 with it. The key stays with the layer, away from anyone who reads the log, and so does the cross-check, which needs it.

DuckDB takes a value in more ways than a quoted string, and a name can hold one too. A regression test for `normalized` types the same learner's email, name and ID into `agent-a05`'s queries in each of those ways. Those requests leave these traces, one record each:

Example `query-shape-hard-cases` (Python 3.12.15 · duckdb 2.0.0 · pyarrow 26.0.1 · pyiceberg 0.12.0 · regopy 1.5.2 · sqlglot 30.33.0), passing in Edition 2026.11.31:

```python
"""Eleven places to type a learner's values into a query, and what each record keeps of it: the query's shape, with a
`?` for each value and each name the catalog doesn't hold, and a numbered alias for each alias or CTE the query defines.
The printed count of records that hold the email, the name or the ID
is the line the prose rests on."""
import json
import re

from lib import audit
from lib.catalog import open_catalog
from lib.enforce import Denied
from lib.platform import load, scratch

log = audit.Log(scratch() / "audit.jsonl")
platform = audit.Audited(open_catalog(["crm.learners"]), log)

# An EMEA learner's email, name and ID, typed into agent-a05's queries as values and as names. None is ever printed.
learner = next(row for row in load("crm.learners").to_pylist() if row["support_region"] == "emea")
email, name, learner_id = learner["email"], learner["full_name"], learner["learner_id"]
cases = {                                                 # each query puts the same values in one more way
    "a string": f"select learner_id from crm.learners where email = '{email}'",
    "a dollar-quoted string": f"select learner_id from crm.learners where email = $${email}$$",
    "an escape string": f"select learner_id from crm.learners where email = e'{email}'",
    "a list": f"select learner_id from crm.learners where email in ['{email}']",
    "a number": f"select email from crm.learners where learner_id = {learner_id}",
    "a comment": f"select email from crm.learners where learner_id = {learner_id} -- {name}",
    "a name in double quotes": f'select learner_id from crm.learners where full_name = "{name}"',    # a column, to DuckDB
    "a column alias": f'select learner_id as "{email}" from crm.learners',
    "a table alias": f'select "{name}".email from crm.learners as "{name}"',
    "a CTE's name": f'with "{email}" as (select learner_id from crm.learners) select learner_id from "{email}"',
    "a table the catalog doesn't hold": f'select learner_id from "{email}"',
}

for query in cases.values():
    try:
        platform.sql(query, "agent-a05").fetchall()      # a denial is recorded, then raised: the script carries on
    except Denied:
        pass

# What each record keeps of its query. The email or the name anywhere in its line, or the ID in what it keeps, is a
# value that got through, and is never printed.
held = []
for case, line in zip(cases, log.lines()):               # one record per query, in the order they ran
    record = json.loads(line)
    kept = record["query"] + (f" (denied: {record['denial']})" if record["denial"] else "")
    held.append(email in line or name in line or bool(re.search(rf"\b{learner_id}\b", kept)))     # the ID as a whole word
    print(f"{case}:", "(holds a value)" if held[-1] else kept)
print("records holding the email, the name or the ID:", sum(held), "of", len(cases))
```

Output:

```text
a string: SELECT learner_id FROM crm.learners WHERE email = ?
a dollar-quoted string: SELECT learner_id FROM crm.learners WHERE email = ?
an escape string: SELECT learner_id FROM crm.learners WHERE email = ?
a list: SELECT learner_id FROM crm.learners WHERE email IN (?)
a number: SELECT email FROM crm.learners WHERE learner_id = ?
a comment: SELECT email FROM crm.learners WHERE learner_id = ?
a name in double quotes: SELECT learner_id FROM crm.learners WHERE full_name = ? (denied: Column '?' could not be resolved)
a column alias: SELECT learner_id AS alias_1 FROM crm.learners
a table alias: SELECT alias_1.email FROM crm.learners AS alias_1
a CTE's name: WITH alias_1 AS (SELECT learner_id FROM crm.learners) SELECT learner_id FROM alias_1
a table the catalog doesn't hold: SELECT learner_id FROM ? (denied: not a table in the catalog: "?")
records holding the email, the name or the ID: 0 of 11
```

Each record keeps the query's shape and none of its values, allowed or denied alike. The last line counts the records that hold the email, the name or the ID: 0 of 11.

Two things are lost. The normalized text drops harmless values and names too: in the [first record](#record-every-decision), `ROUND(AVG(rating), 2)` became `ROUND(AVG(rating), ?)`, and in the second, the aliases `r` and `l` became `alias_2` and `alias_1`. And the log can't answer "who looked up this learner?". When the platform needs that answer, for a learner who asks who has read their data, keep it in a separate access log, with learner IDs, a retention period and no chain, which [chapter 7](#retention-consent-erasure)'s erasure reaches like any other table.

**Working rule:** Record a query's shape, never its values: replace every literal, in every syntax the engine accepts, and every name the catalog doesn't hold, drop comments, take the same values out of denial reasons, and keep the executed SQL only as a keyed digest whose key the log's readers never get. Never use a plain hash of text that may hold personal data. When you need to know who read one person's data, keep that in a separate log that retention and erasure reach, not in the tamper-evident one.

## Replay an old decision under its own policy {#replay-old-decisions}

A [decision record](#record-every-decision) holds the request, but a decision depends on three things: the request, the policy's rules, and the data the policy reads. The last two change all the time. The platform's policy reads `governance/users.toml`, `governance/metadata.toml` and `learning.courses`, as [chapter 5](#generate-the-data-the-policy-reads) set up, so a support agent moved to another region, or a course added to `learning.courses`, changes decisions without anyone touching Rego. If a record says nothing about what the policy was, its decision can't be explained once either changes: replayed later, it gives a different answer, and the log looks wrong. OPA records the policy's version as `bundles[_].revision`. For policies that call non-deterministic built-ins, such as the time or an HTTP request, OPA can also log their results in `nd_builtin_cache`, which exists for debugging and decision replay ([OPA, decision logs](https://www.openpolicyagent.org/docs/management-decision-logs)). The platform's policy calls none.

`lib.audit.bundle` puts together the policy's modules and its [data document](#generate-the-data-the-policy-reads), the facts the policy reads, generated from those three sources, as OPA ships them in a bundle. `lib.audit.revision` is the SHA-256 of that bundle: a fingerprint that changes when any rule changes, or any fact the policy reads. Every record carries the revision that decided it. `lib.audit.Archive` keeps every bundle the layer has used, under its revision, so a decision can be made again under the policy that made it. A bundle holds rules and the attributes of the platform's users, not data about learners, so the archive can be kept as long as the log.

On the platform, `analyst-1` reads learners' emails and gets pseudonyms. Suppose a later pull request adds it to support for EMEA, the change [chapter 6](#test-who-sees-what) caught with its who-sees-what test. The old record is replayed under each revision:

Example `explain-old-decision` (Python 3.12.15 · duckdb 2.0.0 · pyarrow 26.0.1 · pyiceberg 0.12.0 · regopy 1.5.2 · sqlglot 30.33.0), passing in Edition 2026.11.31:

```python
"""Replays a logged decision under the revision it was made with and under the current one, after a change to the users
file alone. The printed email and match lines are the point: only the archived revision explains the record."""
from lib import audit, policy
from lib.catalog import open_catalog
from lib.platform import scratch

folder = scratch()
archive = audit.Archive(folder / "bundles")         # every policy revision that has decided anything, by revision
log = audit.Log(folder / "audit.jsonl")

# analyst-1 reads learners' emails, which the policy pseudonymizes.
platform = audit.Audited(open_catalog(["crm.learners"]), log, archive=archive)
platform.sql("select learner_id, email from crm.learners", "analyst-1").fetchall()
record = log.records()[0]

# Later, a pull request adds analyst-1 to support for EMEA, as in chapter 6. Only users.toml changes, which reaches
# the policy through its data document, and no Rego.
data = policy.data_document()
data["platform"]["users"]["analyst-1"] = {"groups": ["data-science", "support"], "region": "emea"}
current, logged = audit.bundle(data=data), archive.get(record["policy_revision"])     # the bundle the record was decided under
print("logged under revision:", record["policy_revision"][:12])
print("current revision:", audit.revision(current)[:12])
modules_changed = current["modules"] != logged["modules"]
assert not modules_changed                      # no Rego changed, so the policy modules are the same
print("policy modules changed:", modules_changed)
data_changed = current["data"] != logged["data"]
assert data_changed                             # the data document is the only part that changed
print("data document changed:", data_changed)

# Explain the logged decision by making it again, from the request the record holds, under each revision.
print("email, as logged:", record["decisions"][0]["column_masks"].get("email", "shown as stored"))
for name, revision in [("the current revision", current), ("the logged revision", logged)]:
    again = audit.replay(record, revision)
    matches = again == record["decisions"]
    assert matches == (revision is logged)      # only the revision the record was decided under explains it
    print(f"under {name}, email:", again[0]["column_masks"].get("email", "shown as stored"))
    print(f"under {name}, the decision matches the log:", matches)
```

Output:

```text
logged under revision: f761e4ad08d4
current revision: 0350e789b34d
policy modules changed: False
data document changed: True
email, as logged: pseudonymize
under the current revision, email: shown as stored
under the current revision, the decision matches the log: False
under the logged revision, email: pseudonymize
under the logged revision, the decision matches the log: True
```

The revision changed, from `f761e4ad08d4` to `0350e789b34d`, though no Rego did. The policy modules are the same, and the data document, which now lists the new membership, is not. A fingerprint of the policy files alone would have missed the change. Replayed under the current revision, the decision doesn't match the log: `email` is `shown as stored`, because support sees contact details. An investigator holding only the current policy would conclude the log, or the layer, was wrong. Replayed under the logged revision from the archive, `email` is `pseudonymize` again, and the decision matches the log.

Replay explains the decision, not the rows. A row filter is replayed as a rule, such as support's region or the consent filter `learner_consented`, and which rows passed it depended on that day's data, which may have changed since, or been erased. That split is by design: the log says what the rule was, and the tables, with their own snapshots and retention, say what it matched.

**Working rule:** Fingerprint every decision with a hash of everything it depended on besides the request: the policy's modules and its data document, together. Archive each revision for as long as you keep the log, and explain an old decision by replaying it under its own revision, never under the current one. If a policy reads the clock or calls another service, log those results too, as OPA's `nd_builtin_cache` does.

## Chain the records so an edit shows {#chain-the-records}

The platform's [decision records](#record-every-decision) are evidence only if a change to them can be detected. SP 800-53 asks systems to "Protect audit information and audit logging tools from unauthorized access, modification, and deletion" (AU-9), and, in AU-9(3), to "Implement cryptographic mechanisms to protect the integrity of audit information and audit tools". Its AU-9(3) discussion offers, as an example, "signed hash functions using asymmetric cryptography" ([NIST SP 800-53 Rev. 5](https://csrc.nist.gov/pubs/sp/800/53/r5/upd1/final)), which only the holder of a private key can make. The platform's chain, below, is not that, and protects nothing on its own. NIST's guide to log management, SP 800-92, describes the plainest mechanism: a message digest for each log file, stored securely, so that a change to an archived log is detected ([NIST SP 800-92](https://doi.org/10.6028/NIST.SP.800-92)). The guide dates from 2006. It names MD5 and SHA-1 as the common digests of its day, and its own footnote 25 advises SHA-256, which the platform uses.

A digest per file detects a change to a closed file. A **hash chain** detects a change to a log that's still growing: every record carries a sequence number, `seq`, and the SHA-256 of the line before it, `prev_hash`, so changing a record breaks the link to the next one, and removing one leaves a gap. `lib.audit.check` walks the chain and reports each break. A chain is tamper-evident, not tamper-proof. Schneier and Kelsey designed a log in which an attacker who takes over the logging machine can neither read nor falsify the entries written before. Even there, the attacker "can delete a block of entries (or the entire log file)", and the trusted party the log reports to notices only the next time the machine contacts it ([Schneier and Kelsey, 1999](https://www.schneier.com/wp-content/uploads/2016/02/paper-auditlogs.pdf)). A plain chain like the platform's is weaker: anyone who can write the file can rebuild it, renumbering as they go. So what it proves depends on a chain's **head**, the number of records and the hash of the last one, written down where the log's writer can't change it. An edit to, or a removal of, any record up to that head shows, however it's hidden. After the head, a chain rebuilt or cut passes every check, and without a head, so does a chain rebuilt from the start. `check` takes the head as an anchor.

To test the chain, the example replays `lib.audit.DAY`, the 13 queries the platform received on Oct 1, and writes the head down at noon, at record 9. Then it tampers with copies of the log in five ways. Before noon, record 3, `agent-a05`'s lookup of a learner's contact details, is handed to another agent, once as a plain edit and once with the chain rebuilt. After noon, record 12, the denied read by `someone-else`, the user the platform doesn't know, is removed, once plainly and once with the chain rebuilt; and the day's last two records are cut off. Before you run it, predict which of the five the chain finds on its own, and which against the noon head.

Example `chain-catches-edit` (Python 3.12.15 · duckdb 2.0.0 · pyarrow 26.0.1 · pyiceberg 0.12.0 · regopy 1.5.2 · sqlglot 30.33.0), passing in Edition 2026.11.31:

```python
"""Replays the day's queries into a chained log, writes the head down at noon, then tampers with copies of the log five
ways and checks each against the chain alone and against the noon head. The point rests on the two checks of each
change: the chain finds the plain edit and the plain removal on its own, and only the head finds the rebuilt chain."""
import json

from lib import audit
from lib.catalog import open_catalog
from lib.platform import scratch

log = audit.Log(scratch() / "audit.jsonl")
platform = audit.Audited(open_catalog(["crm.learners", "learning.reviews", "billing.payments", "support.tickets"]), log)
audit.run_day(platform, [query for query in audit.DAY if query[0] < "12:00"])     # the morning; times compare as strings
anchor = log.head()                               # the head, written down at noon somewhere the log's writer can't reach
audit.run_day(platform, [query for query in audit.DAY if query[0] >= "12:00"])    # the afternoon
lines = log.lines()
print("records:", len(lines))
print("head written down at noon, at record:", anchor["records"])
print("record 3:", json.loads(lines[2])["user"], "reads", json.loads(lines[2])["query"])     # record 3 is index 2


def rechained(lines: list[str]) -> list[str]:
    """The chain built again, every record renumbered and every prev_hash recomputed: what anyone who can rewrite the
    log's file can do."""
    rebuilt, previous = [], audit.GENESIS          # the first record points at 64 zeros, not at a record
    for seq, line in enumerate(lines, start=1):
        rebuilt.append(audit.canonical(json.loads(line) | {"seq": seq, "prev_hash": previous}))
        previous = audit.sha256(rebuilt[-1])
    return rebuilt


# Before noon, record 3 handed to another agent. After noon, record 12 (someone-else's denied read) removed, and the
# day's last two records cut off.
edited = lines[:2] + [audit.canonical(json.loads(lines[2]) | {"user": "agent-a01"})] + lines[3:]     # index 2 is record 3
removed = lines[:11] + lines[12:]                                                                    # index 11 is record 12
for name, version in [("untouched", lines), ("record 3 edited", edited), ("record 3 edited, chain rebuilt", rechained(edited)),
                      ("record 12 removed", removed), ("record 12 removed, chain rebuilt", rechained(removed)),
                      ("last 2 records removed", lines[:-2])]:
    print(f"{name}, checked alone:", "; ".join(audit.check(version)) or "no problem found")
    print(f"{name}, checked against the noon head:", "; ".join(audit.check(version, anchor)) or "no problem found")
```

Output:

```text
records: 13
head written down at noon, at record: 9
record 3: agent-a05 reads SELECT full_name, email, phone FROM crm.learners WHERE learner_id = ?
untouched, checked alone: no problem found
untouched, checked against the noon head: no problem found
record 3 edited, checked alone: record 3 was changed: record 4 doesn't follow it
record 3 edited, checked against the noon head: record 3 was changed: record 4 doesn't follow it
record 3 edited, chain rebuilt, checked alone: no problem found
record 3 edited, chain rebuilt, checked against the noon head: the chain no longer passes through the head written down at record 9
record 12 removed, checked alone: record 12 is missing: record 13 follows record 11
record 12 removed, checked against the noon head: record 12 is missing: record 13 follows record 11
record 12 removed, chain rebuilt, checked alone: no problem found
record 12 removed, chain rebuilt, checked against the noon head: no problem found
last 2 records removed, checked alone: no problem found
last 2 records removed, checked against the noon head: no problem found
```

On its own, the chain catches the plain edit, `record 3 was changed: record 4 doesn't follow it`, and the plain removal, `record 12 is missing: record 13 follows record 11`, and nothing else. It misses both rebuilt chains, which are consistent from start to end, and the cut, since no record follows the last one to notice.

A head protects only the records written before it. Against the noon head, the edit made before noon still shows with the chain rebuilt, `the chain no longer passes through the head written down at record 9`. Everything hidden after noon passes: the removal with the chain rebuilt gives `no problem found`, and the cut gives `no problem found`.

This is why the [rule about values](#keep-values-out-of-the-record) isn't optional. Erasing a learner from a chained log means changing or removing a record, and wherever a check can see that, it can't tell it from tampering. A log that holds no data about learners never meets [chapter 7](#retention-consent-erasure)'s erasure. The IDs of the people who query are another matter: give the log a retention period, and start a new chain each period, with its last head written down, so a whole period can be deleted when it expires.

Checking a chain means re-reading every record. Crosby and Wallach describe tree-shaped logs that can prove "that individual logged events are still present, and that the log, as seen now, is consistent with how it was seen in the past", with proofs of logarithmic size, an improvement on earlier linear constructions ([Crosby and Wallach, 2009](https://www.usenix.org/legacy/event/sec09/tech/full_papers/crosby.pdf)). For one platform's own log we recommend a chain checked daily, and a tree when outside parties audit a log too large to re-read.

**Working rule:** Chain every record to the one before with SHA-256 and a sequence number, and count on the chain to protect only the records up to the last head written down outside the layer's control. Write the head down often, to an append-only store or to another system under other credentials, as often as the number of unprotected records you can accept; if that number is zero, send each record off the machine as it's written. Check the chain against the last head on a schedule. If you need what AU-9(3) describes, have each head signed by a system whose key the layer never holds. Never chain a record that might one day have to be erased for one person; let whole periods expire instead.

## Cross-check the log against DuckDB's query log {#cross-check-the-query-log}

The platform's [audit log](#record-every-decision) records every query through the enforcement layer, and only those. A query run on the layer's DuckDB connection directly, by a notebook helper or a test that was handed the connection, gets no decision and no record. Noticing it takes a second record, kept by the engine. DuckDB 2.0.0 has one, off by default: its logging framework has a `QueryLog` type, which records the queries DuckDB executes, turned on with `CALL enable_logging('QueryLog')` and read from the `duckdb_logs` view ([DuckDB, logging](https://duckdb.org/docs/2.0/operations_manual/logging/overview.html)). A separate setting, `log_query_path`, names a file to log queries to ([DuckDB, configuration](https://duckdb.org/docs/2.0/configuration/overview.html)), and the layer's connection refuses it, as the [section on production](#audit-in-production) shows.

DuckDB's log isn't an audit log. Each entry holds the SQL text, with connection, transaction and query IDs and a timestamp: no user, and no decision. It sees only the rewritten query, and it belongs to the process it records. What it can do is check the audit log: every query DuckDB ran should be the SQL that some decision record handed it. `Audited` turns DuckDB's log on and locks the connection's configuration when it starts. `Audited.check_engine_log` matches each query in DuckDB's log, by keyed digest, against the `rewritten_digest` of the records written since the last check, then empties DuckDB's log, which holds the queries' values. A relation's query runs "when a method of the Output section reads the relation, and it runs again on every read" ([DuckDB, relational API](https://duckdb.org/docs/2.0/clients/python/relational_api)), and a narrowed relation runs SQL of its own, which no record handed DuckDB. So `Audited.sql` fetches each result itself, once, and hands the rows back as a relation on a connection of its own: whatever the caller does with it, DuckDB's log holds each allowed query once.

The first check replays the day's 13 queries and checks them. Then two more queries go through the layer, one its caller never fetches and one it narrows to five rows, and a notebook helper reads a learner's email with the layer's connection, around the layer. A second check follows:

Example `query-log-cross-check` (Python 3.12.15 · duckdb 2.0.0 · pyarrow 26.0.1 · pyiceberg 0.12.0 · regopy 1.5.2 · sqlglot 30.33.0), passing in Edition 2026.11.31:

```python
"""DuckDB's query log checked against the audit log: first on the day's queries, then after two more queries through the
layer and a read around it. The point rests on the last check's unrecorded query, the one read that bypassed the layer."""
from lib import audit
from lib.catalog import open_catalog
from lib.platform import scratch

log = audit.Log(scratch() / "audit.jsonl")
platform = audit.Audited(open_catalog(["crm.learners", "learning.reviews", "billing.payments", "support.tickets"]), log)
day = audit.run_day(platform)
print("queries allowed:", day["allowed"])
print("queries denied:", day["denied"])

# DuckDB's own query log, kept on the layer's connection, against the records written since the last check. The check
# then empties DuckDB's log, which holds the queries' values.
found = platform.check_engine_log()
print("queries DuckDB ran:", found["engine queries"])
print("queries DuckDB ran with no decision record:", len(found["unrecorded"]))

# Two more queries through the layer: one whose caller never fetches it, and one the caller narrows to five rows.
platform.sql("select learner_id, country from crm.learners", "analyst-1")        # the layer fetches it, so DuckDB logs it once
platform.sql("select learner_id, country from crm.learners", "analyst-1").limit(5).fetchall()    # narrowed after the layer's fetch
# Then a notebook helper that was handed the layer's connection reads a learner's email with it, around the layer.
platform.enforcer.connection.sql("select email from crm.learners where learner_id = 541").fetchall()    # no record for this one
found = platform.check_engine_log()
print("next check, records checked:", " to ".join(map(str, found["records checked"])))
print("next check, queries with no decision record:", len(found["unrecorded"]))
for query in found["unrecorded"]:
    print("unrecorded:", query)
print("next check, decision records DuckDB has no query for:", len(found["not in the engine log"]))
```

Output:

```text
queries allowed: 9
queries denied: 4
queries DuckDB ran: 9
queries DuckDB ran with no decision record: 0
next check, records checked: 14 to 15
next check, queries with no decision record: 1
unrecorded: SELECT email FROM crm.learners WHERE learner_id = ?
next check, decision records DuckDB has no query for: 0
```

The cross-check matches the day's queries to their records: 9 were allowed and 4 denied, and DuckDB ran 9: one for each allowed query and none for the denied ones, and 0 without a record.

The next check finds the read around the layer and nothing else. The check covers records 14 to 15, the two queries through the layer, and raises no alarm. The read around the layer shows up as 1 query with no decision record, printed without its values. The last line checks the other way, for records whose query DuckDB has no trace of: 0 here.

What the check catches depends on who holds the connection. It catches a read around the layer by code that leaves DuckDB's log alone, such as this notebook helper, a test, or a job handed the connection by mistake. It doesn't catch code that holds the connection and means to hide. A read made after a check, with DuckDB's log emptied again before the layer's next query, leaves nothing for either direction to find. And DuckDB's `write_log` function takes the log type as an argument, so it can write `QueryLog` entries ([DuckDB source, v2.0.0](https://github.com/duckdb/duckdb/blob/v2.0.0/src/function/scalar/system/write_log.cpp)); the configuration lock doesn't stop a function. A guarantee against such code needs the engine's log written outside the process that holds the connection.

The check also finds the layer's own shortcuts, such as [chapter 6](#test-who-sees-what)'s `observe`, which reads the stored table on the layer's connection with no user behind it. Give such queries records of their own, with the layer as the user, rather than exceptions in the check.

**Working rule:** Turn on the engine's own query log wherever you enforce policies, fetch every result inside the layer, and check the log on a schedule against the records written since the last check, in both directions: every query the engine ran must match a record, and every allowed record must match a query. Then empty it, since it keeps the queries' values. Treat a mismatch as an incident to explain, not noise to filter. Count on the check to catch mistakes, not code that holds the connection and means to hide: against that, only a log written outside the process helps. Never use the engine's log as the audit log: it has no user and no decision.

## Generate the access review from the log {#generate-the-access-review}

The platform's [audit log](#record-every-decision) says what each person read, and the policy says what each person may read; an access review puts the two side by side. SP 800-53 asks organizations to review accounts for compliance with account management requirements (AC-2 j), and to review the privileges assigned to roles and classes at a set frequency, to validate the need for them, reassigning or removing them where necessary (AC-6(7)) ([NIST SP 800-53 Rev. 5](https://csrc.nist.gov/pubs/sp/800/53/r5/upd1/final)). A list of who holds which role answers neither, because it says nothing about need. Code can compute the facts a review needs: what the policy allows each person, what the log shows they used, what they were denied, and memberships that point at nothing. Whether to remove access stays a decision for the data's owners.

`lib.audit.review` evaluates the platform's policy for every user in `users.toml` against every table, reads the log, and lists for each user the tables they may read, the tables they read, the grants they didn't use, their denied queries with the assistant that sent them, and anything someone should look at. To give it something stale to find, the example adds `analyst-2` to the users, still listed in `bi`, a group the policy no longer has, as happens when a group is retired and nobody edits the users file:

Example `access-review` (Python 3.12.15 · duckdb 2.0.0 · pyarrow 26.0.1 · pyiceberg 0.12.0 · regopy 1.5.2 · sqlglot 30.33.0), passing in Edition 2026.11.31:

```python
"""An access review of every user in the users file, and every user the log names, against the policy and the day's log,
with a stale group membership added to find. The printed `look at` lines and the grants-not-used count are what the prose
rests on."""
from lib import audit, policy
from lib.catalog import open_catalog
from lib.platform import TABLES, scratch

log = audit.Log(scratch() / "audit.jsonl")
platform = audit.Audited(open_catalog(["crm.learners", "learning.reviews", "billing.payments", "support.tickets"]), log)
audit.run_day(platform)      # the day's queries, so that the review has a log to read

# users.toml as the review finds it: analyst-2 is still listed in bi, a group the policy no longer has.
data = policy.data_document()
data["platform"]["users"]["analyst-2"] = {"groups": ["bi"]}
rows = audit.review(audit.bundle(data=data), log.records(), TABLES)      # all eight of the platform's tables

for row in rows:
    print(f"{row['user']}: may read {len(row['may read'])}, read {len(row['read'])}, denied {row['denied']}"
          + (f" (sent by {', '.join(row['denied via'])})" if row["denied via"] else ""))
    if row["unused"]:            # a grant unused: a table the policy lets the user read, and the log never shows them reading
        print("  not used:", ", ".join(row["unused"]))
    for finding in row["findings"]:
        print("  look at:", finding)
print("users reviewed:", len(rows))
print("grants not used:", sum(len(row["unused"]) for row in rows))
print("users to look at:", sum(bool(row["findings"]) for row in rows))     # any finding at all needs a person
print("denied queries flagged at:", audit.settings()["review"]["denied_queries_flagged"])     # from governance/audit.toml
```

Output:

```text
agent-a01: may read 2, read 0, denied 0
  not used: crm.learners, support.tickets
agent-a05: may read 2, read 2, denied 0
agent-a09: may read 2, read 0, denied 0
  not used: crm.learners, support.tickets
analyst-1: may read 6, read 2, denied 0
  not used: billing.payments, learning.courses, learning.enrollments, learning.events
analyst-2: may read 0, read 0, denied 0
  look at: in group bi, which the policy doesn't know
assistant-1: may read 2, read 1, denied 0
  not used: learning.courses
finance-1: may read 3, read 1, denied 0
  not used: learning.courses, learning.enrollments
instructor-17: may read 3, read 1, denied 3 (sent by assistant-1)
  not used: learning.courses, learning.enrollments
  look at: 3 denied queries
marketing-1: may read 2, read 1, denied 0
  not used: learning.courses
someone-else: may read 0, read 0, denied 1
  look at: not in users.toml
users reviewed: 10
grants not used: 14
users to look at: 3
denied queries flagged at: 3
```

Over Oct 1, the day `lib.audit.DAY` replays, 14 grants went unused, and `agent-a01` and `agent-a09` read nothing at all. One day is too short to remove anyone's access, since support works in shifts and finance reads some tables only when it closes the month, so these counts are leads, not decisions. A grant unused for a whole review period, such as a quarter, is what to bring to the table's owner. 3 users need a person to look at them:

- `analyst-2` is in a group no rule knows. The membership grants nothing today, and would grant whatever a future rule for `bi` allows, so remove it.
- `someone-else` isn't in `users.toml`, so a principal the platform doesn't know tried to read learners' emails.
- `instructor-17`'s line reads `may read 3, read 1, denied 3 (sent by assistant-1)`. In the two minutes after answering the instructor's first question, the assistant tried learners, then learners joined to reviews, then tickets. That is an assistant trying tables one by one, and without `via` the review would have blamed the instructor.

The threshold for flagging, 3 denied queries, is a rule of thumb, kept in `governance/audit.toml` with its reasoning, so that changing it is a reviewed change, like a policy change.

**Working rule:** Generate access reviews from the policy and the audit log, never from a list of role memberships: what each person may read against what they read, their denials, and memberships the policy no longer knows. Run them over a period long enough for every job's rhythm, such as a quarter, and send each finding to the owner of the table or group it concerns. Remove a grant nobody used in the period unless its owner says why it's needed.

## Produce evidence as data {#produce-evidence-as-data}

The [chain check](#chain-the-records), the [cross-check](#cross-check-the-query-log) and the [access review](#generate-the-access-review) are worth most when someone outside the team can check them. The EU's General Data Protection Regulation (GDPR) puts that duty on the organization, as its accountability principle: "The controller shall be responsible for, and be able to demonstrate compliance with, paragraph 1 ('accountability')" ([GDPR, Article 5(2)](https://eur-lex.europa.eu/eli/reg/2016/679/oj)). What counts as demonstrating is a question for lawyers; this section shows what code can produce. NIST's assessment procedures for account management name the records an assessor examines, among them access authorization records, account management compliance reviews and system audit records ([NIST SP 800-53A Rev. 5, in NIST's OSCAL catalog](https://github.com/usnistgov/oscal-content/blob/main/nist.gov/SP800-53/rev5/json/NIST_SP-800-53_rev5_catalog.json)). A screenshot of a console is none of these: nobody can run it again, compare it with last quarter's, or tell which policy it showed.

`lib.audit.evidence` builds the platform's report from records and code, in these sections:

- the policy, with the digest of each module and of the data document;
- the policy's unit tests, from [chapter 5](#unit-test-the-policy-in-rego);
- the who-sees-what table, checked against the policy;
- classification coverage, from [chapter 2](#test-the-catalog-against-the-metadata)'s metadata checks;
- the audit log's chain check;
- the cross-check against DuckDB's log;
- the erasures the log records;
- the access review.

Every section names the policy revision it was computed under, the digests of its inputs, and what anyone needs to compute it again: the archived bundle, the test files, `metadata.toml` and the tables' schemas, the audit log. The engine cross-check can be computed again only in part, and says so. It keeps the keyed digest of every query DuckDB ran, so anyone holding the log can redo its two counts, but only the layer could read DuckDB's log, once, before emptying it. Generate the report for Oct 1, with every table and pipeline output in the catalog, [chapter 7](#retention-consent-erasure)'s erasure of a learner logged at 07:00, and the day's queries:

Example `evidence-report` (Python 3.12.15 · duckdb 2.0.0 · openlineage-python 1.55.0 · pyarrow 26.0.1 · pyiceberg 0.12.0 · pyyaml 6.0.3 · regopy 1.5.2 · sqlglot 30.33.0), passing in Edition 2026.11.31:

```python
"""The platform on Oct 1, after an erasure and the day's queries: the first evidence report, written to a file as data, with
each section's result printed. The printed checks, the erasure's lines and the head are what the prose rests on."""
import json
from datetime import timedelta

from lib import audit, retention
from lib.platform import scratch

# The platform on Oct 1: every table and pipeline output, chapter 7's erasure of learner 3590 at 07:00, then the day's
# queries through the audited layer.
catalog, graph = retention.open_platform(statistics_off=True)      # statistics off for personal columns: each table rewritten under them, old files swept
log = audit.Log(scratch() / "audit.jsonl")
_, found, done = retention.erase_everywhere(catalog, graph, 3590, "erasure-0001", min_age=timedelta(0))    # zero: no file age guard
audit.record_erasure(log, "erasure-0001", done, retention.scan(catalog, found), at=audit.MIDNIGHT.replace(hour=7))
platform = audit.Audited(catalog, log)
audit.run_day(platform)

# The first report, so there's no head from an earlier one to check the chain against.
report = audit.evidence(platform, catalog, anchor=None)
reports = scratch()          # in production, another system, which the log's writer can't change
(reports / "evidence-2026-10-01.json").write_text(json.dumps(report, indent=2, sort_keys=True), encoding="utf-8")
result = {name: section["result"] for name, section in report.items()}
print("sections:", ", ".join(report))
print("policy tests passed:", f"{result['policy tests']['passed']} of {result['policy tests']['of']}")
print("who-sees-what cases the policy agrees with:", f"{result['who sees what']['agree']} of {result['who sees what']['of']}")
for problem in result["classification coverage"]["problems"]:
    print("classification problem:", problem)
print("audit log records:", result["audit log"]["records"])
print("chain problems, checked alone:", len(result["audit log"]["chain problems"]))
print("queries DuckDB ran with no decision record:", len(result["engine cross-check"]["unrecorded"]))
print("decision records DuckDB has no query for:", len(result["engine cross-check"]["not in the engine log"]))
for erasure in result["erasures"]:
    print(f"erasure {erasure['request']}: {erasure['rows_deleted']} rows deleted, "
          f"{erasure['files_holding_the_person']} of {erasure['files_read']} files hold the person")
    for kept in erasure["kept"]:
        print(f"erasure {erasure['request']}, kept under an exception: {kept['table']}, decided by {kept['decided_by']},",
              f"{', '.join(kept['columns_erased'])} erased")
print("chain head written down, at record:", report["audit log"]["inputs"]["head"]["records"])     # the next report's anchor
print("engine cross-check, computed again from:", "; ".join(report["engine cross-check"]["computed again from"]))
print("engine cross-check, not computed again:", report["engine cross-check"]["not computed again"])
```

Output:

```text
sections: policy, policy tests, who sees what, classification coverage, audit log, engine cross-check, erasures, access review
policy tests passed: 12 of 12
who-sees-what cases the policy agrees with: 12 of 12
classification problem: ai.review_corpus: no metadata
classification problem: marketing.audience: no metadata
audit log records: 14
chain problems, checked alone: 0
queries DuckDB ran with no decision record: 0
decision records DuckDB has no query for: 0
erasure erasure-0001: 25 rows deleted, 0 of 115 files hold the person
erasure erasure-0001, kept under an exception: billing.payments, decided by legal team, card_number erased
chain head written down, at record: 14
engine cross-check, computed again from: the audit log; the engine digests, for the two counts
engine cross-check, not computed again: the reading of DuckDB's log, and the unrecorded queries' shapes: the layer read that log once, then emptied it
```

The report shows what fails as well as what passes, because a report that can only show passing checks is evidence of nothing. The policy passes 12 of 12 unit tests and agrees with 12 of 12 who-sees-what cases. Classification coverage isn't clean: `ai.review_corpus` and `marketing.audience` have no metadata, the gaps that [chapters 2](#a-catalog-you-can-test) and [4](#lineage-from-pipelines) found and that the policy meanwhile treats as sensitive.

The log's own checks come back clean, but they prove less than they look. The chain has 0 problems, checked alone, since no earlier report wrote a head down. The cross-check catches mistakes, not code that means to hide ([above](#cross-check-the-query-log)). DuckDB ran 0 queries without a decision record, and 0 records have no query.

The erasure's lines, `25 rows deleted, 0 of 115 files hold the person` and `billing.payments, decided by legal team, card_number erased`, come from chapter 7's erasure, as the log records it: the request's ID, the tables, the counts, each table an exception kept with the reason `governance/retention.toml` gives, and the scan's result, never the learner's ID or values, so the record is safe to chain. The European Data Protection Board recommends that controllers "Verify that erasure has been carried out and be able to demonstrate such erasure" ([EDPB, 2026](https://www.edpb.europa.eu/system/files/2026-02/edpb_cef-report_2025_right-to-erasure_en.pdf)); this line records what the platform's own scan found. The report also writes down the chain's head, at record 14. Kept in another system, it is the anchor the next report checks the chain against: an edit to, or a removal of, any record up to it will show then, and a later record is protected only once a later head covers it.

The report claims nothing about any law or framework. It states what was checked, under which policy, from which inputs, and with what result.

**Working rule:** Generate evidence from records with code, on a schedule and on every policy change, and stamp each section with the policy revision, the digests of its inputs and what it takes to compute it again, saying which parts nobody can. Keep the reports where the log's writer can't change them, include failing findings rather than filtering them out, and let each report check the chain against the head the last one wrote down. Never send a screenshot where a report can be generated; [chapter 10](#governance-in-ci) runs checks like these on every pull request.

## What goes wrong in production {#audit-in-production}

Nothing breaks when an audit trail stops being complete, true or read. The failure that matters most is logging that the audited code can switch off, and DuckDB's query log is that kind: it belongs to the process it records. Two enforcement layers face the same attack. The first doesn't lock its configuration, like any layer built without `lib.audit`; code holding its connection tries to log to a file, switches the log off, then reads a learner's email around the layer. The second is `lib.audit`'s, locked; such code tries to switch the log off, reads around the layer, then empties the log:

Example `query-log-switched-off` (Python 3.12.15 · duckdb 2.0.0 · pyarrow 26.0.1 · pyiceberg 0.12.0 · regopy 1.5.2 · sqlglot 30.33.0), passing in Edition 2026.11.31:

```python
"""Code holding a layer's connection tries to switch DuckDB's query log off, first on a layer that doesn't lock its
configuration and then on one that does. The printed lines show the switch working on the first, failing on the second,
and the gap that emptying the log leaves in the second layer's check."""
import duckdb

from lib import audit
from lib.catalog import open_catalog
from lib.platform import scratch

AROUND = "select email from crm.learners where learner_id = 541"       # a read around the enforcement layer
folder = scratch()

# A layer that doesn't lock its configuration: code holding its connection tries a file log, then switches logging off.
unlocked = audit.Audited(open_catalog(["crm.learners"]), audit.Log(folder / "unlocked.jsonl"), lock=False)
connection = unlocked.enforcer.connection
try:
    connection.execute("set log_query_path = 'queries.log'")       # the layer's connection refuses to write a file
except duckdb.Error as error:
    print("a query log in a file, on the layer's connection:", type(error).__name__)
connection.execute("CALL disable_logging()")                       # nothing stops the switch here
connection.sql(AROUND).fetchall()                                  # so this read leaves nothing in DuckDB's log
for query in unlocked.check_engine_log()["unrecorded"]:            # only the statements made before the switch remain
    print("unlocked, unrecorded:", query)

# lib.audit's layer locks its configuration: switching the log off is refused, but emptying it isn't.
log = audit.Log(folder / "locked.jsonl")
locked = audit.Audited(open_catalog(["crm.learners"]), log)
locked.sql("select learner_id, email from crm.learners", "analyst-1").fetchall()     # its entry in DuckDB's log is emptied below
connection = locked.enforcer.connection
try:
    connection.execute("CALL disable_logging()")
except duckdb.Error as error:
    print("locked, disable_logging():", type(error).__name__)
connection.sql(AROUND).fetchall()
connection.execute("CALL truncate_duckdb_logs()")                  # emptying isn't locked, so the log loses its entries
found = locked.check_engine_log()
print("locked, after truncate_duckdb_logs(), queries with no decision record:", len(found["unrecorded"]))
print("locked, decision records DuckDB has no query for:", len(found["not in the engine log"]))
```

Output:

```text
a query log in a file, on the layer's connection: PermissionException
unlocked, unrecorded: CALL disable_logging()
unlocked, unrecorded: SET log_query_path = ?
locked, disable_logging(): InvalidInputException
locked, after truncate_duckdb_logs(), queries with no decision record: 0
locked, decision records DuckDB has no query for: 1
```

On the unlocked layer, `disable_logging()` works, and the read that follows leaves no trace. The [cross-check](#cross-check-the-query-log) finds only the statements before the silence: the refused `SET log_query_path`, and `CALL disable_logging()` itself. The refusal is a `PermissionException`, because [chapter 6](#find-what-a-query-reads) forbids the layer's connection to touch files.

On the locked layer, the switch fails and emptying the log doesn't. Switching off fails with an `InvalidInputException`, but `truncate_duckdb_logs()` works, and afterwards the check finds 0 unrecorded queries. The check in the other direction notices the gap: 1 decision record has no query in DuckDB's log, so something emptied the log. It notices only because a record fell in the emptied stretch; code that writes entries back with `write_log` would leave the log looking whole.

None of this stops code that holds the connection. In production, enforce in an engine that runs as a service of its own, such as Trino with OPA, so that its log is written outside the process of anyone who queries it. Store the audit records on another system, as control AU-9(2) asks: "Store audit records [Assignment: frequency] in a repository that is part of a physically different system or system component than the system or component being audited" ([NIST SP 800-53 Rev. 5](https://csrc.nist.gov/pubs/sp/800/53/r5/upd1/final)). We recommend sending each record as it's written, and holding the chain's head elsewhere again.

Three more ways the trail goes wrong:

- **Logs nobody reads.** NIST's guide to log management makes the case for reading them: routine reviews help identify security incidents, policy violations, fraudulent activity and operational problems shortly after they occur (§2.2, [NIST SP 800-92](https://doi.org/10.6028/NIST.SP.800-92)). A log that is only written costs storage and protects nothing. Schedule the access review and the evidence report, give every finding an owner, and alert on what can't wait for a quarter, such as a user flagged for denied queries.
- **Decisions that can't be explained after a policy changed.** A record without a policy revision, or a revision without an archived bundle, replays under the current policy and contradicts itself. Fingerprint the data document with the rules, since a change to the data document changes decisions as surely as one to the rules.
- **Logs that hold personal data.** Personal data reaches logs by many routes: query literals, the engine's own query log, error messages that print values, as [chapter 6](#column-masks-leak) showed for DuckDB's cast errors, and the layer's debugging output. A parser's warnings are one more: sqlglot's warning when it keeps a statement as text quotes the statement ([sqlglot source, v30.24.0](https://github.com/tobymao/sqlglot/blob/v30.24.0/sqlglot/parser.py)), so `lib.audit` silences sqlglot while it parses a user's query. Keep the audit log to IDs and digests, and give every other log a retention period that erasure reaches.
- **A decision log that isn't complete.** OPA can drop decision events by rule, through `system.log.drop`, and "OPA will drop events if the rate limit is exceeded" ([OPA, decision logs](https://www.openpolicyagent.org/docs/management-decision-logs)). Count the engine's queries against the decision records, as the cross-check does, rather than assume every decision was logged.
- **A head kept next to the log.** An anchor stored where the log's writer can change it detects nothing the chain doesn't. Keep it in another system, under other credentials.

## What this chapter decided {#audit-decisions}

- **One record per query, written by the enforcement layer before the query runs.** `lib.audit.Audited` wraps chapter 6's `Enforcer` without changing it, and fetches each result itself. The record covers AU-3's six items, the principal and the assistant that sent the query among them.
- **Records hold IDs, never values.** Every literal, in any of DuckDB's syntaxes, and every name the catalog doesn't hold is replaced, in the query and in the denial's reason, and comments are removed. The SQL DuckDB ran is kept as an HMAC-SHA256 digest under a key only the layer holds, since with the key the digest confirms a guess. Erasures are recorded by request ID.
- **Every record names its policy revision,** the SHA-256 of the policy's modules and data document, and every revision is archived for replay.
- **Records are chained with SHA-256 and numbered,** and the chain protects only the records up to the last head written down outside the layer's control. Each evidence report writes the head down; we recommend doing it more often.
- **DuckDB's query log is a cross-check for mistakes, not the audit log.** The layer turns it on, locks its configuration, checks it both ways against the records since the last check, then empties it. Code holding the connection can defeat it.
- **Access reviews and evidence reports are generated** from the policy, the log and the catalog, each section stamped with its policy revision and inputs. The threshold for flagging denied queries, 3, is a rule of thumb in `governance/audit.toml`. An audit trail is only as good as what the audited code cannot change, and the checks in this chapter say which parts of the system it can. The record that matters is the one the code being audited has no way to switch off.

# Governing data for AI

By the end of this chapter you can build the data an AI assistant learns from under the same rules as every other read of your platform. You'll admit only records whose authors agreed to that use, replace their personal data with pseudonyms and measure what the detector missed, remove duplicates found with MinHash, keep the source of every record with it, and have the pipeline write and check a dataset card on every run. You'll also be able to quote what the EU AI Act says about training data.

The platform, the online learning platform this book follows, wants a course assistant that answers from what learners write in their reviews. It plans to ground the assistant on that text first, and later to fine-tune it on the same text. `review_corpus`, one of the four SQL pipelines that [chapter 4](#lineage-from-pipelines) captures lineage from, copies every row of `learning.reviews` into `ai.review_corpus`, with each review's course and rating. [Chapter 2](#a-catalog-you-can-test) found that `ai.review_corpus` has no owner and no classification, and the copy has four more problems. It holds every review, whether or not its author agreed to their text training an AI. Its text holds names and emails, which [chapter 3](#finding-personal-data) found in reviews. Most of its reviews repeat each other. And nothing in it says where a record came from, so nobody can show that a review was allowed in, or find it again when its author changes their mind.

This chapter rebuilds the job with the book's shared module for AI data, `lib.ai_data`, and the assistant learns only from that governed build: `pipelines/review_corpus.sql` stays as [chapter 4](#lineage-from-pipelines)'s ungoverned copy, so [CI](#run-every-rule-as-one-job) keeps checking its lineage. The module runs the pipeline's own SQL through the enforcement layer that [chapter 6](#enforcing-policies-on-queries) built, replaces the personal data left in the text, removes duplicates, and writes the corpus, its lineage and its dataset card. Its settings are in `governance/ai_data.toml`, reviewed like a policy, and so are the answers the code needs to legal questions, each a placeholder that the platform's legal team replaces with its answer and reason.

## Admit only reviews whose authors consented {#admit-consented-reviews}

On the platform, consent to one use doesn't cover another, because the question a learner answers names the use. Article 5(1)(b) of the EU's General Data Protection Regulation (GDPR) says that personal data shall be "collected for specified, explicit and legitimate purposes and not further processed in a manner that is incompatible with those purposes" ([GDPR](https://eur-lex.europa.eu/eli/reg/2016/679/oj)). The code reads the recorded answer to whether training an AI is covered from `[legal.consent]` in `governance/ai_data.toml`, and decides consent per record when the corpus is built, not from the account's sign-up and not once for the whole table.

`crm.learners.ai_training_consent` records whether a learner agreed to their text training the platform's AI. [Chapter 5](#narrow-by-attribute) made it an attribute the access policy reads, so a read of `learning.reviews` for the purpose `ai-training` keeps only the reviews whose author's answer is true, with contact details removed. `lib.ai_data.read` runs the `SELECT` of `pipelines/review_corpus.sql` through [the enforcement layer](#enforcing-policies-on-queries) as `analyst-1`, the data-science principal, stating the purpose. `lib.ai_data.build` refuses to run unless the layer filters on the column the consent answer names. Compare three builds: the plain pipeline, `pipelines/review_corpus.sql` run as written; the governed read; and a consent filter written by hand.

Example `corpus-consent-filter` (Python 3.12.15 · datasketch 2.0.0 · duckdb 2.0.0 · mlcroissant 1.1.1 · openlineage-python 1.55.0 · phonenumbers 9.0.43 · presidio-analyzer 2.2.366 · presidio-anonymizer 2.2.366 · pyarrow 26.0.1 · pyiceberg 0.12.0 · pyyaml 6.0.3 · regopy 1.5.2 · sqlglot 30.33.0 · tldextract 5.4.0), passing in Edition 2026.11.31:

```python
"""Counts the reviews that three builds of the corpus would hold, and the personal data the plain copy would keep. The prose's point rests on the "share by learners never asked" line."""
from collections import Counter

from lib import ai_data, pii
from lib.catalog import open_catalog
from lib.enforce import Enforcer

platform = Enforcer(open_catalog(["crm.learners", "learning.courses", "learning.reviews"]))
layer = platform.connection           # the enforcement layer's own access, for counting what each build would hold

# The plain pipeline copies every review, with its text as written.
plain = layer.sql("select count(*) from learning.reviews").fetchall()[0][0]
print("reviews the plain pipeline copies:", f"{plain:,}")

# The governed build: the same SQL, through the enforcement layer, as the job's principal, for its purpose.
# the job's own read: its principal and purpose come from the build settings
governed = ai_data.read(platform)
print(f"reviews the governed build reads, as {ai_data.BUILD['principal']} for {ai_data.BUILD['purpose']}:", f"{len(governed):,}")
# no purpose stated: the layer returns every review, with its text left out
without = ai_data.read(platform, purpose=None)
print("reviews read without the purpose:", f"{len(without):,}")
print("of them, with their text:", sum(r["body"] is not None for r in without))

# The filter that looks right: leave out the learners who said no.
# every review except those whose learner said no; a null answer passes, since only "= false" is excluded
opt_out = layer.sql("""
    select count(*), count(*) filter (where l.ai_training_consent is null)
    from learning.reviews r join crm.learners l using (learner_id)
    where r.learner_id not in (select learner_id from crm.learners where ai_training_consent = false)""").fetchall()[0]
print("reviews a filter on refusals keeps:", f"{opt_out[0]:,}")
print("of them, by learners never asked:", f"{opt_out[1]:,}")
print("share by learners never asked:", f"{opt_out[1] / opt_out[0]:.0%}")
# the signup dates of the learners whose consent is still unknown
never = layer.sql("""select min(signup_at)::date, max(signup_at)::date from crm.learners
                     where ai_training_consent is null""").fetchall()[0]
print("learners never asked signed up:", " to ".join(f"{day:%b} {day.day}, {day.year}" for day in never))

# Personal data the plain pipeline's corpus would hold, from the labelled truth, and how often each email is written.
raw = dict(layer.sql("select review_id, body from learning.reviews").fetchall())
for entity, (placed, _) in ai_data.left(raw).items():
    print(f"plain pipeline, labelled {entity} values:", f"{placed:,}")
# each labelled email's text, counted by how many spans write it
emails = Counter(pii.text_of(s) for s in pii.truth() if s.source == "reviews" and s.entity == "EMAIL_ADDRESS")
print("plain pipeline, distinct labelled EMAIL_ADDRESS values:", len(emails))
print("of them, written once:", sum(n == 1 for n in emails.values()))
```

Output:

```text
reviews the plain pipeline copies: 3,000
reviews the governed build reads, as analyst-1 for ai-training: 436
reviews read without the purpose: 3,000
of them, with their text: 0
reviews a filter on refusals keeps: 2,045
of them, by learners never asked: 1,609
share by learners never asked: 79%
learners never asked signed up: Jan 1, 2024 to May 31, 2025
plain pipeline, labelled EMAIL_ADDRESS values: 120
plain pipeline, labelled PERSON values: 798
plain pipeline, distinct labelled EMAIL_ADDRESS values: 113
of them, written once: 106
```

Copied as written, the plain pipeline would put 120 labelled emails and 798 labelled names into the corpus, from 3,000 reviews.

The governed read keeps 436 reviews, the ones whose authors said yes. Without a purpose, the same principal gets 3,000 reviews, 0 of them with their text, so a job that forgets to state its purpose gets nothing it could train on.

Notice that the filter written by hand looks like respect for consent, which is why it's the one to recognize: it leaves out the learners who said no. It keeps 2,045 reviews, and 1,609 of them, 79%, are by learners who were never asked. The README of the platform's generated data says consent is unknown for sign-ups before Jun 1, 2025, and every learner with no answer signed up from Jan 1, 2024 to May 31, 2025. An unknown answer isn't a yes. A corpus built with that filter would be mostly text that no consent covers, and nothing about it would look wrong: no error, a plausible row count, and a condition on the consent column that a reviewer would read as correct. The enforcement layer's filter tests `IS TRUE`, so an unknown answer counts as no, which [chapter 6](#build-row-filters-from-data) built for exactly this case.

On the platform, every review was written under one set of terms, so the card states one licence, read from `[legal.licence]`. A corpus that mixes sources, such as reviews and forum posts, needs a licence per record too, which is one reason [every record keeps its source table](#carry-provenance).

**Working rule:** Decide consent per record, when the corpus is built, from the answer your legal team says covers the use, and admit only records whose answer is yes: never filter out the refusals and keep the rest. Build the corpus through the enforcement layer with its purpose stated, and have the build check that the layer filters on that answer. Keep the legal team's answers, with their reasons, in a reviewed file the code reads, and carry licences per record when a source mixes them.

## Replace personal data in the text, then measure what's left {#replace-personal-data}

Consent decides whose text may enter the corpus. It doesn't make the text free of personal data: a review names its instructor, or gives its author's email for questions. Before such text trains or grounds a model, the corpus job detects the personal data in it, replaces each value, and measures what's left, because a detector's output lists what it found and never what it missed. [The truth](#measure-against-labelled-spans), chapter 3's labelled spans, marks every piece of personal data the generator placed in the reviews. `lib.ai_data.left` counts how many labelled values are still in the text after a step, matching by value, since every replacement moves the offsets after it.

The reviews arrive from the enforcement layer with its `redact_contacts` mask applied, which replaces emails and phone numbers with `[email]` and `[phone]` ([chapter 6](#write-masks-as-sql)). [Chapter 3](#tune-the-recognizers)'s detector, `lib.pii.platform_detector`, runs Presidio's pattern recognizers with no language model, and no pattern finds a name. So `lib.ai_data.detector` adds a Presidio deny list of the names the platform knows: every word of every instructor's name, which Presidio escapes and matches as a whole word, ignoring case unless told otherwise ([Presidio's pattern recognizer](https://github.com/data-privacy-stack/presidio/blob/2.2.364/presidio-analyzer/presidio_analyzer/pattern_recognizer.py)). The list comes from `crm.instructors`, which `analyst-1` may not read. So the platform team's own job, named in `[names]` in `governance/ai_data.toml`, builds it with the platform's own access, as the enforcement layer reads `crm.learners` for consent, and hands the corpus job the list, never the table. The card records which job built it and how many words it holds. `lib.ai_data.known_names` stands for that job. Each value found becomes a keyed pseudonym, which [the next section](#consistent-pseudonyms) builds. Measure each step on the reviews the corpus reads, then try a longer list, with every learner's name in it too:

Example `corpus-personal-data-left` (Python 3.12.15 · datasketch 2.0.0 · duckdb 2.0.0 · mlcroissant 1.1.1 · openlineage-python 1.55.0 · phonenumbers 9.0.43 · presidio-analyzer 2.2.366 · presidio-anonymizer 2.2.366 · pyarrow 26.0.1 · pyiceberg 0.12.0 · pyyaml 6.0.3 · regopy 1.5.2 · sqlglot 30.33.0 · tldextract 5.4.0), passing in Edition 2026.11.31:

```python
"""Counts the labelled names and emails still in the reviews after each step of the corpus job, and the false positives of the instructors' list and of a longer one. The prose's point rests on the "then the instructors' names, PERSON left" line and the false-positive lines."""
from lib import ai_data, pii
from lib.catalog import open_catalog
from lib.enforce import Enforcer

# The reviews the corpus job reads: consenting authors only, with emails and phone numbers already masked.
rows = ai_data.read(Enforcer(open_catalog(["crm.learners", "learning.courses", "learning.reviews"])))
texts = {row["review_id"]: row["body"] for row in rows}    # keyed by review ID, to match the labelled values row by row

# What's left of the labelled personal data after each step, matched by value against the generator's truth.
detectors = {"then the pattern recognizers": pii.platform_detector(), "then the instructors' names": ai_data.detector()}
# the texts as the enforcement layer leaves them, before any detector runs
stages = {"masked by the enforcement layer": texts}
# each detector runs alone on the masked texts; pseudonymize returns the text first
stages |= {name: {i: ai_data.pseudonymize(engine, text)[0] for i, text in texts.items()} for name, engine in detectors.items()}
for stage, done in stages.items():
    for entity, (placed, still) in ai_data.left(done).items():
        print(f"{stage}, {entity} left: {still} of {placed}")

# the instructors' words, from the governance team's list
names = ai_data.known_names()
print("words in the list of instructors' names:", len(names))
# short words are the ones ordinary text could contain
print("words in the list of 3 characters or fewer:", ", ".join(word for word in names if len(word) <= 3))
false_positives = sum(ai_data.unlabelled(ai_data.detector(), texts).values())
assert false_positives == 0
print("false positives, instructors' names:", false_positives)

# Every name the platform holds, learners' too, on the same reviews: in any case, then only as written.
# the union of the instructors' words and every learner's, which the table holds
longer = sorted(set(names) | set(ai_data.known_names("crm.learners")))
# matched in any case, as the platform's detector matches by default
everyone = ai_data.detector(longer)
wrong = ai_data.unlabelled(everyone, texts)
print("words in the list, with learners' names:", f"{len(longer):,}")
print("false positives, with learners' names:", sum(wrong.values()))
print("most common false positives:", ", ".join(f"{word} ({n})" for word, n in wrong.most_common(2)))
# review 24, which the longer list changes
print("a review after:", ai_data.pseudonymize(everyone, texts[24])[0])
# a word matches only as written, so a lowercase word no longer matches a capitalized name
cased = ai_data.detector(longer, match_case=True)
cased_wrong = sum(ai_data.unlabelled(cased, texts).values())
assert cased_wrong == 0
print("false positives, with learners' names, matching case:", cased_wrong)
# pseudonymize with the cased detector, then count what is left by value
left = ai_data.left({i: ai_data.pseudonymize(cased, text)[0] for i, text in texts.items()})
assert left["PERSON"][1] == 0
print("names left, with learners' names, matching case:", f"{left['PERSON'][1]} of {left['PERSON'][0]}")
```

Output:

```text
masked by the enforcement layer, EMAIL_ADDRESS left: 0 of 19
masked by the enforcement layer, PERSON left: 120 of 120
then the pattern recognizers, EMAIL_ADDRESS left: 0 of 19
then the pattern recognizers, PERSON left: 120 of 120
then the instructors' names, EMAIL_ADDRESS left: 0 of 19
then the instructors' names, PERSON left: 0 of 120
words in the list of instructors' names: 106
words in the list of 3 characters or fewer: Ray, Roy, Ulf, Uss, de, la
false positives, instructors' names: 0
words in the list, with learners' names: 4,316
false positives, with learners' names: 147
most common false positives: best (80), date (67)
a review after: Some videos are out of <PERSON_8e2b63a6>.
false positives, with learners' names, matching case: 0
names left, with learners' names, matching case: 0 of 120
```

The enforcement layer's mask leaves 0 of 19 emails and 120 of 120 names, and the pattern recognizers leave 120 of 120 names, since no pattern finds a name.

Only the instructors' list removes names. With its names added, the detector leaves 0 of 120 names and makes no false positives, although its list holds short words that ordinary text could contain: `Ray, Roy, Ulf, Uss, de, la`.

The longer list, with every learner's name in it too, makes 147 false positives. The most common, `best (80), date (67)`, are surnames that are also English words, and the list turns `Some videos are out of date.` into `Some videos are out of <PERSON_8e2b63a6>.` in the corpus.

Matching case avoids both. Matched only as written, the longer list makes no false positives and leaves no names, because every name in these reviews is capitalized. On real text it would miss the names typed in lowercase that [chapter 3 found in tickets](#where-patterns-fail). The platform keeps the instructors' list, matched in any case: instructors are the people reviews talk about, and the corpus job then holds their names instead of every learner's.

The list's result holds for this data only. The labelled spans are generated, and the generated reviews name only instructors, so they can't contain the case a list of instructors misses, such as a classmate's name. For names nobody listed, the detector to try is a named-entity model. [Chapter 3](#compare-a-named-entity-model) compared one, OpenAI's Privacy Filter, from a recording of its detections, scored against the labelled spans with `lib.pii.score`. Score it the same way, on the reviews the corpus reads:

Example `privacy-filter-on-the-corpus` (Python 3.12.15 · datasketch 2.0.0 · duckdb 2.0.0 · mlcroissant 1.1.1 · openlineage-python 1.55.0 · phonenumbers 9.0.43 · presidio-analyzer 2.2.366 · presidio-anonymizer 2.2.366 · pyarrow 26.0.1 · pyiceberg 0.12.0 · pyyaml 6.0.3 · regopy 1.5.2 · sqlglot 30.33.0 · tldextract 5.4.0), passing in Edition 2026.11.31:

```python
"""Scores OpenAI's Privacy Filter, from its recorded detections, against the labelled spans in the reviews the corpus reads. The prose's point rests on the "names the model finds, exactly" and "names the model misses" lines."""
from collections import Counter

from lib import ai_data, pii
from lib.catalog import open_catalog
from lib.enforce import Enforcer

# Privacy Filter's eight labels, mapped to the labelled spans' entity types. A label with no type matches nothing.
MATCHES = {"private_person": {"PERSON"}, "private_email": {"EMAIL_ADDRESS"}, "private_phone": {"PHONE_NUMBER"},
           "private_address": {"LOCATION"}, "account_number": {"CREDIT_CARD", "IBAN_CODE"},
           "private_date": {"DATE_TIME"}, "private_url": set(), "secret": set()}

# The reviews the corpus job reads, with the model's recorded detections and the labelled spans in them. The model read
# each review as written, before the enforcement layer's mask, which changes no name.
corpus = {row["review_id"] for row in ai_data.read(Enforcer(open_catalog(["crm.learners", "learning.courses",
                                                                          "learning.reviews"])))}
# the model's detections as recorded, not a live call to the model
model, manifest = pii.recorded("pii-entities")
found = [span for span in model if span.source == "reviews" and span.row_id in corpus]
labelled = [span for span in pii.truth() if span.source == "reviews" and span.row_id in corpus]
print("model:", manifest["model"])
print("reviews measured:", len(corpus))

# per entity type: [labelled, found exactly, found overlapping], then per detector label: [found, exact, overlapping]
scores = pii.score(found, labelled, MATCHES)
# the overlap count is unused here: the prose counts exact matches only
placed, exact, _ = scores.recall["PERSON"]
print("labelled names:", placed)
print("names the model finds, exactly:", exact)
# private_person is the model's label for a name, so only that label counts as a match
missed = pii.missed([span for span in labelled if span.entity == "PERSON"], found, {"private_person"})
print("names the model misses:", len(missed))
often = Counter(map(pii.text_of, missed)).most_common(3)
print("names it misses most often:", ", ".join(f"{name} ({n})" for name, n in often))

# What the model finds where no labelled span of the type its label maps to is: text it would replace for nothing.
wrong = sum(n - overlap for n, _, overlap in scores.precision.values())
print("the model's detections:", len(found))
print("of them, overlapping no labelled span of their type:", wrong)
```

Output:

```text
model: openai/privacy-filter
reviews measured: 436
labelled names: 120
names the model finds, exactly: 83
names the model misses: 37
names it misses most often: Marie (4), Liwia (4), Luigi (4)
the model's detections: 102
of them, overlapping no labelled span of their type: 0
```

On these reviews the model can't replace the list, and this data can't show what it adds, since every name in it is on the list. It finds 83 of the 120 labelled names exactly and misses 37, most often `Marie (4), Liwia (4), Luigi (4)`, where the list misses none.

Of its 102 detections, 0 overlap no labelled span of their type. Where text names people no list holds, run both, measured this way first.

The text that comes out can still identify people. A detector misses spans, the text can identify someone without a name, and each record keeps a review ID that leads back to its author through `learning.reviews`. The GDPR's Recital 26 says: "Personal data which have undergone pseudonymisation, which could be attributed to a natural person by the use of additional information should be considered to be information on an identifiable natural person" ([GDPR, Recital 26](https://eur-lex.europa.eu/eli/reg/2016/679/oj)). So we call the corpus pseudonymized, never anonymized.

**Working rule:** Detect personal data in corpus text after the enforcement layer's masks, add the identifiers your platform holds about the people the text is about, and measure what's left against labelled data, by value, after every change to the detector. Prefer a short list of the names your text is about to every name you hold, since a long list matches ordinary words. Have a job that may read the names build the list, and record which list each corpus used. Measure a named-entity model the same way before you rely on it for names no list holds. Keep the list out of the corpus, and call the result pseudonymized.

## Keep pseudonyms consistent with a keyed hash {#consistent-pseudonyms}

A detected name can be replaced by a placeholder, `<PERSON>`, or by a pseudonym: a token for one name, the same wherever that name appears. The platform's reviews name an instructor by first name only, and no two instructors share a first name, so here a token stands for one person. A pseudonym keeps who is who, so two reviews that praise the same instructor still say so.

Presidio's anonymizer has a `hash` operator, and since version 2.2.361 it salts each hash at random by default, for security ([Presidio anonymizer](https://presidio.dataprivacystack.org/anonymizer/)). A random salt makes each hash unlinkable to the next, and so useless as a pseudonym. To get the same hash for the same value, the docs say you "**must** explicitly provide a `salt` parameter", at least 128 bits (16 bytes) long.

The corpus job uses its own keyed hash instead, through Presidio's `custom` operator, which takes a lambda that must return a string. `lib.ai_data.pseudonym` computes an HMAC with SHA-256 of the entity type and the case-folded value, under a key only the corpus job holds, and keeps eight hexadecimal digits. HMAC is the construction RFC 2104 describes for keying a hash, and one of its stated goals is "To have a well understood cryptographic analysis of the strength of the authentication mechanism based on reasonable assumptions on the underlying hash function" ([RFC 2104](https://www.rfc-editor.org/rfc/rfc2104)). Python's standard library has it, as `hmac`, so the job has no reason to key a hash any other way.

Two of the corpus's reviews name the same instructor, and replacing that name both ways shows which way keeps the two linked:

Example `keyed-pseudonyms` (Python 3.12.15 · datasketch 2.0.0 · duckdb 2.0.0 · mlcroissant 1.1.1 · openlineage-python 1.55.0 · phonenumbers 9.0.43 · presidio-analyzer 2.2.366 · presidio-anonymizer 2.2.366 · pyarrow 26.0.1 · pyiceberg 0.12.0 · pyyaml 6.0.3 · regopy 1.5.2 · sqlglot 30.33.0 · tldextract 5.4.0), passing in Edition 2026.11.31:

```python
"""Replaces the same instructor's name in two reviews by Presidio's hash, by the corpus's keyed hash and by the enforcement layer's mask. The prose's point rests on the Presidio hash lines and the keyed pseudonym line."""
from collections import Counter

from presidio_anonymizer import AnonymizerEngine
from presidio_anonymizer.entities import InvalidParamError, OperatorConfig
from sqlglot import exp

from lib import ai_data, enforce
from lib.catalog import open_catalog
from lib.enforce import Enforcer
from lib.platform import load

# Two of the corpus's reviews that name the same instructor, with what the detector found in each.
layer = Enforcer(open_catalog(["crm.learners", "learning.courses", "learning.reviews"]))
engine = ai_data.detector()
found = [(row["body"], engine.analyze(row["body"], language="en")) for row in ai_data.read(layer)]
# the first two reviews whose only detected name is Patrick
named = [(text, spans) for text, spans in found if [text[s.start:s.end] for s in spans] == ["Patrick"]][:2]
print("reviews naming Patrick:", len(named))

def replaced(operator: OperatorConfig) -> list[str]:
    """The value that replaced the name in each of the two reviews."""
    results = [AnonymizerEngine().anonymize(text, spans, {"PERSON": operator}) for text, spans in named]
    return [result.text[item.start:item.end] for result in results for item in result.items]

# Presidio's hash salts each value at random, so the same name gets a different hash in each review
hashed = replaced(OperatorConfig("hash"))
assert hashed[0] != hashed[1]
print("Presidio's hash, the same name in two reviews gets one value:", hashed[0] == hashed[1])
# and hashing the same review again gives it a different hash too
rehashed = replaced(OperatorConfig("hash"))[0]
assert rehashed != hashed[0]
print("Presidio's hash, one review hashed twice gets one value:", rehashed == hashed[0])
# a fixed salt must be at least 16 bytes; this one is 10, so Presidio refuses it
try:
    replaced(OperatorConfig("hash", {"salt": "ten bytes!"}))
except InvalidParamError as error:
    print("a fixed salt of 10 bytes:", error)

# the custom operator's lambda returns the keyed pseudonym as the replacement
keyed = replaced(OperatorConfig("custom", {"lambda": lambda value: ai_data.pseudonym("PERSON", value)}))
print("keyed pseudonyms:", ", ".join(keyed))
print("the name typed in lowercase:", ai_data.pseudonym("PERSON", "patrick"))
# the first word of each instructor's full name
first_names = Counter(name.split()[0] for name in load("crm.instructors").column("full_name").to_pylist())
shared = sum(n for n in first_names.values() if n > 1)
assert shared == 0
print("instructors who share a first name with another:", shared)

# What the enforcement layer's pseudonymize mask makes of the same name, run by the layer under its own key.
mask = enforce.masked("pseudonymize", exp.Literal.string("Patrick"), "VARCHAR").sql(enforce.DIALECT)
print("the enforcement layer's pseudonymize mask, on the same name:", layer.run(f"select {mask}").fetchall()[0][0])
# functions whose name mentions HMAC: the prose says DuckDB has none
hmac = layer.connection.sql("select count(*) from duckdb_functions() where function_name ilike '%hmac%'").fetchall()[0][0]
assert hmac == 0
print("HMAC functions in DuckDB:", hmac)
```

Output:

```text
reviews naming Patrick: 2
Presidio's hash, the same name in two reviews gets one value: False
Presidio's hash, one review hashed twice gets one value: False
a fixed salt of 10 bytes: Salt must be at least 16 bytes (128 bits). Provided salt is 10 bytes.
keyed pseudonyms: <PERSON_d14171be>, <PERSON_d14171be>
the name typed in lowercase: <PERSON_d14171be>
instructors who share a first name with another: 0
the enforcement layer's pseudonymize mask, on the same name: 23b3d8671d01840655296f752121aecc4f7991694260e5e1431d2b57c5e3d53f
HMAC functions in DuckDB: 0
```

Presidio's `hash` gives the same name a different value in each of the two reviews, and gives one review a different value each time it's hashed. Presidio also refuses a fixed salt that's too short, here 10 bytes, with `Salt must be at least 16 bytes (128 bits). Provided salt is 10 bytes.`, as the docs say. The keyed hash gives both reviews `<PERSON_d14171be>, <PERSON_d14171be>`, and the name typed in lowercase gets `<PERSON_d14171be>` too.

The enforcement layer's `pseudonymize` mask, run by the layer on the same name, gives `23b3d8671d01840655296f752121aecc4f7991694260e5e1431d2b57c5e3d53f`. It's SHA-256 over the layer's key followed by the value, as [chapter 6](#write-masks-as-sql) writes it, not an HMAC, because the layer's masks are SQL and DuckDB 2.0.0 has no HMAC function. The two are different constructions under different keys, so the corpus's pseudonyms can't be joined with the ones the layer shows analysts.

Consistency has two costs. Whoever holds the key and a list of names can compute every pseudonym, as chapter 6 showed for emails, so the key lives in a secret manager, never in the corpus, the repository or a log. Keyed pseudonyms also keep apart reviews that placeholders would merge: 96 distinct texts against 21, which [the next section](#near-duplicates-with-minhash) measures. We recommend keyed pseudonyms where the text's meaning depends on telling people apart, as in a ticket that names an agent and a learner, and placeholders where nothing downstream needs to.

**Working rule:** Make pseudonyms with HMAC under a key only the job holds, and give each corpus or audience whose pseudonyms must not join a key of its own. Never use Presidio's `hash` operator without a salt for pseudonyms, since at its pinned version each hash is random. Pseudonymize a person's ID rather than a word of their name when two people can share one, and use a plain placeholder when nothing downstream needs to tell people apart.

## Find near-duplicates with MinHash {#near-duplicates-with-minhash}

Once its names are [pseudonymized](#consistent-pseudonyms), the corpus's next problem is repetition: templates, copies, a review pasted twice, the same sentence from many people. Repetition is a privacy problem as well as a quality one. Kandpal, Wallace and Raffel found that "a sequence that is present 10 times in the training data is on average generated ~1000 times more often than a sequence that is present only once" ([Kandpal et al., 2022](https://arxiv.org/abs/2202.06539)).

Near-copies need a measure of similarity. Broder defined the resemblance of two documents as the Jaccard similarity of their sets of shingles, the overlapping runs of characters or words in a text. He showed that sampling the smallest shingles under a random permutation gives an unbiased estimate of that resemblance ([Broder, 1997](https://doi.org/10.1109/SEQUEN.1997.666900)). Taking the smallest shingle under each of many permutations gives a MinHash signature, which estimates the similarity of two texts without comparing their sets ([Mining of Massive Datasets, section 3.3](http://infolab.stanford.edu/~ullman/mmds/ch3n.pdf)).

`lib.ai_data.deduplicate` removes exact copies first, by SHA-256 of the case-folded text with runs of white space made one. It compares what's left as sets of 5-character shingles with datasketch's `MinHash` and `MinHashLSH`, which returns [candidate pairs](#tune-lsh-then-confirm), and confirms each candidate by its exact Jaccard similarity. A confirmed pair at 0.8 or more is a near-duplicate, unless it differs only in a name, and of each group of duplicates the record with the lowest review ID stays. The duplicate step runs on the pseudonymized reviews, and each stage reports what it removed:

Example `corpus-duplicates` (Python 3.12.15 · datasketch 2.0.0 · duckdb 2.0.0 · mlcroissant 1.1.1 · openlineage-python 1.55.0 · phonenumbers 9.0.43 · presidio-analyzer 2.2.366 · presidio-anonymizer 2.2.366 · pyarrow 26.0.1 · pyiceberg 0.12.0 · pyyaml 6.0.3 · regopy 1.5.2 · sqlglot 30.33.0 · tldextract 5.4.0), passing in Edition 2026.11.31:

```python
"""Runs the corpus's duplicate step on the pseudonymized reviews: exact copies, then near-duplicates by MinHash LSH, each confirmed exactly, and the records kept. The prose's point rests on the "of them, differing only in a name" and "near-duplicates removed" lines."""
from collections import Counter

from lib import ai_data
from lib.catalog import open_catalog
from lib.enforce import Enforcer

# The reviews the corpus job reads, pseudonymized, then deduplicated: exact copies, then near-duplicates.
# the reviews read, each text pseudonymized; the second value counts the replacements
rows, _ = ai_data.read_pseudonymized(Enforcer(open_catalog(["crm.learners", "learning.courses", "learning.reviews"])))
# groups of reviews that stand for one record, and how many each step removed
groups, removed = ai_data.duplicate_groups(rows)
print("reviews read:", len(rows))
# one review per distinct normalized text: the one with the lowest review ID
print("distinct texts:", len(ai_data.exact_distinct(rows)))
for step, count in removed.items():
    print(f"{step}:", count)
print("records kept:", len(groups))
print("share of the reviews read that duplicate another:", f"{1 - len(groups) / len(rows):.0%}")

# identical texts only, before normalization
text, copies = Counter(row["body"] for row in rows).most_common(1)[0]
print("the most repeated text:", text)
print("its copies:", copies)

# A pair MinHash finds and the exact comparison confirms, and whether it differs only in a name.
distinct = [row["body"] for row in ai_data.exact_distinct(rows)]
# the first confirmed pair, by position among the distinct texts
i, j = sorted(ai_data.near_duplicates(distinct).confirmed)[0]
print("a near-duplicate pair:", distinct[i], "|", distinct[j])
print("its Jaccard similarity:", f"{ai_data.jaccard(distinct[i], distinct[j]):.2f}")
# equal once each pseudonym is a placeholder: the pair differs only in which names it holds
print("it differs only in a name:", ai_data.placeholders(distinct[i]) == ai_data.placeholders(distinct[j]))

# Each record kept stands for its group's reviews, under the review ID and course of the first.
# a group whose reviews come from more than one course
several = sum(len({row["course_id"] for row in group}) > 1 for group in groups)
print("records that stand for reviews of several courses:", several)
```

Output:

```text
reviews read: 436
distinct texts: 96
exact duplicates removed: 340
near-duplicate pairs from LSH: 381
near-duplicate pairs confirmed: 8
of them, differing only in a name: 8
near-duplicates removed: 0
records kept: 96
share of the reviews read that duplicate another: 78%
the most repeated text: Clear explanations and good exercises.
its copies: 56
a near-duplicate pair: Some videos are out of date. <PERSON_85783bac> explains things well. Happy to talk about it, write to me at [email]. | Some videos are out of date. <PERSON_967f9ae8> explains things well. Happy to talk about it, write to me at [email].
its Jaccard similarity: 0.81
it differs only in a name: True
records that stand for reviews of several courses: 23
```

Most of the corpus repeats itself. Of the 436 reviews read, 78% repeat another: 340 are exact copies, and only 96 texts are distinct. That's mostly the generator's doing, since it builds reviews from a handful of fixed sentences, so measure the share on your own text before training on it.

The most repeated text, "Clear explanations and good exercises.", alone appears 56 times.

MinHash then confirms 8 pairs at 0.8 or more, such as two reviews at a similarity of 0.81 that differ only in their instructor's pseudonym. Of those pairs, 8 differ only in a name, which the platform keeps apart, so near-duplicates remove 0.

Why pairs that differ only in a name stay apart comes down to what one threshold means at different lengths. Compare the corpus's reviews with the names as written, with keyed pseudonyms and with plain placeholders, then split every pair of distinct texts by whether it differs only in a name:

Example `names-and-near-duplicates` (Python 3.12.15 · datasketch 2.0.0 · duckdb 2.0.0 · mlcroissant 1.1.1 · openlineage-python 1.55.0 · phonenumbers 9.0.43 · presidio-analyzer 2.2.366 · presidio-anonymizer 2.2.366 · pyarrow 26.0.1 · pyiceberg 0.12.0 · pyyaml 6.0.3 · regopy 1.5.2 · sqlglot 30.33.0 · tldextract 5.4.0), passing in Edition 2026.11.31:

```python
"""Counts near-duplicate pairs of the corpus's reviews with names as written, with keyed pseudonyms and with placeholders, then splits the pairs that differ only in a name by length. The prose's point rests on the pair counts and the length ranges."""
from itertools import combinations

from lib import ai_data
from lib.catalog import open_catalog
from lib.enforce import Enforcer

# The reviews the corpus job reads, three ways: names as written, keyed pseudonyms, and plain placeholders.
layer = Enforcer(open_catalog(["crm.learners", "learning.courses", "learning.reviews"]))
keyed, _ = ai_data.read_pseudonymized(layer)
# the same reviews with each pseudonym replaced by a plain placeholder, as the third way
ways = {"names as written": ai_data.read(layer), "keyed pseudonyms": keyed,
        "placeholders": [row | {"body": ai_data.placeholders(row["body"])} for row in keyed]}
# the similarity at which a pair is a near-duplicate
threshold = ai_data.DUPLICATES["threshold"]
for way, rows in ways.items():
    texts = [row["body"] for row in ai_data.exact_distinct(rows)]
    print(f"{way}, distinct texts:", len(texts))
    print(f"{way}, pairs at {threshold} or more:", len(ai_data.near_duplicates(texts).confirmed))

# Every pair of distinct pseudonymized texts, split by whether the pair differs only in a name.
# distinct texts only, so each text appears once in the pairs
texts = [row["body"] for row in ai_data.exact_distinct(keyed)]
name_only, other = [], []
for a, b in combinations(texts, 2):
    # similarity comes first, so min and max over these tuples order the pairs by it
    pair = (ai_data.jaccard(a, b), len(a), a, b)
    (name_only if ai_data.placeholders(a) == ai_data.placeholders(b) else other).append(pair)
print("pairs that differ only in a name:", len(name_only))
# the lowest and highest similarity among the name-only pairs
print("their Jaccard similarity:", f"{min(name_only)[0]:.2f} to {max(name_only)[0]:.2f}")
for label, lengths in [(f"below {threshold}", [n for s, n, _, _ in name_only if s < threshold]),
                       (f"at {threshold} or more", [n for s, n, _, _ in name_only if s >= threshold])]:
    print(f"their length in characters, {label}:", f"{min(lengths)} to {max(lengths)}")
# the most similar pair among those that differ in more than a name
similarity, _, a, b = max(other)
print("the most similar pair that differs in more than a name:", f"{similarity:.2f}")
print("that pair:", a, "|", b)
```

Output:

```text
names as written, distinct texts: 96
names as written, pairs at 0.8 or more: 10
keyed pseudonyms, distinct texts: 96
keyed pseudonyms, pairs at 0.8 or more: 8
placeholders, distinct texts: 21
placeholders, pairs at 0.8 or more: 0
pairs that differ only in a name: 459
their Jaccard similarity: 0.68 to 0.83
their length in characters, below 0.8: 66 to 78
their length in characters, at 0.8 or more: 116 to 119
the most similar pair that differs in more than a name: 0.66
that pair: Clear explanations and good exercises. Happy to talk about it, write to me at [email]. | Clear explanations and good exercises. <PERSON_3026fb67> explains things well. Happy to talk about it, write to me at [email].
```

Which pairs reach 0.8 depends on what stands in for each name. With the names as written, 10 pairs reach 0.8, and with keyed pseudonyms 8, since a pseudonym is longer than most first names and so a larger share of a short review. With plain placeholders, the 21 distinct texts hold 0.

Among the pseudonymized texts, 459 pairs differ only in which pseudonym they hold, at similarities from 0.68 to 0.83, and the threshold splits them by length alone: those below 0.8 are 66 to 78 characters long, and those at 0.8 or more are 116 to 119 characters long.

So the task, not the threshold, decides whether such reviews are duplicates: for a model that learns how learners describe courses, they are, and for one that learns what learners say about each instructor, they aren't. The platform's assistant answers about each course and its instructor, so `name_only_pairs_are_duplicates = false` in `governance/ai_data.toml`, and `deduplicate` never merges two texts that are the same once their pseudonyms are placeholders. The threshold decides the other pairs. The most similar of those, at 0.66, is a review with and without its sentence about the instructor: two different reviews. The platform keeps 0.8, above that pair with a margin.

Deduplicating on text folds reviews of different courses into one record, under the review ID and course of its first review, so the corpus keeps 96 records, and 23 of them stand for reviews of several courses. For training that's the point, since the model learns from the text. For grounding an answer about one course it costs recall, which [the provenance section](#carry-provenance) takes up.

**Working rule:** Deduplicate training text in two steps, exact copies by a hash of the normalized text and then near-duplicates by MinHash, each confirmed by its exact similarity, and report how much each step removed. Pseudonymize first, so that no name reaches the comparison or the sketches and the comparison can tell which pairs differ only in a name. Keep those pairs apart when the text's meaning depends on whom it names, and compare on placeholders when it doesn't. Choose the threshold by reading the pairs close above and below it in your own text.

## Tune LSH for recall, then confirm every pair {#tune-lsh-then-confirm}

[MinHash signatures](#near-duplicates-with-minhash) make each comparison cheap; locality-sensitive hashing (LSH) avoids most comparisons. LSH cuts each signature into b bands of r rows, and two texts become a candidate pair when all the rows of any one band match. Leskovec, Rajaraman and Ullman derive the chance that a pair with Jaccard similarity s becomes a candidate pair ([Mining of Massive Datasets, chapter 3](http://infolab.stanford.edu/~ullman/mmds/ch3n.pdf)):

P(candidate) = 1 − (1 − sʳ)ᵇ

It's an S-shaped curve: a pair well below the threshold is rarely a candidate, a pair well above it nearly always, and near the threshold it's close to a coin toss. The curve's approximate threshold is (1/b)^(1/r). datasketch's documentation warns of both kinds of error: "There will be false positives - sets that do not satisfy your threshold but returned, and false negatives - qualifying sets that are not returned" ([datasketch, MinHash LSH](https://ekzhu.github.io/datasketch/lsh.html)).

With 96 distinct texts, the platform can afford to compare every pair exactly, 4,560 pairs, and that gives the exact pairs to measure LSH against. LSH sees only similarity, so it must return every pair at 0.8, whatever the name rule then decides. Score LSH against the exact comparison, tuned for 0.8 and tuned lower, each under datasketch's default hashing and under the hashing it used before its 2.0.0 release:

Example `lsh-misses-near-duplicates` (Python 3.12.15 · datasketch 2.0.0 · duckdb 2.0.0 · mlcroissant 1.1.1 · openlineage-python 1.55.0 · phonenumbers 9.0.43 · presidio-analyzer 2.2.366 · presidio-anonymizer 2.2.366 · pyarrow 26.0.1 · pyiceberg 0.12.0 · pyyaml 6.0.3 · regopy 1.5.2 · sqlglot 30.33.0 · tldextract 5.4.0), passing in Edition 2026.11.31:

```python
"""Measures the pairs LSH misses against every pair compared exactly, under two tunings and two hashing schemes, and shows what datasketch does with sketches of two schemes. The prose's point rests on the "missed" lines."""
from itertools import combinations

from datasketch import MinHashLSH

from lib import ai_data
from lib.catalog import open_catalog
from lib.enforce import Enforcer

rows, _ = ai_data.read_pseudonymized(Enforcer(open_catalog(["crm.learners", "learning.courses", "learning.reviews"])))
texts = [row["body"] for row in ai_data.exact_distinct(rows)]
# the similarity at which a pair is a near-duplicate
threshold = ai_data.DUPLICATES["threshold"]

# Every pair compared exactly: possible at this size, and the truth LSH is measured against.
pairs = {(i, j) for i, j in combinations(range(len(texts)), 2) if ai_data.jaccard(texts[i], texts[j]) >= threshold}
print("pairs compared exactly:", f"{len(texts) * (len(texts) - 1) // 2:,}")
print(f"pairs at a Jaccard similarity of {threshold} or more:", len(pairs))

found = {}
for scheme in ["affine32", "legacy"]:
    # LSH tuned for the threshold, and for the lower candidate threshold
    for tuned_for in [threshold, ai_data.DUPLICATES["candidate_threshold"]]:
        # the bands and rows LSH chose, its candidates, and the candidates the exact check confirms
        found[scheme, tuned_for] = result = ai_data.near_duplicates(texts, threshold, tuned_for, scheme)
        setting = f"{scheme}, LSH tuned for {tuned_for}"
        print(f"{setting}, bands:", result.bands)
        print(f"{setting}, rows per band:", result.rows)
        print(f"{setting}, candidate pairs:", len(result.candidates))
        print(f"{setting}, confirmed:", len(result.confirmed))
        # pairs the exact comparison found that LSH never proposed as candidates
        missed = len(pairs - result.candidates)
        print(f"{setting}, missed:", missed)
        if tuned_for != threshold:
            assert missed == 0  # tuned lower, LSH finds every pair under either scheme

# The chance that a pair of similarity s becomes a candidate, with b bands of r rows: 1 - (1 - s^r)^b.
# the bands and rows LSH chose for the threshold, as the formula uses them
b, r = found["affine32", threshold].bands, found["affine32", threshold].rows
for s in [0.8, 0.9]:
    print(f"chance of a candidate at similarity {s}, LSH tuned for {threshold}:", f"{1 - (1 - s ** r) ** b:.2f}")
# the S-curve's approximate threshold, which the prose compares with the threshold asked for
print("the approximate threshold, (1/b)^(1/r):", f"{(1 / b) ** (1 / r):.2f}")

# What datasketch does with a sketch made under the other scheme.
# an index takes sketches of one scheme, so the legacy sketch below is refused
index = MinHashLSH(threshold=threshold, num_perm=ai_data.DUPLICATES["num_perm"])
index.insert(0, ai_data.minhash(texts[0], "affine32"))
try:
    index.insert(1, ai_data.minhash(texts[1], "legacy"))
except ValueError as error:
    print("a legacy sketch in an index of affine32 sketches:", error)
```

Output:

```text
pairs compared exactly: 4,560
pairs at a Jaccard similarity of 0.8 or more: 8
affine32, LSH tuned for 0.8, bands: 9
affine32, LSH tuned for 0.8, rows per band: 13
affine32, LSH tuned for 0.8, candidate pairs: 36
affine32, LSH tuned for 0.8, confirmed: 1
affine32, LSH tuned for 0.8, missed: 7
affine32, LSH tuned for 0.6, bands: 18
affine32, LSH tuned for 0.6, rows per band: 7
affine32, LSH tuned for 0.6, candidate pairs: 381
affine32, LSH tuned for 0.6, confirmed: 8
affine32, LSH tuned for 0.6, missed: 0
legacy, LSH tuned for 0.8, bands: 9
legacy, LSH tuned for 0.8, rows per band: 13
legacy, LSH tuned for 0.8, candidate pairs: 52
legacy, LSH tuned for 0.8, confirmed: 6
legacy, LSH tuned for 0.8, missed: 2
legacy, LSH tuned for 0.6, bands: 18
legacy, LSH tuned for 0.6, rows per band: 7
legacy, LSH tuned for 0.6, candidate pairs: 406
legacy, LSH tuned for 0.6, confirmed: 8
legacy, LSH tuned for 0.6, missed: 0
chance of a candidate at similarity 0.8, LSH tuned for 0.8: 0.40
chance of a candidate at similarity 0.9, LSH tuned for 0.8: 0.93
the approximate threshold, (1/b)^(1/r): 0.84
a legacy sketch in an index of affine32 sketches: MinHash scheme 'legacy' does not match scheme 'affine32' of the MinHash previously given to this index
```

Tuned for 0.8, datasketch's default LSH returns 36 candidate pairs, misses 7 of the 8 pairs that reach 0.8, and so confirms only 1 of them.

The formula says why. With the 9 bands of 13 rows datasketch chose, a pair at similarity 0.8 becomes a candidate with a probability of 0.40, one at 0.9 with 0.93, and the approximate threshold is 0.84, above the threshold asked for.

Under the default scheme, tuned for 0.6, LSH finds every pair and returns 381 candidates. Each candidate costs one exact comparison, against 4,560 for comparing every pair, and the exact comparison discards every pair below 0.8. So the platform tunes LSH for 0.6 and confirms at 0.8, both settings in `governance/ai_data.toml`.

datasketch's 2.0.0 release made `affine32` the default permutation scheme of MinHash, and [its release notes](https://github.com/ekzhu/datasketch/releases/tag/v2.0.0) say that hash values differ from earlier versions, so they advise rebuilding persisted sketches and LSH indexes, or passing `MinHash(..., scheme="legacy")`. With the legacy scheme, LSH tuned for 0.8 returns 52 candidates and confirms 6, so a count recorded under one scheme isn't reproduced under the other.

Tuned for 0.6, both schemes find all 8 pairs. With high recall and an exact check, the result depends on the similarity, not on the hash functions.

datasketch refuses to mix schemes in one index, with `MinHash scheme 'legacy' does not match scheme 'affine32' of the MinHash previously given to this index`, but it won't stop you comparing old counts with new ones. The book's examples pin datasketch at 2.0.0 and write `scheme = "affine32"` into the settings, so an upgrade can't change a recorded count without a reviewed change.

**Working rule:** Tune LSH below your similarity threshold, so that it misses few pairs, and confirm every candidate by its exact similarity. Measure LSH's misses against an exhaustive comparison on a sample of your corpus whenever its settings, its library or the text change. Pin the MinHash library, write its hashing scheme into your settings, and never compare counts or stored sketches made under different schemes.

## Carry provenance on every record {#carry-provenance}

A record that can't be traced to its source can't be checked, corrected or removed. Provenance at the level of the record answers three questions: which table it came from, which row, and which run of which job put it in the corpus. [Chapter 4](#send-run-events) gives the platform the last part: every run of a job sends OpenLineage events, with a run ID derived from the run's time and the job's name, and column lineage parsed from the job's SQL.

`lib.ai_data.build` writes every record with its provenance beside the review's own columns: `source_table`, `review_id`, `job` and `run_id`. It sends the job's run events with `lib.lineage.record`. One choice there matters: the lineage is parsed from the query as the enforcement layer ran it, with the consent filter and the mask in it, not from the SQL in `pipelines/review_corpus.sql`. Build the corpus, then follow one record back to its run:

Example `corpus-provenance` (Python 3.12.15 · datasketch 2.0.0 · duckdb 2.0.0 · mlcroissant 1.1.1 · openlineage-python 1.55.0 · phonenumbers 9.0.43 · presidio-analyzer 2.2.366 · presidio-anonymizer 2.2.366 · pyarrow 26.0.1 · pyiceberg 0.12.0 · pyyaml 6.0.3 · regopy 1.5.2 · sqlglot 30.33.0 · tldextract 5.4.0), passing in Edition 2026.11.31:

```python
"""Builds the corpus and follows one record back to its source row, its job and the run that wrote it, then checks that run's lineage against the query as written. The prose's point rests on the "rows filtered by" and "body" lines."""
from lib import ai_data, lineage, pipelines
from lib.catalog import open_catalog

catalog = open_catalog(["crm.learners", "learning.courses", "learning.reviews"])
built = ai_data.build(catalog)
# the second record, whose text holds a pseudonym
record = built.records.to_pylist()[1]
print("a record's text:", record["body"])
for column in ["source_table", "review_id", "job"]:
    print(f"its {column}:", record[column])

# The run that wrote it, from the lineage events: the query as the enforcement layer ran it.
# the job's COMPLETE event, which carries the run's lineage
complete = [e for e in lineage.read_events(built.folder / "events.jsonl") if e["eventType"] == "COMPLETE"][0]
same_run = record["run_id"] == complete["run"]["runId"]
assert same_run
print("its run_id is the COMPLETE event's run:", same_run)
print("the run's inputs:", ", ".join(dataset["name"] for dataset in complete["inputs"]))
# column lineage: the inputs that filter the rows, and how each field is transformed
facet = complete["outputs"][0]["facets"]["columnLineage"]
# a FILTER transformation marks an input whose values decide which rows the output holds
filters = [f"{i['name']}.{i['field']}" for i in facet["dataset"] if any(t["subtype"] == "FILTER" for t in i["transformations"])]
print("rows filtered by, as run:", ", ".join(filters))
# the lineage of the review text, from the reviews table
text = [i for i in facet["fields"]["body"]["inputFields"] if (i["name"], i["field"]) == ("learning.reviews", "body")]
# IDENTITY means copied unchanged; TRANSFORMATION means the value was changed on its way
print("body, as run:", text[0]["transformations"][0]["subtype"])
# the same lineage parsed from the SQL as written, with no enforcement rewrite
fields, rows = lineage.column_lineage(pipelines.sql(ai_data.BUILD["job"]), lineage.schema(catalog))
print("rows filtered by, as written:", ", ".join(i.source for i in rows if i.subtype == "FILTER") or "nothing")
print("body, as written:", fields["body"][0].subtype)

# Back from every record to its review's author, with the platform's own access, against today's answers.
# each record's review ID, checked against the author's consent today, not at build time
ids = built.records.column("review_id").to_pylist()
print("records whose author consented:", f"{ai_data.consenting(catalog, ids)} of {len(ids)}")
```

Output:

```text
a record's text: Too fast in the second half. <PERSON_85783bac> explains things well.
its source_table: learning.reviews
its review_id: 18
its job: platform/review_corpus
its run_id is the COMPLETE event's run: True
the run's inputs: crm.learners, learning.courses, learning.reviews
rows filtered by, as run: crm.learners.ai_training_consent, learning.reviews.learner_id
body, as run: TRANSFORMATION
rows filtered by, as written: nothing
body, as written: IDENTITY
records whose author consented: 96 of 96
```

Following a record back shows what the lineage has to record. The record "Too fast in the second half. <PERSON_85783bac> explains things well." came from `learning.reviews`, review 18, written by the job `platform/review_corpus`, and its `run_id` is the run of the job's `COMPLETE` event.

That run read `crm.learners, learning.courses, learning.reviews`, and its lineage says the rows were filtered by `crm.learners.ai_training_consent, learning.reviews.learner_id`, the [`INDIRECT` inputs](#add-indirect-inputs) that decide which rows exist, and that `body` is a `TRANSFORMATION` of the review's text.

Parsed from the pipeline's SQL as written, the same job's rows are filtered by nothing, and `body` is an `IDENTITY` copy. That would say personal text was copied unchanged, with no consent column involved. The query as run is what happened, so it's what the lineage records.

Provenance makes the corpus checkable after the fact. Joining every record back through `learning.reviews` to its author finds 96 of 96 records whose author consented, a check the platform can run on any old build against today's answers.

Provenance also lets retrieval respect each reader's permissions. OWASP asks applications built on language models to "Implement fine-grained access controls and permission-aware vector and embedding stores" ([OWASP LLM08:2025](https://genai.owasp.org/llmrisk/llm082025-vector-and-embedding-weaknesses/)). The platform answers one person from an index built from this corpus, which keeps each record's `review_id` with its vectors. Before a retrieved record reaches the model, the retrieval layer asks the enforcement layer, as the person being answered, which of those reviews they may read, the way [chapter 6](#query-on-someones-behalf) runs an assistant's queries, and drops the rest. An instructor then gets only records from their own courses. A record that stands for reviews of several courses carries the first review's ID, so the check fails closed: for each of the 23 such records, it may hide a text the person could have read, and never shows one they couldn't.

**Working rule:** Give every record its source table, its row's ID, and the job and run that wrote it, and record the job's lineage from the query as it ran, filters and masks included. Check old builds against today's consent through that provenance. Filter every retrieval by the reader's own decision on each record's source row before a model sees it.

## Generate the dataset card from the run {#generate-the-dataset-card}

A dataset card tells whoever trains or audits a model what's in its data, where it came from and what was done to it. Croissant is MLCommons' metadata format for machine-learning datasets ([Akhtar et al., 2024](https://arxiv.org/abs/2403.19546)). Its 1.1 specification makes a card checkable: "Croissant datasets must declare that they conform to the versioned schema, e.g. http://mlcommons.org/croissant/1.1", and every dataset must give a description, licence, name, URL, creator and publication date. For provenance, the specification recommends the W3C PROV Ontology (PROV-O) ([Croissant 1.1](https://docs.mlcommons.org/croissant/docs/croissant-spec-1.1.html)). Its responsible-AI vocabulary adds properties such as `rai:dataCollection` and `rai:personalSensitiveInformation` ([Croissant RAI 1.0](https://docs.mlcommons.org/croissant/docs/croissant-rai-spec.html)). A card written by hand describes the data someone meant to build; a card generated by the run describes the data it built.

`lib.ai_data.card` generates the platform's card with mlcroissant 1.1.1, MLCommons' Python library for Croissant. The run supplies the counts, the corpus file's SHA-256, a field for every column, and three PROV-O properties: the tables the pipeline's own SQL reads, which the values come from, as `prov:wasDerivedFrom`; the table the enforcement layer's rewrite adds, which only decides which rows exist, as `prov:wasInfluencedBy`; and the run, as `prov:wasGeneratedBy`. mlcroissant writes them through its `extra_properties` argument but adds no `prov` prefix to the card's context, so the module adds it. The creator is the owner of `ai.review_corpus` in `governance/metadata.toml`. The licence, the name, URL, description and use cases come from `governance/ai_data.toml`, and `lib.ai_data.card_problems` is the platform's own check of the result. Generate the card, first with mlcroissant's default `conformsTo`:

Example `corpus-dataset-card` (Python 3.12.15 · datasketch 2.0.0 · duckdb 2.0.0 · mlcroissant 1.1.1 · openlineage-python 1.55.0 · phonenumbers 9.0.43 · presidio-analyzer 2.2.366 · presidio-anonymizer 2.2.366 · pyarrow 26.0.1 · pyiceberg 0.12.0 · pyyaml 6.0.3 · regopy 1.5.2 · sqlglot 30.33.0 · tldextract 5.4.0), passing in Edition 2026.11.31:

```python
"""Generates the corpus's Croissant card twice, first with mlcroissant's default conformsTo and then as the platform writes it, and runs the platform's own check on it. The prose's point rests on the "conformsTo" lines and the "problems" line."""
from lib import ai_data
from lib.catalog import open_catalog

built = ai_data.build(open_catalog(["crm.learners", "learning.courses", "learning.reviews"]))

# mlcroissant's default, then the card the pipeline writes, with Croissant 1.1 written out.
# an empty list leaves conformsTo to mlcroissant, which writes its own default
default, _ = ai_data.card(built, conforms_to=[])
print("conformsTo, mlcroissant's default:", default["conformsTo"])
# the card as the platform writes it: Croissant 1.1 and the RAI vocabulary, written out
card, warnings = ai_data.card(built)
ai_data.write_card(card, built.folder)
print("conformsTo, written out:", ", ".join(card["conformsTo"]))
print("datePublished, the run's:", card["datePublished"])
# the build's date as the major number: a nightly build may remove records
print("version, the run's:", card["version"])
print("the sources the card names:", ", ".join(source["@id"] for source in card["prov:wasDerivedFrom"]))
print("the tables that decided which rows, as the card names them:",
      ", ".join(table["@id"] for table in card["prov:wasInfluencedBy"]))
print("the activity the card names:", card["prov:wasGeneratedBy"]["prov:label"])
# the card names every column of the corpus as a field
print("fields in the record set:", ", ".join(f["name"] for f in card["recordSet"][0]["field"]))
for line in card["rai:dataPreprocessingProtocol"] + card["rai:personalSensitiveInformation"]:
    print("-", line)

# What mlcroissant reports, and what the platform's own check of the card reports.
# the message after mlcroissant's severity tag
print("mlcroissant's warnings:", len(warnings))
for warning in warnings:
    print("- mlcroissant:", warning.split("] ", 1)[1])
# the platform's own check: the library's validator doesn't report every requirement
problems = ai_data.card_problems(card, built.folder)
print("problems the book's check finds:", len(problems))
for problem in problems:
    print("-", problem)
```

Output:

```text
conformsTo, mlcroissant's default: http://mlcommons.org/croissant/1.0
conformsTo, written out: http://mlcommons.org/croissant/1.1, http://mlcommons.org/croissant/RAI/1.0
datePublished, the run's: 2026-10-01T02:00:00Z
version, the run's: 20261001.0.0
the sources the card names: iceberg://platform/learning.courses, iceberg://platform/learning.reviews
the tables that decided which rows, as the card names them: iceberg://platform/crm.learners
the activity the card names: platform/review_corpus
fields in the record set: review_id, course_id, course_title, rating, body, source_table, job, run_id
- Read 436 of 3,000 reviews, filtered on crm.learners.ai_training_consent, learning.reviews.learner_id.
- Replaced personal data with keyed pseudonyms (PERSON 120).
- Removed 340 exact duplicates and 0 near-duplicates (MinHash LSH, Jaccard similarity of 0.8 or more, confirmed exactly; 8 pairs that differ only in a name kept apart).
- Kept 96 records, each with its source table, review ID, job and run.
- Values replaced by keyed pseudonyms: PERSON 120.
- Emails and phone numbers replaced by [email] and [phone] by the enforcement layer's redact_contacts mask.
- Names found with a list of 106 words from crm.instructors, built by the governance team's job governance/instructor_names; the list isn't part of the corpus.
- Measured against labelled spans: 0 of 139 labelled values left in the text.
mlcroissant's warnings: 1
- mlcroissant: Property "http://mlcommons.org/croissant/citeAs" is recommended, but does not exist.
problems the book's check finds: 1
- creator: missing
```

Left to its default, mlcroissant writes `http://mlcommons.org/croissant/1.0`, although its own `LATEST_VERSION` is 1.1 ([mlcroissant source](https://github.com/mlcommons/croissant/blob/v1.1.1/python/mlcroissant/mlcroissant/_src/core/context.py)), so a card would declare a version it wasn't written for. The platform writes `http://mlcommons.org/croissant/1.1, http://mlcommons.org/croissant/RAI/1.0`.

The card's `datePublished`, `2026-10-01T02:00:00Z`, is the build's time, because the platform publishes a card only with the build it describes. Its `version`, `20261001.0.0`, takes the build's date as its major number. Croissant recommends MAJOR.MINOR.PATCH, where a new major version means "the existing data has been changed (edited, removed or shuffled records across splits)", and any nightly build may remove records.

The card names the sources `iceberg://platform/learning.courses, iceberg://platform/learning.reviews`, the influence `iceberg://platform/crm.learners` and the activity `platform/review_corpus`, and lists preprocessing steps with the run's counts.

The warnings mlcroissant reports, 1 in all, are about recommended properties, here `citeAs`, not about the problem the platform's check finds: `creator` is missing. `ai.review_corpus` has no owner in the metadata file, the gap [chapter 2](#test-the-catalog-against-the-metadata) found, so the card stays unpublished until the table has an owner. [Chapter 10](#run-every-rule-as-one-job) reports the same gap on every run, as a finding already on main.

mlcroissant can also read a card back, with `mlc.Dataset`, but at 1.1.1, with rdflib 7.6.0 and pandas 3.1.1, reading emits a `DeprecationWarning` from rdflib and a `Pandas4Warning`, and returns text fields as bytes. So the book's examples only write cards, and check them with the platform's own code.

**Working rule:** Generate the card in the run that builds the data, with every count, hash, source and version taken from that run, and write the format's version out. Check the card with your own code for every property the specification requires, since the library's validator doesn't report them all, and don't publish or train on a corpus whose card fails.

## Quote what the EU AI Act says about training data {#what-the-ai-act-says}

The EU's Artificial Intelligence Act, Regulation (EU) 2024/1689, addresses training data in several places. This section quotes two of them, Article 10 and Article 53(1)(d), and we interpret neither.

For high-risk AI systems, Article 10(2) says that training, validation and testing data sets "shall be subject to data governance and management practices", and lists what those practices concern. They include data collection processes and the origin of data, with the original purpose of collecting personal data, and data preparation such as annotation, labelling, cleaning, updating, enrichment and aggregation ([AI Act](https://eur-lex.europa.eu/eli/reg/2024/1689/oj)).

Article 10(3) asks for data sets that are "relevant, sufficiently representative, and to the best extent possible, free of errors and complete in view of the intended purpose" ([AI Act](https://eur-lex.europa.eu/eli/reg/2024/1689/oj)).

Regulation (EU) 2026/1744, the Digital Omnibus on AI, published Jul 24, 2026, leaves these passages as quoted. It replaces paragraphs 1 and 6 of Article 10, deletes paragraph 5, and inserts an Article 4a, "Processing of special categories of personal data for bias detection and correction". It also sets when Chapter III, Sections 1 to 3, which include Article 10, apply: from 2 December 2027 to high-risk AI systems classified under Article 6(2) and Annex III, and from 2 August 2028 to those classified under Article 6(1) and Annex I ([Regulation (EU) 2026/1744](https://eur-lex.europa.eu/eli/reg/2026/1744/oj)).

For providers of general-purpose AI models, Article 53(1)(d) requires them to "draw up and make publicly available a sufficiently detailed summary about the content used for training of the general-purpose AI model, according to a template provided by the AI Office." The European Commission published that template as C(2025) 8311 ([Commission template](https://digital-strategy.ec.europa.eu/en/library/explanatory-notice-and-template-public-summary-training-content-general-purpose-ai-models), [PDF](https://ec.europa.eu/newsroom/dae/redirection/document/118480)).

Its [explanatory notice](https://digital-strategy.ec.europa.eu/en/library/explanatory-notice-and-template-public-summary-training-content-general-purpose-ai-models) says that a downstream entity that modifies a model already placed on the Union market can become the provider of the resulting model, under the Commission guidelines on general-purpose AI models that the notice names. In that case the modifying entity reports only the training content used for the modification.
The Act's Annex III is titled "High-risk AI systems referred to in Article 6(2)", and its point 3, "Education and vocational training", includes "(b) AI systems intended to be used to evaluate learning outcomes, including when those outcomes are used to steer the learning process of natural persons in educational and vocational training institutions at all levels" ([AI Act](https://eur-lex.europa.eu/eli/reg/2024/1689/oj)). We quote it because the platform is an education platform, and stop there. Whether any system of the platform's is high-risk under Article 6 is another question, and none of this chapter's code depends on the answer.

**Working rule:** Quote the Act as amended, from the Official Journal, with a link and the date you read it, because the Omnibus moved both text and dates. Keep the law out of your pipeline's logic: generate the card's facts from the run, and let the people who answer legal questions decide which passages apply.

## What goes wrong in production {#ai-data-in-production}

A corpus gives no warning. The model trains, the card is published, and the problems surface later, in what the model says or in a request nobody can answer.

### A learner withdraws after the build {#withdrawn-after-the-build}

Like marketing's list in [chapter 7](#consent-at-query-time), a corpus records who had consented on the day it was built. Take the corpus that `lib.ai_data.build` writes in the nightly run of Oct 1, the Oct 1 build, and withdraw the consent of the author of one of its records. Then rebuild the corpus in the same folder the next night, without generating its card again, and check each build against today's consent, as a training run does before it starts:

Example `consent-withdrawn-after-build` (Python 3.12.15 · datasketch 2.0.0 · duckdb 2.0.0 · mlcroissant 1.1.1 · openlineage-python 1.55.0 · phonenumbers 9.0.43 · presidio-analyzer 2.2.366 · presidio-anonymizer 2.2.366 · pyarrow 26.0.1 · pyiceberg 0.12.0 · pyyaml 6.0.3 · regopy 1.5.2 · sqlglot 30.33.0 · tldextract 5.4.0), passing in Edition 2026.11.31:

```python
"""Withdraws one author's consent after a build, rebuilds the corpus in the same folder, and checks both builds against today's consent and the card against the file. The prose's point rests on the "records whose author consents today" and "problems" lines."""
from datetime import timedelta

import pyarrow.compute as pc
import pyarrow.parquet as pq

from lib import ai_data, lineage
from lib.catalog import open_catalog
from lib.platform import load

catalog = open_catalog(["crm.learners", "learning.courses", "learning.reviews"])
first = ai_data.build(catalog)                        # the nightly run of Oct 1, and its card
card, _ = ai_data.card(first)
ai_data.write_card(card, first.folder)

# The next day, the author of one of the corpus's records withdraws their consent to AI training.
# review ID to the learner who wrote it, from the reviews table
authors = dict(zip(*load("learning.reviews").select(["review_id", "learner_id"]).to_pydict().values()))
# the author of the Oct 1 build's second record
learner = authors[first.records.column("review_id")[1].as_py()]
learners = catalog.load_table("crm.learners")
rows = learners.scan().to_arrow()
# that learner's answer becomes False; every other row keeps its answer
consent = pc.if_else(pc.equal(rows["learner_id"], learner), False, rows["ai_training_consent"])
learners.overwrite(rows.set_column(rows.schema.get_field_index("ai_training_consent"), "ai_training_consent", consent))

# The next nightly run rebuilds the corpus in the same folder, and nobody generates the card again. A training run
# checks its build against today's consent before it starts.
# the same folder: the corpus file is replaced, so the first card no longer matches it
second = ai_data.build(catalog, first.folder, lineage.NIGHTLY + timedelta(days=1))
for name, built in [("Oct 1 build", first), ("Oct 2 build", second)]:
    ids = built.records.column("review_id").to_pylist()
    # records in each build whose author is the learner who withdrew
    print(f"{name}, records from the learner who withdrew:", sum(authors[i] == learner for i in ids))
    print(f"{name}, records whose author consents today:", f"{ai_data.consenting(catalog, ids)} of {len(ids)}")
different_runs = first.run_id != second.run_id
assert different_runs
print("the two builds are different runs:", different_runs)
# the legal team's placeholder answer, from governance/ai_data.toml
print("a model made from the Oct 1 build, the legal team's answer:", ai_data.LEGAL["withdrawal"]["models_made_before"])

# the card's last preprocessing step, which states the Oct 1 counts
print("the Oct 1 card says:", card["rai:dataPreprocessingProtocol"][-1])
print("records in the corpus file now:", pq.read_metadata(first.folder / ai_data.CORPUS_FILE).num_rows)
# the platform's check of the Oct 1 card against the corpus file as it is now
problems = ai_data.card_problems(card, first.folder)
print("problems with the Oct 1 card, checked against the corpus now:", len(problems))
for problem in problems:
    print("-", problem)
```

Output:

```text
Oct 1 build, records from the learner who withdrew: 1
Oct 1 build, records whose author consents today: 95 of 96
Oct 2 build, records from the learner who withdrew: 0
Oct 2 build, records whose author consents today: 95 of 95
the two builds are different runs: True
a model made from the Oct 1 build, the legal team's answer: undecided
the Oct 1 card says: Kept 96 records, each with its source table, review ID, job and run.
records in the corpus file now: 95
problems with the Oct 1 card, checked against the corpus now: 2
- creator: missing
- review_corpus.parquet: its SHA-256 isn't the one the card states
```

The Oct 1 build holds 1 record by that learner, and the Oct 2 build holds 0, so the next corpus leaves the learner out. The two builds are different runs, with their own run IDs. The next training run has to leave the learner out too, and so does the next index.

Checked through provenance against today's answers, the Oct 1 build has 95 of 96 records whose author consents, so a training run that makes the check refuses it. The Oct 2 build passes with 95 of 95.

What was made from the Oct 1 build before the withdrawal keeps the review: a copy of its file, an index built from it, a model fine-tuned on it. The European Data Protection Board considers that "AI models trained with personal data cannot, in all cases, be considered anonymous" ([EDPB Opinion 28/2024](https://www.edpb.europa.eu/our-work-tools/our-documents/opinion-board-art-64/opinion-282024-certain-data-protection-aspects_en)).

Article 7(3) of the GDPR gives the right to withdraw consent at any time. It says that withdrawing doesn't affect the lawfulness of processing based on consent before the withdrawal, that the person must be told of this right before giving consent, and that withdrawing must be no harder than giving it ([GDPR](https://eur-lex.europa.eu/eli/reg/2016/679/oj)). [Chapter 7](#consent-at-query-time) quotes the first sentences of the same article and says what a withdrawal changes in the platform's code.

The code's part is to find, through provenance, every build that holds the record, and, by run ID, every model and index made from those builds. It then reads what happens to them from `[legal.withdrawal]` in `governance/ai_data.toml`. The legal team's answer for a model made from the Oct 1 build is `undecided`, a placeholder under which the job lists such models for the legal team and changes nothing. If the answer is that a model must not keep the review, the one way to remove it that we can vouch for is to train the model again on a build without it.

The card failed too, without a sound. Its last preprocessing step still reads "Kept 96 records, each with its source table, review ID, job and run." The file holds 95 records, and the platform's check finds 2 problems: the missing creator, and a file whose SHA-256 isn't the one the card states. A card that isn't generated again with its data describes another dataset.

**Working rule:** Record which build, by run ID, every model and index was made from. Before every training run, check the build's records against today's consent and the file's hash against its card, and refuse a build that fails either. When a learner withdraws, find the builds that hold their records and the models made from them, and let your legal team's answer in config decide what happens to those.

### Other failures {#other-ai-data-failures}

- **A filter on refusals that keeps the unknowns.** Built with it, the corpus held 2,045 reviews instead of 436, mostly by learners never asked. Notice it by counting the corpus's records per consent answer, through their provenance: any answer but yes is a defect.
- **Personal data a model memorizes from its text.** Carlini and colleagues extracted training text from a language model, including public personal information such as names, phone numbers and email addresses, even though each such sequence appeared in a single document ([Carlini et al., 2021](https://arxiv.org/abs/2012.07805)). The plain pipeline's corpus holds 120 labelled emails, 113 of them distinct and 106 written only once, and duplicates repeat whatever a detector misses in every copy. Measure what's left against labelled samples of your own text, and deduplicate before training.
- **A pseudonym key that leaks.** With the corpus's key and a list of instructors' names, anyone can compute every pseudonym in the corpus. When the key may have leaked, rotate it, rebuild every corpus made with it, and find by run ID the copies and models made from builds under the old key, as for a withdrawal.

## What this chapter decided {#ai-data-decisions}

- **The corpus is built through the enforcement layer,** as `analyst-1` for `ai-training`: only reviews whose author's `ai_training_consent` is true, contacts masked, an unknown consent counted as no.
- **Every legal answer the code needs is a placeholder for the legal team** in `[legal]` in `governance/ai_data.toml`: the consent that admits a record, the card's licence, and what happens to models made before a withdrawal.
- **Personal data is detected after the enforcement layer's masks,** with the pattern recognizers and the platform team's list of instructors' names, and replaced by keyed pseudonyms. Privacy Filter is measured, not added. The corpus is pseudonymized, never anonymized.
- **Pseudonyms are HMAC-SHA256 under the corpus job's own key,** not the enforcement layer's SHA-256 over key and value, and never Presidio's randomly salted `hash`.
- **Duplicates are removed in two steps,** exact copies by SHA-256, then near-duplicates from MinHash LSH tuned for 0.6 and confirmed at a Jaccard similarity of 0.8, never merging texts that differ only in a name. datasketch and its scheme, `affine32`, are pinned.
- **Every record keeps its provenance,** `source_table`, `review_id`, `job` and `run_id`, and a training run checks its build against today's consent first.
- **The dataset card is Croissant 1.1 with RAI 1.0,** generated by the run, checked by the platform's own code, and unpublished until `ai.review_corpus` has an owner.
- **Answers for one person come from an index of the corpus,** filtered by that person's decision on each record's source review.
- **The EU AI Act is quoted as amended and linked, never interpreted.** A corpus is only as clean as what was admitted into it. Each record is decided when the corpus is built, each record keeps its source, so a decision can be traced back to the review it came from, and the card says what was done with the rest.

# Governance in CI

By the end of this chapter you can run the governance checks of chapters 2 to 7 as one job on each pull request, judging only what the pull request changes. It fails when a column has no classification, runs the tests of the access policy and the enforcement layer, reports what each schema change reaches downstream, and enforces two rules that breaking-change checkers, the tools that compare two versions of a schema for changes that break its readers, don't. You'll keep ownership in contracts of the Open Data Contract Standard (ODCS), generated from the metadata, with a folder and a code owner per domain, layer each domain's rules on global ones, and measure each check's noise by the findings nobody acts on.

Earlier chapters built the governance checks of an online learning platform whose teams change its schemas every week, and no pull request runs any of them. [Chapter 2](#a-catalog-you-can-test) tests the catalog against the reviewed metadata, and [chapter 4](#lineage-from-pipelines) reads lineage from every pipeline run. [Chapter 5](#access-policies-as-code) tests the access policy, and [chapter 6](#enforcing-policies-on-queries) tests the layer that enforces it on queries. [Chapter 7](#retention-consent-erasure) generates table properties from the classifications. Each runs only when someone runs its example, never when a team changes the platform. Three changes are on their way. The CRM team wants a column for learners' work email, the learning team wants to rename a column the reporting pipelines read, and the data-science team wants to see learners' cities without a mask. Each is a small diff a reviewer would approve, and each changes what breaks or who sees what.

This chapter adds the book's shared module for catching them, `lib.ci`, which is the platform's governance job. In this chapter the job is this CI job, not an OpenLineage one. Each simulated pull request runs offline, as two folders called checkouts, one of the main branch and one of the pull request's branch. Each holds `governance/`, the platform's governance files; `pipelines/`, its pipelines' SQL; `migrations/`, the `alter table` statements a branch changes schemas with, which `lib.ci.migrate` applies as Iceberg schema changes; and the files [generated from the metadata](#a-folder-and-a-code-owner-per-domain). For each side, `lib.ci.build` builds a scratch platform the way the platform runs: the source tables, the side's migrations, the deploy, which writes the metadata into the catalog and sets every table's generated properties, then the nightly run of the side's own pipelines, with lineage. Among those pipelines, `review_corpus` stays [chapter 4](#lineage-from-pipelines)'s ungoverned copy, so the job keeps checking its lineage; the platform trains on [chapter 9's governed build](#admit-consented-reviews). `lib.ci.run` runs the checks and compares the two sides. No git or network is involved, and every date is compared with the book's "today", Oct 1, 2026, never the clock.

## Run every check as one job {#run-every-rule-as-one-job}

A check that runs when someone remembers it is a report, and reports go unread. On a pull request, a check is a gate: it runs on every change, before the change merges, and it fails the pull request through its exit code. That is how Open Policy Agent (OPA) advises running policies in CI, with flags, `--fail` and `--fail-defined`, that set the exit code to 1 based on query results ([OPA CI/CD docs](https://www.openpolicyagent.org/docs/cicd)). A job made of several checks needs four things more:

- **Every check runs and reports all its findings,** so one failure doesn't hide the next. A check that crashes becomes one finding, "the check could not run", which fails the job while the other checks run.
- **Every finding has a stable ID,** so the same problem has the same name on every run and can be discussed, counted and waived. `lib.ci` writes it as the check and the subject, such as `coverage/crm.learners.work_email`.
- **The job judges what the pull request changes.** It runs the main branch's checks too, and compares the two. A finding the pull request introduces fails it. A finding already on main is reported on every run, counted and named with the team that owns it, but it fails no pull request that leaves it as it is. We judge that a check that fails every pull request for findings nobody in it caused gets switched off. Google's code-analysis platform, Tricorder, shows most of its analyses' results only on the lines a change touches, by default ([Sadowski et al., 2015](https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/43322.pdf)).
- **The list of checks lives in a reviewed file,** `governance/ci.toml`, so switching one off is a diff that someone approves.

The platform's job runs these checks, in this order:

| Check | From | Fails when |
| --- | --- | --- |
| `coverage` | Chapters [2](#test-the-catalog-against-the-metadata) and [3](#classify-columns-from-the-schema) | A table has no metadata or owner, or a column no classification or one below its name's suggestion |
| `contract-rules` | [This chapter](#run-global-and-domain-rules) | A contract breaks a global or a domain rule |
| `contracts` | [This chapter](#a-folder-and-a-code-owner-per-domain) | A generated file is out of date |
| `policy-tests` | [Chapter 5](#unit-test-the-policy-in-rego) | A Rego test or a who-sees-what row fails |
| `enforcement-tests` | [Chapter 6](#allow-only-catalog-sources) | A regression case or a who-sees-what query fails |
| `migrations` | [This chapter](#a-broken-file-becomes-a-finding) | Iceberg refuses a migration |
| `pipelines` | [Chapter 4](#fail-on-unreadable-lineage) | A pipeline can't run, or its lineage can't be read |
| `lineage-classifications` | [Chapter 4](#personal-data-tagged-internal) | A column is declared less protected than lineage proposes |
| `owner-review` | [This chapter](#ask-the-owner-before-lowering) | A classification drops, or an owner or domain changes, unapproved |
| `removals` | [This chapter](#deprecate-before-removing) | A column is removed before it was deprecated |
| `properties` | [Chapter 7](#keep-values-out-of-manifests) | A table lacks the properties generated for it |

The first pull request the job sees is the CRM team's. Employers buy team licences for their staff, so the team adds `work_email` to `crm.learners` in one migration, and forgets the column's metadata:

Example `governance-ci-report` (Python 3.12.15 · duckdb 2.0.0 · openlineage-python 1.55.0 · phonenumbers 9.0.43 · presidio-analyzer 2.2.366 · pyarrow 26.0.1 · pyiceberg 0.12.0 · pyyaml 6.0.3 · regopy 1.5.2 · sqlglot 30.33.0 · tldextract 5.4.0), passing in Edition 2026.11.31:

```python
"""The platform's governance job on one pull request, run as a CI step runs it: the CRM team adds a work email to
crm.learners. The prose's point rests on the coverage finding for the new column and on the exit code the step returns."""

from lib import ci
from lib.platform import scratch

main = ci.checkout()                      # the main branch: governance files, pipelines and the files generated from them
branch = ci.checkout(main)                # the pull request's branch, starting from main
(branch / "migrations" / "0001-add-work-email.sql").write_text("alter table crm.learners add column work_email varchar;\n")
print("files the pull request changes:", ", ".join(ci.changed(main, branch)))

# What a CI step runs, `python -m lib.ci MAIN BRANCH --summary FILE`: it prints the report and its totals, writes the
# report as Markdown for the pull request, and exits with the code it returns.
summary = scratch() / "summary.md"
code = ci.main([str(main), str(branch), "--summary", str(summary)])     # 1 when a finding this pull request adds fails it
print("the step exits with:", code)
print("the summary it writes for the pull request:")
print(summary.read_text(encoding="utf-8"), end="")
```

Output:

```text
files the pull request changes: migrations/0001-add-work-email.sql
coverage: failed, findings 3, new 1, on main 2, waived 0, 12 tables checked
contract-rules: passed, findings 0, new 0, on main 0, waived 0, 10 contracts checked
contracts: passed, findings 0, new 0, on main 0, waived 0, 11 files checked
policy-tests: passed, findings 0, new 0, on main 0, waived 0, 24 tests and rows checked
enforcement-tests: passed, findings 0, new 0, on main 0, waived 0, 44 cases checked
migrations: passed, findings 0, new 0, on main 0, waived 0, 1 migrations checked
pipelines: passed, findings 0, new 0, on main 0, waived 0, 4 pipelines checked
lineage-classifications: passed, findings 0, new 0, on main 0, waived 0, 23 columns checked
owner-review: passed, findings 0, new 0, on main 0, waived 0, 77 columns checked
removals: passed, findings 0, new 0, on main 0, waived 0, 87 columns checked
properties: passed, findings 4, new 0, on main 4, waived 0, 12 tables checked
coverage/ai.review_corpus: no metadata; on main, owner team-platform
coverage/crm.learners.work_email: not classified; its name suggests personal
coverage/marketing.audience: no metadata; on main, owner team-platform
properties/ai.review_corpus: 7 of 7 generated properties missing; on main, owner team-platform
properties/analytics.course_performance: 4 of 4 generated properties missing; on main, owner team-analytics
properties/analytics.learner_activity: 4 of 4 generated properties missing; on main, owner team-analytics
properties/marketing.audience: 7 of 7 generated properties missing; on main, owner team-platform
change crm.learners, field 16: added as work_email string; named by 0 pipelines, reaches 0 columns
job: failed
checks run: 11
checks failed: 1
findings: 7
findings the pull request introduces: 1
findings already on main: 6
findings waived: 0
findings approved: 0
exit code: 1
the step exits with: 1
the summary it writes for the pull request:
## Governance job: failed

| Check | Result | New | On main | Waived | Checked |
| --- | --- | ---: | ---: | ---: | --- |
| `coverage` | failed | 1 | 2 | 0 | 12 tables |
| `contract-rules` | passed | 0 | 0 | 0 | 10 contracts |
| `contracts` | passed | 0 | 0 | 0 | 11 files |
| `policy-tests` | passed | 0 | 0 | 0 | 24 tests and rows |
| `enforcement-tests` | passed | 0 | 0 | 0 | 44 cases |
| `migrations` | passed | 0 | 0 | 0 | 1 migrations |
| `pipelines` | passed | 0 | 0 | 0 | 4 pipelines |
| `lineage-classifications` | passed | 0 | 0 | 0 | 23 columns |
| `owner-review` | passed | 0 | 0 | 0 | 77 columns |
| `removals` | passed | 0 | 0 | 0 | 87 columns |
| `properties` | passed | 0 | 4 | 0 | 12 tables |

### To fix in this pull request

- `coverage/crm.learners.work_email`: not classified; its name suggests personal

### Already on main, and whose it is

- `coverage/ai.review_corpus`: no metadata (team-platform)
- `coverage/marketing.audience`: no metadata (team-platform)
- `properties/ai.review_corpus`: 7 of 7 generated properties missing (team-platform)
- `properties/analytics.course_performance`: 4 of 4 generated properties missing (team-analytics)
- `properties/analytics.learner_activity`: 4 of 4 generated properties missing (team-analytics)
- `properties/marketing.audience`: 7 of 7 generated properties missing (team-platform)

### Schema changes

- `crm.learners`, field 16: added as work_email string; named by 0 pipelines, reaches 0 columns
```

The pull request introduces 1 finding, and it is the one that fails the job: `crm.learners.work_email` is not classified; its name suggests personal. The suggestion comes from [chapter 3](#classify-columns-from-the-schema)'s name rules, which match `e_?mail`, so the finding says what to add. Of the job's checks, 1 fails, and the step exits with code 1. The coverage findings on main are `marketing.audience` and `ai.review_corpus`, the tables nobody has described since [chapter 2](#a-catalog-you-can-test) found them. Neither table has an owner, so the job names the platform team, which `ci.toml` makes answerable for tables nobody owns. [Chapter 4](#classifications-from-lineage)'s lineage shows that both hold personal data, and every report repeats them until someone describes them.

The other checks pass, among them chapter 5's policy tests, run on the data the branch's files generate, and chapter 6's enforcement tests, run on the branch's tables. The properties check reports findings on main, such as `4 of 4 generated properties missing; on main, owner team-analytics`: the tables pipelines write lose their generated properties on every nightly run, as [chapter 7](#verify-the-erasure) found, on main as on the branch. The schema change, `added as work_email string; named by 0 pipelines, reaches 0 columns`, reaches nothing yet.

**Working rule:** Run every check as one job on every pull request, each reporting all its findings under stable IDs, and a crashed check as a failing finding. Fail the job by exit code on any finding the pull request introduces that is neither waived nor approved. Report every finding already on main, counted and named with its owner, without failing a pull request that leaves it as it is. Keep the list of checks in a reviewed file.

## Gate the pull request, not the commit {#gate-the-pull-request}

[The platform's job](#run-every-rule-as-one-job) runs in CI. A pre-commit hook is the other place a check can run, on the author's machine, before a commit exists. pre-commit, a framework for git hooks, which usually runs them only on the files a commit changes, and its documentation presents hooks as a way to catch simple problems before code review ([pre-commit](https://pre-commit.com/)). That suits checks that need one file: that `metadata.toml` parses, and that every classification is one of the four values.

It doesn't suit the coverage check. The CRM team's pull request changed `migrations/0001-add-work-email.sql` and no metadata file, so a hook filtered to metadata files would have had nothing to check. The gap sits between the catalog, which the migration changed, and a file the pull request never touched, and only a job that builds the whole catalog sees it. A hook can be set to run anyway, with pre-commit's `always_run`, so that it runs even when no file matches. But it runs only where someone installed it, and anyone can skip it, with pre-commit's `SKIP` variable or git's `--no-verify`, and no reviewer can see either.

So the gate is the job on the pull request, made required. GitHub's branch protection can require that every required status check pass before collaborators can merge changes into the protected branch ([GitHub, protected branches](https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/managing-protected-branches/about-protected-branches)), and a status that isn't required is only advice. Where the findings appear matters as much. Google's Tricorder authors found that analysis use drops off when developers have to open a dashboard or run a separate command-line tool ([Sadowski et al., 2015](https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/43322.pdf)). That is one company and code analysis, not data, but it's why `lib.ci.main` does more than return the exit code. A CI step runs it as `python -m lib.ci MAIN BRANCH`, and it prints one line per finding, with its ID and what to add. It also writes the report as Markdown for the pull request, where the reviewer reads the diff. Posting the summary, and reading approvals from the pull request's reviews, are the CI service's part and outside this book's scope.

**Working rule:** Use hooks for fast checks of single files on the author's machine, and make the CI job on the pull request the gate, required by branch protection on the main branch. Put every finding in the pull request itself, with its ID and what to do about it, never only on a dashboard. Run its checks on a schedule against the production catalog too, for the tables that change outside any pull request.

## Report what a schema change reaches {#report-what-a-schema-change-reaches}

The CRM team's added column reached nothing downstream, but a rename does, and its three kinds of readers, Iceberg, SQL and the governance metadata, judge it differently. Iceberg readers find columns by field ID, and the spec says that "projection must be done using field ids" ([Iceberg table spec, 1.12.0](https://github.com/apache/iceberg/blob/apache-iceberg-1.12.0/format/spec.md)), so a renamed column still reads every file written before the rename. SQL finds columns by name, so the same rename breaks every query and pipeline that names the column. The platform's governance metadata is keyed by field ID, as [chapter 2](#key-column-metadata-by-field-id) decided, so a rename needs only the name beside the ID updated. The spec's valid primitive type promotions are `int` to `long`, `float` to `double`, and a decimal's precision, which may only widen, plus, in format v3, `date` to `timestamp` or `timestamp_ns`, and a column of type `unknown` to any type. Any other change means a new column.

Which SQL a change breaks is a lineage question, and [chapter 4](#impact-of-a-column-change) answered it by walking the column graph. A review has to follow INDIRECT edges too, [chapter 4](#add-indirect-inputs)'s name for inputs that decide which rows an output holds rather than what its values are. OpenLineage's column lineage facet keeps a `dataset` list for column lineage that affects the whole dataset, through filtering, sorting, grouping, joining, window functions and the like ([ColumnLineageDatasetFacet, 1.54.0](https://github.com/OpenLineage/OpenLineage/blob/1.54.0/spec/facets/ColumnLineageDatasetFacet.json)). A column used only to join or group shapes every row it feeds, which a report of copied values misses. `lib.ci.changes` compares the two sides' schemas field ID by field ID. For every column the branch renames, retypes or removes, it walks the main branch's lineage graph: the pipelines whose SQL names the column, the columns whose values come from it, every column it can change, and the teams that own them.

Next comes the learning team's rename of `learning.courses.category` to `subject`, in a migration and in the metadata file. The same pull request also fixes the two pipelines that name the column:

Example `schema-change-impact` (Python 3.12.15 · duckdb 2.0.0 · openlineage-python 1.55.0 · phonenumbers 9.0.43 · presidio-analyzer 2.2.366 · pyarrow 26.0.1 · pyiceberg 0.12.0 · pyyaml 6.0.3 · regopy 1.5.2 · sqlglot 30.33.0 · tldextract 5.4.0), passing in Edition 2026.11.31:

```python
"""What the learning team's rename of learning.courses.category reaches, through Iceberg readers, SQL and lineage. The
prose's point rests on the courses an Iceberg reader still finds, the columns reached, and the job's verdict once the fix merges with the rename."""
import pyarrow.compute as pc

from lib import ci, retention

main = ci.checkout()
branch = ci.checkout(main)
(branch / "migrations" / "0001-rename-category.sql").write_text(
    "alter table learning.courses rename column category to subject;\n")
ci.edit(branch, "governance/metadata.toml", '3 = { name = "category", classification = "public"',
        '3 = { name = "subject", classification = "public"')       # the same field ID, renamed

before, after = ci.build(main), ci.build(branch)
# matched by field ID, so the rename is one change, not a column dropped and another added
change = next(c for c in ci.changes(before, after) if c.table == "learning.courses")
print("change:", change.kind, "from", change.before, "to", change.after)
print("field ID, before and after:", change.field_id)
# read by the new name: Iceberg maps it to the same field ID as before
subjects = after.catalog.load_table("learning.courses").scan(selected_fields=("subject",)).to_arrow()
print("courses whose subject an Iceberg reader finds:", pc.count(subjects["subject"]).as_py())
print("pipelines whose SQL names it:", ", ".join(change.jobs))
for job, why in sorted(after.failed.items()):
    print(f"pipeline {job}, on the branch:", why)
print("columns whose values come from it:", len(change.values))
for column in change.values:
    print("  value of:", column)
print("columns it can change, DIRECT and INDIRECT:", len(change.reached))
# a column's table is the first two parts of its name, schema.table.column
print("tables those columns are in:", ", ".join(sorted({".".join(c.split(".")[:2]) for c in change.reached})))
print("owners the report goes to:", ", ".join(change.owners))
print("tables on main:", len(retention.tables(before.catalog)))
print("tables on the branch:", len(retention.tables(after.catalog)))
coverage = ci.run(before, after, checks=["coverage"]).checks[0]
print("coverage findings on the branch:", ", ".join(finding.id for finding in coverage.findings))

# The same pull request, with both pipelines changed to read the new name. Each side runs its own pipelines/.
ci.edit(branch, "pipelines/course_performance.sql", "    c.category,", "    c.subject as category,")
ci.edit(branch, "pipelines/course_performance.sql", "c.title, c.category", "c.title, c.subject")
sql = (branch / "pipelines" / "marketing_audience.sql").read_text(encoding="utf-8")
(branch / "pipelines" / "marketing_audience.sql").write_text(sql.replace("c.category", "c.subject").replace(
    "f.category", "f.subject"), encoding="utf-8")
fixed = ci.build(branch)
ci.generate(fixed)                        # and the contracts generated again, as the author's hook would
print("files the fixed pull request changes:", ", ".join(ci.changed(main, branch)))
for line in ci.lines(ci.run(before, fixed, checks=["pipelines", "coverage", "contracts", "lineage-classifications"])):
    print("with the fix,", line)
```

Output:

```text
change: renamed from category string to subject string
field ID, before and after: 3
courses whose subject an Iceberg reader finds: 200
pipelines whose SQL names it: course_performance, marketing_audience
pipeline course_performance, on the branch: Unknown column: category
pipeline marketing_audience, on the branch: Unknown column: category
columns whose values come from it: 2
  value of: analytics.course_performance.category
  value of: marketing.audience.favourite_category
columns it can change, DIRECT and INDIRECT: 12
tables those columns are in: analytics.course_performance, marketing.audience
owners the report goes to: no owner, team-analytics
tables on main: 12
tables on the branch: 10
coverage findings on the branch: coverage/ai.review_corpus
files the fixed pull request changes: governance/contracts/learning/courses.odcs.yaml, governance/metadata.toml, migrations/0001-rename-category.sql, pipelines/course_performance.sql, pipelines/marketing_audience.sql
with the fix, pipelines: passed, findings 0, new 0, on main 0, waived 0, 4 pipelines checked
with the fix, coverage: passed, findings 2, new 0, on main 2, waived 0, 12 tables checked
with the fix, contracts: passed, findings 0, new 0, on main 0, waived 0, 11 files checked
with the fix, lineage-classifications: passed, findings 0, new 0, on main 0, waived 0, 23 columns checked
with the fix, coverage/ai.review_corpus: no metadata; on main, owner team-platform
with the fix, coverage/marketing.audience: no metadata; on main, owner team-platform
with the fix, change learning.courses, field 3: renamed from category string to subject string; named by 2 pipelines, reaches 12 columns
with the fix, job: passed
```

For Iceberg, nothing broke. The rename kept the column's field ID, so an Iceberg reader on the branch still finds a subject for 200 courses. For SQL, the pipelines that name the column, course_performance, marketing_audience, each stop before writing anything on the branch, because [chapter 4](#fail-on-unreadable-lineage) made unreadable lineage an error. Through values, the rename reaches 2 columns, the two copies of the category. Through every edge it reaches 12 columns, every column of `analytics.course_performance, marketing.audience`, because categories group the course report and rank each learner's favourite. The report goes to no owner, team-analytics: the analytics team, and nobody for `marketing.audience`, which has no owner yet.

A broken build hides what it breaks. The branch's catalog holds 10 tables, against 12 on main, because the broken pipelines wrote nothing. Coverage reports only `coverage/ai.review_corpus`: a table that doesn't exist can't be found undescribed, so `marketing.audience`'s finding is gone. Each side runs its own `pipelines/`, so the fix travels with the rename: with both pipelines reading `subject`, the job reports `passed`. In production, the merge can also guard what the review saw. A REST catalog commit can carry `AssertCurrentSchemaId`, which requires the table's current schema ID to match the one the requirement names ([REST catalog spec, 1.12.0](https://github.com/apache/iceberg/blob/apache-iceberg-1.12.0/open-api/rest-catalog-open-api.yaml)), so a migration reviewed against one schema fails instead of applying to another.

**Working rule:** Diff schemas by field ID, never by name or schema ID, and report every rename, type change and removal three ways: whether Iceberg readers are affected, which pipelines' SQL names the column, and what it reaches through lineage, INDIRECT edges included. Send the report to the owners of every table it reaches. Merge a rename that breaks a pipeline in the same pull request as the pipeline's fix, never before it.

## Ask the owner before lowering a classification {#ask-the-owner-before-lowering}

Breaking-change checkers protect the readers of a schema. dbt's model contracts flag removing a column, or changing its data type, as breaking changes that could affect downstream consumers ([dbt model contracts](https://docs.getdbt.com/docs/mesh/govern/model-contracts)). datacontract-cli's `breaking` command compares two ODCS contracts and fails a change that breaks a guarantee consumers rely on ([datacontract-cli 1.2.7](https://github.com/datacontract/datacontract-cli/blob/v1.2.7/README.md)). Neither treats protection as a guarantee. In datacontract-cli 1.2.7, no rule covers a property's classification or a team member's role, so a change to either falls to a rule that marks it untested ([rules.py](https://github.com/datacontract/datacontract-cli/blob/v1.2.7/datacontract/breaking/rules.py)). The detector reports such a change at INFO ([detector.py](https://github.com/datacontract/datacontract-cli/blob/v1.2.7/datacontract/breaking/detector.py)), a level that doesn't fail. The command exits 1 only for ERROR entries ([command_breaking.py](https://github.com/datacontract/datacontract-cli/blob/v1.2.7/datacontract/command_breaking.py), [model/breaking.py](https://github.com/datacontract/datacontract-cli/blob/v1.2.7/datacontract/model/breaking.py)). `examples/requirements.txt` doesn't pin it, so no example runs it. On Oct 22, 2026, datacontract-cli 1.2.7, run on the contracts this chapter generates for `crm.learners`, failed a renamed `email` with exit code 1. It read the rename as a removal, although the property kept its `id`, because it matches properties by name ([normalize.py](https://github.com/datacontract/datacontract-cli/blob/v1.2.7/datacontract/changelog/normalize.py)). It passed `email` reclassified from `personal` to `public`, and a change of owner, with exit code 0. ODCS itself says a classification "Can be anything" ([ODCS v3.2.0 schema](https://github.com/bitol-io/open-data-contract-standard/blob/v3.2.0/docs/schema.md)).

Yet every policy reads the classification, so lowering one is among the most consequential changes a metadata file can carry. We add a first rule of our own: lowering a column's classification, changing a table's owner, or moving a table to another domain needs the approval of the team that owns the table on the main branch. A move counts as an ownership change, because a domain's own rules apply to its tables ([a later section](#run-global-and-domain-rules)). The owner comes from the main branch, because otherwise a pull request could hand a table to a new owner, and the new owner could approve the downgrade. [Chapter 2](#measure-where-the-drafts-are-wrong) set the same rule for a model's drafts: only a reviewer lowers a classification, with the reason in the column's description.

The data-science team's pull request comes last. It lowers `crm.learners.city` from `personal` to `internal`, so data scientists can group learners by city. The access policy pseudonymizes personal columns for data scientists, and the downgrade takes `city` out of that rule. The run asks the policy before and after, then runs the checks that look at schemas and policies, and the owner rule:

Example `classification-downgrade` (Python 3.12.15 · duckdb 2.0.0 · openlineage-python 1.55.0 · phonenumbers 9.0.43 · presidio-analyzer 2.2.366 · pyarrow 26.0.1 · pyiceberg 0.12.0 · pyyaml 6.0.3 · regopy 1.5.2 · sqlglot 30.33.0 · tldextract 5.4.0), passing in Edition 2026.11.31:

```python
"""A pull request from the data-science team lowers crm.learners.city from personal to internal. The prose's point rests
on analyst-1's mask on each side, and on the owner rule failing alone."""
from lib import ci, policy

main = ci.checkout()
branch = ci.checkout(main)
ci.edit(branch, "governance/metadata.toml", '6 = { name = "city", classification = "personal"',
        '6 = { name = "city", classification = "internal"')
before, after = ci.build(main), ci.build(branch)

reading_city = policy.request("analyst-1", "crm.learners", ["learner_id", "city"])
for side, name in ((before, "main"), (after, "the branch")):
    rego = policy.interpreter(side.governance / "policies", data=ci.policy_data(side))
    # a column with no mask in the decision is shown as it is stored
    print(f"analyst-1 reading city, on {name}:", policy.decide(reading_city, rego)["column_masks"].get("city", "unmasked"))

checks = ["policy-tests", "enforcement-tests", "removals", "owner-review"]
report = ci.run(before, after, checks=checks)
print("schema changes:", len(report.changes))
for line in ci.lines(report):
    print(line)

# The approvals come from the pull request's reviews: one entry per approving review, named by the reviewer's team.
# The owner rule passes when it raises no unapproved finding.
owner_review = lambda approvals: "passed" if ci.run(before, after, approvals, checks=["owner-review"]).passed else "failed"
print("owner-review, once team-crm approves:", owner_review(["team-crm"]))
print("owner-review, once team-analytics approves:", owner_review(["team-analytics"]))
ci.edit(after.root, "governance/ci.toml", "teams = {}", "teams = { team-crm = 2 }")     # team-crm asks for two
# one approval from team-crm and one from another team: team-crm still needs a second of its own
print("owner-review, two required, one from team-crm:", owner_review(["team-crm", "team-platform"]))
print("owner-review, two required, two from team-crm:", owner_review(["team-crm", "team-crm"]))
```

Output:

```text
analyst-1 reading city, on main: pseudonymize
analyst-1 reading city, on the branch: unmasked
schema changes: 0
policy-tests: passed, findings 0, new 0, on main 0, waived 0, 24 tests and rows checked
enforcement-tests: passed, findings 0, new 0, on main 0, waived 0, 44 cases checked
removals: passed, findings 0, new 0, on main 0, waived 0, 87 columns checked
owner-review: failed, findings 1, new 1, on main 0, waived 0, 77 columns checked
owner-review/crm.learners.city: lowered from personal to internal; needs team-crm's approval
job: failed
owner-review, once team-crm approves: passed
owner-review, once team-analytics approves: failed
owner-review, two required, one from team-crm: failed
owner-review, two required, two from team-crm: passed
```

One word in the metadata file unmasks a column for a whole role, and no check but the owner rule notices. `analyst-1` reads `city` as `pseudonymize` on main, and as `unmasked` on the branch. The job reports 0 schema changes, so a breaking-change checker has nothing to compare. The policy tests pass, `passed, findings 0, new 0, on main 0, waived 0, 24 tests and rows checked`, and so do the enforcement tests, because none of them reads `city`, and the removals check passes because no column was removed. Only the owner rule fails, on `lowered from personal to internal; needs team-crm's approval`.

What settles the owner rule is the owner's approval. Approvals reach `lib.ci.run` as one entry per approving review, named by the reviewer's team. With `team-crm`'s approval, the owner rule reports `passed`, and with `team-analytics`', which doesn't own the table, `failed`.

**Working rule:** Treat a lowered classification, a changed owner and a move to another domain as changes that need the approval of the table's owner on the main branch, and fail the job until it's given; never rely on a breaking-change checker for them. Ask for the reason in the column's description in the same pull request. A raised classification needs no such approval, because it can only hide more.

## Deprecate a column before removing it {#deprecate-before-removing}

[The owner rule](#ask-the-owner-before-lowering) guards a change that breaking-change checkers ignore. Removing a column is the change they all catch, but by then the removal is already under review, too late to help the readers it breaks. ODCS 3.2.0 adds a way to warn readers first. Any element of a contract, a schema object or a property, nested ones included, may set the optional `deprecated` boolean to signal that it is no longer recommended for use. The standard also asks that the `description` point to a replacement and give migration guidance ([ODCS v3.2.0 schema](https://github.com/bitol-io/open-data-contract-standard/blob/v3.2.0/docs/schema.md)). Our second rule makes the deprecation warning a required step: a column may leave a table only once the main branch has marked it deprecated. That takes two pull requests, one that deprecates the column and one that removes it, and the column's readers have the time between them to move off it.

The platform's metadata file had no such field, so this chapter adds one: `deprecated = true` in a column's entry. `lib.metadata.load` reads it as `Column.deprecated`, false when it's absent, and `lib.ci.contract` writes it into the column's contract as ODCS's `deprecated`.

The platform's best candidate is `billing.payments.card_number`. Its own description says it "Should not be stored at all", and [chapter 3](#classify-columns-from-their-values) found that it holds full card numbers. `team-billing` removes it twice, each time by dropping the column in a migration and deleting its metadata entry: once straight away, and once after a pull request that deprecates it has merged:

Example `deprecate-then-remove` (Python 3.12.15 · duckdb 2.0.0 · openlineage-python 1.55.0 · phonenumbers 9.0.43 · presidio-analyzer 2.2.366 · pyarrow 26.0.1 · pyiceberg 0.12.0 · pyyaml 6.0.3 · regopy 1.5.2 · sqlglot 30.33.0 · tldextract 5.4.0), passing in Edition 2026.11.31:

```python
"""team-billing removes the legacy card_number column from billing.payments: once straight away, and once after a pull
request that deprecates it has merged. The prose's point rests on the two removal lines."""
import yaml

from lib import ci

entry = ('8 = { name = "card_number", classification = "sensitive", '
         'description = "The full card number, from a legacy export. Should not be stored at all." }\n')
# the note goes after the description's sentence ending "at all.", and deprecated = true goes inside the entry's braces
deprecated_entry = entry.replace("all.", "all. Deprecated: reconcile payments by payment_id.").replace(" }", ", deprecated = true }")


def remove_card_number(base, line):
    """A pull request on top of `base` that drops the column and deletes its metadata entry."""
    branch = ci.checkout(base)
    (branch / "migrations" / "0001-drop-card-number.sql").write_text("alter table billing.payments drop column card_number;\n")
    ci.edit(branch, "governance/metadata.toml", line, "")
    return branch


main = ci.checkout()
deprecated = ci.checkout(main)            # the first pull request: mark the column deprecated, and say what replaces it
ci.edit(deprecated, "governance/metadata.toml", entry, deprecated_entry)

for base, line, name in ((main, entry, "straight away"), (deprecated, deprecated_entry, "after the deprecation merged")):
    report = ci.run(ci.build(base), ci.build(remove_card_number(base, line)), checks=["removals"])
    print(f"removal, {name}:", "passed" if report.checks[0].passed else "failed")
    for finding in report.checks[0].findings:
        print(f"  {finding.id}:", finding.message)

# property 7 is the eighth column, field 8: card_number, the column the deprecation marks
card_number = ci.contract(ci.build(deprecated), "billing.payments")["schema"][0]["properties"][7]
print("the deprecated column in its ODCS contract:")
print(yaml.safe_dump(card_number, sort_keys=False, width=200), end="")
```

Output:

```text
removal, straight away: failed
  removals/billing.payments.card_number: removed without being deprecated on the main branch first
removal, after the deprecation merged: passed
the deprecated column in its ODCS contract:
id: field-8
name: card_number
logicalType: string
physicalType: string
required: false
description: 'The full card number, from a legacy export. Should not be stored at all. Deprecated: reconcile payments by payment_id.'
classification: sensitive
deprecated: true
```

Straight away, the removal fails, `removed without being deprecated on the main branch first`. After the deprecation has merged, the same removal reports `passed`. In between, the contract carried the warning to everyone who reads contracts, not only to the repository: `deprecated: true`, with a description that says how to reconcile payments without the column. The rule checks the order of the two changes, not the time between them. How long readers get is a decision for the owning team and the column's readers, and [the impact report](#report-what-a-schema-change-reaches) says who those readers are.

**Working rule:** Remove a column in two pull requests: first mark it deprecated, with its replacement in the description, and merge that; then drop it from the table and its entry from the metadata. Fail any removal of a column the main branch hadn't deprecated. Agree with the column's readers how long the deprecation lasts, since the job checks only that it came first.

## Route each domain's reviews by path {#a-folder-and-a-code-owner-per-domain}

The [owner rule](#ask-the-owner-before-lowering) asks a table's team to approve its downgrades, and the repository can route that request. GitHub requests code owners' review automatically when a pull request modifies code they own, and a branch protection rule can require that review ([GitHub's documentation, About code owners](https://github.com/github/docs/blob/d59b2631959db46d71ca5f7748ec04311c6d0870/content/repositories/managing-your-repositorys-settings-and-features/customizing-your-repository/about-code-owners.md)). The platform keeps all its metadata in one file, `governance/metadata.toml`, which suits 5 teams in one repository. A file has one set of owners, though, so the request can reach the right team only if each team's metadata lives under a path of its own.

ODCS draws the same lines. A contract names one [`domain`](#keep-metadata-in-a-reviewed-file), the logical data domain it belongs to ([ODCS v3.2.0 fundamentals](https://github.com/bitol-io/open-data-contract-standard/blob/v3.2.0/docs/fundamentals.md)). ODCS has no owner field: ownership is a free-form `role` on a member of the contract's `team` ([ODCS v3.2.0 team](https://github.com/bitol-io/open-data-contract-standard/blob/v3.2.0/docs/team.md)). `lib.ci.contract` writes each described table as an ODCS contract. It records the domain, the owning team as a member with the role `owner`, and each column as a property whose `id` holds its Iceberg field ID. ODCS defines a property's `id` as "A unique identifier for the element used to create stable, refactor-safe references". `lib.ci.generate` writes each contract into a folder for its domain, and a CODEOWNERS file in `.github/`, the first place GitHub looks for one. Both are generated from the metadata, never typed. Ask whom GitHub would request for [the data-science team's downgrade of `city`](#ask-the-owner-before-lowering):

Example `domain-contracts` (Python 3.12.15 · duckdb 2.0.0 · openlineage-python 1.55.0 · phonenumbers 9.0.43 · presidio-analyzer 2.2.366 · pyarrow 26.0.1 · pyiceberg 0.12.0 · pyyaml 6.0.3 · regopy 1.5.2 · sqlglot 30.33.0 · tldextract 5.4.0), passing in Edition 2026.11.31:

```python
"""Ownership as ODCS contracts, a folder per domain, and who GitHub asks to review the downgrade of crm.learners.city
when the metadata is one shared file, and when each domain has a folder of contracts generated from it. The prose's
point rests on the first review line and on the contracts check failing before the files are generated again."""
import yaml

from lib import ci

main = ci.checkout()                      # the main branch holds the contracts and CODEOWNERS generated from its metadata
branch = ci.checkout(main)
ci.edit(branch, "governance/metadata.toml", '6 = { name = "city", classification = "personal"',
        '6 = { name = "city", classification = "internal"')

shared = ["/governance/ @example-org/team-platform"]          # one file for every domain, owned by the platform team
for path, owners in ci.reviewers(shared, ["governance/metadata.toml"]).items():
    print(f"one shared file, review of {path}:", " ".join(owners))

crm = yaml.safe_load((main / "governance/contracts/crm/learners.odcs.yaml").read_text(encoding="utf-8"))
print(yaml.safe_dump({"domain": crm["domain"], "team": crm["team"]}, sort_keys=False), end="")
lines = (main / ".github/CODEOWNERS").read_text(encoding="utf-8").splitlines()
for line in lines[1:]:                                       # the first line says the file is generated
    print("CODEOWNERS", line)
before, after = ci.build(main), ci.build(branch)          # the pull request changes the metadata only
print("teams that own tables:", len({table.owner for table in before.described.values()}))
# drops the job's verdict, the last line, which this example doesn't print
for line in ci.lines(ci.run(before, after, checks=["contracts"]))[:-1]:
    print("before generating again,", line)
print("files generated again:", len(ci.generate(after)))
print("after generating again, contracts:", "passed" if ci.run(before, after, checks=["contracts"]).passed else "failed")
# the last matching CODEOWNERS line wins, as GitHub applies it
for path, owners in ci.reviewers(lines, ci.changed(main, branch)).items():
    print(f"a folder per domain, review of {path}:", " ".join(owners))
```

Output:

```text
one shared file, review of governance/metadata.toml: @example-org/team-platform
domain: crm
team:
  name: team-crm
  members:
  - username: team-crm
    role: owner
CODEOWNERS /.github/CODEOWNERS @example-org/team-platform
CODEOWNERS /governance/ @example-org/team-platform
CODEOWNERS /migrations/ @example-org/team-platform
CODEOWNERS /pipelines/ @example-org/team-platform
CODEOWNERS /pipelines/course_performance.sql @example-org/team-analytics
CODEOWNERS /pipelines/learner_activity.sql @example-org/team-analytics
CODEOWNERS /governance/contracts/analytics/ @example-org/team-analytics
CODEOWNERS /governance/contracts/billing/ @example-org/team-billing
CODEOWNERS /governance/contracts/crm/ @example-org/team-crm
CODEOWNERS /governance/ci/domains/crm.rego @example-org/team-crm
CODEOWNERS /governance/contracts/learning/ @example-org/team-learning
CODEOWNERS /governance/contracts/support/ @example-org/team-support
teams that own tables: 5
before generating again, contracts: failed, findings 1, new 1, on main 0, waived 0, 11 files checked
before generating again, contracts/governance/contracts/crm/learners.odcs.yaml: differs from what the metadata generates: generate it again
files generated again: 11
after generating again, contracts: passed
a folder per domain, review of governance/contracts/crm/learners.odcs.yaml: @example-org/team-crm
a folder per domain, review of governance/metadata.toml: @example-org/team-platform
```

With one shared file, the downgrade changes `governance/metadata.toml`, and GitHub asks `@example-org/team-platform` to review it. That is the team that owns the file, not `team-crm`, whose column it is. The generated CODEOWNERS gives the platform team the file itself, because About code owners says a repository is fully protected against unauthorized changes only when the CODEOWNERS file itself has an owner ([GitHub's documentation](https://github.com/github/docs/blob/d59b2631959db46d71ca5f7748ec04311c6d0870/content/repositories/managing-your-repositorys-settings-and-features/customizing-your-repository/about-code-owners.md)). The platform team also gets the governance files, the migrations and the pipelines. Each pipeline then goes to the team that owns its table, and each folder of contracts goes to its domain's team. The pull request changed only the metadata, so the `contracts` check, which generates every file again and compares, fails: `differs from what the metadata generates: generate it again`. Generated again, the pull request also changes the crm contract, and its review goes to `@example-org/team-crm`, because GitHub gives the last matching pattern precedence. A hook that generates the files saves the author a failed run. The check makes the routing certain.

When code-owner review is required, an approval from any one of a file's owners meets it ([About code owners](https://github.com/github/docs/blob/d59b2631959db46d71ca5f7748ec04311c6d0870/content/repositories/managing-your-repositorys-settings-and-features/customizing-your-repository/about-code-owners.md)). Branch protection asks for one number of approvals for the whole branch, met by reviewers with write permissions ([GitHub, protected branches](https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/managing-protected-branches/about-protected-branches)), so any such reviewer counts toward it, and it can't ask for two of `team-crm`'s members. The job can, from the approvals it's given. The owner rule counts them per team, against a number per team in `ci.toml`. With `team-crm` asking for two, [the downgrade of `city`](#ask-the-owner-before-lowering) with one `team-crm` approval reports `failed`, and with two it reports `passed`. Iceberg 1.12.0's REST catalog spec offers another place for classifications, `labels` per table and field ID ([REST catalog spec](https://github.com/apache/iceberg/blob/apache-iceberg-1.12.0/open-api/rest-catalog-open-api.yaml)). Coverage and the contract rules could read a catalog server's labels, but changes to them would need the server's own review, since the spec doesn't require that labels be persisted or versioned.

**Working rule:** Keep each domain's metadata under a path of its own, as one ODCS contract per table with one domain and one owning team, generated from the metadata and checked by the job, with a CODEOWNERS line for that team. Give the platform team the CODEOWNERS file, the job's settings, its global rules, the [waivers](#count-waived-findings) and the migrations. Require code-owner review, and count approvals per team in the job when one isn't enough.

## Layer domain rules on the global rules {#run-global-and-domain-rules}

Teams that own [their domains' contracts](#a-folder-and-a-code-owner-per-domain) still share one platform, and some rules have to hold in every domain. Dehghani's description of data mesh calls this federated computational governance, "a set of global rules - rules applied to all data products and their interfaces - to ensure a healthy and interoperable ecosystem". Those global rules sit beside the decisions each domain makes for itself, and the platform executes both ([Dehghani, 2020](https://martinfowler.com/articles/data-mesh-principles.html)). That's a framing, not a measurement, and it maps onto code. The platform team keeps one package of global rules, which the domains agree on, and in Dehghani's version the domain and platform product owners together decide them. Each domain has a package of its own, owned by its team, and one job runs them all.

The OPA project's tool for rules over files is Conftest, which tests structured configuration data with Rego ([Conftest](https://www.conftest.dev/)). `lib.ci` runs the same kind of rules in-process with regopy, as [chapter 5](#access-policies-as-code) runs the access policy. `governance/ci/global.rego` holds the global rules: one owner, whatever the case of its role; one domain, which a missing, empty or blank value fails, since an empty string is a defined value in Rego and `not` alone would pass it; a classification from the four, since ODCS accepts any string; and a description for every column. `governance/ci/domains/crm.rego` holds `team-crm`'s rule: nothing in crm is public, because everything in it describes a learner or an instructor. The `contract-rules` check runs the global package on every contract, and each domain's package on the contracts of that domain on the branch. A rule returns each violation as data, with the rule's name, a subject and a message. A violation's subject and the rule's name make the finding's ID. These files sit outside `governance/policies/`, the access policy's folder, so changing them never changes the policy revision that [chapter 8](#replay-old-decisions) records with every decision.

[The CRM team's change](#run-every-rule-as-one-job) comes back, this time with metadata. `work_email` is classified `public`, on the argument that a work address is business information, and its description is empty. The second try adds a description and moves `crm.learners` to the analytics domain, which has no rules of its own:

Example `global-and-domain-rules` (Python 3.12.15 · duckdb 2.0.0 · openlineage-python 1.55.0 · phonenumbers 9.0.43 · presidio-analyzer 2.2.366 · pyarrow 26.0.1 · pyiceberg 0.12.0 · pyyaml 6.0.3 · regopy 1.5.2 · sqlglot 30.33.0 · tldextract 5.4.0), passing in Edition 2026.11.31:

```python
"""The platform's global rules and the crm domain's own rule, run on the CRM team's two tries at work_email: classified
public with no description, then described with crm.learners moved to the analytics domain, which has no rules of its own.
The prose's point rests on the two contract-rules findings of the first try and on the owner-review finding after the move."""
from lib import ci

main = ci.checkout()
branch = ci.checkout(main)
(branch / "migrations" / "0001-add-work-email.sql").write_text("alter table crm.learners add column work_email varchar;\n")
ci.edit(branch, "governance/metadata.toml", '15 = { name = "notes"',
        '16 = { name = "work_email", classification = "public", description = "" }\n15 = { name = "notes"')

for path in sorted((branch / "governance" / "ci").rglob("*.rego")):
    print("rules:", path.relative_to(branch).as_posix())
before = ci.build(main)
report = ci.run(before, branch, checks=["coverage", "contract-rules"])
for line in ci.lines(report):
    print(line)

# The second try: a description, and the table moved to a domain whose rules allow public columns.
ci.edit(branch, "governance/metadata.toml", 'description = "" }', 'description = "The learner\'s work address." }')
# the table's domain changes from crm to analytics, which has no rules of its own
ci.edit(branch, "governance/metadata.toml", 'owner = "team-crm"\ndomain = "crm"\ndescription = "Every learner',
        'owner = "team-crm"\ndomain = "analytics"\ndescription = "Every learner')
report = ci.run(before, branch, checks=["coverage", "contract-rules", "owner-review"])
for line in ci.lines(report):
    print("moved to analytics,", line)
```

Output:

```text
rules: governance/ci/domains/crm.rego
rules: governance/ci/global.rego
coverage: failed, findings 3, new 1, on main 2, waived 0, 12 tables checked
contract-rules: failed, findings 2, new 2, on main 0, waived 0, 10 contracts checked
coverage/ai.review_corpus: no metadata; on main, owner team-platform
coverage/crm.learners.work_email: classified public; its name suggests personal; needs team-crm's approval
coverage/marketing.audience: no metadata; on main, owner team-platform
contract-rules/crm.learners.work_email/described: no description (global rule described)
contract-rules/crm.learners.work_email/no-public: crm classifies no column public (domains.crm rule no-public)
change crm.learners, field 16: added as work_email string; named by 0 pipelines, reaches 0 columns
job: failed
moved to analytics, coverage: failed, findings 3, new 1, on main 2, waived 0, 12 tables checked
moved to analytics, contract-rules: passed, findings 0, new 0, on main 0, waived 0, 10 contracts checked
moved to analytics, owner-review: failed, findings 1, new 1, on main 0, waived 0, 77 columns checked
moved to analytics, coverage/ai.review_corpus: no metadata; on main, owner team-platform
moved to analytics, coverage/crm.learners.work_email: classified public; its name suggests personal; needs team-crm's approval
moved to analytics, coverage/marketing.audience: no metadata; on main, owner team-platform
moved to analytics, owner-review/crm.learners/domain: domain changed from crm to analytics; needs team-crm's approval
moved to analytics, change crm.learners, field 16: added as work_email string; named by 0 pipelines, reaches 0 columns
moved to analytics, job: failed
```

Coverage flags the classification now: `classified public; its name suggests personal; needs team-crm's approval`. A name is evidence, and a pull request has no values to read, so a classification below the name's suggestion is a finding only the owner on main can answer.

The contract rules ask more. The global package flags `no description (global rule described)`, and the crm package flags `crm classifies no column public (domains.crm rule no-public)`. `team-crm` decided the second rule without asking any other team, and the platform runs it for them on every pull request, including pull requests from `team-crm`.

On the second try, the analytics domain's rules apply, and the contract rules pass: `passed, findings 0, new 0, on main 0, waived 0, 10 contracts checked`. The move itself fails, on `domain changed from crm to analytics; needs team-crm's approval`, and so does the classification.

**Working rule:** Write the rules every domain must follow as one global package, kept by the platform team, and let each domain add a package of its own, owned by its team, that runs on that domain's contracts only. Return violations as data with a stable subject, never as a bare pass or fail. Let domain rules add to the global rules, never replace them, and treat a table's move to another domain as an ownership change, so no team escapes its domain's rules by changing one word.

## Count waived findings per check {#count-waived-findings}

[The checks so far](#run-every-rule-as-one-job) each fail a pull request for a reason. A gate also fails when people stop believing it. Google's Tricorder authors define an effective false positive as any report that a user chooses not to act on, and they keep that rate under 10% at code review. What they act on is narrower, the not-useful rate: an analyzer's Not useful clicks as a share of its Not useful, Please fix and Apply fix clicks. At 10% or more, an analyzer goes on probation, and above 25%, it may be turned off at once ([Sadowski et al., 2015](https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/43322.pdf)). That is one company's code analysis, with thresholds of its own choosing, and we know of no published equivalent for data governance. The measure carries over as an analogy. Count, for each check, the findings nobody acts on, waived or left on main.

On the platform, the decision not to act on a new finding is a waiver. A waiver in `governance/waivers.toml` names one finding by its ID, the team answerable for it, a reason, and the last day it applies. `lib.ci` accepts a waiver that runs at most 90 days past the day the job runs. That limit is a rule of thumb kept in `governance/ci.toml`, so every waiver comes back for a decision. The same file lists the checks no waiver silences, whatever its date: coverage, since a table with no owner or a column with no classification is fixed in the pull request, and the enforcement tests. A finding already on main needs no waiver. An expired waiver silences nothing, and the report lists it. The platform team owns the waivers file, so each waiver is a reviewed change too.

The CRM change needs two waivers. The first covers the unclassified column. The second covers, once the column is classified `personal`, the description it hasn't written yet. The team first asks for the second to run until the summer, then until Oct 31:

Example `waived-findings` (Python 3.12.15 · duckdb 2.0.0 · openlineage-python 1.55.0 · phonenumbers 9.0.43 · presidio-analyzer 2.2.366 · pyarrow 26.0.1 · pyiceberg 0.12.0 · pyyaml 6.0.3 · regopy 1.5.2 · sqlglot 30.33.0 · tldextract 5.4.0), passing in Edition 2026.11.31:

```python
"""Waivers on the CRM team's pull request: one for the unclassified work_email column, which coverage never allows;
then, with the column classified but not yet described, one that asks for too long, and one that fits, on the book's
today and a month later. The prose's point rests on the two refusals and on the expired waiver the report lists."""
from datetime import date

from lib import ci
from lib.platform import TODAY


def waive(branch, finding, expires, reason):
    with open(branch / "governance" / "waivers.toml", "a", encoding="utf-8") as waivers:
        # the date is written unquoted, so TOML reads it as a date, which the job compares with the day it runs
        waivers.write(f'\n[[waivers]]\nfinding = "{finding}"\nowner = "team-crm"\nreason = "{reason}"\nexpires = {expires}\n')


main = ci.checkout()
before = ci.build(main)
unclassified = ci.checkout(main)
(unclassified / "migrations" / "0001-add-work-email.sql").write_text("alter table crm.learners add column work_email varchar;\n")
waive(unclassified, "coverage/crm.learners.work_email", "2026-10-31", "Classified once the sign-up form is live.")
for line in ci.lines(ci.run(before, unclassified, checks=["coverage"])):
    print("unclassified,", line)

undescribed = ci.checkout(main)          # the second try: classified, with its description still to come
(undescribed / "migrations" / "0001-add-work-email.sql").write_text("alter table crm.learners add column work_email varchar;\n")
ci.edit(undescribed, "governance/metadata.toml", '15 = { name = "notes"',
        '16 = { name = "work_email", classification = "personal", description = "" }\n15 = { name = "notes"')
waive(undescribed, "contract-rules/crm.learners.work_email/described", "2027-06-30", "Described once the form's copy is final.")
after = ci.build(undescribed)
for line in ci.lines(ci.run(before, after, checks=["coverage", "contract-rules"])):
    print("undescribed, asked until 2027-06-30,", line)

# the same waiver, shortened to the end of October, within the longest a waiver may run
ci.edit(undescribed, "governance/waivers.toml", "2027-06-30", "2026-10-31")
for day in (TODAY, date(2026, 11, 1)):    # the book's today, then a month later, after the waiver has expired
    report = ci.run(before, after, today=day, checks=["coverage", "contract-rules"])
    print(f"undescribed, asked until 2026-10-31, on {day}, contract-rules:", "passed" if report.checks[1].passed else "failed")
    print(f"undescribed, asked until 2026-10-31, on {day}, findings waived:", sum(len(c.waived) for c in report.checks))
    print(f"undescribed, asked until 2026-10-31, on {day}, waivers that silence nothing:",
          "; ".join(why for _, why in report.unused) or "none")
```

Output:

```text
unclassified, coverage: failed, findings 3, new 1, on main 2, waived 0, 12 tables checked
unclassified, coverage/ai.review_corpus: no metadata; on main, owner team-platform
unclassified, coverage/crm.learners.work_email: not classified; its name suggests personal
unclassified, coverage/marketing.audience: no metadata; on main, owner team-platform
unclassified, change crm.learners, field 16: added as work_email string; named by 0 pipelines, reaches 0 columns
unclassified, waiver coverage/crm.learners.work_email: silences nothing, coverage findings are never waived
unclassified, job: failed
undescribed, asked until 2027-06-30, coverage: passed, findings 2, new 0, on main 2, waived 0, 12 tables checked
undescribed, asked until 2027-06-30, contract-rules: failed, findings 1, new 1, on main 0, waived 0, 10 contracts checked
undescribed, asked until 2027-06-30, coverage/ai.review_corpus: no metadata; on main, owner team-platform
undescribed, asked until 2027-06-30, coverage/marketing.audience: no metadata; on main, owner team-platform
undescribed, asked until 2027-06-30, contract-rules/crm.learners.work_email/described: no description (global rule described)
undescribed, asked until 2027-06-30, change crm.learners, field 16: added as work_email string; named by 0 pipelines, reaches 0 columns
undescribed, asked until 2027-06-30, waiver contract-rules/crm.learners.work_email/described: silences nothing, runs to 2027-06-30, past the longest allowed, 2026-12-30
undescribed, asked until 2027-06-30, job: failed
undescribed, asked until 2026-10-31, on 2026-10-01, contract-rules: passed
undescribed, asked until 2026-10-31, on 2026-10-01, findings waived: 1
undescribed, asked until 2026-10-31, on 2026-10-01, waivers that silence nothing: none
undescribed, asked until 2026-10-31, on 2026-11-01, contract-rules: failed
undescribed, asked until 2026-10-31, on 2026-11-01, findings waived: 0
undescribed, asked until 2026-10-31, on 2026-11-01, waivers that silence nothing: expired on 2026-10-31
```

The waiver for the unclassified column is refused, although it asks for only one month: `silences nothing, coverage findings are never waived`. No waiver can make that pull request pass. The waiver for the description is refused while it runs to the summer, `silences nothing, runs to 2027-06-30, past the longest allowed, 2026-12-30`. Shortened to Oct 31, it silences the finding. On Oct 1 the contract rules report `passed`. On Nov 1 they report `failed`. On Oct 1 the list of waivers that silence nothing is `none`; on Nov 1 it is `expired on 2026-10-31`: the date passed, and the problem didn't.

**Working rule:** Let a waiver name exactly one finding a pull request introduces, with an owning team, a reason and an end date at most a few months away, and list the checks no waiver may silence, such as a missing classification or owner. Report, for each check, how many findings nobody acted on, waived or left on main, and review any check where that's most of them: either the check is wrong or the platform is, and the count hides which.

## What goes wrong in production {#ci-in-production}

On our judgment, a governance job is switched off more often than it breaks: teams remove the checks that are slow or noisy, one line of `governance/ci.toml` at a time, and the gate goes on passing while it guards less. Two staged checks show how, with counts in place of timings, and a third shows a job that breaks.

### A check that reads every row {#a-check-that-reads-every-row}

The support team adds two columns to `support.tickets`: `callback_number`, classified `internal`, and `reply_email`, classified `public`. A team that wants values judged, not only names, then adds [chapter 3](#classify-columns-from-their-values)'s classification of the table to the job, by sample and by every row:

Example `slow-check-every-row` (Python 3.12.15 · duckdb 2.0.0 · openlineage-python 1.55.0 · phonenumbers 9.0.43 · presidio-analyzer 2.2.366 · pyarrow 26.0.1 · pyiceberg 0.12.0 · pyyaml 6.0.3 · regopy 1.5.2 · sqlglot 30.33.0 · tldextract 5.4.0), passing in Edition 2026.11.31:

```python
"""The support team's pull request adds two columns to support.tickets, which coverage judges by their names and chapter
3's value checks judge by reading a sample, or every row. The prose's point rests on the callback number's empty column
and on the values each check reads; counts of values stand in for time, which depends on the machine."""
import pyarrow as pa

from lib import ci, pii

main = ci.checkout()
branch = ci.checkout(main)
(branch / "migrations" / "0001-callback-and-reply.sql").write_text(
    "alter table support.tickets add column callback_number varchar;\n"
    "alter table support.tickets add column reply_email varchar;\n")
ci.edit(branch, "governance/metadata.toml", '8 = { name = "status"',
        '9 = { name = "callback_number", classification = "internal", description = "A number to call the learner back on." }\n'
        '10 = { name = "reply_email", classification = "public", description = "Where the reply goes." }\n8 = { name = "status"')
before, after = ci.build(main), ci.build(branch)

report = ci.run(before, after, checks=["coverage", "classify-sampled", "classify-every-row"])
for name in ("body", "callback_number", "reply_email"):
    values = after.catalog.load_table("support.tickets").scan(selected_fields=(name,)).to_arrow()[name]
    print(f"support.tickets.{name}, values on the branch:", f"{len(values) - values.null_count:,}")
    for check in report.checks:
        found = [f.message for f in check.findings if f.id == f"{check.name}/support.tickets.{name}"]
        print(f"support.tickets.{name}, {check.name}:", found[0] if found else "no finding")
for check in report.checks[1:]:
    print(f"{check.name}, values read:", f"{check.checked:,}")

text_columns, sampled, every_row = 0, 0, 0
for name in sorted(before.described):
    data = before.catalog.load_table(name).scan().to_arrow()
    # the non-null values of each text column, the ones a value check classifies
    filled = [len(column) - column.null_count for column in data.columns if pa.types.is_string(column.type)]
    text_columns += len(filled)
    # the sampled check reads at most pii.ROWS values of each column
    sampled += sum(min(pii.ROWS, count) for count in filled)
    every_row += sum(filled)
print("text columns in the described tables:", text_columns)
print("values the sampled check would read in them:", f"{sampled:,}")
print("values the every-row check would read in them:", f"{every_row:,}")
```

Output:

```text
support.tickets.body, values on the branch: 2,000
support.tickets.body, coverage: no finding
support.tickets.body, classify-sampled: values support sensitive, reviewed as personal
support.tickets.body, classify-every-row: values support sensitive, reviewed as personal
support.tickets.callback_number, values on the branch: 0
support.tickets.callback_number, coverage: no finding
support.tickets.callback_number, classify-sampled: no finding
support.tickets.callback_number, classify-every-row: no finding
support.tickets.reply_email, values on the branch: 0
support.tickets.reply_email, coverage: classified public; its name suggests personal; needs team-support's approval
support.tickets.reply_email, classify-sampled: its name suggests personal, reviewed as public
support.tickets.reply_email, classify-every-row: its name suggests personal, reviewed as public
classify-sampled, values read: 1,000
classify-every-row, values read: 10,000
text columns in the described tables: 32
values the sampled check would read in them: 5,800
values the every-row check would read in them: 429,453
```

Coverage fails `reply_email` by its name, `classified public; its name suggests personal; needs team-support's approval`, and passes `callback_number`, `no finding`, because no name rule matches it. The branch holds 0 values in the new column, so reading values doesn't help yet either. The migration adds the column empty, and every row gives `no finding`. [Chapter 3](#reclassify-on-schema-change)'s scheduled sample catches the phone numbers once the column fills.

The sampled check reads 1,000 values and the every-row check reads 10,000. Both find `support.tickets.body` sensitive under a personal classification, which chapter 3 already sent to `team-support`. Across every text column in the described tables, a sample would read 5,800 values, and every row would read 429,453. The second number grows with the data, on every pull request, whatever the pull request changed. A job that keeps people waiting, we judge, gets skipped, and then removed.

**Working rule:** Size pull-request checks by the change, not by the data: metadata, schemas, policies and the names of the columns a pull request adds. Run checks that read values on a schedule, with a fixed sample, as chapter 3 does, and send their findings to the owning team. Document where the job's checks stop, such as at a column added empty.

### A check that flags every ID {#a-check-that-flags-every-id}

A cautious team adds a check that treats every column named like an ID as personal data. The platform decided otherwise for the IDs it joins on: [chapter 2](#classify-pseudonymous-ids) classifies `learner_id` as `internal` and protects the table that resolves it. The check's findings are new, so each owning team waives them:

Example `noisy-check-waivers` (Python 3.12.15 · duckdb 2.0.0 · openlineage-python 1.55.0 · phonenumbers 9.0.43 · presidio-analyzer 2.2.366 · pyarrow 26.0.1 · pyiceberg 0.12.0 · pyyaml 6.0.3 · regopy 1.5.2 · sqlglot 30.33.0 · tldextract 5.4.0), passing in Edition 2026.11.31:

```python
"""A noisy check added to the job: every column named like an ID is personal. The platform decided otherwise for IDs
used for joining, so each owning team waives its findings, and the waiver count climbs. The prose's point rests on the
not-acted-on line of each check."""
from collections import Counter

from lib import ci

main = ci.checkout()
branch = ci.checkout(main)                          # the platform team adds the check to the job's settings
ci.edit(branch, "governance/ci.toml", '"properties"]', '"properties", "id-columns"]')
before, after = ci.build(main), ci.build(branch)

report = ci.run(before, after)
print("waivers in the file, before:", len(ci.waivers(after.governance)))
noisy = next(check for check in report.checks if check.name == "id-columns")
print("job, before the waivers:", "passed" if report.passed else "failed")
print("id-columns, findings:", len(noisy.findings))
print("id-columns, findings on main:", len(noisy.on_main))
# the last word of each message is the classification
classified = Counter(finding.message.rpartition(" ")[2] for finding in noisy.findings)
for level, count in sorted(classified.items()):
    print(f"id-columns, findings classified {level}:", count)
# the part of the finding's ID after the check's name: the column's full name
print("id-columns, the public one:", ", ".join(f.id.split("/")[1] for f in noisy.findings if f.message.endswith("public")))

with open(after.governance / "waivers.toml", "a", encoding="utf-8") as waivers:
    for finding in noisy.findings:
        waivers.write(f'\n[[waivers]]\nfinding = "{finding.id}"\nowner = "{finding.owner}"\n'
                      'reason = "An ID for joining, internal by decision."\nexpires = 2026-10-31\n')

report = ci.run(before, after)
print("waivers in the file, after:", len(ci.waivers(after.governance)))
print("job, with the waivers:", "passed" if report.passed else "failed")
for check in (check for check in report.checks if check.findings):
    idle = [f for f in check.findings if f.waiver or f.on_main]          # findings nobody acted on
    print(f"{check.name}, not acted on:", f"{len(idle)} of {len(check.findings)} findings, {len(idle) / len(check.findings):.0%}: "
          f"{len(check.waived)} waived, {len(check.on_main)} on main")
```

Output:

```text
waivers in the file, before: 0
job, before the waivers: failed
id-columns, findings: 21
id-columns, findings on main: 0
id-columns, findings classified internal: 20
id-columns, findings classified public: 1
id-columns, the public one: learning.courses.course_id
waivers in the file, after: 21
job, with the waivers: passed
coverage, not acted on: 2 of 2 findings, 100%: 0 waived, 2 on main
properties, not acted on: 4 of 4 findings, 100%: 0 waived, 4 on main
id-columns, not acted on: 21 of 21 findings, 100%: 21 waived, 0 on main
```

The check finds 21 columns, each an ID the platform classified on purpose. 20 of them are `internal`, and `learning.courses.course_id` is `public`.

The waivers file grows from 0 waivers to 21, and the ID check shows `21 of 21 findings, 100%: 21 waived, 0 on main`. That is past any threshold [Tricorder's authors use](#count-waived-findings), as an analogy only, since theirs count clicks, not waivers. The properties check shows the same share, `4 of 4 findings, 100%: 0 waived, 4 on main`, and it's right to: its findings are a fix the pipeline runner needs. The count can't tell the two apart, and the reasons can.

**Working rule:** Before adding a check, run it on the main branch and read every finding; if most would be waived, change the rule rather than waiving its findings. Remove a check whose findings stay mostly waived, and treat a high share on a correct check as work to schedule, with an owner and a date.

### A broken file becomes a finding {#a-broken-file-becomes-a-finding}

A support agent's entry goes into `governance/users.toml` with a quote missing. The same pull request changes `crm.learners.country` from text to a number, and it carries the CRM team's unclassified `work_email`:

Example `broken-files-become-findings` (Python 3.12.15 · duckdb 2.0.0 · openlineage-python 1.55.0 · phonenumbers 9.0.43 · presidio-analyzer 2.2.366 · pyarrow 26.0.1 · pyiceberg 0.12.0 · pyyaml 6.0.3 · regopy 1.5.2 · sqlglot 30.33.0 · tldextract 5.4.0), passing in Edition 2026.11.31:

```python
"""A pull request that breaks two files the job reads: a support agent added to the users file with a quote missing,
and a migration Iceberg refuses. It also carries the CRM team's unclassified work_email column. The prose's point rests
on the two checks that could not run and on the exit code."""
from lib import ci

main = ci.checkout()
branch = ci.checkout(main)
# the closing quote after "support" is missing, so the users file no longer parses
ci.edit(branch, "governance/users.toml", '[users.agent-a09]', '[users.agent-a12]\ngroups = ["support]\nregion = "emea"\n\n'
        '[users.agent-a09]')
(branch / "migrations" / "0001-add-work-email.sql").write_text("alter table crm.learners add column work_email varchar;\n")
# a string column cannot become an int: that isn't one of Iceberg's type promotions, so the migration is refused
(branch / "migrations" / "0002-country-as-number.sql").write_text("alter table crm.learners alter column country type int;\n")
print("files the pull request changes:", ", ".join(ci.changed(main, branch)))

report = ci.run(main, branch)
for line in ci.lines(report):
    print(line)
print("checks that could not run:", ", ".join(check.name for check in report.checks if check.error))
print("checks that ran:", sum(check.error is None for check in report.checks))
print("exit code:", report.exit_code)
```

Output:

```text
files the pull request changes: governance/users.toml, migrations/0001-add-work-email.sql, migrations/0002-country-as-number.sql
coverage: failed, findings 3, new 1, on main 2, waived 0, 12 tables checked
contract-rules: passed, findings 0, new 0, on main 0, waived 0, 10 contracts checked
contracts: passed, findings 0, new 0, on main 0, waived 0, 11 files checked
policy-tests: failed, the check could not run
enforcement-tests: failed, the check could not run
migrations: failed, findings 1, new 1, on main 0, waived 0, 2 migrations checked
pipelines: passed, findings 0, new 0, on main 0, waived 0, 4 pipelines checked
lineage-classifications: passed, findings 0, new 0, on main 0, waived 0, 23 columns checked
owner-review: passed, findings 0, new 0, on main 0, waived 0, 77 columns checked
removals: passed, findings 0, new 0, on main 0, waived 0, 87 columns checked
properties: passed, findings 4, new 0, on main 4, waived 0, 12 tables checked
coverage/ai.review_corpus: no metadata; on main, owner team-platform
coverage/crm.learners.work_email: not classified; its name suggests personal
coverage/marketing.audience: no metadata; on main, owner team-platform
policy-tests/error: the check could not run: governance/users.toml doesn't parse: Illegal character '\n' (at line 13, column 20)
enforcement-tests/error: the check could not run: governance/users.toml doesn't parse: Illegal character '\n' (at line 13, column 20)
migrations/0002-country-as-number.sql: refused: Cannot change column type: country: string -> int
properties/ai.review_corpus: 7 of 7 generated properties missing; on main, owner team-platform
properties/analytics.course_performance: 4 of 4 generated properties missing; on main, owner team-analytics
properties/analytics.learner_activity: 4 of 4 generated properties missing; on main, owner team-analytics
properties/marketing.audience: 7 of 7 generated properties missing; on main, owner team-platform
change crm.learners, field 16: added as work_email string; named by 0 pipelines, reaches 0 columns
job: failed
checks that could not run: policy-tests, enforcement-tests
checks that ran: 9
exit code: 1
```

The users file breaks the two checks that read it: `the check could not run: governance/users.toml doesn't parse: Illegal character '\n' (at line 13, column 20)`. Iceberg refuses the type change, which isn't a promotion, as `refused: Cannot change column type: country: string -> int`. The two checks that could not run don't stop the other checks, migrations among them, so the author also sees `not classified; its name suggests personal`, and the job exits with code 1. Without the catch, the job would stop at the first error, and the author would fix one problem per run.

**Working rule:** Turn a check that crashes into one finding that fails the job, with the error's first line and the file it names, run the other checks anyway, and never let a waiver silence it.

### Other failures {#other-ci-failures}

- **A gate nobody required.** A job whose status check branch protection doesn't require reports its findings, and the pull request merges anyway. Require it, and count merged pull requests whose job failed.
- **A broken build hides findings.** A table that a broken pipeline didn't write can't be found undescribed. Read the `pipelines` check first.
- **Waivers outlive their reason.** A finding gets fixed and its waiver stays, ready to hide the next one with the same ID. The report lists waivers that match no finding; delete them.
- **A rename changes finding IDs.** IDs name columns, so a renamed column's findings come back under new IDs, and the report lists their waivers as matching no finding. Move the waivers in the rename's pull request.
- **An approval outlives what it approved.** `team-crm` approves the downgrade of `city`, then the author pushes a downgrade of `email`, and the approval covers both. Have branch protection "dismiss stale pull request approvals when commits are pushed that affect the diff in the pull request".
- **One approval speaks for a team.** Any one code owner's approval meets GitHub's requirement, and branch protection's count is per branch. Count approvals per team in the job where a domain wants more.
- **The blocked team edits the gate.** A team whose pull request fails removes the check from `ci.toml`, adds a waiver, or edits CODEOWNERS. Give all three the platform team as code owner.
- **Changes outside pull requests.** A notebook creates a table, or a migration runs by hand. Run the job on a schedule against production too, as [chapter 2](#test-the-catalog-against-the-metadata) runs `problems`.

## What this chapter decided {#ci-decisions}

- **One job runs the checks of chapters 2 to 7 on every pull request,** `lib.ci.run`, listed in `governance/ci.toml`; only chapter 3's sampling of values runs on a schedule instead. `lib.ci.main` returns the exit code and writes the report as Markdown.
- **The job judges what the pull request changes.** A finding it introduces fails it unless approved or waived; a finding already on main is reported with its owner and fails nothing. A check that crashes is one failing finding.
- **Every finding has a stable ID,** the check and its subject, such as `coverage/crm.learners.work_email`.
- **A pull request is compared with the main branch on scratch platforms** built the way the platform runs, each with its own migrations and pipelines.
- **Schema changes are diffed by field ID** and reported three ways: Iceberg readers, the pipelines whose SQL names the column, and every column lineage says it reaches, INDIRECT edges included.
- **Two rules of our own:** lowering a classification, changing an owner or moving a table to another domain needs the approval of the table's owner on the main branch, counted per team; and a column is removed only after the main branch has marked it `deprecated`.
- **Ownership is an ODCS contract per table,** with one domain and the owning team in the role `owner`, generated into a folder per domain with a CODEOWNERS line, and checked by the job. The platform team owns CODEOWNERS, the job's settings, its global rules, the waivers and the migrations.
- **Global rules and domain rules are Rego,** run in-process with regopy on every contract, outside the access policy's folder.
- **The planted gaps stay on main:** `marketing.audience` and `ai.review_corpus` appear in every report as coverage findings, with the platform team as their owner, until they're described.
- **Noise is measured as findings nobody acts on, per check: waived, or left on main.** A check whose findings are mostly ignored is either wrong or the platform is, and the count alone can't say which. A waiver names one finding, an owner, a reason and an end date at most 90 days away, and coverage and the enforcement tests are never waived.
- **Pull-request checks are sized by the change;** checks that read values run on a schedule. The checks are only useful while people keep them on, so each one is sized to the change.
