---
title: "Data Governance as Code"
book: data-governance-as-code
edition: 2026.10.1
verified: 2026-10-23
edition_code: 6281-4K4S
canonical: https://rollingedition.com/data-governance-as-code
license: CC BY 4.0
attribution: "Data Governance as Code, Edition 2026.10.1, examples passed Oct 23, 2026. By Rolling Edition, https://rollingedition.com/data-governance-as-code. Text CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Code MIT, copyright Rolling Edition and contributors. Written with AI. Verified by people."
---
Edition 2026.10.1. Examples passed Oct 23, 2026. 108 of 108 examples pass. Written with AI. Verified by people.
To cite a section, name Edition 2026.10.1 and link to the section's permanent link: https://rollingedition.com/data-governance-as-code/s/{section_id}
# Governance end to end
By the end of this chapter you can govern one table of a data platform with code, end to end. You'll catalog it with an owning team and a classification for every column, capture the lineage of a pipeline that copies it, decide who may read which of its rows and enforce that decision on a query, and see a pull request fail because it added a column nobody classified. Each piece is small and runs in seconds, and five later chapters each make one of them solid.
This book is for data, analytics and platform engineers who already build pipelines and warehouses and have been asked to make them governed. You're comfortable with SQL, Python and Parquet or a lakehouse table, and you don't need governance theory: you need governance that runs, is tested, and is reviewed in pull requests like the rest of your code. By the end of the book, you can govern a data platform with code that runs in CI:
- a catalog where every table has an owner and every column a classification;
- personal data found automatically, with what the detector misses measured;
- lineage captured from pipelines;
- access policies as tested code, enforced as row filters and column masks;
- retention and erasure that remove data;
- an audit trail;
- checks that fail a pull request;
- the same rules applied to data that trains or grounds AI.
This book follows one case throughout: an online learning platform that sells professional courses to adult learners, which we call the platform. Its warehouse is a catalog of Iceberg tables, each under the namespace of the team that runs it: learners and instructors under `crm`, courses, enrollments, learning events and course reviews under `learning`, payments under `billing`, and support tickets under `support`. SQL pipelines build reporting tables from them. Support agents, instructors, marketing, finance, data scientists and the platform's course assistant, its AI, each need something different from those tables, and that's what makes the platform worth governing.
The platform's data, which every example in this book reads, is generated. A script makes it once from fixed seeds with Faker, a library for fake data, and the files are committed with a data card that says how they were made. No real person is in the data. Its email addresses are on the domains reserved for examples, so they can't reach anyone, and its IP addresses are in the ranges reserved for documentation, so they point at no one. Its phone numbers, card numbers and IBANs are random in a valid format, so one may match a number or an account in service: never call, message, charge or pay them. The card also lists what is planted for the examples to find, such as full card numbers in a legacy column and personal data in free text.
Every example runs offline, in one Python 3.12.15 process, on pinned libraries: PyIceberg 0.12.0 for the Iceberg catalog and its tables, DuckDB 2.0.0 for queries, sqlglot 30.24.0 to parse SQL, openlineage-python 1.54.0 for lineage, and regopy 1.5.2 for policies written in Rego, the policy language of Open Policy Agent (OPA). Six services that a production platform runs don't run here, and each has a stand-in:
- **Trino**, the SQL engine every tool would query through: DuckDB, behind the book's enforcement layer, which rewrites each query before DuckDB runs it.
- **OPA's server**, which Trino asks for access decisions ([Trino 484, OPA access control](https://trino.io/docs/484/security/opa-access-control.html)): regopy, evaluating the same Rego inside the Python process.
- **Spark**, or whatever engine runs the pipelines at scale: DuckDB, running each pipeline's SQL file.
- **A catalog server or a cloud catalog**, serving table metadata through an API such as the Iceberg REST catalog's ([REST catalog spec, 1.12.0](https://github.com/apache/iceberg/blob/apache-iceberg-1.12.0/open-api/rest-catalog-open-api.yaml)): PyIceberg's SQL catalog, where a SQLite file records where each table's metadata is, and the metadata files sit beside the data on the local disk.
- **A lineage server**: a file of OpenLineage events, which the book's code reads back into a graph.
- **The CI service and git**: two folders, one for the main branch and one for a pull request's branch.
The examples teach the open standards those services work with, Iceberg's table metadata, OpenLineage and Rego, so what you build maps onto them.
On the platform, governance starts with a spreadsheet. Someone listed the columns of `crm.learners`, the table that holds every learner's name, email, phone number and address, with a classification beside each, and that's all there is. No team is named as the table's owner. A nightly pipeline copies learners' names and emails into a table for marketing, and nothing records that it does. Support agents each serve one region, but a query on the warehouse returns every region's learners. And the CRM team is about to add a column. This chapter answers each of those with the code that the later chapters build, at the smallest scale that works: one table, one pipeline, one rule and one check. Only one of the four is a policy engine; the others are metadata, lineage events and a test, kept in the same repository and reviewed the same way.
Five later chapters each make one piece solid. [Chapter 2](#a-catalog-you-can-test) keeps the catalog in step with schema changes and measures a local model's drafted metadata. [Chapter 4](#lineage-from-pipelines) uses lineage for impact analysis and to propose classifications. [Chapter 5](#access-policies-as-code) writes the whole policy, with its tests, and [chapter 6](#enforcing-policies-on-queries) enforces it on every query. [Chapter 10](#governance-in-ci) runs the checks of chapters 2 to 7 in CI, with owners' approvals and waivers. Four chapters add what this one leaves out: finding the personal data the metadata misses ([chapter 3](#finding-personal-data)), retention, consent and erasure ([chapter 7](#retention-consent-erasure)), an audit trail ([chapter 8](#audit-and-evidence)), and the same rules for data that trains or grounds AI ([chapter 9](#governing-data-for-ai)). The examples use the book's shared modules, files in `examples/lib/` that hold the code many examples need, which you can open and read; each section names the ones it uses.
## Catalog a table with its owner and classifications {#catalog-a-table}
The platform's first problem is its spreadsheet: a list of columns that nothing checks. A catalog lists a platform's tables and their schemas, and governance needs two facts the data doesn't carry: who answers for each table, and how each column must be protected. Kept in a spreadsheet or a catalog server's web page, those facts change without review, and no test can say whether they're complete. Kept in a file in the repository, beside the pipelines that create the tables, every change to them is a diff someone approves, and a test can compare the file with the catalog.
The platform's file is `governance/metadata.toml`. It gives each table an **owner**, a **domain**, the part of the business the table belongs to, and a description, and each column a description and a **classification**. Catalog tools mean different things by an owner, as [chapter 2](#keep-metadata-in-a-reviewed-file) compares, and we use one meaning throughout: the team accountable for the table, never a person, so the answer survives someone leaving. A classification is one of four values, from least to most protected:
- `public`: may be shown to anyone outside the platform;
- `internal`: for the platform's staff, and doesn't identify a person by itself;
- `personal`: identifies a person, or tells something about one;
- `sensitive`: data whose leak would directly harm someone, such as a card number.
Columns are keyed by their Iceberg field ID, which survives a rename, with the name beside it for people:
```toml
[tables."crm.learners"]
owner = "team-crm"
domain = "crm"
description = "Every learner who has signed up, with their contact details and consents. One row per learner."
[tables."crm.learners".columns]
3 = { name = "email", classification = "personal", description = "The email the learner signs in with." }
```
Three of the book's shared modules do the work. `lib.catalog.open_catalog` builds a fresh Iceberg catalog of the platform's tables in a temporary folder, which it removes when the example exits. `lib.metadata` loads the file, writes it into the catalog with `apply`, and checks the catalog against it with `problems`, which walks every table the catalog holds and reports a table with no metadata or no owner, and a column with no classification. Catalog `crm.learners`:
Example `catalog-the-learners` (Python 3.12.15 · pyarrow 26.0.0 · pyiceberg 0.12.0), passing in Edition 2026.10.1:
```python
"""Catalog crm.learners: its owner, domain and classifications come from governance/metadata.toml, the reviewed file."""
from collections import Counter
from lib import metadata
from lib.catalog import open_catalog
platform = open_catalog(["crm.learners"]) # a fresh Iceberg catalog holding the learners table
described = metadata.load()
learners = described["crm.learners"]
print("owner:", learners.owner)
print("domain:", learners.domain)
counts = Counter(column.classification for column in learners.columns.values())
print("columns classified personal:", counts["personal"])
print("personal columns:", ", ".join(c.name for c in learners.columns.values() if c.classification == "personal"))
print("columns classified internal:", counts["internal"])
# Write the descriptions into the table's schema, and the owner and domain into its properties.
metadata.apply(platform, described)
table = platform.load_table("crm.learners")
print("table properties:", ", ".join(f"{key}={value}" for key, value in sorted(table.properties.items())))
print("email's description, in the Iceberg schema:", table.schema().find_field("email").doc)
# The check: every table in the catalog has an owner, and every column one of the four classifications.
print("problems the check finds:", len(metadata.problems(platform, described)))
```
Output:
```text
owner: team-crm
domain: crm
columns classified personal: 7
personal columns: full_name, email, phone, city, street_address, date_of_birth, notes
columns classified internal: 8
table properties: domain=crm, owner=team-crm
email's description, in the Iceberg schema: The email the learner signs in with.
problems the check finds: 0
```
The table's owner is `team-crm`, in the domain `crm`. Of its columns, 7 are `personal`: `full_name, email, phone, city, street_address, date_of_birth, notes`. `notes` is among them although its name says nothing, because support agents type whatever they like into it. The other 8 are `internal`: the learner's ID, two dates, their country, their two consent flags, their account's status and their support region. After `apply`, the table's properties read `domain=crm, owner=team-crm`, and `email`'s description sits in the Iceberg schema, where any reader of the table finds it: "The email the learner signs in with." The classifications stay in the file, because that is where the access policy and the checks read them. And `problems` finds 0 problems: the catalog's one table has an owner, and every column one of the four classifications.
That zero is what a spreadsheet can't give you. It comes from walking the catalog, so a column the file doesn't describe would show up as a problem, as `work_email` does in [the section on spreadsheets](#a-spreadsheet-nobody-updates).
**Working rule:** Keep owners, domains, descriptions and classifications in one file in the repository, reviewed like the pipelines, with a team as every table's owner and one of a closed list of classifications on every column. Write the descriptions into the catalog, so every reader of a table sees them, and test the catalog against the file by walking the catalog, never the file. If your organization edits metadata in a catalog server's web page, make one side the source and generate the other from it.
## Capture one pipeline's column lineage {#capture-one-pipelines-lineage}
[The catalog](#catalog-a-table) says what a table holds. Lineage says where its data came from and where it went, and governance needs both: a table built from `crm.learners` may or may not copy its emails. OpenLineage, an open specification for lineage, describes a **job**, such as a pipeline, its **runs**, and the **datasets** each run reads and writes. A run reports itself in events: a START when it begins, then one that ends it, such as COMPLETE or FAIL. Events carry **facets**, OpenLineage's name for blocks of metadata attached to a run, a job or a dataset.
The column lineage facet, on the output of a COMPLETE event, lists for each output column the input columns it came from, each with a type. DIRECT means the "output column value was somehow derived from inputField value", and INDIRECT that the "output column value is impacted by the value of inputField column, but it's not derived from it" ([column lineage facet](https://openlineage.io/docs/spec/facets/dataset-facets/column_lineage_facet)): a join or a filter deciding which rows exist, or a grouping, sort, window or condition.
The client library computes no lineage of its own, as [chapter 4](#send-run-events) finds in its source: it sends what you give it. So the book's module `lib.lineage` parses each pipeline's SQL with sqlglot, adds the INDIRECT inputs that sqlglot's own lineage leaves out, and wraps `lib.pipelines.run`, which runs one of the platform's SQL pipelines on DuckDB and writes its output to the catalog: START before the run, COMPLETE with the lineage after it. The events go through OpenLineage's file transport to one file, stamped with a fixed time and a run ID derived from that time and the job's name, so the example writes the same bytes every time.
The platform's `marketing_audience` pipeline is one `create table ... as select ...` statement that writes `marketing.audience`, the list of learners marketing may email. It selects each learner's ID, name, email and country from `crm.learners`, joins each learner to their favourite course category, ranked from their enrollments, and keeps the learners for whom `l.marketing_consent and l.status = 'active'` holds. Run it with lineage, read the lineage back from the events alone, as a lineage server would, and count what the table holds with `lib.warehouse.connect`, which gives DuckDB a view of each catalog table:
Example `audience-pipeline-lineage` (Python 3.12.15 · duckdb 2.0.0 · openlineage-python 1.54.0 · pyarrow 26.0.0 · pyiceberg 0.12.0 · pyyaml 6.0.3 · sqlglot 30.24.0), passing in Edition 2026.10.1:
```python
"""Run the marketing_audience pipeline with OpenLineage run events, then read its column lineage back from the events."""
from lib import lineage, metadata, warehouse
from lib.catalog import open_catalog
from lib.platform import scratch
platform = open_catalog(["crm.learners", "learning.courses", "learning.enrollments"])
events_file = scratch() / "events.jsonl"
lineage.run(platform, "marketing_audience", lineage.client(events_file))
events = lineage.read_events(events_file)
print("events:", ", ".join(f"{e['eventType']} of {e['job']['name']}" for e in events))
print("the run read:", ", ".join(dataset["name"] for dataset in events[0]["inputs"]))
print("the run wrote:", events[-1]["outputs"][0]["name"])
graph = lineage.graph(events)
for column in ("full_name", "email"):
inputs = [e for e in graph.edges if e.target == f"marketing.audience.{column}"]
print(f"marketing.audience.{column} comes from:", ", ".join(f"{e.source} ({e.type} {e.subtype})" for e in inputs))
for subtype in ("FILTER", "JOIN"):
shaping = [e.source for e in graph.edges if e.target == "marketing.audience" and e.subtype == subtype]
print(f"its rows are shaped by, INDIRECT {subtype}:", ", ".join(shaping))
duckdb = warehouse.connect(platform) # DuckDB on the catalog's tables, to count what the join dropped
consenting = "select count(*) from crm.learners where marketing_consent and status = 'active'"
print("consenting, active learners in crm.learners:", f"{duckdb.sql(consenting).fetchall()[0][0]:,}")
print("rows in marketing.audience:", f"{duckdb.sql('select count(*) from marketing.audience').fetchall()[0][0]:,}")
reached = lineage.columns_reached(graph, lineage.downstream(graph, "crm.learners.email"))
print("a change to crm.learners.email reaches:", ", ".join(reached))
# The catalog check, run on the catalog the pipeline wrote to.
for problem in metadata.problems(platform, metadata.load()):
print("the catalog check finds:", problem)
```
Output:
```text
events: START of marketing_audience, COMPLETE of marketing_audience
the run read: crm.learners, learning.courses, learning.enrollments
the run wrote: marketing.audience
marketing.audience.full_name comes from: crm.learners.full_name (DIRECT IDENTITY)
marketing.audience.email comes from: crm.learners.email (DIRECT IDENTITY)
its rows are shaped by, INDIRECT FILTER: crm.learners.marketing_consent, crm.learners.status
its rows are shaped by, INDIRECT JOIN: crm.learners.learner_id, learning.courses.category, learning.courses.course_id, learning.enrollments.course_id, learning.enrollments.learner_id
consenting, active learners in crm.learners: 1,861
rows in marketing.audience: 1,829
a change to crm.learners.email reaches: marketing.audience.email
the catalog check finds: marketing.audience: no metadata
```
The file holds two events, `START of marketing_audience, COMPLETE of marketing_audience`. The run read `crm.learners, learning.courses, learning.enrollments` and wrote `marketing.audience`. Column by column, the lineage says what the table's name doesn't: `email` comes from `crm.learners.email (DIRECT IDENTITY)`, and `full_name` from `crm.learners.full_name (DIRECT IDENTITY)`, both unchanged copies of personal columns.
Two kinds of INDIRECT input shape the rows. The filters, `crm.learners.marketing_consent, crm.learners.status`, mean that every row of the table says its learner consented. The joins, on `crm.learners.learner_id, learning.courses.category, learning.courses.course_id, learning.enrollments.course_id, learning.enrollments.learner_id`, decide which rows exist too: a learner with no enrollment has no favourite category, so the join drops them. `crm.learners` holds 1,861 consenting, active learners, and `marketing.audience` 1,829. Walking the graph from `crm.learners.email` reaches `marketing.audience.email`, so renaming the email column, or changing its format, reaches one column of this pipeline's table. With all four of the platform's pipelines sending lineage, [chapter 4](#impact-of-a-column-change) finds that it reaches no other.
The catalog check from [the section before](#catalog-a-table), run on the same catalog, finds `marketing.audience: no metadata`. The pipeline created a table that nobody has described, and lineage turns that gap from "a table we know nothing about" into "a copy of two personal columns, chosen by consent". Until someone describes it, the platform's access policy treats every column it can't find a classification for as `sensitive`. So anyone who reads the copy through the platform's enforcement layer, which [a later section](#enforce-one-access-decision) introduces, gets no value from it, as [chapter 6's who-sees-what test](#test-who-sees-what) shows.
**Working rule:** Send a START and then a COMPLETE or FAIL event from the code that runs each job, so no run happens without them, and put the output's column lineage in the COMPLETE event, parsed from the job's SQL, with the inputs that join and filter rows as well as those that supply values. Read lineage back from the events, never from the SQL, so the graph shows what ran. Where a job isn't SQL that a parser can read, such as a Python script, write its lineage in the same code, and review the two together.
## Enforce one access decision on a query {#enforce-one-access-decision}
With the platform's tables [cataloged](#catalog-a-table), the next question is who may read what. Policy as code starts from OPA's premise: it "lets you decouple policy from that software service so that the people responsible for policy can read, write, analyze, version, distribute, and in general manage policy separate from the service itself" ([OPA philosophy](https://www.openpolicyagent.org/docs/philosophy)). A **policy** answers a request with a **decision**, and the platform's policy answers with a document rather than a yes or no: whether to allow the read, the **row filters** that keep only the rows the user may see, and the **column masks** that change how a column is shown. The filters and masks are data, never SQL. A decision does nothing on its own, though. Something between the user and the engine has to apply it to every table a query reads.
On the platform, support agents each serve one region, `americas`, `emea` or `apac`, and need learners' contact details to answer tickets. The policy, `governance/policies/access.rego`, denies every request by default, allows each group, the role a user holds in `governance/users.toml`, the tables its work needs, and reads the agent's region from the same file. This is the rule that keeps an agent to their region:
```rego
row_filters contains {"column": "support_region", "equals": user.region} if {
"support" in groups
table in {"crm.learners", "support.tickets"}
}
```
`lib.policy.decide` evaluates the policy with regopy, and `lib.enforce` holds the **enforcement layer**: its `Enforcer` asks the policy about every table a query reads, replaces each with a subquery that applies the decision, and runs the rewrite on DuckDB. `lib.enforce.answer` is the one function a service calls, and returns rows, never the engine's connection. `agent-a05`, a support agent in EMEA, asks for three columns of `crm.learners`, then counts learners by region, and reads their emails, which the example compares with the platform's data as `lib.platform.load` reads it from the generated files:
Example `support-region-enforced` (Python 3.12.15 · duckdb 2.0.0 · pyarrow 26.0.0 · pyiceberg 0.12.0 · regopy 1.5.2 · sqlglot 30.24.0), passing in Edition 2026.10.1:
```python
"""One access decision from the platform's Rego policy, and the same decision enforced on one query."""
import json
from lib import policy
from lib.catalog import open_catalog
from lib.enforce import Enforcer, answer
from lib.platform import load
# agent-a05, a support agent in EMEA, asks to read three columns of crm.learners.
request = policy.request("agent-a05", "crm.learners", ["learner_id", "email", "support_region"])
decision = policy.decide(request)
print("allow:", decision["allow"])
print("row filters:", json.dumps(decision["row_filters"]))
print("column masks:", json.dumps(decision["column_masks"]))
print("reasons:", "; ".join(decision["reasons"]))
# The enforcement layer asks the policy about every table a query reads, then rewrites the query to apply the answer.
layer = Enforcer(open_catalog(["crm.learners"]))
query = "select support_region, count(*) as learners from crm.learners group by support_region order by support_region"
for region, learners in answer(layer, query, "agent-a05"):
print(f"learners agent-a05 sees in {region}:", f"{learners:,}")
# Compare the emails the agent gets with the platform's data, without printing any.
stored = load("crm.learners")
emails = dict(zip(stored.column("learner_id").to_pylist(), stored.column("email").to_pylist()))
seen = answer(layer, "select learner_id, email from crm.learners", "agent-a05")
print("emails agent-a05 sees, as stored:", f"{sum(email == emails[learner] for learner, email in seen):,} of {len(seen):,}")
print("learners in the platform's data:", f"{stored.num_rows:,}")
```
Output:
```text
allow: True
row filters: [{"column": "support_region", "equals": "emea"}]
column masks: {}
reasons: support may read crm.learners
learners agent-a05 sees in emea: 1,581
emails agent-a05 sees, as stored: 1,581 of 1,581
learners in the platform's data: 5,000
```
The decision allows the read, `allow: True`, with one row filter, `[{"column": "support_region", "equals": "emea"}]`, and no column masks, `{}`, since support is the group that needs contact details. Its reason says which group allowed it: `support may read crm.learners`. The agent's query has no filter on region, yet it returns 1,581 learners, all in EMEA, of the 5,000 the platform holds. Their emails come back as stored, 1,581 of 1,581, because the decision masks nothing. The layer applied the region inside the query, so the agent's SQL didn't have to, and couldn't avoid it.
regopy is the Python binding of rego-cpp, Microsoft's C++ implementation of Rego, which supports "v1.18.1 of Rego as defined by OPA" ([rego-cpp README at v1.5.2](https://github.com/microsoft/rego-cpp/blob/v1.5.2/README.md)). regopy differs from OPA in ways [chapter 5](#where-regopy-differs-from-opa) lists, so OPA stays the reference for anything in production. [Chapter 6](#column-masks-leak) shows why the layer replaces whole tables rather than masking columns where a query names them.
**Working rule:** Write access rules as one policy in the repository that denies by default and returns its decision as data: allow, row filters and column masks. Apply that decision in one layer that rewrites every query before the engine runs it, and give users that layer's answers, never the engine's connection. When an engine can ask the policy itself, as Trino can ask OPA, let the engine be that layer.
## Fail the pull request that adds an unclassified column {#fail-the-pull-request}
[The catalog check](#catalog-a-table) proves the catalog complete on the day it runs, and the platform changes every week. A check that runs when someone remembers it is a report, and reports go unread. Run on every pull request, where its exit code can fail the pull request, the same check becomes a gate, and its findings appear where engineers already work. The authors of Tricorder, Google's platform for code analysis, wrote: "We have repeatedly found that when developers have to navigate to a dashboard or run a standalone command line tool, analysis usage drops off" ([Sadowski et al., 2015](https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/43322.pdf)). That is one company's experience with code analysis, not data, but governance checks meet the same people. A gate also has to judge what the pull request changes, not everything on the platform: a finding already on the main branch is reported with the team that owns it, and fails no pull request that leaves it as it is.
The book's module for this is `lib.ci`, the platform's governance job. On the platform, a schema change is a migration: an `alter table` statement in a file in the branch's `migrations/` folder. The examples hold each side of a pull request in a folder, a checkout: one of the main branch and one of the pull request's branch. For each side, `lib.ci.run` builds a scratch platform, a fresh catalog made the way the platform runs: the source tables, the side's migrations, its metadata written into the catalog, and its pipelines. It runs the checks it's given on the branch's platform, runs them on main's alone to learn what is already there, and compares the findings by ID.
This section runs one check, `coverage`. It is [the catalog check](#catalog-a-table), `lib.metadata.problems`, run on the branch's platform, and for each column nobody classified it adds what a rule on column names suggests, from the rules [chapter 3](#classify-columns-from-the-schema) writes. The CRM team's pull request adds a column for learners' work email, in one migration, and forgets the column's metadata:
Example `unclassified-column-fails` (Python 3.12.15 · duckdb 2.0.0 · openlineage-python 1.54.0 · phonenumbers 9.0.41 · presidio-analyzer 2.2.364 · pyarrow 26.0.0 · pyiceberg 0.12.0 · pyyaml 6.0.3 · regopy 1.5.2 · sqlglot 30.24.0 · tldextract 5.4.0), passing in Edition 2026.10.1:
```python
"""The CRM team's pull request adds a column to crm.learners and no metadata for it. One check runs on it: coverage."""
from lib import ci
main = ci.checkout() # the main branch: the governance files and pipelines, and no migrations yet
branch = ci.checkout(main) # the pull request's branch, starting from main
(branch / "migrations" / "0001-add-work-email.sql").write_text("alter table crm.learners add column work_email varchar;\n")
print("files the pull request changes:", ", ".join(ci.changed(main, branch)))
# Build a scratch platform from each side, run the coverage check on both, and compare their findings by ID.
report = ci.run(main, branch, checks=["coverage"])
for line in ci.lines(report):
if not line.startswith("change "): # schema changes are chapter 10's report
print(line)
for line in ci.totals(report):
if line.startswith(("findings the pull request introduces", "findings already on main", "exit code")):
print(line)
```
Output:
```text
files the pull request changes: migrations/0001-add-work-email.sql
coverage: failed, findings 3, new 1, on main 2, waived 0, 12 tables checked
coverage/ai.review_corpus: no metadata; on main, owner team-platform
coverage/crm.learners.work_email: not classified; its name suggests personal
coverage/marketing.audience: no metadata; on main, owner team-platform
job: failed
findings the pull request introduces: 1
findings already on main: 2
exit code: 1
```
The pull request changes one file, `migrations/0001-add-work-email.sql`. The coverage check reports `failed, findings 3, new 1, on main 2, waived 0, 12 tables checked`: the tables it checked are the platform's eight source tables and the four its pipelines write. Its new finding is the pull request's own: `crm.learners.work_email` is `not classified; its name suggests personal`. The name rule matches `email`, so the finding tells the author what to add, and its ID, `coverage/crm.learners.work_email`, stays the same from run to run, so it can be discussed and counted.
The other two findings were there before the pull request: `ai.review_corpus`, and `marketing.audience`, the table [the lineage section](#capture-one-pipelines-lineage) found undescribed, reported as `no metadata; on main, owner team-platform`. Nobody owns either table, so the job names the platform team, which the job's settings make answerable for tables without an owner. Of the findings, 1 is the pull request's own and 2 are already on main, and the job exits with code 1. The pull request fails until the column has a classification, and the two tables on main fail no pull request that leaves them as they are. [Chapter 10's job](#run-every-rule-as-one-job) runs the checks of chapters 2 to 7 on the same pull request, 11 of them.
**Working rule:** Run every governance check as one job on every pull request, required by branch protection, and fail it by exit code on any finding the pull request introduces. Report the findings already on the main branch with their owners, without failing the pull request for them. Put every finding in the pull request itself, under an ID that stays the same from run to run, with what to do about it. Run the same checks on a schedule against the production catalog too, for what changes outside pull requests.
## What goes wrong in production {#end-to-end-in-production}
Governance fails without an error in two ways that the pieces above answer: [a spreadsheet nobody updates](#a-spreadsheet-nobody-updates), and [a policy that another tool bypasses](#a-policy-another-tool-bypasses). The first is why the metadata is code, and the second why enforcement sits in one place.
### A governance spreadsheet nobody updates {#a-spreadsheet-nobody-updates}
[The platform's governance started](#governance-end-to-end) with a spreadsheet of tables and columns, with a classification beside each, and a spreadsheet describes the schema on the day it was written. Schemas don't hold still. A study of ten open-source database applications, with more than 160,000 revisions between them, lists first among its major findings that "Database schemas evolve frequently during the application lifecycle" ([Qiu, Li and Su, 2013](https://www.cs.ucdavis.edu/~su/publications/fse13-db-study.pdf)). The study is of application databases, not warehouses, but every change that adds a column to a table adds one its spreadsheet doesn't list.
Take the platform's spreadsheet, an inventory of `crm.learners` exported as CSV from the catalog and the metadata, one row per column. Then the CRM team's migration from [the pull request above](#fail-the-pull-request) adds `work_email`. Compare what the spreadsheet says with what the check against the catalog finds:
Example `spreadsheet-misses-new-column` (Python 3.12.15 · duckdb 2.0.0 · openlineage-python 1.54.0 · phonenumbers 9.0.41 · presidio-analyzer 2.2.364 · pyarrow 26.0.0 · pyiceberg 0.12.0 · pyyaml 6.0.3 · regopy 1.5.2 · sqlglot 30.24.0 · tldextract 5.4.0), passing in Edition 2026.10.1:
```python
"""A governance inventory kept as a spreadsheet, written before a schema change, against the check on the live schema."""
import csv
import io
from lib import ci, metadata
from lib.catalog import open_catalog
platform = open_catalog(["crm.learners"])
described = metadata.load()
# The inventory: one row per column of crm.learners, exported on the day someone was asked for one.
inventory = io.StringIO()
writer = csv.writer(inventory)
writer.writerow(["table", "column", "classification"])
for field in platform.load_table("crm.learners").schema().fields:
writer.writerow(["crm.learners", field.name, described["crm.learners"].columns[field.field_id].classification])
# Later, the CRM team's migration adds a column for learners' work email.
ci.migrate(platform, "alter table crm.learners add column work_email varchar;")
rows = list(csv.DictReader(io.StringIO(inventory.getvalue())))
print("columns the spreadsheet lists:", len(rows))
print("of them without a classification, by the spreadsheet:", sum(not row["classification"] for row in rows))
live = [field.name for field in platform.load_table("crm.learners").schema().fields]
print("columns crm.learners has:", len(live))
print("columns the spreadsheet doesn't list:", ", ".join(sorted(set(live) - {row["column"] for row in rows})))
for problem in metadata.problems(platform, described):
print("the check against the catalog finds:", problem)
```
Output:
```text
columns the spreadsheet lists: 15
of them without a classification, by the spreadsheet: 0
columns crm.learners has: 16
columns the spreadsheet doesn't list: work_email
the check against the catalog finds: crm.learners.work_email: not classified
```
By its own account, the spreadsheet is complete: it lists 15 columns, and 0 lack a classification. The table has 16, and the one it doesn't list is `work_email`. Nothing in the spreadsheet can show that, because nothing compares it with the catalog. `lib.metadata.problems`, [the catalog check](#catalog-a-table), walks the catalog and finds `crm.learners.work_email: not classified`: the same finding that fails [the CRM team's pull request](#fail-the-pull-request). As a rule of thumb, a copy of the metadata kept by hand, whether a spreadsheet, a wiki page or a data dictionary, goes stale on its first schema change, and nothing says so.
**Working rule:** Never keep governance metadata in a document that nothing checks. Keep it in a reviewed file that a test compares with the live catalog, on every pull request and on a schedule. If people need a spreadsheet, generate it from the file, and say on it when it was generated.
### A policy one tool enforces and another bypasses {#a-policy-another-tool-bypasses}
A policy protects only the reads that pass through whatever enforces it. Trino applies row filters only when it's told where to fetch them: its `opa.policy.row-filters-uri` setting is "The optional URI for fetching row filters - if not set no row filtering is applied" ([Trino 484, OPA access control](https://trino.io/docs/484/security/opa-access-control.html)). Iceberg's REST catalog specification, at 1.12.0, lets a catalog return read restrictions with a table, and says that "A reader that supports read restrictions must fail any read against the loaded table that cannot apply a returned restriction in full" ([REST catalog spec, 1.12.0](https://github.com/apache/iceberg/blob/apache-iceberg-1.12.0/open-api/rest-catalog-open-api.yaml)). The spec gives a reader no way to say that it supports them, as it does for delegated access to storage, with the `X-Iceberg-Access-Delegation` header: an "Optional signal to the server that the client supports delegated access". So a catalog can't tell which readers will apply its restrictions. PyIceberg 0.12.0's REST client reads none ([source](https://github.com/apache/iceberg-python/blob/pyiceberg-0.12.0/pyiceberg/catalog/rest/__init__.py)). We recommend that a catalog that returns read restrictions give storage access only to engines that apply them.
On the platform, the [enforcement layer](#enforce-one-access-decision) keeps `agent-a05` to EMEA. Read the same table around it twice: as a notebook would, with DuckDB on the same catalog through `lib.warehouse.connect`, which the layer itself uses; and as any reader holding the catalog's access would, with PyIceberg's own scan:
Example `read-around-the-layer` (Python 3.12.15 · duckdb 2.0.0 · pyarrow 26.0.0 · pyiceberg 0.12.0 · regopy 1.5.2 · sqlglot 30.24.0), passing in Edition 2026.10.1:
```python
"""The same table, read three ways: through the enforcement layer, and around it with DuckDB and with PyIceberg."""
from collections import Counter
from lib import warehouse
from lib.catalog import open_catalog
from lib.enforce import Denied, Enforcer, answer
platform = open_catalog(["crm.learners"])
query = "select support_region, count(*) from crm.learners group by support_region order by support_region"
def show(rows):
return ", ".join(f"{region} {learners:,}" for region, learners in rows)
layer = Enforcer(platform)
print("through the enforcement layer, as agent-a05:", show(answer(layer, query, "agent-a05")))
# A notebook with DuckDB on the same catalog: the same SQL, and no policy in the way.
print("DuckDB on the same catalog:", show(warehouse.connect(platform).sql(query).fetchall()))
# Any reader that holds the catalog's access, here PyIceberg's own scan: no SQL to rewrite at all.
scan = platform.load_table("crm.learners").scan(selected_fields=("support_region", "email")).to_arrow()
print("PyIceberg's scan of the table:", show(sorted(Counter(scan.column("support_region").to_pylist()).items())))
print("emails PyIceberg's scan returns:", f"{scan.num_rows - scan.column('email').null_count:,}")
# lib.warehouse builds each table's view on a scan it registers under a name of its own. Through the layer, it's denied.
try:
answer(layer, "select count(*) from scan_crm_learners", "agent-a05")
except Denied as reason:
print("the layer, asked for the raw scan:", reason)
```
Output:
```text
through the enforcement layer, as agent-a05: emea 1,581
DuckDB on the same catalog: americas 2,276, apac 1,143, emea 1,581
PyIceberg's scan of the table: americas 2,276, apac 1,143, emea 1,581
emails PyIceberg's scan returns: 5,000
the layer, asked for the raw scan: not a table in the catalog: scan_crm_learners
```
Through the layer, the agent's query returns `emea 1,581`. The same SQL on DuckDB with no layer in front returns `americas 2,276, apac 1,143, emea 1,581`, and PyIceberg's scan, which has no SQL to rewrite, returns `americas 2,276, apac 1,143, emea 1,581`, with 5,000 emails. The policy didn't fail: it was never asked. The same happens in production whenever a second engine, a Spark job or a script holding storage credentials reads the files that the governed engine reads.
`lib.warehouse` builds each table's view on a scan that it registers under a name of its own, `scan_crm_learners` for this table, so the layer's connection holds the raw scan too. The last line shows what the layer answers when a query names it: `not a table in the catalog: scan_crm_learners`. That's the design [chapter 6](#enforcing-policies-on-queries) follows: enforce in one place, and deny everything else that reads data, [whatever isn't a catalog table](#allow-only-catalog-sources). In production, that place is the engine every tool queries through, as [chapter 6](#engine-enforcement) argues, and since [storage permissions are part of enforcement](#enforcement-in-production), only that engine and the jobs that write the tables should be able to read the files.
**Working rule:** Choose one place that enforces the policy, the engine every tool queries through, and give storage credentials only to it and to the jobs that write the tables, never to people or notebooks. Deny every query in that place that reads data other than through a catalog table. Before you count a table as protected, list everything that can read its files, and treat any reader that doesn't ask the policy as a bypass, unless it's a job whose outputs are cataloged and classified like the tables it reads.
## What this chapter decided {#end-to-end-decisions}
- **Governance is four kinds of code in one repository:** a metadata file, the code that sends each job's lineage events, an access policy in Rego, and checks that run on every pull request. Only one of them is a policy engine.
- **Every table has an owning team,** never a person, and every column one of four classifications: `public`, `internal`, `personal` or `sensitive`. They live in `governance/metadata.toml`, keyed by Iceberg field ID, and the catalog is tested against the file by walking the catalog.
- **Every pipeline run sends OpenLineage events,** with column lineage parsed from its SQL, including the columns that join and filter rows.
- **Access decisions are data:** allow, row filters and column masks, from one policy that denies by default. An enforcement layer applies them to every query, and denies every read that isn't of a catalog table.
- **Missing metadata fails closed:** the access policy treats a column nobody has classified as `sensitive`, and a table nobody has described stays a finding in every report until someone describes it.
- **The governance job runs on every pull request,** fails on the findings the pull request introduces, and reports the findings already on main with their owners.
- **The examples run in one process, offline,** through the standards the production services share. PyIceberg's SQL catalog stands in for a catalog server, DuckDB for Spark, DuckDB behind the enforcement layer for Trino, regopy for OPA's server, a file of events for a lineage server, and two folders for the CI service.
# A catalog you can test
By the end of this chapter you can keep a catalog's metadata as code. Every table gets an owner and a domain, and every column a description and a classification, in a file your team reviews. The file is keyed so a rename can't lose anything, written into the Iceberg catalog, and checked by a test that fails when anything is missing. You'll also have a local model draft descriptions and classifications, and measure how often its drafts are wrong before anyone accepts them.
The platform, the online learning platform this book follows, sells professional courses to adult learners and keeps its warehouse in an Iceberg catalog. Eight source tables sit under the namespaces of the teams that run them, such as `crm.learners` and `support.tickets`, and the platform's pipelines build reporting tables from them. [Chapter 1](#governance-end-to-end) built a rough version of the whole governed platform. This chapter makes its catalog solid, and the catalog has three problems. The pipelines keep creating tables, and nothing says who owns `marketing.audience`, the list of learners marketing may email, or whether `ai.review_corpus`, the text the platform's course assistant reads, holds personal data. A column renamed in a schema change can lose its classification without anyone noticing. And every table a pipeline adds needs a description and a classification for each of its columns, written by hand, which is slow enough that teams skip it.
The chapter's examples use five of the book's shared modules. `lib.catalog.open_catalog` builds a fresh catalog of the platform's tables in a temporary folder, with PyIceberg's SQL catalog on SQLite, which PyIceberg's docs recommend "for exploratory or development purposes" only ([PyIceberg configuration](https://github.com/apache/iceberg-python/blob/pyiceberg-0.12.0/mkdocs/docs/configuration.md)). `lib.pipelines.run` runs one of the platform's pipelines and writes its output to the catalog. `lib.warehouse.connect` opens DuckDB on the catalog's tables, so a query can name them as the platform does, and `lib.policy` evaluates the platform's access policies, which [chapter 5](#access-policies-as-code) writes. And `lib.metadata`, which this chapter builds up, loads the platform's metadata, writes it into the catalog and checks the catalog against it.
## Keep the metadata in a reviewed file {#keep-metadata-in-a-reviewed-file}
A catalog's **metadata** is what you know about each table that the data doesn't say: who owns it, which part of the business it belongs to, what it holds, and how each column must be protected. Catalog servers keep this in their own database and edit it in a web page. That works for browsing, but a change made there is invisible to code review, and no test can say whether the catalog is complete. Keeping the metadata in a file in the repository, beside the pipelines that create the tables, makes every change a diff that someone approves, and gives a test something to compare the catalog with.
The platform's file is `governance/metadata.toml`. For each table it records:
- an **owner**, the team answerable for the table, never a person;
- a **domain**, the part of the business the table belongs to;
- a description of the table;
- for each column, its name, a description and a **classification**.
A classification is one of four values, ordered from least to most protected:
| Classification | Meaning | On the platform |
| --- | --- | --- |
| `public` | May be shown to anyone outside the platform | A course's title |
| `internal` | For the platform's staff, and doesn't identify a person by itself | When an enrollment started |
| `personal` | Identifies a person or tells something about one | A learner's email, or text they wrote |
| `sensitive` | Data whose leak would directly harm someone | A card number, an instructor's bank account |
The order matters: when a column is built from several others, as a pipeline's output columns often are, the stricter classification wins, and `lib.metadata.stricter` returns it. Load the file and count what it says:
Example `metadata-model` (Python 3.12.15 · pyarrow 26.0.0 · pyiceberg 0.12.0), passing in Edition 2026.10.1:
```python
from collections import Counter
from lib import metadata
described = metadata.load() # governance/metadata.toml
columns = [column for table in described.values() for column in table.columns.values()]
print("tables described:", len(described))
print("columns described:", len(columns))
counts = Counter(column.classification for column in columns)
for level in metadata.LEVELS:
print(f"columns classified {level}:", counts[level])
owners = sorted({table.owner for table in described.values()})
print("teams that own tables:", len(owners))
print("owners:", ", ".join(owners))
learners = described["crm.learners"]
print("crm.learners owner:", learners.owner)
print("crm.learners domain:", learners.domain)
for field_id in (1, 3, 5, 6):
column = learners.columns[field_id]
print(f"crm.learners field {field_id}:", column.name, "|", column.classification, "|", column.description)
```
Output:
```text
tables described: 10
columns described: 77
columns classified public: 6
columns classified internal: 57
columns classified personal: 12
columns classified sensitive: 2
teams that own tables: 5
owners: team-analytics, team-billing, team-crm, team-learning, team-support
crm.learners owner: team-crm
crm.learners domain: crm
crm.learners field 1: learner_id | internal | The learner's ID, which the platform's other tables use for the learner. Identifies a person through this table's personal columns.
crm.learners field 3: email | personal | The email the learner signs in with.
crm.learners field 5: country | internal | The learner's country, as an ISO 3166 two-letter code.
crm.learners field 6: city | personal | The learner's city. Identifying together with date of birth.
```
The file describes 10 tables and 77 columns: the eight source tables and the two reporting tables the platform's analysts rely on. Most columns are `internal`, 57 of them, because most of a warehouse is IDs, dates and counts. The file classifies 12 as `personal` and 2 as `sensitive`, the card number in `billing.payments` and the payout account in `crm.instructors`, and every table belongs to one of 5 owning teams. Each column's description says what a reviewer needs in order to check the classification: `country` is `internal` because a country alone doesn't single anyone out, while `city` is `personal` because, as its description says, it identifies a learner together with their date of birth.
The four values and the one-owner rule are ours, not a standard's. The Open Data Contract Standard (ODCS) records a domain, ownership as a free-text role on a team, and a free-text classification ([ODCS v3.2.0](https://github.com/bitol-io/open-data-contract-standard/tree/v3.2.0/docs)), which its JSON schema accepts as any string ([ODCS JSON schema](https://github.com/bitol-io/open-data-contract-standard/blob/v3.2.0/schema/odcs-json-schema-v3.2.0.json)). The catalog servers differ too. DataHub types its owners ([ownership types](https://docs.datahub.com/docs/ownership/ownership-types)), allows an asset "only one Domain at a time" ([domains](https://docs.datahub.com/docs/domains)), and classifies columns with glossary terms ([business glossary](https://docs.datahub.com/docs/glossary/business-glossary)). In OpenMetadata, "either a team or multiple user can be the owner", a rule that "is enforced at the instance level" and is on by default ([data ownership](https://docs.open-metadata.org/v2.0.x/how-to-guides/guide-for-data-users/data-ownership)), and a classification is a controlled vocabulary of tags ([classification](https://docs.open-metadata.org/v2.0.x/how-to-guides/data-governance/classification)), such as `PII.Sensitive` ([PII tags](https://docs.open-metadata.org/v2.0.x/how-to-guides/data-governance/classification/sample-data-using-pii-tag)). In Databricks' Unity Catalog every object has an owner, who holds every privilege on it ([privileges](https://docs.databricks.com/aws/en/data-governance/unity-catalog/manage-privileges)). Tables and columns carry tags ([tags](https://docs.databricks.com/aws/en/database-objects/tags)), and a governed tag takes values "only from a predefined set of allowed values" ([governed tags](https://docs.databricks.com/aws/en/admin/governed-tags/)). A file like the platform's maps onto each of them. What none of these documents describes is the check this chapter's test makes: a team, never a person, as the owner of every table, and one of the four classifications on every column.
**Working rule:** Keep owners, domains, descriptions and classifications in a file in the repository, reviewed like the pipelines. Use a closed list of classifications, ordered by how much protection each needs, and one owning team per table. If your organization edits metadata in a catalog server's web page, make one side the source and generate the other from it; two sources that both accept edits drift apart.
## Key column metadata by field ID {#key-column-metadata-by-field-id}
[The platform's metadata file](#keep-metadata-in-a-reviewed-file) keys each column by a number, not by its name, because names change. Iceberg gives every column a **field ID**: "Each field in the tuple is named and has an integer id that is unique in the table schema", and "Renaming an existing field must change the name, but not the field ID" ([Iceberg table spec](https://iceberg.apache.org/spec/)). Readers find a column's values by that ID too, since "Columns in Iceberg data files are selected by field id", so a renamed column still reads the files written before the rename. Metadata keyed by name has nothing to hold on to: after a rename, the old name points at nothing and the new name has no entry.
The platform's CRM team wants `crm.learners.email` called `contact_email`, to make room for a work email. The example renames it in a scratch catalog, then looks the column up in two ways: in the platform's metadata file, keyed by field ID, and in a copy keyed by name, the way a spreadsheet of columns usually is:
Example `rename-orphans-name-keys` (Python 3.12.15 · pyarrow 26.0.0 · pyiceberg 0.12.0 · regopy 1.5.2), passing in Edition 2026.10.1:
```python
from lib import catalog, metadata, policy
platform = catalog.open_catalog(["crm.learners"])
described = metadata.load()
by_id = described["crm.learners"].columns # the book's metadata: keyed by field ID
by_name = {column.name: column.classification for column in by_id.values()} # a spreadsheet's: keyed by name
learners = platform.load_table("crm.learners")
with learners.update_schema() as update:
update.rename_column("email", "contact_email")
field = learners.schema().find_field("contact_email")
print("contact_email field ID:", field.field_id)
values = learners.scan(selected_fields=("contact_email",)).to_arrow().column("contact_email")
print("values read under the new name:", f"{len(values) - values.null_count:,} of {len(values):,}")
print("classification of contact_email, looked up by name:", by_name.get("contact_email"))
print("classification of contact_email, looked up by field ID:", by_id[field.field_id].classification)
names = {f.name for f in learners.schema().fields}
print("name-keyed entries that match no column:", ", ".join(sorted(set(by_name) - names)))
for problem in metadata.problems(platform, described):
print("check:", problem)
# The policies read classifications by name, generated from the file, which still says email.
decision = policy.decide(policy.request("agent-a01", "crm.learners", ["contact_email"]))
print("a support agent reading contact_email sees it as:", decision["column_masks"].get("contact_email", "shown"))
```
Output:
```text
contact_email field ID: 3
values read under the new name: 5,000 of 5,000
classification of contact_email, looked up by name: None
classification of contact_email, looked up by field ID: personal
name-keyed entries that match no column: email
check: crm.learners.contact_email: renamed from email since it was reviewed
a support agent reading contact_email sees it as: redact
```
The rename kept field ID 3, and every value came through under the new name: 5,000 of 5,000. The two lookups disagree. Keyed by name, `contact_email` has no classification, and the entry for `email` matches no column. Keyed by field ID, the column is still `personal`. A check built on the name-keyed copy would report an unclassified column and miss that a classified one had gone; the ID-keyed file knows it is the same column.
The ID-keyed file does notice the rename, and `lib.metadata.problems`, the check this chapter builds on, reports it: `contact_email: renamed from email since it was reviewed`. The data surviving a rename doesn't make the rename harmless. SQL that names `email` breaks, and so does any system that knows columns only by name. Access policies are one: the platform's policies, which [chapter 5](#access-policies-as-code) writes, receive the columns a query reads by name, so they read classifications from a name-keyed view that `lib.policy` generates from this file. When the check fails, someone updates the name in the file, reads the description again, and the generated view follows. Until then, the view has no entry for `contact_email`, and the policies treat a column they can't find as `sensitive`. A support agent, who sees every other personal column of `crm.learners`, gets `contact_email` as `redact`: the rename fails closed, and support notices first.
**Working rule:** Key column metadata by field ID, with the name beside it for people. Treat a name that no longer matches as a reason to review the column's description, not as a new column. Where a system knows columns only by name, generate its name-keyed view from the ID-keyed file, never maintain it by hand.
## Write descriptions into the table's schema {#write-descriptions-into-the-schema}
[The metadata file](#keep-metadata-in-a-reviewed-file) is where the platform's metadata is reviewed. People and tools that read a table, though, look in the catalog, so some of the metadata has to be copied there. The question is where in the catalog each fact belongs, and Iceberg answers part of it. Each column in an Iceberg schema "may have an optional comment or doc string", its `doc` ([Iceberg table spec](https://iceberg.apache.org/spec/)), which travels with the schema. A table also has properties, a map of strings, but the spec says that map "is used to control settings that affect reading and writing and is not intended to be used for arbitrary metadata."
So we follow the spec as far as we can. Column descriptions go into `doc`. Classifications stay in the file, where the policies and checks read them. The owner and the domain go into two table properties, as a pointer for tools that read only the table, so anyone with a table and no repository can find the team to ask. That last choice goes against the spec's stated intent, and we keep it to those two short values. The Iceberg project's own answer is a separate channel: the REST catalog spec of Iceberg 1.12.0 adds `labels`, ownership or classification that a catalog returns with a table, per table and per field ID, kept out of properties on purpose ([REST catalog spec](https://github.com/apache/iceberg/blob/apache-iceberg-1.12.0/open-api/rest-catalog-open-api.yaml), [pull request 15750](https://github.com/apache/iceberg/pull/15750)). PyIceberg 0.12.0's REST client doesn't read them ([source](https://github.com/apache/iceberg-python/blob/pyiceberg-0.12.0/pyiceberg/catalog/rest/__init__.py)).
`lib.metadata.apply` does the copying: for every table the file describes and the catalog holds, it writes each description into its column's `doc`, and the owner and domain into the properties, changing only what differs. Apply it to a fresh catalog of the platform's eight source tables:
Example `apply-metadata` (Python 3.12.15 · pyarrow 26.0.0 · pyiceberg 0.12.0), passing in Edition 2026.10.1:
```python
from lib import catalog, metadata
platform = catalog.open_catalog()
described = metadata.load()
print("crm.learners schema ID before:", platform.load_table("crm.learners").schema().schema_id)
print("tables changed:", len(metadata.apply(platform, described)))
learners = platform.load_table("crm.learners")
print("crm.learners schema ID after:", learners.schema().schema_id)
print("email's doc, in the Iceberg schema:", learners.schema().find_field("email").doc)
scanned = learners.scan(selected_fields=("email",), limit=1).to_arrow()
print("email's doc, in a scan's Arrow schema:", scanned.schema.field("email").metadata[b"doc"].decode())
print("crm.learners properties:", ", ".join(f"{k}={v}" for k, v in sorted(learners.properties.items())))
print("tables changed by applying again:", len(metadata.apply(platform, described)))
```
Output:
```text
crm.learners schema ID before: 0
tables changed: 8
crm.learners schema ID after: 1
email's doc, in the Iceberg schema: The email the learner signs in with.
email's doc, in a scan's Arrow schema: The email the learner signs in with.
crm.learners properties: domain=crm, owner=team-crm
tables changed by applying again: 0
```
All 8 tables changed. The description of `email` is now in the Iceberg schema and in the Arrow schema that a scan returns, as field metadata under `doc`, so code that reads the table through PyIceberg gets the description with the data, without knowing the file exists. The properties hold `owner=team-crm` and `domain=crm`, and applying a second time changes 0 tables, so `apply` can run on every merge without writing anything when nothing changed. The schema ID of `crm.learners` went from 0 to 1, although no column's ID, name or type changed, which is the subject of [the next subsection](#a-description-edit-is-a-schema-change).
### A description edit is a schema change {#a-description-edit-is-a-schema-change}
Iceberg's spec says that every schema change produces "a new schema that is identified by a unique schema ID, is added to the table's list of schemas, and is set as the table's current schema" ([Iceberg table spec](https://iceberg.apache.org/spec/)). In PyIceberg 0.12.0, editing nothing but a column's `doc` counts as such a change. Edit the description of `learning.reviews.body`, after [`lib.metadata.apply`](#write-descriptions-into-the-schema) has written the platform's descriptions into the schema:
Example `doc-edit-schema-id` (Python 3.12.15 · pyarrow 26.0.0 · pyiceberg 0.12.0), passing in Edition 2026.10.1:
```python
from lib import catalog, metadata
platform = catalog.open_catalog(["learning.reviews"])
metadata.apply(platform, metadata.load())
reviews = platform.load_table("learning.reviews")
before = reviews.schema()
print("schema ID before the edit:", before.schema_id)
with reviews.update_schema() as update: # change one description, nothing else
update.update_column("body", doc="The review's text, as the learner wrote it. May hold names and emails.")
after = reviews.schema()
print("schema ID after the edit:", after.schema_id)
print("schemas the table keeps:", len(reviews.metadata.schemas))
print("schema ID changed:", before.schema_id != after.schema_id)
print("field IDs, names or types changed:", metadata.shape(before) != metadata.shape(after))
```
Output:
```text
schema ID before the edit: 1
schema ID after the edit: 2
schemas the table keeps: 3
schema ID changed: True
field IDs, names or types changed: False
```
The schema ID went from 1 to 2, and the table keeps 3 schemas: the original, the one `apply` wrote, and this one. Yet the field IDs, names and types are the same, which `lib.metadata.shape` compares. A rule such as "classify a table again whenever its schema ID changes" would fire on every description edit, and teams learn to ignore a check that fires for nothing.
**Working rule:** Write column descriptions into Iceberg's `doc`, so every reader of the table gets them, and keep classifications in the reviewed file. Put no more than an owner and a domain in table properties, and say in your code why. To decide whether a schema change needs a review, compare field IDs, names and types, never schema IDs.
## Test the catalog against the metadata {#test-the-catalog-against-the-metadata}
With [the metadata applied to the catalog](#write-descriptions-into-the-schema), the remaining question is whether the file covers everything the catalog holds. A test answers it, and it has to start from the catalog: a check that walks the file finds problems only in tables someone already described. `lib.metadata.problems` lists every table in every namespace of the catalog, and reports a table with no entry or no owner, a column whose field ID has no classification, a classification that isn't one of the four, and a column renamed since it was reviewed.
On the platform, the gaps come from pipelines. Each pipeline is one `create table ... as select ...` in `pipelines/`, and running it creates a table. Run all four of them on a fresh catalog, then check:
Example `metadata-gaps` (Python 3.12.15 · duckdb 2.0.0 · pyarrow 26.0.0 · pyiceberg 0.12.0 · sqlglot 30.24.0), passing in Edition 2026.10.1:
```python
from lib import catalog, metadata, pipelines
platform = catalog.open_catalog()
for name in ("course_performance", "learner_activity", "marketing_audience", "review_corpus"):
print("pipeline output:", pipelines.run(platform, name))
tables = [f"{namespace}.{name}" for (namespace,) in platform.list_namespaces() for _, name in platform.list_tables(namespace)]
print("tables in the catalog:", len(tables))
found = metadata.problems(platform, metadata.load())
print("problems found:", len(found))
for problem in found:
print("problem:", problem)
```
Output:
```text
pipeline output: analytics.course_performance
pipeline output: analytics.learner_activity
pipeline output: marketing.audience
pipeline output: ai.review_corpus
tables in the catalog: 12
problems found: 2
problem: ai.review_corpus: no metadata
problem: marketing.audience: no metadata
```
The catalog holds 12 tables, and the check finds 2 problems: `ai.review_corpus: no metadata` and `marketing.audience: no metadata`. The two analytics tables pass, because the file describes them, with field IDs that match the order of their `select` lists. The two that fail are the ones that matter most. `marketing.audience` copies learners' names and emails out of `crm.learners`, and `ai.review_corpus` copies the free text of reviews, so both hold personal data, and neither has an owner who would answer for it. Until someone classifies them, the platform's access policy, `governance/policies/access.rego`, treats every column it can't find a classification for as `sensitive`, so the gap fails closed rather than open. [Chapter 4](#classifications-from-lineage) proposes their classifications from lineage, from the columns each output column was built from.
The test is then one line, `assert not metadata.problems(catalog, metadata.load())`, run wherever the catalog changes: in the pull request that adds a pipeline, which [chapter 10](#run-every-rule-as-one-job) sets up, and on a schedule against the production catalog, which catches tables created outside any pull request.
**Working rule:** Walk the catalog, not the metadata file, so a table nobody described shows up. Fail the test on any table without an owner and any column without one of the four classifications, and never default a missing classification to anything but the most protected value. A pipeline that creates a table ships with that table's metadata, in the same pull request.
## Catch schema changes before the metadata goes stale {#catch-schema-changes}
[The test against the catalog](#test-the-catalog-against-the-metadata) catches new tables. Tables also change underneath their metadata, through schema evolution, and each kind of change needs its own response. A rename keeps the field ID, so the entry survives, but its description may now be wrong. An added column gets a field ID nobody has reviewed. A dropped column leaves an entry that describes nothing. And dropping a column and adding one with the same name doesn't bring the old column back: the spec says that "adding a new field assigns a new ID for that field" ([Iceberg table spec](https://iceberg.apache.org/spec/)), so it's a different column. Schemas change often enough for this to matter: in a study of ten open-source applications, "80% of the schema changes happened in 20-30% of the tables" ([Qiu, Li and Su, 2013](https://www.cs.ucdavis.edu/~su/publications/fse13-db-study.pdf)). The study is of application databases, not warehouses. If a warehouse's changes concentrate the same way, its busiest tables are the ones whose metadata goes stale first.
Stage two of those changes on `crm.learners` in a scratch catalog: add a column, `preferred_name`, with a description but no classification, then drop `country` and add it back. Run `problems`, and `lib.metadata.dropped`, which reports entries whose field ID the table no longer has. Then read the dropped column's values the way Iceberg's time travel does, by naming a snapshot:
Example `schema-changes-checked` (Python 3.12.15 · pyarrow 26.0.0 · pyiceberg 0.12.0), passing in Edition 2026.10.1:
```python
from pyiceberg.types import StringType
from lib import catalog, metadata
platform = catalog.open_catalog(["crm.learners"])
described = metadata.load()
learners = platform.load_table("crm.learners")
written = learners.current_snapshot() # the snapshot the rows were written in
print("country field ID, before it was dropped:", learners.schema().find_field("country").field_id)
with learners.update_schema() as update: # a column nobody has reviewed
update.add_column("preferred_name", StringType(), doc="What the learner asks to be called.")
with learners.update_schema() as update: # drop a column, then add it back
update.delete_column("country")
with learners.update_schema() as update:
update.add_column("country", StringType())
country = learners.schema().find_field("country")
print("country field ID, after it was added back:", country.field_id)
values = learners.scan(selected_fields=("country",)).to_arrow().column("country")
print("country values after it was added back:", f"{values.null_count:,} nulls of {len(values):,}")
print("snapshots the schema changes wrote:", len(learners.metadata.snapshots) - 1)
old = learners.scan(snapshot_id=written.snapshot_id, selected_fields=("country",)).to_arrow().column("country")
print("country values, read by time travel to that snapshot:", f"{len(old) - old.null_count:,} of {len(old):,}")
for problem in metadata.problems(platform, described):
print("problems():", problem)
for problem in metadata.dropped(platform, described):
print("dropped():", problem)
```
Output:
```text
country field ID, before it was dropped: 5
country field ID, after it was added back: 17
country values after it was added back: 5,000 nulls of 5,000
snapshots the schema changes wrote: 0
country values, read by time travel to that snapshot: 5,000 of 5,000
problems(): crm.learners.preferred_name: not classified
problems(): crm.learners.country: not classified
dropped(): crm.learners.country: field 5 is described but no longer in the table
```
Both added columns are unclassified, and the check says so. The description given to `preferred_name` doesn't count: a description is a doc in the schema, and a classification is a reviewed line in the file. `country` came back as field 17, with 5,000 nulls of 5,000, because a plain scan reads with the current schema, where field 5 no longer exists. The learners' countries haven't gone anywhere. A schema change writes no snapshot (0 here), so the table's snapshot is still the one its rows were written in, and a snapshot records the "ID of the table's current schema when the snapshot was created" ([Iceberg table spec](https://iceberg.apache.org/spec/)). In PyIceberg 0.12.0, a scan that names a snapshot reads with that schema, and it returns 5,000 of 5,000 countries. Every snapshot stays readable this way, each with its own schema, until it expires and the files only it referenced are deleted. Dropping a column deletes nothing: [chapter 7](#retention-consent-erasure) is where the platform erases data for good. To the check, the re-added `country` is a column like any other new one, and `dropped` reports that the reviewed entry for the old field now describes nothing. The rename from [the field ID section](#key-column-metadata-by-field-id) is the third kind of change, and `problems` reports it the same way, as a column to review.
**Working rule:** Run both checks on every schema change: `problems` for columns nobody has classified or renamed ones nobody has re-read, and `dropped` for entries that describe nothing. Remove a dropped column's entry in the same pull request that drops it. Dropping a personal column doesn't remove its values, which time travel reads until the snapshots that hold them expire and their files are deleted. Never drop and re-add a column to rename it: rename it in place, so it keeps its field ID and its data.
## Classify pseudonymous IDs for joining {#classify-pseudonymous-ids}
[The metadata file](#keep-metadata-in-a-reviewed-file) classifies some columns that point at people as `internal`, and the most common of them is `learner_id`. It's a number with no meaning of its own, and the platform's analysts and data scientists need it to join enrollments, events, payments and tickets to each other. An ID like this is a **pseudonymous identifier**: it doesn't name anyone, but it leads to someone through another table. Count the tables that carry it and how many enrollments one join turns into an email, then ask the platform's access policy who can make that join and see names:
Example `pseudonymous-learner-id` (Python 3.12.15 · duckdb 2.0.0 · pyarrow 26.0.0 · pyiceberg 0.12.0 · regopy 1.5.2), passing in Edition 2026.10.1:
```python
from lib import catalog, metadata, policy, warehouse
described = metadata.load()
carrying = {name: table.column("learner_id") for name, table in described.items() if table.column("learner_id")}
print("tables with a learner_id column:", len(carrying))
print("learner_id classified as:", ", ".join(sorted({column.classification for column in carrying.values()})))
personal = [c.name for c in described["crm.learners"].columns.values() if c.classification == "personal"]
print("personal columns in crm.learners:", ", ".join(personal))
con = warehouse.connect(catalog.open_catalog(["crm.learners", "learning.enrollments"]))
found, total = con.execute("""
select count(l.email), count(*)
from learning.enrollments e left join crm.learners l on l.learner_id = e.learner_id
""").fetchall()[0]
print("enrollments whose learner_id leads to an email:", f"{found:,} of {total:,}")
# Who can put a name on another table's learner_id: users who read that table and see crm.learners' names unmasked.
rego, seeing, naming = policy.interpreter(), [], []
for user in policy.data_document()["platform"]["users"]:
names = policy.decide(policy.request(user, "crm.learners", ["full_name"]), rego)
if names["allow"] and "full_name" not in names["column_masks"]:
seeing.append(user)
tables = [t for t in carrying if t != "crm.learners"
and policy.decide(policy.request(user, t, ["learner_id"]), rego)["allow"]]
naming += [f"{user} ({', '.join(tables)})"] if tables else []
print("users who see learners' names in crm.learners:", ", ".join(seeing))
print("of them, users who read learner_id in another table:", ", ".join(naming))
```
Output:
```text
tables with a learner_id column: 7
learner_id classified as: internal
personal columns in crm.learners: full_name, email, phone, city, street_address, date_of_birth, notes
enrollments whose learner_id leads to an email: 20,000 of 20,000
users who see learners' names in crm.learners: agent-a01, agent-a05, agent-a09, marketing-1
of them, users who read learner_id in another table: agent-a01 (support.tickets), agent-a05 (support.tickets), agent-a09 (support.tickets)
```
`learner_id` appears in 7 tables, classified `internal` in every one. One join to `crm.learners` gives every enrollment an email, 20,000 of 20,000, and with it the learner's name, phone number, address and date of birth, which are among the table's `personal` columns. The EU's General Data Protection Regulation (GDPR) addresses data like this in Recital 26: "Personal data which have undergone pseudonymisation, which could be attributed to a natural person by the use of additional information should be considered to be information on an identifiable natural person" ([GDPR, Recital 26](https://eur-lex.europa.eu/eli/reg/2016/679/oj)). What the recital means for a given platform is a question for its lawyers, and we don't answer it.
What we do decide is how the book's code treats the ID. `internal` describes what the column reveals in the table that holds it, and each `learner_id`'s description in the metadata file says the rest, as `learning.enrollments`' does: "The learner who enrolled. Identifies a person when joined with crm.learners." The protection sits on the table that resolves the ID: [chapter 5](#return-the-decision-as-one-document) decides who may read the personal columns of `crm.learners`, and the example's last lines ask its policy who sees learners' names there: agent-a01, agent-a05, agent-a09, marketing-1. Of them, only the support agents also read a `learner_id` in another table: agent-a01 (support.tickets), agent-a05 (support.tickets), agent-a09 (support.tickets). They can put a name on every ticket of their region, and that's their job. Marketing sees names but reads no other table that holds the ID.
So the policy, not the ID's classification, decides who can make the join. Classifying `support.tickets.learner_id` as `personal` would change nothing for support, the only group that reads the table, because the policy already shows support every personal column. Erasure, in [chapter 7](#plan-erasure-from-lineage), has to reach every table that holds a learner's `learner_id`, whatever its classification. We call such IDs pseudonymized, never anonymized.
**Working rule:** Classify a pseudonymous ID by what its own table reveals, so the joins your analysts need keep working, and say in its description what it resolves through. Protect the table that resolves it, and include the ID in every erasure. Who can resolve the ID is an access decision: list the users who may read both the ID and the resolving table's personal columns unmasked, and grant that pair only to those whose job is linking rows to people.
## Draft metadata with a local model {#draft-descriptions-with-a-local-model}
[The test against the catalog](#test-the-catalog-against-the-metadata) tells you which columns nobody has described, but it doesn't write the descriptions. Writing a description and choosing a classification for every column is the slowest part of keeping a catalog, and the work comes back with every table a pipeline adds. A language model can draft both from what a catalog tool can show it: a column's name, its type and a few of its values. Research supports both the help and its limits. On one benchmark, given instructions and a two-step pipeline, ChatGPT labelled the semantic types of table columns with "F1 scores of over 85% in zero- and one-shot setups" ([Korini and Bizer, 2023](https://arxiv.org/abs/2306.00745)), while a study of generated column descriptions found that "models unsurprisingly struggle with columns that exhibit inherent ambiguity, highlighting the need for manual expert input" ([Wretblad et al., 2024](https://arxiv.org/abs/2408.04691)). So a model's draft is a proposal that a person accepts, corrects or rejects in the pull request, and the first job is to measure how wrong the drafts are, so the review knows where to look.
The book's drafts, which the examples call the metadata suggestions, were recorded once by `recordings/metadata_suggestions.py`, and the examples read that recording; none of them calls a model. The script runs Qwen3-1.7B at a pinned revision. The model is open, under the Apache-2.0 licence, so anyone can download the same weights. At 1.7 billion parameters it runs on a CPU, with no GPU, which keeps the sample values on the machine that runs it. And it's instruction-tuned, with a switch that turns off its "thinking", the reasoning it otherwise writes before it answers. The script decodes greedily, always taking the most likely next token, although the model's card suggests sampling in this mode: "Temperature=0.7, TopP=0.8, TopK=20, and MinP=0" ([Qwen3-1.7B](https://huggingface.co/Qwen/Qwen3-1.7B)). A recording has to come out the same when someone records it again, and a sampled answer is one draw among many.
For each column of the platform's eight source tables, the model saw the table's name, the column's name and Iceberg type, five of the column's values chosen with a fixed seed, and the four classifications with the meanings in [the table of classifications](#keep-metadata-in-a-reviewed-file). Each meaning comes with two examples, and none of them is a column the platform has: for `sensitive`, "a government ID number or a health condition". The model was asked for a one-sentence description and one classification, as JSON. The recording's manifest names the model, its revision, a hash of the prompts and settings, and the date, so anyone can record again and compare.
Those five values are personal data whenever the column is: for `support.tickets.body`, the tickets the model read held learners' names, street addresses, a phone number, a date of birth and a bank account number. A model running on your own machine keeps them there. Sending the same prompt to a hosted model sends the platform's personal data to another company, which needs the same decision as any other transfer of it.
Compare the model's classifications with the reviewed ones in the metadata file:
Example `suggestion-agreement` (Python 3.12.15 · pyarrow 26.0.0 · pyiceberg 0.12.0), passing in Edition 2026.10.1:
```python
from collections import Counter
from lib import metadata
reviewed = metadata.load() # governance/metadata.toml, reviewed by people
suggested = metadata.suggestions() # data/recorded/metadata-suggestions
assert all(reviewed[s.table].columns[s.field_id].name == s.column for s in suggested)
print("columns the model was asked about:", len(suggested))
print("answers that parse as the JSON asked for:", sum(s.description is not None for s in suggested))
print("answers with one of the four classifications:", sum(s.classification in metadata.LEVELS for s in suggested))
pairs = Counter((reviewed[s.table].columns[s.field_id].classification, s.classification) for s in suggested)
agree = sum(n for (was, said), n in pairs.items() if was == said)
print("classifications that agree with the review:", f"{agree} of {len(suggested)} ({agree / len(suggested):.0%})")
for level in metadata.LEVELS:
said = sorted(((s, n) for (was, s), n in pairs.items() if was == level), key=lambda p: -p[1])
print(f"reviewed {level}, the model said:", ", ".join(f"{s} {n}" for s, n in said))
for level in ("personal", "sensitive"):
reviewed_as = sum(n for (was, _), n in pairs.items() if was == level)
print(f"{level} columns the model called {level}:", f"{pairs[level, level]} of {reviewed_as}")
```
Output:
```text
columns the model was asked about: 64
answers that parse as the JSON asked for: 64
answers with one of the four classifications: 64
classifications that agree with the review: 40 of 64 (62%)
reviewed public, the model said: internal 3, public 3
reviewed internal, the model said: internal 37, public 5, sensitive 2
reviewed personal, the model said: internal 9, public 3
reviewed sensitive, the model said: internal 2
personal columns the model called personal: 0 of 12
sensitive columns the model called sensitive: 0 of 2
```
All 64 answers are valid JSON with one of the four classifications, which shows that the prompt's format worked and says nothing about whether the answers are right. They agree with the review on 40 of 64 (62%) columns, and that single number hides where the misses are. Most of the agreement is `internal` columns called `internal`. Of the platform's personal columns, the model called 0 of 12 `personal`, and of its `sensitive` ones, the card number and the payout account, 0 of 2: it called both `internal`.
### Examples in a prompt can give the answer away {#prompt-examples-give-answers-away}
A prompt's examples are part of the measurement. The script also records the same model on the same columns under a second prompt, identical but for its examples. It gives examples for `personal` and `sensitive` only, and they're the platform's own columns: `personal` is "a name, contact details, an address or text they wrote", and `sensitive` is "a card number or a bank account". Those two are `billing.payments.card_number` and `crm.instructors.payout_iban`, the platform's only sensitive columns. Compare the two prompts' answers:
Example `suggestions-with-platform-examples` (Python 3.12.15 · pyarrow 26.0.0 · pyiceberg 0.12.0), passing in Edition 2026.10.1:
```python
from lib import metadata
reviewed = metadata.load()
prompts = {"the book's prompt": metadata.SUGGESTIONS,
"the prompt with the platform's columns as examples": metadata.SUGGESTIONS_WITH_PLATFORM_EXAMPLES}
for name, path in prompts.items():
suggested = [(s, reviewed[s.table].columns[s.field_id].classification) for s in metadata.suggestions(path)]
agree = sum(s.classification == was for s, was in suggested)
print(f"{name}, classifications that agree with the review:", f"{agree} of {len(suggested)}")
for level in ("personal", "sensitive"):
reviewed_as = [s for s, was in suggested if was == level]
print(f"{name}, {level} columns called {level}:",
f"{sum(s.classification == level for s in reviewed_as)} of {len(reviewed_as)}")
for s, was in suggested:
if was == "sensitive":
print(f"{name}, {s.table}.{s.column}:", s.classification)
```
Output:
```text
the book's prompt, classifications that agree with the review: 40 of 64
the book's prompt, personal columns called personal: 0 of 12
the book's prompt, sensitive columns called sensitive: 0 of 2
the book's prompt, crm.instructors.payout_iban: internal
the book's prompt, billing.payments.card_number: internal
the prompt with the platform's columns as examples, classifications that agree with the review: 41 of 64
the prompt with the platform's columns as examples, personal columns called personal: 0 of 12
the prompt with the platform's columns as examples, sensitive columns called sensitive: 2 of 2
the prompt with the platform's columns as examples, crm.instructors.payout_iban: sensitive
the prompt with the platform's columns as examples, billing.payments.card_number: sensitive
```
Named in the prompt, the sensitive columns come out right: 2 of 2. With examples from elsewhere, the model finds 0 of 2 and calls both `internal`. Overall agreement hardly moves, 41 of 64 against 40 of 64, so a report of one number would never show the difference. The examples for `personal` didn't help: 0 of 12 personal columns came out `personal` under that prompt too. "A card number" names `card_number` almost word for word, while "a name" leaves the model to decide that `full_name` is one, and it didn't. A prompt whose examples match the catalog makes a model look better than it is, on the columns that need protection most, so the book's prompt takes its examples from outside the platform's data.
**Working rule:** Draft metadata with a model only from inputs you're allowed to send it, preferably to a model on your own machines. Record each answer with the model, its revision, the prompt and the settings, so a reviewer can see what produced each draft, and parse and validate every answer before anything reads it. Take the prompt's examples from outside the tables you measure it on. Report agreement with the reviewed metadata per classification, never as one number.
## Measure where the model's drafts are wrong {#measure-where-the-drafts-are-wrong}
[The recorded drafts](#draft-descriptions-with-a-local-model), a local model's descriptions and classifications for the platform's source tables, agree with the reviewed classifications on most columns. For governance, what matters more is which way they miss. A classification more protected than it should be costs a reviewer a minute and an analyst a masked column. One less protected than it should be leaves personal data where the policies treat it as staff data, or as data anyone may see. Count the misses in each direction:
Example `suggestion-mistakes` (Python 3.12.15 · pyarrow 26.0.0 · pyiceberg 0.12.0), passing in Edition 2026.10.1:
```python
from lib import metadata
reviewed = metadata.load()
rank = metadata.LEVELS.index
weaker, stronger = [], []
for s in metadata.suggestions():
if s.classification not in metadata.LEVELS: # an answer nobody can act on goes to a person
continue
column = reviewed[s.table].columns[s.field_id]
if rank(s.classification) < rank(column.classification):
weaker.append((s, column.classification))
elif rank(s.classification) > rank(column.classification):
stronger.append((s, column.classification))
print("columns the model protects less than the review:", len(weaker))
print("columns the model protects more than the review:", len(stronger))
# Each kind of miss, the most protected review first.
for misses in (weaker, stronger):
for was, said in sorted({(was, s.classification) for s, was in misses}, key=lambda p: (-rank(p[0]), -rank(p[1]))):
names = [f"{s.table}.{s.column}" for s, w in misses if (w, s.classification) == (was, said)]
print(f"{was} columns the model called {said}:", len(names))
print(f"which, {was} called {said}:", ", ".join(names))
```
Output:
```text
columns the model protects less than the review: 19
columns the model protects more than the review: 5
sensitive columns the model called internal: 2
which, sensitive called internal: crm.instructors.payout_iban, billing.payments.card_number
personal columns the model called internal: 9
which, personal called internal: crm.learners.full_name, crm.learners.email, crm.learners.phone, crm.learners.date_of_birth, crm.learners.notes, crm.instructors.full_name, crm.instructors.email, learning.events.ip_address, support.tickets.body
personal columns the model called public: 3
which, personal called public: crm.learners.city, crm.learners.street_address, learning.reviews.body
internal columns the model called public: 5
which, internal called public: learning.events.event_type, learning.events.occurred_at, learning.events.user_agent, learning.reviews.posted_at, support.tickets.opened_at
internal columns the model called sensitive: 2
which, internal called sensitive: crm.learners.marketing_consent, crm.learners.ai_training_consent
public columns the model called internal: 3
which, public called internal: learning.courses.course_id, learning.courses.title, learning.courses.price_usd
```
The model protected 19 columns less than the review did, and 5 more. Every personal and sensitive column is among the less protected. The model called 9 personal columns `internal`, including both tables' names and emails, the learners' phone numbers, dates of birth and notes, the IP addresses of events and the text of support tickets. It called 3 of them `public`: learners' cities and street addresses, and the text of their reviews. The prompt's own meaning for `personal` begins "identifies a person or tells something about one", and a name or an email does both. The other 5 less protected columns are three timestamps, an event's type and its user agent, which the model called `public` where the review chose `internal`: a smaller mistake, but one a reviewer still has to catch. The misses the other way cost less. The model called the 2 consent flags `sensitive`, and 3 public columns of the course catalog `internal`.
Descriptions go wrong differently. Put four of the model's descriptions beside the reviewed ones:
Example `suggestion-descriptions` (Python 3.12.15 · pyarrow 26.0.0 · pyiceberg 0.12.0), passing in Edition 2026.10.1:
```python
from lib import metadata
reviewed = metadata.load()
suggested = {(s.table, s.column): s for s in metadata.suggestions()}
# Four columns where the platform knows more than a name, a type and five values say.
for table, column in [("crm.learners", "ai_training_consent"), ("crm.learners", "support_region"),
("crm.learners", "notes"), ("billing.payments", "card_number")]:
print(f"{table}.{column}, the model:", suggested[table, column].description)
print(f"{table}.{column}, the review:", reviewed[table].column(column).description)
```
Output:
```text
crm.learners.ai_training_consent, the model: This column indicates whether a learner has consented to receive AI training.
crm.learners.ai_training_consent, the review: Whether the learner agreed to their text training the platform's AI. Unknown for sign-ups before the question was asked.
crm.learners.support_region, the model: The 'support_region' column indicates the regional support area for learners in the online learning platform.
crm.learners.support_region, the review: The support team that serves the learner: americas, emea or apac.
crm.learners.notes, the model: Stores notes from learners about their move, billing address, or preferences.
crm.learners.notes, the review: Free text from support agents. May hold anything, so it is treated as personal.
billing.payments.card_number, the model: Stores the card numbers of customers for payment processing.
billing.payments.card_number, the review: The full card number, from a legacy export. Should not be stored at all.
```
Every description is fluent and plausible, and three of the four miss something that only someone who knows how the platform fills the column could write. `ai_training_consent` records whether the learner's text may train the platform's AI, not whether the learner wants training, and a null in it means the question was never asked. `notes` are written by support agents about learners, as the values hint ("Asked about invoices for their employer."), not by learners. And `card_number` is a legacy column that shouldn't exist at all, which is the most useful thing its description can say. The fourth is right in substance: the platform gives each learner the `support_region` of the support team that serves their country, and the model calls it their "regional support area". Nothing in a fluent description says which kind it is, so a reviewer reads every one. The authors of one method for generating column descriptions name the same failure in single-prompt models: "hallucinated or incomplete descriptions" ([Cai et al., 2026](https://arxiv.org/abs/2606.21685)).
These are one small model's answers, decoded greedily, to two prompts. The recording can't tell whether the 0 of 12 for personal columns comes from the model's size or from drafting with only a name, a type and five values to go on. A larger model or a better prompt may get more columns right, and the way to find out is to measure it the same way, against the reviewed file, before trusting it with tables nobody has reviewed.
**Working rule:** Let a model raise a column's classification, never lower it: only a reviewer lowers one, in the pull request, with the reason in the column's description. Treat the model's descriptions as first drafts for someone who knows the column to rewrite. Before a model drafts metadata for tables nobody has reviewed, measure it against your reviewed metadata, per classification and per direction of error, and measure again whenever the model or the prompt changes.
## What goes wrong in production {#catalog-in-production}
A catalog's metadata fails quietly: nothing errors when it goes stale, and every failure below looks like a complete catalog until someone relies on it.
- **Owners go stale.** Teams are renamed, merged and dissolved, and the owner in the file outlives them. A team, unlike a person, survives one departure, but the file still needs a check that every owner is a team that exists, run against your organization's list of teams. If you keep ownership in ODCS, its team members carry a `dateOut` and a `replacedByUsername` ([ODCS v3.2.0](https://github.com/bitol-io/open-data-contract-standard/tree/v3.2.0/docs)), which a check can read to find a team with no current member in the `owner` role.
- **Descriptions describe last year's schema.** A column's meaning changes while its name and type stay the same: `amount` starts holding refunds as negative numbers, or `status` gains a value. No schema check sees it. Code comments fail the same way, and research on them found that a detector catches inconsistencies more accurately when it sees the code change, "before they are committed", than when it sees only the code that results ([Panthaplackel et al., 2021](https://arxiv.org/abs/2010.01625)). The comparison holds for columns only where the change is in a pull request: a pipeline's code changes in a diff, but a source system can change what it writes with no diff anywhere. Where there is a diff, make the pull request that changes a pipeline's logic show the descriptions of the columns it writes, so the reviewer reads both.
- **Metadata keyed by name goes stale on a rename.** A spreadsheet of columns, a dashboard's data dictionary, or a policy written against column names loses a column on every rename, as [the field ID section](#key-column-metadata-by-field-id) shows, and reports nothing. Generate every name-keyed view from the ID-keyed file, and run `problems` on every schema change.
- **Two sources of metadata overwrite each other.** A catalog server's web page and a file in a repository both accept edits, and a sync in either direction loses the other's. DataHub keeps edits made in its UI apart from ingested metadata "to avoid accidental overwrites of user-provided data by ingestion pipelines" ([DataHub dataset entity](https://docs.datahub.com/docs/generated/metamodel/entities/dataset)). Choose one source, and make the other read-only or generated.
- **Description edits trigger reviews nobody needs.** In PyIceberg every doc edit gives the schema a new ID, as [the doc edit above](#a-description-edit-is-a-schema-change) shows, so a review keyed on schema IDs fires for nothing, and people learn to ignore it. Compare field IDs, names and types instead.
- **Drafted classifications are approved in bulk.** A pull request that adds forty model-drafted columns gets its descriptions read and its classifications skimmed, and [the recorded drafts](#measure-where-the-drafts-are-wrong) show where that leads. List each drafted classification separately from the descriptions, least protected first, so the reviewer sees what each draft would leave unprotected.
- **A table created outside the pipelines escapes the test.** A notebook or a one-off backfill creates a table nobody will add to the file. Run [`problems`](#test-the-catalog-against-the-metadata) on a schedule against the production catalog, not only in pull requests.
## What this chapter decided {#catalog-decisions}
- **Metadata is a reviewed file:** owners, domains, descriptions and classifications live in `governance/metadata.toml`, changed only by pull request.
- **Four classifications, ordered:** `public`, `internal`, `personal` and `sensitive`, with the stricter one winning when a column is built from several. A column without one is treated as `sensitive`.
- **One owning team per table,** never a person, and a domain beside it.
- **Columns are keyed by Iceberg field ID,** with the name kept beside it so a rename shows up for review.
- **Descriptions go into Iceberg's `doc`,** written by `lib.metadata.apply`. The owner and domain go into table properties as a pointer, and classifications stay in the file.
- **The catalog is tested against the file:** `problems` walks every table in the catalog, and `dropped` finds entries for columns that are gone. Schema changes are compared by field IDs, names and types, not by schema IDs.
- **`learner_id` is `internal`** in every table, with a description that says it resolves through `crm.learners`. The protection sits on that table, and the access policy decides who may join the two with names in view.
- **`apply` and `problems` run on every change:** `apply` writes the file into the catalog on every merge, and `problems` runs in every pull request and on a schedule against the production catalog.
- **The model that drafts metadata runs on the platform's own machines,** because the sample values it's shown are personal data whenever the column is.
- **Model-drafted metadata is a proposal.** Descriptions and classifications drafted by a local model are recorded, measured against the reviewed file and rewritten by a person. A model may raise a classification, never lower it: the recorded model called 0 of 12 personal columns `personal`, and found the sensitive ones only when the prompt named them.
# Finding personal data
By the end of this chapter you can find personal data on a data platform automatically. You'll propose a classification for every column from its name, its type and a reproducible sample of its values, find personal data inside free text span by span, and measure each detector against labelled data, so you know what it misses before anyone relies on it. You'll run Presidio's pattern recognizers offline with thresholds chosen by measurement, compare them with the recorded detections of a model trained to find personal data, and classify again only the columns a schema change adds.
The platform, the online learning platform this book follows, sells professional courses to adult learners, and keeps its warehouse in an Iceberg catalog of eight source tables, each under the namespace of the team that runs it. [Chapter 2](#a-catalog-you-can-test) gave every column a classification in `governance/metadata.toml`, a file people review. A reviewed file records what its reviewers believe a column holds, and on the platform three beliefs need checking. `billing.payments.card_number` is a legacy column whose description says it holds full card numbers, and nobody has checked. `crm.learners.notes` is free text that support agents type, so its name says nothing about what's in it. And learners write whatever they like in support tickets and course reviews, so a classification of the whole column can't protect that text: the platform has to find each name, email and phone number inside it before the text can be masked span by span, which [chapter 9](#replace-personal-data) does to reviews before the platform's course assistant learns from them.
This chapter builds the book's shared module for this, `lib.pii`: rules on column names, seeded samples of a column's values, a Presidio detector configured to run offline, and code that scores any detector against labelled data. Every detector in this chapter is measured against one file, which we call the truth: `data/platform/pii_spans.parquet`. The script that generated the platform's data writes it beside the tickets and reviews: every piece of personal data it placed in their text, with its exact start and end. Invoice numbers, order numbers and the date a problem happened stay unlabelled, because the platform doesn't treat them as personal data.
The platform's phone numbers, card numbers and IBANs are random in a valid format, so any one may be in service, and no example prints one whole: it masks all but the last four digits.
## Classify columns from the schema {#classify-columns-from-the-schema}
The cheapest evidence about a column is its schema: its name and its type. A rule on names reads no data, so it can run over a whole catalog in a pull request, and it catches the columns whose names say what they hold, such as `email` or `date_of_birth`. Catalog tools start the same way: OpenMetadata's auto-classification begins with a "Column Name Scanner", which validates "the column names of the table against a set of regex rules" ([OpenMetadata docs](https://docs.open-metadata.org/v2.0.x/how-to-guides/data-governance/classification/auto-classification/auto-pii-tagging)). The type decides what comes next. A text column's values can be sampled and scanned. A timestamp can't hold an email, and scanning one with a date pattern would flag every row, so numbers, dates, timestamps and booleans are classified by name alone.
The platform keeps its rules in `governance/pii.toml`, the file that holds every setting of its detector and is reviewed like the metadata. The rules are 8 regular expressions on lowercased column names, each naming the entity it suggests, and the first match wins. The same file maps each entity to a classification: a phone number to `personal`, a card number or an IBAN to `sensitive`. `lib.pii.by_name` applies the rules. Run them over every column of the catalog, and compare them with the reviewed metadata:
Example `column-name-rules` (Python 3.12.15 · duckdb 2.0.0 · phonenumbers 9.0.41 · presidio-analyzer 2.2.364 · pyarrow 26.0.0 · pyiceberg 0.12.0 · pyyaml 6.0.3 · tldextract 5.4.0), passing in Edition 2026.10.1:
```python
from pyiceberg.types import StringType
from lib import metadata, pii
from lib.catalog import open_catalog
catalog = open_catalog()
described = metadata.load()
columns = [(table, f) for (namespace,) in sorted(catalog.list_namespaces())
for _, name in sorted(catalog.list_tables(namespace))
for table in [f"{namespace}.{name}"] for f in catalog.load_table(table).schema().fields]
print("columns in the catalog:", len(columns))
print("name rules:", len(pii.NAME_RULES))
named = [(table, f, pii.by_name(f.name)) for table, f in columns if pii.by_name(f.name)]
print("columns a name rule marks:", len(named))
for table, f, entity in named:
print(f" {table}.{f.name}: {entity}, so {pii.CLASSIFICATION[entity]}")
agree = sum(pii.CLASSIFICATION[entity] == described[table].columns[f.field_id].classification
for table, f, entity in named)
print("of them, agreeing with the metadata:", agree)
text = [(table, f) for table, f in columns if isinstance(f.field_type, StringType)]
print("columns of text, whose values can be sampled:", len(text))
# Columns the metadata, reviewed by people, calls personal or sensitive, that no name rule marks.
missed = [f"{table}.{f.name}" for table, f in columns if not pii.by_name(f.name)
and described[table].columns[f.field_id].classification in ("personal", "sensitive")]
print("personal columns no name rule marks:", ", ".join(missed))
```
Output:
```text
columns in the catalog: 64
name rules: 8
columns a name rule marks: 11
billing.payments.card_number: CREDIT_CARD, so sensitive
crm.instructors.full_name: PERSON, so personal
crm.instructors.email: EMAIL_ADDRESS, so personal
crm.instructors.payout_iban: IBAN_CODE, so sensitive
crm.learners.full_name: PERSON, so personal
crm.learners.email: EMAIL_ADDRESS, so personal
crm.learners.phone: PHONE_NUMBER, so personal
crm.learners.city: LOCATION, so personal
crm.learners.street_address: LOCATION, so personal
crm.learners.date_of_birth: DATE_TIME, so personal
learning.events.ip_address: IP_ADDRESS, so personal
of them, agreeing with the metadata: 11
columns of text, whose values can be sampled: 29
personal columns no name rule marks: crm.learners.notes, learning.reviews.body, support.tickets.body
```
Of the catalog's 64 columns, the rules mark 11, and all 11 agree with the classification people gave them. The rules miss three columns the reviewers classified `personal`: `crm.learners.notes`, `learning.reviews.body` and `support.tickets.body`. All three are free text, where a name describes the container, not its contents. And the one marked column the platform worries about most, `billing.payments.card_number`, gets `sensitive` from its name without anyone learning what it holds: whole card numbers, the last four digits, or a payment provider's token would all match `card_?(number|num|no)`.
Name rules are brittle in a known way. The authors of Sherlock, a model for detecting column types, observed that "these matching-based approaches often are not robust to dirty data and only detect a limited number of types" ([Hulsebos et al., 2019](https://arxiv.org/abs/1905.10688)). So the platform reads values too, in the 29 text columns its catalog holds.
**Working rule:** Run name rules over every column first, because they're free, and let them raise a classification, never lower one. Keep the rules short, in code, and reviewed like any other rule. Treat any free-text column, whatever it's called, as one that needs its values sampled, and classify columns of other types by name alone.
## Run Presidio offline {#run-presidio-offline}
To read values, the platform uses Presidio, an open-source library for finding and masking personal data. Its analyzer runs a set of **recognizers**: regular expressions with checksums, phone-number parsing, and a named-entity model reached through an NLP engine. The engine has two jobs, as Presidio's docs put it: "NER based PII identification, and feature extraction for downstream rule based logic (such as leveraging context words for improved detection)", and its default model is spaCy's `en_core_web_lg` ([Presidio docs](https://github.com/data-privacy-stack/presidio/blob/2.2.364/docs/analyzer/customizing_nlp_models.md)). At 2.2.364, `AnalyzerEngine()` with no arguments downloads that model when it isn't installed ([source](https://github.com/data-privacy-stack/presidio/blob/2.2.364/presidio-analyzer/presidio_analyzer/nlp_engine/spacy_nlp_engine.py)), so a pipeline without network access fails on its first line, and one with access installs a model of about 400 MB as a side effect ([en_core_web_lg 3.8.0](https://github.com/explosion/spacy-models/releases/tag/en_core_web_lg-3.8.0)).
`lib.pii.detector` configures the engine explicitly instead, with `NoOpNlpEngine`, which "loads no external model" ([source](https://github.com/data-privacy-stack/presidio/blob/2.2.364/presidio-analyzer/presidio_analyzer/nlp_engine/no_op_nlp_engine.py)). The cost is context words. Presidio raises a pattern's score when a word such as "phone" appears near the match, by comparing "each recognizer's context terms with the lemma of each token in the sentence" ([Presidio docs](https://github.com/data-privacy-stack/presidio/blob/2.2.364/docs/tutorial/06_context.md)), and with no model there are no lemmas, so the engine "cannot use context words from the analyzed text". One more call reaches the network: the email recognizer checks domains with tldextract ([source](https://github.com/data-privacy-stack/presidio/blob/2.2.364/presidio-analyzer/presidio_analyzer/predefined_recognizers/generic/email_recognizer.py)), which "fetches the latest Public Suffix List on first use" ([tldextract 5.4.0 README](https://github.com/john-kurkowski/tldextract/blob/5.4.0/README.md)). An empty `TLDEXTRACT_PUBLIC_SUFFIX_LIST_URLS` turns that off if it's set "before importing `tldextract`", as the README says, and importing Presidio imports tldextract. So `lib/__init__.py`, which Python runs before any module of the book's `lib`, sets it. When an example imported Presidio first, the same file replaces tldextract's shared extractor ([source](https://github.com/john-kurkowski/tldextract/blob/5.4.0/tldextract/tldextract.py)) with one that has no URLs. `lib.pii` refuses to load if any remain, and tldextract uses the copy it ships with, so the same text gives the same answer on every machine.
Load every predefined recognizer on that engine, and run it on one of the platform's support tickets:
Example `presidio-offline-engine` (Python 3.12.15 · duckdb 2.0.0 · phonenumbers 9.0.41 · presidio-analyzer 2.2.364 · pyarrow 26.0.0 · pyiceberg 0.12.0 · pyyaml 6.0.3 · spacy 3.8.16 · tldextract 5.4.0), passing in Edition 2026.10.1:
```python
import spacy
from lib import pii
from lib.platform import load
# AnalyzerEngine() with no arguments loads spaCy's en_core_web_lg, and downloads it when it isn't installed.
print("en_core_web_lg installed:", spacy.util.is_package("en_core_web_lg"))
print("URLs tldextract would fetch the Public Suffix List from:", len(pii.suffix_list_urls()))
detector = pii.detector() # every predefined recognizer, on an NLP engine that loads no model
names = sorted(r.name for r in detector.registry.recognizers)
print("recognizers loaded:", len(names))
print("recognizers:", ", ".join(names))
print("default score threshold:", detector.default_score_threshold)
# The ticket holds a phone number, which may be in service: masked() prints it with only its last four digits.
ticket = pii.texts("tickets")[1421]
found = sorted(pii.detect(detector, ticket))
print("ticket 1421:", pii.masked(ticket))
for start, end, entity, score in found:
print(f" {entity} on {pii.masked(ticket[start:end])}: {score:.2f}")
# Where is the phone number valid? The recognizer reads it in each of its regions.
phone = next(ticket[start:end] for start, end, entity, _ in found if entity == "PHONE_NUMBER")
[learner] = [row["learner_id"] for row in load("support.tickets").to_pylist() if row["ticket_id"] == 1421]
[country] = [row["country"] for row in load("crm.learners").to_pylist() if row["learner_id"] == learner]
print("the learner's country:", country)
print("default regions where it's valid:",
", ".join(r for r in pii.PhoneRecognizer.DEFAULT_SUPPORTED_REGIONS if pii.valid_phone(phone, r)))
```
Output:
```text
en_core_web_lg installed: False
URLs tldextract would fetch the Public Suffix List from: 0
recognizers loaded: 16
recognizers: CreditCardRecognizer, CryptoRecognizer, DateRecognizer, EmailRecognizer, IbanRecognizer, IpRecognizer, MacAddressRecognizer, MedicalLicenseRecognizer, NhsRecognizer, PhoneRecognizer, UrlRecognizer, UsBankRecognizer, UsItinRecognizer, UsLicenseRecognizer, UsPassportRecognizer, UsSsnRecognizer
default score threshold: 0
ticket 1421: I can't open the videos in Introduction to Design Systems, they stop after a few seconds. It happened on 12/03 around 9pm. My colleague Jennifer Abbott (jennifer.abbott@example.net) wants to enrol as well. You can reach me on •••••••0440 after 6pm.
DATE_TIME on 12/03: 0.10
EMAIL_ADDRESS on jennifer.abbott@example.net: 1.00
URL on example.net: 0.50
PHONE_NUMBER on •••••••0440: 0.40
US_BANK_NUMBER on •••••••0440: 0.05
US_DRIVER_LICENSE on •••••••0440: 0.01
the learner's country: US
default regions where it's valid: BR
```
Asked whether `en_core_web_lg` is installed, spaCy answers False, so a bare `AnalyzerEngine()` would try to download it, and tldextract has 0 URLs to fetch its list from. The no-model engine loads 16 recognizers instead, and none of them looks for names: the colleague in the ticket, Jennifer Abbott, goes unnoticed. Her email is found with a score of 1.00, and its domain is found again as a URL, at 0.50. The learner's phone number, an 11-digit number typed without separators, is found three times: as a phone number at 0.40, a US bank number at 0.05 and a US driver's licence at 0.01. Even the phone number is found by chance: the learner's country is US, and of Presidio's default regions, the US among them, the digits are valid only in BR, Brazil. The date of the problem, `12/03`, which says nothing about anyone, is a date at 0.10.
Those scores aren't probabilities. Each pattern carries a constant score, low for weak patterns, as Presidio's tutorial explains: "Zip regex patterns (essentially 5 digits) are very weak, so we would want the initial confidence to be low" ([Presidio docs](https://github.com/data-privacy-stack/presidio/blob/2.2.364/docs/tutorial/06_context.md)). A US bank number is any run of 8 to 17 digits ([supported entities](https://github.com/data-privacy-stack/presidio/blob/2.2.364/docs/supported_entities.md)), and the FAQ admits that "A driver's license number, for example, could be any 9-digit number" ([FAQ](https://github.com/data-privacy-stack/presidio/blob/2.2.364/docs/faq.md)). And the analyzer's default threshold is 0, so every weak match comes back.
**Working rule:** Configure Presidio's engine explicitly: `NoOpNlpEngine` when you run patterns only, or a pinned model you install yourself when you need its names. Never call `AnalyzerEngine()` bare in a pipeline, and pin anything a recognizer would download, such as the Public Suffix List. Read each score as a constant set for its pattern, never as a probability.
## Sample a column's values the same way every run {#sample-column-values}
Scanning every value of every text column is too slow for a warehouse of any size, so tools sample: OpenMetadata's workflow "samples rows directly from the source and validates them against an Entity Recognition engine" ([OpenMetadata docs](https://docs.open-metadata.org/v2.0.x/how-to-guides/data-governance/classification/auto-classification/auto-pii-tagging)). A sample needs three decisions. **Which rows:** a random sample, not the first rows or a block from a random start. Google's Sensitive Data Protection, for one, offers `RANDOM_START`, where "Scanning starts from a randomly selected row" ([Google Cloud docs](https://cloud.google.com/sensitive-data-protection/docs/inspecting-storage)) and reads on from there, and on a table ordered by sign-up date, that's one cohort. **The same rows every run**, so a classification can be reproduced and a change in it means the data changed. And **how many**, from how rare the data you must catch is.
DuckDB's reservoir sample takes a seed, but its docs warn that "fixing the seed only guarantees a reproducible sample when DuckDB runs single-threaded (`SET threads = 1`)" ([DuckDB 2.0.0 docs](https://duckdb.org/docs/2.0/sql/samples)). `lib.pii.sample` sets one thread, filters out NULLs and then draws a seeded reservoir sample. Draw 20 notes from `crm.learners` twice, then work out how many you'd need:
Example `seeded-column-sample` (Python 3.12.15 · duckdb 2.0.0 · phonenumbers 9.0.41 · presidio-analyzer 2.2.364 · pyarrow 26.0.0 · pyiceberg 0.12.0 · pyyaml 6.0.3 · tldextract 5.4.0), passing in Edition 2026.10.1:
```python
import math
from lib import pii
from lib.catalog import open_catalog
from lib.warehouse import connect
con = connect(open_catalog(["crm.learners"]))
first = pii.sample(con, "crm.learners", "notes", rows=20)
second = pii.sample(con, "crm.learners", "notes", rows=20)
print("second draw, same notes as the first:", "yes" if second == first else "no")
# Notes can hold a phone number, which may be in service: masked() prints it with only its last four digits.
print("first sampled notes:", " | ".join(pii.masked(note) for note in first[:3]))
# How rare is personal data in the column? Run the detector over every note, once, to know.
detector = pii.platform_detector()
[(rows, notes)] = con.sql("select count(*), count(notes) from crm.learners").fetchall()
every_note = [note for (note,) in con.sql("select notes from crm.learners where notes is not null").fetchall()]
holding = sum(bool(pii.detect(detector, note)) for note in every_note)
print("learners:", f"{rows:,}")
print("learners with notes:", notes)
print("notes holding personal data the detector finds:", holding)
# The chance that n sampled values hold at least one of them is 1 - (1 - p)^n, for a share p of the values.
for unit, p in (("rows", holding / rows), ("notes", holding / notes)):
print(f"chance 20 sampled {unit} catch one:", f"{1 - (1 - p) ** 20:.1%}")
print(f"sampled {unit} needed for a 99% chance:", math.ceil(math.log(0.01) / math.log(1 - p)))
print("values the platform samples per text column:", pii.ROWS)
print("rarest share that sample catches 99% of the time:", f"1 in {math.floor(1 / (1 - 0.01 ** (1 / pii.ROWS)))}")
```
Output:
```text
second draw, same notes as the first: yes
first sampled notes: VIP: bought a team licence. | Asked about invoices for their employer. | Called about a refund, call back on (•••)•••-1366x•••.
learners: 5,000
learners with notes: 527
notes holding personal data the detector finds: 138
chance 20 sampled rows catch one: 42.9%
sampled rows needed for a 99% chance: 165
chance 20 sampled notes catch one: 99.8%
sampled notes needed for a 99% chance: 16
values the platform samples per text column: 200
rarest share that sample catches 99% of the time: 1 in 43
```
Asked whether a second draw returns the same 20 notes, the example answers yes, and the third note already holds a phone number, which the output masks. Of 5,000 learners, 527 have notes, and 138 of those hold personal data that the platform's detector, [tuned later in this chapter](#tune-the-recognizers), finds. If p is the share of values holding personal data, the chance that n sampled values include at least one is
chance = 1 − (1 − p)ⁿ
and the n that gives a 99% chance is ln(0.01) / ln(1 − p). This is a rule of thumb: it treats the draws as independent, which a sample without replacement from thousands of rows nearly is. Counted over all rows, NULLs included, personal data is rare, and 20 sampled rows catch it only 42.9% of the time; a 99% chance takes 165 rows. Counted over the notes that exist, 20 catch it 99.8% of the time, and 16 are enough for 99%. The NULLs were diluting the sample.
**Working rule:** Sample non-null values, with a fixed seed, on one thread, and keep the seed with the result. Size the sample from the rarest personal data you must catch, with n = ln(0.01) / ln(1 − p) for 99%. The platform samples 200 values per text column, set in `governance/pii.toml`, which catches personal data held by 1 in 43 values or more 99% of the time. For anything rarer, scan the whole column, or classify it by rule.
## Classify columns from their values {#classify-columns-from-their-values}
[The name rules](#classify-columns-from-the-schema) and [a reproducible sample](#sample-column-values) combine into one proposal per column. `lib.pii.classify` takes the strictest classification that either supports: the entity the name suggests, and every entity found in at least one sampled value. A column with no evidence gets `internal`, because a detector that found nothing has shown nothing, and only a person decides that a column is `public`. Strictest is a choice. Presidio's own tool for tables, Presidio structured, gives each column one entity, and its default is "Most Common", which "Identifies the most frequently occurring PII entity in a data column or field" ([Presidio docs](https://github.com/data-privacy-stack/presidio/blob/2.2.364/docs/structured/index.md)). That suits choosing how to mask a column. For classifying it, a column that holds one card number in 200 values still holds card numbers.
Classify all 64 columns of the catalog, sampling the 29 text columns with the platform's detector, whose recognizers [a later section](#tune-the-recognizers) chooses, and compare the proposals with the reviewed metadata:
Example `classify-sampled-columns` (Python 3.12.15 · duckdb 2.0.0 · phonenumbers 9.0.41 · presidio-analyzer 2.2.364 · pyarrow 26.0.0 · pyiceberg 0.12.0 · pyyaml 6.0.3 · tldextract 5.4.0), passing in Edition 2026.10.1:
```python
from lib import metadata, pii
from lib.catalog import open_catalog
from lib.warehouse import connect
catalog = open_catalog()
findings = pii.classify(catalog, connect(catalog), pii.platform_detector())
print("columns classified:", len(findings))
print("columns sampled:", sum(f.sampled > 0 for f in findings))
holding = [f for f in findings if f.found] # columns where sampled values hold personal data
print("columns whose sampled values hold personal data:", len(holding))
print("of them, holding something their name doesn't suggest:", sum(set(f.found) != {f.named} for f in holding))
for f in holding:
print(f"{f.table}.{f.column}, proposed: {f.classification}")
print(f"{f.table}.{f.column}, found: " + ", ".join(f"{e} in {n} of {f.sampled}" for e, n in f.found.most_common()))
[cards] = [f for f in findings if f.column == "card_number"]
print("sampled card numbers the card recognizer accepts:", f"{cards.found['CREDIT_CARD']} of {cards.sampled}")
named_only = [f for f in findings if pii.CLASSIFICATION.get(f.named, pii.FLOOR) != f.classification]
print("columns the values classify more strictly than the name:", ", ".join(f"{f.table}.{f.column}" for f in named_only))
# Compare with the metadata people reviewed. A column the detector found nothing in proposes only the floor, internal.
described = metadata.load()
evidenced = [f for f in findings if f.named or f.found]
print("columns with evidence:", len(evidenced))
print("of them, equal to the metadata:",
sum(f.classification == described[f.table].columns[f.field_id].classification for f in evidenced))
for f in evidenced:
reviewed = described[f.table].columns[f.field_id].classification
if metadata.stricter(f.classification, reviewed) != reviewed:
print(f"stricter than the metadata, {f.table}.{f.column}: {f.classification}, reviewed as {reviewed}")
```
Output:
```text
columns classified: 64
columns sampled: 29
columns whose sampled values hold personal data: 9
of them, holding something their name doesn't suggest: 5
billing.payments.card_number, proposed: sensitive
billing.payments.card_number, found: CREDIT_CARD in 151 of 200, PHONE_NUMBER in 3 of 200
crm.instructors.email, proposed: personal
crm.instructors.email, found: EMAIL_ADDRESS in 50 of 50
crm.instructors.payout_iban, proposed: sensitive
crm.instructors.payout_iban, found: IBAN_CODE in 50 of 50
crm.learners.email, proposed: personal
crm.learners.email, found: EMAIL_ADDRESS in 200 of 200
crm.learners.phone, proposed: personal
crm.learners.phone, found: PHONE_NUMBER in 175 of 200
crm.learners.notes, proposed: personal
crm.learners.notes, found: PHONE_NUMBER in 29 of 200, EMAIL_ADDRESS in 18 of 200
learning.events.ip_address, proposed: personal
learning.events.ip_address, found: IP_ADDRESS in 200 of 200, PHONE_NUMBER in 39 of 200
learning.reviews.body, proposed: personal
learning.reviews.body, found: EMAIL_ADDRESS in 4 of 200
support.tickets.body, proposed: sensitive
support.tickets.body, found: EMAIL_ADDRESS in 47 of 200, PHONE_NUMBER in 37 of 200, DATE_TIME in 29 of 200, IBAN_CODE in 21 of 200, CREDIT_CARD in 12 of 200
sampled card numbers the card recognizer accepts: 151 of 200
columns the values classify more strictly than the name: crm.learners.notes, learning.reviews.body, support.tickets.body
columns with evidence: 14
of them, equal to the metadata: 13
stricter than the metadata, support.tickets.body: sensitive, reviewed as personal
```
Of the 9 columns whose sampled values hold personal data, some hold only what their name says, such as `crm.learners.email`. These 5 hold something their name doesn't suggest, or have a name that suggests nothing:
| Column | Proposed | Found in the sample |
| --- | --- | --- |
| `billing.payments.card_number` | sensitive | CREDIT_CARD in 151 of 200, PHONE_NUMBER in 3 of 200 |
| `crm.learners.notes` | personal | PHONE_NUMBER in 29 of 200, EMAIL_ADDRESS in 18 of 200 |
| `support.tickets.body` | sensitive | EMAIL_ADDRESS in 47 of 200, PHONE_NUMBER in 37 of 200, DATE_TIME in 29 of 200, IBAN_CODE in 21 of 200, CREDIT_CARD in 12 of 200 |
| `learning.reviews.body` | personal | EMAIL_ADDRESS in 4 of 200 |
| `learning.events.ip_address` | personal | IP_ADDRESS in 200 of 200, PHONE_NUMBER in 39 of 200 |
The values settle what the name couldn't. Of the sampled card numbers, 151 of 200 match Presidio's card pattern and pass the Luhn checksum it checks ([source](https://github.com/data-privacy-stack/presidio/blob/2.2.364/presidio-analyzer/presidio_analyzer/predefined_recognizers/generic/credit_card_recognizer.py)), and the rest are formats the pattern doesn't cover ([see where patterns fail](#where-patterns-fail)). The legacy column holds the full card numbers its description says it "Should not be stored at all", and the sample turns that belief into evidence that `team-billing`, the table's owner, can act on. The notes are `personal` on their values alone. The ticket text is `sensitive`, because learners paste card numbers and IBANs into tickets, which makes it the one proposal stricter than the metadata, where people reviewed it as `personal`. Of the 14 columns with any evidence, 13 match the reviewed classification.
Read the evidence, not only the verdict. Some of the IP addresses are also read as phone numbers. Here it changes nothing, because an IP address is `personal` anyway, but it shows that a detector's label says what a value looks like, not what it is.
**Working rule:** Propose the strictest classification that the name or any sampled value supports, with `internal` as the floor, and show the evidence with the proposal. A proposal stricter than the reviewed metadata is a finding for the table's owner, who raises the classification or records why not; never let a detector lower one. OpenMetadata skips a column "already tagged as `PII`"; we recommend sampling every text column on every run instead, because the reviewed tag is the thing being checked.
## Measure a detector against labelled spans {#measure-against-labelled-spans}
A column-level answer is enough to classify `support.tickets.body`, but not to mask it: masking replaces each span of personal data in each text, and a span the detector misses stays readable. To know what a detector misses, compare its spans with [the truth](#finding-personal-data), the labelled spans of every ticket and review. Two matching rules are standard, and `lib.pii` applies both. **Exact** matching, from the CoNLL-2003 shared task, counts an entity "correct only if it is an exact match of the corresponding entity in the data file" ([Tjong Kim Sang and De Meulder, 2003](https://aclanthology.org/W03-0419.pdf)): the detector found the span with the same boundaries and a label of its type. **Overlap** matching asks only for one shared character, still with a label of its type, which the SemEval-2013 task on drug interactions calls "Type matching" ([Segura-Bedmar et al., 2013](https://aclanthology.org/S13-2056.pdf)). Neither credits a span hidden under another label, so `lib.pii` adds a third measure, **coverage**: whether every character of the span is inside some detection, whatever its label. Each answers a different question. Exact asks whether the detector labelled the whole span right, overlap whether it noticed the span as what it is, and coverage whether masking its detections hides the span.
`lib.pii.score` computes exact and overlap matching as **recall**, the share of labelled spans of each type that a detector finds with a label of that type, and **precision**, the share of each label's detections that match a labelled span. `lib.pii.coverage` gives the share of each type's spans that are covered. `score` also counts the texts in which every labelled character falls inside some detection, the texts masking would leave with nothing personal. The Text Anonymization Benchmark counts a person as protected only "if all of its mentions are masked" ([Pilán et al., 2022](https://aclanthology.org/2022.cl-4.19.pdf)); the platform counts whole texts, which is stricter. And recall comes first: Presidio's evaluation guide says "recall is often more important than precision, as we'd like to avoid missing any PII" ([Presidio docs](https://github.com/data-privacy-stack/presidio/blob/2.2.364/docs/evaluation/index.md)).
Run Presidio with its defaults over all 5,000 tickets and reviews, and score its 3,902 detections against the 4,292 labelled spans:
Example `measure-pattern-recognizers` (Python 3.12.15 · duckdb 2.0.0 · phonenumbers 9.0.41 · presidio-analyzer 2.2.364 · pyarrow 26.0.0 · pyiceberg 0.12.0 · pyyaml 6.0.3 · tldextract 5.4.0), passing in Edition 2026.10.1:
```python
import re
from lib import pii
labelled = pii.truth()
print("texts:", f"{sum(len(pii.texts(source)) for source in pii.SOURCES):,}")
print("labelled spans:", f"{len(labelled):,}")
detector = pii.detector() # Presidio's defaults: every recognizer, threshold 0
found = pii.find(detector)
scores, covered = pii.score(found, labelled), pii.coverage(found, labelled)
print("detections:", f"{len(found):,}")
for entity, (n, exact, overlap) in scores.recall.items():
print(f"{entity}, labelled: {n:,}")
print(f"{entity}, recall, exact: {exact / n:.2f}")
print(f"{entity}, recall, overlap: {overlap / n:.2f}")
print(f"{entity}, covered, any label: {covered[entity][1] / n:.2f}")
for label, (n, exact, overlap) in scores.precision.items():
print(f"{label}, detections: {n:,}")
print(f"{label}, precision, exact: {exact / n:.2f}")
print("precision of all detections, exact:", f"{pii.precision_overall(scores):.2f}")
print("texts holding personal data:", f"{scores.texts:,}")
print("texts with every labelled span masked:", f"{scores.masked:,}")
# The truth's limits. The generator builds every text from a few sentence templates, and writes every date of birth
# one way. Presidio's date patterns don't read every way of writing one.
print("sentence templates the texts are built from:", len(pii.templates()))
births = [pii.text_of(s) for s in labelled if s.entity == "DATE_TIME"]
print("dates of birth written like 1985-03-12:", sum(re.fullmatch(r"\d{4}-\d\d-\d\d", d) is not None for d in births))
for written in ("1985-03-12", "12 March 1985"):
text = f"My date of birth on the account is wrong, it should be {written}."
print(f"dates found in a date of birth written {written}:", sum(e == "DATE_TIME" for _, _, e, _ in pii.detect(detector, text)))
```
Output:
```text
texts: 5,000
labelled spans: 4,292
detections: 3,902
CREDIT_CARD, labelled: 213
CREDIT_CARD, recall, exact: 0.71
CREDIT_CARD, recall, overlap: 0.71
CREDIT_CARD, covered, any label: 0.89
DATE_TIME, labelled: 196
DATE_TIME, recall, exact: 1.00
DATE_TIME, recall, overlap: 1.00
DATE_TIME, covered, any label: 1.00
EMAIL_ADDRESS, labelled: 543
EMAIL_ADDRESS, recall, exact: 1.00
EMAIL_ADDRESS, recall, overlap: 1.00
EMAIL_ADDRESS, covered, any label: 1.00
IBAN_CODE, labelled: 236
IBAN_CODE, recall, exact: 1.00
IBAN_CODE, recall, overlap: 1.00
IBAN_CODE, covered, any label: 1.00
LOCATION, labelled: 210
LOCATION, recall, exact: 0.00
LOCATION, recall, overlap: 0.00
LOCATION, covered, any label: 0.00
PERSON, labelled: 2,455
PERSON, recall, exact: 0.00
PERSON, recall, overlap: 0.00
PERSON, covered, any label: 0.00
PHONE_NUMBER, labelled: 439
PHONE_NUMBER, recall, exact: 0.78
PHONE_NUMBER, recall, overlap: 0.79
PHONE_NUMBER, covered, any label: 0.91
CREDIT_CARD, detections: 151
CREDIT_CARD, precision, exact: 1.00
DATE_TIME, detections: 421
DATE_TIME, precision, exact: 0.47
EMAIL_ADDRESS, detections: 543
EMAIL_ADDRESS, precision, exact: 1.00
IBAN_CODE, detections: 236
IBAN_CODE, precision, exact: 1.00
MEDICAL_LICENSE, detections: 18
MEDICAL_LICENSE, precision, exact: 0.00
PHONE_NUMBER, detections: 384
PHONE_NUMBER, precision, exact: 0.89
UK_NHS, detections: 8
UK_NHS, precision, exact: 0.00
URL, detections: 751
URL, precision, exact: 0.00
US_BANK_NUMBER, detections: 689
US_BANK_NUMBER, precision, exact: 0.00
US_DRIVER_LICENSE, detections: 683
US_DRIVER_LICENSE, precision, exact: 0.00
US_ITIN, detections: 2
US_ITIN, precision, exact: 0.00
US_PASSPORT, detections: 8
US_PASSPORT, precision, exact: 0.00
US_SSN, detections: 8
US_SSN, precision, exact: 0.00
precision of all detections, exact: 0.38
texts holding personal data: 2,586
texts with every labelled span masked: 425
sentence templates the texts are built from: 30
dates of birth written like 1985-03-12: 196
dates found in a date of birth written 1985-03-12: 1
dates found in a date of birth written 12 March 1985: 0
```
| Entity type | Labelled | Recall, exact | Recall, overlap | Covered |
| --- | --- | --- | --- | --- |
| `PERSON` | 2,455 | 0.00 | 0.00 | 0.00 |
| `EMAIL_ADDRESS` | 543 | 1.00 | 1.00 | 1.00 |
| `PHONE_NUMBER` | 439 | 0.78 | 0.79 | 0.91 |
| `IBAN_CODE` | 236 | 1.00 | 1.00 | 1.00 |
| `CREDIT_CARD` | 213 | 0.71 | 0.71 | 0.89 |
| `LOCATION` | 210 | 0.00 | 0.00 | 0.00 |
| `DATE_TIME` | 196 | 1.00 | 1.00 | 1.00 |
The patterns find every email, IBAN and date of birth, most phone numbers and card numbers, and no names or street addresses at all, which is most of what the tickets hold. Precision is poor: 0.38 of all detections match a labelled span. The 689 US bank numbers, 683 driver's licences and 751 URLs are all wrong, with a precision of 0.00: phone, card and invoice numbers, all runs of digits, are read as bank numbers and licences, and pieces of emails and a few addresses as URLs. A wrong label still hides the digits, so phone and card numbers are covered more often than they're found. Of the 2,586 texts that hold personal data, 425 would come out of masking with nothing personal left.
The truth has its own limits, because the text is generated. Every ticket and review is put together from 30 sentence templates in `prep/generate.py`, so each [hard case](#where-patterns-fail) is one sentence frame, and all 196 dates of birth are written like `1985-03-12`. A recall of 1.00 for dates means that one format: in the generator's sentence, Presidio finds 1 date when the date of birth is written `1985-03-12`, and 0 when it's written `12 March 1985`. A measure on this text tells you how a detector behaves on these shapes, not how it will do on yours.
**Working rule:** Measure every detector against labelled spans before it masks anything: recall per entity type by exact and by overlap matching, both with a label of the right type, coverage whatever the label, precision per label, and the share of texts with nothing personal left. Weigh recall first. Before you trust a detector on your text, label a few hundred real records by hand and measure on those too.
## Tune the recognizers against the truth {#tune-the-recognizers}
[Presidio's defaults](#measure-against-labelled-spans) return every weak match, and the FAQ names the remedies: "1. Change the acceptance threshold", and "2. Remove unnecessary PII recognizers, if the dataset does not contain these entities", noting that "there is a trade-off between false positives (falsely detected text) and false negatives" ([FAQ](https://github.com/data-privacy-stack/presidio/blob/2.2.364/docs/faq.md)). Presidio 2.2.364 adds "per-recognizer and per-entity score threshold configuration", with the analyzer's default threshold "as the fallback" ([changelog](https://github.com/data-privacy-stack/presidio/blob/2.2.364/CHANGELOG.md), [release notes](https://github.com/data-privacy-stack/presidio/releases/tag/2.2.364)). The phone recognizer has one more setting that matters on a platform whose learners live in many countries: the regions whose numbering plans it reads ([source](https://github.com/data-privacy-stack/presidio/blob/2.2.364/presidio-analyzer/presidio_analyzer/predefined_recognizers/generic/phone_recognizer.py)).
`governance/pii.toml` keeps 6 recognizers, for the identifiers the platform's text can hold, each with its own threshold. The phone regions aren't a setting: `lib.pii.countries` reads the countries the learners live in from `crm.learners`, so the regions follow the data. Run four settings against the same truth:
Example `tune-pattern-recognizers` (Python 3.12.15 · duckdb 2.0.0 · phonenumbers 9.0.41 · presidio-analyzer 2.2.364 · pyarrow 26.0.0 · pyiceberg 0.12.0 · pyyaml 6.0.3 · tldextract 5.4.0), passing in Edition 2026.10.1:
```python
from lib import pii
labelled = pii.truth()
print("recognizers kept:", len(pii.THRESHOLDS))
print("recognizers kept, with their thresholds:", ", ".join(f"{name} {t}" for name, t in pii.THRESHOLDS.items()))
print("phone regions, Presidio's default:", ", ".join(pii.PhoneRecognizer.DEFAULT_SUPPORTED_REGIONS))
print("phone regions, the learners' countries:", ", ".join(pii.countries()))
settings = {
"Presidio's defaults": pii.detector(),
"one threshold of 0.5": pii.detector(default_threshold=0.5),
"kept recognizers, own thresholds": pii.detector(pii.THRESHOLDS),
"kept recognizers, learners' phone regions": pii.platform_detector(),
}
for name, detector in settings.items():
found = pii.find(detector)
scores, covered = pii.score(found, labelled), pii.coverage(found, labelled)
n, exact, _ = scores.recall["PHONE_NUMBER"]
print(f"{name}, detections: {len(found):,}")
print(f"{name}, precision, exact: {pii.precision_overall(scores):.2f}")
print(f"{name}, phone recall, exact: {exact / n:.2f}")
for entity, short in (("PHONE_NUMBER", "phones"), ("CREDIT_CARD", "cards")):
print(f"{name}, {short} covered, any label: {covered[entity][1] / covered[entity][0]:.2f}")
print(f"{name}, texts with every labelled span masked: {scores.masked:,}")
```
Output:
```text
recognizers kept: 6
recognizers kept, with their thresholds: EmailRecognizer 0.5, PhoneRecognizer 0.4, CreditCardRecognizer 0.5, IbanRecognizer 0.5, DateRecognizer 0.5, IpRecognizer 0.5
phone regions, Presidio's default: US, GB, DE, FR, IL, IN, CA, BR
phone regions, the learners' countries: AU, BR, CA, DE, ES, FR, GB, IN, JP, MX, PL, US
Presidio's defaults, detections: 3,902
Presidio's defaults, precision, exact: 0.38
Presidio's defaults, phone recall, exact: 0.78
Presidio's defaults, phones covered, any label: 0.91
Presidio's defaults, cards covered, any label: 0.89
Presidio's defaults, texts with every labelled span masked: 425
one threshold of 0.5, detections: 1,903
one threshold of 0.5, precision, exact: 0.59
one threshold of 0.5, phone recall, exact: 0.00
one threshold of 0.5, phones covered, any label: 0.01
one threshold of 0.5, cards covered, any label: 0.71
one threshold of 0.5, texts with every labelled span masked: 284
kept recognizers, own thresholds, detections: 1,510
kept recognizers, own thresholds, precision, exact: 0.97
kept recognizers, own thresholds, phone recall, exact: 0.78
kept recognizers, own thresholds, phones covered, any label: 0.78
kept recognizers, own thresholds, cards covered, any label: 0.72
kept recognizers, own thresholds, texts with every labelled span masked: 394
kept recognizers, learners' phone regions, detections: 1,539
kept recognizers, learners' phone regions, precision, exact: 0.97
kept recognizers, learners' phone regions, phone recall, exact: 0.84
kept recognizers, learners' phone regions, phones covered, any label: 0.84
kept recognizers, learners' phone regions, cards covered, any label: 0.72
kept recognizers, learners' phone regions, texts with every labelled span masked: 405
```
| Measure | Defaults | One threshold, 0.5 | Kept recognizers | And learners' regions |
| --- | --- | --- | --- | --- |
| Detections | 3,902 | 1,903 | 1,510 | 1,539 |
| Precision, exact | 0.38 | 0.59 | 0.97 | 0.97 |
| Phone recall, exact | 0.78 | 0.00 | 0.78 | 0.84 |
| Phones covered | 0.91 | 0.01 | 0.78 | 0.84 |
| Cards covered | 0.89 | 0.71 | 0.72 | 0.72 |
| Texts fully masked | 425 | 284 | 394 | 405 |
The obvious fix fails. One threshold of 0.5 removes the weak matches and every phone number with them, because a phone number scores 0.40 when no context word can raise it. Keeping 6 recognizers, with the phone recognizer's threshold at its own score, raises precision to 0.97 and keeps phone recall where it was. Presidio's default regions are US, GB, DE, FR, IL, IN, CA, BR; the learners live in AU, BR, CA, DE, ES, FR, GB, IN, JP, MX, PL, US, and reading their numbering plans raises phone recall to 0.84.
One number goes the wrong way: fewer texts come out fully masked than with the defaults. The recognizers the platform dropped had been masking some phone and card numbers under the wrong label, and a wrong label still hides the digits: without them, coverage falls from 0.91 to 0.78 for phone numbers and from 0.89 to 0.72 for card numbers. We think precision of 0.97 is worth that price, because detections are the evidence a reviewer reads, and with the defaults the tickets and reviews hold 689 bank numbers and 683 driver's licences, none of them real. `governance/pii.toml` maps neither entity to a classification, so they change no proposal, but each is a finding someone has to read and dismiss. Dropping the US recognizers is also a claim about the platform's text: the truth holds no US bank or licence numbers. If the platform ever asks for one, that claim, and the measurement, must change.
**Working rule:** Never run with Presidio's default threshold of 0, or with one threshold for every recognizer. Keep only the recognizers for identifiers your text can hold, set each one's threshold from its own scores, and read phone numbers in the regions your users live in, derived from your data. Put all of it in code, and measure again whenever it changes.
## See where patterns fail {#where-patterns-fail}
[The tuned recognizers](#tune-the-recognizers) score well on totals, and totals hide where they fail. A pattern detector fails in three ways: there's no pattern for the thing at all, its pattern or its validation rejects the value, or it matches something that only looks personal. The platform's tickets plant cases of each kind, as its data card says: names typed in lowercase, initials, first names that are ordinary words, phone numbers typed without separators, and invoice numbers, order numbers and dates of a problem, which [the truth](#finding-personal-data) leaves unlabelled. Count each with the platform's detector. A case counts as caught when some detection overlaps it, whatever its label:
Example `pattern-hard-cases` (Python 3.12.15 · duckdb 2.0.0 · phonenumbers 9.0.41 · presidio-analyzer 2.2.364 · pyarrow 26.0.0 · pyiceberg 0.12.0 · pyyaml 6.0.3 · tldextract 5.4.0), passing in Edition 2026.10.1:
```python
from collections import Counter
from lib import pii
from lib.platform import load
labelled = pii.truth()
found = pii.find(pii.platform_detector())
cases = pii.hard_cases(labelled) # a case is caught when some detection overlaps it, whatever its label
for case, spans in cases.items():
print(f"{case}, labelled: {len(spans)}")
print(f"{case}, caught: {pii.caught(spans, found)}")
print("hard cases of a name, caught:", pii.caught([s for spans in cases.values() for s in spans if s.entity == "PERSON"], found))
for kind, spans in pii.lookalikes().items():
print(f"{kind}, in the tickets: {len(spans)}")
print(f"{kind}, flagged as personal data: {pii.caught(spans, found)}")
# Card numbers the card pattern misses: too short or long for its 13 to 17 digits, or a first digit it doesn't accept.
missed = pii.missed([s for s in labelled if s.entity == "CREDIT_CARD"], found, {"CREDIT_CARD"})
why = lambda digits: f"{len(digits)} digits" if not 13 <= len(digits) <= 17 else f"{len(digits)} digits starting with {digits[0]}"
print("CREDIT_CARD, missed:", len(missed))
for reason, n in Counter(why(pii.text_of(s)) for s in missed).most_common():
print(f"CREDIT_CARD, missed, {reason}: {n}")
print("IBAN_CODE, missed:", len(pii.missed([s for s in labelled if s.entity == "IBAN_CODE"], found, {"IBAN_CODE"})))
# Are the phone numbers the patterns miss valid where the ticket's learner lives? Phones appear only in tickets.
learner_of = {row["ticket_id"]: row["learner_id"] for row in load("support.tickets").to_pylist()}
country_of = {row["learner_id"]: row["country"] for row in load("crm.learners").to_pylist()}
phones = pii.missed([s for s in labelled if s.entity == "PHONE_NUMBER"], found, {"PHONE_NUMBER"})
print("phone numbers missed:", len(phones))
print("of them, invalid in the learner's country:",
sum(not pii.valid_phone(pii.text_of(s), country_of[learner_of[s.row_id]]) for s in phones))
```
Output:
```text
names in lowercase, labelled: 211
names in lowercase, caught: 0
initial and surname, labelled: 163
initial and surname, caught: 0
first names that are words, labelled: 187
first names that are words, caught: 0
instructors named in reviews, labelled: 798
instructors named in reviews, caught: 0
phones typed as digits only, labelled: 257
phones typed as digits only, caught: 220
hard cases of a name, caught: 0
invoice numbers, in the tickets: 224
invoice numbers, flagged as personal data: 31
order numbers, in the tickets: 212
order numbers, flagged as personal data: 0
dates of a problem, in the tickets: 225
dates of a problem, flagged as personal data: 0
CREDIT_CARD, missed: 62
CREDIT_CARD, missed, 19 digits: 23
CREDIT_CARD, missed, 12 digits: 22
CREDIT_CARD, missed, 16 digits starting with 2: 10
CREDIT_CARD, missed, 15 digits starting with 2: 7
IBAN_CODE, missed: 0
phone numbers missed: 65
of them, invalid in the learner's country: 63
```
There is no pattern for a name, so every hard case of a name is missed. The tickets hold 211 names typed in lowercase, 163 initials with a surname and 187 first names such as Grace or Will, and the reviews name an instructor 798 times; the patterns catch 0 of them. Phones typed as digits only do well, 220 of 257, and the price is the look-alikes: 31 of the 224 invoice numbers are read as phone numbers. The 212 order numbers are flagged 0 times, and the 225 dates of a problem, such as `12/03`, 0 times. With [Presidio's defaults](#run-presidio-offline), `12/03` was a date at 0.10, and the date recognizer returned 421 dates where the truth labels 196; its threshold in `governance/pii.toml` keeps such weak matches out.
The card pattern's shape causes the card misses. It accepts 13 to 17 digits starting with 1, 3, 4, 50 to 55 or 6, but no 13-digit number starting with 1 ([source](https://github.com/data-privacy-stack/presidio/blob/2.2.364/presidio-analyzer/presidio_analyzer/predefined_recognizers/generic/credit_card_recognizer.py)), and every card number in the data passes the Luhn checksum, so validation rejects none. Of the 62 card numbers it misses, 23 have 19 digits and 22 have 12, and the rest start with 2: 10 of 16 digits and 7 of 15. Validation causes the other misses. The IBAN recognizer checks each country's format and length and the IBAN's check digits ([source](https://github.com/data-privacy-stack/presidio/blob/2.2.364/presidio-analyzer/presidio_analyzer/predefined_recognizers/generic/iban_recognizer.py)). Every labelled IBAN passes, so it misses 0, but a learner who mistypes one digit of an IBAN writes a value it rejects. The phone recognizer accepts only numbers that python-phonenumbers calls valid ([source](https://github.com/daviddrysdale/python-phonenumbers/blob/v9.0.41/python/phonenumbers/phonenumbermatcher.py)), and the generator doesn't always write those: of the 65 phone numbers the platform's detector misses, 63 aren't valid in the country the ticket's learner lives in, so most of the phone misses are the generator's, not the detector's.
**Working rule:** Read a detector's misses by case, not by total, and keep a list of the cases your text holds. Use patterns for what has a checkable shape, such as emails, IBANs, card numbers, phone numbers and dates, and never for names or addresses. Keep the checksums and numbering plans, because they're what makes the patterns precise. Check the card pattern against the card numbers your payment provider issues, and the date patterns against the ways your users write dates.
## Compare a PII model {#compare-a-named-entity-model}
[The patterns](#where-patterns-fail) have nothing for names or addresses, and those are most of the personal data in the platform's text: 2,455 names and 210 addresses among the 4,292 labelled spans. Finding them takes a model that reads context, and a general named-entity model is the wrong one. The authors of the SPY dataset "define PII entities as those that can be used to identify, contact, or locate a specific individual", and found that "dedicated NER models exhibit limitations when applied to PII-specific contexts" ([Savkin et al., 2025](https://aclanthology.org/2025.naacl-srw.23.pdf)). We use a PII model, one trained for personally identifiable information: OpenAI's Privacy Filter, an Apache-2.0 model with eight fixed labels, such as `private_person`, `private_address` and `account_number` ([model card](https://huggingface.co/openai/privacy-filter/blob/7ffa9a043d54d1be65afb281eddf0ffbe629385b/README.md)). The labels are the model's: it "does not support configuring label policies dynamically at runtime" ([OpenAI, 2026](https://cdn.openai.com/pdf/c66281ed-b638-456a-8ce1-97e9f5264a90/OpenAI-Privacy-Filter-Model-Card.pdf), section 4.2). We chose it over the runner-up among open PII models, GLiNER2-PII, for those fixed labels: GLiNER2-PII "conditions on the labels you provide at inference time" and takes a threshold ([model card](https://huggingface.co/fastino/gliner2-privacy-filter-PII-multi/blob/1cb4166094dc58fa8d836429f060d6c95f62b495/README.md)), so its recording would have to pin both. Privacy Filter has 1.5 billion parameters, 50 million of them active, and runs on a CPU.
The book's examples never run a model, so they read a recording of its output. The recording, `data/recorded/pii-entities/`, which this chapter and [chapter 9](#replace-personal-data) use, holds every span the model found in every ticket and review, with its offsets, its text and the model's label. `recordings/pii_entities.py` made it on a CPU with the model's reference decoder, `opf` ([code](https://github.com/openai/privacy-filter/tree/f7f00ca7fb869683eb732c010299d901457f19c3)), which chooses the best sequence of labels for a whole text, so most spans come back in one piece rather than token by token. It used the default operating point, all six of the decoder's transition biases at 0 ([calibration file](https://huggingface.co/openai/privacy-filter/blob/7ffa9a043d54d1be65afb281eddf0ffbe629385b/original/viterbi_calibration.json)), which can be set "for improved recall" instead (section 2.3.2). The manifest pins the model's revision and the SHA-256 of its weights, so anyone can record it again and compare, and `lib.pii.recorded` refuses spans that don't match the text at their offsets. The recording keeps the model's labels. The example maps them to the truth's entity types, `account_number` to both card numbers and IBANs, so a different mapping never needs a new recording. Score the model, the platform's patterns, and both together, masking whatever either finds:
Example `privacy-filter-spans` (Python 3.12.15 · duckdb 2.0.0 · phonenumbers 9.0.41 · presidio-analyzer 2.2.364 · pyarrow 26.0.0 · pyiceberg 0.12.0 · pyyaml 6.0.3 · tldextract 5.4.0), passing in Edition 2026.10.1:
```python
from lib import pii
# Privacy Filter's eight labels, mapped to the truth's entity types. A label the truth has no type for matches nothing.
MATCHES = {"private_person": {"PERSON"}, "private_email": {"EMAIL_ADDRESS"}, "private_phone": {"PHONE_NUMBER"},
"private_address": {"LOCATION"}, "account_number": {"CREDIT_CARD", "IBAN_CODE"},
"private_date": {"DATE_TIME"}, "private_url": set(), "secret": set()}
model, manifest = pii.recorded("pii-entities")
print("model:", manifest["model"], "decoded with", manifest["settings"]["decode_mode"])
labelled = pii.truth()
detectors = {"patterns": pii.find(pii.platform_detector()), "model": model}
detectors["both"] = detectors["patterns"] + model # mask whatever either one finds
for name, found in detectors.items():
scores = pii.score(found, labelled, MATCHES) # Presidio's labels aren't in MATCHES, so match themselves
print(f"{name}, detections: {len(found):,}")
for entity, (n, exact, overlap) in scores.recall.items():
print(f"{name}, {entity}, recall, exact: {exact / n:.2f}")
print(f"{name}, {entity}, recall, overlap: {overlap / n:.2f}")
print(f"{name}, precision, exact: {pii.precision_overall(scores):.2f}")
print(f"{name}, texts with every labelled span masked: {scores.masked:,} of {scores.texts:,}")
for label, (n, exact, overlap) in pii.score(model, labelled, MATCHES).precision.items():
print(f"model, {label}, detections: {n:,}")
print(f"model, {label}, precision, exact: {exact / n:.2f}")
print(f"model, {label}, precision, overlap: {overlap / n:.2f}")
```
Output:
```text
model: openai/privacy-filter decoded with viterbi
patterns, detections: 1,539
patterns, CREDIT_CARD, recall, exact: 0.71
patterns, CREDIT_CARD, recall, overlap: 0.71
patterns, DATE_TIME, recall, exact: 1.00
patterns, DATE_TIME, recall, overlap: 1.00
patterns, EMAIL_ADDRESS, recall, exact: 1.00
patterns, EMAIL_ADDRESS, recall, overlap: 1.00
patterns, IBAN_CODE, recall, exact: 1.00
patterns, IBAN_CODE, recall, overlap: 1.00
patterns, LOCATION, recall, exact: 0.00
patterns, LOCATION, recall, overlap: 0.00
patterns, PERSON, recall, exact: 0.00
patterns, PERSON, recall, overlap: 0.00
patterns, PHONE_NUMBER, recall, exact: 0.84
patterns, PHONE_NUMBER, recall, overlap: 0.85
patterns, precision, exact: 0.97
patterns, texts with every labelled span masked: 405 of 2,586
model, detections: 4,424
model, CREDIT_CARD, recall, exact: 0.69
model, CREDIT_CARD, recall, overlap: 0.69
model, DATE_TIME, recall, exact: 0.97
model, DATE_TIME, recall, overlap: 0.97
model, EMAIL_ADDRESS, recall, exact: 1.00
model, EMAIL_ADDRESS, recall, overlap: 1.00
model, IBAN_CODE, recall, exact: 0.99
model, IBAN_CODE, recall, overlap: 1.00
model, LOCATION, recall, exact: 0.67
model, LOCATION, recall, overlap: 0.79
model, PERSON, recall, exact: 0.84
model, PERSON, recall, overlap: 0.84
model, PHONE_NUMBER, recall, exact: 0.94
model, PHONE_NUMBER, recall, overlap: 0.94
model, precision, exact: 0.84
model, texts with every labelled span masked: 2,095 of 2,586
both, detections: 5,963
both, CREDIT_CARD, recall, exact: 0.92
both, CREDIT_CARD, recall, overlap: 0.92
both, DATE_TIME, recall, exact: 1.00
both, DATE_TIME, recall, overlap: 1.00
both, EMAIL_ADDRESS, recall, exact: 1.00
both, EMAIL_ADDRESS, recall, overlap: 1.00
both, IBAN_CODE, recall, exact: 1.00
both, IBAN_CODE, recall, overlap: 1.00
both, LOCATION, recall, exact: 0.67
both, LOCATION, recall, overlap: 0.79
both, PERSON, recall, exact: 0.84
both, PERSON, recall, overlap: 0.84
both, PHONE_NUMBER, recall, exact: 0.99
both, PHONE_NUMBER, recall, overlap: 0.99
both, precision, exact: 0.88
both, texts with every labelled span masked: 2,120 of 2,586
model, account_number, detections: 746
model, account_number, precision, exact: 0.51
model, account_number, precision, overlap: 0.51
model, private_address, detections: 177
model, private_address, precision, exact: 0.79
model, private_address, precision, overlap: 1.00
model, private_date, detections: 384
model, private_date, precision, exact: 0.50
model, private_date, precision, overlap: 0.50
model, private_email, detections: 543
model, private_email, precision, exact: 1.00
model, private_email, precision, overlap: 1.00
model, private_person, detections: 2,110
model, private_person, precision, exact: 0.98
model, private_person, precision, overlap: 0.98
model, private_phone, detections: 464
model, private_phone, precision, exact: 0.89
model, private_phone, precision, overlap: 0.89
```
| Measure | Patterns | Model | Both |
| --- | --- | --- | --- |
| Recall, exact, `PERSON` | 0.00 | 0.84 | 0.84 |
| Recall, exact, `LOCATION` | 0.00 | 0.67 | 0.67 |
| Recall, exact, `PHONE_NUMBER` | 0.84 | 0.94 | 0.99 |
| Recall, exact, `CREDIT_CARD` | 0.71 | 0.69 | 0.92 |
| Recall, exact, `IBAN_CODE` | 1.00 | 0.99 | 1.00 |
| Precision, exact | 0.97 | 0.84 | 0.88 |
| Texts fully masked | 405 of 2,586 | 2,095 of 2,586 | 2,120 of 2,586 |
The model finds most of the names and addresses the patterns can't. Card numbers are where the two detectors complement each other: the model and the patterns find about the same share, 0.69 and 0.71, but not the same cards, so together they find 0.92; [a later section](#what-a-wrong-label-costs) shows why the model's share is low. The model pays in precision, 0.84 overall, and only 0.51 for `account_number` and 0.50 for `private_date`. That partly measures a difference in label policy: the truth leaves invoice numbers and dates of a problem unlabelled, and the card warns that "definitions of appropriate label policies and decision boundaries can differ" (section 4.2). Together, the two detectors leave 2,120 of 2,586 texts with nothing personal in them, against 405 of 2,586 for the patterns alone.
### Read the model's misses {#read-the-models-misses}
OpenAI's card for Privacy Filter lists where it fails: "under-detection of uncommon personal names, regional naming conventions, initials, honorific-heavy references, or domain-specific identifiers; over-redaction of public entities, locations, or common nouns when local context is ambiguous" ([OpenAI, 2026](https://cdn.openai.com/pdf/c66281ed-b638-456a-8ce1-97e9f5264a90/OpenAI-Privacy-Filter-Model-Card.pdf), section 4.3). Check that list against [the hard cases](#where-patterns-fail) the tickets plant:
Example `privacy-filter-hard-cases` (Python 3.12.15 · duckdb 2.0.0 · phonenumbers 9.0.41 · presidio-analyzer 2.2.364 · pyarrow 26.0.0 · pyiceberg 0.12.0 · pyyaml 6.0.3 · tldextract 5.4.0), passing in Edition 2026.10.1:
```python
from collections import Counter
from lib import pii
model, _ = pii.recorded("pii-entities")
labelled = pii.truth()
cases = pii.hard_cases(labelled)
for case, spans in cases.items():
print(f"{case}, labelled: {len(spans)}")
print(f"{case}, caught by the model: {pii.caught(spans, model)}")
for kind, spans in pii.lookalikes().items():
print(f"{kind}, flagged by the model: {pii.caught(spans, model)}")
# The names the model misses, by case: read them before trusting a recall figure.
missed = pii.missed([s for s in labelled if s.entity == "PERSON"], model, {"private_person"})
print("names the model misses:", len(missed))
for case in ("first names that are words", "instructors named in reviews"):
often = Counter(pii.text_of(s) for s in pii.missed(cases[case], model, {"private_person"})).most_common(3)
print(f"{case}, missed most often:", ", ".join(name for name, _ in often))
# A first name on its own says less than a full name: how many of the misses are one word?
print("missed names that are one word:", sum(len(pii.text_of(s).split()) == 1 for s in missed))
print("missed names of two words or more:", sum(len(pii.text_of(s).split()) > 1 for s in missed))
print("of them, typed in lowercase:", sum(len(pii.text_of(s).split()) > 1 and pii.text_of(s).islower() for s in missed))
# A lowercase name that neither the model nor the platform's patterns cover any part of, in the first ticket with one.
first = min(pii.missed(cases["names in lowercase"], model + pii.find(pii.platform_detector())), key=lambda s: s.row_id)
print("first ticket with a lowercase name both detectors miss:", first.row_id)
print("the name, as typed:", pii.text_of(first))
```
Output:
```text
names in lowercase, labelled: 211
names in lowercase, caught by the model: 167
initial and surname, labelled: 163
initial and surname, caught by the model: 162
first names that are words, labelled: 187
first names that are words, caught by the model: 128
instructors named in reviews, labelled: 798
instructors named in reviews, caught by the model: 535
phones typed as digits only, labelled: 257
phones typed as digits only, caught by the model: 256
invoice numbers, flagged by the model: 224
order numbers, flagged by the model: 125
dates of a problem, flagged by the model: 188
names the model misses: 387
first names that are words, missed most often: Summer, Will, Hope
instructors named in reviews, missed most often: Marie, Lori, Augustin
missed names that are one word: 337
missed names of two words or more: 50
of them, typed in lowercase: 43
first ticket with a lowercase name both detectors miss: 19
the name, as typed: juan fonseca
```
On the platform's text, initials aren't the problem: the model catches 162 of 163. Lowercase names are mostly caught, 167 of 211. The misses are first names that are also words, 128 of 187 caught, and an instructor's first name alone in a review, 535 of 798 caught. Most often it misses the words Summer, Will, Hope and the instructors' names Marie, Lori, Augustin. The model also flags far more look-alikes than the tuned patterns: 224 invoice numbers, 125 order numbers and 188 dates of a problem, against 31, 0 and 0.
Reading the misses also says how much they matter. Of the 387 names the model misses, 337 are a single word, a first name on its own, and 50 are longer, 43 of those typed in lowercase. The authors of the Text Anonymization Benchmark draw the same line: "failing to detect a direct identifier such as a full person name is much more harmful from a privacy perspective than failing to detect a quasi-identifier" ([Pilán et al., 2022](https://aclanthology.org/2022.cl-4.19.pdf)). A first name alone still needs masking, since a review carries its learner's ID and an instructor's first name in a review of their own course points at one person. The card's own advice is to measure: "Before production use, it's best to evaluate the model in-domain against local policy references" (section 4.5). It also lists education among the settings where "Additional caution is warranted" (section 4.4), and calls the model "a redaction and data minimization aid, not an anonymization, compliance, or a safety guarantee" (section 4.1).
**Working rule:** Mask free text with a PII model and the tuned patterns together, whatever either finds: the model for names and addresses, the patterns' checksums for card numbers and IBANs. Record the model's output with its revision, weights and operating point pinned, keep its own labels, and map them in code. Measure it on your own text, by hard case, before use and after every model change; if it misses too much, try an operating point that leans toward recall and record again. Treat masked text as reduced, never as anonymous.
## Count what a wrong label costs {#what-a-wrong-label-costs}
[The comparison with the patterns](#compare-a-named-entity-model) scored the model by exact and overlap matching, which both need a label of the span's type. Masking doesn't read the label: a card number the model labels `private_phone` is still hidden. Classification does, because `governance/pii.toml` maps each entity type to a classification, a phone number to `personal` and a card number to `sensitive`. Measure the model's coverage beside its recall, and the labels it gives the platform's card numbers, phone numbers and addresses:
Example `privacy-filter-labels` (Python 3.12.15 · duckdb 2.0.0 · phonenumbers 9.0.41 · presidio-analyzer 2.2.364 · pyarrow 26.0.0 · pyiceberg 0.12.0 · pyyaml 6.0.3 · tldextract 5.4.0), passing in Edition 2026.10.1:
```python
from lib import pii
# What the model's labels cost. A span is covered when every character of it is inside some detection, whatever its
# label: that's what masking needs. Exact and overlap matching also need a label of the span's type.
model, _ = pii.recorded("pii-entities")
labelled = pii.truth()
covered = pii.coverage(model, labelled)
for entity in ("CREDIT_CARD", "PHONE_NUMBER", "LOCATION"):
n, fully = covered[entity]
print(f"{entity}, covered by the model, any label: {fully / n:.2f}")
spans = {entity: [s for s in labelled if s.entity == entity] for entity in ("CREDIT_CARD", "PHONE_NUMBER", "LOCATION")}
print("card numbers labelled private_phone:", pii.caught(spans["CREDIT_CARD"], model, {"private_phone"}))
print("phone numbers labelled account_number:", pii.caught(spans["PHONE_NUMBER"], model, {"account_number"}))
# For each address, the labels of the detections that overlap it, and the characters no detection covers.
on = pii.labels_on(spans["LOCATION"], model)
print("addresses with no detection at all:", sum(not labels for labels in on))
print("addresses read, at least in part, as a person:", sum("private_person" in labels for labels in on))
print("of them, with no address label:", sum({"private_person"} <= labels and "private_address" not in labels for labels in on))
left = [pii.uncovered(address, model) for address in spans["LOCATION"]]
print("addresses not fully covered:", sum(bool(chars) for chars in left))
print("of them, with only spaces and punctuation left:", sum(bool(chars) and not any(c.isalnum() for c in chars) for chars in left))
```
Output:
```text
CREDIT_CARD, covered by the model, any label: 0.87
PHONE_NUMBER, covered by the model, any label: 1.00
LOCATION, covered by the model, any label: 0.67
card numbers labelled private_phone: 40
phone numbers labelled account_number: 27
addresses with no detection at all: 24
addresses read, at least in part, as a person: 28
of them, with no address label: 20
addresses not fully covered: 70
of them, with only spaces and punctuation left: 21
```
The model covers 0.87 of card numbers, but labels 40 of the 213 as phone numbers, so its recall for cards is only 0.69. If the platform classified columns from the model's labels, a column of those card numbers would be proposed `personal`, not `sensitive`. That's what the patterns add: their Luhn check keeps card numbers labelled as cards. The mistake also runs the other way, toward the stricter side: the model labels 27 phone numbers `account_number`, which maps to card numbers and IBANs, so its phone coverage is 1.00 and its recall 0.94.
Addresses show what each measure counts. They score 0.67 by exact matching, 0.79 by overlap, and 0.67 by coverage. The model finds nothing at all in 24 addresses. It reads part of 28 as a person, since Faker builds many street names from surnames, as the platform's data card notes, and gives 20 of those no address label at all, so overlap matching misses them too. Other addresses come back in pieces with the right label, such as a street and a town split at a comma, which overlap matching credits and coverage doesn't: of the 70 addresses not fully covered, 21 have every letter and digit masked and only a comma or a space left. Coverage is the measure masking needs, and it's strict.
**Working rule:** Report coverage, whatever the label, beside recall with the right label: masking needs the first, classification the second. Read which labels a detector gives each type of span, and keep a recognizer with a checksum for every identifier whose label decides a classification, such as card numbers.
## Classify again when the schema changes {#reclassify-on-schema-change}
A classification describes a column at one moment, and [chapter 2](#catch-schema-changes) showed how schemas move under it. The question is when to run the classifier again. Not on every schema ID: in PyIceberg, editing a column's description gives the schema a new ID with no new column. What needs classifying is a [field ID](#key-column-metadata-by-field-id), the number Iceberg gives each column, that the metadata has never seen, because Iceberg gives every added column a new one, and a renamed column keeps its ID and so its reviewed classification. Values can also change while the schema stays the same, so the samples of text columns need a rerun on a schedule as well. Vendors promise this as a service: Databricks says its classification's "scanning is incremental and optimized to ensure all new data is classified without manual configuration" ([Databricks docs](https://docs.databricks.com/aws/en/data-governance/unity-catalog/data-classification)), a claim we don't test.
On the platform, the support team wants a number to call learners back on for tickets opened by phone, and adds a column called `callback_number` to `support.tickets`. Edit a description first, then add the column, fill it from the learners' phone numbers, and classify only the field IDs that `governance/metadata.toml` lacks:
Example `reclassify-new-columns` (Python 3.12.15 · duckdb 2.0.0 · phonenumbers 9.0.41 · presidio-analyzer 2.2.364 · pyarrow 26.0.0 · pyiceberg 0.12.0 · pyyaml 6.0.3 · tldextract 5.4.0), passing in Edition 2026.10.1:
```python
from pyiceberg.types import StringType
from lib import metadata, pii
from lib.catalog import fill_column, open_catalog
from lib.platform import load
from lib.warehouse import connect
catalog = open_catalog(["support.tickets"])
tickets = catalog.load_table("support.tickets")
reviewed = metadata.load()["support.tickets"].columns # field ID -> the column as people reviewed it
unclassified = lambda: {f.field_id for f in tickets.schema().fields if f.field_id not in reviewed}
print("schema ID at the start:", tickets.schema().schema_id)
with tickets.update_schema() as update: # a doc edit: a new schema, no new column
update.update_column("body", doc="What the learner wrote, as they wrote it.")
print("schema ID after a doc edit:", tickets.schema().schema_id)
print("fields to classify after the doc edit:", len(unclassified()))
# The support team starts storing the number to call back for tickets opened by phone: the learner's phone number.
with tickets.update_schema() as update:
update.add_column("callback_number", StringType(), doc="Where to call the learner back.")
phone_of = {row["learner_id"]: row["phone"] for row in load("crm.learners").to_pylist()}
fill_column(tickets, "callback_number", lambda row: phone_of[row["learner_id"]] if row["channel"] == "phone" else None)
print("schema ID after adding a column:", tickets.schema().schema_id)
print("fields to classify after adding a column:", len(unclassified()))
for f in pii.classify(catalog, connect(catalog), pii.platform_detector(), only={"support.tickets": unclassified()}):
found = ", ".join(f"{entity} in {n} of {f.sampled}" for entity, n in f.found.items())
print(f"{f.table}.{f.column}, field ID: {f.field_id}")
print(f"{f.table}.{f.column}, entity its name suggests: {f.named}")
print(f"{f.table}.{f.column}, found in its sampled values: {found}")
print(f"{f.table}.{f.column}, proposed classification: {f.classification}")
```
Output:
```text
schema ID at the start: 0
schema ID after a doc edit: 1
fields to classify after the doc edit: 0
schema ID after adding a column: 2
fields to classify after adding a column: 1
support.tickets.callback_number, field ID: 9
support.tickets.callback_number, entity its name suggests: None
support.tickets.callback_number, found in its sampled values: PHONE_NUMBER in 154 of 173
support.tickets.callback_number, proposed classification: personal
```
The doc edit moved the schema ID to 1 and left 0 fields to classify. Adding the column moved it to 2, with 1 field to classify, field 9. No name rule matches it, since `callback_number` contains none of the words the phone rule looks for, which is how a name rule misses a column nobody wrote it for. Its values settle it, PHONE_NUMBER in 154 of 173, and the proposal is personal. The pull request that adds the column is where its classification belongs, beside the metadata entry it needs: [chapter 10](#run-every-rule-as-one-job) fails that pull request until the column has one. A proposal from values, like this one, can come only once the column holds values, so it comes from the sample on a schedule.
**Working rule:** Classify every field ID the metadata doesn't have, in the pull request that adds it, and don't let it merge without a reviewed classification. Key the trigger on field IDs, never on schema IDs. Sample every text column again on a schedule too, because values change without a schema change.
## What goes wrong in production {#personal-data-in-production}
A detector fails quietly. Its output always looks like a list of findings, and what it misses never appears in it:
- **Phone numbers read as bank numbers.** With Presidio's defaults, every run of 8 to 17 digits is a US bank number, and every phone number typed without separators is one. The evidence for the ticket table lists hundreds of bank numbers that aren't there, a report of "bank numbers in tickets" sends someone to the wrong team, and mapping that entity to a classification would raise columns for the wrong reason. Keep only the recognizers your text needs, with thresholds of their own, as [the tuning section](#tune-the-recognizers) does, and notice the problem by counting detections per label: a label with hundreds of hits and no labelled spans is a false positive.
- **Names in free text missed.** Patterns find no names, and [the model](#read-the-models-misses) misses some lowercase names, first names that are also words, and first names standing alone. Ticket 19 still says "this is juan fonseca, i still can't log in" after both detectors have run, and passes every check that counts detections. You notice only by measuring recall against labelled text, per hard case, and by reading a sample of masked output.
- **Personal data in a column called `notes`.** No name rule marks it, and a reviewer who never read it classifies it `internal`. Sample every text column's values, and treat free-text columns as personal until a sample says otherwise.
- **A column more sensitive than its metadata.** Learners paste card numbers into tickets, so `support.tickets.body` holds `sensitive` data under a `personal` classification. Compare every proposal with the reviewed metadata, and route a stricter one to the table's owner.
- **A detector that reaches the network.** `AnalyzerEngine()` downloads a model, and tldextract fetches the Public Suffix List. Offline, the pipeline fails; online, the same text can be classified differently on two machines. A setting read at import holds only if it's made before anything imports the library. Configure the engine, pin what it would fetch, check that the setting held, and run the detector's tests with the network off.
- **Truth shaped by its generator.** The platform's text comes from 30 sentence templates filled by Faker, which writes every date of birth one way, phone numbers that aren't valid where the learner lives, and street names built from people's names. Read the misses before trusting a recall figure, and measure again on labelled samples of your own text.
## What this chapter decided {#personal-data-decisions}
- **Columns are classified from their name, their type and their values.** Name rules run on every column; text columns are sampled; the strictest evidence wins, with `internal` as the floor, and only a person sets `public`.
- **Samples are reproducible:** 200 non-null values per text column, by a seeded reservoir sample on one DuckDB thread.
- **Presidio runs offline,** on `NoOpNlpEngine`, with 6 recognizers, each with its own threshold, phone regions read from the learners' countries, and tldextract's own copy of the Public Suffix List, checked when `lib.pii` loads.
- **Detectors are measured against labelled spans,** by exact and overlap matching with a label of the right type and by coverage whatever the label, as recall per entity type, precision per label and texts left with nothing personal, with recall weighed first.
- **A proposal stricter than the metadata is a finding:** `support.tickets.body` goes to `team-support` as `sensitive`, and `billing.payments.card_number` to `team-billing`, with the evidence that it holds full card numbers.
- **Free text is masked with two detectors:** OpenAI's Privacy Filter, recorded at a pinned revision and operating point, for names and addresses, and the tuned patterns for card numbers, IBANs and phone numbers, whose checksums keep card numbers labelled as cards. Masking hides whatever either finds; neither is trusted to make text anonymous.
- **No example prints a whole phone number, card number or IBAN:** `lib.pii.masked` keeps the last four digits.
- **Reclassification follows field IDs:** every field the metadata lacks is classified in the pull request that adds it, and text columns are sampled again on a schedule.
# Lineage from pipelines
By the end of this chapter you can capture lineage from your pipelines as OpenLineage run events, with column lineage parsed from their SQL, including the columns that only join, filter and group. You'll use the graph those events describe to say what a column change breaks downstream, and to propose classifications for tables nobody has classified. You'll also have checks for the three ways lineage fails in production: events that never arrive, a job that sends none, and a copy of personal data classified as less than personal.
The platform, the online learning platform that [chapter 1](#governance-end-to-end) introduces and this book follows, sells professional courses to adult learners, and four SQL pipelines build reporting tables from its eight source tables. Each is one `create table ... as select ...` in `examples/pipelines/`, and the book's shared module `lib.pipelines` runs it on DuckDB and writes the result back to the catalog:
- `course_performance` writes `analytics.course_performance`: enrollments, completions, ratings and revenue per course.
- `learner_activity` writes `analytics.learner_activity`: each learner's events and last activity.
- `marketing_audience` writes `marketing.audience`: the learners marketing may email, with their name and email.
- `review_corpus` writes `ai.review_corpus`: the review text the platform's course assistant is grounded on.
[Chapter 2](#a-catalog-you-can-test)'s metadata check found that two of those outputs, `marketing.audience` and `ai.review_corpus`, have no metadata at all: no owner, and no classification for any column. Yet one holds learners' names and emails, and the other holds the free text of reviews, where [chapter 3](#classify-columns-from-their-values) found names and emails too. Two questions follow, and the catalog can't answer either. Which columns of those tables are personal data? And if the CRM team renames or reformats `crm.learners.email`, what breaks? Both are questions about lineage: where each column's data came from, and where it goes.
This chapter adds the book's shared module for lineage, `lib.lineage`. It runs each pipeline, tells OpenLineage what the run read and wrote, column by column, and reads the result back as a graph that answers both questions.
## Send a run event from every pipeline {#send-run-events}
OpenLineage, an open specification for lineage, describes it with three objects ([object model](https://openlineage.io/docs/spec/object-model)). A **job** is "a process that consumes or produces Datasets", such as a pipeline. A **run** is one occurrence of a job, identified by a UUID the client generates, and the spec recommends UUIDv7. A **dataset** is a table or file, named by a namespace and a name that together identify it. A run reports itself through **run events**: a START when it begins, then one terminal event, COMPLETE, FAIL or ABORT, and a server treats the events for one run as accumulating ([run cycle](https://openlineage.io/docs/spec/run-cycle)). Lineage comes mostly from the work itself: in the spec's words, a backend "learns about Datasets primarily by receiving information about Jobs".
The Python client, openlineage-python 1.54.0, builds events from `RunEvent`, `Run`, `Job`, `InputDataset` and `OutputDataset`, which come from `openlineage.client.event_v2`, and sends them through a transport ([client docs](https://openlineage.io/docs/client/python)). It ships HTTP and Kafka transports for servers, and a file transport "designed mainly for integration testing", which with `append` set writes one JSON event per line to a single file ([configuration](https://openlineage.io/docs/client/python/configuration)). The client computes no lineage. It serializes what the caller passes in, and adds only metadata of its own: a run facet with its version, the job's source location when it runs in a Git checkout, and any environment variables you configure it to collect (`emit` in `client/client.py`, [openlineage-python](https://pypi.org/project/openlineage-python/1.54.0/)). So the lineage that reaches a server is the lineage your code put in the event.
For the platform, `lib.lineage.run` wraps `lib.pipelines.run`, which runs one of the platform's four SQL pipelines and writes its table to the catalog. It sends START with the tables the pipeline's SQL reads as inputs, runs the pipeline, and then sends COMPLETE with the output dataset and two facets on it. A **facet** is OpenLineage's name for a block of metadata attached to a run, a job or a dataset, and these two hold the output's schema and its column lineage, which the [next section](#trace-output-columns) builds. If the pipeline raises, it sends FAIL instead. `lib.lineage.run_all` does that for all four pipelines into one events file. The datasets need a namespace, and OpenLineage's [naming conventions](https://openlineage.io/docs/spec/naming) have no entry for an Iceberg table in a local SQL catalog. The nearest is the local file system, named by its path, which would put a folder on your machine into every event. So `lib.lineage` uses the placeholder `iceberg://platform`, with the catalog's identifiers, such as `crm.learners`, as names.
The examples need the same bytes every time, and two things in an event change by default: its time, and a random run ID. So `lib.lineage` stamps every event with the nominal start of the nightly run on the book's "today", and derives the run ID with the client's `generate_static_uuid`, which, its docstring says, "always produces the same result" for the same time and data. Here the data is the job's name. Run the four pipelines with lineage twice, into two files, and print what the first file's events say, never the raw JSON:
Example `pipeline-run-events` (Python 3.12.15 · duckdb 2.0.0 · openlineage-python 1.54.0 · pyarrow 26.0.0 · pyiceberg 0.12.0 · pyyaml 6.0.3 · sqlglot 30.24.0), passing in Edition 2026.10.1:
```python
"""Run the platform's four pipelines with OpenLineage run events, then read back what the events say."""
from openlineage.client.uuid import generate_new_uuid, generate_static_uuid
from lib import lineage
from lib.catalog import open_catalog
from lib.platform import scratch
folder = scratch()
catalog = open_catalog()
lineage.run_all(catalog, folder / "first.jsonl")
lineage.run_all(catalog, folder / "second.jsonl") # the same nightly run, sent again
events = lineage.read_events(folder / "first.jsonl")
print("events:", len(events))
for event in events:
job, state = event["job"]["name"], event["eventType"]
if state == "START":
print(f"{job} START, reads:", ", ".join(dataset["name"] for dataset in event["inputs"]))
else:
output = event["outputs"][0]
lineage_fields = output["facets"]["columnLineage"]["fields"]
print(f"{job} {state}, writes:", output["name"], f"({len(lineage_fields)} columns with lineage)")
print("event times:", ", ".join(sorted({event["eventTime"] for event in events})))
print("distinct run IDs:", len({event["run"]["runId"] for event in events}))
print("run facets the client added:", ", ".join(events[0]["run"]["facets"]))
print("same bytes when sent again:", (folder / "first.jsonl").read_bytes() == (folder / "second.jsonl").read_bytes())
print("two random run IDs equal:", generate_new_uuid() == generate_new_uuid())
print("two static run IDs equal:", generate_static_uuid(lineage.NIGHTLY, b"marketing_audience")
== generate_static_uuid(lineage.NIGHTLY, b"marketing_audience"))
```
Output:
```text
events: 8
course_performance START, reads: billing.payments, learning.courses, learning.enrollments, learning.reviews
course_performance COMPLETE, writes: analytics.course_performance (7 columns with lineage)
learner_activity START, reads: crm.learners, learning.events
learner_activity COMPLETE, writes: analytics.learner_activity (6 columns with lineage)
marketing_audience START, reads: crm.learners, learning.courses, learning.enrollments
marketing_audience COMPLETE, writes: marketing.audience (5 columns with lineage)
review_corpus START, reads: learning.courses, learning.reviews
review_corpus COMPLETE, writes: ai.review_corpus (5 columns with lineage)
event times: 2026-10-01T02:00:00+00:00
distinct run IDs: 4
run facets the client added: tags
same bytes when sent again: True
two random run IDs equal: False
two static run IDs equal: True
```
The file holds 8 events, a START and a COMPLETE for each of the 4 runs, all stamped `2026-10-01T02:00:00+00:00`. The `marketing_audience` job's START names the tables its SQL reads, `crm.learners`, `learning.courses` and `learning.enrollments`, and its COMPLETE names `marketing.audience`, with lineage for each of its columns. Comparing the two files prints `True`: sending the same nominal run again wrote the same bytes, which random run IDs would never allow. Even so, the events hold something you didn't put there: a run facet named `tags`, which carries the client's own version ([changelog, 1.38.0](https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md)), and every facet's `_producer` URL names that version too (`DEFAULT_PRODUCER` in the client's `client/constants.py`). So the bytes change with every client release even when the lineage doesn't, and a test should compare what events say, as this example's summary does.
A production emitter stamps COMPLETE with the time the run finished. A derived run ID is still worth keeping there, made from the scheduled time, the job's name and the attempt number: an event sent twice then lands in the same run instead of creating a second one, and a retried job gets a run of its own. Production events should also carry the output's Iceberg snapshot ID, in the `version` dataset facet that the [object model](https://openlineage.io/docs/spec/object-model) names for it. The examples leave it out, because PyIceberg 0.12.0 draws each snapshot ID from a random UUID (`_generate_snapshot_id` in its `pyiceberg/table/metadata.py`).
**Working rule:** Send a START and then a COMPLETE or FAIL event from the code that runs each job, so no run can happen without them, and name datasets by catalog identifiers under a namespace you choose, never a path. In tests, use the file transport with a fixed event time and static run IDs, and compare what events say, never their bytes. In production, send through a transport to your lineage server, and derive each run ID from the scheduled time, the job and the attempt, so a resent event never creates a duplicate run.
## Trace each output column to its sources {#trace-output-columns}
A [run event](#send-run-events) says which tables a job read and wrote. Column lineage says which input columns each output column came from, and that is what governance needs: a table that reads `crm.learners` may or may not copy its emails. OpenLineage puts column lineage in a dataset facet, `columnLineage`, on the output: `fields` maps each output column to its `inputFields`, and an optional `dataset` list holds inputs that affect the whole dataset ([facet schema](https://openlineage.io/spec/facets/1-2-0/ColumnLineageDatasetFacet.json)). Since the client computes none of it, something has to parse each job's SQL.
sqlglot, a SQL parser, does it with `sqlglot.lineage.lineage`. Given a statement and the schema of the tables it reads, it returns, for each output column, a tree whose leaves are source columns, and with `column=None` it returns every output column at once ([sqlglot lineage](https://sqlglot.com/sqlglot/lineage.html)). It needs the schema. Without one, `SELECT *` comes back as a single output named `*`, and an unqualified column in a join as a placeholder with no table. `lib.lineage.schema` builds the schema from the catalog's Iceberg metadata, so the parser sees the columns the warehouse has, at sqlglot 30.24.0.
For the platform, take `marketing_audience`. In words, it picks learners who agreed to marketing email and whose account is active, and adds each one's favourite course category, which a CTE computes by ranking categories by the learner's enrollments. Run sqlglot on it, and on two columns of `course_performance`:
Example `sqlglot-misses-row-inputs` (Python 3.12.15 · duckdb 2.0.0 · openlineage-python 1.54.0 · pyarrow 26.0.0 · pyiceberg 0.12.0 · pyyaml 6.0.3 · sqlglot 30.24.0), passing in Edition 2026.10.1:
```python
"""What sqlglot.lineage says the platform's pipeline outputs come from, and what it leaves out or doesn't mark."""
import sqlglot
from sqlglot import exp
from lib import lineage, pipelines
from lib.catalog import open_catalog
schema = lineage.schema(open_catalog(["crm.learners", "learning.courses", "learning.enrollments", "learning.reviews",
"billing.payments"]))
statement = pipelines.sql("marketing_audience")
found = lineage.sqlglot_sources(statement, schema)
for column, sources in found.items():
print(f"marketing.audience.{column} comes from:", ", ".join(sources))
parsed = sqlglot.parse_one(statement, read="duckdb")
tables = {table.alias_or_name: f"{table.db}.{table.name}" for table in parsed.find_all(exp.Table)}
where = parsed.find(exp.Where)
print("the pipeline's WHERE:", where.this.sql(dialect="duckdb"))
in_lineage = {source for sources in found.values() for source in sources}
for column in where.find_all(exp.Column):
source = f"{tables[column.table]}.{column.name}"
print(f"{source} in any column's lineage:", source in in_lineage)
course_performance = lineage.sqlglot_sources(pipelines.sql("course_performance"), schema)
for column in ("completions", "revenue_usd"):
print(f"analytics.course_performance.{column} comes from:", ", ".join(course_performance[column]))
```
Output:
```text
marketing.audience.learner_id comes from: crm.learners.learner_id
marketing.audience.full_name comes from: crm.learners.full_name
marketing.audience.email comes from: crm.learners.email
marketing.audience.country comes from: crm.learners.country
marketing.audience.favourite_category comes from: learning.courses.category
the pipeline's WHERE: l.marketing_consent AND l.status = 'active'
crm.learners.marketing_consent in any column's lineage: False
crm.learners.status in any column's lineage: False
analytics.course_performance.completions comes from: learning.enrollments.completed_at, learning.enrollments.enrollment_id
analytics.course_performance.revenue_usd comes from: billing.payments.amount, billing.payments.status
```
The values are right. `email` comes from `crm.learners.email`, and `favourite_category` is traced through the CTE to `learning.courses.category`. What's missing is everything that decides which learners are in the table. The pipeline's `WHERE` is `l.marketing_consent AND l.status = 'active'`, and neither column is in any output's lineage: the example's check prints `False` for `marketing_consent` and `False` for `status`. The consent filter is the reason marketing may hold this table at all, and the lineage doesn't mention it. Nor does it mention the join keys from enrollments and courses.
The second problem is the reverse: an input that only filters is listed as if it were a value. `course_performance` sums only succeeded payments, with `filter (where p.status = 'succeeded')`, and sqlglot says `revenue_usd` comes from `billing.payments.amount, billing.payments.status`: the column summed and the column that only decides which payments count, listed alike. `completions` counts enrollments with `filter (where e.completed_at is not null)`, and comes back the same way, from `learning.enrollments.completed_at, learning.enrollments.enrollment_id`. Both problems follow from how sqlglot builds the tree. It collects the columns in each output's expression, so it sees a filter inside an expression and nothing outside it, and its nodes carry no type to say how an input was used.
**Working rule:** Use sqlglot's `lineage` for the columns each output's value is computed from, and always pass the catalog's schema, generated from the table metadata, never typed. Don't read its answer as everything a column depends on: it misses the columns that join, filter and group, and it doesn't say which of its inputs are values.
## Add the inputs that decide which rows exist {#add-indirect-inputs}
[sqlglot's lineage](#trace-output-columns) lists the columns a value is computed from, and OpenLineage has a place for the rest. Each input field carries a list of transformations, each with a type, a subtype and a `masking` flag ([column lineage facet](https://openlineage.io/docs/spec/facets/dataset-facets/column_lineage_facet)). DIRECT means the "output column value was somehow derived from inputField value", as IDENTITY, TRANSFORMATION or AGGREGATION. INDIRECT means the "output column value is impacted by the value of inputField column, but it's not derived from it", as JOIN, GROUP_BY, FILTER, SORT, WINDOW or CONDITIONAL. Inputs that shape every row, such as a `WHERE`, belong in the facet's `dataset` list.
For governance, the INDIRECT side matters, because information flows through conditions as well as values. Denning calls it "implicit flow": a statement that assigns only when a condition holds still tells you the condition's value ([Denning, 1976](https://doi.org/10.1145/360051.360056)). A row in `marketing.audience` tells you that learner consented.
`lib.lineage.column_lineage` adds the two things sqlglot's answer lacks: a type for each value input, and the INDIRECT inputs. It types each value input by where the column sits in the output's expression: a bare column is IDENTITY, a column inside an aggregate is AGGREGATION, and anything else is TRANSFORMATION. A column inside an aggregate's `FILTER`, a window's `PARTITION BY` or `ORDER BY`, or a `CASE`'s condition or operand is INDIRECT, and a hash on the way sets `masking`. `COALESCE`'s arguments count as values: OpenLineage lists `COALESCE` under CONDITIONAL, but any of its arguments can become the output's value. Then it walks every `SELECT` in the statement, CTEs included, and collects the columns of its row clauses, following CTE columns back to the tables they come from:
```python
ROW_CLAUSES = {"joins": "JOIN", "where": "FILTER", "group": "GROUP_BY", "having": "FILTER", "qualify": "FILTER",
"order": "SORT"}
```
Run it on `marketing_audience`, and on the two `course_performance` columns again:
Example `indirect-column-lineage` (Python 3.12.15 · duckdb 2.0.0 · openlineage-python 1.54.0 · pyarrow 26.0.0 · pyiceberg 0.12.0 · pyyaml 6.0.3 · sqlglot 30.24.0), passing in Edition 2026.10.1:
```python
"""The book's column lineage: each column's inputs as OpenLineage types them, plus the inputs that shape every row,
which sqlglot.lineage leaves out."""
from lib import lineage, pipelines
from lib.catalog import open_catalog
schema = lineage.schema(open_catalog(["crm.learners", "learning.courses", "learning.enrollments", "learning.reviews",
"billing.payments"]))
fields, rows = lineage.column_lineage(pipelines.sql("marketing_audience"), schema)
for column, inputs in fields.items():
print(f"marketing.audience.{column}:", ", ".join(f"{dependency.source} {dependency.type} {dependency.subtype}"
for dependency in inputs))
print("inputs that shape every row of marketing.audience:")
for source in sorted({dependency.source for dependency in rows}):
print(f" {source}:", "INDIRECT",
", ".join(dependency.subtype for dependency in rows if dependency.source == source))
values = {dependency.source for inputs in fields.values() for dependency in inputs}
print("source columns, values only:", len(values))
print("source columns, with the rows' inputs:", len(values | {dependency.source for dependency in rows}))
fields, _ = lineage.column_lineage(pipelines.sql("course_performance"), schema)
for column in ("completions", "revenue_usd"):
print(f"analytics.course_performance.{column}:",
", ".join(f"{dependency.source} {dependency.type} {dependency.subtype}" for dependency in fields[column]))
```
Output:
```text
marketing.audience.learner_id: crm.learners.learner_id DIRECT IDENTITY
marketing.audience.full_name: crm.learners.full_name DIRECT IDENTITY
marketing.audience.email: crm.learners.email DIRECT IDENTITY
marketing.audience.country: crm.learners.country DIRECT IDENTITY
marketing.audience.favourite_category: learning.courses.category DIRECT IDENTITY
inputs that shape every row of marketing.audience:
crm.learners.learner_id: INDIRECT JOIN
crm.learners.marketing_consent: INDIRECT FILTER
crm.learners.status: INDIRECT FILTER
learning.courses.category: INDIRECT GROUP_BY, JOIN
learning.courses.course_id: INDIRECT JOIN
learning.enrollments.course_id: INDIRECT JOIN
learning.enrollments.learner_id: INDIRECT GROUP_BY, JOIN
source columns, values only: 5
source columns, with the rows' inputs: 10
analytics.course_performance.completions: learning.enrollments.completed_at INDIRECT FILTER, learning.enrollments.enrollment_id DIRECT AGGREGATION
analytics.course_performance.revenue_usd: billing.payments.amount DIRECT AGGREGATION, billing.payments.status INDIRECT FILTER
```
The values are as [sqlglot found them](#trace-output-columns), each an IDENTITY copy. The inputs that shape every row are new: `crm.learners.marketing_consent` and `crm.learners.status` as FILTER, the learner and course IDs as JOIN, and the enrollments' `learner_id` and the category as GROUP_BY. The lineage of `marketing.audience` grows from 5 source columns to 10. For `revenue_usd`, the two inputs are now told apart: `amount` is a DIRECT AGGREGATION, and `status` an INDIRECT FILTER. So are the two inputs of `completions`: `enrollment_id` is counted, and `completed_at` only filters.
Two details of the facet are worth knowing. The per-field `transformationType` and `transformationDescription` are deprecated, replaced by the per-input `transformations` that `lib.lineage` writes ([changelog, 1.17.1](https://github.com/OpenLineage/OpenLineage/blob/main/CHANGELOG.md)). And OpenLineage 1.53.0 added a Lineage dataset facet for declared lineage, such as a view's, sent on a `DatasetEvent`, which consumers should prefer where both describe the same relationship ([lineage facet](https://openlineage.io/docs/spec/facets/dataset-facets/lineage)). Run events from jobs carry `columnLineage`, as the platform's do.
**Working rule:** Record INDIRECT inputs with every job's column lineage: the join keys, filters and grouping columns of every `SELECT`, CTEs included, in the facet's `dataset` list, and type each value input with OpenLineage's subtypes. Expect more edges, many of them join keys. Keep them anyway: a filter on a consent column decides who may be emailed, and it reaches the table only as an INDIRECT input.
## Fail on statements the parser can't read {#fail-on-unreadable-lineage}
The platform's [column lineage](#add-indirect-inputs) is only as good as the parse behind it, and a parse can be wrong without failing. sqlglot's `lineage` accepts more than a `SELECT`: it finds the query inside a `CREATE TABLE ... AS`, an `INSERT ... SELECT` or a `CREATE VIEW`, and it reads a `MERGE` too ([sqlglot](https://pypi.org/project/sqlglot/30.24.0/), `optimizer/scope.py`). Three cases come back looking fine. An `INSERT`'s outputs are named after its `SELECT`, never after the table's columns. A `MERGE` comes back as the last query sqlglot reaches in it, as if that query were what the `MERGE` writes. When `USING` names a table or a CTE and no subquery follows, sqlglot returns no lineage and no error. When `USING` is a subquery, or a subquery sits in a `WHEN` condition or a `SET`, it returns the columns of the last such subquery, whether the `MERGE` writes them or not. And a misspelt column gets through when it's unqualified in a query on one table: sqlglot assigns it to that table whether the table has the column or not.
All four of the platform's pipelines are `CREATE TABLE ... AS` statements. As tables grow, pipelines like these often become incremental: an `INSERT` that appends a day's revenue to `analytics.daily_revenue`, or a `MERGE` that updates `analytics.learner_activity` in place. Run sqlglot and `lib.lineage.column_lineage` on both, with the `MERGE` in two forms:
Example `insert-and-merge-lineage` (Python 3.12.15 · duckdb 2.0.0 · openlineage-python 1.54.0 · pyarrow 26.0.0 · pyiceberg 0.12.0 · pyyaml 6.0.3 · sqlglot 30.24.0), passing in Edition 2026.10.1:
```python
"""Two incremental statements sqlglot.lineage answers without complaint and gets wrong: an INSERT, whose outputs it
names after the SELECT, and a MERGE, for which it returns nothing, or the columns of the subquery its USING names."""
import pyarrow as pa
from sqlglot.lineage import lineage as sqlglot_lineage
from lib import lineage
from lib.catalog import open_catalog
catalog = open_catalog(["billing.payments", "crm.learners"])
catalog.create_namespace("analytics") # the incremental table, as it exists before the day's INSERT
catalog.create_table("analytics.daily_revenue", schema=pa.schema([("day", pa.date32()), ("revenue_usd", pa.float64())]))
schema = lineage.schema(catalog)
insert = """insert into analytics.daily_revenue
select cast(p.paid_at as date), sum(p.amount) from billing.payments p where p.status = 'succeeded' group by 1"""
merges = {"USING a table": """merge into analytics.learner_activity a using crm.learners l
on a.learner_id = l.learner_id when matched then update set country = l.country""",
"USING a subquery": """merge into analytics.learner_activity a
using (select l.learner_id, l.country, l.email from crm.learners l) s on a.learner_id = s.learner_id
when not matched then insert (learner_id, country) values (s.learner_id, s.country)"""}
print("analytics.daily_revenue columns:", ", ".join(schema["analytics"]["daily_revenue"]))
print("sqlglot, INSERT outputs:", ", ".join(sqlglot_lineage(None, insert, schema, dialect="duckdb")))
fields, rows = lineage.column_lineage(insert, schema)
for column, inputs in fields.items():
print(f"book, INSERT {column}:", ", ".join(f"{dependency.source} {dependency.type} {dependency.subtype}"
for dependency in inputs))
print("book, INSERT rows shaped by:", ", ".join(f"{dependency.source} {dependency.subtype}" for dependency in rows))
for label, merge in merges.items():
outputs = sqlglot_lineage(None, merge, schema, dialect="duckdb")
print(f"sqlglot, MERGE {label}, outputs:", ", ".join(outputs) or "none")
refused = {"INSERT with one value too many": insert.replace("sum(p.amount)", "sum(p.amount), count(*)"),
**{f"MERGE {label}": merge for label, merge in merges.items()}}
for label, statement in refused.items():
try:
lineage.column_lineage(statement, schema)
except lineage.LineageError as error:
print(f"book, {label}:", error)
```
Output:
```text
analytics.daily_revenue columns: day, revenue_usd
sqlglot, INSERT outputs: paid_at, _col_1
book, INSERT day: billing.payments.paid_at DIRECT TRANSFORMATION
book, INSERT revenue_usd: billing.payments.amount DIRECT AGGREGATION
book, INSERT rows shaped by: billing.payments.paid_at GROUP_BY, billing.payments.status FILTER
sqlglot, MERGE USING a table, outputs: none
sqlglot, MERGE USING a subquery, outputs: learner_id, country, email
book, INSERT with one value too many: analytics.daily_revenue: the INSERT writes 2 columns, and its SELECT has 3 values
book, MERGE USING a table: analytics.learner_activity: lineage is read only from CREATE ... AS and INSERT ... SELECT, not from MERGE
book, MERGE USING a subquery: analytics.learner_activity: lineage is read only from CREATE ... AS and INSERT ... SELECT, not from MERGE
```
The table `analytics.daily_revenue` has the columns `day, revenue_usd`, and sqlglot names the `INSERT`'s outputs `paid_at, _col_1`: the first after the column inside the `cast`, the second a generated name for an expression with no alias. Neither is a column of the table. An `INSERT` fills the table's columns by position, so `lib.lineage` matches by position too, and takes the names from the catalog: the columns the `INSERT` lists, or else all of the table's. So `day` comes from `billing.payments.paid_at DIRECT TRANSFORMATION`, and the `INSERT`'s grouping and its filter on `status` shape every row: `billing.payments.paid_at GROUP_BY, billing.payments.status FILTER`. When the `SELECT` has more or fewer values than the columns written, `lib.lineage` raises: `analytics.daily_revenue: the INSERT writes 2 columns, and its SELECT has 3 values`.
For the `MERGE` whose `USING` names a table, sqlglot returns none, and no error. For the one whose `USING` is a subquery, it returns `learner_id, country, email`, though that `MERGE` writes only `learner_id` and `country`. A check for empty lineage catches the first and passes the second. So `lib.lineage` doesn't judge what sqlglot returns for a `MERGE`. Before it calls sqlglot, it refuses every statement that isn't a `CREATE ... AS` or an `INSERT ... SELECT`: `analytics.learner_activity: lineage is read only from CREATE ... AS and INSERT ... SELECT, not from MERGE`.
A misspelt column, the third case, depends on how the column is written. The platform's pipelines qualify every column with its table's alias, as in `l.email`, and sqlglot checks a qualified column against the schema itself:
Example `misspelt-column-lineage` (Python 3.12.15 · duckdb 2.0.0 · openlineage-python 1.54.0 · pyarrow 26.0.0 · pyiceberg 0.12.0 · pyyaml 6.0.3 · sqlglot 30.24.0), passing in Edition 2026.10.1:
```python
"""A misspelt column: sqlglot raises when the column is qualified, and when it isn't, in a query on one table, traces it
to that table anyway. lib.lineage checks every source against the schema."""
from sqlglot.errors import OptimizeError
from lib import lineage
from lib.catalog import open_catalog
schema = lineage.schema(open_catalog(["crm.learners"]))
qualified = "create table marketing.audience as select l.learner_id, l.emial from crm.learners l"
unqualified = "create table marketing.audience as select learner_id, emial from crm.learners"
try:
lineage.sqlglot_sources(qualified, schema)
except OptimizeError as error:
print("sqlglot, qualified l.emial:", f"{type(error).__name__}: {error}")
print("sqlglot, unqualified emial comes from:", ", ".join(lineage.sqlglot_sources(unqualified, schema)["emial"]))
try:
lineage.column_lineage(unqualified, schema)
except lineage.LineageError as error:
print("book, unqualified emial:", error)
```
Output:
```text
sqlglot, qualified l.emial: OptimizeError: Unknown column: emial
sqlglot, unqualified emial comes from: crm.learners.emial
book, unqualified emial: crm.learners has no column emial
```
Qualified, the typo stops sqlglot with `OptimizeError: Unknown column: emial`. Unqualified, in a query on one table, it comes back as `crm.learners.emial`, a confident answer about a column that doesn't exist. In a join, a misspelt unqualified column belongs to none of the tables, and sqlglot returns a placeholder with no table ([sqlglot](https://pypi.org/project/sqlglot/30.24.0/), `lineage.py`), which `lib.lineage` rejects as a source it can't place. For the single-table case, `lib.lineage` checks every source against the schema and raises `crm.learners has no column emial`.
`lib.lineage.record`, which `run` uses, reads the lineage before the job runs, so a pipeline whose lineage can't be read fails before it writes anything. That is strict, and deliberate: a job with empty lineage would vanish from every impact report and every classification check in this chapter, and one with wrong lineage would mislead them. Run the same check in CI on every pull request that changes a pipeline, so the failure shows up in review rather than at night.
**Working rule:** Treat a statement your parser can't read correctly as an error, never as empty or partial lineage. Refuse every kind of statement it gets wrong, such as `MERGE`, before you ask it, and fail on empty lineage, a source the parser can't place and a column the schema doesn't have. Match an `INSERT`'s columns by position against the table's columns in the catalog, never the `SELECT`'s names, and fail when the counts differ. For a `MERGE`, write a `SELECT` of the columns it writes, and record the job's lineage from that, as an `INSERT ... SELECT` into the `MERGE`'s table, until your parser handles `MERGE` itself.
## Find what a column change breaks {#impact-of-a-column-change}
With [column lineage](#add-indirect-inputs) in every COMPLETE event, the platform has a graph: columns are nodes, and each input field is an edge to the column or table it feeds. **Impact analysis** asks what a change to one column can break. It is a walk of that graph: start at the column, follow every edge out of it, and keep going from wherever you land, at any depth. No special method is needed; it's graph reachability. One rule is specific to lineage. An INDIRECT edge into a table means every row of it can change, so the walk continues from all of that table's columns.
`lib.lineage.graph` builds the graph from the events file alone, as a lineage server would, not from the SQL. That way it includes every job that sends events, whatever it's written in, and nothing that doesn't. `lib.lineage.downstream` does the walk.
For the platform, take three changes its teams might make. The CRM team lowercases every email, or renames `crm.learners.email`. A new sign-up flow changes what `crm.learners.marketing_consent` means for older accounts. The learning team renames a category in `learning.courses`. Before you run it, predict how many columns the consent change can reach:
Example `column-impact` (Python 3.12.15 · duckdb 2.0.0 · openlineage-python 1.54.0 · pyarrow 26.0.0 · pyiceberg 0.12.0 · pyyaml 6.0.3 · sqlglot 30.24.0), passing in Edition 2026.10.1:
```python
"""What a change to a source column can break downstream, read from the events the nightly run sent."""
from lib import lineage
from lib.catalog import open_catalog
from lib.platform import scratch
events_file = scratch() / "events.jsonl"
lineage.run_all(open_catalog(), events_file)
graph = lineage.graph(lineage.read_events(events_file))
print("edges in the column graph:", len(graph.edges))
for column in ("crm.learners.email", "crm.learners.marketing_consent", "learning.courses.category"):
reached = lineage.downstream(graph, column)
print(f"{column} reaches:")
for target in dict.fromkeys(edge.target for edge in reached):
how = ", ".join(f"{edge.type} {edge.subtype}" for edge in reached if edge.target == target)
print(f" {target}{', every row' if target.count('.') == 1 else ''}:", how)
values = [edge for edge in reached if edge.type == "DIRECT"]
print(f"{column}, columns reached through values:", len(lineage.columns_reached(graph, values)))
print(f"{column}, columns reached in all:", len(lineage.columns_reached(graph, reached)))
```
Output:
```text
edges in the column graph: 51
crm.learners.email reaches:
marketing.audience.email: DIRECT IDENTITY
crm.learners.email, columns reached through values: 1
crm.learners.email, columns reached in all: 1
crm.learners.marketing_consent reaches:
marketing.audience, every row: INDIRECT FILTER
crm.learners.marketing_consent, columns reached through values: 0
crm.learners.marketing_consent, columns reached in all: 5
learning.courses.category reaches:
analytics.course_performance, every row: INDIRECT GROUP_BY
analytics.course_performance.category: DIRECT IDENTITY
marketing.audience, every row: INDIRECT GROUP_BY, INDIRECT JOIN
marketing.audience.favourite_category: DIRECT IDENTITY
learning.courses.category, columns reached through values: 2
learning.courses.category, columns reached in all: 12
```
The graph has 51 edges. A change to `email` reaches one column, `marketing.audience.email`, so renaming it breaks the marketing pipeline and no other job that sends lineage. The consent change reaches 0 columns through values, yet through the filter it reaches all 5 columns of `marketing.audience`: it changes who gets emailed. A lineage tool that follows values only would report a change to consent, the column that decides who may be emailed, as having no impact. A category rename reaches 2 columns through values, and 12 in all, because categories also group `analytics.course_performance` and rank each learner's favourite.
The same walk serves two later chapters: [chapter 7](#plan-erasure-from-lineage) follows a learner's columns downstream to find every derived table an erasure must reach, and [chapter 10](#report-what-a-schema-change-reaches) writes an impact report for every pull request that changes a schema.
**Working rule:** Run impact analysis on the graph built from events, INDIRECT edges included, and report two lists: columns whose values can change, and tables whose rows can change. Treat a change to a column used only in filters and joins as a change to every column of the tables it filters, and send the report to the owners of each table it reaches.
## Propose classifications from lineage {#classifications-from-lineage}
A derived column holds data from its inputs, so its classification can be derived from theirs. The classic rule comes from Denning's lattice model of information flow: the class of a value computed from several inputs is the least upper bound, the "join", of their classes ([Denning, 1976](https://doi.org/10.1145/360051.360056)). The platform's classifications form a line, `public`, `internal`, `personal` and `sensitive`, so the join is the stricter of them, which `stricter` in `lib.metadata`, the book's metadata module, returns. Which inputs count is a judgement. We count INDIRECT inputs too, following Denning's implicit flows, because a table filtered on consent reveals each learner's consent. That choice lives in `governance/lineage.toml`, so a team can change it in one reviewed line.
Two more choices come from the same caution. OpenLineage's `masking` flag says only that a value was obfuscated, such as by a hash ([column lineage facet](https://openlineage.io/docs/spec/facets/dataset-facets/column_lineage_facet)). Whether a hashed identifier is still personal data is a legal question: [chapter 2](#classify-pseudonymous-ids) quotes the GDPR recital on pseudonymized data and leaves the answer to lawyers. So the book's rule never lowers a classification because of masking. Catalog servers leave the same choice to configuration. Apache Atlas propagates classifications along lineage and lets you block propagation on an edge, for example where a view masks a PII column ([Atlas](https://atlas.apache.org/1.0.0/ClassificationPropagation.html)), and DataHub propagates glossary terms along column lineage, a feature its 1.7.0 documentation describes as a public beta in DataHub Cloud ([DataHub](https://docs.datahub.com/docs/automations/glossary-term-propagation)). And a proposal is a floor, not a verdict: like the drafts a local model writes in [chapter 2](#draft-descriptions-with-a-local-model), it's reviewed before it goes into `governance/metadata.toml`, the platform's file of owners and classifications.
For the platform, `lib.lineage.propose` proposes a classification for every column a job wrote, and `lib.metadata.problems` lists the tables that need one. Before trusting the rule on the two tables with no metadata, `marketing.audience` and `ai.review_corpus`, check it on the two that people did classify, `analytics.course_performance` and `analytics.learner_activity`:
Example `propagate-classifications` (Python 3.12.15 · duckdb 2.0.0 · openlineage-python 1.54.0 · pyarrow 26.0.0 · pyiceberg 0.12.0 · pyyaml 6.0.3 · sqlglot 30.24.0), passing in Edition 2026.10.1:
```python
"""Propose classifications for the pipeline outputs that have no metadata, from their lineage, and check the rule
against the outputs people did classify."""
from lib import lineage, metadata
from lib.catalog import open_catalog
from lib.platform import scratch
catalog = open_catalog()
events_file = scratch() / "events.jsonl"
lineage.run_all(catalog, events_file)
for problem in metadata.problems(catalog, metadata.load()):
print("metadata problem:", problem)
graph, known = lineage.graph(lineage.read_events(events_file)), lineage.declared()
print("propagated through:", " and ".join(lineage.rules()["propagate_through"]))
proposed = lineage.propose(graph, known)
for column, (classification, decided_by) in proposed.items():
if column not in known:
first = decided_by[0]
print(f"{column}: {classification}, from {first.source} ({first.type} {first.subtype})")
print("personal columns proposed:", sum(1 for column, (classification, _) in proposed.items()
if column not in known and classification == "personal"))
classified = [column for column in proposed if column in known]
print("declared classifications checked:", len(classified))
print("classes declared for them:", ", ".join(sorted({known[column] for column in classified})))
for through in (["DIRECT", "INDIRECT"], ["DIRECT"]):
proposals = lineage.propose(graph, known, through)
agree = sum(1 for column in classified if proposals[column][0] == known[column])
print(f"declared classifications reproduced, through {' and '.join(through)}:", f"{agree} of {len(classified)}")
missed = [column for column in classified if proposals[column][0] != known[column]] # with DIRECT inputs only
print("missed with DIRECT inputs only:", len(missed))
for column in missed:
print(f" {column}: declared {known[column]}, DIRECT inputs propose {proposals[column][0]}")
print("declared less protected than lineage proposes:", len(lineage.understated(proposed, known)))
```
Output:
```text
metadata problem: ai.review_corpus: no metadata
metadata problem: marketing.audience: no metadata
propagated through: DIRECT and INDIRECT
marketing.audience.learner_id: internal, from crm.learners.learner_id (DIRECT IDENTITY)
marketing.audience.full_name: personal, from crm.learners.full_name (DIRECT IDENTITY)
marketing.audience.email: personal, from crm.learners.email (DIRECT IDENTITY)
marketing.audience.country: internal, from crm.learners.country (DIRECT IDENTITY)
marketing.audience.favourite_category: internal, from crm.learners.learner_id (INDIRECT JOIN)
ai.review_corpus.review_id: internal, from learning.reviews.review_id (DIRECT IDENTITY)
ai.review_corpus.course_id: internal, from learning.reviews.course_id (DIRECT IDENTITY)
ai.review_corpus.course_title: internal, from learning.reviews.course_id (INDIRECT JOIN)
ai.review_corpus.rating: internal, from learning.reviews.rating (DIRECT IDENTITY)
ai.review_corpus.body: personal, from learning.reviews.body (DIRECT IDENTITY)
personal columns proposed: 3
declared classifications checked: 13
classes declared for them: internal
declared classifications reproduced, through DIRECT and INDIRECT: 13 of 13
declared classifications reproduced, through DIRECT: 10 of 13
missed with DIRECT inputs only: 3
analytics.course_performance.course_id: declared internal, DIRECT inputs propose public
analytics.course_performance.title: declared internal, DIRECT inputs propose public
analytics.course_performance.category: declared internal, DIRECT inputs propose public
declared less protected than lineage proposes: 0
```
The metadata check from [chapter 2](#test-the-catalog-against-the-metadata) still reports the two tables with no metadata, and lineage proposes a classification for each of their columns. It proposes 3 personal columns: `full_name` and `email` in `marketing.audience`, each an IDENTITY copy of a personal column in `crm.learners`, and `body` in `ai.review_corpus`, a copy of the reviews' free text. The rest come out internal, including `favourite_category`, for which the example prints `internal, from crm.learners.learner_id (INDIRECT JOIN)`. Its values are category names from the public course catalog, but the rows they sit in are chosen by a join on learner IDs and a consent filter, and those are internal.
On the 13 columns people did classify, all of them `internal`, counting INDIRECT inputs reproduces 13 of 13, and DIRECT inputs alone reproduce 10 of 13. The 3 that DIRECT inputs miss are `course_id`, `title` and `category` of `analytics.course_performance`, which a DIRECT-only rule calls public, as in the course catalog, while the analytics team chose internal. So the rule proposes no less than people chose here, and lineage finds no declared classification lower than its proposal (0). That is weak evidence: a rule that proposed internal for every column would score as well. And it can't show what the rule does with a personal filter, because none of the platform's filters or joins reads a personal column: even `marketing_consent` is classified internal.
So build a table whose filter does. For a birthday offer, marketing wants the learners who consented and were born this month, with their ID and country, and nothing else:
Example `personal-filter-propagates` (Python 3.12.15 · duckdb 2.0.0 · openlineage-python 1.54.0 · pyarrow 26.0.0 · pyiceberg 0.12.0 · pyyaml 6.0.3 · sqlglot 30.24.0), passing in Edition 2026.10.1:
```python
"""A table chosen by a personal column: the learners with a birthday this month, for a birthday offer. None of its
values is personal, yet each row reveals the month of a learner's birth. Only counting INDIRECT inputs proposes
personal."""
from lib import lineage
from lib.catalog import open_catalog
from lib.platform import scratch
from lib.warehouse import connect
catalog = open_catalog(["crm.learners"])
# The consenting learners born in the month of the nightly run, on the book's "today".
select = ("select l.learner_id, l.country from crm.learners l "
f"where l.marketing_consent and month(l.date_of_birth) = {lineage.NIGHTLY.month}")
def offers() -> None:
rows = connect(catalog).sql(select).to_arrow_table()
catalog.create_namespace_if_not_exists("marketing")
catalog.create_table("marketing.birthday_offers", schema=rows.schema).append(rows)
events_file = scratch() / "events.jsonl"
lineage.record(lineage.client(events_file), "birthday_offers", f"create table marketing.birthday_offers as {select}",
catalog, offers)
offers_table = catalog.load_table("marketing.birthday_offers").scan().to_arrow()
print("learners in marketing.birthday_offers:", f"{len(offers_table):,}")
graph, known = lineage.graph(lineage.read_events(events_file)), lineage.declared()
print("crm.learners.date_of_birth declared:", known["crm.learners.date_of_birth"])
for through in (["DIRECT", "INDIRECT"], ["DIRECT"]):
for column, (classification, decided_by) in lineage.propose(graph, known, through).items():
label = f"{column}, through {' and '.join(through)}"
print(f"{label}:", classification)
print(f"{label}, decided by:", f"{decided_by[0].source} ({decided_by[0].type} {decided_by[0].subtype})")
```
Output:
```text
learners in marketing.birthday_offers: 165
crm.learners.date_of_birth declared: personal
marketing.birthday_offers.learner_id, through DIRECT and INDIRECT: personal
marketing.birthday_offers.learner_id, through DIRECT and INDIRECT, decided by: crm.learners.date_of_birth (INDIRECT FILTER)
marketing.birthday_offers.country, through DIRECT and INDIRECT: personal
marketing.birthday_offers.country, through DIRECT and INDIRECT, decided by: crm.learners.date_of_birth (INDIRECT FILTER)
marketing.birthday_offers.learner_id, through DIRECT: internal
marketing.birthday_offers.learner_id, through DIRECT, decided by: crm.learners.learner_id (DIRECT IDENTITY)
marketing.birthday_offers.country, through DIRECT: internal
marketing.birthday_offers.country, through DIRECT, decided by: crm.learners.country (DIRECT IDENTITY)
```
The table holds 165 learners, and neither of its columns copies anything personal: counting DIRECT inputs only proposes `internal` for `learner_id` and `internal` for `country`. Yet every row says that its learner was born in the month of the offer, part of a date of birth, which `governance/metadata.toml` classifies `personal`. Counting INDIRECT inputs proposes `personal` for both columns, decided by `crm.learners.date_of_birth (INDIRECT FILTER)`. That is the case we count INDIRECT inputs for, and it shows the cost too: one personal filter makes every column of a table personal.
OpenLineage can carry classifications in the events themselves, with the Tags dataset facet, whose `field` key "identifies the field in a dataset if a tag applies to one" ([tags facet](https://openlineage.io/docs/spec/facets/dataset-facets/tag-facet), [schema](https://openlineage.io/spec/facets/1-0-0/TagsDatasetFacet.json)). We keep them in `governance/metadata.toml` instead, the one source that the platform's access policies also read, through `lib.policy`, the book's policy module ([chapter 5](#generate-the-data-the-policy-reads)).
**Working rule:** Propose each derived column's classification as the stricter of its inputs', counting INDIRECT inputs, and send the proposals to the table's owner for review before they become metadata. Never let lineage lower a classification someone declared, and never treat masking as declassification. Count DIRECT inputs only if your pipelines often join or filter on personal columns, such as matching on email, and every output would otherwise come out personal; then review INDIRECT personal inputs by hand.
## What goes wrong in production {#lineage-in-production}
Lineage fails quietly: a missing edge looks exactly like a column with no dependencies, and every check built on the graph, from [impact analysis](#impact-of-a-column-change) to [proposed classifications](#classifications-from-lineage), then passes. Three failures are common, and each has a check.
### Events that never arrive {#events-that-never-arrive}
The client reads `OPENLINEAGE_DISABLED` from the environment, and when it's `true`, events are dropped ([configuration](https://openlineage.io/docs/client/python/configuration)). In the client's source, that check comes first ([openlineage-python](https://pypi.org/project/openlineage-python/1.54.0/), `client/client.py`): before it looks at the transport you passed explicitly, the client picks a transport that does nothing. Its only notice is a log line at INFO level: "OpenLineage is disabled. No events will be emitted." Python's root logger is set to `WARNING` by default, so the line never shows ([logging HOWTO](https://docs.python.org/3.12/howto/logging.html)). A variable left in a shared environment file after a debugging session is enough. Set it, then run the platform's `review_corpus` pipeline with an explicit file transport:
Example `lineage-disabled-silently` (Python 3.12.15 · duckdb 2.0.0 · openlineage-python 1.54.0 · pyarrow 26.0.0 · pyiceberg 0.12.0 · pyyaml 6.0.3 · sqlglot 30.24.0), passing in Edition 2026.10.1:
```python
"""With OPENLINEAGE_DISABLED=true in the environment, an explicit file transport writes nothing and raises nothing: its
only notice is one INFO log line, which Python's default logging level hides."""
import os
from lib import lineage
from lib.catalog import open_catalog
from lib.platform import scratch
os.environ["OPENLINEAGE_DISABLED"] = "true" # say, left in a shared environment file after a debugging session
events_file = scratch() / "events.jsonl"
client = lineage.client(events_file) # a file transport, passed in explicitly
catalog = open_catalog(["learning.courses", "learning.reviews"])
output = lineage.run(catalog, "review_corpus", client)
print("transport the client chose:", client.transport.kind)
print("pipeline wrote:", output)
print("rows written:", f"{len(catalog.load_table(output).scan().to_arrow()):,}")
print("events file exists:", events_file.exists())
missing = lineage.missing_runs(["review_corpus"], lineage.read_events(events_file))
print("jobs that ran with no COMPLETE event:", ", ".join(missing))
```
Output:
```text
transport the client chose: noop
pipeline wrote: ai.review_corpus
rows written: 3,000
events file exists: False
jobs that ran with no COMPLETE event: review_corpus
```
The transport the client chose is `noop`. The pipeline ran and wrote 3,000 rows to `ai.review_corpus`, the check for the events file prints `False`, and nothing raised an error. The check that catches it compares the jobs that ran, which the scheduler knows, with the jobs that sent a COMPLETE event: `lib.lineage.missing_runs` lists `review_corpus`.
**Working rule:** After every scheduled run, compare the jobs that ran with the jobs whose COMPLETE events arrived, and alert on any difference. Don't rely on the emitter to report its own silence.
### A job nobody instrumented {#job-nobody-instrumented}
Lineage covers only the jobs that send it. The platform's marketing team runs a weekly Python script that copies `marketing.audience` into `marketing.campaign_list`, a table its email tool reads. Nobody wired the script to OpenLineage:
Example `uninstrumented-job` (Python 3.12.15 · duckdb 2.0.0 · openlineage-python 1.54.0 · pyarrow 26.0.0 · pyiceberg 0.12.0 · pyyaml 6.0.3 · sqlglot 30.24.0), passing in Edition 2026.10.1:
```python
"""A Python step that copies marketing.audience for the email tool and tells OpenLineage nothing: impact analysis misses
the copy, and a check of the catalog against the lineage finds it."""
from lib import lineage
from lib.catalog import open_catalog
from lib.platform import scratch
from lib.warehouse import connect
catalog = open_catalog()
events_file = scratch() / "events.jsonl"
lineage.run_all(catalog, events_file)
# The marketing team's weekly export, written straight to the catalog: no lineage event.
export = connect(catalog).sql("select learner_id, full_name, email from marketing.audience").to_arrow_table()
catalog.create_table("marketing.campaign_list", schema=export.schema).append(export)
print("rows exported:", f"{len(export):,}")
events = lineage.read_events(events_file)
reached = lineage.downstream(lineage.graph(events), "crm.learners.email")
print("crm.learners.email reaches:", ", ".join(edge.target for edge in reached))
print("tables no job's lineage accounts for:", ", ".join(lineage.unaccounted(catalog, events)))
```
Output:
```text
rows exported: 1,829
crm.learners.email reaches: marketing.audience.email
tables no job's lineage accounts for: marketing.campaign_list
```
The script copied 1,829 learners' names and emails, and impact analysis can't see the copy: `crm.learners.email` still reaches only `marketing.audience.email`. An erasure guided by this lineage would miss the copy. The catalog knows the table exists, though, so `lib.lineage.unaccounted` compares the catalog's tables with the lineage: every table must be one of the platform's sources or the output of a job that lineage knows. It finds `marketing.campaign_list`.
**Working rule:** Check the catalog against the lineage on a schedule, and treat a table no job's lineage accounts for as an incident: find the job, and make it send events.
### Personal data in a table classified internal {#personal-data-tagged-internal}
Instrumenting [the marketing team's export](#job-nobody-instrumented) fixes the graph, and exposes a second problem. When the export's owner created `marketing.campaign_list`, they classified every column internal, because only the marketing team reads it. That's a statement about who reads a table, not what it holds. Wrap the script's work in `lib.lineage.record`, with the `SELECT` it runs as its lineage, and compare the declared classifications with lineage's proposal:
Example `personal-copy-tagged-internal` (Python 3.12.15 · duckdb 2.0.0 · openlineage-python 1.54.0 · pyarrow 26.0.0 · pyiceberg 0.12.0 · pyyaml 6.0.3 · sqlglot 30.24.0), passing in Edition 2026.10.1:
```python
"""The same export, now sending lineage, in a table its owner tagged internal: lineage finds the personal data in it."""
from lib import lineage
from lib.catalog import open_catalog
from lib.platform import scratch
from lib.warehouse import connect
catalog = open_catalog()
events_file = scratch() / "events.jsonl"
lineage.run_all(catalog, events_file)
select = "select a.learner_id, a.full_name, a.email from marketing.audience a"
def export() -> None:
rows = connect(catalog).sql(select).to_arrow_table()
catalog.create_table("marketing.campaign_list", schema=rows.schema).append(rows)
statement = f"create table marketing.campaign_list as {select}"
lineage.record(lineage.client(events_file), "campaign_export", statement, catalog, export)
# Its owner tagged every column internal: "only the marketing team reads it".
known = lineage.declared() | {f"marketing.campaign_list.{c}": "internal" for c in ("learner_id", "full_name", "email")}
graph = lineage.graph(lineage.read_events(events_file))
print("crm.learners.email reaches:", ", ".join(edge.target for edge in lineage.downstream(graph, "crm.learners.email")))
for problem in lineage.understated(lineage.propose(graph, known), known):
print("understated:", problem)
```
Output:
```text
crm.learners.email reaches: marketing.audience.email, marketing.campaign_list.email
understated: marketing.campaign_list.full_name: declared internal, lineage proposes personal
understated: marketing.campaign_list.email: declared internal, lineage proposes personal
```
`crm.learners.email` now reaches `marketing.audience.email, marketing.campaign_list.email`: two hops, the second through the export. And `lib.lineage.understated` reports `full_name` and `email` in `marketing.campaign_list` as declared internal while lineage proposes personal: copies of personal data that any policy reading the declared classification would treat as internal.
**Working rule:** Run `lib.lineage.understated`, or your own check like it, on every lineage update, and fail when a declared classification is lower than lineage proposes. Classify by what a table holds, never by who reads it, and fix such a finding by raising the classification, not by dropping the edge.
## What this chapter decided {#lineage-decisions}
- **Every pipeline run sends OpenLineage run events,** START and then COMPLETE or FAIL, from the code that runs it (`lib.lineage.run`), with the output's schema and column lineage. Datasets are named by catalog identifiers under the placeholder namespace `iceberg://platform`. Examples fix event times and derive run IDs from the nominal run, and compare what events say, not their bytes.
- **Column lineage is sqlglot's value lineage, typed as OpenLineage subtypes, plus INDIRECT inputs:** the join, filter, grouping and sort columns of every `SELECT`, CTEs included, in the facet's `dataset` list.
- **Lineage that can't be read correctly is an error,** never empty or partial lineage. `lib.lineage` reads only `CREATE ... AS` and `INSERT ... SELECT`, and refuses every other statement, `MERGE` included, whatever its `USING` names. Empty lineage, an unplaceable source or an unknown column stops the job before it runs. An `INSERT`'s columns come from the table's schema in the catalog, matched by position, and a count that differs is an error.
- **Impact analysis walks the graph built from events,** INDIRECT edges included, and reports columns whose values can change and tables whose rows can change.
- **Classifications propagate as the stricter of all inputs',** DIRECT and INDIRECT, as set in `governance/lineage.toml`, so a table chosen by a personal filter is proposed personal. Lineage never lowers a declared classification, and masking never declassifies. The proposals for the two unclassified tables, `full_name`, `email` and `body` personal and every other column internal, go to their owners for review before they enter `governance/metadata.toml`.
- **Three checks guard the lineage itself:** jobs that ran with no COMPLETE event, tables no job's lineage accounts for, and declared classifications lower than lineage proposes.
# Access policies as code
By the end of this chapter you can write a data platform's access rules as one policy that denies by default and returns row filters and column masks as data. The policy allows by role and narrows by attribute, and an engine can apply its whole decision. You'll write it in Rego, the policy language of Open Policy Agent (OPA), test it the way `opa test`, OPA's test runner, does, check it against a who-sees-what table that the data's owners can read, know where the in-process engine this book uses differs from OPA, and see the same rules in Cedar, a narrower policy language.
The platform, the online learning platform this book follows, sells professional courses to adult learners, and each of the people and programs that read its warehouse needs something different. Support agents each serve one region, `americas`, `emea` or `apac`, and need learners' contact details to answer tickets. Instructors see the learners and reviews of their own courses, without contact details. Marketing may reach only learners who agreed to marketing email. Finance reconciles payments, and the legacy `card_number` column of `billing.payments` must never show more than its last four digits. Data scientists see `learner_id` as stored, to join tables on it, with personal columns pseudonymized and free text redacted. The platform's course assistant, its AI, reads only the reviews whose authors agreed to AI training. [Chapter 1](#governance-end-to-end) showed one rule, support agents see only their own region, and enforced it on one query. Rules for every group need to live in one place, stated exactly, and tested before anyone relies on them.
This chapter's terms are the ones the rest of the book uses. A **policy** is code that answers whether a request may go ahead; its answer is a **decision**. The **principal** is who asks, a **role** is a group of principals with the same job, and an **attribute** is any other fact a rule reads, about the principal, the data or the request. A **row filter** limits which rows a query returns, and a **column mask** changes how a column's values are shown, such as **pseudonymization**, which replaces an identifier with a consistent stand-in.
The platform's policy is `governance/policies/access.rego`, and its tests are in `governance/policies/access_test.rego`. The book's shared module `lib.policy` loads both into regopy 1.5.2, the Python binding of rego-cpp, Microsoft's C++ implementation of Rego, which evaluates policies inside the Python process with no server to run.
## Shape the request like Trino's {#shape-the-request-like-trinos}
A policy engine answers questions about requests, so the shape of a request decides which systems can ask it. OPA's premise is that policy is taken out of the service that enforces it: OPA "lets you decouple policy from that software service" so that it can be read, versioned and tested on its own ([OPA philosophy](https://www.openpolicyagent.org/docs/philosophy)). The service sends a JSON document describing the request, and the policy reads it as `input`. Trino, an open-source SQL engine, has a plugin that does exactly this: for each access check a query needs, it sends OPA a document with `context.identity`, the user and their groups, and `action`, the operation and the resource, down to the catalog, schema, table and columns ([Trino 484, OPA access control](https://trino.io/docs/484/security/opa-access-control.html)).
The platform's policy reads the same fields, so it decides Trino's request to read columns, `SelectFromColumns`, as Trino sends it. Trino needs more than that one decision, though: it wants each row filter and column mask as SQL, a row filter as `{"expression": "clause"}`, whereas the policy returns a column and a value ([Trino 484, OPA access control](https://trino.io/docs/484/security/opa-access-control.html)), and it also asks about other checks, such as `ExecuteQuery` and `AccessCatalog`, that this policy's `allow` denies ([`OpaAccessControl.java` at Trino 484](https://github.com/trinodb/trino/blob/484/plugin/trino-opa/src/main/java/io/trino/plugin/opa/OpaAccessControl.java)). [Chapter 6](#engine-enforcement) shows what a module beside the policy has to return for them. `lib.policy.request` builds the platform's requests in Trino's shape. Here an `emea` support agent, `agent-a05`, reads three columns of `support.tickets`:
Example `trino-shaped-request` (Python 3.12.15 · pyarrow 26.0.0 · pyiceberg 0.12.0 · regopy 1.5.2), passing in Edition 2026.10.1:
```python
import json
from lib import policy
# A support agent's query reads three columns of the tickets table: the request Trino's OPA plugin would send for it.
request = policy.request("agent-a05", "support.tickets", ["ticket_id", "opened_at", "body"])
print(json.dumps(request, indent=2))
```
Output:
```text
{
"context": {
"identity": {
"user": "agent-a05"
}
},
"action": {
"operation": "SelectFromColumns",
"resource": {
"table": {
"catalogName": "platform",
"schemaName": "support",
"tableName": "tickets",
"columns": [
"ticket_id",
"opened_at",
"body"
]
}
}
}
}
```
The operation is `SelectFromColumns`, Trino's name for a read, and the table sits under `action.resource.table` with its catalog, schema, name and the columns the query touches. The platform's request differs from Trino's in two places that matter. It leaves out `context.identity.groups`, because the policy looks a user's groups up in the same file as their attributes, `governance/users.toml`, so the tests and the policy read one source of truth. And it may carry a top-level `purpose`, such as `ai-training`, which Trino doesn't send. Behind Trino, we recommend deriving the purpose from the identity, for instance by giving the training pipeline a service user of its own, rather than trusting a value the client sets.
**Working rule:** Shape the policy's input like the requests of the engine that will enforce it, field for field, and add only what that engine can't send, at the top level where a reviewer sees it. If your engine isn't Trino, start from its request format instead, and keep the shape in one function, as `lib.policy.request` does, so tests and callers can't drift apart.
## Generate the data the policy reads {#generate-the-data-the-policy-reads}
The [request](#shape-the-request-like-trinos) says who asks for what. Everything else a rule needs, such as each user's groups and each column's classification, comes from a second document that OPA calls `data`. A fact in that document is only as current as its source, and every fact the platform's policy reads has an owner already. Users and their attributes live in `governance/users.toml`. Each column's classification lives in `governance/metadata.toml`, the reviewed metadata file that [chapter 2](#key-column-metadata-by-field-id) keys by Iceberg field ID. And which instructor teaches which course is in the `learning.courses` table itself. Typing any of these into the policy, or copying them into a second file, makes a copy that someone has to remember to update.
`lib.policy.data_document` builds the document from those three sources each time a policy is loaded, and never from anything typed:
Example `generated-policy-data` (Python 3.12.15 · pyarrow 26.0.0 · pyiceberg 0.12.0 · regopy 1.5.2), passing in Edition 2026.10.1:
```python
from lib import policy
# What the policy reads besides the request, built from the files that own each fact.
data = policy.data_document()["platform"]
print("users, from governance/users.toml:", len(data["users"]))
print("agent-a05:", data["users"]["agent-a05"])
print("instructor-17:", data["users"]["instructor-17"])
classifications = data["classifications"]
print("tables, from governance/metadata.toml:", len(classifications))
print("columns classified:", sum(len(columns) for columns in classifications.values()))
print("crm.learners.email:", classifications["crm.learners"]["email"])
print("billing.payments.card_number:", classifications["billing.payments"]["card_number"])
courses = data["instructor_courses"]
print("instructors with courses, from learning.courses:", len(courses))
print("instructor 17's courses:", courses["17"])
```
Output:
```text
users, from governance/users.toml: 8
agent-a05: {'groups': ['support'], 'region': 'emea'}
instructor-17: {'groups': ['instructor'], 'instructor_id': 17}
tables, from governance/metadata.toml: 10
columns classified: 77
crm.learners.email: personal
billing.payments.card_number: sensitive
instructors with courses, from learning.courses: 48
instructor 17's courses: [2, 75, 80, 87, 103, 134]
```
The document holds 8 users, the classifications of 77 columns across 10 tables, and the courses of 48 instructors. `agent-a05` has a group and a region, `instructor-17` a group and an instructor ID, and nothing in `users.toml` says which courses instructor 17 teaches: the document finds [2, 75, 80, 87, 103, 134] in `learning.courses`. When the platform publishes another of their courses, the instructor's row filter includes it the next time the policy's data is generated, with no edit anywhere. A classification works the same way: changing `card_number` from `sensitive` would take one reviewed change to `metadata.toml`, and the policy would follow.
**Working rule:** Generate the policy's data from the file or table that owns each fact, every time you load the policy, and never copy a fact into the policy or into a second file. When a fact has no owner yet, such as a support agent's region, give it one file, reviewed like code, as the platform does with `users.toml`.
## Deny by default, then allow by role {#deny-by-default-then-allow-by-role}
With the [request](#shape-the-request-like-trinos) and its [data](#generate-the-data-the-policy-reads) in place, the first thing a policy has to settle is what happens to a request no rule anticipated. Saltzer and Schroeder's principle of fail-safe defaults answers it: "Base access decisions on permission rather than exclusion" ([Saltzer and Schroeder, 1975](https://web.mit.edu/Saltzer/www/publications/protection/Basic.html)). In Rego, that is a default value. A rule whose body never evaluates to true has no value, which Rego calls **undefined**, and "The default value is used when all the rules sharing the same name are undefined" ([OPA policy language](https://www.openpolicyagent.org/docs/policy-language)), so with `default allow := false`, a request that no `allow` rule matches is denied, whoever sends it and whatever it asks for.
The rules that do allow start from roles: role-based access control, first formalized by Ferraiolo and Kuhn, grants access by the job a principal holds rather than by who they are ([Ferraiolo and Kuhn, 1992](https://csrc.nist.gov/files/pubs/conference/1992/10/13/rolebased-access-controls/final/docs/ferraiolo-kuhn-92.pdf)). On the platform, a role is a group in `users.toml`, and the policy's `readable` object maps each group to the tables it may read. A request is allowed when it's a read, some group of the user may read the table, and nothing the group's row filters depend on is missing:
```rego
default allow := false
allow if {
input.action.operation == "SelectFromColumns"
readable_by_a_group
not missing_attribute
}
```
The last line is a guard against a failure this chapter's [section on production](#policies-in-production) shows by removing it. The example prints the map and decides four requests:
Example `allow-by-role` (Python 3.12.15 · pyarrow 26.0.0 · pyiceberg 0.12.0 · regopy 1.5.2), passing in Edition 2026.10.1:
```python
from lib import policy
rego = policy.interpreter()
# The groups the policy knows, and the tables each may read: the role part of the policy.
readable = policy.evaluate(rego, "data.platform.access.readable", {})
for group, tables in readable.items():
print(f"{group} may read:", ", ".join(tables))
# Who may read what. Anything no rule allows is denied, including a user or an operation the policy has never seen.
requests = [
policy.request("finance-1", "billing.payments", ["amount"]),
policy.request("marketing-1", "billing.payments", ["amount"]),
policy.request("someone-else", "crm.learners", ["learner_id"]),
]
delete = policy.request("finance-1", "billing.payments", ["amount"])
delete["action"]["operation"] = "DeleteFromTable"
for request in requests + [delete]:
decision = policy.decide(request, rego)
who, table = request["context"]["identity"]["user"], request["action"]["resource"]["table"]
print(f"{who}, {request['action']['operation']} on {table['schemaName']}.{table['tableName']}:",
"allowed" if decision["allow"] else "denied", "|", "; ".join(decision["reasons"]))
```
Output:
```text
ai may read: learning.courses, learning.reviews
data-science may read: analytics.course_performance, analytics.learner_activity, billing.payments, crm.learners, learning.courses, learning.enrollments, learning.events, learning.reviews
finance may read: billing.payments, learning.courses, learning.enrollments
instructor may read: learning.courses, learning.enrollments, learning.reviews
marketing may read: crm.learners, learning.courses, marketing.audience
support may read: crm.learners, support.tickets
finance-1, SelectFromColumns on billing.payments: allowed | finance may read billing.payments
marketing-1, SelectFromColumns on billing.payments: denied | no group of this user may read this table
someone-else, SelectFromColumns on crm.learners: denied | no group of this user may read this table
finance-1, DeleteFromTable on billing.payments: denied | DeleteFromTable is not an operation this policy allows
```
Finance may read `billing.payments`, and the reason says which group allowed it. Marketing may not, and neither may `someone-else`, a user the policy has never heard of: nothing grants either of them anything, so the default holds. The fourth request is finance again, on the same table, asking to delete rather than read. No rule mentions `DeleteFromTable`, so it's denied too, and the reason says why. A policy that tried to list what is forbidden would have needed a rule for that operation; this one needs none.
Every decision carries its reasons, so a denial can be explained to the person who got it, and logged.
**Working rule:** Start every policy with `default allow := false`, and allow only through rules that name the role and the resource. Keep the map from roles to tables in one place, where a reviewer can read all of it in one sitting, and give every decision a reason. When a request is denied that should have been allowed, add a rule that allows it; never weaken the default.
## Narrow by attribute: region, consent and purpose {#narrow-by-attribute}
[Roles](#deny-by-default-then-allow-by-role) say which tables a group may read; they can't say which rows. NIST's guide to attribute-based access control defines the general approach: access "is determined by evaluating attributes associated with the subject, object, requested operations, and, in some cases, environment conditions", and it treats roles as one such attribute: "RBAC works on the attribute of 'role'" ([NIST SP 800-162](https://doi.org/10.6028/NIST.SP.800-162)). The alternative, a role for every combination, ends in what the same guide calls "role explosion".
The platform shows why. Support by region would need three support roles; add marketing's consent and a purpose for AI training, and the roles multiply again. Instead, the policy keeps one role per job and reads three attributes. The **region** is an attribute of the principal, from `users.toml`. **Consent** is an attribute of the data: the learner's `marketing_consent` and `ai_training_consent` columns in `crm.learners`. The **purpose** is an attribute of the request. Each one becomes a row filter, a structured condition on a column that the engine adds to the query. The support agent's filter reads the agent's region:
```rego
row_filters contains {"column": "support_region", "equals": user.region} if {
"support" in groups
table in {"crm.learners", "support.tickets"}
}
```
The example asks for the filters of three support agents, an instructor, marketing, and a data scientist reading reviews twice, once with no purpose and once for `ai-training`:
Example `attributes-as-row-filters` (Python 3.12.15 · pyarrow 26.0.0 · pyiceberg 0.12.0 · regopy 1.5.2), passing in Edition 2026.10.1:
```python
import json
from lib import policy
rego = policy.interpreter()
# Region is the agent's attribute, consent the learner's, and purpose the request's. Each becomes a row filter.
reads = [
("agent-a01", "support.tickets", ["ticket_id"], None),
("agent-a05", "support.tickets", ["ticket_id"], None),
("agent-a09", "support.tickets", ["ticket_id"], None),
("instructor-17", "learning.reviews", ["rating"], None),
("marketing-1", "crm.learners", ["email"], None),
("analyst-1", "learning.reviews", ["rating", "body"], None),
("analyst-1", "learning.reviews", ["rating", "body"], "ai-training"),
]
for user, table, columns, purpose in reads:
decision = policy.decide(policy.request(user, table, columns, purpose), rego)
who = user + (f" for {purpose}" if purpose else "")
print(f"{who} reads {table}, row filters:", json.dumps(decision["row_filters"]))
if "body" in columns:
print(f"{who} reads {table}, body shown as:", decision["column_masks"]["body"])
```
Output:
```text
agent-a01 reads support.tickets, row filters: [{"column": "support_region", "equals": "americas"}]
agent-a05 reads support.tickets, row filters: [{"column": "support_region", "equals": "emea"}]
agent-a09 reads support.tickets, row filters: [{"column": "support_region", "equals": "apac"}]
instructor-17 reads learning.reviews, row filters: [{"column": "course_id", "in": [2, 75, 80, 87, 103, 134]}]
marketing-1 reads crm.learners, row filters: [{"column": "marketing_consent", "equals": true}, {"column": "status", "equals": "active"}]
analyst-1 reads learning.reviews, row filters: []
analyst-1 reads learning.reviews, body shown as: redact
analyst-1 for ai-training reads learning.reviews, row filters: [{"learner_consented": "ai_training_consent"}]
analyst-1 for ai-training reads learning.reviews, body shown as: redact_contacts
```
The three agents get the same rule with three values, one per region, from one role. Instructor 17's filter lists their courses, generated from `learning.courses`. Marketing gets two filters, consent given and account active, and an engine applies both. The data scientist's two reads show what the purpose does. With no purpose, every review is visible, but the review text, `body`, is shown as `redact`. For `ai-training`, only the reviews of learners who agreed to AI training remain, and the text comes back as `redact_contacts`, with contact details taken out. That filter, `learner_consented`, tests the review's author, not the review, so enforcing it takes a join to `crm.learners`. A learner who signed up before the platform asked about AI training has no answer in `ai_training_consent`, according to the platform's data card, so enforcing the filter has to count an unknown answer as a no.
The purpose is the one attribute the requester supplies, so it reveals more only under a narrower condition: stating `ai-training` shows review text, but only for consenting learners and without contact details. Leaving it out, or giving any other purpose, gets what the role allows anyway, with the text hidden.
**Working rule:** Model each condition as an attribute of the principal, the data or the request, read by one rule per role, rather than as another role. When an attribute comes from the requester, as a purpose does, let it unlock access only under a narrower condition, so that leaving it out, or getting it wrong, falls back to what the role allows anyway.
## Return the decision as one document {#return-the-decision-as-one-document}
A yes or no can't carry the [row filters](#narrow-by-attribute) that attributes produce, but a Rego decision isn't limited to yes or no: "OPA policies can represent decisions as arbitrary structured data (e.g., booleans, strings, maps, maps of lists of maps, etc.)" ([OPA philosophy](https://www.openpolicyagent.org/docs/philosophy)). The platform's policy uses that to return its whole answer as one document, `data.platform.access.decision`, with four keys: `allow`, the `row_filters`, the `column_masks`, and the `reasons`. `lib.policy.decide` asks for it with one query and returns it as a Python dictionary.
The filters and masks are data, never SQL. A filter names a column and a value or a list of values, or, for consent, the learner's consent column; a mask is a name, one of `redact`, `last4`, `pseudonymize` and `redact_contacts`. Whatever enforces the decision turns each into SQL with its own code, so nothing a request or a user record contains, such as a region typed into `users.toml`, can reach a query as SQL text. Masks follow each column's classification from the metadata. Sensitive columns are hidden from everyone, except the last four digits of a card number for finance. Personal columns are shown to support, which needs contact details, and masked for everyone else as their job requires. And a column the metadata doesn't classify is treated as sensitive:
```rego
classification(column) := level if {
level := data.platform.classifications[table][column]
} else := "sensitive"
```
For the platform, take marketing reading four columns of `crm.learners`. Three are in the metadata. The fourth, `linkedin_url`, stands for a column that a pipeline has added to the table and that nobody has classified yet:
Example `one-decision-document` (Python 3.12.15 · pyarrow 26.0.0 · pyiceberg 0.12.0 · regopy 1.5.2), passing in Edition 2026.10.1:
```python
import json
from lib import policy
# Marketing reads four columns of crm.learners. `linkedin_url` stands for a column added to the table and not yet
# classified in governance/metadata.toml.
request = policy.request("marketing-1", "crm.learners", ["full_name", "email", "phone", "linkedin_url"])
decision = policy.decide(request)
print(json.dumps(decision, indent=2))
print("keys:", ", ".join(sorted(decision)))
```
Output:
```text
{
"allow": true,
"column_masks": {
"linkedin_url": "redact",
"phone": "redact"
},
"reasons": [
"linkedin_url is sensitive, shown as redact",
"marketing may read crm.learners",
"phone is personal, shown as redact"
],
"row_filters": [
{
"column": "marketing_consent",
"equals": true
},
{
"column": "status",
"equals": "active"
}
]
}
keys: allow, column_masks, reasons, row_filters
```
The document allows the read, with marketing's two row filters. Two columns are masked: `phone`, which is personal and which marketing doesn't need for email, and `linkedin_url`, which is unclassified and so treated as sensitive until someone reviews it. `full_name` and `email` have no mask, and a column without a mask is shown as stored: marketing needs both to send an email. The reasons give one line per group that allowed the read and one per mask, so the document explains itself.
A denial carries empty filters and masks. The policy builds the two outcomes with two separate rules, so nothing downstream can mistake a denial for a read with some columns masked.
**Working rule:** Return one decision document per request, holding the allow, the row filters, the column masks and the reasons, and read it as a whole. Keep filters and masks as names and values that your enforcement code translates, never as SQL that the policy writes. If an engine needs SQL from the policy, as Trino does for its row filters and column masks ([Trino 484, OPA access control](https://trino.io/docs/484/security/opa-access-control.html)), generate it in one tested function between the two.
## Unit-test the policy in Rego {#unit-test-the-policy-in-rego}
The [decision document](#return-the-decision-as-one-document) is what a test can compare. That matters, because a policy decides who sees personal data, and a change to it can widen access without anyone seeing it in a diff. OPA's answer is tests written in Rego: a test is a rule whose name is "prefixed with test_", and "If the test rule is undefined or generates a non-true value the test result is reported as FAIL" ([OPA policy testing](https://www.openpolicyagent.org/docs/policy-testing)). A test sets up its request with `with`, which replaces part of `input` or `data` for one expression.
The book's engine, regopy, has no `opa test`, so `lib.policy.run_tests` plays its part: it finds every `test_` rule in the folder's `_test.rego` files, evaluates each, and reports PASS when the rule is true, FAIL when it's false or undefined, and ERROR when evaluation fails. Each of the platform's tests sets the user and the action as two separate paths, each to a literal value, for reasons [the section on regopy](#where-regopy-differs-from-opa) shows:
```rego
test_finance_sees_only_the_last_four_digits if {
decision := access.decision with input.context.identity.user as "finance-1"
with input.action as {"operation": "SelectFromColumns", "resource": {"table": {"schemaName": "billing", "tableName": "payments", "columns": ["amount", "card_number"]}}}
decision.allow
decision.row_filters == []
decision.column_masks == {"card_number": "last4"}
}
```
The tests cover each group's allow, its filters and its masks, and the ways a request is denied: a table the group isn't given, an operation other than a read, a user the policy doesn't know, and a support agent with no region.
Example `policy-unit-tests` (Python 3.12.15 · pyarrow 26.0.0 · pyiceberg 0.12.0 · regopy 1.5.2), passing in Edition 2026.10.1:
```python
from collections import Counter
from lib import policy
# Every `test_` rule in governance/policies/access_test.rego, run by name: PASS, FAIL or ERROR, as `opa test` reports.
results = policy.run_tests()
for test, result in results.items():
print(f"{result}: {test}")
print("tests passed:", f"{Counter(results.values())['PASS']} of {len(results)}")
```
Output:
```text
PASS: test_support_reads_only_its_own_region
PASS: test_support_without_a_region_is_denied
PASS: test_marketing_sees_only_consenting_active_learners
PASS: test_a_group_only_reads_its_own_tables
PASS: test_finance_sees_only_the_last_four_digits
PASS: test_data_science_never_sees_card_numbers
PASS: test_data_science_sees_pseudonyms_not_people
PASS: test_an_unclassified_column_is_hidden
PASS: test_an_unknown_user_is_denied
PASS: test_an_instructor_sees_only_their_courses
PASS: test_the_assistant_reads_only_consented_reviews
PASS: test_training_reads_only_consented_reviews_without_contacts
tests passed: 12 of 12
```
The run passes 12 of 12 tests. A test that checks only `decision.allow` would miss most mistakes, so every test of an allowed read compares the row filters and the column masks whole. In finance's test, `decision.row_filters == []` fails if the policy adds any filter to finance's reads, and `decision.column_masks == {"card_number": "last4"}` fails if it masks one column more or one fewer, or shows the card number another way.
**Working rule:** Write a test for every rule that grants or narrows access, and one for every way a request can be denied, and compare whole filters and masks, not single keys. Run the tests on every change to the policy and to the data the policy reads.
## Check who sees what from a decision table {#check-who-sees-what}
[Unit tests](#unit-test-the-policy-in-rego) are written in Rego by whoever writes the policy. The people who own the data, such as the support lead or the finance team, should be able to check the policy too, without reading Rego. A decision table gives them that: one row per request, saying who reads which table and what they must get back, in the terms of their job.
The platform's table is `lib.policy.WHO_SEES_WHAT`, a list of rows in Python. Each gives a user, a table, the columns read and an optional purpose, then whether the read is allowed, which columns the row filters test and which columns are masked, and how. The example runs every row through `lib.policy.decide`, which returns the [decision document](#return-the-decision-as-one-document), and prints what the policy decided, one line per row: the form to show the data's owners.
Example `who-sees-what` (Python 3.12.15 · pyarrow 26.0.0 · pyiceberg 0.12.0 · regopy 1.5.2), passing in Edition 2026.10.1:
```python
from lib import policy
rego = policy.interpreter()
# Each row of lib.policy.WHO_SEES_WHAT is a request and the decision a reviewer expects for it.
matching = 0
for case in policy.WHO_SEES_WHAT:
decision = policy.decide(policy.request(case.user, case.table, case.columns, case.purpose), rego)
matching += policy.matches(case, decision)
who = case.user + (f" ({case.purpose})" if case.purpose else "")
shown = "allowed" if decision["allow"] else "denied"
filters = ", ".join(policy.filtered_on(decision)) or "-"
masks = ", ".join(f"{column}: {mask}" for column, mask in decision["column_masks"].items()) or "-"
print(f"{who:26} {case.table:17} {shown:8} rows: {filters:29} masks: {masks}")
print("rows of the table the policy matches:", f"{matching} of {len(policy.WHO_SEES_WHAT)}")
```
Output:
```text
agent-a05 crm.learners allowed rows: support_region masks: -
agent-a05 billing.payments denied rows: - masks: -
instructor-17 learning.reviews allowed rows: course_id masks: body: redact_contacts
instructor-17 crm.learners denied rows: - masks: -
marketing-1 crm.learners allowed rows: marketing_consent, status masks: phone: redact
finance-1 billing.payments allowed rows: - masks: card_number: last4
analyst-1 billing.payments allowed rows: - masks: card_number: redact
analyst-1 crm.learners allowed rows: - masks: email: pseudonymize, notes: redact
analyst-1 learning.reviews allowed rows: - masks: body: redact
analyst-1 (ai-training) learning.reviews allowed rows: ai_training_consent masks: body: redact_contacts
assistant-1 learning.reviews allowed rows: ai_training_consent masks: body: redact_contacts
someone-else crm.learners denied rows: - masks: -
rows of the table the policy matches: 12 of 12
```
12 of 12 rows match. Read down the table and each role's rules show up as outcomes: the support agent filtered by region and denied payments; the instructor filtered by course and denied the learners table; finance seeing `card_number` as `last4` where the data scientist sees `redact`; the data scientist's two reads of reviews, with and without a purpose. A reviewer who thinks a row is wrong can say so in a pull request without knowing which rule produced it.
**Working rule:** Keep a who-sees-what table in the repository with the policy, run it with the unit tests, and show its printed form to the data's owners, in terms they can check. When an owner disagrees with a row, change the row first, and let the failing row drive the change to the policy.
## Know where regopy differs from OPA {#where-regopy-differs-from-opa}
The platform's [tests](#unit-test-the-policy-in-rego) run in regopy because the book's examples must run offline, in seconds, inside Python, and OPA is a separate program. regopy 1.5.2 is close to OPA: its engine says "We support v1.18.1 of Rego as defined by OPA" and that it passes "all the non-builtin specific test suites" OPA publishes, and it lists the built-ins it doesn't support, such as `net.*` and `http.send` ([rego-cpp README at v1.5.2](https://github.com/microsoft/rego-cpp/blob/v1.5.2/README.md)). regopy also differs from OPA in ways its README doesn't list. The platform's policy and tests avoid each of the cases that follow, and OPA itself is the reference for anything in production.
The first group runs without any error:
Example `regopy-silent-differences` (Python 3.12.15 · regopy 1.5.2), passing in Edition 2026.10.1:
```python
import json
from regopy import Input, Interpreter
def ask(rules: str, query: str, request: dict) -> str:
"""One query's answer from regopy: the value bound to x, or "undefined"."""
rego = Interpreter()
rego.add_module("example", "package example\n" + rules)
rego.set_input(Input(request))
output = str(rego.query(query))
return output if output == "undefined" else json.dumps(json.loads(output)["bindings"]["x"])
admins_only = 'default allow := false\nallow if input.user == "admin"\n'
print("a false decision, queried on its own:", ask(admins_only, "data.example.allow", {"user": "guest"}))
print("the same decision, bound to a variable:", ask(admins_only, "x := data.example.allow", {"user": "guest"}))
pre_1_0 = 'allow { input.user == "admin" }\n' # no `if`: OPA 1.x refuses to load this
print("a rule in pre-1.0 syntax:", ask(pre_1_0, "x := data.example.allow", {"user": "admin"}))
typo = 'region := "emea"\ndefault allow := false\nallow if not deny\ndeny if input.region != regoin\n'
print("a deny with a misspelt, unbound variable, for apac:", ask(typo, "x := data.example.allow", {"region": "apac"}))
mocks = """region := input.user.region
emea := {"region": "emea"}
with_a_literal := seen if { seen := region with input.user as {"region": "emea"} }
with_a_reference := seen if { seen := region with input.user as emea }
"""
print("region under a mock written as a literal:", ask(mocks, "x := data.example.with_a_literal", {"user": {"region": "apac"}}))
print("region under a mock that refers to a rule:", ask(mocks, "x := data.example.with_a_reference", {"user": {"region": "apac"}}))
```
Output:
```text
a false decision, queried on its own: undefined
the same decision, bound to a variable: false
a rule in pre-1.0 syntax: true
a deny with a misspelt, unbound variable, for apac: true
region under a mock written as a literal: "emea"
region under a mock that refers to a rule: "apac"
```
- **A false decision comes back undefined.** Asked for `data.example.allow` on its own, regopy answers `undefined` where the value is `false`, so "denied" can't be told from "no decision". Bound to a variable, the same query gives `false`. `lib.policy` binds every decision it reads.
- **Pre-1.0 syntax runs.** Since OPA 1.0, "Using `if` for all rule definitions and `contains` for multi-value rules is now mandatory" ([OPA v1.0.0 release notes](https://github.com/open-policy-agent/opa/releases/tag/v1.0.0)), and rego-cpp's changelog says that "Support for v0 Rego has been dropped" ([rego-cpp changelog](https://github.com/microsoft/rego-cpp/blob/v1.5.2/CHANGELOG)). Yet regopy evaluates `allow { ... }` and answers `true`, so a policy can pass here and fail to load in OPA, and so behind Trino.
- **An unbound variable switches a rule off.** OPA requires a variable in a comparison to "appear in another expression in the same rule that would cause the variable to be bound" ([OPA policy language](https://www.openpolicyagent.org/docs/policy-language)), and rejects the policy otherwise. regopy runs it: a `deny` with `regoin` misspelt for `region` is silently undefined, so `allow if not deny` allows an `apac` request: `true`.
- **A mock that refers to another rule is ignored.** In a policy whose `region` rule reads `input.user.region`, a mock written as a literal, `with input.user as {"region": "emea"}`, gives `"emea"`. Written as a reference to a rule holding the same object, it gives the real input's `"apac"`.
The second group fails, and one of the failures takes Python down with it. The first example adds a test to a copy of the platform's tests, and the second formats the same string from nine sources:
Example `regopy-failed-outputs` (Python 3.12.15 · pyarrow 26.0.0 · pyiceberg 0.12.0 · regopy 1.5.2), passing in Edition 2026.10.1:
```python
import signal
import subprocess
import sys
import tempfile
from collections import Counter
from pathlib import Path
from lib import policy
# The platform's tests, plus one that replaces the whole input at once rather than one path at a time.
whole_input = """
test_finance_reads_payments if {
access.decision.allow with input as {"context": {"identity": {"user": "finance-1"}}, "action": {"operation": "SelectFromColumns", "resource": {"table": {"schemaName": "billing", "tableName": "payments", "columns": ["amount"]}}}}
}
"""
with tempfile.TemporaryDirectory() as name:
folder = Path(name)
for path in policy.POLICIES.glob("*.rego"):
(folder / path.name).write_text(path.read_text(encoding="utf-8"), encoding="utf-8")
with open(folder / "access_test.rego", "a", encoding="utf-8") as tests:
tests.write(whole_input)
results = policy.run_tests(folder)
print("tests that end in ERROR:", f"{Counter(results.values())['ERROR']} of {len(results)}")
rego = policy.interpreter(folder, tests=True)
print("the query true succeeds:", rego.query("true").ok())
# lib.policy checks each output before reading it, so a decision asked of this interpreter raises an error.
try:
policy.decide(policy.request("finance-1", "billing.payments", ["amount"]), rego)
except RuntimeError as error:
print("lib.policy.decide raises:", error)
# Printing one of those failed outputs, in a separate Python process.
crash = "import sys; from pathlib import Path; from lib import policy; print(policy.interpreter(Path(sys.argv[1]), tests=True).query('true'))"
ended = subprocess.run([sys.executable, "-c", crash, str(folder)], capture_output=True)
print("printing the failed output ends Python with:", signal.Signals(-ended.returncode).name)
```
Output:
```text
tests that end in ERROR: 13 of 13
the query true succeeds: False
lib.policy.decide raises: regopy couldn't evaluate data.platform.access.decision, so there is no decision to read
printing the failed output ends Python with: SIGSEGV
```
Example `regopy-sprintf-quotes` (Python 3.12.15 · regopy 1.5.2), passing in Edition 2026.10.1:
```python
import json
from regopy import Input, Interpreter
# The same label, sprintf("%s.%s", [schema, input.table]), with the schema "crm" taken from nine places.
helpers = 'crm := "crm"\nschema_of(x) := "crm" if x == 1\n'
cases = [
("from input.schema", "input.schema", "true"),
("from a rule", "crm", "true"),
("from a function", "schema_of(1)", "true"),
("from iterating over a set", "s", 'some s in {"crm"}'),
("from object.get, the key found", 'object.get(input, "schema", "public")', "true"),
("from object.get, its default", 'object.get(input, "namespace", "crm")', "true"),
("from lower(input.schema)", "lower(input.schema)", "true"),
("from concat", 'concat("", ["c", "rm"])', "true"),
("from a local variable set to a literal", "v", 'v := "crm"'),
]
for name, schema, body in cases:
rego = Interpreter()
rego.add_module("example", f'package example\n{helpers}label := sprintf("%s.%s", [{schema}, input.table]) if {{ {body} }}\n')
rego.set_input(Input({"schema": "crm", "table": "learners"}))
try:
print(f"{name}:", json.loads(str(rego.query("x := data.example.label")))["bindings"]["x"])
except json.JSONDecodeError as error:
print(f"{name}, regopy's invalid JSON:", error.doc)
```
Output:
```text
from input.schema: crm.learners
from a rule: crm.learners
from a function: crm.learners
from iterating over a set: crm.learners
from object.get, the key found: crm.learners
from object.get, its default, regopy's invalid JSON: {"expressions":[true], "bindings":{"x":""crm".learners"}}
from lower(input.schema), regopy's invalid JSON: {"expressions":[true], "bindings":{"x":""crm".learners"}}
from concat, regopy's invalid JSON: {"expressions":[true], "bindings":{"x":""crm".learners"}}
from a local variable set to a literal, regopy's invalid JSON: {"expressions":[true], "bindings":{"x":""crm".learners"}}
```
- **Replacing the whole input breaks every query.** One extra test that writes its request as `with input as {...}`, and evaluates rules that call built-ins such as `sprintf`, makes every query in that interpreter fail. Every test then ends in ERROR, 13 of 13, and so does the first query asked of a fresh interpreter loaded with the same files, before any test runs: asked whether the query `true` succeeded, regopy says `False`. Replacing one path at a time, as the platform's tests do, works.
- **Printing a failed output crashes Python.** Converting one of those failed outputs to a string ends the process with `SIGSEGV`. So `lib.policy` checks `output.ok()` before it reads anything, and a decision asked of that interpreter raises an error instead: `regopy couldn't evaluate data.platform.access.decision, so there is no decision to read`.
- **`sprintf` keeps stray quotes.** Formatting a string returned by some built-ins, such as `object.get`'s default, `lower` or `concat`, or a local variable set to a literal, keeps its quotes, and regopy's own output is then invalid JSON, `{"expressions":[true], "bindings":{"x":""crm".learners"}}`, which its Python wrapper fails to parse. Strings from `input`, a rule, a function, iteration and `object.get`'s found value format cleanly, and the platform's policy formats only strings like those.
**Working rule:** Treat regopy as a fast local evaluator and OPA as the reference. Write Rego v1 only, bind every decision to a variable, mock input one path at a time with literal values, and check that an output succeeded before reading it. Before a policy guards production data, check and test it with OPA itself, with `opa check --strict` and `opa test`.
## Write the same rules in Cedar {#the-same-rules-in-cedar}
Rego isn't the only language for policies like the platform's. Cedar, which its designers present as a language for "Expressive, Fast, Safe, and Analyzable Authorization" ([Cutler et al., 2024](https://arxiv.org/abs/2403.04651)), takes a narrower view of a decision. A Cedar request names a principal, an action and a resource, and the answer follows three rules: "default deny", "forbid overrides permit", and "skip on error: if a policy's evaluation returns error, the policy does not factor into the authorization response; it is skipped" ([Cedar, authorization](https://docs.cedarpolicy.com/auth/authorization.html)). A schema declares every type of entity, the kinds of principal and resource a request can name, with its attributes, and "By default, attributes that you define are required" ([Cedar, schema](https://docs.cedarpolicy.com/schema/schema.html)). Validating policies against the schema catches mistakes before any request: "If your policies are deemed valid, they are sure not to exhibit most errors that could arise during request evaluation, for requests that adhere to the expectations defined by the schema" ([Cedar, validation](https://docs.cedarpolicy.com/policies/validation.html)). The book's examples run Cedar with cedarpy 4.12.2, which "helps you use the (Rust) Cedar Policy library from Python" ([cedarpy](https://pypi.org/project/cedarpy/4.12.2/)).
The platform's rules in Cedar are in `governance/policies/access.cedar`, with the schema in `access.cedarschema`. A `permit` per group repeats the Rego policy's [`readable` map](#deny-by-default-then-allow-by-role). A `forbid` keeps support agents to learners in their own region, and another hides sensitive columns from everyone except finance's view of card numbers. `lib.policy.cedar_entities` generates the users, groups, tables and columns from the same [data document](#generate-the-data-the-policy-reads) the Rego policy reads, and the example adds the first learner of `crm.learners` in each region:
Example `cedar-same-rules` (Python 3.12.15 · cedarpy 4.12.2 · pyarrow 26.0.0 · pyiceberg 0.12.0 · regopy 1.5.2), passing in Edition 2026.10.1:
```python
import cedarpy
from lib import policy
from lib.platform import load
policies = policy.CEDAR_POLICIES.read_text(encoding="utf-8")
schema = policy.CEDAR_SCHEMA.read_text(encoding="utf-8")
print("policies valid against the schema:", cedarpy.validate_policies(policies, schema).validation_passed)
typo = cedarpy.validate_policies(policies.replace("principal.region", "principal.regoin"), schema)
print("with region misspelt:", typo.errors[0].error)
# Entities generated from the files the Rego policy reads, plus the first learner of crm.learners in each region.
first = {}
for learner in load("crm.learners").select(["learner_id", "support_region"]).to_pylist():
first.setdefault(learner["support_region"], learner)
entities = policy.cedar_entities(learners=first.values())
def ask(user: str, resource: str) -> cedarpy.AuthzResult:
request = {"principal": f'User::"{user}"', "action": 'Action::"read"', "resource": resource, "context": {}}
return cedarpy.is_authorized(request, policies, entities, schema)
for user, kind, name in [("finance-1", "Column", "billing.payments.card_number"),
("analyst-1", "Column", "billing.payments.card_number"),
("analyst-1", "Column", "billing.payments.amount"),
("marketing-1", "Table", "billing.payments")]:
print(f"{user} reads {name}:", ask(user, f'{kind}::"{name}"').decision.value)
for region, learner in sorted(first.items()):
print(f"agent-a05 (emea) reads a learner in {region}:", ask("agent-a05", f'Learner::"{learner["learner_id"]}"').decision.value)
answer = ask("finance-1", 'Column::"billing.payments.card_number"')
print("everything Cedar returns for finance-1's card numbers:", answer.decision.value, list(answer.diagnostics.reasons), list(answer.diagnostics.errors))
```
Output:
```text
policies valid against the schema: True
with region misspelt: for policy `policy6`, attribute `regoin` on entity type `User` not found
finance-1 reads billing.payments.card_number: Allow
analyst-1 reads billing.payments.card_number: Deny
analyst-1 reads billing.payments.amount: Allow
marketing-1 reads billing.payments: Deny
agent-a05 (emea) reads a learner in americas: Deny
agent-a05 (emea) reads a learner in apac: Deny
agent-a05 (emea) reads a learner in emea: Allow
everything Cedar returns for finance-1's card numbers: Allow ['policy3'] []
```
The policies validate against the schema, and misspelling `region` gives a precise error before anything runs: for policy `policy6`, attribute `regoin` on entity type `User` not found. Rego has no such check by default. The same typo in an attribute, `user.regoin` in the support agent's row filter, is undefined in OPA and regopy alike, and `opa check --strict` passes it, because OPA checks references against a schema only when you give it one ([OPA policy language, schemas](https://www.openpolicyagent.org/docs/policy-language#using-schemas-to-enhance-the-rego-type-checker)), and the platform gives none. The agent's row filter disappears, and only a test that compares the filters, such as `test_support_reads_only_its_own_region`, catches it. The decisions agree with the Rego policy where Cedar can express them. Finance may read the `card_number` column, and the data scientist may not, where Rego allows the read and redacts the column. The data scientist may read `amount`, and marketing may not read payments. The `emea` agent may read the `emea` learner and neither of the others.
The last line of the output is everything Cedar returns for finance's card numbers: a decision, and the ID of the policy that decided it. cedarpy names the policies by position, `policy3` for the fourth, rather than by their `@id` annotations, such as `finance-tables`. Cedar's documentation explains why that's allowed: "@id is not special in the Cedar language … The Cedar CLI uses the @id annotation to set policy IDs, but other interfaces, such as the Cedar APIs, have other ways to set policy IDs" ([Cedar, syntax](https://docs.cedarpolicy.com/policies/syntax-policy.html)). Nothing in the answer says `last4`, the [mask](#return-the-decision-as-one-document) the Rego policy returns for finance's card numbers, because Cedar can permit a column but has no way to say "show the last four digits": that rule would move into the enforcement code, outside the policy and its tests. Rows are the same. Cedar decides one learner at a time, so filtering the platform's learners means one request per learner, or Cedar's experimental partial evaluation, which "evaluates whatever is known and returns residual policies for the parts that aren't", and whose result "is not a final authorization decision" ([cedarpy](https://pypi.org/project/cedarpy/4.12.2/)). Cedar's paper makes the trade-off from the other side: "Rego's added expressiveness is a blessing but also a curse. It provides no built-in authorization concepts such as 'allow' or 'deny'". The same paper reports Cedar 42.8 to 80.8 times faster than Rego at the median, on three workloads with transitive relations, with Cedar 3.0.1 against OPA's Rego 0.61.0. Those are its designers' measurements, not this book's, and they say nothing about regopy.
**Working rule:** Choose Cedar when every question is "may this principal do this to this resource?", as in an application's permission checks, and you want a schema to check every policy before it runs. Choose Rego when the answer has to carry data, as a warehouse's row filters and column masks do, or when the engine already asks OPA, as Trino does. For the platform's warehouse, we recommend Rego.
## What goes wrong in production {#policies-in-production}
We consider the most damaging failure a rule that silently doesn't apply because an attribute is missing, and it happens in both languages. In Rego, "Expressions that refer to undefined values are also undefined" ([OPA policy language](https://www.openpolicyagent.org/docs/policy-language)). The support agent's [row filter](#narrow-by-attribute) takes its value from `user.region`, so for a user without a region the filter doesn't exist at all. A rule that allows by group still allows, and no filter means every row.
To see it, take a copy of the platform's policy and tests, in a temporary folder, with the one line `not missing_attribute` taken out of [`allow`](#deny-by-default-then-allow-by-role). Then add a support agent to the data with their group and no region, as happens when someone is given a role before anyone sets their region, and ask both policies for the agent's tickets:
Example `missing-region-guard` (Python 3.12.15 · pyarrow 26.0.0 · pyiceberg 0.12.0 · regopy 1.5.2), passing in Edition 2026.10.1:
```python
import tempfile
from pathlib import Path
from lib import policy
from lib.platform import load
# A support agent added to users.toml with their group, before anyone has set their region.
data = policy.data_document()
data["platform"]["users"]["agent-new"] = {"groups": ["support"]}
request = policy.request("agent-new", "support.tickets", ["ticket_id", "body"])
regions = sorted(set(load("support.tickets").column("support_region").to_pylist()))
# A copy of the platform's policy and tests, with one line taken out of `allow`: the missing-attribute guard.
with tempfile.TemporaryDirectory() as name:
folder = Path(name)
for path in policy.POLICIES.glob("*.rego"):
(folder / path.name).write_text(path.read_text(encoding="utf-8").replace("\tnot missing_attribute\n", ""), encoding="utf-8")
for label, policies in [("with the guard", policy.POLICIES), ("without the guard", folder)]:
decision = policy.decide(request, policy.interpreter(policies, data=data))
seen = [r for r in regions if decision["allow"] and all(f["equals"] == r for f in decision["row_filters"])]
print(f"{label}, allowed:", decision["allow"])
print(f"{label}, reasons:", "; ".join(decision["reasons"]))
print(f"{label}, row filters:", decision["row_filters"])
print(f"{label}, regions whose tickets the agent sees:", ", ".join(seen) or "none")
failing = [test for test, result in policy.run_tests(folder).items() if result != "PASS"]
print("tests that fail without the guard:", ", ".join(failing))
```
Output:
```text
with the guard, allowed: False
with the guard, reasons: support's row filter needs the user's region, and there is none
with the guard, row filters: []
with the guard, regions whose tickets the agent sees: none
without the guard, allowed: True
without the guard, reasons: support may read support.tickets; support's row filter needs the user's region, and there is none
without the guard, row filters: []
without the guard, regions whose tickets the agent sees: americas, apac, emea
tests that fail without the guard: test_support_without_a_region_is_denied
```
With the guard, the agent is denied, and the reason names what is missing: support's row filter needs the user's region, and there is none. Without it, the request is allowed with row filters `[]`, and the agent sees the tickets of americas, apac, emea: every region, including every learner's message and contact details. The decision's own reasons say the region is missing, but nothing in `allow` asked. `default allow := false` didn't help either, because a rule did allow. One test catches it, `test_support_without_a_region_is_denied`, which is why the policy's tests include the case of the missing attribute.
Cedar fails the same way, through [skip on error](#the-same-rules-in-cedar). Take the support agents' `forbid` from the platform's Cedar policies without its `has region` check, so it reads `principal.region` on an agent who has none:
Example `cedar-missing-region` (Python 3.12.15 · cedarpy 4.12.2 · pyarrow 26.0.0 · pyiceberg 0.12.0 · regopy 1.5.2), passing in Edition 2026.10.1:
```python
import cedarpy
from lib import policy
from lib.platform import load
guarded = policy.CEDAR_POLICIES.read_text(encoding="utf-8")
unguarded = guarded.replace("principal has region && ", "") # the same forbid, without its `has` check
schema = policy.CEDAR_SCHEMA.read_text(encoding="utf-8")
# A new support agent with no region yet, and the first learner of crm.learners in each region.
data = policy.data_document()
data["platform"]["users"]["agent-new"] = {"groups": ["support"]}
first = {}
for learner in load("crm.learners").select(["learner_id", "support_region"]).to_pylist():
first.setdefault(learner["support_region"], learner)
entities = policy.cedar_entities(learners=first.values(), data=data)
for name, policies in [("with `has region`", guarded), ("without it", unguarded)]:
for region, learner in sorted(first.items()):
request = {"principal": 'User::"agent-new"', "action": 'Action::"read"',
"resource": f'Learner::"{learner["learner_id"]}"', "context": {}}
answer = cedarpy.is_authorized(request, policies, entities, schema)
print(f"{name}, a learner in {region}:", answer.decision.value)
print(f"{name}, policies skipped on an error:", list(answer.diagnostics.errors))
print("validation without it:", cedarpy.validate_policies(unguarded, schema).errors[0].error)
```
Output:
```text
with `has region`, a learner in americas: Deny
with `has region`, a learner in apac: Deny
with `has region`, a learner in emea: Deny
with `has region`, policies skipped on an error: []
without it, a learner in americas: Allow
without it, a learner in apac: Allow
without it, a learner in emea: Allow
without it, policies skipped on an error: ['error while evaluating policy `policy6`: `User::"agent-new"` does not have the attribute `region`']
validation without it: for policy `policy6`, unable to guarantee safety of access to optional attribute `region` on entity type `User`
```
With the check, the agent is denied every learner. Without it, evaluating the `forbid` raises an error, Cedar skips the policy, the support `permit` still applies, and the agent may read the learners of all three regions. Passing the schema to `is_authorized` didn't stop it, because the schema declares `region` optional: users outside support have none. Validation did, before any request: for policy `policy6`, unable to guarantee safety of access to optional attribute `region` on entity type `User`.
Other failures to watch for:
- **A misspelt attribute is undefined, not an error.** `user.regoin` for `user.region` passes `opa check --strict`, and the support filter disappears as it does for a missing region. Compare whole filters in the tests, and give OPA a schema if you want the check before any test runs.
- **A deny rule that reads a missing attribute fails open.** The guard fixes `allow`; a `deny` or `forbid` needs the same care. Deny explicitly when a required attribute is absent, and write the test for it before the rule.
- **Policies nobody can test.** Rules kept in an engine's settings or in hand-made views per role can't be unit-tested or diffed, so they drift. Keep the policy, its tests and its decision table in the repository, and run all three on every change.
- **regopy passes and OPA refuses.** Pre-1.0 syntax and unbound variables run in regopy. Check every policy with `opa check --strict` before deploying it to OPA.
- **Data that changed under the policy.** A support agent moved to another region is one edit to `users.toml`, and the decision follows only when the data document is rebuilt. Rebuild it whenever its sources change, and deploy it with the policy.
## What this chapter decided {#policies-decisions}
- **One policy, in Rego,** `governance/policies/access.rego`, denies by default, allows by group, and narrows by three attributes: the agent's region, the learner's consent and the request's purpose.
- **Requests are shaped like Trino's OPA requests,** built by `lib.policy.request`, plus an optional top-level `purpose`; groups come from `users.toml`, not from the request.
- **The policy's data is generated** by `lib.policy.data_document` from `users.toml`, `metadata.toml` and `learning.courses`, and never typed.
- **Each decision is one document** with `allow`, `row_filters`, `column_masks` and `reasons`. Filters and masks are data, never SQL, and the masks are `redact`, `last4`, `pseudonymize` and `redact_contacts`. A denial carries no filters or masks.
- **Missing facts fail closed:** a column the metadata doesn't classify is treated as sensitive, and a user whose row filter needs a missing attribute is denied.
- **The one purpose the policy knows is `ai-training`,** which limits reviews to learners who agreed to AI training and shows their text with contact details removed. A read without a purpose gets what the role allows, with review text hidden.
- **Two kinds of test run on every change:** the Rego unit tests through `lib.policy.run_tests`, and the who-sees-what table, `lib.policy.WHO_SEES_WHAT`.
- **regopy for the examples, OPA as the reference:** Rego v1 only, every decision bound to a variable, input mocked one path at a time with literals, and every output checked before it's read.
- **Cedar is for comparison** (`governance/policies/access.cedar`). It suits allow-or-deny checks on single resources; the platform's warehouse needs filters and masks as data, so it uses Rego.
# Enforcing policies on queries
By the end of this chapter you can enforce access decisions on every query the platform runs. You'll rewrite each query so that every table it reads is replaced by a subquery that keeps only the rows the user may see and masks the columns they may not, deny every query that reads anything else, test who gets what by running real queries, and say where a rewrite in front of DuckDB stops protecting anything.
The platform, the online learning platform this book follows, sells professional courses to adult learners, and its warehouse holds their contact details, their payments and the text they write. [Chapter 5](#access-policies-as-code) wrote the platform's access policy in Rego. Given a user, a table, the columns they want and, optionally, why they want them, the policy returns a **decision**: a document that says whether to `allow` the read, which **row filters** keep the rows the user may see, which **column masks** change how a column is shown (`redact`, `last4`, `pseudonymize` or `redact_contacts`), and the `reasons`. A decision does nothing on its own. A support agent's SQL reaches DuckDB as text, and something between the two has to apply the decision to every table that text reads, however the query is written.
That layer is this chapter's subject: the book's shared module `lib.enforce`, which we call the **enforcement layer**. Its `Enforcer` holds four things: a DuckDB connection with a view for each catalog table (`lib.warehouse.connect`), the tables' schema, the policy and a key for pseudonyms. Users never get the connection. `Enforcer.sql(query, user)` rewrites a user's query and runs the rewrite. A service exposes only `lib.enforce.answer`, which calls it and returns the rows: never the relation, a handle on the connection, or `Enforcer.run`, which executes SQL the layer wrote with the key attached. The examples run sqlglot 30.24.0 and DuckDB 2.0.0.
## Find every table and column a query reads {#find-what-a-query-reads}
A policy can only decide about what it's asked, so before the enforcement layer asks, it has to know every table a query reads and which columns it reads from each. Reading that off SQL text is a parsing problem with traps: `*`, aliases, joins with `USING`, common table expressions (CTEs) and identifiers in any case. sqlglot parses a query into a tree, and its `qualify` step resolves every column against a schema, qualifies it with its table, and expands `*` into the columns it stands for when it has a schema to expand against ([`qualify` at v30.24.0](https://github.com/tobymao/sqlglot/blob/v30.24.0/sqlglot/optimizer/qualify.py)). The enforcement layer builds that schema when it starts: it lists the catalog's tables, and reads each one's columns and types from the view `lib.warehouse.connect` created for it, so a view DuckDB holds for anything else never enters the schema.
For the platform, `Enforcer.reads` returns the columns each query reads from each table:
Example `tables-a-query-reads` (Python 3.12.15 · duckdb 2.0.0 · pyarrow 26.0.0 · pyiceberg 0.12.0 · regopy 1.5.2 · sqlglot 30.24.0), passing in Edition 2026.10.1:
```python
import sqlglot
from sqlglot import exp
from sqlglot.optimizer.normalize_identifiers import normalize_identifiers
from lib.catalog import open_catalog
from lib.enforce import Enforcer
platform = Enforcer(open_catalog(["crm.learners", "support.tickets"]))
# What each query reads, found by sqlglot's qualify against the catalog's schema, with `*` expanded.
queries = {
"every column": "select * from crm.learners",
"a join": """select l.country, count(*) as tickets
from crm.learners as l join support.tickets as t using (learner_id) group by all""",
"a CTE": """with emea as (select * exclude (notes) from crm.learners where support_region = 'emea')
select email from emea""",
"names in capitals": 'select EMAIL from "CRM"."LEARNERS"',
}
for name, sql in queries.items():
for table, columns in platform.reads(sql).items():
print(f"{name}, {table}: {', '.join(columns)}")
# DuckDB resolves quoted capitals to the lowercase table. Whether sqlglot lowercases them depends on the dialect.
capitals = 'select count(*) from "CRM"."LEARNERS"'
print('DuckDB, "CRM"."LEARNERS":', f"{platform.connection.sql(capitals).fetchall()[0][0]:,} rows")
for dialect in ["duckdb", "trino", "postgres"]:
table = normalize_identifiers(sqlglot.parse_one('select * from "LEARNERS"', read=dialect), dialect=dialect).find(exp.Table)
print(f'{dialect} normalizes "LEARNERS" to:', table.name)
```
Output:
```text
every column, crm.learners: ai_training_consent, city, closed_at, country, date_of_birth, email, full_name, learner_id, marketing_consent, notes, phone, signup_at, status, street_address, support_region
a join, crm.learners: country, learner_id
a join, support.tickets: learner_id
a CTE, crm.learners: ai_training_consent, city, closed_at, country, date_of_birth, email, full_name, learner_id, marketing_consent, phone, signup_at, status, street_address, support_region
names in capitals, crm.learners: email
DuckDB, "CRM"."LEARNERS": 5,000 rows
duckdb normalizes "LEARNERS" to: learners
trino normalizes "LEARNERS" to: learners
postgres normalizes "LEARNERS" to: LEARNERS
```
`SELECT *` becomes every column of `crm.learners`, and the join reads `learner_id` from both tables, because the `USING` clause compares them, though neither appears in the output. In the query with a CTE, the outer query returns only `email`, but the CTE names every column of `crm.learners` except `notes`, so the layer asks about all of those: it asks about every column a query names anywhere, which can be more than the query returns, and never fewer. `"CRM"."LEARNERS"` resolves to `crm.learners`, because the layer normalizes identifiers the way DuckDB resolves them before it matches anything: DuckDB returns 5,000 rows of `crm.learners` for that name. The dialect matters: sqlglot lowercases `"LEARNERS"` for DuckDB and Trino, and keeps it as `LEARNERS` for PostgreSQL, so the layer normalizes with the dialect of the engine that runs the query.
`qualify` has one gap that decides the layer's design: it fails on an unknown column of a known table, but it passes over a table that isn't in the schema at all, and leaves that table's `*` unexpanded. Run it on a join with `marketing.audience`, which stands for a table a pipeline wrote after the schema was read, then send the same query and five other reads through the layer:
Example `unknown-tables-denied` (Python 3.12.15 · duckdb 2.0.0 · pyarrow 26.0.0 · pyiceberg 0.12.0 · regopy 1.5.2 · sqlglot 30.24.0), passing in Edition 2026.10.1:
```python
import sqlglot
from sqlglot.optimizer.qualify import qualify
from lib.catalog import open_catalog
from lib.enforce import Denied, Enforcer
platform = Enforcer(open_catalog(["crm.learners"]))
# marketing.audience stands for a table a pipeline wrote after the schema was read. qualify alone lets it through.
query = "select * from crm.learners as l join marketing.audience as a on a.learner_id = l.learner_id"
print("qualify on its own:", qualify(sqlglot.parse_one(query, read="duckdb"), schema=platform.schema, dialect="duckdb").sql("duckdb"))
# The enforcement layer denies anything that isn't a catalog table, before it asks the policy.
attempts = {
"a table missing from the catalog": query,
"a column the table doesn't have": "select shoe_size from crm.learners",
"the scan behind a view": "select count(*) from scan_crm_learners",
"a Parquet file": "select count(*) from read_parquet('data/platform/learners.parquet')",
"a table function": "select count(*) from query_table('crm.learners')",
"a statement that isn't a query": "summarize crm.learners",
}
for name, sql in attempts.items():
try:
platform.rewrite(sql, "analyst-1")
print(f"{name}: allowed")
except Denied as reason:
print(f"{name}: denied, {reason}")
# What those reads would do on the layer's own connection, without the rewrite in front of it.
rows = lambda sql: f"{platform.connection.sql(sql).fetchall()[0][0]:,} rows"
print("the scan, run directly:", rows("select count(*) from scan_crm_learners"))
print("the table function, run directly:", rows("select count(*) from query_table('crm.learners')"))
try:
rows("select count(*) from read_parquet('data/platform/learners.parquet')")
except Exception as error:
print("the file, run directly:", str(error).split(":")[0])
try:
platform.connection.execute("set enable_external_access = true")
except Exception as error:
print("turning external access back on:", str(error).split(": ", 1)[1])
```
Output:
```text
qualify on its own: SELECT * FROM "crm"."learners" AS "l" JOIN "marketing"."audience" AS "a" ON "a"."learner_id" = "l"."learner_id"
a table missing from the catalog: denied, not a table in the catalog: marketing.audience AS a
a column the table doesn't have: denied, Column 'shoe_size' could not be resolved. Line: 1, Col: 16
the scan behind a view: denied, not a table in the catalog: scan_crm_learners
a Parquet file: denied, not a table in the catalog: READ_PARQUET('data/platform/learners.parquet')
a table function: denied, not a table in the catalog: QUERY_TABLE('crm.learners')
a statement that isn't a query: denied, only a single query is allowed
the scan, run directly: 5,000 rows
the table function, run directly: 5,000 rows
the file, run directly: Permission Error
turning external access back on: Cannot enable external access while database is running
```
`qualify` on its own returns the query with `SELECT *` intact and no error, so a layer that trusted it would ask the policy about `crm.learners` and pass `marketing.audience` through unchecked. The enforcement layer denies every table that isn't in the catalog before it asks the policy anything. A column the table doesn't have is denied by `qualify` itself. The other four reads show why that rule is broad. `lib.warehouse` registers each table's scan with DuckDB under a name of its own, such as `scan_crm_learners`. Run directly on the layer's connection, that scan returns 5,000 rows of raw learners. `read_parquet` reads a file, and `query_table` reads whatever table its string names, 5,000 rows when run directly. `SUMMARIZE` isn't a query the rewrite can analyse. The first three aren't catalog tables and the last isn't a query, so all four are denied. The layer's connection also has DuckDB's `enable_external_access` turned off, so a file read that got past the rewrite would still fail, with a `Permission Error` ([Securing DuckDB](https://duckdb.org/docs/2.0/operations_manual/securing_duckdb/overview)).
**Working rule:** Ask the policy about every table and column a query names, with `*` expanded against the catalog's own schema, and deny anything that schema doesn't hold: unknown tables, files, table functions and statements other than a single query. A denial costs a user one question to the platform team; a pass-through costs whatever the unchecked table holds.
## Allow only catalog tables, CTEs and subqueries {#allow-only-catalog-sources}
[Denying names the catalog doesn't hold](#find-what-a-query-reads) isn't enough on its own, because a query can read data through sources that aren't table names. A table function can sit under `LATERAL`, or in a subquery in the select list. Outside a CTE's scope, or before the CTE is defined, DuckDB resolves its name to whatever else the connection holds under that name. A check that inspects only table names, and accepts any name a CTE uses, passes both.
So the enforcement layer checks sources against an allowlist. `Enforcer.check_sources` walks the whole query and requires every `FROM` and `JOIN` source, at every level, to be a catalog table, a CTE in scope or a subquery, and a `LATERAL` to be over a subquery. Everything else is denied, including table functions wherever they appear, `VALUES`, `UNNEST` and joins in parentheses. A CTE is in scope in its own query, below it and in later CTEs, and a recursive one also in its `UNION`'s recursive branch. An allowlist fails closed on a kind of source nobody thought of.
A query through the layer must also change nothing. DuckDB's scalar `write_log` writes into the engine's query log. `duckdb_functions()` marks such functions: `has_side_effects` is "`true` if this function changes the database state" ([DuckDB, `duckdb_functions`](https://duckdb.org/docs/2.0/sql/meta/duckdb_table_functions)). At 2.0.0 it also marks functions that only vary, such as `random` and `uuid`. The layer denies every flagged function, and every macro that calls one, such as `pg_sleep`, reading the list from `duckdb_functions()` when it starts: a list sorted by hand into harmful and harmless is one an upgrade can outdate unnoticed, and a governed read seldom needs a random number.
`lib.enforce` keeps the cases as a regression test, `source_cases`, which [chapter 10's CI](#run-every-rule-as-one-job) runs, as it does [the order test](#gate-columns-on-row-filters) below:
Example `catalog-sources-only` (Python 3.12.15 · duckdb 2.0.0 · pyarrow 26.0.0 · pyiceberg 0.12.0 · regopy 1.5.2 · sqlglot 30.24.0), passing in Edition 2026.10.1:
```python
from lib.catalog import open_catalog
from lib.enforce import MUST_ALLOW, MUST_DENY, Enforcer, source_cases
platform = Enforcer(open_catalog(["crm.learners", "support.tickets"]))
# The allowlist's regression test, lib.enforce.source_cases, which chapter 10's CI runs too: queries that read data
# other than through a catalog table, a call to each function DuckDB marks as having side effects, and a query the
# layer must still allow. A case passes when the layer denies it exactly when it must, and its parse logs nothing.
cases = source_cases(platform)
for case in cases:
if case["case"] in MUST_DENY:
print(f"{case['case']}:", f"denied, {case['reason']}" if case["denied"] else "ALLOWED")
functions = [case for case in cases if case["must"] == "deny" and case["case"] not in MUST_DENY]
print("functions with side effects:", ", ".join(case["case"].removeprefix("the function ") for case in functions))
print("functions with side effects, denied:", f"{sum(case['denied'] for case in functions)} of {len(functions)}")
print("cases whose parse sqlglot logged, quoting the query:", f"{sum(case['logged'] for case in cases)} of {len(cases)}")
passing = [case for case in cases if case["denied"] == (case["must"] == "deny") and not case["logged"]]
print("cases that pass:", f"{len(passing)} of {len(cases)}")
# What two of the cases do on the layer's own connection, without the layer in front of it.
before_the_cte = platform.connection.sql(MUST_DENY["a CTE's name before the CTE"]).fetchall()[0][0]
print("a CTE's name before the CTE, run directly:", f"{before_the_cte:,} rows")
platform.connection.execute("CALL enable_logging('QueryLog')")
platform.connection.sql(MUST_DENY["a function that writes DuckDB's log"]).fetchall()
logged = [message for (message,) in platform.connection.sql("select message from duckdb_logs where type = 'QueryLog'").fetchall()]
print("the log function, run directly, query-log entries it leaves:", logged)
# The query the allowlist must let through, run through the layer.
for name, query in MUST_ALLOW.items():
print(f"{name}, allowed:", f"{platform.sql(query, 'agent-a05').fetchall()[0][0]:,} rows")
```
Output:
```text
a table function in FROM: denied, not a table in the catalog: QUERY_TABLE('crm.learners')
a table function in a scalar subquery: denied, not a table in the catalog: QUERY_TABLE('crm.learners')
a table function under LATERAL: denied, a LATERAL that isn't a subquery: LATERAL QUERY_TABLE('crm.learners')
a catalog-qualified name: denied, not a table in the catalog: memory.crm.learners
an information_schema view: denied, not a table in the catalog: information_schema.tables
a CTE's name outside its scope: denied, not a table in the catalog: other
a CTE's name before the CTE: denied, not a table in the catalog: scan_crm_learners
a function that writes DuckDB's log: denied, a function DuckDB marks as having side effects: write_log
a statement sqlglot keeps as text: denied, only a single query is allowed
functions with side effects: current_connection_id, current_query, current_query_id, current_transaction_id, currval, error, gen_random_uuid, nextval, pg_sleep, random, setseed, setval, sleep_ms, stats, uuid, uuidv4, uuidv7, vector_type, write_log
functions with side effects, denied: 19 of 19
cases whose parse sqlglot logged, quoting the query: 0 of 29
cases that pass: 29 of 29
a CTE's name before the CTE, run directly: 5,000 rows
the log function, run directly, query-log entries it leaves: ["select write_log('an entry', log_type := 'QueryLog')", 'an entry']
a CTE, a join and a scalar subquery, allowed: 61 rows
```
Each case is denied before any policy is asked, with a reason that names the source. The layer parses with sqlglot's warnings silenced, since sqlglot logs any statement it keeps as text, such as a `CALL`, values included ([sqlglot source, v30.24.0](https://github.com/tobymao/sqlglot/blob/v30.24.0/sqlglot/parser.py)), and every case checks that nothing was logged: 29 of 29 cases pass. Two of them, run on the layer's own connection, show what the denials prevent: the name used before its CTE reads the scan of the same name, 5,000 rows of raw learners, and `write_log` leaves `["select write_log('an entry', log_type := 'QueryLog')", 'an entry']`: its own statement, and an entry no query made. Every function with side effects is denied, 19 of 19. The last line checks that the allowlist doesn't deny too much: run as `agent-a05`, the EMEA support agent, a CTE, a join and a scalar subquery over catalog tables return 61 rows.
**Working rule:** Check every source a query reads against an allowlist of catalog tables, CTEs in scope and subqueries, and deny the rest, along with every function the engine marks as having side effects, read from its own list. Keep the cases as a regression test for every change to the layer and every engine upgrade, and add one whenever a new kind of source or function turns up.
## Masking column references leaks {#column-masks-leak}
With the columns known, the obvious enforcement rewrites them: wherever the query selects `email` and the decision masks it, select the mask instead. It needs nothing but the column list from [the first section](#find-what-a-query-reads), and it looks right on the queries people test with. It fails because a query can read a column without selecting it, and DuckDB can return a column without naming it.
On the platform, `analyst-1`, a data scientist, may read `crm.learners` with `email` and the other personal columns pseudonymized, and free text such as `notes` redacted. `Enforcer.mask_select_list` does the column rewrite, kept only to show this failure. The example prints its rewrite of a plain read, then runs five reads aimed at learner 2's email, with any stored email replaced by a placeholder, since the book's examples never print one. Before you run it, predict how many of the five the column rewrite stops:
Example `select-list-masks-leak` (Python 3.12.15 · duckdb 2.0.0 · pyarrow 26.0.0 · pyiceberg 0.12.0 · regopy 1.5.2 · sqlglot 30.24.0), passing in Edition 2026.10.1:
```python
import re
import duckdb
from lib.catalog import open_catalog
from lib.enforce import EMAIL, Enforcer
platform = Enforcer(open_catalog(["crm.learners"]))
# analyst-1, a data scientist, may read crm.learners with `email` pseudonymized. The attacks aim at learner 2.
email = platform.connection.sql("select email from crm.learners where learner_id = 2").fetchall()[0][0]
print("the column rewrite:", platform.mask_select_list("select learner_id, email from crm.learners", "analyst-1"))
attacks = {
"a WHERE on the raw column": f"select learner_id from crm.learners where email = '{email}'",
"the whole row as a struct": "select struct_extract(l, 'email') from crm.learners as l where learner_id = 2",
"COLUMNS(...)": "select columns('^email$') from crm.learners where learner_id = 2",
"to_json": "select to_json(l) ->> 'email' from crm.learners as l where learner_id = 2",
"a cast error": "select learner_id from crm.learners where cast(email as integer) = 0",
}
for name, sql in attacks.items():
try:
answer = str(platform.run(platform.mask_select_list(sql, "analyst-1")).fetchall())
except duckdb.Error as error:
answer = str(error).splitlines()[0]
print(f"{name}:", re.sub(EMAIL, "", answer)) # the book never prints a stored email
```
Output:
```text
the column rewrite: SELECT "learners"."learner_id" AS "learner_id", SHA256($pseudonym_key || CAST("learners"."email" AS TEXT)) AS "email" FROM "crm"."learners" AS "learners"
a WHERE on the raw column: [(2,)]
the whole row as a struct: [('',)]
COLUMNS(...): [('',)]
to_json: [('',)]
a cast error: Conversion Error: Could not convert string '' to INT32 when casting from source column email
```
None. The rewrite of the plain read is correct: `email` comes back as a hash. The five reads get past it in three ways:
- **A `WHERE` on the raw column** never passes through the select list, so it compares the stored email. The analyst guesses an email and gets back learner 2's ID, `[(2,)]`, which links that person to every row they can join on `learner_id`.
- **The whole row as a struct, `COLUMNS(...)` and `to_json`** return the stored email, `[('',)]`, without any column reference named `email` for the rewrite to find: DuckDB expands the row, or matches column names against a pattern, itself.
- **A cast error** prints a stored email in its message. DuckDB's cast errors include the value, and PostgreSQL's documentation warns that "most casting functions include their input values in the error messages they produce" ([PostgreSQL 18, Rules and Privileges](https://www.postgresql.org/docs/18/rules-privileges.html)).
Masking the column wherever the query names it, in the `WHERE` clause too, would stop the first and the last read. It can't stop the middle three, because they never name the column.
**Working rule:** Don't enforce masks by rewriting the column references in a query. However many reference shapes you cover, the engine has more ways to read a column than the rewrite has to recognize one. Mask where the data comes from instead, so every read sees the masked value, which is what the [next section](#replace-each-table) does.
## Replace each table with a filtered, masked subquery {#replace-each-table}
[Rewriting column references](#column-masks-leak) leaks because the raw table stays in the query. The fix replaces the table. `Enforcer.rewrite` replaces each catalog table in the query, under the query's own alias for it, with a subquery built from the user's decision:
- every column of the table, in the table's order, so `*`, a struct or `COLUMNS(...)` sees the shape it expects;
- each column the query reads, as stored or masked as the decision says, inside a `CASE` on the row filters that [a later section](#gate-columns-on-row-filters) explains;
- each column it doesn't read, as a typed `NULL`, since the policy wasn't asked about it;
- a `WHERE` clause with every row filter in the decision.
Everything the user's query does, from its `WHERE` to its struct reads, then happens on that subquery. It's the model Trino's OPA plugin follows, where a row filter "essentially behaves like an additional `WHERE` clause", as the [section on engines](#engine-enforcement) shows.
For the platform, take `instructor-17`, who reads the low ratings among their courses' reviews. The policy filters reviews to the instructor's own courses and shows `body` with contact details removed:
Example `table-rewrite` (Python 3.12.15 · duckdb 2.0.0 · pyarrow 26.0.0 · pyiceberg 0.12.0 · regopy 1.5.2 · sqlglot 30.24.0), passing in Edition 2026.10.1:
```python
import sqlglot
from lib.catalog import open_catalog
from lib.enforce import Enforcer
platform = Enforcer(open_catalog(["learning.reviews"]))
# instructor-17 reads the low ratings of their courses' reviews. The policy filters reviews to their six courses and
# shows `body` with contact details removed.
query = "select course_id, rating, body from learning.reviews where rating <= 2"
rewritten = platform.rewrite(query, "instructor-17")
print(sqlglot.transpile(rewritten, read="duckdb", write="duckdb", pretty=True)[0])
print("reviews returned:", len(platform.run(rewritten).fetchall()))
print("of them holding an email:", platform.run(f"select count(*) from ({rewritten}) where body like '%@%'").fetchall()[0][0])
```
Output:
```text
SELECT
"reviews"."course_id" AS "course_id",
"reviews"."rating" AS "rating",
"reviews"."body" AS "body"
FROM (
SELECT
CAST(NULL AS BIGINT) AS review_id,
CAST(NULL AS BIGINT) AS learner_id,
CASE WHEN source.course_id IN (2, 75, 80, 87, 103, 134) THEN source.course_id END AS course_id,
CASE WHEN source.course_id IN (2, 75, 80, 87, 103, 134) THEN source.rating END AS rating,
CASE
WHEN source.course_id IN (2, 75, 80, 87, 103, 134)
THEN REGEXP_REPLACE(
REGEXP_REPLACE(source.body, '[A-Za-z0-9._%+-]+@[A-Za-z0-9-]+(\.[A-Za-z0-9-]+)+', '[email]', 'g'),
'\+?\(?[0-9][0-9 ().-]{6,}[0-9]',
'[phone]',
'g'
)
END AS body,
CAST(NULL AS TIMESTAMPTZ) AS posted_at
FROM learning.reviews AS source
WHERE
source.course_id IN (2, 75, 80, 87, 103, 134)
) AS "reviews"
WHERE
"reviews"."rating" <= 2
reviews returned: 9
of them holding an email: 0
```
The instructor's query stays on the outside, qualified but otherwise unchanged. Inside, `course_id` and `rating` come through as stored and `body` as the `redact_contacts` mask, each in a `CASE` on the row filter, a literal list of the instructor's courses, and the columns the query doesn't read are typed `NULL`s. The query returns 9 reviews, and 0 of them hold an email.
The five reads that defeated the column rewrite, run by `analyst-1` through the table rewrite, and through `lib.enforce.answer`, the function a service calls at its boundary:
Example `table-rewrite-holds` (Python 3.12.15 · duckdb 2.0.0 · pyarrow 26.0.0 · pyiceberg 0.12.0 · regopy 1.5.2 · sqlglot 30.24.0), passing in Edition 2026.10.1:
```python
import logging
import re
from lib.catalog import open_catalog
from lib.enforce import EMAIL, Enforcer, QueryFailed, answer
platform = Enforcer(open_catalog(["crm.learners"]))
operator_log = [] # the platform team's log, kept in a list here so the example can print it
logging.getLogger("lib.enforce").addHandler(handler := logging.Handler())
handler.emit = lambda record: operator_log.append(record.getMessage().splitlines()[0])
# The five attacks on the column rewrite, run by analyst-1 again: through the table rewrite this time, and through
# `answer`, the function a service calls at its boundary.
email = platform.connection.sql("select email from crm.learners where learner_id = 2").fetchall()[0][0]
attacks = {
"a WHERE on the raw column": f"select learner_id from crm.learners where email = '{email}'",
"the whole row as a struct": "select struct_extract(l, 'email') from crm.learners as l where learner_id = 2",
"COLUMNS(...)": "select columns('^email$') from crm.learners where learner_id = 2",
"to_json": "select to_json(l) ->> 'email' from crm.learners as l where learner_id = 2",
"a cast error": "select learner_id from crm.learners where cast(email as integer) = 0",
}
shorten = lambda text: re.sub(r"\b([0-9a-f]{8})[0-9a-f]{56}\b", r"\1…", re.sub(EMAIL, "", text))
for name, sql in attacks.items():
try:
got = str(answer(platform, sql, "analyst-1"))
except QueryFailed as error:
got = f"{type(error).__name__}, {error}"
print(f"{name}:", shorten(got))
for line in operator_log:
print("in the operator's log:", shorten(line)) # a pseudonym, 64 hex digits, shortened
# Why `answer` returns rows: the relation Enforcer.sql returns is a handle on the layer's connection, and runs any SQL.
relation = platform.sql("select learner_id from crm.learners", "analyst-1")
print("the relation, running SQL of its own:", shorten(str(relation.query("r", "select email from crm.learners where learner_id = 2").fetchall())))
print("what answer returns:", type(answer(platform, "select learner_id from crm.learners", "analyst-1")).__name__)
```
Output:
```text
a WHERE on the raw column: []
the whole row as a struct: [(None,)]
COLUMNS(...): [(None,)]
to_json: [(None,)]
a cast error: QueryFailed, the query failed: the platform team can see why
in the operator's log: a query by analyst-1 failed in the engine: Conversion Error: Could not convert string 'e4f62f1a…' to INT32 when casting from source column email
the relation, running SQL of its own: [('',)]
what answer returns: list
```
The `WHERE` finds no one, `[]`, because inside the subquery `email` is already the pseudonym. The struct, `COLUMNS(...)` and `to_json` reads return `NULL`, not the pseudonym, and that is the typed-`NULL` rule working: those queries name no column of `crm.learners` but `learner_id`, so the policy was asked about no other column, and the subquery gives the rest no value. A read the rewrite can't see fails closed. DuckDB's cast error now holds a pseudonym instead of an email, and the analyst doesn't see even that: `answer` fetches the rows inside a `try`, turns any engine error into `QueryFailed`, and writes DuckDB's message to the platform team's log, the example's last line. The analyst gets `QueryFailed, the query failed: the platform team can see why`. And `answer` returns rows because a DuckDB relation runs any SQL on its connection: the analyst's relation, asked for an email, returns `[('',)]`.
Generating the subquery per query, rather than writing a view per role, is what keeps it in step with the policy. The platform's decisions depend on region, courses, consent and purpose, and a hand-made view for every combination is the "role explosion" NIST describes, where conditions become "numerous roles that are ad hoc and limited in membership" ([NIST SP 800-162](https://doi.org/10.6028/NIST.SP.800-162)). The views would also drift from the policy the first time someone edited one and not the other.
**Working rule:** Replace every protected table with a subquery generated from the decision, under the query's own alias, and leave the user's query otherwise untouched. Give columns the query doesn't read a typed `NULL`. If your users need whole-row reads to show values, ask the policy about every column of each table instead: the masks then cover all of them, at the cost of requests, and audit records, that name columns the query never used.
## Write each mask as SQL in one place {#write-masks-as-sql}
The policy names a mask; the enforcement layer decides what SQL it becomes. Keeping that mapping in one place, the `MASKS` table in `lib.enforce` and the typed `NULL` beside it, means a fix to a mask reaches every table that uses it, and the policy never carries SQL, which would be code arriving as data. The [table rewrite](#replace-each-table) calls `lib.enforce.masked` for every masked column:
Example `masks-as-sql` (Python 3.12.15 · duckdb 2.0.0 · pyarrow 26.0.0 · pyiceberg 0.12.0 · regopy 1.5.2 · sqlglot 30.24.0), passing in Edition 2026.10.1:
```python
import duckdb
from sqlglot import exp
from lib.enforce import masked
# The four masks the policy names, as the SQL the enforcement layer writes for them.
for kind, column, type_ in [("redact", "date_of_birth", "DATE"), ("last4", "card_number", "VARCHAR"),
("pseudonymize", "email", "VARCHAR"), ("redact_contacts", "body", "VARCHAR")]:
print(f"{kind}:", masked(kind, exp.column(column), type_).sql("duckdb"))
# The pseudonymize mask hashes the key followed by the value, because DuckDB has no HMAC function to call.
hmac = duckdb.sql("select count(*) from duckdb_functions() where function_name ilike '%hmac%'").fetchall()[0][0]
print("DuckDB functions named like HMAC:", hmac)
```
Output:
```text
redact: CAST(NULL AS DATE)
last4: RIGHT(REGEXP_REPLACE(card_number, '[^0-9]', '', 'g'), 4)
pseudonymize: SHA256($pseudonym_key || CAST(email AS TEXT))
redact_contacts: REGEXP_REPLACE(REGEXP_REPLACE(body, '[A-Za-z0-9._%+-]+@[A-Za-z0-9-]+(\.[A-Za-z0-9-]+)+', '[email]', 'g'), '\+?\(?[0-9][0-9 ().-]{6,}[0-9]', '[phone]', 'g')
DuckDB functions named like HMAC: 0
```
Each mask does one job:
- **`redact`** is a `NULL` cast to the column's type, so the column keeps its type for anything the query does with it, such as a union or a date function.
- **`last4`** keeps the last four digits of a card number, after removing anything that isn't a digit.
- **`pseudonymize`** is a keyed hash, SHA-256 over the key followed by the value, because DuckDB 2.0.0 has no HMAC function: `duckdb_functions()` lists 0 named like it. Where your engine has one, prefer HMAC ([RFC 2104](https://www.rfc-editor.org/rfc/rfc2104)).
- **`redact_contacts`** replaces emails and phone numbers in free text with `[email]` and `[phone]`.
The key is what makes a pseudonym a pseudonym. The next example plays someone who holds a list of the platform's emails and hashes each one to look for it among the pseudonyms that `analyst-1`, the platform's data scientist, sees in `crm.learners`:
Example `pseudonym-key` (Python 3.12.15 · duckdb 2.0.0 · pyarrow 26.0.0 · pyiceberg 0.12.0 · regopy 1.5.2 · sqlglot 30.24.0), passing in Edition 2026.10.1:
```python
from lib.catalog import open_catalog
from lib.enforce import Enforcer
# The pseudonyms analyst-1, the platform's data scientist, sees in crm.learners. Someone holding a list of the
# platform's emails hashes each one and looks for it among them: without the key the hashes match nothing, and with
# the key they match every learner.
platform = Enforcer(open_catalog(["crm.learners"]), key="example-key-not-a-secret")
seen = platform.sql("select learner_id, email from crm.learners", "analyst-1").to_arrow_table()
platform.connection.register("seen", seen)
for name, guess in [("without the key", "sha256(email)"), ("with the key", "sha256($pseudonym_key || email)")]:
matched = platform.run(f"select count(*) from crm.learners where {guess} in (select email from seen)").fetchall()[0][0]
print(f"pseudonyms matched from a list of emails, {name}: {matched:,} of {seen.num_rows:,}")
# A new key gives every learner a new pseudonym, so data pseudonymized under the old key no longer joins.
platform.key = "example-key-rotated"
rotated = platform.sql("select learner_id, email from crm.learners", "analyst-1").to_arrow_table()
changed = sum(a != b for a, b in zip(seen["email"].to_pylist(), rotated["email"].to_pylist()))
print("pseudonyms that change when the key is rotated:", f"{changed:,} of {seen.num_rows:,}")
```
Output:
```text
pseudonyms matched from a list of emails, without the key: 0 of 5,000
pseudonyms matched from a list of emails, with the key: 5,000 of 5,000
pseudonyms that change when the key is rotated: 5,000 of 5,000
```
Without the key, the hashes match 0 of 5,000. With the key, they match 5,000 of 5,000, so whoever holds the key and a list of emails can name every pseudonym on it. That's why the key belongs to the enforcement layer alone. It isn't in the policy, which is code read in pull requests, or in the query, where a user who chose the key could compute pseudonyms of guessed emails. The layer passes it to DuckDB as a query parameter, so it never appears in the query's text, which DuckDB's query log records, as [chapter 8](#keep-values-out-of-the-record) shows. That's also why the layer denies any user query that carries a parameter: one named `$pseudonym_key` would receive the key. In production, keep the key in a secret manager that only the enforcement layer can read. Rotate it when it may have leaked, and plan for what rotation does: 5,000 of 5,000 pseudonyms change, so data pseudonymized under the old key no longer joins with data under the new one.
`redact_contacts` is two regular expressions, and it's worth measuring what they remove. The platform's tickets and reviews come with every piece of personal data the generator placed in them, labelled by construction, which [chapter 3](#finding-personal-data) uses as its ground truth:
Example `redact-contacts-recall` (Python 3.12.15 · duckdb 2.0.0 · pyarrow 26.0.0 · pyiceberg 0.12.0 · regopy 1.5.2 · sqlglot 30.24.0), passing in Edition 2026.10.1:
```python
import duckdb
from sqlglot import exp
from lib.enforce import masked
from lib.platform import load
# The redact_contacts mask, run over every ticket and review, against the personal data the generator placed in them
# (pii_spans, labelled by construction).
con = duckdb.connect()
for name in ["support.tickets", "learning.reviews", "pii_spans"]:
con.register(name.split(".")[-1], load(name))
mask = masked("redact_contacts", exp.column("body"), "VARCHAR").sql("duckdb")
counts = con.sql(f"""
with masked as (select 'tickets' as source, ticket_id as row_id, {mask} as body from tickets
union all select 'reviews', review_id, {mask} from reviews)
select source, entity, count(*), count(*) filter (where contains(masked.body, pii_spans.text))
from pii_spans join masked using (source, row_id) group by all order by all""").fetchall()
for source, entity, placed, left in counts:
print(f"{source}, {entity}, left in the text: {left:,} of {placed:,}")
# The phone pattern also takes card numbers, IBANs and dates, which are digits too.
replaced = con.sql(f"select sum(len(regexp_extract_all({mask}, '\\[phone\\]'))) from tickets").fetchall()[0][0]
print("[phone] replacements in tickets:", f"{replaced:,}")
```
Output:
```text
reviews, EMAIL_ADDRESS, left in the text: 0 of 120
reviews, PERSON, left in the text: 798 of 798
tickets, CREDIT_CARD, left in the text: 0 of 213
tickets, DATE_TIME, left in the text: 0 of 196
tickets, EMAIL_ADDRESS, left in the text: 0 of 423
tickets, IBAN_CODE, left in the text: 0 of 236
tickets, LOCATION, left in the text: 210 of 210
tickets, PERSON, left in the text: 1,657 of 1,657
tickets, PHONE_NUMBER, left in the text: 0 of 439
[phone] replacements in tickets: 1,520
```
In the reviews, which instructors and the assistant read through this mask, no email is left, 0 of 120, and every name is, 798 of 798. In the tickets, no phone number is left, 0 of 439, but the phone pattern made 1,520 replacements, because card numbers, IBANs and dates are digits too: 0 of 213 card numbers are left, 0 of 236 IBANs and 0 of 196 dates. Names and street addresses stay: the mask removes contact details, and it doesn't make text anonymous.
**Working rule:** Keep every mask's SQL in one table in the enforcement layer, and let the policy name masks, never write them. Key every pseudonym with a secret only the layer holds, passed as a parameter. Measure each text mask against labelled data before a policy relies on it, and name masks for what they remove, so nobody reads `redact_contacts` as "anonymized".
## Build row filters from the decision's data {#build-row-filters-from-data}
Masks change how a value is shown; row filters decide which rows a user gets. They arrive in a decision as data, in three shapes: a column equal to a value, a column in a list of values, and a consent check on the learner a row belongs to. `lib.enforce.row_filter` turns each into a SQL condition, with the values written as SQL literals by sqlglot, never pasted into SQL text. It denies the query for a shape it doesn't know or a column the table doesn't have, and the layer checks a consent filter's column against `crm.learners` before it builds the check. [Chapter 5](#return-the-decision-as-one-document) made the policy return filters as data for exactly this reason: nothing a request carries can reach a query as code.
The consent shape is the one that needs care. The course assistant, `assistant-1`, may read reviews only by learners who agreed to their text training the platform's AI, and that consent lives in `crm.learners`, which the assistant may not read. The layer turns it into a semi-join, a test that the review's `learner_id` is among the learners who consented, and runs it with the layer's own access to `crm.learners`:
Example `row-filters-from-data` (Python 3.12.15 · duckdb 2.0.0 · pyarrow 26.0.0 · pyiceberg 0.12.0 · regopy 1.5.2 · sqlglot 30.24.0), passing in Edition 2026.10.1:
```python
from lib.catalog import open_catalog
from lib.enforce import Enforcer, row_filter
platform = Enforcer(open_catalog(["crm.learners", "learning.reviews", "support.tickets"]))
# Each row filter in a decision, as the condition the enforcement layer writes for it.
for user, table in [("agent-a05", "support.tickets"), ("instructor-17", "learning.reviews"), ("assistant-1", "learning.reviews")]:
decision = platform.decide(user, {table: set()})[table]
for rule in decision["row_filters"]:
print(f"{user}, {table}: {row_filter(rule, platform.columns(table)).sql('duckdb')}")
# A filter's values come from data, so a value that looks like SQL stays a value.
odd = {"column": "support_region", "equals": "emea' or '1' = '1"}
print("a region that looks like SQL:", row_filter(odd, platform.columns("support.tickets")).sql("duckdb"))
# The assistant's consent filter reads crm.learners, which the assistant itself may not read.
total = platform.connection.sql("select count(*) from learning.reviews").fetchall()[0][0]
read = platform.sql("select count(*) from learning.reviews", "assistant-1").fetchall()[0][0]
print("reviews the assistant reads:", f"{read:,} of {total:,}")
by_consent = platform.connection.sql("""select coalesce(l.ai_training_consent::varchar, 'never asked'), count(*)
from learning.reviews r join crm.learners l using (learner_id) group by all order by all""").fetchall()
for consent, reviews in by_consent:
print(f"reviews whose author's AI-training consent is {consent}: {reviews:,}")
try:
platform.sql("select count(*) from crm.learners", "assistant-1")
except Exception as reason:
print("the assistant reading crm.learners:", reason)
```
Output:
```text
agent-a05, support.tickets: source.support_region = 'emea'
instructor-17, learning.reviews: source.course_id IN (2, 75, 80, 87, 103, 134)
assistant-1, learning.reviews: source.learner_id IN (SELECT consent.learner_id FROM crm.learners AS consent WHERE consent.ai_training_consent IS TRUE)
a region that looks like SQL: source.support_region = 'emea'' or ''1'' = ''1'
reviews the assistant reads: 436 of 3,000
reviews whose author's AI-training consent is false: 955
reviews whose author's AI-training consent is never asked: 1,609
reviews whose author's AI-training consent is true: 436
the assistant reading crm.learners: assistant-1 may not read crm.learners: no group of this user may read this table
```
The agent's region and the instructor's courses become an equality and an `IN` list. A region value written to look like SQL stays a string, its quotes doubled. The assistant reads 436 of 3,000 reviews. The consent filter tests `IS TRUE`, so the authors of 1,609 reviews, who signed up before the platform asked the question and have no answer recorded, count as not consenting, along with those who said no. And the assistant asking for `crm.learners` itself is denied, with the policy's reason: the semi-join's read of that table belongs to the enforcement layer, not to the user.
**Working rule:** Build every row filter from the decision's values with a SQL builder, and deny any filter shape or column the layer doesn't recognize. Treat an unknown consent as no consent. When a filter needs a table the user may not read, read it with the layer's own access inside the subquery, and never by widening the user's access.
## Gate every column on the row filters {#gate-columns-on-row-filters}
[The row filters](#build-row-filters-from-data) sit in the subquery's `WHERE`, and the user's query wraps the subquery, so it's tempting to assume DuckDB applies the filters first. SQL promises no order, and an optimizer moves conditions wherever they're cheapest. A user's condition that can fail, such as a cast, can then run on rows the filters drop, and a generic error doesn't help: that the query fails says something about a row the user may not see. PostgreSQL's documentation names the leak: a function "that might throw an error depending on the values received as arguments ... could provide significant information about the unseen rows if applied before the security view's row filters". In PostgreSQL, a view marked `security_barrier` keeps "maliciously-chosen functions and operators from being passed values from rows until after the view has done its work", and only functions marked `LEAKPROOF` may still run first ([PostgreSQL 18, Rules and Privileges](https://www.postgresql.org/docs/18/rules-privileges.html)). DuckDB has neither.
So the enforcement layer makes the order irrelevant. Every column the query reads leaves the subquery as `CASE WHEN THEN END`, as in [the instructor's rewrite](#replace-each-table). The user's conditions then depend on the filters' result: whatever DuckDB evaluates first, on a row the filters drop they see `NULL` in every column.
The book's test, `lib.enforce.order_cases`, checks that for each filter shape the layer writes: the support agent's equality, the instructor's `IN` list and the course assistant's consent semi-join. For each, it copies the filtered table with a `body` that holds digits in the rows the filter keeps and text in the rows it drops. A query that casts `body` to a number then fails only if the cast sees a dropped row's value:
Example `row-filter-order` (Python 3.12.15 · duckdb 2.0.0 · pyarrow 26.0.0 · pyiceberg 0.12.0 · regopy 1.5.2 · sqlglot 30.24.0), passing in Edition 2026.10.1:
```python
from lib.enforce import order_cases
# The evaluation-order regression test, lib.enforce.order_cases, which chapter 10's CI runs on every DuckDB upgrade.
# For each shape of row filter the layer writes, it takes a real user's filter and a copy of the filtered table whose
# `body` holds digits in the rows the filter keeps and text in the rows it drops. A query that casts `body` to a number
# fails only if the cast sees a dropped row's value. It runs over the filter in a WHERE alone, and over the layer's
# subquery.
cases = order_cases()
for case in cases:
for form, safe in [("with the filter in WHERE only", case["safe_with_where_only"]),
("as the layer writes it", case["safe_as_written"])]:
print(f"{case['shape']}, {form}: the cast {'never saw' if safe else 'saw'} a dropped row's value")
print("filter shapes the layer's subquery keeps safe:", f"{sum(case['safe_as_written'] for case in cases)} of {len(cases)}")
```
Output:
```text
equality, with the filter in WHERE only: the cast never saw a dropped row's value
equality, as the layer writes it: the cast never saw a dropped row's value
IN list, with the filter in WHERE only: the cast never saw a dropped row's value
IN list, as the layer writes it: the cast never saw a dropped row's value
consent semi-join, with the filter in WHERE only: the cast saw a dropped row's value
consent semi-join, as the layer writes it: the cast never saw a dropped row's value
filter shapes the layer's subquery keeps safe: 3 of 3
```
With the filter in the `WHERE` alone, the equality and the `IN` list were safe, and the consent semi-join wasn't: DuckDB ran the cast on reviews whose authors didn't consent. With every column gated, all three are safe, 3 of 3. The first two were safe by the optimizer's choice at 2.0.0, not by any promise, so the test runs on every upgrade. The gate costs evaluating the filters again in each column the query reads.
**Working rule:** Never rely on the engine's evaluation order to keep a user's conditions off rows a filter drops. Make every value a user's query can read depend on the row filters, and test each filter shape on every engine upgrade. Where an engine has a barrier of its own, such as PostgreSQL's `security_barrier` views, use it as well.
## Test who sees what by running queries {#test-who-sees-what}
[Chapter 5](#check-who-sees-what) checks the policy with a who-sees-what table, `lib.policy.WHO_SEES_WHAT`: a list of requests, each with the decision a reviewer expects for it. That tests the decisions, not what users get, which also depends on the layer's rewrite, its masks' SQL and the data. `lib.enforce.observe` runs each case's columns through the layer as the case's user, and compares what comes back with what's stored: how many rows, and which columns differ. A case agrees with its decision when the user is denied exactly when the case says so, gets fewer rows than stored exactly when the case expects a row filter, and gets changed values in exactly the columns the case expects masked:
Example `who-sees-what-in-queries` (Python 3.12.15 · duckdb 2.0.0 · pyarrow 26.0.0 · pyiceberg 0.12.0 · regopy 1.5.2 · sqlglot 30.24.0), passing in Edition 2026.10.1:
```python
from lib import pipelines, policy
from lib.catalog import open_catalog
from lib.enforce import Enforcer, agrees, observe
catalog = open_catalog(["crm.learners", "learning.reviews", "billing.payments", "learning.enrollments", "learning.courses"])
pipelines.run(catalog, "marketing_audience") # a copy of learners' names and emails, with no metadata yet
platform = Enforcer(catalog)
# Chapter 5's who-sees-what table, run as queries: what each user gets, against what's stored. A column is "changed"
# when at least one value the user gets differs from the stored one.
agreeing = 0
for case in policy.WHO_SEES_WHAT:
got = observe(platform, case.user, case.table, case.columns, case.purpose)
who = case.user + (f" for {case.purpose}" if case.purpose else "")
if got is None:
print(f"{who}, {case.table}: denied")
else:
print(f"{who}, {case.table}: {got['rows']:,} of {got['stored']:,} rows, changed: {', '.join(got['changed']) or 'none'}")
agreeing += agrees(case, got)
print("cases where what the user gets agrees with the decision:", f"{agreeing} of {len(policy.WHO_SEES_WHAT)}")
# marketing.audience is a table marketing may read, but nobody has classified its columns, so the policy treats each
# one as sensitive: marketing gets every row, and no value in any column.
rows = platform.sql("select * from marketing.audience", "marketing-1").fetchall()
shown = sum(any(value is not None for value in column) for column in zip(*rows))
print("marketing-1, marketing.audience, which has no metadata, rows:", f"{len(rows):,}")
print("marketing-1, marketing.audience, which has no metadata, columns with a value:", f"{shown} of {len(rows[0])}")
```
Output:
```text
agent-a05, crm.learners: 1,581 of 5,000 rows, changed: none
agent-a05, billing.payments: denied
instructor-17, learning.reviews: 69 of 3,000 rows, changed: body
instructor-17, crm.learners: denied
marketing-1, crm.learners: 1,861 of 5,000 rows, changed: phone
finance-1, billing.payments: 17,562 of 17,562 rows, changed: card_number
analyst-1, billing.payments: 17,562 of 17,562 rows, changed: card_number
analyst-1, crm.learners: 5,000 of 5,000 rows, changed: email, notes
analyst-1, learning.reviews: 3,000 of 3,000 rows, changed: body
analyst-1 for ai-training, learning.reviews: 436 of 3,000 rows, changed: body
assistant-1, learning.reviews: 436 of 3,000 rows, changed: body
someone-else, crm.learners: denied
cases where what the user gets agrees with the decision: 12 of 12
marketing-1, marketing.audience, which has no metadata, rows: 1,829
marketing-1, marketing.audience, which has no metadata, columns with a value: 0 of 5
```
What users get agrees with the decision in 12 of 12 cases. In the output, the EMEA support agent gets their region's learners as stored, marketing the consenting, active learners with `phone` changed, and the analyst every learner with `email` and `notes` changed. The last two lines cover `marketing.audience`, which a pipeline writes and nobody has classified yet. Marketing may read it, and gets 1,829 rows with 0 of 5 columns holding a value, because the policy treats an unclassified column as sensitive.
The test earns its place on a change. Suppose a pull request adds `analyst-1` to the support group for EMEA, so they can help with tickets. It's a two-line change to `governance/users.toml`, a group and the region support's row filter needs, and the example below makes it in memory, then runs the same cases:
Example `group-change-caught` (Python 3.12.15 · duckdb 2.0.0 · pyarrow 26.0.0 · pyiceberg 0.12.0 · regopy 1.5.2 · sqlglot 30.24.0), passing in Edition 2026.10.1:
```python
from lib import policy
from lib.catalog import open_catalog
from lib.enforce import Enforcer, agrees, observe
# A pull request changes one user in governance/users.toml, in two lines: analyst-1 joins support, with the region
# support's row filter needs, EMEA, to help with tickets.
data = policy.data_document()
data["platform"]["users"]["analyst-1"] = {"groups": ["data-science", "support"], "region": "emea"}
platform = Enforcer(open_catalog(["crm.learners", "learning.reviews", "billing.payments"]),
rego=policy.interpreter(data=data))
# The who-sees-what cases, run as queries with the change in place. The cases are unchanged: they say what each
# user should get, and the change was meant to touch support tickets only.
failing = 0
for case in policy.WHO_SEES_WHAT:
got = observe(platform, case.user, case.table, case.columns, case.purpose)
if not agrees(case, got):
failing += 1
who = case.user + (f" for {case.purpose}" if case.purpose else "")
print(f"fails, {who}, {case.table}, rows: {got['rows']:,} of {got['stored']:,}")
print(f"fails, {who}, {case.table}, columns changed: {', '.join(got['changed']) or 'none'}, "
f"expected: {', '.join(case.masked)}")
print("cases that fail:", f"{failing} of {len(policy.WHO_SEES_WHAT)}")
# Chapter 5's table, which checks the decisions themselves, on the same change.
decisions = [policy.decide(policy.request(c.user, c.table, c.columns, c.purpose), platform.rego) for c in policy.WHO_SEES_WHAT]
print("cases that fail as decisions:", f"{sum(not policy.matches(c, d) for c, d in zip(policy.WHO_SEES_WHAT, decisions))} of {len(decisions)}")
```
Output:
```text
fails, analyst-1, crm.learners, rows: 1,581 of 5,000
fails, analyst-1, crm.learners, columns changed: none, expected: email, notes
fails, analyst-1, learning.reviews, rows: 3,000 of 3,000
fails, analyst-1, learning.reviews, columns changed: none, expected: body
fails, analyst-1 for ai-training, learning.reviews, rows: 436 of 3,000
fails, analyst-1 for ai-training, learning.reviews, columns changed: none, expected: body
cases that fail: 3 of 12
cases that fail as decisions: 3 of 12
```
The change fails 3 of 12 cases, and none of them is about tickets. On `crm.learners`, the analyst now gets 1,581 of 5,000 rows with no column changed: support's region filter applies to them, and the policy's personal-column masks apply only to users outside support, so every email and note comes through as stored. Review text, which a data scientist otherwise can't read at all, or reads with contacts removed when the purpose is AI training, comes through whole. [Chapter 5's decision-level table](#check-who-sees-what) fails on the same cases, 3 of 12; this one shows what the analyst would actually get.
**Working rule:** Run the who-sees-what cases as queries through the enforcement layer on every change to the policies, the users file, the metadata or the layer itself, and fail the build when any case disagrees. Fix a failing case by changing the policy or the users file, and change a case only when the people who own the data agree that what the user should get has changed.
## Hold back small groups in reports {#hold-back-small-groups}
Row filters and masks decide which rows and values a user sees. Aggregates are a second channel: a count or an average over a few people describes those people. The usual defence is a **minimum group size**, below which a reported group is held back. The number is a rule of thumb, not a measurement. One public data-use agreement from the US Centers for Medicare & Medicaid Services sets 11, and adds that "no use of percentages or other mathematical formulas may be used if they result in the display of a cell less than 11" ([CMS-R-0235L](https://www.cms.gov/medicare/cms-forms/cms-forms/downloads/cms-r-0235l.pdf)). The platform uses the same minimum, 11, and keeps it in `governance/enforcement.toml`, where a change is reviewed like a policy change, and `lib.enforce.minimum_group_size` reads it.
The platform sends each instructor a daily course report: learners and average progress per course and learner country. The report job reads learners' countries, which instructors may not read, and instructors may read their own courses' enrollments with each one's progress. Put those together:
Example `small-groups-difference` (Python 3.12.15 · duckdb 2.0.0 · pyarrow 26.0.0 · pyiceberg 0.12.0 · regopy 1.5.2 · sqlglot 30.24.0), passing in Edition 2026.10.1:
```python
from lib.catalog import open_catalog
from lib.enforce import Enforcer, minimum_group_size
platform = Enforcer(open_catalog(["crm.learners", "learning.courses", "learning.enrollments"]))
minimum = minimum_group_size()
print("minimum group size:", minimum)
count = lambda sql: platform.connection.sql(f"select count(*) from ({sql})").fetchall()[0][0]
# The platform's daily course report for instructor-17: learners and average progress, per course and learner country.
# The report job reads learners' countries, which instructors may not read.
platform.connection.execute("""create view mine as
select e.course_id, e.progress_pct, e.enrolled_at::date as day, l.country
from learning.enrollments e join crm.learners l using (learner_id)
where e.course_id in (select course_id from learning.courses where instructor_id = 17)""")
platform.connection.execute("""create view report as
select course_id, country, count(*) as learners, avg(progress_pct) as progress from mine group by all""")
print("cells in the report:", count("from report"))
print("cells under the minimum:", count(f"from report where learners < {minimum}"))
# instructor-17 may read their courses' enrollments, with each one's progress. A cell of one learner, matched on its
# progress, names that learner's country.
seen = platform.sql("select course_id, progress_pct from learning.enrollments", "instructor-17").to_arrow_table()
platform.connection.register("seen", seen)
print("cells of one learner:", count("from report where learners = 1"))
print("learners a cell of one names:", count("""from report as cell where learners = 1 and (select count(*) from seen
where seen.course_id = cell.course_id and seen.progress_pct = cell.progress) = 1"""))
print("cells shown with the minimum:", count(f"from report where learners >= {minimum}"))
# The minimum doesn't stop differencing. An enrollment that arrives alone in its course on its day, in a cell already
# shown, raises that cell by one in the next day's report, and the instructor sees which enrollment arrived.
revealed = count(f"""from mine as m
where (select count(*) from mine as o where o.course_id = m.course_id and o.day = m.day) = 1
and (select count(*) from mine as o where o.course_id = m.course_id and o.country = m.country and o.day < m.day) >= {minimum}""")
print("learners whose country the daily report reveals anyway:", f"{revealed} of {count('from mine')}")
```
Output:
```text
minimum group size: 11
cells in the report: 63
cells under the minimum: 51
cells of one learner: 6
learners a cell of one names: 3
cells shown with the minimum: 12
learners whose country the daily report reveals anyway: 92 of 482
```
Of the 63 cells of `instructor-17`'s report, 51 are under the minimum, and 6 hold one learner each. A cell of one names its learner's country to anyone who can find that learner, and the instructor can: matching the cell's average progress against their own enrollments names 3 learners exactly. With the minimum, 12 cells are shown and none of those lookups works.
The minimum doesn't stop differencing. An enrollment that arrives alone in its course on its day, into a cell that's already shown, raises that cell by one in the next day's report, and the instructor sees which enrollment arrived. That reveals the country of 92 of 482 learners. This is the attack Denning, Denning and Schwartz called a tracker: they showed that "the compromise of small query sets can in fact almost always be accomplished with the help of characteristic formulas called trackers" ([Denning et al., 1979](https://doi.org/10.1145/320064.320069)). Snowflake, which enforces a minimum group size in its engine, says its policies "do not guarantee a malicious actor could not use deliberate queries to obtain potentially sensitive data" ([Snowflake aggregation policies](https://docs.snowflake.com/en/user-guide/aggregation-policies)). Against differencing, we recommend noise, as in differential privacy ([Dwork and Roth, 2014](https://doi.org/10.1561/0400000042)), rather than higher thresholds, and *Privacy Engineering for Data Teams* covers it.
**Working rule:** Hold back every reported group below the minimum in `governance/enforcement.toml`, and treat the minimum as protection against looking one person up, never against differencing. Don't break a report down by an attribute its reader may not read, sent to a reader who can see the rows behind it: the instructor's report by country is a lookup table for countries. Drop the breakdown, or report it only after it has been made noisy.
## Hand the same policies to an engine that enforces them {#engine-enforcement}
In production, enforcement belongs in the engine every tool queries through, because Saltzer and Schroeder's principle of complete mediation holds that "every access to every object must be checked for authority" ([Saltzer and Schroeder, 1975](https://web.mit.edu/Saltzer/www/publications/protection/Basic.html)), and a policy enforced in one tool and not another fails it. Trino, an open-source distributed SQL engine, can take its access control from Open Policy Agent (OPA) through a plugin, and [chapter 5](#shape-the-request-like-trinos) shaped the platform's requests like the plugin's so the same policy could serve both. Trino 484 offers file-based, OPA and Ranger access control, and none for Cedar ([Trino 484, Security](https://trino.io/docs/484/security.html)).
The plugin asks OPA separate questions at separate URIs ([Trino 484, OPA access control](https://trino.io/docs/484/security/opa-access-control.html)):
- **Whether to allow** an operation, such as `SelectFromColumns`, answered with a boolean `allow`.
- **Row filters**, `GetRowFilters`, for a table. OPA answers with a list of `{"expression": ...}`, and "each expression essentially behaves like an additional WHERE clause". The URI is optional: "if not set no row filtering is applied".
- **Column masks**, `GetColumnMask`, for one column, or a batch of columns at `opa.policy.batch-column-masking-uri`. "Only a single column mask may be returned for a given column."
The operation names and where each request puts its table or column are in the plugin's source at tag 484 ([`OpaAccessControl.java`](https://github.com/trinodb/trino/blob/484/plugin/trino-opa/src/main/java/io/trino/plugin/opa/OpaAccessControl.java), [`OpaHighLevelClient.java`](https://github.com/trinodb/trino/blob/484/plugin/trino-opa/src/main/java/io/trino/plugin/opa/OpaHighLevelClient.java)). The documentation's batch-mask policy reads `filterResources[i].catalogName` where its own request example nests those fields under `.column`, so as printed it wouldn't match. Build the platform's requests in those shapes and give them to its policy:
Example `trino-requests` (Python 3.12.15 · duckdb 2.0.0 · pyarrow 26.0.0 · pyiceberg 0.12.0 · regopy 1.5.2 · sqlglot 30.24.0), passing in Edition 2026.10.1:
```python
import json
from lib import metadata, policy
from lib.enforce import row_filter
# The platform's policy, with a support agent added before anyone set their region (chapter 5's missing-region case).
data = policy.data_document()
data["platform"]["users"]["agent-new"] = {"groups": ["support"]}
rego = policy.interpreter(data=data)
described = metadata.load()
def trino_request(user: str, operation: str, resource: dict | None = None) -> dict:
"""A request in the shape Trino 484's OPA plugin sends (OpaAccessControl, OpaHighLevelClient)."""
action = {"operation": operation} | ({"resource": resource} if resource else {})
return {"context": {"identity": {"user": user, "groups": []}, "softwareStack": {"trinoVersion": "484"}}, "action": action}
# Trino asks `allow` about every operation: ExecuteQuery before any query runs, AccessCatalog for each catalog it names.
for operation, resource in [("ExecuteQuery", None), ("AccessCatalog", {"catalog": {"name": "platform"}})]:
print(f"allow, {operation}, agent-a05:", policy.decide(trino_request("agent-a05", operation, resource), rego)["allow"])
# GetRowFilters names a table. The policy's row_filters rule on its own skips `allow` and its missing-attribute guard,
# so for agent-new it returns no filter, which Trino reads as no filtering. The answer comes from the decision instead,
# for the same read asked as SelectFromColumns: its filters in Trino's SQL, or a filter that keeps no rows.
for user, schema, table in [("agent-a05", "support", "tickets"), ("instructor-17", "learning", "reviews"),
("agent-new", "support", "tickets")]:
request = trino_request(user, "GetRowFilters", {"table": {"catalogName": "platform", "schemaName": schema, "tableName": table}})
print(f"GetRowFilters, {user}, {schema}.{table}, the rule alone:",
json.dumps(policy.evaluate(rego, "data.platform.access.row_filters", request)))
decision = policy.decide(policy.request(user, f"{schema}.{table}", []), rego)
columns = {c.name: "" for c in described[f"{schema}.{table}"].columns.values()}
filters = [{"expression": row_filter(rule, columns, table=None).sql("trino")} for rule in decision["row_filters"]]
print(f"GetRowFilters, {user}, {schema}.{table}, answered:", json.dumps(filters if decision["allow"] else [{"expression": "false"}]))
# GetColumnMask names one column, under action.resource.column. The policy reads the columns of a SelectFromColumns
# request instead, so it masks nothing here until the request is translated.
column = {"catalogName": "platform", "schemaName": "crm", "tableName": "learners", "columnName": "email", "columnType": "varchar"}
decision = policy.decide(trino_request("analyst-1", "GetColumnMask", {"column": column}), rego)
print("GetColumnMask as sent, analyst-1, crm.learners.email:", json.dumps(decision["column_masks"]), decision["reasons"][0])
translated = policy.request("analyst-1", "crm.learners", [column["columnName"]])
print("GetColumnMask translated, analyst-1, crm.learners.email:", json.dumps(policy.decide(translated, rego)["column_masks"]))
```
Output:
```text
allow, ExecuteQuery, agent-a05: False
allow, AccessCatalog, agent-a05: False
GetRowFilters, agent-a05, support.tickets, the rule alone: [{"column": "support_region", "equals": "emea"}]
GetRowFilters, agent-a05, support.tickets, answered: [{"expression": "support_region = 'emea'"}]
GetRowFilters, instructor-17, learning.reviews, the rule alone: [{"column": "course_id", "in": [2, 75, 80, 87, 103, 134]}]
GetRowFilters, instructor-17, learning.reviews, answered: [{"expression": "course_id IN (2, 75, 80, 87, 103, 134)"}]
GetRowFilters, agent-new, support.tickets, the rule alone: []
GetRowFilters, agent-new, support.tickets, answered: [{"expression": "false"}]
GetColumnMask as sent, analyst-1, crm.learners.email: {} GetColumnMask is not an operation this policy allows
GetColumnMask translated, analyst-1, crm.learners.email: {"email": "pseudonymize"}
```
The policy's `allow` admits only `SelectFromColumns`, so pointed at Trino unchanged it would deny the checks Trino makes first: `allow` is `False` for `ExecuteQuery`, asked before any query runs, and for `AccessCatalog`. The `row_filters` rule doesn't test the operation, so it answers `GetRowFilters`, but on its own it skips `allow` and [chapter 5](#deny-by-default-then-allow-by-role)'s missing-attribute guard: for `agent-new`, a support agent with no region yet, it returns `[]`, which Trino reads as no filtering. So the answer has to come from the decision for the same read, asked as `SelectFromColumns`: its filters, written in Trino's SQL by the same [`row_filter`](#build-row-filters-from-data) the rewrite uses, or, when it denies the read, `[{"expression": "false"}]`, which keeps no rows.
`GetColumnMask` names the column under `action.resource.column`, while the policy reads a `SelectFromColumns` request's columns, so as sent it masks nothing and the policy rejects the operation; translated, it gets `pseudonymize`. In production, the translations, the check of `allow` and the Trino SQL belong in a Rego module beside the policy, so OPA can answer Trino itself; the book's examples run no OPA server, so the example shows what that module must return. One more piece maps onto Trino directly: the consent filter reads `crm.learners`, which the assistant may not, and Trino lets a filter carry an `identity` to run as, "such that a filter can target a column the requesting user cannot see".
Since 1.9.0, OPA's Compile API can also write SQL `WHERE` clauses from a policy by partial evaluation ([OPA v1.9.0](https://github.com/open-policy-agent/opa/releases/tag/v1.9.0), [Data filtering with OPA](https://www.openpolicyagent.org/docs/filtering)), but it needs an OPA server, which the examples don't run.
**Working rule:** Put enforcement in the engine every tool queries through, and keep one policy for it and for any rewrite like this chapter's. With Trino, answer every operation the plugin asks about, set both the row-filter and the column-mask URIs, since without them it applies no filters or masks, and return row filters only from a decision that allows the read. Build your tests from the request shapes in the plugin's source rather than from its documentation's example policies.
## Query on someone's behalf {#query-on-someones-behalf}
An assistant that answers questions from the warehouse is one more user of the enforcement layer, with a choice to make: whose identity does its query carry? The platform's course assistant, `assistant-1`, has its own access, for reviews by consenting learners. When it answers a person, it must query as that person, so the answer contains exactly what they could have read themselves. OWASP's guidance for language-model applications says the same: "Implement authorization in downstream systems rather than relying on an LLM to decide if an action is allowed or not" ([OWASP LLM06:2025](https://genai.owasp.org/llmrisk/llm062025-excessive-agency/)). The assistant itself belongs to *Data Assistants in Practice*; this section is about its queries.
`instructor-17` asks the assistant how their courses' reviews look, and the assistant writes a query. The tool it calls takes the user from the signed-in session, never from the model or the conversation, so no prompt can change who the query runs as:
Example `assistant-on-behalf` (Python 3.12.15 · duckdb 2.0.0 · pyarrow 26.0.0 · pyiceberg 0.12.0 · regopy 1.5.2 · sqlglot 30.24.0), passing in Edition 2026.10.1:
```python
from lib import policy
from lib.catalog import open_catalog
from lib.enforce import Enforcer
platform = Enforcer(open_catalog(["crm.learners", "learning.reviews"]))
courses = set(policy.data_document()["platform"]["instructor_courses"]["17"]) # instructor-17's own courses
def query_tool(session_user: str):
"""The tool the course assistant calls to query the warehouse. The model writes the SQL; the user it runs as comes
from the signed-in session, never from the model or the conversation."""
return lambda sql: platform.sql(sql, user=session_user).fetchall()
# instructor-17 asks the assistant how their courses' reviews look. The assistant writes this query.
question = "select course_id, count(*) as reviews, round(avg(rating), 2) as rating from learning.reviews group by all"
for runs_as in ["instructor-17", "assistant-1"]:
answer = query_tool(runs_as)(question)
print(f"run as {runs_as}, courses in the answer:", len(answer))
print(f"run as {runs_as}, reviews behind it:", sum(reviews for _, reviews, _ in answer))
mine = [row for row in answer if row[0] in courses]
print(f"run as {runs_as}, instructor-17's reviews behind it:", sum(reviews for _, reviews, _ in mine))
```
Output:
```text
run as instructor-17, courses in the answer: 6
run as instructor-17, reviews behind it: 69
run as instructor-17, instructor-17's reviews behind it: 69
run as assistant-1, courses in the answer: 130
run as assistant-1, reviews behind it: 436
run as assistant-1, instructor-17's reviews behind it: 7
```
Run as `instructor-17`, the answer covers their 6 courses and 69 reviews. Run as the assistant, it covers 130 courses, most of them other instructors', and only 7 of the instructor's own reviews, the ones whose authors consented to AI training. The assistant's identity is wrong both ways: it shows the instructor what they may not see, and hides what they may. The same holds for retrieval: an assistant grounded on governed text needs "permission-aware vector and embedding stores" ([OWASP LLM08:2025](https://genai.owasp.org/llmrisk/llm082025-vector-and-embedding-weaknesses/)).
**Working rule:** Run an assistant's queries with the identity of the person it answers, taken from their authenticated session, and let the enforcement layer apply that person's decision. Use the assistant's own identity only for work it does for no one in particular, such as a nightly summary of consented reviews for the course pages, and record both identities in the audit log. Whoever builds the assistant's training set queries as themselves, with the purpose `ai-training`, as [chapter 9](#admit-consented-reviews) does.
## What goes wrong in production {#enforcement-in-production}
An enforcement layer fails by letting a read through that nobody tested, and most of these failures look like correct answers:
- **`SELECT *` slips past on a table missing from the schema.** `qualify` leaves the star unexpanded and raises nothing. Deny every table the catalog doesn't hold, and rebuild the schema whenever the catalog changes.
- **Aggregates leak small groups.** A cell of one names its member, and differences between reports name members of cells that pass any minimum, as [the report example](#hold-back-small-groups) showed. Set the minimum, and remove breakdowns by attributes the reader may not read.
- **Joins and copies bring masked columns back.** A pipeline that runs with full access, such as the one that writes `marketing.audience`, copies `email` into a table of its own, protected only by its own classification: unclassified, it's hidden from everyone, as [the who-sees-what test](#test-who-sees-what) showed, and classified `internal` by mistake, it's shown to anyone who may read the table. [Chapter 4](#classifications-from-lineage) propagates classifications along lineage to catch that. Unmasked columns re-identify through joins too: Sweeney found 87% of the US population likely unique on ZIP code, gender and date of birth in 1990 census data ([Sweeney, 2002](https://dataprivacylab.org/dataprivacy/projects/kanonymity/kanonymity.pdf)), and Golle, counting again, found 61% on the same 1990 data and 63% on 2000's ([Golle, 2006](https://doi.org/10.1145/1179601.1179615)). Rocher and colleagues estimate, from a model fitted to census data, that 15 demographic attributes would make 99.98% of people in Massachusetts unique ([Rocher et al., 2019](https://doi.org/10.1038/s41467-019-10933-3)).
- **Direct file reads bypass the rewrite.** `read_parquet`, a path in `FROM` and table functions read data without naming a table. Deny them in the rewrite, and turn off `enable_external_access`, which DuckDB then won't turn back on: "Cannot enable external access while database is running". `lock_configuration` keeps every other setting as the layer left it; set it once the layer's connection is fully configured ([Securing DuckDB](https://duckdb.org/docs/2.0/operations_manual/securing_duckdb/overview)), not when it's created, since [chapter 8](#cross-check-the-query-log) starts DuckDB's query log after that. Anyone who can read the warehouse's files in storage needs no SQL at all, so storage permissions are part of enforcement.
- **Identifier case decides what a name matches.** A layer that matched names without normalizing them would let `"LEARNERS"`, which DuckDB resolves to `learners`, reach the raw table, as [the first section](#find-what-a-query-reads) showed. Normalize with the dialect of the engine that runs the query before matching any name.
- **Error messages print values.** DuckDB's cast errors include the value. The table rewrite makes it a masked one, and [the gate on each column](#gate-columns-on-row-filters) keeps dropped rows out of reach, but the message still reaches the user unless something stops it. Return a generic error at the service boundary, as `lib.enforce.answer` does, and log the engine's message where only the platform team reads it, apart from the audit log, which [chapter 8](#keep-values-out-of-the-record) keeps free of values. PostgreSQL warns that plans and timings remain covert channels even behind a security barrier ([PostgreSQL 18, Rules and Privileges](https://www.postgresql.org/docs/18/rules-privileges.html)).
- **DuckDB is not a security boundary.** "DuckDB executes SQL with the full privileges of the user running it, much like a shell or scripting interpreter" ([Securing DuckDB](https://duckdb.org/docs/2.0/operations_manual/securing_duckdb/overview)). A rewrite inside the same process protects data only while users can't reach DuckDB by any other route. Treat this chapter's layer as a tested model of what an engine such as Trino enforces, and enforce there in production.
## What this chapter decided {#enforcement-decisions}
- **Enforcement replaces tables, not columns.** Each catalog table in a query becomes a subquery generated from the user's decision. Rewriting column references is kept only to show that it leaks.
- **The layer fails closed.** Its sources are an allowlist of catalog tables, CTEs in scope and subqueries. It denies functions with side effects, parameters and anything but a single query. Columns a query doesn't read are typed `NULL`s, and the ones it reads are gated on the row filters.
- **Users get one generic error** from `answer`, and DuckDB's message goes to the platform team.
- **Masks are SQL in one place.** The policy names masks; `lib.enforce` writes their SQL. Pseudonyms are keyed with a secret only the layer holds, passed as a parameter.
- **Row filters are built from data,** with values as literals. Consent is a semi-join the layer runs with its own access, and an unknown consent counts as no.
- **Who sees what is tested as queries:** chapter 5's cases run through the layer on every change.
- **Reports hold back groups under the minimum group size,** 11 in `governance/enforcement.toml`, and don't break down by attributes their readers may not read.
- **Assistants query as the person they answer,** with the identity taken from the session.
- **In production, the engine enforces:** Trino with OPA, from the same policy, with both URIs set and row filters answered only from an allowed decision.
# Retention, consent and erasure
By the end of this chapter you can keep retention rules as reviewed config, make a withdrawal of consent change the next query, and erase one person from every table their data reached. You'll apply the rules against a day you pass in, find those tables from lineage, and keep only what your legal team has recorded an exception for. You'll also remove what a row delete leaves behind in an Iceberg table, old snapshots, unreferenced files and column statistics, and finish with a scan that reads every file left and shows that none of them holds the person.
The platform, the online learning platform this book follows, sells professional courses to adult learners and keeps its warehouse as Iceberg tables in one catalog. It has kept everything since its first sign-up in January 2024: every support ticket, and every learning event with the IP address it came from. Nothing says how long either should stay. Two requests from one learner test the rest of its governance. Learner 3590 withdraws their consent to marketing email, and some weeks later asks the platform to erase their data. The first should change what marketing sees from the next query on. The second should leave no copy of the learner anywhere: not in the tables built from theirs, and not in the old versions of each table that Iceberg keeps on purpose.
Three terms name the chapter's subjects. **Retention** is how long the platform keeps each kind of data. **Consent** is a person's agreement to one use of their data, such as marketing email, which they can withdraw. **Erasure** is removing a person's data on request. The chapter adds two things to the book's shared code: the platform's retention rules, in `governance/retention.toml`, and the module `lib.retention`, which applies them, plans and carries out an erasure, cleans up and checks the files afterwards, and keeps the register of erasures. It uses `lib.lineage` from [chapter 4](#lineage-from-pipelines) to find where a learner's data went, and the classifications in `lib.metadata` from [chapter 2](#a-catalog-you-can-test). Every example runs PyIceberg 0.12.0 on a local catalog.
## Keep retention rules in a reviewed file {#retention-rules-in-a-file}
A retention rule says how long one kind of data is kept, from which date the time counts, and which rows it applies to. The EU's General Data Protection Regulation (GDPR) states the principle as storage limitation: personal data are to be "kept in a form which permits identification of data subjects for no longer than is necessary for the purposes for which the personal data are processed" (Article 5(1)(e)), and its Recital 39 says that "time limits should be established by the controller for erasure or for a periodic review" ([GDPR](https://eur-lex.europa.eu/eli/reg/2016/679/oj)). NIST's catalog of security and privacy controls describes the same job in neutral words, in control SI-12(3): "dispose of, destroy, or erase information following the retention period" ([NIST SP 800-53 Rev. 5](https://csrc.nist.gov/pubs/sp/800/53/r5/upd1/final)). How long is necessary is for the platform's owners and lawyers to decide; the code applies what they decide, the same way every time.
That argues for rules as config, in a file reviewed like the access policies, and for code that applies them against a day it is given rather than the clock. A job that reads the clock deletes different rows depending on when it runs: a test can't pin its result, and a rerun after a failure can delete more than the run it replaces.
The platform's rules are in `governance/retention.toml`. The periods are the platform's own choices for this book's case, not legal advice:
```toml
[tables."support.tickets"]
keep_days = 730
counted_from = "opened_at" # tickets record when they were opened, not when they were closed
only_when = { column = "status", equals = "closed" }
[tables."learning.events"]
keep_days = 365
counted_from = "occurred_at"
[tables."billing.payments"]
keep_days = 3650 # a placeholder for the period the platform's legal team sets for payment records
counted_from = "paid_at"
```
Closed tickets are kept two years, events one year, and payments 3,650 days, a placeholder that [the erasure plan](#plan-erasure-from-lineage) relies on. The tickets rule should count from when a ticket was closed, but the table records only `opened_at`, so a ticket opened 25 months ago and closed last week goes at once. The platform lives with that because adding a `closed_at` column is the support team's change to their own table, not the governance team's; until they make it, the rule errs toward deleting early. `lib.retention.due` turns a rule and a day into an Iceberg filter. Count the rows due on the book's "today", Oct 1, 2026, and 90 days later:
Example `retention-rules-due` (Python 3.12.15 · duckdb 2.0.0 · openlineage-python 1.54.0 · pyarrow 26.0.0 · pyiceberg 0.12.0 · pyyaml 6.0.3 · sqlglot 30.24.0), passing in Edition 2026.10.1:
```python
"""Which rows the platform's retention rules no longer keep, counted against a day the job is given, never the clock."""
from datetime import timedelta
from lib import retention
from lib.catalog import open_catalog
from lib.platform import TODAY
rules = retention.rules()["tables"]
catalog = open_catalog(sorted(rules))
for name, rule in sorted(rules.items()):
state = rule.get("only_when")
print(f"{name}: kept {rule['keep_days']} days from {rule['counted_from']}"
+ (f", once {state['column']} is {state['equals']}" if state else ""))
for day in (TODAY, TODAY + timedelta(days=90)):
for name, rule in sorted(rules.items()):
table = catalog.load_table(name)
print(f"{name} on {day}, kept from:", retention.cutoff(rule, day))
print(f"{name} on {day}, rows due:", f"{retention.count(table, retention.due(rule, day)):,}")
print(f"{name} on {day}, rows in the table:", f"{retention.count(table):,}")
```
Output:
```text
billing.payments: kept 3650 days from paid_at
learning.events: kept 365 days from occurred_at
support.tickets: kept 730 days from opened_at, once status is closed
billing.payments on 2026-10-01, kept from: 2016-10-03
billing.payments on 2026-10-01, rows due: 0
billing.payments on 2026-10-01, rows in the table: 17,562
learning.events on 2026-10-01, kept from: 2025-10-01
learning.events on 2026-10-01, rows due: 9,060
learning.events on 2026-10-01, rows in the table: 100,000
support.tickets on 2026-10-01, kept from: 2024-10-01
support.tickets on 2026-10-01, rows due: 58
support.tickets on 2026-10-01, rows in the table: 2,000
billing.payments on 2026-12-30, kept from: 2017-01-01
billing.payments on 2026-12-30, rows due: 0
billing.payments on 2026-12-30, rows in the table: 17,562
learning.events on 2026-12-30, kept from: 2025-12-30
learning.events on 2026-12-30, rows due: 15,545
learning.events on 2026-12-30, rows in the table: 100,000
support.tickets on 2026-12-30, kept from: 2024-12-30
support.tickets on 2026-12-30, rows due: 111
support.tickets on 2026-12-30, rows in the table: 2,000
```
On Oct 1, 2026, the events rule keeps events from 2025-10-01 on, so 9,060 of the 100,000 events are due, and 58 of the 2,000 tickets are closed and were opened more than two years ago. Ninety days later, the same rules make 15,545 events and 111 tickets due. The payments rule makes 0 of the 17,562 payments due on Oct 1 and 0 on Dec 30. The counts depend on the day passed in and on the data, and on nothing else.
**Working rule:** Keep each table's retention period, the column it counts from and the state a row must be in, in a reviewed file, and apply it against a day the scheduler passes in, never the clock. Count from the date the rule means. When the table doesn't record that date, as the platform's tickets don't record when they closed, ask the table's owner to add the column rather than let the rule count from another date for good, and say in the rule's file which way it errs until then.
## Delete expired rows with row deletes {#delete-expired-rows}
With [the rules in a file](#retention-rules-in-a-file), the retention job deletes what they say. Iceberg writers delete rows in one of two ways, set by the table property `write.delete.mode`, which defaults to `copy-on-write` ([Iceberg configuration](https://iceberg.apache.org/docs/latest/configuration/)). Copy-on-write rewrites each affected data file without the deleted rows. Merge-on-read leaves the data file alone and writes a delete file beside it, which the spec defines in two kinds ([Iceberg table spec](https://iceberg.apache.org/spec/)): "Position deletes -- Mark a row deleted by data file path and the row position in the data file", a deletion vector in tables of version 3 or above, and "Equality deletes -- Mark a row deleted by one or more column values, like id = 5". PyIceberg 0.12.0's `Table.delete` always copies on write: with `merge-on-read` set, it warns "Merge on read is not yet supported, falling back to copy-on-write" and rewrites the files anyway ([source](https://github.com/apache/iceberg-python/blob/pyiceberg-0.12.0/pyiceberg/table/__init__.py)). Either way, a delete is a commit, a new snapshot, and the snapshot before it is still there.
`lib.retention.apply` deletes, for each rule, the rows `due` returns for the day it's given, and records the rule's table and the day in the commit's snapshot summary, so the table's own history says why the rows went. Apply the platform's rules on Oct 1, 2026, apply them again, read each table as it is now and by time travel to the snapshot before the delete, and expire the old snapshots by age:
Example `apply-retention` (Python 3.12.15 · duckdb 2.0.0 · openlineage-python 1.54.0 · pyarrow 26.0.0 · pyiceberg 0.12.0 · pyyaml 6.0.3 · sqlglot 30.24.0), passing in Edition 2026.10.1:
```python
"""Apply the retention rules on the book's "today": Iceberg row deletes, which leave the old rows in older snapshots."""
from datetime import datetime, timedelta, timezone
from lib import retention
from lib.catalog import open_catalog
from lib.platform import TODAY
rules = retention.rules()["tables"]
catalog = open_catalog(sorted(rules))
before = {name: catalog.load_table(name).current_snapshot().snapshot_id for name in rules}
deleted = retention.apply(catalog, TODAY)
for name, rows in deleted.items():
print(f"{name}, rows deleted:", f"{rows:,}")
print("rows deleted by running again for the same day:", sum(retention.apply(catalog, TODAY).values()))
for name in (name for name, rows in deleted.items() if rows):
rule, table = rules[name], catalog.load_table(name)
summary = table.current_snapshot().summary
print(f"{name}, snapshots:", " then ".join(s.summary.operation.value for s in table.snapshots()))
print(f"{name}, rows written again to delete the others:", f"{int(summary['added-records']):,}")
print(f"{name}, the commit records:", summary["retention-rule"], "on", summary["retention-day"])
rows_due = retention.due(rule, TODAY)
print(f"{name}, rows due, read now:", f"{retention.count(table, rows_due):,}")
print(f"{name}, rows due, read by time travel:", f"{retention.count(table, rows_due, before[name]):,}")
# Snapshots carry real commit times, so their age is measured on the clock, by the table's own setting.
print(f"{name}, snapshots expired by age today:", retention.expire_by_age(table))
later = datetime.now(timezone.utc) + timedelta(days=6)
print(f"{name}, snapshots expired by age six days later:", retention.expire_by_age(table, later))
```
Output:
```text
billing.payments, rows deleted: 0
learning.events, rows deleted: 9,060
support.tickets, rows deleted: 58
rows deleted by running again for the same day: 0
learning.events, snapshots: append then overwrite
learning.events, rows written again to delete the others: 90,940
learning.events, the commit records: learning.events on 2026-10-01
learning.events, rows due, read now: 0
learning.events, rows due, read by time travel: 9,060
learning.events, snapshots expired by age today: 0
learning.events, snapshots expired by age six days later: 1
support.tickets, snapshots: append then overwrite
support.tickets, rows written again to delete the others: 1,942
support.tickets, the commit records: support.tickets on 2026-10-01
support.tickets, rows due, read now: 0
support.tickets, rows due, read by time travel: 58
support.tickets, snapshots expired by age today: 0
support.tickets, snapshots expired by age six days later: 1
```
The job deleted 9,060 events, 58 tickets and 0 payments. Running it again for the same day deleted 0 rows and made no commit, so a retried run is harmless. Each table it changed went from an `append` snapshot to an `overwrite`, and the summary shows what copy-on-write costs: to delete the old events, PyIceberg wrote the other 90,940 again, because the table is one unpartitioned file. Partitioned by month of `occurred_at`, the table would let the same delete drop whole files instead, which `Table.delete` does when every row of a file matches. In PyIceberg 0.12.0, writing a table partitioned by month needs the `pyiceberg-core` extra, which this book's examples don't install.
The time-travel lines matter most here. Read as it is now, `learning.events` holds 0 rows past their retention. Read by time travel, the snapshot before the delete still returns all 9,060, IP addresses included, and the Parquet file that holds them is still on disk. `lib.retention.expire_by_age` expires snapshots older than the table's own `history.expire.max-snapshot-age-ms`, five days here, measured on the clock: it expires 0 snapshots of `learning.events` straight after the delete, and 1 six days later. The retention job isn't finished until the old snapshots are [expired](#expire-snapshots) and [the files no snapshot references are removed](#remove-unreferenced-files): the same clean-up an erasure needs.
**Working rule:** Apply retention as Iceberg row deletes, one rule at a time, and record the rule and the day in each commit's snapshot summary. Make the job idempotent for a given day. Partition large tables by the column a rule counts from, so its deletes drop whole files instead of rewriting them; in PyIceberg, install its `pyiceberg-core` extra to write them. Count the job done only once the old snapshots are expired and their files removed.
## Read consent where the query runs {#consent-at-query-time}
[Retention](#retention-rules-in-a-file) removes rows by age. Consent changes who may use them, and withdrawing it isn't the same event as asking for erasure, so the platform's code treats them as two paths. The GDPR's Article 7(3) says: "The data subject shall have the right to withdraw his or her consent at any time. The withdrawal of consent shall not affect the lawfulness of processing based on consent before its withdrawal." ([GDPR](https://eur-lex.europa.eu/eli/reg/2016/679/oj)). Article 17(1)(b) lists a withdrawal as one ground for erasure, "where there is no other legal ground for the processing". Whether that applies to a given request is for the platform's lawyers. In the code, a withdrawal changes one attribute; an erasure deletes rows.
On the platform, consent is an attribute of the data that the access policy reads, as [chapter 5](#narrow-by-attribute) set it up: marketing's row filter keeps only learners whose `marketing_consent` is true, and the course assistant's keeps only the reviews whose authors' `ai_training_consent` is true. So a withdrawal needs no change to the policy. The CRM, the application where learners manage their accounts, syncs the new value into `crm.learners`, and the next query marketing runs through the enforcement layer, which [chapter 6](#enforcing-policies-on-queries) builds to rewrite every query, returns one learner fewer. Withdraw learner 3590's marketing consent, and compare what marketing sees and the tables hold before the withdrawal, after it, and after the marketing pipeline runs again:
Example `consent-withdrawal` (Python 3.12.15 · duckdb 2.0.0 · openlineage-python 1.54.0 · pyarrow 26.0.0 · pyiceberg 0.12.0 · pyyaml 6.0.3 · regopy 1.5.2 · sqlglot 30.24.0), passing in Edition 2026.10.1:
```python
"""A learner withdraws marketing consent: one attribute changes, and marketing's next query returns one learner fewer.
No row is deleted. A table built with the consent filter keeps the learner until its pipeline runs again."""
import json
import pyarrow as pa
from pyiceberg.expressions import EqualTo
from lib import pipelines, policy, retention
from lib.catalog import open_catalog
from lib.enforce import Enforcer
LEARNER = 3590
catalog = open_catalog(["crm.learners", "learning.enrollments", "learning.courses"])
pipelines.run(catalog, "marketing_audience")
decision = policy.decide(policy.request("marketing-1", "crm.learners", ["learner_id"]))
print("marketing's row filters on crm.learners:", json.dumps(decision["row_filters"]))
def report(when: str) -> None:
seen = [row[0] for row in Enforcer(catalog).sql("select learner_id from crm.learners", "marketing-1").fetchall()]
print(f"{when}, learners marketing sees:", f"{len(seen):,}")
print(f"{when}, marketing sees learner {LEARNER}:", LEARNER in seen)
print(f"{when}, rows stored in crm.learners:", f"{retention.count(catalog.load_table('crm.learners')):,}")
audience = retention.count(catalog.load_table("marketing.audience"), EqualTo("learner_id", LEARNER))
print(f"{when}, learner {LEARNER}'s rows in marketing.audience:", audience)
report("before")
learners = catalog.load_table("crm.learners")
row = learners.scan(row_filter=EqualTo("learner_id", LEARNER)).to_arrow()
row = row.set_column(row.schema.get_field_index("marketing_consent"), "marketing_consent", pa.array([False]))
learners.overwrite(row, overwrite_filter=EqualTo("learner_id", LEARNER)) # what the CRM's sync writes
report("after the withdrawal")
pipelines.run(catalog, "marketing_audience")
report("after the audience is built again")
```
Output:
```text
marketing's row filters on crm.learners: [{"column": "marketing_consent", "equals": true}, {"column": "status", "equals": "active"}]
before, learners marketing sees: 1,861
before, marketing sees learner 3590: True
before, rows stored in crm.learners: 5,000
before, learner 3590's rows in marketing.audience: 1
after the withdrawal, learners marketing sees: 1,860
after the withdrawal, marketing sees learner 3590: False
after the withdrawal, rows stored in crm.learners: 5,000
after the withdrawal, learner 3590's rows in marketing.audience: 1
after the audience is built again, learners marketing sees: 1,860
after the audience is built again, marketing sees learner 3590: False
after the audience is built again, rows stored in crm.learners: 5,000
after the audience is built again, learner 3590's rows in marketing.audience: 0
```
Marketing's row filters stay `[{"column": "marketing_consent", "equals": true}, {"column": "status", "equals": "active"}]`. Before the withdrawal, marketing sees 1,861 learners, learner 3590 among them; after it, 1,860, without learner 3590. `crm.learners` still stores 5,000 rows. Nothing was deleted, and the learner's account, enrollments and history are as they were.
The copy is the problem. `marketing.audience`, the list a marketing pipeline builds each night, applies the consent filter when it's built, and the policy gives marketing that table with no consent filter of its own. After the withdrawal it still holds 1 row for learner 3590. Until someone classifies the table's columns, the policy redacts them for every reader; once someone does, as [chapter 4](#classifications-from-lineage) proposes, a campaign sent from the table before its next build reaches someone who said no. Only once the pipeline runs again does it hold 0. A training corpus is built with a consent filter too, and [chapter 9](#withdrawn-after-the-build) checks each build against today's consent before training on it.
A true-or-false column also says too little: `ai_training_consent` is null for everyone who signed up before the platform asked, and false can mean refused or withdrawn. The Data Privacy Vocabulary, a W3C Community Group report rather than a standard, names the states a consent can be in, given, refused, withdrawn, expired and unknown among them ([DPV 2.3](https://w3c-cg.github.io/dpv/2.3/dpv/)).
**Working rule:** Keep consent where the query runs: in one table that the system the learner uses keeps current, read by the policy as a row filter, so a withdrawal changes the next query and deletes nothing. When the platform's lawyers decide that a withdrawal is also a ground for erasure, send it through [the erasure path](#plan-erasure-from-lineage) as well. Rebuild every table built with a consent filter whenever consent changes, or give its readers the same filter. Record each consent as a state and the time it changed, and derive the true-or-false attribute the policy reads from it, counting unknown as not given.
## Plan an erasure from lineage {#plan-erasure-from-lineage}
[A withdrawal](#consent-at-query-time) changes one attribute; an erasure has to find the person everywhere. The GDPR's Article 17(1) opens: "The data subject shall have the right to obtain from the controller the erasure of personal data concerning him or her without undue delay", on the grounds it lists ([GDPR](https://eur-lex.europa.eu/eli/reg/2016/679/oj)). Which grounds apply is a legal question. The platform treats a request as reaching every copy, so the first job is a list of where the person's data is, an engineering question with two halves. The source tables that hold a learner's ID come from the catalog: [chapter 2](#classify-pseudonymous-ids) decided that every table with a `learner_id` is part of every erasure, whatever its classification. The tables built from them come from lineage: [chapter 4](#impact-of-a-column-change)'s walk of the column graph, run from each source table, reaches every table their rows shaped.
Some data stays. Article 17(3) opens: "Paragraphs 1 and 2 shall not apply to the extent that processing is necessary:", and its five points include "(b) for compliance with a legal obligation which requires processing by Union or Member State law to which the controller is subject" ([GDPR](https://eur-lex.europa.eu/eli/reg/2016/679/oj)). Whether an exception applies, to which table, and for how long is the platform's legal team's decision, never the code's. The code gives the decision one place, in `governance/retention.toml`, where the legal team writes each exception with its reason. The book's reason is a placeholder, because we decide no legal question:
```toml
[erasure.exceptions."billing.payments"]
reason = "PLACEHOLDER: a legal obligation to keep payment records, such as tax records. The legal team replaces this with the obligation and the law it comes from."
decided_by = "legal team" # a role, never a person's name
erase_columns = ["card_number"] # what the obligation doesn't need; an empty list keeps the whole row
```
An erasure keeps an excepted table's rows until the table's [retention rule](#retention-rules-in-a-file) deletes them, and erases the columns `erase_columns` lists from them at once: here the full card number, which the platform's metadata says "Should not be stored at all". `lib.retention.exceptions` refuses an exception with no reason, or one for a table no rule ends.
Finding a derived table isn't the same as finding the learner's rows in it. `lib.retention.plan` looks at the lineage edges into each table the walk reaches, for a column copied unchanged, an IDENTITY edge in OpenLineage's terms, from the learner's ID or from the ID of one of their rows, such as a review. If there is one, the plan deletes by that column. If there's none, the learner's rows only shaped the table, and the plan runs the table's job again once the sources are erased. The catalog the plan reads is the platform as the request finds it, weeks after the withdrawal: `crm.learners` holds learner 3590's withdrawn consent, and the four pipelines of [chapter 4](#lineage-from-pipelines), which build the analytics, marketing and AI tables, have run since, with their lineage recorded. Plan learner 3590's erasure:
Example `erasure-plan` (Python 3.12.15 · duckdb 2.0.0 · openlineage-python 1.54.0 · phonenumbers 9.0.41 · presidio-analyzer 2.2.364 · pyarrow 26.0.0 · pyiceberg 0.12.0 · pyyaml 6.0.3 · sqlglot 30.24.0 · tldextract 5.4.0), passing in Edition 2026.10.1:
```python
"""Plan one learner's erasure: the tables that hold their learner_id, then every table lineage says their rows reach,
with the exceptions the platform's legal team recorded in governance/retention.toml."""
from lib import pii, retention
LEARNER = 3590
catalog, graph = retention.open_platform() # weeks after the learner withdrew marketing consent
steps = retention.plan(catalog, graph, LEARNER)
for step in steps:
if step.column is None:
print(f"{step.table}: run {retention.job_for(step.table)} again")
elif step.kept:
print(f"{step.table}: keep, under an exception, and erase {', '.join(step.cleared)}")
else:
print(f"{step.table}: delete where {step.column} in {list(step.values)}"
+ (f", a copy of {step.via}" if step.via else ""))
print("tables in the plan:", len(steps))
print("tables found by their learner_id column:", sum(1 for step in steps if step.column and not step.via))
print("tables found by lineage:", sum(1 for step in steps if step.via or step.column is None))
for step in (step for step in steps if step.kept):
print(f"the reason recorded for {step.table}:", step.kept)
theirs = retention.rows_of([LEARNER])
for name in ("billing.payments", "marketing.audience"):
print(f"the learner's rows in {name}:", retention.count(catalog.load_table(name), theirs))
found = retention.traces(catalog, steps, LEARNER)
# The phone number is printed through lib.pii.masked, which keeps its shape and its last four digits.
print("the learner's identifying values, as the scan searches for them:", pii.masked(", ".join(found.texts)))
named = retention.mentions(catalog, steps, found)
print("rows the plan doesn't delete whose text names the learner:", sum(named.values()))
```
Output:
```text
billing.payments: keep, under an exception, and erase card_number
crm.learners: delete where learner_id in [3590]
learning.enrollments: delete where learner_id in [3590]
learning.events: delete where learner_id in [3590]
learning.reviews: delete where learner_id in [3590]
support.tickets: delete where learner_id in [3590]
analytics.course_performance: run course_performance again
analytics.learner_activity: delete where learner_id in [3590], a copy of crm.learners.learner_id
marketing.audience: delete where learner_id in [3590], a copy of crm.learners.learner_id
ai.review_corpus: delete where review_id in [1199], a copy of learning.reviews.review_id
tables in the plan: 10
tables found by their learner_id column: 6
tables found by lineage: 4
the reason recorded for billing.payments: PLACEHOLDER: a legal obligation to keep payment records, such as tax records. The legal team replaces this with the obligation and the law it comes from.
the learner's rows in billing.payments: 2
the learner's rows in marketing.audience: 0
the learner's identifying values, as the scan searches for them: •••-•••-5880 x•, 87199 jacob rad, alexa reed, alexa590@post.e
rows the plan doesn't delete whose text names the learner: 0
```
The plan visits 10 tables: 6 source tables with a `learner_id` column, and 4 that lineage adds. It keeps the learner's 2 payments under the exception. The plan is decided by columns and lineage, not rows, so it lists two tables that hold nothing of the learner's: `support.tickets`, since learner 3590 never opened a ticket, and `marketing.audience`, which holds 0 rows because the pipeline rebuilt it after the withdrawal. `ai.review_corpus` has no `learner_id` at all, but lineage says its `review_id` is an IDENTITY copy of `learning.reviews.review_id`, so the plan deletes the learner's one review by that ID. `analytics.course_performance` holds only counts and averages per course, so the plan rebuilds it from the erased sources. Whether an average that once counted the learner is still their data is a legal question; rebuilding costs one pipeline run, so the code doesn't need the answer.
Text can name a person in rows the plan doesn't delete, such as another learner's ticket. The example searches every string column for the identifying values `governance/retention.toml` lists, and finds 0 such rows. The search matches the first 15 characters of each value, lowercased, anywhere in the text: here `•••-•••-5880 x•, 87199 jacob rad, alexa reed, alexa590@post.e`, printed with the phone number masked, cut to 15 for the reason [the final scan](#verify-the-erasure) explains. A name typed differently or a phone number without separators needs the detectors of [chapter 3](#finding-personal-data).
**Working rule:** Plan every erasure from the catalog and the lineage graph, not from a list someone maintains: every table with the person's ID column, then every table lineage says their rows reach. Delete from a derived table by a column that lineage shows is an unchanged copy of an ID, and rebuild the derived tables the person's rows only shaped. Keep exceptions in config, written by your legal team with their reasons, and let the plan read and show them; never let the code decide one. Run [chapter 4's check for tables no job's lineage accounts for](#job-nobody-instrumented) first, because the plan can't reach a copy the lineage doesn't show.
## Erase the rows the plan lists {#erase-the-rows}
[The plan](#plan-erasure-from-lineage) says where learner 3590's data is, and `lib.retention.erase` carries it out, in its order. It deletes each step's rows with `Table.delete`, sources first, writes the rows an exception keeps again with the exception's columns empty, and runs the job of each table the plan rebuilds once the sources are done. Every commit records the request's ID, `erasure-0001`, in its snapshot summary. A summary is written into the table's metadata files, which Iceberg keeps after the snapshot expires: all of them by default, and the last `write.metadata.previous-versions-max`, 100 by default, with `write.metadata.delete-after-commit.enabled` set ([Iceberg maintenance](https://iceberg.apache.org/docs/latest/maintenance/), [Iceberg configuration](https://iceberg.apache.org/docs/latest/configuration/)). So the summary holds only the request's ID, which [the erasure register](#erased-learner-comes-back) pairs with a keyed hash of the learner's ID, and never the learner's email. Erase the learner, then read two tables as they are now and by time travel:
Example `erase-learner` (Python 3.12.15 · duckdb 2.0.0 · openlineage-python 1.54.0 · pyarrow 26.0.0 · pyiceberg 0.12.0 · pyyaml 6.0.3 · sqlglot 30.24.0), passing in Edition 2026.10.1:
```python
"""Carry out the erasure plan with Iceberg row deletes. The current snapshots no longer hold the learner; the snapshots
before them still do, and time travel reads them."""
from lib import retention
LEARNER = 3590
catalog, graph = retention.open_platform()
steps = retention.plan(catalog, graph, LEARNER)
before = {step.table: catalog.load_table(step.table).current_snapshot().snapshot_id for step in steps}
done = retention.erase(catalog, steps, "erasure-0001")
for table, rows in done.items():
print(f"{table}, rows deleted:" if rows != "rebuilt" else f"{table}:", rows)
print("tables one row was deleted from:", ", ".join(table for table, rows in done.items() if rows == 1))
theirs = retention.rows_of([LEARNER])
payments = catalog.load_table("billing.payments").scan(row_filter=theirs).to_arrow()
print("billing.payments, the learner's rows kept:", payments.num_rows)
print("billing.payments, card numbers left in them:", payments.num_rows - payments["card_number"].null_count)
for name in ("crm.learners", "learning.events"):
table = catalog.load_table(name)
print(f"the learner's rows in {name}, read now:", retention.count(table, theirs))
print(f"the learner's rows in {name}, read by time travel:", retention.count(table, theirs, before[name]))
summary = catalog.load_table("crm.learners").current_snapshot().summary
print("crm.learners, rows written again to delete one:", f"{int(summary['added-records']):,}")
print("crm.learners, the commit records:", summary["erasure-request"])
holding = [(table, snapshot) for step in steps if step.column is not None
for table in [catalog.load_table(step.table)] for snapshot in table.snapshots()
if not step.kept and retention.count(table, retention.rows(step), snapshot.snapshot_id)]
print("snapshots that still hold the rows the erasure deleted:", len(holding))
print("of them current:", sum(snapshot == table.current_snapshot() for table, snapshot in holding))
```
Output:
```text
billing.payments, rows deleted: 0
crm.learners, rows deleted: 1
learning.enrollments, rows deleted: 2
learning.events, rows deleted: 19
learning.reviews, rows deleted: 1
support.tickets, rows deleted: 0
analytics.course_performance: rebuilt
analytics.learner_activity, rows deleted: 1
marketing.audience, rows deleted: 0
ai.review_corpus, rows deleted: 1
tables one row was deleted from: crm.learners, learning.reviews, analytics.learner_activity, ai.review_corpus
billing.payments, the learner's rows kept: 2
billing.payments, card numbers left in them: 0
the learner's rows in crm.learners, read now: 0
the learner's rows in crm.learners, read by time travel: 1
the learner's rows in learning.events, read now: 0
the learner's rows in learning.events, read by time travel: 19
crm.learners, rows written again to delete one: 4,999
crm.learners, the commit records: erasure-0001
snapshots that still hold the rows the erasure deleted: 6
of them current: 0
```
The erasure deleted the learner's rows from every table in the plan that held any: 19 events, 2 enrollments, and one row from each of `crm.learners, learning.reviews, analytics.learner_activity, ai.review_corpus`. It deleted none from `support.tickets` and `marketing.audience`, which held none, and it rebuilt `analytics.course_performance`. `billing.payments` keeps the learner's 2 payments, with 0 card numbers left in them. Read now, `crm.learners` holds 0 rows for the learner. Read by time travel to the snapshot before the erasure, it returns 1, with name, email, phone and address, and `learning.events` returns all 19 events. To delete one row of `crm.learners`, PyIceberg wrote the other 4,999 into a new file, as copy-on-write does, and the old file, with the learner in it, belongs to the snapshot before.
Across the plan's tables, 6 snapshots still hold the rows the erasure deleted, and 0 of them are current. That is what snapshots are for: a reader can go back to any of them, and a bad commit can be rolled back. For an erasure, it means the delete took the learner out of the table but not out of storage.
**Working rule:** Carry out an erasure as row deletes in the plan's order, and record only a request ID in each commit, never a value of the person's. Treat the delete as the first of three steps: the snapshots before it still hold the person, and anyone allowed to time travel can read them until they're [expired](#expire-snapshots).
## Expire snapshots by the table's own settings {#expire-snapshots}
[After the erasure's deletes](#erase-the-rows), learner 3590's rows live on in the snapshots before them. The Iceberg spec says what comes next: "Table snapshots expire and are removed from metadata to allow removed or replaced data files to be physically deleted" ([Iceberg table spec](https://iceberg.apache.org/spec/)). Each table says how, in its properties ([Iceberg configuration](https://iceberg.apache.org/docs/latest/configuration/)): `gc.enabled` "Allows garbage collection operations such as expiring snapshots and removing orphan files", `history.expire.min-snapshots-to-keep` is how many snapshots to keep however old they are, and `history.expire.max-snapshot-age-ms` is the age at which the rest may go. In Iceberg's Java library, "Regularly expiring snapshots deletes unused data files" ([Iceberg maintenance](https://iceberg.apache.org/docs/latest/maintenance/)).
PyIceberg 0.12.0 does less. Its `table.maintenance.expire_snapshots()` commits one change, which removes the snapshots from the table's metadata, and deletes no file ([source](https://github.com/apache/iceberg-python/blob/pyiceberg-0.12.0/pyiceberg/table/update/snapshot.py)), though its documentation offers it to "clean up table metadata and reduce storage costs" ([PyIceberg API](https://py.iceberg.apache.org/api/)). Nothing in it reads `gc.enabled`, and only its command-line tool reads the `history.expire` settings. An issue asks for expiry to delete files too ([issue 2604](https://github.com/apache/iceberg-python/issues/2604)). Take `crm.learners` with 3 snapshots, the load and two deletes, once with `gc.enabled=false` and once with `history.expire.min-snapshots-to-keep=3`. Expire both by age with PyIceberg, `older_than` the current time, and with `lib.retention.expire`, which reads the two settings first:
Example `expiry-ignores-table-settings` (Python 3.12.15 · duckdb 2.0.0 · openlineage-python 1.54.0 · pyarrow 26.0.0 · pyiceberg 0.12.0 · pyyaml 6.0.3 · sqlglot 30.24.0), passing in Edition 2026.10.1:
```python
"""PyIceberg's snapshot expiry ignores two of the table's own settings, gc.enabled and min-snapshots-to-keep. The
book's expiry reads them first."""
from datetime import datetime, timezone
from pyiceberg.expressions import EqualTo
from lib import retention
from lib.catalog import open_catalog
def learners_with(settings: dict):
"""crm.learners with three snapshots: the load, then two deletes."""
table = open_catalog(["crm.learners"]).load_table("crm.learners")
with table.transaction() as transaction:
transaction.set_properties(settings)
for learner in (3590, 3591):
table.delete(EqualTo("learner_id", learner))
return table
for settings in ({"gc.enabled": "false"}, {"history.expire.min-snapshots-to-keep": "3"}):
setting = ", ".join(f"{key}={value}" for key, value in settings.items())
table = learners_with(settings)
print(f"{setting}, snapshots:", len(table.snapshots()))
try:
print(f"{setting}, the book's expiry expires:", retention.expire(table))
except retention.RetentionError as error:
print(f"{setting}, the book's expiry refuses:", error)
table.maintenance.expire_snapshots().older_than(datetime.now(timezone.utc)).commit() # by age, as of now
print(f"{setting}, snapshots left by PyIceberg's expiry:", len(table.snapshots()))
```
Output:
```text
gc.enabled=false, snapshots: 3
gc.enabled=false, the book's expiry refuses: crm.learners has gc.enabled=false, which turns off garbage collection for the table
gc.enabled=false, snapshots left by PyIceberg's expiry: 1
history.expire.min-snapshots-to-keep=3, snapshots: 3
history.expire.min-snapshots-to-keep=3, the book's expiry expires: 0
history.expire.min-snapshots-to-keep=3, snapshots left by PyIceberg's expiry: 1
```
With garbage collection off, PyIceberg's expiry by age left 1 of the 3 snapshots, and with three to keep, it also left 1: `older_than` expires every snapshot older than the time it's given except those a branch or tag points at, here only the current one. The book's expiry refused the first table, `crm.learners has gc.enabled=false, which turns off garbage collection for the table`, and expired 0 snapshots of the second. An owner sets these properties for reasons the expiring code can't see, and in PyIceberg they are only advice.
For an erasure, both settings mean the job can't finish: a table that keeps a snapshot from before the delete still holds the person. A job that stopped at that table would already have deleted rows from the tables before it, and would leave every table after it unexpired, so the check belongs before the first delete. `lib.retention.erase_everywhere` runs `lib.retention.blocked` on every table in the plan first: garbage collection off, more than one snapshot to keep, or a branch or tag besides `main`. If any table fails, the job refuses to start and names every such table. Turn off garbage collection on `crm.learners`, keep three snapshots of `learning.events`, then run the job and the scan:
Example `erasure-refuses-to-start` (Python 3.12.15 · duckdb 2.0.0 · openlineage-python 1.54.0 · pyarrow 26.0.0 · pyiceberg 0.12.0 · pyyaml 6.0.3 · sqlglot 30.24.0), passing in Edition 2026.10.1:
```python
"""The erasure job checks every table in its plan before it changes any. When a table's own settings would keep the
learner, it refuses to start and names the tables; the scan of every file still runs, and finds the learner."""
from datetime import timedelta
from lib import retention
LEARNER = 3590
catalog, graph = retention.open_platform()
for name, settings in (("crm.learners", {"gc.enabled": "false"}),
("learning.events", {"history.expire.min-snapshots-to-keep": "3"})):
with catalog.load_table(name).transaction() as transaction:
transaction.set_properties(settings)
before = {name: catalog.load_table(name).current_snapshot().snapshot_id for name in retention.tables(catalog)}
try:
steps, found, done = retention.erase_everywhere(catalog, graph, LEARNER, "erasure-0001", min_age=timedelta(0))
except retention.ErasureRefused as refusal:
for name, why in refusal.tables.items():
print(f"the job refuses, because of {name}:", why)
found = refusal.found
print("tables the job changed:", sum(catalog.load_table(name).current_snapshot().snapshot_id != snapshot
for name, snapshot in before.items()))
report = retention.scan(catalog, found)
print("tables whose files hold the learner:", sum(1 for _, holding in report.values() if holding))
print("files holding the learner:", sum(holding for _, holding in report.values()))
```
Output:
```text
the job refuses, because of crm.learners: gc.enabled=false turns off garbage collection for the table
the job refuses, because of learning.events: history.expire.min-snapshots-to-keep=3 keeps the snapshot from before the delete
tables the job changed: 0
tables whose files hold the learner: 7
files holding the learner: 8
```
The job names both tables, `crm.learners` because `gc.enabled=false turns off garbage collection for the table`, and `learning.events` because `history.expire.min-snapshots-to-keep=3 keeps the snapshot from before the delete`, and changed 0 tables. The scan runs either way, from the traces the refusal carries, and finds the learner in 8 files across 7 tables, since nothing was erased. A person then decides with the tables' owners, rather than code overriding an owner's setting.
**Working rule:** Expire snapshots through one function of your own that reads `gc.enabled` and the `history.expire` settings before it expires anything, and refuses when garbage collection is off, because PyIceberg's expiry reads neither; check again at every PyIceberg release. Before an erasure deletes anything, check every table in its plan the same way, and refuse the whole erasure when one would keep the person. For an erasure, expire every snapshot older than the delete. For retention, expire by age, as `history.expire.max-snapshot-age-ms` says, measured on the clock, since snapshots carry real commit times.
## Remove the files no snapshot references {#remove-unreferenced-files}
Once [the old snapshots are expired](#expire-snapshots), the files only they referenced are unreferenced: under the table's location, but in no snapshot's metadata. Iceberg calls them orphans, and its Java library removes them in a separate action, with a warning: "It is dangerous to remove orphan files with a retention interval shorter than the time expected for any write to complete", and "The default interval is 3 days." ([Iceberg maintenance](https://iceberg.apache.org/docs/latest/maintenance/)). PyIceberg 0.12.0 has no such action. Its release notes list none ([release notes](https://github.com/apache/iceberg-python/releases/tag/pyiceberg-0.12.0)), the issue that asked for one was closed by the project's stale bot, marked not planned ([issue 1200](https://github.com/apache/iceberg-python/issues/1200)), and two pull requests that added one were closed without being merged ([1958](https://github.com/apache/iceberg-python/pull/1958), [3358](https://github.com/apache/iceberg-python/pull/3358)). Its only call that deletes data files is `purge_table`, which deletes the whole table ([source](https://github.com/apache/iceberg-python/blob/pyiceberg-0.12.0/pyiceberg/catalog/__init__.py)).
So `lib.retention` removes them itself, from what the metadata tables PyIceberg exposes as `table.inspect` say a table references. `lib.retention.referenced` collects the metadata files, current and logged, every snapshot's manifest list, every manifest (a manifest lists data files, with each file's column statistics), every data and delete file live in any snapshot, and any statistics files. Every other file under the table's location is unreferenced, and `lib.retention.sweep` deletes the ones older than the minimum age its caller passes. `governance/retention.toml` sets that age to 72 hours, Java's default, and `lib.retention.min_file_age` reads it. After the erasure, expire the plan's tables, sweep with that age and then with none, and check every file for the learner:
Example `sweep-unreferenced-files` (Python 3.12.15 · duckdb 2.0.0 · openlineage-python 1.54.0 · pyarrow 26.0.0 · pyiceberg 0.12.0 · pyyaml 6.0.3 · sqlglot 30.24.0), passing in Edition 2026.10.1:
```python
"""After the erasure, expire the old snapshots, then remove the files no snapshot references. PyIceberg's expiry
deletes no file, and it has no orphan-file removal, so lib.retention.sweep does it with PyIceberg's inspect tables."""
from datetime import timedelta
from lib import retention
LEARNER = 3590
catalog, graph = retention.open_platform()
steps = retention.plan(catalog, graph, LEARNER)
found = retention.traces(catalog, steps, LEARNER)
retention.erase(catalog, steps, "erasure-0001")
tables = [catalog.load_table(step.table) for step in steps]
print("snapshots expired:", sum(retention.expire(table) for table in tables))
print("files under the plan's tables:", sum(len(retention.files_under(table.location())) for table in tables))
print("files no metadata references:", sum(len(retention.unreferenced(table)) for table in tables))
print("files holding the learner, after expiry:", sum(held for _, held in retention.scan(catalog, found).values()))
print("files swept with the configured minimum age:", sum(retention.sweep(t, retention.min_file_age()) for t in tables))
# One writer, nothing in flight: no file is too young to remove, so the examples sweep with no minimum age.
print("files swept with no minimum age:", sum(retention.sweep(table, timedelta(0)) for table in tables))
print("files holding the learner, after the sweep:", sum(held for _, held in retention.scan(catalog, found).values()))
for table in tables:
entries = table.inspect.entries()
for status, metrics in zip(entries["status"].to_pylist(), entries["readable_metrics"].to_pylist()):
bounds = metrics.get("email") or {}
kept = (bounds.get("lower_bound"), bounds.get("upper_bound"))
if any(text in bound.lower() for bound in kept if bound for text in found.texts):
print(f"{'.'.join(table.name()[-2:])}, manifest entry with status {status}, email bounds:",
bounds["lower_bound"], "to", bounds["upper_bound"])
```
Output:
```text
snapshots expired: 8
files under the plan's tables: 100
files no metadata references: 28
files holding the learner, after expiry: 9
files swept with the configured minimum age: 0
files swept with no minimum age: 28
files holding the learner, after the sweep: 1
crm.learners, manifest entry with status 2, email bounds: ALEXA590@POST.EX to zofia.haase@exan
```
Expiry removed 8 snapshots and no file. Of the 100 files under the plan's tables, 28 are referenced by nothing, and 9 files hold the learner. A sweep with the configured 72 hours deleted 0 files, since every file was written seconds before. The examples then pass an age of zero, because one process writes their catalog. That sweep deleted 28 files, and 1 file still holds the learner: a manifest that the current snapshot of `crm.learners` references.
The spec says why: "When a file is replaced or deleted from the dataset, its manifest entry fields store the snapshot ID in which the file was deleted and status 2 (deleted)" ([Iceberg table spec](https://iceberg.apache.org/spec/)). That entry keeps the deleted file's column statistics. By default, under the metrics mode `truncate(16)` ([Iceberg configuration](https://iceberg.apache.org/docs/latest/configuration/)), those include each string column's smallest value, cut to 16 characters, and its largest, cut to 16 with the last character raised by one, so that it still bounds every value in the file ([source](https://github.com/apache/iceberg-python/blob/pyiceberg-0.12.0/pyiceberg/utils/truncate.py)). The deleted entry in `crm.learners` keeps the email bounds `ALEXA590@POST.EX to zofia.haase@exan`. The first is learner 3590's email, typed in capitals, which sorts before every other email on the platform. The second is learner 4998's, the platform's largest, whose 16th character, an `m`, became an `n`. No learner is safe from this: each data file's entry keeps the smallest and largest value of every column, so a table of many small files names many learners.
**Working rule:** Remove unreferenced files after every expiry, with a sweep of your own built on the inspect tables, since PyIceberg has none, and keep a minimum age of days in production so that a write in progress keeps its files. Use an age of zero only where a single process writes. Don't count a person erased because no data file holds them: the manifests may still hold their values.
## Keep personal values out of manifests {#keep-values-out-of-manifests}
[The sweep](#remove-unreferenced-files) left the learner's email in a manifest, as a column bound. Iceberg lets a table choose, per column, what its manifests keep, with the property `write.metadata.metrics.column.`: `none` keeps nothing, `counts` keeps value and null counts, `truncate(n)` adds bounds cut to n characters, and `full` adds whole bounds ([Iceberg configuration](https://iceberg.apache.org/docs/latest/configuration/)). PyIceberg 0.12.0 looks the property up by the column's name when it writes a file ([source](https://github.com/apache/iceberg-python/blob/pyiceberg-0.12.0/pyiceberg/io/pyarrow.py)).
Typed by hand, those properties would drift from the classifications. `lib.retention.properties` generates them instead, from `governance/metadata.toml` and the `[statistics]` section of `governance/retention.toml`: `none` for every column classified `personal` or `sensitive`, and for every column nobody has classified, the same fail-closed default the access policy uses. It adds the `history.expire` settings from the rules' `[snapshots]` section. `lib.retention.apply_properties` sets them on every table and rewrites each table's data once, because a metrics mode applies only to the files written after it's set; expiry and a sweep then remove the files from before. Erase learner 4998, whose email is the largest bound in the manifest of `crm.learners`, on two platforms, one with default statistics and one with the generated properties:
Example `statistics-off-for-personal-columns` (Python 3.12.15 · duckdb 2.0.0 · openlineage-python 1.54.0 · pyarrow 26.0.0 · pyiceberg 0.12.0 · pyyaml 6.0.3 · sqlglot 30.24.0), passing in Edition 2026.10.1:
```python
"""Generate each table's properties from the classifications, so manifests keep no values of personal columns, then
erase the learner whose email is the platform's largest on a platform with them and on one without."""
from datetime import timedelta
from lib import retention
LEARNER = 4998
for statistics_off in (False, True):
catalog, graph = retention.open_platform(statistics_off)
label = "statistics off for protected columns" if statistics_off else "default statistics"
if statistics_off:
generated = retention.properties(catalog)
turned_off = [key for wanted in generated.values() for key in wanted if key.startswith(retention.METRICS)]
print("columns with statistics turned off:", len(turned_off))
unclassified = [name for name in generated if name not in retention.metadata.load()]
print("of them in tables nobody has classified:",
sum(key.startswith(retention.METRICS) for name in unclassified for key in generated[name]))
print("crm.learners, properties set:", len(generated["crm.learners"]))
for key, value in sorted(generated["crm.learners"].items()):
print(f" {key} = {value}")
entry = catalog.load_table("crm.learners").inspect.entries()["readable_metrics"].to_pylist()[0]
print(f"{label}, crm.learners email bounds:", entry["email"]["lower_bound"], "to", entry["email"]["upper_bound"])
columns = retention.metadata.load()["crm.learners"].columns.values()
personal = [column.name for column in columns if column.classification == "personal"]
print(f"{label}, crm.learners personal columns:", len(personal))
with_bounds = sum(entry[name]["lower_bound"] is not None for name in personal)
print(f"{label}, of them with bounds in the manifest:", with_bounds)
steps, found, _ = retention.erase_everywhere(catalog, graph, LEARNER, "erasure-0002", min_age=timedelta(0))
print(f"{label}, files holding learner {LEARNER} after erasure, expiry and sweep:",
sum(held for _, held in retention.scan(catalog, found).values()))
```
Output:
```text
default statistics, crm.learners email bounds: ALEXA590@POST.EX to zofia.haase@exan
default statistics, crm.learners personal columns: 7
default statistics, of them with bounds in the manifest: 7
default statistics, files holding learner 4998 after erasure, expiry and sweep: 1
columns with statistics turned off: 24
of them in tables nobody has classified: 10
crm.learners, properties set: 9
history.expire.max-snapshot-age-ms = 432000000
history.expire.min-snapshots-to-keep = 1
write.metadata.metrics.column.city = none
write.metadata.metrics.column.date_of_birth = none
write.metadata.metrics.column.email = none
write.metadata.metrics.column.full_name = none
write.metadata.metrics.column.notes = none
write.metadata.metrics.column.phone = none
write.metadata.metrics.column.street_address = none
statistics off for protected columns, crm.learners email bounds: None to None
statistics off for protected columns, crm.learners personal columns: 7
statistics off for protected columns, of them with bounds in the manifest: 0
statistics off for protected columns, files holding learner 4998 after erasure, expiry and sweep: 0
```
The generated properties turn statistics off for 24 columns, 10 of them in `marketing.audience` and `ai.review_corpus`, the two tables that [chapter 2](#test-the-catalog-against-the-metadata) found with no metadata. `crm.learners` gets 9 properties, the two expiry settings and `none` for each personal column. With default statistics, the manifest's email bounds are `ALEXA590@POST.EX to zofia.haase@exan`, and it keeps bounds for 7 of the table's 7 personal columns. After the erasure, expiry and sweep, 1 file still holds learner 4998, through that raised upper bound. With statistics off, the email bounds are `None to None`, the manifest keeps bounds for 0 personal columns, and 0 files hold the learner.
The cost is pruning. Iceberg uses the bounds in manifests to skip files a query's filter can't match ([Iceberg table spec](https://iceberg.apache.org/spec/)), so without them, a query that filters on `email` reads every file. The platform filters and erases by `learner_id`, which is `internal` and keeps its statistics. `counts` is the other choice: it keeps the counts that help a planner, and no values. And because PyIceberg 0.12.0 keeps the property under the column's old name on a rename, leaving the new name without it, the properties are generated again on every schema change, as [chapter 2](#key-column-metadata-by-field-id) asks of every view of the metadata keyed by name.
**Working rule:** Generate per-column metrics properties from your classifications, `none` or `counts` for every personal, sensitive or unclassified column, and apply them when a table is created or, for an existing table, with one rewrite followed by expiry and a sweep. Keep statistics on the ID columns you filter and erase by. Generate the properties again after every schema change.
## Prove the erasure by reading every file {#verify-the-erasure}
[The delete](#erase-the-rows), [the expiry](#expire-snapshots), [the sweep](#remove-unreferenced-files) and [the statistics settings](#keep-values-out-of-manifests) each removed something the step before left behind, and left something of its own, which is the case for checking the result rather than trusting the steps. The European Data Protection Board's report on how controllers handle erasure, adopted in February 2026, found backups, and anonymisation used in place of deletion, among the weak points, and recommends that controllers "Verify that erasure has been carried out and be able to demonstrate such erasure" ([EDPB, 2026](https://www.edpb.europa.eu/system/files/2026-02/edpb_cef-report_2025_right-to-erasure_en.pdf)).
`lib.retention.scan` reads every file under the catalog's warehouse, each with its own format's reader, and looks for the learner's traces, which `lib.retention.traces` noted before the erasure: the IDs of every row the plan deletes, by column, and the first 15 characters, lowercased, of each identifying value. Fifteen, because a manifest keeps a largest value's first 15 characters unchanged and raises the 16th by one, so a search for 16 misses a person whose value is a file's largest, as learner 4998's email is. A Parquet file is read as rows. A manifest or manifest list, both Avro, is read record by record with PyIceberg's Avro reader: every string in it, bounds included, and the bounds of each ID column, decoded by the schema the manifest carries. A metadata file is read as text. In a table an exception keeps, the learner's IDs are expected, so there the scan looks only for values, including those of the columns the exception erased. `lib.retention.erase_everywhere` runs the whole job: plan, note the traces, check every table, erase, expire, sweep. Run it on the platform with the generated properties, then scan:
Example `verify-erasure` (Python 3.12.15 · duckdb 2.0.0 · openlineage-python 1.54.0 · pyarrow 26.0.0 · pyiceberg 0.12.0 · pyyaml 6.0.3 · sqlglot 30.24.0), passing in Edition 2026.10.1:
```python
"""The whole erasure job on the platform with statistics off for protected columns, ending with a scan of every file
under the warehouse. What it prints is the evidence: IDs and counts, never the learner's values."""
from datetime import timedelta
from lib import retention
LEARNER, REQUEST = 3590, "erasure-0001"
catalog, graph = retention.open_platform(statistics_off=True)
steps, found, done = retention.erase_everywhere(catalog, graph, LEARNER, REQUEST, min_age=timedelta(0))
report = retention.scan(catalog, found)
for table, (files, holding) in report.items():
print(f"{table}: {files} files read, {holding} holding the learner")
print("request:", REQUEST)
print("tables visited:", len(steps))
print("rows deleted:", sum(rows for rows in done.values() if rows != "rebuilt"))
print("tables rebuilt:", ", ".join(table for table, rows in done.items() if rows == "rebuilt"))
print("tables kept under an exception, columns erased:",
"; ".join(f"{step.table}, {', '.join(step.cleared)}" for step in steps if step.kept))
print("snapshots a reader can still time travel to:",
sum(len(catalog.load_table(step.table).snapshots()) - 1 for step in steps))
print("files no metadata references:",
sum(len(retention.unreferenced(catalog.load_table(name))) for name in retention.tables(catalog)))
print("tables missing their generated properties:", ", ".join(retention.missing_properties(catalog)))
print("files read:", sum(files for files, _ in report.values()))
print("files holding the learner:", sum(holding for _, holding in report.values()))
```
Output:
```text
ai.review_corpus: 11 files read, 0 holding the learner
analytics.course_performance: 5 files read, 0 holding the learner
analytics.learner_activity: 11 files read, 0 holding the learner
billing.payments: 12 files read, 0 holding the learner
crm.instructors: 8 files read, 0 holding the learner
crm.learners: 11 files read, 0 holding the learner
learning.courses: 8 files read, 0 holding the learner
learning.enrollments: 11 files read, 0 holding the learner
learning.events: 11 files read, 0 holding the learner
learning.reviews: 11 files read, 0 holding the learner
marketing.audience: 8 files read, 0 holding the learner
support.tickets: 8 files read, 0 holding the learner
request: erasure-0001
tables visited: 10
rows deleted: 25
tables rebuilt: analytics.course_performance
tables kept under an exception, columns erased: billing.payments, card_number
snapshots a reader can still time travel to: 0
files no metadata references: 0
tables missing their generated properties: analytics.course_performance
files read: 115
files holding the learner: 0
```
The scan read 115 files under every table's location, and 0 of them hold the learner. No reader can time travel to the learner, since 0 snapshots older than the current ones remain, and 0 files are unreferenced. The lines after the per-table counts are the evidence: the request's ID, 10 tables visited, 25 rows deleted, the table rebuilt, the table an exception keeps and the column erased from it, `billing.payments, card_number`, with the reason in `governance/retention.toml`, and the scan's counts. They name no value of the learner's, so keeping them puts none back into the platform, which [chapter 8](#produce-evidence-as-data) relies on when it writes them to the audit log.
One line is a warning: `analytics.course_performance` lacks its generated properties, because `lib.pipelines.run` dropped the table and created it again without them. That table has no personal columns. The next table a pipeline rebuilds might have them, and the scan would then report its manifests' bounds as a failed erasure, so the check belongs after every pipeline run.
IDs can stay in a manifest too, since the ID columns keep their statistics. An erased ID that was a file's smallest or largest, such as learner 5000's, stays in the deleted entry's bounds, and the scan reports it. One more commit to the table, followed by an expiry and a sweep, removes it, because PyIceberg's next append leaves out a manifest that holds only deleted files ([source](https://github.com/apache/iceberg-python/blob/pyiceberg-0.12.0/pyiceberg/table/update/snapshot.py)).
The scan covers the warehouse only. Exports, backups, the source systems and logs outside it can hold the learner too, and the same traces can be searched for in each of them the platform can read.
**Working rule:** End every erasure with a scan of every file the platform stores, for the person's IDs and identifying values, read with each format's own reader, and keep its result, counts and IDs only, as the evidence. Search for as much of each value as every bound keeps unchanged, which is one character less than the truncation length. Treat any file that still holds the person as a failed erasure, not a warning. Search the stores outside the warehouse that you control, and list in the record the ones you can't search.
## What goes wrong in production {#retention-in-production}
Retention and erasure fail quietly: a delete that succeeds looks the same whether the data is gone or still in storage. The first failure here undoes a finished erasure.
### An erased learner comes back {#erased-learner-comes-back}
Besides the CRM's sync, the warehouse reloads all of `crm.learners` each night from the application's export, and the erasure ran in the warehouse only. Erase learner 3590 everywhere [the plan](#plan-erasure-from-lineage) says, then run the nightly reload and the pipelines that read it:
Example `erased-learner-returns` (Python 3.12.15 · duckdb 2.0.0 · openlineage-python 1.54.0 · pyarrow 26.0.0 · pyiceberg 0.12.0 · pyyaml 6.0.3 · sqlglot 30.24.0), passing in Edition 2026.10.1:
```python
"""An erased learner comes back: the warehouse's copy of the CRM is reloaded each night from the application's
export, which nobody erased, and the pipelines rebuild from it. A check against the erasure register catches it."""
from datetime import timedelta
from lib import pipelines, retention
LEARNER, REQUEST = 3590, "erasure-0001"
catalog, graph = retention.open_platform()
retention.erase_everywhere(catalog, graph, LEARNER, REQUEST, min_age=timedelta(0))
register = [retention.register_entry(REQUEST, LEARNER)] # the request's ID and a keyed hash of the learner's ID
print("the erasure register's entry:", register[0]["request"], register[0]["learner"])
learners = catalog.load_table("crm.learners")
print("the learner's rows in crm.learners after the erasure:", retention.count(learners, retention.rows_of([LEARNER])))
learners.overwrite(retention.crm_export()) # the nightly reload, from the application's export
for name in ("learner_activity", "marketing_audience"):
pipelines.run(catalog, name)
for table in ("crm.learners", "analytics.learner_activity", "marketing.audience"):
print(f"the learner's rows in {table} after the nightly run:",
retention.count(catalog.load_table(table), retention.rows_of([LEARNER])))
for table, rows in retention.erased_rows(catalog, register).items():
print(f"the check against the register, rows of erased learners in {table}:", rows)
```
Output:
```text
the erasure register's entry: erasure-0001 c25d8a8b12426596f0c0b1ebfb890246b42ea535791f07256efd16fc6c2cb84b
the learner's rows in crm.learners after the erasure: 0
the learner's rows in crm.learners after the nightly run: 1
the learner's rows in analytics.learner_activity after the nightly run: 1
the learner's rows in marketing.audience after the nightly run: 0
the check against the register, rows of erased learners in analytics.learner_activity: 1
the check against the register, rows of erased learners in crm.learners: 1
```
After the erasure, `crm.learners` holds 0 rows for the learner. After the nightly run it holds 1 again, and `analytics.learner_activity`, rebuilt from it, holds 1. `marketing.audience` stays clear only because the learner's consent comes back withdrawn and their enrollments, which it joins, were erased.
The check that catches this keeps an erasure register: for each request, its ID and a keyed hash (HMAC-SHA256) of the erased learner's ID, and nothing else. The key is the erasure job's secret, kept like the enforcement layer's pseudonymization key in [chapter 6](#write-masks-as-sql) but a different one, because whoever holds it can hash every learner ID and find whom the register names. After every load, restore and rebuild, `lib.retention.erased_rows` hashes each `learner_id` in every table with that column and looks it up, so a restore or a backfill is checked without the register holding anyone's ID. It finds the learner in `analytics.learner_activity` and `crm.learners`, and leaves out `billing.payments`, whose rows the exception keeps. The audit log's record of the erasure, in [chapter 8](#produce-evidence-as-data), holds the same request ID and counts, never the learner, so the register is the only place a request meets a learner, as a hash. We treat a keyed hash of an ID as pseudonymized data, never anonymized, so whether the platform may keep the register, and for how long, is a question for its lawyers.
**Working rule:** Erase in the source system first, or in the same request, and keep a register of keyed hashes of erased IDs that every load filters against and every run checks, with the key held as a secret apart from the data. Treat a row with an erased ID as an incident, as you would a leak.
### Other failures {#other-retention-failures}
- **Old snapshots and files keep "deleted" data.** A retention or erasure job that stops after its delete leaves everything readable by time travel, and in PyIceberg the files stay after expiry too. Run [the expiry](#expire-snapshots) and [the sweep](#remove-unreferenced-files) in the same job, then the scan.
- **Another engine deletes by merge-on-read.** PyIceberg reads the deletion vectors other engines write ([release notes](https://github.com/apache/iceberg-python/releases/tag/pyiceberg-0.12.0)). Rows deleted that way stay in their data files until compaction rewrites them, and a delete file may keep their values: "Both position and equality delete files allow encoding deleted row values with a delete" ([Iceberg table spec](https://iceberg.apache.org/spec/)). Compact the affected files before [the expiry](#expire-snapshots) and [the sweep](#remove-unreferenced-files), and scan the delete files too.
- **Backups keep everything the sweep deletes.** They are one of the weak points [the EDPB's report](#verify-the-erasure) names. Decide with the people who answer for the backups how they're erased or aged out, and record it with each request.
- **A derived table nobody erased.** A job that sends no lineage writes a copy the plan can't see, as [chapter 4's uninstrumented export](#job-nobody-instrumented) does. Run `lib.lineage.unaccounted` before every erasure.
- **A rebuilt table loses its properties.** A pipeline that drops and creates its table loses the generated statistics settings, and the next erasure leaves values in its manifests. Apply the properties after every run, or create tables with them.
- **A renamed column loses its setting.** In PyIceberg, the metrics property stays under the column's old name, so after a rename the new name has none. Generate the properties again on every schema change.
- **Retention counted from the wrong date.** A rule that means "after closing" but counts from opening deletes early, and one that counts from a load date deletes late. Store the date the rule means.
## What this chapter decided {#retention-decisions}
- **Retention rules are config,** applied against a day the job is given, never the clock. Events are kept 365 days, payments 3,650, and closed tickets 730 days from opening, because the table has no `closed_at`: the platform's choices, not legal advice.
- **Consent is an attribute the policy reads** at query time; a withdrawal deletes nothing, and tables built with a consent filter are rebuilt.
- **An erasure is planned from the catalog and the lineage:** every table with a `learner_id` column, then every table lineage reaches.
- **Exceptions to erasure are the legal team's,** recorded in `governance/retention.toml` with their reasons; the code keeps what they say until the table's retention rule ends it.
- **Commits record request IDs and rule names,** never a person's values.
- **The book's expiry reads each table's settings,** which PyIceberg's doesn't, and an erasure checks every table in its plan before it deletes anything.
- **The book's sweep removes unreferenced files,** with a minimum age of 72 hours in production.
- **Manifests keep no values of protected columns:** metrics mode `none`, generated from the classifications.
- **Every erasure ends with a scan** of every file, for IDs and 15 characters of each identifying value; its counts are the evidence.
- **The erasure register keeps a keyed hash of each erased ID,** checked after every load, restore and rebuild; whether to keep it, and how long, is for the lawyers.
# Audit and evidence
By the end of this chapter you can keep an audit log of every access decision and query, with no values in it, chained so that an edit shows. You'll write it where the decisions are made, and write down its head, the number and hash of its last record, where the log's writer can't change it, so that an edit to, or a removal of, any record up to that head shows. You'll also explain an old decision after the policy has changed, cross-check the log against the query engine's own record of what it ran, and generate access reviews and evidence reports from the log and the policies, as code that anyone can run again.
The platform, the online learning platform this book follows, sells professional courses to adult learners. [Chapter 5](#access-policies-as-code) wrote its access policy in Rego, and [chapter 6](#enforcing-policies-on-queries) built the **enforcement layer**, `lib.enforce.Enforcer`, which asks the policy for a **decision** about every table a query reads and rewrites the query so that DuckDB returns only what the decision allows. The layer forgets each decision as soon as it has applied it. When the support lead asks why an analyst could read review text last week, or an auditor asks the platform to show that its access rules work, nobody can answer from records. And the obvious fix, logging the text of every query, makes a second copy of what the policy protects: a support agent's query can carry a learner's email in its `WHERE` clause, and [chapter 7](#retention-consent-erasure) has to erase a learner from everywhere their data went.
The chapter's terms, as the rest of the book uses them: an **audit log** is a record of who did what, when, and with what outcome, kept so that someone can check it later. The platform's audit log holds one **decision record** per query. A **policy revision** is a fingerprint of everything a decision depended on besides the request. A log is **tamper-evident** when a change to it can be detected, though not prevented. An **access review** compares what each person may read with what they use. **Evidence** is a report that shows a control works, generated from records rather than captured from a screen.
The code is the book's shared module `lib.audit`. Its `Audited` class wraps [chapter 6](#enforcing-policies-on-queries)'s `Enforcer` without changing it: it writes a record for every query, and then lets the layer run the query. Most examples replay one day of the platform's queries, `lib.audit.DAY`: 13 queries on Oct 1, 2026, from a support agent, finance, an instructor through the course assistant, marketing, a data scientist, the assistant on its own account, and one user the platform doesn't know. The examples run DuckDB 2.0.0, sqlglot 30.24.0 and regopy 1.5.2.
## Record every decision where it's made {#record-every-decision}
An audit record is only as complete as what its writer knows. The query engine knows the SQL it ran, but not who asked or why. The policy knows its decision, but not the query. Only the enforcement layer holds all of it at once: the principal, meaning the user or program that asks, the tables and columns the query reads, each table's decision, and the SQL it hands the engine. That is where Open Policy Agent (OPA) writes its own record: "The decision logs contain events that describe policy queries. Each event includes the policy that was queried, the input to the query, bundle metadata, and other information that enables auditing and offline debugging of policy decisions" ([OPA, decision logs](https://www.openpolicyagent.org/docs/management-decision-logs)). Each event carries a `decision_id`, the `path` of the rule queried, the `input`, the `result`, a `timestamp`, and `bundles[_].revision`, the "Revision of the bundle at the time of evaluation". regopy, which runs the platform's policy inside Python, returns the value of the query and nothing more, so the platform's layer has to build every other field itself.
NIST's catalog of security controls gives a checklist for those fields. Control AU-3 asks that audit records "contain information that establishes the following: a. What type of event occurred; b. When the event occurred; c. Where the event occurred; d. Source of the event; e. Outcome of the event; and f. Identity of any individuals, subjects, or objects/entities associated with the event" ([NIST SP 800-53 Rev. 5](https://csrc.nist.gov/pubs/sp/800/53/r5/upd1/final)).
`lib.audit.Audited` takes a catalog and a `Log`, the audit log's file. Its `sql` method takes the same arguments as `Enforcer.sql`, plus `via`, the assistant that sent the query when one did. It plans the query, asks the policy about every table the query reads, writes one record, and only then hands the rewritten query to DuckDB. So no query reaches DuckDB through the layer without a record, and when the record can't be written, the query doesn't run. [Chapter 6](#query-on-someones-behalf) left this chapter one case: when the course assistant answers `instructor-17`, the query runs as the instructor, and both identities belong in the log. The example sends that query, then has the assistant try to put names to the ratings:
Example `decision-record` (Python 3.12.15 · duckdb 2.0.0 · pyarrow 26.0.0 · pyiceberg 0.12.0 · regopy 1.5.2 · sqlglot 30.24.0), passing in Edition 2026.10.1:
```python
from lib import audit
from lib.catalog import open_catalog
from lib.enforce import Denied
from lib.platform import scratch
# Chapter 6's enforcement layer, wrapped so that every query it's given is recorded before DuckDB runs anything.
log = audit.Log(scratch() / "audit.jsonl")
platform = audit.Audited(open_catalog(["crm.learners", "learning.reviews"]), log)
# instructor-17 asks the course assistant how their reviews look. The assistant writes the SQL; the query runs as the
# instructor, with the assistant recorded as the one that sent it.
question = "select course_id, count(*) as reviews, round(avg(rating), 2) as rating from learning.reviews group by all"
platform.sql(question, user="instructor-17", via="assistant-1").fetchall()
# Then the assistant tries to put names to the ratings, which the instructor may not see.
try:
platform.sql("select l.full_name, r.rating from learning.reviews as r join crm.learners as l using (learner_id)",
user="instructor-17", via="assistant-1").fetchall()
except Denied:
pass
for record in log.records():
print(f"record {record['seq']}:")
print(audit.show(record))
```
Output:
```text
record 1:
seq: 1
event: query
time: 2026-10-01T09:00:00Z
catalog: platform
user: instructor-17
via: assistant-1
purpose: null
query: SELECT course_id, COUNT(*) AS reviews, ROUND(AVG(rating), ?) AS rating FROM learning.reviews GROUP BY ALL
outcome: allowed
denial: null
decision: {"table": "learning.reviews", "columns": ["course_id", "rating"], "allow": true, "row_filters": [{"column": "course_id", "in": [2, 75, 80, 87, 103, 134]}], "column_masks": {}, "reasons": ["instructor may read learning.reviews"]}
policy_revision: f761e4ad08d4...
rewritten_digest: 385c5cdabeae...
prev_hash: 000000000000...
record 2:
seq: 2
event: query
time: 2026-10-01T09:01:00Z
catalog: platform
user: instructor-17
via: assistant-1
purpose: null
query: SELECT alias_1.full_name, alias_2.rating FROM learning.reviews AS alias_2 JOIN crm.learners AS alias_1 USING (learner_id)
outcome: denied
denial: instructor-17 may not read crm.learners: no group of this user may read this table
decision: {"table": "crm.learners", "columns": ["full_name", "learner_id"], "allow": false, "row_filters": [], "column_masks": {}, "reasons": ["no group of this user may read this table"]}
decision: {"table": "learning.reviews", "columns": ["learner_id", "rating"], "allow": true, "row_filters": [{"column": "course_id", "in": [2, 75, 80, 87, 103, 134]}], "column_masks": {}, "reasons": ["instructor may read learning.reviews"]}
policy_revision: f761e4ad08d4...
rewritten_digest: null
prev_hash: 0d3e61eaadeb...
```
Record 1 covers AU-3's six items. `event` is the type, and `time` says when, in UTC with a `Z`, since control AU-8 asks for time stamps that "use Coordinated Universal Time" or state their offset; the examples take it from a fixed clock. `catalog` says where. `user` and `via` are the source: the query ran as `instructor-17`, and `assistant-1` sent it. With the decision's `table`, they are the identities item f asks for. `query` and the decision's `columns` say what was read, and `outcome` and the decision's `allow`, `row_filters`, `column_masks` and `reasons` say how it ended, down to the course IDs the row filter kept. The next sections explain `policy_revision`, `rewritten_digest` and `prev_hash`.
Record 2 is a denial, and it holds both tables' decisions: the join may read `learning.reviews` but not `crm.learners`, and one denied table denies the query. Without `via`, the log would say that the instructor tried to read learners' names. With it, a reviewer knows the assistant wrote that query, and looks at the assistant's behaviour as well as the instructor's.
**Working rule:** Write the audit record in the enforcement layer, where the principal, the query and the decision meet, and write it before the query runs, so that a failed write stops the query. Record every identity involved, the person and any assistant acting for them, and every table's decision with its filters, masks and reasons. When an engine enforces the policy itself, as the SQL engine Trino does through OPA, turn on OPA's decision log instead, and make sure its input names every identity involved: Trino's `opa.identity.extra-credentials-keys`, for example, forwards the credentials it lists "to OPA as extraCredentials in the identity context" ([Trino 484, OPA access control](https://trino.io/docs/484/security/opa-access-control.html)).
## Keep values out of the record {#keep-values-out-of-the-record}
The [decision record](#record-every-decision) holds IDs, names of tables and columns, decisions and digests, and no value from the data its query read, because everything an audit log holds becomes data to protect. SP 800-53's discussion of AU-3 warns that "there is the potential to reveal personally identifiable information in the audit trail, especially if the trail records inputs", and its enhancement AU-3(3) asks systems to "Limit personally identifiable information contained in audit records to the following elements identified in the privacy risk assessment: [Assignment: elements]" ([NIST SP 800-53 Rev. 5](https://csrc.nist.gov/pubs/sp/800/53/r5/upd1/final)). Limit, not remove: AU-3 f asks for the principal, so every record names one, such as `instructor-17`, one instructor's account. The GDPR calls personal data "any information relating to an identified or identifiable natural person", identifiable "directly or indirectly, in particular by reference to an identifier such as a name, an identification number, location data, an online identifier" ([GDPR, Article 4(1)](https://eur-lex.europa.eu/eli/reg/2016/679/oj)), and an account ID like this one can be such an identifier. The platform has a sharper reason to keep learners out: [chapter 7](#retention-consent-erasure) erases a learner from every table their data reached, and erasing one record from the [chain](#chain-the-records) this chapter adds can't be told from tampering.
The risk is the query text. A support agent who looks a learner up by the email in a ticket writes that email into the query. `lib.audit.normalized` keeps the query's shape and drops its values: sqlglot parses the query, and every literal, in any of DuckDB's syntaxes, becomes `?`. So does every name the catalog doesn't hold, such as a value typed in double quotes, except that an alias or a CTE's name the query defines becomes `alias_1`, `alias_2` and so on. Comments, which can hold anything, are left out, and a denial's reason is recorded with the same values taken out. The SQL the layer hands DuckDB is kept only as `rewritten_digest`, an HMAC-SHA256 under a key only the layer holds, which the [cross-check](#cross-check-the-query-log) matches against DuckDB's own log. OPA starts from the whole input instead, and "queries the data.system.log.mask path prior to encoding and uploading decision logs" to remove what that policy names ([OPA, decision logs](https://www.openpolicyagent.org/docs/management-decision-logs)). The platform's layer never writes the values at all.
On the platform, `agent-a05` looks up an EMEA learner by email, then by ID, and `analyst-1` reads learners' emails, which the policy pseudonymizes. The example never prints the email or the ID:
Example `query-without-literals` (Python 3.12.15 · duckdb 2.0.0 · pyarrow 26.0.0 · pyiceberg 0.12.0 · regopy 1.5.2 · sqlglot 30.24.0), passing in Edition 2026.10.1:
```python
from lib import audit
from lib.catalog import open_catalog
from lib.platform import load, scratch
log = audit.Log(scratch() / "audit.jsonl")
platform = audit.Audited(open_catalog(["crm.learners"]), log)
# agent-a05 looks up an EMEA learner by the email in a ticket, then by their ID. Neither value is ever printed.
learners = load("crm.learners").to_pylist()
learner = next(row for row in learners if row["support_region"] == "emea")
email, learner_id = learner["email"], learner["learner_id"]
platform.sql(f"select learner_id, full_name from crm.learners where email = '{email}'", "agent-a05").fetchall()
platform.sql(f"select email, phone from crm.learners where learner_id = {learner_id}", "agent-a05").fetchall()
platform.sql("select learner_id, email from crm.learners", "analyst-1").fetchall() # pseudonymized with the key
for record in log.records():
print("query in the record:", record["query"])
engine = audit.engine_queries(platform.enforcer.connection)
print("the email, in the audit log:", email in log.path.read_text())
print("the email, in DuckDB's query log:", any(email in query for query in engine))
print("the pseudonymization key, in DuckDB's query log:", any(platform.enforcer.key in query for query in engine))
print("its placeholder, in DuckDB's query log:", any("$pseudonym_key" in query for query in engine))
# Guessing which learner the agent looked up, from a list of the platform's emails: rebuild the text DuckDB ran with
# each email, digest it, and compare. First with a plain SHA-256 of the query, as a log might keep, then with the
# record's keyed digest, by someone without the layer's key and by someone who holds it.
ran = next(query for query in engine if email in query)
guesses = [ran.replace(email, row["email"]) for row in learners]
logged = log.records()[0]["rewritten_digest"]
print("emails tried:", f"{len(guesses):,}")
print("emails a plain SHA-256 of the query confirms:", sum(audit.sha256(g) == audit.sha256(ran) for g in guesses))
print("emails the keyed digest confirms, without the layer's key:",
sum(audit.digest(g, "a-guessed-key") == logged for g in guesses))
print("emails the keyed digest confirms, with the layer's key:", sum(audit.digest(g, platform.key) == logged for g in guesses))
```
Output:
```text
query in the record: SELECT learner_id, full_name FROM crm.learners WHERE email = ?
query in the record: SELECT email, phone FROM crm.learners WHERE learner_id = ?
query in the record: SELECT learner_id, email FROM crm.learners
the email, in the audit log: False
the email, in DuckDB's query log: True
the pseudonymization key, in DuckDB's query log: False
its placeholder, in DuckDB's query log: True
emails tried: 5,000
emails a plain SHA-256 of the query confirms: 1
emails the keyed digest confirms, without the layer's key: 0
emails the keyed digest confirms, with the layer's key: 1
```
The records keep `WHERE email = ?` and `WHERE learner_id = ?`, and a search of the audit log for the email finds nothing (`False`). DuckDB's query log does hold it (`True`), because that log records the SQL text DuckDB runs, literals included. The pseudonymization key isn't there (`False`): [chapter 6](#write-masks-as-sql) passes the key as a parameter, so DuckDB's log shows the placeholder `$pseudonym_key` where the key would be.
The last lines show what the key does. The rewrite is code in the repository, so anyone with a list of emails, from an old export or a breach, can rebuild the text DuckDB ran for each one. Of 5,000 emails tried, a plain SHA-256 of that text confirms 1: the learner the agent looked up. The record's keyed digest confirms 0 without the layer's key, and 1 with it. So a keyed digest of a query that held an email is a pseudonym of that email, and the key is what keeps it one, as [chapter 6](#write-masks-as-sql) showed for the platform's pseudonyms. The key stays with the layer, away from anyone who reads the log, and so does the cross-check, which needs it.
DuckDB takes a value in more ways than a quoted string, and a name can hold one too. The second example, a regression test for `normalized`, types the same learner's email, name and ID into `agent-a05`'s queries in each of those ways:
Example `query-shape-hard-cases` (Python 3.12.15 · duckdb 2.0.0 · pyarrow 26.0.0 · pyiceberg 0.12.0 · regopy 1.5.2 · sqlglot 30.24.0), passing in Edition 2026.10.1:
```python
import json
import re
from lib import audit
from lib.catalog import open_catalog
from lib.enforce import Denied
from lib.platform import load, scratch
log = audit.Log(scratch() / "audit.jsonl")
platform = audit.Audited(open_catalog(["crm.learners"]), log)
# An EMEA learner's email, name and ID, typed into agent-a05's queries as values and as names. None is ever printed.
learner = next(row for row in load("crm.learners").to_pylist() if row["support_region"] == "emea")
email, name, learner_id = learner["email"], learner["full_name"], learner["learner_id"]
cases = {
"a string": f"select learner_id from crm.learners where email = '{email}'",
"a dollar-quoted string": f"select learner_id from crm.learners where email = $${email}$$",
"an escape string": f"select learner_id from crm.learners where email = e'{email}'",
"a list": f"select learner_id from crm.learners where email in ['{email}']",
"a number": f"select email from crm.learners where learner_id = {learner_id}",
"a comment": f"select email from crm.learners where learner_id = {learner_id} -- {name}",
"a name in double quotes": f'select learner_id from crm.learners where full_name = "{name}"',
"a column alias": f'select learner_id as "{email}" from crm.learners',
"a table alias": f'select "{name}".email from crm.learners as "{name}"',
"a CTE's name": f'with "{email}" as (select learner_id from crm.learners) select learner_id from "{email}"',
"a table the catalog doesn't hold": f'select learner_id from "{email}"',
}
for query in cases.values():
try:
platform.sql(query, "agent-a05").fetchall()
except Denied:
pass
# What each record keeps of its query. The email or the name anywhere in its line, or the ID in what it keeps, is a
# value that got through, and is never printed.
held = []
for case, line in zip(cases, log.lines()):
record = json.loads(line)
kept = record["query"] + (f" (denied: {record['denial']})" if record["denial"] else "")
held.append(email in line or name in line or bool(re.search(rf"\b{learner_id}\b", kept)))
print(f"{case}:", "(holds a value)" if held[-1] else kept)
print("records holding the email, the name or the ID:", sum(held), "of", len(cases))
```
Output:
```text
a string: SELECT learner_id FROM crm.learners WHERE email = ?
a dollar-quoted string: SELECT learner_id FROM crm.learners WHERE email = ?
an escape string: SELECT learner_id FROM crm.learners WHERE email = ?
a list: SELECT learner_id FROM crm.learners WHERE email IN (?)
a number: SELECT email FROM crm.learners WHERE learner_id = ?
a comment: SELECT email FROM crm.learners WHERE learner_id = ?
a name in double quotes: SELECT learner_id FROM crm.learners WHERE full_name = ? (denied: Column '?' could not be resolved)
a column alias: SELECT learner_id AS alias_1 FROM crm.learners
a table alias: SELECT alias_1.email FROM crm.learners AS alias_1
a CTE's name: WITH alias_1 AS (SELECT learner_id FROM crm.learners) SELECT learner_id FROM alias_1
a table the catalog doesn't hold: SELECT learner_id FROM ? (denied: not a table in the catalog: "?")
records holding the email, the name or the ID: 0 of 11
```
In allowed and denied queries alike, each record keeps the query's shape and none of its values. The last line counts the records that hold the email, the name or the ID: 0 of 11.
Two things are lost. The normalized text drops harmless values and names too: in the [first record](#record-every-decision), `ROUND(AVG(rating), 2)` became `ROUND(AVG(rating), ?)`, and in the second, the aliases `r` and `l` became `alias_2` and `alias_1`. And the log can't answer "who looked up this learner?". When the platform needs that answer, for a learner who asks who has read their data, keep it in a separate access log, with learner IDs, a retention period and no chain, which [chapter 7](#retention-consent-erasure)'s erasure reaches like any other table.
**Working rule:** Record a query's shape, never its values: replace every literal, in every syntax the engine accepts, and every name the catalog doesn't hold, drop comments, take the same values out of denial reasons, and keep the executed SQL only as a keyed digest whose key the log's readers never get. Never use a plain hash of text that may hold personal data. When you need to know who read one person's data, keep that in a separate log that retention and erasure reach, not in the tamper-evident one.
## Replay an old decision under its own policy {#replay-old-decisions}
A [decision record](#record-every-decision) holds the request, but a decision depends on three things: the request, the policy's rules, and the data the policy reads. The last two change all the time. The platform's policy reads `governance/users.toml`, `governance/metadata.toml` and `learning.courses`, as [chapter 5](#generate-the-data-the-policy-reads) set up, so a support agent moved to another region, or a course added to `learning.courses`, changes decisions without anyone touching Rego. If a record says nothing about what the policy was, its decision can't be explained once either changes: replayed later, it gives a different answer, and the log looks wrong. OPA records the policy's version as `bundles[_].revision`. For policies that call non-deterministic built-ins, such as the time or an HTTP request, OPA can also log their results in `nd_builtin_cache`, "Intended for use in debugging and decision replay" ([OPA, decision logs](https://www.openpolicyagent.org/docs/management-decision-logs)). The platform's policy calls none.
`lib.audit.bundle` puts together the policy's modules and its data document, the facts the policy reads, generated from those three sources, as OPA ships them in a bundle. `lib.audit.revision` is the SHA-256 of that bundle: a fingerprint that changes when any rule changes, or any fact the policy reads. Every record carries the revision that decided it. `lib.audit.Archive` keeps every bundle the layer has used, under its revision, so a decision can be made again under the policy that made it. A bundle holds rules and the attributes of the platform's users, not data about learners, so the archive can be kept as long as the log.
On the platform, `analyst-1` reads learners' emails and gets pseudonyms. Later, a pull request adds `analyst-1` to support for EMEA, the change [chapter 6](#test-who-sees-what) caught with its who-sees-what test. Make that change, then replay the old record under each revision:
Example `explain-old-decision` (Python 3.12.15 · duckdb 2.0.0 · pyarrow 26.0.0 · pyiceberg 0.12.0 · regopy 1.5.2 · sqlglot 30.24.0), passing in Edition 2026.10.1:
```python
from lib import audit, policy
from lib.catalog import open_catalog
from lib.platform import scratch
folder = scratch()
archive = audit.Archive(folder / "bundles") # every policy revision that has decided anything, by revision
log = audit.Log(folder / "audit.jsonl")
# analyst-1 reads learners' emails, which the policy pseudonymizes.
platform = audit.Audited(open_catalog(["crm.learners"]), log, archive=archive)
platform.sql("select learner_id, email from crm.learners", "analyst-1").fetchall()
record = log.records()[0]
# Later, a pull request adds analyst-1 to support for EMEA, as in chapter 6. Only users.toml changes.
data = policy.data_document()
data["platform"]["users"]["analyst-1"] = {"groups": ["data-science", "support"], "region": "emea"}
current, logged = audit.bundle(data=data), archive.get(record["policy_revision"])
print("logged under revision:", record["policy_revision"][:12])
print("current revision:", audit.revision(current)[:12])
print("policy modules changed:", current["modules"] != logged["modules"])
print("data document changed:", current["data"] != logged["data"])
# Explain the logged decision by making it again, from the request the record holds, under each revision.
print("email, as logged:", record["decisions"][0]["column_masks"].get("email", "shown as stored"))
for name, revision in [("the current revision", current), ("the logged revision", logged)]:
again = audit.replay(record, revision)
print(f"under {name}, email:", again[0]["column_masks"].get("email", "shown as stored"))
print(f"under {name}, the decision matches the log:", again == record["decisions"])
```
Output:
```text
logged under revision: f761e4ad08d4
current revision: 0350e789b34d
policy modules changed: False
data document changed: True
email, as logged: pseudonymize
under the current revision, email: shown as stored
under the current revision, the decision matches the log: False
under the logged revision, email: pseudonymize
under the logged revision, the decision matches the log: True
```
The revision changed, from `f761e4ad08d4` to `0350e789b34d`, though no Rego did: asked whether the policy modules changed, the example answers `False`, and whether the data document changed, `True`. A fingerprint of the policy files alone would have missed the change. Replayed under the current revision, the decision no longer matches the log: `email` is `shown as stored`, because support sees contact details. An investigator holding only the current policy would conclude that the log, or the layer, was wrong. Replayed under the logged revision from the archive, `email` is `pseudonymize` again, and the decision matches the log: `True`.
Replay explains the decision, not the rows. A row filter is replayed as a rule, such as support's region or the consent filter `learner_consented`, and which rows passed it depended on the data that day, which may have changed since, or been erased. That split is by design: the log says what the rule was, and the tables, with their own snapshots and retention, say what it matched.
**Working rule:** Fingerprint every decision with a hash of everything it depended on besides the request: the policy's modules and its data document, together. Archive each revision for as long as you keep the log, and explain an old decision by replaying it under its own revision, never under the current one. If a policy reads the clock or calls another service, log those results too, as OPA's `nd_builtin_cache` does.
## Chain the records so an edit shows {#chain-the-records}
The platform's [decision records](#record-every-decision) are evidence only if a change to them can be detected. SP 800-53 asks systems to "Protect audit information and audit logging tools from unauthorized access, modification, and deletion" (AU-9), and, in AU-9(3), to "Implement cryptographic mechanisms to protect the integrity of audit information and audit tools". Its discussion of AU-9(3) gives as an example "signed hash functions using asymmetric cryptography" ([NIST SP 800-53 Rev. 5](https://csrc.nist.gov/pubs/sp/800/53/r5/upd1/final)), which only the holder of a private key can make. The platform's chain, below, is not that, and protects nothing on its own. NIST's guide to log management, SP 800-92, describes the plainest mechanism: "Log file integrity checking involves calculating a message digest for each file and storing the message digest securely to ensure that changes to archived logs are detected" ([NIST SP 800-92](https://doi.org/10.6028/NIST.SP.800-92)). The guide dates from 2006. It names MD5 and SHA-1 as the common digests of its day, and its own footnote 25 advises SHA-256, which the platform uses.
A digest per file detects a change to a closed file. A **hash chain** detects a change to a log that's still growing: every record carries a sequence number, `seq`, and the SHA-256 of the line before it, `prev_hash`, so changing a record breaks the link to the next one, and removing one leaves a gap. `lib.audit.check` walks the chain and reports each break. A chain is tamper-evident, not tamper-proof. Schneier and Kelsey designed a log in which an attacker who takes over the logging machine can neither read nor falsify the entries written before. Even there, the attacker "can delete a block of entries (or the entire log file)", and the trusted party the log reports to notices only the next time the machine contacts it ([Schneier and Kelsey, 1999](https://www.schneier.com/wp-content/uploads/2016/02/paper-auditlogs.pdf)). A plain chain like the platform's is weaker: anyone who can write the file can rebuild it, renumbering as they go. So what it proves depends on its **head**, the number of records and the hash of the last one, written down where the log's writer can't change it. An edit to, or a removal of, any record up to that head shows, however it's hidden. After the head, a chain rebuilt or cut passes every check, and without a head, so does a chain rebuilt from the start. `check` takes the head as an anchor.
The example replays `lib.audit.DAY`, the 13 queries the platform received on Oct 1, and writes the head down at noon, at record 9. Then it tampers with copies of the log. Before noon, record 3, `agent-a05`'s lookup of a learner's contact details, is handed to another agent, once as a plain edit and once with the chain rebuilt. After noon, record 12, the denied read by `someone-else`, the user the platform doesn't know, is removed, once plainly and once with the chain rebuilt; and the day's last two records are cut off. Before you run it, predict which of the five the chain finds on its own, and which against the noon head.
Example `chain-catches-edit` (Python 3.12.15 · duckdb 2.0.0 · pyarrow 26.0.0 · pyiceberg 0.12.0 · regopy 1.5.2 · sqlglot 30.24.0), passing in Edition 2026.10.1:
```python
import json
from lib import audit
from lib.catalog import open_catalog
from lib.platform import scratch
log = audit.Log(scratch() / "audit.jsonl")
platform = audit.Audited(open_catalog(["crm.learners", "learning.reviews", "billing.payments", "support.tickets"]), log)
audit.run_day(platform, [query for query in audit.DAY if query[0] < "12:00"]) # the morning
anchor = log.head() # the head, written down at noon somewhere the log's writer can't reach
audit.run_day(platform, [query for query in audit.DAY if query[0] >= "12:00"]) # the afternoon
lines = log.lines()
print("records:", len(lines))
print("head written down at noon, at record:", anchor["records"])
print("record 3:", json.loads(lines[2])["user"], "reads", json.loads(lines[2])["query"])
def rechained(lines: list[str]) -> list[str]:
"""The chain built again, every record renumbered and every prev_hash recomputed: what anyone who can rewrite the
log's file can do."""
rebuilt, previous = [], audit.GENESIS
for seq, line in enumerate(lines, start=1):
rebuilt.append(audit.canonical(json.loads(line) | {"seq": seq, "prev_hash": previous}))
previous = audit.sha256(rebuilt[-1])
return rebuilt
# Before noon, record 3 handed to another agent. After noon, record 12 (someone-else's denied read) removed, and the
# day's last two records cut off.
edited = lines[:2] + [audit.canonical(json.loads(lines[2]) | {"user": "agent-a01"})] + lines[3:]
removed = lines[:11] + lines[12:]
for name, version in [("untouched", lines), ("record 3 edited", edited), ("record 3 edited, chain rebuilt", rechained(edited)),
("record 12 removed", removed), ("record 12 removed, chain rebuilt", rechained(removed)),
("last 2 records removed", lines[:-2])]:
print(f"{name}, checked alone:", "; ".join(audit.check(version)) or "no problem found")
print(f"{name}, checked against the noon head:", "; ".join(audit.check(version, anchor)) or "no problem found")
```
Output:
```text
records: 13
head written down at noon, at record: 9
record 3: agent-a05 reads SELECT full_name, email, phone FROM crm.learners WHERE learner_id = ?
untouched, checked alone: no problem found
untouched, checked against the noon head: no problem found
record 3 edited, checked alone: record 3 was changed: record 4 doesn't follow it
record 3 edited, checked against the noon head: record 3 was changed: record 4 doesn't follow it
record 3 edited, chain rebuilt, checked alone: no problem found
record 3 edited, chain rebuilt, checked against the noon head: the chain no longer passes through the head written down at record 9
record 12 removed, checked alone: record 12 is missing: record 13 follows record 11
record 12 removed, checked against the noon head: record 12 is missing: record 13 follows record 11
record 12 removed, chain rebuilt, checked alone: no problem found
record 12 removed, chain rebuilt, checked against the noon head: no problem found
last 2 records removed, checked alone: no problem found
last 2 records removed, checked against the noon head: no problem found
```
On its own, the chain finds the plain edit, `record 3 was changed: record 4 doesn't follow it`, and the plain removal, `record 12 is missing: record 13 follows record 11`. It misses both rebuilt chains, which are consistent from start to end, and the cut, since no record follows the last one to notice. Against the noon head, it also finds the chain rebuilt before noon: `the chain no longer passes through the head written down at record 9`. Everything hidden after noon passes: against the noon head, the removal with the chain rebuilt gives `no problem found`, and the cut gives `no problem found`. A head protects only the records written before it.
This is why the [rule about values](#keep-values-out-of-the-record) isn't optional. Erasing a learner from a chained log means changing or removing a record, and wherever a check can see that, it can't tell it from tampering. A log that holds no data about learners never meets [chapter 7](#retention-consent-erasure)'s erasure. The IDs of the people who query are another matter: give the log a retention period, and start a new chain each period, with its last head written down, so that a whole period can be deleted when it expires.
Checking a chain means re-reading every record. Crosby and Wallach describe tree-shaped logs that can prove "that individual logged events are still present, and that the log, as seen now, is consistent with how it was seen in the past", with proofs "with logarithmic size and space, improving over previous linear constructions" ([Crosby and Wallach, 2009](https://www.usenix.org/legacy/event/sec09/tech/full_papers/crosby.pdf)). As a rule of thumb, we recommend a chain checked daily for one platform's own log, and a tree when outside parties audit a log too large to re-read.
**Working rule:** Chain every record to the one before with SHA-256 and a sequence number, and count on the chain to protect only the records up to the last head written down outside the layer's control. Write the head down often, to an append-only store or to another system under other credentials, as often as the number of unprotected records you can accept; if that number is zero, send each record off the machine as it's written. Check the chain against the last head on a schedule. If you need what AU-9(3) describes, have each head signed by a system whose key the layer never holds. Never chain a record that might one day have to be erased for one person; let whole periods expire instead.
## Cross-check the log against DuckDB's query log {#cross-check-the-query-log}
The platform's [audit log](#record-every-decision) records every query that goes through the enforcement layer, and only those. A query run on the layer's DuckDB connection directly, by a notebook helper or a test that was handed the connection, gets no decision and no record. Noticing it takes a second record, kept by the engine. DuckDB 2.0.0 has one, off by default: its logging framework includes a `QueryLog` type, which "Logs which queries are executed in DuckDB", turned on with `CALL enable_logging('QueryLog')` and read from the `duckdb_logs` view ([DuckDB, logging](https://duckdb.org/docs/2.0/operations_manual/logging/overview.html)). A separate setting, `log_query_path`, "Specifies the path to which queries should be logged" in a file ([DuckDB, configuration](https://duckdb.org/docs/2.0/configuration/overview.html)), which the layer's connection refuses, as the [section on production](#audit-in-production) shows.
DuckDB's log isn't an audit log. Each entry holds the SQL text, with connection, transaction and query IDs and a timestamp: no user, and no decision. It sees only the rewritten query, and it belongs to the process it records. What it can do is check the audit log: every query DuckDB ran should be the SQL that some decision record handed it. `Audited` turns DuckDB's log on and locks the connection's configuration when it starts. `Audited.check_engine_log` matches each query in DuckDB's log, by keyed digest, against the `rewritten_digest` of the records written since the last check, then empties DuckDB's log, which holds the queries' values. A relation's query runs "when a method of the Output section reads the relation, and it runs again on every read" ([DuckDB, relational API](https://duckdb.org/docs/2.0/clients/python/relational_api)), and a narrowed relation runs SQL of its own, which no record handed DuckDB. So `Audited.sql` fetches each result itself, once, and hands the rows back as a relation on a connection of its own: whatever the caller does with it, DuckDB's log holds each allowed query once.
On the platform, replay `lib.audit.DAY`, the day's 13 queries, and check. Then send two more queries through the layer, one its caller never fetches and one it narrows to five rows, read a learner's email through the layer's connection, around the layer, and check again:
Example `query-log-cross-check` (Python 3.12.15 · duckdb 2.0.0 · pyarrow 26.0.0 · pyiceberg 0.12.0 · regopy 1.5.2 · sqlglot 30.24.0), passing in Edition 2026.10.1:
```python
from lib import audit
from lib.catalog import open_catalog
from lib.platform import scratch
log = audit.Log(scratch() / "audit.jsonl")
platform = audit.Audited(open_catalog(["crm.learners", "learning.reviews", "billing.payments", "support.tickets"]), log)
day = audit.run_day(platform)
print("queries allowed:", day["allowed"])
print("queries denied:", day["denied"])
# DuckDB's own query log, kept on the layer's connection, against the records written since the last check. The check
# then empties DuckDB's log, which holds the queries' values.
found = platform.check_engine_log()
print("queries DuckDB ran:", found["engine queries"])
print("queries DuckDB ran with no decision record:", len(found["unrecorded"]))
# Two more queries through the layer: one whose caller never fetches it, and one the caller narrows to five rows.
platform.sql("select learner_id, country from crm.learners", "analyst-1")
platform.sql("select learner_id, country from crm.learners", "analyst-1").limit(5).fetchall()
# Then a notebook helper that was handed the layer's connection reads a learner's email with it, around the layer.
platform.enforcer.connection.sql("select email from crm.learners where learner_id = 541").fetchall()
found = platform.check_engine_log()
print("next check, records checked:", " to ".join(map(str, found["records checked"])))
print("next check, queries with no decision record:", len(found["unrecorded"]))
for query in found["unrecorded"]:
print("unrecorded:", query)
print("next check, decision records DuckDB has no query for:", len(found["not in the engine log"]))
```
Output:
```text
queries allowed: 9
queries denied: 4
queries DuckDB ran: 9
queries DuckDB ran with no decision record: 0
next check, records checked: 14 to 15
next check, queries with no decision record: 1
unrecorded: SELECT email FROM crm.learners WHERE learner_id = ?
next check, decision records DuckDB has no query for: 0
```
Of the day's queries, 9 were allowed and 4 denied, and DuckDB ran 9: one for each allowed query, none for the denied ones, and 0 without a record. The next check covers records 14 to 15, the two queries through the layer, and neither raises an alarm. The read around the layer shows up as 1 query with no decision record, printed without its values. The last line checks the other direction, records whose query DuckDB has no trace of: 0 here.
What the check catches depends on who holds the connection. It catches a read around the layer by code that leaves DuckDB's log alone, such as this notebook helper, a test, or a job handed the connection by mistake. It doesn't catch code that holds the connection and means to hide. A read made after a check, with DuckDB's log emptied again before the layer's next query, leaves nothing for either direction to find. And DuckDB's `write_log` function takes the log type as an argument, so it can write `QueryLog` entries ([DuckDB source, v2.0.0](https://github.com/duckdb/duckdb/blob/v2.0.0/src/function/scalar/system/write_log.cpp)); the configuration lock doesn't stop a function. A guarantee against such code needs the engine's log written outside the process that holds the connection.
The check also finds the layer's own shortcuts, such as [chapter 6](#test-who-sees-what)'s `observe`, which reads the stored table on the layer's connection with no user behind it. Give such queries records of their own, with the layer as the user, rather than exceptions in the check.
**Working rule:** Turn on the engine's own query log wherever you enforce policies, fetch every result inside the layer, and check the log on a schedule against the records written since the last check, in both directions: every query the engine ran must match a record, and every allowed record must match a query. Then empty it, since it keeps the queries' values. Treat a mismatch as an incident to explain, not noise to filter. Count on the check to catch mistakes, not code that holds the connection and means to hide: against that, only a log written outside the process helps. Never use the engine's log as the audit log: it has no user and no decision.
## Generate the access review from the log {#generate-the-access-review}
The platform's [audit log](#record-every-decision) says what each person read, and the policy says what each person may read; an access review puts the two side by side. SP 800-53 asks organizations to "Review accounts for compliance with account management requirements" (AC-2 j), and to "Review [Assignment: frequency] the privileges assigned to [Assignment: roles and classes] to validate the need for such privileges" and "Reassign or remove privileges, if necessary" (AC-6(7)) ([NIST SP 800-53 Rev. 5](https://csrc.nist.gov/pubs/sp/800/53/r5/upd1/final)). A list of who holds which role answers neither, because it says nothing about need. Code can compute the facts a review needs: what the policy allows each person, what the log shows they used, what they were denied, and memberships that point at nothing. Whether to remove access stays a decision for the data's owners.
`lib.audit.review` evaluates the platform's policy for every user in `users.toml` against every table, reads the log, and lists for each user the tables they may read, the tables they read, the grants they didn't use, their denied queries with the assistant that sent them, and anything someone should look at. To give it something stale to find, the example adds `analyst-2` to the users, still listed in `bi`, a group the policy no longer has, as happens when a group is retired and nobody edits the users file:
Example `access-review` (Python 3.12.15 · duckdb 2.0.0 · pyarrow 26.0.0 · pyiceberg 0.12.0 · regopy 1.5.2 · sqlglot 30.24.0), passing in Edition 2026.10.1:
```python
from lib import audit, policy
from lib.catalog import open_catalog
from lib.platform import TABLES, scratch
log = audit.Log(scratch() / "audit.jsonl")
platform = audit.Audited(open_catalog(["crm.learners", "learning.reviews", "billing.payments", "support.tickets"]), log)
audit.run_day(platform)
# users.toml as the review finds it: analyst-2 is still listed in bi, a group the policy no longer has.
data = policy.data_document()
data["platform"]["users"]["analyst-2"] = {"groups": ["bi"]}
rows = audit.review(audit.bundle(data=data), log.records(), TABLES) # all eight of the platform's tables
for row in rows:
print(f"{row['user']}: may read {len(row['may read'])}, read {len(row['read'])}, denied {row['denied']}"
+ (f" (sent by {', '.join(row['denied via'])})" if row["denied via"] else ""))
if row["unused"]:
print(" not used:", ", ".join(row["unused"]))
for finding in row["findings"]:
print(" look at:", finding)
print("users reviewed:", len(rows))
print("grants not used:", sum(len(row["unused"]) for row in rows))
print("users to look at:", sum(bool(row["findings"]) for row in rows))
print("denied queries flagged at:", audit.settings()["review"]["denied_queries_flagged"])
```
Output:
```text
agent-a01: may read 2, read 0, denied 0
not used: crm.learners, support.tickets
agent-a05: may read 2, read 2, denied 0
agent-a09: may read 2, read 0, denied 0
not used: crm.learners, support.tickets
analyst-1: may read 6, read 2, denied 0
not used: billing.payments, learning.courses, learning.enrollments, learning.events
analyst-2: may read 0, read 0, denied 0
look at: in group bi, which the policy doesn't know
assistant-1: may read 2, read 1, denied 0
not used: learning.courses
finance-1: may read 3, read 1, denied 0
not used: learning.courses, learning.enrollments
instructor-17: may read 3, read 1, denied 3 (sent by assistant-1)
not used: learning.courses, learning.enrollments
look at: 3 denied queries
marketing-1: may read 2, read 1, denied 0
not used: learning.courses
someone-else: may read 0, read 0, denied 1
look at: not in users.toml
users reviewed: 10
grants not used: 14
users to look at: 3
denied queries flagged at: 3
```
Over Oct 1, the day `lib.audit.DAY` replays, 14 grants went unused, and `agent-a01` and `agent-a09` read nothing at all. One day is too short to remove anyone's access: support works in shifts, and finance reads some tables only when it closes the month. Over a review period, such as a quarter, a grant unused for the whole period is what to bring to the table's owner. 3 users need a person to look at them:
- `analyst-2` is in a group no rule knows. The membership grants nothing today, and would grant whatever a future rule for `bi` allows, so remove it.
- `someone-else` isn't in `users.toml`, so a principal the platform doesn't know tried to read learners' emails.
- `instructor-17`'s line reads `may read 3, read 1, denied 3 (sent by assistant-1)`. In the two minutes after answering the instructor's first question, the assistant tried learners, then learners joined to reviews, then tickets. That is an assistant trying tables one by one, and without `via` the review would have blamed the instructor.
The threshold for flagging, 3 denied queries, is a rule of thumb, kept in `governance/audit.toml` with its reasoning, so that changing it is a reviewed change like any change to the policy.
**Working rule:** Generate access reviews from the policy and the audit log, never from a list of role memberships: what each person may read against what they read, their denials, and memberships the policy no longer knows. Run them over a period long enough for every job's rhythm, such as a quarter, and send each finding to the owner of the table or group it concerns. Remove a grant nobody used in the period unless its owner says why it's needed.
## Produce evidence as data {#produce-evidence-as-data}
The [chain check](#chain-the-records), the [cross-check](#cross-check-the-query-log) and the [access review](#generate-the-access-review) are worth most when someone outside the team can check them. The EU's General Data Protection Regulation (GDPR) puts that duty on the organization, as its accountability principle: "The controller shall be responsible for, and be able to demonstrate compliance with, paragraph 1 (‘accountability’)" ([GDPR, Article 5(2)](https://eur-lex.europa.eu/eli/reg/2016/679/oj)). What counts as demonstrating is a question for lawyers; this section shows what code can produce. NIST's assessment procedures for account management list records to examine, among them "access authorization records", "account management compliance reviews" and "system audit records" ([NIST SP 800-53A Rev. 5, in NIST's OSCAL catalog](https://github.com/usnistgov/oscal-content/blob/main/nist.gov/SP800-53/rev5/json/NIST_SP-800-53_rev5_catalog.json)). A screenshot of a console is none of these: nobody can run it again, compare it with last quarter's, or tell which policy it showed.
`lib.audit.evidence` builds the platform's report from records and code, in these sections:
- the policy, with the digest of each module and of the data document;
- the policy's unit tests, from [chapter 5](#unit-test-the-policy-in-rego);
- the who-sees-what table, checked against the policy;
- classification coverage, from [chapter 2](#test-the-catalog-against-the-metadata)'s metadata checks;
- the audit log's chain check;
- the cross-check against DuckDB's log;
- the erasures the log records;
- the access review.
Every section names the policy revision it was computed under, the digests of its inputs, and what anyone needs to compute it again: the archived bundle, the test files, `metadata.toml` and the tables' schemas, the audit log. The engine cross-check can be computed again only in part, and says so. It keeps the keyed digest of every query DuckDB ran, so anyone holding the audit log can redo its two counts, but only the layer could read DuckDB's log, once, before emptying it. Generate the report for the platform on Oct 1, with every table and pipeline output in the catalog, [chapter 7](#retention-consent-erasure)'s erasure of a learner logged at 07:00, and the day's queries:
Example `evidence-report` (Python 3.12.15 · duckdb 2.0.0 · openlineage-python 1.54.0 · pyarrow 26.0.0 · pyiceberg 0.12.0 · pyyaml 6.0.3 · regopy 1.5.2 · sqlglot 30.24.0), passing in Edition 2026.10.1:
```python
import json
from datetime import timedelta
from lib import audit, retention
from lib.platform import scratch
# The platform on Oct 1: every table and pipeline output, chapter 7's erasure of learner 3590 at 07:00, then the day's
# queries through the audited layer.
catalog, graph = retention.open_platform(statistics_off=True)
log = audit.Log(scratch() / "audit.jsonl")
_, found, done = retention.erase_everywhere(catalog, graph, 3590, "erasure-0001", min_age=timedelta(0))
audit.record_erasure(log, "erasure-0001", done, retention.scan(catalog, found), at=audit.MIDNIGHT.replace(hour=7))
platform = audit.Audited(catalog, log)
audit.run_day(platform)
# The first report, so there's no head from an earlier one to check the chain against.
report = audit.evidence(platform, catalog, anchor=None)
reports = scratch() # in production, another system, which the log's writer can't change
(reports / "evidence-2026-10-01.json").write_text(json.dumps(report, indent=2, sort_keys=True), encoding="utf-8")
result = {name: section["result"] for name, section in report.items()}
print("sections:", ", ".join(report))
print("policy tests passed:", f"{result['policy tests']['passed']} of {result['policy tests']['of']}")
print("who-sees-what cases the policy agrees with:", f"{result['who sees what']['agree']} of {result['who sees what']['of']}")
for problem in result["classification coverage"]["problems"]:
print("classification problem:", problem)
print("audit log records:", result["audit log"]["records"])
print("chain problems, checked alone:", len(result["audit log"]["chain problems"]))
print("queries DuckDB ran with no decision record:", len(result["engine cross-check"]["unrecorded"]))
print("decision records DuckDB has no query for:", len(result["engine cross-check"]["not in the engine log"]))
for erasure in result["erasures"]:
print(f"erasure {erasure['request']}: {erasure['rows_deleted']} rows deleted, "
f"{erasure['files_holding_the_person']} of {erasure['files_read']} files hold the person")
for kept in erasure["kept"]:
print(f"erasure {erasure['request']}, kept under an exception: {kept['table']}, decided by {kept['decided_by']},",
f"{', '.join(kept['columns_erased'])} erased")
print("chain head written down, at record:", report["audit log"]["inputs"]["head"]["records"])
print("engine cross-check, computed again from:", "; ".join(report["engine cross-check"]["computed again from"]))
print("engine cross-check, not computed again:", report["engine cross-check"]["not computed again"])
```
Output:
```text
sections: policy, policy tests, who sees what, classification coverage, audit log, engine cross-check, erasures, access review
policy tests passed: 12 of 12
who-sees-what cases the policy agrees with: 12 of 12
classification problem: ai.review_corpus: no metadata
classification problem: marketing.audience: no metadata
audit log records: 14
chain problems, checked alone: 0
queries DuckDB ran with no decision record: 0
decision records DuckDB has no query for: 0
erasure erasure-0001: 25 rows deleted, 0 of 115 files hold the person
erasure erasure-0001, kept under an exception: billing.payments, decided by legal team, card_number erased
chain head written down, at record: 14
engine cross-check, computed again from: the audit log; the engine digests, for the two counts
engine cross-check, not computed again: the reading of DuckDB's log, and the unrecorded queries' shapes: the layer read that log once, then emptied it
```
The policy passes 12 of 12 unit tests and agrees with 12 of 12 who-sees-what cases. Classification coverage isn't clean, and the report says so: `ai.review_corpus` and `marketing.audience` have no metadata, the gaps that [chapters 2](#a-catalog-you-can-test) and [4](#lineage-from-pipelines) found and that the policy meanwhile treats as sensitive. A report that can only show passing checks is evidence of nothing. The chain has 0 problems, checked alone, since no earlier report wrote a head down. DuckDB ran 0 queries without a decision record, and 0 records have no query.
The erasure's lines, `25 rows deleted, 0 of 115 files hold the person` and `billing.payments, decided by legal team, card_number erased`, come from chapter 7's erasure, as the audit log records it: the request's ID, the tables, the counts, each table an exception kept with the reason `governance/retention.toml` gives, and the scan's result, never the learner's ID or values, so the record is safe to chain. The European Data Protection Board recommends that controllers "Verify that erasure has been carried out and be able to demonstrate such erasure" ([EDPB, 2026](https://www.edpb.europa.eu/system/files/2026-02/edpb_cef-report_2025_right-to-erasure_en.pdf)); this line records what the platform's own scan found. The report also writes down the chain's head, at record 14. Kept in another system, it is the anchor the next report checks the chain against: an edit to, or a removal of, any record up to it will show then, and a later record is protected only once a later head covers it.
The report claims nothing about any law or framework. It states what was checked, under which policy, from which inputs, and with what result.
**Working rule:** Generate evidence from records with code, on a schedule and on every policy change, and stamp each section with the policy revision, the digests of its inputs and what it takes to compute it again, saying which parts nobody can. Keep the reports where the log's writer can't change them, include failing findings rather than filtering them out, and let each report check the chain against the head the last one wrote down. Never send a screenshot where a report can be generated; [chapter 10](#governance-in-ci) runs checks like these on every pull request.
## What goes wrong in production {#audit-in-production}
An audit trail fails quietly: nothing breaks when it stops being complete, true or read. The failure that matters most is logging that the audited code can switch off, and DuckDB's query log is that kind: it belongs to the process it records. The example sets up two enforcement layers. The first doesn't lock its configuration, like any layer built without `lib.audit`; code holding its connection tries to log to a file, switches the log off, then reads a learner's email around the layer. The second is `lib.audit`'s, locked; the same kind of code tries to switch the log off, reads around the layer, then empties the log:
Example `query-log-switched-off` (Python 3.12.15 · duckdb 2.0.0 · pyarrow 26.0.0 · pyiceberg 0.12.0 · regopy 1.5.2 · sqlglot 30.24.0), passing in Edition 2026.10.1:
```python
import duckdb
from lib import audit
from lib.catalog import open_catalog
from lib.platform import scratch
AROUND = "select email from crm.learners where learner_id = 541" # a read around the enforcement layer
folder = scratch()
# A layer that doesn't lock its configuration: code holding its connection tries a file log, then switches logging off.
unlocked = audit.Audited(open_catalog(["crm.learners"]), audit.Log(folder / "unlocked.jsonl"), lock=False)
connection = unlocked.enforcer.connection
try:
connection.execute("set log_query_path = 'queries.log'")
except duckdb.Error as error:
print("a query log in a file, on the layer's connection:", type(error).__name__)
connection.execute("CALL disable_logging()")
connection.sql(AROUND).fetchall()
for query in unlocked.check_engine_log()["unrecorded"]:
print("unlocked, unrecorded:", query)
# lib.audit's layer locks its configuration: switching the log off is refused, but emptying it isn't.
log = audit.Log(folder / "locked.jsonl")
locked = audit.Audited(open_catalog(["crm.learners"]), log)
locked.sql("select learner_id, email from crm.learners", "analyst-1").fetchall()
connection = locked.enforcer.connection
try:
connection.execute("CALL disable_logging()")
except duckdb.Error as error:
print("locked, disable_logging():", type(error).__name__)
connection.sql(AROUND).fetchall()
connection.execute("CALL truncate_duckdb_logs()")
found = locked.check_engine_log()
print("locked, after truncate_duckdb_logs(), queries with no decision record:", len(found["unrecorded"]))
print("locked, decision records DuckDB has no query for:", len(found["not in the engine log"]))
```
Output:
```text
a query log in a file, on the layer's connection: PermissionException
unlocked, unrecorded: CALL disable_logging()
unlocked, unrecorded: SET log_query_path = ?
locked, disable_logging(): InvalidInputException
locked, after truncate_duckdb_logs(), queries with no decision record: 0
locked, decision records DuckDB has no query for: 1
```
A log in a file is refused with a `PermissionException`, because [chapter 6](#find-what-a-query-reads) forbids the layer's connection to touch files. On the unlocked layer, `disable_logging()` works, and the read that follows leaves no trace. All the [cross-check](#cross-check-the-query-log) finds is the statements before the silence: the refused `SET log_query_path`, and `CALL disable_logging()` itself. On the locked layer, switching off fails with an `InvalidInputException`, but `truncate_duckdb_logs()` works, and afterwards the check finds 0 unrecorded queries. The check in the other direction notices the gap: 1 decision record has no query in DuckDB's log, so something emptied the log. It notices only because a record fell in the emptied stretch; code that writes entries back with `write_log` would leave the log looking whole.
None of this stops code that holds the connection. In production, enforce in an engine that runs as a service of its own, such as Trino with OPA, so that its log is written outside the process of anyone who queries it. Store the audit records on another system, as control AU-9(2) asks: "Store audit records [Assignment: frequency] in a repository that is part of a physically different system or system component than the system or component being audited" ([NIST SP 800-53 Rev. 5](https://csrc.nist.gov/pubs/sp/800/53/r5/upd1/final)). We recommend sending each record as it's written, and holding the chain's head somewhere else again.
Other failures to watch for:
- **Logs nobody reads.** NIST's guide to log management makes the case for reading them: "Routine log reviews and analysis are beneficial for identifying security incidents, policy violations, fraudulent activity, and operational problems shortly after they have occurred" (§2.2, [NIST SP 800-92](https://doi.org/10.6028/NIST.SP.800-92)). A log that is only written costs storage and protects nothing. Schedule the access review and the evidence report, give every finding an owner, and alert on what can't wait for a quarter, such as a user flagged for denied queries.
- **Decisions that can't be explained after a policy changed.** A record without a policy revision, or a revision without an archived bundle, replays under the current policy and contradicts itself. Fingerprint the data document with the rules, since a change to the data document changes decisions as surely as a change to the rules.
- **Logs that hold personal data.** Personal data reaches logs by many routes: query literals, the engine's own query log, error messages that print values, as [chapter 6](#column-masks-leak) showed for DuckDB's cast errors, and the layer's debugging output. A parser's warnings are one more: sqlglot's warning when it keeps a statement as text quotes the statement ([sqlglot source, v30.24.0](https://github.com/tobymao/sqlglot/blob/v30.24.0/sqlglot/parser.py)), so `lib.audit` silences sqlglot while it parses a user's query. Keep the audit log to IDs and digests, and give every other log a retention period that erasure reaches.
- **A decision log that isn't complete.** OPA can drop decision events by rule, through `system.log.drop`, and "OPA will drop events if the rate limit is exceeded" ([OPA, decision logs](https://www.openpolicyagent.org/docs/management-decision-logs)). Count the engine's queries against the decision records, as the cross-check does, rather than assume every decision was logged.
- **A head kept next to the log.** An anchor stored where the log's writer can change it detects nothing the chain doesn't. Keep it in another system, under other credentials.
## What this chapter decided {#audit-decisions}
- **One record per query, written by the enforcement layer before the query runs.** `lib.audit.Audited` wraps chapter 6's `Enforcer` without changing it, and fetches each result itself. The record covers AU-3's six items, the principal and the assistant that sent the query among them.
- **Records hold IDs, never values.** Every literal, in any of DuckDB's syntaxes, and every name the catalog doesn't hold is replaced, in the query and in the denial's reason, and comments are removed. The SQL DuckDB ran is kept as an HMAC-SHA256 digest under a key only the layer holds, since with the key the digest confirms a guess. Erasures are recorded by request ID.
- **Every record names its policy revision,** the SHA-256 of the policy's modules and data document, and every revision is archived for replay.
- **Records are chained with SHA-256 and numbered,** and the chain protects only the records up to the last head written down outside the layer's control. Each evidence report writes the head down; we recommend doing it more often.
- **DuckDB's query log is a cross-check for mistakes, not the audit log.** The layer turns it on, locks its configuration, checks it both ways against the records since the last check, then empties it. Code holding the connection can defeat it.
- **Access reviews and evidence reports are generated** from the policy, the log and the catalog, each section stamped with its policy revision and inputs. The threshold for flagging denied queries, 3, is a rule of thumb in `governance/audit.toml`.
# Governing data for AI
By the end of this chapter you can build the data an AI assistant learns from under the same rules as every other read of your platform. You'll admit only records whose authors agreed to that use, replace their personal data with pseudonyms and measure what the detector missed, remove duplicates found with MinHash, keep the source of every record with it, and have the pipeline write and check a dataset card on every run. You'll also be able to quote what the EU AI Act says about training data.
The platform, the online learning platform this book follows, sells professional courses to adult learners, and wants a course assistant that answers questions about its courses from what learners say about them. It plans to ground the assistant on the text learners write in their reviews first, and later to fine-tune it on the same text. `review_corpus`, one of the four SQL pipelines that [chapter 4](#lineage-from-pipelines) captures lineage from, copies every row of `learning.reviews` into `ai.review_corpus`, with each review's course and rating. [Chapter 2](#a-catalog-you-can-test) found that `ai.review_corpus` has no owner and no classification, and the copy has four more problems. It holds every review, whether or not its author agreed to their text training an AI. Its text holds names and emails, which [chapter 3](#finding-personal-data) found in reviews. Most of its reviews repeat each other. And nothing in it says where a record came from, so nobody can show that a review was allowed in, or find it again when its author changes their mind.
This chapter rebuilds the job with the book's shared module for AI data, `lib.ai_data`, and the assistant learns only from that governed build: `pipelines/review_corpus.sql` stays as [chapter 4](#lineage-from-pipelines)'s ungoverned copy, so [CI](#run-every-rule-as-one-job) keeps checking its lineage. The module runs the pipeline's own SQL through the enforcement layer that [chapter 6](#enforcing-policies-on-queries) built, replaces the personal data left in the text, removes duplicates, and writes the corpus, its lineage and its dataset card. Its settings are in `governance/ai_data.toml`, reviewed like a policy, and so are the answers the code needs to legal questions, each a placeholder that the platform's legal team replaces with its answer and reason.
## Admit only reviews whose authors consented {#admit-consented-reviews}
On the platform, consent to one use doesn't cover another: the question a learner answers names the use. The EU's General Data Protection Regulation (GDPR) says in Article 5(1) that personal data shall be "collected for specified, explicit and legitimate purposes and not further processed in a manner that is incompatible with those purposes" (point (b)) ([GDPR](https://eur-lex.europa.eu/eli/reg/2016/679/oj)). What that means for a given platform, and which recorded answer covers training an AI, are questions for its legal team, and the code reads the answer to the second from `[legal.consent]` in `governance/ai_data.toml`. Given that answer, the code decides consent per record, when the corpus is built, not from the account's sign-up and not once for the whole table.
`crm.learners.ai_training_consent` records whether a learner agreed to their text training the platform's AI, and [chapter 5](#narrow-by-attribute) made it an attribute the access policy reads: a read of `learning.reviews` for the purpose `ai-training` keeps only the reviews whose author's answer is true, with contact details removed. `lib.ai_data.read` runs the `SELECT` of `pipelines/review_corpus.sql` through [the enforcement layer](#enforcing-policies-on-queries) as `analyst-1`, the data-science principal, stating the purpose, and `lib.ai_data.build` refuses to run unless the layer filters on the column the consent answer names. Compare what the job gets with what the plain pipeline, `pipelines/review_corpus.sql` run as written, copies, and with a consent filter written by hand:
Example `corpus-consent-filter` (Python 3.12.15 · datasketch 2.0.0 · duckdb 2.0.0 · mlcroissant 1.1.1 · openlineage-python 1.54.0 · phonenumbers 9.0.41 · presidio-analyzer 2.2.364 · presidio-anonymizer 2.2.364 · pyarrow 26.0.0 · pyiceberg 0.12.0 · pyyaml 6.0.3 · regopy 1.5.2 · sqlglot 30.24.0 · tldextract 5.4.0), passing in Edition 2026.10.1:
```python
from collections import Counter
from lib import ai_data, pii
from lib.catalog import open_catalog
from lib.enforce import Enforcer
platform = Enforcer(open_catalog(["crm.learners", "learning.courses", "learning.reviews"]))
layer = platform.connection # the enforcement layer's own access, for counting what each build would hold
# The plain pipeline copies every review, with its text as written.
plain = layer.sql("select count(*) from learning.reviews").fetchall()[0][0]
print("reviews the plain pipeline copies:", f"{plain:,}")
# The governed build: the same SQL, through the enforcement layer, as the job's principal, for its purpose.
governed = ai_data.read(platform)
print(f"reviews the governed build reads, as {ai_data.BUILD['principal']} for {ai_data.BUILD['purpose']}:", f"{len(governed):,}")
without = ai_data.read(platform, purpose=None)
print("reviews read without the purpose:", f"{len(without):,}")
print("of them, with their text:", sum(r["body"] is not None for r in without))
# The filter that looks right: leave out the learners who said no.
opt_out = layer.sql("""
select count(*), count(*) filter (where l.ai_training_consent is null)
from learning.reviews r join crm.learners l using (learner_id)
where r.learner_id not in (select learner_id from crm.learners where ai_training_consent = false)""").fetchall()[0]
print("reviews a filter on refusals keeps:", f"{opt_out[0]:,}")
print("of them, by learners never asked:", f"{opt_out[1]:,}")
print("share by learners never asked:", f"{opt_out[1] / opt_out[0]:.0%}")
never = layer.sql("""select min(signup_at)::date, max(signup_at)::date from crm.learners
where ai_training_consent is null""").fetchall()[0]
print("learners never asked signed up:", " to ".join(f"{day:%b} {day.day}, {day.year}" for day in never))
# Personal data the plain pipeline's corpus would hold, from the labelled truth, and how often each email is written.
raw = dict(layer.sql("select review_id, body from learning.reviews").fetchall())
for entity, (placed, _) in ai_data.left(raw).items():
print(f"plain pipeline, labelled {entity} values:", f"{placed:,}")
emails = Counter(pii.text_of(s) for s in pii.truth() if s.source == "reviews" and s.entity == "EMAIL_ADDRESS")
print("plain pipeline, distinct labelled EMAIL_ADDRESS values:", len(emails))
print("of them, written once:", sum(n == 1 for n in emails.values()))
```
Output:
```text
reviews the plain pipeline copies: 3,000
reviews the governed build reads, as analyst-1 for ai-training: 436
reviews read without the purpose: 3,000
of them, with their text: 0
reviews a filter on refusals keeps: 2,045
of them, by learners never asked: 1,609
share by learners never asked: 79%
learners never asked signed up: Jan 1, 2024 to May 31, 2025
plain pipeline, labelled EMAIL_ADDRESS values: 120
plain pipeline, labelled PERSON values: 798
plain pipeline, distinct labelled EMAIL_ADDRESS values: 113
of them, written once: 106
```
The plain pipeline copies all 3,000 reviews, with 120 labelled emails and 798 labelled names in their text. Through the enforcement layer, the job reads 436. Without the purpose, the same principal gets 3,000 reviews, 0 of them with their text, so a job that forgets to state its purpose gets nothing it could train on.
The filter written by hand is the one to recognize, because it looks like respect for consent: leave out the learners who said no. It keeps 2,045 reviews, and 1,609 of them, 79%, are by learners who were never asked. The README of the platform's generated data says consent is unknown for sign-ups before Jun 1, 2025, and every learner with no answer signed up from Jan 1, 2024 to May 31, 2025. An unknown answer isn't a yes. A corpus built with that filter would be mostly text that no consent covers, and nothing about it would look wrong: no error, a plausible row count, and a condition on the consent column that a reviewer would read as correct. The enforcement layer's filter tests `IS TRUE`, so an unknown answer counts as no, which [chapter 6](#build-row-filters-from-data) built for exactly this case.
On the platform, every review was written under one set of terms, so the card states one licence, read from `[legal.licence]`; whether a platform's terms cover training is for its legal team. A corpus that mixes sources, such as reviews and forum posts, needs a licence per record too, which is one reason [every record keeps its source table](#carry-provenance).
**Working rule:** Decide consent per record, when the corpus is built, from the answer your legal team says covers the use, and admit only records whose answer is yes: never filter out the refusals and keep the rest. Build the corpus through the enforcement layer with its purpose stated, and have the build check that the layer filters on that answer. Keep the legal team's answers, with their reasons, in a reviewed file the code reads, and carry licences per record when a source mixes them.
## Replace personal data in the text, then measure what's left {#replace-personal-data}
Consent decides whose text may enter the corpus. It doesn't make the text free of personal data: a review names its instructor, or gives its author's email for questions. Before such text trains or grounds a model, the corpus job detects the personal data in it, replaces each value, and measures what's left, because a detector's output lists what it found and never what it missed. [Chapter 3](#measure-against-labelled-spans) set up the measurement: `data/platform/pii_spans.parquet` labels every piece of personal data the generator placed in the reviews. `lib.ai_data.left` counts how many labelled values are still in the text after a step, matching by value, since every replacement moves the offsets after it.
The reviews arrive from the enforcement layer with its `redact_contacts` mask applied, which replaces emails and phone numbers with `[email]` and `[phone]` ([chapter 6](#write-masks-as-sql)). [Chapter 3](#tune-the-recognizers)'s detector, `lib.pii.platform_detector`, runs Presidio's pattern recognizers with no language model, and no pattern finds a name. So `lib.ai_data.detector` adds a Presidio deny list of the names the platform knows, every word of every instructor's name, which Presidio escapes and matches as a whole word, ignoring case unless told otherwise ([source](https://github.com/data-privacy-stack/presidio/blob/2.2.364/presidio-analyzer/presidio_analyzer/pattern_recognizer.py)). The list comes from `crm.instructors`, which `analyst-1` may not read. So the governance team's own job, named in `[names]` in `governance/ai_data.toml`, builds it with the platform's own access, as the enforcement layer reads `crm.learners` for consent, and hands the corpus job the list, never the table; the card records which job built it and how many words it holds. `lib.ai_data.known_names` stands for that job. Each value found becomes a keyed pseudonym, which [the next section](#consistent-pseudonyms) builds. Measure each step on the reviews the corpus reads, then try a longer list, with every learner's name in it too:
Example `corpus-personal-data-left` (Python 3.12.15 · datasketch 2.0.0 · duckdb 2.0.0 · mlcroissant 1.1.1 · openlineage-python 1.54.0 · phonenumbers 9.0.41 · presidio-analyzer 2.2.364 · presidio-anonymizer 2.2.364 · pyarrow 26.0.0 · pyiceberg 0.12.0 · pyyaml 6.0.3 · regopy 1.5.2 · sqlglot 30.24.0 · tldextract 5.4.0), passing in Edition 2026.10.1:
```python
from lib import ai_data, pii
from lib.catalog import open_catalog
from lib.enforce import Enforcer
# The reviews the corpus job reads: consenting authors only, with emails and phone numbers already masked.
rows = ai_data.read(Enforcer(open_catalog(["crm.learners", "learning.courses", "learning.reviews"])))
texts = {row["review_id"]: row["body"] for row in rows}
# What's left of the labelled personal data after each step, matched by value against the generator's truth.
detectors = {"then the pattern recognizers": pii.platform_detector(), "then the instructors' names": ai_data.detector()}
stages = {"masked by the enforcement layer": texts}
stages |= {name: {i: ai_data.pseudonymize(engine, text)[0] for i, text in texts.items()} for name, engine in detectors.items()}
for stage, done in stages.items():
for entity, (placed, still) in ai_data.left(done).items():
print(f"{stage}, {entity} left: {still} of {placed}")
names = ai_data.known_names()
print("words in the list of instructors' names:", len(names))
print("words in the list of 3 characters or fewer:", ", ".join(word for word in names if len(word) <= 3))
print("false positives, instructors' names:", sum(ai_data.unlabelled(ai_data.detector(), texts).values()))
# Every name the platform holds, learners' too, on the same reviews: in any case, then only as written.
longer = sorted(set(names) | set(ai_data.known_names("crm.learners")))
everyone = ai_data.detector(longer)
wrong = ai_data.unlabelled(everyone, texts)
print("words in the list, with learners' names:", f"{len(longer):,}")
print("false positives, with learners' names:", sum(wrong.values()))
print("most common false positives:", ", ".join(f"{word} ({n})" for word, n in wrong.most_common(2)))
print("a review after:", ai_data.pseudonymize(everyone, texts[24])[0])
cased = ai_data.detector(longer, match_case=True)
print("false positives, with learners' names, matching case:", sum(ai_data.unlabelled(cased, texts).values()))
left = ai_data.left({i: ai_data.pseudonymize(cased, text)[0] for i, text in texts.items()})
print("names left, with learners' names, matching case:", f"{left['PERSON'][1]} of {left['PERSON'][0]}")
```
Output:
```text
masked by the enforcement layer, EMAIL_ADDRESS left: 0 of 19
masked by the enforcement layer, PERSON left: 120 of 120
then the pattern recognizers, EMAIL_ADDRESS left: 0 of 19
then the pattern recognizers, PERSON left: 120 of 120
then the instructors' names, EMAIL_ADDRESS left: 0 of 19
then the instructors' names, PERSON left: 0 of 120
words in the list of instructors' names: 106
words in the list of 3 characters or fewer: Ray, Roy, Ulf, Uss, de, la
false positives, instructors' names: 0
words in the list, with learners' names: 4,316
false positives, with learners' names: 147
most common false positives: best (80), date (67)
a review after: Some videos are out of .
false positives, with learners' names, matching case: 0
names left, with learners' names, matching case: 0 of 120
```
The enforcement layer's mask leaves 0 of 19 emails and every name, 120 of 120. The pattern recognizers change nothing: 120 of 120. With the instructors' names added, the detector leaves 0 of 120 names, with 0 false positives, although the list of 106 words holds short ones that ordinary text could contain: `Ray, Roy, Ulf, Uss, de, la`. With the learners' names added, 4,316 words, it makes 147 false positives. The most common, `best (80), date (67)`, are surnames that are also English words, and the list turns the review "Some videos are out of date." into `Some videos are out of .` in the corpus. Matching case avoids both. Matched only as written, the longer list makes 0 false positives and leaves 0 of 120 names, because every name in these reviews is capitalized. On real text it would miss the names typed in lowercase that [chapter 3 found in tickets](#where-patterns-fail). The platform keeps the instructors' list, matched in any case: instructors are the people reviews talk about, and the corpus job then holds their names instead of every learner's.
No names left is a result about this data. The labelled spans are generated, and the generated reviews name only instructors, so they can't contain the case a list of instructors misses, such as a classmate's name. For names nobody listed, the detector to try is a named-entity model. Chapter 3 [compared one](#compare-a-named-entity-model), OpenAI's Privacy Filter, from a recording of its detections, scored against the labelled spans with `lib.pii.score`. Measure it the same way on the reviews the corpus reads:
Example `privacy-filter-on-the-corpus` (Python 3.12.15 · datasketch 2.0.0 · duckdb 2.0.0 · mlcroissant 1.1.1 · openlineage-python 1.54.0 · phonenumbers 9.0.41 · presidio-analyzer 2.2.364 · presidio-anonymizer 2.2.364 · pyarrow 26.0.0 · pyiceberg 0.12.0 · pyyaml 6.0.3 · regopy 1.5.2 · sqlglot 30.24.0 · tldextract 5.4.0), passing in Edition 2026.10.1:
```python
from collections import Counter
from lib import ai_data, pii
from lib.catalog import open_catalog
from lib.enforce import Enforcer
# Privacy Filter's eight labels, mapped to the labelled spans' entity types. A label with no type matches nothing.
MATCHES = {"private_person": {"PERSON"}, "private_email": {"EMAIL_ADDRESS"}, "private_phone": {"PHONE_NUMBER"},
"private_address": {"LOCATION"}, "account_number": {"CREDIT_CARD", "IBAN_CODE"},
"private_date": {"DATE_TIME"}, "private_url": set(), "secret": set()}
# The reviews the corpus job reads, with the model's recorded detections and the labelled spans in them. The model read
# each review as written, before the enforcement layer's mask, which changes no name.
corpus = {row["review_id"] for row in ai_data.read(Enforcer(open_catalog(["crm.learners", "learning.courses",
"learning.reviews"])))}
model, manifest = pii.recorded("pii-entities")
found = [span for span in model if span.source == "reviews" and span.row_id in corpus]
labelled = [span for span in pii.truth() if span.source == "reviews" and span.row_id in corpus]
print("model:", manifest["model"])
print("reviews measured:", len(corpus))
scores = pii.score(found, labelled, MATCHES)
placed, exact, _ = scores.recall["PERSON"]
print("labelled names:", placed)
print("names the model finds, exactly:", exact)
missed = pii.missed([span for span in labelled if span.entity == "PERSON"], found, {"private_person"})
print("names the model misses:", len(missed))
often = Counter(map(pii.text_of, missed)).most_common(3)
print("names it misses most often:", ", ".join(f"{name} ({n})" for name, n in often))
# What the model finds where no labelled span of the type its label maps to is: text it would replace for nothing.
wrong = sum(n - overlap for n, _, overlap in scores.precision.values())
print("the model's detections:", len(found))
print("of them, overlapping no labelled span of their type:", wrong)
```
Output:
```text
model: openai/privacy-filter
reviews measured: 436
labelled names: 120
names the model finds, exactly: 83
names the model misses: 37
names it misses most often: Marie (4), Liwia (4), Luigi (4)
the model's detections: 102
of them, overlapping no labelled span of their type: 0
```
Of the 120 labelled names in those 436 reviews, the model finds 83 exactly and misses 37, most often `Marie (4), Liwia (4), Luigi (4)`, where the list misses none. Of its 102 detections, 0 overlap no labelled span of their type. On these reviews the model can't replace the list, and this data can't show what it adds, since every name in it is on the list. Where text names people no list holds, run both, measured this way first.
The text that comes out can still identify people. A detector misses spans, the text can identify someone without a name, and each record keeps a review ID that leads back to its author through `learning.reviews`. The GDPR's Recital 26 says: "Personal data which have undergone pseudonymisation, which could be attributed to a natural person by the use of additional information should be considered to be information on an identifiable natural person" ([GDPR, Recital 26](https://eur-lex.europa.eu/eli/reg/2016/679/oj)). So we call the corpus pseudonymized, never anonymized.
**Working rule:** Detect personal data in corpus text after the enforcement layer's masks, add the identifiers your platform holds about the people the text is about, and measure what's left against labelled data, by value, after every change to the detector. Prefer a short list of the names your text is about to every name you hold, since a long list matches ordinary words. Have a job that may read the names build the list, and record which list each corpus used. Measure a named-entity model the same way before you rely on it for names no list holds. Keep the list out of the corpus, and call the result pseudonymized.
## Keep pseudonyms consistent with a keyed hash {#consistent-pseudonyms}
A detected name can be replaced by a placeholder, ``, or by a pseudonym: a token for one name, the same wherever that name appears. The platform's reviews name an instructor by first name only, and 0 instructors share a first name with another, so here a token stands for one person. A pseudonym keeps who is who, so two reviews that praise the same instructor still say so. Presidio's anonymizer has a `hash` operator, but "Starting from version 2.2.361, the hash operator uses random salt by default for security" ([Presidio anonymizer](https://presidio.dataprivacystack.org/anonymizer/)). A random salt makes each hash unlinkable to the next, and useless as a pseudonym. To get the same hash for the same value, the docs say, "you **must** explicitly provide a `salt` parameter", of "at least 128 bits (16 bytes)".
The corpus job uses its own keyed hash instead, through Presidio's `custom` operator, which takes a lambda whose "return type must be a string". `lib.ai_data.pseudonym` computes an HMAC with SHA-256 of the entity type and the case-folded value, under a key only the corpus job holds, and keeps eight hexadecimal digits. HMAC is the construction RFC 2104 describes for keying a hash, and one of its stated goals is "To have a well understood cryptographic analysis of the strength of the authentication mechanism based on reasonable assumptions on the underlying hash function" ([RFC 2104](https://www.rfc-editor.org/rfc/rfc2104)). Python's standard library has it, as `hmac`, so the job has no reason to key a hash any other way. Take two of the corpus's reviews that name the same instructor, and replace the name both ways:
Example `keyed-pseudonyms` (Python 3.12.15 · datasketch 2.0.0 · duckdb 2.0.0 · mlcroissant 1.1.1 · openlineage-python 1.54.0 · phonenumbers 9.0.41 · presidio-analyzer 2.2.364 · presidio-anonymizer 2.2.364 · pyarrow 26.0.0 · pyiceberg 0.12.0 · pyyaml 6.0.3 · regopy 1.5.2 · sqlglot 30.24.0 · tldextract 5.4.0), passing in Edition 2026.10.1:
```python
from collections import Counter
from presidio_anonymizer import AnonymizerEngine
from presidio_anonymizer.entities import InvalidParamError, OperatorConfig
from sqlglot import exp
from lib import ai_data, enforce
from lib.catalog import open_catalog
from lib.enforce import Enforcer
from lib.platform import load
# Two of the corpus's reviews that name the same instructor, with what the detector found in each.
layer = Enforcer(open_catalog(["crm.learners", "learning.courses", "learning.reviews"]))
engine = ai_data.detector()
found = [(row["body"], engine.analyze(row["body"], language="en")) for row in ai_data.read(layer)]
named = [(text, spans) for text, spans in found if [text[s.start:s.end] for s in spans] == ["Patrick"]][:2]
print("reviews naming Patrick:", len(named))
def replaced(operator: OperatorConfig) -> list[str]:
"""The value that replaced the name in each of the two reviews."""
results = [AnonymizerEngine().anonymize(text, spans, {"PERSON": operator}) for text, spans in named]
return [result.text[item.start:item.end] for result in results for item in result.items]
hashed = replaced(OperatorConfig("hash"))
print("Presidio's hash, the same name in two reviews gets one value:", hashed[0] == hashed[1])
print("Presidio's hash, one review hashed twice gets one value:", replaced(OperatorConfig("hash"))[0] == hashed[0])
try:
replaced(OperatorConfig("hash", {"salt": "ten bytes!"}))
except InvalidParamError as error:
print("a fixed salt of 10 bytes:", error)
keyed = replaced(OperatorConfig("custom", {"lambda": lambda value: ai_data.pseudonym("PERSON", value)}))
print("keyed pseudonyms:", ", ".join(keyed))
print("the name typed in lowercase:", ai_data.pseudonym("PERSON", "patrick"))
first_names = Counter(name.split()[0] for name in load("crm.instructors").column("full_name").to_pylist())
print("instructors who share a first name with another:", sum(n for n in first_names.values() if n > 1))
# What the enforcement layer's pseudonymize mask makes of the same name, run by the layer under its own key.
mask = enforce.masked("pseudonymize", exp.Literal.string("Patrick"), "VARCHAR").sql(enforce.DIALECT)
print("the enforcement layer's pseudonymize mask, on the same name:", layer.run(f"select {mask}").fetchall()[0][0])
hmac = layer.connection.sql("select count(*) from duckdb_functions() where function_name ilike '%hmac%'")
print("HMAC functions in DuckDB:", hmac.fetchall()[0][0])
```
Output:
```text
reviews naming Patrick: 2
Presidio's hash, the same name in two reviews gets one value: False
Presidio's hash, one review hashed twice gets one value: False
a fixed salt of 10 bytes: Salt must be at least 16 bytes (128 bits). Provided salt is 10 bytes.
keyed pseudonyms: ,
the name typed in lowercase:
instructors who share a first name with another: 0
the enforcement layer's pseudonymize mask, on the same name: 23b3d8671d01840655296f752121aecc4f7991694260e5e1431d2b57c5e3d53f
HMAC functions in DuckDB: 0
```
Asked whether Presidio's `hash` gives the name one value in the two reviews, the example answers `False`, and asked whether hashing one review twice gives one value, `False` again. A fixed salt that's too short is refused with `Salt must be at least 16 bytes (128 bits). Provided salt is 10 bytes.`, as the docs say. The keyed hash gives the two reviews `, `, and the name typed in lowercase gets `` too.
The enforcement layer's `pseudonymize` mask, run by the layer on the same name, gives `23b3d8671d01840655296f752121aecc4f7991694260e5e1431d2b57c5e3d53f`: SHA-256 over the layer's key followed by the value, as [chapter 6](#write-masks-as-sql) writes it, because the layer's masks are SQL and DuckDB 2.0.0 has no HMAC function (the example counts 0). The two are different constructions under different keys, so the corpus's pseudonyms can't be joined with the ones the layer shows analysts.
Consistency has two costs. Whoever holds the key and a list of names can compute every pseudonym, as chapter 6 showed for emails, so the key lives in a secret manager, never in the corpus, the repository or a log. And keyed pseudonyms keep apart reviews that placeholders would merge: 96 distinct texts against 21, which [the next section](#near-duplicates-with-minhash) measures. We recommend keyed pseudonyms where the text's meaning depends on telling people apart, as in a ticket that names an agent and a learner, and placeholders where nothing downstream needs to.
**Working rule:** Make pseudonyms with HMAC under a key only the job holds, and give each corpus or audience whose pseudonyms must not join a key of its own. Never use Presidio's `hash` operator without a salt for pseudonyms, since at its pinned version each hash is random. Pseudonymize a person's ID rather than a word of their name when two people can share one, and use a plain placeholder when nothing downstream needs to tell people apart.
## Find near-duplicates with MinHash {#near-duplicates-with-minhash}
Once its names are [pseudonymized](#consistent-pseudonyms), the corpus's next problem is repetition: templates, copies, a review pasted twice, the same sentence from many people. Repetition is a privacy problem as well as a quality one: Kandpal, Wallace and Raffel found that "a sequence that is present 10 times in the training data is on average generated ~1000 times more often than a sequence that is present only once" ([Kandpal et al., 2022](https://arxiv.org/abs/2202.06539)). Near-copies need a measure of similarity. Broder defined the resemblance of two documents as the Jaccard similarity of their sets of shingles, the overlapping runs of characters or words in each, and showed that sampling the smallest shingles under a random permutation gives a value that "is an unbiased estimate of the resemblance of A and B" ([Broder, 1997](https://doi.org/10.1109/SEQUEN.1997.666900)). Taking the smallest shingle under each of many permutations gives a MinHash signature, which estimates the similarity of two texts without comparing their sets ([Mining of Massive Datasets, section 3.3](http://infolab.stanford.edu/~ullman/mmds/ch3n.pdf)).
`lib.ai_data.deduplicate` removes exact copies first, by SHA-256 of the case-folded text with runs of white space made one. It compares what's left as sets of 5-character shingles with datasketch's `MinHash` and `MinHashLSH`, which returns candidate pairs, and confirms each candidate by its exact Jaccard similarity. A confirmed pair at 0.8 or more is a near-duplicate, unless it differs only in a name, and of each group of duplicates the record with the lowest review ID stays. Run it on the corpus's pseudonymized reviews:
Example `corpus-duplicates` (Python 3.12.15 · datasketch 2.0.0 · duckdb 2.0.0 · mlcroissant 1.1.1 · openlineage-python 1.54.0 · phonenumbers 9.0.41 · presidio-analyzer 2.2.364 · presidio-anonymizer 2.2.364 · pyarrow 26.0.0 · pyiceberg 0.12.0 · pyyaml 6.0.3 · regopy 1.5.2 · sqlglot 30.24.0 · tldextract 5.4.0), passing in Edition 2026.10.1:
```python
from collections import Counter
from lib import ai_data
from lib.catalog import open_catalog
from lib.enforce import Enforcer
# The reviews the corpus job reads, pseudonymized, then deduplicated: exact copies, then near-duplicates.
rows, _ = ai_data.read_pseudonymized(Enforcer(open_catalog(["crm.learners", "learning.courses", "learning.reviews"])))
groups, removed = ai_data.duplicate_groups(rows)
print("reviews read:", len(rows))
print("distinct texts:", len(ai_data.exact_distinct(rows)))
for step, count in removed.items():
print(f"{step}:", count)
print("records kept:", len(groups))
print("share of the reviews read that duplicate another:", f"{1 - len(groups) / len(rows):.0%}")
text, copies = Counter(row["body"] for row in rows).most_common(1)[0]
print("the most repeated text:", text)
print("its copies:", copies)
# A pair MinHash finds and the exact comparison confirms, and whether it differs only in a name.
distinct = [row["body"] for row in ai_data.exact_distinct(rows)]
i, j = sorted(ai_data.near_duplicates(distinct).confirmed)[0]
print("a near-duplicate pair:", distinct[i], "|", distinct[j])
print("its Jaccard similarity:", f"{ai_data.jaccard(distinct[i], distinct[j]):.2f}")
print("it differs only in a name:", ai_data.placeholders(distinct[i]) == ai_data.placeholders(distinct[j]))
# Each record kept stands for its group's reviews, under the review ID and course of the first.
several = sum(len({row["course_id"] for row in group}) > 1 for group in groups)
print("records that stand for reviews of several courses:", several)
```
Output:
```text
reviews read: 436
distinct texts: 96
exact duplicates removed: 340
near-duplicate pairs from LSH: 381
near-duplicate pairs confirmed: 8
of them, differing only in a name: 8
near-duplicates removed: 0
records kept: 96
share of the reviews read that duplicate another: 78%
the most repeated text: Clear explanations and good exercises.
its copies: 56
a near-duplicate pair: Some videos are out of date. explains things well. Happy to talk about it, write to me at [email]. | Some videos are out of date. explains things well. Happy to talk about it, write to me at [email].
its Jaccard similarity: 0.81
it differs only in a name: True
records that stand for reviews of several courses: 23
```
Of 436 reviews read, only 96 texts are distinct: 340 are exact copies of another. "Clear explanations and good exercises." alone appears 56 times. MinHash then confirms 8 pairs at 0.8 or more, such as two reviews at a similarity of 0.81 that differ only in their instructor's pseudonym. Of those pairs, 8 differ only in a name, which the platform keeps apart, so near-duplicates remove 0. The corpus keeps 96 records, and 78% of the reviews it read repeat another. That's mostly the generator's doing, since it builds reviews from a handful of fixed sentences, so measure the share on your own text before training on it.
Why pairs that differ only in a name stay apart comes down to what one threshold means at different lengths. Compare the corpus's reviews with the names as written, with keyed pseudonyms and with plain placeholders, then split every pair of distinct texts by whether it differs only in a name:
Example `names-and-near-duplicates` (Python 3.12.15 · datasketch 2.0.0 · duckdb 2.0.0 · mlcroissant 1.1.1 · openlineage-python 1.54.0 · phonenumbers 9.0.41 · presidio-analyzer 2.2.364 · presidio-anonymizer 2.2.364 · pyarrow 26.0.0 · pyiceberg 0.12.0 · pyyaml 6.0.3 · regopy 1.5.2 · sqlglot 30.24.0 · tldextract 5.4.0), passing in Edition 2026.10.1:
```python
from itertools import combinations
from lib import ai_data
from lib.catalog import open_catalog
from lib.enforce import Enforcer
# The reviews the corpus job reads, three ways: names as written, keyed pseudonyms, and plain placeholders.
layer = Enforcer(open_catalog(["crm.learners", "learning.courses", "learning.reviews"]))
keyed, _ = ai_data.read_pseudonymized(layer)
ways = {"names as written": ai_data.read(layer), "keyed pseudonyms": keyed,
"placeholders": [row | {"body": ai_data.placeholders(row["body"])} for row in keyed]}
threshold = ai_data.DUPLICATES["threshold"]
for way, rows in ways.items():
texts = [row["body"] for row in ai_data.exact_distinct(rows)]
print(f"{way}, distinct texts:", len(texts))
print(f"{way}, pairs at {threshold} or more:", len(ai_data.near_duplicates(texts).confirmed))
# Every pair of distinct pseudonymized texts, split by whether the pair differs only in a name.
texts = [row["body"] for row in ai_data.exact_distinct(keyed)]
name_only, other = [], []
for a, b in combinations(texts, 2):
pair = (ai_data.jaccard(a, b), len(a), a, b)
(name_only if ai_data.placeholders(a) == ai_data.placeholders(b) else other).append(pair)
print("pairs that differ only in a name:", len(name_only))
print("their Jaccard similarity:", f"{min(name_only)[0]:.2f} to {max(name_only)[0]:.2f}")
for label, lengths in [(f"below {threshold}", [n for s, n, _, _ in name_only if s < threshold]),
(f"at {threshold} or more", [n for s, n, _, _ in name_only if s >= threshold])]:
print(f"their length in characters, {label}:", f"{min(lengths)} to {max(lengths)}")
similarity, _, a, b = max(other)
print("the most similar pair that differs in more than a name:", f"{similarity:.2f}")
print("that pair:", a, "|", b)
```
Output:
```text
names as written, distinct texts: 96
names as written, pairs at 0.8 or more: 10
keyed pseudonyms, distinct texts: 96
keyed pseudonyms, pairs at 0.8 or more: 8
placeholders, distinct texts: 21
placeholders, pairs at 0.8 or more: 0
pairs that differ only in a name: 459
their Jaccard similarity: 0.68 to 0.83
their length in characters, below 0.8: 66 to 78
their length in characters, at 0.8 or more: 116 to 119
the most similar pair that differs in more than a name: 0.66
that pair: Clear explanations and good exercises. Happy to talk about it, write to me at [email]. | Clear explanations and good exercises. explains things well. Happy to talk about it, write to me at [email].
```
With the names as written, 10 pairs reach 0.8, and with keyed pseudonyms 8, since a pseudonym is longer than most first names and so a larger share of a short review. With plain placeholders, the 21 distinct texts hold 0. Among the pseudonymized texts, 459 pairs differ only in which pseudonym they hold, at similarities from 0.68 to 0.83, and the threshold splits them by length alone: those below 0.8 are 66 to 78 characters long, and those at 0.8 or more 116 to 119. So the task, not the threshold, decides whether such reviews are duplicates: for a model that learns how learners describe courses they are, and for one that learns what learners say about each instructor they aren't. The platform's assistant answers about each course and its instructor, so `name_only_pairs_are_duplicates = false` in `governance/ai_data.toml`, and `deduplicate` never merges two texts that are the same once their pseudonyms are placeholders. The threshold decides the other pairs, and the most similar of those, at 0.66, is a review with and without its sentence about the instructor: two different reviews. The platform keeps 0.8, above that pair with a margin.
Deduplicating on text also folds reviews of different courses into one record, under the review ID and course of its first review: 23 of the 96 records stand for reviews of several courses. For training that's the point, since the model learns from the text. For grounding an answer about one course it costs recall, which [the provenance section](#carry-provenance) takes up.
**Working rule:** Deduplicate training text in two steps, exact copies by a hash of the normalized text and then near-duplicates by MinHash, each confirmed by its exact similarity, and report how much each step removed. Pseudonymize first, so that no name reaches the comparison or the sketches and the comparison can tell which pairs differ only in a name. Keep those pairs apart when the text's meaning depends on whom it names, and compare on placeholders when it doesn't. Choose the threshold by reading the pairs close above and below it in your own text.
## Tune LSH for recall, then confirm every pair {#tune-lsh-then-confirm}
[MinHash signatures](#near-duplicates-with-minhash) make each comparison cheap; locality-sensitive hashing (LSH) avoids most comparisons. It cuts each signature into b bands of r rows, and two texts become a candidate pair when all the rows of any one band match. Leskovec, Rajaraman and Ullman derive the chance of that for a pair with Jaccard similarity s ([Mining of Massive Datasets, chapter 3](http://infolab.stanford.edu/~ullman/mmds/ch3n.pdf)):
P(candidate) = 1 − (1 − sʳ)ᵇ
It's an S-shaped curve: a pair well below the threshold is rarely a candidate, a pair well above it nearly always, and near the threshold it's close to a coin toss. "An approximation to the threshold is (1/b)^(1/r)". datasketch's documentation warns: "There will be false positives - sets that do not satisfy your threshold but returned, and false negatives - qualifying sets that are not returned" ([datasketch, MinHash LSH](https://ekzhu.github.io/datasketch/lsh.html)).
With 96 distinct texts, the platform can afford to compare every pair exactly, 4,560 pairs, and that gives the truth to measure LSH against. LSH sees only similarity, so it must return every pair at 0.8, whatever the name rule then decides. Run LSH tuned for the threshold itself, 0.8, and tuned lower, each with datasketch's default hashing and with the hashing it used before its 2.0.0 release:
Example `lsh-misses-near-duplicates` (Python 3.12.15 · datasketch 2.0.0 · duckdb 2.0.0 · mlcroissant 1.1.1 · openlineage-python 1.54.0 · phonenumbers 9.0.41 · presidio-analyzer 2.2.364 · presidio-anonymizer 2.2.364 · pyarrow 26.0.0 · pyiceberg 0.12.0 · pyyaml 6.0.3 · regopy 1.5.2 · sqlglot 30.24.0 · tldextract 5.4.0), passing in Edition 2026.10.1:
```python
from itertools import combinations
from datasketch import MinHashLSH
from lib import ai_data
from lib.catalog import open_catalog
from lib.enforce import Enforcer
rows, _ = ai_data.read_pseudonymized(Enforcer(open_catalog(["crm.learners", "learning.courses", "learning.reviews"])))
texts = [row["body"] for row in ai_data.exact_distinct(rows)]
threshold = ai_data.DUPLICATES["threshold"]
# Every pair compared exactly: possible at this size, and the truth LSH is measured against.
pairs = {(i, j) for i, j in combinations(range(len(texts)), 2) if ai_data.jaccard(texts[i], texts[j]) >= threshold}
print("pairs compared exactly:", f"{len(texts) * (len(texts) - 1) // 2:,}")
print(f"pairs at a Jaccard similarity of {threshold} or more:", len(pairs))
found = {}
for scheme in ["affine32", "legacy"]:
for tuned_for in [threshold, ai_data.DUPLICATES["candidate_threshold"]]:
found[scheme, tuned_for] = result = ai_data.near_duplicates(texts, threshold, tuned_for, scheme)
setting = f"{scheme}, LSH tuned for {tuned_for}"
print(f"{setting}, bands:", result.bands)
print(f"{setting}, rows per band:", result.rows)
print(f"{setting}, candidate pairs:", len(result.candidates))
print(f"{setting}, confirmed:", len(result.confirmed))
print(f"{setting}, missed:", len(pairs - result.candidates))
# The chance that a pair of similarity s becomes a candidate, with b bands of r rows: 1 - (1 - s^r)^b.
b, r = found["affine32", threshold].bands, found["affine32", threshold].rows
for s in [0.8, 0.9]:
print(f"chance of a candidate at similarity {s}, LSH tuned for {threshold}:", f"{1 - (1 - s ** r) ** b:.2f}")
print("the approximate threshold, (1/b)^(1/r):", f"{(1 / b) ** (1 / r):.2f}")
# What datasketch does with a sketch made under the other scheme.
index = MinHashLSH(threshold=threshold, num_perm=ai_data.DUPLICATES["num_perm"])
index.insert(0, ai_data.minhash(texts[0], "affine32"))
try:
index.insert(1, ai_data.minhash(texts[1], "legacy"))
except ValueError as error:
print("a legacy sketch in an index of affine32 sketches:", error)
```
Output:
```text
pairs compared exactly: 4,560
pairs at a Jaccard similarity of 0.8 or more: 8
affine32, LSH tuned for 0.8, bands: 9
affine32, LSH tuned for 0.8, rows per band: 13
affine32, LSH tuned for 0.8, candidate pairs: 36
affine32, LSH tuned for 0.8, confirmed: 1
affine32, LSH tuned for 0.8, missed: 7
affine32, LSH tuned for 0.6, bands: 18
affine32, LSH tuned for 0.6, rows per band: 7
affine32, LSH tuned for 0.6, candidate pairs: 381
affine32, LSH tuned for 0.6, confirmed: 8
affine32, LSH tuned for 0.6, missed: 0
legacy, LSH tuned for 0.8, bands: 9
legacy, LSH tuned for 0.8, rows per band: 13
legacy, LSH tuned for 0.8, candidate pairs: 52
legacy, LSH tuned for 0.8, confirmed: 6
legacy, LSH tuned for 0.8, missed: 2
legacy, LSH tuned for 0.6, bands: 18
legacy, LSH tuned for 0.6, rows per band: 7
legacy, LSH tuned for 0.6, candidate pairs: 406
legacy, LSH tuned for 0.6, confirmed: 8
legacy, LSH tuned for 0.6, missed: 0
chance of a candidate at similarity 0.8, LSH tuned for 0.8: 0.40
chance of a candidate at similarity 0.9, LSH tuned for 0.8: 0.93
the approximate threshold, (1/b)^(1/r): 0.84
a legacy sketch in an index of affine32 sketches: MinHash scheme 'legacy' does not match scheme 'affine32' of the MinHash previously given to this index
```
Compared exactly, 8 pairs reach 0.8. Tuned for 0.8, datasketch chooses 9 bands of 13 rows, returns 36 candidate pairs, and misses 7 of the 8, so LSH confirms only 1 of them. The formula says why: with those bands, a pair at similarity 0.8 becomes a candidate with a probability of 0.40, one at 0.9 with 0.93, and the curve's approximate threshold is 0.84, above the threshold asked for. Tuned for 0.6, it chooses 18 bands of 7 rows, returns 381 candidates and misses 0. Each candidate costs one exact comparison, 381 instead of 4,560, and the exact comparison discards every pair below 0.8. So the platform tunes LSH for 0.6 and confirms at 0.8, both settings in `governance/ai_data.toml`.
datasketch's 2.0.0 release made `affine32` MinHash's default permutation scheme, and its notes warn: "Hash values differ from earlier versions: rebuild persisted sketches and LSH indexes, or pass MinHash(..., scheme="legacy")" ([datasketch release notes](https://github.com/ekzhu/datasketch/releases/tag/v2.0.0)). With the legacy scheme, LSH tuned for 0.8 returns 52 candidates and confirms 6, so a count recorded under one scheme isn't reproduced under the other. Tuned for 0.6, both schemes find all 8 pairs, from 381 and 406 candidates: with high recall and an exact check, the result depends on the similarity, not on the hash functions. datasketch refuses to mix schemes in one index, with `MinHash scheme 'legacy' does not match scheme 'affine32' of the MinHash previously given to this index`, but not to compare old counts with new ones. The book's examples pin datasketch at 2.0.0 and write `scheme = "affine32"` into the settings, so an upgrade can't change a recorded count without a reviewed change.
**Working rule:** Tune LSH below your similarity threshold, so that it misses few pairs, and confirm every candidate by its exact similarity. Measure LSH's misses against an exhaustive comparison on a sample of your corpus whenever its settings, its library or the text change. Pin the MinHash library, write its hashing scheme into your settings, and never compare counts or stored sketches made under different schemes.
## Carry provenance on every record {#carry-provenance}
A record that can't be traced to its source can't be checked, corrected or removed. Provenance at the level of the record answers three questions: which table it came from, which row, and which run of which job put it in the corpus. [Chapter 4](#send-run-events) gives the platform the last part: every run of a job sends OpenLineage events, with a run ID derived from the run's time and the job's name, and column lineage parsed from the job's SQL.
`lib.ai_data.build` writes every record with its provenance beside the review's own columns: `source_table`, `review_id`, `job` and `run_id`. It sends the job's run events with `lib.lineage.record`, and one choice there matters. The lineage is parsed from the query as the enforcement layer ran it, the rewrite with the consent filter and the mask in it, not from the SQL in `pipelines/review_corpus.sql`. Build the corpus, then follow a record back:
Example `corpus-provenance` (Python 3.12.15 · datasketch 2.0.0 · duckdb 2.0.0 · mlcroissant 1.1.1 · openlineage-python 1.54.0 · phonenumbers 9.0.41 · presidio-analyzer 2.2.364 · presidio-anonymizer 2.2.364 · pyarrow 26.0.0 · pyiceberg 0.12.0 · pyyaml 6.0.3 · regopy 1.5.2 · sqlglot 30.24.0 · tldextract 5.4.0), passing in Edition 2026.10.1:
```python
from lib import ai_data, lineage, pipelines
from lib.catalog import open_catalog
catalog = open_catalog(["crm.learners", "learning.courses", "learning.reviews"])
built = ai_data.build(catalog)
record = built.records.to_pylist()[1]
print("a record's text:", record["body"])
for column in ["source_table", "review_id", "job"]:
print(f"its {column}:", record[column])
# The run that wrote it, from the lineage events: the query as the enforcement layer ran it.
complete = [e for e in lineage.read_events(built.folder / "events.jsonl") if e["eventType"] == "COMPLETE"][0]
print("its run_id is the COMPLETE event's run:", record["run_id"] == complete["run"]["runId"])
print("the run's inputs:", ", ".join(dataset["name"] for dataset in complete["inputs"]))
facet = complete["outputs"][0]["facets"]["columnLineage"]
filters = [f"{i['name']}.{i['field']}" for i in facet["dataset"] if any(t["subtype"] == "FILTER" for t in i["transformations"])]
print("rows filtered by, as run:", ", ".join(filters))
text = [i for i in facet["fields"]["body"]["inputFields"] if (i["name"], i["field"]) == ("learning.reviews", "body")]
print("body, as run:", text[0]["transformations"][0]["subtype"])
fields, rows = lineage.column_lineage(pipelines.sql(ai_data.BUILD["job"]), lineage.schema(catalog))
print("rows filtered by, as written:", ", ".join(i.source for i in rows if i.subtype == "FILTER") or "nothing")
print("body, as written:", fields["body"][0].subtype)
# Back from every record to its review's author, with the platform's own access, against today's answers.
ids = built.records.column("review_id").to_pylist()
print("records whose author consented:", f"{ai_data.consenting(catalog, ids)} of {len(ids)}")
```
Output:
```text
a record's text: Too fast in the second half. explains things well.
its source_table: learning.reviews
its review_id: 18
its job: platform/review_corpus
its run_id is the COMPLETE event's run: True
the run's inputs: crm.learners, learning.courses, learning.reviews
rows filtered by, as run: crm.learners.ai_training_consent, learning.reviews.learner_id
body, as run: TRANSFORMATION
rows filtered by, as written: nothing
body, as written: IDENTITY
records whose author consented: 96 of 96
```
The record "Too fast in the second half. explains things well." came from `learning.reviews`, review 18, written by the job `platform/review_corpus`, and the example checks that its `run_id` is the run of the job's `COMPLETE` event: `True`. That run read `crm.learners, learning.courses, learning.reviews`. Its lineage says the rows were filtered by `crm.learners.ai_training_consent, learning.reviews.learner_id`, the [`INDIRECT` inputs](#add-indirect-inputs) that decide which rows exist, and that `body` is a `TRANSFORMATION` of the review's text. Parsed from the pipeline's SQL as written, the same job's rows are filtered by nothing, and `body` is an `IDENTITY` copy: lineage that would say personal text was copied unchanged, with no consent column involved. The query as run is what happened, so it's what the lineage records.
Provenance makes the corpus checkable after the fact. Joining every record back through `learning.reviews` to its author finds 96 of 96 records whose author consented, a check the platform can run on any old build against today's answers.
Provenance also lets retrieval respect each reader's permissions. OWASP asks applications built on language models to "Implement fine-grained access controls and permission-aware vector and embedding stores" ([OWASP LLM08:2025](https://genai.owasp.org/llmrisk/llm082025-vector-and-embedding-weaknesses/)). The platform answers one person from an index built from this corpus, which keeps each record's `review_id` with its vectors. Before a retrieved record reaches the model, the retrieval layer asks the enforcement layer, as the person being answered, which of those reviews they may read, the way [chapter 6](#query-on-someones-behalf) runs an assistant's queries, and drops the rest. An instructor then gets only records from their own courses. A record that stands for reviews of several courses carries the first review's ID, so the check fails closed: for each of the 23 such records, it may hide a text the person could have read, and never shows one they couldn't.
**Working rule:** Give every record its source table, its row's ID, and the job and run that wrote it, and record the job's lineage from the query as it ran, filters and masks included. Check old builds against today's consent through that provenance. Filter every retrieval by the reader's own decision on each record's source row before a model sees it.
## Generate the dataset card from the run {#generate-the-dataset-card}
A dataset card tells whoever trains or audits a model what's in its data, where it came from and what was done to it. Croissant is MLCommons' metadata format for machine-learning datasets ([Akhtar et al., 2024](https://arxiv.org/abs/2403.19546)). Its 1.1 specification makes a card checkable: "Croissant datasets must declare that they conform to the versioned schema, e.g. http://mlcommons.org/croissant/1.1", and every dataset must give a description, licence, name, URL, creator and publication date. For provenance, "Croissant recommends using the W3C PROV Ontology (PROV-O)" ([Croissant 1.1](https://docs.mlcommons.org/croissant/docs/croissant-spec-1.1.html)). Its responsible-AI vocabulary adds properties such as `rai:dataCollection` and `rai:personalSensitiveInformation` ([Croissant RAI 1.0](https://docs.mlcommons.org/croissant/docs/croissant-rai-spec.html)). A card written by hand describes the data someone meant to build; a card generated by the run describes the data it built.
`lib.ai_data.card` generates the platform's card with mlcroissant 1.1.1, MLCommons' Python library for Croissant. The run supplies the counts, the corpus file's SHA-256, a field for every column, and three PROV-O properties: the tables the pipeline's own SQL reads, which the values come from, as `prov:wasDerivedFrom`; the table the enforcement layer's rewrite adds, which only decides which rows exist, as `prov:wasInfluencedBy`; and the run, as `prov:wasGeneratedBy`. mlcroissant writes them through its `extra_properties` argument but adds no `prov` prefix to the card's context, so the module adds it. The creator is the owner of `ai.review_corpus` in `governance/metadata.toml`. The licence, the name, URL, description and use cases come from `governance/ai_data.toml`, and `lib.ai_data.card_problems` is the platform's own check of the result. Generate the card, first with mlcroissant's default `conformsTo`:
Example `corpus-dataset-card` (Python 3.12.15 · datasketch 2.0.0 · duckdb 2.0.0 · mlcroissant 1.1.1 · openlineage-python 1.54.0 · phonenumbers 9.0.41 · presidio-analyzer 2.2.364 · presidio-anonymizer 2.2.364 · pyarrow 26.0.0 · pyiceberg 0.12.0 · pyyaml 6.0.3 · regopy 1.5.2 · sqlglot 30.24.0 · tldextract 5.4.0), passing in Edition 2026.10.1:
```python
from lib import ai_data
from lib.catalog import open_catalog
built = ai_data.build(open_catalog(["crm.learners", "learning.courses", "learning.reviews"]))
# mlcroissant's default, then the card the pipeline writes, with Croissant 1.1 written out.
default, _ = ai_data.card(built, conforms_to=[])
print("conformsTo, mlcroissant's default:", default["conformsTo"])
card, warnings = ai_data.card(built)
ai_data.write_card(card, built.folder)
print("conformsTo, written out:", ", ".join(card["conformsTo"]))
print("datePublished, the run's:", card["datePublished"])
print("version, the run's:", card["version"])
print("the sources the card names:", ", ".join(source["@id"] for source in card["prov:wasDerivedFrom"]))
print("the tables that decided which rows, as the card names them:",
", ".join(table["@id"] for table in card["prov:wasInfluencedBy"]))
print("the activity the card names:", card["prov:wasGeneratedBy"]["prov:label"])
print("fields in the record set:", ", ".join(f["name"] for f in card["recordSet"][0]["field"]))
for line in card["rai:dataPreprocessingProtocol"] + card["rai:personalSensitiveInformation"]:
print("-", line)
# What mlcroissant reports, and what the platform's own check of the card reports.
print("mlcroissant's warnings:", len(warnings))
for warning in warnings:
print("- mlcroissant:", warning.split("] ", 1)[1])
problems = ai_data.card_problems(card, built.folder)
print("problems the book's check finds:", len(problems))
for problem in problems:
print("-", problem)
```
Output:
```text
conformsTo, mlcroissant's default: http://mlcommons.org/croissant/1.0
conformsTo, written out: http://mlcommons.org/croissant/1.1, http://mlcommons.org/croissant/RAI/1.0
datePublished, the run's: 2026-10-01T02:00:00Z
version, the run's: 20261001.0.0
the sources the card names: iceberg://platform/learning.courses, iceberg://platform/learning.reviews
the tables that decided which rows, as the card names them: iceberg://platform/crm.learners
the activity the card names: platform/review_corpus
fields in the record set: review_id, course_id, course_title, rating, body, source_table, job, run_id
- Read 436 of 3,000 reviews, filtered on crm.learners.ai_training_consent, learning.reviews.learner_id.
- Replaced personal data with keyed pseudonyms (PERSON 120).
- Removed 340 exact duplicates and 0 near-duplicates (MinHash LSH, Jaccard similarity of 0.8 or more, confirmed exactly; 8 pairs that differ only in a name kept apart).
- Kept 96 records, each with its source table, review ID, job and run.
- Values replaced by keyed pseudonyms: PERSON 120.
- Emails and phone numbers replaced by [email] and [phone] by the enforcement layer's redact_contacts mask.
- Names found with a list of 106 words from crm.instructors, built by the governance team's job governance/instructor_names; the list isn't part of the corpus.
- Measured against labelled spans: 0 of 139 labelled values left in the text.
mlcroissant's warnings: 1
- mlcroissant: Property "http://mlcommons.org/croissant/citeAs" is recommended, but does not exist.
problems the book's check finds: 1
- creator: missing
```
Left to its default, mlcroissant writes `http://mlcommons.org/croissant/1.0`, although its own `LATEST_VERSION` is 1.1 ([mlcroissant source](https://github.com/mlcommons/croissant/blob/v1.1.1/python/mlcroissant/mlcroissant/_src/core/context.py)), so a card would declare a version it wasn't written for. The platform writes `http://mlcommons.org/croissant/1.1, http://mlcommons.org/croissant/RAI/1.0`. The card's `datePublished`, `2026-10-01T02:00:00Z`, is the build's time, because the platform publishes a card only with the build it describes. Its `version`, `20261001.0.0`, takes the build's date as its major number: Croissant recommends MAJOR.MINOR.PATCH, where a new major version means "the existing data has been changed (edited, removed or shuffled records across splits)", and any nightly build may remove records. The card names the sources `iceberg://platform/learning.courses, iceberg://platform/learning.reviews`, the influence `iceberg://platform/crm.learners` and the activity `platform/review_corpus`, and lists preprocessing steps with the run's counts.
The warnings mlcroissant reports, 1 in all, are about recommended properties, here `citeAs`, and none is about the problem the platform's check finds: `creator` is missing. `ai.review_corpus` has no owner in the metadata file, the gap [chapter 2](#test-the-catalog-against-the-metadata) found, so the card stays unpublished until the table has an owner, and [chapter 10](#run-every-rule-as-one-job) reports the same gap on every run, as a finding already on main. mlcroissant can also read a card back, with `mlc.Dataset`, but at 1.1.1, with rdflib 7.6.0 and pandas 3.1.0, reading emits a `DeprecationWarning` from rdflib and a `Pandas4Warning` and returns text fields as bytes, so the book's examples only write cards, and check them with the platform's own code.
**Working rule:** Generate the card in the run that builds the data, with every count, hash, source and version taken from that run, and write the format's version out. Check the card with your own code for every property the specification requires, since the library's validator doesn't report them all, and don't publish or train on a corpus whose card fails.
## Quote what the EU AI Act says about training data {#what-the-ai-act-says}
The EU's Artificial Intelligence Act, Regulation (EU) 2024/1689, addresses training data in several places. This section quotes two of them, Article 10 and Article 53(1)(d), and we interpret neither: whether any of it applies to the platform's course assistant is a legal question.
For high-risk AI systems, Article 10(2) says that training, validation and testing data sets "shall be subject to data governance and management practices", and lists what those practices concern, including "(b) data collection processes and the origin of data, and in the case of personal data, the original purpose of the data collection" and "(c) relevant data-preparation processing operations, such as annotation, labelling, cleaning, updating, enrichment and aggregation". Article 10(3) asks for data sets that are "relevant, sufficiently representative, and to the best extent possible, free of errors and complete in view of the intended purpose" ([AI Act](https://eur-lex.europa.eu/eli/reg/2024/1689/oj)). Regulation (EU) 2026/1744, the Digital Omnibus on AI, published Jul 24, 2026, leaves these passages as quoted. It replaces paragraphs 1 and 6 of Article 10, deletes its paragraph 5 ("paragraph 5 is deleted"), and inserts an Article 4a, "Processing of special categories of personal data for bias detection and correction". It also sets when Chapter III, Sections 1, 2 and 3, Article 10's section among them, apply: "(i) 2 December 2027 as regards AI systems classified as high-risk pursuant to Article 6(2) and Annex III; and (ii) 2 August 2028 as regards AI systems classified as high-risk pursuant to Article 6(1) and Annex I" ([Regulation (EU) 2026/1744](https://eur-lex.europa.eu/eli/reg/2026/1744/oj)).
For providers of general-purpose AI models, Article 53(1)(d) requires them to "draw up and make publicly available a sufficiently detailed summary about the content used for training of the general-purpose AI model, according to a template provided by the AI Office." The European Commission published that template as C(2025) 8311 ([Commission template](https://digital-strategy.ec.europa.eu/en/library/explanatory-notice-and-template-public-summary-training-content-general-purpose-ai-models), [PDF](https://ec.europa.eu/newsroom/dae/redirection/document/118480)), and its explanatory notice says that "An existing general-purpose AI model already placed on the Union market may be modified by a downstream entity in such a way that the downstream entity becomes the provider of the resulting general-purpose AI model, as specified in the Commission guidelines on General-Purpose AI models", and that then "the information reported by the modifying entity in the Template should be limited to the training content used for the model modification only". Whether fine-tuning a model would make the platform a provider is a question for its legal team, under the guidelines the notice names.
The Act's Annex III is titled "High-risk AI systems referred to in Article 6(2)", and its point 3, "Education and vocational training", includes "(b) AI systems intended to be used to evaluate learning outcomes, including when those outcomes are used to steer the learning process of natural persons in educational and vocational training institutions at all levels" ([AI Act](https://eur-lex.europa.eu/eli/reg/2024/1689/oj)). We quote it because the platform is an education platform, and stop there. Whether any system of the platform's is classified as high-risk under Article 6 is a question for its legal team, and none of this chapter's code depends on the answer.
**Working rule:** Quote the Act as amended, from the Official Journal, with a link and the date you read it, because the Omnibus moved both text and dates. Keep the law out of your pipeline's logic: generate the card's facts from the run, and let the people who answer legal questions decide which passages apply.
## What goes wrong in production {#ai-data-in-production}
A corpus fails quietly. The model trains, the card is published, and the problems surface later, in what the model says or in a request nobody can answer.
### A learner withdraws after the build {#withdrawn-after-the-build}
Like marketing's list in [chapter 7](#consent-at-query-time), a corpus records who had consented on the day it was built. Take the corpus that `lib.ai_data.build` writes in the nightly run of Oct 1, the Oct 1 build, and withdraw the consent of the author of one of its records. Then rebuild the corpus in the same folder the next night, without generating its card again, and check each build against today's consent, as a training run does before it starts:
Example `consent-withdrawn-after-build` (Python 3.12.15 · datasketch 2.0.0 · duckdb 2.0.0 · mlcroissant 1.1.1 · openlineage-python 1.54.0 · phonenumbers 9.0.41 · presidio-analyzer 2.2.364 · presidio-anonymizer 2.2.364 · pyarrow 26.0.0 · pyiceberg 0.12.0 · pyyaml 6.0.3 · regopy 1.5.2 · sqlglot 30.24.0 · tldextract 5.4.0), passing in Edition 2026.10.1:
```python
from datetime import timedelta
import pyarrow.compute as pc
import pyarrow.parquet as pq
from lib import ai_data, lineage
from lib.catalog import open_catalog
from lib.platform import load
catalog = open_catalog(["crm.learners", "learning.courses", "learning.reviews"])
first = ai_data.build(catalog) # the nightly run of Oct 1, and its card
card, _ = ai_data.card(first)
ai_data.write_card(card, first.folder)
# The next day, the author of one of the corpus's records withdraws their consent to AI training.
authors = dict(zip(*load("learning.reviews").select(["review_id", "learner_id"]).to_pydict().values()))
learner = authors[first.records.column("review_id")[1].as_py()]
learners = catalog.load_table("crm.learners")
rows = learners.scan().to_arrow()
consent = pc.if_else(pc.equal(rows["learner_id"], learner), False, rows["ai_training_consent"])
learners.overwrite(rows.set_column(rows.schema.get_field_index("ai_training_consent"), "ai_training_consent", consent))
# The next nightly run rebuilds the corpus in the same folder, and nobody generates the card again. A training run
# checks its build against today's consent before it starts.
second = ai_data.build(catalog, first.folder, lineage.NIGHTLY + timedelta(days=1))
for name, built in [("Oct 1 build", first), ("Oct 2 build", second)]:
ids = built.records.column("review_id").to_pylist()
print(f"{name}, records from the learner who withdrew:", sum(authors[i] == learner for i in ids))
print(f"{name}, records whose author consents today:", f"{ai_data.consenting(catalog, ids)} of {len(ids)}")
print("the two builds are different runs:", first.run_id != second.run_id)
print("a model made from the Oct 1 build, the legal team's answer:", ai_data.LEGAL["withdrawal"]["models_made_before"])
print("the Oct 1 card says:", card["rai:dataPreprocessingProtocol"][-1])
print("records in the corpus file now:", pq.read_metadata(first.folder / ai_data.CORPUS_FILE).num_rows)
problems = ai_data.card_problems(card, first.folder)
print("problems with the Oct 1 card, checked against the corpus now:", len(problems))
for problem in problems:
print("-", problem)
```
Output:
```text
Oct 1 build, records from the learner who withdrew: 1
Oct 1 build, records whose author consents today: 95 of 96
Oct 2 build, records from the learner who withdrew: 0
Oct 2 build, records whose author consents today: 95 of 95
the two builds are different runs: True
a model made from the Oct 1 build, the legal team's answer: undecided
the Oct 1 card says: Kept 96 records, each with its source table, review ID, job and run.
records in the corpus file now: 95
problems with the Oct 1 card, checked against the corpus now: 2
- creator: missing
- review_corpus.parquet: its SHA-256 isn't the one the card states
```
The Oct 1 build holds 1 record by that learner, and the Oct 2 build holds 0, so the next corpus leaves the learner out. Asked whether the two builds are different runs, the example answers `True`. The next training run has to leave the learner out too, and so does the next index. Checked through provenance against today's answers, the Oct 1 build has 95 of 96 records whose author consents, so a training run that makes the check refuses it, and the Oct 2 build passes with 95 of 95.
What was made from the Oct 1 build before the withdrawal keeps the review: a copy of its file, an index built from it, a model fine-tuned on it. The European Data Protection Board considers that "AI models trained with personal data cannot, in all cases, be considered anonymous" ([EDPB Opinion 28/2024](https://www.edpb.europa.eu/our-work-tools/our-documents/opinion-board-art-64/opinion-282024-certain-data-protection-aspects_en)). Whether any of them must change is for the platform's legal team. The GDPR's Article 7(3) says, in full: "The data subject shall have the right to withdraw his or her consent at any time. The withdrawal of consent shall not affect the lawfulness of processing based on consent before its withdrawal. Prior to giving consent, the data subject shall be informed thereof. It shall be as easy to withdraw as to give consent." ([GDPR](https://eur-lex.europa.eu/eli/reg/2016/679/oj)). The code's part is to find, through provenance, every build that holds the record, and by run ID every model and index made from those builds, and to read what happens to them from `[legal.withdrawal]` in `governance/ai_data.toml`. The example reads `undecided`, the placeholder, under which the job lists them for the legal team and changes nothing. If the answer is that a model must not keep the review, the one way to remove it that we can vouch for is to train the model again on a build without it.
The card failed too, without a sound. Its last preprocessing step still reads "Kept 96 records, each with its source table, review ID, job and run." The file holds 95 records, and the platform's check finds 2 problems: the missing creator, and a file whose SHA-256 isn't the one the card states. A card that isn't generated again with its data describes another dataset.
**Working rule:** Record which build, by run ID, every model and index was made from. Before every training run, check the build's records against today's consent and the file's hash against its card, and refuse a build that fails either. When a learner withdraws, find the builds that hold their records and the models made from them, and let your legal team's answer in config decide what happens to those.
### Other failures {#other-ai-data-failures}
- **A filter on refusals that keeps the unknowns.** Built with it, the corpus held 2,045 reviews instead of 436, mostly by learners never asked. Notice it by counting the corpus's records per consent answer, through their provenance: any answer but yes is a defect.
- **Personal data a model memorizes from its text.** Carlini and colleagues extracted training text from a language model that included public personal information, "names, phone numbers, and email addresses", "even though each of the above sequences are included in just one document" ([Carlini et al., 2021](https://arxiv.org/abs/2012.07805)). The plain pipeline's corpus holds 120 labelled emails, 113 of them distinct and 106 written only once, and duplicates repeat whatever a detector misses in every copy. Measure what's left against labelled samples of your own text, and deduplicate before training.
- **A pseudonym key that leaks.** With the corpus's key and a list of instructors' names, anyone can compute every pseudonym in the corpus. When the key may have leaked, rotate it, rebuild every corpus made with it, and find by run ID the copies and models made from builds under the old key, as for a withdrawal.
## What this chapter decided {#ai-data-decisions}
- **The corpus is built through the enforcement layer,** as `analyst-1` for `ai-training`: only reviews whose author's `ai_training_consent` is true, contacts masked, an unknown consent counted as no.
- **Every legal answer the code needs is a placeholder for the legal team** in `[legal]` in `governance/ai_data.toml`: the consent that admits a record, the card's licence, and what happens to models made before a withdrawal.
- **Personal data is detected after the enforcement layer's masks,** with the pattern recognizers and the governance team's list of instructors' names, and replaced by keyed pseudonyms. Privacy Filter is measured, not added. The corpus is pseudonymized, never anonymized.
- **Pseudonyms are HMAC-SHA256 under the corpus job's own key,** not the enforcement layer's SHA-256 over key and value, and never Presidio's randomly salted `hash`.
- **Duplicates are removed in two steps,** exact copies by SHA-256, then near-duplicates from MinHash LSH tuned for 0.6 and confirmed at a Jaccard similarity of 0.8, never merging texts that differ only in a name. datasketch and its scheme, `affine32`, are pinned.
- **Every record keeps its provenance,** `source_table`, `review_id`, `job` and `run_id`, and a training run checks its build against today's consent first.
- **The dataset card is Croissant 1.1 with RAI 1.0,** generated by the run, checked by the platform's own code, and unpublished until `ai.review_corpus` has an owner.
- **Answers for one person come from an index of the corpus,** filtered by that person's decision on each record's source review.
- **The EU AI Act is quoted as amended and linked, never interpreted.**
# Governance in CI
By the end of this chapter you can run the governance checks of chapters 2 to 7 as one job on each pull request, judging only what the pull request changes. It fails when a column has no classification, runs the tests of the access policy and the enforcement layer, reports what each schema change reaches downstream, and enforces two rules that breaking-change checkers, the tools that compare two versions of a schema for changes that break its readers, don't. You'll keep ownership in contracts of the Open Data Contract Standard (ODCS), generated from the metadata, with a folder and a code owner per domain, layer each domain's rules on global ones, and measure each check's noise by the findings nobody acts on.
The platform, the online learning platform this book follows, sells professional courses to adult learners and keeps its warehouse in an Iceberg catalog, with each table under the namespace of the team that runs it. Earlier chapters built its checks: [chapter 2](#a-catalog-you-can-test) tests the catalog against the reviewed metadata, [chapter 4](#lineage-from-pipelines) reads lineage from every pipeline run, [chapter 5](#access-policies-as-code) tests the access policy, [chapter 6](#enforcing-policies-on-queries) tests the layer that enforces it on queries, and [chapter 7](#retention-consent-erasure) generates table properties from the classifications. Each runs only when someone runs its example, and none runs when a team changes the platform. Three changes are on their way. The CRM team wants a column for learners' work email, the learning team wants to rename a column the reporting pipelines read, and the data-science team wants to see learners' cities without a mask. Each is a small diff a reviewer would approve in a minute, and each changes what breaks or who sees what.
This chapter adds the book's shared module for catching them, `lib.ci`, the platform's governance job; in this chapter, the job is this CI job, not an OpenLineage one. The examples simulate a pull request offline, as two folders called checkouts, one of the main branch and one of the pull request's branch. Each holds `governance/`, the platform's governance files; `pipelines/`, its pipelines' SQL; `migrations/`, the `alter table` statements a branch changes schemas with, which `lib.ci.migrate` applies as Iceberg schema changes; and the files [generated from the metadata](#a-folder-and-a-code-owner-per-domain). For each side, `lib.ci.build` builds a scratch platform the way the platform runs: the source tables, the side's migrations, the deploy, which writes the metadata into the catalog and sets every table's generated properties, then the nightly run of the side's own pipelines, with lineage. Among them, `review_corpus` stays [chapter 4](#lineage-from-pipelines)'s ungoverned copy, so the job keeps checking its lineage; the platform trains on [chapter 9's governed build](#admit-consented-reviews). `lib.ci.run` runs the checks and compares the two sides. No git or network is involved, and every date is compared with the book's "today", Oct 1, 2026, never the clock.
## Run every check as one job {#run-every-rule-as-one-job}
A check that runs when someone remembers it is a report, and reports go unread. On a pull request it's a gate: it runs on every change, before the change merges, and fails the pull request through its exit code. That is how Open Policy Agent (OPA) advises running policies in CI: its `--fail` and `--fail-defined` flags "Set the exit code to 1 based on query results" ([OPA CI/CD docs](https://www.openpolicyagent.org/docs/cicd)). A job made of several checks needs four things more:
- **Every check runs and reports all its findings,** so one failure doesn't hide the next. A check that crashes becomes one finding, "the check could not run", which fails the job while the other checks run.
- **Every finding has a stable ID,** so the same problem has the same name on every run and can be discussed, counted and waived. `lib.ci` writes it as the check and the subject, such as `coverage/crm.learners.work_email`.
- **The job judges what the pull request changes.** It runs the main branch's checks too, and compares. A finding the pull request introduces fails it. A finding already on main is reported on every run, counted and named with the team that owns it, but fails no pull request that leaves it as it is. As a rule of thumb, a check that fails every pull request for findings nobody in it caused gets switched off. Google's code-analysis platform, Tricorder, does much the same at code review: "we only display results for most analyses on changed lines by default" ([Sadowski et al., 2015](https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/43322.pdf)).
- **The list of checks lives in a reviewed file,** `governance/ci.toml`, so switching one off is a diff that someone approves.
The platform's job runs these checks, in this order:
| Check | From | Fails when |
| --- | --- | --- |
| `coverage` | Chapters [2](#test-the-catalog-against-the-metadata) and [3](#classify-columns-from-the-schema) | A table has no metadata or owner, or a column no classification or one below its name's suggestion |
| `contract-rules` | [This chapter](#run-global-and-domain-rules) | A contract breaks a global or a domain rule |
| `contracts` | [This chapter](#a-folder-and-a-code-owner-per-domain) | A generated file is out of date |
| `policy-tests` | [Chapter 5](#unit-test-the-policy-in-rego) | A Rego test or a who-sees-what row fails |
| `enforcement-tests` | [Chapter 6](#allow-only-catalog-sources) | A regression case or a who-sees-what query fails |
| `migrations` | [This chapter](#a-broken-file-becomes-a-finding) | Iceberg refuses a migration |
| `pipelines` | [Chapter 4](#fail-on-unreadable-lineage) | A pipeline can't run, or its lineage can't be read |
| `lineage-classifications` | [Chapter 4](#personal-data-tagged-internal) | A column is declared less protected than lineage proposes |
| `owner-review` | [This chapter](#ask-the-owner-before-lowering) | A classification drops, or an owner or domain changes, unapproved |
| `removals` | [This chapter](#deprecate-before-removing) | A column is removed before it was deprecated |
| `properties` | [Chapter 7](#keep-values-out-of-manifests) | A table lacks the properties generated for it |
The first pull request the job sees is the CRM team's. Employers buy team licences for their staff, so the team adds `work_email` to `crm.learners`, in one migration, and forgets the column's metadata:
Example `governance-ci-report` (Python 3.12.15 · duckdb 2.0.0 · openlineage-python 1.54.0 · phonenumbers 9.0.41 · presidio-analyzer 2.2.364 · pyarrow 26.0.0 · pyiceberg 0.12.0 · pyyaml 6.0.3 · regopy 1.5.2 · sqlglot 30.24.0 · tldextract 5.4.0), passing in Edition 2026.10.1:
```python
"""The platform's governance job on one pull request: the CRM team adds a work email to crm.learners."""
from lib import ci
from lib.platform import scratch
main = ci.checkout() # the main branch: governance files, pipelines and the files generated from them
branch = ci.checkout(main) # the pull request's branch, starting from main
(branch / "migrations" / "0001-add-work-email.sql").write_text("alter table crm.learners add column work_email varchar;\n")
print("files the pull request changes:", ", ".join(ci.changed(main, branch)))
# What a CI step runs, `python -m lib.ci MAIN BRANCH --summary FILE`: it prints the report and its totals, writes the
# report as Markdown for the pull request, and exits with the code it returns.
summary = scratch() / "summary.md"
code = ci.main([str(main), str(branch), "--summary", str(summary)])
print("the step exits with:", code)
print("the summary it writes for the pull request:")
print(summary.read_text(encoding="utf-8"), end="")
```
Output:
```text
files the pull request changes: migrations/0001-add-work-email.sql
coverage: failed, findings 3, new 1, on main 2, waived 0, 12 tables checked
contract-rules: passed, findings 0, new 0, on main 0, waived 0, 10 contracts checked
contracts: passed, findings 0, new 0, on main 0, waived 0, 11 files checked
policy-tests: passed, findings 0, new 0, on main 0, waived 0, 24 tests and rows checked
enforcement-tests: passed, findings 0, new 0, on main 0, waived 0, 44 cases checked
migrations: passed, findings 0, new 0, on main 0, waived 0, 1 migrations checked
pipelines: passed, findings 0, new 0, on main 0, waived 0, 4 pipelines checked
lineage-classifications: passed, findings 0, new 0, on main 0, waived 0, 23 columns checked
owner-review: passed, findings 0, new 0, on main 0, waived 0, 77 columns checked
removals: passed, findings 0, new 0, on main 0, waived 0, 87 columns checked
properties: passed, findings 4, new 0, on main 4, waived 0, 12 tables checked
coverage/ai.review_corpus: no metadata; on main, owner team-platform
coverage/crm.learners.work_email: not classified; its name suggests personal
coverage/marketing.audience: no metadata; on main, owner team-platform
properties/ai.review_corpus: 7 of 7 generated properties missing; on main, owner team-platform
properties/analytics.course_performance: 4 of 4 generated properties missing; on main, owner team-analytics
properties/analytics.learner_activity: 4 of 4 generated properties missing; on main, owner team-analytics
properties/marketing.audience: 7 of 7 generated properties missing; on main, owner team-platform
change crm.learners, field 16: added as work_email string; named by 0 pipelines, reaches 0 columns
job: failed
checks run: 11
checks failed: 1
findings: 7
findings the pull request introduces: 1
findings already on main: 6
findings waived: 0
findings approved: 0
exit code: 1
the step exits with: 1
the summary it writes for the pull request:
## Governance job: failed
| Check | Result | New | On main | Waived | Checked |
| --- | --- | ---: | ---: | ---: | --- |
| `coverage` | failed | 1 | 2 | 0 | 12 tables |
| `contract-rules` | passed | 0 | 0 | 0 | 10 contracts |
| `contracts` | passed | 0 | 0 | 0 | 11 files |
| `policy-tests` | passed | 0 | 0 | 0 | 24 tests and rows |
| `enforcement-tests` | passed | 0 | 0 | 0 | 44 cases |
| `migrations` | passed | 0 | 0 | 0 | 1 migrations |
| `pipelines` | passed | 0 | 0 | 0 | 4 pipelines |
| `lineage-classifications` | passed | 0 | 0 | 0 | 23 columns |
| `owner-review` | passed | 0 | 0 | 0 | 77 columns |
| `removals` | passed | 0 | 0 | 0 | 87 columns |
| `properties` | passed | 0 | 4 | 0 | 12 tables |
### To fix in this pull request
- `coverage/crm.learners.work_email`: not classified; its name suggests personal
### Already on main, and whose it is
- `coverage/ai.review_corpus`: no metadata (team-platform)
- `coverage/marketing.audience`: no metadata (team-platform)
- `properties/ai.review_corpus`: 7 of 7 generated properties missing (team-platform)
- `properties/analytics.course_performance`: 4 of 4 generated properties missing (team-analytics)
- `properties/analytics.learner_activity`: 4 of 4 generated properties missing (team-analytics)
- `properties/marketing.audience`: 7 of 7 generated properties missing (team-platform)
### Schema changes
- `crm.learners`, field 16: added as work_email string; named by 0 pipelines, reaches 0 columns
```
The pull request changes one file, `migrations/0001-add-work-email.sql`, and the job exits with code 1: of its 11 checks, 1 fails. The coverage check reports `failed, findings 3, new 1, on main 2, waived 0, 12 tables checked`. The new finding is the pull request's own: `crm.learners.work_email` is `not classified; its name suggests personal`. The suggestion comes from [chapter 3](#classify-columns-from-the-schema)'s name rules, which match `e_?mail`, so the finding says what to add. The two findings on main are `marketing.audience` and `ai.review_corpus`, the tables nobody has described since [chapter 2](#a-catalog-you-can-test) found them: `no metadata; on main, owner team-platform`. Neither table has an owner, so the job names the platform team, which `ci.toml` makes answerable for tables nobody owns. [Chapter 4](#classifications-from-lineage)'s lineage shows that both hold personal data, and every report repeats them until someone describes them.
The other checks pass, among them chapter 5's policy tests, on the data the branch's files generate, and chapter 6's enforcement tests, on the branch's tables. The properties check reports `passed, findings 4, new 0, on main 4, waived 0, 12 tables checked`: the tables pipelines write lose their generated properties on every nightly run, as [chapter 7](#verify-the-erasure) found, on main as on the branch, so each is a finding on main with its owner, such as `4 of 4 generated properties missing; on main, owner team-analytics`. Of the job's 7 findings, 1 is the pull request's own, and the schema change, `added as work_email string; named by 0 pipelines, reaches 0 columns`, reaches nothing yet.
**Working rule:** Run every check as one job on every pull request, each reporting all its findings under stable IDs, and a crashed check as a failing finding. Fail the job by exit code on any finding the pull request introduces that is neither waived nor approved. Report every finding already on main, counted and named with its owner, without failing a pull request that leaves it as it is. Keep the list of checks in a reviewed file.
## Gate the pull request, not the commit {#gate-the-pull-request}
[The platform's job](#run-every-rule-as-one-job) runs in CI, and a pre-commit hook is the other place a check can run: on the author's machine, before a commit exists. pre-commit, a framework for git hooks, runs them on the files a commit changes: "usually pre-commit will only run on the changed files during git hooks", and its documentation describes hooks as "useful for identifying simple issues before submission to code review" ([pre-commit](https://pre-commit.com/)). That suits checks that need one file: that `metadata.toml` parses, that every classification is one of the four values.
It doesn't suit the coverage check. The CRM team's pull request changed `migrations/0001-add-work-email.sql` and no metadata file, so a hook filtered to metadata files had nothing to check. The gap sits between the catalog, which the migration changed, and a file the pull request never touched, and only a job that builds the whole catalog sees it. A hook can be told to run anyway, with pre-commit's `always_run`, under which "this hook will run even if there are no matching files". But it runs only where someone installed it, and anyone can skip it, with pre-commit's `SKIP` variable or git's `--no-verify`, which no reviewer can see.
So the gate is the job on the pull request, made required: GitHub's branch protection can require that "all required status checks must pass before collaborators can merge changes into the protected branch" ([GitHub, protected branches](https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/managing-protected-branches/about-protected-branches)), and a status that isn't required is advice. Where the findings appear matters as much. [Tricorder](#run-every-rule-as-one-job)'s authors found that "when developers have to navigate to a dashboard or run a standalone command line tool, analysis usage drops off" ([Sadowski et al., 2015](https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/43322.pdf)). That is one company and code analysis, not data, but it's why `lib.ci.main`, which a CI step runs as `python -m lib.ci MAIN BRANCH`, does more than return the exit code, `1` for the CRM team. It prints one line per finding, with its ID and what to add, and writes the report as Markdown, the summary at the end of the example's output, for the pull request where the reviewer reads the diff. Posting the summary, and reading approvals from the pull request's reviews, are the CI service's part and outside this book's scope.
**Working rule:** Use hooks for fast checks of single files on the author's machine, and make the CI job on the pull request the gate, required by branch protection on the main branch. Put every finding in the pull request itself, with its ID and what to do about it, never only on a dashboard. Run its checks on a schedule against the production catalog too, for the tables that change outside any pull request.
## Report what a schema change reaches {#report-what-a-schema-change-reaches}
The CRM team's added column reached nothing downstream. A rename does, and its three kinds of readers judge it differently. Iceberg readers find columns by field ID: the spec says "projection must be done using field ids" ([Iceberg table spec, 1.12.0](https://github.com/apache/iceberg/blob/apache-iceberg-1.12.0/format/spec.md)), so a renamed column still reads every file written before the rename. SQL finds columns by name, so the same rename breaks every query and pipeline that names the column. The platform's governance metadata, keyed by field ID as [chapter 2](#key-column-metadata-by-field-id) decided, needs only the name beside the ID updated. Iceberg allows few type changes. The spec's "Valid primitive type promotions" are `int` to `long`, `float` to `double` and a decimal's precision ("Widen precision only"), plus, in format v3, `date` to `timestamp` or `timestamp_ns`, and a column of type `unknown` to any type. Any other change means a new column.
Which SQL a change breaks is a lineage question, and [chapter 4](#impact-of-a-column-change) answered it by walking the column graph. A review needs the walk to follow INDIRECT edges too, [chapter 4](#add-indirect-inputs)'s name for inputs that decide which rows an output holds rather than what its values are. OpenLineage's column lineage facet keeps a `dataset` list for "Column level lineage that affects the whole dataset. This includes filtering, sorting, grouping (aggregates), joining, window functions, etc." ([ColumnLineageDatasetFacet, 1.54.0](https://github.com/OpenLineage/OpenLineage/blob/1.54.0/spec/facets/ColumnLineageDatasetFacet.json)). A column used only to join or group shapes every row it feeds, which a report of copied values misses. `lib.ci.changes` compares the two sides' schemas field ID by field ID, and for every column the branch renames, retypes or removes, it walks the main branch's lineage graph: the pipelines whose SQL names the column, the columns whose values come from it, every column it can change, and the teams that own them.
The learning team's pull request renames `learning.courses.category` to `subject`, in a migration, and updates the name in the metadata file. Then it fixes the two pipelines that name the column, in the same pull request:
Example `schema-change-impact` (Python 3.12.15 · duckdb 2.0.0 · openlineage-python 1.54.0 · phonenumbers 9.0.41 · presidio-analyzer 2.2.364 · pyarrow 26.0.0 · pyiceberg 0.12.0 · pyyaml 6.0.3 · regopy 1.5.2 · sqlglot 30.24.0 · tldextract 5.4.0), passing in Edition 2026.10.1:
```python
"""What the learning team's rename of learning.courses.category reaches: Iceberg readers, SQL and lineage. Then the same
rename merged together with the fix of both pipelines that name the column."""
import pyarrow.compute as pc
from lib import ci, retention
main = ci.checkout()
branch = ci.checkout(main)
(branch / "migrations" / "0001-rename-category.sql").write_text(
"alter table learning.courses rename column category to subject;\n")
ci.edit(branch, "governance/metadata.toml", '3 = { name = "category", classification = "public"',
'3 = { name = "subject", classification = "public"') # the same field ID, renamed
before, after = ci.build(main), ci.build(branch)
change = next(c for c in ci.changes(before, after) if c.table == "learning.courses")
print("change:", change.kind, "from", change.before, "to", change.after)
print("field ID, before and after:", change.field_id)
subjects = after.catalog.load_table("learning.courses").scan(selected_fields=("subject",)).to_arrow()
print("courses whose subject an Iceberg reader finds:", pc.count(subjects["subject"]).as_py())
print("pipelines whose SQL names it:", ", ".join(change.jobs))
for job, why in sorted(after.failed.items()):
print(f"pipeline {job}, on the branch:", why)
print("columns whose values come from it:", len(change.values))
for column in change.values:
print(" value of:", column)
print("columns it can change, DIRECT and INDIRECT:", len(change.reached))
print("tables those columns are in:", ", ".join(sorted({".".join(c.split(".")[:2]) for c in change.reached})))
print("owners the report goes to:", ", ".join(change.owners))
print("tables on main:", len(retention.tables(before.catalog)))
print("tables on the branch:", len(retention.tables(after.catalog)))
coverage = ci.run(before, after, checks=["coverage"]).checks[0]
print("coverage findings on the branch:", ", ".join(finding.id for finding in coverage.findings))
# The same pull request, with both pipelines changed to read the new name. Each side runs its own pipelines/.
ci.edit(branch, "pipelines/course_performance.sql", " c.category,", " c.subject as category,")
ci.edit(branch, "pipelines/course_performance.sql", "c.title, c.category", "c.title, c.subject")
sql = (branch / "pipelines" / "marketing_audience.sql").read_text(encoding="utf-8")
(branch / "pipelines" / "marketing_audience.sql").write_text(sql.replace("c.category", "c.subject").replace(
"f.category", "f.subject"), encoding="utf-8")
fixed = ci.build(branch)
ci.generate(fixed) # and the contracts generated again, as the author's hook would
print("files the fixed pull request changes:", ", ".join(ci.changed(main, branch)))
for line in ci.lines(ci.run(before, fixed, checks=["pipelines", "coverage", "contracts", "lineage-classifications"])):
print("with the fix,", line)
```
Output:
```text
change: renamed from category string to subject string
field ID, before and after: 3
courses whose subject an Iceberg reader finds: 200
pipelines whose SQL names it: course_performance, marketing_audience
pipeline course_performance, on the branch: Unknown column: category
pipeline marketing_audience, on the branch: Unknown column: category
columns whose values come from it: 2
value of: analytics.course_performance.category
value of: marketing.audience.favourite_category
columns it can change, DIRECT and INDIRECT: 12
tables those columns are in: analytics.course_performance, marketing.audience
owners the report goes to: no owner, team-analytics
tables on main: 12
tables on the branch: 10
coverage findings on the branch: coverage/ai.review_corpus
files the fixed pull request changes: governance/contracts/learning/courses.odcs.yaml, governance/metadata.toml, migrations/0001-rename-category.sql, pipelines/course_performance.sql, pipelines/marketing_audience.sql
with the fix, pipelines: passed, findings 0, new 0, on main 0, waived 0, 4 pipelines checked
with the fix, coverage: passed, findings 2, new 0, on main 2, waived 0, 12 tables checked
with the fix, contracts: passed, findings 0, new 0, on main 0, waived 0, 11 files checked
with the fix, lineage-classifications: passed, findings 0, new 0, on main 0, waived 0, 23 columns checked
with the fix, coverage/ai.review_corpus: no metadata; on main, owner team-platform
with the fix, coverage/marketing.audience: no metadata; on main, owner team-platform
with the fix, change learning.courses, field 3: renamed from category string to subject string; named by 2 pipelines, reaches 12 columns
with the fix, job: passed
```
The rename kept field ID 3, and an Iceberg reader on the branch finds a subject for 200 courses: for Iceberg, nothing broke. For SQL, two pipelines name the column, `course_performance, marketing_audience`, and on the branch both stop before writing anything, with `Unknown column: category`, because [chapter 4](#fail-on-unreadable-lineage) made unreadable lineage an error. Through values, the rename reaches 2 columns, the two copies of the category. Through every edge it reaches 12, every column of `analytics.course_performance, marketing.audience`, because categories group the course report and rank each learner's favourite. The report goes to `no owner, team-analytics`: the analytics team, and nobody for `marketing.audience`, which has no owner yet.
The table counts show what a broken build hides. The branch's catalog holds 10 tables, against 12 on main, because the broken pipelines wrote nothing, and coverage reports only `coverage/ai.review_corpus`: a table that doesn't exist can't be found undescribed. Each side runs its own `pipelines/`, so the fix travels with the rename: with both pipelines reading `subject`, the job reports `passed, findings 0, new 0, on main 0, waived 0, 4 pipelines checked`, and `passed`. In production, the merge can also guard what the review saw: a REST catalog commit can carry `AssertCurrentSchemaId`, under which "The table's current schema id must match the requirement's `current-schema-id`" ([REST catalog spec, 1.12.0](https://github.com/apache/iceberg/blob/apache-iceberg-1.12.0/open-api/rest-catalog-open-api.yaml)), so a migration reviewed against one schema fails instead of applying to another.
**Working rule:** Diff schemas by field ID, never by name or schema ID, and report every rename, type change and removal three ways: whether Iceberg readers are affected, which pipelines' SQL names the column, and what it reaches through lineage, INDIRECT edges included. Send the report to the owners of every table it reaches. Merge a rename that breaks a pipeline in the same pull request as the pipeline's fix, never before it.
## Ask the owner before lowering a classification {#ask-the-owner-before-lowering}
Breaking-change checkers protect the readers of a schema. dbt's model contracts look for "breaking changes that could impact downstream consumers", such as "Removing an existing column" and "Changing the data_type of an existing column" ([dbt model contracts](https://docs.getdbt.com/docs/mesh/govern/model-contracts)). datacontract-cli's `breaking` compares two ODCS contracts and fails "when a contract change breaks a guarantee consumers rely on" ([datacontract-cli 1.2.7](https://github.com/datacontract/datacontract-cli/blob/v1.2.7/README.md)). Neither treats protection as a guarantee. In datacontract-cli 1.2.7, no rule covers a property's classification or a team member's role, so a change to either falls to a rule that marks it untested ([rules.py](https://github.com/datacontract/datacontract-cli/blob/v1.2.7/datacontract/breaking/rules.py)), and the detector reports it at INFO ([detector.py](https://github.com/datacontract/datacontract-cli/blob/v1.2.7/datacontract/breaking/detector.py)), a level that doesn't fail: the command exits 1 only for ERROR entries ([command_breaking.py](https://github.com/datacontract/datacontract-cli/blob/v1.2.7/datacontract/command_breaking.py), [model/breaking.py](https://github.com/datacontract/datacontract-cli/blob/v1.2.7/datacontract/model/breaking.py)). `examples/requirements.txt` doesn't pin it, so no example runs it. On Oct 22, 2026, datacontract-cli 1.2.7, run on the contracts this chapter generates for `crm.learners`, failed a renamed `email` with exit code 1, reading it as a removal although the property kept its `id`, because it matches properties by name ([normalize.py](https://github.com/datacontract/datacontract-cli/blob/v1.2.7/datacontract/changelog/normalize.py)). It passed `email` reclassified from `personal` to `public`, and a change of owner, with exit code 0. ODCS itself says a classification "Can be anything" ([ODCS v3.2.0 schema](https://github.com/bitol-io/open-data-contract-standard/blob/v3.2.0/docs/schema.md)).
Yet every policy reads the classification, so lowering one is among the most consequential changes a metadata file can carry. We add a first rule of our own: lowering a column's classification, changing a table's owner, or moving a table to another domain needs the approval of the team that owns the table on the main branch. A move is an ownership change, because a domain's own rules apply to its tables ([a later section](#run-global-and-domain-rules)). Taking the owner from the main branch matters, because otherwise a pull request could hand a table to a new owner, and the new owner could approve the downgrade. [Chapter 2](#measure-where-the-drafts-are-wrong) set the same rule for a model's drafts: only a reviewer lowers a classification, with the reason in the column's description.
The data-science team's pull request lowers `crm.learners.city` from `personal` to `internal`, so data scientists can group learners by city: the access policy pseudonymizes personal columns for data scientists, and this gets around it. Ask the policy before and after, then run the checks that look at schemas and policies, and the owner rule:
Example `classification-downgrade` (Python 3.12.15 · duckdb 2.0.0 · openlineage-python 1.54.0 · phonenumbers 9.0.41 · presidio-analyzer 2.2.364 · pyarrow 26.0.0 · pyiceberg 0.12.0 · pyyaml 6.0.3 · regopy 1.5.2 · sqlglot 30.24.0 · tldextract 5.4.0), passing in Edition 2026.10.1:
```python
"""A pull request from the data-science team lowers crm.learners.city from personal to internal."""
from lib import ci, policy
main = ci.checkout()
branch = ci.checkout(main)
ci.edit(branch, "governance/metadata.toml", '6 = { name = "city", classification = "personal"',
'6 = { name = "city", classification = "internal"')
before, after = ci.build(main), ci.build(branch)
reading_city = policy.request("analyst-1", "crm.learners", ["learner_id", "city"])
for side, name in ((before, "main"), (after, "the branch")):
rego = policy.interpreter(side.governance / "policies", data=ci.policy_data(side))
print(f"analyst-1 reading city, on {name}:", policy.decide(reading_city, rego)["column_masks"].get("city", "unmasked"))
checks = ["policy-tests", "enforcement-tests", "removals", "owner-review"]
report = ci.run(before, after, checks=checks)
print("schema changes:", len(report.changes))
for line in ci.lines(report):
print(line)
# The approvals come from the pull request's reviews: one entry per approving review, named by the reviewer's team.
owner_review = lambda approvals: "passed" if ci.run(before, after, approvals, checks=["owner-review"]).passed else "failed"
print("owner-review, once team-crm approves:", owner_review(["team-crm"]))
print("owner-review, once team-analytics approves:", owner_review(["team-analytics"]))
ci.edit(after.root, "governance/ci.toml", "teams = {}", "teams = { team-crm = 2 }") # team-crm asks for two
print("owner-review, two required, one from team-crm:", owner_review(["team-crm", "team-platform"]))
print("owner-review, two required, two from team-crm:", owner_review(["team-crm", "team-crm"]))
```
Output:
```text
analyst-1 reading city, on main: pseudonymize
analyst-1 reading city, on the branch: unmasked
schema changes: 0
policy-tests: passed, findings 0, new 0, on main 0, waived 0, 24 tests and rows checked
enforcement-tests: passed, findings 0, new 0, on main 0, waived 0, 44 cases checked
removals: passed, findings 0, new 0, on main 0, waived 0, 87 columns checked
owner-review: failed, findings 1, new 1, on main 0, waived 0, 77 columns checked
owner-review/crm.learners.city: lowered from personal to internal; needs team-crm's approval
job: failed
owner-review, once team-crm approves: passed
owner-review, once team-analytics approves: failed
owner-review, two required, one from team-crm: failed
owner-review, two required, two from team-crm: passed
```
On the main branch, `analyst-1` reads `city` as `pseudonymize`; on the branch, `unmasked`. One word in the metadata file unmasks a column for a whole role, and nothing else notices. The schema didn't change, so a breaking-change checker has nothing to compare: the job reports 0 schema changes. The policy tests pass, `passed, findings 0, new 0, on main 0, waived 0, 24 tests and rows checked`, and so do the enforcement tests, because none of them reads `city`, and the removals check passes because no column was removed. Only the owner rule fails: `lowered from personal to internal; needs team-crm's approval`. The example passes the approvals to `lib.ci.run`, one entry per approving review, named by the reviewer's team. With team-crm's approval, the owner rule reports `passed`; with team-analytics', which doesn't own the table, `failed`.
**Working rule:** Treat a lowered classification, a changed owner and a move to another domain as changes that need the approval of the table's owner on the main branch, and fail the job until it's given; never rely on a breaking-change checker for them. Ask for the reason in the column's description in the same pull request. A raised classification needs no such approval, because it can only hide more.
## Deprecate a column before removing it {#deprecate-before-removing}
[The owner rule](#ask-the-owner-before-lowering) guards a change breaking-change checkers ignore. Removing a column is the change they all catch, and catching it doesn't help the readers it breaks: when the check fails, the removal is already the change under review. ODCS 3.2.0 adds a way to warn them first: "Any element (a schema object or a property, including nested properties) may set the optional `deprecated` boolean to signal that it is no longer recommended for use", and the standard asks you to "Use the `description` field to point to a replacement and provide migration guidance" ([ODCS v3.2.0 schema](https://github.com/bitol-io/open-data-contract-standard/blob/v3.2.0/docs/schema.md)). Our second rule makes that warning a required step: a column may leave a table only once the main branch has marked it deprecated. That takes two pull requests, one that deprecates the column and one that removes it, and the column's readers have the time between them to move off it.
The platform's metadata file had no such field, so this chapter adds one: `deprecated = true` in a column's entry, which `lib.metadata.load` reads as `Column.deprecated`, false when it's absent, and `lib.ci.contract` writes into the column's contract as ODCS's `deprecated`.
The platform's best candidate is `billing.payments.card_number`, whose own description says it "Should not be stored at all", and which [chapter 3](#classify-columns-from-their-values) found holds full card numbers. team-billing removes it twice, each time by dropping the column in a migration and deleting its metadata entry: once straight away, and once after a pull request that deprecates it has merged:
Example `deprecate-then-remove` (Python 3.12.15 · duckdb 2.0.0 · openlineage-python 1.54.0 · phonenumbers 9.0.41 · presidio-analyzer 2.2.364 · pyarrow 26.0.0 · pyiceberg 0.12.0 · pyyaml 6.0.3 · regopy 1.5.2 · sqlglot 30.24.0 · tldextract 5.4.0), passing in Edition 2026.10.1:
```python
"""team-billing removes the legacy card_number column from billing.payments: once straight away, and once after a pull
request that deprecates it has merged."""
import yaml
from lib import ci
entry = ('8 = { name = "card_number", classification = "sensitive", '
'description = "The full card number, from a legacy export. Should not be stored at all." }\n')
deprecated_entry = entry.replace("all.", "all. Deprecated: reconcile payments by payment_id.").replace(" }", ", deprecated = true }")
def remove_card_number(base, line):
"""A pull request on top of `base` that drops the column and deletes its metadata entry."""
branch = ci.checkout(base)
(branch / "migrations" / "0001-drop-card-number.sql").write_text("alter table billing.payments drop column card_number;\n")
ci.edit(branch, "governance/metadata.toml", line, "")
return branch
main = ci.checkout()
deprecated = ci.checkout(main) # the first pull request: mark the column deprecated, and say what replaces it
ci.edit(deprecated, "governance/metadata.toml", entry, deprecated_entry)
for base, line, name in ((main, entry, "straight away"), (deprecated, deprecated_entry, "after the deprecation merged")):
report = ci.run(ci.build(base), ci.build(remove_card_number(base, line)), checks=["removals"])
print(f"removal, {name}:", "passed" if report.checks[0].passed else "failed")
for finding in report.checks[0].findings:
print(f" {finding.id}:", finding.message)
card_number = ci.contract(ci.build(deprecated), "billing.payments")["schema"][0]["properties"][7]
print("the deprecated column in its ODCS contract:")
print(yaml.safe_dump(card_number, sort_keys=False, width=200), end="")
```
Output:
```text
removal, straight away: failed
removals/billing.payments.card_number: removed without being deprecated on the main branch first
removal, after the deprecation merged: passed
the deprecated column in its ODCS contract:
id: field-8
name: card_number
logicalType: string
physicalType: string
required: false
description: 'The full card number, from a legacy export. Should not be stored at all. Deprecated: reconcile payments by payment_id.'
classification: sensitive
deprecated: true
```
Straight away, the removal fails, `removed without being deprecated on the main branch first`. After the deprecation has merged, the same removal reports `passed`. Between the two, the contract carried the warning to everyone who reads contracts rather than the repository: `deprecated: true`, and a description that says how to reconcile payments without the column. The rule checks the order of the two changes, not the time between them. How long readers get is a decision for the owning team and the column's readers, and [the impact report](#report-what-a-schema-change-reaches) says who those readers are.
**Working rule:** Remove a column in two pull requests: first mark it deprecated, with its replacement in the description, and merge that; then drop it from the table and its entry from the metadata. Fail any removal of a column the main branch hadn't deprecated. Agree with the column's readers how long the deprecation lasts, since the job checks only that it came first.
## Route each domain's reviews by path {#a-folder-and-a-code-owner-per-domain}
The [owner rule](#ask-the-owner-before-lowering) asks a table's team to approve its downgrades, and the repository can route that request. GitHub's code owners work by path: "Code owners are automatically requested for review when someone opens a pull request that modifies code that they own", and a branch protection rule can require their review ([GitHub, about code owners](https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/customizing-your-repository/about-code-owners)). The platform keeps all its metadata in one file, `governance/metadata.toml`, which suits 5 teams in one repository. A file has one set of owners, though, so the request can reach the right team only if each team's metadata lives under a path of its own.
ODCS draws the same lines. A contract names one `domain`, the "Name of the logical data domain" ([ODCS v3.2.0 fundamentals](https://github.com/bitol-io/open-data-contract-standard/blob/v3.2.0/docs/fundamentals.md)). ODCS has no owner field: ownership is a free-form `role` on a member of the contract's `team` ([ODCS v3.2.0 team](https://github.com/bitol-io/open-data-contract-standard/blob/v3.2.0/docs/team.md)). `lib.ci.contract` writes each described table as an ODCS contract: its domain, its owning team as a member with the role `owner`, and each column as a property whose `id`, which ODCS defines as "A unique identifier for the element used to create stable, refactor-safe references", holds its Iceberg field ID. `lib.ci.generate` writes each contract in a folder for its domain, and a CODEOWNERS file in `.github/`, the first place GitHub looks for one. Both are generated from the metadata, never typed. Take the data-science team's downgrade of `city` again, and ask whom GitHub would request:
Example `domain-contracts` (Python 3.12.15 · duckdb 2.0.0 · openlineage-python 1.54.0 · phonenumbers 9.0.41 · presidio-analyzer 2.2.364 · pyarrow 26.0.0 · pyiceberg 0.12.0 · pyyaml 6.0.3 · regopy 1.5.2 · sqlglot 30.24.0 · tldextract 5.4.0), passing in Edition 2026.10.1:
```python
"""Ownership as ODCS contracts, a folder per domain, and who GitHub asks to review the downgrade of crm.learners.city
when the metadata is one shared file, and when each domain has a folder of contracts generated from it."""
import yaml
from lib import ci
main = ci.checkout() # the main branch holds the contracts and CODEOWNERS generated from its metadata
branch = ci.checkout(main)
ci.edit(branch, "governance/metadata.toml", '6 = { name = "city", classification = "personal"',
'6 = { name = "city", classification = "internal"')
shared = ["/governance/ @example-org/team-platform"] # one file for every domain, owned by the platform team
for path, owners in ci.reviewers(shared, ["governance/metadata.toml"]).items():
print(f"one shared file, review of {path}:", " ".join(owners))
crm = yaml.safe_load((main / "governance/contracts/crm/learners.odcs.yaml").read_text(encoding="utf-8"))
print(yaml.safe_dump({"domain": crm["domain"], "team": crm["team"]}, sort_keys=False), end="")
lines = (main / ".github/CODEOWNERS").read_text(encoding="utf-8").splitlines()
for line in lines[1:]: # the first line says the file is generated
print("CODEOWNERS", line)
before, after = ci.build(main), ci.build(branch) # the pull request changes the metadata only
print("teams that own tables:", len({table.owner for table in before.described.values()}))
for line in ci.lines(ci.run(before, after, checks=["contracts"]))[:-1]:
print("before generating again,", line)
print("files generated again:", len(ci.generate(after)))
print("after generating again, contracts:", "passed" if ci.run(before, after, checks=["contracts"]).passed else "failed")
for path, owners in ci.reviewers(lines, ci.changed(main, branch)).items():
print(f"a folder per domain, review of {path}:", " ".join(owners))
```
Output:
```text
one shared file, review of governance/metadata.toml: @example-org/team-platform
domain: crm
team:
name: team-crm
members:
- username: team-crm
role: owner
CODEOWNERS /.github/CODEOWNERS @example-org/team-platform
CODEOWNERS /governance/ @example-org/team-platform
CODEOWNERS /migrations/ @example-org/team-platform
CODEOWNERS /pipelines/ @example-org/team-platform
CODEOWNERS /pipelines/course_performance.sql @example-org/team-analytics
CODEOWNERS /pipelines/learner_activity.sql @example-org/team-analytics
CODEOWNERS /governance/contracts/analytics/ @example-org/team-analytics
CODEOWNERS /governance/contracts/billing/ @example-org/team-billing
CODEOWNERS /governance/contracts/crm/ @example-org/team-crm
CODEOWNERS /governance/ci/domains/crm.rego @example-org/team-crm
CODEOWNERS /governance/contracts/learning/ @example-org/team-learning
CODEOWNERS /governance/contracts/support/ @example-org/team-support
teams that own tables: 5
before generating again, contracts: failed, findings 1, new 1, on main 0, waived 0, 11 files checked
before generating again, contracts/governance/contracts/crm/learners.odcs.yaml: differs from what the metadata generates: generate it again
files generated again: 11
after generating again, contracts: passed
a folder per domain, review of governance/contracts/crm/learners.odcs.yaml: @example-org/team-crm
a folder per domain, review of governance/metadata.toml: @example-org/team-platform
```
With one shared file, the downgrade changes `governance/metadata.toml`, and GitHub asks `@example-org/team-platform`, the team that owns the file, not team-crm, whose column it is. The generated CODEOWNERS gives the platform team the file itself, since "To protect a repository fully against unauthorized changes, you also need to define an owner for the CODEOWNERS file itself", along with the governance files, the migrations and the pipelines; each pipeline then goes to the team that owns its table, and each folder of contracts to its domain's team. The pull request changed only the metadata, so the `contracts` check, which generates every file again and compares, fails: `differs from what the metadata generates: generate it again`. Generated again, the pull request also changes the crm contract, and its review goes to `@example-org/team-crm`, since "the last matching pattern takes the most precedence". A hook that generates the files saves the author a failed run; the check makes the routing certain.
Required code-owner review has a limit: "an approval from any of the owners is sufficient to meet this requirement". Branch protection's count of approving reviews is one number for the whole branch, met when a pull request is "approved by the required number of reviewers with write permissions" ([GitHub, protected branches](https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/managing-protected-branches/about-protected-branches)), so any reviewer with write access counts toward it, and it can't ask for two of team-crm's members. The job can, from the approvals it's given: the owner rule counts them per team, against a number per team in `ci.toml`. With team-crm asking for two, [the downgrade of `city`](#ask-the-owner-before-lowering) with one team-crm approval reports `failed`, and with two, `passed`. Iceberg 1.12.0's REST catalog spec offers another place for classifications, `labels` per table and field ID ([REST catalog spec](https://github.com/apache/iceberg/blob/apache-iceberg-1.12.0/open-api/rest-catalog-open-api.yaml)). Coverage and the contract rules could read a catalog server's labels, but changes to them would need the server's own review, since the spec doesn't require that they be "persisted or versioned".
**Working rule:** Keep each domain's metadata under a path of its own, as one ODCS contract per table with one domain and one owning team, generated from the metadata and checked by the job, with a CODEOWNERS line for that team. Give the platform team the CODEOWNERS file, the job's settings, its global rules, the waivers and the migrations. Require code-owner review, and count approvals per team in the job when one isn't enough.
## Layer domain rules on the global rules {#run-global-and-domain-rules}
Teams that own [their domains' contracts](#a-folder-and-a-code-owner-per-domain) still share one platform, and some rules have to hold in every domain. Dehghani's description of data mesh calls this federated computational governance: "a set of global rules - rules applied to all data products and their interfaces - to ensure a healthy and interoperable ecosystem", beside the decisions each domain makes for itself, with the platform executing both ([Dehghani, 2020](https://martinfowler.com/articles/data-mesh-principles.html)). That's a framing, not a measurement, and it maps onto code: one package of global rules, which the platform team keeps and the domains agree on (in Dehghani's version, "the federation of domain data product owners and data platform product owners" decides them), a package per domain owned by its team, and one job that runs them all.
The OPA project's tool for rules over files is Conftest, "a utility to help you write tests against structured configuration data", which "relies on the Rego language" ([Conftest](https://www.conftest.dev/)). `lib.ci` runs the same kind of rules in-process with regopy, as [chapter 5](#access-policies-as-code) runs the access policy. `governance/ci/global.rego` holds the global rules: one owner, whatever the case of its role; one domain, which a missing, empty or blank value fails, since an empty string is a defined value in Rego and `not` alone would pass it; a classification from the four, since ODCS accepts any string; and a description for every column. `governance/ci/domains/crm.rego` holds team-crm's rule: nothing in crm is public, because everything in it describes a learner or an instructor. The `contract-rules` check runs the global package on every contract, and each domain's package on the contracts of that domain on the branch. A rule returns each violation as data, with the rule's name, a subject and a message, and the subject and the rule make the finding's ID. These files sit outside `governance/policies/`, the access policy's folder, so changing them never changes the policy revision that [chapter 8](#replay-old-decisions) records with every decision.
[The CRM team's pull request](#run-every-rule-as-one-job) comes back, this time with metadata: `work_email` is classified `public`, on the argument that a work address is business information, and its description is empty. Then it tries again, with a description and with `crm.learners` moved to the analytics domain, which has no rules of its own:
Example `global-and-domain-rules` (Python 3.12.15 · duckdb 2.0.0 · openlineage-python 1.54.0 · phonenumbers 9.0.41 · presidio-analyzer 2.2.364 · pyarrow 26.0.0 · pyiceberg 0.12.0 · pyyaml 6.0.3 · regopy 1.5.2 · sqlglot 30.24.0 · tldextract 5.4.0), passing in Edition 2026.10.1:
```python
"""The platform's global rules and the crm domain's own rule, run on every contract of the CRM team's pull request,
which this time classifies the new work_email column: as public, with no description. Then the same column described,
with crm.learners moved to the analytics domain, which has no rules of its own."""
from lib import ci
main = ci.checkout()
branch = ci.checkout(main)
(branch / "migrations" / "0001-add-work-email.sql").write_text("alter table crm.learners add column work_email varchar;\n")
ci.edit(branch, "governance/metadata.toml", '15 = { name = "notes"',
'16 = { name = "work_email", classification = "public", description = "" }\n15 = { name = "notes"')
for path in sorted((branch / "governance" / "ci").rglob("*.rego")):
print("rules:", path.relative_to(branch).as_posix())
before = ci.build(main)
report = ci.run(before, branch, checks=["coverage", "contract-rules"])
for line in ci.lines(report):
print(line)
# The second try: a description, and the table moved to a domain whose rules allow public columns.
ci.edit(branch, "governance/metadata.toml", 'description = "" }', 'description = "The learner\'s work address." }')
ci.edit(branch, "governance/metadata.toml", 'owner = "team-crm"\ndomain = "crm"\ndescription = "Every learner',
'owner = "team-crm"\ndomain = "analytics"\ndescription = "Every learner')
report = ci.run(before, branch, checks=["coverage", "contract-rules", "owner-review"])
for line in ci.lines(report):
print("moved to analytics,", line)
```
Output:
```text
rules: governance/ci/domains/crm.rego
rules: governance/ci/global.rego
coverage: failed, findings 3, new 1, on main 2, waived 0, 12 tables checked
contract-rules: failed, findings 2, new 2, on main 0, waived 0, 10 contracts checked
coverage/ai.review_corpus: no metadata; on main, owner team-platform
coverage/crm.learners.work_email: classified public; its name suggests personal; needs team-crm's approval
coverage/marketing.audience: no metadata; on main, owner team-platform
contract-rules/crm.learners.work_email/described: no description (global rule described)
contract-rules/crm.learners.work_email/no-public: crm classifies no column public (domains.crm rule no-public)
change crm.learners, field 16: added as work_email string; named by 0 pipelines, reaches 0 columns
job: failed
moved to analytics, coverage: failed, findings 3, new 1, on main 2, waived 0, 12 tables checked
moved to analytics, contract-rules: passed, findings 0, new 0, on main 0, waived 0, 10 contracts checked
moved to analytics, owner-review: failed, findings 1, new 1, on main 0, waived 0, 77 columns checked
moved to analytics, coverage/ai.review_corpus: no metadata; on main, owner team-platform
moved to analytics, coverage/crm.learners.work_email: classified public; its name suggests personal; needs team-crm's approval
moved to analytics, coverage/marketing.audience: no metadata; on main, owner team-platform
moved to analytics, owner-review/crm.learners/domain: domain changed from crm to analytics; needs team-crm's approval
moved to analytics, change crm.learners, field 16: added as work_email string; named by 0 pipelines, reaches 0 columns
moved to analytics, job: failed
```
Coverage now flags the column's classification: `classified public; its name suggests personal; needs team-crm's approval`. A name is evidence, and a pull request has no values to read, so a classification below the name's suggestion is a finding only the owner on main can answer. The contract rules ask more, `failed, findings 2, new 2, on main 0, waived 0, 10 contracts checked`. The global package finds `no description (global rule described)`, and the crm package `crm classifies no column public (domains.crm rule no-public)`. team-crm decided the second rule without asking any other team, and the platform runs it for them on every pull request, including pull requests from team-crm. On the second try, the analytics domain's rules apply, and the contract rules report `passed, findings 0, new 0, on main 0, waived 0, 10 contracts checked`. The move itself fails, `domain changed from crm to analytics; needs team-crm's approval`, and so does the classification.
**Working rule:** Write the rules every domain must follow as one global package, kept by the platform team, and let each domain add a package of its own, owned by its team, that runs on that domain's contracts only. Return violations as data with a stable subject, never as a bare pass or fail. Let domain rules add to the global rules, never replace them, and treat a table's move to another domain as an ownership change, so no team escapes its domain's rules by changing one word.
## Count waived findings per check {#count-waived-findings}
[The checks so far](#run-every-rule-as-one-job) each fail a pull request for a reason. A gate also fails if people stop believing it. [Tricorder](#run-every-rule-as-one-job)'s authors define "an effective false positive as any report from the tool where a user chooses not to take action to resolve the report", and keep that rate under 10% at code review. What they act on is narrower: the "not-useful rate", an analyzer's Not useful clicks as a share of its Not useful, Please fix and Apply fix clicks. "A rate ≥ 10% puts the analyzer on probation", and "If the rate goes above 25%, we may decide to turn the analyzer off immediately" ([Sadowski et al., 2015](https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/43322.pdf)). That is one company's code analysis, with thresholds of its own choosing, and we know of no published equivalent for data governance. The measure carries over as an analogy: count, for each check, the findings nobody acts on, waived or left on main.
On the platform, the decision not to act on a new finding is a waiver. A waiver in `governance/waivers.toml` names one finding by its ID, the team answerable for it, a reason, and the last day it applies. `lib.ci` accepts a waiver that runs at most 90 days past the day the job runs, a rule of thumb kept in `governance/ci.toml`, so every waiver comes back for a decision. The same file lists the checks no waiver silences, whatever its date: coverage, since a table with no owner or a column with no classification is fixed in the pull request, and the enforcement tests. A finding already on main needs no waiver. An expired waiver silences nothing, and the report lists it. The platform team owns the waivers file, so each waiver is a reviewed change too.
[The CRM team's pull request](#run-every-rule-as-one-job) asks for waivers twice: for its unclassified column, then, with the column classified `personal`, for the description it hasn't written, until the summer and then until Oct 31:
Example `waived-findings` (Python 3.12.15 · duckdb 2.0.0 · openlineage-python 1.54.0 · phonenumbers 9.0.41 · presidio-analyzer 2.2.364 · pyarrow 26.0.0 · pyiceberg 0.12.0 · pyyaml 6.0.3 · regopy 1.5.2 · sqlglot 30.24.0 · tldextract 5.4.0), passing in Edition 2026.10.1:
```python
"""Waivers on the CRM team's pull request: one for the unclassified work_email column, which coverage never allows;
then, with the column classified but not yet described, one that asks for too long, and one that fits, on the book's
today and a month later."""
from datetime import date
from lib import ci
from lib.platform import TODAY
def waive(branch, finding, expires, reason):
with open(branch / "governance" / "waivers.toml", "a", encoding="utf-8") as waivers:
waivers.write(f'\n[[waivers]]\nfinding = "{finding}"\nowner = "team-crm"\nreason = "{reason}"\nexpires = {expires}\n')
main = ci.checkout()
before = ci.build(main)
unclassified = ci.checkout(main)
(unclassified / "migrations" / "0001-add-work-email.sql").write_text("alter table crm.learners add column work_email varchar;\n")
waive(unclassified, "coverage/crm.learners.work_email", "2026-10-31", "Classified once the sign-up form is live.")
for line in ci.lines(ci.run(before, unclassified, checks=["coverage"])):
print("unclassified,", line)
undescribed = ci.checkout(main) # the second try: classified, with its description still to come
(undescribed / "migrations" / "0001-add-work-email.sql").write_text("alter table crm.learners add column work_email varchar;\n")
ci.edit(undescribed, "governance/metadata.toml", '15 = { name = "notes"',
'16 = { name = "work_email", classification = "personal", description = "" }\n15 = { name = "notes"')
waive(undescribed, "contract-rules/crm.learners.work_email/described", "2027-06-30", "Described once the form's copy is final.")
after = ci.build(undescribed)
for line in ci.lines(ci.run(before, after, checks=["coverage", "contract-rules"])):
print("undescribed, asked until 2027-06-30,", line)
ci.edit(undescribed, "governance/waivers.toml", "2027-06-30", "2026-10-31")
for day in (TODAY, date(2026, 11, 1)):
report = ci.run(before, after, today=day, checks=["coverage", "contract-rules"])
print(f"undescribed, asked until 2026-10-31, on {day}, contract-rules:", "passed" if report.checks[1].passed else "failed")
print(f"undescribed, asked until 2026-10-31, on {day}, findings waived:", sum(len(c.waived) for c in report.checks))
print(f"undescribed, asked until 2026-10-31, on {day}, waivers that silence nothing:",
"; ".join(why for _, why in report.unused) or "none")
```
Output:
```text
unclassified, coverage: failed, findings 3, new 1, on main 2, waived 0, 12 tables checked
unclassified, coverage/ai.review_corpus: no metadata; on main, owner team-platform
unclassified, coverage/crm.learners.work_email: not classified; its name suggests personal
unclassified, coverage/marketing.audience: no metadata; on main, owner team-platform
unclassified, change crm.learners, field 16: added as work_email string; named by 0 pipelines, reaches 0 columns
unclassified, waiver coverage/crm.learners.work_email: silences nothing, coverage findings are never waived
unclassified, job: failed
undescribed, asked until 2027-06-30, coverage: passed, findings 2, new 0, on main 2, waived 0, 12 tables checked
undescribed, asked until 2027-06-30, contract-rules: failed, findings 1, new 1, on main 0, waived 0, 10 contracts checked
undescribed, asked until 2027-06-30, coverage/ai.review_corpus: no metadata; on main, owner team-platform
undescribed, asked until 2027-06-30, coverage/marketing.audience: no metadata; on main, owner team-platform
undescribed, asked until 2027-06-30, contract-rules/crm.learners.work_email/described: no description (global rule described)
undescribed, asked until 2027-06-30, change crm.learners, field 16: added as work_email string; named by 0 pipelines, reaches 0 columns
undescribed, asked until 2027-06-30, waiver contract-rules/crm.learners.work_email/described: silences nothing, runs to 2027-06-30, past the longest allowed, 2026-12-30
undescribed, asked until 2027-06-30, job: failed
undescribed, asked until 2026-10-31, on 2026-10-01, contract-rules: passed
undescribed, asked until 2026-10-31, on 2026-10-01, findings waived: 1
undescribed, asked until 2026-10-31, on 2026-10-01, waivers that silence nothing: none
undescribed, asked until 2026-10-31, on 2026-11-01, contract-rules: failed
undescribed, asked until 2026-10-31, on 2026-11-01, findings waived: 0
undescribed, asked until 2026-10-31, on 2026-11-01, waivers that silence nothing: expired on 2026-10-31
```
The waiver for the unclassified column is refused, although it asks for one month: `silences nothing, coverage findings are never waived`. No waiver can make that pull request pass. The waiver for the description is refused while it runs to the summer, `silences nothing, runs to 2027-06-30, past the longest allowed, 2026-12-30`. Shortened, it silences the finding, and on Oct 1 the contract rules report `passed`. On Nov 1 they report `failed`, and the report lists the waiver as `expired on 2026-10-31`: the date passed, and the problem didn't.
**Working rule:** Let a waiver name exactly one finding a pull request introduces, with an owning team, a reason and an end date at most a few months away, and list the checks no waiver may silence, such as a missing classification or owner. Report, for each check, how many findings nobody acted on, waived or left on main, and review any check where that's most of them: either the check is wrong or the platform is, and the count hides which.
## What goes wrong in production {#ci-in-production}
As a rule of thumb, a governance job is switched off more often than it breaks: teams remove the checks that are slow or noisy, one line of `governance/ci.toml` at a time, and the gate goes on passing while it guards less. Two staged checks show how, with counts in place of timings, and a third shows a job that breaks.
### A check that reads every row {#a-check-that-reads-every-row}
The support team's pull request adds two columns to `support.tickets`: `callback_number`, classified `internal`, and `reply_email`, classified `public`. Then a team that wants values judged, not only names, adds [chapter 3](#classify-columns-from-their-values)'s classification of the table to the job, by sample and by every row:
Example `slow-check-every-row` (Python 3.12.15 · duckdb 2.0.0 · openlineage-python 1.54.0 · phonenumbers 9.0.41 · presidio-analyzer 2.2.364 · pyarrow 26.0.0 · pyiceberg 0.12.0 · pyyaml 6.0.3 · regopy 1.5.2 · sqlglot 30.24.0 · tldextract 5.4.0), passing in Edition 2026.10.1:
```python
"""The support team's pull request adds two columns to support.tickets. Coverage judges them by their names; chapter
3's classification of the table's values, added to the job, reads a sample or, the slow version, every row. Counts of
values read stand in for time, which depends on the machine."""
import pyarrow as pa
from lib import ci, pii
main = ci.checkout()
branch = ci.checkout(main)
(branch / "migrations" / "0001-callback-and-reply.sql").write_text(
"alter table support.tickets add column callback_number varchar;\n"
"alter table support.tickets add column reply_email varchar;\n")
ci.edit(branch, "governance/metadata.toml", '8 = { name = "status"',
'9 = { name = "callback_number", classification = "internal", description = "A number to call the learner back on." }\n'
'10 = { name = "reply_email", classification = "public", description = "Where the reply goes." }\n8 = { name = "status"')
before, after = ci.build(main), ci.build(branch)
report = ci.run(before, after, checks=["coverage", "classify-sampled", "classify-every-row"])
for name in ("body", "callback_number", "reply_email"):
values = after.catalog.load_table("support.tickets").scan(selected_fields=(name,)).to_arrow()[name]
print(f"support.tickets.{name}, values on the branch:", f"{len(values) - values.null_count:,}")
for check in report.checks:
found = [f.message for f in check.findings if f.id == f"{check.name}/support.tickets.{name}"]
print(f"support.tickets.{name}, {check.name}:", found[0] if found else "no finding")
for check in report.checks[1:]:
print(f"{check.name}, values read:", f"{check.checked:,}")
text_columns, sampled, every_row = 0, 0, 0
for name in sorted(before.described):
data = before.catalog.load_table(name).scan().to_arrow()
filled = [len(column) - column.null_count for column in data.columns if pa.types.is_string(column.type)]
text_columns += len(filled)
sampled += sum(min(pii.ROWS, count) for count in filled)
every_row += sum(filled)
print("text columns in the described tables:", text_columns)
print("values the sampled check would read in them:", f"{sampled:,}")
print("values the every-row check would read in them:", f"{every_row:,}")
```
Output:
```text
support.tickets.body, values on the branch: 2,000
support.tickets.body, coverage: no finding
support.tickets.body, classify-sampled: values support sensitive, reviewed as personal
support.tickets.body, classify-every-row: values support sensitive, reviewed as personal
support.tickets.callback_number, values on the branch: 0
support.tickets.callback_number, coverage: no finding
support.tickets.callback_number, classify-sampled: no finding
support.tickets.callback_number, classify-every-row: no finding
support.tickets.reply_email, values on the branch: 0
support.tickets.reply_email, coverage: classified public; its name suggests personal; needs team-support's approval
support.tickets.reply_email, classify-sampled: its name suggests personal, reviewed as public
support.tickets.reply_email, classify-every-row: its name suggests personal, reviewed as public
classify-sampled, values read: 1,000
classify-every-row, values read: 10,000
text columns in the described tables: 32
values the sampled check would read in them: 5,800
values the every-row check would read in them: 429,453
```
Coverage fails `reply_email`, `classified public; its name suggests personal; needs team-support's approval`, and passes `callback_number`, `no finding`, because no name rule matches it, and the branch holds 0 values in it. Reading values doesn't help, since the migration adds the column empty: every row gives `no finding`. [Chapter 3](#reclassify-on-schema-change)'s scheduled sample catches the phone numbers once the column fills. The sampled check reads 1,000 values and the every-row check 10,000, and both find `support.tickets.body` sensitive under a personal classification, which chapter 3 already sent to team-support. Across the described tables' 32 text columns, a sample reads 5,800 values and every row 429,453. The second number grows with the data, on every pull request, whatever it changed. As a rule of thumb, a job that keeps people waiting gets skipped, and then removed.
**Working rule:** Size pull-request checks by the change, not by the data: metadata, schemas, policies and the names of the columns a pull request adds. Run checks that read values on a schedule, with a fixed sample, as chapter 3 does, and send their findings to the owning team. Document where the job's checks stop, such as at a column added empty.
### A check that flags every ID {#a-check-that-flags-every-id}
A cautious team adds a check that treats every column named like an ID as personal data. The platform decided otherwise for the IDs it joins on: [chapter 2](#classify-pseudonymous-ids) classifies `learner_id` as `internal` and protects the table that resolves it. The check's findings are new, since the main branch's job doesn't run it, so each owning team waives them:
Example `noisy-check-waivers` (Python 3.12.15 · duckdb 2.0.0 · openlineage-python 1.54.0 · phonenumbers 9.0.41 · presidio-analyzer 2.2.364 · pyarrow 26.0.0 · pyiceberg 0.12.0 · pyyaml 6.0.3 · regopy 1.5.2 · sqlglot 30.24.0 · tldextract 5.4.0), passing in Edition 2026.10.1:
```python
"""A noisy check added to the job: every column named like an ID is personal. The platform decided otherwise for IDs
used for joining, so each owning team waives its findings, and the waiver count climbs."""
from collections import Counter
from lib import ci
main = ci.checkout()
branch = ci.checkout(main) # the platform team adds the check to the job's settings
ci.edit(branch, "governance/ci.toml", '"properties"]', '"properties", "id-columns"]')
before, after = ci.build(main), ci.build(branch)
report = ci.run(before, after)
print("waivers in the file, before:", len(ci.waivers(after.governance)))
noisy = next(check for check in report.checks if check.name == "id-columns")
print("job, before the waivers:", "passed" if report.passed else "failed")
print("id-columns, findings:", len(noisy.findings))
print("id-columns, findings on main:", len(noisy.on_main))
classified = Counter(finding.message.rpartition(" ")[2] for finding in noisy.findings)
for level, count in sorted(classified.items()):
print(f"id-columns, findings classified {level}:", count)
print("id-columns, the public one:", ", ".join(f.id.split("/")[1] for f in noisy.findings if f.message.endswith("public")))
with open(after.governance / "waivers.toml", "a", encoding="utf-8") as waivers:
for finding in noisy.findings:
waivers.write(f'\n[[waivers]]\nfinding = "{finding.id}"\nowner = "{finding.owner}"\n'
'reason = "An ID for joining, internal by decision."\nexpires = 2026-10-31\n')
report = ci.run(before, after)
print("waivers in the file, after:", len(ci.waivers(after.governance)))
print("job, with the waivers:", "passed" if report.passed else "failed")
for check in (check for check in report.checks if check.findings):
idle = [f for f in check.findings if f.waiver or f.on_main] # findings nobody acted on
print(f"{check.name}, not acted on:", f"{len(idle)} of {len(check.findings)} findings, {len(idle) / len(check.findings):.0%}: "
f"{len(check.waived)} waived, {len(check.on_main)} on main")
```
Output:
```text
waivers in the file, before: 0
job, before the waivers: failed
id-columns, findings: 21
id-columns, findings on main: 0
id-columns, findings classified internal: 20
id-columns, findings classified public: 1
id-columns, the public one: learning.courses.course_id
waivers in the file, after: 21
job, with the waivers: passed
coverage, not acted on: 2 of 2 findings, 100%: 0 waived, 2 on main
properties, not acted on: 4 of 4 findings, 100%: 0 waived, 4 on main
id-columns, not acted on: 21 of 21 findings, 100%: 21 waived, 0 on main
```
The check finds 21 columns, all IDs the platform classified on purpose: 20 `internal`, and `learning.courses.course_id`, which is `public`. The waivers file grows from 0 waivers to 21, and the ID check shows `21 of 21 findings, 100%: 21 waived, 0 on main`: past any threshold [Tricorder](#count-waived-findings)'s authors use, as an analogy only, since theirs count clicks, not waivers. The properties check shows the same share, `4 of 4 findings, 100%: 0 waived, 4 on main`, and it's right: its findings are a fix the pipeline runner needs. The count can't tell the two apart, and the reasons can.
**Working rule:** Before adding a check, run it on the main branch and read every finding; if most would be waived, change the rule rather than waiving its findings. Remove a check whose findings stay mostly waived, and treat a high share on a correct check as work to schedule, with an owner and a date.
### A broken file becomes a finding {#a-broken-file-becomes-a-finding}
A support agent is added to `governance/users.toml` with a quote missing, in a pull request that also changes `crm.learners.country` from text to a number and carries the CRM team's unclassified `work_email`:
Example `broken-files-become-findings` (Python 3.12.15 · duckdb 2.0.0 · openlineage-python 1.54.0 · phonenumbers 9.0.41 · presidio-analyzer 2.2.364 · pyarrow 26.0.0 · pyiceberg 0.12.0 · pyyaml 6.0.3 · regopy 1.5.2 · sqlglot 30.24.0 · tldextract 5.4.0), passing in Edition 2026.10.1:
```python
"""A pull request that breaks two files the job reads: a support agent added to the users file with a quote missing,
and a migration Iceberg refuses. It also carries the CRM team's unclassified work_email column."""
from lib import ci
main = ci.checkout()
branch = ci.checkout(main)
ci.edit(branch, "governance/users.toml", '[users.agent-a09]', '[users.agent-a12]\ngroups = ["support]\nregion = "emea"\n\n'
'[users.agent-a09]')
(branch / "migrations" / "0001-add-work-email.sql").write_text("alter table crm.learners add column work_email varchar;\n")
(branch / "migrations" / "0002-country-as-number.sql").write_text("alter table crm.learners alter column country type int;\n")
print("files the pull request changes:", ", ".join(ci.changed(main, branch)))
report = ci.run(main, branch)
for line in ci.lines(report):
print(line)
print("checks that could not run:", ", ".join(check.name for check in report.checks if check.error))
print("checks that ran:", sum(check.error is None for check in report.checks))
print("exit code:", report.exit_code)
```
Output:
```text
files the pull request changes: governance/users.toml, migrations/0001-add-work-email.sql, migrations/0002-country-as-number.sql
coverage: failed, findings 3, new 1, on main 2, waived 0, 12 tables checked
contract-rules: passed, findings 0, new 0, on main 0, waived 0, 10 contracts checked
contracts: passed, findings 0, new 0, on main 0, waived 0, 11 files checked
policy-tests: failed, the check could not run
enforcement-tests: failed, the check could not run
migrations: failed, findings 1, new 1, on main 0, waived 0, 2 migrations checked
pipelines: passed, findings 0, new 0, on main 0, waived 0, 4 pipelines checked
lineage-classifications: passed, findings 0, new 0, on main 0, waived 0, 23 columns checked
owner-review: passed, findings 0, new 0, on main 0, waived 0, 77 columns checked
removals: passed, findings 0, new 0, on main 0, waived 0, 87 columns checked
properties: passed, findings 4, new 0, on main 4, waived 0, 12 tables checked
coverage/ai.review_corpus: no metadata; on main, owner team-platform
coverage/crm.learners.work_email: not classified; its name suggests personal
coverage/marketing.audience: no metadata; on main, owner team-platform
policy-tests/error: the check could not run: governance/users.toml doesn't parse: Illegal character '\n' (at line 13, column 20)
enforcement-tests/error: the check could not run: governance/users.toml doesn't parse: Illegal character '\n' (at line 13, column 20)
migrations/0002-country-as-number.sql: refused: Cannot change column type: country: string -> int
properties/ai.review_corpus: 7 of 7 generated properties missing; on main, owner team-platform
properties/analytics.course_performance: 4 of 4 generated properties missing; on main, owner team-analytics
properties/analytics.learner_activity: 4 of 4 generated properties missing; on main, owner team-analytics
properties/marketing.audience: 7 of 7 generated properties missing; on main, owner team-platform
change crm.learners, field 16: added as work_email string; named by 0 pipelines, reaches 0 columns
job: failed
checks that could not run: policy-tests, enforcement-tests
checks that ran: 9
exit code: 1
```
The users file breaks the two checks that read it: `the check could not run: governance/users.toml doesn't parse: Illegal character '\n' (at line 13, column 20)`. Iceberg refuses the type change, which isn't a promotion, and the job reports `refused: Cannot change column type: country: string -> int`. The other 9 checks run, so the author also sees `not classified; its name suggests personal`, and the job exits 1. Without the catch, the job would stop at the first error, and the author would fix one problem per run.
**Working rule:** Turn a check that crashes into one finding that fails the job, with the error's first line and the file it names, run the other checks anyway, and never let a waiver silence it.
### Other failures {#other-ci-failures}
- **A gate nobody required.** A job whose status check branch protection doesn't require reports its findings, and the pull request merges anyway. Require it, and count merged pull requests whose job failed.
- **A broken build hides findings.** A table that a broken pipeline didn't write can't be found undescribed. Read the `pipelines` check first.
- **Waivers outlive their reason.** A finding gets fixed and its waiver stays, ready to hide the next one with the same ID. The report lists waivers that match no finding; delete them.
- **A rename changes finding IDs.** IDs name columns, so a renamed column's findings come back under new IDs, and the report lists their waivers as matching no finding. Move the waivers in the rename's pull request.
- **An approval outlives what it approved.** team-crm approves the downgrade of `city`, then the author pushes a downgrade of `email`, and the approval covers both. Have branch protection "dismiss stale pull request approvals when commits are pushed that affect the diff in the pull request".
- **One approval speaks for a team.** Any one code owner's approval meets GitHub's requirement, and branch protection's count is per branch. Count approvals per team in the job where a domain wants more.
- **The blocked team edits the gate.** A team whose pull request fails removes the check from `ci.toml`, adds a waiver, or edits CODEOWNERS. Give all three the platform team as code owner.
- **Changes outside pull requests.** A notebook creates a table, or a migration runs by hand. Run the job on a schedule against production too, as [chapter 2](#test-the-catalog-against-the-metadata) runs `problems`.
## What this chapter decided {#ci-decisions}
- **One job runs the checks of chapters 2 to 7 on every pull request,** `lib.ci.run`, listed in `governance/ci.toml`; only chapter 3's sampling of values runs on a schedule instead. `lib.ci.main` returns the exit code and writes the report as Markdown.
- **The job judges what the pull request changes.** A finding it introduces fails it unless approved or waived; a finding already on main is reported with its owner and fails nothing. A check that crashes is one failing finding.
- **Every finding has a stable ID,** the check and its subject, such as `coverage/crm.learners.work_email`.
- **A pull request is compared with the main branch on scratch platforms** built the way the platform runs, each with its own migrations and pipelines.
- **Schema changes are diffed by field ID** and reported three ways: Iceberg readers, the pipelines whose SQL names the column, and every column lineage says it reaches, INDIRECT edges included.
- **Two rules of our own:** lowering a classification, changing an owner or moving a table to another domain needs the approval of the table's owner on the main branch, counted per team; and a column is removed only after the main branch has marked it `deprecated`.
- **Ownership is an ODCS contract per table,** with one domain and the owning team in the role `owner`, generated into a folder per domain with a CODEOWNERS line, and checked by the job. The platform team owns CODEOWNERS, the job's settings, its global rules, the waivers and the migrations.
- **Global rules and domain rules are Rego,** run in-process with regopy on every contract, outside the access policy's folder.
- **The planted gaps stay on main:** `marketing.audience` and `ai.review_corpus` appear in every report as coverage findings, with the platform team as their owner, until they're described.
- **Noise is measured as findings nobody acts on, per check:** waived, or left on main. A waiver names one finding, an owner, a reason and an end date at most 90 days away; coverage and the enforcement tests are never waived.
- **Pull-request checks are sized by the change;** checks that read values run on a schedule.