feat(anonymization): suppress complements, within a row and across grains by blarghmatey · Pull Request #36 · mitodl/ol-analytics-api

blarghmatey · 2026-08-20T22:19:14Z

Stacked on #34 (feat/contract-scoped-endpoints) - review that first; this diff is only the commit on top.

Closes the complement-disclosure gap in the k-anonymity floor. Two channels, both reachable once #34 ships.

Within a row

The floor guarded only the small side of a count. A count that nearly fills the cohort containing it identifies the members who did not do the thing just as precisely as a small count identifies the few who did: 42 active learners of whom 40 used the chatbot names exactly 2 abstainers.

CohortPolicy gains contained_in, mapping a secondary count to the cohort it is a strict subset of. A subset whose complement is nonzero and below the floor is nulled along with its derived values. Ancestors are walked transitively, so video_watchers -> engaged_learners -> total_enrolled_learners is checked at both levels.

Every containment declaration was verified against the b2b_analytics dbt SQL on ol-data-platform@main, not assumed:

view	subset	container	why it holds
contract_utilization	active_learners, learners_certified	seats_consumed	both counted off the contract's own enrollments
enrollment_completion_funnel	active/passing/certified_learners	enrolled_learners	grades and certificates join `(user, course_run)` from the enrollment
monthly_engagement_trend	certified_learners, video_watchers, problem_attempters, chatbot_users	monthly_active_learners	each of those actions sets `active_count`
program_funnel	enrolled_via_program, program_course_completers	enrolled_in_contract_courses	same enrollment rows, filtered/joined
content_engagement_depth	engaged_learners	total_enrolled_learners	`active_count > 0` over the same learners
content_engagement_depth	video_watchers, problem_attempters, chatbot_users	engaged_learners	each action sets `active_count`
mit_admin_contract_health	active_learners, certified_learners	seats_consumed	same shape as contract_utilization

Counts that are not subsets stay out of it, declared uncontained with the reason in the model docstring, per the constraint carried over from #33:

enrolling_learners - active_count is set by navigation, discussion, videos, problems, chatbot or certificate activity (organization_administration_report.sql lines 301-307), and enrolled_count is not among them, so enrolling alone never makes a learner active. monthly_active_learners - enrolling_learners can go negative and is not a complement.
certificates_earned in content_engagement_depth - sum(certificate_count) counts events, not learners. Subtracting it from a cohort is arithmetic on two different units.

Leaving a secondary count out of both lists raises at policy-definition time. An undeclared containment silently skips the rule, which is the failure mode this whole change exists to close, so it is not allowed to be an oversight.

A negative complement fails closed (suppresses). It means the declaration is provably wrong for that row, and a bound we cannot trust is not a bound.

Across grains

#34 publishes the same learners at org and contract grain. The org engagement trend aggregates across an org's contracts, and its event sums add up exactly, so a contract-month the floor withholds is recoverable as org_total - sum(the visible contract months). Per the production check recorded on this task on 2026-08-14, 4 of 58 orgs hold more than one contract, so the arithmetic works on real data today; I did not re-run that query for this PR.

/organizations/{id}/engagement-trend now scans mv_b2b_contract_monthly_engagement_trend, runs those rows through suppress_small_cohorts exactly as the contract endpoint would, and blanks its own additive totals for whatever comes back NULL. The scan covers the whole org, not just the requested page, since the caller can subtract rows they read from any page of the contract endpoint. It is bounded by a row cap and fails closed on truncation: a scan that cannot prove it saw every contributing row is not allowed to certify a total as safe.

Blanking is per column, so a month where only the chatbot total is withheld downstream keeps its video and problem totals.

The first version of this asked a cheaper question - which contract rows does the floor drop - and that was wrong. Every guarded column is a derived column at the contract grain, so a contract row can clear its row gate and still publish NULL for one of them because the cohort it is attributable to is sub-floor. Reproduced against the real cohort policy: contract C1 with 30 active learners of whom 2 used the chatbot publishes its row with the chatbot total withheld, C2 publishes 500, the org publishes 517, and 517 - 500 returns the withheld 17 for a cohort of 2. Re-deriving the suppression rule in SQL was the wrong shape regardless - the complement rule is a fixpoint over transitive containment pairs, so a second implementation drifts, and it drifts toward publishing. Reusing the function is the fix; hidden_additive_columns catches all three ways a column goes missing (row dropped, cohort sub-floor, cohort taken by the complement rule).

Two deliberate non-actions:

Learner counts stay published. A learner active under two contracts is counted in both rows, so subtracting them bounds the withheld cohort rather than revealing it. Blanking them would cost the dashboard its headline number to defend against a bound.
The content-engagement pair gets no guard. A course run belongs to exactly one contract, so its org row and contract row hold identical counts (the same 2026-08-14 check found 1093/1093 rows equal on all nine numeric columns; also not re-run here) and the floor makes the same call on both. Nothing is aggregated away, so there is no remainder to subtract. Only the endpoint that needs a probe pays for one.

Known remaining gaps

The no-contract residual. organization_key in the source report falls back to a free-text field, and those rows carry no contract. For an org with such rows, org_total - sum(all contract rows) leaves a residual attributable to learners with no contract, which this guard does not cover because no contract row is withheld. Tracked separately; it is a different cohort from the one this change protects.

Inner funnel complements. EnrollmentCompletionFunnel declares active, passing and certified flat under enrolled_learners rather than chained, so active_learners - certified_learners is not checked. If certified learners really are a subset of active ones, that difference names the actives who did not certify. The chain is not declared because the view's SQL does not enforce it: active_learners filters on enrollment_is_active, and a learner who completed and later unenrolled is certified but not active; certificates are likewise joined on (user, course_run) without reference to is_passing. Declaring a containment that does not hold fails closed and deletes good data, so this needs an empirical check against production before it can be tightened. (ProgramFunnel is flat for a stronger reason: a learner can complete a contract course without having enrolled via the program pathway, so completers are genuinely not inside enrolled_via_program.)

The partition assumption is not asserted here. The argument that content-engagement needs no cross-grain guard rests on a course run belonging to exactly one contract. That is a property of the dbt models, not of this repo, and nothing here would fail if it stopped holding.

Verification

uv run ruff check ., uv run ruff format --check ., uv run mypy src and uv run pytest all pass locally: 234 tests, with core/anonymization.py and routers/organizations.py both at 100% line coverage. New tests cover the complement rule (near-total, zero complement, at-floor, negative, transitive-through-a-suppressed-container, uncontained exemption), every policy validation error, the cross-grain blanking (per key, no-op when nothing is withheld, no probe on endpoints without a finer grain), and the probe SQL. The column-contract test now also asserts that containment columns are real model fields and that each _FinerGrain declaration lines up with the contract endpoint it names.

Nothing here was exercised against live StarRocks. The suppression logic is response-layer and covered by unit and endpoint tests only.

🤖 Generated with Claude Code

https://claude.ai/code/session_01RVrNyjeL7gSVY61gSUavKf

github-actions · 2026-08-24T20:32:11Z

OpenAPI Changes

Show/hide changes

## b2b_dashboard.yaml: added

Unexpected changes? Ensure your branch is up-to-date with main (consider rebasing).

Copilot

Pull request overview

Adds complement-aware k-anonymity within rows and cross-grain suppression for organization engagement totals. It also introduces committed per-tenant OpenAPI contracts and validation tooling.

Changes:

Adds containment-based and cross-grain anonymization safeguards.
Updates B2B cohort policies and comprehensive tests.
Adds OpenAPI generation, drift checks, and diff automation.

Reviewed changes

Copilot reviewed 19 out of 21 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
`src/ol_analytics_api/core/anonymization.py`	Implements complement and cross-grain suppression.
`src/ol_analytics_api/core/db/query.py`	Adds finer-grain scanning and suppression integration.
`src/ol_analytics_api/tenants/b2b_dashboard/models.py`	Declares cohort containment relationships.
`src/ol_analytics_api/tenants/b2b_dashboard/routers/organizations.py`	Applies the contract-grain guard to org trends.
`src/ol_analytics_api/tenants/b2b_dashboard/routers/contracts.py`	Adds stable operation IDs.
`src/ol_analytics_api/tenants/b2b_dashboard/routers/admin.py`	Adds a stable operation ID.
`src/ol_analytics_api/main.py`	Adds stable tenant names.
`src/ol_analytics_api/openapi.py`	Generates tenant OpenAPI documents.
`bin/generate-openapi-spec`	Writes and checks committed specifications.
`openapi/specs/b2b_dashboard.yaml`	Publishes the B2B API contract.
`.github/workflows/openapi-diff.yml`	Reports and checks API changes.
`.github/workflows/ci.yml`	Verifies generated specifications.
`tests/test_anonymization.py`	Tests new suppression behavior.
`tests/test_column_contract.py`	Validates containment and finer-grain declarations.
`tests/test_endpoints.py`	Tests endpoint-level cross-grain suppression.
`tests/test_query_chokepoint.py`	Tests finer-grain query helpers.
`tests/test_openapi_spec.py`	Tests the published API contract.
`tests/test_lifespan.py`	Updates tenant fixtures.
`README.md`	Documents OpenAPI publication.
`pyproject.toml`	Adds OpenAPI tooling dependencies and lint coverage.
`uv.lock`	Locks the added dependencies.

Suppressed comments (1)

src/ol_analytics_api/tenants/b2b_dashboard/routers/organizations.py:242

This scan only observes contract rows, but the coarse org MV also includes source rows with no contract (as noted in the PR description). Therefore the declared additive columns are not actually sums of the scanned rows: an org total of 517 minus a visible contract total of 500 still reveals the 17-event no-contract residual, even when this scan finds nothing hidden. The guard needs to detect and suppress unmatched/no-contract contributions before the org totals can be safely published.

        scan_query = build_grain_scan(
            _SCHEMA,
            spec.finer_grain.mv,
            spec.finer_grain.model,
            filter_columns=(_ORG_FILTER_COLUMN,),
        )

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

…ains The k-anonymity floor only guarded the small side of a count. A count that nearly fills the cohort containing it identifies the members who did NOT do the thing just as precisely: 42 active learners of whom 40 used the chatbot names exactly 2 abstainers. CohortPolicy now declares which secondary counts are strict subsets of which (contained_in), verified against the b2b_analytics dbt SQL, and a subset whose complement is nonzero and below the floor is nulled along with its derived values. Ancestors are walked transitively, so watchers inside engaged inside enrolled is checked at both levels. Counts that are not subsets stay out of it, declared as uncontained with the reason: enrolling does not set active_count, so an org month's enrolling learners are not inside its active learners, and certificates_earned counts events rather than learners, where the subtraction has no meaning at all. Leaving a secondary count out of both lists raises at policy-definition time rather than silently skipping the rule. The second channel opened when the contract-scoped endpoints landed: the same learners are now published at org and contract grain, and the org engagement trend aggregates across an org's contracts. Its event sums add up exactly, so a contract-month the floor withholds is recoverable as org_total minus the visible contract months. 4 of 58 orgs hold more than one contract, so the arithmetic works on production data today. That endpoint now probes the contract-grained view for the months it withholds and blanks its own additive totals for them. The probe compares against the floor in SQL and projects the month alone, so the sub-floor count never enters the process. Learner counts are deliberately left published: a learner active under two contracts is counted in both rows, so subtracting them bounds the withheld cohort rather than revealing it. The content-engagement pair needs no guard at all, since a course run belongs to exactly one contract and its org row and contract row hold identical numbers, leaving no remainder to subtract. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RVrNyjeL7gSVY61gSUavKf

…t it drops The cross-grain guard asked the contract-grained MV a cheaper question than the one that matters: which rows does the floor DROP. Every column it protects is a derived column at that grain, and a contract row can clear its own row gate and still publish NULL for one, because the cohort that column is attributable to is sub-floor. A contract month with 30 active learners of whom 2 used the chatbot publishes its row and withholds its chatbot total; the probe saw nothing hidden, left the org total alone, and org minus the sibling contract handed the withheld number back. Reproduced against the real cohort policy before fixing: 517 - 500 = 17 interactions from a cohort of 2. Re-encoding the rule in SQL was the wrong shape to begin with. The complement rule is a fixpoint over transitive containment pairs, so a second implementation of it drifts, and it drifts toward publishing. The guard now scans the finer grain and runs those rows through suppress_small_cohorts itself, keeping only which additive columns come back NULL per key. One implementation of the governance decision, and it catches all three ways a column goes missing: the row dropped, its cohort sub-floor, or its cohort taken by the complement rule. Blanking is now per column rather than per key, so a month where only the chatbot total is withheld downstream keeps its video and problem totals. The scan is bounded and fails closed: a result that reaches the row cap cannot prove it saw every contributing row, and publishing a total the guard could not check is the failure the guard is for. Also from review: the additive columns were hand-listed with nothing checking the list was complete, so a sixth event sum added to the MV would have left its subtraction open with a green suite. Additive and non-additive must now partition the coarse model's derived columns, enforced at import. And complement_pairs() was documented as computed once per response while being called once per row; it is now hoisted to match. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RVrNyjeL7gSVY61gSUavKf

…#37) * feat(openapi): publish a per-tenant spec and generate clients from it MIT Learn consumes this service through hand-written types and a hand-written axios client that mirror models.py column for column. That was the right call for the first cut - nothing here published a client, and blocking the dashboard on a cross-repo publish pipeline was not worth it - but it means the frontend drifts silently every time a materialized view gains or renames a column. It already has: ol-analytics-api#33 made three engagement totals nullable and added seven columns, and mit-learn's types still say otherwise. Each tenant is a mounted sub-app, so it owns its own /openapi.json and the root app's schema contains none of it. `openapi.py` builds the apps through the same create_app() the server runs and takes each tenant's document from there, with two fixups that exist because the output is for a client generator rather than for the tenant's own /docs: paths are re-prefixed with the mount path, since Starlette strips it before the sub-app sees a request and a client pointed at the service host would otherwise call URLs that do not exist; and the document version is pinned rather than read from the package, so a release that changes no route produces no diff. Every route now names its own operation_id. That string becomes the generated client's method name, and FastAPI's default derives one from the function name and the whole path - `contractUtilizationOrganizationsOrganizationIdContract UtilizationGet`, renamed whenever the path moves. The tag prefix is also what keeps the org and contract routers' identically-named panels apart. Verified rather than assumed, since the org and contract endpoints are registered in a loop over a table of specs with a runtime-parametrized generic: openapi-generator v7.2.0 emits a distinct TypeScript interface per row model and per envelope (no collapse to one untyped OrgAnalyticsResponse), resolves the 3.1 `anyOf: [integer, null]` columns to `number | null`, and the result typechecks clean under `tsc --strict`. A test asserts the non-collapse so a future registration change cannot quietly undo it. The spec is committed because it is a cross-repo interface: ol-infrastructure's api_clients_pipeline watches openapi/specs/*.yaml on release and publishes the TypeScript package from it, the same arrangement behind @mitodl/mitxonline-api-axios. Drift fails CI twice over - as a test, and as a --check run of the generator, which is the only thing that exercises the generator at all. openapi-diff.yml comments the changelog on any PR touching a spec and fails on a breaking change, because breaking one here means breaking a client someone already shipped. Still to land before mit-learn can drop its hand-written client: the ol-analytics-api-clients repo and the PIPELINE_CONFIGS entry pointing at it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01AVSXwars1LvgtqV1YWhrB1 * fix(openapi): diff the union of base/head specs, pin oasdiff, describe the pipeline as future The base-only spec loop skipped added specs in the changelog and let the -f guard silently skip deleted specs in the breaking-change check -- deleting a whole published API would have passed CI. Iterate the union of base and head filenames instead, and fail explicitly on a removed spec. Pin oasdiff by digest in both invocations so an upstream image change can't silently alter breaking-change classification. The README, generator script, and workflow comment described the ol-infrastructure client-publishing pipeline as already wired up; none of it exists yet (no PIPELINE_CONFIGS entry, no release branch, MIT Learn still on its hand-written client). Reworded to the intended future state. Added a test asserting every route's operation_id is explicit: the existing uniqueness check still passes for a route that fell back to FastAPI's path-derived default, which is unique but not stable. --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>

… sums Copilot review on #36: the cross-grain guard only blanked additive event sums, on the reasoning that distinct-learner cohort counts (monthly_active_ learners and friends) aren't exactly additive across a org's contracts, so subtracting them only bounds a withheld cohort rather than revealing it. That reasoning holds when contracts overlap, but not when they don't -- two disjoint contracts sum exactly, so a hidden one's headline count comes back whole from `org_total - visible_contract_total`, the same as a hidden event sum would. Nothing available here can tell overlapping contracts from disjoint ones. CrossGrainAdditives gains guarded_cohorts: the coarse grain's primary and secondary cohort columns, blanked wholesale for any key the finer grain hides anything for, regardless of which specific column triggered it. organizations._register derives it from the coarse model's own cohort_policy, so nothing has to be hand-declared per endpoint the way additive_columns is. monthly_active_learners becomes Optional to carry the blank. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_015gRjAEb8aQ5KXSrqUVT8m7

blarghmatey · 2026-08-28T20:51:11Z

Addressed review feedback: the org/contract learner-count cross-grain leak is fixed in b953e6e (see reply on that thread) — 242 tests pass, mypy/ruff clean, OpenAPI spec regenerated. The other Copilot comment (org rows with no contract_id leaking a residual, at organizations.py:242) restates the "no-contract residual" gap already disclosed in this PR's description under "Known remaining gaps" — a different cohort (learners with no contract) than the one this PR protects, and deliberately out of scope here. No task currently tracks it; flagging that it should get one before it's forgotten.

blarghmatey · 2026-08-31T13:11:13Z

Tracked the no-contract residual as tk-the-no-contract-residual-leaks-a-suppressed-coho-b3e254 in the work graph (p1, linked to the B2B self-serve analytics project).

blarghmatey mentioned this pull request Aug 21, 2026

feat(openapi): publish a per-tenant spec and generate clients from it #37

Merged

blarghmatey requested a balanced review from Copilot August 25, 2026 20:20

Copilot started reviewing on behalf of blarghmatey August 25, 2026 20:21 View session

Copilot AI reviewed Aug 25, 2026

View reviewed changes

Comment thread src/ol_analytics_api/tenants/b2b_dashboard/routers/organizations.py Outdated

Base automatically changed from feat/contract-scoped-endpoints to main August 28, 2026 20:22

blarghmatey and others added 3 commits August 28, 2026 16:24

blarghmatey force-pushed the feat/complement-suppression branch from d23bd05 to 0628d12 Compare August 28, 2026 20:24

Navigation Menu

Uh oh!

feat(anonymization): suppress complements, within a row and across grains - #36

feat(anonymization): suppress complements, within a row and across grains#36
blarghmatey wants to merge 4 commits into
mainfrom
feat/complement-suppression

blarghmatey commented Aug 20, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Aug 24, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

blarghmatey commented Aug 28, 2026

Uh oh!

blarghmatey commented Aug 31, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

blarghmatey commented Aug 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Within a row

Across grains

Known remaining gaps

Verification

Uh oh!

github-actions Bot commented Aug 24, 2026

OpenAPI Changes

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

blarghmatey commented Aug 28, 2026

Uh oh!

blarghmatey commented Aug 31, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

blarghmatey commented Aug 20, 2026 •

edited

Loading