From fac45ed6eb636f0392184a38233a7591ecd02692 Mon Sep 17 00:00:00 2001 From: aliferous3 Date: Sun, 18 Oct 2026 21:18:47 +0400 Subject: [PATCH 1/9] docs: design crash-safe ownership checkpointing (Phase 1C-B0) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Resolves the spec §9.1 persistence-handoff blocker at the design level: proves the four-table model cannot carry exact crash-safe accumulation, and specifies the minimal amendment — one EPHEMERAL pseudonymous work table, a single-transaction CAS page checkpoint, per-scope two-pass identity resolution, and a pinned-revision walk (pending a B0.5 non-persisting spike). Records the proposed spec amendment in §27. Docs only — no code, schema, or config changes. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --- docs/ownership-checkpoint-architecture.md | 598 ++++++++++++++++++++++ docs/ownership-data-privacy-spec.md | 55 ++ 2 files changed, 653 insertions(+) create mode 100644 docs/ownership-checkpoint-architecture.md diff --git a/docs/ownership-checkpoint-architecture.md b/docs/ownership-checkpoint-architecture.md new file mode 100644 index 0000000..ed07dc7 --- /dev/null +++ b/docs/ownership-checkpoint-architecture.md @@ -0,0 +1,598 @@ +# Ownership Checkpoint Architecture — Phase 1C-B0 Design + +> **Status: design only. Nothing in this document is implemented or live.** +> +> This document resolves the §9.1 hard blocker in +> [ownership-data-privacy-spec.md](ownership-data-privacy-spec.md): how to run +> a `history(path:)` walk that spans many GitHub pages and many serverless +> invocations while preserving every privacy/minimization invariant and +> producing results **bit-identical** to the Phase 1C-A one-shot reducer. +> +> Inputs: `migrations/012_ownership.sql`, `lib/ownership/*` (merged via +> PR #78, head `cb0a6fa`), `docs/ownership-graphql-spike-results.md`, +> `scripts/backup-scope.mjs`. + +--- + +## 0. The contradiction, stated plainly + +The Phase 1C-A reducer is only correct when it sees the **complete** node set +of a scope before folding, because Top-25 membership, +`human_contributor_count`, the `other` fold, and the durable-identity +decision are all functions of the whole set. But a scope walk can span tens +or hundreds of pages, and a serverless invocation holds memory only for +~45–60 s. + +- Persist nothing per-contributor → a crash loses the counts; restart must + re-walk the prefix → no completion guarantee (Option A fails, §3.1). +- Persist every contributor's identity provisionally → overflow GitHub + identities reach storage → violates §6.4 durable-identity minimization + and §9 "in memory only". + +**Finding: there is no architecture that satisfies exactness, crash safety, +and identity minimization using only the four existing tables.** The minimum +amendment is one ephemeral, backup-excluded work table holding +*pseudonymous counters only* — no GitHub identity fields — plus one atomic +checkpoint primitive. This is §5 below; the required contract amendment is +recorded in §8 and in the spec's new proposed-amendment appendix. + +--- + +## 1. Fundamental verdict: four tables are NOT sufficient + +Checked against `012_ownership.sql` as deployed: + +| Candidate storage | Why it cannot carry work state | +|---|---| +| `ownership_snapshots` | Contributor rows require `contributor_id` → a durable `ownership_contributors` row → requires `github_node_id`/`display_login` non-null (`ownership_contributors_linked` CHECK). Overflow identities would be written. Also no month-keyed accumulation shape without polluting immutable period facts. | +| `ownership_contributors` | The linked CHECK **forces** `github_node_id` + `display_login` — pseudonym-only provisional rows are impossible by construction, and persisting identities is exactly what §9.1 forbids. | +| `ownership_sync_state` (`history_cursor`/other jsonb) | Technically possible, rejected in §3.2: it is OPERATIONAL (dumped into backups — provisional pseudonyms would leak into every dump), a single JSONB row is a read-modify-write hotspot with unbounded growth, and it cannot be CHECK-constrained per element. | +| `ownership_scopes` | Scope definition only; nothing accumulates there. | + +Conclusion: **migration 013 is required.** The delta is minimal (§11): one +work table + one checkpoint function. No existing column or constraint +changes. + +--- + +## 2. The central privacy question — what may persist provisionally + +Three distinct things must not be conflated: + +| Level | Content | Status | +|---|---|---| +| A. GitHub identity | `author.user.id` (node id), `login`, name, email | Durable only for *represented* contributors (§6.4); prohibited for overflow; emails/names prohibited always | +| B. Final `contributor_key` | `sha256(domain ∥ repo_uuid ∥ node_id)` | **Already durable** for represented contributors; opaque; carries no displayable identity | +| C. Build-scoped work key | any pseudonym valid during a build | Proposed for provisional counters | + +Key observation: **`contributor_key` itself is not an identity field.** It +is a repo-scoped SHA-256 over the internal repository uuid and the node id. +It cannot be rendered as a name, cannot be looked up on GitHub (GraphQL +node ids are opaque, non-enumerable strings — preimage search requires +guessing them), and the contract already persists it durably for every +represented contributor. + +Proposed rule (§8 amendment): **provisional work state may persist the same +`contributor_key` that would be durable if the contributor ends up +represented — and nothing else about identity.** Overflow contributors +therefore leave behind only an opaque 64-hex string that is deleted at scope +finalization, excluded from backups, cascades on repo deletion, and is swept +by TTL on abandoned builds. This is a *narrow* amendment: the §6.4 guarantee +"overflow-only contributors get no durable identity row" is preserved +verbatim — the work row is not an identity row and is never durable. + +Why not a build-nonce key (`hash(domain ∥ nonce ∥ repo ∥ node_id)`)? It adds +nothing the deterministic key doesn't already provide (both are opaque +pseudonyms under the same threat model — an actor who can preimage-attack +sha256 breaks either), and it **breaks Top-25 tie determinism** (§7): the +reducer ranks by `touch_commits desc, contributor_key asc`. With nonce keys +the ranking would need a key-mapping pass and two builds could diverge at +rank-25 ties. Deterministic keys make work-row ordering literally equal to +final ordering — proof by construction, not by argument. + +**Rejected alternative:** persisting `github_node_id`/`display_login` in a +provisional identity map (even ephemeral). That stores real identity for +overflow contributors — a *qualitative* expansion of stored PII, not a +quantitative one. It is the cheap engineering answer and the wrong privacy +answer. The two-pass design (§4) makes it unnecessary. + +--- + +## 3. Options evaluated + +### 3.1 Option A — no provisional persistence (restart-in-invocation) + +Correctness if a scope completes in one invocation: fine. The failure is the +completion guarantee: + +- Measured: ~0.5 s per 100-node history page (spike TEST B). +- Slice budget: 45 s webhook/cron slices inside `maxDuration: 60`. +- → ~80 pages ≈ 8,000 commits per invocation at best. +- A repository scope walks **all** history. A 20k-commit repository needs + ~200+ pages ≈ 100 s — **the repo scope alone can never finish in one + invocation.** Pathological months/dirs make it worse. +- Restart amplification: a crash at page N re-spends N pages of rate limit + and wall-clock; under shared 5,000 pt/hr quota this can starve sync. +- There is **no hard bound** on "a month/scope fits one invocation" — failure + is permanent, not slow. + +**Rejected — no completion guarantee.** + +### 3.2 Option B — JSONB accumulator inside `ownership_sync_state` + +- `ownership_sync_state` is classified **OPERATIONAL → included in every + pg_dump** (`backup-scope.mjs:40-47`). Provisional pseudonyms would be + written into every daily/weekly dump and retained per backup retention — + directly contrary to the ephemeral classification this data needs (§9). +- Single row per repo: every page checkpoint rewrites the whole JSONB — + O(accumulator size) write amplification per 100-node page; MVCC dead + tuples on a hot row; TOAST churn as the blob grows. +- Unbounded: `contributors × months` can reach tens of thousands of entries + (§6); no per-element CHECK constraint possible; a corrupt blob is + unparseable as a unit. +- Breaks the one-scope-at-a-time bound: the jsonb accumulates all in-flight + scopes at once. + +Atomicity is the one thing B gets right (single-row update is atomic), but +everything else fails. **Rejected.** + +### 3.3 Option C — provisional rows in existing ownership tables + +- `ownership_contributors_linked` CHECK *requires* `github_node_id` + + `display_login` for linked rows — the schema itself forbids pseudonym-only + provisional contributors. Weakening the CHECK to admit provisional rows + weakens the durable contract to accommodate the mechanism: backwards. +- Writing overflow identities "temporarily" is still writing them — §9.1 + names this explicitly. +- `ownership_snapshots` monthly rows are immutable period facts; using them + as accumulators forces dummy `contributor_id` values → same identity + problem. +- `active_version` gating does prevent *serving* partials, but the problem + here is *storing* identity, not *serving* it. + +**Rejected on both privacy and schema-fit grounds.** + +### 3.4 Option D — dedicated ephemeral work table (RECOMMENDED) + +One table, `ownership_history_work`, existing only for the duration of a +build. Scope-at-a-time processing bounds it to one scope's state. + +Minimal necessary shape (final column list is 013's; the contract is): + +| Column | Why it is necessary | +|---|---| +| `user_id`, `repository_id` | tenant composite FK → cascade deletes on repo/user removal | +| `scope_id` | composite FK `(scope_id, repository_id) → ownership_scopes` — ties work to one scope; file paths never appear | +| `calculation_version` | isolates build N+1 work from served N; enables failed-build GC by version | +| `period_start` | month key — the aggregation grain | +| `work_key` | `contributor_key` — or the literal `'unlinked'` for the single unresolved bucket; CHECK `work_key='unlinked' or work_key ~ '^[0-9a-f]{64}$'` prevents squatting and makes the two classes structurally disjoint | +| `is_bot` | boolean only — required because `human_contributor_count` needs the bot bit for **every** linked contributor including overflow-only ones, whose login is never persisted | +| `touch_commits` | the accumulated count — the point of the table | +| `first_seen_at`, `last_seen_at` | min/max `committedDate` within the month — preserves exact durable-candidate timestamps (see §5.3) | + +**Never stored here:** `github_node_id`, `login`, email, name, path, raw +node, SHA (the pin lives in `history_cursor`, §10), page payloads. + +It solves exact Top-25 (all contributions for the scope are present at +finalize), exact human count (distinct `work_key` where `not is_bot`), +crash recovery (page deltas + cursor are one transaction, §5.1), privacy +(no identity fields), bounded storage (§6), deletion (FK cascades + scope +delete + TTL sweep), and it is cleanly backup-excludable (§9). + +--- + +## 4. Recommended architecture + +**Option D + scope-sequential processing + per-scope two-pass identity +resolution + atomic page checkpoints + pinned-revision walks.** + +### 4.1 Build state machine + +`ownership_sync_state` (unchanged columns; `history_cursor` jsonb carries +all checkpoint detail): + +``` +pending → history → done | error (phase enum — unchanged) +``` + +`history_cursor` jsonb during `history`: + +```jsonc +{ + "pinned_oid": "<40-hex>", // build boundary, §10 + "scope_order": ["", ...], // deterministic + "current": { + "scope_id": "", + "pass": "accumulate" | "resolve", + "cursor": "" | null + }, + "lease": { "owner": "", "expires": "" } // §14 +} +``` + +`pending_scopes` jsonb: remaining scope ids (existing column, existing +purpose). `blame_cursor` untouched — HEAD blame is a later tranche. + +Per-scope lifecycle (scopes processed in `scope_order`; repo scope first — +its resolve pass captures the bulk of identities once): + +``` +ACCUMULATE walk history(pinned_oid, path=scope) page by page; + each page → own_checkpoint_page CAS tx (§5.1) + until hasNextPage=false +RESOLVE re-walk the same scope's pinned history (pass 2); + per page, inside the SAME CAS mechanism: + recompute work_key for each node; for keys ∈ represented set + (recomputed deterministically from work rows each invocation), + upsert the durable ownership_contributors row + {contributor_key, github_node_id, display_login, is_bot}; + skip pass entirely if every represented key already has a row +COMMIT single tx: load work rows → run the reducer algorithm → + upsert contributor first/last_seen_at → insert snapshot + contributor rows + scope-total rows (building_version) → + delete this scope's work rows → advance pending_scopes / + clear current cursor +``` + +When `pending_scopes` empties: promotion tx — validate zero remaining work +rows for `building_version`, expected snapshot cardinality within §11 +bounds, then `active_version = building_version; building_version = null; +phase = 'done'` in one update. Old-version GC is a separate, later +transaction (active version is already flipped — GC failure is harmless). + +### 4.2 Why the second pass is required, not optional + +At accumulate-time we deliberately cannot know which linked identities will +be represented. Capturing node_id/login for everyone during pass 1 = +provisional identity persistence for overflow contributors (rejected, §2). +The only way to keep identity fields out of provisional storage while still +producing durable contributor rows is to observe identity **after** the +represented set is known — i.e., a second walk of the same scope. + +Cost: pass 2 is another `history(path:)` walk of that scope at cost ≈1 +pt/page (measured). Sum over scopes ≈ 2× total history spend — bounded, and +the skip optimization (all represented keys already resolved by an earlier +scope) makes typical builds cheaper. + +**Resolve-drift edge case (stated honestly):** `author.user` is resolved by +GitHub at query time, not frozen at the pinned commit — a contributor can +unlink between pass 1 and pass 2 (account deleted, email relinked). Policy: +if a represented key resolves to `user = null` during pass 2, that key's +counts are folded into the **unlinked** bucket and finals are recomputed — +the truthful answer for the linkage state GitHub currently reports, exactly +what the one-shot reducer would emit had the stream shown null. If the key +remains *linked but never re-observed* in pass 2 (shouldn't happen under a +pinned revision — same commits), fail closed `OWNERSHIP_HISTORY_FAILED` and +retry once before erroring. The differential test (§12) does not simulate +live drift; equivalence holds against a fixed stream by construction. + +### 4.3 first_seen_at / last_seen_at + +Work rows keep min/max `committedDate` per (scope, key, month). At COMMIT: +contributor first/last = min/max over all the contributor's work rows across +all scopes (upsert `least`/`greatest` when a contributor is represented in +multiple scopes — contributor rows are repo-unique). This is value-identical +to the one-shot reducer's running min/max. No hidden problem — day-precision +is preserved; only the container changed. + +### 4.4 Unlinked + +All `author.user == null` nodes increment work rows with +`work_key='unlinked'` — one bucket, no per-author keys, no name/email ever +touched. Malformed present-but-broken identity still throws +`OwnershipIdentityError` → the page tx aborts → cursor doesn't advance → +error is surfaced via closed-taxonomy `OWNERSHIP_HISTORY_FAILED` and the +build stops (malformed identity is never silently reclassified). Retry +behavior for a persistent malformed node: the walk wedges at that cursor — +fail-closed by design, surfaced for a contract decision rather than skipped. + +### 4.5 Bots + +`is_bot` (login-suffix heuristic, pass 1) travels on the work row as a bare +boolean — the only way `human_contributor_count` can include/exclude +overflow contributors exactly without their logins. + +--- + +## 5. Exactly-once checkpointing + +### 5.1 The primitive — `own_checkpoint_page` (prospective 013 function) + +A SQL function, `security invoker`, `EXECUTE → service_role` only, taking: + +``` +p_user, p_repository, p_version, p_scope, +p_expected_cursor, p_next_cursor, p_pass, +p_deltas jsonb -- accumulate pass: [{period_start, work_key, + -- is_bot, touches, first_seen_at, last_seen_at}] +p_identities jsonb -- resolve pass: [{work_key, github_node_id, + -- display_login, is_bot}] +``` + +One transaction, in order: + +1. `SELECT ... FROM ownership_sync_state WHERE (user_id, repository_id) FOR UPDATE` + — serializes all ownership work for the repo; row is also the lease record. +2. Verify: row exists, `building_version = p_version`, `phase='history'`, + `current.scope_id = p_scope`, `current.pass = p_pass`, + `current.cursor IS NOT DISTINCT FROM p_expected_cursor`. +3. On any mismatch → `RETURN 'stale'` — **no writes.** (Lease expiry is a + caller concern, not a correctness gate.) +4. Apply `p_deltas`: set-valued upsert + `ON CONFLICT (scope_id, calculation_version, period_start, work_key) + DO UPDATE SET touch_commits = w.touch_commits + EXCLUDED.touch_commits, + first_seen_at = least(w.first_seen_at, EXCLUDED.first_seen_at), + last_seen_at = greatest(w.last_seen_at, EXCLUDED.last_seen_at)`. + Apply `p_identities`: upsert `ownership_contributors` on + `(repository_id, contributor_key)` refreshing `display_login`/`is_bot`. + Enforce the work-row cap inside the same statement (count check → raise + → abort → phase 'error' by caller). +5. `jsonb_set(history_cursor, '{current,cursor}', p_next_cursor)`. +6. Commit. + +### 5.2 Proof of the two crash windows + +**Invariant:** durable work state for scope S under version V always +reflects exactly the prefix of pages ending at durable cursor C. + +- Inductive step: a transaction either aborts entirely (state = prefix at C, + cursor still C — CASE B impossible: cursor cannot move without its deltas) + or commits (state = prefix at C′, cursor C′ — CASE A impossible: deltas + cannot land without the cursor moving). Postgres tx atomicity makes the + intermediate state unobservable. ∎ +- Retry after commit-with-lost-response: caller retries with + `p_expected_cursor = C`. Stored cursor is C′ ≠ C → step 2 fails → `'stale'`, + zero writes. The page's effects were applied exactly once — the retry is + a read. ∎ +- `touch_commits + delta` additive upserts are safe **only because** the CAS + guarantees a delta batch can be applied at most once per cursor value — + the rejected "write then advance" / "advance then write" orderings are + exactly CASE A / CASE B. + +Same CAS envelope protects COMMIT (expected "scope still current") and +promotion (expected "pending empty") — one mechanism everywhere. + +--- + +## 6. Working-state bounds (hard guard, fail-closed) + +Work rows exist only for the scope currently in `ACCUMULATE`/`RESOLVE`; +COMMIT deletes them before the next scope starts. + +| Bound | Proposed limit | Rationale | +|---|---|---| +| Distinct `work_key` per scope | 10,000 | ~5× facebook/react's repo-wide contributor count for a *single scope* — absurdly generous | +| Work rows per scope | 100,000 | keys × active months; e.g. 1k contributors × 100 months. ~10 MB transient | +| Page delta rows | ≤100 | one GraphQL page | +| JSONB cursor size | ~1 KB | sha + uuid + cursor strings | + +Breach → `phase='error'`, `error='OWNERSHIP_INTERNAL_ERROR'`, work rows +retained for operator inspection (backup-excluded, so they cost nothing +durable), deleted by the next build start or the TTL sweep. **Never** +silently partial — an over-cap scope is an error, not a folded result. + +TTL/cleanup: rows with `calculation_version` ≠ current `building_version`, +or orphaning an aborted/cleared build, are deleted (a) at build start, (b) +by a periodic sweep of work rows whose repo has no matching building_version +or is gone — plus unconditional cascade on repo/user delete via the +composite FK. + +--- + +## 7. Top-25 tie determinism + +Work key **is** the final `contributor_key`. COMMIT loads work rows and runs +the identical comparator — `touch_commits desc, contributor_key asc` — over +the identical complete set (same pinned revision ⇒ same node stream ⇒ same +counts). Two rebuilds on the same pinned revision + same +`calculation_version` produce identical membership including rank-25 ties. +The differential test (§12) is the executable proof. + +--- + +## 8. Required contract amendment (proposed, not live) + +Recorded in the spec appendix as PROPOSED — effective only if the +implementation tranche is approved: + +1. **Ephemeral pseudonymous working state** — §3 gains an explicit class: + `EPHEMERAL build-scoped counters` — `contributor_key`-keyed + (scope×month×key) counters + `is_bot` + first/last timestamps, existing + only while `building_version` is set, excluded from backups, cascade- + deleted, TTL-swept. **No GitHub identity fields on overflow + contributors** — §6.4's durable-identity guarantee is unchanged. +2. **Second persisted SHA** — `history_cursor.pinned_oid` joins + `last_processed_sha` as the only sanctioned SHAs (build boundary vs + incremental anchor). +3. **Two-pass re-derivation** — `author.user.{id,login}` may be re-read in + a resolve pass solely to populate durable contributor rows for + already-represented keys. + +Nothing else changes: no new identity fields, no finer granularity, no +relaxation of §5's prohibited list. + +--- + +## 9. Backups & restore + +- `ownership_history_work` → **EPHEMERAL** list in `backup-scope.mjs` + (classification is mandatory — an unclassified table fails CI). Not in + `BACKUP_TABLES`; never dumped. +- Verified against the assumption, not accepted on faith: losing work state + costs a restarted scope — cheap. Putting it in dumps would store + pseudonymous counters indefinitely for zero benefit. EPHEMERAL is correct. +- **Restore semantics:** restore brings back sync_state (OPERATIONAL) but no + work rows. On resume the engine detects + `building_version set ∧ work rows absent ∧ current scope mid-walk` → + resets `current.cursor = null` (restart that scope's walk) — pages are + re-fetched, CAS txs rebuild state, and no page can be double-applied + because cursor/delta atomicity is per-transaction, not per-backup. + `active_version` rows are untouched throughout. + +--- + +## 10. Revision pinning — the second blocker + +`defaultBranchRef.target` resolves **at query time**: page 1 can read HEAD@A +and page 40 HEAD@B after a push — silently mixing revisions inside one +scope's aggregates. + +**Design:** resolve `defaultBranchRef.target.oid` once at build start +(cost-1 probe) → `pinned_oid` in `history_cursor` → every walk uses + +```graphql +repository(owner: $owner, name: $name) { + object(oid: $oid) { + ... on Commit { + history(first: 100, path: $path, after: $cursor) { + pageInfo { hasNextPage endCursor } + nodes { committedDate author { user { id login } } } + } + } + } +} +``` + +Tree enumeration likewise pins: `object(oid:)` → `... on Commit { tree { oid } }` +then REST `git/trees/{tree_sha}?recursive=1` (the measured endpoint, now +sha-pinned). + +- Retries always carry `pinned_oid` — same revision by construction. +- Force-push making the oid unresolvable → fail closed + (`OWNERSHIP_HISTORY_FAILED`), `active_version` kept, next build re-pins. + Never continue mid-build on a new HEAD — that mixes revisions. +- **Not measured in Phase 0.5.** `object(oid:)`/`object(expression:)` was + used only for `blame` at literal `"HEAD"`. A pinned-oid `history(path:)` + walk, pagination behavior under `object`, and unreachable-oid semantics + (null vs `NOT_FOUND` error) are all unmeasured. +- **Gate: Phase 1C-B0.5 non-persisting spike** before implementation — + same spike rigor (metadata-only queries, no writes): pinned-oid history + walks + pagination, tree-sha enumeration, unreachable-oid behavior, cost + confirmation, and live `author.user` resolution under a pinned commit. + +--- + +## 11. Prospective migration 013 (DO NOT CREATE) + +`013_ownership_checkpointing.sql` — additive only: + +- `create table public.ownership_history_work` — columns per §3.4; + composite FKs `(repository_id, user_id) → repositories`, + `(scope_id, repository_id) → ownership_scopes(id, repository_id)`, + all `on delete cascade`; unique `(scope_id, calculation_version, + period_start, work_key)`; CHECKs on work_key shape and non-negative + counts; `calculation_version > 0`. +- RLS enabled, zero policies; revoke browser roles; `service_role` DML only. +- `own_checkpoint_page(...)` — §5.1; `security invoker`, `EXECUTE → + service_role`, all other grants revoked. +- `backup-scope.mjs`: append to `EPHEMERAL_TABLES` (same PR — CI requires it). +- Recovery-drill addition: assert the table exists post-migration and is + **absent** from dumps. +- No changes to the four existing tables: phase enum already contains + `history`; `resolve` is cursor-jsonb state, not a new phase; error + taxonomy already covers the failure modes. + +--- + +## 12. Differential correctness test (design) + +`test/ownership-checkpoint-equivalence.test.mjs` (Phase 1C-B deliverable): + +- **Harness:** one synthetic node stream per scope → + (A) `reduceHistoryToMonthly` once → expected output. + (B) simulated engine: partition the stream into pages; drive the + checkpoint algorithm over the pages (in-memory stand-in implementing the + same upsert/CAS semantics), then finalize → actual output. +- **Assert equality** on: per-month `total_touch_commits`, + `touch_commits`/`touch_share` per contributor row, + `human_contributor_count`, `unlinked_touch_share`, Top-25 membership + (exact key sets), `other` counts/shares, the durable + `contributor_candidates` set including `first_seen_at`/`last_seen_at`. +- **Adversarial pagination:** month spanning many pages; boundary splitting + a month mid-contributor-run; a contributor crossing rank 25 on the final + page; 30-way tie straddling rank 25; unlinked/bot nodes spread across + pages; replay of an already-committed page (must no-op via stale cursor); + simulated commit-success/response-lost; timeout between pages. +- The test fails if equivalence depends on pagination — that is the entire + point of the mechanism. + +--- + +## 13. Failure matrix + +| Case | Durable state | Retried? | No double-count | No skip | Cleanup | +|---|---|---|---|---|---| +| Crash before GitHub response | cursor=C, state=prefix(C) | yes, refetch page | page never applied | cursor still C | — | +| Crash after response, before tx | same | yes | tx never ran | same | — | +| Tx commits, response lost | cursor=C′, state=prefix(C′) | retry sees C′≠C → 'stale' | CAS rejects re-apply | page is applied | — | +| Retry same page | — | returns 'stale' | no writes on stale | — | — | +| Rate limit before page | unchanged | resume from C | nothing applied | — | error=GITHUB_RATE_LIMIT, resume later | +| Rate limit after response | response already in hand → tx completes or crashes clean | — | tx atomic | — | — | +| Malformed identity in page | tx aborts; cursor stays C | wedges at C — fail closed | — | page never committed | error=OWNERSHIP_HISTORY_FAILED; operator decides | +| DB timeout before commit | abort | retry applies | — | — | — | +| DB timeout, unknown outcome | maybe committed | retry: cursor==C′ → 'stale' (applied) or C → apply (not) | exactly-once either way | — | — | +| Worker timeout | as crash cases | next worker | CAS | — | lease expiry §14 | +| Deploy during build | in-flight tx aborts or completes | next deploy resumes | CAS | — | code must be read-compatible (additive 013) | +| Gate off mid-build | rows persist harmlessly | no caller → nothing runs | — | — | TTL sweep / next enable | +| Repo disconnected mid-build | cascade unchanged; sync gate stops calls | no | — | — | build stalls; resume on reconnect | +| Repo deleted mid-build | FK cascade removes work+sync rows | n/a | — | — | cascade is total — nothing to clean | +| Installation revoked | tx state intact | fails at next fetch | CAS | — | error=GITHUB_ACCESS_REVOKED | +| Pinned revision disappears | pages error; cursor=C | retry once → fail closed | partial scope state retained but never served (building_version) | — | error=OWNERSHIP_HISTORY_FAILED; next build re-pins | +| calculation_version changed | old-version work rows orphaned | n/a | version-scoped unique keys | — | GC by version at next build / sweep | +| Stale concurrent worker | lease + CAS | loser gets 'stale' | CAS rejects second apply | — | lease takeover after expiry | +| CAS cursor mismatch | — | reload state, refetch from durable C | — | — | — | + +--- + +## 14. Concurrency + +- **Per-repo, not per-user.** `ownership_sync_state` row is the unit of + contention — `user_sync` locking is untouched, so a huge ownership build + can never starve the same user's normal sync. +- **Soft lease** (`history_cursor.lease = {owner, expires}`): a worker + holds the build; a stale worker (expired lease) yields. Reduces wasted + fetches, is not the correctness mechanism. +- **Correctness backstop:** `SELECT FOR UPDATE` + cursor CAS inside + `own_checkpoint_page` — even two racing workers cannot double-apply. + +--- + +## 15. Rate limit / slice budget + +- ≤300 GraphQL pts per slice (spike recommendation — headroom for normal + sync within the 5,000 pt/hr floor). +- Wall-clock deadline ~40 s inside the 45 s slice; stop only at page + boundaries (a checkpoint is the only legal pause point). +- Count **measured** `rateLimit.cost` per page, not assumed 1 — stop when + `remaining` < 500 or the budget is spent; persist nothing about + rate limits beyond the closed `GITHUB_RATE_LIMIT` error code. +- Resume: next scheduled slice picks up at the durable cursor — no + ownership-specific cron wiring in this design (runtime wiring is 1C-B's). + +--- + +## 16. Decision standard — final answers + +| Question | Answer | Basis | +|---|---|---| +| A. Buildable without migration 013? | **NO** | §1 — no existing table can hold per-contributor intermediate state within the contract | +| B. Temporary per-contributor state unavoidable? | **YES** | §0 — exact fold requires complete sets; multi-invocation walks lose memory | +| C. Fifth ephemeral table required? | **YES** | §3 — only D satisfies privacy + bound + backup + cascade simultaneously | +| D. Two-pass identity resolution required? | **YES** | §4.2 — the only way to keep overflow identity fields unpersisted | +| E. Fixed-revision history query required? | **YES** | §10 — `defaultBranchRef` drifts mid-build | +| F. Additional measured spike required? | **YES** | §10 — `object(oid:)` history is unmeasured | +| G. Exact Top-25 preserved? | **YES** | §7 — work key = contributor key, identical comparator | +| H. Overflow identity fields unpersisted? | **YES** | §4.2 — pass 2 captures only represented keys | +| I. Every page transition exactly-once? | **YES** | §5.2 — single-transaction CAS, proven | +| J. Work state backup-excludable safely? | **YES** | §9 — loss = restarted scope, never user data | + +## 17. Readiness verdict + +**Ready to IMPLEMENT Phase 1C-B: NO.** Three prerequisites, in order: + +1. Approval of the §8 contract amendment (ephemeral pseudonymous work + state + pinned build SHA in sync-state jsonb). +2. Phase 1C-B0.5 non-persisting spike measuring pinned-revision history + walks, pinned tree enumeration, and unreachable-oid behavior. +3. Then: migration 013 + the engine implementation behind the existing + hard-off gate. diff --git a/docs/ownership-data-privacy-spec.md b/docs/ownership-data-privacy-spec.md index 5eb5aff..d63931c 100644 --- a/docs/ownership-data-privacy-spec.md +++ b/docs/ownership-data-privacy-spec.md @@ -1203,3 +1203,58 @@ ownership-cycle budget must be measured during staged rollout before GA | 4 | Monthly indefinitely vs compaction? | **Full monthly, no compaction in V1** — sparse writes + top-25/other + 300k/repo cap bound volume; compaction stays a version-gated option | | 5 | Separate HEAD table vs mixed? | **One `ownership_snapshots` table**; `period_granularity='head'` + sentinel `period_start` distinguishes HEAD rows | | 6 | Global vs repo-scoped contributor identity? | **Repository-scoped** — no global contributor table, no cross-tenant index or join, ever | + +--- + +## 27. PROPOSED amendment — ephemeral build working state (NOT LIVE) + +> **Status: proposed only.** Recorded by the Phase 1C-B0 checkpoint design +> ([ownership-checkpoint-architecture.md](ownership-checkpoint-architecture.md)). +> Nothing below is implemented, migrated, or approved for production. It +> becomes contract only if the implementation tranche is approved. + +The §9.1 blocker analysis reached a proved conclusion: **exact** +Top-25/Other membership and `human_contributor_count` require the complete +per-contributor set of a scope, while a `history(path:)` walk can span many +invocations — so exact crash-safe accumulation requires *some* durable +per-contributor intermediate state, and none of the four 012 tables can +hold it without violating this contract (snapshots require a durable +contributor FK; the linked-contributor CHECK forces identity fields; the +sync-state JSONB is backed up, unbounded, and unconstrainable). + +The proposed amendment is deliberately narrow: + +1. **New data class — EPHEMERAL build working state.** Per + (scope × month × contributor_key) touch counters plus the `is_bot` + boolean and min/max `committedDate`, existing only while + `building_version` is set for that repository, keyed by the **same + opaque `contributor_key`** already defined in §5/§6 (sha256 over + domain + repository uuid + node id). The key carries no node id, no + login, no name, no email, no path, and no raw nodes. +2. **The §6.4 guarantee is preserved verbatim** — overflow-only + contributors still get **no durable identity row** and their GitHub + identity fields (`github_node_id`, `display_login`) are still **never + persisted**. The work row is a counter, not an identity row; it is + deleted at scope finalization, is backup-excluded (EPHEMERAL class), + cascades on repository/account deletion, and is swept on abandoned + builds. +3. **Two-pass identity resolution.** GitHub identity fields + (`author.user.{id,login}`) are re-read in a second walk only to write + durable `ownership_contributors` rows for keys already known to be + represented — overflow identities are classified transiently (to + compute their work key) and discarded in memory per page, exactly as + §9 requires. +4. **Build-boundary SHA.** `ownership_sync_state.history_cursor` may carry + the build's pinned commit oid in addition to page cursors — a second + sanctioned SHA beside `last_processed_sha` (§5.12), required because + `defaultBranchRef` resolves at query time and multi-page builds must + not straddle pushes. +5. Rejected within the design: provisional persistence of + `github_node_id`/`display_login` for overflow contributors in any form + (ephemeral identity maps, provisional contributor rows), and JSONB + accumulators inside `ownership_sync_state` (backed up, unbounded). + +If this amendment is rejected, the honest alternatives are: (a) accept +per-scope restart-in-single-invocation with no completion guarantee on +long histories, or (b) accept approximate/page-local folding — both +rejected in the design document. From f84465373b09bd7c749a130e3992e9f2d5154c1f Mon Sep 17 00:00:00 2001 From: aliferous3 Date: Sun, 18 Oct 2026 21:32:59 +0400 Subject: [PATCH 2/9] =?UTF-8?q?docs:=20correct=20checkpoint=20architecture?= =?UTF-8?q?=20=E2=80=94=20atomic=20scope=20commit,=20staged=20identity?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Revision 2: move all durable contributor writes into the single scope-COMMIT transaction (no identity rows during page-by-page resolve), split ephemeral state into accumulation counters plus a represented-key-only identity staging relation, remove the unimplementable resolve-drift fold (a null author.user carries no node id to map back), reclassify work_key as a linkable pseudonymous identifier, and make TTL/cursor bounds concrete. Docs only. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --- docs/ownership-checkpoint-architecture.md | 739 ++++++++++++---------- docs/ownership-data-privacy-spec.md | 65 +- 2 files changed, 463 insertions(+), 341 deletions(-) diff --git a/docs/ownership-checkpoint-architecture.md b/docs/ownership-checkpoint-architecture.md index ed07dc7..b2f858a 100644 --- a/docs/ownership-checkpoint-architecture.md +++ b/docs/ownership-checkpoint-architecture.md @@ -1,6 +1,11 @@ # Ownership Checkpoint Architecture — Phase 1C-B0 Design -> **Status: design only. Nothing in this document is implemented or live.** +> **Status: design only, revision 2. Nothing in this document is implemented +> or live.** Revision 1→2 corrections: durable contributor writes moved +> entirely inside the atomic scope-COMMIT; a second ephemeral relation +> stages pass-2 identities; the unimplementable resolve-drift fold removed; +> `work_key` reclassified as a linkable pseudonymous identifier (not +> anonymous); concrete TTL/cleanup; cursor jsonb slimmed. > > This document resolves the §9.1 hard blocker in > [ownership-data-privacy-spec.md](ownership-data-privacy-spec.md): how to run @@ -31,10 +36,10 @@ or hundreds of pages, and a serverless invocation holds memory only for **Finding: there is no architecture that satisfies exactness, crash safety, and identity minimization using only the four existing tables.** The minimum -amendment is one ephemeral, backup-excluded work table holding -*pseudonymous counters only* — no GitHub identity fields — plus one atomic -checkpoint primitive. This is §5 below; the required contract amendment is -recorded in §8 and in the spec's new proposed-amendment appendix. +amendment is ephemeral, backup-excluded working state — counters plus a +narrow pass-2 identity-staging relation — plus one atomic checkpoint +primitive. §5 below; the required contract amendment is recorded in §8 and +in the spec's proposed-amendment appendix. --- @@ -45,62 +50,84 @@ Checked against `012_ownership.sql` as deployed: | Candidate storage | Why it cannot carry work state | |---|---| | `ownership_snapshots` | Contributor rows require `contributor_id` → a durable `ownership_contributors` row → requires `github_node_id`/`display_login` non-null (`ownership_contributors_linked` CHECK). Overflow identities would be written. Also no month-keyed accumulation shape without polluting immutable period facts. | -| `ownership_contributors` | The linked CHECK **forces** `github_node_id` + `display_login` — pseudonym-only provisional rows are impossible by construction, and persisting identities is exactly what §9.1 forbids. | +| `ownership_contributors` | The linked CHECK **forces** `github_node_id` + `display_login` — pseudonym-only provisional rows are impossible by construction, and persisting identities there is exactly what §9.1 forbids. It is also **retained, version-shared reference state** — not scoped to `calculation_version` — so it cannot be used as build scratch space at all (§4.4). | | `ownership_sync_state` (`history_cursor`/other jsonb) | Technically possible, rejected in §3.2: it is OPERATIONAL (dumped into backups — provisional pseudonyms would leak into every dump), a single JSONB row is a read-modify-write hotspot with unbounded growth, and it cannot be CHECK-constrained per element. | | `ownership_scopes` | Scope definition only; nothing accumulates there. | -Conclusion: **migration 013 is required.** The delta is minimal (§11): one -work table + one checkpoint function. No existing column or constraint -changes. +Conclusion: **migration 013 is required.** The delta is minimal (§11): two +ephemeral work relations + one checkpoint function. No existing column or +constraint changes. --- -## 2. The central privacy question — what may persist provisionally +## 2. Terminology — EPHEMERAL vs RETAINED -Three distinct things must not be conflated: +These words must be exact, because the work rows **are** persisted in +Postgres while a build runs: -| Level | Content | Status | -|---|---|---| -| A. GitHub identity | `author.user.id` (node id), `login`, name, email | Durable only for *represented* contributors (§6.4); prohibited for overflow; emails/names prohibited always | -| B. Final `contributor_key` | `sha256(domain ∥ repo_uuid ∥ node_id)` | **Already durable** for represented contributors; opaque; carries no displayable identity | -| C. Build-scoped work key | any pseudonym valid during a build | Proposed for provisional counters | - -Key observation: **`contributor_key` itself is not an identity field.** It -is a repo-scoped SHA-256 over the internal repository uuid and the node id. -It cannot be rendered as a name, cannot be looked up on GitHub (GraphQL -node ids are opaque, non-enumerable strings — preimage search requires -guessing them), and the contract already persists it durably for every -represented contributor. - -Proposed rule (§8 amendment): **provisional work state may persist the same -`contributor_key` that would be durable if the contributor ends up -represented — and nothing else about identity.** Overflow contributors -therefore leave behind only an opaque 64-hex string that is deleted at scope -finalization, excluded from backups, cascades on repo deletion, and is swept -by TTL on abandoned builds. This is a *narrow* amendment: the §6.4 guarantee -"overflow-only contributors get no durable identity row" is preserved -verbatim — the work row is not an identity row and is never durable. - -Why not a build-nonce key (`hash(domain ∥ nonce ∥ repo ∥ node_id)`)? It adds -nothing the deterministic key doesn't already provide (both are opaque -pseudonyms under the same threat model — an actor who can preimage-attack -sha256 breaks either), and it **breaks Top-25 tie determinism** (§7): the -reducer ranks by `touch_commits desc, contributor_key asc`. With nonce keys -the ranking would need a key-mapping pass and two builds could diverge at -rank-25 ties. Deterministic keys make work-row ordering literally equal to -final ordering — proof by construction, not by argument. - -**Rejected alternative:** persisting `github_node_id`/`display_login` in a -provisional identity map (even ephemeral). That stores real identity for -overflow contributors — a *qualitative* expansion of stored PII, not a -quantitative one. It is the cheap engineering answer and the wrong privacy -answer. The two-pass design (§4) makes it unnecessary. +- **EPHEMERAL / short-lived persisted operational state** — rows written + solely to make a multi-invocation build resumable. Intentionally + short-lived: deleted at scope finalization or on build-abort cleanup, + excluded from backups, never served by any API or RPC, never logged or + telemetered, fully reconstructable by restarting the build. Persistence + ≠ retention. +- **RETAINED ownership data** — `ownership_scopes`, + `ownership_contributors`, `ownership_snapshots` at `active_version`: + durable, backed-up, user-visible, deleted only by disconnect/account + deletion. + +The entire privacy boundary of this design is: *overflow contributors may +appear in ephemeral state only as the pseudonymous work key below, and +their GitHub identity fields may never appear in any persisted state.* + +--- + +## 3. `work_key` privacy classification — corrected + +`contributor_key` (`sha256(domain ∥ repo_uuid ∥ node_id)`) is a +**deterministic, repository-scoped, pseudonymous identifier derived from a +GitHub account identity**. Precisely: + +- contains no plaintext node id or login; +- is not user-displayable; +- is scoped to exactly one repository; +- **is linkable within that repository** — the same GitHub account always + produces the same key, and anyone holding the repository uuid plus a + candidate node id can test membership. It is therefore **not anonymous + data**, only pseudonymous. + +It may be used in persisted build state only under all of these conditions: + +- short-lived (deleted at scope finalization; swept on abandoned builds); +- `service_role`-only access, zero browser/API exposure; +- never logged, telemetered, or placed in error fields; +- excluded from backups; +- cascade-deleted on repository/account deletion. + +The stronger rule stands unchanged: **overflow-only contributors never +receive retained `ownership_contributors` identity rows** — the pseudonym's +ephemeral appearance does not create an identity record of any kind. + +Rejected alternative — a build-nonce key +(`hash(domain ∥ nonce ∥ repo ∥ node_id)`): it adds nothing under the same +threat model (both are sha256 pseudonyms; a preimage attacker breaks either) +and it **breaks Top-25 tie determinism** — the reducer ranks by +`touch_commits desc, contributor_key asc`. With nonce keys the rank-25 +tiebreak changes every build unless a key-mapping pass re-derives final +keys, an unforced correctness risk. Deterministic keys make work-row +ordering literally equal to final ordering — determinism by construction. + +**Rejected outright:** persisting `github_node_id`/`display_login` for +contributors not yet proven represented (an "optimistic" identity map). +That stores real identity for potentially-overflow contributors — a +*qualitative* expansion of stored PII, not a quantitative one. Pass-2 +staging (§4.4) exists only for keys already proven represented. --- -## 3. Options evaluated +## 4. Options evaluated -### 3.1 Option A — no provisional persistence (restart-in-invocation) +### 4.1 Option A — no provisional persistence (restart-in-invocation) Correctness if a scope completes in one invocation: fine. The failure is the completion guarantee: @@ -113,12 +140,12 @@ completion guarantee: invocation.** Pathological months/dirs make it worse. - Restart amplification: a crash at page N re-spends N pages of rate limit and wall-clock; under shared 5,000 pt/hr quota this can starve sync. -- There is **no hard bound** on "a month/scope fits one invocation" — failure - is permanent, not slow. +- There is **no hard bound** on "a month/scope fits one invocation" — + failure is permanent, not slow. **Rejected — no completion guarantee.** -### 3.2 Option B — JSONB accumulator inside `ownership_sync_state` +### 4.2 Option B — JSONB accumulator inside `ownership_sync_state` - `ownership_sync_state` is classified **OPERATIONAL → included in every pg_dump** (`backup-scope.mjs:40-47`). Provisional pseudonyms would be @@ -136,57 +163,93 @@ completion guarantee: Atomicity is the one thing B gets right (single-row update is atomic), but everything else fails. **Rejected.** -### 3.3 Option C — provisional rows in existing ownership tables +### 4.3 Option C — provisional rows in existing ownership tables - `ownership_contributors_linked` CHECK *requires* `github_node_id` + `display_login` for linked rows — the schema itself forbids pseudonym-only provisional contributors. Weakening the CHECK to admit provisional rows weakens the durable contract to accommodate the mechanism: backwards. -- Writing overflow identities "temporarily" is still writing them — §9.1 - names this explicitly. +- `ownership_contributors` is **retained reference state shared across + calculation versions** — a row written during an unfinished build would + survive aborts as an unreferenced identity row: durable identity for a + contributor that may end up represented nowhere. Revision-1 of this + document allowed exactly that; corrected in §4.4/§5. - `ownership_snapshots` monthly rows are immutable period facts; using them - as accumulators forces dummy `contributor_id` values → same identity - problem. -- `active_version` gating does prevent *serving* partials, but the problem - here is *storing* identity, not *serving* it. + as accumulators forces dummy `contributor_id` values → same problem. +- `active_version` gating prevents *serving* partials, but the problem here + is *storing* identity, not *serving* it. -**Rejected on both privacy and schema-fit grounds.** +**Rejected on privacy, retention, and schema-fit grounds.** -### 3.4 Option D — dedicated ephemeral work table (RECOMMENDED) +### 4.4 Option D — dedicated ephemeral working state (RECOMMENDED) -One table, `ownership_history_work`, existing only for the duration of a -build. Scope-at-a-time processing bounds it to one scope's state. +**Two** narrow ephemeral relations — not one. A single wide table would +either duplicate identity fields into every month row or conflate counters +with identity staging; the normalized pair is the cleaner privacy boundary: -Minimal necessary shape (final column list is 013's; the contract is): +**`ownership_history_work`** — accumulation counters only: | Column | Why it is necessary | |---|---| -| `user_id`, `repository_id` | tenant composite FK → cascade deletes on repo/user removal | -| `scope_id` | composite FK `(scope_id, repository_id) → ownership_scopes` — ties work to one scope; file paths never appear | -| `calculation_version` | isolates build N+1 work from served N; enables failed-build GC by version | +| `user_id`, `repository_id` | tenant composite FK → cascade deletes | +| `scope_id` | composite FK `(scope_id, repository_id) → ownership_scopes`; file paths never appear | +| `calculation_version` | isolates build N+1 work from served N; enables failed-build GC | | `period_start` | month key — the aggregation grain | -| `work_key` | `contributor_key` — or the literal `'unlinked'` for the single unresolved bucket; CHECK `work_key='unlinked' or work_key ~ '^[0-9a-f]{64}$'` prevents squatting and makes the two classes structurally disjoint | -| `is_bot` | boolean only — required because `human_contributor_count` needs the bot bit for **every** linked contributor including overflow-only ones, whose login is never persisted | +| `work_key` | `contributor_key`, or the literal `'unlinked'` for the single unresolved bucket; CHECK `work_key='unlinked' or work_key ~ '^[0-9a-f]{64}$'` makes the classes structurally disjoint | +| `is_bot` | boolean only — `human_contributor_count` needs the bot bit for **every** linked contributor including overflow-only ones, whose login is never persisted | | `touch_commits` | the accumulated count — the point of the table | -| `first_seen_at`, `last_seen_at` | min/max `committedDate` within the month — preserves exact durable-candidate timestamps (see §5.3) | +| `first_seen_at`, `last_seen_at` | min/max `committedDate` within the month — preserves exact durable-candidate timestamps | +| `created_at`, `updated_at` | required for the concrete stale-work sweep (§7) | -**Never stored here:** `github_node_id`, `login`, email, name, path, raw -node, SHA (the pin lives in `history_cursor`, §10), page payloads. +**`ownership_history_identity_work`** — pass-2 identity staging, one row +per **represented key only**: -It solves exact Top-25 (all contributions for the scope are present at -finalize), exact human count (distinct `work_key` where `not is_bot`), -crash recovery (page deltas + cursor are one transaction, §5.1), privacy -(no identity fields), bounded storage (§6), deletion (FK cascades + scope -delete + TTL sweep), and it is cleanly backup-excludable (§9). +| Column | Why it is necessary | +|---|---| +| `user_id`, `repository_id`, `scope_id`, `calculation_version` | same boundary/isolation/cascade reasons | +| `work_key` | which represented contributor this stages | +| `github_node_id`, `display_login`, `is_bot` | the identity triple needed to create the durable contributor row at COMMIT — nothing else | + +Hard rules for the staging relation: + +- **only represented keys may ever appear** — the resolve page tx writes a + staged row only when `work_key ∈ represented(set recomputed from + accumulation counters)`; overflow keys observed during pass 2 are + classified transiently and discarded in memory; +- no email, no name, no `databaseId`, no path, no raw nodes; +- EPHEMERAL: backup-excluded, RLS zero-policy, `service_role` only, cascade + on repo/user/scope delete; +- deleted inside the same COMMIT transaction that consumes it, and by the + abandoned-build sweep; +- **it is the only place GitHub identity fields may exist outside retained + tables — and only for already-represented contributors.** + +**Migration 013 therefore needs two work relations.** One table would be +smaller only by conflating two data classes with different sensitivity; +the split keeps the counter relation identity-free *by shape*. --- -## 4. Recommended architecture +## 5. Recommended architecture **Option D + scope-sequential processing + per-scope two-pass identity resolution + atomic page checkpoints + pinned-revision walks.** -### 4.1 Build state machine +### 5.1 The attribution-freeze rule + +**ACCUMULATE defines the build's attribution metrics.** Once a scope's +accumulate walk completes, the counters freeze: + +- the set of work keys, per-month touch counts, unlinked share, + Top-25 membership, `human_contributor_count`, `is_bot` classification. + +RESOLVE **must not mutate those metrics.** It exists solely to rediscover +`id`/`login` for keys pass 1 already proved individually represented. Any +GitHub-side linkage change during the build cannot alter the computed +aggregates — the build either resolves every required key and finalizes, +or fails closed and restarts the scope from a fresh ACCUMULATE (§5.3). + +### 5.2 Build state machine `ownership_sync_state` (unchanged columns; `history_cursor` jsonb carries all checkpoint detail): @@ -195,198 +258,207 @@ all checkpoint detail): pending → history → done | error (phase enum — unchanged) ``` -`history_cursor` jsonb during `history`: +`history_cursor` jsonb during `history` — deliberately small: ```jsonc { - "pinned_oid": "<40-hex>", // build boundary, §10 - "scope_order": ["", ...], // deterministic + "pinned_oid": "<40-hex>", // build boundary, §10 "current": { "scope_id": "", "pass": "accumulate" | "resolve", + "accumulate_done": true, "cursor": "" | null }, - "lease": { "owner": "", "expires": "" } // §14 + "lease": { "owner": "", "expires": "" }, // §14 + "build_started_at": "" } ``` -`pending_scopes` jsonb: remaining scope ids (existing column, existing -purpose). `blame_cursor` untouched — HEAD blame is a later tranche. +Scope order is **not** stored here: `pending_scopes` (existing jsonb +column) holds remaining scope ids, and deterministic order is re-derivable +from `ownership_scopes` (`order by depth, scope_path_hash`) at any resume +point — no duplicated full-scope list in the cursor. Worst-case +`history_cursor` stays ~1 KB; `pending_scopes` carries the ≤500 uuid list +(~20 KB) honestly, once, where the column was designed for it. + +`blame_cursor` untouched — HEAD blame is a later tranche. -Per-scope lifecycle (scopes processed in `scope_order`; repo scope first — -its resolve pass captures the bulk of identities once): +### 5.3 Per-scope lifecycle + +Repo scope first — its resolve pass stages the bulk of identities once. ``` ACCUMULATE walk history(pinned_oid, path=scope) page by page; - each page → own_checkpoint_page CAS tx (§5.1) - until hasNextPage=false + each page → own_checkpoint_page CAS tx with p_deltas + until hasNextPage=false → accumulate_done=true RESOLVE re-walk the same scope's pinned history (pass 2); - per page, inside the SAME CAS mechanism: - recompute work_key for each node; for keys ∈ represented set - (recomputed deterministically from work rows each invocation), - upsert the durable ownership_contributors row - {contributor_key, github_node_id, display_login, is_bot}; - skip pass entirely if every represented key already has a row -COMMIT single tx: load work rows → run the reducer algorithm → - upsert contributor first/last_seen_at → insert snapshot - contributor rows + scope-total rows (building_version) → - delete this scope's work rows → advance pending_scopes / - clear current cursor + per page, the SAME CAS primitive with p_identities: + recompute work_key for each node; stage + {work_key → node_id, login, is_bot} into + ownership_history_identity_work ONLY for keys in the + represented set (recomputed deterministically from the + completed accumulation counters each invocation); + skipped entirely when every represented key is staged +COMMIT ONE transaction — the only place retained rows are written: + 1. verify accumulation complete for the scope + 2. verify a staged identity exists for EVERY represented + linked key (missing → abort → restart scope at a fresh + ACCUMULATE) + 3. upsert ownership_contributors rows for staged keys + (+ unlinked/other synthetic rows on first need) + 4. materialize monthly contributor snapshot rows + 5. materialize monthly scope-total snapshot rows + 6. delete the scope's rows from BOTH work relations + 7. mark scope done: advance pending_scopes, clear current ``` -When `pending_scopes` empties: promotion tx — validate zero remaining work -rows for `building_version`, expected snapshot cardinality within §11 -bounds, then `active_version = building_version; building_version = null; -phase = 'done'` in one update. Old-version GC is a separate, later -transaction (active version is already flipped — GC failure is harmless). - -### 4.2 Why the second pass is required, not optional - -At accumulate-time we deliberately cannot know which linked identities will -be represented. Capturing node_id/login for everyone during pass 1 = -provisional identity persistence for overflow contributors (rejected, §2). -The only way to keep identity fields out of provisional storage while still -producing durable contributor rows is to observe identity **after** the -represented set is known — i.e., a second walk of the same scope. - -Cost: pass 2 is another `history(path:)` walk of that scope at cost ≈1 -pt/page (measured). Sum over scopes ≈ 2× total history spend — bounded, and -the skip optimization (all represented keys already resolved by an earlier -scope) makes typical builds cheaper. - -**Resolve-drift edge case (stated honestly):** `author.user` is resolved by -GitHub at query time, not frozen at the pinned commit — a contributor can -unlink between pass 1 and pass 2 (account deleted, email relinked). Policy: -if a represented key resolves to `user = null` during pass 2, that key's -counts are folded into the **unlinked** bucket and finals are recomputed — -the truthful answer for the linkage state GitHub currently reports, exactly -what the one-shot reducer would emit had the stream shown null. If the key -remains *linked but never re-observed* in pass 2 (shouldn't happen under a -pinned revision — same commits), fail closed `OWNERSHIP_HISTORY_FAILED` and -retry once before erroring. The differential test (§12) does not simulate -live drift; equivalence holds against a fixed stream by construction. - -### 4.3 first_seen_at / last_seen_at +A COMMIT that fails writes nothing retained: no contributor identity, no +snapshots — ephemeral state persists for safe retry. A retry re-enters at +step 1 with the same checks. + +**Resolve-drift semantics (corrected):** `author.user` is resolved by +GitHub at query time and can change independently of the pinned commit — +a pass-1 linked contributor can return `user = null` in pass 2. The +revision-1 "fold into unlinked" policy was **removed as unimplementable**: +a null `author.user` carries no node id, so there is no way to map the +null observation back to the earlier `work_key`, and the history query +deliberately selects no per-commit identity that could perform that +mapping. Semantics now: + +- a required represented key that cannot be resolved by the end of the + complete pass-2 walk → the scope/build **fails closed** and restarts + from a fresh ACCUMULATE (bounded retries, then + `OWNERSHIP_HISTORY_FAILED`); +- a *new* linked key appearing only in pass 2 is **ignored** for this + build — it was not in pass 1's represented set and contributes nothing + the freeze didn't already count; +- an overflow key disappearing in pass 2 is irrelevant — overflow + contributes only counts, which are frozen; +- a login rename under the same node id is a valid refresh — pass-2 stage + captures the current login, which is the correct durable display value. + +### 5.4 first_seen_at / last_seen_at Work rows keep min/max `committedDate` per (scope, key, month). At COMMIT: -contributor first/last = min/max over all the contributor's work rows across -all scopes (upsert `least`/`greatest` when a contributor is represented in -multiple scopes — contributor rows are repo-unique). This is value-identical -to the one-shot reducer's running min/max. No hidden problem — day-precision -is preserved; only the container changed. - -### 4.4 Unlinked - -All `author.user == null` nodes increment work rows with -`work_key='unlinked'` — one bucket, no per-author keys, no name/email ever -touched. Malformed present-but-broken identity still throws -`OwnershipIdentityError` → the page tx aborts → cursor doesn't advance → -error is surfaced via closed-taxonomy `OWNERSHIP_HISTORY_FAILED` and the -build stops (malformed identity is never silently reclassified). Retry -behavior for a persistent malformed node: the walk wedges at that cursor — -fail-closed by design, surfaced for a contract decision rather than skipped. - -### 4.5 Bots - -`is_bot` (login-suffix heuristic, pass 1) travels on the work row as a bare -boolean — the only way `human_contributor_count` can include/exclude -overflow contributors exactly without their logins. +contributor first/last = min/max over all the contributor's work rows +**across scopes** (upsert `least`/`greatest` — contributor rows are +repo-unique). Value-identical to the one-shot reducer's running min/max; +day precision preserved. + +### 5.5 Unlinked and bots + +All `author.user == null` nodes accumulate into `work_key='unlinked'` — +one bucket, no per-author keys, no name/email ever touched. Malformed +present-but-broken identity → `OwnershipIdentityError` → page tx aborts → +cursor doesn't advance → `OWNERSHIP_HISTORY_FAILED` fail-closed; the walk +wedges at that cursor for an operator decision rather than silently +reclassifying. `is_bot` travels on the work row as a bare boolean — the +only way `human_contributor_count` can include/exclude overflow +contributors exactly without their logins. --- -## 5. Exactly-once checkpointing +## 6. Exactly-once checkpointing -### 5.1 The primitive — `own_checkpoint_page` (prospective 013 function) +### 6.1 The primitive — `own_checkpoint_page` (prospective 013 function) -A SQL function, `security invoker`, `EXECUTE → service_role` only, taking: +A SQL function, `security invoker`, `EXECUTE → service_role` only: ``` -p_user, p_repository, p_version, p_scope, -p_expected_cursor, p_next_cursor, p_pass, -p_deltas jsonb -- accumulate pass: [{period_start, work_key, - -- is_bot, touches, first_seen_at, last_seen_at}] -p_identities jsonb -- resolve pass: [{work_key, github_node_id, +p_user, p_repository, p_version, p_scope, p_pass, +p_expected_cursor, p_next_cursor, +p_deltas jsonb -- accumulate: [{period_start, work_key, is_bot, + -- touches, first_seen_at, last_seen_at}] +p_identities jsonb -- resolve: [{work_key, github_node_id, -- display_login, is_bot}] ``` One transaction, in order: -1. `SELECT ... FROM ownership_sync_state WHERE (user_id, repository_id) FOR UPDATE` - — serializes all ownership work for the repo; row is also the lease record. +1. `SELECT ... FROM ownership_sync_state WHERE (user_id, repository_id) + FOR UPDATE` — serializes all ownership work for the repo. 2. Verify: row exists, `building_version = p_version`, `phase='history'`, `current.scope_id = p_scope`, `current.pass = p_pass`, `current.cursor IS NOT DISTINCT FROM p_expected_cursor`. -3. On any mismatch → `RETURN 'stale'` — **no writes.** (Lease expiry is a - caller concern, not a correctness gate.) -4. Apply `p_deltas`: set-valued upsert +3. On any mismatch → `RETURN 'stale'` — **no writes.** +4. Apply `p_deltas`: set-valued upsert into `ownership_history_work` `ON CONFLICT (scope_id, calculation_version, period_start, work_key) DO UPDATE SET touch_commits = w.touch_commits + EXCLUDED.touch_commits, - first_seen_at = least(w.first_seen_at, EXCLUDED.first_seen_at), - last_seen_at = greatest(w.last_seen_at, EXCLUDED.last_seen_at)`. - Apply `p_identities`: upsert `ownership_contributors` on - `(repository_id, contributor_key)` refreshing `display_login`/`is_bot`. - Enforce the work-row cap inside the same statement (count check → raise - → abort → phase 'error' by caller). -5. `jsonb_set(history_cursor, '{current,cursor}', p_next_cursor)`. + first_seen_at = least(...), last_seen_at = greatest(...)`; cap check in + the same statement (§7 bounds → raise → caller marks error). + Apply `p_identities`: `INSERT ... ON CONFLICT DO UPDATE` into + `ownership_history_identity_work` — **staged** rows only, never + `ownership_contributors`. +5. `jsonb_set(history_cursor, '{current,cursor}', p_next_cursor)` and, when + `hasNextPage` was false, set `accumulate_done` / flip `pass`. 6. Commit. -### 5.2 Proof of the two crash windows +### 6.2 Proof of the two crash windows **Invariant:** durable work state for scope S under version V always -reflects exactly the prefix of pages ending at durable cursor C. +reflects exactly the prefix of pages ending at durable cursor C — in both +work relations. -- Inductive step: a transaction either aborts entirely (state = prefix at C, - cursor still C — CASE B impossible: cursor cannot move without its deltas) - or commits (state = prefix at C′, cursor C′ — CASE A impossible: deltas - cannot land without the cursor moving). Postgres tx atomicity makes the +- Inductive step: a transaction either aborts entirely (state = prefix at + C, cursor still C — CASE B impossible: cursor cannot move without its + deltas) or commits (state = prefix at C′, cursor C′ — CASE A impossible: + deltas cannot land without the cursor moving). Atomicity makes the intermediate state unobservable. ∎ - Retry after commit-with-lost-response: caller retries with - `p_expected_cursor = C`. Stored cursor is C′ ≠ C → step 2 fails → `'stale'`, - zero writes. The page's effects were applied exactly once — the retry is - a read. ∎ -- `touch_commits + delta` additive upserts are safe **only because** the CAS - guarantees a delta batch can be applied at most once per cursor value — - the rejected "write then advance" / "advance then write" orderings are - exactly CASE A / CASE B. - -Same CAS envelope protects COMMIT (expected "scope still current") and -promotion (expected "pending empty") — one mechanism everywhere. + `p_expected_cursor = C`; stored cursor is C′ ≠ C → step 2 fails → + `'stale'`, zero writes. The page applied exactly once. ∎ +- Additive `+ delta` upserts are safe **only because** the CAS guarantees a + delta batch applies at most once per cursor value — the rejected + "write then advance" / "advance then write" orderings are exactly + CASE A / CASE B. + +The same CAS envelope protects COMMIT (expected "scope still current" + +accumulate/resolve-complete checks inside the transaction) and promotion +(expected "pending empty, no work rows for version") — one mechanism +everywhere. --- -## 6. Working-state bounds (hard guard, fail-closed) +## 7. Working-state bounds, TTL, and cleanup Work rows exist only for the scope currently in `ACCUMULATE`/`RESOLVE`; COMMIT deletes them before the next scope starts. -| Bound | Proposed limit | Rationale | +| Bound | Limit | Rationale | |---|---|---| -| Distinct `work_key` per scope | 10,000 | ~5× facebook/react's repo-wide contributor count for a *single scope* — absurdly generous | -| Work rows per scope | 100,000 | keys × active months; e.g. 1k contributors × 100 months. ~10 MB transient | +| Distinct `work_key` per scope | 10,000 | ~5× facebook/react's repo-wide contributor count for a *single* scope | +| Accumulation rows per scope | 100,000 | keys × active months; ~10 MB transient worst case | +| Identity-staging rows per scope | ≤ represented keys, ≤10,000 | one row per represented key — the narrowest possible set | | Page delta rows | ≤100 | one GraphQL page | -| JSONB cursor size | ~1 KB | sha + uuid + cursor strings | - -Breach → `phase='error'`, `error='OWNERSHIP_INTERNAL_ERROR'`, work rows -retained for operator inspection (backup-excluded, so they cost nothing -durable), deleted by the next build start or the TTL sweep. **Never** -silently partial — an over-cap scope is an error, not a folded result. - -TTL/cleanup: rows with `calculation_version` ≠ current `building_version`, -or orphaning an aborted/cleared build, are deleted (a) at build start, (b) -by a periodic sweep of work rows whose repo has no matching building_version -or is gone — plus unconditional cascade on repo/user delete via the -composite FK. - ---- - -## 7. Top-25 tie determinism -Work key **is** the final `contributor_key`. COMMIT loads work rows and runs -the identical comparator — `touch_commits desc, contributor_key asc` — over -the identical complete set (same pinned revision ⇒ same node stream ⇒ same -counts). Two rebuilds on the same pinned revision + same -`calculation_version` produce identical membership including rank-25 ties. -The differential test (§12) is the executable proof. +Breach → `phase='error'`, `error='OWNERSHIP_INTERNAL_ERROR'` — **never** +silently partial; an over-cap scope is an error, not a folded result. + +**Cleanup — concrete, not aspirational:** + +- Rows carry `created_at`/`updated_at`; the checkpoint upsert touches + `updated_at` on every application, so **activity is directly visible on + the rows themselves.** +- **Stale threshold:** work/staging rows whose repo build shows no + checkpoint for **14 days** (measured via row `updated_at` joined to + `ownership_sync_state.updated_at`) are abandoned. Two weeks is far + beyond any legitimate pause — slices checkpoint every ~40 s while a + build lives; even a fully paused build must beat 14 days. +- **Active-build protection:** a build that is checkpointing (any page tx + within the window) is by definition not stale — the threshold only + fires on genuinely dead state. +- **Sweep triggers:** (a) unconditionally at every build start for that + repo — rows with `calculation_version ≠ building_version` or whose + scope is gone are deleted; (b) a periodic sweep function + (`own_sweep_stale_work`, proposed in 013) invoked by whatever scheduler + the implementation tranche wires — so cleanup eventually runs even if + the repo never builds again; (c) unconditional FK cascade on + repository/account deletion. +- **Error-build behavior:** `phase='error'` leaves work rows for operator + inspection (they're backup-excluded and bounded) until the 14-day sweep + or the next build start. +- **Gate-off behavior:** identical — no caller runs, rows age out via the + sweep; no special case needed. --- @@ -395,18 +467,22 @@ The differential test (§12) is the executable proof. Recorded in the spec appendix as PROPOSED — effective only if the implementation tranche is approved: -1. **Ephemeral pseudonymous working state** — §3 gains an explicit class: - `EPHEMERAL build-scoped counters` — `contributor_key`-keyed - (scope×month×key) counters + `is_bot` + first/last timestamps, existing - only while `building_version` is set, excluded from backups, cascade- - deleted, TTL-swept. **No GitHub identity fields on overflow - contributors** — §6.4's durable-identity guarantee is unchanged. -2. **Second persisted SHA** — `history_cursor.pinned_oid` joins - `last_processed_sha` as the only sanctioned SHAs (build boundary vs +1. **New data class — EPHEMERAL short-lived persisted operational state:** + (a) per (scope × month × work_key) counters + `is_bot` + min/max + committedDate; (b) per represented-key identity staging + (`work_key`, `github_node_id`, `display_login`, `is_bot`) existing + only between pass-1 completion and scope COMMIT. `work_key` is a + repo-scoped pseudonymous identifier — permitted persisted only under + the §3 conditions. +2. **§6.4 preserved verbatim** — overflow contributors get no retained + identity row and their GitHub identity fields never reach any + persisted state; durable contributor rows are created only inside + scope finalization from staged, already-represented keys. +3. **Second persisted SHA** — `history_cursor.pinned_oid` joins + `last_processed_sha` as a sanctioned SHA (build boundary vs incremental anchor). -3. **Two-pass re-derivation** — `author.user.{id,login}` may be re-read in - a resolve pass solely to populate durable contributor rows for - already-represented keys. +4. **Two-pass re-derivation** — `author.user.{id,login}` may be re-read + in a resolve pass solely to stage identities for represented keys. Nothing else changes: no new identity fields, no finer granularity, no relaxation of §5's prohibited list. @@ -415,30 +491,27 @@ relaxation of §5's prohibited list. ## 9. Backups & restore -- `ownership_history_work` → **EPHEMERAL** list in `backup-scope.mjs` - (classification is mandatory — an unclassified table fails CI). Not in - `BACKUP_TABLES`; never dumped. -- Verified against the assumption, not accepted on faith: losing work state - costs a restarted scope — cheap. Putting it in dumps would store - pseudonymous counters indefinitely for zero benefit. EPHEMERAL is correct. -- **Restore semantics:** restore brings back sync_state (OPERATIONAL) but no - work rows. On resume the engine detects - `building_version set ∧ work rows absent ∧ current scope mid-walk` → - resets `current.cursor = null` (restart that scope's walk) — pages are - re-fetched, CAS txs rebuild state, and no page can be double-applied - because cursor/delta atomicity is per-transaction, not per-backup. - `active_version` rows are untouched throughout. +- Both work relations → **EPHEMERAL** in `backup-scope.mjs` + (classification is mandatory — an unclassified table fails CI). + Verified, not assumed: losing them costs a restarted scope only; + dumping them would persist pseudonymous state for zero benefit. +- **Restore semantics:** restore returns sync_state (OPERATIONAL) but no + work rows. Resume detects `building_version set ∧ work rows absent ∧ + scope mid-walk` → resets `current.cursor = null`, `pass='accumulate'` + for that scope → pages re-fetched, CAS txs rebuild. No page can be + double-applied — cursor/delta atomicity is per-transaction, not + per-backup. `active_version` rows are untouched throughout. --- ## 10. Revision pinning — the second blocker -`defaultBranchRef.target` resolves **at query time**: page 1 can read HEAD@A -and page 40 HEAD@B after a push — silently mixing revisions inside one -scope's aggregates. +`defaultBranchRef.target` resolves **at query time**: page 1 can read +HEAD@A and page 40 HEAD@B after a push — silently mixing revisions inside +one scope's aggregates. **Design:** resolve `defaultBranchRef.target.oid` once at build start -(cost-1 probe) → `pinned_oid` in `history_cursor` → every walk uses +(cost-1 probe) → `pinned_oid` in `history_cursor` → every walk uses: ```graphql repository(owner: $owner, name: $name) { @@ -453,22 +526,26 @@ repository(owner: $owner, name: $name) { } ``` -Tree enumeration likewise pins: `object(oid:)` → `... on Commit { tree { oid } }` -then REST `git/trees/{tree_sha}?recursive=1` (the measured endpoint, now -sha-pinned). +Tree enumeration likewise pins: `object(oid:)` → `... on Commit { tree +{ oid } }` then REST `git/trees/{tree_sha}?recursive=1` — the measured +endpoint form, now sha-pinned. - Retries always carry `pinned_oid` — same revision by construction. - Force-push making the oid unresolvable → fail closed (`OWNERSHIP_HISTORY_FAILED`), `active_version` kept, next build re-pins. Never continue mid-build on a new HEAD — that mixes revisions. -- **Not measured in Phase 0.5.** `object(oid:)`/`object(expression:)` was - used only for `blame` at literal `"HEAD"`. A pinned-oid `history(path:)` - walk, pagination behavior under `object`, and unreachable-oid semantics - (null vs `NOT_FOUND` error) are all unmeasured. +- GitHub's GraphQL schema does expose `Repository.object(oid:)` — but the + **exact ownership query shape is UNMEASURED.** Phase 0.5 used + `object(expression:"HEAD")` for blame only. - **Gate: Phase 1C-B0.5 non-persisting spike** before implementation — - same spike rigor (metadata-only queries, no writes): pinned-oid history - walks + pagination, tree-sha enumeration, unreachable-oid behavior, cost - confirmation, and live `author.user` resolution under a pinned commit. + metadata-only queries, no writes — measuring: + `object(oid:) → Commit → history(path:)`; multi-page cursor stability; + the same pinned oid across repeated requests; public **and** private + repos; unreachable-oid behavior (`null` vs `NOT_FOUND` error); + force-push/deleted-reference behavior if safely reproducible; GraphQL + cost; `tree.oid` from the pinned Commit; the REST recursive tree call + with that exact tree sha; and `author.user` resolution behavior under a + pinned commit. --- @@ -476,21 +553,29 @@ sha-pinned). `013_ownership_checkpointing.sql` — additive only: -- `create table public.ownership_history_work` — columns per §3.4; - composite FKs `(repository_id, user_id) → repositories`, - `(scope_id, repository_id) → ownership_scopes(id, repository_id)`, - all `on delete cascade`; unique `(scope_id, calculation_version, - period_start, work_key)`; CHECKs on work_key shape and non-negative - counts; `calculation_version > 0`. -- RLS enabled, zero policies; revoke browser roles; `service_role` DML only. -- `own_checkpoint_page(...)` — §5.1; `security invoker`, `EXECUTE → +- `create table public.ownership_history_work` — §4.4 columns; composite + FKs `(repository_id, user_id) → repositories` and + `(scope_id, repository_id) → ownership_scopes(id, repository_id)`, all + `on delete cascade`; unique `(scope_id, calculation_version, + period_start, work_key)`; CHECKs on work_key shape, non-negative + counts, `calculation_version > 0`, first ≤ last seen. +- `create table public.ownership_history_identity_work` — §4.4 columns; + same composite FK pattern plus `(scope_id, repository_id)`; unique + `(scope_id, calculation_version, work_key)`; CHECK work_key is a 64-hex + key (never `'unlinked'`/`'other'` — synthetic buckets carry no identity + and must never be staged). +- RLS enabled, zero policies; revoke browser roles; `service_role` DML + only — on **both**. +- `own_checkpoint_page(...)` — §6.1; `security invoker`, `EXECUTE → service_role`, all other grants revoked. -- `backup-scope.mjs`: append to `EPHEMERAL_TABLES` (same PR — CI requires it). -- Recovery-drill addition: assert the table exists post-migration and is - **absent** from dumps. -- No changes to the four existing tables: phase enum already contains - `history`; `resolve` is cursor-jsonb state, not a new phase; error - taxonomy already covers the failure modes. +- `own_sweep_stale_work(...)` — the §7 sweep, same security shape. +- `backup-scope.mjs`: append both tables to `EPHEMERAL_TABLES` (same PR — + CI requires classification). +- Recovery-drill addition: assert the tables exist post-migration and are + absent from dumps. +- No changes to the four existing tables: `phase` enum already contains + `history`; resolve is cursor-jsonb state, not a new phase; the error + taxonomy already covers every failure mode in §13. --- @@ -500,21 +585,31 @@ sha-pinned). - **Harness:** one synthetic node stream per scope → (A) `reduceHistoryToMonthly` once → expected output. - (B) simulated engine: partition the stream into pages; drive the - checkpoint algorithm over the pages (in-memory stand-in implementing the - same upsert/CAS semantics), then finalize → actual output. -- **Assert equality** on: per-month `total_touch_commits`, - `touch_commits`/`touch_share` per contributor row, - `human_contributor_count`, `unlinked_touch_share`, Top-25 membership - (exact key sets), `other` counts/shares, the durable + (B) simulated engine: partition into pages; drive the checkpoint + algorithm with an in-memory stand-in implementing identical upsert/CAS + semantics → finalize → actual output. +- **Equivalence rule:** pass-2 is **metadata resolution only** — it may + stage identity triples; it may not touch counts. Assert equality on: + per-month `total_touch_commits`, per-contributor `touch_commits` / + `touch_share`, `human_contributor_count`, `unlinked_touch_share`, + exact Top-25 key sets, `other` counts/shares, and the durable `contributor_candidates` set including `first_seen_at`/`last_seen_at`. -- **Adversarial pagination:** month spanning many pages; boundary splitting - a month mid-contributor-run; a contributor crossing rank 25 on the final - page; 30-way tie straddling rank 25; unlinked/bot nodes spread across - pages; replay of an already-committed page (must no-op via stale cursor); - simulated commit-success/response-lost; timeout between pages. -- The test fails if equivalence depends on pagination — that is the entire - point of the mechanism. +- **Adversarial pagination:** month spanning many pages; boundary + splitting a month mid-run; contributor crossing rank 25 on the final + page; 30-way tie straddling rank 25; unlinked/bot nodes across pages; + replay of a committed page (must no-op via stale cursor); + commit-success/response-lost; timeout between pages. +- **Resolve-stage failure cases:** + - selected represented key resolves to `user=null` during resolve → + fail closed, restart scope at fresh ACCUMULATE; + - selected key never re-observed in pass 2 → same fail/restart; + - overflow key disappears during resolve → ignored, aggregates frozen; + - new linked key appearing only in resolve → ignored this build (next + fresh build may observe it); + - login rename under same node id → staged login refreshes correctly; + - resolve completes, worker crashes before COMMIT → **zero** durable + contributor rows from that scope — staged rows stay ephemeral and the + next attempt re-verifies them inside the same COMMIT transaction. --- @@ -522,52 +617,56 @@ sha-pinned). | Case | Durable state | Retried? | No double-count | No skip | Cleanup | |---|---|---|---|---|---| -| Crash before GitHub response | cursor=C, state=prefix(C) | yes, refetch page | page never applied | cursor still C | — | +| Crash before GitHub response | cursor=C, state=prefix(C) | yes, refetch | page never applied | cursor still C | — | | Crash after response, before tx | same | yes | tx never ran | same | — | -| Tx commits, response lost | cursor=C′, state=prefix(C′) | retry sees C′≠C → 'stale' | CAS rejects re-apply | page is applied | — | -| Retry same page | — | returns 'stale' | no writes on stale | — | — | -| Rate limit before page | unchanged | resume from C | nothing applied | — | error=GITHUB_RATE_LIMIT, resume later | -| Rate limit after response | response already in hand → tx completes or crashes clean | — | tx atomic | — | — | -| Malformed identity in page | tx aborts; cursor stays C | wedges at C — fail closed | — | page never committed | error=OWNERSHIP_HISTORY_FAILED; operator decides | +| Tx commits, response lost | cursor=C′, state=prefix(C′) | retry sees C′≠C → 'stale' | CAS rejects re-apply | applied | — | +| Retry same page | — | 'stale' | no writes on stale | — | — | +| Rate limit before page | unchanged | resume from C | nothing applied | — | `GITHUB_RATE_LIMIT`, resume later | +| Rate limit after response | response in hand → tx completes or aborts clean | — | tx atomic | — | — | +| Malformed identity in page | tx aborts; cursor stays C | wedges → fail closed | — | page never committed | `OWNERSHIP_HISTORY_FAILED` | | DB timeout before commit | abort | retry applies | — | — | — | -| DB timeout, unknown outcome | maybe committed | retry: cursor==C′ → 'stale' (applied) or C → apply (not) | exactly-once either way | — | — | +| DB timeout, unknown outcome | maybe committed | retry: C′ → 'stale' else apply | exactly-once either way | — | — | | Worker timeout | as crash cases | next worker | CAS | — | lease expiry §14 | -| Deploy during build | in-flight tx aborts or completes | next deploy resumes | CAS | — | code must be read-compatible (additive 013) | -| Gate off mid-build | rows persist harmlessly | no caller → nothing runs | — | — | TTL sweep / next enable | -| Repo disconnected mid-build | cascade unchanged; sync gate stops calls | no | — | — | build stalls; resume on reconnect | -| Repo deleted mid-build | FK cascade removes work+sync rows | n/a | — | — | cascade is total — nothing to clean | -| Installation revoked | tx state intact | fails at next fetch | CAS | — | error=GITHUB_ACCESS_REVOKED | -| Pinned revision disappears | pages error; cursor=C | retry once → fail closed | partial scope state retained but never served (building_version) | — | error=OWNERSHIP_HISTORY_FAILED; next build re-pins | -| calculation_version changed | old-version work rows orphaned | n/a | version-scoped unique keys | — | GC by version at next build / sweep | +| Deploy during build | in-flight tx aborts or completes | next deploy resumes | CAS | — | additive 013 keeps reads compatible | +| Gate off mid-build | ephemeral rows idle | no caller runs | — | — | 14-day sweep / next enable | +| Repo disconnected mid-build | rows persist; calls stop at gate | no | — | — | resume on reconnect; sweep if abandoned | +| Repo deleted mid-build | FK cascade removes work+staging+sync rows | n/a | — | — | cascade is total | +| Installation revoked | state intact | fails at next fetch | CAS | — | `GITHUB_ACCESS_REVOKED` | +| Pinned revision disappears | pages error; cursor=C | retry once → fail closed | partial scope state never served (building_version) | — | `OWNERSHIP_HISTORY_FAILED`; next build re-pins | +| Represented key unresolvable in pass 2 | staging incomplete | bounded retry → restart scope at fresh ACCUMULATE → `OWNERSHIP_HISTORY_FAILED` | — | — | scope restarts; stale staged rows deleted on restart/sweep | +| Crash between resolve-done and COMMIT | staged rows persist, zero retained writes | COMMIT retries from stored state | — | — | staging is ephemeral until consumed | +| calculation_version changed | old-version rows orphaned | n/a | version-scoped unique keys | — | GC by version at next build / sweep | | Stale concurrent worker | lease + CAS | loser gets 'stale' | CAS rejects second apply | — | lease takeover after expiry | -| CAS cursor mismatch | — | reload state, refetch from durable C | — | — | — | +| CAS cursor mismatch | — | reload, refetch from durable C | — | — | — | +| COMMIT succeeds, response lost | scope marked done in same tx | retry sees scope advanced → 'stale' | single tx — no partial | scope already done | work rows deleted by that same tx | --- ## 14. Concurrency -- **Per-repo, not per-user.** `ownership_sync_state` row is the unit of - contention — `user_sync` locking is untouched, so a huge ownership build - can never starve the same user's normal sync. +- **Per-repo, not per-user.** `ownership_sync_state` is the contention + unit — `user_sync` locking untouched, so a large ownership build can + never starve the same user's normal sync. - **Soft lease** (`history_cursor.lease = {owner, expires}`): a worker - holds the build; a stale worker (expired lease) yields. Reduces wasted - fetches, is not the correctness mechanism. -- **Correctness backstop:** `SELECT FOR UPDATE` + cursor CAS inside - `own_checkpoint_page` — even two racing workers cannot double-apply. + holds the build; an expired lease yields to takeover. Reduces wasted + fetches; it is not the correctness mechanism. +- **Correctness backstop:** `SELECT FOR UPDATE` + cursor CAS inside the + checkpoint primitive — two racing workers cannot double-apply even with + racing leases. --- ## 15. Rate limit / slice budget - ≤300 GraphQL pts per slice (spike recommendation — headroom for normal - sync within the 5,000 pt/hr floor). + sync inside the 5,000 pt/hr floor). - Wall-clock deadline ~40 s inside the 45 s slice; stop only at page - boundaries (a checkpoint is the only legal pause point). -- Count **measured** `rateLimit.cost` per page, not assumed 1 — stop when - `remaining` < 500 or the budget is spent; persist nothing about - rate limits beyond the closed `GITHUB_RATE_LIMIT` error code. -- Resume: next scheduled slice picks up at the durable cursor — no - ownership-specific cron wiring in this design (runtime wiring is 1C-B's). + boundaries — a checkpoint is the only legal pause point. +- Count **measured** `rateLimit.cost` per page, never assumed 1; stop + when `remaining` < 500 or budget spent; rate limits surface only as the + closed `GITHUB_RATE_LIMIT` error code. +- Resume: next scheduled slice picks up at the durable cursor (runtime + wiring is Phase 1C-B's job, not this document's). --- @@ -577,22 +676,22 @@ sha-pinned). |---|---|---| | A. Buildable without migration 013? | **NO** | §1 — no existing table can hold per-contributor intermediate state within the contract | | B. Temporary per-contributor state unavoidable? | **YES** | §0 — exact fold requires complete sets; multi-invocation walks lose memory | -| C. Fifth ephemeral table required? | **YES** | §3 — only D satisfies privacy + bound + backup + cascade simultaneously | -| D. Two-pass identity resolution required? | **YES** | §4.2 — the only way to keep overflow identity fields unpersisted | +| C. Fifth ephemeral table required? | **YES — two relations** | §4.4 — counters + identity staging are different data classes; splitting keeps the counter relation identity-free by shape | +| D. Two-pass identity resolution required? | **YES** | §5.1/§5.3 — the only way to keep overflow identity fields unpersisted while still populating durable contributor rows | | E. Fixed-revision history query required? | **YES** | §10 — `defaultBranchRef` drifts mid-build | | F. Additional measured spike required? | **YES** | §10 — `object(oid:)` history is unmeasured | -| G. Exact Top-25 preserved? | **YES** | §7 — work key = contributor key, identical comparator | -| H. Overflow identity fields unpersisted? | **YES** | §4.2 — pass 2 captures only represented keys | -| I. Every page transition exactly-once? | **YES** | §5.2 — single-transaction CAS, proven | +| G. Exact Top-25 preserved? | **YES** | §3 — work key = contributor key, identical comparator, frozen pass-1 metrics | +| H. Overflow identity fields unpersisted? | **YES** | §4.4 — staging accepts represented keys only; durable contributor rows exist only via COMMIT | +| I. Every page transition exactly-once? | **YES** | §6.2 — single-transaction CAS, proven | | J. Work state backup-excludable safely? | **YES** | §9 — loss = restarted scope, never user data | ## 17. Readiness verdict **Ready to IMPLEMENT Phase 1C-B: NO.** Three prerequisites, in order: -1. Approval of the §8 contract amendment (ephemeral pseudonymous work - state + pinned build SHA in sync-state jsonb). +1. Approval of the §8 contract amendment (ephemeral pseudonymous counters + + represented-key identity staging + pinned build SHA). 2. Phase 1C-B0.5 non-persisting spike measuring pinned-revision history - walks, pinned tree enumeration, and unreachable-oid behavior. + walks, pinned tree enumeration, and unreachable-oid behavior (§10). 3. Then: migration 013 + the engine implementation behind the existing hard-off gate. diff --git a/docs/ownership-data-privacy-spec.md b/docs/ownership-data-privacy-spec.md index d63931c..a7d4942 100644 --- a/docs/ownership-data-privacy-spec.md +++ b/docs/ownership-data-privacy-spec.md @@ -1222,37 +1222,60 @@ hold it without violating this contract (snapshots require a durable contributor FK; the linked-contributor CHECK forces identity fields; the sync-state JSONB is backed up, unbounded, and unconstrainable). +Terminology for this amendment is exact: **EPHEMERAL** means short-lived +*persisted* operational state — written to Postgres, but backup-excluded, +never browser/API-exposed, never logged or telemetered, deleted at scope +finalization or on build-abort cleanup, and fully reconstructable by +restarting the build. **RETAINED** means durable, backed-up, user-visible +ownership data (`ownership_scopes`, `ownership_contributors`, +`ownership_snapshots` at `active_version`). Persisted ≠ retained. + The proposed amendment is deliberately narrow: -1. **New data class — EPHEMERAL build working state.** Per - (scope × month × contributor_key) touch counters plus the `is_bot` +1. **New data class — EPHEMERAL build working state, two relations.** (a) + Per (scope × month × work_key) touch counters plus the `is_bot` boolean and min/max `committedDate`, existing only while - `building_version` is set for that repository, keyed by the **same - opaque `contributor_key`** already defined in §5/§6 (sha256 over - domain + repository uuid + node id). The key carries no node id, no - login, no name, no email, no path, and no raw nodes. + `building_version` is set for that repository. (b) Per + represented-work-key identity staging carrying exactly + `github_node_id`, `display_login`, `is_bot`, existing only between a + scope's completed accumulation and its finalization. `work_key` is the + same `contributor_key` defined in §5/§6 — a **deterministic, + repository-scoped pseudonymous identifier** derived from GitHub node + identity: no plaintext identity, not user-displayable, linkable within + the repository, and therefore not anonymous. It may appear in + persisted build state only under the EPHEMERAL conditions above — + service-role-only, backup-excluded, never logged, deleted at + finalization, cascade-deleted, swept on abandoned builds. 2. **The §6.4 guarantee is preserved verbatim** — overflow-only - contributors still get **no durable identity row** and their GitHub - identity fields (`github_node_id`, `display_login`) are still **never - persisted**. The work row is a counter, not an identity row; it is - deleted at scope finalization, is backup-excluded (EPHEMERAL class), - cascades on repository/account deletion, and is swept on abandoned - builds. -3. **Two-pass identity resolution.** GitHub identity fields - (`author.user.{id,login}`) are re-read in a second walk only to write - durable `ownership_contributors` rows for keys already known to be - represented — overflow identities are classified transiently (to - compute their work key) and discarded in memory per page, exactly as - §9 requires. + contributors still get **no retained identity row** and their GitHub + identity fields (`github_node_id`, `display_login`) are **never + persisted in any table**. Staged identity rows are written only for + contributor keys pass 1 already proved individually represented; + `ownership_contributors` rows are created **only inside the atomic + scope-finalization transaction**, never during page-by-page resolve — + so an abandoned scope can never orphan a durable identity. +3. **Two-pass identity resolution, attribution-frozen.** The accumulate + pass fixes every metric (work keys, counts, Top-25, unlinked share, + human counts, bot flags); the resolve pass re-reads + `author.user.{id,login}` solely to stage identities for represented + keys and cannot mutate metrics. A required key that stays + unresolvable fails the scope closed and restarts at a fresh + accumulate — GitHub's account-link resolution can change independently + of the pinned commit, and the build succeeds only while selected keys + remain resolvable. 4. **Build-boundary SHA.** `ownership_sync_state.history_cursor` may carry the build's pinned commit oid in addition to page cursors — a second sanctioned SHA beside `last_processed_sha` (§5.12), required because `defaultBranchRef` resolves at query time and multi-page builds must not straddle pushes. 5. Rejected within the design: provisional persistence of - `github_node_id`/`display_login` for overflow contributors in any form - (ephemeral identity maps, provisional contributor rows), and JSONB - accumulators inside `ownership_sync_state` (backed up, unbounded). + `github_node_id`/`display_login` for contributors not yet proven + represented (optimistic identity maps, provisional contributor rows); + JSONB accumulators inside `ownership_sync_state` (backed up, + unbounded); and folding pass-1-linked contributors into `unlinked` + when they unlink during pass 2 (unimplementable — a null `author.user` + carries no node id mapping back to the work key; pass-2 non-resolution + instead fails the scope closed). If this amendment is rejected, the honest alternatives are: (a) accept per-scope restart-in-single-invocation with no completion guarantee on From 39c1231b614a03d4aec6352591c5afccca9bf48a Mon Sep 17 00:00:00 2001 From: aliferous3 Date: Sun, 18 Oct 2026 21:48:32 +0400 Subject: [PATCH 3/9] docs: freeze bot flags to accumulate, add abort cleanup, fix seen-at semantics MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Revision 3: is_bot is frozen pass-1 attribution — resolve verifies rather than re-derives it and a classification flip fails the scope closed; the staging relation drops is_bot entirely. Adds the explicit pre-flip abandon transaction (versioned snapshot/ephemeral deletes plus anti-join pruning of contributor rows unreferenced in ANY version, preserving active-version references). first_seen_at/last_seen_at now accumulate only across finalized scopes where the contributor was individually represented — recorded in the spec so the field names don't overclaim. Docs only. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --- docs/ownership-checkpoint-architecture.md | 158 ++++++++++++++++++---- docs/ownership-data-privacy-spec.md | 49 ++++--- 2 files changed, 162 insertions(+), 45 deletions(-) diff --git a/docs/ownership-checkpoint-architecture.md b/docs/ownership-checkpoint-architecture.md index b2f858a..88f6e70 100644 --- a/docs/ownership-checkpoint-architecture.md +++ b/docs/ownership-checkpoint-architecture.md @@ -208,7 +208,15 @@ per **represented key only**: |---|---| | `user_id`, `repository_id`, `scope_id`, `calculation_version` | same boundary/isolation/cascade reasons | | `work_key` | which represented contributor this stages | -| `github_node_id`, `display_login`, `is_bot` | the identity triple needed to create the durable contributor row at COMMIT — nothing else | +| `github_node_id`, `display_login` | the identity pair needed to create the durable contributor row at COMMIT — nothing else | +| `created_at`, `updated_at` | same stale-sweep requirement as counters (§7) | + +Deliberately **no `is_bot` column**: bot classification is part of the +frozen pass-1 attribution (§5.1 — `human_contributor_count` depends on +it), and the retained `ownership_contributors.is_bot` for the build comes +from the frozen counter state, not from pass 2. Staging a second +is_bot would create a competing authority; pass 2 instead *verifies* +observed bot status against the frozen value (§5.3). Hard rules for the staging relation: @@ -241,7 +249,10 @@ resolution + atomic page checkpoints + pinned-revision walks.** accumulate walk completes, the counters freeze: - the set of work keys, per-month touch counts, unlinked share, - Top-25 membership, `human_contributor_count`, `is_bot` classification. + Top-25 membership, `human_contributor_count`, and **`is_bot` + classification** — frozen because `human_contributor_count` depends on + it; the retained `ownership_contributors.is_bot` written at COMMIT is + the pass-1 value, never a pass-2 observation. RESOLVE **must not mutate those metrics.** It exists solely to rediscover `id`/`login` for keys pass 1 already proved individually represented. Any @@ -293,12 +304,14 @@ ACCUMULATE walk history(pinned_oid, path=scope) page by page; until hasNextPage=false → accumulate_done=true RESOLVE re-walk the same scope's pinned history (pass 2); per page, the SAME CAS primitive with p_identities: - recompute work_key for each node; stage - {work_key → node_id, login, is_bot} into - ownership_history_identity_work ONLY for keys in the + recompute work_key for each node; for keys in the represented set (recomputed deterministically from the - completed accumulation counters each invocation); - skipped entirely when every represented key is staged + completed accumulation counters each invocation) ALSO + verify observed is_bot(current login) == frozen is_bot — + a mismatch is classification drift → fail closed; + stage {work_key → node_id, login} into + ownership_history_identity_work; skipped entirely when + every represented key is staged and verified COMMIT ONE transaction — the only place retained rows are written: 1. verify accumulation complete for the scope 2. verify a staged identity exists for EVERY represented @@ -334,16 +347,40 @@ mapping. Semantics now: the freeze didn't already count; - an overflow key disappearing in pass 2 is irrelevant — overflow contributes only counts, which are frozen; -- a login rename under the same node id is a valid refresh — pass-2 stage - captures the current login, which is the correct durable display value. +- a login rename under the same node id that **preserves bot + classification** is a valid refresh — pass-2 stage captures the current + login, which is the correct durable display value; +- a login change under the same node id that **flips bot + classification** (ordinary login → `[bot]`-suffixed, or vice versa) is + classification drift: the frozen `is_bot` is part of the attribution + metrics, so the scope **fails closed and restarts at a fresh + ACCUMULATE** — bounded retries, then `OWNERSHIP_HISTORY_FAILED`. Pass 2 + may never silently alter bot classification. + +### 5.4 first_seen_at / last_seen_at — corrected semantics + +Work rows keep min/max `committedDate` per (scope, key, month). At scope +COMMIT, for each represented contributor: + +``` +current_scope_first = min(first_seen_at over that key's CURRENT-scope work rows) +current_scope_last = max(last_seen_at over that key's CURRENT-scope work rows) +``` -### 5.4 first_seen_at / last_seen_at +then the retained `ownership_contributors` row upserts with: -Work rows keep min/max `committedDate` per (scope, key, month). At COMMIT: -contributor first/last = min/max over all the contributor's work rows -**across scopes** (upsert `least`/`greatest` — contributor rows are -repo-unique). Value-identical to the one-shot reducer's running min/max; -day precision preserved. +``` +first_seen_at = least(existing.first_seen_at, current_scope_first) +last_seen_at = greatest(existing.last_seen_at, current_scope_last) +``` + +So retained first/last bounds accumulate **only across finalized scopes in +which the contributor was individually represented**. Activity observed in +an earlier scope where the contributor stayed overflow-only does **not** +contribute — those work rows are destroyed at that scope's COMMIT, and no +claim can be made about them. The spec §27 amendment states this so the +field names never imply an all-repository-touch guarantee the storage +model cannot provide. ### 5.5 Unlinked and bots @@ -356,6 +393,41 @@ reclassifying. `is_bot` travels on the work row as a bare boolean — the only way `human_contributor_count` can include/exclude overflow contributors exactly without their logins. +### 5.6 Pre-flip build abort — explicit cleanup + +Scope COMMITs legitimately create `ownership_contributors` rows and +`building_version` snapshots before the build completes. If the build is +abandoned before promotion (error, gate-off, operator abort), those rows +must not orphan — this is identity minimization, not housekeeping. + +**Abandon transaction (one tx, under the same sync-row lock):** + +1. Verify `(user_id, repository_id)` row, `building_version = V`, + `phase != 'done'`. +2. Delete `ownership_snapshots` where `repository_id` and + `calculation_version = V`. +3. Delete `ownership_history_work` and `ownership_history_identity_work` + for that repo/version. +4. Clear `building_version`, `history_cursor` build state, and + `pending_scopes` for the build. +5. **Prune** `ownership_contributors` rows for the repository not + referenced by *any* `ownership_snapshots` row in *any* version — the + anti-join is what makes it safe: contributors still referenced by the + active/retained version survive; contributors introduced only by the + aborted build are removed. Synthetic `unlinked`/`other` rows prune the + same way — they are reconstructable. +6. `active_version` is never touched; phase falls back to the state implied + by whether an active version exists (`done`/`error` as appropriate). + +Required implementation-time tests: + +- **Initial-build abort:** `active_version=null`, some V1 scopes + committed → abort → zero V1 snapshots, zero V1-only contributor rows, + zero rows in both work relations. +- **N → N+1 abort:** N active, some N+1 scopes committed → abort → all N + rows untouched; contributors referenced by N survive; contributors + introduced only by N+1 pruned; no N+1 snapshots or work rows remain. + --- ## 6. Exactly-once checkpointing @@ -470,14 +542,24 @@ implementation tranche is approved: 1. **New data class — EPHEMERAL short-lived persisted operational state:** (a) per (scope × month × work_key) counters + `is_bot` + min/max committedDate; (b) per represented-key identity staging - (`work_key`, `github_node_id`, `display_login`, `is_bot`) existing - only between pass-1 completion and scope COMMIT. `work_key` is a - repo-scoped pseudonymous identifier — permitted persisted only under - the §3 conditions. + (`work_key`, `github_node_id`, `display_login`) existing only between + pass-1 completion and scope COMMIT. `work_key` is a repo-scoped + pseudonymous identifier — permitted persisted only under the §3 + conditions. 2. **§6.4 preserved verbatim** — overflow contributors get no retained identity row and their GitHub identity fields never reach any persisted state; durable contributor rows are created only inside scope finalization from staged, already-represented keys. + **Attribution freeze:** `is_bot` is frozen by pass 1 along with all + counts; pass 2 verifies rather than re-derives it, and a mismatch + fails the scope closed rather than silently reclassifying. + **Abort cleanup:** abandoned builds delete their versioned snapshots + and ephemeral state, then prune `ownership_contributors` rows + unreferenced by any snapshot in any version — retained-version + contributors always survive (§5.6). + **Seen-at semantics:** retained `first_seen_at`/`last_seen_at` + accumulate only across finalized scopes where the contributor was + individually represented (§5.4) — no all-repository-touch claim. 3. **Second persisted SHA** — `history_cursor.pinned_oid` joins `last_processed_sha` as a sanctioned SHA (build boundary vs incremental anchor). @@ -559,11 +641,13 @@ endpoint form, now sha-pinned. `on delete cascade`; unique `(scope_id, calculation_version, period_start, work_key)`; CHECKs on work_key shape, non-negative counts, `calculation_version > 0`, first ≤ last seen. -- `create table public.ownership_history_identity_work` — §4.4 columns; - same composite FK pattern plus `(scope_id, repository_id)`; unique - `(scope_id, calculation_version, work_key)`; CHECK work_key is a 64-hex - key (never `'unlinked'`/`'other'` — synthetic buckets carry no identity - and must never be staged). +- `create table public.ownership_history_identity_work` — §4.4 columns: + tenant/scope/version + `work_key` + `github_node_id` + `display_login` + + timestamps only (**no `is_bot`** — classification is frozen pass-1 + state, §5.1/§5.3); same composite FK pattern; unique `(scope_id, + calculation_version, work_key)`; CHECK work_key is a 64-hex key (never + `'unlinked'`/`'other'` — synthetic buckets carry no identity and must + never be staged). - RLS enabled, zero policies; revoke browser roles; `service_role` DML only — on **both**. - `own_checkpoint_page(...)` — §6.1; `security invoker`, `EXECUTE → @@ -606,10 +690,28 @@ endpoint form, now sha-pinned. - overflow key disappears during resolve → ignored, aggregates frozen; - new linked key appearing only in resolve → ignored this build (next fresh build may observe it); - - login rename under same node id → staged login refreshes correctly; + - login rename under same node id preserving bot classification → + staged login refreshes correctly; + - same node id, ordinary login → `[bot]` login between passes → fail + closed, restart at fresh ACCUMULATE; + - same node id, `[bot]` → ordinary login between passes → same + fail/restart; + - identical bot classification between passes → valid; - resolve completes, worker crashes before COMMIT → **zero** durable contributor rows from that scope — staged rows stay ephemeral and the - next attempt re-verifies them inside the same COMMIT transaction. + next attempt re-verifies them inside the same COMMIT transaction; + - the retained `is_bot` on every contributor row equals the pass-1 + frozen value, never a pass-2 observation. +- **Candidate/upsert equivalence:** the contributor candidate set and + each candidate's `first_seen_at`/`last_seen_at` must equal the one-shot + result, with the documented merge rule — repo-level bounds accumulate + only over scopes where the contributor was individually represented + (§5.4), and the simulated multi-scope run must exercise the + least/greatest upsert across finalized scopes. +- **Abort cases:** initial-build abort leaves zero V1 snapshots, + zero V1-only contributor rows, zero work/staging rows; N→N+1 abort + preserves all N rows and N-referenced contributors while pruning + N+1-only contributors (§5.6). --- @@ -634,6 +736,10 @@ endpoint form, now sha-pinned. | Installation revoked | state intact | fails at next fetch | CAS | — | `GITHUB_ACCESS_REVOKED` | | Pinned revision disappears | pages error; cursor=C | retry once → fail closed | partial scope state never served (building_version) | — | `OWNERSHIP_HISTORY_FAILED`; next build re-pins | | Represented key unresolvable in pass 2 | staging incomplete | bounded retry → restart scope at fresh ACCUMULATE → `OWNERSHIP_HISTORY_FAILED` | — | — | scope restarts; stale staged rows deleted on restart/sweep | +| Pass-2 bot-classification drift | staging not written for that key | same fail-closed restart at fresh ACCUMULATE | frozen pass-1 `is_bot` is the only authority | — | scope restarts; staged rows deleted | +| Build abort after ≥1 scope COMMITs | committed scope snapshots + contributor rows for `building_version` | abandon tx (§5.6) | — | — | single tx: delete versioned snapshots + both work relations + clear build state | +| Abort cleanup, N active + N+1 building | N rows are served data | n/a — cleanup only touches N+1 | — | — | N snapshots and N-referenced contributors preserved; N+1 artifacts deleted | +| Orphan contributor pruning | contributor rows unreferenced by every version | part of abandon tx | — | — | anti-join delete — only rows unreferenced by ANY snapshot version are removed | | Crash between resolve-done and COMMIT | staged rows persist, zero retained writes | COMMIT retries from stored state | — | — | staging is ephemeral until consumed | | calculation_version changed | old-version rows orphaned | n/a | version-scoped unique keys | — | GC by version at next build / sweep | | Stale concurrent worker | lease + CAS | loser gets 'stale' | CAS rejects second apply | — | lease takeover after expiry | diff --git a/docs/ownership-data-privacy-spec.md b/docs/ownership-data-privacy-spec.md index a7d4942..b95e565 100644 --- a/docs/ownership-data-privacy-spec.md +++ b/docs/ownership-data-privacy-spec.md @@ -248,7 +248,7 @@ filtered in memory during enumeration and never reach this table (§8). | `github_node_id` | text null | durable key for linked accounts only — the GraphQL `User.id` already selected as `author.user.id` (the same value `isAttributedCommit` compares against `users.github_node_id`) | | `display_login` | text null | current GitHub login for display — see §6 decision | | `is_bot` | boolean | bots excludable in UI per RFC §10.3 | -| `first_seen_at`, `last_seen_at` | timestamptz | powers "recent maintainer" without scanning snapshots | +| `first_seen_at`, `last_seen_at` | timestamptz | powers "recent maintainer" without scanning snapshots — bounds accumulate only over scope-periods where the contributor was **individually represented**; overflow-only activity contributes no bounds (§27), so these fields do not claim all-repository-touch coverage | | `created_at`, `updated_at` | timestamptz | housekeeping | **Rejected:** global contributor table; numeric `databaseId` (no concrete @@ -1237,15 +1237,17 @@ The proposed amendment is deliberately narrow: boolean and min/max `committedDate`, existing only while `building_version` is set for that repository. (b) Per represented-work-key identity staging carrying exactly - `github_node_id`, `display_login`, `is_bot`, existing only between a - scope's completed accumulation and its finalization. `work_key` is the - same `contributor_key` defined in §5/§6 — a **deterministic, - repository-scoped pseudonymous identifier** derived from GitHub node - identity: no plaintext identity, not user-displayable, linkable within - the repository, and therefore not anonymous. It may appear in - persisted build state only under the EPHEMERAL conditions above — - service-role-only, backup-excluded, never logged, deleted at - finalization, cascade-deleted, swept on abandoned builds. + `github_node_id` and `display_login` (no `is_bot` — bot + classification is frozen pass-1 attribution, not a second staged + authority), existing only between a scope's completed accumulation and + its finalization. `work_key` is the same `contributor_key` defined in + §5/§6 — a **deterministic, repository-scoped pseudonymous identifier** + derived from GitHub node identity: no plaintext identity, not + user-displayable, linkable within the repository, and therefore not + anonymous. It may appear in persisted build state only under the + EPHEMERAL conditions above — service-role-only, backup-excluded, never + logged, deleted at finalization, cascade-deleted, swept on abandoned + builds. 2. **The §6.4 guarantee is preserved verbatim** — overflow-only contributors still get **no retained identity row** and their GitHub identity fields (`github_node_id`, `display_login`) are **never @@ -1253,16 +1255,25 @@ The proposed amendment is deliberately narrow: contributor keys pass 1 already proved individually represented; `ownership_contributors` rows are created **only inside the atomic scope-finalization transaction**, never during page-by-page resolve — - so an abandoned scope can never orphan a durable identity. + so an abandoned scope can never orphan a durable identity. If a build + is abandoned after scope finalizations but before promotion, the abort + transaction deletes the building-version's snapshots and ephemeral + state and then **prunes `ownership_contributors` rows unreferenced by + any snapshot in any version** — contributors still referenced by the + active/retained version always survive; this pruning is part of + identity minimization, not housekeeping. 3. **Two-pass identity resolution, attribution-frozen.** The accumulate - pass fixes every metric (work keys, counts, Top-25, unlinked share, - human counts, bot flags); the resolve pass re-reads - `author.user.{id,login}` solely to stage identities for represented - keys and cannot mutate metrics. A required key that stays - unresolvable fails the scope closed and restarts at a fresh - accumulate — GitHub's account-link resolution can change independently - of the pinned commit, and the build succeeds only while selected keys - remain resolvable. + pass fixes every metric — work keys, counts, Top-25, unlinked share, + human counts, **and `is_bot` classification** (`human_contributor_count` + depends on it; the retained `ownership_contributors.is_bot` is always + the pass-1 value). The resolve pass re-reads `author.user.{id,login}` + solely to stage identities for represented keys and cannot mutate + metrics — a bot-classification mismatch between passes is identity + drift: the scope fails closed and restarts at a fresh accumulate. A + required key that stays unresolvable fails the scope closed the same + way — GitHub's account-link resolution can change independently of the + pinned commit, and the build succeeds only while selected keys remain + resolvable and consistent. 4. **Build-boundary SHA.** `ownership_sync_state.history_cursor` may carry the build's pinned commit oid in addition to page cursors — a second sanctioned SHA beside `last_processed_sha` (§5.12), required because From e9de4bcf93688282507cca64f4aa5415388fa4c1 Mon Sep 17 00:00:00 2001 From: aliferous3 Date: Sun, 18 Oct 2026 22:07:21 +0400 Subject: [PATCH 4/9] docs: version-isolate contributor metadata via build-level staging MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Revision 3: ownership_contributors is shared across calculation versions but the dashboard RPCs version-filter only snapshots — mid-build mutation of shared contributor metadata would leak into active-version responses and could not be restored on abort. Identity/metadata staging moves to a build-grain ephemeral relation (ownership_contributor_build_work); retained rows are never mutated while a building version is incomplete; promotion applies staged metadata and flips active_version in one transaction; abort deletes building artifacts and anti-join prunes unreferenced contributor rows. is_bot is frozen once per build — drift between passes or across scopes fails closed. Docs only. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --- docs/ownership-checkpoint-architecture.md | 333 ++++++++++++++++------ docs/ownership-data-privacy-spec.md | 67 +++-- 2 files changed, 281 insertions(+), 119 deletions(-) diff --git a/docs/ownership-checkpoint-architecture.md b/docs/ownership-checkpoint-architecture.md index 88f6e70..fb3db99 100644 --- a/docs/ownership-checkpoint-architecture.md +++ b/docs/ownership-checkpoint-architecture.md @@ -1,11 +1,16 @@ # Ownership Checkpoint Architecture — Phase 1C-B0 Design -> **Status: design only, revision 2. Nothing in this document is implemented -> or live.** Revision 1→2 corrections: durable contributor writes moved -> entirely inside the atomic scope-COMMIT; a second ephemeral relation -> stages pass-2 identities; the unimplementable resolve-drift fold removed; -> `work_key` reclassified as a linkable pseudonymous identifier (not -> anonymous); concrete TTL/cleanup; cursor jsonb slimmed. +> **Status: design only, revision 3. Nothing in this document is implemented +> or live.** Revision 2→3 correction — **version isolation**: retained +> `ownership_contributors` rows are shared across calculation versions while +> the dashboard RPCs filter only snapshots by `active_version`; a building +> version mutating shared contributor metadata would leak into active-N +> responses and could not be restored on abort. Contributor identity/metadata +> is therefore staged **build-level** and applied atomically at promotion. +> Earlier corrections: durable contributor writes inside the atomic +> scope-COMMIT only; the unimplementable resolve-drift fold removed; +> `work_key` classified as a linkable pseudonymous identifier; concrete +> TTL/cleanup; slim cursor jsonb. > > This document resolves the §9.1 hard blocker in > [ownership-data-privacy-spec.md](ownership-data-privacy-spec.md): how to run @@ -98,7 +103,8 @@ GitHub account identity**. Precisely: It may be used in persisted build state only under all of these conditions: -- short-lived (deleted at scope finalization; swept on abandoned builds); +- short-lived (counter rows deleted at scope finalization; build-level + staging deleted at promotion/abort; all swept on abandoned builds); - `service_role`-only access, zero browser/API exposure; - never logged, telemetered, or placed in error fields; - excluded from backups; @@ -201,22 +207,43 @@ with identity staging; the normalized pair is the cleaner privacy boundary: | `first_seen_at`, `last_seen_at` | min/max `committedDate` within the month — preserves exact durable-candidate timestamps | | `created_at`, `updated_at` | required for the concrete stale-work sweep (§7) | -**`ownership_history_identity_work`** — pass-2 identity staging, one row -per **represented key only**: +**`ownership_contributor_build_work`** — **build-level** contributor +staging, one row per **represented key per repository build**, grain +`(repository_id, calculation_version, work_key)`: | Column | Why it is necessary | |---|---| -| `user_id`, `repository_id`, `scope_id`, `calculation_version` | same boundary/isolation/cascade reasons | +| `user_id`, `repository_id`, `calculation_version` | tenant boundary + build isolation — the grain is the *build*, not the scope | | `work_key` | which represented contributor this stages | -| `github_node_id`, `display_login` | the identity pair needed to create the durable contributor row at COMMIT — nothing else | -| `created_at`, `updated_at` | same stale-sweep requirement as counters (§7) | +| `github_node_id`, `display_login` | the identity pair rediscovered by pass 2 — populated only for represented keys | +| `is_bot` | the **frozen pass-1** classification — recorded once, verified on every later appearance (§5.3); the retained row's value comes from here, never from pass-2 observations | +| `first_seen_at`, `last_seen_at` | merged represented-scope bounds, accumulated by scope COMMITs via least/greatest (§5.4) | +| `created_at`, `updated_at` | stale-sweep requirement (§7) | + +Why build-level, not scope-level staging (revision-3 change): +`ownership_contributors` is **shared across calculation versions** — the +deployed `dash_ownership_*` RPCs filter snapshots by `active_version` but +join `display_login`/`is_bot` from the shared contributor row +unversioned. Any build-time mutation of a shared contributor row +(display_login refresh, is_bot, seen bounds) would therefore leak N+1 +metadata into live active-N responses, and an abort could not restore the +old values. The build-level row is the complete intended N+1 contributor +metadata; retained rows are **never mutated while a building version +exists** — promotion applies it atomically (§5.7). -Deliberately **no `is_bot` column**: bot classification is part of the -frozen pass-1 attribution (§5.1 — `human_contributor_count` depends on -it), and the retained `ownership_contributors.is_bot` for the build comes -from the frozen counter state, not from pass 2. Staging a second -is_bot would create a competing authority; pass 2 instead *verifies* -observed bot status against the frozen value (§5.3). +Hard rules for the staging relation: + +- **only represented keys may ever appear** — the resolve page tx writes a + staged row only when `work_key ∈ represented`(recomputed from the + completed accumulation counters); overflow keys observed during pass 2 + are classified transiently and discarded in memory; +- no email, no name, no `databaseId`, no path, no raw nodes; +- EPHEMERAL: backup-excluded, RLS zero-policy, `service_role` only, cascade + on repo/user delete; +- deleted on promotion (consumed), on abort, and by the abandoned-build + sweep; +- **it is the only place GitHub identity fields may exist outside retained + tables — and only for already-represented contributors.** Hard rules for the staging relation: @@ -226,9 +253,10 @@ Hard rules for the staging relation: classified transiently and discarded in memory; - no email, no name, no `databaseId`, no path, no raw nodes; - EPHEMERAL: backup-excluded, RLS zero-policy, `service_role` only, cascade - on repo/user/scope delete; -- deleted inside the same COMMIT transaction that consumes it, and by the - abandoned-build sweep; + on repo/user delete; +- deleted by the promotion transaction that consumes it (§5.7), by the + abort transaction (§5.6), and by the abandoned-build sweep — it lives + for the whole build, not one scope; - **it is the only place GitHub identity fields may exist outside retained tables — and only for already-represented contributors.** @@ -251,8 +279,10 @@ accumulate walk completes, the counters freeze: - the set of work keys, per-month touch counts, unlinked share, Top-25 membership, `human_contributor_count`, and **`is_bot` classification** — frozen because `human_contributor_count` depends on - it; the retained `ownership_contributors.is_bot` written at COMMIT is - the pass-1 value, never a pass-2 observation. + it; the retained `ownership_contributors.is_bot` comes from the frozen + pass-1 value staged in `ownership_contributor_build_work` — applied to + the shared row only at promotion (§5.7), never from a pass-2 + observation and never during the build. RESOLVE **must not mutate those metrics.** It exists solely to rediscover `id`/`login` for keys pass 1 already proved individually represented. Any @@ -306,28 +336,47 @@ RESOLVE re-walk the same scope's pinned history (pass 2); per page, the SAME CAS primitive with p_identities: recompute work_key for each node; for keys in the represented set (recomputed deterministically from the - completed accumulation counters each invocation) ALSO - verify observed is_bot(current login) == frozen is_bot — - a mismatch is classification drift → fail closed; - stage {work_key → node_id, login} into - ownership_history_identity_work; skipped entirely when - every represented key is staged and verified + completed accumulation counters each invocation): + - verify observed is_bot(current login) == frozen + build-level is_bot — mismatch is classification + drift → fail closed + - upsert {work_key → node_id, login} into + ownership_contributor_build_work (repo×version grain); + a conflicting stored node_id for the same key is + impossible by construction and fails closed if seen; + skipped entirely when every represented key is staged COMMIT ONE transaction — the only place retained rows are written: 1. verify accumulation complete for the scope - 2. verify a staged identity exists for EVERY represented - linked key (missing → abort → restart scope at a fresh - ACCUMULATE) - 3. upsert ownership_contributors rows for staged keys - (+ unlinked/other synthetic rows on first need) - 4. materialize monthly contributor snapshot rows - 5. materialize monthly scope-total snapshot rows - 6. delete the scope's rows from BOTH work relations + 2. verify a staged build-work row exists for EVERY + represented linked key (missing → abort → restart + scope at fresh ACCUMULATE) + 3. merge this scope's contribution into the build-work row: + is_bot must equal the staged frozen value (drift → + fail closed); first/last seen merge via least/greatest + 4. NEW represented contributor (no ownership_contributors + row): insert the minimum legitimate linked row — + node_id + display_login + frozen is_bot + staged + first/last — required so building snapshots can FK to + it. It is invisible to active-N RPC output because no + active-version snapshot references it; abort prunes it + (§5.6). + EXISTING contributor row: reference its existing id in + building snapshots — update NOTHING on it (no + display_login / is_bot / first_seen_at / last_seen_at + writes while building_version is incomplete). + 5. materialize monthly contributor snapshot rows + + scope-total snapshot rows (building_version) + 6. delete the scope's ownership_history_work rows 7. mark scope done: advance pending_scopes, clear current ``` A COMMIT that fails writes nothing retained: no contributor identity, no snapshots — ephemeral state persists for safe retry. A retry re-enters at -step 1 with the same checks. +step 1 with the same checks. New contributor rows created by a committed +scope COMMIT are *building artifacts*: they exist solely so building +snapshots have a valid FK target, no active-N snapshot references them, +and the abort transaction removes them via the unreferenced-row prune — +their presence is not evidence that active N changed. **Resolve-drift semantics (corrected):** `author.user` is resolved by GitHub at query time and can change independently of the pinned commit — @@ -355,7 +404,14 @@ mapping. Semantics now: classification drift: the frozen `is_bot` is part of the attribution metrics, so the scope **fails closed and restarts at a fresh ACCUMULATE** — bounded retries, then `OWNERSHIP_HISTORY_FAILED`. Pass 2 - may never silently alter bot classification. + may never silently alter bot classification; +- **cross-scope bot consistency:** `is_bot` for a `work_key` is frozen + once per build at first representation. If a later scope's pass-1 + classification for the same key disagrees with the frozen build-level + value, the identity classification changed mid-build → **fail/restart + the repository build from a new pinned boundary**, not just the scope. + One contributor cannot be human in one scope and bot in another within + a single calculation version. ### 5.4 first_seen_at / last_seen_at — corrected semantics @@ -367,20 +423,23 @@ current_scope_first = min(first_seen_at over that key's CURRENT-scope work rows) current_scope_last = max(last_seen_at over that key's CURRENT-scope work rows) ``` -then the retained `ownership_contributors` row upserts with: +then the **build-level staging row** — never the shared retained row — +merges: ``` -first_seen_at = least(existing.first_seen_at, current_scope_first) -last_seen_at = greatest(existing.last_seen_at, current_scope_last) +staged.first_seen_at = least(staged.first_seen_at, current_scope_first) +staged.last_seen_at = greatest(staged.last_seen_at, current_scope_last) ``` -So retained first/last bounds accumulate **only across finalized scopes in -which the contributor was individually represented**. Activity observed in -an earlier scope where the contributor stayed overflow-only does **not** -contribute — those work rows are destroyed at that scope's COMMIT, and no -claim can be made about them. The spec §27 amendment states this so the -field names never imply an all-repository-touch guarantee the storage -model cannot provide. +So the build's intended first/last bounds accumulate **only across +finalized scopes in which the contributor was individually represented**. +Activity observed in an earlier scope where the contributor stayed +overflow-only does **not** contribute — those work rows are destroyed at +that scope's COMMIT, and no claim can be made about them. The staged +bounds land on the shared retained row only inside the promotion +transaction (§5.7). The spec §27 amendment states this so the field names +never imply an all-repository-touch guarantee the storage model cannot +provide. ### 5.5 Unlinked and bots @@ -395,10 +454,17 @@ contributors exactly without their logins. ### 5.6 Pre-flip build abort — explicit cleanup -Scope COMMITs legitimately create `ownership_contributors` rows and -`building_version` snapshots before the build completes. If the build is -abandoned before promotion (error, gate-off, operator abort), those rows -must not orphan — this is identity minimization, not housekeeping. +Scope COMMITs legitimately create `ownership_contributors` rows (for +contributors new to the repository, §5.3) and `building_version` +snapshots before the build completes. If the build is abandoned before +promotion, those rows must not orphan — this is identity minimization, +not housekeeping. + +Crucially, **existing contributor rows referenced by active N require no +save/restore** — under this design they are never mutated during the N+1 +build (§5.7), so there is nothing to roll back. This is a stronger +invariant than save/restore: the shared row cannot drift in the first +place. **Abandon transaction (one tx, under the same sync-row lock):** @@ -406,18 +472,64 @@ must not orphan — this is identity minimization, not housekeeping. `phase != 'done'`. 2. Delete `ownership_snapshots` where `repository_id` and `calculation_version = V`. -3. Delete `ownership_history_work` and `ownership_history_identity_work` +3. Delete `ownership_history_work` and `ownership_contributor_build_work` for that repo/version. 4. Clear `building_version`, `history_cursor` build state, and `pending_scopes` for the build. -5. **Prune** `ownership_contributors` rows for the repository not - referenced by *any* `ownership_snapshots` row in *any* version — the - anti-join is what makes it safe: contributors still referenced by the - active/retained version survive; contributors introduced only by the - aborted build are removed. Synthetic `unlinked`/`other` rows prune the - same way — they are reconstructable. -6. `active_version` is never touched; phase falls back to the state implied - by whether an active version exists (`done`/`error` as appropriate). +5. **Prune** `ownership_contributors` rows for the repository referenced + by *zero* `ownership_snapshots` rows across *all* versions — the + anti-join is what makes it safe: N-referenced contributors survive; + N+1-only rows created solely to satisfy building-snapshot FKs are + removed. Synthetic `unlinked`/`other` rows prune the same way — they + are reconstructable. +6. `active_version` is never touched; phase falls back to the state + implied by whether an active version exists (`done`/`error` as + appropriate). Existing contributor metadata needs no restoration — + it was never mutated. + +### 5.7 Promotion — atomic metadata application + version flip + +When `pending_scopes` is empty, ONE transaction under the sync-row lock: + +1. Verify: `building_version = V`; no `ownership_history_work` rows for V; + no unresolved `ownership_contributor_build_work` rows for V (every + represented key has a complete staged identity); snapshot cardinality + within §11 bounds. +2. Apply staged contributor metadata to retained rows — + `UPDATE ownership_contributors ... FROM ownership_contributor_build_work` + setting `github_node_id` (consistency-checked: same `work_key` must + carry the same node id — mismatch fails closed), `display_login`, + `is_bot` (the frozen pass-1 value), `first_seen_at`, `last_seen_at`. + New contributors inserted at scope COMMITs already carry these values; + the update is idempotent over them. +3. `active_version = V; building_version = null;` clear build cursor state; + `phase = 'done'` — single row update. +4. Delete `ownership_contributor_build_work` for the repo/version. +5. Commit. + +The metadata application and the `active_version` flip are visible +**together**: before this transaction, active-N responses can only read +the old contributor metadata; after it, they read N+1 snapshots joined to +the new metadata. No intermediate mixed state is observable — readers +under read-committed see either fully-pre or fully-post commit. A failed +promotion leaves active N and all shared contributor metadata unchanged +and building V retryable. Post-flip garbage collection of old-version +snapshots is a separate later transaction (spec §13) and can prune +contributors left unreferenced by that GC the same way §5.6 does. + +### 5.8 Why this is required — the shared-metadata leak + +The deployed `dash_ownership_head` / `dash_ownership_timeline` RPCs +(`012_ownership.sql`) filter `ownership_snapshots` by +`calculation_version = st.active_version` but join `display_login` / +`is_bot` from `ownership_contributors` **without any version column to +filter on** — the contributor row is shared reference state by design. +Any in-build UPDATE of `display_login`, `is_bot`, or the seen bounds on a +contributor also referenced by active N would immediately alter live +active-N responses — and, being unversioned, could not be restored by the +abort anti-join prune. Revision-2's scope-level staging still wrote the +final metadata at scope COMMIT, which had exactly this defect. The +build-level relation + promotion-time application is the fix. Required implementation-time tests: @@ -425,8 +537,9 @@ Required implementation-time tests: committed → abort → zero V1 snapshots, zero V1-only contributor rows, zero rows in both work relations. - **N → N+1 abort:** N active, some N+1 scopes committed → abort → all N - rows untouched; contributors referenced by N survive; contributors - introduced only by N+1 pruned; no N+1 snapshots or work rows remain. + rows untouched (snapshots *and* contributor metadata byte-identical); + contributors referenced by N survive; contributors introduced only by + N+1 pruned; no N+1 snapshots or work rows remain. --- @@ -442,7 +555,7 @@ p_expected_cursor, p_next_cursor, p_deltas jsonb -- accumulate: [{period_start, work_key, is_bot, -- touches, first_seen_at, last_seen_at}] p_identities jsonb -- resolve: [{work_key, github_node_id, - -- display_login, is_bot}] + -- display_login, observed_is_bot}] ``` One transaction, in order: @@ -459,8 +572,14 @@ One transaction, in order: first_seen_at = least(...), last_seen_at = greatest(...)`; cap check in the same statement (§7 bounds → raise → caller marks error). Apply `p_identities`: `INSERT ... ON CONFLICT DO UPDATE` into - `ownership_history_identity_work` — **staged** rows only, never - `ownership_contributors`. + `ownership_contributor_build_work` at `(repository_id, + calculation_version, work_key)` grain — **staged** rows only, never + `ownership_contributors`. Each staged row's `is_bot` is written from + the **frozen pass-1** counters (not the pass-2 observation); if + `observed_is_bot` differs from the frozen value the transaction raises + — classification drift fails the scope closed (§5.3). A conflicting + stored `github_node_id` for an existing `work_key` is impossible by + construction and also fails closed. 5. `jsonb_set(history_cursor, '{current,cursor}', p_next_cursor)` and, when `hasNextPage` was false, set `accumulate_done` / flip `pass`. 6. Commit. @@ -500,7 +619,7 @@ COMMIT deletes them before the next scope starts. |---|---|---| | Distinct `work_key` per scope | 10,000 | ~5× facebook/react's repo-wide contributor count for a *single* scope | | Accumulation rows per scope | 100,000 | keys × active months; ~10 MB transient worst case | -| Identity-staging rows per scope | ≤ represented keys, ≤10,000 | one row per represented key — the narrowest possible set | +| Build-work rows per build | ≤ represented keys, ≤10,000 | one row per represented key across the whole build — the narrowest possible set | | Page delta rows | ≤100 | one GraphQL page | Breach → `phase='error'`, `error='OWNERSHIP_INTERNAL_ERROR'` — **never** @@ -542,21 +661,34 @@ implementation tranche is approved: 1. **New data class — EPHEMERAL short-lived persisted operational state:** (a) per (scope × month × work_key) counters + `is_bot` + min/max committedDate; (b) per represented-key identity staging - (`work_key`, `github_node_id`, `display_login`) existing only between - pass-1 completion and scope COMMIT. `work_key` is a repo-scoped - pseudonymous identifier — permitted persisted only under the §3 - conditions. + (`work_key`, `github_node_id`, `display_login`, frozen `is_bot`, + `first_seen_at`, `last_seen_at`) at **repository-build** grain — + `ownership_contributor_build_work`, grain `(repository_id, + calculation_version, work_key)` — existing only while a building + version exists. `work_key` is a repo-scoped pseudonymous identifier — + permitted persisted only under the §3 conditions. 2. **§6.4 preserved verbatim** — overflow contributors get no retained identity row and their GitHub identity fields never reach any - persisted state; durable contributor rows are created only inside - scope finalization from staged, already-represented keys. + persisted state; staged build-work rows exist only for contributor + keys pass 1 already proved individually represented. + **Version isolation:** shared `ownership_contributors` rows are never + mutated while a `building_version` is incomplete — the dashboard RPCs + join contributor metadata unversioned, so any mid-build metadata write + would leak into active-version responses (§5.8). Intended N+1 + metadata lives in build staging and is applied in the **same + transaction** as the `active_version` flip. A scope COMMIT may create + a *new* contributor row solely to satisfy building-snapshot FKs — + invisible to active-N output (no N snapshot references it) and pruned + by the abort path if unreferenced. **Attribution freeze:** `is_bot` is frozen by pass 1 along with all - counts; pass 2 verifies rather than re-derives it, and a mismatch - fails the scope closed rather than silently reclassifying. + counts; pass 2 verifies rather than re-derives it, and a mismatch — + between passes *or across scopes of one build* — fails the build + closed rather than silently reclassifying. **Abort cleanup:** abandoned builds delete their versioned snapshots and ephemeral state, then prune `ownership_contributors` rows unreferenced by any snapshot in any version — retained-version - contributors always survive (§5.6). + contributors always survive (§5.6); their metadata was never mutated, + so nothing needs restoration. **Seen-at semantics:** retained `first_seen_at`/`last_seen_at` accumulate only across finalized scopes where the contributor was individually represented (§5.4) — no all-repository-touch claim. @@ -641,17 +773,20 @@ endpoint form, now sha-pinned. `on delete cascade`; unique `(scope_id, calculation_version, period_start, work_key)`; CHECKs on work_key shape, non-negative counts, `calculation_version > 0`, first ≤ last seen. -- `create table public.ownership_history_identity_work` — §4.4 columns: - tenant/scope/version + `work_key` + `github_node_id` + `display_login` - + timestamps only (**no `is_bot`** — classification is frozen pass-1 - state, §5.1/§5.3); same composite FK pattern; unique `(scope_id, - calculation_version, work_key)`; CHECK work_key is a 64-hex key (never - `'unlinked'`/`'other'` — synthetic buckets carry no identity and must - never be staged). +- `create table public.ownership_contributor_build_work` — §4.4 columns: + tenant/version + `work_key` + `github_node_id` + `display_login` + + frozen `is_bot` + `first_seen_at`/`last_seen_at` + timestamps; + composite tenant FK `(repository_id, user_id) → repositories`; unique + `(repository_id, calculation_version, work_key)`; CHECK work_key is a + 64-hex key (never `'unlinked'`/`'other'` — synthetic buckets carry no + identity and must never be staged). - RLS enabled, zero policies; revoke browser roles; `service_role` DML only — on **both**. - `own_checkpoint_page(...)` — §6.1; `security invoker`, `EXECUTE → - service_role`, all other grants revoked. + service_role`, all other grants revoked. The same transactional pattern + covers scope-COMMIT (§5.3), promotion (§5.7), and abort (§5.6) — whether + as additional functions or a client-managed multi-statement tx is an + implementation detail of 1C-B. - `own_sweep_stale_work(...)` — the §7 sweep, same security shape. - `backup-scope.mjs`: append both tables to `EPHEMERAL_TABLES` (same PR — CI requires classification). @@ -704,14 +839,28 @@ endpoint form, now sha-pinned. frozen value, never a pass-2 observation. - **Candidate/upsert equivalence:** the contributor candidate set and each candidate's `first_seen_at`/`last_seen_at` must equal the one-shot - result, with the documented merge rule — repo-level bounds accumulate - only over scopes where the contributor was individually represented - (§5.4), and the simulated multi-scope run must exercise the - least/greatest upsert across finalized scopes. + result, with the documented merge rule — bounds accumulate only over + scopes where the contributor was individually represented (§5.4), and + the simulated multi-scope run must exercise the least/greatest merge + into the build-level staging row. - **Abort cases:** initial-build abort leaves zero V1 snapshots, zero V1-only contributor rows, zero work/staging rows; N→N+1 abort preserves all N rows and N-referenced contributors while pruning N+1-only contributors (§5.6). +- **Version isolation (§5.7/§5.8):** + - N active + N+1 scope committed: N+1 snapshots exist, but the + active-version RPC output is byte-identical — existing contributors' + `display_login`, `is_bot`, `first/last_seen_at` unchanged; + - N+1 abort: N snapshots and N contributor metadata unchanged, N+1 + snapshots gone, N+1-only contributors pruned, all build staging gone; + - N+1 promotion: staged metadata application and the `active_version` + flip commit in one transaction — post-promotion responses show N+1 + snapshots + new metadata, no observable intermediate mix; + - database error mid-promotion: active N and its contributor metadata + untouched, building N+1 fully retryable; + - contributor represented across several N+1 scopes: build-level + staged bounds aggregate all represented scopes correctly while the + shared row stays unchanged until promotion. --- @@ -745,6 +894,10 @@ endpoint form, now sha-pinned. | Stale concurrent worker | lease + CAS | loser gets 'stale' | CAS rejects second apply | — | lease takeover after expiry | | CAS cursor mismatch | — | reload, refetch from durable C | — | — | — | | COMMIT succeeds, response lost | scope marked done in same tx | retry sees scope advanced → 'stale' | single tx — no partial | scope already done | work rows deleted by that same tx | +| Promotion tx fails mid-flight | abort — no writes | retry re-runs promotion | single tx | — | building V retryable; active N untouched | +| Promotion commits, response lost | V promoted atomically | caller reload sees active_version=V | flip+metadata in one tx | — | staging already deleted by that tx | +| Cross-scope bot drift mid-build | staged frozen is_bot ≠ later scope's pass-1 value | fail closed → fresh build boundary | — | — | abort path cleans build state | +| Shared contributor metadata mutate attempt during build | **none — design forbids it** | n/a | retained rows are read-only during build | — | nothing to roll back (§5.8) | --- @@ -782,7 +935,7 @@ endpoint form, now sha-pinned. |---|---|---| | A. Buildable without migration 013? | **NO** | §1 — no existing table can hold per-contributor intermediate state within the contract | | B. Temporary per-contributor state unavoidable? | **YES** | §0 — exact fold requires complete sets; multi-invocation walks lose memory | -| C. Fifth ephemeral table required? | **YES — two relations** | §4.4 — counters + identity staging are different data classes; splitting keeps the counter relation identity-free by shape | +| C. Fifth ephemeral table required? | **YES — two relations** | §4.4 — per-scope counters + build-level contributor staging are different data classes and different lifecycles | | D. Two-pass identity resolution required? | **YES** | §5.1/§5.3 — the only way to keep overflow identity fields unpersisted while still populating durable contributor rows | | E. Fixed-revision history query required? | **YES** | §10 — `defaultBranchRef` drifts mid-build | | F. Additional measured spike required? | **YES** | §10 — `object(oid:)` history is unmeasured | diff --git a/docs/ownership-data-privacy-spec.md b/docs/ownership-data-privacy-spec.md index b95e565..31ffe52 100644 --- a/docs/ownership-data-privacy-spec.md +++ b/docs/ownership-data-privacy-spec.md @@ -1235,43 +1235,52 @@ The proposed amendment is deliberately narrow: 1. **New data class — EPHEMERAL build working state, two relations.** (a) Per (scope × month × work_key) touch counters plus the `is_bot` boolean and min/max `committedDate`, existing only while - `building_version` is set for that repository. (b) Per - represented-work-key identity staging carrying exactly - `github_node_id` and `display_login` (no `is_bot` — bot - classification is frozen pass-1 attribution, not a second staged - authority), existing only between a scope's completed accumulation and - its finalization. `work_key` is the same `contributor_key` defined in - §5/§6 — a **deterministic, repository-scoped pseudonymous identifier** - derived from GitHub node identity: no plaintext identity, not - user-displayable, linkable within the repository, and therefore not - anonymous. It may appear in persisted build state only under the + `building_version` is set for that repository. (b) **Build-level** + contributor staging at grain `(repository_id, calculation_version, + work_key)` carrying `github_node_id`, `display_login`, frozen + `is_bot`, and `first_seen_at`/`last_seen_at`, existing only for + contributor keys already proved individually represented, and only + while the build lives. `work_key` is the same `contributor_key` + defined in §5/§6 — a **deterministic, repository-scoped pseudonymous + identifier** derived from GitHub node identity: no plaintext identity, + not user-displayable, linkable within the repository, and therefore + not anonymous. It may appear in persisted build state only under the EPHEMERAL conditions above — service-role-only, backup-excluded, never - logged, deleted at finalization, cascade-deleted, swept on abandoned - builds. + logged, cascade-deleted, swept on abandoned builds. 2. **The §6.4 guarantee is preserved verbatim** — overflow-only contributors still get **no retained identity row** and their GitHub identity fields (`github_node_id`, `display_login`) are **never - persisted in any table**. Staged identity rows are written only for - contributor keys pass 1 already proved individually represented; - `ownership_contributors` rows are created **only inside the atomic - scope-finalization transaction**, never during page-by-page resolve — - so an abandoned scope can never orphan a durable identity. If a build - is abandoned after scope finalizations but before promotion, the abort - transaction deletes the building-version's snapshots and ephemeral - state and then **prunes `ownership_contributors` rows unreferenced by - any snapshot in any version** — contributors still referenced by the - active/retained version always survive; this pruning is part of - identity minimization, not housekeeping. + persisted in any table**. Two further guarantees are required because + `ownership_contributors` is shared across calculation versions while + the dashboard RPCs filter only snapshots by `active_version`: + **(a) shared contributor rows are never mutated while a + `building_version` is incomplete** — mid-build metadata writes would + leak into active-version responses; intended building-version metadata + lives only in build staging and is applied to shared rows in the same + transaction as the `active_version` flip. **(b)** a scope + finalization may create a *new* contributor row solely to satisfy + building-snapshot foreign keys — no active-version snapshot can + reference it, so active output is unchanged — and if the build aborts, + the abort transaction deletes the building-version snapshots and + ephemeral state, then **prunes `ownership_contributors` rows + unreferenced by any snapshot in any version**: active-version- + referenced contributors always survive (and were never mutated, so + nothing needs restoring); this pruning is part of identity + minimization, not housekeeping. 3. **Two-pass identity resolution, attribution-frozen.** The accumulate pass fixes every metric — work keys, counts, Top-25, unlinked share, human counts, **and `is_bot` classification** (`human_contributor_count` depends on it; the retained `ownership_contributors.is_bot` is always - the pass-1 value). The resolve pass re-reads `author.user.{id,login}` - solely to stage identities for represented keys and cannot mutate - metrics — a bot-classification mismatch between passes is identity - drift: the scope fails closed and restarts at a fresh accumulate. A - required key that stays unresolvable fails the scope closed the same - way — GitHub's account-link resolution can change independently of the + the frozen pass-1 value, staged at build level and applied at + promotion). The resolve pass re-reads `author.user.{id,login}` solely + to stage identities for represented keys and cannot mutate metrics — + a bot-classification mismatch between passes, **or across scopes of + one build**, is identity drift: the scope fails closed and restarts at + a fresh accumulate (cross-scope drift restarts the repository build + from a new pinned boundary — one contributor cannot be human in one + scope and bot in another within one calculation version). A required + key that stays unresolvable fails the scope closed the same way — + GitHub's account-link resolution can change independently of the pinned commit, and the build succeeds only while selected keys remain resolvable and consistent. 4. **Build-boundary SHA.** `ownership_sync_state.history_cursor` may carry From 2a63e8370ba961871d235ea7a1a0058f0eb3f1cf Mon Sep 17 00:00:00 2001 From: aliferous3 Date: Sun, 18 Oct 2026 22:19:09 +0400 Subject: [PATCH 5/9] docs: ephemeral loss abandons the whole building version MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Revision 4: build-level contributor staging accumulates across already-finalized scopes, so losing any required ephemeral state invalidates the entire building version — piecemeal resume is impossible. Adds the canonical own_abandon_build operation reused by the stale sweep, restore reconciliation, operator abort, and invalid-state recovery; defines build-state validity invariants; forbids overlapping building versions; keeps both relations EPHEMERAL (loss = discarded build, never active data). Docs only. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --- docs/ownership-checkpoint-architecture.md | 197 +++++++++++++++------- docs/ownership-data-privacy-spec.md | 10 ++ 2 files changed, 150 insertions(+), 57 deletions(-) diff --git a/docs/ownership-checkpoint-architecture.md b/docs/ownership-checkpoint-architecture.md index fb3db99..252f578 100644 --- a/docs/ownership-checkpoint-architecture.md +++ b/docs/ownership-checkpoint-architecture.md @@ -1,16 +1,19 @@ # Ownership Checkpoint Architecture — Phase 1C-B0 Design -> **Status: design only, revision 3. Nothing in this document is implemented -> or live.** Revision 2→3 correction — **version isolation**: retained -> `ownership_contributors` rows are shared across calculation versions while -> the dashboard RPCs filter only snapshots by `active_version`; a building -> version mutating shared contributor metadata would leak into active-N -> responses and could not be restored on abort. Contributor identity/metadata -> is therefore staged **build-level** and applied atomically at promotion. -> Earlier corrections: durable contributor writes inside the atomic -> scope-COMMIT only; the unimplementable resolve-drift fold removed; -> `work_key` classified as a linkable pseudonymous identifier; concrete -> TTL/cleanup; slim cursor jsonb. +> **Status: design only, revision 4. Nothing in this document is implemented +> or live.** Revision 3→4 correction — **ephemeral-loss semantics**: because +> `ownership_contributor_build_work` accumulates across already-finalized +> scopes, losing any required ephemeral state invalidates the ENTIRE +> building version — there is one canonical `own_abandon_build` operation +> reused by the stale sweep, restore reconciliation, operator abort, and +> invalid-state recovery; a build never resumes piecemeal. Revision 2→3 — +> **version isolation**: shared `ownership_contributors` rows are never +> mutated during a build (RPCs version-filter snapshots only); metadata is +> staged build-level and applied atomically at promotion. Earlier: +> durable contributor writes inside the atomic scope-COMMIT only; the +> unimplementable resolve-drift fold removed; `work_key` classified as a +> linkable pseudonymous identifier; concrete TTL/cleanup; slim cursor +> jsonb. > > This document resolves the §9.1 hard blocker in > [ownership-data-privacy-spec.md](ownership-data-privacy-spec.md): how to run @@ -452,7 +455,7 @@ reclassifying. `is_bot` travels on the work row as a bare boolean — the only way `human_contributor_count` can include/exclude overflow contributors exactly without their logins. -### 5.6 Pre-flip build abort — explicit cleanup +### 5.6 Build abandonment — THE canonical cleanup operation Scope COMMITs legitimately create `ownership_contributors` rows (for contributors new to the repository, §5.3) and `building_version` @@ -466,26 +469,69 @@ build (§5.7), so there is nothing to roll back. This is a stronger invariant than save/restore: the shared row cannot drift in the first place. -**Abandon transaction (one tx, under the same sync-row lock):** +**The canonical abandon transaction** (`own_abandon_build`, prospective +013 — the ONE semantic abandon operation; the stale-build sweep, restore +reconciliation, operator abort, and invalid-state recovery all reuse it — +there is no second cleanup semantics anywhere): -1. Verify `(user_id, repository_id)` row, `building_version = V`, - `phase != 'done'`. -2. Delete `ownership_snapshots` where `repository_id` and +1. `SELECT FOR UPDATE` the `(user_id, repository_id)` sync-state row. +2. Verify `building_version = V` (the expected version is supplied). +3. Delete `ownership_snapshots` where `repository_id` and `calculation_version = V`. -3. Delete `ownership_history_work` and `ownership_contributor_build_work` +4. Delete `ownership_history_work` and `ownership_contributor_build_work` for that repo/version. -4. Clear `building_version`, `history_cursor` build state, and +5. Clear `building_version`, `history_cursor` build state, and `pending_scopes` for the build. -5. **Prune** `ownership_contributors` rows for the repository referenced +6. **Prune** `ownership_contributors` rows for the repository referenced by *zero* `ownership_snapshots` rows across *all* versions — the anti-join is what makes it safe: N-referenced contributors survive; - N+1-only rows created solely to satisfy building-snapshot FKs are + V-only rows created solely to satisfy building-snapshot FKs are removed. Synthetic `unlinked`/`other` rows prune the same way — they are reconstructable. -6. `active_version` is never touched; phase falls back to the state - implied by whether an active version exists (`done`/`error` as - appropriate). Existing contributor metadata needs no restoration — - it was never mutated. +7. `active_version` is never touched; phase/error fall back to the state + implied by whether an active version exists (closed taxonomy only). + Existing contributor metadata needs no restoration — it was never + mutated. + +### 5.6a The ephemeral-loss invariant (CANONICAL) + +`ownership_contributor_build_work` accumulates metadata across +**already-finalized** scopes of the building version — staged +`display_login`, the frozen build-level `is_bot`, and represented-scope +seen bounds exist nowhere else (`ownership_snapshots` does not carry +them). Therefore: + +> **If `building_version` is non-null and any required EPHEMERAL build +> state is lost, stale, corrupt, or deliberately discarded, the ENTIRE +> building version is invalid. It must never resume piecemeal — not even +> "restart the current scope." The only safe operation is the canonical +> abandon transaction, then a fresh build later from a fresh pinned +> boundary. No previously committed V scope is trusted.** + +A new repository build may not start while `building_version` is set: it +first **resumes** the existing build if §5.6b validity holds, else runs +`own_abandon_build`. `building_version` must be null before a fresh build +initializes — no overlapping building versions, ever. + +### 5.6b Required ephemeral-state validity + +Whether a build is resumable is decided by internal consistency, not by +the mere presence/absence of counter rows: + +| State-machine position | Required present | Allowed absent | +|---|---|---| +| ACCUMULATE mid-scope | `history_work` prefix rows may exist (or be empty pre-first-page); `contributor_build_work` for previously finalized scopes' represented keys | next pages | +| RESOLVE mid-scope | complete `history_work` counters for the scope; `contributor_build_work` for prior scopes | identity rows not yet observed in pass 2 | +| Between scopes | `contributor_build_work` for all represented keys of committed scopes; `pending_scopes` non-empty | `history_work` for the just-committed scope (consumed) | +| Ready to promote | `pending_scopes` empty; complete `contributor_build_work`; zero `history_work` | — | + +Zero `history_work` rows is **not** evidence of loss — it is the legal +state after each scope COMMIT and before promotion. Validation checks: +`building_version` non-null, phase/pass/current-scope consistent with +`history_cursor`, `pending_scopes` well-formed, every represented key of +every already-committed scope having a `contributor_build_work` row, and +no orphaned V snapshots without their staging. If any check fails — never +reconstruct heuristically → `own_abandon_build`. ### 5.7 Promotion — atomic metadata application + version flip @@ -625,29 +671,36 @@ COMMIT deletes them before the next scope starts. Breach → `phase='error'`, `error='OWNERSHIP_INTERNAL_ERROR'` — **never** silently partial; an over-cap scope is an error, not a folded result. -**Cleanup — concrete, not aspirational:** +**Cleanup — concrete, and routed through the canonical abandon:** - Rows carry `created_at`/`updated_at`; the checkpoint upsert touches `updated_at` on every application, so **activity is directly visible on the rows themselves.** -- **Stale threshold:** work/staging rows whose repo build shows no - checkpoint for **14 days** (measured via row `updated_at` joined to - `ownership_sync_state.updated_at`) are abandoned. Two weeks is far - beyond any legitimate pause — slices checkpoint every ~40 s while a - build lives; even a fully paused build must beat 14 days. -- **Active-build protection:** a build that is checkpointing (any page tx - within the window) is by definition not stale — the threshold only - fires on genuinely dead state. -- **Sweep triggers:** (a) unconditionally at every build start for that - repo — rows with `calculation_version ≠ building_version` or whose - scope is gone are deleted; (b) a periodic sweep function - (`own_sweep_stale_work`, proposed in 013) invoked by whatever scheduler - the implementation tranche wires — so cleanup eventually runs even if - the repo never builds again; (c) unconditional FK cascade on +- **Stale threshold:** a repository build showing no checkpoint for + **14 days** (no valid lease; row `updated_at` and + `ownership_sync_state.updated_at` both older) is abandoned. Two weeks + is far beyond any legitimate pause — slices checkpoint every ~40 s + while a build lives. +- **A stale build is never cleaned piecemeal.** `own_sweep_stale_work` + must NOT bare-`DELETE` work rows for a repo whose `building_version` + is still set — that would strand V's snapshots and V-only contributors. + Under the sync-row lock it verifies `building_version = V` and genuine + staleness, then runs `own_abandon_build` (§5.6) — the identical + delete/prune/clear sequence. Only **orphan rows with no + `building_version`** (e.g. leftovers of an already-cleared build) may + be deleted directly. +- **Active-build protection:** a build checkpointing within the window is + by definition not stale — the threshold only fires on genuinely dead + state. +- **Sweep triggers:** (a) at every build start for that repo — validity + check §5.6b → resume or `own_abandon_build`; (b) a periodic sweep + function (`own_sweep_stale_work`, proposed in 013) invoked by whatever + scheduler the implementation tranche wires — so cleanup eventually runs + even if the repo never builds again; (c) unconditional FK cascade on repository/account deletion. -- **Error-build behavior:** `phase='error'` leaves work rows for operator - inspection (they're backup-excluded and bounded) until the 14-day sweep - or the next build start. +- **Error-build behavior:** `phase='error'` leaves build state for + operator inspection (backup-excluded and bounded) until the 14-day + sweep runs the abandon path or the next build start does. - **Gate-off behavior:** identical — no caller runs, rows age out via the sweep; no special case needed. @@ -707,14 +760,28 @@ relaxation of §5's prohibited list. - Both work relations → **EPHEMERAL** in `backup-scope.mjs` (classification is mandatory — an unclassified table fails CI). - Verified, not assumed: losing them costs a restarted scope only; - dumping them would persist pseudonymous state for zero benefit. -- **Restore semantics:** restore returns sync_state (OPERATIONAL) but no - work rows. Resume detects `building_version set ∧ work rows absent ∧ - scope mid-walk` → resets `current.cursor = null`, `pass='accumulate'` - for that scope → pages re-fetched, CAS txs rebuild. No page can be - double-applied — cursor/delta atomicity is per-transaction, not - per-backup. `active_version` rows are untouched throughout. + Verified against the revision-3 state model: losing them never loses + active user data — the recovery cost is **"discard the incomplete + building version and rebuild it,"** not "resume the current scope." + Dumping them would persist pseudonymous state for zero benefit; the + EPHEMERAL classification stands. +- **Restore semantics — abandon, never piecemeal-resume:** restore + returns `ownership_sync_state` (OPERATIONAL) but no work rows. If the + restored row has `building_version = V`, the required build state for V + is by definition absent/incomplete → V is unrecoverable → run + `own_abandon_build`: delete V snapshots, delete any surviving V work + rows, prune unreferenced V-only contributors, clear V build state. Then + a later ownership run begins a NEW complete build — re-enumerate + scopes, obtain a fresh pinned revision, accumulate from the start; **no + previously committed V scope is trusted.** + - *Initial V1 restore* (`active_version=null`, partial V1): remove all + V1 artifacts → repo returns to "ownership not ready" → a fresh V1 + build starts later. + - *N active + N+1 building restore:* remove all partial N+1 artifacts → + N stays byte-identical and active → a fresh N+1 build starts later. + - `active_version` is untouched in both cases — every active row comes + from CORE tables (`ownership_scopes`, `ownership_contributors`, + `ownership_snapshots` at N, `ownership_sync_state`). --- @@ -784,10 +851,20 @@ endpoint form, now sha-pinned. only — on **both**. - `own_checkpoint_page(...)` — §6.1; `security invoker`, `EXECUTE → service_role`, all other grants revoked. The same transactional pattern - covers scope-COMMIT (§5.3), promotion (§5.7), and abort (§5.6) — whether - as additional functions or a client-managed multi-statement tx is an - implementation detail of 1C-B. -- `own_sweep_stale_work(...)` — the §7 sweep, same security shape. + covers scope-COMMIT (§5.3) and promotion (§5.7) — whether as additional + functions or a client-managed multi-statement tx is an implementation + detail of 1C-B. +- `own_abandon_build(p_user, p_repository, p_version)` — **the canonical + build-abandonment primitive** (§5.6): locks the sync-state row, + validates the expected `building_version`, deletes the version's + snapshots and both ephemeral relations, anti-join prunes unreferenced + contributors, clears build state, preserves `active_version`. There is + ONE semantic abandon operation — the stale sweep, restore + reconciliation, operator abort, and invalid-state recovery all call it. +- `own_sweep_stale_work(...)` — repository-build-aware: bare-deletes + orphan work rows only when `building_version` is null; for a genuinely + stale live build (no lease, no activity in 14 days) it calls + `own_abandon_build` — same security shape. - `backup-scope.mjs`: append both tables to `EPHEMERAL_TABLES` (same PR — CI requires classification). - Recovery-drill addition: assert the tables exist post-migration and are @@ -879,8 +956,14 @@ endpoint form, now sha-pinned. | DB timeout, unknown outcome | maybe committed | retry: C′ → 'stale' else apply | exactly-once either way | — | — | | Worker timeout | as crash cases | next worker | CAS | — | lease expiry §14 | | Deploy during build | in-flight tx aborts or completes | next deploy resumes | CAS | — | additive 013 keeps reads compatible | -| Gate off mid-build | ephemeral rows idle | no caller runs | — | — | 14-day sweep / next enable | -| Repo disconnected mid-build | rows persist; calls stop at gate | no | — | — | resume on reconnect; sweep if abandoned | +| Gate off mid-build | ephemeral rows idle | no caller runs | — | — | stale sweep → `own_abandon_build` / next enable | +| Repo disconnected mid-build | rows persist; calls stop at gate | no | — | — | resume on reconnect if §5.6b valid; else canonical abandon | +| **Stale TTL while N+1 building** | N+1 snapshots + staging exist; build dead ≥14d | n/a — whole V discarded | — | — | `own_abandon_build`: V snapshots+work gone, V-only contributors pruned, N untouched | +| **Restore: N+1 sync state, no ephemeral tables** | sync row claims building V; work/staging absent | **never piecemeal-resume** | — | — | `own_abandon_build` → fresh N+1 build later; N stays byte-identical | +| **Restore: initial V1, no ephemeral state** | partial V1 artifacts | never resume | — | — | V1 artifacts removed; `active_version=null` → "not ready"; fresh V1 later | +| **Orphan work rows, `building_version=null`** | rows without a live build | n/a | — | — | direct sweep delete permitted — no build semantics to preserve | +| **Corrupt/inconsistent ephemeral state** | violates §5.6b invariants | never reconstruct heuristically | — | — | `own_abandon_build` → fresh build | +| **New build requested while `building_version` set** | existing build state | resume iff §5.6b passes; else `own_abandon_build` first | — | — | `building_version=null` before new build initializes — no overlapping versions | | Repo deleted mid-build | FK cascade removes work+staging+sync rows | n/a | — | — | cascade is total | | Installation revoked | state intact | fails at next fetch | CAS | — | `GITHUB_ACCESS_REVOKED` | | Pinned revision disappears | pages error; cursor=C | retry once → fail closed | partial scope state never served (building_version) | — | `OWNERSHIP_HISTORY_FAILED`; next build re-pins | @@ -942,7 +1025,7 @@ endpoint form, now sha-pinned. | G. Exact Top-25 preserved? | **YES** | §3 — work key = contributor key, identical comparator, frozen pass-1 metrics | | H. Overflow identity fields unpersisted? | **YES** | §4.4 — staging accepts represented keys only; durable contributor rows exist only via COMMIT | | I. Every page transition exactly-once? | **YES** | §6.2 — single-transaction CAS, proven | -| J. Work state backup-excludable safely? | **YES** | §9 — loss = restarted scope, never user data | +| J. Work state backup-excludable safely? | **YES** | §9/§5.6a — loss = discarded *building* version (canonical abandon + fresh rebuild), never active user data | ## 17. Readiness verdict diff --git a/docs/ownership-data-privacy-spec.md b/docs/ownership-data-privacy-spec.md index 31ffe52..e353e43 100644 --- a/docs/ownership-data-privacy-spec.md +++ b/docs/ownership-data-privacy-spec.md @@ -1296,6 +1296,16 @@ The proposed amendment is deliberately narrow: when they unlink during pass 2 (unimplementable — a null `author.user` carries no node id mapping back to the work key; pass-2 non-resolution instead fails the scope closed). +6. **Ephemeral loss invalidates the building version only.** The EPHEMERAL + relations are intentionally not backed up; their loss never loses + `active_version` data. But because build-level staging accumulates + across already-finalized scopes, an incomplete build whose ephemeral + state is lost, stale, corrupt, or absent after restore is **wholly + discarded** through one canonical abandon operation (delete its + versioned snapshots and ephemeral rows, prune its unreferenced + contributors, preserve `active_version`) — piecemeal resumption, + including restarting only the current scope, is prohibited because the + discarded metadata is not reconstructable from snapshots. If this amendment is rejected, the honest alternatives are: (a) accept per-scope restart-in-single-invocation with no completion guarantee on From dbd19783d1f34b8c2535098ae7e60daffb8e1e5d Mon Sep 17 00:00:00 2001 From: aliferous3 Date: Sun, 18 Oct 2026 22:26:51 +0400 Subject: [PATCH 6/9] docs: split scope identity from versioned membership ownership_scopes is shared across calculation versions and the deployed dash_ownership_scopes enumerates base scope rows unversioned, so a building version could leak new scopes or mutate active exclusion state. Design adds retained ownership_scope_versions (CORE), moves excluded/exclusion_reason to per-version membership, routes abandon/GC and the scopes RPC through membership, and keeps base scope identity immutable. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --- docs/ownership-checkpoint-architecture.md | 286 +++++++++++++++++----- docs/ownership-data-privacy-spec.md | 50 +++- 2 files changed, 262 insertions(+), 74 deletions(-) diff --git a/docs/ownership-checkpoint-architecture.md b/docs/ownership-checkpoint-architecture.md index 252f578..68f35be 100644 --- a/docs/ownership-checkpoint-architecture.md +++ b/docs/ownership-checkpoint-architecture.md @@ -1,15 +1,18 @@ # Ownership Checkpoint Architecture — Phase 1C-B0 Design -> **Status: design only, revision 4. Nothing in this document is implemented -> or live.** Revision 3→4 correction — **ephemeral-loss semantics**: because -> `ownership_contributor_build_work` accumulates across already-finalized -> scopes, losing any required ephemeral state invalidates the ENTIRE -> building version — there is one canonical `own_abandon_build` operation -> reused by the stale sweep, restore reconciliation, operator abort, and -> invalid-state recovery; a build never resumes piecemeal. Revision 2→3 — -> **version isolation**: shared `ownership_contributors` rows are never -> mutated during a build (RPCs version-filter snapshots only); metadata is -> staged build-level and applied atomically at promotion. Earlier: +> **Status: design only, revision 5. Nothing in this document is implemented +> or live.** Revision 4→5 correction — **scope version isolation**: +> `ownership_scopes` is shared/unversioned exactly like +> `ownership_contributors` — `dash_ownership_scopes` enumerates base scope +> rows regardless of version, so a building version's new scopes or +> exclusion-state mutations would leak into active-N responses. The design +> now splits immutable scope identity from versioned membership +> (`ownership_scope_versions`, CORE). Revision 3→4 — **ephemeral-loss +> semantics**: losing any required ephemeral state invalidates the ENTIRE +> building version; one canonical `own_abandon_build` reused by sweep, +> restore, abort, and invalid-state recovery. Revision 2→3 — **version +> isolation**: shared contributor rows never mutated during a build; +> metadata staged build-level, applied atomically at promotion. Earlier: > durable contributor writes inside the atomic scope-COMMIT only; the > unimplementable resolve-drift fold removed; `work_key` classified as a > linkable pseudonymous identifier; concrete TTL/cleanup; slim cursor @@ -62,9 +65,12 @@ Checked against `012_ownership.sql` as deployed: | `ownership_sync_state` (`history_cursor`/other jsonb) | Technically possible, rejected in §3.2: it is OPERATIONAL (dumped into backups — provisional pseudonyms would leak into every dump), a single JSONB row is a read-modify-write hotspot with unbounded growth, and it cannot be CHECK-constrained per element. | | `ownership_scopes` | Scope definition only; nothing accumulates there. | -Conclusion: **migration 013 is required.** The delta is minimal (§11): two -ephemeral work relations + one checkpoint function. No existing column or -constraint changes. +Conclusion: **migration 013 is required.** The delta (§11): one retained +versioned scope-membership table (`ownership_scope_versions`, CORE), two +ephemeral work relations, the checkpoint/abandon/sweep primitives, and an +amended `dash_ownership_scopes`. No changes to migration 012 itself — the +base scope/contributor columns become non-authoritative or versioned +rather than retroactively edited. --- @@ -80,7 +86,8 @@ Postgres while a build runs: telemetered, fully reconstructable by restarting the build. Persistence ≠ retention. - **RETAINED ownership data** — `ownership_scopes`, - `ownership_contributors`, `ownership_snapshots` at `active_version`: + `ownership_scope_versions` (§5.9), `ownership_contributors`, + `ownership_snapshots` at `active_version`: durable, backed-up, user-visible, deleted only by disconnect/account deletion. @@ -236,20 +243,6 @@ exists** — promotion applies it atomically (§5.7). Hard rules for the staging relation: -- **only represented keys may ever appear** — the resolve page tx writes a - staged row only when `work_key ∈ represented`(recomputed from the - completed accumulation counters); overflow keys observed during pass 2 - are classified transiently and discarded in memory; -- no email, no name, no `databaseId`, no path, no raw nodes; -- EPHEMERAL: backup-excluded, RLS zero-policy, `service_role` only, cascade - on repo/user delete; -- deleted on promotion (consumed), on abort, and by the abandoned-build - sweep; -- **it is the only place GitHub identity fields may exist outside retained - tables — and only for already-represented contributors.** - -Hard rules for the staging relation: - - **only represented keys may ever appear** — the resolve page tx writes a staged row only when `work_key ∈ represented(set recomputed from accumulation counters)`; overflow keys observed during pass 2 are @@ -320,7 +313,8 @@ pending → history → done | error (phase enum — unchanged) Scope order is **not** stored here: `pending_scopes` (existing jsonb column) holds remaining scope ids, and deterministic order is re-derivable -from `ownership_scopes` (`order by depth, scope_path_hash`) at any resume +from the build's `ownership_scope_versions` membership +(`order by depth, scope_path_hash` over V members) at any resume point — no duplicated full-scope list in the cursor. Worst-case `history_cursor` stays ~1 KB; `pending_scopes` carries the ≤500 uuid list (~20 KB) honestly, once, where the column was designed for it. @@ -478,20 +472,29 @@ there is no second cleanup semantics anywhere): 2. Verify `building_version = V` (the expected version is supplied). 3. Delete `ownership_snapshots` where `repository_id` and `calculation_version = V`. -4. Delete `ownership_history_work` and `ownership_contributor_build_work` +4. Delete `ownership_scope_versions` where `repository_id` and + `calculation_version = V` — V's scope membership is version state of + the abandoned build (§5.9). +5. Delete `ownership_history_work` and `ownership_contributor_build_work` for that repo/version. -5. Clear `building_version`, `history_cursor` build state, and +6. Clear `building_version`, `history_cursor` build state, and `pending_scopes` for the build. -6. **Prune** `ownership_contributors` rows for the repository referenced +7. **Prune** `ownership_contributors` rows for the repository referenced by *zero* `ownership_snapshots` rows across *all* versions — the anti-join is what makes it safe: N-referenced contributors survive; V-only rows created solely to satisfy building-snapshot FKs are removed. Synthetic `unlinked`/`other` rows prune the same way — they are reconstructable. -7. `active_version` is never touched; phase/error fall back to the state +8. **Prune** `ownership_scopes` base rows now referenced by NO + `ownership_scope_versions` row AND NO `ownership_snapshots` row — + V-only base identities are removed; N-referenced scopes always + survive. The snapshot FK is `ON DELETE CASCADE`, so this explicit + check is what makes the prune safe rather than destructive; a base + scope still referenced by active/retained data is never deleted. +9. `active_version` is never touched; phase/error fall back to the state implied by whether an active version exists (closed taxonomy only). - Existing contributor metadata needs no restoration — it was never - mutated. + Existing contributor metadata and N scope membership need no + restoration — they were never mutated. ### 5.6a The ephemeral-loss invariant (CANONICAL) @@ -528,9 +531,13 @@ the mere presence/absence of counter rows: Zero `history_work` rows is **not** evidence of loss — it is the legal state after each scope COMMIT and before promotion. Validation checks: `building_version` non-null, phase/pass/current-scope consistent with -`history_cursor`, `pending_scopes` well-formed, every represented key of -every already-committed scope having a `contributor_build_work` row, and -no orphaned V snapshots without their staging. If any check fails — never +`history_cursor`, `pending_scopes` well-formed, **V's +`ownership_scope_versions` membership coherent** — every `pending_scopes` +entry and every V snapshot's `scope_id` has a V membership row +(`excluded=false` for pending entries); every represented key of every +already-committed scope having a `contributor_build_work` row; and no +orphaned V snapshots without their staging. Missing or corrupt V +membership is itself a fail-closed reason. If any check fails — never reconstruct heuristically → `own_abandon_build`. ### 5.7 Promotion — atomic metadata application + version flip @@ -560,8 +567,11 @@ the new metadata. No intermediate mixed state is observable — readers under read-committed see either fully-pre or fully-post commit. A failed promotion leaves active N and all shared contributor metadata unchanged and building V retryable. Post-flip garbage collection of old-version -snapshots is a separate later transaction (spec §13) and can prune -contributors left unreferenced by that GC the same way §5.6 does. +snapshots is a separate later transaction (spec §13): it deletes the old +version's `ownership_scope_versions` membership, then prunes +contributors and base scopes left unreferenced by that GC the same way +§5.6 does — membership first, identity prune second, never cascading +away referenced/active rows. ### 5.8 Why this is required — the shared-metadata leak @@ -587,6 +597,76 @@ Required implementation-time tests: contributors referenced by N survive; contributors introduced only by N+1 pruned; no N+1 snapshots or work rows remain. +### 5.9 Scope identity vs version membership — the same leak class + +`ownership_scopes` is shared/unversioned exactly like +`ownership_contributors`, and the deployed `dash_ownership_scopes` +enumerates **base scope rows** filtered only by `active_version is not +null` — it does not require the scope to belong to the active version. +Three distinct leaks follow: + +- **new-scope leak:** an N+1 scope row inserted during a build would + appear in active-N `dash_ownership_scopes` output (with null N + metrics); +- **metadata leak:** mutating `excluded`/`exclusion_reason`/ + `display_path`/`depth` on a shared scope alters active-N output; +- **deletion hazard:** `ownership_snapshots` FK-references scopes + `ON DELETE CASCADE` — deleting a base scope during a build could + cascade-destroy **active-N snapshots**. + +Resolution — split immutable identity from versioned membership: + +**`ownership_scopes` stays the immutable repository/path identity +relation** (`id, user_id, repository_id, scope_path_hash, display_path, +scope_type, depth, created_at, updated_at`): never mutated merely because +another version is building; never deleted while any snapshot or +membership row references it (the FK cascade makes "delete a referenced +scope" a correctness violation, not a cleanup path). Vendor/generated/ +binary/submodule paths still never get a base row; file scopes remain +impossible. + +**`ownership_scope_versions`** — RETAINED, CORE-classified, grain +`(scope_id, calculation_version)` plus `user_id`/`repository_id` for +DB-enforced tenant scoping: + +| Field | Semantics | +|---|---| +| `scope_id`, `calculation_version` | the scope belongs to that version | +| `excluded`, `exclusion_reason` | **version-sensitive** cardinality state — a dir can be inside the ≤500 set in N and outside in N+1 as the tree changes; `excluded=true` permitted only with `'cardinality_cap'` (existing contract preserved); lives here, not on the shared base row | +| `created_at`, `updated_at` | housekeeping | + +Rules: one membership row ⇒ the scope belongs to that version; +`excluded=false` ⇒ analyzed/included; `excluded=true` ⇒ cardinality-cap +marker for that version only. The base row's `excluded`/ +`exclusion_reason` columns become **non-authoritative legacy** after 013 +(new base rows always `excluded=false`; the columns may be dropped only +if a later migration explicitly justifies it — 012 is never edited +retroactively). + +**Build flow:** at N+1 init — pin revision → enumerate eligible +scopes → insert missing base identity rows → insert N+1 membership rows +→ derive `pending_scopes` from N+1 membership where `excluded=false` → +walk. New base rows may exist pre-promotion **only because dashboard +reads will join membership** — without membership for `active_version` +they are invisible. Existing N membership is never touched. + +**RPC:** `dash_ownership_scopes` must be amended to join +`ownership_scope_versions` on `calculation_version = st.active_version` +and take `excluded`/`exclusion_reason` from membership — N+1-only scopes +cannot appear, and N exclusion state cannot move. `dash_ownership_head`/ +`dash_ownership_timeline` reach scopes only through version-filtered +snapshot rows, so they are already correct — but all three RPCs get an +explicit version-consistency audit at implementation. + +**Promotion:** membership needs no rewrite — the flip swaps which +version's membership the RPC joins, atomically with the contributor +metadata application (§5.7). **GC:** when old version N is collected: +delete N's membership rows → then prune base scopes referenced by no +membership row *and* no snapshot — in that order, never cascading away +referenced/active data. **Abandon** (§5.6) additionally deletes V's +membership rows and prunes V-only base scopes that are then referenced +by nothing. + --- ## 6. Exactly-once checkpointing @@ -720,10 +800,25 @@ implementation tranche is approved: calculation_version, work_key)` — existing only while a building version exists. `work_key` is a repo-scoped pseudonymous identifier — permitted persisted only under the §3 conditions. -2. **§6.4 preserved verbatim** — overflow contributors get no retained - identity row and their GitHub identity fields never reach any - persisted state; staged build-work rows exist only for contributor - keys pass 1 already proved individually represented. +2. **§6.4's substantive guarantee preserved** — the live §6.4 text stays + unchanged until this amendment is adopted (it is not claimed verbatim — + the amendment deliberately persists the pseudonymous `work_key` in + EPHEMERAL build state). What is guaranteed exactly as before: overflow + contributors receive **no retained `ownership_contributors` row** and + their `github_node_id`/`display_login` never reach *any* persisted + state — only the pseudonymous `work_key` may exist, and only in the + short-lived EPHEMERAL relations under the §3 conditions. Staged + build-work rows exist only for contributor keys pass 1 already proved + individually represented. + **Scope version isolation (§5.9):** `ownership_scopes` base rows are + immutable identity only; per-version `excluded`/`exclusion_reason` + state moves to retained `ownership_scope_versions` (CORE, grain + `scope_id × calculation_version`). Building versions may add base + scope identities and N+1 membership, but dashboard reads enumerate + only `active_version` membership — a new N+1 scope is invisible and N + exclusion state cannot move. The base row's exclusion columns become + non-authoritative after 013; base scopes referenced by any snapshot or + membership row are never deleted. **Version isolation:** shared `ownership_contributors` rows are never mutated while a `building_version` is incomplete — the dashboard RPCs join contributor metadata unversioned, so any mid-build metadata write @@ -760,17 +855,24 @@ relaxation of §5's prohibited list. - Both work relations → **EPHEMERAL** in `backup-scope.mjs` (classification is mandatory — an unclassified table fails CI). - Verified against the revision-3 state model: losing them never loses - active user data — the recovery cost is **"discard the incomplete - building version and rebuild it,"** not "resume the current scope." - Dumping them would persist pseudonymous state for zero benefit; the - EPHEMERAL classification stands. + `ownership_scope_versions` → **CORE** — it is required to interpret + active ownership state (it carries which scopes belong to + `active_version` and their authoritative exclusion state), so it must + be dumped and restored exactly like `ownership_snapshots`. Verified + against the revision-5 state model: losing the EPHEMERAL relations + never loses active user data — the recovery cost is **"discard the + incomplete building version and rebuild it,"** not "resume the current + scope." Dumping them would persist pseudonymous state for zero benefit; + the EPHEMERAL classification stands. - **Restore semantics — abandon, never piecemeal-resume:** restore - returns `ownership_sync_state` (OPERATIONAL) but no work rows. If the + returns `ownership_sync_state` (OPERATIONAL) plus all CORE ownership + rows — including V's `ownership_scope_versions` membership and V's + snapshots — but no work rows. If the restored row has `building_version = V`, the required build state for V is by definition absent/incomplete → V is unrecoverable → run - `own_abandon_build`: delete V snapshots, delete any surviving V work - rows, prune unreferenced V-only contributors, clear V build state. Then + `own_abandon_build`: delete V snapshots **and V scope membership**, + delete any surviving V work rows, prune unreferenced V-only + contributors and V-only base scopes, clear V build state. Then a later ownership run begins a NEW complete build — re-enumerate scopes, obtain a fresh pinned revision, accumulate from the start; **no previously committed V scope is trusted.** @@ -834,6 +936,27 @@ endpoint form, now sha-pinned. `013_ownership_checkpointing.sql` — additive only: +**RETAINED (CORE):** + +- `create table public.ownership_scope_versions` — §5.9: composite + tenant/scope FKs `(scope_id, repository_id) → ownership_scopes` and + `(repository_id, user_id) → repositories`, all `on delete cascade`; + unique `(scope_id, calculation_version)`; `excluded boolean not null + default false`; CHECK `excluded → exclusion_reason = 'cardinality_cap'` + and `not excluded → exclusion_reason is null` (the spec's + `ownership_scopes_exclusion` contract, versioned); CHECK + `calculation_version > 0`. RLS enabled, zero + policies, `service_role` DML only — the retained-table security model, + **not** the ephemeral one: it is CORE backup state. +- `ownership_scopes.excluded`/`exclusion_reason` become + **non-authoritative legacy**: new base rows always write + `excluded=false, exclusion_reason=null`; authoritative per-version + state lives in `ownership_scope_versions`. The columns may be dropped + only by a later explicitly justified migration — 012 is never edited + retroactively. + +**EPHEMERAL:** + - `create table public.ownership_history_work` — §4.4 columns; composite FKs `(repository_id, user_id) → repositories` and `(scope_id, repository_id) → ownership_scopes(id, repository_id)`, all @@ -857,21 +980,36 @@ endpoint form, now sha-pinned. - `own_abandon_build(p_user, p_repository, p_version)` — **the canonical build-abandonment primitive** (§5.6): locks the sync-state row, validates the expected `building_version`, deletes the version's - snapshots and both ephemeral relations, anti-join prunes unreferenced - contributors, clears build state, preserves `active_version`. There is - ONE semantic abandon operation — the stale sweep, restore - reconciliation, operator abort, and invalid-state recovery all call it. + snapshots **and `ownership_scope_versions` membership**, deletes both + ephemeral relations, anti-join prunes unreferenced contributors **and + unreferenced base scopes** (no membership row AND no snapshot), clears + build state, preserves `active_version`. There is ONE semantic abandon + operation — the stale sweep, restore reconciliation, operator abort, + and invalid-state recovery all call it. - `own_sweep_stale_work(...)` — repository-build-aware: bare-deletes orphan work rows only when `building_version` is null; for a genuinely stale live build (no lease, no activity in 14 days) it calls `own_abandon_build` — same security shape. -- `backup-scope.mjs`: append both tables to `EPHEMERAL_TABLES` (same PR — +- `create or replace function dash_ownership_scopes(uuid, uuid)` — + **amended for version isolation**: joins `ownership_scope_versions` + where `calculation_version = st.active_version` and takes + `excluded`/`exclusion_reason` from membership; returns only + active-version scopes. `dash_ownership_head`/`dash_ownership_timeline` + reach scopes only through version-filtered snapshot rows — audit all + three at implementation; signature stays `security invoker` with the + same tenant checks. +- `backup-scope.mjs`: append `ownership_history_work` and + `ownership_contributor_build_work` to `EPHEMERAL_TABLES`; + `ownership_scope_versions` goes to the CORE ownership set (same PR — CI requires classification). -- Recovery-drill addition: assert the tables exist post-migration and are - absent from dumps. -- No changes to the four existing tables: `phase` enum already contains - `history`; resolve is cursor-jsonb state, not a new phase; the error - taxonomy already covers every failure mode in §13. +- Recovery-drill addition: assert all three tables exist post-migration; + the two work relations are absent from dumps; `ownership_scope_versions` + **is present** in dumps and restores. +- No schema changes to the four existing tables: `phase` enum already + contains `history`; resolve is cursor-jsonb state, not a new phase; the + error taxonomy already covers every failure mode in §13. The only + existing-object change is `create or replace` on `dash_ownership_scopes` + — same signature, same grants, version-correct semantics. --- @@ -938,6 +1076,26 @@ endpoint form, now sha-pinned. - contributor represented across several N+1 scopes: build-level staged bounds aggregate all represented scopes correctly while the shared row stays unchanged until promotion. +- **Scope version isolation (§5.9):** + - N active, N+1 enumerates a brand-new directory scope: base scope row + and N+1 membership may exist, but active-N `dash_ownership_scopes` + output is byte-identical — the new scope is invisible until + promotion; + - N+1 changes the cardinality-cap selection (a dir excluded in N is + included in N+1 or vice versa): active-N `excluded`/ + `exclusion_reason` values unchanged, N+1 state isolated in its own + membership rows; + - N+1 abort: N membership and output byte-identical, N+1 membership + deleted, N+1-only unreferenced base scope identities pruned, + N-referenced scopes preserved; + - N+1 promotion: the active scope set switches atomically with the + `active_version` flip — no observable mixed N/N+1 scope set; + - post-promotion old-version GC: N membership deleted at the retention + point, then base scopes referenced by nothing pruned — active + scopes/snapshots never cascade away; + - a delete attempt on a base scope still referenced by active-version + membership or snapshots is rejected/blocked by the implementation + contract — never a cleanup path. --- @@ -981,6 +1139,10 @@ endpoint form, now sha-pinned. | Promotion commits, response lost | V promoted atomically | caller reload sees active_version=V | flip+metadata in one tx | — | staging already deleted by that tx | | Cross-scope bot drift mid-build | staged frozen is_bot ≠ later scope's pass-1 value | fail closed → fresh build boundary | — | — | abort path cleans build state | | Shared contributor metadata mutate attempt during build | **none — design forbids it** | n/a | retained rows are read-only during build | — | nothing to roll back (§5.8) | +| N+1 inserts new base scope + membership | N+1-only membership + base identity | n/a | dashboard joins only active-version membership — new scope invisible in N output | — | abort/GC prunes unreferenced base scope (§5.9) | +| N+1 changes cardinality-cap selection | per-version exclusion state on N+1 membership | n/a | N's `excluded`/`exclusion_reason` on its membership rows unchanged | — | shared base row never mutated; abort removes N+1 membership | +| Delete of base scope referenced by active data | forbidden by contract — snapshot FK cascade would destroy active rows | n/a | — | — | prune only deletes scopes referenced by NO membership AND NO snapshot (§5.6) | +| Restore: V membership + V snapshots restored, work absent | sync claims building V; CORE rows present, EPHEMERAL absent | never piecemeal-resume | — | — | `own_abandon_build` deletes restored V membership + snapshots too | --- diff --git a/docs/ownership-data-privacy-spec.md b/docs/ownership-data-privacy-spec.md index e353e43..54dc04e 100644 --- a/docs/ownership-data-privacy-spec.md +++ b/docs/ownership-data-privacy-spec.md @@ -1227,8 +1227,9 @@ Terminology for this amendment is exact: **EPHEMERAL** means short-lived never browser/API-exposed, never logged or telemetered, deleted at scope finalization or on build-abort cleanup, and fully reconstructable by restarting the build. **RETAINED** means durable, backed-up, user-visible -ownership data (`ownership_scopes`, `ownership_contributors`, -`ownership_snapshots` at `active_version`). Persisted ≠ retained. +ownership data (`ownership_scopes`, `ownership_scope_versions`, +`ownership_contributors`, `ownership_snapshots` at `active_version`). +Persisted ≠ retained. The proposed amendment is deliberately narrow: @@ -1247,12 +1248,18 @@ The proposed amendment is deliberately narrow: not anonymous. It may appear in persisted build state only under the EPHEMERAL conditions above — service-role-only, backup-excluded, never logged, cascade-deleted, swept on abandoned builds. -2. **The §6.4 guarantee is preserved verbatim** — overflow-only - contributors still get **no retained identity row** and their GitHub - identity fields (`github_node_id`, `display_login`) are **never - persisted in any table**. Two further guarantees are required because - `ownership_contributors` is shared across calculation versions while - the dashboard RPCs filter only snapshots by `active_version`: +2. **The §6.4 guarantee is preserved in substance** — the live §6.4 text + remains unchanged until this amendment is adopted; it is not "verbatim" + because the amendment deliberately persists the pseudonymous + `work_key` in EPHEMERAL build state. What is guaranteed exactly as + before: overflow-only contributors still get **no retained identity + row** and their GitHub identity fields (`github_node_id`, + `display_login`) are **never persisted in any table** — only their + pseudonymous `work_key` may exist, and only in short-lived EPHEMERAL + relations under this amendment's conditions. Two further guarantees + are required because `ownership_contributors` is shared across + calculation versions while the dashboard RPCs filter only snapshots by + `active_version`: **(a) shared contributor rows are never mutated while a `building_version` is incomplete** — mid-build metadata writes would leak into active-version responses; intended building-version metadata @@ -1267,6 +1274,24 @@ The proposed amendment is deliberately narrow: referenced contributors always survive (and were never mutated, so nothing needs restoring); this pruning is part of identity minimization, not housekeeping. + **(c) scope version isolation** — `ownership_scopes` is shared/ + unversioned the same way, and the deployed `dash_ownership_scopes` + enumerates base scope rows regardless of version. The amendment splits + immutable scope identity (`id`, path hash, display path, type, depth) + from **versioned membership**: a retained CORE relation + `ownership_scope_versions` at grain `(scope_id, calculation_version)` + carrying the per-version `excluded`/`exclusion_reason` cardinality + state (a directory may be inside the ≤500 set in N and outside in N+1 + as the tree changes; `excluded=true` remains permitted only with + `'cardinality_cap'`). Building versions may insert new base scope + identities and their own membership rows, but the scopes RPC joins + membership at `active_version` — a new N+1 scope is invisible in N + output and N exclusion state cannot move. The base row's exclusion + columns become non-authoritative legacy; a base scope referenced by + any snapshot or membership row is never deleted (the snapshot FK is + `ON DELETE CASCADE` — deletion would destroy active data). Abandoned + and garbage-collected versions delete their membership rows and then + prune base scopes referenced by nothing. 3. **Two-pass identity resolution, attribution-frozen.** The accumulate pass fixes every metric — work keys, counts, Top-25, unlinked share, human counts, **and `is_bot` classification** (`human_contributor_count` @@ -1302,10 +1327,11 @@ The proposed amendment is deliberately narrow: across already-finalized scopes, an incomplete build whose ephemeral state is lost, stale, corrupt, or absent after restore is **wholly discarded** through one canonical abandon operation (delete its - versioned snapshots and ephemeral rows, prune its unreferenced - contributors, preserve `active_version`) — piecemeal resumption, - including restarting only the current scope, is prohibited because the - discarded metadata is not reconstructable from snapshots. + versioned snapshots **and scope membership** and ephemeral rows, prune + its unreferenced contributors **and unreferenced base scopes**, + preserve `active_version`) — piecemeal resumption, including + restarting only the current scope, is prohibited because the discarded + metadata is not reconstructable from snapshots. If this amendment is rejected, the honest alternatives are: (a) accept per-scope restart-in-single-invocation with no completion guarantee on From 474c8605a611786e175e1a7b81bafd09a00639a3 Mon Sep 17 00:00:00 2001 From: aliferous3 Date: Sun, 18 Oct 2026 23:22:47 +0400 Subject: [PATCH 7/9] docs: DB-enforce snapshot membership and sharpen scope caps A snapshot must be impossible without version membership: prospective 013 adds a composite (scope, repo, tenant, version) FK from ownership_snapshots to ownership_scope_versions with NO ACTION/RESTRICT semantics so cleanup can never cascade-destroy snapshots. Also separates the hard 500-directory enumeration cap (overflow paths never persisted) from the cardinality_cap analysis-omission marker, and states per-version bounds honestly. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --- docs/ownership-checkpoint-architecture.md | 204 ++++++++++++++++++---- docs/ownership-data-privacy-spec.md | 38 +++- 2 files changed, 210 insertions(+), 32 deletions(-) diff --git a/docs/ownership-checkpoint-architecture.md b/docs/ownership-checkpoint-architecture.md index 68f35be..7a8ca78 100644 --- a/docs/ownership-checkpoint-architecture.md +++ b/docs/ownership-checkpoint-architecture.md @@ -1,7 +1,14 @@ # Ownership Checkpoint Architecture — Phase 1C-B0 Design -> **Status: design only, revision 5. Nothing in this document is implemented -> or live.** Revision 4→5 correction — **scope version isolation**: +> **Status: design only, revision 6. Nothing in this document is implemented +> or live.** Revision 5→6 hardening — **membership integrity + cardinality +> precision**: `ownership_snapshots` gains a DB-enforced composite FK to +> `ownership_scope_versions` (same scope/repo/tenant/version; `NO ACTION`, +> so membership deletion can never destroy snapshots); the 500-directory +> hard enumeration cap is distinguished from the `cardinality_cap` +> analysis-omission marker; overflow directories are never persisted at all; +> bounds are stated per version (~≤1000 transient base identities during +> N/N+1 coexistence). Revision 4→5 — **scope version isolation**: > `ownership_scopes` is shared/unversioned exactly like > `ownership_contributors` — `dash_ownership_scopes` enumerates base scope > rows regardless of version, so a building version's new scopes or @@ -47,10 +54,13 @@ or hundreds of pages, and a serverless invocation holds memory only for **Finding: there is no architecture that satisfies exactness, crash safety, and identity minimization using only the four existing tables.** The minimum -amendment is ephemeral, backup-excluded working state — counters plus a -narrow pass-2 identity-staging relation — plus one atomic checkpoint -primitive. §5 below; the required contract amendment is recorded in §8 and -in the spec's proposed-amendment appendix. +amendment is: two ephemeral, backup-excluded work relations (counters plus +build-level identity staging); one retained versioned scope-membership +relation (`ownership_scope_versions`, CORE) with a **DB-enforced composite +FK from `ownership_snapshots`**; the canonical checkpoint/abandon/sweep +primitives; an atomic contributor-metadata promotion; and a version-correct +`dash_ownership_scopes`. §5 below; the required contract amendment is +recorded in §8 and in the spec's proposed-amendment appendix. --- @@ -67,10 +77,12 @@ Checked against `012_ownership.sql` as deployed: Conclusion: **migration 013 is required.** The delta (§11): one retained versioned scope-membership table (`ownership_scope_versions`, CORE), two -ephemeral work relations, the checkpoint/abandon/sweep primitives, and an -amended `dash_ownership_scopes`. No changes to migration 012 itself — the -base scope/contributor columns become non-authoritative or versioned -rather than retroactively edited. +ephemeral work relations, a **composite FK added to the existing +`ownership_snapshots` table** enforcing snapshot→membership integrity, the +checkpoint/abandon/sweep primitives, and an amended +`dash_ownership_scopes`. Migration 012's file is never edited — 013 alters +the *deployed* `ownership_snapshots` by adding the membership FK, and the +base scope exclusion columns become non-authoritative. --- @@ -256,9 +268,10 @@ Hard rules for the staging relation: - **it is the only place GitHub identity fields may exist outside retained tables — and only for already-represented contributors.** -**Migration 013 therefore needs two work relations.** One table would be -smaller only by conflating two data classes with different sensitivity; -the split keeps the counter relation identity-free *by shape*. +**Migration 013 therefore needs two EPHEMERAL work relations** plus the +retained `ownership_scope_versions` (§5.9). One table would be smaller +only by conflating two data classes with different sensitivity; the split +keeps the counter relation identity-free *by shape*. --- @@ -496,6 +509,13 @@ there is no second cleanup semantics anywhere): Existing contributor metadata and N scope membership need no restoration — they were never mutated. +The ordering is **required, not stylistic**: the prospective +`snapshots → scope_versions` FK is `NO ACTION`/`RESTRICT` (§5.9), so any +cleanup that tried membership-first would *fail*, not cascade — and that +is the point: an incorrect cleanup must be incapable of destroying +active/retained data rather than merely unlikely to. No operation ever +deletes a membership row while snapshots still reference it. + ### 5.6a The ephemeral-loss invariant (CANONICAL) `ownership_contributor_build_work` accumulates metadata across @@ -540,6 +560,14 @@ orphaned V snapshots without their staging. Missing or corrupt V membership is itself a fail-closed reason. If any check fails — never reconstruct heuristically → `own_abandon_build`. +Role boundary: §5.6b validates *resumability* — cursor/pending/phase +coherence, ephemeral-state completeness, staging completeness. It is +**not** the integrity mechanism for snapshot↔membership: the +`snapshots → ownership_scope_versions` composite FK (§5.9) is primary and +makes a V snapshot without V membership impossible even if validation +has a bug. Application checks decide "can this build resume?"; the FK +decides "can this row exist?" + ### 5.7 Promotion — atomic metadata application + version flip When `pending_scopes` is empty, ONE transaction under the sync-row lock: @@ -567,11 +595,13 @@ the new metadata. No intermediate mixed state is observable — readers under read-committed see either fully-pre or fully-post commit. A failed promotion leaves active N and all shared contributor metadata unchanged and building V retryable. Post-flip garbage collection of old-version -snapshots is a separate later transaction (spec §13): it deletes the old -version's `ownership_scope_versions` membership, then prunes -contributors and base scopes left unreferenced by that GC the same way -§5.6 does — membership first, identity prune second, never cascading -away referenced/active rows. +snapshots is a separate later transaction (spec §13), ordered to match +the RESTRICT FK: verify the collected version is not `active_version` → +delete old-version **snapshots** → delete old-version +`ownership_scope_versions` membership → prune contributors referenced by +no remaining snapshot → prune base scopes referenced by no membership +row and no snapshot. Membership is never deleted before its snapshots, +and no delete ever relies on cascading membership→snapshot. ### 5.8 Why this is required — the shared-metadata leak @@ -643,6 +673,61 @@ marker for that version only. The base row's `excluded`/ if a later migration explicitly justifies it — 012 is never edited retroactively). +**Exact membership semantics:** + +- **NO membership row for `(scope, V)`** — "this scope does not belong to + calculation version V." Not persisted, not served, not analyzable for V. +- **`excluded=false`** — "this scope belongs to V and is analyzed." +- **`excluded=true, exclusion_reason='cardinality_cap'`** — "this scope + belongs to the *bounded persisted set* for V, but analysis was omitted + because the approved repository snapshot/storage cardinality budget + would be exceeded." This is the **only** permitted exclusion reason. +- `excluded=true` is **never** a generic "the repository contained this + path" inventory marker. + +**The two caps are distinct — do not conflate them:** + +- **Hard directory-enumeration cap (500/version):** + `deriveScopeCandidates().scopes` yields the bounded selected directory + set (repo scope is separate). `deriveScopeCandidates().overflow` — + eligible directories beyond the hard cap — is **never persisted + individually**: no `ownership_scopes` base row, no + `ownership_scope_versions` row, no `display_path` in durable storage, + logs, or telemetry. Persisting overflow as `excluded=true` rows would + make the private-path inventory unbounded and defeat the cap's purpose. +- **Analysis cardinality cap (the legitimate `cardinality_cap` use):** an + *already-selected, inside-the-500* scope whose projected snapshot volume + would breach the repo snapshot budget may be marked + `excluded='cardinality_cap'` — membership stays; analysis is skipped. + +**Bounds, stated honestly:** ≤500 **directory memberships per calculation +version** (repo scope separate). Because active N and building N+1 +coexist, `ownership_scopes` may temporarily hold the *union* of two +bounded sets — worst case ≈≤1000 base directory identities during a +rebuild with zero overlap; normal overlap is far lower. After abort or +old-version GC, identities referenced by neither retained membership nor +snapshots are pruned — **no unbounded accumulation of historical +directory identities.** The lifecycle remains: at most `active + one +building version`; old-version GC runs after promotion. + +**DB-enforced snapshot→membership integrity (hard invariant):** +application validation alone must not be the only thing preventing a +`scope_id/calculation_version` snapshot with no matching membership. +Prospective 013 provides a supporting unique key on +`ownership_scope_versions` over the full tenant boundary — +`(scope_id, repository_id, user_id, calculation_version)` — and `ALTER +ownership_snapshots` adds a composite FK on the same four columns +referencing it, with **`ON DELETE NO ACTION`/`RESTRICT`** (not CASCADE): +deleting a membership row must never be a path that destroys snapshots — +cleanup order is always snapshots first, membership second (§5.6, §5.7). +The composite columns make the DB itself prove same-scope + same-repo + +same-tenant + same-version; a V snapshot without V membership is +**impossible**. Repository/account deletion still cascades completely — +the cascade arrives via the other FKs (`snapshots→repositories`, +`membership→repositories`), so ordering the *membership* FK as RESTRICT +adds no teardown hazard; this is verified against real Postgres during +implementation. + **Build flow:** at N+1 init — pin revision → enumerate eligible scopes → insert missing base identity rows → insert N+1 membership rows → derive `pending_scopes` from N+1 membership where `excluded=false` → @@ -743,6 +828,8 @@ COMMIT deletes them before the next scope starts. | Bound | Limit | Rationale | |---|---|---| +| Directory memberships per version | 500 | hard enumeration cap from `deriveScopeCandidates`; overflow paths get no base row, no membership row, no persisted `display_path` (§5.9) | +| Base scope identities per repo | ≈≤1000 transient | union of active N + building N+1 bounded sets; pruned on abort/GC — never unbounded accumulation | | Distinct `work_key` per scope | 10,000 | ~5× facebook/react's repo-wide contributor count for a *single* scope | | Accumulation rows per scope | 100,000 | keys × active months; ~10 MB transient worst case | | Build-work rows per build | ≤ represented keys, ≤10,000 | one row per represented key across the whole build — the narrowest possible set | @@ -939,11 +1026,16 @@ endpoint form, now sha-pinned. **RETAINED (CORE):** - `create table public.ownership_scope_versions` — §5.9: composite - tenant/scope FKs `(scope_id, repository_id) → ownership_scopes` and - `(repository_id, user_id) → repositories`, all `on delete cascade`; - unique `(scope_id, calculation_version)`; `excluded boolean not null - default false`; CHECK `excluded → exclusion_reason = 'cardinality_cap'` - and `not excluded → exclusion_reason is null` (the spec's + tenant/scope FKs `(repository_id, user_id) → repositories` and + `(scope_id, repository_id) → ownership_scopes(id, repository_id)`, all + `on delete cascade` (repository/account teardown unchanged); unique + `(scope_id, calculation_version)` for membership uniqueness **plus** the + supporting unique `(scope_id, repository_id, user_id, + calculation_version)` that the snapshot FK targets — the four-column + key is what lets the FK *prove* same-tenant/same-repo/same-version, not + merely same-scope; `excluded boolean not null default false`; CHECK + `excluded → exclusion_reason = 'cardinality_cap'` and + `not excluded → exclusion_reason is null` (the spec's `ownership_scopes_exclusion` contract, versioned); CHECK `calculation_version > 0`. RLS enabled, zero policies, `service_role` DML only — the retained-table security model, @@ -955,6 +1047,21 @@ endpoint form, now sha-pinned. only by a later explicitly justified migration — 012 is never edited retroactively. +**EXISTING TABLE CHANGE:** + +- `alter table public.ownership_snapshots add foreign key + (scope_id, repository_id, user_id, calculation_version) references + ownership_scope_versions (scope_id, repository_id, user_id, + calculation_version) on delete no action` — the DB-enforced + membership invariant (§5.9): a snapshot can exist only for a scope + that belongs to the snapshot's own calculation version, repository, + and tenant. `NO ACTION`/`RESTRICT` — **not** CASCADE — so deleting a + membership row can never destroy snapshots; abort/GC ordering is + snapshots first, membership second (§5.6, §5.7). This is the one + change to a 012-deployed table; migration 012's file itself is never + touched. The exact FK action is verified against real Postgres in the + implementation tranche. + **EPHEMERAL:** - `create table public.ownership_history_work` — §4.4 columns; composite @@ -1002,14 +1109,19 @@ endpoint form, now sha-pinned. `ownership_contributor_build_work` to `EPHEMERAL_TABLES`; `ownership_scope_versions` goes to the CORE ownership set (same PR — CI requires classification). -- Recovery-drill addition: assert all three tables exist post-migration; - the two work relations are absent from dumps; `ownership_scope_versions` - **is present** in dumps and restores. -- No schema changes to the four existing tables: `phase` enum already - contains `history`; resolve is cursor-jsonb state, not a new phase; the - error taxonomy already covers every failure mode in §13. The only - existing-object change is `create or replace` on `dash_ownership_scopes` - — same signature, same grants, version-correct semantics. +- Recovery-drill addition: assert all three new tables exist + post-migration; the two work relations are absent from dumps; + `ownership_scope_versions` **is present** in dumps and restores; the + snapshot→membership FK is enforced (a snapshot without membership is + rejected); the legal abort/GC delete ordering works (snapshots, then + membership); repository and account deletion still cascade the complete + ownership graph. +- **One existing-table change**: the `ownership_snapshots` composite FK + above. Otherwise `phase` enum already contains `history`; resolve is + cursor-jsonb state, not a new phase; the error taxonomy already covers + every failure mode in §13; and the only existing-object change is + `create or replace` on `dash_ownership_scopes` — same signature, same + grants, version-correct semantics. --- @@ -1096,6 +1208,32 @@ endpoint form, now sha-pinned. - a delete attempt on a base scope still referenced by active-version membership or snapshots is rejected/blocked by the implementation contract — never a cleanup path. +- **DB-integrity tests — real Postgres (drill/migration tests, not + mocks):** + - membership S/V exists → snapshot S/V succeeds; + - snapshot S/V with no membership S/V → **rejected**; + - membership S/V1 only → snapshot S/V2 **rejected** (wrong version); + - membership from repo A → snapshot claiming repo B **rejected**; + - membership under user A → snapshot under user B **rejected** + (cross-tenant); + - delete membership while a snapshot references it → **rejected** by + the RESTRICT/NO ACTION FK; + - delete V snapshots first, then V membership → succeeds; + - repository delete → full ownership graph cascades; account delete → + same. +- **Cardinality tests:** + - ≤500 selected directory candidates → ≤500 V membership rows (repo + scope separate); + - 501+ eligible dirs → derivation overflow non-empty; overflow paths + get NO base row, NO membership row, and their `display_path` never + reaches DB/logs/telemetry; + - a selected in-cap scope dropped on projected snapshot-volume → + membership `excluded=true, 'cardinality_cap'` — and still counts + inside the ≤500 membership set; + - active N (≤500) + building N+1 (≤500) → base-scope table bounded by + the union, no unbounded accumulation; + - abort N+1 → N+1-only base identities pruned; promote N+1 + GC N → + N-only base identities pruned. --- @@ -1143,6 +1281,9 @@ endpoint form, now sha-pinned. | N+1 changes cardinality-cap selection | per-version exclusion state on N+1 membership | n/a | N's `excluded`/`exclusion_reason` on its membership rows unchanged | — | shared base row never mutated; abort removes N+1 membership | | Delete of base scope referenced by active data | forbidden by contract — snapshot FK cascade would destroy active rows | n/a | — | — | prune only deletes scopes referenced by NO membership AND NO snapshot (§5.6) | | Restore: V membership + V snapshots restored, work absent | sync claims building V; CORE rows present, EPHEMERAL absent | never piecemeal-resume | — | — | `own_abandon_build` deletes restored V membership + snapshots too | +| Snapshot insert without V membership | impossible — composite FK `(scope, repo, user, version)` → `ownership_scope_versions` rejects it | n/a | DB-enforced, not app-only | — | row cannot exist | +| Delete membership while snapshot references it | rejected — FK is NO ACTION/RESTRICT | n/a | incorrect cleanup cannot destroy snapshots | — | ordering enforced by the constraint | +| Hard-cap overflow directory path | no base row, no membership row | n/a | `display_path` never persisted | — | nothing to clean — never written | --- @@ -1188,6 +1329,7 @@ endpoint form, now sha-pinned. | H. Overflow identity fields unpersisted? | **YES** | §4.4 — staging accepts represented keys only; durable contributor rows exist only via COMMIT | | I. Every page transition exactly-once? | **YES** | §6.2 — single-transaction CAS, proven | | J. Work state backup-excludable safely? | **YES** | §9/§5.6a — loss = discarded *building* version (canonical abandon + fresh rebuild), never active user data | +| K. Scope membership versioned + DB-enforced? | **YES** | §5.9 — retained `ownership_scope_versions` (CORE) + composite snapshot→membership FK (NO ACTION); `dash_ownership_scopes` joins membership at `active_version` | ## 17. Readiness verdict diff --git a/docs/ownership-data-privacy-spec.md b/docs/ownership-data-privacy-spec.md index 54dc04e..1cc89ae 100644 --- a/docs/ownership-data-privacy-spec.md +++ b/docs/ownership-data-privacy-spec.md @@ -609,7 +609,10 @@ PK `id`. Composite FK `(repository_id, user_id) → repositories(id, user_id)` cascade. Unique `(repository_id, scope_path_hash)`. Index `(repository_id, depth)` for bounded enumeration. Columns per §4.1. Write: ownership ingestion. Read: dashboard RPCs join through it for labels. -Cardinality: ≤500/repo; ~50–200 typical. +Cardinality: ≤500/repo; ~50–200 typical. *(Under proposed §27 the bound +is stated per version: ≤500 **directory memberships** per +`calculation_version`; the base table may transiently hold the union of +active+building sets — ≈≤1000 worst case, pruned after abort/GC.)* ### `ownership_contributors` PK `id`. Composite FK `(repository_id, user_id) → repositories(id, user_id)` @@ -690,6 +693,17 @@ cap and degrades to shallower scope coverage instead of 1.5M rows. Postgres handles 300k indexed rows trivially — the cap exists for hygiene and per-tenant fairness, not database survival. +*Under proposed §27 the scope bounds decompose precisely — they are four +different counts, not one "≤500 scopes/repo":* **base scope identities** +(≈≤ the union of the active and building directory sets; ≈≤1000 transient +during N/N+1 coexistence); **version memberships** (≤500 directories per +`calculation_version`, repo scope separate — directories beyond the hard +cap are never persisted at all, so membership is never an overflow +inventory); **analyzed memberships** (the `excluded=false` subset — +`excluded='cardinality_cap'` marks only in-set scopes omitted on projected +snapshot volume); **snapshots** (sparse writes, Top-25+Other fan-out, +~300k/repo cap as above). + --- ## 12. Historical retention — **decision: full monthly history, no compaction in V1** @@ -1292,6 +1306,28 @@ The proposed amendment is deliberately narrow: `ON DELETE CASCADE` — deletion would destroy active data). Abandoned and garbage-collected versions delete their membership rows and then prune base scopes referenced by nothing. + **(d) DB-enforced membership integrity** — `ownership_snapshots` gains + a composite FK `(scope_id, repository_id, user_id, + calculation_version) → ownership_scope_versions`, backed by a + supporting unique on the same four columns, with `ON DELETE NO + ACTION`/`RESTRICT` (never CASCADE): a snapshot can exist only for a + scope belonging to its own version/repo/tenant, and deleting a + membership row can never destroy snapshots — cleanup order is always + snapshots first, membership second. Repository/account deletion still + cascades completely through the other FKs. + **(e) private-path bound is precise** — the ≤500 cap is a *hard + directory-enumeration cap per calculation version*: directories beyond + it (`deriveScopeCandidates().overflow`) are never persisted + individually — no base scope row, no membership row, no `display_path` + in storage/logs/telemetry. `excluded='cardinality_cap'` is reserved + for scopes *inside* the bounded persisted set that are omitted from + analysis only because the repo snapshot/storage budget would breach — + it is never a "repo contained this path" inventory marker, and + `ownership_scope_versions` never becomes a full repository path + inventory. While N and N+1 coexist, `ownership_scopes` may hold the + union of two bounded sets (≈≤1000 base identities worst case, + transient); old-only identities are pruned after abort/GC — no + unbounded accumulation. 3. **Two-pass identity resolution, attribution-frozen.** The accumulate pass fixes every metric — work keys, counts, Top-25, unlinked share, human counts, **and `is_bot` classification** (`human_contributor_count` From 2f7a3fe483025578c335eba4c10ba66cfc7941eb Mon Sep 17 00:00:00 2001 From: aliferous3 Date: Mon, 19 Oct 2026 09:43:14 +0400 Subject: [PATCH 8/9] docs: block base-scope cascade deletes and gate builds on retired GC MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Revision 7 state-integrity pass. The snapshot→scope-version FK protected membership, but 012's snapshots→scopes CASCADE and a cascading scope_versions→scopes FK would still let a direct base-scope delete destroy active membership and snapshots — prospective 013 sets both retained child→scope FKs to ON DELETE NO ACTION (repo/account teardown still cascades via the direct repository/user FKs; NO ACTION is named exactly because it checks after the statement's referential actions, unlike RESTRICT). Adds the retired-version lifecycle gate: at most one active plus one building version, fresh-build preflight GCs retired versions in legal order before initializing, making the ≤~1000 base identity bound real. Defines committed-V-provenance staging cleanup on scope restart (build-level staging has no scope provenance and the represented set can drift), and strengthens promotion to require the staged linked key set to exactly equal the represented V key set. Docs only — no migration, no production change. Generated with [Devin](https://devin.ai) Co-Authored-By: Devin <158243242+devin-ai-integration[bot]@users.noreply.github.com> --- docs/ownership-checkpoint-architecture.md | 502 +++++++++++++++++----- docs/ownership-data-privacy-spec.md | 67 ++- 2 files changed, 456 insertions(+), 113 deletions(-) diff --git a/docs/ownership-checkpoint-architecture.md b/docs/ownership-checkpoint-architecture.md index 7a8ca78..53b55cb 100644 --- a/docs/ownership-checkpoint-architecture.md +++ b/docs/ownership-checkpoint-architecture.md @@ -1,7 +1,18 @@ # Ownership Checkpoint Architecture — Phase 1C-B0 Design -> **Status: design only, revision 6. Nothing in this document is implemented -> or live.** Revision 5→6 hardening — **membership integrity + cardinality +> **Status: design only, revision 7. Nothing in this document is implemented +> or live.** Revision 6→7 hardening — **state integrity**: both retained +> child→base-scope FKs become `ON DELETE NO ACTION` in prospective 013 +> (the existing `ownership_snapshots→ownership_scopes` CASCADE and the +> new `ownership_scope_versions→ownership_scopes` FK), so a referenced +> base scope is undeletable while teardown still cascades through the +> direct repository/user FKs; the FK action is named exactly — `NO +> ACTION`, checked after the statement's referential actions — never +> "NO ACTION/RESTRICT"; a retired-version GC gate (§5.6c) makes a third +> coexisting version impossible and the ≈≤1000 base-identity bound real; +> scope-restart staging cleanup (§5.3) is defined by committed-V +> provenance; promotion requires an *exact* staged↔represented key set +> (§5.7). Revision 5→6 — **membership integrity + cardinality > precision**: `ownership_snapshots` gains a DB-enforced composite FK to > `ownership_scope_versions` (same scope/repo/tenant/version; `NO ACTION`, > so membership deletion can never destroy snapshots); the 500-directory @@ -78,11 +89,14 @@ Checked against `012_ownership.sql` as deployed: Conclusion: **migration 013 is required.** The delta (§11): one retained versioned scope-membership table (`ownership_scope_versions`, CORE), two ephemeral work relations, a **composite FK added to the existing -`ownership_snapshots` table** enforcing snapshot→membership integrity, the +`ownership_snapshots` table** enforcing snapshot→membership integrity, +**plus** the deployed `ownership_snapshots → ownership_scopes` FK changed +from CASCADE to `ON DELETE NO ACTION` (§5.9a), the checkpoint/abandon/sweep primitives, and an amended `dash_ownership_scopes`. Migration 012's file is never edited — 013 alters -the *deployed* `ownership_snapshots` by adding the membership FK, and the -base scope exclusion columns become non-authoritative. +the *deployed* `ownership_snapshots` by adding the membership FK and +changing the scope FK's delete action, and the base scope exclusion +columns become non-authoritative. --- @@ -359,7 +373,7 @@ COMMIT ONE transaction — the only place retained rows are written: 1. verify accumulation complete for the scope 2. verify a staged build-work row exists for EVERY represented linked key (missing → abort → restart - scope at fresh ACCUMULATE) + scope at fresh ACCUMULATE — restart cleanup below) 3. merge this scope's contribution into the build-work row: is_bot must equal the staged frozen value (drift → fail closed); first/last seen merge via least/greatest @@ -423,6 +437,42 @@ mapping. Semantics now: One contributor cannot be human in one scope and bot in another within a single calculation version. +**Scope-restart staging cleanup (required — staging has no scope +provenance).** `ownership_contributor_build_work` is deliberately +build-level — `(repository_id, calculation_version, work_key)` — so +RESOLVE writes represented-key identity into it **before** the current +scope's COMMIT. If that scope then fails and restarts from a fresh +ACCUMULATE, staged rows written by the failed attempt can survive. This +matters: `author.user` linkage is allowed to drift between walks, so the +fresh accumulate can produce a *different* represented set — a stale +`work_key` from the failed attempt may no longer belong to any finalized +result. "Delete staged rows on scope restart" cannot be the rule, because +the build-grain relation cannot distinguish current-scope rows by scope. +The safe cleanup is defined by **committed-V provenance, not scope +provenance.** Before restarting the current scope at a fresh ACCUMULATE, +in ONE transaction under the repository ownership lock: + +1. delete/reset `ownership_history_work` rows for the current scope; +2. inspect `ownership_contributor_build_work` for `building_version V`; +3. **preserve** staged keys whose `work_key` maps through + `ownership_contributors.contributor_key` to a contributor row + referenced by at least one **COMMITTED V snapshot** — i.e. keys + already represented by an earlier finalized scope of this build; +4. **delete** staged keys with no committed-V-snapshot reference — + including keys staged only by the uncommitted attempt for + contributors new to the repository (no `ownership_contributors` row + exists, hence no V snapshot) **and** existing active-N contributors + staged only by the failed current scope (they likewise have no V + snapshot for V — their retained rows stay; only the staging row is + removed); +5. reset `current.pass`/`current.cursor` to a fresh ACCUMULATE. + +The distinction is exact: a key represented by an earlier committed V +scope must survive (its staging is consumed by promotion); a key staged +only by the failed/uncommitted current scope must not (the fresh +accumulate may not represent it, and promotion must never apply stale +staging — §5.7's exact-set invariant is the second line of defense). + ### 5.4 first_seen_at / last_seen_at — corrected semantics Work rows keep min/max `committedDate` per (scope, key, month). At scope @@ -501,16 +551,19 @@ there is no second cleanup semantics anywhere): 8. **Prune** `ownership_scopes` base rows now referenced by NO `ownership_scope_versions` row AND NO `ownership_snapshots` row — V-only base identities are removed; N-referenced scopes always - survive. The snapshot FK is `ON DELETE CASCADE`, so this explicit - check is what makes the prune safe rather than destructive; a base - scope still referenced by active/retained data is never deleted. + survive. Prospective 013 sets **both** retained child→base-scope FKs + (`snapshots → scopes`, `scope_versions → scopes`) to `ON DELETE NO + ACTION`, so a base scope referenced by any retained child + **cannot be deleted at all** — the database rejects the delete + (§5.9); this explicit unreferenced check is what makes the prune + succeed rather than fail. 9. `active_version` is never touched; phase/error fall back to the state implied by whether an active version exists (closed taxonomy only). Existing contributor metadata and N scope membership need no restoration — they were never mutated. The ordering is **required, not stylistic**: the prospective -`snapshots → scope_versions` FK is `NO ACTION`/`RESTRICT` (§5.9), so any +`snapshots → scope_versions` FK is `ON DELETE NO ACTION` (§5.9), so any cleanup that tried membership-first would *fail*, not cascade — and that is the point: an incorrect cleanup must be incapable of destroying active/retained data rather than merely unlikely to. No operation ever @@ -534,7 +587,9 @@ them). Therefore: A new repository build may not start while `building_version` is set: it first **resumes** the existing build if §5.6b validity holds, else runs `own_abandon_build`. `building_version` must be null before a fresh build -initializes — no overlapping building versions, ever. +initializes — necessary but not sufficient: the §5.6c retired-version +gate must also pass, so no overlapping building versions and no third +retained version, ever. ### 5.6b Required ephemeral-state validity @@ -546,7 +601,7 @@ the mere presence/absence of counter rows: | ACCUMULATE mid-scope | `history_work` prefix rows may exist (or be empty pre-first-page); `contributor_build_work` for previously finalized scopes' represented keys | next pages | | RESOLVE mid-scope | complete `history_work` counters for the scope; `contributor_build_work` for prior scopes | identity rows not yet observed in pass 2 | | Between scopes | `contributor_build_work` for all represented keys of committed scopes; `pending_scopes` non-empty | `history_work` for the just-committed scope (consumed) | -| Ready to promote | `pending_scopes` empty; complete `contributor_build_work`; zero `history_work` | — | +| Ready to promote | `pending_scopes` empty; `contributor_build_work` exactly equal to the represented V key set (§5.7); zero `history_work` | — | Zero `history_work` rows is **not** evidence of loss — it is the legal state after each scope COMMIT and before promotion. Validation checks: @@ -568,20 +623,78 @@ makes a V snapshot without V membership impossible even if validation has a bug. Application checks decide "can this build resume?"; the FK decides "can this row exist?" -### 5.7 Promotion — atomic metadata application + version flip +### 5.6c Retired-version lifecycle gate (CANONICAL) + +The ≈≤1000 base-identity bound assumes only `active N + building N+1` +ever coexist — but post-promotion old-version GC currently runs in a +separate later transaction. Without a build-start gate, "promote N+1 → +old N awaits GC → start building N+2" leaves **N + N+1 + N+2** in +retained state and the bound is false. The lifecycle is therefore a +hard invariant, not housekeeping: + +> **A repository may contain at most one `active_version` and, +> optionally, one `building_version`. No *retired* calculation version — +> a version whose snapshots/membership remain but which is neither +> `active_version` nor `building_version` — may exist when a NEW +> building version is initialized.** + +After promotion, the old version may coexist transiently with the new +active version **only while the mandatory GC is pending**; while such +retired rows exist a new ownership build MUST NOT start. **No third +ownership version may coexist with a building version.** + +**Fresh-build preflight** — every fresh build starts with, inside ONE +transaction under the repository ownership lock: + +1. `SELECT FOR UPDATE` the `(user_id, repository_id)` sync-state row. +2. Verify `building_version` is null — a non-null value routes to + §5.6b resume-or-abandon instead; fresh init never proceeds past it. +3. Identify **every** `calculation_version` present in + `ownership_snapshots` or `ownership_scope_versions` that is not + `active_version` — the retired set (normally the previous version + awaiting post-promotion GC; empty in the common case). +4. Garbage-collect each retired version in the **legal order**: + a. `ownership_snapshots` rows for the retired version; + b. `ownership_scope_versions` membership rows for the retired version; + c. `ownership_contributors` rows referenced by zero snapshots across + all versions (anti-join); + d. `ownership_scopes` base rows referenced by zero membership rows + and zero snapshots (anti-join). +5. Verify **no retired-version rows remain** — re-check the retired set + is empty inside the same transaction. +6. Only then pin the new revision and initialize `building_version`. + +If preflight GC fails or cannot complete: **`active_version` keeps +serving normally and no new build starts.** The gate re-runs the same +GC on the next build-start attempt (and the scheduled sweep may run it +too); the GC is idempotent, so a partial earlier attempt is safe to +retry. This makes the §5.9 bound real rather than aspirational. When `pending_scopes` is empty, ONE transaction under the sync-row lock: 1. Verify: `building_version = V`; no `ownership_history_work` rows for V; - no unresolved `ownership_contributor_build_work` rows for V (every - represented key has a complete staged identity); snapshot cardinality - within §11 bounds. + snapshot cardinality within §11 bounds; and the **exact-set invariant** + — the staged linked key set must *equal* the represented set, in both + directions: + - every `ownership_contributor_build_work` row for V maps (via + `contributor_key = work_key`) to a contributor referenced by at + least one building-V snapshot — **no extra staged keys** (a stale + row from an abandoned scope attempt must have been removed by the + §5.3 restart cleanup; a surviving one fails this check closed); + - every linked contributor key represented in V snapshots has a + staged build-work row — **no missing keys**. + `set(staged work_key) == set(distinct contributor_key over V + snapshots)` — synthetic `unlinked`/`other` buckets are excluded from + both sides (they never enter contributor staging). Any asymmetry → + promotion fails closed; nothing is applied. 2. Apply staged contributor metadata to retained rows — `UPDATE ownership_contributors ... FROM ownership_contributor_build_work` setting `github_node_id` (consistency-checked: same `work_key` must carry the same node id — mismatch fails closed), `display_login`, `is_bot` (the frozen pass-1 value), `first_seen_at`, `last_seen_at`. - New contributors inserted at scope COMMITs already carry these values; + The update touches exactly the set validated in step 1 — stale + staging rows can never write shared retained metadata. New + contributors inserted at scope COMMITs already carry these values; the update is idempotent over them. 3. `active_version = V; building_version = null;` clear build cursor state; `phase = 'done'` — single row update. @@ -596,12 +709,15 @@ under read-committed see either fully-pre or fully-post commit. A failed promotion leaves active N and all shared contributor metadata unchanged and building V retryable. Post-flip garbage collection of old-version snapshots is a separate later transaction (spec §13), ordered to match -the RESTRICT FK: verify the collected version is not `active_version` → -delete old-version **snapshots** → delete old-version +the `NO ACTION` child FKs: verify the collected version is not +`active_version` → delete old-version **snapshots** → delete old-version `ownership_scope_versions` membership → prune contributors referenced by no remaining snapshot → prune base scopes referenced by no membership row and no snapshot. Membership is never deleted before its snapshots, -and no delete ever relies on cascading membership→snapshot. +and no delete ever relies on cascading membership→snapshot. Until this +GC completes the retired version is still present, so the §5.6c gate +blocks any fresh build start — retired state is transient by +construction, never a third coexisting version. ### 5.8 Why this is required — the shared-metadata leak @@ -640,20 +756,29 @@ Three distinct leaks follow: metrics); - **metadata leak:** mutating `excluded`/`exclusion_reason`/ `display_path`/`depth` on a shared scope alters active-N output; -- **deletion hazard:** `ownership_snapshots` FK-references scopes - `ON DELETE CASCADE` — deleting a base scope during a build could - cascade-destroy **active-N snapshots**. +- **deletion hazard:** under migration 012, `ownership_snapshots` + FK-references scopes `ON DELETE CASCADE` — deleting a base scope + during a build could cascade-destroy **active-N snapshots**. The new + snapshot→membership FK protects membership integrity, but does not by + itself close this path: an erroneous + `DELETE FROM ownership_scopes WHERE id = ` could still + remove active scope-version membership *and* active snapshots through + cascades. Prospective 013 therefore changes the retained child→scope + delete actions to `ON DELETE NO ACTION` (below), so the database + itself rejects any base-scope delete while retained/versioned + children reference it. Resolution — split immutable identity from versioned membership: **`ownership_scopes` stays the immutable repository/path identity relation** (`id, user_id, repository_id, scope_path_hash, display_path, scope_type, depth, created_at, updated_at`): never mutated merely because -another version is building; never deleted while any snapshot or -membership row references it (the FK cascade makes "delete a referenced -scope" a correctness violation, not a cleanup path). Vendor/generated/ -binary/submodule paths still never get a base row; file scopes remain -impossible. +another version is building; **undeletable while any snapshot or +membership row references it** — under prospective 013 both retained +child→base-scope FKs are `ON DELETE NO ACTION`, so "delete a referenced +scope" is rejected by the database, not merely forbidden by contract +(§5.9a). Vendor/generated/binary/submodule paths still never get a base +row; file scopes remain impossible. **`ownership_scope_versions`** — RETAINED, CORE-classified, grain `(scope_id, calculation_version)` plus `user_id`/`repository_id` for @@ -700,15 +825,33 @@ retroactively). would breach the repo snapshot budget may be marked `excluded='cardinality_cap'` — membership stays; analysis is skipped. -**Bounds, stated honestly:** ≤500 **directory memberships per calculation -version** (repo scope separate). Because active N and building N+1 -coexist, `ownership_scopes` may temporarily hold the *union* of two -bounded sets — worst case ≈≤1000 base directory identities during a -rebuild with zero overlap; normal overlap is far lower. After abort or -old-version GC, identities referenced by neither retained membership nor -snapshots are pruned — **no unbounded accumulation of historical -directory identities.** The lifecycle remains: at most `active + one -building version`; old-version GC runs after promotion. +**Bounds, stated honestly — defined per lifecycle phase** (the phases are +the only states in which base identities exist; the retired-version GC +gate in §5.6c is what makes them exhaustive): + +- **NORMAL IDLE** — only `active_version` membership is retained: ≤500 + directory memberships; `ownership_scopes` holds ≤500 base directory + identities plus the repo scope. +- **DURING BUILD** — active N (≤500) + building N+1 (≤500) coexist; + `ownership_scopes` holds the *union* of two bounded sets — worst case + ≈≤1000 base directory identities with zero overlap; normal overlap is + far lower. +- **IMMEDIATELY AFTER PROMOTION** — the previous (retired) version's + membership/snapshots may coexist transiently with the new active + version **only while the mandatory old-version GC is pending** — and + **no new build may start in this state** (§5.6c). This is a + GC-pending state, not a third lifecycle phase for serving. +- **AFTER GC** — identities referenced by neither retained membership + nor snapshots are pruned; `ownership_scopes` returns to the + active-only bound. + +≤500 **directory memberships per calculation version** (repo scope +separate); ≈≤1000 base identities is the durable worst case during a +rebuild — never exceeded, because **no third ownership version may +coexist with a building version** (§5.6c) and a retired version may +exist only in the GC-pending window after promotion. **No unbounded +accumulation of historical directory identities** — abort and GC prune +identities referenced by nothing retained. **DB-enforced snapshot→membership integrity (hard invariant):** application validation alone must not be the only thing preventing a @@ -717,23 +860,85 @@ Prospective 013 provides a supporting unique key on `ownership_scope_versions` over the full tenant boundary — `(scope_id, repository_id, user_id, calculation_version)` — and `ALTER ownership_snapshots` adds a composite FK on the same four columns -referencing it, with **`ON DELETE NO ACTION`/`RESTRICT`** (not CASCADE): -deleting a membership row must never be a path that destroys snapshots — -cleanup order is always snapshots first, membership second (§5.6, §5.7). -The composite columns make the DB itself prove same-scope + same-repo + -same-tenant + same-version; a V snapshot without V membership is -**impossible**. Repository/account deletion still cascades completely — -the cascade arrives via the other FKs (`snapshots→repositories`, -`membership→repositories`), so ordering the *membership* FK as RESTRICT -adds no teardown hazard; this is verified against real Postgres during -implementation. - -**Build flow:** at N+1 init — pin revision → enumerate eligible -scopes → insert missing base identity rows → insert N+1 membership rows -→ derive `pending_scopes` from N+1 membership where `excluded=false` → -walk. New base rows may exist pre-promotion **only because dashboard -reads will join membership** — without membership for `active_version` -they are invisible. Existing N membership is never touched. +referencing it, with **`ON DELETE NO ACTION`** (not CASCADE, not left +unspecified between NO ACTION and RESTRICT — see §5.9a for why the exact +action is named): deleting a membership row must never be a path that +destroys snapshots — cleanup order is always snapshots first, membership +second (§5.6, §5.7). The composite columns make the DB itself prove +same-scope + same-repo + same-tenant + same-version; a V snapshot +without V membership is **impossible**. Repository/account deletion +still cascades completely — the cascade arrives via the other FKs +(`snapshots→repositories`, `membership→repositories`), so the +`NO ACTION` membership FK adds no teardown hazard; this is verified +against real Postgres during implementation (§5.9a proof plan). + +### 5.9a Base-scope delete enforcement — `NO ACTION` on both retained child FKs + +The membership FK above protects snapshots from *membership* deletion, +but a second, older destructive path remains: `DELETE FROM +ownership_scopes` directly. Under 012, `ownership_snapshots → +ownership_scopes` is `ON DELETE CASCADE`, and a cascading +`ownership_scope_versions → ownership_scopes` FK would give an erroneous +direct scope delete the power to remove **active membership and active +snapshots in one statement** — contradicting the invariant that deleting +a referenced base scope is forbidden. Prospective 013 therefore sets: + +| Child → `ownership_scopes` FK | Action | +|---|---| +| `ownership_snapshots(scope_id, repository_id) → ownership_scopes(id, repository_id)` — **existing 012 constraint, altered by 013** | `ON DELETE NO ACTION` | +| `ownership_scope_versions(scope_id, repository_id) → ownership_scopes(id, repository_id)` — new in 013 | `ON DELETE NO ACTION` | + +**No retained child→base-scope relation uses CASCADE.** A base-scope +prune (§5.6 step 8, §5.7 GC, §5.6c preflight GC) succeeds **only after** +snapshots *and* scope-version memberships no longer reference it — +the legal order is snapshots → membership → then the base row, enforced +by the database rather than by convention. The ephemeral +`ownership_history_work → ownership_scopes` FK stays `CASCADE` (it is +not retained child state and plays no part in the protected graph); the +direct `(repository_id, user_id)` and `user_id` FKs on every ownership +table stay `CASCADE`, so repository/account teardown still removes the +entire graph in one statement. + +**Why `NO ACTION`, exactly — not "NO ACTION/RESTRICT":** repository or +account deletion is *one* statement whose referential actions remove +snapshots, memberships, and scopes through their direct +repository/user cascade FKs. `NO ACTION` checks referential validity +**after** the statement's referential actions have occurred — a parent +scope deleted in the same statement as its (already cascade-removed) +children passes. `RESTRICT` checks eagerly and can reject that same +teardown depending on the order the engine walks the constraint graph. +The architecture names **`ON DELETE NO ACTION`** as the exact intended +action for every retained ownership child FK; implementation changes it +only if the real-Postgres migration test proves otherwise. + +**Required real-Postgres proof plan (migration/drill tests, not +mocks):** + +- direct `DELETE FROM ownership_scopes` on a base scope referenced by + **active membership** → rejected; +- direct delete on a base scope referenced by **active snapshots** → + rejected; +- delete the version's snapshots, then its membership, then the base + scope → **succeeds** (the legal prune order); +- `DELETE FROM repositories` → the complete ownership graph (scopes, + scope versions, contributors, snapshots, sync state, both work + relations) cascades away in one statement; +- `DELETE FROM users` → same. + +The last two are the teardown contract §14 requires — they exist to +prove the `NO ACTION` choice does not degrade repository/account +deletion under the real engine, not as theory. + +**Build flow:** at N+1 init — **fresh-build preflight (§5.6c):** lock +the sync-state row → verify `building_version` is null → garbage-collect +every calculation version other than `active_version` in legal order → +verify no retired-version rows remain — **then** pin revision → +enumerate eligible scopes → insert missing base identity rows → insert +N+1 membership rows → derive `pending_scopes` from N+1 membership where +`excluded=false` → walk. New base rows may exist pre-promotion **only +because dashboard reads will join membership** — without membership for +`active_version` they are invisible. Existing N membership is never +touched. **RPC:** `dash_ownership_scopes` must be amended to join `ownership_scope_versions` on `calculation_version = st.active_version` @@ -745,12 +950,16 @@ explicit version-consistency audit at implementation. **Promotion:** membership needs no rewrite — the flip swaps which version's membership the RPC joins, atomically with the contributor -metadata application (§5.7). **GC:** when old version N is collected: -delete N's membership rows → then prune base scopes referenced by no -membership row *and* no snapshot — in that order, never cascading away -referenced/active data. **Abandon** (§5.6) additionally deletes V's -membership rows and prunes V-only base scopes that are then referenced -by nothing. +metadata application (§5.7). **GC:** when old version N is collected, +the order is fixed — delete N's **snapshots** → delete N's +**membership rows** → prune contributors referenced by no remaining +snapshot → prune base scopes referenced by no membership row *and* no +snapshot. `NO ACTION` on both retained child→base-scope FKs (§5.9a) +makes any other order *fail* rather than cascade away referenced/active +data. The GC may run after promotion, but a **fresh build may not start +until it completes** (§5.6c). **Abandon** (§5.6) additionally deletes +V's membership rows and prunes V-only base scopes that are then +referenced by nothing. --- @@ -829,7 +1038,7 @@ COMMIT deletes them before the next scope starts. | Bound | Limit | Rationale | |---|---|---| | Directory memberships per version | 500 | hard enumeration cap from `deriveScopeCandidates`; overflow paths get no base row, no membership row, no persisted `display_path` (§5.9) | -| Base scope identities per repo | ≈≤1000 transient | union of active N + building N+1 bounded sets; pruned on abort/GC — never unbounded accumulation | +| Base scope identities per repo | ≤500 idle; ≈≤1000 during build; transient while retired-version GC is pending | union of active N + building N+1 bounded sets; a third version can never coexist — the §5.6c gate blocks new builds until retired GC completes; pruned on abort/GC — never unbounded accumulation | | Distinct `work_key` per scope | 10,000 | ~5× facebook/react's repo-wide contributor count for a *single* scope | | Accumulation rows per scope | 100,000 | keys × active months; ~10 MB transient worst case | | Build-work rows per build | ≤ represented keys, ≤10,000 | one row per represented key across the whole build — the narrowest possible set | @@ -860,7 +1069,8 @@ silently partial; an over-cap scope is an error, not a folded result. by definition not stale — the threshold only fires on genuinely dead state. - **Sweep triggers:** (a) at every build start for that repo — validity - check §5.6b → resume or `own_abandon_build`; (b) a periodic sweep + check §5.6b → resume or `own_abandon_build`, plus the §5.6c + retired-version GC gate before any fresh init; (b) a periodic sweep function (`own_sweep_stale_work`, proposed in 013) invoked by whatever scheduler the implementation tranche wires — so cleanup eventually runs even if the repo never builds again; (c) unconditional FK cascade on @@ -905,7 +1115,15 @@ implementation tranche is approved: only `active_version` membership — a new N+1 scope is invisible and N exclusion state cannot move. The base row's exclusion columns become non-authoritative after 013; base scopes referenced by any snapshot or - membership row are never deleted. + membership row are **undeletable** — both retained child→scope FKs + become `ON DELETE NO ACTION` (§5.9a), and a no-longer-referenced base + scope is removed only by the explicit prune. + **Lifecycle bound (§5.6c):** at most one `active_version` plus one + `building_version` may exist; a retired version's rows may remain only + while post-promotion GC is pending, and no fresh build may initialize + until retired-version GC completes — a third coexisting version is + impossible, which is what makes the ≈≤1000 base-identity bound a real + bound. **Version isolation:** shared `ownership_contributors` rows are never mutated while a `building_version` is incomplete — the dashboard RPCs join contributor metadata unversioned, so any mid-build metadata write @@ -919,11 +1137,13 @@ implementation tranche is approved: counts; pass 2 verifies rather than re-derives it, and a mismatch — between passes *or across scopes of one build* — fails the build closed rather than silently reclassifying. - **Abort cleanup:** abandoned builds delete their versioned snapshots - and ephemeral state, then prune `ownership_contributors` rows - unreferenced by any snapshot in any version — retained-version - contributors always survive (§5.6); their metadata was never mutated, - so nothing needs restoration. + **Abort cleanup:** abandoned builds delete their versioned snapshots, + their `ownership_scope_versions` membership, and ephemeral state, then + prune `ownership_contributors` rows unreferenced by any snapshot in + any version and base scopes referenced by no membership row and no + snapshot — retained-version contributors and scopes always survive + (§5.6); contributor metadata was never mutated, so nothing needs + restoration. **Seen-at semantics:** retained `first_seen_at`/`last_seen_at` accumulate only across finalized scopes where the contributor was individually represented (§5.4) — no all-repository-touch claim. @@ -1026,9 +1246,11 @@ endpoint form, now sha-pinned. **RETAINED (CORE):** - `create table public.ownership_scope_versions` — §5.9: composite - tenant/scope FKs `(repository_id, user_id) → repositories` and - `(scope_id, repository_id) → ownership_scopes(id, repository_id)`, all - `on delete cascade` (repository/account teardown unchanged); unique + tenant FK `(repository_id, user_id) → repositories` `on delete + cascade` (repository/account teardown unchanged) **plus** scope FK + `(scope_id, repository_id) → ownership_scopes(id, repository_id)` `on + delete no action` (§5.9a — a base scope referenced by membership + cannot be deleted); unique `(scope_id, calculation_version)` for membership uniqueness **plus** the supporting unique `(scope_id, repository_id, user_id, calculation_version)` that the snapshot FK targets — the four-column @@ -1047,7 +1269,9 @@ endpoint form, now sha-pinned. only by a later explicitly justified migration — 012 is never edited retroactively. -**EXISTING TABLE CHANGE:** +**EXISTING TABLE CHANGES (both on 012-deployed `ownership_snapshots`; +migration 012's file itself is never touched — 013 alters the deployed +constraints):** - `alter table public.ownership_snapshots add foreign key (scope_id, repository_id, user_id, calculation_version) references @@ -1055,19 +1279,34 @@ endpoint form, now sha-pinned. calculation_version) on delete no action` — the DB-enforced membership invariant (§5.9): a snapshot can exist only for a scope that belongs to the snapshot's own calculation version, repository, - and tenant. `NO ACTION`/`RESTRICT` — **not** CASCADE — so deleting a + and tenant. `ON DELETE NO ACTION` — **not** CASCADE — so deleting a membership row can never destroy snapshots; abort/GC ordering is - snapshots first, membership second (§5.6, §5.7). This is the one - change to a 012-deployed table; migration 012's file itself is never - touched. The exact FK action is verified against real Postgres in the - implementation tranche. + snapshots first, membership second (§5.6, §5.7). +- `alter table public.ownership_snapshots drop constraint + ownership_snapshots_scope_repo, add foreign key + (scope_id, repository_id) references ownership_scopes + (id, repository_id) on delete no action` — the §5.9a base-scope + invariant: the existing `ON DELETE CASCADE` lets an erroneous direct + scope delete destroy active snapshots and membership; `NO ACTION` + makes the delete fail while any snapshot references the scope. The + constraint is re-created under the same name/keys; only the delete + action changes. + +The exact action is **`ON DELETE NO ACTION`** for all three retained +child→scope/membership relations (§5.9a explains why NO ACTION rather +than RESTRICT or CASCADE); the real-Postgres migration test in §5.9a is +the required proof that the choice is correct, including that single- +statement repository/account teardown still cascades the complete +ownership graph. **EPHEMERAL:** - `create table public.ownership_history_work` — §4.4 columns; composite FKs `(repository_id, user_id) → repositories` and `(scope_id, repository_id) → ownership_scopes(id, repository_id)`, all - `on delete cascade`; unique `(scope_id, calculation_version, + `on delete cascade` (EPHEMERAL rows are not retained children — the + §5.9a NO ACTION rule applies to the two retained child→scope + relations only); unique `(scope_id, calculation_version, period_start, work_key)`; CHECKs on work_key shape, non-negative counts, `calculation_version > 0`, first ≤ last seen. - `create table public.ownership_contributor_build_work` — §4.4 columns: @@ -1081,9 +1320,11 @@ endpoint form, now sha-pinned. only — on **both**. - `own_checkpoint_page(...)` — §6.1; `security invoker`, `EXECUTE → service_role`, all other grants revoked. The same transactional pattern - covers scope-COMMIT (§5.3) and promotion (§5.7) — whether as additional - functions or a client-managed multi-statement tx is an implementation - detail of 1C-B. + covers scope-COMMIT (§5.3), the §5.3 scope-restart staging cleanup + (committed-V-provenance-preserving deletion before fresh ACCUMULATE), + promotion (§5.7), and the §5.6c fresh-build preflight GC gate — + whether as additional functions or a client-managed multi-statement tx + is an implementation detail of 1C-B. - `own_abandon_build(p_user, p_repository, p_version)` — **the canonical build-abandonment primitive** (§5.6): locks the sync-state row, validates the expected `building_version`, deletes the version's @@ -1114,14 +1355,19 @@ endpoint form, now sha-pinned. `ownership_scope_versions` **is present** in dumps and restores; the snapshot→membership FK is enforced (a snapshot without membership is rejected); the legal abort/GC delete ordering works (snapshots, then - membership); repository and account deletion still cascade the complete - ownership graph. -- **One existing-table change**: the `ownership_snapshots` composite FK - above. Otherwise `phase` enum already contains `history`; resolve is - cursor-jsonb state, not a new phase; the error taxonomy already covers - every failure mode in §13; and the only existing-object change is - `create or replace` on `dash_ownership_scopes` — same signature, same - grants, version-correct semantics. + membership); **the §5.9a base-scope enforcement is proven** — direct + delete of a scope referenced by active membership or snapshots is + rejected, while the legal snapshots→membership→scope order succeeds; + repository and account deletion still cascade the complete ownership + graph **in a single statement** after the FK action changes. +- **Two existing-table changes**: the `ownership_snapshots` composite + membership FK above, and the `ownership_snapshots_scope_repo` FK + action change CASCADE → NO ACTION. Otherwise `phase` enum already + contains `history`; resolve is cursor-jsonb state, not a new phase; + the error taxonomy already covers every failure mode in §13; and the + only existing-object change is `create or replace` on + `dash_ownership_scopes` — same signature, same grants, version-correct + semantics. --- @@ -1206,8 +1452,9 @@ endpoint form, now sha-pinned. point, then base scopes referenced by nothing pruned — active scopes/snapshots never cascade away; - a delete attempt on a base scope still referenced by active-version - membership or snapshots is rejected/blocked by the implementation - contract — never a cleanup path. + membership or snapshots is **rejected by the database** — both + retained child→scope FKs are `NO ACTION` (§5.9a); never a cleanup + path. - **DB-integrity tests — real Postgres (drill/migration tests, not mocks):** - membership S/V exists → snapshot S/V succeeds; @@ -1217,10 +1464,20 @@ endpoint form, now sha-pinned. - membership under user A → snapshot under user B **rejected** (cross-tenant); - delete membership while a snapshot references it → **rejected** by - the RESTRICT/NO ACTION FK; + the `NO ACTION` FK; - delete V snapshots first, then V membership → succeeds; - - repository delete → full ownership graph cascades; account delete → - same. + - **direct `DELETE FROM ownership_scopes` on a base scope referenced + by active membership → rejected** (`scope_versions → scopes` is + `NO ACTION`); + - **direct delete on a base scope referenced by active snapshots → + rejected** (`snapshots → scopes` is `NO ACTION` after the 013 + constraint change — an erroneous direct scope delete cannot + cascade-destroy active data); + - delete the version's snapshots → its membership → then the base + scope → **succeeds** (the only legal base-scope prune order); + - repository delete → full ownership graph cascades **in one + statement** (proves `NO ACTION` child FKs do not degrade teardown); + account delete → same. - **Cardinality tests:** - ≤500 selected directory candidates → ≤500 V membership rows (repo scope separate); @@ -1234,6 +1491,47 @@ endpoint form, now sha-pinned. the union, no unbounded accumulation; - abort N+1 → N+1-only base identities pruned; promote N+1 + GC N → N-only base identities pruned. +- **Scope-restart staging tests (§5.3):** + - *current scope RESOLVE partially stages then fails:* prior committed + V scopes exist; current scope stages new key X; a later represented + key fails resolution → scope restarts → X's staged row is removed + (no committed V snapshot references it); prior committed staging is + preserved; + - *existing active-N contributor staged only by the failed V scope:* + contributor row already exists from active N; failed N+1 scope + stages its key; no committed N+1 snapshot references it → restart + removes its build-work row; the retained row itself is untouched and + promotion cannot update it accidentally; + - *key shared with an earlier committed V scope:* the failed current + scope stages a key an earlier V scope already finalized → the + earlier V snapshot references the contributor → restart preserves + the build-work row; + - *represented set changes after fresh ACCUMULATE:* linkage drift + produces a different represented set on restart → stale keys from + the previous attempt survive only if referenced by committed V + snapshots; all others are gone before the new accumulate begins; + - *ready-to-promote with an extra build-work key:* a staged key with + no represented V snapshot → promotion validation rejects/fails + closed (exact-set invariant); + - *ready-to-promote with a missing build-work key:* a represented V + key with no staged row → promotion rejects/fails closed; + - *exact set:* staged linked keys exactly equal the union of + represented linked keys in V snapshots → promotion succeeds and + applies metadata to exactly that set. +- **Lifecycle / GC-gate tests (§5.6c):** + - *no third version:* N promoted over by N+1, retired N rows still + present (GC pending) → a request to initialize N+2 **does not start + a build** — `building_version` stays null; + - *gate completes then builds:* same state → preflight GC removes + retired N rows in legal order (snapshots → membership → unreferenced + contributors → unreferenced base scopes) → only then does the fresh + build initialize `building_version`; + - *retired-version GC failure:* forced GC error mid-preflight → + `active_version` N+1 keeps serving, no `building_version` is set, + next attempt retries the same GC (idempotent partial GC safe); + - *no retired state:* `active_version` only, `building_version` null, + zero retired rows → preflight is a no-op and the build initializes + immediately. --- @@ -1263,8 +1561,8 @@ endpoint form, now sha-pinned. | Repo deleted mid-build | FK cascade removes work+staging+sync rows | n/a | — | — | cascade is total | | Installation revoked | state intact | fails at next fetch | CAS | — | `GITHUB_ACCESS_REVOKED` | | Pinned revision disappears | pages error; cursor=C | retry once → fail closed | partial scope state never served (building_version) | — | `OWNERSHIP_HISTORY_FAILED`; next build re-pins | -| Represented key unresolvable in pass 2 | staging incomplete | bounded retry → restart scope at fresh ACCUMULATE → `OWNERSHIP_HISTORY_FAILED` | — | — | scope restarts; stale staged rows deleted on restart/sweep | -| Pass-2 bot-classification drift | staging not written for that key | same fail-closed restart at fresh ACCUMULATE | frozen pass-1 `is_bot` is the only authority | — | scope restarts; staged rows deleted | +| Represented key unresolvable in pass 2 | staging incomplete | bounded retry → restart scope at fresh ACCUMULATE → `OWNERSHIP_HISTORY_FAILED` | — | — | scope restarts; §5.3 cleanup preserves staging referenced by committed V snapshots, deletes the rest | +| Pass-2 bot-classification drift | staging not written for that key | same fail-closed restart at fresh ACCUMULATE | frozen pass-1 `is_bot` is the only authority | — | scope restarts; §5.3 cleanup deletes uncommitted staging | | Build abort after ≥1 scope COMMITs | committed scope snapshots + contributor rows for `building_version` | abandon tx (§5.6) | — | — | single tx: delete versioned snapshots + both work relations + clear build state | | Abort cleanup, N active + N+1 building | N rows are served data | n/a — cleanup only touches N+1 | — | — | N snapshots and N-referenced contributors preserved; N+1 artifacts deleted | | Orphan contributor pruning | contributor rows unreferenced by every version | part of abandon tx | — | — | anti-join delete — only rows unreferenced by ANY snapshot version are removed | @@ -1279,11 +1577,16 @@ endpoint form, now sha-pinned. | Shared contributor metadata mutate attempt during build | **none — design forbids it** | n/a | retained rows are read-only during build | — | nothing to roll back (§5.8) | | N+1 inserts new base scope + membership | N+1-only membership + base identity | n/a | dashboard joins only active-version membership — new scope invisible in N output | — | abort/GC prunes unreferenced base scope (§5.9) | | N+1 changes cardinality-cap selection | per-version exclusion state on N+1 membership | n/a | N's `excluded`/`exclusion_reason` on its membership rows unchanged | — | shared base row never mutated; abort removes N+1 membership | -| Delete of base scope referenced by active data | forbidden by contract — snapshot FK cascade would destroy active rows | n/a | — | — | prune only deletes scopes referenced by NO membership AND NO snapshot (§5.6) | +| Direct delete of base scope referenced by active data | **rejected by the database** — `snapshots→scopes` and `scope_versions→scopes` are `NO ACTION` (§5.9a) | n/a | active membership/snapshots cannot be cascade-destroyed | — | prune only deletes scopes referenced by NO membership AND NO snapshot (§5.6) | | Restore: V membership + V snapshots restored, work absent | sync claims building V; CORE rows present, EPHEMERAL absent | never piecemeal-resume | — | — | `own_abandon_build` deletes restored V membership + snapshots too | | Snapshot insert without V membership | impossible — composite FK `(scope, repo, user, version)` → `ownership_scope_versions` rejects it | n/a | DB-enforced, not app-only | — | row cannot exist | -| Delete membership while snapshot references it | rejected — FK is NO ACTION/RESTRICT | n/a | incorrect cleanup cannot destroy snapshots | — | ordering enforced by the constraint | +| Delete membership while snapshot references it | rejected — FK is `ON DELETE NO ACTION` | n/a | incorrect cleanup cannot destroy snapshots | — | ordering enforced by the constraint | | Hard-cap overflow directory path | no base row, no membership row | n/a | `display_path` never persisted | — | nothing to clean — never written | +| Fresh build requested while retired version rows remain | previous version awaiting post-promotion GC | preflight GC runs first — build does not initialize while retired rows exist | no third version can coexist (§5.6c) | — | GC in legal order: snapshots → membership → unreferenced contributors → unreferenced base scopes | +| Retired-version GC fails mid-preflight | partial GC progress (idempotent) | retry on next build-start/sweep — never a build start | `active_version` keeps serving normally | — | `building_version` stays null; GC re-runs until retired set is empty | +| Resolve fails after partial build staging | `contributor_build_work` may hold keys staged only by the uncommitted scope | scope restart at fresh ACCUMULATE | restart cleanup preserves only keys referenced by committed V snapshots | — | uncommitted-only staging deleted in the restart transaction (§5.3) | +| Promotion validation finds an extra staged key | staging set ⊋ represented V key set | promotion rejected — fails closed | no shared contributor metadata written | — | exact-set invariant (§5.7): stale staging can never apply | +| Promotion validation finds a missing staged key | staging set ⊊ represented V key set | promotion rejected — fails closed | no shared contributor metadata written | — | same exact-set invariant | --- @@ -1330,6 +1633,9 @@ endpoint form, now sha-pinned. | I. Every page transition exactly-once? | **YES** | §6.2 — single-transaction CAS, proven | | J. Work state backup-excludable safely? | **YES** | §9/§5.6a — loss = discarded *building* version (canonical abandon + fresh rebuild), never active user data | | K. Scope membership versioned + DB-enforced? | **YES** | §5.9 — retained `ownership_scope_versions` (CORE) + composite snapshot→membership FK (NO ACTION); `dash_ownership_scopes` joins membership at `active_version` | +| L. Referenced base scope undeletable? | **YES** | §5.9a — both retained child→scope FKs `ON DELETE NO ACTION`; repo/account teardown still cascades via the direct repository/user FKs | +| M. Third version coexistence impossible? | **YES** | §5.6c — fresh-build preflight GCs every retired version before `building_version` initializes; GC failure = no new build | +| N. Stale staging cannot reach promotion? | **YES** | §5.3 — restart cleanup keeps only committed-V-referenced keys; §5.7 — promotion requires the staged set to *equal* the represented set | ## 17. Readiness verdict diff --git a/docs/ownership-data-privacy-spec.md b/docs/ownership-data-privacy-spec.md index 1cc89ae..30b7139 100644 --- a/docs/ownership-data-privacy-spec.md +++ b/docs/ownership-data-privacy-spec.md @@ -271,7 +271,7 @@ historical row, once written, never changes meaning. |---|---|---| | `id` | uuid pk | | | `user_id`, `repository_id` | uuid FKs | direct tenant scoping | -| `scope_id` | uuid → ownership_scopes, cascade | | +| `scope_id` | uuid → ownership_scopes, cascade (NO ACTION under proposed §27) | | | `contributor_id` | uuid → ownership_contributors, cascade, **null = scope-total row** | per-contributor vs scope summary | | `period_granularity` | text: `month` \| `head` | `head` = current-state snapshot (HEAD blame + recent-window signals) | | `period_start` | timestamptz | month start; fixed sentinel `1970-01-01` for `head` rows | @@ -599,6 +599,9 @@ composite foreign keys (application-side checks alone are insufficient): `foreign key (contributor_id, repository_id) references ownership_contributors(id, repository_id) on delete cascade` — a snapshot cannot pair a scope or contributor from one repository with another's repository_id, and both parents cascade with the repo. + *(Under proposed §27 the `scope_id` FK's delete action becomes `ON + DELETE NO ACTION` — a referenced base scope must be undeletable; the + contributor FK is unaffected.)* - The required supporting uniques (`ownership_scopes(id, repository_id)`, `ownership_contributors(id, repository_id)`) are Phase-1B implementation detail, as is the exact SQL. The contract is: **the database itself @@ -610,9 +613,12 @@ cascade. Unique `(repository_id, scope_path_hash)`. Index `(repository_id, depth)` for bounded enumeration. Columns per §4.1. Write: ownership ingestion. Read: dashboard RPCs join through it for labels. Cardinality: ≤500/repo; ~50–200 typical. *(Under proposed §27 the bound -is stated per version: ≤500 **directory memberships** per -`calculation_version`; the base table may transiently hold the union of -active+building sets — ≈≤1000 worst case, pruned after abort/GC.)* +is stated per version and per lifecycle phase: ≤500 **directory +memberships** per `calculation_version`; the base table holds ≤500+repo +scope idle, the union of active+building sets — ≈≤1000 — during a build, +and may transiently exceed 500 while retired-version GC is pending after +promotion; a fresh build is gated on retired-version GC, so a third +version can never coexist; pruned after abort/GC.)* ### `ownership_contributors` PK `id`. Composite FK `(repository_id, user_id) → repositories(id, user_id)` @@ -695,8 +701,11 @@ per-tenant fairness, not database survival. *Under proposed §27 the scope bounds decompose precisely — they are four different counts, not one "≤500 scopes/repo":* **base scope identities** -(≈≤ the union of the active and building directory sets; ≈≤1000 transient -during N/N+1 coexistence); **version memberships** (≤500 directories per +(≤500+repo scope at idle; ≈≤ the union of the active and building +directory sets, ≈≤1000, during N/N+1 coexistence; transiently >500 only +while post-promotion retired-version GC is pending — a fresh build is +gated on that GC completing, so three versions never coexist); **version +memberships** (≤500 directories per `calculation_version`, repo scope separate — directories beyond the hard cap are never persisted at all, so membership is never an overflow inventory); **analyzed memberships** (the `excluded=false` subset — @@ -751,7 +760,11 @@ Version bump flow (N → N+1, same machinery): 4. When every scope validates complete, `active_version` flips to N+1 and `building_version` clears — a single-row update, so the flip is atomic. 5. **Only after the flip** are old N rows garbage-collected. The active - version is never deleted during an unfinished rebuild. + version is never deleted during an unfinished rebuild. Under proposed + §27 the bound is hardened further: retired-version rows may exist + only while that GC is pending, and **no fresh build may initialize + until retired-version GC completes** — at most one active plus one + building version, ever; a third coexisting version is impossible. Failure/rollback semantics: @@ -1302,19 +1315,29 @@ The proposed amendment is deliberately narrow: membership at `active_version` — a new N+1 scope is invisible in N output and N exclusion state cannot move. The base row's exclusion columns become non-authoritative legacy; a base scope referenced by - any snapshot or membership row is never deleted (the snapshot FK is - `ON DELETE CASCADE` — deletion would destroy active data). Abandoned - and garbage-collected versions delete their membership rows and then - prune base scopes referenced by nothing. + any snapshot or membership row is **undeletable** — the amendment + changes the existing `snapshots → scopes` FK from `ON DELETE CASCADE` + to `ON DELETE NO ACTION` and creates `scope_versions → scopes` as + `NO ACTION`, so an erroneous direct scope delete is rejected by the + database rather than cascade-destroying active data. Abandoned and + garbage-collected versions delete their snapshots, then membership + rows, then prune base scopes referenced by nothing — the only legal + order. **(d) DB-enforced membership integrity** — `ownership_snapshots` gains a composite FK `(scope_id, repository_id, user_id, calculation_version) → ownership_scope_versions`, backed by a supporting unique on the same four columns, with `ON DELETE NO - ACTION`/`RESTRICT` (never CASCADE): a snapshot can exist only for a + ACTION` (never CASCADE; NO ACTION is the exact named action — it is + checked after the statement's referential actions, which is what lets + single-statement teardown work): a snapshot can exist only for a scope belonging to its own version/repo/tenant, and deleting a membership row can never destroy snapshots — cleanup order is always - snapshots first, membership second. Repository/account deletion still - cascades completely through the other FKs. + snapshots first, membership second. Together with (c), **every + retained child→base-scope relation is `NO ACTION`**: repository/ + account deletion still cascades completely, because the direct + `repositories`/`users` FKs on every ownership table remain `CASCADE` + — proven against real Postgres during the migration tranche, not + assumed. **(e) private-path bound is precise** — the ≤500 cap is a *hard directory-enumeration cap per calculation version*: directories beyond it (`deriveScopeCandidates().overflow`) are never persisted @@ -1327,7 +1350,11 @@ The proposed amendment is deliberately narrow: inventory. While N and N+1 coexist, `ownership_scopes` may hold the union of two bounded sets (≈≤1000 base identities worst case, transient); old-only identities are pruned after abort/GC — no - unbounded accumulation. + unbounded accumulation. The transient bound is enforced by lifecycle, + not hope: after promotion a retired version's rows may remain only + while its GC is pending, and a fresh build may not initialize until + that GC completes — three coexisting versions are impossible, so the + ≈≤1000 bound is real. 3. **Two-pass identity resolution, attribution-frozen.** The accumulate pass fixes every metric — work keys, counts, Top-25, unlinked share, human counts, **and `is_bot` classification** (`human_contributor_count` @@ -1368,6 +1395,16 @@ The proposed amendment is deliberately narrow: preserve `active_version`) — piecemeal resumption, including restarting only the current scope, is prohibited because the discarded metadata is not reconstructable from snapshots. +7. **Build staging integrity.** `ownership_contributor_build_work` has no + scope provenance, so a scope that fails mid-resolve and restarts at a + fresh accumulate first deletes staged keys not referenced by any + committed building-version snapshot and preserves keys that are — + linkage drift between walks can change the represented set, and stale + keys must not survive. Promotion then requires the staged linked key + set to *equal* the union of represented linked contributor keys in the + building-version snapshots — no missing keys, no extra keys — before + any shared `ownership_contributors` row is touched; stale staging can + never write retained metadata. If this amendment is rejected, the honest alternatives are: (a) accept per-scope restart-in-single-invocation with no completion guarantee on From faa83cf63bd466ef6a4ae61635e6106202703648 Mon Sep 17 00:00:00 2001 From: aliferous3 <94698925+Aliferous3@users.noreply.github.com> Date: Mon, 19 Oct 2026 09:51:53 +0400 Subject: [PATCH 9/9] docs: repair checkpoint design section references --- docs/ownership-checkpoint-architecture.md | 6 ++++-- docs/ownership-data-privacy-spec.md | 2 +- 2 files changed, 5 insertions(+), 3 deletions(-) diff --git a/docs/ownership-checkpoint-architecture.md b/docs/ownership-checkpoint-architecture.md index 53b55cb..b2eee53 100644 --- a/docs/ownership-checkpoint-architecture.md +++ b/docs/ownership-checkpoint-architecture.md @@ -58,7 +58,7 @@ or hundreds of pages, and a serverless invocation holds memory only for ~45–60 s. - Persist nothing per-contributor → a crash loses the counts; restart must - re-walk the prefix → no completion guarantee (Option A fails, §3.1). + re-walk the prefix → no completion guarantee (Option A fails, §4.1). - Persist every contributor's identity provisionally → overflow GitHub identities reach storage → violates §6.4 durable-identity minimization and §9 "in memory only". @@ -83,7 +83,7 @@ Checked against `012_ownership.sql` as deployed: |---|---| | `ownership_snapshots` | Contributor rows require `contributor_id` → a durable `ownership_contributors` row → requires `github_node_id`/`display_login` non-null (`ownership_contributors_linked` CHECK). Overflow identities would be written. Also no month-keyed accumulation shape without polluting immutable period facts. | | `ownership_contributors` | The linked CHECK **forces** `github_node_id` + `display_login` — pseudonym-only provisional rows are impossible by construction, and persisting identities there is exactly what §9.1 forbids. It is also **retained, version-shared reference state** — not scoped to `calculation_version` — so it cannot be used as build scratch space at all (§4.4). | -| `ownership_sync_state` (`history_cursor`/other jsonb) | Technically possible, rejected in §3.2: it is OPERATIONAL (dumped into backups — provisional pseudonyms would leak into every dump), a single JSONB row is a read-modify-write hotspot with unbounded growth, and it cannot be CHECK-constrained per element. | +| `ownership_sync_state` (`history_cursor`/other jsonb) | Technically possible, rejected in §4.2: it is OPERATIONAL (dumped into backups — provisional pseudonyms would leak into every dump), a single JSONB row is a read-modify-write hotspot with unbounded growth, and it cannot be CHECK-constrained per element. | | `ownership_scopes` | Scope definition only; nothing accumulates there. | Conclusion: **migration 013 is required.** The delta (§11): one retained @@ -670,6 +670,8 @@ GC on the next build-start attempt (and the scheduled sweep may run it too); the GC is idempotent, so a partial earlier attempt is safe to retry. This makes the §5.9 bound real rather than aspirational. +### 5.7 Promotion — atomic metadata application + version flip + When `pending_scopes` is empty, ONE transaction under the sync-row lock: 1. Verify: `building_version = V`; no `ownership_history_work` rows for V; diff --git a/docs/ownership-data-privacy-spec.md b/docs/ownership-data-privacy-spec.md index 30b7139..e423eaf 100644 --- a/docs/ownership-data-privacy-spec.md +++ b/docs/ownership-data-privacy-spec.md @@ -1373,7 +1373,7 @@ The proposed amendment is deliberately narrow: resolvable and consistent. 4. **Build-boundary SHA.** `ownership_sync_state.history_cursor` may carry the build's pinned commit oid in addition to page cursors — a second - sanctioned SHA beside `last_processed_sha` (§5.12), required because + sanctioned SHA beside `last_processed_sha` (§4.4), required because `defaultBranchRef` resolves at query time and multi-page builds must not straddle pushes. 5. Rejected within the design: provisional persistence of