feat(backend): implement Repository Lineage per RFC-0002 (#299)

[parthrohit22]

Closes #299.

Context

This is the highest-risk item in the backlog: a production schema migration introducing cyclic deferred foreign keys between a new repository_lineages table and the existing repositories table. Given that risk, I stopped and flagged it back before starting, and only began implementation after explicit owner go-ahead with three conditions: real-PostgreSQL concurrency verification (not just SQLite), a full disposable-database rehearsal via the existing rehearsal script, and extra-thorough test coverage on the FK-cycle handling specifically. All three are satisfied — see Verification below.

The design itself is not new: it follows RFC-0002 and the already-owner-approved migration plan (approved via PR #328), with the #322 rehearsal/recovery prerequisite already closed. This PR is that plan turned into working, tested code — I verified every one of its non-trivial claims empirically before relying on it (see below), rather than assuming the plan alone was sufficient proof.

What's new

• repository_lineages: owner-scoped canonical key (canonical_source_key + canonical_branch, case-folded GitHub owner/repo), display_name, latest_repository_id, next_sequence, created_at. A partial unique index is both the owner-scoped canonical lookup and the sole arbiter if two imports race to create the same lineage.
• repositories.lineage_id / .sequence: permanently nullable. Uploads and unresolved-ref legacy GitHub rows are unlineaged standalone imports by design (RFC §4.3/§6), not a transitional state.
• The cyclic integrity boundary: repository_lineages.latest_repository_id(+id) → repositories.id/lineage_id, and repositories.lineage_id(+owner_id) → repository_lineages.id/owner_id. Both deferrable/initially-deferred composite FKs — verified empirically (raw SQL, SQLAlchemy create_all(), and the real Alembic migration) on both SQLite and real PostgreSQL before writing the actual implementation, per the plan's own "if SQLite can't preserve this, stop" instruction.
• Two Alembic revisions, 0013_lineage_expand and 0014_lineage_constraints (the plan suggested 0011/0012, but those numbers were taken by unrelated work — invite tokens, waitlist — that landed on dev since the plan was written). Revision A creates the lineage table, expands repositories, runs a strict deterministic backfill of resolvable historical GitHub commits (grouped by owner + case-folded canonical source + exact ref, sorted by created_at/id, UUIDv5-keyed for idempotent reconciliation), verifies every §6.4 invariant, then closes the non-cyclic constraints. Revision B closes the cyclic one. Uploads, unresolved refs, and any URL outside the exact accepted historical grammar stay standalone — nothing is guessed.
• RepositoryRepository.add_with_lineage: the transactional allocator (RFC §5.2) — find-or-create the lineage, atomically claim the next sequence via UPDATE ... SET next_sequence = next_sequence + 1, check for a same-lineage duplicate commit, insert, update the latest pointer, commit once. Bounded retry (5, matching AiConversationRepository's own pattern) on the create-lineage race.
• RepositoryRepository.delete_with_lineage_update: rolls the latest pointer back to the next-highest surviving sequence before deleting, never decrements the counter, keeps an empty lineage (not garbage collected) so a later re-import never reuses a sequence.
• RepositoryService.import_github_repository now always computes a canonical pair (a resolved ref is guaranteed by this point) and persists through the transactional allocator. import_uploaded_repository is unchanged in behaviour — it already never set lineage fields; added a clarifying comment.

Two real bugs found and fixed along the way

1. A pre-existing duplicate-detection bug, now fixed. I removed the old pre-clone find_by_source_revision_for_owner check. It matched on (source_url, revision_value, owner_id) with no branch/ref component — structurally impossible to make branch-aware, since the ref isn't resolved until after cloning — and it incorrectly rejected re-importing the same commit under a different branch, something RFC-0002 explicitly requires to succeed. This was already broken before this PR; my new lineage-scoped duplicate check is now the sole, correct authority. Covered by test_same_commit_is_allowed_in_two_different_branch_lineages.

2. A SQLite/Alembic interaction bug, found and fixed. alembic/env.py now disables PRAGMA foreign_keys for the connection Alembic itself uses to run migrations, SQLite only. Root cause (confirmed by direct reproduction): SQLite refuses to toggle that pragma mid-transaction, and Alembic's own per-migration transaction is already open by the time a revision's upgrade() runs. Revision B's batch-mode recreation of repository_lineages (required to add the cyclic FK — SQLite has no ALTER TABLE ADD CONSTRAINT) drops and rebuilds a table that repositories already has a deferred FK pointing at from revision A; SQLite's deferred-FK bookkeeping doesn't correctly reconcile that recreation against the still-open transaction, so a fully self-consistent final state still fails at COMMIT with a generic "FOREIGN KEY constraint failed" (PRAGMA foreign_key_check reports zero violations immediately beforehand — confirmed directly). This only reproduced once app.core.database had already been imported elsewhere in the process, so it silently passed in isolated test runs and only surfaced in a full-suite run. The regression test now forces that registration itself rather than depending on incidental test-file ordering, and I verified it actually fails without the fix by temporarily reverting it. Runtime enforcement for the real application is unaffected — every normal app connection still gets PRAGMA foreign_keys=ON; this only touches the connection Alembic uses while migrating.

Verification (the three explicit conditions)

1. Real-PostgreSQL concurrency verification — test_repository_lineage_concurrency.py: real threads against a real, separate-connection PostgreSQL database with threading.Barrier synchronization (matching the existing test_concurrent_refresh_on_postgres_mints_one_successor pattern). Two different commits racing into an existing lineage get unique consecutive sequences; two first-imports racing create exactly one lineage; two identical commits racing produce one repository and one correctly-rejected conflict with no burned sequence number; a forced duplicate (lineage_id, sequence) is rejected; a forced cross-owner attachment is rejected by the composite FK; a forced cross-lineage latest pointer is rejected by the composite FK; account deletion cascades a user's lineages without touching another owner's. All 7 pass reliably (verified across 5 repeated runs) and clean up every row they create.
2. Full disposable rehearsal — python scripts/rehearse_migrations.py (SQLite) and --postgres (real, disposable database) both pass: clean upgrade → clean downgrade → re-upgrade, and the representative 0004_ai_provider_configs baseline reaches head. HEAD_REVISION/REQUIRED_HEAD_TABLES updated for the new head.
3. Extra-thorough FK-cycle coverage — beyond the concurrency file's direct FK-forcing tests, test_repository_lineage_migration.py verifies the deferred FK's exact shape (constrained/referred columns, deferrable) after a fresh migration, plus a populated-database backfill test exercising every §6 grouping/exclusion case in one pass (same source/ref groups; different ref/repo/owner separate; a URL variant — mixed-case host, .git suffix, trailing slash — still canonicalizes into the same lineage; malformed URL, unresolved ref, and upload all stay standalone; a deterministic timestamp tie breaks on repository id).

Full backend suite (pytest, no -k) green on both SQLite and real PostgreSQL. ruff check/ruff format --check/mypy clean on every changed file. npm run generate:api-contract -- --check confirms zero frontend contract drift — matches the plan's "no API contract change, no frontend surface" scope exactly.

Docs updated: the migration plan's revision-ID note now reflects the actual 0013/0014 numbering, and the rehearsal runbook's stated head revision is current.

[parthrohit22]

Pushed a fix for the CI failure: test_cross_owner_lineage_attachment_is_rejected_by_the_database_even_if_forced failed on Linux CI ("DID NOT RAISE IntegrityError") but was never reproducible locally on macOS, even with CI's exact command/flags. Rather than guess at the platform-specific SQLite behavior, the test now sets PRAGMA foreign_keys=ON explicitly on its own connection instead of depending on app.core.database's process-wide connect-event listener having already fired. Doesn't touch application/migration code — the equivalent real-Postgres proof already passes reliably. Waiting on CI to confirm.

[parthrohit22]

Update on the CI failure: the first fix (explicitly setting PRAGMA foreign_keys=ON) did NOT resolve it — confirmed from the actual CI log that the fix's code ran, the pragma was set, and COMMIT still didn't raise IntegrityError on the Linux runner, while the identical scenario raises reliably every time on macOS. This is a genuine platform/SQLite-build difference in deferred foreign-key enforcement at commit time, not a test-setup issue.

Rather than keep guessing at that platform-specific commit-time behavior, I rewrote the assertion to use PRAGMA foreign_key_check instead, which SQLite documents as detecting a violation regardless of the connection's foreign_keys/deferred state — it verifies the exact same fact (this row genuinely violates the ownership FK) without depending on whatever is different about commit-time enforcement in CI's environment. Pushed and verified passing locally; waiting on CI. The equivalent real-Postgres proof is unaffected and already passes reliably.

[parthrohit22]

Root cause found and fixed. PRAGMA foreign_key_check itself found no violation on CI's SQLite — not a timing quirk, the constraint genuinely wasn't being tracked for enforcement there.

The actual cause: Base.metadata.create_all() (used by the client/auth_client test fixtures, and by AUTO_CREATE_TABLES=true in local dev) has to resolve this table pair's genuine FK cycle by emitting one table with an inline FK referencing the other before it exists — confirmed directly via echo=True: repositories was created first, with its FK referencing a repository_lineages table that didn't exist yet. At least one real SQLite build doesn't enforce a deferred FK declared that way. The Alembic migration never does this — 0014 adds each cyclic FK in its own revision, after both tables already exist — so a database built by migrating (every real deployment, and this repo's own CI rehearsal) was never affected.

Moved the proof onto a database built via the real migration instead of create_all(), verified on both SQLite and real Postgres, including with CI's exact command/env reproduced locally (999/999). Documented the underlying create_all()-on-SQLite limitation directly on both cyclic FK declarations so it's not a silent surprise later — it's a dev/test schema-bootstrap gap only, not a production one.