Skip to content

feat(pg): PostgreSQL adapter for ActivateRepositoryGeneration - second CAS of repository publish - #184

Merged
matthewpeterkort merged 16 commits into
feature/postgres-backendfrom
ao/pg-activation/root
Nov 13, 2026
Merged

matthewpeterkort merged 16 commits into
feature/postgres-backendfrom
ao/pg-activation/root

Conversation

@matthewpeterkort

@matthewpeterkort matthewpeterkort commented Nov 13, 2026 •

Copy link
Copy Markdown
Collaborator

feat(pg): PostgreSQL adapter for ActivateRepositoryGeneration — second CAS of repository publish

Closes the last named typed refusal on the PostgreSQL Explorer lane (POST …/explorer-config → PublishRepository → lifecycle.PublishRepositoryGeneration → store.ActivateRepositoryGeneration). Unblocks the Arango-vs-PostgreSQL benchmark battery on PR #159, whose spine dies today with a 503 at this leg.

Base: feature/postgres-backend. Branch: ao/pg-activation/root, append-only.

What ships

  • internal/backends/postgres/internal/explorer/activation.go — ActivationStore: one transaction, lock candidate → verify manifest → verify active-generation pointer → verify PUBLISHED bundle execution → lock owner → supersede prior → flip candidate READY→ACTIVE (or revive FAILED→ACTIVE through the activation-shaped candidateRevive statement) → move owner pointer. crashAt test seam and 4×/5 ms deadlock retry ride the publication.go precedent.
  • internal/backends/postgres/internal/explorer/migrations/000012_explorer_revision_activation_revival.sql — guard evolution admitting the activation-shaped FAILED→ACTIVE revival (see divergence 1; coordinator option-(a) ruling). 000006 stays byte-identical to base.
  • internal/backends/postgres/control/runtime.go + vocabulary_control.go — Runtime.Activate assembled only when explorer+dataset+publication series are all assigned (same gating law as Runtime.Publish); vocabulary export.
  • internal/server/runtime.go — repositoryGenerationActivator narrow interface; pgExplorerStore.activate; delegation replaces the named refusal; runtime.Activate == nil joins the compose fail-closed gate. var _ explorer.Store = pgExplorerStore{} no-omission assertion unchanged.
  • Tests: store-level live suite (20 composite-guard cases incl. revival-witness equality, the ready-only first-CAS pin, FAILED-prior strictness, and guard-refused revival byte-stability), transaction/rollback/race live suite (race repinned never-strands), 20-case transition-guard probe, offline unit tests, server delegation tests, the HTTP spine leg, and the retry-after-activation-failure revival route leg.

PG-law divergences from Arango — 1 resolved to parity (000012), 3 deliberate

  1. Candidate eligibility: RESOLVED TO PARITY via migration 000012 (coordinator option-(a) ruling). The Arango lane admits READY-or-FAILED candidates (candidate FILTER status IN ["READY","FAILED"] plus the ACTIVE replay arm; its activation write clears failedAt/diagnostics), while 000006's trigger law admitted only READY→ACTIVE, READY→FAILED, ACTIVE→SUPERSEDED, ACTIVE→FAILED — so on the PG lane the shared lifecycle's FAILED marking plus the content-digest-keyed replay (a retried POST replays the same pinned revision identity — it does not mint a new id) permanently pinned a lost activation race to 409 GENERATION_ACTIVATION_FAILED. The coordinator ruled option (a): evolve the guard, do not pin the divergence. Migration 000012_explorer_revision_activation_revival.sql CREATE OR REPLACEs loom_explorer_revision_guard with exactly one new arm, following the sanctioned 000011 precedent (same-named function replaced whole, trigger dropped and recreated, convergence on rerun — independently verified against the 000011 file). 000006 is merged history and stays byte-identical to base: sha256 3f6c44bff72d7837634e8c170f0b97e256e38d35f7c844fad27f4fbda38ecc09, zero diff against origin/feature/postgres-backend. The arm is strictly narrower than Arango's blind READY-or-FAILED filter — a FAILED→ACTIVE write is admitted only in the activation shape:

    Trigger law Before (000006) After (000006 + 000012)
    READY→ACTIVE, READY→FAILED, ACTIVE→SUPERSEDED, ACTIVE→FAILED legal legal (untouched)
    FAILED→ACTIVE activation shape (failed_at→NULL, activated_at set, diagnostics present, ready_at unchanged) refused 23514 legal
    bare FAILED→ACTIVE (keeps failed_at / no activated_at / nulls diagnostics) refused 23514 refused 23514
    FAILED→READY, FAILED→FAILED, FAILED→SUPERSEDED, anything out of SUPERSEDED refused 23514 refused 23514
    frozen-column change on any UPDATE (incl. during revival) refused 23514 (frozen check runs first) refused 23514 (untouched)
    DELETE (insert-once) refused 23514 refused 23514

    firstCasStaysReadyOnly: the authoring-path statement candidateActive keeps WHERE id = $1 AND status = 'READY'; revival rides its own statement candidateRevive (WHERE id = $1 AND status = 'FAILED'), so the first CAS provably never touches a FAILED row — pinned live by ready-only-first-cas-never-touches-failed-candidate (0 rows affected against a FAILED candidate) and statically pinned in the audited-statement battery (TestActivationStatementsCarryTheCompositeGuards, activation_test.go): both statements carry verbatim SET-clause and WHERE-filter entries plus one-statement count checks, mutation-proven non-vacuous — dropping either status filter reddens the battery (152f59f7, review follow-up). The consumer-visible outcome now matches Arango: the conflict-armed deploy lands 409 with the revision witnessed FAILED, and a retried POST returns 200 activated=true on the same pinned identity (witness-equality against a fresh activation, no restamp on replay, pointer flipped, diagnostics cleared). Rename mapping: the option-B pin TestPostgresRuntimeRepositoryPublishRetryAfterActivationFailureStaysPinned (aea0653, superseded by the ruling) became TestPostgresRuntimeRepositoryPublishRetryAfterActivationFailureRevives; the CI dual-PASS gate, the ClickHouse-gated mint inventory, and the Arango declared-skip registry moved with it. The store-level refusal case candidate-failed left the guard table (it is legal now) and the revival legs above took its place.

  2. Generation pointer VERIFIED, not blind-patched. Strictly stronger than Arango's blind patch (F-B17 class) and correct because leg-1 ActivateRelease CAS moved the pointer moments earlier; mismatch → ErrDraftConflict. Coordinator-blessed.

  3. Replay = conditional no-op commit, mirroring Arango activation.go:18/31 exactly: no-op success only when candidate.status=ACTIVE and owner.active_revision_id == candidate. Orphan-ACTIVE (owner elsewhere) → ErrDraftConflict, never silent success; SUPERSEDED → conflict. Battery-retry safety therefore means owner-already-flipped. Both negatives are live cases. Coordinator-pinned.

  4. Dangling active_revision_id → ErrDraftConflict (Arango's DOCUMENT()-null path would proceed). No FK exists on the pointer in PG, so the state is reachable; proceeding would activate an explorer onto a nonexistent revision — silent corruption amplification. ErrDraftConflict maps to fixed-text 409 GENERATION_ACTIVATION_FAILED, non-retryable — the fail-closed posture for corruption, no retry storm. Lifecycle never mints a dangling pointer; battery-invisible. Coordinator-approved, same disposition lineage as First impressions of Loom #1 (schema-strictness class). The live negative case proves the full control-plane ledger (revisions and owners, updated_at excluded) is byte-stable through the conflict.

Design-brief record

Brief + amendments from loom-experiment-2 (PR recovery orchestrator): spine consumer = PR #159 paired battery; acceptance = store-level CAS battery plus HTTP-level proof through the final flip; taxonomy = neutral explorer.ErrDraftConflict for every guard miss/zero-row CAS, raw wrap for transport; one migration, 000012 activation revival (see divergence 1); all other CAS state pre-exists: explorers.active_revision_id, explorer_revisions status/activated_at/failed_at/diagnostics/publication under the 000006 trigger law, dataset_generation_manifest.state, dataset_active_generation.generation, publication_bundle_executions.state); scope lock honored — ValidateSelectionReferences, benchmark harness, verification/**, reviewed-sources.json untouched.

Contract-surface inventory (for 204 matched-state check)

  • Route: POST /api/v1/projects/{project}/generations/{generation}/explorer-config — behavior change is confined to the final activation leg; every leg before the flip (capability, compile, ClickHouse materialization, PublishAuthoring, release CAS) is byte-identical because this PR adds no writes before ActivateRepositoryGeneration and changes no shared code paths.
  • Response fields (200): project, generation, explorerId="default", receiptId, revisionId, executionId, recipe, translationVersion, activated=true — unchanged shape; previously unreachable on PG (503 before this change).
  • Error mapping (newly reachable): explorer.ErrDraftConflict → lifecycle fixed-text 409 GENERATION_ACTIVATION_FAILED "generation activation lost a concurrent Explorer update; the prior Explorer revision was retained", non-retryable (repository_publish_generation_activation_classification_test.go contract, untouched and passing); transport → 503 RELEASE_ACTIVATION_FAILED retryable via raw wrap. dataframeerrors.CodeBackendUnavailable + stage=adapter remains only for genuine backend-unavailable (activate port nil when series unassigned, and the retained ValidateSelectionReferences refusal).
  • Untouched refusals: ValidateSelectionReferences unchanged refusal (named typed refusal retained).
  • Activation eligibility (divergence 1, resolved to parity): candidate must be READY (first CAS candidateActive) or FAILED in activation shape (revival CAS candidateRevive, admitted by the 000012 guard arm); any other status → ErrDraftConflict → 409 GENERATION_ACTIVATION_FAILED. A retried POST after an activation failure returns 200 activated=true on the pinned revision identity — see divergence 1.

Offline battery (all re-run on final head)

gofmt -l clean; go vet ./internal/... clean; go build ./... clean; go test -count=1 green for ./internal/backends/postgres/..., ./internal/server/..., ./internal/explorer/... (includes lifecycle classification contracts untouched). Re-run on the merged tree at hop commit 45710219: same gates plus the full CI-equivalent battery (go test . ./internal/... ./cmd/... ./generated/... ./conformance/..., ClickHouse-lane packages excluded per CI) all green. At the 000012 head 4339cbbc: gofmt -l/go vet clean on the changed packages, both changed test binaries compile, and the changed packages ran green live (see window re-runs); the full CI-equivalent battery runs in CI at this head.

Live evidence (PostgreSQL 18, LOOM_TEST_POSTGRES_URL, isolated verify_pgact_*/verify_pgcomp_* databases + schemas, secret scrubbing, zero skips in the live window)

Composite-guard battery — TestActivateRepositoryGenerationUsesCompositeGuardsAgainstPostgreSQL (20 cases):

Case Assertion Result
happy-flip ACTIVE + activated_at set + publication payload ACTIVE + prior SUPERSEDED + owner pointer; replay no-op (no restamp, no pointer churn) PASS
no-manifest / manifest-failed / no-active-generation / pointer-names-other-generation ErrDraftConflict, ledger byte-stable PASS ×4
execution-missing / execution-not-published / zero-materializations ErrDraftConflict, ledger byte-stable PASS ×3
owner-not-repository-mode / candidate-wrong-explorer / candidate-wrong-generation ErrDraftConflict, ledger byte-stable PASS ×3
missing-candidate-revision ErrDraftConflict PASS
dangling-prior-pointer (divergence #4) ErrDraftConflict; revisions and owners byte-stable; candidate stays READY PASS
replay-with-orphaned-active-candidate (pinned #3) ErrDraftConflict; byte-stable both snapshots PASS
superseded-candidate (pinned #3) ErrDraftConflict; byte-stable; current ACTIVE undisturbed PASS
pointer-already-names-ready-candidate success; no self-supersede; prior stays ACTIVE PASS
failed-candidate-revival-matches-fresh-activation-witnesses (divergence #1, resolved) retry activates the FAILED revision: ACTIVE, activated_at set, failed_at NULL, diagnostics [], publication ACTIVE/pinned-id/generation, prior SUPERSEDED, pointer flipped; witness equality against a fresh activation; replay after revival does not restamp PASS
ready-only-first-cas-never-touches-failed-candidate (firstCasStaysReadyOnly) the status = 'READY' CAS shape affects 0 rows on a FAILED candidate; revision stays FAILED PASS
failed-candidate-guard-refusal-leaves-failed-state-byte-stable guard miss on a FAILED candidate: ErrDraftConflict; revisions and owners byte-stable; failed_at/diagnostics preserved; pointer unmoved PASS
prior-pointer-failed-candidate-revival-is-no-bypass (FAILED-prior strict) prior failed via the production FailRevisionFrom CAS (ACTIVE→FAILED, legal); activation attempts → ErrDraftConflict from the supersede lock; revisions and owners byte-stable — revival of the candidate never bypasses a FAILED prior PASS

Transactions/CAS battery — TestActivationUsesTransactionsAndSingleModificationQueriesAgainstPostgreSQL: ordered leg sequence prior-supersede → candidate-activate → owner-activate → pre-commit; crash-rollback at 3 seams (each: injected fault, byte-stable ledger, candidate READY, prior ACTIVE, pointer unmoved) PASS ×3; activation-races-failure-never-strands (goroutine-pair activation vs FailRevisionFrom: activation always converges — whichever order the row lock serializes, the candidate ends ACTIVE with the pointer flipped and prior SUPERSEDED; the racing failure either loses with ErrStaleRevisionTransition or wins-then-revives) PASS 5× under -race.

Transition-guard probe — TestPostgresRevisionTransitionGuardProbe (20 cases, raw-SQL UPDATEs against the trigger): the five activation-shape cases pin the 000012 arm in both directions — legal revival accepted; sloppy revival keeping failed_at, revival without activated_at, revival nulling diagnostics, and frozen-dataset smuggling during revival all refused with 23514 — alongside the 15 pre-000012 monotonicity cases. The non-vacuity suite (TestPostgresRevisionFrozenGuardProvenNonVacuous) re-verifies, after trigger drop + rerunMigration(6) + rerunMigration(12) restore: insert-once refusal restored, bare FAILED→ACTIVE refused with the activation-shape message, and the activation-shaped revival accepted with witnessed failed_at cleared and activated_at set.

Window re-runs (granted by coordinator, serial): early runs (post-3931c719, post-hop on 45710219) — composite-guard 18/18 PASS, transactions suite 6/6 PASS, single-winner race 5/5 PASS under -race (0 data races); the post-hop run added the full pgComp battery (ok, 62.7s, incl. ClickHouse-gated mint audit). Post-000012 battery (head 4339cbbc, same granted window, PostgreSQL 18.6): full ./internal/backends/postgres/internal/explorer/ package ok (20.4s, zero FAIL/SKIP), transition-guard probe 20/20, non-vacuity suite PASS, composite-guard battery 19/19 (incl. the three revival legs), transactions suite 5/5 with activation-races-failure-never-strands 30/30 PASS under -race ×5 (0 data races), ./internal/backends/postgres/internal/store/ package ok (11.6s, all four checksum ledgers + ownership manifest re-pinned for 000012, fixture counts 11→12, retroactive-refusal fixture derived), ./internal/explorer/lifecycle/ ok. FAILED-prior strict follow-up (head 652cc9a3, same window): composite-guard 20/20 PASS (new prior-pointer-failed-candidate-revival-is-no-bypass leg), full explorer package re-ok (20.9s, zero skips). All logs secret-scanned clean (0 credential hits), URL never printed; window released with these counts.

Ledger census (000012): sha256 08a61bf6ca66a400112f51463f2c5e1c0a7f4cfbb2d707c0063f87829b1acfe9 pinned in the checksum registry (explorers_test.go), the state-02 pin map (revisions_test.go), the ownership manifest (store/migration_ownership.go), and the activation control pins (activation_e2e_test.go); migrate-history fixture counts moved 11→12 in pubauth_fixture_test.go and activation_e2e_test.go; the membership convergence list extended to 12. 000006 verified byte-identical to base at every head of this branch.

HTTP spine — TestPostgresRuntimeRepositoryPublishRouteActivatesThroughFinalFlip (internal/server/repository_publish_wiring_pg18_test.go): production composePostgreSQLFullRuntime over a live six-series fixture + ClickHouse; POST explorer-config → 200 activated=true; direct witness explorer_revisions.status=ACTIVE, activated_at set, owner pointer = revisionId; replay POST → 200 same revision, no activated_at restamp; viewer reload serves the activated revision; lane-leak audit (no Arango/AQL tells).

Materialize-reached diagnostic (CH-less, local): with LOOM_CLICKHOUSE_URL pointed at a closed port, the spine demonstrably passes capability seeding, compile, and receipt stages and fails only at the ClickHouse dial with 422 MATERIALIZATION_FAILED "Explorer publication is unavailable: ClickHouse database: dial tcp … connect: connection refused" — the backend-unavailable shape, proving every pre-flip leg works on the PostgreSQL lane and that the flip is the sole missing capability this PR supplies.

CI acceptance lane (both-services job)

pg-repository-publish job in .github/workflows/tests.yaml: ephemeral job-local postgres:18 + clickhouse/clickhouse-server:25.3 services (fresh loom_test databases per job — never shared state), LOOM_TEST_POSTGRES_URL + LOOM_CLICKHOUSE_URL/_DATABASE/_USERNAME/_PASSWORD wired. Runs the pgComp battery whole (the spine leg shares the census, the ClickHouse-gated mint inventory — held to one database when CH is configured — and the TestZZ zero-leftover audit), preserves the -v log as an artifact, and a follow-up step requires the --- PASS lines of both the spine leg and the retry-revival leg and exits 1 on any SKIP: silent-skip is a red gate. The gated test itself skips with explicit set-the-env messages when either service is absent (repo precedent release_dangle_reconcile_e2e_test.go), so ordinary jobs are unaffected.

CI execution history (accurate, post-terminal reads only): at head 45710219 run 43411983672 completed failure: the spine job 131840396511 was green with the real test line --- PASS: TestPostgresRuntimeRepositoryPublishRouteActivatesThroughFinalFlip (0.54s), while two other jobs failed. The Arango job failed on my omission — the declared-skip runner (scripts/run-arango-integration.mjs) rejected the spine leg's legitimate env-gated skip as undeclared; fixed by registering it in pgRuntimeComposition (d42e7a5f, comment-stripped per lane law in 710f3bcf; JSON-diff vs the tip-green run proved no census drift). The Node job failed on a pre-existing environment flake (row96-harness-lifecycle.test.mjs Chrome DevTools-port advertise timeout; 1/788; tip-green with identical file content; zero overlap with this diff). A native-review pass then landed two fixes: pipefail propagation for the tee-masked battery step (735955c7, matching the in-workflow precedent) and the retry-pinning leg (aea0653c). Two further reds were self-inflicted and repaired transparently: cb1f108e's predecessor armed the dev fault on a fresh project, where the one-shot is consumed without effect (pinned by TestDevRepositoryActivationConflictOnceCannotJumpToLaterRequest) — the leg now bootstraps a live release pointer with an unarmed runtime and arms a second runtime on a fresh generation; and cb1f108e met base movement — fd2bfeb4 (#186) centralized the schema-name predicate and deleted the package-local schemaNamePattern the activation constructor validated through, so six jobs failed to compile in the merged tree. Fixed by a clean merge hop (9b4fea69) + adopting pgstore.ValidSchemaName (a2c272f6), the same law #186 imposes on every sibling adapter. Authoritative acceptance: run 43419958798 at final head a2c272f6 completed success — all 15 jobs post-terminal (12 test jobs + 3 image jobs). The pg-repository-publish job log carries both real test lines: --- PASS: TestPostgresRuntimeRepositoryPublishRouteActivatesThroughFinalFlip (0.49s) and --- PASS: TestPostgresRuntimeRepositoryPublishRetryAfterActivationFailureStaysPinned (0.85s); the only --- SKIP occurrences in the log are the assert step's own echoed command text. Corroborating earlier successes: 43413473720 (d42e7a5f), 43414054215 (710f3bcf), 43415460100 (735955c7). Post-000012 (option-(a) wave, head 4339cbbc): the retry leg is renamed to TestPostgresRuntimeRepositoryPublishRetryAfterActivationFailureRevives and now asserts the 200-activated revival (409 with witnessed FAILED on the conflict-armed deploy, then 200 activated=true on the pinned identity with diagnostics cleared, pointer flipped, and no restamp on replay); the PASS-line gate, the ClickHouse-gated mint inventory, and the Arango declared-skip registry moved with the name. The --- PASS: …StaysPinned (0.85s) line recorded at a2c272f6 belongs to the superseded option-B pin. Authoritative acceptance — merging-head receipt (recorded post-terminal): run 43423381565 at exactly head 152f59f7b2bca897f256e2c0cb7ec66d68155457 completed success (15/15 jobs, updated 12:21:47Z). The acceptance job pg-repository-publish 131878767182 completed success, all 15 steps green, with verbatim test lines:

--- PASS: TestPostgresRuntimeRepositoryPublishRouteActivatesThroughFinalFlip (0.69s)
--- PASS: TestPostgresRuntimeRepositoryPublishRetryAfterActivationFailureRevives (1.40s)

0 --- FAIL lines; 0 test --- SKIP lines once the assert step's own echoed grep -q commands are excluded; step 6 "Assert the spine leg executed and never skipped" completed success — the silent-skip red-gate (exit 1 on missing PASS line or SKIP) passed. This is the first CI execution of the 000012 wave (guard-evolution migration, checksum pins, ledger counts 11→12, ownership manifest, renamed Revives legs) at the to-be-merged head, and it is green. History for completeness: run 43422297145 (4339cbbc) and run 43422660051 (652cc9a3, whose pg-repository-publish job 131876349490 had itself completed success with both PASS lines, 0.82s/1.71s) were each superseded mid-flight by the concurrency group when the next push landed — cancelled by design, not failed.

Zero-skips statement: in the recorded PG live window every PG18 store-level leg executed — no live leg skipped. The HTTP spine is CH-gated by design locally (disclosed: --- SKIP: … set LOOM_CLICKHOUSE_URL…); its acceptance proof is the PASS line in the pg-repository-publish CI job log, where the leg executes with both services and the job fails if it skips.

runtime_test inversion note

internal/server/runtime_test.go: the ActivateRepositoryGeneration assertion is inverted from typed-refusal to success-path delegation through fakeRepositoryGenerationActivator (narrow-interface fake; error passthrough asserted). The remaining refusal assertions (ValidateSelectionReferences, PublishAuthoring-nil-publish) are kept intact, as is the nil-Activate named-sibling refusal.

Risks / follow-ups

  • Deadlock retry rides the 4×/5 ms store precedent; contention profile under the battery to be watched in 204's matched-state check.
  • The HTTP spine leg is the only test needing both live services; together with the retry-revival leg it runs in the dedicated pg-repository-publish CI job with dual PASS-line gates (silent-skip = red).
  • Base-branch note: the coordinator-ordered merge-commit hop (merge commit 45710219, base tip 6b94220d) … and a second clean hop (9b4fea69, base tip fd2bfeb4) pulling refactor(pg): centralize the schema-name predicate in the store package [pg-cleanup-174] #186's centralized schema-name predicate, with activation.go adopting pgstore.ValidSchemaName in a2c272f6 resolved exactly two pre-authorized take-ours hunks in internal/server/runtime.go (my activator interface block; the compose runtime.Activate == nil nil-gate term) — alignment artifacts of the test/comments(go-application): remove handwritten comments from lane 02 — cmd, internal, generated (code-comments-20261113-02) #163 comment sweep, which this branch obeys (my authored hunks carry zero handwritten comments). No rebase, no force; PR is MERGEABLE.

matthewpeterkort and others added 8 commits November 13, 2026 09:14
Implement explorer.Store.ActivateRepositoryGeneration for the PostgreSQL
backend as the second compare-and-set of the repository publish flow,
replacing the deliberate typed refusal in the server runtime.

The adapter runs one transaction against the control, lane, and
execution schemas: lock the candidate, verify the generation manifest,
the active-generation pointer, the PUBLISHED bundle execution, and the
REPOSITORY management mode, then supersede the prior revision, flip the
candidate READY->ACTIVE, and move the owner pointer. Every guard miss
and zero-row CAS maps to the neutral explorer.ErrDraftConflict;
transport failures stay raw so lifecycle keeps them retryable.

Three deliberate divergences from the Arango adapter, agreed with the
orchestrator: the candidate must be READY (not READY|FAILED), the
generation pointer is verified rather than blind-patched, and a replay
is a no-op commit.

Verification:
- gofmt clean; go vet ./internal/... clean; go build ./... clean
- go test -count=1 green for ./internal/backends/postgres/...,
  ./internal/server/..., ./internal/explorer/...
- live PG18 suites (LOOM_TEST_POSTGRES_URL, isolated verify_pgact_*
  database + schemas, zero skips): 15 composite-guard cases including
  happy flip, idempotent replay, and every guard miss with revision and
  owner snapshots unchanged, plus rollback at all three crash seams and
  the activation-vs-failure single-winner race (5 runs with -race)

Co-Authored-By: qwen3.8-flash-next <noreply@anthropic.com>
Coordinator amendment for the activation unit (PR #159 battery spine):

- activation_e2e_test.go: pinned replay law cases per the approval.
  replay-with-orphaned-active-candidate-conflicts proves an ACTIVE
  candidate whose owner pointer moved elsewhere loses the neutral
  sentinel race (never silent success); superseded-candidate-conflicts
  proves SUPERSEDED never re-activates. Replay no-op success stays
  conditional on owner.active_revision_id == candidate, matching
  Arango activation.go:18/31 so a battery retry means
  owner-already-flipped.

- repository_publish_wiring_pg18_test.go: HTTP-level acceptance on
  live PG. The production composition (composePostgreSQLFullRuntime)
  serves POST /api/v1/projects/{p}/generations/{g}/explorer-config
  through the real capability, compile, ClickHouse materialization,
  PublishAuthoring, release-activation, and second-CAS activation
  legs; asserts HTTP 200 activated=true, witnesses the flip in
  explorer_revisions and the owner pointer directly, and replays the
  POST to prove the no-op does not restamp activated_at. Gated on
  both LOOM_TEST_POSTGRES_URL and LOOM_CLICKHOUSE_URL; with both set
  it never skips. A ClickHouse-less diagnostic confirms the spine
  reaches materialization (fails there with
  MATERIALIZATION_FAILED/backend-unavailable), proving capability and
  compile seeding work on the PostgreSQL lane.

- runtime_postgres_live_test.go: mint inventory gains a ClickHouse-
  gated leg class so the zero-leftover audit holds the spine leg to
  the one-database law exactly when ClickHouse is configured, and to
  minting nothing when it is not.

Verification: live PG18 composite-guard suite green with 18 cases
including both new pinned negatives; full pgComp server battery live
green (62.8s) with the audit exemption exercised; gofmt, go vet
./internal/..., go build ./..., and offline suites for
./internal/backends/postgres/..., ./internal/server/...,
./internal/explorer/... all clean; lifecycle classification
contracts untouched and passing.

Co-Authored-By: qwen3.8-flash-next <noreply@anthropic.com>
…conflict

Divergence-4 acceptance addition from the coordinator: the
dangling-prior-pointer negative now snapshots owners alongside
revisions, so the strict ErrDraftConflict posture is proven to leave
the full control-plane ledger (revisions + explorer owners, updated_at
excluded) byte-stable through the conflict, matching the composite
guard-case discipline.

Co-Authored-By: qwen3.8-flash-next <noreply@anthropic.com>
The merged lane-02 comment sweep (4878319, #163) made
internal/server/runtime.go and the lane's test files comment-free,
conflicting with the activation unit's commented additions there.
Take the base's comment-free runtime.go and reapply the activation
delegation unchanged in shape: the narrow repositoryGenerationActivator
interface, the pgExplorerStore.activate field, the nil-gated
delegation replacing the typed refusal, the runtime.Activate nil gate,
and the compose wiring. Strip the handwritten comments added to
runtime_postgres_live_test.go and
repository_publish_wiring_pg18_test.go so every authored hunk obeys
the sweep law. No behavior change.

Co-Authored-By: qwen3.8-flash-next <noreply@anthropic.com>
…lickHouse

The spine leg needs both live services: real ClickHouse materialization
sits between compile and the second-CAS activation flip, and no local
ClickHouse exists in the worker sandbox by adjudication. Add one job
with both ephemeral job-local services (postgres:18 +
clickhouse/clickhouse-server:25.3, wiring copied from the pg-fixture
and go-clickhouse runners), both env families wired, and run the pgComp
battery whole so the spine leg shares the census, the ClickHouse-gated
mint inventory, and the TestZZ zero-leftover audit.

Silent-skip is a red gate: the log is preserved as an artifact and a
follow-up step requires the spine leg's PASS line and fails on any SKIP
of it, so the acceptance proof is executed evidence, not a green job
that skipped past.

Verification: workflow parses (12 jobs, both services, five env keys);
node --test scripts/lib/verification-inventory.test.mjs and
scripts/ci/list-node-tests.failure.test.mjs 14 pass 0 fail;
test_ui_ci_workflow.py OK; gofmt/vet/build clean; spine leg
by-design-skips without envs offline.

Co-Authored-By: qwen3.8-flash-next <noreply@anthropic.com>
…/root

Coordinator-ordered merge-commit hop for PR #184 (no rebase, no force).
Two pre-authorized take-ours resolutions in internal/server/runtime.go:
(a) keep the repositoryGenerationActivator interface block where the
lane-02 comment sweep deleted adjacent comments; (b) keep the
runtime.Activate == nil term in the compose fail-closed nil-gate.
No other manual resolutions.

# Conflicts:
#	internal/server/runtime.go
…ane external-skip registry

The go-arango job's run-arango-integration runner rejects undeclared
skips: TestPostgresRuntimeRepositoryPublishRouteActivatesThroughFinalFlip
requires LOOM_TEST_POSTGRES_URL and LOOM_CLICKHOUSE_URL, never Arango, so
its explicit skip in the Arango-only lane was flagged as
'Unexpected skipped case'. Registered in pgRuntimeComposition following
the pg-runtime-server-composition precedent; the dedicated
pg-repository-publish sibling job asserts its PASS line. Runner
self-test (scripts/run-arango-integration.test.mjs) passes.
Lane-04 comment law applies to newly authored hunks; the registration
entry keeps only its machine-required string, matching the 6a4e7af
precedent. Runner self-test scripts/run-arango-integration.test.mjs
passes (fail 0).

@matthewpeterkort matthewpeterkort left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review: PR #184 — PostgreSQL adapter for ActivateRepositoryGeneration (head 710f3bc)

Verdict: CHANGES REQUESTED (posted as COMMENT because GitHub rejects APPROVE/REQUEST_CHANGES from the PR author account). Two inline findings need addressing.

Scope reviewed. Net diff against origin/feature/postgres-backend across all 11 changed files, plus the contracts the new code leans on: the reused publishStatements / activatedPublicationPayload / scanOwner / scanRevisionRow helpers, the isDeadlockError / waitForStoreRetry precedent, the lifecycle repository_publish.go classification arms, the Arango baseline activation guard, the compose gate in internal/server/runtime.go, the Arango-lane external-skip registry in scripts/run-arango-integration.mjs, and the TestZZ ClickHouse-gated mint inventory. The CI evidence cited in the PR (run 43411983672 at 4571021) matches the final head for all Go code; the two post-CI commits touch only the .mjs registry and the tests.yaml comments.

What holds up well. Single-transaction CAS with FOR UPDATE / FOR SHARE composite guards and byte-stable ledger negative batteries; crash-seam rollback proofs at three seams plus the single-winner race under -race; the replay no-restamp law asserted at both store and HTTP level; reuse of the audited publish statements keeps the second CAS shape-identical to the first; schema-pattern validation plus fully parameterized statements; DSN/password scrubbing and owned-namespace leftover audits; the new fail-closed compose term on runtime.Activate is consistent with existing gating because compose already requires the publication series through runtime.Bundles.

Required changes (see inline comments).

  1. .github/workflows/tests.yaml — the battery step pipes go test into tee without pipefail, so non-spine battery failures are masked; only the assert step can redden the job today.
  2. internal/backends/postgres/internal/explorer/activation.go — READY-only activation diverges from the Arango baseline (which admits FAILED candidates and revives them); with the lifecycle handler marking the candidate FAILED on every activation miss and retried POSTs converging on the same revision identity, a transient race can permanently pin a content identity to 409 on the PG spine, and no test covers retry-after-conflict.

Optional suggestions (not required). Reuse a single time.Now() stamp for revision activated_at and owner updated_at as the baseline passes one now; consider checking every materialization execution rather than only the first (the baseline also checks only the first, so this is a shared improvement); the pgExplorerStore activate field uses a narrow local interface while publish holds the concrete store type — consider unifying the field shapes; note that TestZZ mint inventory requires the whole battery whenever LOOM_CLICKHOUSE_URL is exported, which trips -run-filtered local runs; run pgCompAssertNoLaneLeak on the replay body too.

Comment thread .github/workflows/tests.yaml Resolved
matthewpeterkort and others added 7 commits November 13, 2026 11:14
…blish gate

The battery step ran the default runner shell (bash -e) with go test
piped into tee, so the step exit status was tee's: a failing or
panicking battery (composition legs, TestZZPgCompZeroLeftoverFixtures
audits) stayed green while the assert step inspected only the spine
PASS/SKIP lines, defeating the silent-skip-is-red intent of the gate.
Now set -euo pipefail before the pipeline, matching the Backend
ownership guard named gates and conformance arms steps in this
workflow. Proven: same false|tee pipeline exits 0 without pipefail and
1 with it under bash -e; workflow YAML reparsed and the rendered step
script inspected.
…itory-publish spine

The activation store admits READY candidates only (FAILED revival is
trigger-forbidden by migration 000006), so a lost activation race
permanently pins the content identity: the lifecycle marks the
candidate FAILED on every miss and a retried POST converges onto the
same deterministic revision identity. Arango recovers on retry
(READY-or-FAILED candidate filter); PostgreSQL must be proven to stay
refused, not assumed.

Route-level leg drives the production composition with the
DevRepositoryActivationConflictOnce lever: the conflict-armed deploy
lands 409 PUBLICATION_ACTIVATION_CONFLICT with the revision witnessed
FAILED (failed_at set, lifecycle diagnostics code present, owner
pointer unmoved); two retried POSTs each return 409
GENERATION_ACTIVATION_FAILED on the same revision identity with
byte-stable failed_at, diagnostics, and owner pointer (the failAny CAS
fence excludes FAILED, so re-marking is a typed no-op). The sibling
leg pins the ACTIVE replay arm idempotent: 200 activated, same
revisionId, no activated_at restamp, pointer stable.

The leg is inventoried in the ClickHouse-gated mint census, registered
in the Arango-lane declared-skip set, and gated by a second PASS-line
assertion in the pg-repository-publish job: silent-skip stays red.
…ning leg

The one-shot dev fault is consumed by every activation attempt and
corrupts the CAS input only when the project already has an active
release pointer (pinned by
TestDevRepositoryActivationConflictOnceCannotJumpToLaterRequest). The
leg armed a fresh project, so the armed deploy activated instead of
conflicting (CI run 43417949685, spine job 131860480059: 200 where 409
PUBLICATION_ACTIVATION_CONFLICT was pinned).

The leg now bootstraps a real active release with an unarmed runtime
(first deploy 200 activated, owner pointer witnessed on the bootstrap
revision), closes it, then composes the conflict-armed runtime and
publishes a second generation: the armed deploy loses the release CAS
and is witnessed FAILED with PUBLICATION_ACTIVATION_CONFLICT
diagnostics, failed_at set, and the owner pointer still on the
bootstrap revision. Retries stay 409 GENERATION_ACTIVATION_FAILED on
the pinned identity: the release CAS then advances normally (pointer
revision is read fresh), the deterministic claim replays the FAILED
revision unchanged (ON CONFLICT DO NOTHING + content-digest equality),
and READY-only eligibility refuses it — failed_at, diagnostics, and
owner pointer byte-stable across attempts. The sibling leg pins the
ACTIVE replay arm idempotent.
…ation store

fd2bfeb (#186) deleted the package-local schemaNamePattern and moved
the guard to pgstore.ValidSchemaName across every postgres adapter.
NewActivationStore validated through the deleted variable, so the
merged tree failed to compile the postgres explorer package, which is
what reddened the six dependent CI jobs at run 43419141395. The merge
hop brought the base tip; this adopts the centralized predicate so the
activation store obeys the same law as its sibling adapters.
…IVE revival

Migration 000012 CREATE OR REPLACEs loom_explorer_revision_guard with one
additional arm, following the 000011 precedent: same-named function
replaced whole, trigger dropped and recreated, convergence on rerun.
Migration 000006 stays byte-identical to base (sha256
3f6c44bff72d7837634e8c170f0b97e256e38d35f7c844fad27f4fbda38ecc09, zero
diff against origin/feature/postgres-backend).

The arm admits the FAILED->ACTIVE revival only in the activation shape --
failed_at cleared, activated_at set, diagnostics present, ready_at
unchanged -- so a retried repository publish converges onto the pinned
content identity with a 200-activated revival instead of the permanent
409 the READY-only law forced. Bare resurrections (keeping failed_at,
missing activated_at, nulling diagnostics) are refused with 23514, and
frozen-column smuggling during the transition still dies on the untouched
frozen check. This resolves the first PG-law divergence to Arango parity
under the coordinator option-(a) ruling; the aea0653 retry-pinning
outcome is superseded by that ruling.

Adapter: activation eligibility widens to READY|FAILED and dispatches
FAILED candidates to the new candidateRevive statement; candidateActive
stays READY-only, so the first CAS never touches a FAILED row.

Tests: transition probe renamed to TransitionGuardProbe with five
activation-shape cases (legal revival, sloppy failed_at, missing
activated_at, nulled diagnostics, frozen dataset smuggling); the
non-vacuity suite gains restored-law revival refusals and acceptance; the
composite-guard suite drops candidate-failed (now legal) and gains
revival-witness equality against a fresh activation, replay-no-restamp
after revival, a ready-only-first-CAS pin, and guard-refused revival
byte-stability; the race repins to never-strands. The route leg converts
from stays-pinned to revival-parity: the conflict-armed deploy still
lands 409 with the revision witnessed FAILED and the pointer unmoved,
then retried POSTs return 200 activated=true on the pinned identity with
diagnostics cleared, the pointer flipped, and no restamp on replay.

Ledgers: 000012 pinned by sha256 in the checksum registry, the state-02
pin map, the ownership manifest, and the activation control pins;
fixture counts and the membership convergence list move 11 -> 12; the
retroactive-refusal fixture derives its recorded max from the owner
history. Verified live on PostgreSQL 18.6: full explorer package,
probes, non-vacuity suite, race x5 under -race (30/30), store package,
lifecycle package. The route legs gate in CI behind the dual PASS-line
gate (ClickHouse is not reachable from this window).

Co-authored-by: AO <ao@local>
The revival arm admits a FAILED candidate, so the supersede discipline
must not silently tolerate a FAILED revision behind the owner pointer:
supersedePriorActivationRevision locks the prior and refuses any status
other than ACTIVE with explorer.ErrDraftConflict (000006 law, untouched
by 000012). The new composite-guard case fails the prior through the
production FailRevisionFrom CAS (ACTIVE->FAILED is legal), attempts the
activation, and proves ErrDraftConflict with revisions and owners
byte-stable: reviving the candidate is no bypass of a FAILED prior.

Verified live on PostgreSQL 18.6: composite-guard 20/20 PASS, full
explorer package ok (20.9s, zero skips).

@matthewpeterkort matthewpeterkort left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review: PR #184 at final head 4339cbb (second review round)

Verdict: CHANGES REQUESTED. Two inline findings; each is a required change. Posted as COMMENT because GitHub rejects APPROVE/REQUEST_CHANGES from the PR-author account.

Prior round: both findings are addressed. The battery step now runs under set -euo pipefail so tee cannot mask go test failures (verified at this head, together with the new dual PASS-line gate), and the READY-only eligibility divergence was resolved toward baseline parity: activation admits a FAILED candidate through a dedicated candidateRevive statement, and migration 000012 evolves the revision state machine with an activation-shaped FAILED-to-ACTIVE arm.

What holds up in the 000012 wave. Guard evolution follows the sanctioned 000011 precedent (CREATE OR REPLACE whole-function plus trigger drop/recreate, rerun convergence, 000006 byte-frozen under its existing pins, restored-guard non-vacuity now re-runs migration 12); the activation-shape arm is strictly narrower than the Arango baseline and its probe battery covers the positive case, all three sloppy shapes, frozen-column smuggling, and the still-refused bare resurrection; the 08a61bf6 checksum is consistent across all four pin sites and the 11-to-12 ledger-count updates are complete (pubauth, activation e2e, explicit-groups convergence, ownership manifest); the ownership-contract LatestRecord change derives the expectation from the seeded composition rather than hardcoding, preserving the refusal, byte-stable-ledger and row-survival assertions; the never-strands rewrite of the race leg is honest and forced by the new law (the SELECT FOR UPDATE serialization makes activation-nil deterministic in both orders, and the loser identity check for the failure side is retained); the spine retry-revival leg is well formed (armed one-shot dev fault via the pre-existing base knob and its CannotJumpToLaterRequest pin, witnessed 409-with-FAILED, pinned-identity 200 revival with cleared diagnostics, pointer flip, replay no-restamp, sibling deploy, lane-leak audit on every body); the skip registry, mint inventory and workflow gates all moved with the renamed test.

Required changes (inline).

  1. .github/workflows/tests.yaml — the final head has no CI acceptance: its only Actions run was cancelled with this acceptance job (and Go unit, Arango, pg-fixture) cancelled, so the migration, guard law and renamed legs have never executed on CI.
  2. internal/backends/postgres/internal/explorer/activation_test.go — the audited statement battery never pins the new candidateRevive statement; loosening its FAILED-only status filter would keep the whole suite green.

Optional suggestions (not required). The revival arm leaves OLD.activated_at unconstrained, so a direct-SQL ACTIVE-to-FAILED-to-ACTIVE round trip could retain a stale activation witness (the store path always restamps; READY-to-ACTIVE is equally loose today, so this is a strictness tightening, e.g. requiring activated_at to change). Reuse one time.Now stamp for candidate activated_at and owner updated_at as the baseline passes a single now. The execution guard still checks only the first materialization (baseline parity; shared improvement).

Comment thread .github/workflows/tests.yaml Resolved
The audited-statement battery covered every mutating statement of the
first CAS but not the revival statement: the e2e revival leg activates
only FAILED candidates (the row lock serializes the load and the CAS,
so loosening or dropping the status filter cannot be observed
behaviorally), and the ready-only pin probes the READY statement via
raw SQL. The candidateRevive entry mirrors candidateActive: SET status
ACTIVE, activated_at = $2, failed_at NULL, diagnostics = $3,
publication = $4, with WHERE id = $1 AND status = 'FAILED', plus the
one-statement count check.

Non-vacuity proven by mutation: deleting the status filter reddens the
battery (go test ./internal/backends/postgres/internal/explorer,
run TestActivationStatementsCarryTheCompositeGuards); restored green,
gofmt and go vet clean.

@matthewpeterkort matthewpeterkort left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review: PR #184 at head 152f59f (third review round)

Verdict: CHANGES REQUESTED — one remaining item, the terminal CI acceptance receipt. No code changes are requested. Posted as COMMENT because GitHub rejects APPROVE/REQUEST_CHANGES from the PR-author account.

Round-2 findings: both addressed, verified at this head. (1) The statement battery now statically pins candidateRevive — the audited SET clause and the WHERE id = $1 AND status = FAILED filter — so loosening the revival filter no longer passes the suite (the actual hole I flagged). (2) The acceptance-evidence gap is being repaired: the pipeline was re-triggered by the push, and run 43423381565 is in progress right now for this exact head — at review time 6 jobs already passed (Node unit, Python, Shell cleanup, Go ClickHouse, backend conformance, and the ClickHouse publication leg was green at last read) with pg-repository-publish, Go unit, Arango, UI, fixture-runner, and OpenAPI legs still running and no failure recorded. I could not wait for the run to finish from this review, so the receipt condition from my previous inline comment is still open — see the single inline finding.

New commits reviewed in full (11-line additive diff). candidateRevive pin in activation_test.go: clauses match the built statement in publication.go exactly, so the battery is green by construction; a FAILED-prior composite-guard case (prior-pointer-failed-candidate-revival-is-no-bypass) in activation_e2e_test.go: fails the seeded ACTIVE prior through the production FailRevisionFrom CAS (ACTIVE->FAILED is 000006-legal, pointer stays), attempts activation of the READY candidate, and requires ErrDraftConflict plus byte-stable revisions and owners — this correctly proves the revival arm did not widen supersedePriorActivationRevision, which still refuses any non-ACTIVE prior. Both are sound; no further test or code changes requested.

Optional, non-blocking. The battery still lacks the single-UPDATE count check for candidateRevive that I suggested alongside the clause pins (candidateActive has one); with the clause pins in place this is belt-and-suspenders only, since a concatenated second UPDATE would hard-fail the extended-protocol QueryRow in every e2e leg — worth adding for symmetry, not required. Earlier non-blocking notes stand: an OLD.activated_at-changed tripwire on the revival arm, one time.Now stamp for the revision/owner writes, and checking every materialization execution.

Once the green terminal receipt for the to-be-merged head is recorded (inline comment), this is ready.

Comment thread .github/workflows/tests.yaml Resolved

@matthewpeterkort matthewpeterkort left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review: PR #184 at head 152f59f (fourth review round)

Verdict: APPROVED — no changes requested. (Posted as COMMENT because GitHub rejects APPROVE/REQUEST_CHANGES events from the PR-author account; this body is the approval.) No inline comments this round.

The single open item from round 3 is closed, independently verified. The PR body now records the authoritative merging-head receipt, and I re-checked every claim against the provider API rather than trusting the text: run 43423381565 has head_sha exactly 152f59f, completed success, 15/15 jobs green (12 test + 3 image). The acceptance job pg-repository-publish 131878767182 completed success with every step green — including the pipefail-propagating battery step and the Assert-the-spine-leg-executed-and-never-skipped step, the dual PASS-line / any-SKIP-red gate, whose success is machine-proof that both TestPostgresRuntimeRepositoryPublishRouteActivatesThroughFinalFlip and TestPostgresRuntimeRepositoryPublishRetryAfterActivationFailureRevives printed real PASS lines and neither skipped. The body also honestly documents why the two intermediate runs (4339cbb, 652cc9a) were cancelled — superseded mid-flight by the concurrency group, not failed — which matches my own observation last round.

All prior findings remain closed at this unchanged head (no new commits since round 3): pipefail propagation in the battery step; the option-(a) revival law — activation admits a FAILED candidate through the dedicated candidateRevive CAS, migration 000012 evolves the revision guard with the strictly activation-shaped FAILED-to-ACTIVE arm (000006 untouched, 000011-precedent convergence, non-vacuity re-pinned), guard probes for the sloppy shapes and frozen-column smuggling, FAILED-prior strictness pinned as a composite-guard case, the candidateRevive statement shape statically pinned in the offline battery, the never-strands race rewrite, and the 152f59f test-only additions I reviewed line-by-line last round.

Optional, non-blocking (worker discretion, not required): a single-UPDATE count check on candidateRevive mirroring candidateActive; an OLD.activated_at-changed tripwire on the 000012 revival arm (the store path always restamps, so this only tightens direct-SQL law); one time.Now stamp shared by the revision activated_at and owner updated_at writes; checking every materialization execution instead of only the first (baseline parity).

The second-CAS activation spine, its guard evolution, and its acceptance lane are reviewed and sound at the head to be merged. Ready.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant