[matthewpeterkort]

Problem

observeObservationCodePivot (internal/catalog/write_profiler.go) called stat.setPivotDefaults(family, columnSelector, valueSelector) once per Observation document with that document's per-payload value arm (observationValueSelectorFromPayload: quantity > codeableConcept(text > coding[].display) > string > …). setPivotDefaults overwrote on every non-empty value (last-write-wins), so the stored pivot_value_selector was the arm of the last-arrived Observation document. Per-backend arrival order differs, the FieldEnrichment digest diverged, and every downstream identity chained from it (SnapshotIdentity.fieldDigest → snapshotToken → digestID idSeed → all candidate/node/edge IDs → published workspace bytes). Evidence: /tmp/opencode/diag-159-evidence/wave2 — exactly one of 92 field-enrichment rows diverged for the identical authored dataset: Observation/code.pivot_value_selector = valueCodeableConcept.coding[].display (Arango) vs valueString (PG). This aborted the #159 paired-benchmark prepare stage (matched-bytes abort, driver.go:296).

Fix

The profiler tracks the observed value-arm and column-arm sets in fieldCatalogStats (set-normalized, merge-unioned) and derives the stored pivot_value_selector / pivot_column_selector at export as the first arm present per observationValueSelectorFromPayload precedence (value: quantity → codeableConcept text → coding display → string → …; column: code.text → code.coding[].display). #159 fixture effect: both backends now store valueQuantity.value (both arms observed, quantity highest) → identical FieldDigest → matching prepare-stage bytes.

Same-class repairs found by the R2 audit (not just the one that burned):

• pivot_column_selector had the identical per-document arrival-order bug (code.text key presence); fixed via the same arm-set mechanism.
• pivot_candidate, pivot_kind, and the object → codeable_concept/coding kind upgrade were creation-order- and merge-order-dependent (e.g. a code:{} doc with a value arm creating the stat left pivot_item_resource_type/pivot_value_selectors empty, while a codeable-concept doc creating it filled them); they now converge monotonically (candidate OR, kind precedence observation > codeable_concept_display_value, plan-mirrored kind upgrade) in ensureStat and Merge.
• Observer-created stats now apply the default pivot spec so spec slots no longer depend on which document created the stat.

Documented as constant-per-(resourceType, path) (no change needed; all writers emit equal values, so non-empty-wins converges regardless of order): pivot_family (observer writes only the OBSERVATION_CODE_VALUE constant; spec agrees for Observation/code), pivot_item_source, pivot_item_resource_type, pivot_value_selectors (spec-only writers).

Consumer behavior delta (intentional)

Stored hints are consumed by graphql internal/api/graphql/graph/query/input_resolution.go:116 (pivot hints when the author left selectors nil) and internal/dataframe/recipe/schema/discovery.go (observedValuePath + pivot column/value selectors). For datasets whose last-arrived arm differed from the highest-precedence observed arm, the stored hint changes to the highest-precedence arm; for all other datasets it is unchanged. The highest-precedence value arm equals the DefaultPivotSpec default (valueQuantity.value first choice), i.e. the well-trodden spec-default resolution path.

Tests

• R1 (root-cause proof): four new tests in internal/catalog/write_profiler_pivot_determinism_test.go — full 720-permutation arrival-order independence over an all-three-arms mix, arm-precedence with reversed arrival (the exact Arango-vs-PG evidence arms: coding-display vs string → valueCodeableConcept.coding[].display), merge-direction stability, and creation-order/merge convergence (empty-code-with-arm vs cc-without-arm). All four fail at the pre-fix tip and pass post-fix.
• R4 (closes the banked conformance gap): new TestBackendArchitectureConformance/crossBackendCandidateIdentityParity ingests one identical authored dataset (Observations spanning valueString / valueCodeableConcept coding-display / text / valueQuantity / mixed / bare) into fresh Arango and PostgreSQL stores, then pins cross-backend equality of the snapshot token and the sorted candidate-ID, node-ID, and edge-ID multisets from the production builder route, with non-vacuity assertions (non-empty, Observation candidates present). Registered in the fail-closed CI event gate (fails if skipped despite both services) and in the offline skipset capture/allowlist (verifier green: 3 identities).
• R3 consumer regressions: go test -count=1 ./internal/..., ./cmd/..., ./conformance/... green offline (includes graphql graph/query, dataframe/recipe/schema, api/recipe, ingest merge fan-in, explorer capability/lifecycle). Live PG/Arango legs are adjudicated by the dedicated backend-conformance CI job (both services; this sandbox has no docker — shared-window leg available from the coordinator on request, queue empty).
• gofmt / go vet / go build ./... clean.

Program laws

Append-only history (new branch, no rebase/force/amend); new hunks carry zero handwritten comments; pinned paths untouched (no dual-ledger classification needed); no changes to PR #159, the benchmark harness, matcher law, or 000006/000012.

Unblocks #159 paired benchmark prepare stage.

Skip-set adjudication (settled explicitly per recovery instruction)

The parity pin is env-gated by design (option 2): cross-backend parity requires BOTH live services, so the test skips when either LOOM_TEST_POSTGRES_URL or LOOM_TEST_ARANGO_URL is absent — the honest offline state. It therefore legitimately appears in the skip-set capture alongside the two arm legs, and scripts/run-arango-integration.test.mjs was updated in commit 6cfb14c to pin the three-identity capture (capture.json + runner registry + test expectation agree in one append-only commit).

The silent-skip red law holds and was proven, not asserted: in the backend-conformance CI job (which boots both services and exports both env vars), the leg is in the fail-closed required-events list. checkGoEvents (scripts/run-postgres-integration.mjs, reused via scripts/lib/check-go-test-events.mjs) refuses, per required name, (a) a skip event, (b) a missing run event, (c) a missing pass event, and additionally asserts zero skips package-wide. Simulated locally: a synthetic event stream where crossBackendCandidateIdentityParity skips while everything else passes yields gate exit 1 with Required real integration case was skipped, a blocker and never acceptance. So if both arms boot and the parity test somehow skipped unnoticed, the job is red — the pin cannot pass vacuously. Offline, the same leg's skip is only tolerated under the frozen allowlist that backend-conformance-skipset-verify.mjs (CAPTURE_SHA256-pinned) verifies before the live arms run.

Handoff reconciliation (worker 204)

Adopted the root-cause diagnosis and evidence dumps. The delivered design supersedes the scratch documentPivotColumnSelect/documentPivotValueSelect firstNonEmpty slots with observed arm SETS + precedence derivation (equivalent outcome, plus merge/creation-order convergence the scratch patch did not cover). Their audit suggestions were executed: setPivotScope arms audited (constant-per-family spec writers, non-empty-wins converges), the Merge new-stat copy block audited (now merge-unions arm sets and converges pivot evidence), the column-selector same-shape bug (code.text arrival order) is fixed via the column arm precedence table, and rank order reuses observationValueSelectorFromPayload precedence exactly. The conformance gap they flagged (candidate non-empty only) is the R4 pin above. Consumer sites (graphql input_resolution.go, recipe/schema discovery, api/recipe discovery, graphql fieldrefs) are covered by the R3 regression runs; #159 untouched per handoff instruction.

Review thread PRRT_kwDOS-yV5s63IWlo (fixed in 10bddda)

The reviewer found the same class inside the kind lattice: kindUpgrade only upgraded from object, so a coding-vs-codeable_concept conflict for one path resolved to whichever shard/document arrived first, and Documents emits stat.kind verbatim into the stored profile and FieldDigest. kindUpgrade now honors a total-order lattice (codeable_concept > coding > object), shared by all three first-upgrade-wins sites — convergePivotEvidence, upgradePivotStat, and addFieldPlan (plan-layer same-path merge, a third site with the same guard). Kinds outside the lattice are untouched. Two new tests (merge both directions; document arrival both orders, asserting kind = codeable_concept) fail pre-fix and pass post-fix; thread replied and resolved; Loom CI run 43443343682 all-green at head 10bddda.

Adjacent same-class finding (separate unit, per coordination): semanticObservationKey excludes Status while mergeSemanticObservations/addSemanticObservation are first-writer-wins on the whole observation record, so two same-key observations whose shape-derived statuses differ (e.g. UNSUPPORTED_VALUE_PROJECTION for a coding-only valueCodeableConcept vs UNRESOLVED_SYSTEM for a text value, under one code) converge to the first arrival. Repro captured; not fixed here because the correct status-combination lattice is a semantic-schema product decision.

Kind lattice adjudication (10bddda)

kindUpgrade now enforces a single total-order upgrade lattice, codeable_concept > coding > object: an upgrade fires only when both the current and incoming kinds sit inside the lattice and rank(incoming) > rank(current), so the join is a commutative higher-rank-wins merge and therefore arrival-order-independent. One shared function governs all three sites where the old current != object guard let a coding-vs-codeable_concept conflict through: convergePivotEvidence (merge direction), upgradePivotStat (plan arrival inside one profiler), and addFieldPlan (plan-layer same-path merge — a third site with the same object-only guard, found by the recovery session beyond the reviewer thread).

Preservation argument (independently git-verified pre-fix): today object -> {coding, codeable_concept} already upgrades at all three sites, and array/string/scalar never entered the lattice because the old code only promoted when the incoming kind was codeable_concept or coding while the current kind was exactly object. Under the lattice the entry set is identical (rank(current) >= rank(object) && rank(incoming) >= rank(coding) admits exactly the same pairs, strictly ordered), so coding -> codeable_concept is the ONLY new transition, and it is the one the reviewer thread flagged.

Plan-layer pivotKind ruling (8e18343)

Audit of addFieldPlan's pivotCandidate branch found it performed an unconditional existing.PivotKind = pivotKind, a raw last-write-wins assignment, while the stat-layer counterpart convergePivotEvidence resolves the same conflict monotonically via pivotKindRank. Reachability analysis first: the branch is entered only by classifyObjectShape's codeable-concept arm, which always pairs pivotCandidate=true with pivotKindCodeableConcept; every other addFieldPlan caller passes (false, ""), and pivotKindObservation never reaches the plan layer because observeObservationCodePivot writes it directly onto stats. So under today's callers the unconditional write always assigns the same constant and cannot diverge.

Ruling: the latent hazard is removed rather than merely documented. The branch now applies the same monotone join as the stat layer, reusing pivotKindRank (observation > codeable_concept): higher rank wins; equal rank keeps the incumbent (today's only equal-rank pair is the identical constant, so behavior is unchanged for every current caller); an empty or unknown pivot kind (rank 0) can never demote a populated lattice kind. New test TestFieldCatalogPlanPivotKindConvergesAcrossMergeOrder merges conflicting same-path candidate plans through addFieldPlan in both directions and pins observation both ways; it fails on the pre-fix unconditional assignment and passes post-fix.

semanticObservationKey deferral (separate unit)

While pinning the kind-merge convergence, the fixture exposed an adjacent same-class finding that is deliberately NOT fixed in this PR: semanticObservationKey excludes Status, while mergeSemanticObservations and addSemanticObservation are first-writer-wins on the whole observation record. Two same-key observations whose shape-derived statuses differ — e.g. a coding-only valueCodeableConcept (UNSUPPORTED_VALUE_PROJECTION) and a text valueCodeableConcept (UNRESOLVED_SYSTEM) under one code — converge to whichever arrived first. Repro evidence: /tmp/opencode/kind-merge-fail.txt (status field diverges across merge directions on identical source). The correct status-combination lattice is a semantic-schema product decision, so it is tracked as the deferred ledger ticket semanticObservationKey-first-writer-wins carrying this repro pointer, not silently guessed here.

Empirical tripwire (coordinator): the #159 paired run after merge is the live check — wave-1 evidence proved semanticObservations are byte-compared in availability responses. If snapshot tokens or candidate IDs still diverge cross-backend, this unit is re-armed as a blocker.

[matthewpeterkort]

Review of head 6cfb14c (2 commits) against feature/postgres-backend.

The arm-based pivot-default mechanism is sound: observePivotArms union + deriveObservedPivotSelector precedence + Documents-time derivation make PivotColumnSelect/PivotValueSelect arrival-order independent, Merge unions the arms and rank-converges pivotKind, and the new permutation/merge-direction unit tests plus the crossBackendCandidateIdentityParity conformance leg are wired consistently through ab10MintInventory, the skip-set capture, the Node skip-set test, and the CI event gate (all four pin sites match).

Verdict: REQUESTING CHANGES. One finding, posted as an inline comment: the new kind convergence in convergePivotEvidence is not commutative for the coding-vs-codeable_concept conflict, so the stored profile Kind remains merge-direction dependent - the exact class of nondeterminism this PR exists to remove.

Notes that do not require changes:

• The conformance lane now runs three sequential heavy arms under a single go test -timeout=420s with a 480s outer timeout; the parity leg alone allocates a 420s context. Wall-clock headroom is unverified here, so watch the first CI run for timeout pressure.
• The CAPTURE_SHA256 bump in scripts/backend-conformance-skipset-verify.mjs could not be recomputed locally under reviewer command constraints; the offline verify step in the workflow will catch a mismatch, and the capture content matches the new subtest name at every pin site.
• The column selector now prefers code.text whenever any corpus document carries code.text (previously last-observed document won). This matches the schema preference in defaultCodeableColumnSelector; assumed intentional.
• The new pivotColumnArms/pivotValueArms maps are not charged to the retention budget, but they are bounded by the fixed arm vocabulary, which is acceptable.

[matthewpeterkort]

Re-review of head 8e18343 (two new commits since 6cfb14c: 10bddda total-order kind upgrades, 8e18343 monotone pivotKind join at the plan layer), diffed against the prior reviewed head and against base feature/postgres-backend.

Verdict: APPROVED. The prior required change is resolved and I have no remaining blocking findings; posting no inline comments.

How I verified the fix: kindUpgrade is now a rank-guarded lattice join (codeable_concept > coding > object, with scalar/array/unknown excluded on both sides) applied uniformly at the stat layer (convergePivotEvidence, upgradePivotStat, pivotStatNeedsUpgrade) and the plan layer (addFieldPlan). I checked all pairs in the object/coding/codeable_concept lattice: the result is max under the rank order, so it is commutative and associative, and the previously divergent coding-vs-codeable_concept merge now converges in both directions. pivotKind now uses the same rank-max join at both layers, which also removes the prior plan-layer demotion (existing.PivotKind could be clobbered with an empty kind). All other merge inputs are commutative joins: arms union, pivotCandidate/pivotSpecApplied OR, and family/itemSource/valueSelectors are DefaultPivotSpec(resourceType, path)-derived under the enforced profiler identity, so first-non-empty is unobservable. Documents() derives selectors from the arm union under fixed precedence, direction-independent. pivotStatNeedsUpgrade uses the same kindUpgrade predicate, so the upgrade is idempotent and cannot oscillate.

The three new tests (TestFieldCatalogKindConvergesAcrossMergeDirection, TestFieldCatalogKindConvergesAcrossDocumentArrivalOrder, TestFieldCatalogPlanPivotKindConvergesAcrossMergeOrder) cover exactly the previously broken scenarios and would have failed at 6cfb14c; the rest of the branch is unchanged since the prior review and the skip-set/workflow pin sites remain mutually consistent.

Notes (not blocking):

• A genuine same-path shape conflict across shards (scalar or array kind in one shard, codeable_concept in another) still resolves first-writer-wins, because rank-0 kinds are excluded from the join on both sides. That is pre-existing base behavior rather than a regression from this PR, and it is outside the pivot-defaults contract, but it is a natural follow-up if kind convergence should extend to true shape conflicts.
• The plan-layer test synthesizes pivotKindObservation at the plan level, which production plan classification never produces (only the stat layer sets it); harmless, but worth knowing it is a synthetic scenario.
• Carried over from the prior round and still unverified locally: CAPTURE_SHA256 in scripts/backend-conformance-skipset-verify.mjs (the workflow offline verify step gates it), and CI wall-clock headroom for three sequential arms under the single 420s go test budget. Tests were not executed during this review per reviewer constraints.