Repository navigation
diag: paired catalog capture #159 (throwaway - DO NOT MERGE) - #187
Closed
matthewpeterkort wants to merge 48 commits into
Closed
matthewpeterkort wants to merge 48 commits into
matthewpeterkort wants to merge 48 commits into
Conversation
…chmark Adds internal/dataframe/benchmark: the offline-verifiable core of the two-backend dataframe benchmark (ticket pg-benchmark-arango-postgres-dataframe). - fixtures.go: deterministic seeded FHIR NDJSON generator (Patient/ Observation/Specimen/Condition/ResearchStudy) with facts sidecar and sha256 manifest digest; tiny/small/representative presets; designed-in duplicate contributors, null/missing values, selective filters. Fixes during testing: metastatic specimen stride moved to the global cursor (per-patient index could never reach it, making the filter vacuous). - canonical.go: total type-tagged canonicalization; null vs missing, bag multiplicity, numeric spelling equivalence; explicit storage-ID -> logical-ID mapping so backends are never compared on storage identity. - gates.go: correctness gates BEFORE timing: oracle rows, cross-backend bag equality, column contract, row count, completeness (truncated is never complete), negative auth (denial needs a non-empty baseline and errors are not denials), arm outcome publication. No path from an empty or absent result to a pass; unsupported cases are published, never hidden; empty matrices are refused, not passes. - oracle.go: independent tiny-preset oracles derived from Facts with plain Go arithmetic (no planner/compiler/storage involvement). - measure.go: warmup exclusion, interleaved rounds, nearest-rank p50/p95 with declared method, failed samples kept in raw and excluded from statistics, unavailable metrics never zero, measured throughput windows, injectable clock. - matrix.go: closed workload battery with column contracts, categories, and per-entry oracle-consistency validation. - report.go: report.json schema whose every derived number is recomputed from raw evidence by VerifyReport; controls (stability CV limit, self-ratio, empty floor, ratio-only-both-passed) are mandatory; passed runs require claims and both arms' evidence. The 18-case forgery suite (report_test.go) proves each tamper path REDs (ruling ii). Offline validation: go test -count=1 ./internal/dataframe/benchmark (all pass), go vet, gofmt clean. Live Arango/PG legs, harness wiring, and the node orchestrator land in following waves; PG window not yet granted. AO native adversarial review is pending the PR (coordinator-gated).
Second wave of the two-backend dataframe benchmark core. - provision.go: spawns one loom process per arm from generated config (ClickHouse disabled both arms per the matched-reader ruling; calypr auth against a mock fence), waits for readiness, and owns restarts so the process-cold cache label is only ever earned, never claimed. Arm logs pass through a secret-scrubbing writer (connection strings never reach evidence). MockFence implements the verified authscope contract: GET /user keyed by bearer JWT sub, iss-bound tokens, read/* grants; requests counted for the control's evidence line. - harness.go: public-route client only (multipart dataset upload with auth_resource_path, explorer-config publication, authoring/v2 preview execution). Preview results canonicalize through the shared bag path and carry sampled=true through, never silently. MeasureArms runs the bound protocol: warmup phase, fixed-order interleaved round-robin so drift hits both arms alike, failed samples kept raw with identity and timing. NegativeAuthControl is the pre-timing gate using the production denial shape (restricted intersection -> zero rows). - harness_test.go: offline fake arms prove route usage, digest path, interleaving order, failure retention in raw samples, the full negative-auth rule incl. leak detection, fence grants, bearer shape rejection, log scrubbing, and the both-arms ClickHouse-disabled config. Offline validation: go test -count=1 ./internal/dataframe/benchmark (all pass), go vet, gofmt clean. Live legs still gated on the PG window; execution-route deviations (receipt acquisition, preview pagination) are explicit live-rehearsal items to reconcile against source at run time.
Go-side authority for the benchmark artifact (ruling ii): - verify --report re-derives every number via benchmark.LoadReport; proved end-to-end: a valid report verifies OK, a median-tampered report is REJECTED naming the cell that fails to recompute. - run fails closed (exit 3, blocked) until the granted exclusive window wave wires preflight + the live driver; no partial environment can emit a success-shaped artifact.
Author one ExplorerBuilderWorkspace v2 document per workload using only
wire shapes copied from testdata/acceptance/ncpi-tcga-brca/workspace.json
(field columns, aggregate COUNT/SUM/MIN/MAX/DISTINCT_VALUES with where
{path,equals}, extensionByUrl, fixedFilters {column,values}). This makes
the live flow four public routes with zero builder-command payloads:
explorer-config publish -> create explorer from source (draft = active
revision) -> reconcile (compile receipt) -> preview (source-row
execution, limit <= engine max).
AuthoringVerification() asks the repository's own decoder about every
template before any service is contacted: rejected shapes ship as
planned/not_run with the decoder error as the named capability (same
honesty mechanism as the CI-targeted Arango rows); shapes without
acceptance precedent are labeled unverified-shape for rehearsal
classification. OutputIDFor maps workload ids to decoder-legal output
ids (underscores).
Offline: all 10 templates DecodeWorkspace clean incl. fixedFilters and
route child shapes; full package test suite green, vet and build clean.
…arkdown
postgres-only worker sessions must say so in a way the verifier enforces,
not by silent absence:
- RunPlan {scope, legs[], plannedNotRun[]} stored in report.json;
ValidateLegPlan refuses not_run legs without a named capability,
postgres-only scope with a run arango leg, and unknown scopes/modes.
- VerifyReport now recomputes status via RunStatusScoped and enforces the
un-run-leg contract: arm blocked in every gate report, disclosure row per
workload, zero samples/summaries/claims, no version, self-ratio exactly
0 (never a forged 1.0); run legs keep live-version and 1±0.05 self-ratio
requirements. Disclosure rows for a leg that ran are also refused.
- A postgres-only pass is legitimate with zero speed claims (it certifies
correctness+timings, not a comparison); both-arms passes still require
claims and both arms' evidence.
- RenderMarkdown emits report.md from verified content only: leg table with
capabilities, disclosure rows, controls with plan language
(ratio-only-both-passed, CV ceiling, self-ratio, empty floor), gate and
timing tables, claims or the named reason there are none.
- armplan_test.go: honest postgres-only round-trip + 8-case forgery suite
(faked passed arm, smuggled samples, dropped disclosure, blanked
capability, forged version/self-ratio, injected speed claim, scope lie)
plus the converse disclosure-for-ran-leg case.
All 10 battery workloads still decode; package tests, vet, build green.
ControlRuns values now have producers the verifier can independently recheck: - ControlUnstableCells: warm passed samples per workload/backend; CV over the ceiling (0.35) marks the cell unstable; fewer than 3 samples is unstable, never silently trusted. Warmup and failed samples excluded. - ControlSelfRatio: backend-vs-itself across halves of the (round, iteration) sequence, worst deviation across cells (>=1); ok=false when a cell cannot be split, so a missing control cannot forge its way past the verifier's 1±0.05 bound. - ControlEmptyFloorMillis + Arm.MeasureEmptyFloor: median of an identical request whose expectation is zero rows; any rows back is a protocol error (the control stopped being empty). Harness gains the four public authoring routes, matching openapi_explorer_direct.go and the reconcile/preview direct handlers: CreateExplorer (POST /explorers, sourceExplorerId=default so the draft equals the active revision), FetchBuilderState (GET authoring/v2/builder -> draftVersion/draftDigest/catalog.snapshotToken; refuses a missing token), Reconcile (POST authoring/v2/reconcile -> receiptId), and AuthorWorkload composing publish -> create -> builder -> reconcile. Offline tests pin route order, the draft token round-trip (stale token rejected), missing-token refusal, and the empty-floor non-empty guard. Full package suite green.
… per-scope reports RunBenchmark orchestrates the full matched run against public routes: - arms attached (BaseURL) or spawned (ServerSpec via injectable SpawnFunc; process-cold restarts only earned from real restarts); - dataset upload, per-workload authoring through the four-route flow, with an abort when StableExplorerID diverges across arms (matched comparison is invalid); - pre-timing authorization gate (ruling iii) on the representative workload, including a single-arm variant; a leak aborts before any artifact exists; - interleaved MeasureArms timing; MeasureArms now skips non-participating arms instead of panicking; - gate evaluation via the real evaluators (test seam only), oracles and storage-ID mapping from the same deterministic facts; - control runs: CV stability, self-ratios for run legs only (0 = absent for unrun legs), empty-result floor from an authored zero-row workspace (BuildEmptyFloorWorkspace) whose non-empty response is a protocol error; - live versions are mandatory from VersionProbe for every run leg; the report only exists if BuildReport's internal VerifyReport passes. Offline driver tests pin: both-arms pass with claims + control values, postgres-only pass with zero claims/samples/version/self-ratio for the unrun leg, auth-leak abort before artifact, divergent-explorer abort, source-commit validation, using the injectable scripted clock (host jitter tripping the self-ratio control in-test is the rule working as designed). Full package suite green, vet clean.
… probes benchcli run drives RunBenchmark end to end and writes report.json, report.md, and arms.json (operational manifest: window note, arm evidence, fence request count). Exit 3 = refused to run, exit 1 = ran and failed. Preflight refuses before any work unless: - LOOM_BENCH_LIVE=1 (coordinator-granted exclusive window) and a --window-note recording granter and time; - --source-commit is the full 40-hex SHA; - --out has no existing report.json (no silent overwrite); - scope/flag coherence: postgres-only must not receive Arango flags; both-arms requires both arms; attached arms require both tokens; all participating arms are attached or spawned, never mixed (an attached server's trusted issuer is not the benchmark fence); - namespace guard: PostgreSQL database and Arango database must be under loom_bench_* — a shared instance can never be written by a benchmark run outside its namespace. DSN error text is scrubbed. Spawned arms: one MockFence is the sole auth authority; the scoped principal is granted only the decoy path, so the negative-auth control exercises the production restricted-intersection denial. Configs render via PostgresBackendYAML (six-series ladder spelled out — without it the server first-query-legs and authoring routes would silently be absent) and ArangoBackendYAML; DSNs are scrubbed from arm logs. Version evidence comes from the live services: pgx SELECT version() and ArangoDB /_api/version (shape-checked), never hand-written. CLI smoke checks: every preflight refusal exits 3 with a named reason; namespace guards refuse loom_test and prod_db.
…sclosed not_run benchmark-backends.mjs is the node-side orchestrator; verification authority stays in Go (benchcli verify), so node tests can never satisfy a live check (ruling ii): - --verify-only --report: exits 0 only when the Go verifier says OK; prints the plan's evidence line check live-two-backend-correctness. - --run --out <dir> -- <benchcli flags>: drives benchcli (which fails closed without the granted window), then verifies the produced report.json before reporting pass. - Exit 3 from benchcli is disclosed as not_run with the preflight reason (run stays planned/not_run), never a pass, never a silent skip. - --harness-selftest: node-only sanity, clearly labelled. - --benchcli <path> runs the built binary; the default invokes go run. Hermetic node --test suite (9 tests, no network, no Go): arg parsing, invocation shapes, exit classification, the not_run disclosure path, verify-only pass/fail authority, run-then-verify ordering, report summary fields. Negative e2e: a hand-crafted report is REJECTED by benchcli and the orchestrator reports the live check as fail.
…e bench namespace The server's full runtime opens with SetupNone (the factory's arrangement law: the caller arranges schemas, migrations belong to setup). setup-pg follows the canonical live-composition path: CREATE DATABASE (refused if it exists: fresh per run keeps evidence attributable), CREATE SCHEMA per assigned series under a bench-prefixed name, then the public factory with SetupMigrate. drop-pg force-drops only bench-namespace databases. Admin DSN comes from LOOM_BENCH_ADMIN_DSN (never argv, never echoed); all error text is credential-scrubbed.
…n the _series suffix)
…d arm configs The server refuses to configure without a >=16-char population_mapping_cursor_secret. BuildServerConfigYAML now emits one from a caller-supplied per-run value that the deployer also adds to the log scrub list.
… config --config makes the YAML authoritative for server.listen, so the earlier --listen argv was ignored and the arm bound :0 behind our back. The deployer now renders ListenPlaceholder and the driver substitutes the FreePort-selected address before spawning.
Dataset ingest and explorer publication authorize with PermissionWrite; a read-only grant list 403s every upload. A benchmark grant is a real principal grant in both directions; the negative-auth scoped principal still holds no grant on the dataset path at all.
Pre-window disclosure per the loom-experiment-2 ruling: benchcli run --scope planned writes a fully-disclosed artifact (both legs not_run with named capabilities, every workload blocked, zero controls, samples, summaries, claims) and exits 3; it refuses every live flag, needs no window, and the live driver refuses planned scope outright. - armplan.go: RunScopePlanned, PlannedPlan, PlannedGateReports; ValidateLegPlan requires both legs not_run WITH named capability. - report.go: VerifyReport branches on run-leg count — a report where nothing ran must have wholly absent control values (stolen limits, borrowed floors, and forged self-ratios RED), while any run leg keeps the full controls requirement. - cmd/benchcli: planned mode, per-backend disclosure reasons, shared writeRunArtifacts so planned and live artifacts share one writer. - scripts/benchmark-backends.mjs: classifyPlannedDisclosure enforces skip-capture discipline (any sample/summary/claim in a planned artifact FAILS the check) and still routes the artifact through the Go verifier for authority. - README for the package; runner pointer in scripts/README.md. Live legs cite finding PG-PG-AUTH-PATH-DISCOVERY-ALLGENS (scoped calypr ingest blocked on postgres; fix owned by arch-backend-09). Validation: go build ./... and go vet clean; offline package suite green with 42 forgery-RED cases (14 new planned paths); node tests 12/12 in scripts/benchmark-backends.test.mjs and 167/167 across scripts/*.test.mjs; planned artifact produced by the CLI passes benchcli verify.
…ployer refusals The cmd/benchcli package had no test file, leaving the fail-closed CLI surface unpinned: loom_bench_* namespace guards, credential redaction in error text, DSN parsing, admin DSN env handling, battery disclosure totaledness, artifact writing, and deployer topology refusals. - guardPostgresNamespace: benchmark db passes; loom_test/postgres/ prefix-lookalike refused; error text never carries the password. - parsePostgresDSN / parseArangoDSN: full field split, default port, scheme/database/namespace refusals, no credential leakage. - benchDatabaseRe: accepts loom_bench_<4..48>, rejects loom_test, postgres, case-violating and short names. - setupPG schema-prefix guard and withDatabase URL-form law are checked hermetically (refusal happens before any connect). - planNotRunMatchesBattery: complete plans pass, dropped rows and rows for ran legs fail. - writeRunArtifacts: the three files land and report.json re-verifies through LoadReport/VerifyReport; arms.json stays valid JSON. - deployer: mixed topologies, missing tokens/dsn/binary, and caller tokens against the fence all refuse; the valid spawned shape wires the fence issuer into the config, mints distinct tokens, scrubs the DSN from arm logs, and grants the scoped subject everything EXCEPT the dataset path — verified through the fence's real /user surface. Offline only; scripts/** untouched per the runner-serialization ruling. go build/vet clean; benchmark + benchcli packages green (12 new test functions); node suite still 12/12.
…LLGENS The two scratch proofs the fix (513b052, PR #156) cites were held in the session artifacts dir but absent from the pushed head; verifier 208 flagged the gap and the coordinator ruled them ticket evidence to keep in-repo. - scratch_authrepro_test.go: hermetic unit-level branch discrimination against a real ScopeResolver + FenceUserAccessClient + MockFence. Variant A (bearer principal, matching grant, empty lister) proves the harness token/issuer wiring is correct by PASSING AuthorizeWrite; nil-principal and header-less variants pin the 'outside caller scope' shape the fake-based driver tests rely on. Runs always. - scratch_liveupload_test.go: live proof. Opt-in by substrate (skips without the scratch DSN file; binary path overridable via LOOM_BENCH_SCRATCH_BINARY). Pins TWO-WORLD CONSISTENCY rather than defect-era absolutes, so it does not false-red after the fix: the HTTP upload outcome and the store-level AllGenerations listing must agree (defect = both refuse, fix = both succeed); disagreement is substrate drift and fails. This is what caught the defect live: fence hit exactly 1x, then 403, while the listing refused. Behavior-only assertions: no coupling to the normalizer's output shape. Offline green: both pass here (defect-era binary + library = consistent refusal world); CI skips the live probe without the scratch DSN file.
…tatus The first consolidated live run refused activation: GENERATION_LOAD_INCOMPLETE with validation_errors=32 and Patient/Condition committing zero rows, while Observation/Specimen/ResearchStudy loaded. The count was exactly the tiny preset's 12 patients + 20 conditions, and the server-side row builder (internal/ingest loadRowGeneratedInternal) showed two fixture defects, both harness bugs rather than backend behaviour: - HumanName.family is a single string in FHIR R4; the generator emitted a JSON array, so every Patient line failed decode. Each family name now gets its own name entry, so name[].family still yields every family a patient carries and DISTINCT-vs-ALL stays selective. The given list rides on the first entry only, so name[].given[] cannot duplicate a given value across entries and the sort-limit oracle keeps exactly one row per patient. - Condition.clinicalStatus is required (1..1); it was never emitted, so every Condition line failed Validate(). It is now a constant 'active' coding on the standard condition-clinical system. No workload predicate reads it, so it stays out of every oracle. Add TestGeneratedFilesAreValidFHIR, which decodes and validates every line of the tiny and small presets through the same generated concrete FHIR structs the loader uses, plus the non-blank and non-empty-id checks the loader applies. The client-side preflight does not run this validator, which is why the defect reached the server: this test closes that gap offline instead of discovering shape errors during a live window. Facts, oracles, and canonical comparison are unaffected: they read PatientFacts and ConditionFacts, not the documents. Offline suite green (benchmark + benchcli + node 12/12); go build and vet clean.
The consolidated live run cleared generation upload and stopped at
publication: POST /api/v1/projects/{project}/generations/{generation}/explorer-config
returned 403 {"error":"forbidden"}. Cause: publishRepositoryExplorerConfig
calls AuthorizeWrite with the authResourcePath taken from the query
parameter, and with scoped authorization enabled AuthorizeWrite rejects an
empty path outright ("auth_resource_path is required when scoped
authorization is enabled", internal/authscope/scope_resolver.go). The
benchmark harness only sent the path as the upload form field, so every
durable write - publish, create-explorer, reconcile - was unreachable under
a scoped deployment. The run reached publish first because publication
precedes explorer creation in the authoring sequence.
Add Arm.AuthResourcePath, set from the driver options at both arm
construction sites, and apply it as a query parameter on the three write
routes through Arm.scoped. Read routes are untouched: preview authorizes via
authoringReadDirect, and an empty path sends no parameter so unscoped
deployments are unaffected (an unrestricted principal ignores it for writes,
and a spurious scope would only narrow reads).
TestWriteRoutesCarryAuthResourcePath asserts the parameter per route and
asserts its absence for an unscoped arm; it is RED with the scoped() helper
short-circuited, so a regression cannot pass silently. Offline suite green
(benchmark + benchcli + node 12/12), go build and vet clean.
…ve probe Two live-window defects in the probe itself, both found while running the granted Phase B session at server 513b052: - The server resolves --schema relative to the CWD (default schemas/graph-fhir.json), and SpawnServer inherits the spawner's directory. The probe runs under go test (package dir), so generation load failed with stat schemas/graph-fhir.json: no such file or directory and the upload returned 500 INTERNAL_ERROR (fs.PathError, request_id 29da9ff4). benchcli run only worked because the runbook launches it from the repository root. Add ServerSpec.WorkDir, honored only when set, and have the probe pin the repository root (auto-discovered by walking up for schemas/graph-fhir.json, LOOM_BENCH_SCRATCH_WORKDIR override) so probe and benchcli spawn identically regardless of caller CWD. - The probe uploaded samples.ndjson, which was never a loadable resource name; the 403-era probe never reached load, so nobody noticed. After the auth fix the file reaches the loader. Upload the real GenerateFiles fixtures instead, which is the path the benchmark actually measures. Probe outcome against loom-server-513b0520 (fixed) from this pre-merge checkout (library pre-fix): upload accepted (fence requests=1) while the store-level AllGenerations listing still refuses -> surface drift, correctly reported by the consistency pin. It goes green when 207's fix lands at this branch's base.
…an excluded plan The publication chain is ClickHouse-backed end to end: publication.Target has exactly one production implementation (ClickHouseBundleStore) and POST .../explorer-config materializes through it before the revision that seeds the authoring draft persists. Publish, authoring, reconcile, receipt previews, workload gates, timing, and controls therefore share one gate behind the service (coordinator ruling 2026-11-13: no ClickHouse chasing in the container; non-publication stages must go green). RunPlan gains a checked Publication declaration. --publication=excluded (default) runs spawn, upload, and version probes for real and emits a report where VerifyReport refuses any samples, summaries, claims, or control values, requires every gate arm blocked, and one disclosure row per battery workload per ran leg under the capability CH-<CI pin>-unavailable-in-container derived from ClickHouseCIVersion (tracked against the clickhouse/clickhouse-server pin in .github/workflows/tests.yaml). --publication=run requires --clickhouse-enabled; a ClickHouse-disabled identity claiming the chain ran no longer verifies. Driver protocol fixtures now state their ClickHouse posture explicitly instead of relying on a hardcoded flag. Validation: go test ./internal/dataframe/benchmark/... green including the new publication_test.go (honest round-trip, driver excluded protocol, 8 forgery RED cases, CI-pin drift guard); node orchestrator 12/12; go build and vet clean. Runner freeze respected: no scripts/** changes.
…ublic postgres port
Two CI failures on the benchmark branch:
1. scripts/package_audit.py refused the new packages (Python tests lane).
Curation added for internal/dataframe/benchmark and its benchcli
entry point; docs/PACKAGE_AUDIT.{csv,md} regenerated. package_audit_verify
passed for both locations; the measured-importer column updates are the
new packages appearing as importers, as expected. Not runner territory:
no CI selector or orchestrator files touched, runner freeze respected.
2. TestPostgresFactoryOwnershipContract fired: production files import
the PostgreSQL driver outside the register (internal/backends/ is the
driver's only legitimate production resident). benchcli's version
probe and setup-pg/drop-pg imported pgx/v5 directly. The contract test
is arch-backend-09's; the fix stays in the benchmark:
- probePostgresVersion now opens the public factory (no schema
assignment, SetupNone) and runs SELECT version() on a connection
acquired from Runtime.ConnSource(), the factory's public
single-connection port. Zero jackc imports remain in the package.
- setup-pg/drop-pg run their DDL on the same port; identifiers pass a
closed regex before SQL-standard double-quote rendering.
- DSN parsing moved from pgx.ParseConfig to net/url; the lane already
mandates URL-form DSNs (guards suite unchanged).
Validation: TestPostgresFactoryOwnershipContract green; go test
-count=1 ./internal/dataframe/benchmark/... green; python suite 48+14
OK; node 12/12; go build/vet clean. Live smoke against the test PG
lane: setup-pg created database + six-series ladder and drop-pg
removed it (exit 0 both), and the probe returned the full
PostgreSQL 18.6 version string. The internal/ci vet-gate failure
reproduces on clean base faffab3 in this sandbox (TMPDIR relpath
quirk), so it is environmental, not this branch.
…o/backend-benchmark/pg-benchmark-dataframe # Conflicts: # docs/PACKAGE_AUDIT.csv # docs/PACKAGE_AUDIT.md
…reasons Addresses review r4559430560: derive-age-band was unpassable by construction (the workspace authors DISTINCT_VALUES over raw valueQuantity.value while the oracle emits band labels; fixture ages [1000,10000) make the young/middle branches dead), and sort-limit-topn carries an unproven first-only assumption over multi-valued name[].family. Live battery law: a live leg runs exactly the authoring-proven workloads. Registry-excluded workloads are never authored, previewed, or timed; they ship as blocked gate rows with the LiveBatteryExclusion marker, one disclosure row per running leg naming the exact defect, and zero timing evidence. VerifyReport ties marker, disclosure text, arm posture, and sample-freedom back to LiveBatteryExclusions(); RunStatusScoped skips disclosed exclusions; benchcli's plan matcher and both report fixtures follow the same law. The pre-timing authorization gate moves to a proven workload (group-count-by-patient) since filter-active-gender is now registry-excluded. Validation: go test ./internal/dataframe/benchmark/... green (both packages, incl. 8 new live-battery forgery subtests and TestLiveBatteryExclusionsRegistry); node scripts/benchmark-backends.test.mjs 12/12; go build/vet clean.
…ew r4559430564/568) MeasureArms labels process-cold only on the sample that directly follows that arm's own successful restart; attached arms (no Proc) are labeled service-warm regardless of the peer being spawned; a failed cold restart records a labeled error sample at its position with Round/Iteration set. Controls split into passedStable (claims + CV, steady regimes only) and passedMeasured (round-half self-ratio across all regimes, so a spawned timing leg never aborts for lack of cells). buildClaims and VerifyReport share claimCellPair: warm preferred, service-warm fallback, never process-cold. Tests: TestMeasureArmsColdLabelContract spawns a real sleeper process as the arm's process owner and pins restart counts, per-round cold-first labeling, and service-warm attached arms; TestMeasureArmsFailedColdRestart IsLabeled pins the failure-sample contract.
Age observations now carry a days identifier under urn:loom:benchmark:age-days alongside valueQuantity; the fixture domain is [1,90] days so the corpus straddles both band thresholds. The workspace is Observation-rooted and derives age_band via EXACT_CATEGORY_RECODE with an exhaustive map generated from the shared AgeBandFor() the oracle uses, so author and oracle cannot drift; unknownPolicy is ERROR and the fixed filter selects the three band labels. The workload stays authoring-unverified (recode has no acceptance precedent) and remains a disclosed live-battery exclusion until the ClickHouse lane rehearses it; TestDeriveAgeBandOfflineWitness proves the authored shape reproduces the oracle bag offline against the uploaded fixture.
…de, live exit contract Operator acceptance amendment (2026-11-13): the benchmark must actually compare the two real drivers. The claim that timing sat behind ClickHouse was stale - the backend-conformance lane runs the whole authoring battery on both real drivers with clickhouse.enabled=false, and the PostgreSQL backend forbids clickhouse.enabled in this slice. What sits behind the service is only the dataset->bundle->ClickHouseBundleStore chain. - PublicationPlan gains the timed mode (Battery): chain disclosed not_run under PublicationCapabilityTimed while the battery - authoring, preview, timing, gates, controls - runs for real. VerifyReport ties timing evidence to the flag: flipping battery off under a report that carries samples is refused (forgery test). - benchcli runLive now classifies by report status + measured-sample presence (finding 4): passed -> 0; measured-but-failed -> 1; non-passed zero-timing (the default publication=excluded shape) -> 3 with the status reasons printed - disclosed not_run, never the success shape of exit 0. Pinned by TestClassifyLiveExit. - scripts/benchmark-backends.mjs gains --paired: discovers both services from LOOM_TEST_ARANGO_URL + LOOM_TEST_POSTGRES_URL, runs the matched battery through benchcli (spawned arms, benchmark fence, timed mode), hands authority to the Go verifier, then fails unless BOTH drivers ran and each carries >=5 non-warmup passed timings; writes comparison.md with median/min/max/range per workload per driver plus versions, source SHA, fixture digest, and warmup/cache/rounds/concurrency conditions. Missing services, planned/not_run/excluded arms, and zero timing are hard failures, never skips or passes (20 node tests). - tests.yaml registers job factory/paired-driver-benchmark on the backend-conformance service pattern (arangodb:3.12 + postgres:18 in one job): builds loom-server + benchcli from the SHA, runs the exact paired command with --iterations 5 --warmup 1 --rounds 5, prints the comparison table in the log, and uploads the artifact set. The job goes red rather than report scaffolding as a comparison. - README: exit-code table, the three publication modes, the paired lane, and the corrected chain-vs-battery ClickHouse claim.
CI run 43366558961 of the new factory/paired-driver-benchmark job proved the lane end to end up to the first real posture the PostgreSQL backend takes: readiness answers schema_missing -- the caller owns schema lifecycle. The Arango arm reached ready 200; the Postgres arm could not, so the check correctly went red with no comparison table. The paired run now bootstraps like the server's live composition fixtures do: when LOOM_BENCH_ADMIN_DSN is set it runs benchcli drop-pg (best effort) and setup-pg for the DSN's loom_bench_* database -- database, six-series schema ladder, migrated tables -- before spawning arms, and fails the check before any timing attempt if setup-pg fails. The workflow supplies the admin DSN and lets the script own provisioning so the documented command is the whole lane. Verified against the failed run's artifact: the arm log's 503 reason and the rendered config were byte-checked against PostgresBackendYAML's six-series contract.
scripts/tests/test_ui_ci_workflow.py pins every tests.yaml job to timeout-minutes: 10; the new paired job shipped at 20 and failed the retained Python contract. The job's own measured wall time (Go builds, provisioning, both services, the tiny battery) sits well inside ten minutes -- the 20 was unmeasured cushion, not evidence.
CI run 43367221616 proved the fail-closed provisioning gate: benchcli
answered 'database loom_bench_ci is outside the benchmark namespace'
(its rule is loom_bench_[a-z0-9_]{4,48}) and the check reported the
setup failure before any timing attempt, exactly as designed. The lane
had been inventing its own looser loom_bench_* prefix rule; it now
mirrors the benchcli regex, and the workflow's postgres DSN names
loom_bench_paired. A test pins the CI-unlegal two-char suffix case.
…ers no paths CI run 43368609701 of factory/paired-driver-benchmark was the first upload ever attempted against a spawned arm, and the Arango arm refused it: POST /api/v1/datasets/loom-bench/generations/... returned 403 FORBIDDEN in 1 ms with no fence involvement in the arm log. Root cause is a chicken-and-egg at the first authorized write. The authscope F-B27 project-binding gate asks DiscoverExistingAuthResource- Paths which paths the project has recorded before ingest can bootstrap fhir_field_catalog -- that collection is created only by the ingest bootstrap spec. On a fresh database the collection does not exist, the discovery AQL faults, the handler wraps any discovery error as CodeForbidden, and the zero-path first-generation bootstrap branch (the branch this gate is built for) is never reached. The database can never receive its first generation. The PostgreSQL arm never showed this because setup-pg's SetupMigrate pre-creates the table. Discovery now answers zero recorded paths when the field catalog collection is absent, the same posture retainedSourceFieldProfileExists already takes for a missing collection, and the same posture an empty table gives the postgres backend. A missing collection is no data, not an authorization-backend outage; a mid-request deletion still fails the query honestly. Controls: offline fake extended so accepted requests pin exactly one existence probe plus one query, rejected requests pin zero of both, and a new TestFB27MissingFieldCatalogIsNoRecordedPaths proves no AQL fires when the collection is absent; a live LOOM_TEST_ARANGO_URL proof opens a fresh database without bootstrapping and asserts both discovery shapes return empty. The gen-bind control header already named this exact failure shape: a storage fault masked as 403.
The server's explorer-config body bind is strict and closed: AggregateSource.Where is json:"-", so any workspace with aggregate.where faults with unknown field "where" before ingest gates run. The battery's v3 templates stay authored offline; each live arm now fetches its builder catalog from the seeded default explorer once and migrates predicate workspaces through the repository's own MigrateLegacyContributors, publishing canonical v4 Column.Contributor bytes - the same path the acceptance lane runs. The driver refuses to time unmatched arms when the two catalogs migrate the same workload to different bytes, and an unresolvable or cross-generation predicate path fails closed before any timing. FetchContributorCatalog refuses incomplete, truncated, or scope-less catalogs. Removes the committed scratch repro probes for the already-fixed auth-path-discovery defect; the regression proofs live in internal/backends/arango/internal/catalog.
… reads pass The first scoped GET .../authoring/v2/builder in CI returned 403 (stage authorization). compose.go wires the explorer read gate to authscope.AuthorizeReadProject, which admits a caller only when a fence grant normalizes to projectid.Legacy(project). The harness project was "loom-bench" while its fence resources lived under /programs/loom/projects/loom-bench/, whose calypr key is "loom-loom-bench" - no grant could ever equal the project key. The write gate hid the drift because it compares caller paths against the dataset path itself, so upload and publish passed while every scoped authoring read was doomed. The defaults now model program loom / project bench, addressed by the legacy key loom-bench - the shape the repo's own authscope contract tests pin. Dataset and decoy paths stay raw-distinct, so the scoped negative-auth control still denies by row-path equality, and the write gate keeps passing with both sides normalizing to the project key. TestDefaultFenceGrantsSatisfyTheProjectReadGate pins the whole default set against the gate, including the foreign-subject refusal.
The planned-leg default still asserted in the present tense that the scoped calypr lane cannot ingest on postgres (PG-PG-AUTH-PATH-DISCOVERY- ALLGENS). #156 fixed that finding and the paired lane has since uploaded on both drivers; a stale present-tense reason in a not_run disclosure is exactly the misreport class the report verifier exists to refuse. The capability now names the actual reason a planned postgres leg happens: no service/DSN in the environment.
The local postgres rehearsal disproved the chain-less-battery premise: PublishRepository always materializes (repository_publish.go requires MaterializeReceipt; publication.Target has exactly one production implementation, ClickHouseBundleStore), so the first explorer-config publish answers 422 MATERIALIZATION_FAILED without ClickHouse. There is no battery without the chain: the timed mode is removed, and --publication is excluded | run. - driver: bootstrap the default explorer by publishing the empty-floor workspace before reading the builder catalog (the default exists only after the first publish), then fetch each arm's contributor catalog. - harness: Publication binds the real explorer-config response fields (receiptId, revisionId, recipe, translationVersion); output identity belongs to the caller that authored the single-output workspace. - benchcli: --clickhouse-url/-user/-password/-recipe spawn flags; each arm materializes into its own loom_bench_* database on the shared service; run mode preflights url+recipe+enabled. - scripts: paired classification now requires plan.publication.run and identity.clickhouseDisabled=false; comparison.md states the chain ran. - CI: the paired job adds clickhouse/clickhouse-server:25.3 and invokes --publication=run through it. Validation: go test ./internal/dataframe/benchmark/... and ./internal/dataframe/... green (run-mode classification, config YAML both variants, bootstrap-first publish ordering pinned in driver_test); node scripts/benchmark-backends.test.mjs 23/23; --harness-selftest pass. The full chain (publish with materialization) has no container engine in this worker, so it is adjudicated by the paired CI job itself, which now provisions the service.
…o/backend-benchmark/pg-benchmark-dataframe # Conflicts: # internal/backends/arango/internal/catalog/all_generations_auth_path_conflict_test.go
The paired job's ClickHouse service ran bare, but the image refuses the anonymous default login from the runner network: the arango arm logged "code: 516 ... Authentication failed" at EnsureDatabase, the publication plane recorded a startup degradation, ClickHouseReadyCheck returned it verbatim, and /readyz held 503 for the whole spawn budget. The two ClickHouse jobs in this workflow already create loom_ci with CLICKHOUSE_DEFAULT_ACCESS_MANAGEMENT=1; the paired service now carries the same env, and the job exports LOOM_BENCH_CH_USER/PASSWORD the way the conformance lane exports LOOM_CLICKHOUSE_USERNAME. The orchestrator resolves ClickHouse credentials in order: explicit --clickhouse-user/-password flags, then those two environment variables, then the bare server default. Pinned by the node suite (env fills, flags win); 24/24 green. The paired job at this head is the adjudicator for the materialization chain.
The verifier comment reword at 76054ef lost tab indentation, which the tracked-files gofmt gate (internal/ci) adjudicates. go test ./internal/ci -run TestGofmtGateTrackedFilesAreCanonical now passes, and gofmt -l is clean across internal/ and cmd/.
… publication-mode README Review r4561031154: the paired lane ran --iterations 5 --rounds 5, so a spawned arm's only measured sample per round was the post-restart process-cold one; claimCellPair cites warm/service-warm only, so the report verified as a passed run with no claims and benchcli exited 1. MeasureArms now refuses a spawned arm below two measured iterations per round; the paired lane runs 10 iterations over 5 interleaved rounds (workflow, mjs defaults, and paired validation each encode iterations >= 2*rounds, rounds >= 5). Adds controls_test.go (claim regime + stability-control exclusion) and a spawned full-driver test asserting claims are produced. Review r4561031163: README documented a timed mode, a PublicationCapabilityTimed capability, and a chain-less battery that the 2026-11-13 rehearsal disproved. Section rewritten to the implemented two modes (excluded | run), the ClickHouse-backed chain the authoring battery requires (first publish answers 422 MATERIALIZATION_FAILED without a target), the per-arm databases and credential precedence, and the live-battery exclusion marker (currently six workloads).
…ed, matched-state proof Ruling-B route-independent conditions, ahead of the seeding-path decision: - PublishReceipt (POST authoring/v2/publish) verifies its answer: receipt identity round-trips, revision identity exists, and every output is READY with a materialization identity (condition ii: materialization evidence consumed, never decorative). - doArmRequest wraps activation-style refusals (typed activation codes, activate-ExplorerConfigV2, retained-revision prose) in ErrActivationRefusal centrally, so no route can tolerate, retry, or special-case one (condition iii); detection is content-based, verified not to mislabel generic 5xx. - SetupProofEvidence + MatchedSetupProof give the pre-timing matched-state proof: content-derived draft digest equality, exact output-set match, materialization-count equality, and agreeing setup-preview rows across arms (condition i). Revision ids are recorded, not compared. Wiring into AuthorWorkload waits on the coordinator's seeding-path selection (command-replay vs store-backed seed route); these primitives survive either path. Fakes serve the two new routes. Suites green.
…apter posture, not prose Coordinator ruling (2026-11-13, Option A sequence): the fail-closed detector keys on the typed activation codes plus the structured CodeBackendUnavailable + stage=adapter posture (postgreSQLRuntime Unbiased shape in runtime.go), never on message prose. Prose-only bodies with no typed code are ordinary failures; the paired 503 shape is still detected via GENERATION_ACTIVATION_FAILED, and the remaining named refusal (ValidateSelectionReferences) is detected via the adapter posture. Tests pin all three cases.
…driver setup path Review r4561644037: PublishReceipt, FetchExplorerActiveState, SetupProofEvidence, and MatchedSetupProof were referenced only by their own tests — the driver authored, reconciled, and went straight to the authorization gate and MeasureArms, so ruling conditions (i) and (ii) were enforced only against a fake. RunBenchmark now, per live workload: after AuthorWorkload on every arm, each arm publishes its reconciled receipt through the authoring route (untimed setup; every output must answer READY with a materialization identity) and reads back its active state; timing starts only after MatchedSetupProof passes — equal content-derived draft digest, equal output set with a materialization identity each, equal setup-preview rows. Divergence aborts with no report and no samples. Activation-style refusals on the new route abort fail-closed through the central doArmRequest keying. The repository explorer-config spine is unchanged (Option A); the authoring publish is added setup verification, same route and payloads on both arms. Single-arm lanes keep per-arm verification; cross-arm equality applies when both arms run. Tests: both-arms protocol now asserts one authoring publish per live workload per arm; a digest-divergence test proves the abort happens before timing (only the two first-workload proof previews may exist). README documents the pre-timing proof and the fail-closed refusal posture. gofmt/node/benchmark suites green; internal/ci green except the pre-existing shallow-checkout vet probe (CI-depth unaffected).
…ed on steady regime Review r4561644049: VerifyReport accepted SelfRatioArango/ SelfRatioPostgres, UnstableCells, and StabilityCVLimit on faith -- a crafted artifact could borrow a perfect 1.0 self-ratio or omit an unstable cell and still verify. The verifier now recomputes the self-ratio from raw samples (exact-match within 1e-9), recomputes the unstable-cell list, and pins the CV limit to the exported constant. Two new forgery cases pin the paths (self-ratio-borrowed-perfect, unstable-cell-hidden-from-control), and the report/armplan fixtures now state the honest control values their samples derive. Review r4561644054: ControlUnstableCells computed CV over all regimes while discovering cells via passedStable -- the mandated spawned protocol's one process-cold sample per round inflated every cell's CV and would have suppressed all claims, the round-2 no-claims failure re-entering through the control path. CV is now judged over the steady (warm/service-warm) regime, matching the docstring and claimCellPair. TestControlUnstableCells pins the cold-outlier-must-not-flag invariant, a warm-relabelled control of the control, genuine instability, the sub-3-sample rule, and cold-only cells.
…et guards Review r4562338071: the two new verifier guards needed cases that fail only on them. - unstable-cell-hidden-from-control was confounded: injecting three wild samples desynced the summaries and pushed the half-median deviation far past the band, so an earlier check could reject for reasons unrelated to the unstable-cell rule. Rewritten per the review: the primary timing's arango warm samples become alternating 1,100,1,100 (equal half medians keep the honest self-ratio match), the summary row is recomputed with Summarize, claims are rebuilt from those summaries, and Controls.UnstableCells stays cleared -- only the ControlUnstableCells recompute catches this shape. - stability-limit-smuggled-ceiling: 0.9 passes every rule the old verifier had (it rejected only <= 0), exercising solely the new equality with the exported constant. - forgeErr lets cases name the rejecting rule; the three guard cases now pin their rule by error message. Verified by deleting each guard: all three artifacts then survive verification, so each case is the exclusive witness of its rule.
…er) into benchmark branch Hop-merge per coordinator order: pulls the #184 ActivateRepositoryGeneration adapter + 000012 activation-shape FAILED->ACTIVE revival wave onto the benchmark head so the paired-driver job can go green at the real spine. Merge-commit hop; no rebase, no force-push.
Throwaway diagnostic for the cross-arm matched-bytes red at ba53916 (aggregate-sum-age-by-patient publishes divergent workspace bytes). Additive only: an env-gated (LOOM_BENCH_DIAG_DIR) raw /builder catalog dump per arm in FetchContributorCatalog, the env on the paired run step, and an always() artifact upload (diag-159-catalogs) so both arms' catalogs survive the expected first-predicate abort. Matchers untouched (zero diff on driver/report/controls/authoringpublish/ harness/gates/measure and the runner script). DO NOT MERGE.
Divergence is fully localized to the snapshot token: every response element (availability incl. examples/counts, nodes, edges, full candidate payloads) matches byte-for-byte across arms; only ID-derived fields differ. Identity digests are recomputed read-time from raw backend evidence values hidden from the response (DistinctValues, PivotColumns, ExtensionValues, SemanticObservations, MaxItems). Adds env-gated (LOOM_BENCH_DIAG_DIR) dumps of the full CapabilityEvidence and resolved SnapshotIdentity per arm at the shared read site. No behavior change when the env var is unset. Matchers zero-diff attested. DO NOT MERGE.
matthewpeterkort
left a comment
matthewpeterkort
left a comment
Collaborator
Author
There was a problem hiding this comment.
Review of PR #187 (head 6c5e5d0, throwaway diag-159). Scoped diff against the approved #159 head ba53916: exactly two commits, four files (+52 lines), everything else byte-identical to the round-6 approval. Verified the diag plumbing is real, not broken: LOOM_BENCH_DIAG_DIR reaches the spawned loom-server children because SpawnServer uses cmd.Env = os.Environ() + spec.Env (provision.go:91); the artifact path matches the diag dir; the dumps carry no credentials (CapabilityEvidenceOptions is project/generation/auth-paths/resource-types only; CI creds are the throwaway values already disclosed in the workflow); the new imports compile (internal/catalog was already imported by explorer_capability.go). Verdict: CHANGES REQUESTED. Three required inline findings: (1) the wave-2 evidence and identity dumps collide across the two concurrently running arm servers — same binary, same %T tag, same shared directory, last-writer-wins with no atomic rename — which silently destroys exactly the cross-arm comparison this capture exists to make; (2) the identity dump puts an untested env-gated disk write on the production capability-resolution path; (3) merge posture: the PR is self-declared DO NOT MERGE and must be kept unmergeable after the evidence run. Non-blocking suggestions (no action required): for a PR whose deliverable IS the artifact, consider if-no-files-found: error on diag-159-catalogs so a plumbing regression cannot look like an empty diagnostic; DiagDump runs MarshalIndent on every capability read — harmless while env-gated, do not copy the pattern; the name parameter is unsanitized while tag is (fine for the current constant call sites). Reviewer notes: review-only pass — no tests, builds, or CI executed; no code in the local checkout was modified; this is same-model review coverage, not cross-model evidence.
Collaborator
Author
|
Diagnostic complete: both harvest waves captured; root cause localized to write_profiler pivot_value_selector last-write-wins (arrival-order dependent). Part-2 package delivered to coordinator. Closing throwaway PR. |
matthewpeterkort
left a comment
matthewpeterkort
left a comment
Collaborator
Author
There was a problem hiding this comment.
Review of PR #187 (head 6c5e5d0, throwaway diag-159). Companion to summary review 5892035560; this review carries the three required inline findings. Scoped diff against the approved #159 head ba53916: exactly two commits, four files (+52 lines), everything else byte-identical to the round-6 approval. Verified the diag plumbing is real: LOOM_BENCH_DIAG_DIR reaches spawned loom-server children (SpawnServer uses cmd.Env = os.Environ() + spec.Env, provision.go:91); the artifact path matches the diag dir; the dumps carry no credentials (CapabilityEvidenceOptions is project/generation/auth-paths/resource-types only; CI creds are the throwaway values already disclosed in the workflow); imports compile (internal/catalog already imported by explorer_capability.go). Verdict: CHANGES REQUESTED — three required inline findings: cross-arm dump filename collision defeating the capture purpose, an untested env-gated disk write on the production capability-resolution path, and merge posture for the self-declared DO NOT MERGE throwaway. Non-blocking suggestions (no action required): for a PR whose deliverable IS the artifact, consider if-no-files-found: error on diag-159-catalogs so a plumbing regression cannot masquerade as an empty diagnostic; DiagDump runs MarshalIndent on every capability read — harmless while env-gated, do not copy the pattern; the name parameter is unsanitized while tag is (fine for the current constant call sites). Reviewer notes: review-only pass — no tests, builds, or CI executed; the local checkout was not modified; same-model review coverage, not cross-model evidence.
| if err != nil { | ||
| return | ||
| } | ||
| _ = os.WriteFile(filepath.Join(dir, name+"-"+tag+".json"), raw, 0o644) |
Collaborator
Author
There was a problem hiding this comment.
Required change: this filename scheme silently defeats the cross-arm comparison this capture exists to make. In the paired lane both loom-server processes run concurrently on one runner, inherit the same LOOM_BENCH_DIAG_DIR, and — being the same binary — produce identical %T tags, so the arango and postgres arms last-writer-wins collide on evidence-.json and identity-.json; direct WriteFile to the final path with no lock and no atomic rename can also publish torn JSON. The driver-side dump in contributors.go is correctly per-backend (catalog-.json); the server-side dumps are not. Make every writer filename unique — for example have the driver pass LOOM_BENCH_DIAG_TAG=arango|postgres through ServerSpec.Env and include it in the name (or append the pid) — and stop swallowing write errors: log failures to stderr so a missing or truncated artifact can explain itself instead of quietly returning one arm-silenced dataset.
| if err != nil { | ||
| return capability.Snapshot{}, fmt.Errorf("digest semantic field identity: %w", err) | ||
| } | ||
| catalog.DiagDump("identity", struct { |
Collaborator
Author
There was a problem hiding this comment.
Required change: this call site puts an env-gated disk write on the production capability-resolution path. Any deployment or inherited environment that happens to set LOOM_BENCH_DIAG_DIR turns every Resolve into a serializing file writer with every error ignored, and no test pins behavior for either env-set or env-unset. After the diag-159 evidence run, remove this hook from the production path rather than leaving it landed: if the capability ever needs to survive, ship it as a build-tagged file or an observer injected only by the benchmark driver, with a unit test pinning (a) no write when the env is unset and (b) unique arm-tagged files when it is set. As written it also inherits the cross-arm filename collision from the capability_evidence.go comment and needs the same per-arm unique naming.
| if: always() | ||
| uses: actions/upload-artifact@v4 | ||
| with: | ||
| name: diag-159-catalogs |
Collaborator
Author
There was a problem hiding this comment.
Required change: merge posture. This PR is a self-declared throwaway diagnostic (title: DO NOT MERGE) and the wave-2 code above is not landable as-is, so the concrete change is to keep this PR unmergeable once the paired job has produced the diag-159-catalogs artifact — convert to draft or close it, record the artifact link in the PR #159 evidence trail, and do not merge this commit set to main. If any part of the diagnostic needs to survive, extract it as its own reviewable change with the per-arm naming, error visibility, and tests required by the other two comments.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Throwaway diagnostic PR for the paired-driver-benchmark cross-arm byte divergence on #159 (head ba53916). Adds ONLY: env-gated per-arm /builder catalog dumper (LOOM_BENCH_DIAG_DIR) + paired-step env + if: always() artifact upload diag-159-catalogs. The paired job is EXPECTED to stay red (first-predicate matched-bytes abort); the artifact is the product. Matcher files zero-diff attested. Close+delete after harvest.