fix(strix): distinguish recovered provider faults from inconsistent terminal scan evidence

Fresh exact canary

ContextualWisdomLab/Orgmetra#64@818950960c4fe0a4e7b026bd5499fe9cca1002be produced a new required Strix terminal failure in run 34131638369, job 101787200568.

The job is not a startup/queue/sandbox failure. Exact-head admission, trusted source materialization, contextual-orchestrator sidecar provisioning, Strix installation, and model/base preparation all succeeded. Run Strix (quick) ran from 2026-09-07T15:52:49Z to 18:52:48Z, then the artifact was uploaded.

Artifact strix-reports id 10030120846 is exact-head bound and has GitHub digest sha256:e61c95771ab6c37f7a60431f01eff4ada91adbcfbd2f6eaf79a803aacd7f9d5c. Independent local hashing of the downloaded ZIP matched that digest.

Inside that immutable artifact:

run.json (sha256:3fab449a5fcc1d4507c163d58109c6d414b786dddc92f950e9f00beea73e6166) records status="completed", scan_results.scan_completed=true, scan_results.success=true, and 191 LLM requests.
findings.sarif (sha256:32434df418c1d319e32cb5c174697ccd70bff77b93b74459fec9632c762225ab) contains an empty results array.
penetration_test_report.md (sha256:037d479f18d3b8cbdaeed06eb95e8814536eff98333681be8ee6d5c41fed1566) says the penetration test completed with no high-impact vulnerabilities, but its Technical Analysis also says it identified "several low-impact issues", including potential SQL injection and employment-version-validation issues. Those claims are not represented as SARIF findings.
gate-console.log (sha256:97b3bcb9752ce00efab0ae4d76d2c03d1952d3d152ad22faf8fb2a66a6206a2d) contains a real recovered openai.InternalServerError/HTTP 500 during the scan, then later prints Penetration test completed and Vulnerabilities 0, followed by Strix report artifacts emitted warning/fatal/denied/timeout output; failing closed., Strix run emitted provider infrastructure or failure-signal output; failing closed., and finally STRIX_PROVIDER_UNAVAILABLE: contextual-orchestrator/orchestrator/free exhausted.

No rerun, elapsed-time cap, external provider fallback, manual success status, or bypass was used.

Source contradiction on protected `main`

Protected .github main@78a4937c684a54ca8e415822c913742f41c6efc4 currently documents in scripts/ci/strix_quick_gate.sh that the sticky infrastructure flag prevents incomplete-evidence bypass, but that a successful Strix exit ignores the flag because the scanner produced a complete result set. The same script then makes strict mode (STRIX_FAIL_ON_PROVIDER_SIGNAL=1) return failure for rc=0 whenever the newest report contains a warning/fatal/denied/timeout or provider failure signal. The required workflow explicitly sets STRIX_FAIL_ON_PROVIDER_SIGNAL: "1" and forbids external fallback because orchestrator/free owns discovery/failover.

The fresh Orgmetra canary therefore exposes two different questions that the current verdict collapses into STRIX_PROVIDER_UNAVAILABLE:

Recovered provider transport faults: a model request failed during a long scan, but Strix later emitted status=completed, scan_completed=true, and success=true.
Terminal evidence consistency: the terminal narrative itself names low-impact potential issues while SARIF contains no findings. That is an evidence-integrity concern independent of provider availability.

The current gate cannot tell the operator which of these actually invalidated the scan. In this canary it also says the gateway was exhausted even though the same artifact contains a terminal completed scan after the recovered fault. Do not treat that diagnostic as proof that the gateway had no usable route at termination.

Safety boundary

This issue does not propose converting recovered provider errors into automatic success. The Orgmetra artifact is not acceptable GREEN evidence as-is because the terminal narrative/SARIF representation is internally inconsistent. It also does not propose an elapsed-time timeout, direct provider/model fallback, orchestrator/auto, paid fallback, or weakening #891 fail-closed behavior.

The owner repair should preserve orchestrator/free and fail closed, but classify the failure causally.

RED acceptance

A deterministic fixture reproduces all of the following at once:

Strix process exits 0 and terminal structured metadata says completed/success;
a bounded/recovered provider exception exists earlier in the same run;
terminal report/SARIF disagree about whether findings exist;
current strict gate emits the generic provider-unavailable/exhausted verdict without distinguishing recovered transport telemetry from terminal evidence inconsistency.

Also retain fixtures where provider failure actually prevents terminal scan evidence; those must remain STRIX_PROVIDER_UNAVAILABLE/non-passing.

GREEN acceptance

Define a typed terminal-evidence state machine for at least: complete-consistent scan, incomplete/provider-unavailable scan, and complete-but-internally-inconsistent scan.
Recovered provider faults remain observable and auditable, but do not by themselves prove gateway exhaustion after a later terminal scan receipt exists.
A completed scan whose narrative/run metadata/SARIF disagree about findings fails closed as an evidence-integrity verdict, not a fabricated provider-exhaustion diagnosis.
Missing/partial/skipped/cancelled/neutral/provider-exhausted evidence remains non-passing per [Security] Fail closed when Strix produces no authoritative scan evidence #891.
Preserve orchestrator/free + gateway token only; no provider/model/group hard-code or external/paid fallback.
Preserve model timeout default-null semantics. Do not classify a healthy reasoning/stream/tool-call solely by elapsed time; Strix run ended "cancelled" at 6h with no verdict: 909 provider attempts, 92% on one route, 0 exhaustion events, request identity absent from evidence #2000's attempt/identity investigation stays separate.
Add exact artifact fixtures for recovered provider fault + complete consistent evidence, recovered provider fault + inconsistent terminal evidence, and genuine incomplete/provider-unavailable evidence.
After protected merge, rerun unchanged Orgmetra#64@818950960c4fe0a4e7b026bd5499fe9cca1002be and require a typed terminal verdict whose diagnosis matches the emitted artifact. Do not reuse predecessor evidence.

Related owner work

[Security] Fail closed when Strix produces no authoritative scan evidence #891: fail closed when no authoritative Strix evidence exists.
fix(strix): bare Fatal/Denied/Warn/Warning match lacks provider-context guard, causing false-positive fail-closed on clean scans #1291: false positives from bare warning/fatal/denied tokens; this canary is broader because it contains a real provider 500 plus a later terminal scan.
Strix run ended "cancelled" at 6h with no verdict: 909 provider attempts, 92% on one route, 0 exhaustion events, request identity absent from evidence #2000: long-run request identity/exhaustion evidence; do not solve that by adding a wall-clock model timeout.
[CO owner] eliminate orchestrator/free provider-family SPOF before gateway release #1915/Review sidecar probes and replays spend the per-key free-tier rate budget the review itself needs #1948: orchestrator/free provider-family availability and free-tier pressure.

Fresh downstream RED from ContextualWisdomLab/fast-mlsirm#1773@fa3d06a1a55ecc91e561603d1daf9f2c4de2c348 sharpens this issue beyond the existing Orgmetra inconsistent-terminal-evidence canary.

Exact required Strix run/job: 34284217353 / 102256794796. Exact retained artifact: strix-reports id 10080622334, GitHub digest sha256:feb402d0062244db0455c203395ae3b04ab95132f835de70db44f9d94bae620f.

I downloaded and inspected that immutable artifact rather than classifying from the check conclusion alone:

run.json: top-level status="completed"; scan_results.scan_completed=true; scan_results.success=true.
findings.sarif: valid SARIF 2.1.0 with results=[].
penetration_test_report.md: repeatedly states no actionable vulnerabilities in the declared two-file PR scope; unlike the original Orgmetra canary, the terminal narrative observed here is consistent with zero SARIF findings.
strix.log contains exactly one WARNING-level record: strix.tools.web_search.tool: web_search invoked without PERPLEXITY_API_KEY configured. No ERROR/CRITICAL/FATAL record is present in the report log.
gate-attempts/001-orchestrator_free-rc0.log also contains Strix's ordinary MODEL QUALITY WARNING banner because openai/orchestrator/free is not one of Strix's hard-coded recommended model names.
Nevertheless the scanner reaches Penetration test completed and Vulnerabilities 0, after which the central gate emits Strix report artifacts emitted warning/fatal/denied/timeout output; failing closed, then Strix run emitted provider infrastructure or failure-signal output, and finally fabricates STRIX_PROVIDER_UNAVAILABLE: contextual-orchestrator/orchestrator/free exhausted.

The gateway was demonstrably not exhausted at admission: the same job's sidecar preflight selected 24 free candidates, found 6 ready routes, and the gateway chat/completions preflight completed on attempt 1 with finish_reason=stop; priced selected routes remained 0. The scan then ran for roughly 36 minutes and produced terminal structured evidence.

This adds a distinct RED class to #2026: complete + internally consistent terminal evidence plus an auxiliary-tool / model-quality warning is currently collapsed into provider exhaustion. A provider-exhausted verdict is causally false here even if policy ultimately decides the missing auxiliary web-search capability should keep the security gate non-passing.

Please preserve fail-closed semantics but add an exact fixture/state for this shape. At minimum, terminal classification must distinguish (a) genuine gateway/provider exhaustion, (b) completed evidence with recovered provider telemetry, (c) completed evidence with auxiliary capability unavailable (here PERPLEXITY_API_KEY), and (d) internally inconsistent report/SARIF evidence. Do not silence unknown warnings globally and do not convert this artifact to GREEN merely because the process exited 0; the point is to stop diagnosing an unrelated auxiliary/tool warning as orchestrator/free exhausted and to bind the final verdict to typed terminal evidence.

Owner boundary: no fast-mlsirm workflow/provider fallback was added, and orchestrator/free remains the only model route.

Fresh exact Orgmetra canary adds the complete-but-internally-inconsistent class already required by this issue, and shows why the terminal verdict must not be collapsed into provider exhaustion.

Consumer: ContextualWisdomLab/Orgmetra#63@d88800a5ca3ca15df332e8def5e25064c46e4005.
Required Strix run/job: 34294884985 / 102289341649.
Immutable strix-reports artifact: id 10083683583, GitHub digest sha256:0936587636b5689136fb28221d55b84982fa8241950d087cb52818cb2299887f.

The job passed exact-head admission, trusted source materialization, CO sidecar provisioning, Strix installation, and orchestrator/free model/base preparation. Run Strix (quick) then ran from 2026-09-09T00:38:00Z to 01:02:16Z and emitted terminal artifacts. run.json says status=completed, scan_results.scan_completed=true, and scan_results.success=true; its executive/technical summaries say no high-impact vulnerabilities. In the same terminal artifact set, however, vulnerabilities.json and SARIF contain one HIGH CWE-863 result, and the console says Vulnerabilities HIGH: 1 (Total: 1). This is terminal evidence inconsistency, independent of earlier provider/tool telemetry.

The gate then emits Strix report artifacts emitted warning/fatal/denied/timeout output; failing closed, reports exit code 2, and finally diagnoses STRIX_PROVIDER_UNAVAILABLE: contextual-orchestrator/orchestrator/free exhausted. That diagnosis is causally wrong for a run that reached terminal structured evidence with 204 LLM requests; the artifact is non-passing because its terminal evidence conflicts, not because terminal evidence is absent.

Please add this exact fixture shape to the typed terminal-evidence state machine: completed/success metadata + narrative claiming no high-impact finding + SARIF/vulnerability ledger containing a HIGH finding. Preserve fail-closed semantics. Do not turn this run GREEN, suppress the finding, add a direct provider/model fallback, or classify a long-running reasoning/tool path by elapsed time. The required output is a typed evidence-integrity verdict distinct from genuine CO/provider exhaustion.

Related validation evidence from PR #2089 exact head 693e618b6c11cc4e1286734940a1c3b21a50d074: the new Strix location validator passes trusted-state 2 for both ../outside.py and a file reached through a symlinked ancestor inside the scan tree (scan/link -> target, link/secret.py). This means the current validator checks only the final file and does not bind finding paths to the trusted scan-tree root. Finding and reproduction are recorded on PR #2089: #2089 (comment).

Fresh Wardnet consumer RED confirms the complete + apparently consistent terminal evidence + recovered provider telemetry class and gives a current central-workflow canary.

Consumer: ContextualWisdomLab/wardnet#174@e7e8bdcf3922f6a07647478befeda86866254a9b against protected base main@f8260f1e03836039ff9463dd99fa982e4e270c4b.
Required Strix run/job: 34686246615 / 103534063940.
Trusted central workflow source materialized at exact .github SHA 691fb78932eff5fbe52db69077848134b0b4e053; contextual-orchestrator sidecar was vendored at exact SHA 414f22973658c4ddc3d4320fcf7acd9b4e8ba991. The model contract remained contextual-orchestrator/orchestrator/free; no direct provider/model fallback was introduced.

The gateway admission path did encounter several bounded provider-discovery failures (Bytez 500, NVIDIA/OpenRouter timeout/429/404 responses) before recovering and reporting five ready routes. Strix then completed the scan and emitted the terminal console verdict Penetration test completed / Vulnerabilities 0 (No exploitable vulnerabilities detected) on the exact PR scope. Immediately afterward the central quick gate emitted Strix report artifacts emitted warning/fatal/denied/timeout output; failing closed., then Strix run emitted provider infrastructure or failure-signal output; failing closed., and finally STRIX_PROVIDER_UNAVAILABLE: contextual-orchestrator/orchestrator/free exhausted.

This is the same causal misclassification #2026 is meant to eliminate: recovered route-discovery/provider telemetry is being retained as a sticky terminal infrastructure signal after the scanner has reached a completed authoritative result. The terminal error therefore claims gateway exhaustion even though the same run documents successful failover and a completed scan.

Please use this unchanged Wardnet head as another post-fix canary. GREEN must preserve #891 fail-closed semantics while distinguishing genuine incomplete/provider-unavailable evidence from a completed, internally consistent scan that merely contains recovered provider telemetry. Recovered failures remain auditable, but must not by themselves manufacture a terminal orchestrator/free exhausted diagnosis after valid terminal scan evidence exists. No Wardnet-local workflow copy/workaround should be required; repair belongs to the central Strix owner.

Fresh complete-consistent + recovered-provider-fault canary from ContextualWisdomLab/LineageWeave#1046@f80c0ec5f35fd4fc0a867125bc46dfd2d2dee9a9, Strix run 34692420821, artifact 10298596767 (sha256:0ab02471f2d81667914ae48281aaa0808bd239330bb5b8188a51cf2725b5fd8b). I downloaded and inspected the immutable artifact rather than classifying from workflow metadata.

Terminal evidence is internally consistent:

run.json sha256 dddd6e57e477e0ec9e17a487c33e95b536b0efafbe557a4508495a0eb94cc31a: status="completed", scan_results.scan_completed=true, scan_results.success=true, 55 recorded LLM requests.
findings.sarif sha256 32434df418c1d319e32cb5c174697ccd70bff77b93b74459fec9632c762225ab: empty results.
penetration_test_report.md sha256 32aa14d3b8816d7b97a0b5705459733478e26c10eea96c23d848e2c0bb1bbd09: explicitly reports low/healthy posture, 0 known-CVE dependency findings, no secrets, no integrity issues, and no exploitable vulnerability demonstrated. Unlike the Orgmetra canary recorded in this issue, the narrative does not name latent low-impact findings that are absent from SARIF.

Recovered provider telemetry is nevertheless noisy:

sidecar stderr sha256 858b60eed31a7ddbf875c2061233dd1a890c1c1048e82efedf44025ae943b0cb: 116 provider_attempt starts, 53 provider_attempt_failed, 41 request_failed status=500 code=internal_error, 0 provider_exhausted, and no request_id= evidence;
99/116 attempts (85.3%) selected nvidia_nim_deepseek_ai_deepseek_v4_flash_0731; 42 of its attempts failed;
error telemetry includes TimeoutError and HTTPError, while the scan still reached a successful terminal receipt.

The strict wrapper then emits, after the successful Strix completion block:
Strix report artifacts emitted warning/fatal/denied/timeout output; failing closed.
Strix run emitted provider infrastructure or failure-signal output; failing closed.
STRIX_PROVIDER_UNAVAILABLE: contextual-orchestrator/orchestrator/free exhausted; ...

That final exhausted diagnosis is contradicted by the sidecar evidence (provider_exhausted=0) and by the later complete-consistent terminal scan receipt. This canary therefore supplies the exact fixture this issue asks for: recovered provider fault + complete consistent evidence, distinct from both genuine incomplete/provider-unavailable evidence and the Orgmetra complete-but-internally-inconsistent case.

No leaf bypass was applied. LineageWeave #1046 has been returned to Draft because the required Strix context is terminal failure even though the retained scan evidence itself is complete and clean.

Fresh recurrence from ContextualWisdomLab/fast-mlsirm#1829@6782cdab34015d62a211c8c7a919c486e021e5ad: required Strix run 34702334353, job 103577031062.

The retained full job log records the exact MODEL QUALITY WARNING that openai/orchestrator/free is not a recommended frontier model for Strix, followed later by Strix run succeeded for model 'orchestrator/free'. The workflow also contains explicit policy logic accepting orchestrator/free / contextual-orchestrator/orchestrator/free as the configured model and treating provider/backend signals separately. This recurrence therefore confirms the warning is emitted for the configured alias/model label; it does not by itself prove a capability failure or gateway exhaustion.

Please classify the warning source and its relationship to the existing typed terminal-evidence work in this issue. Preserve the orchestrator/free policy and fail-closed behavior; do not replace providers/models, add paid fallback, or silence the warning globally. Evidence is retained locally as pr1829-strix-678-full.log (warning lines 2181-2186; policy lines 2018-2024).

Fresh Orgmetra canary supplies the genuine incomplete/provider-unavailable side of this state machine, distinct from the earlier completed-but-inconsistent examples.

Consumer: ContextualWisdomLab/Orgmetra#64@c8d1c3993ce1eb4e8bdabfe4b666c61eda57fcff; required Strix run/job 34704247956 / 103581586116; trusted reusable workflow .github@fb17ef556f94f673234aa557254ae52779e9a7b0; CO 414f22973658c4ddc3d4320fcf7acd9b4e8ba991.

Exact-head admission, trusted source materialization, CO sidecar provisioning, installation/configuration, model/base preparation, and gateway chat/completions preflight all succeeded. Preflight selected only free routes (priced_selected_count=0) and returned finish_reason=stop. The workflow/compat boundary explicitly set LLM_TIMEOUT=0, STRIX_MEMORY_COMPRESSOR_TIMEOUT=0, STRIX_PROCESS_TIMEOUT_SECONDS=0, STRIX_TOTAL_TIMEOUT_SECONDS=0, and CWL_STRIX_UNBOUNDED_INFERENCE=1.

The actual Strix request later terminated after about 1880 s with LLM CONNECTION FAILED / Could not establish connection to the language model / Error: Request timed out. Strix exited 1, and no completed vulnerability report/SARIF verdict was produced. Retained strix-reports artifact id 10302600731, digest sha256:b00d613097ac0cd838d08bc1a95200697e4ad23ec4f644f8ffc9699d4d78029e, contains log/evidence only. In this shape STRIX_PROVIDER_UNAVAILABLE is appropriately non-passing; unlike the older Orgmetra canary, there is no later completed scan receipt whose evidence would make provider-unavailable a causally false terminal classification.

Please retain this exact fixture shape when implementing the typed terminal-evidence state machine: healthy admission/preflight + no priced/direct fallback + later terminal model-request timeout + no authoritative scan verdict => incomplete/provider-unavailable, fail closed. Separately, #2000 now has the same canary for request-identity/timeout-provenance investigation because the zero/null timeout contract did not demonstrably survive end-to-end. No elapsed-time failure policy or external/provider fallback is requested.

Fresh complete-consistent Orgmetra canary for the terminal-evidence state machine:

consumer: ContextualWisdomLab/Orgmetra#64@d9cc516d54b4642f59fe126c331a19945fdf75f1
required run/job: 34731009530 / 103653717441
reusable workflow exact source reported by the job: .github@749668517580255635ce44052b34c2cc05bca82f
immutable strix-reports artifact: id 10311173510, GitHub digest sha256:3cb9623b44ba03143a0955b96644251d81f8902514664708d4e9d9fb78679878

The artifact is materially different from the earlier inconsistent-narrative fixture. run.json records status="completed", scan_results.scan_completed=true; the terminal report says 0 confirmed/exploitable vulnerabilities and explicitly labels its digest/CI/tenant-boundary items as defense-in-depth recommendations rather than vulnerabilities; findings.sarif is valid SARIF with 0 results; strix.log ends finish_scan: completed scan with 0 vulnerability report(s). So terminal narrative/run/SARIF agree on the vulnerability verdict.

Earlier provider faults were nevertheless recovered during the scan. The sidecar records 297 provider-attempt starts, 169 failed attempts, repeated TimeoutError / request_failed status=500 code=internal_error, and no provider_exhausted event. After that telemetry, Strix still reached a complete terminal scan at 2026-09-13T04:06:32Z.

Current strict gate then emits, after the completed Vulnerabilities 0 banner:

Strix report artifacts emitted warning/fatal/denied/timeout output; failing closed.
Strix run emitted provider infrastructure or failure-signal output; failing closed.
STRIX_PROVIDER_UNAVAILABLE: contextual-orchestrator/orchestrator/free exhausted; ...

This is the requested recovered provider fault + complete consistent evidence fixture. The final provider exhausted diagnosis is not supported by the same artifact (provider_exhausted=0, complete consistent terminal evidence exists). Please preserve the provider-fault telemetry but classify the terminal evidence causally rather than collapsing this case into provider-unavailable. No direct/provider/paid fallback, elapsed-time success rule, or gate weakening is requested.

Navigation Menu

Repository navigation

fix(strix): distinguish recovered provider faults from inconsistent terminal scan evidence #2026

Fresh exact canary

Source contradiction on protected `main`

Safety boundary

RED acceptance

GREEN acceptance

Related owner work

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

fix(strix): distinguish recovered provider faults from inconsistent terminal scan evidence #2026

Description

Fresh exact canary

Source contradiction on protected main

Safety boundary

RED acceptance

GREEN acceptance

Related owner work

Activity

seonghobae commented on Sep 9, 2026

seonghobae commented on Sep 9, 2026

seonghobae commented on Sep 11, 2026

seonghobae commented on Sep 12, 2026

seonghobae commented on Sep 12, 2026

seonghobae commented on Sep 12, 2026

seonghobae commented on Sep 12, 2026

seonghobae commented on Sep 13, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Source contradiction on protected `main`