Noema required-review retry loses valid head on malformed verdict, then exhausts via 502

Affected consumer

ContextualWisdomLab/naruon#1505 exact head 7da2a242e463f59d4580cb38e7591f1ba4b4049e.

Required Noema Review run 33460498090, job 99742587317, using trusted central workflow commit 44a3c740f7c46c06e7500174d4127413f3f581eb.

Exact failure boundary

The job correctly bound the target to the live PR head and provisioned the repository-scoped cwl-noema-review App token. The contextual-orchestrator sidecar reached a ready gateway and reported four usable free routes.

The first model response reached noema_review_gate.py::validate_substantive_verdict but failed the deterministic schema/semantic contract:

RuntimeError: Noema adversarial probe 1 outcome must be falsified or confirmed

call_llm() then retried through the model path. After the long retry budget, the retry terminated at the sidecar with urllib.error.HTTPError: HTTP Error 502: Bad Gateway, and the required check failed after ~88 minutes.

This is central review-control/model-output handling evidence, not a Naruon source failure. Naruon repository CI, Security Scan, Dependency Review, Semgrep, Bandit and image/build workflows for this exact head were independently terminal-success; all PR inline findings are resolved. The Noema failure remains non-passing and must not be converted to source approval or bypassed.

Required repair

Add a deterministic RED fixture where a model verdict is structurally parseable but an adversarial-probe outcome is outside the closed falsified|confirmed domain.
Keep that malformed model output non-passing, but classify it as model-output/schema failure rather than a Naruon source finding.
Bound retry/fallback so one malformed verdict cannot consume the full required-review wall-clock and then end only as an opaque gateway 502; preserve exact failure receipts for each materially distinct attempt.
Retry/fallback only through contextual-orchestrator-owned provider routing; do not add direct provider calls or weaken the deterministic verdict validator.
Preserve exact-head/live-base binding, repository-scoped reviewer credentials, and fail-closed behavior.
GREEN acceptance: on unchanged naruon#1505@7da2a242e463f59d4580cb38e7591f1ba4b4049e, a fresh required Noema run either produces a schema-valid current-head formal review verdict or a typed model/provider unavailable result within the bounded budget; it must not synthesize a source defect from infrastructure/model-output failure.

Related organization queue saturation issue #1531 may affect scheduling, but this failure had a runner and executed for ~88 minutes, so the first causal boundary here is verdict-validation/retry handling rather than runner acquisition.

Fresh same-class Noema verdict-validation recurrence from ContextualWisdomLab/OriginWeave#82, extending this issue's acceptance surface beyond an invalid probe outcome value.

Exact consumer evidence:

protected live base OriginWeave/main@542ca1e9c0a863595b8b6697790005d2471f5413
exact PR head 90c4e9a8f31eb94eb46243343160d3f6c96921ef
required Noema run 33497088638, attempt 1, job 99821658651
trusted central workflow source ContextualWisdomLab/.github@035269a76ffa6e176ee467f452467cbf4c2158a8
exact-head/live-PR validation, repository-scoped cwl-noema-review App token, target visibility, and contextual-orchestrator orchestrator/free sidecar all succeeded
sidecar eventually had 3 ready free routes and gateway chat/completions preflight succeeded
first terminal failing boundary in scripts.ci.noema_review_gate: Noema adversarial validation requires at least 2 concrete probe(s)
same exact OriginWeave head independently has successful repository CI, MV3 Compatibility, SAST Semgrep, and Security Scan; current inline review threads are resolved

A bounded unchanged-head rerun of run 33497088638 was accepted as attempt 2 and is currently queued, so attempt 1 remains non-passing and is not converted to source failure or approval.

Please extend the existing RED/GREEN contract so a structurally parseable review that contains fewer than the required concrete adversarial probes is treated as typed MODEL_OUTPUT_INVALID / review-unavailable evidence, receives bounded deterministic corrective feedback/retry through contextual-orchestrator only, preserves exact attempt receipts, and cannot consume the full review budget or synthesize a leaf source defect. GREEN canary should include the unchanged OriginWeave #82 head above in addition to the existing malformed-outcome case.

Fresh Orgmetra Noema consumer canary adds a related provider/review-budget failure mode. ContextualWisdomLab/Orgmetra#40 exact unchanged head 6917e41f9053fab6f7e99f8185f2137e8fc5fca5, required Noema run 33294991587, job/check 99436298297, trusted workflow source 1cbb6aaf0a24c3628d24c3dd6d9dcaa8a7eec0c5: repository-scoped cwl-noema-review token minting, target visibility, contextual-orchestrator sidecar provisioning, /healthz, provider-route preflight and gateway chat/completions preflight all succeeded. The sidecar had 3 ready free routes. The first causal failure was then inside scripts.ci.noema_review_gate while waiting for /v1/chat/completions: Python urllib raised TimeoutError: timed out at its 120s request boundary. This is not runner acquisition and not an Orgmetra source finding. RED acceptance should also cover a healthy sidecar whose review request exceeds the per-attempt deadline; GREEN should preserve fail-closed semantics while returning typed model/provider/review-unavailable evidence with bounded retry/attempt receipts rather than an opaque traceback or fabricated source verdict. I re-ran this exact failed Noema job without mutating the Orgmetra head; any fresh verdict must bind the same SHA.

Reopened after a post-closure Wardnet recurrence proves the bounded provider/review-unavailable acceptance is not yet protected truth.

Fresh consumer evidence:

ContextualWisdomLab/wardnet#77
exact unchanged head 7cb69730930653f78acf954cff957291d0b1320c
protected live base main@cc15cc2c34daf8c104eeb83d52a6a66f3cd6e128
required Noema run 33531072968, job/check 99934082777
trusted central workflow source ContextualWisdomLab/.github@cb38cc30284a02d0986cb55a14ff0a65ef390937

The runner was acquired on Ubuntu 24.04, live PR/head binding succeeded, repository-scoped cwl-noema-review App token minting succeeded, and the contextual-orchestrator sidecar completed /healthz, provider-route and gateway chat/completions preflight. It reported five ready free routes. The model/review phase then ran from 03:16:26Z until 04:35:38Z and terminated only as:

HTTP Error 502: Bad Gateway

No typed model/provider/review-unavailable envelope was produced. This exact head independently has repository CI, Fuzz, Security Scan, SAST, CodeQL, Semgrep, OSV, dependency-review and Trivy/Scorecard evidence terminal-success, and every current inline review thread is resolved. coverage-evidence remains queued independently, so the PR remains non-passing and is not a bypass candidate.

This recurrence is slightly broader than the original malformed-verdict path: even when the log exposes no earlier schema-validation error, a healthy preflight followed by a long review request must not consume ~79 minutes and end as an opaque 502. Extend the existing RED/GREEN contract to cover this provider/review-budget path directly: bounded materially distinct attempts through contextual-orchestrator only; exact attempt receipts; typed MODEL_PROVIDER_UNAVAILABLE / REVIEW_UNAVAILABLE (or equivalent) when the review cannot complete; no fabricated source verdict; exact PR/head binding preserved. After central repair reaches protected truth, rerun unchanged Wardnet #77 and the existing Naruon/Orgmetra canaries.

docs(item23): Noema review-gate failure retrospective and improvement…

Fresh LifeOS recurrence confirms the same healthy-preflight → long opaque-502 class and adds exact gateway attempt telemetry that narrows the owner split.

consumer: ContextualWisdomLab/life-os#198
exact unchanged head: 387dc146cb9971acb040de893feadf1fb642c600
Required Noema Review run 33728189150, job 100561893763
trusted central workflow source: ContextualWisdomLab/.github@ff332ea88fc93371b032539a12b9985ef249dfbe
contextual-orchestrator source pinned by the sidecar: 464da4715b495b5eaaa593eba3796e2d976ee0c9
repository-scoped reviewer token, live-head validation, target visibility, /healthz, provider-route preflight and gateway chat/completions preflight all succeeded
orchestrator/free reported 52 admitted routes, 12 selected candidates, six ready candidates, three-account diversity and priced_selected_count=0
actual verdict request ended after 3647.8s as HTTP Error 502: Bad Gateway, with caller attempts=1, phase=connecting, served_model=unknown; no publishable verdict was produced

This remains review infrastructure evidence, not a LifeOS source finding or approval. The central caller is correctly issuing one gateway request with gateway token + orchestrator/free; it must not add provider/model/group/paid fallback in .github or LifeOS.

I filed the corresponding gateway-owner defect as ContextualWisdomLab/contextual-orchestrator#1045, with RED/GREEN acceptance for request-correlated candidate/failover evidence and retryable 502 handling while keeping default model timeout null and user cancel/provider end/admin timeout distinct. This issue should retain the central Noema-side acceptance: translate the owner gateway's typed exhausted/unavailable result into bounded MODEL_PROVIDER_UNAVAILABLE / REVIEW_UNAVAILABLE evidence, preserve exact-head binding, never fabricate a source verdict, and rerun unchanged consumer canaries only after the owner repair reaches an immutable reviewed boundary.

Additional current-head Noema specimen from Context Fabric, without any leaf source failure: context-graph-contracts#4@3b1bdf0f47087bf06c082180e5a919efeaffb8dd, Required Noema Review run 33873623613, job 101065579624, trusted central workflow f893b473bda88dee56ed89af2d9b006a317d2b95.

The job acquired ubuntu-24.04, validated the exact live PR head, minted the repository-scoped cwl-noema-review token, and provisioned contextual-orchestrator. Sidecar preflight admitted 61 free routes and selected 12; the runtime probe found three ready routes and the gateway chat/completions preflight succeeded. The actual verdict phase then terminated after 592.4s with Noema gateway transport failed: HTTPError: HTTP Error 502: Bad Gateway, phase=response_error, served_model=google/gemma-4-31b-it; publication correctly stayed skipped/fail-closed.

This specimen does not first fail on malformed verdict validation, so it extends the same bounded-review requirement: provider/gateway transport failure itself must not spend ~10 minutes after a successful preflight and surface only as an opaque required-review failure. Keep contextual-orchestrator as sole routing/failover authority and exact-head binding unchanged. GREEN should include a bounded typed MODEL/PROVIDER_UNAVAILABLE receipt for this direct 502 class (or a valid verdict on the unchanged head), without inventing a CGC source finding or weakening the gate.

Fresh Wardnet canary adds a same-owner decision/probe consistency failure class on an unchanged exact head; this is central Noema model-output handling, not a Wardnet source finding.

consumer: ContextualWisdomLab/wardnet#155
exact head: f74ff25a321dfb1d7109719e2a1fc77e47dc4898
live protected base: wardnet/main@5829a0f08d78de464dd24393ce5d0f25fba9d126
required Noema run/job: 33904630596 / 101187432177
immutable trusted central workflow source: .github@67fd7e5d03a0aa602989a88f55b10cf298610c38
runner acquired normally on Ubuntu 24.04; exact-head validation, repository-scoped reviewer token, target visibility, CO sidecar, provider-route preflight, and gateway chat/completions preflight all succeeded
sidecar reported 5 ready orchestrator/free routes; verdict request was served by deepseek-ai/deepseek-v4-pro-0813
first terminal causal boundary after 196.4s: Noema model output failed local validation: Noema approve cannot contain a confirmed adversarial probe
publication was correctly skipped; no review/source finding was synthesized.

RED: a structurally parseable model result can carry a top-level approve decision while simultaneously carrying a confirmed adversarial probe. The validator correctly rejects the contradiction, but the required lane terminates as a generic failed process instead of bounded typed model-output/review-unavailable evidence.

GREEN acceptance: add deterministic fixtures for approve + confirmed probe (and symmetric decision/probe contradictions), keep the contradiction fail-closed, provide bounded corrective feedback/retry through contextual-orchestrator only, preserve exact attempt/model/duration receipts, and terminate as typed MODEL_OUTPUT_INVALID / REVIEW_UNAVAILABLE if repair cannot produce a coherent envelope. Never coerce confirmed to falsified, downgrade the finding, fabricate a source defect, or convert unavailable review to PASS. Re-run unchanged wardnet#155@f74ff25... after the owner repair reaches protected .github/main.

Fresh owner-plane recurrence on 2026-09-07 KST confirms the same healthy-preflight → long opaque-502 class, now on the PR that repairs organization ruleset governance itself.

consumer: ContextualWisdomLab/.github#1644
exact unchanged head: 5f5d7a9e849984c052c72a08eba34f68cb694024
live protected base: main@c9052e607e5f3cc76e73207e7786b21500721b79
required Noema run/job: 34056426529 / 101553094642
repository-scoped cwl-noema-review[bot] token minting, live-head validation, target visibility, contextual-orchestrator sidecar provisioning, and gateway preflight all succeeded
preflight reported target_ready=8; among the observed candidates were successful ready routes as well as typed 404/timeout rejections
the single Noema caller request then ran for 2049.8s and terminated as HTTP Error 502: Bad Gateway, phase=response_error, served_model=deepseek-ai/deepseek-v4-flash-0731
publication remained skipped and the run uploaded noema-sidecar-evidence artifact 9997785570; no source finding or synthetic approval was emitted

This is not runner starvation and not a #1644 source defect. The runner executed the gate normally; the remaining owner boundary is the contextual-orchestrator-owned free-route repair/failover and typed review-unavailable contract already tracked here. Keep the caller at one gateway request, preserve fail-closed exact-head binding, and do not add direct-provider fallback or extend the required-job wall-clock. GREEN is a bounded typed outcome or a schema-valid exact-head review, followed by the existing callback/publication path.

Fresh Wardnet current-head recurrence adds an owner-boundary defect to the existing Noema 502/review-budget RED. Consumer is ContextualWisdomLab/wardnet#173@7d0006b0f1fd3311c891bf359bd0c3e66a1831ec; Required Noema run 34025816776, job/check 101472521995, trusted central workflow source .github@43024633eba9d96b0456970391360da5a171fbda.

Exact-head admission, repository-scoped cwl-noema-review token minting, target visibility and sidecar startup all succeeded. The sidecar then logged vendoring contextual-orchestrator @ 414f22973658c4ddc3d4320fcf7acd9b4e8ba991, observed five provider secrets, and performed direct provider/model discovery attempts before reporting the orchestrator/free gateway ready. The actual Noema call used NOEMA_LLM_MODEL=orchestrator/free, but after 1644.8s it failed only as HTTP Error 502: Bad Gateway, phase=response_error, served_model=deepseek-ai/deepseek-v4-flash-0731, with no typed provider/review-unavailable terminal envelope. OpenCode subsequently issued CHANGES_REQUESTED solely because this Required Noema check was failed; this is non-passing evidence, not a Wardnet docs/source finding.

Please extend the causal repair so central model-backed review does not vendor a mutable contextual-orchestrator source commit or own provider/model discovery/secrets. GREEN for this class: consume an immutable released CO API/client/schema or trusted released gateway boundary; pass only the gateway credential and orchestrator/free; keep provider/model/failover authority inside CO; return typed provider-end/review-unavailable evidence with materially bounded attempts rather than an opaque long-running 502; preserve exact repo/PR/head/run/job binding and fail-closed semantics. Then rerun the unchanged Wardnet #173 head and publish a terminal Noema review/verdict without leaf source churn, direct-provider fallback, predecessor evidence, or gate weakening.

Fresh Wardnet current-head canary reproduces the direct healthy-preflight → opaque 502 class on a newer gateway pin.

consumer: ContextualWisdomLab/wardnet#140@13da5928e22d195b5a2f3cbfcee415060747125b
protected base at dispatch: wardnet/main@a52ccd0a24a727d9349bb32def7713882d8cad1e
Required Noema Review run 34166153353, job 101878838694
trusted central workflow source: ContextualWisdomLab/.github@78a4937c684a54ca8e415822c913742f41c6efc4
contextual-orchestrator pin: 414f22973658c4ddc3d4320fcf7acd9b4e8ba991
exact-head binding, repository-scoped reviewer token, sidecar /healthz, provider-route preflight and gateway chat/completions preflight succeeded
policy admitted 59 free routes / selected 24 / three-account diversity / zero priced candidates; runtime preflight found 2 ready routes
actual verdict request failed after 267.4s with HTTP 502, caller attempts=1, phase=response_error, served_model=unknown; no verdict envelope was published

This remains review-infrastructure evidence, not a Wardnet source finding or approval. I handed the gateway-owner specimen to contextual-orchestrator#1045 as well. Central acceptance remains: after the owner returns a typed exhausted/unavailable result, Noema must preserve exact-head binding, expose bounded MODEL_PROVIDER_UNAVAILABLE / REVIEW_UNAVAILABLE evidence, and never synthesize a source verdict. Do not add caller-side provider/model/group/paid fallback.

New exact-head reproduction on .github#2110@8371f623de21f4560d48972b06d6b63e0cfc5927: required Noema run https://github.com/ContextualWisdomLab/.github/actions/runs/34698137613, job 103565841613. Discovery admitted 8 ready free candidates, the review sidecar and repository-scoped token were provisioned, then the model phase failed after 251.9s with HTTPError: HTTP Error 502: Bad Gateway; served model deepseek-ai/deepseek-v4-flash-0731, caller attempts=1, phase=response_error. No verdict envelope or review was published, so the required job correctly remained failure. This is fresh evidence for the existing gateway-owned retry/failover gap; no leaf retry, paid bypass, timeout shortening, or source churn was applied.

Additional consumer evidence, 2026-09-13: contextual-orchestrator PR #1154 at d555ec4af1c0c039d67b123c4627fc28fdf728bf, Noema job.

The raw job log records Noema model output failed local validation: Noema approve cannot contain a confirmed adversarial probe, followed by the warning outcome=failed phase=validating duration=284.3s; caller attempts=1. This is a contradictory verdict envelope, not proof that the proposed source is safe or defective. Preserve the rejecting validator and exact-head binding. No valid formal review was published by this job.

Owner: central Noema verdict handling, with CO responsible for authorized gateway recovery. Next action: reproduce this specific approve+confirmed-probe envelope against the canonical validator and inspect the existing recovery path; preserve a typed failed-verdict receipt rather than accepting or inventing approval. Internal review date: 2026-09-14, chosen because the failure currently prevents required review. Completion requires a valid exact-head review or explicit unavailable outcome and verified gateway-owned recovery behavior; this evidence comment does not resolve the issue. No caller-side provider fallback or paid call was added.

Exact current-consumer model-output specimen for this existing Noema validation/recovery issue, 2026-10-02 KST. No source approval or recovery completion is claimed.

OriginWeave#340 head ad50387bebad48bec1b6b1267cbe55f961d55924, base 87c4daa1830bac5a5228b6036752ad5633232085; immutable central workflow/source 37b10243cec3d160ecc9c1be75c71428b160a703. Required Noema run 36926739978, job 110585910545, completed failure at 2026-10-01T21:35:31Z (2026-10-02 06:35:31 KST).

Raw causal diagnostic: Noema model output failed local validation: Noema approve requires adversarial_validation.status=passed; caller attempts=1, duration=1118.4s, phase=validating, served_model=google/gemma-4-31b-it.

Artifact 11195517237 (noema-sidecar-evidence) archive SHA256 abfa8be0a8a8c134d51179d6a83ad3c2b8581b1d0946d3a58234478ccabdb62f matched the API digest. Actual sidecar request 2c4e8e43adcc4c09928ad0bf7f17e06f returned HTTP200 after1118410.3ms. Its prior route chain contains primary/sub DeepSeek provider-response errors and Gemma attempts. Caller attempts1 is one gateway request, not one provider call. This is not PR337's HTTP400 or PR339's final429/capacity continuation403.

The artifact includes only sidecar stderr and plain-chat preflight JSON. It does not retain the rejected verdict's status field, so whether that actual field was failed, missing, or malformed is unknown. No raw model body or credential is requested for public logging.

Read-only reproduction used exact tracked 37b10243 validator/classifier/schema source, not the different local central checkout HEAD. A stdlib-only pure AST slice exercised the actual PR340 diff with its two-probe floor: an explicitly labeled offline consistent diagnostic control passes; changing only its adversarial status to failed raises the exact typed NoemaModelOutputError above. This fixture was never a historical model response, hosted review, or published approval. The rejecting validator is correct and must remain unchanged as an acceptance gate.

The pinned structured schema allows status passed|failed independently of decision, while local semantic validation enforces approve→passed and forbids confirmed probes on approve. The gateway's structural repair alone therefore does not establish this cross-field condition. The hidden historical value prevents attributing the cause more narrowly than the observed semantic-validation boundary. Exact pinned two-phase preparation writes a private verdict envelope only after successful validation and emits bounded capacity-continuation outputs only for NoemaTransportError; model-output validation is not that class. Consequently continue-noema-transport job 110595621077 was skipped rather than executing a capacity retry. Do not turn that skip into pass or trigger blind leaf redispatch.

Feasible next action for existing central owner: inspect sanitized decision/status/validator outcome with the correlation above, and evaluate a bounded gateway-owned semantic repair/exclusion route with an explicit contradictory approve/status regression. Preserve exact-head/base binding, reviewer/dispatcher separation and fail-closed gates; independently review/test any central change. The consumer has not modified central source/workflows, credentials, model routing or branch rules and has not requested a duplicate dispatch.

Current-head CodeQL/OpenCode actual downstream producers separately exist (36929151093, 36928381456) but their admission jobs are queued on the named single-runner groups; no terminal verdict/countable approval has been verified. Native exact-head Rust/Python/coverage and MV3 evidence is verified but does not substitute for this required review. Required acceptance remains either a valid current-head formal review or an explicit typed unavailable outcome, followed by the repository's unchanged protected merge gates. This evidence comment does not resolve the issue.

Navigation Menu

Repository navigation

Noema required-review retry loses valid head on malformed verdict, then exhausts via 502 #1611

Affected consumer

Exact failure boundary

Required repair

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Noema required-review retry loses valid head on malformed verdict, then exhausts via 502 #1611

Description

Affected consumer

Exact failure boundary

Required repair

Activity

seonghobae commented on Sep 1, 2026

seonghobae commented on Sep 1, 2026

seonghobae commented on Sep 2, 2026

seonghobae commented on Sep 4, 2026

seonghobae commented on Sep 4, 2026

seonghobae commented on Sep 5, 2026

seonghobae commented on Sep 6, 2026

seonghobae commented on Sep 7, 2026

seonghobae commented on Sep 8, 2026

seonghobae commented on Sep 12, 2026

seonghobae commented on Sep 13, 2026

seonghobae commented on Oct 1, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions