Skip to content

fix(review): retire leaf routing after contextual-orchestrator free-pool release #2042

Description

@seonghobae

Dependency

Canonical owner issue: ContextualWisdomLab/contextual-orchestrator#1106.

Protected consumer source main@7fd571dbcdbae6acf29d8f4ee704d7ba6297e4db still makes provider/model routing and test-time-compute decisions in scripts/ci/contextual_orchestrator_review_launcher.py and scripts/ci/contextual_orchestrator_review_policy.py. This is a temporary consumer bridge, not an approved owner boundary.

Current exact evidence (2026-09-08 UTC)

Repeated current-head Noema attempts all used model: orchestrator/free, caller attempts=1, and no paid/local fallback, but the leaf preflight exposes fixed routing/admission parameters and then the gateway fails:

The launcher currently owns fixed candidate/probe/readiness/account-skip/escalation/token budgets and a priority penalty. These values affect eligibility, selection and test-time compute without an identified statistical loss/utility model or authoritative provider allocation standard. It also imports credential/provider policy and performs completion probes in the leaf.

Fail-closed bridge rule

Until contextual-orchestrator#1106 ships an immutable released contract:

  • do not add another threshold, weight, score, timeout, retry count, token guess, tie-break, provider/model/group pin, credential rule or paid/local fallback;
  • do not copy owner source or read an owner branch;
  • preserve full sanitized evidence and exact-head failure;
  • treat model unavailability as non-passing rather than weakening review gates.

Consumer repair after owner release

Create a RED workflow/contract fixture first, then:

  1. bump the immutable contextual-orchestrator owner release;
  2. pass a list/array of configured credential sources only through the owner bootstrap boundary, with secrets redacted;
  3. remove provider/credential eligibility and OpenAI exclusion from the leaf; CO alone enforces the free-pool policy;
  4. remove candidate discovery/catalog ordering, completion probing, readiness thresholds, account-429 counters, token/escalation budgets, priority penalties and fallback selection from the leaf;
  5. make Noema/OpenCode/Strix model-backed workflows call only the gateway with model: orchestrator/free and the gateway token;
  6. keep missing capability and missing evidence fail closed;
  7. verify exact-SHA source/API-schema/behavior/security/SBOM/provenance, then delete the temporary bridge files/flags rather than retaining two routing authorities.

Do not close this issue merely because the owner issue is Proposed or a branch exists. Completion requires immutable owner release plus exact-head consumer GREEN and ordinary protected integration.

Activity

  1. added
    bugSomething isn't working
    type: bugDefect or incorrect behavior
    on Sep 8, 2026
  2. seonghobae commented on Sep 8, 2026

    @seonghobae
    ContributorAuthor

    Fresh consumer/owner handoff — 2026-09-08 UTC

    Naruon #1602 remains an unchanged exact-head reproduction for this bridge-removal issue: ContextualWisdomLab/naruon@4450b97bc9e32a0150b7bd67888860d24f56f007, Required Noema Review run 34249618694, job 102140984266. The job used .github@7fd571dbcdbae6acf29d8f4ee704d7ba6297e4db, vendored contextual-orchestrator 414f22973658c4ddc3d4320fcf7acd9b4e8ba991, requested only model=orchestrator/free, and used caller attempts=1. Leaf preflight admitted 59 routes / selected 24 and reached one ready route plus a successful gateway preflight; the substantive verdict request nevertheless failed closed with HTTP 429 after 155.6 s (phase=response_error, served model deepseek-ai/deepseek-v4-flash-0731).

    Canonical owner contextual-orchestrator#1106 is still open/high-priority. Broad owner PR #971 has advanced non-force to 838cbcb77218f6a87c9ec222b5a26b0c9066ec3f, but that head still assigns admitted review agents priority=0 and constructs ModelClient(max_output_tokens=32768) while delegating subsequent selection to the existing routing layer. It has not published the request-scoped admission/fallback/provenance contract needed to delete this consumer's probing/selection bridge. Contextual-orchestrator currently has no GitHub Release, so there is no immutable version to bump here yet.

    Keep this issue fail-closed and source-neutral until the owner produces RED→GREEN plus an immutable release. Then create the consumer RED first, bump only that released version, delete the leaf eligibility/catalog/probing/readiness/account-counter/token-budget/priority/fallback logic, and replay unchanged Naruon #1602 through Noema/OpenCode/Strix. Do not add a consumer retry, provider/model pin, paid fallback, or another heuristic while waiting.

  3. seonghobae commented on Sep 8, 2026

    @seonghobae
    ContributorAuthor

    Fresh consumer evidence for the CO-release succession contract: Naruon #1612 exact 3da3ae8e60e1bb049f59ae86bfe82db12b7e3cc7, Required Noema Review run 34271416210 / job 102213908311, reusable workflow .github@7fd571dbcdbae6acf29d8f4ee704d7ba6297e4db, vendored CO 414f22973658c4ddc3d4320fcf7acd9b4e8ba991.

    orchestrator/free, caller attempts=1: 61 routes admitted, 24 selected, 3 reported ready; gateway preflight succeeded. The actual verdict request then failed after ~365.3 s with HTTP 502 on served model google/gemma-4-31b-it, so no current-head Noema verdict was published and the gate correctly failed closed. Leaf heuristics (account_skip_after_429=2, probe_budget=16, escalation_budget=4) are still present, but this run is further evidence not to grow them: #1602 already produced a post-preflight 429 and #1612 now produces a post-preflight 502.

    Keep this issue's succession unchanged: wait for the canonical CO request-scoped admission/routing/fallback/provenance fix and immutable release, bump that released version here, delete the redundant consumer probing/selection bridge, then replay the unchanged Naruon exact head. No provider/model pin, paid fallback, or synthetic success in the consumer.

  4. seonghobae commented on Sep 8, 2026

    @seonghobae
    ContributorAuthor

    Fresh consumer evidence from ContextualWisdomLab/naruon#1620 exact 3871aa4623e4fdcd2023e79ec7d959eb3e7afa51: Required Noema Review run 34287675066 / job 102267705820 used .github@7fd571dbcdbae6acf29d8f4ee704d7ba6297e4db, model=orchestrator/free, caller attempts=1, and no paid/local/direct-provider fallback. Leaf sidecar admitted 61 routes / selected 24; fixed bridge knobs were visible (probe_budget=16, account_skip_after_429=2, escalation_budget=4), with 1 ready route and successful gateway preflight. The substantive verdict then failed after 221.6 s with HTTP 429 (phase=response_error, served model deepseek-ai/deepseek-v4-flash-0731).

    This strengthens the existing cleanup requirement rather than justifying larger leaf budgets. Keep the bridge fail-closed; after contextual-orchestrator#1106 ships an immutable request-scoped contract, bump that release here and delete provider/credential eligibility, completion probing, readiness thresholds, account counters, token/escalation budgets, priority/fallback selection from the leaf. Naruon source remains unchanged for this owner failure.

  5. seonghobae commented on Sep 9, 2026

    @seonghobae
    ContributorAuthor

    Fresh consumer evidence to bind the post-owner-release cleanup: ContextualWisdomLab/naruon#1623@e8a54fc5156ac3ffbb79bc8418c5125d7dcdea60, Required Noema run 34308231412 / job 102329519489.

    The current bridge at .github@7fd571dbcdbae6acf29d8f4ee704d7ba6297e4db admitted 61, selected 24, probed 16, marked 1 ready and reported the gateway preflight ready, while still exposing account_skip_after_429=2, probe_budget=16, escalation_budget=4, target_ready=8. The actual unchanged-head verdict call (model=orchestrator/free, caller attempts=1) then failed after 214.0s with HTTP 429, phase=response_error, served deepseek-ai/deepseek-v4-pro-0813.

    Canonical owner handoff is now on contextual-orchestrator#1106 comment 5595655747. Preserve fail-closed behavior here; do not add another consumer retry/probe/provider/model heuristic. After an immutable CO release, bump that release and remove the leaf admission/probing/selection bridge, then replay Noema/OpenCode/Strix on the unchanged consumer head with exact-SHA behavior/security/SBOM/provenance evidence.

  6. seonghobae commented on Sep 9, 2026

    @seonghobae
    ContributorAuthor

    Fresh consumer acceptance evidence from ContextualWisdomLab/naruon#1623@e8a54fc5156ac3ffbb79bc8418c5125d7dcdea60:

    Strix required run 34308231478, job 102329702263, used .github@7fd571dbcdbae6acf29d8f4ee704d7ba6297e4db and CO 414f22973658c4ddc3d4320fcf7acd9b4e8ba991. orchestrator/free preflight admitted 61 routes, selected 24, probed 16 and marked 3 ready; gateway chat/completions preflight succeeded. The authoritative Strix 1.5.3 scan then ran against 3 changed files, with sandbox image ready, but after 1883 s ended LLM CONNECTION FAILED / request timeout. No vulnerability report was produced, so the gate correctly remained non-passing as STRIX_PROVIDER_UNAVAILABLE.

    This is not a reason to add another .github timeout/retry/probe/selection heuristic. It is a second consumer reproduction of the request-scoped availability gap already tracked in contextual-orchestrator#1106 (the same Naruon head's Noema request failed HTTP 429 after 214.0 s despite successful preflight). Preserve the current fail-closed single orchestrator/free invocation. After #1106 ships an immutable released contract, bump that exact release here and remove the redundant leaf admission/probing/selection bridge before replaying unchanged Naruon heads.

  7. seonghobae commented on Sep 9, 2026

    @seonghobae
    ContributorAuthor

    Fresh consumer canary for the post-owner-release bridge deletion: ContextualWisdomLab/Orgmetra#96@0e69f0799898b7e19edcf19bca2d2cf6483f65fe, Required Strix run 34343437671, job 102439501245.

    The job reached the real scanner path after exact-head admission/materialization, credential gate, contextual-orchestrator sidecar provisioning, Strix installation, and orchestrator/free preparation all succeeded. Immutable artifact strix-reports id 10102316354, digest sha256:95562de6303e1e8e135e7e63bb829e1a0c8ff4778842bc8e8517db144bcc4dc9, records the current bridge preflight as candidate_count=24, probe_budget=16, probed_count=16, ready_count=3, target_ready=8, account_skip_after_429=2; gateway preflight itself was ready in one attempt.

    The substantive Strix call then failed after 1,870 s with Request timed out and no vulnerability report. Sanitized sidecar evidence contains repeated ~90 s TimeoutError attempts against deepseek-ai/deepseek-v4-flash-0731, repeatedly using three-attempt sequences before provider_exhausted, with other ready-model traffic interleaved. This is incomplete model/transport evidence, not an Orgmetra security verdict; fail-closed behavior is correct.

    This should not grow the temporary consumer bridge. In particular, do not add a larger/smaller leaf timeout, another retry count, provider/model pin, paid fallback, or extra readiness threshold in response. Canonical RCA and the default-null model-timeout / request-scoped routing contract belong to contextual-orchestrator#1106. After an immutable owner release, create the consumer RED, bump that exact release here, delete leaf candidate discovery/probing/readiness/account counters/token/escalation/timeout/retry/fallback policy, then replay the unchanged Orgmetra head.

  8. seonghobae commented on Sep 9, 2026

    @seonghobae
    ContributorAuthor

    Fresh owner-dependency readback: contextual-orchestrator #971 has advanced to actual exact head d9c2f57701f6e7fe194582b510255759ffb30990 (its PR body still names an older checkpoint), but the current review_gateway.py still constructs all admitted review agents at priority=0 and ModelClient(max_output_tokens=32768) and exposes no typed request-scoped Fugu/Conductor/TRINITY allocation/provenance contract. I recorded the exact continuity finding on contextual-orchestrator#1106 as comment 5607242383.

    Same-head CO checks are not release-ready: Security Scan 34350400621, Security and Quality 34350400662, and SAST Semgrep 34350400657 succeed; CodeQL PR 34350400648 fails. Accordingly this consumer issue must remain fail-closed. Do not bump a mutable #971 branch or delete the leaf bridge yet. Required boundary remains immutable CO owner release first, then RED-first .github exact-release adoption/deletion and unchanged Noema/OpenCode/Strix consumer replay.

  9. seonghobae commented on Sep 10, 2026

    @seonghobae
    ContributorAuthor

    Fresh exact consumer evidence, 2026-09-10, from OriginWeave#229@7ec83c1be1a8e8724d37c2d6ebbdb215b1b10e23:

    • OpenCode required run 34428243675 / job 102723776992: bootstrap, superseded-run cleanup and exact-head admission succeeded; the request for current-head review execution succeeded; the job then failed closed because no current-head OpenCode verdict was available.
    • Noema required run 34428243682 / job 102720491195: trusted Noema source, exact-head validation, credential selection and repository-scoped token succeeded. The contextual-orchestrator review sidecar provisioned successfully, then Prepare Noema model verdict failed; failure evidence upload succeeded and verdict publication was skipped.
    • Same OriginWeave head: native CI 34428243419, Security Scan 34428243446, SAST Semgrep 34428243481 and Strix 34428243671 are GREEN. Required CodeQL 34428243635 is still separately in progress at this observation.

    This is a useful consumer canary for #1106: preserve fail-closed behavior and sanitized evidence, but do not add leaf provider/model/group routing, retry/timeout heuristics, paid fallback or copied CO source. After an immutable CO release, delete the temporary routing/admission bridge and regenerate the exact-head reviews.

  10. seonghobae commented on Sep 11, 2026

    @seonghobae
    ContributorAuthor

    Fresh TEPP consumer canary: ContextualWisdomLab/TEPP#310@ecbe7aa7ace9094e7631d59d942869594c9d7d00. Required Noema run 34181514731, original job 101923023946, verified exact live head and repository-scoped reviewer token, provisioned contextual-orchestrator sidecar pinned at 414f22973658c4ddc3d4320fcf7acd9b4e8ba991, and reached gateway preflight with model: orchestrator/free. The model phase then failed closed after 206.8 s with HTTP Error 502: Bad Gateway, served_model meta/llama-3.2-11b-vision-instruct; no Noema verdict was published. Sidecar evidence artifact 10039548659, archive SHA256 4b5cc2faa70bc914354d0570abbdb6b782645c1173ccefaa2bec49fd25425c44. A lane-local rerun attempt 2 (103208877918) is queued on the unchanged head. This is additional review-gateway availability/ownership evidence, not a TEPP source finding; no leaf provider/model fallback or gate weakening is being introduced.

  11. seonghobae commented on Sep 11, 2026

    @seonghobae
    ContributorAuthor

    Fresh unchanged-head TEPP consumer recovery, 2026-09-11 KST. ContextualWisdomLab/TEPP#310@ecbe7aa7ace9094e7631d59d942869594c9d7d00, Required Noema Review run 34181514731 attempt 2, job 103208877918 completed success without a consumer source change.

    The job successfully passed exact-live-head admission, provisioned the contextual-orchestrator review sidecar, and completed Prepare Noema model verdict. The predecessor HTTP 502 therefore does not reproduce on this unchanged consumer head. Because TEPP #310 is Draft, Publish prepared Noema verdict on the exact live head was skipped, so this is gateway/runtime recovery evidence only; it is not a published Noema review or approval.

    This narrows the earlier 502 to a transient execution failure but does not close #2042. The central leaf still owns the temporary routing/admission bridge described here and contextual-orchestrator still lacks the immutable released free-pool contract required by this issue. Do not convert this recovery into a leaf timeout/provider fallback or treat Draft-suppressed publication as review evidence. Owner GREEN remains: immutable CO release -> consumer bump/removal of leaf routing authority -> Ready current-head execution with a published exact-head verdict and normal protected integration.

  12. seonghobae commented on Sep 13, 2026

    @seonghobae
    ContributorAuthor

    Fresh Strix security evidence on current central review/bootstrap code exposes a concrete credential-destination binding defect in the temporary leaf bridge this issue already owns.

    Exact evidence: .github#2106@611ccd73460ab0188e0085956ade6180bb28a91a, Strix run 34742109546, job 103684345530, artifact 10314266999, digest sha256:a05a1bd70db3383ecb0c9d4755e92416bdd3426b7cbe9e632cb7bef6cc69da11. Finding is CWE-915 / low severity in protected-main scripts/ci/contextual_orchestrator_review_policy.py.

    parse_discovery_report validates provider identity and credential_key, but currently lets discovery rows override the trusted provider registry through:

    "base_url": row.get("base_url") or PROVIDER_BASE_URLS[provider],
    "auth_scheme": row.get("auth_scheme") or PROVIDER_AUTH_SCHEMES[provider],

    Those values are later emitted to the runtime catalog. Thus an attacker-influenced discovery report can keep the expected credential name while redirecting the credentialed route to an arbitrary base_url / auth scheme. That weakens source-identity ↔ destination-identity binding at exactly the temporary routing bridge #2042 intends to retire.

    Canonical repair direction: first add a focused RED contract proving a registered provider row cannot alter destination/auth identity; then, while this bridge still exists, derive base_url and auth_scheme only from the trusted provider registry (or a strictly versioned provider-specific allowlist if overrides are genuinely part of an owner contract). Do not copy this bridge into consumers, add direct provider fallbacks, or wait for eventual CO retirement while leaving a credential-redirection path unguarded.

    The eventual #2042 GREEN remains owner release → immutable consumer bump → bridge deletion, but the interim bridge must remain safe until then.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingpriority: highHigh-priority or P1 worktype: bugDefect or incorrect behavior

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions