fix(core): derive the OpenCode Free model set from the models.dev snapshot - #3429
fix(core): derive the OpenCode Free model set from the models.dev snapshot#3429cunninghamcard-bit wants to merge 4 commits into
Conversation
…pshot The opencode-free provider pinned four model ids as a compile-time constant. The free tier rotates fast: three of the four now fail live probes (429 FreeUsageLimitError, 400 model unavailable) and newer working free models never surfaced, because nothing refreshed the pin. The metadata sync now marks models.dev zero-input-cost models with isFree, and opencodeFreeModelIds / OPENCODE_FREE_DEFAULT_ENABLED_MODELS become a derivation over the snapshot (isFree, tool-capable, not deprecated, default model first) with a module-load guard that fails loudly if the snapshot ever drops the default. Free models now rotate in and out with routine snapshot refreshes instead of rotting in a pin. Also refreshes the models.dev snapshot (2026-08-21), which prunes the deprecated free ids and surfaces x-preview-f-free; the Vercel thinking test moves to openai/gpt-5.1-thinking because upstream renamed xai/grok-4.3 to spacexai/grok-4.3 and dropped its reasoning options. Fixes apache#3409 Generated-by: Claude Code
8e36626 to
0cbaba4
Compare
Astro-Han
left a comment
There was a problem hiding this comment.
Thanks for removing the hand-maintained OpenCode Free pin and reusing the generated metadata/provider-registry seam. That is the right direction and avoids another one-off model-list patch.
I left two P2 comments because the current implementation still conflates the snapshot candidate catalog with live anonymous availability, and it only updates fresh or newly created connections. Neither issue requires another static allowlist or provider-specific renderer path.
The clean final state would keep the responsibilities separate:
- models.dev owns free/capability/lifecycle candidate facts;
- the existing discovery seam intersects those candidates with the anonymously served catalog;
- Runtime Host owns connection-local health, fallback, and persisted selection;
- a versioned built-in seed migration updates untouched legacy defaults without overwriting user choices.
I am leaving this as a non-blocking COMMENT rather than requesting changes, but I do not think the current head fully resolves #3409 yet.
AI-assisted review disclosure: OpenAI Codex coordinated two independent reviewer agents to inspect the exact-head diff, related Runtime Host and execution paths, live anonymous endpoint behavior, tests, and severity calibration. I reviewed and approved these comments before posting.
| 'OpenCode Free', | ||
| Object.fromEntries( | ||
| Object.entries(GENERATED_MODELS_DEV_METADATA.opencode).filter( | ||
| ([, model]) => model.isFree === true && model.lifecycle !== 'deprecated', |
There was a problem hiding this comment.
[P2] Thanks for replacing the manually maintained pin with structured metadata. The remaining issue is that this filter answers “free, tool-capable, and not deprecated,” but the result is also used as the set that is currently runnable and enabled by default. Those are different contracts: on the current exact head, anonymous probes still return 429 for big-pickle and mimo-v2.5-free, while muse-spark-1.2-contributor-free does not produce a usable completion. The fallback probing in testConnection only runs for an explicit connection test; it neither updates enabledModelIds nor participates in normal turns.
Could we keep this derived set as the candidate inventory, but separate it from the default-enabled set? The smallest safe version would seed only the known-working default model. For the clean final state, models.dev should own price/capability/lifecycle eligibility, anonymous /models discovery should intersect the served catalog with those candidates, and Runtime Host should own per-connection health and reversible fallback when inference returns availability or rate-limit failures. That keeps the dynamic facts at the existing Host seam instead of encoding them into a global snapshot.
There was a problem hiding this comment.
Thanks — agreed the two contracts were conflated. Addressed in 2f2e980: OPENCODE_FREE_DEFAULT_ENABLED_MODELS now seeds only the probe-verified default model, while the derived set stays as the opt-in candidate inventory in fallbackModels. The fuller separation you describe (models.dev eligibility ∩ anonymously served catalog at the discovery seam, with Runtime Host owning connection-local health and reversible fallback) matches where we'd like this to go too; leaving that as a follow-up since it touches the discovery and Host seams beyond this fix.
There was a problem hiding this comment.
Correction and a consolidated plan — apologies for the churn on this thread (2f2e980 narrowed the seed to one model, 62de05b reverted it; both were wrong ways to answer your point, and I should have settled the design here first).
Fresh anonymous probes just now: 5 of 7 candidates complete successfully — including both big-pickle and mimo-v2.5-free that probed 429 earlier (FreeUsageLimitError is per-IP and transient), while nemotron-3-ultra-free itself was failing with a transient upstream 502 at probe time. So availability is a point-in-time, per-IP signal: encoding any probe snapshot into the compile-time constant is wrong in both directions.
Which is, I now understand, your actual point: eligibility and availability are different contracts, and availability must be resolved at runtime. Final shape for this PR:
- models.dev snapshot keeps owning eligibility (free, tool-capable, not deprecated) — unchanged;
- the connection's model list is resolved at the existing discovery seam as candidates ∩ anonymously served
/zen/v1/models, falling back to the static candidates when the endpoint is unreachable — this removes the "declared free but not anonymously served" ids; - transient 429/5xx per-request failures stay a Runtime Host health concern (your item 3), which I'd leave to the Persisted connection catalogs keep model ids that the snapshot has retired #3354-adjacent follow-up rather than bolt onto this PR.
Implementing the intersection now; will push it as one change.
There was a problem hiding this comment.
Landed in 2ecd384 (with 62de05b), completing the plan above: eligibility from the snapshot; the connection's list resolved at the existing discovery seam as candidates ∩ anonymously served /zen/v1/models (anonymous, filter: 'fallback-models'), falling back to the static candidates when unreachable.
One disclosure beyond that plan: your muse-spark observation was confirmed and is worse than an availability blip — it persistently answers HTTP 200 with an empty completion while billing the full token budget (4 consecutive anonymous probes, max_tokens 8–200). That failure shape is invisible to both the /models intersection and any future 429/5xx health seam, so it is quarantined via an evidence-dated deny-list enforced at catalog AND execution admission (including inventories stored before the entry existed) — deny-only, the opposite failure mode of the old pin. Transient per-request failures (429 quota, upstream 5xx) remain with the Runtime Host health/failover follow-up you sketched as item 3.
| ]); | ||
| // The free set is derived from the models.dev snapshot and rotates with | ||
| // refreshes; assert the structural contract, not today's ids. | ||
| assert.deepEqual(free?.enabledModelIds, [...OPENCODE_FREE_DEFAULT_ENABLED_MODELS]); |
There was a problem hiding this comment.
[P2] Thanks for covering the fresh-Host wiring here. This does not cover the normal upgrade path, though: ensureBootstrapRuntimePolicy() returns as soon as any catalog already exists, and ensureConnection() preserves an existing opencode-free row unchanged. Users upgrading from the old seed therefore retain the retired/dead IDs and do not receive the newly derived candidates.
The clean fix should be a versioned built-in seed policy in the Runtime Host catalog authority: migrate only a connection that can still be proven to match the previous system-owned seed, make the migration idempotent and revision-safe, and always preserve a user-modified enabledModelIds selection. If that belongs in #3354 instead, could this PR explicitly limit its claim to fresh/new connections and avoid closing the existing-user part of #3409?
There was a problem hiding this comment.
Good catch — fresh installs only was a real gap. Addressed in 2f2e980 with a seed migration that avoids a version field: bootstrap now migrates an existing opencode-free row only when its enabledModelIds still exactly match one of the three historical system seeds (['big-pickle'] from #1720, ['nemotron-3-ultra-free'] from #2166, or the three-model list from #2431) — exact match is the proof it is still system-owned. Any other inventory is treated as a user selection and never touched. The migration is idempotent (no-op once the row equals the current seed) and commits through the catalog's revision CAS. Tests cover the migrated, user-modified, and repeat-bootstrap cases. If maintainers prefer the versioned seed-policy design from #3354 instead, happy to align — this version keeps the provable-match property without new persisted state.
There was a problem hiding this comment.
Completed in 2ecd384. On top of the exact-match seed migration described above (unchanged: three verbatim historical seeds, revision-CAS, idempotent, user-modified rows — including reordered lists — never touched), the migration now also repairs the default target when the migration itself nulled it (a previously cleared default stays cleared), and the update write refreshes the stored non-fetched models inventory to the current build's candidates, so a migrated row's picker and execution admission agree. The exact-match enumeration is documented as deliberately lossy — every seed change must append the prior value — with the versioned seed policy from #3354 as the durable replacement; happy to align this to that design when it lands.
…ed legacy opencode-free rows Review follow-up on apache#3429: The snapshot-derived free set answers candidate eligibility (free, tool-capable, not deprecated), not live anonymous availability — several candidates still 429/400 on anonymous probes. Default-enable only the probe-verified default model and keep the derived set as opt-in candidate inventory in fallbackModels. Existing installs never re-ran bootstrap seeding, so upgrades kept retired ids. Bootstrap now migrates an opencode-free row whose enabledModelIds still exactly match a historical system seed (['big-pickle'], ['nemotron-3-ultra-free'], or the three-model list) to the current seed; any other inventory is a user selection and is never touched. The migration is idempotent and commits through the catalog's revision CAS. Generated-by: Claude Code
…free Narrowing the default-enabled set to one probe-verified model encoded a point-in-time availability snapshot into a compile-time constant — the same mistake in the opposite direction. Fresh anonymous probes complete successfully on 5 of 7 candidates including both ids the earlier probe reported as 429 (per-IP FreeUsageLimitError is transient), while the single retained default was itself failing with a transient upstream 502 at probe time. Availability cannot be pinned; per-model health belongs to a runtime seam. The legacy-seed migration is unchanged and follows the constant. Generated-by: Claude Code
…mous catalog and heal stale inventories Review follow-up on apache#3429, closing the free-vs-actually-usable gap: Eligibility stays with the models.dev snapshot, minus a deny-list of persistently broken models with dated probe evidence (muse-spark answers 200 with an empty completion and bills the full token budget). The quarantine is enforced wherever a model can be offered or executed — catalog entries and execution admission alike — so ids carried by inventories stored before the entry was added are refused too, not just excluded from fresh derivations. Deny-only: a stale entry hides at most one healthy model, the opposite failure mode of the allow-list pin this replaced. Serving moves to the existing discovery seam: opencode-free switches to protocol discovery (anonymous /models, no credential) intersected with the snapshot candidates via the existing fallback-models filter, and storage pre-fills the candidate inventory at create for credential-free providers backed by a models.dev catalog, so a fresh install is usable before its first fetch. Discovery failure writes nothing. A stored non-fetched model list is a cache of the writing build's static inventory, not wire data: connection updates refresh it at the existing write seam (which already owned the endpoint-change reset), fetched results are preserved verbatim, and an inventory change invalidates lastTest — a verified badge must not outlive the inventory it was tested against. The document stays the single authority every read serves, so the picker and execution admission cannot disagree. The legacy-seed migration gains the pre-deny-list derived seed as a fourth historical value; its update write refreshes the stored inventory, and it re-seeds the default target only when the migration itself nulled it (a previously cleared default stays cleared), with bounded retries. Per-request availability (429 quota, transient upstream errors) remains a Runtime Host health concern for a follow-up. Generated-by: Claude Code
Summary
The
opencode-freeprovider pinned four free model ids as a compile-time constant; the free tier rotates fast, three of the four now fail live probes, and new free models never surfaced. #2165 already swapped this pin once by hand, so this fixes the mechanism, in three layers with different owners:scripts/sync-model-metadata.mjsmarks models.dev zero-input-cost modelsisFree; the candidate set is derived asisFree && functionCalling && !deprecated && !broken, default model first, with a module-load guard. Rotation now flows through routine snapshot refreshes./models, no credential) intersected with the candidates via the existingfallback-modelsfilter; failure writes nothing and reads keep the candidates, so offline behaves exactly as before. Storage pre-fills the candidate inventory at create for credential-free providers backed by a models.dev catalog, keeping a fresh install usable before its first fetch.muse-spark-1.2-contributor-freepersistently answers HTTP 200 with an empty completion while billing the full token budget (4 consecutive anonymous probes, 2026-08-21, max_tokens 8–200). It is deny-listed with that evidence and refused wherever a model can be offered or executed — catalog and execution admission alike, including inventories stored before the entry existed. Deny-only: a stale entry hides at most one healthy model, the opposite failure mode of the old allow-list pin.Existing installs heal on upgrade, split by data ownership:
enabledModelIdsis user-editable, so bootstrap migrates it only when it still exactly matches a historical system seed (lossy heuristic by design; the versioned seed policy from #3354 is the durable replacement), re-seeding the default target only when the migration itself nulled it. Stored non-fetchedmodelslists are caches of the writing build's statics, so connection updates refresh them at the existing write seam and an inventory change invalidateslastTest. The document stays the single authority every read serves, so the picker and execution admission cannot disagree.Known limits, deliberately out of scope: transient 429/5xx (per-IP quota is a point-in-time signal — probes showed the same id failing and succeeding within hours, so no list can encode it) stays with a Runtime Host health/failover follow-up; the pre-existing deepseek admission carve-out and the enabled-ids-vs-inventory drift on user-modified rows are unchanged.
Fixes #3409
Verification
fetchProviderModels('opencode-free')returns the served∩candidates intersection (7 ids at probe time); side-by-side fresh desktop builds show pre-fix seeding the stale 3-model set and post-fix seeding the derived set with the default model answering normally.AI use
Select exactly one:
Tool(s) and scope: Claude Code (Claude Fable 5) authored the code and tests across all four commits, ran the snapshot refresh and live endpoint probes, and drafted this description; adversarial verification used Claude subagent workflows and an OpenAI Codex CLI review, whose confirmed findings (stored-inventory quarantine gap, lastTest basis, default-target repair) are incorporated. The human contributor of record reviewed the changes and verified both desktop builds. All commits carry the
Generated-by: Claude Codetrailer.Checklist
Does this PR entail a change in behavior?