Skip to content

fix(core): derive the OpenCode Free model set from the models.dev snapshot - #3429

Open
cunninghamcard-bit wants to merge 4 commits into
apache:mainfrom
cunninghamcard-bit:fix/opencode-free-derive-from-snapshot
Open

fix(core): derive the OpenCode Free model set from the models.dev snapshot#3429
cunninghamcard-bit wants to merge 4 commits into
apache:mainfrom
cunninghamcard-bit:fix/opencode-free-derive-from-snapshot

Conversation

@cunninghamcard-bit

@cunninghamcard-bit cunninghamcard-bit commented Aug 21, 2026

Copy link
Copy Markdown

Summary

The opencode-free provider pinned four free model ids as a compile-time constant; the free tier rotates fast, three of the four now fail live probes, and new free models never surfaced. #2165 already swapped this pin once by hand, so this fixes the mechanism, in three layers with different owners:

  • Eligibility (changes weekly) — scripts/sync-model-metadata.mjs marks models.dev zero-input-cost models isFree; the candidate set is derived as isFree && functionCalling && !deprecated && !broken, default model first, with a module-load guard. Rotation now flows through routine snapshot refreshes.
  • Served (changes daily) — opencode-free switches to the existing protocol discovery (anonymous /models, no credential) intersected with the candidates via the existing fallback-models filter; failure writes nothing and reads keep the candidates, so offline behaves exactly as before. Storage pre-fills the candidate inventory at create for credential-free providers backed by a models.dev catalog, keeping a fresh install usable before its first fetch.
  • Responsive (changes per request) — mostly out of scope (see below), except one evidence-backed quarantine: muse-spark-1.2-contributor-free persistently answers HTTP 200 with an empty completion while billing the full token budget (4 consecutive anonymous probes, 2026-08-21, max_tokens 8–200). It is deny-listed with that evidence and refused wherever a model can be offered or executed — catalog and execution admission alike, including inventories stored before the entry existed. Deny-only: a stale entry hides at most one healthy model, the opposite failure mode of the old allow-list pin.

Existing installs heal on upgrade, split by data ownership: enabledModelIds is user-editable, so bootstrap migrates it only when it still exactly matches a historical system seed (lossy heuristic by design; the versioned seed policy from #3354 is the durable replacement), re-seeding the default target only when the migration itself nulled it. Stored non-fetched models lists are caches of the writing build's statics, so connection updates refresh them at the existing write seam and an inventory change invalidates lastTest. The document stays the single authority every read serves, so the picker and execution admission cannot disagree.

Known limits, deliberately out of scope: transient 429/5xx (per-IP quota is a point-in-time signal — probes showed the same id failing and succeeding within hours, so no list can encode it) stays with a Runtime Host health/failover follow-up; the pre-existing deepseek admission carve-out and the enabled-ids-vs-inventory drift on user-modified rows are unchanged.

Fixes #3409

Verification

  • Full workspace typecheck and test suite pass (10 workspaces, 0 failures); biome clean on changed files. New tests: seed migration (migrated / user-modified incl. reordered / repeat-bootstrap / default repair), inventory refresh on update, execution-read inventory, quarantine admission.
  • Live anonymous probes: fetchProviderModels('opencode-free') returns the served∩candidates intersection (7 ids at probe time); side-by-side fresh desktop builds show pre-fix seeding the stale 3-model set and post-fix seeding the derived set with the default model answering normally.

AI use

Select exactly one:

  • No generative tool made a substantive contribution
  • Generative tooling made a substantive contribution

Tool(s) and scope: Claude Code (Claude Fable 5) authored the code and tests across all four commits, ran the snapshot refresh and live endpoint probes, and drafted this description; adversarial verification used Claude subagent workflows and an OpenAI Codex CLI review, whose confirmed findings (stored-inventory quarantine gap, lastTest basis, default-target repair) are incorporated. The human contributor of record reviewed the changes and verified both desktop builds. All commits carry the Generated-by: Claude Code trailer.

Checklist

  • Tests cover the change and fail without it
  • Lint, format, typecheck and the affected suites pass locally

Does this PR entail a change in behavior?

  • Yes — described under Summary above
  • No

…pshot

The opencode-free provider pinned four model ids as a compile-time
constant. The free tier rotates fast: three of the four now fail live
probes (429 FreeUsageLimitError, 400 model unavailable) and newer
working free models never surfaced, because nothing refreshed the pin.

The metadata sync now marks models.dev zero-input-cost models with
isFree, and opencodeFreeModelIds / OPENCODE_FREE_DEFAULT_ENABLED_MODELS
become a derivation over the snapshot (isFree, tool-capable, not
deprecated, default model first) with a module-load guard that fails
loudly if the snapshot ever drops the default. Free models now rotate
in and out with routine snapshot refreshes instead of rotting in a pin.

Also refreshes the models.dev snapshot (2026-08-21), which prunes the
deprecated free ids and surfaces x-preview-f-free; the Vercel thinking
test moves to openai/gpt-5.1-thinking because upstream renamed
xai/grok-4.3 to spacexai/grok-4.3 and dropped its reasoning options.

Fixes apache#3409

Generated-by: Claude Code
@cunninghamcard-bit
cunninghamcard-bit force-pushed the fix/opencode-free-derive-from-snapshot branch from 8e36626 to 0cbaba4 Compare August 21, 2026 13:56

@Astro-Han Astro-Han left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for removing the hand-maintained OpenCode Free pin and reusing the generated metadata/provider-registry seam. That is the right direction and avoids another one-off model-list patch.

I left two P2 comments because the current implementation still conflates the snapshot candidate catalog with live anonymous availability, and it only updates fresh or newly created connections. Neither issue requires another static allowlist or provider-specific renderer path.

The clean final state would keep the responsibilities separate:

  1. models.dev owns free/capability/lifecycle candidate facts;
  2. the existing discovery seam intersects those candidates with the anonymously served catalog;
  3. Runtime Host owns connection-local health, fallback, and persisted selection;
  4. a versioned built-in seed migration updates untouched legacy defaults without overwriting user choices.

I am leaving this as a non-blocking COMMENT rather than requesting changes, but I do not think the current head fully resolves #3409 yet.

AI-assisted review disclosure: OpenAI Codex coordinated two independent reviewer agents to inspect the exact-head diff, related Runtime Host and execution paths, live anonymous endpoint behavior, tests, and severity calibration. I reviewed and approved these comments before posting.

Comment thread packages/core/src/provider-registry.ts Outdated
'OpenCode Free',
Object.fromEntries(
Object.entries(GENERATED_MODELS_DEV_METADATA.opencode).filter(
([, model]) => model.isFree === true && model.lifecycle !== 'deprecated',

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Thanks for replacing the manually maintained pin with structured metadata. The remaining issue is that this filter answers “free, tool-capable, and not deprecated,” but the result is also used as the set that is currently runnable and enabled by default. Those are different contracts: on the current exact head, anonymous probes still return 429 for big-pickle and mimo-v2.5-free, while muse-spark-1.2-contributor-free does not produce a usable completion. The fallback probing in testConnection only runs for an explicit connection test; it neither updates enabledModelIds nor participates in normal turns.

Could we keep this derived set as the candidate inventory, but separate it from the default-enabled set? The smallest safe version would seed only the known-working default model. For the clean final state, models.dev should own price/capability/lifecycle eligibility, anonymous /models discovery should intersect the served catalog with those candidates, and Runtime Host should own per-connection health and reversible fallback when inference returns availability or rate-limit failures. That keeps the dynamic facts at the existing Host seam instead of encoding them into a global snapshot.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks — agreed the two contracts were conflated. Addressed in 2f2e980: OPENCODE_FREE_DEFAULT_ENABLED_MODELS now seeds only the probe-verified default model, while the derived set stays as the opt-in candidate inventory in fallbackModels. The fuller separation you describe (models.dev eligibility ∩ anonymously served catalog at the discovery seam, with Runtime Host owning connection-local health and reversible fallback) matches where we'd like this to go too; leaving that as a follow-up since it touches the discovery and Host seams beyond this fix.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correction and a consolidated plan — apologies for the churn on this thread (2f2e980 narrowed the seed to one model, 62de05b reverted it; both were wrong ways to answer your point, and I should have settled the design here first).

Fresh anonymous probes just now: 5 of 7 candidates complete successfully — including both big-pickle and mimo-v2.5-free that probed 429 earlier (FreeUsageLimitError is per-IP and transient), while nemotron-3-ultra-free itself was failing with a transient upstream 502 at probe time. So availability is a point-in-time, per-IP signal: encoding any probe snapshot into the compile-time constant is wrong in both directions.

Which is, I now understand, your actual point: eligibility and availability are different contracts, and availability must be resolved at runtime. Final shape for this PR:

  • models.dev snapshot keeps owning eligibility (free, tool-capable, not deprecated) — unchanged;
  • the connection's model list is resolved at the existing discovery seam as candidates ∩ anonymously served /zen/v1/models, falling back to the static candidates when the endpoint is unreachable — this removes the "declared free but not anonymously served" ids;
  • transient 429/5xx per-request failures stay a Runtime Host health concern (your item 3), which I'd leave to the Persisted connection catalogs keep model ids that the snapshot has retired #3354-adjacent follow-up rather than bolt onto this PR.

Implementing the intersection now; will push it as one change.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Landed in 2ecd384 (with 62de05b), completing the plan above: eligibility from the snapshot; the connection's list resolved at the existing discovery seam as candidates ∩ anonymously served /zen/v1/models (anonymous, filter: 'fallback-models'), falling back to the static candidates when unreachable.

One disclosure beyond that plan: your muse-spark observation was confirmed and is worse than an availability blip — it persistently answers HTTP 200 with an empty completion while billing the full token budget (4 consecutive anonymous probes, max_tokens 8–200). That failure shape is invisible to both the /models intersection and any future 429/5xx health seam, so it is quarantined via an evidence-dated deny-list enforced at catalog AND execution admission (including inventories stored before the entry existed) — deny-only, the opposite failure mode of the old pin. Transient per-request failures (429 quota, upstream 5xx) remain with the Runtime Host health/failover follow-up you sketched as item 3.

]);
// The free set is derived from the models.dev snapshot and rotates with
// refreshes; assert the structural contract, not today's ids.
assert.deepEqual(free?.enabledModelIds, [...OPENCODE_FREE_DEFAULT_ENABLED_MODELS]);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Thanks for covering the fresh-Host wiring here. This does not cover the normal upgrade path, though: ensureBootstrapRuntimePolicy() returns as soon as any catalog already exists, and ensureConnection() preserves an existing opencode-free row unchanged. Users upgrading from the old seed therefore retain the retired/dead IDs and do not receive the newly derived candidates.

The clean fix should be a versioned built-in seed policy in the Runtime Host catalog authority: migrate only a connection that can still be proven to match the previous system-owned seed, make the migration idempotent and revision-safe, and always preserve a user-modified enabledModelIds selection. If that belongs in #3354 instead, could this PR explicitly limit its claim to fresh/new connections and avoid closing the existing-user part of #3409?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch — fresh installs only was a real gap. Addressed in 2f2e980 with a seed migration that avoids a version field: bootstrap now migrates an existing opencode-free row only when its enabledModelIds still exactly match one of the three historical system seeds (['big-pickle'] from #1720, ['nemotron-3-ultra-free'] from #2166, or the three-model list from #2431) — exact match is the proof it is still system-owned. Any other inventory is treated as a user selection and never touched. The migration is idempotent (no-op once the row equals the current seed) and commits through the catalog's revision CAS. Tests cover the migrated, user-modified, and repeat-bootstrap cases. If maintainers prefer the versioned seed-policy design from #3354 instead, happy to align — this version keeps the provable-match property without new persisted state.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Completed in 2ecd384. On top of the exact-match seed migration described above (unchanged: three verbatim historical seeds, revision-CAS, idempotent, user-modified rows — including reordered lists — never touched), the migration now also repairs the default target when the migration itself nulled it (a previously cleared default stays cleared), and the update write refreshes the stored non-fetched models inventory to the current build's candidates, so a migrated row's picker and execution admission agree. The exact-match enumeration is documented as deliberately lossy — every seed change must append the prior value — with the versioned seed policy from #3354 as the durable replacement; happy to align this to that design when it lands.

…ed legacy opencode-free rows

Review follow-up on apache#3429:

The snapshot-derived free set answers candidate eligibility (free,
tool-capable, not deprecated), not live anonymous availability — several
candidates still 429/400 on anonymous probes. Default-enable only the
probe-verified default model and keep the derived set as opt-in
candidate inventory in fallbackModels.

Existing installs never re-ran bootstrap seeding, so upgrades kept
retired ids. Bootstrap now migrates an opencode-free row whose
enabledModelIds still exactly match a historical system seed
(['big-pickle'], ['nemotron-3-ultra-free'], or the three-model list) to
the current seed; any other inventory is a user selection and is never
touched. The migration is idempotent and commits through the catalog's
revision CAS.

Generated-by: Claude Code
…free

Narrowing the default-enabled set to one probe-verified model encoded a
point-in-time availability snapshot into a compile-time constant — the
same mistake in the opposite direction. Fresh anonymous probes complete
successfully on 5 of 7 candidates including both ids the earlier probe
reported as 429 (per-IP FreeUsageLimitError is transient), while the
single retained default was itself failing with a transient upstream
502 at probe time. Availability cannot be pinned; per-model health
belongs to a runtime seam. The legacy-seed migration is unchanged and
follows the constant.

Generated-by: Claude Code
…mous catalog and heal stale inventories

Review follow-up on apache#3429, closing the free-vs-actually-usable gap:

Eligibility stays with the models.dev snapshot, minus a deny-list of
persistently broken models with dated probe evidence (muse-spark
answers 200 with an empty completion and bills the full token budget).
The quarantine is enforced wherever a model can be offered or executed
— catalog entries and execution admission alike — so ids carried by
inventories stored before the entry was added are refused too, not just
excluded from fresh derivations. Deny-only: a stale entry hides at most
one healthy model, the opposite failure mode of the allow-list pin this
replaced.

Serving moves to the existing discovery seam: opencode-free switches to
protocol discovery (anonymous /models, no credential) intersected with
the snapshot candidates via the existing fallback-models filter, and
storage pre-fills the candidate inventory at create for credential-free
providers backed by a models.dev catalog, so a fresh install is usable
before its first fetch. Discovery failure writes nothing.

A stored non-fetched model list is a cache of the writing build's
static inventory, not wire data: connection updates refresh it at the
existing write seam (which already owned the endpoint-change reset),
fetched results are preserved verbatim, and an inventory change
invalidates lastTest — a verified badge must not outlive the inventory
it was tested against. The document stays the single authority every
read serves, so the picker and execution admission cannot disagree.

The legacy-seed migration gains the pre-deny-list derived seed as a
fourth historical value; its update write refreshes the stored
inventory, and it re-seeds the default target only when the migration
itself nulled it (a previously cleared default stays cleared), with
bounded retries.

Per-request availability (429 quota, transient upstream errors) remains
a Runtime Host health concern for a follow-up.

Generated-by: Claude Code
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug(core): OpenCode Free pinned model list has rotted — 3 of 4 pinned models fail (429/400) and new free models never appear

2 participants