fix(provider-setup): enable the model the run targets, not every model (#1666)

[Victor-w-Madeira]

Closes #1666

Verdict: writes lost, reader right, product not at fault

#1666 left two readings open — the writes were genuinely lost, or the server-side
confirmation added the day before could not report anything else — and required the
product to be ruled out first.

Reproduced on a clean langflowai/langflow-nightly:latest (1.12.0.dev45), local, no
load, --workers=1: 0 of 36 / 0 of 9 / 0 of 30, identical to the daily. So the
environment was not the cause.

With every enabled_models call logged, the sweep's 74 toggles produced ZERO
POST /api/v1/models/enabled_models for its whole duration. Langflow never received a
write, so this is not a product regression, and the #1651 reader is right in
substance. useModelToggleQueue batches behind a 1000 ms debounce and cancels +
discards the pending batch on both paths the sweep takes — its unmount cleanup
("an explicit close already consumes its batch through flushPendingChanges before
unmount") and its identity effect, which fires the moment the selected provider
changes. The loop clicks faster than the debounce, so the timer never fired mid-loop.

Only the last provider's batch left at all, ~5 s after the sweep had moved on — which
is why #1651's read reported 0 of 30 for a google whose 30 models were enabled seconds
later. One of the three readings was a race; the other two were real.

Why the fix is not "make the 74 writes land"

That was prototyped and rejected on measured cost. update_enabled_models calls
validate_model_provider_key once per model, synchronously, inside the request, and
the lanes run LANGFLOW_WORKERS=1. On an idle container:

Operation	Cost
enable 1 model	0.42–1.01 s
enable 30 models	103 s
disable the same 30	0.02 s (no validation on that path)

All 74 would cost ~250 s of blocked backend per sweep, twice per shard — the wedge the
lanes already carry a health gate for (#922/#927/#1045). The prototype confirmed it: the
sweep went 40 s → 120 s+, google's write was unanswered at 45 s, and the confirmation read
then timed out as well.

That per-model validation is also the mechanism behind the eight 1 write(s) started, 0 finished give-ups #1672 taught the picker read to name instead of blaming
MODEL_PICKER_DEFECT.

What this does

The sweep now only reads model names for models.json. ensureTargetModelsEnabled
writes the one model per active provider that the key axis settled on — all any spec
picks — then confirms it against the server. Measured ~2 s, and the sweep is faster
than before (22–35 s against 40 s) because it no longer clicks 74 toggles for nothing. The
server ends at the five MIN_DEFAULT_MODELS defaults plus the target.

The write outranks a negative read. Two states leave a model reading "not enabled" and
only one is a defect: a 2xx write plus a model still off is real; a write that never
answered 2xx means the read describes the state before an enable that never happened. On
2026-09-01 the daily measured 19 outages and four WORKER TIMEOUT → SIGKILL cycles, so a
dropped write is the shape of a bad day — failing on it would redden pr-validation.yml's
whole E2E job, where this pre-flight is a hard gate. A refused write is in that branch
too: 400 Cannot enable not supported model: o3 is definite and reachable whenever the key
axis settles on one of OpenAI's not_supported entries, but its cause is the candidate
choice, so it is unknown carrying the server's own words rather than "the enable did not
take effect".

providers.json records four states, not a boolean — off (listed, disabled) and
absent (not listed at all: a name/catalog mismatch, or a policy-hidden provider) send a
reader to different places, and collapsing them is the ambiguity this issue spent an
investigation resolving.

The two decisions #1666 asked to be recorded

Warn or fail → FAIL, scoped. Three things keep it off the provider-health coupling
Collect models spent #915/#910/#911 learning to avoid, and off the #980 trade:

• an inactive provider has no settled model and never reaches the check, so a drained
  key still cannot redden it;
• unknown warns — it covers a wedged backend and a refused enable alike, and never
  reads as clean (shardguard counts blob files, not tests — a 0-test run reaches triage looking benign #1012);
• it is scoped through COLLECT_REQUIRED_PROVIDERS, the mechanism the collector-stall
  step already uses, so an enable failure on a provider pr-validation.yml's pinned run
  (pr-validation: the lane pays for multi-provider LLM coverage it does not owe (141 runs / 3.5 days on claude-sonnet-5) #1169) will never target cannot kill its E2E job. Unset — the daily, manual.yml, local
  — still requires every env-keyed provider, and a non-required provider's cold state is
  still reported.

The cold path is observable at run level. The enablement state prints on every run,
green included, and is recorded per provider in providers.json, so the next daily's
triage reads it instead of inferring it from a warning's absence.

Two hardening details worth the reviewer's eye

Both data files are written BEFORE the new step, and providers.json is patched after.
The step spends up to one write per provider plus a read at the very end of a minutes-long
sweep, so a test.setTimeout abort inside it used to discard the whole catalog —
resolveTestTargets() then resolves one (fallback) target and every parametrized spec
skips green (#570/#1012).

The per-call ceiling is now counted explicitly against the pre-flight timeout, because
a per-call constant can silently break the #1385 sizing invariant. It already had, at 60 s
(450 + 90 + 240 > 720), with that assertion passing.

waitForToggleChecked is deleted with its tests: it polled aria-checked, flipped
synchronously before any request leaves the browser, so it could only ever confirm the
optimistic client cache — and with no clicks left there is nothing to wait on.

Known cost this change introduces (measured, not assumed)

Google was the last provider, so its ~30 models used to be left enabled by the escaping
batch and setupGoogle clicked nothing. It now clicks ~24 and spends the give-up budget.
The write does land just after the give-up, so it is paid once per instance per
provider, not per spec: two consecutive agent-current-date-tool runs measured 3.3 min
then 1.1 min, server ending at openai 41 / google 35 enabled. Pointing those helpers at
the target model is #1651's surface and is tracked separately.

Validation

• tests/collect-models.spec.ts — 4 clean --retries=0 runs, fresh container and
  already-configured, 22.0 / 22.6 / 28.6 / 35.2 s.
• Forced failure, both modes. Write suppressed while reporting 2xx: unscoped, both
  providers fail naming off; with COLLECT_REQUIRED_PROVIDERS=openai, openai fails and
  google warns "Not fatal on this lane".
• agent-current-date-tool.spec.ts (the spec [Daily #1665] collect-models confirms 0 of 74 enable writes on all four shards, warns, and the whole run proceeds on the MIN_DEFAULT_MODELS cold path #1666 names) — 4 passed / 2 skipped, twice
  at --retries=0. The 2 skips are anthropic, whose key reports credit balance is too low, so this PR could not exercise the third provider end to end.
• npm run typecheck clean · npm run lint 0 errors · npm run test:units 884/884 ·
  npm run test:scripts 901/901 · both QA-CHECKLIST guards ✓ · watch-upstream-areas --mode=check-docs reports no unresolved path in the changed doc (both new src/… tokens
  verified on origin/main and release-1.12.0).
• 18 new unit tests cover ensureTargetModelsEnabled and targetEnablementVerdict against
  a duck-typed APIRequestContext — one POST per provider (never one batch: the endpoint
  raises on the first rejected update before persisting any), the body shape, the typed
  map winning over the flat one the backend documents as OR-ing across model types, and
  every path that must report rather than throw.

Nothing was quarantined for #1666, so there is no test.fixme to remove and no
@stable to restore.

🤖 Generated with Claude Code

[Copilot]

🟡 Changes recommended

confirmEnabledOnServer can throw on unexpected API payload shapes due to using the in operator on an unvalidated JSON-cast map, risking a hard abort of collect-models.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Este PR ajusta o fluxo de provider setup do E2E para que o collect-models deixe de tentar habilitar todos os modelos via UI (onde os writes estavam sendo descartados) e passe a habilitar/confirmar apenas o(s) modelo(s) que o run realmente vai mirar, registrando esse veredito em providers.json e adicionando um gate na spec utilitária.

Changes:

• collect-models agora só lê os nomes dos modelos na UI durante o sweep e move a habilitação para um passo explícito via API para os target models por provider.
• Introduz veredito persistido por provider (targetEnablement + targetEnablementDetail) e um gate em tests/collect-models.spec.ts para falhar em off/absent quando o provider for “required”.
• Atualiza e expande testes unitários e documentação para refletir o novo contrato/ordem de passos.

File summaries

File	Description
tests/helpers/provider-setup/collect-models.ts	Move a habilitação para ensureTargetModelsEnabled, adiciona targetEnablement* ao providers.json e reforça a confirmação no servidor (typed map vs flat).
tests/helpers/provider-setup/collect-models.test.ts	Atualiza a suíte de unit tests: remove waitForToggleChecked e adiciona cobertura para ensureTargetModelsEnabled/targetEnablementVerdict + sizing invariant.
tests/collect-models.spec.ts	Adiciona o gate “target model enabled” (falha scoping por COLLECT_REQUIRED_PROVIDERS, warns para unknown).
docs/collect-models.md	Documenta o novo contrato: “enable target model, not every model”, estados (enabled/off/absent/unknown/null) e racional de custo/robustez.

Review details

• Files reviewed: 4/4 changed files
• Comments generated: 1
• Review effort level: Lite

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.