From 787f68469567508162a13a027d1a5bfa7ec76a8c Mon Sep 17 00:00:00 2001 From: mnkj0021 Date: Mon, 28 Sep 2026 19:34:31 +0000 Subject: [PATCH 1/5] docs: document model lifecycle and deprecation --- website/astro.config.mjs | 1 + .../docs/guides/choosing-a-coding-agent.md | 4 +- .../content/docs/guides/model-lifecycle.md | 139 ++++++++++++++++++ 3 files changed, 143 insertions(+), 1 deletion(-) create mode 100644 website/src/content/docs/guides/model-lifecycle.md diff --git a/website/astro.config.mjs b/website/astro.config.mjs index c637dd8c7..018d40137 100644 --- a/website/astro.config.mjs +++ b/website/astro.config.mjs @@ -123,6 +123,7 @@ export default defineConfig({ { label: 'Agent profiles', slug: 'guides/defining-profiles' }, { label: 'Prompt features', slug: 'guides/prompt-features' }, { label: 'Choose a coding agent', slug: 'guides/choosing-a-coding-agent' }, + { label: 'Model lifecycle & deprecation', slug: 'guides/model-lifecycle' }, { label: 'Software stacks', slug: 'guides/software-stacks' }, ], }, diff --git a/website/src/content/docs/guides/choosing-a-coding-agent.md b/website/src/content/docs/guides/choosing-a-coding-agent.md index f07316b58..0eaad8ba1 100644 --- a/website/src/content/docs/guides/choosing-a-coding-agent.md +++ b/website/src/content/docs/guides/choosing-a-coding-agent.md @@ -39,7 +39,9 @@ any supported coding agent. - **Available models.** Each coding agent exposes its own model list. GitHub Copilot CLI and Claude Code CLI advertise the - models their respective agent supports. + models their respective agent supports. See + [Model lifecycle and deprecation](/guides/model-lifecycle/) for how Scope + discovers, retires, and restores models. - **Extension support.** Only VS Code Copilot accepts `extensions` in a profile. The two CLI-based agents reject extensions with HTTP 400 — see diff --git a/website/src/content/docs/guides/model-lifecycle.md b/website/src/content/docs/guides/model-lifecycle.md new file mode 100644 index 000000000..142c5f6ea --- /dev/null +++ b/website/src/content/docs/guides/model-lifecycle.md @@ -0,0 +1,139 @@ +--- +title: Model lifecycle and deprecation +description: How Scope discovers new coding-agent models, handles provider deprecations, and treats models that disappear. +--- + +Scope does not maintain a permanent hand-written list of coding-agent models. +For providers that support model discovery, **model scanners** periodically ask +the provider what is available and reconcile that result with Scope's model +catalog. + +This means the model list can change independently of a Scope release. A new +provider model can appear after a scan, and a retired model can disappear after +the provider stops advertising it. + +## Lifecycle at a glance + +A model normally moves through these states: + +1. **Discovered** — a scanner sees the model for the first time. +2. **Active** — later scans continue to return it. +3. **Planned for retirement** *(when the provider supplies a date)* — Scope + records the provider's end-of-life/deprecation date, but the model remains + active while the provider still advertises it. +4. **Disappeared** — a scan no longer returns a model that Scope previously + saw. +5. **Restored** — if the provider advertises a disappeared model again, Scope + clears the disappearance marker and treats the model as active again. + +Scope keeps lifecycle history rather than deleting the model record. This lets +older runs retain the model ID they actually used. + +## How new models are added + +Each provider scanner returns a model ID and, when available, metadata such as: + +- the provider-reported availability date; +- the provider-reported end-of-life/deprecation date; +- model capabilities such as supported reasoning-effort levels, tool calling, + vision, streaming, or adaptive thinking. + +The scanner sends that inventory to Scope's model-sync API. A model Scope has +never seen before is inserted with a `firstSeenAt` timestamp. Models already in +the catalog get their `lastSeenAt` timestamp and provider metadata refreshed. + +After reconciliation, the coding agent's `supportedModels` list is rebuilt from +models that are currently active. The Portal and request APIs use that list for +new submissions. + +### Default model changes + +If an agent has no default model, or its current default is no longer active, +Scope automatically selects the newest active model. "Newest" is determined by +the provider availability date when supplied, otherwise by the time Scope first +saw the model. + +A model appearing in the catalog therefore does **not** necessarily change the +default immediately. The automatic default selection happens when a default is +missing or has disappeared. + +## What happens when a model is deprecated + +Providers do not all expose deprecation information in the same way. When a +scanner receives a planned end-of-life date, Scope stores it as +`providerEndOfLife`. That date is informational lifecycle metadata; Scope does +not remove the model merely because the date exists. + +Actual availability is determined by provider discovery. Once a previously +active model is absent from a scan, Scope records `disappearedAt` and removes it +from the agent's active `supportedModels` list. + +This distinction matters because a provider may announce retirement before the +model stops working, or temporarily omit a model from discovery. + +## Submission behavior after a model disappears + +Scope deliberately has a short grace period for a newly disappeared model: + +- **Less than 24 hours since `disappearedAt`:** submission is allowed, but the + request receives a warning that the model may not be available at runtime. +- **24 hours or more since `disappearedAt`:** new submissions using that model + are rejected with `model_unavailable_for_worker`. + +The grace period reduces false failures from scanner lag or a short provider +inventory outage without allowing a known-unavailable model to remain +selectable indefinitely. + +If the model reappears in a later provider scan, its `disappearedAt` marker is +cleared and it returns to the active list. + +## What users should expect + +### Portal and inline submissions + +The model picker reflects the active models for the selected coding agent. If a +model disappears, it will no longer be offered for new inline selections after +reconciliation. + +A request that explicitly references a model may receive the temporary warning +or eventual hard rejection described above. + +### Profiles + +Profiles are versioned configuration records, so an older profile version can +still name a model that has since disappeared. Keeping that historical value is +important for reproducibility: Scope should not silently rewrite what an old +benchmark was configured to use. + +When you create new runs, prefer an active model. If a long-lived benchmark +profile points at a retired model, create a new profile version with its +replacement instead of changing the meaning of the historical version. + +### Existing runs and reports + +Existing run records keep the model ID that was used at execution time. Model +retirement does not rewrite completed run history or reports. + +This is why lifecycle records are marked as disappeared instead of being +removed from the database. + +## Operator checklist for rotating models + +When a provider introduces or retires a model: + +1. Run or wait for the relevant provider scanner. +2. Verify the model catalog (`GET /api/v1/models`) shows the expected active or + disappeared state. +3. Check the affected coding agent's `supportedModels` and `defaultModel`. +4. Review `providerEndOfLife` dates where the provider supplies them. +5. Update benchmark profiles that should move to a replacement model by + creating new profile versions. +6. Keep historical runs/profile versions unchanged so comparisons remain + traceable. + +## See also + +- [Choosing a coding agent](/guides/choosing-a-coding-agent/) +- [Defining profiles](/guides/defining-profiles/) +- [Submitting requests from the Portal](/guides/submitting-requests-portal/) +- [Coding agents & capabilities](/reference/workers/) From 901f104066ea3ea4448b460aa70c899ab3d61c25 Mon Sep 17 00:00:00 2001 From: mnkj0021 <98058999+mnkj0021@users.noreply.github.com> Date: Fri, 2 Oct 2026 17:12:27 +0500 Subject: [PATCH 2/5] docs: clarify model selection and disappearance validation --- .../content/docs/guides/model-lifecycle.md | 44 +++++++++++++------ 1 file changed, 30 insertions(+), 14 deletions(-) diff --git a/website/src/content/docs/guides/model-lifecycle.md b/website/src/content/docs/guides/model-lifecycle.md index 142c5f6ea..3c99b856f 100644 --- a/website/src/content/docs/guides/model-lifecycle.md +++ b/website/src/content/docs/guides/model-lifecycle.md @@ -43,8 +43,13 @@ never seen before is inserted with a `firstSeenAt` timestamp. Models already in the catalog get their `lastSeenAt` timestamp and provider metadata refreshed. After reconciliation, the coding agent's `supportedModels` list is rebuilt from -models that are currently active. The Portal and request APIs use that list for -new submissions. +models that are currently active. Request validation uses that agent-specific +list when checking an explicitly selected model. + +The Portal gets model capability and selection data separately from the model +catalog by querying `GET /api/v1/models` for the selected agent with +`status=active`. That active-catalog query does not consume the agent's +`supportedModels` field. ### Default model changes @@ -73,30 +78,41 @@ model stops working, or temporarily omit a model from discovery. ## Submission behavior after a model disappears -Scope deliberately has a short grace period for a newly disappeared model: +Request submission first validates the selected model against the coding +agent's current `supportedModels` list. Because reconciliation removes +disappeared models from that list, a reconciled request can be rejected at this +step with `agent_model_unsupported`. + +If the model still passes that agent-target validation, the request route also +checks the model catalog's `disappearedAt` timestamp: - **Less than 24 hours since `disappearedAt`:** submission is allowed, but the request receives a warning that the model may not be available at runtime. -- **24 hours or more since `disappearedAt`:** new submissions using that model - are rejected with `model_unavailable_for_worker`. +- **24 hours or more since `disappearedAt`:** submission is rejected with + `model_unavailable_for_worker`. -The grace period reduces false failures from scanner lag or a short provider -inventory outage without allowing a known-unavailable model to remain -selectable indefinitely. +The 24-hour check is therefore a secondary safety net, not a guarantee that +every disappeared model remains submittable for 24 hours. In the common case +where agent reconciliation has already removed the model from +`supportedModels`, the earlier agent-target validation rejects it first. If the model reappears in a later provider scan, its `disappearedAt` marker is -cleared and it returns to the active list. +cleared and it returns to the active model catalog and agent reconciliation can +restore it to `supportedModels`. ## What users should expect ### Portal and inline submissions -The model picker reflects the active models for the selected coding agent. If a -model disappears, it will no longer be offered for new inline selections after -reconciliation. +Portal model capability and model-selection data comes from the active model +catalog for the selected coding agent. Once a model is no longer active, it +drops out of that catalog query. -A request that explicitly references a model may receive the temporary warning -or eventual hard rejection described above. +Submission is validated independently against the agent's +`supportedModels`. Historical configuration can still reference an older +model, but a new request using it may be rejected by agent-target validation. +Only models that pass that validation reach the later `disappearedAt` +warning/rejection check described above. ### Profiles From bad2face13a09b522ea6e501532e69ccac3b26d9 Mon Sep 17 00:00:00 2001 From: mnkj0021 <98058999+mnkj0021@users.noreply.github.com> Date: Wed, 7 Oct 2026 23:28:23 +0500 Subject: [PATCH 3/5] docs: address model lifecycle review feedback --- website/astro.config.mjs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/astro.config.mjs b/website/astro.config.mjs index 79eca4542..356a84f49 100644 --- a/website/astro.config.mjs +++ b/website/astro.config.mjs @@ -130,7 +130,7 @@ export default defineConfig({ { label: 'Agent profiles', slug: 'guides/defining-profiles' }, { label: 'Prompt features', slug: 'guides/prompt-features' }, { label: 'Choose a coding agent', slug: 'guides/choosing-a-coding-agent' }, - { label: 'Model lifecycle & deprecation', slug: 'guides/model-lifecycle' }, + { label: 'Model catalog & lifecycle', slug: 'guides/model-lifecycle' }, { label: 'Software stacks', slug: 'guides/software-stacks' }, ], }, From b57c64bb76a20702626961c0173dfc008af6af93 Mon Sep 17 00:00:00 2001 From: mnkj0021 <98058999+mnkj0021@users.noreply.github.com> Date: Thu, 8 Oct 2026 07:28:47 +0500 Subject: [PATCH 4/5] docs: clarify model catalog lifecycle and submission behavior --- .../content/docs/guides/model-lifecycle.md | 73 +++++++++++-------- 1 file changed, 43 insertions(+), 30 deletions(-) diff --git a/website/src/content/docs/guides/model-lifecycle.md b/website/src/content/docs/guides/model-lifecycle.md index 3c99b856f..fb034d788 100644 --- a/website/src/content/docs/guides/model-lifecycle.md +++ b/website/src/content/docs/guides/model-lifecycle.md @@ -1,5 +1,5 @@ --- -title: Model lifecycle and deprecation +title: Model catalog and lifecycle description: How Scope discovers new coding-agent models, handles provider deprecations, and treats models that disappear. --- @@ -17,17 +17,16 @@ the provider stops advertising it. A model normally moves through these states: 1. **Discovered** — a scanner sees the model for the first time. -2. **Active** — later scans continue to return it. -3. **Planned for retirement** *(when the provider supplies a date)* — Scope - records the provider's end-of-life/deprecation date, but the model remains - active while the provider still advertises it. -4. **Disappeared** — a scan no longer returns a model that Scope previously - saw. -5. **Restored** — if the provider advertises a disappeared model again, Scope +2. **Active** — later scans continue to return it. A planned retirement date, + when supplied, is metadata on an active model, not a separate status. +3. **Disappeared** — a scan no longer returns a model that Scope previously saw. +4. **Restored** — if the provider advertises a disappeared model again, Scope clears the disappearance marker and treats the model as active again. -Scope keeps lifecycle history rather than deleting the model record. This lets -older runs retain the model ID they actually used. +Scope keeps the model record (marked as disappeared) rather than deleting it. +The marker is cleared on restore, so earlier disappear/restore cycles are not +preserved as a full lifecycle history. Older runs still retain the model ID +they actually used. ## How new models are added @@ -38,6 +37,10 @@ Each provider scanner returns a model ID and, when available, metadata such as: - model capabilities such as supported reasoning-effort levels, tool calling, vision, streaming, or adaptive thinking. +The Copilot and Anthropic scanners expose different metadata. Only Copilot +reports `providerEndOfLife`; Anthropic reports `providerAvailableFrom` but +not an end-of-life date. + The scanner sends that inventory to Scope's model-sync API. A model Scope has never seen before is inserted with a `firstSeenAt` timestamp. Models already in the catalog get their `lastSeenAt` timestamp and provider metadata refreshed. @@ -60,7 +63,9 @@ saw the model. A model appearing in the catalog therefore does **not** necessarily change the default immediately. The automatic default selection happens when a default is -missing or has disappeared. +missing or has disappeared. Restoring the previously selected model does not +automatically restore it as the default; the replacement stays selected until +the default is changed again. ## What happens when a model is deprecated @@ -78,23 +83,24 @@ model stops working, or temporarily omit a model from discovery. ## Submission behavior after a model disappears -Request submission first validates the selected model against the coding -agent's current `supportedModels` list. Because reconciliation removes -disappeared models from that list, a reconciled request can be rejected at this -step with `agent_model_unsupported`. +In the normal scanner reconciliation flow, the disappeared model is removed +from the agent's `supportedModels` list at the same time that `disappearedAt` +is set. New submissions targeting that model therefore fail **immediately** +with `agent_model_unsupported`, rather than receiving a 24-hour grace period. -If the model still passes that agent-target validation, the request route also -checks the model catalog's `disappearedAt` timestamp: +The request route only reaches the later `disappearedAt` check if agent-target +validation still accepts the model. This can happen if an agent's +`supportedModels` list has been explicitly supplied by an agent update or +registration instead of reflecting the latest scanner reconciliation: - **Less than 24 hours since `disappearedAt`:** submission is allowed, but the request receives a warning that the model may not be available at runtime. - **24 hours or more since `disappearedAt`:** submission is rejected with `model_unavailable_for_worker`. -The 24-hour check is therefore a secondary safety net, not a guarantee that -every disappeared model remains submittable for 24 hours. In the common case -where agent reconciliation has already removed the model from -`supportedModels`, the earlier agent-target validation rejects it first. +This 24-hour warning/rejection check is a **secondary safeguard**, not the +normal submission behavior after a scan. It does not grant a grace period to +models removed from the agent's `supportedModels` list. If the model reappears in a later provider scan, its `disappearedAt` marker is cleared and it returns to the active model catalog and agent reconciliation can @@ -114,16 +120,22 @@ model, but a new request using it may be rejected by agent-target validation. Only models that pass that validation reach the later `disappearedAt` warning/rejection check described above. +For inline requests that omit `model`, Scope resolves the agent's current +`defaultModel` **at submission time**. If a scan rotates the default, later +inline requests can silently use a different model. Specify an explicit +`model`, or use a versioned profile, for reproducible benchmark comparisons. + ### Profiles -Profiles are versioned configuration records, so an older profile version can -still name a model that has since disappeared. Keeping that historical value is -important for reproducibility: Scope should not silently rewrite what an old -benchmark was configured to use. +Profile creation saves the **resolved model** in the profile version. If no +model was explicitly provided, Scope saves the agent's current default at that +time. Existing profile versions therefore **do not follow later default-model +rotation**; they keep their original model for reproducibility. -When you create new runs, prefer an active model. If a long-lived benchmark -profile points at a retired model, create a new profile version with its -replacement instead of changing the meaning of the historical version. +If a saved profile version refers to a disappeared model, new runs and reruns +using that version fail agent-target validation with +`agent_model_unsupported`. You must create a **new profile version** with an +active replacement model. The historical version is not silently rewritten. ### Existing runs and reports @@ -138,8 +150,9 @@ removed from the database. When a provider introduces or retires a model: 1. Run or wait for the relevant provider scanner. -2. Verify the model catalog (`GET /api/v1/models`) shows the expected active or - disappeared state. +2. Query `GET /api/v1/models?agentId=&status=disappeared` to + inspect disappeared models for that agent; use `status=active` for the + active catalog. The endpoint also accepts a `provider` filter. 3. Check the affected coding agent's `supportedModels` and `defaultModel`. 4. Review `providerEndOfLife` dates where the provider supplies them. 5. Update benchmark profiles that should move to a replacement model by From 223087da80e42ee7a75c9fac7e45a50330c68c17 Mon Sep 17 00:00:00 2001 From: mnkj0021 <98058999+mnkj0021@users.noreply.github.com> Date: Thu, 8 Oct 2026 10:33:59 +0500 Subject: [PATCH 5/5] docs: align model lifecycle guide cross-link with reviewer naming --- website/src/content/docs/guides/choosing-a-coding-agent.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/src/content/docs/guides/choosing-a-coding-agent.md b/website/src/content/docs/guides/choosing-a-coding-agent.md index 0eaad8ba1..0153deedd 100644 --- a/website/src/content/docs/guides/choosing-a-coding-agent.md +++ b/website/src/content/docs/guides/choosing-a-coding-agent.md @@ -40,7 +40,7 @@ any supported coding agent. - **Available models.** Each coding agent exposes its own model list. GitHub Copilot CLI and Claude Code CLI advertise the models their respective agent supports. See - [Model lifecycle and deprecation](/guides/model-lifecycle/) for how Scope + [Model catalog and lifecycle](/guides/model-lifecycle/) for how Scope discovers, retires, and restores models. - **Extension support.** Only VS Code Copilot accepts `extensions` in a profile. The two CLI-based agents reject