Skip to content

Harden runtime model selection and release checks #16

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
QUSETIONS merged 2 commits into from
Jul 17, 2026
Merged

Harden runtime model selection and release checks #16

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
f8edf34
Harden runtime model selection and release checks
QUSETIONS Jul 17, 2026
c60b1ed
Fix Windows command execution fallbacks
QUSETIONS Jul 17, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions benchmarks/release_readiness.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,20 @@
BENCHMARKS_DIR = REPO_ROOT / "benchmarks"


def _clear_test_provider_environment(env: dict[str, str]) -> None:
"""Keep the release test suite hermetic while live smoke uses real config."""
for key in list(env):
if key.startswith((
"MINI_CODE_",
"OPENAI_",
"ANTHROPIC_",
"OPENROUTER_",
"CUSTOM_",
"DEEPSEEK_",
)):
env.pop(key, None)


def _normalize_evidence_paths(
value: object,
*,
Expand All @@ -61,6 +75,8 @@ def _normalize_evidence_paths(

def _run_command(label: str, command: list[str], *, cwd: Path, timeout: int = 1800) -> ReleaseCheck:
env = dict(os.environ)
if label == "pytest-q":
_clear_test_provider_environment(env)
existing_pythonpath = env.get("PYTHONPATH", "")
env["PYTHONPATH"] = (
str(REPO_ROOT)
Expand Down
92 changes: 46 additions & 46 deletions benchmarks/release_readiness_results.json
View file
Open in desktop

Large diffs are not rendered by default.

24 changes: 12 additions & 12 deletions benchmarks/release_readiness_results.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# MiniCode Release Readiness

- Generated at: 2026-07-10T04:23:46.564243+00:00
- Generated at: 2026-07-17T06:54:02.128764+00:00
- Status: blocked
- Local gates: blocked
- Provider status: at-risk
Expand All @@ -16,7 +16,7 @@
| check | status | exit_code | summary |
| --- | --- | ---: | --- |
| compileall | passed | 0 | compileall completed. |
| pytest-q | passed | 0 | 1297 passed, 2 skipped in 106.00s (0:01:46) |
| pytest-q | passed | 0 | 1310 passed, 2 skipped in 27.20s |
| runtime-profile-eval | passed | 0 | benchmarks/runtime_profile_eval_results.md |
| structure-compliance | passed | 0 | quality gate findings: 0 |

Expand Down Expand Up @@ -52,7 +52,7 @@

| label | outcome | failure_category | retryable | ownership | recovery_action | risk_scope | error_code | request_id | exit_code | summary |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | ---: | --- |
| headless-smoke | provider_channel_unavailable | configuration | false | local-configuration | Repair model-to-provider channel configuration. | provider-config | - | - | 1 | 2026-07-10 12:25:41,644 [WARNING] minicode.config: Project .mcp.json found at .mcp.json but NOT loaded (security: use --trust-project-mcp or MINI_CODE_TRUST_PROJECT_MCP=1). |
| headless-smoke | provider_channel_unavailable | configuration | false | local-configuration | Repair model-to-provider channel configuration. | provider-config | - | - | 1 | 2026-07-17 14:57:56,583 [WARNING] minicode.config: Project .mcp.json found at .mcp.json but NOT loaded (security: use --trust-project-mcp or MINI_CODE_TRUST_PROJECT_MCP=1). |

## Provider Action Items

Expand Down Expand Up @@ -119,13 +119,13 @@

| label | exists | size_bytes | sha256 | path |
| --- | --- | ---: | --- | --- |
| bundle_manifest_json | yes | 1526 | 16da44b19a63 | .temp/readiness-bundle/readiness-artifact-manifest.json |
| doctor_markdown | yes | 3307 | 66b1462dca50 | .temp/readiness-doctor.md |
| fallback_examples_json | yes | 876 | ceee505c6b4d | .temp/readiness-fallback-examples.json |
| bundle_manifest_json | yes | 1346 | e1756d9f9d97 | .temp/readiness-bundle/readiness-artifact-manifest.json |
| doctor_markdown | yes | 3277 | 1cee8e1f80c8 | .temp/readiness-doctor.md |
| fallback_examples_json | yes | 882 | f3664ba7de4a | .temp/readiness-fallback-examples.json |
| fallback_simulations_json | yes | 875 | 1355d9236755 | .temp/readiness-bundle/readiness-fallback-simulations.json |
| headless_trace | yes | 7947 | 642d9adad0bc | .temp/headless-provider-smoke-trace.json |
| json | yes | 6227 | bc160c17a80f | benchmarks/runtime_profile_eval_results.json |
| markdown | yes | 2374 | 4b88fb8c3795 | benchmarks/runtime_profile_eval_results.md |
| patch_preview_json | yes | 1611 | d3f81131954a | .temp/readiness-fallback-patch-preview.json |
| repair_plan_json | yes | 2473 | d037b4854529 | .temp/readiness-repair-plan.json |
| structure_compliance | yes | 89486 | ae1d66d73deb | .temp/structure-compliance.json |
| headless_trace | yes | 7935 | 1bfddc2dd3f9 | .temp/headless-provider-smoke-trace.json |
| json | yes | 6289 | 12acb4235389 | benchmarks/runtime_profile_eval_results.json |
| markdown | yes | 2373 | 5a41364c9693 | benchmarks/runtime_profile_eval_results.md |
| patch_preview_json | yes | 1617 | 87f7c97f4566 | .temp/readiness-fallback-patch-preview.json |
| repair_plan_json | yes | 2482 | 5d9d0551e73c | .temp/readiness-repair-plan.json |
| structure_compliance | yes | 98320 | a0d95a4c996e | .temp/structure-compliance.json |
22 changes: 11 additions & 11 deletions benchmarks/runtime_profile_eval_results.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
"scenario": "depth-budget-floor",
"condition": "single",
"runtime_profile": "single",
"wall_time_ms": 12.948627001605928,
"wall_time_ms": 4.455542002688162,
"model_calls": 1,
"tool_starts": 0,
"tool_results": 0,
Expand All @@ -29,7 +29,7 @@
"scenario": "depth-budget-floor",
"condition": "single-deep",
"runtime_profile": "single-deep",
"wall_time_ms": 2.634717006003484,
"wall_time_ms": 0.971665998804383,
"model_calls": 2,
"tool_starts": 0,
"tool_results": 0,
Expand All @@ -54,7 +54,7 @@
"scenario": "widening-escalation",
"condition": "single",
"runtime_profile": "single",
"wall_time_ms": 2.0254789997125044,
"wall_time_ms": 0.6387499888660386,
"model_calls": 1,
"tool_starts": 0,
"tool_results": 0,
Expand All @@ -79,7 +79,7 @@
"scenario": "widening-escalation",
"condition": "single-deep",
"runtime_profile": "single-deep",
"wall_time_ms": 4.325524998421315,
"wall_time_ms": 1.3504160015145317,
"model_calls": 10,
"tool_starts": 0,
"tool_results": 0,
Expand Down Expand Up @@ -117,15 +117,15 @@
"total_tool_starts": 0,
"total_tool_results": 0,
"total_runtime_events": 4,
"total_wall_time_ms": 14.974106001318432,
"total_wall_time_ms": 5.094291991554201,
"completion_rate": 0.0,
"widened_rate": 0.0,
"verification_guard_rate": 0.0,
"avg_model_calls": 1.0,
"avg_tool_starts": 0.0,
"avg_tool_results": 0.0,
"avg_runtime_events": 2.0,
"avg_wall_time_ms": 7.487053000659216
"avg_wall_time_ms": 2.5471459957771003
},
"single-deep": {
"runs": 2,
Expand All @@ -136,26 +136,26 @@
"total_tool_starts": 0,
"total_tool_results": 0,
"total_runtime_events": 9,
"total_wall_time_ms": 6.960242004424799,
"total_wall_time_ms": 2.3220820003189147,
"completion_rate": 1.0,
"widened_rate": 0.5,
"verification_guard_rate": 0.0,
"avg_model_calls": 6.0,
"avg_tool_starts": 0.0,
"avg_tool_results": 0.0,
"avg_runtime_events": 4.5,
"avg_wall_time_ms": 3.4801210022123996
"avg_wall_time_ms": 1.1610410001594573
}
},
"provider_diagnostics": [
{
"label": "headless-smoke",
"outcome": "provider_channel_unavailable",
"command": "/usr/bin/python3 -m minicode.headless Reply with exactly OK.",
"command": "~/.cache/codex-runtimes/codex-primary-runtime/dependencies/python/bin/python3 -m minicode.headless Reply with exactly OK.",
"exit_code": 1,
"summary": "2026-07-10 12:25:41,644 [WARNING] minicode.config: Project .mcp.json found at .mcp.json but NOT loaded (security: use --trust-project-mcp or MINI_CODE_TRUST_PROJECT_MCP=1).",
"summary": "2026-07-17 14:57:56,583 [WARNING] minicode.config: Project .mcp.json found at .mcp.json but NOT loaded (security: use --trust-project-mcp or MINI_CODE_TRUST_PROJECT_MCP=1).",
"stdout": "",
"stderr": "2026-07-10 12:25:41,644 [WARNING] minicode.config: Project .mcp.json found at .mcp.json but NOT loaded (security: use --trust-project-mcp or MINI_CODE_TRUST_PROJECT_MCP=1).\n2026-07-10 12:25:41,645 [ERROR] minicode.headless: Config load failed: No model configured. Set ~/.mini-code/settings.json or ANTHROPIC_MODEL.\nTraceback (most recent call last):\n File \"minicode/headless.py\", line 178, in run_headless\n runtime = load_runtime_config(cwd)\n ^^^^^^^^^^^^^^^^^^^^^^^^\n File \"minicode/config.py\", line 560, in load_runtime_config\n raise RuntimeError(\"No model configured. Set ~/.mini-code/settings.json or ANTHROPIC_MODEL.\")\nRuntimeError: No model configured. Set ~/.mini-code/settings.json or ANTHROPIC_MODEL.\n2026-07-10 12:25:41,648 [WARNING] minicode.config: Project .mcp.json found at .mcp.json but NOT loaded (security: use --trust-project-mcp or MINI_CODE_TRUST_PROJECT_MCP=1).\nConfig error: No model configured. Set ~/.mini-code/settings.json or ANTHROPIC_MODEL.\n",
"stderr": "2026-07-17 14:57:56,583 [WARNING] minicode.config: Project .mcp.json found at .mcp.json but NOT loaded (security: use --trust-project-mcp or MINI_CODE_TRUST_PROJECT_MCP=1).\n2026-07-17 14:57:56,583 [ERROR] minicode.headless: Config load failed: No model configured. Set ~/.mini-code/settings.json or ANTHROPIC_MODEL.\nTraceback (most recent call last):\n File \"minicode/headless.py\", line 178, in run_headless\n runtime = load_runtime_config(cwd)\n ^^^^^^^^^^^^^^^^^^^^^^^^\n File \"minicode/config.py\", line 608, in load_runtime_config\n raise RuntimeError(\"No model configured. Set ~/.mini-code/settings.json or ANTHROPIC_MODEL.\")\nRuntimeError: No model configured. Set ~/.mini-code/settings.json or ANTHROPIC_MODEL.\n2026-07-17 14:57:56,585 [WARNING] minicode.config: Project .mcp.json found at .mcp.json but NOT loaded (security: use --trust-project-mcp or MINI_CODE_TRUST_PROJECT_MCP=1).\nConfig error: No model configured. Set ~/.mini-code/settings.json or ANTHROPIC_MODEL.\n",
"risk_scope": "provider-config",
"error_code": "",
"request_id": "",
Expand Down
14 changes: 7 additions & 7 deletions benchmarks/runtime_profile_eval_results.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,17 +4,17 @@

| condition | runs | completion_rate | widened_rate | verification_guard_rate | avg_model_calls | avg_runtime_events | avg_wall_time_ms |
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| single | 2 | 0.00 | 0.00 | 0.00 | 1.00 | 2.00 | 7.49 |
| single-deep | 2 | 1.00 | 0.50 | 0.00 | 6.00 | 4.50 | 3.48 |
| single | 2 | 0.00 | 0.00 | 0.00 | 1.00 | 2.00 | 2.55 |
| single-deep | 2 | 1.00 | 0.50 | 0.00 | 6.00 | 4.50 | 1.16 |

## Scenario Rows

| scenario | condition | completed | stop_reason | widened | verification_guard | runtime_events | model_calls | wall_time_ms | final_message |
| --- | --- | --- | --- | --- | --- | ---: | ---: | ---: | --- |
| depth-budget-floor | single | no | max_steps | no | no | 2 | 1 | 12.95 | Reached the maximum tool step limit for this turn. |
| depth-budget-floor | single-deep | yes | done | no | no | 2 | 2 | 2.63 | done |
| widening-escalation | single | no | max_steps | no | no | 2 | 1 | 2.03 | Reached the maximum tool step limit for this turn. |
| widening-escalation | single-deep | yes | done | yes | no | 7 | 10 | 4.33 | done with a broader plan |
| depth-budget-floor | single | no | max_steps | no | no | 2 | 1 | 4.46 | Reached the maximum tool step limit for this turn. |
| depth-budget-floor | single-deep | yes | done | no | no | 2 | 2 | 0.97 | done |
| widening-escalation | single | no | max_steps | no | no | 2 | 1 | 0.64 | Reached the maximum tool step limit for this turn. |
| widening-escalation | single-deep | yes | done | yes | no | 7 | 10 | 1.35 | done with a broader plan |

## Runtime Timelines

Expand All @@ -27,7 +27,7 @@

| label | outcome | category | retryable | ownership | recovery_action | risk_scope | readiness | repair_steps | trace | error_code | request_id | exit_code | summary |
| --- | --- | --- | --- | --- | --- | --- | --- | ---: | --- | --- | --- | ---: | --- |
| headless-smoke | provider_channel_unavailable | configuration | no | local-configuration | Repair model-to-provider channel configuration. | provider-config | blocked | 6 | .temp/headless-provider-smoke-trace.json | - | - | 1 | 2026-07-10 12:25:41,644 [WARNING] minicode.config: Project .mcp.json found at .mcp.json but NOT loaded (security: use --... |
| headless-smoke | provider_channel_unavailable | configuration | no | local-configuration | Repair model-to-provider channel configuration. | provider-config | blocked | 6 | .temp/headless-provider-smoke-trace.json | - | - | 1 | 2026-07-17 14:57:56,583 [WARNING] minicode.config: Project .mcp.json found at .mcp.json but NOT loaded (security: use --... |

Guidance for `headless-smoke`:
- Verify the selected model group and provider channel configuration.
Expand Down
8 changes: 6 additions & 2 deletions minicode/agent_loop.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -751,6 +751,7 @@ def run_agent_turn(
on_assistant_stream_chunk: Callable[[str], None] | None = None,
on_thinking_chunk: Callable[[str], None] | None = None,
context_manager: ContextManager | None = None,
memory_manager: MemoryManager | None = None,
runtime: dict | None = None,
metrics_collector: AgentMetricsCollector | None = None,
system_prompt: str = "",
Expand Down Expand Up @@ -955,7 +956,11 @@ def emit_runtime_event(
circuit_breaker_limit=3,
session_memory_enabled=True,
)
memory_mgr = MemoryManager(project_root=cwd)
memory_mgr = (
memory_manager
if memory_manager is not None
else MemoryManager(project_root=cwd)
)
# 将 memory_mgr 注入 ReflectionEngine，使自省经验持久化
if reflection_engine:
reflection_engine.memory = memory_mgr
Expand Down Expand Up @@ -2416,4 +2421,3 @@ def emit_runtime_event(
supervisor_report.risk_level.value,
"; ".join(supervisor_report.recommended_actions[:3]),
)

111 changes: 88 additions & 23 deletions minicode/config.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -259,6 +259,56 @@ def _known_openai_exposed_models(runtime: dict[str, Any] | None) -> list[str]:
return list(list_openai_exposed_models(runtime))


def _known_openai_agent_models(runtime: dict[str, Any] | None) -> list[str]:
from minicode.model_registry import model_id_supports_agent_tools

return [
model
for model in _known_openai_exposed_models(runtime)
if model_id_supports_agent_tools(model)
]


def discovered_openai_fallbacks(
runtime: dict[str, Any] | None,
current_model: str | None = None,
*,
probe_openai_models: bool = True,
) -> list[str]:
"""Return provider-exposed fallback models for a custom OpenAI host.

A custom OpenAI-compatible gateway is not guaranteed to expose the built-in
OpenAI defaults. Prefer the gateway's own model catalog when it is
available. Network probing is opt-in so local readiness surfaces stay
deterministic; the runtime switcher enables it when it needs recovery.
"""
runtime = runtime or {}
if not _uses_custom_openai_compatible_host(runtime):
return []

from minicode.model_registry import (
list_openai_exposed_models,
model_id_supports_agent_tools,
probe_openai_exposed_models,
)

active_model = str(current_model or runtime.get("model", "")).strip()
exposed_models = (
probe_openai_exposed_models(runtime)
if probe_openai_models
else list_openai_exposed_models(runtime)
)
return [
model
for model in exposed_models
if (
model
and model != active_model
and model_id_supports_agent_tools(model)
)
]


def describe_fallback_guidance(
runtime: dict[str, Any] | None,
provider_name: str | None = None,
Expand All @@ -278,7 +328,7 @@ def describe_fallback_guidance(
configured = configured_model_fallbacks(runtime, provider_key)
defaults = default_model_fallbacks(runtime, provider_key, current_model=active_model)
exposed_openai_models = (
_known_openai_exposed_models(runtime)
_known_openai_agent_models(runtime)
if provider_key == "openai" and _uses_custom_openai_compatible_host(runtime)
else []
)
Expand All @@ -301,25 +351,23 @@ def describe_fallback_guidance(
)

if not configured:
if defaults:
if exposed_openai_models:
exposed_preview = ", ".join(exposed_openai_models[:3])
guidance.append(
"Default failover is already available for this runtime through "
"provider-exposed models. This OpenAI-compatible provider currently "
f"exposes: {exposed_preview}. Set fallbackModels or "
f"{provider_specific_key} to one of the exposed models."
)
elif defaults:
preview = ", ".join(defaults[:3])
if exposed_openai_models:
exposed_preview = ", ".join(exposed_openai_models[:3])
guidance.append(
"Default failover is already available for this runtime"
f"{': ' + preview if preview else '.'}"
f" This OpenAI-compatible provider currently exposes: {exposed_preview}. "
f"Set fallbackModels or {provider_specific_key} to one of the exposed models "
"if the defaults are unavailable on the current provider."
)
else:
guidance.append(
"Default failover is already available for this runtime"
f"{': ' + preview if preview else '.'}"
" If those models are still unavailable on the current provider, "
f"set fallbackModels or {provider_specific_key} to models that the provider actually exposes, "
"or switch provider credentials."
)
guidance.append(
"Default failover is already available for this runtime"
f"{': ' + preview if preview else '.'}"
" If those models are still unavailable on the current provider, "
f"set fallbackModels or {provider_specific_key} to models that the provider actually exposes, "
"or switch provider credentials."
)
else:
guidance.append(
f"Add fallbackModels or {provider_specific_key} to enable model failover."
Expand Down Expand Up @@ -669,7 +717,11 @@ def _is_valid_http_url(value: str | None) -> bool:
return parsed.scheme in {"http", "https"} and bool(parsed.netloc)


def validate_provider_runtime(runtime: dict[str, Any]) -> list[str]:
def validate_provider_runtime(
runtime: dict[str, Any],
*,
probe_openai_models: bool = True,
) -> list[str]:
"""Validate the auth/base-url required by the detected provider.

A generic API key is not enough: if the selected model routes to OpenAI,
Expand All @@ -679,7 +731,11 @@ def validate_provider_runtime(runtime: dict[str, Any]) -> list[str]:
from minicode.model_registry import Provider, detect_provider

model = str(runtime.get("model", "")).strip()
provider = detect_provider(model, runtime)
provider = detect_provider(
model,
runtime,
probe_openai_models=probe_openai_models,
)
errors: list[str] = []

if provider == Provider.OPENAI:
Expand Down Expand Up @@ -755,7 +811,12 @@ def validate_config(cwd: str | Path | None = None) -> tuple[bool, list[str]]:

try:
config = load_runtime_config(cwd)
errors.extend(validate_provider_runtime(config))
errors.extend(
validate_provider_runtime(
config,
probe_openai_models=False,
)
)

# 检查模型名称拼写
model = config.get("model", "")
Expand Down Expand Up @@ -847,7 +908,11 @@ def format_config_diagnostic(cwd: str | Path | None = None) -> str:

# Show provider info
from minicode.model_registry import detect_provider, Provider
provider = detect_provider(model_name, config)
provider = detect_provider(
model_name,
config,
probe_openai_models=False,
)
lines.append(f" Provider: {provider.value}")
lines.append(f" Channel: {describe_provider_channel(config, provider.value)}")

Expand Down
1 change: 1 addition & 0 deletions minicode/headless.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -233,6 +233,7 @@ def run_headless(prompt: str | None = None, allow_edits: bool = False) -> str:
messages=messages,
cwd=cwd,
permissions=permissions,
memory_manager=memory_mgr,
runtime=runtime,
)

Expand Down
Loading
Loading