From f7b9dce83bd59e864fe71c93fb62f21270214f7b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E5=93=80=E6=B4=9B=E8=8A=99?= Date: Sat, 29 Aug 2026 21:57:30 +0800 Subject: [PATCH 01/39] docs: design 1.1.3 storage governance --- ...6-08-29-storage-governance-1.1.3-design.md | 151 ++++++++++++++++++ 1 file changed, 151 insertions(+) create mode 100644 docs/superpowers/specs/2026-08-29-storage-governance-1.1.3-design.md diff --git a/docs/superpowers/specs/2026-08-29-storage-governance-1.1.3-design.md b/docs/superpowers/specs/2026-08-29-storage-governance-1.1.3-design.md new file mode 100644 index 0000000..a68918f --- /dev/null +++ b/docs/superpowers/specs/2026-08-29-storage-governance-1.1.3-design.md @@ -0,0 +1,151 @@ +# 2718lab DevKit 1.1.3 存储治理设计 + +## 状态与选型 + +本文是 1.1.3 的实现基线。选定方案为“宿主权威的存储准入、持久租约账本与显式清理三层方案”（方案 A）。本设计只定义边界和证据，不执行清理、不改变现有用户文件。 + +## 背景与问题界定 + +G 盘空间下降的主要证据指向重复的 Cargo `target` 根：主 Host target、集成 target、孤立的 rmcp target 各自保存了大量 `incremental` 和 `deps` 文件。当前恢复树的 DevKit 源码约为数百 MiB，Codex/DevKit 会话数据不是这次增长的主因。1.1.3 的首要修复是阻止同一构建语义因任务目录、worktree 或重启而无限复制 target；会话治理是独立的保守收尾措施，不能被用来解释或掩盖 target 泄漏。 + +本机所有临时产物继续限定在 `G:\2718lab\_codex\.codex-task-temp` 下。宿主是路径、进程、磁盘统计和删除动作的唯一权威；DevKit 只产生经过规范化和哈希绑定的 storage intent，不能声称已经取得租约或已经清理。 + +## 目标与非目标 + +目标是：为 Cargo、Python、MCP 打包和 Fast Lane 任务提供确定性产物根；在写入前预留文件数、字节数和最低剩余空间；跨重启恢复租约；以 preview/candidate/recheck/apply 证据链执行有限清理；在 GitHub 可达且用户明确授权精确路径时才允许源代码清理；对归档会话只做 CAS 去重。 + +非目标是：全盘扫描后按大小删除、自动删除未知目录、活动任务、脏 worktree、源代码或活动/未归档会话；从目录名推断拥有者；以“已上传 GitHub”单独替代路径授权；引入远程会话同步、压缩策略或绕过 Fast Lane 的 route/lease/context/capability 证明。 + +## 方案比较 + +### 方案 A：Host 权威三层治理（选定） + +Host 统一分配确定性 target 根，持久化 task storage lease ledger，所有清理都经过候选哈希和再次核验。优点是能同时约束 DevKit 与 Codex Host，能恢复中断状态，且删除范围可审计；代价是需要 ledger 迁移、进程身份证明和少量 Host 接口。适合当前重复 target 的根因和“不能误删”的要求。 + +### 方案 B：每个插件自行配额与定时清理 + +每个插件管理自己的 target 和临时目录。实现初期较小，但无法识别跨插件的重复 Cargo target，重启后也无法确认 owner；Host、DevKit 和插件会互相绕过配额。该方案不选。 + +### 方案 C：先上传远端，再删除本地 + +把构建产物、源码和会话先归档到远端，再由远端策略回收本地。它依赖网络、远端权限和同步语义，且不能解决本地 target 在上传前的无界增长。它可作为将来归档扩展，不进入 1.1.3 的删除路径。 + +## 三层架构 + +### 第一层：存储准入与确定性根（Storage Firewall） + +Host 在进程启动前验证 approved task root、磁盘统计、target key、当前 lease 和预算。写入进程只能收到 Host 分配的绝对产物根；未登记的 `CARGO_TARGET_DIR`、临时根或 package cache 一律拒绝。低空间、统计失败、策略缺失、路径越出批准根或预算不足均 fail-closed，不通过杀死其他进程来“腾空间”。 + +### 第二层：所有权、配额与重启恢复（Lease Ledger） + +Host 将每个任务的 generated storage 写入持久账本。账本以事务和原子替换保存，包含预留、心跳、过期、重启代数、进程启动身份及最后 receipt。活动租约不因清理扫描而失效；没有足够证据的过期租约进入 recovery/quarantine，而不是直接删除。 + +### 第三层：显式清理与远端/会话保护（Cleanup Governance） + +清理是独立的、有限的 preview 到 apply 流程。仅有账本标记为 disposable、无 owner、内容哈希仍相同且位于批准 generated 根中的候选才能 apply。源代码清理还要经过 GitHub reachability 和用户精确路径授权。会话只允许对不可变、无活动引用的 archived CAS 对象做去重。 + +## 确定性 target key 与数据流 + +`target_key` 不含绝对路径、task id、随机数或 worktree 临时目录名，按 UTF-8 canonical JSON 计算： + +```text +{ + "schema":"2718lab.storage.target.v1", + "artifact_kind":"cargo-target", + "repository_identity":, + "workspace_manifest_hash":, + "cargo_lock_hash":, + "toolchain_digest":, + "target_triple":, + "profile":, + "features_hash":, + "build_env_class": +} +``` + +`target_key = SHA-256(canonical_json)`，Host 将其映射到批准根下的固定目录，并拒绝同 key 的不相容字段。不同构建语义必须产生不同 key；相同 key 的并发写入须先取得同一 target-family exclusive lease，不能靠共享目录的偶然行为。Python/MCP 产物使用同样的 schema 规则，以 `artifact_kind` 和其锁文件/解释器摘要替换 Cargo 字段。 + +数据流固定为：DevKit Fast Lane 编译器产生绑定 `task_id/plan_binding/context_hash/storage_intent` 的请求；Host 验证 route、lease、context、capability 后规范化 intent，计算 target key，读取 G 盘统计并在 ledger 中 reserve；Host 启动 worker 并注入已分配的产物根；worker 周期性回报 bytes/files/receipt；Host 在 terminal receipt 后释放 lease；只有随后独立生成的 cleanup preview 才能进入清理链。Project Index 缺失不会给 storage intent 赋予路径或 owner，仍须 Host 自动重建并回 receipt。 + +## Task Storage Lease Ledger + +每条记录的 schema 为 `2718lab.storage.lease.v1`，至少包含以下字段： + +| 字段 | 约束 | +| --- | --- | +| `ledger_epoch`, `schema_version` | 单调 epoch 与精确 schema；迁移前后可核验 | +| `lease_id`, `task_id`, `assignment_id`, `plan_binding` | 唯一、不可改写并绑定 Fast Lane receipt | +| `project_identity`, `repository_identity`, `worktree_identity` | Host 已证明的项目与 worktree 身份 | +| `artifact_kind`, `target_key`, `path_identity` | generated 类型、确定性 key、批准根内规范绝对路径 | +| `owner_epoch`, `owner_kind`, `process_id`, `process_start_time`, `host_instance_id` | 防 PID 重用的 owner 证明；不是目录名推断 | +| `state` | `reserved`、`active`、`released`、`recovery_pending`、`quarantined` 或 `cleanup_eligible` | +| `created_at`, `last_heartbeat`, `expires_at`, `restart_generation` | 单调时钟与重启恢复信息 | +| `reserved_bytes`, `reserved_files`, `observed_bytes`, `observed_files` | 预留与实测值均不可超过策略 | +| `free_space_before`, `free_space_after_reserve`, `free_space_floor` | 准入时的磁盘证据 | +| `candidate_hash`, `receipt_hash`, `release_reason`, `cleanup_policy_hash` | 清理和终结证据绑定 | + +状态变更只能由 Host 事务完成：`reserved -> active -> released`；重启或 owner 证明缺失时为 `recovery_pending`；任何路径、内容、owner 或账本不确定时为 `quarantined`；`cleanup_eligible` 只是候选资格，不是删除动作。 + +## 配额、文件数与剩余空间门槛 + +Host policy 必须明确登记 `task_byte_limit`、`task_file_limit`、`target_family_byte_limit`、`target_family_file_limit`、`global_reserved_byte_limit`、`global_reserved_file_limit`、`free_space_floor_bytes` 和 `emergency_floor_bytes`；缺少或溢出任何值返回 `STORAGE_POLICY_MISSING`。准入要求同时满足： + +```text +requested_bytes <= task_byte_limit +requested_files <= task_file_limit +family_observed + family_reserved + requested <= family_limit +global_reserved + requested <= global_reserved_limit +free_before - (global_reserved + requested) >= free_space_floor_bytes +``` + +心跳按实测 bytes/files 重新核验。超出 byte/file 限额或 `free_space` 低于 floor 时，阻止新的 storage reservation 并返回稳定错误；不自动终止不属于本 lease 的进程，不删除活动目录。低于 emergency floor 时进入全局 pressure 状态，只允许释放、恢复和只读 preview；统计失败同样 fail-closed。 + +## 重启恢复 + +Host 启动先锁定 ledger，读取上一次 epoch 并增加 `restart_generation`。对每条 `active`/`reserved` 记录，只有同时匹配 `host_instance_id`、进程 PID、进程启动时间、owner epoch 和有效心跳的进程才可恢复为 active；其他记录转为 `recovery_pending`。Host 重新测量批准根并验证 target key、ledger path 和内容 manifest；缺失 receipt、路径变化、脏状态、未知文件或锁检测失败均转为 `quarantined`。恢复未完成前不允许 apply。中断中的 apply 带有 journal；重启后必须重新执行 candidate hash 和 owner 检查，不能按“已开始删除”继续盲删。 + +## Preview、候选哈希、复核与 Apply + +1. `preview` 只读扫描 ledger 已登记的 approved generated roots，按规范路径排序，记录每个候选的 `path_identity`、类型、bytes、files、content/manifest hash、owner 状态、dirty/source/session 分类、原因、ledger epoch 和 policy hash，生成 `candidate_hash = SHA-256(canonical manifest)`。 +2. 调用者提交精确的 `candidate_hash`、`policy_hash`、`ledger_epoch` 和有限 batch 上限；Host 取得 storage writer fence 后重新 stat、重新哈希并重新读取 lease/process/lock/Git 状态。 +3. 任一值变化返回 `STORAGE_CANDIDATE_STALE`，释放 fence，不产生删除。任一候选是 unknown、active、dirty、source 或 session，整个候选项保持保护并记录稳定错误。 +4. 只有全部复核通过的 disposable generated candidates 才能 apply；每个动作写入 journal 和 receipt，删除后立即验证路径不存在、账本状态为 released，并报告实际 bytes/files。部分失败保留未完成项并返回 `STORAGE_APPLY_INCOMPLETE`，不得扩大下一批范围。 + +清理永远不扫描批准根之外的目录，不跟随 reparse point，不依据大小、最近时间或目录名称猜测资格。未知、活动、脏 worktree、任意 source 文件、活动或未归档 session 永不自动删除。 + +## GitHub 可达性与精确路径授权 + +任何源代码、worktree 或本地代码目录的删除都是独立的 `source_cleanup` 操作。Host 必须同时证明：精确 commit 已在配置的 GitHub remote/repository 可达；remote identity 与本地 repository identity 匹配；worktree `status --porcelain` 为空；分支不是当前/受保护分支；没有 active lease、运行进程、待审查候选或未完成 receipt；并持有未过期的 `path_authorization`，其绑定 exact absolute path、repository identity、commit/tree hash、授权者、签发时间和 expiry。上传成功或 GitHub 可达本身不能替代该授权。remote 不可达、commit 不可证明、路径不精确或任一 owner 状态不明，分别返回 `GITHUB_REACHABILITY_UNAVAILABLE`、`GITHUB_COMMIT_NOT_REACHABLE` 或 `PATH_AUTHORIZATION_REQUIRED`，本地代码保持不变。 + +## 归档会话的 CAS 去重 + +CAS 去重不是通用清理。仅当 session 状态为 `archived`、对象不可变、content hash 与长度匹配、没有活动会话/lease/checkpoint/reference、且 ledger CAS transaction 成功时，Host 才能把重复对象的引用原子地指向唯一对象，再删除重复对象并写 receipt。任何 hash、引用计数、状态或锁读取失败都保留两个对象并返回 `STORAGE_CAS_MISMATCH` 或 `STORAGE_CAS_REFERENCE_ACTIVE`。活跃、未归档、内容不完整或未知来源的 session 不参与去重。 + +## 稳定错误 + +接口只返回固定 code、canonical detail 和 receipt identity。核心 code 为：`STORAGE_ROOT_NOT_APPROVED`、`STORAGE_TARGET_KEY_INVALID`、`STORAGE_POLICY_MISSING`、`STORAGE_QUOTA_EXCEEDED`、`STORAGE_FILE_LIMIT_EXCEEDED`、`STORAGE_FREE_SPACE_FLOOR`、`STORAGE_STAT_UNAVAILABLE`、`STORAGE_LEASE_CONFLICT`、`STORAGE_RECOVERY_REQUIRED`、`STORAGE_CANDIDATE_STALE`、`STORAGE_PROTECTED_UNKNOWN`、`STORAGE_PROTECTED_ACTIVE`、`STORAGE_PROTECTED_DIRTY`、`STORAGE_PROTECTED_SOURCE`、`STORAGE_PROTECTED_SESSION`、`STORAGE_APPLY_INCOMPLETE`、`STORAGE_POSTCHECK_FAILED`、`GITHUB_REACHABILITY_UNAVAILABLE`、`GITHUB_COMMIT_NOT_REACHABLE`、`PATH_AUTHORIZATION_REQUIRED`、`STORAGE_CAS_MISMATCH` 和 `STORAGE_CAS_REFERENCE_ACTIVE`。同一事实和输入必须得到同一 code，不以自然语言猜测替换 code。 + +## 最小 TDD 与编译验收 + +1. target key 的相同语义复用、任一构建字段变化分叉； +2. byte/file/floor/global reservation 四类准入和超限 fail-closed； +3. ledger reserve/heartbeat/release 及重启后 PID 重用防护； +4. preview hash 在内容、owner、ledger epoch 变化后失效； +5. unknown/active/dirty/source/session 五类保护零删除； +6. GitHub reachable 但无 exact path authorization 仍拒绝； +7. archived CAS 等哈希去重可提交，不匹配保持双对象； +8. 旧 ledger/无 ledger 的迁移和回滚保持未知根保护。 + +每项先产生可复现 RED，再以最小改动转 GREEN；不添加与边界无关的大型测试矩阵，不以单测替代 Host 运行证据。实现验收至少包括 Host 受影响 crate 的 `cargo check --locked -j1`、DevKit 变更 Python 文件的 `py_compile`、manifest/schema 校验、`git diff --check`，以及一个受控的 preview/apply 运行回执。磁盘 pressure 或统计失败时测试/build 也应停止新增 target 并报告稳定错误。 + +## 迁移与回滚 + +迁移使用 `storage-ledger-v1` 的事务性新表/文件和原子提交，先生成只读 inventory 与备份，再把已知 generated 根登记为 `recovery_pending`；旧 target、未知目录和 source 不因迁移自动删除。迁移失败恢复旧账本快照，所有产物保留。1.1.2 客户端没有 storage intent 时，Host 只允许受保护的兼容观察，不允许无账本写入。 + +回滚 1.1.3 时停止新的 storage admission，完成或标记现有 lease 的 recovery receipt，保留 ledger、journal 和所有未验证根；1.1.2 可读取既有代码但不能执行 1.1.3 的 apply。重新启用时从 ledger epoch 继续，不重建随机 target。任何部分迁移或 apply 失败均可回到上一个 ledger snapshot，绝不通过 `reset`、全盘删除或隐式路径扩张恢复。 + +## 1.1.3 版本边界 + +本版本交付：确定性 generated target 根；Host storage lease ledger；task/family/global byte-file-free gates；重启恢复与 pressure fail-closed；preview/candidate/recheck/apply 和 receipts；GitHub reachability 加 exact path authorization；archived session CAS dedupe；稳定错误、迁移、回滚和最小验收工具。 + +本版本不交付：远程 session/Atlas 同步、自动删除源代码或普通 session、全盘清理器、透明压缩、跨机器共享 target、额度绕过、Fast Lane route/lease/context/capability 降级，以及未经过用户精确授权的分支/worktree 删除。1.1.3 的完成条件是这些边界在 Host 与 DevKit 的生产路径中均有实现和 receipt 证据，而不是仅有设计文档或局部静态测试。 From 4585e6b9830cec3c87e699929557ab309485ce1c Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E5=93=80=E6=B4=9B=E8=8A=99?= Date: Sat, 29 Aug 2026 23:20:28 +0800 Subject: [PATCH 02/39] docs: plan 1.1.3 storage governance --- .../plans/2026-08-29-owned-cleanup-1.1.3.md | 552 ++++++++++ ...26-08-29-source-session-retention-1.1.3.md | 957 ++++++++++++++++++ .../2026-08-29-storage-firewall-1.1.3.md | 592 +++++++++++ 3 files changed, 2101 insertions(+) create mode 100644 docs/superpowers/plans/2026-08-29-owned-cleanup-1.1.3.md create mode 100644 docs/superpowers/plans/2026-08-29-source-session-retention-1.1.3.md create mode 100644 docs/superpowers/plans/2026-08-29-storage-firewall-1.1.3.md diff --git a/docs/superpowers/plans/2026-08-29-owned-cleanup-1.1.3.md b/docs/superpowers/plans/2026-08-29-owned-cleanup-1.1.3.md new file mode 100644 index 0000000..07b9c06 --- /dev/null +++ b/docs/superpowers/plans/2026-08-29-owned-cleanup-1.1.3.md @@ -0,0 +1,552 @@ +# Storage Lease Ledger and Owned Cleanup 1.1.3 Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Persist every admitted generated-storage lease across process restarts and make generated-cache cleanup a bounded preview/recheck/apply transaction owned by the Codex Host. + +**Architecture:** Plan 1 supplies the validated `StorageAdmissionReceipt` and deterministic target root. This plan adds a host-owned JSON ledger with atomic replacement, owner/process fencing, restart recovery, byte/file/free-space accounting, and a fair pressure state; the DevKit only forwards typed status/preview/apply requests over the authenticated bridge. A cleanup candidate is immutable evidence, not permission: only a fresh candidate hash, policy hash, ledger epoch, writer fence, and post-stat match can authorize a bounded generated-cache deletion. + +**Tech Stack:** Rust 2021 (`serde`, `serde_json`, `sha2`, `tokio`, `std::fs`, `std::time`), Python 3.11 (`dataclasses`, `hashlib`, `json`, `pathlib`), FastMCP/Pydantic, and the existing atomic JSON replacement and authenticated inherited-handle transport. + +--- + +## Scope and file map + +Line ranges refer to the Plan 1 baseline (`37029a9` for DevKit and +`552fe8035d` for Host). Re-read the named symbol before editing because Plan 1 +will add the storage admission types. + +DevKit: + +- Create `mcp-tools/devkit_runtime/storage_ledger.py`: typed status, preview, + and apply request/receipt projections; it must never inspect or delete a host + path. +- Modify `mcp-tools/devkit_runtime/host_bridge.py:218-264,916-1045` to carry + `storage_status`, `storage_preview`, and `storage_apply` request/receipt + frames through the authenticated session. +- Modify `mcp-tools/devkit_runtime/host_session.py:159-335,636-735` with + `storage_status()`, `storage_preview()`, and `storage_apply()` methods that + return only stable codes, hashes, counts, and opaque receipt identities. +- Modify `mcp-tools/server.py:179-289,1008-1314` and + `mcp-tools/devkit_runtime/tool_metadata.py:1-28` to add the read-only + `storage_status`/`storage_preview` tools and the explicitly destructive + `storage_apply` tool; the apply model requires candidate/policy/epoch hashes + and a finite batch limit. +- Create `mcp-tools/tests/test_storage_ledger.py` and modify + `mcp-tools/tests/test_mcp_contract.py:240-360` for tool annotations and + exact request validation. + +Codex Host: + +- Create `codex-rs/core/src/fast_lane_host_dispatch/storage_ledger.rs`: the + `LeaseRecord`, ledger snapshot, atomic journal, owner probe, recovery state, + quota accounting, candidate manifest, and generated apply transaction. +- Create `codex-rs/core/src/fast_lane_host_dispatch/storage_ledger_tests.rs`. +- Modify `codex-rs/core/src/fast_lane_host_dispatch/mod.rs:1-49` to register and + export the ledger types to the coordinator. +- Modify `codex-rs/core/src/fast_lane_host_dispatch/storage_firewall.rs` at + its admission/release methods to call the ledger rather than maintaining + process-local counters. +- Modify `codex-rs/core/src/fast_lane_host_dispatch/registry.rs:567-2060` + and `coordinator.rs:393-580,1218-1260` to recover/open the ledger at host + startup, reserve/heartbeat/release records, and block admission in pressure + or recovery state. +- Modify `codex-rs/rmcp-client/src/inherited_host_bridge_protocol.rs:35-240` + and `.../envelope.rs:25-220` for exact ledger/preview/apply wire schemas. +- Modify `codex-rs/rmcp-client/src/inherited_host_bridge_protocol/session.rs:159-240,247-350,412-570` + to expose the typed operation queue without creating a second receiver. + +No source/session deletion, GitHub reachability, or CAS deduplication belongs +to this plan; those are Plan 3. No unknown directory can enter this ledger. + +## Shared lease schema and public operation contract + +This plan consumes Plan 1's `StorageAdmissionReceipt` and uses this exact +record shape for `schema == "2718lab.storage.lease.v1"`: + +```rust +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +pub(crate) struct LeaseRecord { + pub(crate) ledger_epoch: u64, + pub(crate) schema_version: String, + pub(crate) lease_id: String, + pub(crate) task_id: String, + pub(crate) assignment_id: String, + pub(crate) plan_binding: String, + pub(crate) project_identity: String, + pub(crate) repository_identity: String, + pub(crate) worktree_identity: String, + pub(crate) artifact_kind: String, + pub(crate) target_key: String, + pub(crate) path_identity: String, + pub(crate) owner_epoch: u64, + pub(crate) owner_kind: String, + pub(crate) process_id: u32, + pub(crate) process_start_time: u64, + pub(crate) host_instance_id: String, + pub(crate) state: LeaseState, + pub(crate) created_at: u64, + pub(crate) last_heartbeat: u64, + pub(crate) expires_at: u64, + pub(crate) restart_generation: u64, + pub(crate) reserved_bytes: u64, + pub(crate) reserved_files: u64, + pub(crate) observed_bytes: u64, + pub(crate) observed_files: u64, + pub(crate) free_space_before: u64, + pub(crate) free_space_after_reserve: u64, + pub(crate) free_space_floor: u64, + pub(crate) candidate_hash: Option, + pub(crate) receipt_hash: Option, + pub(crate) release_reason: Option, + pub(crate) cleanup_policy_hash: Option, +} +``` + +`LeaseState` is exactly `reserved | active | released | recovery_pending | +quarantined | cleanup_eligible`. The only legal automatic transitions are +`reserved -> active -> released`; restart evidence can move an active or +reserved record to `recovery_pending`, and failed verification can move it to +`quarantined`. `cleanup_eligible` never deletes anything by itself. +`StorageLedgerError::LeaseConflict` maps exactly to +`STORAGE_LEASE_CONFLICT`; it is returned for a stale owner proof, duplicate +activation, heartbeat after release, and release by a different owner. + +The public status/preview/apply shapes are: + +```json +{ + "schema": "2718lab.storage.preview.v1", + "ledger_epoch": 12, + "policy_hash": "sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "candidate_hash": "sha256:bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "candidates": [ + { + "path_identity": "sha256:cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc", + "artifact_kind": "cargo-target", + "bytes": 1024, + "files": 3, + "content_hash": "sha256:dddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd", + "owner_state": "none", + "classification": "generated-disposable", + "lease_id": "sha256:eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee" + } + ] +} +``` + +`storage_apply` accepts exactly `candidate_hash`, `policy_hash`, +`ledger_epoch`, and `batch_limit` (1 through 16). It returns +`STORAGE_CANDIDATE_STALE`, `STORAGE_PROTECTED_UNKNOWN`, +`STORAGE_PROTECTED_ACTIVE`, `STORAGE_PROTECTED_DIRTY`, or +`STORAGE_APPLY_INCOMPLETE` without deleting when any recheck differs. + +## Implementation tasks + +### Task 1: Define ledger and operation RED tests + +**Files:** +- Create: `mcp-tools/tests/test_storage_ledger.py` +- Create: `codex-rs/core/src/fast_lane_host_dispatch/storage_ledger_tests.rs` +- Modify: `mcp-tools/tests/test_mcp_contract.py:240-360` + +- [ ] **Step 1: Add the Python RED test for exact apply fields and bounded batch.** + +```python +def test_storage_apply_rejects_path_and_unbounded_batch(): + from devkit_runtime.storage_ledger import StorageApplyRequest, StorageLedgerError + + try: + StorageApplyRequest.from_mapping({ + "candidate_hash": "sha256:" + "a" * 64, + "policy_hash": "sha256:" + "b" * 64, + "ledger_epoch": 1, + "batch_limit": 17, + "path": "G:/source" + }) + except StorageLedgerError as error: + assert error.code == "STORAGE_CANDIDATE_STALE" + else: + raise AssertionError("invalid apply request was accepted") +``` + +- [ ] **Step 2: Add the Rust RED test for an invalid transition.** + +```rust +#[test] +fn released_lease_cannot_receive_a_heartbeat() { + let mut ledger = test_ledger(); + let lease = ledger.reserve(test_admission()).unwrap(); + ledger.activate(&lease.lease_id, owner()).unwrap(); + ledger.release(&lease.lease_id, "terminal").unwrap(); + assert_eq!( + ledger.heartbeat( + &lease.lease_id, + owner(), + ObservedStorage { bytes: 100, files: 1 }, + ), + Err(StorageLedgerError::LeaseConflict), + ); +} +``` + +- [ ] **Step 3: Run only the new RED tests.** + +```powershell +$env:PYTEST_DISABLE_PLUGIN_AUTOLOAD='1'; python -m pytest tests/test_storage_ledger.py::test_storage_apply_rejects_path_and_unbounded_batch -q -o cache_dir=G:\2718lab\_codex\.codex-task-temp\storage-113-ledger-pytest +Push-Location 'G:\2718lab\_codex\.codex-task-temp\codex-host-mcp-fix-recovery2\codex-rs'; $env:CARGO_TARGET_DIR='G:\2718lab\_codex\.codex-task-temp\storage-113-ledger-rust-target'; cargo test -p codex-core released_lease_cannot_receive_a_heartbeat --locked -j1; Pop-Location +``` + +Expected: both commands fail because the ledger types do not exist. The +failure must occur before any production path or deletion call. + +- [ ] **Step 4: Commit only the RED contract.** + +```powershell +Push-Location 'G:\2718lab\_codex\.codex-task-temp\devkit-1.1.2-recovery'; git add mcp-tools/tests/test_storage_ledger.py mcp-tools/tests/test_mcp_contract.py; git commit -m 'test: define owned storage ledger contract'; Pop-Location +Push-Location 'G:\2718lab\_codex\.codex-task-temp\codex-host-mcp-fix-recovery2'; git add codex-rs/core/src/fast_lane_host_dispatch/storage_ledger_tests.rs; git commit -m 'test: define owned storage ledger contract'; Pop-Location +``` + +### Task 2: Implement atomic ledger snapshots and schema migration + +**Files:** +- Create: `codex-rs/core/src/fast_lane_host_dispatch/storage_ledger.rs` +- Modify: `codex-rs/core/src/fast_lane_host_dispatch/mod.rs:1-49` +- Test: `codex-rs/core/src/fast_lane_host_dispatch/storage_ledger_tests.rs` + +- [ ] **Step 1: Define the store and exact snapshot envelope.** + +```rust +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +pub(crate) struct LedgerSnapshot { + pub(crate) schema: String, + pub(crate) ledger_epoch: u64, + pub(crate) restart_generation: u64, + pub(crate) host_instance_id: String, + pub(crate) policy_hash: String, + pub(crate) leases: Vec, + pub(crate) journal: Option, +} + +pub(crate) struct StorageLedger { + path: PathBuf, + snapshot: LedgerSnapshot, + owner_probe: Box, + capacity: Box, +} +``` + +`open` must reject a symlink/reparse-point ledger file, malformed JSON, a +non-monotonic epoch, unknown state, duplicate `lease_id`, or a path whose +canonical parent is outside the approved generated root. An absent file is +opened as a zero-lease `storage-ledger-v1` snapshot only after the parent root +has been proved approved; it is not an authorization to write arbitrary roots. + +- [ ] **Step 2: Implement atomic replacement with a journal.** + +```rust +fn persist(&mut self, next: LedgerSnapshot) -> Result<(), StorageLedgerError> { + validate_snapshot(&next)?; + let temporary = self.path.with_extension("json.stage"); + let bytes = serde_json::to_vec(&next).map_err(|_| StorageLedgerError::StatUnavailable)?; + let mut file = OpenOptions::new().write(true).create_new(true).open(&temporary) + .map_err(|_| StorageLedgerError::StatUnavailable)?; + file.write_all(&bytes).map_err(|_| StorageLedgerError::StatUnavailable)?; + file.sync_all().map_err(|_| StorageLedgerError::StatUnavailable)?; + replace_file_durably(&temporary, &self.path)?; + self.snapshot = next; + Ok(()) +} +``` + +The Windows replace helper must use the same write-through replacement +semantics already used by `registry.rs:4281-4380`; Unix uses `rename` after +`sync_all`. A failed replacement leaves the prior snapshot and the stage file +is removed only when its identity still matches the stage created by this +operation. + +- [ ] **Step 3: Add migration and rollback tests, then turn the RED schema tests green.** + +```rust +#[test] +fn old_or_missing_ledger_becomes_recovery_pending_without_deletion() { + let mut ledger = open_fixture_with_legacy_snapshot(); + let result = ledger.recover_after_restart(current_owner_set_empty()); + assert!(result.is_ok()); + assert!(ledger.records().iter().all(|record| record.state == LeaseState::RecoveryPending)); + assert!(fixture_generated_file().exists()); +} +``` + +```powershell +Push-Location 'G:\2718lab\_codex\.codex-task-temp\codex-host-mcp-fix-recovery2\codex-rs'; $env:CARGO_TARGET_DIR='G:\2718lab\_codex\.codex-task-temp\storage-113-ledger-rust-target'; cargo test -p codex-core storage_ledger --locked -j1; Pop-Location +``` + +Expected: the focused ledger tests pass; migration failure returns to the +previous snapshot and keeps every generated file. + +- [ ] **Step 4: Commit the durable ledger core.** + +```powershell +Push-Location 'G:\2718lab\_codex\.codex-task-temp\codex-host-mcp-fix-recovery2'; git add codex-rs/core/src/fast_lane_host_dispatch/storage_ledger.rs codex-rs/core/src/fast_lane_host_dispatch/storage_ledger_tests.rs codex-rs/core/src/fast_lane_host_dispatch/mod.rs; git commit -m 'feat: persist storage lease ledger atomically'; Pop-Location +``` + +### Task 3: Enforce reserve, heartbeat, release, and pressure gates + +**Files:** +- Modify: `codex-rs/core/src/fast_lane_host_dispatch/storage_firewall.rs` +- Modify: `codex-rs/core/src/fast_lane_host_dispatch/registry.rs:567-1668` +- Modify: `codex-rs/core/src/fast_lane_host_dispatch/coordinator.rs:393-580,1218-1260` +- Test: `codex-rs/core/src/fast_lane_host_dispatch/storage_ledger_tests.rs` + +- [ ] **Step 1: Add RED accounting tests for all four admission equations.** + +```rust +#[test] +fn byte_file_global_and_floor_limits_fail_closed() { + let cases = [ + (AdmissionMutation::TaskBytes, "STORAGE_QUOTA_EXCEEDED"), + (AdmissionMutation::TaskFiles, "STORAGE_FILE_LIMIT_EXCEEDED"), + (AdmissionMutation::GlobalReserved, "STORAGE_QUOTA_EXCEEDED"), + (AdmissionMutation::FreeFloor, "STORAGE_FREE_SPACE_FLOOR"), + ]; + for (mutation, code) in cases { + let firewall = fixture_firewall(mutation); + assert_eq!(firewall.admit(test_intent()).unwrap_err().code(), code); + assert!(fixture_target_root().read_dir().unwrap().next().is_none()); + } +} +``` + +- [ ] **Step 2: Implement lease state methods with owner fencing.** + +```rust +pub(crate) fn reserve(&mut self, admission: StorageAdmissionReceipt, now: u64) -> Result; +pub(crate) fn activate(&mut self, lease_id: &str, owner: OwnerProof) -> Result<(), StorageLedgerError>; +pub(crate) fn heartbeat(&mut self, lease_id: &str, owner: OwnerProof, observed: ObservedStorage) -> Result; +pub(crate) fn release(&mut self, lease_id: &str, owner: OwnerProof, reason: &str) -> Result; +``` + +`heartbeat` remeasures bytes/files and applies the task, family, global, and +free-space equations before persisting. If an observation exceeds a limit, +new reservations return the stable pressure/quota code; the active lease is +not killed and its directory is not deleted. At or below +`emergency_floor_bytes`, the ledger enters `pressure=true` and allows only +release, recovery, and read-only preview operations. + +- [ ] **Step 3: Connect `registry.rs` and the coordinator to the ledger.** + +```rust +let admission = storage_firewall.admit(intent)?; +let lease = storage_ledger.reserve(admission, clock.now()?)?; +let prepared = adapter.prepare_batch_with_storage(batch, lease.clone()).await?; +storage_ledger.activate(&lease.lease_id, owner_probe.current()?)?; +``` + +Every failed preparation calls `release` with `"prepare_failed"`; every +terminal/recovery path calls it with its exact reason. Releasing the Fast Lane +scope lease and releasing storage are separate journal entries bound by the +same `assignment_id`, `plan_binding`, and receipt hash. + +- [ ] **Step 4: Run the focused accounting and core compile gates.** + +```powershell +Push-Location 'G:\2718lab\_codex\.codex-task-temp\codex-host-mcp-fix-recovery2\codex-rs'; $env:CARGO_TARGET_DIR='G:\2718lab\_codex\.codex-task-temp\storage-113-ledger-rust-target'; cargo test -p codex-core byte_file_global_and_floor_limits_fail_closed --locked -j1; cargo check -p codex-core --lib --locked -j1; Pop-Location +``` + +Expected: the four cases pass and `cargo check` finishes with zero warnings. +If disk statistics fail, the result is `STORAGE_STAT_UNAVAILABLE` and no new +target root is created. + +- [ ] **Step 5: Commit the quota and lifecycle wiring.** + +```powershell +Push-Location 'G:\2718lab\_codex\.codex-task-temp\codex-host-mcp-fix-recovery2'; git add codex-rs/core/src/fast_lane_host_dispatch/storage_firewall.rs codex-rs/core/src/fast_lane_host_dispatch/registry.rs codex-rs/core/src/fast_lane_host_dispatch/coordinator.rs codex-rs/core/src/fast_lane_host_dispatch/storage_ledger_tests.rs; git commit -m 'feat: bind storage leases to quota lifecycle'; Pop-Location +``` + +### Task 4: Implement restart owner recovery and fail-closed quarantine + +**Files:** +- Modify: `codex-rs/core/src/fast_lane_host_dispatch/storage_ledger.rs` +- Modify: `codex-rs/core/src/fast_lane_host_dispatch/registry.rs:3809-4380` +- Test: `codex-rs/core/src/fast_lane_host_dispatch/storage_ledger_tests.rs` + +- [ ] **Step 1: Add RED tests for PID reuse, changed path, unknown files, and locked-stat recovery.** + +```rust +#[test] +fn restart_requires_instance_pid_start_and_owner_epoch() { + let mut ledger = open_fixture_with_active_lease(owner_with(41, 900, 7)); + ledger.recover_after_restart(owner_with(41, 901, 7)).unwrap(); + assert_eq!(ledger.records()[0].state, LeaseState::RecoveryPending); + assert_eq!(ledger.records()[0].restart_generation, 2); +} +``` + +- [ ] **Step 2: Implement `OwnerProbe` and recovery validation.** + +```rust +pub(crate) trait OwnerProbe: Send + Sync { + fn current(&self) -> Result; + fn matches(&self, owner: &OwnerProof) -> Result; +} + +fn recover_record(record: &mut LeaseRecord, owner_probe: &dyn OwnerProbe, root: &Path) -> Result<(), StorageLedgerError> { + if record.state != LeaseState::Active && record.state != LeaseState::Reserved { + return Ok(()); + } + if !owner_probe.matches(&OwnerProof::from_record(record))? + || !verify_target_identity(root, record)? + || !manifest_matches(record)? + { + record.state = LeaseState::Quarantined; + return Ok(()); + } + record.state = LeaseState::Active; + Ok(()) +} +``` + +The host increments `restart_generation` under the ledger lock before examining +records. Missing receipt, path change, dirty state, unknown file, or a failed +lock/stat check becomes `quarantined`; an owner that cannot be proved becomes +`recovery_pending` until a later explicit recovery receipt. `apply` is blocked +while any recovery remains unresolved. + +- [ ] **Step 3: Add a restart recovery receipt and verify no deletion occurred.** + +```rust +assert_eq!(receipt.code(), "STORAGE_RECOVERY_REQUIRED"); +assert_eq!(receipt.restart_generation(), 2); +assert!(fixture_generated_file().exists()); +``` + +- [ ] **Step 4: Run the focused recovery probe and compile gate.** + +```powershell +Push-Location 'G:\2718lab\_codex\.codex-task-temp\codex-host-mcp-fix-recovery2\codex-rs'; $env:CARGO_TARGET_DIR='G:\2718lab\_codex\.codex-task-temp\storage-113-ledger-rust-target'; cargo test -p codex-core restart_requires_instance_pid_start_and_owner_epoch --locked -j1; cargo check -p codex-core --lib --locked -j1; Pop-Location +``` + +Expected: `1 passed`, then a zero-warning compile. + +- [ ] **Step 5: Commit restart recovery.** + +```powershell +Push-Location 'G:\2718lab\_codex\.codex-task-temp\codex-host-mcp-fix-recovery2'; git add codex-rs/core/src/fast_lane_host_dispatch/storage_ledger.rs codex-rs/core/src/fast_lane_host_dispatch/registry.rs codex-rs/core/src/fast_lane_host_dispatch/storage_ledger_tests.rs; git commit -m 'feat: recover storage ownership across restarts'; Pop-Location +``` + +### Task 5: Add preview hash, recheck fence, and bounded generated apply + +**Files:** +- Modify: `codex-rs/core/src/fast_lane_host_dispatch/storage_ledger.rs` +- Modify: `codex-rs/rmcp-client/src/inherited_host_bridge_protocol.rs:35-240` +- Modify: `codex-rs/rmcp-client/src/inherited_host_bridge_protocol/envelope.rs:25-220` +- Modify: `codex-rs/rmcp-client/src/inherited_host_bridge_protocol/session.rs:412-570` +- Test: `codex-rs/core/src/fast_lane_host_dispatch/storage_ledger_tests.rs` + +- [ ] **Step 1: Add RED tests for candidate invalidation and protected classifications.** + +```rust +#[test] +fn preview_hash_invalidates_on_epoch_owner_or_content_change() { + let mut ledger = test_ledger_with_disposable_candidate(); + let preview = ledger.preview().unwrap(); + ledger.bump_epoch_for_test(); + let error = ledger.apply(&ApplyRequest::from_preview(&preview, 1)).unwrap_err(); + assert_eq!(error.code(), "STORAGE_CANDIDATE_STALE"); + assert!(fixture_candidate_path().exists()); +} +``` + +- [ ] **Step 2: Implement canonical candidate manifest and preview hash.** + +```rust +pub(crate) fn preview(&self) -> Result { + let mut candidates = self.scan_registered_generated_roots()?; + candidates.sort_by(|left, right| left.path_identity.cmp(&right.path_identity)); + let manifest = serde_json::json!({ + "schema": "2718lab.storage.preview.v1", + "ledger_epoch": self.snapshot.ledger_epoch, + "policy_hash": self.snapshot.policy_hash, + "candidates": candidates, + }); + Ok(StoragePreview { manifest, candidate_hash: canonical_hash(&manifest)? }) +} +``` + +The scan follows no reparse point, visits only ledger-registered generated +roots, and labels active/unknown/dirty/source/session entries as protected. +It does not select by size, age, or directory name. + +- [ ] **Step 3: Implement apply as recheck, journal, delete, postcheck.** + +```rust +pub(crate) fn apply(&mut self, request: ApplyRequest) -> Result { + let preview = self.preview()?; + if request.candidate_hash != preview.candidate_hash + || request.policy_hash != self.snapshot.policy_hash + || request.ledger_epoch != self.snapshot.ledger_epoch + { + return Err(StorageLedgerError::CandidateStale); + } + let fence = self.writer_fence()?; + let selected = preview.generated_disposable(request.batch_limit)?; + for candidate in selected { + self.recheck_candidate(&candidate, &fence)?; + self.write_journal_started(&candidate)?; + self.remove_verified_generated_path(&candidate)?; + if candidate.path_exists()? { + return Err(StorageLedgerError::PostcheckFailed); + } + self.mark_released(&candidate.lease_id)?; + } + self.write_receipt_and_release_fence() +} +``` + +A failed item returns `STORAGE_APPLY_INCOMPLETE` and leaves all later items +untouched. A changed candidate releases the writer fence without deletion. +Apply cannot run while the ledger is in pressure recovery or while any +candidate is active, unknown, dirty, source, or session classified. + +- [ ] **Step 4: Add the exact bridge operations and DevKit projections.** + +```python +def storage_apply(self, request: StorageApplyRequest) -> dict[str, object]: + if request.batch_limit < 1 or request.batch_limit > 16: + return {"code": "STORAGE_CANDIDATE_STALE"} + response = _host_session().storage_apply(request.to_wire()) + return project_storage_receipt(response) +``` + +The bridge validator requires exact schemas +`2718lab.storage.status.v1`, `2718lab.storage.preview.v1`, and +`2718lab.storage.apply.v1`; `storage_apply` is the only destructive operation. +The DevKit projection strips absolute paths and owner/process values before +returning an MCP result. + +- [ ] **Step 5: Run preview/apply focused tests and compile-first gates.** + +```powershell +Push-Location 'G:\2718lab\_codex\.codex-task-temp\codex-host-mcp-fix-recovery2\codex-rs'; $env:CARGO_TARGET_DIR='G:\2718lab\_codex\.codex-task-temp\storage-113-ledger-rust-target'; cargo test -p codex-core preview_hash_invalidates_on_epoch_owner_or_content_change --locked -j1; cargo check -p codex-core --lib --locked -j1; Pop-Location +$env:PYTEST_DISABLE_PLUGIN_AUTOLOAD='1'; Push-Location 'G:\2718lab\_codex\.codex-task-temp\devkit-1.1.2-recovery\mcp-tools'; python -m pytest tests/test_storage_ledger.py -q -o cache_dir=G:\2718lab\_codex\.codex-task-temp\storage-113-ledger-pytest; python -m py_compile devkit_runtime/storage_ledger.py devkit_runtime/host_bridge.py devkit_runtime/host_session.py server.py; Pop-Location +``` + +Expected: focused Rust and Python tests pass, both compile gates are silent, +and only the named task-local target/cache roots are touched. + +- [ ] **Step 6: Commit preview/apply and bridge integration.** + +```powershell +Push-Location 'G:\2718lab\_codex\.codex-task-temp\devkit-1.1.2-recovery'; git add mcp-tools/devkit_runtime/storage_ledger.py mcp-tools/devkit_runtime/host_bridge.py mcp-tools/devkit_runtime/host_session.py mcp-tools/devkit_runtime/tool_metadata.py mcp-tools/server.py mcp-tools/tests/test_storage_ledger.py mcp-tools/tests/test_mcp_contract.py; git commit -m 'feat: add owned storage preview and apply'; Pop-Location +Push-Location 'G:\2718lab\_codex\.codex-task-temp\codex-host-mcp-fix-recovery2'; git add codex-rs/core/src/fast_lane_host_dispatch/storage_ledger.rs codex-rs/core/src/fast_lane_host_dispatch/storage_ledger_tests.rs codex-rs/rmcp-client/src/inherited_host_bridge_protocol.rs codex-rs/rmcp-client/src/inherited_host_bridge_protocol/envelope.rs codex-rs/rmcp-client/src/inherited_host_bridge_protocol/session.rs; git commit -m 'feat: add owned storage preview and apply'; Pop-Location +``` + +## Plan 2 acceptance gate and handoff + +- [ ] Re-read the design sections “Task Storage Lease Ledger”, “重启恢复”, “Preview、候选哈希、复核与 Apply”, and “稳定错误”; map every listed field/code to a task above. +- [ ] Run `git diff --check` in both worktrees and verify only the mapped files changed. +- [ ] Run DevKit `py_compile` and Host `cargo check -p codex-core --lib --locked -j1` with the one named task target; zero warnings are required before any package build. +- [ ] Record receipts for reserve, heartbeat, release, restart recovery, candidate stale, protected candidate, successful one-item generated apply, and partial apply. Each receipt must include ledger epoch and receipt hash. +- [ ] Verify a missing/legacy ledger migrates to recovery protection, a failed atomic write leaves the previous snapshot, pressure blocks new reservations, and no operation kills another process or scans outside registered generated roots. +- [ ] Do not implement GitHub source deletion, ordinary/active session deletion, CAS dedupe, compression, or remote synchronization. Plan 3 consumes the ledger's protected classifications and apply fence. diff --git a/docs/superpowers/plans/2026-08-29-source-session-retention-1.1.3.md b/docs/superpowers/plans/2026-08-29-source-session-retention-1.1.3.md new file mode 100644 index 0000000..4fcd139 --- /dev/null +++ b/docs/superpowers/plans/2026-08-29-source-session-retention-1.1.3.md @@ -0,0 +1,957 @@ +# Source and Session Retention 1.1.3 Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Require GitHub reachability plus an unexpired exact-path authorization before any local source removal, repair the session index activity timestamp after durable rollout writes, and deduplicate only immutable archived CAS objects with no live reference. + +**Architecture:** Plans 1 and 2 provide deterministic generated roots, the persistent storage ledger, owner probes, and the preview/recheck/apply writer fence. This plan adds a separate Host-only retention authority: source cleanup is a two-phase candidate transaction guarded by remote identity, clean/unpushed/current-branch/active-lease checks and an opaque path authorization; session retention updates the append-only activity index after durable writes and limits CAS dedupe to the dedicated archived CAS object root. DevKit validates and forwards exact requests but never proves Git state, mints authorization, scans sessions, or removes a path. + +**Tech Stack:** Rust 2021 (`serde`, `serde_json`, `sha2`, Tokio, `std::process::Command`), Git CLI, Python 3.11 (`dataclasses`, `hashlib`, `json`), FastMCP/Pydantic, the existing authenticated inherited-handle bridge, `codex-rollout`, and the Plan 2 storage ledger/journal. + +--- + +## Scope and file map + +Line references start at DevKit commit `37029a9b1677aade4a2874d7fb3e30cb61f01092` +and Host commit `552fe8035d8bb928467fff428773fc4d5d34c168`. Plans 1 and 2 +will move some Host line numbers, so re-read every named symbol before editing. + +DevKit files: + +- Create `mcp-tools/devkit_runtime/storage_retention.py`: exact source-preview, + source-apply, session-CAS-preview, and session-CAS-apply request projections. + It accepts an opaque Host authorization token but never creates one. +- Modify `mcp-tools/devkit_runtime/host_bridge.py:218-264,2298-2677` and + `mcp-tools/devkit_runtime/host_session.py:159-335,636-735`: carry the four + retention operations over the existing authenticated single-reader/session + queue. +- Modify `mcp-tools/server.py:179-289,1008-1314` and + `mcp-tools/devkit_runtime/tool_metadata.py:5-23`: expose read-only preview + tools and destructive apply tools with exact bounded inputs. +- Create `mcp-tools/tests/test_storage_retention.py` and modify + `mcp-tools/tests/test_mcp_contract.py:240-360`: reject public path guessing, + missing authorization, unbounded batches, and destructive annotation drift. + +Codex Host files: + +- Create `codex-rs/core/src/fast_lane_host_dispatch/source_session_retention.rs`: + exact path authorization verification, GitHub remote reachability, source + candidate manifests, source apply receipts, and the adapter to archived CAS. +- Create + `codex-rs/core/src/fast_lane_host_dispatch/source_session_retention_tests.rs` + and modify `codex-rs/core/src/fast_lane_host_dispatch/mod.rs:1-49`. +- Modify `codex-rs/core/src/fast_lane_host_dispatch/storage_ledger.rs` from + Plan 2 to expose read-only active-lease/path checks and the existing writer + fence/journal to retention; no second ledger is created. +- Modify `codex-rs/core/src/fast_lane_host_dispatch/worktree.rs:65-119` and + `codex-rs/core/src/fast_lane_host_dispatch/codex_adapter.rs:802-979`: stop + automatic integrated-worktree removal, register a protected source cleanup + candidate, and route later removal through the retention authority. +- Modify `codex-rs/rmcp-client/src/inherited_host_bridge_protocol.rs:35-240`, + `codex-rs/rmcp-client/src/inherited_host_bridge_protocol/envelope.rs:25-220`, + and + `codex-rs/rmcp-client/src/inherited_host_bridge_protocol/session.rs:412-570` + with exact retention wire schemas. +- Modify `codex-rs/rollout/src/session_index.rs:20-294` and + `codex-rs/rollout/src/session_index_tests.rs:1-430`: append an activity row + after a durable rollout write while retaining the last explicit thread name. +- Modify `codex-rs/rollout/src/recorder.rs:829-1024,1624-1867` and + `codex-rs/rollout/src/recorder_tests.rs:90-190`: carry `codex_home/thread_id` + into the writer, preserve deferred creation, create a missing YYYY/MM/DD + directory on first persistence, and touch the activity index only after the + rollout flush succeeds. +- Create `codex-rs/rollout/src/archived_session_cas.rs` and + `codex-rs/rollout/src/archived_session_cas_tests.rs`; modify + `codex-rs/rollout/src/lib.rs:60-140` and + `codex-rs/rollout/src/rollout_reference_index.rs:19-90`: transactionally + deduplicate only registered `.blob` objects below + `archived_sessions/cas/objects`, using rollout reference evidence. + +The worker must not edit ordinary rollout bodies, compression policy, +`delete_thread.rs`, arbitrary project directories, or any file outside this +map. A source directory that is not a registered linked worktree is protected +in 1.1.3; support for deleting standalone clones requires a later design. + +## Exact contracts and invariants + +`source_cleanup_apply` accepts this exact shape. `exact_path` is required on +the destructive call so the user-visible authorization and the Host recheck +bind the same literal target; public receipts expose only `path_identity`. + +```json +{ + "schema": "2718lab.storage.source-cleanup-apply.v1", + "exact_path": "G:\\2718lab\\_codex\\.codex-task-temp\\writer-01", + "repository_identity": "sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "commit": "0123456789abcdef0123456789abcdef01234567", + "tree": "89abcdef0123456789abcdef0123456789abcdef", + "remote_name": "origin", + "candidate_hash": "sha256:bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "ledger_epoch": 14, + "authorization_token": "pathauth-v1.opaque-host-signed-value", + "batch_limit": 1 +} +``` + +The Host-decoded `PathAuthorization` has exact fields `schema`, +`authorization_id`, `exact_path`, `path_identity`, `repository_identity`, +`commit`, `tree`, `authorizer`, `issued_at`, `expires_at`, `nonce`, and +`authorization_hash`. It is signed or MAC-verified by a Host-injected +`PathAuthorizationVerifier`; DevKit cannot mint it. The authorization expires +after at most 15 minutes and authorizes exactly one literal canonical path, +repository identity, commit, and tree. + +The source operation is fail-closed unless all of these fresh facts agree: + +1. The target is one registered linked worktree below the approved task root, + not the repository root, current Host cwd, current worktree, or a protected + branch. +2. `git status --porcelain=v1 -z --untracked-files=all` is empty and `HEAD` plus + `HEAD^{tree}` exactly match the authorization. +3. The configured remote URL canonicalizes to the same + `github.com//` identity as the ledger record. +4. A Host-owned, quota-admitted bare verifier fetch proves the commit is an + ancestor of at least one currently advertised GitHub head. A failed network + or stat call is `GITHUB_REACHABILITY_UNAVAILABLE`; no matching head is + `GITHUB_COMMIT_NOT_REACHABLE` and therefore also covers unpushed commits. +5. Plan 2 reports no active/recovery/quarantined storage lease, Fast Lane scope + lease, running owner, pending integration, or incomplete cleanup receipt for + the exact path identity. +6. Candidate hash, ledger epoch, authorization, Git facts, and path identity + still match after acquiring the Plan 2 writer fence. + +Archived-session CAS uses a separate exact root and never treats a rollout +body as an object. Eligible paths have the form +`archived_sessions/cas/objects//<62 lowercase hex>.blob`. +Every object must be immutable, hash/length verified, referenced only by +archived records, absent from `RolloutReferenceIndex` live/current lineage, +and absent from active writer/lease/checkpoint/reference probes. Ordinary +`.jsonl`, `.jsonl.zst`, `sessions/**`, the current rollout, and any unknown file +are always protected. A dedupe transaction atomically repoints archived CAS +references to the lexically first verified object, persists the CAS index, +removes only redundant `.blob` objects, then writes a receipt. + +## Implementation tasks + +### Task 1: Freeze retention requests and protection failures with RED tests + +**Files:** +- Create: `mcp-tools/tests/test_storage_retention.py` +- Create: `codex-rs/core/src/fast_lane_host_dispatch/source_session_retention_tests.rs` +- Create: `codex-rs/rollout/src/archived_session_cas_tests.rs` +- Modify: `mcp-tools/tests/test_mcp_contract.py:240-360` + +- [ ] **Step 1: Add the Python RED test for mandatory exact path authorization.** + +```python +def test_source_cleanup_apply_requires_exact_path_and_host_authorization(): + from devkit_runtime.storage_retention import RetentionContractError, SourceCleanupApply + + value = { + "schema": "2718lab.storage.source-cleanup-apply.v1", + "repository_identity": "sha256:" + "a" * 64, + "commit": "0" * 40, + "tree": "1" * 40, + "remote_name": "origin", + "candidate_hash": "sha256:" + "b" * 64, + "ledger_epoch": 1, + "authorization_token": "", + "batch_limit": 1, + } + try: + SourceCleanupApply.from_mapping(value) + except RetentionContractError as error: + assert error.code == "PATH_AUTHORIZATION_REQUIRED" + else: + raise AssertionError("source cleanup accepted no exact authorization") +``` + +- [ ] **Step 2: Add Host RED tests for dirty, unpushed, active, and current worktrees.** + +```rust +#[test] +fn source_cleanup_protects_dirty_unpushed_active_and_current_worktrees() { + for (mutation, code) in [ + (SourceMutation::Dirty, "STORAGE_PROTECTED_DIRTY"), + (SourceMutation::Unpushed, "GITHUB_COMMIT_NOT_REACHABLE"), + (SourceMutation::ActiveLease, "STORAGE_PROTECTED_ACTIVE"), + (SourceMutation::CurrentWorktree, "STORAGE_PROTECTED_SOURCE"), + ] { + let fixture = source_fixture(mutation); + let error = fixture.authority.preview(fixture.request).unwrap_err(); + assert_eq!(error.code(), code); + assert!(fixture.worktree.exists()); + } +} +``` + +- [ ] **Step 3: Add the archived CAS RED test proving rollout bodies are outside the deletion domain.** + +```rust +#[test] +fn cas_dedupe_never_selects_current_active_or_rollout_body() { + let fixture = archived_cas_fixture_with_duplicate_and_rollout_body(); + let preview = fixture.store.preview(&fixture.protection).unwrap(); + assert_eq!(preview.groups.len(), 1); + assert!(preview.groups[0].objects.iter().all(|item| item.path.extension().unwrap() == "blob")); + assert!(fixture.current_rollout.exists()); + assert!(fixture.archived_rollout_body.exists()); +} +``` + +- [ ] **Step 4: Run only the new RED probes.** + +From the DevKit `mcp-tools` directory: + +```powershell +$env:PYTEST_DISABLE_PLUGIN_AUTOLOAD='1'; python -m pytest tests/test_storage_retention.py::test_source_cleanup_apply_requires_exact_path_and_host_authorization -q -o cache_dir=G:\2718lab\_codex\.codex-task-temp\storage-113-retention-pytest +``` + +From the Host `codex-rs` directory: + +```powershell +$env:CARGO_TARGET_DIR='G:\2718lab\_codex\.codex-task-temp\storage-113-retention-rust-target'; cargo test -p codex-core source_cleanup_protects_dirty_unpushed_active_and_current_worktrees --locked -j1 +$env:CARGO_TARGET_DIR='G:\2718lab\_codex\.codex-task-temp\storage-113-retention-rust-target'; cargo test -p codex-rollout cas_dedupe_never_selects_current_active_or_rollout_body --locked -j1 +``` + +Expected: all three commands fail because the retention modules do not yet +exist. No source, rollout, or CAS object is removed. + +- [ ] **Step 5: Commit RED tests separately in their owning repositories.** + +```powershell +Push-Location 'G:\2718lab\_codex\.codex-task-temp\devkit-1.1.2-recovery'; git add mcp-tools/tests/test_storage_retention.py mcp-tools/tests/test_mcp_contract.py; git commit -m 'test: define source and session retention contract'; Pop-Location +Push-Location 'G:\2718lab\_codex\.codex-task-temp\codex-host-mcp-fix-recovery2'; git add codex-rs/core/src/fast_lane_host_dispatch/source_session_retention_tests.rs codex-rs/rollout/src/archived_session_cas_tests.rs; git commit -m 'test: define source and session retention contract'; Pop-Location +``` + +### Task 2: Implement exact DevKit retention contracts and Host stable types + +**Files:** +- Create: `mcp-tools/devkit_runtime/storage_retention.py` +- Create: `codex-rs/core/src/fast_lane_host_dispatch/source_session_retention.rs` +- Modify: `codex-rs/core/src/fast_lane_host_dispatch/mod.rs:1-49` +- Test: `mcp-tools/tests/test_storage_retention.py` +- Test: `codex-rs/core/src/fast_lane_host_dispatch/source_session_retention_tests.rs` + +- [ ] **Step 1: Implement the exact Python apply parser.** + +```python +from dataclasses import dataclass +import re +from typing import Mapping + +_DIGEST = re.compile(r"sha256:[0-9a-f]{64}\Z") +_OBJECT_ID = re.compile(r"(?:[0-9a-f]{40}|[0-9a-f]{64})\Z") +_REMOTE = re.compile(r"[A-Za-z0-9._-]{1,64}\Z") +_APPLY_FIELDS = frozenset({ + "schema", "exact_path", "repository_identity", "commit", "tree", + "remote_name", "candidate_hash", "ledger_epoch", + "authorization_token", "batch_limit", +}) + + +class RetentionContractError(ValueError): + def __init__(self, code: str) -> None: + self.code = code + super().__init__(code) + + +@dataclass(frozen=True, slots=True) +class SourceCleanupApply: + exact_path: str + repository_identity: str + commit: str + tree: str + remote_name: str + candidate_hash: str + ledger_epoch: int + authorization_token: str + batch_limit: int + + @classmethod + def from_mapping(cls, value: object) -> "SourceCleanupApply": + if type(value) is not dict or set(value) != _APPLY_FIELDS: + raise RetentionContractError("PATH_AUTHORIZATION_REQUIRED") + if value.get("schema") != "2718lab.storage.source-cleanup-apply.v1": + raise RetentionContractError("PATH_AUTHORIZATION_REQUIRED") + path = value.get("exact_path") + token = value.get("authorization_token") + epoch = value.get("ledger_epoch") + limit = value.get("batch_limit") + if ( + type(path) is not str or not path or not _is_absolute_literal(path) + or type(token) is not str or not token.startswith("pathauth-v1.") + or len(token) > 4096 + or type(epoch) is not int or isinstance(epoch, bool) or epoch < 1 + or type(limit) is not int or isinstance(limit, bool) or limit != 1 + or type(value.get("repository_identity")) is not str + or _DIGEST.fullmatch(value["repository_identity"]) is None + or type(value.get("candidate_hash")) is not str + or _DIGEST.fullmatch(value["candidate_hash"]) is None + or type(value.get("commit")) is not str + or _OBJECT_ID.fullmatch(value["commit"]) is None + or type(value.get("tree")) is not str + or _OBJECT_ID.fullmatch(value["tree"]) is None + or type(value.get("remote_name")) is not str + or _REMOTE.fullmatch(value["remote_name"]) is None + ): + raise RetentionContractError("PATH_AUTHORIZATION_REQUIRED") + return cls(path, value["repository_identity"], value["commit"], value["tree"], value["remote_name"], value["candidate_hash"], epoch, token, limit) + + def to_wire(self) -> dict[str, object]: + return { + "schema": "2718lab.storage.source-cleanup-apply.v1", + "exact_path": self.exact_path, + "repository_identity": self.repository_identity, + "commit": self.commit, + "tree": self.tree, + "remote_name": self.remote_name, + "candidate_hash": self.candidate_hash, + "ledger_epoch": self.ledger_epoch, + "authorization_token": self.authorization_token, + "batch_limit": self.batch_limit, + } + + +def _is_absolute_literal(value: str) -> bool: + return ( + len(value) >= 4 + and value[1:3] == ":\\" + and value[0].isalpha() + and "/" not in value + and "\\.\\" not in value + and "\\..\\" not in value + and not value.endswith(("\\.", "\\..")) + ) +``` + +Add concrete `from_mapping` and `to_wire` methods for +`SourceCleanupPreview(exact_path, repository_identity, commit, tree, +remote_name)`, `SessionCasPreview()` and +`SessionCasApply(candidate_hash, ledger_epoch, batch_limit)`. Their exact +field sets are respectively `{schema, exact_path, repository_identity, +commit, tree, remote_name}`, `{schema}`, and `{schema, candidate_hash, +ledger_epoch, batch_limit}`; the schemas are the four constants in Task 7. +Reuse `_is_absolute_literal`, `_DIGEST`, `_OBJECT_ID`, and `_REMOTE` above, +require `ledger_epoch >= 1` and `1 <= batch_limit <= 16`, and return a dict +containing exactly those fields from each `to_wire`. No parser accepts a +wildcard, relative path, free-form Git URL, CAS path, or delete flag. + +- [ ] **Step 2: Define Host types and exact stable codes.** + +```rust +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +pub(crate) struct PathAuthorization { + pub(crate) schema: String, + pub(crate) authorization_id: String, + pub(crate) exact_path: PathBuf, + pub(crate) path_identity: String, + pub(crate) repository_identity: String, + pub(crate) commit: String, + pub(crate) tree: String, + pub(crate) authorizer: String, + pub(crate) issued_at: u64, + pub(crate) expires_at: u64, + pub(crate) nonce: String, + pub(crate) authorization_hash: String, +} + +pub(crate) trait PathAuthorizationVerifier: Send + Sync { + fn verify(&self, opaque: &str, now: u64) -> Result; +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq, thiserror::Error)] +pub(crate) enum RetentionError { + #[error("GitHub reachability unavailable")] + GithubReachabilityUnavailable, + #[error("commit is not reachable from a configured GitHub head")] + GithubCommitNotReachable, + #[error("exact path authorization is required")] + PathAuthorizationRequired, + #[error("candidate is stale")] + CandidateStale, + #[error("active owner protects the target")] + ProtectedActive, + #[error("dirty source protects the target")] + ProtectedDirty, + #[error("source target is protected")] + ProtectedSource, + #[error("session target is protected")] + ProtectedSession, + #[error("CAS evidence does not match")] + CasMismatch, + #[error("CAS reference is active")] + CasReferenceActive, + #[error("retention apply is incomplete")] + ApplyIncomplete, + #[error("retention postcheck failed")] + PostcheckFailed, +} + +impl RetentionError { + pub(crate) fn code(self) -> &'static str { + match self { + Self::GithubReachabilityUnavailable => "GITHUB_REACHABILITY_UNAVAILABLE", + Self::GithubCommitNotReachable => "GITHUB_COMMIT_NOT_REACHABLE", + Self::PathAuthorizationRequired => "PATH_AUTHORIZATION_REQUIRED", + Self::CandidateStale => "STORAGE_CANDIDATE_STALE", + Self::ProtectedActive => "STORAGE_PROTECTED_ACTIVE", + Self::ProtectedDirty => "STORAGE_PROTECTED_DIRTY", + Self::ProtectedSource => "STORAGE_PROTECTED_SOURCE", + Self::ProtectedSession => "STORAGE_PROTECTED_SESSION", + Self::CasMismatch => "STORAGE_CAS_MISMATCH", + Self::CasReferenceActive => "STORAGE_CAS_REFERENCE_ACTIVE", + Self::ApplyIncomplete => "STORAGE_APPLY_INCOMPLETE", + Self::PostcheckFailed => "STORAGE_POSTCHECK_FAILED", + } + } +} +``` + +- [ ] **Step 3: Turn the contract probes green and compile the new Python module.** + +```powershell +Push-Location 'G:\2718lab\_codex\.codex-task-temp\devkit-1.1.2-recovery\mcp-tools'; $env:PYTEST_DISABLE_PLUGIN_AUTOLOAD='1'; python -m pytest tests/test_storage_retention.py::test_source_cleanup_apply_requires_exact_path_and_host_authorization -q -o cache_dir=G:\2718lab\_codex\.codex-task-temp\storage-113-retention-pytest; python -m py_compile devkit_runtime/storage_retention.py; Pop-Location +Push-Location 'G:\2718lab\_codex\.codex-task-temp\codex-host-mcp-fix-recovery2\codex-rs'; $env:CARGO_TARGET_DIR='G:\2718lab\_codex\.codex-task-temp\storage-113-retention-rust-target'; cargo test -p codex-core source_retention_contract --locked -j1; Pop-Location +``` + +Expected: the Python test passes, `py_compile` is silent, and the focused Host +contract tests pass. + +- [ ] **Step 4: Commit the contract implementation separately.** + +```powershell +Push-Location 'G:\2718lab\_codex\.codex-task-temp\devkit-1.1.2-recovery'; git add mcp-tools/devkit_runtime/storage_retention.py mcp-tools/tests/test_storage_retention.py; git commit -m 'feat: validate exact retention requests'; Pop-Location +Push-Location 'G:\2718lab\_codex\.codex-task-temp\codex-host-mcp-fix-recovery2'; git add codex-rs/core/src/fast_lane_host_dispatch/source_session_retention.rs codex-rs/core/src/fast_lane_host_dispatch/source_session_retention_tests.rs codex-rs/core/src/fast_lane_host_dispatch/mod.rs; git commit -m 'feat: define host retention authority'; Pop-Location +``` + +### Task 3: Repair durable session activity indexing and day-directory creation + +**Files:** +- Modify: `codex-rs/rollout/src/session_index.rs:20-294` +- Modify: `codex-rs/rollout/src/session_index_tests.rs:1-430` +- Modify: `codex-rs/rollout/src/recorder.rs:829-1024,1624-1867` +- Modify: `codex-rs/rollout/src/recorder_tests.rs:90-190` + +- [ ] **Step 1: Add a RED test that a durable write refreshes the index and creates the missing day directory.** + +```rust +#[tokio::test] +async fn persist_creates_day_directory_and_refreshes_session_index_activity() -> std::io::Result<()> { + let home = tempfile::tempdir()?; + let config = test_config(home.path()); + let thread_id = ThreadId::new(); + append_session_index_entry(home.path(), &SessionIndexEntry { + id: thread_id, + thread_name: "saved-thread".into(), + updated_at: "2024-01-01T00:00:00Z".into(), + }).await?; + let recorder = create_test_recorder(&config, thread_id).await?; + let parent = recorder.rollout_path().parent().unwrap().to_path_buf(); + assert!(!parent.exists()); + recorder.record_canonical_items(&[test_event("saved")]).await?; + recorder.persist().await?; + assert!(parent.is_dir()); + let entry = latest_session_index_entry(home.path(), thread_id).await?.unwrap(); + assert_eq!(entry.thread_name, "saved-thread"); + assert_ne!(entry.updated_at, "2024-01-01T00:00:00Z"); + Ok(()) +} +``` + +- [ ] **Step 2: Add a lock-safe activity touch that preserves the latest explicit name.** + +```rust +pub async fn touch_thread_updated_at( + codex_home: &Path, + thread_id: ThreadId, +) -> std::io::Result { + let _guard = SESSION_INDEX_LOCK + .lock() + .map_err(|err| std::io::Error::other(err.to_string()))?; + let path = session_index_path(codex_home); + let thread_name = if path.exists() { + scan_index_from_end_by_id(&path, &thread_id)? + .map(|entry| entry.thread_name) + .unwrap_or_default() + } else { + String::new() + }; + let entry = SessionIndexEntry { + id: thread_id, + thread_name, + updated_at: now_rfc3339()?, + }; + append_session_index_entry_locked(codex_home, &entry)?; + Ok(entry) +} +``` + +Refactor `append_session_index_entry` to acquire `SESSION_INDEX_LOCK` and call +the same synchronous `append_session_index_entry_locked`. Make +`find_thread_name_by_id` ignore an empty name. This keeps old rows readable, +prevents a touch/rename race, and allows unnamed persisted sessions to receive +an activity timestamp without inventing a title. + +- [ ] **Step 3: Carry the Host-owned identity into the writer and touch only after flush succeeds.** + +```rust +struct RolloutWriterState { + writer: Option, + deferred_creation: bool, + pending_items: Vec, + meta: Option, + cwd: PathBuf, + codex_home: PathBuf, + thread_id: ThreadId, + rollout_path: PathBuf, + ordinal_state: RolloutOrdinalState, + last_logged_error: Option, +} + +async fn write_pending_once(&mut self) -> std::io::Result<()> { + self.ensure_writer_open().await?; + self.write_session_meta_if_needed().await?; + self.write_pending_items_once().await?; + if let Some(writer) = self.writer.as_mut() { + writer.file.flush().await?; + } + super::session_index::touch_thread_updated_at(&self.codex_home, self.thread_id).await?; + Ok(()) +} +``` + +For `Create`, set `thread_id=conversation_id`; for `Resume`, derive the thread +ID with the existing `parse_timestamp_uuid_from_filename` result and fail the +recorder open if it does not match the rollout metadata. Keep +`precompute_new_rollout_path` side-effect free. The existing +`open_log_file -> fs::create_dir_all(parent)` remains the only first-persist +day-directory creation seam, so constructing an unused conversation still +creates neither a rollout nor a date directory. + +- [ ] **Step 4: Run the two focused rollout probes and compile the crate.** + +```powershell +Push-Location 'G:\2718lab\_codex\.codex-task-temp\codex-host-mcp-fix-recovery2\codex-rs'; $env:CARGO_TARGET_DIR='G:\2718lab\_codex\.codex-task-temp\storage-113-retention-rust-target'; cargo test -p codex-rollout persist_creates_day_directory_and_refreshes_session_index_activity --locked -j1; cargo test -p codex-rollout touch_thread_updated_at_preserves_latest_name --locked -j1; cargo check -p codex-rollout --locked -j1; Pop-Location +``` + +Expected: both focused tests pass and the crate check finishes with no new +warning. The first test must read the saved rollout before asserting the index +timestamp, proving the body was durable before the activity row. + +- [ ] **Step 5: Commit the session durability repair.** + +```powershell +Push-Location 'G:\2718lab\_codex\.codex-task-temp\codex-host-mcp-fix-recovery2'; git add codex-rs/rollout/src/session_index.rs codex-rs/rollout/src/session_index_tests.rs codex-rs/rollout/src/recorder.rs codex-rs/rollout/src/recorder_tests.rs; git commit -m 'fix: refresh session activity after durable writes'; Pop-Location +``` + +### Task 4: Prove GitHub reachability and build source cleanup previews + +**Files:** +- Modify: `codex-rs/core/src/fast_lane_host_dispatch/source_session_retention.rs` +- Modify: `codex-rs/core/src/fast_lane_host_dispatch/storage_ledger.rs` +- Test: `codex-rs/core/src/fast_lane_host_dispatch/source_session_retention_tests.rs` + +- [ ] **Step 1: Add deterministic RED fixtures for remote mismatch and exact ancestor reachability.** + +```rust +#[test] +fn preview_requires_matching_github_identity_and_remote_ancestor() { + let reachable = source_fixture(SourceMutation::None); + let preview = reachable.authority.preview(reachable.request).unwrap(); + assert_eq!(preview.repository_identity, reachable.repository_identity); + let mismatch = source_fixture(SourceMutation::RemoteMismatch); + assert_eq!(mismatch.authority.preview(mismatch.request).unwrap_err().code(), "GITHUB_REACHABILITY_UNAVAILABLE"); +} +``` + +- [ ] **Step 2: Implement the injected GitHub reachability verifier.** + +```rust +pub(crate) trait GithubReachability: Send + Sync { + fn prove( + &self, + repository_root: &Path, + remote_name: &str, + expected_identity: &str, + commit: &str, + ) -> Result; +} + +fn prove_with_bare_verifier(&self, request: &GithubRequest) -> Result { + let remote_url = git_stdout(&request.repository_root, ["config", "--get", &format!("remote.{}.url", request.remote_name)]) + .map_err(|_| RetentionError::GithubReachabilityUnavailable)?; + let identity = canonical_github_identity(remote_url.trim()) + .ok_or(RetentionError::GithubReachabilityUnavailable)?; + if identity != request.expected_identity { + return Err(RetentionError::GithubReachabilityUnavailable); + } + let verifier = self.admitted_verifier_root(&request.request_hash)?; + run_git(&verifier, ["init", "--bare", "--quiet"]) + .map_err(|_| RetentionError::GithubReachabilityUnavailable)?; + run_git(&verifier, ["fetch", "--quiet", "--no-tags", "--filter=blob:none", remote_url.trim(), "+refs/heads/*:refs/remotes/verified/*"]) + .map_err(|_| RetentionError::GithubReachabilityUnavailable)?; + let heads = git_lines(&verifier, ["for-each-ref", "--format=%(refname)", "refs/remotes/verified"]) + .map_err(|_| RetentionError::GithubReachabilityUnavailable)?; + if !heads.iter().any(|head| git_success(&verifier, ["merge-base", "--is-ancestor", request.commit.as_str(), head.as_str()])) { + return Err(RetentionError::GithubCommitNotReachable); + } + Ok(GithubReachabilityReceipt::new(identity, request.commit.clone(), heads)) +} +``` + +The verifier root is obtained from Plan 1 admission with its own byte/file +budget and released through Plan 2; it is never created below the source +worktree. `canonical_github_identity` accepts only exact GitHub HTTPS or SSH +remote forms and hashes the lowercase `github.com/owner/repository` identity. +Do not pass credentials or remote URL into any public receipt. + +- [ ] **Step 3: Build the canonical source candidate only after every read-only gate passes.** + +```rust +pub(crate) fn preview(&self, request: SourcePreviewRequest) -> Result { + let source = self.resolve_registered_worktree(&request.exact_path)?; + self.reject_repository_current_or_protected(&source)?; + self.ledger.require_no_path_owner(&source.path_identity)?; + require_empty_porcelain(&source.path)?; + require_exact_git_object(&source.path, "HEAD", &request.commit)?; + require_exact_git_object(&source.path, "HEAD^{tree}", &request.tree)?; + let remote = self.github.prove(&source.repository_root, &request.remote_name, &request.repository_identity, &request.commit)?; + let manifest = SourceCandidateManifest::new(self.ledger.epoch(), source, request, remote); + Ok(SourceCleanupPreview::from_manifest(manifest, canonical_hash(&manifest)?)) +} +``` + +`StorageLedger::require_no_path_owner` returns `STORAGE_PROTECTED_ACTIVE` for +active/reserved/recovery records and `STORAGE_PROTECTED_SOURCE` for +quarantined or incomplete-receipt records. The manifest contains the exact +commit/tree, registered-worktree identity, clean-status hash, remote heads +hash, ledger epoch, policy hash, and path identity. + +- [ ] **Step 4: Run the focused preview tests and the core compile gate.** + +```powershell +Push-Location 'G:\2718lab\_codex\.codex-task-temp\codex-host-mcp-fix-recovery2\codex-rs'; $env:CARGO_TARGET_DIR='G:\2718lab\_codex\.codex-task-temp\storage-113-retention-rust-target'; cargo test -p codex-core preview_requires_matching_github_identity_and_remote_ancestor --locked -j1; cargo test -p codex-core source_cleanup_protects_dirty_unpushed_active_and_current_worktrees --locked -j1; cargo check -p codex-core --lib --locked -j1; Pop-Location +``` + +Expected: both tests pass and core compiles with no new warning. Tests use a +local injected `GithubReachability` fixture; they make no network call. + +- [ ] **Step 5: Commit reachability and preview.** + +```powershell +Push-Location 'G:\2718lab\_codex\.codex-task-temp\codex-host-mcp-fix-recovery2'; git add codex-rs/core/src/fast_lane_host_dispatch/source_session_retention.rs codex-rs/core/src/fast_lane_host_dispatch/source_session_retention_tests.rs codex-rs/core/src/fast_lane_host_dispatch/storage_ledger.rs; git commit -m 'feat: prove source cleanup reachability'; Pop-Location +``` + +### Task 5: Require authorization at apply and remove automatic worktree deletion + +**Files:** +- Modify: `codex-rs/core/src/fast_lane_host_dispatch/source_session_retention.rs` +- Modify: `codex-rs/core/src/fast_lane_host_dispatch/worktree.rs:65-119` +- Modify: `codex-rs/core/src/fast_lane_host_dispatch/codex_adapter.rs:802-979` +- Modify: `codex-rs/core/src/fast_lane_host_dispatch/storage_ledger.rs` +- Test: `codex-rs/core/src/fast_lane_host_dispatch/source_session_retention_tests.rs` + +- [ ] **Step 1: Add a RED test that remote reachability without exact authorization still preserves the worktree.** + +```rust +#[test] +fn reachable_clean_source_without_exact_authorization_is_not_removed() { + let fixture = source_fixture(SourceMutation::None); + let preview = fixture.authority.preview(fixture.request.clone()).unwrap(); + let request = SourceApplyRequest::without_authorization(&preview); + assert_eq!(fixture.authority.apply(request).unwrap_err().code(), "PATH_AUTHORIZATION_REQUIRED"); + assert!(fixture.worktree.exists()); +} +``` + +- [ ] **Step 2: Verify authorization, re-preview under the writer fence, journal, remove, and postcheck.** + +```rust +pub(crate) fn apply(&mut self, request: SourceApplyRequest) -> Result { + let authorization = self.authorization.verify(&request.authorization_token, self.clock.now()?)?; + require_authorization_match(&authorization, &request)?; + let fence = self.ledger.writer_fence()?; + let preview = self.preview(request.preview_request())?; + if preview.candidate_hash != request.candidate_hash || preview.ledger_epoch != request.ledger_epoch { + return Err(RetentionError::CandidateStale); + } + require_authorization_match_preview(&authorization, &preview)?; + self.ledger.write_source_cleanup_started(&preview, &authorization, &fence)?; + self.worktrees.remove_authorized_worktree(&preview.exact_path, &preview.commit)?; + if preview.exact_path.exists() || self.worktrees.is_registered(&preview.exact_path)? { + return Err(RetentionError::PostcheckFailed); + } + self.ledger.commit_source_cleanup_receipt(&preview, &authorization, &fence) +} +``` + +`require_authorization_match` verifies exact canonical path equality, path +identity, repository identity, commit, tree, `issued_at <= now < expires_at`, +maximum 15-minute lifetime, nonce uniqueness, and authorization hash. The +receipt schema is `2718lab.storage.source-cleanup-receipt.v1` and includes +only `code`, `path_identity`, `repository_identity`, `commit`, `tree`, +`candidate_hash`, `ledger_epoch`, `authorization_id`, `remote_receipt_hash`, +`removed`, and `receipt_hash`. + +- [ ] **Step 3: Replace the production auto-remove seam with candidate registration.** + +```rust +// codex_adapter.rs, after successful integration and writer shutdown +for worktree in &successful_worktrees.created { + self.retention.register_integrated_source_candidate( + worktree, + &integration_receipt, + self.registry.active_lease_set_hash(), + )?; +} +// Do not call remove_integrated_batch here. Source cleanup is a later, +// explicitly authorized operation. +``` + +Rename `GitWorktreeBroker::remove_integrated_batch` to +`remove_authorized_worktree(path, expected_head)` and make it `pub(crate)` only +to `SourceSessionRetention`. It rechecks direct-child scope, registration, +clean porcelain, and exact HEAD immediately before `git worktree remove`; no +`--force` is allowed. Failed integration, dirty writers, and missing receipts +remain quarantined. + +- [ ] **Step 4: Run the authorization and production-seam tests, then compile core.** + +```powershell +Push-Location 'G:\2718lab\_codex\.codex-task-temp\codex-host-mcp-fix-recovery2\codex-rs'; $env:CARGO_TARGET_DIR='G:\2718lab\_codex\.codex-task-temp\storage-113-retention-rust-target'; cargo test -p codex-core reachable_clean_source_without_exact_authorization_is_not_removed --locked -j1; cargo test -p codex-core integrated_worktree_is_retained_until_authorized_cleanup --locked -j1; cargo check -p codex-core --lib --locked -j1; Pop-Location +``` + +Expected: both tests pass, the integrated worktree still exists before apply, +the authorized one-item fixture is removed only after all rechecks, and core +compiles with no new warning. + +- [ ] **Step 5: Commit source apply and production wiring.** + +```powershell +Push-Location 'G:\2718lab\_codex\.codex-task-temp\codex-host-mcp-fix-recovery2'; git add codex-rs/core/src/fast_lane_host_dispatch/source_session_retention.rs codex-rs/core/src/fast_lane_host_dispatch/source_session_retention_tests.rs codex-rs/core/src/fast_lane_host_dispatch/worktree.rs codex-rs/core/src/fast_lane_host_dispatch/codex_adapter.rs codex-rs/core/src/fast_lane_host_dispatch/storage_ledger.rs; git commit -m 'feat: require authorization for source cleanup'; Pop-Location +``` + +### Task 6: Implement archived no-live-reference CAS dedupe + +**Files:** +- Create: `codex-rs/rollout/src/archived_session_cas.rs` +- Create: `codex-rs/rollout/src/archived_session_cas_tests.rs` +- Modify: `codex-rs/rollout/src/lib.rs:60-140` +- Modify: `codex-rs/rollout/src/rollout_reference_index.rs:19-90` +- Modify: `codex-rs/core/src/fast_lane_host_dispatch/source_session_retention.rs` +- Test: `codex-rs/core/src/fast_lane_host_dispatch/source_session_retention_tests.rs` + +- [ ] **Step 1: Define the exact archived CAS index and protection snapshot.** + +```rust +#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] +pub struct ArchivedCasObject { + pub object_id: String, + pub content_hash: String, + pub byte_length: u64, + pub relative_path: PathBuf, + pub immutable: bool, + pub archived_thread_ids: Vec, +} + +#[derive(Clone, Debug, Default, PartialEq, Eq)] +pub struct SessionProtectionSnapshot { + pub active_thread_ids: HashSet, + pub current_rollout_ids: HashSet, + pub checkpoint_hashes: HashSet, + pub active_lease_hashes: HashSet, +} + +pub struct ArchivedSessionCas { + root: PathBuf, + index_path: PathBuf, + objects: Vec, +} +``` + +`ArchivedSessionCas::open` canonicalizes +`archived_sessions/cas/objects`, rejects any reparse point, validates exact +lowercase hash fan-out paths, rejects duplicate object IDs, and reads the +index with the same atomic replacement pattern used by Plan 2. It never scans +`sessions/**` or ordinary files in `archived_sessions/**`. + +- [ ] **Step 2: Implement preview and transactional apply.** + +```rust +pub fn preview(&self, protection: &SessionProtectionSnapshot, references: &RolloutReferenceIndex) -> Result { + let mut groups = group_verified_objects_by_hash_and_length(&self.objects, &self.root)?; + groups.retain(|group| group.objects.len() > 1); + for group in &groups { + require_archived_inactive_unreferenced(group, protection, references)?; + } + groups.sort_by(|left, right| left.content_hash.cmp(&right.content_hash)); + CasDedupePreview::new(groups) +} + +pub fn apply(&mut self, request: &CasDedupeApply, protection: &SessionProtectionSnapshot, references: &RolloutReferenceIndex) -> Result { + let preview = self.preview(protection, references)?; + if request.candidate_hash != preview.candidate_hash { + return Err(ArchivedCasError::Mismatch); + } + let selected = preview.groups.into_iter().take(request.batch_limit).collect::>(); + let next = repoint_archived_references_to_lexical_canonical(&self.objects, &selected)?; + self.persist_index_atomically(&next)?; + for redundant in redundant_blob_paths(&selected) { + remove_verified_blob_only(&self.root, &redundant)?; + } + verify_canonical_blobs_and_rollout_bodies_unchanged(&self.root, &selected)?; + self.objects = next; + CasDedupeReceipt::from_applied(selected) +} +``` + +`require_archived_inactive_unreferenced` returns +`STORAGE_CAS_REFERENCE_ACTIVE` if any thread is active/current, any rollout +lineage references the object, or any checkpoint/lease reference exists. Hash, +length, index, lock, path, or immutable-state mismatch returns +`STORAGE_CAS_MISMATCH` and keeps every object. Apply does not create a CAS +object from rollout JSONL and does not delete a final canonical object. + +- [ ] **Step 3: Bind CAS apply to the Plan 2 fence and receipt journal.** + +```rust +let fence = self.ledger.writer_fence()?; +let references = RolloutReferenceIndex::scan(&self.codex_home) + .await + .map_err(|_| RetentionError::CasReferenceActive)?; +let protection = self.session_owners.snapshot()?; +let receipt = self.archived_cas.apply(&request, &protection, &references) + .map_err(RetentionError::from)?; +self.ledger.commit_session_cas_receipt(&receipt, &fence)?; +``` + +The Host response strips all filesystem paths and thread IDs. It exposes +`candidate_hash`, `ledger_epoch`, `canonical_object_count`, +`redundant_object_count`, `reclaimed_bytes`, `code`, and `receipt_hash`. + +- [ ] **Step 4: Run the CAS protection/commit tests and compile both affected crates.** + +```powershell +Push-Location 'G:\2718lab\_codex\.codex-task-temp\codex-host-mcp-fix-recovery2\codex-rs'; $env:CARGO_TARGET_DIR='G:\2718lab\_codex\.codex-task-temp\storage-113-retention-rust-target'; cargo test -p codex-rollout cas_dedupe_never_selects_current_active_or_rollout_body --locked -j1; cargo test -p codex-rollout archived_equal_hash_cas_dedupe_commits_reference_transaction --locked -j1; cargo check -p codex-rollout --locked -j1; cargo check -p codex-core --lib --locked -j1; Pop-Location +``` + +Expected: focused tests pass and both compile gates finish with no new warning. +The post-test fixture must prove both rollout bodies still exist byte-for-byte. + +- [ ] **Step 5: Commit archived CAS dedupe.** + +```powershell +Push-Location 'G:\2718lab\_codex\.codex-task-temp\codex-host-mcp-fix-recovery2'; git add codex-rs/rollout/src/archived_session_cas.rs codex-rs/rollout/src/archived_session_cas_tests.rs codex-rs/rollout/src/lib.rs codex-rs/rollout/src/rollout_reference_index.rs codex-rs/core/src/fast_lane_host_dispatch/source_session_retention.rs codex-rs/core/src/fast_lane_host_dispatch/source_session_retention_tests.rs; git commit -m 'feat: deduplicate protected archived session cas'; Pop-Location +``` + +### Task 7: Wire authenticated retention tools and final compile gates + +**Files:** +- Modify: `mcp-tools/devkit_runtime/host_bridge.py:218-264,2298-2677` +- Modify: `mcp-tools/devkit_runtime/host_session.py:159-335,636-735` +- Modify: `mcp-tools/server.py:179-289,1008-1314` +- Modify: `mcp-tools/devkit_runtime/tool_metadata.py:5-23` +- Modify: `mcp-tools/tests/test_storage_retention.py` +- Modify: `mcp-tools/tests/test_mcp_contract.py:240-360` +- Modify: `codex-rs/rmcp-client/src/inherited_host_bridge_protocol.rs:35-240` +- Modify: `codex-rs/rmcp-client/src/inherited_host_bridge_protocol/envelope.rs:25-220` +- Modify: `codex-rs/rmcp-client/src/inherited_host_bridge_protocol/session.rs:412-570` + +- [ ] **Step 1: Add RED tests for exact wire fields and destructive annotations.** + +```python +def test_retention_tools_have_exact_safety_annotations(): + from devkit_runtime.tool_metadata import TOOL_ANNOTATIONS + + assert TOOL_ANNOTATIONS["source_cleanup_preview"] == (True, False, True, True) + assert TOOL_ANNOTATIONS["source_cleanup_apply"] == (False, True, False, True) + assert TOOL_ANNOTATIONS["session_cas_preview"] == (True, False, True, False) + assert TOOL_ANNOTATIONS["session_cas_apply"] == (False, True, False, False) +``` + +- [ ] **Step 2: Add exact authenticated bridge schemas on Python and Rust sides.** + +```rust +pub(crate) const SOURCE_CLEANUP_PREVIEW_SCHEMA: &str = + "2718lab.storage.source-cleanup-preview.v1"; +pub(crate) const SOURCE_CLEANUP_APPLY_SCHEMA: &str = + "2718lab.storage.source-cleanup-apply.v1"; +pub(crate) const SESSION_CAS_PREVIEW_SCHEMA: &str = + "2718lab.storage.session-cas-preview.v1"; +pub(crate) const SESSION_CAS_APPLY_SCHEMA: &str = + "2718lab.storage.session-cas-apply.v1"; +``` + +Each envelope uses the existing session ID, monotonic correlation ID, request +hash, size bound, replay cache, and single receiver. Require exact fields from +Task 2; reject unknown fields, duplicate correlation IDs, expired sessions, +and responses whose candidate/ledger/authorization binding differs from the +request. Absolute paths appear only inside the encrypted/authenticated private +request and never in public results. + +- [ ] **Step 3: Register the four tools and project path-free receipts.** + +```python +@mcp.tool(annotations=_tool_annotations("source_cleanup_apply")) +def source_cleanup_apply(request: dict[str, object]) -> dict[str, object]: + from devkit_runtime.storage_retention import RetentionContractError, SourceCleanupApply + try: + exact = SourceCleanupApply.from_mapping(request) + except RetentionContractError as error: + return _failure(error.code) + return _project_retention_receipt(_host_session().source_cleanup_apply(exact.to_wire())) + + +def _project_retention_receipt(value: object) -> dict[str, object]: + allowed = { + "code", "path_identity", "repository_identity", "commit", "tree", + "candidate_hash", "ledger_epoch", "authorization_id", + "remote_receipt_hash", "removed", "canonical_object_count", + "redundant_object_count", "reclaimed_bytes", "receipt_hash", + } + if type(value) is not dict or not set(value) <= allowed: + return _failure("INTERNAL_ERROR") + return {key: value[key] for key in sorted(value)} +``` + +Register `source_cleanup_preview`, `session_cas_preview`, and +`session_cas_apply` by parsing their corresponding Task 2 type, passing +`parsed.to_wire()` to the same-named `HostSession` method, and returning +`_project_retention_receipt(response)`. Each catches +`RetentionContractError` and returns `_failure(error.code)`. No tool accepts a +recursive flag, wildcard, root expansion, force flag, or ordinary session ID. + +- [ ] **Step 4: Run focused contract tests and compile-first gates.** + +```powershell +Push-Location 'G:\2718lab\_codex\.codex-task-temp\devkit-1.1.2-recovery\mcp-tools'; $env:PYTEST_DISABLE_PLUGIN_AUTOLOAD='1'; python -m pytest tests/test_storage_retention.py tests/test_mcp_contract.py -q -o cache_dir=G:\2718lab\_codex\.codex-task-temp\storage-113-retention-pytest; python -m py_compile devkit_runtime/storage_retention.py devkit_runtime/host_bridge.py devkit_runtime/host_session.py server.py; Pop-Location +Push-Location 'G:\2718lab\_codex\.codex-task-temp\codex-host-mcp-fix-recovery2\codex-rs'; $env:CARGO_TARGET_DIR='G:\2718lab\_codex\.codex-task-temp\storage-113-retention-rust-target'; cargo test -p codex-rmcp-client source_cleanup_protocol --locked -j1; cargo test -p codex-rmcp-client session_cas_protocol --locked -j1; cargo check -p codex-rmcp-client -p codex-mcp --lib --locked -j1; cargo check -p codex-core --lib --locked -j1; Pop-Location +``` + +Expected: focused Python/Rust probes pass, Python compile is silent, and both +Rust checks finish with no new warning. Do not run the full workspace suite. + +- [ ] **Step 5: Commit the bridge and public tool surface separately.** + +```powershell +Push-Location 'G:\2718lab\_codex\.codex-task-temp\devkit-1.1.2-recovery'; git add mcp-tools/devkit_runtime/storage_retention.py mcp-tools/devkit_runtime/host_bridge.py mcp-tools/devkit_runtime/host_session.py mcp-tools/devkit_runtime/tool_metadata.py mcp-tools/server.py mcp-tools/tests/test_storage_retention.py mcp-tools/tests/test_mcp_contract.py; git commit -m 'feat: expose governed retention operations'; Pop-Location +Push-Location 'G:\2718lab\_codex\.codex-task-temp\codex-host-mcp-fix-recovery2'; git add codex-rs/rmcp-client/src/inherited_host_bridge_protocol.rs codex-rs/rmcp-client/src/inherited_host_bridge_protocol/envelope.rs codex-rs/rmcp-client/src/inherited_host_bridge_protocol/session.rs; git commit -m 'feat: authenticate retention bridge operations'; Pop-Location +``` + +## Plan 3 acceptance gate and 1.1.3 completion + +- [ ] Re-read the design sections “GitHub 可达性与精确路径授权”, “归档会话的 CAS 去重”, “稳定错误”, “迁移与回滚”, and “1.1.3 版本边界”; map every requirement and stable code to a task above. +- [ ] Run `git diff --check` in both worktrees and verify only the Plan 3 file map changed since the Plan 2 commits. +- [ ] Run the named DevKit `py_compile`, Host `cargo check -p codex-rollout --locked -j1`, `cargo check -p codex-rmcp-client -p codex-mcp --lib --locked -j1`, and `cargo check -p codex-core --lib --locked -j1` commands using the single retention target root. +- [ ] Record a source preview receipt, an unreachable/unpushed refusal, a missing-authorization refusal, an active-lease refusal, one explicitly authorized worktree removal receipt, a session-index activity receipt, a protected-current-session CAS refusal, and one archived duplicate-CAS receipt. Every apply receipt must bind candidate hash and ledger epoch. +- [ ] Verify source cleanup refuses the repository root, current worktree, protected branch, dirty tree, unpushed commit, mismatched GitHub identity, expired authorization, changed HEAD/tree, active/recovery/quarantined lease, and stale candidate without removing a path. +- [ ] Verify a saved rollout refreshes `session_index.jsonl` only after durable body flush, first persistence creates its missing YYYY/MM/DD directory, and an unused deferred session creates neither file nor directory. +- [ ] Verify CAS dedupe never scans or removes `sessions/**`, current/active rollouts, ordinary archived `.jsonl`/`.jsonl.zst` bodies, final canonical blobs, unknown files, or objects with active checkpoint/lease/rollout references. +- [ ] Run one controlled Host production receipt flow for each apply operation. A network, stat, lock, ledger, authorization, or postcheck failure is a protected refusal; it never expands the candidate set or invokes force deletion. + +Plan 1, Plan 2, and Plan 3 together complete 1.1.3. They do not authorize a +local deletion merely because a commit was pushed, do not delete ordinary +Codex conversations, and do not add remote session synchronization, +compression, full-disk scanning, cross-machine target sharing, or any Fast +Lane route/lease/context/capability fallback. diff --git a/docs/superpowers/plans/2026-08-29-storage-firewall-1.1.3.md b/docs/superpowers/plans/2026-08-29-storage-firewall-1.1.3.md new file mode 100644 index 0000000..5768119 --- /dev/null +++ b/docs/superpowers/plans/2026-08-29-storage-firewall-1.1.3.md @@ -0,0 +1,592 @@ +# Storage Firewall 1.1.3 Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Make every Cargo, Python, MCP-package, and Fast Lane write begin with one host-approved deterministic generated root and a fail-closed byte/file/free-space reservation. + +**Architecture:** DevKit emits a bounded, path-free `StorageIntent` whose hash is bound to the Fast Lane task, source plan, execution context, and project identity. The Codex Host validates the intent, computes the canonical target key, checks the approved root and disk policy, and returns an opaque admission receipt containing the assigned root; the worker never selects `CARGO_TARGET_DIR` or a temporary directory. Lease persistence, cleanup, GitHub source authorization, and session CAS are separate follow-up plans and are not hidden in this admission path. + +**Tech Stack:** Python 3.11 standard library (`dataclasses`, `hashlib`, `json`, `pathlib`), MCP FastMCP/Pydantic, Rust 2021, `serde`/`serde_json`, `sha2`, Tokio, platform filesystem-capacity APIs, and the existing authenticated inherited-handle bridge. + +--- + +## Scope and file map + +All line references are against commit `37029a9` in the DevKit worktree and +commit `552fe8035d` in the Codex Host worktree. A worker must re-read the +symbol at the listed line before editing because earlier tasks can shift line +numbers. + +DevKit files: + +- Create `mcp-tools/devkit_runtime/storage_intent.py`: immutable intent and + canonical target-descriptor validation; it contains no filesystem write and + no owner claim. +- Modify `mcp-tools/devkit_fastlane/scripts/authenticated_v5_planner.py:14-415` + and `mcp-tools/devkit_fastlane/scripts/authenticated_v5_projection.py:1-318`: + bind one intent to every compiler assignment and retain it through initial + and successor waves. +- Modify `mcp-tools/devkit_runtime/fastlane_host_intent.py:333-595`: + structurally parse the new intent and reject an intent whose binding does not + equal the assignment's task/context/plan hashes. +- Modify `mcp-tools/devkit_runtime/host_bridge.py:218-264,2298-2677` and + `mcp-tools/devkit_runtime/host_session.py:159-335,636-735`: add the private + `storage_admit` request/receipt exchange to the already authenticated bridge. +- Modify `mcp-tools/server.py:1008-1314` only at the authenticated Fast Lane + dispatch seam so the production path forwards the intent and exposes the + returned root as a private execution binding; no public path argument is + added. +- Create `mcp-tools/tests/test_storage_firewall.py` and modify + `mcp-tools/tests/test_fastlane_host_adapter.py:894-1014` for the smallest + projection-to-host regression. + +Codex Host files: + +- Create `codex-rs/core/src/fast_lane_host_dispatch/storage_firewall.rs`: the + canonical target key, approved-root fence, capacity provider, quota policy, + admission receipt, and stable errors. +- Modify `codex-rs/core/src/fast_lane_host_dispatch/mod.rs:1-49` to register the + module and keep its types `pub(crate)`. +- Modify `codex-rs/core/src/fast_lane_host_dispatch/contract.rs:1-842` only in + the assignment/skeleton projection to carry and hash `storage_intent`. +- Modify `codex-rs/rmcp-client/src/inherited_host_bridge_protocol.rs:1-420` + and `codex-rs/rmcp-client/src/inherited_host_bridge_protocol/envelope.rs:1-310` + to validate the exact wire payload and its size before it enters core. +- Modify `codex-rs/core/src/fast_lane_host_dispatch/registry.rs:567-1668` to + reserve the target-family key before writer preparation and + `coordinator.rs:393-580,1218-1260` to release the admission on terminal or + recovery paths. +- Modify `codex-rs/core/src/fast_lane_host_dispatch/codex_adapter.rs:1-1444` + at worker environment construction so the host-issued root is the sole + `CARGO_TARGET_DIR`/task-temp value. +- Create `codex-rs/core/src/fast_lane_host_dispatch/storage_firewall_tests.rs` + and register it from the new module with `#[path = ...]`. +- Modify `codex-rs/core/Cargo.toml:80-145` by adding + `[target.'cfg(target_os = "windows")'.dependencies] windows-sys = { + version = "0.52", features = ["Win32_Storage_FileSystem"] }`, matching the + existing CLI dependency version; then update `codex-rs/Cargo.lock` and + `codex-rs/MODULE.bazel.lock` in the same Host commit. + +The worker must not edit any file outside this map. The ledger, preview/apply, +source authorization, and session CAS changes belong to Plans 2 and 3. + +## Shared wire contract + +The following exact JSON shape is the only storage admission payload. The +`target_descriptor` is the sole input to `target_key`; request sizes and root +bindings are policy inputs and are not smuggled into the target-key identity. + +```json +{ + "schema": "2718lab.storage.intent.v1", + "task_id": "task-01", + "plan_binding": "sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", + "context_hash": "sha256:bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", + "storage_intent_hash": "sha256:cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc", + "requested_bytes": 104857600, + "requested_files": 4096, + "target_descriptor": { + "schema": "2718lab.storage.target.v1", + "artifact_kind": "cargo-target", + "repository_identity": "sha256:dddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd", + "workspace_manifest_hash": "sha256:eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee", + "cargo_lock_hash": "sha256:ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff", + "toolchain_digest": "sha256:1111111111111111111111111111111111111111111111111111111111111111", + "target_triple": "x86_64-pc-windows-msvc", + "profile": "dev", + "features_hash": "sha256:2222222222222222222222222222222222222222222222222222222222222222", + "build_env_class": "windows-msvc" + } +} +``` + +The canonical `target_key` is the SHA-256 of the UTF-8 canonical JSON of the +ten `target_descriptor` fields. The host returns a `StorageAdmissionReceipt` +with `storage_intent_hash`, `target_key`, `assigned_root_identity`, +`target_family_lease_id`, `reserved_bytes`, `reserved_files`, +`free_space_before`, `free_space_after_reserve`, and `free_space_floor`. + +## Implementation tasks + +### Task 1: Freeze the intent contract with RED tests + +**Files:** +- Create: `mcp-tools/tests/test_storage_firewall.py` +- Create: `codex-rs/core/src/fast_lane_host_dispatch/storage_firewall_tests.rs` +- Modify: `mcp-tools/tests/test_fastlane_host_adapter.py:894-1014` + +- [ ] **Step 1: Add the Python RED fixture.** + +```python +def test_storage_intent_rejects_absolute_path_and_unknown_descriptor_key(): + from devkit_runtime.storage_intent import StorageIntentError, parse_storage_intent + + value = { + "schema": "2718lab.storage.intent.v1", + "task_id": "task-01", + "plan_binding": "sha256:" + "a" * 64, + "context_hash": "sha256:" + "b" * 64, + "requested_bytes": 1, + "requested_files": 1, + "target_descriptor": { + "schema": "2718lab.storage.target.v1", + "artifact_kind": "cargo-target", + "repository_identity": "sha256:" + "c" * 64, + "workspace_manifest_hash": "sha256:" + "d" * 64, + "cargo_lock_hash": "sha256:" + "e" * 64, + "toolchain_digest": "sha256:" + "f" * 64, + "target_triple": "x86_64-pc-windows-msvc", + "profile": "dev", + "features_hash": "sha256:" + "1" * 64, + "build_env_class": "windows-msvc", + "path": "G:/unapproved" + } + } + try: + parse_storage_intent(value) + except StorageIntentError as error: + assert error.code == "STORAGE_TARGET_KEY_INVALID" + else: + raise AssertionError("invalid descriptor was accepted") +``` + +- [ ] **Step 2: Add the Rust RED assertions for policy failure.** + +```rust +#[test] +fn missing_policy_is_stable_and_does_not_create_a_root() { + let firewall = StorageFirewall::new( + PathBuf::from(r"G:\2718lab\_codex\.codex-task-temp"), + CapacitySnapshot { free_bytes: 8 * GIB, free_files: 1_000_000 }, + None, + ); + let error = firewall.admit(intent()).expect_err("missing policy must fail closed"); + assert_eq!(error.code(), "STORAGE_POLICY_MISSING"); + assert!(!Path::new(r"G:\2718lab\_codex\.codex-task-temp\targets").exists()); +} +``` + +- [ ] **Step 3: Run only the new RED probes.** + +Run from `G:\2718lab\_codex\.codex-task-temp\devkit-1.1.2-recovery\mcp-tools`: + +```powershell +$env:PYTEST_DISABLE_PLUGIN_AUTOLOAD='1'; python -m pytest tests/test_storage_firewall.py::test_storage_intent_rejects_absolute_path_and_unknown_descriptor_key -q -o cache_dir=G:\2718lab\_codex\.codex-task-temp\storage-113-firewall-pytest +``` + +Run from `G:\2718lab\_codex\.codex-task-temp\codex-host-mcp-fix-recovery2\codex-rs`: + +```powershell +$env:CARGO_TARGET_DIR='G:\2718lab\_codex\.codex-task-temp\storage-113-firewall-rust-target'; cargo test -p codex-core missing_policy_is_stable_and_does_not_create_a_root --locked -j1 +``` + +Expected: both commands fail because the new module/types do not exist; +neither command may create a production target root. + +- [ ] **Step 4: Commit the RED tests only.** + +```powershell +Push-Location 'G:\2718lab\_codex\.codex-task-temp\devkit-1.1.2-recovery'; git add mcp-tools/tests/test_storage_firewall.py mcp-tools/tests/test_fastlane_host_adapter.py; git commit -m 'test: define storage firewall admission boundary'; Pop-Location +Push-Location 'G:\2718lab\_codex\.codex-task-temp\codex-host-mcp-fix-recovery2'; git add codex-rs/core/src/fast_lane_host_dispatch/storage_firewall_tests.rs; git commit -m 'test: define storage firewall admission boundary'; Pop-Location +``` + +### Task 2: Implement canonical Python intents + +**Files:** +- Create: `mcp-tools/devkit_runtime/storage_intent.py` +- Modify: `mcp-tools/devkit_runtime/__init__.py:1-18` + +- [ ] **Step 1: Define the bounded immutable records and stable error.** + +```python +@dataclass(frozen=True, slots=True) +class StorageIntent: + task_id: str + plan_binding: str + context_hash: str + storage_intent_hash: str + requested_bytes: int + requested_files: int + target_descriptor: Mapping[str, str] + + +def parse_storage_intent(value: object) -> StorageIntent: + mapping = _exact_mapping(value, _INTENT_FIELDS) + descriptor = _exact_mapping(mapping["target_descriptor"], _TARGET_FIELDS) + task_id = _bounded_identifier(mapping["task_id"], "task_id") + plan_binding = _digest(mapping["plan_binding"], "plan_binding") + context_hash = _digest(mapping["context_hash"], "context_hash") + requested_bytes = _bounded_positive(mapping["requested_bytes"], "requested_bytes") + requested_files = _bounded_positive(mapping["requested_files"], "requested_files") + target = _canonical_target(descriptor) + expected = _sha256_json({"target_descriptor": target, "task_id": task_id, "plan_binding": plan_binding, "context_hash": context_hash, "requested_bytes": requested_bytes, "requested_files": requested_files}) + storage_intent_hash = _digest(mapping["storage_intent_hash"], "storage_intent_hash") + if expected != storage_intent_hash: + raise StorageIntentError("STORAGE_TARGET_KEY_INVALID") + return StorageIntent(task_id, plan_binding, context_hash, storage_intent_hash, requested_bytes, requested_files, target) +``` + +The implementation must reject absolute paths, reparse-point hints, unknown +keys, booleans used as integers, zero/overflow values, non-lowercase digests, +and artifact kinds outside `cargo-target`, `python-cache`, `mcp-package`, and +`fastlane-task`. It must use `json.dumps(value, sort_keys=True, separators=(",", ":"), ensure_ascii=True)` before hashing. + +- [ ] **Step 2: Export only the parser and record.** + +```python +from .storage_intent import StorageIntent, StorageIntentError, parse_storage_intent + +__all__ = ["StorageIntent", "StorageIntentError", "parse_storage_intent"] +``` + +- [ ] **Step 3: Turn the Python RED green and compile the changed modules.** + +```powershell +$env:PYTEST_DISABLE_PLUGIN_AUTOLOAD='1'; python -m pytest tests/test_storage_firewall.py::test_storage_intent_rejects_absolute_path_and_unknown_descriptor_key -q -o cache_dir=G:\2718lab\_codex\.codex-task-temp\storage-113-firewall-pytest +python -m py_compile devkit_runtime/storage_intent.py devkit_runtime/__init__.py +``` + +Expected: `1 passed`, then a successful compile with no output. + +- [ ] **Step 4: Commit the Python contract.** + +```powershell +git add mcp-tools/devkit_runtime/storage_intent.py mcp-tools/devkit_runtime/__init__.py +git commit -m "feat: add canonical storage intent" +``` + +### Task 3: Bind intents to every Fast Lane assignment + +**Files:** +- Modify: `mcp-tools/devkit_fastlane/scripts/authenticated_v5_planner.py:124-333` +- Modify: `mcp-tools/devkit_fastlane/scripts/authenticated_v5_projection.py:186-284` +- Modify: `mcp-tools/devkit_runtime/fastlane_host_intent.py:333-595` +- Test: `mcp-tools/tests/test_storage_firewall.py` + +- [ ] **Step 1: Add a RED assertion that initial and successor assignments carry the same exact intent binding.** + +```python +def test_every_fastlane_wave_carries_plan_context_bound_storage_intent(): + from devkit_fastlane.scripts.authenticated_v5_planner import compile_skeletons + + first, successor = compile_skeletons(SOURCE_UNITS, source_plan_hash=PLAN_HASH, context=CONTEXT) + assert first[0]["storage_intent"]["task_id"] == first[0]["task_id"] + assert successor[0]["storage_intent"]["plan_binding"] == PLAN_HASH + assert successor[0]["storage_intent"]["context_hash"] == CONTEXT["execution_context_hash"] +``` + +- [ ] **Step 2: Add `storage_intent` to the exact planner/projection field sets and construct it from normalized assignment data.** + +```python +intent = make_storage_intent( + task_id=unit["task_id"], + plan_binding=source_hash, + context_hash=context["execution_context_hash"], + artifact_kind="fastlane-task", + repository_identity=context["repository_identity"], + workspace_manifest_hash=context["workspace_manifest_hash"], + cargo_lock_hash=context["cargo_lock_hash"], + toolchain_digest=context["toolchain_digest"], + target_triple=context["target_triple"], + profile=context["profile"], + features_hash=context["features_hash"], + build_env_class=context["build_env_class"], + requested_bytes=unit["storage_budget"]["bytes"], + requested_files=unit["storage_budget"]["files"], +) +assignment["storage_intent"] = intent +``` + +The compiler may not synthesize a default budget. A source unit without the +two positive budget values fails with `STORAGE_POLICY_MISSING` during compile; +it never gets a guessed path or a guessed owner. The projection hash must +include the complete `storage_intent` object before `dispatch_binding_hash` is +computed. + +- [ ] **Step 3: Parse and compare the binding at the host-intent boundary.** + +```python +parsed = parse_storage_intent(candidate["storage_intent"]) +if parsed.task_id != candidate["task_id"]: + raise ValueError("STORAGE_TARGET_KEY_INVALID") +if parsed.plan_binding != source_plan_hash: + raise ValueError("STORAGE_TARGET_KEY_INVALID") +if parsed.context_hash != candidate["execution_context_hash"]: + raise ValueError("STORAGE_TARGET_KEY_INVALID") +``` + +- [ ] **Step 4: Run the focused RED/GREEN probe and Python compile gate.** + +```powershell +$env:PYTEST_DISABLE_PLUGIN_AUTOLOAD='1'; python -m pytest tests/test_storage_firewall.py::test_every_fastlane_wave_carries_plan_context_bound_storage_intent -q -o cache_dir=G:\2718lab\_codex\.codex-task-temp\storage-113-firewall-pytest +python -m py_compile devkit_fastlane/scripts/authenticated_v5_planner.py devkit_fastlane/scripts/authenticated_v5_projection.py devkit_runtime/fastlane_host_intent.py +``` + +Expected: `1 passed` and a successful compile. Do not run the broad Fast Lane +matrix in this plan. + +- [ ] **Step 5: Commit the binding change.** + +```powershell +git add mcp-tools/devkit_fastlane/scripts/authenticated_v5_planner.py mcp-tools/devkit_fastlane/scripts/authenticated_v5_projection.py mcp-tools/devkit_runtime/fastlane_host_intent.py mcp-tools/tests/test_storage_firewall.py +git commit -m "feat: bind storage intents to fast lane waves" +``` + +### Task 4: Add the authenticated bridge exchange + +**Files:** +- Modify: `mcp-tools/devkit_runtime/host_bridge.py:218-264,2298-2677` +- Modify: `mcp-tools/devkit_runtime/host_session.py:159-335,636-735` +- Modify: `codex-rs/rmcp-client/src/inherited_host_bridge_protocol.rs:35-240` +- Modify: `codex-rs/rmcp-client/src/inherited_host_bridge_protocol/envelope.rs:25-220` +- Test: `mcp-tools/tests/test_storage_firewall.py` +- Test: `codex-rs/rmcp-client/src/inherited_host_bridge_protocol_tests.rs` + +- [ ] **Step 1: Add the wire RED test for exact operation fields and replay identity.** + +```python +def test_storage_admission_request_is_session_bound_and_replay_stable(): + request = build_storage_admission_request(INTENT) + assert request["schema"] == "2718lab.storage.admission-request.v1" + assert set(request) == {"schema", "correlation_id", "storage_intent", "request_hash"} + assert request["request_hash"] == canonical_hash({key: request[key] for key in request if key != "request_hash"}) +``` + +- [ ] **Step 2: Implement exact validation on both sides.** + +```rust +pub(crate) const STORAGE_ADMISSION_REQUEST_SCHEMA: &str = + "2718lab.storage.admission-request.v1"; +pub(crate) const STORAGE_ADMISSION_RECEIPT_SCHEMA: &str = + "2718lab.storage.admission-receipt.v1"; + +fn validate_storage_admission(payload: &Map) -> Result { + require_exact_fields(payload, &["schema", "correlation_id", "storage_intent", "request_hash"])?; + if value_str(payload, "schema")? != STORAGE_ADMISSION_REQUEST_SCHEMA { + return Err(HostBridgeProtocolError::InvalidOperation); + } + let intent = validate_storage_intent(value_object(payload, "storage_intent")?)?; + let request_hash = strict_digest(value_str(payload, "request_hash")?)?; + if digest(&canonical_bytes(&without_field(payload, "request_hash")?)?) != request_hash { + return Err(HostBridgeProtocolError::InvalidOperation); + } + Ok(StorageAdmissionRequest::new(value_str(payload, "correlation_id")?.into(), intent, request_hash)) +} +``` + +The Python and Rust validators must reject different sessions, duplicate +correlation IDs, frame payloads above the existing `MAX_OPERATION_BYTES`, +unknown fields, and a receipt whose `storage_intent_hash` or `target_key` does +not match the request. No public MCP response may include the absolute root. + +- [ ] **Step 3: Add `HostSession.request_storage_admission(intent)` and its typed receipt.** + +```python +def request_storage_admission(self, intent: StorageIntent) -> StorageAdmissionReceipt | str: + request = build_storage_admission_request(intent) + response = self._bridge.request_storage_admission(request) + if not isinstance(response, StorageAdmissionReceipt): + return "STORAGE_STAT_UNAVAILABLE" + return response +``` + +The Rust session routes this message through the existing single writer queue; +it must not add a second receiver for the same bridge direction. + +- [ ] **Step 4: Run the bridge-focused probes and compile gates.** + +```powershell +$env:PYTEST_DISABLE_PLUGIN_AUTOLOAD='1'; python -m pytest tests/test_storage_firewall.py::test_storage_admission_request_is_session_bound_and_replay_stable -q -o cache_dir=G:\2718lab\_codex\.codex-task-temp\storage-113-firewall-pytest +$env:CARGO_TARGET_DIR='G:\2718lab\_codex\.codex-task-temp\storage-113-firewall-rust-target'; cargo test -p codex-rmcp-client storage_admission --locked -j1 +``` + +Expected: Python `1 passed`; Rust storage protocol tests pass with no new +target root outside the named task target. + +- [ ] **Step 5: Commit the protocol slice.** + +```powershell +Push-Location 'G:\2718lab\_codex\.codex-task-temp\devkit-1.1.2-recovery'; git add mcp-tools/devkit_runtime/host_bridge.py mcp-tools/devkit_runtime/host_session.py mcp-tools/tests/test_storage_firewall.py; git commit -m 'feat: carry storage admission over host bridge'; Pop-Location +Push-Location 'G:\2718lab\_codex\.codex-task-temp\codex-host-mcp-fix-recovery2'; git add codex-rs/rmcp-client/src/inherited_host_bridge_protocol.rs codex-rs/rmcp-client/src/inherited_host_bridge_protocol/envelope.rs codex-rs/rmcp-client/src/inherited_host_bridge_protocol_tests.rs; git commit -m 'feat: carry storage admission over host bridge'; Pop-Location +``` + +### Task 5: Implement host target-key and capacity admission + +**Files:** +- Create: `codex-rs/core/src/fast_lane_host_dispatch/storage_firewall.rs` +- Create: `codex-rs/core/src/fast_lane_host_dispatch/storage_firewall_tests.rs` +- Modify: `codex-rs/core/src/fast_lane_host_dispatch/mod.rs:1-49` +- Modify: `codex-rs/core/Cargo.toml:80-145` +- Modify: `codex-rs/Cargo.lock` and `codex-rs/MODULE.bazel.lock` + +- [ ] **Step 1: Add the Rust RED target-key vectors.** + +```rust +#[test] +fn target_key_reuses_only_identical_build_semantics() { + let first = target_key(&descriptor("dev", "sha256:".to_owned() + &"1".repeat(64))).unwrap(); + let same = target_key(&descriptor("dev", "sha256:".to_owned() + &"1".repeat(64))).unwrap(); + let profile = target_key(&descriptor("release", "sha256:".to_owned() + &"1".repeat(64))).unwrap(); + let features = target_key(&descriptor("dev", "sha256:".to_owned() + &"2".repeat(64))).unwrap(); + assert_eq!(first, same); + assert_ne!(first, profile); + assert_ne!(first, features); +} +``` + +- [ ] **Step 2: Implement the exact policy and deterministic root mapping.** + +```rust +#[derive(Clone, Debug, PartialEq, Eq)] +pub(crate) struct StoragePolicy { + pub(crate) task_byte_limit: u64, + pub(crate) task_file_limit: u64, + pub(crate) target_family_byte_limit: u64, + pub(crate) target_family_file_limit: u64, + pub(crate) global_reserved_byte_limit: u64, + pub(crate) global_reserved_file_limit: u64, + pub(crate) free_space_floor_bytes: u64, + pub(crate) emergency_floor_bytes: u64, +} + +pub(crate) fn admit(&self, intent: &StorageIntent) -> Result { + let policy = self.policy.as_ref().ok_or(StorageError::PolicyMissing)?; + let free_before = self.capacity.free_bytes().map_err(|_| StorageError::StatUnavailable)?; + if intent.requested_bytes > policy.task_byte_limit { + return Err(StorageError::QuotaExceeded); + } + if intent.requested_files > policy.task_file_limit { + return Err(StorageError::FileLimitExceeded); + } + if self.family_observed_bytes + self.family_reserved_bytes + intent.requested_bytes + > policy.target_family_byte_limit + { + return Err(StorageError::QuotaExceeded); + } + if self.family_observed_files + self.family_reserved_files + intent.requested_files + > policy.target_family_file_limit + { + return Err(StorageError::FileLimitExceeded); + } + if self.global_reserved_bytes + intent.requested_bytes > policy.global_reserved_byte_limit + || self.global_reserved_files + intent.requested_files > policy.global_reserved_file_limit + { + return Err(StorageError::QuotaExceeded); + } + if free_before < policy.free_space_floor_bytes + || free_before - (self.global_reserved_bytes + intent.requested_bytes) + < policy.free_space_floor_bytes + { + return Err(StorageError::FreeSpaceFloor); + } + let target_key = target_key(&intent.target_descriptor)?; + let assigned_root = self.approved_root.join("generated").join(&target_key[7..]); + verify_strict_child(&self.approved_root, &assigned_root)?; + Ok(self.reserve(target_key, assigned_root, free_before, intent, policy)) +} +``` + +`StorageError::code()` must return exactly one of the Plan 1 codes +`STORAGE_ROOT_NOT_APPROVED`, `STORAGE_TARGET_KEY_INVALID`, +`STORAGE_POLICY_MISSING`, `STORAGE_QUOTA_EXCEEDED`, +`STORAGE_FILE_LIMIT_EXCEEDED`, `STORAGE_FREE_SPACE_FLOOR`, and +`STORAGE_STAT_UNAVAILABLE`. A missing/overflowed policy field is never treated +as zero and never treated as unlimited. + +- [ ] **Step 3: Implement platform capacity providers using the existing CLI doctor implementation as the reference.** + +On Unix call `libc::statvfs`; on Windows call +`GetDiskFreeSpaceExW`; on unsupported platforms return +`STORAGE_STAT_UNAVAILABLE`. The provider is injected as a trait in tests so +tests do not query the real G drive. `StorageFirewall::new` must not create the +approved root or target directory during construction or failed admission. + +- [ ] **Step 4: Run only the target-key and admission probes.** + +```powershell +$env:CARGO_TARGET_DIR='G:\2718lab\_codex\.codex-task-temp\storage-113-firewall-rust-target'; cargo test -p codex-core target_key_reuses_only_identical_build_semantics --locked -j1 +$env:CARGO_TARGET_DIR='G:\2718lab\_codex\.codex-task-temp\storage-113-firewall-rust-target'; cargo test -p codex-core missing_policy_is_stable_and_does_not_create_a_root --locked -j1 +``` + +Expected: both tests pass. If free-space statistics fail, the command must +stop with the stable error and must not create a second Cargo target. + +- [ ] **Step 5: Commit the host firewall.** + +```powershell +Push-Location 'G:\2718lab\_codex\.codex-task-temp\codex-host-mcp-fix-recovery2'; git add codex-rs/core/src/fast_lane_host_dispatch/storage_firewall.rs codex-rs/core/src/fast_lane_host_dispatch/storage_firewall_tests.rs codex-rs/core/src/fast_lane_host_dispatch/mod.rs codex-rs/core/Cargo.toml codex-rs/Cargo.lock codex-rs/MODULE.bazel.lock; git commit -m 'feat: enforce deterministic storage admission'; Pop-Location +``` + +### Task 6: Connect admission to preparation, worker environment, and terminal release + +**Files:** +- Modify: `codex-rs/core/src/fast_lane_host_dispatch/registry.rs:567-1668` +- Modify: `codex-rs/core/src/fast_lane_host_dispatch/coordinator.rs:393-580,1218-1260` +- Modify: `codex-rs/core/src/fast_lane_host_dispatch/codex_adapter.rs:1-1444` +- Modify: `mcp-tools/server.py:1008-1314` +- Test: `mcp-tools/tests/test_storage_firewall.py` +- Test: `codex-rs/core/src/fast_lane_host_dispatch/storage_firewall_tests.rs` + +- [ ] **Step 1: Add a RED production-path assertion that an admitted root is present only in host-owned worker facts.** + +```python +def test_fastlane_dispatch_forwards_receipt_identity_without_public_path(): + result = dispatch_fixture_with_storage_intent() + assert result["storage_admission"]["assigned_root_identity"].startswith("sha256:") + assert "assigned_root" not in result + assert "CARGO_TARGET_DIR" not in result +``` + +- [ ] **Step 2: Reserve before `prepare_batch`, inject after reservation, and release on every terminal/recovery branch.** + +```rust +let admission = self.storage.admit(&assignment.storage_intent)?; +let prepared = self.prepare_worktrees(&batch, &admission)?; +let mut environment = self.worker_environment(&prepared); +environment.insert("CARGO_TARGET_DIR".into(), admission.assigned_root().display().to_string()); +environment.insert("CODEX_TASK_TEMP".into(), admission.assigned_temp_root().display().to_string()); +``` + +The `assigned_root` and `assigned_temp_root` values remain in a private +`HostWriterContext`; only their identity hashes enter public receipts. On +`dispatch_all` error, terminal quarantine, successful integration, and +`recover_batch`, call `StorageFirewall::release` exactly once with the lease +ID. A release failure returns `STORAGE_POSTCHECK_FAILED` and retains the +ledger/lease state for Plan 2 recovery; it never triggers a broad delete. + +- [ ] **Step 3: Make the DevKit Fast Lane result expose only stable storage code and receipt identity.** + +```python +public = { + "storage_code": receipt.code, + "storage_receipt_hash": receipt.receipt_hash, + "target_key": receipt.target_key, +} +``` + +- [ ] **Step 4: Run the one production-path probe plus compile-first gates.** + +```powershell +$env:PYTEST_DISABLE_PLUGIN_AUTOLOAD='1'; python -m pytest tests/test_storage_firewall.py::test_fastlane_dispatch_forwards_receipt_identity_without_public_path -q -o cache_dir=G:\2718lab\_codex\.codex-task-temp\storage-113-firewall-pytest +python -m py_compile server.py devkit_runtime/host_bridge.py devkit_runtime/host_session.py +Push-Location 'G:\2718lab\_codex\.codex-task-temp\codex-host-mcp-fix-recovery2\codex-rs'; $env:CARGO_TARGET_DIR='G:\2718lab\_codex\.codex-task-temp\storage-113-firewall-rust-target'; cargo check -p codex-core --lib --locked -j1; Pop-Location +``` + +Expected: Python probe passes, `py_compile` is silent, and `cargo check` +finishes successfully with no new warning. This is the compile-first gate for +Plan 1; do not run the full workspace suite. + +- [ ] **Step 5: Commit the production wiring.** + +```powershell +Push-Location 'G:\2718lab\_codex\.codex-task-temp\devkit-1.1.2-recovery'; git add mcp-tools/server.py mcp-tools/devkit_runtime/host_bridge.py mcp-tools/devkit_runtime/host_session.py mcp-tools/tests/test_storage_firewall.py; git commit -m 'feat: bind admitted storage roots to fast lane workers'; Pop-Location +Push-Location 'G:\2718lab\_codex\.codex-task-temp\codex-host-mcp-fix-recovery2'; git add codex-rs/core/src/fast_lane_host_dispatch/registry.rs codex-rs/core/src/fast_lane_host_dispatch/coordinator.rs codex-rs/core/src/fast_lane_host_dispatch/codex_adapter.rs codex-rs/core/src/fast_lane_host_dispatch/storage_firewall_tests.rs; git commit -m 'feat: bind admitted storage roots to fast lane workers'; Pop-Location +``` + +## Plan 1 acceptance gate and handoff + +- [ ] Re-read the design sections “确定性 target key 与数据流” and “配额、文件数与剩余空间门槛”; verify every field and every fail-closed code is represented by a task above. +- [ ] Run `git diff --check` in both worktrees and verify only the mapped files changed. +- [ ] Run `python -m py_compile` on every changed DevKit Python file and `cargo check -p codex-core --lib --locked -j1` with the one named task target. +- [ ] Record one controlled admission receipt proving same semantics reuse one target key and one changed semantic forks it; record one low-space/policy-failure receipt proving no directory was created. +- [ ] Do not implement ledger persistence, preview/apply, source deletion, session deletion, compression, or remote synchronization in this plan. Plan 2 consumes `StorageAdmissionReceipt`; Plan 3 consumes the released/observed storage records. From f0fb64f54fe7bef317c7e1f8258a27a44d616333 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E5=93=80=E6=B4=9B=E8=8A=99?= Date: Sat, 29 Aug 2026 23:29:42 +0800 Subject: [PATCH 03/39] test: define storage firewall admission boundary --- mcp-tools/tests/test_storage_firewall.py | 35 ++++++++++++++++++++++++ 1 file changed, 35 insertions(+) create mode 100644 mcp-tools/tests/test_storage_firewall.py diff --git a/mcp-tools/tests/test_storage_firewall.py b/mcp-tools/tests/test_storage_firewall.py new file mode 100644 index 0000000..4b389e1 --- /dev/null +++ b/mcp-tools/tests/test_storage_firewall.py @@ -0,0 +1,35 @@ +from __future__ import annotations + + +def test_storage_intent_rejects_absolute_path_and_unknown_descriptor_key() -> None: + from devkit_runtime.storage_intent import StorageIntentError, parse_storage_intent + + value = { + "schema": "2718lab.storage.intent.v1", + "task_id": "task-01", + "plan_binding": "sha256:" + "a" * 64, + "context_hash": "sha256:" + "b" * 64, + "storage_intent_hash": "sha256:" + "c" * 64, + "requested_bytes": 1, + "requested_files": 1, + "target_descriptor": { + "schema": "2718lab.storage.target.v1", + "artifact_kind": "cargo-target", + "repository_identity": "sha256:" + "d" * 64, + "workspace_manifest_hash": "sha256:" + "e" * 64, + "cargo_lock_hash": "sha256:" + "f" * 64, + "toolchain_digest": "sha256:" + "1" * 64, + "target_triple": "x86_64-pc-windows-msvc", + "profile": "dev", + "features_hash": "sha256:" + "2" * 64, + "build_env_class": "windows-msvc", + "path": "G:/unapproved", + }, + } + + try: + parse_storage_intent(value) + except StorageIntentError as error: + assert error.code == "STORAGE_TARGET_KEY_INVALID" + else: + raise AssertionError("invalid descriptor was accepted") From 11be8aa639653d200455b94289cb41f7bd2d4489 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E5=93=80=E6=B4=9B=E8=8A=99?= Date: Sat, 29 Aug 2026 23:51:20 +0800 Subject: [PATCH 04/39] test: tighten storage intent rejection contract --- mcp-tools/tests/test_storage_firewall.py | 74 ++++++++++++++++++------ 1 file changed, 57 insertions(+), 17 deletions(-) diff --git a/mcp-tools/tests/test_storage_firewall.py b/mcp-tools/tests/test_storage_firewall.py index 4b389e1..75e3719 100644 --- a/mcp-tools/tests/test_storage_firewall.py +++ b/mcp-tools/tests/test_storage_firewall.py @@ -1,31 +1,59 @@ from __future__ import annotations +import hashlib +import json + + +def _canonical_hash(value: object) -> str: + encoded = json.dumps( + value, + ensure_ascii=True, + sort_keys=True, + separators=(",", ":"), + ).encode("utf-8") + return "sha256:" + hashlib.sha256(encoded).hexdigest() -def test_storage_intent_rejects_absolute_path_and_unknown_descriptor_key() -> None: - from devkit_runtime.storage_intent import StorageIntentError, parse_storage_intent - value = { +def _intent_with_unknown_descriptor_key( + key: str, value: str +) -> dict[str, object]: + descriptor: dict[str, str] = { + "schema": "2718lab.storage.target.v1", + "artifact_kind": "cargo-target", + "repository_identity": "sha256:" + "d" * 64, + "workspace_manifest_hash": "sha256:" + "e" * 64, + "cargo_lock_hash": "sha256:" + "f" * 64, + "toolchain_digest": "sha256:" + "1" * 64, + "target_triple": "x86_64-pc-windows-msvc", + "profile": "dev", + "features_hash": "sha256:" + "2" * 64, + "build_env_class": "windows-msvc", + } + intent: dict[str, object] = { "schema": "2718lab.storage.intent.v1", "task_id": "task-01", "plan_binding": "sha256:" + "a" * 64, "context_hash": "sha256:" + "b" * 64, - "storage_intent_hash": "sha256:" + "c" * 64, "requested_bytes": 1, "requested_files": 1, - "target_descriptor": { - "schema": "2718lab.storage.target.v1", - "artifact_kind": "cargo-target", - "repository_identity": "sha256:" + "d" * 64, - "workspace_manifest_hash": "sha256:" + "e" * 64, - "cargo_lock_hash": "sha256:" + "f" * 64, - "toolchain_digest": "sha256:" + "1" * 64, - "target_triple": "x86_64-pc-windows-msvc", - "profile": "dev", - "features_hash": "sha256:" + "2" * 64, - "build_env_class": "windows-msvc", - "path": "G:/unapproved", - }, + "target_descriptor": descriptor, } + intent["storage_intent_hash"] = _canonical_hash( + { + "target_descriptor": descriptor, + "task_id": intent["task_id"], + "plan_binding": intent["plan_binding"], + "context_hash": intent["context_hash"], + "requested_bytes": intent["requested_bytes"], + "requested_files": intent["requested_files"], + } + ) + descriptor[key] = value + return intent + + +def _assert_storage_intent_rejected(value: dict[str, object]) -> None: + from devkit_runtime.storage_intent import StorageIntentError, parse_storage_intent try: parse_storage_intent(value) @@ -33,3 +61,15 @@ def test_storage_intent_rejects_absolute_path_and_unknown_descriptor_key() -> No assert error.code == "STORAGE_TARGET_KEY_INVALID" else: raise AssertionError("invalid descriptor was accepted") + + +def test_storage_intent_rejects_absolute_path_and_unknown_descriptor_key() -> None: + _assert_storage_intent_rejected( + _intent_with_unknown_descriptor_key("path", "G:/unapproved") + ) + + +def test_storage_intent_rejects_plain_unknown_descriptor_key() -> None: + _assert_storage_intent_rejected( + _intent_with_unknown_descriptor_key("unexpected", "cache") + ) From dc27bfb67b8f6cd3018466c3e51f1bf2c80a89c6 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E5=93=80=E6=B4=9B=E8=8A=99?= Date: Sat, 29 Aug 2026 23:59:12 +0800 Subject: [PATCH 05/39] feat: add canonical storage intent --- mcp-tools/devkit_runtime/__init__.py | 4 + mcp-tools/devkit_runtime/storage_intent.py | 249 +++++++++++++++++++++ 2 files changed, 253 insertions(+) create mode 100644 mcp-tools/devkit_runtime/storage_intent.py diff --git a/mcp-tools/devkit_runtime/__init__.py b/mcp-tools/devkit_runtime/__init__.py index b02ac35..40f9372 100644 --- a/mcp-tools/devkit_runtime/__init__.py +++ b/mcp-tools/devkit_runtime/__init__.py @@ -5,9 +5,13 @@ VerifiedSqliteSnapshot, open_verified_sqlite_snapshot, ) +from .storage_intent import StorageIntent, StorageIntentError, parse_storage_intent __all__ = [ "SqliteSnapshotError", "VerifiedSqliteSnapshot", "open_verified_sqlite_snapshot", + "StorageIntent", + "StorageIntentError", + "parse_storage_intent", ] diff --git a/mcp-tools/devkit_runtime/storage_intent.py b/mcp-tools/devkit_runtime/storage_intent.py new file mode 100644 index 0000000..88af310 --- /dev/null +++ b/mcp-tools/devkit_runtime/storage_intent.py @@ -0,0 +1,249 @@ +"""Canonical, path-free storage intent validation for the DevKit runtime. + +The DevKit emits this value as an input to the authenticated Host storage +firewall. This module only validates and normalizes the value; it never +chooses a filesystem path, creates a directory, or claims ownership of a +storage lease. +""" + +from __future__ import annotations + +import hashlib +import json +import re +from collections.abc import Mapping +from dataclasses import dataclass +from types import MappingProxyType +from typing import Final + + +STORAGE_INTENT_SCHEMA: Final = "2718lab.storage.intent.v1" +TARGET_DESCRIPTOR_SCHEMA: Final = "2718lab.storage.target.v1" +STORAGE_TARGET_KEY_INVALID: Final = "STORAGE_TARGET_KEY_INVALID" + +_INTENT_FIELDS: Final = frozenset( + { + "schema", + "task_id", + "plan_binding", + "context_hash", + "storage_intent_hash", + "requested_bytes", + "requested_files", + "target_descriptor", + } +) +_TARGET_FIELDS: Final = frozenset( + { + "schema", + "artifact_kind", + "repository_identity", + "workspace_manifest_hash", + "cargo_lock_hash", + "toolchain_digest", + "target_triple", + "profile", + "features_hash", + "build_env_class", + } +) +_TARGET_FIELD_ORDER: Final = ( + "schema", + "artifact_kind", + "repository_identity", + "workspace_manifest_hash", + "cargo_lock_hash", + "toolchain_digest", + "target_triple", + "profile", + "features_hash", + "build_env_class", +) +_ARTIFACT_KINDS: Final = frozenset( + {"cargo-target", "python-cache", "mcp-package", "fastlane-task"} +) +_SHA256: Final = re.compile(r"sha256:[0-9a-f]{64}\Z") +_IDENTIFIER: Final = re.compile(r"[A-Za-z0-9][A-Za-z0-9._-]{0,255}\Z") +_MAX_IDENTIFIER_BYTES: Final = 256 +_MAX_U64: Final = (1 << 64) - 1 + + +class StorageIntentError(ValueError): + """A malformed storage intent represented by a stable error code.""" + + code: str + + def __init__(self, code: str = STORAGE_TARGET_KEY_INVALID) -> None: + self.code = code + super().__init__(code) + + +@dataclass(frozen=True, slots=True) +class StorageIntent: + """Validated immutable storage admission input. + + ``target_descriptor`` is copied into a read-only mapping so callers cannot + mutate the target semantics after validation. The record contains no + absolute path or owner/lease field; those remain Host-owned facts. + """ + + task_id: str + plan_binding: str + context_hash: str + storage_intent_hash: str + requested_bytes: int + requested_files: int + target_descriptor: Mapping[str, str] + + def __post_init__(self) -> None: + if not isinstance(self.target_descriptor, Mapping): + raise StorageIntentError() + descriptor = dict(self.target_descriptor) + if any( + type(key) is not str or type(value) is not str + for key, value in descriptor.items() + ): + raise StorageIntentError() + object.__setattr__( + self, + "target_descriptor", + MappingProxyType(descriptor), + ) + + def to_dict(self) -> dict[str, object]: + """Return a plain JSON-compatible projection for a Host boundary.""" + return { + "schema": STORAGE_INTENT_SCHEMA, + "task_id": self.task_id, + "plan_binding": self.plan_binding, + "context_hash": self.context_hash, + "storage_intent_hash": self.storage_intent_hash, + "requested_bytes": self.requested_bytes, + "requested_files": self.requested_files, + "target_descriptor": dict(self.target_descriptor), + } + + +def parse_storage_intent(value: object) -> StorageIntent: + """Parse one exact, canonical storage intent or fail closed. + + Unknown fields are rejected before any supplied digest is compared. The + intent digest binds the task/context, requested budgets, and complete + canonical target descriptor, as defined by the 1.1.3 storage contract. + """ + mapping = _exact_mapping(value, _INTENT_FIELDS) + descriptor = _exact_mapping(mapping["target_descriptor"], _TARGET_FIELDS) + + if mapping["schema"] != STORAGE_INTENT_SCHEMA: + raise StorageIntentError() + if descriptor["schema"] != TARGET_DESCRIPTOR_SCHEMA: + raise StorageIntentError() + + task_id = _bounded_identifier(mapping["task_id"]) + plan_binding = _digest(mapping["plan_binding"]) + context_hash = _digest(mapping["context_hash"]) + requested_bytes = _bounded_positive(mapping["requested_bytes"]) + requested_files = _bounded_positive(mapping["requested_files"]) + target = _canonical_target(descriptor) + + storage_intent_hash = _digest(mapping["storage_intent_hash"]) + expected = _sha256_json( + { + "target_descriptor": target, + "task_id": task_id, + "plan_binding": plan_binding, + "context_hash": context_hash, + "requested_bytes": requested_bytes, + "requested_files": requested_files, + } + ) + if expected != storage_intent_hash: + raise StorageIntentError() + + return StorageIntent( + task_id=task_id, + plan_binding=plan_binding, + context_hash=context_hash, + storage_intent_hash=storage_intent_hash, + requested_bytes=requested_bytes, + requested_files=requested_files, + target_descriptor=MappingProxyType(target), + ) + + +def _exact_mapping(value: object, fields: frozenset[str]) -> dict[str, object]: + if not isinstance(value, Mapping): + raise StorageIntentError() + try: + keys = tuple(value.keys()) + except (AttributeError, TypeError, ValueError) as error: + raise StorageIntentError() from error + if len(keys) != len(fields) or any(type(key) is not str for key in keys): + raise StorageIntentError() + if frozenset(keys) != fields: + raise StorageIntentError() + try: + return {key: value[key] for key in keys} + except (KeyError, TypeError, ValueError) as error: + raise StorageIntentError() from error + + +def _canonical_target(descriptor: Mapping[str, object]) -> dict[str, str]: + artifact_kind = descriptor["artifact_kind"] + if type(artifact_kind) is not str or artifact_kind not in _ARTIFACT_KINDS: + raise StorageIntentError() + + target: dict[str, str] = {} + for field in _TARGET_FIELD_ORDER: + raw = descriptor[field] + if type(raw) is not str: + raise StorageIntentError() + if field.endswith("_hash") or field in { + "repository_identity", + "toolchain_digest", + "features_hash", + }: + _digest(raw) + elif field not in {"schema", "artifact_kind"}: + _bounded_identifier(raw) + target[field] = raw + return target + + +def _bounded_identifier(value: object) -> str: + if type(value) is not str: + raise StorageIntentError() + if not value or len(value.encode("utf-8")) > _MAX_IDENTIFIER_BYTES: + raise StorageIntentError() + if _IDENTIFIER.fullmatch(value) is None: + raise StorageIntentError() + return value + + +def _digest(value: object) -> str: + if type(value) is not str or _SHA256.fullmatch(value) is None: + raise StorageIntentError() + return value + + +def _bounded_positive(value: object) -> int: + if type(value) is not int or not 0 < value <= _MAX_U64: + raise StorageIntentError() + return value + + +def _sha256_json(value: object) -> str: + try: + encoded = json.dumps( + value, + sort_keys=True, + separators=(",", ":"), + ensure_ascii=True, + allow_nan=False, + ).encode("utf-8") + except (TypeError, ValueError, UnicodeError) as error: + raise StorageIntentError() from error + return "sha256:" + hashlib.sha256(encoded).hexdigest() + + +__all__ = ["StorageIntent", "StorageIntentError", "parse_storage_intent"] From d787193a20583b8d5086842cedcd4252337040f5 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E5=93=80=E6=B4=9B=E8=8A=99?= Date: Sun, 30 Aug 2026 00:05:06 +0800 Subject: [PATCH 06/39] fix: normalize invalid storage identifiers --- mcp-tools/devkit_runtime/storage_intent.py | 4 ++-- mcp-tools/tests/test_storage_firewall.py | 17 +++++++++++++++++ 2 files changed, 19 insertions(+), 2 deletions(-) diff --git a/mcp-tools/devkit_runtime/storage_intent.py b/mcp-tools/devkit_runtime/storage_intent.py index 88af310..0a3636e 100644 --- a/mcp-tools/devkit_runtime/storage_intent.py +++ b/mcp-tools/devkit_runtime/storage_intent.py @@ -213,9 +213,9 @@ def _canonical_target(descriptor: Mapping[str, object]) -> dict[str, str]: def _bounded_identifier(value: object) -> str: if type(value) is not str: raise StorageIntentError() - if not value or len(value.encode("utf-8")) > _MAX_IDENTIFIER_BYTES: + if not value or _IDENTIFIER.fullmatch(value) is None: raise StorageIntentError() - if _IDENTIFIER.fullmatch(value) is None: + if len(value.encode("utf-8")) > _MAX_IDENTIFIER_BYTES: raise StorageIntentError() return value diff --git a/mcp-tools/tests/test_storage_firewall.py b/mcp-tools/tests/test_storage_firewall.py index 75e3719..4ebe615 100644 --- a/mcp-tools/tests/test_storage_firewall.py +++ b/mcp-tools/tests/test_storage_firewall.py @@ -73,3 +73,20 @@ def test_storage_intent_rejects_plain_unknown_descriptor_key() -> None: _assert_storage_intent_rejected( _intent_with_unknown_descriptor_key("unexpected", "cache") ) + + +def test_storage_intent_rejects_isolated_surrogate_with_stable_code() -> None: + value = _intent_with_unknown_descriptor_key("unexpected", "cache") + descriptor = value["target_descriptor"] + assert isinstance(descriptor, dict) + descriptor.pop("unexpected") + descriptor["target_triple"] = "\ud800" + + from devkit_runtime.storage_intent import StorageIntentError, parse_storage_intent + + try: + parse_storage_intent(value) + except StorageIntentError as error: + assert error.code == "STORAGE_TARGET_KEY_INVALID" + else: + raise AssertionError("invalid surrogate was accepted") From dce930e3a9ea2140686e820de5ad8de55324d690 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E5=93=80=E6=B4=9B=E8=8A=99?= Date: Sun, 30 Aug 2026 00:33:36 +0800 Subject: [PATCH 07/39] feat: bind storage intents to fast lane waves --- .../scripts/authenticated_v5_planner.py | 368 ++++++++++++++++-- .../scripts/authenticated_v5_projection.py | 129 ++++++ .../devkit_runtime/fastlane_host_intent.py | 98 ++++- mcp-tools/tests/test_storage_firewall.py | 185 +++++++++ 4 files changed, 745 insertions(+), 35 deletions(-) diff --git a/mcp-tools/devkit_fastlane/scripts/authenticated_v5_planner.py b/mcp-tools/devkit_fastlane/scripts/authenticated_v5_planner.py index 8916d2b..56d1fdf 100644 --- a/mcp-tools/devkit_fastlane/scripts/authenticated_v5_planner.py +++ b/mcp-tools/devkit_fastlane/scripts/authenticated_v5_planner.py @@ -16,11 +16,271 @@ "dispatch_order", "index_context_hash", "predecessor_hash", + "storage_budget", + "storage_intent", } ) +_LEGACY_UNIT_FIELDS = _UNIT_FIELDS - {"storage_budget", "storage_intent"} +_BUDGET_UNIT_FIELDS = _LEGACY_UNIT_FIELDS | {"storage_budget"} +_INTENT_UNIT_FIELDS = _LEGACY_UNIT_FIELDS | {"storage_intent"} _CONTEXTUAL_UNIT_FIELDS = (_UNIT_FIELDS - {"predecessor_hash"}) | {"workflow_id_hash"} +_LEGACY_CONTEXTUAL_UNIT_FIELDS = _CONTEXTUAL_UNIT_FIELDS - { + "storage_budget", + "storage_intent", +} +_BUDGET_CONTEXTUAL_UNIT_FIELDS = _LEGACY_CONTEXTUAL_UNIT_FIELDS | {"storage_budget"} +_INTENT_CONTEXTUAL_UNIT_FIELDS = _LEGACY_CONTEXTUAL_UNIT_FIELDS | {"storage_intent"} _ATTESTATION_ITEM_FIELDS = frozenset({"task_id", "request_binding_hash", "attestation"}) _CONCURRENCY_MODES = frozenset({"parallel", "serial", "isolated_worktree"}) +_STORAGE_INTENT_SCHEMA = "2718lab.storage.intent.v1" +_STORAGE_TARGET_SCHEMA = "2718lab.storage.target.v1" +_STORAGE_POLICY_MISSING = "STORAGE_POLICY_MISSING" +_STORAGE_TARGET_KEY_INVALID = "STORAGE_TARGET_KEY_INVALID" +_STORAGE_DESCRIPTOR_FIELDS = ( + "repository_identity", + "workspace_manifest_hash", + "cargo_lock_hash", + "toolchain_digest", + "target_triple", + "profile", + "features_hash", + "build_env_class", +) + + +def _storage_context_for_task( + context: Mapping[str, Any] | None, + task_id: str, +) -> Mapping[str, Any]: + """Resolve a task context without inventing any storage facts.""" + + if isinstance(context, Sequence) and not isinstance( + context, (str, bytes, bytearray) + ): + for item in context: + if isinstance(item, Mapping) and item.get("task_id") == task_id: + return item + return {} + if not isinstance(context, Mapping): + return {} + if "task_id" in context and context.get("task_id") != task_id: + return {} + direct = context.get(task_id) + if isinstance(direct, Mapping): + return direct + for field in ("by_task", "contexts", "execution_contexts"): + grouped = context.get(field) + if isinstance(grouped, Mapping): + direct = grouped.get(task_id) + if isinstance(direct, Mapping): + return direct + elif isinstance(grouped, Sequence) and not isinstance( + grouped, (str, bytes, bytearray) + ): + for item in grouped: + if isinstance(item, Mapping) and item.get("task_id") == task_id: + return item + return context + + +def _storage_value( + context: Mapping[str, Any], + source_unit: Mapping[str, Any], + field: str, +) -> object: + """Read one attested descriptor value from context or source unit.""" + + candidates: list[Mapping[str, Any]] = [context] + for container_name in ( + "storage_descriptor", + "target_descriptor", + "storage_target", + "build_context", + "bootstrap_plan", + "storage", + ): + container = context.get(container_name) + if isinstance(container, Mapping): + candidates.insert(0, container) + for candidate in candidates: + if field in candidate: + return candidate[field] + if field in source_unit: + return source_unit[field] + task = source_unit.get("task") + if isinstance(task, Mapping) and field in task: + return task[field] + return None + + +def _storage_execution_context_hash( + context: Mapping[str, Any], source_unit: Mapping[str, Any] +) -> object: + value = context.get("execution_context_hash") + if value is None: + value = source_unit.get("execution_context_hash") + return value + + +def _storage_budget( + source_unit: Mapping[str, Any], +) -> tuple[int, int]: + budget = source_unit.get("storage_budget") + if budget is None: + task = source_unit.get("task") + if isinstance(task, Mapping): + budget = task.get("storage_budget") + if not isinstance(budget, Mapping): + raise ValueError(_STORAGE_POLICY_MISSING) + requested_bytes = budget.get("bytes") + requested_files = budget.get("files") + if ( + type(requested_bytes) is not int + or requested_bytes <= 0 + or requested_bytes > (1 << 64) - 1 + or type(requested_files) is not int + or requested_files <= 0 + or requested_files > (1 << 64) - 1 + ): + raise ValueError(_STORAGE_POLICY_MISSING) + return requested_bytes, requested_files + + +def _make_storage_intent( + api: Any, + source_unit: Mapping[str, Any], + *, + task_id: str, + source_plan_hash: object, + context: Mapping[str, Any] | None, +) -> dict[str, object]: + """Build one path-free intent from a normalized unit and context.""" + + source_hash = api._hash(source_plan_hash, "source_plan_hash") + task_context = _storage_context_for_task(context, task_id) + context_hash = _storage_execution_context_hash(task_context, source_unit) + try: + context_hash = api._hash( + context_hash, + f"storage context {task_id}.execution_context_hash", + ) + except Exception as error: + raise ValueError(_STORAGE_POLICY_MISSING) from error + requested_bytes, requested_files = _storage_budget(source_unit) + descriptor = { + "schema": _STORAGE_TARGET_SCHEMA, + "artifact_kind": "fastlane-task", + **{ + field: _storage_value(task_context, source_unit, field) + for field in _STORAGE_DESCRIPTOR_FIELDS + }, + } + if any(descriptor[field] is None for field in _STORAGE_DESCRIPTOR_FIELDS): + raise ValueError(_STORAGE_POLICY_MISSING) + intent_without_hash = { + "schema": _STORAGE_INTENT_SCHEMA, + "task_id": task_id, + "plan_binding": source_hash, + "context_hash": context_hash, + "requested_bytes": requested_bytes, + "requested_files": requested_files, + "target_descriptor": descriptor, + } + intent = { + **intent_without_hash, + "storage_intent_hash": api._sha256_json( + { + "target_descriptor": descriptor, + "task_id": task_id, + "plan_binding": source_hash, + "context_hash": context_hash, + "requested_bytes": requested_bytes, + "requested_files": requested_files, + } + ), + } + try: + from devkit_runtime.storage_intent import parse_storage_intent + + return parse_storage_intent(intent).to_dict() + except Exception as error: + code = getattr(error, "code", _STORAGE_TARGET_KEY_INVALID) + raise ValueError(code) from error + + +def _validate_storage_intent( + value: object, + *, + task_id: str, + source_plan_hash: str, + context: Mapping[str, Any] | None, + source_unit: Mapping[str, Any], + api: Any, +) -> dict[str, object]: + try: + from devkit_runtime.storage_intent import parse_storage_intent + + parsed = parse_storage_intent(value) + except Exception as error: + code = getattr(error, "code", _STORAGE_TARGET_KEY_INVALID) + raise ValueError(code) from error + task_context = _storage_context_for_task(context, task_id) + expected_context = _storage_execution_context_hash(task_context, source_unit) + if expected_context is not None: + try: + expected_context = api._hash( + expected_context, + f"storage context {task_id}.execution_context_hash", + ) + except Exception as error: + raise ValueError(_STORAGE_TARGET_KEY_INVALID) from error + if "storage_budget" in source_unit: + try: + requested_bytes, requested_files = _storage_budget(source_unit) + except ValueError as error: + raise ValueError(_STORAGE_POLICY_MISSING) from error + if ( + parsed.requested_bytes != requested_bytes + or parsed.requested_files != requested_files + ): + raise ValueError(_STORAGE_TARGET_KEY_INVALID) + if ( + parsed.task_id != task_id + or parsed.plan_binding != source_plan_hash + or ( + expected_context is not None + and parsed.context_hash != expected_context + ) + ): + raise ValueError(_STORAGE_TARGET_KEY_INVALID) + return parsed.to_dict() + + +def _unit_storage_intent( + api: Any, + unit: Mapping[str, Any], + *, + task_id: str, + source_plan_hash: str, + context: Mapping[str, Any] | None, +) -> dict[str, object]: + supplied = unit.get("storage_intent") + if supplied is None: + return _make_storage_intent( + api, + unit, + task_id=task_id, + source_plan_hash=source_plan_hash, + context=context, + ) + return _validate_storage_intent( + supplied, + task_id=task_id, + source_plan_hash=source_plan_hash, + context=context, + source_unit=unit, + api=api, + ) def owned_scope_hash(api: Any, task_id: object, write_scope: object) -> str: @@ -49,7 +309,16 @@ def normalize_units( normalized: list[dict[str, Any]] = [] for index, raw_unit in enumerate(units): unit = api._mapping(raw_unit, f"authenticated V5 units[{index}]") - if set(unit) not in {_UNIT_FIELDS, _CONTEXTUAL_UNIT_FIELDS}: + if set(unit) not in { + _UNIT_FIELDS, + _CONTEXTUAL_UNIT_FIELDS, + _LEGACY_UNIT_FIELDS, + _LEGACY_CONTEXTUAL_UNIT_FIELDS, + _BUDGET_UNIT_FIELDS, + _BUDGET_CONTEXTUAL_UNIT_FIELDS, + _INTENT_UNIT_FIELDS, + _INTENT_CONTEXTUAL_UNIT_FIELDS, + }: raise ValueError(f"authenticated V5 units[{index}] has unsupported fields") task = dict(api._mapping(unit["task"], f"authenticated V5 units[{index}].task")) api._task_id( @@ -79,37 +348,53 @@ def normalize_units( or not 0 <= dispatch_order < 16 ): raise ValueError("authenticated V5 dispatch facts are invalid") - normalized.append( - { - "task": task, - "dependency_state": json.loads( - api._canonical_json(unit["dependency_state"]) - ), - "write_scope": write_scope, - "concurrency_mode": concurrency_mode, - "dispatch_order": dispatch_order, - "index_context_hash": api._hash( - unit["index_context_hash"], - f"authenticated V5 units[{index}].index_context_hash", - ), - "predecessor_hash": ( - api._hash( - unit["predecessor_hash"], - f"authenticated V5 units[{index}].predecessor_hash", - ) - if "predecessor_hash" in unit - else None - ), - "workflow_id_hash": ( - api._hash( - unit["workflow_id_hash"], - f"authenticated V5 units[{index}].workflow_id_hash", - ) - if "workflow_id_hash" in unit - else None - ), - } - ) + normalized_unit = { + "task": task, + "dependency_state": json.loads( + api._canonical_json(unit["dependency_state"]) + ), + "write_scope": write_scope, + "concurrency_mode": concurrency_mode, + "dispatch_order": dispatch_order, + "index_context_hash": api._hash( + unit["index_context_hash"], + f"authenticated V5 units[{index}].index_context_hash", + ), + "predecessor_hash": ( + api._hash( + unit["predecessor_hash"], + f"authenticated V5 units[{index}].predecessor_hash", + ) + if "predecessor_hash" in unit + else None + ), + "workflow_id_hash": ( + api._hash( + unit["workflow_id_hash"], + f"authenticated V5 units[{index}].workflow_id_hash", + ) + if "workflow_id_hash" in unit + else None + ), + } + if "storage_budget" in unit: + budget = unit["storage_budget"] + if not isinstance(budget, Mapping): + raise ValueError(_STORAGE_POLICY_MISSING) + normalized_unit["storage_budget"] = json.loads( + api._canonical_json(budget) + ) + if "storage_intent" in unit: + try: + from devkit_runtime.storage_intent import parse_storage_intent + + normalized_unit["storage_intent"] = parse_storage_intent( + unit["storage_intent"] + ).to_dict() + except Exception as error: + code = getattr(error, "code", _STORAGE_TARGET_KEY_INVALID) + raise ValueError(code) from error + normalized.append(normalized_unit) task_ids = [str(item["task"]["task_id"]) for item in normalized] orders = [int(item["dispatch_order"]) for item in normalized] if ( @@ -175,6 +460,7 @@ def compile_skeletons( source_plan_hash: object, routing_requests: Sequence[Mapping[str, Any]], attestation_items: Sequence[Mapping[str, Any]], + context: Mapping[str, Any] | None = None, ) -> dict[str, list[dict[str, Any]]]: source_hash = api._hash(source_plan_hash, "source_plan_hash") normalized_units = normalize_units(api, units) @@ -259,6 +545,13 @@ def compile_skeletons( route_pairs: set[tuple[str, str]] = set() for unit in normalized_units: task_id = str(unit["task"]["task_id"]) + storage_intent = _unit_storage_intent( + api, + unit, + task_id=task_id, + source_plan_hash=source_hash, + context=context, + ) request = request_by_task.get(task_id) attestation = attestation_by_task.get(task_id) if request is None or attestation is None: @@ -328,6 +621,7 @@ def compile_skeletons( "index_context_hash": unit["index_context_hash"], "predecessor_hash": predecessor_hash, "source_plan_hash": source_hash, + "storage_intent": storage_intent, } ) route_pairs.add((model, effort)) @@ -387,6 +681,16 @@ def validate_skeleton_package( skeleton.get("task_id"), f"authenticated V5 {wave_name} skeletons[{index}].task_id", ) + storage_intent = _validate_storage_intent( + skeleton.get("storage_intent"), + task_id=task_id, + source_plan_hash=source_hash, + context=None, + source_unit={}, + api=api, + ) + if skeleton.get("storage_intent") != storage_intent: + raise ValueError("authenticated V5 storage intent is not canonical") order = skeleton.get("dispatch_order") if type(order) is not int or not 0 <= order < len(source_ids): raise ValueError("authenticated V5 package dispatch order is invalid") diff --git a/mcp-tools/devkit_fastlane/scripts/authenticated_v5_projection.py b/mcp-tools/devkit_fastlane/scripts/authenticated_v5_projection.py index 91f5a94..df7e808 100644 --- a/mcp-tools/devkit_fastlane/scripts/authenticated_v5_projection.py +++ b/mcp-tools/devkit_fastlane/scripts/authenticated_v5_projection.py @@ -5,6 +5,123 @@ from collections.abc import Mapping, Sequence from typing import Any +_STORAGE_INTENT_SCHEMA = "2718lab.storage.intent.v1" +_STORAGE_TARGET_SCHEMA = "2718lab.storage.target.v1" +_STORAGE_POLICY_MISSING = "STORAGE_POLICY_MISSING" +_STORAGE_TARGET_KEY_INVALID = "STORAGE_TARGET_KEY_INVALID" +_STORAGE_DESCRIPTOR_FIELDS = ( + "repository_identity", + "workspace_manifest_hash", + "cargo_lock_hash", + "toolchain_digest", + "target_triple", + "profile", + "features_hash", + "build_env_class", +) + + +def _storage_context_value( + context: Mapping[str, Any], source_unit: Mapping[str, Any], field: str +) -> object: + candidates: list[Mapping[str, Any]] = [context] + for container_name in ( + "storage_descriptor", + "target_descriptor", + "storage_target", + "build_context", + "bootstrap_plan", + "storage", + ): + container = context.get(container_name) + if isinstance(container, Mapping): + candidates.insert(0, container) + for candidate in candidates: + if field in candidate: + return candidate[field] + task = source_unit.get("task") + if isinstance(task, Mapping) and field in task: + return task[field] + return source_unit.get(field) + + +def _make_storage_intent( + api: Any, + source_unit: Mapping[str, Any], + context: Mapping[str, Any], + *, + task_id: str, + source_plan_hash: str, +) -> dict[str, object]: + context_hash = context.get("execution_context_hash") + if context_hash is None: + context_hash = source_unit.get("execution_context_hash") + try: + context_hash = api._hash( + context_hash, + f"storage context {task_id}.execution_context_hash", + ) + except Exception as error: + raise ValueError(_STORAGE_POLICY_MISSING) from error + budget = source_unit.get("storage_budget") + if budget is None: + task = source_unit.get("task") + if isinstance(task, Mapping): + budget = task.get("storage_budget") + if not isinstance(budget, Mapping): + raise ValueError(_STORAGE_POLICY_MISSING) + requested_bytes = budget.get("bytes") + requested_files = budget.get("files") + if ( + type(requested_bytes) is not int + or requested_bytes <= 0 + or requested_bytes > (1 << 64) - 1 + or type(requested_files) is not int + or requested_files <= 0 + or requested_files > (1 << 64) - 1 + ): + raise ValueError(_STORAGE_POLICY_MISSING) + descriptor = { + "schema": _STORAGE_TARGET_SCHEMA, + "artifact_kind": "fastlane-task", + **{ + field: _storage_context_value(context, source_unit, field) + for field in _STORAGE_DESCRIPTOR_FIELDS + }, + } + if any(descriptor[field] is None for field in _STORAGE_DESCRIPTOR_FIELDS): + raise ValueError(_STORAGE_POLICY_MISSING) + intent_preimage = { + "target_descriptor": descriptor, + "task_id": task_id, + "plan_binding": source_plan_hash, + "context_hash": context_hash, + "requested_bytes": requested_bytes, + "requested_files": requested_files, + } + intent = { + "schema": _STORAGE_INTENT_SCHEMA, + **{ + key: intent_preimage[key] + for key in ( + "task_id", + "plan_binding", + "context_hash", + "requested_bytes", + "requested_files", + "target_descriptor", + ) + }, + "storage_intent_hash": api._sha256_json(intent_preimage), + } + try: + from devkit_runtime.storage_intent import parse_storage_intent + + return parse_storage_intent(intent).to_dict() + except Exception as error: + code = getattr(error, "code", _STORAGE_TARGET_KEY_INVALID) + raise ValueError(code) from error + def project_units( api: Any, @@ -220,6 +337,13 @@ def project_slice(task_ids: Sequence[str]) -> list[dict[str, Any]]: **dependency_without_hash, "dependency_state_hash": api._sha256_json(dependency_without_hash), } + storage_intent = _make_storage_intent( + api, + source_unit, + context_by_task[task_id], + task_id=task_id, + source_plan_hash=source_plan_hash, + ) criticality = { "Terra High": "normal", "Terra Max": "high", @@ -231,6 +355,10 @@ def project_slice(task_ids: Sequence[str]) -> list[dict[str, Any]]: "source_unit": source_unit, "target_gates": target, "dependency_state": dependency_state, + # Keep the complete intent in the projection preimage. A + # later dispatch binding therefore cannot omit storage + # semantics while retaining the same profile evidence hash. + "storage_intent": storage_intent, } task = { "schema": "2718lab-devkit/task-routing-profile-v5", @@ -276,6 +404,7 @@ def project_slice(task_ids: Sequence[str]) -> list[dict[str, Any]]: "dispatch_order": dispatch_order, "index_context_hash": index_hash, "workflow_id_hash": workflow_hash, + "storage_intent": storage_intent, } ) return projected_slice diff --git a/mcp-tools/devkit_runtime/fastlane_host_intent.py b/mcp-tools/devkit_runtime/fastlane_host_intent.py index fb7901d..4990083 100644 --- a/mcp-tools/devkit_runtime/fastlane_host_intent.py +++ b/mcp-tools/devkit_runtime/fastlane_host_intent.py @@ -14,6 +14,13 @@ from dataclasses import field as dataclass_field from typing import Final, Literal, cast +from .storage_intent import ( + STORAGE_TARGET_KEY_INVALID, + StorageIntent, + StorageIntentError, + parse_storage_intent, +) + NO_SAFE_WORK: Final = "NO_SAFE_WORK" UNSPLITTABLE: Final = "UNSPLITTABLE" @@ -46,6 +53,7 @@ "intent_hash", } ) +_ROOT_STORAGE_KEYS: Final = _ROOT_KEYS | {"storage_intent", "execution_context_hash"} _ASSIGNMENT_KEYS: Final = frozenset( { "assignment_id", @@ -54,6 +62,10 @@ "assignment_binding_hash", } ) +_ASSIGNMENT_STORAGE_KEYS: Final = _ASSIGNMENT_KEYS | { + "storage_intent", + "execution_context_hash", +} _PREDECESSOR_KEYS: Final = frozenset( { "schema", @@ -240,6 +252,8 @@ class HostExecutionExpectationProjection: lease_owner: str lease_epoch: int lease_fencing_token: str + storage_intent: StorageIntent | None = None + execution_context_hash: str | None = None @dataclass(frozen=True, slots=True) @@ -287,6 +301,8 @@ class ParsedHostExecutionIntent: lease_epoch: int lease_fencing_token: str lease_binding_hash: str + storage_intent: StorageIntent | None = None + execution_context_hash: str | None = None @dataclass(frozen=True, slots=True) @@ -391,7 +407,11 @@ def classify_host_scheduler_topology( def _parse(candidate: object) -> ParsedHostExecutionIntent | None: - root = _bound_mapping(candidate, _ROOT_KEYS, "intent_hash") + root = _bound_mapping_variant( + candidate, + (_ROOT_KEYS, _ROOT_STORAGE_KEYS), + "intent_hash", + ) if root is None or _text(root, "schema") != _SCHEMA: return None @@ -407,8 +427,10 @@ def _parse(candidate: object) -> ParsedHostExecutionIntent | None: ): return None - assignment = _bound_mapping( - root["assignment"], _ASSIGNMENT_KEYS, "assignment_binding_hash" + assignment = _bound_mapping_variant( + root["assignment"], + (_ASSIGNMENT_KEYS, _ASSIGNMENT_STORAGE_KEYS), + "assignment_binding_hash", ) route = _bound_mapping(root["route"], _ROUTE_KEYS, "route_binding_hash") packets = _bound_mapping(root["packets"], _PACKET_KEYS, "packet_binding_hash") @@ -488,6 +510,31 @@ def _parse(candidate: object) -> ParsedHostExecutionIntent | None: active_lease_set_hash, ) = validated_predecessor + storage_intent: StorageIntent | None = None + execution_context_hash: str | None = None + raw_storage_intent = root.get("storage_intent") + if raw_storage_intent is None and assignment is not None: + raw_storage_intent = assignment.get("storage_intent") + if raw_storage_intent is not None: + try: + storage_intent = parse_storage_intent(raw_storage_intent) + except StorageIntentError: + raise + except Exception as error: + raise StorageIntentError(STORAGE_TARGET_KEY_INVALID) from error + raw_execution_context_hash = root.get("execution_context_hash") + if raw_execution_context_hash is None and assignment is not None: + raw_execution_context_hash = assignment.get("execution_context_hash") + if not _is_hash_value(raw_execution_context_hash): + raise StorageIntentError(STORAGE_TARGET_KEY_INVALID) + execution_context_hash = cast(str, raw_execution_context_hash) + if ( + storage_intent.task_id != task_id + or storage_intent.plan_binding != source_plan_hash + or storage_intent.context_hash != execution_context_hash + ): + raise StorageIntentError(STORAGE_TARGET_KEY_INVALID) + capability_facts = _validate_capability_facts(root["capability_facts"]) if capability_facts is None or not _has_candidate_capability_claim( capability_facts, @@ -593,6 +640,8 @@ def _parse(candidate: object) -> ParsedHostExecutionIntent | None: lease_epoch=lease_epoch, lease_fencing_token=lease_fencing_token, lease_binding_hash=lease_binding_hash, + storage_intent=storage_intent, + execution_context_hash=execution_context_hash, ) @@ -969,6 +1018,15 @@ def matches_host_execution_expectation( and expectation_projection.lease_epoch == parsed_intent.lease_epoch and expectation_projection.lease_fencing_token == parsed_intent.lease_fencing_token + and ( + expectation_projection.storage_intent is None + or expectation_projection.storage_intent == parsed_intent.storage_intent + ) + and ( + expectation_projection.execution_context_hash is None + or expectation_projection.execution_context_hash + == parsed_intent.execution_context_hash + ) ) @@ -979,6 +1037,14 @@ def _is_expectation_projection( if not _capability_expectations_are_valid(expectation.capability_facts): return False + if expectation.storage_intent is not None and type( + expectation.storage_intent + ) is not StorageIntent: + return False + if expectation.execution_context_hash is not None and not _is_hash_value( + expectation.execution_context_hash + ): + return False hash_values = ( expectation.candidate_intent_hash, expectation.projection_hash, @@ -1128,6 +1194,22 @@ def _bound_mapping( return mapping if _canonical_hash(unbound) == binding_hash else None +def _bound_mapping_variant( + value: object, + expected_keys: tuple[frozenset[str], ...], + binding_field: str, +) -> dict[str, object] | None: + mapping = _exact_mapping_variant(value, expected_keys) + if mapping is None: + return None + binding_hash = _valid_hash(mapping, binding_field) + if binding_hash is None: + return None + unbound = dict(mapping) + del unbound[binding_field] + return mapping if _canonical_hash(unbound) == binding_hash else None + + def _exact_mapping( value: object, expected_keys: frozenset[str], @@ -1138,6 +1220,16 @@ def _exact_mapping( return mapping if set(mapping) == expected_keys else None +def _exact_mapping_variant( + value: object, + expected_keys: tuple[frozenset[str], ...], +) -> dict[str, object] | None: + if type(value) is not dict: + return None + mapping = cast(dict[str, object], value) + return mapping if any(set(mapping) == keys for keys in expected_keys) else None + + def _text(mapping: dict[str, object], field: str) -> str | None: value = mapping[field] return value if type(value) is str else None diff --git a/mcp-tools/tests/test_storage_firewall.py b/mcp-tools/tests/test_storage_firewall.py index 4ebe615..a94472e 100644 --- a/mcp-tools/tests/test_storage_firewall.py +++ b/mcp-tools/tests/test_storage_firewall.py @@ -90,3 +90,188 @@ def test_storage_intent_rejects_isolated_surrogate_with_stable_code() -> None: assert error.code == "STORAGE_TARGET_KEY_INVALID" else: raise AssertionError("invalid surrogate was accepted") + + +class _StorageBindingRoutingCore: + def load_policy_v5(self) -> dict[str, object]: + return {} + + def policy_hash_v5(self, policy: object) -> str: + del policy + return _canonical_hash({"policy": "storage-binding"}) + + def _normalise_request_v5( + self, request: dict[str, object], policy: object + ) -> dict[str, object]: + del policy + return request + + def v5_request_binding_hash(self, request: object) -> str: + return _canonical_hash(request) + + def route_v5( + self, request: dict[str, object], *, policy: object + ) -> dict[str, object]: + del policy + task = request["task"] + assert isinstance(task, dict) + return { + "schema": "2718lab-devkit/fastlane-routing-result-v5", + "status": "resolved", + "task_id": task["task_id"], + "route": { + "model": "gpt-5.6-luna", + "effort": "max", + "inherit_current_session_model": False, + }, + } + + +class _StorageBindingApi: + def __init__(self) -> None: + self.core = _StorageBindingRoutingCore() + + def _mapping(self, value: object, field: str) -> dict[str, object]: + assert isinstance(value, dict), field + return value + + def _task_id(self, value: object, field: str) -> str: + assert isinstance(value, str), field + return value + + def _normalised_scopes(self, value: object, field: str = "scope") -> list[str]: + assert isinstance(value, list), field + return value + + def _canonical_json(self, value: object) -> str: + return json.dumps(value, ensure_ascii=True, sort_keys=True, separators=(",", ":")) + + def _sha256_json(self, value: object) -> str: + return _canonical_hash(value) + + def _hash(self, value: object, field: str) -> str: + assert isinstance(value, str), field + return value + + def _text(self, value: object, field: str, *, maximum: int) -> str: + assert isinstance(value, str) and len(value) <= maximum, field + return value + + def _exact_keys( + self, value: dict[str, object], expected: frozenset[str], field: str + ) -> None: + assert set(value) == expected, field + + def _fast_lane_routing_core(self) -> _StorageBindingRoutingCore: + return self.core + + +def _storage_binding_unit(task_id: str, dispatch_order: int) -> dict[str, object]: + return { + "task": { + "schema": "2718lab-devkit/task-routing-profile-v5", + "task_id": task_id, + "role": "execution", + "access": "workspace_write", + "write_scope_count": 1, + "overlap_risk": "none", + "overlap_count": 0, + }, + "dependency_state": {"task_id": task_id}, + "write_scope": [f"src/{task_id}.py"], + "concurrency_mode": "parallel", + "dispatch_order": dispatch_order, + "index_context_hash": "sha256:" + "b" * 64, + "workflow_id_hash": "sha256:" + "c" * 64, + "storage_budget": {"bytes": 4096, "files": 8}, + } + + +def _storage_binding_context() -> dict[str, object]: + return { + "execution_context_hash": "sha256:" + "4" * 64, + "repository_identity": "sha256:" + "5" * 64, + "workspace_manifest_hash": "sha256:" + "6" * 64, + "cargo_lock_hash": "sha256:" + "7" * 64, + "toolchain_digest": "sha256:" + "8" * 64, + "target_triple": "x86_64-pc-windows-msvc", + "profile": "dev", + "features_hash": "sha256:" + "9" * 64, + "build_env_class": "windows-msvc", + } + + +def _storage_binding_request( + api: _StorageBindingApi, unit: dict[str, object], source_plan_hash: str +) -> tuple[dict[str, object], dict[str, object]]: + task = unit["task"] + assert isinstance(task, dict) + request: dict[str, object] = { + "task": task, + "scheduler_facts": {"route_epoch": 1}, + "child_route_attestation": None, + } + binding_hash = api.core.v5_request_binding_hash(request) + attestation: dict[str, object] = { + "request_binding_hash": binding_hash, + "attestation": { + "status": "attested", + "request_binding_hash": binding_hash, + }, + } + attestation_payload = attestation["attestation"] + assert isinstance(attestation_payload, dict) + attestation_payload["attestation_hash"] = _canonical_hash( + { + key: value + for key, value in attestation_payload.items() + if key != "attestation_hash" + } + ) + item = { + "task_id": task["task_id"], + "request_binding_hash": binding_hash, + "attestation": attestation_payload, + } + del source_plan_hash + return request, item + + +def test_every_fastlane_wave_carries_plan_context_bound_storage_intent() -> None: + from devkit_fastlane.scripts.authenticated_v5_planner import compile_skeletons + + api = _StorageBindingApi() + plan_hash = "sha256:" + "a" * 64 + context = _storage_binding_context() + first_unit = _storage_binding_unit("task-01", 0) + successor_unit = _storage_binding_unit("task-02", 1) + + first_request, first_attestation = _storage_binding_request( + api, first_unit, plan_hash + ) + successor_request, successor_attestation = _storage_binding_request( + api, successor_unit, plan_hash + ) + first = compile_skeletons( + api, + [first_unit], + source_plan_hash=plan_hash, + routing_requests=[first_request], + attestation_items=[first_attestation], + context=context, + ) + successor = compile_skeletons( + api, + [successor_unit], + source_plan_hash=plan_hash, + routing_requests=[successor_request], + attestation_items=[successor_attestation], + context=context, + ) + + for wave in (first["assignment_skeletons"], successor["assignment_skeletons"]): + for assignment in wave: + intent = assignment["storage_intent"] + assert intent["task_id"] == assignment["task_id"] + assert intent["plan_binding"] == plan_hash + assert intent["context_hash"] == context["execution_context_hash"] From a5063c47c9fb46cb638e0f00ad1666e5974bc8ca Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E5=93=80=E6=B4=9B=E8=8A=99?= Date: Sun, 30 Aug 2026 00:55:02 +0800 Subject: [PATCH 08/39] fix: migrate verified legacy runtime stores --- mcp-tools/orchestrator/store.py | 215 +++++++++++++------- mcp-tools/tests/test_runtime_composition.py | 35 ++++ 2 files changed, 175 insertions(+), 75 deletions(-) diff --git a/mcp-tools/orchestrator/store.py b/mcp-tools/orchestrator/store.py index 3a6922c..2cb7d9d 100644 --- a/mcp-tools/orchestrator/store.py +++ b/mcp-tools/orchestrator/store.py @@ -7073,6 +7073,125 @@ def _canonicalize_external_bootstrap_expiries(cursor: sqlite3.Cursor) -> None: ), ) + @classmethod + def _validate_legacy_atlas_outbox_shape(cls, cursor: sqlite3.Cursor) -> None: + """Prove the v6-v10 outbox is the known legacy DDL before rebuilding it.""" + + expected_columns = tuple( + ( + name, + column_type.casefold(), + 0 if name == "ingestion_key" else not_null, + primary_key, + 0, + ) + for name, column_type, not_null, primary_key in _ATLAS_OUTBOX_COLUMN_CONTRACT + ) + if _table_column_contract(cursor, "atlas_ingestion_outbox") != expected_columns: + raise StoreError("orchestrator store is not prepared") + + def index_columns(index_name: object) -> tuple[str, ...]: + if type(index_name) is not str: + raise ValueError("index name is invalid") + identifier = index_name.replace('"', '""') + return tuple( + str(row["name"]) + for row in cursor.execute(f'PRAGMA index_info("{identifier}")').fetchall() + ) + + unique_indexes: set[tuple[str, ...]] = set() + non_unique_indexes: set[tuple[str, ...]] = set() + index_rows = cursor.execute( + "PRAGMA index_list(atlas_ingestion_outbox)" + ).fetchall() + for index in index_rows: + if int(index["partial"]): + raise StoreError("orchestrator store is not prepared") + columns = index_columns(index["name"]) + (unique_indexes if int(index["unique"]) else non_unique_indexes).add(columns) + expected_unique_indexes = { + ("ingestion_key",), + ("acceptance_id",), + ("payload_hash",), + _ATLAS_OUTBOX_IDENTITY, + } + expected_non_unique_indexes = {("state", "created_at", "ingestion_key")} + if ( + unique_indexes != expected_unique_indexes + or non_unique_indexes != expected_non_unique_indexes + or len(index_rows) + != len(expected_unique_indexes) + len(expected_non_unique_indexes) + ): + raise StoreError("orchestrator store is not prepared") + + expected_foreign_keys = frozenset( + { + ( + ( + "acceptance_id", + "code_task_acceptances", + "acceptance_id", + "no action", + "no action", + "none", + ), + ) + } + ) + if ( + _foreign_key_contract(cursor, "atlas_ingestion_outbox") + != expected_foreign_keys + or _sqlite_check_expressions( + _table_sql(cursor, "atlas_ingestion_outbox") + ) + != _ATLAS_OUTBOX_REQUIRED_CHECKS + ): + raise StoreError("orchestrator store is not prepared") + + @classmethod + def _validate_legacy_atlas_outbox_rows(cls, cursor: sqlite3.Cursor) -> None: + """Reject null or non-convertible legacy rows before the table swap.""" + + text_columns = { + "ingestion_key", + "acceptance_id", + "payload_json", + "payload_hash", + "state", + "last_error_code", + "reason_codes_json", + "created_at", + "updated_at", + } + rows = cursor.execute("SELECT * FROM atlas_ingestion_outbox").fetchall() + for row in rows: + if any(row[column] is None for column in _ATLAS_OUTBOX_COLUMNS): + raise StoreError("legacy atlas outbox row is invalid") + if any(type(row[column]) is not str for column in text_columns): + raise StoreError("legacy atlas outbox row is invalid") + if type(row["attempt_count"]) is not int: + raise StoreError("legacy atlas outbox row is invalid") + if ( + row["state"] not in {"pending", "projected", "quarantined"} + or not 0 <= row["attempt_count"] <= cls._MAX_ATLAS_OUTBOX_ATTEMPTS + or row["ingestion_key"] != row["payload_hash"] + ): + raise StoreError("legacy atlas outbox row is invalid") + try: + cls._safe_acceptance_identifier("ingestion_key", row["ingestion_key"]) + cls._safe_acceptance_identifier("acceptance_id", row["acceptance_id"]) + cls._safe_acceptance_identifier("payload_hash", row["payload_hash"]) + cls._safe_outbox_code( + "last_error_code", row["last_error_code"], allow_empty=True + ) + reason_codes = _decode_outbox_reason_codes(row["reason_codes_json"]) + if len(reason_codes) > cls._MAX_SAFE_OUTBOX_REASON_COUNT: + raise ValueError("too many outbox reason codes") + for reason_code in reason_codes: + cls._safe_outbox_code("reason_code", reason_code) + except (StoreError, TypeError, ValueError, json.JSONDecodeError) as error: + raise StoreError("legacy atlas outbox row is invalid") from error + @classmethod def _migrate_atlas_outbox_ingestion_key_not_null( cls, cursor: sqlite3.Cursor @@ -7100,85 +7219,31 @@ def _migrate_atlas_outbox_ingestion_key_not_null( } if set(columns) != required_columns: raise StoreError("orchestrator store is not prepared") - ingestion_key = columns["ingestion_key"] - if int(ingestion_key["notnull"]): - return source_version_row = cursor.execute( "SELECT value FROM schema_metadata WHERE key = 'schema_version'" ).fetchone() - source_version = ( - None if source_version_row is None else int(source_version_row["value"]) - ) - - def index_columns(index_name: object) -> tuple[str, ...]: - if type(index_name) is not str: - raise ValueError("index name is invalid") - identifier = index_name.replace('"', '""') - return tuple( - str(row["name"]) - for row in cursor.execute(f'PRAGMA index_info("{identifier}")') - ) - - outbox_not_null_columns = { - str(row["name"]) - for row in columns.values() - if int(row["notnull"]) - } - outbox_primary_key = tuple( - str(row["name"]) - for row in sorted( - columns.values(), key=lambda row: int(row["pk"]) - ) - if int(row["pk"]) - ) - outbox_unique_columns = { - index_columns(row["name"]) - for row in cursor.execute( - "PRAGMA index_list(atlas_ingestion_outbox)" - ).fetchall() - if int(row["unique"]) and not int(row["partial"]) - } - outbox_foreign_keys = { - (str(row["from"]), str(row["table"]), str(row["to"])) - for row in cursor.execute( - "PRAGMA foreign_key_list(atlas_ingestion_outbox)" - ) - } - outbox_row = cursor.execute( - """ - SELECT sql FROM sqlite_master - WHERE type = 'table' AND name = 'atlas_ingestion_outbox' - """ - ).fetchone() - outbox_checks = ( - frozenset() - if outbox_row is None - else _sqlite_check_expressions(outbox_row["sql"]) - ) - required_not_null_columns = required_columns - {"ingestion_key"} - has_complete_v10_layout = ( - required_not_null_columns.issubset(outbox_not_null_columns) - and outbox_primary_key == ("ingestion_key",) - and ("acceptance_id",) in outbox_unique_columns - and ("payload_hash",) in outbox_unique_columns - and ( - "acceptance_id", - "code_task_acceptances", - "acceptance_id", - ) - in outbox_foreign_keys - and _ATLAS_OUTBOX_REQUIRED_CHECKS.issubset(outbox_checks) - ) - if source_version != 10 or not has_complete_v10_layout: + ingestion_key = columns["ingestion_key"] + if source_version_row is None: + if int(ingestion_key["notnull"]): + return raise StoreError("orchestrator store is not prepared") - if cursor.execute( - """ - SELECT 1 FROM atlas_ingestion_outbox - WHERE ingestion_key IS NULL - LIMIT 1 - """ - ).fetchone() is not None: - raise StoreError("legacy atlas outbox row is invalid") + source_version_value = source_version_row["value"] + if type(source_version_value) is not str: + raise StoreError("orchestrator store is not prepared") + try: + source_version = int(source_version_value) + except (TypeError, ValueError) as error: + raise StoreError("orchestrator store is not prepared") from error + if str(source_version) != source_version_value or not ( + 6 <= source_version <= cls._SCHEMA_VERSION + ): + raise StoreError("orchestrator store is not prepared") + if int(ingestion_key["notnull"]): + return + if source_version not in range(6, 11): + raise StoreError("orchestrator store is not prepared") + cls._validate_legacy_atlas_outbox_shape(cursor) + cls._validate_legacy_atlas_outbox_rows(cursor) cursor.execute( """ CREATE TABLE atlas_ingestion_outbox_v11 ( diff --git a/mcp-tools/tests/test_runtime_composition.py b/mcp-tools/tests/test_runtime_composition.py index f01bd1a..968b155 100644 --- a/mcp-tools/tests/test_runtime_composition.py +++ b/mcp-tools/tests/test_runtime_composition.py @@ -1243,6 +1243,41 @@ def test_sqlite_store_migrates_v10_outbox_to_reject_null_ingestion_keys( store.close() +@pytest.mark.parametrize("schema_version", (6, 7, 8, 9, 10)) +def test_sqlite_store_bootstraps_verified_legacy_empty_outbox( + tmp_path: Path, schema_version: int +) -> None: + database, _, _ = _legacy_v10_atlas_outbox_database( + tmp_path, ingestion_key=f"sha256:{'a' * 64}" + ) + connection = sqlite3.connect(database) + try: + for trigger_name in ( + "atlas_finalizations_no_update", + "atlas_finalizations_no_delete", + "atlas_finalizations_require_projected_outbox", + ): + connection.execute(f"DROP TRIGGER {trigger_name}") + connection.execute("DROP TABLE atlas_finalizations") + connection.execute("DELETE FROM atlas_ingestion_outbox") + connection.execute( + "UPDATE schema_metadata SET value = ? WHERE key = 'schema_version'", + (str(schema_version),), + ) + connection.commit() + finally: + connection.close() + + store = SQLiteStore(database) + try: + assert store.schema_version() == 13 + assert store._connection.execute( + "SELECT COUNT(*) FROM atlas_ingestion_outbox" + ).fetchone()[0] == 0 + finally: + store.close() + + def test_sqlite_store_fails_closed_for_legacy_null_outbox_key(tmp_path: Path) -> None: database, _, _ = _legacy_v10_atlas_outbox_database(tmp_path, ingestion_key=None) From c1e88389fa7ec2d1c7d660e8ce06a4291dfa6984 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E5=93=80=E6=B4=9B=E8=8A=99?= Date: Sun, 30 Aug 2026 01:23:02 +0800 Subject: [PATCH 09/39] fix: validate legacy stores before migration ddl --- mcp-tools/orchestrator/store.py | 64 +++- mcp-tools/tests/test_runtime_composition.py | 331 +++++++++++++++++++- 2 files changed, 379 insertions(+), 16 deletions(-) diff --git a/mcp-tools/orchestrator/store.py b/mcp-tools/orchestrator/store.py index 2cb7d9d..e10d544 100644 --- a/mcp-tools/orchestrator/store.py +++ b/mcp-tools/orchestrator/store.py @@ -7099,8 +7099,8 @@ def index_columns(index_name: object) -> tuple[str, ...]: for row in cursor.execute(f'PRAGMA index_info("{identifier}")').fetchall() ) - unique_indexes: set[tuple[str, ...]] = set() - non_unique_indexes: set[tuple[str, ...]] = set() + unique_indexes: set[tuple[tuple[str, ...], str]] = set() + non_unique_indexes: set[tuple[tuple[str, ...], str, str]] = set() index_rows = cursor.execute( "PRAGMA index_list(atlas_ingestion_outbox)" ).fetchall() @@ -7108,14 +7108,19 @@ def index_columns(index_name: object) -> tuple[str, ...]: if int(index["partial"]): raise StoreError("orchestrator store is not prepared") columns = index_columns(index["name"]) - (unique_indexes if int(index["unique"]) else non_unique_indexes).add(columns) + origin = str(index["origin"]).casefold() + if int(index["unique"]): + unique_indexes.add((columns, origin)) + else: + non_unique_indexes.add((columns, origin, str(index["name"]))) expected_unique_indexes = { - ("ingestion_key",), - ("acceptance_id",), - ("payload_hash",), - _ATLAS_OUTBOX_IDENTITY, + (("ingestion_key",), "pk"), + (("acceptance_id",), "u"), + (("payload_hash",), "u"), + } + expected_non_unique_indexes = { + (("state", "created_at", "ingestion_key"), "c", "idx_atlas_outbox_pending") } - expected_non_unique_indexes = {("state", "created_at", "ingestion_key")} if ( unique_indexes != expected_unique_indexes or non_unique_indexes != expected_non_unique_indexes @@ -7175,9 +7180,27 @@ def _validate_legacy_atlas_outbox_rows(cls, cursor: sqlite3.Cursor) -> None: row["state"] not in {"pending", "projected", "quarantined"} or not 0 <= row["attempt_count"] <= cls._MAX_ATLAS_OUTBOX_ATTEMPTS or row["ingestion_key"] != row["payload_hash"] + or ( + row["state"] == "projected" + and row["last_error_code"] != "" + ) + or ( + row["state"] == "quarantined" + and not row["last_error_code"] + ) + or ( + row["state"] == "pending" + and row["attempt_count"] > 0 + and not row["last_error_code"] + ) ): raise StoreError("legacy atlas outbox row is invalid") try: + if ( + row["created_at"] != _utc_timestamp(row["created_at"]) + or row["updated_at"] != _utc_timestamp(row["updated_at"]) + ): + raise ValueError("outbox timestamps are not canonical UTC") cls._safe_acceptance_identifier("ingestion_key", row["ingestion_key"]) cls._safe_acceptance_identifier("acceptance_id", row["acceptance_id"]) cls._safe_acceptance_identifier("payload_hash", row["payload_hash"]) @@ -7242,7 +7265,6 @@ def _migrate_atlas_outbox_ingestion_key_not_null( return if source_version not in range(6, 11): raise StoreError("orchestrator store is not prepared") - cls._validate_legacy_atlas_outbox_shape(cursor) cls._validate_legacy_atlas_outbox_rows(cursor) cursor.execute( """ @@ -7305,6 +7327,27 @@ def _migrate_atlas_outbox_ingestion_key_not_null( except (IndexError, TypeError, ValueError, sqlite3.DatabaseError) as error: raise StoreError("orchestrator schema is corrupt") from error + @classmethod + def _preflight_legacy_atlas_outbox_before_schema_ddl( + cls, cursor: sqlite3.Cursor, *, fresh_database: bool + ) -> None: + """Validate legacy outbox DDL before current CREATE/INDEX statements run.""" + + if fresh_database: + return + try: + source_version = _schema_version_from_connection(cursor) + except (IndexError, TypeError, ValueError, sqlite3.DatabaseError) as error: + raise StoreError("orchestrator store is not prepared") from error + if source_version is None or source_version not in range(6, 11): + return + try: + cls._validate_legacy_atlas_outbox_shape(cursor) + except StoreError: + raise + except (IndexError, TypeError, ValueError, sqlite3.DatabaseError) as error: + raise StoreError("orchestrator store is not prepared") from error + @staticmethod def _drop_atlas_finalization_projection_trigger_for_outbox_rebuild( cursor: sqlite3.Cursor, @@ -7361,6 +7404,9 @@ def _create_schema(self) -> None: ).fetchone() is None except sqlite3.DatabaseError as error: raise StoreError("orchestrator schema is corrupt") from error + self._preflight_legacy_atlas_outbox_before_schema_ddl( + cursor, fresh_database=fresh_database + ) _execute_schema_statements( cursor, """ diff --git a/mcp-tools/tests/test_runtime_composition.py b/mcp-tools/tests/test_runtime_composition.py index 968b155..fefc8f4 100644 --- a/mcp-tools/tests/test_runtime_composition.py +++ b/mcp-tools/tests/test_runtime_composition.py @@ -889,6 +889,137 @@ def not_null(column: str) -> str: """ +def _legacy_v6_schema() -> str: + """Build the complete v6 schema without passing through the current store.""" + + return f""" + CREATE TABLE schema_metadata (key TEXT PRIMARY KEY, value TEXT NOT NULL); + CREATE TABLE workflows ( + id TEXT PRIMARY KEY, kind TEXT NOT NULL, title TEXT NOT NULL, + product_summary TEXT NOT NULL, state TEXT NOT NULL, + version INTEGER NOT NULL, policy_version TEXT NOT NULL, + created_at TEXT NOT NULL, updated_at TEXT NOT NULL + ); + CREATE TABLE tasks ( + id TEXT PRIMARY KEY, workflow_id TEXT NOT NULL REFERENCES workflows(id), + title TEXT NOT NULL, owner_role TEXT NOT NULL, state TEXT NOT NULL, + write_scope TEXT NOT NULL, card_hash TEXT NOT NULL, + result_hash TEXT NOT NULL, version INTEGER NOT NULL, + task_kind TEXT NOT NULL DEFAULT 'general', + intent_id TEXT NOT NULL DEFAULT '', language TEXT NOT NULL DEFAULT '', + framework TEXT NOT NULL DEFAULT '' + ); + CREATE INDEX idx_tasks_workflow_state ON tasks(workflow_id, state); + CREATE TABLE code_task_acceptances ( + acceptance_id TEXT PRIMARY KEY, workflow_id TEXT NOT NULL REFERENCES workflows(id), + code_task_id TEXT NOT NULL UNIQUE REFERENCES tasks(id), code_task_version INTEGER NOT NULL, + input_snapshot_id TEXT NOT NULL, output_snapshot_id TEXT NOT NULL, + indexed_diff_hash TEXT NOT NULL, intent_id TEXT NOT NULL, + language TEXT NOT NULL, framework TEXT NOT NULL, + payload_json TEXT NOT NULL, payload_hash TEXT NOT NULL UNIQUE, + created_at TEXT NOT NULL, + CHECK (acceptance_id = payload_hash) + ); + CREATE INDEX idx_code_task_acceptances_workflow + ON code_task_acceptances(workflow_id, created_at, acceptance_id); + CREATE TABLE code_task_receipt_attestations ( + task_id TEXT PRIMARY KEY REFERENCES tasks(id), workflow_id TEXT NOT NULL REFERENCES workflows(id), + code_task_version INTEGER NOT NULL, input_snapshot_id TEXT NOT NULL, + output_snapshot_id TEXT NOT NULL, workspace_hash TEXT NOT NULL, + execution_receipt_ids TEXT NOT NULL, attestation_hash TEXT NOT NULL UNIQUE, + created_at TEXT NOT NULL, + UNIQUE (task_id, code_task_version, attestation_hash) + ); + CREATE INDEX idx_code_task_receipt_attestations_workflow + ON code_task_receipt_attestations(workflow_id, code_task_version, task_id); + CREATE TABLE code_task_receipt_owners ( + receipt_id TEXT PRIMARY KEY, task_id TEXT NOT NULL, + code_task_version INTEGER NOT NULL, attestation_hash TEXT NOT NULL, + FOREIGN KEY (task_id, code_task_version, attestation_hash) + REFERENCES code_task_receipt_attestations(task_id, code_task_version, attestation_hash) + ); + CREATE INDEX idx_code_task_receipt_owners_task + ON code_task_receipt_owners(task_id, code_task_version, attestation_hash, receipt_id); + {_atlas_outbox_schema()} + CREATE INDEX idx_atlas_outbox_pending + ON atlas_ingestion_outbox(state, created_at, ingestion_key); + CREATE TABLE task_dependencies ( + task_id TEXT NOT NULL REFERENCES tasks(id), dependency_id TEXT NOT NULL REFERENCES tasks(id), + PRIMARY KEY (task_id, dependency_id) + ); + CREATE INDEX idx_task_dependencies_dependency ON task_dependencies(dependency_id); + CREATE TABLE lease_epochs (task_id TEXT PRIMARY KEY REFERENCES tasks(id), epoch INTEGER NOT NULL); + CREATE TABLE leases ( + task_id TEXT PRIMARY KEY REFERENCES tasks(id), owner TEXT NOT NULL, + epoch INTEGER NOT NULL, expires_at TEXT NOT NULL, + heartbeat_at TEXT NOT NULL, host_target TEXT + ); + CREATE TABLE events ( + sequence INTEGER PRIMARY KEY AUTOINCREMENT, + workflow_id TEXT NOT NULL REFERENCES workflows(id), task_id TEXT REFERENCES tasks(id), + event_type TEXT NOT NULL, redacted_payload TEXT NOT NULL, + payload_hash TEXT NOT NULL, created_at TEXT NOT NULL + ); + CREATE INDEX idx_events_workflow_sequence ON events(workflow_id, sequence); + CREATE TABLE artifacts ( + content_hash TEXT PRIMARY KEY, kind TEXT NOT NULL, safe_path TEXT NOT NULL, + size INTEGER NOT NULL, redaction_version TEXT NOT NULL, created_at TEXT NOT NULL + ); + CREATE TABLE task_inputs (task_id TEXT PRIMARY KEY REFERENCES tasks(id), input_hash TEXT NOT NULL); + CREATE TABLE artifact_owners (content_hash TEXT PRIMARY KEY REFERENCES artifacts(content_hash), task_id TEXT NOT NULL REFERENCES tasks(id)); + CREATE TABLE task_cards (task_id TEXT PRIMARY KEY REFERENCES tasks(id), card_hash TEXT NOT NULL, card_body TEXT NOT NULL); + CREATE TABLE task_contract_subscriptions ( + task_id TEXT NOT NULL REFERENCES tasks(id), contract_hash TEXT NOT NULL, + PRIMARY KEY (task_id, contract_hash) + ); + CREATE TABLE task_required_evidence ( + task_id TEXT NOT NULL REFERENCES tasks(id), position INTEGER NOT NULL, + evidence TEXT NOT NULL, + PRIMARY KEY (task_id, position) + ); + CREATE TABLE task_index_bindings ( + task_id TEXT PRIMARY KEY REFERENCES tasks(id), workspace_root TEXT NOT NULL DEFAULT '', + workspace_id TEXT NOT NULL DEFAULT '', input_snapshot_id TEXT NOT NULL, + output_snapshot_id TEXT NOT NULL, task_node_ids TEXT NOT NULL, + contract_node_ids TEXT NOT NULL, checkpoint_id TEXT NOT NULL, + indexed_diff_hash TEXT NOT NULL, fallback_count INTEGER NOT NULL + ); + CREATE TABLE task_index_query_receipts ( + task_id TEXT NOT NULL REFERENCES tasks(id), trace_id TEXT NOT NULL, + snapshot_id TEXT NOT NULL, miss_escape_used INTEGER NOT NULL, + recorded_at TEXT NOT NULL, + PRIMARY KEY (task_id, trace_id) + ); + CREATE INDEX idx_task_index_query_snapshot ON task_index_query_receipts(task_id, snapshot_id); + CREATE TABLE task_index_verification_artifacts ( + task_id TEXT NOT NULL REFERENCES tasks(id), + content_hash TEXT NOT NULL REFERENCES artifacts(content_hash), snapshot_id TEXT NOT NULL, + PRIMARY KEY (task_id, content_hash) + ); + CREATE TABLE task_index_binding_events ( + sequence INTEGER PRIMARY KEY AUTOINCREMENT, task_id TEXT NOT NULL REFERENCES tasks(id), + event_type TEXT NOT NULL, snapshot_id TEXT NOT NULL, + trace_id TEXT NOT NULL, created_at TEXT NOT NULL + ); + CREATE TABLE peer_capabilities ( + workflow_id TEXT NOT NULL REFERENCES workflows(id), sender_task_id TEXT NOT NULL REFERENCES tasks(id), + recipient_task_id TEXT NOT NULL REFERENCES tasks(id), relationship TEXT NOT NULL, + capability TEXT NOT NULL UNIQUE, + PRIMARY KEY (workflow_id, sender_task_id, recipient_task_id, relationship) + ); + CREATE TABLE messages ( + sequence INTEGER PRIMARY KEY AUTOINCREMENT, delivery_id TEXT NOT NULL UNIQUE, + workflow_id TEXT NOT NULL REFERENCES workflows(id), sender_task_id TEXT NOT NULL REFERENCES tasks(id), + recipient_task_id TEXT NOT NULL REFERENCES tasks(id), correlation_id TEXT NOT NULL, + artifact_hash TEXT NOT NULL REFERENCES artifacts(content_hash), redacted_metadata TEXT NOT NULL, + created_at TEXT NOT NULL, expires_at TEXT NOT NULL, + delivery_state TEXT NOT NULL, acknowledged_at TEXT, + UNIQUE (workflow_id, sender_task_id, recipient_task_id, correlation_id) + ); + CREATE INDEX idx_messages_recipient_inbox ON messages(workflow_id, recipient_task_id, sequence); + """ + + def _runtime_with_malformed_atlas_outbox( tmp_path: Path, outbox_schema: str ) -> RuntimeConfig: @@ -1042,7 +1173,7 @@ def prepare_proof_registry(database_path: Path) -> None: def _insert_legacy_atlas_acceptance( connection: sqlite3.Connection, *, suffix: str ) -> tuple[str, str]: - timestamp = "2026-08-09T00:00:00Z" + timestamp = "2026-08-09T00:00:00+00:00" workflow_id = "legacy-workflow" task_id = f"legacy-task-{suffix}" acceptance_id = f"sha256:{suffix * 64}" @@ -1112,6 +1243,49 @@ def _insert_legacy_atlas_acceptance( return acceptance_id, timestamp +def _legacy_v6_atlas_outbox_database( + tmp_path: Path, *, ingestion_key: str | None +) -> tuple[Path, str, str]: + """Create a complete historical v6 store, without bootstrapping v13 first.""" + + database = tmp_path / "legacy-v6-atlas-outbox.sqlite3" + connection = sqlite3.connect(database) + try: + connection.execute("PRAGMA foreign_keys = ON") + connection.executescript(_legacy_v6_schema()) + acceptance_id, timestamp = _insert_legacy_atlas_acceptance( + connection, suffix="a" + ) + connection.execute( + "INSERT INTO schema_metadata (key, value) VALUES (?, ?)", + ("schema_version", "6"), + ) + connection.execute( + """ + INSERT INTO atlas_ingestion_outbox ( + ingestion_key, acceptance_id, payload_json, payload_hash, state, + attempt_count, last_error_code, reason_codes_json, created_at, updated_at + ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?) + """, + ( + ingestion_key, + acceptance_id, + "{}", + acceptance_id, + "pending", + 0, + "", + "[]", + timestamp, + timestamp, + ), + ) + connection.commit() + finally: + connection.close() + return database, acceptance_id, timestamp + + def _legacy_v10_atlas_outbox_database( tmp_path: Path, *, ingestion_key: str | None ) -> tuple[Path, str, str]: @@ -1139,6 +1313,10 @@ def _legacy_v10_atlas_outbox_database( ) connection.execute("DROP TABLE atlas_ingestion_outbox") connection.executescript(_atlas_outbox_schema()) + connection.execute( + "CREATE INDEX idx_atlas_outbox_pending " + "ON atlas_ingestion_outbox(state, created_at, ingestion_key)" + ) connection.execute( """ INSERT INTO atlas_ingestion_outbox ( @@ -1243,9 +1421,152 @@ def test_sqlite_store_migrates_v10_outbox_to_reject_null_ingestion_keys( store.close() -@pytest.mark.parametrize("schema_version", (6, 7, 8, 9, 10)) +@pytest.mark.parametrize("missing_object", ("table", "pending-index")) +def test_sqlite_store_rejects_legacy_v6_outbox_drift_before_current_ddl( + tmp_path: Path, missing_object: str +) -> None: + database, _, _ = _legacy_v6_atlas_outbox_database( + tmp_path, ingestion_key=f"sha256:{'a' * 64}" + ) + connection = sqlite3.connect(database) + try: + if missing_object == "table": + connection.execute("DROP TABLE atlas_ingestion_outbox") + else: + connection.execute("DROP INDEX idx_atlas_outbox_pending") + connection.commit() + finally: + connection.close() + + with pytest.raises(StoreError, match="orchestrator store is not prepared"): + SQLiteStore(database) + + connection = sqlite3.connect(database) + try: + assert connection.execute( + "SELECT value FROM schema_metadata WHERE key = 'schema_version'" + ).fetchone()[0] == "6" + if missing_object == "table": + assert connection.execute( + "SELECT 1 FROM sqlite_master WHERE type = 'table' " + "AND name = 'atlas_ingestion_outbox'" + ).fetchone() is None + else: + assert connection.execute( + "SELECT 1 FROM sqlite_master WHERE type = 'index' " + "AND name = 'idx_atlas_outbox_pending'" + ).fetchone() is None + finally: + connection.close() + + +def test_sqlite_store_bootstraps_true_legacy_v6_empty_outbox( + tmp_path: Path, +) -> None: + database, _, _ = _legacy_v6_atlas_outbox_database( + tmp_path, ingestion_key=f"sha256:{'a' * 64}" + ) + connection = sqlite3.connect(database) + try: + connection.execute("DELETE FROM atlas_ingestion_outbox") + assert connection.execute( + "SELECT value FROM schema_metadata WHERE key = 'schema_version'" + ).fetchone() == ("6",) + assert connection.execute( + "SELECT 1 FROM sqlite_master WHERE type = 'table' " + "AND name = 'atlas_finalizations'" + ).fetchone() is None + connection.commit() + finally: + connection.close() + + store = SQLiteStore(database) + try: + assert store.schema_version() == 13 + assert store._connection.execute( + "SELECT COUNT(*) FROM atlas_ingestion_outbox" + ).fetchone()[0] == 0 + assert store._connection.execute( + "SELECT 1 FROM sqlite_master WHERE type = 'table' " + "AND name = 'atlas_finalizations'" + ).fetchone() is not None + finally: + store.close() + + +@pytest.mark.parametrize( + "outbox_update", + ( + {"state": "projected", "attempt_count": 0, "last_error_code": "ERR"}, + {"state": "quarantined", "attempt_count": 0, "last_error_code": ""}, + {"state": "pending", "attempt_count": 1, "last_error_code": ""}, + { + "state": "pending", + "attempt_count": 0, + "last_error_code": "", + "created_at": "2026-08-09T00:00:00+08:00", + "updated_at": "2026-08-09T00:00:00+08:00", + }, + ), + ids=( + "projected-error", + "quarantined-missing-error", + "pending-retry-missing-error", + "non-utc-timestamp", + ), +) +def test_sqlite_store_rejects_legacy_atlas_outbox_row_contract_drift( + tmp_path: Path, outbox_update: dict[str, object] +) -> None: + database, _, _ = _legacy_v10_atlas_outbox_database( + tmp_path, ingestion_key=f"sha256:{'a' * 64}" + ) + connection = sqlite3.connect(database) + try: + assignments = { + "state": outbox_update.get("state", "pending"), + "attempt_count": outbox_update.get("attempt_count", 0), + "last_error_code": outbox_update.get("last_error_code", ""), + "created_at": outbox_update.get( + "created_at", "2026-08-09T00:00:00+00:00" + ), + "updated_at": outbox_update.get( + "updated_at", "2026-08-09T00:00:00+00:00" + ), + } + connection.execute( + """ + UPDATE atlas_ingestion_outbox + SET state = ?, attempt_count = ?, last_error_code = ?, + created_at = ?, updated_at = ? + """, + tuple(assignments.values()), + ) + connection.commit() + finally: + connection.close() + + with pytest.raises(StoreError, match="legacy atlas outbox row is invalid"): + SQLiteStore(database) + + connection = sqlite3.connect(database) + try: + assert connection.execute( + "SELECT value FROM schema_metadata WHERE key = 'schema_version'" + ).fetchone()[0] == "10" + columns = { + str(row[1]): int(row[3]) + for row in connection.execute( + "PRAGMA table_info(atlas_ingestion_outbox)" + ).fetchall() + } + assert columns["ingestion_key"] == 0 + finally: + connection.close() + + def test_sqlite_store_bootstraps_verified_legacy_empty_outbox( - tmp_path: Path, schema_version: int + tmp_path: Path, ) -> None: database, _, _ = _legacy_v10_atlas_outbox_database( tmp_path, ingestion_key=f"sha256:{'a' * 64}" @@ -1260,10 +1581,6 @@ def test_sqlite_store_bootstraps_verified_legacy_empty_outbox( connection.execute(f"DROP TRIGGER {trigger_name}") connection.execute("DROP TABLE atlas_finalizations") connection.execute("DELETE FROM atlas_ingestion_outbox") - connection.execute( - "UPDATE schema_metadata SET value = ? WHERE key = 'schema_version'", - (str(schema_version),), - ) connection.commit() finally: connection.close() From b7ea0dc85646e397aa365358e8840d0da0f01ed2 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E5=93=80=E6=B4=9B=E8=8A=99?= Date: Sun, 30 Aug 2026 01:31:43 +0800 Subject: [PATCH 10/39] fix: reject semantic legacy schema drift --- mcp-tools/orchestrator/store.py | 73 +++++++++++++++++-- mcp-tools/tests/test_runtime_composition.py | 77 +++++++++++++++++---- 2 files changed, 132 insertions(+), 18 deletions(-) diff --git a/mcp-tools/orchestrator/store.py b/mcp-tools/orchestrator/store.py index e10d544..fcadc1e 100644 --- a/mcp-tools/orchestrator/store.py +++ b/mcp-tools/orchestrator/store.py @@ -7084,12 +7084,34 @@ def _validate_legacy_atlas_outbox_shape(cls, cursor: sqlite3.Cursor) -> None: 0 if name == "ingestion_key" else not_null, primary_key, 0, + None, ) for name, column_type, not_null, primary_key in _ATLAS_OUTBOX_COLUMN_CONTRACT ) - if _table_column_contract(cursor, "atlas_ingestion_outbox") != expected_columns: + actual_columns = tuple( + ( + str(row["name"]), + str(row["type"]).casefold(), + int(row["notnull"]), + int(row["pk"]), + int(row["hidden"]), + row["dflt_value"], + ) + for row in cursor.execute( + "PRAGMA table_xinfo(atlas_ingestion_outbox)" + ).fetchall() + ) + if actual_columns != expected_columns: raise StoreError("orchestrator store is not prepared") + table_sql = _table_sql(cursor, "atlas_ingestion_outbox") + table_tokens = _sqlite_schema_tokens(table_sql) + for index, token in enumerate(table_tokens): + if token == "collate" and ( + index + 1 == len(table_tokens) or table_tokens[index + 1] != "binary" + ): + raise StoreError("orchestrator store is not prepared") + def index_columns(index_name: object) -> tuple[str, ...]: if type(index_name) is not str: raise ValueError("index name is invalid") @@ -7099,6 +7121,44 @@ def index_columns(index_name: object) -> tuple[str, ...]: for row in cursor.execute(f'PRAGMA index_info("{identifier}")').fetchall() ) + def validate_index_xinfo( + index_name: object, expected_columns: tuple[str, ...] + ) -> None: + if type(index_name) is not str: + raise ValueError("index name is invalid") + identifier = index_name.replace('"', '""') + info_rows = cursor.execute( + f'PRAGMA index_xinfo("{identifier}")' + ).fetchall() + key_rows = [row for row in info_rows if int(row["key"])] + if len(key_rows) != len(expected_columns): + raise StoreError("orchestrator store is not prepared") + for sequence, (row, expected_column) in enumerate( + zip(key_rows, expected_columns, strict=True) + ): + if ( + int(row["seqno"]) != sequence + or int(row["cid"]) != _ATLAS_OUTBOX_COLUMNS.index(expected_column) + or row["name"] != expected_column + or str(row["coll"]).casefold() != "binary" + or int(row["desc"]) != 0 + or int(row["key"]) != 1 + ): + raise StoreError("orchestrator store is not prepared") + non_key_rows = [row for row in info_rows if not int(row["key"])] + if len(non_key_rows) != 1: + raise StoreError("orchestrator store is not prepared") + non_key = non_key_rows[0] + if ( + int(non_key["seqno"]) != len(expected_columns) + or int(non_key["cid"]) != -1 + or non_key["name"] is not None + or str(non_key["coll"]).casefold() != "binary" + or int(non_key["desc"]) != 0 + or int(non_key["key"]) != 0 + ): + raise StoreError("orchestrator store is not prepared") + unique_indexes: set[tuple[tuple[str, ...], str]] = set() non_unique_indexes: set[tuple[tuple[str, ...], str, str]] = set() index_rows = cursor.execute( @@ -7108,6 +7168,7 @@ def index_columns(index_name: object) -> tuple[str, ...]: if int(index["partial"]): raise StoreError("orchestrator store is not prepared") columns = index_columns(index["name"]) + validate_index_xinfo(index["name"], columns) origin = str(index["origin"]).casefold() if int(index["unique"]): unique_indexes.add((columns, origin)) @@ -7146,10 +7207,7 @@ def index_columns(index_name: object) -> tuple[str, ...]: if ( _foreign_key_contract(cursor, "atlas_ingestion_outbox") != expected_foreign_keys - or _sqlite_check_expressions( - _table_sql(cursor, "atlas_ingestion_outbox") - ) - != _ATLAS_OUTBOX_REQUIRED_CHECKS + or _sqlite_check_expressions(table_sql) != _ATLAS_OUTBOX_REQUIRED_CHECKS ): raise StoreError("orchestrator store is not prepared") @@ -7193,6 +7251,11 @@ def _validate_legacy_atlas_outbox_rows(cls, cursor: sqlite3.Cursor) -> None: and row["attempt_count"] > 0 and not row["last_error_code"] ) + or ( + row["state"] == "pending" + and row["attempt_count"] == 0 + and row["last_error_code"] != "" + ) ): raise StoreError("legacy atlas outbox row is invalid") try: diff --git a/mcp-tools/tests/test_runtime_composition.py b/mcp-tools/tests/test_runtime_composition.py index fefc8f4..097445f 100644 --- a/mcp-tools/tests/test_runtime_composition.py +++ b/mcp-tools/tests/test_runtime_composition.py @@ -841,21 +841,25 @@ def _atlas_outbox_schema( equality_check: str = "CHECK (ingestion_key = payload_hash)", state_check: str = "CHECK (state IN ('pending', 'projected', 'quarantined'))", attempt_check: str = "CHECK (attempt_count BETWEEN 0 AND 16)", + column_collation: str = "", ) -> str: def not_null(column: str) -> str: return "" if nullable_column == column else " NOT NULL" + def text_type(column: str) -> str: + return f"TEXT{column_collation}{not_null(column)}" + payload_json = ( - f"payload_json TEXT{not_null('payload_json')}," + f"payload_json {text_type('payload_json')}," if include_payload_json else "" ) if partial_unique_indexes: acceptance_id = ( - f"acceptance_id TEXT{not_null('acceptance_id')} " + f"acceptance_id {text_type('acceptance_id')} " "REFERENCES code_task_acceptances(acceptance_id)," ) - payload_hash = f"payload_hash TEXT{not_null('payload_hash')}," + payload_hash = f"payload_hash {text_type('payload_hash')}," unique_indexes = """ CREATE UNIQUE INDEX atlas_outbox_acceptance_partial ON atlas_ingestion_outbox(acceptance_id) @@ -866,23 +870,23 @@ def not_null(column: str) -> str: """ else: acceptance_id = ( - f"acceptance_id TEXT{not_null('acceptance_id')} UNIQUE " + f"acceptance_id {text_type('acceptance_id')} UNIQUE " "REFERENCES code_task_acceptances(acceptance_id)," ) - payload_hash = f"payload_hash TEXT{not_null('payload_hash')} UNIQUE," + payload_hash = f"payload_hash {text_type('payload_hash')} UNIQUE," unique_indexes = "" return f""" CREATE TABLE atlas_ingestion_outbox ( - ingestion_key TEXT PRIMARY KEY, + ingestion_key TEXT{column_collation} PRIMARY KEY, {acceptance_id} {payload_json} {payload_hash} - state TEXT{not_null('state')} {state_check}, + state {text_type('state')} {state_check}, attempt_count INTEGER{not_null('attempt_count')} {attempt_check}, - last_error_code TEXT{not_null('last_error_code')}, - reason_codes_json TEXT{not_null('reason_codes_json')}, - created_at TEXT{not_null('created_at')}, - updated_at TEXT{not_null('updated_at')}, + last_error_code {text_type('last_error_code')}, + reason_codes_json {text_type('reason_codes_json')}, + created_at {text_type('created_at')}, + updated_at {text_type('updated_at')}, {equality_check} ); {unique_indexes} @@ -940,9 +944,11 @@ def _legacy_v6_schema() -> str: ); CREATE INDEX idx_code_task_receipt_owners_task ON code_task_receipt_owners(task_id, code_task_version, attestation_hash, receipt_id); - {_atlas_outbox_schema()} + {_atlas_outbox_schema(column_collation=" COLLATE BINARY")} CREATE INDEX idx_atlas_outbox_pending - ON atlas_ingestion_outbox(state, created_at, ingestion_key); + ON atlas_ingestion_outbox( + state ASC, created_at ASC, ingestion_key ASC + ); CREATE TABLE task_dependencies ( task_id TEXT NOT NULL REFERENCES tasks(id), dependency_id TEXT NOT NULL REFERENCES tasks(id), PRIMARY KEY (task_id, dependency_id) @@ -1494,12 +1500,56 @@ def test_sqlite_store_bootstraps_true_legacy_v6_empty_outbox( store.close() +@pytest.mark.parametrize( + "semantic_drift", ("column-nocase", "pending-nocase-desc") +) +def test_sqlite_store_rejects_legacy_v6_semantic_shape_drift( + tmp_path: Path, semantic_drift: str +) -> None: + database, _, _ = _legacy_v6_atlas_outbox_database( + tmp_path, ingestion_key=f"sha256:{'a' * 64}" + ) + connection = sqlite3.connect(database) + try: + if semantic_drift == "column-nocase": + connection.execute("DROP TABLE atlas_ingestion_outbox") + connection.executescript( + _atlas_outbox_schema(column_collation=" COLLATE NOCASE") + ) + connection.execute( + "CREATE INDEX idx_atlas_outbox_pending " + "ON atlas_ingestion_outbox(state ASC, created_at ASC, ingestion_key ASC)" + ) + else: + connection.execute("DROP INDEX idx_atlas_outbox_pending") + connection.execute( + "CREATE INDEX idx_atlas_outbox_pending " + "ON atlas_ingestion_outbox(" + "state COLLATE NOCASE DESC, created_at ASC, ingestion_key ASC)" + ) + connection.commit() + finally: + connection.close() + + with pytest.raises(StoreError, match="orchestrator store is not prepared"): + SQLiteStore(database) + + connection = sqlite3.connect(database) + try: + assert connection.execute( + "SELECT value FROM schema_metadata WHERE key = 'schema_version'" + ).fetchone() == ("6",) + finally: + connection.close() + + @pytest.mark.parametrize( "outbox_update", ( {"state": "projected", "attempt_count": 0, "last_error_code": "ERR"}, {"state": "quarantined", "attempt_count": 0, "last_error_code": ""}, {"state": "pending", "attempt_count": 1, "last_error_code": ""}, + {"state": "pending", "attempt_count": 0, "last_error_code": "ERR"}, { "state": "pending", "attempt_count": 0, @@ -1512,6 +1562,7 @@ def test_sqlite_store_bootstraps_true_legacy_v6_empty_outbox( "projected-error", "quarantined-missing-error", "pending-retry-missing-error", + "pending-initial-error", "non-utc-timestamp", ), ) From 48c81e57d9f18d5c914e3e004ab69d048bc8e485 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E5=93=80=E6=B4=9B=E8=8A=99?= Date: Sun, 30 Aug 2026 01:36:33 +0800 Subject: [PATCH 11/39] fix: bind legacy outbox rows to acceptances --- mcp-tools/orchestrator/store.py | 14 +++++++ mcp-tools/tests/test_runtime_composition.py | 42 +++++++++++++++++++++ 2 files changed, 56 insertions(+) diff --git a/mcp-tools/orchestrator/store.py b/mcp-tools/orchestrator/store.py index fcadc1e..b6974c1 100644 --- a/mcp-tools/orchestrator/store.py +++ b/mcp-tools/orchestrator/store.py @@ -7238,6 +7238,7 @@ def _validate_legacy_atlas_outbox_rows(cls, cursor: sqlite3.Cursor) -> None: row["state"] not in {"pending", "projected", "quarantined"} or not 0 <= row["attempt_count"] <= cls._MAX_ATLAS_OUTBOX_ATTEMPTS or row["ingestion_key"] != row["payload_hash"] + or row["acceptance_id"] != row["ingestion_key"] or ( row["state"] == "projected" and row["last_error_code"] != "" @@ -7258,6 +7259,19 @@ def _validate_legacy_atlas_outbox_rows(cls, cursor: sqlite3.Cursor) -> None: ) ): raise StoreError("legacy atlas outbox row is invalid") + acceptance = cursor.execute( + "SELECT acceptance_id, payload_hash, payload_json " + "FROM code_task_acceptances WHERE acceptance_id = ?", + (row["acceptance_id"],), + ).fetchone() + if ( + acceptance is None + or row["acceptance_id"] != acceptance["acceptance_id"] + or row["ingestion_key"] != acceptance["payload_hash"] + or row["acceptance_id"] != acceptance["payload_hash"] + or row["payload_json"] != acceptance["payload_json"] + ): + raise StoreError("legacy atlas outbox row is invalid") try: if ( row["created_at"] != _utc_timestamp(row["created_at"]) diff --git a/mcp-tools/tests/test_runtime_composition.py b/mcp-tools/tests/test_runtime_composition.py index 097445f..0908e2d 100644 --- a/mcp-tools/tests/test_runtime_composition.py +++ b/mcp-tools/tests/test_runtime_composition.py @@ -1616,6 +1616,48 @@ def test_sqlite_store_rejects_legacy_atlas_outbox_row_contract_drift( connection.close() +@pytest.mark.parametrize("binding_drift", ("identity", "payload")) +def test_sqlite_store_rejects_legacy_outbox_acceptance_binding_drift( + tmp_path: Path, binding_drift: str +) -> None: + database, acceptance_id, _ = _legacy_v10_atlas_outbox_database( + tmp_path, ingestion_key=f"sha256:{'a' * 64}" + ) + connection = sqlite3.connect(database) + try: + if binding_drift == "identity": + other_acceptance_id, _ = _insert_legacy_atlas_acceptance( + connection, suffix="b" + ) + connection.execute( + "UPDATE atlas_ingestion_outbox SET acceptance_id = ?", + (other_acceptance_id,), + ) + else: + connection.execute( + "UPDATE code_task_acceptances SET payload_json = ? " + "WHERE acceptance_id = ?", + ('{"different":true}', acceptance_id), + ) + connection.commit() + finally: + connection.close() + + with pytest.raises(StoreError, match="legacy atlas outbox row is invalid"): + SQLiteStore(database) + + connection = sqlite3.connect(database) + try: + assert connection.execute( + "SELECT value FROM schema_metadata WHERE key = 'schema_version'" + ).fetchone() == ("10",) + assert connection.execute( + "SELECT COUNT(*) FROM atlas_ingestion_outbox" + ).fetchone()[0] == 1 + finally: + connection.close() + + def test_sqlite_store_bootstraps_verified_legacy_empty_outbox( tmp_path: Path, ) -> None: From f5dfc7e0d12cb35e663fbec8ea88cef2acc8c805 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E5=93=80=E6=B4=9B=E8=8A=99?= Date: Sun, 30 Aug 2026 01:37:26 +0800 Subject: [PATCH 12/39] fix: enforce storage intent at fast lane boundaries --- .../scripts/authenticated_v5_planner.py | 32 +- .../scripts/authenticated_v5_projection.py | 199 +++++++++-- .../devkit_runtime/fastlane_host_intent.py | 76 ++-- mcp-tools/tests/test_storage_firewall.py | 336 ++++++++++-------- 4 files changed, 403 insertions(+), 240 deletions(-) diff --git a/mcp-tools/devkit_fastlane/scripts/authenticated_v5_planner.py b/mcp-tools/devkit_fastlane/scripts/authenticated_v5_planner.py index 56d1fdf..1fa4f94 100644 --- a/mcp-tools/devkit_fastlane/scripts/authenticated_v5_planner.py +++ b/mcp-tools/devkit_fastlane/scripts/authenticated_v5_planner.py @@ -85,10 +85,9 @@ def _storage_context_for_task( def _storage_value( context: Mapping[str, Any], - source_unit: Mapping[str, Any], field: str, ) -> object: - """Read one attested descriptor value from context or source unit.""" + """Read one attested descriptor value from the canonical context.""" candidates: list[Mapping[str, Any]] = [context] for container_name in ( @@ -105,31 +104,17 @@ def _storage_value( for candidate in candidates: if field in candidate: return candidate[field] - if field in source_unit: - return source_unit[field] - task = source_unit.get("task") - if isinstance(task, Mapping) and field in task: - return task[field] return None -def _storage_execution_context_hash( - context: Mapping[str, Any], source_unit: Mapping[str, Any] -) -> object: - value = context.get("execution_context_hash") - if value is None: - value = source_unit.get("execution_context_hash") - return value +def _storage_execution_context_hash(context: Mapping[str, Any]) -> object: + return context.get("execution_context_hash") def _storage_budget( source_unit: Mapping[str, Any], ) -> tuple[int, int]: budget = source_unit.get("storage_budget") - if budget is None: - task = source_unit.get("task") - if isinstance(task, Mapping): - budget = task.get("storage_budget") if not isinstance(budget, Mapping): raise ValueError(_STORAGE_POLICY_MISSING) requested_bytes = budget.get("bytes") @@ -158,7 +143,7 @@ def _make_storage_intent( source_hash = api._hash(source_plan_hash, "source_plan_hash") task_context = _storage_context_for_task(context, task_id) - context_hash = _storage_execution_context_hash(task_context, source_unit) + context_hash = _storage_execution_context_hash(task_context) try: context_hash = api._hash( context_hash, @@ -171,7 +156,7 @@ def _make_storage_intent( "schema": _STORAGE_TARGET_SCHEMA, "artifact_kind": "fastlane-task", **{ - field: _storage_value(task_context, source_unit, field) + field: _storage_value(task_context, field) for field in _STORAGE_DESCRIPTOR_FIELDS }, } @@ -225,7 +210,9 @@ def _validate_storage_intent( code = getattr(error, "code", _STORAGE_TARGET_KEY_INVALID) raise ValueError(code) from error task_context = _storage_context_for_task(context, task_id) - expected_context = _storage_execution_context_hash(task_context, source_unit) + expected_context = _storage_execution_context_hash(task_context) + if context is not None and expected_context is None: + raise ValueError(_STORAGE_POLICY_MISSING) if expected_context is not None: try: expected_context = api._hash( @@ -264,6 +251,9 @@ def _unit_storage_intent( source_plan_hash: str, context: Mapping[str, Any] | None, ) -> dict[str, object]: + # A pre-bound intent is not permission to invent a budget at compile time. + # Every compiler unit must carry the explicit request/source-unit budget. + _storage_budget(unit) supplied = unit.get("storage_intent") if supplied is None: return _make_storage_intent( diff --git a/mcp-tools/devkit_fastlane/scripts/authenticated_v5_projection.py b/mcp-tools/devkit_fastlane/scripts/authenticated_v5_projection.py index df7e808..0e1707f 100644 --- a/mcp-tools/devkit_fastlane/scripts/authenticated_v5_projection.py +++ b/mcp-tools/devkit_fastlane/scripts/authenticated_v5_projection.py @@ -19,30 +19,142 @@ "features_hash", "build_env_class", ) +_STORAGE_CONTEXT_FIELDS = frozenset( + {"execution_context_hash", *_STORAGE_DESCRIPTOR_FIELDS} +) +_STORAGE_REQUEST_FIELDS = frozenset({"storage_budgets", "storage_contexts"}) +_STORAGE_CONTEXT_CONTAINERS = ("storage_context", "storage_descriptor") + + +def _storage_record_for_task( + value: object, task_id: str, field: str +) -> Mapping[str, Any] | None: + """Return one explicitly keyed storage record without selecting a default.""" + + if value is None: + return None + if isinstance(value, Mapping): + direct = value.get(task_id) + if isinstance(direct, Mapping): + return direct + if value.get("task_id") == task_id: + return value + if any(key in value for key in _STORAGE_CONTEXT_FIELDS) or field == "budget": + raise ValueError(_STORAGE_POLICY_MISSING) + return None + if isinstance(value, Sequence) and not isinstance(value, (str, bytes, bytearray)): + matches = [ + item + for item in value + if isinstance(item, Mapping) and item.get("task_id") == task_id + ] + if len(matches) > 1: + raise ValueError(_STORAGE_TARGET_KEY_INVALID) + return matches[0] if matches else None + raise ValueError(_STORAGE_POLICY_MISSING) + + +def _storage_context_record(value: Mapping[str, Any]) -> Mapping[str, Any] | None: + """Collect only explicit context facts and reject conflicting duplicates.""" + + records: list[Mapping[str, Any]] = [] + direct = {key: value[key] for key in _STORAGE_CONTEXT_FIELDS if key in value} + if direct: + records.append(direct) + for name in _STORAGE_CONTEXT_CONTAINERS: + nested = value.get(name) + if isinstance(nested, Mapping): + records.append( + {key: nested[key] for key in _STORAGE_CONTEXT_FIELDS if key in nested} + ) + if not records: + return None + merged: dict[str, Any] = {} + for record in records: + for key, item in record.items(): + if key in merged and merged[key] != item: + raise ValueError(_STORAGE_TARGET_KEY_INVALID) + merged[key] = item + return merged + + +def _canonical_storage_context(value: Mapping[str, Any]) -> dict[str, Any]: + if set(value) != _STORAGE_CONTEXT_FIELDS: + raise ValueError(_STORAGE_POLICY_MISSING) + return dict(value) + + +def _merge_storage_contexts( + *records: Mapping[str, Any] | None, +) -> dict[str, Any] | None: + merged: dict[str, Any] = {} + for record in records: + if record is None: + continue + for key, item in record.items(): + if key in merged and merged[key] != item: + raise ValueError(_STORAGE_TARGET_KEY_INVALID) + merged[key] = item + return merged or None + + +def _storage_context_without_facts(value: object) -> object: + if not isinstance(value, Mapping): + return value + cleaned = dict(value) + for key in _STORAGE_CONTEXT_FIELDS: + cleaned.pop(key, None) + for name in _STORAGE_CONTEXT_CONTAINERS: + cleaned.pop(name, None) + return cleaned + + +def _storage_request_without_extensions( + value: Mapping[str, Any], api: Any +) -> None: + base = { + key: item for key, item in value.items() if key not in _STORAGE_REQUEST_FIELDS + } + api._exact_keys(base, api._FAST_LANE_REQUEST_FIELDS, "fast-lane request") + + +def _attach_storage_budget( + source_unit: Mapping[str, Any], + request: Mapping[str, Any], + *, + task_id: str, +) -> dict[str, Any]: + result = dict(source_unit) + source_budget = source_unit.get("storage_budget") + request_record = _storage_record_for_task( + request.get("storage_budgets"), task_id, "budget" + ) + request_budget: object = request_record + if isinstance(request_record, Mapping) and "storage_budget" in request_record: + request_budget = request_record["storage_budget"] + if source_budget is not None and request_budget is not None: + if source_budget != request_budget: + raise ValueError(_STORAGE_TARGET_KEY_INVALID) + elif source_budget is None: + source_budget = request_budget + if source_budget is None: + raise ValueError(_STORAGE_POLICY_MISSING) + result["storage_budget"] = source_budget + return result def _storage_context_value( - context: Mapping[str, Any], source_unit: Mapping[str, Any], field: str + context: Mapping[str, Any], field: str ) -> object: candidates: list[Mapping[str, Any]] = [context] - for container_name in ( - "storage_descriptor", - "target_descriptor", - "storage_target", - "build_context", - "bootstrap_plan", - "storage", - ): + for container_name in _STORAGE_CONTEXT_CONTAINERS: container = context.get(container_name) if isinstance(container, Mapping): candidates.insert(0, container) for candidate in candidates: if field in candidate: return candidate[field] - task = source_unit.get("task") - if isinstance(task, Mapping) and field in task: - return task[field] - return source_unit.get(field) + return None def _make_storage_intent( @@ -54,8 +166,6 @@ def _make_storage_intent( source_plan_hash: str, ) -> dict[str, object]: context_hash = context.get("execution_context_hash") - if context_hash is None: - context_hash = source_unit.get("execution_context_hash") try: context_hash = api._hash( context_hash, @@ -64,10 +174,6 @@ def _make_storage_intent( except Exception as error: raise ValueError(_STORAGE_POLICY_MISSING) from error budget = source_unit.get("storage_budget") - if budget is None: - task = source_unit.get("task") - if isinstance(task, Mapping): - budget = task.get("storage_budget") if not isinstance(budget, Mapping): raise ValueError(_STORAGE_POLICY_MISSING) requested_bytes = budget.get("bytes") @@ -85,7 +191,7 @@ def _make_storage_intent( "schema": _STORAGE_TARGET_SCHEMA, "artifact_kind": "fastlane-task", **{ - field: _storage_context_value(context, source_unit, field) + field: _storage_context_value(context, field) for field in _STORAGE_DESCRIPTOR_FIELDS }, } @@ -166,7 +272,7 @@ def project_units_with_waves( """ candidate = api._mapping(request, "fast-lane request") - api._exact_keys(candidate, api._FAST_LANE_REQUEST_FIELDS, "fast-lane request") + _storage_request_without_extensions(candidate, api) if ( candidate.get("schema") != "team-efficiency/fast-lane-request-v1" or len(api._json_bytes(candidate)) > api.MAX_MANIFEST_INPUT_BYTES @@ -174,9 +280,26 @@ def project_units_with_waves( ): raise ValueError("authenticated V5 raw request is unsupported") source_plan = api.decompose(candidate["work_package"]) + raw_context_by_task: dict[str, Mapping[str, Any]] = {} + raw_execution_contexts = candidate["execution_contexts"] + if isinstance(raw_execution_contexts, Sequence) and not isinstance( + raw_execution_contexts, (str, bytes, bytearray) + ): + for raw_context in raw_execution_contexts: + if not isinstance(raw_context, Mapping): + continue + raw_task_id = raw_context.get("task_id") + if not isinstance(raw_task_id, str): + continue + record = _storage_context_record(raw_context) + if record is not None: + existing = raw_context_by_task.get(raw_task_id) + raw_context_by_task[raw_task_id] = _merge_storage_contexts( + existing, record + ) or {} + source_plan_hash = api._sha256_json(source_plan) if source_plan.get("status") != "planned": raise ValueError("authenticated V5 source plan is not schedulable") - source_plan_hash = api._sha256_json(source_plan) project_binding = api._validated_project_binding(candidate["project_binding"]) project_authority = api._mapping( source_plan.get("project_authority"), "source plan.project_authority" @@ -197,7 +320,10 @@ def project_units_with_waves( candidate["target_gates"], source_plan ) execution_contexts, read_contexts = api._validated_fast_lane_contexts( - candidate["execution_contexts"], + [ + _storage_context_without_facts(item) + for item in candidate["execution_contexts"] + ], candidate["read_contexts"], source_plan, candidate["scheduler_state"], @@ -215,7 +341,31 @@ def project_units_with_waves( if remediation is not None or state["phase"] != "execution": raise ValueError("authenticated V5 scheduler phase is unsupported") - units_by_task = api._fast_lane_unit_index(source_plan) + storage_context_by_task: dict[str, dict[str, Any]] = {} + for normalized_context in execution_contexts: + task_id = str(normalized_context["task_id"]) + request_context = _storage_record_for_task( + candidate.get("storage_contexts"), task_id, "context" + ) + raw_context = raw_context_by_task.get(task_id) + merged = _merge_storage_contexts(raw_context, request_context) + if merged is None: + raise ValueError(_STORAGE_POLICY_MISSING) + storage_context_by_task[task_id] = _canonical_storage_context(merged) + execution_contexts = [ + {**context, **storage_context_by_task[str(context["task_id"])]} + for context in execution_contexts + ] + + raw_units_by_task = api._fast_lane_unit_index(source_plan) + units_by_task = { + task_id: _attach_storage_budget( + source_unit, + candidate, + task_id=task_id, + ) + for task_id, source_unit in raw_units_by_task.items() + } if not 1 <= len(units_by_task) <= 16: raise ValueError("authenticated V5 source plan exceeds the bounded queue") config_capacity = source_plan.get("capacity") @@ -404,6 +554,7 @@ def project_slice(task_ids: Sequence[str]) -> list[dict[str, Any]]: "dispatch_order": dispatch_order, "index_context_hash": index_hash, "workflow_id_hash": workflow_hash, + "storage_budget": source_unit["storage_budget"], "storage_intent": storage_intent, } ) diff --git a/mcp-tools/devkit_runtime/fastlane_host_intent.py b/mcp-tools/devkit_runtime/fastlane_host_intent.py index 4990083..860b315 100644 --- a/mcp-tools/devkit_runtime/fastlane_host_intent.py +++ b/mcp-tools/devkit_runtime/fastlane_host_intent.py @@ -62,10 +62,6 @@ "assignment_binding_hash", } ) -_ASSIGNMENT_STORAGE_KEYS: Final = _ASSIGNMENT_KEYS | { - "storage_intent", - "execution_context_hash", -} _PREDECESSOR_KEYS: Final = frozenset( { "schema", @@ -363,12 +359,14 @@ def validate_host_execution_intent( def parse_host_execution_intent( candidate: object, -) -> ParsedHostExecutionIntent | Literal["NO_SAFE_WORK"]: +) -> ParsedHostExecutionIntent | StorageIntentError | Literal["NO_SAFE_WORK"]: """Parse a candidate structurally; the result is never an authorization.""" try: parsed = _parse(candidate) - except Exception: + except StorageIntentError as error: + return error + except (AttributeError, IndexError, KeyError, TypeError, ValueError, UnicodeError): return NO_SAFE_WORK return parsed if parsed is not None else NO_SAFE_WORK @@ -407,11 +405,21 @@ def classify_host_scheduler_topology( def _parse(candidate: object) -> ParsedHostExecutionIntent | None: - root = _bound_mapping_variant( - candidate, - (_ROOT_KEYS, _ROOT_STORAGE_KEYS), - "intent_hash", - ) + if isinstance(candidate, dict) and candidate.get("schema") == _SCHEMA: + assignment_candidate = candidate.get("assignment") + if ( + "storage_intent" not in candidate + or "execution_context_hash" not in candidate + or ( + isinstance(assignment_candidate, dict) + and ( + "storage_intent" in assignment_candidate + or "execution_context_hash" in assignment_candidate + ) + ) + ): + raise StorageIntentError(STORAGE_TARGET_KEY_INVALID) + root = _bound_mapping(candidate, _ROOT_STORAGE_KEYS, "intent_hash") if root is None or _text(root, "schema") != _SCHEMA: return None @@ -427,10 +435,8 @@ def _parse(candidate: object) -> ParsedHostExecutionIntent | None: ): return None - assignment = _bound_mapping_variant( - root["assignment"], - (_ASSIGNMENT_KEYS, _ASSIGNMENT_STORAGE_KEYS), - "assignment_binding_hash", + assignment = _bound_mapping( + root["assignment"], _ASSIGNMENT_KEYS, "assignment_binding_hash" ) route = _bound_mapping(root["route"], _ROUTE_KEYS, "route_binding_hash") packets = _bound_mapping(root["packets"], _PACKET_KEYS, "packet_binding_hash") @@ -510,30 +516,22 @@ def _parse(candidate: object) -> ParsedHostExecutionIntent | None: active_lease_set_hash, ) = validated_predecessor - storage_intent: StorageIntent | None = None - execution_context_hash: str | None = None - raw_storage_intent = root.get("storage_intent") - if raw_storage_intent is None and assignment is not None: - raw_storage_intent = assignment.get("storage_intent") - if raw_storage_intent is not None: - try: - storage_intent = parse_storage_intent(raw_storage_intent) - except StorageIntentError: - raise - except Exception as error: - raise StorageIntentError(STORAGE_TARGET_KEY_INVALID) from error - raw_execution_context_hash = root.get("execution_context_hash") - if raw_execution_context_hash is None and assignment is not None: - raw_execution_context_hash = assignment.get("execution_context_hash") - if not _is_hash_value(raw_execution_context_hash): - raise StorageIntentError(STORAGE_TARGET_KEY_INVALID) - execution_context_hash = cast(str, raw_execution_context_hash) - if ( - storage_intent.task_id != task_id - or storage_intent.plan_binding != source_plan_hash - or storage_intent.context_hash != execution_context_hash - ): - raise StorageIntentError(STORAGE_TARGET_KEY_INVALID) + try: + storage_intent = parse_storage_intent(root["storage_intent"]) + except StorageIntentError: + raise + except (AttributeError, KeyError, TypeError, ValueError, UnicodeError) as error: + raise StorageIntentError(STORAGE_TARGET_KEY_INVALID) from error + raw_execution_context_hash = root["execution_context_hash"] + if not _is_hash_value(raw_execution_context_hash): + raise StorageIntentError(STORAGE_TARGET_KEY_INVALID) + execution_context_hash = cast(str, raw_execution_context_hash) + if ( + storage_intent.task_id != task_id + or storage_intent.plan_binding != source_plan_hash + or storage_intent.context_hash != execution_context_hash + ): + raise StorageIntentError(STORAGE_TARGET_KEY_INVALID) capability_facts = _validate_capability_facts(root["capability_facts"]) if capability_facts is None or not _has_candidate_capability_claim( diff --git a/mcp-tools/tests/test_storage_firewall.py b/mcp-tools/tests/test_storage_firewall.py index a94472e..4b3599d 100644 --- a/mcp-tools/tests/test_storage_firewall.py +++ b/mcp-tools/tests/test_storage_firewall.py @@ -1,7 +1,13 @@ from __future__ import annotations +import copy import hashlib +import importlib.util import json +import os +from pathlib import Path + +import pytest def _canonical_hash(value: object) -> str: @@ -92,101 +98,6 @@ def test_storage_intent_rejects_isolated_surrogate_with_stable_code() -> None: raise AssertionError("invalid surrogate was accepted") -class _StorageBindingRoutingCore: - def load_policy_v5(self) -> dict[str, object]: - return {} - - def policy_hash_v5(self, policy: object) -> str: - del policy - return _canonical_hash({"policy": "storage-binding"}) - - def _normalise_request_v5( - self, request: dict[str, object], policy: object - ) -> dict[str, object]: - del policy - return request - - def v5_request_binding_hash(self, request: object) -> str: - return _canonical_hash(request) - - def route_v5( - self, request: dict[str, object], *, policy: object - ) -> dict[str, object]: - del policy - task = request["task"] - assert isinstance(task, dict) - return { - "schema": "2718lab-devkit/fastlane-routing-result-v5", - "status": "resolved", - "task_id": task["task_id"], - "route": { - "model": "gpt-5.6-luna", - "effort": "max", - "inherit_current_session_model": False, - }, - } - - -class _StorageBindingApi: - def __init__(self) -> None: - self.core = _StorageBindingRoutingCore() - - def _mapping(self, value: object, field: str) -> dict[str, object]: - assert isinstance(value, dict), field - return value - - def _task_id(self, value: object, field: str) -> str: - assert isinstance(value, str), field - return value - - def _normalised_scopes(self, value: object, field: str = "scope") -> list[str]: - assert isinstance(value, list), field - return value - - def _canonical_json(self, value: object) -> str: - return json.dumps(value, ensure_ascii=True, sort_keys=True, separators=(",", ":")) - - def _sha256_json(self, value: object) -> str: - return _canonical_hash(value) - - def _hash(self, value: object, field: str) -> str: - assert isinstance(value, str), field - return value - - def _text(self, value: object, field: str, *, maximum: int) -> str: - assert isinstance(value, str) and len(value) <= maximum, field - return value - - def _exact_keys( - self, value: dict[str, object], expected: frozenset[str], field: str - ) -> None: - assert set(value) == expected, field - - def _fast_lane_routing_core(self) -> _StorageBindingRoutingCore: - return self.core - - -def _storage_binding_unit(task_id: str, dispatch_order: int) -> dict[str, object]: - return { - "task": { - "schema": "2718lab-devkit/task-routing-profile-v5", - "task_id": task_id, - "role": "execution", - "access": "workspace_write", - "write_scope_count": 1, - "overlap_risk": "none", - "overlap_count": 0, - }, - "dependency_state": {"task_id": task_id}, - "write_scope": [f"src/{task_id}.py"], - "concurrency_mode": "parallel", - "dispatch_order": dispatch_order, - "index_context_hash": "sha256:" + "b" * 64, - "workflow_id_hash": "sha256:" + "c" * 64, - "storage_budget": {"bytes": 4096, "files": 8}, - } - - def _storage_binding_context() -> dict[str, object]: return { "execution_context_hash": "sha256:" + "4" * 64, @@ -201,77 +112,190 @@ def _storage_binding_context() -> dict[str, object]: } -def _storage_binding_request( - api: _StorageBindingApi, unit: dict[str, object], source_plan_hash: str -) -> tuple[dict[str, object], dict[str, object]]: - task = unit["task"] - assert isinstance(task, dict) - request: dict[str, object] = { - "task": task, - "scheduler_facts": {"route_epoch": 1}, - "child_route_attestation": None, - } - binding_hash = api.core.v5_request_binding_hash(request) - attestation: dict[str, object] = { - "request_binding_hash": binding_hash, - "attestation": { - "status": "attested", - "request_binding_hash": binding_hash, +def _host_storage_intent( + *, task_id: str, plan_binding: str, context_hash: str +) -> dict[str, object]: + context = _storage_binding_context() + descriptor = { + "schema": "2718lab.storage.target.v1", + "artifact_kind": "fastlane-task", + **{ + key: value + for key, value in context.items() + if key != "execution_context_hash" }, } - attestation_payload = attestation["attestation"] - assert isinstance(attestation_payload, dict) - attestation_payload["attestation_hash"] = _canonical_hash( + intent = { + "schema": "2718lab.storage.intent.v1", + "task_id": task_id, + "plan_binding": plan_binding, + "context_hash": context_hash, + "requested_bytes": 4096, + "requested_files": 8, + "target_descriptor": descriptor, + } + intent["storage_intent_hash"] = _canonical_hash( { - key: value - for key, value in attestation_payload.items() - if key != "attestation_hash" + key: intent[key] + for key in ( + "target_descriptor", + "task_id", + "plan_binding", + "context_hash", + "requested_bytes", + "requested_files", + ) } ) - item = { - "task_id": task["task_id"], - "request_binding_hash": binding_hash, - "attestation": attestation_payload, - } - del source_plan_hash - return request, item + return intent -def test_every_fastlane_wave_carries_plan_context_bound_storage_intent() -> None: - from devkit_fastlane.scripts.authenticated_v5_planner import compile_skeletons +def test_host_intent_requires_one_canonical_storage_binding_and_typed_failures() -> None: + from test_fastlane_host_intent import _intent, _with_binding + from devkit_runtime.fastlane_host_intent import ( + NO_SAFE_WORK, + STORAGE_TARGET_KEY_INVALID, + StorageIntentError, + parse_host_execution_intent, + validate_host_execution_intent, + ) - api = _StorageBindingApi() - plan_hash = "sha256:" + "a" * 64 - context = _storage_binding_context() - first_unit = _storage_binding_unit("task-01", 0) - successor_unit = _storage_binding_unit("task-02", 1) + legacy = _intent() + legacy_result = parse_host_execution_intent(legacy) + assert isinstance(legacy_result, StorageIntentError) + assert legacy_result.code == STORAGE_TARGET_KEY_INVALID + assert validate_host_execution_intent(legacy) is NO_SAFE_WORK + + candidate = _intent() + task_id = candidate["assignment"]["predecessor"]["task_id"] + source_plan_hash = candidate["source_plan_hash"] + context_hash = "sha256:" + "4" * 64 + storage_intent = _host_storage_intent( + task_id=task_id, + plan_binding=source_plan_hash, + context_hash=context_hash, + ) + candidate["storage_intent"] = storage_intent + candidate["execution_context_hash"] = context_hash + candidate["intent_hash"] = _canonical_hash( + {key: value for key, value in candidate.items() if key != "intent_hash"} + ) + assert parse_host_execution_intent(candidate).storage_intent is not None - first_request, first_attestation = _storage_binding_request( - api, first_unit, plan_hash + duplicate = copy.deepcopy(candidate) + duplicate["assignment"]["storage_intent"] = storage_intent + duplicate["assignment"] = _with_binding( + duplicate["assignment"], "assignment_binding_hash" ) - successor_request, successor_attestation = _storage_binding_request( - api, successor_unit, plan_hash + duplicate["intent_hash"] = _canonical_hash( + {key: value for key, value in duplicate.items() if key != "intent_hash"} ) - first = compile_skeletons( - api, - [first_unit], - source_plan_hash=plan_hash, - routing_requests=[first_request], - attestation_items=[first_attestation], - context=context, + duplicate_result = parse_host_execution_intent(duplicate) + conflict = copy.deepcopy(candidate) + conflict["storage_intent"] = _host_storage_intent( + task_id=task_id, + plan_binding=source_plan_hash, + context_hash="sha256:" + "3" * 64, ) - successor = compile_skeletons( - api, - [successor_unit], - source_plan_hash=plan_hash, - routing_requests=[successor_request], - attestation_items=[successor_attestation], - context=context, + conflict = _with_binding(conflict, "intent_hash") + conflict_result = parse_host_execution_intent(conflict) + for result in (duplicate_result, conflict_result): + assert isinstance(result, StorageIntentError) + assert result.code == STORAGE_TARGET_KEY_INVALID + + +def _real_storage_request() -> tuple[ + object, + object, + dict[str, object], + dict[str, object], + dict[str, object], + str | None, +]: + test_module_path = ( + Path(__file__).resolve().parents[1] + / "devkit_fastlane" + / "tests" + / "test_team_efficiency.py" ) - - for wave in (first["assignment_skeletons"], successor["assignment_skeletons"]): - for assignment in wave: - intent = assignment["storage_intent"] - assert intent["task_id"] == assignment["task_id"] - assert intent["plan_binding"] == plan_hash - assert intent["context_hash"] == context["execution_context_hash"] + spec = importlib.util.spec_from_file_location( + "storage_test_team_efficiency_tests", test_module_path + ) + assert spec is not None and spec.loader is not None + tests_module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(tests_module) + + previous_task_temp = os.environ.get("CODEX_TASK_TEMP") + if previous_task_temp is None: + os.environ["CODEX_TASK_TEMP"] = str( + Path(__file__).resolve().parents[3] + / ".codex-task-temp" + ) + fixture = tests_module.TeamEfficiencyTests("runTest") + fixture.setUp() + helper = tests_module.load_efficiency() + request = fixture.fast_lane_request(helper) + task_ids = [item["task_id"] for item in request["execution_contexts"]] + descriptor = { + key: value + for key, value in _storage_binding_context().items() + if key != "execution_context_hash" + } + for context in request["execution_contexts"]: + context.update( + { + **descriptor, + "execution_context_hash": _canonical_hash( + {"storage_context": context["task_id"]} + ), + } + ) + request["storage_budgets"] = { + task_id: {"bytes": 4096 + index, "files": 8 + index} + for index, task_id in enumerate(task_ids) + } + host_status = fixture.fast_lane_host_status(helper, request) + route_request = host_status["routing_context"]["routes"][0]["request"] + host = copy.deepcopy(route_request["host_capabilities"]) + host["models"] = [ + {**model, "efforts": sorted(model["efforts"])} for model in host["models"] + ] + scheduler = route_request["scheduler_facts"] + return fixture, helper, request, host, scheduler, previous_task_temp + + +def test_real_prepare_entry_binds_initial_successor_and_missing_facts_fail_closed() -> None: + fixture, helper, request, host, scheduler, previous_task_temp = _real_storage_request() + try: + prepared = helper.prepare_authenticated_v5_routing_from_request( + request, + index_context_hash=helper._sha256_json({"index": "storage-real-entry"}), + host_capabilities=host, + scheduler_facts=scheduler, + ) + for wave in (prepared["units"], prepared["remaining_units"]): + assert wave + for unit in wave: + intent = unit["storage_intent"] + assert intent["task_id"] == unit["task"]["task_id"] + assert intent["plan_binding"] == prepared["source_plan_hash"] + assert intent["context_hash"] == _canonical_hash( + {"storage_context": unit["task"]["task_id"]} + ) + + missing = copy.deepcopy(request) + for context in missing["execution_contexts"]: + context.pop("execution_context_hash") + with pytest.raises(ValueError, match="STORAGE_POLICY_MISSING"): + helper.prepare_authenticated_v5_routing_from_request( + missing, + index_context_hash=helper._sha256_json({"index": "storage-real-entry"}), + host_capabilities=host, + scheduler_facts=scheduler, + ) + finally: + fixture.tearDown() + if previous_task_temp is None: + os.environ.pop("CODEX_TASK_TEMP", None) + else: + os.environ["CODEX_TASK_TEMP"] = previous_task_temp From a028f384a9d79bd5df2b210b21450844b878a4b4 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E5=93=80=E6=B4=9B=E8=8A=99?= Date: Sun, 30 Aug 2026 01:52:06 +0800 Subject: [PATCH 13/39] fix: verify legacy acceptance content addresses --- mcp-tools/orchestrator/store.py | 54 ++++++++++++++++----- mcp-tools/tests/test_runtime_composition.py | 53 +++++++++++++++----- 2 files changed, 83 insertions(+), 24 deletions(-) diff --git a/mcp-tools/orchestrator/store.py b/mcp-tools/orchestrator/store.py index b6974c1..51528ad 100644 --- a/mcp-tools/orchestrator/store.py +++ b/mcp-tools/orchestrator/store.py @@ -7259,20 +7259,48 @@ def _validate_legacy_atlas_outbox_rows(cls, cursor: sqlite3.Cursor) -> None: ) ): raise StoreError("legacy atlas outbox row is invalid") - acceptance = cursor.execute( - "SELECT acceptance_id, payload_hash, payload_json " - "FROM code_task_acceptances WHERE acceptance_id = ?", - (row["acceptance_id"],), - ).fetchone() - if ( - acceptance is None - or row["acceptance_id"] != acceptance["acceptance_id"] - or row["ingestion_key"] != acceptance["payload_hash"] - or row["acceptance_id"] != acceptance["payload_hash"] - or row["payload_json"] != acceptance["payload_json"] - ): - raise StoreError("legacy atlas outbox row is invalid") try: + acceptance = cursor.execute( + """ + SELECT acceptance_id, workflow_id, code_task_id, + code_task_version, input_snapshot_id, + output_snapshot_id, indexed_diff_hash, intent_id, + language, framework, payload_json, payload_hash + FROM code_task_acceptances + WHERE acceptance_id = ? + """, + (row["acceptance_id"],), + ).fetchone() + if acceptance is None: + raise ValueError("referenced acceptance is missing") + canonical_acceptance_payload = ( + cls._canonical_code_task_acceptance_payload( + workflow_id=acceptance["workflow_id"], + task_id=acceptance["code_task_id"], + task_version=acceptance["code_task_version"], + input_snapshot_id=acceptance["input_snapshot_id"], + output_snapshot_id=acceptance["output_snapshot_id"], + indexed_diff_hash=acceptance["indexed_diff_hash"], + intent_id=acceptance["intent_id"], + language=acceptance["language"], + framework=acceptance["framework"], + ) + ) + canonical_acceptance_hash = _payload_hash( + canonical_acceptance_payload + ) + if ( + row["acceptance_id"] != acceptance["acceptance_id"] + or row["ingestion_key"] != acceptance["payload_hash"] + or row["acceptance_id"] != acceptance["payload_hash"] + or row["payload_json"] != acceptance["payload_json"] + or acceptance["payload_json"] != canonical_acceptance_payload + or acceptance["payload_hash"] != canonical_acceptance_hash + or acceptance["acceptance_id"] != canonical_acceptance_hash + or row["payload_hash"] != _payload_hash(row["payload_json"]) + or row["payload_hash"] != canonical_acceptance_hash + ): + raise ValueError("legacy acceptance content address mismatch") if ( row["created_at"] != _utc_timestamp(row["created_at"]) or row["updated_at"] != _utc_timestamp(row["updated_at"]) diff --git a/mcp-tools/tests/test_runtime_composition.py b/mcp-tools/tests/test_runtime_composition.py index 0908e2d..4bd5857 100644 --- a/mcp-tools/tests/test_runtime_composition.py +++ b/mcp-tools/tests/test_runtime_composition.py @@ -13,7 +13,7 @@ from devkit_runtime.config import RuntimeConfig, RuntimeConfigError from devkit_runtime.relay_runtime import RelayRuntime from devkit_runtime.uow import RuntimeAdapterFactories, RuntimeUnitOfWork -from orchestrator.store import SQLiteStore, StoreError +from orchestrator.store import SQLiteStore, StoreError, _payload_hash def test_runtime_config_load_prefers_plugin_data_without_writing( @@ -1182,7 +1182,21 @@ def _insert_legacy_atlas_acceptance( timestamp = "2026-08-09T00:00:00+00:00" workflow_id = "legacy-workflow" task_id = f"legacy-task-{suffix}" - acceptance_id = f"sha256:{suffix * 64}" + input_snapshot_id = f"sha256:{'e' * 64}" + output_snapshot_id = f"sha256:{'f' * 64}" + indexed_diff_hash = f"sha256:{'0' * 64}" + payload_json = SQLiteStore._canonical_code_task_acceptance_payload( + workflow_id=workflow_id, + task_id=task_id, + task_version=1, + input_snapshot_id=input_snapshot_id, + output_snapshot_id=output_snapshot_id, + indexed_diff_hash=indexed_diff_hash, + intent_id="legacy", + language="python", + framework="pytest", + ) + acceptance_id = _payload_hash(payload_json) connection.execute( """ INSERT INTO workflows ( @@ -1235,13 +1249,13 @@ def _insert_legacy_atlas_acceptance( workflow_id, task_id, 1, - f"sha256:{'e' * 64}", - f"sha256:{'f' * 64}", - f"sha256:{'0' * 64}", + input_snapshot_id, + output_snapshot_id, + indexed_diff_hash, "legacy", "python", "pytest", - "{}", + payload_json, acceptance_id, timestamp, ), @@ -1262,6 +1276,11 @@ def _legacy_v6_atlas_outbox_database( acceptance_id, timestamp = _insert_legacy_atlas_acceptance( connection, suffix="a" ) + payload_json = connection.execute( + "SELECT payload_json FROM code_task_acceptances WHERE acceptance_id = ?", + (acceptance_id,), + ).fetchone()[0] + outbox_ingestion_key = acceptance_id if ingestion_key is not None else None connection.execute( "INSERT INTO schema_metadata (key, value) VALUES (?, ?)", ("schema_version", "6"), @@ -1274,9 +1293,9 @@ def _legacy_v6_atlas_outbox_database( ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?) """, ( - ingestion_key, + outbox_ingestion_key, acceptance_id, - "{}", + payload_json, acceptance_id, "pending", 0, @@ -1304,6 +1323,11 @@ def _legacy_v10_atlas_outbox_database( acceptance_id, timestamp = _insert_legacy_atlas_acceptance( connection, suffix="a" ) + payload_json = connection.execute( + "SELECT payload_json FROM code_task_acceptances WHERE acceptance_id = ?", + (acceptance_id,), + ).fetchone()[0] + outbox_ingestion_key = acceptance_id if ingestion_key is not None else None connection.execute("DROP TABLE schema_metadata") connection.execute( """ @@ -1331,9 +1355,9 @@ def _legacy_v10_atlas_outbox_database( ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?) """, ( - ingestion_key, + outbox_ingestion_key, acceptance_id, - "{}", + payload_json, acceptance_id, "pending", 0, @@ -1356,6 +1380,7 @@ def test_sqlite_store_migrates_v10_outbox_to_reject_null_ingestion_keys( database, acceptance_id, timestamp = _legacy_v10_atlas_outbox_database( tmp_path, ingestion_key=ingestion_key ) + ingestion_key = acceptance_id store = SQLiteStore(database) try: @@ -1616,7 +1641,7 @@ def test_sqlite_store_rejects_legacy_atlas_outbox_row_contract_drift( connection.close() -@pytest.mark.parametrize("binding_drift", ("identity", "payload")) +@pytest.mark.parametrize("binding_drift", ("identity", "payload-both")) def test_sqlite_store_rejects_legacy_outbox_acceptance_binding_drift( tmp_path: Path, binding_drift: str ) -> None: @@ -1639,6 +1664,10 @@ def test_sqlite_store_rejects_legacy_outbox_acceptance_binding_drift( "WHERE acceptance_id = ?", ('{"different":true}', acceptance_id), ) + connection.execute( + "UPDATE atlas_ingestion_outbox SET payload_json = ?", + ('{"different":true}',), + ) connection.commit() finally: connection.close() @@ -1756,6 +1785,7 @@ def _legacy_v10_incomplete_atlas_outbox_database( database, acceptance_id, timestamp = _legacy_v10_atlas_outbox_database( tmp_path, ingestion_key=ingestion_key ) + ingestion_key = acceptance_id connection = sqlite3.connect(database) try: connection.execute("DROP TABLE atlas_ingestion_outbox") @@ -1841,6 +1871,7 @@ def _malformed_v11_nullable_atlas_outbox_database( database, acceptance_id, _ = _legacy_v10_atlas_outbox_database( tmp_path, ingestion_key=ingestion_key ) + ingestion_key = acceptance_id connection = sqlite3.connect(database) try: connection.execute( From 686b57df9a58be00bf907de2187eb1ee80586813 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E5=93=80=E6=B4=9B=E8=8A=99?= Date: Sun, 30 Aug 2026 01:53:18 +0800 Subject: [PATCH 14/39] fix: bind storage intent to routing projection --- .../scripts/authenticated_v5_planner.py | 39 +++++++++++++ .../scripts/authenticated_v5_projection.py | 54 ++++++++++++++---- mcp-tools/tests/test_storage_firewall.py | 56 +++++++++++++++++++ 3 files changed, 137 insertions(+), 12 deletions(-) diff --git a/mcp-tools/devkit_fastlane/scripts/authenticated_v5_planner.py b/mcp-tools/devkit_fastlane/scripts/authenticated_v5_planner.py index 1fa4f94..41be7b2 100644 --- a/mcp-tools/devkit_fastlane/scripts/authenticated_v5_planner.py +++ b/mcp-tools/devkit_fastlane/scripts/authenticated_v5_planner.py @@ -46,6 +46,18 @@ "features_hash", "build_env_class", ) +_PROFILE_EVIDENCE_SCHEMA = "team-efficiency/fast-lane-v5-profile-evidence-v1" +_PROFILE_UNIT_FIELDS = ( + "task", + "dependency_state", + "write_scope", + "concurrency_mode", + "dispatch_order", + "index_context_hash", + "workflow_id_hash", + "storage_budget", + "storage_intent", +) def _storage_context_for_task( @@ -273,6 +285,21 @@ def _unit_storage_intent( ) +def _routing_profile_material( + source_plan_hash: str, unit: Mapping[str, Any] +) -> dict[str, Any]: + task = dict(unit["task"]) + task.pop("profile_evidence_hash", None) + return { + "schema": _PROFILE_EVIDENCE_SCHEMA, + "source_plan_hash": source_plan_hash, + "unit": { + field: task if field == "task" else unit[field] + for field in _PROFILE_UNIT_FIELDS + }, + } + + def owned_scope_hash(api: Any, task_id: object, write_scope: object) -> str: normalized_task_id = api._task_id(task_id, "authenticated V5 task_id") normalized_scope = api._normalised_scopes( @@ -546,6 +573,18 @@ def compile_skeletons( attestation = attestation_by_task.get(task_id) if request is None or attestation is None: raise ValueError("authenticated V5 routing response is incomplete") + if request["task"] != unit["task"]: + raise ValueError("authenticated V5 routing profile binding is invalid") + profile_hash = api._sha256_json( + _routing_profile_material( + source_hash, + {**unit, "storage_intent": storage_intent}, + ) + ) + if not hmac.compare_digest( + str(unit["task"].get("profile_evidence_hash")), profile_hash + ): + raise ValueError("authenticated V5 routing profile binding is invalid") attested_request = {**request, "child_route_attestation": attestation} try: normalized_request = core._normalise_request_v5(attested_request, policy) diff --git a/mcp-tools/devkit_fastlane/scripts/authenticated_v5_projection.py b/mcp-tools/devkit_fastlane/scripts/authenticated_v5_projection.py index 0e1707f..34eb80e 100644 --- a/mcp-tools/devkit_fastlane/scripts/authenticated_v5_projection.py +++ b/mcp-tools/devkit_fastlane/scripts/authenticated_v5_projection.py @@ -24,6 +24,33 @@ ) _STORAGE_REQUEST_FIELDS = frozenset({"storage_budgets", "storage_contexts"}) _STORAGE_CONTEXT_CONTAINERS = ("storage_context", "storage_descriptor") +_PROFILE_EVIDENCE_SCHEMA = "team-efficiency/fast-lane-v5-profile-evidence-v1" +_PROFILE_UNIT_FIELDS = ( + "task", + "dependency_state", + "write_scope", + "concurrency_mode", + "dispatch_order", + "index_context_hash", + "workflow_id_hash", + "storage_budget", + "storage_intent", +) + + +def _routing_profile_material( + source_plan_hash: str, unit: Mapping[str, Any] +) -> dict[str, Any]: + task = dict(unit["task"]) + task.pop("profile_evidence_hash", None) + return { + "schema": _PROFILE_EVIDENCE_SCHEMA, + "source_plan_hash": source_plan_hash, + "unit": { + field: task if field == "task" else unit[field] + for field in _PROFILE_UNIT_FIELDS + }, + } def _storage_record_for_task( @@ -499,17 +526,6 @@ def project_slice(task_ids: Sequence[str]) -> list[dict[str, Any]]: "Terra Max": "high", "Sol High": "critical", }.get(str(source_unit.get("recommended_route")), "critical") - profile_material = { - "schema": "team-efficiency/fast-lane-v5-profile-evidence-v1", - "source_plan_hash": source_plan_hash, - "source_unit": source_unit, - "target_gates": target, - "dependency_state": dependency_state, - # Keep the complete intent in the projection preimage. A - # later dispatch binding therefore cannot omit storage - # semantics while retaining the same profile evidence hash. - "storage_intent": storage_intent, - } task = { "schema": "2718lab-devkit/task-routing-profile-v5", "task_id": task_id, @@ -543,8 +559,22 @@ def project_slice(task_ids: Sequence[str]) -> list[dict[str, Any]]: "narrow_decoupling_eligible": False, "strike": None, "gate_matrix_hash": api._sha256_json(target), - "profile_evidence_hash": api._sha256_json(profile_material), } + profile_material = _routing_profile_material( + source_plan_hash, + { + "task": task, + "dependency_state": dependency_state, + "write_scope": write_scope, + "concurrency_mode": "parallel", + "dispatch_order": dispatch_order, + "index_context_hash": index_hash, + "workflow_id_hash": workflow_hash, + "storage_budget": source_unit["storage_budget"], + "storage_intent": storage_intent, + }, + ) + task["profile_evidence_hash"] = api._sha256_json(profile_material) projected_slice.append( { "task": task, diff --git a/mcp-tools/tests/test_storage_firewall.py b/mcp-tools/tests/test_storage_firewall.py index 4b3599d..54475df 100644 --- a/mcp-tools/tests/test_storage_firewall.py +++ b/mcp-tools/tests/test_storage_firewall.py @@ -5,6 +5,7 @@ import importlib.util import json import os +import sys from pathlib import Path import pytest @@ -299,3 +300,58 @@ def test_real_prepare_entry_binds_initial_successor_and_missing_facts_fail_close os.environ.pop("CODEX_TASK_TEMP", None) else: os.environ["CODEX_TASK_TEMP"] = previous_task_temp + + +def test_attested_routing_rejects_storage_rebind_after_attestation() -> None: + fixture, helper, request, host, scheduler, previous_task_temp = _real_storage_request() + try: + prepared = helper.prepare_authenticated_v5_routing_from_request( + request, + index_context_hash=helper._sha256_json({"index": "storage-real-entry"}), + host_capabilities=host, + scheduler_facts=scheduler, + ) + routing_request = prepared["routing_requests"][0] + core = sys.modules["fastlane_routing"] + request_binding_hash = core.v5_request_binding_hash(routing_request) + attestation = { + "schema": "2718lab-devkit/host-child-route-attestation-v1", + "status": "attested", + "request_binding_hash": request_binding_hash, + "host_id_hash": host["host_id_hash"], + "capability_epoch": 1, + "lease_epoch": 0, + "issued_event_seq": 1, + "expires_event_seq": 1, + "route": {"lane": "sol", "model": "gpt-5.6-sol", "effort": "high", "rank": 40}, + "inherit_current_session_model": False, + "refusal_code": None, + } + attestation["attestation_hash"] = helper._sha256_json(attestation) + tampered = copy.deepcopy(prepared["units"][0]) + budget = {"bytes": tampered["storage_budget"]["bytes"] + 1, "files": tampered["storage_budget"]["files"] + 1} + tampered["storage_budget"] = budget + intent = tampered["storage_intent"] = copy.deepcopy(tampered["storage_intent"]) + intent.update({"requested_bytes": budget["bytes"], "requested_files": budget["files"]}) + intent["storage_intent_hash"] = _canonical_hash( + {key: intent[key] for key in ("target_descriptor", "task_id", "plan_binding", "context_hash", "requested_bytes", "requested_files")} + ) + with pytest.raises(ValueError, match="routing profile"): + helper.compile_authenticated_v5_assignment_skeletons( + [tampered], + source_plan_hash=prepared["source_plan_hash"], + routing_requests=[routing_request], + attestation_items=[ + { + "task_id": routing_request["task"]["task_id"], + "request_binding_hash": request_binding_hash, + "attestation": attestation, + } + ], + ) + finally: + fixture.tearDown() + if previous_task_temp is None: + os.environ.pop("CODEX_TASK_TEMP", None) + else: + os.environ["CODEX_TASK_TEMP"] = previous_task_temp From 3e0fa2a9d014ab31a1a1d8f594baeb667aa91a8f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E5=93=80=E6=B4=9B=E8=8A=99?= Date: Sun, 30 Aug 2026 04:12:20 +0800 Subject: [PATCH 15/39] feat: consume verified host storage profiles --- .../scripts/authenticated_v5_planner.py | 398 +++++----- .../scripts/authenticated_v5_projection.py | 354 +++------ .../scripts/team_efficiency.py | 4 +- .../devkit_runtime/fastlane_host_adapter.py | 253 +++++- .../devkit_runtime/fastlane_host_intent.py | 46 +- mcp-tools/devkit_runtime/host_bridge.py | 333 ++++++++ mcp-tools/devkit_runtime/host_session.py | 177 ++++- mcp-tools/tests/test_storage_firewall.py | 733 +++++++++++------- 8 files changed, 1548 insertions(+), 750 deletions(-) diff --git a/mcp-tools/devkit_fastlane/scripts/authenticated_v5_planner.py b/mcp-tools/devkit_fastlane/scripts/authenticated_v5_planner.py index 41be7b2..d4bf111 100644 --- a/mcp-tools/devkit_fastlane/scripts/authenticated_v5_planner.py +++ b/mcp-tools/devkit_fastlane/scripts/authenticated_v5_planner.py @@ -4,6 +4,7 @@ import hmac import json +import re from collections.abc import Mapping, Sequence from typing import Any @@ -17,19 +18,13 @@ "index_context_hash", "predecessor_hash", "storage_budget", - "storage_intent", } ) -_LEGACY_UNIT_FIELDS = _UNIT_FIELDS - {"storage_budget", "storage_intent"} +_LEGACY_UNIT_FIELDS = _UNIT_FIELDS - {"storage_budget"} _BUDGET_UNIT_FIELDS = _LEGACY_UNIT_FIELDS | {"storage_budget"} -_INTENT_UNIT_FIELDS = _LEGACY_UNIT_FIELDS | {"storage_intent"} _CONTEXTUAL_UNIT_FIELDS = (_UNIT_FIELDS - {"predecessor_hash"}) | {"workflow_id_hash"} -_LEGACY_CONTEXTUAL_UNIT_FIELDS = _CONTEXTUAL_UNIT_FIELDS - { - "storage_budget", - "storage_intent", -} +_LEGACY_CONTEXTUAL_UNIT_FIELDS = _CONTEXTUAL_UNIT_FIELDS - {"storage_budget"} _BUDGET_CONTEXTUAL_UNIT_FIELDS = _LEGACY_CONTEXTUAL_UNIT_FIELDS | {"storage_budget"} -_INTENT_CONTEXTUAL_UNIT_FIELDS = _LEGACY_CONTEXTUAL_UNIT_FIELDS | {"storage_intent"} _ATTESTATION_ITEM_FIELDS = frozenset({"task_id", "request_binding_hash", "attestation"}) _CONCURRENCY_MODES = frozenset({"parallel", "serial", "isolated_worktree"}) _STORAGE_INTENT_SCHEMA = "2718lab.storage.intent.v1" @@ -46,6 +41,35 @@ "features_hash", "build_env_class", ) +_STORAGE_PROFILE_FIELDS = frozenset( + { + "schema", + "call_intent_hash", + "preparation_id", + "task_id", + "source_plan_hash", + "index_attestation_hash", + "execution_context_hash", + *_STORAGE_DESCRIPTOR_FIELDS, + "profile_hash", + "attestation_hash", + } +) +_STORAGE_PROFILE_SCHEMA = "2718lab-devkit/storage-profile-v1" +_STORAGE_PROFILE_SCALAR = re.compile(r"[A-Za-z0-9][A-Za-z0-9._+-]{0,127}\Z") +_PREPARATION_ID = re.compile(r"[a-z0-9][a-z0-9._-]{0,127}\Z") +_SKELETON_FIELDS = frozenset( + { + "task_id", + "routing_proof", + "write_scope", + "concurrency_mode", + "dispatch_order", + "index_context_hash", + "predecessor_hash", + "source_plan_hash", + } +) _PROFILE_EVIDENCE_SCHEMA = "team-efficiency/fast-lane-v5-profile-evidence-v1" _PROFILE_UNIT_FIELDS = ( "task", @@ -55,74 +79,9 @@ "dispatch_order", "index_context_hash", "workflow_id_hash", - "storage_budget", - "storage_intent", ) -def _storage_context_for_task( - context: Mapping[str, Any] | None, - task_id: str, -) -> Mapping[str, Any]: - """Resolve a task context without inventing any storage facts.""" - - if isinstance(context, Sequence) and not isinstance( - context, (str, bytes, bytearray) - ): - for item in context: - if isinstance(item, Mapping) and item.get("task_id") == task_id: - return item - return {} - if not isinstance(context, Mapping): - return {} - if "task_id" in context and context.get("task_id") != task_id: - return {} - direct = context.get(task_id) - if isinstance(direct, Mapping): - return direct - for field in ("by_task", "contexts", "execution_contexts"): - grouped = context.get(field) - if isinstance(grouped, Mapping): - direct = grouped.get(task_id) - if isinstance(direct, Mapping): - return direct - elif isinstance(grouped, Sequence) and not isinstance( - grouped, (str, bytes, bytearray) - ): - for item in grouped: - if isinstance(item, Mapping) and item.get("task_id") == task_id: - return item - return context - - -def _storage_value( - context: Mapping[str, Any], - field: str, -) -> object: - """Read one attested descriptor value from the canonical context.""" - - candidates: list[Mapping[str, Any]] = [context] - for container_name in ( - "storage_descriptor", - "target_descriptor", - "storage_target", - "build_context", - "bootstrap_plan", - "storage", - ): - container = context.get(container_name) - if isinstance(container, Mapping): - candidates.insert(0, container) - for candidate in candidates: - if field in candidate: - return candidate[field] - return None - - -def _storage_execution_context_hash(context: Mapping[str, Any]) -> object: - return context.get("execution_context_hash") - - def _storage_budget( source_unit: Mapping[str, Any], ) -> tuple[int, int]: @@ -143,32 +102,120 @@ def _storage_budget( return requested_bytes, requested_files +def _normalize_host_storage_profile( + api: Any, + value: object, + *, + task_id: str, + source_plan_hash: str, +) -> dict[str, Any]: + """Normalize one profile emitted by the private Host. + + The compiler deliberately accepts only the closed profile shape. Host + request/ref/routing bindings are validated by ``host_bridge`` before this + helper is called; this second check protects callers that use the pure + planner/compiler helper directly. + """ + + profile = api._mapping(value, f"Host storage profile {task_id}") + if set(profile) != _STORAGE_PROFILE_FIELDS: + raise ValueError(_STORAGE_POLICY_MISSING) + if ( + profile.get("schema") != _STORAGE_PROFILE_SCHEMA + or profile.get("task_id") != task_id + or profile.get("source_plan_hash") != source_plan_hash + ): + raise ValueError(_STORAGE_TARGET_KEY_INVALID) + digest_fields = { + "index_attestation_hash", + "execution_context_hash", + "repository_identity", + "workspace_manifest_hash", + "cargo_lock_hash", + "toolchain_digest", + "features_hash", + } + for field in ( + "index_attestation_hash", + "execution_context_hash", + *_STORAGE_DESCRIPTOR_FIELDS, + ): + item = profile.get(field) + if type(item) is not str: + raise ValueError(_STORAGE_POLICY_MISSING) + if field in digest_fields: + try: + api._hash(item, f"Host storage profile {task_id}.{field}") + except Exception as error: + raise ValueError(_STORAGE_TARGET_KEY_INVALID) from error + if field == "target_triple" and _STORAGE_PROFILE_SCALAR.fullmatch(item) is None: + raise ValueError(_STORAGE_TARGET_KEY_INVALID) + if field == "profile" and item != "dev": + raise ValueError(_STORAGE_TARGET_KEY_INVALID) + if field == "build_env_class" and item not in { + "managed_read_only", + "managed_workspace", + "disabled", + "external", + }: + raise ValueError(_STORAGE_TARGET_KEY_INVALID) + call_intent_hash = profile.get("call_intent_hash") + preparation_id = profile.get("preparation_id") + if ( + type(call_intent_hash) is not str + or len(call_intent_hash) != 64 + or any(character not in "0123456789abcdef" for character in call_intent_hash) + or type(preparation_id) is not str + or _PREPARATION_ID.fullmatch(preparation_id) is None + ): + raise ValueError(_STORAGE_TARGET_KEY_INVALID) + for field in ("profile_hash", "attestation_hash"): + item = profile.get(field) + try: + api._hash(item, f"Host storage profile {task_id}.{field}") + except Exception as error: + raise ValueError(_STORAGE_TARGET_KEY_INVALID) + unsigned = { + key: profile[key] + for key in _STORAGE_PROFILE_FIELDS + if key not in {"profile_hash", "attestation_hash"} + } + expected_binding = api._sha256_json(unsigned) + if not hmac.compare_digest(str(profile["profile_hash"]), expected_binding): + raise ValueError(_STORAGE_TARGET_KEY_INVALID) + # The complete attestation preimage is bound to the private request, + # routing result, and index reference by ``host_bridge``. This pure + # compiler helper deliberately cannot recreate those Host-only facts; it + # still requires the attestation to be a canonical digest and never treats + # a caller-supplied descriptor as evidence. + return {key: profile[key] for key in sorted(_STORAGE_PROFILE_FIELDS)} + + def _make_storage_intent( api: Any, source_unit: Mapping[str, Any], *, task_id: str, source_plan_hash: object, - context: Mapping[str, Any] | None, + profile: Mapping[str, Any], ) -> dict[str, object]: - """Build one path-free intent from a normalized unit and context.""" + """Build one path-free intent from caller budget and Host facts.""" source_hash = api._hash(source_plan_hash, "source_plan_hash") - task_context = _storage_context_for_task(context, task_id) - context_hash = _storage_execution_context_hash(task_context) - try: - context_hash = api._hash( - context_hash, - f"storage context {task_id}.execution_context_hash", - ) - except Exception as error: - raise ValueError(_STORAGE_POLICY_MISSING) from error + if set(profile) != _STORAGE_PROFILE_FIELDS: + raise ValueError(_STORAGE_POLICY_MISSING) + if profile.get("task_id") != task_id: + raise ValueError(_STORAGE_TARGET_KEY_INVALID) + context_hash = api._hash( + profile.get("execution_context_hash"), + f"Host storage profile {task_id}.execution_context_hash", + ) requested_bytes, requested_files = _storage_budget(source_unit) descriptor = { "schema": _STORAGE_TARGET_SCHEMA, "artifact_kind": "fastlane-task", **{ - field: _storage_value(task_context, field) + field: profile.get(field) for field in _STORAGE_DESCRIPTOR_FIELDS }, } @@ -205,83 +252,23 @@ def _make_storage_intent( raise ValueError(code) from error -def _validate_storage_intent( - value: object, - *, - task_id: str, - source_plan_hash: str, - context: Mapping[str, Any] | None, - source_unit: Mapping[str, Any], - api: Any, -) -> dict[str, object]: - try: - from devkit_runtime.storage_intent import parse_storage_intent - - parsed = parse_storage_intent(value) - except Exception as error: - code = getattr(error, "code", _STORAGE_TARGET_KEY_INVALID) - raise ValueError(code) from error - task_context = _storage_context_for_task(context, task_id) - expected_context = _storage_execution_context_hash(task_context) - if context is not None and expected_context is None: - raise ValueError(_STORAGE_POLICY_MISSING) - if expected_context is not None: - try: - expected_context = api._hash( - expected_context, - f"storage context {task_id}.execution_context_hash", - ) - except Exception as error: - raise ValueError(_STORAGE_TARGET_KEY_INVALID) from error - if "storage_budget" in source_unit: - try: - requested_bytes, requested_files = _storage_budget(source_unit) - except ValueError as error: - raise ValueError(_STORAGE_POLICY_MISSING) from error - if ( - parsed.requested_bytes != requested_bytes - or parsed.requested_files != requested_files - ): - raise ValueError(_STORAGE_TARGET_KEY_INVALID) - if ( - parsed.task_id != task_id - or parsed.plan_binding != source_plan_hash - or ( - expected_context is not None - and parsed.context_hash != expected_context - ) - ): - raise ValueError(_STORAGE_TARGET_KEY_INVALID) - return parsed.to_dict() - - def _unit_storage_intent( api: Any, unit: Mapping[str, Any], *, task_id: str, source_plan_hash: str, - context: Mapping[str, Any] | None, + profile: Mapping[str, Any], ) -> dict[str, object]: # A pre-bound intent is not permission to invent a budget at compile time. # Every compiler unit must carry the explicit request/source-unit budget. _storage_budget(unit) - supplied = unit.get("storage_intent") - if supplied is None: - return _make_storage_intent( - api, - unit, - task_id=task_id, - source_plan_hash=source_plan_hash, - context=context, - ) - return _validate_storage_intent( - supplied, + return _make_storage_intent( + api, + unit, task_id=task_id, source_plan_hash=source_plan_hash, - context=context, - source_unit=unit, - api=api, + profile=profile, ) @@ -290,12 +277,15 @@ def _routing_profile_material( ) -> dict[str, Any]: task = dict(unit["task"]) task.pop("profile_evidence_hash", None) + fields = _PROFILE_UNIT_FIELDS + ( + ("storage_budget",) if "storage_budget" in unit else () + ) return { "schema": _PROFILE_EVIDENCE_SCHEMA, "source_plan_hash": source_plan_hash, "unit": { field: task if field == "task" else unit[field] - for field in _PROFILE_UNIT_FIELDS + for field in fields }, } @@ -333,8 +323,6 @@ def normalize_units( _LEGACY_CONTEXTUAL_UNIT_FIELDS, _BUDGET_UNIT_FIELDS, _BUDGET_CONTEXTUAL_UNIT_FIELDS, - _INTENT_UNIT_FIELDS, - _INTENT_CONTEXTUAL_UNIT_FIELDS, }: raise ValueError(f"authenticated V5 units[{index}] has unsupported fields") task = dict(api._mapping(unit["task"], f"authenticated V5 units[{index}].task")) @@ -396,21 +384,21 @@ def normalize_units( } if "storage_budget" in unit: budget = unit["storage_budget"] - if not isinstance(budget, Mapping): + if not isinstance(budget, Mapping) or set(budget) != {"bytes", "files"}: raise ValueError(_STORAGE_POLICY_MISSING) - normalized_unit["storage_budget"] = json.loads( - api._canonical_json(budget) - ) - if "storage_intent" in unit: - try: - from devkit_runtime.storage_intent import parse_storage_intent - - normalized_unit["storage_intent"] = parse_storage_intent( - unit["storage_intent"] - ).to_dict() - except Exception as error: - code = getattr(error, "code", _STORAGE_TARGET_KEY_INVALID) - raise ValueError(code) from error + requested_bytes = budget.get("bytes") + requested_files = budget.get("files") + if ( + type(requested_bytes) is not int + or not 0 < requested_bytes <= (1 << 64) - 1 + or type(requested_files) is not int + or not 0 < requested_files <= (1 << 64) - 1 + ): + raise ValueError(_STORAGE_POLICY_MISSING) + normalized_unit["storage_budget"] = { + "bytes": requested_bytes, + "files": requested_files, + } normalized.append(normalized_unit) task_ids = [str(item["task"]["task_id"]) for item in normalized] orders = [int(item["dispatch_order"]) for item in normalized] @@ -477,8 +465,8 @@ def compile_skeletons( source_plan_hash: object, routing_requests: Sequence[Mapping[str, Any]], attestation_items: Sequence[Mapping[str, Any]], - context: Mapping[str, Any] | None = None, -) -> dict[str, list[dict[str, Any]]]: + storage_profiles: object = None, +) -> dict[str, object]: source_hash = api._hash(source_plan_hash, "source_plan_hash") normalized_units = normalize_units(api, units) if ( @@ -493,6 +481,30 @@ def compile_skeletons( core = api._fast_lane_routing_core() if core is None: raise ValueError("authenticated V5 routing core is unavailable") + storage_required = any("storage_budget" in unit for unit in normalized_units) + profiles_by_task: dict[str, dict[str, Any]] = {} + if storage_profiles is not None: + if not storage_required or any( + "storage_budget" not in unit for unit in normalized_units + ): + raise ValueError(_STORAGE_TARGET_KEY_INVALID) + if ( + not isinstance(storage_profiles, Sequence) + or isinstance(storage_profiles, (str, bytes, bytearray)) + or len(storage_profiles) != len(normalized_units) + ): + raise ValueError(_STORAGE_POLICY_MISSING) + for unit, raw_profile in zip(normalized_units, storage_profiles, strict=True): + task_id = str(unit["task"]["task_id"]) + profile = _normalize_host_storage_profile( + api, + raw_profile, + task_id=task_id, + source_plan_hash=source_hash, + ) + profiles_by_task[task_id] = profile + if len(profiles_by_task) != len(normalized_units): + raise ValueError(_STORAGE_TARGET_KEY_INVALID) policy = core.load_policy_v5() request_by_task: dict[str, dict[str, Any]] = {} for raw_request in routing_requests: @@ -559,16 +571,24 @@ def compile_skeletons( attestation_by_task[task_id] = attestation skeletons: list[dict[str, Any]] = [] + storage_intents: list[dict[str, object]] = [] route_pairs: set[tuple[str, str]] = set() for unit in normalized_units: task_id = str(unit["task"]["task_id"]) - storage_intent = _unit_storage_intent( - api, - unit, - task_id=task_id, - source_plan_hash=source_hash, - context=context, - ) + profile = profiles_by_task.get(task_id) + if storage_profiles is not None: + _storage_budget(unit) + if profile is None: + raise ValueError(_STORAGE_POLICY_MISSING) + storage_intents.append( + _unit_storage_intent( + api, + unit, + task_id=task_id, + source_plan_hash=source_hash, + profile=profile, + ) + ) request = request_by_task.get(task_id) attestation = attestation_by_task.get(task_id) if request is None or attestation is None: @@ -578,7 +598,7 @@ def compile_skeletons( profile_hash = api._sha256_json( _routing_profile_material( source_hash, - {**unit, "storage_intent": storage_intent}, + unit, ) ) if not hmac.compare_digest( @@ -650,16 +670,21 @@ def compile_skeletons( "index_context_hash": unit["index_context_hash"], "predecessor_hash": predecessor_hash, "source_plan_hash": source_hash, - "storage_intent": storage_intent, } ) route_pairs.add((model, effort)) - return { + result: dict[str, object] = { "assignment_skeletons": skeletons, "requested_route_pairs": [ {"model": model, "effort": effort} for model, effort in sorted(route_pairs) ], } + if storage_profiles is not None: + result["storage_profiles"] = [ + profiles_by_task[str(unit["task"]["task_id"])] for unit in normalized_units + ] + result["storage_intents"] = storage_intents + return result def validate_skeleton_package( @@ -692,6 +717,7 @@ def validate_skeleton_package( ) if not 1 <= len(source_ids) <= 16 or len(set(source_ids)) != len(source_ids): raise ValueError("authenticated V5 source plan task coverage is invalid") + normalize_units(api, source_plan_units) combined: list[dict[str, Any]] = [] for wave_name, wave in ( @@ -704,22 +730,16 @@ def validate_skeleton_package( skeleton = dict( api._mapping(raw_skeleton, f"authenticated V5 {wave_name} skeletons[{index}]") ) + if set(skeleton) != _SKELETON_FIELDS: + raise ValueError("authenticated V5 skeleton fields are invalid") if skeleton.get("source_plan_hash") != source_hash: raise ValueError("authenticated V5 skeleton source hash is invalid") task_id = api._task_id( skeleton.get("task_id"), f"authenticated V5 {wave_name} skeletons[{index}].task_id", ) - storage_intent = _validate_storage_intent( - skeleton.get("storage_intent"), - task_id=task_id, - source_plan_hash=source_hash, - context=None, - source_unit={}, - api=api, - ) - if skeleton.get("storage_intent") != storage_intent: - raise ValueError("authenticated V5 storage intent is not canonical") + if "storage_intent" in skeleton: + raise ValueError("authenticated V5 pre-host skeleton carries storage intent") order = skeleton.get("dispatch_order") if type(order) is not int or not 0 <= order < len(source_ids): raise ValueError("authenticated V5 package dispatch order is invalid") diff --git a/mcp-tools/devkit_fastlane/scripts/authenticated_v5_projection.py b/mcp-tools/devkit_fastlane/scripts/authenticated_v5_projection.py index 34eb80e..76b3c4d 100644 --- a/mcp-tools/devkit_fastlane/scripts/authenticated_v5_projection.py +++ b/mcp-tools/devkit_fastlane/scripts/authenticated_v5_projection.py @@ -5,8 +5,6 @@ from collections.abc import Mapping, Sequence from typing import Any -_STORAGE_INTENT_SCHEMA = "2718lab.storage.intent.v1" -_STORAGE_TARGET_SCHEMA = "2718lab.storage.target.v1" _STORAGE_POLICY_MISSING = "STORAGE_POLICY_MISSING" _STORAGE_TARGET_KEY_INVALID = "STORAGE_TARGET_KEY_INVALID" _STORAGE_DESCRIPTOR_FIELDS = ( @@ -19,11 +17,20 @@ "features_hash", "build_env_class", ) +_STORAGE_REQUEST_FIELDS = frozenset({"storage_budgets"}) _STORAGE_CONTEXT_FIELDS = frozenset( {"execution_context_hash", *_STORAGE_DESCRIPTOR_FIELDS} ) -_STORAGE_REQUEST_FIELDS = frozenset({"storage_budgets", "storage_contexts"}) -_STORAGE_CONTEXT_CONTAINERS = ("storage_context", "storage_descriptor") +_STORAGE_PUBLIC_DESCRIPTOR_KEYS = _STORAGE_CONTEXT_FIELDS | frozenset( + { + "storage_context", + "storage_contexts", + "storage_descriptor", + "storage_profile", + "storage_profiles", + "target_descriptor", + } +) _PROFILE_EVIDENCE_SCHEMA = "team-efficiency/fast-lane-v5-profile-evidence-v1" _PROFILE_UNIT_FIELDS = ( "task", @@ -33,8 +40,6 @@ "dispatch_order", "index_context_hash", "workflow_id_hash", - "storage_budget", - "storage_intent", ) @@ -43,168 +48,48 @@ def _routing_profile_material( ) -> dict[str, Any]: task = dict(unit["task"]) task.pop("profile_evidence_hash", None) + fields = _PROFILE_UNIT_FIELDS + ( + ("storage_budget",) if "storage_budget" in unit else () + ) return { "schema": _PROFILE_EVIDENCE_SCHEMA, "source_plan_hash": source_plan_hash, "unit": { field: task if field == "task" else unit[field] - for field in _PROFILE_UNIT_FIELDS + for field in fields }, } -def _storage_record_for_task( - value: object, task_id: str, field: str -) -> Mapping[str, Any] | None: - """Return one explicitly keyed storage record without selecting a default.""" - - if value is None: - return None - if isinstance(value, Mapping): - direct = value.get(task_id) - if isinstance(direct, Mapping): - return direct - if value.get("task_id") == task_id: - return value - if any(key in value for key in _STORAGE_CONTEXT_FIELDS) or field == "budget": - raise ValueError(_STORAGE_POLICY_MISSING) - return None - if isinstance(value, Sequence) and not isinstance(value, (str, bytes, bytearray)): - matches = [ - item - for item in value - if isinstance(item, Mapping) and item.get("task_id") == task_id - ] - if len(matches) > 1: - raise ValueError(_STORAGE_TARGET_KEY_INVALID) - return matches[0] if matches else None - raise ValueError(_STORAGE_POLICY_MISSING) - - -def _storage_context_record(value: Mapping[str, Any]) -> Mapping[str, Any] | None: - """Collect only explicit context facts and reject conflicting duplicates.""" - - records: list[Mapping[str, Any]] = [] - direct = {key: value[key] for key in _STORAGE_CONTEXT_FIELDS if key in value} - if direct: - records.append(direct) - for name in _STORAGE_CONTEXT_CONTAINERS: - nested = value.get(name) - if isinstance(nested, Mapping): - records.append( - {key: nested[key] for key in _STORAGE_CONTEXT_FIELDS if key in nested} - ) - if not records: - return None - merged: dict[str, Any] = {} - for record in records: - for key, item in record.items(): - if key in merged and merged[key] != item: - raise ValueError(_STORAGE_TARGET_KEY_INVALID) - merged[key] = item - return merged - - -def _canonical_storage_context(value: Mapping[str, Any]) -> dict[str, Any]: - if set(value) != _STORAGE_CONTEXT_FIELDS: - raise ValueError(_STORAGE_POLICY_MISSING) - return dict(value) - - -def _merge_storage_contexts( - *records: Mapping[str, Any] | None, -) -> dict[str, Any] | None: - merged: dict[str, Any] = {} - for record in records: - if record is None: - continue - for key, item in record.items(): - if key in merged and merged[key] != item: - raise ValueError(_STORAGE_TARGET_KEY_INVALID) - merged[key] = item - return merged or None - - -def _storage_context_without_facts(value: object) -> object: - if not isinstance(value, Mapping): - return value - cleaned = dict(value) - for key in _STORAGE_CONTEXT_FIELDS: - cleaned.pop(key, None) - for name in _STORAGE_CONTEXT_CONTAINERS: - cleaned.pop(name, None) - return cleaned - - def _storage_request_without_extensions( value: Mapping[str, Any], api: Any ) -> None: + if "storage_contexts" in value: + raise ValueError(_STORAGE_TARGET_KEY_INVALID) base = { key: item for key, item in value.items() if key not in _STORAGE_REQUEST_FIELDS } api._exact_keys(base, api._FAST_LANE_REQUEST_FIELDS, "fast-lane request") -def _attach_storage_budget( - source_unit: Mapping[str, Any], - request: Mapping[str, Any], - *, - task_id: str, -) -> dict[str, Any]: - result = dict(source_unit) - source_budget = source_unit.get("storage_budget") - request_record = _storage_record_for_task( - request.get("storage_budgets"), task_id, "budget" - ) - request_budget: object = request_record - if isinstance(request_record, Mapping) and "storage_budget" in request_record: - request_budget = request_record["storage_budget"] - if source_budget is not None and request_budget is not None: - if source_budget != request_budget: - raise ValueError(_STORAGE_TARGET_KEY_INVALID) - elif source_budget is None: - source_budget = request_budget - if source_budget is None: - raise ValueError(_STORAGE_POLICY_MISSING) - result["storage_budget"] = source_budget - return result - +def _reject_public_storage_facts(value: object) -> None: + """Reject descriptor facts supplied through the public request.""" -def _storage_context_value( - context: Mapping[str, Any], field: str -) -> object: - candidates: list[Mapping[str, Any]] = [context] - for container_name in _STORAGE_CONTEXT_CONTAINERS: - container = context.get(container_name) - if isinstance(container, Mapping): - candidates.insert(0, container) - for candidate in candidates: - if field in candidate: - return candidate[field] - return None + if isinstance(value, Mapping): + if set(value).intersection(_STORAGE_PUBLIC_DESCRIPTOR_KEYS): + raise ValueError(_STORAGE_TARGET_KEY_INVALID) + for item in value.values(): + _reject_public_storage_facts(item) + elif isinstance(value, Sequence) and not isinstance(value, (str, bytes, bytearray)): + for item in value: + _reject_public_storage_facts(item) -def _make_storage_intent( - api: Any, - source_unit: Mapping[str, Any], - context: Mapping[str, Any], - *, - task_id: str, - source_plan_hash: str, -) -> dict[str, object]: - context_hash = context.get("execution_context_hash") - try: - context_hash = api._hash( - context_hash, - f"storage context {task_id}.execution_context_hash", - ) - except Exception as error: - raise ValueError(_STORAGE_POLICY_MISSING) from error - budget = source_unit.get("storage_budget") - if not isinstance(budget, Mapping): +def _validated_storage_budget(value: object) -> dict[str, int]: + if not isinstance(value, Mapping) or set(value) != {"bytes", "files"}: raise ValueError(_STORAGE_POLICY_MISSING) - requested_bytes = budget.get("bytes") - requested_files = budget.get("files") + requested_bytes = value.get("bytes") + requested_files = value.get("files") if ( type(requested_bytes) is not int or requested_bytes <= 0 @@ -214,46 +99,46 @@ def _make_storage_intent( or requested_files > (1 << 64) - 1 ): raise ValueError(_STORAGE_POLICY_MISSING) - descriptor = { - "schema": _STORAGE_TARGET_SCHEMA, - "artifact_kind": "fastlane-task", - **{ - field: _storage_context_value(context, field) - for field in _STORAGE_DESCRIPTOR_FIELDS - }, - } - if any(descriptor[field] is None for field in _STORAGE_DESCRIPTOR_FIELDS): + return {"bytes": requested_bytes, "files": requested_files} + + +def _validated_request_storage_budgets( + value: object, task_ids: Sequence[str] +) -> dict[str, dict[str, int]]: + """Accept only exact per-task public budgets; no default or surplus task.""" + + if value is None: + return {} + if not isinstance(value, Mapping) or set(value) != set(task_ids): raise ValueError(_STORAGE_POLICY_MISSING) - intent_preimage = { - "target_descriptor": descriptor, - "task_id": task_id, - "plan_binding": source_plan_hash, - "context_hash": context_hash, - "requested_bytes": requested_bytes, - "requested_files": requested_files, - } - intent = { - "schema": _STORAGE_INTENT_SCHEMA, - **{ - key: intent_preimage[key] - for key in ( - "task_id", - "plan_binding", - "context_hash", - "requested_bytes", - "requested_files", - "target_descriptor", - ) - }, - "storage_intent_hash": api._sha256_json(intent_preimage), + return { + task_id: _validated_storage_budget(value[task_id]) + for task_id in task_ids } - try: - from devkit_runtime.storage_intent import parse_storage_intent - return parse_storage_intent(intent).to_dict() - except Exception as error: - code = getattr(error, "code", _STORAGE_TARGET_KEY_INVALID) - raise ValueError(code) from error + +def _attach_storage_budget( + source_unit: Mapping[str, Any], + request_budgets: Mapping[str, Mapping[str, int]], + *, + task_id: str, +) -> dict[str, Any]: + result = dict(source_unit) + source_budget = source_unit.get("storage_budget") + request_budget = request_budgets.get(task_id) + if source_budget is not None: + source_budget = _validated_storage_budget(source_budget) + if request_budget is not None: + request_budget = _validated_storage_budget(request_budget) + if source_budget is not None and request_budget is not None: + if source_budget != request_budget: + raise ValueError(_STORAGE_TARGET_KEY_INVALID) + elif source_budget is None: + source_budget = request_budget + if source_budget is None: + return result + result["storage_budget"] = _validated_storage_budget(source_budget) + return result def project_units( @@ -307,23 +192,8 @@ def project_units_with_waves( ): raise ValueError("authenticated V5 raw request is unsupported") source_plan = api.decompose(candidate["work_package"]) - raw_context_by_task: dict[str, Mapping[str, Any]] = {} raw_execution_contexts = candidate["execution_contexts"] - if isinstance(raw_execution_contexts, Sequence) and not isinstance( - raw_execution_contexts, (str, bytes, bytearray) - ): - for raw_context in raw_execution_contexts: - if not isinstance(raw_context, Mapping): - continue - raw_task_id = raw_context.get("task_id") - if not isinstance(raw_task_id, str): - continue - record = _storage_context_record(raw_context) - if record is not None: - existing = raw_context_by_task.get(raw_task_id) - raw_context_by_task[raw_task_id] = _merge_storage_contexts( - existing, record - ) or {} + _reject_public_storage_facts(raw_execution_contexts) source_plan_hash = api._sha256_json(source_plan) if source_plan.get("status") != "planned": raise ValueError("authenticated V5 source plan is not schedulable") @@ -347,10 +217,7 @@ def project_units_with_waves( candidate["target_gates"], source_plan ) execution_contexts, read_contexts = api._validated_fast_lane_contexts( - [ - _storage_context_without_facts(item) - for item in candidate["execution_contexts"] - ], + candidate["execution_contexts"], candidate["read_contexts"], source_plan, candidate["scheduler_state"], @@ -368,27 +235,21 @@ def project_units_with_waves( if remediation is not None or state["phase"] != "execution": raise ValueError("authenticated V5 scheduler phase is unsupported") - storage_context_by_task: dict[str, dict[str, Any]] = {} - for normalized_context in execution_contexts: - task_id = str(normalized_context["task_id"]) - request_context = _storage_record_for_task( - candidate.get("storage_contexts"), task_id, "context" - ) - raw_context = raw_context_by_task.get(task_id) - merged = _merge_storage_contexts(raw_context, request_context) - if merged is None: - raise ValueError(_STORAGE_POLICY_MISSING) - storage_context_by_task[task_id] = _canonical_storage_context(merged) - execution_contexts = [ - {**context, **storage_context_by_task[str(context["task_id"])]} - for context in execution_contexts - ] - raw_units_by_task = api._fast_lane_unit_index(source_plan) + request_budgets = _validated_request_storage_budgets( + candidate.get("storage_budgets"), tuple(raw_units_by_task) + ) + source_budget_task_ids = { + task_id + for task_id, source_unit in raw_units_by_task.items() + if "storage_budget" in source_unit + } + if source_budget_task_ids and source_budget_task_ids != set(raw_units_by_task): + raise ValueError(_STORAGE_POLICY_MISSING) units_by_task = { task_id: _attach_storage_budget( source_unit, - candidate, + request_budgets, task_id=task_id, ) for task_id, source_unit in raw_units_by_task.items() @@ -439,7 +300,6 @@ def project_units_with_waves( ) ] index_hash = api._hash(index_context_hash, "index_context_hash") - context_by_task = {item["task_id"]: item for item in execution_contexts} target_by_task = {item["task_id"]: item for item in target_gates} workflow_hash = api._sha256_json({"workflow_id": project_binding["workflow_id"]}) @@ -487,8 +347,7 @@ def project_slice(task_ids: Sequence[str]) -> list[dict[str, Any]]: dispatch_order = package_order[task_id] source_unit = units_by_task[task_id] if ( - context_by_task.get(task_id) is None - or target_by_task.get(task_id) is None + target_by_task.get(task_id) is None ): raise ValueError("authenticated V5 execution context is incomplete") target = target_by_task[task_id] @@ -514,13 +373,6 @@ def project_slice(task_ids: Sequence[str]) -> list[dict[str, Any]]: **dependency_without_hash, "dependency_state_hash": api._sha256_json(dependency_without_hash), } - storage_intent = _make_storage_intent( - api, - source_unit, - context_by_task[task_id], - task_id=task_id, - source_plan_hash=source_plan_hash, - ) criticality = { "Terra High": "normal", "Terra Max": "high", @@ -560,34 +412,22 @@ def project_slice(task_ids: Sequence[str]) -> list[dict[str, Any]]: "strike": None, "gate_matrix_hash": api._sha256_json(target), } - profile_material = _routing_profile_material( - source_plan_hash, - { - "task": task, - "dependency_state": dependency_state, - "write_scope": write_scope, - "concurrency_mode": "parallel", - "dispatch_order": dispatch_order, - "index_context_hash": index_hash, - "workflow_id_hash": workflow_hash, - "storage_budget": source_unit["storage_budget"], - "storage_intent": storage_intent, - }, - ) + profile_unit: dict[str, Any] = { + "task": task, + "dependency_state": dependency_state, + "write_scope": write_scope, + "concurrency_mode": "parallel", + "dispatch_order": dispatch_order, + "index_context_hash": index_hash, + "workflow_id_hash": workflow_hash, + } + if "storage_budget" in source_unit: + profile_unit["storage_budget"] = source_unit["storage_budget"] + profile_material = _routing_profile_material(source_plan_hash, profile_unit) task["profile_evidence_hash"] = api._sha256_json(profile_material) - projected_slice.append( - { - "task": task, - "dependency_state": dependency_state, - "write_scope": write_scope, - "concurrency_mode": "parallel", - "dispatch_order": dispatch_order, - "index_context_hash": index_hash, - "workflow_id_hash": workflow_hash, - "storage_budget": source_unit["storage_budget"], - "storage_intent": storage_intent, - } - ) + projected_unit = dict(profile_unit) + projected_unit["task"] = task + projected_slice.append(projected_unit) return projected_slice return ( diff --git a/mcp-tools/devkit_fastlane/scripts/team_efficiency.py b/mcp-tools/devkit_fastlane/scripts/team_efficiency.py index ee616e1..99938ee 100644 --- a/mcp-tools/devkit_fastlane/scripts/team_efficiency.py +++ b/mcp-tools/devkit_fastlane/scripts/team_efficiency.py @@ -4886,7 +4886,8 @@ def compile_authenticated_v5_assignment_skeletons( source_plan_hash: object, routing_requests: Sequence[Mapping[str, Any]], attestation_items: Sequence[Mapping[str, Any]], -) -> dict[str, list[dict[str, Any]]]: + storage_profiles: object = None, +) -> dict[str, object]: planner = _authenticated_v5_helper_module("authenticated_v5_planner") return planner.compile_skeletons( _AuthenticatedV5Api(), @@ -4894,6 +4895,7 @@ def compile_authenticated_v5_assignment_skeletons( source_plan_hash=source_plan_hash, routing_requests=routing_requests, attestation_items=attestation_items, + storage_profiles=storage_profiles, ) diff --git a/mcp-tools/devkit_runtime/fastlane_host_adapter.py b/mcp-tools/devkit_runtime/fastlane_host_adapter.py index 9227f12..69e88fa 100644 --- a/mcp-tools/devkit_runtime/fastlane_host_adapter.py +++ b/mcp-tools/devkit_runtime/fastlane_host_adapter.py @@ -7,6 +7,7 @@ from __future__ import annotations import hashlib +import hmac import json import re import unicodedata @@ -25,7 +26,48 @@ NO_SAFE_WORK: Final = "NO_SAFE_WORK" _HASH: Final = re.compile(r"sha256:[0-9a-f]{64}\Z") +_RAW_HASH: Final = re.compile(r"[0-9a-f]{64}\Z") +_STORAGE_PROFILE_SCHEMA: Final = "2718lab-devkit/storage-profile-v1" +_STORAGE_INTENT_SCHEMA: Final = "2718lab.storage.intent.v1" +_STORAGE_TARGET_SCHEMA: Final = "2718lab.storage.target.v1" +_STORAGE_PROFILE_FIELDS: Final = frozenset( + { + "schema", + "call_intent_hash", + "preparation_id", + "task_id", + "source_plan_hash", + "index_attestation_hash", + "execution_context_hash", + "repository_identity", + "workspace_manifest_hash", + "cargo_lock_hash", + "toolchain_digest", + "target_triple", + "profile", + "features_hash", + "build_env_class", + "profile_hash", + "attestation_hash", + } +) +_STORAGE_DESCRIPTOR_FIELDS: Final = ( + "repository_identity", + "workspace_manifest_hash", + "cargo_lock_hash", + "toolchain_digest", + "target_triple", + "profile", + "features_hash", + "build_env_class", +) _LABEL: Final = re.compile(r"[A-Za-z0-9][A-Za-z0-9._-]{0,127}\Z") +_PREPARATION_ID: Final = re.compile(r"[a-z0-9][a-z0-9._-]{0,127}\Z") +_FAST_LANE_TASK_ID: Final = re.compile(r"[A-Za-z0-9][A-Za-z0-9._:-]{0,95}\Z") +_STORAGE_PROFILE_SCALAR: Final = re.compile(r"[A-Za-z0-9][A-Za-z0-9._+-]{0,127}\Z") +_STORAGE_PROFILE_BUILD_ENV_CLASSES: Final = frozenset( + {"managed_read_only", "managed_workspace", "disabled", "external"} +) _PATH_PART: Final = re.compile(r"[A-Za-z0-9][A-Za-z0-9._-]*\Z") _WINDOWS_RESERVED_NAMES: Final = frozenset( { @@ -124,6 +166,181 @@ class _PreparedHostFacts: evidence_expires_at: int | None = None preparation_id: str | None = None call_intent_hash: str | None = None + storage_budgets: tuple[tuple[str, int, int], ...] = () + + +def _normalized_storage_budgets( + value: object, +) -> tuple[tuple[str, int, int], ...]: + """Normalize caller budgets kept outside the pre-Host skeleton.""" + + if value is None: + return () + if not isinstance(value, Mapping) or len(value) > 16: + raise ValueError("storage budgets are unavailable") + normalized: list[tuple[str, int, int]] = [] + for raw_task_id, raw_budget in value.items(): + if ( + type(raw_task_id) is not str + or _FAST_LANE_TASK_ID.fullmatch(raw_task_id) is None + or not isinstance(raw_budget, Mapping) + ): + raise ValueError("storage budget is invalid") + task_id = raw_task_id + if set(raw_budget) != {"bytes", "files"}: + raise ValueError("storage budget is invalid") + requested_bytes = raw_budget.get("bytes") + requested_files = raw_budget.get("files") + if ( + type(requested_bytes) is not int + or not 0 < requested_bytes <= (1 << 64) - 1 + or type(requested_files) is not int + or not 0 < requested_files <= (1 << 64) - 1 + ): + raise ValueError("storage budget is invalid") + normalized.append((task_id, requested_bytes, requested_files)) + normalized.sort(key=lambda item: item[0]) + if len({item[0] for item in normalized}) != len(normalized): + raise ValueError("storage budget tasks are duplicated") + return tuple(normalized) + + +def _storage_intents_for_profiles( + profiles: tuple[dict[str, object], ...], + assignments: Sequence[Mapping[str, object]], + budgets: tuple[tuple[str, int, int], ...], +) -> list[dict[str, object]]: + """Construct post-Host intents from Host facts and external budgets.""" + + if not profiles: + if budgets: + raise ValueError("Host storage profiles are unavailable") + return [] + if not budgets or len(profiles) != len(assignments): + raise ValueError("Host storage profiles are incomplete") + budget_by_task = {task_id: (bytes_, files) for task_id, bytes_, files in budgets} + assignment_task_ids: list[str] = [] + for assignment in assignments: + task_id = assignment.get("task_id") + source_plan_hash = assignment.get("source_plan_hash") + if ( + type(task_id) is not str + or _FAST_LANE_TASK_ID.fullmatch(task_id) is None + or type(source_plan_hash) is not str + or _HASH.fullmatch(source_plan_hash) is None + ): + raise ValueError("storage profile assignment binding is invalid") + assignment_task_ids.append(task_id) + if ( + len(set(assignment_task_ids)) != len(assignment_task_ids) + or set(budget_by_task) != set(assignment_task_ids) + ): + raise ValueError("storage budget/profile bindings are invalid") + profile_task_ids: list[str] = [] + for profile in profiles: + if type(profile) is not dict or set(profile) != _STORAGE_PROFILE_FIELDS: + raise ValueError("Host storage profile fields are invalid") + task_id = profile.get("task_id") + call_intent_hash = profile.get("call_intent_hash") + preparation_id = profile.get("preparation_id") + if ( + type(task_id) is not str + or _FAST_LANE_TASK_ID.fullmatch(task_id) is None + or type(call_intent_hash) is not str + or _RAW_HASH.fullmatch(call_intent_hash) is None + or type(preparation_id) is not str + or _PREPARATION_ID.fullmatch(preparation_id) is None + or profile.get("schema") != _STORAGE_PROFILE_SCHEMA + ): + raise ValueError("Host storage profile is invalid") + for field_name in ( + "source_plan_hash", + "index_attestation_hash", + "execution_context_hash", + "repository_identity", + "workspace_manifest_hash", + "cargo_lock_hash", + "toolchain_digest", + "features_hash", + "profile_hash", + "attestation_hash", + ): + value = profile.get(field_name) + if type(value) is not str or _HASH.fullmatch(value) is None: + raise ValueError("Host storage profile hashes are invalid") + target_triple = profile.get("target_triple") + if ( + type(target_triple) is not str + or _STORAGE_PROFILE_SCALAR.fullmatch(target_triple) is None + or profile.get("profile") != "dev" + ): + raise ValueError("Host storage profile scalar is invalid") + if profile.get("build_env_class") not in _STORAGE_PROFILE_BUILD_ENV_CLASSES: + raise ValueError("Host storage profile build environment is invalid") + unsigned = { + key: profile[key] + for key in _STORAGE_PROFILE_FIELDS + if key not in {"profile_hash", "attestation_hash"} + } + if not hmac.compare_digest( + cast(str, profile["profile_hash"]), _canonical_hash(unsigned) + ): + raise ValueError("Host storage profile binding is invalid") + profile_task_ids.append(task_id) + if profile_task_ids != assignment_task_ids: + raise ValueError("storage budget/profile bindings are invalid") + intents: list[dict[str, object]] = [] + for assignment, profile in zip(assignments, profiles, strict=True): + task_id = assignment.get("task_id") + source_plan_hash = assignment.get("source_plan_hash") + assert type(task_id) is str + budget = budget_by_task.get(task_id) + if ( + budget is None + or profile.get("source_plan_hash") != source_plan_hash + ): + raise ValueError("storage profile assignment binding is invalid") + context_hash = profile.get("execution_context_hash") + if ( + type(source_plan_hash) is not str + or _HASH.fullmatch(source_plan_hash) is None + or type(context_hash) is not str + or _HASH.fullmatch(context_hash) is None + ): + raise ValueError("storage profile hashes are invalid") + descriptor = { + "schema": _STORAGE_TARGET_SCHEMA, + "artifact_kind": "fastlane-task", + **{field: profile.get(field) for field in _STORAGE_DESCRIPTOR_FIELDS}, + } + if any( + type(descriptor[field]) is not str or not descriptor[field] + for field in _STORAGE_DESCRIPTOR_FIELDS + ): + raise ValueError("storage profile descriptor is invalid") + requested_bytes, requested_files = budget + intent_preimage = { + "target_descriptor": descriptor, + "task_id": task_id, + "plan_binding": source_plan_hash, + "context_hash": context_hash, + "requested_bytes": requested_bytes, + "requested_files": requested_files, + } + intent = { + "schema": _STORAGE_INTENT_SCHEMA, + "task_id": task_id, + "plan_binding": source_plan_hash, + "context_hash": context_hash, + "storage_intent_hash": _canonical_hash(intent_preimage), + "requested_bytes": requested_bytes, + "requested_files": requested_files, + "target_descriptor": descriptor, + } + from .storage_intent import parse_storage_intent + + intents.append(parse_storage_intent(intent).to_dict()) + return intents def prepare_verified_host_facts( @@ -136,6 +353,7 @@ def prepare_verified_host_facts( request: object = None, reasoning_effort: object = None, requested_routes: Sequence[HostRoute] | object = None, + storage_budgets: object = None, ) -> _PreparedHostFacts | str: """Accept no public substitute for session-owned compiler evidence.""" @@ -148,6 +366,7 @@ def prepare_verified_host_facts( ): return NO_SAFE_WORK try: + normalized_storage_budgets = _normalized_storage_budgets(storage_budgets) bridge_attested = False if request is not None or reasoning_effort is not None: normalized_request = _planner_request(request) @@ -183,15 +402,26 @@ def prepare_verified_host_facts( return NO_SAFE_WORK else: requested_routes = tuple(requested_routes) + skeletons = tuple( + cast( + list[dict[str, object]], + normalized_request["assignment_skeletons"], + ) + ) + storage_task_ids = tuple( + cast(str, skeleton["task_id"]) for skeleton in skeletons + ) + if normalized_storage_budgets and { + task_id for task_id, _bytes, _files in normalized_storage_budgets + } != set(storage_task_ids): + return NO_SAFE_WORK bridge_attested = session.bind_compiler_request( preparation_id=normalized_preparation_id, call_intent_hash=cast(str, call_intent_hash), request_hash=_hash_bytes(request_bytes), reasoning_effort=reasoning_effort, requested_routes=requested_routes, - assignment_skeletons=tuple( - cast(list[dict[str, object]], normalized_request["assignment_skeletons"]) - ), + assignment_skeletons=skeletons, project_index_attestation_refs=tuple( cast( list[dict[str, object]], @@ -201,10 +431,13 @@ def prepare_verified_host_facts( routing_registry_binding_hash=cast( str, routing_registry_binding_hash ), + storage_task_ids=(storage_task_ids if normalized_storage_budgets else ()), ) if not bridge_attested: return NO_SAFE_WORK else: + if normalized_storage_budgets: + return NO_SAFE_WORK scheduling = session.scheduling_facts(tuple(capability_facts)) if type(scheduling) is not HostSchedulingFacts: return NO_SAFE_WORK @@ -226,6 +459,7 @@ def prepare_verified_host_facts( call_intent_hash=( cast(str, call_intent_hash) if bridge_attested else None ), + storage_budgets=normalized_storage_budgets, ) except Exception: return NO_SAFE_WORK @@ -331,6 +565,19 @@ def compile_fast_lane_with_host_facts( ): return NO_SAFE_WORK _validate_batch_fences(facts) + # Storage intents are local compiler proof material only in Task4a. Do + # not extend the established dispatch-batch schema before Task4b owns + # Host admission/execution. Constructing them here still validates the + # Host profile order, the caller budgets, and every profile/dispatch + # binding after the full compiler evidence response has been verified. + if prepared.storage_budgets: + _storage_intents_for_profiles( + material.storage_profiles, + fact_mappings, + prepared.storage_budgets, + ) + elif material.storage_profiles: + return NO_SAFE_WORK batch: dict[str, object] = { "schema": "2718lab-devkit/fastlane-host-dispatch-batch-v1", "action": "dispatch_all", diff --git a/mcp-tools/devkit_runtime/fastlane_host_intent.py b/mcp-tools/devkit_runtime/fastlane_host_intent.py index 860b315..2f56918 100644 --- a/mcp-tools/devkit_runtime/fastlane_host_intent.py +++ b/mcp-tools/devkit_runtime/fastlane_host_intent.py @@ -24,7 +24,8 @@ NO_SAFE_WORK: Final = "NO_SAFE_WORK" UNSPLITTABLE: Final = "UNSPLITTABLE" -_SCHEMA: Final = "2718lab-devkit/fastlane-host-execution-intent-v2" +_SCHEMA_V2: Final = "2718lab-devkit/fastlane-host-execution-intent-v2" +_SCHEMA_V3: Final = "2718lab-devkit/fastlane-host-execution-intent-v3" _RELAY_HOST_SCHEDULER_SLOT_SCHEMA: Final = ( "2718lab-devkit/relay-host-scheduler-slot-v1" ) @@ -405,7 +406,12 @@ def classify_host_scheduler_topology( def _parse(candidate: object) -> ParsedHostExecutionIntent | None: - if isinstance(candidate, dict) and candidate.get("schema") == _SCHEMA: + schema = candidate.get("schema") if isinstance(candidate, dict) else None + if schema == _SCHEMA_V2: + root = _bound_mapping(candidate, _ROOT_KEYS, "intent_hash") + storage_intent: StorageIntent | None = None + execution_context_hash: str | None = None + elif schema == _SCHEMA_V3: assignment_candidate = candidate.get("assignment") if ( "storage_intent" not in candidate @@ -419,8 +425,22 @@ def _parse(candidate: object) -> ParsedHostExecutionIntent | None: ) ): raise StorageIntentError(STORAGE_TARGET_KEY_INVALID) - root = _bound_mapping(candidate, _ROOT_STORAGE_KEYS, "intent_hash") - if root is None or _text(root, "schema") != _SCHEMA: + root = _bound_mapping(candidate, _ROOT_STORAGE_KEYS, "intent_hash") + if root is None: + raise StorageIntentError(STORAGE_TARGET_KEY_INVALID) + try: + storage_intent = parse_storage_intent(root["storage_intent"]) + except StorageIntentError: + raise + except (AttributeError, KeyError, TypeError, ValueError, UnicodeError) as error: + raise StorageIntentError(STORAGE_TARGET_KEY_INVALID) from error + raw_execution_context_hash = root["execution_context_hash"] + if not _is_hash_value(raw_execution_context_hash): + raise StorageIntentError(STORAGE_TARGET_KEY_INVALID) + execution_context_hash = cast(str, raw_execution_context_hash) + else: + return None + if root is None or _text(root, "schema") != schema: return None projection_hash = _valid_hash(root, "projection_hash") @@ -516,18 +536,10 @@ def _parse(candidate: object) -> ParsedHostExecutionIntent | None: active_lease_set_hash, ) = validated_predecessor - try: - storage_intent = parse_storage_intent(root["storage_intent"]) - except StorageIntentError: - raise - except (AttributeError, KeyError, TypeError, ValueError, UnicodeError) as error: - raise StorageIntentError(STORAGE_TARGET_KEY_INVALID) from error - raw_execution_context_hash = root["execution_context_hash"] - if not _is_hash_value(raw_execution_context_hash): - raise StorageIntentError(STORAGE_TARGET_KEY_INVALID) - execution_context_hash = cast(str, raw_execution_context_hash) - if ( - storage_intent.task_id != task_id + if schema == _SCHEMA_V3 and ( + storage_intent is None + or execution_context_hash is None + or storage_intent.task_id != task_id or storage_intent.plan_binding != source_plan_hash or storage_intent.context_hash != execution_context_hash ): @@ -597,7 +609,7 @@ def _parse(candidate: object) -> ParsedHostExecutionIntent | None: return None return ParsedHostExecutionIntent( - schema=_SCHEMA, + schema=cast(str, schema), intent_hash=intent_hash, projection_hash=projection_hash, source_plan_hash=source_plan_hash, diff --git a/mcp-tools/devkit_runtime/host_bridge.py b/mcp-tools/devkit_runtime/host_bridge.py index de04139..2c618a6 100644 --- a/mcp-tools/devkit_runtime/host_bridge.py +++ b/mcp-tools/devkit_runtime/host_bridge.py @@ -45,6 +45,10 @@ _COMPILER_EVIDENCE_RESPONSE_SCHEMA: Final = ( "2718lab-devkit/compiler-evidence-response-v1" ) +_STORAGE_PROFILE_REQUEST_SCHEMA: Final = ( + "2718lab-devkit/storage-profile-request-v1" +) +_STORAGE_PROFILE_SCHEMA: Final = "2718lab-devkit/storage-profile-v1" _PROJECT_INDEX_ATTESTATION_SCHEMA: Final = ( project_index_attestation_protocol.ATTESTATION_SCHEMA ) @@ -74,6 +78,47 @@ _MAX_TERMINAL_OPERATION_TOMBSTONES: Final = 256 _MAX_COMPILER_EVIDENCE_BYTES: Final = 40 * 1024 _COMPILER_EVIDENCE_TTL_SECONDS: Final = 120 +_MAX_STORAGE_PROFILE_BYTES: Final = 8 * 1024 +_STORAGE_DESCRIPTOR_FIELDS: Final = ( + "repository_identity", + "workspace_manifest_hash", + "cargo_lock_hash", + "toolchain_digest", + "target_triple", + "profile", + "features_hash", + "build_env_class", +) +_STORAGE_PROFILE_FIELDS: Final = frozenset( + { + "schema", + "call_intent_hash", + "preparation_id", + "task_id", + "source_plan_hash", + "index_attestation_hash", + "execution_context_hash", + *_STORAGE_DESCRIPTOR_FIELDS, + "profile_hash", + "attestation_hash", + } +) +_STORAGE_PROFILE_REQUEST_FIELDS: Final = frozenset( + { + "schema", + "call_intent_hash", + "preparation_id", + "task_id", + "source_plan_hash", + "index_attestation_hash", + "nonce", + "request_hash", + } +) +_STORAGE_PROFILE_BUILD_ENV_CLASSES: Final = frozenset( + {"managed_read_only", "managed_workspace", "disabled", "external"} +) +_STORAGE_PROFILE_SCALAR: Final = re.compile(r"[A-Za-z0-9][A-Za-z0-9._+-]{0,127}\Z") _CAPABILITY_V2_TTL_SECONDS: Final = 120 _MAX_PROJECT_INDEX_ATTESTATION_BYTES: Final = ( project_index_attestation_protocol.MAX_ATTESTATION_BYTES @@ -113,6 +158,8 @@ "proof_continuation", "compiler_evidence_request", "compiler_evidence_response", + "storage_profile_request", + "storage_profile_response", "project_index_attestation", "routing_attestation_request", "routing_attestation_response", @@ -242,6 +289,19 @@ class CompilerEvidenceRequest: expires_at: int +@dataclass(frozen=True) +class StorageProfileRequest: + """One private, replay-bound request for Host-owned storage profile facts.""" + + call_intent_hash: str + preparation_id: str + task_id: str + source_plan_hash: str + index_attestation_hash: str + nonce: str = field(repr=False) + request_hash: str + + @dataclass(frozen=True) class FastLaneRefillRegistryRequest: """One authenticated queue of remaining V5 skeletons. @@ -342,6 +402,8 @@ def __init__( self._received_operations: dict[str, OperationReceipt] = {} self._pending_compiler_evidence: dict[str, CompilerEvidenceRequest] = {} self._received_compiler_evidence: set[str] = set() + self._pending_storage_profiles: dict[str, StorageProfileRequest] = {} + self._received_storage_profiles: set[str] = set() self._sent_fast_lane_refill_registries: set[str] = set() self._received_fast_lane_refill_registries: set[str] = set() self._received_project_index_attestations: set[str] = set() @@ -1383,6 +1445,109 @@ def receive_compiler_evidence_response( del self._pending_compiler_evidence[request.preparation_id] return normalized + def send_storage_profile_request( + self, + *, + call_intent_hash: str, + preparation_id: str, + task_id: str, + source_plan_hash: str, + index_attestation_hash: str, + ) -> StorageProfileRequest: + """Ask the Host for one path-free profile tied to a private session.""" + + nonce = _b64encode(secrets.token_bytes(32)) + unsigned = { + "schema": _STORAGE_PROFILE_REQUEST_SCHEMA, + "call_intent_hash": call_intent_hash, + "preparation_id": preparation_id, + "task_id": task_id, + "source_plan_hash": source_plan_hash, + "index_attestation_hash": index_attestation_hash, + "nonce": nonce, + } + request = _normalize_storage_profile_request( + {**unsigned, "request_hash": _private_payload_hash(unsigned)} + ) + if request.request_hash in self._pending_storage_profiles: + raise HostBridgeError("HOST_BRIDGE_STORAGE_PROFILE_INVALID") + payload = _storage_profile_request_payload(request) + _validate_private_packet_size(payload, _MAX_STORAGE_PROFILE_BYTES) + self._send_validated_private( + kind="storage_profile_request", + action_id=_storage_profile_action_id(request), + payload=payload, + ) + self._pending_storage_profiles[request.request_hash] = request + return request + + def receive_storage_profile_request(self) -> StorageProfileRequest: + """Receive exactly one Host-bound storage profile request once.""" + + try: + message = self._receive_private() + request = _parse_storage_profile_request(message) + if request.request_hash in self._received_storage_profiles: + raise HostBridgeError("HOST_BRIDGE_STORAGE_PROFILE_INVALID") + except HostBridgeError: + self._poison() + raise + self._received_storage_profiles.add(request.request_hash) + return request + + def send_storage_profile_response( + self, + *, + request: StorageProfileRequest, + response: Mapping[str, object], + ) -> None: + """Return one exact Host profile to the request's private session.""" + + normalized_request = _normalize_storage_profile_request( + _storage_profile_request_payload(request) + ) + if normalized_request.request_hash not in self._received_storage_profiles: + raise HostBridgeError("HOST_BRIDGE_STORAGE_PROFILE_INVALID") + normalized = _normalize_storage_profile_response( + response, request=normalized_request + ) + _validate_private_packet_size(normalized, _MAX_STORAGE_PROFILE_BYTES) + self._send_validated_private( + kind="storage_profile_response", + action_id=_storage_profile_action_id(normalized_request), + payload=normalized, + ) + self._received_storage_profiles.remove(normalized_request.request_hash) + + def receive_storage_profile_response( + self, *, request: StorageProfileRequest + ) -> dict[str, object]: + """Consume one exact response bound to its still-pending request.""" + + normalized_request = _normalize_storage_profile_request( + _storage_profile_request_payload(request) + ) + if ( + self._pending_storage_profiles.get(normalized_request.request_hash) + != normalized_request + ): + raise HostBridgeError("HOST_BRIDGE_STORAGE_PROFILE_INVALID") + try: + message = self._receive_private() + if ( + message.kind != "storage_profile_response" + or message.action_id != _storage_profile_action_id(normalized_request) + ): + raise HostBridgeError("HOST_BRIDGE_STORAGE_PROFILE_INVALID") + normalized = _normalize_storage_profile_response( + message.payload, request=normalized_request + ) + except HostBridgeError: + self._poison() + raise + del self._pending_storage_profiles[normalized_request.request_hash] + return normalized + def receive_operation( self, *, @@ -2841,6 +3006,174 @@ def _is_index_correlation(value: object) -> bool: return project_index_attestation_protocol.is_index_correlation(value) +def _normalize_storage_profile_request(value: object) -> StorageProfileRequest: + """Validate one exact, nonce-bound private profile request.""" + + if ( + type(value) is not dict + or set(value) != _STORAGE_PROFILE_REQUEST_FIELDS + or value.get("schema") != _STORAGE_PROFILE_REQUEST_SCHEMA + ): + raise HostBridgeError("HOST_BRIDGE_STORAGE_PROFILE_INVALID") + call_intent_hash = value.get("call_intent_hash") + preparation_id = value.get("preparation_id") + task_id = value.get("task_id") + source_plan_hash = value.get("source_plan_hash") + index_attestation_hash = value.get("index_attestation_hash") + nonce = value.get("nonce") + request_hash = value.get("request_hash") + if ( + type(call_intent_hash) is not str + or len(call_intent_hash) != 64 + or any(character not in "0123456789abcdef" for character in call_intent_hash) + or type(preparation_id) is not str + or _IDENTIFIER.fullmatch(preparation_id) is None + or type(task_id) is not str + or _FAST_LANE_TASK_ID.fullmatch(task_id) is None + or type(source_plan_hash) is not str + or _DIGEST.fullmatch(source_plan_hash) is None + or type(index_attestation_hash) is not str + or _DIGEST.fullmatch(index_attestation_hash) is None + or type(nonce) is not str + or type(request_hash) is not str + or _DIGEST.fullmatch(request_hash) is None + ): + raise HostBridgeError("HOST_BRIDGE_STORAGE_PROFILE_INVALID") + try: + decoded_nonce = _b64decode(nonce) + except ValueError as error: + raise HostBridgeError("HOST_BRIDGE_STORAGE_PROFILE_INVALID") from error + if len(decoded_nonce) != 32 or _b64encode(decoded_nonce) != nonce: + raise HostBridgeError("HOST_BRIDGE_STORAGE_PROFILE_INVALID") + unsigned = dict(value) + unsigned.pop("request_hash") + if not hmac.compare_digest(request_hash, _private_payload_hash(unsigned)): + raise HostBridgeError("HOST_BRIDGE_STORAGE_PROFILE_INVALID") + return StorageProfileRequest( + call_intent_hash=call_intent_hash, + preparation_id=preparation_id, + task_id=task_id, + source_plan_hash=source_plan_hash, + index_attestation_hash=index_attestation_hash, + nonce=nonce, + request_hash=request_hash, + ) + + +def _storage_profile_request_payload( + request: StorageProfileRequest, +) -> dict[str, object]: + if type(request) is not StorageProfileRequest: + raise HostBridgeError("HOST_BRIDGE_STORAGE_PROFILE_INVALID") + return { + "schema": _STORAGE_PROFILE_REQUEST_SCHEMA, + "call_intent_hash": request.call_intent_hash, + "preparation_id": request.preparation_id, + "task_id": request.task_id, + "source_plan_hash": request.source_plan_hash, + "index_attestation_hash": request.index_attestation_hash, + "nonce": request.nonce, + "request_hash": request.request_hash, + } + + +def _storage_profile_action_id(request: StorageProfileRequest) -> str: + normalized = _normalize_storage_profile_request( + _storage_profile_request_payload(request) + ) + return normalized.request_hash[7:] + + +def _parse_storage_profile_request( + message: PrivateHostMessage, +) -> StorageProfileRequest: + if message.kind != "storage_profile_request": + raise HostBridgeError("HOST_BRIDGE_STORAGE_PROFILE_INVALID") + request = _normalize_storage_profile_request(message.payload) + if message.action_id != _storage_profile_action_id(request): + raise HostBridgeError("HOST_BRIDGE_STORAGE_PROFILE_INVALID") + _validate_private_packet_size(message.payload, _MAX_STORAGE_PROFILE_BYTES) + return request + + +def _normalize_storage_profile_response( + value: object, + *, + request: StorageProfileRequest, +) -> dict[str, object]: + """Validate the exact Host-owned profile and its request bindings. + + ``attestation_hash`` deliberately remains opaque: the Host includes its + private bridge generation and expiry deadline in that attestation, neither + of which crosses this Python protocol boundary. The authenticated frame, + pending request, and exact profile hash still make substitution fail closed. + """ + + if type(request) is not StorageProfileRequest: + raise HostBridgeError("HOST_BRIDGE_STORAGE_PROFILE_INVALID") + normalized_request = _normalize_storage_profile_request( + _storage_profile_request_payload(request) + ) + if ( + type(value) is not dict + or set(value) != _STORAGE_PROFILE_FIELDS + or value.get("schema") != _STORAGE_PROFILE_SCHEMA + or value.get("call_intent_hash") != normalized_request.call_intent_hash + or value.get("preparation_id") != normalized_request.preparation_id + or value.get("task_id") != normalized_request.task_id + or value.get("source_plan_hash") != normalized_request.source_plan_hash + or value.get("index_attestation_hash") + != normalized_request.index_attestation_hash + ): + raise HostBridgeError("HOST_BRIDGE_STORAGE_PROFILE_INVALID") + call_intent_hash = value.get("call_intent_hash") + preparation_id = value.get("preparation_id") + task_id = value.get("task_id") + if ( + type(call_intent_hash) is not str + or len(call_intent_hash) != 64 + or any(character not in "0123456789abcdef" for character in call_intent_hash) + or type(preparation_id) is not str + or _IDENTIFIER.fullmatch(preparation_id) is None + or type(task_id) is not str + or _FAST_LANE_TASK_ID.fullmatch(task_id) is None + ): + raise HostBridgeError("HOST_BRIDGE_STORAGE_PROFILE_INVALID") + for field_name in ( + "source_plan_hash", + "index_attestation_hash", + "execution_context_hash", + "repository_identity", + "workspace_manifest_hash", + "cargo_lock_hash", + "toolchain_digest", + "features_hash", + "profile_hash", + "attestation_hash", + ): + item = value.get(field_name) + if type(item) is not str or _DIGEST.fullmatch(item) is None: + raise HostBridgeError("HOST_BRIDGE_STORAGE_PROFILE_INVALID") + target_triple = value.get("target_triple") + if ( + type(target_triple) is not str + or _STORAGE_PROFILE_SCALAR.fullmatch(target_triple) is None + or value.get("profile") != "dev" + ): + raise HostBridgeError("HOST_BRIDGE_STORAGE_PROFILE_INVALID") + if value.get("build_env_class") not in _STORAGE_PROFILE_BUILD_ENV_CLASSES: + raise HostBridgeError("HOST_BRIDGE_STORAGE_PROFILE_INVALID") + unsigned = { + field_name: value[field_name] + for field_name in _STORAGE_PROFILE_FIELDS + if field_name not in {"profile_hash", "attestation_hash"} + } + profile_hash = cast(str, value["profile_hash"]) + if not hmac.compare_digest(profile_hash, _private_payload_hash(unsigned)): + raise HostBridgeError("HOST_BRIDGE_STORAGE_PROFILE_INVALID") + return {field_name: value[field_name] for field_name in sorted(_STORAGE_PROFILE_FIELDS)} + + def _normalize_compiler_evidence_response( value: object, *, request: CompilerEvidenceRequest, now: int ) -> dict[str, object]: diff --git a/mcp-tools/devkit_runtime/host_session.py b/mcp-tools/devkit_runtime/host_session.py index dc6ec9e..7f81f75 100644 --- a/mcp-tools/devkit_runtime/host_session.py +++ b/mcp-tools/devkit_runtime/host_session.py @@ -33,11 +33,39 @@ _NO_SAFE_WORK: Final = "NO_SAFE_WORK" _HASH_PREFIX: Final = "sha256:" _IDENTIFIER: Final = re.compile(r"[a-z0-9][a-z0-9._-]{0,127}\Z") +_FAST_LANE_TASK_ID: Final = re.compile(r"[A-Za-z0-9][A-Za-z0-9._:-]{0,95}\Z") +_RAW_HASH: Final = re.compile(r"[0-9a-f]{64}\Z") _REASON_SESSION_UNAVAILABLE: Final = "HOST_SESSION_UNAVAILABLE" _REASON_CAPABILITY_UNAVAILABLE: Final = "HOST_CAPABILITY_UNAVAILABLE" _REASON_EXECUTION_EVIDENCE_UNAVAILABLE: Final = "HOST_EXECUTION_EVIDENCE_UNAVAILABLE" _MAX_HOST_WRITER_ACTIONS: Final = 9 _MAX_HOST_READER_ACTIONS: Final = 189 +_STORAGE_PROFILE_SCHEMA: Final = "2718lab-devkit/storage-profile-v1" +_STORAGE_PROFILE_FIELDS: Final = frozenset( + { + "schema", + "call_intent_hash", + "preparation_id", + "task_id", + "source_plan_hash", + "index_attestation_hash", + "execution_context_hash", + "repository_identity", + "workspace_manifest_hash", + "cargo_lock_hash", + "toolchain_digest", + "target_triple", + "profile", + "features_hash", + "build_env_class", + "profile_hash", + "attestation_hash", + } +) +_STORAGE_PROFILE_BUILD_ENV_CLASSES: Final = frozenset( + {"managed_read_only", "managed_workspace", "disabled", "external"} +) +_STORAGE_PROFILE_SCALAR: Final = re.compile(r"[A-Za-z0-9][A-Za-z0-9._+-]{0,127}\Z") class HostCapabilityState(StrEnum): @@ -148,6 +176,7 @@ class _CompilerInvocationBinding: verified_lease_scope_bindings: tuple[str, ...] dispatch_facts: tuple[object, ...] = () dispatch_binding_hashes: tuple[str, ...] = () + storage_profiles: tuple[dict[str, object], ...] = field(default=(), repr=False) registry_binding_hash: str | None = None evidence_expires_at: int | None = None @@ -174,6 +203,7 @@ class _CompilerInvocation: expires_at: float binding_hash: str registry_binding_hash: str | None = None + storage_profiles: tuple[dict[str, object], ...] = field(default=(), repr=False) @dataclass(frozen=True) @@ -182,9 +212,10 @@ class _CompilerRequestContext: request_hash: str reasoning_effort: str requested_routes: tuple[HostRoute, ...] + routing_registry_binding_hash: str assignment_skeletons: tuple[dict[str, object], ...] = field(repr=False) project_index_attestation_refs: tuple[dict[str, object], ...] = field(repr=False) - routing_registry_binding_hash: str + storage_task_ids: tuple[str, ...] = () @dataclass(frozen=True) @@ -540,6 +571,8 @@ def prepare_compiler_evidence( invocation_binding["dispatch_binding_hashes"] = ( binding.dispatch_binding_hashes ) + if binding.storage_profiles: + invocation_binding["storage_profiles"] = binding.storage_profiles material = _CompilerInvocation( schema="2718lab-devkit/compiler-invocation-v2", preparation_id=preparation_id, @@ -553,6 +586,7 @@ def prepare_compiler_evidence( expires_at=expires_at, binding_hash=_hash(invocation_binding), registry_binding_hash=binding.registry_binding_hash, + storage_profiles=binding.storage_profiles, ) material_state = _compiler_invocation_state(material) try: @@ -588,6 +622,7 @@ def bind_compiler_request( assignment_skeletons: tuple[dict[str, object], ...], project_index_attestation_refs: tuple[dict[str, object], ...], routing_registry_binding_hash: str, + storage_task_ids: tuple[str, ...] = (), ) -> bool: """Bind public request identity before a private registry round trip.""" @@ -604,13 +639,27 @@ def bind_compiler_request( or not _is_hash(request_hash) or not _is_hash(routing_registry_binding_hash) or reasoning_effort not in {"low", "medium", "high", "xhigh", "max"} + or type(storage_task_ids) is not tuple + or len(storage_task_ids) > 16 + or any( + type(task_id) is not str + or _FAST_LANE_TASK_ID.fullmatch(task_id) is None + for task_id in storage_task_ids + ) + or len(set(storage_task_ids)) != len(storage_task_ids) ): return False try: normalized_routes = _normalized_routes(requested_routes) + skeleton_task_ids = tuple( + _required_fast_lane_task_id(_mapping(item).get("task_id")) + for item in assignment_skeletons + ) trusted_now = int(self._read_trusted_clock()) except (TypeError, ValueError): return False + if storage_task_ids and storage_task_ids != skeleton_task_ids: + return False routing_snapshot = self._routing_attestation_snapshots.get( (call_intent_hash, preparation_id) ) @@ -631,6 +680,7 @@ def bind_compiler_request( assignment_skeletons=assignment_skeletons, project_index_attestation_refs=project_index_attestation_refs, routing_registry_binding_hash=routing_registry_binding_hash, + storage_task_ids=storage_task_ids, ) return True @@ -665,6 +715,53 @@ def _resolve_bridge_compiler_invocation( route_hashes = cast(list[object], response["verified_route_result_hashes"]) lease_hashes = cast(list[object], response["verified_lease_scope_bindings"]) dispatch_hashes = cast(list[object], response["dispatch_binding_hashes"]) + dispatch_facts = tuple( + _dispatch_fact_from_mapping(value) for value in facts_value + ) + storage_profiles: tuple[dict[str, object], ...] = () + if context.storage_task_ids: + skeleton_task_ids = tuple( + _required_fast_lane_task_id(_mapping(item).get("task_id")) + for item in context.assignment_skeletons + ) + if skeleton_task_ids != context.storage_task_ids: + return None + profiles: list[dict[str, object]] = [] + for skeleton, index_ref, fact in zip( + context.assignment_skeletons, + context.project_index_attestation_refs, + dispatch_facts, + strict=True, + ): + skeleton_mapping = _mapping(skeleton) + index_mapping = _mapping(index_ref) + task_id = _required_fast_lane_task_id( + skeleton_mapping.get("task_id") + ) + source_plan_hash = _required_hash( + skeleton_mapping.get("source_plan_hash") + ) + index_attestation_hash = _required_hash( + index_mapping.get("attestation_hash") + ) + if ( + index_mapping.get("task_id") != task_id + or fact.task_id != task_id + or fact.source_plan_hash != source_plan_hash + or fact.index_context_hash != skeleton_mapping.get("index_context_hash") + ): + return None + profile_request = bridge.send_storage_profile_request( + call_intent_hash=context.call_intent_hash, + preparation_id=preparation_id, + task_id=task_id, + source_plan_hash=source_plan_hash, + index_attestation_hash=index_attestation_hash, + ) + profiles.append( + bridge.receive_storage_profile_response(request=profile_request) + ) + storage_profiles = tuple(profiles) return _CompilerInvocationBinding( request_hash=_required_hash(response["request_hash"]), reasoning_effort=str(response["reasoning_effort"]), @@ -676,14 +773,13 @@ def _resolve_bridge_compiler_invocation( _required_hash(value) for value in lease_hashes ), - dispatch_facts=tuple( - _dispatch_fact_from_mapping(value) for value in facts_value - ), + dispatch_facts=dispatch_facts, dispatch_binding_hashes=tuple( _required_hash(value) for value in dispatch_hashes ), registry_binding_hash=_required_hash(response["registry_binding_hash"]), evidence_expires_at=_required_positive_int(response["expires_at"]), + storage_profiles=storage_profiles, ) def send_project_index_attestation( @@ -1695,6 +1791,12 @@ def _required_hash(value: object) -> str: return value +def _required_fast_lane_task_id(value: object) -> str: + if type(value) is not str or _FAST_LANE_TASK_ID.fullmatch(value) is None: + raise ValueError("expected Fast Lane task id") + return value + + def _is_hash(value: object) -> bool: return ( isinstance(value, str) @@ -1736,6 +1838,68 @@ def _optional_ordered_hash_tuple(value: object) -> bool: ) +def _normalized_storage_profiles( + value: object, +) -> tuple[dict[str, object], ...]: + if type(value) is not tuple or len(value) > 16: + raise ValueError("compiler storage profiles are invalid") + normalized: list[dict[str, object]] = [] + for profile in value: + mapping = dict(_mapping(profile)) + if set(mapping) != _STORAGE_PROFILE_FIELDS: + raise ValueError("compiler storage profile fields are invalid") + if mapping.get("schema") != _STORAGE_PROFILE_SCHEMA: + raise ValueError("compiler storage profile schema is invalid") + call_intent_hash = mapping.get("call_intent_hash") + preparation_id = mapping.get("preparation_id") + task_id = mapping.get("task_id") + if ( + type(call_intent_hash) is not str + or _RAW_HASH.fullmatch(call_intent_hash) is None + or type(preparation_id) is not str + or _IDENTIFIER.fullmatch(preparation_id) is None + or type(task_id) is not str + or _FAST_LANE_TASK_ID.fullmatch(task_id) is None + ): + raise ValueError("compiler storage profile binding is invalid") + target_triple = mapping.get("target_triple") + if ( + type(target_triple) is not str + or _STORAGE_PROFILE_SCALAR.fullmatch(target_triple) is None + or mapping.get("profile") != "dev" + ): + raise ValueError("compiler storage profile scalar is invalid") + for field_name in ( + "source_plan_hash", + "index_attestation_hash", + "execution_context_hash", + "repository_identity", + "workspace_manifest_hash", + "cargo_lock_hash", + "toolchain_digest", + "features_hash", + "profile_hash", + "attestation_hash", + ): + _required_hash(mapping.get(field_name)) + if mapping.get("build_env_class") not in _STORAGE_PROFILE_BUILD_ENV_CLASSES: + raise ValueError("compiler storage profile build environment is invalid") + unsigned = { + field_name: mapping[field_name] + for field_name in _STORAGE_PROFILE_FIELDS + if field_name not in {"profile_hash", "attestation_hash"} + } + if mapping["profile_hash"] != _hash(unsigned): + raise ValueError("compiler storage profile hash is invalid") + normalized.append( + {field_name: mapping[field_name] for field_name in sorted(_STORAGE_PROFILE_FIELDS)} + ) + task_ids = [cast(str, profile["task_id"]) for profile in normalized] + if len(task_ids) != len(set(task_ids)): + raise ValueError("compiler storage profile tasks are duplicated") + return tuple(normalized) + + def _normalized_compiler_invocation_binding( value: object, ) -> _CompilerInvocationBinding: @@ -1750,6 +1914,8 @@ def _normalized_compiler_invocation_binding( or len(value.dispatch_facts) > 16 or not _optional_ordered_hash_tuple(value.dispatch_binding_hashes) or len(value.dispatch_facts) != len(value.dispatch_binding_hashes) + or type(value.storage_profiles) is not tuple + or len(value.storage_profiles) > 16 or ( value.registry_binding_hash is not None and not _is_hash(value.registry_binding_hash) @@ -1760,6 +1926,7 @@ def _normalized_compiler_invocation_binding( ) ): raise ValueError("compiler invocation binding is invalid") + storage_profiles = _normalized_storage_profiles(value.storage_profiles) return _CompilerInvocationBinding( request_hash=value.request_hash, reasoning_effort=value.reasoning_effort, @@ -1767,6 +1934,7 @@ def _normalized_compiler_invocation_binding( verified_lease_scope_bindings=value.verified_lease_scope_bindings, dispatch_facts=value.dispatch_facts, dispatch_binding_hashes=value.dispatch_binding_hashes, + storage_profiles=storage_profiles, registry_binding_hash=value.registry_binding_hash, evidence_expires_at=value.evidence_expires_at, ) @@ -1784,6 +1952,7 @@ def _compiler_invocation_state(value: _CompilerInvocation) -> tuple[object, ...] value.verified_lease_scope_bindings, value.dispatch_facts, value.dispatch_binding_hashes, + tuple(_hash(profile) for profile in value.storage_profiles), value.issued_at, value.expires_at, value.binding_hash, diff --git a/mcp-tools/tests/test_storage_firewall.py b/mcp-tools/tests/test_storage_firewall.py index 54475df..fdb2fcc 100644 --- a/mcp-tools/tests/test_storage_firewall.py +++ b/mcp-tools/tests/test_storage_firewall.py @@ -1,132 +1,316 @@ from __future__ import annotations +import base64 import copy import hashlib -import importlib.util import json import os import sys +import threading from pathlib import Path import pytest +MCP_TOOLS = Path(__file__).resolve().parents[1] +TESTS = Path(__file__).resolve().parent +for _path in (MCP_TOOLS, TESTS): + if str(_path) not in sys.path: + sys.path.insert(0, str(_path)) + + +def _hash(character: str) -> str: + return "sha256:" + character * 64 + + def _canonical_hash(value: object) -> str: - encoded = json.dumps( - value, - ensure_ascii=True, - sort_keys=True, - separators=(",", ":"), - ).encode("utf-8") - return "sha256:" + hashlib.sha256(encoded).hexdigest() + return ( + "sha256:" + + hashlib.sha256( + json.dumps( + value, + ensure_ascii=False, + sort_keys=True, + separators=(",", ":"), + allow_nan=False, + ).encode("utf-8") + ).hexdigest() + ) -def _intent_with_unknown_descriptor_key( - key: str, value: str -) -> dict[str, object]: - descriptor: dict[str, str] = { - "schema": "2718lab.storage.target.v1", - "artifact_kind": "cargo-target", - "repository_identity": "sha256:" + "d" * 64, - "workspace_manifest_hash": "sha256:" + "e" * 64, - "cargo_lock_hash": "sha256:" + "f" * 64, - "toolchain_digest": "sha256:" + "1" * 64, - "target_triple": "x86_64-pc-windows-msvc", - "profile": "dev", - "features_hash": "sha256:" + "2" * 64, - "build_env_class": "windows-msvc", +def _authenticated_v5_fixture() -> dict[str, object]: + """Build a canonical legacy V5 skeleton without any storage field.""" + + from devkit_fastlane.scripts import fastlane_routing, team_efficiency + + hash_a = _canonical_hash({"fixture": "a"}) + hash_b = _canonical_hash({"fixture": "b"}) + source_plan_hash = _canonical_hash({"fixture": "plan"}) + dependency: dict[str, object] = { + "schema": "2718lab-devkit/dependency-state-v1", + "graph_epoch": 1, + "direct_dependency_ids": [], + "completed_dependency_ids": [], } - intent: dict[str, object] = { - "schema": "2718lab.storage.intent.v1", - "task_id": "task-01", - "plan_binding": "sha256:" + "a" * 64, - "context_hash": "sha256:" + "b" * 64, - "requested_bytes": 1, - "requested_files": 1, - "target_descriptor": descriptor, + dependency["dependency_state_hash"] = _canonical_hash(dependency) + scheduler = { + "event_seq": 1, + "route_epoch": 1, + "override_epoch": 0, + "recovery_epoch": 0, + "ready_event_seq": 1, + "dispatch_cause": "task_ready", + "transport_state": "connected", + "execution_state": "unknown", + "lease_state": "unclaimed", + "evidence_state": "none", + "lease_epoch": 0, + "recovery_probe_count_epoch": 0, + "fence_count_epoch": 0, + "fenced_replacement_count_task": 0, } - intent["storage_intent_hash"] = _canonical_hash( + host = { + "schema": "2718lab-devkit/host-capabilities-v1", + "host_id_hash": hash_a, + "capability_epoch": 1, + "total_slots": 4, + "model_slot_limits": {"luna": 4, "terra": 0, "sol": 0, "spark": 0}, + "models": [ + { + "model_id": "gpt-5.6-luna", + "status": "available", + "efforts": ["max"], + } + ], + "entitlements": [], + } + unit = { + "task": { + "schema": "2718lab-devkit/task-routing-profile-v5", + "task_id": "TASK-V5", + "role": "execution", + "access": "workspace_write", + "write_scope_count": 1, + "write_scope_breadth": "single_file", + "read_scope_count": 0, + "read_scope_breadth": "none", + "overlap_risk": "none", + "overlap_count": 0, + "dependency_depth": 0, + "downstream_critical_count": 0, + "critical_path": False, + "criticality": "low", + "cross_module": False, + "database_work": False, + "migration": False, + "security_sensitive": False, + "destructive": False, + "external_boundary": False, + "architecture_conflict": False, + "design_ambiguity": False, + "verification_cost": "none", + "blocker_severity": "none", + "authorization": "not_required", + "authorization_evidence_hash": None, + "narrow_decoupling_eligible": False, + "strike": None, + "gate_matrix_hash": hash_a, + "profile_evidence_hash": hash_b, + }, + "dependency_state": dependency, + "write_scope": ["src/task_v5.py"], + "concurrency_mode": "parallel", + "dispatch_order": 0, + "index_context_hash": hash_a, + "predecessor_hash": hash_b, + } + api = team_efficiency._AuthenticatedV5Api() + planner = team_efficiency._authenticated_v5_helper_module("authenticated_v5_planner") + normalized_unit = planner.normalize_units(api, [unit])[0] + unit["task"]["profile_evidence_hash"] = team_efficiency._sha256_json( + planner._routing_profile_material(source_plan_hash, normalized_unit) + ) + routing_requests = team_efficiency.prepare_authenticated_v5_routing_requests( + [unit], + source_plan_hash=source_plan_hash, + host_capabilities=host, + scheduler_facts=scheduler, + ) + request_binding_hash = fastlane_routing.v5_request_binding_hash(routing_requests[0]) + attestation: dict[str, object] = { + "schema": "2718lab-devkit/host-child-route-attestation-v1", + "status": "attested", + "request_binding_hash": request_binding_hash, + "host_id_hash": hash_a, + "capability_epoch": 1, + "lease_epoch": 0, + "issued_event_seq": 1, + "expires_event_seq": 1, + "route": { + "lane": "luna", + "model": "gpt-5.6-luna", + "effort": "max", + "rank": 40, + }, + "inherit_current_session_model": False, + "refusal_code": None, + } + attestation["attestation_hash"] = _canonical_hash(attestation) + attestation_items = [ { - "target_descriptor": descriptor, - "task_id": intent["task_id"], - "plan_binding": intent["plan_binding"], - "context_hash": intent["context_hash"], - "requested_bytes": intent["requested_bytes"], - "requested_files": intent["requested_files"], + "task_id": "TASK-V5", + "request_binding_hash": request_binding_hash, + "attestation": attestation, } + ] + compiled = team_efficiency.compile_authenticated_v5_assignment_skeletons( + [unit], + source_plan_hash=source_plan_hash, + routing_requests=routing_requests, + attestation_items=attestation_items, ) - descriptor[key] = value - return intent - - -def _assert_storage_intent_rejected(value: dict[str, object]) -> None: - from devkit_runtime.storage_intent import StorageIntentError, parse_storage_intent - - try: - parse_storage_intent(value) - except StorageIntentError as error: - assert error.code == "STORAGE_TARGET_KEY_INVALID" - else: - raise AssertionError("invalid descriptor was accepted") + index_ref = { + "task_id": "TASK-V5", + "correlation_id": "index-" + "c" * 64, + "workspace_id": _hash("d"), + "workspace_binding_hash": _hash("e"), + "root_identity_hash": _hash("f"), + "snapshot_id": _hash("0"), + "snapshot_attestation_hash": _hash("1"), + "query_receipt_hash": _hash("6"), + "index_context_hash": hash_a, + "attestation_hash": _hash("7"), + } + return { + "call_intent_hash": "a" * 64, + "preparation_id": "dispatch-v5-1", + "host": host, + "scheduler": scheduler, + "source_plan_hash": source_plan_hash, + "routing_requests": routing_requests, + "attestation_items": attestation_items, + "compiled": compiled, + "planner_request": { + "schema": "2718lab-devkit/fastlane-host-planner-request-v1", + "action": "plan_dispatch", + "assignment_skeletons": compiled["assignment_skeletons"], + "project_index_attestation_refs": [index_ref], + }, + } -def test_storage_intent_rejects_absolute_path_and_unknown_descriptor_key() -> None: - _assert_storage_intent_rejected( - _intent_with_unknown_descriptor_key("path", "G:/unapproved") +def _v5_dispatch_fact(adapter: object, fixture: dict[str, object]) -> object: + skeleton = fixture["compiled"]["assignment_skeletons"][0] + proof = skeleton["routing_proof"] + result_route = proof["result"]["route"] + return adapter._HostDispatchFact( + task_id=skeleton["task_id"], + route=adapter._HostDispatchRoute( + model=result_route["model"], + reasoning_effort=result_route["effort"], + routing_context_hash=proof["routing_context_hash"], + routing_result_hash=proof["routing_result_hash"], + require_explicit_route=True, + ), + lease_id="lease-task-v5", + lease_epoch=1, + task_version=1, + assignment_token=_hash("3"), + write_scope=tuple(skeleton["write_scope"]), + concurrency_mode=skeleton["concurrency_mode"], + dispatch_order=skeleton["dispatch_order"], + index_context_hash=skeleton["index_context_hash"], + worktree_identity=_hash("4"), + worktree_base=_hash("5"), + integration_head=_hash("6"), + predecessor_hash=skeleton["predecessor_hash"], + source_plan_hash=skeleton["source_plan_hash"], + ledger_epoch=11, + active_lease_set_hash=_hash("b"), ) -def test_storage_intent_rejects_plain_unknown_descriptor_key() -> None: - _assert_storage_intent_rejected( - _intent_with_unknown_descriptor_key("unexpected", "cache") +def _pipe_pair() -> tuple[object, object]: + from devkit_runtime.host_bridge import InheritedHandleHostBridge + + child_to_host_read, child_to_host_write = os.pipe() + host_to_child_read, host_to_child_write = os.pipe() + return ( + InheritedHandleHostBridge.from_file_descriptors( + read_fd=host_to_child_read, + write_fd=child_to_host_write, + session_key=b"k" * 32, + session_nonce=b"fastlane-storage-private-nonce", + ), + InheritedHandleHostBridge.from_file_descriptors( + read_fd=child_to_host_read, + write_fd=host_to_child_write, + session_key=b"k" * 32, + session_nonce=b"fastlane-storage-private-nonce", + ), ) -def test_storage_intent_rejects_isolated_surrogate_with_stable_code() -> None: - value = _intent_with_unknown_descriptor_key("unexpected", "cache") - descriptor = value["target_descriptor"] - assert isinstance(descriptor, dict) - descriptor.pop("unexpected") - descriptor["target_triple"] = "\ud800" - - from devkit_runtime.storage_intent import StorageIntentError, parse_storage_intent - - try: - parse_storage_intent(value) - except StorageIntentError as error: - assert error.code == "STORAGE_TARGET_KEY_INVALID" - else: - raise AssertionError("invalid surrogate was accepted") - - -def _storage_binding_context() -> dict[str, object]: - return { - "execution_context_hash": "sha256:" + "4" * 64, - "repository_identity": "sha256:" + "5" * 64, - "workspace_manifest_hash": "sha256:" + "6" * 64, - "cargo_lock_hash": "sha256:" + "7" * 64, - "toolchain_digest": "sha256:" + "8" * 64, +def _storage_profile_response(request: object) -> dict[str, object]: + from devkit_runtime.host_bridge import StorageProfileRequest + + assert isinstance(request, StorageProfileRequest) + response: dict[str, object] = { + "schema": "2718lab-devkit/storage-profile-v1", + "call_intent_hash": request.call_intent_hash, + "preparation_id": request.preparation_id, + "task_id": request.task_id, + "source_plan_hash": request.source_plan_hash, + "index_attestation_hash": request.index_attestation_hash, + "repository_identity": _hash("1"), + "workspace_manifest_hash": _hash("2"), + "cargo_lock_hash": _hash("3"), + "toolchain_digest": _hash("4"), "target_triple": "x86_64-pc-windows-msvc", "profile": "dev", - "features_hash": "sha256:" + "9" * 64, - "build_env_class": "windows-msvc", + "features_hash": _hash("5"), + "build_env_class": "managed_workspace", + "execution_context_hash": _hash("6"), + } + response["profile_hash"] = _canonical_hash(response) + # The Host alone binds generation and TTL into this opaque attestation. + response["attestation_hash"] = _hash("7") + return response + + +def _storage_profile_request() -> object: + from devkit_runtime import host_bridge + + nonce = base64.urlsafe_b64encode(b"n" * 32).decode("ascii").rstrip("=") + request = { + "schema": "2718lab-devkit/storage-profile-request-v1", + "call_intent_hash": "a" * 64, + "preparation_id": "storage-profile-1", + "task_id": "TASK-V5", + "source_plan_hash": _hash("8"), + "index_attestation_hash": _hash("9"), + "nonce": nonce, } + request["request_hash"] = _canonical_hash(request) + return host_bridge._normalize_storage_profile_request(request) -def _host_storage_intent( +def _storage_intent( *, task_id: str, plan_binding: str, context_hash: str ) -> dict[str, object]: - context = _storage_binding_context() descriptor = { "schema": "2718lab.storage.target.v1", "artifact_kind": "fastlane-task", - **{ - key: value - for key, value in context.items() - if key != "execution_context_hash" - }, + "repository_identity": _hash("1"), + "workspace_manifest_hash": _hash("2"), + "cargo_lock_hash": _hash("3"), + "toolchain_digest": _hash("4"), + "target_triple": "x86_64-pc-windows-msvc", + "profile": "dev", + "features_hash": _hash("5"), + "build_env_class": "managed_workspace", } - intent = { + intent: dict[str, object] = { "schema": "2718lab.storage.intent.v1", "task_id": task_id, "plan_binding": plan_binding, @@ -137,25 +321,169 @@ def _host_storage_intent( } intent["storage_intent_hash"] = _canonical_hash( { - key: intent[key] - for key in ( - "target_descriptor", - "task_id", - "plan_binding", - "context_hash", - "requested_bytes", - "requested_files", - ) + "target_descriptor": descriptor, + "task_id": task_id, + "plan_binding": plan_binding, + "context_hash": context_hash, + "requested_bytes": 4096, + "requested_files": 8, } ) return intent -def test_host_intent_requires_one_canonical_storage_binding_and_typed_failures() -> None: +def test_public_request_rejects_caller_storage_descriptors() -> None: + from devkit_fastlane.scripts import authenticated_v5_projection as projection + + with pytest.raises(ValueError, match="STORAGE_TARGET_KEY_INVALID"): + projection._reject_public_storage_facts( + {"storage_contexts": {"TASK-V5": {"repository_identity": _hash("1")}}} + ) + with pytest.raises(ValueError, match="STORAGE_TARGET_KEY_INVALID"): + projection._reject_public_storage_facts( + [{"execution_context_hash": _hash("2")}] + ) + + +def test_pre_host_skeleton_remains_the_legacy_exact_eight_fields() -> None: + fixture = _authenticated_v5_fixture() + skeleton = fixture["compiled"]["assignment_skeletons"][0] + assert set(skeleton) == { + "task_id", + "routing_proof", + "write_scope", + "concurrency_mode", + "dispatch_order", + "index_context_hash", + "predecessor_hash", + "source_plan_hash", + } + + +def test_verified_private_profile_round_trip_constructs_local_intent( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """Exercise the framed Host bridge and local post-profile compilation.""" + + from devkit_runtime import fastlane_host_adapter as adapter + import devkit_runtime.host_session as host_session + from devkit_runtime.host_bridge import InheritedHandleHostBridge + + fixture = _authenticated_v5_fixture() + child, host = _pipe_pair() + fact = _v5_dispatch_fact(adapter, fixture) + fact_mapping = adapter._dispatch_fact_mapping(fact) + lease_hash = adapter._lease_scope_binding_hash(fact) + failure: list[BaseException] = [] + + def host_reply() -> None: + try: + probe = host.receive_capability_probe_v2(now=1_700_000_000) + host.send_capability_report_v2( + probe=probe, + host_capabilities=fixture["host"], + scheduler_facts=fixture["scheduler"], + now=1_700_000_000, + ) + routing_request = host.receive_routing_attestation_request(now=1_700_000_000) + host.send_routing_attestation_response( + request=routing_request, + attestations=fixture["attestation_items"], + now=1_700_000_000, + ) + evidence_request = host.receive_compiler_evidence_request(now=1_700_000_000) + response = { + "schema": "2718lab-devkit/compiler-evidence-response-v1", + "preparation_id": evidence_request.preparation_id, + "request_hash": evidence_request.request_hash, + "reasoning_effort": evidence_request.reasoning_effort, + "verified_route_result_hashes": [fact.route.routing_result_hash], + "verified_lease_scope_bindings": [lease_hash], + "dispatch_facts": [fact_mapping], + "dispatch_binding_hashes": [fact_mapping["dispatch_binding_hash"]], + "nonce": evidence_request.nonce, + "expires_at": evidence_request.expires_at, + } + response["registry_binding_hash"] = _canonical_hash(response) + host.send_compiler_evidence_response( + request=evidence_request, + response=response, + now=1_700_000_000, + ) + profile_request = host.receive_storage_profile_request() + host.send_storage_profile_response( + request=profile_request, + response=_storage_profile_response(profile_request), + ) + except BaseException as error: # report peer errors after the round trip + failure.append(error) + + worker = threading.Thread(target=host_reply, daemon=True) + worker.start() + monkeypatch.setattr( + InheritedHandleHostBridge, + "from_environment", + classmethod(lambda cls, environ=None, *, platform=None: child), + ) + session = host_session.HostSession.from_environment( + environ={}, platform="posix", clock=lambda: 1_700_000_000 + ) + try: + assert session.resolve_capability_snapshot_v2( + call_intent_hash=fixture["call_intent_hash"], + preparation_id=fixture["preparation_id"], + ) is not None + routing = session.resolve_routing_attestations( + call_intent_hash=fixture["call_intent_hash"], + preparation_id=fixture["preparation_id"], + routing_requests=fixture["routing_requests"], + ) + assert routing is not None + prepared = adapter.prepare_verified_host_facts( + session, + preparation_id=fixture["preparation_id"], + call_intent_hash=fixture["call_intent_hash"], + routing_registry_binding_hash=routing.routing_registry_binding_hash, + request=fixture["planner_request"], + reasoning_effort="max", + storage_budgets={"TASK-V5": {"bytes": 4096, "files": 8}}, + ) + assert type(prepared).__name__ == "_PreparedHostFacts" + batch = adapter.compile_fast_lane_with_host_facts( + fixture["planner_request"], + reasoning_effort="max", + verified_host_facts=prepared, + ) + assert isinstance(batch, dict) + assert "storage_intents" not in batch + finally: + session.close() + host.close() + worker.join(timeout=2) + assert not worker.is_alive() + assert not failure + + +def test_profile_tamper_or_missing_field_fails_closed() -> None: + from devkit_runtime import host_bridge + from devkit_runtime.host_bridge import HostBridgeError + + request = _storage_profile_request() + profile = _storage_profile_response(request) + tampered = copy.deepcopy(profile) + tampered["profile_hash"] = _hash("f") + with pytest.raises(HostBridgeError, match="HOST_BRIDGE_STORAGE_PROFILE_INVALID"): + host_bridge._normalize_storage_profile_response(tampered, request=request) + missing = copy.deepcopy(profile) + missing.pop("attestation_hash") + with pytest.raises(HostBridgeError, match="HOST_BRIDGE_STORAGE_PROFILE_INVALID"): + host_bridge._normalize_storage_profile_response(missing, request=request) + + +def test_v2_remains_compatible_while_v3_requires_one_root_storage_intent() -> None: from test_fastlane_host_intent import _intent, _with_binding from devkit_runtime.fastlane_host_intent import ( NO_SAFE_WORK, - STORAGE_TARGET_KEY_INVALID, StorageIntentError, parse_host_execution_intent, validate_host_execution_intent, @@ -163,195 +491,42 @@ def test_host_intent_requires_one_canonical_storage_binding_and_typed_failures() legacy = _intent() legacy_result = parse_host_execution_intent(legacy) - assert isinstance(legacy_result, StorageIntentError) - assert legacy_result.code == STORAGE_TARGET_KEY_INVALID + assert legacy_result != NO_SAFE_WORK + assert not isinstance(legacy_result, StorageIntentError) + assert legacy_result.schema.endswith("-v2") + assert legacy_result.storage_intent is None assert validate_host_execution_intent(legacy) is NO_SAFE_WORK candidate = _intent() + candidate["schema"] = "2718lab-devkit/fastlane-host-execution-intent-v3" task_id = candidate["assignment"]["predecessor"]["task_id"] - source_plan_hash = candidate["source_plan_hash"] - context_hash = "sha256:" + "4" * 64 - storage_intent = _host_storage_intent( + context_hash = _hash("a") + candidate["storage_intent"] = _storage_intent( task_id=task_id, - plan_binding=source_plan_hash, + plan_binding=candidate["source_plan_hash"], context_hash=context_hash, ) - candidate["storage_intent"] = storage_intent candidate["execution_context_hash"] = context_hash - candidate["intent_hash"] = _canonical_hash( - {key: value for key, value in candidate.items() if key != "intent_hash"} - ) - assert parse_host_execution_intent(candidate).storage_intent is not None + candidate = _with_binding(candidate, "intent_hash") + parsed = parse_host_execution_intent(candidate) + assert not isinstance(parsed, StorageIntentError) + assert parsed != NO_SAFE_WORK + assert parsed.schema.endswith("-v3") + assert parsed.storage_intent is not None + assert validate_host_execution_intent(candidate) is NO_SAFE_WORK duplicate = copy.deepcopy(candidate) - duplicate["assignment"]["storage_intent"] = storage_intent + duplicate["assignment"]["storage_intent"] = copy.deepcopy( + candidate["storage_intent"] + ) duplicate["assignment"] = _with_binding( duplicate["assignment"], "assignment_binding_hash" ) - duplicate["intent_hash"] = _canonical_hash( - {key: value for key, value in duplicate.items() if key != "intent_hash"} - ) - duplicate_result = parse_host_execution_intent(duplicate) - conflict = copy.deepcopy(candidate) - conflict["storage_intent"] = _host_storage_intent( - task_id=task_id, - plan_binding=source_plan_hash, - context_hash="sha256:" + "3" * 64, - ) - conflict = _with_binding(conflict, "intent_hash") - conflict_result = parse_host_execution_intent(conflict) - for result in (duplicate_result, conflict_result): + duplicate = _with_binding(duplicate, "intent_hash") + missing = copy.deepcopy(candidate) + missing.pop("storage_intent") + missing = _with_binding(missing, "intent_hash") + for malformed in (duplicate, missing): + result = parse_host_execution_intent(malformed) assert isinstance(result, StorageIntentError) - assert result.code == STORAGE_TARGET_KEY_INVALID - - -def _real_storage_request() -> tuple[ - object, - object, - dict[str, object], - dict[str, object], - dict[str, object], - str | None, -]: - test_module_path = ( - Path(__file__).resolve().parents[1] - / "devkit_fastlane" - / "tests" - / "test_team_efficiency.py" - ) - spec = importlib.util.spec_from_file_location( - "storage_test_team_efficiency_tests", test_module_path - ) - assert spec is not None and spec.loader is not None - tests_module = importlib.util.module_from_spec(spec) - spec.loader.exec_module(tests_module) - - previous_task_temp = os.environ.get("CODEX_TASK_TEMP") - if previous_task_temp is None: - os.environ["CODEX_TASK_TEMP"] = str( - Path(__file__).resolve().parents[3] - / ".codex-task-temp" - ) - fixture = tests_module.TeamEfficiencyTests("runTest") - fixture.setUp() - helper = tests_module.load_efficiency() - request = fixture.fast_lane_request(helper) - task_ids = [item["task_id"] for item in request["execution_contexts"]] - descriptor = { - key: value - for key, value in _storage_binding_context().items() - if key != "execution_context_hash" - } - for context in request["execution_contexts"]: - context.update( - { - **descriptor, - "execution_context_hash": _canonical_hash( - {"storage_context": context["task_id"]} - ), - } - ) - request["storage_budgets"] = { - task_id: {"bytes": 4096 + index, "files": 8 + index} - for index, task_id in enumerate(task_ids) - } - host_status = fixture.fast_lane_host_status(helper, request) - route_request = host_status["routing_context"]["routes"][0]["request"] - host = copy.deepcopy(route_request["host_capabilities"]) - host["models"] = [ - {**model, "efforts": sorted(model["efforts"])} for model in host["models"] - ] - scheduler = route_request["scheduler_facts"] - return fixture, helper, request, host, scheduler, previous_task_temp - - -def test_real_prepare_entry_binds_initial_successor_and_missing_facts_fail_closed() -> None: - fixture, helper, request, host, scheduler, previous_task_temp = _real_storage_request() - try: - prepared = helper.prepare_authenticated_v5_routing_from_request( - request, - index_context_hash=helper._sha256_json({"index": "storage-real-entry"}), - host_capabilities=host, - scheduler_facts=scheduler, - ) - for wave in (prepared["units"], prepared["remaining_units"]): - assert wave - for unit in wave: - intent = unit["storage_intent"] - assert intent["task_id"] == unit["task"]["task_id"] - assert intent["plan_binding"] == prepared["source_plan_hash"] - assert intent["context_hash"] == _canonical_hash( - {"storage_context": unit["task"]["task_id"]} - ) - - missing = copy.deepcopy(request) - for context in missing["execution_contexts"]: - context.pop("execution_context_hash") - with pytest.raises(ValueError, match="STORAGE_POLICY_MISSING"): - helper.prepare_authenticated_v5_routing_from_request( - missing, - index_context_hash=helper._sha256_json({"index": "storage-real-entry"}), - host_capabilities=host, - scheduler_facts=scheduler, - ) - finally: - fixture.tearDown() - if previous_task_temp is None: - os.environ.pop("CODEX_TASK_TEMP", None) - else: - os.environ["CODEX_TASK_TEMP"] = previous_task_temp - - -def test_attested_routing_rejects_storage_rebind_after_attestation() -> None: - fixture, helper, request, host, scheduler, previous_task_temp = _real_storage_request() - try: - prepared = helper.prepare_authenticated_v5_routing_from_request( - request, - index_context_hash=helper._sha256_json({"index": "storage-real-entry"}), - host_capabilities=host, - scheduler_facts=scheduler, - ) - routing_request = prepared["routing_requests"][0] - core = sys.modules["fastlane_routing"] - request_binding_hash = core.v5_request_binding_hash(routing_request) - attestation = { - "schema": "2718lab-devkit/host-child-route-attestation-v1", - "status": "attested", - "request_binding_hash": request_binding_hash, - "host_id_hash": host["host_id_hash"], - "capability_epoch": 1, - "lease_epoch": 0, - "issued_event_seq": 1, - "expires_event_seq": 1, - "route": {"lane": "sol", "model": "gpt-5.6-sol", "effort": "high", "rank": 40}, - "inherit_current_session_model": False, - "refusal_code": None, - } - attestation["attestation_hash"] = helper._sha256_json(attestation) - tampered = copy.deepcopy(prepared["units"][0]) - budget = {"bytes": tampered["storage_budget"]["bytes"] + 1, "files": tampered["storage_budget"]["files"] + 1} - tampered["storage_budget"] = budget - intent = tampered["storage_intent"] = copy.deepcopy(tampered["storage_intent"]) - intent.update({"requested_bytes": budget["bytes"], "requested_files": budget["files"]}) - intent["storage_intent_hash"] = _canonical_hash( - {key: intent[key] for key in ("target_descriptor", "task_id", "plan_binding", "context_hash", "requested_bytes", "requested_files")} - ) - with pytest.raises(ValueError, match="routing profile"): - helper.compile_authenticated_v5_assignment_skeletons( - [tampered], - source_plan_hash=prepared["source_plan_hash"], - routing_requests=[routing_request], - attestation_items=[ - { - "task_id": routing_request["task"]["task_id"], - "request_binding_hash": request_binding_hash, - "attestation": attestation, - } - ], - ) - finally: - fixture.tearDown() - if previous_task_temp is None: - os.environ.pop("CODEX_TASK_TEMP", None) - else: - os.environ["CODEX_TASK_TEMP"] = previous_task_temp + assert result.code == "STORAGE_TARGET_KEY_INVALID" From 9900ad49e231c56ee56ee29f78812764984a4a26 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E5=93=80=E6=B4=9B=E8=8A=99?= Date: Sun, 30 Aug 2026 04:32:49 +0800 Subject: [PATCH 16/39] fix: bind storage intent proofs to compiler state --- .../devkit_runtime/fastlane_host_adapter.py | 57 ++- mcp-tools/devkit_runtime/host_bridge.py | 7 +- mcp-tools/devkit_runtime/host_session.py | 343 ++++++++++++++++-- mcp-tools/server.py | 15 + mcp-tools/tests/test_storage_firewall.py | 3 + 5 files changed, 393 insertions(+), 32 deletions(-) diff --git a/mcp-tools/devkit_runtime/fastlane_host_adapter.py b/mcp-tools/devkit_runtime/fastlane_host_adapter.py index 69e88fa..51e26f9 100644 --- a/mcp-tools/devkit_runtime/fastlane_host_adapter.py +++ b/mcp-tools/devkit_runtime/fastlane_host_adapter.py @@ -167,6 +167,7 @@ class _PreparedHostFacts: preparation_id: str | None = None call_intent_hash: str | None = None storage_budgets: tuple[tuple[str, int, int], ...] = () + storage_intents: tuple[dict[str, object], ...] = field(default=(), repr=False) def _normalized_storage_budgets( @@ -368,6 +369,7 @@ def prepare_verified_host_facts( try: normalized_storage_budgets = _normalized_storage_budgets(storage_budgets) bridge_attested = False + skeletons: tuple[dict[str, object], ...] = () if request is not None or reasoning_effort is not None: normalized_request = _planner_request(request) request_bytes = _canonical_bytes(normalized_request) @@ -432,6 +434,7 @@ def prepare_verified_host_facts( str, routing_registry_binding_hash ), storage_task_ids=(storage_task_ids if normalized_storage_budgets else ()), + storage_budget_bindings=normalized_storage_budgets, ) if not bridge_attested: return NO_SAFE_WORK @@ -449,6 +452,24 @@ def prepare_verified_host_facts( expires_at = session.compiler_evidence_expires_at(evidence) if expires_at is None: return NO_SAFE_WORK + storage_intents: tuple[dict[str, object], ...] = () + if normalized_storage_budgets: + profiles = session.storage_profiles_for_compiler_evidence(evidence) + if type(profiles) is not tuple: + return NO_SAFE_WORK + storage_intents = tuple( + _storage_intents_for_profiles( + profiles, + skeletons, + normalized_storage_budgets, + ) + ) + if not session.bind_storage_intent_proof( + evidence, + storage_budgets=normalized_storage_budgets, + storage_intents=storage_intents, + ): + return NO_SAFE_WORK return _PreparedHostFacts( session=session, evidence=evidence, @@ -460,6 +481,7 @@ def prepare_verified_host_facts( cast(str, call_intent_hash) if bridge_attested else None ), storage_budgets=normalized_storage_budgets, + storage_intents=storage_intents, ) except Exception: return NO_SAFE_WORK @@ -567,16 +589,35 @@ def compile_fast_lane_with_host_facts( _validate_batch_fences(facts) # Storage intents are local compiler proof material only in Task4a. Do # not extend the established dispatch-batch schema before Task4b owns - # Host admission/execution. Constructing them here still validates the - # Host profile order, the caller budgets, and every profile/dispatch - # binding after the full compiler evidence response has been verified. + # Host admission/execution. They are constructed and sealed into the + # private compiler invocation after the profile exchange, then + # re-derived here to reject a changed budget/profile/intent binding. if prepared.storage_budgets: - _storage_intents_for_profiles( - material.storage_profiles, - fact_mappings, - prepared.storage_budgets, + storage_intents = tuple( + _storage_intents_for_profiles( + material.storage_profiles, + fact_mappings, + prepared.storage_budgets, + ) ) - elif material.storage_profiles: + storage_intent_hashes = tuple( + cast(str, intent["storage_intent_hash"]) + for intent in storage_intents + ) + if ( + material.storage_budget_bindings != prepared.storage_budgets + or material.storage_intents != storage_intents + or material.storage_intent_hashes != storage_intent_hashes + or prepared.storage_intents != storage_intents + ): + return NO_SAFE_WORK + elif ( + material.storage_profiles + or material.storage_budget_bindings + or material.storage_intents + or material.storage_intent_hashes + or prepared.storage_intents + ): return NO_SAFE_WORK batch: dict[str, object] = { "schema": "2718lab-devkit/fastlane-host-dispatch-batch-v1", diff --git a/mcp-tools/devkit_runtime/host_bridge.py b/mcp-tools/devkit_runtime/host_bridge.py index 2c618a6..a689c23 100644 --- a/mcp-tools/devkit_runtime/host_bridge.py +++ b/mcp-tools/devkit_runtime/host_bridge.py @@ -3081,7 +3081,12 @@ def _storage_profile_action_id(request: StorageProfileRequest) -> str: normalized = _normalize_storage_profile_request( _storage_profile_request_payload(request) ) - return normalized.request_hash[7:] + # The Host multiplexes the private profile exchange under the same + # preparation action as compiler evidence. ``request_hash`` is still + # verified from the exact eight-field payload, but it is not a wire action + # identifier; Rust therefore requires this exact preparation id on both + # the request and response frames. + return normalized.preparation_id def _parse_storage_profile_request( diff --git a/mcp-tools/devkit_runtime/host_session.py b/mcp-tools/devkit_runtime/host_session.py index 7f81f75..df8f045 100644 --- a/mcp-tools/devkit_runtime/host_session.py +++ b/mcp-tools/devkit_runtime/host_session.py @@ -12,7 +12,7 @@ import math import re from collections.abc import Callable, Mapping, Sequence -from dataclasses import dataclass, field +from dataclasses import dataclass, field, replace from enum import StrEnum from threading import Event, RLock, Thread, current_thread from typing import Final, TypeAlias, cast @@ -66,6 +66,18 @@ {"managed_read_only", "managed_workspace", "disabled", "external"} ) _STORAGE_PROFILE_SCALAR: Final = re.compile(r"[A-Za-z0-9][A-Za-z0-9._+-]{0,127}\Z") +_STORAGE_INTENT_SCHEMA: Final = "2718lab.storage.intent.v1" +_STORAGE_TARGET_SCHEMA: Final = "2718lab.storage.target.v1" +_STORAGE_DESCRIPTOR_FIELDS: Final = ( + "repository_identity", + "workspace_manifest_hash", + "cargo_lock_hash", + "toolchain_digest", + "target_triple", + "profile", + "features_hash", + "build_env_class", +) class HostCapabilityState(StrEnum): @@ -176,6 +188,9 @@ class _CompilerInvocationBinding: verified_lease_scope_bindings: tuple[str, ...] dispatch_facts: tuple[object, ...] = () dispatch_binding_hashes: tuple[str, ...] = () + storage_budget_bindings: tuple[tuple[str, int, int], ...] = field( + default=(), repr=False + ) storage_profiles: tuple[dict[str, object], ...] = field(default=(), repr=False) registry_binding_hash: str | None = None evidence_expires_at: int | None = None @@ -203,7 +218,12 @@ class _CompilerInvocation: expires_at: float binding_hash: str registry_binding_hash: str | None = None + storage_budget_bindings: tuple[tuple[str, int, int], ...] = field( + default=(), repr=False + ) storage_profiles: tuple[dict[str, object], ...] = field(default=(), repr=False) + storage_intents: tuple[dict[str, object], ...] = field(default=(), repr=False) + storage_intent_hashes: tuple[str, ...] = field(default=(), repr=False) @dataclass(frozen=True) @@ -216,6 +236,9 @@ class _CompilerRequestContext: assignment_skeletons: tuple[dict[str, object], ...] = field(repr=False) project_index_attestation_refs: tuple[dict[str, object], ...] = field(repr=False) storage_task_ids: tuple[str, ...] = () + storage_budget_bindings: tuple[tuple[str, int, int], ...] = field( + default=(), repr=False + ) @dataclass(frozen=True) @@ -554,25 +577,6 @@ def prepare_compiler_evidence( ): return _NO_SAFE_WORK preparation = _CompilerPreparation() - invocation_binding = { - "preparation_id": preparation_id, - "request_hash": binding.request_hash, - "reasoning_effort": binding.reasoning_effort, - "verified_route_result_hashes": binding.verified_route_result_hashes, - "verified_lease_scope_bindings": binding.verified_lease_scope_bindings, - "issued_at": issued_at, - "expires_at": expires_at, - } - if binding.registry_binding_hash is not None: - invocation_binding["registry_binding_hash"] = ( - binding.registry_binding_hash - ) - if binding.dispatch_binding_hashes: - invocation_binding["dispatch_binding_hashes"] = ( - binding.dispatch_binding_hashes - ) - if binding.storage_profiles: - invocation_binding["storage_profiles"] = binding.storage_profiles material = _CompilerInvocation( schema="2718lab-devkit/compiler-invocation-v2", preparation_id=preparation_id, @@ -584,10 +588,15 @@ def prepare_compiler_evidence( dispatch_binding_hashes=binding.dispatch_binding_hashes, issued_at=issued_at, expires_at=expires_at, - binding_hash=_hash(invocation_binding), + binding_hash="", registry_binding_hash=binding.registry_binding_hash, + storage_budget_bindings=binding.storage_budget_bindings, storage_profiles=binding.storage_profiles, ) + material = replace( + material, + binding_hash=_hash(_compiler_invocation_binding_material(material)), + ) material_state = _compiler_invocation_state(material) try: provider_material = provider(preparation) @@ -623,6 +632,7 @@ def bind_compiler_request( project_index_attestation_refs: tuple[dict[str, object], ...], routing_registry_binding_hash: str, storage_task_ids: tuple[str, ...] = (), + storage_budget_bindings: tuple[tuple[str, int, int], ...] = (), ) -> bool: """Bind public request identity before a private registry round trip.""" @@ -647,10 +657,14 @@ def bind_compiler_request( for task_id in storage_task_ids ) or len(set(storage_task_ids)) != len(storage_task_ids) + or type(storage_budget_bindings) is not tuple ): return False try: normalized_routes = _normalized_routes(requested_routes) + normalized_storage_budgets = _normalized_storage_budget_bindings( + storage_budget_bindings + ) skeleton_task_ids = tuple( _required_fast_lane_task_id(_mapping(item).get("task_id")) for item in assignment_skeletons @@ -658,7 +672,18 @@ def bind_compiler_request( trusted_now = int(self._read_trusted_clock()) except (TypeError, ValueError): return False - if storage_task_ids and storage_task_ids != skeleton_task_ids: + if ( + bool(storage_task_ids) != bool(normalized_storage_budgets) + or (storage_task_ids and storage_task_ids != skeleton_task_ids) + or ( + normalized_storage_budgets + and { + task_id + for task_id, _requested_bytes, _requested_files in normalized_storage_budgets + } + != set(skeleton_task_ids) + ) + ): return False routing_snapshot = self._routing_attestation_snapshots.get( (call_intent_hash, preparation_id) @@ -681,6 +706,7 @@ def bind_compiler_request( project_index_attestation_refs=project_index_attestation_refs, routing_registry_binding_hash=routing_registry_binding_hash, storage_task_ids=storage_task_ids, + storage_budget_bindings=normalized_storage_budgets, ) return True @@ -779,6 +805,7 @@ def _resolve_bridge_compiler_invocation( ), registry_binding_hash=_required_hash(response["registry_binding_hash"]), evidence_expires_at=_required_positive_int(response["expires_at"]), + storage_budget_bindings=context.storage_budget_bindings, storage_profiles=storage_profiles, ) @@ -820,6 +847,109 @@ def project_index_query_attestation( return None return dict(value) + def storage_profiles_for_compiler_evidence( + self, evidence: object + ) -> tuple[dict[str, object], ...] | str: + """Expose copies of profile facts only through a live opaque handle. + + This is deliberately narrower than consuming compiler evidence: the + adapter must construct and bind its local storage-intent proof before + the one-shot compiler material can be consumed for dispatch. + """ + + with self._compiler_evidence_lock: + if ( + self._closed + or self._frozen + or type(evidence) is not _CompilerEvidenceHandle + ): + return _NO_SAFE_WORK + material = self._compiler_evidence.get(evidence) + if material is None: + return _NO_SAFE_WORK + try: + now = self._read_trusted_clock() + except (TypeError, ValueError): + return _NO_SAFE_WORK + if ( + now >= material.expires_at + or material.binding_hash + != _hash(_compiler_invocation_binding_material(material)) + or bool(material.storage_budget_bindings) + != bool(material.storage_profiles) + ): + return _NO_SAFE_WORK + return tuple(dict(profile) for profile in material.storage_profiles) + + def bind_storage_intent_proof( + self, + evidence: object, + *, + storage_budgets: object, + storage_intents: object, + ) -> bool: + """Seal post-Host budgets and intent hashes into compiler state once. + + The Host profile exchange is already complete at this point. This + method accepts no profile replacement: it validates the adapter's + intents against the retained Host profiles and dispatch facts, then + folds the exact budgets and intents into the private invocation hash. + """ + + with self._compiler_evidence_lock: + if ( + self._closed + or self._frozen + or type(evidence) is not _CompilerEvidenceHandle + ): + return False + material = self._compiler_evidence.get(evidence) + if material is None: + return False + try: + now = self._read_trusted_clock() + budget_bindings = _normalized_storage_budget_bindings( + storage_budgets + ) + if ( + now >= material.expires_at + or not budget_bindings + or budget_bindings != material.storage_budget_bindings + or not material.storage_profiles + or material.storage_intents + or material.storage_intent_hashes + or material.binding_hash + != _hash(_compiler_invocation_binding_material(material)) + ): + return False + intents = _normalized_storage_intent_proof( + storage_intents, + budget_bindings=budget_bindings, + profiles=material.storage_profiles, + dispatch_facts=material.dispatch_facts, + preparation_id=material.preparation_id, + ) + intent_hashes = tuple( + _required_hash(intent["storage_intent_hash"]) + for intent in intents + ) + rebound = replace( + material, + storage_intents=intents, + storage_intent_hashes=intent_hashes, + binding_hash="", + ) + rebound = replace( + rebound, + binding_hash=_hash( + _compiler_invocation_binding_material(rebound) + ), + ) + except (KeyError, TypeError, ValueError): + return False + self._compiler_evidence[evidence] = rebound + return True + def consume_compiler_evidence(self, evidence: object) -> object | str: """Exchange a session-issued handle once, rejecting public substitutes.""" @@ -837,7 +967,11 @@ def consume_compiler_evidence(self, evidence: object) -> object | str: now = self._read_trusted_clock() except (TypeError, ValueError): return _NO_SAFE_WORK - if now >= material.expires_at: + if ( + now >= material.expires_at + or material.binding_hash + != _hash(_compiler_invocation_binding_material(material)) + ): return _NO_SAFE_WORK return material @@ -1838,6 +1972,34 @@ def _optional_ordered_hash_tuple(value: object) -> bool: ) +def _normalized_storage_budget_bindings( + value: object, +) -> tuple[tuple[str, int, int], ...]: + """Normalize exact task budgets retained outside the public skeleton.""" + + if type(value) is not tuple or len(value) > 16: + raise ValueError("compiler storage budgets are invalid") + normalized: list[tuple[str, int, int]] = [] + for item in value: + if type(item) is not tuple or len(item) != 3: + raise ValueError("compiler storage budget is invalid") + task_id, requested_bytes, requested_files = item + if ( + type(task_id) is not str + or _FAST_LANE_TASK_ID.fullmatch(task_id) is None + or type(requested_bytes) is not int + or not 0 < requested_bytes <= (1 << 64) - 1 + or type(requested_files) is not int + or not 0 < requested_files <= (1 << 64) - 1 + ): + raise ValueError("compiler storage budget is invalid") + normalized.append((task_id, requested_bytes, requested_files)) + normalized.sort(key=lambda item: item[0]) + if len({task_id for task_id, _bytes, _files in normalized}) != len(normalized): + raise ValueError("compiler storage budgets are duplicated") + return tuple(normalized) + + def _normalized_storage_profiles( value: object, ) -> tuple[dict[str, object], ...]: @@ -1900,6 +2062,127 @@ def _normalized_storage_profiles( return tuple(normalized) +def _normalized_storage_intent_proof( + value: object, + *, + budget_bindings: tuple[tuple[str, int, int], ...], + profiles: tuple[dict[str, object], ...], + dispatch_facts: tuple[object, ...], + preparation_id: str, +) -> tuple[dict[str, object], ...]: + """Tie adapter-built intents back to this invocation's Host facts once.""" + + if ( + type(value) is not tuple + or not value + or len(value) > 16 + or len(value) != len(profiles) + or len(value) != len(dispatch_facts) + or _IDENTIFIER.fullmatch(preparation_id) is None + ): + raise ValueError("compiler storage intent proof is invalid") + budget_by_task = { + task_id: (requested_bytes, requested_files) + for task_id, requested_bytes, requested_files in budget_bindings + } + try: + fact_task_ids = tuple( + _required_fast_lane_task_id(getattr(fact, "task_id")) + for fact in dispatch_facts + ) + fact_source_plan_hashes = tuple( + _required_hash(getattr(fact, "source_plan_hash")) + for fact in dispatch_facts + ) + except (AttributeError, TypeError, ValueError) as error: + raise ValueError("compiler storage dispatch facts are invalid") from error + if len(set(fact_task_ids)) != len(fact_task_ids) or set(budget_by_task) != set( + fact_task_ids + ): + raise ValueError("compiler storage budget binding is invalid") + + from .storage_intent import parse_storage_intent + + normalized: list[dict[str, object]] = [] + for intent, profile, task_id, source_plan_hash in zip( + value, + profiles, + fact_task_ids, + fact_source_plan_hashes, + strict=True, + ): + if type(intent) is not dict: + raise ValueError("compiler storage intent is invalid") + parsed = parse_storage_intent(intent) + normalized_intent = parsed.to_dict() + if normalized_intent != intent: + raise ValueError("compiler storage intent is not canonical") + if ( + profile.get("preparation_id") != preparation_id + or profile.get("task_id") != task_id + or profile.get("source_plan_hash") != source_plan_hash + or parsed.task_id != task_id + or parsed.plan_binding != source_plan_hash + or parsed.context_hash != profile.get("execution_context_hash") + or (parsed.requested_bytes, parsed.requested_files) + != budget_by_task.get(task_id) + ): + raise ValueError("compiler storage intent binding is invalid") + descriptor = { + "schema": _STORAGE_TARGET_SCHEMA, + "artifact_kind": "fastlane-task", + **{ + field_name: profile.get(field_name) + for field_name in _STORAGE_DESCRIPTOR_FIELDS + }, + } + if ( + parsed.to_dict().get("schema") != _STORAGE_INTENT_SCHEMA + or parsed.to_dict().get("target_descriptor") != descriptor + ): + raise ValueError("compiler storage target binding is invalid") + normalized.append(normalized_intent) + return tuple(normalized) + + +def _compiler_invocation_binding_material( + value: _CompilerInvocation, +) -> dict[str, object]: + """Render private invocation proof without extending legacy wire schemas.""" + + material: dict[str, object] = { + "preparation_id": value.preparation_id, + "request_hash": value.request_hash, + "reasoning_effort": value.reasoning_effort, + "verified_route_result_hashes": value.verified_route_result_hashes, + "verified_lease_scope_bindings": value.verified_lease_scope_bindings, + "issued_at": value.issued_at, + "expires_at": value.expires_at, + } + if value.registry_binding_hash is not None: + material["registry_binding_hash"] = value.registry_binding_hash + if value.dispatch_binding_hashes: + material["dispatch_binding_hashes"] = value.dispatch_binding_hashes + if value.storage_budget_bindings: + material["storage_budget_bindings"] = [ + { + "task_id": task_id, + "bytes": requested_bytes, + "files": requested_files, + } + for task_id, requested_bytes, requested_files in value.storage_budget_bindings + ] + if value.storage_profiles: + material["storage_profiles"] = [ + dict(profile) for profile in value.storage_profiles + ] + if value.storage_intents: + material["storage_intents"] = [dict(intent) for intent in value.storage_intents] + if value.storage_intent_hashes: + material["storage_intent_hashes"] = value.storage_intent_hashes + return material + + def _normalized_compiler_invocation_binding( value: object, ) -> _CompilerInvocationBinding: @@ -1914,6 +2197,7 @@ def _normalized_compiler_invocation_binding( or len(value.dispatch_facts) > 16 or not _optional_ordered_hash_tuple(value.dispatch_binding_hashes) or len(value.dispatch_facts) != len(value.dispatch_binding_hashes) + or type(value.storage_budget_bindings) is not tuple or type(value.storage_profiles) is not tuple or len(value.storage_profiles) > 16 or ( @@ -1926,7 +2210,16 @@ def _normalized_compiler_invocation_binding( ) ): raise ValueError("compiler invocation binding is invalid") + storage_budget_bindings = _normalized_storage_budget_bindings( + value.storage_budget_bindings + ) storage_profiles = _normalized_storage_profiles(value.storage_profiles) + if bool(storage_budget_bindings) != bool(storage_profiles): + raise ValueError("compiler storage proof is incomplete") + if storage_budget_bindings and { + task_id for task_id, _requested_bytes, _requested_files in storage_budget_bindings + } != {cast(str, profile["task_id"]) for profile in storage_profiles}: + raise ValueError("compiler storage profile budget bindings are invalid") return _CompilerInvocationBinding( request_hash=value.request_hash, reasoning_effort=value.reasoning_effort, @@ -1934,6 +2227,7 @@ def _normalized_compiler_invocation_binding( verified_lease_scope_bindings=value.verified_lease_scope_bindings, dispatch_facts=value.dispatch_facts, dispatch_binding_hashes=value.dispatch_binding_hashes, + storage_budget_bindings=storage_budget_bindings, storage_profiles=storage_profiles, registry_binding_hash=value.registry_binding_hash, evidence_expires_at=value.evidence_expires_at, @@ -1952,7 +2246,10 @@ def _compiler_invocation_state(value: _CompilerInvocation) -> tuple[object, ...] value.verified_lease_scope_bindings, value.dispatch_facts, value.dispatch_binding_hashes, + value.storage_budget_bindings, tuple(_hash(profile) for profile in value.storage_profiles), + tuple(_hash(intent) for intent in value.storage_intents), + value.storage_intent_hashes, value.issued_at, value.expires_at, value.binding_hash, diff --git a/mcp-tools/server.py b/mcp-tools/server.py index 74a5aa9..7010a3a 100644 --- a/mcp-tools/server.py +++ b/mcp-tools/server.py @@ -1130,6 +1130,20 @@ def _fastlane_authenticated_dispatch( return _failure("FASTLANE_HOST_AUTHORITY_UNAVAILABLE") initial_units = projected["units"] remaining_units = projected["remaining_units"] + initial_storage_budgets = { + unit["task"]["task_id"]: unit["storage_budget"] + for unit in initial_units + if "storage_budget" in unit + } + # The compiler/profile exchange can attest only the live initial + # skeletons. Remaining work is materialized later by the Host-owned + # refill registry, which has no storage-profile proof channel yet. + # Budgeted successors therefore fail closed before publication. + if ( + len(initial_storage_budgets) not in {0, len(initial_units)} + or any("storage_budget" in unit for unit in remaining_units) + ): + return _failure("FASTLANE_HOST_AUTHORITY_UNAVAILABLE") initial_task_ids = { unit["task"]["task_id"] for unit in initial_units } @@ -1228,6 +1242,7 @@ def _fastlane_authenticated_dispatch( request=planner_request, reasoning_effort=reasoning_effort, requested_routes=requested_routes, + storage_budgets=initial_storage_budgets, ) if prepared == NO_SAFE_WORK: return _failure("FASTLANE_HOST_AUTHORITY_UNAVAILABLE") diff --git a/mcp-tools/tests/test_storage_firewall.py b/mcp-tools/tests/test_storage_firewall.py index fdb2fcc..f269404 100644 --- a/mcp-tools/tests/test_storage_firewall.py +++ b/mcp-tools/tests/test_storage_firewall.py @@ -123,6 +123,7 @@ def _authenticated_v5_fixture() -> dict[str, object]: "dispatch_order": 0, "index_context_hash": hash_a, "predecessor_hash": hash_b, + "storage_budget": {"bytes": 4096, "files": 8}, } api = team_efficiency._AuthenticatedV5Api() planner = team_efficiency._authenticated_v5_helper_module("authenticated_v5_planner") @@ -449,6 +450,8 @@ def host_reply() -> None: storage_budgets={"TASK-V5": {"bytes": 4096, "files": 8}}, ) assert type(prepared).__name__ == "_PreparedHostFacts" + assert len(prepared.storage_intents) == 1 + assert prepared.storage_intents[0]["task_id"] == "TASK-V5" batch = adapter.compile_fast_lane_with_host_facts( fixture["planner_request"], reasoning_effort="max", From 6f32231ade85cbb10ac0a0f48ac56292c96ad3dc Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E5=93=80=E6=B4=9B=E8=8A=99?= Date: Sun, 30 Aug 2026 04:35:10 +0800 Subject: [PATCH 17/39] fix: permit validated storage profile frames --- mcp-tools/devkit_runtime/host_bridge.py | 2 ++ 1 file changed, 2 insertions(+) diff --git a/mcp-tools/devkit_runtime/host_bridge.py b/mcp-tools/devkit_runtime/host_bridge.py index a689c23..13e1497 100644 --- a/mcp-tools/devkit_runtime/host_bridge.py +++ b/mcp-tools/devkit_runtime/host_bridge.py @@ -177,6 +177,8 @@ "proof_continuation", "compiler_evidence_request", "compiler_evidence_response", + "storage_profile_request", + "storage_profile_response", "project_index_attestation", "routing_attestation_request", "routing_attestation_response", From 8bc0973314ae4a00147f2a7da945ca3e1e9f6fcd Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E5=93=80=E6=B4=9B=E8=8A=99?= Date: Sun, 30 Aug 2026 05:37:53 +0800 Subject: [PATCH 18/39] docs: align storage admission plan with host ownership --- .../2026-08-29-storage-firewall-1.1.3.md | 471 +++++++++++------- 1 file changed, 295 insertions(+), 176 deletions(-) diff --git a/docs/superpowers/plans/2026-08-29-storage-firewall-1.1.3.md b/docs/superpowers/plans/2026-08-29-storage-firewall-1.1.3.md index 5768119..32beef5 100644 --- a/docs/superpowers/plans/2026-08-29-storage-firewall-1.1.3.md +++ b/docs/superpowers/plans/2026-08-29-storage-firewall-1.1.3.md @@ -4,10 +4,25 @@ **Goal:** Make every Cargo, Python, MCP-package, and Fast Lane write begin with one host-approved deterministic generated root and a fail-closed byte/file/free-space reservation. -**Architecture:** DevKit emits a bounded, path-free `StorageIntent` whose hash is bound to the Fast Lane task, source plan, execution context, and project identity. The Codex Host validates the intent, computes the canonical target key, checks the approved root and disk policy, and returns an opaque admission receipt containing the assigned root; the worker never selects `CARGO_TARGET_DIR` or a temporary directory. Lease persistence, cleanup, GitHub source authorization, and session CAS are separate follow-up plans and are not hidden in this admission path. +**Architecture:** DevKit emits a bounded, path-free `StorageIntent` whose hash is bound to the Fast Lane task, source plan, execution context, and project identity. The Codex Host resolves an already-issued profile, validates the intent, and reserves a deterministic target family for a Host-owned wave with per-task member grants. Receipts contain identities only; private worker facts carry paths. Existing route/lease batch hashes remain unchanged. Lease persistence, cleanup, GitHub source authorization, and session CAS are separate follow-up plans. **Tech Stack:** Python 3.11 standard library (`dataclasses`, `hashlib`, `json`, `pathlib`), MCP FastMCP/Pydantic, Rust 2021, `serde`/`serde_json`, `sha2`, Tokio, platform filesystem-capacity APIs, and the existing authenticated inherited-handle bridge. +**Revision status (2026-08-30):** This is an unpublished protocol revision made +after the existing Task 4a slice passed its compile checks. That result does +not mean admission, group reservations, or successor production wiring below +is implemented. The unpublished admission-v1 request changes from exact4 to +exact5; intent/target/profile-v1 stay unchanged. If exact4 has been deployed +outside these worktrees, use admission-v2 instead and reject downgrade. + +**Compile-first execution:** Reuse the Host's existing +`G:\2718lab\_codex\.codex-task-temp\codex-host-mcp-fix-recovery2\codex-rs\target`, +set `CARGO_INCREMENTAL=0`, and use `--locked -j1`. Check the changed crates +before running a small relevant probe. Do not recreate RED failures from +completed slices, repeatedly run full suites, or create a storage-specific +Cargo target. Unchecked tasks below are implementation/acceptance work, not +claims that the revised contract is complete. + --- ## Scope and file map @@ -22,10 +37,13 @@ DevKit files: - Create `mcp-tools/devkit_runtime/storage_intent.py`: immutable intent and canonical target-descriptor validation; it contains no filesystem write and no owner claim. +- Modify `mcp-tools/devkit_runtime/__init__.py` only for the intent exports. - Modify `mcp-tools/devkit_fastlane/scripts/authenticated_v5_planner.py:14-415` and `mcp-tools/devkit_fastlane/scripts/authenticated_v5_projection.py:1-318`: - bind one intent to every compiler assignment and retain it through initial - and successor waves. + retain explicit budgets through initial and successor waves; construct + initial intents only from verified Host profiles, not caller facts. +- Modify `mcp-tools/devkit_runtime/fastlane_host_adapter.py`: forward initial + intent/profile references without rewriting attested route/lease facts. - Modify `mcp-tools/devkit_runtime/fastlane_host_intent.py:333-595`: structurally parse the new intent and reject an intent whose binding does not equal the assignment's task/context/plan hashes. @@ -33,25 +51,37 @@ DevKit files: `mcp-tools/devkit_runtime/host_session.py:159-335,636-735`: add the private `storage_admit` request/receipt exchange to the already authenticated bridge. - Modify `mcp-tools/server.py:1008-1314` only at the authenticated Fast Lane - dispatch seam so the production path forwards the intent and exposes the - returned root as a private execution binding; no public path argument is - added. + dispatch seam so the production path forwards intent/profile references + and returns receipt identities; paths never cross into the DevKit response. - Create `mcp-tools/tests/test_storage_firewall.py` and modify `mcp-tools/tests/test_fastlane_host_adapter.py:894-1014` for the smallest projection-to-host regression. Codex Host files: +- Create `codex-rs/rmcp-client/src/storage_intent.rs` for the single strict + Rust intent/target codec and export it from `codex-rs/rmcp-client/src/lib.rs`. + Reuse rmcp-client's existing `sha2`/`serde_json`; do not add a protocol-crate + dependency or make rmcp-client depend on core. - Create `codex-rs/core/src/fast_lane_host_dispatch/storage_firewall.rs`: the canonical target key, approved-root fence, capacity provider, quota policy, - admission receipt, and stable errors. + admission receipt, and stable errors. Re-export the rmcp-client intent type + here for the existing RED imports; do not create a second Rust codec. - Modify `codex-rs/core/src/fast_lane_host_dispatch/mod.rs:1-49` to register the module and keep its types `pub(crate)`. -- Modify `codex-rs/core/src/fast_lane_host_dispatch/contract.rs:1-842` only in - the assignment/skeleton projection to carry and hash `storage_intent`. +- Modify `codex-rs/core/src/fast_lane_host_dispatch/contract.rs:1-842` to define + the separate Host-owned `StorageAssignmentBinding`; do not append storage + fields to, or rehash, an already-attested route/lease assignment or batch. - Modify `codex-rs/rmcp-client/src/inherited_host_bridge_protocol.rs:1-420` and `codex-rs/rmcp-client/src/inherited_host_bridge_protocol/envelope.rs:1-310` to validate the exact wire payload and its size before it enters core. +- Modify `codex-rs/rmcp-client/src/inherited_host_bridge_protocol/session.rs` + and `pump.rs` for session-local completed-profile lookup and admission I/O; + modify `codex-rs/core/src/fast_lane_host_dispatch/receiver.rs` to record + successful profile transmission before accepting admission. +- Modify `codex-rs/core/src/fast_lane_host_dispatch/storage_profile.rs` only + to share Host-derived profile construction with selected-wave authority; + successor materialization must not fabricate a bridge request. - Modify `codex-rs/core/src/fast_lane_host_dispatch/registry.rs:567-1668` to reserve the target-family key before writer preparation and `coordinator.rs:393-580,1218-1260` to release the admission on terminal or @@ -59,20 +89,22 @@ Codex Host files: - Modify `codex-rs/core/src/fast_lane_host_dispatch/codex_adapter.rs:1-1444` at worker environment construction so the host-issued root is the sole `CARGO_TARGET_DIR`/task-temp value. -- Create `codex-rs/core/src/fast_lane_host_dispatch/storage_firewall_tests.rs` - and register it from the new module with `#[path = ...]`. +- Modify the existing `codex-rs/core/src/fast_lane_host_dispatch/storage_firewall_tests.rs`; + preserve its sibling-module registration in `mod.rs`. - Modify `codex-rs/core/Cargo.toml:80-145` by adding `[target.'cfg(target_os = "windows")'.dependencies] windows-sys = { version = "0.52", features = ["Win32_Storage_FileSystem"] }`, matching the existing CLI dependency version; then update `codex-rs/Cargo.lock` and - `codex-rs/MODULE.bazel.lock` in the same Host commit. + `MODULE.bazel.lock` at the Host repository root in the same Host commit + when dependency changes require them. The worker must not edit any file outside this map. The ledger, preview/apply, source authorization, and session CAS changes belong to Plans 2 and 3. ## Shared wire contract -The following exact JSON shape is the only storage admission payload. The +The following exact JSON shape is the storage intent, nested in the Task 4 +admission request. The `target_descriptor` is the sole input to `target_key`; request sizes and root bindings are policy inputs and are not smuggled into the target-key identity. @@ -101,10 +133,23 @@ bindings are policy inputs and are not smuggled into the target-key identity. ``` The canonical `target_key` is the SHA-256 of the UTF-8 canonical JSON of the -ten `target_descriptor` fields. The host returns a `StorageAdmissionReceipt` -with `storage_intent_hash`, `target_key`, `assigned_root_identity`, +ten `target_descriptor` fields. It never includes task/wave/member identifiers. +The exact successful `StorageAdmissionReceipt` decision has these fields: + +`schema`, `admission_id`, `profile_attestation_hash`, `storage_intent_hash`, +`storage_binding_hash`, `target_key`, `assigned_root_identity`, `target_family_lease_id`, `reserved_bytes`, `reserved_files`, -`free_space_before`, `free_space_after_reserve`, and `free_space_floor`. +`free_space_before`, `free_space_after_reserve`, `free_space_floor`, +`expires_at`, `receipt_hash`. + +Its schema is `2718lab.storage.admission-receipt.v1`; `receipt_hash` hashes +all decision fields except itself. The successful private response envelope +is exact `{schema, correlation_id, request_hash, receipt}` with schema +`2718lab.storage.admission-response.v1`. A failure uses the existing bounded +transport error path with a stable code, never a fabricated zero reservation. +The internal transport-completion receipt separately records response hash, +session binding, bridge generation, expiry, and completion time. It is proof +of transmission, not a second capacity reservation or a public path carrier. ## Implementation tasks @@ -112,7 +157,7 @@ with `storage_intent_hash`, `target_key`, `assigned_root_identity`, **Files:** - Create: `mcp-tools/tests/test_storage_firewall.py` -- Create: `codex-rs/core/src/fast_lane_host_dispatch/storage_firewall_tests.rs` +- Modify: `codex-rs/core/src/fast_lane_host_dispatch/storage_firewall_tests.rs` - Modify: `mcp-tools/tests/test_fastlane_host_adapter.py:894-1014` - [ ] **Step 1: Add the Python RED fixture.** @@ -155,18 +200,19 @@ def test_storage_intent_rejects_absolute_path_and_unknown_descriptor_key(): ```rust #[test] fn missing_policy_is_stable_and_does_not_create_a_root() { + let approved_root = tempfile::tempdir().unwrap(); let firewall = StorageFirewall::new( - PathBuf::from(r"G:\2718lab\_codex\.codex-task-temp"), + approved_root.path().to_path_buf(), CapacitySnapshot { free_bytes: 8 * GIB, free_files: 1_000_000 }, None, ); let error = firewall.admit(intent()).expect_err("missing policy must fail closed"); assert_eq!(error.code(), "STORAGE_POLICY_MISSING"); - assert!(!Path::new(r"G:\2718lab\_codex\.codex-task-temp\targets").exists()); + assert!(!approved_root.path().join("generated").exists()); } ``` -- [ ] **Step 3: Run only the new RED probes.** +- [ ] **Step 3: Keep the existing RED evidence; run a new narrow probe only after compilation.** Run from `G:\2718lab\_codex\.codex-task-temp\devkit-1.1.2-recovery\mcp-tools`: @@ -177,11 +223,15 @@ $env:PYTEST_DISABLE_PLUGIN_AUTOLOAD='1'; python -m pytest tests/test_storage_fir Run from `G:\2718lab\_codex\.codex-task-temp\codex-host-mcp-fix-recovery2\codex-rs`: ```powershell -$env:CARGO_TARGET_DIR='G:\2718lab\_codex\.codex-task-temp\storage-113-firewall-rust-target'; cargo test -p codex-core missing_policy_is_stable_and_does_not_create_a_root --locked -j1 +$env:CARGO_TARGET_DIR='G:\2718lab\_codex\.codex-task-temp\codex-host-mcp-fix-recovery2\codex-rs\target' +$env:CARGO_INCREMENTAL='0' +cargo check -p codex-core --lib --locked -j1 +cargo test -p codex-core missing_policy_is_stable_and_does_not_create_a_root --locked -j1 ``` -Expected: both commands fail because the new module/types do not exist; -neither command may create a production target root. +Historical missing-module RED is not a reason to rerun a failed build. Stop +if the compile gate fails; after implementation, run only the selected +boundary probes. Neither probe may create a production generated root. - [ ] **Step 4: Commit the RED tests only.** @@ -239,14 +289,14 @@ from .storage_intent import StorageIntent, StorageIntentError, parse_storage_int __all__ = ["StorageIntent", "StorageIntentError", "parse_storage_intent"] ``` -- [ ] **Step 3: Turn the Python RED green and compile the changed modules.** +- [ ] **Step 3: Compile changed modules, then run the Python boundary probe once.** ```powershell -$env:PYTEST_DISABLE_PLUGIN_AUTOLOAD='1'; python -m pytest tests/test_storage_firewall.py::test_storage_intent_rejects_absolute_path_and_unknown_descriptor_key -q -o cache_dir=G:\2718lab\_codex\.codex-task-temp\storage-113-firewall-pytest python -m py_compile devkit_runtime/storage_intent.py devkit_runtime/__init__.py +$env:PYTEST_DISABLE_PLUGIN_AUTOLOAD='1'; python -m pytest tests/test_storage_firewall.py::test_storage_intent_rejects_absolute_path_and_unknown_descriptor_key -q -o cache_dir=G:\2718lab\_codex\.codex-task-temp\storage-113-firewall-pytest ``` -Expected: `1 passed`, then a successful compile with no output. +Expected: successful compile with no output, then `1 passed`. - [ ] **Step 4: Commit the Python contract.** @@ -255,83 +305,79 @@ git add mcp-tools/devkit_runtime/storage_intent.py mcp-tools/devkit_runtime/__in git commit -m "feat: add canonical storage intent" ``` -### Task 3: Bind intents to every Fast Lane assignment +### Task 3: Bind storage alongside immutable Fast Lane assignments **Files:** - Modify: `mcp-tools/devkit_fastlane/scripts/authenticated_v5_planner.py:124-333` - Modify: `mcp-tools/devkit_fastlane/scripts/authenticated_v5_projection.py:186-284` - Modify: `mcp-tools/devkit_runtime/fastlane_host_intent.py:333-595` +- Modify: `mcp-tools/devkit_runtime/fastlane_host_adapter.py` +- Modify: `codex-rs/core/src/fast_lane_host_dispatch/contract.rs` +- Modify: `codex-rs/core/src/fast_lane_host_dispatch/registry.rs` - Test: `mcp-tools/tests/test_storage_firewall.py` -- [ ] **Step 1: Add a RED assertion that initial and successor assignments carry the same exact intent binding.** - -```python -def test_every_fastlane_wave_carries_plan_context_bound_storage_intent(): - from devkit_fastlane.scripts.authenticated_v5_planner import compile_skeletons - - first, successor = compile_skeletons(SOURCE_UNITS, source_plan_hash=PLAN_HASH, context=CONTEXT) - assert first[0]["storage_intent"]["task_id"] == first[0]["task_id"] - assert successor[0]["storage_intent"]["plan_binding"] == PLAN_HASH - assert successor[0]["storage_intent"]["context_hash"] == CONTEXT["execution_context_hash"] -``` - -- [ ] **Step 2: Add `storage_intent` to the exact planner/projection field sets and construct it from normalized assignment data.** - -```python -intent = make_storage_intent( - task_id=unit["task_id"], - plan_binding=source_hash, - context_hash=context["execution_context_hash"], - artifact_kind="fastlane-task", - repository_identity=context["repository_identity"], - workspace_manifest_hash=context["workspace_manifest_hash"], - cargo_lock_hash=context["cargo_lock_hash"], - toolchain_digest=context["toolchain_digest"], - target_triple=context["target_triple"], - profile=context["profile"], - features_hash=context["features_hash"], - build_env_class=context["build_env_class"], - requested_bytes=unit["storage_budget"]["bytes"], - requested_files=unit["storage_budget"]["files"], -) -assignment["storage_intent"] = intent -``` - -The compiler may not synthesize a default budget. A source unit without the -two positive budget values fails with `STORAGE_POLICY_MISSING` during compile; -it never gets a guessed path or a guessed owner. The projection hash must -include the complete `storage_intent` object before `dispatch_binding_hash` is -computed. +- [ ] **Step 1: Keep one focused binding regression.** + +The initial case constructs intents from real Host profile fixtures and +explicit budgets, rejects changed task/plan/context/profile facts, and asserts +that original assignment and batch hashes are unchanged. The successor case +belongs at Rust `consume_refill_queue`: Python does not materialize successor +assignments. Until that path is wired, budgeted remaining work must fail closed +before dispatch; do not move `all_units` into the initial capacity/lease wave. + +- [ ] **Step 2: Construct initial intents from verified profiles and retain explicit successor budgets.** + +Use the existing `_storage_intents_for_profiles` path in +`fastlane_host_adapter.py`; the public caller supplies requested budgets, not +repository/toolchain/context facts. Missing positive budgets fail with +`STORAGE_POLICY_MISSING`, never a default. Retain remaining budgets keyed by +task in the exact refill request/ledger and include them in its canonical +binding; Host checks exact remaining-task coverage and verified index refs. +Registration does not reserve storage for unselected successors. + +`claim_evidence` already registers runtime, `by_batch`, `batch_intents`, and +scope leases using the original route/lease batch hash before profile exchange. +Keep those hashes and their preimages unchanged. Define a separate Host-owned +`StorageAssignmentBinding` whose canonical proof binds original batch hash, +task ID, verified profile provenance, storage intent hash, admission ID, and +family lease ID. For initial work the provenance references the completed +profile attestation; for successors it references native selected-wave +authority. Store the proof and private member grant in Host runtime facts; +the digest alone is not authority. Do not append storage and recompute the +original `dispatch_binding_hash` or `batch_hash`. - [ ] **Step 3: Parse and compare the binding at the host-intent boundary.** ```python -parsed = parse_storage_intent(candidate["storage_intent"]) -if parsed.task_id != candidate["task_id"]: +parsed = parse_storage_intent(storage_candidate) +if parsed.task_id != verified_profile["task_id"]: raise ValueError("STORAGE_TARGET_KEY_INVALID") if parsed.plan_binding != source_plan_hash: raise ValueError("STORAGE_TARGET_KEY_INVALID") -if parsed.context_hash != candidate["execution_context_hash"]: +if parsed.context_hash != verified_profile["execution_context_hash"]: raise ValueError("STORAGE_TARGET_KEY_INVALID") ``` -- [ ] **Step 4: Run the focused RED/GREEN probe and Python compile gate.** +- [ ] **Step 4: Compile changed Python modules, then run the focused binding probe once.** ```powershell -$env:PYTEST_DISABLE_PLUGIN_AUTOLOAD='1'; python -m pytest tests/test_storage_firewall.py::test_every_fastlane_wave_carries_plan_context_bound_storage_intent -q -o cache_dir=G:\2718lab\_codex\.codex-task-temp\storage-113-firewall-pytest -python -m py_compile devkit_fastlane/scripts/authenticated_v5_planner.py devkit_fastlane/scripts/authenticated_v5_projection.py devkit_runtime/fastlane_host_intent.py +python -m py_compile devkit_fastlane/scripts/authenticated_v5_planner.py devkit_fastlane/scripts/authenticated_v5_projection.py devkit_runtime/fastlane_host_intent.py devkit_runtime/fastlane_host_adapter.py +$env:PYTEST_DISABLE_PLUGIN_AUTOLOAD='1'; python -m pytest tests/test_storage_firewall.py -k storage_binding -q -o cache_dir=G:\2718lab\_codex\.codex-task-temp\storage-113-firewall-pytest ``` -Expected: `1 passed` and a successful compile. Do not run the broad Fast Lane -matrix in this plan. +Expected: successful compile and the selected binding regression passes. +An empty test selection is not a pass. Do not run the broad Fast Lane matrix. - [ ] **Step 5: Commit the binding change.** ```powershell -git add mcp-tools/devkit_fastlane/scripts/authenticated_v5_planner.py mcp-tools/devkit_fastlane/scripts/authenticated_v5_projection.py mcp-tools/devkit_runtime/fastlane_host_intent.py mcp-tools/tests/test_storage_firewall.py +git add mcp-tools/devkit_fastlane/scripts/authenticated_v5_planner.py mcp-tools/devkit_fastlane/scripts/authenticated_v5_projection.py mcp-tools/devkit_runtime/fastlane_host_intent.py mcp-tools/devkit_runtime/fastlane_host_adapter.py mcp-tools/tests/test_storage_firewall.py git commit -m "feat: bind storage intents to fast lane waves" ``` +Commit Host binding/refill changes with the Task 6 Host slice; do not treat +the Python commit as complete successor production wiring. + ### Task 4: Add the authenticated bridge exchange **Files:** @@ -339,6 +385,10 @@ git commit -m "feat: bind storage intents to fast lane waves" - Modify: `mcp-tools/devkit_runtime/host_session.py:159-335,636-735` - Modify: `codex-rs/rmcp-client/src/inherited_host_bridge_protocol.rs:35-240` - Modify: `codex-rs/rmcp-client/src/inherited_host_bridge_protocol/envelope.rs:25-220` +- Modify: `codex-rs/rmcp-client/src/inherited_host_bridge_protocol/session.rs` +- Modify: `codex-rs/rmcp-client/src/inherited_host_bridge_protocol/pump.rs` +- Modify: `codex-rs/core/src/fast_lane_host_dispatch/receiver.rs` +- Modify: `codex-rs/core/src/fast_lane_host_dispatch/registry.rs` - Test: `mcp-tools/tests/test_storage_firewall.py` - Test: `codex-rs/rmcp-client/src/inherited_host_bridge_protocol_tests.rs` @@ -346,44 +396,55 @@ git commit -m "feat: bind storage intents to fast lane waves" ```python def test_storage_admission_request_is_session_bound_and_replay_stable(): - request = build_storage_admission_request(INTENT) + request = build_storage_admission_request(INTENT, profile_attestation_hash=PROFILE_ATTESTATION_HASH) assert request["schema"] == "2718lab.storage.admission-request.v1" - assert set(request) == {"schema", "correlation_id", "storage_intent", "request_hash"} + assert set(request) == {"schema", "correlation_id", "profile_attestation_hash", "storage_intent", "request_hash"} assert request["request_hash"] == canonical_hash({key: request[key] for key in request if key != "request_hash"}) ``` - [ ] **Step 2: Implement exact validation on both sides.** -```rust -pub(crate) const STORAGE_ADMISSION_REQUEST_SCHEMA: &str = - "2718lab.storage.admission-request.v1"; -pub(crate) const STORAGE_ADMISSION_RECEIPT_SCHEMA: &str = - "2718lab.storage.admission-receipt.v1"; - -fn validate_storage_admission(payload: &Map) -> Result { - require_exact_fields(payload, &["schema", "correlation_id", "storage_intent", "request_hash"])?; - if value_str(payload, "schema")? != STORAGE_ADMISSION_REQUEST_SCHEMA { - return Err(HostBridgeProtocolError::InvalidOperation); - } - let intent = validate_storage_intent(value_object(payload, "storage_intent")?)?; - let request_hash = strict_digest(value_str(payload, "request_hash")?)?; - if digest(&canonical_bytes(&without_field(payload, "request_hash")?)?) != request_hash { - return Err(HostBridgeProtocolError::InvalidOperation); - } - Ok(StorageAdmissionRequest::new(value_str(payload, "correlation_id")?.into(), intent, request_hash)) -} -``` +The exact request is `{schema, correlation_id, profile_attestation_hash, +storage_intent, request_hash}` with schema +`2718lab.storage.admission-request.v1`. `request_hash` hashes all fields +except itself. Frame `action_id` must equal `correlation_id`; correlation is +only a reply/replay key and must never be parsed or guessed into an owner. + +After authenticated frame verification, resolve `profile_attestation_hash` +only in this `SessionCore`'s completed-profile table. Extend the existing +`StorageProfileCompletion` beyond preparation/task/expiry to retain the full +verified profile binding and response identity. A digest supplied by the +caller, or a successful recomputation of an attestation hash, is not evidence +that Host issued or sent that profile. Cross-session references fail closed. + +Core `claim_storage_profile` must retain an `IssuedStorageProfile` record: +call intent, preparation, task, source plan, index attestation, full profile, +profile/request/attestation hashes, original runtime batch, generation, +expiry, and `Issued`/`Sent` state. After the existing receiver obtains the +actual profile transport receipt, `mark_storage_profile_sent` validates it +against that record and marks `Sent`. Only then may `claim_storage_admission` +accept the bridge-resolved reference. Do not reinterpret the existing +request-specific `session_binding_hash` as a universal session identifier. + +Admission rechecks core's `Sent` record, the current preparation and verified +index binding, and exact intent task/plan/context/descriptor equality. The +caller can request bytes/files, but cannot mint profile facts, paths, a group +owner, generation, or expiry. The Host-derived deadline is the minimum of +transport, issued profile, preparation, and index authority deadlines; the +pending record, response, and completion receipt use that same deadline. +Use the shared rmcp-client intent codec and then construct the private-field +`HostStorageAdmissionRequest` from the resolved binding, not from JSON alone. The Python and Rust validators must reject different sessions, duplicate correlation IDs, frame payloads above the existing `MAX_OPERATION_BYTES`, unknown fields, and a receipt whose `storage_intent_hash` or `target_key` does not match the request. No public MCP response may include the absolute root. -- [ ] **Step 3: Add `HostSession.request_storage_admission(intent)` and its typed receipt.** +- [ ] **Step 3: Add the bound HostSession call and separated decision/transport receipts.** ```python -def request_storage_admission(self, intent: StorageIntent) -> StorageAdmissionReceipt | str: - request = build_storage_admission_request(intent) +def request_storage_admission(self, intent: StorageIntent, *, profile_attestation_hash: str) -> StorageAdmissionReceipt | str: + request = build_storage_admission_request(intent, profile_attestation_hash=profile_attestation_hash) response = self._bridge.request_storage_admission(request) if not isinstance(response, StorageAdmissionReceipt): return "STORAGE_STAT_UNAVAILABLE" @@ -391,33 +452,47 @@ def request_storage_admission(self, intent: StorageIntent) -> StorageAdmissionRe ``` The Rust session routes this message through the existing single writer queue; -it must not add a second receiver for the same bridge direction. +it must not add a second receiver for the same bridge direction. Return the +decision and response envelope defined in the shared wire contract; retain +transport completion separately. A new correlation for the same verified +member/intent may retrieve the same decision, never reserve again. Exact +correlation replay remains rejected. A changed intent for an existing member +is a conflict, not a budget update. -- [ ] **Step 4: Run the bridge-focused probes and compile gates.** +- [ ] **Step 4: Run compile gates first, then the bridge-focused probes once.** ```powershell +python -m py_compile devkit_runtime/host_bridge.py devkit_runtime/host_session.py $env:PYTEST_DISABLE_PLUGIN_AUTOLOAD='1'; python -m pytest tests/test_storage_firewall.py::test_storage_admission_request_is_session_bound_and_replay_stable -q -o cache_dir=G:\2718lab\_codex\.codex-task-temp\storage-113-firewall-pytest -$env:CARGO_TARGET_DIR='G:\2718lab\_codex\.codex-task-temp\storage-113-firewall-rust-target'; cargo test -p codex-rmcp-client storage_admission --locked -j1 +Push-Location 'G:\2718lab\_codex\.codex-task-temp\codex-host-mcp-fix-recovery2\codex-rs' +$env:CARGO_TARGET_DIR='G:\2718lab\_codex\.codex-task-temp\codex-host-mcp-fix-recovery2\codex-rs\target' +$env:CARGO_INCREMENTAL='0' +cargo check -p codex-rmcp-client -p codex-core --lib --locked -j1 +cargo test -p codex-rmcp-client storage_admission --locked -j1 +Pop-Location ``` -Expected: Python `1 passed`; Rust storage protocol tests pass with no new -target root outside the named task target. +Run the selected tests only after their compile gate succeeds. Expected: +Python `1 passed`; selected Rust protocol tests pass using the existing Host +target. Do not turn this slice into a full-suite run. - [ ] **Step 5: Commit the protocol slice.** ```powershell Push-Location 'G:\2718lab\_codex\.codex-task-temp\devkit-1.1.2-recovery'; git add mcp-tools/devkit_runtime/host_bridge.py mcp-tools/devkit_runtime/host_session.py mcp-tools/tests/test_storage_firewall.py; git commit -m 'feat: carry storage admission over host bridge'; Pop-Location -Push-Location 'G:\2718lab\_codex\.codex-task-temp\codex-host-mcp-fix-recovery2'; git add codex-rs/rmcp-client/src/inherited_host_bridge_protocol.rs codex-rs/rmcp-client/src/inherited_host_bridge_protocol/envelope.rs codex-rs/rmcp-client/src/inherited_host_bridge_protocol_tests.rs; git commit -m 'feat: carry storage admission over host bridge'; Pop-Location +Push-Location 'G:\2718lab\_codex\.codex-task-temp\codex-host-mcp-fix-recovery2'; git add codex-rs/rmcp-client/src/inherited_host_bridge_protocol.rs codex-rs/rmcp-client/src/inherited_host_bridge_protocol/envelope.rs codex-rs/rmcp-client/src/inherited_host_bridge_protocol/session.rs codex-rs/rmcp-client/src/inherited_host_bridge_protocol/pump.rs codex-rs/rmcp-client/src/inherited_host_bridge_protocol_tests.rs codex-rs/core/src/fast_lane_host_dispatch/receiver.rs codex-rs/core/src/fast_lane_host_dispatch/registry.rs; git commit -m 'feat: carry storage admission over host bridge'; Pop-Location ``` ### Task 5: Implement host target-key and capacity admission **Files:** +- Create: `codex-rs/rmcp-client/src/storage_intent.rs` +- Modify: `codex-rs/rmcp-client/src/lib.rs` - Create: `codex-rs/core/src/fast_lane_host_dispatch/storage_firewall.rs` -- Create: `codex-rs/core/src/fast_lane_host_dispatch/storage_firewall_tests.rs` +- Modify: `codex-rs/core/src/fast_lane_host_dispatch/storage_firewall_tests.rs` - Modify: `codex-rs/core/src/fast_lane_host_dispatch/mod.rs:1-49` - Modify: `codex-rs/core/Cargo.toml:80-145` -- Modify: `codex-rs/Cargo.lock` and `codex-rs/MODULE.bazel.lock` +- Modify if dependencies change: `codex-rs/Cargo.lock` and Host-root `MODULE.bazel.lock` - [ ] **Step 1: Add the Rust RED target-key vectors.** @@ -434,7 +509,13 @@ fn target_key_reuses_only_identical_build_semantics() { } ``` -- [ ] **Step 2: Implement the exact policy and deterministic root mapping.** +- [ ] **Step 2: Implement the shared codec, exact policy, and Host-only group/member admission.** + +`rmcp-client/src/storage_intent.rs` owns the one strict Rust `StorageIntent` +and target codec, using the crate's existing `sha2`/`serde_json`. Export from +`lib.rs`; core `storage_firewall.rs` re-exports the intent type to preserve +the existing RED struct literals/imports. Admission protocol and firewall +must use this same codec; no duplicate validator or reversed core dependency. ```rust #[derive(Clone, Debug, PartialEq, Eq)] @@ -448,50 +529,49 @@ pub(crate) struct StoragePolicy { pub(crate) free_space_floor_bytes: u64, pub(crate) emergency_floor_bytes: u64, } - -pub(crate) fn admit(&self, intent: &StorageIntent) -> Result { - let policy = self.policy.as_ref().ok_or(StorageError::PolicyMissing)?; - let free_before = self.capacity.free_bytes().map_err(|_| StorageError::StatUnavailable)?; - if intent.requested_bytes > policy.task_byte_limit { - return Err(StorageError::QuotaExceeded); - } - if intent.requested_files > policy.task_file_limit { - return Err(StorageError::FileLimitExceeded); - } - if self.family_observed_bytes + self.family_reserved_bytes + intent.requested_bytes - > policy.target_family_byte_limit - { - return Err(StorageError::QuotaExceeded); - } - if self.family_observed_files + self.family_reserved_files + intent.requested_files - > policy.target_family_file_limit - { - return Err(StorageError::FileLimitExceeded); - } - if self.global_reserved_bytes + intent.requested_bytes > policy.global_reserved_byte_limit - || self.global_reserved_files + intent.requested_files > policy.global_reserved_file_limit - { - return Err(StorageError::QuotaExceeded); - } - if free_before < policy.free_space_floor_bytes - || free_before - (self.global_reserved_bytes + intent.requested_bytes) - < policy.free_space_floor_bytes - { - return Err(StorageError::FreeSpaceFloor); - } - let target_key = target_key(&intent.target_descriptor)?; - let assigned_root = self.approved_root.join("generated").join(&target_key[7..]); - verify_strict_child(&self.approved_root, &assigned_root)?; - Ok(self.reserve(target_key, assigned_root, free_before, intent, policy)) -} ``` -`StorageError::code()` must return exactly one of the Plan 1 codes +Define `StorageAdmissionAuthority`, `StorageGroupOwner`, and member grants as +Host-only records, never deserializable caller claims. The initial owner is +the verified pending runtime batch; a successor owner is the Host-created +selected-wave attempt. `reserve_member_once` accepts validated authority and +intent; `seal_group(expected_tasks)` requires the exact Host task set, with no +missing/extra/duplicate member, before any writer may prepare. + +For each new member, check positive policy/budget values and task limits, then +family observed + reserved + requested bytes/files, global reserved + requested +bytes/files, and free bytes minus global reserved/requested against the floor. +Use checked arithmetic throughout; any overflow or unavailable measurement +fails closed. Below the emergency floor, do not admit new writers. Serialize +reservation updates under one state lock and count each member budget once. + +The same owner and target key share one family lease. An identical member and +intent returns its existing grant without charging again; a changed intent +conflicts. A different owner conflicts while the lease remains held. Sum +member budgets rather than reserving an entire family again per assignment. +This permits scope-disjoint writers in the same wave instead of making the +second same-key assignment permanently `NO_SAFE_WORK`. + +Map the family to `approved_root/generated/` using strict +child/reparse checks. Member scratch directories are separate Host-assigned +children; the Cargo cache remains one shared canonical target root. Do not +put task/member suffixes in `target_key` or duplicate the Cargo cache. Shared +cache writes require the supported Cargo locking contract; other outputs +must remain in disjoint member grants. Family ownership survives until the +last member's confirmed shutdown and successful postcheck; cloning a grant +or replaying a request neither reserves nor releases capacity. Admission +expiry is a deadline for starting/attaching a grant, not permission to release +a running member without confirmed shutdown and postcheck. + +`StorageError::code()` must include the Plan 1 admission codes `STORAGE_ROOT_NOT_APPROVED`, `STORAGE_TARGET_KEY_INVALID`, `STORAGE_POLICY_MISSING`, `STORAGE_QUOTA_EXCEEDED`, `STORAGE_FILE_LIMIT_EXCEEDED`, `STORAGE_FREE_SPACE_FLOOR`, and -`STORAGE_STAT_UNAVAILABLE`. A missing/overflowed policy field is never treated -as zero and never treated as unlimited. +`STORAGE_STAT_UNAVAILABLE`, plus `STORAGE_LEASE_CONFLICT` for a different owner +or changed member intent and `STORAGE_POSTCHECK_FAILED` for retained ownership +after failed postcheck. A missing/overflowed policy field is never treated as +zero or unlimited. Preserve the existing missing-policy `admit(intent())` +test seam, but production admission requires verified Host authority. - [ ] **Step 3: Implement platform capacity providers using the existing CLI doctor implementation as the reference.** @@ -501,20 +581,25 @@ On Unix call `libc::statvfs`; on Windows call tests do not query the real G drive. `StorageFirewall::new` must not create the approved root or target directory during construction or failed admission. -- [ ] **Step 4: Run only the target-key and admission probes.** +- [ ] **Step 4: Compile both changed crates, then run only selected boundary probes once.** ```powershell -$env:CARGO_TARGET_DIR='G:\2718lab\_codex\.codex-task-temp\storage-113-firewall-rust-target'; cargo test -p codex-core target_key_reuses_only_identical_build_semantics --locked -j1 -$env:CARGO_TARGET_DIR='G:\2718lab\_codex\.codex-task-temp\storage-113-firewall-rust-target'; cargo test -p codex-core missing_policy_is_stable_and_does_not_create_a_root --locked -j1 +$env:CARGO_TARGET_DIR='G:\2718lab\_codex\.codex-task-temp\codex-host-mcp-fix-recovery2\codex-rs\target' +$env:CARGO_INCREMENTAL='0' +cargo check -p codex-rmcp-client -p codex-core --lib --locked -j1 +cargo test -p codex-core target_key_reuses_only_identical_build_semantics --locked -j1 +cargo test -p codex-core missing_policy_is_stable_and_does_not_create_a_root --locked -j1 ``` -Expected: both tests pass. If free-space statistics fail, the command must -stop with the stable error and must not create a second Cargo target. +Stop on compile failure. Expected afterward: both selected tests pass. Keep +one small same-owner sharing/cross-owner conflict/seal regression with the +kernel, not a broad matrix. If space or statistics are unavailable, stop; +do not create a second Cargo target or repeatedly rerun the suite. - [ ] **Step 5: Commit the host firewall.** ```powershell -Push-Location 'G:\2718lab\_codex\.codex-task-temp\codex-host-mcp-fix-recovery2'; git add codex-rs/core/src/fast_lane_host_dispatch/storage_firewall.rs codex-rs/core/src/fast_lane_host_dispatch/storage_firewall_tests.rs codex-rs/core/src/fast_lane_host_dispatch/mod.rs codex-rs/core/Cargo.toml codex-rs/Cargo.lock codex-rs/MODULE.bazel.lock; git commit -m 'feat: enforce deterministic storage admission'; Pop-Location +Push-Location 'G:\2718lab\_codex\.codex-task-temp\codex-host-mcp-fix-recovery2'; git add codex-rs/rmcp-client/src/storage_intent.rs codex-rs/rmcp-client/src/lib.rs codex-rs/core/src/fast_lane_host_dispatch/storage_firewall.rs codex-rs/core/src/fast_lane_host_dispatch/storage_firewall_tests.rs codex-rs/core/src/fast_lane_host_dispatch/mod.rs codex-rs/core/Cargo.toml codex-rs/Cargo.lock MODULE.bazel.lock; git commit -m 'feat: enforce deterministic storage admission'; Pop-Location ``` ### Task 6: Connect admission to preparation, worker environment, and terminal release @@ -523,6 +608,14 @@ Push-Location 'G:\2718lab\_codex\.codex-task-temp\codex-host-mcp-fix-recovery2'; - Modify: `codex-rs/core/src/fast_lane_host_dispatch/registry.rs:567-1668` - Modify: `codex-rs/core/src/fast_lane_host_dispatch/coordinator.rs:393-580,1218-1260` - Modify: `codex-rs/core/src/fast_lane_host_dispatch/codex_adapter.rs:1-1444` +- Modify: `codex-rs/core/src/fast_lane_host_dispatch/contract.rs` +- Modify: `codex-rs/core/src/fast_lane_host_dispatch/storage_profile.rs` +- Modify: `codex-rs/rmcp-client/src/inherited_host_bridge_protocol.rs` +- Modify: `codex-rs/rmcp-client/src/inherited_host_bridge_protocol/session.rs` +- Modify: `codex-rs/rmcp-client/src/inherited_host_bridge_protocol/pump.rs` +- Modify: `mcp-tools/devkit_runtime/fastlane_host_adapter.py` +- Modify: `mcp-tools/devkit_runtime/host_bridge.py` +- Modify: `mcp-tools/devkit_runtime/host_session.py` - Modify: `mcp-tools/server.py:1008-1314` - Test: `mcp-tools/tests/test_storage_firewall.py` - Test: `codex-rs/core/src/fast_lane_host_dispatch/storage_firewall_tests.rs` @@ -537,22 +630,41 @@ def test_fastlane_dispatch_forwards_receipt_identity_without_public_path(): assert "CARGO_TARGET_DIR" not in result ``` -- [ ] **Step 2: Reserve before `prepare_batch`, inject after reservation, and release on every terminal/recovery branch.** - -```rust -let admission = self.storage.admit(&assignment.storage_intent)?; -let prepared = self.prepare_worktrees(&batch, &admission)?; -let mut environment = self.worker_environment(&prepared); -environment.insert("CARGO_TARGET_DIR".into(), admission.assigned_root().display().to_string()); -environment.insert("CODEX_TASK_TEMP".into(), admission.assigned_temp_root().display().to_string()); -``` - -The `assigned_root` and `assigned_temp_root` values remain in a private -`HostWriterContext`; only their identity hashes enter public receipts. On -`dispatch_all` error, terminal quarantine, successful integration, and -`recover_batch`, call `StorageFirewall::release` exactly once with the lease -ID. A release failure returns `STORAGE_POSTCHECK_FAILED` and retains the -ledger/lease state for Plan 2 recovery; it never triggers a broad delete. +- [ ] **Step 2: Connect both authority sources to one reservation service and consume sealed grants exactly once.** + +Initial `claim_storage_admission` resolves the Task 4 `Sent` profile and calls +`reserve_member_once` for the Host's original runtime batch owner. Store each +decision, `StorageAssignmentBinding`, and private grant in registry/runtime +state. `consume_batch` validates the original batch and exact member coverage, +seals the group, and transfers/attaches those grants to `HostWriterContext`. +It does not regenerate route/lease facts, rekey batch indexes, or reserve again. + +Successors are materialized natively by `consume_refill_queue`, not by a new +Python prepare call. Extend the exact refill codec/ledger with explicit +per-task budgets and bind them into the queue hash. `register_refill_queue` +validates coverage against its existing verified remaining skeletons and +index refs; it must not allocate capacity for all remaining work. After the +selected wave passes dependency/scope gates, resolve and revalidate its Host +profile provenance, construct intent and `StorageAssignmentBinding`, and call +the same `reserve_member_once` service with native selected-wave authority. +Do not manufacture bridge requests, correlations, nonces, or transport +receipts for this internal path. Attach sealed grants before the selected +wave can prepare. Failed admission does not advance the queue cursor; an old +queue without required budget/provenance is not silently upgraded or admitted. + +`CodexHostDispatchAdapter::prepare_batch` must require the already-sealed +grants before `worktree_broker.reserve_batch`, and must never call admission +again. At worker config construction, inject the private common Cargo target +and per-member scratch into `config.permissions.shell_environment_policy.r#set` +as `CARGO_TARGET_DIR` and `CODEX_TASK_TEMP`. These paths stay in private +`HostWriterContext`, not the caller's receipt or bounded public context. + +On prepare/dispatch failure, revoke unused member grants; after a writer has +started, terminal/recovery/integration paths must first confirm child shutdown +and perform postcheck. Release each member once and the family lease only +after the last member succeeds. A clone, timeout, or terminal ACK alone is +not release evidence. Postcheck failure returns `STORAGE_POSTCHECK_FAILED` +and retains ownership for later recovery; it never triggers a broad delete. - [ ] **Step 3: Make the DevKit Fast Lane result expose only stable storage code and receipt identity.** @@ -567,26 +679,33 @@ public = { - [ ] **Step 4: Run the one production-path probe plus compile-first gates.** ```powershell -$env:PYTEST_DISABLE_PLUGIN_AUTOLOAD='1'; python -m pytest tests/test_storage_firewall.py::test_fastlane_dispatch_forwards_receipt_identity_without_public_path -q -o cache_dir=G:\2718lab\_codex\.codex-task-temp\storage-113-firewall-pytest python -m py_compile server.py devkit_runtime/host_bridge.py devkit_runtime/host_session.py -Push-Location 'G:\2718lab\_codex\.codex-task-temp\codex-host-mcp-fix-recovery2\codex-rs'; $env:CARGO_TARGET_DIR='G:\2718lab\_codex\.codex-task-temp\storage-113-firewall-rust-target'; cargo check -p codex-core --lib --locked -j1; Pop-Location +Push-Location 'G:\2718lab\_codex\.codex-task-temp\codex-host-mcp-fix-recovery2\codex-rs' +$env:CARGO_TARGET_DIR='G:\2718lab\_codex\.codex-task-temp\codex-host-mcp-fix-recovery2\codex-rs\target' +$env:CARGO_INCREMENTAL='0' +cargo check -p codex-rmcp-client -p codex-core --lib --locked -j1 +Pop-Location +$env:PYTEST_DISABLE_PLUGIN_AUTOLOAD='1'; python -m pytest tests/test_storage_firewall.py::test_fastlane_dispatch_forwards_receipt_identity_without_public_path -q -o cache_dir=G:\2718lab\_codex\.codex-task-temp\storage-113-firewall-pytest ``` Expected: Python probe passes, `py_compile` is silent, and `cargo check` -finishes successfully with no new warning. This is the compile-first gate for -Plan 1; do not run the full workspace suite. +finishes successfully with no new warning. Stop before probes if compilation +fails. Keep one bounded Host successor/shared-group regression; Python-only +initial evidence is not full production acceptance. Do not run the full +workspace suite or repeatedly rebuild unchanged slices. - [ ] **Step 5: Commit the production wiring.** ```powershell -Push-Location 'G:\2718lab\_codex\.codex-task-temp\devkit-1.1.2-recovery'; git add mcp-tools/server.py mcp-tools/devkit_runtime/host_bridge.py mcp-tools/devkit_runtime/host_session.py mcp-tools/tests/test_storage_firewall.py; git commit -m 'feat: bind admitted storage roots to fast lane workers'; Pop-Location -Push-Location 'G:\2718lab\_codex\.codex-task-temp\codex-host-mcp-fix-recovery2'; git add codex-rs/core/src/fast_lane_host_dispatch/registry.rs codex-rs/core/src/fast_lane_host_dispatch/coordinator.rs codex-rs/core/src/fast_lane_host_dispatch/codex_adapter.rs codex-rs/core/src/fast_lane_host_dispatch/storage_firewall_tests.rs; git commit -m 'feat: bind admitted storage roots to fast lane workers'; Pop-Location +Push-Location 'G:\2718lab\_codex\.codex-task-temp\devkit-1.1.2-recovery'; git add mcp-tools/server.py mcp-tools/devkit_runtime/fastlane_host_adapter.py mcp-tools/devkit_runtime/host_bridge.py mcp-tools/devkit_runtime/host_session.py mcp-tools/tests/test_storage_firewall.py; git commit -m 'feat: bind admitted storage roots to fast lane workers'; Pop-Location +Push-Location 'G:\2718lab\_codex\.codex-task-temp\codex-host-mcp-fix-recovery2'; git add codex-rs/core/src/fast_lane_host_dispatch/registry.rs codex-rs/core/src/fast_lane_host_dispatch/coordinator.rs codex-rs/core/src/fast_lane_host_dispatch/codex_adapter.rs codex-rs/core/src/fast_lane_host_dispatch/contract.rs codex-rs/core/src/fast_lane_host_dispatch/storage_profile.rs codex-rs/core/src/fast_lane_host_dispatch/storage_firewall_tests.rs codex-rs/rmcp-client/src/inherited_host_bridge_protocol.rs codex-rs/rmcp-client/src/inherited_host_bridge_protocol/session.rs codex-rs/rmcp-client/src/inherited_host_bridge_protocol/pump.rs; git commit -m 'feat: bind admitted storage roots to fast lane workers'; Pop-Location ``` ## Plan 1 acceptance gate and handoff - [ ] Re-read the design sections “确定性 target key 与数据流” and “配额、文件数与剩余空间门槛”; verify every field and every fail-closed code is represented by a task above. - [ ] Run `git diff --check` in both worktrees and verify only the mapped files changed. -- [ ] Run `python -m py_compile` on every changed DevKit Python file and `cargo check -p codex-core --lib --locked -j1` with the one named task target. +- [ ] Run `python -m py_compile` on changed DevKit Python files and `cargo check -p codex-rmcp-client -p codex-core --lib --locked -j1` with the existing Host `codex-rs\target` and `CARGO_INCREMENTAL=0`; retain current compile evidence instead of rerunning unchanged slices. - [ ] Record one controlled admission receipt proving same semantics reuse one target key and one changed semantic forks it; record one low-space/policy-failure receipt proving no directory was created. +- [ ] Verify exact5 session lookup/core `Sent` binding, unchanged original batch hashes, same-owner member sharing with exact sealing, cross-owner conflict, and a native selected-successor admission without a second reservation. Do not label initial-only wiring complete. - [ ] Do not implement ledger persistence, preview/apply, source deletion, session deletion, compression, or remote synchronization in this plan. Plan 2 consumes `StorageAdmissionReceipt`; Plan 3 consumes the released/observed storage records. From 7a9e541e7769b43803840f782fef90542f102fe2 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E5=93=80=E6=B4=9B=E8=8A=99?= Date: Sun, 30 Aug 2026 05:51:05 +0800 Subject: [PATCH 19/39] feat: carry storage admission over host bridge --- mcp-tools/devkit_runtime/host_bridge.py | 244 ++++++++++++++++++++++- mcp-tools/devkit_runtime/host_session.py | 96 +++++++++ mcp-tools/tests/test_storage_firewall.py | 118 ++++++++++- 3 files changed, 455 insertions(+), 3 deletions(-) diff --git a/mcp-tools/devkit_runtime/host_bridge.py b/mcp-tools/devkit_runtime/host_bridge.py index 13e1497..b6f822e 100644 --- a/mcp-tools/devkit_runtime/host_bridge.py +++ b/mcp-tools/devkit_runtime/host_bridge.py @@ -20,7 +20,7 @@ import stat import struct import time -from collections.abc import Mapping, Sequence +from collections.abc import Callable, Mapping, Sequence from dataclasses import dataclass, field from threading import Event, Lock, RLock from typing import Final, cast @@ -30,6 +30,7 @@ host_envelopes, project_index_attestation_protocol, ) +from .storage_intent import StorageIntent, StorageIntentError, parse_storage_intent _FRAME_SCHEMA: Final = "2718lab-devkit/host-bridge-v1" _CAPABILITY_PROBE_SCHEMA: Final = "2718lab-devkit/host-capability-probe-v1" @@ -49,6 +50,19 @@ "2718lab-devkit/storage-profile-request-v1" ) _STORAGE_PROFILE_SCHEMA: Final = "2718lab-devkit/storage-profile-v1" +_STORAGE_ADMISSION_REQUEST_SCHEMA: Final = "2718lab.storage.admission-request.v1" +_STORAGE_ADMISSION_RESPONSE_SCHEMA: Final = "2718lab.storage.admission-response.v1" +_STORAGE_ADMISSION_RECEIPT_SCHEMA: Final = "2718lab.storage.admission-receipt.v1" +_STORAGE_ADMISSION_REQUEST_FIELDS: Final = frozenset( + {"schema", "correlation_id", "profile_attestation_hash", "storage_intent", "request_hash"} +) +_STORAGE_ADMISSION_RECEIPT_FIELDS: Final = frozenset( + {"schema", "admission_id", "profile_attestation_hash", "storage_intent_hash", + "storage_binding_hash", "target_key", "assigned_root_identity", + "target_family_lease_id", "reserved_bytes", "reserved_files", + "free_space_before", "free_space_after_reserve", "free_space_floor", + "expires_at", "receipt_hash"} +) _PROJECT_INDEX_ATTESTATION_SCHEMA: Final = ( project_index_attestation_protocol.ATTESTATION_SCHEMA ) @@ -160,6 +174,8 @@ "compiler_evidence_response", "storage_profile_request", "storage_profile_response", + "storage_admission_request", + "storage_admission_response", "project_index_attestation", "routing_attestation_request", "routing_attestation_response", @@ -179,6 +195,8 @@ "compiler_evidence_response", "storage_profile_request", "storage_profile_response", + "storage_admission_request", + "storage_admission_response", "project_index_attestation", "routing_attestation_request", "routing_attestation_response", @@ -304,6 +322,45 @@ class StorageProfileRequest: request_hash: str +@dataclass(frozen=True, slots=True) +class StorageAdmissionReceipt: + """Host decision, never a local reservation or a private filesystem path.""" + + schema: str + admission_id: str + profile_attestation_hash: str + storage_intent_hash: str + storage_binding_hash: str + target_key: str + assigned_root_identity: str + target_family_lease_id: str + reserved_bytes: int + reserved_files: int + free_space_before: int + free_space_after_reserve: int + free_space_floor: int + expires_at: int + receipt_hash: str + + def to_dict(self) -> dict[str, object]: + return {name: getattr(self, name) for name in sorted(_STORAGE_ADMISSION_RECEIPT_FIELDS)} + + +@dataclass(frozen=True) +class _StorageAdmissionCompletion: + """Local authenticated reception, separate from the Host capacity decision. + + Host generation stays opaque in profile-v1; bridge identity below only + identifies this Python transport instance, not a fabricated Host generation. + """ + + request_hash: str + response_hash: str + bridge_identity: object = field(repr=False) + expires_at: int + completed_at: int + + @dataclass(frozen=True) class FastLaneRefillRegistryRequest: """One authenticated queue of remaining V5 skeletons. @@ -406,6 +463,11 @@ def __init__( self._received_compiler_evidence: set[str] = set() self._pending_storage_profiles: dict[str, StorageProfileRequest] = {} self._received_storage_profiles: set[str] = set() + self._completed_storage_profiles: dict[str, dict[str, object]] = {} + self._storage_admission_correlations: set[str] = set() + self._storage_admission_decisions: dict[str, StorageAdmissionReceipt] = {} + self._storage_admission_completions: dict[str, _StorageAdmissionCompletion] = {} + self._storage_transport_identity = object() self._sent_fast_lane_refill_registries: set[str] = set() self._received_fast_lane_refill_registries: set[str] = set() self._received_project_index_attestations: set[str] = set() @@ -1548,8 +1610,84 @@ def receive_storage_profile_response( self._poison() raise del self._pending_storage_profiles[normalized_request.request_hash] + attestation = cast(str, normalized["attestation_hash"]) + previous = self._completed_storage_profiles.get(attestation) + if previous is not None and previous != normalized: + self._poison() + raise HostBridgeError("HOST_BRIDGE_STORAGE_PROFILE_INVALID") + self._completed_storage_profiles[attestation] = dict(normalized) return normalized + def request_storage_admission( + self, request: Mapping[str, object], *, now: int, expires_at: int, + clock: Callable[[], float], + ) -> StorageAdmissionReceipt: + """Exchange one decision before terminal reception starts; never admit locally. + + The owning HostSession serializes this round trip with preparation and + terminal-reader startup. The existing bridge I/O lock owns both frames. + """ + with self._io_lock: + normalized = _normalize_storage_admission_request(request) + correlation = cast(str, normalized["correlation_id"]) + attestation = cast(str, normalized["profile_attestation_hash"]) + profile = self._completed_storage_profiles.get(attestation) + intent = parse_storage_intent(normalized["storage_intent"]) + if profile is None: + raise HostBridgeError("HOST_BRIDGE_STORAGE_PROFILE_INVALID") + _validate_storage_admission_profile(intent, profile) + if ( + type(now) is not int or type(expires_at) is not int + or not 0 <= now < expires_at <= (1 << 64) - 1 + or correlation in self._storage_admission_correlations + ): + raise HostBridgeError("HOST_BRIDGE_STORAGE_ADMISSION_INVALID") + previous = self._storage_admission_decisions.get(attestation) + if previous is not None and previous.storage_intent_hash != intent.storage_intent_hash: + raise HostBridgeError("STORAGE_LEASE_CONFLICT") + # Burn before writing, including transport failure; retry must use a + # fresh correlation and still obtain a decision from the Host. + self._storage_admission_correlations.add(correlation) + try: + self._send_validated_private( + kind="storage_admission_request", action_id=correlation, payload=normalized + ) + message = self._receive_private() + if message.kind != "storage_admission_response" or message.action_id != correlation: + raise HostBridgeError("HOST_BRIDGE_STORAGE_ADMISSION_INVALID") + completed_time = clock() + if ( + type(completed_time) not in (int, float) + or not math.isfinite(completed_time) or completed_time < now + ): + raise HostBridgeError("HOST_BRIDGE_STORAGE_ADMISSION_INVALID") + completed_at = int(completed_time) + receipt = _normalize_storage_admission_response( + message.payload, request=normalized, now=completed_at, expires_at=expires_at + ) + if previous is not None and previous != receipt: + raise HostBridgeError("HOST_BRIDGE_STORAGE_ADMISSION_INVALID") + except HostBridgeError: + self._poison() + raise + self._storage_admission_decisions[attestation] = receipt + self._storage_admission_completions[correlation] = _StorageAdmissionCompletion( + request_hash=cast(str, normalized["request_hash"]), + response_hash=_private_payload_hash(message.payload), + bridge_identity=self._storage_transport_identity, + expires_at=receipt.expires_at, + completed_at=completed_at, + ) + return receipt + + def has_completed_storage_profile(self, profile: Mapping[str, object]) -> bool: + """Lookup transport-completed facts; caller-supplied hashes cannot enroll.""" + attestation = profile.get("attestation_hash") + return ( + self.is_available and type(attestation) is str + and self._completed_storage_profiles.get(attestation) == profile + ) + def receive_operation( self, *, @@ -3008,6 +3146,110 @@ def _is_index_correlation(value: object) -> bool: return project_index_attestation_protocol.is_index_correlation(value) +def build_storage_admission_request( + intent: StorageIntent, *, profile_attestation_hash: str +) -> dict[str, object]: + """Build exact5; a canonical digest is a reference, never proof of issuance.""" + if type(intent) is not StorageIntent: + raise HostBridgeError("HOST_BRIDGE_STORAGE_ADMISSION_INVALID") + unsigned = { + "schema": _STORAGE_ADMISSION_REQUEST_SCHEMA, + "correlation_id": "storage-admit-" + secrets.token_hex(32), + "profile_attestation_hash": profile_attestation_hash, + "storage_intent": intent.to_dict(), + } + return _normalize_storage_admission_request( + {**unsigned, "request_hash": _private_payload_hash(unsigned)} + ) + + +def _normalize_storage_admission_request(value: object) -> dict[str, object]: + if ( + type(value) is not dict or set(value) != _STORAGE_ADMISSION_REQUEST_FIELDS + or value.get("schema") != _STORAGE_ADMISSION_REQUEST_SCHEMA + ): + raise HostBridgeError("HOST_BRIDGE_STORAGE_ADMISSION_INVALID") + _validate_private_packet_size(value, _MAX_OPERATION_PACKET_BYTES) + correlation = value.get("correlation_id") + if type(correlation) is not str or _IDENTIFIER.fullmatch(correlation) is None: + raise HostBridgeError("HOST_BRIDGE_STORAGE_ADMISSION_INVALID") + for name in ("profile_attestation_hash", "request_hash"): + if type(value[name]) is not str or _DIGEST.fullmatch(value[name]) is None: + raise HostBridgeError("HOST_BRIDGE_STORAGE_ADMISSION_INVALID") + try: + intent = parse_storage_intent(value["storage_intent"]) + except StorageIntentError as error: + raise HostBridgeError(error.code) from error + normalized = {**value, "storage_intent": intent.to_dict()} + if normalized["request_hash"] != _private_payload_hash( + {name: item for name, item in normalized.items() if name != "request_hash"} + ): + raise HostBridgeError("HOST_BRIDGE_STORAGE_ADMISSION_INVALID") + return normalized + + +def _validate_storage_admission_profile( + intent: StorageIntent, profile: Mapping[str, object] +) -> None: + if ( + intent.task_id != profile.get("task_id") + or intent.plan_binding != profile.get("source_plan_hash") + or intent.context_hash != profile.get("execution_context_hash") + or intent.target_descriptor["artifact_kind"] != "fastlane-task" + or any(intent.target_descriptor[name] != profile.get(name) + for name in _STORAGE_DESCRIPTOR_FIELDS) + ): + raise HostBridgeError("STORAGE_TARGET_KEY_INVALID") + + +def _normalize_storage_admission_response( + value: object, *, request: Mapping[str, object], now: int, expires_at: int +) -> StorageAdmissionReceipt: + request = _normalize_storage_admission_request(request) + if ( + type(value) is not dict + or set(value) != {"schema", "correlation_id", "request_hash", "receipt"} + or value.get("schema") != _STORAGE_ADMISSION_RESPONSE_SCHEMA + or value.get("correlation_id") != request["correlation_id"] + or value.get("request_hash") != request["request_hash"] + ): + raise HostBridgeError("HOST_BRIDGE_STORAGE_ADMISSION_INVALID") + _validate_private_packet_size(value, _MAX_OPERATION_PACKET_BYTES) + receipt = value["receipt"] + if ( + type(receipt) is not dict or set(receipt) != _STORAGE_ADMISSION_RECEIPT_FIELDS + or receipt.get("schema") != _STORAGE_ADMISSION_RECEIPT_SCHEMA + ): + raise HostBridgeError("HOST_BRIDGE_STORAGE_ADMISSION_INVALID") + # Both Host-minted IDs are frozen SHA-256 identities, not UUIDs or paths. + for name in ("admission_id", "target_family_lease_id", "profile_attestation_hash", + "storage_intent_hash", "storage_binding_hash", "target_key", + "assigned_root_identity", "receipt_hash"): + if type(receipt[name]) is not str or _DIGEST.fullmatch(receipt[name]) is None: + raise HostBridgeError("HOST_BRIDGE_STORAGE_ADMISSION_INVALID") + for name in ("reserved_bytes", "reserved_files", "free_space_before", + "free_space_after_reserve", "free_space_floor", "expires_at"): + if type(receipt[name]) is not int or not 0 <= receipt[name] <= (1 << 64) - 1: + raise HostBridgeError("HOST_BRIDGE_STORAGE_ADMISSION_INVALID") + intent = parse_storage_intent(request["storage_intent"]) + if ( + type(now) is not int or type(expires_at) is not int + or not 0 <= now < receipt["expires_at"] <= expires_at <= (1 << 64) - 1 + or receipt["profile_attestation_hash"] != request["profile_attestation_hash"] + or receipt["storage_intent_hash"] != intent.storage_intent_hash + or receipt["target_key"] != _private_payload_hash(dict(intent.target_descriptor)) + or receipt["reserved_bytes"] != intent.requested_bytes + or receipt["reserved_files"] != intent.requested_files + or receipt["free_space_after_reserve"] > receipt["free_space_before"] - receipt["reserved_bytes"] + or receipt["free_space_after_reserve"] < receipt["free_space_floor"] + or receipt["receipt_hash"] != _private_payload_hash( + {name: item for name, item in receipt.items() if name != "receipt_hash"} + ) + ): + raise HostBridgeError("HOST_BRIDGE_STORAGE_ADMISSION_INVALID") + return StorageAdmissionReceipt(**receipt) + + def _normalize_storage_profile_request(value: object) -> StorageProfileRequest: """Validate one exact, nonce-bound private profile request.""" diff --git a/mcp-tools/devkit_runtime/host_session.py b/mcp-tools/devkit_runtime/host_session.py index df8f045..17250c6 100644 --- a/mcp-tools/devkit_runtime/host_session.py +++ b/mcp-tools/devkit_runtime/host_session.py @@ -23,12 +23,16 @@ HostBridgeError, InheritedHandleHostBridge, OperationReceipt, + StorageAdmissionReceipt, + _validate_storage_admission_profile, + build_storage_admission_request, ) from .host_scheduler_topology_adapter import ( HostAuthoritativeActionFact, HostSchedulerTopologyFact, construct_host_scheduler_topology, ) +from .storage_intent import StorageIntent, StorageIntentError, parse_storage_intent _NO_SAFE_WORK: Final = "NO_SAFE_WORK" _HASH_PREFIX: Final = "sha256:" @@ -268,6 +272,17 @@ class _PendingFastLaneTerminal: lease_expires_at: int = field(repr=False) +@dataclass(frozen=True) +class _CompletedStorageProfile: + """Local completion tied to a verified preparation and this exact bridge.""" + + profile: dict[str, object] = field(repr=False) + bridge: InheritedHandleHostBridge = field(repr=False) + expires_at: int + requested_bytes: int + requested_files: int + + class _CompilerPreparation: """A zero-field marker that a provider can only return unchanged.""" @@ -307,6 +322,7 @@ def __init__( self._compiler_evidence_lock = RLock() self._compiler_evidence: dict[_CompilerEvidenceHandle, _CompilerInvocation] = {} self._compiler_request_contexts: dict[str, _CompilerRequestContext] = {} + self._completed_storage_profiles: dict[str, _CompletedStorageProfile] = {} self._capability_snapshots_v2: dict[ tuple[str, str], _HostCapabilitySnapshotV2 ] = {} @@ -514,6 +530,7 @@ def close(self) -> None: self._frozen = True self._compiler_evidence.clear() self._compiler_request_contexts.clear() + self._completed_storage_profiles.clear() self._capability_snapshots_v2.clear() self._preparation_expiry_caps.clear() self._routing_attestation_snapshots.clear() @@ -617,6 +634,27 @@ def prepare_compiler_evidence( ): return _NO_SAFE_WORK evidence = _CompilerEvidenceHandle() + # Only the actual authenticated profile round trip can enroll a + # reference; an injected resolver/provider cannot mint authority. + bridge = self._bridge + if bridge is not None and binding_resolver == self._resolve_bridge_compiler_invocation: + budgets = {task: (byte_count, files) for task, byte_count, files + in material.storage_budget_bindings} + completed_profiles: dict[str, _CompletedStorageProfile] = {} + for profile in material.storage_profiles: + if not bridge.has_completed_storage_profile(profile): + return _NO_SAFE_WORK + attestation = cast(str, profile["attestation_hash"]) + byte_count, files = budgets[cast(str, profile["task_id"])] + completed = _CompletedStorageProfile( + profile=dict(profile), bridge=bridge, expires_at=int(expires_at), + requested_bytes=byte_count, requested_files=files, + ) + previous = completed_profiles.get(attestation) or self._completed_storage_profiles.get(attestation) + if previous is not None and previous != completed: + return _NO_SAFE_WORK + completed_profiles[attestation] = completed + self._completed_storage_profiles.update(completed_profiles) self._compiler_evidence[evidence] = material return evidence @@ -847,6 +885,63 @@ def project_index_query_attestation( return None return dict(value) + def request_storage_admission( + self, intent: StorageIntent, *, profile_attestation_hash: str + ) -> StorageAdmissionReceipt | str: + """Ask Host before dispatch; no profile hash or local cache admits work. + + Profile-v1 keeps Host generation/index deadlines opaque. Their final + validation belongs to Rust's completed-profile/Sent records. Locally + the same bridge, accepted preparation, exact profile and budgets are + mandatory, and the Host decision cannot extend preparation authority. + """ + with self._compiler_evidence_lock: + bridge = self._bridge + if not self.is_available or bridge is None: + return "STORAGE_STAT_UNAVAILABLE" + terminal_thread = self._fast_lane_terminal_thread + if terminal_thread is not None and terminal_thread.is_alive(): + return "STORAGE_STAT_UNAVAILABLE" + if type(profile_attestation_hash) is not str: + return "STORAGE_TARGET_KEY_INVALID" + completed = self._completed_storage_profiles.get(profile_attestation_hash) + if completed is None or completed.bridge is not bridge: + return "STORAGE_TARGET_KEY_INVALID" + try: + now = int(self._read_trusted_clock()) + if now >= completed.expires_at or not bridge.has_completed_storage_profile(completed.profile): + return "STORAGE_STAT_UNAVAILABLE" + if type(intent) is not StorageIntent: + return "STORAGE_TARGET_KEY_INVALID" + parsed = parse_storage_intent(intent.to_dict()) + _validate_storage_admission_profile(parsed, completed.profile) + if (parsed.requested_bytes, parsed.requested_files) != ( + completed.requested_bytes, completed.requested_files + ): + return "STORAGE_LEASE_CONFLICT" + request = build_storage_admission_request( + parsed, profile_attestation_hash=profile_attestation_hash + ) + receipt = bridge.request_storage_admission( + request, now=now, expires_at=completed.expires_at, + clock=self._read_trusted_clock, + ) + # Re-read the trusted clock after blocking I/O. A valid frame + # received after its deadline must not authorize a writer. + if int(self._read_trusted_clock()) >= receipt.expires_at: + return "STORAGE_STAT_UNAVAILABLE" + return receipt + except StorageIntentError as error: + return error.code + except HostBridgeError as error: + if error.code in {"STORAGE_TARGET_KEY_INVALID", "STORAGE_LEASE_CONFLICT"}: + return error.code + self._freeze() + return "STORAGE_STAT_UNAVAILABLE" + except (TypeError, ValueError): + self._freeze() + return "STORAGE_STAT_UNAVAILABLE" + def storage_profiles_for_compiler_evidence( self, evidence: object ) -> tuple[dict[str, object], ...] | str: @@ -1895,6 +1990,7 @@ def _freeze(self) -> None: self._frozen = True self._compiler_evidence.clear() self._compiler_request_contexts.clear() + self._completed_storage_profiles.clear() if self._bridge is not None: self._bridge.close() diff --git a/mcp-tools/tests/test_storage_firewall.py b/mcp-tools/tests/test_storage_firewall.py index f269404..34214ee 100644 --- a/mcp-tools/tests/test_storage_firewall.py +++ b/mcp-tools/tests/test_storage_firewall.py @@ -11,7 +11,6 @@ import pytest - MCP_TOOLS = Path(__file__).resolve().parents[1] TESTS = Path(__file__).resolve().parent for _path in (MCP_TOOLS, TESTS): @@ -364,10 +363,47 @@ def test_pre_host_skeleton_remains_the_legacy_exact_eight_fields() -> None: def test_verified_private_profile_round_trip_constructs_local_intent( monkeypatch: pytest.MonkeyPatch, ) -> None: + _verified_profile_round_trip(monkeypatch, admit=False) + + +def test_storage_admission_request_is_session_bound_and_replay_stable( + monkeypatch: pytest.MonkeyPatch, +) -> None: + _verified_profile_round_trip(monkeypatch, admit=True) + + +def _admission_response(request: dict[str, object]) -> dict[str, object]: + intent = request["storage_intent"] + receipt = { + "schema": "2718lab.storage.admission-receipt.v1", + "admission_id": _hash("a"), + "profile_attestation_hash": request["profile_attestation_hash"], + "storage_intent_hash": intent["storage_intent_hash"], + "storage_binding_hash": _hash("b"), + "target_key": _canonical_hash(intent["target_descriptor"]), + "assigned_root_identity": _hash("c"), + "target_family_lease_id": _hash("d"), + "reserved_bytes": intent["requested_bytes"], + "reserved_files": intent["requested_files"], + "free_space_before": 8192, + "free_space_after_reserve": 4096, + "free_space_floor": 1024, + "expires_at": 1_700_000_060, + } + receipt["receipt_hash"] = _canonical_hash(receipt) + return { + "schema": "2718lab.storage.admission-response.v1", + "correlation_id": request["correlation_id"], + "request_hash": request["request_hash"], + "receipt": receipt, + } + + +def _verified_profile_round_trip(monkeypatch: pytest.MonkeyPatch, *, admit: bool) -> None: """Exercise the framed Host bridge and local post-profile compilation.""" - from devkit_runtime import fastlane_host_adapter as adapter import devkit_runtime.host_session as host_session + from devkit_runtime import fastlane_host_adapter as adapter from devkit_runtime.host_bridge import InheritedHandleHostBridge fixture = _authenticated_v5_fixture() @@ -376,6 +412,7 @@ def test_verified_private_profile_round_trip_constructs_local_intent( fact_mapping = adapter._dispatch_fact_mapping(fact) lease_hash = adapter._lease_scope_binding_hash(fact) failure: list[BaseException] = [] + admission_requests: list[dict[str, object]] = [] def host_reply() -> None: try: @@ -416,6 +453,22 @@ def host_reply() -> None: request=profile_request, response=_storage_profile_response(profile_request), ) + if admit: + for _ in range(2): + message = host._receive_private() + request = message.payload + assert message.kind == "storage_admission_request" + assert message.action_id == request["correlation_id"] + assert set(request) == {"schema", "correlation_id", "profile_attestation_hash", "storage_intent", "request_hash"} + assert request["schema"] == "2718lab.storage.admission-request.v1" + assert request["request_hash"] == _canonical_hash( + {key: value for key, value in request.items() if key != "request_hash"} + ) + admission_requests.append(request) + host._send_validated_private( + kind="storage_admission_response", action_id=message.action_id, + payload=_admission_response(request), + ) except BaseException as error: # report peer errors after the round trip failure.append(error) @@ -452,6 +505,31 @@ def host_reply() -> None: assert type(prepared).__name__ == "_PreparedHostFacts" assert len(prepared.storage_intents) == 1 assert prepared.storage_intents[0]["task_id"] == "TASK-V5" + if admit: + from devkit_runtime.host_bridge import ( + HostBridgeError, + StorageAdmissionReceipt, + ) + from devkit_runtime.storage_intent import parse_storage_intent + + intent = parse_storage_intent(prepared.storage_intents[0]) + # Same attestation/profile facts on another Python session do not + # become an issued reference, even when the bridge object matches. + foreign = host_session.HostSession(bridge=child, clock=lambda: 1_700_000_000) + assert foreign.request_storage_admission(intent, profile_attestation_hash=_hash("7")) == "STORAGE_TARGET_KEY_INVALID" + first = session.request_storage_admission(intent, profile_attestation_hash=_hash("7")) + second = session.request_storage_admission(intent, profile_attestation_hash=_hash("7")) + assert isinstance(first, StorageAdmissionReceipt) + assert first == second + assert len(admission_requests) == 2 # no local decision/cache admission + assert admission_requests[0]["correlation_id"] != admission_requests[1]["correlation_id"] + assert len(child._storage_admission_completions) == 2 + with pytest.raises(HostBridgeError, match="HOST_BRIDGE_STORAGE_ADMISSION_INVALID"): + child.request_storage_admission( + admission_requests[0], now=1_700_000_000, expires_at=1_700_000_120, + clock=lambda: 1_700_000_000, + ) + assert set(first.to_dict()) == set(_admission_response(admission_requests[0])["receipt"]) batch = adapter.compile_fast_lane_with_host_facts( fixture["planner_request"], reasoning_effort="max", @@ -467,6 +545,41 @@ def host_reply() -> None: assert not failure +def test_storage_admission_rejects_substitution_and_unknown_fields() -> None: + from devkit_runtime import host_bridge + from devkit_runtime.storage_intent import parse_storage_intent + + intent = parse_storage_intent(_storage_intent( + task_id="TASK-V5", plan_binding=_hash("8"), context_hash=_hash("6") + )) + request = host_bridge.build_storage_admission_request(intent, profile_attestation_hash=_hash("7")) + response = _admission_response(request) + malformed = [dict(request, assigned_root="G:/private"), {key: value for key, value in request.items() if key != "profile_attestation_hash"}] + for candidate in malformed: + with pytest.raises(host_bridge.HostBridgeError): + host_bridge._normalize_storage_admission_request(candidate) + for field, value in (("target_key", _hash("f")), ("storage_intent_hash", _hash("f")), + ("profile_attestation_hash", _hash("f")), ("reserved_bytes", True), + ("reserved_files", 9), ("expires_at", 1_700_000_000), + ("admission_id", "G:/private"), ("assigned_root", "G:/private")): + candidate = copy.deepcopy(response) + candidate["receipt"][field] = value + candidate["receipt"]["receipt_hash"] = _canonical_hash( + {key: value for key, value in candidate["receipt"].items() if key != "receipt_hash"} + ) + with pytest.raises(host_bridge.HostBridgeError): + host_bridge._normalize_storage_admission_response( + candidate, request=request, now=1_700_000_000, expires_at=1_700_000_120 + ) + child, host = _pipe_pair() + try: + with pytest.raises(host_bridge.HostBridgeError, match="HOST_BRIDGE_STORAGE_PROFILE_INVALID"): + child.request_storage_admission(request, now=1_700_000_000, expires_at=1_700_000_120, clock=lambda: 1_700_000_000) + finally: + child.close() + host.close() + + def test_profile_tamper_or_missing_field_fails_closed() -> None: from devkit_runtime import host_bridge from devkit_runtime.host_bridge import HostBridgeError @@ -485,6 +598,7 @@ def test_profile_tamper_or_missing_field_fails_closed() -> None: def test_v2_remains_compatible_while_v3_requires_one_root_storage_intent() -> None: from test_fastlane_host_intent import _intent, _with_binding + from devkit_runtime.fastlane_host_intent import ( NO_SAFE_WORK, StorageIntentError, From 5af16d2188a668d7dc3dff5bdc9bd3813252737b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E5=93=80=E6=B4=9B=E8=8A=99?= Date: Sun, 30 Aug 2026 05:54:08 +0800 Subject: [PATCH 20/39] fix: reject zero storage admission free-space floor --- mcp-tools/devkit_runtime/host_bridge.py | 1 + mcp-tools/tests/test_storage_firewall.py | 1 + 2 files changed, 2 insertions(+) diff --git a/mcp-tools/devkit_runtime/host_bridge.py b/mcp-tools/devkit_runtime/host_bridge.py index b6f822e..c9d8fe3 100644 --- a/mcp-tools/devkit_runtime/host_bridge.py +++ b/mcp-tools/devkit_runtime/host_bridge.py @@ -3240,6 +3240,7 @@ def _normalize_storage_admission_response( or receipt["target_key"] != _private_payload_hash(dict(intent.target_descriptor)) or receipt["reserved_bytes"] != intent.requested_bytes or receipt["reserved_files"] != intent.requested_files + or receipt["free_space_floor"] == 0 or receipt["free_space_after_reserve"] > receipt["free_space_before"] - receipt["reserved_bytes"] or receipt["free_space_after_reserve"] < receipt["free_space_floor"] or receipt["receipt_hash"] != _private_payload_hash( diff --git a/mcp-tools/tests/test_storage_firewall.py b/mcp-tools/tests/test_storage_firewall.py index 34214ee..4cba2b1 100644 --- a/mcp-tools/tests/test_storage_firewall.py +++ b/mcp-tools/tests/test_storage_firewall.py @@ -561,6 +561,7 @@ def test_storage_admission_rejects_substitution_and_unknown_fields() -> None: for field, value in (("target_key", _hash("f")), ("storage_intent_hash", _hash("f")), ("profile_attestation_hash", _hash("f")), ("reserved_bytes", True), ("reserved_files", 9), ("expires_at", 1_700_000_000), + ("free_space_floor", 0), ("admission_id", "G:/private"), ("assigned_root", "G:/private")): candidate = copy.deepcopy(response) candidate["receipt"][field] = value From 65bf285a7e6ccbe11f3295b1a780f3da60f6a391 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E5=93=80=E6=B4=9B=E8=8A=99?= Date: Sun, 30 Aug 2026 06:04:01 +0800 Subject: [PATCH 21/39] docs: map storage runtime integration and release gates --- .../2026-08-29-storage-firewall-1.1.3.md | 399 +++++++++++++----- 1 file changed, 287 insertions(+), 112 deletions(-) diff --git a/docs/superpowers/plans/2026-08-29-storage-firewall-1.1.3.md b/docs/superpowers/plans/2026-08-29-storage-firewall-1.1.3.md index 32beef5..0541537 100644 --- a/docs/superpowers/plans/2026-08-29-storage-firewall-1.1.3.md +++ b/docs/superpowers/plans/2026-08-29-storage-firewall-1.1.3.md @@ -8,12 +8,16 @@ **Tech Stack:** Python 3.11 standard library (`dataclasses`, `hashlib`, `json`, `pathlib`), MCP FastMCP/Pydantic, Rust 2021, `serde`/`serde_json`, `sha2`, Tokio, platform filesystem-capacity APIs, and the existing authenticated inherited-handle bridge. -**Revision status (2026-08-30):** This is an unpublished protocol revision made -after the existing Task 4a slice passed its compile checks. That result does -not mean admission, group reservations, or successor production wiring below -is implemented. The unpublished admission-v1 request changes from exact4 to -exact5; intent/target/profile-v1 stay unchanged. If exact4 has been deployed -outside these worktrees, use admission-v2 instead and reject downgrade. +**Revision status (2026-08-30):** This is an unpublished protocol revision. +Preserve the implemented Task 4 profile/Sent/admission exchange and Task 5 +shared codec/kernel contracts and their recorded compile evidence; the Host +aggregate two-crate check at `99e5` was reported exit 0. That is not Task 6 +production acceptance. Task 6 below records the remaining configuration, +runtime sharing, filesystem ordering, process-proof, and postcheck work; no +checkbox is completed by this documentation update. The unpublished +admission-v1 request is exact5, replacing exact4; intent/target/profile-v1 stay +unchanged. If exact4 has been deployed outside these worktrees, use admission-v2 +instead and reject downgrade. **Compile-first execution:** Reuse the Host's existing `G:\2718lab\_codex\.codex-task-temp\codex-host-mcp-fix-recovery2\codex-rs\target`, @@ -27,10 +31,10 @@ claims that the revised contract is complete. ## Scope and file map -All line references are against commit `37029a9` in the DevKit worktree and -commit `552fe8035d` in the Codex Host worktree. A worker must re-read the -symbol at the listed line before editing because earlier tasks can shift line -numbers. +Original line references are against commit `37029a9` in the DevKit worktree +and commit `552fe8035d` in the Codex Host worktree. Task 6's symbol map was +refreshed from the Host `f26f6ae` working tree and concurrent Task 4/5 repairs. +Re-read each named symbol before editing; later commits shift line numbers. DevKit files: @@ -83,20 +87,49 @@ Codex Host files: to share Host-derived profile construction with selected-wave authority; successor materialization must not fabricate a bridge request. - Modify `codex-rs/core/src/fast_lane_host_dispatch/registry.rs:567-1668` to - reserve the target-family key before writer preparation and - `coordinator.rs:393-580,1218-1260` to release the admission on terminal or - recovery paths. + reserve/seal before filesystem materialization and consume before writer + preparation; modify `coordinator.rs:393-580,1218-1260` to settle only after + confirmed process-tree shutdown and successful family postcheck. - Modify `codex-rs/core/src/fast_lane_host_dispatch/codex_adapter.rs:1-1444` at worker environment construction so the host-issued root is the sole `CARGO_TARGET_DIR`/task-temp value. - Modify the existing `codex-rs/core/src/fast_lane_host_dispatch/storage_firewall_tests.rs`; preserve its sibling-module registration in `mod.rs`. -- Modify `codex-rs/core/Cargo.toml:80-145` by adding - `[target.'cfg(target_os = "windows")'.dependencies] windows-sys = { - version = "0.52", features = ["Win32_Storage_FileSystem"] }`, matching the - existing CLI dependency version; then update `codex-rs/Cargo.lock` and - `MODULE.bazel.lock` at the Host repository root in the same Host commit - when dependency changes require them. +- Modify `codex-rs/config/src/config_toml.rs` (`ConfigToml`), + `codex-rs/core/src/config/mod.rs` (`Config::load_config_with_layer_stack`), + and generated `codex-rs/core/config.schema.json` for the explicit Host + storage block. Read existing `config/src/config_layer_source.rs` for trust + provenance and `config/src/schema.rs::write_config_schema` for generation. +- Modify `codex-rs/app-server/src/message_processor.rs`, + `codex-rs/core/src/thread_manager.rs` (`ThreadManagerState`), + `codex-rs/core/src/session/mod.rs` (`SessionSpawnArgs`), + `codex-rs/core/src/session/session.rs`, and + `codex-rs/core/src/state/service.rs` (`SessionServices`) to inject one + Host-runtime service Arc into all registries. Create + `codex-rs/core/src/fast_lane_host_dispatch/storage_service.rs` for that + authority/ledger facade, registering it in the existing dispatch `mod.rs`. +- Modify `codex-rs/core/src/mcp_tool_call.rs` and + `codex-rs/core/src/fast_lane_host_dispatch/worktree.rs` for planned versus + materialized roots; no pre-admission `create_dir_all` remains. +- Modify `codex-rs/core/src/unified_exec/mod.rs`, `process.rs`, and + `process_manager.rs`; `codex-rs/utils/pty/src/process.rs` and `win/job.rs`; + and `codex-rs/core/src/session/handlers.rs` and `agent/control/legacy.rs` + for actual owned-process termination evidence. Read/reuse + `codex-rs/exec-server/src/process.rs` lifecycle events; modify that boundary + only if its existing exit events cannot carry the required confirmation. +- Modify `codex-rs/protocol/src/shell_environment.rs` at final environment + assembly, alongside core permissions/worker configuration, so reserved + environment values survive filtering and cannot be replaced by per-call + overrides in the storage-managed execution path. +- Create `codex-rs/core/src/fast_lane_host_dispatch/storage_postcheck.rs` and + register it in `mod.rs`: real no-follow family/member observations after + the process fence, not caller-supplied counters. + +The capacity provider now reuses existing platform FFI; no new Windows +dependency is required. Do not change manifests/locks for this documentation +or add a dependency merely because an older example requested it. +`MODULE.bazel.lock`, if a separately justified dependency change needs it, +is at the Host repository root, not under `codex-rs`. The worker must not edit any file outside this map. The ledger, preview/apply, source authorization, and session CAS changes belong to Plans 2 and 3. @@ -143,8 +176,12 @@ The exact successful `StorageAdmissionReceipt` decision has these fields: `expires_at`, `receipt_hash`. Its schema is `2718lab.storage.admission-receipt.v1`; `receipt_hash` hashes -all decision fields except itself. The successful private response envelope -is exact `{schema, correlation_id, request_hash, receipt}` with schema +all decision fields except itself. Both Host-minted IDs, `admission_id` and +`target_family_lease_id` (the internal family lease ID), are strictly +`sha256:` followed by 64 lowercase hexadecimal characters, not UUIDs, paths, +or arbitrary opaque strings. Keep that exact wire field name; do not add a +second `family_lease_id` alias to the exact receipt. The successful private +response envelope is exact `{schema, correlation_id, request_hash, receipt}` with schema `2718lab.storage.admission-response.v1`. A failure uses the existing bounded transport error path with a stable code, never a fabricated zero reservation. The internal transport-completion receipt separately records response hash, @@ -491,8 +528,6 @@ Push-Location 'G:\2718lab\_codex\.codex-task-temp\codex-host-mcp-fix-recovery2'; - Create: `codex-rs/core/src/fast_lane_host_dispatch/storage_firewall.rs` - Modify: `codex-rs/core/src/fast_lane_host_dispatch/storage_firewall_tests.rs` - Modify: `codex-rs/core/src/fast_lane_host_dispatch/mod.rs:1-49` -- Modify: `codex-rs/core/Cargo.toml:80-145` -- Modify if dependencies change: `codex-rs/Cargo.lock` and Host-root `MODULE.bazel.lock` - [ ] **Step 1: Add the Rust RED target-key vectors.** @@ -553,11 +588,17 @@ This permits scope-disjoint writers in the same wave instead of making the second same-key assignment permanently `NO_SAFE_WORK`. Map the family to `approved_root/generated/` using strict -child/reparse checks. Member scratch directories are separate Host-assigned -children; the Cargo cache remains one shared canonical target root. Do not -put task/member suffixes in `target_key` or duplicate the Cargo cache. Shared -cache writes require the supported Cargo locking contract; other outputs -must remain in disjoint member grants. Family ownership survives until the +child/reparse checks. Every admitted artifact kind has the common +`/cargo-target` child, including the existing `fastlane-task` intent; +`cargo_target_root()` supplies the worker's Cargo cache independently of its +primary artifact/output root. Scratch and non-Cargo outputs are private +`/members///{scratch,output}` children. +For `cargo-target`, the assigned primary root may be the common cache; never +send another artifact's output into that cache. Do not put task/member +suffixes in `target_key`, rewrite `fastlane-task` on the wire, or trust the +caller to select a Host-authorized job kind. Shared cache writes require the +supported Cargo locking contract; other outputs remain in disjoint member +grants. Family ownership survives until the last member's confirmed shutdown and successful postcheck; cloning a grant or replaying a request neither reserves nor releases capacity. Admission expiry is a deadline for starting/attaching a grant, not permission to release @@ -575,8 +616,9 @@ test seam, but production admission requires verified Host authority. - [ ] **Step 3: Implement platform capacity providers using the existing CLI doctor implementation as the reference.** -On Unix call `libc::statvfs`; on Windows call -`GetDiskFreeSpaceExW`; on unsupported platforms return +On Unix call the existing `libc::statvfs` binding; on Windows reuse the kernel's +existing `GetDiskFreeSpaceExW` FFI without adding a new crate dependency; +on unsupported platforms return `STORAGE_STAT_UNAVAILABLE`. The provider is injected as a trait in tests so tests do not query the real G drive. `StorageFirewall::new` must not create the approved root or target directory during construction or failed admission. @@ -599,84 +641,208 @@ do not create a second Cargo target or repeatedly rerun the suite. - [ ] **Step 5: Commit the host firewall.** ```powershell -Push-Location 'G:\2718lab\_codex\.codex-task-temp\codex-host-mcp-fix-recovery2'; git add codex-rs/rmcp-client/src/storage_intent.rs codex-rs/rmcp-client/src/lib.rs codex-rs/core/src/fast_lane_host_dispatch/storage_firewall.rs codex-rs/core/src/fast_lane_host_dispatch/storage_firewall_tests.rs codex-rs/core/src/fast_lane_host_dispatch/mod.rs codex-rs/core/Cargo.toml codex-rs/Cargo.lock MODULE.bazel.lock; git commit -m 'feat: enforce deterministic storage admission'; Pop-Location +Push-Location 'G:\2718lab\_codex\.codex-task-temp\codex-host-mcp-fix-recovery2'; git add codex-rs/rmcp-client/src/storage_intent.rs codex-rs/rmcp-client/src/lib.rs codex-rs/core/src/fast_lane_host_dispatch/storage_firewall.rs codex-rs/core/src/fast_lane_host_dispatch/storage_firewall_tests.rs codex-rs/core/src/fast_lane_host_dispatch/mod.rs; git commit -m 'feat: enforce deterministic storage admission'; Pop-Location ``` ### Task 6: Connect admission to preparation, worker environment, and terminal release -**Files:** -- Modify: `codex-rs/core/src/fast_lane_host_dispatch/registry.rs:567-1668` -- Modify: `codex-rs/core/src/fast_lane_host_dispatch/coordinator.rs:393-580,1218-1260` -- Modify: `codex-rs/core/src/fast_lane_host_dispatch/codex_adapter.rs:1-1444` -- Modify: `codex-rs/core/src/fast_lane_host_dispatch/contract.rs` -- Modify: `codex-rs/core/src/fast_lane_host_dispatch/storage_profile.rs` -- Modify: `codex-rs/rmcp-client/src/inherited_host_bridge_protocol.rs` -- Modify: `codex-rs/rmcp-client/src/inherited_host_bridge_protocol/session.rs` -- Modify: `codex-rs/rmcp-client/src/inherited_host_bridge_protocol/pump.rs` -- Modify: `mcp-tools/devkit_runtime/fastlane_host_adapter.py` -- Modify: `mcp-tools/devkit_runtime/host_bridge.py` -- Modify: `mcp-tools/devkit_runtime/host_session.py` -- Modify: `mcp-tools/server.py:1008-1314` -- Test: `mcp-tools/tests/test_storage_firewall.py` -- Test: `codex-rs/core/src/fast_lane_host_dispatch/storage_firewall_tests.rs` - -- [ ] **Step 1: Add a RED production-path assertion that an admitted root is present only in host-owned worker facts.** - -```python -def test_fastlane_dispatch_forwards_receipt_identity_without_public_path(): - result = dispatch_fixture_with_storage_intent() - assert result["storage_admission"]["assigned_root_identity"].startswith("sha256:") - assert "assigned_root" not in result - assert "CARGO_TARGET_DIR" not in result -``` - -- [ ] **Step 2: Connect both authority sources to one reservation service and consume sealed grants exactly once.** +**Independent file groups:** All paths below are within the scope map above. + +| Slice | Files and existing attachment points | +| --- | --- | +| 6a configuration/service | `config/src/config_toml.rs::ConfigToml`, `core/src/config/mod.rs::Config::load_config_with_layer_stack`, generated `core/config.schema.json`; new `core/src/fast_lane_host_dispatch/storage_service.rs` and `mod.rs`; `app-server/src/message_processor.rs`, `core/src/thread_manager.rs::ThreadManagerState`, `core/src/session/mod.rs::SessionSpawnArgs`, `core/src/session/session.rs`, `core/src/state/service.rs::SessionServices` | +| 6b authority/write ordering | `core/src/fast_lane_host_dispatch/{registry,contract,storage_profile,worktree}.rs`, `core/src/mcp_tool_call.rs`; existing rmcp-client refill protocol/session/pump; DevKit `fastlane_host_adapter.py`, `host_bridge.py`, `host_session.py`, `server.py` | +| 6c worker isolation | `core/src/fast_lane_host_dispatch/codex_adapter.rs::{HostWriterContext,CodexHostDispatchFacts,prepare_batch}`, `core/src/config/mod.rs::Permissions`, `protocol/src/shell_environment.rs`, `core/src/unified_exec/process_manager.rs` | +| 6d real termination | `core/src/unified_exec/{mod,process,process_manager}.rs`, `utils/pty/src/{process,win/job}.rs`, `core/src/session/handlers.rs`, `core/src/agent/control/legacy.rs`; reuse `exec-server/src/process.rs` event boundary | +| 6e postcheck/settlement | new `core/src/fast_lane_host_dispatch/storage_postcheck.rs`, `mod.rs`, existing `codex_adapter.rs` terminal/recovery methods and `coordinator.rs` observers; existing storage firewall tests | + +Host paths in this table are relative to `codex-rs`; DevKit runtime filenames +are under `mcp-tools/devkit_runtime`, with `server.py` under `mcp-tools`. +6a can compile independently with storage disabled. 6d can be implemented +independently of admission. 6b depends on 6a and the preserved Task 4/5 +contracts, plus the bounded control-allocation prerequisite below for durable +refill registration; 6c depends on 6b. Successful release in 6e requires both phases of +6d, not just worker wiring or a green compile. Coordinate shared files rather +than concurrently editing registry/adapter/process-manager from two slices. + +- [ ] **Step 6a: Load explicit trusted configuration and inject one runtime-owned service.** + +Add an optional `storage_firewall` block to `ConfigToml`, containing exactly +nine required fields: `approved_root`, `task_byte_limit`, `task_file_limit`, +`target_family_byte_limit`, `target_family_file_limit`, +`global_reserved_byte_limit`, `global_reserved_file_limit`, +`free_space_floor_bytes`, and `emergency_floor_bytes`. The root must be an +explicit absolute Host-approved directory; each numeric value must pass the +kernel's positive/checked policy validation, including emergency <= floor. +Do not supply numerical defaults, derive authority from free disk space, or +create an approved root while loading configuration. Missing block disables +storage admission with `STORAGE_POLICY_MISSING`; malformed blocks are rejected. + +Use the existing `ConfigLayerSource` provenance, not merged values alone. +System/enterprise-managed/operator user configuration may supply the block; +workspace `Project` configuration cannot set or override it. Treat session +flags as authority only when the Host startup path explicitly validates an +operator override, never when copied from worker/caller facts. Reject an +untrusted storage override instead of silently merging individual fields. +Generate `core/config.schema.json` through the existing +`config/src/schema.rs::write_config_schema` path. + +Construct one service at the process-scoped Host runtime's ThreadManager +creation in `app-server/src/message_processor.rs`; store the same Arc in +`ThreadManagerState`, forward through `SessionSpawnArgs`/`SessionServices`, +and give every `FastLaneHostFactsRegistry` a reference to it. The new +`storage_service.rs` facade owns the single kernel accounting state and +immutable resolved policy/root, not another independent reservation ledger. +Adapt every constructor/call site; test-only construction may be explicitly +disabled or use an injected fixture, never a permissive production default. +Current `session/session.rs` creates registries per Session: keep that facts +scope but do not create a firewall there. Per-thread config reload must not +reset global reservations or replace policy/root while grants are retained. + +This is cross-session sharing inside one Host runtime, not a cross-process +ledger. An independent Host process must not claim the same active storage +root without exclusive root authority; multi-process persistence/recovery +remains Plan 2. An in-memory Arc cannot prove that exclusivity by itself. + +- [ ] **Step 6b: Admit/seal before any task-root write, then consume once.** Initial `claim_storage_admission` resolves the Task 4 `Sent` profile and calls -`reserve_member_once` for the Host's original runtime batch owner. Store each -decision, `StorageAssignmentBinding`, and private grant in registry/runtime -state. `consume_batch` validates the original batch and exact member coverage, -seals the group, and transfers/attaches those grants to `HostWriterContext`. -It does not regenerate route/lease facts, rekey batch indexes, or reserve again. - -Successors are materialized natively by `consume_refill_queue`, not by a new -Python prepare call. Extend the exact refill codec/ledger with explicit -per-task budgets and bind them into the queue hash. `register_refill_queue` -validates coverage against its existing verified remaining skeletons and -index refs; it must not allocate capacity for all remaining work. After the -selected wave passes dependency/scope gates, resolve and revalidate its Host -profile provenance, construct intent and `StorageAssignmentBinding`, and call -the same `reserve_member_once` service with native selected-wave authority. -Do not manufacture bridge requests, correlations, nonces, or transport -receipts for this internal path. Attach sealed grants before the selected -wave can prepare. Failed admission does not advance the queue cursor; an old -queue without required budget/provenance is not silently upgraded or admitted. - -`CodexHostDispatchAdapter::prepare_batch` must require the already-sealed -grants before `worktree_broker.reserve_batch`, and must never call admission -again. At worker config construction, inject the private common Cargo target -and per-member scratch into `config.permissions.shell_environment_policy.r#set` -as `CARGO_TARGET_DIR` and `CODEX_TASK_TEMP`. These paths stay in private -`HostWriterContext`, not the caller's receipt or bounded public context. - -On prepare/dispatch failure, revoke unused member grants; after a writer has -started, terminal/recovery/integration paths must first confirm child shutdown -and perform postcheck. Release each member once and the family lease only -after the last member succeeds. A clone, timeout, or terminal ACK alone is -not release evidence. Postcheck failure returns `STORAGE_POSTCHECK_FAILED` -and retains ownership for later recovery; it never triggers a broad delete. - -- [ ] **Step 3: Make the DevKit Fast Lane result expose only stable storage code and receipt identity.** - -```python -public = { - "storage_code": receipt.code, - "storage_receipt_hash": receipt.receipt_hash, - "target_key": receipt.target_key, -} -``` - -- [ ] **Step 4: Run the one production-path probe plus compile-first gates.** +`reserve_member_once` for the original Host runtime batch. Each task retains +its own profile/attestation hash; group authority comes from verified original +batch/preparation/call/generation, not equality of per-task profile hashes. +Keep each decision, `StorageAssignmentBinding`, and grant in private Host +state. Seal against the original exact task set before filesystem writes. +`consume_batch` revalidates that sealed set and consumes/transfers the grants +before setting `BatchConsumed`, removing `by_batch`, or activating scope +leases. On failure preserve a recoverable consistent state. It neither +rewrites original route/lease hashes nor reserves a second time. + +`mcp_tool_call.rs` currently creates a hardcoded per-session/call task root +before constructing `GitWorktreeBroker`; remove this pre-admission mkdir. +Split `GitWorktreeBroker::new`'s current existing-root canonicalization into +planned-root validation against the approved existing parent and later +materialization with identity/reparse revalidation. Materialize only after +successful admission/seal and before the first durable queue write or +`reserve_batch`; adapter preparation revalidates the materialized binding. + +Python currently registers the refill queue before dispatch, while Host +`refill_authority_root_binding` canonicalizes an already-existing task root. +Move initial admission/seal before queue registration; delaying mkdir only +until `prepare_batch` is insufficient. Put queue metadata, atomic-replacement +temporaries, and worktree roots under explicit accounted ownership, but do +not place durable queue records in a member's releasable scratch/output. +Their lifecycle spans waves: releasing/cleaning the initial member must not +erase an active queue, and retaining its Cargo family lease for the queue +would permanently block a different-owner same-key successor. + +Durable refill registration therefore depends on an independently bounded +Host ledger/control allocation with a queue-lifetime owner, explicit positive +byte/file allowance, safe private root, and terminal queue settlement. Plan 2 +must provide that control-allocation contract before this part of 6b is +enabled; implement no ledger or new artifact/wire field in this Plan 1 update. +Until it exists, reject durable registration before any write. Do not invent +a control task/lease, take an unrequested budget, or exempt metadata from +accounting. Its owner must not hold the shared Cargo family lease; verify +that initial member release preserves the queue while same-key successors +can acquire their own group lease. This prerequisite changes the execution +order, not the frozen admission-v1 or target-v1 contract. + +Successors remain native to `consume_refill_queue`. Extend the exact refill +codec and durable snapshot with task-keyed remaining budgets covered by the +queue hash; verify exact remaining-task coverage and existing index refs at +registration. Allocate only the selected wave after dependency/scope gates. +Use its verified native authority to build profile provenance, intent, and +binding and invoke the same service; no synthetic bridge request/correlation +or fabricated transport receipt. Seal/attach before prepare, and commit queue +cursor/budget consumption consistently with the attempt outcome. Admission +failure does not advance the queue. Old snapshots without required budget +provenance fail closed, not silent upgrade or all-backlog initial reservation. + +- [ ] **Step 6c: Attach grants to workers and enforce reserved paths at final environment/permission assembly.** + +`CodexHostDispatchAdapter::prepare_batch` checks sealed grants before +`worktree_broker.reserve_batch`; it never invokes admission again. Carry +private grant/binding state in `HostWriterContext`/`CodexHostDispatchFacts`. +Use `cargo_target_root()` for `CARGO_TARGET_DIR` for every artifact kind, +including `fastlane-task`, and the private scratch getter for +`CODEX_TASK_TEMP`; non-Cargo outputs use the member output root. No path is +put into the decision, public bounded context, or DevKit result. + +Setting `config.permissions.shell_environment_policy.r#set` alone is not +sufficient: `include_only` is applied afterward and per-call overrides can +replace values. Preserve/reject overrides of reserved values at final +storage-managed environment assembly. Update actual `Permissions` using its +workspace-root/effective-profile APIs as well as `Config.workspace_roots`, +granting only the member paths and shared cache while respecting managed +denies. Do not make all of approved_root writable. Environment variables alone +do not confine arbitrary shell writes; the sandbox and descendant fence must +cover those writes, or the managed-storage execution path remains unavailable. + +- [ ] **Step 6d.1: Retain process handles and await real exit, without minting terminal proof yet.** + +`ProcessStore` currently has no wait operation. Its +`terminate_all_processes` drains entries then issues non-confirming terminate +calls; an empty map is not shutdown proof. `UnifiedExecProcess::terminate_confirmed` +also synthesizes `signal_exit` after termination without waiting for a real +local exit. Replace this use with an asynchronous confirmed boundary that +closes new process admission, retains ownership/handles, propagates kill +errors, and waits for the existing local `SpawnedPty.exit_rx`/wait task or +ExecServer `Exited` event. Distinguish actual exit from closed channels, +unknown exit, timeout, and synthetic status. Preserve retained grants and +recovery handles on failure; never turn an error into success by draining. + +`utils/pty::ProcessHandle::terminate` currently ignores killer errors and +drops its wait handle; extend that boundary so the caller can await actual +termination. This phase establishes managed root-process exit only, not proof +that all descendants have stopped writing. + +- [ ] **Step 6d.2: Fence owned descendants and shutdown-time writers before issuing TerminalProof.** + +`shutdown_agent_tree` waits for agent threads, not an OS process tree. +For storage-managed processes enforce retained, non-escaping process-family +ownership; on Windows the JobObject path must not use `preserve_descendants` +and must confirm no active owned processes remain after termination. Use an +equivalent owned process-family fence on supported platforms; unsupported or +unconfirmable cases retain ownership/fail closed. Do not scan/kill unrelated +processes or equate root PID exit with descendant exit. + +Cover code-mode, MCP, prewarm, and hooks that can write into grants. +`shutdown_session_runtime` currently runs session-end hooks after terminating +unified exec: run all permitted shutdown work before the final write fence, +or explicitly deny those paths once fenced. Prevent post-proof process or +writer creation. Bind the Host-created terminal evidence to the actual +member/group/generation and owned process set. Neither assistant `Completed`, +terminal ACK, nor `ShutdownComplete` establishes this evidence. + +A never-started cancellation is allowed only when Host lifecycle state proves +no worker/process was ever created. A prepared endpoint with no assistant +input can already have prewarm/hooks; it requires the same termination fence. +Failed prepare/recovery must leave storage ownership in the shared service +even if adapter runtime objects are removed. Until both 6d phases hold, do not +wire a synthetic TerminalProof just to enable successful release. + +- [ ] **Step 6e: Scan real family contents after the fence and settle exactly once.** + +`WriterIntegrationCandidate::collect_terminal` proves Git/worktree state, not +disk usage. Implement the new `storage_postcheck.rs` collector over private +grant paths: bounded no-follow/reparse-safe traversal, checked byte/file +sums, and path identity revalidation. Inaccessible, unstable, or unbounded +observations return `STORAGE_POSTCHECK_FAILED` and retain ownership. Never +accept caller counters or delete data to make a postcheck pass. + +Attach member terminal handling at adapter +`shutdown_and_validate_terminal_success` and its recovery paths, with +coordinator observers retaining the group barrier. A fenced member's private +scratch/output can be checked independently; shared cache/family counts +require every active member in that family to be quiet. Count shared files +once, retain family observed usage across released tombstones, and release +the family lease only after the last required postcheck succeeds. Scope-lease +release and terminal ACK are not substitutes for storage settlement. + +- [ ] **Step 6f: Compile changed slices first, then keep only bounded acceptance probes.** + +Retain the current Task 4/5 compile evidence rather than rerunning unchanged +slices. For newly changed wiring, run from DevKit `mcp-tools`: ```powershell python -m py_compile server.py devkit_runtime/host_bridge.py devkit_runtime/host_session.py @@ -688,18 +854,23 @@ Pop-Location $env:PYTEST_DISABLE_PLUGIN_AUTOLOAD='1'; python -m pytest tests/test_storage_firewall.py::test_fastlane_dispatch_forwards_receipt_identity_without_public_path -q -o cache_dir=G:\2718lab\_codex\.codex-task-temp\storage-113-firewall-pytest ``` -Expected: Python probe passes, `py_compile` is silent, and `cargo check` -finishes successfully with no new warning. Stop before probes if compilation -fails. Keep one bounded Host successor/shared-group regression; Python-only -initial evidence is not full production acceptance. Do not run the full -workspace suite or repeatedly rebuild unchanged slices. +Also compile changed app-server/PTY/protocol callers in the relevant slice, +using the same existing target, `CARGO_INCREMENTAL=0`, and `--locked -j1`. +The two-crate check alone does not cover an edited app-server constructor. +Stop before probes on compile failure. Preserve one selected public-path +non-disclosure probe, one native-successor/shared-group regression, and one +bounded termination-failure/retained-ownership probe. Do not run the full +workspace suite, regenerate old RED evidence, or repeatedly rebuild unchanged +slices. Python-only initial evidence is not full production acceptance. -- [ ] **Step 5: Commit the production wiring.** +- [ ] **Step 6g: Commit each compiled file group with explicit remaining gates.** -```powershell -Push-Location 'G:\2718lab\_codex\.codex-task-temp\devkit-1.1.2-recovery'; git add mcp-tools/server.py mcp-tools/devkit_runtime/fastlane_host_adapter.py mcp-tools/devkit_runtime/host_bridge.py mcp-tools/devkit_runtime/host_session.py mcp-tools/tests/test_storage_firewall.py; git commit -m 'feat: bind admitted storage roots to fast lane workers'; Pop-Location -Push-Location 'G:\2718lab\_codex\.codex-task-temp\codex-host-mcp-fix-recovery2'; git add codex-rs/core/src/fast_lane_host_dispatch/registry.rs codex-rs/core/src/fast_lane_host_dispatch/coordinator.rs codex-rs/core/src/fast_lane_host_dispatch/codex_adapter.rs codex-rs/core/src/fast_lane_host_dispatch/contract.rs codex-rs/core/src/fast_lane_host_dispatch/storage_profile.rs codex-rs/core/src/fast_lane_host_dispatch/storage_firewall_tests.rs codex-rs/rmcp-client/src/inherited_host_bridge_protocol.rs codex-rs/rmcp-client/src/inherited_host_bridge_protocol/session.rs codex-rs/rmcp-client/src/inherited_host_bridge_protocol/pump.rs; git commit -m 'feat: bind admitted storage roots to fast lane workers'; Pop-Location -``` +Stage only that slice's named files after `git diff --check`; do not use a +broad `git add` or include another worker's dirty adapter/registry changes. +Record the compile scope and unverified process/platform/config conditions +with each handoff. A disabled 6a service or root-process-only 6d.1 can be a +compiled intermediate slice, but cannot be reported as Task 6 storage release +or complete Plan 1 acceptance. ## Plan 1 acceptance gate and handoff @@ -708,4 +879,8 @@ Push-Location 'G:\2718lab\_codex\.codex-task-temp\codex-host-mcp-fix-recovery2'; - [ ] Run `python -m py_compile` on changed DevKit Python files and `cargo check -p codex-rmcp-client -p codex-core --lib --locked -j1` with the existing Host `codex-rs\target` and `CARGO_INCREMENTAL=0`; retain current compile evidence instead of rerunning unchanged slices. - [ ] Record one controlled admission receipt proving same semantics reuse one target key and one changed semantic forks it; record one low-space/policy-failure receipt proving no directory was created. - [ ] Verify exact5 session lookup/core `Sent` binding, unchanged original batch hashes, same-owner member sharing with exact sealing, cross-owner conflict, and a native selected-successor admission without a second reservation. Do not label initial-only wiring complete. +- [ ] Verify explicit trusted root plus eight policy values, one Host-runtime service shared across Sessions, and no pre-admission task/control/queue directory writes. Record the single-runtime versus independent-Host-process ownership boundary. +- [ ] Before enabling durable refill, obtain Plan 2's bounded queue-lifetime control allocation: initial member release cannot remove active metadata, and that allocation cannot retain the Cargo family lease or block same-key successors. Missing allocation rejects registration before writes; no free budget or fabricated task is allowed. +- [ ] Verify actual process and descendant shutdown evidence, denial of post-proof writers, and a real all-members-quiet family scan before release. Failure/timeout retains ownership; `Completed`, `ShutdownComplete`, ACK, or an empty ProcessStore is not sufficient. +- [ ] Obtain the operator's approved absolute root and eight policy values before production enablement; do not invent them or reuse example test capacities as configuration. Record any unsupported process-containment platform as a remaining activation gate. - [ ] Do not implement ledger persistence, preview/apply, source deletion, session deletion, compression, or remote synchronization in this plan. Plan 2 consumes `StorageAdmissionReceipt`; Plan 3 consumes the released/observed storage records. From 7493d164de3b858a0ba3b065f5b3b1a9de8e6f42 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E5=93=80=E6=B4=9B=E8=8A=99?= Date: Sun, 30 Aug 2026 06:05:08 +0800 Subject: [PATCH 22/39] fix: bound storage admission transport and cancel before session close --- mcp-tools/devkit_runtime/host_bridge.py | 242 ++++++++++++++++++++--- mcp-tools/devkit_runtime/host_session.py | 5 + mcp-tools/tests/test_storage_firewall.py | 71 +++++++ 3 files changed, 295 insertions(+), 23 deletions(-) diff --git a/mcp-tools/devkit_runtime/host_bridge.py b/mcp-tools/devkit_runtime/host_bridge.py index c9d8fe3..ecb62df 100644 --- a/mcp-tools/devkit_runtime/host_bridge.py +++ b/mcp-tools/devkit_runtime/host_bridge.py @@ -20,7 +20,8 @@ import stat import struct import time -from collections.abc import Callable, Mapping, Sequence +from collections.abc import Callable, Iterator, Mapping, Sequence +from contextlib import contextmanager from dataclasses import dataclass, field from threading import Event, Lock, RLock from typing import Final, cast @@ -640,7 +641,7 @@ def accept_from_file_descriptors( @property def is_available(self) -> bool: - return not self._closed + return not self._closed and not self._cancel_event.is_set() def prepare_capability( self, @@ -1627,7 +1628,7 @@ def request_storage_admission( The owning HostSession serializes this round trip with preparation and terminal-reader startup. The existing bridge I/O lock owns both frames. """ - with self._io_lock: + with self._storage_admission_io(now=now, expires_at=expires_at) as deadline: normalized = _normalize_storage_admission_request(request) correlation = cast(str, normalized["correlation_id"]) attestation = cast(str, normalized["profile_attestation_hash"]) @@ -1650,9 +1651,10 @@ def request_storage_admission( self._storage_admission_correlations.add(correlation) try: self._send_validated_private( - kind="storage_admission_request", action_id=correlation, payload=normalized + kind="storage_admission_request", action_id=correlation, payload=normalized, + deadline=deadline, ) - message = self._receive_private() + message = self._receive_private(deadline=deadline) if message.kind != "storage_admission_response" or message.action_id != correlation: raise HostBridgeError("HOST_BRIDGE_STORAGE_ADMISSION_INVALID") completed_time = clock() @@ -1680,6 +1682,27 @@ def request_storage_admission( ) return receipt + @contextmanager + def _storage_admission_io(self, *, now: int, expires_at: int) -> Iterator[float]: + if ( + type(now) is not int or type(expires_at) is not int + or not 0 <= now < expires_at <= (1 << 64) - 1 + ): + raise HostBridgeError("HOST_BRIDGE_STORAGE_ADMISSION_INVALID") + # One monotonic deadline covers lock acquisition, bootstrap/request + # writes, reply length prefix, and reply payload. Partial progress must + # never restart the authority-derived transport budget. + deadline = time.monotonic() + (expires_at - now) + while not self._io_lock.acquire( + timeout=min(0.1, _remaining_io_time(deadline, self._cancel_event)) + ): + pass + try: + self._ensure_open() + yield deadline + finally: + self._io_lock.release() + def has_completed_storage_profile(self, profile: Mapping[str, object]) -> bool: """Lookup transport-completed facts; caller-supplied hashes cannot enroll.""" attestation = profile.get("attestation_hash") @@ -1903,16 +1926,18 @@ def send_private( self._send_private(kind=kind, action_id=action_id, payload=payload) def _send_validated_private( - self, *, kind: str, action_id: str, payload: Mapping[str, object] + self, *, kind: str, action_id: str, payload: Mapping[str, object], + deadline: float | None = None, ) -> None: """Write a packet only after its typed private validator has succeeded.""" if kind not in _VALIDATED_PRIVATE_KINDS: raise HostBridgeError("HOST_BRIDGE_FRAME_INVALID") - self._send_private(kind=kind, action_id=action_id, payload=payload) + self._send_private(kind=kind, action_id=action_id, payload=payload, deadline=deadline) def _send_private( - self, *, kind: str, action_id: str, payload: Mapping[str, object] + self, *, kind: str, action_id: str, payload: Mapping[str, object], + deadline: float | None = None, ) -> None: with self._io_lock: if kind not in _MESSAGE_KINDS or kind == "session_open": @@ -1941,9 +1966,9 @@ def _send_private( self._poison() raise HostBridgeError("HOST_BRIDGE_FRAME_INVALID") from None if bootstrap is not None: - self._write_complete(bootstrap) + self._write_complete(bootstrap, deadline=deadline) self._bootstrap_sent = True - self._write_complete(private_frame) + self._write_complete(private_frame, deadline=deadline) self._next_out += 1 def receive(self) -> PrivateHostMessage: @@ -1955,7 +1980,7 @@ def receive(self) -> PrivateHostMessage: raise HostBridgeError("HOST_BRIDGE_FRAME_INVALID") return message - def _receive_private(self) -> PrivateHostMessage: + def _receive_private(self, *, deadline: float | None = None) -> PrivateHostMessage: """Read one framed message for a typed private validator.""" with self._io_lock: @@ -1966,6 +1991,7 @@ def _receive_private(self) -> PrivateHostMessage: self._read_fd, cancel_event=self._cancel_event, cancel_fd=self._cancel_read_fd, + deadline=deadline, ) ) self._verify_frame(frame, expected_sequence=self._next_in) @@ -1991,6 +2017,20 @@ def _receive_private(self) -> PrivateHostMessage: payload=payload, ) + def cancel_read(self) -> None: + """Cancel blocked I/O without waiting for its lock or closing its fd.""" + with self._close_lock: + self._signal_cancel_locked() + + def _signal_cancel_locked(self) -> None: + if self._cancel_event.is_set(): + return + self._cancel_event.set() + try: + os.write(self._cancel_write_fd, b"\x00") + except OSError: + pass + def close(self) -> None: """Close only the descriptor(s) this bridge owns.""" @@ -1998,11 +2038,7 @@ def close(self) -> None: if self._closed: return self._closed = True - self._cancel_event.set() - try: - os.write(self._cancel_write_fd, b"\\x00") - except OSError: - pass + self._signal_cancel_locked() cancel_descriptors = { self._cancel_read_fd, self._cancel_write_fd, @@ -2094,21 +2130,48 @@ def _read_raw_frame( *, cancel_event: Event | None = None, cancel_fd: int | None = None, + deadline: float | None = None, ) -> bytes: header = _read_exact( - descriptor, 4, cancel_event=cancel_event, cancel_fd=cancel_fd + descriptor, 4, cancel_event=cancel_event, cancel_fd=cancel_fd, + deadline=deadline, ) size = struct.unpack("!I", header)[0] if size == 0 or size > _MAX_FRAME_BYTES: raise HostBridgeError("HOST_BRIDGE_FRAME_INVALID") return _read_exact( - descriptor, size, cancel_event=cancel_event, cancel_fd=cancel_fd + descriptor, size, cancel_event=cancel_event, cancel_fd=cancel_fd, + deadline=deadline, ) - def _write_complete(self, payload: bytes) -> None: + def _write_complete(self, payload: bytes, *, deadline: float | None = None) -> None: try: + if deadline is not None: + with _nonblocking_pipe_writer(self._write_fd) as write: + offset = 0 + while offset < len(payload): + remaining = _remaining_io_time(deadline, self._cancel_event) + if os.name != "nt": + readable, writable, _ = select.select( + [self._cancel_read_fd], [self._write_fd], [], min(0.1, remaining) + ) + if readable: + raise HostBridgeError("HOST_BRIDGE_UNAVAILABLE") + if not writable: + continue + try: + written = write(payload[offset:]) + except BlockingIOError: + written = 0 + if written < 0 or written > len(payload) - offset: + raise HostBridgeError("HOST_BRIDGE_UNAVAILABLE") + offset += written + if written == 0: + self._cancel_event.wait(min(0.05, _remaining_io_time(deadline, self._cancel_event))) + _remaining_io_time(deadline, self._cancel_event) + return written = os.write(self._write_fd, payload) - except OSError as error: + except (HostBridgeError, OSError, ValueError) as error: self._poison() raise HostBridgeError("HOST_BRIDGE_UNAVAILABLE") from error if written != len(payload): @@ -2172,7 +2235,7 @@ def _prune_terminal_operation_tombstones(self, *, now: int) -> None: del self._terminal_operation_tombstones[key] def _ensure_open(self) -> None: - if self._closed or self._read_fd < 0 or self._write_fd < 0: + if self._closed or self._cancel_event.is_set() or self._read_fd < 0 or self._write_fd < 0: raise HostBridgeError("HOST_BRIDGE_UNAVAILABLE") @@ -3996,17 +4059,127 @@ class _IoStatusBlock(ctypes.Structure): return access_mask.value +def _remaining_io_time(deadline: float, cancel_event: Event | None) -> float: + if cancel_event is not None and cancel_event.is_set(): + raise HostBridgeError("HOST_BRIDGE_UNAVAILABLE") + if type(deadline) not in (int, float) or not math.isfinite(deadline): + raise HostBridgeError("HOST_BRIDGE_UNAVAILABLE") + remaining = deadline - time.monotonic() + if remaining <= 0: + raise HostBridgeError("HOST_BRIDGE_UNAVAILABLE") + return remaining + + +@contextmanager +def _nonblocking_pipe_writer(descriptor: int) -> Iterator[Callable[[bytes], int]]: + """Temporarily use synchronous nonblocking writes; never spawn an I/O worker. + + Windows PIPE_NOWAIT supports our synchronous named and anonymous handles. + Query/verify/restore the exact original mode; unsupported handles fail + closed, never fall back to a blocking WriteFile. Partial/zero writes are + handled by the caller's single absolute deadline loop. + """ + if os.name != "nt": + previous = os.get_blocking(descriptor) + os.set_blocking(descriptor, False) + try: + yield lambda payload: os.write(descriptor, payload) + finally: + os.set_blocking(descriptor, previous) + return + + import ctypes + import msvcrt + + kernel32 = ctypes.WinDLL("kernel32", use_last_error=True) + handle = msvcrt.get_osfhandle(descriptor) + get_state = kernel32.GetNamedPipeHandleStateW + get_state.argtypes = ( + ctypes.c_void_p, ctypes.POINTER(ctypes.c_ulong), ctypes.c_void_p, + ctypes.c_void_p, ctypes.c_void_p, ctypes.c_void_p, ctypes.c_ulong, + ) + get_state.restype = ctypes.c_int + set_state = kernel32.SetNamedPipeHandleState + set_state.argtypes = ( + ctypes.c_void_p, ctypes.POINTER(ctypes.c_ulong), ctypes.c_void_p, ctypes.c_void_p, + ) + set_state.restype = ctypes.c_int + write_file = kernel32.WriteFile + write_file.argtypes = ( + ctypes.c_void_p, ctypes.c_void_p, ctypes.c_ulong, + ctypes.POINTER(ctypes.c_ulong), ctypes.c_void_p, + ) + write_file.restype = ctypes.c_int + + def read_mode() -> int: + state = ctypes.c_ulong() + if not get_state(handle, ctypes.byref(state), None, None, None, None, 0): + raise HostBridgeError("HOST_BRIDGE_UNAVAILABLE") + if state.value & ~3: # only PIPE_NOWAIT | PIPE_READMODE_MESSAGE + raise HostBridgeError("HOST_BRIDGE_UNAVAILABLE") + return state.value + + def set_mode(mode: int) -> None: + state = ctypes.c_ulong(mode) + if not set_state(handle, ctypes.byref(state), None, None): + raise HostBridgeError("HOST_BRIDGE_UNAVAILABLE") + if read_mode() != mode: + raise HostBridgeError("HOST_BRIDGE_UNAVAILABLE") + + def write(payload: bytes) -> int: + buffer = ctypes.create_string_buffer(payload) + written = ctypes.c_ulong() + if not write_file(handle, buffer, len(payload), ctypes.byref(written), None): + raise HostBridgeError("HOST_BRIDGE_UNAVAILABLE") + return written.value + + original = read_mode() + try: + set_mode(original | 1) # PIPE_NOWAIT; retain the original read mode + yield write + finally: + set_mode(original) + + def _read_exact( descriptor: int, size: int, *, cancel_event: Event | None = None, cancel_fd: int | None = None, + deadline: float | None = None, ) -> bytes: chunks: list[bytes] = [] remaining = size while remaining: - if cancel_event is not None: + read_size = remaining + if deadline is not None: + timeout = min(0.1, _remaining_io_time(deadline, cancel_event)) + if os.name == "nt": + # Only this receiver can consume these bytes. Never request + # more than PeekNamedPipe reported, or read an unknown handle. + available = _windows_pipe_available_bytes(descriptor) + if not available: + if cancel_event is not None: + cancel_event.wait(min(0.05, timeout)) + else: + time.sleep(min(0.05, timeout)) + continue + read_size = min(remaining, available) + else: + wait_fds = [descriptor] + if cancel_fd is not None and cancel_fd >= 0: + wait_fds.append(cancel_fd) + try: + readable, _, _ = select.select(wait_fds, [], [], timeout) + except (OSError, ValueError) as error: + raise HostBridgeError("HOST_BRIDGE_UNAVAILABLE") from error + if cancel_fd is not None and cancel_fd in readable: + raise HostBridgeError("HOST_BRIDGE_UNAVAILABLE") + if descriptor not in readable: + continue + _remaining_io_time(deadline, cancel_event) + elif cancel_event is not None: if cancel_event.is_set(): raise HostBridgeError("HOST_BRIDGE_UNAVAILABLE") # POSIX pipes can be waited on together with the private wake fd. @@ -4031,16 +4204,39 @@ def _read_exact( time.sleep(0.05) continue try: - chunk = os.read(descriptor, remaining) + chunk = os.read(descriptor, read_size) except OSError as error: raise HostBridgeError("HOST_BRIDGE_UNAVAILABLE") from error if not chunk: raise HostBridgeError("HOST_BRIDGE_UNAVAILABLE") chunks.append(chunk) remaining -= len(chunk) + if deadline is not None: + _remaining_io_time(deadline, cancel_event) return b"".join(chunks) +def _windows_pipe_available_bytes(descriptor: int) -> int: + """A bounded-read prerequisite; unsupported/closed handles fail closed.""" + try: + import ctypes + import msvcrt + + kernel32 = ctypes.WinDLL("kernel32", use_last_error=True) + peek = kernel32.PeekNamedPipe + peek.argtypes = ( + ctypes.c_void_p, ctypes.c_void_p, ctypes.c_ulong, + ctypes.c_void_p, ctypes.POINTER(ctypes.c_ulong), ctypes.c_void_p, + ) + peek.restype = ctypes.c_int + available = ctypes.c_ulong() + if not peek(msvcrt.get_osfhandle(descriptor), None, 0, None, ctypes.byref(available), None): + raise HostBridgeError("HOST_BRIDGE_UNAVAILABLE") + return available.value + except (AttributeError, ImportError, OSError, OverflowError, ValueError) as error: + raise HostBridgeError("HOST_BRIDGE_UNAVAILABLE") from error + + def _windows_pipe_readable(descriptor: int) -> bool | None: """Return pipe readiness on Windows, or None when the handle is unknown.""" diff --git a/mcp-tools/devkit_runtime/host_session.py b/mcp-tools/devkit_runtime/host_session.py index 17250c6..b231ba7 100644 --- a/mcp-tools/devkit_runtime/host_session.py +++ b/mcp-tools/devkit_runtime/host_session.py @@ -523,6 +523,11 @@ def resolve_routing_attestations( def close(self) -> None: """Close the owned private transport once; no session can be revived.""" + # Admission owns the business lock across its single-reader round trip. + # Wake it first without closing/reusing descriptors while it restores + # transport mode; only close fds after that owner releases this lock. + if self._bridge is not None: + self._bridge.cancel_read() with self._compiler_evidence_lock: if self._closed: return diff --git a/mcp-tools/tests/test_storage_firewall.py b/mcp-tools/tests/test_storage_firewall.py index 4cba2b1..2b14898 100644 --- a/mcp-tools/tests/test_storage_firewall.py +++ b/mcp-tools/tests/test_storage_firewall.py @@ -7,6 +7,7 @@ import os import sys import threading +import time from pathlib import Path import pytest @@ -581,6 +582,76 @@ def test_storage_admission_rejects_substitution_and_unknown_fields() -> None: host.close() +def test_storage_admission_deadline_and_close_cancel_blocked_io() -> None: + from devkit_runtime import host_bridge, host_session + from devkit_runtime.storage_intent import parse_storage_intent + + # Real pipe I/O with a fixed trusted clock: the transport deadline must + # still advance monotonically. Peer never returns an admission response. + for scenario in ("no_reply", "full_pipe", "close"): + child, host = _pipe_pair() + profile_request = _storage_profile_request() + pending = child.send_storage_profile_request(**{ + name: getattr(profile_request, name) for name in ( + "call_intent_hash", "preparation_id", "task_id", + "source_plan_hash", "index_attestation_hash", + ) + }) + peer_request = host.receive_storage_profile_request() + host.send_storage_profile_response(request=peer_request, response=_storage_profile_response(peer_request)) + profile = child.receive_storage_profile_response(request=pending) + session = host_session.HostSession(bridge=child, clock=lambda: 1_700_000_000) + # This test isolates transport lifecycle after actual profile delivery; + # preparation enrollment itself is covered by the round-trip test. + session._completed_storage_profiles[_hash("7")] = host_session._CompletedStorageProfile( + profile=profile, bridge=child, + expires_at=1_700_000_060 if scenario == "close" else 1_700_000_001, + requested_bytes=4096, requested_files=8, + ) + intent = parse_storage_intent(_storage_intent( + task_id="TASK-V5", plan_binding=_hash("8"), context_hash=_hash("6") + )) + if scenario == "full_pipe": + with host_bridge._nonblocking_pipe_writer(child._write_fd) as write: + for _ in range(1024): + try: + if write(b"x" * 4096) == 0: + break + except BlockingIOError: + break + else: + raise AssertionError("fixture pipe did not reach bounded capacity") + result: list[object] = [] + worker = threading.Thread(target=lambda: result.append(session.request_storage_admission( + intent, profile_attestation_hash=_hash("7") + )), daemon=True) + closer: threading.Thread | None = None + worker.start() + try: + if scenario != "full_pipe": + message = host._receive_private(deadline=time.monotonic() + 2) + assert message.kind == "storage_admission_request" + if scenario == "close": + closer = threading.Thread(target=session.close, daemon=True) + closer.start() + closer.join(timeout=2) + assert not closer.is_alive(), "close waited behind admission's business lock" + worker.join(timeout=3) + assert not worker.is_alive(), "admission did not terminate at its transport deadline" + assert result == ["STORAGE_STAT_UNAVAILABLE"] + assert not child._storage_admission_completions + assert not child._storage_admission_decisions + finally: + # Only test-owned descriptors/threads; ensure a failing regression + # cannot leave a blocked peer alive in the test process. + child.close() + host.close() + worker.join(timeout=2) + if closer is not None: + closer.join(timeout=2) + session.close() + + def test_profile_tamper_or_missing_field_fails_closed() -> None: from devkit_runtime import host_bridge from devkit_runtime.host_bridge import HostBridgeError From 725ffeb0bd1d73029a836b8af6c9b8ba6bf3769c Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E5=93=80=E6=B4=9B=E8=8A=99?= Date: Sun, 30 Aug 2026 06:10:07 +0800 Subject: [PATCH 23/39] fix: defer bridge descriptor close until active io cleanup --- mcp-tools/devkit_runtime/host_bridge.py | 75 ++++++++++++++++-------- mcp-tools/tests/test_storage_firewall.py | 38 +++++++++++- 2 files changed, 85 insertions(+), 28 deletions(-) diff --git a/mcp-tools/devkit_runtime/host_bridge.py b/mcp-tools/devkit_runtime/host_bridge.py index ecb62df..3de3b62 100644 --- a/mcp-tools/devkit_runtime/host_bridge.py +++ b/mcp-tools/devkit_runtime/host_bridge.py @@ -438,6 +438,8 @@ def __init__( raise HostBridgeError("HOST_BRIDGE_UNAVAILABLE") from error self._cancel_event = Event() self._close_lock = Lock() + self._active_io_count = 0 + self._descriptors_closed = False # A session has one framed sequence in each direction. Serialize all # bridge I/O so concurrent Fast Lane waves cannot interleave bytes or # consume one another's sequence slot. @@ -1983,7 +1985,7 @@ def receive(self) -> PrivateHostMessage: def _receive_private(self, *, deadline: float | None = None) -> PrivateHostMessage: """Read one framed message for a typed private validator.""" - with self._io_lock: + with self._io_lock, self._active_io(): self._ensure_open() try: frame = _decode_frame( @@ -2032,34 +2034,51 @@ def _signal_cancel_locked(self) -> None: pass def close(self) -> None: - """Close only the descriptor(s) this bridge owns.""" + """Cancel immediately; close owned fds after the last active I/O exits. + + This does not wait on the I/O lock, including when _poison calls us + reentrantly. A legacy blocking write may retain its fds until that OS + call returns; close returning is not evidence of kernel I/O shutdown. + """ with self._close_lock: - if self._closed: - return - self._closed = True - self._signal_cancel_locked() - cancel_descriptors = { - self._cancel_read_fd, - self._cancel_write_fd, - } - self._cancel_read_fd = -1 - self._cancel_write_fd = -1 - for descriptor in cancel_descriptors: - try: - os.close(descriptor) - except OSError: - pass - if not self._owns_descriptors: - return - descriptors = {self._read_fd, self._write_fd} + if not self._closed: + self._closed = True + self._signal_cancel_locked() + if self._active_io_count == 0: + self._close_descriptors_locked() + + @contextmanager + def _active_io(self) -> Iterator[None]: + # Enrollment and close's unavailable transition are atomic. No new + # operation may borrow an fd after close/cancel has begun. + with self._close_lock: + self._ensure_open() + self._active_io_count += 1 + try: + yield + finally: + with self._close_lock: + self._active_io_count -= 1 + if self._closed and self._active_io_count == 0: + self._close_descriptors_locked() + + def _close_descriptors_locked(self) -> None: + if self._descriptors_closed: + return + self._descriptors_closed = True + descriptors = {self._cancel_read_fd, self._cancel_write_fd} + self._cancel_read_fd = -1 + self._cancel_write_fd = -1 + if self._owns_descriptors: + descriptors.update((self._read_fd, self._write_fd)) self._read_fd = -1 self._write_fd = -1 - for descriptor in descriptors: - try: - os.close(descriptor) - except OSError: - pass + for descriptor in descriptors: + try: + os.close(descriptor) + except OSError: + pass def _frame_bytes( self, *, kind: str, action_id: str, sequence: int, payload: dict[str, object] @@ -2145,6 +2164,12 @@ def _read_raw_frame( ) def _write_complete(self, payload: bytes, *, deadline: float | None = None) -> None: + # Keep the descriptor/HANDLE alive through the writer's finally mode + # restoration, including direct bridge.close() from another caller. + with self._active_io(): + self._write_complete_active(payload, deadline=deadline) + + def _write_complete_active(self, payload: bytes, *, deadline: float | None) -> None: try: if deadline is not None: with _nonblocking_pipe_writer(self._write_fd) as write: diff --git a/mcp-tools/tests/test_storage_firewall.py b/mcp-tools/tests/test_storage_firewall.py index 2b14898..dbb5f11 100644 --- a/mcp-tools/tests/test_storage_firewall.py +++ b/mcp-tools/tests/test_storage_firewall.py @@ -8,6 +8,7 @@ import sys import threading import time +from contextlib import contextmanager from pathlib import Path import pytest @@ -582,13 +583,15 @@ def test_storage_admission_rejects_substitution_and_unknown_fields() -> None: host.close() -def test_storage_admission_deadline_and_close_cancel_blocked_io() -> None: +def test_storage_admission_deadline_and_close_cancel_blocked_io( + monkeypatch: pytest.MonkeyPatch, +) -> None: from devkit_runtime import host_bridge, host_session from devkit_runtime.storage_intent import parse_storage_intent # Real pipe I/O with a fixed trusted clock: the transport deadline must # still advance monotonically. Peer never returns an admission response. - for scenario in ("no_reply", "full_pipe", "close"): + for scenario in ("no_reply", "full_pipe", "close", "direct_close"): child, host = _pipe_pair() profile_request = _storage_profile_request() pending = child.send_storage_profile_request(**{ @@ -605,7 +608,7 @@ def test_storage_admission_deadline_and_close_cancel_blocked_io() -> None: # preparation enrollment itself is covered by the round-trip test. session._completed_storage_profiles[_hash("7")] = host_session._CompletedStorageProfile( profile=profile, bridge=child, - expires_at=1_700_000_060 if scenario == "close" else 1_700_000_001, + expires_at=1_700_000_060 if scenario in {"close", "direct_close"} else 1_700_000_001, requested_bytes=4096, requested_files=8, ) intent = parse_storage_intent(_storage_intent( @@ -621,6 +624,20 @@ def test_storage_admission_deadline_and_close_cancel_blocked_io() -> None: break else: raise AssertionError("fixture pipe did not reach bounded capacity") + restore_reached = threading.Event() + restore_allowed = threading.Event() + original_writer = host_bridge._nonblocking_pipe_writer + if scenario == "direct_close": + @contextmanager + def paused_restore(descriptor: int): + with original_writer(descriptor) as write: + try: + yield write + finally: + restore_reached.set() + assert restore_allowed.wait(timeout=3), "mode-restore barrier was not released" + + monkeypatch.setattr(host_bridge, "_nonblocking_pipe_writer", paused_restore) result: list[object] = [] worker = threading.Thread(target=lambda: result.append(session.request_storage_admission( intent, profile_attestation_hash=_hash("7") @@ -636,20 +653,35 @@ def test_storage_admission_deadline_and_close_cancel_blocked_io() -> None: closer.start() closer.join(timeout=2) assert not closer.is_alive(), "close waited behind admission's business lock" + if scenario == "direct_close": + assert restore_reached.wait(timeout=2) + borrowed_fd = child._write_fd + child.close() + child.close() # repeated close must not bypass deferred cleanup + assert not child.is_available + assert child._active_io_count == 1 + assert not child._descriptors_closed + assert child._write_fd == borrowed_fd + os.fstat(borrowed_fd) # mode restoration still owns this exact fd + restore_allowed.set() worker.join(timeout=3) assert not worker.is_alive(), "admission did not terminate at its transport deadline" assert result == ["STORAGE_STAT_UNAVAILABLE"] assert not child._storage_admission_completions assert not child._storage_admission_decisions + assert child._active_io_count == 0 + assert child._descriptors_closed finally: # Only test-owned descriptors/threads; ensure a failing regression # cannot leave a blocked peer alive in the test process. + restore_allowed.set() child.close() host.close() worker.join(timeout=2) if closer is not None: closer.join(timeout=2) session.close() + monkeypatch.setattr(host_bridge, "_nonblocking_pipe_writer", original_writer) def test_profile_tamper_or_missing_field_fails_closed() -> None: From 012927ab410dcbacf18771fbd43b686be9e0fab3 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E5=93=80=E6=B4=9B=E8=8A=99?= Date: Sun, 30 Aug 2026 06:11:43 +0800 Subject: [PATCH 24/39] docs: align fast lane storage config and schema generation --- .../2026-08-29-storage-firewall-1.1.3.md | 46 +++++++++++++------ 1 file changed, 33 insertions(+), 13 deletions(-) diff --git a/docs/superpowers/plans/2026-08-29-storage-firewall-1.1.3.md b/docs/superpowers/plans/2026-08-29-storage-firewall-1.1.3.md index 0541537..b65c7b6 100644 --- a/docs/superpowers/plans/2026-08-29-storage-firewall-1.1.3.md +++ b/docs/superpowers/plans/2026-08-29-storage-firewall-1.1.3.md @@ -98,8 +98,8 @@ Codex Host files: - Modify `codex-rs/config/src/config_toml.rs` (`ConfigToml`), `codex-rs/core/src/config/mod.rs` (`Config::load_config_with_layer_stack`), and generated `codex-rs/core/config.schema.json` for the explicit Host - storage block. Read existing `config/src/config_layer_source.rs` for trust - provenance and `config/src/schema.rs::write_config_schema` for generation. + `fast_lane_storage` block. Read existing `config/src/config_layer_source.rs` + for trust provenance and `config/src/schema.rs::write_config_schema` for generation. - Modify `codex-rs/app-server/src/message_processor.rs`, `codex-rs/core/src/thread_manager.rs` (`ThreadManagerState`), `codex-rs/core/src/session/mod.rs` (`SessionSpawnArgs`), @@ -667,13 +667,15 @@ than concurrently editing registry/adapter/process-manager from two slices. - [ ] **Step 6a: Load explicit trusted configuration and inject one runtime-owned service.** -Add an optional `storage_firewall` block to `ConfigToml`, containing exactly -nine required fields: `approved_root`, `task_byte_limit`, `task_file_limit`, +Preserve the implemented optional `fast_lane_storage` block in `ConfigToml`, +containing exactly nine required fields: `approved_root`, `task_byte_limit`, `task_file_limit`, `target_family_byte_limit`, `target_family_file_limit`, `global_reserved_byte_limit`, `global_reserved_file_limit`, `free_space_floor_bytes`, and `emergency_floor_bytes`. The root must be an -explicit absolute Host-approved directory; each numeric value must pass the -kernel's positive/checked policy validation, including emergency <= floor. +explicit absolute Host-approved directory; all eight numeric fields use +`NonZeroU64` and must also pass the kernel's checked policy validation, +including emergency <= floor. Keep the root and all eight values pending the +operator's explicit choice; do not populate them in this plan revision. Do not supply numerical defaults, derive authority from free disk space, or create an approved root while loading configuration. Missing block disables storage admission with `STORAGE_POLICY_MISSING`; malformed blocks are rejected. @@ -684,20 +686,38 @@ workspace `Project` configuration cannot set or override it. Treat session flags as authority only when the Host startup path explicitly validates an operator override, never when copied from worker/caller facts. Reject an untrusted storage override instead of silently merging individual fields. -Generate `core/config.schema.json` through the existing -`config/src/schema.rs::write_config_schema` path. +Local configuration rejection uses `STORAGE_CONFIG_SOURCE_NOT_TRUSTED` and +`STORAGE_CONFIG_RESTART_REQUIRED`, alongside `STORAGE_ROOT_NOT_APPROVED` and +`STORAGE_POLICY_MISSING` where applicable. These are local configuration +diagnostics, not new public admission-wire fields or a change to the exact +request/receipt shape; do not expose private configuration paths in responses. -Construct one service at the process-scoped Host runtime's ThreadManager -creation in `app-server/src/message_processor.rs`; store the same Arc in -`ThreadManagerState`, forward through `SessionSpawnArgs`/`SessionServices`, -and give every `FastLaneHostFactsRegistry` a reference to it. The new +Generate only `core/config.schema.json` with the lightweight config example; +do not rebuild a core binary solely to emit schema. The implementation command +below is documentation, not an instruction to run it during plan-only edits: + +```powershell +Push-Location 'G:\2718lab\_codex\.codex-task-temp\codex-host-mcp-fix-recovery2\codex-rs' +$env:CARGO_TARGET_DIR='G:\2718lab\_codex\.codex-task-temp\codex-host-mcp-fix-recovery2\codex-rs\target' +$env:CARGO_INCREMENTAL='0' +cargo run --locked -p codex-config --example write_config_schema -j1 -- 'G:\2718lab\_codex\.codex-task-temp\codex-host-mcp-fix-recovery2\codex-rs\core\config.schema.json' +Pop-Location +``` + +Construct the sole service Arc from the manager's base `Config` in +`ThreadManager` initialization, not from per-session or per-turn config. +The Host runtime caller in `app-server/src/message_processor.rs` supplies +that base configuration. Store the same Arc in `ThreadManagerState`, forward +through `SessionSpawnArgs`/`SessionServices`, and give every +`FastLaneHostFactsRegistry` a reference to it. The new `storage_service.rs` facade owns the single kernel accounting state and immutable resolved policy/root, not another independent reservation ledger. Adapt every constructor/call site; test-only construction may be explicitly disabled or use an injected fixture, never a permissive production default. Current `session/session.rs` creates registries per Session: keep that facts scope but do not create a firewall there. Per-thread config reload must not -reset global reservations or replace policy/root while grants are retained. +reset global reservations or replace policy/root; reject a differing storage +configuration with the local `STORAGE_CONFIG_RESTART_REQUIRED` diagnostic. This is cross-session sharing inside one Host runtime, not a cross-process ledger. An independent Host process must not claim the same active storage From 51db3e89ad54d379d16fbe1c890a6eb41a70285f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E5=93=80=E6=B4=9B=E8=8A=99?= Date: Sun, 30 Aug 2026 06:23:44 +0800 Subject: [PATCH 25/39] docs: align owned ledger with shared storage lifecycle --- .../plans/2026-08-29-owned-cleanup-1.1.3.md | 1005 ++++++++--------- 1 file changed, 474 insertions(+), 531 deletions(-) diff --git a/docs/superpowers/plans/2026-08-29-owned-cleanup-1.1.3.md b/docs/superpowers/plans/2026-08-29-owned-cleanup-1.1.3.md index 07b9c06..46d3639 100644 --- a/docs/superpowers/plans/2026-08-29-owned-cleanup-1.1.3.md +++ b/docs/superpowers/plans/2026-08-29-owned-cleanup-1.1.3.md @@ -2,551 +2,494 @@ > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. -**Goal:** Persist every admitted generated-storage lease across process restarts and make generated-cache cleanup a bounded preview/recheck/apply transaction owned by the Codex Host. +**Goal:** Give the existing Host storage service one durable family/member/control accounting authority, then add explicitly authorized, bounded generated-cache cleanup. -**Architecture:** Plan 1 supplies the validated `StorageAdmissionReceipt` and deterministic target root. This plan adds a host-owned JSON ledger with atomic replacement, owner/process fencing, restart recovery, byte/file/free-space accounting, and a fair pressure state; the DevKit only forwards typed status/preview/apply requests over the authenticated bridge. A cleanup candidate is immutable evidence, not permission: only a fresh candidate hash, policy hash, ledger epoch, writer fence, and post-stat match can authorize a bounded generated-cache deletion. +**Architecture:** Plan 1 supplies strict intents, verified group/member authority, and a single service per runtime. Plan 2 moves kernel state into a root-shared transactional ledger: every Host transaction takes an OS fence, reloads current state, rechecks epoch/owners, transitions and atomically persists. Admission is one transition, not an admission followed by another reservation. Independently owned control allocations keep bootstrap/queue metadata bounded across waves without holding Cargo family leases. Cleanup is a later handle-fenced transaction over proven disposable objects, not a consequence of release, expiry, size, or age. -**Tech Stack:** Rust 2021 (`serde`, `serde_json`, `sha2`, `tokio`, `std::fs`, `std::time`), Python 3.11 (`dataclasses`, `hashlib`, `json`, `pathlib`), FastMCP/Pydantic, and the existing atomic JSON replacement and authenticated inherited-handle transport. +**Tech Stack:** Existing Rust `serde`/`serde_json`/`sha2`, platform filesystem/process handles, existing durable replacement helpers, and the authenticated bridge; Python is a path-free projection only. ---- - -## Scope and file map - -Line ranges refer to the Plan 1 baseline (`37029a9` for DevKit and -`552fe8035d` for Host). Re-read the named symbol before editing because Plan 1 -will add the storage admission types. - -DevKit: - -- Create `mcp-tools/devkit_runtime/storage_ledger.py`: typed status, preview, - and apply request/receipt projections; it must never inspect or delete a host - path. -- Modify `mcp-tools/devkit_runtime/host_bridge.py:218-264,916-1045` to carry - `storage_status`, `storage_preview`, and `storage_apply` request/receipt - frames through the authenticated session. -- Modify `mcp-tools/devkit_runtime/host_session.py:159-335,636-735` with - `storage_status()`, `storage_preview()`, and `storage_apply()` methods that - return only stable codes, hashes, counts, and opaque receipt identities. -- Modify `mcp-tools/server.py:179-289,1008-1314` and - `mcp-tools/devkit_runtime/tool_metadata.py:1-28` to add the read-only - `storage_status`/`storage_preview` tools and the explicitly destructive - `storage_apply` tool; the apply model requires candidate/policy/epoch hashes - and a finite batch limit. -- Create `mcp-tools/tests/test_storage_ledger.py` and modify - `mcp-tools/tests/test_mcp_contract.py:240-360` for tool annotations and - exact request validation. - -Codex Host: - -- Create `codex-rs/core/src/fast_lane_host_dispatch/storage_ledger.rs`: the - `LeaseRecord`, ledger snapshot, atomic journal, owner probe, recovery state, - quota accounting, candidate manifest, and generated apply transaction. -- Create `codex-rs/core/src/fast_lane_host_dispatch/storage_ledger_tests.rs`. -- Modify `codex-rs/core/src/fast_lane_host_dispatch/mod.rs:1-49` to register and - export the ledger types to the coordinator. -- Modify `codex-rs/core/src/fast_lane_host_dispatch/storage_firewall.rs` at - its admission/release methods to call the ledger rather than maintaining - process-local counters. -- Modify `codex-rs/core/src/fast_lane_host_dispatch/registry.rs:567-2060` - and `coordinator.rs:393-580,1218-1260` to recover/open the ledger at host - startup, reserve/heartbeat/release records, and block admission in pressure - or recovery state. -- Modify `codex-rs/rmcp-client/src/inherited_host_bridge_protocol.rs:35-240` - and `.../envelope.rs:25-220` for exact ledger/preview/apply wire schemas. -- Modify `codex-rs/rmcp-client/src/inherited_host_bridge_protocol/session.rs:159-240,247-350,412-570` - to expose the typed operation queue without creating a second receiver. - -No source/session deletion, GitHub reachability, or CAS deduplication belongs -to this plan; those are Plan 3. No unknown directory can enter this ledger. - -## Shared lease schema and public operation contract - -This plan consumes Plan 1's `StorageAdmissionReceipt` and uses this exact -record shape for `schema == "2718lab.storage.lease.v1"`: - -```rust -#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] -pub(crate) struct LeaseRecord { - pub(crate) ledger_epoch: u64, - pub(crate) schema_version: String, - pub(crate) lease_id: String, - pub(crate) task_id: String, - pub(crate) assignment_id: String, - pub(crate) plan_binding: String, - pub(crate) project_identity: String, - pub(crate) repository_identity: String, - pub(crate) worktree_identity: String, - pub(crate) artifact_kind: String, - pub(crate) target_key: String, - pub(crate) path_identity: String, - pub(crate) owner_epoch: u64, - pub(crate) owner_kind: String, - pub(crate) process_id: u32, - pub(crate) process_start_time: u64, - pub(crate) host_instance_id: String, - pub(crate) state: LeaseState, - pub(crate) created_at: u64, - pub(crate) last_heartbeat: u64, - pub(crate) expires_at: u64, - pub(crate) restart_generation: u64, - pub(crate) reserved_bytes: u64, - pub(crate) reserved_files: u64, - pub(crate) observed_bytes: u64, - pub(crate) observed_files: u64, - pub(crate) free_space_before: u64, - pub(crate) free_space_after_reserve: u64, - pub(crate) free_space_floor: u64, - pub(crate) candidate_hash: Option, - pub(crate) receipt_hash: Option, - pub(crate) release_reason: Option, - pub(crate) cleanup_policy_hash: Option, -} -``` - -`LeaseState` is exactly `reserved | active | released | recovery_pending | -quarantined | cleanup_eligible`. The only legal automatic transitions are -`reserved -> active -> released`; restart evidence can move an active or -reserved record to `recovery_pending`, and failed verification can move it to -`quarantined`. `cleanup_eligible` never deletes anything by itself. -`StorageLedgerError::LeaseConflict` maps exactly to -`STORAGE_LEASE_CONFLICT`; it is returned for a stale owner proof, duplicate -activation, heartbeat after release, and release by a different owner. - -The public status/preview/apply shapes are: - -```json -{ - "schema": "2718lab.storage.preview.v1", - "ledger_epoch": 12, - "policy_hash": "sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", - "candidate_hash": "sha256:bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", - "candidates": [ - { - "path_identity": "sha256:cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc", - "artifact_kind": "cargo-target", - "bytes": 1024, - "files": 3, - "content_hash": "sha256:dddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd", - "owner_state": "none", - "classification": "generated-disposable", - "lease_id": "sha256:eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee" - } - ] -} -``` - -`storage_apply` accepts exactly `candidate_hash`, `policy_hash`, -`ledger_epoch`, and `batch_limit` (1 through 16). It returns -`STORAGE_CANDIDATE_STALE`, `STORAGE_PROTECTED_UNKNOWN`, -`STORAGE_PROTECTED_ACTIVE`, `STORAGE_PROTECTED_DIRTY`, or -`STORAGE_APPLY_INCOMPLETE` without deleting when any recheck differs. - -## Implementation tasks - -### Task 1: Define ledger and operation RED tests - -**Files:** -- Create: `mcp-tools/tests/test_storage_ledger.py` -- Create: `codex-rs/core/src/fast_lane_host_dispatch/storage_ledger_tests.rs` -- Modify: `mcp-tools/tests/test_mcp_contract.py:240-360` - -- [ ] **Step 1: Add the Python RED test for exact apply fields and bounded batch.** - -```python -def test_storage_apply_rejects_path_and_unbounded_batch(): - from devkit_runtime.storage_ledger import StorageApplyRequest, StorageLedgerError - - try: - StorageApplyRequest.from_mapping({ - "candidate_hash": "sha256:" + "a" * 64, - "policy_hash": "sha256:" + "b" * 64, - "ledger_epoch": 1, - "batch_limit": 17, - "path": "G:/source" - }) - except StorageLedgerError as error: - assert error.code == "STORAGE_CANDIDATE_STALE" - else: - raise AssertionError("invalid apply request was accepted") -``` - -- [ ] **Step 2: Add the Rust RED test for an invalid transition.** - -```rust -#[test] -fn released_lease_cannot_receive_a_heartbeat() { - let mut ledger = test_ledger(); - let lease = ledger.reserve(test_admission()).unwrap(); - ledger.activate(&lease.lease_id, owner()).unwrap(); - ledger.release(&lease.lease_id, "terminal").unwrap(); - assert_eq!( - ledger.heartbeat( - &lease.lease_id, - owner(), - ObservedStorage { bytes: 100, files: 1 }, - ), - Err(StorageLedgerError::LeaseConflict), - ); -} -``` - -- [ ] **Step 3: Run only the new RED tests.** - -```powershell -$env:PYTEST_DISABLE_PLUGIN_AUTOLOAD='1'; python -m pytest tests/test_storage_ledger.py::test_storage_apply_rejects_path_and_unbounded_batch -q -o cache_dir=G:\2718lab\_codex\.codex-task-temp\storage-113-ledger-pytest -Push-Location 'G:\2718lab\_codex\.codex-task-temp\codex-host-mcp-fix-recovery2\codex-rs'; $env:CARGO_TARGET_DIR='G:\2718lab\_codex\.codex-task-temp\storage-113-ledger-rust-target'; cargo test -p codex-core released_lease_cannot_receive_a_heartbeat --locked -j1; Pop-Location -``` - -Expected: both commands fail because the ledger types do not exist. The -failure must occur before any production path or deletion call. - -- [ ] **Step 4: Commit only the RED contract.** - -```powershell -Push-Location 'G:\2718lab\_codex\.codex-task-temp\devkit-1.1.2-recovery'; git add mcp-tools/tests/test_storage_ledger.py mcp-tools/tests/test_mcp_contract.py; git commit -m 'test: define owned storage ledger contract'; Pop-Location -Push-Location 'G:\2718lab\_codex\.codex-task-temp\codex-host-mcp-fix-recovery2'; git add codex-rs/core/src/fast_lane_host_dispatch/storage_ledger_tests.rs; git commit -m 'test: define owned storage ledger contract'; Pop-Location -``` - -### Task 2: Implement atomic ledger snapshots and schema migration - -**Files:** -- Create: `codex-rs/core/src/fast_lane_host_dispatch/storage_ledger.rs` -- Modify: `codex-rs/core/src/fast_lane_host_dispatch/mod.rs:1-49` -- Test: `codex-rs/core/src/fast_lane_host_dispatch/storage_ledger_tests.rs` - -- [ ] **Step 1: Define the store and exact snapshot envelope.** - -```rust -#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)] -pub(crate) struct LedgerSnapshot { - pub(crate) schema: String, - pub(crate) ledger_epoch: u64, - pub(crate) restart_generation: u64, - pub(crate) host_instance_id: String, - pub(crate) policy_hash: String, - pub(crate) leases: Vec, - pub(crate) journal: Option, -} - -pub(crate) struct StorageLedger { - path: PathBuf, - snapshot: LedgerSnapshot, - owner_probe: Box, - capacity: Box, -} -``` - -`open` must reject a symlink/reparse-point ledger file, malformed JSON, a -non-monotonic epoch, unknown state, duplicate `lease_id`, or a path whose -canonical parent is outside the approved generated root. An absent file is -opened as a zero-lease `storage-ledger-v1` snapshot only after the parent root -has been proved approved; it is not an authorization to write arbitrary roots. - -- [ ] **Step 2: Implement atomic replacement with a journal.** - -```rust -fn persist(&mut self, next: LedgerSnapshot) -> Result<(), StorageLedgerError> { - validate_snapshot(&next)?; - let temporary = self.path.with_extension("json.stage"); - let bytes = serde_json::to_vec(&next).map_err(|_| StorageLedgerError::StatUnavailable)?; - let mut file = OpenOptions::new().write(true).create_new(true).open(&temporary) - .map_err(|_| StorageLedgerError::StatUnavailable)?; - file.write_all(&bytes).map_err(|_| StorageLedgerError::StatUnavailable)?; - file.sync_all().map_err(|_| StorageLedgerError::StatUnavailable)?; - replace_file_durably(&temporary, &self.path)?; - self.snapshot = next; - Ok(()) -} -``` - -The Windows replace helper must use the same write-through replacement -semantics already used by `registry.rs:4281-4380`; Unix uses `rename` after -`sync_all`. A failed replacement leaves the prior snapshot and the stage file -is removed only when its identity still matches the stage created by this -operation. - -- [ ] **Step 3: Add migration and rollback tests, then turn the RED schema tests green.** - -```rust -#[test] -fn old_or_missing_ledger_becomes_recovery_pending_without_deletion() { - let mut ledger = open_fixture_with_legacy_snapshot(); - let result = ledger.recover_after_restart(current_owner_set_empty()); - assert!(result.is_ok()); - assert!(ledger.records().iter().all(|record| record.state == LeaseState::RecoveryPending)); - assert!(fixture_generated_file().exists()); -} -``` - -```powershell -Push-Location 'G:\2718lab\_codex\.codex-task-temp\codex-host-mcp-fix-recovery2\codex-rs'; $env:CARGO_TARGET_DIR='G:\2718lab\_codex\.codex-task-temp\storage-113-ledger-rust-target'; cargo test -p codex-core storage_ledger --locked -j1; Pop-Location -``` - -Expected: the focused ledger tests pass; migration failure returns to the -previous snapshot and keeps every generated file. - -- [ ] **Step 4: Commit the durable ledger core.** - -```powershell -Push-Location 'G:\2718lab\_codex\.codex-task-temp\codex-host-mcp-fix-recovery2'; git add codex-rs/core/src/fast_lane_host_dispatch/storage_ledger.rs codex-rs/core/src/fast_lane_host_dispatch/storage_ledger_tests.rs codex-rs/core/src/fast_lane_host_dispatch/mod.rs; git commit -m 'feat: persist storage lease ledger atomically'; Pop-Location -``` - -### Task 3: Enforce reserve, heartbeat, release, and pressure gates - -**Files:** -- Modify: `codex-rs/core/src/fast_lane_host_dispatch/storage_firewall.rs` -- Modify: `codex-rs/core/src/fast_lane_host_dispatch/registry.rs:567-1668` -- Modify: `codex-rs/core/src/fast_lane_host_dispatch/coordinator.rs:393-580,1218-1260` -- Test: `codex-rs/core/src/fast_lane_host_dispatch/storage_ledger_tests.rs` - -- [ ] **Step 1: Add RED accounting tests for all four admission equations.** - -```rust -#[test] -fn byte_file_global_and_floor_limits_fail_closed() { - let cases = [ - (AdmissionMutation::TaskBytes, "STORAGE_QUOTA_EXCEEDED"), - (AdmissionMutation::TaskFiles, "STORAGE_FILE_LIMIT_EXCEEDED"), - (AdmissionMutation::GlobalReserved, "STORAGE_QUOTA_EXCEEDED"), - (AdmissionMutation::FreeFloor, "STORAGE_FREE_SPACE_FLOOR"), - ]; - for (mutation, code) in cases { - let firewall = fixture_firewall(mutation); - assert_eq!(firewall.admit(test_intent()).unwrap_err().code(), code); - assert!(fixture_target_root().read_dir().unwrap().next().is_none()); - } -} -``` - -- [ ] **Step 2: Implement lease state methods with owner fencing.** - -```rust -pub(crate) fn reserve(&mut self, admission: StorageAdmissionReceipt, now: u64) -> Result; -pub(crate) fn activate(&mut self, lease_id: &str, owner: OwnerProof) -> Result<(), StorageLedgerError>; -pub(crate) fn heartbeat(&mut self, lease_id: &str, owner: OwnerProof, observed: ObservedStorage) -> Result; -pub(crate) fn release(&mut self, lease_id: &str, owner: OwnerProof, reason: &str) -> Result; -``` +**Revision status (2026-08-30):** This replaces the unpublished flat lease-v1 design with internal snapshot-v2 aligned to Host `937ae14` kernel/service symbols and Plan 1's Task 6 map. It does not claim implementation, migration, cleanup, or activation is complete. Preserve admission-v1 exact5, existing intent/target/profile contracts, SHA-256 admission/family IDs, and the strict `fast_lane_storage` root-plus-eight policy shape. No new artifact kind, public admission field, or user control-budget field is introduced. -`heartbeat` remeasures bytes/files and applies the task, family, global, and -free-space equations before persisting. If an observation exceeds a limit, -new reservations return the stable pressure/quota code; the active lease is -not killed and its directory is not deleted. At or below -`emergency_floor_bytes`, the ledger enters `pressure=true` and allows only -release, recovery, and read-only preview operations. +The main thread records Host `841fdaf` three-crate compile exit 0 and DevKit +`b2aff` three selected final tests exit 0 for the existing slices. Retain that +scope of evidence; it does not prove the new Plan 2 ledger/cleanup exists. +Production cleanup and the 1.1.3 release remain incomplete. -- [ ] **Step 3: Connect `registry.rs` and the coordinator to the ledger.** +**Compile first:** Reuse only +`G:\2718lab\_codex\.codex-task-temp\codex-host-mcp-fix-recovery2\codex-rs\target`, +with `CARGO_INCREMENTAL=0` and `--locked -j1`. Retain prior compile evidence; +compile changed crates before at most the two core boundary probes below. +Do not create a ledger-specific Cargo target or repeatedly run full suites. +Commands here are implementation instructions, not commands to run during +this documentation-only revision. No live cleanup/configuration is authorized. -```rust -let admission = storage_firewall.admit(intent)?; -let lease = storage_ledger.reserve(admission, clock.now()?)?; -let prepared = adapter.prepare_batch_with_storage(batch, lease.clone()).await?; -storage_ledger.activate(&lease.lease_id, owner_probe.current()?)?; -``` - -Every failed preparation calls `release` with `"prepare_failed"`; every -terminal/recovery path calls it with its exact reason. Releasing the Fast Lane -scope lease and releasing storage are separate journal entries bound by the -same `assignment_id`, `plan_binding`, and receipt hash. - -- [ ] **Step 4: Run the focused accounting and core compile gates.** - -```powershell -Push-Location 'G:\2718lab\_codex\.codex-task-temp\codex-host-mcp-fix-recovery2\codex-rs'; $env:CARGO_TARGET_DIR='G:\2718lab\_codex\.codex-task-temp\storage-113-ledger-rust-target'; cargo test -p codex-core byte_file_global_and_floor_limits_fail_closed --locked -j1; cargo check -p codex-core --lib --locked -j1; Pop-Location -``` - -Expected: the four cases pass and `cargo check` finishes with zero warnings. -If disk statistics fail, the result is `STORAGE_STAT_UNAVAILABLE` and no new -target root is created. - -- [ ] **Step 5: Commit the quota and lifecycle wiring.** - -```powershell -Push-Location 'G:\2718lab\_codex\.codex-task-temp\codex-host-mcp-fix-recovery2'; git add codex-rs/core/src/fast_lane_host_dispatch/storage_firewall.rs codex-rs/core/src/fast_lane_host_dispatch/registry.rs codex-rs/core/src/fast_lane_host_dispatch/coordinator.rs codex-rs/core/src/fast_lane_host_dispatch/storage_ledger_tests.rs; git commit -m 'feat: bind storage leases to quota lifecycle'; Pop-Location -``` - -### Task 4: Implement restart owner recovery and fail-closed quarantine - -**Files:** -- Modify: `codex-rs/core/src/fast_lane_host_dispatch/storage_ledger.rs` -- Modify: `codex-rs/core/src/fast_lane_host_dispatch/registry.rs:3809-4380` -- Test: `codex-rs/core/src/fast_lane_host_dispatch/storage_ledger_tests.rs` - -- [ ] **Step 1: Add RED tests for PID reuse, changed path, unknown files, and locked-stat recovery.** - -```rust -#[test] -fn restart_requires_instance_pid_start_and_owner_epoch() { - let mut ledger = open_fixture_with_active_lease(owner_with(41, 900, 7)); - ledger.recover_after_restart(owner_with(41, 901, 7)).unwrap(); - assert_eq!(ledger.records()[0].state, LeaseState::RecoveryPending); - assert_eq!(ledger.records()[0].restart_generation, 2); -} -``` - -- [ ] **Step 2: Implement `OwnerProbe` and recovery validation.** - -```rust -pub(crate) trait OwnerProbe: Send + Sync { - fn current(&self) -> Result; - fn matches(&self, owner: &OwnerProof) -> Result; -} - -fn recover_record(record: &mut LeaseRecord, owner_probe: &dyn OwnerProbe, root: &Path) -> Result<(), StorageLedgerError> { - if record.state != LeaseState::Active && record.state != LeaseState::Reserved { - return Ok(()); - } - if !owner_probe.matches(&OwnerProof::from_record(record))? - || !verify_target_identity(root, record)? - || !manifest_matches(record)? - { - record.state = LeaseState::Quarantined; - return Ok(()); - } - record.state = LeaseState::Active; - Ok(()) -} -``` - -The host increments `restart_generation` under the ledger lock before examining -records. Missing receipt, path change, dirty state, unknown file, or a failed -lock/stat check becomes `quarantined`; an owner that cannot be proved becomes -`recovery_pending` until a later explicit recovery receipt. `apply` is blocked -while any recovery remains unresolved. - -- [ ] **Step 3: Add a restart recovery receipt and verify no deletion occurred.** - -```rust -assert_eq!(receipt.code(), "STORAGE_RECOVERY_REQUIRED"); -assert_eq!(receipt.restart_generation(), 2); -assert!(fixture_generated_file().exists()); -``` - -- [ ] **Step 4: Run the focused recovery probe and compile gate.** - -```powershell -Push-Location 'G:\2718lab\_codex\.codex-task-temp\codex-host-mcp-fix-recovery2\codex-rs'; $env:CARGO_TARGET_DIR='G:\2718lab\_codex\.codex-task-temp\storage-113-ledger-rust-target'; cargo test -p codex-core restart_requires_instance_pid_start_and_owner_epoch --locked -j1; cargo check -p codex-core --lib --locked -j1; Pop-Location -``` - -Expected: `1 passed`, then a zero-warning compile. - -- [ ] **Step 5: Commit restart recovery.** - -```powershell -Push-Location 'G:\2718lab\_codex\.codex-task-temp\codex-host-mcp-fix-recovery2'; git add codex-rs/core/src/fast_lane_host_dispatch/storage_ledger.rs codex-rs/core/src/fast_lane_host_dispatch/registry.rs codex-rs/core/src/fast_lane_host_dispatch/storage_ledger_tests.rs; git commit -m 'feat: recover storage ownership across restarts'; Pop-Location -``` - -### Task 5: Add preview hash, recheck fence, and bounded generated apply - -**Files:** -- Modify: `codex-rs/core/src/fast_lane_host_dispatch/storage_ledger.rs` -- Modify: `codex-rs/rmcp-client/src/inherited_host_bridge_protocol.rs:35-240` -- Modify: `codex-rs/rmcp-client/src/inherited_host_bridge_protocol/envelope.rs:25-220` -- Modify: `codex-rs/rmcp-client/src/inherited_host_bridge_protocol/session.rs:412-570` -- Test: `codex-rs/core/src/fast_lane_host_dispatch/storage_ledger_tests.rs` - -- [ ] **Step 1: Add RED tests for candidate invalidation and protected classifications.** - -```rust -#[test] -fn preview_hash_invalidates_on_epoch_owner_or_content_change() { - let mut ledger = test_ledger_with_disposable_candidate(); - let preview = ledger.preview().unwrap(); - ledger.bump_epoch_for_test(); - let error = ledger.apply(&ApplyRequest::from_preview(&preview, 1)).unwrap_err(); - assert_eq!(error.code(), "STORAGE_CANDIDATE_STALE"); - assert!(fixture_candidate_path().exists()); -} -``` +--- -- [ ] **Step 2: Implement canonical candidate manifest and preview hash.** +## Dependency order and bounded file map + +Implement **P2-base** (Tasks 1-3: root guard, unique ledger transaction, +bootstrap/control allocations and restart protection) before enabling Plan 1 +Task 6's durable refill registration. Plan 1 does not depend on **P2-apply** +(Tasks 4-5: preview/delete tooling). Conversely P2-apply depends on Plan 1's +real process/descendant terminal fence and a correct family postcheck. +Never unblock a circular dependency with free metadata writes or fake proof. +The next production integration order is P2-base/control, then Plan 1's +remaining lifecycle/terminal wiring, then P2-apply. The root and eight policy +values are still awaiting the user's confirmation; this plan requires no +additional control-budget choice and fills in no values on the user's behalf. + +All Host paths below are relative to +`G:\2718lab\_codex\.codex-task-temp\codex-host-mcp-fix-recovery2`. +DevKit paths are relative to +`G:\2718lab\_codex\.codex-task-temp\devkit-1.1.2-recovery`. +Re-read current symbols before editing; unrelated dirty work stays untouched. +Bare Host module filenames below resolve within +`codex-rs/core/src/fast_lane_host_dispatch/`; other paths are explicitly prefixed. + +| Slice | Files and ownership | +| --- | --- | +| Base state/transactions | Create `codex-rs/core/src/fast_lane_host_dispatch/storage_ledger.rs` for typed snapshots, `RootStorageLedger` transactions, bounded commit/recovery and control ownership; its local mutex serializes only this Host's callers, never replaces the cross-process fence/reload; register in `mod.rs`. | +| Real OS boundary | Create `codex-rs/core/src/fast_lane_host_dispatch/storage_fs.rs` for root/process exclusion, owned directory/file handles, bounded durable replacement and deletion primitives; no permissive path-string fallback. | +| Existing kernel integration | Modify `storage_firewall.rs` to evaluate transitions against freshly loaded transaction state instead of its own independent `Mutex`; modify `storage_service.rs::HostStorageService` to own one ledger/checker entry point per runtime into the shared root state. | +| Runtime/base initialization | Modify `codex-rs/core/src/thread_manager.rs` and existing service/session construction only as required to initialize the shared base-Config service once. Registries do not reopen a ledger per session. | +| Authority/queue lifecycle | Modify `fast_lane_host_dispatch/registry.rs` at initial admission, `consume_batch`, `register_refill_queue`, `consume_refill_queue` and queue persistence; modify `codex_adapter.rs`/`coordinator.rs` only at lifecycle settlement seams. Keep original route/lease hashes unchanged. | +| Later cleanup | Create `fast_lane_host_dispatch/storage_cleanup.rs` for candidate classification/manifest and apply state machine; reuse `storage_fs.rs` and Plan 1 terminal/postcheck evidence. | +| Later wire projection | Modify `codex-rs/rmcp-client/src/inherited_host_bridge_protocol.rs` and its `envelope.rs`/`session.rs`/`pump.rs` only for status/preview/apply; keep the single authenticated writer/receiver arrangement. | +| Later DevKit projection | Create `mcp-tools/devkit_runtime/storage_ledger.py`; modify existing `host_bridge.py`, `host_session.py`, `mcp-tools/server.py` and `devkit_runtime/tool_metadata.py` for typed read-only status/preview and explicitly destructive apply. | +| Bounded verification | Create `codex-rs/core/src/fast_lane_host_dispatch/storage_ledger_tests.rs`; reuse existing firewall tests. Add only the necessary exact Python request/tool-annotation assertion in `mcp-tools/tests/test_storage_ledger.py` / `test_mcp_contract.py`. | + +No source/session deletion, GitHub reachability, CAS, compression, remote sync, +new dependency, or live configuration change belongs to this revision. +Unknown existing directories are protected, not automatically imported. + +## Internal snapshot-v2: family, group, member, control + +The existing kernel has `Family { lease, observed }`, +`FamilyLease { owner, lease_id, reserved, allowance }`, +`Group { authority_subject, sealed, members }` and +`MemberPhase::{Reserved, Consumed, Released}`. Preserve those semantics; +do not flatten them into one task per family lease or invent an `Active` +phase that changes the consume-before-prepare boundary. + +The following are **planned private persistence records**, not wire authority +types. Decode with exact fields, bounded lengths/counts, checked arithmetic +and duplicate-key rejection. `Usage` contains `bytes: u64, files: u64`. + +| Record / unique key | Required contents | +| --- | --- | +| `LedgerSnapshotV2` / one approved root | `schema = "2718lab.storage.ledger-snapshot.v2"`, root-wide `ledger_epoch`, root identity, policy hash, owner/family/group/member/control collections and one bounded optional pending operation. Restart generations belong to individual owners, not a global takeover generation. | +| `FamilySnapshot` / `target_key` | Canonical descriptor/root identity, retained `observed: Usage` and optional `FamilyLeaseSnapshot`. Shared Cargo data is observed once per family, not once per member. Every existing artifact kind retains its common `cargo-target` child. | +| `FamilyLeaseSnapshot` / active `target_family_lease_id` | One `owner_id`, `reserved: Usage` and `allowance: Usage`. One family has at most one active lease/owner; many same-owner members may reference it. IDs remain strict lowercase SHA-256 digests. | +| `GroupSnapshot` / `owner_id` | Original Host batch/selected-wave authority subject and provenance binding, exact Host-owner reference, optional exact sealed task set, authority deadline/epoch, and recovery disposition. It is not keyed by each task's distinct Sent profile hash. | +| `MemberSnapshot` / (`owner_id`, `task_id`) | Original intent, exact decision/receipt identity including admission/family IDs, `StorageAssignmentBinding`, derived private path identities, `MemberPhase`, and terminal/postcheck evidence references. Released records retain replay tombstones. | +| `ControlSnapshot` / `control_id` | Internal purpose (`Bootstrap` or `RefillQueue`, not an artifact kind), verified queue/Host provenance, exact Host-owner reference and independent lifecycle owner, safe control-path identity, bounded payload identity, committed observation, reserved physical footprint/growth and state. No Cargo family lease or fabricated task ID. | +| `OwnerBinding` / (`host_instance_id`, `owner_epoch`) | Host instance identity, PID plus creation identity, this owner's restart generation and recovery disposition. Mutations require the live root transaction guard and matching owner evidence; serialized fields alone never substitute for process handles. | +| `PendingOperation` / at most one root transaction | Operation/expected epoch, exact affected identities, before/after state hashes, reserved metadata footprint and stage/cleanup progress. No append-only unbounded journal. | + +`RecoveryDisposition::{Current, RecoveryPending, Quarantined}` is orthogonal +to member phase. A persisted `Released` tombstone does not become executable +again after restart. Duplicate member keys and inconsistent family references +are invalid; repeated references to the **same** family lease from distinct +members are expected. A different owner cannot reference that active lease. +Released tombstones retain their historical lease/receipt identities even +after the family has no lease or a later owner has acquired a new one. + +Private paths are derived/reopened against the approved root and checked OS +identities. Deserializing a subject hash must not call +`StorageAdmissionAuthority::from_verified_batch` as if provenance had been +verified. Rehydration must pass the service's recovery boundary first. +Do not persist or deserialize live handle/termination capabilities. + +### Single ownership and arithmetic + +`RootStorageLedger::transact` is the sole state-mutation/commit boundary. +Every transaction acquires the real cross-process root fence, reloads the +latest bounded snapshot, rechecks epoch/policy/owners, computes a transition +and atomically persists before releasing the fence. `HostStorageService` +remains one entry point per runtime, constructed from its manager's base +Config and shared across its sessions. Multiple Host processes coordinate +through this same root transaction protocol, not separate cached counters. +The kernel evaluates only the current transaction state. An optional cached +snapshot is non-authoritative for reservation, release, recovery or deletion. + +Move the current `State` into this durable authority rather than mirroring it. Do not expose a new +`ledger.reserve(receipt)` after `reserve_member_once`. Receipt delivery/replay, +group sealing and member attachment do not reserve again. Kernel public +wrappers must delegate into the same ledger transaction, never recursively +lock the old firewall while a ledger transaction is held. + +Compute/validate aggregate caches from authoritative records at load and +commit; never trust a serialized global total independently: + +- Family reserved = sum of original budgets of its non-Released members. +- Global reserved = those member budgets plus independently reserved control + footprints, each physical control extent counted once. +- Family observed remains after the lease is removed; private member counts + are not added to it again. Observed family bytes already occupy disk and + are not newly added to global growth reservation. +- A new member adds its budget once to family reserved, family allowance and + global reserved. Same member/intent replay returns the original grant; + changed intent or a Released member cannot reserve again. +- For unused `Reserved` cancellation, remove that member's budget from the + prior allowance before postcheck. For `Consumed` settlement, require the + real termination evidence and bounded scan; then retire unused allowance: + `next_allowance = min(adjusted_allowance, observed + remaining_reserved)` + componentwise with checked arithmetic. Never lend a released allowance to + another writer. When no members remain, remove the active family lease but + retain observed usage and tombstones. +- Over-budget postcheck updates conservative known usage without releasing + ownership/counters. Preserve Plan 1's all-members-quiet barrier for the + final shared-family observation; a mutex is not a filesystem writer fence. + +Capacity uses the existing root-plus-eight policy. Windows physical free-file +capacity remains `None`, not infinity; enforce logical file budgets from +owned observations/reservations. Metadata fits under the same global and +free-space checks. Below emergency pressure, no new positive reservation is +allowed; already-reserved bounded recovery/settlement space remains usable. +No pressure state authorizes killing a process or deleting data. + +## Real authority and filesystem interfaces + +Implement these interfaces as private RAII/handle types in `storage_fs.rs`; +the names describe new implementation work, not an existing capability: ```rust -pub(crate) fn preview(&self) -> Result { - let mut candidates = self.scan_registered_generated_roots()?; - candidates.sort_by(|left, right| left.path_identity.cmp(&right.path_identity)); - let manifest = serde_json::json!({ - "schema": "2718lab.storage.preview.v1", - "ledger_epoch": self.snapshot.ledger_epoch, - "policy_hash": self.snapshot.policy_hash, - "candidates": candidates, - }); - Ok(StoragePreview { manifest, candidate_hash: canonical_hash(&manifest)? }) +trait RootOwnershipProvider { + fn acquire(&self, root: &ApprovedRoot) -> Result; } -``` - -The scan follows no reparse point, visits only ledger-registered generated -roots, and labels active/unknown/dirty/source/session entries as protected. -It does not select by size, age, or directory name. - -- [ ] **Step 3: Implement apply as recheck, journal, delete, postcheck.** - -```rust -pub(crate) fn apply(&mut self, request: ApplyRequest) -> Result { - let preview = self.preview()?; - if request.candidate_hash != preview.candidate_hash - || request.policy_hash != self.snapshot.policy_hash - || request.ledger_epoch != self.snapshot.ledger_epoch - { - return Err(StorageLedgerError::CandidateStale); - } - let fence = self.writer_fence()?; - let selected = preview.generated_disposable(request.batch_limit)?; - for candidate in selected { - self.recheck_candidate(&candidate, &fence)?; - self.write_journal_started(&candidate)?; - self.remove_verified_generated_path(&candidate)?; - if candidate.path_exists()? { - return Err(StorageLedgerError::PostcheckFailed); - } - self.mark_released(&candidate.lease_id)?; - } - self.write_receipt_and_release_fence() +trait OwnedFilesystem { + fn open_child_no_follow( + &self, parent: &OwnedDirectoryHandle, child: &SingleComponent, + ) -> Result; + fn begin_mutation( + &self, root: &RootTransactionGuard, subject: &OwnedEntryHandle, + ) -> Result; + fn remove_verified_tree( + &self, fence: &NamespaceMutationFence, entry: &OwnedEntryHandle, + limit: &RemovalBound, + ) -> Result; } ``` -A failed item returns `STORAGE_APPLY_INCOMPLETE` and leaves all later items -untouched. A changed candidate releases the writer fence without deletion. -Apply cannot run while the ledger is in pressure recovery or while any -candidate is active, unknown, dirty, source, or session classified. - -- [ ] **Step 4: Add the exact bridge operations and DevKit projections.** - -```python -def storage_apply(self, request: StorageApplyRequest) -> dict[str, object]: - if request.batch_limit < 1 or request.batch_limit > 16: - return {"code": "STORAGE_CANDIDATE_STALE"} - response = _host_session().storage_apply(request.to_wire()) - return project_storage_receipt(response) -``` - -The bridge validator requires exact schemas -`2718lab.storage.status.v1`, `2718lab.storage.preview.v1`, and -`2718lab.storage.apply.v1`; `storage_apply` is the only destructive operation. -The DevKit projection strips absolute paths and owner/process values before -returning an MCP result. - -- [ ] **Step 5: Run preview/apply focused tests and compile-first gates.** - -```powershell -Push-Location 'G:\2718lab\_codex\.codex-task-temp\codex-host-mcp-fix-recovery2\codex-rs'; $env:CARGO_TARGET_DIR='G:\2718lab\_codex\.codex-task-temp\storage-113-ledger-rust-target'; cargo test -p codex-core preview_hash_invalidates_on_epoch_owner_or_content_change --locked -j1; cargo check -p codex-core --lib --locked -j1; Pop-Location -$env:PYTEST_DISABLE_PLUGIN_AUTOLOAD='1'; Push-Location 'G:\2718lab\_codex\.codex-task-temp\devkit-1.1.2-recovery\mcp-tools'; python -m pytest tests/test_storage_ledger.py -q -o cache_dir=G:\2718lab\_codex\.codex-task-temp\storage-113-ledger-pytest; python -m py_compile devkit_runtime/storage_ledger.py devkit_runtime/host_bridge.py devkit_runtime/host_session.py server.py; Pop-Location -``` - -Expected: focused Rust and Python tests pass, both compile gates are silent, -and only the named task-local target/cache roots are touched. - -- [ ] **Step 6: Commit preview/apply and bridge integration.** +`ApprovedRoot`, `SingleComponent` and handles have private constructors. +`RootTransactionGuard` retains the opened root identity and actual exclusive +OS lock for one reload/recheck/transition/persist transaction. Release it +after commit so another Host can transact against the new epoch; never keep +one Host's cached state authoritative after releasing it. +`NamespaceMutationFence` excludes admissions +and namespace writers for the exact owned subtree until final observation/ +commit. Neither type is serde, a boolean, a deadline, or a caller token. + +On Windows, acquire a machine-wide named mutex keyed from the opened local +volume/file identity before creating lock/ledger files; validate ownership, +abandonment and collisions. Retain no-follow directory handles and compare +volume/file IDs; reject reparse points. Child deletion uses verified handles +and the platform disposition API, not `remove_dir_all` on a reconstructed +string. Sharing/ACL rules and the process fence must exclude rename/replacement +by writers during the operation. The guard must also cover every Host process +using that root, not merely one Rust mutex. + +On Unix, use an actual exclusive OS lock on an already-open approved +directory where supported, and directory-relative no-follow opens plus +`fstat` identity checks. Relative unlink operations still require a real +namespace-mutation fence; `openat` or a final string comparison alone does +not close a leaf-replacement race. If the backend cannot prove that fence, +apply is unavailable. Do not claim a portable secure delete from a trait +stub. Remote/shared filesystems without a cross-host locking guarantee are +unsupported for this local-root implementation. + +Keep lock order explicit: OS root transaction guard, then this runtime's +transaction mutex, then the specific namespace fence. Reload after taking +the OS guard, not before. Await actual writer shutdown +**before** the transaction and validate the Host-owned terminal evidence +nonblockingly inside it, as required by `HostStorageTerminationEvidence`. +Do not hold the OS/accounting locks while waiting for child exit or call back +into the firewall from a proof provider. Busy/unavailable/abandoned/unknown +results fail closed and never trigger takeover by TTL. + +## P2-base implementation tasks + +### Task 1: Own bootstrap and control writes before opening a ledger + +**Files:** `storage_fs.rs`, `storage_ledger.rs`, `storage_service.rs`, `mod.rs`. + +- [ ] Open/validate the configured root without writing, obtain its OS + transaction guard, reload any current root snapshot, verify the unchanged + trusted base policy and measure available capacity. An existing valid + ledger is joined transactionally, never overwritten with an empty state. + No new user fields are needed or allowed: all policy values remain pending + the user's root-plus-eight choice; absent policy is not zero/unlimited. +- [ ] Calculate an internal bootstrap reservation before the first metadata + write. Its bound includes the encoded empty snapshot/header and all files + simultaneously alive during creation/replacement: old snapshot, stage, + bounded pending journal/receipt, and any on-disk lock representation. + Charge directories/files according to the same deterministic counting rule + used by observations; do not hide stage/lock overhead. +- [ ] Use a bounded counting serializer and existing per-record protocol + limits to prove the encoded maximum. Account for maximum numeric widths, + current collection sizes and the proposed records. Stop encoding/scanning + at the remaining admitted bound. Reject overflow or an unprovable bound; + do not invent a record count, unlimited log or free bootstrap exception. +- [ ] Before a ledger exists, the live root transaction guard owns this startup + reservation in memory and excludes concurrent root transactions. Write the first durable + snapshot containing that same control reservation, then release the guard + and publish the service. For an existing ledger reserve only this operation's + required delta against the reloaded shared state. + This transfers ownership of one reservation; it does not charge twice. + A crash leaving only stage/unknown files enters recovery, never fresh-empty + initialization over those files. +- [ ] Before every growth/replacement, calculate peak physical coexistence and + reserve the positive delta from existing global bytes/files limits while + checking the free-space floor. Only then create/write. Shrink/release a + control footprint only after durable commit and verified old/stage removal. + Already-accounted file extents are not summed again as parent and child. +- [ ] The bootstrap allocation belongs to the root lifetime, not the first + Host process. Record its last mutator for audit, but do not release it on + that Host's exit or reserve it again when another Host opens the ledger. +- [ ] Provide `reserve_control_once`, `replace_control_payload` and + `settle_control` on the same service transaction, taking verified Host + queue provenance and bounded encoded payloads, not caller paths/owners. + Queue control lives under the private `approved_root/control` namespace, + outside `generated//members` and outside the common Cargo cache. +- [ ] A queue allocation survives initial member release and all intermediate + waves. It settles only after the verified queue is exhausted/cancelled, + no queue writer remains and its final payload/postcheck is durable. + It never holds a Cargo family lease. Initial and same-key successor groups + can therefore settle/reacquire their family without destroying live queue + metadata. Queue suballocations transfer reserved extents; they are not a + second global charge on top of a parent footprint. +- [ ] Keep retained control metadata and replay tombstones bounded. If a next + snapshot cannot fit, reject the mutation before write. Do not discard + tombstones or live queue records to make it fit. Terminal metadata retirement + needs its own proven lifecycle/epoch rule; no age-only trimming. + +### Task 2: Make kernel transitions one durable transaction + +**Files:** `storage_ledger.rs`, `storage_firewall.rs`, `storage_service.rs`; +later attachment points in `registry.rs`, `codex_adapter.rs`, `coordinator.rs`. + +- [ ] Replace the private kernel state mutex with the reloaded ledger transaction state. + Preserve `create_group`, `reserve_member_once`, `seal_group`, + `consume_member_once`, `revoke_unused_member` and + `release_member_after_postcheck` semantics. Internal transition evaluators + take transaction state rather than reacquiring a separate lock. +- [ ] Under a newly acquired root guard/transaction lock: reload the committed + snapshot, verify expected epoch, current and other recorded owners/policy, + exact authority and namespace identities; compute the + next family/group/member/control snapshot and peak metadata reservation. + Run all fallible validation/checked arithmetic before granting an action. +- [ ] Persist one bounded next snapshot with a unique owned stage handle, + sync file contents, atomically replace through the held parent handles, and + complete the platform durability barrier. Reuse existing + `registry.rs::persist_json_file` replacement semantics as a reference, not + as proof that its path-based helper already supplies the required fence. + On Unix also sync the parent; on Windows use the existing durable replacement + behavior plus identity-safe handles. +- [ ] Return a grant and refresh any read-only cache only after commit. + A serialization/write failure before replacement keeps the old state and + releases only proven-unused operation space. An uncertain replacement/ + durability outcome freezes the service in recovery; do not report the old + or next epoch as certainly committed. Leftover stage files remain accounted + and protected until their recorded identity can be verified. +- [ ] Persist Reserved admission before returning its decision and persist + Consumed before **any** writer preparation/materialization is permitted. + Successful transport is not another reservation. `consume_batch` and the + native selected-wave path use this same service and original frozen batch + hashes; no synthetic bridge exchange is needed. +- [ ] Only an unconsumed Reserved grant may use `revoke_unused_member`. + Consumed prepare failure, cancellation, timeout and terminal ACK require + actual never-launched or all-writing-descendants-stopped evidence plus + postcheck before settlement. Do not unconditionally release on prepare error. + Preserve ownership if an adapter runtime object is removed during recovery. +- [ ] Heartbeats/observations are bounded state updates, not authority renewal. + They cannot reactivate Released members, expand budgets, or erase shared + observed usage. A root snapshot contains the exact effect once. + +### Task 3: Recover epochs/owners without inventing live authority + +**Files:** `storage_ledger.rs`, `storage_fs.rs`, `storage_service.rs` and queue +recovery seams in `registry.rs`; use Plan 1 process-lifecycle evidence. + +- [ ] Acquire the real root transaction guard and reload before advancing state. + Validate exact snapshot-v2 structure, checksum/predecessor epoch and pending + operation evidence under bounded reads. Unknown/malformed/regressing state + is protected; a self-consistent hash alone is not trusted owner authority. +- [ ] Register the starting Host's real instance/PID/creation identity and + owner epoch through a durable transaction before its first grant. A restart + generation applies only to the corresponding prior Host instance, never + the entire root. Preserve other live Host owners and all their member/ + control reservations in the shared global totals. Starting this Host does + not migrate, reclaim or relabel another active Host's members. +- [ ] Reconcile a prior owner only using actual process/Job handles and full + ownership/path evidence. PID equality, PID absence, heartbeat expiry and + OS mutex abandonment do not prove old descendants stopped. Uncertain old + owners become RecoveryPending while their reservation/allowance/observation + remains charged; live other owners remain live. Do not resume execution + solely from stored hashes or reconstruct Sent/native authority from a string. +- [ ] Await the real owned-process fence outside the transaction; inside it + verify the bound terminal evidence and remeasure under the namespace fence. + Only a committed recovery settlement can release capacity. Unknown owner, + missing evidence, replaced root, failed stat or unreconciled journal keeps + admission/apply blocked. Quarantine is a logical protected state, not an + automatic move/delete operation. An abandoned guard or unmatched stage + triggers bounded pending-operation reconciliation, not automatic deletion + or a guess about which Host owned the stage. +- [ ] Treat legacy flat lease-v1 as read-only recovery input. It cannot + reconstruct exact sealed member sets, shared allowance or control ownership + unambiguously, so do not auto-migrate it into active snapshot-v2 or duplicate + one family lease per task. Unknown versions fail closed. Existing data and + raw legacy evidence remain untouched pending an explicit audited recovery. + A missing ledger never authorizes adoption/deletion of unknown artifacts. +- [ ] Report P2-base ready only when the shared ledger, bounded bootstrap/queue + allocations and fail-closed restart gate are wired and compiled. This lets + Plan 1 Task 6 proceed; it does not enable cleanup apply. + +## P2-apply: classification, preview and locked deletion + +### Task 4: Classify only proven disposable, unowned generated objects + +**Files:** `storage_cleanup.rs`, `storage_fs.rs`, `storage_ledger.rs`. + +- [ ] Build candidates from registered family/member generated roots only. + Eligibility requires verified producer/classification evidence, released + ownership, terminal/postcheck proof, no live group/member/control reference, + stable no-follow identity and a bounded content manifest. Unreleased family + data, unknown files, dirty output, source/worktree data, sessions, metadata + and live queues are protected. Size/age/name may order already-eligible + candidates but never make an object eligible. +- [ ] Shared Cargo cleanup targets the family cache once, never one task's + duplicate lease reference. Member scratch/output is a separate bounded + object and is not disposable merely because its phase is Released. + Non-Cargo output can contain valuable results; require explicit classification + and requested cleanup scope. Do not infer disposability from the four + artifact-kind names alone. +- [ ] Preview is read-only and path-free. Under a consistent ledger view, + bind root/policy/epoch, exact candidate set, last ownership/evidence, + content/identity manifest and finite limits into the candidate hash. + Counts and hashes are observations, not deletion capabilities; caller + recomputation does not mint Host classification or a handle fence. +- [ ] Retain the planned bounded public apply payload: + `{candidate_hash, policy_hash, ledger_epoch, batch_limit}`, with batch_limit + 1 through 16 and no path/owner fields, inside the authenticated operation + envelope. Explicit user authorization for that destructive scope is still + required. Status/preview are read-only; no automatic pressure cleanup. + +### Task 5: Acquire fences first, then recheck and journal apply + +**Files:** `storage_cleanup.rs`, `storage_fs.rs`, `storage_ledger.rs`; +only afterward the mapped rmcp-client and DevKit projection files. + +- [ ] Resolve the Host-held preview reference, acquire the root transaction + lock and real namespace-mutation fence, then freshly reopen/remeasure the + exact candidate set through retained no-follow handles. Recheck owner/ + member/control references, policy, expected epoch, classification, identity, + content manifest and finite bounds **inside** that fenced interval. + Never compute a new preview first and acquire the writer fence afterward. +- [ ] Any discrepancy returns `STORAGE_CANDIDATE_STALE` or the precise + protected code before deletion. Unknown/active/dirty/recovery/control data + remains protected. No path string, TTL, `owner_state = none` label or a + successful hash comparison substitutes for this fenced recheck. +- [ ] Reserve the bounded apply journal/receipt peak before writing it, using + the same control accounting. Persist an Applying operation for the exact + selected identities/expected epoch before the first delete. The root lock + prevents another admission during this transition; retain the namespace + fence through deletion, post-stat and final ledger commit. +- [ ] Delete only via `OwnedFilesystem::remove_verified_tree` with finite + bytes/files/depth/time limits, no links/mount traversal, stable opened parent + identities and the supported platform's actual mutation exclusion. + Reject multiply linked or unowned entries when ownership cannot be proved. + No generic `remove_dir_all` fallback or wildcard deletion is permitted. +- [ ] Postcheck the exact result and update retained family/control observation + once before committing the bounded receipt. Deletion does not release an + active lease; only already-settled eligible data enters apply. On partial + removal or an uncertain final commit, persist/protect the pending operation, + return `STORAGE_APPLY_INCOMPLETE`, stop later items and require recovery. + Never roll back by claiming removed bytes/files still exist or attempt an + unrelated cleanup. Crash recovery does not auto-continue deletion. +- [ ] Expose only stable codes, hashes, counts and receipt identity through + `storage_status`/`storage_preview`/`storage_apply`. Keep absolute paths, + owner PID/creation identity and handles private. Stable failures include + `STORAGE_CANDIDATE_STALE`, `STORAGE_PROTECTED_UNKNOWN`, + `STORAGE_PROTECTED_ACTIVE`, `STORAGE_PROTECTED_DIRTY`, + `STORAGE_APPLY_INCOMPLETE` and existing admission/recovery codes. + Internal snapshot-v2 is not a change to public admission-v1. + +## Compile-first verification and commits + +Keep only two new core boundary cases; preserve existing kernel vectors. +Use temporary, owned fixture roots and injected filesystem/process backends, +never the configured live generated root. + +1. `shared_family_control_transaction_is_single_charge`: two distinct members + share one family lease; repeating one intent is not a new reservation. + Persist/reopen a protected snapshot with allowance and retained observation + intact. Alternate transactions through two runtime service fixtures and + verify each reloads the latest epoch and retains the other live owner's + reservations. Initial settlement keeps a live independent queue; a same-key + successor can reserve without inheriting its predecessor's active lease. + Inject commit failure and verify no uncommitted grant is returned and + retained control/stage capacity is not silently freed. +2. `apply_rechecks_identity_under_fence_and_retains_failed_shutdown`: a + Consumed member lacking real terminal proof cannot release or become a + candidate. Change candidate identity/epoch between preview and fence; + apply rejects before the delete primitive. The fixture backend verifies + that the successful recheck/deletion interval actually holds both guards. + +These are acceptance cases to implement, not claims that the test functions +already exist. Keep one small Python assertion rejecting paths/unknown fields +and unbounded apply batches if the bridge projection changes. Do not recreate +old missing-module RED failures or introduce a broad test matrix. + +After a changed slice compiles, run the relevant case once, from Host +`codex-rs`: ```powershell -Push-Location 'G:\2718lab\_codex\.codex-task-temp\devkit-1.1.2-recovery'; git add mcp-tools/devkit_runtime/storage_ledger.py mcp-tools/devkit_runtime/host_bridge.py mcp-tools/devkit_runtime/host_session.py mcp-tools/devkit_runtime/tool_metadata.py mcp-tools/server.py mcp-tools/tests/test_storage_ledger.py mcp-tools/tests/test_mcp_contract.py; git commit -m 'feat: add owned storage preview and apply'; Pop-Location -Push-Location 'G:\2718lab\_codex\.codex-task-temp\codex-host-mcp-fix-recovery2'; git add codex-rs/core/src/fast_lane_host_dispatch/storage_ledger.rs codex-rs/core/src/fast_lane_host_dispatch/storage_ledger_tests.rs codex-rs/rmcp-client/src/inherited_host_bridge_protocol.rs codex-rs/rmcp-client/src/inherited_host_bridge_protocol/envelope.rs codex-rs/rmcp-client/src/inherited_host_bridge_protocol/session.rs; git commit -m 'feat: add owned storage preview and apply'; Pop-Location +$env:CARGO_TARGET_DIR='G:\2718lab\_codex\.codex-task-temp\codex-host-mcp-fix-recovery2\codex-rs\target' +$env:CARGO_INCREMENTAL='0' +cargo check -p codex-rmcp-client -p codex-core --lib --locked -j1 +cargo test -p codex-core shared_family_control_transaction_is_single_charge --locked -j1 +cargo test -p codex-core apply_rechecks_identity_under_fence_and_retains_failed_shutdown --locked -j1 ``` -## Plan 2 acceptance gate and handoff - -- [ ] Re-read the design sections “Task Storage Lease Ledger”, “重启恢复”, “Preview、候选哈希、复核与 Apply”, and “稳定错误”; map every listed field/code to a task above. -- [ ] Run `git diff --check` in both worktrees and verify only the mapped files changed. -- [ ] Run DevKit `py_compile` and Host `cargo check -p codex-core --lib --locked -j1` with the one named task target; zero warnings are required before any package build. -- [ ] Record receipts for reserve, heartbeat, release, restart recovery, candidate stale, protected candidate, successful one-item generated apply, and partial apply. Each receipt must include ledger epoch and receipt hash. -- [ ] Verify a missing/legacy ledger migrates to recovery protection, a failed atomic write leaves the previous snapshot, pressure blocks new reservations, and no operation kills another process or scans outside registered generated roots. -- [ ] Do not implement GitHub source deletion, ordinary/active session deletion, CAS dedupe, compression, or remote synchronization. Plan 3 consumes the ledger's protected classifications and apply fence. +Stop before probes if compilation fails; no repeated full-suite runs. Compile +any changed runtime caller separately with the same target/settings rather +than treating the two-crate gate as coverage of an edited app-server. +For changed Python modules, run only their `py_compile` and the selected +exact request assertion after the Rust gate. No package build or live cleanup +is needed for these document/base slices. + +- [ ] Commit P2-base state/control work separately from later cleanup/wire + changes. Use explicit named-file staging after `git diff --check`; do not + stage other workers' changes or commit an uncompiled layer as accepted. +- [ ] Record which OS handle/descendant fence has real implementation and + verification. An unsupported backend stays unavailable; a mock guard or + compile success is not production deletion approval. + +## Plan 2 acceptance gates and handoff + +- [ ] One service per runtime enters the same cross-process root transaction: + OS fence, reload/recheck epoch/owner state, transition, durable commit. + Other live Host owners remain active and globally counted; no independent + per-Host authoritative counter cache exists. Same-key members share the lease, and no path performs + firewall admission followed by independent ledger reservation. +- [ ] Root-plus-eight trusted configuration remains unchanged and numerically + pending the user. Bootstrap/queue/snapshot/stage/journal bytes/files are + bounded and charged before writes from the existing global limits; + unprovable capacity/bounds fail closed without creating metadata. +- [ ] P2-base enables Plan 1's durable refill dependency independently of + cleanup apply: live queues survive initial member settlement without + occupying its Cargo family lease or releasable member paths. +- [ ] Real cross-process root exclusion, epoch/creation identity, recovery + protection and no-follow handle fences exist. TTL/strings/PID equality + cannot authorize takeover, release or deletion. +- [ ] Released tombstones and retained family observations survive restart; + legacy/missing/ambiguous state never grants fresh ownership over old data. +- [ ] Consumed prepare failure and terminal settlement require actual process + proof and postcheck. Uncertain commit/shutdown/stat leaves ownership intact. +- [ ] Cleanup requires explicit scope authorization and fresh locked/fenced + eligibility/identity checks, with bounded journal/delete/postcheck. + Record a fixture stale-candidate/no-delete result before any separately + authorized live apply; this plan edit itself authorizes none. +- [ ] No generated/source/session/queue data was deleted merely to complete a + test, satisfy pressure, or repair a snapshot. Plan 3 receives protected + classifications and the real fence, not permission for broader deletion. From 9131ac44d36c797b67b18a4b9b4c4fcbd913a51a Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E5=93=80=E6=B4=9B=E8=8A=99?= Date: Sun, 30 Aug 2026 21:20:35 +0800 Subject: [PATCH 26/39] docs: plan protected storage broker --- .../plans/2026-08-30-storage-broker-1.1.3.md | 512 ++++++++++++++++++ 1 file changed, 512 insertions(+) create mode 100644 docs/superpowers/plans/2026-08-30-storage-broker-1.1.3.md diff --git a/docs/superpowers/plans/2026-08-30-storage-broker-1.1.3.md b/docs/superpowers/plans/2026-08-30-storage-broker-1.1.3.md new file mode 100644 index 0000000..ab44891 --- /dev/null +++ b/docs/superpowers/plans/2026-08-30-storage-broker-1.1.3.md @@ -0,0 +1,512 @@ +# Windows Protected Storage Broker 1.1.3 Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Build a Windows-only, explicitly provisioned machine service that is the sole writer for the 1.1.3 protected storage root, while the Host becomes an authenticated client that fails closed whenever the broker is disabled, absent, untrusted, mismatched, or recovering. + +**Architecture:** Follow `RECOMMEND_C` and treat `NEW_BROKER_REQUIRED` as resolved only by a separate `codex-storage-broker-windows` crate. A LocalSystem SCM service with an unrestricted Service SID owns the protected root and all ledger/control mutations; the Host verifies the connected service process and root identity and never falls back to its current in-process writer. Provisioning is a separate, explicit elevated action, never a side effect of build, package install, Host startup, configuration load, or this plan. + +**Tech Stack:** Rust 2024, Cargo workspace, `windows-sys` 0.52 Win32 APIs, length-prefixed binary IPC over a local named pipe, SHA-256, existing handle/no-reparse storage primitives, Bazel `codex_rust_crate`, PowerShell acceptance probes, GitHub Windows release packaging. + +--- + +## Status, repositories, and non-goals + +- Host repository: `G:\2718lab\_codex\.codex-task-temp\codex-host-mcp-fix-recovery2`. +- DevKit repository: `G:\2718lab\_codex\.codex-task-temp\devkit-1.1.2-recovery`. +- Host baseline commits `d41c566`, `11e9330`, `9ca17fd`, and `f80ab0b` have already passed their separate specification and quality reviews. Preserve their fail-closed bootstrap behavior while moving write authority out of `codex-core`. +- Existing unrelated Host edits in `codex-rs/core/src/fast_lane_host_dispatch/codex_adapter.rs` and `coordinator.rs` are outside this plan until the later Plan 1 lifecycle slice explicitly owns them. Never stage them accidentally. +- `FastLaneStorageConfigToml` remains the exact approved root plus eight non-zero policy values. Missing configuration is the default-disabled state. Present configuration with no valid provisioning is `STORAGE_BROKER_UNPROVISIONED`, not permission to create a local ledger. +- This plan does not run provisioning, create a service, edit SCM, change an ACL, create the production root, activate live configuration, clean generated data, or delete a legacy/staged object. +- This plan does not implement generated cleanup. Cleanup stays in Plan 2 `P2-apply` after the broker, Plan 1 lifecycle proof, preview, and deletion fences exist. + +## Existing code to reuse deliberately + +| Existing path and symbol | Reuse boundary | +| --- | --- | +| `codex-rs/core/src/fast_lane_host_dispatch/storage_fs.rs::{ApprovedRoot, PlatformRootOwnershipProvider, write_stage, publish_stage, StageDestination}` | Move the handle-based root writer into the broker crate. `ReplaceVerified` must become a real broker-only operation; it must remain fail closed until destination identity, parent identity, stage identity, and durable replacement are all verified. | +| `codex-rs/core/src/fast_lane_host_dispatch/storage_ledger.rs::{RootStorageLedger, reserve_control_once, replace_control_payload, settle_control}` | Move the reviewed bootstrap transaction and the three control API shapes into broker ownership. Do not leave a second compilable writer in `codex-core`. | +| `codex-rs/core/src/fast_lane_host_dispatch/storage_service.rs::{ValidatedStorageConfig, HostStorageService}` | Keep trusted config provenance and the one-service-per-runtime object; replace local ledger construction with one cached, authenticated `BrokerClient`. | +| `codex-rs/rmcp-client/src/inherited_host_bridge.rs::create_platform_bridge` | Copy the local-only pipe, protected DACL, `FILE_FLAG_FIRST_PIPE_INSTANCE`, PID, and process-creation binding patterns; do not reuse its same-user bearer authority. | +| `codex-rs/rmcp-client/src/stdio_server_launcher.rs` | Reuse the `GetNamedPipeServerProcessId` plus `OpenProcess`/`GetProcessTimes` anti-PID-reuse sequence. | +| `codex-rs/windows-sandbox-rs/src/elevated/runner_pipe.rs::{create_named_pipe, connect_pipe}` | Reuse `PIPE_REJECT_REMOTE_CLIENTS`, client PID discovery, and explicit SDDL construction patterns. Broker authentication is stricter than the runner helper. | +| `codex-rs/windows-sandbox-rs/src/bin/setup_main/win/no_reparse_dir.rs::open_or_create_no_reparse` | Port the `OBJ_DONT_REPARSE` directory-open pattern into broker provisioning/root code; do not add a dependency from the broker to the large sandbox crate. | +| `codex-rs/windows-sandbox-rs/src/acl.rs` | Follow its `SetNamedSecurityInfoW`, protected DACL, SID conversion, and post-write ACL verification patterns. | +| `.github/workflows/rust-release-windows.yml`, `.github/scripts/build-codex-package-archive.sh`, `scripts/codex_package/{targets,cargo,cli,layout}.py` | Add, sign, stage, archive, and validate the service and provisioner as Windows resources. Packaging them never provisions them. | + +## Locked security and wire contracts + +The implementation must use these names and bounds consistently across tasks: + +```rust +pub const SERVICE_NAME: &str = "2718labStorageBroker"; +pub const SERVICE_ACCOUNT: &str = "LocalSystem"; +pub const PIPE_NAME: &str = r"\\.\pipe\2718lab-storage-broker-v1"; +pub const PROTOCOL_MAGIC: [u8; 8] = *b"2718SB01"; +pub const PROTOCOL_VERSION: u16 = 1; +pub const MAX_CONTROL_PAYLOAD_BYTES: usize = 1024 * 1024; +pub const MAX_FRAME_BODY_BYTES: usize = MAX_CONTROL_PAYLOAD_BYTES + 4096; +pub const MAX_FRAME_BYTES: usize = 16 + MAX_FRAME_BODY_BYTES; +pub const MAX_REQUESTS_PER_CONNECTION: u32 = 64; +pub const MAX_CONCURRENT_CONNECTIONS: usize = 8; +pub const IO_DEADLINE: Duration = Duration::from_secs(5); +pub const BROKER_RELEASE_SEQUENCE: u64 = 1_001_003; +``` + +The 16-byte frame header is exactly magic `[u8; 8]`, protocol `u16` little-endian, opcode `u16` little-endian, and body length `u32` little-endian. Decoding rejects a wrong magic/version, unknown opcode, a body over the cap, truncation, trailing bytes, duplicate fields, a zero identifier, and any non-canonical enum value before allocating the declared body. + +```rust +pub type Digest32 = [u8; 32]; + +#[derive(Clone, Copy, Debug, Eq, PartialEq)] +pub struct ProcessIdentity { + pub pid: u32, + pub creation_time_100ns: u64, + pub image_sha256: Digest32, +} + +#[derive(Clone, Copy, Debug, Eq, PartialEq)] +pub struct RootIdentity { + pub volume_serial_number: u64, + pub file_id: [u8; 16], +} + +#[derive(Clone, Debug, Eq, PartialEq)] +pub struct ControlAuthority { + pub host_owner_id: Digest32, + pub queue_id: Digest32, + pub queue_epoch: NonZeroU64, + pub active_lease_set_hash: Digest32, +} + +#[derive(Clone, Debug, Eq, PartialEq)] +pub enum ControlMutation { + ReserveOnce { + control_id: Digest32, + expected_ledger_epoch: u64, + authority: ControlAuthority, + payload: BoundedControlPayload, + }, + ReplaceVerified { + control_id: Digest32, + expected_ledger_epoch: u64, + expected_payload_sha256: Digest32, + authority: ControlAuthority, + payload: BoundedControlPayload, + }, + Settle { + control_id: Digest32, + expected_ledger_epoch: u64, + expected_payload_sha256: Digest32, + authority: ControlAuthority, + }, +} + +pub struct BrokerRequest { + pub request_id: Digest32, + pub connection_nonce: Digest32, + pub root_identity: RootIdentity, + pub mutation: ControlMutation, +} + +pub struct BrokerReceipt { + pub request_id: Digest32, + pub previous_ledger_epoch: u64, + pub committed_ledger_epoch: u64, + pub snapshot_sha256: Digest32, + pub payload_sha256: Digest32, + pub root_identity: RootIdentity, + pub service_process: ProcessIdentity, +} +``` + +No broker request contains a path, path component, environment-derived directory, owner SID string, service name, byte/file limit, release number, or deletion instruction. The service takes its single root, policy, accepted Host image hashes, release floor, and service binary hash only from its machine-protected active record. + +The Host/client handshake is ordered and mutually bound: + +1. Host loads `%ProgramData%\2718lab\StorageBroker\active-v1.json` through a no-reparse handle and verifies owner `SYSTEM` plus a protected DACL with no non-administrator write ACE. +2. Host opens only `PIPE_NAME` with `SECURITY_SQOS_PRESENT | SECURITY_IDENTIFICATION`; UNC/remote pipe names are not representable by the API. +3. Host obtains the pipe server PID, matches it to `QueryServiceStatusEx(SERVICE_NAME)`, opens the process, records creation time, validates service SID membership, validates the protected version-directory image path and exact manifest SHA-256, then rechecks PID/creation after the handshake. +4. `ServerHello` repeats service PID/creation, broker release sequence, service image hash, root identity, provisioning generation, and both nonces. Every field must match the protected record or live handles. +5. Service obtains the client PID with `GetNamedPipeClientProcessId`, binds its creation time, image path, exact authorized Host SHA-256, and Authenticode publisher, then rechecks PID/creation before each mutation. A merely same-user process is rejected. + +## State machines and fail-closed outcomes + +Root transaction states are: + +```text +Locked -> Reloaded -> PeakReserved -> StageSynced -> ReplaceVerified -> Committed + | | | | | + +----------+-------------+---------------+---------------+-> RecoveryRequired +``` + +- The broker is the only process allowed a write-capable root handle. +- `ReplaceVerified` checks the still-open destination, parent, and stage identities immediately before replacement and verifies the new identity/hash immediately afterward. +- Failure before replacement leaves the old committed snapshot authoritative and retains the deterministic stage as protected recovery input. +- An uncertain replacement or durability barrier enters `RecoveryRequired`; it never guesses old/new state, removes the stage, recreates an empty ledger, or serves another mutation. +- Startup with a stage/journal reconciles only an exact recorded predecessor/next hash and identity. Any ambiguity remains recovery-required for explicit administrator repair. + +Provisioning states are: + +```text +Absent/ActiveOld -> CandidateVerified -> CandidateProtected -> ScmSwitched + -> HealthChecked -> ActiveRecordPublished +``` + +- The active record is published last. Before that publication, Host clients continue trusting only the old record. +- A candidate release lower than the protected release floor is rejected. The same sequence is idempotent only when every binary/root/Host hash is identical. +- Failure before `ScmSwitched` leaves the old service/root/record untouched. Failure after it restores the old SCM image path and restarts the old service; the old version directory and root are retained. +- Candidate/stage artifacts are never age-cleaned. A later explicit `status` or `repair` command reports/reconciles them. + +## Task 1: Scaffold the independent protocol/client crate + +**Files:** +- Create: `codex-rs/storage-broker-windows/Cargo.toml` +- Create: `codex-rs/storage-broker-windows/BUILD.bazel` +- Create: `codex-rs/storage-broker-windows/src/lib.rs` +- Create: `codex-rs/storage-broker-windows/src/protocol.rs` +- Create: `codex-rs/storage-broker-windows/src/error.rs` +- Create: `codex-rs/storage-broker-windows/src/provision_record.rs` +- Modify: `codex-rs/Cargo.toml` +- Modify: `codex-rs/Cargo.lock` +- Modify: `MODULE.bazel.lock` + +- [ ] Add workspace member `storage-broker-windows` and workspace dependency `codex-storage-broker-windows = { path = "storage-broker-windows" }`. The crate exposes protocol/client types on every OS; service/provisioning modules are `cfg(windows)`, and non-Windows `BrokerClient::connect` returns `STORAGE_BROKER_UNSUPPORTED`. +- [ ] Implement the locked header, `BoundedControlPayload::try_from(Vec)`, exact encode/decode functions, and a closed `BrokerErrorCode` enum. Use checked arithmetic before allocation and return only stable codes plus bounded diagnostic text. +- [ ] Implement strict `ActiveProvisionRecordV1` decoding with `serde(deny_unknown_fields)`, a maximum 16 KiB file, one active and at most one rollback Host hash, exact `BROKER_RELEASE_SEQUENCE`, fixed service/pipe names, fixed protected base directories, and `RootIdentity`. Reject environment-supplied Program Files/ProgramData paths; Windows code resolves known folders. +- [ ] Keep public API minimal and path-free: + +```rust +pub fn encode_request(request: &BrokerRequest) -> Result, BrokerError>; +pub fn decode_request(frame: &[u8]) -> Result; +pub fn encode_response(response: &BrokerResponse) -> Result, BrokerError>; +pub fn decode_response(frame: &[u8]) -> Result; +``` + +- [ ] Refresh Bazel lock state because a new workspace crate changes Cargo metadata, even when all external versions already exist. +- [ ] Compile before any probe: + +```powershell +$env:CARGO_TARGET_DIR='G:\2718lab\_codex\.codex-task-temp\codex-host-mcp-fix-recovery2\codex-rs\target' +$env:CARGO_INCREMENTAL='0' +cargo check --locked -p codex-storage-broker-windows --lib -j1 +``` + +Expected: exit `0`; no service, directory, pipe, ACL, or root is created. + +- [ ] Run `git diff --check`, stage only the Task 1 files, and commit: + +```powershell +git commit -m "feat: define bounded storage broker protocol" +``` + +## Task 2: Implement SCM service identity and authenticated local pipe + +**Files:** +- Create: `codex-rs/storage-broker-windows/src/identity.rs` +- Create: `codex-rs/storage-broker-windows/src/pipe.rs` +- Create: `codex-rs/storage-broker-windows/src/service.rs` +- Create: `codex-rs/storage-broker-windows/src/bin/storage_broker_service.rs` +- Modify: `codex-rs/storage-broker-windows/Cargo.toml` +- Modify: `codex-rs/storage-broker-windows/BUILD.bazel` + +- [ ] Add binary `codex-storage-broker-service`. Its only accepted entry is SCM `StartServiceCtrlDispatcherW`; console launch returns `STORAGE_BROKER_SCM_REQUIRED`. Register STOP/SHUTDOWN controls, report exact service states, and drain bounded in-flight requests before stopping. +- [ ] Require LocalSystem plus the unrestricted `NT SERVICE\2718labStorageBroker` SID in the live token. At startup, compare SCM-configured image path, process image handle/hash, protected version directory identity/DACL, release sequence, and active record. Any mismatch reports `SERVICE_STOPPED` with a deterministic service-specific exit code. +- [ ] Create one local byte-mode pipe with `FILE_FLAG_FIRST_PIPE_INSTANCE`, `PIPE_REJECT_REMOTE_CLIENTS`, the fixed name, a protected DACL, eight maximum instances, and five-second overlapped I/O cancellation. Never accept a caller-selected pipe name. +- [ ] Implement both endpoint verifiers around live handles: + +```rust +fn verify_service_endpoint( + pipe: &OwnedHandle, + record: &ActiveProvisionRecordV1, +) -> Result; + +fn verify_host_client( + pipe: &OwnedHandle, + record: &ActiveProvisionRecordV1, +) -> Result; +``` + +The first uses `GetNamedPipeServerProcessId`, `QueryServiceStatusEx`, `OpenProcess`, `GetProcessTimes`, `OpenProcessToken`, `CheckTokenMembership`, `QueryFullProcessImageNameW`, and SHA-256. The second uses `GetNamedPipeClientProcessId`, the same PID/creation/image sequence, exact Host hash allowlisting, and `WinVerifyTrust`. Recheck process creation after hashing and after handshake. +- [ ] Bind requests to two 32-byte OS-random nonces and the verified process/root record. Close the connection on a repeated request ID, request-count overflow, timeout, short write/read, unknown response, or identity drift. +- [ ] Compile the library and service binary only: + +```powershell +cargo check --locked -p codex-storage-broker-windows --lib --bin codex-storage-broker-service -j1 +``` + +Expected: exit `0`; running the binary directly is not part of this task. + +- [ ] Run `git diff --check`, stage only Task 2 files, and commit: + +```powershell +git commit -m "feat: authenticate storage broker pipe endpoints" +``` + +## Task 3: Move the root writer and three control transactions into the broker + +**Files:** +- Create: `codex-rs/storage-broker-windows/src/root_fs.rs` +- Create: `codex-rs/storage-broker-windows/src/ledger.rs` +- Create: `codex-rs/storage-broker-windows/src/ledger_codec.rs` +- Modify: `codex-rs/storage-broker-windows/src/service.rs` +- Modify: `codex-rs/storage-broker-windows/src/lib.rs` +- Source to migrate: `codex-rs/core/src/fast_lane_host_dispatch/storage_fs.rs` +- Source to migrate: `codex-rs/core/src/fast_lane_host_dispatch/storage_ledger.rs` + +- [ ] Port the reviewed no-follow handles, root identity, OS root exclusion, bounded read, stage sync, and snapshot codec into the independent crate. Keep modules below 500 lines by separating filesystem handles, snapshot codec, and transaction logic. +- [ ] Make `BrokerState` own exactly one `RootWriter` for the protected record: + +```rust +pub struct RootWriter { + root: ApprovedRoot, + policy: StoragePolicy, + transaction: Mutex<()>, + recovery: AtomicBool, +} + +impl RootWriter { + pub fn apply( + &self, + client: &VerifiedHostClient, + request: BrokerRequest, + ) -> Result; +} +``` + +- [ ] Implement `ReserveOnce`, `ReplaceVerified`, and `Settle` as one-shot, epoch-checked transactions. Reload under the OS root guard, verify root/policy/owner/control/payload identities, calculate peak bytes/files with checked arithmetic, check configured limits/floor, persist a bounded next snapshot, and return a receipt only after durable commit. +- [ ] Implement `StageDestination::ReplaceVerified` only here. Open the old destination and stage without following reparse points; compare expected old payload hash and file identity; use handle-relative replacement; sync; reopen; compare new identity/hash; retain stage/journal and freeze on uncertainty. +- [ ] Keep the protocol path-free. Map `control_id` deterministically to one private broker-owned component; reject collisions and unexpected entries. The broker never exposes generic create/write/rename/delete RPCs. +- [ ] Startup joins only an exact snapshot-v2 and exact protected root identity. Missing ledger bootstraps with accounted metadata; known stage/journal is reconciled; malformed/legacy/unknown entries produce `STORAGE_LEDGER_RECOVERY_REQUIRED`. No startup branch removes data. +- [ ] Compile before moving Host call sites: + +```powershell +cargo check --locked -p codex-storage-broker-windows --lib --bin codex-storage-broker-service -j1 +``` + +Expected: exit `0`; root mutation is reachable only after verified SCM startup and provisioning. + +- [ ] Run `git diff --check`, stage only Task 3 files, and commit: + +```powershell +git commit -m "feat: make broker the protected root writer" +``` + +## Task 4: Convert Host storage service to a client with no local fallback + +**Files:** +- Modify: `codex-rs/core/Cargo.toml` +- Modify: `codex-rs/core/BUILD.bazel` +- Create: `codex-rs/core/src/fast_lane_host_dispatch/storage_broker.rs` +- Modify: `codex-rs/core/src/fast_lane_host_dispatch/storage_service.rs` +- Modify: `codex-rs/core/src/fast_lane_host_dispatch/mod.rs` +- Delete after migration: `codex-rs/core/src/fast_lane_host_dispatch/storage_fs.rs` +- Delete after migration: `codex-rs/core/src/fast_lane_host_dispatch/storage_ledger.rs` +- Modify: `codex-rs/Cargo.lock` +- Modify: `MODULE.bazel.lock` + +- [ ] Add the broker crate dependency. `HostStorageService` keeps validated root-plus-eight configuration and the in-memory policy evaluator, but replaces `RootStorageLedger` with a cached `Result, BrokerError>` frozen per runtime. +- [ ] Preserve the existing `Arc` sharing from `ThreadManager` through ordinary and delegated sessions. A session cannot construct a broker, select a pipe/root, replace the cached client, or obtain a session-local writer; all Host processes still converge on the service's single root transaction. +- [ ] `HostStorageService::new` performs no writes and does not start/install the service. Connection remains lazy. Missing config returns `STORAGE_POLICY_MISSING`; configured but missing active record/service returns `STORAGE_BROKER_UNPROVISIONED`; identity/protocol/recovery failures retain their exact stable code. +- [ ] Expose the three path-free client calls using the sealed Host provenance type: + +```rust +pub(super) fn reserve_control_once( + &self, + config: &Config, + provenance: &VerifiedControlProvenance, + payload: &Value, +) -> io::Result; + +pub(super) fn replace_control_payload( + &self, + config: &Config, + provenance: &VerifiedControlProvenance, + expected_payload_sha256: Digest32, + payload: &Value, +) -> io::Result; + +pub(super) fn settle_control( + &self, + config: &Config, + provenance: &VerifiedControlProvenance, + expected_payload_sha256: Digest32, +) -> io::Result; +``` + +- [ ] Encode JSON into `BoundedControlPayload` before connecting. Preserve the current unconstructable provenance barrier until Plan 2 `P2-base Task 1` wires verified queue ownership; do not add a public constructor or synthesize owner/queue hashes. +- [ ] Remove both local writer modules from `mod.rs` and the tree after the broker migration is compiled. A search with PowerShell `Select-String` over tracked Rust files must show no remaining Host call to `ApprovedRoot::open`, `RootStorageLedger::open`, `write_stage`, or `publish_stage`. +- [ ] Compile the broker and Host, once: + +```powershell +cargo check --locked -p codex-storage-broker-windows --bins -p codex-core --lib -j1 +``` + +Expected: exit `0`; existing unrelated dirty adapter/coordinator files remain unstaged. + +- [ ] Run `git diff --check`, explicitly stage Task 4 paths, verify `git diff --cached --name-only`, and commit: + +```powershell +git commit -m "refactor: route storage writes through broker" +``` + +## Task 5: Add explicit administrator and enterprise provisioning source + +**Files:** +- Create: `codex-rs/storage-broker-windows/src/provisioning.rs` +- Create: `codex-rs/storage-broker-windows/src/bin/storage_broker_provision.rs` +- Create: `codex-rs/storage-broker-windows/build.rs` +- Create: `codex-rs/storage-broker-windows/codex-storage-broker-provision.manifest` +- Modify: `codex-rs/storage-broker-windows/Cargo.toml` +- Modify: `codex-rs/storage-broker-windows/BUILD.bazel` + +- [ ] Add `codex-storage-broker-provision` with only `status`, `provision`, and `repair-status`. Embed `requireAdministrator`; still verify an elevated administrator token at runtime. Do not add auto-provision, uninstall, cleanup, root-reset, force-downgrade, arbitrary service-binary, or arbitrary pipe options. +- [ ] Require one explicit mode: + +```text +provision --mode administrator --root --host-binary --confirm-root-id +provision --mode enterprise-managed +status +repair-status +``` + +Administrator mode validates the exact displayed root identity confirmation. Enterprise mode reads only `HKLM\SOFTWARE\Policies\2718lab\StorageBroker` values `RootPath`, `AuthorizedHostPath`, and `ConfirmedRootId`; it accepts no path flags. +- [ ] Locate the service binary only as the fixed sibling `codex-storage-broker-service.exe`. Verify its Authenticode publisher matches the provisioner, its embedded release sequence equals `BROKER_RELEASE_SEQUENCE`, and its SHA-256 is stable across copy. Reject an unsigned, user-replaced, reparse-backed, alternate-name, or lower-sequence binary. +- [ ] Resolve Program Files and ProgramData through Known Folder APIs. Install the service binary under `%ProgramFiles%\2718lab\StorageBroker\versions\00000000001001003\`; store `active-v1.json`, `release-floor-v1.json`, and deterministic candidate status under `%ProgramData%\2718lab\StorageBroker\`. +- [ ] Apply and then read back protected DACLs. Version/state locations are owned by SYSTEM; SYSTEM and Administrators have full control, the service SID has required read/execute, ordinary users have read-only access only to the active record/service image needed for Host verification, and no ordinary-user write ACE exists. The root grants full control only to SYSTEM, Administrators, and the service SID and is opened with no-reparse semantics. +- [ ] Create/update SCM with `CreateServiceW`/`ChangeServiceConfigW`, LocalSystem, fixed service name, fixed protected image path, automatic start, and `ChangeServiceConfig2W(SERVICE_CONFIG_SERVICE_SID_INFO, SERVICE_SID_TYPE_UNRESTRICTED)`. Read back every property before switching. +- [ ] Perform the locked provisioning state machine. Health check the candidate service as an elevated maintenance client, publish the active record last with atomic replace, and update the release floor monotonically. On failure restore the old SCM image path/start state and retain the old root/version/record; report candidate evidence without deleting it. +- [ ] `status` is read-only and reports active/candidate versions, hashes, SCM PID/state, service SID type, root identity, DACL verdict, and recovery state. `repair-status` only reconciles/verifies status metadata; it never repairs ledger contents or deletes stages. +- [ ] Compile only; do not run the provisioner: + +```powershell +cargo check --locked -p codex-storage-broker-windows --bins -j1 +``` + +Expected: exit `0`; `Get-Service 2718labStorageBroker -ErrorAction SilentlyContinue` is unchanged before and after compilation. + +- [ ] Run `git diff --check`, stage only Task 5 files, and commit: + +```powershell +git commit -m "feat: add explicit storage broker provisioning" +``` + +## Task 6: Wire Cargo, Bazel, signing, and package manifests + +**Files:** +- Modify: `codex-rs/storage-broker-windows/BUILD.bazel` +- Modify: `codex-rs/Cargo.toml` +- Modify: `codex-rs/Cargo.lock` +- Modify: `MODULE.bazel.lock` +- Modify: `.github/workflows/rust-release-windows.yml` +- Modify: `.github/scripts/build-codex-package-archive.sh` +- Modify: `scripts/codex_package/targets.py` +- Modify: `scripts/codex_package/cargo.py` +- Modify: `scripts/codex_package/cli.py` +- Modify: `scripts/codex_package/layout.py` +- Modify: `scripts/codex_package/test_cargo.py` +- Modify: `scripts/codex_package/test_layout.py` + +- [ ] Make `codex_rust_crate` expose library, service, and provisioner targets compatible only with Windows binaries. Mirror the existing sandbox setup manifest resource handling so Cargo and both Bazel Windows ABIs embed the elevation manifest correctly. +- [ ] Add both broker executables to `WINDOWS_BINARIES`, helper build/stage/PDB/sign/verify/symbol lists, and Windows package resource inputs. Do not add them to non-Windows target builds. +- [ ] Extend package types with explicit fields `codex_storage_broker_service_bin` and `codex_storage_broker_provision_bin`; add matching CLI/archive-script flags; source-build them for Windows; copy them to `codex-resources/`; require both during Windows package validation. +- [ ] Preserve the package/install boundary: archive creation only copies signed files. It does not invoke the provisioner, create Program Files/ProgramData directories, call SCM, edit registry, create the root, or change ACLs. +- [ ] Update only the two existing package tests to assert the exact Windows resource list and non-Windows rejection. Do not add a broad packaging matrix. +- [ ] Refresh lock state and compile before focused package assertions: + +```powershell +just bazel-lock-update +cargo check --locked -p codex-storage-broker-windows --bins -p codex-core --lib -j1 +python -m unittest scripts.codex_package.test_cargo scripts.codex_package.test_layout +bazel build --platforms=//:local_windows_msvc //codex-rs/storage-broker-windows:codex-storage-broker-service //codex-rs/storage-broker-windows:codex-storage-broker-provision +``` + +Expected: all four commands exit `0`; Python assertions confirm package inclusion only, not provisioning. + +- [ ] Run `git diff --check`, explicitly stage Task 6 files, inspect the staged list, and commit: + +```powershell +git commit -m "build: package signed storage broker binaries" +``` + +## Task 7: Add exactly two boundary probes and close the dependency handoff + +**Files:** +- Create: `codex-rs/storage-broker-windows/src/protocol_identity_tests.rs` +- Create: `codex-rs/storage-broker-windows/src/root_writer_tests.rs` +- Create: `codex-rs/core/src/fast_lane_host_dispatch/storage_broker_tests.rs` +- Modify: `codex-rs/storage-broker-windows/src/lib.rs` +- Modify: `codex-rs/core/src/fast_lane_host_dispatch/mod.rs` +- Modify: `docs/superpowers/plans/2026-08-29-owned-cleanup-1.1.3.md` in the DevKit repository +- Modify: `docs/superpowers/plans/2026-08-29-storage-firewall-1.1.3.md` in the DevKit repository + +- [ ] Implement Probe 1, `unprovisioned_or_spoofed_broker_never_writes`, as one Windows-focused boundary case. It covers: no active record/service; a same-user fake pipe server; wrong service PID/creation; missing Service SID; wrong service binary hash/root identity; oversized/unknown/path-bearing frames. Assert the exact fail-closed code and an unchanged owned fixture root inventory. +- [ ] Implement Probe 2, `uncertain_replace_retains_old_snapshot_and_stage`, with an injected broker filesystem backend. Fail once after stage sync and once at replacement verification; change destination identity before `ReplaceVerified`; assert the old snapshot remains authoritative when known, the stage stays present/accounted, reopening enters recovery when outcome is uncertain, and no delete primitive is called. +- [ ] Keep test seams private and capability-shaped. Do not expose a public fake client, fake identity constructor, path mutation API, or production bypass. Compare whole receipts/snapshots rather than field-by-field assertions. +- [ ] Amend the two existing DevKit plans with this exact dependency order: + +```text +storage broker Tasks 1-6 + -> Plan 2 P2-base Task 1 control transactions + -> Plan 1 Task 6 durable refill/lifecycle wiring + -> Plan 2 remaining P2-base owner recovery + -> Plan 2 P2-apply preview and generated cleanup +``` + +State explicitly that broker completion does not authorize cleanup or live activation. Move Plan 2's root-writer file ownership from `codex-core` to `codex-storage-broker-windows`; keep Host as authenticated client. +- [ ] Compile once, then run only the two named probes: + +```powershell +cargo check --locked -p codex-storage-broker-windows --bins -p codex-core --lib -j1 +cargo test --locked -p codex-core unprovisioned_or_spoofed_broker_never_writes -j1 +cargo test --locked -p codex-storage-broker-windows uncertain_replace_retains_old_snapshot_and_stage -j1 +``` + +Expected: compile exit `0`; each command reports one selected passing test. Do not run a workspace/full suite in this slice. + +- [ ] Perform installed-but-unprovisioned acceptance on a disposable package directory, without elevation: + +```powershell +$packageRoot='G:\2718lab\_codex\.codex-task-temp\storage-broker-unprovisioned-package' +python scripts/build_codex_package.py --target x86_64-pc-windows-msvc --variant codex --cargo-profile dev --package-dir $packageRoot --force +Test-Path -LiteralPath (Join-Path $packageRoot 'codex-resources\codex-storage-broker-service.exe') +Test-Path -LiteralPath (Join-Path $packageRoot 'codex-resources\codex-storage-broker-provision.exe') +Get-Service 2718labStorageBroker -ErrorAction SilentlyContinue +``` + +Expected: both `Test-Path` calls are `True`; packaging exits `0`; no new service exists, no active machine record/root is created, and the focused Host probe returns `STORAGE_BROKER_UNPROVISIONED` without writing its fixture root. + +- [ ] Run `just fmt` once after all Rust edits. Do not rerun probes after formatting per repository guidance. Run `git diff --check`, stage only Task 7-owned paths, inspect the staged list, and commit Host test work and DevKit plan amendments in their respective repositories: + +```powershell +git commit -m "test: prove storage broker fail-closed boundaries" +git commit -m "docs: gate storage lifecycle on protected broker" +``` + +## Final acceptance gates + +- [ ] `codex-storage-broker-windows` is an independent crate and the only compiled owner of write-capable protected-root handles. +- [ ] The SCM service runs as LocalSystem with the exact unrestricted Service SID and fixed local-only pipe; Host and service bind PID plus creation time and exact signed image hashes in both directions. +- [ ] Host verifies protected active record, SCM PID, service token SID, service image/hash, handshake nonces, provisioning generation, and root identity before accepting a receipt. +- [ ] No same-user process, user-writable broker binary, remote pipe, caller path, alternate root, or arbitrary opcode can reach the writer. +- [ ] Multiple ordinary/delegated sessions share one Host client, and multiple Host processes serialize through the broker's one OS-fenced ledger; session identity never becomes root authority. +- [ ] Protocol allocation, request count, payload, concurrency, and I/O time are hard-capped before use. +- [ ] The broker owns `ReserveOnce`, `ReplaceVerified`, and `Settle`; receipt publication follows durable commit; uncertain state freezes and protects stage/recovery data. +- [ ] Missing configuration is disabled. Installed but unprovisioned, unsupported OS, service absent, signature mismatch, downgrade, root mismatch, or recovery state fails closed with no local fallback. +- [ ] Provisioning occurs only through an explicitly elevated administrator command or the fixed enterprise policy registry source. Build, package, install, Host startup, and config load never invoke it. +- [ ] Upgrades reject rollback/downgrade and publish active state last. Every failure retains the old service version, old active record, and old root; no candidate or stage is automatically deleted. +- [ ] Windows release builds sign and package both binaries, while package assembly proves no SCM/ACL/root mutation. +- [ ] Only the two named boundary probes are added/run. Compile-first evidence is recorded separately from live provisioning acceptance. +- [ ] Broker delivery unblocks only Plan 2 `P2-base Task 1`; control lifecycle, Plan 1 Task 6, owner recovery, preview, and generated cleanup remain ordered downstream. + +## Plan self-review checklist + +- [ ] Every created/modified/deleted path is named above and exists in the mapped Cargo/Bazel/package structure. +- [ ] `BrokerRequest`, `ControlMutation`, `BrokerReceipt`, `ProcessIdentity`, `RootIdentity`, and all constants have one spelling and one definition across tasks. +- [ ] PowerShell `Select-String` over this plan finds none of the forbidden placeholder phrases from the planning skill. +- [ ] No step asks an implementation worker to run provisioning, edit production ACLs, create a live root, delete generated data, or stage unrelated dirty files. +- [ ] `git diff --check` passes in both repositories, and each commit stages explicit named files only. + +## Execution handoff + +Use subagent-driven execution: one fresh implementation worker per task, then a specification review and a quality review before the next task. Tasks 1-6 are Host-repository commits; Task 7 closes with one Host probe commit and one DevKit dependency-document commit. Stop immediately on compile failure, identity ambiguity, disk-pressure threshold, or any attempted live provisioning/cleanup, and report the exact failing gate without substituting a local writer. From b5e0a99a6735254f218c4025ce31d7c1e168fea9 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E5=93=80=E6=B4=9B=E8=8A=99?= Date: Sun, 30 Aug 2026 21:23:10 +0800 Subject: [PATCH 27/39] docs: classify broker runtime failures --- .../plans/2026-08-30-storage-broker-1.1.3.md | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-) diff --git a/docs/superpowers/plans/2026-08-30-storage-broker-1.1.3.md b/docs/superpowers/plans/2026-08-30-storage-broker-1.1.3.md index ab44891..83dac0b 100644 --- a/docs/superpowers/plans/2026-08-30-storage-broker-1.1.3.md +++ b/docs/superpowers/plans/2026-08-30-storage-broker-1.1.3.md @@ -172,7 +172,7 @@ Absent/ActiveOld -> CandidateVerified -> CandidateProtected -> ScmSwitched - Modify: `MODULE.bazel.lock` - [ ] Add workspace member `storage-broker-windows` and workspace dependency `codex-storage-broker-windows = { path = "storage-broker-windows" }`. The crate exposes protocol/client types on every OS; service/provisioning modules are `cfg(windows)`, and non-Windows `BrokerClient::connect` returns `STORAGE_BROKER_UNSUPPORTED`. -- [ ] Implement the locked header, `BoundedControlPayload::try_from(Vec)`, exact encode/decode functions, and a closed `BrokerErrorCode` enum. Use checked arithmetic before allocation and return only stable codes plus bounded diagnostic text. +- [ ] Implement the locked header, `BoundedControlPayload::try_from(Vec)`, exact encode/decode functions, and a closed `BrokerErrorCode` enum. Use checked arithmetic before allocation and return only stable codes plus bounded diagnostic text. Define `ErrorDisposition::{Retryable, Sticky}` locally with an exhaustive match: only `STORAGE_BROKER_BUSY`, `STORAGE_BROKER_CONNECT_TIMEOUT`, and `STORAGE_BROKER_SERVICE_STARTING` are retryable; identity/protocol/root mismatch, recovery-required, commit-uncertain, downgrade, unprovisioned, and every unknown code are sticky. The wire has no `retryable` field, so a service response cannot choose or widen its disposition. - [ ] Implement strict `ActiveProvisionRecordV1` decoding with `serde(deny_unknown_fields)`, a maximum 16 KiB file, one active and at most one rollback Host hash, exact `BROKER_RELEASE_SEQUENCE`, fixed service/pipe names, fixed protected base directories, and `RootIdentity`. Reject environment-supplied Program Files/ProgramData paths; Windows code resolves known folders. - [ ] Keep public API minimal and path-free: @@ -305,9 +305,15 @@ git commit -m "feat: make broker the protected root writer" - Modify: `codex-rs/Cargo.lock` - Modify: `MODULE.bazel.lock` -- [ ] Add the broker crate dependency. `HostStorageService` keeps validated root-plus-eight configuration and the in-memory policy evaluator, but replaces `RootStorageLedger` with a cached `Result, BrokerError>` frozen per runtime. +- [ ] Add the broker crate dependency. `HostStorageService` keeps validated root-plus-eight configuration and the in-memory policy evaluator, but replaces `RootStorageLedger` with this runtime state (or a type-equivalent state): + +```rust +broker: Mutex, BrokerError>>> +``` + +`None` means not attempted or a prior retryable attempt was deliberately not cached, `Some(Ok(client))` is the verified shared client, and `Some(Err(error))` contains only a sticky error frozen for the runtime. Never store a retryable error in the mutex. - [ ] Preserve the existing `Arc` sharing from `ThreadManager` through ordinary and delegated sessions. A session cannot construct a broker, select a pipe/root, replace the cached client, or obtain a session-local writer; all Host processes still converge on the service's single root transaction. -- [ ] `HostStorageService::new` performs no writes and does not start/install the service. Connection remains lazy. Missing config returns `STORAGE_POLICY_MISSING`; configured but missing active record/service returns `STORAGE_BROKER_UNPROVISIONED`; identity/protocol/recovery failures retain their exact stable code. +- [ ] `HostStorageService::new` performs no writes and does not start/install the service. Connection remains lazy. Each storage operation makes at most one bounded connect/mutation attempt. `BUSY`, pipe-connect timeout, or service-starting returns to that caller without caching the error; the next storage operation may make one fresh bounded attempt. Identity/protocol/root mismatch, recovery-required, commit-uncertain, downgrade, unprovisioned, and unknown errors are cached as sticky. Missing config returns `STORAGE_POLICY_MISSING`; configured but missing active record/service returns sticky `STORAGE_BROKER_UNPROVISIONED` and requires explicit provisioning plus Host restart. There is no spin, background poll, timer/TTL takeover, or retry while holding the runtime mutex. - [ ] Expose the three path-free client calls using the sealed Host provenance type: ```rust @@ -493,6 +499,7 @@ git commit -m "docs: gate storage lifecycle on protected broker" - [ ] Protocol allocation, request count, payload, concurrency, and I/O time are hard-capped before use. - [ ] The broker owns `ReserveOnce`, `ReplaceVerified`, and `Settle`; receipt publication follows durable commit; uncertain state freezes and protects stage/recovery data. - [ ] Missing configuration is disabled. Installed but unprovisioned, unsupported OS, service absent, signature mismatch, downgrade, root mismatch, or recovery state fails closed with no local fallback. +- [ ] Retry disposition is an exhaustive Host-side match: only busy/connect-timeout/service-starting remain uncached for one attempt on a later storage operation; all safety/identity/protocol/root/recovery/commit/downgrade/unprovisioned/unknown failures freeze the runtime, with no self-retry loop, background poll, TTL, or takeover. - [ ] Provisioning occurs only through an explicitly elevated administrator command or the fixed enterprise policy registry source. Build, package, install, Host startup, and config load never invoke it. - [ ] Upgrades reject rollback/downgrade and publish active state last. Every failure retains the old service version, old active record, and old root; no candidate or stage is automatically deleted. - [ ] Windows release builds sign and package both binaries, while package assembly proves no SCM/ACL/root mutation. From 72e7791758718662f67faea3a812cc1ccb3f4099 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E5=93=80=E6=B4=9B=E8=8A=99?= Date: Mon, 31 Aug 2026 15:09:17 +0800 Subject: [PATCH 28/39] docs: gate storage lifecycle on protected broker --- .../plans/2026-08-29-owned-cleanup-1.1.3.md | 246 +++++++++++------- .../2026-08-29-storage-firewall-1.1.3.md | 88 +++++-- 2 files changed, 210 insertions(+), 124 deletions(-) diff --git a/docs/superpowers/plans/2026-08-29-owned-cleanup-1.1.3.md b/docs/superpowers/plans/2026-08-29-owned-cleanup-1.1.3.md index 46d3639..8ad4484 100644 --- a/docs/superpowers/plans/2026-08-29-owned-cleanup-1.1.3.md +++ b/docs/superpowers/plans/2026-08-29-owned-cleanup-1.1.3.md @@ -2,9 +2,9 @@ > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. -**Goal:** Give the existing Host storage service one durable family/member/control accounting authority, then add explicitly authorized, bounded generated-cache cleanup. +**Goal:** Give the protected storage broker one durable family/member/control accounting authority, keep the Host as its authenticated client, then add explicitly authorized, bounded generated-cache cleanup. -**Architecture:** Plan 1 supplies strict intents, verified group/member authority, and a single service per runtime. Plan 2 moves kernel state into a root-shared transactional ledger: every Host transaction takes an OS fence, reloads current state, rechecks epoch/owners, transitions and atomically persists. Admission is one transition, not an admission followed by another reservation. Independently owned control allocations keep bootstrap/queue metadata bounded across waves without holding Cargo family leases. Cleanup is a later handle-fenced transaction over proven disposable objects, not a consequence of release, expiry, size, or age. +**Architecture:** Plan 1 supplies strict intents, verified group/member authority, and a single Host service per runtime. Plan 2 extends the root-shared transactional ledger owned by `codex-storage-broker-windows`: each protected-root transaction runs inside the broker under its OS fence, reloads current state, rechecks epoch/owners, transitions and atomically persists. The Host is only an authenticated, path-free client and has no local writer fallback. Admission is one transition, not an admission followed by another reservation. Independently owned control allocations keep bootstrap/queue metadata bounded across waves without holding Cargo family leases. Cleanup is a later handle-fenced broker transaction over proven disposable objects, not a consequence of release, expiry, size, or age. **Tech Stack:** Existing Rust `serde`/`serde_json`/`sha2`, platform filesystem/process handles, existing durable replacement helpers, and the authenticated bridge; Python is a path-free projection only. @@ -15,6 +15,10 @@ The main thread records Host `841fdaf` three-crate compile exit 0 and DevKit scope of evidence; it does not prove the new Plan 2 ledger/cleanup exists. Production cleanup and the 1.1.3 release remain incomplete. +**Protected-broker handoff (2026-08-31):** Storage broker Tasks 1-6 are a +prerequisite, not work completed by this plan revision. This edit claims no +Task 7 probe result, elevated provisioning/acceptance, or Bazel completion. + **Compile first:** Reuse only `G:\2718lab\_codex\.codex-task-temp\codex-host-mcp-fix-recovery2\codex-rs\target`, with `CARGO_INCREMENTAL=0` and `--locked -j1`. Retain prior compile evidence; @@ -27,36 +31,45 @@ this documentation-only revision. No live cleanup/configuration is authorized. ## Dependency order and bounded file map -Implement **P2-base** (Tasks 1-3: root guard, unique ledger transaction, -bootstrap/control allocations and restart protection) before enabling Plan 1 -Task 6's durable refill registration. Plan 1 does not depend on **P2-apply** -(Tasks 4-5: preview/delete tooling). Conversely P2-apply depends on Plan 1's -real process/descendant terminal fence and a correct family postcheck. -Never unblock a circular dependency with free metadata writes or fake proof. -The next production integration order is P2-base/control, then Plan 1's -remaining lifecycle/terminal wiring, then P2-apply. The root and eight policy -values are still awaiting the user's confirmation; this plan requires no -additional control-budget choice and fills in no values on the user's behalf. - -All Host paths below are relative to +Use this exact implementation dependency order: + +```text +storage broker Tasks 1-6 + -> Plan 2 P2-base Task 1 control transactions + -> Plan 1 Task 6 durable refill/lifecycle wiring + -> Plan 2 remaining P2-base owner recovery + -> Plan 2 P2-apply preview and generated cleanup +``` + +Broker completion unblocks only Plan 2 P2-base Task 1; it does not authorize +cleanup or live activation. Plan 1 does not depend on **P2-apply** (Tasks 4-5), +while P2-apply depends on Plan 1's real process/descendant terminal fence and +a correct family postcheck plus the remaining P2-base owner recovery. Never +unblock a dependency with free metadata writes or fake proof. The root and +eight policy values are still awaiting the user's confirmation; this plan +requires no additional control-budget choice and fills in no values on the +user's behalf. + +All Host-repository paths below are relative to `G:\2718lab\_codex\.codex-task-temp\codex-host-mcp-fix-recovery2`. DevKit paths are relative to `G:\2718lab\_codex\.codex-task-temp\devkit-1.1.2-recovery`. Re-read current symbols before editing; unrelated dirty work stays untouched. -Bare Host module filenames below resolve within -`codex-rs/core/src/fast_lane_host_dispatch/`; other paths are explicitly prefixed. +Protected-root writer modules resolve within +`codex-rs/storage-broker-windows/src/`; Host client modules resolve within +`codex-rs/core/src/fast_lane_host_dispatch/`. Do not recreate a writer in core. | Slice | Files and ownership | | --- | --- | -| Base state/transactions | Create `codex-rs/core/src/fast_lane_host_dispatch/storage_ledger.rs` for typed snapshots, `RootStorageLedger` transactions, bounded commit/recovery and control ownership; its local mutex serializes only this Host's callers, never replaces the cross-process fence/reload; register in `mod.rs`. | -| Real OS boundary | Create `codex-rs/core/src/fast_lane_host_dispatch/storage_fs.rs` for root/process exclusion, owned directory/file handles, bounded durable replacement and deletion primitives; no permissive path-string fallback. | -| Existing kernel integration | Modify `storage_firewall.rs` to evaluate transitions against freshly loaded transaction state instead of its own independent `Mutex`; modify `storage_service.rs::HostStorageService` to own one ledger/checker entry point per runtime into the shared root state. | -| Runtime/base initialization | Modify `codex-rs/core/src/thread_manager.rs` and existing service/session construction only as required to initialize the shared base-Config service once. Registries do not reopen a ledger per session. | +| Base state/transactions | Extend broker-owned `codex-rs/storage-broker-windows/src/{ledger,ledger_codec,service}.rs` for typed snapshots, bounded commit/recovery and control ownership. The broker transaction mutex and OS fence/reload remain authoritative; no second compilable writer may remain in `codex-core`. | +| Real OS boundary | Extend broker-owned `codex-rs/storage-broker-windows/src/root_fs.rs` for root/process exclusion, owned directory/file handles, bounded durable replacement and later deletion primitives; no permissive path-string fallback. | +| Existing kernel integration | Modify core `storage_firewall.rs` to evaluate transitions through the broker-backed operation boundary instead of its own independent `Mutex`; keep `storage_service.rs::HostStorageService` plus `storage_broker.rs` as one authenticated client entry point per runtime, with no local ledger/root handles or fallback. | +| Runtime/base initialization | Modify `codex-rs/core/src/thread_manager.rs` and existing service/session construction only as required to initialize the shared base-Config client service once. Registries do not construct a broker client or writer per session. | | Authority/queue lifecycle | Modify `fast_lane_host_dispatch/registry.rs` at initial admission, `consume_batch`, `register_refill_queue`, `consume_refill_queue` and queue persistence; modify `codex_adapter.rs`/`coordinator.rs` only at lifecycle settlement seams. Keep original route/lease hashes unchanged. | -| Later cleanup | Create `fast_lane_host_dispatch/storage_cleanup.rs` for candidate classification/manifest and apply state machine; reuse `storage_fs.rs` and Plan 1 terminal/postcheck evidence. | +| Later cleanup | Add the candidate classification/manifest and apply state machine under `codex-rs/storage-broker-windows/src/`, reusing broker `root_fs.rs` and Plan 1 terminal/postcheck evidence; core exposes only the authenticated path-free client operation. | | Later wire projection | Modify `codex-rs/rmcp-client/src/inherited_host_bridge_protocol.rs` and its `envelope.rs`/`session.rs`/`pump.rs` only for status/preview/apply; keep the single authenticated writer/receiver arrangement. | | Later DevKit projection | Create `mcp-tools/devkit_runtime/storage_ledger.py`; modify existing `host_bridge.py`, `host_session.py`, `mcp-tools/server.py` and `devkit_runtime/tool_metadata.py` for typed read-only status/preview and explicitly destructive apply. | -| Bounded verification | Create `codex-rs/core/src/fast_lane_host_dispatch/storage_ledger_tests.rs`; reuse existing firewall tests. Add only the necessary exact Python request/tool-annotation assertion in `mcp-tools/tests/test_storage_ledger.py` / `test_mcp_contract.py`. | +| Bounded verification | Put protected-root transaction/apply cases in `codex-storage-broker-windows`; reuse core firewall/client tests only for Host policy and fail-closed client behavior. Add only the necessary exact Python request/tool-annotation assertion in `mcp-tools/tests/test_storage_ledger.py` / `test_mcp_contract.py`. | No source/session deletion, GitHub reachability, CAS, compression, remote sync, new dependency, or live configuration change belongs to this revision. @@ -102,21 +115,23 @@ Do not persist or deserialize live handle/termination capabilities. ### Single ownership and arithmetic -`RootStorageLedger::transact` is the sole state-mutation/commit boundary. -Every transaction acquires the real cross-process root fence, reloads the -latest bounded snapshot, rechecks epoch/policy/owners, computes a transition -and atomically persists before releasing the fence. `HostStorageService` -remains one entry point per runtime, constructed from its manager's base -Config and shared across its sessions. Multiple Host processes coordinate -through this same root transaction protocol, not separate cached counters. -The kernel evaluates only the current transaction state. An optional cached -snapshot is non-authoritative for reservation, release, recovery or deletion. - -Move the current `State` into this durable authority rather than mirroring it. Do not expose a new -`ledger.reserve(receipt)` after `reserve_member_once`. Receipt delivery/replay, -group sealing and member attachment do not reserve again. Kernel public -wrappers must delegate into the same ledger transaction, never recursively -lock the old firewall while a ledger transaction is held. +The broker's `RootWriter::apply`/ledger transaction is the sole +state-mutation/commit boundary. Every transaction acquires the broker's real +cross-process root fence, reloads the latest bounded snapshot, rechecks +epoch/policy/owners, computes a transition and atomically persists before +releasing the fence. `HostStorageService` remains one authenticated client +entry point per runtime, constructed from its manager's base Config and shared +across its sessions. Multiple Host processes coordinate through the broker, +not separate cached counters or local writers. The kernel evaluates only the +current broker transaction state. A Host-side cached snapshot is +non-authoritative for reservation, release, recovery or deletion. + +Move the current `State` into the broker's durable authority rather than +mirroring it in core. Do not expose a new `ledger.reserve(receipt)` after +`reserve_member_once`. Receipt delivery/replay, group sealing and member +attachment do not reserve again. Kernel public wrappers must issue sealed, +path-free authenticated broker operations into the same transaction, never +open the protected root or recursively lock the old firewall. Compute/validate aggregate caches from authoritative records at load and commit; never trust a serialized global total independently: @@ -150,8 +165,9 @@ No pressure state authorizes killing a process or deleting data. ## Real authority and filesystem interfaces -Implement these interfaces as private RAII/handle types in `storage_fs.rs`; -the names describe new implementation work, not an existing capability: +Implement these interfaces as private RAII/handle types in broker-owned +`codex-rs/storage-broker-windows/src/root_fs.rs`; the names describe planned +Plan 2 capability, not work completed by the broker dependency handoff: ```rust trait RootOwnershipProvider { @@ -171,50 +187,55 @@ trait OwnedFilesystem { } ``` -`ApprovedRoot`, `SingleComponent` and handles have private constructors. +`ApprovedRoot`, `SingleComponent` and handles have broker-private constructors. +The Host authenticated client cannot construct, receive, or serialize them. `RootTransactionGuard` retains the opened root identity and actual exclusive OS lock for one reload/recheck/transition/persist transaction. Release it -after commit so another Host can transact against the new epoch; never keep -one Host's cached state authoritative after releasing it. +after commit so another authenticated Host client can transact against the +new epoch; never keep one Host's cached state authoritative after releasing it. `NamespaceMutationFence` excludes admissions and namespace writers for the exact owned subtree until final observation/ commit. Neither type is serde, a boolean, a deadline, or a caller token. -On Windows, acquire a machine-wide named mutex keyed from the opened local -volume/file identity before creating lock/ledger files; validate ownership, +On Windows, the broker acquires a machine-wide named mutex keyed from the +opened local volume/file identity before creating lock/ledger files; validate ownership, abandonment and collisions. Retain no-follow directory handles and compare volume/file IDs; reject reparse points. Child deletion uses verified handles and the platform disposition API, not `remove_dir_all` on a reconstructed string. Sharing/ACL rules and the process fence must exclude rename/replacement -by writers during the operation. The guard must also cover every Host process -using that root, not merely one Rust mutex. - -On Unix, use an actual exclusive OS lock on an already-open approved -directory where supported, and directory-relative no-follow opens plus -`fstat` identity checks. Relative unlink operations still require a real -namespace-mutation fence; `openat` or a final string comparison alone does -not close a leaf-replacement race. If the backend cannot prove that fence, -apply is unavailable. Do not claim a portable secure delete from a trait -stub. Remote/shared filesystems without a cross-host locking guarantee are -unsupported for this local-root implementation. - -Keep lock order explicit: OS root transaction guard, then this runtime's -transaction mutex, then the specific namespace fence. Reload after taking -the OS guard, not before. Await actual writer shutdown -**before** the transaction and validate the Host-owned terminal evidence -nonblockingly inside it, as required by `HostStorageTerminationEvidence`. -Do not hold the OS/accounting locks while waiting for child exit or call back -into the firewall from a proof provider. Busy/unavailable/abandoned/unknown -results fail closed and never trigger takeover by TTL. +by writers during the operation. The broker guard must serialize every Host +client using that root, not merely one core Rust mutex. + +On non-Windows platforms the protected broker is unsupported and writer/apply +operations remain unavailable. Do not reintroduce a core-local Unix writer or +claim a portable secure delete from a trait stub. Remote/shared filesystems +without the broker's locking guarantee are unsupported. + +Keep broker lock order explicit: OS root transaction guard, then the broker +transaction mutex, then the specific namespace fence. Reload after taking the +OS guard, not before. The Host awaits actual writer shutdown **before** its +authenticated request; the broker validates the sealed Host-owned terminal +evidence nonblockingly inside the transaction, as required by +`HostStorageTerminationEvidence`. Do not hold broker OS/accounting locks while +waiting for child exit or call back into the firewall from a proof provider. +Busy/unavailable/abandoned/unknown results fail closed and never trigger a +Host fallback or takeover by TTL. ## P2-base implementation tasks -### Task 1: Own bootstrap and control writes before opening a ledger +### Task 1: Wire bootstrap and control transactions through the broker -**Files:** `storage_fs.rs`, `storage_ledger.rs`, `storage_service.rs`, `mod.rs`. +**Files:** broker `root_fs.rs`, `ledger.rs`, `ledger_codec.rs`, `service.rs`, and +`protocol.rs`; Host client `storage_broker.rs`, `storage_service.rs`, `mod.rs`, +and the bounded queue-provenance seam in `registry.rs`. -- [ ] Open/validate the configured root without writing, obtain its OS - transaction guard, reload any current root snapshot, verify the unchanged +**Precondition:** Storage broker Tasks 1-6 are delivered and compiled. This +task does not recreate their root writer; it binds verified Plan 2 control +ownership/lifecycle to the broker's three path-free transactions. Every root +open/write below is broker-side. The Host remains an authenticated client. + +- [ ] In the broker, open/validate the configured root without writing, obtain + its OS transaction guard, reload any current root snapshot, verify the unchanged trusted base policy and measure available capacity. An existing valid ledger is joined transactionally, never overwritten with an empty state. No new user fields are needed or allowed: all policy values remain pending @@ -230,7 +251,7 @@ results fail closed and never trigger takeover by TTL. current collection sizes and the proposed records. Stop encoding/scanning at the remaining admitted bound. Reject overflow or an unprovable bound; do not invent a record count, unlimited log or free bootstrap exception. -- [ ] Before a ledger exists, the live root transaction guard owns this startup +- [ ] Before a ledger exists, the broker's live root transaction guard owns this startup reservation in memory and excludes concurrent root transactions. Write the first durable snapshot containing that same control reservation, then release the guard and publish the service. For an existing ledger reserve only this operation's @@ -245,10 +266,11 @@ results fail closed and never trigger takeover by TTL. Already-accounted file extents are not summed again as parent and child. - [ ] The bootstrap allocation belongs to the root lifetime, not the first Host process. Record its last mutator for audit, but do not release it on - that Host's exit or reserve it again when another Host opens the ledger. -- [ ] Provide `reserve_control_once`, `replace_control_payload` and - `settle_control` on the same service transaction, taking verified Host - queue provenance and bounded encoded payloads, not caller paths/owners. + that Host's exit or reserve it again when another authenticated Host client + joins the same broker-owned ledger. +- [ ] Wire the broker-owned `reserve_control_once`, `replace_control_payload` + and `settle_control` operations through the same service transaction, taking + verified Host queue provenance and bounded encoded payloads, not caller paths/owners. Queue control lives under the private `approved_root/control` namespace, outside `generated//members` and outside the common Cargo cache. - [ ] A queue allocation survives initial member release and all intermediate @@ -263,17 +285,30 @@ results fail closed and never trigger takeover by TTL. tombstones or live queue records to make it fit. Terminal metadata retirement needs its own proven lifecycle/epoch rule; no age-only trimming. +- [ ] Report this control-transaction slice ready only after the broker and + Host client compile together with verified queue provenance still sealed. + Only then may Plan 1 Task 6 durable refill/lifecycle wiring proceed. This + does not complete remaining P2-base recovery or authorize cleanup/activation. + +### Remaining P2-base after Plan 1 Task 6 + ### Task 2: Make kernel transitions one durable transaction -**Files:** `storage_ledger.rs`, `storage_firewall.rs`, `storage_service.rs`; -later attachment points in `registry.rs`, `codex_adapter.rs`, `coordinator.rs`. +**Files:** broker `ledger.rs`, `ledger_codec.rs`, `service.rs`, and `protocol.rs`; +Host client/evaluator `storage_broker.rs`, `storage_firewall.rs`, and +`storage_service.rs`; later attachment points in `registry.rs`, +`codex_adapter.rs`, and `coordinator.rs`. + +**Precondition:** Plan 1 Task 6 durable refill/lifecycle wiring follows the +broker-backed Task 1 control boundary above. This remaining P2-base slice must +not be pulled ahead of that lifecycle wiring. - [ ] Replace the private kernel state mutex with the reloaded ledger transaction state. Preserve `create_group`, `reserve_member_once`, `seal_group`, `consume_member_once`, `revoke_unused_member` and `release_member_after_postcheck` semantics. Internal transition evaluators take transaction state rather than reacquiring a separate lock. -- [ ] Under a newly acquired root guard/transaction lock: reload the committed +- [ ] Under a newly acquired broker root guard/transaction lock: reload the committed snapshot, verify expected epoch, current and other recorded owners/policy, exact authority and namespace identities; compute the next family/group/member/control snapshot and peak metadata reservation. @@ -307,10 +342,12 @@ later attachment points in `registry.rs`, `codex_adapter.rs`, `coordinator.rs`. ### Task 3: Recover epochs/owners without inventing live authority -**Files:** `storage_ledger.rs`, `storage_fs.rs`, `storage_service.rs` and queue -recovery seams in `registry.rs`; use Plan 1 process-lifecycle evidence. +**Files:** broker `ledger.rs`, `ledger_codec.rs`, `root_fs.rs`, and `service.rs`; +Host client `storage_broker.rs`/`storage_service.rs` and queue recovery seams in +`registry.rs`; use Plan 1 process-lifecycle evidence. -- [ ] Acquire the real root transaction guard and reload before advancing state. +- [ ] Have the broker acquire the real root transaction guard and reload before + advancing state. Validate exact snapshot-v2 structure, checksum/predecessor epoch and pending operation evidence under bounded reads. Unknown/malformed/regressing state is protected; a self-consistent hash alone is not trusted owner authority. @@ -340,15 +377,21 @@ recovery seams in `registry.rs`; use Plan 1 process-lifecycle evidence. one family lease per task. Unknown versions fail closed. Existing data and raw legacy evidence remain untouched pending an explicit audited recovery. A missing ledger never authorizes adoption/deletion of unknown artifacts. -- [ ] Report P2-base ready only when the shared ledger, bounded bootstrap/queue - allocations and fail-closed restart gate are wired and compiled. This lets - Plan 1 Task 6 proceed; it does not enable cleanup apply. +- [ ] Report remaining P2-base ready only after Plan 1 Task 6 and the + broker-backed owner recovery/restart gate are wired and compiled. It does + not enable cleanup apply or live activation. ## P2-apply: classification, preview and locked deletion ### Task 4: Classify only proven disposable, unowned generated objects -**Files:** `storage_cleanup.rs`, `storage_fs.rs`, `storage_ledger.rs`. +**Files:** broker-owned cleanup/classification module plus `root_fs.rs`, +`ledger.rs`, `ledger_codec.rs`, `service.rs`, and the path-free protocol; +Host retains only its authenticated client/projection seams. + +**Precondition:** Broker Tasks 1-6, Plan 2 P2-base Task 1, Plan 1 Task 6, and +the remaining P2-base owner recovery are complete in that order. None of those +prerequisites authorizes preview/apply or live activation. - [ ] Build candidates from registered family/member generated roots only. Eligibility requires verified producer/classification evidence, released @@ -376,11 +419,12 @@ recovery seams in `registry.rs`; use Plan 1 process-lifecycle evidence. ### Task 5: Acquire fences first, then recheck and journal apply -**Files:** `storage_cleanup.rs`, `storage_fs.rs`, `storage_ledger.rs`; -only afterward the mapped rmcp-client and DevKit projection files. +**Files:** the broker-owned cleanup module, `root_fs.rs`, `ledger.rs`, +`ledger_codec.rs`, `service.rs`, and protocol; only afterward the Host +authenticated-client, mapped rmcp-client, and DevKit projection files. -- [ ] Resolve the Host-held preview reference, acquire the root transaction - lock and real namespace-mutation fence, then freshly reopen/remeasure the +- [ ] Resolve the broker-issued preview reference, have the broker acquire the + root transaction lock and real namespace-mutation fence, then freshly reopen/remeasure the exact candidate set through retained no-follow handles. Recheck owner/ member/control references, policy, expected epoch, classification, identity, content manifest and finite bounds **inside** that fenced interval. @@ -394,7 +438,7 @@ only afterward the mapped rmcp-client and DevKit projection files. selected identities/expected epoch before the first delete. The root lock prevents another admission during this transition; retain the namespace fence through deletion, post-stat and final ledger commit. -- [ ] Delete only via `OwnedFilesystem::remove_verified_tree` with finite +- [ ] Delete only through the broker's `OwnedFilesystem::remove_verified_tree` with finite bytes/files/depth/time limits, no links/mount traversal, stable opened parent identities and the supported platform's actual mutation exclusion. Reject multiply linked or unowned entries when ownership cannot be proved. @@ -441,14 +485,14 @@ and unbounded apply batches if the bridge projection changes. Do not recreate old missing-module RED failures or introduce a broad test matrix. After a changed slice compiles, run the relevant case once, from Host -`codex-rs`: +`codex-rs`; protected-root transaction/apply cases belong to the broker crate: ```powershell $env:CARGO_TARGET_DIR='G:\2718lab\_codex\.codex-task-temp\codex-host-mcp-fix-recovery2\codex-rs\target' $env:CARGO_INCREMENTAL='0' -cargo check -p codex-rmcp-client -p codex-core --lib --locked -j1 -cargo test -p codex-core shared_family_control_transaction_is_single_charge --locked -j1 -cargo test -p codex-core apply_rechecks_identity_under_fence_and_retains_failed_shutdown --locked -j1 +cargo check -p codex-storage-broker-windows --bins -p codex-rmcp-client -p codex-core --lib --locked -j1 +cargo test -p codex-storage-broker-windows shared_family_control_transaction_is_single_charge --locked -j1 +cargo test -p codex-storage-broker-windows apply_rechecks_identity_under_fence_and_retains_failed_shutdown --locked -j1 ``` Stop before probes if compilation fails; no repeated full-suite runs. Compile @@ -467,17 +511,19 @@ is needed for these document/base slices. ## Plan 2 acceptance gates and handoff -- [ ] One service per runtime enters the same cross-process root transaction: - OS fence, reload/recheck epoch/owner state, transition, durable commit. - Other live Host owners remain active and globally counted; no independent - per-Host authoritative counter cache exists. Same-key members share the lease, and no path performs +- [ ] One authenticated Host client per runtime submits path-free operations to + the broker's cross-process root transaction: OS fence, reload/recheck + epoch/owner state, transition, durable commit. Other live Host owners remain + active and globally counted; no core-local writer or per-Host authoritative + counter cache exists. Same-key members share the lease, and no path performs firewall admission followed by independent ledger reservation. - [ ] Root-plus-eight trusted configuration remains unchanged and numerically pending the user. Bootstrap/queue/snapshot/stage/journal bytes/files are bounded and charged before writes from the existing global limits; unprovable capacity/bounds fail closed without creating metadata. -- [ ] P2-base enables Plan 1's durable refill dependency independently of - cleanup apply: live queues survive initial member settlement without +- [ ] After broker Tasks 1-6, P2-base Task 1 alone enables Plan 1's durable + refill/lifecycle dependency; only afterward may remaining P2-base owner + recovery proceed. Live queues survive initial member settlement without occupying its Cargo family lease or releasable member paths. - [ ] Real cross-process root exclusion, epoch/creation identity, recovery protection and no-follow handle fences exist. TTL/strings/PID equality @@ -490,6 +536,10 @@ is needed for these document/base slices. eligibility/identity checks, with bounded journal/delete/postcheck. Record a fixture stale-candidate/no-delete result before any separately authorized live apply; this plan edit itself authorizes none. +- [ ] Broker completion, package presence, compile/probe success, or later + provisioning does not authorize cleanup or live activation. Record those as + separate gates; this revision claims no Task 7 probes, elevated acceptance, + or Bazel completion. - [ ] No generated/source/session/queue data was deleted merely to complete a test, satisfy pressure, or repair a snapshot. Plan 3 receives protected classifications and the real fence, not permission for broader deletion. diff --git a/docs/superpowers/plans/2026-08-29-storage-firewall-1.1.3.md b/docs/superpowers/plans/2026-08-29-storage-firewall-1.1.3.md index b65c7b6..3656c2e 100644 --- a/docs/superpowers/plans/2026-08-29-storage-firewall-1.1.3.md +++ b/docs/superpowers/plans/2026-08-29-storage-firewall-1.1.3.md @@ -4,7 +4,7 @@ **Goal:** Make every Cargo, Python, MCP-package, and Fast Lane write begin with one host-approved deterministic generated root and a fail-closed byte/file/free-space reservation. -**Architecture:** DevKit emits a bounded, path-free `StorageIntent` whose hash is bound to the Fast Lane task, source plan, execution context, and project identity. The Codex Host resolves an already-issued profile, validates the intent, and reserves a deterministic target family for a Host-owned wave with per-task member grants. Receipts contain identities only; private worker facts carry paths. Existing route/lease batch hashes remain unchanged. Lease persistence, cleanup, GitHub source authorization, and session CAS are separate follow-up plans. +**Architecture:** DevKit emits a bounded, path-free `StorageIntent` whose hash is bound to the Fast Lane task, source plan, execution context, and project identity. The Codex Host resolves an already-issued profile, validates the intent, and reserves a deterministic target family for a Host-owned wave with per-task member grants. The Host remains an authenticated client of `codex-storage-broker-windows`; it never owns protected-root writer handles or falls back to a local ledger. Receipts contain identities only; private worker facts carry paths. Existing route/lease batch hashes remain unchanged. Lease persistence, cleanup, GitHub source authorization, and session CAS are separate follow-up plans. **Tech Stack:** Python 3.11 standard library (`dataclasses`, `hashlib`, `json`, `pathlib`), MCP FastMCP/Pydantic, Rust 2021, `serde`/`serde_json`, `sha2`, Tokio, platform filesystem-capacity APIs, and the existing authenticated inherited-handle bridge. @@ -19,6 +19,11 @@ admission-v1 request is exact5, replacing exact4; intent/target/profile-v1 stay unchanged. If exact4 has been deployed outside these worktrees, use admission-v2 instead and reject downgrade. +**Protected-broker handoff (2026-08-31):** The dependency text below treats +storage broker Tasks 1-6 as prerequisites, not completed work. Broker delivery +does not authorize cleanup or live activation. This revision claims no Task 7 +probe result, elevated provisioning/acceptance, or Bazel completion. + **Compile-first execution:** Reuse the Host's existing `G:\2718lab\_codex\.codex-task-temp\codex-host-mcp-fix-recovery2\codex-rs\target`, set `CARGO_INCREMENTAL=0`, and use `--locked -j1`. Check the changed crates @@ -105,9 +110,11 @@ Codex Host files: `codex-rs/core/src/session/mod.rs` (`SessionSpawnArgs`), `codex-rs/core/src/session/session.rs`, and `codex-rs/core/src/state/service.rs` (`SessionServices`) to inject one - Host-runtime service Arc into all registries. Create + Host-runtime service Arc into all registries. Reuse/modify `codex-rs/core/src/fast_lane_host_dispatch/storage_service.rs` for that - authority/ledger facade, registering it in the existing dispatch `mod.rs`. + authenticated client/policy facade, registering it in the existing dispatch + `mod.rs`; reuse the broker plan's `storage_broker.rs` and do not recreate a + root writer or local fallback in core. - Modify `codex-rs/core/src/mcp_tool_call.rs` and `codex-rs/core/src/fast_lane_host_dispatch/worktree.rs` for planned versus materialized roots; no pre-admission `create_dir_all` remains. @@ -125,11 +132,11 @@ Codex Host files: register it in `mod.rs`: real no-follow family/member observations after the process fence, not caller-supplied counters. -The capacity provider now reuses existing platform FFI; no new Windows -dependency is required. Do not change manifests/locks for this documentation -or add a dependency merely because an older example requested it. -`MODULE.bazel.lock`, if a separately justified dependency change needs it, -is at the Host repository root, not under `codex-rs`. +The Task 5 capacity provider itself still reuses existing platform FFI. Task 6 +consumes the broker crate dependency established by storage broker Tasks 1-6; +do not add another writer dependency or alter manifests/locks in this +documentation-only revision. `MODULE.bazel.lock` remains a broker-plan gate at +the Host repository root, not evidence that Bazel completion already occurred. The worker must not edit any file outside this map. The ledger, preview/apply, source authorization, and session CAS changes belong to Plans 2 and 3. @@ -646,11 +653,24 @@ Push-Location 'G:\2718lab\_codex\.codex-task-temp\codex-host-mcp-fix-recovery2'; ### Task 6: Connect admission to preparation, worker environment, and terminal release +Use this exact implementation dependency order: + +```text +storage broker Tasks 1-6 + -> Plan 2 P2-base Task 1 control transactions + -> Plan 1 Task 6 durable refill/lifecycle wiring + -> Plan 2 remaining P2-base owner recovery + -> Plan 2 P2-apply preview and generated cleanup +``` + +Broker completion unblocks only Plan 2 P2-base Task 1. It does not authorize +cleanup or live activation, and it does not by itself complete this Task 6. + **Independent file groups:** All paths below are within the scope map above. | Slice | Files and existing attachment points | | --- | --- | -| 6a configuration/service | `config/src/config_toml.rs::ConfigToml`, `core/src/config/mod.rs::Config::load_config_with_layer_stack`, generated `core/config.schema.json`; new `core/src/fast_lane_host_dispatch/storage_service.rs` and `mod.rs`; `app-server/src/message_processor.rs`, `core/src/thread_manager.rs::ThreadManagerState`, `core/src/session/mod.rs::SessionSpawnArgs`, `core/src/session/session.rs`, `core/src/state/service.rs::SessionServices` | +| 6a configuration/service | `config/src/config_toml.rs::ConfigToml`, `core/src/config/mod.rs::Config::load_config_with_layer_stack`, generated `core/config.schema.json`; core `storage_service.rs`, broker-plan `storage_broker.rs`, and `mod.rs` as the authenticated Host client only; `app-server/src/message_processor.rs`, `core/src/thread_manager.rs::ThreadManagerState`, `core/src/session/mod.rs::SessionSpawnArgs`, `core/src/session/session.rs`, `core/src/state/service.rs::SessionServices` | | 6b authority/write ordering | `core/src/fast_lane_host_dispatch/{registry,contract,storage_profile,worktree}.rs`, `core/src/mcp_tool_call.rs`; existing rmcp-client refill protocol/session/pump; DevKit `fastlane_host_adapter.py`, `host_bridge.py`, `host_session.py`, `server.py` | | 6c worker isolation | `core/src/fast_lane_host_dispatch/codex_adapter.rs::{HostWriterContext,CodexHostDispatchFacts,prepare_batch}`, `core/src/config/mod.rs::Permissions`, `protocol/src/shell_environment.rs`, `core/src/unified_exec/process_manager.rs` | | 6d real termination | `core/src/unified_exec/{mod,process,process_manager}.rs`, `utils/pty/src/{process,win/job}.rs`, `core/src/session/handlers.rs`, `core/src/agent/control/legacy.rs`; reuse `exec-server/src/process.rs` event boundary | @@ -659,11 +679,11 @@ Push-Location 'G:\2718lab\_codex\.codex-task-temp\codex-host-mcp-fix-recovery2'; Host paths in this table are relative to `codex-rs`; DevKit runtime filenames are under `mcp-tools/devkit_runtime`, with `server.py` under `mcp-tools`. 6a can compile independently with storage disabled. 6d can be implemented -independently of admission. 6b depends on 6a and the preserved Task 4/5 -contracts, plus the bounded control-allocation prerequisite below for durable -refill registration; 6c depends on 6b. Successful release in 6e requires both phases of -6d, not just worker wiring or a green compile. Coordinate shared files rather -than concurrently editing registry/adapter/process-manager from two slices. +independently of admission. Durable 6b depends on storage broker Tasks 1-6, +Plan 2 P2-base Task 1, 6a, and the preserved Task 4/5 contracts; 6c depends on +6b. Successful release in 6e requires both phases of 6d, not just worker wiring +or a green compile. Coordinate shared files rather than concurrently editing +registry/adapter/process-manager from two slices. - [ ] **Step 6a: Load explicit trusted configuration and inject one runtime-owned service.** @@ -710,8 +730,9 @@ The Host runtime caller in `app-server/src/message_processor.rs` supplies that base configuration. Store the same Arc in `ThreadManagerState`, forward through `SessionSpawnArgs`/`SessionServices`, and give every `FastLaneHostFactsRegistry` a reference to it. The new -`storage_service.rs` facade owns the single kernel accounting state and -immutable resolved policy/root, not another independent reservation ledger. +`storage_service.rs` facade owns the single Host-side policy evaluator and +authenticated broker client for the immutable resolved policy/root, not an +independent reservation ledger or protected-root handle. Adapt every constructor/call site; test-only construction may be explicitly disabled or use an injected fixture, never a permissive production default. Current `session/session.rs` creates registries per Session: keep that facts @@ -719,10 +740,11 @@ scope but do not create a firewall there. Per-thread config reload must not reset global reservations or replace policy/root; reject a differing storage configuration with the local `STORAGE_CONFIG_RESTART_REQUIRED` diagnostic. -This is cross-session sharing inside one Host runtime, not a cross-process -ledger. An independent Host process must not claim the same active storage -root without exclusive root authority; multi-process persistence/recovery -remains Plan 2. An in-memory Arc cannot prove that exclusivity by itself. +This is cross-session sharing of one authenticated client inside one Host +runtime. Cross-process serialization, persistence, and root ownership belong +only to `codex-storage-broker-windows`; an independent Host process is another +authenticated client and cannot claim the root. Remaining owner recovery stays +in Plan 2 after this Task 6. An in-memory Arc is never root authority. - [ ] **Step 6b: Admit/seal before any task-root write, then consume once.** @@ -755,10 +777,12 @@ Their lifecycle spans waves: releasing/cleaning the initial member must not erase an active queue, and retaining its Cargo family lease for the queue would permanently block a different-owner same-key successor. -Durable refill registration therefore depends on an independently bounded -Host ledger/control allocation with a queue-lifetime owner, explicit positive -byte/file allowance, safe private root, and terminal queue settlement. Plan 2 -must provide that control-allocation contract before this part of 6b is +Durable refill registration therefore depends first on storage broker Tasks +1-6 and then on Plan 2 P2-base Task 1 binding an independently bounded +broker-owned control allocation to a verified queue-lifetime owner, explicit +positive byte/file allowance, safe private root, and terminal queue settlement. +The Host accesses those control transactions only as an authenticated client. +Plan 2 must provide that control-allocation contract before this part of 6b is enabled; implement no ledger or new artifact/wire field in this Plan 1 update. Until it exists, reject durable registration before any write. Do not invent a control task/lease, take an unrequested budget, or exempt metadata from @@ -899,8 +923,20 @@ or complete Plan 1 acceptance. - [ ] Run `python -m py_compile` on changed DevKit Python files and `cargo check -p codex-rmcp-client -p codex-core --lib --locked -j1` with the existing Host `codex-rs\target` and `CARGO_INCREMENTAL=0`; retain current compile evidence instead of rerunning unchanged slices. - [ ] Record one controlled admission receipt proving same semantics reuse one target key and one changed semantic forks it; record one low-space/policy-failure receipt proving no directory was created. - [ ] Verify exact5 session lookup/core `Sent` binding, unchanged original batch hashes, same-owner member sharing with exact sealing, cross-owner conflict, and a native selected-successor admission without a second reservation. Do not label initial-only wiring complete. -- [ ] Verify explicit trusted root plus eight policy values, one Host-runtime service shared across Sessions, and no pre-admission task/control/queue directory writes. Record the single-runtime versus independent-Host-process ownership boundary. -- [ ] Before enabling durable refill, obtain Plan 2's bounded queue-lifetime control allocation: initial member release cannot remove active metadata, and that allocation cannot retain the Cargo family lease or block same-key successors. Missing allocation rejects registration before writes; no free budget or fabricated task is allowed. +- [ ] Verify explicit trusted root plus eight policy values, one authenticated + Host client service shared across Sessions, and no pre-admission + task/control/queue directory writes. Verify the broker is the sole root writer + and no Host-local fallback exists across independent Host processes. +- [ ] Before enabling durable refill, complete storage broker Tasks 1-6 and + obtain Plan 2 P2-base Task 1's broker-backed bounded queue-lifetime control + allocation: initial member release cannot remove active metadata, and that + allocation cannot retain the Cargo family lease or block same-key successors. + Missing allocation rejects registration before writes; no free budget or + fabricated task is allowed. Remaining P2-base owner recovery follows this + Task 6; P2-apply follows only after that recovery. - [ ] Verify actual process and descendant shutdown evidence, denial of post-proof writers, and a real all-members-quiet family scan before release. Failure/timeout retains ownership; `Completed`, `ShutdownComplete`, ACK, or an empty ProcessStore is not sufficient. - [ ] Obtain the operator's approved absolute root and eight policy values before production enablement; do not invent them or reuse example test capacities as configuration. Record any unsupported process-containment platform as a remaining activation gate. - [ ] Do not implement ledger persistence, preview/apply, source deletion, session deletion, compression, or remote synchronization in this plan. Plan 2 consumes `StorageAdmissionReceipt`; Plan 3 consumes the released/observed storage records. +- [ ] Treat broker completion, package presence, compile/probe success, and any + later elevated provisioning acceptance as separate gates. None authorizes + cleanup or live activation; this plan revision claims none complete. From a580021e92e8f8c652edb8f532b2e609ae1ebe67 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E5=93=80=E6=B4=9B=E8=8A=99?= Date: Tue, 1 Sep 2026 17:26:31 +0800 Subject: [PATCH 29/39] release: prepare DevKit 1.1.3 --- .codex-plugin/plugin.json | 2 +- .github/ISSUE_TEMPLATE/bug_report.yml | 2 +- CHANGELOG.md | 39 ++++++++++++++++++++++ README.md | 12 +++---- README.zh-CN.md | 12 +++---- mcp-tools/pyproject.toml | 2 +- mcp-tools/tests/test_bugkiller_metadata.py | 2 +- mcp-tools/tests/test_primary_artifact.py | 4 +-- mcp-tools/uv.lock | 2 +- 9 files changed, 58 insertions(+), 19 deletions(-) diff --git a/.codex-plugin/plugin.json b/.codex-plugin/plugin.json index 6963596..706b1e7 100644 --- a/.codex-plugin/plugin.json +++ b/.codex-plugin/plugin.json @@ -1,6 +1,6 @@ { "name": "2718lab-devkit", - "version": "1.1.2", + "version": "1.1.3", "description": "Local MCP server for developer workflow coordination, indexing, and evidence handling.", "author": { "name": "2718lab", diff --git a/.github/ISSUE_TEMPLATE/bug_report.yml b/.github/ISSUE_TEMPLATE/bug_report.yml index d0a632b..b4e8d9f 100644 --- a/.github/ISSUE_TEMPLATE/bug_report.yml +++ b/.github/ISSUE_TEMPLATE/bug_report.yml @@ -28,7 +28,7 @@ body: id: version attributes: label: DevKit version - placeholder: "v1.1.2" + placeholder: "v1.1.3" validations: required: true diff --git a/CHANGELOG.md b/CHANGELOG.md index f3ad3dc..0aec67f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,45 @@ only after the CI and artifact checks pass. ## [Unreleased] +## [1.1.3] - 2026-09-01 + +### Added + +- Added a canonical, exact-key, path-free storage-intent contract for Cargo + targets, Python caches, MCP packages, and Fast Lane task storage. Intent + hashes bind task, plan, context, byte/file budgets, and the complete target + descriptor without allowing DevKit to choose a filesystem path or claim a + storage lease. +- Added DevKit-side storage profile/admission validation and compiler-proof + plumbing for compatible Hosts. Budgets participate in wave/profile evidence + and intent proofs can be privately bound, while the legacy eight-field + pre-Host skeleton and public dispatch batch remain free of storage intents; + production worker admission is not activated by this repository. + +### Fixed + +- Hardened verified legacy runtime-store migration so physical and semantic + schema shape, metadata, content addresses, acceptance identities, and exact + Atlas outbox/finalization bindings are checked before current DDL can run. + Schema drift, null/orphan rows, half-upgraded state, or content-address + mismatch fail closed. +- Bounded storage-admission frames, deadlines, cancellation, and active bridge + I/O teardown; zero free-space floors and mismatched or unknown receipt fields + are rejected before a storage decision can be consumed. + +### Security + +- On an ordinary Host, the missing compatible private profile/authority + exchange keeps budgeted Fast Lane fail-closed with + `FASTLANE_HOST_AUTHORITY_UNAVAILABLE`/`NO_SAFE_WORK`. DevKit provides the + path-free intent and private protocol, but does not authenticate or provision + the Windows protected broker, create its root, activate cleanup, or provide a + local writer fallback. +- The compatible Host boundary is still awaiting final protected-broker + compile, probe, and runtime receipts from the separate Host branch. This + release does not claim that upstream Codex or an ordinary Codex Host supports + protected-broker storage execution. + ## [1.1.2] - 2026-08-27 ### Fixed diff --git a/README.md b/README.md index ceebf1b..e832292 100644 --- a/README.md +++ b/README.md @@ -1,14 +1,14 @@ [简体中文](README.zh-CN.md) -# 2718lab DevKit — Codex + MCP v1.1.2 +# 2718lab DevKit — Codex + MCP v1.1.3 -[![version](https://img.shields.io/badge/version-v1.1.2-blue)](./.codex-plugin/plugin.json) +[![version](https://img.shields.io/badge/version-v1.1.3-blue)](./.codex-plugin/plugin.json) [![license](https://img.shields.io/badge/license-AGPL--3.0-blue)](LICENSE) 2718lab DevKit is a Codex-first engineering toolkit: a local, stdio-only MCP runtime for bounded project indexing, Atlas evidence, Relay lifecycle coordination, and deterministic Fast Lane planning, plus a compact Skill bundle -of reference manuals. This repository carries the versioned v1.1.2 package. +of reference manuals. This repository carries the versioned v1.1.3 package. The checked-in manifest and allowlist define the executable runtime surface; the manual map, install, build, and verification sections below describe the supported workflow. @@ -168,7 +168,7 @@ source of record remains `main` and immutable release tags. Maintainers build that snapshot with the dedicated marketplace allowlist: - python .codex-plugin/build_main_artifact.py --plugin-root . --allowlist .codex-plugin/marketplace-artifact-allowlist.json --output /2718lab-devkit-marketplace-v1.1.2.zip + python .codex-plugin/build_main_artifact.py --plugin-root . --allowlist .codex-plugin/marketplace-artifact-allowlist.json --output /2718lab-devkit-marketplace-v1.1.3.zip ## Install and run locally @@ -215,7 +215,7 @@ handles or falls back to an unrelated local start. The allowlisted builder creates a deterministic ZIP outside the plugin source tree. Choose an output directory outside the source tree: - python .codex-plugin/build_main_artifact.py --plugin-root . --output /2718lab-devkit-v1.1.2.zip + python .codex-plugin/build_main_artifact.py --plugin-root . --output /2718lab-devkit-v1.1.3.zip The artifact contains the manifest, .mcp.json, LICENSE, the locked Python project, and the runtime files selected by @@ -381,7 +381,7 @@ freeze a transient regression count. ## Version -This repository represents the versioned v1.1.2 package. Release notes are +This repository represents the versioned v1.1.3 package. Release notes are in [CHANGELOG.md](CHANGELOG.md); build and install from the checked-in manifest, artifact allowlist, and locked dependency set. A maintainer dispatches Release from current `main`; it validates all declared gates, creates the annotated tag, diff --git a/README.zh-CN.md b/README.zh-CN.md index 7d7c06b..81b3204 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -1,14 +1,14 @@ [English](README.md) -# 2718lab DevKit —— Codex + MCP v1.1.2 +# 2718lab DevKit —— Codex + MCP v1.1.3 -[![版本](https://img.shields.io/badge/version-v1.1.2-blue)](./.codex-plugin/plugin.json) +[![版本](https://img.shields.io/badge/version-v1.1.3-blue)](./.codex-plugin/plugin.json) [![许可证](https://img.shields.io/badge/license-AGPL--3.0-blue)](LICENSE) 2718lab DevKit 是一个 Codex-first 工程工具包：它包含一个本地、仅 stdio 传输的 MCP 运行时，用于有边界的项目索引、Atlas 证据、Relay 生命周期协调和 确定性的 Fast Lane 规划；同时还包含一组精简的 Skill 说明书。本仓库承载版本化的 -v1.1.2 包；已提交的 manifest 和 allowlist 定义可执行运行时范围，说明书导航、 +v1.1.3 包；已提交的 manifest 和 allowlist 定义可执行运行时范围，说明书导航、 安装、构建和验证章节共同给出支持的工作流。 当前版本保留刻意 fail-closed 的 Fast Lane 预览。公共编译器和 CLI 固定返回 @@ -148,7 +148,7 @@ Fast Lane 不含额度协调器合同；公共编译器和 CLI 不读取、协 维护者使用专用的 marketplace allowlist 构建该快照： - python .codex-plugin/build_main_artifact.py --plugin-root . --allowlist .codex-plugin/marketplace-artifact-allowlist.json --output /2718lab-devkit-marketplace-v1.1.2.zip + python .codex-plugin/build_main_artifact.py --plugin-root . --allowlist .codex-plugin/marketplace-artifact-allowlist.json --output /2718lab-devkit-marketplace-v1.1.3.zip ## 本地安装与运行 @@ -188,7 +188,7 @@ RELAY_CAPABILITY_BROKER_UNAVAILABLE。服务器不会暴露原始 handle，也 allowlist builder 会在插件源码树之外生成确定性的 ZIP。请选择源码树之外的输出目录： - python .codex-plugin/build_main_artifact.py --plugin-root . --output /2718lab-devkit-v1.1.2.zip + python .codex-plugin/build_main_artifact.py --plugin-root . --output /2718lab-devkit-v1.1.3.zip 产物包含 manifest、.mcp.json、LICENSE、锁定的 Python 项目，以及 .codex-plugin/main-artifact-allowlist.json 选中的运行时文件。它的可执行运行时 @@ -329,7 +329,7 @@ CI 和全新产物检查才是当前测试计数的唯一来源。它们验证 ## 版本 -本仓库代表版本化的 v1.1.2 包。发布说明见 +本仓库代表版本化的 v1.1.3 包。发布说明见 [CHANGELOG.md](CHANGELOG.md)；构建和安装请以已提交的 manifest、产物 allowlist 和锁定依赖为准。维护者从 current `main` 手动 dispatch Release；它通过全部 gates 后才创建注释 tag 并发布匹配的 GitHub Release。单独 push tag 不会触发发布。 diff --git a/mcp-tools/pyproject.toml b/mcp-tools/pyproject.toml index c7fc5f1..fd9823d 100644 --- a/mcp-tools/pyproject.toml +++ b/mcp-tools/pyproject.toml @@ -1,6 +1,6 @@ [project] name = "2718lab-devkit-mcp" -version = "1.1.2" +version = "1.1.3" description = "MCP runtime for the 2718lab DevKit primary plugin." requires-python = ">=3.11" dependencies = [ diff --git a/mcp-tools/tests/test_bugkiller_metadata.py b/mcp-tools/tests/test_bugkiller_metadata.py index c4c5c5b..6192c4b 100644 --- a/mcp-tools/tests/test_bugkiller_metadata.py +++ b/mcp-tools/tests/test_bugkiller_metadata.py @@ -39,7 +39,7 @@ def test_primary_plugin_manifest_is_stable_v1_and_has_no_prompt_runtime_surface( self, ) -> None: codex = load_json(".codex-plugin/plugin.json") - self.assertEqual("1.1.2", codex["version"]) + self.assertEqual("1.1.3", codex["version"]) self.assertEqual("./.mcp.json", codex["mcpServers"]) for legacy_surface in ("skills", "agents", "commands", "hooks"): self.assertNotIn(legacy_surface, codex) diff --git a/mcp-tools/tests/test_primary_artifact.py b/mcp-tools/tests/test_primary_artifact.py index 1d90be7..0f59b97 100644 --- a/mcp-tools/tests/test_primary_artifact.py +++ b/mcp-tools/tests/test_primary_artifact.py @@ -231,14 +231,14 @@ def test_python_project_and_lock_use_pep440_stable_v1_metadata() -> None: with project_path.open("rb") as project_file: project = tomllib.load(project_file) - assert project["project"]["version"] == "1.1.2" + assert project["project"]["version"] == "1.1.3" assert project["project"]["dependencies"] == ["mcp[cli]>=1,<2"] assert "devkit_atlas" in project["tool"]["pyright"]["include"] assert "devkit_runtime" in project["tool"]["pyright"]["include"] assert "code_atlas" not in project["tool"]["pyright"]["include"] lock_text = lock_path.read_text(encoding="utf-8") assert 'name = "2718lab-devkit-mcp"' in lock_text - assert 'version = "1.1.2"' in lock_text + assert 'version = "1.1.3"' in lock_text def test_two_builds_are_byte_identical_with_normalized_zip_metadata( diff --git a/mcp-tools/uv.lock b/mcp-tools/uv.lock index c5bc3a4..96cd914 100644 --- a/mcp-tools/uv.lock +++ b/mcp-tools/uv.lock @@ -10,7 +10,7 @@ resolution-markers = [ [[package]] name = "2718lab-devkit-mcp" -version = "1.1.2" +version = "1.1.3" source = { virtual = "." } dependencies = [ { name = "mcp", extra = ["cli"] }, From 9fc9218c66b6b01e9e151c8e0c7091e1f31b8196 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E5=93=80=E6=B4=9B=E8=8A=99?= Date: Tue, 1 Sep 2026 17:32:54 +0800 Subject: [PATCH 30/39] fix: satisfy 1.1.3 release static gates --- mcp-tools/devkit_fastlane/scripts/authenticated_v5_planner.py | 4 ++-- mcp-tools/devkit_runtime/storage_intent.py | 1 - 2 files changed, 2 insertions(+), 3 deletions(-) diff --git a/mcp-tools/devkit_fastlane/scripts/authenticated_v5_planner.py b/mcp-tools/devkit_fastlane/scripts/authenticated_v5_planner.py index d4bf111..14233de 100644 --- a/mcp-tools/devkit_fastlane/scripts/authenticated_v5_planner.py +++ b/mcp-tools/devkit_fastlane/scripts/authenticated_v5_planner.py @@ -174,7 +174,7 @@ def _normalize_host_storage_profile( try: api._hash(item, f"Host storage profile {task_id}.{field}") except Exception as error: - raise ValueError(_STORAGE_TARGET_KEY_INVALID) + raise ValueError(_STORAGE_TARGET_KEY_INVALID) from error unsigned = { key: profile[key] for key in _STORAGE_PROFILE_FIELDS @@ -734,7 +734,7 @@ def validate_skeleton_package( raise ValueError("authenticated V5 skeleton fields are invalid") if skeleton.get("source_plan_hash") != source_hash: raise ValueError("authenticated V5 skeleton source hash is invalid") - task_id = api._task_id( + api._task_id( skeleton.get("task_id"), f"authenticated V5 {wave_name} skeletons[{index}].task_id", ) diff --git a/mcp-tools/devkit_runtime/storage_intent.py b/mcp-tools/devkit_runtime/storage_intent.py index 0a3636e..6bbd068 100644 --- a/mcp-tools/devkit_runtime/storage_intent.py +++ b/mcp-tools/devkit_runtime/storage_intent.py @@ -16,7 +16,6 @@ from types import MappingProxyType from typing import Final - STORAGE_INTENT_SCHEMA: Final = "2718lab.storage.intent.v1" TARGET_DESCRIPTOR_SCHEMA: Final = "2718lab.storage.target.v1" STORAGE_TARGET_KEY_INVALID: Final = "STORAGE_TARGET_KEY_INVALID" From c578c308f652c9108a10442e8ceae2d48cc37957 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E5=93=80=E6=B4=9B=E8=8A=99?= Date: Tue, 1 Sep 2026 17:36:15 +0800 Subject: [PATCH 31/39] style: satisfy 1.1.3 release formatting gate --- .../scripts/authenticated_v5_planner.py | 18 +- .../scripts/authenticated_v5_projection.py | 18 +- .../devkit_runtime/fastlane_host_adapter.py | 32 +- .../devkit_runtime/fastlane_host_intent.py | 14 +- mcp-tools/devkit_runtime/host_bridge.py | 308 ++++++++++------ mcp-tools/devkit_runtime/host_session.py | 123 ++++--- mcp-tools/orchestrator/store.py | 340 ++++++++++++------ mcp-tools/server.py | 43 +-- mcp-tools/tests/test_runtime_composition.py | 304 +++++++++------- mcp-tools/tests/test_storage_firewall.py | 202 ++++++++--- 10 files changed, 883 insertions(+), 519 deletions(-) diff --git a/mcp-tools/devkit_fastlane/scripts/authenticated_v5_planner.py b/mcp-tools/devkit_fastlane/scripts/authenticated_v5_planner.py index 14233de..7200399 100644 --- a/mcp-tools/devkit_fastlane/scripts/authenticated_v5_planner.py +++ b/mcp-tools/devkit_fastlane/scripts/authenticated_v5_planner.py @@ -214,10 +214,7 @@ def _make_storage_intent( descriptor = { "schema": _STORAGE_TARGET_SCHEMA, "artifact_kind": "fastlane-task", - **{ - field: profile.get(field) - for field in _STORAGE_DESCRIPTOR_FIELDS - }, + **{field: profile.get(field) for field in _STORAGE_DESCRIPTOR_FIELDS}, } if any(descriptor[field] is None for field in _STORAGE_DESCRIPTOR_FIELDS): raise ValueError(_STORAGE_POLICY_MISSING) @@ -283,10 +280,7 @@ def _routing_profile_material( return { "schema": _PROFILE_EVIDENCE_SCHEMA, "source_plan_hash": source_plan_hash, - "unit": { - field: task if field == "task" else unit[field] - for field in fields - }, + "unit": {field: task if field == "task" else unit[field] for field in fields}, } @@ -728,7 +722,9 @@ def validate_skeleton_package( raise ValueError(f"authenticated V5 {wave_name} skeletons are invalid") for index, raw_skeleton in enumerate(wave): skeleton = dict( - api._mapping(raw_skeleton, f"authenticated V5 {wave_name} skeletons[{index}]") + api._mapping( + raw_skeleton, f"authenticated V5 {wave_name} skeletons[{index}]" + ) ) if set(skeleton) != _SKELETON_FIELDS: raise ValueError("authenticated V5 skeleton fields are invalid") @@ -739,7 +735,9 @@ def validate_skeleton_package( f"authenticated V5 {wave_name} skeletons[{index}].task_id", ) if "storage_intent" in skeleton: - raise ValueError("authenticated V5 pre-host skeleton carries storage intent") + raise ValueError( + "authenticated V5 pre-host skeleton carries storage intent" + ) order = skeleton.get("dispatch_order") if type(order) is not int or not 0 <= order < len(source_ids): raise ValueError("authenticated V5 package dispatch order is invalid") diff --git a/mcp-tools/devkit_fastlane/scripts/authenticated_v5_projection.py b/mcp-tools/devkit_fastlane/scripts/authenticated_v5_projection.py index 76b3c4d..fffff10 100644 --- a/mcp-tools/devkit_fastlane/scripts/authenticated_v5_projection.py +++ b/mcp-tools/devkit_fastlane/scripts/authenticated_v5_projection.py @@ -54,16 +54,11 @@ def _routing_profile_material( return { "schema": _PROFILE_EVIDENCE_SCHEMA, "source_plan_hash": source_plan_hash, - "unit": { - field: task if field == "task" else unit[field] - for field in fields - }, + "unit": {field: task if field == "task" else unit[field] for field in fields}, } -def _storage_request_without_extensions( - value: Mapping[str, Any], api: Any -) -> None: +def _storage_request_without_extensions(value: Mapping[str, Any], api: Any) -> None: if "storage_contexts" in value: raise ValueError(_STORAGE_TARGET_KEY_INVALID) base = { @@ -111,10 +106,7 @@ def _validated_request_storage_budgets( return {} if not isinstance(value, Mapping) or set(value) != set(task_ids): raise ValueError(_STORAGE_POLICY_MISSING) - return { - task_id: _validated_storage_budget(value[task_id]) - for task_id in task_ids - } + return {task_id: _validated_storage_budget(value[task_id]) for task_id in task_ids} def _attach_storage_budget( @@ -346,9 +338,7 @@ def project_slice(task_ids: Sequence[str]) -> list[dict[str, Any]]: # remains the complete 0..N-1 sequence. dispatch_order = package_order[task_id] source_unit = units_by_task[task_id] - if ( - target_by_task.get(task_id) is None - ): + if target_by_task.get(task_id) is None: raise ValueError("authenticated V5 execution context is incomplete") target = target_by_task[task_id] write_scope = api._normalised_scopes(source_unit.get("write_scope", [])) diff --git a/mcp-tools/devkit_runtime/fastlane_host_adapter.py b/mcp-tools/devkit_runtime/fastlane_host_adapter.py index 51e26f9..bff645a 100644 --- a/mcp-tools/devkit_runtime/fastlane_host_adapter.py +++ b/mcp-tools/devkit_runtime/fastlane_host_adapter.py @@ -232,10 +232,9 @@ def _storage_intents_for_profiles( ): raise ValueError("storage profile assignment binding is invalid") assignment_task_ids.append(task_id) - if ( - len(set(assignment_task_ids)) != len(assignment_task_ids) - or set(budget_by_task) != set(assignment_task_ids) - ): + if len(set(assignment_task_ids)) != len(assignment_task_ids) or set( + budget_by_task + ) != set(assignment_task_ids): raise ValueError("storage budget/profile bindings are invalid") profile_task_ids: list[str] = [] for profile in profiles: @@ -296,10 +295,7 @@ def _storage_intents_for_profiles( source_plan_hash = assignment.get("source_plan_hash") assert type(task_id) is str budget = budget_by_task.get(task_id) - if ( - budget is None - or profile.get("source_plan_hash") != source_plan_hash - ): + if budget is None or profile.get("source_plan_hash") != source_plan_hash: raise ValueError("storage profile assignment binding is invalid") context_hash = profile.get("execution_context_hash") if ( @@ -430,10 +426,10 @@ def prepare_verified_host_facts( normalized_request["project_index_attestation_refs"], ) ), - routing_registry_binding_hash=cast( - str, routing_registry_binding_hash + routing_registry_binding_hash=cast(str, routing_registry_binding_hash), + storage_task_ids=( + storage_task_ids if normalized_storage_budgets else () ), - storage_task_ids=(storage_task_ids if normalized_storage_budgets else ()), storage_budget_bindings=normalized_storage_budgets, ) if not bridge_attested: @@ -477,9 +473,7 @@ def prepare_verified_host_facts( bridge_attested=bridge_attested, evidence_expires_at=expires_at, preparation_id=normalized_preparation_id, - call_intent_hash=( - cast(str, call_intent_hash) if bridge_attested else None - ), + call_intent_hash=(cast(str, call_intent_hash) if bridge_attested else None), storage_budgets=normalized_storage_budgets, storage_intents=storage_intents, ) @@ -601,8 +595,7 @@ def compile_fast_lane_with_host_facts( ) ) storage_intent_hashes = tuple( - cast(str, intent["storage_intent_hash"]) - for intent in storage_intents + cast(str, intent["storage_intent_hash"]) for intent in storage_intents ) if ( material.storage_budget_bindings != prepared.storage_budgets @@ -1105,9 +1098,10 @@ def _dispatch_fact_from_mapping(value: object) -> _HostDispatchFact: ledger_epoch=cast(int, normalized["ledger_epoch"]), active_lease_set_hash=cast(str, normalized["active_lease_set_hash"]), ) - if normalized["dispatch_binding_hash"] != _dispatch_fact_mapping(fact)[ - "dispatch_binding_hash" - ]: + if ( + normalized["dispatch_binding_hash"] + != _dispatch_fact_mapping(fact)["dispatch_binding_hash"] + ): raise ValueError("dispatch binding hash is invalid") return fact diff --git a/mcp-tools/devkit_runtime/fastlane_host_intent.py b/mcp-tools/devkit_runtime/fastlane_host_intent.py index 2f56918..c2b91df 100644 --- a/mcp-tools/devkit_runtime/fastlane_host_intent.py +++ b/mcp-tools/devkit_runtime/fastlane_host_intent.py @@ -26,9 +26,7 @@ _SCHEMA_V2: Final = "2718lab-devkit/fastlane-host-execution-intent-v2" _SCHEMA_V3: Final = "2718lab-devkit/fastlane-host-execution-intent-v3" -_RELAY_HOST_SCHEDULER_SLOT_SCHEMA: Final = ( - "2718lab-devkit/relay-host-scheduler-slot-v1" -) +_RELAY_HOST_SCHEDULER_SLOT_SCHEMA: Final = "2718lab-devkit/relay-host-scheduler-slot-v1" _HOST_TOPOLOGY_SCHEMA: Final = "2718lab-devkit/host-scheduler-topology-v1" _PREDECESSOR_SCHEMA: Final = "2718lab-devkit/fastlane-external-lease-predecessor-v2" _HASH_PATTERN: Final = re.compile(r"sha256:[0-9a-f]{64}\Z") @@ -464,8 +462,7 @@ def _parse(candidate: object) -> ParsedHostExecutionIntent | None: create = _bound_mapping(root["create"], _CREATE_KEYS, "create_binding_hash") lease = _bound_mapping(root["lease"], _LEASE_KEYS, "lease_binding_hash") if any( - value is None - for value in (assignment, route, packets, source, create, lease) + value is None for value in (assignment, route, packets, source, create, lease) ): return None @@ -1047,9 +1044,10 @@ def _is_expectation_projection( if not _capability_expectations_are_valid(expectation.capability_facts): return False - if expectation.storage_intent is not None and type( - expectation.storage_intent - ) is not StorageIntent: + if ( + expectation.storage_intent is not None + and type(expectation.storage_intent) is not StorageIntent + ): return False if expectation.execution_context_hash is not None and not _is_hash_value( expectation.execution_context_hash diff --git a/mcp-tools/devkit_runtime/host_bridge.py b/mcp-tools/devkit_runtime/host_bridge.py index 3de3b62..847fa09 100644 --- a/mcp-tools/devkit_runtime/host_bridge.py +++ b/mcp-tools/devkit_runtime/host_bridge.py @@ -41,28 +41,42 @@ _OPERATION_REQUEST_SCHEMA: Final = "2718lab-devkit/host-operation-request-v1" _TERMINAL_RESULT_SCHEMA: Final = "2718lab-devkit/host-terminal-result-v1" _PROOF_CONTINUATION_SCHEMA: Final = "2718lab-devkit/host-proof-continuation-v1" -_COMPILER_EVIDENCE_REQUEST_SCHEMA: Final = ( - "2718lab-devkit/compiler-evidence-request-v1" -) +_COMPILER_EVIDENCE_REQUEST_SCHEMA: Final = "2718lab-devkit/compiler-evidence-request-v1" _COMPILER_EVIDENCE_RESPONSE_SCHEMA: Final = ( "2718lab-devkit/compiler-evidence-response-v1" ) -_STORAGE_PROFILE_REQUEST_SCHEMA: Final = ( - "2718lab-devkit/storage-profile-request-v1" -) +_STORAGE_PROFILE_REQUEST_SCHEMA: Final = "2718lab-devkit/storage-profile-request-v1" _STORAGE_PROFILE_SCHEMA: Final = "2718lab-devkit/storage-profile-v1" _STORAGE_ADMISSION_REQUEST_SCHEMA: Final = "2718lab.storage.admission-request.v1" _STORAGE_ADMISSION_RESPONSE_SCHEMA: Final = "2718lab.storage.admission-response.v1" _STORAGE_ADMISSION_RECEIPT_SCHEMA: Final = "2718lab.storage.admission-receipt.v1" _STORAGE_ADMISSION_REQUEST_FIELDS: Final = frozenset( - {"schema", "correlation_id", "profile_attestation_hash", "storage_intent", "request_hash"} + { + "schema", + "correlation_id", + "profile_attestation_hash", + "storage_intent", + "request_hash", + } ) _STORAGE_ADMISSION_RECEIPT_FIELDS: Final = frozenset( - {"schema", "admission_id", "profile_attestation_hash", "storage_intent_hash", - "storage_binding_hash", "target_key", "assigned_root_identity", - "target_family_lease_id", "reserved_bytes", "reserved_files", - "free_space_before", "free_space_after_reserve", "free_space_floor", - "expires_at", "receipt_hash"} + { + "schema", + "admission_id", + "profile_attestation_hash", + "storage_intent_hash", + "storage_binding_hash", + "target_key", + "assigned_root_identity", + "target_family_lease_id", + "reserved_bytes", + "reserved_files", + "free_space_before", + "free_space_after_reserve", + "free_space_floor", + "expires_at", + "receipt_hash", + } ) _PROJECT_INDEX_ATTESTATION_SCHEMA: Final = ( project_index_attestation_protocol.ATTESTATION_SCHEMA @@ -77,9 +91,7 @@ fastlane_terminal_protocol.TERMINAL_RESULT_SCHEMA ) _FAST_LANE_TERMINAL_ACK_SCHEMA: Final = fastlane_terminal_protocol.TERMINAL_ACK_SCHEMA -_FAST_LANE_REFILL_REGISTRY_SCHEMA: Final = ( - "2718lab-devkit/fast_lane_refill_registry-v1" -) +_FAST_LANE_REFILL_REGISTRY_SCHEMA: Final = "2718lab-devkit/fast_lane_refill_registry-v1" _FAST_LANE_REFILL_REGISTRY_ACTION_PREFIX: Final = "refill-registry-" _FRAME_FIELDS: Final = frozenset( {"schema", "kind", "action_id", "session_nonce", "sequence", "payload", "mac"} @@ -148,9 +160,7 @@ _ENDPOINT = re.compile(r"[A-Za-z0-9][A-Za-z0-9._:/-]{0,255}\Z") _DIGEST = re.compile(r"sha256:[0-9a-f]{64}\Z") _MAC = re.compile(r"[0-9a-f]{64}\Z") -_FAST_LANE_REFILL_REGISTRY_ACTION = re.compile( - r"refill-registry-[0-9a-f]{64}\Z" -) +_FAST_LANE_REFILL_REGISTRY_ACTION = re.compile(r"refill-registry-[0-9a-f]{64}\Z") _FD_SELECTOR = re.compile(r"[0-9]{1,18}\Z") _WINDOWS_PIPE_SELECTOR = re.compile( r"pipe:(?Pcodex-devkit-(?P[1-9][0-9]{0,9})-" @@ -344,7 +354,10 @@ class StorageAdmissionReceipt: receipt_hash: str def to_dict(self) -> dict[str, object]: - return {name: getattr(self, name) for name in sorted(_STORAGE_ADMISSION_RECEIPT_FIELDS)} + return { + name: getattr(self, name) + for name in sorted(_STORAGE_ADMISSION_RECEIPT_FIELDS) + } @dataclass(frozen=True) @@ -855,9 +868,10 @@ def send_routing_attestation_response( attestations: Sequence[Mapping[str, object]], now: int, ) -> dict[str, object]: - if self._received_routing_attestations.get( - request.routing_request_set_hash - ) != request: + if ( + self._received_routing_attestations.get(request.routing_request_set_hash) + != request + ): raise HostBridgeError("HOST_BRIDGE_ROUTING_ATTESTATION_INVALID") unsigned: dict[str, object] = { "schema": _ROUTING_ATTESTATION_RESPONSE_SCHEMA, @@ -885,9 +899,10 @@ def send_routing_attestation_response( def receive_routing_attestation_response( self, *, request: RoutingAttestationRequest, now: int ) -> dict[str, object]: - if self._pending_routing_attestations.get( - request.routing_request_set_hash - ) != request: + if ( + self._pending_routing_attestations.get(request.routing_request_set_hash) + != request + ): raise HostBridgeError("HOST_BRIDGE_ROUTING_ATTESTATION_INVALID") try: message = self._receive_private() @@ -1139,7 +1154,8 @@ def receive_fast_lane_refill_registry_request( if ( message.kind != "fast_lane_refill_registry" or message.action_id != expected_action - or request.queue_registry_hash in self._received_fast_lane_refill_registries + or request.queue_registry_hash + in self._received_fast_lane_refill_registries ): raise HostBridgeError("HOST_BRIDGE_FAST_LANE_REFILL_INVALID") except HostBridgeError: @@ -1377,9 +1393,7 @@ def send_compiler_evidence_request( "reasoning_effort": reasoning_effort, "requested_route_pairs": list(requested_route_pairs), "assignment_skeletons": list(assignment_skeletons), - "project_index_attestation_refs": list( - project_index_attestation_refs - ), + "project_index_attestation_refs": list(project_index_attestation_refs), "routing_registry_binding_hash": routing_registry_binding_hash, "nonce": nonce, "expires_at": now + _COMPILER_EVIDENCE_TTL_SECONDS, @@ -1408,9 +1422,7 @@ def send_project_index_attestation( assert type(attestation_hash) is str if attestation_hash in self._received_project_index_attestations: raise HostBridgeError("HOST_BRIDGE_PROJECT_INDEX_ATTESTATION_INVALID") - _validate_private_packet_size( - normalized, _MAX_PROJECT_INDEX_ATTESTATION_BYTES - ) + _validate_private_packet_size(normalized, _MAX_PROJECT_INDEX_ATTESTATION_BYTES) self._send_validated_private( kind="project_index_attestation", action_id=cast(str, normalized["correlation_id"]), @@ -1425,22 +1437,14 @@ def receive_project_index_attestation(self, *, now: int) -> dict[str, object]: try: message = self._receive_private() if message.kind != "project_index_attestation": - raise HostBridgeError( - "HOST_BRIDGE_PROJECT_INDEX_ATTESTATION_INVALID" - ) - normalized = _normalize_project_index_attestation( - message.payload, now=now - ) + raise HostBridgeError("HOST_BRIDGE_PROJECT_INDEX_ATTESTATION_INVALID") + normalized = _normalize_project_index_attestation(message.payload, now=now) if message.action_id != normalized["correlation_id"]: - raise HostBridgeError( - "HOST_BRIDGE_PROJECT_INDEX_ATTESTATION_INVALID" - ) + raise HostBridgeError("HOST_BRIDGE_PROJECT_INDEX_ATTESTATION_INVALID") attestation_hash = normalized["attestation_hash"] assert type(attestation_hash) is str if attestation_hash in self._received_project_index_attestations: - raise HostBridgeError( - "HOST_BRIDGE_PROJECT_INDEX_ATTESTATION_INVALID" - ) + raise HostBridgeError("HOST_BRIDGE_PROJECT_INDEX_ATTESTATION_INVALID") except HostBridgeError: self._poison() raise @@ -1622,7 +1626,11 @@ def receive_storage_profile_response( return normalized def request_storage_admission( - self, request: Mapping[str, object], *, now: int, expires_at: int, + self, + request: Mapping[str, object], + *, + now: int, + expires_at: int, clock: Callable[[], float], ) -> StorageAdmissionReceipt: """Exchange one decision before terminal reception starts; never admit locally. @@ -1640,34 +1648,47 @@ def request_storage_admission( raise HostBridgeError("HOST_BRIDGE_STORAGE_PROFILE_INVALID") _validate_storage_admission_profile(intent, profile) if ( - type(now) is not int or type(expires_at) is not int + type(now) is not int + or type(expires_at) is not int or not 0 <= now < expires_at <= (1 << 64) - 1 or correlation in self._storage_admission_correlations ): raise HostBridgeError("HOST_BRIDGE_STORAGE_ADMISSION_INVALID") previous = self._storage_admission_decisions.get(attestation) - if previous is not None and previous.storage_intent_hash != intent.storage_intent_hash: + if ( + previous is not None + and previous.storage_intent_hash != intent.storage_intent_hash + ): raise HostBridgeError("STORAGE_LEASE_CONFLICT") # Burn before writing, including transport failure; retry must use a # fresh correlation and still obtain a decision from the Host. self._storage_admission_correlations.add(correlation) try: self._send_validated_private( - kind="storage_admission_request", action_id=correlation, payload=normalized, + kind="storage_admission_request", + action_id=correlation, + payload=normalized, deadline=deadline, ) message = self._receive_private(deadline=deadline) - if message.kind != "storage_admission_response" or message.action_id != correlation: + if ( + message.kind != "storage_admission_response" + or message.action_id != correlation + ): raise HostBridgeError("HOST_BRIDGE_STORAGE_ADMISSION_INVALID") completed_time = clock() if ( type(completed_time) not in (int, float) - or not math.isfinite(completed_time) or completed_time < now + or not math.isfinite(completed_time) + or completed_time < now ): raise HostBridgeError("HOST_BRIDGE_STORAGE_ADMISSION_INVALID") completed_at = int(completed_time) receipt = _normalize_storage_admission_response( - message.payload, request=normalized, now=completed_at, expires_at=expires_at + message.payload, + request=normalized, + now=completed_at, + expires_at=expires_at, ) if previous is not None and previous != receipt: raise HostBridgeError("HOST_BRIDGE_STORAGE_ADMISSION_INVALID") @@ -1675,19 +1696,22 @@ def request_storage_admission( self._poison() raise self._storage_admission_decisions[attestation] = receipt - self._storage_admission_completions[correlation] = _StorageAdmissionCompletion( - request_hash=cast(str, normalized["request_hash"]), - response_hash=_private_payload_hash(message.payload), - bridge_identity=self._storage_transport_identity, - expires_at=receipt.expires_at, - completed_at=completed_at, + self._storage_admission_completions[correlation] = ( + _StorageAdmissionCompletion( + request_hash=cast(str, normalized["request_hash"]), + response_hash=_private_payload_hash(message.payload), + bridge_identity=self._storage_transport_identity, + expires_at=receipt.expires_at, + completed_at=completed_at, + ) ) return receipt @contextmanager def _storage_admission_io(self, *, now: int, expires_at: int) -> Iterator[float]: if ( - type(now) is not int or type(expires_at) is not int + type(now) is not int + or type(expires_at) is not int or not 0 <= now < expires_at <= (1 << 64) - 1 ): raise HostBridgeError("HOST_BRIDGE_STORAGE_ADMISSION_INVALID") @@ -1709,7 +1733,8 @@ def has_completed_storage_profile(self, profile: Mapping[str, object]) -> bool: """Lookup transport-completed facts; caller-supplied hashes cannot enroll.""" attestation = profile.get("attestation_hash") return ( - self.is_available and type(attestation) is str + self.is_available + and type(attestation) is str and self._completed_storage_profiles.get(attestation) == profile ) @@ -1928,17 +1953,27 @@ def send_private( self._send_private(kind=kind, action_id=action_id, payload=payload) def _send_validated_private( - self, *, kind: str, action_id: str, payload: Mapping[str, object], + self, + *, + kind: str, + action_id: str, + payload: Mapping[str, object], deadline: float | None = None, ) -> None: """Write a packet only after its typed private validator has succeeded.""" if kind not in _VALIDATED_PRIVATE_KINDS: raise HostBridgeError("HOST_BRIDGE_FRAME_INVALID") - self._send_private(kind=kind, action_id=action_id, payload=payload, deadline=deadline) + self._send_private( + kind=kind, action_id=action_id, payload=payload, deadline=deadline + ) def _send_private( - self, *, kind: str, action_id: str, payload: Mapping[str, object], + self, + *, + kind: str, + action_id: str, + payload: Mapping[str, object], deadline: float | None = None, ) -> None: with self._io_lock: @@ -2152,14 +2187,20 @@ def _read_raw_frame( deadline: float | None = None, ) -> bytes: header = _read_exact( - descriptor, 4, cancel_event=cancel_event, cancel_fd=cancel_fd, + descriptor, + 4, + cancel_event=cancel_event, + cancel_fd=cancel_fd, deadline=deadline, ) size = struct.unpack("!I", header)[0] if size == 0 or size > _MAX_FRAME_BYTES: raise HostBridgeError("HOST_BRIDGE_FRAME_INVALID") return _read_exact( - descriptor, size, cancel_event=cancel_event, cancel_fd=cancel_fd, + descriptor, + size, + cancel_event=cancel_event, + cancel_fd=cancel_fd, deadline=deadline, ) @@ -2178,7 +2219,10 @@ def _write_complete_active(self, payload: bytes, *, deadline: float | None) -> N remaining = _remaining_io_time(deadline, self._cancel_event) if os.name != "nt": readable, writable, _ = select.select( - [self._cancel_read_fd], [self._write_fd], [], min(0.1, remaining) + [self._cancel_read_fd], + [self._write_fd], + [], + min(0.1, remaining), ) if readable: raise HostBridgeError("HOST_BRIDGE_UNAVAILABLE") @@ -2192,7 +2236,12 @@ def _write_complete_active(self, payload: bytes, *, deadline: float | None) -> N raise HostBridgeError("HOST_BRIDGE_UNAVAILABLE") offset += written if written == 0: - self._cancel_event.wait(min(0.05, _remaining_io_time(deadline, self._cancel_event))) + self._cancel_event.wait( + min( + 0.05, + _remaining_io_time(deadline, self._cancel_event), + ) + ) _remaining_io_time(deadline, self._cancel_event) return written = os.write(self._write_fd, payload) @@ -2260,7 +2309,12 @@ def _prune_terminal_operation_tombstones(self, *, now: int) -> None: del self._terminal_operation_tombstones[key] def _ensure_open(self) -> None: - if self._closed or self._cancel_event.is_set() or self._read_fd < 0 or self._write_fd < 0: + if ( + self._closed + or self._cancel_event.is_set() + or self._read_fd < 0 + or self._write_fd < 0 + ): raise HostBridgeError("HOST_BRIDGE_UNAVAILABLE") @@ -2480,9 +2534,10 @@ def _normalize_routing_attestation_response( raise HostBridgeError("HOST_BRIDGE_ROUTING_ATTESTATION_INVALID") task_id = original_request["task"]["task_id"] binding_hash = fastlane_routing.v5_request_binding_hash(original_request) - if item.get("task_id") != task_id or item.get( - "request_binding_hash" - ) != binding_hash: + if ( + item.get("task_id") != task_id + or item.get("request_binding_hash") != binding_hash + ): raise HostBridgeError("HOST_BRIDGE_ROUTING_ATTESTATION_INVALID") routed_request = dict(original_request) routed_request["child_route_attestation"] = item.get("attestation") @@ -3024,10 +3079,11 @@ def _validate_skeleton_package_coverage( or len(set(source_ids)) != len(source_ids) ): raise HostBridgeError("HOST_BRIDGE_FAST_LANE_REFILL_INVALID") - if not isinstance(initial_skeletons, Sequence) or isinstance( - initial_skeletons, (str, bytes, bytearray) - ) or not isinstance(remaining_skeletons, Sequence) or isinstance( - remaining_skeletons, (str, bytes, bytearray) + if ( + not isinstance(initial_skeletons, Sequence) + or isinstance(initial_skeletons, (str, bytes, bytearray)) + or not isinstance(remaining_skeletons, Sequence) + or isinstance(remaining_skeletons, (str, bytes, bytearray)) ): raise HostBridgeError("HOST_BRIDGE_FAST_LANE_REFILL_INVALID") combined = [*initial_skeletons, *remaining_skeletons] @@ -3173,8 +3229,7 @@ def _normalize_fast_lane_refill_registry_request( if not hmac.compare_digest(skeleton_package_hash, expected_package_hash): raise HostBridgeError("HOST_BRIDGE_FAST_LANE_REFILL_INVALID") if any( - item["index_context_hash"] != index_context_hash - for item in normalized_initial + item["index_context_hash"] != index_context_hash for item in normalized_initial ): raise HostBridgeError("HOST_BRIDGE_FAST_LANE_REFILL_INVALID") unsigned = dict(value) @@ -3253,7 +3308,8 @@ def build_storage_admission_request( def _normalize_storage_admission_request(value: object) -> dict[str, object]: if ( - type(value) is not dict or set(value) != _STORAGE_ADMISSION_REQUEST_FIELDS + type(value) is not dict + or set(value) != _STORAGE_ADMISSION_REQUEST_FIELDS or value.get("schema") != _STORAGE_ADMISSION_REQUEST_SCHEMA ): raise HostBridgeError("HOST_BRIDGE_STORAGE_ADMISSION_INVALID") @@ -3284,8 +3340,10 @@ def _validate_storage_admission_profile( or intent.plan_binding != profile.get("source_plan_hash") or intent.context_hash != profile.get("execution_context_hash") or intent.target_descriptor["artifact_kind"] != "fastlane-task" - or any(intent.target_descriptor[name] != profile.get(name) - for name in _STORAGE_DESCRIPTOR_FIELDS) + or any( + intent.target_descriptor[name] != profile.get(name) + for name in _STORAGE_DESCRIPTOR_FIELDS + ) ): raise HostBridgeError("STORAGE_TARGET_KEY_INVALID") @@ -3305,33 +3363,51 @@ def _normalize_storage_admission_response( _validate_private_packet_size(value, _MAX_OPERATION_PACKET_BYTES) receipt = value["receipt"] if ( - type(receipt) is not dict or set(receipt) != _STORAGE_ADMISSION_RECEIPT_FIELDS + type(receipt) is not dict + or set(receipt) != _STORAGE_ADMISSION_RECEIPT_FIELDS or receipt.get("schema") != _STORAGE_ADMISSION_RECEIPT_SCHEMA ): raise HostBridgeError("HOST_BRIDGE_STORAGE_ADMISSION_INVALID") # Both Host-minted IDs are frozen SHA-256 identities, not UUIDs or paths. - for name in ("admission_id", "target_family_lease_id", "profile_attestation_hash", - "storage_intent_hash", "storage_binding_hash", "target_key", - "assigned_root_identity", "receipt_hash"): + for name in ( + "admission_id", + "target_family_lease_id", + "profile_attestation_hash", + "storage_intent_hash", + "storage_binding_hash", + "target_key", + "assigned_root_identity", + "receipt_hash", + ): if type(receipt[name]) is not str or _DIGEST.fullmatch(receipt[name]) is None: raise HostBridgeError("HOST_BRIDGE_STORAGE_ADMISSION_INVALID") - for name in ("reserved_bytes", "reserved_files", "free_space_before", - "free_space_after_reserve", "free_space_floor", "expires_at"): + for name in ( + "reserved_bytes", + "reserved_files", + "free_space_before", + "free_space_after_reserve", + "free_space_floor", + "expires_at", + ): if type(receipt[name]) is not int or not 0 <= receipt[name] <= (1 << 64) - 1: raise HostBridgeError("HOST_BRIDGE_STORAGE_ADMISSION_INVALID") intent = parse_storage_intent(request["storage_intent"]) if ( - type(now) is not int or type(expires_at) is not int + type(now) is not int + or type(expires_at) is not int or not 0 <= now < receipt["expires_at"] <= expires_at <= (1 << 64) - 1 or receipt["profile_attestation_hash"] != request["profile_attestation_hash"] or receipt["storage_intent_hash"] != intent.storage_intent_hash - or receipt["target_key"] != _private_payload_hash(dict(intent.target_descriptor)) + or receipt["target_key"] + != _private_payload_hash(dict(intent.target_descriptor)) or receipt["reserved_bytes"] != intent.requested_bytes or receipt["reserved_files"] != intent.requested_files or receipt["free_space_floor"] == 0 - or receipt["free_space_after_reserve"] > receipt["free_space_before"] - receipt["reserved_bytes"] + or receipt["free_space_after_reserve"] + > receipt["free_space_before"] - receipt["reserved_bytes"] or receipt["free_space_after_reserve"] < receipt["free_space_floor"] - or receipt["receipt_hash"] != _private_payload_hash( + or receipt["receipt_hash"] + != _private_payload_hash( {name: item for name, item in receipt.items() if name != "receipt_hash"} ) ): @@ -3509,7 +3585,9 @@ def _normalize_storage_profile_response( profile_hash = cast(str, value["profile_hash"]) if not hmac.compare_digest(profile_hash, _private_payload_hash(unsigned)): raise HostBridgeError("HOST_BRIDGE_STORAGE_PROFILE_INVALID") - return {field_name: value[field_name] for field_name in sorted(_STORAGE_PROFILE_FIELDS)} + return { + field_name: value[field_name] for field_name in sorted(_STORAGE_PROFILE_FIELDS) + } def _normalize_compiler_evidence_response( @@ -3580,13 +3658,17 @@ def _normalize_compiler_evidence_response( ) except Exception as error: raise HostBridgeError("HOST_BRIDGE_COMPILER_EVIDENCE_INVALID") from error - requested_pairs = {(model, effort) for model, effort in request.requested_route_pairs} + requested_pairs = { + (model, effort) for model, effort in request.requested_route_pairs + } fact_pairs = { (fact.route.model, fact.route.reasoning_effort) for fact in normalized_facts } skeletons = request.assignment_skeletons skeleton_by_task = {item["task_id"]: item for item in skeletons} - if len(skeleton_by_task) != len(skeletons) or len(normalized_facts) != len(skeletons): + if len(skeleton_by_task) != len(skeletons) or len(normalized_facts) != len( + skeletons + ): raise HostBridgeError("HOST_BRIDGE_COMPILER_EVIDENCE_INVALID") for fact, mapping in zip(normalized_facts, normalized_mappings, strict=True): skeleton = skeleton_by_task.get(fact.task_id) @@ -3628,7 +3710,9 @@ def _normalize_compiler_evidence_response( if ( type(registry_binding_hash) is not str or _DIGEST.fullmatch(registry_binding_hash) is None - or not hmac.compare_digest(registry_binding_hash, _private_payload_hash(unsigned)) + or not hmac.compare_digest( + registry_binding_hash, _private_payload_hash(unsigned) + ) ): raise HostBridgeError("HOST_BRIDGE_COMPILER_EVIDENCE_INVALID") _validate_private_packet_size(value, _MAX_COMPILER_EVIDENCE_BYTES) @@ -4120,19 +4204,30 @@ def _nonblocking_pipe_writer(descriptor: int) -> Iterator[Callable[[bytes], int] handle = msvcrt.get_osfhandle(descriptor) get_state = kernel32.GetNamedPipeHandleStateW get_state.argtypes = ( - ctypes.c_void_p, ctypes.POINTER(ctypes.c_ulong), ctypes.c_void_p, - ctypes.c_void_p, ctypes.c_void_p, ctypes.c_void_p, ctypes.c_ulong, + ctypes.c_void_p, + ctypes.POINTER(ctypes.c_ulong), + ctypes.c_void_p, + ctypes.c_void_p, + ctypes.c_void_p, + ctypes.c_void_p, + ctypes.c_ulong, ) get_state.restype = ctypes.c_int set_state = kernel32.SetNamedPipeHandleState set_state.argtypes = ( - ctypes.c_void_p, ctypes.POINTER(ctypes.c_ulong), ctypes.c_void_p, ctypes.c_void_p, + ctypes.c_void_p, + ctypes.POINTER(ctypes.c_ulong), + ctypes.c_void_p, + ctypes.c_void_p, ) set_state.restype = ctypes.c_int write_file = kernel32.WriteFile write_file.argtypes = ( - ctypes.c_void_p, ctypes.c_void_p, ctypes.c_ulong, - ctypes.POINTER(ctypes.c_ulong), ctypes.c_void_p, + ctypes.c_void_p, + ctypes.c_void_p, + ctypes.c_ulong, + ctypes.POINTER(ctypes.c_ulong), + ctypes.c_void_p, ) write_file.restype = ctypes.c_int @@ -4250,12 +4345,23 @@ def _windows_pipe_available_bytes(descriptor: int) -> int: kernel32 = ctypes.WinDLL("kernel32", use_last_error=True) peek = kernel32.PeekNamedPipe peek.argtypes = ( - ctypes.c_void_p, ctypes.c_void_p, ctypes.c_ulong, - ctypes.c_void_p, ctypes.POINTER(ctypes.c_ulong), ctypes.c_void_p, + ctypes.c_void_p, + ctypes.c_void_p, + ctypes.c_ulong, + ctypes.c_void_p, + ctypes.POINTER(ctypes.c_ulong), + ctypes.c_void_p, ) peek.restype = ctypes.c_int available = ctypes.c_ulong() - if not peek(msvcrt.get_osfhandle(descriptor), None, 0, None, ctypes.byref(available), None): + if not peek( + msvcrt.get_osfhandle(descriptor), + None, + 0, + None, + ctypes.byref(available), + None, + ): raise HostBridgeError("HOST_BRIDGE_UNAVAILABLE") return available.value except (AttributeError, ImportError, OSError, OverflowError, ValueError) as error: diff --git a/mcp-tools/devkit_runtime/host_session.py b/mcp-tools/devkit_runtime/host_session.py index b231ba7..ad2099d 100644 --- a/mcp-tools/devkit_runtime/host_session.py +++ b/mcp-tools/devkit_runtime/host_session.py @@ -350,11 +350,7 @@ def __init__( self._clock = clock self._last_trusted_clock: float | None = None self._closed = False - self._frozen = ( - bridge is None - or not callable(clock) - or not bridge.is_available - ) + self._frozen = bridge is None or not callable(clock) or not bridge.is_available self._attested_capabilities: dict[tuple[str, str], _CapabilityRecord] = {} self._consumed_predecessors: set[tuple[str, str, str, str, str]] = set() self._last_unavailable: HostUnavailableFacts | None = None @@ -427,7 +423,9 @@ def resolve_capability_snapshot_v2( or not self.is_available or type(call_intent_hash) is not str or len(call_intent_hash) != 64 - or any(character not in "0123456789abcdef" for character in call_intent_hash) + or any( + character not in "0123456789abcdef" for character in call_intent_hash + ) or type(preparation_id) is not str or _IDENTIFIER.fullmatch(preparation_id) is None or ( @@ -642,9 +640,14 @@ def prepare_compiler_evidence( # Only the actual authenticated profile round trip can enroll a # reference; an injected resolver/provider cannot mint authority. bridge = self._bridge - if bridge is not None and binding_resolver == self._resolve_bridge_compiler_invocation: - budgets = {task: (byte_count, files) for task, byte_count, files - in material.storage_budget_bindings} + if ( + bridge is not None + and binding_resolver == self._resolve_bridge_compiler_invocation + ): + budgets = { + task: (byte_count, files) + for task, byte_count, files in material.storage_budget_bindings + } completed_profiles: dict[str, _CompletedStorageProfile] = {} for profile in material.storage_profiles: if not bridge.has_completed_storage_profile(profile): @@ -652,10 +655,15 @@ def prepare_compiler_evidence( attestation = cast(str, profile["attestation_hash"]) byte_count, files = budgets[cast(str, profile["task_id"])] completed = _CompletedStorageProfile( - profile=dict(profile), bridge=bridge, expires_at=int(expires_at), - requested_bytes=byte_count, requested_files=files, + profile=dict(profile), + bridge=bridge, + expires_at=int(expires_at), + requested_bytes=byte_count, + requested_files=files, ) - previous = completed_profiles.get(attestation) or self._completed_storage_profiles.get(attestation) + previous = completed_profiles.get( + attestation + ) or self._completed_storage_profiles.get(attestation) if previous is not None and previous != completed: return _NO_SAFE_WORK completed_profiles[attestation] = completed @@ -688,7 +696,10 @@ def bind_compiler_request( or _IDENTIFIER.fullmatch(preparation_id) is None or type(call_intent_hash) is not str or len(call_intent_hash) != 64 - or any(character not in "0123456789abcdef" for character in call_intent_hash) + or any( + character not in "0123456789abcdef" + for character in call_intent_hash + ) or not _is_hash(request_hash) or not _is_hash(routing_registry_binding_hash) or reasoning_effort not in {"low", "medium", "high", "xhigh", "max"} @@ -804,9 +815,7 @@ def _resolve_bridge_compiler_invocation( ): skeleton_mapping = _mapping(skeleton) index_mapping = _mapping(index_ref) - task_id = _required_fast_lane_task_id( - skeleton_mapping.get("task_id") - ) + task_id = _required_fast_lane_task_id(skeleton_mapping.get("task_id")) source_plan_hash = _required_hash( skeleton_mapping.get("source_plan_hash") ) @@ -817,7 +826,8 @@ def _resolve_bridge_compiler_invocation( index_mapping.get("task_id") != task_id or fact.task_id != task_id or fact.source_plan_hash != source_plan_hash - or fact.index_context_hash != skeleton_mapping.get("index_context_hash") + or fact.index_context_hash + != skeleton_mapping.get("index_context_hash") ): return None profile_request = bridge.send_storage_profile_request( @@ -835,12 +845,10 @@ def _resolve_bridge_compiler_invocation( request_hash=_required_hash(response["request_hash"]), reasoning_effort=str(response["reasoning_effort"]), verified_route_result_hashes=tuple( - _required_hash(value) - for value in route_hashes + _required_hash(value) for value in route_hashes ), verified_lease_scope_bindings=tuple( - _required_hash(value) - for value in lease_hashes + _required_hash(value) for value in lease_hashes ), dispatch_facts=dispatch_facts, dispatch_binding_hashes=tuple( @@ -914,21 +922,27 @@ def request_storage_admission( return "STORAGE_TARGET_KEY_INVALID" try: now = int(self._read_trusted_clock()) - if now >= completed.expires_at or not bridge.has_completed_storage_profile(completed.profile): + if ( + now >= completed.expires_at + or not bridge.has_completed_storage_profile(completed.profile) + ): return "STORAGE_STAT_UNAVAILABLE" if type(intent) is not StorageIntent: return "STORAGE_TARGET_KEY_INVALID" parsed = parse_storage_intent(intent.to_dict()) _validate_storage_admission_profile(parsed, completed.profile) if (parsed.requested_bytes, parsed.requested_files) != ( - completed.requested_bytes, completed.requested_files + completed.requested_bytes, + completed.requested_files, ): return "STORAGE_LEASE_CONFLICT" request = build_storage_admission_request( parsed, profile_attestation_hash=profile_attestation_hash ) receipt = bridge.request_storage_admission( - request, now=now, expires_at=completed.expires_at, + request, + now=now, + expires_at=completed.expires_at, clock=self._read_trusted_clock, ) # Re-read the trusted clock after blocking I/O. A valid frame @@ -939,7 +953,10 @@ def request_storage_admission( except StorageIntentError as error: return error.code except HostBridgeError as error: - if error.code in {"STORAGE_TARGET_KEY_INVALID", "STORAGE_LEASE_CONFLICT"}: + if error.code in { + "STORAGE_TARGET_KEY_INVALID", + "STORAGE_LEASE_CONFLICT", + }: return error.code self._freeze() return "STORAGE_STAT_UNAVAILABLE" @@ -1008,9 +1025,7 @@ def bind_storage_intent_proof( return False try: now = self._read_trusted_clock() - budget_bindings = _normalized_storage_budget_bindings( - storage_budgets - ) + budget_bindings = _normalized_storage_budget_bindings(storage_budgets) if ( now >= material.expires_at or not budget_bindings @@ -1030,8 +1045,7 @@ def bind_storage_intent_proof( preparation_id=material.preparation_id, ) intent_hashes = tuple( - _required_hash(intent["storage_intent_hash"]) - for intent in intents + _required_hash(intent["storage_intent_hash"]) for intent in intents ) rebound = replace( material, @@ -1041,9 +1055,7 @@ def bind_storage_intent_proof( ) rebound = replace( rebound, - binding_hash=_hash( - _compiler_invocation_binding_material(rebound) - ), + binding_hash=_hash(_compiler_invocation_binding_material(rebound)), ) except (KeyError, TypeError, ValueError): return False @@ -1067,10 +1079,8 @@ def consume_compiler_evidence(self, evidence: object) -> object | str: now = self._read_trusted_clock() except (TypeError, ValueError): return _NO_SAFE_WORK - if ( - now >= material.expires_at - or material.binding_hash - != _hash(_compiler_invocation_binding_material(material)) + if now >= material.expires_at or material.binding_hash != _hash( + _compiler_invocation_binding_material(material) ): return _NO_SAFE_WORK return material @@ -1112,7 +1122,10 @@ def send_fast_lane_dispatch_batch( if ( type(call_intent_hash) is not str or len(call_intent_hash) != 64 - or any(character not in "0123456789abcdef" for character in call_intent_hash) + or any( + character not in "0123456789abcdef" + for character in call_intent_hash + ) or type(preparation_id) is not str or _IDENTIFIER.fullmatch(preparation_id) is None ): @@ -1142,7 +1155,9 @@ def send_fast_lane_dispatch_batch( "lease_epoch": assignment.get("lease_epoch"), "task_version": assignment.get("task_version"), "assignment_token": assignment.get("assignment_token"), - "dispatch_binding_hash": assignment.get("dispatch_binding_hash"), + "dispatch_binding_hash": assignment.get( + "dispatch_binding_hash" + ), "routing_result_hash": route.get("routing_result_hash"), "worktree_identity": assignment.get("worktree_identity"), "worktree_base": assignment.get("worktree_base"), @@ -1175,8 +1190,12 @@ def send_fast_lane_dispatch_batch( now=now, ) self._pending_fast_lane_terminals.update(pending) - if refill_callback is not None and not self.start_fast_lane_terminal_receiver( - batch_hash=cast(str, batch_hash), refill_callback=refill_callback + if ( + refill_callback is not None + and not self.start_fast_lane_terminal_receiver( + batch_hash=cast(str, batch_hash), + refill_callback=refill_callback, + ) ): raise ValueError("Fast Lane terminal receiver was not started") return receipt @@ -1303,7 +1322,9 @@ def start_fast_lane_terminal_receiver( with self._compiler_evidence_lock: if ( not self.is_available - or not any(key[0] == batch_hash for key in self._pending_fast_lane_terminals) + or not any( + key[0] == batch_hash for key in self._pending_fast_lane_terminals + ) or batch_hash in self._fast_lane_refill_callbacks ): return False @@ -1651,8 +1672,7 @@ def _resolve_host_scheduler_topology( type(action) is not HostAuthoritativeActionFact or not _is_hash(action.plan_hash) or _IDENTIFIER.fullmatch(action.task_id) is None - or action.kind - not in {"implementation", "prewarm", "design"} + or action.kind not in {"implementation", "prewarm", "design"} or _IDENTIFIER.fullmatch(action.target) is None or not _is_hash(action.group_binding_hash) or action.plan_hash != fact.plan_hash @@ -1681,9 +1701,7 @@ def _resolve_host_scheduler_topology( ) for action in fact.authoritative_actions ) - or len( - {action.task_id for action in fact.authoritative_actions} - ) + or len({action.task_id for action in fact.authoritative_actions}) != len(fact.authoritative_actions) or fact.scheduler_id in facts_by_scheduler ): @@ -2155,7 +2173,10 @@ def _normalized_storage_profiles( if mapping["profile_hash"] != _hash(unsigned): raise ValueError("compiler storage profile hash is invalid") normalized.append( - {field_name: mapping[field_name] for field_name in sorted(_STORAGE_PROFILE_FIELDS)} + { + field_name: mapping[field_name] + for field_name in sorted(_STORAGE_PROFILE_FIELDS) + } ) task_ids = [cast(str, profile["task_id"]) for profile in normalized] if len(task_ids) != len(set(task_ids)): @@ -2192,8 +2213,7 @@ def _normalized_storage_intent_proof( for fact in dispatch_facts ) fact_source_plan_hashes = tuple( - _required_hash(getattr(fact, "source_plan_hash")) - for fact in dispatch_facts + _required_hash(getattr(fact, "source_plan_hash")) for fact in dispatch_facts ) except (AttributeError, TypeError, ValueError) as error: raise ValueError("compiler storage dispatch facts are invalid") from error @@ -2318,7 +2338,8 @@ def _normalized_compiler_invocation_binding( if bool(storage_budget_bindings) != bool(storage_profiles): raise ValueError("compiler storage proof is incomplete") if storage_budget_bindings and { - task_id for task_id, _requested_bytes, _requested_files in storage_budget_bindings + task_id + for task_id, _requested_bytes, _requested_files in storage_budget_bindings } != {cast(str, profile["task_id"]) for profile in storage_profiles}: raise ValueError("compiler storage profile budget bindings are invalid") return _CompilerInvocationBinding( diff --git a/mcp-tools/orchestrator/store.py b/mcp-tools/orchestrator/store.py index 51528ad..847adb0 100644 --- a/mcp-tools/orchestrator/store.py +++ b/mcp-tools/orchestrator/store.py @@ -288,7 +288,13 @@ def _validate_quiescent_wal_pair( _u32(frame_header, 20, byteorder="big"), ): raise StoreError("orchestrator store is not prepared") - if frame_count and _u32(wal_bytes, 32 + (frame_count - 1) * (24 + page_size) + 4, byteorder="big") == 0: + if ( + frame_count + and _u32( + wal_bytes, 32 + (frame_count - 1) * (24 + page_size) + 4, byteorder="big" + ) + == 0 + ): raise StoreError("orchestrator store is not prepared") if frame_count and frame_checksum != ( _u32(wal_index, 24, byteorder=native_byteorder), @@ -305,9 +311,7 @@ def _schema_version_from_connection(connection: sqlite3.Connection) -> int | Non return None if len(row) != 1 or row[0] != "table": raise StoreError("orchestrator store is not prepared") - rows = connection.execute( - "SELECT key, value FROM schema_metadata" - ).fetchall() + rows = connection.execute("SELECT key, value FROM schema_metadata").fetchall() if len(rows) != 1 or rows[0][0] != "schema_version": raise StoreError("orchestrator store is not prepared") try: @@ -327,7 +331,9 @@ def _main_database_path(connection: sqlite3.Connection) -> Path: raise StoreError("orchestrator store is not prepared") -def _transition_to_delete_journal(connection: sqlite3.Connection, database: Path) -> None: +def _transition_to_delete_journal( + connection: sqlite3.Connection, database: Path +) -> None: """Checkpoint a valid legacy WAL exactly, then make DELETE durable.""" row = connection.execute("PRAGMA journal_mode").fetchone() if row is None or len(row) != 1 or not isinstance(row[0], str): @@ -364,6 +370,7 @@ def _transition_to_delete_journal(connection: sqlite3.Connection, database: Path for sidecar in _sqlite_wal_sidecars(database): _require_absent_sqlite_sidecar(sidecar) + _ATLAS_OUTBOX_REQUIRED_CHECKS = frozenset( { ("ingestion_key", "=", "payload_hash"), @@ -392,7 +399,11 @@ def _sqlite_check_expressions(table_sql: object) -> frozenset[tuple[str, ...]]: checks: set[tuple[str, ...]] = set() index = 0 while index < len(tokens): - if tokens[index] != "check" or index + 1 == len(tokens) or tokens[index + 1] != "(": + if ( + tokens[index] != "check" + or index + 1 == len(tokens) + or tokens[index + 1] != "(" + ): index += 1 continue expression, index = _sqlite_parenthesized_tokens(tokens, index + 1) @@ -546,9 +557,7 @@ def _foreign_key_contract( ) -def _table_sql( - executor: sqlite3.Connection | sqlite3.Cursor, table_name: str -) -> str: +def _table_sql(executor: sqlite3.Connection | sqlite3.Cursor, table_name: str) -> str: row = executor.execute( "SELECT sql FROM sqlite_master WHERE type = 'table' AND name = ?", (table_name,), @@ -1014,7 +1023,13 @@ def _schema_metadata_layout( for row in sorted(columns.values(), key=lambda row: int(row["pk"])) if int(row["pk"]) ) - except (IndexError, OSError, TypeError, ValueError, sqlite3.DatabaseError) as error: + except ( + IndexError, + OSError, + TypeError, + ValueError, + sqlite3.DatabaseError, + ) as error: raise StoreError("orchestrator schema is corrupt") from error return columns, primary_key @@ -1069,8 +1084,7 @@ def _migrate_schema_metadata_key_not_null( or type(metadata_rows[0]["key"]) is not str or metadata_rows[0]["key"] != "schema_version" or type(metadata_rows[0]["value"]) is not str - or metadata_rows[0]["value"] - not in {"12", str(cls._SCHEMA_VERSION)} + or metadata_rows[0]["value"] not in {"12", str(cls._SCHEMA_VERSION)} ): raise StoreError("orchestrator store is not prepared") return @@ -1109,9 +1123,7 @@ def _migrate_schema_metadata_key_not_null( """ ) cursor.execute("DROP TABLE schema_metadata") - cursor.execute( - "ALTER TABLE schema_metadata_v12 RENAME TO schema_metadata" - ) + cursor.execute("ALTER TABLE schema_metadata_v12 RENAME TO schema_metadata") except sqlite3.IntegrityError as error: raise StoreError("orchestrator store is not prepared") from error except sqlite3.DatabaseError as error: @@ -1188,7 +1200,13 @@ def validate_prepared_connection(cls, connection: sqlite3.Connection) -> None: foreign_key_violation = connection.execute( "PRAGMA foreign_key_check" ).fetchone() - except (IndexError, OSError, TypeError, ValueError, sqlite3.DatabaseError) as error: + except ( + IndexError, + OSError, + TypeError, + ValueError, + sqlite3.DatabaseError, + ) as error: raise StoreError("orchestrator schema is corrupt") from error if ( not required_tables.issubset(tables) @@ -1231,7 +1249,9 @@ def _validate_atlas_outbox_shape( ) } ) - unique_indexes = set(_unique_index_contract(connection, "atlas_ingestion_outbox")) + unique_indexes = set( + _unique_index_contract(connection, "atlas_ingestion_outbox") + ) if ( _table_column_contract(connection, "atlas_ingestion_outbox") != expected_columns @@ -1242,7 +1262,9 @@ def _validate_atlas_outbox_shape( ) or _foreign_key_contract(connection, "atlas_ingestion_outbox") != expected_foreign_keys - or _sqlite_check_expressions(_table_sql(connection, "atlas_ingestion_outbox")) + or _sqlite_check_expressions( + _table_sql(connection, "atlas_ingestion_outbox") + ) != _ATLAS_OUTBOX_REQUIRED_CHECKS ): raise StoreError("orchestrator store is not prepared") @@ -1309,8 +1331,7 @@ def _validate_atlas_finalization_shape( """Verify the immutable certificate table and its exact anchors.""" expected_columns = tuple( (name, column_type.casefold(), not_null, primary_key, 0) - for name, column_type, not_null, primary_key - in _ATLAS_FINALIZATION_COLUMN_CONTRACT + for name, column_type, not_null, primary_key in _ATLAS_FINALIZATION_COLUMN_CONTRACT ) expected_foreign_keys = frozenset( { @@ -1357,7 +1378,8 @@ def _validate_atlas_finalization_shape( if type(row["sql"]) is str } if ( - _table_column_contract(connection, "atlas_finalizations") != expected_columns + _table_column_contract(connection, "atlas_finalizations") + != expected_columns or set(_unique_index_contract(connection, "atlas_finalizations")) != {("finalization_hash",), ("acceptance_id",), ("ingestion_key",)} or _foreign_key_contract(connection, "atlas_finalizations") @@ -1369,9 +1391,7 @@ def _validate_atlas_finalization_shape( raise StoreError("orchestrator store is not prepared") @classmethod - def _migrate_atlas_finalization_binding( - cls, cursor: sqlite3.Cursor - ) -> None: + def _migrate_atlas_finalization_binding(cls, cursor: sqlite3.Cursor) -> None: """Upgrade only the previous v13 certificate layout to exact outbox binding.""" try: @@ -1381,8 +1401,7 @@ def _migrate_atlas_finalization_binding( pass expected_columns = tuple( (name, column_type.casefold(), not_null, primary_key, 0) - for name, column_type, not_null, primary_key - in _ATLAS_FINALIZATION_COLUMN_CONTRACT + for name, column_type, not_null, primary_key in _ATLAS_FINALIZATION_COLUMN_CONTRACT ) expected_previous_foreign_keys = frozenset( { @@ -1653,14 +1672,20 @@ def admit_external_bootstrap( } ) with self._transaction() as cursor: - batch_exists = cursor.execute( - "SELECT 1 FROM external_bootstrap_batches WHERE batch_hash = ?", - (batch.batch_hash,), - ).fetchone() is not None - grant_exists = cursor.execute( - "SELECT 1 FROM external_dispatch_grants WHERE grant_id = ?", - (grant.grant_id,), - ).fetchone() is not None + batch_exists = ( + cursor.execute( + "SELECT 1 FROM external_bootstrap_batches WHERE batch_hash = ?", + (batch.batch_hash,), + ).fetchone() + is not None + ) + grant_exists = ( + cursor.execute( + "SELECT 1 FROM external_dispatch_grants WHERE grant_id = ?", + (grant.grant_id,), + ).fetchone() + is not None + ) self._require_external_payload( cursor, "external_bootstrap_descriptors", @@ -1690,7 +1715,9 @@ def admit_external_bootstrap( existing_idempotency is not None and str(existing_idempotency["batch_hash"]) != batch.batch_hash ): - raise ExternalBootstrapConflictError("bootstrap idempotency binding conflicts") + raise ExternalBootstrapConflictError( + "bootstrap idempotency binding conflicts" + ) existing_grant_binding = cursor.execute( """ SELECT grant_id FROM external_dispatch_grants @@ -1706,7 +1733,9 @@ def admit_external_bootstrap( existing_grant_binding is not None and str(existing_grant_binding["grant_id"]) != grant.grant_id ): - raise ExternalBootstrapConflictError("external dispatch grant binding conflicts") + raise ExternalBootstrapConflictError( + "external dispatch grant binding conflicts" + ) existing_composite_binding = cursor.execute( "SELECT * FROM external_dispatch_grant_bindings WHERE grant_id = ?", (grant.grant_id,), @@ -1755,7 +1784,9 @@ def admit_external_bootstrap( ), ) for item in batch.items: - item_payload = _canonical_payload_json(self._external_batch_item_payload(item)) + item_payload = _canonical_payload_json( + self._external_batch_item_payload(item) + ) row = cursor.execute( """ SELECT payload_json FROM external_bootstrap_batch_items @@ -1764,7 +1795,9 @@ def admit_external_bootstrap( (batch.batch_hash, item.item_index), ).fetchone() if row is not None and str(row["payload_json"]) != item_payload: - raise ExternalBootstrapConflictError("bootstrap batch item conflicts") + raise ExternalBootstrapConflictError( + "bootstrap batch item conflicts" + ) cursor.execute( """ INSERT OR IGNORE INTO external_bootstrap_batch_items @@ -1870,9 +1903,16 @@ def admit_external_bootstrap( (grant.grant_id,), ).fetchone() self._validate_external_grant_binding_at_read(grant_row, cursor=cursor) - return descriptor, batch, self._external_outbox_from_row(outbox_row), self._external_grant_from_row(grant_row) + return ( + descriptor, + batch, + self._external_outbox_from_row(outbox_row), + self._external_grant_from_row(grant_row), + ) - def get_external_dispatch_grant(self, grant_id: str) -> ExternalDispatchGrant | None: + def get_external_dispatch_grant( + self, grant_id: str + ) -> ExternalDispatchGrant | None: row = self._connection.execute( "SELECT * FROM external_dispatch_grants WHERE grant_id = ?", (grant_id,) ).fetchone() @@ -1920,7 +1960,9 @@ def consume_external_dispatch_grant( ), ) if cursor.rowcount != 1: - raise ExternalDispatchGrantError("external dispatch grant is expired, replayed, or unbound") + raise ExternalDispatchGrantError( + "external dispatch grant is expired, replayed, or unbound" + ) row = cursor.execute( "SELECT * FROM external_dispatch_grants WHERE grant_id = ?", (grant_id,) ).fetchone() @@ -2368,7 +2410,9 @@ def enqueue_message( workflow_id=workflow_id, recipient_task_id=recipient_task_id, recipient_epoch=( - int(recipient_lease["epoch"]) if recipient_lease is not None else None + int(recipient_lease["epoch"]) + if recipient_lease is not None + else None ), now=now_utc, incoming_bytes=artifact_size, @@ -2460,7 +2504,9 @@ def enqueue_role_envelope( datetime.fromisoformat(now_utc) + timedelta(seconds=ttl_seconds) ).isoformat() with self._transaction() as cursor: - self._require_current_lease(cursor, sender_task_id, owner, epoch, now=now_utc) + self._require_current_lease( + cursor, sender_task_id, owner, epoch, now=now_utc + ) self._require_task_in_workflow( cursor, workflow_id, sender_task_id, RoleEnvelopeForbiddenError ) @@ -2792,7 +2838,9 @@ def record_host_archive_result( "SELECT owner_role FROM tasks WHERE id = ?", (task_id,) ).fetchone() if task is None or str(task["owner_role"]) != "worker": - raise RoleEnvelopeForbiddenError("archive report is not owned by a worker") + raise RoleEnvelopeForbiddenError( + "archive report is not owned by a worker" + ) existing = cursor.execute( "SELECT * FROM host_operation_receipts WHERE operation_id = ?", (operation_id,), @@ -4820,7 +4868,9 @@ def _role_direction( try: return RoleEnvelopeDirection(value) except (TypeError, ValueError) as error: - raise RoleEnvelopeInvalidError("role envelope direction is not supported") from error + raise RoleEnvelopeInvalidError( + "role envelope direction is not supported" + ) from error @classmethod def _safe_role_identifier(cls, value: object) -> bool: @@ -4833,7 +4883,10 @@ def _safe_role_identifier(cls, value: object) -> bool: @classmethod def _require_role_hash(cls, value: object, label: str) -> str: - if not isinstance(value, str) or cls._SHA256_IDENTIFIER_PATTERN.fullmatch(value) is None: + if ( + not isinstance(value, str) + or cls._SHA256_IDENTIFIER_PATTERN.fullmatch(value) is None + ): raise RoleEnvelopeInvalidError(f"{label} must be a sha256 reference") return value @@ -4853,9 +4906,7 @@ def _role_hashes(cls, value: object, label: str) -> tuple[str, ...]: return hashes @classmethod - def _role_risk_items( - cls, value: object - ) -> tuple[RoleRiskItem, ...]: + def _role_risk_items(cls, value: object) -> tuple[RoleRiskItem, ...]: if not isinstance(value, tuple) or len(value) > cls._MAX_ROLE_RISK_ITEMS: raise RoleEnvelopeInvalidError("risk items are outside the bounded schema") items: list[RoleRiskItem] = [] @@ -4884,7 +4935,11 @@ def _role_risk_items( ): raise RoleEnvelopeInvalidError("risk item is not a bounded reference") items.append( - RoleRiskItem(code, severity, cls._require_role_hash(evidence_hash, "risk evidence")) + RoleRiskItem( + code, + severity, + cls._require_role_hash(evidence_hash, "risk evidence"), + ) ) if len({item.code for item in items}) != len(items): raise RoleEnvelopeInvalidError("risk item codes must be unique") @@ -4905,18 +4960,28 @@ def _role_coordinator_binding( now: str, ) -> tuple[str, int]: if direction is RoleEnvelopeDirection.COORDINATOR_TO_WORKER: - if coordinator_task_id not in (None, sender_task_id) or coordinator_epoch not in ( + if coordinator_task_id not in ( + None, + sender_task_id, + ) or coordinator_epoch not in ( None, sender_epoch, ): - raise RoleEnvelopeInvalidError("coordinator binding does not match sender") + raise RoleEnvelopeInvalidError( + "coordinator binding does not match sender" + ) return sender_task_id, sender_epoch if direction is RoleEnvelopeDirection.WORKER_TO_COORDINATOR: - if coordinator_task_id not in (None, recipient_task_id) or coordinator_epoch not in ( + if coordinator_task_id not in ( + None, + recipient_task_id, + ) or coordinator_epoch not in ( None, recipient_epoch, ): - raise RoleEnvelopeInvalidError("coordinator binding does not match recipient") + raise RoleEnvelopeInvalidError( + "coordinator binding does not match recipient" + ) return recipient_task_id, recipient_epoch if ( not self._safe_role_identifier(coordinator_task_id) @@ -4924,7 +4989,9 @@ def _role_coordinator_binding( or isinstance(coordinator_epoch, bool) or coordinator_epoch < 1 ): - raise RoleEnvelopeInvalidError("peer envelope needs an exact coordinator binding") + raise RoleEnvelopeInvalidError( + "peer envelope needs an exact coordinator binding" + ) self._require_task_in_workflow( cursor, workflow_id, coordinator_task_id, RoleEnvelopeForbiddenError ) @@ -4955,13 +5022,18 @@ def _require_role_task_roles( RoleEnvelopeDirection.PEER_TO_PEER: ("worker", "worker"), }[direction] if (sender_role, recipient_role) != expected: - raise RoleEnvelopeInvalidError("role direction does not match sender and recipient") + raise RoleEnvelopeInvalidError( + "role direction does not match sender and recipient" + ) rows = cursor.execute( "SELECT id, owner_role FROM tasks WHERE id IN (?, ?)", (sender_task_id, recipient_task_id), ).fetchall() roles = {str(row["id"]): str(row["owner_role"]) for row in rows} - if roles.get(sender_task_id) != sender_role or roles.get(recipient_task_id) != recipient_role: + if ( + roles.get(sender_task_id) != sender_role + or roles.get(recipient_task_id) != recipient_role + ): raise RoleEnvelopeForbiddenError("task role is not authoritative") @staticmethod @@ -4973,11 +5045,7 @@ def _require_current_recipient_lease( row = cursor.execute( "SELECT epoch, expires_at FROM leases WHERE task_id = ?", (task_id,) ).fetchone() - if ( - row is None - or int(row["epoch"]) != epoch - or str(row["expires_at"]) <= now - ): + if row is None or int(row["epoch"]) != epoch or str(row["expires_at"]) <= now: raise StaleLeaseError(f"lease is stale for task {task_id!r}") def _require_role_peer_capability( @@ -5007,7 +5075,9 @@ def _require_role_peer_capability( or not isinstance(capability, str) or not secrets.compare_digest(str(row["capability"]), capability) ): - raise CapabilityInvalidError("delivery capability is not valid for this peer") + raise CapabilityInvalidError( + "delivery capability is not valid for this peer" + ) return _payload_hash(capability) def _role_envelope_payload( @@ -5071,7 +5141,9 @@ def _role_envelope_payload( terminal = self._require_role_hash(terminal_result_hash, "terminal result") risks = self._role_risk_items(risk_items) if not {risk.evidence_hash for risk in risks}.issubset(set(evidence)): - raise RoleEnvelopeInvalidError("risk evidence must be an envelope evidence reference") + raise RoleEnvelopeInvalidError( + "risk evidence must be an envelope evidence reference" + ) if direction is RoleEnvelopeDirection.COORDINATOR_TO_WORKER: if ( not card_hash @@ -5083,7 +5155,9 @@ def _role_envelope_payload( or risks or recipient_capability_hash ): - raise RoleEnvelopeInvalidError("coordinator envelope schema is not exact") + raise RoleEnvelopeInvalidError( + "coordinator envelope schema is not exact" + ) elif direction is RoleEnvelopeDirection.WORKER_TO_COORDINATOR: if ( card_hash @@ -5172,7 +5246,9 @@ def _require_role_direction_references( "SELECT card_hash FROM task_cards WHERE task_id = ?", (recipient_task_id,) ).fetchone() if card is None or str(card["card_hash"]) != payload["task_card_hash"]: - raise RoleEnvelopeForbiddenError("task card reference is not recipient-bound") + raise RoleEnvelopeForbiddenError( + "task card reference is not recipient-bound" + ) allowed_contracts = { str(row["contract_hash"]) for row in cursor.execute( @@ -5181,8 +5257,12 @@ def _require_role_direction_references( ).fetchall() } requested_contracts = tuple(str(item) for item in payload["contract_hashes"]) - if not requested_contracts or not set(requested_contracts).issubset(allowed_contracts): - raise RoleEnvelopeForbiddenError("contract references are not recipient-bound") + if not requested_contracts or not set(requested_contracts).issubset( + allowed_contracts + ): + raise RoleEnvelopeForbiddenError( + "contract references are not recipient-bound" + ) def _require_live_role_assignment( self, @@ -5234,7 +5314,9 @@ def _require_live_role_assignment( ), ).fetchone() if row is None: - raise RoleEnvelopeForbiddenError("no live coordinator assignment matches envelope") + raise RoleEnvelopeForbiddenError( + "no live coordinator assignment matches envelope" + ) def _role_reference_bytes( self, @@ -6266,7 +6348,9 @@ def _validate_external_identifier(cls, value: str, label: str) -> None: not isinstance(value, str) or cls._SAFE_ACCEPTANCE_IDENTIFIER_PATTERN.fullmatch(value) is None ): - raise ExternalBootstrapConflictError(f"{label} is outside the bounded schema") + raise ExternalBootstrapConflictError( + f"{label} is outside the bounded schema" + ) @classmethod def _external_descriptor_payload( @@ -6306,7 +6390,9 @@ def _external_batch_item_payload( } @classmethod - def _external_batch_payload(cls, batch: ExternalBootstrapBatch) -> dict[str, object]: + def _external_batch_payload( + cls, batch: ExternalBootstrapBatch + ) -> dict[str, object]: return { "availability": batch.availability, "batch_hash": batch.batch_hash, @@ -6531,7 +6617,10 @@ def _validate_external_grant_binding_at_read( raise ValueError("grant does not bind a batch") batch = self._external_batch_from_row( batch_row, - tuple(self._external_batch_item_from_row(item_row) for item_row in item_rows), + tuple( + self._external_batch_item_from_row(item_row) + for item_row in item_rows + ), ) if ( descriptor.descriptor_hash != grant.descriptor_hash @@ -6600,7 +6689,9 @@ def _validate_external_grant_binding_at_read( ) ) ): - raise ValueError("external bootstrap commitment chain is absent or mismatched") + raise ValueError( + "external bootstrap commitment chain is absent or mismatched" + ) except ( KeyError, TypeError, @@ -6632,7 +6723,9 @@ def _validate_external_bootstrap_records( or not batch.items or len(batch.items) > cls._MAX_EXTERNAL_BOOTSTRAP_BATCH_ITEMS ): - raise ExternalBootstrapConflictError("external bootstrap batch is not pending") + raise ExternalBootstrapConflictError( + "external bootstrap batch is not pending" + ) cls._validate_external_hash(batch.batch_hash, "batch_hash") cls._validate_external_hash(batch.descriptor_hash, "descriptor_hash") cls._validate_external_identifier(batch.idempotency_key, "idempotency_key") @@ -6648,14 +6741,18 @@ def _validate_external_bootstrap_records( or isinstance(item.lease_epoch, bool) or item.lease_epoch < 0 ): - raise ExternalBootstrapConflictError("batch item ordering or lease is invalid") + raise ExternalBootstrapConflictError( + "batch item ordering or lease is invalid" + ) cls._validate_external_identifier(item.workflow_id, "workflow_id") cls._validate_external_identifier(item.task_id, "task_id") for label, value in cls._external_batch_item_payload(item).items(): if str(label).endswith("_hash"): cls._validate_external_hash(value, str(label)) if item.assignment_hash in assignment_hashes: - raise ExternalBootstrapConflictError("batch assignment hashes must be unique") + raise ExternalBootstrapConflictError( + "batch assignment hashes must be unique" + ) assignment_hashes.add(item.assignment_hash) if ( not isinstance(grant, ExternalDispatchGrant) @@ -6663,7 +6760,9 @@ def _validate_external_bootstrap_records( or grant.availability != cls._EXTERNAL_BOOTSTRAP_AVAILABILITY or grant.consumed_at is not None ): - raise ExternalBootstrapConflictError("external dispatch grant is not pending") + raise ExternalBootstrapConflictError( + "external dispatch grant is not pending" + ) cls._validate_external_identifier(grant.grant_id, "grant_id") for label, value in cls._external_grant_payload(grant).items(): if str(label).endswith("_hash"): @@ -6675,7 +6774,9 @@ def _validate_external_bootstrap_records( or grant.expires_at != batch.expires_at or grant.assignment_hash not in assignment_hashes ): - raise ExternalBootstrapConflictError("grant is not bound to descriptor batch assignment") + raise ExternalBootstrapConflictError( + "grant is not bound to descriptor batch assignment" + ) @staticmethod def _insert_or_require_external_commitment( @@ -6702,7 +6803,9 @@ def _insert_or_require_external_commitment( ) return if any(str(row[column]) != value for column, value in values.items()): - raise ExternalBootstrapConflictError("external bootstrap commitment conflicts") + raise ExternalBootstrapConflictError( + "external bootstrap commitment conflicts" + ) @staticmethod def _require_external_payload( @@ -6864,9 +6967,7 @@ def _preflight_external_batch_item_row( or payload["lease_epoch"] < 0 or type(payload["workflow_id"]) is not str or type(payload["task_id"]) is not str - or cls._SAFE_ACCEPTANCE_IDENTIFIER_PATTERN.fullmatch( - payload["workflow_id"] - ) + or cls._SAFE_ACCEPTANCE_IDENTIFIER_PATTERN.fullmatch(payload["workflow_id"]) is None or cls._SAFE_ACCEPTANCE_IDENTIFIER_PATTERN.fullmatch(payload["task_id"]) is None @@ -6908,15 +7009,13 @@ def _preflight_external_batch_row( if set(payload) != expected_fields or type(payload["items"]) is not list: raise ValueError("batch payload shape is invalid") if any( - type(payload[field]) is not str - for field in expected_fields - {"items"} + type(payload[field]) is not str for field in expected_fields - {"items"} ): raise ValueError("batch payload values are invalid") if ( payload["state"] != ExternalBootstrapState.PENDING.value or payload["availability"] != cls._EXTERNAL_BOOTSTRAP_AVAILABILITY - or cls._SHA256_IDENTIFIER_PATTERN.fullmatch(payload["batch_hash"]) - is None + or cls._SHA256_IDENTIFIER_PATTERN.fullmatch(payload["batch_hash"]) is None or cls._SHA256_IDENTIFIER_PATTERN.fullmatch(payload["descriptor_hash"]) is None or cls._SAFE_ACCEPTANCE_IDENTIFIER_PATTERN.fullmatch( @@ -7049,10 +7148,11 @@ def _canonicalize_external_bootstrap_expiries(cursor: sqlite3.Cursor) -> None: if ( str(row["expires_at"]) != raw_expiry or str(row["payload_json"]) != raw_payload_json - or str(row["payload_hash"]) - != _payload_hash(raw_payload_json) + or str(row["payload_hash"]) != _payload_hash(raw_payload_json) ): - raise ValueError("raw expiry column, payload, and hash disagree") + raise ValueError( + "raw expiry column, payload, and hash disagree" + ) payload["expires_at"] = _utc_timestamp(raw_expiry) canonical_payload = _canonical_payload_json(payload) except (KeyError, TypeError, ValueError, json.JSONDecodeError) as error: @@ -7118,7 +7218,9 @@ def index_columns(index_name: object) -> tuple[str, ...]: identifier = index_name.replace('"', '""') return tuple( str(row["name"]) - for row in cursor.execute(f'PRAGMA index_info("{identifier}")').fetchall() + for row in cursor.execute( + f'PRAGMA index_info("{identifier}")' + ).fetchall() ) def validate_index_xinfo( @@ -7127,9 +7229,7 @@ def validate_index_xinfo( if type(index_name) is not str: raise ValueError("index name is invalid") identifier = index_name.replace('"', '""') - info_rows = cursor.execute( - f'PRAGMA index_xinfo("{identifier}")' - ).fetchall() + info_rows = cursor.execute(f'PRAGMA index_xinfo("{identifier}")').fetchall() key_rows = [row for row in info_rows if int(row["key"])] if len(key_rows) != len(expected_columns): raise StoreError("orchestrator store is not prepared") @@ -7239,14 +7339,8 @@ def _validate_legacy_atlas_outbox_rows(cls, cursor: sqlite3.Cursor) -> None: or not 0 <= row["attempt_count"] <= cls._MAX_ATLAS_OUTBOX_ATTEMPTS or row["ingestion_key"] != row["payload_hash"] or row["acceptance_id"] != row["ingestion_key"] - or ( - row["state"] == "projected" - and row["last_error_code"] != "" - ) - or ( - row["state"] == "quarantined" - and not row["last_error_code"] - ) + or (row["state"] == "projected" and row["last_error_code"] != "") + or (row["state"] == "quarantined" and not row["last_error_code"]) or ( row["state"] == "pending" and row["attempt_count"] > 0 @@ -7286,9 +7380,7 @@ def _validate_legacy_atlas_outbox_rows(cls, cursor: sqlite3.Cursor) -> None: framework=acceptance["framework"], ) ) - canonical_acceptance_hash = _payload_hash( - canonical_acceptance_payload - ) + canonical_acceptance_hash = _payload_hash(canonical_acceptance_payload) if ( row["acceptance_id"] != acceptance["acceptance_id"] or row["ingestion_key"] != acceptance["payload_hash"] @@ -7301,10 +7393,9 @@ def _validate_legacy_atlas_outbox_rows(cls, cursor: sqlite3.Cursor) -> None: or row["payload_hash"] != canonical_acceptance_hash ): raise ValueError("legacy acceptance content address mismatch") - if ( - row["created_at"] != _utc_timestamp(row["created_at"]) - or row["updated_at"] != _utc_timestamp(row["updated_at"]) - ): + if row["created_at"] != _utc_timestamp(row["created_at"]) or row[ + "updated_at" + ] != _utc_timestamp(row["updated_at"]): raise ValueError("outbox timestamps are not canonical UTC") cls._safe_acceptance_identifier("ingestion_key", row["ingestion_key"]) cls._safe_acceptance_identifier("acceptance_id", row["acceptance_id"]) @@ -7403,9 +7494,7 @@ def _migrate_atlas_outbox_ingestion_key_not_null( FROM atlas_ingestion_outbox """ ) - cls._drop_atlas_finalization_projection_trigger_for_outbox_rebuild( - cursor - ) + cls._drop_atlas_finalization_projection_trigger_for_outbox_rebuild(cursor) cursor.execute("DROP TABLE atlas_ingestion_outbox") cursor.execute( "ALTER TABLE atlas_ingestion_outbox_v11 RENAME TO atlas_ingestion_outbox" @@ -7500,13 +7589,16 @@ def _restore_atlas_finalization_projection_trigger_after_outbox_rebuild( def _create_schema(self) -> None: with self._transaction() as cursor: try: - fresh_database = cursor.execute( - """ + fresh_database = ( + cursor.execute( + """ SELECT 1 FROM sqlite_master WHERE type = 'table' AND name NOT LIKE 'sqlite_%' LIMIT 1 """ - ).fetchone() is None + ).fetchone() + is None + ) except sqlite3.DatabaseError as error: raise StoreError("orchestrator schema is corrupt") from error self._preflight_legacy_atlas_outbox_before_schema_ddl( @@ -8178,7 +8270,9 @@ def _role_envelope_from_row(self, row: sqlite3.Row) -> RoleEnvelope: raise TypeError("role payload is not an object") if payload.get("schema_version") != self._ROLE_ENVELOPE_SCHEMA_VERSION: raise ValueError("role payload schema is not current") - if _payload_hash(_canonical_payload_json(payload)) != str(row["envelope_hash"]): + if _payload_hash(_canonical_payload_json(payload)) != str( + row["envelope_hash"] + ): raise ValueError("role envelope hash is corrupt") direction = RoleEnvelopeDirection(str(row["direction"])) bindings = { @@ -8203,7 +8297,9 @@ def _role_envelope_from_row(self, row: sqlite3.Row) -> RoleEnvelope: if any(payload.get(key) != value for key, value in bindings.items()): raise ValueError("role envelope bindings are corrupt") contracts = tuple(str(item) for item in payload["contract_hashes"]) - index_evidence = tuple(str(item) for item in payload["index_evidence_hashes"]) + index_evidence = tuple( + str(item) for item in payload["index_evidence_hashes"] + ) evidence = tuple(str(item) for item in payload["evidence_hashes"]) dependencies = tuple(str(item) for item in payload["dependency_hashes"]) risks = self._role_risk_items(tuple(payload["risk_items"])) @@ -8247,7 +8343,9 @@ def _role_envelope_from_row(self, row: sqlite3.Row) -> RoleEnvelope: str(row["envelope_hash"]), ) except (KeyError, TypeError, ValueError, json.JSONDecodeError) as error: - raise RoleEnvelopeInvalidError("durable role envelope is corrupt") from error + raise RoleEnvelopeInvalidError( + "durable role envelope is corrupt" + ) from error def _host_operation_receipt_from_row( self, row: sqlite3.Row @@ -8269,7 +8367,9 @@ def _host_operation_receipt_from_row( "status_code": str(row["status_code"]), "outcome": str(row["outcome"]), } - if _payload_hash(_canonical_payload_json(payload)) != str(row["receipt_hash"]): + if _payload_hash(_canonical_payload_json(payload)) != str( + row["receipt_hash"] + ): raise ValueError("host operation receipt hash is corrupt") return HostOperationReceipt( str(row["operation_id"]), @@ -8289,7 +8389,9 @@ def _host_operation_receipt_from_row( str(row["reported_at"]), ) except (KeyError, TypeError, ValueError) as error: - raise HostOperationConflictError("durable host operation receipt is corrupt") from error + raise HostOperationConflictError( + "durable host operation receipt is corrupt" + ) from error @staticmethod def _last_lease_epoch(cursor: sqlite3.Cursor, task_id: str) -> int: diff --git a/mcp-tools/server.py b/mcp-tools/server.py index 7010a3a..45fc632 100644 --- a/mcp-tools/server.py +++ b/mcp-tools/server.py @@ -1008,9 +1008,7 @@ def relay_compile(request: RelayCompileRequest) -> dict[str, object]: @mcp.tool(annotations=_tool_annotations("fastlane_compile")) def fastlane_compile( request: dict[str, object], - reasoning_effort: Literal[ - "low", "medium", "high", "xhigh", "max" - ], + reasoning_effort: Literal["low", "medium", "high", "xhigh", "max"], enable: bool = False, ) -> dict[str, object]: """Compile inert Fast Lane descriptors without receiving host-private evidence. @@ -1077,10 +1075,7 @@ def _fastlane_authenticated_dispatch( if reasoning_effort == "ultra": return _failure("FASTLANE_HOST_AUTHORITY_UNAVAILABLE") session = _host_session() - if ( - type(session) is not HostSession - or not session.is_available - ): + if type(session) is not HostSession or not session.is_available: return _failure("FASTLANE_HOST_AUTHORITY_UNAVAILABLE") index_attestation = session.project_index_query_attestation( correlation_id=index_query_correlation @@ -1139,14 +1134,11 @@ def _fastlane_authenticated_dispatch( # skeletons. Remaining work is materialized later by the Host-owned # refill registry, which has no storage-profile proof channel yet. # Budgeted successors therefore fail closed before publication. - if ( - len(initial_storage_budgets) not in {0, len(initial_units)} - or any("storage_budget" in unit for unit in remaining_units) + if len(initial_storage_budgets) not in {0, len(initial_units)} or any( + "storage_budget" in unit for unit in remaining_units ): return _failure("FASTLANE_HOST_AUTHORITY_UNAVAILABLE") - initial_task_ids = { - unit["task"]["task_id"] for unit in initial_units - } + initial_task_ids = {unit["task"]["task_id"] for unit in initial_units} initial_requests: list[dict[str, object]] = [] remaining_requests: list[dict[str, object]] = [] for item in routing_snapshot.routing_requests: @@ -1283,24 +1275,25 @@ def _fastlane_authenticated_dispatch( def refill_callback(trigger: Mapping[str, object]) -> dict[str, object]: """Record the real next-boundary result for this fully dispatched plan.""" - request_hash = "sha256:" + hashlib.sha256( - json.dumps( - request, - ensure_ascii=False, - sort_keys=True, - separators=(",", ":"), - allow_nan=False, - ).encode("utf-8") - ).hexdigest() + request_hash = ( + "sha256:" + + hashlib.sha256( + json.dumps( + request, + ensure_ascii=False, + sort_keys=True, + separators=(",", ":"), + allow_nan=False, + ).encode("utf-8") + ).hexdigest() + ) queued_ids = [ skeleton["task_id"] for skeleton in compiled_remaining["assignment_skeletons"] ] return { "schema": "2718lab-devkit/fastlane-refill-receipt-v1", - "state": ( - "QUEUED_WAVE_PENDING" if queued_ids else "NO_QUEUED_WORK" - ), + "state": ("QUEUED_WAVE_PENDING" if queued_ids else "NO_QUEUED_WORK"), "request_hash": request_hash, "refill_trigger_hash": trigger["refill_trigger_hash"], "queue_registry_hash": ( diff --git a/mcp-tools/tests/test_runtime_composition.py b/mcp-tools/tests/test_runtime_composition.py index 4bd5857..091b227 100644 --- a/mcp-tools/tests/test_runtime_composition.py +++ b/mcp-tools/tests/test_runtime_composition.py @@ -850,9 +850,7 @@ def text_type(column: str) -> str: return f"TEXT{column_collation}{not_null(column)}" payload_json = ( - f"payload_json {text_type('payload_json')}," - if include_payload_json - else "" + f"payload_json {text_type('payload_json')}," if include_payload_json else "" ) if partial_unique_indexes: acceptance_id = ( @@ -881,12 +879,12 @@ def text_type(column: str) -> str: {acceptance_id} {payload_json} {payload_hash} - state {text_type('state')} {state_check}, - attempt_count INTEGER{not_null('attempt_count')} {attempt_check}, - last_error_code {text_type('last_error_code')}, - reason_codes_json {text_type('reason_codes_json')}, - created_at {text_type('created_at')}, - updated_at {text_type('updated_at')}, + state {text_type("state")} {state_check}, + attempt_count INTEGER{not_null("attempt_count")} {attempt_check}, + last_error_code {text_type("last_error_code")}, + reason_codes_json {text_type("reason_codes_json")}, + created_at {text_type("created_at")}, + updated_at {text_type("updated_at")}, {equality_check} ); {unique_indexes} @@ -1071,9 +1069,7 @@ def prepare_proof_registry(database_path: Path) -> None: ) ), _atlas_outbox_schema( - attempt_check=( - "CHECK (1) /* CHECK (attempt_count BETWEEN 0 AND 16) */" - ) + attempt_check=("CHECK (1) /* CHECK (attempt_count BETWEEN 0 AND 16) */") ), ), ids=( @@ -1418,9 +1414,7 @@ def test_sqlite_store_migrates_v10_outbox_to_reject_null_ingestion_keys( 0, ) - next_acceptance_id, _ = _insert_legacy_atlas_acceptance( - connection, suffix="b" - ) + next_acceptance_id, _ = _insert_legacy_atlas_acceptance(connection, suffix="b") with pytest.raises( sqlite3.IntegrityError, match="NOT NULL constraint failed: atlas_ingestion_outbox.ingestion_key", @@ -1445,9 +1439,12 @@ def test_sqlite_store_migrates_v10_outbox_to_reject_null_ingestion_keys( timestamp, ), ) - assert connection.execute( - "SELECT COUNT(*) FROM atlas_ingestion_outbox" - ).fetchone()[0] == 1 + assert ( + connection.execute( + "SELECT COUNT(*) FROM atlas_ingestion_outbox" + ).fetchone()[0] + == 1 + ) finally: store.close() @@ -1474,19 +1471,28 @@ def test_sqlite_store_rejects_legacy_v6_outbox_drift_before_current_ddl( connection = sqlite3.connect(database) try: - assert connection.execute( - "SELECT value FROM schema_metadata WHERE key = 'schema_version'" - ).fetchone()[0] == "6" + assert ( + connection.execute( + "SELECT value FROM schema_metadata WHERE key = 'schema_version'" + ).fetchone()[0] + == "6" + ) if missing_object == "table": - assert connection.execute( - "SELECT 1 FROM sqlite_master WHERE type = 'table' " - "AND name = 'atlas_ingestion_outbox'" - ).fetchone() is None + assert ( + connection.execute( + "SELECT 1 FROM sqlite_master WHERE type = 'table' " + "AND name = 'atlas_ingestion_outbox'" + ).fetchone() + is None + ) else: - assert connection.execute( - "SELECT 1 FROM sqlite_master WHERE type = 'index' " - "AND name = 'idx_atlas_outbox_pending'" - ).fetchone() is None + assert ( + connection.execute( + "SELECT 1 FROM sqlite_master WHERE type = 'index' " + "AND name = 'idx_atlas_outbox_pending'" + ).fetchone() + is None + ) finally: connection.close() @@ -1503,10 +1509,13 @@ def test_sqlite_store_bootstraps_true_legacy_v6_empty_outbox( assert connection.execute( "SELECT value FROM schema_metadata WHERE key = 'schema_version'" ).fetchone() == ("6",) - assert connection.execute( - "SELECT 1 FROM sqlite_master WHERE type = 'table' " - "AND name = 'atlas_finalizations'" - ).fetchone() is None + assert ( + connection.execute( + "SELECT 1 FROM sqlite_master WHERE type = 'table' " + "AND name = 'atlas_finalizations'" + ).fetchone() + is None + ) connection.commit() finally: connection.close() @@ -1514,20 +1523,24 @@ def test_sqlite_store_bootstraps_true_legacy_v6_empty_outbox( store = SQLiteStore(database) try: assert store.schema_version() == 13 - assert store._connection.execute( - "SELECT COUNT(*) FROM atlas_ingestion_outbox" - ).fetchone()[0] == 0 - assert store._connection.execute( - "SELECT 1 FROM sqlite_master WHERE type = 'table' " - "AND name = 'atlas_finalizations'" - ).fetchone() is not None + assert ( + store._connection.execute( + "SELECT COUNT(*) FROM atlas_ingestion_outbox" + ).fetchone()[0] + == 0 + ) + assert ( + store._connection.execute( + "SELECT 1 FROM sqlite_master WHERE type = 'table' " + "AND name = 'atlas_finalizations'" + ).fetchone() + is not None + ) finally: store.close() -@pytest.mark.parametrize( - "semantic_drift", ("column-nocase", "pending-nocase-desc") -) +@pytest.mark.parametrize("semantic_drift", ("column-nocase", "pending-nocase-desc")) def test_sqlite_store_rejects_legacy_v6_semantic_shape_drift( tmp_path: Path, semantic_drift: str ) -> None: @@ -1603,12 +1616,8 @@ def test_sqlite_store_rejects_legacy_atlas_outbox_row_contract_drift( "state": outbox_update.get("state", "pending"), "attempt_count": outbox_update.get("attempt_count", 0), "last_error_code": outbox_update.get("last_error_code", ""), - "created_at": outbox_update.get( - "created_at", "2026-08-09T00:00:00+00:00" - ), - "updated_at": outbox_update.get( - "updated_at", "2026-08-09T00:00:00+00:00" - ), + "created_at": outbox_update.get("created_at", "2026-08-09T00:00:00+00:00"), + "updated_at": outbox_update.get("updated_at", "2026-08-09T00:00:00+00:00"), } connection.execute( """ @@ -1627,9 +1636,12 @@ def test_sqlite_store_rejects_legacy_atlas_outbox_row_contract_drift( connection = sqlite3.connect(database) try: - assert connection.execute( - "SELECT value FROM schema_metadata WHERE key = 'schema_version'" - ).fetchone()[0] == "10" + assert ( + connection.execute( + "SELECT value FROM schema_metadata WHERE key = 'schema_version'" + ).fetchone()[0] + == "10" + ) columns = { str(row[1]): int(row[3]) for row in connection.execute( @@ -1680,9 +1692,12 @@ def test_sqlite_store_rejects_legacy_outbox_acceptance_binding_drift( assert connection.execute( "SELECT value FROM schema_metadata WHERE key = 'schema_version'" ).fetchone() == ("10",) - assert connection.execute( - "SELECT COUNT(*) FROM atlas_ingestion_outbox" - ).fetchone()[0] == 1 + assert ( + connection.execute( + "SELECT COUNT(*) FROM atlas_ingestion_outbox" + ).fetchone()[0] + == 1 + ) finally: connection.close() @@ -1710,9 +1725,12 @@ def test_sqlite_store_bootstraps_verified_legacy_empty_outbox( store = SQLiteStore(database) try: assert store.schema_version() == 13 - assert store._connection.execute( - "SELECT COUNT(*) FROM atlas_ingestion_outbox" - ).fetchone()[0] == 0 + assert ( + store._connection.execute( + "SELECT COUNT(*) FROM atlas_ingestion_outbox" + ).fetchone()[0] + == 0 + ) finally: store.close() @@ -1725,12 +1743,18 @@ def test_sqlite_store_fails_closed_for_legacy_null_outbox_key(tmp_path: Path) -> connection = sqlite3.connect(database) try: - assert connection.execute( - "SELECT ingestion_key FROM atlas_ingestion_outbox" - ).fetchone()[0] is None - assert connection.execute( - "SELECT value FROM schema_metadata WHERE key = 'schema_version'" - ).fetchone()[0] == "10" + assert ( + connection.execute( + "SELECT ingestion_key FROM atlas_ingestion_outbox" + ).fetchone()[0] + is None + ) + assert ( + connection.execute( + "SELECT value FROM schema_metadata WHERE key = 'schema_version'" + ).fetchone()[0] + == "10" + ) finally: connection.close() @@ -1745,9 +1769,7 @@ def test_sqlite_store_fails_closed_for_noncanonical_finalization_guard_during_v1 ) connection = sqlite3.connect(database) try: - connection.execute( - "DROP TRIGGER atlas_finalizations_require_projected_outbox" - ) + connection.execute("DROP TRIGGER atlas_finalizations_require_projected_outbox") connection.execute( """ CREATE TRIGGER atlas_finalizations_require_projected_outbox @@ -1818,8 +1840,8 @@ def _legacy_v10_incomplete_atlas_outbox_database( def test_sqlite_store_rolls_back_incomplete_v10_outbox_upgrade( tmp_path: Path, ) -> None: - database, ingestion_key, acceptance_id = _legacy_v10_incomplete_atlas_outbox_database( - tmp_path + database, ingestion_key, acceptance_id = ( + _legacy_v10_incomplete_atlas_outbox_database(tmp_path) ) with pytest.raises(StoreError): @@ -1839,13 +1861,19 @@ def test_sqlite_store_rolls_back_incomplete_v10_outbox_upgrade( for row in connection.execute("PRAGMA table_info(schema_metadata)") } assert metadata_columns["key"] == 0 - assert connection.execute( - "SELECT 1 FROM sqlite_master WHERE type = 'table' " - "AND name = 'schema_metadata_v12'" - ).fetchone() is None - assert connection.execute( - "SELECT value FROM schema_metadata WHERE key = 'schema_version'" - ).fetchone()[0] == "10" + assert ( + connection.execute( + "SELECT 1 FROM sqlite_master WHERE type = 'table' " + "AND name = 'schema_metadata_v12'" + ).fetchone() + is None + ) + assert ( + connection.execute( + "SELECT value FROM schema_metadata WHERE key = 'schema_version'" + ).fetchone()[0] + == "10" + ) assert tuple( connection.execute( """ @@ -1887,8 +1915,8 @@ def _malformed_v11_nullable_atlas_outbox_database( def test_sqlite_store_preserves_malformed_v11_nullable_outbox( tmp_path: Path, ) -> None: - database, ingestion_key, acceptance_id = _malformed_v11_nullable_atlas_outbox_database( - tmp_path + database, ingestion_key, acceptance_id = ( + _malformed_v11_nullable_atlas_outbox_database(tmp_path) ) with pytest.raises(StoreError): @@ -1902,9 +1930,12 @@ def test_sqlite_store_preserves_malformed_v11_nullable_outbox( for row in connection.execute("PRAGMA table_info(atlas_ingestion_outbox)") } assert columns["ingestion_key"] == 0 - assert connection.execute( - "SELECT value FROM schema_metadata WHERE key = 'schema_version'" - ).fetchone()[0] == "11" + assert ( + connection.execute( + "SELECT value FROM schema_metadata WHERE key = 'schema_version'" + ).fetchone()[0] + == "11" + ) assert tuple( connection.execute( """ @@ -2086,12 +2117,18 @@ def test_sqlite_store_rejects_null_schema_metadata_key( for row in connection.execute("PRAGMA table_info(schema_metadata)") } assert columns["key"] == 0 - assert connection.execute( - "SELECT value FROM schema_metadata WHERE key IS NULL" - ).fetchone()[0] == "invalid-null-key" - assert connection.execute( - "SELECT value FROM schema_metadata WHERE key = 'schema_version'" - ).fetchone()[0] == "11" + assert ( + connection.execute( + "SELECT value FROM schema_metadata WHERE key IS NULL" + ).fetchone()[0] + == "invalid-null-key" + ) + assert ( + connection.execute( + "SELECT value FROM schema_metadata WHERE key = 'schema_version'" + ).fetchone()[0] + == "11" + ) finally: connection.close() @@ -2179,15 +2216,21 @@ def test_sqlite_store_rejects_untrusted_legacy_schema_metadata( for row in connection.execute("PRAGMA table_info(schema_metadata)") } assert columns["key"] == 0 - assert tuple( - connection.execute( - "SELECT key, value FROM schema_metadata ORDER BY key, value" + assert ( + tuple( + connection.execute( + "SELECT key, value FROM schema_metadata ORDER BY key, value" + ) ) - ) == expected_rows - assert connection.execute( - "SELECT 1 FROM sqlite_master WHERE type = 'table' " - "AND name = 'schema_metadata_v12'" - ).fetchone() is None + == expected_rows + ) + assert ( + connection.execute( + "SELECT 1 FROM sqlite_master WHERE type = 'table' " + "AND name = 'schema_metadata_v12'" + ).fetchone() + is None + ) finally: connection.close() @@ -2257,11 +2300,14 @@ def test_sqlite_store_rejects_noncanonical_strict_schema_metadata( for row in connection.execute("PRAGMA table_info(schema_metadata)") } assert columns["key"] == 1 - assert tuple( - connection.execute( - "SELECT key, value FROM schema_metadata ORDER BY key, value" + assert ( + tuple( + connection.execute( + "SELECT key, value FROM schema_metadata ORDER BY key, value" + ) ) - ) == expected_rows + == expected_rows + ) finally: connection.close() @@ -2316,13 +2362,19 @@ def test_sqlite_store_rejects_generated_schema_metadata_column( str(row["name"]) for row in connection.execute("PRAGMA table_xinfo(schema_metadata)") } == {"key", "value", "generated_marker"} - assert connection.execute( - "SELECT value FROM schema_metadata WHERE key = 'schema_version'" - ).fetchone()["value"] == "12" - assert connection.execute( - "SELECT 1 FROM sqlite_master WHERE type = 'table' " - "AND name = 'schema_metadata_v12'" - ).fetchone() is None + assert ( + connection.execute( + "SELECT value FROM schema_metadata WHERE key = 'schema_version'" + ).fetchone()["value"] + == "12" + ) + assert ( + connection.execute( + "SELECT 1 FROM sqlite_master WHERE type = 'table' " + "AND name = 'schema_metadata_v12'" + ).fetchone() + is None + ) finally: connection.close() @@ -2379,13 +2431,19 @@ def test_sqlite_store_rejects_generated_schema_metadata_value( for row in connection.execute("PRAGMA table_xinfo(schema_metadata)") } assert columns == {"key": (1, 0), "value": (1, 3)} - assert connection.execute( - "SELECT value FROM schema_metadata WHERE key = 'schema_version'" - ).fetchone()["value"] == "12" - assert connection.execute( - "SELECT 1 FROM sqlite_master WHERE type = 'table' " - "AND name = 'schema_metadata_v12'" - ).fetchone() is None + assert ( + connection.execute( + "SELECT value FROM schema_metadata WHERE key = 'schema_version'" + ).fetchone()["value"] + == "12" + ) + assert ( + connection.execute( + "SELECT 1 FROM sqlite_master WHERE type = 'table' " + "AND name = 'schema_metadata_v12'" + ).fetchone() + is None + ) finally: connection.close() @@ -2433,12 +2491,18 @@ def test_sqlite_store_rejects_legacy_generated_schema_metadata_value( for row in connection.execute("PRAGMA table_xinfo(schema_metadata)") } assert columns == {"key": (0, 0), "value": (1, 3)} - assert connection.execute( - "SELECT value FROM schema_metadata WHERE key = 'schema_version'" - ).fetchone()["value"] == "11" - assert connection.execute( - "SELECT 1 FROM sqlite_master WHERE type = 'table' " - "AND name = 'schema_metadata_v12'" - ).fetchone() is None + assert ( + connection.execute( + "SELECT value FROM schema_metadata WHERE key = 'schema_version'" + ).fetchone()["value"] + == "11" + ) + assert ( + connection.execute( + "SELECT 1 FROM sqlite_master WHERE type = 'table' " + "AND name = 'schema_metadata_v12'" + ).fetchone() + is None + ) finally: connection.close() diff --git a/mcp-tools/tests/test_storage_firewall.py b/mcp-tools/tests/test_storage_firewall.py index dbb5f11..11d978a 100644 --- a/mcp-tools/tests/test_storage_firewall.py +++ b/mcp-tools/tests/test_storage_firewall.py @@ -127,7 +127,9 @@ def _authenticated_v5_fixture() -> dict[str, object]: "storage_budget": {"bytes": 4096, "files": 8}, } api = team_efficiency._AuthenticatedV5Api() - planner = team_efficiency._authenticated_v5_helper_module("authenticated_v5_planner") + planner = team_efficiency._authenticated_v5_helper_module( + "authenticated_v5_planner" + ) normalized_unit = planner.normalize_units(api, [unit])[0] unit["task"]["profile_evidence_hash"] = team_efficiency._sha256_json( planner._routing_profile_material(source_plan_hash, normalized_unit) @@ -401,7 +403,9 @@ def _admission_response(request: dict[str, object]) -> dict[str, object]: } -def _verified_profile_round_trip(monkeypatch: pytest.MonkeyPatch, *, admit: bool) -> None: +def _verified_profile_round_trip( + monkeypatch: pytest.MonkeyPatch, *, admit: bool +) -> None: """Exercise the framed Host bridge and local post-profile compilation.""" import devkit_runtime.host_session as host_session @@ -425,7 +429,9 @@ def host_reply() -> None: scheduler_facts=fixture["scheduler"], now=1_700_000_000, ) - routing_request = host.receive_routing_attestation_request(now=1_700_000_000) + routing_request = host.receive_routing_attestation_request( + now=1_700_000_000 + ) host.send_routing_attestation_response( request=routing_request, attestations=fixture["attestation_items"], @@ -461,14 +467,25 @@ def host_reply() -> None: request = message.payload assert message.kind == "storage_admission_request" assert message.action_id == request["correlation_id"] - assert set(request) == {"schema", "correlation_id", "profile_attestation_hash", "storage_intent", "request_hash"} + assert set(request) == { + "schema", + "correlation_id", + "profile_attestation_hash", + "storage_intent", + "request_hash", + } assert request["schema"] == "2718lab.storage.admission-request.v1" assert request["request_hash"] == _canonical_hash( - {key: value for key, value in request.items() if key != "request_hash"} + { + key: value + for key, value in request.items() + if key != "request_hash" + } ) admission_requests.append(request) host._send_validated_private( - kind="storage_admission_response", action_id=message.action_id, + kind="storage_admission_response", + action_id=message.action_id, payload=_admission_response(request), ) except BaseException as error: # report peer errors after the round trip @@ -485,10 +502,13 @@ def host_reply() -> None: environ={}, platform="posix", clock=lambda: 1_700_000_000 ) try: - assert session.resolve_capability_snapshot_v2( - call_intent_hash=fixture["call_intent_hash"], - preparation_id=fixture["preparation_id"], - ) is not None + assert ( + session.resolve_capability_snapshot_v2( + call_intent_hash=fixture["call_intent_hash"], + preparation_id=fixture["preparation_id"], + ) + is not None + ) routing = session.resolve_routing_attestations( call_intent_hash=fixture["call_intent_hash"], preparation_id=fixture["preparation_id"], @@ -517,21 +537,41 @@ def host_reply() -> None: intent = parse_storage_intent(prepared.storage_intents[0]) # Same attestation/profile facts on another Python session do not # become an issued reference, even when the bridge object matches. - foreign = host_session.HostSession(bridge=child, clock=lambda: 1_700_000_000) - assert foreign.request_storage_admission(intent, profile_attestation_hash=_hash("7")) == "STORAGE_TARGET_KEY_INVALID" - first = session.request_storage_admission(intent, profile_attestation_hash=_hash("7")) - second = session.request_storage_admission(intent, profile_attestation_hash=_hash("7")) + foreign = host_session.HostSession( + bridge=child, clock=lambda: 1_700_000_000 + ) + assert ( + foreign.request_storage_admission( + intent, profile_attestation_hash=_hash("7") + ) + == "STORAGE_TARGET_KEY_INVALID" + ) + first = session.request_storage_admission( + intent, profile_attestation_hash=_hash("7") + ) + second = session.request_storage_admission( + intent, profile_attestation_hash=_hash("7") + ) assert isinstance(first, StorageAdmissionReceipt) assert first == second assert len(admission_requests) == 2 # no local decision/cache admission - assert admission_requests[0]["correlation_id"] != admission_requests[1]["correlation_id"] + assert ( + admission_requests[0]["correlation_id"] + != admission_requests[1]["correlation_id"] + ) assert len(child._storage_admission_completions) == 2 - with pytest.raises(HostBridgeError, match="HOST_BRIDGE_STORAGE_ADMISSION_INVALID"): + with pytest.raises( + HostBridgeError, match="HOST_BRIDGE_STORAGE_ADMISSION_INVALID" + ): child.request_storage_admission( - admission_requests[0], now=1_700_000_000, expires_at=1_700_000_120, + admission_requests[0], + now=1_700_000_000, + expires_at=1_700_000_120, clock=lambda: 1_700_000_000, ) - assert set(first.to_dict()) == set(_admission_response(admission_requests[0])["receipt"]) + assert set(first.to_dict()) == set( + _admission_response(admission_requests[0])["receipt"] + ) batch = adapter.compile_fast_lane_with_host_facts( fixture["planner_request"], reasoning_effort="max", @@ -551,24 +591,45 @@ def test_storage_admission_rejects_substitution_and_unknown_fields() -> None: from devkit_runtime import host_bridge from devkit_runtime.storage_intent import parse_storage_intent - intent = parse_storage_intent(_storage_intent( - task_id="TASK-V5", plan_binding=_hash("8"), context_hash=_hash("6") - )) - request = host_bridge.build_storage_admission_request(intent, profile_attestation_hash=_hash("7")) + intent = parse_storage_intent( + _storage_intent( + task_id="TASK-V5", plan_binding=_hash("8"), context_hash=_hash("6") + ) + ) + request = host_bridge.build_storage_admission_request( + intent, profile_attestation_hash=_hash("7") + ) response = _admission_response(request) - malformed = [dict(request, assigned_root="G:/private"), {key: value for key, value in request.items() if key != "profile_attestation_hash"}] + malformed = [ + dict(request, assigned_root="G:/private"), + { + key: value + for key, value in request.items() + if key != "profile_attestation_hash" + }, + ] for candidate in malformed: with pytest.raises(host_bridge.HostBridgeError): host_bridge._normalize_storage_admission_request(candidate) - for field, value in (("target_key", _hash("f")), ("storage_intent_hash", _hash("f")), - ("profile_attestation_hash", _hash("f")), ("reserved_bytes", True), - ("reserved_files", 9), ("expires_at", 1_700_000_000), - ("free_space_floor", 0), - ("admission_id", "G:/private"), ("assigned_root", "G:/private")): + for field, value in ( + ("target_key", _hash("f")), + ("storage_intent_hash", _hash("f")), + ("profile_attestation_hash", _hash("f")), + ("reserved_bytes", True), + ("reserved_files", 9), + ("expires_at", 1_700_000_000), + ("free_space_floor", 0), + ("admission_id", "G:/private"), + ("assigned_root", "G:/private"), + ): candidate = copy.deepcopy(response) candidate["receipt"][field] = value candidate["receipt"]["receipt_hash"] = _canonical_hash( - {key: value for key, value in candidate["receipt"].items() if key != "receipt_hash"} + { + key: value + for key, value in candidate["receipt"].items() + if key != "receipt_hash" + } ) with pytest.raises(host_bridge.HostBridgeError): host_bridge._normalize_storage_admission_response( @@ -576,8 +637,15 @@ def test_storage_admission_rejects_substitution_and_unknown_fields() -> None: ) child, host = _pipe_pair() try: - with pytest.raises(host_bridge.HostBridgeError, match="HOST_BRIDGE_STORAGE_PROFILE_INVALID"): - child.request_storage_admission(request, now=1_700_000_000, expires_at=1_700_000_120, clock=lambda: 1_700_000_000) + with pytest.raises( + host_bridge.HostBridgeError, match="HOST_BRIDGE_STORAGE_PROFILE_INVALID" + ): + child.request_storage_admission( + request, + now=1_700_000_000, + expires_at=1_700_000_120, + clock=lambda: 1_700_000_000, + ) finally: child.close() host.close() @@ -594,26 +662,42 @@ def test_storage_admission_deadline_and_close_cancel_blocked_io( for scenario in ("no_reply", "full_pipe", "close", "direct_close"): child, host = _pipe_pair() profile_request = _storage_profile_request() - pending = child.send_storage_profile_request(**{ - name: getattr(profile_request, name) for name in ( - "call_intent_hash", "preparation_id", "task_id", - "source_plan_hash", "index_attestation_hash", - ) - }) + pending = child.send_storage_profile_request( + **{ + name: getattr(profile_request, name) + for name in ( + "call_intent_hash", + "preparation_id", + "task_id", + "source_plan_hash", + "index_attestation_hash", + ) + } + ) peer_request = host.receive_storage_profile_request() - host.send_storage_profile_response(request=peer_request, response=_storage_profile_response(peer_request)) + host.send_storage_profile_response( + request=peer_request, response=_storage_profile_response(peer_request) + ) profile = child.receive_storage_profile_response(request=pending) session = host_session.HostSession(bridge=child, clock=lambda: 1_700_000_000) # This test isolates transport lifecycle after actual profile delivery; # preparation enrollment itself is covered by the round-trip test. - session._completed_storage_profiles[_hash("7")] = host_session._CompletedStorageProfile( - profile=profile, bridge=child, - expires_at=1_700_000_060 if scenario in {"close", "direct_close"} else 1_700_000_001, - requested_bytes=4096, requested_files=8, + session._completed_storage_profiles[_hash("7")] = ( + host_session._CompletedStorageProfile( + profile=profile, + bridge=child, + expires_at=1_700_000_060 + if scenario in {"close", "direct_close"} + else 1_700_000_001, + requested_bytes=4096, + requested_files=8, + ) + ) + intent = parse_storage_intent( + _storage_intent( + task_id="TASK-V5", plan_binding=_hash("8"), context_hash=_hash("6") + ) ) - intent = parse_storage_intent(_storage_intent( - task_id="TASK-V5", plan_binding=_hash("8"), context_hash=_hash("6") - )) if scenario == "full_pipe": with host_bridge._nonblocking_pipe_writer(child._write_fd) as write: for _ in range(1024): @@ -628,6 +712,7 @@ def test_storage_admission_deadline_and_close_cancel_blocked_io( restore_allowed = threading.Event() original_writer = host_bridge._nonblocking_pipe_writer if scenario == "direct_close": + @contextmanager def paused_restore(descriptor: int): with original_writer(descriptor) as write: @@ -635,13 +720,20 @@ def paused_restore(descriptor: int): yield write finally: restore_reached.set() - assert restore_allowed.wait(timeout=3), "mode-restore barrier was not released" + assert restore_allowed.wait(timeout=3), ( + "mode-restore barrier was not released" + ) monkeypatch.setattr(host_bridge, "_nonblocking_pipe_writer", paused_restore) result: list[object] = [] - worker = threading.Thread(target=lambda: result.append(session.request_storage_admission( - intent, profile_attestation_hash=_hash("7") - )), daemon=True) + worker = threading.Thread( + target=lambda: result.append( + session.request_storage_admission( + intent, profile_attestation_hash=_hash("7") + ) + ), + daemon=True, + ) closer: threading.Thread | None = None worker.start() try: @@ -652,7 +744,9 @@ def paused_restore(descriptor: int): closer = threading.Thread(target=session.close, daemon=True) closer.start() closer.join(timeout=2) - assert not closer.is_alive(), "close waited behind admission's business lock" + assert not closer.is_alive(), ( + "close waited behind admission's business lock" + ) if scenario == "direct_close": assert restore_reached.wait(timeout=2) borrowed_fd = child._write_fd @@ -665,7 +759,9 @@ def paused_restore(descriptor: int): os.fstat(borrowed_fd) # mode restoration still owns this exact fd restore_allowed.set() worker.join(timeout=3) - assert not worker.is_alive(), "admission did not terminate at its transport deadline" + assert not worker.is_alive(), ( + "admission did not terminate at its transport deadline" + ) assert result == ["STORAGE_STAT_UNAVAILABLE"] assert not child._storage_admission_completions assert not child._storage_admission_decisions @@ -681,7 +777,9 @@ def paused_restore(descriptor: int): if closer is not None: closer.join(timeout=2) session.close() - monkeypatch.setattr(host_bridge, "_nonblocking_pipe_writer", original_writer) + monkeypatch.setattr( + host_bridge, "_nonblocking_pipe_writer", original_writer + ) def test_profile_tamper_or_missing_field_fails_closed() -> None: From c436b739d5e50b13b4c0f9a575154f7ebbd40d93 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E5=93=80=E6=B4=9B=E8=8A=99?= Date: Tue, 1 Sep 2026 17:38:49 +0800 Subject: [PATCH 32/39] fix: narrow compiled Fast Lane skeletons --- mcp-tools/server.py | 29 +++++++++++++++++++++-------- 1 file changed, 21 insertions(+), 8 deletions(-) diff --git a/mcp-tools/server.py b/mcp-tools/server.py index 45fc632..1a6c546 100644 --- a/mcp-tools/server.py +++ b/mcp-tools/server.py @@ -1192,11 +1192,27 @@ def _fastlane_authenticated_dispatch( "index_context_hash", "attestation_hash", ) - skeletons = compiled["assignment_skeletons"] + raw_skeletons = compiled.get("assignment_skeletons") + raw_remaining_skeletons = compiled_remaining.get("assignment_skeletons") + if type(raw_skeletons) is not list or type(raw_remaining_skeletons) is not list: + return _failure("FASTLANE_HOST_AUTHORITY_UNAVAILABLE") + skeletons: list[dict[str, object]] = [] + remaining_skeletons: list[dict[str, object]] = [] + for raw_skeleton, destination in ( + (raw_skeletons, skeletons), + (raw_remaining_skeletons, remaining_skeletons), + ): + for skeleton in raw_skeleton: + if ( + type(skeleton) is not dict + or type(skeleton.get("task_id")) is not str + ): + return _failure("FASTLANE_HOST_AUTHORITY_UNAVAILABLE") + destination.append(skeleton) skeleton_package_hash = validate_authenticated_v5_skeleton_package( projected["all_units"], skeletons, - compiled_remaining["assignment_skeletons"], + remaining_skeletons, source_plan_hash=projected["source_plan_hash"], ) index_refs = [ @@ -1255,7 +1271,7 @@ def _fastlane_authenticated_dispatch( unit["task"]["task_id"] for unit in projected["all_units"] ], initial_skeletons=skeletons, - remaining_skeletons=compiled_remaining["assignment_skeletons"], + remaining_skeletons=remaining_skeletons, index_attestation_refs=[ { "task_id": skeleton["task_id"], @@ -1264,7 +1280,7 @@ def _fastlane_authenticated_dispatch( for field in attestation_ref_fields }, } - for skeleton in compiled_remaining["assignment_skeletons"] + for skeleton in remaining_skeletons ], skeleton_package_hash=skeleton_package_hash, now=now, @@ -1287,10 +1303,7 @@ def refill_callback(trigger: Mapping[str, object]) -> dict[str, object]: ).encode("utf-8") ).hexdigest() ) - queued_ids = [ - skeleton["task_id"] - for skeleton in compiled_remaining["assignment_skeletons"] - ] + queued_ids = [skeleton["task_id"] for skeleton in remaining_skeletons] return { "schema": "2718lab-devkit/fastlane-refill-receipt-v1", "state": ("QUEUED_WAVE_PENDING" if queued_ids else "NO_QUEUED_WORK"), From 45b4cd5d0c36548779ad85a5b629e0c25e2bb46a Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E5=93=80=E6=B4=9B=E8=8A=99?= Date: Tue, 1 Sep 2026 17:45:31 +0800 Subject: [PATCH 33/39] test: use real v10 schema in metadata migration --- mcp-tools/tests/test_runtime_composition.py | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/mcp-tools/tests/test_runtime_composition.py b/mcp-tools/tests/test_runtime_composition.py index 091b227..f361c3b 100644 --- a/mcp-tools/tests/test_runtime_composition.py +++ b/mcp-tools/tests/test_runtime_composition.py @@ -2168,7 +2168,12 @@ def _legacy_metadata_database( def test_sqlite_store_migrates_trustworthy_legacy_schema_metadata( tmp_path: Path, legacy_version: str ) -> None: - database = _legacy_metadata_database(tmp_path, version=legacy_version) + if legacy_version == "10": + database, _, _ = _legacy_v10_atlas_outbox_database( + tmp_path, ingestion_key=f"sha256:{'a' * 64}" + ) + else: + database = _legacy_metadata_database(tmp_path, version=legacy_version) store = SQLiteStore(database) try: From abe81ec94d2eb7516ed4c7cb0224744973eb72ea Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E5=93=80=E6=B4=9B=E8=8A=99?= Date: Tue, 1 Sep 2026 17:51:42 +0800 Subject: [PATCH 34/39] test: bind authenticated V5 profile evidence --- mcp-tools/devkit_fastlane/tests/test_team_efficiency.py | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/mcp-tools/devkit_fastlane/tests/test_team_efficiency.py b/mcp-tools/devkit_fastlane/tests/test_team_efficiency.py index c6aebc1..dbf8e22 100644 --- a/mcp-tools/devkit_fastlane/tests/test_team_efficiency.py +++ b/mcp-tools/devkit_fastlane/tests/test_team_efficiency.py @@ -8781,6 +8781,11 @@ def test_authenticated_v5_planner_emits_exact_proofs_and_skeletons(self) -> None "index_context_hash": hash_a, "predecessor_hash": hash_b, } + planner = helper._authenticated_v5_helper_module("authenticated_v5_planner") + normalized_unit = helper._authenticated_v5_units([unit])[0] + unit["task"]["profile_evidence_hash"] = helper._sha256_json( + planner._routing_profile_material(source_plan_hash, normalized_unit) + ) requests = helper.prepare_authenticated_v5_routing_requests( [unit], source_plan_hash=source_plan_hash, From f53d0ea80802b19cb2b40be4b1fcdf822e23e452 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E5=93=80=E6=B4=9B=E8=8A=99?= Date: Tue, 1 Sep 2026 22:05:52 +0800 Subject: [PATCH 35/39] test: bind Host adapter V5 profile evidence --- mcp-tools/tests/test_fastlane_host_adapter.py | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/mcp-tools/tests/test_fastlane_host_adapter.py b/mcp-tools/tests/test_fastlane_host_adapter.py index b49d924..4b66b64 100644 --- a/mcp-tools/tests/test_fastlane_host_adapter.py +++ b/mcp-tools/tests/test_fastlane_host_adapter.py @@ -127,6 +127,13 @@ def _authenticated_v5_fixture() -> dict[str, object]: "index_context_hash": hash_a, "predecessor_hash": hash_b, } + planner = team_efficiency._authenticated_v5_helper_module( + "authenticated_v5_planner" + ) + normalized_unit = team_efficiency._authenticated_v5_units([unit])[0] + unit["task"]["profile_evidence_hash"] = team_efficiency._sha256_json( + planner._routing_profile_material(source_plan_hash, normalized_unit) + ) routing_requests = team_efficiency.prepare_authenticated_v5_routing_requests( [unit], source_plan_hash=source_plan_hash, From 653d300d3d71dd67c398f660821ca7a374999643 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E5=93=80=E6=B4=9B=E8=8A=99?= Date: Tue, 1 Sep 2026 22:14:24 +0800 Subject: [PATCH 36/39] docs: pin 1.1.3 Host compatibility source --- CHANGELOG.md | 12 ++++++++---- README.md | 9 +++++++++ README.zh-CN.md | 7 +++++++ 3 files changed, 24 insertions(+), 4 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 0aec67f..84801d9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -42,10 +42,14 @@ only after the CI and artifact checks pass. path-free intent and private protocol, but does not authenticate or provision the Windows protected broker, create its root, activate cleanup, or provide a local writer fallback. -- The compatible Host boundary is still awaiting final protected-broker - compile, probe, and runtime receipts from the separate Host branch. This - release does not claim that upstream Codex or an ordinary Codex Host supports - protected-broker storage execution. +- Compatible, buildable Host source exists only on Ayleovelle's user-fork + [`codex/host-1.1.3-storage-governance-upstream`](https://github.com/Ayleovelle/codex/tree/codex/host-1.1.3-storage-governance-upstream) + branch, fixed at immutable commit + [`c3dde23bec21c45d10740f2eec09d9a1b87cd329`](https://github.com/Ayleovelle/codex/commit/c3dde23bec21c45d10740f2eec09d9a1b87cd329). + This identifies the separate Host source boundary only: it is not merged into + OpenAI upstream and is not shipped or activated by this plugin. Stock Codex + Hosts still fail closed, and final protected-broker compile, probe, and + runtime receipts remain Host-side release evidence rather than DevKit claims. ## [1.1.2] - 2026-08-27 diff --git a/README.md b/README.md index e832292..af285d6 100644 --- a/README.md +++ b/README.md @@ -18,6 +18,15 @@ and CLI return `NO_SAFE_WORK` with zero assignments: they do not consume host status or live-account inputs, and have no worktree execution path. Host execution remains an external Desktop-host bridge requirement. +For storage-governed execution, v1.1.3 references compatible Host source only +on Ayleovelle's user-fork +[`codex/host-1.1.3-storage-governance-upstream`](https://github.com/Ayleovelle/codex/tree/codex/host-1.1.3-storage-governance-upstream) +branch, pinned to immutable commit +[`c3dde23bec21c45d10740f2eec09d9a1b87cd329`](https://github.com/Ayleovelle/codex/commit/c3dde23bec21c45d10740f2eec09d9a1b87cd329). +That fork is buildable Host source, not an OpenAI upstream merge or a component +shipped by this package. Stock Codex Hosts have no attested protected broker +and continue to fail closed. + > [!IMPORTANT] > **Workflow reminder:** route from bounded evidence. Parallel A1/A2/A3 work is > allowed only with disjoint, exclusively owned write scopes and independent G: diff --git a/README.zh-CN.md b/README.zh-CN.md index 81b3204..5f2d4e5 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -15,6 +15,13 @@ v1.1.3 包；已提交的 manifest 和 allowlist 定义可执行运行时范围 `NO_SAFE_WORK` 与零 assignments：不会消费 host-status 或实时账号输入，也没有 worktree 执行路径。宿主执行属于未来外部 Desktop-host bridge 合同的要求。 +对于受存储治理的执行，v1.1.3 只引用 Ayleovelle 用户 fork 上的兼容 Host 源码： +[`codex/host-1.1.3-storage-governance-upstream`](https://github.com/Ayleovelle/codex/tree/codex/host-1.1.3-storage-governance-upstream) +分支，固定到不可变提交 +[`c3dde23bec21c45d10740f2eec09d9a1b87cd329`](https://github.com/Ayleovelle/codex/commit/c3dde23bec21c45d10740f2eec09d9a1b87cd329)。 +该 fork 是可构建的 Host 源码，不是 OpenAI upstream 合并，也不随本包交付。 +stock Codex Host 没有经过证明的 protected broker，继续 fail-closed。 + > [!IMPORTANT] > **工作流提醒：** 先用有界证据路由。A1/A2/A3 并行只允许发生在互不重叠、独占 > 的写入范围，并各自使用 G: 盘隔离任务根。执行前必须 claim 并 bind；prewarm From 3480b2b52580d9104dd5ca6693d6e5707521a53e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E5=93=80=E6=B4=9B=E8=8A=99?= Date: Tue, 1 Sep 2026 22:31:36 +0800 Subject: [PATCH 37/39] fix: preserve pre-atlas store migrations --- mcp-tools/orchestrator/store.py | 6 ++- .../test_orchestrator_store_service_api.py | 41 +++++++++++++++++++ 2 files changed, 46 insertions(+), 1 deletion(-) diff --git a/mcp-tools/orchestrator/store.py b/mcp-tools/orchestrator/store.py index 847adb0..89dd1be 100644 --- a/mcp-tools/orchestrator/store.py +++ b/mcp-tools/orchestrator/store.py @@ -7454,9 +7454,13 @@ def _migrate_atlas_outbox_ingestion_key_not_null( except (TypeError, ValueError) as error: raise StoreError("orchestrator store is not prepared") from error if str(source_version) != source_version_value or not ( - 6 <= source_version <= cls._SCHEMA_VERSION + 1 <= source_version <= cls._SCHEMA_VERSION ): raise StoreError("orchestrator store is not prepared") + if source_version < 6: + if not int(ingestion_key["notnull"]): + raise StoreError("orchestrator store is not prepared") + return if int(ingestion_key["notnull"]): return if source_version not in range(6, 11): diff --git a/mcp-tools/tests/test_orchestrator_store_service_api.py b/mcp-tools/tests/test_orchestrator_store_service_api.py index d1448e8..66df9ec 100644 --- a/mcp-tools/tests/test_orchestrator_store_service_api.py +++ b/mcp-tools/tests/test_orchestrator_store_service_api.py @@ -20,6 +20,7 @@ CardHashMismatchError, LeaseConflictError, SQLiteStore, + StoreError, StrictIndexError, VersionConflictError, WorkflowCancelledError, @@ -377,6 +378,46 @@ def test_version_one_database_migrates_without_treating_legacy_lease_as_online( finally: migrated.close() + def test_pre_v6_database_with_nullable_atlas_outbox_fails_closed(self) -> None: + legacy_database = ( + Path(self._temporary_directory.name) / "legacy-v5-outbox.sqlite" + ) + seeded = SQLiteStore(legacy_database) + seeded.close() + + connection = sqlite3.connect(legacy_database) + try: + connection.execute("DROP TABLE atlas_ingestion_outbox") + connection.executescript( + """ + CREATE TABLE atlas_ingestion_outbox ( + ingestion_key TEXT PRIMARY KEY, + acceptance_id TEXT NOT NULL UNIQUE + REFERENCES code_task_acceptances(acceptance_id), + payload_json TEXT NOT NULL, + payload_hash TEXT NOT NULL UNIQUE, + state TEXT NOT NULL + CHECK (state IN ('pending', 'projected', 'quarantined')), + attempt_count INTEGER NOT NULL + CHECK (attempt_count BETWEEN 0 AND 16), + last_error_code TEXT NOT NULL, + reason_codes_json TEXT NOT NULL, + created_at TEXT NOT NULL, + updated_at TEXT NOT NULL, + CHECK (ingestion_key = payload_hash) + ); + CREATE INDEX idx_atlas_outbox_pending + ON atlas_ingestion_outbox(state, created_at, ingestion_key); + """ + ) + _replace_schema_metadata_with_legacy_version(connection, "5") + connection.commit() + finally: + connection.close() + + with self.assertRaisesRegex(StoreError, "orchestrator store is not prepared"): + SQLiteStore(legacy_database) + def test_v4_database_additively_migrates_receipt_trust_tables_and_keeps_data( self, ) -> None: From 6da5295b83e2900f5865c71b9929166dfcf4b260 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E5=93=80=E6=B4=9B=E8=8A=99?= Date: Tue, 1 Sep 2026 22:34:35 +0800 Subject: [PATCH 38/39] test: restore authentic legacy outbox fixtures --- .../test_orchestrator_external_bootstrap.py | 28 +++++++ .../test_orchestrator_store_messaging.py | 73 +++++++++++++++---- 2 files changed, 88 insertions(+), 13 deletions(-) diff --git a/mcp-tools/tests/test_orchestrator_external_bootstrap.py b/mcp-tools/tests/test_orchestrator_external_bootstrap.py index 099a500..12f74e3 100644 --- a/mcp-tools/tests/test_orchestrator_external_bootstrap.py +++ b/mcp-tools/tests/test_orchestrator_external_bootstrap.py @@ -36,6 +36,34 @@ def _replace_schema_metadata_with_legacy_version( connection: sqlite3.Connection, version: str, ) -> None: + if version in {"6", "7", "8", "9", "10"}: + connection.execute("DROP TABLE atlas_ingestion_outbox") + connection.execute( + """ + CREATE TABLE atlas_ingestion_outbox ( + ingestion_key TEXT PRIMARY KEY, + acceptance_id TEXT NOT NULL UNIQUE + REFERENCES code_task_acceptances(acceptance_id), + payload_json TEXT NOT NULL, + payload_hash TEXT NOT NULL UNIQUE, + state TEXT NOT NULL + CHECK (state IN ('pending', 'projected', 'quarantined')), + attempt_count INTEGER NOT NULL + CHECK (attempt_count BETWEEN 0 AND 16), + last_error_code TEXT NOT NULL, + reason_codes_json TEXT NOT NULL, + created_at TEXT NOT NULL, + updated_at TEXT NOT NULL, + CHECK (ingestion_key = payload_hash) + ) + """ + ) + connection.execute( + """ + CREATE INDEX idx_atlas_outbox_pending + ON atlas_ingestion_outbox(state, created_at, ingestion_key) + """ + ) connection.execute("DROP TABLE schema_metadata") connection.execute( """ diff --git a/mcp-tools/tests/test_orchestrator_store_messaging.py b/mcp-tools/tests/test_orchestrator_store_messaging.py index 1a65651..fba2bd3 100644 --- a/mcp-tools/tests/test_orchestrator_store_messaging.py +++ b/mcp-tools/tests/test_orchestrator_store_messaging.py @@ -299,7 +299,10 @@ def setUp(self) -> None: contract_subscriptions=(self._hash("contract"),), ) self.coordinator_lease = self.store.acquire_lease( - "coordinator", "coordinator-owner", "2026-08-03T01:00:00+00:00", now=self._NOW + "coordinator", + "coordinator-owner", + "2026-08-03T01:00:00+00:00", + now=self._NOW, ) self.worker_one_lease = self.store.acquire_lease( "worker-one", "worker-one-owner", "2026-08-03T01:00:00+00:00", now=self._NOW @@ -324,7 +327,9 @@ def tearDown(self) -> None: self.store.close() self._temporary_directory.cleanup() - def _assignment_kwargs(self, *, correlation_id: str = "assignment-one") -> dict[str, object]: + def _assignment_kwargs( + self, *, correlation_id: str = "assignment-one" + ) -> dict[str, object]: return { "recipient_epoch": self.worker_one_lease.epoch, "direction": "coordinator_to_worker", @@ -345,7 +350,9 @@ def _assignment_kwargs(self, *, correlation_id: str = "assignment-one") -> dict[ def _role_row_count(self) -> int: return int( - self.store._connection.execute("SELECT COUNT(*) FROM role_envelopes").fetchone()[0] + self.store._connection.execute( + "SELECT COUNT(*) FROM role_envelopes" + ).fetchone()[0] ) def test_sensitive_fields_each_reject_without_any_role_row_or_event(self) -> None: @@ -375,7 +382,9 @@ def test_sensitive_fields_each_reject_without_any_role_row_or_event(self) -> Non self.assertEqual(0, self._role_row_count()) self.assertEqual((), self.store.list_events(self.workflow.id)) - def test_terminal_sensitive_fields_each_reject_without_a_new_row_or_event(self) -> None: + def test_terminal_sensitive_fields_each_reject_without_a_new_row_or_event( + self, + ) -> None: self.store.enqueue_role_envelope( self.workflow.id, "coordinator", @@ -449,7 +458,9 @@ def test_terminal_sensitive_fields_each_reject_without_a_new_row_or_event(self) ) self.assertEqual("ROLE_ENVELOPE_INVALID", captured.exception.code) self.assertEqual(baseline_rows, self._role_row_count()) - self.assertEqual(baseline_events, len(self.store.list_events(self.workflow.id))) + self.assertEqual( + baseline_events, len(self.store.list_events(self.workflow.id)) + ) def test_peer_sensitive_fields_each_reject_without_a_new_row_or_event(self) -> None: self.store.enqueue_role_envelope( @@ -522,7 +533,9 @@ def test_peer_sensitive_fields_each_reject_without_a_new_row_or_event(self) -> N ) self.assertEqual("ROLE_ENVELOPE_INVALID", captured.exception.code) self.assertEqual(baseline_rows, self._role_row_count()) - self.assertEqual(baseline_events, len(self.store.list_events(self.workflow.id))) + self.assertEqual( + baseline_events, len(self.store.list_events(self.workflow.id)) + ) def test_wrong_peer_capability_fails_without_a_new_role_envelope(self) -> None: assignment = self.store.enqueue_role_envelope( @@ -573,11 +586,15 @@ def test_wrong_peer_capability_fails_without_a_new_role_envelope(self) -> None: now=self._NOW, ) self.assertEqual("CAPABILITY_INVALID", captured.exception.code) - self.assertEqual(assignment.delivery_id, self.store._role_envelope_from_row( - self.store._connection.execute( - "SELECT * FROM role_envelopes WHERE delivery_id = ?", (assignment.delivery_id,) - ).fetchone() - ).delivery_id) + self.assertEqual( + assignment.delivery_id, + self.store._role_envelope_from_row( + self.store._connection.execute( + "SELECT * FROM role_envelopes WHERE delivery_id = ?", + (assignment.delivery_id,), + ).fetchone() + ).delivery_id, + ) self.assertEqual(1, self._role_row_count()) self.assertEqual(event_count, len(self.store.list_events(self.workflow.id))) @@ -585,7 +602,9 @@ def test_wrong_peer_capability_fails_without_a_new_role_envelope(self) -> None: class SQLiteStoreRoleEnvelopeSchemaTests(unittest.TestCase): """The v6 upgrade creates the same unique envelope hash surface as a fresh v7 DB.""" - def test_v6_upgrade_and_fresh_v7_have_the_partial_unique_envelope_hash_index(self) -> None: + def test_v6_upgrade_and_fresh_v7_have_the_partial_unique_envelope_hash_index( + self, + ) -> None: temporary = tempfile.TemporaryDirectory() self.addCleanup(temporary.cleanup) legacy_database = Path(temporary.name) / "legacy.sqlite" @@ -617,6 +636,32 @@ def test_v6_upgrade_and_fresh_v7_have_the_partial_unique_envelope_hash_index(sel ) """ ) + connection.execute( + """ + CREATE TABLE atlas_ingestion_outbox ( + ingestion_key TEXT PRIMARY KEY, + acceptance_id TEXT NOT NULL UNIQUE + REFERENCES code_task_acceptances(acceptance_id), + payload_json TEXT NOT NULL, + payload_hash TEXT NOT NULL UNIQUE, + state TEXT NOT NULL + CHECK (state IN ('pending', 'projected', 'quarantined')), + attempt_count INTEGER NOT NULL + CHECK (attempt_count BETWEEN 0 AND 16), + last_error_code TEXT NOT NULL, + reason_codes_json TEXT NOT NULL, + created_at TEXT NOT NULL, + updated_at TEXT NOT NULL, + CHECK (ingestion_key = payload_hash) + ) + """ + ) + connection.execute( + """ + CREATE INDEX idx_atlas_outbox_pending + ON atlas_ingestion_outbox(state, created_at, ingestion_key) + """ + ) connection.commit() finally: connection.close() @@ -629,7 +674,9 @@ def test_v6_upgrade_and_fresh_v7_have_the_partial_unique_envelope_hash_index(sel for store in (legacy, fresh): indexes = { str(row["name"]): int(row["unique"]) - for row in store._connection.execute("PRAGMA index_list(role_envelopes)").fetchall() + for row in store._connection.execute( + "PRAGMA index_list(role_envelopes)" + ).fetchall() } self.assertEqual(1, indexes[expected_index]) sql = store._connection.execute( From 832d9e1dfc9df20b791843a791f8f0a2f5243c95 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E5=93=80=E6=B4=9B=E8=8A=99?= Date: Tue, 1 Sep 2026 23:17:36 +0800 Subject: [PATCH 39/39] test: close rejected registry probe cleanly --- mcp-tools/tests/test_fastlane_host_adapter.py | 110 +++++++++++------- 1 file changed, 65 insertions(+), 45 deletions(-) diff --git a/mcp-tools/tests/test_fastlane_host_adapter.py b/mcp-tools/tests/test_fastlane_host_adapter.py index 4b66b64..b644f39 100644 --- a/mcp-tools/tests/test_fastlane_host_adapter.py +++ b/mcp-tools/tests/test_fastlane_host_adapter.py @@ -140,9 +140,7 @@ def _authenticated_v5_fixture() -> dict[str, object]: host_capabilities=host, scheduler_facts=scheduler, ) - request_binding_hash = fastlane_routing.v5_request_binding_hash( - routing_requests[0] - ) + request_binding_hash = fastlane_routing.v5_request_binding_hash(routing_requests[0]) attestation: dict[str, object] = { "schema": "2718lab-devkit/host-child-route-attestation-v1", "status": "attested", @@ -914,9 +912,10 @@ def test_environment_session_round_trips_evidence_and_typed_dispatch( received: list[OperationReceipt] = [] def host_reply() -> None: - assert host.receive_project_index_attestation(now=1_700_000_000) == fixture[ - "index_query" - ] + assert ( + host.receive_project_index_attestation(now=1_700_000_000) + == fixture["index_query"] + ) probe = host.receive_capability_probe_v2(now=1_700_000_000) host.send_capability_report_v2( probe=probe, @@ -924,9 +923,7 @@ def host_reply() -> None: scheduler_facts=fixture["scheduler"], now=1_700_000_000, ) - routing_request = host.receive_routing_attestation_request( - now=1_700_000_000 - ) + routing_request = host.receive_routing_attestation_request(now=1_700_000_000) assert list(routing_request.routing_requests) == fixture["routing_requests"] host.send_routing_attestation_response( request=routing_request, @@ -964,9 +961,10 @@ def host_reply() -> None: session = host_session.HostSession.from_environment( environ={}, platform="posix", clock=lambda: 1_700_000_000 ) - assert session.send_project_index_attestation(fixture["index_query"]) == fixture[ - "index_query" - ] + assert ( + session.send_project_index_attestation(fixture["index_query"]) + == fixture["index_query"] + ) capability = session.resolve_capability_snapshot_v2( call_intent_hash=fixture["call_intent_hash"], preparation_id=fixture["preparation_id"], @@ -1040,9 +1038,7 @@ def test_fast_lane_terminal_ack_removes_only_completed_assignment() -> None: write_scope=("src/task_v5_b.py",), dispatch_order=1, ) - mappings = [ - adapter._dispatch_fact_mapping(fact) for fact in (fact_a, fact_b) - ] + mappings = [adapter._dispatch_fact_mapping(fact) for fact in (fact_a, fact_b)] batch: dict[str, object] = { "schema": "2718lab-devkit/fastlane-host-dispatch-batch-v1", "action": "dispatch_all", @@ -1076,6 +1072,7 @@ def refill_callback(trigger: Mapping[str, object]) -> dict[str, object]: receipt = {"state": "NO_QUEUED_WORK", **dict(trigger)} refill_receipts.append(receipt) return receipt + dispatch_reader = threading.Thread( target=lambda: received_dispatch.append( host.receive_fast_lane_dispatch_batch(now=1_700_000_000) @@ -1096,7 +1093,9 @@ def refill_callback(trigger: Mapping[str, object]) -> dict[str, object]: dispatch_reader.join(timeout=2) assert received_dispatch == [receipt] expected = dict( - session._pending_fast_lane_terminals[(batch["batch_hash"], fact_a.task_id)].expected + session._pending_fast_lane_terminals[ + (batch["batch_hash"], fact_a.task_id) + ].expected ) terminal: dict[str, object] = { "schema": "2718lab-devkit/fastlane-worker-terminal-result-v1", @@ -1162,7 +1161,10 @@ def refill_callback(trigger: Mapping[str, object]) -> dict[str, object]: time.sleep(0.01) assert len(refill_receipts) == 1 assert ack["refill_trigger_hash"] == refill_receipts[0]["refill_trigger_hash"] - assert (batch["batch_hash"], fact_a.task_id) not in session._pending_fast_lane_terminals + assert ( + batch["batch_hash"], + fact_a.task_id, + ) not in session._pending_fast_lane_terminals assert (batch["batch_hash"], fact_b.task_id) in session._pending_fast_lane_terminals close_started = time.monotonic() session.close() @@ -1192,16 +1194,22 @@ def test_compiler_evidence_cross_language_fixed_vector() -> None: "predecessor_hash", "source_plan_hash", } - assert host_bridge._normalize_compiler_evidence_response( - vector["response"], request=request, now=1_700_000_000 - ) == vector["response"] + assert ( + host_bridge._normalize_compiler_evidence_response( + vector["response"], request=request, now=1_700_000_000 + ) + == vector["response"] + ) assert vector["frame"]["canonical_payload_hash"] == _canonical_hash( vector["request"] ) for item in vector["project_index_attestations"]: - assert host_bridge._normalize_project_index_attestation( - item["payload"], now=1_700_000_000 - ) == item["payload"] + assert ( + host_bridge._normalize_project_index_attestation( + item["payload"], now=1_700_000_000 + ) + == item["payload"] + ) read_fd, write_fd = os.pipe() bridge = host_bridge.InheritedHandleHostBridge.from_file_descriptors( read_fd=read_fd, @@ -1235,34 +1243,42 @@ def test_registry_hash_tamper_never_issues_compiler_evidence( ) -> None: adapter = _adapter() import devkit_runtime.host_session as host_session - from devkit_runtime.host_bridge import InheritedHandleHostBridge + from devkit_runtime.host_bridge import HostBridgeError, InheritedHandleHostBridge child, host = _pipe_pair() fact = _dispatch_fact(adapter, task="task-1", scope="src/a.py") request = _planner_request(adapter, (fact,)) fact_mapping = adapter._dispatch_fact_mapping(fact) + shutdown = threading.Event() + evidence_request_received = threading.Event() + unexpected_errors: list[HostBridgeError] = [] def host_reply() -> None: - evidence_request = host.receive_compiler_evidence_request(now=1_700_000_000) - host._send_private( - kind="compiler_evidence_response", - action_id=evidence_request.preparation_id, - payload={ - "schema": "2718lab-devkit/compiler-evidence-response-v1", - "preparation_id": evidence_request.preparation_id, - "request_hash": evidence_request.request_hash, - "reasoning_effort": evidence_request.reasoning_effort, - "verified_route_result_hashes": [fact.route.routing_result_hash], - "verified_lease_scope_bindings": [ - adapter._lease_scope_binding_hash(fact) - ], - "dispatch_facts": [fact_mapping], - "dispatch_binding_hashes": [fact_mapping["dispatch_binding_hash"]], - "nonce": evidence_request.nonce, - "expires_at": evidence_request.expires_at, - "registry_binding_hash": _hash("0"), - }, - ) + try: + evidence_request = host.receive_compiler_evidence_request(now=1_700_000_000) + evidence_request_received.set() + host._send_private( + kind="compiler_evidence_response", + action_id=evidence_request.preparation_id, + payload={ + "schema": "2718lab-devkit/compiler-evidence-response-v1", + "preparation_id": evidence_request.preparation_id, + "request_hash": evidence_request.request_hash, + "reasoning_effort": evidence_request.reasoning_effort, + "verified_route_result_hashes": [fact.route.routing_result_hash], + "verified_lease_scope_bindings": [ + adapter._lease_scope_binding_hash(fact) + ], + "dispatch_facts": [fact_mapping], + "dispatch_binding_hashes": [fact_mapping["dispatch_binding_hash"]], + "nonce": evidence_request.nonce, + "expires_at": evidence_request.expires_at, + "registry_binding_hash": _hash("0"), + }, + ) + except HostBridgeError as error: + if not shutdown.is_set(): + unexpected_errors.append(error) thread = threading.Thread(target=host_reply, daemon=True) thread.start() @@ -1285,9 +1301,13 @@ def host_reply() -> None: == adapter.NO_SAFE_WORK ) finally: - thread.join(timeout=2) + shutdown.set() child.close() + thread.join(timeout=2) host.close() + assert not thread.is_alive() + assert not evidence_request_received.is_set() + assert unexpected_errors == [] @pytest.mark.parametrize(