From eef9df6ca682ff641ac9e2dc09c0c6833f471a42 Mon Sep 17 00:00:00 2001 From: Juan Antonio Osorio Date: Tue, 15 Sep 2026 18:47:50 +0300 Subject: [PATCH 01/16] docs: draft native Kubernetes execution provider plan Record the separate optional provider boundary, acceptance scenarios, and unresolved technical contracts. Keep implementation as a draft stack above the plan branch. Co-Authored-By: Mecatl --- docs/acceptance/README.md | 4 + .../acceptance/native-kubernetes-execution.md | 194 ++++++++++++++++++ docs/adr/0343-native-kubernetes-execution.md | 176 ++++++++++++++++ docs/adr/README.md | 1 + 4 files changed, 375 insertions(+) create mode 100644 docs/acceptance/native-kubernetes-execution.md create mode 100644 docs/adr/0343-native-kubernetes-execution.md diff --git a/docs/acceptance/README.md b/docs/acceptance/README.md index d918c46efb..9b56772fe7 100644 --- a/docs/acceptance/README.md +++ b/docs/acceptance/README.md @@ -111,6 +111,10 @@ PR after verification. There is no cleanup or status-only PR. ## Plans +- [Native Kubernetes execution provider](native-kubernetes-execution.md) — separately deployed, + optional controller/executor service in this monorepo, with retained workspaces and kind + qualification. Status: draft; implementation is planned as a draft PR stacked on the plan PR. + - [Development-spine work classification](adr-scope-classification.md) — routes Spike, Routine, Bounded, and Architectural work; keeps acceptance planning for substantive work while reserving ADRs for genuinely durable architecture decisions. Status: landed in this diff --git a/docs/acceptance/native-kubernetes-execution.md b/docs/acceptance/native-kubernetes-execution.md new file mode 100644 index 0000000000..2c7035fe7d --- /dev/null +++ b/docs/acceptance/native-kubernetes-execution.md @@ -0,0 +1,194 @@ +# Native Kubernetes execution — acceptance plan + +**Contract:** human-reviewed/v2 +**Work classification:** Architectural — introduces a controller/executor trust boundary, durable environment ownership, Kubernetes resources, and an optional mecak8s deployment integration. +**Decision record:** [ADR 0343](../adr/0343-native-kubernetes-execution.md) +**Phase:** native Kubernetes execution first slice +**Status:** draft, 2026-09-15. Planning artifact only; material protocol and lifecycle decisions remain open. +**Delivery:** Split. Human-authorized draft stack: the Plan / Interface PR targets `main`; the subsequent draft Implementation PR targets `plan/native-kubernetes-execution`. This explicitly permits draft implementation before plan merge, but neither PR is approved or merge-ready by that authorization alone. +**Expected tasks:** deferred to orchestration after the exact interfaces are specified. +**Issue:** None — planning request has no linked issue. +**Plan PR:** draft on `plan/native-kubernetes-execution`. +**Approved baseline:** absent; observed repository baseline `16a8e3b735bdc26f4c958cdf24fd1e7ac00a38f9` is evidence only. + +All named verification tests below describe future implementation proof, not existing coverage. + +The smallest demonstrable slice is an independently deployed execution environment provider service +with its own controller and executors. Mecak8s is an optional client: its adapter obtains a persistent +Kubernetes workspace and bound Shell through the provider API without owning Pods, PVCs, or the +controller lifecycle. The provider keeps the existing `tool.Environment` and file-tool schemas, +uses an operator-selected profile, proves behavior offline in kind, and makes no remote-project-ingestion claim. +ADR 0048 continues to govern storage-free mecak8s; this separate provider does not supersede it. + +## Human decisions + +- [x] Keep the execution environment provider a separate service in this monorepo. — Decision: the provider owns its controller, CRDs, Pods, PVCs, binaries, images, chart, and lifecycle; mecak8s consumes its API through an optional adapter and remains storage-free under ADR 0048, which is not superseded. +- [x] Choose a purpose-built controller rather than Agent Sandbox. — Decision: own the controller/executor boundary and its CRD. +- [x] Keep Kubernetes execution optional. — Decision: disabled/default deployments remain Kubernetes-execution-free. +- [x] Retain one PVC per logical environment by default. — Decision: session deletion does not delete a workspace; committed data is removed only through explicit retirement. Storage class, size, and quotas are operator-configured. +- [x] Set the first-slice source and execution boundary. — Decision: use a blank tool-seeded workspace, existing filesystem tools, foreground Shell, and operator-global instructions only, with no warm pools or automatic idle suspension. Defer Git clone, private credentials, remote project ingestion, background commands, and isolated delegation; schedules remain excluded. +- [x] Set the initial security and qualification posture. — Decision: initial use is authenticated and authorized in an operator-controlled cluster, not hardened hostile multitenancy; use mock/kind first, then an optional bounded real-model smoke. +- [x] Choose mTLS service links plus least-privilege grants scoped to environment, operation, and ownership epoch. — Decision: uncertain takeover fails closed; old-writer termination or external fencing is required before replacement. +- [x] Handle live-provider credentials only through a trusted runtime loader. — Decision: it may be consumed directly from its protected file channel to a narrowly scoped HARNESS-only Secret, with redacted failures, before child processes or tools can inherit it. The agent never accesses the file or Secret. No agent or tool reads, displays, or loads it into model/tool output, argv, Helm values, disk manifests, git, logs, artifacts, or the execution workload; mock tests do not access it. Endpoint, protocol, and model selection still require verified nonsecret configuration. +- [ ] Draft and review the exact private control/data-plane API: wire fields/errors, allocation transaction, ownership/reference lifecycle, idempotency, attach/reattach, command stream/cancel/status, and compatibility behavior. +- [ ] Draft and review the exact CRD and configuration schemas, names, versioning, validation, and compatibility/migration rules; approved defaults do not approve exact field names, sizes, or values. +- [ ] Draft and security-review the grant trust, issuance, verification, rotation/revocation, replay bounds, cancellation/termination acknowledgement, and external-fencing proof contract. + +## Interface contract + +This Plan / Interface PR contains documentation only. The user requested a later draft implementation +PR stacked on this plan branch. Record its exact plan commit as the working baseline, not as a merged +approval. Resolve the open technical contracts in this plan before dispatching affected implementation; +keep both PRs draft until their respective review gates are satisfied. + +These are **candidate** surfaces for review, not implemented or approved contracts. + +- **gRPC / protobuf:** Public Harness protobuf remains unchanged in the first slice. A private mTLS execution protocol is required for `ValidateProfile`, idempotent `EnsureEnvironment`, exact `AttachEnvironment`, bounded command start/stream/cancel/status, and retirement. Exact messages, field numbers, status/error mapping, fencing fields, stream resume semantics, and authorization metadata remain an unchecked Human decision; the plan cannot become `proposed` until they are written exactly. +- **Exported Go APIs / interfaces:** No `engine/` exported API change is proposed: `tool.Environment` remains immutable identity + Workspace + ReadLedger + optional bound CommandRunner (`engine/tool/environment.go:9`). Candidate host-only seams are a side-effect-free configuration validator/describer and an allocation/reattachment provider implemented under `internal/adapter`; their exact signatures and whether they extend or wrap the placement provider remain open. `Bind` cannot be used for startup validation because its request has no allocation identity and may allocate (`internal/adapter/server/placement.go:99-117`), while exact `Reattach` is distinct (`internal/adapter/server/placement.go:149`). +- **Tool schemas:** Read/ListDir/Edit/Write/Copy/Move/Remove/Grep/Glob/Shell schemas remain byte-compatible. The remote catalogue is explicit attenuation: the first slice advertises supported filesystem tools and Shell, excludes unsupported SkillDraft/Parallel/delegation/schedule paths as decided, and never silently routes an unsupported remote operation to local execution. Existing versioned CreateFile/ReplaceFile semantics remain (`engine/tool/tool.go:353-424`); Shell's documented POSIX bypass of file CAS remains honest (`engine/tool/tool.go:380-390`). +- **CLI / config:** Kubernetes execution remains disabled by default. The provider is packaged and released in this monorepo with its own binaries, images, and `deploy/helm/mecatl-execution` chart; the default mecak8s chart installs none of them and creates no Kubernetes client, informer, or execution goroutine. Candidate mecak8s configuration includes an enablement switch, endpoint, operator-selected profile, mTLS references, and operator-configured storage class/size/quotas. Exact names, precedence, validation behavior, Secret key layout, and schema compatibility remain to be drafted and reviewed. +- **Events / persistence:** The CRD and configuration schema remain future technical specifications. The approved storage behavior is one PVC per logical environment, retained by default across session deletion and executor replacement; session deletion is not workspace deletion, and committed data retires only through explicit authorized retirement. Storage class, size, and quotas are operator-configured, with exact field names and values still to be defined. `EnvironmentRef.Revision` remains immutable persisted identity, distinct from a transient execution-fence epoch; Pod UID is never durable session identity. The allocation transaction, ownership/reference lifecycle, status shape, exact mapping, and lifecycle protocol remain open for authoring and review. +- **Security / authority:** Initial qualification is for authenticated, authorized use in an operator-controlled cluster, not hardened hostile multitenancy. The chosen approach is mTLS service links plus least-privilege grants scoped to the exact environment, operation, and ownership epoch. The supervisor/controller and arbitrary-shell workload remain separate trust domains; no controller credential, signing key, provider key, or default service-account token enters the workload. The exact grant trust, issuance, verification, revocation, cancellation/termination acknowledgement, and fencing contract remains unresolved and blocks proposal. Any uncertain takeover fails closed: old-writer termination or external compute/storage fencing must be proven before replacement. +- **Compatibility / migration:** Additive and opt-in: existing local and `no-fs` sessions retain current behavior, and disabled mecak8s performs no Kubernetes-execution API calls or resource installation. Remote selection is explicit and fails closed when unavailable, with no local fallback. Existing Clear/Fork behavior is not environment-serialized today; the candidate contract would add environment exclusion while preserving same-`EnvironmentRef` successor semantics, subject to the unchecked lifecycle decision. PR #580 may inform behavioral conformance but is neither a dependency nor a protocol source. + +## In scope — 7 scenarios, in implementation order + +### Scenario 1 — Disabled means absent, including startup validation + +The optional deployment boundary follows [draft ADR 0343](../adr/0343-native-kubernetes-execution.md). +Current startup provider validation reaches `Bind` (`docs/design/IMPLEMENTATION-NOTES.md:6549-6560`), +so execution configuration needs a side-effect-free validation path rather than a fake allocation. + +**Acceptance:** +- AC1.1: With execution disabled, mecak8s creates no Kubernetes execution client, informer, goroutine, CRD, RBAC, controller, executor, PVC, or Pod, and performs no execution API call. + - verify: `TestNativeKubernetesExecution_Scenario1_DisabledHasNoEffects` +- AC1.2: The provider service deploys, restarts, and reconciles independently of mecak8s. The default mecak8s chart has no dependency on the provider chart and installs no provider resource; enabling the client adapter requires explicit endpoint/profile configuration. The adapter needs no Pod/PVC/controller management RBAC. + - verify: `TestNativeKubernetesExecution_Scenario1_DefaultChartIndependent` +- AC1.3: Preflight validates configured profiles and endpoint/provider compatibility without allocating an environment. Allocation occurs only at an authorized session-binding operation; enabled configuration that is incomplete fails startup before allocation, and a configured but unavailable endpoint fails validation clearly with no local fallback. + - verify: `TestADR_0343_PreflightNeverAllocates` +- AC1.4: Disabling the mecak8s client integration neither adopts nor deletes existing `ExecutionEnvironment` CRs, PVCs, or Pods and does not stop the separate provider's reconciliation for its existing allocations or other authorized clients. + - verify: `TestNativeKubernetesExecution_Scenario1_DisabledDoesNotAdoptOrDeleteExistingResources` + +### Scenario 2 — One idempotent logical environment is allocated per binding + +The controller owns a logical environment, PVC, and executor lifecycle rather than equating identity +with an ephemeral Pod, as decided in [draft ADR 0343](../adr/0343-native-kubernetes-execution.md). + +**Acceptance:** +- AC2.1: Repeating the same authorized ensure request with the same final idempotency identity returns the same environment generation and does not create a second PVC or Pod. + - verify: `TestNativeKubernetesExecution_Scenario2_IdempotentEnsure` +- AC2.2: A stale/different profile, digest, storage request, generation, or owner fails with the final reviewed conflict/precondition error and never adopts or mutates an unrelated allocation. + - verify: `TestADR_0343_AllocationIdentityFailsClosed` +- AC2.3: Only operator-configured profile references select digest-pinned images and storage; public clients/models cannot submit arbitrary images, Pod specs, paths, URLs, source credentials, or Kubernetes object names. + - verify: `TestNativeKubernetesExecution_Scenario2_ProfileIsOperatorSelected` + +### Scenario 3 — Existing file tools and Shell are transparent but honest + +The adapter preserves the immutable environment and independent read ledger +(`engine/tool/environment.go:9`; `engine/tool/ledger.go:1`) and the existing WorkspaceReader seam +(`engine/tool/tool.go:314`), under [draft ADR 0343](../adr/0343-native-kubernetes-execution.md). + +**Acceptance:** +- AC3.1: A fresh environment can create, read, conditionally edit, replace, copy, move, remove, list, grep, and glob a small Go fixture with unchanged tool schemas and the existing read-before-edit/version behavior where that contract applies. + - verify: `TestNativeKubernetesExecution_Scenario3_FileToolConformance` +- AC3.2: Read/Edit/Write preserve their applicable recorded-read and CreateFile/ReplaceFile CAS contracts; Copy/Move/Remove preserve their own positive conformance contracts and do not require an unrelated content read ledger/CAS precondition. + - verify: `TestInvariant_remote_execution_preserves_file_version_protocol` +- AC3.3: File access is physically confined through symlink traversal rather than lexical checks alone, and every read, mutate, list, search, status, stream, and cancel request is authorized for the exact environment; stale, expired, revoked, or wrong-environment grants are denied. + - verify: `TestADR_0343_RemoteFilesystemAndOperationAuthorizationFailClosed` +- AC3.4: Shell is bound to the same environment namespace as Workspace. Cancellation distinguishes requested cancellation, acknowledged complete process termination, and externally proven compute/storage fencing; namespace teardown includes detached descendants or fails closed. Output is capped and valid UTF-8 repaired, and Shell-created file changes are not falsely claimed to participate in Workspace CAS. + - verify: `TestNativeKubernetesExecution_Scenario3_ShellNamespaceCancelAndCASDisclosure` + +### Scenario 4 — Caller isolation and execution ownership survive failures + +A session lease is session-only and its token is not consulted for writes +(`engine/port/lease.go:37-48`); [draft ADR 0343](../adr/0343-native-kubernetes-execution.md) +therefore requires independent environment execution fencing. + +**Acceptance:** +- AC4.1: Caller A cannot discover, attach, execute in, stream from, cancel, or retire caller B's environment; hidden and absent allocations are indistinguishable at the public boundary. + - verify: `TestNativeKubernetesExecution_Scenario4_CallerEnvironmentIsolation` +- AC4.2: Every command is authorized for one immutable environment revision and a distinct transient execution-fence epoch; lease loss or grant revocation prevents new commands and cancels/fences active work without exposing capability credentials to the workload or another caller's output. Same-Pod placement alone is not a security boundary. + - verify: `TestADR_0343_ExecutionGrantIsEnvironmentAndFenceScoped` +- AC4.3: Timeout, Pod deletion, controller restart, Lease timeout, or network partition alone never authorizes a replacement executor while an old writer may run; unknown fencing state fails closed and requires the reviewed manual/external fencing path. + - verify: `TestADR_0343_PartitionCannotAuthorizeTakeoverByTimeout` + +### Scenario 5 — Restart preserves workspace while retirement is deliberate + +Exact reattachment and safe lifecycle behavior follow [draft ADR 0343](../adr/0343-native-kubernetes-execution.md) +and remain separate from the session-only lease contract (`engine/port/lease.go:37-48`). + +**Acceptance:** +- AC5.1: After mecak8s and controller restart, exact reattachment to the persisted environment revision restores the same PVC data and a correctly bound runner; an unavailable/mismatched generation fails with no default/local fallback. + - verify: `TestNativeKubernetesExecution_Scenario5_RestartReattachesPVC` +- AC5.2: Executor Pod replacement preserves the logical environment and PVC while current Pod UID may change without a spec `metadata.generation` change; `observedGeneration` records processed spec generation, and status records current Pod/PVC UID references without using them as durable session identity. + - verify: `TestADR_0343_LogicalIdentityOutlivesPodUID` +- AC5.3: The lifecycle matrix is explicit: removal of one reference retains the environment; removal of the last reference retains it by default; committed data retires only through explicit authorized retirement; a retiring environment rejects new bindings and successors. An unreadable store/reference is not an orphan and is retained. + - verify: `TestNativeKubernetesExecution_Scenario5_RetentionAndSafeRetirement` +- AC5.4: A pending allocation whose session association might have committed is not garbage-collected merely after TTL; collection waits for conclusive reconciliation that publication is absent. Explicit retirement of a live/shared environment is refused while any live reference exists unless a separately reviewed retire/quiesce policy authorizes it. + - verify: `TestADR_0343_PendingAllocationAndLiveReferenceRetirementFailClosed` +- AC5.5: Default chart uninstall never deletes runtime CRs/PVCs; deletion of CRDs while live resources exist is destructive and unsupported, and no automatic hook deletes CRDs. Owner references alone are not a lifecycle proof. + - verify: `TestADR_0343_UninstallRetentionAndCRDDeletionSafety` +- AC5.6: PVC persistence is not documented as node-disaster recovery; kind host-local storage proves restart survival only, and a dead or unobservable node is outside that positive guarantee. + - verify: inspection — human review of deployment documentation; `task docs` checks links and structure after authorized tracked changes + +### Scenario 6 — Remote catalog and project-source boundaries are explicit + +The first slice avoids claiming that local project trust applies remotely: current admission is rooted +in local composition (`internal/app/project_ingestion.go:20`) and AGENTS discovery reads Workspace +(`engine/prompt/builder.go:337`). The boundary is recorded in [draft ADR 0343](../adr/0343-native-kubernetes-execution.md). + +**Acceptance:** +- AC6.1: A remote execution session receives operator-global instructions and only its explicitly supported catalog; project AGENTS/rules/skills and Git source ingestion are off unless a later reviewed source/trust contract enables them. + - verify: `TestNativeKubernetesExecution_Scenario6_NoFalseRemoteProjectIngestion` +- AC6.2: Schedules remain excluded. Background commands, isolated Subagent/Parallel/Team delegation, SkillDraft, Git/source operations, and unsupported successor modes are rejected with named capability/precondition errors rather than running locally or silently omitting work; `no-fs` remains unchanged. + - verify: `TestNativeKubernetesExecution_Scenario6_UnsupportedPathsNeverFallBackLocal` +- AC6.3: The candidate Clear/Fork contract adds environment exclusion while preserving same-remote-`EnvironmentRef` successor semantics; it makes no claim that current Clear/Fork already serializes shared environments, adds no public Harness field, and never lets a model choose an environment/profile. + - verify: `TestNativeKubernetesExecution_Scenario6_SuccessorSemantics` + +### Scenario 7 — kind proves an offline coding flow; live qualification is separate + +The authoritative e2e is a focused mock-only task, following [draft ADR 0343](../adr/0343-native-kubernetes-execution.md). +The existing broad suite creates cluster/images before key capture (`e2e/k8s/suite_test.go:32-80`), +so the focused task must not inherit ambient provider credentials. + +**Acceptance:** +- AC7.1: Candidate `task e2e:k8s:execution` runs on the supported generic toolchain, including CI, with a unique owned kind cluster, a scratch kubeconfig, and explicit `--kubeconfig`/context on every command. It deploys the separate chart plus explicitly connected mecak8s and uses only the deterministic mock provider. + - verify: `TestNativeKubernetesExecution_Scenario7_FocusedKindMockFlow` +- AC7.2: The mock flow seeds a small Go fixture through file tools, asks the harness to implement a function/tests, runs `go test` through bound Shell, and verifies the persisted artifact and restart/reattach behavior. + - verify: `TestNativeKubernetesExecution_Scenario7_MockCodingFlow` +- AC7.3: Failure artifacts are bounded and sanitized; cleanup targets only the recorded owned cluster. Destructive live cleanup requires confirmation and never mutates ambient kubeconfig/context. + - verify: `TestNativeKubernetesExecution_Scenario7_OwnershipAndSanitizedArtifacts` +- AC7.4: Candidate `task e2e:k8s:execution:live` is a separate explicit, bounded real-model smoke after mock/kind qualification. It runs only against verified nonsecret endpoint, protocol, and model configuration in an authenticated, authorized operator-controlled cluster; it does not claim a hard dollar cap from token/run limits. + - verify: `TestNativeKubernetesExecution_Scenario7_LiveQualificationIsExplicit` +- AC7.5: A trusted runtime-only loader may consume the approved credential directly from its protected file channel and create a narrowly scoped HARNESS-only Secret through a protected channel before child processes or tools can inherit it. The agent never accesses the file or Secret; failures are redacted, and the credential never appears in model/tool output, argv, Helm values, disk manifests, git, logs, artifacts, or the execution workload. Mock tests do not access it. + - verify: `TestNativeKubernetesExecution_Scenario7_LiveSecretAndSpendBoundary` + +## Out of scope + +| Item | Defer-to | Decision | +|---|---|---| +| GitHub cloning, private-repository credentials, and authoritative remote project trust/AGENTS/rules/skills | Separate source-ingestion architecture plan | Blank/PVC workspace and tool-seeded fixture are sufficient for slice 1. | +| Agent Sandbox integration | Not planned for this capability | Human selected a purpose-built controller. | +| Public Harness placement/profile/image/path fields | Later architecture only if justified | First slice keeps the public API unchanged and server-owned. | +| Generic capabilities framework | Later, only with multiple proven consumers | Use narrow optional interfaces/catalog attenuation rather than speculative abstraction. | +| Schedules and remote delegation/fork support | Separate reviewed expansion | First slice rejects unsupported paths without local fallback. | +| Automated force takeover under uncertain fencing | Not permitted | Unknown old executor state is fail-closed/manual until a reviewed external fencing proof exists. | +| Node-disaster PVC recovery and production multi-zone storage | Production storage design | kind host-local PVC is only a restart-survival proof. | +| Provider credential discovery, arbitrary endpoint forwarding, or live model inference beyond the approved runtime-loader boundary | Separate explicit qualification | The approved credential is not read in planning/mock tests; verified nonsecret endpoint/protocol/model configuration governs any optional smoke. | +| Depending on or copying PR #580's transport | Independent work | Share behavioral conformance where useful; do not inherit unqualified cancellation/auth protocol. | + +## Definition of done + +1. The outstanding technical specifications — private API, allocation/ownership lifecycle, CRD/config schema and compatibility, and grant/revocation/termination/fencing proof — are authored and reviewed in this draft Plan / Interface PR before the affected implementation is dispatched. +2. The authorized implementation passes applicable `task lint`, `task test`, `task docs`, `task api:check`, `task site:build`, and `go run ./cmd/mecademo` gates. +3. `task ac-trace-strict` resolves every named proof when the plan becomes `landed`. +4. The focused mock-only kind task passes on the supported toolchain with a unique owned cluster; optional live qualification is reported separately and never substitutes for it. +5. The draft implementation PR targets `plan/native-kubernetes-execution`, links the draft Plan / Interface PR and exact working baseline, reports interface conformance, and completes panel review before becoming merge-ready. Neither PR is merged automatically. + +## Deferred decisions and known risks + +- Exact wire fields/errors, lifecycle signatures, CRD schema, and auth/fencing protocol are material open decisions in `## Human decisions`; implementation must not fill them in opportunistically. +- A PVC can preserve files while a stale command still mutates them. Session leasing does not fence command execution, and a timeout is not proof that a partitioned executor stopped. +- Default kind does not enforce NetworkPolicy. A negative isolation test needs a network-policy-capable CNI and must state what was actually enforced. +- Controller/executor credentials, signing material, and provider secrets must never enter arbitrary-shell workloads. +- The future controller, informer/cache, execution client, connection pool, grant verifier/rotation state, and stream/cancellation registry all outlive one tool call and must be inventoried in `docs/adr/0027-cloud-native.md` when tracked design work is authorized. diff --git a/docs/adr/0343-native-kubernetes-execution.md b/docs/adr/0343-native-kubernetes-execution.md new file mode 100644 index 0000000000..7e993276e8 --- /dev/null +++ b/docs/adr/0343-native-kubernetes-execution.md @@ -0,0 +1,176 @@ +# ADR 0343 — Optional native Kubernetes execution environments + +- Status: Draft +- Date: 2026-09-15 +- Scope: independently deployed execution environment provider service; mecak8s is an optional client; controller/executor, logical environment, PVC, Pod, authorization, fencing, and kind qualification boundaries +- Supersedes: none + +## Context + +Mecak8s externalizes session state to Redis and uses Kubernetes Leases for session-level +single-writer ownership, but it does not provide a production remote execution environment. +The current remote environment adapter is a fake (`internal/adapter/remoteenv/remoteenv.go:1`). +`tool.Environment` already binds immutable identity, Workspace, an independent read ledger, and an +optional CommandRunner (`engine/tool/environment.go:9`), so a Kubernetes implementation should +meet that runtime contract without widening the engine API. + +A session lease does not fence an already-running remote command: its token is intentionally not +consulted on session writes (`engine/port/lease.go:37-48`). Kubernetes Pod replacement, network +partition, and controller restart therefore introduce a separate execution-ownership problem. +Likewise, startup placement validation currently reaches `Bind` +(`docs/design/IMPLEMENTATION-NOTES.md:6549-6560`), but using an allocating operation as preflight +would leak PVCs and Pods. + +The human has approved the direction and defaults: a purpose-built execution environment provider service in this monorepo, separate from mecak8s, with its own binaries, images, chart lifecycle, optional client integration, and mock/kind-first qualification. The companion +[acceptance plan](../acceptance/native-kubernetes-execution.md) and this ADR remain draft because the +exact private protocol, lifecycle, CRD/configuration schema, and grant/fencing specifications still +need authoring and review. + +## Approved direction and defaults; technical specifications remain draft + +This ADR records the approved direction: a separate execution environment provider in this monorepo; +it does **not supersede ADR 0048**. ADR 0048 governs storage-free mecak8s. Mecak8s remains a client of +the provider, keeps session state in its existing stores, and neither mounts provider workspace PVCs nor +owns executor/controller lifecycle. The provider owns its CRDs, Pods, PVCs, authorization, +reconciliation, binaries, images, and chart lifecycle. Its API is not tied to a mecak8s process or +release; other authorized Mecatl compositions can use an adapter without embedding the controller. +Provider allocation handles remain separate from harness session IDs. Exact API/version compatibility +still requires specification and review. + +### 1. Own a narrow controller/executor boundary + +Ship a dedicated controller and executor integration rather than Agent Sandbox. The controller +reconciles a logical execution environment consisting of a future `ExecutionEnvironment` custom +resource, persistent workspace PVC, and replaceable executor Pod. Durable identity is the logical +environment plus immutable `EnvironmentRef.Revision`, which persists unchanged across run ownership +turnover; it is not a fence epoch or restart counter. Pod and PVC names and UIDs are observed references +used to detect replacement/adoption errors. + +The future CRD must express operator-selected, digest-pinned profile/image/storage policy and distinguish +processed spec generation from the current executor Pod UID. Only operator-configured profiles may choose +image, storage, or source policy. Public clients and models cannot submit arbitrary Pod specs, images, +paths, URLs, credentials, or Kubernetes names. Its exact group/version/kind, schema, conditions, +mutability rules, and source metadata remain to be authored and reviewed. + +### 2. Preserve the core environment and tool contracts + +Keep `engine/tool` and public Harness APIs unchanged. A host-side adapter maps an exact persisted +`session.EnvironmentRef` to a complete remote `tool.Environment`. Existing filesystem tool schemas, +read-before-edit ledger, CreateFile/ReplaceFile version protocol, and bound Shell semantics remain. +Shell writes retain their documented ability to bypass Workspace CAS +(`engine/tool/tool.go:353-424`); the product must disclose that honestly rather than promising a +stronger remote filesystem transaction. + +Introduce a side-effect-free configuration/profile validation operation distinct from idempotent +allocation. Candidate private operations include profile validation, ensure, exact attach, command +start/stream/status/cancel, and retirement. Their precise Go signatures and private wire protocol +are deliberately undecided and block promotion of this ADR. + +### 3. Make installation and connection explicitly optional + +The provider has its own binaries, images, and Helm chart lifecycle in this monorepo. The mecak8s chart +has no default dependency on it. Kubernetes execution is disabled by default; an explicit endpoint and +operator-selected profile connect the optional client. Disabled means no execution Kubernetes client, +informer, goroutine, API call, CRD/RBAC install, or execution resource. + +An explicitly selected remote environment fails clearly at validation if its configured endpoint is +unavailable, and fails closed at runtime if the provider is lost or its exact immutable revision cannot +reattach. It never falls back to the local project root. Existing local and `no-fs` behavior is unchanged. + +Exact packaging names, values/flag precedence, mTLS Secret layout, storage class/size/quota configuration, +and compatibility rules remain to be authored and reviewed. + +### 4. Separate execution fencing from session leasing + +The chosen approach is mTLS service links plus a least-privilege execution grant scoped to the exact +environment, operation, and ownership epoch. Initial qualification is authenticated and authorized use in +an operator-controlled cluster, not hardened hostile multitenancy. The arbitrary-shell workload receives +no controller credential, signing key, provider key, or default service-account token. Same-Pod or +localhost placement is not treated as a trust boundary. + +Grant revocation prevents new commands. Cancellation request, acknowledged complete process termination, +and externally proven compute/storage fencing are distinct states. Namespace process teardown must include +detached descendants or fail closed. Pod deletion, Lease timeout, a timeout, controller restart, or +partition is not evidence that an old command stopped and cannot alone authorize takeover. If the old +executor cannot be positively terminated or externally fenced, replacement fails closed. + +The exact grant trust, issuer/verifier, claims, audience, rotation/revocation, replay protection, command +cancellation acknowledgement, process-group termination, and takeover/fencing proof remain to be authored +and security-reviewed. No existing SPIFFE issuer is assumed. + +### 5. Start with a persistent blank workspace and an attenuated catalogue + +The first slice provisions a blank/PVC-backed workspace that existing filesystem tools seed. It supports +foreground Shell and operator-global instructions only. It has no warm pools or automatic idle suspension. +Git clone, private source credentials, remote project ingestion, background commands, and isolated delegation +are deferred. Project AGENTS/rules/skills remain disabled until a separate root/source trust design exists. Current +project admission is local-root based (`internal/app/project_ingestion.go:20`). + +Schedules remain excluded. SkillDraft and unsupported remote Subagent/Parallel/Team paths return explicit +unsupported/precondition results and never execute locally. Existing Clear/Fork is not environment- +serialized today; the future lifecycle contract must preserve same-`EnvironmentRef` successor semantics +while adding appropriate environment exclusion. Exact capability/error and lifecycle mechanics remain +technical specifications, not new Human direction decisions. + +### 6. Retain committed workspaces by default + +One persistent PVC per logical environment is retained across executor and mecak8s/controller restarts. +Session deletion does not delete a workspace. The lifecycle default is: removal of one reference retains +the environment; removal of the last reference retains it; committed data retires only through explicit +authorized retirement; and a retiring environment rejects new bindings and successors. An unreadable +store/reference is not an orphan and is retained. A pending allocation whose session association might +have committed cannot be collected merely after TTL: publication absence must first be conclusively +reconciled. Explicit retirement of a live/shared environment is refused while any live reference exists +unless a later reviewed retire/quiesce policy specifies otherwise. + +Default chart uninstall never deletes runtime CRs/PVCs. Deleting CRDs while live resources exist is +destructive and unsupported; no automatic hook deletes CRDs. Owner references alone do not prove safe +lifecycle ownership. PVC persistence does not claim node-disaster recovery; kind host-local storage +proves only process/Pod restart survival, not recovery from a dead or unobservable node. + +Storage class, size, and quotas are operator-configured. Their exact configuration schema, sharing, +committed-state marker, retention mechanics, and retirement authorization remain to be authored and +reviewed. + +### 7. Qualify offline before optional live inference + +Add a focused mock-only kind task separate from the broad e2e suite. The task must run on the +supported generic toolchain, including CI. +It uses a unique owned cluster, a scratch kubeconfig, explicit kubeconfig/context on every command, +sanitized bounded artifacts, and confirmation before destructive live cleanup. Default kind is not +accepted as negative NetworkPolicy evidence; isolation testing needs a network-policy-capable CNI. + +A later real-model smoke is separate and bounded, after mock/kind qualification, for authenticated and +authorized use in an operator-controlled cluster. Endpoint, protocol, and model selection require verified +nonsecret configuration; credential usability does not authorize arbitrary endpoint forwarding. The approved +credential is not accessed by planning or mock tests. A trusted runtime-only loader may consume its protected +file directly through a protected channel to create a narrowly scoped HARNESS-only Secret before child +processes or tools can inherit it; the agent never accesses the file or Secret. Failures are redacted, and +the credential never enters model/tool output, +argv, Helm values, disk manifests, git, logs, artifacts, or the execution workload. Token/run limits do not +claim a hard dollar cap. One bounded coding prompt is qualitative; deterministic mock behavior remains the +correctness gate. + +## Consequences + +**Positive.** Mecak8s can optionally bind tools to a persistent Kubernetes namespace without changing +tool schemas, engine APIs, or public placement authority. Disabled deployments remain free of CRDs, +RBAC, informers, clients, and resources. Logical identity survives Pod replacement, and kind can prove +the complete coding path offline. + +**Costs and risks.** Mecatl would own a controller, CRD, private protocol, PVC lifecycle, command +streaming/cancellation, and a security-critical fencing system. A persistent volume does not stop a +partitioned writer. Arbitrary Shell makes the executor a hostile-workload boundary. Network policy, +resource quotas, output caps, non-root execution, service-account-token suppression, and credential +separation are required but not sufficient. + +**Draft boundary.** The direction and defaults above are approved, but this ADR is not implementation-ready. +The next planning work authors and reviews the exact private wire fields/errors, allocation transaction and +ownership/reference lifecycle, CRD/configuration schema and compatibility, and grant trust/revocation/ +termination/fencing proof. No implementation may invent those contracts. + +## See also + +- [Native Kubernetes execution acceptance plan](../acceptance/native-kubernetes-execution.md) +- [ADR 0048 — mecak8s](0048-mecak8s.md) +- [ADR 0291 — server-owned session placement](0291-server-owned-session-placement.md) diff --git a/docs/adr/README.md b/docs/adr/README.md index d778e6204a..943c74d60e 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -83,6 +83,7 @@ Documentation/citation conventions are in [`docs/design/README.md`](../design/RE - [0348 — TypeScript SDK MCP authorization lifecycle](./0348-typescript-sdk-mcp-authorization-lifecycle.md) *(proposed; supersedes ADR 0304 Decision 3 only for the authorization-lifecycle resource)* - [0342 - Gate runs on unresolved live context windows](./0342-context-window-admission.md) *(supersedes ADR 0016 only for pre-swap run admission)* - [0346 - Prompt-cache breakpoints are protocol-native, never vendor-keyed](./0346-unified-prompt-cache-dialect.md) *(supersedes ADR 0100's prompt_cache_breakpoint deferral, its root cache_control dialect arm, and its OpenRouter TTL deferral; extends ADR 0334 to OpenRouter)* +- [0343 — Optional native Kubernetes execution provider](./0343-native-kubernetes-execution.md) *(draft; separate service, preserves ADR 0048)* - [0036 — `engine/` is its own Go module (monorepo via `go.work`)](./0036-engine-module.md) - [0037 — Engine public-API stability contract](./0037-engine-stability-contract.md) - [0038 — Event-sourced SessionStore rehydration (the reference fold)](./0038-event-sourced-rehydration.md) *(Decision 3 origin-opacity and no-public-replay clauses proposed to be superseded by 0337)* From 819b6af8d69cbe205f9bc066f48441e225058ed7 Mon Sep 17 00:00:00 2001 From: Juan Antonio Osorio Date: Wed, 16 Sep 2026 21:31:54 +0200 Subject: [PATCH 02/16] docs: select gRPC and real-provider kind qualification Co-Authored-By: Mecatl --- docs/acceptance/native-kubernetes-execution.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/acceptance/native-kubernetes-execution.md b/docs/acceptance/native-kubernetes-execution.md index 2c7035fe7d..b6f7ea5164 100644 --- a/docs/acceptance/native-kubernetes-execution.md +++ b/docs/acceptance/native-kubernetes-execution.md @@ -24,6 +24,8 @@ ADR 0048 continues to govern storage-free mecak8s; this separate provider does n - [x] Keep the execution environment provider a separate service in this monorepo. — Decision: the provider owns its controller, CRDs, Pods, PVCs, binaries, images, chart, and lifecycle; mecak8s consumes its API through an optional adapter and remains storage-free under ADR 0048, which is not superseded. - [x] Choose a purpose-built controller rather than Agent Sandbox. — Decision: own the controller/executor boundary and its CRD. +- [x] Use typed gRPC for the private provider network protocol. — Decision: replace HTTP/JSON with protobuf RPCs over mandatory mTLS; keep the public Harness API unchanged and preserve the separate provider service boundary. +- [x] Qualify the gRPC coding flow with a real OpenRouter credential. — Decision: run the explicit live kind target after mock qualification using only the trusted runtime loader; prove model tool execution and independently verify workspace artifacts and a successful test command, then restore mock configuration and delete only the run-owned Secret by UID. - [x] Keep Kubernetes execution optional. — Decision: disabled/default deployments remain Kubernetes-execution-free. - [x] Retain one PVC per logical environment by default. — Decision: session deletion does not delete a workspace; committed data is removed only through explicit retirement. Storage class, size, and quotas are operator-configured. - [x] Set the first-slice source and execution boundary. — Decision: use a blank tool-seeded workspace, existing filesystem tools, foreground Shell, and operator-global instructions only, with no warm pools or automatic idle suspension. Defer Git clone, private credentials, remote project ingestion, background commands, and isolated delegation; schedules remain excluded. @@ -43,7 +45,7 @@ keep both PRs draft until their respective review gates are satisfied. These are **candidate** surfaces for review, not implemented or approved contracts. -- **gRPC / protobuf:** Public Harness protobuf remains unchanged in the first slice. A private mTLS execution protocol is required for `ValidateProfile`, idempotent `EnsureEnvironment`, exact `AttachEnvironment`, bounded command start/stream/cancel/status, and retirement. Exact messages, field numbers, status/error mapping, fencing fields, stream resume semantics, and authorization metadata remain an unchecked Human decision; the plan cannot become `proposed` until they are written exactly. +- **gRPC / protobuf:** Public Harness protobuf remains unchanged. The private provider uses `mecatl.execution.v1.ExecutionProviderService`, defined in `contracts/proto/mecatl/execution/v1/execution.proto` and generated through `task generate`, over mandatory mTLS. Candidate unary RPCs are `ValidateProfile`, `EnsureEnvironment`, `AttachEnvironment`, `ReleaseReference`, `RetireEnvironment`, `Files`, `StartCommand`, `CommandStatus`, and `CancelCommand`. File payloads and opaque versions are protobuf bytes; file operations and command states are closed enums; sanitized gRPC statuses carry a validated stable error classification. Client targets are `host:port`; there is no provider REST compatibility path or automatic mutation retry. Credential-free Kubernetes exec stdin framing is a separate helper protocol, not the provider network API. Exact lifecycle and grant semantics remain under draft review; selecting gRPC does not mark the whole contract approved. - **Exported Go APIs / interfaces:** No `engine/` exported API change is proposed: `tool.Environment` remains immutable identity + Workspace + ReadLedger + optional bound CommandRunner (`engine/tool/environment.go:9`). Candidate host-only seams are a side-effect-free configuration validator/describer and an allocation/reattachment provider implemented under `internal/adapter`; their exact signatures and whether they extend or wrap the placement provider remain open. `Bind` cannot be used for startup validation because its request has no allocation identity and may allocate (`internal/adapter/server/placement.go:99-117`), while exact `Reattach` is distinct (`internal/adapter/server/placement.go:149`). - **Tool schemas:** Read/ListDir/Edit/Write/Copy/Move/Remove/Grep/Glob/Shell schemas remain byte-compatible. The remote catalogue is explicit attenuation: the first slice advertises supported filesystem tools and Shell, excludes unsupported SkillDraft/Parallel/delegation/schedule paths as decided, and never silently routes an unsupported remote operation to local execution. Existing versioned CreateFile/ReplaceFile semantics remain (`engine/tool/tool.go:353-424`); Shell's documented POSIX bypass of file CAS remains honest (`engine/tool/tool.go:380-390`). - **CLI / config:** Kubernetes execution remains disabled by default. The provider is packaged and released in this monorepo with its own binaries, images, and `deploy/helm/mecatl-execution` chart; the default mecak8s chart installs none of them and creates no Kubernetes client, informer, or execution goroutine. Candidate mecak8s configuration includes an enablement switch, endpoint, operator-selected profile, mTLS references, and operator-configured storage class/size/quotas. Exact names, precedence, validation behavior, Secret key layout, and schema compatibility remain to be drafted and reviewed. From a5030545e87e81aad12fba4c49f0c9933a634b5e Mon Sep 17 00:00:00 2001 From: Juan Antonio Osorio Date: Wed, 16 Sep 2026 21:50:32 +0200 Subject: [PATCH 03/16] docs: renumber native Kubernetes execution ADR Co-Authored-By: Mecatl --- .../acceptance/native-kubernetes-execution.md | 32 +++++++++---------- ...md => 0349-native-kubernetes-execution.md} | 2 +- docs/adr/README.md | 2 +- 3 files changed, 18 insertions(+), 18 deletions(-) rename docs/adr/{0343-native-kubernetes-execution.md => 0349-native-kubernetes-execution.md} (99%) diff --git a/docs/acceptance/native-kubernetes-execution.md b/docs/acceptance/native-kubernetes-execution.md index b6f7ea5164..fe47f92b0b 100644 --- a/docs/acceptance/native-kubernetes-execution.md +++ b/docs/acceptance/native-kubernetes-execution.md @@ -2,7 +2,7 @@ **Contract:** human-reviewed/v2 **Work classification:** Architectural — introduces a controller/executor trust boundary, durable environment ownership, Kubernetes resources, and an optional mecak8s deployment integration. -**Decision record:** [ADR 0343](../adr/0343-native-kubernetes-execution.md) +**Decision record:** [ADR 0349](../adr/0349-native-kubernetes-execution.md) **Phase:** native Kubernetes execution first slice **Status:** draft, 2026-09-15. Planning artifact only; material protocol and lifecycle decisions remain open. **Delivery:** Split. Human-authorized draft stack: the Plan / Interface PR targets `main`; the subsequent draft Implementation PR targets `plan/native-kubernetes-execution`. This explicitly permits draft implementation before plan merge, but neither PR is approved or merge-ready by that authorization alone. @@ -57,7 +57,7 @@ These are **candidate** surfaces for review, not implemented or approved contrac ### Scenario 1 — Disabled means absent, including startup validation -The optional deployment boundary follows [draft ADR 0343](../adr/0343-native-kubernetes-execution.md). +The optional deployment boundary follows [draft ADR 0349](../adr/0349-native-kubernetes-execution.md). Current startup provider validation reaches `Bind` (`docs/design/IMPLEMENTATION-NOTES.md:6549-6560`), so execution configuration needs a side-effect-free validation path rather than a fake allocation. @@ -67,20 +67,20 @@ so execution configuration needs a side-effect-free validation path rather than - AC1.2: The provider service deploys, restarts, and reconciles independently of mecak8s. The default mecak8s chart has no dependency on the provider chart and installs no provider resource; enabling the client adapter requires explicit endpoint/profile configuration. The adapter needs no Pod/PVC/controller management RBAC. - verify: `TestNativeKubernetesExecution_Scenario1_DefaultChartIndependent` - AC1.3: Preflight validates configured profiles and endpoint/provider compatibility without allocating an environment. Allocation occurs only at an authorized session-binding operation; enabled configuration that is incomplete fails startup before allocation, and a configured but unavailable endpoint fails validation clearly with no local fallback. - - verify: `TestADR_0343_PreflightNeverAllocates` + - verify: `TestADR_0349_PreflightNeverAllocates` - AC1.4: Disabling the mecak8s client integration neither adopts nor deletes existing `ExecutionEnvironment` CRs, PVCs, or Pods and does not stop the separate provider's reconciliation for its existing allocations or other authorized clients. - verify: `TestNativeKubernetesExecution_Scenario1_DisabledDoesNotAdoptOrDeleteExistingResources` ### Scenario 2 — One idempotent logical environment is allocated per binding The controller owns a logical environment, PVC, and executor lifecycle rather than equating identity -with an ephemeral Pod, as decided in [draft ADR 0343](../adr/0343-native-kubernetes-execution.md). +with an ephemeral Pod, as decided in [draft ADR 0349](../adr/0349-native-kubernetes-execution.md). **Acceptance:** - AC2.1: Repeating the same authorized ensure request with the same final idempotency identity returns the same environment generation and does not create a second PVC or Pod. - verify: `TestNativeKubernetesExecution_Scenario2_IdempotentEnsure` - AC2.2: A stale/different profile, digest, storage request, generation, or owner fails with the final reviewed conflict/precondition error and never adopts or mutates an unrelated allocation. - - verify: `TestADR_0343_AllocationIdentityFailsClosed` + - verify: `TestADR_0349_AllocationIdentityFailsClosed` - AC2.3: Only operator-configured profile references select digest-pinned images and storage; public clients/models cannot submit arbitrary images, Pod specs, paths, URLs, source credentials, or Kubernetes object names. - verify: `TestNativeKubernetesExecution_Scenario2_ProfileIsOperatorSelected` @@ -88,7 +88,7 @@ with an ephemeral Pod, as decided in [draft ADR 0343](../adr/0343-native-kuberne The adapter preserves the immutable environment and independent read ledger (`engine/tool/environment.go:9`; `engine/tool/ledger.go:1`) and the existing WorkspaceReader seam -(`engine/tool/tool.go:314`), under [draft ADR 0343](../adr/0343-native-kubernetes-execution.md). +(`engine/tool/tool.go:314`), under [draft ADR 0349](../adr/0349-native-kubernetes-execution.md). **Acceptance:** - AC3.1: A fresh environment can create, read, conditionally edit, replace, copy, move, remove, list, grep, and glob a small Go fixture with unchanged tool schemas and the existing read-before-edit/version behavior where that contract applies. @@ -96,40 +96,40 @@ The adapter preserves the immutable environment and independent read ledger - AC3.2: Read/Edit/Write preserve their applicable recorded-read and CreateFile/ReplaceFile CAS contracts; Copy/Move/Remove preserve their own positive conformance contracts and do not require an unrelated content read ledger/CAS precondition. - verify: `TestInvariant_remote_execution_preserves_file_version_protocol` - AC3.3: File access is physically confined through symlink traversal rather than lexical checks alone, and every read, mutate, list, search, status, stream, and cancel request is authorized for the exact environment; stale, expired, revoked, or wrong-environment grants are denied. - - verify: `TestADR_0343_RemoteFilesystemAndOperationAuthorizationFailClosed` + - verify: `TestADR_0349_RemoteFilesystemAndOperationAuthorizationFailClosed` - AC3.4: Shell is bound to the same environment namespace as Workspace. Cancellation distinguishes requested cancellation, acknowledged complete process termination, and externally proven compute/storage fencing; namespace teardown includes detached descendants or fails closed. Output is capped and valid UTF-8 repaired, and Shell-created file changes are not falsely claimed to participate in Workspace CAS. - verify: `TestNativeKubernetesExecution_Scenario3_ShellNamespaceCancelAndCASDisclosure` ### Scenario 4 — Caller isolation and execution ownership survive failures A session lease is session-only and its token is not consulted for writes -(`engine/port/lease.go:37-48`); [draft ADR 0343](../adr/0343-native-kubernetes-execution.md) +(`engine/port/lease.go:37-48`); [draft ADR 0349](../adr/0349-native-kubernetes-execution.md) therefore requires independent environment execution fencing. **Acceptance:** - AC4.1: Caller A cannot discover, attach, execute in, stream from, cancel, or retire caller B's environment; hidden and absent allocations are indistinguishable at the public boundary. - verify: `TestNativeKubernetesExecution_Scenario4_CallerEnvironmentIsolation` - AC4.2: Every command is authorized for one immutable environment revision and a distinct transient execution-fence epoch; lease loss or grant revocation prevents new commands and cancels/fences active work without exposing capability credentials to the workload or another caller's output. Same-Pod placement alone is not a security boundary. - - verify: `TestADR_0343_ExecutionGrantIsEnvironmentAndFenceScoped` + - verify: `TestADR_0349_ExecutionGrantIsEnvironmentAndFenceScoped` - AC4.3: Timeout, Pod deletion, controller restart, Lease timeout, or network partition alone never authorizes a replacement executor while an old writer may run; unknown fencing state fails closed and requires the reviewed manual/external fencing path. - - verify: `TestADR_0343_PartitionCannotAuthorizeTakeoverByTimeout` + - verify: `TestADR_0349_PartitionCannotAuthorizeTakeoverByTimeout` ### Scenario 5 — Restart preserves workspace while retirement is deliberate -Exact reattachment and safe lifecycle behavior follow [draft ADR 0343](../adr/0343-native-kubernetes-execution.md) +Exact reattachment and safe lifecycle behavior follow [draft ADR 0349](../adr/0349-native-kubernetes-execution.md) and remain separate from the session-only lease contract (`engine/port/lease.go:37-48`). **Acceptance:** - AC5.1: After mecak8s and controller restart, exact reattachment to the persisted environment revision restores the same PVC data and a correctly bound runner; an unavailable/mismatched generation fails with no default/local fallback. - verify: `TestNativeKubernetesExecution_Scenario5_RestartReattachesPVC` - AC5.2: Executor Pod replacement preserves the logical environment and PVC while current Pod UID may change without a spec `metadata.generation` change; `observedGeneration` records processed spec generation, and status records current Pod/PVC UID references without using them as durable session identity. - - verify: `TestADR_0343_LogicalIdentityOutlivesPodUID` + - verify: `TestADR_0349_LogicalIdentityOutlivesPodUID` - AC5.3: The lifecycle matrix is explicit: removal of one reference retains the environment; removal of the last reference retains it by default; committed data retires only through explicit authorized retirement; a retiring environment rejects new bindings and successors. An unreadable store/reference is not an orphan and is retained. - verify: `TestNativeKubernetesExecution_Scenario5_RetentionAndSafeRetirement` - AC5.4: A pending allocation whose session association might have committed is not garbage-collected merely after TTL; collection waits for conclusive reconciliation that publication is absent. Explicit retirement of a live/shared environment is refused while any live reference exists unless a separately reviewed retire/quiesce policy authorizes it. - - verify: `TestADR_0343_PendingAllocationAndLiveReferenceRetirementFailClosed` + - verify: `TestADR_0349_PendingAllocationAndLiveReferenceRetirementFailClosed` - AC5.5: Default chart uninstall never deletes runtime CRs/PVCs; deletion of CRDs while live resources exist is destructive and unsupported, and no automatic hook deletes CRDs. Owner references alone are not a lifecycle proof. - - verify: `TestADR_0343_UninstallRetentionAndCRDDeletionSafety` + - verify: `TestADR_0349_UninstallRetentionAndCRDDeletionSafety` - AC5.6: PVC persistence is not documented as node-disaster recovery; kind host-local storage proves restart survival only, and a dead or unobservable node is outside that positive guarantee. - verify: inspection — human review of deployment documentation; `task docs` checks links and structure after authorized tracked changes @@ -137,7 +137,7 @@ and remain separate from the session-only lease contract (`engine/port/lease.go: The first slice avoids claiming that local project trust applies remotely: current admission is rooted in local composition (`internal/app/project_ingestion.go:20`) and AGENTS discovery reads Workspace -(`engine/prompt/builder.go:337`). The boundary is recorded in [draft ADR 0343](../adr/0343-native-kubernetes-execution.md). +(`engine/prompt/builder.go:337`). The boundary is recorded in [draft ADR 0349](../adr/0349-native-kubernetes-execution.md). **Acceptance:** - AC6.1: A remote execution session receives operator-global instructions and only its explicitly supported catalog; project AGENTS/rules/skills and Git source ingestion are off unless a later reviewed source/trust contract enables them. @@ -149,7 +149,7 @@ in local composition (`internal/app/project_ingestion.go:20`) and AGENTS discove ### Scenario 7 — kind proves an offline coding flow; live qualification is separate -The authoritative e2e is a focused mock-only task, following [draft ADR 0343](../adr/0343-native-kubernetes-execution.md). +The authoritative e2e is a focused mock-only task, following [draft ADR 0349](../adr/0349-native-kubernetes-execution.md). The existing broad suite creates cluster/images before key capture (`e2e/k8s/suite_test.go:32-80`), so the focused task must not inherit ambient provider credentials. diff --git a/docs/adr/0343-native-kubernetes-execution.md b/docs/adr/0349-native-kubernetes-execution.md similarity index 99% rename from docs/adr/0343-native-kubernetes-execution.md rename to docs/adr/0349-native-kubernetes-execution.md index 7e993276e8..a0ea302551 100644 --- a/docs/adr/0343-native-kubernetes-execution.md +++ b/docs/adr/0349-native-kubernetes-execution.md @@ -1,4 +1,4 @@ -# ADR 0343 — Optional native Kubernetes execution environments +# ADR 0349 — Optional native Kubernetes execution environments - Status: Draft - Date: 2026-09-15 diff --git a/docs/adr/README.md b/docs/adr/README.md index 943c74d60e..30cda2b1e9 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -83,7 +83,7 @@ Documentation/citation conventions are in [`docs/design/README.md`](../design/RE - [0348 — TypeScript SDK MCP authorization lifecycle](./0348-typescript-sdk-mcp-authorization-lifecycle.md) *(proposed; supersedes ADR 0304 Decision 3 only for the authorization-lifecycle resource)* - [0342 - Gate runs on unresolved live context windows](./0342-context-window-admission.md) *(supersedes ADR 0016 only for pre-swap run admission)* - [0346 - Prompt-cache breakpoints are protocol-native, never vendor-keyed](./0346-unified-prompt-cache-dialect.md) *(supersedes ADR 0100's prompt_cache_breakpoint deferral, its root cache_control dialect arm, and its OpenRouter TTL deferral; extends ADR 0334 to OpenRouter)* -- [0343 — Optional native Kubernetes execution provider](./0343-native-kubernetes-execution.md) *(draft; separate service, preserves ADR 0048)* +- [0349 — Optional native Kubernetes execution provider](./0349-native-kubernetes-execution.md) *(draft; separate service, preserves ADR 0048)* - [0036 — `engine/` is its own Go module (monorepo via `go.work`)](./0036-engine-module.md) - [0037 — Engine public-API stability contract](./0037-engine-stability-contract.md) - [0038 — Event-sourced SessionStore rehydration (the reference fold)](./0038-event-sourced-rehydration.md) *(Decision 3 origin-opacity and no-public-replay clauses proposed to be superseded by 0337)* From 622a57ce02b2ac518e59147f5fe50f0efd13d38d Mon Sep 17 00:00:00 2001 From: Juan Antonio Osorio Date: Mon, 21 Sep 2026 19:42:19 +0300 Subject: [PATCH 04/16] docs: reconcile native Kubernetes shipping contract Co-Authored-By: OpenAI --- .../acceptance/native-kubernetes-execution.md | 365 +++++++++++++++--- docs/adr/0349-native-kubernetes-execution.md | 189 ++++++--- 2 files changed, 444 insertions(+), 110 deletions(-) diff --git a/docs/acceptance/native-kubernetes-execution.md b/docs/acceptance/native-kubernetes-execution.md index fe47f92b0b..514a52900e 100644 --- a/docs/acceptance/native-kubernetes-execution.md +++ b/docs/acceptance/native-kubernetes-execution.md @@ -4,14 +4,17 @@ **Work classification:** Architectural — introduces a controller/executor trust boundary, durable environment ownership, Kubernetes resources, and an optional mecak8s deployment integration. **Decision record:** [ADR 0349](../adr/0349-native-kubernetes-execution.md) **Phase:** native Kubernetes execution first slice -**Status:** draft, 2026-09-15. Planning artifact only; material protocol and lifecycle decisions remain open. -**Delivery:** Split. Human-authorized draft stack: the Plan / Interface PR targets `main`; the subsequent draft Implementation PR targets `plan/native-kubernetes-execution`. This explicitly permits draft implementation before plan merge, but neither PR is approved or merge-ready by that authorization alone. -**Expected tasks:** deferred to orchestration after the exact interfaces are specified. +**Status:** draft, amended 2026-09-21. Exact proposed contracts are authored below; API, schema, and security review approvals remain outstanding. +**Delivery:** Split. Human-authorized draft stack: the Plan / Interface PR targets `main`; the draft Implementation PR targets `plan/native-kubernetes-execution`. Authorization includes developing this amendment before plan merge, not approval or merge readiness. Human alone merges. +**Expected tasks:** draft implementation decomposition remains run-local; no approved orchestration baseline. **Issue:** None — planning request has no linked issue. -**Plan PR:** draft on `plan/native-kubernetes-execution`. -**Approved baseline:** absent; observed repository baseline `16a8e3b735bdc26f4c958cdf24fd1e7ac00a38f9` is evidence only. +**Plan PR:** [#1579](https://github.com/stacklok/mecatl/pull/1579), draft on `plan/native-kubernetes-execution`; implementation [#1614](https://github.com/stacklok/mecatl/pull/1614) remains draft. +**Approved baseline:** absent. Plan worktree baseline `a5030545e87e81aad12fba4c49f0c9933a634b5e` and implementation reference `deaf1c3d1dfe7ea92afc8fe826f1bc613080f219` are evidence, not approval. -All named verification tests below describe future implementation proof, not existing coverage. +The [implementation proof map](#implementation-proof-map) distinguishes existing draft coverage and +successful qualification from amendment proofs still pending. Implementation-only source paths below +refer to that pinned implementation reference; they are future code proofs on this documentation-only +branch, not claims that the files or tests already exist here. The smallest demonstrable slice is an independently deployed execution environment provider service with its own controller and executors. Mecak8s is an optional client: its adapter obtains a persistent @@ -29,29 +32,231 @@ ADR 0048 continues to govern storage-free mecak8s; this separate provider does n - [x] Keep Kubernetes execution optional. — Decision: disabled/default deployments remain Kubernetes-execution-free. - [x] Retain one PVC per logical environment by default. — Decision: session deletion does not delete a workspace; committed data is removed only through explicit retirement. Storage class, size, and quotas are operator-configured. - [x] Set the first-slice source and execution boundary. — Decision: use a blank tool-seeded workspace, existing filesystem tools, foreground Shell, and operator-global instructions only, with no warm pools or automatic idle suspension. Defer Git clone, private credentials, remote project ingestion, background commands, and isolated delegation; schedules remain excluded. -- [x] Set the initial security and qualification posture. — Decision: initial use is authenticated and authorized in an operator-controlled cluster, not hardened hostile multitenancy; use mock/kind first, then an optional bounded real-model smoke. +- [x] Set the initial security and qualification posture. — Decision: the first production slice is authenticated and authorized in an operator-controlled cluster, with the limits below, not hardened hostile multitenancy. Deterministic mock/kind qualification precedes the separately invoked native-provider OpenRouter coding qualification required by the checked live decision above; this is not an always-on live CI dependency. - [x] Choose mTLS service links plus least-privilege grants scoped to environment, operation, and ownership epoch. — Decision: uncertain takeover fails closed; old-writer termination or external fencing is required before replacement. - [x] Handle live-provider credentials only through a trusted runtime loader. — Decision: it may be consumed directly from its protected file channel to a narrowly scoped HARNESS-only Secret, with redacted failures, before child processes or tools can inherit it. The agent never accesses the file or Secret. No agent or tool reads, displays, or loads it into model/tool output, argv, Helm values, disk manifests, git, logs, artifacts, or the execution workload; mock tests do not access it. Endpoint, protocol, and model selection still require verified nonsecret configuration. -- [ ] Draft and review the exact private control/data-plane API: wire fields/errors, allocation transaction, ownership/reference lifecycle, idempotency, attach/reattach, command stream/cancel/status, and compatibility behavior. -- [ ] Draft and review the exact CRD and configuration schemas, names, versioning, validation, and compatibility/migration rules; approved defaults do not approve exact field names, sizes, or values. -- [ ] Draft and security-review the grant trust, issuance, verification, rotation/revocation, replay bounds, cancellation/termination acknowledgement, and external-fencing proof contract. +- [x] Develop scoped administrative authority in this existing draft stack. — Decision: a distinct administrator URI may administer only explicitly named creator URIs through `administratorFor`; this grants no data-plane or owner-attestation authority. Existing self-administration remains compatible, with no namespace-wide administrator grant. +- [ ] Review and approve the authored private control/data-plane API, reference transaction, run ownership, errors, and compatibility contract below. +- [ ] Review and approve the authored CRD/status schema v2 and configuration contract, including scoped administration, continuing provisioning retries, and retained-resource Helm lifecycle qualification. +- [ ] Security-review and approve the authored grant trust, rotation/revocation, replay bounds, termination acknowledgement, fencing, and retained authority-history contract. +- [ ] Decide the release maturity designation. The target remains an operator-controlled first production slice with explicit limits; the CRD name `v1alpha1` does not settle an “alpha” product label or waive any completion gate. ## Interface contract -This Plan / Interface PR contains documentation only. The user requested a later draft implementation -PR stacked on this plan branch. Record its exact plan commit as the working baseline, not as a merged -approval. Resolve the open technical contracts in this plan before dispatching affected implementation; -keep both PRs draft until their respective review gates are satisfied. - -These are **candidate** surfaces for review, not implemented or approved contracts. - -- **gRPC / protobuf:** Public Harness protobuf remains unchanged. The private provider uses `mecatl.execution.v1.ExecutionProviderService`, defined in `contracts/proto/mecatl/execution/v1/execution.proto` and generated through `task generate`, over mandatory mTLS. Candidate unary RPCs are `ValidateProfile`, `EnsureEnvironment`, `AttachEnvironment`, `ReleaseReference`, `RetireEnvironment`, `Files`, `StartCommand`, `CommandStatus`, and `CancelCommand`. File payloads and opaque versions are protobuf bytes; file operations and command states are closed enums; sanitized gRPC statuses carry a validated stable error classification. Client targets are `host:port`; there is no provider REST compatibility path or automatic mutation retry. Credential-free Kubernetes exec stdin framing is a separate helper protocol, not the provider network API. Exact lifecycle and grant semantics remain under draft review; selecting gRPC does not mark the whole contract approved. -- **Exported Go APIs / interfaces:** No `engine/` exported API change is proposed: `tool.Environment` remains immutable identity + Workspace + ReadLedger + optional bound CommandRunner (`engine/tool/environment.go:9`). Candidate host-only seams are a side-effect-free configuration validator/describer and an allocation/reattachment provider implemented under `internal/adapter`; their exact signatures and whether they extend or wrap the placement provider remain open. `Bind` cannot be used for startup validation because its request has no allocation identity and may allocate (`internal/adapter/server/placement.go:99-117`), while exact `Reattach` is distinct (`internal/adapter/server/placement.go:149`). -- **Tool schemas:** Read/ListDir/Edit/Write/Copy/Move/Remove/Grep/Glob/Shell schemas remain byte-compatible. The remote catalogue is explicit attenuation: the first slice advertises supported filesystem tools and Shell, excludes unsupported SkillDraft/Parallel/delegation/schedule paths as decided, and never silently routes an unsupported remote operation to local execution. Existing versioned CreateFile/ReplaceFile semantics remain (`engine/tool/tool.go:353-424`); Shell's documented POSIX bypass of file CAS remains honest (`engine/tool/tool.go:380-390`). -- **CLI / config:** Kubernetes execution remains disabled by default. The provider is packaged and released in this monorepo with its own binaries, images, and `deploy/helm/mecatl-execution` chart; the default mecak8s chart installs none of them and creates no Kubernetes client, informer, or execution goroutine. Candidate mecak8s configuration includes an enablement switch, endpoint, operator-selected profile, mTLS references, and operator-configured storage class/size/quotas. Exact names, precedence, validation behavior, Secret key layout, and schema compatibility remain to be drafted and reviewed. -- **Events / persistence:** The CRD and configuration schema remain future technical specifications. The approved storage behavior is one PVC per logical environment, retained by default across session deletion and executor replacement; session deletion is not workspace deletion, and committed data retires only through explicit authorized retirement. Storage class, size, and quotas are operator-configured, with exact field names and values still to be defined. `EnvironmentRef.Revision` remains immutable persisted identity, distinct from a transient execution-fence epoch; Pod UID is never durable session identity. The allocation transaction, ownership/reference lifecycle, status shape, exact mapping, and lifecycle protocol remain open for authoring and review. -- **Security / authority:** Initial qualification is for authenticated, authorized use in an operator-controlled cluster, not hardened hostile multitenancy. The chosen approach is mTLS service links plus least-privilege grants scoped to the exact environment, operation, and ownership epoch. The supervisor/controller and arbitrary-shell workload remain separate trust domains; no controller credential, signing key, provider key, or default service-account token enters the workload. The exact grant trust, issuance, verification, revocation, cancellation/termination acknowledgement, and fencing contract remains unresolved and blocks proposal. Any uncertain takeover fails closed: old-writer termination or external compute/storage fencing must be proven before replacement. -- **Compatibility / migration:** Additive and opt-in: existing local and `no-fs` sessions retain current behavior, and disabled mecak8s performs no Kubernetes-execution API calls or resource installation. Remote selection is explicit and fails closed when unavailable, with no local fallback. Existing Clear/Fork behavior is not environment-serialized today; the candidate contract would add environment exclusion while preserving same-`EnvironmentRef` successor semantics, subject to the unchecked lifecycle decision. PR #580 may inform behavioral conformance but is neither a dependency nor a protocol source. +This documentation-only amendment reconciles the draft implementation at `deaf1c3d1` with the +proposed contract. It does not turn authored specifications, advisory architect/devil review, or passing +CI into human approval. Scoped administration, provisioning convergence, and Helm lifecycle safeguards +are explicit amendment requirements still to be implemented and qualified. Other details below describe +the inspected draft, subject to the unchecked reviews above. + +- **gRPC / protobuf:** Public Harness protobuf is unchanged. The private `mecatl.execution.v1.ExecutionProviderService` uses mandatory mTLS, a `host:port` endpoint, protocol identifier `execution-grpc/1`, and an 8 MiB message bound. The exact wire schema is the implementation reference's `contracts/proto/mecatl/execution/v1/execution.proto`, generated with `task generate`. The RPC/field summary below is the review contract; there is no HTTP/JSON compatibility endpoint. +- **Exported Go APIs / interfaces:** No `engine/` exported API change. `tool.Environment` remains identity + Workspace + independent ReadLedger + optional bound CommandRunner (`engine/tool/environment.go`). Host-only seams in the implementation's `internal/adapter/server/placement.go` are specified below; Kubernetes dependencies stay outside the engine. +- **Tool schemas:** Existing filesystem tools and foreground Shell remain byte-compatible, including recorded reads and conditional CreateFile/ReplaceFile. Copy/Move/Remove retain their own conformance contracts. Shell writes bypass file CAS (`engine/tool/tool.go`). Remote catalog attenuation and its model-visible posture must be tested through the real composition factory. Unsupported operations never execute locally. +- **CLI / config:** Disabled by default. Mecak8s accepts `--execution-enabled`, `--execution-endpoint`, `--execution-profile`, `--execution-tls-ca`, `--execution-tls-cert`, and `--execution-tls-key`. Enablement requires all five string settings and OIDC caller ownership enforcement; it conflicts with `--workspace` and `--redis-filesystem`. The separate `deploy/helm/mecatl-execution` chart owns the provider deployment. The default mecak8s chart has no provider dependency. Exact provider profiles and security configuration are below. +- **Events / persistence:** No public event widening. Provider CR status, reference intents, run/operation claims, lifecycle receipts, security high-water state, and capacity reservations are durable provider state. Exact session `EnvironmentRef.Revision` remains immutable and separate from transient execution epoch and grant generation. The namespaced CRD and schema-v2 fields are below. +- **Security / authority:** TLS 1.3 with one canonical allowlisted client URI SAN, short-lived signed grants, exact owner/reference checks, and independent environment fencing. Scoped administration extends only the provider's existing client policy. No controller credentials, signing keys, provider secrets, or service-account token enter arbitrary-shell workloads. This is an operator-controlled cluster boundary, not a hostile-multitenancy claim. +- **Compatibility / migration:** Additive and opt-in for existing local and `no-fs` sessions, with no local fallback for remote failures. Private schema migration is explicit and quiesced; CRD upgrade is manual. The `administratorFor` extension changes neither public Harness nor engine nor private protobuf. Old strict decoders reject it; use a quiesced provider upgrade, not a claimed mixed-version rolling upgrade. Neither downgrade nor reinstall may reset authority high-water state. + +## Proposed technical specifications + +These sections define the proposed interface details referenced by the categories above. + +### Private RPC and host contract + +Wire fields use protobuf snake_case. `EnvironmentRef = {id, revision}` and `Owner = {issuer, subject}`. +`RequestContext = {environment, owner, binding_id, epoch, grant, run_id, claim_id, grant_generation}`. +Environment and owner matches are exact; a client cannot select a creator through any new wire field. +The host maps this ref to `session.EnvironmentRef{Kind:"kubernetes", ID:id, Revision:revision}`; +reattachment never substitutes a current default or Pod UID. + +| RPCs | Exact request/result and semantics | +|---|---| +| `ValidateProfile` | `{profile}` → `{profile, digest, capabilities, max_file_bytes, max_command_bytes, max_command_duration_millis}`. Side-effect-free preflight, never `Bind` or allocation. | +| `EnsureEnvironment` | `{binding_id, profile, owner, operation_id}` → `{environment, epoch, ready, grant, grant_expires_at, grant_generation}`. Server-minted final binding ID, authenticated creator, owner, and profile fingerprint identify one allocation; drift conflicts instead of adopting another environment. | +| `AttachEnvironment` | `{context, purpose:"session"}` → the same result fields as Ensure. Exact persisted ref and reference membership; attach does not confer an active execution claim. | +| `AcquireRun` | `{environment, owner, binding_id, run_id, operation_id, ttl_millis}` → `RunClaimResponse{environment, binding_id, run_id, claim_id, epoch, grant_generation, grant, expires_at}`. One published binding holds environment-wide execution ownership at a time. TTL is 30s–5m; adapter default 1m. | +| `RenewRun`, `ReleaseRun` | Both carry `{environment, owner, binding_id, run_id, claim_id, epoch, operation_id, grant_generation}`; Renew adds `ttl_millis` and returns RunClaimResponse; Release returns Empty. Exact resourceVersion CAS and operation replay checks prevent stale renewal/release. | +| `CommitReference`, `AbortReference`, `PrepareReferenceDelete`, `ConfirmReferenceDelete`, `CancelReferenceDelete` | `{environment, owner, binding_id, operation_id}` → Empty. Durable reference transaction described below. `ReleaseReference{context}` is a compatibility entry, not permission to drop ambiguous or active references. | +| `ReserveSuccessor` | `{environment, owner, source_binding_id, destination_binding_id, operation_id}` → `{environment}`. Reserve another reference to the same exact environment under exclusion; not a filesystem clone. | +| `ListReferenceIntents` | `{owner, limit, environment, binding_id}` → bounded `intents[{environment, binding_id, state, operation_id, source_binding_id, created_at, owner}]`. Client-scoped reconciliation; attested owner information returns only to the owning creator client. | +| `RetireEnvironment`, `ReplaceExecutor`, `RecoverEnvironment` | `{environment, owner, expected_execution_epoch, expected_pod_uid, expected_pvc_uid, operation_id}` → Empty. Administrative admission retains exact epoch/UID checks, reference/idle preconditions, and operation receipts. Recovery accepts independently observed exact terminal Pod proof, never a caller boolean asserting fencing. | +| `DeleteRetiredEnvironment` | `{environment, owner, expected_pvc_uid, operation_id}` → Empty. Explicit destructive retained-PVC deletion after retirement and reference exclusion, with UID-preconditioned deletion and capacity release; not default uninstall behavior. | +| `MigrateEnvironment` | `{environment, owner, optional expected_schema_version, expected_pod_uid, expected_pvc_uid, operation_id}` → Empty. Presence is mandatory; 0 or 1 selects the compatible prototype migration to 2. Unknown/current versions are not a general migration or rollback API. | +| `RevokeEnvironment` | `{environment, owner, expected_grant_generation, operation_id}` → `{grant_generation}`. CAS advances revocation generation without changing execution epoch; repeat exact operations return the receipt rather than revoking twice. | +| `Files` | `{context, operation, path, destination, pattern, data:bytes, version:bytes, limit}` → `{data:bytes, version:bytes, info, entries, paths, matches, authority_target, authority_workspace}`. Closed enum: READ, RESOLVE_AUTHORITY, STAT, CREATE, REPLACE, LIST, REMOVE, RENAME, COPY, GLOB, GREP; UNSPECIFIED is rejected. FileInfo carries `{name, size, mode, mod_time, is_dir}`; GrepMatch carries `{path, line, text}`. Paths remain confined inside the bound workspace. | +| `StartCommand` | `{context, command, timeout_millis}` → `{command_id, state, result}`. Foreground unary RPC, not a detached job or server stream. Result carries `{command_id, state, exit_code, stdout:bytes, stderr:bytes, next_offset, truncated, terminal_receipt}`. Closed states are RUNNING, SUCCEEDED, FAILED, CANCELLED, FENCE_UNKNOWN (UNSPECIFIED rejected). Context cancellation is not proof of process termination. | +| `CommandStatus`, `CancelCommand` | `{context, command_id, offset}`. Authenticate/authorize and validate first, then return gRPC `Unimplemented` for this foreground-only slice. No command-stream RPC is implemented. | + +Sanitized errors carry `ErrorDetail{code, retryable}`. Stable mappings are `invalid_argument`→InvalidArgument, +`unauthenticated`→Unauthenticated, `permission_denied`→PermissionDenied, `not_found`→NotFound, +`already_exists`→AlreadyExists, `conflict`/`version_mismatch`→Aborted, `not_ready`→Unavailable, +`fence_unknown`→FailedPrecondition, `resource_exhausted`→ResourceExhausted, and `internal`→Internal. +Only `not_ready` and `unauthenticated` can carry `retryable:true`; this is not permission to retry a +mutation automatically. Grant-expiry refresh is read-only. Backend details never cross the boundary; +absent and out-of-scope environments have the same non-disclosing denial. Foreground detached-control +`Unimplemented` is the explicit exception to the structured error vocabulary. + +Host-only interfaces keep allocation and execution separate: + +- `PlacementValidator.ValidatePlacement(context.Context) error` validates without allocation. +- `PlacementBindRequest.BindingID session.SessionID` is the final server-minted binding; + `PlacementBinding.Commit func(context.Context) error` publishes after store persistence, and + `Close func() error` aborts only a conclusively uncommitted reservation. +- `ExecutionAccess` supplies `Applies(session.EnvironmentRef) bool` and + `AcquireRun(context.Context, ExecutionRunRequest) (ExecutionRunHandle, error)`; + the request contains `Ref, Principal, BindingID, RunID`, and the handle supplies + `Environment() tool.Environment`, `Renew(context.Context) error`, and `Release(context.Context) error`. +- `PlacementSuccessorReservoir.ReserveSuccessor(context.Context, PlacementSuccessorRequest) (PlacementBinding, error)` + and `ReferenceLifecycle.PrepareReferenceDelete(context.Context, PlacementSuccessorRequest) (ReferenceDeleteHandle, error)` + use `Ref, Principal, SourceBindingID, DestinationBindingID`. ReferenceLifecycle also has `Applies`; + the delete handle has `Confirm(context.Context) error` and `Cancel(context.Context) error`. +- `ReferenceIntentLifecycle.ListReferenceIntents(context.Context, int) ([]ReferenceIntent, error)` and + `CommitReferenceIntent`, `ConfirmReferenceIntentDelete`, `CancelReferenceIntentDelete` (each + `(context.Context, ReferenceIntent) error`) reconcile durable ambiguity. ReferenceIntent contains + `Ref, Principal, BindingID, SourceBindingID, OperationID, PendingDelete`. +- `PlacementReattacher.Reattach(context.Context, PlacementReattachRequest) (PlacementBinding, error)` + uses exact `Ref, Principal, Scope, BindingID`. Public callers receive display metadata only. + +### Reference transaction, schema v2, and fencing + +Ensure reserves `PendingCreate` before session-store publication; confirmed publication permits +Commit→`Published`. Abort is safe only when publication is conclusively absent. A lost commit response +remains recoverable, never TTL garbage. Delete uses `Published`→`PendingDelete` before store deletion, +then Confirm removes the reference or Cancel restores Published after confirmed non-deletion. +Store unavailability retains the intent. Clear/Fork reserve a destination reference under environment +exclusion, publish it, and preserve the source on failure. Clear creates empty history, Fork copies valid +history, both keep the same remote ref; neither is a workspace clone. A retiring environment admits no +new reference or successor. Retirement requires no references, including pending ones. + +The implementation CRD is namespaced `execution.mecatl.dev/v1alpha1`, kind `ExecutionEnvironment`, +plural `executionenvironments`, with a status subresource. `v1alpha1` is the Kubernetes API name, +independent of the unresolved release maturity designation. The structural schema is in the implementation's +`deploy/helm/mecatl-execution/crds/executionenvironment.yaml`: + +| Area | Exact fields and constraints | +|---|---| +| `spec` | `schemaVersion=2`; immutable `allocationID, revision, ownerHash, clientHash, bindingID, requestFingerprint, profile, profileDigest, image, storageClass, storageSize, resources`. Optional immutable `ownerIssuer, ownerSubject` retain attested ownership. `desired` is Active or Retiring. Hashes are 64 lowercase hex characters; profileDigest is `sha256:` plus 64 hex; image is digest-pinned. `allocationID`/profile ≤63 characters, revision ≤64, bindingID ≤253. | +| Core `status` | `schemaVersion=2, observedGeneration, epoch, grantGeneration, fenceState, references, pvc{name,uid}, pod{name,uid}, conditions`. Epoch and grantGeneration are positive signed-int64-bounded counters; fenceState is Healthy or FenceUnknown. Conditions carry `type,status,reason,message,observedGeneration,lastTransitionTime`; they describe observed spec generation, not durable identity. | +| References | At most 64 records, map-keyed by `bindingID`, with `state, operationID, sourceBindingID, createdAt`; state is PendingCreate, Published, or PendingDelete. | +| `activeRun` | `bindingID, runID, claimID, operationID, ownerHash, clientHash, epoch, grantGeneration, expiresAt`. | +| `activeOperation` | `id, operation, startedAt, claimID, runID, epoch, holderID, renewedAt, expiresAt`. Lease-holder identity is replica-specific; an expired holder remains recorded for recovery. | +| Receipts | `renewReceipts[{operationID,fingerprint,expiresAt}]` and `revocationReceipts[{operationID,fingerprint,expectedGrantGeneration,grantGeneration}]`, each capped at 32. Receipt eviction does not authorize replay past current claim/generation preconditions. | +| Lifecycle | `lifecycleOperation{id,type,phase,expectedEpoch,expectedPodUID,expectedPVCUID,createdAt}`; types ReplaceExecutor, RetireEnvironment, DeleteRetiredEnvironment; phases Quiescing, WaitingForTermination, RemovingPodFinalizer, WaitingForPodDeletion, CreatingReplacement, DeletingPVC, ReleasingSlot. | +| Recovery/migration | `terminationProof{operationID,podUID,pvcUID,epoch,podPhase,observedAt}` (Succeeded/Failed only); `lastReplacement{operationID,previousPodUID,replacementPodUID,pvcUID,previousEpoch,replacementEpoch}`; `migrationOperation{id,fromSchema,expectedPodUID,expectedPVCUID}` and `lastMigrationOperationID`. Schema 0/1 is accepted only for explicit compatible migration, never ordinary execution; unknown schemas fail closed. | + +Run claims serialize environment use independently of session leases. A separate 30s operation-holder +lease renews every TTL/3. Every operation rechecks current claim, owner, creator, revision, epoch, and +grant generation against resourceVersion-CAS state. Holder loss, transport cancellation, revocation, or +uncertain helper completion cancels work and preserves fencing uncertainty; it does not clear operation +identity to let another writer start. Exact terminal proof is persisted before removing the executor +finalizer or replacing the Pod. Missing/unobservable Pods and missing or mismatched authoritative PVC/Pod +UIDs never justify automatic replacement. External compute/storage fencing requires an operator runbook; +the API has no force-takeover assertion. Helper process teardown must include detached descendants or +fail closed. Credential-free Kubernetes exec stdin framing is an internal helper transport, not another +provider network protocol or a trusted workload-supplied proof of node termination. + +**Provisioning amendment:** transient PVC/Pod create failures, including quota saturation, retain +sanitized Ready=false conditions and schedule rate-limited retries for as long as the CR exists. +Restoring quota must converge without another reference/spec change, restart, or manual reconcile. +There is no finite retry count that silently forgets an existing unready allocation. Identity/ownership +failures remain fail-closed with no replacement, however often reconciliation runs. The inspected queue +already rate-limits returned errors; create-error branches currently return the condition-write result, +which can be nil. Merely retaining that queue is not proof of continuing provisioning retries. + +### Provider configuration, rotation, and scoped administration + +The strict profile YAML is `profiles: {: ProfileSpec}`. ProfileSpec requires +`image, storageClass, storageSize, cpuRequest, memoryRequest, cpuLimit, memoryLimit, +ephemeralStorageRequest, ephemeralStorageLimit, tmpSizeLimit, runtimeClassName, +maxFileBytes, maxCommandBytes, maxCommandDuration, maxEnvironments`. Positive resource quantities, +request≤limit, available RuntimeClass, and digest-pinned images are required. Protocol ceilings are +5 MiB file payloads and 1 MiB command input/output bound; profiles tighten them. Command duration is +positive and ≤30m; maxEnvironments is 1–10,000. Helm values require +`provider.image, provider.securitySecretName, provider.securityManifest`, explicit ingress selectors, +API/DNS CIDRs, and enabled NetworkPolicy/resource governance. Replicas default to 2; stream/RPC/per-client +RPC limits default to 64/128/32. Arbitrary workload Pod specs and paths are not public inputs. + +`provider.securityManifest` is a JSON **string**, decoded strictly by the provider; it is not a new Helm +RBAC object. Manifest version 1 has `version, generation, issuer, audience, activeKeyID, grantTTL, +clockSkew, keys, tls, clients`. `keys[]` has `id, version, file, publicKeySHA256, activateAt, verifyUntil, +state` (active, verify-only, revoked); TLS has `certificateFile, privateKeyFile, clientCAFile`, confined +relative to the mounted security directory. Secret keys are the operator-chosen filenames referenced +by that manifest, not credential literals in values. At most 64 keys and 256 clients; grantTTL is +positive and ≤5m, clockSkew is non-negative and smaller than TTL. No SPIFFE issuer deployment is assumed. + +Ed25519 grants use header `{alg:"Ed25519", kid, typ:"MECATL-GRANT"}` and claims +`iss, aud, client, owner, binding_id, run_id, claim_id, environment, epoch, grant_generation, +operations, not_before, expires_at, nonce`. The provider verifies exact operation and current durable +claim, not just signature/expiry. Rotation reloads an immutable snapshot (default 2s polling) and checks +a durable ConfigMap high-water ledger `{generation,digest,fingerprints}` on each RPC, including existing +connections. Key windows, CA/client removal, equal-generation policy drift, key-ID/version reuse with +different material, and rollback fail closed. RevokeEnvironment advances the separate per-environment +grantGeneration. Neither revocation nor a changed security generation constitutes termination proof. + +**Scoped-admin amendment, exact proposed configuration:** add optional `administratorFor: []string` +to each `clients[]` entry, matching existing lowerCamelCase `mayAttestOwner` and `administrator`. +For example, the nonsecret manifest fragment is: + +```json +{"clients":[ + {"uri":"spiffe://example.com/mecatl/harness","mayAttestOwner":true,"administrator":false}, + {"uri":"spiffe://example.com/mecatl/operations","mayAttestOwner":false,"administrator":true, + "administratorFor":["spiffe://example.com/mecatl/harness"]} +]} +``` + +This is a policy fragment, not a complete install manifest. The proposed internal field is +`ClientPolicy.AdministratorFor []string`; no new public API, engine seam, proto field, or general RBAC +layer is needed. The strict manifest decoder owns validation because Helm's existing field is a string: + +- Absent/empty list preserves self-admin only when `administrator:true`. A nonempty list requires + `administrator:true`, contains at most 256 unique canonical URI strings, and rejects wildcards, + duplicates, malformed entries, and noncanonical forms. Reuse the existing URI validator: nonempty + scheme/host, lowercase canonical scheme/host, no userinfo, query, or fragment. No prefix matching. +- Entries name immutable **creator** URIs, not owners, namespaces, or admin-to-admin delegation. + A creator need not remain in the current client allowlist: revoking its login must not prevent + explicitly scoped retirement of its retained allocations. The list alone grants no owner attestation. +- One shared administrative subject decision loads the environment, preserves its immutable + `spec.clientHash`, and matches it against the authenticated admin's own URI hash or an exact listed + creator URI hash. Keep the actual caller separate from the matched creator; do not rewrite creation + identity or trust a client-supplied creator. Require exact ownerHash and revision before disclosing + lifecycle state, then retain each RPC's epoch/UID/schema/generation and operation-replay preconditions. +- Apply the same decision to RetireEnvironment, ReplaceExecutor, RecoverEnvironment, + DeleteRetiredEnvironment, MigrateEnvironment, and RevokeEnvironment, including replay and CAS retry + paths. A receipt is never an authorization bypass. Out-of-scope and missing targets are indistinguishable. +- Administration never grants AttachEnvironment, Files, Shell/StartCommand, run claims, reference + discovery/mutation, or `MayAttestOwner`. Their existing creator and owner checks remain unchanged. +- Include the sorted scope list in the canonical authority digest. Scope removal at a higher generation + must deny subsequent RPCs on already-open connections, including receipt replay; equal-generation scope + edits fail closed. Already admitted lifecycle intents may finish safe reconciliation, but removal + authorizes no new admin request. Old binaries reject the unknown field. Quiesce and upgrade all provider + replicas before publishing it; do not claim mixed-version rolling compatibility. + +### Helm lifecycle amendment + +Default uninstall retains runtime CRs/PVCs and may leave executor Pods running. Consequently it must +also retain workload default-deny/profile NetworkPolicies, the security authority high-water ConfigMap, +and `mecatl-execution-profile-allocations` capacity ledger. Those resources live as long as retained +allocations/executors, not just the provider Deployment. Never render empty data over a retained ledger, +reset its generation, or recreate capacity as empty. Retain operator-owned security material and the +nonsecret profile/security configuration needed for exact reattachment; no chart hook deletes them. + +Supported reinstall uses the **same release name and namespace** and unchanged resource names/profile +identity. It verifies retained resource identity and Helm ownership metadata, explicitly reuses/adopts +only that release's resources, preserves ledger contents, and fails on missing authority history or +foreign/ambiguous ownership while allocations survive. Fresh install initializes ledgers only when +there are no retained allocations to authorize. Changed release/namespace adoption, namespace deletion, +CRD deletion with live resources, force cleanup, arbitrary chart rollback, and authority-generation +reset are unsupported. CRDs require manual upgrade before a quiesced compatible schema migration; +unknown schema versions cannot be downgraded. Ordinary compatible upgrades also preserve live ledger data. + +Qualification must execute real Helm install→upgrade→uninstall→same-release reinstall with retained +PVC data and executors. During provider absence, enforcing-CNI probes must still deny forbidden traffic. +After reinstall, current credentials reattach exactly, old authority/grants stay rejected, capacity +reservations still constrain allocation, and mismatched ownership/history fails safely. Annotation +string checks and the existing no-deletion-hook test are necessary but insufficient. Inventory both +ledgers and retained isolation/configuration in the resource and rehydration ledgers in +`docs/adr/0027-cloud-native.md` during implementation; no destructive hooks or automatic force deletion. ## In scope — 7 scenarios, in implementation order @@ -83,6 +288,8 @@ with an ephemeral Pod, as decided in [draft ADR 0349](../adr/0349-native-kuberne - verify: `TestADR_0349_AllocationIdentityFailsClosed` - AC2.3: Only operator-configured profile references select digest-pinned images and storage; public clients/models cannot submit arbitrary images, Pod specs, paths, URLs, source credentials, or Kubernetes object names. - verify: `TestNativeKubernetesExecution_Scenario2_ProfileIsOperatorSelected` +- AC2.4: After transient PVC or Pod creation failure, including prolonged quota saturation, an existing allocation converges when the failure clears without unrelated CR/reference changes, restart, or manual reconcile. Retries remain rate-limited and continue while the object exists; wrong ownership and missing/mismatched authoritative UIDs never trigger adoption or replacement. + - verify: `TestNativeKubernetesExecution_Scenario2_ProvisioningRetriesUntilRecovery`; `TestKindExecutionProductionProvisioningConvergence` (new amendment proofs, PENDING) ### Scenario 3 — Existing file tools and Shell are transparent but honest @@ -95,7 +302,7 @@ The adapter preserves the immutable environment and independent read ledger - verify: `TestNativeKubernetesExecution_Scenario3_FileToolConformance` - AC3.2: Read/Edit/Write preserve their applicable recorded-read and CreateFile/ReplaceFile CAS contracts; Copy/Move/Remove preserve their own positive conformance contracts and do not require an unrelated content read ledger/CAS precondition. - verify: `TestInvariant_remote_execution_preserves_file_version_protocol` -- AC3.3: File access is physically confined through symlink traversal rather than lexical checks alone, and every read, mutate, list, search, status, stream, and cancel request is authorized for the exact environment; stale, expired, revoked, or wrong-environment grants are denied. +- AC3.3: File access is physically confined through symlink traversal rather than lexical checks alone, and every supported read/mutate/list/search/command request is authorized for the exact environment; stale, expired, revoked, or wrong-environment grants are denied. Detached status/cancel requests authorize before returning Unimplemented; this slice exposes no command-stream RPC. - verify: `TestADR_0349_RemoteFilesystemAndOperationAuthorizationFailClosed` - AC3.4: Shell is bound to the same environment namespace as Workspace. Cancellation distinguishes requested cancellation, acknowledged complete process termination, and externally proven compute/storage fencing; namespace teardown includes detached descendants or fails closed. Output is capped and valid UTF-8 repaired, and Shell-created file changes are not falsely claimed to participate in Workspace CAS. - verify: `TestNativeKubernetesExecution_Scenario3_ShellNamespaceCancelAndCASDisclosure` @@ -107,12 +314,16 @@ A session lease is session-only and its token is not consulted for writes therefore requires independent environment execution fencing. **Acceptance:** -- AC4.1: Caller A cannot discover, attach, execute in, stream from, cancel, or retire caller B's environment; hidden and absent allocations are indistinguishable at the public boundary. +- AC4.1: Ordinary creator client A cannot discover, attach, execute in, stream from, cancel, or retire creator client B's environment; hidden and absent allocations are indistinguishable at the public boundary. The explicit scoped administrative exception in AC4.4 permits only its named administrative RPCs. - verify: `TestNativeKubernetesExecution_Scenario4_CallerEnvironmentIsolation` - AC4.2: Every command is authorized for one immutable environment revision and a distinct transient execution-fence epoch; lease loss or grant revocation prevents new commands and cancels/fences active work without exposing capability credentials to the workload or another caller's output. Same-Pod placement alone is not a security boundary. - verify: `TestADR_0349_ExecutionGrantIsEnvironmentAndFenceScoped` - AC4.3: Timeout, Pod deletion, controller restart, Lease timeout, or network partition alone never authorizes a replacement executor while an old writer may run; unknown fencing state fails closed and requires the reviewed manual/external fencing path. - verify: `TestADR_0349_PartitionCannotAuthorizeTakeoverByTimeout` +- AC4.4: A distinct `administrator:true` client with `administratorFor:[creatorURI]` can administer that creator's environment through all six administrative RPCs, including migration, revocation, and retained deletion, while preserving immutable creator identity and exact owner/revision/epoch/UID/operation checks. A second creator, wrong owner, stale identity, or replay with changed inputs is denied without existence disclosure. Self-admin compatibility remains explicit; no Files/Shell/attach/run/reference or MayAttestOwner authority is gained. + - verify: `TestNativeKubernetesExecution_Scenario4_ScopedAdministratorAllPaths`; `TestKindExecutionProductionScopedAdministrator` (new amendment proofs, PENDING) +- AC4.5: Strict `administratorFor` validation and canonical digest inclusion prevent ambiguous scope. Removal at a higher security generation takes effect on existing connections and receipt replay; an equal-generation scope edit fails closed. Old binaries reject the field and the documented quiesced upgrade preserves authority history. + - verify: `TestNativeKubernetesExecution_Scenario4_AdministratorScopeRotation`; `TestNativeKubernetesExecution_Scenario4_AdministratorScopeValidation` (new amendment proofs, PENDING) ### Scenario 5 — Restart preserves workspace while retirement is deliberate @@ -128,10 +339,12 @@ and remain separate from the session-only lease contract (`engine/port/lease.go: - verify: `TestNativeKubernetesExecution_Scenario5_RetentionAndSafeRetirement` - AC5.4: A pending allocation whose session association might have committed is not garbage-collected merely after TTL; collection waits for conclusive reconciliation that publication is absent. Explicit retirement of a live/shared environment is refused while any live reference exists unless a separately reviewed retire/quiesce policy authorizes it. - verify: `TestADR_0349_PendingAllocationAndLiveReferenceRetirementFailClosed` -- AC5.5: Default chart uninstall never deletes runtime CRs/PVCs; deletion of CRDs while live resources exist is destructive and unsupported, and no automatic hook deletes CRDs. Owner references alone are not a lifecycle proof. - - verify: `TestADR_0349_UninstallRetentionAndCRDDeletionSafety` +- AC5.5: Default chart uninstall retains runtime CRs/PVCs and preserves workload network isolation and security authority history while executors survive. Retained security/capacity ledgers and required configuration outlive the provider Deployment. Deletion of CRDs with live resources is destructive and unsupported; no destructive hook or force cleanup is provided. Owner references or retention annotation strings alone are not lifecycle proof. + - verify: `TestChartHasNoDeletionHook`; `TestKindExecutionProductionHelmRetentionAndReinstall` (behavioral amendment proof, PENDING) - AC5.6: PVC persistence is not documented as node-disaster recovery; kind host-local storage proves restart survival only, and a dead or unobservable node is outside that positive guarantee. - verify: inspection — human review of deployment documentation; `task docs` checks links and structure after authorized tracked changes +- AC5.7: Real Helm install/compatible upgrade/uninstall/reinstall of the same release and namespace preserves exact runtime identity, PVC contents, NetworkPolicy enforcement during provider absence, security high-water/key history, and capacity reservations. Safe adoption validates ownership; missing ledgers with retained allocations, foreign resources, stale authority, arbitrary downgrade, or incompatible profile/schema fail closed without overwrite. CRD upgrades remain manual. + - verify: `TestKindExecutionProductionHelmRetentionAndReinstall`; `TestNativeKubernetesExecution_Scenario5_HelmRetainedAuthorityFailsClosed` (new amendment proofs, PENDING) ### Scenario 6 — Remote catalog and project-source boundaries are explicit @@ -144,27 +357,86 @@ in local composition (`internal/app/project_ingestion.go:20`) and AGENTS discove - verify: `TestNativeKubernetesExecution_Scenario6_NoFalseRemoteProjectIngestion` - AC6.2: Schedules remain excluded. Background commands, isolated Subagent/Parallel/Team delegation, SkillDraft, Git/source operations, and unsupported successor modes are rejected with named capability/precondition errors rather than running locally or silently omitting work; `no-fs` remains unchanged. - verify: `TestNativeKubernetesExecution_Scenario6_UnsupportedPathsNeverFallBackLocal` -- AC6.3: The candidate Clear/Fork contract adds environment exclusion while preserving same-remote-`EnvironmentRef` successor semantics; it makes no claim that current Clear/Fork already serializes shared environments, adds no public Harness field, and never lets a model choose an environment/profile. +- AC6.3: Clear/Fork use environment exclusion and reference reservation/publication while preserving same-remote-`EnvironmentRef` successor semantics and the source on publication failure. Clear/Fork are not workspace cloning; no public Harness field is added and no model chooses an environment/profile. - verify: `TestNativeKubernetesExecution_Scenario6_SuccessorSemantics` ### Scenario 7 — kind proves an offline coding flow; live qualification is separate -The authoritative e2e is a focused mock-only task, following [draft ADR 0349](../adr/0349-native-kubernetes-execution.md). -The existing broad suite creates cluster/images before key capture (`e2e/k8s/suite_test.go:32-80`), -so the focused task must not inherit ambient provider credentials. +The deterministic correctness gate is a focused mock-only task, following [draft ADR 0349](../adr/0349-native-kubernetes-execution.md). +The stack additionally requires the explicitly invoked native-provider OpenRouter coding qualification +recorded in Human decisions; it is completion evidence, not an always-on CI dependency. The broad mecak8s +live suite is not native-provider proof. The focused mock task must not inherit ambient provider credentials. **Acceptance:** -- AC7.1: Candidate `task e2e:k8s:execution` runs on the supported generic toolchain, including CI, with a unique owned kind cluster, a scratch kubeconfig, and explicit `--kubeconfig`/context on every command. It deploys the separate chart plus explicitly connected mecak8s and uses only the deterministic mock provider. +- AC7.1: `task e2e:k8s:execution` and the enforcing-CNI `task e2e:k8s:execution:production` run on the supported generic toolchain, including CI, with a unique owned kind cluster, a scratch kubeconfig, and explicit `--kubeconfig`/context on every command. They deploy the separate chart plus explicitly connected mecak8s and use only the deterministic mock provider. The final production-profile run uses Calico and exercises negative network isolation, rotation/revocation, replica/lifecycle recovery, holder loss, quota recovery, pending delete, Clear/Fork, migration, sanitized artifacts, and the amendment regressions. - verify: `TestNativeKubernetesExecution_Scenario7_FocusedKindMockFlow` - AC7.2: The mock flow seeds a small Go fixture through file tools, asks the harness to implement a function/tests, runs `go test` through bound Shell, and verifies the persisted artifact and restart/reattach behavior. - verify: `TestNativeKubernetesExecution_Scenario7_MockCodingFlow` - AC7.3: Failure artifacts are bounded and sanitized; cleanup targets only the recorded owned cluster. Destructive live cleanup requires confirmation and never mutates ambient kubeconfig/context. - verify: `TestNativeKubernetesExecution_Scenario7_OwnershipAndSanitizedArtifacts` -- AC7.4: Candidate `task e2e:k8s:execution:live` is a separate explicit, bounded real-model smoke after mock/kind qualification. It runs only against verified nonsecret endpoint, protocol, and model configuration in an authenticated, authorized operator-controlled cluster; it does not claim a hard dollar cap from token/run limits. - - verify: `TestNativeKubernetesExecution_Scenario7_LiveQualificationIsExplicit` +- AC7.4: `task e2e:k8s:execution:live` is a separate explicit native-provider OpenRouter coding qualification, required to complete this stack after final deterministic qualification. It proves model-driven file/tool execution, independently verified persisted workspace artifacts, and a successful bound `go test` command. It uses verified nonsecret endpoint/protocol/model configuration in an authenticated operator-controlled cluster, then restores mock configuration and deletes only the run-owned Secret by recorded UID. It is not an always-on live CI dependency and claims no hard dollar cap from token/run limits. + - verify: `TestKindExecutionLiveQualification`; inspection — record successful native-provider live evidence for the final implementation candidate (PENDING, not satisfied by generic mecak8s live CI) - AC7.5: A trusted runtime-only loader may consume the approved credential directly from its protected file channel and create a narrowly scoped HARNESS-only Secret through a protected channel before child processes or tools can inherit it. The agent never accesses the file or Secret; failures are redacted, and the credential never appears in model/tool output, argv, Helm values, disk manifests, git, logs, artifacts, or the execution workload. Mock tests do not access it. - verify: `TestNativeKubernetesExecution_Scenario7_LiveSecretAndSpendBoundary` +## Implementation proof map + +Source/test locations in this table refer to implementation commit +`deaf1c3d1dfe7ea92afc8fe826f1bc613080f219`, not files supplied by this plan branch. +Existing `TestNativeKubernetesExecution_*`/`TestADR_0349_*` verify labels that do not yet resolve are +future acceptance obligations, not invented test results. Before completion, replace those labels with +exact executable proofs that cover the full AC, or implement the missing proof, and pass strict trace. +A row naming a unit/fake-client test establishes only that boundary; it does not prove the entire AC. + +**Observed qualification:** the [production Kind+Calico job](https://github.com/stacklok/mecatl/actions/runs/35589357385/job/106299991747) +completed **SUCCESS** at the exact implementation reference. This includes network isolation, +rotation/revocation, replicas/lifecycle, holder loss, quota saturation, pending-delete recovery, +Clear/Fork, compatible prototype migration, and artifact isolation. Historical local keyring/toolchain +failures are not active blockers for that run. This is baseline evidence, not final-candidate evidence +for the new scoped-admin, continuing-retry, or Helm uninstall/reinstall requirements. + +Draft CI skipped race lanes; non-race success is not a race result. The generic live compaction lane +failure is addressed by [merged PR #1728](https://github.com/stacklok/mecatl/pull/1728) on main, not a +native-provider defect or proof that its fix is present in this draft. Generic mecak8s live success is +also not AC7.4 evidence. Native-provider live qualification and all new amendment regressions remain +**PENDING**. No final live execution or human contract/panel approval is asserted here. + +| AC | Existing implementation proof at the reference commit; remaining evidence | +|---|---| +| AC1.1 | `internal/app/remote_execution_test.go`: `TestRemoteDeploymentNoFSUsesLocalAttenuationWithoutProviderCall`; full disabled-effects obligation remains. | +| AC1.2 | `deploy/helm/mecatl-execution/chart_test.go`: `TestChartRetainsCRDAndDoesNotGrantSecretAPI`; production job separately deploys provider and client. | +| AC1.3 | `internal/adapter/executioncontroller/handler_test.go`: `TestHandlerValidateIsReadOnlyAndEnsureIdempotent`; `controller_test.go`: `TestInitializeRefusesMissingRuntimeClassBeforeCreatingPods`. | +| AC1.4 | `internal/adapter/executioncontroller/controller_test.go`: `TestStartupDoesNotFenceLivePeerOperation`; this is peer-restart evidence, not a complete disabled-client lifecycle proof. | +| AC2.1 | `internal/adapter/executioncontroller/handler_test.go`: `TestHandlerValidateIsReadOnlyAndEnsureIdempotent`; `e2e/k8s_execution/qualification_test.go`: `TestKindExecutionQualification`. | +| AC2.2 | `internal/adapter/executioncontroller/store_test.go`: `TestStoreEnsureUsesStableLookupAndRejectsFingerprintDrift`; `controller_test.go`: `TestReconcileRefusesForeignExistingPVCWithoutPersistingUID`. | +| AC2.3 | `internal/adapter/executioncontroller/profiles_test.go`: `TestLoadProfilesStrictAndDigestPinned`; production `TestKindExecutionProductionQuotaSaturation`. | +| AC2.4 | **PENDING** continuing retries after prolonged PVC/Pod failure and quota restoration with no unrelated event; current quota test alone is insufficient. | +| AC3.1 | `internal/adapter/executionclient/client_test.go`: `TestProviderThroughRealGRPCSignedHandlerRefreshesAndReattachesExactly`; Kind coding flow succeeds, but full per-tool conformance matrix remains to be traced/qualified. | +| AC3.2 | `internal/adapter/executionclient/service_integration_test.go`: `TestServiceUsesRealMTLSProviderStoreAndReleasesOnlyAfterDrain`; retain explicit file-version and Copy/Move/Remove conformance proofs. | +| AC3.3 | `internal/adapter/executioncontroller/security_test.go`: `TestSecurityManagerGuardsEveryRPCOnExistingConnection`; `handler_test.go`: `TestHandlerRequiresAllowlistedCanonicalURISAN`; production `TestKindExecutionProductionSecurityRotation`. | +| AC3.4 | `internal/adapter/executioncontroller/store_test.go`: `TestCancelledStoreOperationStaysActiveUntilBackendStopsThenFences`; production `TestKindExecutionProductionHolderLossFencesActiveOperation`. | +| AC4.1 | `internal/adapter/executioncontroller/store_lifecycle_test.go`: `TestClientScopedIntentListReturnsAttestedOwnerOnlyToOwningClient`; Kind isolation and pending-delete tests. Scoped-admin exception needs new positive/negative proofs. | +| AC4.2 | `internal/adapter/executioncontroller/store_revoke_test.go`: `TestRevokeEnvironmentFencesOldClaimWithoutChangingExecutionEpoch`; `store_concurrency_test.go`: `TestAcquireRunAndRevokeUseResourceVersionCAS`; production security rotation. | +| AC4.3 | `internal/adapter/executioncontroller/production_lifecycle_test.go`: `TestExpiredOperationLeaseFencesWithoutClearingIdentity`, `TestRecoverMissingPodRemainsFenceUnknown`, `TestRecoveredTerminalProofCanStartExactReplacement`; production holder loss. | +| AC4.4–AC4.5 | **PENDING** all-admin-path scope matrix, data-plane non-escalation, immutable creator/owner and replay checks, strict decoder/digest tests, scope removal on open connections, and Kind distinct-admin qualification. | +| AC5.1 | `internal/adapter/executionclient/client_test.go`: exact reattachment test above; production `TestKindExecutionProductionReplicaLifecycle` and security rotation. | +| AC5.2 | `internal/adapter/executioncontroller/production_lifecycle_test.go`: `TestReplacementPersistsTerminalProofBeforeRemovingPodFinalizer`; production replica lifecycle. | +| AC5.3 | `internal/adapter/executioncontroller/production_lifecycle_test.go`: `TestReplacementQuiescesAndPendingReferenceBlocksRetirement`. | +| AC5.4 | `internal/adapter/executionclient/reference_ambiguity_test.go`: `TestAmbiguousCommitIsRetainedAndNeverAbortedByCleanup`; production `TestKindExecutionProductionPendingDeleteOutageRecovery`. | +| AC5.5, AC5.7 | `deploy/helm/mecatl-execution/chart_test.go`: `TestChartHasNoDeletionHook` is static partial coverage only. **PENDING** actual Helm lifecycle, retained network isolation, security/capacity-ledger survival, ownership adoption, and missing-history rejection. | +| AC5.6 | Deployment-documentation inspection and docs gates required on the final implementation; Kind is restart proof only. | +| AC6.1–AC6.2 | `internal/app/remote_execution_test.go`: `TestRemoteExecutionRealFactoryCarriesPostureAndAttenuatedCatalog`, `TestRemoteDeploymentNoFSUsesLocalAttenuationWithoutProviderCall`. | +| AC6.3 | `internal/adapter/executioncontroller/store_lifecycle_test.go`: `TestReferenceTransactionsRetainUnknownAndNeverChangeSource`; production `TestKindExecutionProductionClearForkLifecycle`. | +| AC7.1 | `.github/workflows/k8s-e2e.yml` runs `task e2e:k8s:execution:production`; the reference job succeeded. `e2e/k8s_execution/aa_production_migration_test.go`: `TestKindExecutionProductionCompatiblePrototypeMigration`. Final amended-candidate run **PENDING**. | +| AC7.2 | `e2e/k8s_execution/qualification_test.go`: `TestKindExecutionQualification`, run by the successful production profile. | +| AC7.3 | Production `TestKindExecutionProductionFailureArtifactBoundary`; workflow bounded uploads and ownership-scoped cleanup. | +| AC7.4 | `e2e/k8s_execution/live_qualification_test.go`: `TestKindExecutionLiveQualification` exists; successful final native-provider OpenRouter evidence **PENDING**. | +| AC7.5 | `e2e/k8s_execution/fixture/credentialloader/main_test.go`: `TestLoadCredentialAndRedaction`, `TestDeletePinsReceiptUIDAndLeavesReplacement`; these offline tests do not substitute for live execution. | + +Production test names without another path above reside in the implementation's +`e2e/k8s_execution/production_qualification_test.go`. Do not import implementation files into this +plan worktree to make citations or trace appear complete. + ## Out of scope | Item | Defer-to | Decision | @@ -176,21 +448,22 @@ so the focused task must not inherit ambient provider credentials. | Schedules and remote delegation/fork support | Separate reviewed expansion | First slice rejects unsupported paths without local fallback. | | Automated force takeover under uncertain fencing | Not permitted | Unknown old executor state is fail-closed/manual until a reviewed external fencing proof exists. | | Node-disaster PVC recovery and production multi-zone storage | Production storage design | kind host-local PVC is only a restart-survival proof. | -| Provider credential discovery, arbitrary endpoint forwarding, or live model inference beyond the approved runtime-loader boundary | Separate explicit qualification | The approved credential is not read in planning/mock tests; verified nonsecret endpoint/protocol/model configuration governs any optional smoke. | +| Provider credential discovery, arbitrary endpoint forwarding, or live inference beyond the approved native qualification | Separate explicit authorization | This stack requires native OpenRouter coding evidence within the protected runtime-loader boundary; planning/mock tests never access credentials. | | Depending on or copying PR #580's transport | Independent work | Share behavioral conformance where useful; do not inherit unqualified cancellation/auth protocol. | ## Definition of done -1. The outstanding technical specifications — private API, allocation/ownership lifecycle, CRD/config schema and compatibility, and grant/revocation/termination/fencing proof — are authored and reviewed in this draft Plan / Interface PR before the affected implementation is dispatched. -2. The authorized implementation passes applicable `task lint`, `task test`, `task docs`, `task api:check`, `task site:build`, and `go run ./cmd/mecademo` gates. -3. `task ac-trace-strict` resolves every named proof when the plan becomes `landed`. -4. The focused mock-only kind task passes on the supported toolchain with a unique owned cluster; optional live qualification is reported separately and never substitutes for it. -5. The draft implementation PR targets `plan/native-kubernetes-execution`, links the draft Plan / Interface PR and exact working baseline, reports interface conformance, and completes panel review before becoming merge-ready. Neither PR is merged automatically. +1. Human API/schema/security review approves the authored interface and amendment requirements, and the release maturity designation is explicit. Both PRs remain draft with no approved baseline until human contract approval; authorized pre-merge development does not waive that gate. +2. The final implementation candidate passes `task lint`, full offline `task test` including race and engine-standalone coverage, `task build`, `task api:check`, `task docs`, `task site:build`, and `go run ./cmd/mecademo`. A skipped draft race lane is not a pass. Generated contracts/references must be fresh; no arbitrary prose-pinning tests. +3. `task ac-trace-strict` resolves every final `verify:` proof, including new AC2.4, AC4.4–AC4.5, and AC5.7. Replace provisional labels with executable proofs covering the whole AC; the evidence map alone is not strict trace completion. +4. Fresh deterministic Kind+Calico production qualification passes at the final candidate, including the expanded scope/retry/Helm lifecycle matrix. Then record successful stack-specific native-provider OpenRouter coding qualification under AC7.4–AC7.5. Neither historical deterministic success nor generic mecak8s live CI substitutes for those final results. +5. Complete independent Spec/Standards/Test adequacy/Domain panel review and resolve merge-blocking findings. Update the owning `user-docs/building/deployment/mecak8s.md` and implementation's `user-docs/features/execution-environments.md`, plus resource/rehydration inventories, in the same implementation stack; keep this plan branch documentation-only. +6. The existing implementation PR targets `plan/native-kubernetes-execution`, links plan PR #1579 and its exact working commit, reports conformance/amendments and final evidence. Human alone approves and merges; no new PR, automatic merge, or unsupported landed claim. ## Deferred decisions and known risks -- Exact wire fields/errors, lifecycle signatures, CRD schema, and auth/fencing protocol are material open decisions in `## Human decisions`; implementation must not fill them in opportunistically. -- A PVC can preserve files while a stale command still mutates them. Session leasing does not fence command execution, and a timeout is not proof that a partitioned executor stopped. -- Default kind does not enforce NetworkPolicy. A negative isolation test needs a network-policy-capable CNI and must state what was actually enforced. -- Controller/executor credentials, signing material, and provider secrets must never enter arbitrary-shell workloads. -- The future controller, informer/cache, execution client, connection pool, grant verifier/rotation state, and stream/cancellation registry all outlive one tool call and must be inventoried in `docs/adr/0027-cloud-native.md` when tracked design work is authorized. +- Exact specifications are authored, but human API/schema/security approval and the release maturity designation remain open. Draft development is authorized; promotion/merge is not. Do not turn `v1alpha1` into an unreviewed product “alpha” decision. +- A PVC can preserve files while a stale command still mutates them. Session leasing does not fence command execution; missing old-executor evidence remains a fail-closed manual recovery boundary. +- Only an enforcing CNI supplies negative NetworkPolicy proof. Retaining storage while uninstall removes isolation or authority history is unsafe; the Helm lifecycle amendment is a completion requirement, not deferred polish. +- Every outlives-a-call resource, including retained security/capacity ledgers, reload state, informer/cache, connection pools, claims, and lifecycle/reference reconciliation, needs an accurate resource/rehydration inventory in `docs/adr/0027-cloud-native.md` on the implementation branch. +- The new scope, retry, and Helm behavioral regressions and final native live evidence are pending. Existing source/proof locations absent on the plan branch must stay explicitly implementation-scoped; plan docs gates must not be “fixed” by adding implementation code. diff --git a/docs/adr/0349-native-kubernetes-execution.md b/docs/adr/0349-native-kubernetes-execution.md index a0ea302551..7f5f032265 100644 --- a/docs/adr/0349-native-kubernetes-execution.md +++ b/docs/adr/0349-native-kubernetes-execution.md @@ -1,7 +1,7 @@ # ADR 0349 — Optional native Kubernetes execution environments - Status: Draft -- Date: 2026-09-15 +- Date: 2026-09-15; draft amended 2026-09-21 - Scope: independently deployed execution environment provider service; mecak8s is an optional client; controller/executor, logical environment, PVC, Pod, authorization, fencing, and kind qualification boundaries - Supersedes: none @@ -21,12 +21,19 @@ Likewise, startup placement validation currently reaches `Bind` (`docs/design/IMPLEMENTATION-NOTES.md:6549-6560`), but using an allocating operation as preflight would leak PVCs and Pods. -The human has approved the direction and defaults: a purpose-built execution environment provider service in this monorepo, separate from mecak8s, with its own binaries, images, chart lifecycle, optional client integration, and mock/kind-first qualification. The companion -[acceptance plan](../acceptance/native-kubernetes-execution.md) and this ADR remain draft because the -exact private protocol, lifecycle, CRD/configuration schema, and grant/fencing specifications still -need authoring and review. +The human authorized the direction and defaults: a purpose-built execution environment provider in this +monorepo, separate from mecak8s, with its own binaries, images, chart lifecycle, optional client integration, +and deterministic kind qualification followed by native-provider OpenRouter coding evidence. The target +is an operator-controlled first production slice with explicit limits; release maturity remains undecided. -## Approved direction and defaults; technical specifications remain draft +The existing plan PR #1579 and implementation PR #1614 form a human-authorized **draft** stack. The +[acceptance plan](../acceptance/native-kubernetes-execution.md) now authors the exact proposed contract, +reconciled against implementation `deaf1c3d1dfe7ea92afc8fe826f1bc613080f219`. API, schema, and security +review approvals remain unchecked; neither authored specs nor draft code is a merged approved baseline. +This amendment also authorizes development of scoped administration, continuing provisioning retries, +and retained-resource Helm lifecycle safeguards on those existing PRs. Human alone merges. + +## Proposed decision within the authorized draft stack This ADR records the approved direction: a separate execution environment provider in this monorepo; it does **not supersede ADR 0048**. ADR 0048 governs storage-free mecak8s. Mecak8s remains a client of @@ -34,23 +41,30 @@ the provider, keeps session state in its existing stores, and neither mounts pro owns executor/controller lifecycle. The provider owns its CRDs, Pods, PVCs, authorization, reconciliation, binaries, images, and chart lifecycle. Its API is not tied to a mecak8s process or release; other authorized Mecatl compositions can use an adapter without embedding the controller. -Provider allocation handles remain separate from harness session IDs. Exact API/version compatibility -still requires specification and review. +Provider allocation handles remain separate from harness session IDs. The +[exact interface contract](../acceptance/native-kubernetes-execution.md#interface-contract) is authored +for review in the acceptance plan; this ADR records the durable boundaries rather than duplicating the spec. ### 1. Own a narrow controller/executor boundary Ship a dedicated controller and executor integration rather than Agent Sandbox. The controller -reconciles a logical execution environment consisting of a future `ExecutionEnvironment` custom -resource, persistent workspace PVC, and replaceable executor Pod. Durable identity is the logical -environment plus immutable `EnvironmentRef.Revision`, which persists unchanged across run ownership -turnover; it is not a fence epoch or restart counter. Pod and PVC names and UIDs are observed references -used to detect replacement/adoption errors. - -The future CRD must express operator-selected, digest-pinned profile/image/storage policy and distinguish -processed spec generation from the current executor Pod UID. Only operator-configured profiles may choose -image, storage, or source policy. Public clients and models cannot submit arbitrary Pod specs, images, -paths, URLs, credentials, or Kubernetes names. Its exact group/version/kind, schema, conditions, -mutability rules, and source metadata remain to be authored and reviewed. +reconciles a namespaced `execution.mecatl.dev/v1alpha1` `ExecutionEnvironment`, a persistent workspace +PVC, and a replaceable executor Pod. Spec and status use explicit schema version 2. The Kubernetes +API name does not settle the release maturity label. Durable identity is the logical environment plus +immutable `EnvironmentRef.Revision`; run epoch, grant generation, Pod UID, and PVC UID have distinct roles. +`observedGeneration` records processed spec generation, not executor identity. + +Only operator-configured, digest-pinned profiles select image, storage, resource/operation bounds, and +RuntimeClass. Public clients/models cannot submit arbitrary Pod specs, images, paths, URLs, credentials, +or Kubernetes names. Creator `clientHash`, owner identity, allocation fingerprint, and revision remain +immutable. The [schema and transaction contract](../acceptance/native-kubernetes-execution.md#reference-transaction-schema-v2-and-fencing) +records exact fields, receipts, conditions, and migration behavior. + +Transient PVC/Pod create errors and quota saturation must recover through continuing rate-limited +reconciliation while the CR exists. Quota restoration alone must be sufficient; no unrelated reference +change, restart, or manual reconcile is required. A finite retry count or successful condition write +must not strand provisioning. Wrong ownership or missing/mismatched authoritative UIDs still fail +closed without replacement, regardless of retry scheduling. ### 2. Preserve the core environment and tool contracts @@ -61,10 +75,14 @@ Shell writes retain their documented ability to bypass Workspace CAS (`engine/tool/tool.go:353-424`); the product must disclose that honestly rather than promising a stronger remote filesystem transaction. -Introduce a side-effect-free configuration/profile validation operation distinct from idempotent -allocation. Candidate private operations include profile validation, ensure, exact attach, command -start/stream/status/cancel, and retirement. Their precise Go signatures and private wire protocol -are deliberately undecided and block promotion of this ADR. +The private typed `mecatl.execution.v1.ExecutionProviderService` uses mTLS, with no REST fallback. +`ValidateProfile` is side-effect-free; Ensure reserves a reference, exact Attach reattaches, and an +independent Acquire/Renew/ReleaseRun claim gates operations. Reference and administrative lifecycle +RPCs carry exact identities and replay preconditions. `Files` transports bounded bytes and opaque +versions. `StartCommand` is foreground unary; authorized detached CommandStatus/CancelCommand calls +return Unimplemented. Do not advertise a command stream or background service absent from this slice. +The [RPC and host contract](../acceptance/native-kubernetes-execution.md#private-rpc-and-host-contract) +records exact messages, error classifications, and host-only placement/reference/run interfaces. ### 3. Make installation and connection explicitly optional @@ -77,8 +95,13 @@ An explicitly selected remote environment fails clearly at validation if its con unavailable, and fails closed at runtime if the provider is lost or its exact immutable revision cannot reattach. It never falls back to the local project root. Existing local and `no-fs` behavior is unchanged. -Exact packaging names, values/flag precedence, mTLS Secret layout, storage class/size/quota configuration, -and compatibility rules remain to be authored and reviewed. +The separate chart is `deploy/helm/mecatl-execution`. Mecak8s enables the client through +`--execution-enabled`, endpoint/profile flags, and three mounted TLS-file flags; OIDC ownership is +required. Profiles and the versioned security manifest are strict operator configuration, not model +inputs. The [configuration contract](../acceptance/native-kubernetes-execution.md#provider-configuration-rotation-and-scoped-administration) +records exact names and bounds. Old binaries reject the scoped-admin field: quiesce and upgrade provider +replicas before introducing it. CRD upgrade remains manual; there is no arbitrary downgrade or +mixed-version compatibility claim for this amendment. ### 4. Separate execution fencing from session leasing @@ -94,9 +117,28 @@ detached descendants or fail closed. Pod deletion, Lease timeout, a timeout, con partition is not evidence that an old command stopped and cannot alone authorize takeover. If the old executor cannot be positively terminated or externally fenced, replacement fails closed. -The exact grant trust, issuer/verifier, claims, audience, rotation/revocation, replay protection, command -cancellation acknowledgement, process-group termination, and takeover/fencing proof remain to be authored -and security-reviewed. No existing SPIFFE issuer is assumed. +Ed25519 grants bind issuer/audience, creator, owner, environment revision, binding/run/claim, operation, +epoch, grant generation, and validity window. ResourceVersion CAS gates run claims and the separate +replica-owned operation lease. Expiry retains uncertain operation identity; it does not authorize a +second writer. Exact terminal Pod/PVC proof is persisted before finalizer removal and replacement; +RecoverEnvironment has no caller-supplied force-fencing assertion. The external-fencing runbook remains +necessary for unobservable nodes. + +Rotation publishes a validated security snapshot backed by a durable generation/digest/key-fingerprint +high-water ledger. Every RPC rechecks current authority, including requests over established connections. +CA/key/client removal, scope removal, rollback, or equal-generation policy drift fail closed. Neither +security generation nor per-environment grant revocation is proof of termination. No existing SPIFFE +issuer is assumed, and no credential crosses the provider-to-workload stdin boundary. + +Scoped administration uses one narrow extension to the existing client policy: +`administrator:true` plus optional `administratorFor:[canonicalCreatorURI]`. Absent/empty scope preserves +self-admin compatibility, not namespace-wide privilege. A distinct operations URI may administer only +the listed creators' allocations. The shared decision derives creator identity from the loaded +immutable `clientHash` and keeps exact owner, revision, epoch, UID, schema/generation, and operation replay +checks. It applies to all six administrative RPCs, including migration, revocation, and retained deletion. +A scope never grants attach, Files, Shell, run/reference access, or MayAttestOwner. No public Harness, +engine, or private protobuf widening is needed. The acceptance contract specifies validation and digest +inclusion; authoring this proposal is not the outstanding human security approval. ### 5. Start with a persistent blank workspace and an attenuated catalogue @@ -107,10 +149,11 @@ are deferred. Project AGENTS/rules/skills remain disabled until a separate root/ project admission is local-root based (`internal/app/project_ingestion.go:20`). Schedules remain excluded. SkillDraft and unsupported remote Subagent/Parallel/Team paths return explicit -unsupported/precondition results and never execute locally. Existing Clear/Fork is not environment- -serialized today; the future lifecycle contract must preserve same-`EnvironmentRef` successor semantics -while adding appropriate environment exclusion. Exact capability/error and lifecycle mechanics remain -technical specifications, not new Human direction decisions. +unsupported/precondition results and never execute locally. Clear/Fork reserve and publish references +under environment exclusion while preserving the same remote `EnvironmentRef`; neither clones the +workspace. Clear creates empty history and Fork copies valid history. Publication failure preserves the +source; ambiguous reference outcomes stay durable for reconciliation. No public placement selector or +model-chosen profile is added. ### 6. Retain committed workspaces by default @@ -123,33 +166,49 @@ have committed cannot be collected merely after TTL: publication absence must fi reconciled. Explicit retirement of a live/shared environment is refused while any live reference exists unless a later reviewed retire/quiesce policy specifies otherwise. -Default chart uninstall never deletes runtime CRs/PVCs. Deleting CRDs while live resources exist is -destructive and unsupported; no automatic hook deletes CRDs. Owner references alone do not prove safe -lifecycle ownership. PVC persistence does not claim node-disaster recovery; kind host-local storage -proves only process/Pod restart survival, not recovery from a dead or unobservable node. - -Storage class, size, and quotas are operator-configured. Their exact configuration schema, sharing, -committed-state marker, retention mechanics, and retirement authorization remain to be authored and -reviewed. - -### 7. Qualify offline before optional live inference - -Add a focused mock-only kind task separate from the broad e2e suite. The task must run on the -supported generic toolchain, including CI. -It uses a unique owned cluster, a scratch kubeconfig, explicit kubeconfig/context on every command, -sanitized bounded artifacts, and confirmation before destructive live cleanup. Default kind is not -accepted as negative NetworkPolicy evidence; isolation testing needs a network-policy-capable CNI. - -A later real-model smoke is separate and bounded, after mock/kind qualification, for authenticated and -authorized use in an operator-controlled cluster. Endpoint, protocol, and model selection require verified -nonsecret configuration; credential usability does not authorize arbitrary endpoint forwarding. The approved -credential is not accessed by planning or mock tests. A trusted runtime-only loader may consume its protected -file directly through a protected channel to create a narrowly scoped HARNESS-only Secret before child -processes or tools can inherit it; the agent never accesses the file or Secret. Failures are redacted, and -the credential never enters model/tool output, -argv, Helm values, disk manifests, git, logs, artifacts, or the execution workload. Token/run limits do not -claim a hard dollar cap. One bounded coding prompt is qualitative; deterministic mock behavior remains the -correctness gate. +Default chart uninstall retains runtime CRs/PVCs. While executors survive, workload isolation, +security high-water/key history, and the profile-capacity ledger must survive too. Their lifetime follows +retained allocations, not the provider Deployment. A keep annotation or no-deletion-hook assertion is not +behavioral proof. Uninstall must not silently remove NetworkPolicies or reset authority on reinstall. + +The supported reinstall is the same release and namespace, with explicit validation/reuse of that +release's retained resources and unchanged allocation/profile identity. Preserve ledger contents and +resource ownership; reject foreign/ambiguous resources and missing authority history while allocations +survive. Compatible upgrades also preserve state. Fresh install may initialize empty ledgers only in +the absence of retained allocations. No destructive hook, forced adoption/cleanup, authority-generation +reset, or arbitrary rollback is supported. Manual CRD upgrade precedes explicit compatible schema-v2 +migration; CRD/namespace deletion with live resources is destructive and unsupported. + +The [Helm lifecycle contract](../acceptance/native-kubernetes-execution.md#helm-lifecycle-amendment) +requires actual install/upgrade/uninstall/reinstall qualification: persistent data, exact reattachment, +negative enforcing-CNI probes during provider absence, rejection of stale authority after restart, and +capacity accounting. Add retained ledgers/isolation/configuration to the resource and rehydration +inventories in `docs/adr/0027-cloud-native.md` during implementation. PVC persistence does not claim +node-disaster recovery; kind host-local storage proves restart survival only. + +### 7. Qualify deterministically, then prove native-provider live coding + +The focused mock-only kind tasks run on the supported generic toolchain, including CI, with a unique +owned cluster, scratch kubeconfig, explicit context on every command, bounded sanitized artifacts, and +ownership-scoped cleanup. Default kind does not prove NetworkPolicy enforcement; the production profile +uses Calico. The acceptance plan records a successful production job at `deaf1c3d1`, distinct from the +pending final candidate and new scope/retry/Helm regression qualification. Historical local keyring +failures no longer block that baseline. Skipped draft race lanes remain unproven. + +The checked human live decision requires the separate `task e2e:k8s:execution:live` OpenRouter coding +qualification **for this stack**, after deterministic qualification. It is not an always-on live CI +dependency. Prove model-driven tools, independently verified persistent artifacts, and a successful +bound test command; generic mecak8s live CI is not native-provider evidence. No final native live run is +claimed. The generic compaction live-lane correction in merged #1728 is separate from this capability. + +Use verified nonsecret endpoint/protocol/model configuration in an authenticated operator-controlled +cluster. A trusted runtime-only loader consumes the approved protected file channel into a narrowly +scoped HARNESS-only Secret before tools/child processes can inherit it. Planning and mock tests never +access it. Neither agent nor tool reads the credential file or Secret; failures are redacted, and no +credential enters model/tool output, argv, Helm values, disk manifests, git, logs, artifacts, or the +execution workload. Restore mock configuration and delete only the run-owned Secret by recorded UID. +Token/run limits do not establish a hard dollar cap. Live coding evidence supplements, never replaces, +the deterministic correctness gate. ## Consequences @@ -159,15 +218,17 @@ RBAC, informers, clients, and resources. Logical identity survives Pod replaceme the complete coding path offline. **Costs and risks.** Mecatl would own a controller, CRD, private protocol, PVC lifecycle, command -streaming/cancellation, and a security-critical fencing system. A persistent volume does not stop a +execution/cancellation, and a security-critical fencing system. A persistent volume does not stop a partitioned writer. Arbitrary Shell makes the executor a hostile-workload boundary. Network policy, resource quotas, output caps, non-root execution, service-account-token suppression, and credential separation are required but not sufficient. -**Draft boundary.** The direction and defaults above are approved, but this ADR is not implementation-ready. -The next planning work authors and reviews the exact private wire fields/errors, allocation transaction and -ownership/reference lifecycle, CRD/configuration schema and compatibility, and grant trust/revocation/ -termination/fencing proof. No implementation may invent those contracts. +**Draft boundary.** The exact proposed interfaces are authored in the acceptance plan; human API/schema/ +security approval and release maturity remain unresolved. Authorized pre-merge development may proceed +on the existing draft stack, but no review, approval, landed status, or merge is inferred. Completion +requires strict AC trace, full offline/race/build/API/docs/site/demo gates, independent panel review, +final deterministic qualification, and stack-specific native live evidence. New scope, retry, and Helm +behavioral proofs remain pending. Human alone merges. ## See also From 2a123e0911574ab718dab2405c5a0e3169f7b89b Mon Sep 17 00:00:00 2001 From: Juan Antonio Osorio Date: Mon, 21 Sep 2026 20:52:52 +0300 Subject: [PATCH 05/16] fix(tooling): drain acceptance checker category pipelines Preserve first-match extraction without closing the pipe early under pipefail. Cover oversized Split and Combined contracts with executable regression fixtures. Co-Authored-By: OpenAI --- .../scripts/check-acceptance-plan-test.sh | 14 ++++++++++++++ .../scripts/check-acceptance-plan.sh | 4 ++-- 2 files changed, 16 insertions(+), 2 deletions(-) diff --git a/.claude/skills/to-acceptance-plan/scripts/check-acceptance-plan-test.sh b/.claude/skills/to-acceptance-plan/scripts/check-acceptance-plan-test.sh index d031018428..9f11a66ba9 100755 --- a/.claude/skills/to-acceptance-plan/scripts/check-acceptance-plan-test.sh +++ b/.claude/skills/to-acceptance-plan/scripts/check-acceptance-plan-test.sh @@ -100,6 +100,20 @@ sed_in_place 's/\*\*Expected tasks:\*\* deferred to orchestration/\*\*Expected t sed_in_place 's||**Combined rationale:** The fixture is one indivisible documentation check, so separate plan review adds no value.|' "$combined_valid" bash "$checker" "$combined_valid" >/dev/null +# Exceed pipe capacity after the first category: an early-exiting reader must +# not turn a valid Split or Combined contract into a SIGPIPE failure. +for source in "$valid" "$combined_valid"; do + large="$root/acceptance/large-${source##*/}" + awk ' + { print } + /^- \*\*gRPC \/ protobuf:\*\*/ { + for (i = 0; i < 16384; i++) + print " Additional interface detail keeps the contract larger than a pipe buffer." + } + ' "$source" >"$large" + bash "$checker" "$large" >/dev/null +done + for case_name in missing-contract invalid-contract missing-category bare-none placeholder bad-delivery bad-status proposed-unchecked missing-human placeholder-human checked-without-decision; do cp "$valid" "$root/acceptance/$case_name.md" done diff --git a/.claude/skills/to-acceptance-plan/scripts/check-acceptance-plan.sh b/.claude/skills/to-acceptance-plan/scripts/check-acceptance-plan.sh index 4c82f608ab..a7b21442ad 100755 --- a/.claude/skills/to-acceptance-plan/scripts/check-acceptance-plan.sh +++ b/.claude/skills/to-acceptance-plan/scripts/check-acceptance-plan.sh @@ -264,7 +264,7 @@ for label in "${category_labels[@]}"; do continue fi - category_line=$(printf '%s\n' "$interface_block" | awk -v label="$label" 'index($0, label) == 1 && substr($0, length(label) + 1, 1) == " " { print; exit }') + category_line=$(printf '%s\n' "$interface_block" | awk -v label="$label" '!found && index($0, label) == 1 && substr($0, length(label) + 1, 1) == " " { print; found = 1 }') category_content=${category_line#"$label"} category_content=${category_content#" "} if [[ -z "$category_content" ]]; then @@ -292,7 +292,7 @@ if [[ "$combined" -eq 1 ]]; then '- **Security / authority:**' ) for label in "${combined_labels[@]}"; do - category_line=$(printf '%s\n' "$interface_block" | awk -v label="$label" 'index($0, label) == 1 && substr($0, length(label) + 1, 1) == " " { print; exit }') + category_line=$(printf '%s\n' "$interface_block" | awk -v label="$label" '!found && index($0, label) == 1 && substr($0, length(label) + 1, 1) == " " { print; found = 1 }') category_content=${category_line#"$label"} category_content=${category_content#" "} if ! [[ "$category_content" =~ ^None[[:space:]]+(-|–|—)[[:space:]]+[^[:space:]].* ]]; then From 1efd98378c34a347d2d2e5d4130c182754883c14 Mon Sep 17 00:00:00 2001 From: Juan Antonio Osorio Date: Mon, 21 Sep 2026 20:52:52 +0300 Subject: [PATCH 06/16] docs: resolve native Kubernetes draft ADR number collision Main now owns ADR 0349 for SDK decoding. Renumber the native Kubernetes draft and its references to 0350 without changing the authored contract. Co-Authored-By: OpenAI --- .../acceptance/native-kubernetes-execution.md | 32 +++++++++---------- ...md => 0350-native-kubernetes-execution.md} | 2 +- docs/adr/README.md | 2 +- 3 files changed, 18 insertions(+), 18 deletions(-) rename docs/adr/{0349-native-kubernetes-execution.md => 0350-native-kubernetes-execution.md} (99%) diff --git a/docs/acceptance/native-kubernetes-execution.md b/docs/acceptance/native-kubernetes-execution.md index 514a52900e..0c5b245af9 100644 --- a/docs/acceptance/native-kubernetes-execution.md +++ b/docs/acceptance/native-kubernetes-execution.md @@ -2,7 +2,7 @@ **Contract:** human-reviewed/v2 **Work classification:** Architectural — introduces a controller/executor trust boundary, durable environment ownership, Kubernetes resources, and an optional mecak8s deployment integration. -**Decision record:** [ADR 0349](../adr/0349-native-kubernetes-execution.md) +**Decision record:** [ADR 0350](../adr/0350-native-kubernetes-execution.md) **Phase:** native Kubernetes execution first slice **Status:** draft, amended 2026-09-21. Exact proposed contracts are authored below; API, schema, and security review approvals remain outstanding. **Delivery:** Split. Human-authorized draft stack: the Plan / Interface PR targets `main`; the draft Implementation PR targets `plan/native-kubernetes-execution`. Authorization includes developing this amendment before plan merge, not approval or merge readiness. Human alone merges. @@ -262,7 +262,7 @@ ledgers and retained isolation/configuration in the resource and rehydration led ### Scenario 1 — Disabled means absent, including startup validation -The optional deployment boundary follows [draft ADR 0349](../adr/0349-native-kubernetes-execution.md). +The optional deployment boundary follows [draft ADR 0350](../adr/0350-native-kubernetes-execution.md). Current startup provider validation reaches `Bind` (`docs/design/IMPLEMENTATION-NOTES.md:6549-6560`), so execution configuration needs a side-effect-free validation path rather than a fake allocation. @@ -272,20 +272,20 @@ so execution configuration needs a side-effect-free validation path rather than - AC1.2: The provider service deploys, restarts, and reconciles independently of mecak8s. The default mecak8s chart has no dependency on the provider chart and installs no provider resource; enabling the client adapter requires explicit endpoint/profile configuration. The adapter needs no Pod/PVC/controller management RBAC. - verify: `TestNativeKubernetesExecution_Scenario1_DefaultChartIndependent` - AC1.3: Preflight validates configured profiles and endpoint/provider compatibility without allocating an environment. Allocation occurs only at an authorized session-binding operation; enabled configuration that is incomplete fails startup before allocation, and a configured but unavailable endpoint fails validation clearly with no local fallback. - - verify: `TestADR_0349_PreflightNeverAllocates` + - verify: `TestADR_0350_PreflightNeverAllocates` - AC1.4: Disabling the mecak8s client integration neither adopts nor deletes existing `ExecutionEnvironment` CRs, PVCs, or Pods and does not stop the separate provider's reconciliation for its existing allocations or other authorized clients. - verify: `TestNativeKubernetesExecution_Scenario1_DisabledDoesNotAdoptOrDeleteExistingResources` ### Scenario 2 — One idempotent logical environment is allocated per binding The controller owns a logical environment, PVC, and executor lifecycle rather than equating identity -with an ephemeral Pod, as decided in [draft ADR 0349](../adr/0349-native-kubernetes-execution.md). +with an ephemeral Pod, as decided in [draft ADR 0350](../adr/0350-native-kubernetes-execution.md). **Acceptance:** - AC2.1: Repeating the same authorized ensure request with the same final idempotency identity returns the same environment generation and does not create a second PVC or Pod. - verify: `TestNativeKubernetesExecution_Scenario2_IdempotentEnsure` - AC2.2: A stale/different profile, digest, storage request, generation, or owner fails with the final reviewed conflict/precondition error and never adopts or mutates an unrelated allocation. - - verify: `TestADR_0349_AllocationIdentityFailsClosed` + - verify: `TestADR_0350_AllocationIdentityFailsClosed` - AC2.3: Only operator-configured profile references select digest-pinned images and storage; public clients/models cannot submit arbitrary images, Pod specs, paths, URLs, source credentials, or Kubernetes object names. - verify: `TestNativeKubernetesExecution_Scenario2_ProfileIsOperatorSelected` - AC2.4: After transient PVC or Pod creation failure, including prolonged quota saturation, an existing allocation converges when the failure clears without unrelated CR/reference changes, restart, or manual reconcile. Retries remain rate-limited and continue while the object exists; wrong ownership and missing/mismatched authoritative UIDs never trigger adoption or replacement. @@ -295,7 +295,7 @@ with an ephemeral Pod, as decided in [draft ADR 0349](../adr/0349-native-kuberne The adapter preserves the immutable environment and independent read ledger (`engine/tool/environment.go:9`; `engine/tool/ledger.go:1`) and the existing WorkspaceReader seam -(`engine/tool/tool.go:314`), under [draft ADR 0349](../adr/0349-native-kubernetes-execution.md). +(`engine/tool/tool.go:314`), under [draft ADR 0350](../adr/0350-native-kubernetes-execution.md). **Acceptance:** - AC3.1: A fresh environment can create, read, conditionally edit, replace, copy, move, remove, list, grep, and glob a small Go fixture with unchanged tool schemas and the existing read-before-edit/version behavior where that contract applies. @@ -303,23 +303,23 @@ The adapter preserves the immutable environment and independent read ledger - AC3.2: Read/Edit/Write preserve their applicable recorded-read and CreateFile/ReplaceFile CAS contracts; Copy/Move/Remove preserve their own positive conformance contracts and do not require an unrelated content read ledger/CAS precondition. - verify: `TestInvariant_remote_execution_preserves_file_version_protocol` - AC3.3: File access is physically confined through symlink traversal rather than lexical checks alone, and every supported read/mutate/list/search/command request is authorized for the exact environment; stale, expired, revoked, or wrong-environment grants are denied. Detached status/cancel requests authorize before returning Unimplemented; this slice exposes no command-stream RPC. - - verify: `TestADR_0349_RemoteFilesystemAndOperationAuthorizationFailClosed` + - verify: `TestADR_0350_RemoteFilesystemAndOperationAuthorizationFailClosed` - AC3.4: Shell is bound to the same environment namespace as Workspace. Cancellation distinguishes requested cancellation, acknowledged complete process termination, and externally proven compute/storage fencing; namespace teardown includes detached descendants or fails closed. Output is capped and valid UTF-8 repaired, and Shell-created file changes are not falsely claimed to participate in Workspace CAS. - verify: `TestNativeKubernetesExecution_Scenario3_ShellNamespaceCancelAndCASDisclosure` ### Scenario 4 — Caller isolation and execution ownership survive failures A session lease is session-only and its token is not consulted for writes -(`engine/port/lease.go:37-48`); [draft ADR 0349](../adr/0349-native-kubernetes-execution.md) +(`engine/port/lease.go:37-48`); [draft ADR 0350](../adr/0350-native-kubernetes-execution.md) therefore requires independent environment execution fencing. **Acceptance:** - AC4.1: Ordinary creator client A cannot discover, attach, execute in, stream from, cancel, or retire creator client B's environment; hidden and absent allocations are indistinguishable at the public boundary. The explicit scoped administrative exception in AC4.4 permits only its named administrative RPCs. - verify: `TestNativeKubernetesExecution_Scenario4_CallerEnvironmentIsolation` - AC4.2: Every command is authorized for one immutable environment revision and a distinct transient execution-fence epoch; lease loss or grant revocation prevents new commands and cancels/fences active work without exposing capability credentials to the workload or another caller's output. Same-Pod placement alone is not a security boundary. - - verify: `TestADR_0349_ExecutionGrantIsEnvironmentAndFenceScoped` + - verify: `TestADR_0350_ExecutionGrantIsEnvironmentAndFenceScoped` - AC4.3: Timeout, Pod deletion, controller restart, Lease timeout, or network partition alone never authorizes a replacement executor while an old writer may run; unknown fencing state fails closed and requires the reviewed manual/external fencing path. - - verify: `TestADR_0349_PartitionCannotAuthorizeTakeoverByTimeout` + - verify: `TestADR_0350_PartitionCannotAuthorizeTakeoverByTimeout` - AC4.4: A distinct `administrator:true` client with `administratorFor:[creatorURI]` can administer that creator's environment through all six administrative RPCs, including migration, revocation, and retained deletion, while preserving immutable creator identity and exact owner/revision/epoch/UID/operation checks. A second creator, wrong owner, stale identity, or replay with changed inputs is denied without existence disclosure. Self-admin compatibility remains explicit; no Files/Shell/attach/run/reference or MayAttestOwner authority is gained. - verify: `TestNativeKubernetesExecution_Scenario4_ScopedAdministratorAllPaths`; `TestKindExecutionProductionScopedAdministrator` (new amendment proofs, PENDING) - AC4.5: Strict `administratorFor` validation and canonical digest inclusion prevent ambiguous scope. Removal at a higher security generation takes effect on existing connections and receipt replay; an equal-generation scope edit fails closed. Old binaries reject the field and the documented quiesced upgrade preserves authority history. @@ -327,18 +327,18 @@ therefore requires independent environment execution fencing. ### Scenario 5 — Restart preserves workspace while retirement is deliberate -Exact reattachment and safe lifecycle behavior follow [draft ADR 0349](../adr/0349-native-kubernetes-execution.md) +Exact reattachment and safe lifecycle behavior follow [draft ADR 0350](../adr/0350-native-kubernetes-execution.md) and remain separate from the session-only lease contract (`engine/port/lease.go:37-48`). **Acceptance:** - AC5.1: After mecak8s and controller restart, exact reattachment to the persisted environment revision restores the same PVC data and a correctly bound runner; an unavailable/mismatched generation fails with no default/local fallback. - verify: `TestNativeKubernetesExecution_Scenario5_RestartReattachesPVC` - AC5.2: Executor Pod replacement preserves the logical environment and PVC while current Pod UID may change without a spec `metadata.generation` change; `observedGeneration` records processed spec generation, and status records current Pod/PVC UID references without using them as durable session identity. - - verify: `TestADR_0349_LogicalIdentityOutlivesPodUID` + - verify: `TestADR_0350_LogicalIdentityOutlivesPodUID` - AC5.3: The lifecycle matrix is explicit: removal of one reference retains the environment; removal of the last reference retains it by default; committed data retires only through explicit authorized retirement; a retiring environment rejects new bindings and successors. An unreadable store/reference is not an orphan and is retained. - verify: `TestNativeKubernetesExecution_Scenario5_RetentionAndSafeRetirement` - AC5.4: A pending allocation whose session association might have committed is not garbage-collected merely after TTL; collection waits for conclusive reconciliation that publication is absent. Explicit retirement of a live/shared environment is refused while any live reference exists unless a separately reviewed retire/quiesce policy authorizes it. - - verify: `TestADR_0349_PendingAllocationAndLiveReferenceRetirementFailClosed` + - verify: `TestADR_0350_PendingAllocationAndLiveReferenceRetirementFailClosed` - AC5.5: Default chart uninstall retains runtime CRs/PVCs and preserves workload network isolation and security authority history while executors survive. Retained security/capacity ledgers and required configuration outlive the provider Deployment. Deletion of CRDs with live resources is destructive and unsupported; no destructive hook or force cleanup is provided. Owner references or retention annotation strings alone are not lifecycle proof. - verify: `TestChartHasNoDeletionHook`; `TestKindExecutionProductionHelmRetentionAndReinstall` (behavioral amendment proof, PENDING) - AC5.6: PVC persistence is not documented as node-disaster recovery; kind host-local storage proves restart survival only, and a dead or unobservable node is outside that positive guarantee. @@ -350,7 +350,7 @@ and remain separate from the session-only lease contract (`engine/port/lease.go: The first slice avoids claiming that local project trust applies remotely: current admission is rooted in local composition (`internal/app/project_ingestion.go:20`) and AGENTS discovery reads Workspace -(`engine/prompt/builder.go:337`). The boundary is recorded in [draft ADR 0349](../adr/0349-native-kubernetes-execution.md). +(`engine/prompt/builder.go:337`). The boundary is recorded in [draft ADR 0350](../adr/0350-native-kubernetes-execution.md). **Acceptance:** - AC6.1: A remote execution session receives operator-global instructions and only its explicitly supported catalog; project AGENTS/rules/skills and Git source ingestion are off unless a later reviewed source/trust contract enables them. @@ -362,7 +362,7 @@ in local composition (`internal/app/project_ingestion.go:20`) and AGENTS discove ### Scenario 7 — kind proves an offline coding flow; live qualification is separate -The deterministic correctness gate is a focused mock-only task, following [draft ADR 0349](../adr/0349-native-kubernetes-execution.md). +The deterministic correctness gate is a focused mock-only task, following [draft ADR 0350](../adr/0350-native-kubernetes-execution.md). The stack additionally requires the explicitly invoked native-provider OpenRouter coding qualification recorded in Human decisions; it is completion evidence, not an always-on CI dependency. The broad mecak8s live suite is not native-provider proof. The focused mock task must not inherit ambient provider credentials. @@ -383,7 +383,7 @@ live suite is not native-provider proof. The focused mock task must not inherit Source/test locations in this table refer to implementation commit `deaf1c3d1dfe7ea92afc8fe826f1bc613080f219`, not files supplied by this plan branch. -Existing `TestNativeKubernetesExecution_*`/`TestADR_0349_*` verify labels that do not yet resolve are +Existing `TestNativeKubernetesExecution_*`/`TestADR_0350_*` verify labels that do not yet resolve are future acceptance obligations, not invented test results. Before completion, replace those labels with exact executable proofs that cover the full AC, or implement the missing proof, and pass strict trace. A row naming a unit/fake-client test establishes only that boundary; it does not prove the entire AC. diff --git a/docs/adr/0349-native-kubernetes-execution.md b/docs/adr/0350-native-kubernetes-execution.md similarity index 99% rename from docs/adr/0349-native-kubernetes-execution.md rename to docs/adr/0350-native-kubernetes-execution.md index 7f5f032265..fd24e7aca3 100644 --- a/docs/adr/0349-native-kubernetes-execution.md +++ b/docs/adr/0350-native-kubernetes-execution.md @@ -1,4 +1,4 @@ -# ADR 0349 — Optional native Kubernetes execution environments +# ADR 0350 — Optional native Kubernetes execution environments - Status: Draft - Date: 2026-09-15; draft amended 2026-09-21 diff --git a/docs/adr/README.md b/docs/adr/README.md index 0b77b2e022..2ae70f9d5e 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -84,7 +84,7 @@ Documentation/citation conventions are in [`docs/design/README.md`](../design/RE - [0349 - Cause-free TypeScript SDK malformed-success decoding](./0349-typescript-sdk-malformed-success-decoding.md) *(proposed; narrows decoder diagnostics at the HTTP successful-response boundary)* - [0342 - Gate runs on unresolved live context windows](./0342-context-window-admission.md) *(supersedes ADR 0016 only for pre-swap run admission)* - [0346 - Prompt-cache breakpoints are protocol-native, never vendor-keyed](./0346-unified-prompt-cache-dialect.md) *(supersedes ADR 0100's prompt_cache_breakpoint deferral, its root cache_control dialect arm, and its OpenRouter TTL deferral; extends ADR 0334 to OpenRouter)* -- [0349 — Optional native Kubernetes execution provider](./0349-native-kubernetes-execution.md) *(draft; separate service, preserves ADR 0048)* +- [0350 — Optional native Kubernetes execution provider](./0350-native-kubernetes-execution.md) *(draft; separate service, preserves ADR 0048)* - [0036 — `engine/` is its own Go module (monorepo via `go.work`)](./0036-engine-module.md) - [0037 — Engine public-API stability contract](./0037-engine-stability-contract.md) - [0038 — Event-sourced SessionStore rehydration (the reference fold)](./0038-event-sourced-rehydration.md) *(Decision 3 origin-opacity and no-public-replay clauses proposed to be superseded by 0337)* From 5f49a36e98ba9909c99ade7c54627e79a951f361 Mon Sep 17 00:00:00 2001 From: Juan Antonio Osorio Date: Mon, 21 Sep 2026 21:52:47 +0300 Subject: [PATCH 07/16] docs: repair custom-provider metadata proof pointer Co-Authored-By: OpenAI --- docs/acceptance/operator-defined-llm-providers.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/acceptance/operator-defined-llm-providers.md b/docs/acceptance/operator-defined-llm-providers.md index 003691ee1b..4ff3f3c4f3 100644 --- a/docs/acceptance/operator-defined-llm-providers.md +++ b/docs/acceptance/operator-defined-llm-providers.md @@ -135,7 +135,7 @@ follows the live-catalog fallback and capability-truth discipline in - verify: `TestOperatorDefinedLLMProviders_Scenario4_ListingFallback` - AC4.4: unknown live model metadata never overclaims image, audio, reasoning, output, or context-window capabilities. - - verify: `TestInvariant_custom_provider_live_metadata_conservative` + - verify: `TestInvariant_custom_provider_omitted_live_modalities_use_adapter` --- @@ -240,7 +240,7 @@ session rehydration. - `TestADR_0238_BuiltinOverrideAllowlist` - `TestInvariant_custom_provider_authfile_strict` - `TestInvariant_custom_provider_has_no_builtin_private_options` -- `TestInvariant_custom_provider_live_metadata_conservative` +- `TestInvariant_custom_provider_omitted_live_modalities_use_adapter` - `TestOperatorDefinedLLMProviders_Scenario1_ValidDefinition` - `TestOperatorDefinedLLMProviders_Scenario2_SeparateAuthFileKeys` - `TestOperatorDefinedLLMProviders_Scenario3_DeclaredFlavorSelectsAdapter` From f6fed8e1b9636f88d91e18d740ff6f815280b9a9 Mon Sep 17 00:00:00 2001 From: Juan Antonio Osorio Date: Mon, 21 Sep 2026 21:52:47 +0300 Subject: [PATCH 08/16] docs: specify exact native migration receipt replay Co-Authored-By: OpenAI --- docs/acceptance/native-kubernetes-execution.md | 2 +- docs/adr/0350-native-kubernetes-execution.md | 4 +++- 2 files changed, 4 insertions(+), 2 deletions(-) diff --git a/docs/acceptance/native-kubernetes-execution.md b/docs/acceptance/native-kubernetes-execution.md index 0c5b245af9..984a7e5562 100644 --- a/docs/acceptance/native-kubernetes-execution.md +++ b/docs/acceptance/native-kubernetes-execution.md @@ -142,7 +142,7 @@ independent of the unresolved release maturity designation. The structural schem | `activeOperation` | `id, operation, startedAt, claimID, runID, epoch, holderID, renewedAt, expiresAt`. Lease-holder identity is replica-specific; an expired holder remains recorded for recovery. | | Receipts | `renewReceipts[{operationID,fingerprint,expiresAt}]` and `revocationReceipts[{operationID,fingerprint,expectedGrantGeneration,grantGeneration}]`, each capped at 32. Receipt eviction does not authorize replay past current claim/generation preconditions. | | Lifecycle | `lifecycleOperation{id,type,phase,expectedEpoch,expectedPodUID,expectedPVCUID,createdAt}`; types ReplaceExecutor, RetireEnvironment, DeleteRetiredEnvironment; phases Quiescing, WaitingForTermination, RemovingPodFinalizer, WaitingForPodDeletion, CreatingReplacement, DeletingPVC, ReleasingSlot. | -| Recovery/migration | `terminationProof{operationID,podUID,pvcUID,epoch,podPhase,observedAt}` (Succeeded/Failed only); `lastReplacement{operationID,previousPodUID,replacementPodUID,pvcUID,previousEpoch,replacementEpoch}`; `migrationOperation{id,fromSchema,expectedPodUID,expectedPVCUID}` and `lastMigrationOperationID`. Schema 0/1 is accepted only for explicit compatible migration, never ordinary execution; unknown schemas fail closed. | +| Recovery/migration | `terminationProof{operationID,podUID,pvcUID,epoch,podPhase,observedAt}` (Succeeded/Failed only); `lastReplacement{operationID,previousPodUID,replacementPodUID,pvcUID,previousEpoch,replacementEpoch}`; `migrationOperation{id,fromSchema,expectedPodUID,expectedPVCUID}` and `lastMigrationOperationID, lastMigrationFromSchema` (integer 0 or 1; presence required for completed replay). Completed and concurrent replay must match the operation, source schema, exact subject, and runtime UIDs; old receipts without source-schema presence fail closed. Schema 0/1 is accepted only for explicit compatible migration, never ordinary execution; unknown schemas fail closed. | Run claims serialize environment use independently of session leases. A separate 30s operation-holder lease renews every TTL/3. Every operation rechecks current claim, owner, creator, revision, epoch, and diff --git a/docs/adr/0350-native-kubernetes-execution.md b/docs/adr/0350-native-kubernetes-execution.md index fd24e7aca3..773810294b 100644 --- a/docs/adr/0350-native-kubernetes-execution.md +++ b/docs/adr/0350-native-kubernetes-execution.md @@ -58,7 +58,9 @@ Only operator-configured, digest-pinned profiles select image, storage, resource RuntimeClass. Public clients/models cannot submit arbitrary Pod specs, images, paths, URLs, credentials, or Kubernetes names. Creator `clientHash`, owner identity, allocation fingerprint, and revision remain immutable. The [schema and transaction contract](../acceptance/native-kubernetes-execution.md#reference-transaction-schema-v2-and-fencing) -records exact fields, receipts, conditions, and migration behavior. +records exact fields, receipts, conditions, and migration behavior. Completed migration receipts retain +the source schema with explicit presence (including legacy schema zero); replay requires the exact +request identity, and receipts missing that information fail closed. Transient PVC/Pod create errors and quota saturation must recover through continuing rate-limited reconciliation while the CR exists. Quota restoration alone must be sufficient; no unrelated reference From 7172ce89e294a7d70e3ad0636afdc270c5553a23 Mon Sep 17 00:00:00 2001 From: Juan Antonio Osorio Date: Mon, 21 Sep 2026 22:19:35 +0300 Subject: [PATCH 09/16] docs: trace native execution acceptance to implemented proofs Co-Authored-By: OpenAI --- .../acceptance/native-kubernetes-execution.md | 113 ++++++++++-------- docs/adr/0350-native-kubernetes-execution.md | 4 +- 2 files changed, 63 insertions(+), 54 deletions(-) diff --git a/docs/acceptance/native-kubernetes-execution.md b/docs/acceptance/native-kubernetes-execution.md index 984a7e5562..b911e574d1 100644 --- a/docs/acceptance/native-kubernetes-execution.md +++ b/docs/acceptance/native-kubernetes-execution.md @@ -9,7 +9,7 @@ **Expected tasks:** draft implementation decomposition remains run-local; no approved orchestration baseline. **Issue:** None — planning request has no linked issue. **Plan PR:** [#1579](https://github.com/stacklok/mecatl/pull/1579), draft on `plan/native-kubernetes-execution`; implementation [#1614](https://github.com/stacklok/mecatl/pull/1614) remains draft. -**Approved baseline:** absent. Plan worktree baseline `a5030545e87e81aad12fba4c49f0c9933a634b5e` and implementation reference `deaf1c3d1dfe7ea92afc8fe826f1bc613080f219` are evidence, not approval. +**Approved baseline:** absent. Plan worktree baseline `a5030545e87e81aad12fba4c49f0c9933a634b5e` and implementation source reference `9bb4d89b8aa497c9d2cd6aa4a94ab9262a913364` are evidence, not approval. The [implementation proof map](#implementation-proof-map) distinguishes existing draft coverage and successful qualification from amendment proofs still pending. Implementation-only source paths below @@ -43,11 +43,10 @@ ADR 0048 continues to govern storage-free mecak8s; this separate provider does n ## Interface contract -This documentation-only amendment reconciles the draft implementation at `deaf1c3d1` with the -proposed contract. It does not turn authored specifications, advisory architect/devil review, or passing -CI into human approval. Scoped administration, provisioning convergence, and Helm lifecycle safeguards -are explicit amendment requirements still to be implemented and qualified. Other details below describe -the inspected draft, subject to the unchecked reviews above. +This documentation-only amendment traces the draft implementation at `9bb4d89b8` against the +proposed contract. Authored specifications, advisory review, and passing CI do not constitute human +approval. Scoped administration, continuing provisioning retries, and Helm lifecycle proofs exist in +that source; final runtime qualification and the unchecked reviews above remain pending. - **gRPC / protobuf:** Public Harness protobuf is unchanged. The private `mecatl.execution.v1.ExecutionProviderService` uses mandatory mTLS, a `host:port` endpoint, protocol identifier `execution-grpc/1`, and an 8 MiB message bound. The exact wire schema is the implementation reference's `contracts/proto/mecatl/execution/v1/execution.proto`, generated with `task generate`. The RPC/field summary below is the review contract; there is no HTTP/JSON compatibility endpoint. - **Exported Go APIs / interfaces:** No `engine/` exported API change. `tool.Environment` remains identity + Workspace + independent ReadLedger + optional bound CommandRunner (`engine/tool/environment.go`). Host-only seams in the implementation's `internal/adapter/server/placement.go` are specified below; Kubernetes dependencies stay outside the engine. @@ -142,7 +141,7 @@ independent of the unresolved release maturity designation. The structural schem | `activeOperation` | `id, operation, startedAt, claimID, runID, epoch, holderID, renewedAt, expiresAt`. Lease-holder identity is replica-specific; an expired holder remains recorded for recovery. | | Receipts | `renewReceipts[{operationID,fingerprint,expiresAt}]` and `revocationReceipts[{operationID,fingerprint,expectedGrantGeneration,grantGeneration}]`, each capped at 32. Receipt eviction does not authorize replay past current claim/generation preconditions. | | Lifecycle | `lifecycleOperation{id,type,phase,expectedEpoch,expectedPodUID,expectedPVCUID,createdAt}`; types ReplaceExecutor, RetireEnvironment, DeleteRetiredEnvironment; phases Quiescing, WaitingForTermination, RemovingPodFinalizer, WaitingForPodDeletion, CreatingReplacement, DeletingPVC, ReleasingSlot. | -| Recovery/migration | `terminationProof{operationID,podUID,pvcUID,epoch,podPhase,observedAt}` (Succeeded/Failed only); `lastReplacement{operationID,previousPodUID,replacementPodUID,pvcUID,previousEpoch,replacementEpoch}`; `migrationOperation{id,fromSchema,expectedPodUID,expectedPVCUID}` and `lastMigrationOperationID, lastMigrationFromSchema` (integer 0 or 1; presence required for completed replay). Completed and concurrent replay must match the operation, source schema, exact subject, and runtime UIDs; old receipts without source-schema presence fail closed. Schema 0/1 is accepted only for explicit compatible migration, never ordinary execution; unknown schemas fail closed. | +| Recovery/migration | `terminationProof{operationID,podUID,pvcUID,epoch,podPhase,observedAt}` (Succeeded/Failed only); `lastReplacement{operationID,previousPodUID,replacementPodUID,pvcUID,previousEpoch,replacementEpoch}`; `migrationOperation{id,fromSchema,expectedPodUID,expectedPVCUID}` and `lastMigrationOperationID, lastMigrationFromSchema` (integer 0 or 1; presence required for completed replay). Completed receipts expire atomically when replacement publishes the new Pod identity; neither the old request nor the same operation with replacement UIDs may revive them. Completed and concurrent replay must match the operation, source schema, exact subject, and runtime UIDs; old receipts without source-schema presence fail closed. Schema 0/1 is accepted only for explicit compatible migration, never ordinary execution; unknown schemas fail closed. | Run claims serialize environment use independently of session leases. A separate 30s operation-holder lease renews every TTL/3. Every operation rechecks current claim, owner, creator, revision, epoch, and @@ -268,13 +267,13 @@ so execution configuration needs a side-effect-free validation path rather than **Acceptance:** - AC1.1: With execution disabled, mecak8s creates no Kubernetes execution client, informer, goroutine, CRD, RBAC, controller, executor, PVC, or Pod, and performs no execution API call. - - verify: `TestNativeKubernetesExecution_Scenario1_DisabledHasNoEffects` + - verify: `TestRemoteDeploymentNoFSUsesLocalAttenuationWithoutProviderCall`; inspection — full disabled-startup side-effect proof is MISSING (the no-fs test is partial, not disabled integration). - AC1.2: The provider service deploys, restarts, and reconciles independently of mecak8s. The default mecak8s chart has no dependency on the provider chart and installs no provider resource; enabling the client adapter requires explicit endpoint/profile configuration. The adapter needs no Pod/PVC/controller management RBAC. - - verify: `TestNativeKubernetesExecution_Scenario1_DefaultChartIndependent` + - verify: `TestChartRetainsCRDAndDoesNotGrantSecretAPI`, `TestKindExecutionProductionReplicaLifecycle`; inspection — default chart dependency/RBAC review and final Kind execution PENDING. - AC1.3: Preflight validates configured profiles and endpoint/provider compatibility without allocating an environment. Allocation occurs only at an authorized session-binding operation; enabled configuration that is incomplete fails startup before allocation, and a configured but unavailable endpoint fails validation clearly with no local fallback. - - verify: `TestADR_0350_PreflightNeverAllocates` + - verify: `TestHandlerValidateIsReadOnlyAndEnsureIdempotent`, `TestGRPCReadinessAndMTLSAreMandatory`, `TestKindExecutionQualification`; inspection — incomplete enabled-config startup-before-allocation coverage remains MISSING; final Kind run PENDING. - AC1.4: Disabling the mecak8s client integration neither adopts nor deletes existing `ExecutionEnvironment` CRs, PVCs, or Pods and does not stop the separate provider's reconciliation for its existing allocations or other authorized clients. - - verify: `TestNativeKubernetesExecution_Scenario1_DisabledDoesNotAdoptOrDeleteExistingResources` + - verify: `TestStartupDoesNotFenceLivePeerOperation`; inspection — peer restart is partial coverage; disabled-client retained-resource lifecycle proof remains MISSING. ### Scenario 2 — One idempotent logical environment is allocated per binding @@ -283,13 +282,13 @@ with an ephemeral Pod, as decided in [draft ADR 0350](../adr/0350-native-kuberne **Acceptance:** - AC2.1: Repeating the same authorized ensure request with the same final idempotency identity returns the same environment generation and does not create a second PVC or Pod. - - verify: `TestNativeKubernetesExecution_Scenario2_IdempotentEnsure` + - verify: `TestHandlerValidateIsReadOnlyAndEnsureIdempotent`, `TestKindExecutionQualification` (final Kind execution PENDING). - AC2.2: A stale/different profile, digest, storage request, generation, or owner fails with the final reviewed conflict/precondition error and never adopts or mutates an unrelated allocation. - - verify: `TestADR_0350_AllocationIdentityFailsClosed` + - verify: `TestStoreEnsureUsesStableLookupAndRejectsFingerprintDrift`, `TestReconcileRefusesForeignExistingPVCWithoutPersistingUID`. - AC2.3: Only operator-configured profile references select digest-pinned images and storage; public clients/models cannot submit arbitrary images, Pod specs, paths, URLs, source credentials, or Kubernetes object names. - - verify: `TestNativeKubernetesExecution_Scenario2_ProfileIsOperatorSelected` + - verify: `TestLoadProfilesStrictAndDigestPinned`, `TestReconcileCreatesTokenlessNonRootPodAndRetainedPVC`; inspection — public request/schema authority review PENDING. - AC2.4: After transient PVC or Pod creation failure, including prolonged quota saturation, an existing allocation converges when the failure clears without unrelated CR/reference changes, restart, or manual reconcile. Retries remain rate-limited and continue while the object exists; wrong ownership and missing/mismatched authoritative UIDs never trigger adoption or replacement. - - verify: `TestNativeKubernetesExecution_Scenario2_ProvisioningRetriesUntilRecovery`; `TestKindExecutionProductionProvisioningConvergence` (new amendment proofs, PENDING) + - verify: `TestProvisioningWorkerConvergesWithoutAnotherEvent`, `TestProvisioningPreservesCreateAndConditionErrors`, `TestProvisioningRetriesNeverReplaceAuthoritativeResources`, `TestKindExecutionProductionQuotaSaturation` (Kind quota-restoration proof exists; final execution PENDING). ### Scenario 3 — Existing file tools and Shell are transparent but honest @@ -299,13 +298,13 @@ The adapter preserves the immutable environment and independent read ledger **Acceptance:** - AC3.1: A fresh environment can create, read, conditionally edit, replace, copy, move, remove, list, grep, and glob a small Go fixture with unchanged tool schemas and the existing read-before-edit/version behavior where that contract applies. - - verify: `TestNativeKubernetesExecution_Scenario3_FileToolConformance` + - verify: `TestFileOperationsAreConfinedAndVersioned`, `TestProviderThroughRealGRPCSignedHandlerRefreshesAndReattachesExactly`, `TestKindExecutionQualification`; inspection — full native per-tool positive conformance matrix remains MISSING; final Kind execution PENDING. - AC3.2: Read/Edit/Write preserve their applicable recorded-read and CreateFile/ReplaceFile CAS contracts; Copy/Move/Remove preserve their own positive conformance contracts and do not require an unrelated content read ledger/CAS precondition. - - verify: `TestInvariant_remote_execution_preserves_file_version_protocol` + - verify: `TestFileOperationsAreConfinedAndVersioned`, `TestRenameAndCopyNeverClobber`; inspection — native recorded-read and positive Copy/Move/Remove conformance remain MISSING (negative clobber tests are not positive conformance). - AC3.3: File access is physically confined through symlink traversal rather than lexical checks alone, and every supported read/mutate/list/search/command request is authorized for the exact environment; stale, expired, revoked, or wrong-environment grants are denied. Detached status/cancel requests authorize before returning Unimplemented; this slice exposes no command-stream RPC. - - verify: `TestADR_0350_RemoteFilesystemAndOperationAuthorizationFailClosed` + - verify: `TestFileOperationsAreConfinedAndVersioned`, `TestSecurityManagerGuardsEveryRPCOnExistingConnection`, `TestGrantRoundTripAndExactBindings`, `TestGrantRejectsExpiryRevocationAndUnknownFields`, `TestCommandStatusIsUnimplementedOnlyAfterAuthorization`; inspection — per-file-operation confinement/authorization matrix remains partial. - AC3.4: Shell is bound to the same environment namespace as Workspace. Cancellation distinguishes requested cancellation, acknowledged complete process termination, and externally proven compute/storage fencing; namespace teardown includes detached descendants or fails closed. Output is capped and valid UTF-8 repaired, and Shell-created file changes are not falsely claimed to participate in Workspace CAS. - - verify: `TestNativeKubernetesExecution_Scenario3_ShellNamespaceCancelAndCASDisclosure` + - verify: `TestForegroundCommandReturnsReceiptAndTimeoutTerminatesDescendants`, `TestSuccessfulDetachedChildIsKilledBeforeTerminalReceipt`, `TestCancelledStoreOperationStaysActiveUntilBackendStopsThenFences`, `TestKindExecutionProductionHolderLossFencesActiveOperation`; inspection — native output-cap/UTF-8 and Shell-CAS disclosure proof remains partial; final Kind execution PENDING. ### Scenario 4 — Caller isolation and execution ownership survive failures @@ -315,15 +314,15 @@ therefore requires independent environment execution fencing. **Acceptance:** - AC4.1: Ordinary creator client A cannot discover, attach, execute in, stream from, cancel, or retire creator client B's environment; hidden and absent allocations are indistinguishable at the public boundary. The explicit scoped administrative exception in AC4.4 permits only its named administrative RPCs. - - verify: `TestNativeKubernetesExecution_Scenario4_CallerEnvironmentIsolation` + - verify: `TestClientScopedIntentListReturnsAttestedOwnerOnlyToOwningClient`, `TestScopedAdminAllRoutesOverMTLS`, `TestScopedAdminWithOwnerAttestationCannotUseAnotherCreatorsDataPlane`, `TestKindExecutionQualification` (final Kind execution PENDING). - AC4.2: Every command is authorized for one immutable environment revision and a distinct transient execution-fence epoch; lease loss or grant revocation prevents new commands and cancels/fences active work without exposing capability credentials to the workload or another caller's output. Same-Pod placement alone is not a security boundary. - - verify: `TestADR_0350_ExecutionGrantIsEnvironmentAndFenceScoped` + - verify: `TestGrantRoundTripAndExactBindings`, `TestRevokeEnvironmentFencesOldClaimWithoutChangingExecutionEpoch`, `TestAcquireRunAndRevokeUseResourceVersionCAS`, `TestKindExecutionProductionSecurityRotation`, `TestKindExecutionProductionHolderLossFencesActiveOperation` (final Kind execution PENDING). - AC4.3: Timeout, Pod deletion, controller restart, Lease timeout, or network partition alone never authorizes a replacement executor while an old writer may run; unknown fencing state fails closed and requires the reviewed manual/external fencing path. - - verify: `TestADR_0350_PartitionCannotAuthorizeTakeoverByTimeout` + - verify: `TestExpiredOperationLeaseFencesWithoutClearingIdentity`, `TestRecoverMissingPodRemainsFenceUnknown`, `TestRecoveredTerminalProofCanStartExactReplacement`, `TestKindExecutionProductionHolderLossFencesActiveOperation` (final Kind execution PENDING); inspection — external fencing runbook review PENDING. - AC4.4: A distinct `administrator:true` client with `administratorFor:[creatorURI]` can administer that creator's environment through all six administrative RPCs, including migration, revocation, and retained deletion, while preserving immutable creator identity and exact owner/revision/epoch/UID/operation checks. A second creator, wrong owner, stale identity, or replay with changed inputs is denied without existence disclosure. Self-admin compatibility remains explicit; no Files/Shell/attach/run/reference or MayAttestOwner authority is gained. - - verify: `TestNativeKubernetesExecution_Scenario4_ScopedAdministratorAllPaths`; `TestKindExecutionProductionScopedAdministrator` (new amendment proofs, PENDING) + - verify: `TestScopedAdminAllRoutesOverMTLS`, `TestScopedAdminCASRetryRechecksSubject`, `TestScopedAdminMigrationReceiptRetainsUIDPreconditions`, `TestMigrationCompletedCASReplayRequiresExactSourceSchema`, `TestMigrationReceiptExpiresAfterReconciledReplacement`, `TestScopedAdminDoesNotGrantAttestationOrDataPlane`, `TestScopedAdminWithOwnerAttestationCannotUseAnotherCreatorsDataPlane`, `TestKindExecutionProductionScopedAdministrator`; inspection — replacement-expiry regression is in the targeted repair, not the pinned baseline; final Kind execution PENDING (Kind covers replacement/non-escalation, not all six routes). - AC4.5: Strict `administratorFor` validation and canonical digest inclusion prevent ambiguous scope. Removal at a higher security generation takes effect on existing connections and receipt replay; an equal-generation scope edit fails closed. Old binaries reject the field and the documented quiesced upgrade preserves authority history. - - verify: `TestNativeKubernetesExecution_Scenario4_AdministratorScopeRotation`; `TestNativeKubernetesExecution_Scenario4_AdministratorScopeValidation` (new amendment proofs, PENDING) + - verify: `TestAdministratorScopeManifestValidation`, `TestAdministratorScopeDigestIsNormalizedAndAuthorityBound`, `TestScopedAdminAllRoutesOverMTLS`, `TestScopedAdminEqualGenerationDriftFailsClosed`; inspection — old-binary rejection/quiesced-upgrade compatibility review PENDING. ### Scenario 5 — Restart preserves workspace while retirement is deliberate @@ -332,19 +331,19 @@ and remain separate from the session-only lease contract (`engine/port/lease.go: **Acceptance:** - AC5.1: After mecak8s and controller restart, exact reattachment to the persisted environment revision restores the same PVC data and a correctly bound runner; an unavailable/mismatched generation fails with no default/local fallback. - - verify: `TestNativeKubernetesExecution_Scenario5_RestartReattachesPVC` + - verify: `TestProviderThroughRealGRPCSignedHandlerRefreshesAndReattachesExactly`, `TestKindExecutionQualification`, `TestKindExecutionProductionReplicaLifecycle` (final Kind restart/data-survival execution PENDING). - AC5.2: Executor Pod replacement preserves the logical environment and PVC while current Pod UID may change without a spec `metadata.generation` change; `observedGeneration` records processed spec generation, and status records current Pod/PVC UID references without using them as durable session identity. - - verify: `TestADR_0350_LogicalIdentityOutlivesPodUID` + - verify: `TestReplacementPersistsTerminalProofBeforeRemovingPodFinalizer`, `TestStaleReplacementObservationCannotOverwriteCompletion`, `TestKindExecutionProductionReplicaLifecycle`; inspection — observedGeneration/spec-generation distinction review and final Kind execution PENDING. - AC5.3: The lifecycle matrix is explicit: removal of one reference retains the environment; removal of the last reference retains it by default; committed data retires only through explicit authorized retirement; a retiring environment rejects new bindings and successors. An unreadable store/reference is not an orphan and is retained. - - verify: `TestNativeKubernetesExecution_Scenario5_RetentionAndSafeRetirement` + - verify: `TestReplacementQuiescesAndPendingReferenceBlocksRetirement`, `TestReferenceTransactionsRetainUnknownAndNeverChangeSource`, `TestKindExecutionProductionPendingDeleteOutageRecovery` (final Kind execution PENDING). - AC5.4: A pending allocation whose session association might have committed is not garbage-collected merely after TTL; collection waits for conclusive reconciliation that publication is absent. Explicit retirement of a live/shared environment is refused while any live reference exists unless a separately reviewed retire/quiesce policy authorizes it. - - verify: `TestADR_0350_PendingAllocationAndLiveReferenceRetirementFailClosed` + - verify: `TestAmbiguousCommitIsRetainedAndNeverAbortedByCleanup`, `TestReplacementQuiescesAndPendingReferenceBlocksRetirement`, `TestReferenceIntentReconciliationIsOwnerAndRefExact`. - AC5.5: Default chart uninstall retains runtime CRs/PVCs and preserves workload network isolation and security authority history while executors survive. Retained security/capacity ledgers and required configuration outlive the provider Deployment. Deletion of CRDs with live resources is destructive and unsupported; no destructive hook or force cleanup is provided. Owner references or retention annotation strings alone are not lifecycle proof. - - verify: `TestChartHasNoDeletionHook`; `TestKindExecutionProductionHelmRetentionAndReinstall` (behavioral amendment proof, PENDING) + - verify: `TestChartHasNoDeletionHook`, `TestChartRetainedLifetime`, `TestKindExecutionProductionHelmLifetime` (offline rendering is partial; actual Helm lifecycle execution PENDING). - AC5.6: PVC persistence is not documented as node-disaster recovery; kind host-local storage proves restart survival only, and a dead or unobservable node is outside that positive guarantee. - verify: inspection — human review of deployment documentation; `task docs` checks links and structure after authorized tracked changes - AC5.7: Real Helm install/compatible upgrade/uninstall/reinstall of the same release and namespace preserves exact runtime identity, PVC contents, NetworkPolicy enforcement during provider absence, security high-water/key history, and capacity reservations. Safe adoption validates ownership; missing ledgers with retained allocations, foreign resources, stale authority, arbitrary downgrade, or incompatible profile/schema fail closed without overwrite. CRD upgrades remain manual. - - verify: `TestKindExecutionProductionHelmRetentionAndReinstall`; `TestNativeKubernetesExecution_Scenario5_HelmRetainedAuthorityFailsClosed` (new amendment proofs, PENDING) + - verify: `TestChartRetainedLifetime`, `TestSecurityLedgerRejectsKeyVersionRollbackAcrossRestart`, `TestKindExecutionProductionHelmLifetime` (real Helm lifecycle test exists; final execution PENDING). ### Scenario 6 — Remote catalog and project-source boundaries are explicit @@ -354,11 +353,11 @@ in local composition (`internal/app/project_ingestion.go:20`) and AGENTS discove **Acceptance:** - AC6.1: A remote execution session receives operator-global instructions and only its explicitly supported catalog; project AGENTS/rules/skills and Git source ingestion are off unless a later reviewed source/trust contract enables them. - - verify: `TestNativeKubernetesExecution_Scenario6_NoFalseRemoteProjectIngestion` + - verify: `TestRemoteExecutionRealFactoryCarriesPostureAndAttenuatedCatalog`; inspection — the test pins host AGENTS exclusion/catalog posture; complete operator-global and remote rules/skills/Git source matrix remains partial. - AC6.2: Schedules remain excluded. Background commands, isolated Subagent/Parallel/Team delegation, SkillDraft, Git/source operations, and unsupported successor modes are rejected with named capability/precondition errors rather than running locally or silently omitting work; `no-fs` remains unchanged. - - verify: `TestNativeKubernetesExecution_Scenario6_UnsupportedPathsNeverFallBackLocal` + - verify: `TestRemoteExecutionRealFactoryCarriesPostureAndAttenuatedCatalog`, `TestRemoteDeploymentNoFSUsesLocalAttenuationWithoutProviderCall`, `TestCommandStatusIsUnimplementedOnlyAfterAuthorization`; inspection — full unsupported-path invocation/error matrix remains partial. - AC6.3: Clear/Fork use environment exclusion and reference reservation/publication while preserving same-remote-`EnvironmentRef` successor semantics and the source on publication failure. Clear/Fork are not workspace cloning; no public Harness field is added and no model chooses an environment/profile. - - verify: `TestNativeKubernetesExecution_Scenario6_SuccessorSemantics` + - verify: `TestReferenceTransactionsRetainUnknownAndNeverChangeSource`, `TestKindExecutionProductionClearForkLifecycle` (final Kind execution PENDING); inspection — public API/profile authority review PENDING. ### Scenario 7 — kind proves an offline coding flow; live qualification is separate @@ -369,27 +368,33 @@ live suite is not native-provider proof. The focused mock task must not inherit **Acceptance:** - AC7.1: `task e2e:k8s:execution` and the enforcing-CNI `task e2e:k8s:execution:production` run on the supported generic toolchain, including CI, with a unique owned kind cluster, a scratch kubeconfig, and explicit `--kubeconfig`/context on every command. They deploy the separate chart plus explicitly connected mecak8s and use only the deterministic mock provider. The final production-profile run uses Calico and exercises negative network isolation, rotation/revocation, replica/lifecycle recovery, holder loss, quota recovery, pending delete, Clear/Fork, migration, sanitized artifacts, and the amendment regressions. - - verify: `TestNativeKubernetesExecution_Scenario7_FocusedKindMockFlow` + - verify: `TestKindExecutionQualification`, `TestKindExecutionProductionNetworkPolicyEnforced`, `TestKindExecutionProductionSecurityRotation`, `TestKindExecutionProductionReplicaLifecycle`, `TestKindExecutionProductionHolderLossFencesActiveOperation`, `TestKindExecutionProductionQuotaSaturation`, `TestKindExecutionProductionPendingDeleteOutageRecovery`, `TestKindExecutionProductionClearForkLifecycle`, `TestKindExecutionProductionCompatiblePrototypeMigration`, `TestKindExecutionProductionScopedAdministrator`, `TestKindExecutionProductionHelmLifetime` (names exist; final production Kind+Calico run PENDING). - AC7.2: The mock flow seeds a small Go fixture through file tools, asks the harness to implement a function/tests, runs `go test` through bound Shell, and verifies the persisted artifact and restart/reattach behavior. - - verify: `TestNativeKubernetesExecution_Scenario7_MockCodingFlow` + - verify: `TestKindExecutionQualification` (coding/artifact/restart assertions exist; final Kind execution PENDING). - AC7.3: Failure artifacts are bounded and sanitized; cleanup targets only the recorded owned cluster. Destructive live cleanup requires confirmation and never mutates ambient kubeconfig/context. - - verify: `TestNativeKubernetesExecution_Scenario7_OwnershipAndSanitizedArtifacts` + - verify: `TestKindExecutionProductionFailureArtifactBoundary`, `TestLifetimeOutputRejectsOversizedArtifacts`; inspection — owned-cluster cleanup/context script review and final Kind execution PENDING. - AC7.4: `task e2e:k8s:execution:live` is a separate explicit native-provider OpenRouter coding qualification, required to complete this stack after final deterministic qualification. It proves model-driven file/tool execution, independently verified persisted workspace artifacts, and a successful bound `go test` command. It uses verified nonsecret endpoint/protocol/model configuration in an authenticated operator-controlled cluster, then restores mock configuration and deletes only the run-owned Secret by recorded UID. It is not an always-on live CI dependency and claims no hard dollar cap from token/run limits. - verify: `TestKindExecutionLiveQualification`; inspection — record successful native-provider live evidence for the final implementation candidate (PENDING, not satisfied by generic mecak8s live CI) - AC7.5: A trusted runtime-only loader may consume the approved credential directly from its protected file channel and create a narrowly scoped HARNESS-only Secret through a protected channel before child processes or tools can inherit it. The agent never accesses the file or Secret; failures are redacted, and the credential never appears in model/tool output, argv, Helm values, disk manifests, git, logs, artifacts, or the execution workload. Mock tests do not access it. - - verify: `TestNativeKubernetesExecution_Scenario7_LiveSecretAndSpendBoundary` + - verify: `TestLoadCredentialAndRedaction`, `TestDeletePinsReceiptUIDAndLeavesReplacement`, `TestSecretReferencesFindEveryCredentialEscape`, `TestExpectedHarnessCredentialReferenceIsExact`, `TestKindExecutionLiveQualification`; inspection — offline synthetic-credential tests do not establish live handling; final authorized native live evidence PENDING. ## Implementation proof map -Source/test locations in this table refer to implementation commit -`deaf1c3d1dfe7ea92afc8fe826f1bc613080f219`, not files supplied by this plan branch. -Existing `TestNativeKubernetesExecution_*`/`TestADR_0350_*` verify labels that do not yet resolve are -future acceptance obligations, not invented test results. Before completion, replace those labels with -exact executable proofs that cover the full AC, or implement the missing proof, and pass strict trace. +Source/test locations in this table and the `verify:` lines refer to implementation commit +`9bb4d89b8aa497c9d2cd6aa4a94ab9262a913364`, not files supplied by this plan branch. +The labels name existing tests, not successful executions or complete behavioral coverage. +MISSING/partial annotations preserve broader proof gaps outside the targeted receipt repair; +PENDING runtime evidence means the test exists but still needs final-candidate execution. +The migration-receipt replacement-expiry regression `TestMigrationReceiptExpiresAfterReconciledReplacement` +is in the targeted repair's `internal/adapter/executioncontroller/production_lifecycle_test.go`, not this +source baseline; verify it against the implementation candidate containing this plan merge. +Run `actrace --strict --plan ` from the implementation worktree +for source resolution, then rerun after its normal plan merge. Draft status is intentional: +strict mode gates landed plans only, so inspect the native report for missing proofs as well. A row naming a unit/fake-client test establishes only that boundary; it does not prove the entire AC. **Observed qualification:** the [production Kind+Calico job](https://github.com/stacklok/mecatl/actions/runs/35589357385/job/106299991747) -completed **SUCCESS** at the exact implementation reference. This includes network isolation, +completed **SUCCESS** at historical implementation `deaf1c3d1dfe7ea92afc8fe826f1bc613080f219`. This includes network isolation, rotation/revocation, replicas/lifecycle, holder loss, quota saturation, pending-delete recovery, Clear/Fork, compatible prototype migration, and artifact isolation. Historical local keyring/toolchain failures are not active blockers for that run. This is baseline evidence, not final-candidate evidence @@ -398,8 +403,9 @@ for the new scoped-admin, continuing-retry, or Helm uninstall/reinstall requirem Draft CI skipped race lanes; non-race success is not a race result. The generic live compaction lane failure is addressed by [merged PR #1728](https://github.com/stacklok/mecatl/pull/1728) on main, not a native-provider defect or proof that its fix is present in this draft. Generic mecak8s live success is -also not AC7.4 evidence. Native-provider live qualification and all new amendment regressions remain -**PENDING**. No final live execution or human contract/panel approval is asserted here. +also not AC7.4 evidence. Final-candidate native-provider live and amended production Kind qualification +remain **PENDING**. Source resolution below does not assert new-head CI, final live execution, or human +contract/panel approval. | AC | Existing implementation proof at the reference commit; remaining evidence | |---|---| @@ -410,20 +416,21 @@ also not AC7.4 evidence. Native-provider live qualification and all new amendmen | AC2.1 | `internal/adapter/executioncontroller/handler_test.go`: `TestHandlerValidateIsReadOnlyAndEnsureIdempotent`; `e2e/k8s_execution/qualification_test.go`: `TestKindExecutionQualification`. | | AC2.2 | `internal/adapter/executioncontroller/store_test.go`: `TestStoreEnsureUsesStableLookupAndRejectsFingerprintDrift`; `controller_test.go`: `TestReconcileRefusesForeignExistingPVCWithoutPersistingUID`. | | AC2.3 | `internal/adapter/executioncontroller/profiles_test.go`: `TestLoadProfilesStrictAndDigestPinned`; production `TestKindExecutionProductionQuotaSaturation`. | -| AC2.4 | **PENDING** continuing retries after prolonged PVC/Pod failure and quota restoration with no unrelated event; current quota test alone is insufficient. | +| AC2.4 | `internal/adapter/executioncontroller/provisioning_retry_test.go`: `TestProvisioningWorkerConvergesWithoutAnotherEvent`, `TestProvisioningPreservesCreateAndConditionErrors`, `TestProvisioningRetriesNeverReplaceAuthoritativeResources` exercise PVC and Pod retry/confinement paths. Production `TestKindExecutionProductionQuotaSaturation` restores quota without a new reference event; final runtime evidence PENDING. | | AC3.1 | `internal/adapter/executionclient/client_test.go`: `TestProviderThroughRealGRPCSignedHandlerRefreshesAndReattachesExactly`; Kind coding flow succeeds, but full per-tool conformance matrix remains to be traced/qualified. | -| AC3.2 | `internal/adapter/executionclient/service_integration_test.go`: `TestServiceUsesRealMTLSProviderStoreAndReleasesOnlyAfterDrain`; retain explicit file-version and Copy/Move/Remove conformance proofs. | -| AC3.3 | `internal/adapter/executioncontroller/security_test.go`: `TestSecurityManagerGuardsEveryRPCOnExistingConnection`; `handler_test.go`: `TestHandlerRequiresAllowlistedCanonicalURISAN`; production `TestKindExecutionProductionSecurityRotation`. | +| AC3.2 | `internal/executionexecutor/executor_test.go`: `TestFileOperationsAreConfinedAndVersioned`, `TestRenameAndCopyNeverClobber`; native recorded-read and positive Copy/Move/Remove conformance remain MISSING. | +| AC3.3 | `internal/executionexecutor/executor_test.go`: confinement/version tests; `internal/executionenv/protocol_test.go`: exact binding/expiry/revocation tests; `internal/adapter/executionclient/client_test.go`: authorized detached-control rejection; `internal/adapter/executioncontroller/security_test.go`: `TestSecurityManagerGuardsEveryRPCOnExistingConnection`. Per-file-operation matrix remains partial. | | AC3.4 | `internal/adapter/executioncontroller/store_test.go`: `TestCancelledStoreOperationStaysActiveUntilBackendStopsThenFences`; production `TestKindExecutionProductionHolderLossFencesActiveOperation`. | -| AC4.1 | `internal/adapter/executioncontroller/store_lifecycle_test.go`: `TestClientScopedIntentListReturnsAttestedOwnerOnlyToOwningClient`; Kind isolation and pending-delete tests. Scoped-admin exception needs new positive/negative proofs. | +| AC4.1 | `internal/adapter/executioncontroller/store_lifecycle_test.go`: `TestClientScopedIntentListReturnsAttestedOwnerOnlyToOwningClient`; Kind isolation and pending-delete tests; scoped-admin non-escalation is exercised by the AC4.4 tests. | | AC4.2 | `internal/adapter/executioncontroller/store_revoke_test.go`: `TestRevokeEnvironmentFencesOldClaimWithoutChangingExecutionEpoch`; `store_concurrency_test.go`: `TestAcquireRunAndRevokeUseResourceVersionCAS`; production security rotation. | | AC4.3 | `internal/adapter/executioncontroller/production_lifecycle_test.go`: `TestExpiredOperationLeaseFencesWithoutClearingIdentity`, `TestRecoverMissingPodRemainsFenceUnknown`, `TestRecoveredTerminalProofCanStartExactReplacement`; production holder loss. | -| AC4.4–AC4.5 | **PENDING** all-admin-path scope matrix, data-plane non-escalation, immutable creator/owner and replay checks, strict decoder/digest tests, scope removal on open connections, and Kind distinct-admin qualification. | +| AC4.4 | `internal/adapter/executioncontroller/admin_scope_test.go`: `TestScopedAdminAllRoutesOverMTLS`, `TestScopedAdminCASRetryRechecksSubject`, `TestScopedAdminMigrationReceiptRetainsUIDPreconditions`, `TestMigrationCompletedCASReplayRequiresExactSourceSchema`, `TestMigrationReceiptExpiresAfterReconciledReplacement`, `TestScopedAdminDoesNotGrantAttestationOrDataPlane`, `TestScopedAdminWithOwnerAttestationCannotUseAnotherCreatorsDataPlane`. Replacement-expiry test is in `internal/adapter/executioncontroller/production_lifecycle_test.go` in the targeted repair, not the pinned baseline. `e2e/k8s_execution/ab_production_admin_test.go`: `TestKindExecutionProductionScopedAdministrator` exists (replacement/non-escalation, not all six routes); final Kind evidence PENDING. | +| AC4.5 | `internal/adapter/executioncontroller/admin_scope_security_test.go`: `TestAdministratorScopeManifestValidation`, `TestAdministratorScopeDigestIsNormalizedAndAuthorityBound`; `internal/adapter/executioncontroller/admin_scope_test.go`: all-routes scope removal plus `TestScopedAdminEqualGenerationDriftFailsClosed`. Compatibility/upgrade human review PENDING. | | AC5.1 | `internal/adapter/executionclient/client_test.go`: exact reattachment test above; production `TestKindExecutionProductionReplicaLifecycle` and security rotation. | | AC5.2 | `internal/adapter/executioncontroller/production_lifecycle_test.go`: `TestReplacementPersistsTerminalProofBeforeRemovingPodFinalizer`; production replica lifecycle. | | AC5.3 | `internal/adapter/executioncontroller/production_lifecycle_test.go`: `TestReplacementQuiescesAndPendingReferenceBlocksRetirement`. | | AC5.4 | `internal/adapter/executionclient/reference_ambiguity_test.go`: `TestAmbiguousCommitIsRetainedAndNeverAbortedByCleanup`; production `TestKindExecutionProductionPendingDeleteOutageRecovery`. | -| AC5.5, AC5.7 | `deploy/helm/mecatl-execution/chart_test.go`: `TestChartHasNoDeletionHook` is static partial coverage only. **PENDING** actual Helm lifecycle, retained network isolation, security/capacity-ledger survival, ownership adoption, and missing-history rejection. | +| AC5.5, AC5.7 | `deploy/helm/mecatl-execution/chart_test.go`: `TestChartHasNoDeletionHook`; `deploy/helm/mecatl-execution/lifetime_test.go`: `TestChartRetainedLifetime` exercises lookup/adoption, missing history, foreign ownership, profile drift, and retained policies/ledgers offline. `e2e/k8s_execution/zz_production_helm_lifetime_test.go`: `TestKindExecutionProductionHelmLifetime` exercises actual upgrade/uninstall/reinstall and enforcement; final runtime evidence PENDING. | | AC5.6 | Deployment-documentation inspection and docs gates required on the final implementation; Kind is restart proof only. | | AC6.1–AC6.2 | `internal/app/remote_execution_test.go`: `TestRemoteExecutionRealFactoryCarriesPostureAndAttenuatedCatalog`, `TestRemoteDeploymentNoFSUsesLocalAttenuationWithoutProviderCall`. | | AC6.3 | `internal/adapter/executioncontroller/store_lifecycle_test.go`: `TestReferenceTransactionsRetainUnknownAndNeverChangeSource`; production `TestKindExecutionProductionClearForkLifecycle`. | @@ -455,7 +462,7 @@ plan worktree to make citations or trace appear complete. 1. Human API/schema/security review approves the authored interface and amendment requirements, and the release maturity designation is explicit. Both PRs remain draft with no approved baseline until human contract approval; authorized pre-merge development does not waive that gate. 2. The final implementation candidate passes `task lint`, full offline `task test` including race and engine-standalone coverage, `task build`, `task api:check`, `task docs`, `task site:build`, and `go run ./cmd/mecademo`. A skipped draft race lane is not a pass. Generated contracts/references must be fresh; no arbitrary prose-pinning tests. -3. `task ac-trace-strict` resolves every final `verify:` proof, including new AC2.4, AC4.4–AC4.5, and AC5.7. Replace provisional labels with executable proofs covering the whole AC; the evidence map alone is not strict trace completion. +3. `task ac-trace-strict` resolves every final `verify:` proof, including AC2.4, AC4.4–AC4.5, and AC5.7. Resolve the MISSING/partial behavioral coverage noted above; named-test resolution alone is not AC completion. 4. Fresh deterministic Kind+Calico production qualification passes at the final candidate, including the expanded scope/retry/Helm lifecycle matrix. Then record successful stack-specific native-provider OpenRouter coding qualification under AC7.4–AC7.5. Neither historical deterministic success nor generic mecak8s live CI substitutes for those final results. 5. Complete independent Spec/Standards/Test adequacy/Domain panel review and resolve merge-blocking findings. Update the owning `user-docs/building/deployment/mecak8s.md` and implementation's `user-docs/features/execution-environments.md`, plus resource/rehydration inventories, in the same implementation stack; keep this plan branch documentation-only. 6. The existing implementation PR targets `plan/native-kubernetes-execution`, links plan PR #1579 and its exact working commit, reports conformance/amendments and final evidence. Human alone approves and merges; no new PR, automatic merge, or unsupported landed claim. @@ -466,4 +473,4 @@ plan worktree to make citations or trace appear complete. - A PVC can preserve files while a stale command still mutates them. Session leasing does not fence command execution; missing old-executor evidence remains a fail-closed manual recovery boundary. - Only an enforcing CNI supplies negative NetworkPolicy proof. Retaining storage while uninstall removes isolation or authority history is unsafe; the Helm lifecycle amendment is a completion requirement, not deferred polish. - Every outlives-a-call resource, including retained security/capacity ledgers, reload state, informer/cache, connection pools, claims, and lifecycle/reference reconciliation, needs an accurate resource/rehydration inventory in `docs/adr/0027-cloud-native.md` on the implementation branch. -- The new scope, retry, and Helm behavioral regressions and final native live evidence are pending. Existing source/proof locations absent on the plan branch must stay explicitly implementation-scoped; plan docs gates must not be “fixed” by adding implementation code. +- Scope, retry, and Helm behavioral tests exist in the pinned implementation source; final production Kind/native live evidence remains pending. Broader missing/partial coverage in the verify lines remains an acceptance obligation, not part of the targeted receipt repair. Source/proof locations absent on the plan branch stay explicitly implementation-scoped; do not add implementation code to satisfy plan-branch gates. diff --git a/docs/adr/0350-native-kubernetes-execution.md b/docs/adr/0350-native-kubernetes-execution.md index 773810294b..fe663fbb50 100644 --- a/docs/adr/0350-native-kubernetes-execution.md +++ b/docs/adr/0350-native-kubernetes-execution.md @@ -60,7 +60,9 @@ or Kubernetes names. Creator `clientHash`, owner identity, allocation fingerprin immutable. The [schema and transaction contract](../acceptance/native-kubernetes-execution.md#reference-transaction-schema-v2-and-fencing) records exact fields, receipts, conditions, and migration behavior. Completed migration receipts retain the source schema with explicit presence (including legacy schema zero); replay requires the exact -request identity, and receipts missing that information fail closed. +request identity, and receipts missing that information fail closed. Replacement atomically expires +completed migration receipts when it publishes the new Pod identity; replay with either the original or +replacement UIDs cannot revive an expired receipt. Transient PVC/Pod create errors and quota saturation must recover through continuing rate-limited reconciliation while the CR exists. Quota restoration alone must be sufficient; no unrelated reference From 9f3d287ddfc32f3e4c7508cba463cb6ecd58a3e0 Mon Sep 17 00:00:00 2001 From: Juan Antonio Osorio Date: Mon, 21 Sep 2026 23:11:33 +0300 Subject: [PATCH 10/16] docs: map native execution coverage to verified candidate Co-Authored-By: OpenAI --- .../acceptance/native-kubernetes-execution.md | 89 +++++++++++-------- 1 file changed, 54 insertions(+), 35 deletions(-) diff --git a/docs/acceptance/native-kubernetes-execution.md b/docs/acceptance/native-kubernetes-execution.md index b911e574d1..e62deb735f 100644 --- a/docs/acceptance/native-kubernetes-execution.md +++ b/docs/acceptance/native-kubernetes-execution.md @@ -88,7 +88,7 @@ reattachment never substitutes a current default or Pod UID. Sanitized errors carry `ErrorDetail{code, retryable}`. Stable mappings are `invalid_argument`→InvalidArgument, `unauthenticated`→Unauthenticated, `permission_denied`→PermissionDenied, `not_found`→NotFound, -`already_exists`→AlreadyExists, `conflict`/`version_mismatch`→Aborted, `not_ready`→Unavailable, +`already_exists`→AlreadyExists, `conflict`/`version_mismatch`/`directory_not_empty`→Aborted, `not_ready`→Unavailable, `fence_unknown`→FailedPrecondition, `resource_exhausted`→ResourceExhausted, and `internal`→Internal. Only `not_ready` and `unauthenticated` can carry `retryable:true`; this is not permission to retry a mutation automatically. Grant-expiry refresh is read-only. Backend details never cross the boundary; @@ -262,18 +262,18 @@ ledgers and retained isolation/configuration in the resource and rehydration led ### Scenario 1 — Disabled means absent, including startup validation The optional deployment boundary follows [draft ADR 0350](../adr/0350-native-kubernetes-execution.md). -Current startup provider validation reaches `Bind` (`docs/design/IMPLEMENTATION-NOTES.md:6549-6560`), -so execution configuration needs a side-effect-free validation path rather than a fake allocation. +Startup preflight uses side-effect-free `ValidatePlacement`/`ValidateProfile`, not `Bind` +or a throwaway allocation. CLI and real composition tests below exercise that boundary. **Acceptance:** - AC1.1: With execution disabled, mecak8s creates no Kubernetes execution client, informer, goroutine, CRD, RBAC, controller, executor, PVC, or Pod, and performs no execution API call. - - verify: `TestRemoteDeploymentNoFSUsesLocalAttenuationWithoutProviderCall`; inspection — full disabled-startup side-effect proof is MISSING (the no-fs test is partial, not disabled integration). + - verify: `TestDisabledExecutionStartupDoesNotContactExecutionOrKubernetes`, `TestDisabledCompositionPreservesAllocationsAndIndependentReconciliation`; offline CLI `run()` reaches listener setup with poison execution TLS paths and a test-only kubeconfig/API endpoint untouched. Real composition runs and closes without provider-store actions; constructor/informer absence also follows from the disabled CLI branch and independent provider packaging (not goroutine counts). - AC1.2: The provider service deploys, restarts, and reconciles independently of mecak8s. The default mecak8s chart has no dependency on the provider chart and installs no provider resource; enabling the client adapter requires explicit endpoint/profile configuration. The adapter needs no Pod/PVC/controller management RBAC. - verify: `TestChartRetainsCRDAndDoesNotGrantSecretAPI`, `TestKindExecutionProductionReplicaLifecycle`; inspection — default chart dependency/RBAC review and final Kind execution PENDING. - AC1.3: Preflight validates configured profiles and endpoint/provider compatibility without allocating an environment. Allocation occurs only at an authorized session-binding operation; enabled configuration that is incomplete fails startup before allocation, and a configured but unavailable endpoint fails validation clearly with no local fallback. - - verify: `TestHandlerValidateIsReadOnlyAndEnsureIdempotent`, `TestGRPCReadinessAndMTLSAreMandatory`, `TestKindExecutionQualification`; inspection — incomplete enabled-config startup-before-allocation coverage remains MISSING; final Kind run PENDING. + - verify: `TestExecutionPreflightRejectsIncompleteAndIncompatibleCLI`, `TestRemoteExecutionPreflightNeverAllocates`, `TestHandlerValidateIsReadOnlyAndEnsureIdempotent`, `TestGRPCReadinessAndMTLSAreMandatory`, `TestKindExecutionQualification`; offline CLI/composition and signed-handler validation cover rejection without allocation; final Kind run PENDING. - AC1.4: Disabling the mecak8s client integration neither adopts nor deletes existing `ExecutionEnvironment` CRs, PVCs, or Pods and does not stop the separate provider's reconciliation for its existing allocations or other authorized clients. - - verify: `TestStartupDoesNotFenceLivePeerOperation`; inspection — peer restart is partial coverage; disabled-client retained-resource lifecycle proof remains MISSING. + - verify: `TestDisabledCompositionPreservesAllocationsAndIndependentReconciliation`, `TestStartupDoesNotFenceLivePeerOperation`; real disabled Build/run/Close leaves test-owned CR/Pod/PVC state unchanged, then the separate reconciler advances the existing allocation to Ready. This is offline lifecycle evidence, not a cluster restart run. ### Scenario 2 — One idempotent logical environment is allocated per binding @@ -298,11 +298,11 @@ The adapter preserves the immutable environment and independent read ledger **Acceptance:** - AC3.1: A fresh environment can create, read, conditionally edit, replace, copy, move, remove, list, grep, and glob a small Go fixture with unchanged tool schemas and the existing read-before-edit/version behavior where that contract applies. - - verify: `TestFileOperationsAreConfinedAndVersioned`, `TestProviderThroughRealGRPCSignedHandlerRefreshesAndReattachesExactly`, `TestKindExecutionQualification`; inspection — full native per-tool positive conformance matrix remains MISSING; final Kind execution PENDING. + - verify: `TestRemoteWorkspaceConformanceThroughSignedGRPC`, `TestRemoteFileToolsPreserveLedgerAndNamespaceContracts`, `TestKindExecutionQualification`; both shared Workspace conformance suites and all nine actual file tools run over loopback mTLS/signed gRPC with the real executor in a test-owned directory. Kind invokes the added remote-tool matrix, but final Kind execution remains PENDING. - AC3.2: Read/Edit/Write preserve their applicable recorded-read and CreateFile/ReplaceFile CAS contracts; Copy/Move/Remove preserve their own positive conformance contracts and do not require an unrelated content read ledger/CAS precondition. - - verify: `TestFileOperationsAreConfinedAndVersioned`, `TestRenameAndCopyNeverClobber`; inspection — native recorded-read and positive Copy/Move/Remove conformance remain MISSING (negative clobber tests are not positive conformance). + - verify: `TestRemoteWorkspaceConformanceThroughSignedGRPC`, `TestRemoteFileToolsPreserveLedgerAndNamespaceContracts`; shared tests cover create/replace races, stale/zero versions, namespace positives and non-clobbering failures; actual tools cover unread/stale Edit/Write refusal and successful replacement. Namespace operations have no unrelated content-read precondition. - AC3.3: File access is physically confined through symlink traversal rather than lexical checks alone, and every supported read/mutate/list/search/command request is authorized for the exact environment; stale, expired, revoked, or wrong-environment grants are denied. Detached status/cancel requests authorize before returning Unimplemented; this slice exposes no command-stream RPC. - - verify: `TestFileOperationsAreConfinedAndVersioned`, `TestSecurityManagerGuardsEveryRPCOnExistingConnection`, `TestGrantRoundTripAndExactBindings`, `TestGrantRejectsExpiryRevocationAndUnknownFields`, `TestCommandStatusIsUnimplementedOnlyAfterAuthorization`; inspection — per-file-operation confinement/authorization matrix remains partial. + - verify: `TestEveryFileOperationRequiresExactCurrentGrant`, `TestFileOperationsNeverTraverseExternalSymlink`, `TestRemoteWorkspaceConformanceThroughSignedGRPC`, `TestSecurityManagerGuardsEveryRPCOnExistingConnection`, `TestRevokeEnvironmentFencesOldClaimWithoutChangingExecutionEpoch`, `TestCommandStatusIsUnimplementedOnlyAfterAuthorization`; every supported file operation has a valid positive control before wrong-client/owner/environment/revision/epoch/operation, expired, and revoked-nonce rejection at the shared handler gate. Executor tests cover symlink source/destination traversal and search leakage; durable generation revocation remains covered at the store boundary. - AC3.4: Shell is bound to the same environment namespace as Workspace. Cancellation distinguishes requested cancellation, acknowledged complete process termination, and externally proven compute/storage fencing; namespace teardown includes detached descendants or fails closed. Output is capped and valid UTF-8 repaired, and Shell-created file changes are not falsely claimed to participate in Workspace CAS. - verify: `TestForegroundCommandReturnsReceiptAndTimeoutTerminatesDescendants`, `TestSuccessfulDetachedChildIsKilledBeforeTerminalReceipt`, `TestCancelledStoreOperationStaysActiveUntilBackendStopsThenFences`, `TestKindExecutionProductionHolderLossFencesActiveOperation`; inspection — native output-cap/UTF-8 and Shell-CAS disclosure proof remains partial; final Kind execution PENDING. @@ -320,7 +320,7 @@ therefore requires independent environment execution fencing. - AC4.3: Timeout, Pod deletion, controller restart, Lease timeout, or network partition alone never authorizes a replacement executor while an old writer may run; unknown fencing state fails closed and requires the reviewed manual/external fencing path. - verify: `TestExpiredOperationLeaseFencesWithoutClearingIdentity`, `TestRecoverMissingPodRemainsFenceUnknown`, `TestRecoveredTerminalProofCanStartExactReplacement`, `TestKindExecutionProductionHolderLossFencesActiveOperation` (final Kind execution PENDING); inspection — external fencing runbook review PENDING. - AC4.4: A distinct `administrator:true` client with `administratorFor:[creatorURI]` can administer that creator's environment through all six administrative RPCs, including migration, revocation, and retained deletion, while preserving immutable creator identity and exact owner/revision/epoch/UID/operation checks. A second creator, wrong owner, stale identity, or replay with changed inputs is denied without existence disclosure. Self-admin compatibility remains explicit; no Files/Shell/attach/run/reference or MayAttestOwner authority is gained. - - verify: `TestScopedAdminAllRoutesOverMTLS`, `TestScopedAdminCASRetryRechecksSubject`, `TestScopedAdminMigrationReceiptRetainsUIDPreconditions`, `TestMigrationCompletedCASReplayRequiresExactSourceSchema`, `TestMigrationReceiptExpiresAfterReconciledReplacement`, `TestScopedAdminDoesNotGrantAttestationOrDataPlane`, `TestScopedAdminWithOwnerAttestationCannotUseAnotherCreatorsDataPlane`, `TestKindExecutionProductionScopedAdministrator`; inspection — replacement-expiry regression is in the targeted repair, not the pinned baseline; final Kind execution PENDING (Kind covers replacement/non-escalation, not all six routes). + - verify: `TestScopedAdminAllRoutesOverMTLS`, `TestScopedAdminCASRetryRechecksSubject`, `TestScopedAdminMigrationReceiptRetainsUIDPreconditions`, `TestMigrationCompletedCASReplayRequiresExactSourceSchema`, `TestMigrationReceiptExpiresAfterReconciledReplacement`, `TestScopedAdminDoesNotGrantAttestationOrDataPlane`, `TestScopedAdminWithOwnerAttestationCannotUseAnotherCreatorsDataPlane`, `TestKindExecutionProductionScopedAdministrator`; inspection — replacement-expiry regression is present in the implementation candidate; final Kind execution PENDING (Kind covers replacement/non-escalation, not all six routes). - AC4.5: Strict `administratorFor` validation and canonical digest inclusion prevent ambiguous scope. Removal at a higher security generation takes effect on existing connections and receipt replay; an equal-generation scope edit fails closed. Old binaries reject the field and the documented quiesced upgrade preserves authority history. - verify: `TestAdministratorScopeManifestValidation`, `TestAdministratorScopeDigestIsNormalizedAndAuthorityBound`, `TestScopedAdminAllRoutesOverMTLS`, `TestScopedAdminEqualGenerationDriftFailsClosed`; inspection — old-binary rejection/quiesced-upgrade compatibility review PENDING. @@ -353,9 +353,9 @@ in local composition (`internal/app/project_ingestion.go:20`) and AGENTS discove **Acceptance:** - AC6.1: A remote execution session receives operator-global instructions and only its explicitly supported catalog; project AGENTS/rules/skills and Git source ingestion are off unless a later reviewed source/trust contract enables them. - - verify: `TestRemoteExecutionRealFactoryCarriesPostureAndAttenuatedCatalog`; inspection — the test pins host AGENTS exclusion/catalog posture; complete operator-global and remote rules/skills/Git source matrix remains partial. + - verify: `TestRemoteExecutionRealFactoryCarriesPostureAndAttenuatedCatalog`; real factory invocation plants host and remote AGENTS, host rules/skills/commands, invokes the forbidden project skill, and proves operator-global rules/skills survive. Removing remote admission makes the test fail on leaked rules and skill body. Git snapshot exclusion uses the same admission seam; a dedicated native Git fixture is still partial evidence. - AC6.2: Schedules remain excluded. Background commands, isolated Subagent/Parallel/Team delegation, SkillDraft, Git/source operations, and unsupported successor modes are rejected with named capability/precondition errors rather than running locally or silently omitting work; `no-fs` remains unchanged. - - verify: `TestRemoteExecutionRealFactoryCarriesPostureAndAttenuatedCatalog`, `TestRemoteDeploymentNoFSUsesLocalAttenuationWithoutProviderCall`, `TestCommandStatusIsUnimplementedOnlyAfterAuthorization`; inspection — full unsupported-path invocation/error matrix remains partial. + - verify: `TestRemoteExecutionRealFactoryCarriesPostureAndAttenuatedCatalog`, `TestRemoteDeploymentNoFSUsesLocalAttenuationWithoutProviderCall`, `TestExecutionPreflightRejectsIncompleteAndIncompatibleCLI`, `TestServiceUsesRealMTLSProviderStoreAndReleasesOnlyAfterDrain`, `TestCommandStatusIsUnimplementedOnlyAfterAuthorization`; real factory tests invoke Subagent/Parallel/Team/SkillDraft/Schedule and require model-visible errors, reject worktree discovery, and withhold command listing/expansion instead of opening host paths. The native mTLS/store integration rejects background Shell with its named runner-capability error and no executor dispatch, while foreground Shell succeeds. Unsupported-successor invocation coverage remains partial; do not infer it from catalog absence. - AC6.3: Clear/Fork use environment exclusion and reference reservation/publication while preserving same-remote-`EnvironmentRef` successor semantics and the source on publication failure. Clear/Fork are not workspace cloning; no public Harness field is added and no model chooses an environment/profile. - verify: `TestReferenceTransactionsRetainUnknownAndNeverChangeSource`, `TestKindExecutionProductionClearForkLifecycle` (final Kind execution PENDING); inspection — public API/profile authority review PENDING. @@ -368,7 +368,7 @@ live suite is not native-provider proof. The focused mock task must not inherit **Acceptance:** - AC7.1: `task e2e:k8s:execution` and the enforcing-CNI `task e2e:k8s:execution:production` run on the supported generic toolchain, including CI, with a unique owned kind cluster, a scratch kubeconfig, and explicit `--kubeconfig`/context on every command. They deploy the separate chart plus explicitly connected mecak8s and use only the deterministic mock provider. The final production-profile run uses Calico and exercises negative network isolation, rotation/revocation, replica/lifecycle recovery, holder loss, quota recovery, pending delete, Clear/Fork, migration, sanitized artifacts, and the amendment regressions. - - verify: `TestKindExecutionQualification`, `TestKindExecutionProductionNetworkPolicyEnforced`, `TestKindExecutionProductionSecurityRotation`, `TestKindExecutionProductionReplicaLifecycle`, `TestKindExecutionProductionHolderLossFencesActiveOperation`, `TestKindExecutionProductionQuotaSaturation`, `TestKindExecutionProductionPendingDeleteOutageRecovery`, `TestKindExecutionProductionClearForkLifecycle`, `TestKindExecutionProductionCompatiblePrototypeMigration`, `TestKindExecutionProductionScopedAdministrator`, `TestKindExecutionProductionHelmLifetime` (names exist; final production Kind+Calico run PENDING). + - verify: `TestLegacyFixtureWaitsForQuotaAccountingBeforeCreate`, `TestKindExecutionQualification`, `TestKindExecutionProductionNetworkPolicyEnforced`, `TestKindExecutionProductionSecurityRotation`, `TestKindExecutionProductionReplicaLifecycle`, `TestKindExecutionProductionHolderLossFencesActiveOperation`, `TestKindExecutionProductionQuotaSaturation`, `TestKindExecutionProductionPendingDeleteOutageRecovery`, `TestKindExecutionProductionClearForkLifecycle`, `TestKindExecutionProductionCompatiblePrototypeMigration`, `TestKindExecutionProductionScopedAdministrator`, `TestKindExecutionProductionHelmLifetime` (names exist; final production Kind+Calico run PENDING). - AC7.2: The mock flow seeds a small Go fixture through file tools, asks the harness to implement a function/tests, runs `go test` through bound Shell, and verifies the persisted artifact and restart/reattach behavior. - verify: `TestKindExecutionQualification` (coding/artifact/restart assertions exist; final Kind execution PENDING). - AC7.3: Failure artifacts are bounded and sanitized; cleanup targets only the recorded owned cluster. Destructive live cleanup requires confirmation and never mutates ambient kubeconfig/context. @@ -381,18 +381,26 @@ live suite is not native-provider proof. The focused mock task must not inherit ## Implementation proof map Source/test locations in this table and the `verify:` lines refer to implementation commit -`9bb4d89b8aa497c9d2cd6aa4a94ab9262a913364`, not files supplied by this plan branch. -The labels name existing tests, not successful executions or complete behavioral coverage. -MISSING/partial annotations preserve broader proof gaps outside the targeted receipt repair; -PENDING runtime evidence means the test exists but still needs final-candidate execution. +`0df361480b2b9c1b9a02ca5c7f55362d0e257966`, not files supplied by this plan branch. +This is the intended candidate for the normal merge of this proof-map update into implementation. +The labels name actual tests; source resolution alone is not successful execution or complete +behavioral coverage. Offline boundaries and remaining partial proof are stated per AC; +PENDING runtime evidence still requires final-candidate execution. The migration-receipt replacement-expiry regression `TestMigrationReceiptExpiresAfterReconciledReplacement` -is in the targeted repair's `internal/adapter/executioncontroller/production_lifecycle_test.go`, not this -source baseline; verify it against the implementation candidate containing this plan merge. +is already present in that candidate's `internal/adapter/executioncontroller/production_lifecycle_test.go`. Run `actrace --strict --plan ` from the implementation worktree for source resolution, then rerun after its normal plan merge. Draft status is intentional: strict mode gates landed plans only, so inspect the native report for missing proofs as well. A row naming a unit/fake-client test establishes only that boundary; it does not prove the entire AC. +**Offline candidate verification:** the implementation content pinned above passed targeted race +packages (execution client/controller/executor/protocol, mecak8s, and remote composition), the +build-tagged quota test, compilation of the Kind qualification and fixture packages without running +Kind, full `task lint`, full `task test` (including race and standalone module checks), `task build`, +`task docs`, `task site:build`, and the offline demo. Local logs are under the implementation's ignored +`.scratch/orchestrate/native-kubernetes-execution/coverage-*.log`. Source conformance does not qualify +Kubernetes admission, CNI enforcement, or live-model behavior; those remain the runtime gates below. + **Observed qualification:** the [production Kind+Calico job](https://github.com/stacklok/mecatl/actions/runs/35589357385/job/106299991747) completed **SUCCESS** at historical implementation `deaf1c3d1dfe7ea92afc8fe826f1bc613080f219`. This includes network isolation, rotation/revocation, replicas/lifecycle, holder loss, quota saturation, pending-delete recovery, @@ -400,31 +408,42 @@ Clear/Fork, compatible prototype migration, and artifact isolation. Historical l failures are not active blockers for that run. This is baseline evidence, not final-candidate evidence for the new scoped-admin, continuing-retry, or Helm uninstall/reinstall requirements. -Draft CI skipped race lanes; non-race success is not a race result. The generic live compaction lane -failure is addressed by [merged PR #1728](https://github.com/stacklok/mecatl/pull/1728) on main, not a -native-provider defect or proof that its fix is present in this draft. Generic mecak8s live success is -also not AC7.4 evidence. Final-candidate native-provider live and amended production Kind qualification -remain **PENDING**. Source resolution below does not assert new-head CI, final live execution, or human -contract/panel approval. +Draft CI skipped race lanes; non-race success is not a race result. The newer +[production Kind+Calico job](https://github.com/stacklok/mecatl/actions/runs/35642635684/job/106475375010) +at `9bb4d89b8aa497c9d2cd6aa4a94ab9262a913364` failed **before qualification tests** while +seeding `legacy-migration`: ResourceQuota accounting for +`count/executionenvironments.execution.mecatl.dev` was still unknown. Provider Deployment +readiness was not quota readiness. This supersedes a test-race diagnosis for that job. +The implementation adds a two-minute, context-aware quota-accounting wait with delayed-status, +timeout, cancellation, and forbidden-response controls in +`TestLegacyFixtureWaitsForQuotaAccountingBeforeCreate`. It does not disable quota or retry +arbitrary forbidden creates. Those offline controls and tagged compilation are not a rerun +of the production job. + +[PR #1728](https://github.com/stacklok/mecatl/pull/1728), commit +`6501b5924`, is already integrated in the implementation ancestry. Its generic live-compaction +repair is not native-provider qualification. Final-candidate native-provider live and amended +production Kind qualification remain **PENDING**. Source resolution below does not assert +new-head CI, final live execution, or human contract/panel approval. | AC | Existing implementation proof at the reference commit; remaining evidence | |---|---| -| AC1.1 | `internal/app/remote_execution_test.go`: `TestRemoteDeploymentNoFSUsesLocalAttenuationWithoutProviderCall`; full disabled-effects obligation remains. | +| AC1.1 | `cmd/mecak8s/execution_startup_test.go`: `TestDisabledExecutionStartupDoesNotContactExecutionOrKubernetes` exercises CLI `run()` through composition to listener setup with poison TLS paths and no test-endpoint calls. `internal/adapter/executioncontroller/disabled_composition_test.go` exercises disabled Build/run/Close beside retained provider resources. Constructor/informer absence is also inspected at the CLI enablement branch; no flaky goroutine census. | | AC1.2 | `deploy/helm/mecatl-execution/chart_test.go`: `TestChartRetainsCRDAndDoesNotGrantSecretAPI`; production job separately deploys provider and client. | -| AC1.3 | `internal/adapter/executioncontroller/handler_test.go`: `TestHandlerValidateIsReadOnlyAndEnsureIdempotent`; `controller_test.go`: `TestInitializeRefusesMissingRuntimeClassBeforeCreatingPods`. | -| AC1.4 | `internal/adapter/executioncontroller/controller_test.go`: `TestStartupDoesNotFenceLivePeerOperation`; this is peer-restart evidence, not a complete disabled-client lifecycle proof. | +| AC1.3 | `cmd/mecak8s/execution_startup_test.go`: `TestExecutionPreflightRejectsIncompleteAndIncompatibleCLI`; `internal/app/remote_execution_test.go`: `TestRemoteExecutionPreflightNeverAllocates`; handler read-only validation and mTLS/readiness tests cover the transport. | +| AC1.4 | `internal/adapter/executioncontroller/disabled_composition_test.go`: `TestDisabledCompositionPreservesAllocationsAndIndependentReconciliation` compares retained objects across disabled Build/run/Close, then observes independent reconciliation. `controller_test.go`: peer-restart test remains complementary. | | AC2.1 | `internal/adapter/executioncontroller/handler_test.go`: `TestHandlerValidateIsReadOnlyAndEnsureIdempotent`; `e2e/k8s_execution/qualification_test.go`: `TestKindExecutionQualification`. | | AC2.2 | `internal/adapter/executioncontroller/store_test.go`: `TestStoreEnsureUsesStableLookupAndRejectsFingerprintDrift`; `controller_test.go`: `TestReconcileRefusesForeignExistingPVCWithoutPersistingUID`. | | AC2.3 | `internal/adapter/executioncontroller/profiles_test.go`: `TestLoadProfilesStrictAndDigestPinned`; production `TestKindExecutionProductionQuotaSaturation`. | | AC2.4 | `internal/adapter/executioncontroller/provisioning_retry_test.go`: `TestProvisioningWorkerConvergesWithoutAnotherEvent`, `TestProvisioningPreservesCreateAndConditionErrors`, `TestProvisioningRetriesNeverReplaceAuthoritativeResources` exercise PVC and Pod retry/confinement paths. Production `TestKindExecutionProductionQuotaSaturation` restores quota without a new reference event; final runtime evidence PENDING. | -| AC3.1 | `internal/adapter/executionclient/client_test.go`: `TestProviderThroughRealGRPCSignedHandlerRefreshesAndReattachesExactly`; Kind coding flow succeeds, but full per-tool conformance matrix remains to be traced/qualified. | -| AC3.2 | `internal/executionexecutor/executor_test.go`: `TestFileOperationsAreConfinedAndVersioned`, `TestRenameAndCopyNeverClobber`; native recorded-read and positive Copy/Move/Remove conformance remain MISSING. | -| AC3.3 | `internal/executionexecutor/executor_test.go`: confinement/version tests; `internal/executionenv/protocol_test.go`: exact binding/expiry/revocation tests; `internal/adapter/executionclient/client_test.go`: authorized detached-control rejection; `internal/adapter/executioncontroller/security_test.go`: `TestSecurityManagerGuardsEveryRPCOnExistingConnection`. Per-file-operation matrix remains partial. | +| AC3.1 | `internal/adapter/executionclient/files_conformance_test.go`: `TestRemoteWorkspaceConformanceThroughSignedGRPC` runs both existing shared suites; `TestRemoteFileToolsPreserveLedgerAndNamespaceContracts` invokes all nine tool bodies against the signed handler and real executor. `e2e/k8s_execution/file_tools_test.go` adds the tool matrix called by Kind qualification; runtime execution PENDING. | +| AC3.2 | Same gRPC suites cover concurrent create/CAS, missing/stale/zero versions, non-clobbering namespace operations, and actual Read-ledger behavior. The suite exposed and now guards missing-file replacement precedence and typed non-empty-directory removal errors. | +| AC3.3 | `internal/adapter/executioncontroller/file_authorization_test.go`: every supported file operation has positive and negative signed-grant controls; `internal/executionexecutor/confinement_test.go`: physical symlink traversal/search checks. Existing security manager, protocol, durable revocation, and detached-control tests remain complementary. | | AC3.4 | `internal/adapter/executioncontroller/store_test.go`: `TestCancelledStoreOperationStaysActiveUntilBackendStopsThenFences`; production `TestKindExecutionProductionHolderLossFencesActiveOperation`. | | AC4.1 | `internal/adapter/executioncontroller/store_lifecycle_test.go`: `TestClientScopedIntentListReturnsAttestedOwnerOnlyToOwningClient`; Kind isolation and pending-delete tests; scoped-admin non-escalation is exercised by the AC4.4 tests. | | AC4.2 | `internal/adapter/executioncontroller/store_revoke_test.go`: `TestRevokeEnvironmentFencesOldClaimWithoutChangingExecutionEpoch`; `store_concurrency_test.go`: `TestAcquireRunAndRevokeUseResourceVersionCAS`; production security rotation. | | AC4.3 | `internal/adapter/executioncontroller/production_lifecycle_test.go`: `TestExpiredOperationLeaseFencesWithoutClearingIdentity`, `TestRecoverMissingPodRemainsFenceUnknown`, `TestRecoveredTerminalProofCanStartExactReplacement`; production holder loss. | -| AC4.4 | `internal/adapter/executioncontroller/admin_scope_test.go`: `TestScopedAdminAllRoutesOverMTLS`, `TestScopedAdminCASRetryRechecksSubject`, `TestScopedAdminMigrationReceiptRetainsUIDPreconditions`, `TestMigrationCompletedCASReplayRequiresExactSourceSchema`, `TestMigrationReceiptExpiresAfterReconciledReplacement`, `TestScopedAdminDoesNotGrantAttestationOrDataPlane`, `TestScopedAdminWithOwnerAttestationCannotUseAnotherCreatorsDataPlane`. Replacement-expiry test is in `internal/adapter/executioncontroller/production_lifecycle_test.go` in the targeted repair, not the pinned baseline. `e2e/k8s_execution/ab_production_admin_test.go`: `TestKindExecutionProductionScopedAdministrator` exists (replacement/non-escalation, not all six routes); final Kind evidence PENDING. | +| AC4.4 | `internal/adapter/executioncontroller/admin_scope_test.go`: `TestScopedAdminAllRoutesOverMTLS`, `TestScopedAdminCASRetryRechecksSubject`, `TestScopedAdminMigrationReceiptRetainsUIDPreconditions`, `TestMigrationCompletedCASReplayRequiresExactSourceSchema`, `TestMigrationReceiptExpiresAfterReconciledReplacement`, `TestScopedAdminDoesNotGrantAttestationOrDataPlane`, `TestScopedAdminWithOwnerAttestationCannotUseAnotherCreatorsDataPlane`. Replacement-expiry test is in `internal/adapter/executioncontroller/production_lifecycle_test.go` in the pinned implementation candidate. `e2e/k8s_execution/ab_production_admin_test.go`: `TestKindExecutionProductionScopedAdministrator` exists (replacement/non-escalation, not all six routes); final Kind evidence PENDING. | | AC4.5 | `internal/adapter/executioncontroller/admin_scope_security_test.go`: `TestAdministratorScopeManifestValidation`, `TestAdministratorScopeDigestIsNormalizedAndAuthorityBound`; `internal/adapter/executioncontroller/admin_scope_test.go`: all-routes scope removal plus `TestScopedAdminEqualGenerationDriftFailsClosed`. Compatibility/upgrade human review PENDING. | | AC5.1 | `internal/adapter/executionclient/client_test.go`: exact reattachment test above; production `TestKindExecutionProductionReplicaLifecycle` and security rotation. | | AC5.2 | `internal/adapter/executioncontroller/production_lifecycle_test.go`: `TestReplacementPersistsTerminalProofBeforeRemovingPodFinalizer`; production replica lifecycle. | @@ -432,9 +451,9 @@ contract/panel approval. | AC5.4 | `internal/adapter/executionclient/reference_ambiguity_test.go`: `TestAmbiguousCommitIsRetainedAndNeverAbortedByCleanup`; production `TestKindExecutionProductionPendingDeleteOutageRecovery`. | | AC5.5, AC5.7 | `deploy/helm/mecatl-execution/chart_test.go`: `TestChartHasNoDeletionHook`; `deploy/helm/mecatl-execution/lifetime_test.go`: `TestChartRetainedLifetime` exercises lookup/adoption, missing history, foreign ownership, profile drift, and retained policies/ledgers offline. `e2e/k8s_execution/zz_production_helm_lifetime_test.go`: `TestKindExecutionProductionHelmLifetime` exercises actual upgrade/uninstall/reinstall and enforcement; final runtime evidence PENDING. | | AC5.6 | Deployment-documentation inspection and docs gates required on the final implementation; Kind is restart proof only. | -| AC6.1–AC6.2 | `internal/app/remote_execution_test.go`: `TestRemoteExecutionRealFactoryCarriesPostureAndAttenuatedCatalog`, `TestRemoteDeploymentNoFSUsesLocalAttenuationWithoutProviderCall`. | +| AC6.1–AC6.2 | `internal/app/remote_execution_test.go`: real factory poison-source and operator-global positive controls, forbidden-tool invocation errors, command/worktree discovery, and unchanged no-FS catalog/no-allocation checks. Native background Shell rejection is exercised in `internal/adapter/executionclient/service_integration_test.go` alongside a successful foreground command. Native Git snapshot and unsupported-successor invocation evidence remains partial; the shared admission gate and catalog inspection alone do not close that matrix. | | AC6.3 | `internal/adapter/executioncontroller/store_lifecycle_test.go`: `TestReferenceTransactionsRetainUnknownAndNeverChangeSource`; production `TestKindExecutionProductionClearForkLifecycle`. | -| AC7.1 | `.github/workflows/k8s-e2e.yml` runs `task e2e:k8s:execution:production`; the reference job succeeded. `e2e/k8s_execution/aa_production_migration_test.go`: `TestKindExecutionProductionCompatiblePrototypeMigration`. Final amended-candidate run **PENDING**. | +| AC7.1 | `.github/workflows/k8s-e2e.yml` runs `task e2e:k8s:execution:production`; the historical reference job succeeded, but the newer job failed during quota-dependent fixture setup before tests. `internal/adapter/executioncontroller/legacy_fixture_kind_test.go` covers the bounded accounting wait offline. `e2e/k8s_execution/aa_production_migration_test.go`: `TestKindExecutionProductionCompatiblePrototypeMigration`. Final amended-candidate run **PENDING**. | | AC7.2 | `e2e/k8s_execution/qualification_test.go`: `TestKindExecutionQualification`, run by the successful production profile. | | AC7.3 | Production `TestKindExecutionProductionFailureArtifactBoundary`; workflow bounded uploads and ownership-scoped cleanup. | | AC7.4 | `e2e/k8s_execution/live_qualification_test.go`: `TestKindExecutionLiveQualification` exists; successful final native-provider OpenRouter evidence **PENDING**. | @@ -473,4 +492,4 @@ plan worktree to make citations or trace appear complete. - A PVC can preserve files while a stale command still mutates them. Session leasing does not fence command execution; missing old-executor evidence remains a fail-closed manual recovery boundary. - Only an enforcing CNI supplies negative NetworkPolicy proof. Retaining storage while uninstall removes isolation or authority history is unsafe; the Helm lifecycle amendment is a completion requirement, not deferred polish. - Every outlives-a-call resource, including retained security/capacity ledgers, reload state, informer/cache, connection pools, claims, and lifecycle/reference reconciliation, needs an accurate resource/rehydration inventory in `docs/adr/0027-cloud-native.md` on the implementation branch. -- Scope, retry, and Helm behavioral tests exist in the pinned implementation source; final production Kind/native live evidence remains pending. Broader missing/partial coverage in the verify lines remains an acceptance obligation, not part of the targeted receipt repair. Source/proof locations absent on the plan branch stay explicitly implementation-scoped; do not add implementation code to satisfy plan-branch gates. +- Scope, retry, Helm lifecycle, disabled startup, remote filesystem conformance, and source-admission behavioral tests exist in the pinned implementation source. Final production Kind/native live evidence and human reviews remain pending. The remaining partial evidence in AC3.4 and AC6.1–AC6.2 is still an acceptance obligation. Source/proof locations absent on the plan branch stay explicitly implementation-scoped; do not add implementation code to satisfy plan-branch gates. From a81d773c88e98ce8c6555514183e20de2195d5f8 Mon Sep 17 00:00:00 2001 From: Juan Antonio Osorio Date: Mon, 21 Sep 2026 23:44:09 +0300 Subject: [PATCH 11/16] docs: close bounded native execution proofs and clarify evidence limits Co-Authored-By: OpenAI --- .../acceptance/native-kubernetes-execution.md | 39 ++++++++++++------- 1 file changed, 26 insertions(+), 13 deletions(-) diff --git a/docs/acceptance/native-kubernetes-execution.md b/docs/acceptance/native-kubernetes-execution.md index e62deb735f..2efab54065 100644 --- a/docs/acceptance/native-kubernetes-execution.md +++ b/docs/acceptance/native-kubernetes-execution.md @@ -267,13 +267,13 @@ or a throwaway allocation. CLI and real composition tests below exercise that bo **Acceptance:** - AC1.1: With execution disabled, mecak8s creates no Kubernetes execution client, informer, goroutine, CRD, RBAC, controller, executor, PVC, or Pod, and performs no execution API call. - - verify: `TestDisabledExecutionStartupDoesNotContactExecutionOrKubernetes`, `TestDisabledCompositionPreservesAllocationsAndIndependentReconciliation`; offline CLI `run()` reaches listener setup with poison execution TLS paths and a test-only kubeconfig/API endpoint untouched. Real composition runs and closes without provider-store actions; constructor/informer absence also follows from the disabled CLI branch and independent provider packaging (not goroutine counts). + - verify: `TestDisabledExecutionStartupDoesNotContactExecutionOrKubernetes`, `TestDisabledCompositionPreservesAllocationsAndIndependentReconciliation`; offline CLI `run()` reaches listener setup with poison execution TLS paths and a test-only kubeconfig/API endpoint untouched. That poisoned-endpoint test guards disabled startup contact; the lifecycle fixture's fake clients are not injected into app.Build, so its action log proves only independent reconciliation, not absence of an app-created client. Constructor/informer absence also follows from the disabled CLI branch and independent provider packaging (not goroutine counts). - AC1.2: The provider service deploys, restarts, and reconciles independently of mecak8s. The default mecak8s chart has no dependency on the provider chart and installs no provider resource; enabling the client adapter requires explicit endpoint/profile configuration. The adapter needs no Pod/PVC/controller management RBAC. - verify: `TestChartRetainsCRDAndDoesNotGrantSecretAPI`, `TestKindExecutionProductionReplicaLifecycle`; inspection — default chart dependency/RBAC review and final Kind execution PENDING. - AC1.3: Preflight validates configured profiles and endpoint/provider compatibility without allocating an environment. Allocation occurs only at an authorized session-binding operation; enabled configuration that is incomplete fails startup before allocation, and a configured but unavailable endpoint fails validation clearly with no local fallback. - verify: `TestExecutionPreflightRejectsIncompleteAndIncompatibleCLI`, `TestRemoteExecutionPreflightNeverAllocates`, `TestHandlerValidateIsReadOnlyAndEnsureIdempotent`, `TestGRPCReadinessAndMTLSAreMandatory`, `TestKindExecutionQualification`; offline CLI/composition and signed-handler validation cover rejection without allocation; final Kind run PENDING. - AC1.4: Disabling the mecak8s client integration neither adopts nor deletes existing `ExecutionEnvironment` CRs, PVCs, or Pods and does not stop the separate provider's reconciliation for its existing allocations or other authorized clients. - - verify: `TestDisabledCompositionPreservesAllocationsAndIndependentReconciliation`, `TestStartupDoesNotFenceLivePeerOperation`; real disabled Build/run/Close leaves test-owned CR/Pod/PVC state unchanged, then the separate reconciler advances the existing allocation to Ready. This is offline lifecycle evidence, not a cluster restart run. + - verify: `TestDisabledCompositionPreservesAllocationsAndIndependentReconciliation`, `TestStartupDoesNotFenceLivePeerOperation`; the separate reconciler advances its retained allocation to Ready after an unrelated disabled Build/run/Close. Its fake clients are test-owned, not injected into app.Build; unchanged objects/actions are a limited independent-lifecycle oracle, not a client-construction guard or cluster restart run. `TestDisabledExecutionStartupDoesNotContactExecutionOrKubernetes` supplies the actual poisoned-endpoint CLI guard. ### Scenario 2 — One idempotent logical environment is allocated per binding @@ -304,7 +304,7 @@ The adapter preserves the immutable environment and independent read ledger - AC3.3: File access is physically confined through symlink traversal rather than lexical checks alone, and every supported read/mutate/list/search/command request is authorized for the exact environment; stale, expired, revoked, or wrong-environment grants are denied. Detached status/cancel requests authorize before returning Unimplemented; this slice exposes no command-stream RPC. - verify: `TestEveryFileOperationRequiresExactCurrentGrant`, `TestFileOperationsNeverTraverseExternalSymlink`, `TestRemoteWorkspaceConformanceThroughSignedGRPC`, `TestSecurityManagerGuardsEveryRPCOnExistingConnection`, `TestRevokeEnvironmentFencesOldClaimWithoutChangingExecutionEpoch`, `TestCommandStatusIsUnimplementedOnlyAfterAuthorization`; every supported file operation has a valid positive control before wrong-client/owner/environment/revision/epoch/operation, expired, and revoked-nonce rejection at the shared handler gate. Executor tests cover symlink source/destination traversal and search leakage; durable generation revocation remains covered at the store boundary. - AC3.4: Shell is bound to the same environment namespace as Workspace. Cancellation distinguishes requested cancellation, acknowledged complete process termination, and externally proven compute/storage fencing; namespace teardown includes detached descendants or fails closed. Output is capped and valid UTF-8 repaired, and Shell-created file changes are not falsely claimed to participate in Workspace CAS. - - verify: `TestForegroundCommandReturnsReceiptAndTimeoutTerminatesDescendants`, `TestSuccessfulDetachedChildIsKilledBeforeTerminalReceipt`, `TestCancelledStoreOperationStaysActiveUntilBackendStopsThenFences`, `TestKindExecutionProductionHolderLossFencesActiveOperation`; inspection — native output-cap/UTF-8 and Shell-CAS disclosure proof remains partial; final Kind execution PENDING. + - verify: `TestForegroundCommandReturnsReceiptAndTimeoutTerminatesDescendants`, `TestSuccessfulDetachedChildIsKilledBeforeTerminalReceipt`, `TestCancelledStoreOperationStaysActiveUntilBackendStopsThenFences`, `TestNativeShellCapsCombinedOutputAndRepairsUTF8`, `TestNativeShellWriteInvalidatesRecordedRead`, `TestProviderThroughRealGRPCSignedHandlerRefreshesAndReattachesExactly`, `TestServiceUsesRealMTLSProviderStoreAndReleasesOnlyAfterDrain`, `TestKindExecutionProductionHolderLossFencesActiveOperation`; the native helper executes oversized stdout/stderr and split malformed-byte output with combined-cap/truncation assertions, and writes after a recorded Read so actual Edit rejects the stale version without overwriting Shell's content. Private protobuf bytes may remain malformed; signed gRPC runner and model/client-event assertions cover text repair separately. Final Kind execution PENDING. ### Scenario 4 — Caller isolation and execution ownership survive failures @@ -352,10 +352,10 @@ in local composition (`internal/app/project_ingestion.go:20`) and AGENTS discove (`engine/prompt/builder.go:337`). The boundary is recorded in [draft ADR 0350](../adr/0350-native-kubernetes-execution.md). **Acceptance:** -- AC6.1: A remote execution session receives operator-global instructions and only its explicitly supported catalog; project AGENTS/rules/skills and Git source ingestion are off unless a later reviewed source/trust contract enables them. - - verify: `TestRemoteExecutionRealFactoryCarriesPostureAndAttenuatedCatalog`; real factory invocation plants host and remote AGENTS, host rules/skills/commands, invokes the forbidden project skill, and proves operator-global rules/skills survive. Removing remote admission makes the test fail on leaked rules and skill body. Git snapshot exclusion uses the same admission seam; a dedicated native Git fixture is still partial evidence. +- AC6.1: A remote execution session receives operator-global instructions and only its explicitly supported catalog; project AGENTS/rules/skills and Git source ingestion are off unless a later reviewed source/trust contract enables them. Existing operator-global and explicit-file permission policy remains effective, including configured Ask/Deny under auto; these authorization rules are distinct from prompt rules. Project permission sources are excluded, without a new trust decision. + - verify: `TestRemoteExecutionRealFactoryCarriesPostureAndAttenuatedCatalog`, `TestRemoteExecutionPreservesOperatorPermissionsUnderAuto`; real factory invocation plants host and remote AGENTS, host rules/skills/commands, invokes the forbidden project skill, and proves operator-global prompt rules/skills survive. A test-owned Git repository yields a nonempty snapshot in the local factory request and none in the remote request. Under auto, both conventional global and explicit-file Deny/configured Ask survive remote selection, while project Deny/Ask are excluded and allowed Shell controls execute only on the in-memory runner. - AC6.2: Schedules remain excluded. Background commands, isolated Subagent/Parallel/Team delegation, SkillDraft, Git/source operations, and unsupported successor modes are rejected with named capability/precondition errors rather than running locally or silently omitting work; `no-fs` remains unchanged. - - verify: `TestRemoteExecutionRealFactoryCarriesPostureAndAttenuatedCatalog`, `TestRemoteDeploymentNoFSUsesLocalAttenuationWithoutProviderCall`, `TestExecutionPreflightRejectsIncompleteAndIncompatibleCLI`, `TestServiceUsesRealMTLSProviderStoreAndReleasesOnlyAfterDrain`, `TestCommandStatusIsUnimplementedOnlyAfterAuthorization`; real factory tests invoke Subagent/Parallel/Team/SkillDraft/Schedule and require model-visible errors, reject worktree discovery, and withhold command listing/expansion instead of opening host paths. The native mTLS/store integration rejects background Shell with its named runner-capability error and no executor dispatch, while foreground Shell succeeds. Unsupported-successor invocation coverage remains partial; do not infer it from catalog absence. + - verify: `TestRemoteExecutionRealFactoryCarriesPostureAndAttenuatedCatalog`, `TestRemoteDeploymentNoFSUsesLocalAttenuationWithoutProviderCall`, `TestExecutionPreflightRejectsIncompleteAndIncompatibleCLI`, `TestServiceUsesRealMTLSProviderStoreAndReleasesOnlyAfterDrain`, `TestCommandStatusIsUnimplementedOnlyAfterAuthorization`; real factory tests invoke Subagent/Parallel/Team/SkillDraft/Schedule and require model-visible errors, reject worktree discovery, and withhold command listing/expansion instead of opening host paths. The native mTLS/store integration rejects background Shell with its named runner-capability error and no executor dispatch, while foreground Shell succeeds. Actual Service Clear/Fork calls with a nonempty unsupported worktree selector return `ErrInvalidPlacementSelection`, publish no destination, and preserve source ref/history and provider references. A direct native Bind with a valid binding ID distinguishes the unsupported-selector gate from missing binding identity or malformed Service input. - AC6.3: Clear/Fork use environment exclusion and reference reservation/publication while preserving same-remote-`EnvironmentRef` successor semantics and the source on publication failure. Clear/Fork are not workspace cloning; no public Harness field is added and no model chooses an environment/profile. - verify: `TestReferenceTransactionsRetainUnknownAndNeverChangeSource`, `TestKindExecutionProductionClearForkLifecycle` (final Kind execution PENDING); inspection — public API/profile authority review PENDING. @@ -381,7 +381,7 @@ live suite is not native-provider proof. The focused mock task must not inherit ## Implementation proof map Source/test locations in this table and the `verify:` lines refer to implementation commit -`0df361480b2b9c1b9a02ca5c7f55362d0e257966`, not files supplied by this plan branch. +`773a203df5de9b27612d190a774a857590702cbd`, not files supplied by this plan branch. This is the intended candidate for the normal merge of this proof-map update into implementation. The labels name actual tests; source resolution alone is not successful execution or complete behavioral coverage. Offline boundaries and remaining partial proof are stated per AC; @@ -393,7 +393,8 @@ for source resolution, then rerun after its normal plan merge. Draft status is i strict mode gates landed plans only, so inspect the native report for missing proofs as well. A row naming a unit/fake-client test establishes only that boundary; it does not prove the entire AC. -**Offline candidate verification:** the implementation content pinned above passed targeted race +**Prior offline verification:** implementation `0df361480b2b9c1b9a02ca5c7f55362d0e257966` +passed targeted race packages (execution client/controller/executor/protocol, mecak8s, and remote composition), the build-tagged quota test, compilation of the Kind qualification and fixture packages without running Kind, full `task lint`, full `task test` (including race and standalone module checks), `task build`, @@ -401,6 +402,18 @@ Kind, full `task lint`, full `task test` (including race and standalone module c `.scratch/orchestrate/native-kubernetes-execution/coverage-*.log`. Source conformance does not qualify Kubernetes admission, CNI enforcement, or live-model behavior; those remain the runtime gates below. +**Bounded follow-up verification:** `773a203df` preserves operator permission sources through the +existing workspace-pinned resolver with no workspace. The real-factory auto-posture regression first +failed with both Deny/Ask commands executed and no configured ask, for global and explicit-file sources; +it passes after the fix. Local/no-FS policy selection and model configuration are unchanged. Default, +selector, and client-bound remote factory paths share `remoteSessionConfiguration` before catalog and +engine assembly. The added AC3.4/AC6 proofs pass targeted race tests; full lint, docs, site build, +Taskfile build, and offline demo pass. Logs are under the implementation's ignored +`.scratch/orchestrate/native-kubernetes-execution/final-bounded-*.log`. The first full-suite attempt +hit the unrelated status-line clock-refresh timeout; its isolated race rerun and one final full +`task test` rerun passed, including all module/standalone checks. +These are offline results, not new-head CI, live qualification, or approval. + **Observed qualification:** the [production Kind+Calico job](https://github.com/stacklok/mecatl/actions/runs/35589357385/job/106299991747) completed **SUCCESS** at historical implementation `deaf1c3d1dfe7ea92afc8fe826f1bc613080f219`. This includes network isolation, rotation/revocation, replicas/lifecycle, holder loss, quota saturation, pending-delete recovery, @@ -428,10 +441,10 @@ new-head CI, final live execution, or human contract/panel approval. | AC | Existing implementation proof at the reference commit; remaining evidence | |---|---| -| AC1.1 | `cmd/mecak8s/execution_startup_test.go`: `TestDisabledExecutionStartupDoesNotContactExecutionOrKubernetes` exercises CLI `run()` through composition to listener setup with poison TLS paths and no test-endpoint calls. `internal/adapter/executioncontroller/disabled_composition_test.go` exercises disabled Build/run/Close beside retained provider resources. Constructor/informer absence is also inspected at the CLI enablement branch; no flaky goroutine census. | +| AC1.1 | `cmd/mecak8s/execution_startup_test.go`: `TestDisabledExecutionStartupDoesNotContactExecutionOrKubernetes` exercises CLI `run()` through composition to listener setup with poison TLS paths and no test-endpoint calls. `internal/adapter/executioncontroller/disabled_composition_test.go` exercises independent reconciliation beside disabled Build/run/Close; its fake clients are not injected into app.Build, so their empty action log does not detect app-created clients. Constructor/informer absence is also inspected at the CLI enablement branch; no flaky goroutine census. | | AC1.2 | `deploy/helm/mecatl-execution/chart_test.go`: `TestChartRetainsCRDAndDoesNotGrantSecretAPI`; production job separately deploys provider and client. | | AC1.3 | `cmd/mecak8s/execution_startup_test.go`: `TestExecutionPreflightRejectsIncompleteAndIncompatibleCLI`; `internal/app/remote_execution_test.go`: `TestRemoteExecutionPreflightNeverAllocates`; handler read-only validation and mTLS/readiness tests cover the transport. | -| AC1.4 | `internal/adapter/executioncontroller/disabled_composition_test.go`: `TestDisabledCompositionPreservesAllocationsAndIndependentReconciliation` compares retained objects across disabled Build/run/Close, then observes independent reconciliation. `controller_test.go`: peer-restart test remains complementary. | +| AC1.4 | `internal/adapter/executioncontroller/disabled_composition_test.go`: `TestDisabledCompositionPreservesAllocationsAndIndependentReconciliation` observes independent reconciliation after disabled Build/run/Close. Retained objects and fake action counts are only a limited lifecycle oracle because the clients are not injected into app.Build; the poisoned-endpoint CLI test in AC1.1 guards startup contact. `controller_test.go`: peer-restart test remains complementary. | | AC2.1 | `internal/adapter/executioncontroller/handler_test.go`: `TestHandlerValidateIsReadOnlyAndEnsureIdempotent`; `e2e/k8s_execution/qualification_test.go`: `TestKindExecutionQualification`. | | AC2.2 | `internal/adapter/executioncontroller/store_test.go`: `TestStoreEnsureUsesStableLookupAndRejectsFingerprintDrift`; `controller_test.go`: `TestReconcileRefusesForeignExistingPVCWithoutPersistingUID`. | | AC2.3 | `internal/adapter/executioncontroller/profiles_test.go`: `TestLoadProfilesStrictAndDigestPinned`; production `TestKindExecutionProductionQuotaSaturation`. | @@ -439,7 +452,7 @@ new-head CI, final live execution, or human contract/panel approval. | AC3.1 | `internal/adapter/executionclient/files_conformance_test.go`: `TestRemoteWorkspaceConformanceThroughSignedGRPC` runs both existing shared suites; `TestRemoteFileToolsPreserveLedgerAndNamespaceContracts` invokes all nine tool bodies against the signed handler and real executor. `e2e/k8s_execution/file_tools_test.go` adds the tool matrix called by Kind qualification; runtime execution PENDING. | | AC3.2 | Same gRPC suites cover concurrent create/CAS, missing/stale/zero versions, non-clobbering namespace operations, and actual Read-ledger behavior. The suite exposed and now guards missing-file replacement precedence and typed non-empty-directory removal errors. | | AC3.3 | `internal/adapter/executioncontroller/file_authorization_test.go`: every supported file operation has positive and negative signed-grant controls; `internal/executionexecutor/confinement_test.go`: physical symlink traversal/search checks. Existing security manager, protocol, durable revocation, and detached-control tests remain complementary. | -| AC3.4 | `internal/adapter/executioncontroller/store_test.go`: `TestCancelledStoreOperationStaysActiveUntilBackendStopsThenFences`; production `TestKindExecutionProductionHolderLossFencesActiveOperation`. | +| AC3.4 | `internal/executionexecutor/executor_test.go`: `TestNativeShellCapsCombinedOutputAndRepairsUTF8`, `TestNativeShellWriteInvalidatesRecordedRead` execute the native helper for combined output bounds, malformed bytes, and actual Shell-write→Edit version refusal. `internal/adapter/executionclient/client_test.go`: `TestProviderThroughRealGRPCSignedHandlerRefreshesAndReattachesExactly` checks runner text repair after private protobuf bytes; `internal/adapter/executionclient/service_integration_test.go`: `TestServiceUsesRealMTLSProviderStoreAndReleasesOnlyAfterDrain` checks identical valid UTF-8 in client events and model history. `internal/adapter/executioncontroller/store_test.go`: `TestCancelledStoreOperationStaysActiveUntilBackendStopsThenFences`; production `TestKindExecutionProductionHolderLossFencesActiveOperation` remains runtime qualification. | | AC4.1 | `internal/adapter/executioncontroller/store_lifecycle_test.go`: `TestClientScopedIntentListReturnsAttestedOwnerOnlyToOwningClient`; Kind isolation and pending-delete tests; scoped-admin non-escalation is exercised by the AC4.4 tests. | | AC4.2 | `internal/adapter/executioncontroller/store_revoke_test.go`: `TestRevokeEnvironmentFencesOldClaimWithoutChangingExecutionEpoch`; `store_concurrency_test.go`: `TestAcquireRunAndRevokeUseResourceVersionCAS`; production security rotation. | | AC4.3 | `internal/adapter/executioncontroller/production_lifecycle_test.go`: `TestExpiredOperationLeaseFencesWithoutClearingIdentity`, `TestRecoverMissingPodRemainsFenceUnknown`, `TestRecoveredTerminalProofCanStartExactReplacement`; production holder loss. | @@ -451,7 +464,7 @@ new-head CI, final live execution, or human contract/panel approval. | AC5.4 | `internal/adapter/executionclient/reference_ambiguity_test.go`: `TestAmbiguousCommitIsRetainedAndNeverAbortedByCleanup`; production `TestKindExecutionProductionPendingDeleteOutageRecovery`. | | AC5.5, AC5.7 | `deploy/helm/mecatl-execution/chart_test.go`: `TestChartHasNoDeletionHook`; `deploy/helm/mecatl-execution/lifetime_test.go`: `TestChartRetainedLifetime` exercises lookup/adoption, missing history, foreign ownership, profile drift, and retained policies/ledgers offline. `e2e/k8s_execution/zz_production_helm_lifetime_test.go`: `TestKindExecutionProductionHelmLifetime` exercises actual upgrade/uninstall/reinstall and enforcement; final runtime evidence PENDING. | | AC5.6 | Deployment-documentation inspection and docs gates required on the final implementation; Kind is restart proof only. | -| AC6.1–AC6.2 | `internal/app/remote_execution_test.go`: real factory poison-source and operator-global positive controls, forbidden-tool invocation errors, command/worktree discovery, and unchanged no-FS catalog/no-allocation checks. Native background Shell rejection is exercised in `internal/adapter/executionclient/service_integration_test.go` alongside a successful foreground command. Native Git snapshot and unsupported-successor invocation evidence remains partial; the shared admission gate and catalog inspection alone do not close that matrix. | +| AC6.1–AC6.2 | `internal/app/remote_execution_test.go`: `TestRemoteExecutionRealFactoryCarriesPostureAndAttenuatedCatalog` includes a nonempty LOCAL Git snapshot control and REMOTE exclusion, poison-source and operator-global prompt controls, forbidden-tool errors and command/worktree discovery. `TestRemoteExecutionPreservesOperatorPermissionsUnderAuto` exercises global/explicit-file Deny and configured Ask, excluding project permissions, with allowed in-memory Shell controls. `TestRemoteDeploymentNoFSUsesLocalAttenuationWithoutProviderCall` retains the no-FS check. `internal/adapter/executionclient/service_integration_test.go`: `TestServiceUsesRealMTLSProviderStoreAndReleasesOnlyAfterDrain` rejects background Shell and actual Clear/Fork unsupported-selector calls with named errors, no destination publication, and unchanged source ref/history and provider references. | | AC6.3 | `internal/adapter/executioncontroller/store_lifecycle_test.go`: `TestReferenceTransactionsRetainUnknownAndNeverChangeSource`; production `TestKindExecutionProductionClearForkLifecycle`. | | AC7.1 | `.github/workflows/k8s-e2e.yml` runs `task e2e:k8s:execution:production`; the historical reference job succeeded, but the newer job failed during quota-dependent fixture setup before tests. `internal/adapter/executioncontroller/legacy_fixture_kind_test.go` covers the bounded accounting wait offline. `e2e/k8s_execution/aa_production_migration_test.go`: `TestKindExecutionProductionCompatiblePrototypeMigration`. Final amended-candidate run **PENDING**. | | AC7.2 | `e2e/k8s_execution/qualification_test.go`: `TestKindExecutionQualification`, run by the successful production profile. | @@ -492,4 +505,4 @@ plan worktree to make citations or trace appear complete. - A PVC can preserve files while a stale command still mutates them. Session leasing does not fence command execution; missing old-executor evidence remains a fail-closed manual recovery boundary. - Only an enforcing CNI supplies negative NetworkPolicy proof. Retaining storage while uninstall removes isolation or authority history is unsafe; the Helm lifecycle amendment is a completion requirement, not deferred polish. - Every outlives-a-call resource, including retained security/capacity ledgers, reload state, informer/cache, connection pools, claims, and lifecycle/reference reconciliation, needs an accurate resource/rehydration inventory in `docs/adr/0027-cloud-native.md` on the implementation branch. -- Scope, retry, Helm lifecycle, disabled startup, remote filesystem conformance, and source-admission behavioral tests exist in the pinned implementation source. Final production Kind/native live evidence and human reviews remain pending. The remaining partial evidence in AC3.4 and AC6.1–AC6.2 is still an acceptance obligation. Source/proof locations absent on the plan branch stay explicitly implementation-scoped; do not add implementation code to satisfy plan-branch gates. +- Scope, retry, Helm lifecycle, disabled startup, remote filesystem conformance, and source-admission behavioral tests exist in the pinned implementation source. Final production Kind/native live evidence and human reviews remain pending. The requested offline output/CAS and Git/unsupported-successor proofs in AC3.4 and AC6.1–AC6.2 are present; they do not replace final runtime qualification. Source/proof locations absent on the plan branch stay explicitly implementation-scoped; do not add implementation code to satisfy plan-branch gates. From 3761f6370bbebcc70f08f679d7f06505b29a0fb2 Mon Sep 17 00:00:00 2001 From: Juan Antonio Osorio Date: Mon, 21 Sep 2026 23:58:30 +0300 Subject: [PATCH 12/16] docs(plan): bind rotation to immutable material publication Co-Authored-By: OpenAI --- .../acceptance/native-kubernetes-execution.md | 43 ++++++++++++++++++- docs/adr/0350-native-kubernetes-execution.md | 22 +++++++++- 2 files changed, 63 insertions(+), 2 deletions(-) diff --git a/docs/acceptance/native-kubernetes-execution.md b/docs/acceptance/native-kubernetes-execution.md index 2efab54065..fd38700034 100644 --- a/docs/acceptance/native-kubernetes-execution.md +++ b/docs/acceptance/native-kubernetes-execution.md @@ -192,6 +192,25 @@ connections. Key windows, CA/client removal, equal-generation policy drift, key- different material, and rollback fail closed. RevokeEnvironment advances the separate per-environment grantGeneration. Neither revocation nor a changed security generation constitutes termination proof. +**Rotation publication amendment:** use the existing filename fields as immutable material identities. +Every changed signing key, server certificate, server private key, or client-CA bundle gets a new, +generation-specific filename; never overwrite a referenced name with different bytes. Stage the added +Secret entries before publishing the higher-generation manifest, retaining old entries until no live +or in-flight manifest references them. ConfigMap and Secret projections are not atomic together. +A manifest that arrives before its new files must fail closed without advancing the ledger; a material +projection that arrives first must leave the old manifest's authority unchanged. The loader reads one +manifest and confines each named file through `os.Root`; no projection-pinning helper, new config field, +proto, or engine API is introduced. Runtime byte immutability is an operator publication obligation, +not a newly enforced file-history registry. If reused filenames have already published mixed material, +retries cannot repair equal-generation digest drift: publish a complete bundle at a higher generation, +never reset the ledger. + +A peer may advance the shared ledger after a pinned replica successfully acquires a claim. Its next +read must return structured retryable `not_ready` before dispatch until that replica reloads. Positive +qualification uses a bounded read-only content poll for **only** `CodeNotReady && Retryable`; wrong +content and all other errors fail immediately. This amendment adds no mutating retry. Negative +old-client and revoked-key checks use a current, nonexpired claim, not a released phase claim. + **Scoped-admin amendment, exact proposed configuration:** add optional `administratorFor: []string` to each `clients[]` entry, matching existing lowerCamelCase `mayAttestOwner` and `administrator`. For example, the nonsecret manifest fragment is: @@ -433,6 +452,28 @@ timeout, cancellation, and forbidden-response controls in arbitrary forbidden creates. Those offline controls and tagged compilation are not a rerun of the production job. +**Latest reported production context:** the operator supplied run +[35650128625, job 106500190798](https://github.com/stacklok/mecatl/actions/runs/35650128625/job/106500190798) +at `f7919f6c`: 10/11 production tests passed, including scoped administration, Helm lifecycle/quota, +and general file tools. Security rotation failed after about 250 seconds on the final-authority Files +READ immediately after successful AcquireRun (`production_qualification_test.go`, former line 279). +This report was not independently re-fetched during the offline repair. It supersedes the earlier +setup failure as the current reported runtime blocker, not as a final-candidate success. + +**Deterministic repair evidence:** implementation `96e16e926` adds +`internal/adapter/executioncontroller/security_rotation_test.go`: +`TestSecurityRotationPinnedReplicaReadConvergesBeforeDispatch` proves peer advancement → retryable +`not_ready` with zero Files dispatch → reload → the same valid bridge-k2 claim succeeds. +`TestSecurityRotationMutableNamesPoisonSameGeneration` proves that a new manifest with old +`tls.crt`/`tls.key`/`clients.pem` can publish a mixed digest that the later complete bundle cannot replace +at that generation. This is a reproduced defect, **not proof that it caused the CI failure**. +`TestSecurityRotationImmutableNamesHandleProjectionSkew` proves both projection orders recover using +new immutable filenames while retaining old entries. All three pass offline with `-race`, without +sleeps or a cluster. The native exact-cap stdout/stderr/combined repair-expansion regression first +failed with `Truncated=false`, then passed with `-race` after tracking clipping after UTF-8 repair; +earlier bounded-writer truncation remains ORed into the result. Fixture publication/barrier changes +and final runtime qualification are still separate evidence; no live inference or cluster run is claimed. + [PR #1728](https://github.com/stacklok/mecatl/pull/1728), commit `6501b5924`, is already integrated in the implementation ancestry. Its generic live-compaction repair is not native-provider qualification. Final-candidate native-provider live and amended @@ -454,7 +495,7 @@ new-head CI, final live execution, or human contract/panel approval. | AC3.3 | `internal/adapter/executioncontroller/file_authorization_test.go`: every supported file operation has positive and negative signed-grant controls; `internal/executionexecutor/confinement_test.go`: physical symlink traversal/search checks. Existing security manager, protocol, durable revocation, and detached-control tests remain complementary. | | AC3.4 | `internal/executionexecutor/executor_test.go`: `TestNativeShellCapsCombinedOutputAndRepairsUTF8`, `TestNativeShellWriteInvalidatesRecordedRead` execute the native helper for combined output bounds, malformed bytes, and actual Shell-write→Edit version refusal. `internal/adapter/executionclient/client_test.go`: `TestProviderThroughRealGRPCSignedHandlerRefreshesAndReattachesExactly` checks runner text repair after private protobuf bytes; `internal/adapter/executionclient/service_integration_test.go`: `TestServiceUsesRealMTLSProviderStoreAndReleasesOnlyAfterDrain` checks identical valid UTF-8 in client events and model history. `internal/adapter/executioncontroller/store_test.go`: `TestCancelledStoreOperationStaysActiveUntilBackendStopsThenFences`; production `TestKindExecutionProductionHolderLossFencesActiveOperation` remains runtime qualification. | | AC4.1 | `internal/adapter/executioncontroller/store_lifecycle_test.go`: `TestClientScopedIntentListReturnsAttestedOwnerOnlyToOwningClient`; Kind isolation and pending-delete tests; scoped-admin non-escalation is exercised by the AC4.4 tests. | -| AC4.2 | `internal/adapter/executioncontroller/store_revoke_test.go`: `TestRevokeEnvironmentFencesOldClaimWithoutChangingExecutionEpoch`; `store_concurrency_test.go`: `TestAcquireRunAndRevokeUseResourceVersionCAS`; production security rotation. | +| AC4.2 | `internal/adapter/executioncontroller/store_revoke_test.go`: `TestRevokeEnvironmentFencesOldClaimWithoutChangingExecutionEpoch`; `store_concurrency_test.go`: `TestAcquireRunAndRevokeUseResourceVersionCAS`; the three deterministic rotation tests listed above at repair commit `96e16e926`. Latest reported production rotation failed; final runtime success PENDING. | | AC4.3 | `internal/adapter/executioncontroller/production_lifecycle_test.go`: `TestExpiredOperationLeaseFencesWithoutClearingIdentity`, `TestRecoverMissingPodRemainsFenceUnknown`, `TestRecoveredTerminalProofCanStartExactReplacement`; production holder loss. | | AC4.4 | `internal/adapter/executioncontroller/admin_scope_test.go`: `TestScopedAdminAllRoutesOverMTLS`, `TestScopedAdminCASRetryRechecksSubject`, `TestScopedAdminMigrationReceiptRetainsUIDPreconditions`, `TestMigrationCompletedCASReplayRequiresExactSourceSchema`, `TestMigrationReceiptExpiresAfterReconciledReplacement`, `TestScopedAdminDoesNotGrantAttestationOrDataPlane`, `TestScopedAdminWithOwnerAttestationCannotUseAnotherCreatorsDataPlane`. Replacement-expiry test is in `internal/adapter/executioncontroller/production_lifecycle_test.go` in the pinned implementation candidate. `e2e/k8s_execution/ab_production_admin_test.go`: `TestKindExecutionProductionScopedAdministrator` exists (replacement/non-escalation, not all six routes); final Kind evidence PENDING. | | AC4.5 | `internal/adapter/executioncontroller/admin_scope_security_test.go`: `TestAdministratorScopeManifestValidation`, `TestAdministratorScopeDigestIsNormalizedAndAuthorityBound`; `internal/adapter/executioncontroller/admin_scope_test.go`: all-routes scope removal plus `TestScopedAdminEqualGenerationDriftFailsClosed`. Compatibility/upgrade human review PENDING. | diff --git a/docs/adr/0350-native-kubernetes-execution.md b/docs/adr/0350-native-kubernetes-execution.md index fe663fbb50..72eeebcd5a 100644 --- a/docs/adr/0350-native-kubernetes-execution.md +++ b/docs/adr/0350-native-kubernetes-execution.md @@ -134,6 +134,21 @@ CA/key/client removal, scope removal, rollback, or equal-generation policy drift security generation nor per-environment grant revocation is proof of termination. No existing SPIFFE issuer is assumed, and no credential crosses the provider-to-workload stdin boundary. +Material publication uses the existing manifest filenames as immutable identities. Changed signing, +server-certificate, server-key, and client-CA bytes require new generation-specific names. Stage them +before publishing the higher-generation manifest and retain overlap files while any live/in-flight +manifest references them. Secret and ConfigMap updates are not atomic together. One loaded manifest +binds the confined immutable names even across projection changes; missing material fails closed before +ledger publication. No loader projection-pinning helper or new version/config API is required under +this operator-managed publication contract. Reusing names can publish a mixed-generation digest; +correcting that bundle at the same generation is then rejected permanently. Recover only by forward +publication, never by resetting authority history. + +Peer-ledger advancement can transiently reject a pinned replica's read before dispatch even after +successful AcquireRun. Qualification may poll only read-only structured retryable `not_ready`, with a +bound and immediate failure on wrong content or other errors. It must not hide mixed-material drift +or count a released/expired claim as evidence of old-key/client rejection. + Scoped administration uses one narrow extension to the existing client policy: `administrator:true` plus optional `administratorFor:[canonicalCreatorURI]`. Absent/empty scope preserves self-admin compatibility, not namespace-wide privilege. A distinct operations URI may administer only @@ -197,7 +212,12 @@ owned cluster, scratch kubeconfig, explicit context on every command, bounded sa ownership-scoped cleanup. Default kind does not prove NetworkPolicy enforcement; the production profile uses Calico. The acceptance plan records a successful production job at `deaf1c3d1`, distinct from the pending final candidate and new scope/retry/Helm regression qualification. Historical local keyring -failures no longer block that baseline. Skipped draft race lanes remain unproven. +failures no longer block that baseline. Skipped draft race lanes remain unproven. The latest +operator-reported run `35650128625` at `f7919f6c` passed 10/11 production tests, including scoped admin, +Helm lifecycle/quota, and general file tools; rotation failed at the final-authority read. Offline tests +at repair commit `96e16e926` reproduce both safe pre-dispatch replica lag and persistent mixed-material +ledger poisoning, and prove immutable-name publication across both projection orders. They do not +identify which mechanism caused that CI failure or establish final runtime success. The checked human live decision requires the separate `task e2e:k8s:execution:live` OpenRouter coding qualification **for this stack**, after deterministic qualification. It is not an always-on live CI From 11597f5e48707dfc123cc52934f06c110f40db84 Mon Sep 17 00:00:00 2001 From: Juan Antonio Osorio Date: Tue, 22 Sep 2026 00:09:32 +0300 Subject: [PATCH 13/16] docs(plan): record deterministic rotation repair evidence Co-Authored-By: OpenAI --- .../acceptance/native-kubernetes-execution.md | 28 +++++++++++++++---- 1 file changed, 22 insertions(+), 6 deletions(-) diff --git a/docs/acceptance/native-kubernetes-execution.md b/docs/acceptance/native-kubernetes-execution.md index fd38700034..1be98bdb34 100644 --- a/docs/acceptance/native-kubernetes-execution.md +++ b/docs/acceptance/native-kubernetes-execution.md @@ -200,8 +200,10 @@ or in-flight manifest references them. ConfigMap and Secret projections are not A manifest that arrives before its new files must fail closed without advancing the ledger; a material projection that arrives first must leave the old manifest's authority unchanged. The loader reads one manifest and confines each named file through `os.Root`; no projection-pinning helper, new config field, -proto, or engine API is introduced. Runtime byte immutability is an operator publication obligation, -not a newly enforced file-history registry. If reused filenames have already published mixed material, +proto, or engine API is introduced. Before a positive rotation barrier, qualification observes the +intended durable ledger generation; cached PodReady alone cannot establish it. Runtime byte immutability +is an operator publication obligation, not a newly enforced file-history registry. If reused filenames +have already published mixed material, retries cannot repair equal-generation digest drift: publish a complete bundle at a higher generation, never reset the ledger. @@ -335,7 +337,7 @@ therefore requires independent environment execution fencing. - AC4.1: Ordinary creator client A cannot discover, attach, execute in, stream from, cancel, or retire creator client B's environment; hidden and absent allocations are indistinguishable at the public boundary. The explicit scoped administrative exception in AC4.4 permits only its named administrative RPCs. - verify: `TestClientScopedIntentListReturnsAttestedOwnerOnlyToOwningClient`, `TestScopedAdminAllRoutesOverMTLS`, `TestScopedAdminWithOwnerAttestationCannotUseAnotherCreatorsDataPlane`, `TestKindExecutionQualification` (final Kind execution PENDING). - AC4.2: Every command is authorized for one immutable environment revision and a distinct transient execution-fence epoch; lease loss or grant revocation prevents new commands and cancels/fences active work without exposing capability credentials to the workload or another caller's output. Same-Pod placement alone is not a security boundary. - - verify: `TestGrantRoundTripAndExactBindings`, `TestRevokeEnvironmentFencesOldClaimWithoutChangingExecutionEpoch`, `TestAcquireRunAndRevokeUseResourceVersionCAS`, `TestKindExecutionProductionSecurityRotation`, `TestKindExecutionProductionHolderLossFencesActiveOperation` (final Kind execution PENDING). + - verify: `TestGrantRoundTripAndExactBindings`, `TestRevokeEnvironmentFencesOldClaimWithoutChangingExecutionEpoch`, `TestAcquireRunAndRevokeUseResourceVersionCAS`, `TestSecurityRotationPinnedReplicaReadConvergesBeforeDispatch`, `TestSecurityRotationMutableNamesPoisonSameGeneration`, `TestSecurityRotationImmutableNamesHandleProjectionSkew`, `TestReadCurrentAuthorityContentRetriesOnlyPreDispatchLag`, `TestReadCurrentAuthorityContentBoundAndCancellation`, `TestKindExecutionProductionSecurityRotation`, `TestKindExecutionProductionHolderLossFencesActiveOperation` (final Kind execution PENDING). - AC4.3: Timeout, Pod deletion, controller restart, Lease timeout, or network partition alone never authorizes a replacement executor while an old writer may run; unknown fencing state fails closed and requires the reviewed manual/external fencing path. - verify: `TestExpiredOperationLeaseFencesWithoutClearingIdentity`, `TestRecoverMissingPodRemainsFenceUnknown`, `TestRecoveredTerminalProofCanStartExactReplacement`, `TestKindExecutionProductionHolderLossFencesActiveOperation` (final Kind execution PENDING); inspection — external fencing runbook review PENDING. - AC4.4: A distinct `administrator:true` client with `administratorFor:[creatorURI]` can administer that creator's environment through all six administrative RPCs, including migration, revocation, and retained deletion, while preserving immutable creator identity and exact owner/revision/epoch/UID/operation checks. A second creator, wrong owner, stale identity, or replay with changed inputs is denied without existence disclosure. Self-admin compatibility remains explicit; no Files/Shell/attach/run/reference or MayAttestOwner authority is gained. @@ -471,8 +473,22 @@ at that generation. This is a reproduced defect, **not proof that it caused the new immutable filenames while retaining old entries. All three pass offline with `-race`, without sleeps or a cluster. The native exact-cap stdout/stderr/combined repair-expansion regression first failed with `Truncated=false`, then passed with `-race` after tracking clipping after UTF-8 repair; -earlier bounded-writer truncation remains ORed into the result. Fixture publication/barrier changes -and final runtime qualification are still separate evidence; no live inference or cluster run is claimed. +earlier bounded-writer truncation remains ORed into the result. Implementation `faacfcde7` stages and +retains immutable fixture names, observes the requested ledger generation, and uses a bounded read-only +barrier (30 seconds, at most 60 calls) that admits only structured retryable `not_ready`. +`TestReadCurrentAuthorityContentRetriesOnlyPreDispatchLag` and +`TestReadCurrentAuthorityContentBoundAndCancellation` pin success, immediate wrong-content/nonretryable/ +wrong-code failure, persistent lag, and cancellation without sleeps. The rotation generator's +`TestForwardRestoreTrustBundleSupportsFixtureClient` pins the actual manifest filename references; +its new filename assertions failed before the fixture fix and pass afterward. Tagged race tests and +compilation pass. Old-client rejection uses the current successful final claim; the revoked-k1 probe +reuses Helm's re-sign/current-claim positive controls. No new mutating retry was added; the existing +bounded AcquireRun helper is unchanged. Full untagged `task lint` and one full `task test` invocation +(including module race and standalone checks), docs/checker, site build, strict +native-plan trace (32 ACs, zero missing; draft/report-only), targeted four-package race tests, and the +offline demo pass. Tagged whole-fixture lint exposes unrelated pre-existing findings; the changed-line +gate against baseline `2d32864cee8378939cb55137e1ab369e70a18349` passes with zero issues. +These results do not replace final runtime qualification; no live inference or cluster run is claimed. [PR #1728](https://github.com/stacklok/mecatl/pull/1728), commit `6501b5924`, is already integrated in the implementation ancestry. Its generic live-compaction @@ -507,7 +523,7 @@ new-head CI, final live execution, or human contract/panel approval. | AC5.6 | Deployment-documentation inspection and docs gates required on the final implementation; Kind is restart proof only. | | AC6.1–AC6.2 | `internal/app/remote_execution_test.go`: `TestRemoteExecutionRealFactoryCarriesPostureAndAttenuatedCatalog` includes a nonempty LOCAL Git snapshot control and REMOTE exclusion, poison-source and operator-global prompt controls, forbidden-tool errors and command/worktree discovery. `TestRemoteExecutionPreservesOperatorPermissionsUnderAuto` exercises global/explicit-file Deny and configured Ask, excluding project permissions, with allowed in-memory Shell controls. `TestRemoteDeploymentNoFSUsesLocalAttenuationWithoutProviderCall` retains the no-FS check. `internal/adapter/executionclient/service_integration_test.go`: `TestServiceUsesRealMTLSProviderStoreAndReleasesOnlyAfterDrain` rejects background Shell and actual Clear/Fork unsupported-selector calls with named errors, no destination publication, and unchanged source ref/history and provider references. | | AC6.3 | `internal/adapter/executioncontroller/store_lifecycle_test.go`: `TestReferenceTransactionsRetainUnknownAndNeverChangeSource`; production `TestKindExecutionProductionClearForkLifecycle`. | -| AC7.1 | `.github/workflows/k8s-e2e.yml` runs `task e2e:k8s:execution:production`; the historical reference job succeeded, but the newer job failed during quota-dependent fixture setup before tests. `internal/adapter/executioncontroller/legacy_fixture_kind_test.go` covers the bounded accounting wait offline. `e2e/k8s_execution/aa_production_migration_test.go`: `TestKindExecutionProductionCompatiblePrototypeMigration`. Final amended-candidate run **PENDING**. | +| AC7.1 | `.github/workflows/k8s-e2e.yml` runs `task e2e:k8s:execution:production`; the historical reference job succeeded. The latest operator-reported job at `f7919f6c` passed 10/11 production tests and failed rotation's final read; it progressed past the earlier quota setup failure. Offline rotation evidence and read-only barrier controls are listed above. `internal/adapter/executioncontroller/legacy_fixture_kind_test.go` covers the bounded quota-accounting wait. Final amended-candidate run **PENDING**. | | AC7.2 | `e2e/k8s_execution/qualification_test.go`: `TestKindExecutionQualification`, run by the successful production profile. | | AC7.3 | Production `TestKindExecutionProductionFailureArtifactBoundary`; workflow bounded uploads and ownership-scoped cleanup. | | AC7.4 | `e2e/k8s_execution/live_qualification_test.go`: `TestKindExecutionLiveQualification` exists; successful final native-provider OpenRouter evidence **PENDING**. | From 7d9e37a128b17107e2d83467c7767146e9c5efb8 Mon Sep 17 00:00:00 2001 From: Juan Antonio Osorio Date: Tue, 22 Sep 2026 00:21:19 +0300 Subject: [PATCH 14/16] docs(plan): bound manual native execution completion qualification Co-Authored-By: OpenAI --- .../acceptance/native-kubernetes-execution.md | 23 +++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/docs/acceptance/native-kubernetes-execution.md b/docs/acceptance/native-kubernetes-execution.md index 1be98bdb34..44bdffbd4e 100644 --- a/docs/acceptance/native-kubernetes-execution.md +++ b/docs/acceptance/native-kubernetes-execution.md @@ -387,6 +387,29 @@ The stack additionally requires the explicitly invoked native-provider OpenRoute recorded in Human decisions; it is completion evidence, not an always-on CI dependency. The broad mecak8s live suite is not native-provider proof. The focused mock task must not inherit ambient provider credentials. +The existing `e2e-live.yml` workflow may run this qualification only through explicit +`workflow_dispatch` with `native_execution: true` (default false) in `stacklok/mecatl`. +An optional `expected_sha` must match the checked-out event SHA exactly before credential +staging; record that public SHA in the summary. Maintainer dispatch, repository restriction, +and repository-secret access are the trust gates; there is no environment-review gate. +Run production qualification without automatic cluster deletion, then the live target in the +same job only after production succeeds. The ownership record alone is not a success receipt. +Only then stage `OPENROUTER_API_KEY` in a uniquely created private CI directory/file +(0700/0600); absence fails qualification. Remove the key environment before executing the +live script. Its trusted loader, UID-pinned Secret cleanup, and mock restoration remain +mandatory; cleanup failures fail the job. Always attempt deletion of only the recorded, +validated CI-owned cluster and credential file. Upload only the bounded `live-summary.json` +and sanitized qualification status, never credentials, receipts, kubeconfigs, PKI, Helm +values, or transcripts. A 100-minute job bound covers production, mock rerun, live smoke, +builds, and cleanup; it is not a dollar cap. Final-candidate runtime evidence and independent +security review remain PENDING. + +The rotation restart proof must compare the observed grant generation and replayed revoke +receipt exactly with the original revoke result. A current, successful post-restart claim +must bracket a probe signed by the current trusted key with only its grant generation +changed to the revoked value. Require the structured claim-generation rejection, not an +arbitrary error from a released/expired claim, retired key, or unavailable provider. + **Acceptance:** - AC7.1: `task e2e:k8s:execution` and the enforcing-CNI `task e2e:k8s:execution:production` run on the supported generic toolchain, including CI, with a unique owned kind cluster, a scratch kubeconfig, and explicit `--kubeconfig`/context on every command. They deploy the separate chart plus explicitly connected mecak8s and use only the deterministic mock provider. The final production-profile run uses Calico and exercises negative network isolation, rotation/revocation, replica/lifecycle recovery, holder loss, quota recovery, pending delete, Clear/Fork, migration, sanitized artifacts, and the amendment regressions. - verify: `TestLegacyFixtureWaitsForQuotaAccountingBeforeCreate`, `TestKindExecutionQualification`, `TestKindExecutionProductionNetworkPolicyEnforced`, `TestKindExecutionProductionSecurityRotation`, `TestKindExecutionProductionReplicaLifecycle`, `TestKindExecutionProductionHolderLossFencesActiveOperation`, `TestKindExecutionProductionQuotaSaturation`, `TestKindExecutionProductionPendingDeleteOutageRecovery`, `TestKindExecutionProductionClearForkLifecycle`, `TestKindExecutionProductionCompatiblePrototypeMigration`, `TestKindExecutionProductionScopedAdministrator`, `TestKindExecutionProductionHelmLifetime` (names exist; final production Kind+Calico run PENDING). From fd2571798110a1243a0c5f53eaea83ff72f3a544 Mon Sep 17 00:00:00 2001 From: Juan Antonio Osorio Date: Tue, 22 Sep 2026 02:13:11 +0300 Subject: [PATCH 15/16] docs(plan): synchronize native execution repair evidence Co-Authored-By: OpenAI --- .../acceptance/native-kubernetes-execution.md | 63 ++++++++++++++++--- 1 file changed, 56 insertions(+), 7 deletions(-) diff --git a/docs/acceptance/native-kubernetes-execution.md b/docs/acceptance/native-kubernetes-execution.md index 44bdffbd4e..b63cfdfc93 100644 --- a/docs/acceptance/native-kubernetes-execution.md +++ b/docs/acceptance/native-kubernetes-execution.md @@ -471,19 +471,20 @@ at `9bb4d89b8aa497c9d2cd6aa4a94ab9262a913364` failed **before qualification test seeding `legacy-migration`: ResourceQuota accounting for `count/executionenvironments.execution.mecatl.dev` was still unknown. Provider Deployment readiness was not quota readiness. This supersedes a test-race diagnosis for that job. -The implementation adds a two-minute, context-aware quota-accounting wait with delayed-status, +The implementation adds a six-minute, context-aware quota-accounting wait (the +five-minute Kubernetes default usage resync plus margin) with delayed-status, timeout, cancellation, and forbidden-response controls in `TestLegacyFixtureWaitsForQuotaAccountingBeforeCreate`. It does not disable quota or retry arbitrary forbidden creates. Those offline controls and tagged compilation are not a rerun of the production job. -**Latest reported production context:** the operator supplied run +**Earlier reported production context:** the operator supplied run [35650128625, job 106500190798](https://github.com/stacklok/mecatl/actions/runs/35650128625/job/106500190798) at `f7919f6c`: 10/11 production tests passed, including scoped administration, Helm lifecycle/quota, and general file tools. Security rotation failed after about 250 seconds on the final-authority Files READ immediately after successful AcquireRun (`production_qualification_test.go`, former line 279). This report was not independently re-fetched during the offline repair. It supersedes the earlier -setup failure as the current reported runtime blocker, not as a final-candidate success. +setup failure for that reported run, not as a final-candidate success. **Deterministic repair evidence:** implementation `96e16e926` adds `internal/adapter/executioncontroller/security_rotation_test.go`: @@ -513,6 +514,54 @@ offline demo pass. Tagged whole-fixture lint exposes unrelated pre-existing find gate against baseline `2d32864cee8378939cb55137e1ab369e70a18349` passes with zero issues. These results do not replace final runtime qualification; no live inference or cluster run is claimed. +**Current repair evidence (offline, after `0e8c8a5ca`):** the operator reports all +11 native production tests passing at `7e498bd73`, followed by production failures +at `2e1aa6bbf` before live qualification. Manual run +[35659529618](https://github.com/stacklok/mecatl/actions/runs/35659529618) failed before +provider credential staging; no provider credentials were used. These reports +were not independently re-fetched during this repair. The production timeouts +remain unresolved incidents, not evidence that either source defect below caused them. + +- `TestRetainedDeletePeerBetweenFinalizerUpdateAndDelete` deterministically failed + when a second reconciler re-added the environment finalizer between Update and + Delete, stranding the authorized operation behind the generic deletion branch. + It also failed for an already-terminating authorized CR. Both pass after routing + only admitted retained deletion ahead of generic finalization. The test checks + one successful CR/PVC deletion and released capacity. Foreign-CR identity and + unapproved direct deletion have separate fail-closed controls; Delete also pins + resourceVersion and UID. No CRD, protobuf, or authorization contract changed. +- The holder-loss fixture's `nohup` timer was a descendant of the executor command. + Disconnect cancellation kills the process group and reaps descendants, so that + timer could legitimately disappear without terminating the Pod. The fixture now + proves FenceUnknown and overlap denial first, then uses exact-owned-Pod UID-guarded + Kubernetes deletion with normal grace. It waits for Succeeded/Failed and every + declared container's terminated state; absence is never proof. Offline tests + reject foreign UID/owner and incomplete terminal evidence. This causal mechanism + is source-proven; the historical timeout's cause is not. +- The legacy quota fixture waits up to six minutes, respects cancellation, fails + immediately on forbidden responses, and reports only allowlisted missing/mismatched + quota keys and elapsed time on deadline. Tiny-context offline tests cover readiness, + deadline, and cancellation without a six-minute test or blanket create retries. +- The manual workflow collects bounded, closed-vocabulary diagnostics before owned + cleanup when production fails. Offline fixture tests verify redaction, UID-match + booleans, known finalizers, quota key names, and collection even when deletion fails. + The CI-owned path uses the same collector. No raw manifests, commands, grants, + credentials, private URLs, or PKI enter its artifact allowlist. + +The retained-deletion, terminal-proof, quota-diagnostic, and pre-cleanup evidence +regressions were observed red then green offline. Final amended-candidate production +and native live runs remain **PENDING**; no cluster, live model, dispatch, or push +was performed for this repair. `0e8c8a5ca` removed SC2016, but earlier local +actionlint ran without ShellCheck and is not a ShellCheck pass. The plan baseline +remains `7d9e37a128b17107e2d83467c7767146e9c5efb8` in the normal-merge ancestry. +Reported offline gates at `53ab45453` passed: targeted controller/chart/contract and tagged fixture +race tests, `task lint`, tagged changed-line lint, one full `task test` attempt +(including standalone modules), `task docs` plus final docs/checker checks, +`task site:build`, draft/report-only native-plan trace (32 ACs, zero missing), +and the offline demo. Local actionlint passed with ShellCheck explicitly disabled; +ShellCheck was unavailable on the host and in the existing dev toolbox. CI remains +the authority for ShellCheck and final runtime qualification. + [PR #1728](https://github.com/stacklok/mecatl/pull/1728), commit `6501b5924`, is already integrated in the implementation ancestry. Its generic live-compaction repair is not native-provider qualification. Final-candidate native-provider live and amended @@ -534,21 +583,21 @@ new-head CI, final live execution, or human contract/panel approval. | AC3.3 | `internal/adapter/executioncontroller/file_authorization_test.go`: every supported file operation has positive and negative signed-grant controls; `internal/executionexecutor/confinement_test.go`: physical symlink traversal/search checks. Existing security manager, protocol, durable revocation, and detached-control tests remain complementary. | | AC3.4 | `internal/executionexecutor/executor_test.go`: `TestNativeShellCapsCombinedOutputAndRepairsUTF8`, `TestNativeShellWriteInvalidatesRecordedRead` execute the native helper for combined output bounds, malformed bytes, and actual Shell-write→Edit version refusal. `internal/adapter/executionclient/client_test.go`: `TestProviderThroughRealGRPCSignedHandlerRefreshesAndReattachesExactly` checks runner text repair after private protobuf bytes; `internal/adapter/executionclient/service_integration_test.go`: `TestServiceUsesRealMTLSProviderStoreAndReleasesOnlyAfterDrain` checks identical valid UTF-8 in client events and model history. `internal/adapter/executioncontroller/store_test.go`: `TestCancelledStoreOperationStaysActiveUntilBackendStopsThenFences`; production `TestKindExecutionProductionHolderLossFencesActiveOperation` remains runtime qualification. | | AC4.1 | `internal/adapter/executioncontroller/store_lifecycle_test.go`: `TestClientScopedIntentListReturnsAttestedOwnerOnlyToOwningClient`; Kind isolation and pending-delete tests; scoped-admin non-escalation is exercised by the AC4.4 tests. | -| AC4.2 | `internal/adapter/executioncontroller/store_revoke_test.go`: `TestRevokeEnvironmentFencesOldClaimWithoutChangingExecutionEpoch`; `store_concurrency_test.go`: `TestAcquireRunAndRevokeUseResourceVersionCAS`; the three deterministic rotation tests listed above at repair commit `96e16e926`. Latest reported production rotation failed; final runtime success PENDING. | +| AC4.2 | `internal/adapter/executioncontroller/store_revoke_test.go`: `TestRevokeEnvironmentFencesOldClaimWithoutChangingExecutionEpoch`; `store_concurrency_test.go`: `TestAcquireRunAndRevokeUseResourceVersionCAS`; the three deterministic rotation tests listed above at repair commit `96e16e926`. The reported production rotation at `f7919f6c` failed; final runtime success PENDING. | | AC4.3 | `internal/adapter/executioncontroller/production_lifecycle_test.go`: `TestExpiredOperationLeaseFencesWithoutClearingIdentity`, `TestRecoverMissingPodRemainsFenceUnknown`, `TestRecoveredTerminalProofCanStartExactReplacement`; production holder loss. | | AC4.4 | `internal/adapter/executioncontroller/admin_scope_test.go`: `TestScopedAdminAllRoutesOverMTLS`, `TestScopedAdminCASRetryRechecksSubject`, `TestScopedAdminMigrationReceiptRetainsUIDPreconditions`, `TestMigrationCompletedCASReplayRequiresExactSourceSchema`, `TestMigrationReceiptExpiresAfterReconciledReplacement`, `TestScopedAdminDoesNotGrantAttestationOrDataPlane`, `TestScopedAdminWithOwnerAttestationCannotUseAnotherCreatorsDataPlane`. Replacement-expiry test is in `internal/adapter/executioncontroller/production_lifecycle_test.go` in the pinned implementation candidate. `e2e/k8s_execution/ab_production_admin_test.go`: `TestKindExecutionProductionScopedAdministrator` exists (replacement/non-escalation, not all six routes); final Kind evidence PENDING. | | AC4.5 | `internal/adapter/executioncontroller/admin_scope_security_test.go`: `TestAdministratorScopeManifestValidation`, `TestAdministratorScopeDigestIsNormalizedAndAuthorityBound`; `internal/adapter/executioncontroller/admin_scope_test.go`: all-routes scope removal plus `TestScopedAdminEqualGenerationDriftFailsClosed`. Compatibility/upgrade human review PENDING. | | AC5.1 | `internal/adapter/executionclient/client_test.go`: exact reattachment test above; production `TestKindExecutionProductionReplicaLifecycle` and security rotation. | | AC5.2 | `internal/adapter/executioncontroller/production_lifecycle_test.go`: `TestReplacementPersistsTerminalProofBeforeRemovingPodFinalizer`; production replica lifecycle. | -| AC5.3 | `internal/adapter/executioncontroller/production_lifecycle_test.go`: `TestReplacementQuiescesAndPendingReferenceBlocksRetirement`. | +| AC5.3 | `internal/adapter/executioncontroller/production_lifecycle_test.go`: `TestReplacementQuiescesAndPendingReferenceBlocksRetirement`; `internal/adapter/executioncontroller/retained_delete_race_test.go`: `TestRetainedDeletePeerBetweenFinalizerUpdateAndDelete`, `TestRetainedDeleteDoesNotRemoveForeignCR`, `TestDirectCRDeleteStillBlocked` (offline race controls; final runtime evidence PENDING). | | AC5.4 | `internal/adapter/executionclient/reference_ambiguity_test.go`: `TestAmbiguousCommitIsRetainedAndNeverAbortedByCleanup`; production `TestKindExecutionProductionPendingDeleteOutageRecovery`. | | AC5.5, AC5.7 | `deploy/helm/mecatl-execution/chart_test.go`: `TestChartHasNoDeletionHook`; `deploy/helm/mecatl-execution/lifetime_test.go`: `TestChartRetainedLifetime` exercises lookup/adoption, missing history, foreign ownership, profile drift, and retained policies/ledgers offline. `e2e/k8s_execution/zz_production_helm_lifetime_test.go`: `TestKindExecutionProductionHelmLifetime` exercises actual upgrade/uninstall/reinstall and enforcement; final runtime evidence PENDING. | | AC5.6 | Deployment-documentation inspection and docs gates required on the final implementation; Kind is restart proof only. | | AC6.1–AC6.2 | `internal/app/remote_execution_test.go`: `TestRemoteExecutionRealFactoryCarriesPostureAndAttenuatedCatalog` includes a nonempty LOCAL Git snapshot control and REMOTE exclusion, poison-source and operator-global prompt controls, forbidden-tool errors and command/worktree discovery. `TestRemoteExecutionPreservesOperatorPermissionsUnderAuto` exercises global/explicit-file Deny and configured Ask, excluding project permissions, with allowed in-memory Shell controls. `TestRemoteDeploymentNoFSUsesLocalAttenuationWithoutProviderCall` retains the no-FS check. `internal/adapter/executionclient/service_integration_test.go`: `TestServiceUsesRealMTLSProviderStoreAndReleasesOnlyAfterDrain` rejects background Shell and actual Clear/Fork unsupported-selector calls with named errors, no destination publication, and unchanged source ref/history and provider references. | | AC6.3 | `internal/adapter/executioncontroller/store_lifecycle_test.go`: `TestReferenceTransactionsRetainUnknownAndNeverChangeSource`; production `TestKindExecutionProductionClearForkLifecycle`. | -| AC7.1 | `.github/workflows/k8s-e2e.yml` runs `task e2e:k8s:execution:production`; the historical reference job succeeded. The latest operator-reported job at `f7919f6c` passed 10/11 production tests and failed rotation's final read; it progressed past the earlier quota setup failure. Offline rotation evidence and read-only barrier controls are listed above. `internal/adapter/executioncontroller/legacy_fixture_kind_test.go` covers the bounded quota-accounting wait. Final amended-candidate run **PENDING**. | +| AC7.1 | `.github/workflows/k8s-e2e.yml` runs `task e2e:k8s:execution:production`; historical and operator-reported runs are distinguished above. The reported 11/11 pass at `7e498bd73` does not supersede later failures at `2e1aa6bbf`. `internal/adapter/executioncontroller/legacy_fixture_kind_test.go` covers the six-minute quota-accounting bound with tiny-context tests. `e2e/k8s_execution/holder_loss_test.go` covers the exact-owned terminal stimulus and complete kubelet evidence offline. Final amended-candidate run **PENDING**. | | AC7.2 | `e2e/k8s_execution/qualification_test.go`: `TestKindExecutionQualification`, run by the successful production profile. | -| AC7.3 | Production `TestKindExecutionProductionFailureArtifactBoundary`; workflow bounded uploads and ownership-scoped cleanup. | +| AC7.3 | Production `TestKindExecutionProductionFailureArtifactBoundary`; `deploy/mecatl-execution-kind/failure_evidence_test.go` tests closed-vocabulary projection and API-error redaction; `TestNativeWorkflowCleanupOwnership` proves evidence collection before failed cleanup. Workflow bounded uploads and ownership-scoped cleanup; final runtime evidence PENDING. | | AC7.4 | `e2e/k8s_execution/live_qualification_test.go`: `TestKindExecutionLiveQualification` exists; successful final native-provider OpenRouter evidence **PENDING**. | | AC7.5 | `e2e/k8s_execution/fixture/credentialloader/main_test.go`: `TestLoadCredentialAndRedaction`, `TestDeletePinsReceiptUIDAndLeavesReplacement`; these offline tests do not substitute for live execution. | From dfc17054a28a502ee6cd9f8939d6e646e957ceb5 Mon Sep 17 00:00:00 2001 From: Juan Antonio Osorio Date: Tue, 22 Sep 2026 11:22:40 +0300 Subject: [PATCH 16/16] docs(execution): record completed native candidate qualification Co-Authored-By: OpenAI --- .../acceptance/native-kubernetes-execution.md | 211 +++++++++++++----- docs/adr/0350-native-kubernetes-execution.md | 34 +-- 2 files changed, 176 insertions(+), 69 deletions(-) diff --git a/docs/acceptance/native-kubernetes-execution.md b/docs/acceptance/native-kubernetes-execution.md index b63cfdfc93..203bceb44b 100644 --- a/docs/acceptance/native-kubernetes-execution.md +++ b/docs/acceptance/native-kubernetes-execution.md @@ -9,12 +9,12 @@ **Expected tasks:** draft implementation decomposition remains run-local; no approved orchestration baseline. **Issue:** None — planning request has no linked issue. **Plan PR:** [#1579](https://github.com/stacklok/mecatl/pull/1579), draft on `plan/native-kubernetes-execution`; implementation [#1614](https://github.com/stacklok/mecatl/pull/1614) remains draft. -**Approved baseline:** absent. Plan worktree baseline `a5030545e87e81aad12fba4c49f0c9933a634b5e` and implementation source reference `9bb4d89b8aa497c9d2cd6aa4a94ab9262a913364` are evidence, not approval. +**Approved baseline:** absent. Plan worktree baseline `a5030545e87e81aad12fba4c49f0c9933a634b5e` and tested runtime source `2020e59934369e8f771059d0c173fa4cbf964c7e` are evidence, not approval. -The [implementation proof map](#implementation-proof-map) distinguishes existing draft coverage and -successful qualification from amendment proofs still pending. Implementation-only source paths below -refer to that pinned implementation reference; they are future code proofs on this documentation-only -branch, not claims that the files or tests already exist here. +The [implementation proof map](#implementation-proof-map) records completed candidate runtime +qualification separately from open human reviews and limited behavioral proof. Implementation-only +source paths below refer to that tested runtime SHA; they are future code proofs on this +documentation-only branch, not claims that the files or tests already exist here. The smallest demonstrable slice is an independently deployed execution environment provider service with its own controller and executors. Mecak8s is an optional client: its adapter obtains a persistent @@ -43,10 +43,10 @@ ADR 0048 continues to govern storage-free mecak8s; this separate provider does n ## Interface contract -This documentation-only amendment traces the draft implementation at `9bb4d89b8` against the -proposed contract. Authored specifications, advisory review, and passing CI do not constitute human -approval. Scoped administration, continuing provisioning retries, and Helm lifecycle proofs exist in -that source; final runtime qualification and the unchecked reviews above remain pending. +This documentation-only receipt traces the draft implementation at tested runtime SHA +`2020e59934369e8f771059d0c173fa4cbf964c7e` against the proposed contract. Candidate runtime +qualification is complete; authored specifications and passing CI do not constitute human +approval. The unchecked API, schema, security, and release-maturity decisions remain open. - **gRPC / protobuf:** Public Harness protobuf is unchanged. The private `mecatl.execution.v1.ExecutionProviderService` uses mandatory mTLS, a `host:port` endpoint, protocol identifier `execution-grpc/1`, and an 8 MiB message bound. The exact wire schema is the implementation reference's `contracts/proto/mecatl/execution/v1/execution.proto`, generated with `task generate`. The RPC/field summary below is the review contract; there is no HTTP/JSON compatibility endpoint. - **Exported Go APIs / interfaces:** No `engine/` exported API change. `tool.Environment` remains identity + Workspace + independent ReadLedger + optional bound CommandRunner (`engine/tool/environment.go`). Host-only seams in the implementation's `internal/adapter/server/placement.go` are specified below; Kubernetes dependencies stay outside the engine. @@ -290,9 +290,9 @@ or a throwaway allocation. CLI and real composition tests below exercise that bo - AC1.1: With execution disabled, mecak8s creates no Kubernetes execution client, informer, goroutine, CRD, RBAC, controller, executor, PVC, or Pod, and performs no execution API call. - verify: `TestDisabledExecutionStartupDoesNotContactExecutionOrKubernetes`, `TestDisabledCompositionPreservesAllocationsAndIndependentReconciliation`; offline CLI `run()` reaches listener setup with poison execution TLS paths and a test-only kubeconfig/API endpoint untouched. That poisoned-endpoint test guards disabled startup contact; the lifecycle fixture's fake clients are not injected into app.Build, so its action log proves only independent reconciliation, not absence of an app-created client. Constructor/informer absence also follows from the disabled CLI branch and independent provider packaging (not goroutine counts). - AC1.2: The provider service deploys, restarts, and reconciles independently of mecak8s. The default mecak8s chart has no dependency on the provider chart and installs no provider resource; enabling the client adapter requires explicit endpoint/profile configuration. The adapter needs no Pod/PVC/controller management RBAC. - - verify: `TestChartRetainsCRDAndDoesNotGrantSecretAPI`, `TestKindExecutionProductionReplicaLifecycle`; inspection — default chart dependency/RBAC review and final Kind execution PENDING. + - verify: `TestChartRetainsCRDAndDoesNotGrantSecretAPI`, `TestKindExecutionProductionReplicaLifecycle`; inspection — default chart dependency/RBAC review PENDING; final Kind execution passed at the tested runtime SHA. - AC1.3: Preflight validates configured profiles and endpoint/provider compatibility without allocating an environment. Allocation occurs only at an authorized session-binding operation; enabled configuration that is incomplete fails startup before allocation, and a configured but unavailable endpoint fails validation clearly with no local fallback. - - verify: `TestExecutionPreflightRejectsIncompleteAndIncompatibleCLI`, `TestRemoteExecutionPreflightNeverAllocates`, `TestHandlerValidateIsReadOnlyAndEnsureIdempotent`, `TestGRPCReadinessAndMTLSAreMandatory`, `TestKindExecutionQualification`; offline CLI/composition and signed-handler validation cover rejection without allocation; final Kind run PENDING. + - verify: `TestExecutionPreflightRejectsIncompleteAndIncompatibleCLI`, `TestRemoteExecutionPreflightNeverAllocates`, `TestHandlerValidateIsReadOnlyAndEnsureIdempotent`, `TestGRPCReadinessAndMTLSAreMandatory`, `TestKindExecutionQualification`; offline CLI/composition and signed-handler validation cover rejection without allocation; final Kind run passed at the tested runtime SHA. - AC1.4: Disabling the mecak8s client integration neither adopts nor deletes existing `ExecutionEnvironment` CRs, PVCs, or Pods and does not stop the separate provider's reconciliation for its existing allocations or other authorized clients. - verify: `TestDisabledCompositionPreservesAllocationsAndIndependentReconciliation`, `TestStartupDoesNotFenceLivePeerOperation`; the separate reconciler advances its retained allocation to Ready after an unrelated disabled Build/run/Close. Its fake clients are test-owned, not injected into app.Build; unchanged objects/actions are a limited independent-lifecycle oracle, not a client-construction guard or cluster restart run. `TestDisabledExecutionStartupDoesNotContactExecutionOrKubernetes` supplies the actual poisoned-endpoint CLI guard. @@ -303,13 +303,13 @@ with an ephemeral Pod, as decided in [draft ADR 0350](../adr/0350-native-kuberne **Acceptance:** - AC2.1: Repeating the same authorized ensure request with the same final idempotency identity returns the same environment generation and does not create a second PVC or Pod. - - verify: `TestHandlerValidateIsReadOnlyAndEnsureIdempotent`, `TestKindExecutionQualification` (final Kind execution PENDING). + - verify: `TestHandlerValidateIsReadOnlyAndEnsureIdempotent`, `TestKindExecutionQualification` (final Kind execution passed at the tested runtime SHA). - AC2.2: A stale/different profile, digest, storage request, generation, or owner fails with the final reviewed conflict/precondition error and never adopts or mutates an unrelated allocation. - verify: `TestStoreEnsureUsesStableLookupAndRejectsFingerprintDrift`, `TestReconcileRefusesForeignExistingPVCWithoutPersistingUID`. - AC2.3: Only operator-configured profile references select digest-pinned images and storage; public clients/models cannot submit arbitrary images, Pod specs, paths, URLs, source credentials, or Kubernetes object names. - verify: `TestLoadProfilesStrictAndDigestPinned`, `TestReconcileCreatesTokenlessNonRootPodAndRetainedPVC`; inspection — public request/schema authority review PENDING. - AC2.4: After transient PVC or Pod creation failure, including prolonged quota saturation, an existing allocation converges when the failure clears without unrelated CR/reference changes, restart, or manual reconcile. Retries remain rate-limited and continue while the object exists; wrong ownership and missing/mismatched authoritative UIDs never trigger adoption or replacement. - - verify: `TestProvisioningWorkerConvergesWithoutAnotherEvent`, `TestProvisioningPreservesCreateAndConditionErrors`, `TestProvisioningRetriesNeverReplaceAuthoritativeResources`, `TestKindExecutionProductionQuotaSaturation` (Kind quota-restoration proof exists; final execution PENDING). + - verify: `TestProvisioningWorkerConvergesWithoutAnotherEvent`, `TestProvisioningPreservesCreateAndConditionErrors`, `TestProvisioningRetriesNeverReplaceAuthoritativeResources`, `TestKindExecutionProductionQuotaSaturation` (Kind quota-restoration proof exists; final execution passed at the tested runtime SHA). ### Scenario 3 — Existing file tools and Shell are transparent but honest @@ -319,13 +319,13 @@ The adapter preserves the immutable environment and independent read ledger **Acceptance:** - AC3.1: A fresh environment can create, read, conditionally edit, replace, copy, move, remove, list, grep, and glob a small Go fixture with unchanged tool schemas and the existing read-before-edit/version behavior where that contract applies. - - verify: `TestRemoteWorkspaceConformanceThroughSignedGRPC`, `TestRemoteFileToolsPreserveLedgerAndNamespaceContracts`, `TestKindExecutionQualification`; both shared Workspace conformance suites and all nine actual file tools run over loopback mTLS/signed gRPC with the real executor in a test-owned directory. Kind invokes the added remote-tool matrix, but final Kind execution remains PENDING. + - verify: `TestRemoteWorkspaceConformanceThroughSignedGRPC`, `TestRemoteFileToolsPreserveLedgerAndNamespaceContracts`, `TestKindExecutionQualification`; both shared Workspace conformance suites and all nine actual file tools run over loopback mTLS/signed gRPC with the real executor in a test-owned directory. Kind invokes the added remote-tool matrix; final Kind execution passed at the tested runtime SHA. - AC3.2: Read/Edit/Write preserve their applicable recorded-read and CreateFile/ReplaceFile CAS contracts; Copy/Move/Remove preserve their own positive conformance contracts and do not require an unrelated content read ledger/CAS precondition. - verify: `TestRemoteWorkspaceConformanceThroughSignedGRPC`, `TestRemoteFileToolsPreserveLedgerAndNamespaceContracts`; shared tests cover create/replace races, stale/zero versions, namespace positives and non-clobbering failures; actual tools cover unread/stale Edit/Write refusal and successful replacement. Namespace operations have no unrelated content-read precondition. - AC3.3: File access is physically confined through symlink traversal rather than lexical checks alone, and every supported read/mutate/list/search/command request is authorized for the exact environment; stale, expired, revoked, or wrong-environment grants are denied. Detached status/cancel requests authorize before returning Unimplemented; this slice exposes no command-stream RPC. - verify: `TestEveryFileOperationRequiresExactCurrentGrant`, `TestFileOperationsNeverTraverseExternalSymlink`, `TestRemoteWorkspaceConformanceThroughSignedGRPC`, `TestSecurityManagerGuardsEveryRPCOnExistingConnection`, `TestRevokeEnvironmentFencesOldClaimWithoutChangingExecutionEpoch`, `TestCommandStatusIsUnimplementedOnlyAfterAuthorization`; every supported file operation has a valid positive control before wrong-client/owner/environment/revision/epoch/operation, expired, and revoked-nonce rejection at the shared handler gate. Executor tests cover symlink source/destination traversal and search leakage; durable generation revocation remains covered at the store boundary. - AC3.4: Shell is bound to the same environment namespace as Workspace. Cancellation distinguishes requested cancellation, acknowledged complete process termination, and externally proven compute/storage fencing; namespace teardown includes detached descendants or fails closed. Output is capped and valid UTF-8 repaired, and Shell-created file changes are not falsely claimed to participate in Workspace CAS. - - verify: `TestForegroundCommandReturnsReceiptAndTimeoutTerminatesDescendants`, `TestSuccessfulDetachedChildIsKilledBeforeTerminalReceipt`, `TestCancelledStoreOperationStaysActiveUntilBackendStopsThenFences`, `TestNativeShellCapsCombinedOutputAndRepairsUTF8`, `TestNativeShellWriteInvalidatesRecordedRead`, `TestProviderThroughRealGRPCSignedHandlerRefreshesAndReattachesExactly`, `TestServiceUsesRealMTLSProviderStoreAndReleasesOnlyAfterDrain`, `TestKindExecutionProductionHolderLossFencesActiveOperation`; the native helper executes oversized stdout/stderr and split malformed-byte output with combined-cap/truncation assertions, and writes after a recorded Read so actual Edit rejects the stale version without overwriting Shell's content. Private protobuf bytes may remain malformed; signed gRPC runner and model/client-event assertions cover text repair separately. Final Kind execution PENDING. + - verify: `TestForegroundCommandReturnsReceiptAndTimeoutTerminatesDescendants`, `TestSuccessfulDetachedChildIsKilledBeforeTerminalReceipt`, `TestCancelledStoreOperationStaysActiveUntilBackendStopsThenFences`, `TestNativeShellCapsCombinedOutputAndRepairsUTF8`, `TestNativeShellWriteInvalidatesRecordedRead`, `TestProviderThroughRealGRPCSignedHandlerRefreshesAndReattachesExactly`, `TestServiceUsesRealMTLSProviderStoreAndReleasesOnlyAfterDrain`, `TestKindExecutionProductionHolderLossFencesActiveOperation`; the native helper executes oversized stdout/stderr and split malformed-byte output with combined-cap/truncation assertions, and writes after a recorded Read so actual Edit rejects the stale version without overwriting Shell's content. Private protobuf bytes may remain malformed; signed gRPC runner and model/client-event assertions cover text repair separately. Final Kind execution passed at the tested runtime SHA. ### Scenario 4 — Caller isolation and execution ownership survive failures @@ -335,13 +335,13 @@ therefore requires independent environment execution fencing. **Acceptance:** - AC4.1: Ordinary creator client A cannot discover, attach, execute in, stream from, cancel, or retire creator client B's environment; hidden and absent allocations are indistinguishable at the public boundary. The explicit scoped administrative exception in AC4.4 permits only its named administrative RPCs. - - verify: `TestClientScopedIntentListReturnsAttestedOwnerOnlyToOwningClient`, `TestScopedAdminAllRoutesOverMTLS`, `TestScopedAdminWithOwnerAttestationCannotUseAnotherCreatorsDataPlane`, `TestKindExecutionQualification` (final Kind execution PENDING). + - verify: `TestClientScopedIntentListReturnsAttestedOwnerOnlyToOwningClient`, `TestScopedAdminAllRoutesOverMTLS`, `TestScopedAdminWithOwnerAttestationCannotUseAnotherCreatorsDataPlane`, `TestKindExecutionQualification` (final Kind execution passed at the tested runtime SHA). - AC4.2: Every command is authorized for one immutable environment revision and a distinct transient execution-fence epoch; lease loss or grant revocation prevents new commands and cancels/fences active work without exposing capability credentials to the workload or another caller's output. Same-Pod placement alone is not a security boundary. - - verify: `TestGrantRoundTripAndExactBindings`, `TestRevokeEnvironmentFencesOldClaimWithoutChangingExecutionEpoch`, `TestAcquireRunAndRevokeUseResourceVersionCAS`, `TestSecurityRotationPinnedReplicaReadConvergesBeforeDispatch`, `TestSecurityRotationMutableNamesPoisonSameGeneration`, `TestSecurityRotationImmutableNamesHandleProjectionSkew`, `TestReadCurrentAuthorityContentRetriesOnlyPreDispatchLag`, `TestReadCurrentAuthorityContentBoundAndCancellation`, `TestKindExecutionProductionSecurityRotation`, `TestKindExecutionProductionHolderLossFencesActiveOperation` (final Kind execution PENDING). + - verify: `TestGrantRoundTripAndExactBindings`, `TestRevokeEnvironmentFencesOldClaimWithoutChangingExecutionEpoch`, `TestAcquireRunAndRevokeUseResourceVersionCAS`, `TestSecurityRotationPinnedReplicaReadConvergesBeforeDispatch`, `TestSecurityRotationMutableNamesPoisonSameGeneration`, `TestSecurityRotationImmutableNamesHandleProjectionSkew`, `TestReadCurrentAuthorityContentRetriesOnlyPreDispatchLag`, `TestReadCurrentAuthorityContentBoundAndCancellation`, `TestKindExecutionProductionSecurityRotation`, `TestKindExecutionProductionHolderLossFencesActiveOperation` (final Kind execution passed at the tested runtime SHA). - AC4.3: Timeout, Pod deletion, controller restart, Lease timeout, or network partition alone never authorizes a replacement executor while an old writer may run; unknown fencing state fails closed and requires the reviewed manual/external fencing path. - - verify: `TestExpiredOperationLeaseFencesWithoutClearingIdentity`, `TestRecoverMissingPodRemainsFenceUnknown`, `TestRecoveredTerminalProofCanStartExactReplacement`, `TestKindExecutionProductionHolderLossFencesActiveOperation` (final Kind execution PENDING); inspection — external fencing runbook review PENDING. + - verify: `TestExpiredOperationLeaseFencesWithoutClearingIdentity`, `TestRecoverMissingPodRemainsFenceUnknown`, `TestRecoveredTerminalProofCanStartExactReplacement`, `TestKindExecutionProductionHolderLossFencesActiveOperation` (final Kind execution passed at the tested runtime SHA); inspection — external fencing runbook review PENDING. - AC4.4: A distinct `administrator:true` client with `administratorFor:[creatorURI]` can administer that creator's environment through all six administrative RPCs, including migration, revocation, and retained deletion, while preserving immutable creator identity and exact owner/revision/epoch/UID/operation checks. A second creator, wrong owner, stale identity, or replay with changed inputs is denied without existence disclosure. Self-admin compatibility remains explicit; no Files/Shell/attach/run/reference or MayAttestOwner authority is gained. - - verify: `TestScopedAdminAllRoutesOverMTLS`, `TestScopedAdminCASRetryRechecksSubject`, `TestScopedAdminMigrationReceiptRetainsUIDPreconditions`, `TestMigrationCompletedCASReplayRequiresExactSourceSchema`, `TestMigrationReceiptExpiresAfterReconciledReplacement`, `TestScopedAdminDoesNotGrantAttestationOrDataPlane`, `TestScopedAdminWithOwnerAttestationCannotUseAnotherCreatorsDataPlane`, `TestKindExecutionProductionScopedAdministrator`; inspection — replacement-expiry regression is present in the implementation candidate; final Kind execution PENDING (Kind covers replacement/non-escalation, not all six routes). + - verify: `TestScopedAdminAllRoutesOverMTLS`, `TestScopedAdminCASRetryRechecksSubject`, `TestScopedAdminMigrationReceiptRetainsUIDPreconditions`, `TestMigrationCompletedCASReplayRequiresExactSourceSchema`, `TestMigrationReceiptExpiresAfterReconciledReplacement`, `TestScopedAdminDoesNotGrantAttestationOrDataPlane`, `TestScopedAdminWithOwnerAttestationCannotUseAnotherCreatorsDataPlane`, `TestKindExecutionProductionScopedAdministrator`; inspection — replacement-expiry regression is present in the implementation candidate; final Kind execution passed at the tested runtime SHA (Kind covers replacement/non-escalation, not all six routes). - AC4.5: Strict `administratorFor` validation and canonical digest inclusion prevent ambiguous scope. Removal at a higher security generation takes effect on existing connections and receipt replay; an equal-generation scope edit fails closed. Old binaries reject the field and the documented quiesced upgrade preserves authority history. - verify: `TestAdministratorScopeManifestValidation`, `TestAdministratorScopeDigestIsNormalizedAndAuthorityBound`, `TestScopedAdminAllRoutesOverMTLS`, `TestScopedAdminEqualGenerationDriftFailsClosed`; inspection — old-binary rejection/quiesced-upgrade compatibility review PENDING. @@ -352,19 +352,19 @@ and remain separate from the session-only lease contract (`engine/port/lease.go: **Acceptance:** - AC5.1: After mecak8s and controller restart, exact reattachment to the persisted environment revision restores the same PVC data and a correctly bound runner; an unavailable/mismatched generation fails with no default/local fallback. - - verify: `TestProviderThroughRealGRPCSignedHandlerRefreshesAndReattachesExactly`, `TestKindExecutionQualification`, `TestKindExecutionProductionReplicaLifecycle` (final Kind restart/data-survival execution PENDING). + - verify: `TestProviderThroughRealGRPCSignedHandlerRefreshesAndReattachesExactly`, `TestKindExecutionQualification`, `TestKindExecutionProductionReplicaLifecycle` (final Kind restart/data-survival execution passed at the tested runtime SHA). - AC5.2: Executor Pod replacement preserves the logical environment and PVC while current Pod UID may change without a spec `metadata.generation` change; `observedGeneration` records processed spec generation, and status records current Pod/PVC UID references without using them as durable session identity. - - verify: `TestReplacementPersistsTerminalProofBeforeRemovingPodFinalizer`, `TestStaleReplacementObservationCannotOverwriteCompletion`, `TestKindExecutionProductionReplicaLifecycle`; inspection — observedGeneration/spec-generation distinction review and final Kind execution PENDING. + - verify: `TestReplacementPersistsTerminalProofBeforeRemovingPodFinalizer`, `TestStaleReplacementObservationCannotOverwriteCompletion`, `TestKindExecutionProductionReplicaLifecycle`; inspection — observedGeneration/spec-generation distinction review PENDING; final Kind execution passed at the tested runtime SHA. - AC5.3: The lifecycle matrix is explicit: removal of one reference retains the environment; removal of the last reference retains it by default; committed data retires only through explicit authorized retirement; a retiring environment rejects new bindings and successors. An unreadable store/reference is not an orphan and is retained. - - verify: `TestReplacementQuiescesAndPendingReferenceBlocksRetirement`, `TestReferenceTransactionsRetainUnknownAndNeverChangeSource`, `TestKindExecutionProductionPendingDeleteOutageRecovery` (final Kind execution PENDING). + - verify: `TestReplacementQuiescesAndPendingReferenceBlocksRetirement`, `TestReferenceTransactionsRetainUnknownAndNeverChangeSource`, `TestKindExecutionProductionPendingDeleteOutageRecovery` (final Kind execution passed at the tested runtime SHA). - AC5.4: A pending allocation whose session association might have committed is not garbage-collected merely after TTL; collection waits for conclusive reconciliation that publication is absent. Explicit retirement of a live/shared environment is refused while any live reference exists unless a separately reviewed retire/quiesce policy authorizes it. - verify: `TestAmbiguousCommitIsRetainedAndNeverAbortedByCleanup`, `TestReplacementQuiescesAndPendingReferenceBlocksRetirement`, `TestReferenceIntentReconciliationIsOwnerAndRefExact`. - AC5.5: Default chart uninstall retains runtime CRs/PVCs and preserves workload network isolation and security authority history while executors survive. Retained security/capacity ledgers and required configuration outlive the provider Deployment. Deletion of CRDs with live resources is destructive and unsupported; no destructive hook or force cleanup is provided. Owner references or retention annotation strings alone are not lifecycle proof. - - verify: `TestChartHasNoDeletionHook`, `TestChartRetainedLifetime`, `TestKindExecutionProductionHelmLifetime` (offline rendering is partial; actual Helm lifecycle execution PENDING). + - verify: `TestChartHasNoDeletionHook`, `TestChartRetainedLifetime`, `TestKindExecutionProductionHelmLifetime` (offline rendering is partial; actual Helm lifecycle execution passed at the tested runtime SHA). - AC5.6: PVC persistence is not documented as node-disaster recovery; kind host-local storage proves restart survival only, and a dead or unobservable node is outside that positive guarantee. - verify: inspection — human review of deployment documentation; `task docs` checks links and structure after authorized tracked changes - AC5.7: Real Helm install/compatible upgrade/uninstall/reinstall of the same release and namespace preserves exact runtime identity, PVC contents, NetworkPolicy enforcement during provider absence, security high-water/key history, and capacity reservations. Safe adoption validates ownership; missing ledgers with retained allocations, foreign resources, stale authority, arbitrary downgrade, or incompatible profile/schema fail closed without overwrite. CRD upgrades remain manual. - - verify: `TestChartRetainedLifetime`, `TestSecurityLedgerRejectsKeyVersionRollbackAcrossRestart`, `TestKindExecutionProductionHelmLifetime` (real Helm lifecycle test exists; final execution PENDING). + - verify: `TestChartRetainedLifetime`, `TestSecurityLedgerRejectsKeyVersionRollbackAcrossRestart`, `TestKindExecutionProductionHelmLifetime` (real Helm lifecycle test exists; final execution passed at the tested runtime SHA). ### Scenario 6 — Remote catalog and project-source boundaries are explicit @@ -378,7 +378,7 @@ in local composition (`internal/app/project_ingestion.go:20`) and AGENTS discove - AC6.2: Schedules remain excluded. Background commands, isolated Subagent/Parallel/Team delegation, SkillDraft, Git/source operations, and unsupported successor modes are rejected with named capability/precondition errors rather than running locally or silently omitting work; `no-fs` remains unchanged. - verify: `TestRemoteExecutionRealFactoryCarriesPostureAndAttenuatedCatalog`, `TestRemoteDeploymentNoFSUsesLocalAttenuationWithoutProviderCall`, `TestExecutionPreflightRejectsIncompleteAndIncompatibleCLI`, `TestServiceUsesRealMTLSProviderStoreAndReleasesOnlyAfterDrain`, `TestCommandStatusIsUnimplementedOnlyAfterAuthorization`; real factory tests invoke Subagent/Parallel/Team/SkillDraft/Schedule and require model-visible errors, reject worktree discovery, and withhold command listing/expansion instead of opening host paths. The native mTLS/store integration rejects background Shell with its named runner-capability error and no executor dispatch, while foreground Shell succeeds. Actual Service Clear/Fork calls with a nonempty unsupported worktree selector return `ErrInvalidPlacementSelection`, publish no destination, and preserve source ref/history and provider references. A direct native Bind with a valid binding ID distinguishes the unsupported-selector gate from missing binding identity or malformed Service input. - AC6.3: Clear/Fork use environment exclusion and reference reservation/publication while preserving same-remote-`EnvironmentRef` successor semantics and the source on publication failure. Clear/Fork are not workspace cloning; no public Harness field is added and no model chooses an environment/profile. - - verify: `TestReferenceTransactionsRetainUnknownAndNeverChangeSource`, `TestKindExecutionProductionClearForkLifecycle` (final Kind execution PENDING); inspection — public API/profile authority review PENDING. + - verify: `TestReferenceTransactionsRetainUnknownAndNeverChangeSource`, `TestKindExecutionProductionClearForkLifecycle` (final Kind execution passed at the tested runtime SHA); inspection — public API/profile authority review PENDING. ### Scenario 7 — kind proves an offline coding flow; live qualification is separate @@ -401,8 +401,8 @@ mandatory; cleanup failures fail the job. Always attempt deletion of only the re validated CI-owned cluster and credential file. Upload only the bounded `live-summary.json` and sanitized qualification status, never credentials, receipts, kubeconfigs, PKI, Helm values, or transcripts. A 100-minute job bound covers production, mock rerun, live smoke, -builds, and cleanup; it is not a dollar cap. Final-candidate runtime evidence and independent -security review remain PENDING. +builds, and cleanup; it is not a dollar cap. Candidate runtime evidence is recorded below; +independent security review remains PENDING. The rotation restart proof must compare the observed grant generation and replayed revoke receipt exactly with the original revoke result. A current, successful post-restart claim @@ -412,24 +412,25 @@ arbitrary error from a released/expired claim, retired key, or unavailable provi **Acceptance:** - AC7.1: `task e2e:k8s:execution` and the enforcing-CNI `task e2e:k8s:execution:production` run on the supported generic toolchain, including CI, with a unique owned kind cluster, a scratch kubeconfig, and explicit `--kubeconfig`/context on every command. They deploy the separate chart plus explicitly connected mecak8s and use only the deterministic mock provider. The final production-profile run uses Calico and exercises negative network isolation, rotation/revocation, replica/lifecycle recovery, holder loss, quota recovery, pending delete, Clear/Fork, migration, sanitized artifacts, and the amendment regressions. - - verify: `TestLegacyFixtureWaitsForQuotaAccountingBeforeCreate`, `TestKindExecutionQualification`, `TestKindExecutionProductionNetworkPolicyEnforced`, `TestKindExecutionProductionSecurityRotation`, `TestKindExecutionProductionReplicaLifecycle`, `TestKindExecutionProductionHolderLossFencesActiveOperation`, `TestKindExecutionProductionQuotaSaturation`, `TestKindExecutionProductionPendingDeleteOutageRecovery`, `TestKindExecutionProductionClearForkLifecycle`, `TestKindExecutionProductionCompatiblePrototypeMigration`, `TestKindExecutionProductionScopedAdministrator`, `TestKindExecutionProductionHelmLifetime` (names exist; final production Kind+Calico run PENDING). + - verify: `TestLegacyFixtureWaitsForQuotaAccountingBeforeCreate`, `TestKindExecutionQualification`, `TestKindExecutionProductionNetworkPolicyEnforced`, `TestKindExecutionProductionSecurityRotation`, `TestKindExecutionProductionReplicaLifecycle`, `TestKindExecutionProductionHolderLossFencesActiveOperation`, `TestKindExecutionProductionQuotaSaturation`, `TestKindExecutionProductionPendingDeleteOutageRecovery`, `TestKindExecutionProductionClearForkLifecycle`, `TestKindExecutionProductionCompatiblePrototypeMigration`, `TestKindExecutionProductionScopedAdministrator`, `TestKindExecutionProductionHelmLifetime` (final production Kind+Calico run passed at the tested runtime SHA). - AC7.2: The mock flow seeds a small Go fixture through file tools, asks the harness to implement a function/tests, runs `go test` through bound Shell, and verifies the persisted artifact and restart/reattach behavior. - - verify: `TestKindExecutionQualification` (coding/artifact/restart assertions exist; final Kind execution PENDING). + - verify: `TestKindExecutionQualification` (coding/artifact/restart assertions exist; final Kind execution passed at the tested runtime SHA). - AC7.3: Failure artifacts are bounded and sanitized; cleanup targets only the recorded owned cluster. Destructive live cleanup requires confirmation and never mutates ambient kubeconfig/context. - - verify: `TestKindExecutionProductionFailureArtifactBoundary`, `TestLifetimeOutputRejectsOversizedArtifacts`; inspection — owned-cluster cleanup/context script review and final Kind execution PENDING. + - verify: `TestKindExecutionProductionFailureArtifactBoundary`, `TestLifetimeOutputRejectsOversizedArtifacts`; inspection — owned-cluster cleanup/context script review PENDING; final Kind execution passed at the tested runtime SHA. - AC7.4: `task e2e:k8s:execution:live` is a separate explicit native-provider OpenRouter coding qualification, required to complete this stack after final deterministic qualification. It proves model-driven file/tool execution, independently verified persisted workspace artifacts, and a successful bound `go test` command. It uses verified nonsecret endpoint/protocol/model configuration in an authenticated operator-controlled cluster, then restores mock configuration and deletes only the run-owned Secret by recorded UID. It is not an always-on live CI dependency and claims no hard dollar cap from token/run limits. - - verify: `TestKindExecutionLiveQualification`; inspection — record successful native-provider live evidence for the final implementation candidate (PENDING, not satisfied by generic mecak8s live CI) + - verify: `TestKindExecutionLiveQualification`; inspection — successful exact-head native-provider live evidence is recorded in the candidate completion receipt (not generic mecak8s live CI). - AC7.5: A trusted runtime-only loader may consume the approved credential directly from its protected file channel and create a narrowly scoped HARNESS-only Secret through a protected channel before child processes or tools can inherit it. The agent never accesses the file or Secret; failures are redacted, and the credential never appears in model/tool output, argv, Helm values, disk manifests, git, logs, artifacts, or the execution workload. Mock tests do not access it. - - verify: `TestLoadCredentialAndRedaction`, `TestDeletePinsReceiptUIDAndLeavesReplacement`, `TestSecretReferencesFindEveryCredentialEscape`, `TestExpectedHarnessCredentialReferenceIsExact`, `TestKindExecutionLiveQualification`; inspection — offline synthetic-credential tests do not establish live handling; final authorized native live evidence PENDING. + - verify: `TestLoadCredentialAndRedaction`, `TestDeletePinsReceiptUIDAndLeavesReplacement`, `TestSecretReferencesFindEveryCredentialEscape`, `TestExpectedHarnessCredentialReferenceIsExact`, `TestKindExecutionLiveQualification`; inspection — offline synthetic-credential tests do not establish live handling; authorized exact-head native live and cleanup evidence is recorded in the candidate completion receipt. Human security review remains PENDING. ## Implementation proof map -Source/test locations in this table and the `verify:` lines refer to implementation commit -`773a203df5de9b27612d190a774a857590702cbd`, not files supplied by this plan branch. -This is the intended candidate for the normal merge of this proof-map update into implementation. +Source/test locations in this table and the `verify:` lines refer to tested runtime commit +`2020e59934369e8f771059d0c173fa4cbf964c7e`, not files supplied by this plan branch. The labels name actual tests; source resolution alone is not successful execution or complete -behavioral coverage. Offline boundaries and remaining partial proof are stated per AC; -PENDING runtime evidence still requires final-candidate execution. +behavioral coverage. Offline boundaries and remaining partial proof are stated per AC. +The following completion receipt supersedes historical pending-runtime status, not human review +or missing behavioral coverage. Documentation-only descendants retain this tested-runtime attribution; +they are not represented as newly live-qualified heads. The migration-receipt replacement-expiry regression `TestMigrationReceiptExpiresAfterReconciledReplacement` is already present in that candidate's `internal/adapter/executioncontroller/production_lifecycle_test.go`. Run `actrace --strict --plan ` from the implementation worktree @@ -437,6 +438,61 @@ for source resolution, then rerun after its normal plan merge. Draft status is i strict mode gates landed plans only, so inspect the native report for missing proofs as well. A row naming a unit/fake-client test establishes only that boundary; it does not prove the entire AC. +### Candidate completion receipt + +The operator supplied the following verified results for runtime SHA +`2020e59934369e8f771059d0c173fa4cbf964c7e`. This documentation-only receipt performs no +new live run, credential access, or cluster operation. + +- **Required native qualification: SUCCESS.** Manual exact-head run + [35700303737](https://github.com/stacklok/mecatl/actions/runs/35700303737) passed all + 11 production Kind+Calico scenarios, then the real OpenRouter coding flow using + `anthropic/claude-haiku-4.5` (30,521 input / 713 output tokens). The model used Write + and Shell and ran exactly `go test ./...` with exit 0. An independent typed-gRPC + verifier attached to the exact persisted ref, acquired run ownership, verified both + generated files including the nonce and tests, and ran its own exact `go test ./...` + with exit 0. The safe artifact is `live-summary.json`. +- **Cleanup and credential boundary: passed.** The filtered log records + `cleanup verification passed: mock harness restored; run-scoped Secret receipt cleared`. + Explicit workflow private-file deletion and the owned-cluster absence check also passed. + No actual credential bytes were accessed by the agent or tools. The offered local file + was not used; the CI secret reached the harness through the trusted loader. +- **Current PR checks: 31 success, 8 skipped, 0 failed, 0 pending.** Successful runs: + [ordinary CI 35700310195](https://github.com/stacklok/mecatl/actions/runs/35700310195), + [native/Kind 35700310187](https://github.com/stacklok/mecatl/actions/runs/35700310187), + [ordinary live 35700310242](https://github.com/stacklok/mecatl/actions/runs/35700310242), + [performance 35700310255](https://github.com/stacklok/mecatl/actions/runs/35700310255), and + [Deslop 35700310122](https://github.com/stacklok/mecatl/actions/runs/35700310122). + The PR synthetic merge SHA is `c2a9e9a87f36dfbaea6658c1491426847320aecd`, distinct + from the manual run's exact runtime head. Ordinary live CI is complementary, not the + native qualification proof. Full-race CI remains skipped while draft; no execution of + that lane is claimed. Earlier runtime revisions passed local full `task test` with race; + the latest small verifier changes passed targeted race tests. The earlier unrelated + server-lease-test flake did not recur; this is not a claim that it was fixed. + +**Resolution of the historical live blockers:** removing the live-only helper-image rewrite +preserves the already-qualified Kind storage configuration. That rewrite selected executor UID +65532 for a storage helper which must create a directory beneath a root-owned 0755 parent. +The readiness hang is gone; session creation completed in 8 seconds in the successful run. +The independent verifier's later Ensure call used the wrong operation identity for an already +published binding and was correctly rejected. It now reuses read-only `environmentForBinding` +lookup and typed exact-ref Attach plus run acquisition; provider semantics were not weakened. +Instrumented debug-stage capture runs before restoration with a safe bounded allowlist. +The forensic history below retains what each earlier observation did and did not establish. + +**Remaining gates:** human API/schema/security approval, release-maturity designation, +independent final panel review and final human review/merge remain open. Passing scenarios do +not expand their assertion scope: in particular, the scoped-admin Kind case is replacement/ +non-escalation, not all six routes, and the disabled-composition fake-client limitations remain. +Strict trace's 32 resolved ACs are a source map, not blanket behavioral acceptance. Native live +qualification is no longer pending or a blocker. Neither PR nor ADR 0350 is approved or shipped. + +### Historical qualification and repairs + +Historical failures below are retained for diagnosis. Their pending-run statements describe +those revisions only; the candidate receipt above supplies the subsequent successful runtime +resolution without assigning an unproved root cause to every earlier failure. + **Prior offline verification:** implementation `0df361480b2b9c1b9a02ca5c7f55362d0e257966` passed targeted race packages (execution client/controller/executor/protocol, mecak8s, and remote composition), the @@ -514,13 +570,14 @@ offline demo pass. Tagged whole-fixture lint exposes unrelated pre-existing find gate against baseline `2d32864cee8378939cb55137e1ab369e70a18349` passes with zero issues. These results do not replace final runtime qualification; no live inference or cluster run is claimed. -**Current repair evidence (offline, after `0e8c8a5ca`):** the operator reports all +**Historical repair evidence (offline, after `0e8c8a5ca`):** the operator reports all 11 native production tests passing at `7e498bd73`, followed by production failures at `2e1aa6bbf` before live qualification. Manual run [35659529618](https://github.com/stacklok/mecatl/actions/runs/35659529618) failed before provider credential staging; no provider credentials were used. These reports were not independently re-fetched during this repair. The production timeouts -remain unresolved incidents, not evidence that either source defect below caused them. +remain historically unexplained, not active qualification blockers after the successful candidate +run and not evidence that either source defect below caused them. - `TestRetainedDeletePeerBetweenFinalizerUpdateAndDelete` deterministically failed when a second reconciler re-added the environment finalizer between Update and @@ -549,9 +606,9 @@ remain unresolved incidents, not evidence that either source defect below caused credentials, private URLs, or PKI enter its artifact allowlist. The retained-deletion, terminal-proof, quota-diagnostic, and pre-cleanup evidence -regressions were observed red then green offline. Final amended-candidate production -and native live runs remain **PENDING**; no cluster, live model, dispatch, or push -was performed for this repair. `0e8c8a5ca` removed SC2016, but earlier local +regressions were observed red then green offline. At that repair, amended-candidate production +and native live runs were **PENDING**; no cluster, live model, dispatch, or push +was performed for that repair. `0e8c8a5ca` removed SC2016, but earlier local actionlint ran without ShellCheck and is not a ShellCheck pass. The plan baseline remains `7d9e37a128b17107e2d83467c7767146e9c5efb8` in the normal-merge ancestry. Reported offline gates at `53ab45453` passed: targeted controller/chart/contract and tagged fixture @@ -562,11 +619,53 @@ and the offline demo. Local actionlint passed with ShellCheck explicitly disable ShellCheck was unavailable on the host and in the existing dev toolbox. CI remains the authority for ShellCheck and final runtime qualification. +**Historical live-only storage repair (offline, after `2ff780a64145f427ee6ec49f181fe6d0abb97134`):** +the operator reports production green in run +[35692050845](https://github.com/stacklok/mecatl/actions/runs/35692050845), followed by +same-session-hash ID probe success (0 ms), Ensure success (87 ms), and Attach call 1 +`not_ready_nonretryable`, with no Attach end, factory, or save. Resource evidence +reports Ready=False/Reconciled, a Pending executor, and ProvisioningFailed. These +reports were not independently re-fetched here; the safe projection omits the +provisioning message and does not correlate that event to the session binding. + +Source inspection found an independent live-only storage mutation: `live.sh` +replaced local-path's helper image with the executor image, while `run.sh` leaves +Kind's installed storage configuration intact. [Kind v0.33.0's manifest](https://github.com/kubernetes-sigs/kind/blob/v0.33.0/pkg/build/nodeimage/const_storage.go) +uses a Kind helper image (not BusyBox), no helper Pod/container `runAsUser`, and +`mkdir -m 0777 -p "$VOL_DIR"` for setup. Its +[pinned provisioner build](https://github.com/kubernetes-sigs/kind/blob/69b56db7/images/local-path-provisioner/Makefile) +selects v0.0.34; [helper construction](https://github.com/rancher/local-path-provisioner/blob/v0.0.34/provisioner.go#L616-L726) +copies the template without adding a UID override and mounts the parent directory +as HostPath DirectoryOrCreate. [Kubernetes v1.35.8](https://github.com/kubernetes/kubernetes/blob/v1.35.8/pkg/kubelet/kuberuntime/security_context.go#L51-L57) +uses the image UID without an override; its [host-path creation](https://github.com/kubernetes/kubernetes/blob/v1.35.8/pkg/volume/hostpath/host_path.go#L501-L511) +uses mode 0755. `build/execution-workload/Dockerfile` sets USER 65532:65532, so the +rewritten helper runs as UID 65532 and cannot create a child in the normal +root-owned parent. The recovery-only `local-path.yaml` does specify UID 0, but +neither production nor live applies it. No root override is added by this repair. + +`TestLivePreservesQualifiedStorageAndHelmDigestThroughRestoration` failed on the +old script's forbidden local-path ConfigMap read and replacement, then passed +with `-race` after removing that entire mutation. Its fake CLI allows only mock +configuration/readiness operations and verifies mock test, synthetic credential +staging, live test, and digest-only mock/live/mock Helm transitions. Cleanup and +credential-loader failure tests remain unchanged. This establishes the source +regression, not the exact historical ProvisioningFailed cause or why mock passed. +`internal/adapter/executioncontroller/store.go` (`Store.Attach`) returns NotReady +without Retryable when Ready is false; `internal/adapter/executionclient/client.go` +(`waitForBinding`) still polls all NotReady errors until context cancellation, +logging only state changes. A single logged nonretryable state therefore does not +mean only one Attach request occurred. This classification mismatch and the +local-fallback policy are unchanged. Runtime qualification was pending at this +repair; the completion receipt above records its subsequent successful resolution. +Human reviews remain pending. + [PR #1728](https://github.com/stacklok/mecatl/pull/1728), commit `6501b5924`, is already integrated in the implementation ancestry. Its generic live-compaction -repair is not native-provider qualification. Final-candidate native-provider live and amended -production Kind qualification remain **PENDING**. Source resolution below does not assert -new-head CI, final live execution, or human contract/panel approval. +repair is not native-provider qualification. The subsequent native-provider live and amended +production Kind success is recorded in the candidate completion receipt, not inferred from #1728. +Source resolution below does not assert human contract/panel approval. + +### Current proof boundaries | AC | Existing implementation proof at the reference commit; remaining evidence | |---|---| @@ -577,29 +676,29 @@ new-head CI, final live execution, or human contract/panel approval. | AC2.1 | `internal/adapter/executioncontroller/handler_test.go`: `TestHandlerValidateIsReadOnlyAndEnsureIdempotent`; `e2e/k8s_execution/qualification_test.go`: `TestKindExecutionQualification`. | | AC2.2 | `internal/adapter/executioncontroller/store_test.go`: `TestStoreEnsureUsesStableLookupAndRejectsFingerprintDrift`; `controller_test.go`: `TestReconcileRefusesForeignExistingPVCWithoutPersistingUID`. | | AC2.3 | `internal/adapter/executioncontroller/profiles_test.go`: `TestLoadProfilesStrictAndDigestPinned`; production `TestKindExecutionProductionQuotaSaturation`. | -| AC2.4 | `internal/adapter/executioncontroller/provisioning_retry_test.go`: `TestProvisioningWorkerConvergesWithoutAnotherEvent`, `TestProvisioningPreservesCreateAndConditionErrors`, `TestProvisioningRetriesNeverReplaceAuthoritativeResources` exercise PVC and Pod retry/confinement paths. Production `TestKindExecutionProductionQuotaSaturation` restores quota without a new reference event; final runtime evidence PENDING. | -| AC3.1 | `internal/adapter/executionclient/files_conformance_test.go`: `TestRemoteWorkspaceConformanceThroughSignedGRPC` runs both existing shared suites; `TestRemoteFileToolsPreserveLedgerAndNamespaceContracts` invokes all nine tool bodies against the signed handler and real executor. `e2e/k8s_execution/file_tools_test.go` adds the tool matrix called by Kind qualification; runtime execution PENDING. | +| AC2.4 | `internal/adapter/executioncontroller/provisioning_retry_test.go`: `TestProvisioningWorkerConvergesWithoutAnotherEvent`, `TestProvisioningPreservesCreateAndConditionErrors`, `TestProvisioningRetriesNeverReplaceAuthoritativeResources` exercise PVC and Pod retry/confinement paths. Production `TestKindExecutionProductionQuotaSaturation` restores quota without a new reference event; final runtime evidence recorded in the candidate completion receipt. | +| AC3.1 | `internal/adapter/executionclient/files_conformance_test.go`: `TestRemoteWorkspaceConformanceThroughSignedGRPC` runs both existing shared suites; `TestRemoteFileToolsPreserveLedgerAndNamespaceContracts` invokes all nine tool bodies against the signed handler and real executor. `e2e/k8s_execution/file_tools_test.go` adds the tool matrix called by Kind qualification; runtime execution passed at the tested runtime SHA. | | AC3.2 | Same gRPC suites cover concurrent create/CAS, missing/stale/zero versions, non-clobbering namespace operations, and actual Read-ledger behavior. The suite exposed and now guards missing-file replacement precedence and typed non-empty-directory removal errors. | | AC3.3 | `internal/adapter/executioncontroller/file_authorization_test.go`: every supported file operation has positive and negative signed-grant controls; `internal/executionexecutor/confinement_test.go`: physical symlink traversal/search checks. Existing security manager, protocol, durable revocation, and detached-control tests remain complementary. | | AC3.4 | `internal/executionexecutor/executor_test.go`: `TestNativeShellCapsCombinedOutputAndRepairsUTF8`, `TestNativeShellWriteInvalidatesRecordedRead` execute the native helper for combined output bounds, malformed bytes, and actual Shell-write→Edit version refusal. `internal/adapter/executionclient/client_test.go`: `TestProviderThroughRealGRPCSignedHandlerRefreshesAndReattachesExactly` checks runner text repair after private protobuf bytes; `internal/adapter/executionclient/service_integration_test.go`: `TestServiceUsesRealMTLSProviderStoreAndReleasesOnlyAfterDrain` checks identical valid UTF-8 in client events and model history. `internal/adapter/executioncontroller/store_test.go`: `TestCancelledStoreOperationStaysActiveUntilBackendStopsThenFences`; production `TestKindExecutionProductionHolderLossFencesActiveOperation` remains runtime qualification. | | AC4.1 | `internal/adapter/executioncontroller/store_lifecycle_test.go`: `TestClientScopedIntentListReturnsAttestedOwnerOnlyToOwningClient`; Kind isolation and pending-delete tests; scoped-admin non-escalation is exercised by the AC4.4 tests. | -| AC4.2 | `internal/adapter/executioncontroller/store_revoke_test.go`: `TestRevokeEnvironmentFencesOldClaimWithoutChangingExecutionEpoch`; `store_concurrency_test.go`: `TestAcquireRunAndRevokeUseResourceVersionCAS`; the three deterministic rotation tests listed above at repair commit `96e16e926`. The reported production rotation at `f7919f6c` failed; final runtime success PENDING. | +| AC4.2 | `internal/adapter/executioncontroller/store_revoke_test.go`: `TestRevokeEnvironmentFencesOldClaimWithoutChangingExecutionEpoch`; `store_concurrency_test.go`: `TestAcquireRunAndRevokeUseResourceVersionCAS`; the three deterministic rotation tests listed above at repair commit `96e16e926`. The reported production rotation at `f7919f6c` failed historically; final runtime success is recorded in the candidate completion receipt. | | AC4.3 | `internal/adapter/executioncontroller/production_lifecycle_test.go`: `TestExpiredOperationLeaseFencesWithoutClearingIdentity`, `TestRecoverMissingPodRemainsFenceUnknown`, `TestRecoveredTerminalProofCanStartExactReplacement`; production holder loss. | -| AC4.4 | `internal/adapter/executioncontroller/admin_scope_test.go`: `TestScopedAdminAllRoutesOverMTLS`, `TestScopedAdminCASRetryRechecksSubject`, `TestScopedAdminMigrationReceiptRetainsUIDPreconditions`, `TestMigrationCompletedCASReplayRequiresExactSourceSchema`, `TestMigrationReceiptExpiresAfterReconciledReplacement`, `TestScopedAdminDoesNotGrantAttestationOrDataPlane`, `TestScopedAdminWithOwnerAttestationCannotUseAnotherCreatorsDataPlane`. Replacement-expiry test is in `internal/adapter/executioncontroller/production_lifecycle_test.go` in the pinned implementation candidate. `e2e/k8s_execution/ab_production_admin_test.go`: `TestKindExecutionProductionScopedAdministrator` exists (replacement/non-escalation, not all six routes); final Kind evidence PENDING. | +| AC4.4 | `internal/adapter/executioncontroller/admin_scope_test.go`: `TestScopedAdminAllRoutesOverMTLS`, `TestScopedAdminCASRetryRechecksSubject`, `TestScopedAdminMigrationReceiptRetainsUIDPreconditions`, `TestMigrationCompletedCASReplayRequiresExactSourceSchema`, `TestMigrationReceiptExpiresAfterReconciledReplacement`, `TestScopedAdminDoesNotGrantAttestationOrDataPlane`, `TestScopedAdminWithOwnerAttestationCannotUseAnotherCreatorsDataPlane`. Replacement-expiry test is in `internal/adapter/executioncontroller/production_lifecycle_test.go` in the pinned implementation candidate. `e2e/k8s_execution/ab_production_admin_test.go`: `TestKindExecutionProductionScopedAdministrator` exists (replacement/non-escalation, not all six routes); final Kind evidence recorded in the candidate completion receipt. | | AC4.5 | `internal/adapter/executioncontroller/admin_scope_security_test.go`: `TestAdministratorScopeManifestValidation`, `TestAdministratorScopeDigestIsNormalizedAndAuthorityBound`; `internal/adapter/executioncontroller/admin_scope_test.go`: all-routes scope removal plus `TestScopedAdminEqualGenerationDriftFailsClosed`. Compatibility/upgrade human review PENDING. | | AC5.1 | `internal/adapter/executionclient/client_test.go`: exact reattachment test above; production `TestKindExecutionProductionReplicaLifecycle` and security rotation. | | AC5.2 | `internal/adapter/executioncontroller/production_lifecycle_test.go`: `TestReplacementPersistsTerminalProofBeforeRemovingPodFinalizer`; production replica lifecycle. | -| AC5.3 | `internal/adapter/executioncontroller/production_lifecycle_test.go`: `TestReplacementQuiescesAndPendingReferenceBlocksRetirement`; `internal/adapter/executioncontroller/retained_delete_race_test.go`: `TestRetainedDeletePeerBetweenFinalizerUpdateAndDelete`, `TestRetainedDeleteDoesNotRemoveForeignCR`, `TestDirectCRDeleteStillBlocked` (offline race controls; final runtime evidence PENDING). | +| AC5.3 | `internal/adapter/executioncontroller/production_lifecycle_test.go`: `TestReplacementQuiescesAndPendingReferenceBlocksRetirement`; `internal/adapter/executioncontroller/retained_delete_race_test.go`: `TestRetainedDeletePeerBetweenFinalizerUpdateAndDelete`, `TestRetainedDeleteDoesNotRemoveForeignCR`, `TestDirectCRDeleteStillBlocked` (offline race controls; final runtime evidence recorded in the candidate completion receipt). | | AC5.4 | `internal/adapter/executionclient/reference_ambiguity_test.go`: `TestAmbiguousCommitIsRetainedAndNeverAbortedByCleanup`; production `TestKindExecutionProductionPendingDeleteOutageRecovery`. | -| AC5.5, AC5.7 | `deploy/helm/mecatl-execution/chart_test.go`: `TestChartHasNoDeletionHook`; `deploy/helm/mecatl-execution/lifetime_test.go`: `TestChartRetainedLifetime` exercises lookup/adoption, missing history, foreign ownership, profile drift, and retained policies/ledgers offline. `e2e/k8s_execution/zz_production_helm_lifetime_test.go`: `TestKindExecutionProductionHelmLifetime` exercises actual upgrade/uninstall/reinstall and enforcement; final runtime evidence PENDING. | +| AC5.5, AC5.7 | `deploy/helm/mecatl-execution/chart_test.go`: `TestChartHasNoDeletionHook`; `deploy/helm/mecatl-execution/lifetime_test.go`: `TestChartRetainedLifetime` exercises lookup/adoption, missing history, foreign ownership, profile drift, and retained policies/ledgers offline. `e2e/k8s_execution/zz_production_helm_lifetime_test.go`: `TestKindExecutionProductionHelmLifetime` exercises actual upgrade/uninstall/reinstall and enforcement; final runtime evidence recorded in the candidate completion receipt. | | AC5.6 | Deployment-documentation inspection and docs gates required on the final implementation; Kind is restart proof only. | | AC6.1–AC6.2 | `internal/app/remote_execution_test.go`: `TestRemoteExecutionRealFactoryCarriesPostureAndAttenuatedCatalog` includes a nonempty LOCAL Git snapshot control and REMOTE exclusion, poison-source and operator-global prompt controls, forbidden-tool errors and command/worktree discovery. `TestRemoteExecutionPreservesOperatorPermissionsUnderAuto` exercises global/explicit-file Deny and configured Ask, excluding project permissions, with allowed in-memory Shell controls. `TestRemoteDeploymentNoFSUsesLocalAttenuationWithoutProviderCall` retains the no-FS check. `internal/adapter/executionclient/service_integration_test.go`: `TestServiceUsesRealMTLSProviderStoreAndReleasesOnlyAfterDrain` rejects background Shell and actual Clear/Fork unsupported-selector calls with named errors, no destination publication, and unchanged source ref/history and provider references. | | AC6.3 | `internal/adapter/executioncontroller/store_lifecycle_test.go`: `TestReferenceTransactionsRetainUnknownAndNeverChangeSource`; production `TestKindExecutionProductionClearForkLifecycle`. | -| AC7.1 | `.github/workflows/k8s-e2e.yml` runs `task e2e:k8s:execution:production`; historical and operator-reported runs are distinguished above. The reported 11/11 pass at `7e498bd73` does not supersede later failures at `2e1aa6bbf`. `internal/adapter/executioncontroller/legacy_fixture_kind_test.go` covers the six-minute quota-accounting bound with tiny-context tests. `e2e/k8s_execution/holder_loss_test.go` covers the exact-owned terminal stimulus and complete kubelet evidence offline. Final amended-candidate run **PENDING**. | +| AC7.1 | `.github/workflows/k8s-e2e.yml` runs `task e2e:k8s:execution:production`; historical and operator-reported runs are distinguished above. The reported 11/11 pass at `7e498bd73` does not supersede later failures at `2e1aa6bbf`. `internal/adapter/executioncontroller/legacy_fixture_kind_test.go` covers the six-minute quota-accounting bound with tiny-context tests. `e2e/k8s_execution/holder_loss_test.go` covers the exact-owned terminal stimulus and complete kubelet evidence offline. Final amended-candidate production run passed all 11 scenarios at the tested runtime SHA. | | AC7.2 | `e2e/k8s_execution/qualification_test.go`: `TestKindExecutionQualification`, run by the successful production profile. | -| AC7.3 | Production `TestKindExecutionProductionFailureArtifactBoundary`; `deploy/mecatl-execution-kind/failure_evidence_test.go` tests closed-vocabulary projection and API-error redaction; `TestNativeWorkflowCleanupOwnership` proves evidence collection before failed cleanup. Workflow bounded uploads and ownership-scoped cleanup; final runtime evidence PENDING. | -| AC7.4 | `e2e/k8s_execution/live_qualification_test.go`: `TestKindExecutionLiveQualification` exists; successful final native-provider OpenRouter evidence **PENDING**. | -| AC7.5 | `e2e/k8s_execution/fixture/credentialloader/main_test.go`: `TestLoadCredentialAndRedaction`, `TestDeletePinsReceiptUIDAndLeavesReplacement`; these offline tests do not substitute for live execution. | +| AC7.3 | Production `TestKindExecutionProductionFailureArtifactBoundary`; `deploy/mecatl-execution-kind/failure_evidence_test.go` tests closed-vocabulary projection and API-error redaction; `TestNativeWorkflowCleanupOwnership` proves evidence collection before failed cleanup. Workflow bounded uploads and ownership-scoped cleanup; final runtime evidence recorded in the candidate completion receipt. | +| AC7.4 | `e2e/k8s_execution/live_qualification_test.go`: `TestKindExecutionLiveQualification` passed; successful exact-head native-provider OpenRouter evidence and independent file/test verification are recorded in the candidate completion receipt. | +| AC7.5 | `e2e/k8s_execution/fixture/credentialloader/main_test.go`: `TestLoadCredentialAndRedaction`, `TestDeletePinsReceiptUIDAndLeavesReplacement`; these offline tests do not substitute for live execution. The candidate receipt separately records successful trusted-loader live handling, mock restoration, receipt clearance, private-file deletion, and owned-cluster absence. | Production test names without another path above reside in the implementation's `e2e/k8s_execution/production_qualification_test.go`. Do not import implementation files into this @@ -634,4 +733,4 @@ plan worktree to make citations or trace appear complete. - A PVC can preserve files while a stale command still mutates them. Session leasing does not fence command execution; missing old-executor evidence remains a fail-closed manual recovery boundary. - Only an enforcing CNI supplies negative NetworkPolicy proof. Retaining storage while uninstall removes isolation or authority history is unsafe; the Helm lifecycle amendment is a completion requirement, not deferred polish. - Every outlives-a-call resource, including retained security/capacity ledgers, reload state, informer/cache, connection pools, claims, and lifecycle/reference reconciliation, needs an accurate resource/rehydration inventory in `docs/adr/0027-cloud-native.md` on the implementation branch. -- Scope, retry, Helm lifecycle, disabled startup, remote filesystem conformance, and source-admission behavioral tests exist in the pinned implementation source. Final production Kind/native live evidence and human reviews remain pending. The requested offline output/CAS and Git/unsupported-successor proofs in AC3.4 and AC6.1–AC6.2 are present; they do not replace final runtime qualification. Source/proof locations absent on the plan branch stay explicitly implementation-scoped; do not add implementation code to satisfy plan-branch gates. +- Scope, retry, Helm lifecycle, disabled startup, remote filesystem conformance, and source-admission behavioral tests exist in the pinned implementation source. Final production Kind/native live evidence is complete at the tested runtime SHA; human reviews remain pending. The requested offline output/CAS and Git/unsupported-successor proofs in AC3.4 and AC6.1–AC6.2 are present; they do not replace final runtime qualification. Source/proof locations absent on the plan branch stay explicitly implementation-scoped; do not add implementation code to satisfy plan-branch gates. diff --git a/docs/adr/0350-native-kubernetes-execution.md b/docs/adr/0350-native-kubernetes-execution.md index 72eeebcd5a..00eecbdf04 100644 --- a/docs/adr/0350-native-kubernetes-execution.md +++ b/docs/adr/0350-native-kubernetes-execution.md @@ -28,7 +28,7 @@ is an operator-controlled first production slice with explicit limits; release m The existing plan PR #1579 and implementation PR #1614 form a human-authorized **draft** stack. The [acceptance plan](../acceptance/native-kubernetes-execution.md) now authors the exact proposed contract, -reconciled against implementation `deaf1c3d1dfe7ea92afc8fe826f1bc613080f219`. API, schema, and security +reconciled against tested runtime `2020e59934369e8f771059d0c173fa4cbf964c7e`. API, schema, and security review approvals remain unchecked; neither authored specs nor draft code is a merged approved baseline. This amendment also authorizes development of scoped administration, continuing provisioning retries, and retained-resource Helm lifecycle safeguards on those existing PRs. Human alone merges. @@ -210,20 +210,27 @@ node-disaster recovery; kind host-local storage proves restart survival only. The focused mock-only kind tasks run on the supported generic toolchain, including CI, with a unique owned cluster, scratch kubeconfig, explicit context on every command, bounded sanitized artifacts, and ownership-scoped cleanup. Default kind does not prove NetworkPolicy enforcement; the production profile -uses Calico. The acceptance plan records a successful production job at `deaf1c3d1`, distinct from the -pending final candidate and new scope/retry/Helm regression qualification. Historical local keyring -failures no longer block that baseline. Skipped draft race lanes remain unproven. The latest -operator-reported run `35650128625` at `f7919f6c` passed 10/11 production tests, including scoped admin, -Helm lifecycle/quota, and general file tools; rotation failed at the final-authority read. Offline tests -at repair commit `96e16e926` reproduce both safe pre-dispatch replica lag and persistent mixed-material -ledger poisoning, and prove immutable-name publication across both projection orders. They do not -identify which mechanism caused that CI failure or establish final runtime success. +uses Calico. The acceptance plan retains historical production success at `deaf1c3d1` and the +later reported run `35650128625` at `f7919f6c` which passed 10/11 scenarios but failed rotation +at the final-authority read. Offline tests at repair commit `96e16e926` reproduce safe +pre-dispatch replica lag and persistent mixed-material ledger poisoning, and prove immutable-name +publication across both projection orders. They do not identify which mechanism caused that +historical failure. Historical local keyring failures and qualification failures are no longer +current blockers: the [candidate completion receipt](../acceptance/native-kubernetes-execution.md#candidate-completion-receipt) +records all 11 production Kind+Calico scenarios passing at runtime SHA +`2020e59934369e8f771059d0c173fa4cbf964c7e`, including scope/retry/Helm lifecycle qualification. +Draft full-race CI remained skipped; earlier local full-race and latest targeted-race results +are distinguished in that receipt, not represented as a draft full-race CI execution. The checked human live decision requires the separate `task e2e:k8s:execution:live` OpenRouter coding qualification **for this stack**, after deterministic qualification. It is not an always-on live CI dependency. Prove model-driven tools, independently verified persistent artifacts, and a successful -bound test command; generic mecak8s live CI is not native-provider evidence. No final native live run is -claimed. The generic compaction live-lane correction in merged #1728 is separate from this capability. +bound test command; generic mecak8s live CI is not native-provider evidence. Manual exact-head run +[35700303737](https://github.com/stacklok/mecatl/actions/runs/35700303737) supplied that evidence +after production qualification: real OpenRouter coding, independent exact-ref typed-gRPC +reattachment/run acquisition and file/test verification, followed by verified cleanup. This is +qualification of the tested runtime SHA, not approval or a claim that documentation-only descendant +heads executed live. The generic compaction live-lane correction in merged #1728 remains separate. Use verified nonsecret endpoint/protocol/model configuration in an authenticated operator-controlled cluster. A trusted runtime-only loader consumes the approved protected file channel into a narrowly @@ -251,8 +258,9 @@ separation are required but not sufficient. security approval and release maturity remain unresolved. Authorized pre-merge development may proceed on the existing draft stack, but no review, approval, landed status, or merge is inferred. Completion requires strict AC trace, full offline/race/build/API/docs/site/demo gates, independent panel review, -final deterministic qualification, and stack-specific native live evidence. New scope, retry, and Helm -behavioral proofs remain pending. Human alone merges. +final deterministic qualification, and stack-specific native live evidence. The latter two now have +successful candidate receipts, including scope, retry, and Helm scenarios; the plan preserves partial +proof boundaries and outstanding human/panel reviews. Human alone merges. ## See also