diff --git a/.claude/skills/to-acceptance-plan/scripts/check-acceptance-plan-test.sh b/.claude/skills/to-acceptance-plan/scripts/check-acceptance-plan-test.sh index f455110e5..49c2dcc1a 100755 --- a/.claude/skills/to-acceptance-plan/scripts/check-acceptance-plan-test.sh +++ b/.claude/skills/to-acceptance-plan/scripts/check-acceptance-plan-test.sh @@ -113,6 +113,20 @@ sed_in_place 's/\*\*Expected tasks:\*\* deferred to orchestration/\*\*Expected t sed_in_place 's||**Combined rationale:** The fixture is one indivisible documentation check, so separate plan review adds no value.|' "$combined_valid" bash "$checker" "$combined_valid" >/dev/null +# Exceed pipe capacity after the first category: an early-exiting reader must +# not turn a valid Split or Combined contract into a SIGPIPE failure. +for source in "$valid" "$combined_valid"; do + large="$root/acceptance/large-${source##*/}" + awk ' + { print } + /^- \*\*gRPC \/ protobuf:\*\*/ { + for (i = 0; i < 16384; i++) + print " Additional interface detail keeps the contract larger than a pipe buffer." + } + ' "$source" >"$large" + bash "$checker" "$large" >/dev/null +done + for case_name in missing-contract invalid-contract missing-category bare-none placeholder bad-delivery bad-status proposed-unchecked missing-human placeholder-human checked-without-decision; do cp "$valid" "$root/acceptance/$case_name.md" done diff --git a/docs/acceptance/README.md b/docs/acceptance/README.md index 90bfc09d8..458610d65 100644 --- a/docs/acceptance/README.md +++ b/docs/acceptance/README.md @@ -111,6 +111,9 @@ PR after verification. There is no cleanup or status-only PR. ## Plans +- [Native Kubernetes execution provider](native-kubernetes-execution.md) — separately deployed, + optional controller/executor service in this monorepo, with retained workspaces and kind + qualification. Status: draft; implementation is planned as a draft PR stacked on the plan PR. - [Purpose-attributed auxiliary token usage](auxiliary-token-usage.md) — records every reachable session-associated auxiliary LLM call in the canonical ledger under a purpose-specific provider/model-attributed kind, without changing main-run accounting or budgets. Status: proposed. - [Development-spine work classification](adr-scope-classification.md) — routes Spike, Routine, Bounded, and Architectural work; keeps acceptance planning for substantive work diff --git a/docs/acceptance/native-kubernetes-execution.md b/docs/acceptance/native-kubernetes-execution.md new file mode 100644 index 000000000..78b52eed3 --- /dev/null +++ b/docs/acceptance/native-kubernetes-execution.md @@ -0,0 +1,736 @@ +# Native Kubernetes execution — acceptance plan + +**Contract:** human-reviewed/v2 +**Work classification:** Architectural — introduces a controller/executor trust boundary, durable environment ownership, Kubernetes resources, and an optional mecak8s deployment integration. +**Decision record:** [ADR 0354](../adr/0354-native-kubernetes-execution.md) +**Phase:** native Kubernetes execution first slice +**Status:** draft, amended 2026-09-21. Exact proposed contracts are authored below; API, schema, and security review approvals remain outstanding. +**Delivery:** Split. Human-authorized draft stack: the Plan / Interface PR targets `main`; the draft Implementation PR targets `plan/native-kubernetes-execution`. Authorization includes developing this amendment before plan merge, not approval or merge readiness. Human alone merges. +**Expected tasks:** draft implementation decomposition remains run-local; no approved orchestration baseline. +**Issue:** None — planning request has no linked issue. +**Plan PR:** [#1579](https://github.com/stacklok/mecatl/pull/1579), draft on `plan/native-kubernetes-execution`; implementation [#1614](https://github.com/stacklok/mecatl/pull/1614) remains draft. +**Approved baseline:** absent. Plan worktree baseline `a5030545e87e81aad12fba4c49f0c9933a634b5e` and tested runtime source `2020e59934369e8f771059d0c173fa4cbf964c7e` are evidence, not approval. + +The [implementation proof map](#implementation-proof-map) records completed candidate runtime +qualification separately from open human reviews and limited behavioral proof. Implementation-only +source paths below refer to that tested runtime SHA; they are future code proofs on this +documentation-only branch, not claims that the files or tests already exist here. + +The smallest demonstrable slice is an independently deployed execution environment provider service +with its own controller and executors. Mecak8s is an optional client: its adapter obtains a persistent +Kubernetes workspace and bound Shell through the provider API without owning Pods, PVCs, or the +controller lifecycle. The provider keeps the existing `tool.Environment` and file-tool schemas, +uses an operator-selected profile, proves behavior offline in kind, and makes no remote-project-ingestion claim. +ADR 0048 continues to govern storage-free mecak8s; this separate provider does not supersede it. + +## Human decisions + +- [x] Keep the execution environment provider a separate service in this monorepo. — Decision: the provider owns its controller, CRDs, Pods, PVCs, binaries, images, chart, and lifecycle; mecak8s consumes its API through an optional adapter and remains storage-free under ADR 0048, which is not superseded. +- [x] Choose a purpose-built controller rather than Agent Sandbox. — Decision: own the controller/executor boundary and its CRD. +- [x] Use typed gRPC for the private provider network protocol. — Decision: replace HTTP/JSON with protobuf RPCs over mandatory mTLS; keep the public Harness API unchanged and preserve the separate provider service boundary. +- [x] Qualify the gRPC coding flow with a real OpenRouter credential. — Decision: run the explicit live kind target after mock qualification using only the trusted runtime loader; prove model tool execution and independently verify workspace artifacts and a successful test command, then restore mock configuration and delete only the run-owned Secret by UID. +- [x] Keep Kubernetes execution optional. — Decision: disabled/default deployments remain Kubernetes-execution-free. +- [x] Retain one PVC per logical environment by default. — Decision: session deletion does not delete a workspace; committed data is removed only through explicit retirement. Storage class, size, and quotas are operator-configured. +- [x] Set the first-slice source and execution boundary. — Decision: use a blank tool-seeded workspace, existing filesystem tools, foreground Shell, and operator-global instructions only, with no warm pools or automatic idle suspension. Defer Git clone, private credentials, remote project ingestion, background commands, and isolated delegation; schedules remain excluded. +- [x] Set the initial security and qualification posture. — Decision: the first production slice is authenticated and authorized in an operator-controlled cluster, with the limits below, not hardened hostile multitenancy. Deterministic mock/kind qualification precedes the separately invoked native-provider OpenRouter coding qualification required by the checked live decision above; this is not an always-on live CI dependency. +- [x] Choose mTLS service links plus least-privilege grants scoped to environment, operation, and ownership epoch. — Decision: uncertain takeover fails closed; old-writer termination or external fencing is required before replacement. +- [x] Handle live-provider credentials only through a trusted runtime loader. — Decision: it may be consumed directly from its protected file channel to a narrowly scoped HARNESS-only Secret, with redacted failures, before child processes or tools can inherit it. The agent never accesses the file or Secret. No agent or tool reads, displays, or loads it into model/tool output, argv, Helm values, disk manifests, git, logs, artifacts, or the execution workload; mock tests do not access it. Endpoint, protocol, and model selection still require verified nonsecret configuration. +- [x] Develop scoped administrative authority in this existing draft stack. — Decision: a distinct administrator URI may administer only explicitly named creator URIs through `administratorFor`; this grants no data-plane or owner-attestation authority. Existing self-administration remains compatible, with no namespace-wide administrator grant. +- [ ] Review and approve the authored private control/data-plane API, reference transaction, run ownership, errors, and compatibility contract below. +- [ ] Review and approve the authored CRD/status schema v2 and configuration contract, including scoped administration, continuing provisioning retries, and retained-resource Helm lifecycle qualification. +- [ ] Security-review and approve the authored grant trust, rotation/revocation, replay bounds, termination acknowledgement, fencing, and retained authority-history contract. +- [ ] Decide the release maturity designation. The target remains an operator-controlled first production slice with explicit limits; the CRD name `v1alpha1` does not settle an “alpha” product label or waive any completion gate. + +## Interface contract + +This documentation-only receipt traces the draft implementation at tested runtime SHA +`2020e59934369e8f771059d0c173fa4cbf964c7e` against the proposed contract. Candidate runtime +qualification is complete; authored specifications and passing CI do not constitute human +approval. The unchecked API, schema, security, and release-maturity decisions remain open. + +- **gRPC / protobuf:** Public Harness protobuf is unchanged. The private `mecatl.execution.v1.ExecutionProviderService` uses mandatory mTLS, a `host:port` endpoint, protocol identifier `execution-grpc/1`, and an 8 MiB message bound. The exact wire schema is the implementation reference's `contracts/proto/mecatl/execution/v1/execution.proto`, generated with `task generate`. The RPC/field summary below is the review contract; there is no HTTP/JSON compatibility endpoint. +- **Exported Go APIs / interfaces:** No `engine/` exported API change. `tool.Environment` remains identity + Workspace + independent ReadLedger + optional bound CommandRunner (`engine/tool/environment.go`). Host-only seams in the implementation's `internal/adapter/server/placement.go` are specified below; Kubernetes dependencies stay outside the engine. +- **Tool schemas:** Existing filesystem tools and foreground Shell remain byte-compatible, including recorded reads and conditional CreateFile/ReplaceFile. Copy/Move/Remove retain their own conformance contracts. Shell writes bypass file CAS (`engine/tool/tool.go`). Remote catalog attenuation and its model-visible posture must be tested through the real composition factory. Unsupported operations never execute locally. +- **CLI / config:** Disabled by default. Mecak8s accepts `--execution-enabled`, `--execution-endpoint`, `--execution-profile`, `--execution-tls-ca`, `--execution-tls-cert`, and `--execution-tls-key`. Enablement requires all five string settings and OIDC caller ownership enforcement; it conflicts with `--workspace` and `--redis-filesystem`. The separate `deploy/helm/mecatl-execution` chart owns the provider deployment. The default mecak8s chart has no provider dependency. Exact provider profiles and security configuration are below. +- **Events / persistence:** No public event widening. Provider CR status, reference intents, run/operation claims, lifecycle receipts, security high-water state, and capacity reservations are durable provider state. Exact session `EnvironmentRef.Revision` remains immutable and separate from transient execution epoch and grant generation. The namespaced CRD and schema-v2 fields are below. +- **Security / authority:** TLS 1.3 with one canonical allowlisted client URI SAN, short-lived signed grants, exact owner/reference checks, and independent environment fencing. Scoped administration extends only the provider's existing client policy. No controller credentials, signing keys, provider secrets, or service-account token enter arbitrary-shell workloads. This is an operator-controlled cluster boundary, not a hostile-multitenancy claim. +- **Compatibility / migration:** Additive and opt-in for existing local and `no-fs` sessions, with no local fallback for remote failures. Private schema migration is explicit and quiesced; CRD upgrade is manual. The `administratorFor` extension changes neither public Harness nor engine nor private protobuf. Old strict decoders reject it; use a quiesced provider upgrade, not a claimed mixed-version rolling upgrade. Neither downgrade nor reinstall may reset authority high-water state. + +## Proposed technical specifications + +These sections define the proposed interface details referenced by the categories above. + +### Private RPC and host contract + +Wire fields use protobuf snake_case. `EnvironmentRef = {id, revision}` and `Owner = {issuer, subject}`. +`RequestContext = {environment, owner, binding_id, epoch, grant, run_id, claim_id, grant_generation}`. +Environment and owner matches are exact; a client cannot select a creator through any new wire field. +The host maps this ref to `session.EnvironmentRef{Kind:"kubernetes", ID:id, Revision:revision}`; +reattachment never substitutes a current default or Pod UID. + +| RPCs | Exact request/result and semantics | +|---|---| +| `ValidateProfile` | `{profile}` → `{profile, digest, capabilities, max_file_bytes, max_command_bytes, max_command_duration_millis}`. Side-effect-free preflight, never `Bind` or allocation. | +| `EnsureEnvironment` | `{binding_id, profile, owner, operation_id}` → `{environment, epoch, ready, grant, grant_expires_at, grant_generation}`. Server-minted final binding ID, authenticated creator, owner, and profile fingerprint identify one allocation; drift conflicts instead of adopting another environment. | +| `AttachEnvironment` | `{context, purpose:"session"}` → the same result fields as Ensure. Exact persisted ref and reference membership; attach does not confer an active execution claim. | +| `AcquireRun` | `{environment, owner, binding_id, run_id, operation_id, ttl_millis}` → `RunClaimResponse{environment, binding_id, run_id, claim_id, epoch, grant_generation, grant, expires_at}`. One published binding holds environment-wide execution ownership at a time. TTL is 30s–5m; adapter default 1m. | +| `RenewRun`, `ReleaseRun` | Both carry `{environment, owner, binding_id, run_id, claim_id, epoch, operation_id, grant_generation}`; Renew adds `ttl_millis` and returns RunClaimResponse; Release returns Empty. Exact resourceVersion CAS and operation replay checks prevent stale renewal/release. | +| `CommitReference`, `AbortReference`, `PrepareReferenceDelete`, `ConfirmReferenceDelete`, `CancelReferenceDelete` | `{environment, owner, binding_id, operation_id}` → Empty. Durable reference transaction described below. `ReleaseReference{context}` is a compatibility entry, not permission to drop ambiguous or active references. | +| `ReserveSuccessor` | `{environment, owner, source_binding_id, destination_binding_id, operation_id}` → `{environment}`. Reserve another reference to the same exact environment under exclusion; not a filesystem clone. | +| `ListReferenceIntents` | `{owner, limit, environment, binding_id}` → bounded `intents[{environment, binding_id, state, operation_id, source_binding_id, created_at, owner}]`. Client-scoped reconciliation; attested owner information returns only to the owning creator client. | +| `RetireEnvironment`, `ReplaceExecutor`, `RecoverEnvironment` | `{environment, owner, expected_execution_epoch, expected_pod_uid, expected_pvc_uid, operation_id}` → Empty. Administrative admission retains exact epoch/UID checks, reference/idle preconditions, and operation receipts. Recovery accepts independently observed exact terminal Pod proof, never a caller boolean asserting fencing. | +| `DeleteRetiredEnvironment` | `{environment, owner, expected_pvc_uid, operation_id}` → Empty. Explicit destructive retained-PVC deletion after retirement and reference exclusion, with UID-preconditioned deletion and capacity release; not default uninstall behavior. | +| `MigrateEnvironment` | `{environment, owner, optional expected_schema_version, expected_pod_uid, expected_pvc_uid, operation_id}` → Empty. Presence is mandatory; 0 or 1 selects the compatible prototype migration to 2. Unknown/current versions are not a general migration or rollback API. | +| `RevokeEnvironment` | `{environment, owner, expected_grant_generation, operation_id}` → `{grant_generation}`. CAS advances revocation generation without changing execution epoch; repeat exact operations return the receipt rather than revoking twice. | +| `Files` | `{context, operation, path, destination, pattern, data:bytes, version:bytes, limit}` → `{data:bytes, version:bytes, info, entries, paths, matches, authority_target, authority_workspace}`. Closed enum: READ, RESOLVE_AUTHORITY, STAT, CREATE, REPLACE, LIST, REMOVE, RENAME, COPY, GLOB, GREP; UNSPECIFIED is rejected. FileInfo carries `{name, size, mode, mod_time, is_dir}`; GrepMatch carries `{path, line, text}`. Paths remain confined inside the bound workspace. | +| `StartCommand` | `{context, command, timeout_millis}` → `{command_id, state, result}`. Foreground unary RPC, not a detached job or server stream. Result carries `{command_id, state, exit_code, stdout:bytes, stderr:bytes, next_offset, truncated, terminal_receipt}`. Closed states are RUNNING, SUCCEEDED, FAILED, CANCELLED, FENCE_UNKNOWN (UNSPECIFIED rejected). Context cancellation is not proof of process termination. | +| `CommandStatus`, `CancelCommand` | `{context, command_id, offset}`. Authenticate/authorize and validate first, then return gRPC `Unimplemented` for this foreground-only slice. No command-stream RPC is implemented. | + +Sanitized errors carry `ErrorDetail{code, retryable}`. Stable mappings are `invalid_argument`→InvalidArgument, +`unauthenticated`→Unauthenticated, `permission_denied`→PermissionDenied, `not_found`→NotFound, +`already_exists`→AlreadyExists, `conflict`/`version_mismatch`/`directory_not_empty`→Aborted, `not_ready`→Unavailable, +`fence_unknown`→FailedPrecondition, `resource_exhausted`→ResourceExhausted, and `internal`→Internal. +Only `not_ready` and `unauthenticated` can carry `retryable:true`; this is not permission to retry a +mutation automatically. Grant-expiry refresh is read-only. Backend details never cross the boundary; +absent and out-of-scope environments have the same non-disclosing denial. Foreground detached-control +`Unimplemented` is the explicit exception to the structured error vocabulary. + +Host-only interfaces keep allocation and execution separate: + +- `PlacementValidator.ValidatePlacement(context.Context) error` validates without allocation. +- `PlacementBindRequest.BindingID session.SessionID` is the final server-minted binding; + `PlacementBinding.Commit func(context.Context) error` publishes after store persistence, and + `Close func() error` aborts only a conclusively uncommitted reservation. +- `ExecutionAccess` supplies `Applies(session.EnvironmentRef) bool` and + `AcquireRun(context.Context, ExecutionRunRequest) (ExecutionRunHandle, error)`; + the request contains `Ref, Principal, BindingID, RunID`, and the handle supplies + `Environment() tool.Environment`, `Renew(context.Context) error`, and `Release(context.Context) error`. +- `PlacementSuccessorReservoir.ReserveSuccessor(context.Context, PlacementSuccessorRequest) (PlacementBinding, error)` + and `ReferenceLifecycle.PrepareReferenceDelete(context.Context, PlacementSuccessorRequest) (ReferenceDeleteHandle, error)` + use `Ref, Principal, SourceBindingID, DestinationBindingID`. ReferenceLifecycle also has `Applies`; + the delete handle has `Confirm(context.Context) error` and `Cancel(context.Context) error`. +- `ReferenceIntentLifecycle.ListReferenceIntents(context.Context, int) ([]ReferenceIntent, error)` and + `CommitReferenceIntent`, `ConfirmReferenceIntentDelete`, `CancelReferenceIntentDelete` (each + `(context.Context, ReferenceIntent) error`) reconcile durable ambiguity. ReferenceIntent contains + `Ref, Principal, BindingID, SourceBindingID, OperationID, PendingDelete`. +- `PlacementReattacher.Reattach(context.Context, PlacementReattachRequest) (PlacementBinding, error)` + uses exact `Ref, Principal, Scope, BindingID`. Public callers receive display metadata only. + +### Reference transaction, schema v2, and fencing + +Ensure reserves `PendingCreate` before session-store publication; confirmed publication permits +Commit→`Published`. Abort is safe only when publication is conclusively absent. A lost commit response +remains recoverable, never TTL garbage. Delete uses `Published`→`PendingDelete` before store deletion, +then Confirm removes the reference or Cancel restores Published after confirmed non-deletion. +Store unavailability retains the intent. Clear/Fork reserve a destination reference under environment +exclusion, publish it, and preserve the source on failure. Clear creates empty history, Fork copies valid +history, both keep the same remote ref; neither is a workspace clone. A retiring environment admits no +new reference or successor. Retirement requires no references, including pending ones. + +The implementation CRD is namespaced `execution.mecatl.dev/v1alpha1`, kind `ExecutionEnvironment`, +plural `executionenvironments`, with a status subresource. `v1alpha1` is the Kubernetes API name, +independent of the unresolved release maturity designation. The structural schema is in the implementation's +`deploy/helm/mecatl-execution/crds/executionenvironment.yaml`: + +| Area | Exact fields and constraints | +|---|---| +| `spec` | `schemaVersion=2`; immutable `allocationID, revision, ownerHash, clientHash, bindingID, requestFingerprint, profile, profileDigest, image, storageClass, storageSize, resources`. Optional immutable `ownerIssuer, ownerSubject` retain attested ownership. `desired` is Active or Retiring. Hashes are 64 lowercase hex characters; profileDigest is `sha256:` plus 64 hex; image is digest-pinned. `allocationID`/profile ≤63 characters, revision ≤64, bindingID ≤253. | +| Core `status` | `schemaVersion=2, observedGeneration, epoch, grantGeneration, fenceState, references, pvc{name,uid}, pod{name,uid}, conditions`. Epoch and grantGeneration are positive signed-int64-bounded counters; fenceState is Healthy or FenceUnknown. Conditions carry `type,status,reason,message,observedGeneration,lastTransitionTime`; they describe observed spec generation, not durable identity. | +| References | At most 64 records, map-keyed by `bindingID`, with `state, operationID, sourceBindingID, createdAt`; state is PendingCreate, Published, or PendingDelete. | +| `activeRun` | `bindingID, runID, claimID, operationID, ownerHash, clientHash, epoch, grantGeneration, expiresAt`. | +| `activeOperation` | `id, operation, startedAt, claimID, runID, epoch, holderID, renewedAt, expiresAt`. Lease-holder identity is replica-specific; an expired holder remains recorded for recovery. | +| Receipts | `renewReceipts[{operationID,fingerprint,expiresAt}]` and `revocationReceipts[{operationID,fingerprint,expectedGrantGeneration,grantGeneration}]`, each capped at 32. Receipt eviction does not authorize replay past current claim/generation preconditions. | +| Lifecycle | `lifecycleOperation{id,type,phase,expectedEpoch,expectedPodUID,expectedPVCUID,createdAt}`; types ReplaceExecutor, RetireEnvironment, DeleteRetiredEnvironment; phases Quiescing, WaitingForTermination, RemovingPodFinalizer, WaitingForPodDeletion, CreatingReplacement, DeletingPVC, ReleasingSlot. | +| Recovery/migration | `terminationProof{operationID,podUID,pvcUID,epoch,podPhase,observedAt}` (Succeeded/Failed only); `lastReplacement{operationID,previousPodUID,replacementPodUID,pvcUID,previousEpoch,replacementEpoch}`; `migrationOperation{id,fromSchema,expectedPodUID,expectedPVCUID}` and `lastMigrationOperationID, lastMigrationFromSchema` (integer 0 or 1; presence required for completed replay). Completed receipts expire atomically when replacement publishes the new Pod identity; neither the old request nor the same operation with replacement UIDs may revive them. Completed and concurrent replay must match the operation, source schema, exact subject, and runtime UIDs; old receipts without source-schema presence fail closed. Schema 0/1 is accepted only for explicit compatible migration, never ordinary execution; unknown schemas fail closed. | + +Run claims serialize environment use independently of session leases. A separate 30s operation-holder +lease renews every TTL/3. Every operation rechecks current claim, owner, creator, revision, epoch, and +grant generation against resourceVersion-CAS state. Holder loss, transport cancellation, revocation, or +uncertain helper completion cancels work and preserves fencing uncertainty; it does not clear operation +identity to let another writer start. Exact terminal proof is persisted before removing the executor +finalizer or replacing the Pod. Missing/unobservable Pods and missing or mismatched authoritative PVC/Pod +UIDs never justify automatic replacement. External compute/storage fencing requires an operator runbook; +the API has no force-takeover assertion. Helper process teardown must include detached descendants or +fail closed. Credential-free Kubernetes exec stdin framing is an internal helper transport, not another +provider network protocol or a trusted workload-supplied proof of node termination. + +**Provisioning amendment:** transient PVC/Pod create failures, including quota saturation, retain +sanitized Ready=false conditions and schedule rate-limited retries for as long as the CR exists. +Restoring quota must converge without another reference/spec change, restart, or manual reconcile. +There is no finite retry count that silently forgets an existing unready allocation. Identity/ownership +failures remain fail-closed with no replacement, however often reconciliation runs. The inspected queue +already rate-limits returned errors; create-error branches currently return the condition-write result, +which can be nil. Merely retaining that queue is not proof of continuing provisioning retries. + +### Provider configuration, rotation, and scoped administration + +The strict profile YAML is `profiles: {: ProfileSpec}`. ProfileSpec requires +`image, storageClass, storageSize, cpuRequest, memoryRequest, cpuLimit, memoryLimit, +ephemeralStorageRequest, ephemeralStorageLimit, tmpSizeLimit, runtimeClassName, +maxFileBytes, maxCommandBytes, maxCommandDuration, maxEnvironments`. Positive resource quantities, +request≤limit, available RuntimeClass, and digest-pinned images are required. Protocol ceilings are +5 MiB file payloads and 1 MiB command input/output bound; profiles tighten them. Command duration is +positive and ≤30m; maxEnvironments is 1–10,000. Helm values require +`provider.image, provider.securitySecretName, provider.securityManifest`, explicit ingress selectors, +API/DNS CIDRs, and enabled NetworkPolicy/resource governance. Replicas default to 2; stream/RPC/per-client +RPC limits default to 64/128/32. Arbitrary workload Pod specs and paths are not public inputs. + +`provider.securityManifest` is a JSON **string**, decoded strictly by the provider; it is not a new Helm +RBAC object. Manifest version 1 has `version, generation, issuer, audience, activeKeyID, grantTTL, +clockSkew, keys, tls, clients`. `keys[]` has `id, version, file, publicKeySHA256, activateAt, verifyUntil, +state` (active, verify-only, revoked); TLS has `certificateFile, privateKeyFile, clientCAFile`, confined +relative to the mounted security directory. Secret keys are the operator-chosen filenames referenced +by that manifest, not credential literals in values. At most 64 keys and 256 clients; grantTTL is +positive and ≤5m, clockSkew is non-negative and smaller than TTL. No SPIFFE issuer deployment is assumed. + +Ed25519 grants use header `{alg:"Ed25519", kid, typ:"MECATL-GRANT"}` and claims +`iss, aud, client, owner, binding_id, run_id, claim_id, environment, epoch, grant_generation, +operations, not_before, expires_at, nonce`. The provider verifies exact operation and current durable +claim, not just signature/expiry. Rotation reloads an immutable snapshot (default 2s polling) and checks +a durable ConfigMap high-water ledger `{generation,digest,fingerprints}` on each RPC, including existing +connections. Key windows, CA/client removal, equal-generation policy drift, key-ID/version reuse with +different material, and rollback fail closed. RevokeEnvironment advances the separate per-environment +grantGeneration. Neither revocation nor a changed security generation constitutes termination proof. + +**Rotation publication amendment:** use the existing filename fields as immutable material identities. +Every changed signing key, server certificate, server private key, or client-CA bundle gets a new, +generation-specific filename; never overwrite a referenced name with different bytes. Stage the added +Secret entries before publishing the higher-generation manifest, retaining old entries until no live +or in-flight manifest references them. ConfigMap and Secret projections are not atomic together. +A manifest that arrives before its new files must fail closed without advancing the ledger; a material +projection that arrives first must leave the old manifest's authority unchanged. The loader reads one +manifest and confines each named file through `os.Root`; no projection-pinning helper, new config field, +proto, or engine API is introduced. Before a positive rotation barrier, qualification observes the +intended durable ledger generation; cached PodReady alone cannot establish it. Runtime byte immutability +is an operator publication obligation, not a newly enforced file-history registry. If reused filenames +have already published mixed material, +retries cannot repair equal-generation digest drift: publish a complete bundle at a higher generation, +never reset the ledger. + +A peer may advance the shared ledger after a pinned replica successfully acquires a claim. Its next +read must return structured retryable `not_ready` before dispatch until that replica reloads. Positive +qualification uses a bounded read-only content poll for **only** `CodeNotReady && Retryable`; wrong +content and all other errors fail immediately. This amendment adds no mutating retry. Negative +old-client and revoked-key checks use a current, nonexpired claim, not a released phase claim. + +**Scoped-admin amendment, exact proposed configuration:** add optional `administratorFor: []string` +to each `clients[]` entry, matching existing lowerCamelCase `mayAttestOwner` and `administrator`. +For example, the nonsecret manifest fragment is: + +```json +{"clients":[ + {"uri":"spiffe://example.com/mecatl/harness","mayAttestOwner":true,"administrator":false}, + {"uri":"spiffe://example.com/mecatl/operations","mayAttestOwner":false,"administrator":true, + "administratorFor":["spiffe://example.com/mecatl/harness"]} +]} +``` + +This is a policy fragment, not a complete install manifest. The proposed internal field is +`ClientPolicy.AdministratorFor []string`; no new public API, engine seam, proto field, or general RBAC +layer is needed. The strict manifest decoder owns validation because Helm's existing field is a string: + +- Absent/empty list preserves self-admin only when `administrator:true`. A nonempty list requires + `administrator:true`, contains at most 256 unique canonical URI strings, and rejects wildcards, + duplicates, malformed entries, and noncanonical forms. Reuse the existing URI validator: nonempty + scheme/host, lowercase canonical scheme/host, no userinfo, query, or fragment. No prefix matching. +- Entries name immutable **creator** URIs, not owners, namespaces, or admin-to-admin delegation. + A creator need not remain in the current client allowlist: revoking its login must not prevent + explicitly scoped retirement of its retained allocations. The list alone grants no owner attestation. +- One shared administrative subject decision loads the environment, preserves its immutable + `spec.clientHash`, and matches it against the authenticated admin's own URI hash or an exact listed + creator URI hash. Keep the actual caller separate from the matched creator; do not rewrite creation + identity or trust a client-supplied creator. Require exact ownerHash and revision before disclosing + lifecycle state, then retain each RPC's epoch/UID/schema/generation and operation-replay preconditions. +- Apply the same decision to RetireEnvironment, ReplaceExecutor, RecoverEnvironment, + DeleteRetiredEnvironment, MigrateEnvironment, and RevokeEnvironment, including replay and CAS retry + paths. A receipt is never an authorization bypass. Out-of-scope and missing targets are indistinguishable. +- Administration never grants AttachEnvironment, Files, Shell/StartCommand, run claims, reference + discovery/mutation, or `MayAttestOwner`. Their existing creator and owner checks remain unchanged. +- Include the sorted scope list in the canonical authority digest. Scope removal at a higher generation + must deny subsequent RPCs on already-open connections, including receipt replay; equal-generation scope + edits fail closed. Already admitted lifecycle intents may finish safe reconciliation, but removal + authorizes no new admin request. Old binaries reject the unknown field. Quiesce and upgrade all provider + replicas before publishing it; do not claim mixed-version rolling compatibility. + +### Helm lifecycle amendment + +Default uninstall retains runtime CRs/PVCs and may leave executor Pods running. Consequently it must +also retain workload default-deny/profile NetworkPolicies, the security authority high-water ConfigMap, +and `mecatl-execution-profile-allocations` capacity ledger. Those resources live as long as retained +allocations/executors, not just the provider Deployment. Never render empty data over a retained ledger, +reset its generation, or recreate capacity as empty. Retain operator-owned security material and the +nonsecret profile/security configuration needed for exact reattachment; no chart hook deletes them. + +Supported reinstall uses the **same release name and namespace** and unchanged resource names/profile +identity. It verifies retained resource identity and Helm ownership metadata, explicitly reuses/adopts +only that release's resources, preserves ledger contents, and fails on missing authority history or +foreign/ambiguous ownership while allocations survive. Fresh install initializes ledgers only when +there are no retained allocations to authorize. Changed release/namespace adoption, namespace deletion, +CRD deletion with live resources, force cleanup, arbitrary chart rollback, and authority-generation +reset are unsupported. CRDs require manual upgrade before a quiesced compatible schema migration; +unknown schema versions cannot be downgraded. Ordinary compatible upgrades also preserve live ledger data. + +Qualification must execute real Helm install→upgrade→uninstall→same-release reinstall with retained +PVC data and executors. During provider absence, enforcing-CNI probes must still deny forbidden traffic. +After reinstall, current credentials reattach exactly, old authority/grants stay rejected, capacity +reservations still constrain allocation, and mismatched ownership/history fails safely. Annotation +string checks and the existing no-deletion-hook test are necessary but insufficient. Inventory both +ledgers and retained isolation/configuration in the resource and rehydration ledgers in +`docs/adr/0027-cloud-native.md` during implementation; no destructive hooks or automatic force deletion. + +## In scope — 7 scenarios, in implementation order + +### Scenario 1 — Disabled means absent, including startup validation + +The optional deployment boundary follows [draft ADR 0354](../adr/0354-native-kubernetes-execution.md). +Startup preflight uses side-effect-free `ValidatePlacement`/`ValidateProfile`, not `Bind` +or a throwaway allocation. CLI and real composition tests below exercise that boundary. + +**Acceptance:** +- AC1.1: With execution disabled, mecak8s creates no Kubernetes execution client, informer, goroutine, CRD, RBAC, controller, executor, PVC, or Pod, and performs no execution API call. + - verify: `TestDisabledExecutionStartupDoesNotContactExecutionOrKubernetes`, `TestDisabledCompositionPreservesAllocationsAndIndependentReconciliation`; offline CLI `run()` reaches listener setup with poison execution TLS paths and a test-only kubeconfig/API endpoint untouched. That poisoned-endpoint test guards disabled startup contact; the lifecycle fixture's fake clients are not injected into app.Build, so its action log proves only independent reconciliation, not absence of an app-created client. Constructor/informer absence also follows from the disabled CLI branch and independent provider packaging (not goroutine counts). +- AC1.2: The provider service deploys, restarts, and reconciles independently of mecak8s. The default mecak8s chart has no dependency on the provider chart and installs no provider resource; enabling the client adapter requires explicit endpoint/profile configuration. The adapter needs no Pod/PVC/controller management RBAC. + - verify: `TestChartRetainsCRDAndDoesNotGrantSecretAPI`, `TestKindExecutionProductionReplicaLifecycle`; inspection — default chart dependency/RBAC review PENDING; final Kind execution passed at the tested runtime SHA. +- AC1.3: Preflight validates configured profiles and endpoint/provider compatibility without allocating an environment. Allocation occurs only at an authorized session-binding operation; enabled configuration that is incomplete fails startup before allocation, and a configured but unavailable endpoint fails validation clearly with no local fallback. + - verify: `TestExecutionPreflightRejectsIncompleteAndIncompatibleCLI`, `TestRemoteExecutionPreflightNeverAllocates`, `TestHandlerValidateIsReadOnlyAndEnsureIdempotent`, `TestGRPCReadinessAndMTLSAreMandatory`, `TestKindExecutionQualification`; offline CLI/composition and signed-handler validation cover rejection without allocation; final Kind run passed at the tested runtime SHA. +- AC1.4: Disabling the mecak8s client integration neither adopts nor deletes existing `ExecutionEnvironment` CRs, PVCs, or Pods and does not stop the separate provider's reconciliation for its existing allocations or other authorized clients. + - verify: `TestDisabledCompositionPreservesAllocationsAndIndependentReconciliation`, `TestStartupDoesNotFenceLivePeerOperation`; the separate reconciler advances its retained allocation to Ready after an unrelated disabled Build/run/Close. Its fake clients are test-owned, not injected into app.Build; unchanged objects/actions are a limited independent-lifecycle oracle, not a client-construction guard or cluster restart run. `TestDisabledExecutionStartupDoesNotContactExecutionOrKubernetes` supplies the actual poisoned-endpoint CLI guard. + +### Scenario 2 — One idempotent logical environment is allocated per binding + +The controller owns a logical environment, PVC, and executor lifecycle rather than equating identity +with an ephemeral Pod, as decided in [draft ADR 0354](../adr/0354-native-kubernetes-execution.md). + +**Acceptance:** +- AC2.1: Repeating the same authorized ensure request with the same final idempotency identity returns the same environment generation and does not create a second PVC or Pod. + - verify: `TestHandlerValidateIsReadOnlyAndEnsureIdempotent`, `TestKindExecutionQualification` (final Kind execution passed at the tested runtime SHA). +- AC2.2: A stale/different profile, digest, storage request, generation, or owner fails with the final reviewed conflict/precondition error and never adopts or mutates an unrelated allocation. + - verify: `TestStoreEnsureUsesStableLookupAndRejectsFingerprintDrift`, `TestReconcileRefusesForeignExistingPVCWithoutPersistingUID`. +- AC2.3: Only operator-configured profile references select digest-pinned images and storage; public clients/models cannot submit arbitrary images, Pod specs, paths, URLs, source credentials, or Kubernetes object names. + - verify: `TestLoadProfilesStrictAndDigestPinned`, `TestReconcileCreatesTokenlessNonRootPodAndRetainedPVC`; inspection — public request/schema authority review PENDING. +- AC2.4: After transient PVC or Pod creation failure, including prolonged quota saturation, an existing allocation converges when the failure clears without unrelated CR/reference changes, restart, or manual reconcile. Retries remain rate-limited and continue while the object exists; wrong ownership and missing/mismatched authoritative UIDs never trigger adoption or replacement. + - verify: `TestProvisioningWorkerConvergesWithoutAnotherEvent`, `TestProvisioningPreservesCreateAndConditionErrors`, `TestProvisioningRetriesNeverReplaceAuthoritativeResources`, `TestKindExecutionProductionQuotaSaturation` (Kind quota-restoration proof exists; final execution passed at the tested runtime SHA). + +### Scenario 3 — Existing file tools and Shell are transparent but honest + +The adapter preserves the immutable environment and independent read ledger +(`engine/tool/environment.go:9`; `engine/tool/ledger.go:1`) and the existing WorkspaceReader seam +(`engine/tool/tool.go:314`), under [draft ADR 0354](../adr/0354-native-kubernetes-execution.md). + +**Acceptance:** +- AC3.1: A fresh environment can create, read, conditionally edit, replace, copy, move, remove, list, grep, and glob a small Go fixture with unchanged tool schemas and the existing read-before-edit/version behavior where that contract applies. + - verify: `TestRemoteWorkspaceConformanceThroughSignedGRPC`, `TestRemoteFileToolsPreserveLedgerAndNamespaceContracts`, `TestKindExecutionQualification`; both shared Workspace conformance suites and all nine actual file tools run over loopback mTLS/signed gRPC with the real executor in a test-owned directory. Kind invokes the added remote-tool matrix; final Kind execution passed at the tested runtime SHA. +- AC3.2: Read/Edit/Write preserve their applicable recorded-read and CreateFile/ReplaceFile CAS contracts; Copy/Move/Remove preserve their own positive conformance contracts and do not require an unrelated content read ledger/CAS precondition. + - verify: `TestRemoteWorkspaceConformanceThroughSignedGRPC`, `TestRemoteFileToolsPreserveLedgerAndNamespaceContracts`; shared tests cover create/replace races, stale/zero versions, namespace positives and non-clobbering failures; actual tools cover unread/stale Edit/Write refusal and successful replacement. Namespace operations have no unrelated content-read precondition. +- AC3.3: File access is physically confined through symlink traversal rather than lexical checks alone, and every supported read/mutate/list/search/command request is authorized for the exact environment; stale, expired, revoked, or wrong-environment grants are denied. Detached status/cancel requests authorize before returning Unimplemented; this slice exposes no command-stream RPC. + - verify: `TestEveryFileOperationRequiresExactCurrentGrant`, `TestFileOperationsNeverTraverseExternalSymlink`, `TestRemoteWorkspaceConformanceThroughSignedGRPC`, `TestSecurityManagerGuardsEveryRPCOnExistingConnection`, `TestRevokeEnvironmentFencesOldClaimWithoutChangingExecutionEpoch`, `TestCommandStatusIsUnimplementedOnlyAfterAuthorization`; every supported file operation has a valid positive control before wrong-client/owner/environment/revision/epoch/operation, expired, and revoked-nonce rejection at the shared handler gate. Executor tests cover symlink source/destination traversal and search leakage; durable generation revocation remains covered at the store boundary. +- AC3.4: Shell is bound to the same environment namespace as Workspace. Cancellation distinguishes requested cancellation, acknowledged complete process termination, and externally proven compute/storage fencing; namespace teardown includes detached descendants or fails closed. Output is capped and valid UTF-8 repaired, and Shell-created file changes are not falsely claimed to participate in Workspace CAS. + - verify: `TestForegroundCommandReturnsReceiptAndTimeoutTerminatesDescendants`, `TestSuccessfulDetachedChildIsKilledBeforeTerminalReceipt`, `TestCancelledStoreOperationStaysActiveUntilBackendStopsThenFences`, `TestNativeShellCapsCombinedOutputAndRepairsUTF8`, `TestNativeShellWriteInvalidatesRecordedRead`, `TestProviderThroughRealGRPCSignedHandlerRefreshesAndReattachesExactly`, `TestServiceUsesRealMTLSProviderStoreAndReleasesOnlyAfterDrain`, `TestKindExecutionProductionHolderLossFencesActiveOperation`; the native helper executes oversized stdout/stderr and split malformed-byte output with combined-cap/truncation assertions, and writes after a recorded Read so actual Edit rejects the stale version without overwriting Shell's content. Private protobuf bytes may remain malformed; signed gRPC runner and model/client-event assertions cover text repair separately. Final Kind execution passed at the tested runtime SHA. + +### Scenario 4 — Caller isolation and execution ownership survive failures + +A session lease is session-only and its token is not consulted for writes +(`engine/port/lease.go:37-48`); [draft ADR 0354](../adr/0354-native-kubernetes-execution.md) +therefore requires independent environment execution fencing. + +**Acceptance:** +- AC4.1: Ordinary creator client A cannot discover, attach, execute in, stream from, cancel, or retire creator client B's environment; hidden and absent allocations are indistinguishable at the public boundary. The explicit scoped administrative exception in AC4.4 permits only its named administrative RPCs. + - verify: `TestClientScopedIntentListReturnsAttestedOwnerOnlyToOwningClient`, `TestScopedAdminAllRoutesOverMTLS`, `TestScopedAdminWithOwnerAttestationCannotUseAnotherCreatorsDataPlane`, `TestKindExecutionQualification` (final Kind execution passed at the tested runtime SHA). +- AC4.2: Every command is authorized for one immutable environment revision and a distinct transient execution-fence epoch; lease loss or grant revocation prevents new commands and cancels/fences active work without exposing capability credentials to the workload or another caller's output. Same-Pod placement alone is not a security boundary. + - verify: `TestGrantRoundTripAndExactBindings`, `TestRevokeEnvironmentFencesOldClaimWithoutChangingExecutionEpoch`, `TestAcquireRunAndRevokeUseResourceVersionCAS`, `TestSecurityRotationPinnedReplicaReadConvergesBeforeDispatch`, `TestSecurityRotationMutableNamesPoisonSameGeneration`, `TestSecurityRotationImmutableNamesHandleProjectionSkew`, `TestReadCurrentAuthorityContentRetriesOnlyPreDispatchLag`, `TestReadCurrentAuthorityContentBoundAndCancellation`, `TestKindExecutionProductionSecurityRotation`, `TestKindExecutionProductionHolderLossFencesActiveOperation` (final Kind execution passed at the tested runtime SHA). +- AC4.3: Timeout, Pod deletion, controller restart, Lease timeout, or network partition alone never authorizes a replacement executor while an old writer may run; unknown fencing state fails closed and requires the reviewed manual/external fencing path. + - verify: `TestExpiredOperationLeaseFencesWithoutClearingIdentity`, `TestRecoverMissingPodRemainsFenceUnknown`, `TestRecoveredTerminalProofCanStartExactReplacement`, `TestKindExecutionProductionHolderLossFencesActiveOperation` (final Kind execution passed at the tested runtime SHA); inspection — external fencing runbook review PENDING. +- AC4.4: A distinct `administrator:true` client with `administratorFor:[creatorURI]` can administer that creator's environment through all six administrative RPCs, including migration, revocation, and retained deletion, while preserving immutable creator identity and exact owner/revision/epoch/UID/operation checks. A second creator, wrong owner, stale identity, or replay with changed inputs is denied without existence disclosure. Self-admin compatibility remains explicit; no Files/Shell/attach/run/reference or MayAttestOwner authority is gained. + - verify: `TestScopedAdminAllRoutesOverMTLS`, `TestScopedAdminCASRetryRechecksSubject`, `TestScopedAdminMigrationReceiptRetainsUIDPreconditions`, `TestMigrationCompletedCASReplayRequiresExactSourceSchema`, `TestMigrationReceiptExpiresAfterReconciledReplacement`, `TestScopedAdminDoesNotGrantAttestationOrDataPlane`, `TestScopedAdminWithOwnerAttestationCannotUseAnotherCreatorsDataPlane`, `TestKindExecutionProductionScopedAdministrator`; inspection — replacement-expiry regression is present in the implementation candidate; final Kind execution passed at the tested runtime SHA (Kind covers replacement/non-escalation, not all six routes). +- AC4.5: Strict `administratorFor` validation and canonical digest inclusion prevent ambiguous scope. Removal at a higher security generation takes effect on existing connections and receipt replay; an equal-generation scope edit fails closed. Old binaries reject the field and the documented quiesced upgrade preserves authority history. + - verify: `TestAdministratorScopeManifestValidation`, `TestAdministratorScopeDigestIsNormalizedAndAuthorityBound`, `TestScopedAdminAllRoutesOverMTLS`, `TestScopedAdminEqualGenerationDriftFailsClosed`; inspection — old-binary rejection/quiesced-upgrade compatibility review PENDING. + +### Scenario 5 — Restart preserves workspace while retirement is deliberate + +Exact reattachment and safe lifecycle behavior follow [draft ADR 0354](../adr/0354-native-kubernetes-execution.md) +and remain separate from the session-only lease contract (`engine/port/lease.go:37-48`). + +**Acceptance:** +- AC5.1: After mecak8s and controller restart, exact reattachment to the persisted environment revision restores the same PVC data and a correctly bound runner; an unavailable/mismatched generation fails with no default/local fallback. + - verify: `TestProviderThroughRealGRPCSignedHandlerRefreshesAndReattachesExactly`, `TestKindExecutionQualification`, `TestKindExecutionProductionReplicaLifecycle` (final Kind restart/data-survival execution passed at the tested runtime SHA). +- AC5.2: Executor Pod replacement preserves the logical environment and PVC while current Pod UID may change without a spec `metadata.generation` change; `observedGeneration` records processed spec generation, and status records current Pod/PVC UID references without using them as durable session identity. + - verify: `TestReplacementPersistsTerminalProofBeforeRemovingPodFinalizer`, `TestStaleReplacementObservationCannotOverwriteCompletion`, `TestKindExecutionProductionReplicaLifecycle`; inspection — observedGeneration/spec-generation distinction review PENDING; final Kind execution passed at the tested runtime SHA. +- AC5.3: The lifecycle matrix is explicit: removal of one reference retains the environment; removal of the last reference retains it by default; committed data retires only through explicit authorized retirement; a retiring environment rejects new bindings and successors. An unreadable store/reference is not an orphan and is retained. + - verify: `TestReplacementQuiescesAndPendingReferenceBlocksRetirement`, `TestReferenceTransactionsRetainUnknownAndNeverChangeSource`, `TestKindExecutionProductionPendingDeleteOutageRecovery` (final Kind execution passed at the tested runtime SHA). +- AC5.4: A pending allocation whose session association might have committed is not garbage-collected merely after TTL; collection waits for conclusive reconciliation that publication is absent. Explicit retirement of a live/shared environment is refused while any live reference exists unless a separately reviewed retire/quiesce policy authorizes it. + - verify: `TestAmbiguousCommitIsRetainedAndNeverAbortedByCleanup`, `TestReplacementQuiescesAndPendingReferenceBlocksRetirement`, `TestReferenceIntentReconciliationIsOwnerAndRefExact`. +- AC5.5: Default chart uninstall retains runtime CRs/PVCs and preserves workload network isolation and security authority history while executors survive. Retained security/capacity ledgers and required configuration outlive the provider Deployment. Deletion of CRDs with live resources is destructive and unsupported; no destructive hook or force cleanup is provided. Owner references or retention annotation strings alone are not lifecycle proof. + - verify: `TestChartHasNoDeletionHook`, `TestChartRetainedLifetime`, `TestKindExecutionProductionHelmLifetime` (offline rendering is partial; actual Helm lifecycle execution passed at the tested runtime SHA). +- AC5.6: PVC persistence is not documented as node-disaster recovery; kind host-local storage proves restart survival only, and a dead or unobservable node is outside that positive guarantee. + - verify: inspection — human review of deployment documentation; `task docs` checks links and structure after authorized tracked changes +- AC5.7: Real Helm install/compatible upgrade/uninstall/reinstall of the same release and namespace preserves exact runtime identity, PVC contents, NetworkPolicy enforcement during provider absence, security high-water/key history, and capacity reservations. Safe adoption validates ownership; missing ledgers with retained allocations, foreign resources, stale authority, arbitrary downgrade, or incompatible profile/schema fail closed without overwrite. CRD upgrades remain manual. + - verify: `TestChartRetainedLifetime`, `TestSecurityLedgerRejectsKeyVersionRollbackAcrossRestart`, `TestKindExecutionProductionHelmLifetime` (real Helm lifecycle test exists; final execution passed at the tested runtime SHA). + +### Scenario 6 — Remote catalog and project-source boundaries are explicit + +The first slice avoids claiming that local project trust applies remotely: current admission is rooted +in local composition (`internal/app/project_ingestion.go:20`) and AGENTS discovery reads Workspace +(`engine/prompt/builder.go:337`). The boundary is recorded in [draft ADR 0354](../adr/0354-native-kubernetes-execution.md). + +**Acceptance:** +- AC6.1: A remote execution session receives operator-global instructions and only its explicitly supported catalog; project AGENTS/rules/skills and Git source ingestion are off unless a later reviewed source/trust contract enables them. Existing operator-global and explicit-file permission policy remains effective, including configured Ask/Deny under auto; these authorization rules are distinct from prompt rules. Project permission sources are excluded, without a new trust decision. + - verify: `TestRemoteExecutionRealFactoryCarriesPostureAndAttenuatedCatalog`, `TestRemoteExecutionPreservesOperatorPermissionsUnderAuto`; real factory invocation plants host and remote AGENTS, host rules/skills/commands, invokes the forbidden project skill, and proves operator-global prompt rules/skills survive. A test-owned Git repository yields a nonempty snapshot in the local factory request and none in the remote request. Under auto, both conventional global and explicit-file Deny/configured Ask survive remote selection, while project Deny/Ask are excluded and allowed Shell controls execute only on the in-memory runner. +- AC6.2: Schedules remain excluded. Background commands, isolated Subagent/Parallel/Team delegation, SkillDraft, Git/source operations, and unsupported successor modes are rejected with named capability/precondition errors rather than running locally or silently omitting work; `no-fs` remains unchanged. + - verify: `TestRemoteExecutionRealFactoryCarriesPostureAndAttenuatedCatalog`, `TestRemoteDeploymentNoFSUsesLocalAttenuationWithoutProviderCall`, `TestExecutionPreflightRejectsIncompleteAndIncompatibleCLI`, `TestServiceUsesRealMTLSProviderStoreAndReleasesOnlyAfterDrain`, `TestCommandStatusIsUnimplementedOnlyAfterAuthorization`; real factory tests invoke Subagent/Parallel/Team/SkillDraft/Schedule and require model-visible errors, reject worktree discovery, and withhold command listing/expansion instead of opening host paths. The native mTLS/store integration rejects background Shell with its named runner-capability error and no executor dispatch, while foreground Shell succeeds. Actual Service Clear/Fork calls with a nonempty unsupported worktree selector return `ErrInvalidPlacementSelection`, publish no destination, and preserve source ref/history and provider references. A direct native Bind with a valid binding ID distinguishes the unsupported-selector gate from missing binding identity or malformed Service input. +- AC6.3: Clear/Fork use environment exclusion and reference reservation/publication while preserving same-remote-`EnvironmentRef` successor semantics and the source on publication failure. Clear/Fork are not workspace cloning; no public Harness field is added and no model chooses an environment/profile. + - verify: `TestReferenceTransactionsRetainUnknownAndNeverChangeSource`, `TestKindExecutionProductionClearForkLifecycle` (final Kind execution passed at the tested runtime SHA); inspection — public API/profile authority review PENDING. + +### Scenario 7 — kind proves an offline coding flow; live qualification is separate + +The deterministic correctness gate is a focused mock-only task, following [draft ADR 0354](../adr/0354-native-kubernetes-execution.md). +The stack additionally requires the explicitly invoked native-provider OpenRouter coding qualification +recorded in Human decisions; it is completion evidence, not an always-on CI dependency. The broad mecak8s +live suite is not native-provider proof. The focused mock task must not inherit ambient provider credentials. + +The existing `e2e-live.yml` workflow may run this qualification only through explicit +`workflow_dispatch` with `native_execution: true` (default false) in `stacklok/mecatl`. +An optional `expected_sha` must match the checked-out event SHA exactly before credential +staging; record that public SHA in the summary. Maintainer dispatch, repository restriction, +and repository-secret access are the trust gates; there is no environment-review gate. +Run production qualification without automatic cluster deletion, then the live target in the +same job only after production succeeds. The ownership record alone is not a success receipt. +Only then stage `OPENROUTER_API_KEY` in a uniquely created private CI directory/file +(0700/0600); absence fails qualification. Remove the key environment before executing the +live script. Its trusted loader, UID-pinned Secret cleanup, and mock restoration remain +mandatory; cleanup failures fail the job. Always attempt deletion of only the recorded, +validated CI-owned cluster and credential file. Upload only the bounded `live-summary.json` +and sanitized qualification status, never credentials, receipts, kubeconfigs, PKI, Helm +values, or transcripts. A 100-minute job bound covers production, mock rerun, live smoke, +builds, and cleanup; it is not a dollar cap. Candidate runtime evidence is recorded below; +independent security review remains PENDING. + +The rotation restart proof must compare the observed grant generation and replayed revoke +receipt exactly with the original revoke result. A current, successful post-restart claim +must bracket a probe signed by the current trusted key with only its grant generation +changed to the revoked value. Require the structured claim-generation rejection, not an +arbitrary error from a released/expired claim, retired key, or unavailable provider. + +**Acceptance:** +- AC7.1: `task e2e:k8s:execution` and the enforcing-CNI `task e2e:k8s:execution:production` run on the supported generic toolchain, including CI, with a unique owned kind cluster, a scratch kubeconfig, and explicit `--kubeconfig`/context on every command. They deploy the separate chart plus explicitly connected mecak8s and use only the deterministic mock provider. The final production-profile run uses Calico and exercises negative network isolation, rotation/revocation, replica/lifecycle recovery, holder loss, quota recovery, pending delete, Clear/Fork, migration, sanitized artifacts, and the amendment regressions. + - verify: `TestLegacyFixtureWaitsForQuotaAccountingBeforeCreate`, `TestKindExecutionQualification`, `TestKindExecutionProductionNetworkPolicyEnforced`, `TestKindExecutionProductionSecurityRotation`, `TestKindExecutionProductionReplicaLifecycle`, `TestKindExecutionProductionHolderLossFencesActiveOperation`, `TestKindExecutionProductionQuotaSaturation`, `TestKindExecutionProductionPendingDeleteOutageRecovery`, `TestKindExecutionProductionClearForkLifecycle`, `TestKindExecutionProductionCompatiblePrototypeMigration`, `TestKindExecutionProductionScopedAdministrator`, `TestKindExecutionProductionHelmLifetime` (final production Kind+Calico run passed at the tested runtime SHA). +- AC7.2: The mock flow seeds a small Go fixture through file tools, asks the harness to implement a function/tests, runs `go test` through bound Shell, and verifies the persisted artifact and restart/reattach behavior. + - verify: `TestKindExecutionQualification` (coding/artifact/restart assertions exist; final Kind execution passed at the tested runtime SHA). +- AC7.3: Failure artifacts are bounded and sanitized; cleanup targets only the recorded owned cluster. Destructive live cleanup requires confirmation and never mutates ambient kubeconfig/context. + - verify: `TestKindExecutionProductionFailureArtifactBoundary`, `TestLifetimeOutputRejectsOversizedArtifacts`; inspection — owned-cluster cleanup/context script review PENDING; final Kind execution passed at the tested runtime SHA. +- AC7.4: `task e2e:k8s:execution:live` is a separate explicit native-provider OpenRouter coding qualification, required to complete this stack after final deterministic qualification. It proves model-driven file/tool execution, independently verified persisted workspace artifacts, and a successful bound `go test` command. It uses verified nonsecret endpoint/protocol/model configuration in an authenticated operator-controlled cluster, then restores mock configuration and deletes only the run-owned Secret by recorded UID. It is not an always-on live CI dependency and claims no hard dollar cap from token/run limits. + - verify: `TestKindExecutionLiveQualification`; inspection — successful exact-head native-provider live evidence is recorded in the candidate completion receipt (not generic mecak8s live CI). +- AC7.5: A trusted runtime-only loader may consume the approved credential directly from its protected file channel and create a narrowly scoped HARNESS-only Secret through a protected channel before child processes or tools can inherit it. The agent never accesses the file or Secret; failures are redacted, and the credential never appears in model/tool output, argv, Helm values, disk manifests, git, logs, artifacts, or the execution workload. Mock tests do not access it. + - verify: `TestLoadCredentialAndRedaction`, `TestDeletePinsReceiptUIDAndLeavesReplacement`, `TestSecretReferencesFindEveryCredentialEscape`, `TestExpectedHarnessCredentialReferenceIsExact`, `TestKindExecutionLiveQualification`; inspection — offline synthetic-credential tests do not establish live handling; authorized exact-head native live and cleanup evidence is recorded in the candidate completion receipt. Human security review remains PENDING. + +## Implementation proof map + +Source/test locations in this table and the `verify:` lines refer to tested runtime commit +`2020e59934369e8f771059d0c173fa4cbf964c7e`, not files supplied by this plan branch. +The labels name actual tests; source resolution alone is not successful execution or complete +behavioral coverage. Offline boundaries and remaining partial proof are stated per AC. +The following completion receipt supersedes historical pending-runtime status, not human review +or missing behavioral coverage. Documentation-only descendants retain this tested-runtime attribution; +they are not represented as newly live-qualified heads. +The migration-receipt replacement-expiry regression `TestMigrationReceiptExpiresAfterReconciledReplacement` +is already present in that candidate's `internal/adapter/executioncontroller/production_lifecycle_test.go`. +Run `actrace --strict --plan ` from the implementation worktree +for source resolution, then rerun after its normal plan merge. Draft status is intentional: +strict mode gates landed plans only, so inspect the native report for missing proofs as well. +A row naming a unit/fake-client test establishes only that boundary; it does not prove the entire AC. + +### Candidate completion receipt + +The operator supplied the following verified results for runtime SHA +`2020e59934369e8f771059d0c173fa4cbf964c7e`. This documentation-only receipt performs no +new live run, credential access, or cluster operation. + +- **Required native qualification: SUCCESS.** Manual exact-head run + [35700303737](https://github.com/stacklok/mecatl/actions/runs/35700303737) passed all + 11 production Kind+Calico scenarios, then the real OpenRouter coding flow using + `anthropic/claude-haiku-4.5` (30,521 input / 713 output tokens). The model used Write + and Shell and ran exactly `go test ./...` with exit 0. An independent typed-gRPC + verifier attached to the exact persisted ref, acquired run ownership, verified both + generated files including the nonce and tests, and ran its own exact `go test ./...` + with exit 0. The safe artifact is `live-summary.json`. +- **Cleanup and credential boundary: passed.** The filtered log records + `cleanup verification passed: mock harness restored; run-scoped Secret receipt cleared`. + Explicit workflow private-file deletion and the owned-cluster absence check also passed. + No actual credential bytes were accessed by the agent or tools. The offered local file + was not used; the CI secret reached the harness through the trusted loader. +- **Current PR checks: 31 success, 8 skipped, 0 failed, 0 pending.** Successful runs: + [ordinary CI 35700310195](https://github.com/stacklok/mecatl/actions/runs/35700310195), + [native/Kind 35700310187](https://github.com/stacklok/mecatl/actions/runs/35700310187), + [ordinary live 35700310242](https://github.com/stacklok/mecatl/actions/runs/35700310242), + [performance 35700310255](https://github.com/stacklok/mecatl/actions/runs/35700310255), and + [Deslop 35700310122](https://github.com/stacklok/mecatl/actions/runs/35700310122). + The PR synthetic merge SHA is `c2a9e9a87f36dfbaea6658c1491426847320aecd`, distinct + from the manual run's exact runtime head. Ordinary live CI is complementary, not the + native qualification proof. Full-race CI remains skipped while draft; no execution of + that lane is claimed. Earlier runtime revisions passed local full `task test` with race; + the latest small verifier changes passed targeted race tests. The earlier unrelated + server-lease-test flake did not recur; this is not a claim that it was fixed. + +**Resolution of the historical live blockers:** removing the live-only helper-image rewrite +preserves the already-qualified Kind storage configuration. That rewrite selected executor UID +65532 for a storage helper which must create a directory beneath a root-owned 0755 parent. +The readiness hang is gone; session creation completed in 8 seconds in the successful run. +The independent verifier's later Ensure call used the wrong operation identity for an already +published binding and was correctly rejected. It now reuses read-only `environmentForBinding` +lookup and typed exact-ref Attach plus run acquisition; provider semantics were not weakened. +Instrumented debug-stage capture runs before restoration with a safe bounded allowlist. +The forensic history below retains what each earlier observation did and did not establish. + +**Remaining gates:** human API/schema/security approval, release-maturity designation, +independent final panel review and final human review/merge remain open. Passing scenarios do +not expand their assertion scope: in particular, the scoped-admin Kind case is replacement/ +non-escalation, not all six routes, and the disabled-composition fake-client limitations remain. +Strict trace's 32 resolved ACs are a source map, not blanket behavioral acceptance. Native live +qualification is no longer pending or a blocker. Neither PR nor ADR 0354 is approved or shipped. + +### Historical qualification and repairs + +Historical failures below are retained for diagnosis. Their pending-run statements describe +those revisions only; the candidate receipt above supplies the subsequent successful runtime +resolution without assigning an unproved root cause to every earlier failure. + +**Prior offline verification:** implementation `0df361480b2b9c1b9a02ca5c7f55362d0e257966` +passed targeted race +packages (execution client/controller/executor/protocol, mecak8s, and remote composition), the +build-tagged quota test, compilation of the Kind qualification and fixture packages without running +Kind, full `task lint`, full `task test` (including race and standalone module checks), `task build`, +`task docs`, `task site:build`, and the offline demo. Local logs are under the implementation's ignored +`.scratch/orchestrate/native-kubernetes-execution/coverage-*.log`. Source conformance does not qualify +Kubernetes admission, CNI enforcement, or live-model behavior; those remain the runtime gates below. + +**Bounded follow-up verification:** `773a203df` preserves operator permission sources through the +existing workspace-pinned resolver with no workspace. The real-factory auto-posture regression first +failed with both Deny/Ask commands executed and no configured ask, for global and explicit-file sources; +it passes after the fix. Local/no-FS policy selection and model configuration are unchanged. Default, +selector, and client-bound remote factory paths share `remoteSessionConfiguration` before catalog and +engine assembly. The added AC3.4/AC6 proofs pass targeted race tests; full lint, docs, site build, +Taskfile build, and offline demo pass. Logs are under the implementation's ignored +`.scratch/orchestrate/native-kubernetes-execution/final-bounded-*.log`. The first full-suite attempt +hit the unrelated status-line clock-refresh timeout; its isolated race rerun and one final full +`task test` rerun passed, including all module/standalone checks. +These are offline results, not new-head CI, live qualification, or approval. + +**Observed qualification:** the [production Kind+Calico job](https://github.com/stacklok/mecatl/actions/runs/35589357385/job/106299991747) +completed **SUCCESS** at historical implementation `deaf1c3d1dfe7ea92afc8fe826f1bc613080f219`. This includes network isolation, +rotation/revocation, replicas/lifecycle, holder loss, quota saturation, pending-delete recovery, +Clear/Fork, compatible prototype migration, and artifact isolation. Historical local keyring/toolchain +failures are not active blockers for that run. This is baseline evidence, not final-candidate evidence +for the new scoped-admin, continuing-retry, or Helm uninstall/reinstall requirements. + +Draft CI skipped race lanes; non-race success is not a race result. The newer +[production Kind+Calico job](https://github.com/stacklok/mecatl/actions/runs/35642635684/job/106475375010) +at `9bb4d89b8aa497c9d2cd6aa4a94ab9262a913364` failed **before qualification tests** while +seeding `legacy-migration`: ResourceQuota accounting for +`count/executionenvironments.execution.mecatl.dev` was still unknown. Provider Deployment +readiness was not quota readiness. This supersedes a test-race diagnosis for that job. +The implementation adds a six-minute, context-aware quota-accounting wait (the +five-minute Kubernetes default usage resync plus margin) with delayed-status, +timeout, cancellation, and forbidden-response controls in +`TestLegacyFixtureWaitsForQuotaAccountingBeforeCreate`. It does not disable quota or retry +arbitrary forbidden creates. Those offline controls and tagged compilation are not a rerun +of the production job. + +**Earlier reported production context:** the operator supplied run +[35650128625, job 106500190798](https://github.com/stacklok/mecatl/actions/runs/35650128625/job/106500190798) +at `f7919f6c`: 10/11 production tests passed, including scoped administration, Helm lifecycle/quota, +and general file tools. Security rotation failed after about 250 seconds on the final-authority Files +READ immediately after successful AcquireRun (`production_qualification_test.go`, former line 279). +This report was not independently re-fetched during the offline repair. It supersedes the earlier +setup failure for that reported run, not as a final-candidate success. + +**Deterministic repair evidence:** implementation `96e16e926` adds +`internal/adapter/executioncontroller/security_rotation_test.go`: +`TestSecurityRotationPinnedReplicaReadConvergesBeforeDispatch` proves peer advancement → retryable +`not_ready` with zero Files dispatch → reload → the same valid bridge-k2 claim succeeds. +`TestSecurityRotationMutableNamesPoisonSameGeneration` proves that a new manifest with old +`tls.crt`/`tls.key`/`clients.pem` can publish a mixed digest that the later complete bundle cannot replace +at that generation. This is a reproduced defect, **not proof that it caused the CI failure**. +`TestSecurityRotationImmutableNamesHandleProjectionSkew` proves both projection orders recover using +new immutable filenames while retaining old entries. All three pass offline with `-race`, without +sleeps or a cluster. The native exact-cap stdout/stderr/combined repair-expansion regression first +failed with `Truncated=false`, then passed with `-race` after tracking clipping after UTF-8 repair; +earlier bounded-writer truncation remains ORed into the result. Implementation `faacfcde7` stages and +retains immutable fixture names, observes the requested ledger generation, and uses a bounded read-only +barrier (30 seconds, at most 60 calls) that admits only structured retryable `not_ready`. +`TestReadCurrentAuthorityContentRetriesOnlyPreDispatchLag` and +`TestReadCurrentAuthorityContentBoundAndCancellation` pin success, immediate wrong-content/nonretryable/ +wrong-code failure, persistent lag, and cancellation without sleeps. The rotation generator's +`TestForwardRestoreTrustBundleSupportsFixtureClient` pins the actual manifest filename references; +its new filename assertions failed before the fixture fix and pass afterward. Tagged race tests and +compilation pass. Old-client rejection uses the current successful final claim; the revoked-k1 probe +reuses Helm's re-sign/current-claim positive controls. No new mutating retry was added; the existing +bounded AcquireRun helper is unchanged. Full untagged `task lint` and one full `task test` invocation +(including module race and standalone checks), docs/checker, site build, strict +native-plan trace (32 ACs, zero missing; draft/report-only), targeted four-package race tests, and the +offline demo pass. Tagged whole-fixture lint exposes unrelated pre-existing findings; the changed-line +gate against baseline `2d32864cee8378939cb55137e1ab369e70a18349` passes with zero issues. +These results do not replace final runtime qualification; no live inference or cluster run is claimed. + +**Historical repair evidence (offline, after `0e8c8a5ca`):** the operator reports all +11 native production tests passing at `7e498bd73`, followed by production failures +at `2e1aa6bbf` before live qualification. Manual run +[35659529618](https://github.com/stacklok/mecatl/actions/runs/35659529618) failed before +provider credential staging; no provider credentials were used. These reports +were not independently re-fetched during this repair. The production timeouts +remain historically unexplained, not active qualification blockers after the successful candidate +run and not evidence that either source defect below caused them. + +- `TestRetainedDeletePeerBetweenFinalizerUpdateAndDelete` deterministically failed + when a second reconciler re-added the environment finalizer between Update and + Delete, stranding the authorized operation behind the generic deletion branch. + It also failed for an already-terminating authorized CR. Both pass after routing + only admitted retained deletion ahead of generic finalization. The test checks + one successful CR/PVC deletion and released capacity. Foreign-CR identity and + unapproved direct deletion have separate fail-closed controls; Delete also pins + resourceVersion and UID. No CRD, protobuf, or authorization contract changed. +- The holder-loss fixture's `nohup` timer was a descendant of the executor command. + Disconnect cancellation kills the process group and reaps descendants, so that + timer could legitimately disappear without terminating the Pod. The fixture now + proves FenceUnknown and overlap denial first, then uses exact-owned-Pod UID-guarded + Kubernetes deletion with normal grace. It waits for Succeeded/Failed and every + declared container's terminated state; absence is never proof. Offline tests + reject foreign UID/owner and incomplete terminal evidence. This causal mechanism + is source-proven; the historical timeout's cause is not. +- The legacy quota fixture waits up to six minutes, respects cancellation, fails + immediately on forbidden responses, and reports only allowlisted missing/mismatched + quota keys and elapsed time on deadline. Tiny-context offline tests cover readiness, + deadline, and cancellation without a six-minute test or blanket create retries. +- The manual workflow collects bounded, closed-vocabulary diagnostics before owned + cleanup when production fails. Offline fixture tests verify redaction, UID-match + booleans, known finalizers, quota key names, and collection even when deletion fails. + The CI-owned path uses the same collector. No raw manifests, commands, grants, + credentials, private URLs, or PKI enter its artifact allowlist. + +The retained-deletion, terminal-proof, quota-diagnostic, and pre-cleanup evidence +regressions were observed red then green offline. At that repair, amended-candidate production +and native live runs were **PENDING**; no cluster, live model, dispatch, or push +was performed for that repair. `0e8c8a5ca` removed SC2016, but earlier local +actionlint ran without ShellCheck and is not a ShellCheck pass. The plan baseline +remains `7d9e37a128b17107e2d83467c7767146e9c5efb8` in the normal-merge ancestry. +Reported offline gates at `53ab45453` passed: targeted controller/chart/contract and tagged fixture +race tests, `task lint`, tagged changed-line lint, one full `task test` attempt +(including standalone modules), `task docs` plus final docs/checker checks, +`task site:build`, draft/report-only native-plan trace (32 ACs, zero missing), +and the offline demo. Local actionlint passed with ShellCheck explicitly disabled; +ShellCheck was unavailable on the host and in the existing dev toolbox. CI remains +the authority for ShellCheck and final runtime qualification. + +**Historical live-only storage repair (offline, after `2ff780a64145f427ee6ec49f181fe6d0abb97134`):** +the operator reports production green in run +[35692050845](https://github.com/stacklok/mecatl/actions/runs/35692050845), followed by +same-session-hash ID probe success (0 ms), Ensure success (87 ms), and Attach call 1 +`not_ready_nonretryable`, with no Attach end, factory, or save. Resource evidence +reports Ready=False/Reconciled, a Pending executor, and ProvisioningFailed. These +reports were not independently re-fetched here; the safe projection omits the +provisioning message and does not correlate that event to the session binding. + +Source inspection found an independent live-only storage mutation: `live.sh` +replaced local-path's helper image with the executor image, while `run.sh` leaves +Kind's installed storage configuration intact. [Kind v0.33.0's manifest](https://github.com/kubernetes-sigs/kind/blob/v0.33.0/pkg/build/nodeimage/const_storage.go) +uses a Kind helper image (not BusyBox), no helper Pod/container `runAsUser`, and +`mkdir -m 0777 -p "$VOL_DIR"` for setup. Its +[pinned provisioner build](https://github.com/kubernetes-sigs/kind/blob/69b56db7/images/local-path-provisioner/Makefile) +selects v0.0.34; [helper construction](https://github.com/rancher/local-path-provisioner/blob/v0.0.34/provisioner.go#L616-L726) +copies the template without adding a UID override and mounts the parent directory +as HostPath DirectoryOrCreate. [Kubernetes v1.35.8](https://github.com/kubernetes/kubernetes/blob/v1.35.8/pkg/kubelet/kuberuntime/security_context.go#L51-L57) +uses the image UID without an override; its [host-path creation](https://github.com/kubernetes/kubernetes/blob/v1.35.8/pkg/volume/hostpath/host_path.go#L501-L511) +uses mode 0755. `build/execution-workload/Dockerfile` sets USER 65532:65532, so the +rewritten helper runs as UID 65532 and cannot create a child in the normal +root-owned parent. The recovery-only `local-path.yaml` does specify UID 0, but +neither production nor live applies it. No root override is added by this repair. + +`TestLivePreservesQualifiedStorageAndHelmDigestThroughRestoration` failed on the +old script's forbidden local-path ConfigMap read and replacement, then passed +with `-race` after removing that entire mutation. Its fake CLI allows only mock +configuration/readiness operations and verifies mock test, synthetic credential +staging, live test, and digest-only mock/live/mock Helm transitions. Cleanup and +credential-loader failure tests remain unchanged. This establishes the source +regression, not the exact historical ProvisioningFailed cause or why mock passed. +`internal/adapter/executioncontroller/store.go` (`Store.Attach`) returns NotReady +without Retryable when Ready is false; `internal/adapter/executionclient/client.go` +(`waitForBinding`) still polls all NotReady errors until context cancellation, +logging only state changes. A single logged nonretryable state therefore does not +mean only one Attach request occurred. This classification mismatch and the +local-fallback policy are unchanged. Runtime qualification was pending at this +repair; the completion receipt above records its subsequent successful resolution. +Human reviews remain pending. + +[PR #1728](https://github.com/stacklok/mecatl/pull/1728), commit +`6501b5924`, is already integrated in the implementation ancestry. Its generic live-compaction +repair is not native-provider qualification. The subsequent native-provider live and amended +production Kind success is recorded in the candidate completion receipt, not inferred from #1728. +Source resolution below does not assert human contract/panel approval. + +### Current proof boundaries + +| AC | Existing implementation proof at the reference commit; remaining evidence | +|---|---| +| AC1.1 | `cmd/mecak8s/execution_startup_test.go`: `TestDisabledExecutionStartupDoesNotContactExecutionOrKubernetes` exercises CLI `run()` through composition to listener setup with poison TLS paths and no test-endpoint calls. `internal/adapter/executioncontroller/disabled_composition_test.go` exercises independent reconciliation beside disabled Build/run/Close; its fake clients are not injected into app.Build, so their empty action log does not detect app-created clients. Constructor/informer absence is also inspected at the CLI enablement branch; no flaky goroutine census. | +| AC1.2 | `deploy/helm/mecatl-execution/chart_test.go`: `TestChartRetainsCRDAndDoesNotGrantSecretAPI`; production job separately deploys provider and client. | +| AC1.3 | `cmd/mecak8s/execution_startup_test.go`: `TestExecutionPreflightRejectsIncompleteAndIncompatibleCLI`; `internal/app/remote_execution_test.go`: `TestRemoteExecutionPreflightNeverAllocates`; handler read-only validation and mTLS/readiness tests cover the transport. | +| AC1.4 | `internal/adapter/executioncontroller/disabled_composition_test.go`: `TestDisabledCompositionPreservesAllocationsAndIndependentReconciliation` observes independent reconciliation after disabled Build/run/Close. Retained objects and fake action counts are only a limited lifecycle oracle because the clients are not injected into app.Build; the poisoned-endpoint CLI test in AC1.1 guards startup contact. `controller_test.go`: peer-restart test remains complementary. | +| AC2.1 | `internal/adapter/executioncontroller/handler_test.go`: `TestHandlerValidateIsReadOnlyAndEnsureIdempotent`; `e2e/k8s_execution/qualification_test.go`: `TestKindExecutionQualification`. | +| AC2.2 | `internal/adapter/executioncontroller/store_test.go`: `TestStoreEnsureUsesStableLookupAndRejectsFingerprintDrift`; `controller_test.go`: `TestReconcileRefusesForeignExistingPVCWithoutPersistingUID`. | +| AC2.3 | `internal/adapter/executioncontroller/profiles_test.go`: `TestLoadProfilesStrictAndDigestPinned`; production `TestKindExecutionProductionQuotaSaturation`. | +| AC2.4 | `internal/adapter/executioncontroller/provisioning_retry_test.go`: `TestProvisioningWorkerConvergesWithoutAnotherEvent`, `TestProvisioningPreservesCreateAndConditionErrors`, `TestProvisioningRetriesNeverReplaceAuthoritativeResources` exercise PVC and Pod retry/confinement paths. Production `TestKindExecutionProductionQuotaSaturation` restores quota without a new reference event; final runtime evidence recorded in the candidate completion receipt. | +| AC3.1 | `internal/adapter/executionclient/files_conformance_test.go`: `TestRemoteWorkspaceConformanceThroughSignedGRPC` runs both existing shared suites; `TestRemoteFileToolsPreserveLedgerAndNamespaceContracts` invokes all nine tool bodies against the signed handler and real executor. `e2e/k8s_execution/file_tools_test.go` adds the tool matrix called by Kind qualification; runtime execution passed at the tested runtime SHA. | +| AC3.2 | Same gRPC suites cover concurrent create/CAS, missing/stale/zero versions, non-clobbering namespace operations, and actual Read-ledger behavior. The suite exposed and now guards missing-file replacement precedence and typed non-empty-directory removal errors. | +| AC3.3 | `internal/adapter/executioncontroller/file_authorization_test.go`: every supported file operation has positive and negative signed-grant controls; `internal/executionexecutor/confinement_test.go`: physical symlink traversal/search checks. Existing security manager, protocol, durable revocation, and detached-control tests remain complementary. | +| AC3.4 | `internal/executionexecutor/executor_test.go`: `TestNativeShellCapsCombinedOutputAndRepairsUTF8`, `TestNativeShellWriteInvalidatesRecordedRead` execute the native helper for combined output bounds, malformed bytes, and actual Shell-write→Edit version refusal. `internal/adapter/executionclient/client_test.go`: `TestProviderThroughRealGRPCSignedHandlerRefreshesAndReattachesExactly` checks runner text repair after private protobuf bytes; `internal/adapter/executionclient/service_integration_test.go`: `TestServiceUsesRealMTLSProviderStoreAndReleasesOnlyAfterDrain` checks identical valid UTF-8 in client events and model history. `internal/adapter/executioncontroller/store_test.go`: `TestCancelledStoreOperationStaysActiveUntilBackendStopsThenFences`; production `TestKindExecutionProductionHolderLossFencesActiveOperation` remains runtime qualification. | +| AC4.1 | `internal/adapter/executioncontroller/store_lifecycle_test.go`: `TestClientScopedIntentListReturnsAttestedOwnerOnlyToOwningClient`; Kind isolation and pending-delete tests; scoped-admin non-escalation is exercised by the AC4.4 tests. | +| AC4.2 | `internal/adapter/executioncontroller/store_revoke_test.go`: `TestRevokeEnvironmentFencesOldClaimWithoutChangingExecutionEpoch`; `store_concurrency_test.go`: `TestAcquireRunAndRevokeUseResourceVersionCAS`; the three deterministic rotation tests listed above at repair commit `96e16e926`. The reported production rotation at `f7919f6c` failed historically; final runtime success is recorded in the candidate completion receipt. | +| AC4.3 | `internal/adapter/executioncontroller/production_lifecycle_test.go`: `TestExpiredOperationLeaseFencesWithoutClearingIdentity`, `TestRecoverMissingPodRemainsFenceUnknown`, `TestRecoveredTerminalProofCanStartExactReplacement`; production holder loss. | +| AC4.4 | `internal/adapter/executioncontroller/admin_scope_test.go`: `TestScopedAdminAllRoutesOverMTLS`, `TestScopedAdminCASRetryRechecksSubject`, `TestScopedAdminMigrationReceiptRetainsUIDPreconditions`, `TestMigrationCompletedCASReplayRequiresExactSourceSchema`, `TestMigrationReceiptExpiresAfterReconciledReplacement`, `TestScopedAdminDoesNotGrantAttestationOrDataPlane`, `TestScopedAdminWithOwnerAttestationCannotUseAnotherCreatorsDataPlane`. Replacement-expiry test is in `internal/adapter/executioncontroller/production_lifecycle_test.go` in the pinned implementation candidate. `e2e/k8s_execution/ab_production_admin_test.go`: `TestKindExecutionProductionScopedAdministrator` exists (replacement/non-escalation, not all six routes); final Kind evidence recorded in the candidate completion receipt. | +| AC4.5 | `internal/adapter/executioncontroller/admin_scope_security_test.go`: `TestAdministratorScopeManifestValidation`, `TestAdministratorScopeDigestIsNormalizedAndAuthorityBound`; `internal/adapter/executioncontroller/admin_scope_test.go`: all-routes scope removal plus `TestScopedAdminEqualGenerationDriftFailsClosed`. Compatibility/upgrade human review PENDING. | +| AC5.1 | `internal/adapter/executionclient/client_test.go`: exact reattachment test above; production `TestKindExecutionProductionReplicaLifecycle` and security rotation. | +| AC5.2 | `internal/adapter/executioncontroller/production_lifecycle_test.go`: `TestReplacementPersistsTerminalProofBeforeRemovingPodFinalizer`; production replica lifecycle. | +| AC5.3 | `internal/adapter/executioncontroller/production_lifecycle_test.go`: `TestReplacementQuiescesAndPendingReferenceBlocksRetirement`; `internal/adapter/executioncontroller/retained_delete_race_test.go`: `TestRetainedDeletePeerBetweenFinalizerUpdateAndDelete`, `TestRetainedDeleteDoesNotRemoveForeignCR`, `TestDirectCRDeleteStillBlocked` (offline race controls; final runtime evidence recorded in the candidate completion receipt). | +| AC5.4 | `internal/adapter/executionclient/reference_ambiguity_test.go`: `TestAmbiguousCommitIsRetainedAndNeverAbortedByCleanup`; production `TestKindExecutionProductionPendingDeleteOutageRecovery`. | +| AC5.5, AC5.7 | `deploy/helm/mecatl-execution/chart_test.go`: `TestChartHasNoDeletionHook`; `deploy/helm/mecatl-execution/lifetime_test.go`: `TestChartRetainedLifetime` exercises lookup/adoption, missing history, foreign ownership, profile drift, and retained policies/ledgers offline. `e2e/k8s_execution/zz_production_helm_lifetime_test.go`: `TestKindExecutionProductionHelmLifetime` exercises actual upgrade/uninstall/reinstall and enforcement; final runtime evidence recorded in the candidate completion receipt. | +| AC5.6 | Deployment-documentation inspection and docs gates required on the final implementation; Kind is restart proof only. | +| AC6.1–AC6.2 | `internal/app/remote_execution_test.go`: `TestRemoteExecutionRealFactoryCarriesPostureAndAttenuatedCatalog` includes a nonempty LOCAL Git snapshot control and REMOTE exclusion, poison-source and operator-global prompt controls, forbidden-tool errors and command/worktree discovery. `TestRemoteExecutionPreservesOperatorPermissionsUnderAuto` exercises global/explicit-file Deny and configured Ask, excluding project permissions, with allowed in-memory Shell controls. `TestRemoteDeploymentNoFSUsesLocalAttenuationWithoutProviderCall` retains the no-FS check. `internal/adapter/executionclient/service_integration_test.go`: `TestServiceUsesRealMTLSProviderStoreAndReleasesOnlyAfterDrain` rejects background Shell and actual Clear/Fork unsupported-selector calls with named errors, no destination publication, and unchanged source ref/history and provider references. | +| AC6.3 | `internal/adapter/executioncontroller/store_lifecycle_test.go`: `TestReferenceTransactionsRetainUnknownAndNeverChangeSource`; production `TestKindExecutionProductionClearForkLifecycle`. | +| AC7.1 | `.github/workflows/k8s-e2e.yml` runs `task e2e:k8s:execution:production`; historical and operator-reported runs are distinguished above. The reported 11/11 pass at `7e498bd73` does not supersede later failures at `2e1aa6bbf`. `internal/adapter/executioncontroller/legacy_fixture_kind_test.go` covers the six-minute quota-accounting bound with tiny-context tests. `e2e/k8s_execution/holder_loss_test.go` covers the exact-owned terminal stimulus and complete kubelet evidence offline. Final amended-candidate production run passed all 11 scenarios at the tested runtime SHA. | +| AC7.2 | `e2e/k8s_execution/qualification_test.go`: `TestKindExecutionQualification`, run by the successful production profile. | +| AC7.3 | Production `TestKindExecutionProductionFailureArtifactBoundary`; `deploy/mecatl-execution-kind/failure_evidence_test.go` tests closed-vocabulary projection and API-error redaction; `TestNativeWorkflowCleanupOwnership` proves evidence collection before failed cleanup. Workflow bounded uploads and ownership-scoped cleanup; final runtime evidence recorded in the candidate completion receipt. | +| AC7.4 | `e2e/k8s_execution/live_qualification_test.go`: `TestKindExecutionLiveQualification` passed; successful exact-head native-provider OpenRouter evidence and independent file/test verification are recorded in the candidate completion receipt. | +| AC7.5 | `e2e/k8s_execution/fixture/credentialloader/main_test.go`: `TestLoadCredentialAndRedaction`, `TestDeletePinsReceiptUIDAndLeavesReplacement`; these offline tests do not substitute for live execution. The candidate receipt separately records successful trusted-loader live handling, mock restoration, receipt clearance, private-file deletion, and owned-cluster absence. | + +Production test names without another path above reside in the implementation's +`e2e/k8s_execution/production_qualification_test.go`. Do not import implementation files into this +plan worktree to make citations or trace appear complete. + +## Out of scope + +| Item | Defer-to | Decision | +|---|---|---| +| GitHub cloning, private-repository credentials, and authoritative remote project trust/AGENTS/rules/skills | Separate source-ingestion architecture plan | Blank/PVC workspace and tool-seeded fixture are sufficient for slice 1. | +| Agent Sandbox integration | Not planned for this capability | Human selected a purpose-built controller. | +| Public Harness placement/profile/image/path fields | Later architecture only if justified | First slice keeps the public API unchanged and server-owned. | +| Generic capabilities framework | Later, only with multiple proven consumers | Use narrow optional interfaces/catalog attenuation rather than speculative abstraction. | +| Schedules and remote delegation/fork support | Separate reviewed expansion | First slice rejects unsupported paths without local fallback. | +| Automated force takeover under uncertain fencing | Not permitted | Unknown old executor state is fail-closed/manual until a reviewed external fencing proof exists. | +| Node-disaster PVC recovery and production multi-zone storage | Production storage design | kind host-local PVC is only a restart-survival proof. | +| Provider credential discovery, arbitrary endpoint forwarding, or live inference beyond the approved native qualification | Separate explicit authorization | This stack requires native OpenRouter coding evidence within the protected runtime-loader boundary; planning/mock tests never access credentials. | +| Depending on or copying PR #580's transport | Independent work | Share behavioral conformance where useful; do not inherit unqualified cancellation/auth protocol. | + +## Definition of done + +1. Human API/schema/security review approves the authored interface and amendment requirements, and the release maturity designation is explicit. Both PRs remain draft with no approved baseline until human contract approval; authorized pre-merge development does not waive that gate. +2. The final implementation candidate passes `task lint`, full offline `task test` including race and engine-standalone coverage, `task build`, `task api:check`, `task docs`, `task site:build`, and `go run ./cmd/mecademo`. A skipped draft race lane is not a pass. Generated contracts/references must be fresh; no arbitrary prose-pinning tests. +3. `task ac-trace-strict` resolves every final `verify:` proof, including AC2.4, AC4.4–AC4.5, and AC5.7. Resolve the MISSING/partial behavioral coverage noted above; named-test resolution alone is not AC completion. +4. Fresh deterministic Kind+Calico production qualification passes at the final candidate, including the expanded scope/retry/Helm lifecycle matrix. Then record successful stack-specific native-provider OpenRouter coding qualification under AC7.4–AC7.5. Neither historical deterministic success nor generic mecak8s live CI substitutes for those final results. +5. Complete independent Spec/Standards/Test adequacy/Domain panel review and resolve merge-blocking findings. Update the owning `user-docs/building/deployment/mecak8s.md` and implementation's `user-docs/features/execution-environments.md`, plus resource/rehydration inventories, in the same implementation stack; keep this plan branch documentation-only. +6. The existing implementation PR targets `plan/native-kubernetes-execution`, links plan PR #1579 and its exact working commit, reports conformance/amendments and final evidence. Human alone approves and merges; no new PR, automatic merge, or unsupported landed claim. + +## Deferred decisions and known risks + +- Exact specifications are authored, but human API/schema/security approval and the release maturity designation remain open. Draft development is authorized; promotion/merge is not. Do not turn `v1alpha1` into an unreviewed product “alpha” decision. +- A PVC can preserve files while a stale command still mutates them. Session leasing does not fence command execution; missing old-executor evidence remains a fail-closed manual recovery boundary. +- Only an enforcing CNI supplies negative NetworkPolicy proof. Retaining storage while uninstall removes isolation or authority history is unsafe; the Helm lifecycle amendment is a completion requirement, not deferred polish. +- Every outlives-a-call resource, including retained security/capacity ledgers, reload state, informer/cache, connection pools, claims, and lifecycle/reference reconciliation, needs an accurate resource/rehydration inventory in `docs/adr/0027-cloud-native.md` on the implementation branch. +- Scope, retry, Helm lifecycle, disabled startup, remote filesystem conformance, and source-admission behavioral tests exist in the pinned implementation source. Final production Kind/native live evidence is complete at the tested runtime SHA; human reviews remain pending. The requested offline output/CAS and Git/unsupported-successor proofs in AC3.4 and AC6.1–AC6.2 are present; they do not replace final runtime qualification. Source/proof locations absent on the plan branch stay explicitly implementation-scoped; do not add implementation code to satisfy plan-branch gates. diff --git a/docs/adr/0354-native-kubernetes-execution.md b/docs/adr/0354-native-kubernetes-execution.md new file mode 100644 index 000000000..836ee6b76 --- /dev/null +++ b/docs/adr/0354-native-kubernetes-execution.md @@ -0,0 +1,269 @@ +# ADR 0354 — Optional native Kubernetes execution environments + +- Status: Draft +- Date: 2026-09-15; draft amended 2026-09-21 +- Scope: independently deployed execution environment provider service; mecak8s is an optional client; controller/executor, logical environment, PVC, Pod, authorization, fencing, and kind qualification boundaries +- Supersedes: none + +## Context + +Mecak8s externalizes session state to Redis and uses Kubernetes Leases for session-level +single-writer ownership, but it does not provide a production remote execution environment. +The current remote environment adapter is a fake (`internal/adapter/remoteenv/remoteenv.go:1`). +`tool.Environment` already binds immutable identity, Workspace, an independent read ledger, and an +optional CommandRunner (`engine/tool/environment.go:9`), so a Kubernetes implementation should +meet that runtime contract without widening the engine API. + +A session lease does not fence an already-running remote command: its token is intentionally not +consulted on session writes (`engine/port/lease.go:37-48`). Kubernetes Pod replacement, network +partition, and controller restart therefore introduce a separate execution-ownership problem. +Likewise, startup placement validation currently reaches `Bind` +through `internal/adapter/server/placement.go` (`configuredPlacementBinder`), but using an allocating operation as preflight +would leak PVCs and Pods. + +The human authorized the direction and defaults: a purpose-built execution environment provider in this +monorepo, separate from mecak8s, with its own binaries, images, chart lifecycle, optional client integration, +and deterministic kind qualification followed by native-provider OpenRouter coding evidence. The target +is an operator-controlled first production slice with explicit limits; release maturity remains undecided. + +The existing plan PR #1579 and implementation PR #1614 form a human-authorized **draft** stack. The +[acceptance plan](../acceptance/native-kubernetes-execution.md) now authors the exact proposed contract, +reconciled against tested runtime `2020e59934369e8f771059d0c173fa4cbf964c7e`. API, schema, and security +review approvals remain unchecked; neither authored specs nor draft code is a merged approved baseline. +This amendment also authorizes development of scoped administration, continuing provisioning retries, +and retained-resource Helm lifecycle safeguards on those existing PRs. Human alone merges. + +## Proposed decision within the authorized draft stack + +This ADR records the approved direction: a separate execution environment provider in this monorepo; +it does **not supersede ADR 0048**. ADR 0048 governs storage-free mecak8s. Mecak8s remains a client of +the provider, keeps session state in its existing stores, and neither mounts provider workspace PVCs nor +owns executor/controller lifecycle. The provider owns its CRDs, Pods, PVCs, authorization, +reconciliation, binaries, images, and chart lifecycle. Its API is not tied to a mecak8s process or +release; other authorized Mecatl compositions can use an adapter without embedding the controller. +Provider allocation handles remain separate from harness session IDs. The +[exact interface contract](../acceptance/native-kubernetes-execution.md#interface-contract) is authored +for review in the acceptance plan; this ADR records the durable boundaries rather than duplicating the spec. + +### 1. Own a narrow controller/executor boundary + +Ship a dedicated controller and executor integration rather than Agent Sandbox. The controller +reconciles a namespaced `execution.mecatl.dev/v1alpha1` `ExecutionEnvironment`, a persistent workspace +PVC, and a replaceable executor Pod. Spec and status use explicit schema version 2. The Kubernetes +API name does not settle the release maturity label. Durable identity is the logical environment plus +immutable `EnvironmentRef.Revision`; run epoch, grant generation, Pod UID, and PVC UID have distinct roles. +`observedGeneration` records processed spec generation, not executor identity. + +Only operator-configured, digest-pinned profiles select image, storage, resource/operation bounds, and +RuntimeClass. Public clients/models cannot submit arbitrary Pod specs, images, paths, URLs, credentials, +or Kubernetes names. Creator `clientHash`, owner identity, allocation fingerprint, and revision remain +immutable. The [schema and transaction contract](../acceptance/native-kubernetes-execution.md#reference-transaction-schema-v2-and-fencing) +records exact fields, receipts, conditions, and migration behavior. Completed migration receipts retain +the source schema with explicit presence (including legacy schema zero); replay requires the exact +request identity, and receipts missing that information fail closed. Replacement atomically expires +completed migration receipts when it publishes the new Pod identity; replay with either the original or +replacement UIDs cannot revive an expired receipt. + +Transient PVC/Pod create errors and quota saturation must recover through continuing rate-limited +reconciliation while the CR exists. Quota restoration alone must be sufficient; no unrelated reference +change, restart, or manual reconcile is required. A finite retry count or successful condition write +must not strand provisioning. Wrong ownership or missing/mismatched authoritative UIDs still fail +closed without replacement, regardless of retry scheduling. + +### 2. Preserve the core environment and tool contracts + +Keep `engine/tool` and public Harness APIs unchanged. A host-side adapter maps an exact persisted +`session.EnvironmentRef` to a complete remote `tool.Environment`. Existing filesystem tool schemas, +read-before-edit ledger, CreateFile/ReplaceFile version protocol, and bound Shell semantics remain. +Shell writes retain their documented ability to bypass Workspace CAS +(`engine/tool/tool.go:353-424`); the product must disclose that honestly rather than promising a +stronger remote filesystem transaction. + +The private typed `mecatl.execution.v1.ExecutionProviderService` uses mTLS, with no REST fallback. +`ValidateProfile` is side-effect-free; Ensure reserves a reference, exact Attach reattaches, and an +independent Acquire/Renew/ReleaseRun claim gates operations. Reference and administrative lifecycle +RPCs carry exact identities and replay preconditions. `Files` transports bounded bytes and opaque +versions. `StartCommand` is foreground unary; authorized detached CommandStatus/CancelCommand calls +return Unimplemented. Do not advertise a command stream or background service absent from this slice. +The [RPC and host contract](../acceptance/native-kubernetes-execution.md#private-rpc-and-host-contract) +records exact messages, error classifications, and host-only placement/reference/run interfaces. + +### 3. Make installation and connection explicitly optional + +The provider has its own binaries, images, and Helm chart lifecycle in this monorepo. The mecak8s chart +has no default dependency on it. Kubernetes execution is disabled by default; an explicit endpoint and +operator-selected profile connect the optional client. Disabled means no execution Kubernetes client, +informer, goroutine, API call, CRD/RBAC install, or execution resource. + +An explicitly selected remote environment fails clearly at validation if its configured endpoint is +unavailable, and fails closed at runtime if the provider is lost or its exact immutable revision cannot +reattach. It never falls back to the local project root. Existing local and `no-fs` behavior is unchanged. + +The separate chart is `deploy/helm/mecatl-execution`. Mecak8s enables the client through +`--execution-enabled`, endpoint/profile flags, and three mounted TLS-file flags; OIDC ownership is +required. Profiles and the versioned security manifest are strict operator configuration, not model +inputs. The [configuration contract](../acceptance/native-kubernetes-execution.md#provider-configuration-rotation-and-scoped-administration) +records exact names and bounds. Old binaries reject the scoped-admin field: quiesce and upgrade provider +replicas before introducing it. CRD upgrade remains manual; there is no arbitrary downgrade or +mixed-version compatibility claim for this amendment. + +### 4. Separate execution fencing from session leasing + +The chosen approach is mTLS service links plus a least-privilege execution grant scoped to the exact +environment, operation, and ownership epoch. Initial qualification is authenticated and authorized use in +an operator-controlled cluster, not hardened hostile multitenancy. The arbitrary-shell workload receives +no controller credential, signing key, provider key, or default service-account token. Same-Pod or +localhost placement is not treated as a trust boundary. + +Grant revocation prevents new commands. Cancellation request, acknowledged complete process termination, +and externally proven compute/storage fencing are distinct states. Namespace process teardown must include +detached descendants or fail closed. Pod deletion, Lease timeout, a timeout, controller restart, or +partition is not evidence that an old command stopped and cannot alone authorize takeover. If the old +executor cannot be positively terminated or externally fenced, replacement fails closed. + +Ed25519 grants bind issuer/audience, creator, owner, environment revision, binding/run/claim, operation, +epoch, grant generation, and validity window. ResourceVersion CAS gates run claims and the separate +replica-owned operation lease. Expiry retains uncertain operation identity; it does not authorize a +second writer. Exact terminal Pod/PVC proof is persisted before finalizer removal and replacement; +RecoverEnvironment has no caller-supplied force-fencing assertion. The external-fencing runbook remains +necessary for unobservable nodes. + +Rotation publishes a validated security snapshot backed by a durable generation/digest/key-fingerprint +high-water ledger. Every RPC rechecks current authority, including requests over established connections. +CA/key/client removal, scope removal, rollback, or equal-generation policy drift fail closed. Neither +security generation nor per-environment grant revocation is proof of termination. No existing SPIFFE +issuer is assumed, and no credential crosses the provider-to-workload stdin boundary. + +Material publication uses the existing manifest filenames as immutable identities. Changed signing, +server-certificate, server-key, and client-CA bytes require new generation-specific names. Stage them +before publishing the higher-generation manifest and retain overlap files while any live/in-flight +manifest references them. Secret and ConfigMap updates are not atomic together. One loaded manifest +binds the confined immutable names even across projection changes; missing material fails closed before +ledger publication. No loader projection-pinning helper or new version/config API is required under +this operator-managed publication contract. Reusing names can publish a mixed-generation digest; +correcting that bundle at the same generation is then rejected permanently. Recover only by forward +publication, never by resetting authority history. + +Peer-ledger advancement can transiently reject a pinned replica's read before dispatch even after +successful AcquireRun. Qualification may poll only read-only structured retryable `not_ready`, with a +bound and immediate failure on wrong content or other errors. It must not hide mixed-material drift +or count a released/expired claim as evidence of old-key/client rejection. + +Scoped administration uses one narrow extension to the existing client policy: +`administrator:true` plus optional `administratorFor:[canonicalCreatorURI]`. Absent/empty scope preserves +self-admin compatibility, not namespace-wide privilege. A distinct operations URI may administer only +the listed creators' allocations. The shared decision derives creator identity from the loaded +immutable `clientHash` and keeps exact owner, revision, epoch, UID, schema/generation, and operation replay +checks. It applies to all six administrative RPCs, including migration, revocation, and retained deletion. +A scope never grants attach, Files, Shell, run/reference access, or MayAttestOwner. No public Harness, +engine, or private protobuf widening is needed. The acceptance contract specifies validation and digest +inclusion; authoring this proposal is not the outstanding human security approval. + +### 5. Start with a persistent blank workspace and an attenuated catalogue + +The first slice provisions a blank/PVC-backed workspace that existing filesystem tools seed. It supports +foreground Shell and operator-global instructions only. It has no warm pools or automatic idle suspension. +Git clone, private source credentials, remote project ingestion, background commands, and isolated delegation +are deferred. Project AGENTS/rules/skills remain disabled until a separate root/source trust design exists. Current +project admission is local-root based (`internal/app/project_ingestion.go:20`). + +Schedules remain excluded. SkillDraft and unsupported remote Subagent/Parallel/Team paths return explicit +unsupported/precondition results and never execute locally. Clear/Fork reserve and publish references +under environment exclusion while preserving the same remote `EnvironmentRef`; neither clones the +workspace. Clear creates empty history and Fork copies valid history. Publication failure preserves the +source; ambiguous reference outcomes stay durable for reconciliation. No public placement selector or +model-chosen profile is added. + +### 6. Retain committed workspaces by default + +One persistent PVC per logical environment is retained across executor and mecak8s/controller restarts. +Session deletion does not delete a workspace. The lifecycle default is: removal of one reference retains +the environment; removal of the last reference retains it; committed data retires only through explicit +authorized retirement; and a retiring environment rejects new bindings and successors. An unreadable +store/reference is not an orphan and is retained. A pending allocation whose session association might +have committed cannot be collected merely after TTL: publication absence must first be conclusively +reconciled. Explicit retirement of a live/shared environment is refused while any live reference exists +unless a later reviewed retire/quiesce policy specifies otherwise. + +Default chart uninstall retains runtime CRs/PVCs. While executors survive, workload isolation, +security high-water/key history, and the profile-capacity ledger must survive too. Their lifetime follows +retained allocations, not the provider Deployment. A keep annotation or no-deletion-hook assertion is not +behavioral proof. Uninstall must not silently remove NetworkPolicies or reset authority on reinstall. + +The supported reinstall is the same release and namespace, with explicit validation/reuse of that +release's retained resources and unchanged allocation/profile identity. Preserve ledger contents and +resource ownership; reject foreign/ambiguous resources and missing authority history while allocations +survive. Compatible upgrades also preserve state. Fresh install may initialize empty ledgers only in +the absence of retained allocations. No destructive hook, forced adoption/cleanup, authority-generation +reset, or arbitrary rollback is supported. Manual CRD upgrade precedes explicit compatible schema-v2 +migration; CRD/namespace deletion with live resources is destructive and unsupported. + +The [Helm lifecycle contract](../acceptance/native-kubernetes-execution.md#helm-lifecycle-amendment) +requires actual install/upgrade/uninstall/reinstall qualification: persistent data, exact reattachment, +negative enforcing-CNI probes during provider absence, rejection of stale authority after restart, and +capacity accounting. Add retained ledgers/isolation/configuration to the resource and rehydration +inventories in `docs/adr/0027-cloud-native.md` during implementation. PVC persistence does not claim +node-disaster recovery; kind host-local storage proves restart survival only. + +### 7. Qualify deterministically, then prove native-provider live coding + +The focused mock-only kind tasks run on the supported generic toolchain, including CI, with a unique +owned cluster, scratch kubeconfig, explicit context on every command, bounded sanitized artifacts, and +ownership-scoped cleanup. Default kind does not prove NetworkPolicy enforcement; the production profile +uses Calico. The acceptance plan retains historical production success at `deaf1c3d1` and the +later reported run `35650128625` at `f7919f6c` which passed 10/11 scenarios but failed rotation +at the final-authority read. Offline tests at repair commit `96e16e926` reproduce safe +pre-dispatch replica lag and persistent mixed-material ledger poisoning, and prove immutable-name +publication across both projection orders. They do not identify which mechanism caused that +historical failure. Historical local keyring failures and qualification failures are no longer +current blockers: the [candidate completion receipt](../acceptance/native-kubernetes-execution.md#candidate-completion-receipt) +records all 11 production Kind+Calico scenarios passing at runtime SHA +`2020e59934369e8f771059d0c173fa4cbf964c7e`, including scope/retry/Helm lifecycle qualification. +Draft full-race CI remained skipped; earlier local full-race and latest targeted-race results +are distinguished in that receipt, not represented as a draft full-race CI execution. + +The checked human live decision requires the separate `task e2e:k8s:execution:live` OpenRouter coding +qualification **for this stack**, after deterministic qualification. It is not an always-on live CI +dependency. Prove model-driven tools, independently verified persistent artifacts, and a successful +bound test command; generic mecak8s live CI is not native-provider evidence. Manual exact-head run +[35700303737](https://github.com/stacklok/mecatl/actions/runs/35700303737) supplied that evidence +after production qualification: real OpenRouter coding, independent exact-ref typed-gRPC +reattachment/run acquisition and file/test verification, followed by verified cleanup. This is +qualification of the tested runtime SHA, not approval or a claim that documentation-only descendant +heads executed live. The generic compaction live-lane correction in merged #1728 remains separate. + +Use verified nonsecret endpoint/protocol/model configuration in an authenticated operator-controlled +cluster. A trusted runtime-only loader consumes the approved protected file channel into a narrowly +scoped HARNESS-only Secret before tools/child processes can inherit it. Planning and mock tests never +access it. Neither agent nor tool reads the credential file or Secret; failures are redacted, and no +credential enters model/tool output, argv, Helm values, disk manifests, git, logs, artifacts, or the +execution workload. Restore mock configuration and delete only the run-owned Secret by recorded UID. +Token/run limits do not establish a hard dollar cap. Live coding evidence supplements, never replaces, +the deterministic correctness gate. + +## Consequences + +**Positive.** Mecak8s can optionally bind tools to a persistent Kubernetes namespace without changing +tool schemas, engine APIs, or public placement authority. Disabled deployments remain free of CRDs, +RBAC, informers, clients, and resources. Logical identity survives Pod replacement, and kind can prove +the complete coding path offline. + +**Costs and risks.** Mecatl would own a controller, CRD, private protocol, PVC lifecycle, command +execution/cancellation, and a security-critical fencing system. A persistent volume does not stop a +partitioned writer. Arbitrary Shell makes the executor a hostile-workload boundary. Network policy, +resource quotas, output caps, non-root execution, service-account-token suppression, and credential +separation are required but not sufficient. + +**Draft boundary.** The exact proposed interfaces are authored in the acceptance plan; human API/schema/ +security approval and release maturity remain unresolved. Authorized pre-merge development may proceed +on the existing draft stack, but no review, approval, landed status, or merge is inferred. Completion +requires strict AC trace, full offline/race/build/API/docs/site/demo gates, independent panel review, +final deterministic qualification, and stack-specific native live evidence. The latter two now have +successful candidate receipts, including scope, retry, and Helm scenarios; the plan preserves partial +proof boundaries and outstanding human/panel reviews. Human alone merges. + +## See also + +- [Native Kubernetes execution acceptance plan](../acceptance/native-kubernetes-execution.md) +- [ADR 0048 — mecak8s](0048-mecak8s.md) +- [ADR 0291 — server-owned session placement](0291-server-owned-session-placement.md) diff --git a/docs/adr/README.md b/docs/adr/README.md index e64379d59..94d666ea9 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -87,6 +87,7 @@ Documentation/citation conventions are in [`docs/design/README.md`](../design/RE - [0351 — Mecatl Studio: in-repo web UI behind a BFF over the published SDK](./0351-mecatl-studio-in-repo-web-ui.md) *(proposed; `apps/` workspace, one-origin image, `MECATL_*`/`STUDIO_*` split)* - [0342 - Gate runs on unresolved live context windows](./0342-context-window-admission.md) *(supersedes ADR 0016 only for pre-swap run admission)* - [0346 - Prompt-cache breakpoints are protocol-native, never vendor-keyed](./0346-unified-prompt-cache-dialect.md) *(supersedes ADR 0100's prompt_cache_breakpoint deferral, its root cache_control dialect arm, and its OpenRouter TTL deferral; extends ADR 0334 to OpenRouter)* +- [0354 — Optional native Kubernetes execution provider](./0354-native-kubernetes-execution.md) *(draft; separate service, preserves ADR 0048)* - [0036 — `engine/` is its own Go module (monorepo via `go.work`)](./0036-engine-module.md) - [0037 — Engine public-API stability contract](./0037-engine-stability-contract.md) - [0038 — Event-sourced SessionStore rehydration (the reference fold)](./0038-event-sourced-rehydration.md) *(Decision 3 origin-opacity and no-public-replay clauses proposed to be superseded by 0337)*