Skip to content

feat: draft native Kubernetes execution provider - #1614

Draft
JAORMX wants to merge 12 commits into
plan/native-kubernetes-executionfrom
impl/native-kubernetes-execution
Draft

JAORMX wants to merge 12 commits into
plan/native-kubernetes-executionfrom
impl/native-kubernetes-execution

Conversation

@JAORMX

@JAORMX JAORMX commented Sep 15, 2026

Copy link
Copy Markdown
Contributor

Stage: production completion (draft, stacked)

Plan: #1579
Base: plan/native-kubernetes-execution
The operator requested completion of production readiness. This PR now contains the ownership, lifecycle, rotation, deployment, and qualification implementations, but production qualification is not yet complete. No merge or release publication is implied.

Implemented

  • Separate optional provider service/controller in this monorepo; mecak8s is a storage-free client.
  • Typed gRPC over mandatory mTLS, owner/client/environment/run/epoch/generation-bound grants, bounded admission and sanitized errors.
  • Environment-wide run acquisition/renewal/release, exact lease-loss fencing, post-drain/shutdown teardown, and preserved local/no-FS behavior.
  • Transactional create/successor/delete references, bounded owner-safe intent reconciliation, and conservative retention after unknown store outcomes.
  • Schema-v2 CRD, explicit compatible legacy migration, safe exact-Pod termination proof, healthy replacement, recovery, two-stage retirement and UID-preconditioned retained-PVC deletion.
  • Durable revocation receipts, atomic keyring/TLS/CA/policy reload, complete authority digest/high-water checks, per-request expiry enforcement, and ConfigMap-backed allocation capacity.
  • RuntimeClass/StorageClass preflight, quotas, network restrictions, multi-replica controller safety, bounded health/admission, hardened image/chart release definitions, and operator runbooks.
  • Calico-backed production CI/qualification, direct replica-pinned RPCs, API-server legacy-CRD upgrade fixture, exact negative-network probe outcomes, lifecycle/rotation/quota/holder-loss/Clear-Fork cases, and protected live OpenRouter runner.

Verification and current blocker

  • Full task test, targeted race suites, lint/actions, build, API check, docs/site and offline demo passed during the production repair iterations. The latest migration/network fixture edits passed full tests, lint, docs and build.
  • Two independent review rounds identified concrete issues; repairs include CRD admission/pruning, expired-renewal rejection, termination/retirement replay, exact intent lookup, capacity release recovery, complete security digest/expiry checks, and trusted immutable release provenance.
  • Fresh production Kind reached Kubernetes/Calico readiness. The real pre-upgrade CRD/string-reference migration and failure-artifact tests passed. Allowed workload endpoint access and executor TCP denials were observed; subsequent test fixes still require rerun.
  • Local production qualification is blocked by rootless kernel keyring exhaustion (kernel.keys.maxkeys=200, kernel.keys.maxbytes=20000): runc cannot create later Pod sandboxes. No host setting was changed by the agent. Operator approval for a temporary increase is pending.
  • Latest CI is being monitored; the previous production CI CRD validation failure was fixed and now has full Kubernetes CRD validation coverage.
  • Rotation, remaining lifecycle/quota/holder-loss/Clear-Fork cases must all execute successfully before production readiness is claimed. The full acceptance panel and final 28-AC evidence closure remain gates, not declarations.

Live credential status

A prior typed-gRPC live OpenRouter coding run passed (28,003 input / 783 output tokens, actual Write and exact go test ./... Shell exit 0, independent gRPC artifact/test verification). That is pre-production-hardening evidence, not proof of this new lifecycle implementation. The final protected live smoke is withheld until deterministic production qualification is green.

The supplied OpenRouter credential is read only by the trusted runtime loader, never directly by agent tools/model context. Only the harness receives its run-owned Secret; cleanup is UID-pinned and restores mock mode. No credential was read or staged during the blocked production run.

Scope retained

Foreground Shell and filesystem operations are supported. Git bootstrap, remote project ingestion, background/status/streaming command APIs, schedules, and isolated delegation remain explicitly outside this first production scope. Unobservable-node recovery fails closed without independent fencing proof. Cluster administrators and the selected runtime/CNI remain deployment trust assumptions.

Implement a separately deployed provider/controller, confined executor, optional mecak8s mTLS client, charts, and mock kind qualification. Keep lifecycle, distributed ownership, and acceptance gaps explicit for draft review.

Co-Authored-By: Mecatl <mecatl@users.noreply.github.com>
JAORMX and others added 2 commits September 16, 2026 01:08
Hold the actual model stream until both cleanup guards are checked instead of racing a fast finite mock stream.

Co-Authored-By: Mecatl <mecatl@users.noreply.github.com>
Co-Authored-By: Mecatl <mecatl@users.noreply.github.com>
@JAORMX
JAORMX added this pull request to stack #1618 September 16, 2026 07:51
JAORMX and others added 4 commits September 16, 2026 21:31
Replace the private REST protocol with typed protobuf RPCs over mTLS. Add a real-key kind coding smoke with runtime-only credential loading, UID-pinned Secret cleanup, mock restoration, and independent gRPC artifact and command verification.

Co-Authored-By: Mecatl <mecatl@users.noreply.github.com>
Co-Authored-By: Mecatl <mecatl@users.noreply.github.com>
Replace finite mock output with a context-bound stream so cancellation assertions do not race normal completion.

Co-Authored-By: Mecatl <mecatl@users.noreply.github.com>
Preserve the gRPC execution-provider implementation alongside current command-runner composition and teach canonical self-knowledge about the two optional execution binaries.

Co-Authored-By: Mecatl <mecatl@users.noreply.github.com>
JAORMX and others added 2 commits September 17, 2026 04:21
Add run claims, transactional reference reconciliation, UID-fenced executor replacement and retirement, schema migration, durable grant revocation, reloadable TLS/keyring authority, deployment limits and release definitions. Qualification and independent review remain in progress.

Co-Authored-By: Mecatl <mecatl@users.noreply.github.com>
Close ownership, schema pruning, rotation and termination-proof review findings. Add Calico-backed lifecycle, quota, rotation and failure scenarios, strict release provenance, and operational runbooks. Local production qualification remains blocked by host inotify capacity; CI proof is pending.

Co-Authored-By: Mecatl <mecatl@users.noreply.github.com>
Fix CRD admission, termination and replay edges, bounded renewal, exact intent lookup and release provenance. Add real API-server legacy migration and replica/network proof corrections. Final local production qualification is blocked by kernel keyring quota, not reported as passed.

Co-Authored-By: Mecatl <mecatl@users.noreply.github.com>
JAORMX and others added 2 commits September 17, 2026 10:57
Restore forward-generation trust for new connections, reacquire expired test claims safely, use the post-release epoch, and surface holder start failures without weakening lifecycle checks.

Co-Authored-By: Mecatl <mecatl@users.noreply.github.com>
Restore owner attestation in intent responses, verify presented intermediate chains against current trust, and tie readiness to authoritative state. Qualify phase-scoped claims and recreate endpoint connections after rollouts.

Co-Authored-By: Mecatl <mecatl@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant