[Epic] Adopt kagent Agent Substrate — DevAI-authored agents, sandboxed full lifecycle

[Sam123ben]

Epic — Adopt kagent Agent Substrate as DevAI's sandboxed agent runtime

One-line: let DevAI users author their own agents in the dashboard (adhering to the registry design), then have kagent's Agent Substrate run them as isolated, on-demand, sandboxed Actors — through a full dev → test → deploy → run lifecycle.

Why now

Earlier we decommissioned kagent on the prod cluster because the classic kagent model is one always-warm Deployment per agent (× model variant) — a standing-pod explosion that does not fit our 3-node cluster. The user's instinct was right: "I thought these would be like jobs — pop up when called, not always running." So today every agent runs as an on-demand K8s Job (default JobRunnerStage), kagent is dormant behind the operator kill-switch (DEVAI_KAGENT_ENABLED=false).

The Agent Substrate (solo.io + kubernetes-sigs/agent-sandbox + GKE Agent Sandbox) changes the calculus. Its WorkerPool + Actor model gives "fast cold starts, low resource overhead, and secure gVisor/Firecracker isolation per agent" — agents are lightweight Actors multiplexed inside a shared WorkerPool, not a pod each. That directly answers both reasons we backed off kagent (standing-pod cost → low overhead; "should pop up when called" → fast cold start), and it adds strong per-agent sandboxing — exactly what we need to let users run their own agent code safely.

The end-to-end we're building

USER authors agent in DevAI dashboard (ArtifactEditor, registry schema/layout)
   → validate (schema + lint + handover types)            [Authoring]
   → BUILD (resolve refs) → TEST (eval) → SECURITY gate    [Lifecycle harness]
   → PUBLISH to agentic-registry                            [Registry]
   → agent-sync renders SandboxAgent + ActorTemplate        [Registry→Substrate]
   → kagent Substrate runs it as a gVisor-sandboxed Actor   [Runtime]
      in a shared WorkerPool, on demand, on the user's key  [Dispatch + per-user keys]
   → DevAI dispatches a run to the Actor over A2A           [Dispatch]
   → live status + cost surfaced in the dashboard           [Observability]

Sub-issues (decide + sequence here)

• [Substrate] GO/NO-GO: WorkerPool+Actor resource model on 3-node GKE #70 — GO/NO-GO: Substrate resource model on 3-node GKE (the gate; reconcile with the decommission)
• [Substrate] Install & configure Agent Substrate (tesserix-k8s / ArgoCD) #71 — Install & configure Agent Substrate (tesserix-k8s / ArgoCD; CRDs, WorkerPool, gVisor)
• [Substrate] User-authored agents in DevAI (dashboard → registry schema → publish) #72 — User-authored agents in DevAI adhering to the registry design/layouts (dashboard → schema → publish)
• [Substrate] Registry → Substrate render (SandboxAgent + ActorTemplate; resolve promptRef) #73 — Registry → Substrate render (agentic-registry emits SandboxAgent + ActorTemplate/AgentHarnessSubstrateSpec; agent-sync reconciles)
• [Substrate] Dispatch integration — route runs to a Substrate Actor vs Job #74 — Dispatch integration (route a run to a Substrate Actor vs an ephemeral Job; kill-switch; on-demand semantics)
• [Substrate] Full agent lifecycle gate — dev → test → deploy → run (harness) #75 — Full lifecycle gate: dev → test → deploy → run (the agent-harness: build/test/security before publish+deploy)
• [Substrate] Security & multi-tenant isolation (gVisor, agentgateway, per-user keys) #76 — Security & multi-tenant isolation (gVisor/Firecracker, agentgateway egress/ingress, per-user/tenant + passthrough keys)
• [Substrate] Observability & UI — Actor/lifecycle status in the dashboard #77 — Observability & UI (substrate Actor/lifecycle status in the dashboard; cold-start, per-agent health)
• [Substrate] Cost & autoscaling — KEDA on the WorkerPool, 3-node budget #78 — Cost & autoscaling (KEDA on the WorkerPool, scale-to-zero, 3-node budget)

Guardrails / constraints

• No prod standing-pod explosion. Adoption is conditional on [Substrate] GO/NO-GO: WorkerPool+Actor resource model on 3-node GKE #70 proving the WorkerPool model fits 3 nodes. If it doesn't, we stay on on-demand Jobs.
• GitOps only (tesserix-k8s + ArgoCD); no manual kubectl apply. connect-local first for any cluster op.
• Additive + reversible. Substrate is opt-in behind the existing DEVAI_KAGENT_ENABLED operator switch; Jobs remain the default until Substrate is proven.

References

• Article: https://www.cloudnativedeepdive.com/kagent-agent-substrate-configuration-setup/
• Agent Substrate (solo.io): https://www.solo.io/blog/agent-substrate-powers-kubernetes-agents-with-kagent
• kubernetes-sigs/agent-sandbox: https://github.com/kubernetes-sigs/agent-sandbox · docs https://agent-sandbox.sigs.k8s.io/docs/
• GKE Agent Sandbox: https://cloud.google.com/blog/products/containers-kubernetes/bringing-you-agent-sandbox-on-gke-and-agent-substrate
• DevAI context: docs/agentic/KAGENT-INTEGRATION.md §0/§0a, docs/plans/{agent-harness,kagent-backend,kagent-dynamic-provisioning}/

Sequencing (dependencies)

#70 GO/NO-GO ──gate──> everything below
   └─> #71 Install ──> #73 Registry→Substrate render ──> #74 Dispatch ──> #77 Observability
        #72 Authoring ──┘ (feeds #73)                       │
        #75 Lifecycle gate (build/test/security) ───────────┤ (blocks "deploy to Substrate")
        #76 Security/isolation ──underpins──> #71,#74,#75
        #78 Cost/autoscaling ──depends──> #70,#71

Decision order: start with #70 (does it fit 3 nodes?). If GO → #71 + #72 in parallel → #73 → #74 → #75/#76/#77/#78. If NO-GO → stay on on-demand Jobs and revisit when node budget grows.

[Sam123ben]

Progress — the SOFTWARE chain is done; the LIVE runtime is gated on #70

Shipped + tested this pass (additive, dormant until kagent is re-enabled):

• [Substrate] Registry → Substrate render (SandboxAgent + ActorTemplate; resolve promptRef) #73 promptRef→systemMessage resolution (the real schema mismatch) — agentic-registry.
• [Substrate] Registry → Substrate render (SandboxAgent + ActorTemplate; resolve promptRef) #73 cross-validation API GET /v0/agents/{name}/export/kagent?validate=true → {ok, issues} + RegistryClient.kagent_validate() (DevAI). [[Substrate] User-authored agents in DevAI (dashboard → registry schema → publish) #72/[Substrate] Full agent lifecycle gate — dev → test → deploy → run (harness) #75]
• [Substrate] Registry → Substrate render (SandboxAgent + ActorTemplate; resolve promptRef) #73 SandboxAgent rendering (?workerPool=<pool> → spec.substrate.workerPoolRef).

So DevAI-agent → validated → rendered-as-SandboxAgent → dispatchable works end to end in code, all unit-tested.

The one thing not done — and deliberately so: the LIVE Substrate runtime (install Substrate + stand a WorkerPool + gVisor) on the cluster. That is #70's GO/NO-GO and it must run on the sandbox (connect-local) first — we will NOT push a WorkerPool + gVisor onto the 3-node prod we just protected from pod-explosion without proving it fits. SandboxAgent's premise (Actors multiplexed in a shared pool, not a pod each) is what makes it plausibly fit, but that's a measurement, not an assumption.

Next concrete step: run #70 on the sandbox — stand up Substrate + a 1-replica WorkerPool, apply a rendered SandboxAgent, measure per-Actor overhead + cold-start, decide GO/NO-GO.