[Substrate] Security & multi-tenant isolation (gVisor, agentgateway, per-user keys)

[Sam123ben]

Part of the Agent Substrate epic ⬆️. The reason Substrate is worth it for user-authored agents.

Goal

Run untrusted, user-authored agent code with strong isolation and per-user/tenant boundaries, on each user's own LLM key.

Today (grounded)

We have: ambient-mesh AuthorizationPolicies, NetworkPolicies, agentgateway, per-user passthrough ModelConfigs (forward the user's key as the A2A bearer — no shared key), the tenancy model (private-by-default, per-tenant uniqueness). Isolation guardrails in dispatch: HUMAN-principal only, connector-scoped keys, per-principal, never logged.

Tasks

• Sandbox runtime: gVisor/Firecracker per Actor (kernel-level isolation) — confirm + enforce via WorkerPool config (#A/#B).
• Network: route all Actor traffic through agentgateway with fine-grained egress/ingress policies (Substrate's model); deny-by-default egress, allowlist the LLM provider + approved MCP.
• Per-user keys on Substrate: carry the passthrough ModelConfig pattern to SandboxAgent so an Actor runs on the triggering user's key (per-principal, never cross-user).
• Multi-tenant: per-user/team/org isolation (gVisor + namespace/account boundaries); a user's agent can't read another tenant's tools/secrets.
• Secrets: no platform secrets mounted into a user Actor unless explicitly granted; audit grants.
• Threat-model doc: what a malicious user-authored agent can/can't do.

Acceptance

A user-authored agent runs gVisor-sandboxed, on the user's own key, with deny-by-default egress through agentgateway, and provably cannot reach another tenant's data.

Refs: KAGENT-INTEGRATION.md §7 (isolation), tesserix-k8s/.../istio-auth-policies, agentgateway; memory: SCM Per-User, Team/Org Connectors, User Connector Secrets.

— Part of #69

[Sam123ben]

App-side per-user A2A credential isolation is deployed.

• PR fix: restrict kagent passthrough to personal keys #247 merged as 3dc7f27.
• Principal settings overlays now retain source scope for each override.
• kagent A2A passthrough accepts only credentials sourced from USER scope. Tenant, organization, team, and global provider keys remain usable by the normal Job path but cannot cross the A2A bearer boundary.
• Non-object dispatch responses fail closed to the Job path.
• Regression coverage proves the previous shared tenant-key forwarding path is blocked.
• Local verification: 1,393 passed, 1 skipped; Ruff lint/format and strict mypy passed.
• Production freight 4f95b32bf05decd0fa0e5578cfff06e4ef97975b promoted and verified successfully.
• Production verification: 15/15 DevAI Argo applications Synced/Healthy; 11/11 running pods Ready; zero restarts; affected workloads on main-3dc7f27; authenticated agentic status HTTP 200 with registry, AgentGateway, AI Gateway, kagent, and sandbox storage reachable; anonymous status HTTP 401.

This issue remains open. The remaining acceptance criteria are infrastructure/GitOps controls: sandbox RuntimeClass enforcement, default-deny and least-privilege ingress/egress, removal of metadata/public egress, dedicated WorkerPool service account, and non-privileged resource-bounded WorkerPool execution. Those require a separately approved production security-boundary rollout in tesserix-k8s.