feat: schedule hosted agent sandboxes alongside MCP server pods

[ibuildthecloud]

Problem

Hosted agent sandboxes share the MCP namespace, but their pod spec sets no affinity, tolerations, or runtimeClassName — see instance.go, which sets only PriorityClassName, security context, volumes, and pull secrets.

So on a deployment that sets a node pool aside for MCP servers — selected by mcpServerDefaults.affinity and reached through a taint tolerated by mcpServerDefaults.tolerations — MCP server pods land there and sandboxes land anywhere.

That matters more for a sandbox than for a single MCP pod. The pool PVC uses volumeBindingMode: WaitForFirstConsumer and binds to whichever node the first sandbox is placed on, so one misplaced sandbox strands the entire pool off the intended nodes for as long as the volume lives.

Approach

Sandboxes now take those three settings from the same K8sSettings singleton pkg/mcp/kubernetes.go reads, rather than from chart values of their own.

A second set of values would only give the two a way to disagree, and an operator would have to keep them in sync by hand to get the co-location they already asked for once. Reading the singleton per apply — rather than capturing it at startup — is also what lets an admin retarget both at once through the settings API on an install where the values did not come from Helm (SetViaHelm false).

The backend takes this as an Options.Scheduling func(context.Context) Scheduling provider rather than plain fields. pkg/agentbackend/kubernetes currently imports exactly one Obot package, and injecting a storage client instead would pull storage/apis/obot.obot.ai/v1 and system into a package that so far speaks only in agentbackend and raw Kubernetes types. The provider returns a plain struct, so knowledge of where the settings live stays in pkg/services/config.go.

A failed read logs and falls back to unconstrained placement, matching what the MCP backend does on the same error — a settings read that fails should not stop a sandbox from starting.

Changes

• pkg/agentbackend/kubernetes/backend.go — Scheduling struct and the Options.Scheduling provider
• pkg/agentbackend/kubernetes/instance.go — instanceObjects takes a ctx and applies all three to the sandbox PodSpec
• pkg/services/config.go — hostedAgentsScheduling reads the K8sSettings singleton
• chart/values.yaml — comments only; mcpServerDefaults' three scheduling values now govern sandboxes too. No new values or env vars.

Tests

TestInstanceObjectsApplyScheduling asserts all three reach the pod spec; TestInstanceObjectsWithoutSchedulingAreUnconstrained pins the nil-provider behaviour.

Not included

The cleanup Job pod in instance.go still sets no tolerations. It runs in the pool and has to reach the pool volume's node, so on a node pool reached through a taint it will be unschedulable and instance deletion never completes — BackoffLimit: 4 does not help, since the pod is never placed at all. Left out to keep this PR to the sandbox path; happy to fold it in here if preferred.

🤖 Generated with Claude Code

https://claude.ai/code/session_011egKVweRVwSS79iMK1887y

[Copilot]

Pull request overview

This PR updates the hosted agent Kubernetes backend so sandbox pods inherit the same scheduling constraints (affinity, tolerations, runtimeClassName) as MCP server pods, ensuring both workloads co-locate on the intended node pool and preventing unintended PVC binding to the wrong nodes.

Changes:

• Added a Scheduling struct and Options.Scheduling provider to supply per-reconcile pod placement settings.
• Updated sandbox object generation to apply affinity/tolerations/runtimeClassName to the sandbox PodSpec.
• Wired the provider from pkg/services/config.go by reading the K8sSettings singleton; added/updated unit tests and Helm values comments.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
pkg/services/config.go	Adds a scheduling provider that reads K8sSettings per call and injects it into the hosted agent Kubernetes backend.
pkg/services/config_test.go	Updates the hosted agent backend constructor call for the new parameter.
pkg/agentbackend/kubernetes/backend.go	Introduces Scheduling + Options.Scheduling provider hook to supply sandbox pod placement.
pkg/agentbackend/kubernetes/instance.go	Applies scheduling values to the sandbox pod spec and threads ctx into instanceObjects.
pkg/agentbackend/kubernetes/objects_test.go	Adds scheduling propagation tests and updates existing tests for the new ctx parameter.
pkg/agentbackend/kubernetes/subdir_test.go	Updates calls to instanceObjects to pass a context.
pkg/agentbackend/kubernetes/podsecurity_test.go	Updates calls to instanceObjects to pass a context.
chart/values.yaml	Clarifies via comments that MCP scheduling values also govern hosted agent sandbox pods.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.