diff --git a/docs-site/content/kagent/1.x/agents/human-in-the-loop.md b/docs-site/content/kagent/1.x/agents/human-in-the-loop.md
index 04f95ba8..914c7c8f 100644
--- a/docs-site/content/kagent/1.x/agents/human-in-the-loop.md
+++ b/docs-site/content/kagent/1.x/agents/human-in-the-loop.md
@@ -7,11 +7,11 @@ author: kagent.dev
Some agents only answer questions, and those can run unattended. Others call tools that change a cluster, spend money, or read sensitive information. An agent that takes actions like those should get a person's approval first. The human in the loop (HITL) mechanism lets an agent stop mid-turn, return a question or a pending tool call to a person, and continue once that person answers.
-A working HITL setup needs support from both the agent and the client. The agent must pause, which you configure by setting `requireApproval` on a tool binding in an {{< gloss "AgentTemplate" >}}AgentTemplate{{< /gloss >}}. The client must support the Agent2Agent (A2A) human-in-the-loop extension, so that the client can present the request to a person and return the decision to the agent. A client without the extension still sees the agent stop, but cannot answer. Confirm that the client you plan to use supports the extension before you configure an agent to pause. For the clients that support the extension, see [Client support](#client-support).
+A working HITL setup needs support from both the agent and the client. To make the agent pause, set `requireApproval` on a tool binding in an {{< gloss "AgentTemplate" >}}AgentTemplate{{< /gloss >}}. The client must support the Agent2Agent (A2A) human-in-the-loop extension, so that the client can present the request to a person and return the decision to the agent. Before you configure an agent to pause, confirm that the client that you plan to use supports the extension. For the clients that support the extension, see [Client support](#client-support).
## Pause lifecycle
-A client opens every turn by telling the agent which optional A2A extensions it can handle. A client that requests the HITL extension on that message can answer a pause later in the turn. A client that does not request the extension still receives a reply when the turn finishes. When the agent stops for a person, that client does not receive a payload to answer.
+A client opens every turn by telling the agent which optional A2A extensions the client can handle. A client that requests the HITL extension on that message can answer a pause later in the turn. A client that does not request the extension still receives a reply when the turn finishes. When the agent stops for a person, that client does not receive a payload to answer.
The following diagram traces one turn in which the agent stops.
@@ -21,7 +21,7 @@ flowchart TB
working["Agent works on the turn"]
decision{"Does the agent need
a person?"}
pause["Task state becomes
INPUT_REQUIRED"]
- request["Status message carries a
tool_approval_request
or ask_user_request"]
+ request["Status message contains a
tool_approval_request
or ask_user_request"]
answer["Client sends a response message
on the same task"]
done["Agent finishes the turn"]
@@ -34,7 +34,7 @@ flowchart TB
answer --> working
```
-The agent works until it either finishes the turn or needs a person. When it needs a person, the task moves to `INPUT_REQUIRED` and its status message carries either a `tool_approval_request` or an `ask_user_request`. The client answers by sending a response message on the same task, and the agent resumes the turn where it left off.
+The agent works until it either finishes the turn or needs a person. When it needs a person, the task moves to `INPUT_REQUIRED` and its status message contains either a `tool_approval_request` or an `ask_user_request`. The client answers by sending a response message on the same task, and the agent resumes the turn where it left off.
## Pause kinds
@@ -56,7 +56,7 @@ The `kagent` harness composes this text from the pending calls. The wording depe
| A tool pause with no hints | `Approval is required for tool(s): delete_file` |
| A pause that does not include a tool call | `Human input is required before the agent can continue.` |
-The `codex` and `claude` harnesses compose the text differently. Both pass through the hint that Codex or Claude Code supplied with the request, so the wording comes from the coding agent rather than from kagent.
+The `codex` and `claude` harnesses compose the text differently. Both pass through the hint that Codex or Claude Code supplied with the request, so the wording comes from the coding agent.
### Disable questions
@@ -73,7 +73,7 @@ The value removes only the question tool. A binding that sets `requireApproval`
## Require approval for a tool
-Approval is a property of a tool binding. For example, if you set `requireApproval` on an `mcp` tool binding in the AgentTemplate, the agent stops before each call to a tool that the binding exposes.
+If you set `requireApproval` on an `mcp` tool binding in the AgentTemplate, the agent stops before each call to a tool that the binding exposes.
### Harness support
@@ -86,7 +86,7 @@ The Harness that runs an agent decides which pauses the agent can raise, and how
| `claude` | Supported | Unsupported, because the upstream Claude Code tool that backed it was removed. A `claude` agent pauses for tool approval only. | Rejected, with `RemoteMCPServer "" is bound more than once`. A binding whose tool selection kagent cannot verify against the server's discovered tools exposes the whole server and reports a warning. |
| `byo` | Decided by your image | Decided by your image | Not applicable |
-kagent passes each binding's `requireApproval` setting to a `byo` image, and the image decides whether to honor it. The agent card for a `byo` Agent never declares the HITL extension, so a client cannot learn from the card that the agent pauses, even when the image implements the extension. For what else that card omits, see [Bring your own agent]({{< link path="agents/bring-your-own-agent#compare-byo-and-declarative-agents" >}}).
+kagent passes each binding's `requireApproval` setting to a `byo` image, and the image decides whether to honor it. The agent card for a `byo` Agent never declares the HITL extension. The card omits it even when the image implements the extension, so a client cannot learn from the card that the agent pauses. For what else that card omits, see [Bring your own agent]({{< link path="agents/bring-your-own-agent#compare-byo-and-declarative-agents" >}}).
### Configure a binding
@@ -120,13 +120,13 @@ spec:
Approval belongs to the binding rather than to the tool name, so one server can supply tools that require approval alongside tools that do not. Bind the tools that require approval in a binding that sets `requireApproval`, and bind the tools that do not in a second binding that omits it. The `codex` and `claude` harnesses reject a second binding to the same server. An agent on those harnesses applies one approval setting to every tool that the server offers.
> [!NOTE]
-> kagent 0.x named the tools that needed approval in a `requireApproval` list on the `Agent` resource, which matched tool names across every server. In 1.x, approval is a property of one binding, so the same tool name can pause for one server and run freely for another.
+> kagent 0.x named the tools that needed approval in a `requireApproval` list on the Agent resource, which matched tool names across every server.
-A tool that no binding covers with `requireApproval` runs without a pause. These tools include every built-in tool, for example file access, shell, and web search, and every Model Context Protocol (MCP) tool on a binding that omits the field. The sandbox contains those calls. For more information, see [Sandboxing]({{< link path="about/substrate-runtime/sandboxing" >}}).
+A tool runs without a pause unless a binding that sets `requireApproval` exposes it. These tools include every built-in tool, such as file access, shell, and web search, and every Model Context Protocol (MCP) tool on a binding that omits the field. The Actor's sandbox confines those calls. For more information, see [Sandboxing]({{< link path="about/substrate-runtime/sandboxing" >}}).
## Implement a client that supports human approval
-HITL is an [A2A](https://a2a-protocol.org) message extension, identified by a versioned URI. The following sections follow a client through one pause, from the header that it sends to the response that it returns.
+HITL is an [A2A](https://a2a-protocol.org) message extension, identified by a versioned Uniform Resource Identifier (URI). The following sections follow a client through one pause, from the header that it sends to the response that it returns.
### Request the extension
@@ -136,12 +136,13 @@ A client requests the extension by setting the extension URI as the `A2A-Extensi
A2A-Extensions: https://kagent.dev/extensions/hitl/v1
```
-kagent activates the extension only for calls that request it. kagent then echoes the activated URI back and attaches a structured pause payload only to an activated call. An agent's card declares the extension, so a client can discover HITL support before it sends anything.
+kagent activates the extension only for calls that request it. kagent then echoes the activated URI back and attaches a structured pause payload only to an activated call. A declarative agent's card declares the extension, so a client can discover HITL support before it sends anything.
-A gRPC call names the {{< gloss "Agent" >}}Agent{{< /gloss >}} in the standard A2A `tenant` field, as `/`, and requests the extension through a header. Port-forward the controller's gRPC port first, as in [Install kagent]({{< link path="setup/installation" >}}).
+A gRPC call specifies the {{< gloss "Agent" >}}Agent{{< /gloss >}} in the standard A2A `tenant` field, as `/`, and requests the extension through a header. Every A2A call also sends the `A2A-Version: 1.0` header. Port-forward the controller's gRPC port first, as in [Install kagent]({{< link path="setup/installation" >}}).
```bash
grpcurl -plaintext \
+ -H 'A2A-Version: 1.0' \
-H 'A2A-Extensions: https://kagent.dev/extensions/hitl/v1' \
-d '{
"tenant": "kagent/cluster-operator",
@@ -201,33 +202,60 @@ An `ask_user_request` contains an `id` and a list of `questions`. The response e
}
```
-> [!WARNING]
-> Put the extension URI in the response message's `extensions` array as well as in its `metadata`. The `kagent` harness ignores a `metadata` payload that the array does not declare, and delivers the message as ordinary text. The task resumes and the agent replies, so the mistake is silent. The `codex` and `claude` harnesses reject the message with `tool approval response must declare https://kagent.dev/extensions/hitl/v1`.
-
## Resume a paused task
A paused task waits for the client to send a message that contains the response payload, on the same task and context. kagent rejects a resume attempt on a task that is not waiting.
+The response message must list the extension URI in its `extensions` array as well as in its `metadata`. The `kagent` harness ignores a `metadata` payload that the array does not declare, and delivers the message as ordinary text. The task resumes and the agent replies, so the mistake is silent. The `codex` and `claude` harnesses reject the message with `tool approval response must declare https://kagent.dev/extensions/hitl/v1`.
+
+The following message answers the `ask_user_request` in [Answer a question](#answer-a-question). The `taskId` and `contextId` values come from the paused task, and the `id` in the payload comes from the request.
+
+```bash
+grpcurl -plaintext \
+ -H 'A2A-Version: 1.0' \
+ -H 'A2A-Extensions: https://kagent.dev/extensions/hitl/v1' \
+ -d '{
+ "tenant": "kagent/cluster-operator",
+ "message": {
+ "messageId": "msg-2",
+ "role": "ROLE_USER",
+ "taskId": "",
+ "contextId": "",
+ "extensions": ["https://kagent.dev/extensions/hitl/v1"],
+ "metadata": {
+ "https://kagent.dev/extensions/hitl/v1": {
+ "type": "ask_user_response",
+ "id": "",
+ "answers": [{ "answer": ["us-east-1"] }]
+ }
+ },
+ "parts": [{"text": "us-east-1"}]
+ }
+ }' localhost:8083 lf.a2a.v1.A2AService/SendMessage
+```
+
+The agent continues the turn with the answer. `SendMessage` returns after the turn finishes, with the task in `TASK_STATE_COMPLETED`.
+
While the task waits, kagent suspends the {{< gloss "Actor" >}}Actor{{< /gloss >}} to a durable {{< gloss "Snapshot" >}}snapshot{{< /gloss >}} in the same way as at the end of a turn. The snapshot keeps the durable volume and discards the running process. Each harness records the pending request in that volume and continues the turn when the answer arrives.
-The `claude` harness resumes its deferred tool call. The `codex` harness runs the approved MCP call itself and continues with the result. The {{< gloss "Worker" >}}Worker{{< /gloss >}} is released in the meantime, so a task that waits at `INPUT_REQUIRED` does not consume pool capacity. The wait also survives the loss of the node. For more information, see [Suspend and resume]({{< link path="about/substrate-runtime/suspend-and-resume" >}}).
+The {{< gloss "Worker" >}}Worker{{< /gloss >}} is released in the meantime, so a task that waits at `INPUT_REQUIRED` does not consume pool capacity. The wait also survives the loss of the node. For more information, see [Suspend and resume]({{< link path="about/substrate-runtime/suspend-and-resume" >}}).
-On the `codex` and `claude` harnesses, only one pending request survives the suspension. The `codex` harness cancels any other request along with the interrupted turn. The `claude` harness denies a protected call that arrives in a parallel batch, but a single remaining call can still pause for approval. If an approved call fails after it starts, the harness cannot resume the task. To continue the work, start a new Session.
+The `claude` harness resumes its deferred tool call. The `codex` harness runs the approved MCP call itself and continues with the result. On these two harnesses, only one pending request survives the suspension. The `codex` harness cancels any other request along with the interrupted turn. The `claude` harness denies a protected call that arrives in a parallel batch, but a single remaining call can still pause for approval. If an approved call fails after it starts, the harness cannot resume the task. To continue the work, start a new Session.
The {{< gloss "Transcript" >}}transcript{{< /gloss >}} only grows, so the question and the answer both stay in the task history. A later reader can see what was asked and what a person decided.
## Task states
-An A2A task moves through several states over its life. The following states mean that the task has stopped and is waiting.
+The following A2A task states mean that the task has stopped and is waiting.
| State | Meaning |
| ----- | ------- |
| `INPUT_REQUIRED` | The agent is waiting for a person. A tool approval or a question produces this state. |
-| `AUTH_REQUIRED` | The agent is waiting for credentials. kagent's own harnesses never set this state. The gateway accepts a resume from it, so a `byo` image that produces it works. |
+| `AUTH_REQUIRED` | The agent is waiting for credentials. kagent's own harnesses never set this state. kagent accepts a resume from `AUTH_REQUIRED`, so a `byo` image that sets this state works. |
## Agents bound as tools
-An agent that a parent binds as a tool can raise a pause of its own. The request then includes a `nested` block that identifies the subagent, along with its task and context. A client can therefore tell the person which agent is asking rather than attributing the question to the parent.
+An agent that a parent binds as a tool can raise a pause of its own. The request then includes a `nested` block that identifies the subagent, along with its task and context. A client can therefore tell the person which agent is asking.
## Client support
@@ -235,9 +263,9 @@ The AgentTemplate decides that a turn pauses, and the client that raised the tur
| Client | HITL |
| ------ | ---- |
-| Your own A2A client | Full. Request the extension URI and handle every payload type. |
-| An {{< gloss "MCP" >}}MCP{{< /gloss >}} client that supports tasks | Supported. The `invoke_session` tool returns a task, and input requests surface as MCP elicitations. |
-| The kagent CLI | Not supported. `kagent agent invoke` does not request the extension, so a turn that pauses stays at `INPUT_REQUIRED` and the CLI cannot answer it. The CLI reports `Input required to continue this Session.` and stops there. Send the turn again from a client that requests the extension. |
+| Your own A2A client | Supported. Request the extension URI and handle every payload type. |
+| An {{< gloss "MCP" >}}MCP{{< /gloss >}} client that supports tasks | Supported. The `invoke_agent` tool returns a task, and input requests surface as MCP elicitations. |
+| The kagent CLI | Not supported. `kagent agent invoke` does not request the extension, so a turn that pauses stays at `INPUT_REQUIRED` and the CLI cannot answer it. The CLI reports `Input required to continue this Session.` and stops there. |
## Next steps
diff --git a/docs-site/content/kagent/1.x/examples/a2a-agents.md b/docs-site/content/kagent/1.x/examples/a2a-agents.md
index f06ebe5e..10f10318 100644
--- a/docs-site/content/kagent/1.x/examples/a2a-agents.md
+++ b/docs-site/content/kagent/1.x/examples/a2a-agents.md
@@ -15,6 +15,8 @@ This example uses [grpcurl](https://github.com/fullstorydev/grpcurl) to show the
The controller serves `lf.a2a.v1.A2AService` on port `8083`, alongside its REST API and its MCP endpoint. The kagent CLI reaches the same port and the same service.
+Every A2A call must send the `A2A-Version: 1.0` header, as gRPC metadata or as an HTTP header. Without it, the controller rejects the call with `this version is not supported: requested "0.3", served "1.0"`. The A2A client libraries set the header automatically, but a hand-assembled request such as a `grpcurl` call must set it.
+
A gRPC caller identifies the Agent in the standard A2A `tenant` field, as `/`, and the controller routes the call to that Agent. The same controller serves A2A over HTTP JSON-RPC, where the URL path selects the Agent instead:
| Transport | How a caller selects the Agent |
@@ -58,6 +60,7 @@ An A2A client typically starts by reading the agent card, which tells it what th
1. Fetch the card for your Agent.
```bash
grpcurl -plaintext \
+ -H 'A2A-Version: 1.0' \
-d '{"tenant":"'"$AGENT"'"}' \
localhost:8083 lf.a2a.v1.A2AService/GetExtendedAgentCard
```
@@ -122,6 +125,7 @@ A message part can also contain a file. Put the file's bytes in the part's `raw`
1. Send a message to the Agent. Omitting a `contextId` starts a new Session.
```bash
grpcurl -plaintext \
+ -H 'A2A-Version: 1.0' \
-d '{"tenant":"'"$AGENT"'","message":{"messageId":"'"$(uuidgen)"'","role":"ROLE_USER","parts":[{"text":"What is 7 times 6? Answer with just the number."}]}}' \
localhost:8083 lf.a2a.v1.A2AService/SendMessage
```
@@ -165,6 +169,7 @@ A message part can also contain a file. Put the file's bytes in the part's `raw`
1. Send a message on the streaming method.
```bash
grpcurl -plaintext \
+ -H 'A2A-Version: 1.0' \
-d '{"tenant":"'"$AGENT"'","message":{"messageId":"'"$(uuidgen)"'","contextId":"'"$SESSION_ID"'","role":"ROLE_USER","parts":[{"text":"Count from 1 to 3."}]}}' \
localhost:8083 lf.a2a.v1.A2AService/SendStreamingMessage
```
@@ -186,6 +191,7 @@ A task outlives the call that created it, so a caller that lost its connection c
1. Read the task by ID.
```bash
grpcurl -plaintext \
+ -H 'A2A-Version: 1.0' \
-d '{"tenant":"'"$AGENT"'","id":"'"$TASK_ID"'"}' \
localhost:8083 lf.a2a.v1.A2AService/GetTask
```
@@ -353,6 +359,7 @@ Answering a pause needs the human-in-the-loop extension, which a caller requests
1. Send a message that requests the extension.
```bash
grpcurl -plaintext \
+ -H 'A2A-Version: 1.0' \
-H "A2A-Extensions: https://kagent.dev/extensions/hitl/v1" \
-d '{"tenant":"'"$AGENT"'","message":{"messageId":"'"$(uuidgen)"'","role":"ROLE_USER","extensions":["https://kagent.dev/extensions/hitl/v1"],"parts":[{"text":"Should I increase the replica count?"}]}}' \
localhost:8083 lf.a2a.v1.A2AService/SendMessage
@@ -397,6 +404,7 @@ Answering a pause needs the human-in-the-loop extension, which a caller requests
4. Answer on the same task. The response goes in the same metadata key, references the request `id` that it answers, and sets `taskId` so that it answers the paused task rather than starting a new turn.
```bash
grpcurl -plaintext \
+ -H 'A2A-Version: 1.0' \
-H "A2A-Extensions: https://kagent.dev/extensions/hitl/v1" \
-d '{"tenant":"'"$AGENT"'","message":{"messageId":"'"$(uuidgen)"'","taskId":"'"$PAUSED_TASK_ID"'","contextId":"'"$PAUSED_SESSION_ID"'","role":"ROLE_USER","extensions":["https://kagent.dev/extensions/hitl/v1"],"parts":[{"text":"staging"}],"metadata":{"https://kagent.dev/extensions/hitl/v1":{"type":"ask_user_response","id":"'"$REQUEST_ID"'","answers":[{"answer":["staging"]}]}}}}' \
localhost:8083 lf.a2a.v1.A2AService/SendMessage
diff --git a/docs-site/content/kagent/1.x/examples/agents-via-mcp.md b/docs-site/content/kagent/1.x/examples/agents-via-mcp.md
index ef28df02..256d4d8f 100644
--- a/docs-site/content/kagent/1.x/examples/agents-via-mcp.md
+++ b/docs-site/content/kagent/1.x/examples/agents-via-mcp.md
@@ -1,22 +1,22 @@
---
title: Use agents from an MCP client
-description: Connect Claude Code, Cursor, or another agent to kagent's MCP server, then discover and invoke your Sessions as tools.
+description: Connect Claude Code, Cursor, or another MCP client to kagent's MCP server, then discover your agents and invoke them as tools.
weight: 20
author: kagent.dev
---
-The kagent controller runs a {{< gloss "Model Context Protocol" >}}Model Context Protocol{{< /gloss >}} (MCP) server that exposes your {{< gloss "Session" >}}Sessions{{< /gloss >}} as tools. Any MCP client can then discover the agents in your cluster and delegate work to them. This mechanism allows one agent to orchestrate another as a sub-agent.
+The kagent controller runs a {{< gloss "Model Context Protocol" >}}Model Context Protocol{{< /gloss >}} (MCP) server that exposes your agents and their {{< gloss "Session" >}}Sessions{{< /gloss >}} as tools. Any MCP client can then discover the agents in your cluster and delegate work to them. This mechanism allows one agent to orchestrate another as a subagent.
This example runs in the opposite direction to [Your first MCP tool]({{< link path="get-started/your-first-mcp-tool" >}}). There, kagent is the MCP client and an external server provides the tools. Here, kagent is the MCP server and your agents are the tools.
-You can do this work from an MCP client such as Claude Code or Cursor, which builds the calls for you, or with raw `curl` requests when you want to see those calls or drive the endpoint without a client.
+An MCP client such as Claude Code or Cursor builds each call. To see each call, or to drive the endpoint without a client, send raw `curl` requests.
## About the kagent MCP server
-The MCP server is part of the controller's HTTP port rather than a separate deployment, so a default installation already serves it at `/mcp` on port `8083`.
+The MCP server runs on the controller's HTTP port, so a default installation already serves it at `/mcp` on port `8083`.
- **Transport**: Clients connect over Streamable HTTP. Standalone Server-Sent Events (SSE) and stdio are unsupported.
-- **Sessions**: The handler is stateless, so each request stands alone and a client does not need to establish a session first.
+- **State**: The handler is stateless. Each request stands alone, and a client does not need to establish an MCP session first.
- **Extensions**: The server advertises the `io.modelcontextprotocol/tasks` extension, which changes how invocations behave. For more information, see [Invoke without waiting](#invoke-without-waiting).
{{< reuse "kagent-docs/snippets/examples/mcp-endpoint-warning.md" >}}
@@ -25,12 +25,14 @@ The MCP server is part of the controller's HTTP port rather than a separate depl
1. [Install kagent]({{< link path="setup/installation" >}}), and confirm that your installation sets `config.server.grpcReflection=true`. Reflection lets a gRPC client discover the controller's methods without a local copy of kagent's protocol buffer definitions.
-2. [Create your first agent]({{< link path="get-started/your-first-agent" >}}), so that you have at least one Session in the `READY` state. The MCP server lists ready Sessions only.
+2. [Create your first agent]({{< link path="get-started/your-first-agent" >}}), so that the cluster has an {{< gloss "Agent" >}}Agent{{< /gloss >}} to invoke.
-3. Install [grpcurl](https://github.com/fullstorydev/grpcurl). Deleting a checkpoint has neither an MCP tool nor a kagent command. Cleaning one up calls `CheckpointService` directly.
+3. Install [grpcurl](https://github.com/fullstorydev/grpcurl) to delete the checkpoint in [Clean up](#clean-up).
## Connect a client
+A client reaches the MCP server through a port-forward to the controller's HTTP port. After the port-forward runs, register the endpoint in your client, or confirm the endpoint with a `curl` request.
+
1. Port-forward the controller's HTTP port, and leave the command running.
```bash
kubectl port-forward -n kagent svc/kagent-controller 8083:8083
@@ -39,7 +41,7 @@ The MCP server is part of the controller's HTTP port rather than a separate depl
2. Add `http://localhost:8083/mcp` to your client.
{{< tabs >}}
{{% tab name="Claude Code" %}}
- Add `--scope project` to limit the entry to the current project rather than your user configuration.
+ To limit the entry to the current project, add `--scope project`.
```bash
claude mcp add --transport http kagent http://localhost:8083/mcp
```
@@ -61,7 +63,7 @@ The MCP server is part of the controller's HTTP port rather than a separate depl
Continue with the steps in [Use agents from Claude Code or Cursor](#use-agents-from-claude-code-or-cursor).
{{% /tab %}}
{{% tab name="curl" %}}
- Nothing needs to be registered, because each request stands alone. Confirm the endpoint before you continue. Streamable HTTP replies are framed as events, so each response arrives on a `data:` line.
+ A `curl` client does not need to register the server. Before you continue, confirm the endpoint. Streamable HTTP replies are framed as events, so each response arrives on a `data:` line.
```bash
curl -s -X POST http://localhost:8083/mcp \
-H 'Content-Type: application/json' \
@@ -80,64 +82,68 @@ The MCP server is part of the controller's HTTP port rather than a separate depl
## Use agents from Claude Code or Cursor
-Claude Code and Cursor read the tool schemas and build each call, so you work in plain language rather than JSON. You discover the agents in your cluster, hold a conversation with one, answer the agent when it stops to ask you something, and pin its state so that a second agent can start from that point.
+Claude Code and Cursor read the tool schemas and build each call, so you work in plain language. You discover the agents in your cluster, start a Session with one, answer the agent when it stops to ask you something, and checkpoint its Session.
### List and invoke an agent
-1. Ask for your conversations. The client calls `list_sessions` and reports one line per Session.
+The `list_agents` tool returns the Agents that the caller can see. The `invoke_agent` tool sends one of them a message and starts a Session.
+
+1. Ask for your agents. The client calls `list_agents` and reports the namespace and name of each Agent. A listed Agent is not necessarily ready, and the caller might lack permission to invoke it.
```console
- > List my kagent sessions.
+ > List my kagent agents.
```
- The tool returns only the Sessions that the calling identity created, and cannot widen that scope. An empty list therefore means that this identity has not created a Session. The kagent command line interface and this endpoint both default to `admin@kagent.dev`, so they see each other's Sessions. When you created the Session as another user with `kagent --user-id `, send a matching `X-User-Id` header from the client.
-
-2. Ask the agent a question by naming the Session you want to use. The client calls `invoke_session` and fills in the arguments from the tool schema.
+2. Ask the agent a question by naming the Agent. The client calls `invoke_agent` with the Agent's namespace and name. The call starts a new Session, and the reply includes the Session ID.
```console
- > Ask kagent session : what is 2+2?
+ > Ask the kagent agent my-first-agent in the kagent namespace: what is 2+2?
Answer with just the number.
```
-3. Ask a follow-up that depends on the previous answer, such as `Multiply that by 10.`, against the same Session. The word "that" points back to the earlier answer. The model can follow the reference because the transcript belongs to the Session rather than to the client.
+ If your client declares the `io.modelcontextprotocol/tasks` extension, the reply is a task ID instead, and the client polls the task in the background. A client declares its own capabilities on each request, so nothing on the server decides whether a call blocks.
- Because a Session _is_ one conversation, a second message to the same `session_id` continues where the first left off. The reply's `context_id` is the Session's own ID. To hold two independent conversations with one {{< gloss "Agent" >}}Agent{{< /gloss >}}, create two Sessions.
+3. Ask a follow-up that depends on the previous answer, such as `Multiply that by 10.`, in the same Session. The client calls `invoke_agent` again with the `session_id` from the first reply. The word "that" points back to the earlier answer. The model can follow the reference because the transcript belongs to the Session rather than to the client.
+
+ Because a Session _is_ one conversation, a second message to the same `session_id` continues where the first left off. The reply's `context_id` is the Session's own ID. To hold two independent conversations with one Agent, invoke the Agent by name twice, which starts two Sessions.
+
+4. To continue a Session that already exists, ask the client to list your Sessions.
+ ```console
+ > List my kagent sessions.
+ ```
-> [!NOTE]
-> Nothing here configures whether an invocation blocks or returns a task to poll, because a client declares its own capabilities on each request. A reply means your client did not declare the `io.modelcontextprotocol/tasks` extension and the call waited for the agent to finish. A task ID means it did, and the client polls in the background so that a long agent run never holds a request open.
+ The `list_sessions` tool returns only the Sessions that the calling identity created, and cannot widen that scope. The kagent command line interface (CLI) and this endpoint both default to `admin@kagent.dev`, so they see each other's Sessions. If you created the Session as another user with `kagent --user-id `, send a matching `X-User-Id` header from the client.
### Answer an agent's question
-When an agent pauses to ask something, kagent returns the question as an MCP elicitation and your client presents its own prompt: the agent's question, and a fixed set of choices where the agent offered them. When you answer it, the agent resumes the turn where it left off. When you refuse it, the agent is told that the person declined, which it can adapt to rather than treating it as an error.
+When an agent pauses to ask something, kagent returns the question as an MCP elicitation, and your client presents its own prompt with the agent's question and any choices that the agent offered. When you answer the prompt, the agent resumes the turn where it left off. When you refuse it, the agent is told that the person declined, and the agent can adapt to the refusal.
-> [!IMPORTANT]
-> Only a client that declares the tasks extension can answer an agent. A blocking `invoke_session` call has nowhere to surface the question, so an agent that pauses leaves that call waiting.
+Only a client that declares the tasks extension can answer an agent. A blocking `invoke_agent` call cannot show the question, so an agent that pauses leaves that call waiting.
-This is the same pause that any other client sees, reached through MCP instead of A2A. For the pause types, the approval model, and what the agent receives, see [Human in the loop]({{< link path="agents/human-in-the-loop" >}}).
+The pause is the same one that any other client sees, reached through MCP instead of Agent2Agent (A2A). For the pause kinds, the approval model, and what the agent receives, see [Human in the loop]({{< link path="agents/human-in-the-loop" >}}).
### Checkpoint a Session
-You can checkpoint a Session before letting an agent try something risky. The client calls the {{< gloss "Checkpoint" >}}checkpoint{{< /gloss >}} tools, which give the client the same checkpoint operations that the [Agent Substrate example]({{< link path="examples/agent-substrate" >}}) performs from the command line.
+Before an agent tries something risky, you can checkpoint its Session. The {{< gloss "Checkpoint" >}}checkpoint{{< /gloss >}} tools give the client the same checkpoint operations that are performed from the command line in the [Agent Substrate example]({{< link path="examples/agent-substrate" >}}).
1. Ask the client to checkpoint the Session.
```console
> Checkpoint kagent session .
```
- The client reports the checkpoint's own ID, the turn that it pinned, and a state of `CHECKPOINT_STATE_READY`. Because the client holds that result in context, you can refer to the checkpoint without repeating its ID.
-
-2. Keep the checkpoint ID if you need to inspect or delete the checkpoint later.
+ The client reports the checkpoint's own ID, the turn that it pinned, and a state of `CHECKPOINT_STATE_READY`. Because the client keeps that result in context, you can refer to the checkpoint without repeating its ID. To inspect or delete the checkpoint later, keep its ID.
You can now safely [clean up these resources](#clean-up).
## Use agents from curl
-With `curl` you build each request yourself, so every field is visible: the tool name, its arguments, and the `_meta` field that decides whether a call blocks or returns a task to poll. You list the agents in your cluster, hold a conversation with one, answer the agent when it pauses, and checkpoint an instance. Every request is a `tools/call` to `/mcp` unless it names a `tasks/` method, and no request needs an `initialize` handshake first.
+With `curl` you build each request yourself, so every field is visible, including the tool name, its arguments, and the `_meta` field that decides whether a call blocks or returns a task to poll. You list the agents in your cluster, start a Session with one, answer the agent when it pauses, and checkpoint the Session. Every request is a `tools/call` to `/mcp` unless it names a `tasks/` method, and no request needs an `initialize` handshake first.
-> [!NOTE]
-> Do not put `io.modelcontextprotocol/protocolVersion` in a request's `_meta`. The server then requires a matching `Mcp-Protocol-Version` header and rejects the call without one. Neither field is necessary for any of these requests.
+Do not put `io.modelcontextprotocol/protocolVersion` in a request's `_meta`. If a request includes it, the server requires a matching `Mcp-Protocol-Version` header and rejects the call without one. None of these requests need either field.
-### List and invoke an agent
+### List and invoke an agent by using curl
-1. List your ready Sessions.
+Each request in this section is a blocking `tools/call`. The reply returns after the agent finishes the turn.
+
+1. List the Agents that you can see. To limit the list to one namespace, add a `namespace` argument.
```bash
curl -s -X POST http://localhost:8083/mcp \
-H 'Content-Type: application/json' \
@@ -147,23 +153,22 @@ With `curl` you build each request yourself, so every field is visible: the tool
"id": 1,
"method": "tools/call",
"params": {
- "name": "list_sessions",
+ "name": "list_agents",
"arguments": {}
}
}'
```
- The reply carries one line per Session as text, plus the same data as structured content. Example output:
- ```console
- 01a068e3-aeb6-7abc-8d6f-5ba9becd3143 (my-first-agent)
- ```
-
-2. Save the ID of the Session that you want to use.
- ```bash
- export SESSION_ID=
+ The reply contains an `agents` array of namespace and name references, plus a `next_page_token` when more Agents remain. A listed Agent is not necessarily ready, and the caller might lack permission to invoke it. Example output:
+ ```json
+ {
+ "agents": [
+ { "namespace": "kagent", "name": "my-first-agent" }
+ ]
+ }
```
-3. Send a message and wait for the reply. A blocking call needs no `_meta`, because the tool's default behavior asks nothing of the client.
+2. Start a Session by sending the Agent a message, and wait for the reply. The `agent` argument takes a reference from the `list_agents` result. A blocking call does not need `_meta`, because by default the tool does not require a client capability.
```bash
curl -s -X POST http://localhost:8083/mcp \
-H 'Content-Type: application/json' \
@@ -173,27 +178,33 @@ With `curl` you build each request yourself, so every field is visible: the tool
"id": 2,
"method": "tools/call",
"params": {
- "name": "invoke_session",
+ "name": "invoke_agent",
"arguments": {
- "session_id": "'"$SESSION_ID"'",
+ "agent": { "namespace": "kagent", "name": "my-first-agent" },
"message": "What is 2+2? Answer with just the number."
}
}
}'
```
- The reply text comes back as the tool's content, with the task identifiers alongside it. Example output:
+ The reply text comes back as the tool's content, with the new Session's ID and the task identifiers alongside it. Example output:
```json
{
"session_id": "01a068e3-aeb6-7abc-8d6f-5ba9becd3143",
"task_id": "01a06d0d-5fcf-7b07-aae3-1f470a8ee157",
- "context_id": "ce5a10b8-7789-4ba7-8395-e60a339de763",
+ "context_id": "01a068e3-aeb6-7abc-8d6f-5ba9becd3143",
"state": "TASK_STATE_COMPLETED",
"text": "4"
}
```
-4. Send the same request again with `Multiply that by 10.` as the `message`. The word "that" points back to the earlier answer. The model can follow the reference because the transcript belongs to the Session. Note that `context_id` is unchanged while `task_id` is new, so the second turn joined the first conversation instead of starting its own.
+3. Save the Session ID and the task ID from the reply. The checkpoint steps use the task ID.
+ ```bash
+ export SESSION_ID=
+ export HEAD_TASK_ID=
+ ```
+
+4. Send `Multiply that by 10.` to the same Session. This time, `session_id` replaces `agent`, because the call takes exactly one of the two. The word "that" points back to the earlier answer. The model can follow the reference because the transcript belongs to the Session. In the reply, `session_id` and `context_id` are unchanged and `task_id` is new, so the second turn continued the same Session. Save the new task ID.
```bash
curl -s -X POST http://localhost:8083/mcp \
-H 'Content-Type: application/json' \
@@ -203,7 +214,7 @@ With `curl` you build each request yourself, so every field is visible: the tool
"id": 2,
"method": "tools/call",
"params": {
- "name": "invoke_session",
+ "name": "invoke_agent",
"arguments": {
"session_id": "'"$SESSION_ID"'",
"message": "Multiply that by 10."
@@ -216,17 +227,20 @@ With `curl` you build each request yourself, so every field is visible: the tool
{
"session_id": "01a068e3-aeb6-7abc-8d6f-5ba9becd3143",
"task_id": "01a06d0d-6864-79f1-a4cb-8547f77638ba",
- "context_id": "ce5a10b8-7789-4ba7-8395-e60a339de763",
+ "context_id": "01a068e3-aeb6-7abc-8d6f-5ba9becd3143",
"state": "TASK_STATE_COMPLETED",
"text": "40"
}
```
+ ```bash
+ export HEAD_TASK_ID=
+ ```
### Invoke without waiting
-Declaring the `io.modelcontextprotocol/tasks` extension changes the same tool's result. Rather than blocking, `invoke_session` returns immediately with a task to poll, which keeps a long agent run from holding a request open. Because the handler is stateless, every request repeats the declaration in its `_meta` rather than establishing it once, and that includes each poll.
+Declaring the `io.modelcontextprotocol/tasks` extension changes the same tool's result. Rather than blocking, `invoke_agent` returns immediately with a task to poll, which keeps a long agent run from holding a request open. Because the handler is stateless, every request, including each poll, repeats the declaration in its `_meta`. The `tasks/cancel` method stops a run that is still working, and the `tasks/update` method answers an agent that is waiting on a person.
-1. Invoke the agent with the extension declared in `params._meta`. Leave the declaration out and the tool blocks instead, as in [List and invoke an agent](#list-and-invoke-an-agent-1).
+1. Invoke the agent with the extension declared in `params._meta`. If you omit the declaration, the tool blocks, as in [List and invoke an agent by using curl](#list-and-invoke-an-agent-by-using-curl).
```bash
curl -s -X POST http://localhost:8083/mcp \
-H 'Content-Type: application/json' \
@@ -236,7 +250,7 @@ Declaring the `io.modelcontextprotocol/tasks` extension changes the same tool's
"id": 3,
"method": "tools/call",
"params": {
- "name": "invoke_session",
+ "name": "invoke_agent",
"arguments": {
"session_id": "'"$SESSION_ID"'",
"message": "Count to three."
@@ -250,7 +264,7 @@ Declaring the `io.modelcontextprotocol/tasks` extension changes the same tool's
}'
```
- The task carries an opaque ID of the form `v1.` that identifies the namespace, the instance, and the A2A task together. Pass it back verbatim rather than parsing it. Example output:
+ The task has an opaque ID of the form `v1.` that identifies the Session and the A2A task. Pass the ID back verbatim. The checkpoint tools do not accept this ID. Example output:
```json
{
"taskId": "v1.eyJuYW1lc3BhY2UiOiJrYWdlbnQiLCJpbnN0YW5jZUlkIjoi...",
@@ -268,7 +282,7 @@ Declaring the `io.modelcontextprotocol/tasks` extension changes the same tool's
export TASK_ID=''
```
-3. Poll the task with `tasks/get`. Carry the same `_meta` declaration on every poll: a request that omits it is rejected with `-32021 tasks capability required but not declared by client` rather than falling back to a blocking read.
+3. Poll the task with `tasks/get`. Include the same `_meta` declaration on every poll. The server rejects a poll that omits it with `-32021 tasks capability required but not declared by client`.
```bash
curl -s -X POST http://localhost:8083/mcp \
-H 'Content-Type: application/json' \
@@ -288,33 +302,38 @@ Declaring the `io.modelcontextprotocol/tasks` extension changes the same tool's
}'
```
-4. If the `status` reports `working` or `cancelled`, send the request from the previous step again, waiting the interval that `pollIntervalMs` suggests between calls. A status of `input_required` means the agent is waiting on a person, which is explored in [Answer an agent's question](#answer-an-agents-question-1).
+4. If the `status` reports `working`, send the `tasks/get` request again after the interval that `pollIntervalMs` suggests. On a `working` task, `statusMessage` contains the raw task record, because the agent has not produced text yet, so do not present it as progress text. A status of `input_required` means the agent is waiting on a person. To answer it, see [Answer an agent's question by using curl](#answer-an-agents-question-by-using-curl). A status of `cancelled` means that a `tasks/cancel` call stopped the run, so stop polling.
-5. When the `status` reads `completed`, stop polling. The `result` field holds the same content that a blocking call would have returned as both `content` and `structuredContent`. Example output:
+5. When the `status` reads `completed`, stop polling. Read `status`, not `resultType`, to decide that a run is over, because a paused task reports `resultType` as `complete` while its `status` is still `input_required`. The `result` field contains the same `content` and `structuredContent` that a blocking call returns. A run that fails also reports `completed`, so read `structuredContent.state` to tell success from failure. Example output:
```json
{
"status": "completed",
- "statusMessage": "1, 2, 3.",
+ "statusMessage": "1, 2, 3",
"resultType": "complete",
"result": {
- "content": [{ "type": "text", "text": "1, 2, 3." }]
+ "content": [{ "type": "text", "text": "1, 2, 3" }],
+ "structuredContent": {
+ "session_id": "01a068e3-aeb6-7abc-8d6f-5ba9becd3143",
+ "task_id": "01a06d0d-6f2a-7c41-9b5e-31d0c2a7e8f4",
+ "context_id": "01a068e3-aeb6-7abc-8d6f-5ba9becd3143",
+ "state": "TASK_STATE_COMPLETED",
+ "text": "1, 2, 3"
+ }
}
}
```
-Two more methods complete the set. `tasks/cancel` stops a run that is still working, and `tasks/update` answers an agent that is waiting on a person.
-
-> [!WARNING]
-> Read `status` rather than `resultType` to decide that a run is over. A paused task reports `resultType` as `complete` while its `status` is still `input_required`. `resultType` alone therefore does not mean an answer is waiting.
+6. Save the `structuredContent.task_id` value. The checkpoint steps use this task ID, not the `v1.` task ID.
+ ```bash
+ export HEAD_TASK_ID=
+ ```
-> [!NOTE]
-> `statusMessage` changes meaning with the status. On a `working` task it holds the raw task record rather than a readable sentence, because the agent has not produced any text yet. Do not present it to a person as progress text. On an `input_required` task it holds the agent's question, and on a `completed` task it holds the reply.
-### Answer an agent's question
+### Answer an agent's question by using curl
-When an agent pauses to ask something, the task's status becomes `input_required` and `tasks/get` returns an `inputRequests` object describing what the agent needs. kagent builds that as an MCP elicitation, and answering it is a `tasks/update` call.
+When an agent pauses to ask something, the task's status becomes `input_required` and `tasks/get` returns an `inputRequests` object describing what the agent needs. kagent builds the request as an MCP elicitation. To answer it, send a `tasks/update` call.
-1. Poll the paused task and read `inputRequests`. It is keyed by request ID, and each entry is an `elicitation/create` call whose `requestedSchema` is the schema kagent built for the pause. Note the key, because answering needs it. Example output:
+1. Poll the paused task and read `inputRequests`. `inputRequests` is keyed by request ID, and each entry is an `elicitation/create` call whose `requestedSchema` is the schema that kagent built for the pause. Note the key, because the `tasks/update` call requires it. Example output:
```json
{
"status": "input_required",
@@ -352,7 +371,7 @@ When an agent pauses to ask something, the task's status becomes `input_required
export INPUT_REQUEST_ID=
```
-3. Send the answer with `tasks/update`, keying `inputResponses` by the same request ID. An elicitation result of `accept` sends the answers on, while `decline` and `cancel` tell the agent that the person refused. An agent can adapt to a refusal rather than treating it as an error.
+3. Send the answer with `tasks/update`, keying `inputResponses` by the same request ID. An `accept` result sends the answers to the agent. A `decline` or `cancel` result tells the agent that the person refused. If the response key does not match the `inputRequests` key, the server rejects the response with an `Invalid params` error (`-32602`), and the task stays `input_required`. Read the key from `tasks/get`, and do not reuse a task ID or a checkpoint ID.
```bash
curl -s -X POST http://localhost:8083/mcp \
-H 'Content-Type: application/json' \
@@ -383,7 +402,7 @@ When an agent pauses to ask something, the task's status becomes `input_required
{ "resultType": "complete" }
```
-4. Poll the task again to collect the resumed turn. The agent picks up where it paused, so the status returns to `working` and then `completed` with the answer your response produced. Example output:
+4. Poll the task again to collect the resumed turn. The agent picks up where it paused, so the status returns to `working` and then `completed` with the answer that your response produced. Example output:
```json
{
"status": "completed",
@@ -395,16 +414,13 @@ When an agent pauses to ask something, the task's status becomes `input_required
}
```
-> [!NOTE]
-> A response whose key does not match the `inputRequests` key is rejected with an `Invalid params` error (`-32602`) naming the key that the task expects, and the task stays `input_required`. A value that is not an elicitation result is rejected the same way. Read the key from `tasks/get` rather than reusing a task ID or a checkpoint ID.
-
-For the pause types, the approval model, and what the agent receives, see [Human in the loop]({{< link path="agents/human-in-the-loop" >}}).
+For the pause kinds, the approval model, and what the agent receives, see [Human in the loop]({{< link path="agents/human-in-the-loop" >}}).
-### Checkpoint a Session
+### Checkpoint a Session by using curl
-The checkpoint tools pin a Session's state. These tools do not need the tasks extension, so they do not carry `_meta`.
+The checkpoint tools pin a Session's state. These tools do not need the tasks extension, so their requests omit `_meta`.
-1. Create a checkpoint. `expected_head_task_id` names the finished turn to pin. When the conversation has advanced past that turn, the call fails rather than pinning the wrong one. Use the `$TASK_ID` that an earlier invocation returned. `request_id` is an idempotency key. Repeating the call with the same value returns the checkpoint that the earlier call created rather than pinning a second one.
+1. Create a checkpoint of the latest finished turn. `expected_head_task_id` identifies the turn to pin. If the Session has advanced past that turn, the call fails. Use the `$HEAD_TASK_ID` from the latest turn. The `v1.` task ID that `tasks/get` uses fails with `conversation advanced beyond the expected task`. `request_id` is an idempotency key, so a repeated call with the same value returns the existing checkpoint.
```bash
curl -s -X POST http://localhost:8083/mcp \
-H 'Content-Type: application/json' \
@@ -417,14 +433,14 @@ The checkpoint tools pin a Session's state. These tools do not need the tasks ex
"name": "create_session_checkpoint",
"arguments": {
"session_id": "'"$SESSION_ID"'",
- "expected_head_task_id": "'"$TASK_ID"'",
+ "expected_head_task_id": "'"$HEAD_TASK_ID"'",
"request_id": "my-first-checkpoint"
}
}
}'
```
- The result identifies the checkpoint and the turn it pinned. Example output:
+ The result identifies the checkpoint and the turn that it pinned. Example output:
```json
{
"checkpoint": {
@@ -485,16 +501,44 @@ The checkpoint tools pin a Session's state. These tools do not need the tasks ex
}
```
-To start a second Session from a checkpoint, call `fork_session` with the checkpoint ID. For what a fork replays and where it diverges, see [Suspend and resume]({{< link path="about/substrate-runtime/suspend-and-resume#checkpoints" >}}) and the [Agent Substrate example]({{< link path="examples/agent-substrate" >}}).
+4. Start a second Session from the checkpoint. `request_id` is an idempotency key, as in the checkpoint request. For what a fork replays and where it diverges, see [Suspend and resume]({{< link path="about/substrate-runtime/suspend-and-resume#checkpoints" >}}).
+ ```bash
+ curl -s -X POST http://localhost:8083/mcp \
+ -H 'Content-Type: application/json' \
+ -H 'Accept: application/json, text/event-stream' \
+ -d '{
+ "jsonrpc": "2.0",
+ "id": 8,
+ "method": "tools/call",
+ "params": {
+ "name": "fork_session",
+ "arguments": {
+ "checkpoint_id": "'"$CHECKPOINT_ID"'",
+ "request_id": "my-first-fork"
+ }
+ }
+ }'
+ ```
+
+ The result identifies the new Session, which is ready for a message. Example output:
+ ```json
+ {
+ "session": {
+ "agent": "my-first-agent",
+ "id": "01a06d1a-2b7c-7d40-8e13-5c9f04b1a6d2",
+ "state": "RUNTIME_STATE_READY"
+ }
+ }
+ ```
You can now safely [clean up these resources](#clean-up).
## Clean up
-1. Delete the checkpoint that you created. `CheckpointService` listens on the same port that serves MCP, so the port-forward from [Connect a client](#connect-a-client) already reaches the service.
+1. Delete the checkpoint that you created. If you used Claude Code or Cursor, set `CHECKPOINT_ID` to the ID that the client reported. `CheckpointService` listens on the same port that serves MCP, so the port-forward from [Connect a client](#connect-a-client) already reaches the service.
```bash
grpcurl -plaintext \
- -d '{"checkpointId":""}' \
+ -d '{"checkpointId":"'"$CHECKPOINT_ID"'"}' \
localhost:8083 kagent.api.v1.CheckpointService/DeleteCheckpoint
```
@@ -504,15 +548,16 @@ You can now safely [clean up these resources](#clean-up).
## MCP tool reference
-The server exposes five Session tools. Two cover discovery and conversation. Three expose the {{< gloss "Checkpoint" >}}checkpoint{{< /gloss >}} operations, so a client can pin an agent's state as well as talk to it. No tool takes a namespace. A Session is addressed by its own UUID. No tool deletes an object either. Deleting a Session takes the `kagent agent session delete` command, and deleting a checkpoint takes a direct call to the gRPC API, because the CLI has no checkpoint command. The server also exposes a set of [standalone sandbox]({{< link path="about/substrate-runtime/standalone-sandboxes" >}}) tools, which this example does not cover.
+The server exposes the following agent and Session tools. Some cover discovery and conversation, and the rest expose the {{< gloss "Checkpoint" >}}checkpoint{{< /gloss >}} operations, so a client can pin an agent's state as well as talk to it. An Agent is addressed by its namespace and name, and a Session by its own universally unique identifier (UUID). None of these tools deletes a Session or a checkpoint. To delete a Session, run `kagent agent session delete`. To delete a checkpoint, call the gRPC API directly, because the CLI has no checkpoint command. The server also exposes a set of [standalone sandbox]({{< link path="about/substrate-runtime/standalone-sandboxes" >}}) tools, which this example does not cover.
| Tool | Required arguments | What it does |
| ---- | ------------------ | ------------ |
+| `list_agents` | None | Lists the Agents that the caller can see. Takes `namespace` to filter, and `page_size` and `page_token`. |
| `list_sessions` | None | Lists the ready Sessions that the caller created. Takes `page_size` and `page_token`. |
-| `invoke_session` | `session_id`, `message` | Sends a message and returns the agent's reply. Takes `message_id` for idempotency. |
-| `create_session_checkpoint` | `session_id`, `expected_head_task_id` | Pins the conversation at a turn boundary. The call fails if the conversation has advanced past that task. Takes `request_id` for idempotency. |
+| `invoke_agent` | `message`, and exactly one of `agent` or `session_id` | Starts a new Session with `agent`, or continues one with `session_id`, and returns the agent's reply with the Session and task IDs. Takes `message_id` for idempotency. |
+| `create_session_checkpoint` | `session_id`, `expected_head_task_id` | Pins the Session at a turn boundary. The call fails if the Session has advanced past that task. Takes `request_id` for idempotency. |
| `list_session_checkpoints` | `session_id` | Lists that Session's checkpoints. Takes `page_size` and `page_token`. |
-| `fork_session` | `checkpoint_id` | Starts a second Session from a checkpoint, replaying the conversation through the pinned turn. Takes `request_id` for idempotency. |
+| `fork_session` | `checkpoint_id` | Starts a second Session from a checkpoint, replaying the Session's transcript through the pinned turn. Takes `request_id` for idempotency. |
## Next steps
diff --git a/docs-site/content/kagent/1.x/operations/multiple-namespaces.md b/docs-site/content/kagent/1.x/operations/multiple-namespaces.md
index d0d4b04c..df971ca8 100644
--- a/docs-site/content/kagent/1.x/operations/multiple-namespaces.md
+++ b/docs-site/content/kagent/1.x/operations/multiple-namespaces.md
@@ -5,42 +5,47 @@ weight: 25
author: kagent.dev
---
-By default, {{< reuse "kagent-docs/snippets/name-product.md" >}} runs agents only in the kagent namespace. The controller watches every namespace in the cluster. When several teams run agents, give each team a namespace of its own. This guide prepares a namespace to run agents, limits the controller to the namespaces that you list, and points to the settings that share a model or an MCP server across namespaces.
+When several teams run agents, give each team a namespace of its own. In this guide, you prepare a namespace to run agents, limit the controller to the namespaces that you list, and find the settings that share a model or an MCP server across namespaces.
## Default namespace layout
-A {{< reuse "kagent-docs/snippets/name-product.md" >}} installation places its components and permissions in a few namespaces. The layout depends on how you installed kagent. Each default links to the section of this guide that changes it.
+A {{< reuse "kagent-docs/snippets/name-product.md" >}} installation places its components and permissions in a few namespaces. The layout depends on how you installed kagent. Where a section of this guide changes a default, the default links to that section.
-When you install kagent with `kagent install`, the installation sets up the following defaults:
+Both the Helm and CLI installation methods set up the following defaults:
-* The `kagent` namespace holds the controller, the UI, the tool server, the development database, and the `kagent-default` WorkerPool. It is the only namespace that can run agents. To run agents in another namespace, see [Prepare a namespace to run agents](#prepare-a-namespace-to-run-agents).
-* The `ate-system` namespace holds Agent Substrate. The `podcertificate-controller-system` namespace holds its Pod Certificate controller.
* The controller watches every namespace. The chart grants the controller cluster-scoped role-based access control (RBAC). To narrow both scopes, see [Restrict kagent to specific namespaces](#restrict-kagent-to-specific-namespaces).
-* Agent Substrate has one credential policy. The policy lets agents in `kagent` read Secrets in `kagent` only. Each namespace that you prepare gets a policy of its own.
+* The `ate-system` namespace runs Agent Substrate.
* The `default-model-config` ModelConfig and the `kagent-tool-server` RemoteMCPServer admit references from the `kagent` namespace only. To use them from another namespace, see [Share configuration across namespaces](#share-configuration-across-namespaces).
-
-When you install the charts directly with Helm, the installation sets up the following defaults:
-
-* The `kagent` namespace holds the controller, the UI, and the tool server. The namespace has no WorkerPool, because the kagent chart sets `substrateWorkerPool.create` to `false`. No namespace can run agents, including `kagent`, until you prepare it as described in [Prepare a namespace to run agents](#prepare-a-namespace-to-run-agents).
-* The `ate-system` namespace holds Agent Substrate.
-* The controller watches every namespace. The chart grants the controller cluster-scoped RBAC. To narrow both scopes, see [Restrict kagent to specific namespaces](#restrict-kagent-to-specific-namespaces).
-* Agent Substrate has no credential policy, because the Agent Substrate chart sets `credentialProvider.namespacePolicies` to an empty list. Each namespace that you prepare gets a policy of its own.
-* Agent Substrate has no identity pools. Only `kagent install` creates the certificate authority pools and the JWT authority pool. For more information, see [The Agent Substrate plane]({{< link path="about/substrate-runtime/identity#the-agent-substrate-plane" >}}).
-* The `default-model-config` ModelConfig and the `kagent-tool-server` RemoteMCPServer admit references from the `kagent` namespace only. The chart creates `default-model-config` for the provider that `providers.default` names, which defaults to `openAI`. To use either resource from another namespace, see [Share configuration across namespaces](#share-configuration-across-namespaces).
+* Each namespace that you prepare gets an Agent Substrate credential policy of its own.
+
+The other defaults depend on how you installed kagent:
+
+{{< tabs >}}
+{{% tab name="Helm" %}}
+* The `kagent` namespace contains the controller, the UI, and the tool server. The namespace does not have a WorkerPool, because the kagent chart sets `substrateWorkerPool.create` to `false`. No namespace can run agents, including `kagent`, until you prepare it as described in [Prepare a namespace to run agents](#prepare-a-namespace-to-run-agents).
+* Agent Substrate does not have a credential policy, because the Agent Substrate chart sets `credentialProvider.namespacePolicies` to an empty list.
+* The chart creates `default-model-config` for the provider that `providers.default` sets, which defaults to `openAI`.
+{{% /tab %}}
+{{% tab name="kagent install" %}}
+* The `kagent` namespace contains the controller, the UI, the tool server, the development database, and the `kagent-default` WorkerPool. It is the only namespace that can run agents. To run agents in another namespace, see [Prepare a namespace to run agents](#prepare-a-namespace-to-run-agents).
+* The `podcertificate-controller-system` namespace runs the Agent Substrate Pod Certificate controller.
+* Agent Substrate has one credential policy. The policy lets agents in `kagent` read only the Secrets in `kagent`.
+{{% /tab %}}
+{{< /tabs >}}
## Before you begin
1. [Install kagent]({{< link path="setup/installation" >}}).
-2. Install [yq](https://github.com/mikefarah/yq) version 4. The steps use yq to edit the Agent Substrate Helm values.
+2. Install [yq](https://github.com/mikefarah/yq) version 4 and [jq](https://jqlang.org/download/). The steps use yq to edit the Agent Substrate Helm values, and jq to read a Session ID.
## Prepare a namespace to run agents
-An Agent needs a WorkerPool and an Agent Substrate credential policy in its own namespace. A Harness can select only a WorkerPool in the Harness's namespace. Agent Substrate denies every Secret read from a namespace that has no credential policy. The denial includes reads of that namespace's own Secrets. Without the policy, the Agent becomes ready. However, every model call fails with a `403 Forbidden` error.
+An Agent needs a WorkerPool and an Agent Substrate credential policy in its own namespace. A Harness can select only a WorkerPool in the Harness's namespace. Agent Substrate denies every request to read a Secret from a namespace that does not have a credential policy. The denial includes reads of that namespace's own Secrets. Without the policy, the Agent becomes ready. However, every model call fails with a `403 Forbidden` error.
The following steps prepare the `team-a` namespace.
-1. Create a ServiceAccount and a WorkerPool in the namespace. The pool must set both placement labels to its own name and namespace. For more information about the labels, see [Keep placement scoped to the pool's namespace]({{< link path="operations/tune-agent-substrate#workerpool-placement-labels" >}}).
- ```yaml
+1. Create the namespace, a ServiceAccount, and a WorkerPool. The pool must set both placement labels to its own name and namespace. For more information about the labels, see [Keep placement scoped to the pool's namespace]({{< link path="operations/tune-agent-substrate#workerpool-placement-labels" >}}).
+ ```bash
kubectl create namespace team-a
kubectl create serviceaccount team-a-worker -n team-a
kubectl apply -f - < substrate-current-values.yaml
```
- 2. Write a values file that keeps the other namespaces' policies and sets the policy for this namespace. The command replaces any existing policy for the namespace. The `atespace` value is the namespace of the Agent. The `allowedNamespaces` value lists the namespaces whose Secrets its agents can read.
+ 2. Write a values file that keeps the other namespaces' policies and sets the policy for this namespace. The command replaces any existing policy for the namespace. The `atespace` value is the namespace of the Agent. The `allowedNamespaces` value lists the namespaces whose Secrets the agents in that namespace can read.
```bash
yq '{"credentialProvider": {"namespacePolicies": ((.credentialProvider.namespacePolicies // []) | map(select(.atespace != "team-a"))) + [{"atespace": "team-a", "allowedNamespaces": ["team-a"]}]}}' \
substrate-current-values.yaml > substrate-namespace-policies.yaml
@@ -86,7 +91,7 @@ The following steps prepare the `team-a` namespace.
allowedNamespaces:
- team-a
```
- 3. Upgrade the Agent Substrate release with the values file.
+ 3. Upgrade the Agent Substrate release by using the values file.
```bash
helm upgrade substrate \
{{< reuse "kagent-docs/snippets/helm-substrate.md" >}} \
@@ -96,18 +101,18 @@ The following steps prepare the `team-a` namespace.
-f substrate-namespace-policies.yaml
```
-3. Confirm that the WorkerPool has a ready Worker.
+3. Confirm that the WorkerPool has a ready Worker. The `READY` column for `team-a-pool` lists `1`.
```bash
kubectl get workerpools -n team-a
```
- The `READY` column for the `team-a-pool` lists `1`:
+ Example output:
```console
NAME DESIRED REPLICAS READY SANDBOXCONFIG RESOLVED AGE
team-a-pool 1 1 1 gvisor-default True 6s
```
-4. Create a Harness in the namespace that selects the pool. The `spec.substrate.workerPoolRef` field names the pool. For the other Harness fields, see [Agent harness]({{< link path="agents/agent-harness" >}}).
- ```yaml
+4. Create a Harness in the namespace that selects the pool. The `spec.substrate.workerPoolRef` field references the pool. For the other Harness fields, see [Agent harness]({{< link path="agents/agent-harness" >}}).
+ ```bash
kubectl apply -f - <}}) for OpenAI, and replace `kagent` with `team-a`. To use a ModelConfig from another namespace instead, see [Share configuration across namespaces](#share-configuration-across-namespaces).
+5. Create a ModelConfig and its API key Secret in the namespace. For the commands, see the page for your provider, such as [Create the ModelConfig]({{< link path="setup/model-providers/openai#create-the-modelconfig" >}}) for OpenAI, and replace `kagent` with `team-a`. To use a ModelConfig from another namespace instead, see [Share configuration across namespaces](#share-configuration-across-namespaces).
+
+6. Create an AgentTemplate and an Agent in the namespace, and confirm that a turn reaches the model. A reply confirms the WorkerPool, the credential policy, and the ModelConfig together. Without the credential policy, the Agent still becomes ready, but the turn fails with a `403 Forbidden` error.
+ ```bash
+ kubectl apply -f - <
+ systemPrompt: You are a concise, helpful assistant.
+ ---
+ apiVersion: api.kagent.dev/v1alpha3
+ kind: Agent
+ metadata:
+ name: team-a-agent
+ namespace: team-a
+ spec:
+ template:
+ ref:
+ name: team-a-template
+ harnessRef:
+ name: team-a-harness
+ EOF
+ kubectl wait agent/team-a-agent -n team-a --for=condition=Ready --timeout=180s
+ export SESSION_ID=$(kagent -n team-a agent session create --agent team-a-agent -o json | jq -r '.session.id')
+ kagent -n team-a agent invoke --session $SESSION_ID --task "Reply with the word ready."
+ ```
+ Example output:
+ ```console
+ agent.api.kagent.dev/team-a-agent condition met
+ ready
+ ```
## Restrict kagent to specific namespaces
-By default, the controller watches every namespace. The chart creates ClusterRoles and ClusterRoleBindings for cluster-scoped RBAC. To limit the controller, restrict it to the namespaces that run agents. Then, verify that an unwatched namespace stays unmanaged.
+To limit the controller, restrict it to the namespaces that run agents. Then, verify that an unwatched namespace stays unmanaged.
-### Set the watched namespaces
+`global.watchNamespaces` sets two scopes from one value. The RBAC scope is the set of namespaces where the chart grants the controller RBAC. The watch scope is the set of namespaces that the controller watches. For each listed namespace, the chart replaces the controller's reconciliation and management ClusterRoles and the tool server's ClusterRole with a Role and RoleBinding each. The ClusterRole for service account authentication and the metrics ClusterRoles stay cluster-scoped. The chart appends the install namespace to the list, so the release's own resources remain managed. You can set `global.watchNamespaces` at installation or in a later `helm upgrade` of the kagent release.
-`global.watchNamespaces` sets two scopes from one value: the namespaces where the chart grants the controller RBAC, and the namespaces that the controller watches. For each listed namespace, the chart replaces the controller's getter and writer ClusterRoles and the tool server's ClusterRole with a Role and RoleBinding. The ClusterRole for service account authentication and the metrics ClusterRoles stay cluster-scoped. The chart also adds each listed namespace to the watch. The chart appends the install namespace to the list, so the release's own resources remain managed. Because one value sets both scopes, the two scopes cannot disagree.
+The following steps limit the controller to the `team-a` namespace from [Prepare a namespace to run agents](#prepare-a-namespace-to-run-agents), and use its `team-a-agent` Agent to verify the result.
-Set `global.watchNamespaces` when you install kagent, or later with a `helm upgrade` of the kagent release. Create every listed namespace before upgrading, because the chart creates a Role and RoleBinding in each one.
+1. Confirm that every namespace that you plan to list exists. The chart creates a Role and RoleBinding in each listed namespace, so the upgrade fails with `namespaces "" not found` for a namespace that does not exist.
+ ```bash
+ kubectl get namespace team-a
+ ```
-```bash
-helm upgrade kagent \
- {{< reuse "kagent-docs/snippets/helm-path.md" >}}/{{< reuse "kagent-docs/snippets/helm-kagent.md" >}} \
- --version {{< reuse "kagent-docs/versions/kagent.md" >}} \
- --namespace kagent \
- --reuse-values \
- --set 'global.watchNamespaces={prod-agents,staging-agents}'
-```
+2. Upgrade the kagent release with the list of namespaces. The change restarts the controller with the new watch scope. To return to watching every namespace later, set the value to an empty list, such as `--set-json 'global.watchNamespaces=[]'`.
+ ```bash
+ helm upgrade kagent \
+ {{< reuse "kagent-docs/snippets/helm-path.md" >}}/{{< reuse "kagent-docs/snippets/helm-kagent.md" >}} \
+ --version {{< reuse "kagent-docs/versions/kagent.md" >}} \
+ --namespace kagent \
+ --reuse-values \
+ --set 'global.watchNamespaces={team-a}'
+ ```
-The change restarts the controller with the new watch scope. To return to watching every namespace, set the value to an empty list, such as `--set-json 'global.watchNamespaces=[]'`.
+3. Create a namespace that `global.watchNamespaces` does not list, and create an Agent in it. The Agent's AgentTemplate and Harness do not need to exist, because the controller never reads the Agent.
+ ```bash
+ kubectl create namespace dev-agents
+ kubectl apply -f - <}}) apply to each watched namespace.
-
-1. Create one Agent in each watched namespace. An Agent outside the kagent namespace needs a WorkerPool and an Agent Substrate credential policy in its namespace before it can run. For the steps, see [Prepare a namespace to run agents](#prepare-a-namespace-to-run-agents).
- ```bash
- kubectl apply -n prod-agents -f my-agent.yaml
- kubectl apply -n staging-agents -f my-agent.yaml
- ```
+The controller picks up the grant without a restart. Within a few seconds, the Agent's `ResolvedRefs` condition becomes `True`.
-2. Create one Agent in a namespace that the list does not cover.
- ```bash
- kubectl apply -n dev-agents -f my-agent.yaml
- ```
+### Set the RBAC and watch scopes separately
-3. Confirm that the controller reconciles the watched Agents.
- ```bash
- kubectl get agents -n prod-agents
- kubectl get agents -n staging-agents
- ```
- Example output:
- ```console
- NAME READY AGE
- my-agent True 30s
- NAME READY AGE
- my-agent True 30s
- ```
+The chart also has a separate key for each scope. The `rbac.namespaces` key sets the RBAC scope. The `config.kubernetes.watchNamespaces` key sets the watch scope. If the two scopes must differ, use these keys. Otherwise, use `global.watchNamespaces`. The following rules apply to the separate keys:
-4. Confirm that the Agent in the unwatched namespace stays unmanaged. The controller does not have a watch entry for that namespace. The Agent's `READY` value therefore stays empty, and the conditions command prints nothing.
- ```bash
- kubectl get agents -n dev-agents
- kubectl get agent my-agent -n dev-agents -o jsonpath='{.status.conditions}'
- ```
- Example output:
- ```console
- NAME READY AGE
- my-agent 30s
- ```
+* `rbac.namespaces` takes precedence over `global.watchNamespaces` whenever the key is present. An explicit `rbac.namespaces: []` restores cluster-scoped RBAC. When `rbac.namespaces` is set, it must include the install namespace. Otherwise, the render fails.
+* `config.kubernetes.watchNamespaces` replaces the watch scope with exactly the listed namespaces. The chart does not add the install namespace to the watch scope, so to keep watching the install namespace, list it explicitly.
+* When `rbac.namespaces` is set, a `config.kubernetes.watchNamespaces` entry outside the RBAC scope fails the render, because the controller would loop on `Forbidden` errors at runtime.
+* When `global.watchNamespaces` sets the RBAC scope, the chart adds the `config.kubernetes.watchNamespaces` entries to the RBAC scope, so the render does not fail.
## Share configuration across namespaces
-An Agent in one namespace can use a ModelConfig or a RemoteMCPServer in another namespace when the target resource admits the Agent's namespace. A platform team can then keep one model configuration and its credentials in a shared namespace. For the conditions and the steps, see the following sections:
+If the target resource admits the Agent's namespace, an Agent can use a ModelConfig or a RemoteMCPServer in another namespace. A platform team can then keep one model configuration and its credentials in a shared namespace. For the conditions and the steps, see the following sections:
* [Share a ModelConfig across namespaces]({{< link path="setup/model-providers/about-model-providers#share-a-modelconfig-across-namespaces" >}})
* [Share an MCP server across namespaces]({{< link path="skills-and-mcp/about-tools#share-an-mcp-server-across-namespaces" >}})
@@ -242,7 +294,9 @@ A namespace separates the agents of one team from another team's agents in the f
* An Agent or SandboxTemplate can select only the EgressPolicies in its own namespace. For more information, see [Access control]({{< link path="agents/egress-policies#access-control" >}}).
* A ModelConfig or RemoteMCPServer admits references only from the namespaces that its `allowedNamespaces` field selects.
-A namespace does not separate the callers of the kagent API. The open source authorizer permits every API check. A caller that reaches the controller can start Sessions with an Agent in any watched namespace. The caller can also write AgentTemplates and Harnesses through the API without Kubernetes RBAC. Creator scoping keeps one caller's Sessions out of another caller's list. However, creator scoping is not a security boundary. For how kagent identifies and authorizes callers, see [Identity]({{< link path="about/substrate-runtime/identity" >}}).
+A namespace does not separate the callers of the kagent API. The open source authorizer permits every API check. A caller that reaches the controller can start Sessions with an Agent in any watched namespace. The caller can also write AgentTemplates and Harnesses through the API without Kubernetes RBAC.
+
+Creator scoping keeps one caller's Sessions out of another caller's list. However, creator scoping is not a security boundary. For how kagent identifies and authorizes callers, see [Identity]({{< link path="about/substrate-runtime/identity" >}}).
## Next steps
diff --git a/docs-site/content/kagent/1.x/operations/tune-agent-substrate.md b/docs-site/content/kagent/1.x/operations/tune-agent-substrate.md
index 28a990f6..995a312c 100644
--- a/docs-site/content/kagent/1.x/operations/tune-agent-substrate.md
+++ b/docs-site/content/kagent/1.x/operations/tune-agent-substrate.md
@@ -1,13 +1,13 @@
---
title: Tune Agent Substrate
-description: Learn how to size a WorkerPool, configure snapshot storage, and keep Workers on the sandbox class that kagent requires.
+description: Learn how to size a WorkerPool, configure snapshot storage, and move a pool to another sandbox class.
weight: 20
author: kagent.dev
---
-{{< reuse "kagent-docs/snippets/name-product.md" >}} runs every agent on [Agent Substrate]({{< link path="about/architecture/agent-substrate" >}}). A default installation creates one {{< gloss "WorkerPool" >}}WorkerPool{{< /gloss >}} named `kagent-default`, holding a single Worker on the `gvisor` sandbox class, and it writes snapshots to the object store that the Agent Substrate chart bundles.
+{{< reuse "kagent-docs/snippets/name-product.md" >}} runs every agent on [Agent Substrate]({{< link path="about/architecture/agent-substrate" >}}). An installation with `kagent install` creates one {{< gloss "WorkerPool" >}}WorkerPool{{< /gloss >}} named `kagent-default` with a single Worker on the `gvisor` sandbox class. The installation writes snapshots to the object store that the Agent Substrate chart bundles.
-This single Worker reports capacity for 1000 Actors, so a default pool serves many concurrent {{< gloss "Session" >}}Sessions{{< /gloss >}} before it runs out of slots. Three settings still need tuning before the installation takes real traffic: how many Workers to run, where snapshots are stored durably, and what resource requests the Worker pods set. This page covers each setting, and the steps for moving a pool to the `microvm` sandbox class.
+The single Worker reports capacity for 1000 Actors, so a default pool serves many concurrent {{< gloss "Session" >}}Sessions{{< /gloss >}} before it runs out of slots. Before the installation serves production traffic, tune the number of Workers, the snapshot storage location, and the Worker resource requests. You can also move a pool to the `microvm` sandbox class.
## Before you begin
@@ -15,11 +15,11 @@ This single Worker reports capacity for 1000 Actors, so a default pool serves ma
2. Confirm that you have administrative access to the cluster, because this guide scales WorkerPools and reads cluster-scoped resources.
-3. Install [grpcurl](https://github.com/fullstorydev/grpcurl) and [jq](https://jqlang.org/download/). The kagent controller reports the runtime state through its `SystemService` methods, and no kubectl or kagent command covers them.
+3. Install [grpcurl](https://github.com/fullstorydev/grpcurl) and [jq](https://jqlang.org/download/). The kagent controller reports the runtime state through its `SystemService` methods, and kubectl and the kagent CLI do not expose these methods.
## Inspect the runtime
-Read the current state of the runtime before you change it. The rest of this guide uses three values from this section: the pool name and its `READY` count, the snapshot path that Actors are writing to, and the Worker image that the pool runs.
+Read the current state of the runtime before you change it. The rest of this guide uses values from this section, including the pool name and its `READY` count, the snapshot path that Actors are writing to, and the Worker image that the pool runs.
1. Check the WorkerPool's Worker counts. These counts show whether the pool is producing the Workers that you asked for.
```bash
@@ -67,7 +67,7 @@ Read the current state of the runtime before you change it. The rest of this gui
]
}
```
- An Actor in `ACTOR_STATE_RUNNING` is running a turn on a Worker. A suspended Actor occupies no Worker.
+ An Actor in `ACTOR_STATE_RUNNING` is running a turn on a Worker. A suspended Actor does not occupy a Worker.
4. List the Actors in an {{< gloss "Atespace" >}}atespace{{< /gloss >}} with their latest snapshot. The atespace is the namespace of the Agent. Each Actor runs one {{< gloss "Session" >}}Session{{< /gloss >}} and is named after it.
```bash
@@ -105,27 +105,27 @@ Read the current state of the runtime before you change it. The rest of this gui
}
```
-`ListSubstrateActors` and `ListSubstrateWorkers` return 50 entries per page by default. To request up to 100, set `page.limit`. When a response contains a `page.nextPageToken` value, pass it as `page.pageToken` in the next request, such as `{"atespace":"kagent","page":{"pageToken":""}}`.
+ `ListSubstrateActors` and `ListSubstrateWorkers` return 50 entries per page by default. To request up to 100, set `page.limit`. If a response contains a `page.nextPageToken` value, pass it as `page.pageToken` in the next request, such as `{"atespace":"kagent","page":{"pageToken":""}}`.
## Size a WorkerPool
-The replica count is the only capacity dial on a pool, and setting it well depends on knowing what actually consumes a Worker.
+The replica count is the only capacity setting on a pool. To set it, first determine what consumes Worker capacity.
-A {{< gloss "Worker" >}}Worker{{< /gloss >}} hosts many Actors at the same time, and it holds each one only while a turn is running. kagent suspends an Actor at every turn boundary and frees the Actor's share of its Worker, as described in [Suspend and resume]({{< link path="about/substrate-runtime/suspend-and-resume" >}}). An idle {{< gloss "Session" >}}Session{{< /gloss >}} therefore occupies no Worker capacity at all.
+A {{< gloss "Worker" >}}Worker{{< /gloss >}} runs each Actor only while a turn is running. kagent suspends an Actor at every turn boundary and frees the Actor's share of the Worker, as described in [Suspend and resume]({{< link path="about/substrate-runtime/suspend-and-resume" >}}). An idle {{< gloss "Session" >}}Session{{< /gloss >}} therefore does not use Worker capacity.
-Each Worker reports an Actor capacity when it starts, fixed by the Worker build rather than by a field on the WorkerPool. The gVisor and micro-VM Workers that Agent Substrate ships both report a capacity of 1000 Actors, and no setting raises or lowers it.
+Each Worker reports an Actor capacity when it starts. The Worker build fixes this capacity, and no WorkerPool field changes it. The `gvisor` and `microvm` Worker builds both report a capacity of 1000 Actors.
-Agent Substrate checks this capacity on every placement. It weighs a Worker's spare CPU and memory only when the ActorTemplate states how much CPU and memory one Actor needs. An ActorTemplate that the kagent controller compiles does not state per-Actor CPU or memory, so a pool places a Session's Actor on a Worker's available slots alone, regardless of how loaded the Worker's node already is. Worker resource limits still matter, because they govern how Kubernetes packs Workers onto nodes.
+Agent Substrate checks this capacity on every placement. It weighs a Worker's spare CPU and memory only when the ActorTemplate states how much CPU and memory one Actor needs. An ActorTemplate that the kagent controller compiles does not state per-Actor CPU or memory, so a pool places a Session's Actor only on a Worker's free slots, regardless of how loaded the Worker's node already is. Worker resource limits still matter, because they govern how Kubernetes packs Workers onto nodes.
-Size the pool for the number of turns that run at the same time, not for the number of Agents or Sessions that you have created. A cluster with hundreds of Sessions that are each used occasionally needs far fewer Workers than the session count suggests.
+Size the pool for the number of turns that run at the same time, not for the number of Agents or Sessions that you have created. A cluster with hundreds of Sessions that are each used occasionally needs far fewer Workers than the number of Sessions suggests.
-Scale a pool in place with the standard Kubernetes scale command, because the WorkerPool exposes a scale subresource. Use the pool name that you read in [Inspect the runtime](#inspect-the-runtime).
+Because the WorkerPool exposes a scale subresource, you can scale a pool in place by using `kubectl scale`. Use the pool name that you read in [Inspect the runtime](#inspect-the-runtime).
```bash
kubectl scale workerpool kagent-default -n kagent --replicas=8
```
-To change these settings in your Helm values instead, run an upgrade of your kagent installation with the following settings:
+To set the replica count in your Helm values, set `substrateWorkerPool.replicas` and upgrade your kagent release.
```yaml
substrateWorkerPool:
@@ -134,9 +134,9 @@ substrateWorkerPool:
### Recognize an undersized pool
-Agent Substrate reports pool capacity through its metrics rather than through an error that reaches the caller. Each Worker publishes a state of `idle`, `partial`, `at_capacity`, or `unschedulable`, which together show how much of a pool is in use. A placement that does not find a suitable Worker records a `no_capacity` outcome. Agent Substrate treats this outcome as a normal result rather than a failure. It does not set an error type, and it does not raise an alert on its own. Watch the share of Workers at `at_capacity` and the rate of `no_capacity` placements to tell a pool that is merely busy from one that is too small.
+Agent Substrate reports pool capacity through its metrics. Each Worker publishes a state of `idle`, `partial`, `at_capacity`, or `unschedulable`, which together show how much of a pool is in use. A placement that does not find a suitable Worker records a `no_capacity` outcome. Agent Substrate treats this outcome as a normal result. It does not set an error type, and it does not raise an alert on its own. To tell a pool that is merely busy from one that is too small, watch the share of Workers at `at_capacity` and the rate of `no_capacity` placements.
-None of those signals reach the caller. A turn that cannot get a Worker waits for one, and a turn that waits too long fails with a generic timeout that names neither capacity nor the pool.
+None of those signals reach the caller. A turn that cannot get a Worker waits for one, and a turn that waits too long fails with a generic timeout that does not mention capacity or the pool.
```console
ERROR:
@@ -144,13 +144,13 @@ ERROR:
Message: actor "ai-01a087c0-1d72-775a-9a32-566acac7b685" request timed out
```
-Because short turns release their capacity quickly, a pool under mild pressure absorbs the load and only sheds the requests that wait past the deadline. The result is intermittent timeouts under concurrency rather than a clean failure. Neither the kagent controller log nor the Agent Substrate API log records a capacity message when this happens, so confirm the diagnosis from the `at_capacity` and `no_capacity` signals and from the pool's `READY` count. If the pool is under pressure, raise the replica count.
+Because short turns release their capacity quickly, a pool under mild pressure absorbs the load and only sheds the requests that wait past the deadline. The result is intermittent timeouts under concurrency rather than a clean failure. The kagent controller log and the Agent Substrate API log do not record a capacity message for these timeouts. To confirm the diagnosis, check the `at_capacity` and `no_capacity` signals and the pool's `READY` count. If the pool is under pressure, raise the replica count.
### Set resource requests on Workers
-Generated Worker pods do not set resource requests or limits unless you supply them, and the effect on scheduling is easy to miss.
+Generated Worker pods do not set resource requests or limits unless you supply them.
-A Worker with no limits reports its node's entire allocatable capacity as its own. Every Worker on that node reports the same figures, so Agent Substrate cannot use resource capacity to place Actors, and Kubernetes cannot stop you from packing more Workers onto a node than it can carry. Set requests and limits on any pool that carries production traffic.
+A Worker with no limits reports its node's entire allocatable capacity as its own. Every Worker on that node reports the same figures, so Kubernetes cannot stop more Workers from being scheduled onto a node than the node can run. Set requests and limits on any pool that serves production traffic.
```yaml
substrateWorkerPool:
@@ -168,9 +168,13 @@ The same `template` block also accepts `nodeSelector`, `tolerations`, `priorityC
+## Configure Worker identity and placement
+
+Each Worker pool runs as its own ServiceAccount, and kagent places Actors on a pool by two labels. Both settings live in the `substrateWorkerPool` block of the kagent Helm values.
+
### Give Workers their own ServiceAccount
-Worker pods run as a ServiceAccount that the chart creates, named after the pool. On a default install that name is `kagent-default`. The chart binds no Role or ClusterRole to the account. Its purpose is to keep Workers off the namespace's `default` ServiceAccount, and to give you a Worker identity that you can attach your own policy to, such as an IAM role or an admission rule.
+Worker pods run as a ServiceAccount that the chart creates, named after the pool. On a default installation that name is `kagent-default`. The chart does not bind a Role or ClusterRole to the account. Its purpose is to keep Workers off the namespace's `default` ServiceAccount, and to give you a Worker identity that you can attach your own policy to, such as an IAM role or an admission rule.
To run Workers as an account that you manage outside the chart, stop the chart from creating one and name your own account instead. That account must already exist in the release namespace, because neither the chart nor Kubernetes creates it on demand.
@@ -183,16 +187,14 @@ substrateWorkerPool:
| Field | Description |
| ----- | ----------- |
-| `substrateWorkerPool.serviceAccount.create` | Whether the chart creates the Worker ServiceAccount. Defaults to `true`. An account that the chart creates gets the image pull secrets from `imagePullSecrets` and `global.imagePullSecrets`, with duplicate entries removed. Workers can then pull a `workerImage` from a private or mirrored registry. Set the field to `false` to use an account that you manage yourself. In that case, add the image pull secrets to your account, because the chart does not change an account that it does not create. |
-| `substrateWorkerPool.serviceAccount.name` | The ServiceAccount that Worker pods run as. Omit it to use the pool name. |
-
-The chart writes `serviceAccountName` into the pool's pod template. A `serviceAccountName` that you set under `substrateWorkerPool.template` is overwritten rather than merged. Set the account through `substrateWorkerPool.serviceAccount.name`.
+| `substrateWorkerPool.serviceAccount.create` | Whether the chart creates the Worker ServiceAccount. Defaults to `true`. An account that the chart creates gets the image pull secrets from `imagePullSecrets` and `global.imagePullSecrets`, with duplicate entries removed. Workers can then pull a `workerImage` from a private or mirrored registry. To use an account that you manage yourself, set the field to `false`. In that case, add the image pull secrets to your account, because the chart does not change an account that it does not create. |
+| `substrateWorkerPool.serviceAccount.name` | The ServiceAccount that Worker pods run as. Omit it to use the pool name. The chart writes this value into the pool's pod template and overwrites any `serviceAccountName` under `substrateWorkerPool.template`. |
### Keep placement scoped to the pool's namespace {#workerpool-placement-labels}
-kagent selects a WorkerPool by name and namespace when it compiles an ActorTemplate, and the generated ActorTemplate matches Workers on the `kagent.dev/worker-pool-name` and `kagent.dev/worker-pool-namespace` labels. Agent Substrate matches those labels across every pool rather than checking the WorkerPool's namespace. A pool that sets only the name label can therefore be selected from another namespace.
+kagent selects a WorkerPool by name and namespace when it compiles an ActorTemplate, and the generated ActorTemplate matches Workers on the `kagent.dev/worker-pool-name` and `kagent.dev/worker-pool-namespace` labels. Agent Substrate matches those labels across every pool and does not check the WorkerPool's namespace. A pool that sets only the name label can therefore be selected from another namespace.
-The chart sets both labels from the pool's name and namespace, and applies them after `substrateWorkerPool.labels`. An extra label that reuses either key is overwritten rather than honored.
+The chart sets both labels from the pool's name and namespace, and applies them after `substrateWorkerPool.labels`. An extra label that reuses either key is overwritten. To add your own labels, set `substrateWorkerPool.labels`.
```yaml
substrateWorkerPool:
@@ -219,7 +221,7 @@ spec:
workerImage: "ghcr.io/kagent-dev/substrate/ateworker-gvisor:v{{< reuse "kagent-docs/versions/agent-substrate.md" >}}"
```
-Earlier releases placed Actors with a single `kagent.dev/worker-pool` label, and the chart no longer sets it. A Session or Sandbox that is pinned to a revision from one of those releases still selects that key. Keep the old label on the pool until you recreate them.
+Earlier releases placed Actors with a single `kagent.dev/worker-pool` label, and the chart no longer sets it. A Session or Sandbox that is pinned to a revision from one of those releases still selects that key. Keep the old label on the pool until you recreate those Sessions and Sandboxes.
```yaml
substrateWorkerPool:
@@ -229,64 +231,96 @@ substrateWorkerPool:
## Configure snapshot storage
-Agent Substrate writes every suspended Actor to an object store, so the bucket stores the Session state that a resume restores. Two places decide where a snapshot lands:
+Agent Substrate writes every suspended Actor to an object store, so the bucket stores the Session state that a resume restores. The following settings decide where a snapshot lands.
-- In the Agent Substrate Helm values, `atelet.storageBackend` selects the storage client and `rustfs.enabled` decides whether the chart supplies credentials for it. These two values cover both the components that write snapshots and the ones that manage them.
-- In each {{< gloss "Harness" >}}Harness{{< /gloss >}}, `spec.substrate.snapshotPolicy.location`, names the bucket and prefix that its Actors write under. The field is required, and it must point into a bucket that the selected client can reach.
+- In the Agent Substrate Helm values, `atelet.storageBackend` selects the storage client. `rustfs.enabled` decides whether the chart supplies credentials for that client.
+- In each {{< gloss "Harness" >}}Harness{{< /gloss >}}, `spec.substrate.snapshotPolicy.location` sets the bucket and prefix that the Harness's Actors write under. The field is required, and it must point into a bucket that the selected client can reach.
### The bundled object store
-Because `rustfs.enabled` defaults to `true`, the Agent Substrate chart installs an in-cluster object store and points the `atelet` DaemonSet and the `ate-api-server` Deployment at it. The store claims a 1Gi PersistentVolumeClaim, and it accepts an access key and secret key that the chart sets to published defaults.
+Because `rustfs.enabled` defaults to `true`, the Agent Substrate chart installs an in-cluster object store and configures the `atelet` DaemonSet and the `ate-api-server` Deployment to use it. The store claims a 1Gi PersistentVolumeClaim, and it accepts an access key and secret key that the chart sets to published defaults.
> [!IMPORTANT]
-> The bundled store is not durable. Its credentials are shared defaults that every installation receives. Snapshots store agent Session state, so [move a production installation to your own bucket](#move-snapshots-to-an-external-bucket) before it carries real traffic.
+> The bundled store is not durable, and every installation receives the same credentials for it. Before a production installation serves real traffic, [move its snapshots to your own bucket](#move-snapshots-to-an-external-bucket).
### Move snapshots to an external bucket
-1. Turn the bundled store off and select the backend in your Agent Substrate Helm values. Setting `atelet.storageBackend` to `s3` selects the S3 client, and setting it to `gcs` selects Google Cloud Storage. Agent Substrate matches only the `s3` value. Every other value, an empty one included, selects Google Cloud Storage.
+Replace the bundled store with a bucket in Google Cloud Storage, Amazon S3, or an S3-compatible store. The Agent Substrate backend and every Harness location must point at the same bucket.
- ```yaml
+1. Save an Agent Substrate values file that disables the bundled store and selects the storage backend. The `s3` value selects the S3 client. Any other value, including an empty one, selects Google Cloud Storage.
+ ```bash
+ cat > substrate-storage.yaml < \
+ --from-literal=secret-access-key=
+ cat >> substrate-storage.yaml <}} \
+ --version {{< reuse "kagent-docs/versions/agent-substrate-chart.md" >}} \
+ --namespace ate-system \
+ --reuse-values \
+ -f substrate-storage.yaml
+ ```
- ```yaml
- spec:
- substrate:
- workerPoolRef:
- name: kagent-default
- snapshotPolicy:
- location: s3://ate-snapshots/kagent/
+4. Set `spec.substrate.snapshotPolicy.location` on each Harness to the new bucket. Repeat the command for each Harness.
+ ```bash
+ kubectl patch harness -n kagent --type merge \
+ -p '{"spec":{"substrate":{"snapshotPolicy":{"location":"s3:///kagent/"}}}}'
```
-4. Run a turn on an agent, suspend its Actor, and compare the Actor's `snapshotUri` path from [`ListSubstrateActors`](#inspect-the-runtime) against the `location` that you set. Because the backend and the location are configured separately, a mismatch between them breaks resuming an Actor from its snapshot later.
+5. Confirm that a new snapshot lands under the `location` that you set. Send a message to a Session, so that kagent suspends the Session's Actor when the turn ends. Then read the Actor's latest `snapshotUri`. Because the backend and the location are configured separately, a mismatch between them prevents the Actor from resuming from its snapshot.
+ ```bash
+ grpcurl -plaintext -d '{"atespace":"kagent"}' \
+ localhost:8083 kagent.api.v1.SystemService/ListSubstrateActors \
+ | jq '.actors[] | select(.metadata.name == "session-") | {state: .status.state, snapshotUri: .status.snapshots[-1].durableSnapshot.object.snapshotUri}'
+ ```
+ Example output, with the `snapshotUri` value under the bucket and prefix that the Harness sets:
+ ```json
+ {
+ "state": "ACTOR_STATE_SUSPENDED",
+ "snapshotUri": "s3:///kagent/atespaces/kagent/actors/b0a84d88-9eab-4fb1-97f0-9c93a6fc89a4/snapshots/82d37284-bb19-4351-90fb-b84c6f26595b"
+ }
+ ```
-### How the chart reaches the components
+### Storage variables on the snapshot components
-The `atelet` DaemonSet writes a snapshot, and the `ate-api-server` Deployment manages the snapshot. Each pod reaches the bucket through an `objectstore-plugin` sidecar container, and only that container reads the storage variables. The chart renders `atelet.storageBackend` as the `ATE_STORAGE_BACKEND` variable on both sidecars, so one Helm value keeps the two in agreement. The `atelet` and `ate-api-server` containers fail to start when `ATE_STORAGE_BACKEND` is set on them directly.
+The `atelet` DaemonSet writes a snapshot. The `ate-api-server` Deployment manages the snapshot. Each pod reaches the bucket through an `objectstore-plugin` sidecar container, and only that container reads the storage variables. The chart renders `atelet.storageBackend` as the `ATE_STORAGE_BACKEND` variable on both sidecars, so one Helm value keeps the two in agreement. If `ATE_STORAGE_BACKEND` is set directly on the `atelet` or `ate-api-server` container, that container fails to start.
-Reading these variables off the `objectstore-plugin` container helps when a snapshot fails to write. Edit the Helm values rather than the pod environment because the next Helm upgrade overwrites a direct edit.
+If a snapshot fails to write, read these variables from the `objectstore-plugin` container. Because the next Helm upgrade overwrites a direct edit to the pod environment, change these variables in the Helm values.
| Variable | Purpose |
| -------- | ------- |
@@ -298,13 +332,13 @@ Reading these variables off the `objectstore-plugin` container helps when a snap
### Snapshot layout in the bucket
-Agent Substrate builds a path per Actor under the Harness prefix, so one bucket serves many Harnesses without collision. The `` segment is the Actor's `metadata.uid`, not the Session ID in the Actor's name. Read the `snapshotUri` value off the Actor rather than matching a path to an Actor by eye.
+Agent Substrate builds a path per Actor under the Harness prefix, so one bucket serves many Harnesses without collision. The `` segment is the Actor's `metadata.uid`, not the Session ID in the Actor's name. To find an Actor's snapshot, read the `snapshotUri` value from the Actor.
```console
s3://ate-snapshots/kagent/atespaces//actors//snapshots/
```
-A {{< gloss "Checkpoint" >}}checkpoint{{< /gloss >}} uses a different path. kagent creates an Agent Substrate {{< gloss "Tag" >}}Tag{{< /gloss >}} for each checkpoint, and tagging copies the Actor's snapshot into the Tag's own prefix. A Tag names exactly one snapshot, so its objects sit directly under `tags/` with no `snapshots/` segment. The `` segment is the Tag's `metadata.uid`, so nothing in the path records the `checkpoint-` name that kagent gave the Tag.
+A {{< gloss "Checkpoint" >}}checkpoint{{< /gloss >}} uses a different path. kagent creates an Agent Substrate {{< gloss "Tag" >}}Tag{{< /gloss >}} for each checkpoint, and tagging copies the Actor's snapshot into the Tag's own prefix. A Tag references exactly one snapshot. Its objects sit directly under `tags/` without a `snapshots/` segment. The `` segment is the Tag's `metadata.uid`, and the path does not contain the `checkpoint-` name that kagent gave the Tag.
```console
s3://ate-snapshots/kagent/atespaces//tags/
@@ -312,29 +346,35 @@ s3://ate-snapshots/kagent/atespaces//tags/
A backup or lifecycle policy must cover both prefixes, because one scoped to `actors/` omits every checkpoint and understates how much the bucket stores.
-## Keep pools on the sandbox class they serve
+## Keep pools on the sandbox class that they serve
-kagent configures the sandbox class for you. A default installation runs the `gvisor` class, and every {{< gloss "ActorTemplate" >}}ActorTemplate{{< /gloss >}} that kagent generates inherits the class of the pool that hosts it. Edit this setting only to move a pool to the `microvm` class or to change a pool's Worker build. For what each class isolates, see [Sandboxing]({{< link path="about/substrate-runtime/sandboxing#sandbox-classes" >}}).
+kagent sets the sandbox class on the pool that the chart creates. A default installation runs the `gvisor` class, and every {{< gloss "ActorTemplate" >}}ActorTemplate{{< /gloss >}} that kagent generates inherits the class of the pool that hosts it. Change this setting only if you move a pool to the `microvm` class or change a pool's Worker build. For what each class isolates, see [Sandboxing]({{< link path="about/substrate-runtime/sandboxing#sandbox-classes" >}}).
-One rule governs the change. A pool's class, its Worker image, and the cluster's SandboxConfig must all name the same sandbox class, and Agent Substrate rejects the Actor rather than falling back when they disagree.
+A pool's class, its Worker image, and the cluster's SandboxConfig must all name the same sandbox class. If the three settings disagree, Agent Substrate rejects the Actor.
### Set a pool's sandbox class
-1. Set the class and the matching Worker build in your kagent Helm values.
+Set the pool's class, its Worker image, and its SandboxConfig together.
+
+1. Save a kagent values file that sets the class and the matching Worker build.
{{< tabs >}}
{{% tab name="gvisor" %}}
- ```yaml
+ ```bash
+ cat > kagent-sandbox-class.yaml <}}"
+ EOF
```
{{% /tab %}}
{{% tab name="microvm" %}}
- ```yaml
+ ```bash
+ cat > kagent-sandbox-class.yaml <}}"
+ EOF
```
{{% /tab %}}
{{< /tabs >}}
@@ -344,12 +384,10 @@ One rule governs the change. A pool's class, its Worker image, and the cluster's
| `sandboxClass` | The sandbox runtime family for this pool's Workers, and for every ActorTemplate that kagent generates for the pool. Omitting the field selects `gvisor`. |
| `workerImage` | The Worker build that provides the runtime. `ateworker-gvisor` serves the `gvisor` class and `ateworker-microvm` serves `microvm`. A build that does not match its pool's class never registers a Worker. |
-2. Confirm that the cluster has a SandboxConfig for this class. The chart sets the pool's `configRef` to a fixed SandboxConfig name for each class, `gvisor-default` for a `gvisor` pool and `microvm` for a `microvm` pool. To use another SandboxConfig, set `substrateWorkerPool.sandboxConfigRef`.
-
+2. Confirm that the cluster has a SandboxConfig for this class. The chart sets the pool's `configRef` to a fixed SandboxConfig name for each class, `gvisor-default` for a `gvisor` pool and `microvm` for a `microvm` pool. To use another SandboxConfig, set `substrateWorkerPool.sandboxConfigRef`. A default installation creates only the `gvisor` class, because the Agent Substrate chart applies `sandboxconfig-gvisor.yaml` and kagent installs nothing for `microvm`.
```bash
kubectl get sandboxconfigs
```
-
Example output:
```console
NAME CLASS AGE
@@ -357,10 +395,22 @@ One rule governs the change. A pool's class, its Worker image, and the cluster's
microvm microvm 26h
```
- A default installation creates only the `gvisor` class, because the Agent Substrate chart applies `sandboxconfig-gvisor.yaml` and kagent installs nothing for `microvm`. The `microvm` configuration and its runtime assets are staged out of band by the Agent Substrate installer, such as through `hack/install-microvm-deps.sh --install`.
+ If the output does not list `microvm` and the pool uses that class, add it. Run the Agent Substrate `ate-setup` tool from a checkout of the [Agent Substrate repository](https://github.com/kagent-dev/substrate) at the release that matches your chart. The tool stages the runtime assets in the cluster's object store and applies the `microvm` SandboxConfig.
+ ```bash
+ go run ./cmd/ate-setup deploy sandboxconfig --sandbox-classes=microvm
+ ```
-3. Verify that the pool's Workers came up on the build that you set. A Worker built for another class never registers, so the pool's `READY` count stays below its `DESIRED` count. Confirm that the two counts match.
+3. Upgrade the kagent release by using the values file.
+ ```bash
+ helm upgrade kagent \
+ {{< reuse "kagent-docs/snippets/helm-path.md" >}}/{{< reuse "kagent-docs/snippets/helm-kagent.md" >}} \
+ --version {{< reuse "kagent-docs/versions/kagent.md" >}} \
+ --namespace kagent \
+ --reuse-values \
+ -f kagent-sandbox-class.yaml
+ ```
+4. Confirm that the pool's `READY` count matches its `DESIRED` count.
```bash
kubectl get workerpools -n kagent
```
@@ -369,15 +419,22 @@ One rule governs the change. A pool's class, its Worker image, and the cluster's
Each of the three settings fails differently when it disagrees with the others.
-- A Worker whose pool class does not match an ActorTemplate's class reports `RevisionInvalid` with `unsupported sandbox class ""`, and placement of this Actor fails with `WorkerPoolNotFound`.
-- A missing SandboxConfig leaves the pool's class unresolved. The `Resolved` condition under the WorkerPool's `status.sandboxClasses` is `False`. The reason is `SandboxConfigNotFound` when the SandboxConfig that `configRef` references does not exist, or `DefaultConfigNotFound` when the pool sets no `configRef` and the class has no default.
+- A Worker whose pool class does not match an ActorTemplate's class reports `RevisionInvalid` with `unsupported sandbox class ""`, and placement of the Actor fails with `WorkerPoolNotFound`.
+- A missing SandboxConfig leaves the pool's class unresolved. The `Resolved` condition under the WorkerPool's `status.sandboxClasses` is `False`. The reason is `SandboxConfigNotFound` when the SandboxConfig that `configRef` references does not exist, or `DefaultConfigNotFound` when the pool does not set a `configRef` and the class does not have a default.
- A Worker image built for another class never registers, so the pool reports fewer ready Workers than its replica count.
### Considerations before changing a running pool
-Two constraints apply to a pool that already serves traffic.
+The following constraints apply to a pool that already serves traffic.
+
+- **Snapshots do not move between sandbox classes.** An Actor cannot resume from a snapshot that a different class produced, so changing the class of a pool that hosts live Actors strands their state.
+- **Nodes fetch `gvisor` runtime assets from a public Google Cloud Storage URL**, named in the SandboxConfig, independently of the storage backend that you configured for snapshots. A cluster with restricted egress needs a path to this URL, or a SandboxConfig that points at a copy that you host. The `microvm` assets come from the cluster's object store, where `ate-setup` stages them.
+- **A Worker image change rolls every Worker pod.** The pods take a one-hour termination grace period, so an in-flight turn finishes. A rollout of a busy pool takes as long as its longest running turn.
-- **Snapshots do not move between sandbox classes.** An Actor cannot resume from a snapshot that a different class produced, so changing the class of a pool that holds live Actors strands their state.
-- **Nodes fetch sandbox runtime assets from a public Google Cloud Storage URL**, named in the SandboxConfig, independently of the storage backend that you configured for snapshots. A cluster with restricted egress needs a path to this URL, or a SandboxConfig that points at a copy you host.
+## Next steps
-Replacing `workerImage` rolls the Worker pods through the generated Deployment. The pods take a one-hour termination grace period, so an in-flight turn finishes rather than being cut off, and a rollout of a busy pool takes as long as its longest running turn.
+{{< cards >}}
+ {{< card link=`{{< link path="about/substrate-runtime/suspend-and-resume" >}}` title="Suspend and resume" subtitle="Learn when an Actor releases its Worker and how it resumes from a snapshot." >}}
+ {{< card link=`{{< link path="about/substrate-runtime/sandboxing" >}}` title="Sandboxing" subtitle="Compare what the gvisor and microvm sandbox classes isolate." >}}
+ {{< card link=`{{< link path="operations/multiple-namespaces" >}}` title="Run agents in multiple namespaces" subtitle="Prepare a WorkerPool and a credential policy in each namespace that runs agents." >}}
+{{< /cards >}}
diff --git a/docs-site/content/kagent/1.x/operations/upgrade-from-0x.md b/docs-site/content/kagent/1.x/operations/upgrade-from-0x.md
index 1ae18b32..1beb7d51 100644
--- a/docs-site/content/kagent/1.x/operations/upgrade-from-0x.md
+++ b/docs-site/content/kagent/1.x/operations/upgrade-from-0x.md
@@ -5,7 +5,7 @@ weight: 40
author: kagent.dev
---
-{{< reuse "kagent-docs/snippets/name-product.md" >}} 1.0 does not have an in-place upgrade path from the 0.10.x line. Moving to 1.0 means installing kagent fresh against a new database and recreating your resources on it. When your installation runs 1.0, you can perform regular in-place upgrades for later minor releases. To review upgrade paths and versions, see [Version support]({{< link path="reference/versions#release-support-and-compatibility" >}}). For a summary of what 1.0 changes before you plan the move, see [Release notes]({{< link path="reference/release-notes/1.0#100" >}}).
+{{< reuse "kagent-docs/snippets/name-product.md" >}} 1.0 does not have an in-place upgrade path from the 0.10.x line. Moving to 1.0 means installing kagent fresh against a new database and recreating your resources on it. When your installation runs 1.0, you can upgrade in place to later minor releases. For a summary of what 1.0 changes before you plan the move, see [Release notes]({{< link path="reference/release-notes/1.0#100" >}}).
## In-place upgrade blockers
@@ -15,18 +15,18 @@ Review the following changes between 0.10.x and 1.0 before you continue. Each ch
| ------ | ----------- |
| The database schema is a clean baseline | 1.0 replaces golang-migrate with goose and starts from a single baseline migration. An existing 0.10.x database does not have a migration path to that baseline, and 1.0 refuses to run against a 0.10.x database. |
| The custom resources moved to a new API group | 1.0 serves its resources under `api.kagent.dev`, where 0.10.x serves them under `kagent.dev`. kagent does not convert resources between the groups, so resources stored under the old group are not visible through the new custom resource definitions (CRDs). |
-| The resource model is replaced | 0.10.x's `Agent` is gone. What the 0.10.x `Agent` described is now split between an AgentTemplate and a {{< gloss "Harness" >}}Harness{{< /gloss >}}. A new {{< gloss "Agent" >}}Agent{{< /gloss >}} resource in the `api.kagent.dev` group pairs the two. A conversation is a {{< gloss "Session" >}}Session{{< /gloss >}} that is created against that Agent. The two `Agent` kinds share a name and nothing else. For the model itself, see [Core concepts]({{< link path="about/core-concepts" >}}). |
+| The resource model is replaced | The 0.10.x Agent kind does not exist in 1.0. Its configuration moves to an AgentTemplate and a {{< gloss "Harness" >}}Harness{{< /gloss >}}, which a new {{< gloss "Agent" >}}Agent{{< /gloss >}} kind in the `api.kagent.dev` group pairs. A conversation is a {{< gloss "Session" >}}Session{{< /gloss >}} that is created against that Agent. For the mapping, see [Agents become an AgentTemplate, a Harness, and an Agent](#agents-become-an-agenttemplate-a-harness-and-an-agent). |
-The two releases also cannot run side by side on one cluster. The group rename separates the custom resources, but both charts name their cluster-scoped ClusterRoles after the release, such as `kagent-getter-role`. When both releases use the default `kagent` release name, the 1.0 install collides with the ClusterRoles that 0.10.x owns.
+The two releases also cannot run side by side on one cluster. The group rename separates the custom resources, but both charts create the `kagent-tools-cluster-admin-role` ClusterRole and its ClusterRoleBinding, named after the release. When both releases use the default `kagent` release name, the 1.0 install collides with the ClusterRole and ClusterRoleBinding that 0.10.x owns.
> [!WARNING]
-> Downgrading from 1.0 back to 0.10.x is unsupported. Treat the cutover as one-way, and keep the 0.10.x installation running until you have verified the new one.
+> Downgrading from 1.0 to 0.10.x is not supported. Treat the cutover as one-way, and keep the 0.10.x installation running until you have verified the new one.
## Before you begin
1. Read the [kagent releases](https://github.com/kagent-dev/kagent/releases) for the breaking changes in the version that you are moving to, and [Version support]({{< link path="reference/versions#release-support-and-compatibility" >}}) for what 1.0 does and does not promise.
-2. Decide where you will install 1.0. A second cluster is the safer choice, because it leaves your 0.10.x installation untouched. Installing on the same cluster means uninstalling 0.10.x first, and that step is not reversible.
+2. Decide where to install 1.0. A second cluster is the safer choice, because it leaves your 0.10.x installation untouched. Installing on the same cluster means uninstalling 0.10.x first, and that step is not reversible. Keep the 0.10.x installation running until you verify the new one.
3. Provision an empty PostgreSQL database for 1.0. The database must be a new database rather than a new schema in the old one. If you point 1.0 at a 0.10.x database, the startup fails with the following error.
```console
@@ -42,9 +42,11 @@ The two releases also cannot run side by side on one cluster. The group rename s
5. Install [`yq`](https://github.com/mikefarah/yq#install) version 4 or later. The steps in [Resources that you can convert](#resources-that-you-can-convert) use `yq` to convert the exported resources.
+6. Install [`jq`](https://jqlang.org/download/). The steps in [Verify the new installation](#verify-the-new-installation) use `jq` to read the Session ID.
+
## Export your 0.10.x resources
-Export your resources before the 1.0 CRDs replace the 0.10.x CRDs. Some resources convert to 1.0, and the rest serve as a reference when you write their 1.0 replacements. 1.0 cannot read the conversation history of 0.10.x.
+Export your resources before you uninstall 0.10.x. Some resources convert to 1.0, and the rest serve as a reference when you write their 1.0 replacements.
1. Export every kagent resource in your namespace.
```bash
@@ -52,19 +54,21 @@ Export your resources before the 1.0 CRDs replace the 0.10.x CRDs. Some resource
-n kagent -o yaml > kagent-0.10-resources.yaml
```
-2. Note the Secrets that your ModelConfigs reference. Secrets are ordinary Kubernetes resources that kagent does not own, so recreate them on the new cluster by the same method that you used originally.
+2. List the Secrets that your ModelConfigs reference. kagent does not own these Secrets, so the export does not include them.
```bash
kubectl get modelconfigs -n kagent \
-o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.apiKeySecret}{"\n"}{end}'
```
+3. If you install 1.0 on the same cluster, uninstall 0.10.x now. For the steps, see [Uninstall kagent]({{< link path="operations/uninstall#uninstall-with-helm" version="0.x" >}}) in the 0.x documentation.
+
## Install kagent 1.0
-Install 1.0 against the empty database that you provisioned. kagent and Agent Substrate both store their state in that database, each with an owner login that applies migrations and a read/write login that serves runtime queries. `kagent install` normally creates a development database for both. The `--skip-database-setup` flag skips that database. The releases then read the connection Secrets that you create in this section.
+Install 1.0 against the empty database that you provisioned. kagent and Agent Substrate both store their state in that database, each with an owner login that applies migrations and a read/write login that serves runtime queries. `kagent install` normally creates a development database for both. When you pass `--skip-database-setup`, `kagent install` does not create the development database. The kagent and Agent Substrate charts then read the connection Secrets that you create in this section.
Before you start, complete the [Before you begin]({{< link path="setup/installation#before-you-begin" >}}) steps of the installation guide. Those steps prepare a cluster at Kubernetes 1.37 or later and your provider API key. Then install the CLI tools in step 1 of [Install with the CLI]({{< link path="setup/installation#install-kagent" >}}).
-1. Create the namespaces that contain the connection Secrets. `kagent install` uses the namespaces when they already exist.
+1. Create the namespaces that contain the connection Secrets. If a namespace already exists, `kagent install` uses it.
```bash
kubectl create namespace kagent
kubectl create namespace ate-system
@@ -118,6 +122,8 @@ Before you start, complete the [Before you begin]({{< link path="setup/installat
EOF
```
+ These values connect kagent to the external database and turn on the features that later steps use.
+
| Value | Why |
| ----- | --- |
| `database.postgres.ownerConnectionStringSecretRef.name` and `database.postgres.readWriteConnectionStringSecretRef.name` | Read the two connection strings from the Secrets that you created. The keys default to `ownerConnectionString` and `readWriteConnectionString`. |
@@ -125,9 +131,7 @@ Before you start, complete the [Before you begin]({{< link path="setup/installat
| `config.database.vectorEnabled` | Applies the pgvector migrations, which [long-term memory]({{< link path="agents/agent-memory" >}}) depends on. If your database does not have the pgvector extension installed, omit the value. |
| `config.server.grpcReflection` | Lets the guides that use `grpcurl` discover the controller API, as in the installation guide. |
- The controller applies migrations at startup. To apply them from a pipeline instead, set `config.database.skipMigrations=true` and run `kagent db migrate up` with the owner login yourself. The controller then verifies the schema and fails if the schema is not already current.
-
-6. Save a CLI configuration file that references those values. The `kagent db` commands read `database.vectorEnabled` from this file, not from the Helm values. Set `database.vectorEnabled` to match `config.database.vectorEnabled`. If you omitted `config.database.vectorEnabled`, omit `database.vectorEnabled` too.
+6. Save a CLI configuration file that references `kagent-values.yaml`. The `kagent db` commands read `database.vectorEnabled` from this file, not from the Helm values. Set `database.vectorEnabled` to match `config.database.vectorEnabled`. If you omitted `config.database.vectorEnabled`, omit `database.vectorEnabled` too.
```bash
cat > kagent-config.yaml <}}).
-1. Before you apply the ModelConfigs, recreate the Secrets that they reference. Otherwise, the ModelConfigs cannot resolve their Secrets.
+`v1alpha3` reshapes ModelProviderConfig. Model discovery accepts only the `OpenAI` type, which covers any OpenAI-compatible endpoint. A ModelProviderConfig of another type does not have a `v1alpha3` equivalent. The `spec.secretRef` field moves to `spec.discoveryAuth.secretRef`, which references both the Secret and the key within it. kagent uses that credential only to list models, never to invoke them.
+
+1. Recreate the Secrets that your ModelConfigs reference. Without them, the ModelConfigs cannot resolve their API keys.
+ ```bash
+ kubectl create secret generic -n kagent --from-literal =
+ ```
-2. Convert these resources to `v1alpha3`. The export also contains Agents and other kinds that `v1alpha3` does not serve, so the command keeps only ModelConfig, ModelProviderConfig, and RemoteMCPServer resources. The command sets each `apiVersion`, renames the `apiKeyPassthrough` field to `tokenPassthrough`, deletes the `spec.openAI.tokenExchange` field that `v1alpha3` removed, and removes the metadata that the 0.10.x API server set.
+2. Convert the ModelConfig, RemoteMCPServer, and `OpenAI` ModelProviderConfig resources to `v1alpha3`. The command drops every other kind and makes the following changes.
+ - Sets each `apiVersion`.
+ - Renames the `apiKeyPassthrough` field to `tokenPassthrough`.
+ - Deletes the ModelConfig fields that `v1alpha3` removed.
+ - Removes the metadata that the 0.10.x API server set.
+ - Moves the ModelProviderConfig Secret name to `spec.discoveryAuth.secretRef`, with a `` placeholder for the key.
+ - Drops the `kagent-grafana-mcp` RemoteMCPServer, because 1.0 does not deploy the Grafana Model Context Protocol (MCP) server behind it.
```bash
- yq '.items |= map(select(.kind == "ModelConfig" or .kind == "ModelProviderConfig" or .kind == "RemoteMCPServer"))
+ yq '.items |= map(select(.kind == "ModelConfig" or .kind == "RemoteMCPServer" or (.kind == "ModelProviderConfig" and .spec.type == "OpenAI")))
| .items[].apiVersion = "api.kagent.dev/v1alpha3"
| del(.items[].status, .items[].metadata.resourceVersion, .items[].metadata.uid, .items[].metadata.creationTimestamp, .items[].metadata.generation, .items[].metadata.managedFields, .items[].metadata.annotations["kubectl.kubernetes.io/last-applied-configuration"])
| (.items[] | select(.kind == "ModelConfig" and has("spec") and (.spec | has("apiKeyPassthrough"))) | .spec) |= (.tokenPassthrough = .apiKeyPassthrough | del(.apiKeyPassthrough))
- | del(.items[] | select(.kind == "ModelConfig") | .spec.openAI.tokenExchange)' \
+ | del(.items[] | select(.kind == "ModelConfig") | (.spec.openAI.tokenExchange, .spec.tls.caCertSecretRef, .spec.tls.caCertSecretKey, .spec.tls.disableSystemCAs))
+ | del(.items[] | select(.kind == "RemoteMCPServer" and .metadata.name == "kagent-grafana-mcp"))
+ | (.items[] | select(.kind == "ModelProviderConfig" and (.spec | has("secretRef"))) | .spec) |= (.discoveryAuth.secretRef = {"name": .secretRef.name, "key": ""} | del(.secretRef))' \
kagent-0.10-resources.yaml > kagent-1.0-resources.yaml
```
-3. If the converted file includes the `kagent-grafana-mcp` RemoteMCPServer, delete it, because 1.0 does not deploy the Grafana Model Context Protocol (MCP) server behind it.
+3. If the converted file includes a ModelProviderConfig with a `` placeholder, replace the placeholder with the key that contains the API key. A 0.10.x discovery Secret contains exactly one key. To find the key, list the keys of the Secret that `discoveryAuth.secretRef.name` references.
+ ```bash
+ kubectl get secret -n -o go-template='{{range $k, $v := .data}}{{$k}}{{"\n"}}{{end}}'
+ ```
4. Apply the converted resources.
```bash
@@ -192,16 +212,16 @@ kagent 1.0 cannot pass a mounted credential or certificate file to an agent. In
### Agents become an AgentTemplate, a Harness, and an Agent
-A 0.10.x `Agent` described both what the agent does and how it runs. In 1.0, an AgentTemplate defines the agent's behavior, and a Harness defines the runtime and infrastructure. The 1.0 `Agent` resource pairs one AgentTemplate with one Harness. One Harness serves many Agents, so expect fewer Harnesses than you had 0.10.x Agents.
+A 0.10.x Agent described both what the agent does and how it runs. In 1.0, an AgentTemplate defines the agent's behavior, and a Harness defines the runtime and infrastructure. The 1.0 Agent resource pairs one AgentTemplate with one Harness. One Harness serves many Agents, so expect fewer Harnesses than you had 0.10.x Agents.
-| 0.10.x `Agent` field | Equivalent field in 1.0 |
+| 0.10.x Agent field | Equivalent field in 1.0 |
| -------------------- | -------------------- |
| `spec.description` | `AgentTemplate.spec.description` |
| `spec.declarative.systemMessage` | `AgentTemplate.spec.systemPrompt` |
| `spec.declarative.systemMessageFrom` | `AgentTemplate.spec.systemPromptFrom` |
| `spec.declarative.promptTemplate` | `AgentTemplate.spec.promptTemplate`, unchanged |
| `spec.declarative.modelConfig` | `AgentTemplate.spec.modelConfig`, now an object with a `name` rather than a bare string |
-| `spec.declarative.tools` | `AgentTemplate.spec.tools`, reshaped. See the following note. |
+| `spec.declarative.tools` | `AgentTemplate.spec.tools`. A 0.10.x tool set `type: McpServer` or `type: Agent` alongside a matching block. A 1.0 binding contains either an `mcp` block or a `subAgent` block and does not have a `type` field. The block that you set determines the binding's kind. For more information, see [About tools]({{< link path="skills-and-mcp/about-tools" >}}). |
| `spec.declarative.memory` | `Harness.spec.kagent.memory`, so memory is now a property of the runtime rather than of one agent |
| `spec.declarative.runtime` | `Harness.spec.workload.image`, through the runtime that the Harness selects |
| `spec.declarative.deployment` | `Harness.spec.workload` and `Harness.spec.substrate`. Agents no longer run as Deployments. |
@@ -210,51 +230,78 @@ A 0.10.x `Agent` described both what the agent does and how it runs. In 1.0, an
| `spec.iconUrl`, `spec.documentationUrl`, `spec.version`, `spec.provider` | Nothing. kagent builds the agent card from the AgentTemplate's name and description. |
| `spec.declarative.stream`, `executeCodeBlocks`, `shareTools`, `context` | Nothing. `v1alpha3` does not have an equivalent field. |
-> [!NOTE]
-> Tool bindings changed shape. A 0.10.x tool set `type: McpServer` or `type: Agent` alongside a matching block. A 1.0 `ToolBinding` contains either an `mcp` block or a `subAgent` block and has no `type` field, so the block that you set determines the binding's kind. For what each binding does, see [About tools]({{< link path="skills-and-mcp/about-tools" >}}).
+To recreate your agents, write the three 1.0 resources for each 0.10.x Agent, then apply them. For a worked example of all three resources, see [Your first agent]({{< link path="get-started/your-first-agent#create-a-harness-an-agenttemplate-and-an-agent" >}}).
+
+1. Write one AgentTemplate for each 0.10.x Agent. Use the field mapping in the preceding table.
+
+2. Write one Harness for each distinct runtime and infrastructure combination that your agents need. For the Harness fields, see [Agent harness]({{< link path="agents/agent-harness#configure-a-harness" >}}).
-Write one Harness for each distinct runtime and infrastructure combination that your agents need. Then write one Agent for each 0.10.x Agent. Each Agent references the AgentTemplate and the Harness that it pairs. Neither reusable resource references the other, so the Agent makes a template runnable. For the full field reference, see [Agent harness]({{< link path="agents/agent-harness#configure-a-harness" >}}), and for a worked example, see [Your first agent]({{< link path="get-started/your-first-agent#create-a-harness-an-agenttemplate-and-an-agent" >}}).
+3. Write one Agent for each 0.10.x Agent. Each Agent references its AgentTemplate in `spec.template.ref.name` and its Harness in `spec.harnessRef.name`.
-### Resources with no 1.0 equivalent
+4. Apply the resources. Apply the Harnesses and AgentTemplates first, so that each Agent resolves its references when you create it.
+ ```bash
+ kubectl apply -f
+ kubectl apply -f
+ kubectl apply -f
+ ```
+
+### Removed resources
-The following resource kinds are removed without a replacement, so plan for each one before you cut over.
+The following resource kinds do not exist in 1.0. Plan for each one before you cut over.
| Removed resource | What to do |
| ---------------- | ---------- |
-| `AgentHarness` | No equivalent. It provisioned OpenClaw and Hermes coding-agent sandboxes with Slack and Telegram channels. 1.0's `Harness` shares part of the name and nothing else. |
-| `Memory` | Configure memory on the Harness with `spec.kagent.memory` instead. For more information, see [Agent memory]({{< link path="agents/agent-memory#enable-memory" >}}). |
-| `SandboxAgent` | No equivalent, and none is needed. Every 1.0 agent runs in a gVisor sandbox by default. |
-| `ToolServer` | Use `RemoteMCPServer`, which 0.10.x already served alongside it. |
+| AgentHarness | No equivalent. It provisioned OpenClaw and Hermes coding-agent sandboxes with Slack and Telegram channels. The 1.0 Harness shares part of the name and nothing else. |
+| Memory | Configure memory on the Harness by using `spec.kagent.memory`. For more information, see [Agent memory]({{< link path="agents/agent-memory#enable-memory" >}}). |
+| SandboxAgent | No equivalent, and none is needed. Every 1.0 agent runs in a gVisor sandbox by default. |
+| ToolServer | Use RemoteMCPServer, which 0.10.x already served alongside it. |
## Verify the new installation
Confirm that the resources resolved before you retire anything. Harness and AgentTemplate resources do not report their own statuses, so check for issues in the Agent that references them both.
-1. Check that each Agent is ready.
+1. Check that each Agent is ready. Every migrated Agent reports `True` in the `READY` column.
```bash
kagent agent list
```
+ Example output:
+ ```console
+ +----------------+-------+----------------------+
+ | NAME | READY | CREATED |
+ +----------------+-------+----------------------+
+ | my-first-agent | True | 2026-11-04T23:30:29Z |
+ +----------------+-------+----------------------+
+ ```
2. Read the conditions on any Agent that is not ready. `ResolvedRefs` reports a missing reference. The most common cause is a Harness that references a WorkerPool that does not exist.
```bash
kubectl describe agent -n kagent
```
-3. Create a Session against a migrated Agent and send it a message. A reply confirms the whole path, from the revision to the model credentials.
+3. Create a Session against a migrated Agent, and save its ID.
```bash
- kagent agent session create --agent
+ export SESSION_ID=$(kagent agent session create --agent -o json | jq -r '.session.id')
+ ```
+
+4. Send a message to the Session. A reply confirms the whole path, from the revision to the model credentials.
+ ```bash
+ kagent agent invoke --session $SESSION_ID --task "Reply with the word ready."
+ ```
+ Example output:
+ ```console
+ ready
```
## Retire the 0.10.x installation
-After the new installation answers correctly, remove the old one. Follow the 0.x procedure rather than the 1.x procedure, because a 0.10.x installation does not have Agent Substrate, `ate.dev` resources, or identity material to clean up. For those steps, see [Uninstall kagent]({{< relref "/kagent/0.x/operations/uninstall" >}}#uninstall-with-helm) in the 0.x documentation.
+If you installed 1.0 on a second cluster, remove the old installation after the new one answers correctly. Follow the 0.x procedure, because a 0.10.x installation does not have Agent Substrate, `ate.dev` resources, or identity material to clean up. For those steps, see [Uninstall kagent]({{< link path="operations/uninstall#uninstall-with-helm" version="0.x" >}}) in the 0.x documentation.
-Keep the database backup after the uninstall for your own records. The backup is the only remaining copy of the 0.10.x conversation history, and nothing in 1.0 can read it.
+Keep the database backup after the uninstall. The backup is the only remaining copy of the 0.10.x conversation history.
## Next steps
{{< cards >}}
{{< card link=`{{< link path="about/core-concepts" >}}` title="Core concepts" subtitle="Learn the Harness, AgentTemplate, Agent, and Session model that replaces the 0.x Agent resource." >}}
- {{< card link=`{{< link path="reference/versions" >}}` title="Version support" subtitle="Check which upgrade paths kagent supports from 1.0 onward." >}}
- {{< card link=`{{< link path="operations/operational-considerations" >}}` title="Operational considerations" subtitle="Replace the evaluation defaults before the new installation carries real traffic." >}}
+ {{< card link=`{{< link path="reference/versions#release-support-and-compatibility" >}}` title="Version support" subtitle="Check which upgrade paths kagent supports from 1.0 onward." >}}
+ {{< card link=`{{< link path="operations/operational-considerations" >}}` title="Operational considerations" subtitle="Replace the evaluation defaults before the new installation serves production traffic." >}}
{{< /cards >}}
diff --git a/docs-site/content/kagent/1.x/operations/upgrade.md b/docs-site/content/kagent/1.x/operations/upgrade.md
index 8423a5cf..db123159 100644
--- a/docs-site/content/kagent/1.x/operations/upgrade.md
+++ b/docs-site/content/kagent/1.x/operations/upgrade.md
@@ -7,9 +7,7 @@ author: kagent.dev
An installation of {{< reuse "kagent-docs/snippets/name-product.md" >}} consists of five Helm releases. Upgrade the releases in the order that you installed them, starting with the CRDs, then Agent Substrate, and then the kagent control plane.
-Upgrade from one stable release to the next, and do not skip a stable release. Before you move to the next stable release, upgrade to the latest patch version of your current stable release. Patch releases do not change the database schema, so you can upgrade directly to the latest patch release without the patch releases in between. To move a 0.10.x installation to 1.x, see [Upgrade from 0.x]({{< link path="operations/upgrade-from-0x" >}}) instead.
-
-
+Upgrade from one minor release to the next, such as from 1.0 to 1.1, and do not skip a minor release. Before you move to the next minor release, upgrade to the latest patch version of your current minor release. Patch releases do not change the database schema, so you can upgrade directly to the latest patch release without the patch releases in between. To move a 0.10.x installation to 1.x, see [Upgrade from 0.x]({{< link path="operations/upgrade-from-0x" >}}) instead.
## Prepare to upgrade
@@ -49,7 +47,7 @@ Complete every step in this section before you change any release. The steps mak
```
For an installation that you exported from the kagent CLI, the `kagent-values.yaml` file contains your model provider API key in plain text. Store the files securely.
-6. Save the target versions in environment variables. The following values are the versions that this documentation describes. To upgrade across more than one stable release, set the versions of the next stable release, and repeat the upgrade for each stable release.
+6. Save the target versions in environment variables. The following values are the versions that this documentation describes. To upgrade across more than one minor release, set the versions of the next minor release, and repeat the upgrade for each minor release.
```bash
export KAGENT_VERSION={{< reuse "kagent-docs/versions/kagent.md" >}}
export SUBSTRATE_VERSION={{< reuse "kagent-docs/versions/agent-substrate-chart.md" >}}
diff --git a/docs-site/content/kagent/1.x/reference/release-notes/1.0.md b/docs-site/content/kagent/1.x/reference/release-notes/1.0.md
index e29d1962..23418188 100644
--- a/docs-site/content/kagent/1.x/reference/release-notes/1.0.md
+++ b/docs-site/content/kagent/1.x/reference/release-notes/1.0.md
@@ -34,47 +34,45 @@ For the component versions that this release pins and the Kubernetes version tha
### How you declare an agent
-The single 0.10.x Agent resource described both what an agent does and how it runs. 1.0 separates those concerns across two reusable resources, pairs them in a third, and tracks each running Session as a fourth object.
+The single 0.10.x Agent resource described both what an agent does and how it runs. 1.0 splits that description across the following resources, and represents each running conversation as a Session.
- **A {{< gloss "Harness" >}}Harness{{< /gloss >}} defines how an agent is allowed to run**, including its runtime, workload image, {{< gloss "WorkerPool" >}}WorkerPool{{< /gloss >}}, and snapshot storage. One Harness serves many agents, so expect fewer Harnesses than you had 0.10.x Agents.
- **An {{< gloss "AgentTemplate" >}}AgentTemplate{{< /gloss >}} defines what an agent does**, including its model, system prompt, tools, skills, and plugins.
- **An {{< gloss "Agent" >}}Agent{{< /gloss >}} pairs one AgentTemplate with one Harness.** `spec.template` takes either a reference to an existing AgentTemplate or a complete inline spec. `spec.harnessRef` takes only a reference. The Agent reports readiness. Callers address the Agent, not the AgentTemplate or the Harness.
- **A Session is one running conversation** with an Agent. Unlike a Harness, an AgentTemplate, or an Agent, a Session is not a custom resource. kagent's gRPC API creates each Session, and kagent's database stores it.
-A Harness and an AgentTemplate do not reference each other. Neither resource has a field that names the other, so only an Agent connects them. For the full model, see [Core concepts]({{< link path="about/core-concepts#agent" >}}).
+A Harness and an AgentTemplate do not reference each other, so only an Agent connects them. For the full model, see [Core concepts]({{< link path="about/core-concepts#agent" >}}).
The custom resource definition (CRD) API group also moves from `kagent.dev/v1alpha2` to `api.kagent.dev/v1alpha3`. kagent does not convert resources between the two groups. 1.0 no longer bundles kmcp, so it does not install the `mcpservers.kagent.dev` CRD. 1.0 also no longer bundles the Grafana Model Context Protocol (MCP) server, so it does not create the `kagent-grafana-mcp` RemoteMCPServer. 0.10.x and 1.0 cannot run side by side on one cluster, because both charts name their ClusterRoles after the release. To keep 0.10.x serving while you move, install 1.0 on a second cluster.
### New capabilities
-The following features are new in 1.0 and have no 0.10.x counterpart.
+The following features are new in 1.0 and do not have a 0.10.x counterpart.
- **Multi-harness support:** Besides the built-in `kagent` harness, a Harness can run Codex, Claude, or a container image of your own that implements kagent's Agent2Agent (A2A) contract. For what each harness supports, see [Choose a harness]({{< link path="agents/agent-harness#choose-a-harness" >}}).
- **Actors on Micro-VMs:** gVisor remains the default sandbox class. The `microvm` class places a hypervisor boundary between the workload and the host. For how a pool selects its class, see [Sandbox classes]({{< link path="about/substrate-runtime/sandboxing#sandbox-classes" >}}).
-- **Access to destinations with EgressPolicy resources:** You can apply an EgressPolicy to an Agent or a SandboxTemplate to inject a credential into requests that the Agent or SandboxTemplate sends to a destination. For the procedure, see [Grant access to a destination]({{< link path="agents/egress-policies#grant-access-to-a-destination" >}}).
+- **Access to destinations with EgressPolicy resources:** To inject a credential into requests that an Agent or a SandboxTemplate sends to a destination, apply an EgressPolicy to that Agent or SandboxTemplate. For the procedure, see [Grant access to a destination]({{< link path="agents/egress-policies#grant-access-to-a-destination" >}}).
- **Standalone sandboxes with SandboxTemplates:** A SandboxTemplate defines a scratch environment that is independent of any Session. A person, or an agent through the kagent MCP server, creates a Sandbox from the template to run commands and transfer files. For the procedure, see [Create a SandboxTemplate]({{< link path="about/substrate-runtime/standalone-sandboxes#create-a-sandboxtemplate" >}}).
- **Actor identity for model authentication:** A ModelConfig can authenticate to a model endpoint as the calling Actor instead of with a stored credential. When you set `actorJwt`, the egress gateway sends a token that identifies the Actor to a model endpoint that you run, so the ModelConfig does not need a Secret. For the fields, see [Authenticate as the Actor]({{< link path="setup/model-providers/about-model-providers#authenticate-as-the-actor" >}}).
- **Context compaction:** The `kagent` harness summarizes older Session events according to the `spec.kagent.compaction` block on the Harness. For the settings, see [Configure compaction]({{< link path="agents/context-management#configure-compaction" >}}).
- **Structured output:** You can constrain an agent's final answer to a JSON Schema, so that callers receive a response in a predictable shape. The AgentTemplate sets the schema in its `outputSchema` field or reads it from a ConfigMap. For an example, see [Set the schema inline]({{< link path="agents/structured-output#set-the-schema-inline" >}}).
- **Session sharing with share tokens:** A share token gives another account read-only access to one Session. For the access model, see [Share a Session]({{< link path="agents/share-a-session#a-share-is-a-capability-not-a-grant-to-a-user" >}}).
-- **Scheduled runs:** You can run an Agent on a cron expression. Each execution sends a fixed prompt to a new Session, so scheduled runs do not continue an earlier conversation. For the procedure, see [Create a schedule in the UI]({{< link path="agents/scheduled-runs#create-a-schedule-in-the-ui" >}}).
+- **Scheduled runs:** You can run an Agent on a cron expression. Each execution sends a fixed prompt to a new Session, so scheduled runs do not continue an earlier Session. For the procedure, see [Create a schedule in the UI]({{< link path="agents/scheduled-runs#create-a-schedule-in-the-ui" >}}).
### Removed and replaced resources
-Each 0.10.x resource kind is replaced, removed, or carried forward in 1.0. The distinction matters when you plan a move. A replaced kind has a destination for its configuration. A removed kind does not.
+Each 0.10.x resource kind is replaced, removed, carried forward, or carried forward with changes in 1.0. A replaced kind has a destination for its configuration. A removed kind does not.
| 0.10.x resource | Status in 1.0 |
| --------------- | ------------- |
| Agent | Replaced by an AgentTemplate, a Harness, and a new Agent in the `api.kagent.dev` group that pairs them. The two Agent kinds share a name and nothing else. |
| ToolServer | Replaced by {{< gloss "RemoteMCPServer" >}}RemoteMCPServer{{< /gloss >}}, which 0.10.x already served alongside it. |
| Memory | Replaced by `Harness.spec.kagent.memory`, so memory is a property of the runtime rather than of one agent. |
-| SandboxAgent | Removed with no replacement. Every 1.0 agent runs in a sandbox by default. |
+| SandboxAgent | Removed, with no equivalent. Every 1.0 agent runs in a sandbox by default. |
| MCPServer | Removed, because 1.0 no longer bundles kmcp or its controller. Run the MCP server outside of kagent, and set its URL on a RemoteMCPServer. The `kagent mcp` commands are removed with it. |
-| AgentHarness | Removed, with no equivalent. It provisioned OpenClaw and Hermes coding-agent sandboxes with Slack and Telegram channels. |
-| ModelConfig, ModelProviderConfig, RemoteMCPServer | Carried forward. `v1alpha3` keeps every field that `v1alpha2` defines except one, so an exported resource needs a new `apiVersion` and one field rename. The ModelConfig field `apiKeyPassthrough` is now `tokenPassthrough`, the same name that RemoteMCPServer uses for the same behavior. |
-
-> [!IMPORTANT]
-> The 1.0 Harness and the 0.10.x AgentHarness share part of a name and nothing else. AgentHarness provisioned coding-agent sandboxes, while Harness governs how any agent is allowed to run. Read Harness as a new resource rather than a renamed one.
+| AgentHarness | Removed, with no equivalent. AgentHarness provisioned OpenClaw and Hermes coding-agent sandboxes with Slack and Telegram channels. The 1.0 Harness is a new resource, not a renamed AgentHarness. |
+| ModelConfig, RemoteMCPServer | Carried forward. An exported resource needs a new `apiVersion`. The ModelConfig field `apiKeyPassthrough` is now `tokenPassthrough`, the same name that RemoteMCPServer uses for the same behavior. `v1alpha3` removes the ModelConfig fields `spec.openAI.tokenExchange`, `spec.tls.caCertSecretRef`, `spec.tls.caCertSecretKey`, and `spec.tls.disableSystemCAs`. |
+| ModelProviderConfig | Carried forward with changes. Model discovery accepts only the `OpenAI` type, so a ModelProviderConfig of another type does not have a 1.0 equivalent. The `spec.secretRef` field moves to `spec.discoveryAuth.secretRef`, which references both the Secret and the key within it. |
Several Agent fields do not have a `v1alpha3` equivalent, including `a2aConfig`, `executeCodeBlocks`, `shareTools`, and `context`. A2A is always on in 1.0. A caller addresses an Agent at `/agents/{namespace}/{name}` over HTTP, or through the A2A `tenant` field over gRPC. Tool bindings also have a new shape. A 1.0 `ToolBinding` contains an `mcp` or `subAgent` block and does not have a `type` discriminator. For the field-by-field mapping, see [Agents become an AgentTemplate, a Harness, and an Agent]({{< link path="operations/upgrade-from-0x#agents-become-an-agenttemplate-a-harness-and-an-agent" >}}).
@@ -86,4 +84,4 @@ Compatibility promises start at 1.0. Plan a migration from 0.10.x, not an upgrad
- **Downgrade from 1.0 to 0.10.x is not supported.**
- **1.0 to a later minor release is supported.** From 1.0 onward, migrations are append-only and each minor release retains compatibility with the previous release line.
-Moving an existing installation therefore means a migration. Install 1.0 on a second cluster, recreate your resources on it, and retire 0.10.x once the new installation answers correctly. Installing on the same cluster requires uninstalling 0.10.x first, so your agents are unavailable until 1.0 is running. Because 1.0 cannot read a 0.10.x database, Session history does not move to the new installation. For the full procedure, see [Upgrade from 0.x]({{< link path="operations/upgrade-from-0x#in-place-upgrade-blockers" >}}).
+Moving an existing installation therefore means a migration. Install 1.0 on a second cluster, recreate your resources on it, and retire 0.10.x after the new installation answers correctly. 0.10.x and 1.0 cannot run side by side on one cluster, because both charts name their ClusterRoles after the release. Installing on the same cluster requires uninstalling 0.10.x first, so your agents are unavailable until 1.0 is running. Because 1.0 cannot read a 0.10.x database, 0.10.x conversation history does not move to the new installation. For the full procedure, see [Upgrade from 0.x]({{< link path="operations/upgrade-from-0x" >}}).
\ No newline at end of file
diff --git a/docs-site/content/kagent/1.x/setup/model-providers/gemini.md b/docs-site/content/kagent/1.x/setup/model-providers/gemini.md
index 602184bf..7f122ddd 100644
--- a/docs-site/content/kagent/1.x/setup/model-providers/gemini.md
+++ b/docs-site/content/kagent/1.x/setup/model-providers/gemini.md
@@ -5,7 +5,7 @@ weight: 20
author: kagent.dev
---
-The `Gemini` provider reaches Google's Gemini models through the Google AI Studio API, authenticating with a single API key. Gemini models served through Google Cloud Vertex AI do not run on kagent 1.0. For the reason, see [Google Vertex AI]({{< link path="setup/model-providers/google-vertexai" >}}).
+The `Gemini` provider reaches Google's Gemini models through the Google AI Studio API or a custom endpoint that serves the same API. kagent 1.0 rejects the `GeminiVertexAI` provider, so a ModelConfig cannot reach Gemini models through Google Cloud Vertex AI. For the reason, see [Google Vertex AI]({{< link path="setup/model-providers/google-vertexai" >}}).
## Before you begin
@@ -13,7 +13,7 @@ Make sure that your Google Cloud account has a project with the Gemini API enabl
## Create the ModelConfig
-1. Get an API key from [Google AI Studio](https://ai.google.dev/), and save it as an environment variable.
+1. Save your [Google AI Studio](https://ai.google.dev/) API key as an environment variable.
```bash
export GOOGLE_API_KEY=
```
@@ -40,6 +40,8 @@ Make sure that your Google Cloud account has a project with the Gemini API enabl
EOF
```
+ The following fields connect the ModelConfig to the `kagent-gemini` Secret and select the model.
+
| Field | Description |
| ----- | ----------- |
| `apiKeySecret` | The name of the Kubernetes Secret that stores the API key, in the same namespace as this ModelConfig. |
@@ -50,12 +52,49 @@ Make sure that your Google Cloud account has a project with the Gemini API enabl
## Gemini provider settings
-The `gemini` block takes one optional setting. For its type, default, and validation rules, see the [API reference]({{< link path="reference/api-ref#geminiconfig" >}}).
+The `gemini` block takes the following optional settings. For their types, defaults, and validation rules, see the [API reference]({{< link path="reference/api-ref#geminiconfig" >}}).
+
+```yaml
+spec:
+ gemini:
+ baseUrl: https://gateway.example.com/team/v1beta
+ maxOutputTokens:
+```
| Field | Description |
| ----- | ----------- |
+| `baseUrl` | The complete API prefix of a custom Gemini endpoint. For the format, see [Use a custom Gemini endpoint](#use-a-custom-gemini-endpoint). |
| `maxOutputTokens` | A cap on the tokens generated in one response. |
+## Use a custom Gemini endpoint
+
+To route Gemini calls through a gateway or proxy that serves the Gemini API, set `gemini.baseUrl` to the endpoint's complete API prefix, including any version segment such as `/v1beta`. kagent appends only `/models/:generateContent` or its streaming equivalent to that prefix.
+
+The endpoint and the credential are independent settings. Choose the credential that the endpoint accepts.
+
+- kagent sends the API key from `apiKeySecret` in the `x-goog-api-key` header.
+- The egress gateway sends the Actor token from `actorJwt` in the `Authorization: Bearer` header. For the `actorJwt` fields, see [Authenticate as the Actor]({{< link path="setup/model-providers/about-model-providers#authenticate-as-the-actor" >}}).
+- If the endpoint does not require a credential, omit both fields.
+
+The following example uses the API key from the `kagent-gemini` Secret.
+
+```yaml
+kubectl apply -f - <}}).
@@ -69,6 +108,6 @@ spec:
## Next steps
{{< cards >}}
- {{< card link=`{{< link path="get-started/your-first-agent" >}}` title="Your first agent" subtitle="Create an agent that uses this model, and hold a conversation with it." >}}
+ {{< card link=`{{< link path="get-started/your-first-agent" >}}` title="Your first agent" subtitle="Create a Harness, an AgentTemplate, and an Agent, and start a Session with the Agent." >}}
{{< card link=`{{< link path="setup/model-providers/about-model-providers" >}}` title="About model providers" subtitle="Understand how a ModelConfig reaches a running agent." >}}
{{< /cards >}}
diff --git a/docs-site/playwright/tests/chat.spec.ts b/docs-site/playwright/tests/chat.spec.ts
index fc3de99f..00fc2c17 100644
--- a/docs-site/playwright/tests/chat.spec.ts
+++ b/docs-site/playwright/tests/chat.spec.ts
@@ -39,10 +39,13 @@ test('conversation: the seeded transcript', async ({ page }) => {
await gotoUI(page, `/agents/${SEEDED_INSTANCE}/chat`);
- // The composer is the last thing to mount, so it is the honest signal that the
- // conversation has finished loading rather than that its shell has.
+ // The composer mounts alongside the loading skeletons, so it proves only that the
+ // shell has rendered. The session list and the absence of every skeleton are the
+ // signal that the conversation itself has loaded.
await expect(page.getByTestId('chat-composer')).toBeVisible({ timeout: 30_000 });
await expect(page.getByTestId('chat-input')).toBeVisible();
+ await expect(page.getByTestId('chat-sessions-list')).toBeVisible({ timeout: 30_000 });
+ await expect(page.locator('.ant-skeleton')).toHaveCount(0, { timeout: 30_000 });
// Guards against capturing the empty state, which renders the same chrome.
await expect(page.getByTestId('chat-empty')).toHaveCount(0);
await settle(page);