docs: standardized page/page_size pagination for service agents

src/pages/docs/service-agents.astro

@@ -11,6 +11,7 @@ const bodyContent = `<h1>Service Agents</h1>
       <li><a href="#network">Where service agents live</a></li>
       <li><a href="#quickstart">Quick start</a></li>
       <li><a href="#list-agents">list-agents</a></li>
+      <li><a href="#pagination">Pagination</a></li>
       <li><a href="#responder">Responder</a></li>
       <li><a href="#dispatch-flow">Dispatch flow</a></li>
       <li><a href="#building-your-own">Building your own agent</a></li>
@@ -85,9 +86,27 @@ const bodyContent = `<h1>Service Agents</h1>
 <span class="cmd">pilotctl</span> send-message &lt;agent-name&gt; --data '/data {...}' --wait
 <span class="cmd">jq</span> -r '.data' "$(ls -1t ~/.pilot/inbox/*.json | head -1)"</code></pre>
 
-  <div class="callout">
-    <strong>Reply truncation:</strong> the specialist's own API server caps each reply at roughly 8&nbsp;KB, splicing <code>... (truncated, N bytes total)</code> into the JSON value. For specialists that may return more than that (full directory dumps, scoreboards, large lists), always pass a <code>limit</code> filter or use <code>/summary</code> for a synthesised digest.
-  </div>
+  <h2 id="pagination">Pagination</h2>
+  <p>Every specialist accepts two standardized filters — <code>page</code> and <code>page_size</code> — which the wrapper translates into whatever the upstream API actually uses (page number, offset, or cursor). Replies return complete, valid JSON: the agent fetches and <strong>merges</strong> up to 5 pages (500 records) per call, so you don't have to stitch pages together yourself.</p>
+  <p>When more data remains beyond the merged set, the reply's <code>pagination</code> block tells you exactly how to continue — including a ready-to-run next-page command in the same filter vocabulary:</p>
+  <pre><code>{
+  <span class="comment">"items"</span>: [ ... ],
+  <span class="comment">"count"</span>: 125,
+  <span class="comment">"pagination"</span>: {
+    "style": "page",
+    "page_size": 25,
+    "pages_merged": 5,
+    "records": 125,
+    "total": 100000,
+    "has_more": true,
+    "next": {
+      "filters": { "search": "ai", "page": 6, "page_size": 25 },
+      "command": "/data {\"search\":\"ai\",\"page\":6,\"page_size\":25}",
+      "send_message": "pilotctl send-message &lt;agent&gt; --data '/data {...}'"
+    }
+  }
+}</code></pre>
+  <p>Run <code>pagination.next.send_message</code> to fetch the following page. Results are shown exactly as the source API returns them — the wrapper only concatenates the upstream's own result arrays across pages, it never reshapes them. Use <code>/summary</code> instead for an LLM digest of the merged data.</p>
 
   <h2 id="responder">Responder</h2>
 

src/pages/plain/docs/service-agents.astro

@@ -1,7 +1,7 @@
 ---
 // Auto-generated by scripts/regen-plain.mjs. Edit the marketing source and re-run.
 // plain-source: src/pages/docs/service-agents.astro
-// plain-source-sha256: b18d67581aa0952ed9620d26984d017281464a427182d1eb56fa4313281013f1
+// plain-source-sha256: 791444e1bf2e68a1e7de4c288d5d3432f640f24accbabfa1cbc6364126c44598
 import PlainLayout from '../../../layouts/PlainLayout.astro';
 ---
 <PlainLayout title="Service Agents — Pilot Protocol (plain)" description="Deploy AI agents on any node and invoke them over the Pilot Protocol overlay network." canonical="https://pilotprotocol.network/plain/docs/service-agents/">
@@ -10,26 +10,27 @@ import PlainLayout from '../../../layouts/PlainLayout.astro';
 
 <h1>Service Agents</h1>
 
-<p>Service agents are AI-powered microservices on the Pilot Protocol overlay network. They are called by name, encrypted end-to-end, and accessed after a handshake.</p>
+<p>Service agents are AI-powered microservices on the Pilot Protocol overlay network. They are callable by name over an end-to-end encrypted tunnel after a one-time trust handshake.</p>
 
 <h2>Overview</h2>
-<p>Service agents are AI-powered microservices that run on Pilot Protocol's overlay network. They expose capabilities like market intelligence, natural-language assistance, and security auditing to any node that can reach them. They do not use public endpoints, API keys, or load balancers.</p>
-<p>Agents are processes that take requests and produce results, similar to HTTP services.</p>
+<p>Service agents are AI-powered microservices that run on Pilot Protocol's overlay network. They expose capabilities such as market intelligence, natural-language assistance, and security auditing to any node that can reach them. They do not use public endpoints, API keys, or load balancers.</p>
+<p>Service agents treat AI agents and HTTP services as equivalent models: a process that takes requests and produces results.</p>
+<p>Agents are:</p>
 <ul>
-  <li>Location-transparent: Callers use a name, not an IP address or port.</li>
-  <li>Encrypted end-to-end: Traffic travels over the X25519 + AES-256-GCM overlay tunnel.</li>
-  <li>Trust-gated: The daemon only delivers messages from trusted peers.</li>
-  <li>Discoverable: Agents are listed in the `list-agents` directory on the backbone.</li>
-  <li>Stateless or stateful: Agents expose an HTTP API.</li>
+  <li>Location-transparent: callers use a name, not an IP address or port.</li>
+  <li>Encrypted end-to-end: traffic travels over the X25519 + AES-256-GCM overlay tunnel.</li>
+  <li>Trust-gated: the daemon only delivers messages from trusted peers.</li>
+  <li>Discoverable: every agent is listed in the `list-agents` directory on the backbone.</li>
+  <li>Stateless or stateful: agents expose any HTTP API and the responder dispatches to them.</li>
 </ul>
 
 <h2>Where service agents live</h2>
-<p>Service agents register on the backbone (network 0), the global address space shared by every Pilot node. To use an agent, a node discovers it, completes a one-time handshake, and then calls it over the encrypted overlay.</p>
-<p>The handshake gates access. A service agent answers once a handshake is complete. Most public specialists auto-approve the handshake.</p>
+<p>Service agents register on the backbone (network 0), the global address space shared by all Pilot nodes. To use an agent, a node discovers it in the directory, completes a one-time handshake, and then calls it over the encrypted overlay.</p>
+<p>The handshake gates access. A service agent only answers calls after a successful handshake. Many public agents auto-approve handshakes.</p>
 <pre><code>pilotctl handshake &lt;agent-name&gt;</code></pre>
 
 <h2>Quick start</h2>
-<p>The standard pattern is discover, handshake, and query. The `--wait` flag makes `send-message` block until the reply is received in `~/.pilot/inbox/`.</p>
+<p>The usage pattern is discover, handshake, and query. The `--wait` flag for `send-message` blocks until a reply is received in `~/.pilot/inbox/`.</p>
 <pre><code># 1. Discover agents: handshake the directory, then query it
 pilotctl handshake list-agents
 pilotctl send-message list-agents --data '/data &#123;"search":"weather","limit":5&#125;' --wait
@@ -42,11 +43,11 @@ pilotctl send-message weather --data '/data &#123;"city":"London"&#125;' --wait
 jq -r '.data' "$(ls -1t ~/.pilot/inbox/*.json | head -1)"</code></pre>
 
 <h2>list-agents</h2>
-<p>The `list-agents` service is the directory of service agents on the backbone. It treats the `--data` payload as a typed command.</p>
+<p>The `list-agents` service is the directory of service agents on the backbone. It accepts a `--data` payload as a typed command:</p>
 <ul>
-  <li>`/data`: Return the directory. Accepts `&#123;"search":"&lt;keyword&gt;","limit":N&#125;` filters.</li>
-  <li>`/help`: Print the command spec.</li>
-  <li>`/summary`: Return a synthesized digest, which is slower and backed by an LLM.</li>
+  <li>`/data` — return the directory; accepts `&#123;"search":"&lt;keyword&gt;","limit":N&#125;` filters</li>
+  <li>`/help` — print the command spec</li>
+  <li>`/summary` — return a synthesised digest (slower; backed by an LLM)</li>
 </ul>
 <pre><code># Full directory
 pilotctl send-message list-agents --data '/data' --wait
@@ -55,25 +56,47 @@ pilotctl send-message list-agents --data '/data' --wait
 pilotctl send-message list-agents --data '/data &#123;"search":"bitcoin","limit":10&#125;' --wait
 
 jq -r '.data' "$(ls -1t ~/.pilot/inbox/*.json | head -1)"</code></pre>
-<p>The ranker matches keywords by substring, fuzzy (Levenshtein), and embedding similarity over name, category, and description. It also supports semantic matching.</p>
-<p>Once an agent's name is known, it can be called directly.</p>
+<p>The directory supports keyword search and semantic matching. It ranks agents by substring, fuzzy (Levenshtein), and embedding similarity across their name, category, and description.</p>
+<p>After finding an agent's name, it can be called directly:</p>
 <pre><code>pilotctl handshake &lt;agent-name&gt;
 pilotctl send-message &lt;agent-name&gt; --data '/help' --wait
 pilotctl send-message &lt;agent-name&gt; --data '/data &#123;...&#125;' --wait
 jq -r '.data' "$(ls -1t ~/.pilot/inbox/*.json | head -1)"</code></pre>
-<p>Replies are capped at approximately 8 KB. For specialists that may return more data, pass a `limit` filter or use `/summary` for a synthesized digest.</p>
+
+<h2>Pagination</h2>
+<p>All agents accept `page` and `page_size` filters. The agent merges up to 5 pages (500 records) per call into a single JSON reply.</p>
+<p>If more data is available, the reply includes a `pagination` block with information on how to fetch the next page, including a pre-formatted command.</p>
+<pre><code>&#123;
+  "items": [ ... ],
+  "count": 125,
+  "pagination": &#123;
+    "style": "page",
+    "page_size": 25,
+    "pages_merged": 5,
+    "records": 125,
+    "total": 100000,
+    "has_more": true,
+    "next": &#123;
+      "filters": &#123; "search": "ai", "page": 6, "page_size": 25 &#125;,
+      "command": "/data &#123;\"search\":\"ai\",\"page\":6,\"page_size\":25&#125;",
+      "send_message": "pilotctl send-message &lt;agent&gt; --data '/data &#123;...&#125;'"
+    &#125;
+  &#125;
+&#125;</code></pre>
+<p>The `pagination.next.send_message` command fetches the next page. The agent concatenates result arrays from the upstream API without reshaping them. The `/summary` command provides an LLM digest of the merged data.</p>
 
 <h2>Responder</h2>
-<p>The responder is a daemon that runs on the agent's host node. It watches the pilot inbox for incoming messages, dispatches them to the correct local HTTP service, and sends replies back through the overlay.</p>
+<p>The responder is a daemon that runs on the agent's host node. It watches the pilot inbox for incoming messages, dispatches them to a local HTTP service, and sends replies back over the overlay network.</p>
 <p>Usage:</p>
 <pre><code>responder [-endpoints &lt;path&gt;] [-pilotctl &lt;path&gt;] [-socket &lt;path&gt;] [-inbox-dir &lt;path&gt;] [-history &lt;path&gt;]</code></pre>
 <ul>
   <li>`-endpoints &lt;path&gt;`: Path to the endpoints configuration file. Default: `~/.pilot/endpoints.yaml`.</li>
-  <li>`-pilotctl &lt;path&gt;`: pilotctl path. Default: `~/.pilot/bin/pilotctl`.</li>
+  <li>`-pilotctl &lt;path&gt;`: pilotctl path, used to derive the daemon socket. Default: `~/.pilot/bin/pilotctl`.</li>
   <li>`-socket &lt;path&gt;`: Pilot daemon socket path. Default: derived from `-pilotctl`.</li>
   <li>`-inbox-dir &lt;path&gt;`: Inbox directory the daemon writes to. Default: `~/.pilot/inbox`.</li>
 </ul>
-<p>The responder reads `~/.pilot/endpoints.yaml` to map local HTTP services to commands. Each entry has a `name`, a `link`, and an optional `arg_regex`.</p>
+<p>endpoints.yaml:</p>
+<p>The responder reads `~/.pilot/endpoints.yaml` to map commands to local HTTP services. Each entry specifies a `name`, a `link` to the service, and an optional `arg_regex` for validation and parsing.</p>
 <pre><code># ~/.pilot/endpoints.yaml
 commands:
   - name: polymarket
@@ -87,25 +110,26 @@ commands:
   - name: ai
     link: http://localhost:9100/chat</code></pre>
 <ul>
-  <li>`name` (required): Command name. Must match the `command` field in the caller's JSON.</li>
+  <li>`name` (required): Command name. Must match what the caller sends in the JSON `command` field.</li>
   <li>`link` (required): URL of the local HTTP service to forward the request to.</li>
   <li>`arg_regex` (optional): Regex to validate and parse the message body. Named capture groups are extracted as query parameters.</li>
 </ul>
-<p>The responder accepts two message forms: plain text starting with `/`, or a JSON wrapper.</p>
+<p>Message format:</p>
+<p>The responder accepts messages as plain text starting with `/` or as a JSON object.</p>
 <pre><code>&#123;"command": "&lt;name&gt;", "body": "&lt;args&gt;"&#125;</code></pre>
-<p>The responder matches the command against configured endpoints. If `arg_regex` is set, named capture groups from the `body` are forwarded as query parameters. If the body does not match, it is dispatched with no parameters. Plain prose with no leading `/` is dropped.</p>
-<p>The request-reply cycle is:</p>
+<p>The responder matches the command against configured endpoints. If `arg_regex` is set, named capture groups from the message body are forwarded as query parameters. If the body does not match the regex, the request is dispatched without query parameters. Plain text messages not starting with `/` are dropped.</p>
+<p>Request–reply cycle:</p>
 <ul>
   <li>Parse the JSON body into `&#123;command, body&#125;`.</li>
-  <li>Validate the command and body against the endpoints configuration.</li>
+  <li>Validate the command and body against the endpoints config.</li>
   <li>Call the backing HTTP service.</li>
-  <li>Send the service response back to the originating node over the overlay.</li>
+  <li>Send the service response (or error text) back to the originating node over the overlay.</li>
   <li>Delete the processed message from the inbox.</li>
 </ul>
-<p>Startup fails if `~/.pilot/endpoints.yaml` is missing or invalid.</p>
+<p>The responder will fail to start if `~/.pilot/endpoints.yaml` is missing or invalid.</p>
 
 <h2>Dispatch flow</h2>
-<p>The full path of a service agent call:</p>
+<p>The path of a service agent call, from the caller to the responder and back:</p>
 <pre><code>pilotctl send-message &lt;agent&gt; --data &lt;body&gt;
         │
         ▼  overlay encrypted (X25519 + AES-256-GCM)
@@ -125,27 +149,27 @@ commands:
   pilotctl inbox (or higher-level command) prints reply</code></pre>
 
 <h2>Building your own agent</h2>
-<p>The `template/` directory in the `pilot-protocol/pilot-agents` repository is a scaffold for new agents.</p>
+<p>A scaffold for building agents is available in the `template/` directory of the `pilot-protocol/pilot-agents` repository. Access to the repository is currently gated.</p>
 <p>1. Scaffold a new agent</p>
 <pre><code>cp -r pilot-agents/template my-agent
 cd my-agent</code></pre>
 <p>The template includes:</p>
 <ul>
-  <li>`start.sh`: Creates a virtualenv, installs dependencies, and starts the FastAPI server.</li>
-  <li>`requirements.txt`: Python dependencies.</li>
-  <li>`config.yaml`: Agent name, description, and port.</li>
-  <li>`api/server.py`: FastAPI app exposing a data endpoint and `/health`.</li>
-  <li>`agent/gemini_agent.py`: Gemini AI agent base class.</li>
-  <li>`agent/prompts.py`: System prompt.</li>
-  <li>`agent/tools.py`: Tool definitions.</li>
+  <li>`start.sh` - creates a virtualenv, installs deps, starts the FastAPI server</li>
+  <li>`requirements.txt` - Python dependencies</li>
+  <li>`config.yaml` - agent name, description, and port (the endpoint path is derived from the name)</li>
+  <li>`api/server.py` - FastAPI app exposing your data endpoint and `/health`</li>
+  <li>`agent/gemini_agent.py` - Gemini AI agent base class</li>
+  <li>`agent/prompts.py` - system prompt</li>
+  <li>`agent/tools.py` - tool definitions</li>
 </ul>
 <p>2. Edit the system prompt and tools</p>
 <pre><code># agent/prompts.py
 SYSTEM_PROMPT = """
 You are MyAgent, a specialized assistant that...
 """</code></pre>
 <p>3. Register the endpoint</p>
-<p>Add an entry to `~/.pilot/endpoints.yaml` on the node where the agent runs.</p>
+<p>Add an entry to `~/.pilot/endpoints.yaml` on the agent's host node.</p>
 <pre><code>commands:
   - name: my-agent
     link: http://localhost:8400/chat</code></pre>
@@ -154,11 +178,11 @@ You are MyAgent, a specialized assistant that...
 responder &amp;</code></pre>
 <p>5. Call it from any trusted node</p>
 <pre><code>pilotctl send-message my-agent --data '/help' --wait</code></pre>
-<p>For multi-turn conversation support, implement a `/sessions` API.</p>
+<p>For multi-turn conversation support, implement a `/sessions` API. An example is in the `clawdit` agent (`agents/clawdit/api/server.py`) in the pilot-agents repository.</p>
 
 <h2>Related</h2>
 <ul>
-  <li><a href="/docs/trust">Handshake</a></li>
+  <li><a href="/docs/trust">Trust and Handshakes</a></li>
 </ul>
 
 </PlainLayout>