diff --git a/.claude/skills/netclaw-docs/SKILL.md b/.claude/skills/netclaw-docs/SKILL.md index 3a22ecb..1719657 100644 --- a/.claude/skills/netclaw-docs/SKILL.md +++ b/.claude/skills/netclaw-docs/SKILL.md @@ -3,64 +3,93 @@ name: netclaw-docs description: Voice and style contract for netclaw.dev docs (Astro + Starlight, dark-mode only). The how-it-should-sound layer on top of CLAUDE.md's authoring process. --- -# netclaw.dev Docs Voice +# netclaw.dev docs voice -Write like you're explaining netclaw to a sharp coworker who knows Linux, Docker, and Slack but has never touched netclaw. Confident, casual-technical, code-first. Open by telling the reader what the thing does and when they'd reach for it, then get out of the way and show commands. The published pages earn trust by being correct and dense, not by being long. CLAUDE.md owns the process (page types, templates, the critique/humanizer pipeline); this skill owns the voice. +Write like you're explaining netclaw to a sharp coworker who knows Linux, Docker, and Slack but has never touched netclaw. Confident, casual-technical, code-first. Open by saying what the thing does and when you'd reach for it, then show commands. The pages earn trust by being correct and dense, not long. CLAUDE.md owns the process (page types, templates, critique and humanizer steps). This skill owns how the page reads. -## Do - -- **Open with the payload.** First sentence states what the feature does and when you'd use it. No "In this section" runway. See `getting-started/installation.md`: "Netclaw has two components..." -- **Use direct imperative voice.** "Run `netclaw status`." "Store tokens with `netclaw secrets`." Never "You can run" or "Users may wish to." -- **Show, don't tell.** A config block, command, or screenshot beats a paragraph. If CLI help already explains a flag, show the output instead of rewriting it. -- **Tables for anything with a shape** - flags, config fields (Field / Type / Default / Description), scopes, error→cause→fix, mode comparisons. Prose is for the one idea a table can't hold. -- **Name the #1 mistake.** The best pages flag the common footgun in bold ("This is the number one setup mistake"). Lead readers away from the trap. -- **Use Starlight callouts** `:::caution` / `:::note` for breaking changes, footguns, and security boundaries. Reserve them for things that bite. -- **Comment real screenshots in, stub honest TODOs out.** `` and `` are the house pattern; never ship fake "coming soon" prose. -- **Close with cross-links.** End with Related pages + Resources (2+ external links to upstream docs/RFCs). +## Relationship to aarons-voice and humanizer -## Don't +For anything under `src/content/docs/`, this skill supersedes `aarons-voice` and both humanizer skills. It already contains the parts of `aarons-voice` that apply to product docs, so don't load it alongside this one. Deliberately not carried over: bare URLs (use markdown links with descriptive text), the no-bold rule (see Bold), first-person "I", exclamation points, and the email phrase list. Humanizer runs on prose only. Where it disagrees with this skill, this skill wins. -- No filler: "It is important to note that", "This section describes", "The following table shows", "In this guide we will". Cut the sentence; keep the table. -- No hedging: "you may want to", "it is recommended that", "it should be noted". State it. -- No rule-of-three padding ("fast, simple, and powerful") or negative parallelism ("not X, but Y") used for rhythm. -- No inflated symbolism, no "seamless / robust / leverage / delve". Plain verbs. -- Don't explain the obvious. If a reader of this audience already knows it, delete it. -- Don't restate code in prose right after showing it. +## Two registers -## Punctuation +- **Reference** (`cli/`, `configuration/`, `channels/`, `deployment/`, `observability/`, most `guides/`): "you" plus imperative. netclaw is the subject of behavior sentences ("netclaw resolves symlinks first"). No "we", no "I". Opinions stop at "use X unless Y". Exact beats charming. +- **Narrative** (design philosophy, comparisons, tutorials, the landing page): "we" is fine for the project's decisions and opinions. State opinions as opinions. A little roughness is fine. Still never "I". -Use the spaced hyphen ` - ` (space-hyphen-space), not the em-dash (`—`), for parenthetical asides and appositives. As of 2026-07-22 the docs drop the em-dash to match the house writing voice, where the em-dash reads as an AI tell and ` - ` is the connector that replaces it. Outbound email and marketing already ban the em-dash, so there's now one rule everywhere - no docs-versus-email split to remember. +A page is one or the other. If a reference page needs a "why", link to the narrative page. -Style notes for ` - `: it joins clauses mid-sentence only, never after a terminal `.`, `?`, or `!`. If the prior sentence ended, start a new one. Don't stack more than one per sentence, and skip it where a period, colon, comma, or parentheses read cleaner. +## Do -Migration: use ` - ` in every new page and in any page you edit. The ~50 legacy pages still contain em-dashes (`security-model.md` alone has ~38) and stay as they are until you edit the page for another reason, or a dedicated sweep converts them. Some channel pages use the double-hyphen ` -- ` instead. Normalize those to ` - ` when you touch them. +- **Open with the payload.** First sentence says what the feature does and when you'd use it. +- **Lead guides with the answer.** Put an "I want / do this" table first where it fits, then explain. +- **Use the words people search for.** Titles and headings say "turn off", "stop asking", "disable", not internal names. +- **Name the thing.** "Slack, Discord, or Mattermost", not "your chat app". +- **Show, don't tell.** A config block, command or screenshot beats a paragraph. Tables for anything with a shape. Prose is for the one idea a table can't hold, and never restates the table. +- **Name the number one mistake**, once per section at most. +- **Use `:::caution` and `:::note`** for breaking changes, footguns and security boundaries only. +- **One concept page, one guide, one reference per topic.** Other pages link instead of re-explaining. +- **Stub honestly.** `` and ``. Never ship "coming soon" prose. -## Per-page-type quick reference +## Don't -- **CLI** (`cli/`): one-line intent → `## Usage` → options table → examples with a screenshot each → Related Commands. -- **Tutorial** (`getting-started/`): what we're building → Prerequisites (with versions) → numbered Steps with exact commands + expected output → Verify It Works → Next Steps. -- **Guide** (`guides/`): the problem in one line → Before You Begin → steps (call out conditional branches) → Troubleshooting (symptom / cause / fix). -- **Concept** (`architecture/`, `security/`): 2–3 paragraph mental model → key-concept sections → Limitations. -- **Configuration** (`configuration/`): what routes through this → minimal JSON block → full fields table (Field/Type/Default) → invalid-config behavior → diagnosing-errors table. -- **Integration** (`channels/`): transport in one line → Prerequisites → third-party setup (app/tokens/scopes, scopes table) → Configure netclaw → Access control → Verify it works → Troubleshooting. -- **Operations** (`deployment/`): when you need a non-default mode → modes table → per-mode config → restart/validation behavior → `netclaw doctor` troubleshooting. +- No filler ("It is important to note", "This page covers", "The following table shows") and no hedging ("you may want to", "it is recommended"). +- No "No X, no Y, no Z" more than once per page. No slogan triads or punchline fragments for rhythm ("That's unrealistic."). +- No screenshot captions shaped like `Label - fragment; fragment`. A caption is one sentence. +- Cut: "surface" as a noun, "just" as a minimiser, "walks through", "the full breakdown", "out of the box" (say "by default"), "first-class", "at a glance", "worth knowing". +- Don't explain what this audience already knows. Don't restate code in prose after showing it. +- Don't sand good sentences flat to chase a clean score. Fix the tell and keep the sentence. A page that passes every check and reads like a compliance form is worse than one with a stray dash. -## Terminology & product facts +## Punctuation -- **netclaw** is lowercase in running prose. CLI is `netclaw`; daemon is `netclawd`. -- Commands and flags go in backticks: `netclaw status`, `--channel beta`. Config keys too: `Search.Backend`, `Daemon.ExposureMode`. -- **`netclaw init`** = focused 5-step first-run setup (provider, identity, security posture, features) that also powers factory reset. Don't pile post-install config into it. -- **`netclaw config`** = the re-entrant, menu-driven dashboard for everything else (channels, search, exposure, webhooks, skills); safe to re-run, autosaves on completion. -- Cross-link slugs: `/cli//`, `/channels//`, `/configuration//`, `/deployment//`, `/security//`, `/getting-started//`. Trailing slash, lowercase. -- Config lives in `~/.netclaw/config/netclaw.json`; secrets in `~/.netclaw/config/secrets.json` (encrypted at rest, written via `netclaw secrets set`). Env override prefix is `NETCLAW_` with `__` path separators. +- **Contractions everywhere:** it's, don't, you'll, can't. Spell out "do not" or "cannot" only inside a `:::caution` or a security boundary statement, where the stiffness is the point. +- **No semicolons in prose.** Write two sentences. +- **No exclamation points.** No emoji, except status glyphs that mirror real CLI output. +- **Sentence-case headings:** `## Related pages`, `## Before you begin`. Proper nouns and command names keep their case. +- **No em dashes at all.** Pick the replacement by position: + - List separators and table cells: ` - ` or a colon. An empty cell is `-`. + - Headings: a colon (`### 401 Unauthorized: secret mismatch`). Heading edits change anchors, so check inbound links. + - Prose: rewrite with a period, colon or comma first. Keep ` - ` for a real aside, about once per paragraph at most, never after a terminal `.`, `?` or `!`. + - Don't search-and-replace. Forty em dashes turned into forty spaced hyphens is a new tic. Normalize ` -- ` the same way. + +## Bold + +Four uses only: UI labels the reader looks for, one warning per section, Symptom / Cause / Fix labels, and a term where it's defined. No bolded thesis sentences. No bold-label bullet lists where a table or subheadings would do. + +## Structure + +- Every page ends with exactly `## Related pages` (three to five internal links) then `## Resources` (two or more external links). No "Next steps", "See also" or "Keep reading". Tutorials may put `## What's next` before them. +- Every screenshot has the instruction before it and a one-sentence caption after it that says something the alt text doesn't. +- Page-type shapes: + - **CLI:** intent, `## Usage`, options table, examples with a screenshot each. + - **Tutorial:** goal, prerequisites with versions, numbered steps with expected output, verify it works. + - **Guide:** problem in one line, before you begin, steps, troubleshooting (Symptom / Cause / Fix). + - **Concept:** two or three paragraphs of mental model, key sections, limitations. + - **Configuration:** what routes through it, minimal JSON, fields table (Field / Type / Default), invalid-config behavior. + - **Integration:** transport in one line, prerequisites, third-party setup, configure netclaw, access control, verify, troubleshooting. + - **Operations:** when you need a non-default mode, modes table, per-mode config, restart behavior, `netclaw doctor`. + +## Terminology and product facts (0.27.1) + +- netclaw is lowercase in running prose. The CLI is `netclaw`, the daemon is `netclawd`. Commands, flags and config keys go in backticks. +- `netclaw init` is first-run bootstrap only: provider, identity, security posture, then health check. That's four steps for Personal and five for Team or Public, which add a feature-selection step. Channels, search, exposure, webhooks and skill sources live in `netclaw config`, the re-entrant menu dashboard. +- Default base directory is `~/.netclaw`, or `$NETCLAW_HOME` if set. Config is `config/netclaw.json`, secrets are `config/secrets.json` (written with `netclaw secrets set`). Env overrides use the `NETCLAW_` prefix with `__` between path segments. +- `Ctrl+Q` quits every TUI. Escape is per screen: it never quits the config, provider, model, init or chat screens (in chat it denies a pending approval), but it does quit from the `netclaw mcp` server list. State keys per screen from the key-handling code, never as a global rule. +- There is no `netclaw daemon restart`. The daemon restarts itself, after draining sessions, when `netclaw.json` changes. Nothing else triggers it, so secrets and environment changes need a manual restart: + - native: `netclaw daemon stop`, then `netclaw daemon start` + - systemd user unit: `systemctl --user restart netclaw` + - Docker: `docker restart ` (the image's entrypoint supervises the daemon) +- Screenshots and command output come from the release container (`ghcr.io/netclaw-dev/netclaw:`), never a host binary. No private hostnames in samples or screenshots. Use `llm.example.internal` or a named public provider. +- Cross-link slugs are lowercase with a trailing slash: `/cli//`, `/channels//`, `/configuration//`. ## Pre-ship checklist -- [ ] Frontmatter `title` + `description` set. -- [ ] Opens with what/when, not runway. No filler or hedge phrases. -- [ ] Reference material is in tables; code/screenshots carry the weight. -- [ ] netclaw lowercase; commands and config keys backticked; cross-links use correct trailing-slash slugs. -- [ ] No placeholder prose - only `` comments for gaps; all referenced screenshots exist. -- [ ] 2+ external resource links. -- [ ] Every runnable snippet was actually run against the current release. -- [ ] `/humanizer` pass done; `npm run build` passes. +- [ ] Frontmatter `title` and `description` set. Headings in sentence case and built from search words. +- [ ] Opens with what and when. Guides lead with the answer. No filler, hedges or cut-list phrases. +- [ ] No em dashes, ` -- `, or semicolons in prose. ` - ` within budget. Contractions used. +- [ ] Reference page has no "we". No page has "I". Bold only in the four permitted uses. +- [ ] Tables carry reference material and aren't restated in prose. +- [ ] Every screenshot has an instruction before it and a one-sentence caption after it, and the file exists. +- [ ] Ends with `## Related pages` then `## Resources` (2+ external links). +- [ ] Every runnable snippet was run against the release container. No private hostnames. +- [ ] No placeholder prose. Only `` comments for gaps. +- [ ] `humanizer:humanizer` pass done on prose only. `npm run build` passes. diff --git a/.github/workflows/deploy.yml b/.github/workflows/deploy.yml index d107a59..a507863 100644 --- a/.github/workflows/deploy.yml +++ b/.github/workflows/deploy.yml @@ -11,7 +11,8 @@ on: jobs: deploy: name: "Deploy to Cloudflare Pages" - runs-on: ubuntu-latest + # Pinned: the Playwright version in package-lock.json has no Chromium build for ubuntu-latest (26.04). + runs-on: ubuntu-24.04 environment: name: production url: https://netclaw.dev @@ -31,6 +32,25 @@ jobs: - name: Install dependencies run: npm ci + - name: Cache Playwright browser + id: playwright-cache + uses: actions/cache@v4 + with: + path: ~/.cache/ms-playwright + key: playwright-${{ runner.os }}-${{ hashFiles('package-lock.json') }} + + # Mermaid diagrams render at build time through a headless browser. Without it + # the build still succeeds but pages with diagrams ship with an empty body + # (scripts/check-rendered-pages.mjs fails the build when that happens). + # The browser revision follows the playwright version pinned in package-lock.json. + - name: Install Playwright browser + run: | + if [ "${{ steps.playwright-cache.outputs.cache-hit }}" = "true" ]; then + npx playwright install-deps chromium-headless-shell + else + npx playwright install --with-deps chromium-headless-shell + fi + - name: Build site run: npm run build env: diff --git a/.github/workflows/pr-validation.yml b/.github/workflows/pr-validation.yml index 3a590a5..7f02316 100644 --- a/.github/workflows/pr-validation.yml +++ b/.github/workflows/pr-validation.yml @@ -8,7 +8,8 @@ on: jobs: build-site: name: "Build Astro" - runs-on: ubuntu-latest + # Pinned: the Playwright version in package-lock.json has no Chromium build for ubuntu-latest (26.04). + runs-on: ubuntu-24.04 steps: - name: Checkout uses: actions/checkout@v4 @@ -22,12 +23,32 @@ jobs: - name: Install dependencies run: npm ci + - name: Cache Playwright browser + id: playwright-cache + uses: actions/cache@v4 + with: + path: ~/.cache/ms-playwright + key: playwright-${{ runner.os }}-${{ hashFiles('package-lock.json') }} + + # Mermaid diagrams render at build time through a headless browser. Without it + # the build still succeeds but pages with diagrams ship with an empty body + # (scripts/check-rendered-pages.mjs fails the build when that happens). + # The browser revision follows the playwright version pinned in package-lock.json. + - name: Install Playwright browser + run: | + if [ "${{ steps.playwright-cache.outputs.cache-hit }}" = "true" ]; then + npx playwright install-deps chromium-headless-shell + else + npx playwright install --with-deps chromium-headless-shell + fi + - name: Build site run: npm run build link-validation: name: "Link and URL Validation" - runs-on: ubuntu-latest + # Pinned: the Playwright version in package-lock.json has no Chromium build for ubuntu-latest (26.04). + runs-on: ubuntu-24.04 steps: - name: Checkout uses: actions/checkout@v4 @@ -41,6 +62,25 @@ jobs: - name: Install dependencies run: npm ci + - name: Cache Playwright browser + id: playwright-cache + uses: actions/cache@v4 + with: + path: ~/.cache/ms-playwright + key: playwright-${{ runner.os }}-${{ hashFiles('package-lock.json') }} + + # Mermaid diagrams render at build time through a headless browser. Without it + # the build still succeeds but pages with diagrams ship with an empty body + # (scripts/check-rendered-pages.mjs fails the build when that happens). + # The browser revision follows the playwright version pinned in package-lock.json. + - name: Install Playwright browser + run: | + if [ "${{ steps.playwright-cache.outputs.cache-hit }}" = "true" ]; then + npx playwright install-deps chromium-headless-shell + else + npx playwright install --with-deps chromium-headless-shell + fi + - name: Build site run: npm run build diff --git a/AGENTS.md b/AGENTS.md index 29b4b45..be1e61b 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -4,7 +4,7 @@ netclaw.dev documentation site — Astro 6.2 + Starlight, dark-mode only, deploy Two guides govern work here. Read both before writing or editing any doc page: -- **[CLAUDE.md](./CLAUDE.md)** — the authoring *process*: page types, per-type templates, the screenshot pipeline, the critique + `/humanizer` review steps, and build/preview commands. +- **[CLAUDE.md](./CLAUDE.md)** — the authoring *process*: page types, per-type templates, the screenshot pipeline, the critique + `humanizer:humanizer` review steps, and build/preview commands. - **[`.claude/skills/netclaw-docs/SKILL.md`](./.claude/skills/netclaw-docs/SKILL.md)** — the *voice and style* contract: the "less is more" house voice, punctuation policy, per-page-type quick reference, terminology, and the pre-ship checklist. CLAUDE.md owns *what* to write and *how to ship it*; the `netclaw-docs` skill owns *how it reads*. Use both. diff --git a/CLAUDE.md b/CLAUDE.md index 51815c4..867ade9 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -26,9 +26,9 @@ Read the target markdown stub in `src/content/docs/` to get the title and descri Before asking the user anything, collect available information: -1. **Netclaw source repo** — Clone `https://github.com/netclaw-dev/netclaw` (or locate a local clone). Key paths relative to that repo root: +1. **Netclaw source repo** — The local clone is at `~/repositories/netclaw-dev/netclaw` (or clone `https://github.com/netclaw-dev/netclaw`). Check out the release tag the docs target (`src/version.json`). Key paths relative to that repo root: - `src/Netclaw.Cli/` — CLI commands, flags, help text - - `src/Netclaw.Configuration/` — Config schemas, defaults, validation + - `src/Netclaw.Configuration/` — Config schemas, defaults, validation. The JSON schema is `src/Netclaw.Configuration/Schemas/netclaw-config.v1.schema.json` (webhook routes: `webhook-route.v1.schema.json` in the same folder) - `src/Netclaw.Daemon/` — Daemon architecture, startup, health checks - `src/Netclaw.Channels.Slack/` and `Netclaw.Channels.Discord/` — Channel integrations - `src/Netclaw.Security/` — Security model, ACLs, approval gates @@ -37,6 +37,7 @@ Before asking the user anything, collect available information: - `docs/adr/` — Architecture decision records (tool schemas, context discovery, progressive tool disclosure) - `docs/runbooks/` — Operational runbooks (background jobs, behavior debugging, daemon upgrades, memory health, subagents, tool approval gates) - `docs/spec/configuration.md` — Configuration specification + - **Warning:** `docs/spec/SPEC-004-cli-contract.md` and `docs/spec/configuration.md` are stale in places. They list commands and keys that were never implemented. When they disagree with the C# source, the JSON schema, or real `--help` output from the release container, the source, schema and `--help` win. - `CLAUDE.md`, `PROJECT_CONTEXT.md`, `CONTRIBUTING.md` — Project context and conventions 2. **Screenshots** — Check `screenshots/output/` for assets matching the page topic. Map them to the page. Screenshots are served at `/screenshots/output/`. 3. **Landing page** — `src/pages/index.astro` contains feature descriptions, architecture overview, and product positioning that inform doc content. @@ -144,9 +145,13 @@ netclaw -## Related Commands +## Related pages -- [`netclaw `](/cli//) — +- [`netclaw `](/cli//) - + +## Resources + +- []() - ``` #### Tutorial Template @@ -180,13 +185,21 @@ description: "" ... -## Verify It Works +## Verify it works -## Next Steps +## What's next + +- []() - + +## Related pages -- []() — +- []() - + +## Resources + +- []() - ``` #### Guide Template @@ -199,7 +212,7 @@ description: "" -## Before You Begin +## Before you begin - @@ -211,7 +224,15 @@ description: "" ### - +**Symptom:** ... **Cause:** ... **Fix:** ... + +## Related pages + +- []() - + +## Resources + +- []() - ``` #### Concept Template @@ -231,6 +252,14 @@ description: "" ## Limitations - + +## Related pages + +- []() - + +## Resources + +- []() - ``` ### Step 5: Screenshot Integration @@ -239,19 +268,20 @@ When including screenshots: - Use `![alt text](/screenshots/output/.png)` for static screenshots - Prefer `.png` over `.gif` unless the animation is essential to understanding -- Write a caption sentence below each screenshot explaining what it shows +- Put the instruction before the screenshot and a one-sentence caption after it that says something the alt text doesn't +- **Captures come from the release container** (`ghcr.io/netclaw-dev/netclaw:`), never a host-installed binary. A host build can be an unreleased prototype. No private hostnames in samples or screenshots. - If a screenshot shows a multi-step flow (like `init`), break it into individual step screenshots: `init-01-provider-list.png`, `init-02-security-posture.png`, etc. -Available screenshot sets: -- `init-*` — Full `netclaw init` wizard flow (10+ steps) -- `status.*` — System status display -- `doctor.*` — Health check diagnostics -- `stats.*` — Usage statistics -- `provider.*` / `provider-manager.png` — Provider management -- `model.*` / `model-manager.png` — Model management -- `mcp-tools.*` / `mcp-tools-*.png` — MCP tool server management -- `webhooks.*` / `webhooks-list.png` — Webhook configuration -- `reminder.*` / `reminder-list.png` — Reminder management +Available screenshot sets (in `screenshots/output/`): +- `init-*` — `netclaw init` wizard steps (provider, identity, security posture, health check, start-over) +- `config-*` — `netclaw config` dashboard and areas (channels, search, exposure, browser, telemetry, webhooks, skills, workspaces) +- `status.*`, `doctor.*`, `stats.*` — system status, health checks, usage statistics +- `provider.*` / `provider-manager.png`, `model.*` / `model-manager.png` — provider and model management +- `mcp-tools.*` / `mcp-tools-*.png` — MCP server management +- `approvals.*` / `approvals-*.png` — tool approvals list and TUI +- `webhooks.*` / `webhooks-list.png`, `reminder.*` / `reminder-list.png` +- `chat-*`, `single-shot-*` — interactive chat and single-shot CLI runs +- `slack-*`, `discord-*`, `mattermost-*` — third-party channel setup and conversations (kept as is) ### Step 6: Critique (3 parallel agents) @@ -280,7 +310,7 @@ After writing the draft, spawn three review agents in parallel. Pass each the fu ### Step 7: Humanizer Pass -Run `/humanizer` on the drafted page. This is mandatory — no page ships without it. The humanizer catches AI writing patterns that slip through manual review: over-explaining, passive voice, hedge words, repetitive structure, and generic filler that makes documentation feel soulless. +Run `humanizer:humanizer` (the newer, rewrite-oriented plugin skill, not the older `humanizer`) on the drafted page. It runs on prose paragraphs only, never on tables, code blocks or frontmatter. Where it disagrees with the `netclaw-docs` skill, the skill wins (for example, ignore suggestions to add first person or tangents on reference pages). This is mandatory — no page ships without it. The humanizer catches AI writing patterns that slip through manual review: over-explaining, passive voice, hedge words, repetitive structure, and generic filler that makes documentation feel soulless. ### Step 8: Revise @@ -304,6 +334,8 @@ Before declaring the page done: - [ ] At least 2 external links to relevant resources - [ ] No robotic phrasing or unnecessary hedging - [ ] `npm run build` passes with no errors +- [ ] Screenshots and command output come from the release container, not a host binary +- [ ] No private hostnames in samples or screenshots - [ ] **Every runnable example actually runs.** Any `docker run`, `docker compose`, or shell snippet a reader is told to execute MUST be run locally against the current release first and verified crash-free — the container reaches `healthy` and `docker logs` shows no fatal error / `Aborted (core dumped)`. Never ship a command you haven't executed. (This is how the stale `Daemon.Host=0.0.0.0` Docker example shipped a guaranteed startup crash.) ## Autonomous Doc Writing Mode @@ -316,13 +348,13 @@ When invoked with a page slug (e.g., `cli/status`), run the full pipeline withou 4. Run `npm run build` to verify 5. Stage and commit the single page with message: `docs: write
/` -The netclaw source repo is expected at `~/repositories/stannardlabs/netclaw/`. If not found, check common locations or skip source-dependent content. +The netclaw source repo is expected at `~/repositories/netclaw-dev/netclaw`. If not found, check common locations or skip source-dependent content. ## Build & Preview ```bash npm run dev -- --host 0.0.0.0 # Dev server with Tailscale access -npm run build # Production build (43 pages) +npm run build # Production build (165 pages built) ``` ## Project Structure diff --git a/astro.config.mjs b/astro.config.mjs index 1e01f94..4d29df7 100644 --- a/astro.config.mjs +++ b/astro.config.mjs @@ -4,8 +4,13 @@ import rehypeMermaid from 'rehype-mermaid'; export default defineConfig({ site: 'https://netclaw.dev', + // Mirrored in public/_redirects (301 on Cloudflare Pages). These also work in `astro preview`. + redirects: { + '/guides/custom-subagents': '/skills/custom-subagents/', + '/guides/connecting-slack': '/channels/slack/', + }, markdown: { - rehypePlugins: [[rehypeMermaid, { strategy: 'inline-svg', dark: true }]], + rehypePlugins: [[rehypeMermaid, { strategy: 'inline-svg', dark: true, mermaidConfig: { htmlLabels: false, flowchart: { htmlLabels: false, wrappingWidth: 280 } } }]], }, vite: { server: { @@ -41,18 +46,24 @@ export default defineConfig({ items: [ { label: 'Installation', slug: 'getting-started/installation' }, { label: 'Quickstart', slug: 'getting-started/quickstart' }, - { label: 'Your First Conversation', slug: 'getting-started/first-conversation' }, + { label: 'Your first conversation', slug: 'getting-started/first-conversation' }, + { label: 'Build your first assistant', slug: 'getting-started/first-assistant' }, ], }, { - label: 'Architecture & Philosophy', + label: 'Guides', items: [ - { label: 'Design Philosophy', slug: 'architecture/design-philosophy' }, - { label: 'Overview', slug: 'architecture/overview' }, - { label: 'Sessions & Input Model', slug: 'architecture/sessions' }, - { label: 'Context Structure', slug: 'architecture/context-structure' }, - { label: 'Memory Model', slug: 'architecture/memory-model' }, - { label: 'Security Model', slug: 'architecture/security-model' }, + { label: 'Reduce or turn off approval prompts', slug: 'guides/reduce-approval-prompts' }, + { label: 'Run unattended', slug: 'guides/unattended-runs' }, + { label: 'Connect an OAuth MCP server', slug: 'guides/oauth-mcp-servers' }, + { label: 'Run netclaw with Mattermost on Docker Compose', slug: 'guides/mattermost-docker-compose' }, + { label: 'Upgrade netclaw', slug: 'guides/upgrading' }, + { label: 'Back up, restore and move netclaw', slug: 'guides/backup-and-restore' }, + { label: 'Use the CLI from another machine', slug: 'guides/pairing-remote-devices' }, + { label: 'MCP tool permissions', slug: 'guides/mcp-tool-permissions' }, + { label: 'Scripting headless sessions', slug: 'guides/scripting-sessions' }, + { label: 'Images, audio and other file types', slug: 'guides/multimodal-files' }, + { label: 'Migrating model configuration', slug: 'guides/migrating-model-config' }, ], }, { @@ -65,42 +76,56 @@ export default defineConfig({ ], }, { - label: 'Skills', + label: 'Skills & Agents', items: [ { label: 'Overview', slug: 'skills/overview' }, - { label: 'Native Manifest & Sync', slug: 'skills/native-manifest' }, - { label: 'Bundling Resources with Skills', slug: 'skills/bundled-resources' }, - { label: 'Skill Server', slug: 'skills/skill-server' }, + { label: 'Native manifest & sync', slug: 'skills/native-manifest' }, + { label: 'Bundling resources with skills', slug: 'skills/bundled-resources' }, + { label: 'Custom subagents', slug: 'skills/custom-subagents' }, + { label: 'Skill feeds', slug: 'skills/skill-feeds' }, + { label: 'External skills', slug: 'skills/external-skills' }, + { label: 'Skill server', slug: 'skills/skill-server' }, { label: 'skillserver CLI', slug: 'skills/skillserver-cli' }, - { label: 'External Skills', slug: 'skills/external-skills' }, - { label: 'Skill Feeds', slug: 'skills/skill-feeds' }, + { label: 'Distributing subagents', slug: 'skills/distributing-subagents' }, + ], + }, + { + label: 'Configuration', + items: [ + { label: 'Managed providers', slug: 'configuration/managed-providers' }, + { label: 'Self-hosted providers', slug: 'configuration/self-hosted-providers' }, + { label: 'Models', slug: 'configuration/models' }, + { label: 'Memory and embeddings', slug: 'configuration/memory' }, + { label: 'Search providers', slug: 'configuration/search-providers' }, + { label: 'MCP servers', slug: 'configuration/mcp-servers' }, + { label: 'Webhooks', slug: 'configuration/webhooks' }, + { label: 'Reminders', slug: 'configuration/reminders' }, + { label: 'Data retention', slug: 'configuration/data-retention' }, + { label: 'Configuration reference', slug: 'configuration/reference' }, ], }, { label: 'Security', items: [ - { label: 'Security Model', slug: 'security/security-model' }, + { label: 'Security model', slug: 'security/security-model' }, + { label: 'How tool approvals work', slug: 'security/approvals' }, { label: 'Hardening', slug: 'security/hardening' }, - { label: 'Secrets Management', slug: 'security/secrets' }, + { label: 'Secrets management', slug: 'security/secrets' }, ], }, { - label: 'Configuration', + label: 'Deployment', items: [ - { label: 'Managed Providers', slug: 'configuration/managed-providers' }, - { label: 'Self-Hosted Providers', slug: 'configuration/self-hosted-providers' }, - { label: 'Models', slug: 'configuration/models' }, - { label: 'Search Providers', slug: 'configuration/search-providers' }, - { label: 'MCP Servers', slug: 'configuration/mcp-servers' }, - { label: 'Webhooks', slug: 'configuration/webhooks' }, - { label: 'Reminders', slug: 'configuration/reminders' }, + { label: 'Docker', slug: 'deployment/docker' }, + { label: 'systemd', slug: 'deployment/systemd' }, + { label: 'Exposure modes', slug: 'deployment/exposure-modes' }, ], }, { label: 'Notifications & Observability', items: [ - { label: 'Operational Alerts', slug: 'observability/operational-alerts' }, - { label: 'Health Checks', slug: 'observability/health-checks' }, + { label: 'Operational alerts', slug: 'observability/operational-alerts' }, + { label: 'Health checks', slug: 'observability/health-checks' }, { label: 'OpenTelemetry', slug: 'observability/opentelemetry' }, ], }, @@ -115,9 +140,12 @@ export default defineConfig({ { label: 'status', slug: 'cli/status' }, { label: 'doctor', slug: 'cli/doctor' }, { label: 'stats', slug: 'cli/stats' }, + { label: 'daemon', slug: 'cli/daemon' }, + { label: 'update', slug: 'cli/update' }, { label: 'provider', slug: 'cli/provider' }, { label: 'model', slug: 'cli/model' }, - { label: 'mcp-tools', slug: 'cli/mcp-tools' }, + { label: 'memory', slug: 'cli/memory' }, + { label: 'mcp', slug: 'cli/mcp-tools' }, { label: 'webhooks', slug: 'cli/webhooks' }, { label: 'reminder', slug: 'cli/reminder' }, { label: 'skill', slug: 'cli/skill' }, @@ -126,27 +154,30 @@ export default defineConfig({ ], }, { - label: 'Deployment', + label: 'Reference', items: [ - { label: 'Docker', slug: 'deployment/docker' }, - { label: 'systemd', slug: 'deployment/systemd' }, - { label: 'Exposure Modes', slug: 'deployment/exposure-modes' }, + { label: 'Slash commands', slug: 'reference/slash-commands' }, + { label: 'Built-in tools', slug: 'reference/tools' }, + { label: 'Data directory layout', slug: 'reference/data-directory' }, + { label: 'Glossary', slug: 'reference/glossary' }, ], }, { - label: 'Guides', + label: 'Architecture & Philosophy', items: [ - { label: 'MCP Tool Permissions', slug: 'guides/mcp-tool-permissions' }, - { label: 'Migrating Model Configuration', slug: 'guides/migrating-model-config' }, - { label: 'Custom Subagents', slug: 'guides/custom-subagents' }, - { label: 'Images, Audio, and Other File Types', slug: 'guides/multimodal-files' }, - { label: 'Pairing Remote Devices', slug: 'guides/pairing-remote-devices' }, - { label: 'Scripting Headless Sessions', slug: 'guides/scripting-sessions' }, + { label: 'Design philosophy', slug: 'architecture/design-philosophy' }, + { label: 'Overview', slug: 'architecture/overview' }, + { label: 'Sessions & input model', slug: 'architecture/sessions' }, + { label: 'Context structure', slug: 'architecture/context-structure' }, + { label: 'Memory model', slug: 'architecture/memory-model' }, + { label: 'Security model', slug: 'architecture/security-model' }, ], }, { label: 'Troubleshooting', items: [ + { label: 'Troubleshooting by symptom', slug: 'troubleshooting/common-problems' }, + { label: 'FAQ', slug: 'troubleshooting/faq' }, { label: 'llama.cpp', slug: 'troubleshooting/llama-cpp' }, { label: 'vLLM', slug: 'troubleshooting/vllm' }, ], @@ -158,6 +189,7 @@ export default defineConfig({ { label: 'vs. Hermes Agent', slug: 'comparisons/hermes-agent' }, ], }, + { label: 'Changelog', link: '/changelog' }, ], }), ], diff --git a/docker-compose.screenshots.yml b/docker-compose.screenshots.yml index e97cd0b..716532e 100644 --- a/docker-compose.screenshots.yml +++ b/docker-compose.screenshots.yml @@ -1,74 +1,66 @@ -services: - ollama: - image: ollama/ollama:latest - restart: unless-stopped - ports: - - "11434:11434" - volumes: - - screenshot-ollama-data:/root/.ollama +# Base of the docs capture / verification stack. Do not run this file directly: +# use screenshots/stack.sh or screenshots/capture.sh, which assemble it with the +# overlays under screenshots/compose/ and set the variables below. +# +# base netns + netclawd (no model config: first-run state) + capture +# screenshots/compose/configured.yml adds Models.Definitions/Roles, the demo MCP server +# screenshots/compose/ollama.yml adds the hermetic Ollama inference backend +# +# Every service shares ONE network namespace (the `netns` container) so the daemon, +# the CLI inside the capture container and any helper all reach each other on +# 127.0.0.1, exactly like a real user's machine. Nothing is published to the host. +# No container_name and no host ports: run as many projects side by side as you like +# (COMPOSE_PROJECT_NAME=dr04 ...). - ollama-init: +x-netclaw-image: &netclaw-image ${NETCLAW_IMAGE:-ghcr.io/netclaw-dev/netclaw:${NETCLAW_VERSION:?set by screenshots/stack.sh}} + +services: + netns: + # Holds the shared network namespace and nothing else. image: curlimages/curl:8.13.0 - entrypoint: ["/bin/sh", "/usr/local/bin/ollama-init.sh"] + entrypoint: ["sleep", "infinity"] + stop_grace_period: 1s + + netclawd: + image: *netclaw-image + network_mode: "service:netns" environment: - OLLAMA_MODEL: ${OLLAMA_MODEL:-qwen2:0.5b} - depends_on: - - ollama + NETCLAW_Daemon__Port: "5199" + NETCLAW_Daemon__DisableSelfUpdate: "true" volumes: - - ./screenshots/scripts/ollama-init.sh:/usr/local/bin/ollama-init.sh:ro - - mcp-demo: - build: ./screenshots/mcp-server - expose: - - "8080" + # The daemon runs as the non-root `netclaw` user; its home is /home/netclaw/.netclaw. + - netclaw-home:/home/netclaw/.netclaw healthcheck: - test: ["CMD", "python", "-c", "import socket; s=socket.create_connection(('127.0.0.1',8080),2); s.close()"] - interval: 5s + test: ["CMD", "curl", "-sf", "http://127.0.0.1:5199/api/health/ready"] + interval: 3s timeout: 5s - retries: 10 + retries: 40 start_period: 5s + stop_grace_period: 10s - netclawd: - # The published image is `netclaw` (not `netclawd`). Pinned to a release tag - # via NETCLAW_VERSION so screenshots track a known build; capture.sh sources - # NETCLAW_VERSION from src/version.json. Override NETCLAW_IMAGE to use :dev. - image: ${NETCLAW_IMAGE:-ghcr.io/netclaw-dev/netclaw:${NETCLAW_VERSION:-0.22.1}} - ports: - - "${NETCLAW_PORT:-5299}:5199" + # VHS + the netclaw CLI from the same release image as the daemon. Only started by + # `docker compose run` (profile `capture`); see screenshots/capture/Dockerfile. + capture: + profiles: ["capture"] + image: netclaw-docs-capture:${NETCLAW_IMAGE_TAG:?set by screenshots/stack.sh} + build: + context: ./screenshots/capture + args: + NETCLAW_IMAGE: *netclaw-image + network_mode: "service:netns" depends_on: - ollama-init: - condition: service_completed_successfully - mcp-demo: + netclawd: condition: service_healthy environment: - # NOTE (Phase 5): 0.22.1's `local` exposure mode binds loopback and aborts on a - # non-loopback Daemon.Host. The old host-reachability model here (Host=0.0.0.0 + - # published port + host-side `curl`/`netclaw` in capture.sh) crashes under 0.22.1. - # The Phase 5 re-shoot must drive the CLI via `docker exec` (loopback-local) or move - # to reverse-proxy mode + Daemon.TrustedProxies, and re-validate end to end. + HOST_UID: ${HOST_UID:-1000} + HOST_GID: ${HOST_GID:-1000} NETCLAW_Daemon__Port: "5199" - NETCLAW_Daemon__DisableSelfUpdate: "true" - NETCLAW_Providers__local-ollama__Type: ollama - NETCLAW_Providers__local-ollama__Endpoint: http://ollama:11434 - NETCLAW_Models__Main__Provider: local-ollama - NETCLAW_Models__Main__ModelId: ${OLLAMA_MODEL:-qwen2:0.5b} - NETCLAW_Models__Fallback__Provider: local-ollama - NETCLAW_Models__Fallback__ModelId: ${OLLAMA_MODEL:-qwen2:0.5b} - NETCLAW_Models__Compaction__Provider: local-ollama - NETCLAW_Models__Compaction__ModelId: ${OLLAMA_MODEL:-qwen2:0.5b} - NETCLAW_McpServers__demo-utilities__Transport: http - NETCLAW_McpServers__demo-utilities__Url: http://mcp-demo:8080/mcp - NETCLAW_McpServers__demo-utilities__Enabled: "true" + NETCLAW_CONTAINER_SUPERVISOR: entrypoint volumes: - # 0.22.1 runs as the non-root `netclaw` user; its home is /home/netclaw, not /root. - - screenshot-netclaw-home:/home/netclaw/.netclaw - healthcheck: - test: ["CMD", "curl", "-sf", "http://127.0.0.1:5199/api/health/ready"] - interval: 5s - timeout: 5s - retries: 30 - start_period: 10s + - netclaw-home:/home/netclaw/.netclaw + - ./screenshots/tapes:/tapes:ro + - ./screenshots/scripts:/work/screenshots/scripts:ro + - ${CAPTURE_OUT_DIR:-./screenshots/output}:/out volumes: - screenshot-ollama-data: - screenshot-netclaw-home: + netclaw-home: diff --git a/examples/mattermost-compose/.env.example b/examples/mattermost-compose/.env.example new file mode 100644 index 0000000..84bbcd5 --- /dev/null +++ b/examples/mattermost-compose/.env.example @@ -0,0 +1,14 @@ +# Copy to .env and fill in. Compose reads .env automatically. + +# Postgres password for the mattermost database user. +POSTGRES_PASSWORD=change-me-to-a-long-random-string + +# Where you open Mattermost in a browser. Use your real https URL later. +MATTERMOST_SITE_URL=http://localhost:8065 + +# Inference: any OpenAI-compatible endpoint (llama.cpp, vLLM, ...). The address works with or without a /v1 suffix. +# configure-netclaw.sh reads these. The compose file doesn't use them. +INFERENCE_BASE_URL=http://llm.example.internal:8080 +INFERENCE_MODEL_ID=your-model-id + +# provision.sh appends MM_CHANNEL_ID and MM_USER_ID here. Don't set them by hand. diff --git a/examples/mattermost-compose/.gitignore b/examples/mattermost-compose/.gitignore new file mode 100644 index 0000000..2cf01c2 --- /dev/null +++ b/examples/mattermost-compose/.gitignore @@ -0,0 +1,2 @@ +.env +.bot-token diff --git a/examples/mattermost-compose/README.md b/examples/mattermost-compose/README.md new file mode 100644 index 0000000..319442c --- /dev/null +++ b/examples/mattermost-compose/README.md @@ -0,0 +1,26 @@ +# netclaw + Mattermost on Docker Compose + +A private assistant stack: netclaw 0.28.2, Mattermost Team Edition 11.11.1 and +Postgres 17.11 on one Compose network. The full walkthrough, with expected +output at every step, is +[Run netclaw with Mattermost on Docker Compose](https://netclaw.dev/guides/mattermost-docker-compose/). + +| File | What it does | +|------|--------------| +| `compose.yaml` | The three services. netclaw publishes no ports and runs read-only with all capabilities dropped. | +| `.env.example` | Copy to `.env`. Passwords and the inference endpoint. | +| `provision.sh` | Creates the admin, a team, a private channel, one member and one bot with a token. Needs bash and curl. | +| `configure-netclaw.sh` | Adds the inference provider, sets the channel audience, stores the bot token in netclaw's vault and restarts. | + +```bash +cp .env.example .env && chmod 600 .env # edit POSTGRES_PASSWORD, INFERENCE_BASE_URL, INFERENCE_MODEL_ID +docker compose up -d --wait +./provision.sh # prints the generated passwords once +./configure-netclaw.sh +``` + +Then sign in at as the member user and mention the bot +in the `assistant` channel. + +Everything lives in named volumes. `docker compose down` keeps your data. +`docker compose down -v` deletes it. diff --git a/examples/mattermost-compose/compose.yaml b/examples/mattermost-compose/compose.yaml new file mode 100644 index 0000000..c9a3e90 --- /dev/null +++ b/examples/mattermost-compose/compose.yaml @@ -0,0 +1,99 @@ +# netclaw + Mattermost (Team Edition) + Postgres on one private network. +# Only Mattermost's web UI is published, and only on the host's loopback. + +services: + postgres: + image: postgres:17.11-alpine + restart: unless-stopped + environment: + POSTGRES_DB: mattermost + POSTGRES_USER: mattermost + POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:?Set POSTGRES_PASSWORD in .env} + volumes: + - postgres-data:/var/lib/postgresql/data + healthcheck: + test: ["CMD-SHELL", "pg_isready -U mattermost -d mattermost"] + interval: 5s + timeout: 5s + retries: 12 + networks: [chat] + + mattermost: + image: mattermost/mattermost-team-edition:11.11.1 + restart: unless-stopped + depends_on: + postgres: + condition: service_healthy + environment: + MM_SQLSETTINGS_DRIVERNAME: postgres + MM_SQLSETTINGS_DATASOURCE: postgres://mattermost:${POSTGRES_PASSWORD}@postgres:5432/mattermost?sslmode=disable&connect_timeout=10 + MM_SERVICESETTINGS_SITEURL: ${MATTERMOST_SITE_URL:-http://localhost:8065} + # Bots and personal access tokens are off by default. netclaw needs both. + MM_SERVICESETTINGS_ENABLEBOTACCOUNTCREATION: "true" + MM_SERVICESETTINGS_ENABLEUSERACCESSTOKENS: "true" + # Nobody can self-register. Accounts come from provision.sh or an admin. + MM_TEAMSETTINGS_ENABLEOPENSERVER: "false" + # Local mode: lets the healthcheck use mmctl over a container-internal socket. + MM_SERVICESETTINGS_ENABLELOCALMODE: "true" + ports: + - "127.0.0.1:8065:8065" # host loopback only + volumes: + - mattermost-config:/mattermost/config + - mattermost-data:/mattermost/data + - mattermost-plugins:/mattermost/plugins + security_opt: + - no-new-privileges:true + healthcheck: + # The image has no shell or curl, so use mmctl. + test: ["CMD", "/mattermost/bin/mmctl", "--local", "system", "status"] + interval: 10s + timeout: 5s + retries: 18 + start_period: 30s + networks: [chat] + + netclaw: + image: ghcr.io/netclaw-dev/netclaw:0.27.1 + restart: unless-stopped + depends_on: + mattermost: + condition: service_healthy + # No ports: the daemon stays in the default `local` exposure mode (loopback). + # You drive it with `docker compose exec`, and Mattermost never calls it. + environment: + # The daemon is a single-file .NET binary that unpacks itself on start. + # The root filesystem is read-only, so unpack into a small exec tmpfs. + DOTNET_BUNDLE_EXTRACT_BASE_DIR: /var/netclaw-bundle + # Mattermost. The bot token is NOT here. It goes in netclaw's vault. + NETCLAW_Mattermost__Enabled: "true" + NETCLAW_Mattermost__ServerUrl: http://mattermost:8065 + NETCLAW_Mattermost__MentionOnly: "true" + NETCLAW_Mattermost__AllowDirectMessages: "true" + # Allow-lists live here and only here. Env entries merge with netclaw.json + # lists by position, so never set the same list in both places. + # Blank until provision.sh writes the IDs to .env. + NETCLAW_Mattermost__AllowedChannelIds__0: ${MM_CHANNEL_ID:-} + NETCLAW_Mattermost__AllowedUserIds__0: ${MM_USER_ID:-} + volumes: + - netclaw-home:/home/netclaw/.netclaw + # Hardening + user: "1654:1654" # start as the netclaw user, so the entrypoint never needs root + read_only: true # root filesystem is immutable + tmpfs: + - /tmp:rw,noexec,nosuid,size=64m # scratch space only + - /var/netclaw-bundle:rw,exec,nosuid,size=128m,uid=1654,gid=1654 # the unpacked daemon (43 MB), rebuilt on every start + cap_drop: [ALL] # no Linux capabilities + security_opt: + - no-new-privileges:true # no setuid escalation + pids_limit: 256 # fork-bomb ceiling + networks: [chat] + +volumes: + postgres-data: + mattermost-config: + mattermost-data: + mattermost-plugins: + netclaw-home: + +networks: + chat: diff --git a/examples/mattermost-compose/configure-netclaw.sh b/examples/mattermost-compose/configure-netclaw.sh new file mode 100755 index 0000000..b2b28b2 --- /dev/null +++ b/examples/mattermost-compose/configure-netclaw.sh @@ -0,0 +1,40 @@ +#!/usr/bin/env bash +# Finish wiring netclaw after provision.sh: apply the IDs from .env, add the +# inference provider, set the channel audience, and store the bot token in +# netclaw's encrypted vault. Run it once. +set -euo pipefail +cd "$(dirname "$0")" + +[ -f .bot-token ] || { echo "Run ./provision.sh first." >&2; exit 1; } +set -a; . ./.env; set +a +: "${MM_CHANNEL_ID:?missing from .env}" "${INFERENCE_BASE_URL:?missing from .env}" "${INFERENCE_MODEL_ID:?missing from .env}" + +nc() { docker compose exec -T netclaw netclaw "$@"; } + +# Recreate netclaw so it picks up MM_CHANNEL_ID and MM_USER_ID from .env. +docker compose up -d --wait netclaw + +nc provider add main openai-compatible --endpoint "$INFERENCE_BASE_URL" +nc model set main main "$INFERENCE_MODEL_ID" + +# Personal posture (what `netclaw init` writes for one operator) and a personal +# audience for the private channel, so the agent can ask to run shell commands. +# Each command still needs your approval. Editing netclaw.json makes the daemon +# restart itself. +docker compose exec -T -e CH="$MM_CHANNEL_ID" netclaw sh -c ' + f=$HOME/.netclaw/config/netclaw.json + jq --arg c "$CH" " + .Security.DeploymentPosture = \"Personal\" + | .Security.ShellExecutionMode = \"HostAllowed\" + | .Security.StrictDefaults = true + | .Tools.ShellMode = \"HostAllowed\" + | .Mattermost.ChannelAudiences[\$c] = \"personal\"" "$f" > /tmp/netclaw.json + cat /tmp/netclaw.json > "$f"; rm -f /tmp/netclaw.json' + +# Token goes in over stdin, so it never appears in a process list or in .env. +docker compose exec -T netclaw sh -c 'netclaw secrets set Mattermost.BotToken "$(cat)"' < .bot-token + +# Secrets changes need a restart. +docker compose restart netclaw +docker compose up -d --wait netclaw +nc status diff --git a/examples/mattermost-compose/provision.sh b/examples/mattermost-compose/provision.sh new file mode 100755 index 0000000..fffb176 --- /dev/null +++ b/examples/mattermost-compose/provision.sh @@ -0,0 +1,105 @@ +#!/usr/bin/env bash +# Provision a fresh Mattermost: admin, team, private channel, one human user, +# one bot account with an access token. Needs only bash and curl. +# +# ./provision.sh # Mattermost at http://localhost:8065 +# MM_URL=https://chat.example.com ./provision.sh +# +# Writes MM_CHANNEL_ID and MM_USER_ID to .env (compose reads them), the bot +# token to .bot-token (mode 600), and prints the generated passwords once. +# Passwords, the admin session token and request bodies reach curl over stdin +# or a private temp file, never on its command line, so `ps` can't show them. +set -euo pipefail +umask 077 # every file this script creates is readable by you only +cd "$(dirname "$0")" + +MM_URL="${MM_URL:-http://localhost:8065}" +TEAM_NAME="${TEAM_NAME:-home}" +CHANNEL_NAME="${CHANNEL_NAME:-assistant}" +BOT_NAME="${BOT_NAME:-netclaw}" +ADMIN_USER="${ADMIN_USER:-admin}" +ADMIN_EMAIL="${ADMIN_EMAIL:-admin@example.com}" +HUMAN_USER="${HUMAN_USER:-alice}" +HUMAN_EMAIL="${HUMAN_EMAIL:-alice@example.com}" + +# The values are pasted into JSON below, so keep them to plain characters. +for v in TEAM_NAME CHANNEL_NAME BOT_NAME ADMIN_USER HUMAN_USER; do + [[ "${!v}" =~ ^[a-z0-9_-]+$ ]] || { echo "$v must be lowercase letters, digits, - or _." >&2; exit 1; } +done +for v in ADMIN_EMAIL HUMAN_EMAIL; do + [[ "${!v}" =~ ^[A-Za-z0-9._+-]+@[A-Za-z0-9.-]+$ ]] || { echo "$v doesn't look like an email address." >&2; exit 1; } +done + +[ -e .bot-token ] && { echo "Refusing to run twice: .bot-token already exists." >&2; exit 1; } + +rand() { od -An -N12 -tx1 /dev/urandom | tr -d ' \n'; } +# First string value for a key in compact JSON. Mattermost returns compact JSON. +field() { sed -n "s/.*\"$1\":\"\([^\"]*\)\".*/\1/p" | head -n1; } +# Stop if an ID came back empty, so a changed response shape fails loudly. +need() { [ -n "$2" ] || { echo "Mattermost returned no $1. Run 'docker compose down -v' and start over." >&2; exit 1; }; } + +ADMIN_PASS="$(rand)" +HUMAN_PASS="$(rand)" + +# The admin token lives in this private file and curl reads it with -H @file. +HDR="$(mktemp)" +trap 'rm -f "$HDR"' EXIT + +api() { # api METHOD PATH [JSON] (sends the admin token once login has run) + local args=(-sS --fail-with-body -X "$1" "$MM_URL/api/v4$2" -H 'Content-Type: application/json') + [ ! -s "$HDR" ] || args+=(-H "@$HDR") + if [ -n "${3:-}" ]; then + printf '%s' "$3" | curl "${args[@]}" --data-binary @- + else + curl "${args[@]}" + fi +} + +echo "Waiting for Mattermost at $MM_URL ..." +until curl -sf "$MM_URL/api/v4/system/ping" >/dev/null; do sleep 2; done + +# The first account on a fresh server becomes the system admin. +api POST /users "{\"email\":\"$ADMIN_EMAIL\",\"username\":\"$ADMIN_USER\",\"password\":\"$ADMIN_PASS\"}" >/dev/null + +# Log in. The session token comes back in the Token response header. +TOKEN="$(printf '{"login_id":"%s","password":"%s"}' "$ADMIN_USER" "$ADMIN_PASS" \ + | curl -sS -D - -o /dev/null -X POST "$MM_URL/api/v4/users/login" \ + -H 'Content-Type: application/json' --data-binary @- \ + | tr -d '\r' | sed -n 's/^[Tt]oken: //p')" +[ -n "$TOKEN" ] || { echo "Login failed." >&2; exit 1; } +printf 'Authorization: Bearer %s\n' "$TOKEN" > "$HDR" + +TEAM_ID="$(api POST /teams "{\"name\":\"$TEAM_NAME\",\"display_name\":\"Home\",\"type\":\"I\"}" | field id)" +need team "$TEAM_ID" +CHANNEL_ID="$(api POST /channels "{\"team_id\":\"$TEAM_ID\",\"name\":\"$CHANNEL_NAME\",\"display_name\":\"Assistant\",\"type\":\"P\"}" | field id)" +need channel "$CHANNEL_ID" + +HUMAN_ID="$(api POST /users "{\"email\":\"$HUMAN_EMAIL\",\"username\":\"$HUMAN_USER\",\"password\":\"$HUMAN_PASS\"}" | field id)" +need user "$HUMAN_ID" +api POST "/teams/$TEAM_ID/members" "{\"team_id\":\"$TEAM_ID\",\"user_id\":\"$HUMAN_ID\"}" >/dev/null +api POST "/channels/$CHANNEL_ID/members" "{\"user_id\":\"$HUMAN_ID\"}" >/dev/null + +BOT_ID="$(api POST /bots "{\"username\":\"$BOT_NAME\",\"display_name\":\"netclaw\"}" | field user_id)" +need bot "$BOT_ID" +api POST "/teams/$TEAM_ID/members" "{\"team_id\":\"$TEAM_ID\",\"user_id\":\"$BOT_ID\"}" >/dev/null +api POST "/channels/$CHANNEL_ID/members" "{\"user_id\":\"$BOT_ID\"}" >/dev/null +BOT_TOKEN="$(api POST "/users/$BOT_ID/tokens" '{"description":"netclaw"}' | field token)" +need "bot token" "$BOT_TOKEN" + +printf '%s' "$BOT_TOKEN" > .bot-token +[ ! -s .env ] || [ -z "$(tail -c1 .env)" ] || echo >> .env # ensure a trailing newline +{ + echo "MM_CHANNEL_ID=$CHANNEL_ID" + echo "MM_USER_ID=$HUMAN_ID" +} >> .env + +cat < - - - - - - - - - diff --git a/public/assets/netclaw-horizontal-dark.svg b/public/assets/netclaw-horizontal-dark.svg deleted file mode 100644 index c48c75e..0000000 --- a/public/assets/netclaw-horizontal-dark.svg +++ /dev/null @@ -1,26 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - \ No newline at end of file diff --git a/public/assets/netclaw-stacked-purple.svg b/public/assets/netclaw-stacked-purple.svg deleted file mode 100644 index e189634..0000000 --- a/public/assets/netclaw-stacked-purple.svg +++ /dev/null @@ -1,26 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - \ No newline at end of file diff --git a/screenshots/.env.example b/screenshots/.env.example new file mode 100644 index 0000000..32a2313 --- /dev/null +++ b/screenshots/.env.example @@ -0,0 +1,12 @@ +# Copy to screenshots/.env (gitignored). Never commit a real hostname or model name. +# +# Real inference without leaking the endpoint (relay mode): set the real https:// base URL and +# the model id it serves. The forwarder (compose/forwarder.yml) presents it inside the stack as +# http://llm.example.internal:8081 with the model id demo-model. +# INFERENCE_UPSTREAM=https://inference.internal.example +# INFERENCE_UPSTREAM_MODEL=the-real-model-id +# +# Or point netclaw straight at an endpoint (external mode; it ends up in netclaw.json, so don't +# capture provider lists this way). Without either, the stack uses a tiny local Ollama model. +# INFERENCE_BASE_URL=http://llm.example.internal:8080 +# INFERENCE_MODEL_ID=your-model-id diff --git a/screenshots/.gitignore b/screenshots/.gitignore new file mode 100644 index 0000000..fcdbee0 --- /dev/null +++ b/screenshots/.gitignore @@ -0,0 +1,3 @@ +.env +.netclaw-screenshot-home/ +mattermost/.bootstrap*.json diff --git a/screenshots/README.md b/screenshots/README.md index 1a2379f..b3bbe5c 100644 --- a/screenshots/README.md +++ b/screenshots/README.md @@ -1,56 +1,169 @@ -# Screenshot capture pipeline +# Screenshot and verification stack -Source of truth for every image under `output/` (served at `/screenshots/output/`). +Everything that produces an image under `output/` (served at `/screenshots/output/`) or proves a command works runs in containers against the **released** Netclaw image. Docker is the only prerequisite. + +**Rule: never use a host `netclaw` binary.** Whatever is on your `PATH` may be a prototype or a dev build. The CLI in every recording and every verification command comes from `ghcr.io/netclaw-dev/netclaw:`. ## Layout | Path | What it is | |------|-----------| -| `capture.sh` | Orchestrator: brings up the Docker stack, runs every VHS tape, tears down | -| `tapes/*.tape` | [VHS](https://github.com/charmbracelet/vhs) recordings — one per CLI command/flow | -| `scripts/` | Helpers (`ollama-init.sh` model pull, `seed-approvals.sh` fixture seeding) | -| `mcp-server/` | Demo MCP server used by the `mcp-tools` capture | -| `output/` | Generated PNGs/GIFs (committed; consumed by the docs) | +| `stack.sh` | Bring up, reset, exec into and tear down a disposable daemon (verification, no recording) | +| `capture.sh` | Record VHS tapes inside the capture container against that daemon | +| `shoot-page.sh` | Screenshot a page of the locally built docs site in the Playwright container | +| `mattermost/run.sh` | Mattermost + daemon + Playwright, all in containers; regenerates `mattermost-*.png` | +| `tapes/*.tape` | [VHS](https://github.com/charmbracelet/vhs) recordings, one per CLI command or flow | +| `capture/` | Capture image: VHS (ttyd, ffmpeg, Chromium) plus the released CLI | +| `playwright/` | Playwright image matching the version locked in `package-lock.json` | +| `compose/` | Compose overlays (`configured.yml`, `ollama.yml`, `forwarder.yml`, `projects.yml`, `skillserver.yml`, `docker-host.yml`); the base is `../docker-compose.screenshots.yml` | +| `scripts/`, `mcp-server/` | Ollama model pull, the seed scripts tapes run out of sight (`seed-*.sh`), a Mattermost API fixture, the third-party redaction script, the demo MCP server | +| `lib.sh`, `.env.example` | Shared settings; copy `.env.example` to `.env` (gitignored) for real inference | + +## Pick the release + +The image tag comes from one place: `documentedVersion` in `src/version.json`. Override per run: + +```bash +NETCLAW_VERSION=0.27.1 ./screenshots/stack.sh up # a specific release +NETCLAW_IMAGE=ghcr.io/netclaw-dev/netclaw:dev ./screenshots/stack.sh up # any image reference +``` + +`src/version.json` may lag the release you're documenting, so pass `NETCLAW_VERSION=0.27.1` until it's bumped. Every command prints the image it used. + +## Verify a command against a clean daemon + +```bash +export COMPOSE_PROJECT_NAME=dr04 # your own name, see "Run several at once" +./screenshots/stack.sh up # clean ~/.netclaw, daemon healthy, provider + model + MCP configured +./screenshots/stack.sh exec netclaw doctor # runs as the netclaw user, like the daemon +./screenshots/stack.sh exec netclaw status +./screenshots/stack.sh daemon-log 100 # the daemon's log file (container stdout is empty) +./screenshots/stack.sh down # removes containers, network and volumes +``` + +Daemon states: + +| State | Command | What you get | +|-------|---------|--------------| +| configured (default) | `stack.sh up` | Provider `inference`, model roles and the demo MCP server `demo-utilities`, added with `netclaw provider add`, `netclaw model set` and `netclaw mcp add`, plus the Security section `netclaw init` writes. This is what a user's `netclaw.json` looks like after init. | +| first run | `stack.sh up --fresh` | Healthy daemon, empty `~/.netclaw`, no model. `netclaw status` says "daemon not configured". Use it for `netclaw init`. | +| bare | `stack.sh up --bare` | Like first run, plus the demo MCP server container. Tapes that seed their own config (`scripts/seed-*.sh`) use it. | + +`stack.sh reset [--fresh]` wipes `~/.netclaw` and restarts the daemon without rebuilding the rest of the stack. Run `./screenshots/stack.sh` with no arguments for the full command list. + +The configured state writes `netclaw.json` instead of setting `NETCLAW_*` env vars because the CLI refuses to run most commands without that file, and the `provider`, `model` and `mcp` TUIs edit it. The 0.25+ env shape still works if you need it for a throwaway daemon: + +```bash +docker run -d --name x -e NETCLAW_Providers__inference__Type=ollama \ + -e NETCLAW_Providers__inference__Endpoint=http://ollama:11434 \ + -e NETCLAW_Models__Definitions__main__Provider=inference \ + -e NETCLAW_Models__Definitions__main__ModelId=qwen2:0.5b \ + -e NETCLAW_Models__Roles__Main=main ghcr.io/netclaw-dev/netclaw:0.27.1 +``` + +The Mattermost flow below uses exactly this env shape. + +## Inference + +| Mode | When | How | +|------|------|-----| +| hermetic (default) | Shots that don't need a real answer | Ollama with `qwen2:0.5b` in the stack. The first `up` of a project pulls the model (about 350 MB), so allow a minute or two. | +| relay | You need real answers in a screenshot | Set `INFERENCE_UPSTREAM` and `INFERENCE_UPSTREAM_MODEL` in `screenshots/.env` (see below). The only mode that keeps the real endpoint and model name out of the config and the frame. | +| external | You need a real reply and nothing is captured that prints the endpoint | Set `INFERENCE_BASE_URL` (an OpenAI-compatible endpoint) and `INFERENCE_MODEL_ID` in the environment or in `screenshots/.env` | + +```bash +INFERENCE_BASE_URL=http://llm.example.internal:8080 INFERENCE_MODEL_ID=my-model \ + ./screenshots/stack.sh up +``` + +External mode writes the endpoint into the daemon's `netclaw.json`, so commands like `netclaw provider list` print it. Capture those screenshots in hermetic mode. + +### Real inference without leaking the endpoint + +Set `INFERENCE_UPSTREAM` (the real `https://` base URL) and `INFERENCE_UPSTREAM_MODEL` (the model id it serves) in the gitignored `screenshots/.env`. `lib.sh` then switches to relay mode and adds `compose/forwarder.yml`: an nginx container in the stack's network namespace relays port 8081 to the upstream and rewrites the upstream's model id to `demo-model` in every response, and `llm.example.internal` resolves to loopback for everything in the namespace. netclaw only ever sees `http://llm.example.internal:8081` and the model `demo-model`, so provider lists, `netclaw init` and chat status bars are safe to capture, and no real model file name reaches a status bar. Check the first frame of every chat tape anyway. + +This is the only relay, and both tape families use it. The seed scripts (`seed-assistant.sh`, `seed-providers.sh`) point their `lab-llama` provider at it, and the configured daemon (`stack.sh up`) registers it as provider `inference`. Ollama stays up alongside it, so `netclaw init` and the provider and model tapes still have a reachable local provider. Without `INFERENCE_UPSTREAM`, tapes that need a real answer (`stats`, `single-shot-cli-execution`, the chat and approval tapes) can't produce one: the 0.5B Ollama model can't call tools. + +## Standard tape settings + +Every terminal image is shown in a docs column about 650 px wide, so a capture wider than about 1000 px shrinks its text below what a reader can read without the lightbox. Use this block, and change only `Height` (crop by terminal height, not by editing the PNG): + +``` +Set Shell "bash" +Set Theme "Dracula" +Set FontSize 16 +Set Width 1000 +Set Height 540 +Set Padding 20 +Set TypingSpeed 40ms +``` -The daemon image and the demo Ollama model come from `../docker-compose.screenshots.yml`. +At `FontSize 16` a character cell is about 10 px wide, so 1000 px holds about 93 columns, and scaled into the column the text lands at about 11 px. Widen a tape only as far as a table needs it (`model.tape` uses 1200 for the long OpenAI-compatible provider label, `doctor.tape` 1700 for its longest check line) and make the page caption say the image can be enlarged. -## Regenerate everything (VHS / TUI shots) +The status-like tapes (`status`, `doctor`, `stats`, `sessions`, `model`, `provider`, `skill-list`, `skill-sync`, `webhooks`, `reminder-list`, `config`, `single-shot-cli-execution`) all start from `scripts/seed-demo-install.sh`, so the images agree on providers, roles, skills, webhook routes, reminders and the MCP server. Add new status-like captures to that family instead of seeding a separate install. + +## Record tapes ```bash -./screenshots/capture.sh +./screenshots/capture.sh status # one tape +./screenshots/capture.sh status doctor # a few +./screenshots/capture.sh # all tapes +OUT_DIR=/tmp/shots ./screenshots/capture.sh status # write elsewhere (default: screenshots/output/) ``` -The daemon image is **pinned to the documented release**: `capture.sh` reads -`src/version.json` and pins `ghcr.io/netclaw-dev/netclaw:`. To -capture against a different build, override before running: +`capture.sh` starts a stack, resets `~/.netclaw` before every tape, runs each tape inside the capture container, copies the results to the output directory owned by you, and always tears the stack down. It exits non-zero if VHS fails or a tape doesn't produce every `Output` and `Screenshot` file it declares. + +How the recording reaches the daemon: the capture container shares the daemon container's network namespace and `~/.netclaw` volume, so `netclaw status` typed in a tape talks to `127.0.0.1:5199` exactly like a real user's CLI. Nothing is published to the host, and no `docker exec` shows up in a recording. + +Tape conventions: + +- Paths are relative to `/work`, so `Output screenshots/output/status.gif` and `Screenshot screenshots/output/status.png` land in the output directory. +- VHS starts bash with a clean `> ` prompt. No hidden preamble is needed unless a tape has to stage state first (see `chat-approval.tape`). +- Put `# stack: fresh` in a tape to run it against the first-run daemon (see `init-wizard.tape`), or `# stack: bare` for the same empty home with the demo MCP server up. +- Setup that isn't the subject of the image runs between `Hide` and `Show`: `bash /work/screenshots/scripts/seed-*.sh` for config, or `Source "/tapes/include/init-personal.tape"` to run the real `netclaw init` wizard (Personal, Ollama). Wait for a seed to finish with `Wait+Screen@4m /seeded: .../` instead of a fixed sleep. +- `# image: 0.27.0` runs one tape against another release (only `update-check.tape` does, to be honestly one version behind). `# host-docker: true` gives a tape the host's Docker socket (only `docker-health.tape`, which filters everything it prints to its own container). +- Only `.png` files are published from a run. A tape still needs an `Output` line for VHS, and its GIF stays in the container. +- VHS applies `Set Width`/`Set Height` once, so size each tape for its tallest screen. `Screenshot` is slightly asynchronous: leave a `Sleep` after every one, and don't issue two in a row. +- Shared setup goes in `tapes/include/*.tape` and is pulled in with `Source "/tapes/include/.tape"`. `capture.sh` only runs top-level tapes. Hide setup with `Hide`/`Show`, and wait for a marker the typed command doesn't contain (`echo SE''EDED`, then `Wait+Screen /SEEDED/`). +- Tapes that need a Git workspace run `scripts/seed-projects.sh` first. It creates `/home/netclaw/projects/website` on a volume the daemon shares (`compose/projects.yml`) and declares it as the workspace root, so shell approval prompts offer "This repository". Skill captures use `scripts/seed-skills.sh`, which publishes demo skills to the stack's skill server (`compose/skillserver.yml`, port 8090) with the `skillserver` CLI baked into the capture image. +- Tapes that drive a real model (`chat-*`, `approvals-*`, `reminder-run`) must make each prompt self-limiting ("then stop, run nothing else") and wait on text the prompt didn't type, such as a tool result line. A model that wanders into an extra tool call leaves a prompt open and the wait times out. +- A trailing `Screenshot` is lost by VHS if nothing follows it, so `capture.sh` appends `Sleep 1s` to the copy it runs. You don't have to. +- In 0.27.1, `Escape` doesn't quit a TUI. Use `Ctrl+Q`. +- The init wizard prints "Step N of 5" until you pick a posture, then "of 4" for Personal. `init-wizard.tape` (Personal) and `init-team.tape` (Team, the fifth step) record that as shown. + +## Mattermost and other browser captures ```bash -NETCLAW_VERSION=0.22.1 ./screenshots/capture.sh # a specific release -NETCLAW_IMAGE=ghcr.io/netclaw-dev/netclaw:dev ./screenshots/capture.sh # bleeding edge -OLLAMA_MODEL=qwen2:0.5b ./screenshots/capture.sh # demo inference model +INFERENCE_BASE_URL=http://llm.example.internal:8080 INFERENCE_MODEL_ID=my-model \ + OUT_DIR=/tmp/shots ./screenshots/mattermost/run.sh ``` -Keep `src/version.json` in step with the release you profiled — the -`docs_version_check` workflow opens an issue when the docs fall behind. +Mattermost, the daemon and Playwright share one compose network, and the daemon reaches Mattermost at `http://mattermost:8065`. The flow is: bootstrap Mattermost (`bootstrap.mjs`), start the daemon, store the bot token with `netclaw secrets set Mattermost.BotToken` (no token in env or compose), enable the channel, post an `@netclaw` message as the test user (`post-message.mjs`) and wait for the reply, then capture the System Console, bot account and thread (`capture.mjs`). `run.sh` sets `Mattermost.CallbackUrl` (through a small forwarder sidecar, because the daemon listens on loopback only) and allow-lists the daemon host in Mattermost (`MM_SERVICESETTINGS_ALLOWEDUNTRUSTEDINTERNALCONNECTIONS`), so tool approvals render as buttons. `SCRIPTED_APPROVAL=1 ./screenshots/mattermost/run.sh` needs no model: `scripted-llm.mjs` answers with one `demo-utilities/http_fetch` MCP call (the demo MCP server, gated to `Approval` for the Personal audience; `SCRIPTED_TOOL=shell` requests a `shell_execute` call instead), `approval.mjs` captures the prompt and `frame-approval.mjs` frames it as `approval-prompt.png` (rounded dark card, purple glow, 2x). The normal flow needs external inference (a 0.5B model won't produce a presentable answer). The bootstrap file with the short-lived bot token is deleted on exit. -## Mattermost capture (web setup + live conversation) +The Compose-sample screenshots (`mattermost-compose-*.png`, plus the recaptured `mattermost-bot-accounts.png`, `mattermost-create-bot.png`, `mattermost-conversation.png` and the landing page's `public/assets/approval-prompt.png`) come from `mattermost/capture-compose.mjs`, which signs in to the stack in `examples/mattermost-compose/` after `provision.sh` and `configure-netclaw.sh` have run. It needs the admin and `alice` passwords that `provision.sh` printed, and a Git repository for the approval prompt (so "This repository" appears). Token IDs are overwritten in the page before each shot. The Slack and Discord redactions are `scripts/redact-third-party.py`. -The Mattermost shots aren't a VHS tape — they're browser screenshots driven by -Playwright. Everything lives in `mattermost/`: +Screenshot a page of the docs site you built locally: -| File | Role | -|------|------| -| `docker-compose.mattermost.yml` | the Mattermost (`mattermost-preview`) container | -| `bootstrap.mjs` | seeds admin / team / bot + token / channel / test user via the REST API → `.bootstrap.json` | -| `run.sh` | orchestrator: compose up → bootstrap → start the pinned netclaw image (inference pointed at an OpenAI-compatible endpoint) → post an `@netclaw` message → run `capture.mjs` → tear down | -| `capture.mjs` | Playwright: the System Console bot-accounts setting, the bot + its token, and the live `@netclaw` thread | +```bash +npm run build +./screenshots/shoot-page.sh cli/status # docs-logs/screenshots/cli-status.png +``` + +`shoot-page.sh` runs `scripts/screenshot-page.mjs` in the same Playwright image and serves `dist/` itself. To shoot a dev server instead, start it with `npm run dev -- --host 0.0.0.0` and set `SCREENSHOT_BASE_URL=http://host.docker.internal:4321`. + +## Run several at once -Regenerate them with: +Every container, network and volume is namespaced by `COMPOSE_PROJECT_NAME`. There are no fixed container names and no published ports, so two agents can run side by side: ```bash -./screenshots/mattermost/run.sh +COMPOSE_PROJECT_NAME=dr04 ./screenshots/stack.sh up +COMPOSE_PROJECT_NAME=dr07 ./screenshots/capture.sh status ``` -Point inference at your own endpoint/model with `SPARK_ENDPOINT` and `SPARK_MODEL_ID` -(the defaults match the local netclaw instance). `.bootstrap.json` holds a short-lived -bot token and is gitignored. +Use your own name every time, and `down` when you're done. The Mattermost flow appends `-mm` to the name, so it doesn't collide with a stack of the same name. Only built images are shared across projects (`netclaw-docs-capture:`, `netclaw-docs-playwright:`, `netclaw-docs-mcp-demo:1`), which is safe because they're deterministic. + +## Design notes + +- **Why the capture image starts from VHS.** The release image is `ubuntu:24.04` (glibc) and the CLI is a self-contained .NET single-file binary. The VHS image is Debian trixie (glibc 2.41) with ICU and libsodium already installed, so the CLI and its self-dropping launcher are copied out of the release image unmodified. The recorded CLI is byte-for-byte the released one. Putting VHS into the release image would have meant installing Chromium, ttyd and ffmpeg on an Ubuntu base that ships Chromium only as a snap. +- **One network namespace.** A tiny `netns` container owns the stack's network namespace, and the daemon, capture container, Ollama and demo MCP server join it. Everything talks over `127.0.0.1`, the daemon stays in its default loopback-only `local` exposure mode, and the `netclaw init` wizard's default Ollama endpoint (`http://localhost:11434`) just works. +- **Ownership.** The capture container runs the tape as the `netclaw` user (uid 1654, same as the daemon) so it can use the shared home, then copies the files out with your uid and gid. diff --git a/screenshots/capture.sh b/screenshots/capture.sh index 91f7238..af3f216 100755 --- a/screenshots/capture.sh +++ b/screenshots/capture.sh @@ -1,58 +1,73 @@ #!/bin/bash -# Orchestrates the screenshot capture pipeline. -# Brings up Docker Compose environment, runs VHS tapes, tears down. -set -eu +# Record VHS tapes inside the capture container against a disposable released daemon. +# Docker is the only prerequisite; the host netclaw is never used. +# +# ./screenshots/capture.sh run every tape +# ./screenshots/capture.sh status run one tape (screenshots/tapes/status.tape) +# ./screenshots/capture.sh status doctor +# +# Output goes to screenshots/output/ (override: OUT_DIR=/some/dir), owned by you. +# A tape starts from a clean ~/.netclaw. A tape whose first line comments contain +# `# stack: fresh` runs against the first-run state (no model config); `# stack: bare` is the +# same empty home plus the demo MCP server container, for tapes that seed their own config; +# all others run against the configured daemon. Exits non-zero if any tape fails; the stack is always +# torn down. Image, project name and inference settings: see stack.sh and README.md. +set -euo pipefail +# shellcheck source=lib.sh +. "$(dirname "${BASH_SOURCE[0]}")/lib.sh" -SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" -PROJECT_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)" -TAPES_DIR="$SCRIPT_DIR/tapes" -OUTPUT_DIR="$SCRIPT_DIR/output" -COMPOSE_FILE="$PROJECT_ROOT/docker-compose.screenshots.yml" +OUT_DIR="${OUT_DIR:-$SHOTS_DIR/output}" +mkdir -p "$OUT_DIR" +export CAPTURE_OUT_DIR="$(cd "$OUT_DIR" && pwd)" -# Use a clean netclaw home for the init wizard so we get first-run screens -export NETCLAW_SCREENSHOT_HOME="$SCRIPT_DIR/.netclaw-screenshot-home" - -mkdir -p "$OUTPUT_DIR" +tapes=("$@") +if [ "${#tapes[@]}" -eq 0 ]; then + for f in "$SHOTS_DIR"/tapes/*.tape; do tapes+=("$(basename "$f" .tape)"); done +fi -NETCLAW_PORT="${NETCLAW_PORT:-5299}" +cleanup() { + echo "==> Tearing down project $COMPOSE_PROJECT_NAME" + "$SHOTS_DIR/stack.sh" down >/dev/null 2>&1 || true +} +trap cleanup EXIT -# Pin the daemon image to the version the docs are written against (src/version.json), -# unless the caller already set NETCLAW_VERSION or NETCLAW_IMAGE. Keeps screenshots -# in lockstep with the documented release. -if [ -z "${NETCLAW_VERSION:-}" ] && [ -f "$PROJECT_ROOT/src/version.json" ]; then - NETCLAW_VERSION="$(node -e "process.stdout.write(require('$PROJECT_ROOT/src/version.json').documentedVersion)" 2>/dev/null || true)" -fi -export NETCLAW_VERSION -echo "==> Capturing against netclaw ${NETCLAW_IMAGE:-ghcr.io/netclaw-dev/netclaw:${NETCLAW_VERSION:-0.22.1}}" +echo "==> Capturing against $NETCLAW_IMAGE ($INFERENCE_MODE inference), output $CAPTURE_OUT_DIR" -echo "==> Starting screenshot environment (building images if needed)..." -docker compose -f "$COMPOSE_FILE" up -d --build --wait +failed=() +for name in "${tapes[@]}"; do + tape="$SHOTS_DIR/tapes/$name.tape" + [ -f "$tape" ] || { echo "capture: no such tape: $name" >&2; exit 2; } + state=configured + if grep -qE '^#[[:space:]]*stack:[[:space:]]*fresh' "$tape"; then state=fresh; fi + if grep -qE '^#[[:space:]]*stack:[[:space:]]*bare' "$tape"; then state=bare; fi -echo "==> Waiting for netclawd health check on port $NETCLAW_PORT..." -for i in $(seq 1 60); do - if curl -sf "http://127.0.0.1:${NETCLAW_PORT}/api/health/ready" >/dev/null 2>&1; then - echo " netclawd is healthy." - break - fi - if [ "$i" -eq 60 ]; then - echo " ERROR: netclawd did not become healthy in time." - docker compose -f "$COMPOSE_FILE" logs netclawd - docker compose -f "$COMPOSE_FILE" down - exit 1 - fi - sleep 2 -done + # `# image: 0.27.0` runs this one tape against another release (update-check.tape needs an + # install that is genuinely one version behind). `# host-docker: true` lets the tape drive the + # host's Docker through its socket (docker-health.tape). + image_version="$(sed -nE 's/^#[[:space:]]*image:[[:space:]]*([0-9A-Za-z._-]+).*/\1/p' "$tape" | head -1)" + host_docker="" + if grep -qE '^#[[:space:]]*host-docker:[[:space:]]*true' "$tape"; then host_docker=1; fi -echo "==> Running VHS tapes..." -for tape in "$TAPES_DIR"/*.tape; do - if [ -f "$tape" ]; then - echo " Running: $(basename "$tape")" - vhs "$tape" || echo " WARNING: $(basename "$tape") failed" + echo "==> $name: clean daemon ($state)" + if ! ( + if [ -n "$image_version" ]; then + unset NETCLAW_IMAGE + export NETCLAW_VERSION="$image_version" + # shellcheck source=lib.sh + . "$SHOTS_DIR/lib.sh" + echo "==> $name: this tape pins image $NETCLAW_IMAGE" + fi + if [ -n "$host_docker" ]; then export CAPTURE_HOST_DOCKER=1; fi + # reset = wipe ~/.netclaw and (re)start the daemon; also creates the stack on first use. + "$SHOTS_DIR/stack.sh" reset $([ "$state" != configured ] && echo "--$state") && + dc "$state" run --rm --no-deps -T --build capture "$name" + ); then + failed+=("$name") fi done -echo "==> Tearing down environment..." -docker compose -f "$COMPOSE_FILE" down - -echo "==> Screenshots saved to: $OUTPUT_DIR" -ls -la "$OUTPUT_DIR"/*.png 2>/dev/null || echo " No PNG files generated." +if [ "${#failed[@]}" -gt 0 ]; then + echo "capture: FAILED tapes: ${failed[*]}" >&2 + exit 1 +fi +echo "==> Done: ${tapes[*]}" diff --git a/screenshots/capture/Dockerfile b/screenshots/capture/Dockerfile new file mode 100644 index 0000000..fce7e0b --- /dev/null +++ b/screenshots/capture/Dockerfile @@ -0,0 +1,48 @@ +# Capture image: VHS (ttyd, ffmpeg, headless chromium) + the released netclaw CLI. +# +# Why this shape: the release image (ubuntu:24.04, glibc) ships the CLI as a +# self-contained .NET single-file binary, and the VHS image (debian trixie, glibc 2.41, +# ICU and libsodium already present) runs it unmodified. So we take the VHS image and +# copy the CLI and its self-dropping launcher out of the release image. The CLI in a +# tape is therefore byte-for-byte the released one, never a host build. +# +# The container shares the daemon's network namespace and ~/.netclaw volume (see +# docker-compose.screenshots.yml), so `netclaw status` typed in a recording reaches the +# daemon over loopback the way a real user's CLI does. +ARG NETCLAW_IMAGE +ARG VHS_IMAGE=ghcr.io/charmbracelet/vhs:v0.12.1 + +FROM ${NETCLAW_IMAGE} AS netclaw + +FROM ${VHS_IMAGE} + +RUN apt-get update \ + && apt-get install -y --no-install-recommends gosu curl ca-certificates git jq \ + && rm -rf /var/lib/apt/lists/* + +# Same uid/gid as the daemon so the shared ~/.netclaw volume is readable and writable. +RUN groupadd --gid 1654 netclaw \ + && useradd --uid 1654 --gid netclaw --create-home netclaw \ + && mkdir -p /home/netclaw/.netclaw /work/screenshots/output \ + && chown -R netclaw:netclaw /home/netclaw /work + +# The skillserver CLI, used by the seed scripts to publish demo skills to the stack's skill server. +ARG SKILLSERVER_VERSION=0.4.1 +RUN arch="$(uname -m)"; case "$arch" in x86_64) rid=x64 ;; aarch64) rid=arm64 ;; esac \ + && curl -fsSL "https://github.com/netclaw-dev/skill-server/releases/download/${SKILLSERVER_VERSION}/skillserver-${SKILLSERVER_VERSION}-linux-${rid}.tar.gz" \ + | tar -xz -C /usr/local/bin \ + && chmod +x /usr/local/bin/skillserver + +COPY --from=netclaw /opt/netclaw/cli/netclaw /opt/netclaw/cli/netclaw +COPY --from=netclaw /usr/local/bin/netclaw /usr/local/bin/netclaw + +COPY entrypoint.sh /usr/local/bin/capture-entrypoint +RUN chmod +x /usr/local/bin/capture-entrypoint + +ENV HOME=/home/netclaw \ + NETCLAW_HOME=/home/netclaw/.netclaw \ + NETCLAW_CONTAINER_SUPERVISOR=entrypoint \ + VHS_NO_SANDBOX=true + +WORKDIR /work +ENTRYPOINT ["/usr/local/bin/capture-entrypoint"] diff --git a/screenshots/capture/entrypoint.sh b/screenshots/capture/entrypoint.sh new file mode 100755 index 0000000..a8d7993 --- /dev/null +++ b/screenshots/capture/entrypoint.sh @@ -0,0 +1,63 @@ +#!/bin/bash +# Runs inside the capture container. Arguments are tape names (without .tape). +# Each tape is recorded as the unprivileged `netclaw` user (same as the daemon), into a +# staging dir, then copied to /out owned by the invoking host user. +# Exits non-zero as soon as any tape fails or does not produce a file it declares. +set -uo pipefail + +STAGE=/work/screenshots/output +OUT=/out +HOST_UID="${HOST_UID:-0}" +HOST_GID="${HOST_GID:-0}" + +if [ "$#" -eq 0 ]; then + echo "capture: no tapes given" >&2 + exit 2 +fi + +# Tapes marked `# host-docker: true` get the host's Docker socket (compose/docker-host.yml). The +# netclaw user needs the socket's group to use it. +if [ -S /var/run/docker.sock ]; then + sock_gid="$(stat -c %g /var/run/docker.sock)" + getent group "$sock_gid" >/dev/null || groupadd --gid "$sock_gid" dockerhost + usermod -aG "$(getent group "$sock_gid" | cut -d: -f1)" netclaw +fi + +for name in "$@"; do + tape="/tapes/${name}.tape" + if [ ! -f "$tape" ]; then + echo "capture: no such tape: $name" >&2 + exit 2 + fi + + echo "==> Recording $name ($(netclaw --version))" + rm -rf "${STAGE:?}"/* + chown netclaw:netclaw "$STAGE" + + # VHS writes a Screenshot asynchronously, so one that is the LAST command in a tape is + # lost when the recording ends. Run a copy with a trailing Sleep so any tape is safe. + run_tape="/tmp/run-${name}.tape" + { cat "$tape"; printf '\nSleep 1s\n'; } > "$run_tape" + chmod 644 "$run_tape" + + if ! gosu netclaw vhs "$run_tape"; then + echo "capture: FAILED: vhs exited non-zero for $name" >&2 + exit 1 + fi + + # Every Output/Screenshot path the tape declares must exist. + missing=0 + while read -r path; do + if [ ! -s "/work/$path" ]; then + echo "capture: FAILED: $name did not produce $path" >&2 + missing=1 + fi + done < <(sed -nE 's/^[[:space:]]*(Output|Screenshot)[[:space:]]+"?([^" ]+)"?.*$/\2/p' "$tape" | grep -v '^#') + [ "$missing" -eq 0 ] || exit 1 + + # Only stills are published. A tape needs an Output line for VHS to run; its GIF stays in staging. + while IFS= read -r -d '' f; do + install -m 0644 -o "$HOST_UID" -g "$HOST_GID" "$f" "$OUT/$(basename "$f")" + echo " wrote $(basename "$f")" + done < <(find "$STAGE" -maxdepth 1 -type f -name '*.png' -print0) +done diff --git a/screenshots/compose/configured.yml b/screenshots/compose/configured.yml new file mode 100644 index 0000000..9303652 --- /dev/null +++ b/screenshots/compose/configured.yml @@ -0,0 +1,29 @@ +# Overlay: the "initialized" daemon. Adds the demo MCP server container. The daemon's +# provider, model roles and MCP entry are NOT env vars: stack.sh writes them with the +# real CLI (`netclaw provider add`, `netclaw model set`, `netclaw mcp add`) so +# ~/.netclaw/config/netclaw.json looks exactly like a user's after `netclaw init`, and +# the CLI's own config preflight and TUIs see it. +# Relative paths resolve against the directory of docker-compose.screenshots.yml. +services: + mcp-demo: + image: netclaw-docs-mcp-demo:1 + build: ./screenshots/mcp-server + network_mode: "service:netns" + depends_on: + - netns + healthcheck: + test: ["CMD", "python", "-c", "import socket; s=socket.create_connection(('127.0.0.1',8080),2); s.close()"] + interval: 3s + timeout: 5s + retries: 20 + start_period: 3s + + netclawd: + depends_on: + mcp-demo: + condition: service_healthy + environment: + # Real inference endpoints can be slow to first token. + NETCLAW_Session__TurnLlmTimeoutSeconds: "600" + NETCLAW_Session__FirstTokenTimeoutSeconds: "600" + NETCLAW_Session__PrefillTimeoutSeconds: "600" diff --git a/screenshots/compose/docker-host.yml b/screenshots/compose/docker-host.yml new file mode 100644 index 0000000..16e3794 --- /dev/null +++ b/screenshots/compose/docker-host.yml @@ -0,0 +1,12 @@ +# Overlay: lets a tape drive the host's Docker (docker-health.tape), by mounting the Docker socket +# and the host's docker client into the capture container. Added by lib.sh only for tapes marked +# `# host-docker: true`. The tape must filter everything it prints to its own container, so nothing +# else on the host shows up in the screenshot. +services: + capture: + environment: + NETCLAW_IMAGE: ${NETCLAW_IMAGE} + COMPOSE_PROJECT_NAME: ${COMPOSE_PROJECT_NAME} + volumes: + - /var/run/docker.sock:/var/run/docker.sock + - ${DOCKER_CLI_PATH:-/usr/bin/docker}:/usr/local/bin/docker:ro diff --git a/screenshots/compose/forwarder.conf.template b/screenshots/compose/forwarder.conf.template new file mode 100644 index 0000000..566f0f2 --- /dev/null +++ b/screenshots/compose/forwarder.conf.template @@ -0,0 +1,23 @@ +# Relays plain HTTP on :8081 to the real inference endpoint, so netclaw is configured with +# http://llm.example.internal:8081 and the real hostname never appears in netclaw.json, +# the TUI or a recording. The upstream's model id is rewritten to a neutral one in every +# response (so /v1/models lists the neutral id and `netclaw init` / `model discover` store it); +# requests keep the neutral id, which llama.cpp style servers accept for any model name. +# Rendered by the nginx image from INFERENCE_UPSTREAM, INFERENCE_UPSTREAM_HOST, +# INFERENCE_UPSTREAM_MODEL and INFERENCE_MODEL_ID (screenshots/.env, gitignored). +server { + listen 8081; + location / { + proxy_pass ${INFERENCE_UPSTREAM}; + proxy_http_version 1.1; + proxy_ssl_server_name on; + proxy_set_header Host ${INFERENCE_UPSTREAM_HOST}; + proxy_set_header Accept-Encoding ""; + proxy_buffering off; + proxy_read_timeout 900s; + proxy_send_timeout 900s; + sub_filter_types application/json text/event-stream; + sub_filter_once off; + sub_filter '${INFERENCE_UPSTREAM_MODEL}' '${INFERENCE_MODEL_ID}'; + } +} diff --git a/screenshots/compose/forwarder.yml b/screenshots/compose/forwarder.yml new file mode 100644 index 0000000..b2e849d --- /dev/null +++ b/screenshots/compose/forwarder.yml @@ -0,0 +1,32 @@ +# Overlay: keeps a real inference endpoint out of every frame. Added by lib.sh when +# INFERENCE_UPSTREAM is set (an https:// URL in screenshots/.env, together with +# INFERENCE_UPSTREAM_MODEL, the model id the upstream really serves). +# +# - `forwarder` joins the stack's network namespace and relays :8081 to the upstream, rewriting +# the upstream's model id to the neutral INFERENCE_MODEL_ID (default demo-model) in responses. +# - `extra_hosts` on the namespace owner maps llm.example.internal to loopback, and every +# container that shares the namespace (daemon, capture, helpers) inherits the entry. +# +# The configured endpoint is http://llm.example.internal:8081 and the model id is demo-model. +# Port 8081 because the demo MCP server owns 8080 in the shared network namespace. +services: + netns: + extra_hosts: + - "llm.example.internal:127.0.0.1" + + forwarder: + image: nginx:1.27-alpine + network_mode: "service:netns" + depends_on: + - netns + environment: + INFERENCE_UPSTREAM: ${INFERENCE_UPSTREAM:?set in screenshots/.env} + INFERENCE_UPSTREAM_HOST: ${INFERENCE_UPSTREAM_HOST:?set in screenshots/.env} + INFERENCE_UPSTREAM_MODEL: ${INFERENCE_UPSTREAM_MODEL:?set in screenshots/.env} + INFERENCE_MODEL_ID: ${INFERENCE_MODEL_ID:?set in screenshots/.env} + volumes: + - ./screenshots/compose/forwarder.conf.template:/etc/nginx/templates/default.conf.template:ro + + netclawd: + depends_on: + - forwarder diff --git a/screenshots/compose/ollama.yml b/screenshots/compose/ollama.yml new file mode 100644 index 0000000..ef2c1fa --- /dev/null +++ b/screenshots/compose/ollama.yml @@ -0,0 +1,31 @@ +# Overlay: hermetic inference. A tiny Ollama model, no external network use after the +# one-time model pull. Shares the stack's network namespace, so the endpoint is +# http://127.0.0.1:11434 (which is also what the `netclaw init` wizard defaults to). +# Relative paths resolve against the directory of docker-compose.screenshots.yml. +services: + ollama: + image: ollama/ollama:0.32.0 + network_mode: "service:netns" + depends_on: + - netns + volumes: + - ollama-data:/root/.ollama + + ollama-init: + image: curlimages/curl:8.13.0 + network_mode: "service:netns" + entrypoint: ["/bin/sh", "/usr/local/bin/ollama-init.sh"] + environment: + OLLAMA_MODEL: ${OLLAMA_MODEL:-qwen2:0.5b} + depends_on: + - ollama + volumes: + - ./screenshots/scripts/ollama-init.sh:/usr/local/bin/ollama-init.sh:ro + + netclawd: + depends_on: + ollama-init: + condition: service_completed_successfully + +volumes: + ollama-data: diff --git a/screenshots/compose/projects.yml b/screenshots/compose/projects.yml new file mode 100644 index 0000000..155ed9f --- /dev/null +++ b/screenshots/compose/projects.yml @@ -0,0 +1,25 @@ +# Overlay: a neutral projects directory shared by the daemon and the capture container, +# so a tape can create a small Git repository at /home/netclaw/projects/website that the +# daemon's shell tool can also see (approval prompts offer "This repository" only inside +# a Git repository). Created and owned by the netclaw uid (1654) by a one-shot init. +services: + projects-init: + image: alpine:3.20 + network_mode: "none" + command: ["sh", "-c", "mkdir -p /projects && chown 1654:1654 /projects"] + volumes: + - projects:/projects + + netclawd: + depends_on: + projects-init: + condition: service_completed_successfully + volumes: + - projects:/home/netclaw/projects + + capture: + volumes: + - projects:/home/netclaw/projects + +volumes: + projects: diff --git a/screenshots/compose/skillserver.yml b/screenshots/compose/skillserver.yml new file mode 100644 index 0000000..abdb677 --- /dev/null +++ b/screenshots/compose/skillserver.yml @@ -0,0 +1,31 @@ +# Overlay: a skill server on the stack's network namespace (port 8090, because the demo MCP +# server owns 8080), for the skill-list and skill-sync captures. It +# is seeded with a throwaway demo key (publishing needs one; reads stay open). Pinned to the release the docs describe. +services: + skillserver-init: + image: ghcr.io/netclaw-dev/skillserver:0.4.1 + user: "0:0" + network_mode: "none" + entrypoint: ["sh", "-c", "chown -R 1654:1654 /data"] + restart: "no" + volumes: + - skill-data:/data + + skillserver: + image: ghcr.io/netclaw-dev/skillserver:0.4.1 + network_mode: "service:netns" + depends_on: + netns: + condition: service_started + skillserver-init: + condition: service_completed_successfully + environment: + SKILLSERVER__DATAPATH: /data + SKILLSERVER__BASEURL: http://127.0.0.1:8090 + SKILLSERVER__APIKEY: demo-publish-key + ASPNETCORE_URLS: http://+:8090 + volumes: + - skill-data:/data + +volumes: + skill-data: diff --git a/screenshots/lib.sh b/screenshots/lib.sh new file mode 100755 index 0000000..5beee83 --- /dev/null +++ b/screenshots/lib.sh @@ -0,0 +1,109 @@ +#!/bin/bash +# Shared helpers for stack.sh and capture.sh. Source it, don't run it. +# +# Resolves, in order: ./screenshots/.env (gitignored), then defaults. +# COMPOSE_PROJECT_NAME isolates containers, network and volumes (default netclaw-shots) +# NETCLAW_VERSION release tag; default = documentedVersion in src/version.json +# NETCLAW_IMAGE full image reference; wins over NETCLAW_VERSION +# INFERENCE_BASE_URL external OpenAI-compatible endpoint (unset = hermetic Ollama) +# INFERENCE_MODEL_ID model id served there (required with INFERENCE_BASE_URL) +# OLLAMA_MODEL hermetic model (default qwen2:0.5b) +# INFERENCE_UPSTREAM base URL of a real OpenAI-compatible server (relay mode). Adds the forwarder +# (compose/forwarder.yml) that answers to http://llm.example.internal:8081 +# inside the stack and rewrites the upstream's model id to `demo-model` in +# every response, so tapes get real answers while neither the real hostname +# nor the real model name reaches netclaw.json or a screenshot. Needs +# INFERENCE_UPSTREAM_MODEL (the id the upstream really serves). Ollama stays +# available alongside it, for the tapes that need a reachable local provider. + +SHOTS_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +PROJECT_ROOT="$(cd "$SHOTS_DIR/.." && pwd)" +BASE_COMPOSE="$PROJECT_ROOT/docker-compose.screenshots.yml" + +if [ -f "$SHOTS_DIR/.env" ]; then + set -a + # shellcheck disable=SC1091 + . "$SHOTS_DIR/.env" + set +a +fi + +export COMPOSE_PROJECT_NAME="${COMPOSE_PROJECT_NAME:-netclaw-shots}" +export COMPOSE_PROGRESS="${COMPOSE_PROGRESS:-quiet}" # set COMPOSE_PROGRESS=plain to see build/pull output +# Compose project names must be lowercase alphanumerics, '-' or '_'. +case "$COMPOSE_PROJECT_NAME" in + *[!a-z0-9_-]* | "") echo "COMPOSE_PROJECT_NAME must match [a-z0-9_-]+ (got '$COMPOSE_PROJECT_NAME')" >&2; exit 2 ;; +esac + +# Version: one source of truth (src/version.json) unless overridden. +if [ -z "${NETCLAW_IMAGE:-}" ]; then + if [ -z "${NETCLAW_VERSION:-}" ]; then + NETCLAW_VERSION="$(sed -n 's/.*"documentedVersion": *"\([^"]*\)".*/\1/p' "$PROJECT_ROOT/src/version.json")" + [ -n "$NETCLAW_VERSION" ] || { echo "could not read documentedVersion from src/version.json" >&2; exit 2; } + fi + NETCLAW_IMAGE="ghcr.io/netclaw-dev/netclaw:${NETCLAW_VERSION}" +fi +export NETCLAW_IMAGE +export NETCLAW_VERSION="${NETCLAW_IMAGE##*:}" +export NETCLAW_IMAGE_TAG="$(printf '%s' "$NETCLAW_VERSION" | tr -c 'A-Za-z0-9_.-' '_')" + +# Inference: three modes. +# relay INFERENCE_UPSTREAM set: real answers through the forwarder, neutral hostname and model +# external INFERENCE_BASE_URL set: a real endpoint written into netclaw.json as is +# hermetic neither: Ollama with a tiny model, no external network use +if [ -n "${INFERENCE_UPSTREAM:-}" ]; then + : "${INFERENCE_UPSTREAM_MODEL:?INFERENCE_UPSTREAM_MODEL is required with INFERENCE_UPSTREAM (the model id the upstream serves)}" + INFERENCE_UPSTREAM_HOST="${INFERENCE_UPSTREAM#*://}"; INFERENCE_UPSTREAM_HOST="${INFERENCE_UPSTREAM_HOST%%/*}" + INFERENCE_BASE_URL="${INFERENCE_BASE_URL:-http://llm.example.internal:8081}" + INFERENCE_MODEL_ID="${INFERENCE_MODEL_ID:-demo-model}" + INFERENCE_MODE=relay + export INFERENCE_UPSTREAM INFERENCE_UPSTREAM_HOST INFERENCE_UPSTREAM_MODEL INFERENCE_MODEL_ID + export NC_INFERENCE_TYPE=openai-compatible + export NC_INFERENCE_ENDPOINT="$INFERENCE_BASE_URL" + export NC_INFERENCE_MODEL="$INFERENCE_MODEL_ID" +elif [ -n "${INFERENCE_BASE_URL:-}" ]; then + : "${INFERENCE_MODEL_ID:?INFERENCE_MODEL_ID is required when INFERENCE_BASE_URL is set}" + INFERENCE_MODE=external + export NC_INFERENCE_TYPE=openai-compatible + export NC_INFERENCE_ENDPOINT="$INFERENCE_BASE_URL" + export NC_INFERENCE_MODEL="$INFERENCE_MODEL_ID" +else + INFERENCE_MODE=hermetic + export NC_INFERENCE_TYPE=ollama + export NC_INFERENCE_ENDPOINT=http://127.0.0.1:11434 + export NC_INFERENCE_MODEL="${OLLAMA_MODEL:-qwen2:0.5b}" +fi + +# Playwright image/module version: the one locked in package-lock.json. +if [ -z "${PLAYWRIGHT_VERSION:-}" ]; then + PLAYWRIGHT_VERSION="$(grep -A1 '"node_modules/playwright":' "$PROJECT_ROOT/package-lock.json" | sed -n 's/.*"version": *"\([^"]*\)".*/\1/p')" +fi +export PLAYWRIGHT_VERSION + +export HOST_UID="${HOST_UID:-$(id -u)}" +export HOST_GID="${HOST_GID:-$(id -g)}" + +# compose_files +# bare: like configured (demo MCP server container up) but stack.sh does not seed any config +# configured: daemon with a provider, model roles and the demo MCP server +# fresh: daemon with no model config at all (first-run state for `netclaw init`) +# all: every overlay, for commands (exec/logs/down) that must see every service +compose_files() { + local state="$1" + local files=(-f "$BASE_COMPOSE") + if [ "$INFERENCE_MODE" != external ]; then files+=(-f "$SHOTS_DIR/compose/ollama.yml"); fi + if [ "$INFERENCE_MODE" = relay ]; then files+=(-f "$SHOTS_DIR/compose/forwarder.yml"); fi + if [ "$state" != fresh ]; then files+=(-f "$SHOTS_DIR/compose/configured.yml" -f "$SHOTS_DIR/compose/skillserver.yml"); fi + files+=(-f "$SHOTS_DIR/compose/projects.yml") + if [ -n "${CAPTURE_HOST_DOCKER:-}" ]; then files+=(-f "$SHOTS_DIR/compose/docker-host.yml"); fi + printf '%s\0' "${files[@]}" +} + +# dc +dc() { + local state="$1"; shift + local files=() + while IFS= read -r -d '' f; do files+=("$f"); done < <(compose_files "$state") + docker compose "${files[@]}" "$@" +} + +home_volume() { printf '%s_netclaw-home' "$COMPOSE_PROJECT_NAME"; } diff --git a/screenshots/mattermost/approval.mjs b/screenshots/mattermost/approval.mjs new file mode 100644 index 0000000..373e5b6 --- /dev/null +++ b/screenshots/mattermost/approval.mjs @@ -0,0 +1,95 @@ +#!/usr/bin/env node +// Captures the landing page's approval prompt (public/assets/approval-prompt.png): the thread +// post that netclaw sends when shell_execute needs approval, with its buttons, framed by +// frame-approval.mjs. Runs in the Playwright container (run.sh with SCRIPTED_APPROVAL=1). +// +// Env: BOOTSTRAP_FILE, OUT_DIR, REPLY_TIMEOUT_SECONDS (default 120) +import { chromium } from 'playwright'; +import { readFileSync, mkdirSync } from 'fs'; +import { dirname, join } from 'path'; +import { fileURLToPath } from 'url'; +import { frame } from './frame-approval.mjs'; + +const __dirname = dirname(fileURLToPath(import.meta.url)); +const boot = JSON.parse(readFileSync(process.env.BOOTSTRAP_FILE || join(__dirname, '.bootstrap.json'), 'utf8')); +const OUT = process.env.OUT_DIR || join(__dirname, '..', 'output'); +const TIMEOUT_MS = Number(process.env.REPLY_TIMEOUT_SECONDS || 120) * 1000; +const BASE = boot.serverUrl; +mkdirSync(OUT, { recursive: true }); + +// Mattermost's Onyx theme. +const ONYX = { + type: 'Onyx', sidebarBg: '#18181a', sidebarText: '#ffffff', sidebarUnreadText: '#ffffff', + sidebarTextHoverBg: '#25262a', sidebarTextActiveBorder: '#afbe5d', sidebarTextActiveColor: '#ffffff', + sidebarHeaderBg: '#1f1f1f', sidebarTeamBarBg: '#121212', sidebarHeaderTextColor: '#ffffff', + onlineIndicator: '#a4c940', awayIndicator: '#dcbd4e', dndIndicator: '#d24b4e', + mentionBg: '#ffffff', mentionColor: '#1f1f1f', centerChannelBg: '#1f1f1f', centerChannelColor: '#dddddd', + newMessageSeparator: '#cc8f00', linkColor: '#a4ffeb', buttonBg: '#a4ffeb', buttonColor: '#1f1f1f', + errorTextColor: '#ff6461', mentionHighlightBg: '#784098', mentionHighlightLink: '#a4ffeb', codeTheme: 'solarized-dark', +}; + +async function api(method, path, body, token) { + const res = await fetch(`${BASE}/api/v4${path}`, { + method, + headers: { 'Content-Type': 'application/json', ...(token ? { Authorization: `Bearer ${token}` } : {}) }, + body: body ? JSON.stringify(body) : undefined, + }); + if (!res.ok) throw new Error(`${method} ${path} -> ${res.status}: ${(await res.text()).slice(0, 200)}`); + return res; +} + +const login = await api('POST', '/users/login', { login_id: boot.testuser.username, password: boot.testuser.password }); +const token = login.headers.get('Token'); +const me = await login.json(); +await api('PUT', `/users/${me.id}/preferences`, [ + { user_id: me.id, category: 'theme', name: '', value: JSON.stringify(ONYX) }, + { user_id: me.id, category: 'tutorial_step', name: me.id, value: '999' }, + { user_id: me.id, category: 'onboarding_task_list', name: 'onboarding_task_list_show', value: 'false' }, + { user_id: me.id, category: 'recommended_next_steps', name: 'hide', value: 'true' }, +], token); + +const root = await (await api('POST', '/posts', { channel_id: boot.channel.id, message: '@netclaw push the website changes' }, token)).json(); +const thread = async () => { + const d = await (await api('GET', `/posts/${root.id}/thread`, null, token)).json(); + return d.order.map(id => d.posts[id]).sort((a, b) => a.create_at - b.create_at); +}; +let prompt; +for (const t0 = Date.now(); Date.now() - t0 < TIMEOUT_MS && !prompt; await new Promise(r => setTimeout(r, 2000))) + prompt = (await thread()).find(p => p.user_id === boot.bot.userId && /Tool approval required/i.test(p.message)); +if (!prompt) { console.error('no approval prompt within the timeout'); process.exit(1); } +const actions = (prompt.props?.attachments?.[0]?.actions || []).map(a => a.name); +console.log('prompt text:\n' + prompt.message + '\nbuttons: ' + JSON.stringify(actions)); +if (!actions.length) { console.error('the prompt has no buttons; CallbackUrl is not taking effect'); process.exit(1); } + +const browser = await chromium.launch(); +const ctx = await browser.newContext({ viewport: { width: 1280, height: 900 }, deviceScaleFactor: 2, colorScheme: 'dark' }); +await ctx.addInitScript(() => { try { localStorage.setItem('__landingPageSeen__', 'true'); localStorage.setItem('onboarding_complete', 'true'); } catch {} }); +const page = await ctx.newPage(); +await page.goto(`${BASE}/login`, { waitUntil: 'networkidle' }); +for (const rx of [/view in browser/i, /continue in browser/i]) { + try { const l = page.getByText(rx).first(); if (await l.isVisible({ timeout: 1500 })) { await l.click(); break; } } catch {} +} +await page.waitForSelector('input[type="password"]', { timeout: 30000 }); +await page.locator('input[type="text"], input[type="email"]').first().fill(boot.testuser.username); +await page.locator('input[type="password"]').first().fill(boot.testuser.password); +await page.getByRole('button', { name: /log in|sign in/i }).first().click(); +await page.waitForURL(u => !/\/login/.test(new URL(u).pathname), { timeout: 20000 }); +await page.waitForLoadState('networkidle', { timeout: 15000 }).catch(() => {}); + +await page.goto(`${BASE}/${boot.team.name}/pl/${root.id}`, { waitUntil: 'networkidle' }); +await page.waitForTimeout(2500); +await page.getByRole('button', { name: /\d+ repl(y|ies)/i }).last().click(); +await page.waitForTimeout(2500); +await page.getByRole('button', { name: /expand/i }).first().click(); // widen the panel so the buttons fit one row +await page.waitForTimeout(1500); +await page.mouse.move(640, 880); +await page.addStyleTag({ content: '.NotificationSeparator{display:none !important}' }); // the "New Messages" line +await page.waitForTimeout(500); +const post = page.locator(`#sidebar-right #post_${prompt.id}, #sidebar-right [id$="${prompt.id}"]`).first(); +await post.waitFor({ timeout: 15000 }); +await page.screenshot({ path: join(OUT, '_approval-full.png') }); +const raw = join(OUT, '_approval-raw.png'); +await post.screenshot({ path: raw }); +await frame(browser, raw, join(OUT, 'approval-prompt.png')); +await browser.close(); +console.log(' ok approval-prompt.png'); diff --git a/screenshots/mattermost/bootstrap.mjs b/screenshots/mattermost/bootstrap.mjs index c6a8c35..4bb79b8 100644 --- a/screenshots/mattermost/bootstrap.mjs +++ b/screenshots/mattermost/bootstrap.mjs @@ -2,16 +2,19 @@ // Bootstraps a fresh mattermost-preview the same way the netclaw repo's Aspire // demo does: first user (auto system-admin), team, bot + access token, channel, // and a non-admin test user. Writes the resulting IDs/token/credentials to -// .bootstrap.json for run.sh (netclaw config) and capture.mjs (Playwright). +// BOOTSTRAP_FILE for run.sh (netclaw config) and capture.mjs (Playwright). // -// Usage: node bootstrap.mjs [serverUrl] (default http://localhost:8065) +// Runs inside the Playwright container (run.sh), on the same compose network as Mattermost. +// Usage: node bootstrap.mjs [serverUrl] (default http://mattermost:8065) +// Env: BOOTSTRAP_FILE where to write the result (default ./.bootstrap.json) import { writeFileSync } from 'fs'; import { dirname, join } from 'path'; import { fileURLToPath } from 'url'; -const SERVER = (process.argv[2] || 'http://localhost:8065').replace(/\/$/, ''); +const SERVER = (process.argv[2] || 'http://mattermost:8065').replace(/\/$/, ''); const __dirname = dirname(fileURLToPath(import.meta.url)); +const BOOTSTRAP_FILE = process.env.BOOTSTRAP_FILE || join(__dirname, '.bootstrap.json'); const ADMIN = { email: 'admin@test.local', username: 'admin', password: 'Admin1234!' }; const USER = { email: 'testuser@test.local', username: 'testuser', password: 'TestUser1234!' }; @@ -97,9 +100,9 @@ async function main() { channel: { id: channel.id, name: channel.name }, bot: { userId: bot.user_id, username: bot.username, token: tok.token }, }; - writeFileSync(join(__dirname, '.bootstrap.json'), JSON.stringify(out, null, 2)); + writeFileSync(BOOTSTRAP_FILE, JSON.stringify(out, null, 2)); console.log(`\n✅ Bootstrap complete. channel=${channel.id} bot=${bot.user_id}`); - console.log(` Wrote ${join(__dirname, '.bootstrap.json')}`); + console.log(` Wrote ${BOOTSTRAP_FILE}`); } main().catch(err => { console.error('❌ Bootstrap failed:', err.message); process.exit(1); }); diff --git a/screenshots/mattermost/capture-compose.mjs b/screenshots/mattermost/capture-compose.mjs new file mode 100644 index 0000000..d181de5 --- /dev/null +++ b/screenshots/mattermost/capture-compose.mjs @@ -0,0 +1,266 @@ +#!/usr/bin/env node +// Captures the Mattermost screenshots for channels/mattermost.md and +// guides/mattermost-docker-compose.md from the stack in examples/mattermost-compose/ +// (provisioned with provision.sh, wired with configure-netclaw.sh). +// +// One consistent look: Chromium, 1280x800 viewport, device scale factor 2, the Onyx +// dark theme, no browser chrome (page screenshots, cropped to the relevant region). +// Token IDs on screen are overwritten in the page before the screenshot is taken. +// +// Usage (Playwright container or any Node with playwright installed): +// MM_URL=http://localhost:8065 ADMIN_PASSWORD=... ALICE_PASSWORD=... \ +// node capture-compose.mjs [step ...] +// Steps (default: all): admin bots thread approval +// Env: TEAM (home), CHANNEL (assistant), OUT_DIR (../output), +// REPO_DIR (directory the approval prompt runs in), REPLY_TIMEOUT_SECONDS (300) + +import { chromium } from 'playwright'; +import { mkdirSync } from 'fs'; +import { dirname, join } from 'path'; +import { fileURLToPath } from 'url'; + +const here = dirname(fileURLToPath(import.meta.url)); +const BASE = (process.env.MM_URL || 'http://localhost:8065').replace(/\/$/, ''); +const TEAM = process.env.TEAM || 'home'; +const CHANNEL = process.env.CHANNEL || 'assistant'; +const OUT = process.env.OUT_DIR || join(here, '..', 'output'); +const REPO_DIR = process.env.REPO_DIR || '~/.netclaw/projects/website'; +const TIMEOUT_MS = Number(process.env.REPLY_TIMEOUT_SECONDS || 300) * 1000; +const ADMIN = { username: 'admin', password: process.env.ADMIN_PASSWORD }; +const ALICE = { username: 'alice', password: process.env.ALICE_PASSWORD }; +if (!ADMIN.password || !ALICE.password) { console.error('ADMIN_PASSWORD and ALICE_PASSWORD are required'); process.exit(2); } +mkdirSync(OUT, { recursive: true }); + +const steps = new Set(process.argv.slice(2).length ? process.argv.slice(2) : ['admin', 'bots', 'thread', 'approval']); +const VIEWPORT = { width: 1280, height: 800 }; +const MASK = 'x'.repeat(26); + +// Mattermost's Onyx theme. +const ONYX = { + type: 'Onyx', sidebarBg: '#18181a', sidebarText: '#ffffff', sidebarUnreadText: '#ffffff', + sidebarTextHoverBg: '#25262a', sidebarTextActiveBorder: '#afbe5d', sidebarTextActiveColor: '#ffffff', + sidebarHeaderBg: '#1f1f1f', sidebarTeamBarBg: '#121212', sidebarHeaderTextColor: '#ffffff', + onlineIndicator: '#a4c940', awayIndicator: '#dcbd4e', dndIndicator: '#d24b4e', + mentionBg: '#ffffff', mentionColor: '#1f1f1f', centerChannelBg: '#1f1f1f', centerChannelColor: '#dddddd', + newMessageSeparator: '#cc8f00', linkColor: '#a4ffeb', buttonBg: '#a4ffeb', buttonColor: '#1f1f1f', + errorTextColor: '#ff6461', mentionHighlightBg: '#784098', mentionHighlightLink: '#a4ffeb', codeTheme: 'solarized-dark', +}; + +async function api(method, path, body, token) { + const res = await fetch(`${BASE}/api/v4${path}`, { + method, + headers: { 'Content-Type': 'application/json', ...(token ? { Authorization: `Bearer ${token}` } : {}) }, + body: body ? JSON.stringify(body) : undefined, + }); + if (!res.ok) throw new Error(`${method} ${path} -> ${res.status}: ${(await res.text()).slice(0, 200)}`); + return res; +} + +async function apiLogin(user) { + const res = await api('POST', '/users/login', { login_id: user.username, password: user.password }); + const token = res.headers.get('Token'); + const me = await res.json(); + user.token = token; user.id = me.id; + await api('PUT', `/users/${me.id}/preferences`, [ + { user_id: me.id, category: 'theme', name: '', value: JSON.stringify(ONYX) }, + { user_id: me.id, category: 'tutorial_step', name: me.id, value: '999' }, + { user_id: me.id, category: 'onboarding_task_list', name: 'onboarding_task_list_show', value: 'false' }, + { user_id: me.id, category: 'recommended_next_steps', name: 'hide', value: 'true' }, + ], token).catch(e => console.warn(' preferences:', e.message)); + return token; +} + +async function newPage(browser, user, viewport = VIEWPORT) { + const ctx = await browser.newContext({ viewport, deviceScaleFactor: 2, colorScheme: 'dark' }); + await ctx.addInitScript(() => { + try { localStorage.setItem('__landingPageSeen__', 'true'); localStorage.setItem('onboarding_complete', 'true'); } catch {} + }); + const page = await ctx.newPage(); + await page.goto(`${BASE}/login`, { waitUntil: 'networkidle' }); + for (const rx of [/view in browser/i, /continue in browser/i]) { + try { const l = page.getByText(rx).first(); if (await l.isVisible({ timeout: 1500 })) { await l.click(); break; } } catch {} + } + await page.waitForSelector('input[type="password"]', { timeout: 30000 }); + await page.locator('input[type="text"], input[type="email"]').first().fill(user.username); + await page.locator('input[type="password"]').first().fill(user.password); + await page.getByRole('button', { name: /log in|sign in/i }).first().click(); + await page.waitForURL(u => !/\/login/.test(new URL(u).pathname), { timeout: 20000 }); + await page.waitForLoadState('networkidle', { timeout: 15000 }).catch(() => {}); + return { ctx, page }; + +} + +async function dismissModals(page) { + for (const rx of [/view in browser/i, /no thanks/i, /skip/i, /got it/i, /not now/i]) { + for (const role of ['link', 'button']) { + try { + const el = page.getByRole(role, { name: rx }).first(); + if (await el.isVisible({ timeout: 300 })) { await el.click({ timeout: 1000 }); await page.waitForTimeout(300); } + } catch {} + } + } +} + +// The "Preview Mode: Email notifications have not been configured" banner is a stack detail. +// Dismiss it the way a person would, with its close button. +async function closeBanner(page) { + try { + const bar = page.locator('.announcement-bar').first(); + if (await bar.isVisible({ timeout: 1500 })) { + await bar.locator('.announcement-bar__close, .close, [aria-label="Close"]').first().click({ timeout: 2000 }); + await page.waitForTimeout(500); + } + } catch {} +} + +const maskTokenIds = page => page.evaluate(mask => { + // Overwrite anything shaped like a 26-character Mattermost ID inside a "Token ID" line. + const walker = document.createTreeWalker(document.body, NodeFilter.SHOW_TEXT); + const hits = []; + while (walker.nextNode()) { + const n = walker.currentNode; + if (/^[a-z0-9]{26}$/.test(n.nodeValue.trim()) && /Token ID/i.test(n.parentElement?.parentElement?.textContent || '')) hits.push(n); + } + for (const n of hits) n.nodeValue = mask; + return hits.length; +}, MASK); + +const shot = async (page, name, clip) => { + await page.screenshot({ path: join(OUT, name), ...(clip ? { clip } : {}) }); + console.log(` ok ${name}`); +}; + +async function waitFor(fn, what) { + const start = Date.now(); + while (Date.now() - start < TIMEOUT_MS) { + const v = await fn(); + if (v) return v; + await new Promise(r => setTimeout(r, 2500)); + } + throw new Error(`timed out waiting for ${what}`); +} + +const browser = await chromium.launch(); +let failures = 0; +const attempt = async (name, fn) => { try { await fn(); } catch (e) { failures++; console.error(` FAIL ${name}: ${e.message}`); } }; + +await apiLogin(ADMIN); +await apiLogin(ALICE); +const channel = (await (await api('GET', `/teams/name/${TEAM}/channels/name/${CHANNEL}`, null, ALICE.token)).json()); + +if (steps.has('admin')) await attempt('admin', async () => { + const { ctx, page } = await newPage(browser, ADMIN); + await page.goto(`${BASE}/admin_console/integrations/bot_accounts`, { waitUntil: 'networkidle' }); + await page.waitForTimeout(2000); + await dismissModals(page); + await closeBanner(page); + await shot(page, 'mattermost-bot-accounts.png', { x: 220, y: 44, width: 1060, height: 376 }); + await ctx.close(); +}); + +if (steps.has('bots')) await attempt('bots', async () => { + const { ctx, page } = await newPage(browser, ADMIN); + await page.goto(`${BASE}/${TEAM}/integrations/bots`, { waitUntil: 'networkidle' }); + await page.waitForTimeout(2000); + await dismissModals(page); + await closeBanner(page); + console.log(` masked ${await maskTokenIds(page)} token id(s)`); + await shot(page, 'mattermost-create-bot.png', { x: 16, y: 76, width: 1240, height: 610 }); + await shot(page, 'mattermost-compose-bot-account.png', { x: 290, y: 372, width: 940, height: 150 }); + await ctx.close(); +}); + +const botUserId = async () => (await (await api('GET', '/users/username/netclaw', null, ALICE.token)).json()).id; + +async function postAsAlice(message) { + const res = await api('POST', '/posts', { channel_id: channel.id, message }, ALICE.token); + return (await res.json()); +} + +// Start each conversation from an empty channel so only that conversation is on screen. +async function clearChannel() { + const data = await (await api('GET', `/channels/${channel.id}/posts?per_page=200`, null, ADMIN.token)).json(); + for (const id of data.order) { + const p = data.posts[id]; + if (p.type === '' && !p.root_id) await api('DELETE', `/posts/${id}`, null, ADMIN.token).catch(() => {}); + } + await new Promise(r => setTimeout(r, 1000)); +} + +async function threadPosts(rootId) { + const data = await (await api('GET', `/posts/${rootId}/thread`, null, ALICE.token)).json(); + return data.order.map(id => data.posts[id]).sort((a, b) => a.create_at - b.create_at); +} + +async function openThread(page, rootId) { + await page.goto(`${BASE}/${TEAM}/pl/${rootId}`, { waitUntil: 'networkidle' }); + await page.waitForTimeout(2500); + await dismissModals(page); + await closeBanner(page); + // The permalink only highlights the post. Open the thread panel like a person would. + await page.getByRole('button', { name: /\d+ repl(y|ies)/i }).last().click(); + await page.waitForTimeout(2500); + await page.mouse.move(640, 790); +} + +if (steps.has('thread')) await attempt('thread', async () => { + const bot = await botUserId(); + await clearChannel(); + const root = await postAsAlice('@netclaw reply with a one-sentence greeting'); + await waitFor(async () => (await threadPosts(root.id)).some(p => p.user_id === bot && p.message), 'the bot reply'); + const { ctx, page } = await newPage(browser, ALICE); + await openThread(page, root.id); + await shot(page, 'mattermost-conversation.png'); + await shot(page, 'mattermost-compose-thread.png', { x: 270, y: 45, width: 965, height: 755 }); + await ctx.close(); +}); + +if (steps.has('approval')) await attempt('approval', async () => { + const bot = await botUserId(); + await clearChannel(); + const root = await postAsAlice(`@netclaw Use the shell to run exactly this, nothing else: git -C ${REPO_DIR} push origin main`); + const prompt = await waitFor(async () => (await threadPosts(root.id)).find(p => p.user_id === bot && /Tool approval required/i.test(p.message)), 'the approval prompt'); + console.log('--- prompt text ---\n' + prompt.message + '\n-------------------'); + const { ctx, page } = await newPage(browser, ALICE, { width: 1280, height: 900 }); + await openThread(page, root.id); + // Widen the thread panel so the command doesn't wrap. + await page.getByRole('button', { name: /expand/i }).first().click(); + await page.waitForTimeout(1500); + await page.mouse.move(640, 880); + const rhs = page.locator('#sidebar-right'); + const rb = await rhs.boundingBox(); + const posts = rhs.locator('.post'); + const fb = await posts.first().boundingBox(); + const lb = await posts.last().boundingBox(); + console.log('rhs', JSON.stringify(rb), 'first', JSON.stringify(fb), 'last', JSON.stringify(lb)); + await page.screenshot({ path: join(OUT, '_pending-full.png') }); + // Landing-page crop (public/assets/approval-prompt.png): just the pending prompt post. + await page.screenshot({ path: join(OUT, 'approval-prompt.png'), clip: { x: rb.x + 2, y: lb.y + 8, width: rb.width - 4, height: 270 } }); + console.log(' ok approval-prompt.png'); + // The guide's image: alice's request, the pending prompt with all six choices, and her + // one-letter reply typed in the thread box. A prompt resolves the instant the reply is + // sent (the post is edited in place), so this is the last frame that still shows the choices. + await page.locator('#reply_textbox').click(); + await page.keyboard.type('A'); + await page.waitForTimeout(400); + const y0 = fb.y - 24; + await page.screenshot({ path: join(OUT, 'mattermost-compose-approval.png'), clip: { x: rb.x + 2, y: y0, width: rb.width - 4, height: rb.y + rb.height - 22 - y0 } }); + console.log(' ok mattermost-compose-approval.png'); + await page.keyboard.press('Enter'); + await waitFor(async () => { + const posts = await threadPosts(root.id); + const last = posts[posts.length - 1]; + return posts.some(p => /Tool approval resolved/i.test(p.message)) && last.user_id === bot && !/Tool approval/i.test(last.message); + }, 'the approval to resolve and the bot to finish'); + await page.waitForTimeout(2500); + await page.mouse.move(640, 880); + const top = (await posts.first().boundingBox()).y - 24; + const boxTop = (await page.locator('#reply_textbox').boundingBox()).y; + await page.screenshot({ path: join(OUT, 'mattermost-compose-approval-resolved.png'), clip: { x: rb.x + 2, y: top, width: rb.width - 4, height: boxTop - 8 - top } }); + console.log(' ok mattermost-compose-approval-resolved.png'); + await ctx.close(); +}); + +await browser.close(); +if (failures) { console.error(`${failures} step(s) failed`); process.exit(1); } +console.log('Done.'); diff --git a/screenshots/mattermost/capture.mjs b/screenshots/mattermost/capture.mjs index f042351..59fd316 100644 --- a/screenshots/mattermost/capture.mjs +++ b/screenshots/mattermost/capture.mjs @@ -1,10 +1,12 @@ #!/usr/bin/env node // Drives Playwright against the bootstrapped Mattermost to capture the // channels/mattermost.md screenshots: the System Console settings, the bot -// account, and a live netclaw conversation. Reads .bootstrap.json. +// account, and a live netclaw conversation. Runs in the Playwright container (run.sh). // -// Each shot is independent (try/catch) so one failure doesn't lose the rest. +// Each shot is independent (try/catch) so one failure doesn't lose the rest; the +// process exits non-zero if any shot failed. // Usage: node capture.mjs +// Env: BOOTSTRAP_FILE (default ./.bootstrap.json), OUT_DIR (default ../output) import { chromium } from 'playwright'; import { readFileSync, mkdirSync } from 'fs'; @@ -12,10 +14,11 @@ import { dirname, join } from 'path'; import { fileURLToPath } from 'url'; const __dirname = dirname(fileURLToPath(import.meta.url)); -const boot = JSON.parse(readFileSync(join(__dirname, '.bootstrap.json'), 'utf8')); -const OUT = join(__dirname, '..', 'output'); +const boot = JSON.parse(readFileSync(process.env.BOOTSTRAP_FILE || join(__dirname, '.bootstrap.json'), 'utf8')); +const OUT = process.env.OUT_DIR || join(__dirname, '..', 'output'); mkdirSync(OUT, { recursive: true }); const BASE = boot.serverUrl; +let failures = 0; const shot = (page, name) => page.screenshot({ path: join(OUT, name) }).then(() => console.log(` ✓ ${name}`)); @@ -86,7 +89,7 @@ try { await dismissModals(page); await shot(page, 'mattermost-create-bot.png'); await ctx.close(); -} catch (e) { console.error(' ✗ admin shots:', e.message); } +} catch (e) { failures++; console.error(' ✗ admin shots:', e.message); } // 3: live conversation as the test user try { @@ -118,7 +121,8 @@ try { await page.waitForTimeout(1000); await shot(page, 'mattermost-conversation.png'); await ctx.close(); -} catch (e) { console.error(' ✗ conversation shot:', e.message); } +} catch (e) { failures++; console.error(' ✗ conversation shot:', e.message); } await browser.close(); +if (failures) { console.error(`${failures} capture step(s) failed`); process.exit(1); } console.log('Done.'); diff --git a/screenshots/mattermost/docker-compose.mattermost.yml b/screenshots/mattermost/docker-compose.mattermost.yml index 2769691..963e5db 100644 --- a/screenshots/mattermost/docker-compose.mattermost.yml +++ b/screenshots/mattermost/docker-compose.mattermost.yml @@ -1,24 +1,106 @@ -# Mattermost server for capturing the channels/mattermost.md screenshots. -# Brought up by run.sh, which then bootstraps it (bootstrap.mjs), starts the -# pinned netclaw image against it, and drives Playwright (capture.mjs). +# Mattermost capture topology. Everything runs in containers on one compose network; +# nothing is published to the host and there are no fixed container names, so projects +# do not collide. Driven by run.sh (do not run directly). # -# mattermost-preview is an all-in-one testing image (bundled Postgres). amd64-only. +# mattermost all-in-one testing image (bundled Postgres, amd64 only) +# netclawd the released daemon; reaches Mattermost at http://mattermost:8065 +# playwright bootstrap + browser captures, reaching Mattermost by the same name +# +# Models use the 0.25+ Models.Definitions / Models.Roles env shape. The bot token is +# NOT an env var: run.sh stores it with `netclaw secrets set Mattermost.BotToken`. +# run.sh sets Mattermost.CallbackUrl to http://netclawd:5299/api/mattermost/actions (a forwarder +# sidecar, because the daemon listens on loopback only) so approvals +# render as buttons. Mattermost refuses integration callbacks to private addresses, so it +# allow-lists the netclawd host below. services: mattermost: - image: mattermost/mattermost-preview:latest - ports: - - "8065:8065" + image: ${MATTERMOST_IMAGE:-mattermost/mattermost-preview:11.7.7} environment: - MM_SERVICESETTINGS_SITEURL: http://localhost:8065 + MM_SERVICESETTINGS_SITEURL: http://mattermost:8065 MM_TEAMSETTINGS_ENABLEOPENSERVER: "true" MM_SERVICESETTINGS_ENABLETESTING: "true" - # NOTE: EnableBotAccountCreation / EnableUserAccessTokens are intentionally - # NOT set here. bootstrap.mjs flips them via the System Console config API so - # the captured screenshot shows them as editable toggles (an env var would - # render them locked: "set through an environment variable"). + # Button callbacks go to the daemon on the private compose network; Mattermost blocks that + # unless the host is listed. + MM_SERVICESETTINGS_ALLOWEDUNTRUSTEDINTERNALCONNECTIONS: netclawd + # EnableBotAccountCreation / EnableUserAccessTokens are intentionally NOT set here. + # bootstrap.mjs flips them through the config API so the captured System Console + # page shows editable toggles (an env var would render them locked). + # No healthcheck: the 11.x preview image ships neither curl nor wget. bootstrap.mjs + # polls /api/v4/system/ping until the server is ready. + + netclawd: + image: ${NETCLAW_IMAGE:?set by run.sh} + depends_on: + - mattermost + environment: + NETCLAW_Daemon__DisableSelfUpdate: "true" + NETCLAW_Providers__inference__Type: ${NC_INFERENCE_TYPE:?set by run.sh} + NETCLAW_Providers__inference__Endpoint: ${NC_INFERENCE_ENDPOINT:?set by run.sh} + NETCLAW_Models__Definitions__main__Provider: inference + NETCLAW_Models__Definitions__main__ModelId: ${NC_INFERENCE_MODEL:?set by run.sh} + NETCLAW_Models__Roles__Main: main + NETCLAW_Models__Roles__Fallback: main + NETCLAW_Models__Roles__Compaction: main + # Real inference endpoints can be slow to first token. + NETCLAW_Session__TurnLlmTimeoutSeconds: "600" + NETCLAW_Session__FirstTokenTimeoutSeconds: "600" + NETCLAW_Session__PrefillTimeoutSeconds: "600" + volumes: + - netclaw-home:/home/netclaw/.netclaw healthcheck: - test: ["CMD", "curl", "-sf", "http://localhost:8065/api/v4/system/ping"] - interval: 5s + test: ["CMD", "curl", "-sf", "http://127.0.0.1:5199/api/health/ready"] + interval: 3s timeout: 5s - retries: 48 - start_period: 20s + retries: 40 + start_period: 5s + + # Shares netclawd's network namespace so it can reach the loopback-only daemon listener and + # re-expose it on :5299 for Mattermost's button callbacks. + callbackfwd: + image: netclaw-docs-playwright:${PLAYWRIGHT_VERSION:?set by run.sh} + network_mode: service:netclawd + user: "${HOST_UID:-1000}:${HOST_GID:-1000}" + command: ["node", "/work/tcp-forward.mjs"] + depends_on: + - netclawd + volumes: + - .:/work:ro + + # Only started by run.sh with SCRIPTED_APPROVAL=1: answers with one scripted shell_execute call (no real model). + scriptedllm: + profiles: ["scripted"] + image: netclaw-docs-playwright:${PLAYWRIGHT_VERSION:?set by run.sh} + user: "${HOST_UID:-1000}:${HOST_GID:-1000}" + command: ["node", "/work/scripted-llm.mjs"] + environment: + SCRIPTED_TOOL: ${SCRIPTED_TOOL:-mcp} + SCRIPTED_COMMAND: ${SCRIPTED_COMMAND:-git push origin main} + volumes: + - .:/work:ro + + # The stack's demo MCP server (screenshots/mcp-server); approval.sh-style runs register it + # with netclawd and gate http_fetch behind approval. + mcp-demo: + profiles: ["scripted"] + image: netclaw-docs-mcp-demo:1 + build: + context: ../mcp-server + + playwright: + profiles: ["tools"] + image: netclaw-docs-playwright:${PLAYWRIGHT_VERSION:?set by run.sh} + build: + context: ../playwright + args: + PLAYWRIGHT_VERSION: ${PLAYWRIGHT_VERSION} + # Run as the invoking user so files written to the mounts are yours, not root's. + user: "${HOST_UID:-1000}:${HOST_GID:-1000}" + environment: + BOOTSTRAP_FILE: /work/.bootstrap.${COMPOSE_PROJECT_NAME}.json + OUT_DIR: /out + volumes: + - .:/work + - ${CAPTURE_OUT_DIR:?set by run.sh}:/out + +volumes: + netclaw-home: diff --git a/screenshots/mattermost/frame-approval.mjs b/screenshots/mattermost/frame-approval.mjs new file mode 100644 index 0000000..2796afc --- /dev/null +++ b/screenshots/mattermost/frame-approval.mjs @@ -0,0 +1,34 @@ +#!/usr/bin/env node +// Frames a raw crop of the approval prompt the way the landing page shows it: a rounded dark +// card with padding and a soft purple glow on a near-black background (colours sampled from the +// original Slack-era image). Output is a PNG at 2x. +// +// Library: import { frame } from './frame-approval.mjs'; await frame(browser, rawPath, outPath, { bg }) +// CLI: node frame-approval.mjs raw.png out.png [cardBackgroundCss] +import { readFileSync } from 'fs'; +import { fileURLToPath } from 'url'; + +export async function frame(browser, rawPath, outPath, { cardBg = '#1f1f1f', dsf = 2 } = {}) { + const b64 = readFileSync(rawPath).toString('base64'); + const ctx = await browser.newContext({ deviceScaleFactor: dsf, viewport: { width: 1400, height: 900 } }); + const page = await ctx.newPage(); + await page.setContent(`
`); + await page.waitForFunction(() => document.getElementById('shot').complete); + // Show the raw capture at its CSS size (it was taken at the same scale factor). + await page.evaluate(d => { const i = document.getElementById('shot'); i.style.width = (i.naturalWidth / d) + 'px'; }, dsf); + await page.locator('#stage').screenshot({ path: outPath }); + await ctx.close(); +} + +if (process.argv[1] === fileURLToPath(import.meta.url)) { + const { chromium } = await import('playwright'); + const browser = await chromium.launch(); + await frame(browser, process.argv[2], process.argv[3], process.argv[4] ? { cardBg: process.argv[4] } : {}); + await browser.close(); +} diff --git a/screenshots/mattermost/post-message.mjs b/screenshots/mattermost/post-message.mjs new file mode 100644 index 0000000..c985103 --- /dev/null +++ b/screenshots/mattermost/post-message.mjs @@ -0,0 +1,44 @@ +#!/usr/bin/env node +// Posts an @netclaw message as the test user, then waits until the bot replies. +// Exits non-zero if no reply arrives in time (the daemon or inference is broken). +// Runs in the Playwright container (run.sh). +// +// Usage: node post-message.mjs ["message text"] +// Env: BOOTSTRAP_FILE (default ./.bootstrap.json), REPLY_TIMEOUT_SECONDS (default 300) + +import { readFileSync } from 'fs'; +import { dirname, join } from 'path'; +import { fileURLToPath } from 'url'; + +const __dirname = dirname(fileURLToPath(import.meta.url)); +const boot = JSON.parse(readFileSync(process.env.BOOTSTRAP_FILE || join(__dirname, '.bootstrap.json'), 'utf8')); +const MESSAGE = process.argv[2] || '@netclaw hey! In one sentence, what are you and what can you help with?'; +const TIMEOUT_MS = Number(process.env.REPLY_TIMEOUT_SECONDS || 300) * 1000; + +async function api(method, path, body, token) { + const res = await fetch(`${boot.serverUrl}${path}`, { + method, + headers: { 'Content-Type': 'application/json', ...(token ? { Authorization: `Bearer ${token}` } : {}) }, + body: body ? JSON.stringify(body) : undefined, + }); + if (!res.ok) throw new Error(`${method} ${path} -> ${res.status}: ${(await res.text()).slice(0, 200)}`); + return res; +} + +const login = await api('POST', '/api/v4/users/login', { login_id: boot.testuser.username, password: boot.testuser.password }); +const userToken = login.headers.get('Token'); +await api('POST', '/api/v4/posts', { channel_id: boot.channel.id, message: MESSAGE }, userToken); +console.log(`posted: ${MESSAGE}`); + +const start = Date.now(); +while (Date.now() - start < TIMEOUT_MS) { + const data = await (await api('GET', `/api/v4/channels/${boot.channel.id}/posts`, null, boot.bot.token)).json(); + const reply = data.order.map(id => data.posts[id]).find(p => p.user_id === boot.bot.userId && p.root_id && p.message); + if (reply) { + console.log(`bot replied after ${Math.round((Date.now() - start) / 1000)}s: ${reply.message.slice(0, 200).replace(/\s+/g, ' ')}`); + process.exit(0); + } + await new Promise(r => setTimeout(r, 3000)); +} +console.error(`no bot reply within ${TIMEOUT_MS / 1000}s`); +process.exit(1); diff --git a/screenshots/mattermost/run.sh b/screenshots/mattermost/run.sh index e7c8d89..3a89804 100755 --- a/screenshots/mattermost/run.sh +++ b/screenshots/mattermost/run.sh @@ -1,83 +1,137 @@ #!/bin/bash -# Orchestrates the Mattermost screenshot capture end to end: -# 1. brings up the Mattermost container -# 2. bootstraps it (admin/team/bot/token/channel/test user) -> .bootstrap.json -# 3. starts the pinned netclaw daemon against it, with inference pointed at -# an OpenAI-compatible endpoint (spark2 by default) -# 4. posts an @netclaw message as the test user so the bot replies -# 5. drives Playwright (capture.mjs) to shoot the System Console + live thread -# 6. tears everything down +# Regenerate the Mattermost screenshots, entirely in containers (Docker is the only +# prerequisite): Mattermost, the released netclaw daemon and Playwright share one +# compose network. Needs real inference so the bot can answer: # -# The daemon uses `--network host` so its loopback control plane is reachable -# from the host, and Mattermost is reached at http://localhost:8065. +# INFERENCE_BASE_URL=http://llm.example.internal:8080 INFERENCE_MODEL_ID= \ +# ./screenshots/mattermost/run.sh # -# Overridable env: -# NETCLAW_IMAGE default ghcr.io/netclaw-dev/netclaw:0.22.1 -# SPARK_ENDPOINT OpenAI-compatible inference endpoint (default spark2) -# SPARK_MODEL_ID model id served there (matches the local netclaw instance) +# Landing-page approval prompt (public/assets/approval-prompt.png) needs no model: set +# SCRIPTED_APPROVAL=1 and a scripted endpoint (scripted-llm.mjs) requests one demo-utilities http_fetch call +# (SCRIPTED_TOOL=shell for a shell_execute call instead); +# the run then captures the prompt with its buttons, framed by frame-approval.mjs, into OUT_DIR. +# +# (or put INFERENCE_UPSTREAM and INFERENCE_UPSTREAM_MODEL in screenshots/.env, gitignored). Other knobs: +# COMPOSE_PROJECT_NAME prefix for this run; "-mm" is appended (default netclaw-shots-mm) +# NETCLAW_VERSION / NETCLAW_IMAGE daemon image (default: src/version.json) +# OUT_DIR where PNGs go (default screenshots/output) +# REPLY_TIMEOUT_SECONDS how long to wait for the bot's reply (default 300) +# +# Steps: compose up Mattermost -> bootstrap (admin, team, bot, channel, test user) -> +# start netclawd -> store the bot token as an encrypted secret and enable the Mattermost +# channel -> post an @netclaw message as the test user and wait for the reply -> +# Playwright captures -> tear everything down. Exits non-zero on any failure. set -euo pipefail -SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" -cd "$SCRIPT_DIR" +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +COMPOSE_PROJECT_NAME="${COMPOSE_PROJECT_NAME:-netclaw-shots}-mm" +export COMPOSE_PROJECT_NAME +# shellcheck source=../lib.sh +. "$HERE/../lib.sh" + +if [ -n "${SCRIPTED_APPROVAL:-}" ]; then + export NC_INFERENCE_TYPE=openai-compatible NC_INFERENCE_ENDPOINT=http://scriptedllm:8080/v1 NC_INFERENCE_MODEL=demo-model +elif [ "$INFERENCE_MODE" = relay ]; then + # This flow runs on its own compose network without the forwarder, and no frame shows the + # daemon's configuration, so the daemon talks to the upstream directly. + export NC_INFERENCE_ENDPOINT="$INFERENCE_UPSTREAM" NC_INFERENCE_MODEL="$INFERENCE_UPSTREAM_MODEL" +elif [ "$INFERENCE_MODE" != external ]; then + echo "mattermost: set INFERENCE_UPSTREAM (or INFERENCE_BASE_URL and INFERENCE_MODEL_ID); the bot needs a real model to answer" >&2 + exit 2 +fi -NETCLAW_IMAGE="${NETCLAW_IMAGE:-ghcr.io/netclaw-dev/netclaw:0.22.1}" -SPARK_ENDPOINT="${SPARK_ENDPOINT:-https://spark2.testlab.petabridge.net}" -SPARK_MODEL_ID="${SPARK_MODEL_ID:-Qwen/Qwen3.6-35B-A3B-FP8}" -SERVER="http://localhost:8065" +OUT_DIR="${OUT_DIR:-$SHOTS_DIR/output}" +mkdir -p "$OUT_DIR" +export CAPTURE_OUT_DIR="$(cd "$OUT_DIR" && pwd)" +BOOTSTRAP=".bootstrap.${COMPOSE_PROJECT_NAME}.json" + +mm() { docker compose -f "$HERE/docker-compose.mattermost.yml" "$@"; } +pw() { mm run --rm --no-deps -T playwright "$@"; } +nc() { mm exec -T -u netclaw netclawd netclaw "$@"; } cleanup() { - echo "==> Tearing down" - docker rm -f netclaw-mm-shot >/dev/null 2>&1 || true - docker compose -f docker-compose.mattermost.yml down -v >/dev/null 2>&1 || true - rm -f .bootstrap.json + if [ -n "${KEEP_STACK:-}" ]; then echo "==> KEEP_STACK set: leaving project $COMPOSE_PROJECT_NAME running"; return; fi + echo "==> Tearing down project $COMPOSE_PROJECT_NAME" + mm --profile scripted --profile tools down -v --remove-orphans >/dev/null 2>&1 || true + rm -f "$HERE/$BOOTSTRAP" } trap cleanup EXIT -echo "==> Bringing up Mattermost" -docker compose -f docker-compose.mattermost.yml up -d +echo "==> Bringing up Mattermost (${MATTERMOST_IMAGE:-mattermost/mattermost-preview:11.7.7}) and building the Playwright image ($PLAYWRIGHT_VERSION)" +mm up -d mattermost +mm build playwright +if [ -n "${SCRIPTED_APPROVAL:-}" ]; then + export SCRIPTED_TOOL="${SCRIPTED_TOOL:-mcp}" # mcp (default) or shell + mm --profile scripted up -d --build scriptedllm mcp-demo +fi echo "==> Bootstrapping Mattermost" -node bootstrap.mjs "$SERVER" +pw node /work/bootstrap.mjs http://mattermost:8065 + +BOT_TOKEN="$(pw node -p "require('/work/$BOOTSTRAP').bot.token")" +CHANNEL_ID="$(pw node -p "require('/work/$BOOTSTRAP').channel.id")" -BOT_TOKEN="$(node -e "process.stdout.write(require('./.bootstrap.json').bot.token)")" -CHANNEL_ID="$(node -e "process.stdout.write(require('./.bootstrap.json').channel.id)")" +echo "==> Starting netclawd ($NETCLAW_IMAGE), models via NETCLAW_Models__Definitions/Roles env" +mm up -d --wait netclawd +mm up -d callbackfwd -echo "==> Starting netclaw ($NETCLAW_IMAGE) against Mattermost + $SPARK_ENDPOINT" -docker rm -f netclaw-mm-shot >/dev/null 2>&1 || true -docker run -d --name netclaw-mm-shot --network host \ - -e NETCLAW_Daemon__Host=127.0.0.1 -e NETCLAW_Daemon__Port=5299 \ - -e NETCLAW_Daemon__ExposureMode=local -e NETCLAW_Daemon__DisableSelfUpdate=true \ - -e NETCLAW_Session__TurnLlmTimeoutSeconds=600 \ - -e NETCLAW_Session__FirstTokenTimeoutSeconds=600 \ - -e NETCLAW_Session__PrefillTimeoutSeconds=600 \ - -e NETCLAW_Mattermost__Enabled=true \ - -e NETCLAW_Mattermost__ServerUrl="$SERVER" \ - -e NETCLAW_Mattermost__BotToken="$BOT_TOKEN" \ - -e NETCLAW_Mattermost__DefaultChannelId="$CHANNEL_ID" \ - -e NETCLAW_Mattermost__MentionOnly=true \ - -e "NETCLAW_Mattermost__ChannelAudiences__${CHANNEL_ID}=team" \ - -e NETCLAW_Providers__spark__Type=openai-compatible \ - -e NETCLAW_Providers__spark__Endpoint="$SPARK_ENDPOINT" \ - -e NETCLAW_Models__Main__Provider=spark -e NETCLAW_Models__Main__ModelId="$SPARK_MODEL_ID" \ - -e NETCLAW_Models__Fallback__Provider=spark -e NETCLAW_Models__Fallback__ModelId="$SPARK_MODEL_ID" \ - -e NETCLAW_Models__Compaction__Provider=spark -e NETCLAW_Models__Compaction__ModelId="$SPARK_MODEL_ID" \ - "$NETCLAW_IMAGE" +echo "==> Storing the bot token as an encrypted secret and enabling the Mattermost channel" +nc secrets set Mattermost.BotToken "$BOT_TOKEN" >/dev/null +# The scripted approval run needs shell_execute, which the "team" audience doesn't get. +AUDIENCE="${CHANNEL_AUDIENCE:-$([ -n "${SCRIPTED_APPROVAL:-}" ] && echo personal || echo team)}" +mm exec -T -u netclaw -e SERVER=http://mattermost:8065 -e CH="$CHANNEL_ID" -e AUD="$AUDIENCE" netclawd sh -c ' + f=/home/netclaw/.netclaw/config/netclaw.json + [ -f "$f" ] || echo "{}" > "$f" + jq --arg s "$SERVER" --arg c "$CH" ".configVersion = 1 + | .Security = {DeploymentPosture: \"Personal\", ShellExecutionMode: \"HostAllowed\", StrictDefaults: true} + | .Mattermost = {Enabled: true, ServerUrl: \$s, CallbackUrl: \"http://netclawd:5299/api/mattermost/actions\", DefaultChannelId: \$c, MentionOnly: true, ChannelAudiences: {(\$c): \$a}}" --arg a "$AUD" \ + "$f" > "$f.tmp" && mv "$f.tmp" "$f"' -echo "==> Waiting for daemon health" -for i in $(seq 1 30); do - curl -sf http://127.0.0.1:5299/api/health/ready >/dev/null 2>&1 && break +echo "==> Waiting for the Mattermost channel to connect" +connected=0 +for _ in $(seq 1 45); do + if mm exec -T -u netclaw netclawd sh -c 'cat /home/netclaw/.netclaw/logs/daemon-*.log' 2>/dev/null \ + | grep -i 'mattermost' | grep -iE 'connected|started' | tail -n 3 | grep -q .; then connected=1; break; fi sleep 2 done +if [ "$connected" -ne 1 ]; then + echo "mattermost: netclawd never reported a Mattermost connection" >&2 + mm exec -T -u netclaw netclawd sh -c 'grep -i mattermost /home/netclaw/.netclaw/logs/daemon-*.log | tail -n 20' >&2 || true + exit 1 +fi +mm exec -T -u netclaw netclawd sh -c 'grep -i mattermost /home/netclaw/.netclaw/logs/daemon-*.log | tail -n 5' + +if [ -n "${SCRIPTED_APPROVAL:-}" ] && [ "$SCRIPTED_TOOL" = mcp ]; then + echo "==> Registering the demo MCP server and gating its tools behind approval" + nc mcp add --transport http demo-utilities http://mcp-demo:8080/mcp >/dev/null + mm exec -T -u netclaw netclawd sh -c ' + f=/home/netclaw/.netclaw/config/netclaw.json + jq ".Tools.AudienceProfiles.Personal.ApprovalPolicy.McpServerDefaults[\"demo-utilities\"] = \"Approval\"" "$f" > "$f.tmp" && mv "$f.tmp" "$f"' + sleep 12 # the daemon restarts itself when netclaw.json changes + for _ in $(seq 1 30); do nc status 2>/dev/null | grep -q 'mcp:demo-utilities: healthy' && break; sleep 2; done +fi +if [ -n "${SCRIPTED_APPROVAL:-}" ]; then + # A Git repository for the shell call to run in, so the prompt looks like real project work. + mm exec -T -u netclaw netclawd sh -c ' + mkdir -p ~/projects/website && cd ~/projects/website && git init -q -b main . \ + && git config user.name alice && git config user.email alice@example.com \ + && echo "# website" > README.md && git add . && git commit -q -m "Initial commit" \ + && git remote add origin git@example.com:acme/website.git' + echo "==> Capturing the approval prompt" + pw node /work/approval.mjs || { + echo "==> approval capture failed; scripted endpoint and daemon logs follow" >&2 + mm logs --tail 20 scriptedllm >&2 || true + mm exec -T -u netclaw netclawd sh -c 'tail -n 40 /home/netclaw/.netclaw/logs/daemon-*.log' >&2 || true + exit 1 + } + echo "==> Done. approval-prompt.png written to $CAPTURE_OUT_DIR" + exit 0 +fi -echo "==> Posting an @netclaw message as the test user" -TOKEN="$(curl -s -i -X POST "$SERVER/api/v4/users/login" -H 'Content-Type: application/json' \ - -d '{"login_id":"testuser","password":"TestUser1234!"}' | grep -i '^token:' | awk '{print $2}' | tr -d '\r')" -curl -s -X POST "$SERVER/api/v4/posts" -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \ - -d "{\"channel_id\":\"$CHANNEL_ID\",\"message\":\"@netclaw hey! In one sentence, what are you and what can you help with?\"}" >/dev/null -echo " posted; waiting for the bot to reply" -sleep 25 +echo "==> Posting an @netclaw message and waiting for the reply" +pw node /work/post-message.mjs echo "==> Capturing screenshots (Playwright)" -node capture.mjs +pw node /work/capture.mjs -echo "==> Done. Shots written to ../output/mattermost-*.png" +echo "==> Done. Shots written to $CAPTURE_OUT_DIR/mattermost-*.png" diff --git a/screenshots/mattermost/scripted-llm.mjs b/screenshots/mattermost/scripted-llm.mjs new file mode 100644 index 0000000..550765d --- /dev/null +++ b/screenshots/mattermost/scripted-llm.mjs @@ -0,0 +1,59 @@ +#!/usr/bin/env node +// A tiny OpenAI-compatible endpoint that answers every first turn with one scripted +// shell_execute tool call, so the approval prompt shows up without a real model. +// Runs in the Playwright image (approval.sh). After the tool result it replies "Done." +// +// Env: PORT (8080), SCRIPTED_TOOL (mcp|shell, default mcp), SCRIPTED_COMMAND (shell command), +// SCRIPTED_MCP_URL (URL the MCP call fetches) +import { createServer } from 'http'; + +const COMMAND = process.env.SCRIPTED_COMMAND || 'git push origin main'; +const KIND = process.env.SCRIPTED_TOOL || 'mcp'; +const MCP_URL = process.env.SCRIPTED_MCP_URL || 'https://status.example.com/api/v1/incidents'; +const PORT = Number(process.env.PORT || 8080); +const id = () => 'chatcmpl-' + Math.random().toString(36).slice(2, 10); + +function completion(body) { + const tools = body.tools || []; + const shell = KIND === 'shell' && tools.find(t => (t.function?.name || t.name) === 'shell_execute'); + const mcp = KIND === 'mcp' && tools.find(t => /http_fetch/.test(t.function?.name || t.name || '')); + const answered = (body.messages || []).some(m => m.role === 'tool'); + const calledMcp = (body.messages || []).some(m => (m.tool_calls || []).some(c => /http_fetch/.test(c.function?.name || ''))); + const loader = KIND === 'mcp' && !mcp && !answered && tools.find(t => (t.function?.name || t.name) === 'load_tool'); + if (loader) { + // MCP tools are deferred: the first turn loads the tool, the next one calls it. + return { message: { role: 'assistant', content: null, tool_calls: [{ id: 'call_ap0', type: 'function', function: { name: 'load_tool', arguments: JSON.stringify({ name: 'demo-utilities/http_fetch', _rationale: 'Load the HTTP fetch tool.' }) } }] }, finish: 'tool_calls' }; + } + if (mcp && !calledMcp) { + const name = mcp.function?.name || mcp.name; + return { message: { role: 'assistant', content: null, tool_calls: [{ id: 'call_ap1', type: 'function', function: { name, arguments: JSON.stringify({ url: MCP_URL, method: 'POST', _rationale: 'Open a status page incident for the website deploy.' }) } }] }, finish: 'tool_calls' }; + } + if (shell && !answered) { + return { message: { role: 'assistant', content: null, tool_calls: [{ id: 'call_ap1', type: 'function', function: { name: 'shell_execute', arguments: JSON.stringify({ command: COMMAND, _rationale: 'Push the pending website changes to origin.' }) } }] }, finish: 'tool_calls' }; + } + return { message: { role: 'assistant', content: 'Done.' }, finish: 'stop' }; +} + +createServer((req, res) => { + let raw = ''; + req.on('data', c => (raw += c)); + req.on('end', () => { + if (req.method === 'GET') { res.setHeader('Content-Type', 'application/json'); return res.end(JSON.stringify({ object: 'list', data: [{ id: 'demo-model', object: 'model' }] })); } + let body = {}; + try { body = JSON.parse(raw || '{}'); } catch {} + const { message, finish } = completion(body); + console.log(`${req.method} ${req.url} stream=${!!body.stream} -> ${finish}`); + const base = { id: id(), object: 'chat.completion', created: Math.floor(Date.now() / 1000), model: 'demo-model' }; + if (body.stream) { + res.writeHead(200, { 'Content-Type': 'text/event-stream' }); + const delta = { ...message }; + if (delta.tool_calls) delta.tool_calls = delta.tool_calls.map((t, index) => ({ index, ...t })); + res.write(`data: ${JSON.stringify({ ...base, object: 'chat.completion.chunk', choices: [{ index: 0, delta, finish_reason: null }] })}\n\n`); + res.write(`data: ${JSON.stringify({ ...base, object: 'chat.completion.chunk', choices: [{ index: 0, delta: {}, finish_reason: finish }], usage: { prompt_tokens: 10, completion_tokens: 10, total_tokens: 20 } })}\n\n`); + res.end('data: [DONE]\n\n'); + } else { + res.setHeader('Content-Type', 'application/json'); + res.end(JSON.stringify({ ...base, choices: [{ index: 0, message, finish_reason: finish }], usage: { prompt_tokens: 10, completion_tokens: 10, total_tokens: 20 } })); + } + }); +}).listen(PORT, () => console.log(`scripted llm on :${PORT}`)); diff --git a/screenshots/mattermost/tcp-forward.mjs b/screenshots/mattermost/tcp-forward.mjs new file mode 100644 index 0000000..491e148 --- /dev/null +++ b/screenshots/mattermost/tcp-forward.mjs @@ -0,0 +1,14 @@ +#!/usr/bin/env node +// Forwards 0.0.0.0:LISTEN to 127.0.0.1:TARGET inside the netclawd network namespace. +// The daemon's default `local` exposure mode binds loopback only, so Mattermost (another +// container) can't reach /api/mattermost/actions directly. This sidecar is the "forwarder" +// the Mattermost channel docs describe; the daemon still sees loopback traffic. +// Env: LISTEN (5299), TARGET (5199) +import net from 'net'; +const LISTEN = Number(process.env.LISTEN || 5299), TARGET = Number(process.env.TARGET || 5199); +net.createServer(c => { + const u = net.connect(TARGET, '127.0.0.1'); + c.pipe(u).pipe(c); + const end = () => { c.destroy(); u.destroy(); }; + c.on('error', end); u.on('error', end); +}).listen(LISTEN, '0.0.0.0', () => console.log(`forwarding :${LISTEN} -> 127.0.0.1:${TARGET}`)); diff --git a/screenshots/mcp-server/Dockerfile b/screenshots/mcp-server/Dockerfile index 76ae002..e60adb3 100644 --- a/screenshots/mcp-server/Dockerfile +++ b/screenshots/mcp-server/Dockerfile @@ -1,6 +1,7 @@ FROM python:3.12-slim WORKDIR /app -RUN pip install --no-cache-dir "mcp[cli]" +# server.py uses the 1.x FastMCP API; mcp 2.x renamed it. +RUN pip install --no-cache-dir "mcp[cli]<2" COPY server.py . EXPOSE 8080 CMD ["python", "server.py"] diff --git a/screenshots/mcp-server/Dockerfile.oauth b/screenshots/mcp-server/Dockerfile.oauth new file mode 100644 index 0000000..60c30f0 --- /dev/null +++ b/screenshots/mcp-server/Dockerfile.oauth @@ -0,0 +1,6 @@ +FROM python:3.12-slim +WORKDIR /app +RUN pip install --no-cache-dir "mcp[cli]<2" +COPY oauth_server.py . +EXPOSE 8080 +CMD ["python", "oauth_server.py"] diff --git a/screenshots/mcp-server/oauth_server.py b/screenshots/mcp-server/oauth_server.py new file mode 100644 index 0000000..64fc900 --- /dev/null +++ b/screenshots/mcp-server/oauth_server.py @@ -0,0 +1,246 @@ +"""OAuth-protected demo MCP server for testing `netclaw mcp auth`. + +One process plays both roles: the MCP resource server at /mcp and a minimal +OAuth 2.1 authorization server (authorization code + PKCE, refresh tokens with +rotation). The authorize endpoint auto-approves, so a script can follow the +redirect with curl instead of a browser. + +MODE (environment variable): + confidential Pre-registered client only. No dynamic registration. The token + endpoint rejects a wrong or missing client secret. + public Dynamic client registration (RFC 7591), public client, PKCE only. + anonymous Like public, but initialize and tools/list work without a token. + Only tools/call returns 401 (models servers that allow anonymous + catalog discovery). + +Other settings: BASE_URL (public URL of this server), CLIENT_ID, CLIENT_SECRET, +TOKEN_TTL (access token lifetime in seconds, default 3600). +""" + +import base64 +import hashlib +import json +import os +import secrets +import time +from urllib.parse import urlencode + +import uvicorn +from mcp.server.fastmcp import FastMCP +from starlette.requests import Request +from starlette.responses import JSONResponse, RedirectResponse, Response + +MODE = os.environ.get("MODE", "confidential") +BASE_URL = os.environ.get("BASE_URL", "http://localhost:8080").rstrip("/") +CLIENT_ID = os.environ.get("CLIENT_ID", "demo-client") +CLIENT_SECRET = os.environ.get("CLIENT_SECRET", "demo-secret") +TOKEN_TTL = int(os.environ.get("TOKEN_TTL", "3600")) + +mcp = FastMCP("oauth-demo", stateless_http=True, host="0.0.0.0", port=8080) + + +@mcp.tool() +def whoami() -> str: + """Return the identity this server associates with the access token.""" + return "demo-user" + + +@mcp.tool() +def generate_uuid() -> str: + """Generate a random UUID v4.""" + import uuid + return str(uuid.uuid4()) + + +@mcp.prompt() +def review_checklist(topic: str) -> str: + """Walk through a short review checklist for a topic.""" + return f"Review {topic}: list the risks, the owner of each risk, and the next action." + + +codes: dict[str, dict] = {} +access_tokens: dict[str, float] = {} +refresh_tokens: dict[str, dict] = {} +dynamic_clients: dict[str, dict] = {} + + +def log(msg: str) -> None: + print(f"[oauth-demo] {msg}", flush=True) + + +def metadata_url() -> str: + return f"{BASE_URL}/.well-known/oauth-protected-resource/mcp" + + +@mcp.custom_route("/.well-known/oauth-protected-resource/mcp", ["GET"]) +@mcp.custom_route("/.well-known/oauth-protected-resource", ["GET"]) +async def protected_resource(_: Request) -> Response: + return JSONResponse({ + "resource": f"{BASE_URL}/mcp", + "authorization_servers": [BASE_URL], + "bearer_methods_supported": ["header"], + }) + + +@mcp.custom_route("/.well-known/oauth-authorization-server", ["GET"]) +async def as_metadata(_: Request) -> Response: + doc = { + "issuer": BASE_URL, + "authorization_endpoint": f"{BASE_URL}/authorize", + "token_endpoint": f"{BASE_URL}/token", + "response_types_supported": ["code"], + "grant_types_supported": ["authorization_code", "refresh_token"], + "code_challenge_methods_supported": ["S256"], + "token_endpoint_auth_methods_supported": + ["client_secret_post", "client_secret_basic"] if MODE == "confidential" else ["none"], + } + if MODE != "confidential": + doc["registration_endpoint"] = f"{BASE_URL}/register" + return JSONResponse(doc) + + +@mcp.custom_route("/register", ["POST"]) +async def register(request: Request) -> Response: + if MODE == "confidential": + return JSONResponse({"error": "registration_not_supported"}, status_code=404) + body = await request.json() + client_id = "dyn-" + secrets.token_hex(6) + dynamic_clients[client_id] = {"redirect_uris": body.get("redirect_uris", [])} + log(f"registered dynamic client {client_id}") + return JSONResponse({ + "client_id": client_id, + "redirect_uris": body.get("redirect_uris", []), + "token_endpoint_auth_method": "none", + "grant_types": ["authorization_code", "refresh_token"], + "response_types": ["code"], + }, status_code=201) + + +def known_client(client_id: str, redirect_uri: str) -> bool: + if MODE == "confidential": + return client_id == CLIENT_ID and redirect_uri.endswith("/api/mcp/oauth/callback") + return client_id in dynamic_clients and redirect_uri in dynamic_clients[client_id]["redirect_uris"] + + +@mcp.custom_route("/authorize", ["GET"]) +async def authorize(request: Request) -> Response: + q = request.query_params + client_id, redirect_uri = q.get("client_id", ""), q.get("redirect_uri", "") + if not known_client(client_id, redirect_uri): + log(f"authorize rejected: unknown client {client_id} or redirect {redirect_uri}") + return JSONResponse({"error": "invalid_request", + "error_description": "unknown client or redirect_uri"}, status_code=400) + if not q.get("code_challenge"): + return JSONResponse({"error": "invalid_request", "error_description": "PKCE required"}, status_code=400) + code = secrets.token_urlsafe(16) + codes[code] = {"client_id": client_id, "redirect_uri": redirect_uri, + "challenge": q["code_challenge"], "scope": q.get("scope", "")} + log(f"authorize ok for {client_id}, scope='{q.get('scope', '')}'") + params = {"code": code} + if q.get("state"): + params["state"] = q["state"] + return RedirectResponse(f"{redirect_uri}?{urlencode(params)}", status_code=302) + + +def client_credentials(request: Request, form: dict) -> tuple[str, str | None]: + auth = request.headers.get("authorization", "") + if auth.lower().startswith("basic "): + raw = base64.b64decode(auth[6:]).decode() + cid, _, sec = raw.partition(":") + return cid, sec + return form.get("client_id", ""), form.get("client_secret") + + +@mcp.custom_route("/token", ["POST"]) +async def token(request: Request) -> Response: + form = dict((await request.form()).items()) + cid, sec = client_credentials(request, form) + grant = form.get("grant_type") + log(f"token request grant={grant} client={cid} secret_presented={sec is not None}") + if MODE == "confidential" and (cid != CLIENT_ID or sec != CLIENT_SECRET): + return JSONResponse({"error": "invalid_client", + "error_description": "client authentication failed"}, status_code=401) + if grant == "authorization_code": + info = codes.pop(form.get("code", ""), None) + if info is None or info["client_id"] != cid: + return JSONResponse({"error": "invalid_grant"}, status_code=400) + verifier = form.get("code_verifier", "") + digest = base64.urlsafe_b64encode(hashlib.sha256(verifier.encode()).digest()).rstrip(b"=").decode() + if digest != info["challenge"]: + return JSONResponse({"error": "invalid_grant", "error_description": "PKCE mismatch"}, status_code=400) + return issue(cid, info["scope"]) + if grant == "refresh_token": + info = refresh_tokens.pop(form.get("refresh_token", ""), None) + if info is None or info["client_id"] != cid: + log("refresh rejected: unknown or already-used refresh token") + return JSONResponse({"error": "invalid_grant"}, status_code=400) + return issue(cid, info["scope"]) + return JSONResponse({"error": "unsupported_grant_type"}, status_code=400) + + +def issue(client_id: str, scope: str) -> Response: + access = "at-" + secrets.token_urlsafe(16) + refresh = "rt-" + secrets.token_urlsafe(16) + access_tokens[access] = time.time() + TOKEN_TTL + refresh_tokens[refresh] = {"client_id": client_id, "scope": scope} + log(f"issued access token (ttl {TOKEN_TTL}s) and refresh token") + return JSONResponse({"access_token": access, "token_type": "Bearer", "expires_in": TOKEN_TTL, + "refresh_token": refresh, "scope": scope}) + + +def valid_bearer(headers: dict) -> bool: + auth = headers.get(b"authorization", b"").decode() + if not auth.lower().startswith("bearer "): + return False + expires = access_tokens.get(auth[7:]) + return expires is not None and expires > time.time() + + +class Gate: + """Return 401 with an RFC 9728 challenge when /mcp has no valid bearer token.""" + + def __init__(self, app): + self.app = app + + async def __call__(self, scope, receive, send): + if scope["type"] != "http" or scope["path"].rstrip("/") != "/mcp": + return await self.app(scope, receive, send) + + headers = dict(scope["headers"]) + body = b"" + while True: + message = await receive() + body += message.get("body", b"") + if not message.get("more_body"): + break + + methods = [] + try: + parsed = json.loads(body) if body else {} + methods = [m.get("method") for m in (parsed if isinstance(parsed, list) else [parsed])] + except ValueError: + pass + + needs_token = MODE != "anonymous" or "tools/call" in methods + ok = valid_bearer(headers) + log(f"/mcp {scope['method']} methods={methods} bearer_valid={ok}") + if needs_token and not ok: + resp = Response(status_code=401, headers={ + "WWW-Authenticate": f'Bearer resource_metadata="{metadata_url()}"'}) + return await resp(scope, receive, send) + + replayed = False + + async def replay(): + nonlocal replayed + if not replayed: + replayed = True + return {"type": "http.request", "body": body, "more_body": False} + return await receive() + + return await self.app(scope, replay, send) + + +if __name__ == "__main__": + log(f"mode={MODE} base_url={BASE_URL}") + uvicorn.run(Gate(mcp.streamable_http_app()), host="0.0.0.0", port=8080, log_level="warning") diff --git a/screenshots/output/approval-prompt-tui.png b/screenshots/output/approval-prompt-tui.png new file mode 100644 index 0000000..ef76f7a Binary files /dev/null and b/screenshots/output/approval-prompt-tui.png differ diff --git a/screenshots/output/approvals-list-json.png b/screenshots/output/approvals-list-json.png deleted file mode 100644 index 353b85f..0000000 Binary files a/screenshots/output/approvals-list-json.png and /dev/null differ diff --git a/screenshots/output/approvals-list.png b/screenshots/output/approvals-list.png index 0550946..b02ab08 100644 Binary files a/screenshots/output/approvals-list.png and b/screenshots/output/approvals-list.png differ diff --git a/screenshots/output/approvals-tui-revoke-confirm.png b/screenshots/output/approvals-tui-revoke-confirm.png index 1939940..8041170 100644 Binary files a/screenshots/output/approvals-tui-revoke-confirm.png and b/screenshots/output/approvals-tui-revoke-confirm.png differ diff --git a/screenshots/output/approvals-tui-revoke-done.png b/screenshots/output/approvals-tui-revoke-done.png deleted file mode 100644 index 578b4f3..0000000 Binary files a/screenshots/output/approvals-tui-revoke-done.png and /dev/null differ diff --git a/screenshots/output/approvals-tui-selected.png b/screenshots/output/approvals-tui-selected.png deleted file mode 100644 index b2c395f..0000000 Binary files a/screenshots/output/approvals-tui-selected.png and /dev/null differ diff --git a/screenshots/output/approvals-tui.png b/screenshots/output/approvals-tui.png index 72251a9..5211720 100644 Binary files a/screenshots/output/approvals-tui.png and b/screenshots/output/approvals-tui.png differ diff --git a/screenshots/output/approvals.gif b/screenshots/output/approvals.gif deleted file mode 100644 index 0ba76a8..0000000 Binary files a/screenshots/output/approvals.gif and /dev/null differ diff --git a/screenshots/output/chat-approval-prompt.png b/screenshots/output/chat-approval-prompt.png new file mode 100644 index 0000000..1595e7d Binary files /dev/null and b/screenshots/output/chat-approval-prompt.png differ diff --git a/screenshots/output/chat-first-greeting.png b/screenshots/output/chat-first-greeting.png new file mode 100644 index 0000000..a1bfa3f Binary files /dev/null and b/screenshots/output/chat-first-greeting.png differ diff --git a/screenshots/output/chat-intro-greeting.png b/screenshots/output/chat-intro-greeting.png deleted file mode 100644 index a4502ad..0000000 Binary files a/screenshots/output/chat-intro-greeting.png and /dev/null differ diff --git a/screenshots/output/chat-profile-questions.png b/screenshots/output/chat-profile-questions.png deleted file mode 100644 index 8023a5c..0000000 Binary files a/screenshots/output/chat-profile-questions.png and /dev/null differ diff --git a/screenshots/output/chat-profile-update-soul.md.png b/screenshots/output/chat-profile-update-soul.md.png deleted file mode 100644 index 456808d..0000000 Binary files a/screenshots/output/chat-profile-update-soul.md.png and /dev/null differ diff --git a/screenshots/output/chat-tui-approval-prompt.png b/screenshots/output/chat-tui-approval-prompt.png new file mode 100644 index 0000000..fb37f65 Binary files /dev/null and b/screenshots/output/chat-tui-approval-prompt.png differ diff --git a/screenshots/output/chat-tui-idle.png b/screenshots/output/chat-tui-idle.png new file mode 100644 index 0000000..5362069 Binary files /dev/null and b/screenshots/output/chat-tui-idle.png differ diff --git a/screenshots/output/config-browser.png b/screenshots/output/config-browser.png deleted file mode 100644 index 4e01817..0000000 Binary files a/screenshots/output/config-browser.png and /dev/null differ diff --git a/screenshots/output/config-channel-audiences.png b/screenshots/output/config-channel-audiences.png new file mode 100644 index 0000000..bac5c60 Binary files /dev/null and b/screenshots/output/config-channel-audiences.png differ diff --git a/screenshots/output/config-channels-menu.png b/screenshots/output/config-channels-menu.png index 1d8cd98..9c65512 100644 Binary files a/screenshots/output/config-channels-menu.png and b/screenshots/output/config-channels-menu.png differ diff --git a/screenshots/output/config-dashboard.png b/screenshots/output/config-dashboard.png index 12a0e6f..9e554a5 100644 Binary files a/screenshots/output/config-dashboard.png and b/screenshots/output/config-dashboard.png differ diff --git a/screenshots/output/config-exposure.png b/screenshots/output/config-exposure.png index fd89799..634f83f 100644 Binary files a/screenshots/output/config-exposure.png and b/screenshots/output/config-exposure.png differ diff --git a/screenshots/output/config-inbound-webhooks.png b/screenshots/output/config-inbound-webhooks.png index b76c2c8..bfc84e2 100644 Binary files a/screenshots/output/config-inbound-webhooks.png and b/screenshots/output/config-inbound-webhooks.png differ diff --git a/screenshots/output/config-retention.png b/screenshots/output/config-retention.png new file mode 100644 index 0000000..15921a0 Binary files /dev/null and b/screenshots/output/config-retention.png differ diff --git a/screenshots/output/config-search.png b/screenshots/output/config-search.png index 1699a37..9034a80 100644 Binary files a/screenshots/output/config-search.png and b/screenshots/output/config-search.png differ diff --git a/screenshots/output/config-skills.png b/screenshots/output/config-skills.png index 3831821..268f782 100644 Binary files a/screenshots/output/config-skills.png and b/screenshots/output/config-skills.png differ diff --git a/screenshots/output/config-telemetry.png b/screenshots/output/config-telemetry.png deleted file mode 100644 index c4b1d77..0000000 Binary files a/screenshots/output/config-telemetry.png and /dev/null differ diff --git a/screenshots/output/config-workspaces.png b/screenshots/output/config-workspaces.png deleted file mode 100644 index 4f4502e..0000000 Binary files a/screenshots/output/config-workspaces.png and /dev/null differ diff --git a/screenshots/output/daemon-pair.png b/screenshots/output/daemon-pair.png new file mode 100644 index 0000000..0286824 Binary files /dev/null and b/screenshots/output/daemon-pair.png differ diff --git a/screenshots/output/discord-setup-applications.png b/screenshots/output/discord-setup-applications.png index f75b187..ac26bc8 100644 Binary files a/screenshots/output/discord-setup-applications.png and b/screenshots/output/discord-setup-applications.png differ diff --git a/screenshots/output/discord-setup-authorize-dialog.png b/screenshots/output/discord-setup-authorize-dialog.png index 5b945a8..fcbffeb 100644 Binary files a/screenshots/output/discord-setup-authorize-dialog.png and b/screenshots/output/discord-setup-authorize-dialog.png differ diff --git a/screenshots/output/discord-setup-installation-provided-link.png b/screenshots/output/discord-setup-installation-provided-link.png deleted file mode 100644 index 3e6c7fb..0000000 Binary files a/screenshots/output/discord-setup-installation-provided-link.png and /dev/null differ diff --git a/screenshots/output/discord-setup-oauth-url.png b/screenshots/output/discord-setup-oauth-url.png index 215d250..9f0ef49 100644 Binary files a/screenshots/output/discord-setup-oauth-url.png and b/screenshots/output/discord-setup-oauth-url.png differ diff --git a/screenshots/output/docker-health.png b/screenshots/output/docker-health.png new file mode 100644 index 0000000..dc4d431 Binary files /dev/null and b/screenshots/output/docker-health.png differ diff --git a/screenshots/output/doctor.gif b/screenshots/output/doctor.gif deleted file mode 100644 index d713b02..0000000 Binary files a/screenshots/output/doctor.gif and /dev/null differ diff --git a/screenshots/output/doctor.png b/screenshots/output/doctor.png index 1206394..3b6aedb 100644 Binary files a/screenshots/output/doctor.png and b/screenshots/output/doctor.png differ diff --git a/screenshots/output/init-01-endpoint.png b/screenshots/output/init-01-endpoint.png index 75288cc..377e45f 100644 Binary files a/screenshots/output/init-01-endpoint.png and b/screenshots/output/init-01-endpoint.png differ diff --git a/screenshots/output/init-01-model-select.png b/screenshots/output/init-01-model-select.png index f90ee23..52f07f5 100644 Binary files a/screenshots/output/init-01-model-select.png and b/screenshots/output/init-01-model-select.png differ diff --git a/screenshots/output/init-01-provider-list.png b/screenshots/output/init-01-provider-list.png index 12d5471..c26b4b3 100644 Binary files a/screenshots/output/init-01-provider-list.png and b/screenshots/output/init-01-provider-list.png differ diff --git a/screenshots/output/init-01-provider-ollama.png b/screenshots/output/init-01-provider-ollama.png index 75288cc..76bd1a1 100644 Binary files a/screenshots/output/init-01-provider-ollama.png and b/screenshots/output/init-01-provider-ollama.png differ diff --git a/screenshots/output/init-02-identity.png b/screenshots/output/init-02-identity.png index 8d39d3d..4a374d5 100644 Binary files a/screenshots/output/init-02-identity.png and b/screenshots/output/init-02-identity.png differ diff --git a/screenshots/output/init-02-security-posture.png b/screenshots/output/init-02-security-posture.png index 1145cab..0b9508f 100644 Binary files a/screenshots/output/init-02-security-posture.png and b/screenshots/output/init-02-security-posture.png differ diff --git a/screenshots/output/init-04-channel-audiences.png b/screenshots/output/init-04-channel-audiences.png deleted file mode 100644 index 2b53e1c..0000000 Binary files a/screenshots/output/init-04-channel-audiences.png and /dev/null differ diff --git a/screenshots/output/init-04-enabled-features.png b/screenshots/output/init-04-enabled-features.png index dfe8e98..32a1096 100644 Binary files a/screenshots/output/init-04-enabled-features.png and b/screenshots/output/init-04-enabled-features.png differ diff --git a/screenshots/output/init-05-health-check.png b/screenshots/output/init-05-health-check.png index aaa6814..f4c92c6 100644 Binary files a/screenshots/output/init-05-health-check.png and b/screenshots/output/init-05-health-check.png differ diff --git a/screenshots/output/init-07-external-skills.png b/screenshots/output/init-07-external-skills.png deleted file mode 100644 index ae7f20b..0000000 Binary files a/screenshots/output/init-07-external-skills.png and /dev/null differ diff --git a/screenshots/output/init-09-exposure.png b/screenshots/output/init-09-exposure.png deleted file mode 100644 index cb749a2..0000000 Binary files a/screenshots/output/init-09-exposure.png and /dev/null differ diff --git a/screenshots/output/init-existing-menu.png b/screenshots/output/init-existing-menu.png index 0c1ddde..def8b06 100644 Binary files a/screenshots/output/init-existing-menu.png and b/screenshots/output/init-existing-menu.png differ diff --git a/screenshots/output/init-start-over-scope.png b/screenshots/output/init-start-over-scope.png index 84c2b82..fe02e71 100644 Binary files a/screenshots/output/init-start-over-scope.png and b/screenshots/output/init-start-over-scope.png differ diff --git a/screenshots/output/mattermost-bot-accounts.png b/screenshots/output/mattermost-bot-accounts.png index 79a09a9..c1a2ea5 100644 Binary files a/screenshots/output/mattermost-bot-accounts.png and b/screenshots/output/mattermost-bot-accounts.png differ diff --git a/screenshots/output/mattermost-compose-approval-resolved.png b/screenshots/output/mattermost-compose-approval-resolved.png new file mode 100644 index 0000000..8cbc22b Binary files /dev/null and b/screenshots/output/mattermost-compose-approval-resolved.png differ diff --git a/screenshots/output/mattermost-compose-approval.png b/screenshots/output/mattermost-compose-approval.png new file mode 100644 index 0000000..54ba2d3 Binary files /dev/null and b/screenshots/output/mattermost-compose-approval.png differ diff --git a/screenshots/output/mattermost-compose-bot-account.png b/screenshots/output/mattermost-compose-bot-account.png new file mode 100644 index 0000000..7d3be6a Binary files /dev/null and b/screenshots/output/mattermost-compose-bot-account.png differ diff --git a/screenshots/output/mattermost-compose-thread.png b/screenshots/output/mattermost-compose-thread.png new file mode 100644 index 0000000..10e59c6 Binary files /dev/null and b/screenshots/output/mattermost-compose-thread.png differ diff --git a/screenshots/output/mattermost-conversation.png b/screenshots/output/mattermost-conversation.png index f3b4a01..91add4a 100644 Binary files a/screenshots/output/mattermost-conversation.png and b/screenshots/output/mattermost-conversation.png differ diff --git a/screenshots/output/mattermost-create-bot.png b/screenshots/output/mattermost-create-bot.png index d6e7843..a1e31ec 100644 Binary files a/screenshots/output/mattermost-create-bot.png and b/screenshots/output/mattermost-create-bot.png differ diff --git a/screenshots/output/mcp-tools-confirm.png b/screenshots/output/mcp-tools-confirm.png new file mode 100644 index 0000000..2ae736a Binary files /dev/null and b/screenshots/output/mcp-tools-confirm.png differ diff --git a/screenshots/output/mcp-tools-personal.png b/screenshots/output/mcp-tools-personal.png index 3e589db..1f49f5c 100644 Binary files a/screenshots/output/mcp-tools-personal.png and b/screenshots/output/mcp-tools-personal.png differ diff --git a/screenshots/output/mcp-tools-public.png b/screenshots/output/mcp-tools-public.png deleted file mode 100644 index 12523d3..0000000 Binary files a/screenshots/output/mcp-tools-public.png and /dev/null differ diff --git a/screenshots/output/mcp-tools-server-list-2.png b/screenshots/output/mcp-tools-server-list-2.png deleted file mode 100644 index 0797d99..0000000 Binary files a/screenshots/output/mcp-tools-server-list-2.png and /dev/null differ diff --git a/screenshots/output/mcp-tools-server-list.png b/screenshots/output/mcp-tools-server-list.png index 3d4869d..8aed8d9 100644 Binary files a/screenshots/output/mcp-tools-server-list.png and b/screenshots/output/mcp-tools-server-list.png differ diff --git a/screenshots/output/mcp-tools-server2-personal.png b/screenshots/output/mcp-tools-server2-personal.png deleted file mode 100644 index 3d95516..0000000 Binary files a/screenshots/output/mcp-tools-server2-personal.png and /dev/null differ diff --git a/screenshots/output/mcp-tools-team.png b/screenshots/output/mcp-tools-team.png index 1c6dbf5..387388b 100644 Binary files a/screenshots/output/mcp-tools-team.png and b/screenshots/output/mcp-tools-team.png differ diff --git a/screenshots/output/mcp-tools.gif b/screenshots/output/mcp-tools.gif deleted file mode 100644 index 2900c5f..0000000 Binary files a/screenshots/output/mcp-tools.gif and /dev/null differ diff --git a/screenshots/output/model-manager.png b/screenshots/output/model-manager.png index eed76ed..df43475 100644 Binary files a/screenshots/output/model-manager.png and b/screenshots/output/model-manager.png differ diff --git a/screenshots/output/model-properties.png b/screenshots/output/model-properties.png new file mode 100644 index 0000000..698442f Binary files /dev/null and b/screenshots/output/model-properties.png differ diff --git a/screenshots/output/model.gif b/screenshots/output/model.gif deleted file mode 100644 index a629cf6..0000000 Binary files a/screenshots/output/model.gif and /dev/null differ diff --git a/screenshots/output/provider-add-openai-compatible.png b/screenshots/output/provider-add-openai-compatible.png new file mode 100644 index 0000000..0feb361 Binary files /dev/null and b/screenshots/output/provider-add-openai-compatible.png differ diff --git a/screenshots/output/provider-manager.png b/screenshots/output/provider-manager.png index 0d717df..2a07f48 100644 Binary files a/screenshots/output/provider-manager.png and b/screenshots/output/provider-manager.png differ diff --git a/screenshots/output/provider.gif b/screenshots/output/provider.gif deleted file mode 100644 index bf71a5f..0000000 Binary files a/screenshots/output/provider.gif and /dev/null differ diff --git a/screenshots/output/reminder-builder-review.png b/screenshots/output/reminder-builder-review.png new file mode 100644 index 0000000..8e165e3 Binary files /dev/null and b/screenshots/output/reminder-builder-review.png differ diff --git a/screenshots/output/reminder-list.png b/screenshots/output/reminder-list.png index 01d1083..73df3ad 100644 Binary files a/screenshots/output/reminder-list.png and b/screenshots/output/reminder-list.png differ diff --git a/screenshots/output/reminder-run-approval.png b/screenshots/output/reminder-run-approval.png new file mode 100644 index 0000000..8756e8f Binary files /dev/null and b/screenshots/output/reminder-run-approval.png differ diff --git a/screenshots/output/reminder.gif b/screenshots/output/reminder.gif index d4fd796..e132899 100644 Binary files a/screenshots/output/reminder.gif and b/screenshots/output/reminder.gif differ diff --git a/screenshots/output/sessions-tui.png b/screenshots/output/sessions-tui.png new file mode 100644 index 0000000..f9b18df Binary files /dev/null and b/screenshots/output/sessions-tui.png differ diff --git a/screenshots/output/single-shot-chat-web-fetch.png b/screenshots/output/single-shot-chat-web-fetch.png deleted file mode 100644 index 7aa7db5..0000000 Binary files a/screenshots/output/single-shot-chat-web-fetch.png and /dev/null differ diff --git a/screenshots/output/single-shot-cli-execution.png b/screenshots/output/single-shot-cli-execution.png index efe878c..17483ac 100644 Binary files a/screenshots/output/single-shot-cli-execution.png and b/screenshots/output/single-shot-cli-execution.png differ diff --git a/screenshots/output/skill-list.png b/screenshots/output/skill-list.png new file mode 100644 index 0000000..90450b1 Binary files /dev/null and b/screenshots/output/skill-list.png differ diff --git a/screenshots/output/skill-sync.png b/screenshots/output/skill-sync.png new file mode 100644 index 0000000..8b17694 Binary files /dev/null and b/screenshots/output/skill-sync.png differ diff --git a/screenshots/output/slack-channel-integration.png b/screenshots/output/slack-channel-integration.png deleted file mode 100644 index c7d2abf..0000000 Binary files a/screenshots/output/slack-channel-integration.png and /dev/null differ diff --git a/screenshots/output/slack-install-app.png b/screenshots/output/slack-install-app.png index f5b1b07..4403239 100644 Binary files a/screenshots/output/slack-install-app.png and b/screenshots/output/slack-install-app.png differ diff --git a/screenshots/output/slash-unknown-command.png b/screenshots/output/slash-unknown-command.png new file mode 100644 index 0000000..1d69694 Binary files /dev/null and b/screenshots/output/slash-unknown-command.png differ diff --git a/screenshots/output/stats.gif b/screenshots/output/stats.gif deleted file mode 100644 index 7c9df47..0000000 Binary files a/screenshots/output/stats.gif and /dev/null differ diff --git a/screenshots/output/stats.png b/screenshots/output/stats.png index ae0a077..e5f38ad 100644 Binary files a/screenshots/output/stats.png and b/screenshots/output/stats.png differ diff --git a/screenshots/output/status.gif b/screenshots/output/status.gif deleted file mode 100644 index 4f7c742..0000000 Binary files a/screenshots/output/status.gif and /dev/null differ diff --git a/screenshots/output/status.png b/screenshots/output/status.png index 51597f8..3b9c8d2 100644 Binary files a/screenshots/output/status.png and b/screenshots/output/status.png differ diff --git a/screenshots/output/update-check.png b/screenshots/output/update-check.png new file mode 100644 index 0000000..f21f1c6 Binary files /dev/null and b/screenshots/output/update-check.png differ diff --git a/screenshots/output/webhooks-list.png b/screenshots/output/webhooks-list.png index e21be6d..3c7b166 100644 Binary files a/screenshots/output/webhooks-list.png and b/screenshots/output/webhooks-list.png differ diff --git a/screenshots/output/webhooks.gif b/screenshots/output/webhooks.gif deleted file mode 100644 index 0e9b8fe..0000000 Binary files a/screenshots/output/webhooks.gif and /dev/null differ diff --git a/screenshots/playwright/Dockerfile b/screenshots/playwright/Dockerfile new file mode 100644 index 0000000..73238e3 --- /dev/null +++ b/screenshots/playwright/Dockerfile @@ -0,0 +1,16 @@ +# Playwright runner for browser captures (Mattermost flow, docs page screenshots). +# Based on Microsoft's Playwright image so Chromium and its OS deps are preinstalled; +# the version is the one locked in package-lock.json (see screenshots/lib.sh), so the +# `playwright` module and the bundled browsers always match. +# +# The module is installed at /node_modules so scripts mounted anywhere under / resolve +# `import ... from 'playwright'` (ESM ignores NODE_PATH). +ARG PLAYWRIGHT_VERSION +FROM mcr.microsoft.com/playwright:v${PLAYWRIGHT_VERSION}-noble +ARG PLAYWRIGHT_VERSION +RUN mkdir /opt/pw && cd /opt/pw \ + && npm install --no-save --no-audit --no-fund --ignore-scripts playwright@${PLAYWRIGHT_VERSION} \ + && ln -s /opt/pw/node_modules /node_modules \ + && node -e "console.log('playwright', require('playwright/package.json').version)" +ENV HOME=/tmp +WORKDIR /work diff --git a/screenshots/scripts/docker-health-down.sh b/screenshots/scripts/docker-health-down.sh new file mode 100755 index 0000000..f003daf --- /dev/null +++ b/screenshots/scripts/docker-health-down.sh @@ -0,0 +1,10 @@ +#!/bin/bash +# Remove the container docker-health-up.sh started (found by its label, so a container that merely +# shares the name is left alone). Safe to run when there is nothing to remove. +set -u +ids="$(docker ps -aq --filter "label=netclaw-docs-capture=docker-health")" +if [ -n "$ids" ]; then + # shellcheck disable=SC2086 + docker rm -fv $ids >/dev/null +fi +echo "docker-health: cleaned up" diff --git a/screenshots/scripts/docker-health-up.sh b/screenshots/scripts/docker-health-up.sh new file mode 100755 index 0000000..c4a7d22 --- /dev/null +++ b/screenshots/scripts/docker-health-up.sh @@ -0,0 +1,33 @@ +#!/bin/bash +# Start a release container for docker-health.tape through the host's Docker and wait until Docker +# reports it healthy, then give it the name the docs use. Runs inside the capture container, which +# has the host's docker socket for this tape only (compose/docker-host.yml). +# +# It never touches a container it didn't start: if something on the host is already called +# `netclaw`, it stops instead of removing it. The tape removes its own container afterwards by label +# (docker-health-down.sh). +set -eu + +IMAGE="${NETCLAW_IMAGE:-ghcr.io/netclaw-dev/netclaw:0.28.2}" +TMP_NAME="${COMPOSE_PROJECT_NAME:-netclaw-shots}-docker-health" +LABEL="netclaw-docs-capture=docker-health" + +if docker inspect netclaw >/dev/null 2>&1; then + echo "docker-health: a container named 'netclaw' already exists on this host. Stop it or remove it first." >&2 + exit 1 +fi + +docker run -d --name "$TMP_NAME" --label "$LABEL" --stop-timeout 60 \ + -e NETCLAW_Daemon__DisableSelfUpdate=true "$IMAGE" >/dev/null + +for _ in $(seq 1 60); do + status="$(docker inspect --format '{{.State.Health.Status}}' "$TMP_NAME")" + if [ "$status" = healthy ]; then + docker rename "$TMP_NAME" netclaw + echo "container is healthy" + exit 0 + fi + sleep 2 +done +echo "docker-health: container never became healthy" >&2 +exit 1 diff --git a/screenshots/scripts/lib-seed.sh b/screenshots/scripts/lib-seed.sh new file mode 100755 index 0000000..2c7e4dc --- /dev/null +++ b/screenshots/scripts/lib-seed.sh @@ -0,0 +1,34 @@ +#!/bin/bash +# Helpers for the seed-*.sh scripts. Source it. These run inside the capture container as the +# `netclaw` user, which shares the daemon's network namespace and ~/.netclaw volume, so the +# plain `netclaw` CLI talks to the daemon exactly as a user's shell would. The capture image has +# python3 but no jq. + +NC_CONFIG="${HOME}/.netclaw/config/netclaw.json" + +# wait_daemon: block until `netclaw status` reports a healthy daemon. The daemon restarts +# itself in-process after every netclaw.json change, so call this after each write. +wait_daemon() { + local i + sleep 3 + for i in $(seq 1 60); do + if netclaw status 2>/dev/null | grep -q '^overall: healthy'; then return 0; fi + sleep 2 + done + echo "seed: daemon did not become healthy" >&2 + return 1 +} + +# json_set
: merge a JSON object into one top-level section of netclaw.json +# (replacing the keys it names, keeping the rest of the section). +json_set() { + python3 - "$NC_CONFIG" "$1" "$2" <<'PY' +import json, sys +path, section, patch = sys.argv[1], sys.argv[2], json.loads(sys.argv[3]) +with open(path) as f: + cfg = json.load(f) +cfg.setdefault(section, {}).update(patch) +with open(path, "w") as f: + json.dump(cfg, f, indent=2) +PY +} diff --git a/screenshots/scripts/mock-mattermost.py b/screenshots/scripts/mock-mattermost.py new file mode 100755 index 0000000..31e5420 --- /dev/null +++ b/screenshots/scripts/mock-mattermost.py @@ -0,0 +1,45 @@ +#!/usr/bin/env python3 +"""Just enough of the Mattermost REST API for `netclaw config` to resolve channel labels. + +The channel permissions screen looks channel IDs up through GET /api/v4/users/me, the bot's teams +and each team's channels. Nothing else is implemented, so this is a fixture for one screenshot +(config-channel-audiences.png), not a Mattermost server. Listens on 127.0.0.1:8065. +""" +import json +from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer + +TEAM = "t4m1d9q2w6x8c3v5b7n1k0j2hz" +CHANNELS = [ + {"id": "c1a5s7i9s3t0a8n2t6b4o1t5k7", "name": "assistant", "display_name": "Assistant", "type": "O"}, + {"id": "c2o8p4s6r1o9o3m5d7e0m2o4k6", "name": "ops-room", "display_name": "Ops Room", "type": "O"}, + {"id": "c3l9u5n1c7h2t4a6b8l0e1d3k5", "name": "lunch", "display_name": "Lunch", "type": "O"}, +] + + +class Handler(BaseHTTPRequestHandler): + def _send(self, body): + data = json.dumps(body).encode() + self.send_response(200) + self.send_header("Content-Type", "application/json") + self.send_header("Content-Length", str(len(data))) + self.end_headers() + self.wfile.write(data) + + def do_GET(self): + path = self.path.split("?")[0] + if path == "/api/v4/users/me": + self._send({"id": "u7b0t2i4d6n8e0t2c4l6a8w0ka", "username": "netclaw-bot"}) + elif path == "/api/v4/users/me/teams": + self._send([{"id": TEAM, "name": "demo"}]) + elif path == f"/api/v4/users/me/teams/{TEAM}/channels": + self._send(CHANNELS) + else: + self.send_response(404) + self.end_headers() + + def log_message(self, *args): + pass + + +if __name__ == "__main__": + ThreadingHTTPServer(("127.0.0.1", 8065), Handler).serve_forever() diff --git a/screenshots/scripts/ollama-init.sh b/screenshots/scripts/ollama-init.sh index 5b3b5a2..c2809cc 100755 --- a/screenshots/scripts/ollama-init.sh +++ b/screenshots/scripts/ollama-init.sh @@ -2,10 +2,12 @@ set -eu MODEL="${OLLAMA_MODEL:-qwen2:0.5b}" +# Runs in the stack's shared network namespace, so Ollama is on loopback. +OLLAMA="http://127.0.0.1:11434" echo "Waiting for Ollama API to become available..." for i in $(seq 1 30); do - if curl -fsS http://ollama:11434/api/tags >/dev/null; then + if curl -fsS "$OLLAMA/api/tags" >/dev/null; then echo "Ollama is ready." break fi @@ -14,9 +16,9 @@ for i in $(seq 1 30); do done echo "Pulling model: $MODEL" -curl -fsS -X POST http://ollama:11434/api/pull -d "{\"name\":\"$MODEL\"}" +curl -fsS -X POST "$OLLAMA/api/pull" -d "{\"name\":\"$MODEL\"}" echo "Verifying model is available..." -curl -fsS http://ollama:11434/api/tags +curl -fsS "$OLLAMA/api/tags" echo "Model init completed." diff --git a/screenshots/scripts/redact-third-party.py b/screenshots/scripts/redact-third-party.py new file mode 100644 index 0000000..33b56ad --- /dev/null +++ b/screenshots/scripts/redact-third-party.py @@ -0,0 +1,65 @@ +#!/usr/bin/env python3 +"""Minimal, declared redactions for the third-party (Slack and Discord) screenshots. + +These images were captured on the maintainer's machine. We keep them, but remove: + + slack-install-app.png app switcher with a truncated real app name (solid box) + discord-setup-applications.png browser toolbar: URL bar, extension icons (crop) + discord-setup-oauth-url.png client_id value (solid box) + discord-setup-authorize-dialog.png the signed-in Discord handle and the debug app's name, + twice (solid boxes; the dialog still shows the + permissions list and the Authorize button) + +Boxes use the colour sampled next to the box, so they read as empty space. No blur. +Each edit checks the source dimensions first, so running the script twice is a no-op. + +Usage (Pillow required): python3 screenshots/scripts/redact-third-party.py +""" +import os +import sys +from PIL import Image + +OUT = os.path.join(os.path.dirname(os.path.abspath(__file__)), "..", "output") + + +def load(name, expected_size): + path = os.path.join(OUT, name) + img = Image.open(path).convert("RGB") + if img.size != expected_size: + print(f"skip {name}: {img.size} is not the original {expected_size} (already edited?)") + return None, path + return img, path + + +def box(img, rect, sample_at): + """Fill rect (x0, y0, x1, y1) with the colour at sample_at.""" + colour = img.getpixel(sample_at) + img.paste(colour, rect) + + +def main(): + img, path = load("slack-install-app.png", (992, 334)) + if img: + box(img, (40, 44, 254, 100), (20, 70)) # app switcher, left rail background + img.save(path); print("ok slack-install-app.png") + + img, path = load("discord-setup-applications.png", (1522, 539)) + if img: + img.crop((0, 41, 1522, 539)).save(path); print("ok discord-setup-applications.png") + + img, path = load("discord-setup-oauth-url.png", (1406, 208)) + if img: + box(img, (383, 156, 542, 180), (1000, 168)) # client_id digits, input background + img.save(path); print("ok discord-setup-oauth-url.png") + + + img, path = load("discord-setup-authorize-dialog.png", (506, 804)) + if img: + box(img, (144, 148, 318, 172), (110, 160)) # app name in the heading, dialog background + box(img, (209, 207, 306, 226), (110, 160)) # handle after "Signed in as", dialog background + box(img, (255, 283, 374, 300), (430, 292)) # app name in the permissions sentence, panel background + img.save(path); print("ok discord-setup-authorize-dialog.png") + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/screenshots/scripts/seed-approvals.sh b/screenshots/scripts/seed-approvals.sh deleted file mode 100755 index 7e7b8f2..0000000 --- a/screenshots/scripts/seed-approvals.sh +++ /dev/null @@ -1,42 +0,0 @@ -#!/bin/bash -# Seeds a realistic tool-approvals.json for screenshot capture. -# Called by the approvals VHS tape before recording. -set -eu - -NETCLAW_HOME="${NETCLAW_HOME:-$HOME/.netclaw}" -CONFIG_DIR="$NETCLAW_HOME/config" -APPROVALS_FILE="$CONFIG_DIR/tool-approvals.json" - -mkdir -p "$CONFIG_DIR" - -# Clean up any existing approvals and quarantine files -rm -f "$APPROVALS_FILE" "$APPROVALS_FILE.v1.bak" "$APPROVALS_FILE.invalid" - -cat > "$APPROVALS_FILE" << 'SEED' -{ - "version": 2, - "audiences": { - "personal": { - "shell_execute": [ - { "verb": "git push", "directory": null }, - { "verb": "git pull", "directory": null }, - { "verb": "git remote", "directory": "/home/user/repos/netclaw/" }, - { "verb": "docker compose", "directory": "/home/user/repos/netclaw/" }, - { "verb": "npm run", "directory": "/home/user/repos/netclaw-website/" }, - { "verb": "ls", "directory": null } - ], - "mcp:demo-utilities:write_file": [ - { "verb": "write_file", "directory": "/home/user/repos/netclaw/" } - ] - }, - "team": { - "shell_execute": [ - { "verb": "git pull", "directory": null }, - { "verb": "git status", "directory": null } - ] - } - } -} -SEED - -echo "Seeded $APPROVALS_FILE" diff --git a/screenshots/scripts/seed-assistant.sh b/screenshots/scripts/seed-assistant.sh new file mode 100755 index 0000000..c51b743 --- /dev/null +++ b/screenshots/scripts/seed-assistant.sh @@ -0,0 +1,37 @@ +#!/bin/bash +# Seed for the tapes that need a working assistant (status, doctor, stats, sessions, chat): one +# OpenAI-compatible provider answered by the stack's forwarder (INFERENCE_UPSTREAM, see +# compose/forwarder.yml), all three model roles on it, and the demo MCP server. +# +# The model ID is the neutral name the forwarder presents (demo-model). It rewrites the upstream's +# real model id to that name in responses, and llama.cpp serves whatever model it has loaded +# regardless of the name in the request, so the real name never reaches netclaw.json or the screen. +# +# Also writes the Security and Identity sections `netclaw init` writes (Personal posture), so the +# config is complete and `netclaw doctor` has no failures. Needs an empty ~/.netclaw +# (`# stack: bare`, which brings up the demo MCP server container). +set -eu +# shellcheck source=lib-seed.sh +. "$(dirname "${BASH_SOURCE[0]}")/lib-seed.sh" + +MODEL_ID="${ASSISTANT_MODEL_ID:-demo-model}" + +netclaw provider add lab-llama openai-compatible --endpoint "${LAB_LLAMA_ENDPOINT:-http://llm.example.internal:8081}" >/dev/null +for role in main fallback compaction; do + netclaw model set "$role" lab-llama "$MODEL_ID" >/dev/null +done +netclaw mcp add --transport http demo-utilities http://127.0.0.1:8080/mcp >/dev/null + +json_set Security '{"DeploymentPosture": "Personal", "ShellExecutionMode": "HostAllowed", "StrictDefaults": true}' +json_set Tools '{"ShellMode": "HostAllowed"}' +json_set Identity '{"AgentName": "Netclaw", "CommunicationStyle": "Concise & casual", "UserTimezone": "Etc/UTC", "UserName": "alice"}' + +for i in $(seq 1 45); do + if netclaw status 2>/dev/null | grep -q 'mcp:demo-utilities: healthy'; then + echo "seeded: provider lab-llama, model $MODEL_ID, MCP demo-utilities" + exit 0 + fi + sleep 2 +done +echo "seed: demo-utilities did not report healthy" >&2 +exit 1 diff --git a/screenshots/scripts/seed-channel-audiences.sh b/screenshots/scripts/seed-channel-audiences.sh new file mode 100755 index 0000000..b1a8135 --- /dev/null +++ b/screenshots/scripts/seed-channel-audiences.sh @@ -0,0 +1,22 @@ +#!/bin/bash +# Seed for the channel audiences screen: enables a Mattermost adapter with three channels and a mix +# of audiences, pointed at the API fixture (mock-mattermost.py, which the tape starts first). The +# adapter can't actually connect, which only affects `netclaw status`, not the config screen. +set -eu +# shellcheck source=lib-seed.sh +. "$(dirname "${BASH_SOURCE[0]}")/lib-seed.sh" + +netclaw secrets set Mattermost.BotToken demo-bot-token-0000 >/dev/null +json_set Mattermost '{ + "Enabled": true, + "ServerUrl": "http://127.0.0.1:8065", + "AllowDirectMessages": true, + "AllowedChannelIds": ["c1a5s7i9s3t0a8n2t6b4o1t5k7", "c2o8p4s6r1o9o3m5d7e0m2o4k6", "c3l9u5n1c7h2t4a6b8l0e1d3k5"], + "ChannelAudiences": { + "c1a5s7i9s3t0a8n2t6b4o1t5k7": "Team", + "c2o8p4s6r1o9o3m5d7e0m2o4k6": "Team", + "c3l9u5n1c7h2t4a6b8l0e1d3k5": "Public" + } +}' +sleep 5 +echo "seeded: Mattermost adapter with three channels" diff --git a/screenshots/scripts/seed-config-demo.sh b/screenshots/scripts/seed-config-demo.sh new file mode 100755 index 0000000..b7d3402 --- /dev/null +++ b/screenshots/scripts/seed-config-demo.sh @@ -0,0 +1,32 @@ +#!/bin/bash +# Seed for the `netclaw config` tape, on top of a completed `netclaw init` (Personal): a +# projects folder for the workspaces picker, one inbound webhook route, and the Claude Code +# skills folder as an external skill source. Neutral names throughout. +set -eu +# shellcheck source=lib-seed.sh +. "$(dirname "${BASH_SOURCE[0]}")/lib-seed.sh" + +mkdir -p "$HOME/projects/website" "$HOME/projects/notes" "$HOME/projects/infra" +json_set Workspaces '{"Directory": "/home/netclaw/projects"}' +wait_daemon + +netclaw webhooks set github-issues \ + --prompt "Triage the new GitHub issue and post a one-paragraph summary." \ + --secret demo-webhook-secret \ + --signature-header X-Hub-Signature-256 --signature-prefix "sha256=" \ + --events issues.opened,issues.closed >/dev/null +json_set Webhooks '{"Enabled": true}' +wait_daemon + +mkdir -p "$HOME/.claude/skills/release-notes" +cat > "$HOME/.claude/skills/release-notes/SKILL.md" <<'SKILL' +--- +name: release-notes +description: Draft release notes from a list of merged pull requests. +--- + +Group the merged pull requests by area and write one sentence per change. +SKILL +netclaw skill source add claude-code --well-known claude-code >/dev/null +wait_daemon +echo "seeded: workspaces, webhook route github-issues, skill source claude-code" diff --git a/screenshots/scripts/seed-demo-install.sh b/screenshots/scripts/seed-demo-install.sh new file mode 100755 index 0000000..4752b0b --- /dev/null +++ b/screenshots/scripts/seed-demo-install.sh @@ -0,0 +1,81 @@ +#!/bin/bash +# ONE coherent demo install for every status-like capture: status, doctor, stats, sessions, model, +# provider, skill list, webhooks list, reminder list, the config dashboard and its sub-screens. +# Because they all start from this script, no two images disagree about what is installed. +# +# providers lab-llama (OpenAI-compatible, answered by the stack's forwarder), my-ollama +# (the stack's local Ollama), my-deepseek (dummy key) +# roles main on lab-llama / demo-model, fallback and compaction on my-ollama / qwen2:0.5b +# MCP demo-utilities (the stack's demo server) +# skills two local, two from the corp-skills feed, one external source (claude-code) +# webhooks github-issues and deploy-notify +# reminders daily-uptime-check and weekly-report +# workspaces /home/netclaw/projects with website (a Git repo), notes and infra +# search DuckDuckGo, set the way the Search screen saves it +# +# Run inside the capture container against an empty ~/.netclaw (`# stack: bare`, which also brings +# up the demo MCP and skill servers). Prints "seeded: demo install" when done. +set -eu +here="$(dirname "${BASH_SOURCE[0]}")" +# shellcheck source=lib-seed.sh +. "$here/lib-seed.sh" + +netclaw provider add lab-llama openai-compatible --endpoint "${LAB_LLAMA_ENDPOINT:-http://llm.example.internal:8081}" >/dev/null +netclaw provider add my-ollama ollama --endpoint http://127.0.0.1:11434 >/dev/null +netclaw provider add my-deepseek deepseek --api-key sk-demo-0000 >/dev/null +netclaw model set main lab-llama "${ASSISTANT_MODEL_ID:-demo-model}" >/dev/null +netclaw model set fallback my-ollama qwen2:0.5b >/dev/null +netclaw model set compaction my-ollama qwen2:0.5b >/dev/null +netclaw mcp add --transport http demo-utilities http://127.0.0.1:8080/mcp >/dev/null + +# The sections `netclaw init` writes for a Personal install, plus a Search backend. +json_set Security '{"DeploymentPosture": "Personal", "ShellExecutionMode": "HostAllowed", "StrictDefaults": true}' +json_set Tools '{"ShellMode": "HostAllowed"}' +json_set Identity '{"AgentName": "Netclaw", "CommunicationStyle": "Concise & casual", "UserTimezone": "Etc/UTC", "UserName": "alice"}' +json_set Search '{"Backend": "duckduckgo"}' +wait_daemon + +# Workspaces: the website repository plus two plain folders for the picker. +bash "$here/seed-projects.sh" +mkdir -p "$HOME/projects/notes" "$HOME/projects/infra" +wait_daemon + +# Skills: local skills, the corp-skills feed, one external source. +bash "$here/seed-skills.sh" +mkdir -p "$HOME/.claude/skills/release-notes" +cat > "$HOME/.claude/skills/release-notes/SKILL.md" <<'SKILL' +--- +name: release-notes +description: Draft release notes from a list of merged pull requests. +--- + +Group the merged pull requests by area and write one sentence per change. +SKILL +netclaw skill source add claude-code --well-known claude-code >/dev/null +wait_daemon +netclaw skill sync >/dev/null + +# Two inbound webhook routes. +netclaw webhooks set github-issues \ + --prompt "Triage the new GitHub issue and post a one-paragraph summary." \ + --secret demo-webhook-secret \ + --signature-header X-Hub-Signature-256 --signature-prefix "sha256=" \ + --events issues.opened,issues.closed --audience personal >/dev/null +netclaw webhooks set deploy-notify \ + --prompt "A deployment finished. Summarize the event payload in one line." \ + --secret demo-webhook-secret --audience personal >/dev/null +json_set Webhooks '{"Enabled": true}' +wait_daemon + +# Two reminders. +bash "$here/seed-reminders.sh" + +for i in $(seq 1 45); do + if netclaw status 2>/dev/null | grep -q 'mcp:demo-utilities: healthy'; then + echo "seeded: demo install" + exit 0 + fi + sleep 2 +done +echo "seed: demo-utilities did not report healthy" >&2 +exit 1 diff --git a/screenshots/scripts/seed-paired-device.sh b/screenshots/scripts/seed-paired-device.sh new file mode 100755 index 0000000..a879635 --- /dev/null +++ b/screenshots/scripts/seed-paired-device.sh @@ -0,0 +1,10 @@ +#!/bin/bash +# Pair one device, "alice-laptop", the way a second machine would: take a code from the daemon host +# and enter it on the device. The device side runs with its own NETCLAW_HOME, so the token and the +# endpoint land there and the daemon host's CLI configuration stays untouched. +set -eu + +code="$(netclaw daemon pair | awk '/Pairing code/ {print $3}')" +mkdir -p /tmp/alice-laptop +printf '%s\nalice-laptop\n' "$code" | NETCLAW_HOME=/tmp/alice-laptop netclaw pair http://127.0.0.1:5199 >/dev/null +echo "seeded: paired device alice-laptop" diff --git a/screenshots/scripts/seed-projects.sh b/screenshots/scripts/seed-projects.sh new file mode 100644 index 0000000..58aa121 --- /dev/null +++ b/screenshots/scripts/seed-projects.sh @@ -0,0 +1,28 @@ +#!/bin/bash +# Creates the neutral demo workspace used by the chat, approvals and reminder tapes: a small +# Git repository at /home/netclaw/projects/website (shared with the daemon by +# screenshots/compose/projects.yml) declared as the workspace root, so shell calls run +# inside the repository and approval prompts offer "This repository". Idempotent. +# Run inside the capture container. +set -eu + +repo="$HOME/projects/website" +mkdir -p "$repo" +cd "$repo" +if [ ! -d .git ]; then + git init -q -b main . + git config user.name alice + git config user.email alice@example.com + printf '# website\n\nStatic marketing site. Edit files under src/.\n' > AGENTS.md + mkdir -p src + printf '

Hello

\n' > src/index.html + git add . + git commit -q -m "Initial commit" +fi + +# Declare the workspace root. The daemon restarts itself when netclaw.json changes. +cfg="$NETCLAW_HOME/config/netclaw.json" +if [ "$(jq -r '.Workspaces.Directory // ""' "$cfg")" != "$HOME/projects" ]; then + jq --arg d "$HOME/projects" '.Workspaces = {Directory: $d}' "$cfg" > "$cfg.tmp" && mv "$cfg.tmp" "$cfg" + sleep 8 +fi diff --git a/screenshots/scripts/seed-providers.sh b/screenshots/scripts/seed-providers.sh new file mode 100755 index 0000000..7678c05 --- /dev/null +++ b/screenshots/scripts/seed-providers.sh @@ -0,0 +1,31 @@ +#!/bin/bash +# Seed for the provider and model manager tapes: three providers with neutral names and all three +# model roles assigned, written with the real CLI the way a user would after init. +# +# my-ollama ollama the stack's local Ollama (reachable) +# lab-llama openai-compatible http://llm.example.internal:8081, answered by the stack's +# forwarder (compose/forwarder.yml) when INFERENCE_UPSTREAM is set, +# unreachable otherwise +# my-deepseek deepseek a dummy key, so the provider can't authenticate +# +# Also writes the Security and Identity sections `netclaw init` writes, so the config is complete. +# Needs an empty ~/.netclaw (`# stack: fresh` or `# stack: bare`). +set -eu +# shellcheck source=lib-seed.sh +. "$(dirname "${BASH_SOURCE[0]}")/lib-seed.sh" + +netclaw provider add my-ollama ollama --endpoint http://127.0.0.1:11434 >/dev/null +netclaw provider add lab-llama openai-compatible --endpoint "${LAB_LLAMA_ENDPOINT:-http://llm.example.internal:8081}" >/dev/null +netclaw provider add my-deepseek deepseek --api-key sk-demo-0000 >/dev/null + +# The OpenAI-compatible type label is long enough to push the model manager's columns out of line, +# so lab-llama stays out of the role table. +netclaw model set main my-deepseek deepseek-chat >/dev/null +netclaw model set fallback my-ollama qwen2:0.5b >/dev/null +netclaw model set compaction my-ollama qwen2:0.5b >/dev/null + +json_set Security '{"DeploymentPosture": "Personal", "ShellExecutionMode": "HostAllowed", "StrictDefaults": true}' +json_set Tools '{"ShellMode": "HostAllowed"}' +json_set Identity '{"AgentName": "Netclaw", "CommunicationStyle": "Concise & casual", "UserTimezone": "Etc/UTC", "UserName": "alice"}' +wait_daemon +echo "seeded: providers my-ollama, lab-llama, my-deepseek; roles main (my-deepseek), fallback and compaction (my-ollama)" diff --git a/screenshots/scripts/seed-reminders.sh b/screenshots/scripts/seed-reminders.sh new file mode 100644 index 0000000..c175b33 --- /dev/null +++ b/screenshots/scripts/seed-reminders.sh @@ -0,0 +1,10 @@ +#!/bin/bash +# Neutral reminders for the reminder tapes: a daily cron reminder and a weekly one. +# Run inside the capture container (talks to the stack's daemon). +set -eu +netclaw reminder create daily-uptime-check cron "0 9 * * *" \ + "Check that the website and API respond, and note anything slower than usual." \ + --name "Daily uptime check" >/dev/null +netclaw reminder create weekly-report cron "CRON_TZ=America/Chicago 0 16 * * FRI" \ + "Summarize this week's completed tasks and open follow-ups in five bullets." \ + --name "Weekly report" >/dev/null diff --git a/screenshots/scripts/seed-skills.sh b/screenshots/scripts/seed-skills.sh new file mode 100644 index 0000000..6c659e6 --- /dev/null +++ b/screenshots/scripts/seed-skills.sh @@ -0,0 +1,91 @@ +#!/bin/bash +# Stages the skills the skill-list and skill-sync captures show: +# - two local skills in ~/.netclaw/skills (hello-greeter with a references/ file, flat-note) +# - two skills and one subagent published to the stack's skill server (port 8090) with the +# skillserver CLI, and the server added as the `corp-skills` feed in netclaw.json. +# Run inside the capture container. With `no-feed` as the first argument, only the local +# skills are created (nothing is published and no feed is configured). +set -eu + +skills="$HOME/.netclaw/skills" +mkdir -p "$skills/hello-greeter/references" "$skills/flat-note" +cat > "$skills/hello-greeter/SKILL.md" <<'SKILL' +--- +name: hello-greeter +description: Greet the user by name and mention one useful thing about their day. +metadata: + version: 1.0.0 +--- + +# Hello greeter + +Greet the user by name. Read `references/greetings.md` for tone before you answer. +SKILL +cat > "$skills/hello-greeter/references/greetings.md" <<'REF' +Keep greetings to one sentence. Warm, never gushing. +REF +cat > "$skills/flat-note/SKILL.md" <<'SKILL' +--- +name: flat-note +description: Capture a quick note as a single markdown bullet. +--- + +# Flat note + +Write the note as one markdown bullet and confirm it back in a sentence. +SKILL + +[ "${1:-}" = "no-feed" ] && exit 0 + +server=http://127.0.0.1:8090 +export SKILLSERVER_API_KEY=demo-publish-key +src="$(mktemp -d)" +agents="$(mktemp -d)" +mkdir -p "$src/k8s-log-triage/references" "$src/incident-summary" +cat > "$src/k8s-log-triage/SKILL.md" <<'SKILL' +--- +name: k8s-log-triage +description: Triage Kubernetes pod logs, group repeated errors and suggest the next check. +metadata: + version: 0.2.0 +--- + +# Kubernetes log triage + +Read `references/patterns.md`, group the errors by pattern and propose one next check. +SKILL +cat > "$src/k8s-log-triage/references/patterns.md" <<'REF' +CrashLoopBackOff: check the previous container's logs first. +ImagePullBackOff: check the registry credentials and the tag. +REF +cat > "$src/incident-summary/SKILL.md" <<'SKILL' +--- +name: incident-summary +description: Summarize an incident timeline in five bullets for a status update. +metadata: + version: 1.1.0 +--- + +# Incident summary + +Turn the timeline into five bullets: impact, cause, fix, follow-ups, owner. +SKILL +cat > "$agents/release-notes-writer.md" <<'AGENT' +--- +name: release-notes-writer +description: Draft release notes from a list of merged changes +modelRole: Compaction +timeoutSeconds: 90 +--- + +You write short release notes. Group the changes by user impact and keep each line under 100 characters. +AGENT + +skillserver publish-all "$src" --server-url "$server" >/dev/null 2>&1 || skillserver publish-all "$src" --server-url "$server" +skillserver publish-subagent "$agents/release-notes-writer.md" --version 1.0.0 --server-url "$server" >/dev/null +rm -rf "$src" "$agents" + +cfg="$NETCLAW_HOME/config/netclaw.json" +jq '.SkillFeeds = {SyncIntervalMinutes: 60, Feeds: [{Name: "corp-skills", Url: "http://127.0.0.1:8090", Enabled: true, TimeoutSeconds: 30}]}' \ + "$cfg" > "$cfg.tmp" && mv "$cfg.tmp" "$cfg" +sleep 15 diff --git a/screenshots/scripts/seed-webhooks.sh b/screenshots/scripts/seed-webhooks.sh new file mode 100644 index 0000000..e9775be --- /dev/null +++ b/screenshots/scripts/seed-webhooks.sh @@ -0,0 +1,16 @@ +#!/bin/bash +# Two neutral webhook routes for the webhooks tape. The secret is a throwaway demo value +# written to a temp file and read once by `set`. Run inside the capture container. +set -eu +cfg="$NETCLAW_HOME/config/netclaw.json" +if [ "$(jq -r '.Webhooks.Enabled // false' "$cfg")" != "true" ]; then + jq '.Webhooks = {Enabled: true}' "$cfg" > "$cfg.tmp" && mv "$cfg.tmp" "$cfg" + sleep 12 +fi +secret="$(mktemp)" +printf 'demo-signing-secret' > "$secret" +netclaw webhooks set github-issues --prompt "Triage this issue: label it and summarize it in two sentences." \ + --secret-file "$secret" --audience personal >/dev/null +netclaw webhooks set deploy-notify --prompt "A deployment finished. Summarize the event payload in one line." \ + --secret-file "$secret" --audience personal >/dev/null +rm -f "$secret" diff --git a/screenshots/shoot-page.sh b/screenshots/shoot-page.sh new file mode 100755 index 0000000..2687f18 --- /dev/null +++ b/screenshots/shoot-page.sh @@ -0,0 +1,30 @@ +#!/bin/bash +# Screenshot a page of the locally built docs site (dist/) in the Playwright container. +# +# npm run build +# ./screenshots/shoot-page.sh cli/status -> docs-logs/screenshots/cli-status.png +# SCREENSHOT_OUT=/some/dir ./screenshots/shoot-page.sh cli/status +# SCREENSHOT_BASE_URL=http://host.docker.internal:4321 ./screenshots/shoot-page.sh cli/status +# (a dev server started with `npm run dev -- --host 0.0.0.0`) +# +# Runs scripts/screenshot-page.mjs with the same pinned Playwright image as the +# Mattermost capture. No host browser or host node needed beyond building the site. +set -euo pipefail +# shellcheck source=lib.sh +. "$(dirname "${BASH_SOURCE[0]}")/lib.sh" + +[ "$#" -ge 1 ] || { echo "usage: shoot-page.sh e.g. cli/status" >&2; exit 2; } + +OUT="${SCREENSHOT_OUT:-$PROJECT_ROOT/docs-logs/screenshots}" +mkdir -p "$OUT" +OUT="$(cd "$OUT" && pwd)" + +IMAGE="netclaw-docs-playwright:$PLAYWRIGHT_VERSION" +docker build -q -t "$IMAGE" --build-arg "PLAYWRIGHT_VERSION=$PLAYWRIGHT_VERSION" "$SHOTS_DIR/playwright" >/dev/null + +env_args=(-e SCREENSHOT_OUT=/out) +[ -z "${SCREENSHOT_BASE_URL:-}" ] || env_args+=(-e "SCREENSHOT_BASE_URL=$SCREENSHOT_BASE_URL") + +docker run --rm --user "$HOST_UID:$HOST_GID" --add-host host.docker.internal:host-gateway \ + "${env_args[@]}" -v "$PROJECT_ROOT:/site:ro" -v "$OUT:/out" \ + "$IMAGE" node /site/scripts/screenshot-page.mjs "$1" | sed "s#^/out#$OUT#" diff --git a/screenshots/stack.sh b/screenshots/stack.sh new file mode 100755 index 0000000..3d938d2 --- /dev/null +++ b/screenshots/stack.sh @@ -0,0 +1,111 @@ +#!/bin/bash +# Bring up a disposable Netclaw daemon (the released image) for verifying commands and +# for tape recordings. Docker is the only prerequisite. Never uses a host netclaw. +# +# ./screenshots/stack.sh up [--fresh|--bare] start the stack and wait until the daemon is healthy +# ./screenshots/stack.sh reset [--fresh|--bare] wipe ~/.netclaw and restart the daemon (inference cache is kept) +# ./screenshots/stack.sh exec run a command in the daemon container as the netclaw user +# ./screenshots/stack.sh daemon-log [N] last N lines (default 200) of the daemon's log file +# ./screenshots/stack.sh logs [args] docker compose logs for the stack (container stdout) +# ./screenshots/stack.sh ps list the stack's containers +# ./screenshots/stack.sh down remove containers, network and volumes +# +# States: +# configured (default) provider, model roles and the demo MCP server are added with the real +# CLI (provider add / model set / mcp add), as a user would after init +# --fresh no model config and an empty ~/.netclaw: first-run state for `netclaw init` +# --bare empty ~/.netclaw like --fresh, but the demo MCP server container is up; the +# tape seeds its own config (see screenshots/scripts/seed-*.sh) +# +# Examples: +# COMPOSE_PROJECT_NAME=dr04 ./screenshots/stack.sh up +# COMPOSE_PROJECT_NAME=dr04 ./screenshots/stack.sh exec netclaw doctor +# COMPOSE_PROJECT_NAME=dr04 ./screenshots/stack.sh down +# +# Image: NETCLAW_VERSION=0.27.1 (default: src/version.json) or NETCLAW_IMAGE=. +# Inference: hermetic Ollama by default; set INFERENCE_BASE_URL + INFERENCE_MODEL_ID +# (env or screenshots/.env) to use an external OpenAI-compatible endpoint. +set -euo pipefail +# shellcheck source=lib.sh +. "$(dirname "${BASH_SOURCE[0]}")/lib.sh" + +state=configured +cmd="${1:-}"; [ "$#" -gt 0 ] && shift +if [ "${1:-}" = "--fresh" ]; then state=fresh; shift; fi +if [ "${1:-}" = "--bare" ]; then state=bare; shift; fi + +wait_healthy() { + local cid status i + cid="$(dc "$state" ps -q netclawd)" + for i in $(seq 1 90); do + status="$(docker inspect -f '{{.State.Health.Status}}' "$cid" 2>/dev/null || echo starting)" + if [ "$status" = healthy ]; then + echo "==> netclawd is healthy ($NETCLAW_IMAGE, project $COMPOSE_PROJECT_NAME, state $state, inference $INFERENCE_MODE)" + return 0 + fi + sleep 2 + done + echo "stack: netclawd did not become healthy" >&2 + dc "$state" logs --tail 80 netclawd >&2 || true + return 1 +} + +# Run the CLI in the daemon container as the netclaw user (no TTY). +nc() { dc all exec -T -u netclaw netclawd netclaw "$@"; } + +# Initialize a configured daemon the way a user would, with the real CLI. The daemon +# restarts itself in-process after each config write, so wait for MCP to connect. +seed() { + local i + nc provider add inference "$NC_INFERENCE_TYPE" --endpoint "$NC_INFERENCE_ENDPOINT" >/dev/null + for role in main fallback compaction; do + nc model set "$role" inference "$NC_INFERENCE_MODEL" >/dev/null + done + nc mcp add --transport http demo-utilities http://127.0.0.1:8080/mcp >/dev/null + # `netclaw init` also writes the Security section (no CLI command for it); add what the + # wizard writes for the Personal posture so `netclaw doctor` passes the Security Policy check. + dc all exec -T -u netclaw netclawd sh -c ' + f=/home/netclaw/.netclaw/config/netclaw.json + jq ".Security = {DeploymentPosture: \"Personal\", ShellExecutionMode: \"HostAllowed\", StrictDefaults: true}" "$f" > "$f.tmp" && mv "$f.tmp" "$f"' + for i in $(seq 1 45); do + if nc status 2>/dev/null | grep -q 'mcp:demo-utilities: healthy'; then + echo "==> seeded: provider inference ($NC_INFERENCE_TYPE), model $NC_INFERENCE_MODEL, MCP demo-utilities" + return 0 + fi + sleep 2 + done + echo "stack: daemon did not report demo-utilities healthy after seeding" >&2 + nc status >&2 || true + return 1 +} + +bring_up() { + dc "$state" up -d --build --remove-orphans + wait_healthy + if [ "$state" = configured ]; then seed; fi +} + +case "$cmd" in + up) + echo "==> Starting $NETCLAW_IMAGE as project $COMPOSE_PROJECT_NAME ($state, $INFERENCE_MODE inference)" + bring_up + ;; + reset) + dc all rm -sf netclawd >/dev/null + docker volume rm "$(home_volume)" >/dev/null 2>&1 || true + bring_up + ;; + daemon-log) dc all exec -T -u netclaw netclawd sh -c 'tail -n "${1:-200}" /home/netclaw/.netclaw/logs/daemon-*.log' _ "$@" ;; + exec) + [ "$#" -gt 0 ] || { echo "usage: stack.sh exec " >&2; exit 2; } + tty_flag=(); [ -t 0 ] || tty_flag=(-T) + dc all exec "${tty_flag[@]}" -u netclaw netclawd "$@" + ;; + logs) dc all logs "$@" ;; + ps) dc all ps "$@" ;; + down) dc all down -v --remove-orphans ;; + *) + sed -n '2,28p' "${BASH_SOURCE[0]}" | sed 's/^# \{0,1\}//' + exit 2 + ;; +esac diff --git a/screenshots/tapes/approvals-list.tape b/screenshots/tapes/approvals-list.tape new file mode 100644 index 0000000..f9a81a2 --- /dev/null +++ b/screenshots/tapes/approvals-list.tape @@ -0,0 +1,18 @@ +# approvals-list.png: `netclaw approvals list` with grants made by trust-verb and by real +# prompt answers (see lib/seed-approvals.tape). +Set Shell "bash" +Set WaitTimeout 240s +Set Theme "Dracula" +Set FontSize 16 +Set Width 1000 +Set Height 380 +Set Padding 20 +Set TypingSpeed 40ms + +Source "/tapes/include/seed-approvals.tape" + +Type "netclaw approvals list" +Enter +Sleep 2s +Screenshot screenshots/output/approvals-list.png +Sleep 2s diff --git a/screenshots/tapes/approvals-tui.tape b/screenshots/tapes/approvals-tui.tape new file mode 100644 index 0000000..834670a --- /dev/null +++ b/screenshots/tapes/approvals-tui.tape @@ -0,0 +1,28 @@ +# approvals-tui.png and approvals-tui-revoke-confirm.png: the approvals manager and its +# revoke confirmation, over the same grants as approvals-list.tape. +Set Shell "bash" +Set WaitTimeout 240s +Set Theme "Dracula" +Set FontSize 16 +Set Width 1000 +Set Height 430 +Set Padding 20 +Set TypingSpeed 40ms + +Source "/tapes/include/seed-approvals.tape" + +Type "netclaw approvals" +Enter +Sleep 3s +Screenshot screenshots/output/approvals-tui.png +Sleep 2s +Down +Sleep 500ms +Type "r" +Sleep 1s +Screenshot screenshots/output/approvals-tui-revoke-confirm.png +Sleep 2s +Escape +Sleep 500ms +Ctrl+Q +Sleep 1s diff --git a/screenshots/tapes/approvals.tape b/screenshots/tapes/approvals.tape deleted file mode 100644 index d19b914..0000000 --- a/screenshots/tapes/approvals.tape +++ /dev/null @@ -1,94 +0,0 @@ -# netclaw approvals — persistent tool approval management -# Seeds a realistic tool-approvals.json, then captures list + TUI + revoke flow -Output screenshots/output/approvals.gif - -Set Shell "bash" -Set Theme "Dracula" -Set Width 1200 -Set Height 700 -Set FontSize 16 -Set Padding 20 -Set TypingSpeed 50ms - -# Seed approvals data (hidden from recording) -Hide -Type "bash screenshots/scripts/seed-approvals.sh" -Enter -Sleep 500ms -Type "clear" -Enter -Sleep 500ms -Show - -# ═══════════════════════════════════════════════ -# SCENE 1: netclaw approvals list -# ═══════════════════════════════════════════════ - -Type "netclaw approvals list" -Enter -Sleep 2s - -Screenshot screenshots/output/approvals-list.png -Sleep 1s - -Type "clear" -Enter -Sleep 500ms - -# ═══════════════════════════════════════════════ -# SCENE 2: netclaw approvals list --json -# ═══════════════════════════════════════════════ - -Type "netclaw approvals list --json" -Enter -Sleep 2s - -Screenshot screenshots/output/approvals-list-json.png -Sleep 1s - -Type "clear" -Enter -Sleep 500ms - -# ═══════════════════════════════════════════════ -# SCENE 3: netclaw approvals TUI — browse and revoke -# ═══════════════════════════════════════════════ - -Type "netclaw approvals" -Enter -Sleep 3s - -# TUI list view — all approvals -Screenshot screenshots/output/approvals-tui.png -Sleep 500ms - -# Navigate down to an entry -Down -Sleep 300ms -Down -Sleep 300ms -Down -Sleep 300ms - -Screenshot screenshots/output/approvals-tui-selected.png -Sleep 500ms - -# Press Delete to initiate revoke -Delete -Sleep 1s - -# Revoke confirmation dialog -Screenshot screenshots/output/approvals-tui-revoke-confirm.png -Sleep 500ms - -# Confirm the revoke -Enter -Sleep 1s - -# Post-revoke state -Screenshot screenshots/output/approvals-tui-revoke-done.png -Sleep 1s - -# Exit TUI -Ctrl+Q -Sleep 1s diff --git a/screenshots/tapes/channel-audiences.tape b/screenshots/tapes/channel-audiences.tape new file mode 100644 index 0000000..3a702b6 --- /dev/null +++ b/screenshots/tapes/channel-audiences.tape @@ -0,0 +1,51 @@ +# netclaw config > Channels > Mattermost > Channels & Permissions: the per-channel audience screen +# stack: fresh +# The old ten-step init wizard had a Channels step. In 0.27.1 the same screen lives in `netclaw config`, once an adapter is enabled and has channels. Runs the +# real `netclaw init` out of sight, then enables a Mattermost adapter whose channels resolve against +# a small API fixture (scripts/mock-mattermost.py) so the rows show names, not raw IDs. +Output screenshots/output/channel-audiences.gif + +Set Shell "bash" +Set Theme "Dracula" +Set FontSize 16 +Set Width 1000 +Set Height 520 +Set Padding 20 +Set TypingSpeed 40ms + +Hide +Source "/tapes/include/init-personal.tape" +Type "nohup python3 /work/screenshots/scripts/mock-mattermost.py >/dev/null 2>&1 &" +Enter +Sleep 2s +Type "bash /work/screenshots/scripts/seed-channel-audiences.sh" +Enter +Sleep 25s +Type "clear" +Enter +Sleep 1s +Show + +Type "netclaw config" +Enter +Sleep 3s + +# 3. Channels, then Mattermost (the third adapter) +Down +Down +Sleep 400ms +Enter +Sleep 2s +Down +Down +Sleep 400ms +Enter +Sleep 3s + +# Manage channels and permissions +Enter +Sleep 4s +Screenshot screenshots/output/config-channel-audiences.png +Sleep 500ms +Ctrl+Q +Sleep 1s diff --git a/screenshots/tapes/chat-approval.tape b/screenshots/tapes/chat-approval.tape new file mode 100644 index 0000000..e3e4efe --- /dev/null +++ b/screenshots/tapes/chat-approval.tape @@ -0,0 +1,61 @@ +# Chat approval prompts: chat-approval-prompt.png (getting-started: home directory, five options), +# chat-tui-approval-prompt.png (cli/chat) and approval-prompt-tui.png (security/approvals). The last +# two run in the demo repository /home/netclaw/projects/website so all six options are offered. +# The session is idle at the prompt when the image is taken, so nothing is answered. +Set Shell "bash" +Set WaitTimeout 240s +Set Theme "Dracula" +Set FontSize 16 +Set Width 1000 +Set Height 540 +Set Padding 20 +Set TypingSpeed 40ms + +# First image: the getting-started prompt, run from the home directory (no repository), so the +# prompt offers five options, exactly as the tutorial's text block says. +Type "netclaw chat" +Enter +Sleep 3s +Type "Use curl to download https://example.com and save it as ~/example.html, then tell me how many bytes it is." +Enter +Wait+Screen@240s /5\. Deny/ +Sleep 1s +Screenshot screenshots/output/chat-approval-prompt.png +Sleep 2s +Ctrl+Q +Sleep 2s + +# The other two run inside the demo repository. +Hide +Type "bash screenshots/scripts/seed-projects.sh && cd ~/projects/website && clear" +Enter +Sleep 4s +Show + +Type "netclaw chat" +Enter +Sleep 3s +Type "Set the working directory to /home/netclaw/projects/website, then use shell_execute to run exactly: npm run build" +Enter +Wait+Screen@240s /6\. Deny/ +Sleep 1s +Screenshot screenshots/output/chat-tui-approval-prompt.png +Sleep 2s +Ctrl+Q +Sleep 2s + +Hide +Type "clear" +Enter +Show +Type "netclaw chat" +Enter +Sleep 3s +Type "Set the working directory to /home/netclaw/projects/website, then use shell_execute to run exactly: git push origin main" +Enter +Wait+Screen@240s /6\. Deny/ +Sleep 1s +Screenshot screenshots/output/approval-prompt-tui.png +Sleep 2s +Ctrl+Q +Sleep 2s diff --git a/screenshots/tapes/chat-first-greeting.tape b/screenshots/tapes/chat-first-greeting.tape new file mode 100644 index 0000000..3d685ad --- /dev/null +++ b/screenshots/tapes/chat-first-greeting.tape @@ -0,0 +1,44 @@ +# stack: fresh +# chat-first-greeting.png: the chat that `netclaw init` hands off to, with the agent's +# greeting and first question. Walks the real wizard (OpenAI-compatible provider, Personal +# posture) against the stack's inference endpoint; the wizard steps are not part of the image. +Set Shell "bash" +Set WaitTimeout 300s +Set Theme "Dracula" +Set FontSize 16 +Set Width 1000 +Set Height 600 +Set Padding 20 +Set TypingSpeed 40ms + +Hide +Type "netclaw init" +Enter +Sleep 4s +Type "6" +Sleep 2s +Backspace 30 +Type "http://llm.example.internal:8081" +Sleep 500ms +Enter +Sleep 2s +Enter +Wait+Screen@60s /Select a model/ +Enter +Sleep 2s +Enter +Sleep 2s +Enter +Sleep 2s +Type "Sam" +Enter +Sleep 2s +Enter +Sleep 2s +Enter +Wait+Screen@240s /Netclaw Chat/ +Wait+Screen@300s /Ready +\| +demo-model/ +Sleep 2s +Show +Screenshot screenshots/output/chat-first-greeting.png +Sleep 2s diff --git a/screenshots/tapes/chat-session.tape b/screenshots/tapes/chat-session.tape new file mode 100644 index 0000000..05a3f15 --- /dev/null +++ b/screenshots/tapes/chat-session.tape @@ -0,0 +1,44 @@ +# chat-tui-idle.png: a resumed session (replayed history, idle status bar) and +# slash-unknown-command.png: the reply to an unknown slash command (no model call). +Set Shell "bash" +Set WaitTimeout 240s +Set Theme "Dracula" +Set FontSize 16 +Set Width 1000 +Set Height 600 +Set Padding 20 +Set TypingSpeed 40ms + +Hide +Type "netclaw chat -p --resume demo-notes 'In two sentences, what is a good structure for weekly meeting notes?' > /dev/null && netclaw chat -p --resume demo-notes 'Turn that into a three-item checklist.' > /dev/null; echo SE''EDED" +Enter +Wait+Screen@240s /SEEDED/ +Type "clear" +Enter +Sleep 1s +Show + +Type "netclaw chat --resume demo-notes" +Enter +Wait+Screen@60s /End of history/ +Sleep 3s +Screenshot screenshots/output/chat-tui-idle.png +Sleep 2s +Ctrl+Q +Sleep 2s + +Hide +Type "clear" +Enter +Show +Type "netclaw chat" +Enter +Sleep 3s +Type "/nope" +Enter +Wait+Screen@30s /Unknown command/ +Sleep 2s +Screenshot screenshots/output/slash-unknown-command.png +Sleep 2s +Ctrl+Q +Sleep 2s diff --git a/screenshots/tapes/config.tape b/screenshots/tapes/config.tape new file mode 100644 index 0000000..eb010da --- /dev/null +++ b/screenshots/tapes/config.tape @@ -0,0 +1,133 @@ +# netclaw config: the settings dashboard and the areas the docs show, on the shared demo install +# stack: bare +# scripts/seed-demo-install.sh builds the install every status-like capture shares. 0.27.1 shows +# thirteen rows of the dashboard, all visible in a 520 px terminal. +# +# Produces: config-dashboard, config-retention, config-channels-menu, config-inbound-webhooks, config-skills, +# config-search, config-exposure. +Output screenshots/output/config.gif + +Set Shell "bash" +Set Theme "Dracula" +Set FontSize 16 +Set Width 1000 +Set Height 520 +Set Padding 20 +Set TypingSpeed 40ms + +Hide +Type "bash /work/screenshots/scripts/seed-demo-install.sh" +Enter +Wait+Screen@8m /seeded: demo install/ +Type "clear" +Enter +Sleep 1s +Show + +Type "netclaw config" +Enter +Sleep 3s +Screenshot screenshots/output/config-dashboard.png +Sleep 500ms + +# 11. Data Retention: a fresh install has no Retention key, so the row shows the default +Down +Down +Down +Down +Down +Down +Down +Down +Down +Down +Sleep 400ms +Enter +Sleep 2s +Screenshot screenshots/output/config-retention.png +Escape +Sleep 1s +Up +Up +Up +Up +Up +Up +Up +Up +Up +Up +Sleep 400ms + +# 3. Channels +Down +Down +Sleep 400ms +Enter +Sleep 2s +Screenshot screenshots/output/config-channels-menu.png +Escape +Sleep 1s + +# 4. Inbound Webhooks +Down +Down +Down +Sleep 400ms +Enter +Sleep 2s +Screenshot screenshots/output/config-inbound-webhooks.png +Escape +Sleep 1s + +# 5. Skill Sources +Down +Down +Down +Down +Sleep 400ms +Enter +Sleep 2s +Screenshot screenshots/output/config-skills.png +Escape +Sleep 1s + +# 6. Search +Down +Down +Down +Down +Down +Sleep 400ms +Enter +Sleep 2s +Screenshot screenshots/output/config-search.png +Escape +Sleep 1s + +# 9. Security & Access > Exposure Mode +Down +Down +Down +Down +Down +Down +Down +Down +Sleep 400ms +Enter +Sleep 1s +Down +Down +Down +Sleep 400ms +Enter +Sleep 2s +Screenshot screenshots/output/config-exposure.png +Escape +Sleep 1s +Escape +Sleep 1s + +Ctrl+Q +Sleep 1s diff --git a/screenshots/tapes/daemon-pair.tape b/screenshots/tapes/daemon-pair.tape new file mode 100644 index 0000000..d64b29f --- /dev/null +++ b/screenshots/tapes/daemon-pair.tape @@ -0,0 +1,32 @@ +# netclaw daemon pair and netclaw daemon devices: a pairing code and the paired-device list +# stack: fresh +# A device named alice-laptop is paired out of sight (scripts/seed-paired-device.sh runs the device +# side with its own NETCLAW_HOME), so the visible commands show a fresh code and a one-device list. +# The config seeded first (scripts/seed-providers.sh) only gives the CLI something to run against. +Output screenshots/output/daemon-pair.gif + +Set Shell "bash" +Set Theme "Dracula" +Set FontSize 16 +Set Width 1000 +Set Height 360 +Set Padding 20 +Set TypingSpeed 40ms + +Hide +Type "bash /work/screenshots/scripts/seed-providers.sh && bash /work/screenshots/scripts/seed-paired-device.sh" +Enter +Wait+Screen@3m /seeded: paired device/ +Type "clear" +Enter +Sleep 1s +Show + +Type "netclaw daemon pair" +Enter +Sleep 3s +Type "netclaw daemon devices" +Enter +Sleep 3s +Screenshot screenshots/output/daemon-pair.png +Sleep 500ms diff --git a/screenshots/tapes/docker-health.tape b/screenshots/tapes/docker-health.tape new file mode 100644 index 0000000..8eeb7a2 --- /dev/null +++ b/screenshots/tapes/docker-health.tape @@ -0,0 +1,44 @@ +# A netclaw container reaching healthy, as the Docker quick start checks it +# stack: fresh +# host-docker: true +# The only tape that talks to the host's Docker (the capture container gets the socket and the host's +# docker client, see compose/docker-host.yml). It starts its own container from the release image, +# gives it the name the docs use, and every command it prints is filtered to that one container, so +# nothing else on the host appears. Names its container -docker-health while it starts and +# renames it to `netclaw` just before the visible part; the end of the tape removes it (by label, so a container that merely shares the name is never touched). +Output screenshots/output/docker-health.gif + +Set Shell "bash" +Set Theme "Dracula" +Set FontSize 16 +Set Width 1600 +Set Height 300 +Set Padding 20 +Set TypingSpeed 40ms + +Hide +Type "bash /work/screenshots/scripts/docker-health-up.sh" +Enter +Wait+Screen@3m /container is healthy/ +Type "clear" +Enter +Sleep 1s +Show + +Type "docker ps --filter name=^netclaw$" +Enter +Sleep 2s +Type "docker inspect --format '{{.State.Health.Status}}' netclaw" +Enter +Sleep 2s +Type "docker exec netclaw netclaw daemon status" +Enter +Sleep 3s +Screenshot screenshots/output/docker-health.png +Sleep 500ms + +Hide +Type "bash /work/screenshots/scripts/docker-health-down.sh" +Enter +Sleep 5s +Show diff --git a/screenshots/tapes/doctor.tape b/screenshots/tapes/doctor.tape index 0fd471c..5618b05 100644 --- a/screenshots/tapes/doctor.tape +++ b/screenshots/tapes/doctor.tape @@ -1,16 +1,30 @@ -# netclaw doctor — configuration diagnostics +# netclaw doctor: all 24 checks and the exit-code line on a typical Personal install +# stack: bare +# Same demo install as status.tape (scripts/seed-demo-install.sh). A normal install has no secrets.json (nothing to put +# in it), so the one WARN is the real one and the exit code is 2. No check fails. The terminal is +# wide enough that no line wraps and tall enough for every check. Output screenshots/output/doctor.gif Set Shell "bash" Set Theme "Dracula" -Set Width 1200 +Set FontSize 16 +Set Width 1700 Set Height 700 -Set FontSize 14 Set Padding 20 -Set TypingSpeed 50ms +Set TypingSpeed 40ms -Type "netclaw doctor" +Hide +Type "bash /work/screenshots/scripts/seed-demo-install.sh" +Enter +Wait+Screen@8m /seeded: demo install/ +Sleep 20s +Type "clear" Enter -Sleep 5s +Sleep 1s +Show +Type "netclaw doctor" +Enter +Sleep 8s Screenshot screenshots/output/doctor.png +Sleep 500ms diff --git a/screenshots/tapes/help.tape b/screenshots/tapes/help.tape index 84f7667..5087f40 100644 --- a/screenshots/tapes/help.tape +++ b/screenshots/tapes/help.tape @@ -3,7 +3,7 @@ Output screenshots/output/help.png Set Shell "bash" Set Theme "Dracula" -Set Width 1200 +Set Width 1000 Set Height 600 Set FontSize 16 Set Padding 20 diff --git a/screenshots/tapes/include/init-personal.tape b/screenshots/tapes/include/init-personal.tape new file mode 100644 index 0000000..c12dfc7 --- /dev/null +++ b/screenshots/tapes/include/init-personal.tape @@ -0,0 +1,31 @@ +# Setup helper, not a tape: Source it between Hide and Show to run the real `netclaw init` wizard +# (Personal posture, Ollama, the stack's one model, identity "Netclaw" / "alice") against the +# first-run daemon. Leaves the shell at a clean prompt with a completed Personal setup, so a +# tape can start from what a reader has right after the quickstart. +# Needs `# stack: fresh` or `# stack: bare` (empty ~/.netclaw, Ollama on localhost:11434). +Type "netclaw init" +Enter +Sleep 3s +Type "4" +Sleep 1500ms +Enter +Sleep 5s +Enter +Sleep 2s +Enter +Sleep 1s +Enter +Sleep 1s +Type "alice" +Enter +Sleep 1s +Enter +Sleep 2s +Enter +# Health check, then the wizard hands over to chat +Sleep 14s +Ctrl+Q +Sleep 2s +Type "clear" +Enter +Sleep 1s diff --git a/screenshots/tapes/include/seed-approvals.tape b/screenshots/tapes/include/seed-approvals.tape new file mode 100644 index 0000000..3cbc357 --- /dev/null +++ b/screenshots/tapes/include/seed-approvals.tape @@ -0,0 +1,43 @@ +# Shared by approvals-list.tape and approvals-tui.tape (VHS `Source`). Everything between +# Hide and Show is not recorded. Builds the grants the approvals images show, through the +# real CLI and real prompt answers: two trust-verb grants, an MCP grant, a team grant, and +# a folder ("Always here") and a repository ("This repository") grant answered in the chat TUI. +Hide +Type "bash screenshots/scripts/seed-projects.sh && cd ~/projects/website && clear" +Enter +Sleep 2s +Type "netclaw approvals trust-verb 'git push' && netclaw approvals trust-verb gh && netclaw approvals trust-verb 'npm run' --audience team && netclaw approvals trust-verb demo-utilities/calculate --tool demo-utilities/calculate; echo SE''EDED" +Enter +Wait+Screen@60s /SEEDED/ +Type "clear" +Enter +Sleep 1s + +Type "netclaw chat" +Enter +Sleep 3s +Type "Call set_working_directory with /home/netclaw/projects/website, then stop. Run nothing else and ask me nothing." +Enter +Wait+Screen@240s /set_working_directory . \/home\/netclaw\/projects/ +Wait+Screen@240s /Ready +\| +demo-model/ +Sleep 1s +Type "Now use shell_execute to run exactly: npm run build" +Enter +Wait+Screen@240s /6\. Deny/ +Sleep 1s +Type "3" +Wait+Screen@240s /Ready +\| +demo-model/ +Sleep 1s +Type "Now use shell_execute to run exactly: make test" +Enter +Wait+Screen@240s /6\. Deny/ +Sleep 1s +Type "4" +Wait+Screen@240s /Ready +\| +demo-model/ +Sleep 1s +Ctrl+Q +Sleep 2s +Type "clear" +Enter +Sleep 1s +Show diff --git a/screenshots/tapes/init-existing.tape b/screenshots/tapes/init-existing.tape new file mode 100644 index 0000000..c59e6e5 --- /dev/null +++ b/screenshots/tapes/init-existing.tape @@ -0,0 +1,35 @@ +# netclaw init on an existing install: the action menu and the start-over scope chooser +# stack: fresh +# Runs the real wizard out of sight first, so the daemon has a completed Personal setup. +# Produces init-existing-menu and init-start-over-scope. +Output screenshots/output/init-existing.gif + +Set Shell "bash" +Set Theme "Dracula" +Set FontSize 16 +Set Width 1100 +Set Height 380 +Set Padding 20 +Set TypingSpeed 40ms + +Hide +Source "/tapes/include/init-personal.tape" +Show + +Type "netclaw init" +Enter +Sleep 3s +Screenshot screenshots/output/init-existing-menu.png +Sleep 500ms + +# Start over from scratch +Down +Sleep 300ms +Down +Sleep 300ms +Enter +Sleep 2s +Screenshot screenshots/output/init-start-over-scope.png +Sleep 500ms +Ctrl+Q +Sleep 1s diff --git a/screenshots/tapes/init-team.tape b/screenshots/tapes/init-team.tape new file mode 100644 index 0000000..781040b --- /dev/null +++ b/screenshots/tapes/init-team.tape @@ -0,0 +1,50 @@ +# netclaw init: the Team posture, which adds a feature-selection step ("Step 4 of 5") +# stack: fresh +# Same first-run state and local Ollama as init-wizard.tape. Produces init-04-enabled-features +# (feature selection, Team posture) (the five-step flow's fourth step). +Output screenshots/output/init-team.gif + +Set Shell "bash" +Set Theme "Dracula" +Set FontSize 16 +Set Width 1000 +Set Height 420 +Set Padding 20 +Set TypingSpeed 40ms + +Type "netclaw init" +Enter +Sleep 3s + +# Ollama, default endpoint, the one model +Type "4" +Sleep 1500ms +Enter +Sleep 5s +Enter +Sleep 2s + +# Identity: name, style, your name, timezone +Enter +Sleep 1s +Enter +Sleep 1s +Type "alice" +Enter +Sleep 1s +Enter +Sleep 2s + +# Security posture: Team +Down +Sleep 500ms +Enter +Sleep 2s + +# Feature selection: all six on for Team +Screenshot screenshots/output/init-04-enabled-features.png +Sleep 500ms +Enter +Sleep 3s +Ctrl+Q +Sleep 1s diff --git a/screenshots/tapes/init-wizard.tape b/screenshots/tapes/init-wizard.tape index 374cf33..d3dcec3 100644 --- a/screenshots/tapes/init-wizard.tape +++ b/screenshots/tapes/init-wizard.tape @@ -1,227 +1,83 @@ -# Netclaw Init Wizard - Full walkthrough (Personal + Ollama + DuckDuckGo) -# Requires: Ollama running on localhost:11434 with qwen2:0.5b -# Captures every wizard screen for documentation. -Output screenshots/output/init-full.gif +# netclaw init: the Personal walkthrough on 0.27.1 (provider, identity, security posture, health check) +# stack: fresh +# Runs against the first-run daemon (empty ~/.netclaw) with the stack's local Ollama on +# localhost:11434 (qwen2:0.5b, the stack's default). Nothing private is involved. +# +# The header reads "Step N of 5" until a posture is chosen, then "Step 4 of 4" for Personal. +# That is what 0.27.1 prints, so the provider, identity and posture images say "of 5" and the +# health check says "of 4". +# +# Produces: init-01-provider-list, init-01-endpoint, init-01-provider-ollama, init-01-model-select, +# init-02-identity, init-02-security-posture, init-05-health-check. +Output screenshots/output/init-wizard.gif Set Shell "bash" Set Theme "Dracula" -Set Width 1200 -Set Height 700 Set FontSize 16 +Set Width 1000 +Set Height 520 Set Padding 20 -Set TypingSpeed 50ms - -# Clean netclaw home for fresh wizard -Hide -Type "export NETCLAW_HOME=/tmp/netclaw-screenshot-home" -Enter -Sleep 300ms -Type "rm -rf /tmp/netclaw-screenshot-home && mkdir -p /tmp/netclaw-screenshot-home" -Enter -Sleep 300ms -Type "clear" -Enter -Sleep 500ms -Show - -# ═══════════════════════════════════════════════ -# STEP 1: LLM Provider -# ═══════════════════════════════════════════════ +Set TypingSpeed 40ms Type "netclaw init" Enter Sleep 3s -# Sub-step 0: Provider list (Anthropic highlighted by default) +# Provider list: eight entries Screenshot screenshots/output/init-01-provider-list.png -Sleep 300ms +Sleep 500ms -# Move to Ollama and capture (0.22 provider list: Anthropic, GitHub Copilot, Ollama) -Down -Sleep 300ms -Down -Sleep 300ms +# 4 = Ollama: its endpoint step +Type "4" +Sleep 1500ms Screenshot screenshots/output/init-01-provider-ollama.png -Sleep 300ms - -# Select Ollama -Enter -Sleep 2s - -# Sub-step 2: Endpoint input (default http://localhost:11434) +Sleep 500ms +Escape +Sleep 800ms +Escape +Sleep 1s + +# 6 = OpenAI-compatible: the endpoint field arrives pre-filled with http://localhost:11434, which +# is the stack's Ollama (it serves the OpenAI-compatible API too), so the wizard carries on with it. +Type "6" +Sleep 1500ms Screenshot screenshots/output/init-01-endpoint.png -Sleep 300ms +Sleep 500ms -# Accept default endpoint +# Accept the endpoint, skip the optional API key, wait for the connectivity check and model discovery +Enter +Sleep 1500ms Enter Sleep 5s - -# Sub-step 3/4: Model discovery + selection (qwen2:0.5b found) Screenshot screenshots/output/init-01-model-select.png -Sleep 300ms - -# Select qwen2:0.5b -Enter -Sleep 2s - -# ═══════════════════════════════════════════════ -# STEP 2: Security Posture -# ═══════════════════════════════════════════════ -Screenshot screenshots/output/init-02-security-posture.png -Sleep 300ms - -# Select Personal (already highlighted) -Enter -Sleep 2s - -# (Feature Selection is SKIPPED for Personal posture) - -# ═══════════════════════════════════════════════ -# STEP 3: Communication Channels (Channel Picker) -# ═══════════════════════════════════════════════ -Screenshot screenshots/output/init-03-channels.png -Sleep 300ms - -# Press 'd' to skip channels (done without selecting any) -Type "d" -Sleep 2s - -# ═══════════════════════════════════════════════ -# STEP 4: Web Search -# ═══════════════════════════════════════════════ -Screenshot screenshots/output/init-04-search.png -Sleep 300ms - -# Select DuckDuckGo (default, first item) -Enter -Sleep 2s - -# ═══════════════════════════════════════════════ -# STEP 5: Browser Automation -# ═══════════════════════════════════════════════ -Screenshot screenshots/output/init-05-browser.png -Sleep 300ms - -# Select "No" (skip browser automation, first item) -Enter -Sleep 2s - -# ═══════════════════════════════════════════════ -# STEP 6: Identity -# ═══════════════════════════════════════════════ - -# Sub-step 0: Agent name -Screenshot screenshots/output/init-06-identity-name.png -Sleep 300ms - -# Accept default name "Netclaw" -Enter -Sleep 2s - -# Sub-step 1: Communication style -Screenshot screenshots/output/init-06-identity-style.png -Sleep 300ms - -# Select "Concise & casual" (first item) -Enter -Sleep 2s - -# Sub-step 2: User name -Screenshot screenshots/output/init-06-identity-user.png -Sleep 300ms - -# Type a name -Type "Aaron" -Sleep 300ms -Enter -Sleep 2s - -# Sub-step 3: Timezone -Screenshot screenshots/output/init-06-identity-timezone.png -Sleep 300ms - -# Accept default timezone -Enter -Sleep 2s - -# Sub-step 4: Workspaces directory -Screenshot screenshots/output/init-06-identity-workspaces.png -Sleep 300ms - -# Accept default -Enter -Sleep 2s - -# Sub-step 5: Webhook URL (optional) -Screenshot screenshots/output/init-06-identity-webhook.png -Sleep 300ms - -# Skip (leave blank) +Sleep 500ms Enter Sleep 2s -# ═══════════════════════════════════════════════ -# STEP 7: External Skills (conditional - Claude Code detected) -# ═══════════════════════════════════════════════ -Screenshot screenshots/output/init-07-external-skills.png -Sleep 300ms - -# Accept detected skills and continue +# Identity: name, style, your name, timezone Enter -Sleep 2s - -# Sub-step 1: Custom skill directory path (optional) -Screenshot screenshots/output/init-07-custom-skills-path.png -Sleep 300ms - -# Skip (leave blank) +Sleep 1500ms +Screenshot screenshots/output/init-02-identity.png +Sleep 500ms Enter -Sleep 2s - -# ═══════════════════════════════════════════════ -# STEP 8: Skill Feeds -# ═══════════════════════════════════════════════ -Screenshot screenshots/output/init-08-skill-feeds.png -Sleep 300ms - -# Select "No — skip" (second item) -Down -Sleep 300ms +Sleep 1s +Type "alice" Enter -Sleep 2s - -# ═══════════════════════════════════════════════ -# STEP 9: Network Exposure -# ═══════════════════════════════════════════════ -Screenshot screenshots/output/init-09-exposure.png -Sleep 300ms - -# Select "Local" (first item, recommended) +Sleep 1s Enter Sleep 2s -# Sub-step: Inbound Webhooks toggle -Screenshot screenshots/output/init-09-webhooks.png -Sleep 300ms - -# Select "No" (second item) -Down -Sleep 300ms +# Security posture: Personal is highlighted +Screenshot screenshots/output/init-02-security-posture.png +Sleep 500ms Enter -Sleep 2s - -# ═══════════════════════════════════════════════ -# STEP 10: Health Check -# ═══════════════════════════════════════════════ -Screenshot screenshots/output/init-10-health-check.png -Sleep 300ms -# Press Enter to start health check -Enter +# Health check starts on its own and hands over to chat within a couple of seconds +# The finished checklist is on screen for about a second and a half before chat takes over, +# so wait for its last line instead of sleeping a fixed time. +Wait+Screen@60s /Configuration written/ +Sleep 200ms +Screenshot screenshots/output/init-05-health-check.png Sleep 8s - -# Capture final health check results -Screenshot screenshots/output/init-10-health-check-complete.png - -Sleep 2s -Ctrl+C +Ctrl+Q +Sleep 1s diff --git a/screenshots/tapes/mcp-tools.tape b/screenshots/tapes/mcp-tools.tape index 0eebabb..54aed89 100644 --- a/screenshots/tapes/mcp-tools.tape +++ b/screenshots/tapes/mcp-tools.tape @@ -1,45 +1,42 @@ -# netclaw mcp tools — MCP tool permissions TUI -# Shows server list, then tool grid with audience toggling -Output screenshots/output/mcp-tools.gif - +# MCP Permissions TUI (`netclaw mcp permissions`): server list, then the tool grid for the +# demo-utilities server under the Personal, Team and Public audiences, plus the Team "Grant all" prompt. Set Shell "bash" +Set WaitTimeout 60s Set Theme "Dracula" -Set Width 1200 -Set Height 700 Set FontSize 16 +Set Width 1000 +Set Height 560 Set Padding 20 -Set TypingSpeed 50ms +Set TypingSpeed 40ms -Type "netclaw mcp tools" +Type "netclaw mcp permissions" Enter -Sleep 3s - -# Server list showing connected MCP servers +Wait+Screen /demo-utilities/ +Sleep 1s Screenshot screenshots/output/mcp-tools-server-list.png -Sleep 300ms +Sleep 1s -# Select first server to see tool grid Enter Sleep 2s - -# Tool grid — Personal audience (default) Screenshot screenshots/output/mcp-tools-personal.png -Sleep 300ms +Sleep 1s -# Switch to Team audience (right arrow) Right Sleep 1s Screenshot screenshots/output/mcp-tools-team.png -Sleep 300ms +Sleep 1s -# Switch to Public audience (right arrow again) -Right +# E on a server that is not enabled for Team asks before granting anything +Type "e" +Sleep 1s +Screenshot screenshots/output/mcp-tools-confirm.png +Sleep 1s +Type "n" Sleep 1s -Screenshot screenshots/output/mcp-tools-public.png -Sleep 300ms -# Return to server list and exit -Escape +Right Sleep 1s -Escape +Screenshot screenshots/output/mcp-tools-public.png Sleep 1s + +Ctrl+Q diff --git a/screenshots/tapes/model-properties.tape b/screenshots/tapes/model-properties.tape new file mode 100644 index 0000000..ea74e9a --- /dev/null +++ b/screenshots/tapes/model-properties.tape @@ -0,0 +1,35 @@ +# model-properties.png: press P on Main in `netclaw model` to open the Model properties panel. +# stack: bare +# Same demo install as model.tape. lab-llama gets ThinkingDialect ReasoningEffort so the thinking +# level field is editable, as it is for a provider that offers it. +Output screenshots/output/model-properties.gif + +Set Shell "bash" +Set Theme "Dracula" +Set FontSize 16 +Set Width 1000 +Set Height 460 +Set Padding 20 +Set TypingSpeed 40ms + +Hide +Type "bash /work/screenshots/scripts/seed-demo-install.sh" +Enter +Wait+Screen@8m /seeded: demo install/ +Type `python3 -c "import json,os;p=os.path.expanduser('~/.netclaw/config/netclaw.json');c=json.load(open(p));c['Providers']['lab-llama'].setdefault('VendorOptions',{})['ThinkingDialect']='ReasoningEffort';json.dump(c,open(p,'w'),indent=2)"` +Enter +Sleep 8s +Type "clear" +Enter +Sleep 1s +Show + +Type "netclaw model" +Enter +Sleep 4s +Type "p" +Sleep 5s +Screenshot screenshots/output/model-properties.png +Sleep 500ms +Ctrl+Q +Sleep 1s diff --git a/screenshots/tapes/model.tape b/screenshots/tapes/model.tape index bfd14a9..79c79e7 100644 --- a/screenshots/tapes/model.tape +++ b/screenshots/tapes/model.tape @@ -1,20 +1,32 @@ -# netclaw model — model management TUI +# netclaw model: the model manager with Main, Fallback and Compaction all assigned +# stack: bare +# Same three neutral providers as provider.tape (scripts/seed-demo-install.sh). The Provider column is capped +# at 40 characters, so the long OpenAI-compatible label ends in an ellipsis at any width. +# +# Produces model-manager. Output screenshots/output/model.gif Set Shell "bash" Set Theme "Dracula" -Set Width 1200 -Set Height 700 Set FontSize 16 +Set Width 1000 +Set Height 340 Set Padding 20 -Set TypingSpeed 50ms +Set TypingSpeed 40ms -Type "netclaw model" +Hide +Type "bash /work/screenshots/scripts/seed-demo-install.sh" +Enter +Wait+Screen@8m /seeded: demo install/ +Type "clear" Enter -Sleep 3s +Sleep 1s +Show +Type "netclaw model" +Enter +Sleep 4s Screenshot screenshots/output/model-manager.png Sleep 500ms - -Escape +Ctrl+Q Sleep 1s diff --git a/screenshots/tapes/provider.tape b/screenshots/tapes/provider.tape index 9cdbaa7..13859ce 100644 --- a/screenshots/tapes/provider.tape +++ b/screenshots/tapes/provider.tape @@ -1,20 +1,63 @@ -# netclaw provider — provider management TUI +# netclaw provider: the provider manager, and its "add a provider" flow for OpenAI-compatible +# stack: bare +# Three providers with neutral names (scripts/seed-demo-install.sh). my-deepseek has a dummy key, so +# it shows the warning glyph. lab-llama is answered by the stack's forwarder when +# INFERENCE_UPSTREAM is set. The Provider column is capped at 40 characters (in builds after +# 0.27.1), so the long OpenAI-compatible label ends in an ellipsis at any width. +# +# Produces provider-manager and provider-add-openai-compatible. Output screenshots/output/provider.gif Set Shell "bash" Set Theme "Dracula" -Set Width 1200 -Set Height 700 Set FontSize 16 +Set Width 1100 +Set Height 500 Set Padding 20 -Set TypingSpeed 50ms +Set TypingSpeed 40ms -Type "netclaw provider" +Hide +Type "bash /work/screenshots/scripts/seed-demo-install.sh" +Enter +Wait+Screen@8m /seeded: demo install/ +Type "clear" Enter -Sleep 3s +Sleep 1s +Show +Type "netclaw provider" +Enter +# Wait for the health probes to settle +Sleep 8s +Up +Up +Sleep 500ms Screenshot screenshots/output/provider-manager.png Sleep 500ms -Escape +# + Add new provider... > OpenAI-compatible > name > the Endpoint step +Down +Down +Down +Down +Down +Down +Down +Down +Sleep 500ms +Enter +Sleep 2s +Down +Down +Down +Down +Down +Sleep 500ms +Enter +Sleep 2s +Enter +Sleep 2s +Screenshot screenshots/output/provider-add-openai-compatible.png +Sleep 500ms +Ctrl+Q Sleep 1s diff --git a/screenshots/tapes/reminder-list.tape b/screenshots/tapes/reminder-list.tape new file mode 100644 index 0000000..2986021 --- /dev/null +++ b/screenshots/tapes/reminder-list.tape @@ -0,0 +1,26 @@ +# reminder-list.png: `netclaw reminder list` over two neutral reminders. +# `reminder list` prints a table with one row per reminder; 1350 px keeps each row on one line. +# stack: bare +Set Shell "bash" +Set WaitTimeout 120s +Set Theme "Dracula" +Set FontSize 16 +Set Width 1350 +Set Height 260 +Set Padding 20 +Set TypingSpeed 40ms + +Hide +Type "bash /work/screenshots/scripts/seed-demo-install.sh" +Enter +Wait+Screen@8m /seeded: demo install/ +Type "clear" +Enter +Sleep 1s +Show + +Type "netclaw reminder list" +Enter +Sleep 2s +Screenshot screenshots/output/reminder-list.png +Sleep 2s diff --git a/screenshots/tapes/reminder-run.tape b/screenshots/tapes/reminder-run.tape new file mode 100644 index 0000000..5c5c5e6 --- /dev/null +++ b/screenshots/tapes/reminder-run.tape @@ -0,0 +1,34 @@ +# reminder-run-approval.png: `netclaw reminder run disk-note` stopped at the approval prompt +# for the reminder's shell call, with "Always here" highlighted. Nothing is answered. +Set Shell "bash" +Set WaitTimeout 240s +Set Theme "Dracula" +Set FontSize 16 +Set Width 1000 +Set Height 700 +Set Padding 20 +Set TypingSpeed 40ms + +Hide +Type "netclaw reminder create disk-note cron '0 8 * * *' 'Use shell_execute to run exactly: touch /tmp/disk-note.txt' --name 'Disk note' > /dev/null; clear" +Enter +Sleep 2s +Show + +Type "netclaw reminder run disk-note" +Enter +Sleep 5s +Wait+Screen@240s /Ready +\| +demo-model/ +Sleep 1s +Type "Run the reminder disk-note now." +Enter +Wait+Screen@240s /5\. Deny/ +Sleep 1s +Down +Sleep 300ms +Down +Sleep 1s +Screenshot screenshots/output/reminder-run-approval.png +Sleep 2s +Ctrl+Q +Sleep 1s diff --git a/screenshots/tapes/reminder.tape b/screenshots/tapes/reminder.tape index c8e9108..bde2cd7 100644 --- a/screenshots/tapes/reminder.tape +++ b/screenshots/tapes/reminder.tape @@ -1,16 +1,46 @@ -# netclaw reminder list — scheduled reminders +# reminder.gif: the Reminder Builder from the first step to Create, about 15 seconds. +# reminder-builder-review.png: the review step, taken just before Create. +# The recording starts inside the builder (the launch is hidden) and ends on the result. Output screenshots/output/reminder.gif Set Shell "bash" +Set WaitTimeout 60s Set Theme "Dracula" -Set Width 1200 -Set Height 600 Set FontSize 16 +Set Width 1000 +Set Height 440 Set Padding 20 -Set TypingSpeed 50ms +Set TypingSpeed 45ms -Type "netclaw reminder list" +Hide +Type "netclaw reminder ui" Enter -Sleep 3s +Wait+Screen /Step 1 of 6/ +Show -Screenshot screenshots/output/reminder-list.png +Sleep 300ms +Type "daily-uptime-check" +Sleep 600ms +Enter +Sleep 1200ms +Down +Sleep 400ms +Down +Sleep 600ms +Enter +Sleep 1200ms +Type "0 9 * * *" +Sleep 600ms +Enter +Sleep 1200ms +Type "Check that the website and API respond." +Sleep 600ms +Enter +Sleep 1200ms +Enter +Wait+Screen /Step 6 of 6/ +Sleep 1500ms +Screenshot screenshots/output/reminder-builder-review.png +Sleep 2s +Enter +Sleep 2500ms diff --git a/screenshots/tapes/sessions.tape b/screenshots/tapes/sessions.tape new file mode 100644 index 0000000..33bf2f3 --- /dev/null +++ b/screenshots/tapes/sessions.tape @@ -0,0 +1,79 @@ +# netclaw sessions: the sessions browser with three titled conversations +# stack: bare +# Needs real inference: set INFERENCE_UPSTREAM (see compose/forwarder.yml), because the stack's +# 0.5B Ollama model can't use tools. seed-demo-install.sh builds the shared demo install, then the tape holds +# three short chats out of sight (each one a session), which takes a few minutes. +# +# Produces sessions-tui. Same three chats as stats.tape, in a shorter window. +Output screenshots/output/sessions.gif + +Set Shell "bash" +Set Theme "Dracula" +Set FontSize 16 +Set Width 1000 +Set Height 320 +Set Padding 20 +Set TypingSpeed 40ms + +Hide +Type "bash /work/screenshots/scripts/seed-demo-install.sh" +Enter +Wait+Screen@8m /seeded: demo install/ +Type "clear" +Enter +Sleep 1s + +# Conversation demo-notes: an interactive chat, so the daemon has time to title the session +Type "netclaw chat" +Enter +Sleep 6s +Type "Give me a three-item checklist for tidying a desk." +Enter +Sleep 6s +Wait+Screen@4m /Ready/ +Sleep 30s +Ctrl+Q +Sleep 2s +Type "clear" +Enter +Sleep 1s + +# Conversation weekly-report: an interactive chat, so the daemon has time to title the session +Type "netclaw chat" +Enter +Sleep 6s +Type "Write a two-sentence status update: the migration is finished and docs work starts Monday." +Enter +Sleep 6s +Wait+Screen@4m /Ready/ +Sleep 30s +Ctrl+Q +Sleep 2s +Type "clear" +Enter +Sleep 1s + +# Conversation trip-planning: an interactive chat, so the daemon has time to title the session +Type "netclaw chat" +Enter +Sleep 6s +Type "Suggest a name for a small weekend hiking club." +Enter +Sleep 6s +Wait+Screen@4m /Ready/ +Sleep 30s +Ctrl+Q +Sleep 2s +Type "clear" +Enter +Sleep 1s + +Show + +Type "netclaw sessions" +Enter +Sleep 3s +Screenshot screenshots/output/sessions-tui.png +Sleep 500ms +Ctrl+Q +Sleep 1s diff --git a/screenshots/tapes/single-shot-cli-execution.tape b/screenshots/tapes/single-shot-cli-execution.tape new file mode 100644 index 0000000..ff63ff5 --- /dev/null +++ b/screenshots/tapes/single-shot-cli-execution.tape @@ -0,0 +1,31 @@ +# netclaw chat -p: a one-shot headless prompt that calls a tool, with usage lines +# stack: bare +# Needs real inference (INFERENCE_UPSTREAM, see compose/forwarder.yml): the tool calls come from +# the model. The prompt is benign and the answer comes from the stack's demo MCP server. The tape +# waits for the shell prompt to come back, because how long the model takes isn't fixed. +Output screenshots/output/single-shot-cli-execution.gif + +Set Shell "bash" +Set Theme "Dracula" +Set FontSize 16 +Set Width 1000 +Set Height 420 +Set Padding 20 +Set TypingSpeed 40ms + +Hide +Type "bash /work/screenshots/scripts/seed-demo-install.sh" +Enter +Wait+Screen@8m /seeded: demo install/ +Type "clear" +Enter +Sleep 1s +Show + +Type `netclaw chat -p "Search your tools for a UUID generator and answer in under fifteen words."` +Enter +Sleep 10s +Wait+Screen@5m /(?m)^> *$/ +Sleep 1s +Screenshot screenshots/output/single-shot-cli-execution.png +Sleep 500ms diff --git a/screenshots/tapes/skill-list.tape b/screenshots/tapes/skill-list.tape new file mode 100644 index 0000000..cd66e1f --- /dev/null +++ b/screenshots/tapes/skill-list.tape @@ -0,0 +1,25 @@ +# skill-list.png: `netclaw skill list` with local skills, the system skills and two feed skills. +# stack: bare +Set Shell "bash" +Set WaitTimeout 120s +Set Theme "Dracula" +Set FontSize 16 +Set Width 1000 +Set Height 520 +Set Padding 20 +Set TypingSpeed 40ms + +Hide +Type "bash /work/screenshots/scripts/seed-demo-install.sh" +Enter +Wait+Screen@8m /seeded: demo install/ +Type "clear" +Enter +Sleep 1s +Show + +Type "netclaw skill list" +Enter +Sleep 3s +Screenshot screenshots/output/skill-list.png +Sleep 2s diff --git a/screenshots/tapes/skill-sync.tape b/screenshots/tapes/skill-sync.tape new file mode 100644 index 0000000..47ac038 --- /dev/null +++ b/screenshots/tapes/skill-sync.tape @@ -0,0 +1,26 @@ +# skill-sync.png: `netclaw skill sync` against the stack's skill server (feed `corp-skills`). +# stack: bare +# The first sync runs hidden so the recorded one reports the steady state. +Set Shell "bash" +Set WaitTimeout 120s +Set Theme "Dracula" +Set FontSize 16 +Set Width 1000 +Set Height 300 +Set Padding 20 +Set TypingSpeed 40ms + +Hide +Type "bash /work/screenshots/scripts/seed-demo-install.sh" +Enter +Wait+Screen@8m /seeded: demo install/ +Type "clear" +Enter +Sleep 1s +Show + +Type "netclaw skill sync" +Enter +Sleep 6s +Screenshot screenshots/output/skill-sync.png +Sleep 2s diff --git a/screenshots/tapes/stats.tape b/screenshots/tapes/stats.tape index 7d34595..fb73bc4 100644 --- a/screenshots/tapes/stats.tape +++ b/screenshots/tapes/stats.tape @@ -1,16 +1,78 @@ -# netclaw stats — usage statistics +# netclaw stats, after a handful of real conversations +# stack: bare +# Needs real inference: set INFERENCE_UPSTREAM (see compose/forwarder.yml), because the stack's +# 0.5B Ollama model can't use tools. seed-demo-install.sh builds the shared demo install, then the tape holds +# three short chats out of sight (each one a session), which takes a few minutes. +# +# Produces stats. sessions.tape holds the same conversations and photographs the sessions browser. Output screenshots/output/stats.gif Set Shell "bash" Set Theme "Dracula" -Set Width 1200 -Set Height 600 Set FontSize 16 +Set Width 1000 +Set Height 640 Set Padding 20 -Set TypingSpeed 50ms +Set TypingSpeed 40ms + +Hide +Type "bash /work/screenshots/scripts/seed-demo-install.sh" +Enter +Wait+Screen@8m /seeded: demo install/ +Type "clear" +Enter +Sleep 1s + +# Conversation demo-notes: an interactive chat, so the daemon has time to title the session +Type "netclaw chat" +Enter +Sleep 6s +Type "Give me a three-item checklist for tidying a desk." +Enter +Sleep 6s +Wait+Screen@4m /Ready/ +Sleep 30s +Ctrl+Q +Sleep 2s +Type "clear" +Enter +Sleep 1s + +# Conversation weekly-report: an interactive chat, so the daemon has time to title the session +Type "netclaw chat" +Enter +Sleep 6s +Type "Write a two-sentence status update: the migration is finished and docs work starts Monday." +Enter +Sleep 6s +Wait+Screen@4m /Ready/ +Sleep 30s +Ctrl+Q +Sleep 2s +Type "clear" +Enter +Sleep 1s + +# Conversation trip-planning: an interactive chat, so the daemon has time to title the session +Type "netclaw chat" +Enter +Sleep 6s +Type "Suggest a name for a small weekend hiking club." +Enter +Sleep 6s +Wait+Screen@4m /Ready/ +Sleep 30s +Ctrl+Q +Sleep 2s +Type "clear" +Enter +Sleep 1s + +Show Type "netclaw stats" Enter Sleep 3s - Screenshot screenshots/output/stats.png +Sleep 500ms + diff --git a/screenshots/tapes/status.tape b/screenshots/tapes/status.tape index 486421a..7bdbd65 100644 --- a/screenshots/tapes/status.tape +++ b/screenshots/tapes/status.tape @@ -1,16 +1,31 @@ -# netclaw status — daemon health and connector status +# netclaw status: a healthy daemon with its connectors, one of them an MCP server +# stack: bare +# seed-demo-install.sh builds the one demo install every status-like capture shares (providers, +# model roles, the demo MCP server, skills, webhooks, reminders), written with the real CLI. The model endpoint is the stack's forwarder +# (INFERENCE_UPSTREAM) when set. Nothing in the output names it. Output screenshots/output/status.gif Set Shell "bash" Set Theme "Dracula" -Set Width 1200 -Set Height 600 Set FontSize 16 +Set Width 1000 +Set Height 600 Set Padding 20 -Set TypingSpeed 50ms +Set TypingSpeed 40ms + +Hide +Type "bash /work/screenshots/scripts/seed-demo-install.sh" +Enter +Wait+Screen@8m /seeded: demo install/ +# Let the daemon run for a minute or two so the uptime reads like a real one +Sleep 75s +Type "clear" +Enter +Sleep 1s +Show Type "netclaw status" Enter Sleep 3s - Screenshot screenshots/output/status.png +Sleep 500ms diff --git a/screenshots/tapes/update-check.tape b/screenshots/tapes/update-check.tape new file mode 100644 index 0000000..6232986 --- /dev/null +++ b/screenshots/tapes/update-check.tape @@ -0,0 +1,24 @@ +# netclaw update --check on an install that is one release behind +# stack: fresh +# image: 0.28.1 +# This is the one tape that does not run the documented release: it runs the previous one (0.28.1), +# which is genuinely a version behind, so the check reports a real update to v0.28.2 instead of a +# staged one. The visible `netclaw --version` makes that plain. +Output screenshots/output/update-check.gif + +Set Shell "bash" +Set Theme "Dracula" +Set FontSize 16 +Set Width 1000 +Set Height 200 +Set Padding 20 +Set TypingSpeed 40ms + +Type "netclaw --version" +Enter +Sleep 2s +Type "netclaw update --check" +Enter +Sleep 5s +Screenshot screenshots/output/update-check.png +Sleep 500ms diff --git a/screenshots/tapes/webhooks.tape b/screenshots/tapes/webhooks.tape index dd011b8..ccb2bd4 100644 --- a/screenshots/tapes/webhooks.tape +++ b/screenshots/tapes/webhooks.tape @@ -1,16 +1,25 @@ -# netclaw webhooks list — inbound webhook routes -Output screenshots/output/webhooks.gif - +# webhooks-list.png: `netclaw webhooks list` with two routes (see scripts/seed-webhooks.sh). +# stack: bare Set Shell "bash" +Set WaitTimeout 120s Set Theme "Dracula" -Set Width 1200 -Set Height 600 Set FontSize 16 +Set Width 1000 +Set Height 300 Set Padding 20 -Set TypingSpeed 50ms +Set TypingSpeed 40ms -Type "netclaw webhooks list" +Hide +Type "bash /work/screenshots/scripts/seed-demo-install.sh" +Enter +Wait+Screen@8m /seeded: demo install/ +Type "clear" Enter -Sleep 3s +Sleep 1s +Show +Type "netclaw webhooks list" +Enter +Sleep 2s Screenshot screenshots/output/webhooks-list.png +Sleep 2s diff --git a/scripts/check-links.mjs b/scripts/check-links.mjs new file mode 100644 index 0000000..d1116c6 --- /dev/null +++ b/scripts/check-links.mjs @@ -0,0 +1,75 @@ +// Crawls dist/ and verifies every internal link, image and #anchor resolves. +// Complements the external link-validator in CI (which does not check anchors). +// Usage: npm run check-links (after npm run build) +import { readFileSync, readdirSync, statSync, existsSync } from 'node:fs'; +import { join, dirname, resolve, relative } from 'node:path'; +import { fileURLToPath } from 'node:url'; + +const ROOT = join(dirname(fileURLToPath(import.meta.url)), '..'); +const DIST = join(ROOT, 'dist'); +if (!existsSync(DIST)) { + console.error('dist/ not found. Run `npm run build` first.'); + process.exit(2); +} + +const walk = (dir) => + readdirSync(dir).flatMap((n) => { + const p = join(dir, n); + return statSync(p).isDirectory() ? walk(p) : [p]; + }); + +const pages = walk(DIST).filter((f) => f.endsWith('.html')); +const ids = new Map(); // abs html path -> Set of ids +const decode = (s) => s.replace(/&/g, '&').replace(/"/g, '"').replace(/'/g, "'"); + +for (const f of pages) { + const html = readFileSync(f, 'utf8'); + const set = new Set(); + for (const m of html.matchAll(/\s(?:id|name)=("([^"]*)"|'([^']*)'|([^\s>]+))/g)) { + set.add(decode(m[2] ?? m[3] ?? m[4])); + } + ids.set(f, set); +} + +function resolveTarget(pathname, fromFile) { + let p = pathname.startsWith('/') ? join(DIST, pathname) : resolve(dirname(fromFile), pathname); + try { + p = decodeURIComponent(p); + } catch {} + return [p, join(p, 'index.html'), p + '.html'].find((c) => existsSync(c) && statSync(c).isFile()); +} + +const problems = []; +let checked = 0; +for (const f of pages) { + const html = readFileSync(f, 'utf8'); + const rel = '/' + relative(DIST, f).replace(/index\.html$/, ''); + for (const m of html.matchAll(/<(a|img|source)\s[^>]*?(href|src)=("([^"]*)"|'([^']*)')/g)) { + const raw = decode(m[4] ?? m[5]); + if (!raw || /^(https?:|mailto:|tel:|data:|javascript:|\/\/)/.test(raw)) continue; + checked++; + const [beforeHash, hash] = raw.split('#'); + const [pathname] = beforeHash.split('?'); + const target = pathname === '' ? f : resolveTarget(pathname, f); + if (!target) { + problems.push(`${rel} missing target: ${raw}`); + continue; + } + if (hash && target.endsWith('.html')) { + let h = hash; + try { + h = decodeURIComponent(hash); + } catch {} + if (!ids.get(target)?.has(h)) problems.push(`${rel} missing anchor: ${raw}`); + } + } +} + +const unique = [...new Set(problems)]; +console.log(`Checked ${checked} internal references across ${pages.length} pages.`); +if (unique.length) { + console.error(`\n${unique.length} broken:`); + for (const p of unique) console.error(' ' + p); + process.exit(1); +} +console.log('No broken internal links or anchors.'); diff --git a/scripts/check-rendered-pages.mjs b/scripts/check-rendered-pages.mjs new file mode 100644 index 0000000..88e2e9b --- /dev/null +++ b/scripts/check-rendered-pages.mjs @@ -0,0 +1,81 @@ +// Post-build guard. Fails the build when a built docs page is empty or missing diagrams. +// +// Why: Mermaid fences are rendered at build time through a headless browser +// (rehype-mermaid, inline-svg). When the browser is not installed the build still +// succeeds, but every page with a diagram ships with an EMPTY body. This script +// catches that, and any other page whose main content collapsed. +// +// Checks, for every page in src/content/docs: +// 1. dist//index.html exists +// 2. main content has at least MIN_WORDS words of prose (SVG, scripts and tags stripped) +// 3. each ```mermaid fence in the source has a rendered +// +// Usage: node scripts/check-rendered-pages.mjs [--verbose] +import { readFileSync, readdirSync, statSync, existsSync } from 'node:fs'; +import { join, dirname, relative } from 'node:path'; +import { fileURLToPath } from 'node:url'; + +const ROOT = join(dirname(fileURLToPath(import.meta.url)), '..'); +const DOCS = join(ROOT, 'src', 'content', 'docs'); +const DIST = join(ROOT, 'dist'); +const MIN_WORDS = 150; +const verbose = process.argv.includes('--verbose'); + +const walk = (dir) => + readdirSync(dir).flatMap((n) => { + const p = join(dir, n); + return statSync(p).isDirectory() ? walk(p) : [p]; + }); + +if (!existsSync(DIST)) { + console.error('dist/ not found. Run `astro build` first.'); + process.exit(2); +} + +const failures = []; +const rows = []; + +for (const file of walk(DOCS).filter((f) => /\.mdx?$/.test(f))) { + const slug = relative(DOCS, file).replace(/\.mdx?$/, '').replace(/(^|\/)index$/, ''); + const src = readFileSync(file, 'utf8'); + const fences = (src.match(/^\s*```mermaid\b/gm) || []).length; + const out = join(DIST, slug, 'index.html'); + if (!existsSync(out)) { + failures.push(`${slug}: no built page at dist/${slug}/index.html`); + continue; + } + const html = readFileSync(out, 'utf8'); + const start = html.indexOf('class="sl-markdown-content"'); + const end = html.indexOf('', start); + const main = start === -1 ? '' : html.slice(start, end === -1 ? undefined : end); + const text = main + .replace(//g, ' ') + .replace(/<(script|style)[\s\S]*?<\/\1>/g, ' ') + .replace(/<[^>]+>/g, ' '); + const words = (text.match(/\S+/g) || []).length; + const svgs = (main.match(/]*\bid="mermaid-/g) || []).length; + + rows.push({ slug, fences, svgs, words }); + if (start === -1) failures.push(`${slug}: main content container not found`); + else if (words < MIN_WORDS) failures.push(`${slug}: main content has only ${words} words (minimum ${MIN_WORDS})`); + if (svgs < fences) failures.push(`${slug}: ${fences} mermaid fence(s) but ${svgs} rendered SVG(s)`); +} + +if (verbose) { + for (const r of rows.filter((r) => r.fences > 0)) { + console.log(` diagram page ${r.slug}: ${r.svgs}/${r.fences} rendered, ${r.words} words`); + } +} + +const diagramPages = rows.filter((r) => r.fences > 0).length; +if (failures.length) { + console.error(`\ncheck-rendered-pages: ${failures.length} problem(s)`); + for (const f of failures) console.error(' ' + f); + console.error( + '\nIf diagrams are missing, the headless browser is probably not installed: npx playwright install chromium-headless-shell. Astro caches rendered markdown, so after fixing it delete node_modules/.astro and rebuild.', + ); + process.exit(1); +} +console.log( + `check-rendered-pages: ${rows.length} pages OK (${diagramPages} with diagrams, min ${Math.min(...rows.map((r) => r.words))} words)`, +); diff --git a/scripts/config-reference-overrides.json b/scripts/config-reference-overrides.json new file mode 100644 index 0000000..92de1b4 --- /dev/null +++ b/scripts/config-reference-overrides.json @@ -0,0 +1,890 @@ +{ + "_readme": [ + "Hand-maintained inputs for scripts/gen-config-reference.mjs. The JSON schema in the product repo is the structural source of truth (keys, types, enums, min/max).", + "The schema omits many defaults and descriptions, so this file supplies them. Every value here was read from the C# options classes of the release named in _verifiedAgainst.", + "Each entry has a 'source' that names the file to re-read at the next release. When a default in C# changes, edit it here and re-run the script.", + "keys: overrides for keys that exist in the schema (default, description, type, allowed). An override for a key that has left the schema fails the run.", + "shared: overrides applied to each prefix in 'prefixes' (the three chat channels). An explicit entry in 'keys' wins.", + "extra: keys the daemon reads that the schema does not list (chat channel secrets, MCP client secret). 'after' pins the row position.", + "sections: one-paragraph orientation per top-level key, rendered above each table. 'notes' renders below it.", + "skip: schema paths that are not rendered as rows." + ], + "_verifiedAgainst": "netclaw 0.28.2 release source (version string 0.28.2)", + "mapKeyNames": { + "Models.Definitions": "name", + "McpServers": "server", + "Providers": "name" + }, + "shared": { + "prefixes": [ + "Slack", + "Discord", + "Mattermost" + ], + "keys": { + "Enabled": { + "default": "`false`", + "description": "Start this channel adapter. Needs the bot token in `secrets.json`.", + "source": "Netclaw.Channels./ChannelOptions.cs" + }, + "DefaultChannelId": { + "default": "unset", + "description": "Default channel for messages the bot starts itself, such as reminder deliveries. It also counts as an allowed channel.", + "source": "SlackChannelOptions.cs, SlackAclPolicy.IsAllowedChannel, ChannelIntegrationRegistrationExtensions.cs" + }, + "MentionOnly": { + "default": "`true`", + "description": "Respond in channels only when the bot is mentioned.", + "source": "SlackChannelOptions.cs, DiscordChannelOptions.cs, MattermostChannelOptions.cs" + }, + "AllowDirectMessages": { + "default": "`false`", + "description": "Accept direct messages. DMs use the audience mapped to the `dm` key in `ChannelAudiences`.", + "source": "SlackChannelOptions.cs, SlackAclPolicy.EvaluateInbound" + }, + "MentionRequiredInDm": { + "default": "`false`", + "description": "Require a mention in DMs too. Applies only when `AllowDirectMessages` is `true`.", + "source": "SlackChannelOptions.cs" + }, + "MentionRequiredInThreadByChannel": { + "default": "`{}`", + "description": "Per-channel switch, keyed by channel ID. `true` requires a mention in thread replies even when the thread already has an active session. A channel with no entry doesn't.", + "source": "SlackChannelOptions.MentionRequiredInThreadFor" + }, + "AllowedChannelIds": { + "default": "`[]`", + "description": "Channel IDs the bot responds in, in addition to `DefaultChannelId`. Messages from any other channel are denied.", + "source": "SlackAclPolicy.IsAllowedChannel" + }, + "AllowedUserIds": { + "default": "`[]`", + "description": "User IDs allowed to talk to the bot. An empty list allows every user in an allowed channel. A listed user is treated as a trusted internal principal.", + "source": "SlackAclPolicy.IsAllowedUser (empty list allows all)" + }, + "ChannelAudiences": { + "default": "`{}`", + "description": "Audience per channel, keyed by channel ID or `dm`. Unmapped channels fall back to the ACL heuristic. The audience picks the `Tools.AudienceProfiles` entry that applies.", + "source": "SlackChannelOptions.cs" + } + } + }, + "keys": { + "Slack.SocketMode": { + "default": "`true`", + "description": "Connect over Slack Socket Mode. Needs the app-level token (`Slack.AppToken`).", + "source": "SlackChannelOptions.cs" + }, + "Slack.DefaultChannelName": { + "default": "unset", + "description": "A channel name the daemon resolves to an ID at startup when `DefaultChannelId` is unset. An unresolved name logs a warning. The `netclaw config` Slack editor deletes it and writes the ID.", + "source": "SlackChannelOptions.cs" + }, + "Logging.LogLevel.Default": { + "default": "`\"Information\"`", + "description": "Minimum level written to `daemon-.log`. An unset or unrecognized value gives `Information`. Other keys under `Logging.LogLevel` (for example `Microsoft`) set per-category levels.", + "source": "Netclaw.Daemon/Configuration/LoggingRegistrationExtensions.ResolveLogLevel" + }, + "Logging.Console.Enabled": { + "default": "`false`", + "description": "Also write single-line log output to the console. Docker and systemd capture it.", + "source": "LoggingRegistrationExtensions.ConfigureNetclawLogging" + }, + "Telemetry.Enabled": { + "default": "`false`", + "description": "Export metrics and logs over OTLP. netclaw has no tracing exporter.", + "source": "Netclaw.Daemon/Configuration/TelemetryOptions.cs" + }, + "Telemetry.Otlp.Endpoint": { + "default": "`\"http://127.0.0.1:4317\"`", + "description": "OTLP collector base URL. Must be an absolute URI. Over HTTP/protobuf netclaw posts to `v1/logs` and `v1/metrics` under it, and reduces a full signal URL to its base. gRPC uses the URL unchanged.", + "source": "TelemetryOptions.cs, TelemetryOptionsValidator.cs" + }, + "Session.MemoryObserverIdleSeconds": { + "default": "`90`", + "description": "Seconds of quiet before the memory observer distills the conversation into memories.", + "source": "SessionConfig.cs" + }, + "Session.IdleTimeout": { + "description": "How long a session sits idle before it saves a snapshot and passivates. Format `HH:mm:ss`. `00:00:00` disables passivation. Background jobs owned by the session are killed when it passivates.", + "source": "SessionConfig.cs", + "default": "`\"01:00:00\"`" + }, + "Session.TurnLlmTimeoutSeconds": { + "default": "`180`", + "description": "Timeout for the main per-turn model call. Also feeds the `FirstTokenTimeoutSeconds` fallback.", + "source": "SessionConfig.RawSessionConfig" + }, + "Session.ToolExecutionTimeoutSeconds": { + "default": "`90`", + "description": "Inactivity budget per tool call. The call must produce its first result within this time and each later streamed item resets it. A stuck tool fails alone without affecting the others in the batch. Background shell jobs aren't bound by it.", + "source": "SessionConfig.RawSessionConfig, docs/runbooks/background-jobs.md" + }, + "Session.SidecarLlmTimeoutSeconds": { + "default": "`90`", + "description": "Timeout for background model calls: title generation, memory distillation and compaction notes. Raise it for slow models.", + "source": "SessionConfig.cs" + }, + "Memory.Curation.NominatorSimilarityThreshold": { + "default": "`0.86`", + "description": "Embedding cosine similarity threshold above which an existing memory is nominated for the curation call to adjudicate.", + "source": "Netclaw.Configuration/Schemas/netclaw-config.v1.schema.json" + }, + "Session.FirstTokenTimeoutSeconds": { + "default": "`600`", + "description": "Maximum silence between streamed deltas once the first one arrives. Subagent runs use it too, since agent definitions no longer carry `timeoutSeconds`. If unset, it uses `TurnLlmTimeoutSeconds` when you changed that, otherwise 600.", + "source": "SessionConfig.BindFromConfiguration" + }, + "Session.PrefillTimeoutSeconds": { + "default": "`1800`", + "description": "Maximum wait for the first streamed delta. Covers queue wait and prompt prefill, which can take minutes on self-hosted backends.", + "source": "SessionConfig.BindFromConfiguration" + }, + "Session.NoProgressTimeoutSeconds": { + "default": "`1200`", + "description": "Hard ceiling on time without a real output token. Keepalive heartbeats don't reset it, so a backend that heartbeats forever without producing a token is killed when it elapses. The value also sets the thinking cap: one model call that streams only thinking, with no answer text and no tool call, stops at three times this value.", + "source": "SessionConfig.BindFromConfiguration" + }, + "Session.MaxToolIterationsPerTurn": { + "default": "`60`", + "description": "Legacy key kept so existing files still load. It is validated (1 to 500) but no longer limits a turn. netclaw refuses a call that repeats the same work, and the watchdog timeouts bound the rest.", + "source": "SessionConfig.cs" + }, + "Session.CompactionThreshold": { + "default": "`0.75`", + "description": "Fraction of the context window that triggers compaction. Same as `Session.Tuning.CompactionThreshold`. The `Tuning` value wins when both are set.", + "source": "SessionTuning.cs, SessionConfig.BindTuning" + }, + "Session.SnapshotInterval": { + "default": "`20`", + "description": "Turns between persistence snapshots. Same as `Session.Tuning.SnapshotInterval`.", + "source": "SessionTuning.cs" + }, + "Session.KeepRecentToolResults": { + "default": "`3`", + "description": "Recent tool call and result pairs kept in full when compaction clears older results. Same as `Session.Tuning.KeepRecentToolResults`.", + "source": "SessionTuning.cs" + }, + "Session.MaxInlineToolResultChars": { + "default": "`12000`", + "description": "Largest tool result placed inline in the conversation. For longer output the last this-many characters stay inline, the rest is dropped from the inline text, and the full output goes to a session file the model reads with `tool_output_read`. `shell_execute` has its own smaller budget of 2,000. Same as `Session.Tuning.MaxInlineToolResultChars`.", + "source": "SessionTuning.cs" + }, + "Session.DiscoveredToolRetentionTurns": { + "default": "`3`", + "description": "User turns a tool loaded with `load_tool` stays available. `0` makes the model load it again every turn. Same as `Session.Tuning.DiscoveredToolRetentionTurns`.", + "source": "SessionTuning.cs" + }, + "Session.DiscoveredToolMaxCount": { + "default": "`12`", + "description": "Most loaded deferred or MCP tools kept across turns. The oldest are evicted first. Same as `Session.Tuning.DiscoveredToolMaxCount`.", + "source": "SessionTuning.cs" + }, + "Session.KeepRecentMessages": { + "default": "`6`", + "description": "Recent messages kept verbatim after compaction summarizes the rest. Counts raw messages, not turn pairs. Same as `Session.Tuning.KeepRecentMessages`.", + "source": "SessionTuning.cs" + }, + "Session.TitleGenerationInterval": { + "default": "`10`", + "description": "Turns between session title refreshes. The title is always generated on turn 1. `0` disables refreshes. Same as `Session.Tuning.TitleGenerationInterval`.", + "source": "SessionTuning.cs" + }, + "Session.DeterministicRetrievalEnabled": { + "default": "`true`", + "description": "Plan automatic memory recall deterministically on each turn. Scheduled for removal, leave it on. Same as `Session.Tuning.DeterministicRetrievalEnabled`.", + "source": "SessionTuning.cs" + }, + "Session.Tuning": { + "default": "-", + "description": "Internal tuning. Every key here has a same-named key directly under `Session`, and the `Tuning` value wins.", + "source": "SessionConfig.BindTuning" + }, + "Session.Tuning.CompactionThreshold": { + "default": "`0.75`", + "description": "See `Session.CompactionThreshold`.", + "source": "SessionTuning.cs" + }, + "Session.Tuning.SnapshotInterval": { + "default": "`20`", + "description": "See `Session.SnapshotInterval`.", + "source": "SessionTuning.cs" + }, + "Session.Tuning.KeepRecentToolResults": { + "default": "`3`", + "description": "See `Session.KeepRecentToolResults`.", + "source": "SessionTuning.cs" + }, + "Session.Tuning.MaxInlineToolResultChars": { + "default": "`12000`", + "description": "See `Session.MaxInlineToolResultChars`.", + "source": "SessionTuning.cs" + }, + "Session.Tuning.DiscoveredToolRetentionTurns": { + "default": "`3`", + "description": "See `Session.DiscoveredToolRetentionTurns`.", + "source": "SessionTuning.cs" + }, + "Session.Tuning.DiscoveredToolMaxCount": { + "default": "`12`", + "description": "See `Session.DiscoveredToolMaxCount`.", + "source": "SessionTuning.cs" + }, + "Session.Tuning.KeepRecentMessages": { + "default": "`6`", + "description": "See `Session.KeepRecentMessages`.", + "source": "SessionTuning.cs" + }, + "Session.Tuning.TitleGenerationInterval": { + "default": "`10`", + "description": "See `Session.TitleGenerationInterval`.", + "source": "SessionTuning.cs" + }, + "Session.Tuning.DeterministicRetrievalEnabled": { + "default": "`true`", + "description": "See `Session.DeterministicRetrievalEnabled`.", + "source": "SessionTuning.cs" + }, + "Session.Tuning.MemoryDistillationTurnInterval": { + "default": "`5`", + "description": "Completed turns between memory distillation passes, regardless of idle state, so memories form during long active sessions. `0` leaves distillation to `MemoryObserverIdleSeconds`.", + "source": "SessionTuning.cs" + }, + "Session.Tuning.MinimumRecallCompositeScore": { + "default": "unset", + "description": "Minimum composite score for a memory to be injected by automatic recall. Unset uses the coordinator's built-in floor. Lower values let weaker matches through, `0` removes the floor.", + "source": "SessionTuning.cs" + }, + "Session.Tuning.MaxRecallInjectedChars": { + "default": "`2000`", + "description": "Character budget for memory text injected by automatic recall in one turn. Whole items are dropped rather than truncated. `0` removes the budget.", + "source": "SessionTuning.cs" + }, + "Security.DeploymentPosture": { + "default": "unset", + "description": "Who talks to this install: `Personal` (you), `Team` or `Public`. `netclaw init` writes it. When it's unset and `StrictDefaults` is `true`, netclaw behaves as `Public` and denies `shell_execute`. It selects the default tool profiles and shell mode, and it narrows the audience of every turn: a channel mapped to `Personal` under a `Team` posture runs as `Team`.", + "source": "TrustContextPolicy.cs SecurityPolicyDefaults.ResolveDeploymentPosture" + }, + "Security.ShellExecutionMode": { + "default": "unset", + "description": "Whether `shell_execute` can run. Unset gives `HostAllowed` for a `Personal` posture and `Off` for the others. `Tools.ShellMode` overrides it. `SandboxOnly` denies every shell call with `shell_requires_sandbox_backend`.", + "source": "SecurityPolicyDefaults.ResolveShellExecutionMode, ToolAccessPolicy.ResolveShellMode" + }, + "Security.StrictDefaults": { + "default": "`true`", + "description": "Fail closed when posture or shell mode is unset. Set `false` only to get `Personal` and `HostAllowed` as the implicit defaults.", + "source": "SecurityPolicyConfig" + }, + "Tools.ShellMode": { + "default": "unset", + "description": "Overrides `Security.ShellExecutionMode` for the shell tools. Unset follows the security setting.", + "source": "ToolConfig.ShellMode, ToolAccessPolicy.ResolveShellMode" + }, + "Tools.MaxOutputChars": { + "default": "`256000`", + "description": "Most output a tool call captures in memory for the spill file. It isn't what the model sees inline, which is `Session.MaxInlineToolResultChars`. Output past the ceiling is discarded.", + "source": "ToolConfig.cs" + }, + "Tools.AudienceProfiles": { + "default": "-", + "description": "Per-audience tool, file and MCP grants. Public, Team and Personal share one shape. A list you set replaces the default list and doesn't merge with it.", + "source": "ToolAudienceProfiles.cs" + }, + "Tools.AudienceProfiles.": { + "default": "-", + "description": "One entry each for `Public`, `Team` and `Personal`.", + "source": "ToolAudienceProfiles.cs" + }, + "Tools.AudienceProfiles..ToolsMode": { + "default": "`Allowlist` (Personal: `All`)", + "description": "`Allowlist` grants only the tools in `AllowedTools`. `All` grants every profile-managed tool.", + "source": "ToolAudienceProfileDefaults.CreatePublic/CreateTeam/CreatePersonal" + }, + "Tools.AudienceProfiles..AllowedTools": { + "default": "Public and Team defaults, see [Built-in tools](/reference/tools/#default-allow-lists)", + "description": "Profile-managed tools this audience can call. Tools outside the profile-managed set are always available. Setting the key replaces the default list, and a value that isn't a list stops startup.", + "source": "ToolAudienceProfileToolCatalog, ToolConfig.BindFromConfiguration" + }, + "Tools.AudienceProfiles..McpServersMode": { + "default": "`Allowlist` (Personal: `All`)", + "description": "`Allowlist` exposes only the servers in `AllowedMcpServers`. `All` exposes every configured server.", + "source": "ToolAudienceProfileDefaults" + }, + "Tools.AudienceProfiles..AllowedMcpServers": { + "default": "`[]`", + "description": "MCP server names this audience can use when `McpServersMode` is `Allowlist`. MCP stays off for Public and Team until you add a server.", + "source": "ToolAudienceProfile" + }, + "Tools.AudienceProfiles..McpServerToolGrants": { + "default": "unset", + "description": "Per-server tool allowlists. A server listed here exposes only the named tools. A server not listed exposes all of its tools. Ignored when `McpServersMode` is `All`.", + "source": "ToolAudienceProfileResolver.IsMcpToolAllowed" + }, + "Tools.AudienceProfiles..ReadFiles.Mode": { + "default": "`Roots` (Personal: `All`)", + "description": "`None` blocks it, `Roots` limits it to `Roots`, `All` allows any path that isn't protected. For the `Personal` audience, `file_read`, `file_list`, `file_search` and `attach_file` reach at least as far as `WriteFiles` does.", + "source": "ToolAudienceProfileDefaults.CreateSessionScopedFilesystemAccess" + }, + "Tools.AudienceProfiles..ReadFiles.Roots": { + "default": "`[\"{session_dir}\"]`", + "description": "Directories used when `Mode` is `Roots`. Tokens: `{session_dir}`, `{skills_dir}`, `{identity_dir}`, `{workspaces_dir}`.", + "source": "ToolAudienceProfileDefaults, ToolAudienceProfileResolver.ResolveToken" + }, + "Tools.AudienceProfiles..WriteFiles.Mode": { + "default": "`Roots` (Personal: `All`)", + "description": "Same values as `ReadFiles.Mode`, for `file_write` and `file_edit`.", + "source": "ToolAudienceProfileDefaults" + }, + "Tools.AudienceProfiles..WriteFiles.Roots": { + "default": "`[\"{session_dir}\"]`", + "description": "Directories used when `Mode` is `Roots`. Same tokens as `ReadFiles.Roots`.", + "source": "ToolAudienceProfileDefaults" + }, + "Tools.AudienceProfiles..AttachFiles.Mode": { + "default": "`Roots` (Personal: `All`)", + "description": "Same values as `ReadFiles.Mode`, for `attach_file`.", + "source": "ToolAudienceProfileDefaults" + }, + "Tools.AudienceProfiles..AttachFiles.Roots": { + "default": "`[\"{session_dir}\"]`", + "description": "Directories used when `Mode` is `Roots`. Same tokens as `ReadFiles.Roots`.", + "source": "ToolAudienceProfileDefaults" + }, + "Tools.AudienceProfiles..ApprovalPolicy": { + "default": "unset (Personal posture: `shell_execute` is `Approval`)", + "description": "Which tools need a human to approve each call. See [How tool approvals work](/security/approvals/).", + "source": "ToolAudienceProfileDefaults.CreateProfilesForPosture" + }, + "Tools.AudienceProfiles..ApprovalPolicy.DefaultMode": { + "default": "`\"Auto\"`", + "description": "Mode for tools with no override.", + "source": "ToolApprovalConfig.cs" + }, + "Tools.AudienceProfiles..ApprovalPolicy.ToolOverrides": { + "default": "`{}`", + "description": "Mode per tool. Keys are tool names (`shell_execute`) or MCP tools as `server/tool` (or `server__tool`), matched without regard to case. If two keys differ only by case, the more restrictive mode wins. A key that contains `:` or has no value is ignored, so the tool keeps the audience default, and `netclaw doctor` reports it. Writes to the config directory are denied before any approval mode is read.", + "source": "ToolApprovalConfig.cs" + }, + "Tools.AudienceProfiles..ApprovalPolicy.McpServerDefaults": { + "default": "`{}`", + "description": "Mode for every tool of one MCP server, keyed by server name and matched without regard to case. An exact `ToolOverrides` entry wins. A key that contains `:` is ignored and `netclaw doctor` reports it.", + "source": "ToolApprovalConfig.TryGetExplicitMode" + }, + "Tools.AudienceProfiles..ChannelAttachments": { + "default": "-", + "description": "Which files from chat messages this audience can send in.", + "source": "ChannelAttachmentPolicy.cs" + }, + "Tools.AudienceProfiles..ChannelAttachments.AllowedCategories": { + "default": "Public: `Image`. Team: `Image`, `Pdf`, `Document`, `Archive`, `Media`. Personal: those plus `Other`.", + "description": "Attachment categories accepted. An empty list rejects every attachment.", + "source": "ToolAudienceProfileDefaults.CreatePublic/Team/PersonalChannelAttachments" + }, + "Tools.AudienceProfiles..ChannelAttachments.MaxFileBytes": { + "default": "`26214400`", + "description": "Largest file accepted, in bytes (25 MiB). Must be above 0 when any category is allowed.", + "source": "ChannelAttachmentPolicy.DefaultMaxFileBytes, ToolAudienceProfiles.ValidateChannelAttachments" + }, + "Tools.AudienceProfiles..ChannelAttachments.MaxFilesPerMessage": { + "default": "`10`", + "description": "Most files accepted on one message. A message over the limit loses all its attachments. Must be above 0 when any category is allowed.", + "source": "ChannelAttachmentPolicy.DefaultMaxFilesPerMessage" + }, + "Tools.AudienceProfiles.GlobalReadRoots": { + "default": "`[\"{skills_dir}\", \"{identity_dir}\", \"{workspaces_dir}\"]`", + "description": "Directories every audience can read, so skill loading, identity files and project discovery work under Team and Public. Setting the key replaces the list.", + "source": "ToolAudienceProfiles.GlobalReadRoots" + }, + "Tools.WebFetch.RequireHttps": { + "default": "`true`", + "description": "Reject plain `http://` URLs in `web_fetch` unless the host is in `HttpAllowList`.", + "source": "WebFetchConfig.cs" + }, + "Tools.WebFetch.HttpAllowList": { + "default": "`[\"localhost\", \"127.0.0.1\", \"::1\", \"[::1]\"]`", + "description": "Hosts that may use plain HTTP when `RequireHttps` is `true`. Setting the key replaces the list, so keep the loopback entries if you still need them.", + "source": "WebFetchConfig.cs, ToolConfig.DefaultedLists" + }, + "Tools.HardDenyPatterns": { + "default": "`[]`", + "description": "Extra shell verb-chain prefixes that are always blocked and can't be approved. They add to the built-in hard-deny list, they don't replace it.", + "source": "ToolConfig.HardDenyPatterns" + }, + "Providers": { + "type": "map of object", + "default": "`{}`", + "description": "LLM provider connections keyed by a name you choose. `Models.Definitions` entries refer to these names.", + "source": "ProviderEntry.cs" + }, + "Models.Definitions": { + "default": "-", + "description": "Model definitions keyed by a name you choose, at least one. Names are case-insensitive and must be unique.", + "source": "NamedModelConfiguration.cs" + }, + "Models.Definitions.": { + "default": "-", + "description": "One model: which provider serves it and how it behaves.", + "source": "ModelReference.cs" + }, + "Models.Definitions..Provider": { + "default": "required", + "description": "Key into `Providers`.", + "source": "ModelReference.cs" + }, + "Models.Definitions..ModelId": { + "default": "required", + "description": "Model identifier sent to the provider.", + "source": "ModelReference.cs" + }, + "Models.Definitions..ContextWindow": { + "default": "detected", + "description": "Context window in tokens. When set, it caps the value the provider reports.", + "source": "ModelReference.cs" + }, + "Models.Definitions..Provenance": { + "default": "unset", + "description": "Where the model data came from: `Live`, `Defaults` or `Manual`. netclaw sets it. You don't need to.", + "source": "ModelReference.cs" + }, + "Models.Definitions..InputModalities": { + "default": "detected", + "description": "Manual override for what the model accepts, comma-separated from `Text`, `Image`, `Audio`, `Video`. Turns off automatic detection.", + "source": "ModelReference.cs", + "allowed": "comma-separated `Text`, `Image`, `Audio`, `Video`" + }, + "Models.Definitions..OutputModalities": { + "default": "detected", + "description": "Manual override for what the model produces. Same format as `InputModalities`.", + "source": "ModelReference.cs", + "allowed": "comma-separated `Text`, `Image`, `Audio`, `Video`" + }, + "Models.Definitions..MaxOutputTokens": { + "default": "unset", + "description": "Output limit in tokens for every call of this model. Thinking tokens count inside it. Must be below `ContextWindow` when that is set. Unset sends 32000, clamped per call to the model maximum and to the context window minus the estimated prompt.", + "source": "ModelReference.cs" + }, + "Models.Definitions..Thinking": { + "default": "unset", + "description": "Provider-neutral thinking level for every call of this model. `Max` is the highest level the provider offers. A level the provider dialect can't send fails startup. Unset uses the model's default.", + "source": "ModelReference.cs" + }, + "Models.Roles": { + "default": "-", + "description": "Which definition plays each role. Values are names from `Definitions`.", + "source": "NamedModelConfiguration.cs" + }, + "Models.Roles.Main": { + "default": "required", + "description": "Model for all interactions.", + "source": "ModelSelection.cs" + }, + "Models.Roles.Fallback": { + "default": "unset", + "description": "Failover model. Falls back to `Main` when unset.", + "source": "ModelSelection.cs" + }, + "Models.Roles.Compaction": { + "default": "unset", + "description": "Cheaper or faster model for compaction. Falls back to `Main` when unset.", + "source": "ModelSelection.cs" + }, + "Memory.Recall.MinCosineSimilarity": { + "default": "unset (follows the model, 0.24 for the default)", + "description": "Cosine floor for automatic recall. Candidates below it are dropped before ranking. The meaning is specific to the active embedding model, so recalibrate before you set it.", + "source": "MemoryConfig.cs MemoryRecallConfig, doctor 'Memory Embeddings' check" + }, + "Memory.Recall.RelevanceGate.Enabled": { + "default": "unset (follows `Memory.Embeddings.Enabled`)", + "description": "Run the cross-encoder relevance gate on recall candidates. Set it explicitly to turn the gate on or off independently of embeddings.", + "source": "MemoryConfig.cs MemoryRelevanceGateConfig" + }, + "Memory.Recall.RelevanceGate.Threshold": { + "default": "unset (model default, 0.02)", + "description": "Score below which the gate drops a candidate. Unset uses the calibrated value for the provisioned relevance model.", + "source": "MemoryConfig.cs, doctor 'Memory Relevance Gate' check" + }, + "ExternalSkills.Sources": { + "default": "`[]`", + "description": "Ordered list of external skill sources. Earlier entries win name collisions and native netclaw skills always win.", + "source": "ExternalSkillsConfig.cs" + }, + "ExternalSkills.Sources[].Name": { + "default": "required", + "description": "Unique name for the source, for example `claude-code`.", + "source": "ExternalSkillsConfig.cs" + }, + "SkillFeeds.Feeds": { + "default": "`[]`", + "description": "Ordered list of skill server feeds. Earlier feeds win name collisions.", + "source": "SkillFeedsConfig.cs" + }, + "SkillFeeds.Feeds[].Name": { + "default": "required", + "description": "Unique feed name, lowercase letters, digits and hyphens. Used as the directory name.", + "source": "SkillFeedsConfig.cs" + }, + "SkillFeeds.Feeds[].Url": { + "default": "required", + "description": "Skill server base URL.", + "source": "SkillFeedsConfig.cs" + }, + "Notifications.Webhooks": { + "default": "`[]`", + "description": "Targets that receive operational alerts by HTTP POST. Empty means alerts only go to the log.", + "source": "NotificationsConfig.cs" + }, + "Notifications.Webhooks[].Url": { + "default": "required", + "description": "HTTP or HTTPS endpoint to POST alerts to.", + "source": "NotificationsConfig.cs WebhookTarget" + }, + "Notifications.Webhooks[].Name": { + "default": "unset", + "description": "Display name used in logs.", + "source": "NotificationsConfig.cs WebhookTarget" + }, + "Notifications.Webhooks[].Headers": { + "default": "unset", + "description": "Static headers sent with every POST, for example `Authorization`.", + "source": "NotificationsConfig.cs WebhookTarget" + }, + "Webhooks.Enabled": { + "default": "`false`", + "description": "Register the inbound webhook endpoint and the `set_webhook`, `list_webhooks` and `delete_webhook` tools.", + "source": "WebhooksConfig.cs, Netclaw.Daemon/Program.cs" + }, + "Webhooks.ExecutionTimeoutSeconds": { + "default": "`300`", + "description": "Longest a webhook-triggered run can take before the daemon marks it failed.", + "source": "WebhooksConfig.cs, WebhookExecutionActor.cs" + }, + "Workspaces.Directory": { + "default": "`~/.netclaw/workspaces`", + "description": "Root for project workspaces. Projects are Git repositories with an `AGENTS.md`, at any depth. A leading `~` expands to your home directory.", + "source": "NetclawPaths.cs" + }, + "McpServers..Transport": { + "default": "`\"stdio\"`", + "description": "How netclaw connects: `stdio` runs `Command`, `sse` and `http` connect to `Url`.", + "source": "McpServerEntry.cs" + }, + "McpServers..Command": { + "default": "unset", + "description": "Executable to run for `stdio`.", + "source": "McpServerEntry.cs" + }, + "McpServers..Arguments": { + "default": "unset", + "description": "Arguments for `Command`.", + "source": "McpServerEntry.cs" + }, + "McpServers..Url": { + "default": "unset", + "description": "Endpoint for `sse` and `http`.", + "source": "McpServerEntry.cs" + }, + "McpServers..EnvironmentVariables": { + "default": "unset", + "description": "Environment variables passed to the `stdio` process. Values stored in `secrets.json` are decrypted first.", + "source": "McpServerEntry.cs" + }, + "McpServers..Headers": { + "default": "unset", + "description": "HTTP headers for `sse` and `http`. Values stored in `secrets.json` are decrypted first.", + "source": "McpServerEntry.cs" + }, + "McpServers..Enabled": { + "default": "`true`", + "description": "Disabled servers are skipped at startup.", + "source": "McpServerEntry.cs" + }, + "McpServers..GrantCategory": { + "default": "`mcp:`", + "description": "Tool metadata category for the server's tools. Authorization doesn't read it: use the audience's `AllowedMcpServers` and `McpServerToolGrants`.", + "source": "McpServerEntry.cs" + }, + "McpServers..OAuthClientId": { + "default": "unset", + "description": "Static OAuth client ID for servers without dynamic client registration.", + "source": "McpServerEntry.cs" + }, + "McpServers..OAuthScope": { + "default": "unset", + "description": "Space-separated OAuth scopes to request, overriding the server's defaults.", + "source": "McpServerEntry.cs" + }, + "Search.BraveApiKey": { + "default": "unset", + "description": "Brave Search API token. Required when `Backend` is `brave`. Lives in `secrets.json`: `netclaw secrets set Search.BraveApiKey `.", + "source": "SearchConfig.cs ([ConfigValue] PersistTo SecretsJson)" + }, + "Search.SearXngEndpoint": { + "default": "unset", + "description": "SearXNG instance base URL, for example `http://searxng.local:8080`. Required when `Backend` is `searxng`.", + "source": "SearchConfig.cs" + }, + "Memory.Embeddings": { + "default": "-", + "description": "Local ONNX embedding model for semantic memory.", + "source": "MemoryConfig.cs MemoryEmbeddingsConfig" + }, + "Memory.Curation": { + "default": "-", + "description": "How new memories are checked against existing ones before they're saved.", + "source": "MemoryConfig.cs MemoryCurationConfig" + }, + "Memory.Recall": { + "default": "-", + "description": "How automatic recall ranks and filters memories, mixing keyword and vector scores.", + "source": "MemoryConfig.cs MemoryRecallConfig" + }, + "Memory.Embeddings.ModelId": { + "default": "`\"snowflake-arctic-embed-m-int8\"`", + "description": "Embedding model to use. Allowed ids are `snowflake-arctic-embed-m-int8`, `snowflake-arctic-embed-m` and `mxbai-embed-large-v1`. An id outside the allowlist is a configuration error. Changing it re-embeds existing memories under the new id.", + "source": "MemoryConfig.cs MemoryEmbeddingsConfig.ModelId" + }, + "SubAgents.PrefillTimeoutSeconds": { + "default": "`1800`", + "description": "Longest wait for a subagent's first streamed token, which covers queue wait and prompt prefill. Same idea as `Session.PrefillTimeoutSeconds`. After the first token the `Session.FirstTokenTimeoutSeconds` budget applies. An agent definition can override it with `prefillTimeoutSeconds` in its frontmatter.", + "source": "SubAgentConfig.cs" + }, + "SubAgents.NoProgressTimeoutSeconds": { + "default": "`1200`", + "description": "Hard ceiling on a subagent run with no real output token. Keepalives don't reset it. Like `Session.NoProgressTimeoutSeconds`, it also sets the thinking cap at three times the value.", + "source": "SubAgentConfig.cs" + }, + "Daemon.UpdateChannel": { + "default": "`\"stable\"`", + "description": "Release channel the update check follows. `stable` only sees stable releases. `beta` also sees prereleases and moves to a stable release once one supersedes the beta.", + "source": "DaemonConfig.cs" + }, + "Notifications.Webhooks[].Format": { + "default": "`\"Generic\"`", + "description": "Payload format: `Generic` is a JSON envelope, `Slack` is Slack Block Kit.", + "source": "NotificationsConfig.cs WebhookFormatDetection" + }, + "Mattermost.ServerUrl": { + "default": "unset", + "description": "Base URL of the Mattermost server, for example `https://mm.example.com`.", + "source": "MattermostChannelOptions.cs" + }, + "Mattermost.CallbackUrl": { + "default": "unset", + "description": "URL Mattermost can reach for interactive button callbacks, for example `http://netclaw-host:5199/api/mattermost/actions`. When unset, approvals fall back to text replies, except for a subagent's approval request, which needs buttons: the binding refuses it and the session withdraws it.", + "source": "MattermostChannelOptions.cs" + }, + "Providers..Type": { + "type": "string", + "default": "`\"ollama\"`", + "description": "Provider type: `anthropic`, `deepseek`, `github-copilot`, `ollama`, `openai`, `openai-compatible`, `openrouter` or `veniceai`.", + "source": "ProviderEntry.cs, IProviderDescriptor.TypeKey implementations" + }, + "Providers..Endpoint": { + "type": "string", + "default": "`\"\"` (the provider type supplies its own default URL)", + "description": "Base URL of the provider API.", + "source": "ProviderEntry.cs" + }, + "Providers..AuthMethod": { + "type": "string", + "default": "`\"None\"`", + "allowed": "`None`, `ApiKey`, `OAuthDevice`, `OAuthPkce`", + "description": "How netclaw authenticates to the provider.", + "source": "ProviderEntry.cs, AuthMethod.cs" + }, + "Providers..ApiKey": { + "type": "string (secret)", + "default": "unset", + "description": "API key. Lives in `secrets.json`: `netclaw secrets set Providers..ApiKey `.", + "source": "ProviderEntry.cs" + }, + "Providers..VendorOptions": { + "type": "object", + "default": "unset", + "description": "Provider-specific non-secret options, for example `GitHubHost` and `GitHubApiBase` for `github-copilot`, `ThinkingDialect` for `openai-compatible`, `DisableThinking` for `ollama`, `ExcludeReasoning` for `openrouter` and `IncludeVeniceSystemPrompt` for `veniceai`. Each provider type defines its own keys.", + "source": "ProviderEntry.VendorOptions" + }, + "Providers.": { + "description": "One provider connection. The name is yours to choose and `Models.Definitions` entries refer to it.", + "source": "ProviderEntry.cs" + }, + "Providers..OAuthAccessToken": { + "type": "string (secret)", + "description": "OAuth access token. netclaw writes it to `secrets.json` when you sign in with an OAuth auth method.", + "source": "ProviderEntry.cs" + }, + "Providers..OAuthRefreshToken": { + "type": "string (secret)", + "description": "OAuth refresh token. netclaw writes it to `secrets.json` when you sign in with an OAuth auth method.", + "source": "ProviderEntry.cs" + }, + "Providers..OAuthAccountId": { + "type": "string (secret)", + "description": "OAuth account identifier. netclaw writes it to `secrets.json` when you sign in with an OAuth auth method.", + "source": "ProviderEntry.cs" + }, + "Providers..OAuthTokenExpiry": { + "type": "string (secrets.json, unencrypted)", + "description": "Expiry time of the OAuth access token, ISO 8601. netclaw writes it to `secrets.json` beside the tokens when you sign in with an OAuth auth method. It is the one value there that isn't encrypted. A key left in `netclaw.json` is retired: `netclaw doctor` reports it and `netclaw doctor --fix` moves it, after a backup.", + "source": "ProviderEntry.cs" + }, + "Session.MemoryDistillationTurnInterval": { + "description": "Same as `Session.Tuning.MemoryDistillationTurnInterval`.", + "source": "SessionTuning.cs, SessionConfig.BindTuning" + }, + "Session.MinimumRecallCompositeScore": { + "description": "Same as `Session.Tuning.MinimumRecallCompositeScore`.", + "source": "SessionTuning.cs, SessionConfig.BindTuning" + }, + "Session.MaxToolDescriptionChars": { + "description": "Same as `Session.Tuning.MaxToolDescriptionChars`.", + "source": "SessionTuning.cs, SessionConfig.BindTuning" + }, + "Session.MaxToolSchemaWarnChars": { + "description": "Same as `Session.Tuning.MaxToolSchemaWarnChars`.", + "source": "SessionTuning.cs, SessionConfig.BindTuning" + }, + "Session.Tuning.MaxToolDescriptionChars": { + "description": "Longest MCP tool description kept at registration. Longer ones are truncated to protect the context window. `0` turns truncation off.", + "source": "schema" + }, + "Session.Tuning.MaxToolSchemaWarnChars": { + "description": "Size of an MCP tool's JSON schema that triggers a warning in the log. The schema itself is never truncated, so this only reports bloated tool definitions. `0` turns the warning off.", + "source": "schema" + }, + "Retention.Logs.Days": { + "default": "`14`", + "description": "Days to keep `daemon-yyyy-MM-dd.log`, its numbered segments `daemon-yyyy-MM-dd.N.log` (the daemon starts a new segment at 10 MiB) and `crash-*.log` files in the logs directory. The daemon deletes older files 30 seconds after it starts and every 12 hours after that. The newest 3 daemon logs, segments counted as files, and the newest 3 crash logs are always kept. `0` keeps them forever. Per-session logs are never touched. See [Data retention](/configuration/data-retention/).", + "source": "RetentionSettings.cs, RetentionPolicy.cs, DaemonLogRetention" + }, + "Session.Tuning.StreamingRetryPolicy.BaseDelay": { + "allowed": "`HH:mm:ss`", + "source": "schema" + }, + "Session.Tuning.StreamingRetryPolicy.MaxDelay": { + "allowed": "`HH:mm:ss`", + "source": "schema" + } + }, + "extra": { + "McpServers..OAuthClientSecret": { + "type": "string (secret)", + "default": "unset", + "description": "Client secret for a pre-registered confidential OAuth client. Vault only: the schema rejects it in `netclaw.json`, so doctor fails the Config Schema check. Set it with `netclaw mcp add --client-id --client-secret ` or `netclaw secrets set`, which store it in `secrets.json`. Needs `OAuthClientId`.", + "after": "McpServers..OAuthClientId", + "source": "McpServerEntry.cs, McpCommand.cs" + }, + "Slack.BotToken": { + "type": "string (secret)", + "default": "unset", + "description": "Bot token, `xoxb-`. Lives in `secrets.json`: `netclaw secrets set Slack.BotToken `.", + "after": "Slack.Enabled", + "source": "SlackChannelOptions.cs, ChannelsEditorModel.cs" + }, + "Slack.AppToken": { + "type": "string (secret)", + "default": "unset", + "description": "App-level token, `xapp-`, for Socket Mode. Lives in `secrets.json`.", + "after": "Slack.BotToken", + "source": "SlackChannelOptions.cs" + }, + "Discord.BotToken": { + "type": "string (secret)", + "default": "unset", + "description": "Bot token. Lives in `secrets.json`: `netclaw secrets set Discord.BotToken `.", + "after": "Discord.Enabled", + "source": "DiscordChannelOptions.cs" + }, + "Mattermost.BotToken": { + "type": "string (secret)", + "default": "unset", + "description": "Bot access token. Lives in `secrets.json`: `netclaw secrets set Mattermost.BotToken `.", + "after": "Mattermost.Enabled", + "source": "MattermostChannelOptions.cs" + }, + "Providers..VendorOptions.ThinkingDialect": { + "type": "string", + "default": "unset", + "allowed": "`ChatTemplateKwargs`, `ReasoningEffort`, `DeepSeek`, `None`", + "description": "`openai-compatible` only. How netclaw sends a thinking level to the server. Without it only the output limit works. `netclaw model set` saves `ReasoningEffort` when it detects llama.cpp. Set it by hand for vLLM, SGLang or a proxy.", + "after": "Providers..VendorOptions", + "source": "OpenAiCompatibleDescriptor.cs, ThinkingDialectMap.cs" + } + }, + "sections": { + "Slack": { + "summary": "The Slack adapter. [Set up Slack](/channels/slack/) walks through the app manifest and tokens." + }, + "Discord": { + "summary": "The Discord adapter. [Set up Discord](/channels/discord/) covers the bot, intents and tokens." + }, + "Mattermost": { + "summary": "The Mattermost adapter. [Set up Mattermost](/channels/mattermost/) covers the bot account and callback URL." + }, + "Logging": { + "summary": "Daemon log level and console output. Logs go to `daemon-.log` under the data directory regardless of these keys, rolling into numbered segments at 10 MiB." + }, + "Telemetry": { + "summary": "OpenTelemetry export. [OpenTelemetry](/observability/opentelemetry/) shows how to wire a collector." + }, + "Session": { + "summary": "Per-conversation timeouts, compaction and memory-capture timing. The timeout keys take whole seconds, `IdleTimeout` takes `HH:mm:ss`. [Sessions](/architecture/sessions/) explains the lifecycle these keys tune.", + "notes": "A value below a key's minimum fails `netclaw doctor` schema validation. The daemon itself enforces a 1 second minimum on `TurnLlmTimeoutSeconds`, `ToolExecutionTimeoutSeconds` and `SidecarLlmTimeoutSeconds`." + }, + "Security": { + "summary": "Deployment posture and shell mode, which choose the default tool profiles. [Security model](/security/security-model/) explains the three postures." + }, + "Tools": { + "summary": "Tool grants per audience, file roots, approval policy and `web_fetch` limits. A configured list replaces the default list, so a list with one tool grants one tool. [Built-in tools](/reference/tools/) lists the tools and defaults, and [How tool approvals work](/security/approvals/) covers `ApprovalPolicy`." + }, + "Providers": { + "summary": "Where each model is served. [Managed providers](/configuration/managed-providers/) and [self-hosted providers](/configuration/self-hosted-providers/) have one example per provider type." + }, + "Models": { + "summary": "Which model plays Main, Fallback and Compaction, and each definition's context window, modalities, output limit and thinking level. [Model configuration](/configuration/models/) has examples. The older inline shape (`Models.Main`, `Models.Fallback`, `Models.Compaction` as objects) still loads. Mixing it with `Definitions` and `Roles` stops startup." + }, + "Memory": { + "summary": "Cross-session memory, embeddings and recall. [Memory and embeddings](/configuration/memory/) explains how the pieces fit together. The embedding model downloads on first start unless `Embeddings.AutoDownload` is `false`.", + "notes": "Set `Memory.Enabled` to `false` and the memory tools and automatic recall disappear for every audience." + }, + "Search": { + "summary": "The `web_search` backend. [Search providers](/configuration/search-providers/) covers Brave and SearXNG setup." + }, + "SkillSync": { + "summary": "Switch for the skill index tools. [Skills overview](/skills/overview/) explains where skills come from." + }, + "ExternalSkills": { + "summary": "Skill directories from other tools, such as Claude Code. [External skills](/skills/external-skills/) has the setup." + }, + "SkillFeeds": { + "summary": "Private skill servers that netclaw syncs from at startup and on a timer. [Skill feeds](/skills/skill-feeds/) has the setup." + }, + "SubAgents": { + "summary": "Switches and watchdogs for `spawn_agent`. [Custom subagents](/skills/custom-subagents/) shows how to define agents." + }, + "Notifications": { + "summary": "Where operational alerts go. [Operational alerts](/observability/operational-alerts/) lists the alert types." + }, + "Webhooks": { + "summary": "The inbound webhook endpoint. The routes themselves aren't in `netclaw.json`: each one is a file under `config/webhooks/`, managed with `netclaw webhooks` or the webhook tools. [Webhooks](/configuration/webhooks/) covers routes and verification." + }, + "Scheduling": { + "summary": "Switch for reminders and scheduled tasks. [Reminders](/configuration/reminders/) covers creating and running them.", + "notes": "Set `Scheduling.Enabled` to `false` and the five reminder tools aren't registered." + }, + "Daemon": { + "summary": "Bind address, port and how the daemon is exposed to the network. Changing any of these in `netclaw.json` restarts the daemon. [Exposure modes](/deployment/exposure-modes/) explains each mode and [Docker](/deployment/docker/) covers the container settings.", + "notes": "The Docker image's health check probes port 5199. A container with a different `Daemon.Port` reports `unhealthy` even though the daemon is up, so override the health check when you change the port.\n\nIn `Local` exposure mode the daemon answers 401 to a request whose `Host` isn't `localhost` or a loopback IP literal (`127.0.0.1`, `[::1]`), or whose `Origin`, when present, isn't `http://` plus the same host. A client that reaches the daemon by another host name, or a browser app on another port, is refused. `Daemon.Host` as a bind value still accepts spellings such as `127.1`." + }, + "Workspaces": { + "summary": "Where netclaw looks for your project repositories. [Context structure](/architecture/context-structure/) explains how projects feed the prompt." + }, + "McpServers": { + "summary": "MCP server profiles keyed by name. Add them with `netclaw mcp add` or edit the file. [MCP servers](/configuration/mcp-servers/) has the walkthrough and [MCP tool permissions](/guides/mcp-tool-permissions/) covers per-audience access." + }, + "Retention": { + "summary": "How long the daemon keeps log files before a scheduled clean-up deletes them. [Data retention](/configuration/data-retention/) explains what is and isn't deleted." + } + }, + "skip": [] +} diff --git a/scripts/gen-config-reference.mjs b/scripts/gen-config-reference.mjs new file mode 100644 index 0000000..c39a23c --- /dev/null +++ b/scripts/gen-config-reference.mjs @@ -0,0 +1,374 @@ +#!/usr/bin/env node +/** + * Generates the option tables in src/content/docs/configuration/reference.md + * from the netclaw JSON schema, plus a hand-maintained overrides file. + * + * Run by hand at each release, from the repo root. Do NOT wire this into + * `npm run build`: the product repo is not available in CI. + * + * node scripts/gen-config-reference.mjs + * node scripts/gen-config-reference.mjs --schema /path/to/netclaw-config.v1.schema.json + * NETCLAW_SCHEMA=/path/to/schema.json node scripts/gen-config-reference.mjs + * node scripts/gen-config-reference.mjs --check # exit 1 if the page is stale + * + * Inputs + * --schema JSON schema (default: $NETCLAW_SCHEMA, then the product clone at + * ~/repositories/netclaw-dev/netclaw/src/Netclaw.Configuration/Schemas/) + * --overrides scripts/config-reference-overrides.json (default). Defaults, + * descriptions and keys that the schema lacks, each with the C# file + * it came from. Re-verify these against the options classes at each release. + * --out the page (default src/content/docs/configuration/reference.md) + * + * The script rewrites only the text between the BEGIN/END GENERATED markers in the + * page. Everything else in the page is hand-written. It exits non-zero when a key + * in the schema still has no default or no description after overrides are applied, + * or when an override names a key that no longer exists, so a new release can't + * silently ship an empty cell. + */ +import { readFileSync, writeFileSync } from 'node:fs'; +import { homedir } from 'node:os'; +import { dirname, join, resolve } from 'node:path'; +import { fileURLToPath } from 'node:url'; + +const here = dirname(fileURLToPath(import.meta.url)); +const root = resolve(here, '..'); + +const args = process.argv.slice(2); +const arg = (name) => { + const i = args.indexOf(name); + return i >= 0 ? args[i + 1] : undefined; +}; +const checkOnly = args.includes('--check'); + +const defaultSchema = join( + homedir(), + 'repositories/netclaw-dev/netclaw/src/Netclaw.Configuration/Schemas/netclaw-config.v1.schema.json', +); +const schemaPath = resolve(arg('--schema') ?? process.env.NETCLAW_SCHEMA ?? defaultSchema); +const overridesPath = resolve(arg('--overrides') ?? join(here, 'config-reference-overrides.json')); +const outPath = resolve(arg('--out') ?? join(root, 'src/content/docs/configuration/reference.md')); + +const BEGIN = ''; +const END = ''; + +const schema = JSON.parse(readFileSync(schemaPath, 'utf8')); +const overrides = JSON.parse(readFileSync(overridesPath, 'utf8')); +const keyOverrides = { ...(overrides.keys ?? {}) }; +for (const prefix of overrides.shared?.prefixes ?? []) { + for (const [key, spec] of Object.entries(overrides.shared?.keys ?? {})) { + const path = `${prefix}.${key}`; + if (!(path in keyOverrides)) keyOverrides[path] = spec; + } +} +const extraKeys = overrides.extra ?? {}; +const sectionMeta = overrides.sections ?? {}; +const mapKeyNames = overrides.mapKeyNames ?? {}; +const skipKeys = new Set(overrides.skip ?? []); + +const problems = []; +const usedOverrides = new Set(); + +// ---- schema helpers ------------------------------------------------------- + +function deref(node) { + while (node && typeof node === 'object' && node.$ref) { + const parts = node.$ref.replace(/^#\//, '').split('/'); + let target = schema; + for (const p of parts) target = target[p]; + const { $ref, ...rest } = node; + node = { ...target, ...rest }; + } + return node; +} + +/** Collapse anyOf [X, null] and a multi-type with null into X plus nullable. */ +function normalize(node) { + node = deref(node); + if (node === false) return { forbidden: true }; + if (node === true || node === undefined) return { type: 'any' }; + let nullable = false; + if (Array.isArray(node.anyOf)) { + const nonNull = node.anyOf.filter((n) => !(n.type === 'null')); + nullable = nonNull.length !== node.anyOf.length; + if (nonNull.length === 1) { + const { anyOf, ...rest } = node; + node = { ...deref(nonNull[0]), ...rest }; + } + } + if (Array.isArray(node.oneOf)) { + // Models: legacy inline roles or named Definitions/Roles. Document the named shape. + const named = node.oneOf.map(deref).find((n) => n.properties && n.properties.Definitions); + if (named) { + const { oneOf, ...rest } = node; + node = { ...named, ...rest }; + } + } + if (Array.isArray(node.type)) { + nullable = nullable || node.type.includes('null'); + const types = node.type.filter((t) => t !== 'null'); + node = { ...node, type: types.length === 1 ? types[0] : types }; + } + return { ...node, nullable }; +} + +function typeLabel(node) { + const n = normalize(node); + if (n.type === 'array') { + const item = normalize(n.items ?? {}); + const inner = item.type === 'object' ? 'object' : (item.enum ? 'string' : (item.type ?? 'any')); + return `array of ${inner}` + (n.nullable ? ' (nullable)' : ''); + } + if (n.type === 'object' && n.additionalProperties && typeof n.additionalProperties === 'object' && !n.properties) { + const v = normalize(n.additionalProperties); + return `map of ${v.type === 'object' ? 'object' : (v.type ?? 'any')}` + (n.nullable ? ' (nullable)' : ''); + } + const t = Array.isArray(n.type) ? n.type.join(' or ') : (n.type ?? 'any'); + return t + (n.nullable ? ' (nullable)' : ''); +} + +const code = (s) => '`' + String(s).replace(/\|/g, '\\|') + '`'; +// Escape pipes that code() has not already escaped, and keep em dashes out of the page. +const cell = (s) => String(s ?? '-').replace(/\r?\n/g, ' ').replace(/(? v !== null).map((v) => code(v)).join(', ')); + else if (n.type === 'array') { + const item = normalize(n.items ?? {}); + if (item.enum) parts.push('items: ' + item.enum.map((v) => code(v)).join(', ')); + } else if (n.type === 'object' && n.additionalProperties && typeof n.additionalProperties === 'object') { + const v = normalize(n.additionalProperties); + if (v.enum) parts.push('values: ' + v.enum.map((x) => code(x)).join(', ')); + } + if (n.minimum !== undefined && n.maximum !== undefined) parts.push(`${n.minimum} to ${n.maximum}`); + else if (n.minimum !== undefined) parts.push(`${n.minimum} or more`); + else if (n.maximum !== undefined) parts.push(`${n.maximum} or less`); + if (n.pattern) parts.push('pattern ' + code(n.pattern)); + return parts.length ? parts.join('; ') : '-'; +} + +// ---- flattening ----------------------------------------------------------- + +const AUDIENCES = ['Public', 'Team', 'Personal']; +const rows = []; // { section, path, type, def, allowed, desc, secret } + +function mapKeyName(path) { + return mapKeyNames[path] ?? 'key'; +} + +function pushRow(section, path, node, extra = {}) { + const ov = keyOverrides[path]; + if (ov) usedOverrides.add(path); + const n = normalize(node); + const schemaDefault = n.default !== undefined ? renderDefault(n.default) : undefined; + let def = ov?.default ?? schemaDefault; + if (def === undefined && n.nullable) def = 'unset'; + const desc = ov?.description ?? n.description; + const row = { + section, + path, + type: ov?.type ?? typeLabel(node), + def, + allowed: ov?.allowed ?? allowed(node), + desc, + ...extra, + }; + if (def === undefined) problems.push(`no default: ${path}`); + if (!desc) problems.push(`no description: ${path}`); + rows.push(row); +} + +function visit(section, path, node, isRoot = false) { + const n = normalize(node); + if (n.forbidden) return; + if (skipKeys.has(path)) return; + + if (n.type === 'object' && n.properties) { + if (!isRoot && (n.description || keyOverrides[path])) { + // A container with its own description gets a row so the section reads completely. + pushContainer(section, path, n); + } + const props = Object.entries(n.properties); + const audienceProps = props.filter(([k]) => AUDIENCES.includes(k)); + const sameShape = + audienceProps.length === 3 && + new Set(audienceProps.map(([, v]) => v.$ref)).size === 1 && + audienceProps[0][1].$ref; + let emittedAudience = false; + for (const [key, child] of props) { + if (sameShape && AUDIENCES.includes(key)) { + if (!emittedAudience) { + emittedAudience = true; + visit(section, `${path}.`, child); + } + continue; + } + visit(section, `${path}.${key}`, child); + } + return; + } + + if (n.type === 'object' && n.additionalProperties && typeof n.additionalProperties === 'object') { + const v = normalize(n.additionalProperties); + if (v.type === 'object' && v.properties) { + if (keyOverrides[path]) pushContainer(section, path, { ...n, type: 'object' }); + pushContainer(section, `${path}.<${mapKeyName(path)}>`, { ...v, description: keyOverrides[`${path}.<${mapKeyName(path)}>`]?.description ?? v.description ?? n.description, type: 'object' }); + for (const [key, child] of Object.entries(v.properties)) { + visit(section, `${path}.<${mapKeyName(path)}>.${key}`, child); + } + return; + } + pushRow(section, path, node); + return; + } + + if (n.type === 'array') { + const item = normalize(n.items ?? {}); + if (item.type === 'object' && item.properties) { + pushRow(section, path, node); + for (const [key, child] of Object.entries(item.properties)) { + visit(section, `${path}[].${key}`, child); + } + return; + } + } + + pushRow(section, path, node); +} + +function pushContainer(section, path, n) { + const ov = keyOverrides[path]; + if (ov) usedOverrides.add(path); + const desc = ov?.description ?? n.description; + if (!desc) problems.push(`no description: ${path}`); + rows.push({ + section, + path, + type: ov?.type ?? 'object', + def: ov?.default ?? '-', + allowed: '-', + desc, + }); +} + +// ---- walk the schema ------------------------------------------------------ + +const sectionNames = Object.keys(schema.properties).filter((k) => k !== 'configVersion'); +const forbiddenSections = []; + +for (const name of sectionNames) { + const node = schema.properties[name]; + if (node === false) { + forbiddenSections.push(name); + continue; + } + visit(name, name, node, true); +} + +// Keys the schema doesn't carry (secrets, provider entries, keys the daemon reads that the schema omits). +for (const [path, spec] of Object.entries(extraKeys)) { + const section = path.split('.')[0]; + if (!sectionNames.includes(section)) { + problems.push(`extra key in unknown section: ${path}`); + continue; + } + rows.push({ + section, + path, + type: spec.type, + def: spec.default ?? 'unset', + allowed: spec.allowed ?? '-', + desc: spec.description, + after: spec.after, + }); + if (!spec.description || !spec.type) problems.push(`incomplete extra key: ${path}`); +} + +for (const path of Object.keys(keyOverrides)) { + if (!usedOverrides.has(path)) problems.push(`override names a key that is not in the schema: ${path}`); +} +for (const name of sectionNames) { + if (!forbiddenSections.includes(name) && !sectionMeta[name]) problems.push(`no section summary: ${name}`); +} + +// ---- render --------------------------------------------------------------- + +function renderRows(list) { + const out = []; + out.push('| Key | Type | Default | Allowed values | Description |'); + out.push('|-----|------|---------|----------------|-------------|'); + for (const r of list) { + out.push(`| ${code(r.path)} | ${cell(r.type)} | ${r.def === undefined ? '-' : cell(r.def)} | ${cell(r.allowed)} | ${cell(r.desc)} |`); + } + return out.join('\n'); +} + +/** Order rows so an extra key with `after` lands right behind the key it names. */ +function orderRows(list) { + const base = list.filter((r) => !r.after); + for (const r of list.filter((x) => x.after)) { + const idx = base.findIndex((b) => b.path === r.after); + if (idx < 0) base.push(r); + else base.splice(idx + 1, 0, r); + } + return base; +} + +const out = []; +for (const name of sectionNames) { + if (forbiddenSections.includes(name)) continue; + const meta = sectionMeta[name] ?? {}; + const list = orderRows(rows.filter((r) => r.section === name)); + out.push(`### ${name}`); + out.push(''); + if (meta.summary) { + out.push(meta.summary); + out.push(''); + } + if (list.length) { + out.push(renderRows(list)); + out.push(''); + } + if (meta.notes) { + out.push(meta.notes); + out.push(''); + } +} + +const generated = out.join('\n').replace(/\n+$/, '\n'); + +// ---- write ---------------------------------------------------------------- + +const page = readFileSync(outPath, 'utf8'); +const start = page.indexOf(BEGIN); +const end = page.indexOf(END); +if (start < 0 || end < 0 || end < start) { + console.error(`Markers not found in ${outPath}. Expected:\n${BEGIN}\n${END}`); + process.exit(2); +} +const next = page.slice(0, start + BEGIN.length) + '\n\n' + generated + '\n' + page.slice(end); + +if (problems.length) { + console.error('Config reference problems:\n ' + problems.join('\n ')); +} + +if (checkOnly) { + if (next !== page) { + console.error(`${outPath} is stale. Run: node scripts/gen-config-reference.mjs`); + process.exit(1); + } +} else if (next !== page) { + writeFileSync(outPath, next); + console.log(`Wrote ${outPath}`); +} else { + console.log(`${outPath} already up to date`); +} +console.log(`${sectionNames.length} sections, ${rows.length} rows, ${forbiddenSections.length} forbidden section(s): ${forbiddenSections.join(', ') || 'none'}`); +if (problems.length) process.exit(1); diff --git a/scripts/gen-tools-reference.mjs b/scripts/gen-tools-reference.mjs new file mode 100644 index 0000000..e47bd51 --- /dev/null +++ b/scripts/gen-tools-reference.mjs @@ -0,0 +1,355 @@ +#!/usr/bin/env node +/** + * Generates the tool tables in src/content/docs/reference/tools.md from the product source. + * + * Run by hand at each release, from the repo root. Do NOT wire this into + * `npm run build`: the product repo is not available in CI. + * + * node scripts/gen-tools-reference.mjs + * node scripts/gen-tools-reference.mjs --src /path/to/netclaw/src + * NETCLAW_SRC=/path/to/netclaw/src node scripts/gen-tools-reference.mjs + * node scripts/gen-tools-reference.mjs --check # exit 1 if the page is stale + * + * What it reads from the product source (re-read these files if the script's output looks wrong): + * - every [NetclawTool(...)] attribute: tool name, grant category, LLM-facing description + * - Netclaw.Daemon/Configuration/Channel*Tools.cs: the three channel tools declared with `Name => "..."` + * - Register / RegisterCore / ReplaceCore calls: whether a tool is loaded up front (core) or on demand (deferred) + * - Netclaw.Configuration/ToolAudienceProfiles.cs: the Public and Team default allow lists, + * which tools are profile-managed, and the Personal profile (ToolsMode All) + * - Netclaw.Actors/Tools/ToolAccessPolicy.cs: shell-coupled tools and the subsystem feature gates + * - each tool's own source file: an `audience == TrustAudience.Public` guard that makes the tool return + * "This tool is not available." for Public even when the allow list would let it through + * + * Hand-written inputs live in scripts/tools-reference-overrides.json: the one-line summary shown for each + * tool, the approval default and the config keys. The run fails when a tool has no summary, when an + * override names a tool that no longer exists, or when a source assertion no longer holds. + * + * The script rewrites only the text between the BEGIN/END GENERATED markers in the page. + */ +import { readdirSync, readFileSync, statSync, writeFileSync } from 'node:fs'; +import { homedir } from 'node:os'; +import { dirname, join, relative, resolve } from 'node:path'; +import { fileURLToPath } from 'node:url'; + +const here = dirname(fileURLToPath(import.meta.url)); +const root = resolve(here, '..'); +const args = process.argv.slice(2); +const arg = (name) => { + const i = args.indexOf(name); + return i >= 0 ? args[i + 1] : undefined; +}; +const checkOnly = args.includes('--check'); + +const srcDir = resolve(arg('--src') ?? process.env.NETCLAW_SRC ?? join(homedir(), 'repositories/netclaw-dev/netclaw/src')); +const overridesPath = resolve(arg('--overrides') ?? join(here, 'tools-reference-overrides.json')); +const outPath = resolve(arg('--out') ?? join(root, 'src/content/docs/reference/tools.md')); + +const BEGIN = ''; +const END = ''; + +const overrides = JSON.parse(readFileSync(overridesPath, 'utf8')); +const toolOverrides = overrides.tools ?? {}; +const groups = overrides.groups ?? []; +const problems = []; + +// ---- read the source tree ------------------------------------------------- + +function* walk(dir) { + for (const entry of readdirSync(dir)) { + if (entry === 'bin' || entry === 'obj' || entry === 'node_modules') continue; + const full = join(dir, entry); + if (statSync(full).isDirectory()) { + if (/Tests$/.test(entry)) continue; + yield* walk(full); + } else if (entry.endsWith('.cs')) { + yield full; + } + } +} + +const files = [...walk(srcDir)].map((path) => ({ path, rel: relative(srcDir, path), text: readFileSync(path, 'utf8') })); +const file = (rel) => { + const f = files.find((x) => x.rel === rel); + if (!f) { + problems.push(`source file not found: ${rel}`); + return { rel, text: '' }; + } + return f; +}; + +/** Returns the balanced-parenthesis argument text that starts at text[open] === '('. */ +function balanced(text, open) { + let depth = 0; + let inString = false; + for (let i = open; i < text.length; i++) { + const c = text[i]; + if (inString) { + if (c === '\\') i++; + else if (c === '"') inString = false; + continue; + } + if (c === '"') inString = true; + else if (c === '(') depth++; + else if (c === ')') { + depth--; + if (depth === 0) return text.slice(open + 1, i); + } + } + return ''; +} + +/** Splits on commas that are not inside a string or nested brackets. */ +function splitArgs(text) { + const parts = []; + let depth = 0; + let inString = false; + let current = ''; + for (let i = 0; i < text.length; i++) { + const c = text[i]; + if (inString) { + current += c; + if (c === '\\') current += text[++i]; + else if (c === '"') inString = false; + continue; + } + if (c === '"') inString = true; + if ('([{'.includes(c)) depth++; + if (')]}'.includes(c)) depth--; + if (c === ',' && depth === 0) { + parts.push(current.trim()); + current = ''; + } else current += c; + } + if (current.trim()) parts.push(current.trim()); + return parts; +} + +const stringLiterals = (expr) => [...expr.matchAll(/"((?:[^"\\]|\\.)*)"/g)].map((m) => m[1].replace(/\\"/g, '"')).join(''); + +// ---- tools declared with [NetclawTool] ----------------------------------- + +const tools = new Map(); // name -> { name, className, grant, description, source } + +for (const f of files) { + let from = 0; + for (;;) { + const at = f.text.indexOf('[NetclawTool(', from); + if (at < 0) break; + from = at + 1; + const open = at + '[NetclawTool'.length; + const argText = balanced(f.text, open); + const rest = f.text.slice(open + argText.length + 2); + const cls = /class\s+(\w+)/.exec(rest)?.[1]; + if (!cls || !argText) continue; + const parts = splitArgs(argText); + if (parts.length < 2) continue; + let name; + if (parts[0].startsWith('"')) name = stringLiterals(parts[0]); + else { + const constName = parts[0].trim(); + name = new RegExp(`const\\s+string\\s+${constName}\\s*=\\s*"([^"]+)"`).exec(f.text)?.[1]; + } + if (!name || !/^[a-z_]+$/.test(name)) continue; // skips doc-comment mentions and non-tool uses + const grant = /Grant\s*=\s*"([^"]+)"/.exec(argText)?.[1] ?? 'default'; + tools.set(name, { name, className: cls, grant, description: stringLiterals(parts[1]), source: f.rel }); + } +} + +// ---- channel tools (IChannelTool implementations) ------------------------- + +const channelRegistration = file('Netclaw.Daemon/Configuration/ChannelToolRegistration.cs').text; +const channelToolsDeferred = /registry\.Register\(tool\)/.test(channelRegistration); +if (!channelToolsDeferred) problems.push('ChannelToolRegistration no longer calls registry.Register(tool): re-check how channel tools load'); + +for (const rel of ['Netclaw.Daemon/Configuration/ChannelSendTools.cs', 'Netclaw.Daemon/Configuration/ChannelLookupTools.cs']) { + const f = file(rel); + for (const m of f.text.matchAll(/class\s+(\w+)[^{]*?\{[\s\S]*?string Name\s*=>\s*"(\w+)";/g)) { + tools.set(m[2], { name: m[2], className: m[1], grant: 'channel', description: '', source: rel, channel: true }); + } +} + +// Channel-specific lookup tools declared with [NetclawTool] already have a grant of "default" and live in the channel projects. + +// ---- loading tier --------------------------------------------------------- + +const tierByClass = new Map(); +for (const f of files) { + for (const m of f.text.matchAll(/\.(RegisterCore|Register|ReplaceCore)\(\s*new\s+(\w+)\(/g)) { + const tier = m[1] === 'Register' ? 'Deferred' : 'Core'; + if (tierByClass.get(m[2]) !== 'Core') tierByClass.set(m[2], tier); + } +} + +// ---- audience defaults ---------------------------------------------------- + +const profilesFile = file('Netclaw.Configuration/ToolAudienceProfiles.cs'); +const catalogText = /static class ToolAudienceProfileToolCatalog[\s\S]*?static class ToolAudienceProfileDefaults/.exec(profilesFile.text)?.[0] ?? ''; +const constants = new Map([...catalogText.matchAll(/public const string (\w+) = "([^"]+)";/g)].map((m) => [m[1], m[2]])); +const lists = new Map(); +for (const m of catalogText.matchAll(/public static IReadOnlyList (\w+) \{ get; \} =\s*\[([\s\S]*?)\];/g)) { + const items = []; + for (const raw of m[2].split(',').map((s) => s.trim()).filter(Boolean)) { + if (raw.startsWith('..')) items.push(...(lists.get(raw.slice(2).trim()) ?? [])); + else if (constants.has(raw)) items.push(constants.get(raw)); + else problems.push(`unresolved catalog item ${raw} in ${m[1]}`); + } + lists.set(m[1], items); +} +const publicDefault = lists.get('PublicDefaultAllowedTools') ?? []; +const teamDefault = lists.get('TeamDefaultAllowedTools') ?? []; +const profileManaged = new Set(lists.get('ProfileManagedTools') ?? []); +if (!publicDefault.length || !teamDefault.length || !profileManaged.size) problems.push('could not read the audience default lists from ToolAudienceProfiles.cs'); + +const personalAll = /CreatePersonal\(\)[\s\S]*?ToolsMode\s*=\s*ToolProfileMode\.All/.test(profilesFile.text); +if (!personalAll) problems.push('CreatePersonal no longer sets ToolsMode = All: the Personal column is wrong'); +const shellApproval = /CreateProfilesForPosture[\s\S]*?\[ToolAudienceProfileToolCatalog\.ShellExecute\]\s*=\s*ToolApprovalMode\.Approval/.test(profilesFile.text); +if (!shellApproval) problems.push('the Personal posture no longer defaults shell_execute to Approval: re-check the approval column'); + +const policy = file('Netclaw.Actors/Tools/ToolAccessPolicy.cs').text; +const shellCoupled = /IsShellCoupledTool\(INetclawTool tool\)\s*=>\s*IsShellTool\(tool\)\s*\|\|\s*string\.Equals\(tool\.Name, CheckBackgroundJobTool\.ToolName/.test(policy); +if (!shellCoupled) problems.push('IsShellCoupledTool changed: re-check which tools need shell execution'); +const shellCoupledTools = ['shell_execute', 'check_background_job']; + +const gateConfig = { + MemoryEnabled: 'Memory.Enabled', + SearchEnabled: 'Search.Enabled', + SkillSyncEnabled: 'SkillSync.Enabled', + SubAgentsEnabled: 'SubAgents.Enabled', + SchedulingEnabled: 'Scheduling.Enabled', +}; +const gateByTool = new Map(); +const gateBlock = /IsFeatureDisabledTool\(string toolName\)[\s\S]*?\n \}/.exec(policy)?.[0] ?? ''; +for (const m of gateBlock.matchAll(/((?:"\w+"\s*(?:or\s*)?)+)\s*=>\s*!_featureGates\.(\w+)/g)) { + for (const t of m[1].matchAll(/"(\w+)"/g)) gateByTool.set(t[1], gateConfig[m[2]] ?? m[2]); +} +if (!gateByTool.size) problems.push('could not read the feature gates from ToolAccessPolicy.IsFeatureDisabledTool'); + +// ---- compute rows --------------------------------------------------------- + +// Tools whose own code refuses Public regardless of the allow lists. +const publicGuard = /^[^/\n]*(?:audience|Audience)\s*==\s*TrustAudience\.Public/m; +const publicBlocked = new Set([...tools.values()].filter((t) => publicGuard.test(files.find((f) => f.rel === t.source)?.text ?? '')).map((t) => t.name)); +for (const name of Object.keys(toolOverrides)) { + if (toolOverrides[name].publicBlocked && !publicBlocked.has(name)) problems.push(`override says ${name} is blocked for Public but the source has no guard`); +} +for (const name of publicBlocked) { + if (!toolOverrides[name]?.publicBlocked) problems.push(`source blocks ${name} for Public: add "publicBlocked": true to its override`); +} + +function availability(tool) { + const n = tool.name; + if (publicBlocked.has(n)) { + return { Public: 'No', Team: profileManaged.has(n) && !teamDefault.includes(n) ? 'No' : 'Yes', Personal: 'Yes', profileManaged: profileManaged.has(n) }; + } + if (!profileManaged.has(n)) return { Public: 'Yes', Team: 'Yes', Personal: 'Yes', profileManaged: false }; + const inPublic = publicDefault.includes(n); + const inTeam = teamDefault.includes(n); + return { + Public: inPublic ? 'Yes' : 'No', + Team: inTeam ? 'Yes' : 'No', + Personal: 'Yes', + profileManaged: true, + }; +} + +const rows = [...tools.values()].map((t) => { + const ov = toolOverrides[t.name]; + if (!ov) problems.push(`no override (summary, group) for tool: ${t.name}`); + const a = availability(t); + const requires = []; + if (shellCoupledTools.includes(t.name)) requires.push('`Tools.ShellMode` or `Security.ShellExecutionMode`, which must resolve to `HostAllowed`'); + if (gateByTool.has(t.name)) requires.push(`\`${gateByTool.get(t.name)}\``); + if (ov?.requires) requires.push(...ov.requires); + const tier = (t.channel || t.source.startsWith('Netclaw.Channels.')) ? 'Deferred' : (tierByClass.get(t.className) ?? 'Deferred'); + if (!t.channel && !t.source.startsWith('Netclaw.Channels.') && !tierByClass.has(t.className)) problems.push(`no Register call found for ${t.className}: loading tier assumed Deferred`); + const shell = shellCoupledTools.includes(t.name); + return { + ...t, + ...a, + Public: shell ? 'No' : a.Public, + Team: shell ? 'No' : a.Team, + tier, + requires, + summary: ov?.summary, + group: ov?.group, + approval: ov?.approval ?? 'Runs without a prompt', + config: ov?.config ?? [], + }; +}); + +for (const name of Object.keys(toolOverrides)) { + if (!tools.has(name)) problems.push(`override names a tool that is not in the source: ${name}`); +} +for (const g of groups) { + if (!rows.some((r) => r.group === g.id)) problems.push(`group with no tools: ${g.id}`); +} +for (const r of rows) { + if (r.group && !groups.some((g) => g.id === r.group)) problems.push(`unknown group ${r.group} for ${r.name}`); +} + +// ---- render --------------------------------------------------------------- + +const cell = (s) => String(s).replace(/(? r.tier === 'Core').length; +out.push(`netclaw 0.27.1 registers ${count} built-in tools. ${coreCount} are loaded into every prompt (core) and the rest are found with \`search_tools\` and loaded with \`load_tool\` (deferred). The table columns say which audiences get each tool by default. A tool outside the profile-managed set is available to every audience when its subsystem is enabled, except that Public never gets ${[...publicBlocked].sort().map((t) => `\`${t}\``).join(', ').replace(/, ([^,]*)$/, ' or $1')}: those tools refuse Public in their own code.`.replace('0.27.1', overrides.version ?? '0.27.1')); +out.push(''); + +for (const g of groups) { + const list = rows.filter((r) => r.group === g.id).sort((x, y) => (toolOverrides[x.name]?.order ?? 0) - (toolOverrides[y.name]?.order ?? 0) || x.name.localeCompare(y.name)); + if (!list.length) continue; + out.push(`### ${g.title}`); + out.push(''); + if (g.intro) { + out.push(g.intro); + out.push(''); + } + out.push('| Tool | What it does | Loaded | Public | Team | Personal | Approval by default | Controlled by |'); + out.push('|------|--------------|--------|--------|------|----------|---------------------|---------------|'); + for (const r of list) { + const needs = [...r.requires, ...r.config.map((c) => `\`${c}\``)]; + out.push(`| \`${r.name}\` | ${cell(r.summary ?? '')} | ${r.tier} | ${r.Public} | ${r.Team} | ${r.Personal} | ${cell(r.approval)} | ${cell(needs.length ? needs.join(', ') : '-')} |`); + } + out.push(''); +} + +out.push('### Default allow lists'); +out.push(''); +out.push('The lists below are the 0.27.1 defaults for `Tools.AudienceProfiles..AllowedTools`. Personal uses `ToolsMode: All` and has no list.'.replace('0.27.1', overrides.version ?? '0.27.1')); +out.push(''); +out.push('| Audience | Default `AllowedTools` |'); +out.push('|----------|------------------------|'); +out.push(`| Public | ${publicDefault.map((t) => `\`${t}\``).join(', ')} |`); +out.push(`| Team | ${teamDefault.map((t) => `\`${t}\``).join(', ')} |`); +out.push(''); + +const generated = out.join('\n'); + +// ---- write ---------------------------------------------------------------- + +const page = readFileSync(outPath, 'utf8'); +const start = page.indexOf(BEGIN); +const end = page.indexOf(END); +if (start < 0 || end < 0 || end < start) { + console.error(`Markers not found in ${outPath}. Expected:\n${BEGIN}\n${END}`); + process.exit(2); +} +const next = page.slice(0, start + BEGIN.length) + '\n\n' + generated + '\n' + page.slice(end); + +if (problems.length) console.error('Tools reference problems:\n ' + problems.join('\n ')); + +if (checkOnly) { + if (next !== page) { + console.error(`${outPath} is stale. Run: node scripts/gen-tools-reference.mjs`); + process.exit(1); + } +} else if (next !== page) { + writeFileSync(outPath, next); + console.log(`Wrote ${outPath}`); +} else { + console.log(`${outPath} already up to date`); +} +console.log(`${count} tools (${coreCount} core), public default ${publicDefault.length}, team default ${teamDefault.length}`); +if (problems.length) process.exit(1); diff --git a/scripts/generate-social-images.mjs b/scripts/generate-social-images.mjs index 06e8ac9..50aaafb 100644 --- a/scripts/generate-social-images.mjs +++ b/scripts/generate-social-images.mjs @@ -24,6 +24,9 @@ const SECTION_MAP = { 'deployment': 'Deployment', 'architecture': 'Architecture', 'guides': 'Guides', + 'reference': 'Reference', + 'troubleshooting': 'Troubleshooting', + 'comparisons': 'Comparisons', }; function loadManifest() { diff --git a/scripts/screenshot-page.mjs b/scripts/screenshot-page.mjs index 9c2cdf3..1418ef1 100755 --- a/scripts/screenshot-page.mjs +++ b/scripts/screenshot-page.mjs @@ -4,16 +4,23 @@ // Usage: node scripts/screenshot-page.mjs // Example: node scripts/screenshot-page.mjs cli/status // -// Requires: npm run build (must be run first) +// Run it in the Playwright container so the browser is the pinned, containerized one: +// ./screenshots/shoot-page.sh cli/status +// +// Serves dist/ itself with a tiny static server (no astro needed), so it runs anywhere +// the `playwright` module resolves. Requires `npm run build` first. // Output: docs-logs/screenshots/.png -// Timeout: 60 seconds total, then kills everything and exits non-zero. +// +// Env: +// SCREENSHOT_OUT output directory (default docs-logs/screenshots) +// SCREENSHOT_BASE_URL screenshot a running server (e.g. `npm run dev`) instead of dist/ +// Timeout: 60 seconds total, then exits non-zero. import { chromium } from 'playwright'; -import { spawn } from 'child_process'; -import { mkdirSync, existsSync } from 'fs'; -import { dirname, join } from 'path'; +import { createServer } from 'http'; +import { mkdirSync, existsSync, statSync, createReadStream } from 'fs'; +import { dirname, join, extname, normalize } from 'path'; import { fileURLToPath } from 'url'; -import { createServer } from 'net'; const TIMEOUT_MS = 60_000; @@ -27,64 +34,67 @@ if (!slug) { process.exit(1); } -const outputDir = join(projectDir, 'docs-logs', 'screenshots'); +const outputDir = process.env.SCREENSHOT_OUT || join(projectDir, 'docs-logs', 'screenshots'); const outputFile = join(outputDir, slug.replace(/\//g, '-') + '.png'); mkdirSync(outputDir, { recursive: true }); const distDir = join(projectDir, 'dist'); -if (!existsSync(distDir)) { - console.error('dist/ not found — run npm run build first'); +const baseUrl = process.env.SCREENSHOT_BASE_URL?.replace(/\/$/, ''); +if (!baseUrl && !existsSync(distDir)) { + console.error('dist/ not found, run npm run build first'); process.exit(1); } -function findFreePort() { - return new Promise((resolve, reject) => { - const srv = createServer(); - srv.listen(0, () => { - const port = srv.address().port; - srv.close(() => resolve(port)); - }); - srv.on('error', reject); - }); +const MIME = { + '.html': 'text/html; charset=utf-8', '.css': 'text/css', '.js': 'text/javascript', '.mjs': 'text/javascript', + '.json': 'application/json', '.svg': 'image/svg+xml', '.png': 'image/png', '.jpg': 'image/jpeg', + '.gif': 'image/gif', '.webp': 'image/webp', '.ico': 'image/x-icon', '.woff': 'font/woff', + '.woff2': 'font/woff2', '.txt': 'text/plain', '.xml': 'application/xml', +}; + +function resolveFile(urlPath) { + const rel = normalize(decodeURIComponent(urlPath.split('?')[0])).replace(/^(\.\.[/\\])+/, ''); + let file = join(distDir, rel); + if (existsSync(file) && statSync(file).isDirectory()) file = join(file, 'index.html'); + else if (!existsSync(file) && existsSync(join(file, 'index.html'))) file = join(file, 'index.html'); + return existsSync(file) && statSync(file).isFile() ? file : null; } -let preview; +let server; const killTimer = setTimeout(() => { console.error('Screenshot timed out after 60s'); - if (preview) preview.kill('SIGKILL'); + server?.close(); process.exit(1); }, TIMEOUT_MS); +let browser; try { - const port = await findFreePort(); - const url = `http://localhost:${port}/${slug}/`; - - preview = spawn('npx', ['astro', 'preview', '--port', String(port)], { - cwd: projectDir, - stdio: ['ignore', 'pipe', 'pipe'], - }); - - const start = Date.now(); - while (Date.now() - start < 15_000) { - try { - const res = await fetch(url); - if (res.ok) break; - } catch {} - await new Promise(r => setTimeout(r, 300)); + let origin = baseUrl; + if (!origin) { + server = createServer((req, res) => { + const file = resolveFile(req.url); + if (!file) { res.writeHead(404); res.end('not found'); return; } + res.writeHead(200, { 'Content-Type': MIME[extname(file)] || 'application/octet-stream' }); + createReadStream(file).pipe(res); + }); + await new Promise(resolve => server.listen(0, '127.0.0.1', resolve)); + origin = `http://127.0.0.1:${server.address().port}`; } + const url = `${origin}/${slug}/`; - const browser = await chromium.launch(); + browser = await chromium.launch(); const page = await browser.newPage({ viewport: { width: 1280, height: 900 } }); - await page.goto(url, { waitUntil: 'networkidle', timeout: 15_000 }); + const response = await page.goto(url, { waitUntil: 'networkidle', timeout: 15_000 }); + if (!response || !response.ok()) throw new Error(`${url} returned ${response?.status()}`); await page.waitForTimeout(500); await page.screenshot({ path: outputFile, fullPage: true }); - await browser.close(); console.log(outputFile); } catch (err) { console.error('Screenshot failed:', err.message); - process.exit(1); + process.exitCode = 1; } finally { clearTimeout(killTimer); - if (preview) preview.kill('SIGTERM'); + await browser?.close(); + server?.close(); } diff --git a/scripts/tools-reference-overrides.json b/scripts/tools-reference-overrides.json new file mode 100644 index 0000000..732b66d --- /dev/null +++ b/scripts/tools-reference-overrides.json @@ -0,0 +1,196 @@ +{ + "_readme": [ + "Hand-maintained inputs for scripts/gen-tools-reference.mjs.", + "The script discovers tool names, loading tier, audience availability, feature gates and shell coupling from the product source. This file adds what source can't say in user terms:", + "a one-line summary, the display group, the default approval behaviour and the config keys a reader would reach for.", + "Approval defaults come from Netclaw.Configuration/ToolAudienceProfiles.cs (CreateProfilesForPosture), Netclaw.Actors/Tools/FilePathApprovalMatcher.cs and docs/runbooks/tool-approval-gates.md. Re-read them at each release.", + "A tool that appears in source but not here fails the run, so a new tool can't ship undocumented." + ], + "version": "0.28.2", + + "groups": [ + { "id": "files", "title": "Files", "intro": "File tools respect the audience's `ReadFiles`, `WriteFiles` and `AttachFiles` roots. By default Public reaches only its own session directory, Team adds `GlobalReadRoots` for reads, and Personal reaches any path that isn't protected." }, + { "id": "shell", "title": "Shell and background jobs" }, + { "id": "web", "title": "Web" }, + { "id": "memory", "title": "Memory" }, + { "id": "skills", "title": "Skills and tool discovery" }, + { "id": "reminders", "title": "Reminders" }, + { "id": "webhooks", "title": "Webhooks" }, + { "id": "agents", "title": "Subagents" }, + { "id": "channels", "title": "Chat channels" } + ], + + "tools": { + "file_read": { + "group": "files", "order": 1, + "summary": "Read a text file. Images load for models that accept them, and PDFs, media and archives return metadata.", + "approval": "Runs without a prompt. Protected files such as `secrets.json` and credential folders such as `~/.ssh` are denied." + }, + "file_list": { + "group": "files", "order": 2, + "summary": "List one directory without recursing." + }, + "file_search": { + "group": "files", "order": 3, + "summary": "Search authorized files by name or UTF-8 text, recursively, without a shell." + }, + "tool_output_read": { + "group": "files", "order": 4, + "summary": "Read the rest of a truncated tool result by call ID. Large output is spilled to a session file and this tool pages through it." + }, + "file_write": { + "group": "files", "order": 5, + "summary": "Create or replace a file, creating parent directories.", + "approval": "Runs without a prompt. Writes under the config directory are always denied." + }, + "file_edit": { + "group": "files", "order": 6, + "summary": "Replace literal text in a file, or write the whole file.", + "approval": "Runs without a prompt. Edits under the config directory are always denied." + }, + "attach_file": { + "group": "files", "order": 7, + "summary": "Send an existing file to the user in the chat." + }, + "set_working_directory": { + "group": "files", "order": 8, + "summary": "Declare the project directory for the session. It scopes path checks and shell approvals and loads the project's `AGENTS.md`." + }, + + "shell_execute": { + "group": "shell", "order": 1, + "summary": "Run a shell command. Pass `_background: true` to run it as a background job.", + "approval": "Prompts on Personal unless a saved grant or the reviewed-safe command list covers it. Never available to Team or Public." + }, + "check_background_job": { + "group": "shell", "order": 2, + "summary": "Check the status and output tail of a background job, or cancel it." + }, + + "web_search": { + "group": "web", "order": 1, + "summary": "Search the web and return titles, URLs and snippets.", + "config": ["Search.Backend"] + }, + "web_fetch": { + "group": "web", "order": 2, + "summary": "Download a URL to a local file and return a preview.", + "config": ["Tools.WebFetch.RequireHttps", "Tools.WebFetch.HttpAllowList"] + }, + + "store_memory": { + "group": "memory", "order": 1, + "summary": "Save knowledge to cross-session memory." + }, + "find_memories": { + "group": "memory", "order": 2, + "summary": "Search memory. Returns IDs, titles, scores and snippets." + }, + "get_memories": { + "group": "memory", "order": 3, + "summary": "Load the full text of memories by ID." + }, + "update_memory": { + "group": "memory", "order": 4, + "summary": "Edit or delete a memory by ID." + }, + + "skill_load": { + "publicBlocked": true, + "group": "skills", "order": 1, + "summary": "Load a skill's instructions and list its reference files." + }, + "skill_read_resource": { + "publicBlocked": true, + "group": "skills", "order": 2, + "summary": "Read a resource file from a skill directory." + }, + "skill_manage": { + "group": "skills", "order": 3, + "summary": "Create, edit, patch or delete skills and their resource files." + }, + "search_tools": { + "group": "skills", "order": 4, + "summary": "Search deferred tools by keyword, or list MCP servers. Doesn't load anything." + }, + "load_tool": { + "group": "skills", "order": 5, + "summary": "Load a deferred tool by exact name so the model can call it.", + "config": ["Session.DiscoveredToolRetentionTurns", "Session.DiscoveredToolMaxCount"] + }, + + "set_reminder": { + "group": "reminders", "order": 1, + "summary": "Create or update a reminder from a relative time, an `every 6h` interval or a cron expression." + }, + "list_reminders": { + "group": "reminders", "order": 2, + "summary": "List reminders with schedules, status and next run." + }, + "cancel_reminder": { + "group": "reminders", "order": 3, + "summary": "Cancel a reminder. A one-shot is deleted and its history stays readable for 12 days. A recurring reminder is disabled and kept." + }, + "get_reminder_history": { + "group": "reminders", "order": 4, + "summary": "Show recent runs of a reminder with outcome, duration and session ID." + }, + "run_reminder": { + "group": "reminders", "order": 5, + "summary": "Run a reminder's prompt in the current chat so you can answer approval prompts and save grants for the unattended run." + }, + + "set_webhook": { + "group": "webhooks", "order": 1, + "summary": "Create or update an inbound webhook route.", + "requires": ["`Webhooks.Enabled`"] + }, + "list_webhooks": { + "group": "webhooks", "order": 2, + "summary": "List webhook routes and whether each one loads.", + "requires": ["`Webhooks.Enabled`"] + }, + "delete_webhook": { + "group": "webhooks", "order": 3, + "summary": "Delete a webhook route by name.", + "requires": ["`Webhooks.Enabled`"] + }, + + "spawn_agent": { + "publicBlocked": true, + "group": "agents", "order": 1, + "summary": "Hand a task to a subagent. It returns a run ID at once, the child works in the background, and the result arrives in a later turn as a notice that names its `result.md`.", + "config": ["SubAgents.PrefillTimeoutSeconds", "SubAgents.NoProgressTimeoutSeconds"] + }, + "check_agent_run": { + "group": "agents", "order": 2, + "summary": "Read the status of a child run you started, or ask it to cancel. Subagents can't call it." + }, + + "send_channel_message": { + "group": "channels", "order": 1, + "summary": "Post to a channel or send a direct message on Slack, Discord or Mattermost, using IDs from the lookup tools.", + "requires": ["a chat channel enabled"] + }, + "lookup_channel_user": { + "group": "channels", "order": 2, + "summary": "Resolve a person to a stable user ID on an enabled channel.", + "requires": ["a chat channel enabled"] + }, + "lookup_channel_destination": { + "group": "channels", "order": 3, + "summary": "Resolve a channel name to a stable ID, or list every destination the bot can reach.", + "requires": ["a chat channel enabled"] + }, + "lookup_slack_user": { + "group": "channels", "order": 4, + "summary": "Find a Slack user by name, display name or email.", + "requires": ["`Slack.Enabled`"] + }, + "lookup_mattermost_user": { + "group": "channels", "order": 5, + "summary": "Find a Mattermost user by username or email.", + "requires": ["`Mattermost.Enabled`"] + } + } +} diff --git a/src/content/docs/architecture/context-structure.md b/src/content/docs/architecture/context-structure.md index 40bad3a..2452e0a 100644 --- a/src/content/docs/architecture/context-structure.md +++ b/src/content/docs/architecture/context-structure.md @@ -1,77 +1,77 @@ --- -title: "Context Structure" -description: "How netclaw assembles the model's context each turn — the layers, their order, and how subagents inherit them." +title: "Context structure" +description: "How netclaw assembles the model's context each turn: the layers, their order, and how subagents inherit them." --- -Everything the model sees on a turn is assembled from layers, in a fixed order. Netclaw doesn't hand the model one blob of text — it stacks a persisted system prompt, a frozen block of static context, the conversation history, and a fresh per-turn nudge, then attaches the tool schemas separately on the request. +Everything the model sees on a turn is assembled from layers, in a fixed order. netclaw doesn't hand the model one blob of text. It stacks a persisted system prompt, a block of static context, the conversation history and a fresh per-turn nudge, then attaches the tool schemas separately on the request. -The order is deliberate, and it's built for [prompt caching](https://www.anthropic.com/news/prompt-caching). The stable material sits at the front so the provider can reuse the cached prefix across turns; the volatile, changes-every-turn material sits at the back near the model's generation point. That's why, for example, the current git state is injected late and your persona sits first. +The order is deliberate, and it's built for [prompt caching](https://www.anthropic.com/news/prompt-caching). The stable material sits at the front so the provider can reuse the cached prefix across turns, and the material that changes every turn sits at the back near the model's generation point. That's why the current git state is injected late and your persona sits first. -What each layer contains — and whether it appears at all — depends on the session's [audience](/security/security-model/#trust-audiences). A `Public` turn gets a deliberately thin stack; `Team` and `Personal` get the full one. +What each layer contains, and whether it appears at all, depends on the session's [audience](/security/security-model/#trust-audiences). A `Public` turn gets a deliberately thin stack. `Team` and `Personal` get the full one. ## The layers ```mermaid flowchart TD - subgraph TA["Tier A · Persisted system prompt (cached head)"] - A1["SOUL.md — persona, your name"] --> A2["AGENTS.md — operating rules"] --> A3["TOOLING.md — host capabilities"] --> A4["Project instructions"] - end - subgraph TB2["Tier B · Static context (frozen once at startup)"] - B1["Tool index"] --> B2["Skill index"] --> B3["Memory status"] --> B4["session id + dir"] - end - subgraph TC["Tier C · Conversation history (compacted as it grows)"] - C1["Prior turns"] - end - subgraph TD2["Tier D · Per-turn volatile nudge"] - D1["memory-recall"] --> D2["available-subagents"] --> D3["current-time"] --> D4["working-context — git + recent files"] --> D5["active-background-jobs"] - end - TA --> TB2 --> TC --> TD2 --> MODEL["Model generates the reply"] - TOOLS["Tool schemas — on the request, not prompt text"] -.-> MODEL + A["Tier A, persisted system prompt (cached head): SOUL.md, AGENTS.md, TOOLING.md, project instructions"] + B["Tier B, static context (first call, and again after compaction): tool index, skill index, memory status, session paths"] + C["Tier C, conversation history (compacted as it grows): prior turns, each with its nudge"] + D["Tier D, per-turn nudge (saved into history): memory-recall, skill hint, subagents, time, working-context, background jobs"] + M["Model generates the reply"] + T["Tool schemas, on the request and not in the prompt"] + A --> B --> C --> D --> M + T -.-> M ``` -### Tier A — the persisted system prompt +### Tier A: the persisted system prompt -Built once and stored as the head of the conversation, rebuilt only when its inputs change (byte-identical content is left alone to preserve the cache). Four identity layers, concatenated in order: +Built once and stored as the head of the conversation, then rebuilt only when its inputs change. Byte-identical content is left alone to preserve the cache. Four identity layers, concatenated in order: | Layer | What it is | Source | |-------|-----------|--------| -| `SOUL.md` | Persona, tone, your name | `~/.netclaw/SOUL.md` | -| `AGENTS.md` | Operating rules — the built-in core plus your deployment playbook | embedded in netclaw, plus `~/.netclaw/AGENTS.md` | -| `TOOLING.md` | What the host can actually do | `~/.netclaw/TOOLING.md` | +| `SOUL.md` | Persona, tone, your name | `~/.netclaw/identity/SOUL.md` | +| `AGENTS.md` | Operating rules: the built-in core plus your deployment playbook | embedded in netclaw, plus `~/.netclaw/identity/AGENTS.md` | +| `TOOLING.md` | What the host can actually do | `~/.netclaw/identity/TOOLING.md` | | Project instructions | Repo-specific rules | first of `.netclaw/AGENTS.md`, `CLAUDE.md`, `AGENTS.md`, `CONTEXT.md` in the project directory | -The `AGENTS.md` layer has two halves: an operating core compiled into netclaw (a stripped-down variant on `Public`) and the deployment playbook you write at `~/.netclaw/AGENTS.md`. Project instructions follow the same `AGENTS.md` / `CLAUDE.md` convention other coding tools use, but read from the project directory. +The `AGENTS.md` layer has two halves: an operating core compiled into netclaw (a stripped-down variant on `Public`) and the deployment playbook you write at `~/.netclaw/identity/AGENTS.md`. Project instructions follow the same `AGENTS.md` and `CLAUDE.md` convention other coding tools use, but are read from the project directory. -### Tier B — static context, frozen at startup +### Tier B: static context -Rendered once when the session starts and then held byte-stable, so it never breaks the cache prefix mid-conversation: +Layers that don't change over a session's life. They're rendered on the first model call and again right after a compaction, which resets the context, and otherwise they stay out of the way so they never churn the cached prefix: | Layer | What it is | Empty when | |-------|-----------|------------| -| Tool index | A compressed, audience-filtered catalog of available tools — for awareness, not invocation | — | +| Tool index | A compressed, audience-filtered catalog of available tools, for awareness and not invocation | - | | Skill index | The loaded skill inventory | `Public`, or skills disabled | -| Memory status | Whether the memory subsystem is healthy | `Public`, or memory disabled | -| `[session]` | Session id, plus the session directory for non-`Public` turns | — | -| `[attachment]` hint | How to handle attached files | `file_read` not granted | +| Memory status | Whether the memory subsystem is healthy, and how to use its tools | `Public`, or memory disabled | +| `[session]` | Session id plus five paths: `session_dir`, `temp_dir`, `artifact_dir`, `worktree_dir` and `log_path`, with short guidance on using them | id only on `Public` | +| `[attachments]` hint | How to handle attached files | `file_read` not granted | -The tool index is awareness only. The actual tool **schemas** ride on the request itself (as the provider's tool parameter), resolved and audience-filtered every turn — so a tool has two surfaces: a compressed name in the prompt for discovery, and its full schema on the wire. +The tool index is awareness only. The actual tool **schemas** ride on the request itself (as the provider's tool parameter), resolved and audience-filtered every turn. A tool has two surfaces: a compressed name in the prompt for discovery, and its full schema on the wire. -### Tier C — conversation history +### Tier C: conversation history -The turns so far, verbatim. As it grows past the model's budget it gets [compacted](/architecture/sessions/#compaction) into a summary. The durable working context (below) survives compaction; the raw transcript doesn't. +The turns so far, verbatim. When it grows past the budget it gets [compacted](/architecture/sessions/#compaction): old tool output is cleared, older messages are dropped, and a short block of compaction notes about what was dropped takes their place. The durable working context (below) survives compaction. The raw transcript doesn't. -### Tier D — the per-turn volatile nudge +### Tier D: the per-turn nudge -This is the freshest layer, rebuilt every turn. One structural quirk worth knowing: it isn't a trailing system message. Netclaw injects it as a `[system: …]` block wrapped in a **user-role message, placed just before your latest message** — which keeps it inside the cacheable history prefix and sidesteps the "a fresh user turn just started" confusion some models have with trailing system text. Order within it: +This is the freshest layer, rebuilt every turn. One quirk is worth knowing: it isn't a trailing system message. netclaw wraps it in a `[system: ...]` block inside a **user-role message placed just before your latest message** and saves it into history. That keeps it inside the cacheable prefix on later turns, and it sidesteps the "a fresh user turn just started" confusion some models have with trailing system text. The order inside it: | Layer | What it is | Suppressed on `Public` | |-------|-----------|:----------------------:| -| `[memory-recall]` | Memories recalled for this turn | — | -| Skill hint | A nudge toward a relevant skill | — | +| `[memory-recall]` | Memories recalled for this turn, if any cleared the relevance gate | ✓ | +| Skill hint | A nudge toward a relevant skill | - | | `[available-subagents]` | Subagents this session can spawn | ✓ | -| `[current-time]` | UTC, local, day, timezone | — | -| `[working-context]` | Git state + project dir + recent files | ✓ | -| `[active-background-jobs]` | Running background work | ✓ | +| `[current-time]` | UTC, local, day, timezone | - | +| `[working-context]` | Shell, git state, project dir and recent files | ✓ | +| `[active-background-jobs]` | Running or reaped background work | ✓ | +| Slash-command skill content | The body of a skill invoked with `/name` | - | +| Session prompt overlay | A per-session instruction added on top | - | +| Turn restart notice | Tells the model the daemon restarted and it's resuming | - | +| Interrupted-automation note | Carried by the first model call after a person ends an automation turn: the automated turn stopped unfinished because a person sent a message. Later turns don't carry it | - | + +Most turns carry no `[memory-recall]` block. The relevance gate drops weak matches, and when nothing survives the block is omitted. See the [memory model](/architecture/memory-model/#how-recall-works). ## Audience scoping @@ -80,67 +80,73 @@ This is the freshest layer, rebuilt every turn. One structural quirk worth knowi | | `Public` | `Team` / `Personal` | |---|:---:|:---:| | SOUL.md, AGENTS.md core, tool index, `[current-time]` | ✓ (public AGENTS variant) | ✓ | -| TOOLING.md, project instructions | — | ✓ | -| Skill index, memory, subagents, working-context, background jobs | — | ✓ | -| `[session]` | id only | id + directory | +| TOOLING.md, project instructions | - | ✓ | +| Skill index, memory, subagents, working-context, background jobs | - | ✓ | +| `[session]` | id only | id and paths | ## Git working context -On `Team` and `Personal` turns where a project directory is set, netclaw derives a live snapshot of your git state and drops it into the volatile nudge as a `[working-context]` block. It's how the model knows what branch you're on and what you've been touching without you spelling it out. +On `Team` and `Personal` turns where a project directory is set, netclaw derives a live snapshot of your git state and drops it into the nudge as a `[working-context]` block. It's how the model knows what branch you're on and what you've been touching without you spelling it out. -It's computed fresh at the start of every turn by shelling out to `git` — so `git` has to be on the daemon's PATH — under a hard **2-second budget**. Nothing is persisted; the snapshot is rebuilt each turn, and the durable half (project directory and recent files) is what survives compaction and restarts. +It's computed fresh at the start of every turn by shelling out to `git`, so `git` has to be on the daemon's PATH, under a hard **2-second budget**. Nothing is persisted. The snapshot is rebuilt each turn, and the durable half (project directory and recent files) is what survives compaction and restarts. -What the block actually contains: +What the block contains: -- `project_dir` — set via the `set_working_directory` tool. -- `recent_files` — the last 10 files read or written, most-recent first. -- `git:` — `worktree`, `branch` (or `(detached)`), `head`, and `upstream` / `ahead` / `behind` **only when the branch tracks an upstream**, then **counts** of staged, modified, and untracked files. +- `shell:` on `Personal` turns only: `platform`, `executable`, `grammar` and, for PowerShell, `dialect`. It tells the model which shell syntax to write. +- `project_dir`, set with the `set_working_directory` tool. +- `recent_files`: the last 10 files read or written, most recent first. +- `git:` with `worktree`, `common_dir`, `branch` (or `(detached)`), `head`, and `upstream`, `ahead` and `behind` **only when the branch tracks an upstream**, then **counts** of staged, modified and untracked files. :::caution Two things the block does **not** do, despite how it's easy to read the feature: -- **Changed files appear as counts, not names.** The model sees "3 modified", not which three. The individual paths are tracked internally (for subagent handoff) but never rendered into the prompt. -- **There's no per-turn config.** It's fully automatic — no on/off switch, no tunable limits. The 2-second budget, the 256 KB output ceiling, and the 10-file recent list are all fixed. +- **Changed files appear as counts, not names.** The model sees "3 modified", not which three. The paths are tracked internally for subagent handoff but never rendered into the prompt. +- **There's no per-turn config.** It's fully automatic, with no on/off switch and no tunable limits. The 2-second budget, the 256 KB output ceiling and the 10-file recent list are all fixed. ::: -If git isn't reachable, the run is slow, or the directory isn't a repo, the block degrades to a short `unavailable` note with a reason instead of failing the turn. +If git isn't reachable, the run is slow, or the directory isn't a repo, the block degrades to a short `unavailable` note with a reason (for example `git executable not found`) instead of failing the turn. ## Subagent context -A [subagent](/guides/custom-subagents/) doesn't get a copy of the parent's context. It runs a much thinner stack, assembled fresh for a one-shot task: +A [subagent](/skills/custom-subagents/) doesn't get a copy of the parent's context. It runs a much thinner stack, assembled fresh for a one-shot task: ``` AGENTS.md operating rules + project instructions + the subagent's own markdown body + + a compressed tool index + a headless execution contract ``` +Its first user message carries a `Context: ... Task: ...` pair, the working-context block, and its own `[session]` block. + What a subagent **inherits** from its parent at spawn time: -- The `AGENTS.md` operating rules and project instructions (audience-appropriate) — so a child plays by the same house rules as the parent. -- The project directory, the recent-files list, and the **same working directory** — there's no worktree isolation; a subagent edits the same files the parent would. -- The parent's [audience](/security/security-model/#trust-audiences), and a tool policy filtered to it — **minus `spawn_agent`**, so subagents can't recursively spawn. +- The `AGENTS.md` operating rules and project instructions (audience-appropriate), so a child plays by the same house rules as the parent. +- The project directory, the recent-files list, and the **same working directory**. A subagent edits the same files the parent would. +- The parent's [audience](/security/security-model/#trust-audiences) and a tool policy filtered to it, **minus `spawn_agent`, `check_agent_run` and `attach_file`**, so subagents can't spawn their own. - An initial git snapshot taken at spawn. -What it **doesn't** get: your SOUL.md persona, TOOLING.md, memory recall, the skill or memory indexes, subagent discovery, the `[current-time]` layer, background jobs, or the parent's conversation history. Its entire user message is `Context: … Task: …`. +Each child gets its own `temp_dir`, `artifact_dir` and `log_path`. On Linux and macOS the child's `temp_dir` is `/tmp/netclaw-//c<8 hex>`, and the other two sit under the parent's session folder. A successful `spawn_agent` call returns only the run id and `"state":"accepted"`. When the child ends, a notice names its `result.md`, which lists the log path and the artifact directory, and the parent reads what the child produced with `file_read`. + +What it **doesn't** get: your SOUL.md persona, TOOLING.md, memory recall, the skill and memory indexes, subagent discovery, `[current-time]`, background jobs, or the parent's conversation history. -When a subagent finishes, it hands back a working-context delta. The parent merges **only the files a successful child confirmed it wrote or edited** into its own recent-files list — a failed or cancelled run merges nothing, and files the child only read aren't merged. This is an in-memory recent-files handoff, not a git merge: since parent and child share one working directory, the edits are already on disk. The merge just keeps the parent's sense of "what we've been touching" current. +When a subagent finishes, it hands back a working-context delta. The parent merges **only the files a successful child confirmed it wrote or edited** into its own recent-files list. A failed or cancelled run merges nothing, and files the child only read aren't merged. This is an in-memory recent-files handoff and not a git merge: since parent and child share one working directory, the edits are already on disk. The merge just keeps the parent's sense of "what we've been touching" current. ## Limitations -- The git working context needs `git` on the daemon's PATH and a project directory set; without both, the `[working-context]` block is absent, not an error. -- Changed-file **names** are never exposed to the model — only counts, and only the recent-files list carries specific paths. -- None of the layer sizes are configurable. Recent files cap at 10, git inspection at a 2-second budget. -- Subagents share the parent's working directory. There is no filesystem sandbox between a parent and its subagent beyond the audience's tool policy. +- The git working context needs a project directory. Without `git` on the daemon's PATH the block is present with an `unavailable` note, and without a project directory (on `Team`) it's absent. +- Changed-file **names** are never exposed to the model. Only counts, and the recent-files list, carry paths. +- None of the layer sizes are configurable. Recent files cap at 10 and git inspection at a 2-second budget. +- Subagents share the parent's working directory. Nothing isolates a parent from its subagent beyond the audience's tool policy. Every session also has a managed `worktree_dir`, and the agent can create a git worktree there with `git worktree add` and switch into it with `set_working_directory`, but that's something it chooses to do, not a default. ## Related pages -- [Sessions & Input Model](/architecture/sessions/) — the turn lifecycle these layers feed, and compaction -- [Memory Model](/architecture/memory-model/) — how the memory layers (recall, index) are formed and scoped -- [Custom Subagents](/guides/custom-subagents/) — defining the subagents whose context this page describes -- [Security Model](/security/security-model/#trust-audiences) — the audiences that gate each layer +- [Sessions and input model](/architecture/sessions/) - the turn lifecycle these layers feed, and compaction +- [Memory model](/architecture/memory-model/) - how the memory layers (recall, index) are formed and scoped +- [Custom subagents](/skills/custom-subagents/) - defining the subagents whose context this page describes +- [Security model](/security/security-model/#trust-audiences) - the audiences that gate each layer ## Resources -- [Prompt caching (Anthropic)](https://www.anthropic.com/news/prompt-caching) — why the stable-prefix ordering matters -- [AGENTS.md](https://agents.md/) — the operating-rules file convention netclaw follows for project instructions +- [Prompt caching (Anthropic)](https://www.anthropic.com/news/prompt-caching) - why the stable-prefix ordering matters +- [AGENTS.md](https://agents.md/) - the operating-rules file convention netclaw follows for project instructions diff --git a/src/content/docs/architecture/design-philosophy.md b/src/content/docs/architecture/design-philosophy.md index cda913f..e7d1746 100644 --- a/src/content/docs/architecture/design-philosophy.md +++ b/src/content/docs/architecture/design-philosophy.md @@ -1,85 +1,89 @@ --- -title: "Design Philosophy" +title: "Design philosophy" description: "Why netclaw exists, what we believe about agents, and the tradeoffs we chose." --- -Netclaw exists because we disagreed with the direction every other open-source agent project was heading. [OpenClaw](https://openclaw.ai/) (the most popular open-source personal AI assistant, 360K+ GitHub stars, dozens of platform integrations) and [Hermes Agent](https://hermes-agent.nousresearch.com/) (Nous Research's autonomous agent with a self-improving skill system) proved the category and inspired this project. We respect the hell out of what they've built. But we think their philosophies are bad for their projects long-term, and that's why we built something new. +We built netclaw because we disagreed with the direction every other open-source agent project was heading. [OpenClaw](https://openclaw.ai/) (the most popular open-source personal AI assistant, 360K+ GitHub stars, dozens of platform integrations) and [Hermes Agent](https://hermes-agent.nousresearch.com/) (Nous Research's autonomous agent with a self-improving skill system) proved the category and inspired this project. We respect the hell out of what they've built. But we think their philosophies are bad for their projects long-term, and that's why we built something new. ## Simplicity is a feature -[Petabridge](https://petabridge.com/) has maintained [Akka.NET](https://getakka.net/) since 2014. Over a decade of shepherding a large open-source framework through breaking changes, security patches, and community demands. The lesson: **the biggest threat to any project is its own complexity.** +[Petabridge](https://petabridge.com/) has maintained [Akka.NET](https://getakka.net/) since 2014. A decade of shepherding a large open-source framework through breaking changes, security patches, and community demands taught us one thing: **the biggest threat to any project is its own complexity.** -Every feature you add is a feature you have to maintain, test, secure, and explain. The most productive thing a maintainer can do is say no. Most agent projects treat feature count and velocity as proof of progress — hundreds of PRs per release, thousands of community integrations, autonomous skill lifecycle management. We think that's antithetical to how you should build software. +Every feature you add is a feature you have to maintain, test, secure, and explain. The most productive thing a maintainer can do is say no. Most agent projects treat feature count and velocity as proof of progress: hundreds of PRs per release, thousands of community integrations, autonomous skill lifecycle management. We think that's antithetical to how you should build software. -Netclaw's codebase is deliberately small. So is the configuration footprint. When we're tempted to add a feature, the first question is always: can this live in an MCP server someone else maintains instead? +The codebase is deliberately small. So is the configuration footprint. When we're tempted to add a feature, the first question is always: can this live in an MCP server someone else maintains instead? ### What we don't build -You'll probably never see voice mode in netclaw. That's a property of the model you're using and something your operating system will handle natively. We don't build web dashboards for daily interaction — your chat app already has auth, threading, mobile apps, notifications, and search. We don't build an autonomous skill lifecycle that generates, archives, and self-improves skills without human oversight. +You'll probably never see voice mode in netclaw. That's a property of the model you're using and something your operating system will handle natively. We don't build web dashboards for daily interaction. Your chat app already has auth, threading, mobile apps, notifications, and search. We don't build an autonomous skill lifecycle that generates, scores, archives, and rewrites skills without human oversight. The agent has a tool to create and edit skills, but nothing in netclaw curates them behind your back. -Every feature we don't build is a feature that can't break, can't have a CVE, and can't confuse your deployment. +A feature we never build can't break, can't have a CVE, and can't confuse your deployment. ## Chat-first interaction -Your chat app is the interface. Slack, Discord, Mattermost — whatever your team already uses. Every thread you spin up with the agent is its own session: reentrant (pick up where you left off), persisted to disk, and recoverable across daemon restarts. The agent doesn't need its own UI because it lives where your team already works. +Your chat app is the interface. Slack, Discord, Mattermost, whatever your team already uses. Every thread you spin up with the agent is its own session: reentrant (pick up where you left off), persisted to disk, and recoverable across daemon restarts. The agent doesn't need its own UI because it lives where your team already works. -At the session layer, the source of a message is irrelevant. Slack, a webhook, a scheduled timer, a CLI chat — they all produce the same kind of message routed to the same kind of [actor](https://en.wikipedia.org/wiki/Actor_model). The [architecture overview](/architecture/overview/#everything-is-just-input) covers how this works mechanically, but the philosophy is simple: the agent shouldn't care how you talk to it, and you shouldn't have to learn a new interface to use it. +At the session layer, the source of a message is irrelevant. Slack, a webhook, a scheduled timer, a CLI chat: they all produce the same kind of message routed to the same kind of [actor](https://en.wikipedia.org/wiki/Actor_model). The [architecture overview](/architecture/overview/#everything-is-just-input) covers how this works mechanically, but the philosophy is simple. The agent shouldn't care how you talk to it, and you shouldn't have to learn a new interface to use it. -This means adding a new channel — Mattermost, Signal, a web UI for remote management — never touches session logic. The codebase stays small because the transport layer is someone else's problem. +This means adding a new channel, whether Signal or a web UI for remote management, never touches session logic. The codebase stays small because the transport layer is someone else's problem. ## Default-deny security -Giving an LLM shell access and tool-calling ability is giving it the keys to your infrastructure. The question isn't whether something will go wrong — it's what happens when it does. +Giving an LLM shell access and tool-calling ability is giving it the keys to your infrastructure. The question isn't whether something will go wrong, it's what happens when it does. -Most agent frameworks start permissive and add restrictions after security incidents. Netclaw works the other way around. During [`netclaw init`](/cli/init/), you choose an [audience disposition](/architecture/security-model/#audience-as-the-trust-primitive) — Personal, Team, or Public — and that choice sets the baseline for what the agent can do. Some common operations are allowed out of the box, but anything potentially destructive requires explicit approval. You can approve a tool call for the current session or persistently, and the agent makes the distinction clear with a structured approval prompt in the TUI. +Most agent frameworks start permissive and add restrictions after security incidents. This project works the other way around. During [`netclaw init`](/cli/init/), you choose an [audience](/architecture/security-model/#audience-as-the-trust-primitive) (Personal, Team, or Public) and that choice sets the baseline for what the agent can do. Some common operations are allowed by default, but anything potentially destructive requires explicit approval. You can approve a tool call for the current session or persistently, and the agent makes the distinction clear with a structured approval prompt in the TUI or in your chat channel. -![Security posture selection during netclaw init](/screenshots/output/init-02-security-posture.png) +![Netclaw Setup, Step 3 of 5, Security Posture, offering Personal, Team and Public](/screenshots/output/init-02-security-posture.png) + +The choice is made once, in the third step of setup. This is the [principle of least privilege](https://csrc.nist.gov/glossary/term/least_privilege) applied to an LLM agent. -When something fails or is misconfigured, it fails loudly. Fallbacks hide bugs, and on security-relevant paths they can silently escalate privileges. Inbound message checks run in the [gateway boundary](/architecture/overview/#three-boundaries) before messages ever reach the session layer. And the security model stacks four independent layers — hard denies, path restrictions, audience-scoped grants, and human approval gates — so no single check is the only thing between the agent and a destructive action. The [security architecture](/architecture/security-model/) covers the full stack. +When something fails or is misconfigured, it fails loudly. Fallbacks hide bugs, and on security-relevant paths they can silently escalate privileges. Inbound message checks run in the [gateway boundary](/architecture/overview/#three-boundaries) before messages ever reach the session layer. The security model stacks four independent layers (hard denies, path restrictions, audience-scoped grants, and human approval gates) so no single check is the only thing between the agent and a destructive action. The [security architecture](/architecture/security-model/) covers the full stack. ### Determinism boundaries Traditional service architecture thinks in terms of service boundaries. Agent architecture needs to think in terms of **determinism boundaries**: where is it safe for an LLM to operate non-deterministically, and where isn't it? -LLMs are inherently non-deterministic — same input, different output, every time. That's fine for research, summarization, and conversation. It's not fine for `git push --force` or deleting a database. +LLMs are inherently non-deterministic: same input, different output, every time. That's fine for research, summarization, and conversation. It's not fine for `git push --force` or deleting a database. -The operating principle is **reversibility**: if the agent is about to perform an action that can't be undone, that action needs a deterministic gate — an [approval prompt](/architecture/security-model/#human-in-the-loop-as-a-layer-not-a-crutch) where a human confirms intent. If you want to turn those gates off, that's your decision. You live with the consequences. +The operating principle is **reversibility**. If the agent is about to perform an action that can't be undone, that action needs a deterministic gate, an [approval prompt](/architecture/security-model/#human-in-the-loop-as-a-layer-not-a-crutch) where a human confirms intent. If you want to [turn those gates off](/guides/reduce-approval-prompts/), that's your decision. You live with the consequences. ## Lightweight access control without an identity provider -The real-world use case that drove this: a bot in a team Slack channel that can answer questions about live service status for anyone, but only operators — the IT folks — can trigger high-privilege actions like exporting log dumps or restarting services. +The real-world use case that drove this: a bot in a team Slack channel that can answer questions about live service status for anyone, but only operators (the IT folks) can trigger high-privilege actions like exporting log dumps or restarting services. -Traditional role-based access control solves this, but it requires a fully managed identity provider — something most self-hosted deployments don't have and shouldn't need. To keep netclaw simple and self-hostable, we distilled two lightweight primitives that are adjacent to RBAC without the infrastructure dependency: +Traditional role-based access control solves this, but it requires a fully managed identity provider, something most self-hosted deployments don't have and shouldn't need. To keep netclaw simple and self-hostable, we distilled two lightweight primitives that are adjacent to RBAC without the infrastructure dependency: -1. **Audience dispositions** classify trust by deployment topology and channel type. Personal (you at your machine), Team (your Slack workspace), or Public (untrusted sources). The disposition determines what tools are visible, what memory is accessible, and whether approval gates are active. +1. **Audience dispositions** classify trust by deployment topology and channel type. Personal (you at your machine), Team (your Slack, Discord or Mattermost workspace), or Public (untrusted sources). The disposition determines what tools are visible, what memory is accessible, and whether approval gates are active. 2. **Listing owners** are the specific user IDs in each channel who actually operate the bot. They get elevated trust within their audience tier. | Audience | Trust | Typical source | |----------|-------|----------------| | **Personal** | Full access | Local TUI, operator at the machine | -| **Team** | Scoped access | Slack workspace, Discord server | +| **Team** | Scoped access | Slack, Discord or Mattermost workspace | | **Public** | Minimal access | Unknown or untrusted channels | -During [`netclaw init`](/cli/init/), you assign each connected channel to an audience tier. The same Slack workspace can have DMs classified as Personal and a shared ops channel classified as Team: +In [`netclaw config`](/cli/config/), you assign each connected channel to an audience tier. The same Slack workspace can have DMs classified as Personal and a shared ops channel classified as Team: + +![netclaw config Channels & Permissions screen for Mattermost, listing the Assistant and Ops Room channels as Team, Lunch as Public and direct messages as Personal](/screenshots/output/config-channel-audiences.png) -![Per-channel audience configuration during netclaw init](/screenshots/output/init-04-channel-audiences.png) +The screen is Mattermost's, and Slack and Discord use the same layout. Left and right set a channel's audience, and a public lunch channel can sit next to a team ops room. -This isn't a better idea than RBAC. It's a pragmatic simplification that lets you deploy an agent with meaningful access control today, on your own hardware, without standing up Entra or Okta first. Every other security decision flows from the audience disposition. See [audience scoping](/architecture/security-model/#audience-as-the-trust-primitive) for how it works in practice. +We're not claiming this beats RBAC. It's a pragmatic simplification that lets you deploy an agent with meaningful access control today, on your own hardware, without standing up Entra or Okta first. Every other security decision flows from the audience. See [audience scoping](/architecture/security-model/#audience-as-the-trust-primitive) for how it works in practice. ## MCP-first tooling -LLMs calling command-line tools is powerful and composable — especially when those tools are in the model's training data or expose their own help documentation. But shell access is where all the security problems come from. Agents deleting production databases, sending unauthorized emails, running destructive commands — that's shell territory. +LLMs calling command-line tools is powerful and composable, especially when those tools are in the model's training data or expose their own help documentation. But shell access is where the security problems come from. Agents deleting production databases, sending unauthorized emails, running destructive commands: that's shell territory. -For operators and developers, shell access makes sense. Netclaw supports it with strong [approval gates](/architecture/security-model/#human-in-the-loop-as-a-layer-not-a-crutch) that extract verb-chains from commands (so approving `git push` covers `git push origin main`) and make shell access as safe as possible. +For operators and developers, shell access makes sense. netclaw supports it with strong [approval gates](/architecture/security-model/#human-in-the-loop-as-a-layer-not-a-crutch) that extract verb-chains from commands (so approving `git push` covers `git push origin main`) and make shell access as safe as possible. -But for everyone else in an organization? Shell access is a non-starter. The accounting team isn't going to install the Google Cloud CLI on a sandboxed VM over SSH. That's unrealistic. +But for everyone else in an organization? Shell access is a non-starter. The accounting team isn't going to install the Google Cloud CLI on a sandboxed VM over SSH, and shouldn't have to. -[MCP](https://modelcontextprotocol.io/) solves this. It lets you deploy agents that connect to remote tool servers with scoped permissions — read-only access to your CRM, delegated service accounts for cloud resources, controlled access to internal APIs. The agent gets purpose-built tools without shell exposure, and every tool call goes through the same [approval gates](/architecture/security-model/#mcp-security-two-gate-model) as shell commands. +[MCP](https://modelcontextprotocol.io/) solves this. It lets you deploy agents that connect to remote tool servers with scoped permissions: read-only access to your CRM, delegated service accounts for cloud resources, controlled access to internal APIs. The agent gets purpose-built tools without shell exposure, and every tool call goes through the same [approval gates](/architecture/security-model/#mcp-security-two-gate-model) as shell commands. -What makes MCP work at scale is [progressive tool disclosure](/architecture/overview/#tool-system). The agent sees server-level summaries — "browser tools," "memory tools," "CRM tools" — and loads specific tools on demand. This keeps the prompt small and prevents the model from misfiring on similarly-named tools across servers. +What makes MCP work at scale is [progressive tool disclosure](/architecture/overview/#tool-system). The agent sees server-level summaries (browser tools, CRM tools, and so on) and loads specific tools on demand. This keeps the prompt small and prevents the model from misfiring on similarly named tools across servers. If an MCP server definition changes and new tools appear, operators can set the default approval status for unknown tools: auto-deny, auto-approve, or require human confirmation. MCP provides a deterministic security boundary that shell calling fundamentally cannot. @@ -91,44 +95,46 @@ Most agent platforms assume frontier models. OpenClaw's documentation recommends We think that's the wrong default. If your agent is working deals in a CRM, responding to GitHub webhooks, generating invoices, or triaging support tickets, you don't need a frontier model at $10 per million input tokens. -Netclaw runs production workloads on [Qwen 3.6 27B](https://huggingface.co/Qwen/Qwen3.6-27B). We've tested with models as small as Qwen 3.5 9B and gotten reliable tool-calling results. The architecture is designed to make this work: +netclaw runs production workloads on [Qwen 3.6 27B](https://huggingface.co/Qwen/Qwen3.6-27B). We've tested with models as small as Qwen 3.5 9B and gotten reliable tool-calling results. The architecture is designed to make this work: -Progressive tool disclosure reduces the noise in the prompt — what the model has to sort through to pick the right tool — so smaller models can navigate tool graphs without misfiring. Compressed skill and tool indexes keep the system prompt lean, and server-first MCP discovery means the model picks "which server" before it ever sees individual tool definitions. Much simpler decision space. +Progressive tool disclosure cuts the noise in the prompt, so smaller models can navigate tool graphs without misfiring. Compressed skill and tool indexes keep the system prompt lean, and server-first MCP discovery means the model picks "which server" before it ever sees individual tool definitions. That's a much simpler decision space. -There's an economic argument here too. A $3,000 GPU with 64GB VRAM runs local inference indefinitely — bounded capital expenditure instead of unbounded API billing. Local inference also means your data never leaves your infrastructure, which matters when the agent handles contracts, financials, or internal communications. +There's an economic argument here too. A $3,000 GPU with 64GB VRAM runs local inference indefinitely: bounded capital expenditure instead of unbounded API billing. Local inference also means your data never leaves your infrastructure, which matters when the agent handles contracts, financials, or internal communications. [Self-hosted providers](/configuration/self-hosted-providers/) covers the setup. -For tasks that genuinely need frontier capabilities — complex agentic coding, multi-step research across large codebases — netclaw delegates to tools like [Claude Code](https://docs.anthropic.com/en/docs/agents-and-tools/claude-code/overview) or [OpenCode](https://opencode.ai/) in headless mode via skills. Use your existing subscriptions for frontier work. Use local models for everything else. +For tasks that genuinely need frontier capabilities, like complex agentic coding or multi-step research across large codebases, netclaw delegates to tools like [Claude Code](https://docs.anthropic.com/en/docs/agents-and-tools/claude-code/overview) or [OpenCode](https://opencode.ai/) in headless mode via skills. Use your existing subscriptions for frontier work. Use local models for everything else. ## Automatic memory formation Agents that forget everything between conversations aren't useful for real work. But agents that require users to manually save context don't get used either. -Netclaw forms memories automatically. A per-session observation sidecar watches the conversation and, when the session goes idle, distills key facts, decisions, and project context into durable memory — without an extra LLM call during your turn. A background curation pipeline validates proposals, deduplicates against existing memories, blocks secret content, and persists what survives to SQLite. +netclaw forms memories automatically. A per-session memory observer watches the conversation and, when the session goes idle or every few turns, distills key facts, decisions, and project context into durable memory. That's a background call that never sits inside your turn. A curation pipeline validates proposals, deduplicates against existing memories, blocks secret content, and persists what survives to SQLite. + +Before each turn, in any session, netclaw recalls the memories that bear on what you just said. It combines full-text search with semantic search that runs on your own CPU, on two small ONNX models (on by default since 0.27.0). A relevance gate throws out weak matches, so most turns inject nothing, and what does get through arrives as context the model sees before it starts generating, not as a tool call. There's no external vector database, no cloud embedding service, and no third party in the recall path. The one network touch is a model download from huggingface.co on first start, and you can [pre-provision the files](/configuration/memory/#run-offline) to skip it. The agent remembers that your project uses PostgreSQL, that you prefer Terraform over Pulumi, that the deploy target changed last Tuesday, because it observed those facts and stored them. -On the next turn — in any session — relevant memories are recalled automatically via deterministic full-text search and injected into the system prompt. No vector database, no embeddings, no external service. The agent remembers that your project uses PostgreSQL, that you prefer Terraform over Pulumi, that the deploy target changed last Tuesday — because it observed those facts and stored them. +The cost is RAM. With embeddings on, the daemon sits around 480 MB at idle instead of about 235 MB, and you can [turn them off](/configuration/memory/#turn-embeddings-off) if that matters more to you than semantic recall. -This is the kind of feature that belongs in core, not in a plugin. Memory touches the session lifecycle, the compaction pipeline, the security model (public sessions get no memory access), and the system prompt assembly. Pushing it to the edges would mean every integration reimplements it badly. The [memory architecture](/architecture/memory-model/) covers the full system. +This is the kind of feature that belongs in core, not in a plugin. Memory touches the session lifecycle, the compaction pipeline, the security model (public sessions get no memory access), and context assembly. Pushing it to the edges would mean every integration reimplements it badly. The [memory architecture](/architecture/memory-model/) covers the full system. ## The philosophy of no Open-source maintainers tend to say yes too much. Every accepted contribution is a maintenance commitment. Every feature that lands in core is a feature that can't be pushed to a plugin, an MCP server, or a skill that someone else maintains. -Netclaw takes the opposite approach: functionality belongs at the edges unless there's a strong reason to centralize it. Want voice mode? That's a model capability and an OS integration — not an agent framework feature. Want a custom skill marketplace? Run your own [skill feed](/skills/skill-feeds/) or [skill server](/skills/skill-server/). Want a specialized tool for your workflow? Build an MCP server and connect it. +netclaw takes the opposite approach: functionality belongs at the edges unless there's a strong reason to centralize it. Want voice mode? That's a model capability and an OS integration, not an agent framework feature. Want a custom skill marketplace? Run your own [skill feed](/skills/skill-feeds/) or [skill server](/skills/skill-server/). Want a specialized tool for your workflow? Build an MCP server and connect it. The core stays small, and so does the surface area for bugs, CVEs, and breaking changes. When your competitors' complexity is the source of their security vulnerabilities, your simplicity is your security story. -## Keep reading +## Related pages -- [Architecture Overview](/architecture/overview/) — how these principles become a running system -- [Security Architecture](/architecture/security-model/) — the four-layer security model in detail -- [Quickstart](/getting-started/quickstart/) — try it yourself -- [MCP Tool Permissions](/guides/mcp-tool-permissions/) — practical guide to scoping MCP access -- [Self-Hosted Providers](/configuration/self-hosted-providers/) — setting up local model inference +- [Architecture overview](/architecture/overview/) - how these principles become a running system +- [Security architecture](/architecture/security-model/) - the four-layer security model in detail +- [Memory model](/architecture/memory-model/) - the full memory system behind the section above +- [Quickstart](/getting-started/quickstart/) - try it yourself +- [MCP tool permissions](/guides/mcp-tool-permissions/) - practical guide to scoping MCP access ## Resources -- [Actor model](https://en.wikipedia.org/wiki/Actor_model) — the concurrency model behind netclaw's session architecture -- [Akka.NET](https://getakka.net/) — the actor framework that implements it, maintained by Petabridge since 2014 -- [Model Context Protocol specification](https://modelcontextprotocol.io/) — the open standard for agent tool integration -- [Principle of Least Privilege (NIST)](https://csrc.nist.gov/glossary/term/least_privilege) — the access control philosophy behind default-deny -- [OWASP LLM Top 10](https://genai.owasp.org/llm-top-10/) — the attack taxonomy netclaw's security model is designed against +- [Actor model](https://en.wikipedia.org/wiki/Actor_model) - the concurrency model behind netclaw's session architecture +- [Akka.NET](https://getakka.net/) - the actor framework that implements it, maintained by Petabridge since 2014 +- [Model Context Protocol specification](https://modelcontextprotocol.io/) - the open standard for agent tool integration +- [Principle of Least Privilege (NIST)](https://csrc.nist.gov/glossary/term/least_privilege) - the access control philosophy behind default-deny +- [OWASP LLM Top 10](https://genai.owasp.org/llm-top-10/) - the attack taxonomy netclaw's security model is designed against diff --git a/src/content/docs/architecture/memory-model.md b/src/content/docs/architecture/memory-model.md index 7e43c86..8151279 100644 --- a/src/content/docs/architecture/memory-model.md +++ b/src/content/docs/architecture/memory-model.md @@ -1,9 +1,37 @@ --- -title: "Memory Model" -description: "How netclaw forms, stores, and recalls memories across sessions without manual effort." +title: "Memory model" +description: "How netclaw forms, curates and recalls memories across sessions, with local semantic search on by default." --- -Agents that forget everything between conversations aren't useful for real work. Agents that require users to manually tag and save context don't get used. Netclaw's memory system sits between these extremes: it forms memories automatically from conversation, stores them durably in SQLite, and recalls relevant ones before each turn — no manual effort, no external services. If you're not familiar with how [sessions](/architecture/sessions/) work, start there. +Agents that forget everything between conversations aren't useful for real work, and agents that make you tag and save context don't get used. netclaw sits between the two. A background memory observer watches each conversation, a curation pipeline decides what's worth keeping, and before every turn a recall step injects the few memories that actually bear on what you just said. Everything lives in one SQLite file, and since 0.27.0 the semantic search runs on your own CPU with two small ONNX models. If you're not familiar with how [sessions](/architecture/sessions/) work, start there. + +The mental model is two paths that share one store. The write path is slow, asynchronous and uses an LLM sparingly. The read path is fast, local and gated hard, so most turns inject nothing at all. + +Writing happens in the background. The memory observer's LLM call fires after 90 seconds idle, every 5 turns, and at passivation. + +```mermaid +flowchart TD + T["Session transcript"] --> O["Memory observer
(LLM call)"] + O --> Q[("Checkpoint queue")] + X["store_memory tool"] --> Q + K["Compaction notes"] --> Q + Q --> C["Curate and embed
locally"] + C --> S[("Memory store")] + C -->|"near-duplicate
or ambiguous"| L["Curation model call"] + L --> S +``` + +Reading happens on every turn, inside a 300 ms budget: + +```mermaid +flowchart TD + M["Your message"] --> F["Full-text
candidates"] + M --> V["Embed query,
top 50 by cosine"] + F --> R["Fuse, recency decay,
similarity floor"] + V --> R + R --> G["Cross-encoder
relevance gate"] + G --> I["Up to 3 memories
into the turn"] +``` ## What memory looks like in practice @@ -21,15 +49,15 @@ netclaw: Here are some United flights from Houston Intercontinental to Amsterdam departing October 1–7... ``` -After this conversation ends, the memory sidecar (a companion actor running alongside the session) observes two things worth remembering: +Once the conversation goes quiet, the memory observer (a companion actor running alongside the session) proposes three things worth keeping: | What it stores | Memory class | Why | |----------------|-------------|-----| -| You're a United MileagePlus member | **Durable fact** | Airline preference — useful indefinitely | -| Your usual origin airport is IAH | **Durable fact** | Travel pattern — useful indefinitely | -| You need a flight to Amsterdam in early October | **Evidence** | Specific task — expires after 30 days | +| You're a United MileagePlus member | Durable fact | Airline preference, useful indefinitely | +| Your usual origin airport is IAH | Durable fact | Travel pattern, useful indefinitely | +| You need a flight to Amsterdam in early October | Evidence | Specific task, expires after 30 days | -The agent didn't ask you to save any of this. It observed, classified, and stored it. +You didn't ask it to save any of this. ### Conversation 2: A week later @@ -41,11 +69,11 @@ netclaw: Here are some United flights from IAH to Boston departing award availability too? ``` -The agent recalled your airline preference and home airport from durable memory. You didn't repeat yourself. The Amsterdam trip evidence is still in memory too, but it's a different task — the agent doesn't confuse the two. +The agent recalled your airline and home airport from durable memory without a tool call. The Amsterdam evidence is still stored, but it's a different task, so it didn't clear the relevance gate for a Boston question. ### Living documents: memory that updates over time -Not all memories are write-once. If you ask the agent to track RAM prices every week, it doesn't create a new memory each time. It updates the same document — appending the latest prices alongside the historical record: +Not all memories are write-once. If you ask the agent to track RAM prices every week, it doesn't create a new memory each time. It updates the same document and keeps the history: ``` You: Check DDR5 64GB kit prices on Newegg and update your @@ -56,133 +84,136 @@ netclaw: Updated the DDR5 price tracker. Current best: $142.99 data point to the tracking document. ``` -The memory system supports two update patterns: **merge-document** (revise and consolidate, like the price tracker) and **immutable-record** (append-only evidence — "completed the database migration on May 2nd" is a historical fact that shouldn't be revised, just recorded). The LLM picks the pattern based on the content. - -## How it works - -Memory formation happens outside the conversation turn — the user never waits for it. - -```text -┌────────────────────────┐ -│ Session Actor │◂──────────┐ -└───────────┬────────────┘ │ - │ │ - ▾ transcript stream │ -┌────────────────────────┐ │ -│ Observation Sidecar │ │ -└───────────┬────────────┘ │ - │ │ - ▾ idle timeout │ -┌────────────────────────┐ │ -│ Sidecar LLM Call │ │ -└───────────┬────────────┘ │ - │ │ - ▾ memory proposals │ -┌────────────────────────┐ │ -│ Checkpoint Queue │ (SQLite) │ -└───────────┬────────────┘ │ - │ │ - ▾ │ -┌────────────────────────┐ │ -│ Curation Pipeline │ │ -└───────────┬────────────┘ │ - │ │ - ▾ validated memories │ -┌────────────────────────┐ │ -│ Memory Store (SQLite) │───────────┘ -└────────────────────────┘ auto-recall -``` +Memory has two update patterns. **Merge-document** memories get revised and consolidated, like the price tracker. **Immutable-record** memories are append-only evidence, like "completed the database migration on May 2nd". The LLM picks the pattern from the content. -**Observation sidecar.** Each session has a companion [actor](/architecture/sessions/#session-identity) that accumulates the conversation transcript — user messages, assistant replies, tool calls. When the session goes idle, the sidecar makes a separate LLM call (not part of the conversation) to distill observations into memory proposals. +## How memories form -**Checkpoint queue.** Proposals land in a durable SQLite table (`memory_checkpoints`), not an in-memory queue. If the daemon crashes between observation and curation, pending proposals survive the restart. A background worker continuously leases and processes pending checkpoints with retry logic (up to 5 attempts per proposal). +Formation happens outside your turn, so you never wait for it. -**Curation pipeline.** The background worker applies rules-first extraction (pattern-matching for project facts, milestones, preferences), deduplicates against existing memories via fingerprinting, blocks anything that looks like a secret or credential, and persists what survives. +**Memory observer.** Each session has a companion actor that accumulates the transcript: your messages, assistant replies, tool call names, recalled memories and loaded skills. It makes a separate LLM call to distill that transcript into memory proposals when the session has been idle for 90 seconds (`Session.MemoryObserverIdleSeconds`), after every 5 completed turns (`Session.Tuning.MemoryDistillationTurnInterval`), and once more when an idle session shuts down. Information already shown as a recalled memory is excluded from new proposals. -**Automatic recall.** Before each user turn, a deterministic retrieval coordinator runs a lightweight [retrieval-augmented generation](https://en.wikipedia.org/wiki/Retrieval-augmented_generation) (RAG) pipeline. It tokenizes the user's message and recent context, builds a query plan, searches the memory store via full-text search (SQLite FTS5), and scores candidates using a composite of lexical match strength and memory class rank. The top results are injected directly into the system prompt — not as a tool call, but as context the model sees before it starts generating. +**Other entry points.** The agent's `store_memory` tool queues a proposal directly. After a [compaction](/architecture/sessions/#compaction), the session also queues a checkpoint carrying the compaction summary. Subagents can't write memory themselves. They return structured findings to the parent session, which reviews them and queues only the accepted ones. -That's why the agent already knows your airline preference in Conversation 2. It didn't call a tool — the retrieval coordinator found the memory, scored it as relevant, and injected it into the system prompt. +**Checkpoint queue.** Proposals land in the `memory_checkpoints` table, not in an in-memory queue. A background worker leases them, retries up to 5 times, and survives a daemon crash between observation and curation. -Retrieval is deterministic by design — no embeddings, no neural ranking, no non-determinism in the recall path. The same query against the same memory store produces the same results every time. Durable facts are heavily prioritized for auto-recall; evidence can also surface when it scores highly enough, but traces are excluded entirely. Recall runs on a 300ms budget so it never delays the conversation. +**Curation.** The worker applies rules first: pattern matching for project facts, milestones and preferences, fingerprint deduplication, and a block on anything that looks like a secret or credential. Then it embeds the proposal and looks up existing memories with similarity above 0.86 (up to 5). Similarity only nominates candidates. It never merges anything by itself. When there's a nominee, or the rules can't decide, a curation call to your Main model picks create, update, consolidate or skip. A merged body is validated against its sources before it's written, and if validation fails the new text is appended instead, so nothing gets dropped. A proposal that matches nothing already stored is created without an LLM call. -## Memory classes +Put together, the memory-related calls that reach your model provider are the memory observer call per trigger, the curation call when a near-duplicate turns up or the rules can't decide, and the compaction notes call. Embedding and relevance scoring run locally and cost no provider tokens. -The LLM classifies each memory into one of three classes based on how long the information is useful: +## Memory classes | Class | Lifespan | Auto-recalled | Example | |-------|----------|---------------|---------| -| **Durable fact** | Permanent | Yes (prioritized) | "Prefers TypeScript over JavaScript" | -| **Evidence** | 30 days | Sometimes (if score is high enough) | "Completed the database migration on May 2nd" | +| **Durable fact** | Permanent | Yes, if it clears the gate | "Prefers TypeScript over JavaScript" | +| **Evidence** | 30 days | Yes, if it clears the gate | "Completed the database migration on May 2nd" | | **Trace** | 72 hours | Never | Raw conversation logs | -Durable facts are what make the agent feel like it knows you. Evidence — work product, findings, completed tasks — matters for weeks but not forever. When something goes wrong with memory, traces are how you debug it. +Durable facts are what make the agent feel like it knows you. Evidence (work product, findings, completed tasks) matters for weeks but not forever. Traces are for debugging memory itself. -Expired evidence stays in the database but drops out of search results. The query layer filters expired records at read time — no garbage collection runs. +Expired evidence stays in the database but drops out of search results. The query layer filters it at read time, and no garbage collection runs. ## The knowledge graph -Memories are organized around **anchors** — entities like projects, people, preferences, workflows, and milestones. Anchors can relate to each other (`related_to`, `depends_on`), forming a lightweight knowledge graph. The LLM manages the graph structure behind the scenes as it creates and updates memories. +Memories hang off **anchors**: entities like projects, people, preferences, workflows and milestones. Anchors can be linked by edges, which makes a lightweight graph. The LLM manages it as it creates and updates memories. + +That's what connects the dots across conversations. When the agent stores "uses PostgreSQL" as a durable fact anchored to your backend project, and later you ask about that project's infrastructure, the PostgreSQL fact comes along because the anchor links them. + +## How recall works -This is what connects the dots across conversations. When the agent stores "uses PostgreSQL" as a durable fact anchored to your backend project, and later you ask about the backend project's infrastructure, the PostgreSQL preference comes along for the ride — because the anchor links them. +Before each user turn, a retrieval coordinator runs a small [retrieval-augmented generation](https://en.wikipedia.org/wiki/Retrieval-augmented_generation) pipeline, all inside the daemon. + +1. **Plan.** It tokenizes your message and recent context into a lexical query and anchor hints. +2. **Candidates.** SQLite FTS5 returns lexical matches. In parallel the query is embedded with the local model and up to 50 nearest vectors are fetched by cosine similarity. The two sets are merged. +3. **Rank.** Each candidate gets a fused score: 0.7 times cosine similarity, plus 0.3 times its squashed lexical score, plus a small prior for its memory class. That score is multiplied by a recency factor, `0.85 + 0.15 × 2^(-age/30 days)`, so old memories lose a little but never vanish on age alone. +4. **Floor.** A candidate whose vector is below the similarity floor (0.24 for the default model) is dropped. A memory with no vector yet, because it predates the backfill, can't be measured, so it skips the floor and competes on its lexical score. Records (the immutable-record class) have no embeddings by design, so they always compete on lexical score and don't count as a recall coverage gap. +5. **Gate.** The top candidates, at most 3 by default, go to a cross-encoder that reads the query and the memory together. Anything scoring below 0.02 is dropped. +6. **Inject.** What's left goes into the turn as a `[memory-recall]` block, capped at 2,000 characters. Zero survivors means no block at all, which is a healthy result and the common one. + +The whole thing runs on a 300 ms budget, with 150 ms for embedding the query and 120 ms for the gate. If a stage runs out of time or a model isn't loaded, recall degrades to the stage before it instead of delaying your turn. With embeddings off or unavailable, recall is full-text only: the same query against the same store gives the same results every time. + +The recalled block rides in the per-turn nudge, not the system prompt, so it doesn't break prompt caching. [Context structure](/architecture/context-structure/) shows where. + +### Why the gate exists + +The 0.27.0 release notes report this out-of-sample test on 450 real-traffic queries: + +| Metric | Hybrid floor only | With the cross-encoder gate | +|--------|------------------:|----------------------------:| +| Correct zero-injection | 7.3% | 86.8% | +| Relevant recall retained | n/a | 98.3% | +| Mean injected items per turn | 2.538 | 0.251 | + +The floor alone lets weakly related memories through nearly every turn. The gate cuts that to about one turn in four while keeping almost all the useful hits. + +### The models + +| | Embedder | Relevance gate | +|---|---|---| +| Model | `snowflake-arctic-embed-m-int8` | `ms-marco-minilm-l-6-v2` | +| Job | Turns text into a 768-dimension vector | Scores a query and a memory together | +| Size on disk | 110 MB | 23 MB | +| Runs | At write time, and on each recall query | Per candidate on recall | + +Both are downloaded from huggingface.co on first daemon start into `~/.netclaw/models/`, checked against pinned SHA-256 hashes, and loaded in-process through ONNX Runtime. Vectors live in the `memory_embeddings` table, and the daemon keeps a brute-force index in memory. Expect roughly 480 MB resident with them loaded and about 235 MB without. The [memory configuration page](/configuration/memory/) has measured numbers, the offline procedure, and the switch to turn it all off. ## Audience scoping -Netclaw classifies every session into a [trust tier](/architecture/design-philosophy/#lightweight-access-control-without-an-identity-provider) based on its source channel — Personal, Team, or Public. Memory respects these tiers: +netclaw classifies every session into a [trust tier](/architecture/design-philosophy/#lightweight-access-control-without-an-identity-provider) from its source channel: Personal, Team or Public. Memory respects the tiers: -- **Public** sessions have no memory access — no recall, no writes, no search. Memories authored by Public sessions are excluded from all audiences. +- **Public** sessions have no memory access. No recall, no writes, no search. Memories authored by Public sessions are excluded from all audiences. - **Team** and **Personal** sessions have full memory access, scoped by policy boundary. -Each memory carries the audience and boundary context of the session that created it. This prevents a memory formed in a private Personal session from surfacing in a shared Team channel unless the policy explicitly allows it. +Each memory carries the audience and boundary of the session that created it. A memory formed in a Personal session doesn't surface in a shared Team channel unless policy allows it. ## Compaction and memory -When a session's context window fills up and [compaction](/architecture/sessions/#compaction) kicks in, the memory system gets involved at Phase 2 — before the lossy summarization step. Key facts, decisions, and action items are extracted and checkpointed into the memory pipeline, ensuring they survive even if the summary misses them. - -This is the safety net: compaction can lose nuance, but durable memories persist independently of the conversation history. +When a session's context fills up, [compaction](/architecture/sessions/#compaction) shrinks the history and a model call writes compaction notes about what was discarded. After the compaction is saved, those notes are queued as a `compaction-boundary` checkpoint and go through the same curation pipeline as everything else. That's the safety net: compaction can lose nuance, but durable memories persist independently of the conversation history. ## Explicit tools -The agent also has four tools for direct memory management, available in Team and Personal sessions: +The agent has four tools for direct memory management, available in Team and Personal sessions: | Tool | Purpose | |------|---------| -| `find_memories` | Search durable facts and current evidence by keyword (includes evidence that auto-recall may not surface) | -| `get_memories` | Retrieve full content of specific memories by ID | -| `store_memory` | Explicitly save a memory (bypasses most curation filters) | -| `update_memory` | Correct or supersede an existing memory | +| `find_memories` | Search durable facts and current evidence, optionally including expired evidence | +| `get_memories` | Load the full content of specific memories by id | +| `store_memory` | Save a memory on request | +| `update_memory` | Edit a memory (find and replace, or full rewrite), supersede a record, or delete one | -These are the manual-control layer. Most of the time, automatic observation handles memory formation. Explicit tools are for when you want to correct something the agent got wrong or store something it wouldn't have observed on its own. If the agent recalls something incorrect, tell it — "that's wrong, I switched to Postgres last month" — and it will update or supersede the old memory. +Most of the time, automatic observation handles formation and automatic recall handles retrieval. The tools cover the gaps: correcting something the agent got wrong, or searching when recall injected nothing. If you say "that's wrong, I switched to Postgres last month", it finds the memory and edits it. To find, correct or remove a wrong memory yourself, see [fix a wrong memory](/configuration/memory/#fix-a-wrong-memory). ## Observability -Memory formation is invisible during normal use — the agent just gets smarter over time. To see what's happening: +Memory is invisible in normal use. To see what it's doing: -- **`netclaw stats`** shows memory recall counts and checkpoint queue depth -- **`netclaw doctor`** runs a health check on the checkpoint queue (warns if pending checkpoints exceed 25) -- **Debug logs** emit query plans, recall scores, and curation decisions for troubleshooting +- **`netclaw status`** shows `memory: sqlite (healthy)` or degraded. +- **`netclaw stats`** shows how many memories were formed and recalled this process, the store's anchor, document, record and edge counts, and pending checkpoints. +- **`netclaw doctor`** checks the checkpoint backlog (warns above 25 pending), curation model health, embedding coverage and the relevance gate. +- **The daemon log** carries a `memory_retrieval_final` line per turn with how many candidates the floor and the gate dropped, and `memory_recall_*_degraded` lines when a stage was skipped. ## Configuration -Memory is enabled by default. The key settings in `netclaw.json`: +Memory is on by default, and the defaults suit most installs. The settings that matter are `Memory.Enabled`, `Memory.Embeddings.Enabled`, `Memory.Embeddings.AutoDownload` and `Memory.AutoRecallMaxItems`. All of them are covered, with defaults, on [Memory and embeddings](/configuration/memory/). -| Setting | Default | What it controls | -|---------|---------|-----------------| -| `Memory.Enabled` | `true` | Master switch — disables all memory when `false` | -| `Memory.RecallTimeoutMs` | `300` | Time budget for auto-recall per turn | -| `Memory.AutoRecallMaxItems` | `3` | Maximum memories injected per turn | +## Limitations -SQLite is the only storage backend — no vector database, no external service, no API keys. Netclaw should work on a single machine with nothing beyond what ships in the binary. FTS5 handles the current recall workload; vector search may arrive later. +- One machine, one SQLite file. There's no external vector database, and no remote embedding service to point at. +- The first start with embeddings on needs to reach huggingface.co, or the model files copied in beforehand. +- There's no `netclaw memory` command to list, export or delete memories. The agent's tools and the SQLite file are the routes. +- A memory edited through the agent is re-embedded as part of the edit. One edited directly in the SQLite file loses its vector until the next daemon start or `netclaw memory backfill-embeddings`. -## Keep reading +## Related pages -- [Sessions & Input Model](/architecture/sessions/) — session lifecycle and compaction (where memory extraction fits) -- [Context Structure](/architecture/context-structure/) — how the recall and memory-index layers slot into each turn's context -- [Security Architecture](/architecture/security-model/) — audience scoping that governs memory access -- [Design Philosophy](/architecture/design-philosophy/#automatic-memory-formation) — why memory belongs in core +- [Sessions and input model](/architecture/sessions/) - the lifecycle and compaction that feed memory +- [Context structure](/architecture/context-structure/) - where recalled memories sit in each turn +- [Memory and embeddings](/configuration/memory/) - configuration, resource cost, offline use, fixing a wrong memory +- [`netclaw memory`](/cli/memory/) - backfill vectors after an upgrade +- [Design philosophy](/architecture/design-philosophy/#automatic-memory-formation) - why memory belongs in core ## Resources -- [SQLite FTS5](https://www.sqlite.org/fts5.html) — the full-text search engine behind memory recall -- [Actor model](https://en.wikipedia.org/wiki/Actor_model) — the concurrency model that makes the observation sidecar possible -- [Akka.NET Persistence](https://getakka.net/articles/persistence/event-sourcing.html) — journal and snapshot recovery for the checkpoint queue -- [Write-Ahead Logging](https://www.sqlite.org/wal.html) — how SQLite ensures memory durability +- [SQLite FTS5](https://www.sqlite.org/fts5.html) - the full-text engine behind lexical recall +- [ONNX Runtime](https://onnxruntime.ai/) - runs the embedding and relevance models +- [snowflake-arctic-embed-m](https://huggingface.co/Snowflake/snowflake-arctic-embed-m) - the embedding model netclaw ships as its default +- [Write-ahead logging](https://www.sqlite.org/wal.html) - how SQLite keeps the memory store durable +- [Akka.NET persistence](https://getakka.net/articles/persistence/event-sourcing.html) - journal and snapshot recovery behind the observer diff --git a/src/content/docs/architecture/overview.md b/src/content/docs/architecture/overview.md index 1ae13f0..5defdf4 100644 --- a/src/content/docs/architecture/overview.md +++ b/src/content/docs/architecture/overview.md @@ -1,195 +1,230 @@ --- -title: "Architecture Overview" -description: "Daemon + CLI architecture, Akka.NET, and design principles." +title: "Architecture overview" +description: "The daemon and CLI, the Akka.NET actor system, and how a message becomes a turn." --- -Netclaw is a single-process .NET 10 application that runs as an always-on autonomous agent. It takes input from Slack, Discord, webhooks, timers, and a local TUI, and treats all of them the same way: as messages routed into persistent conversation actors. - -Two binaries ship together: `netclawd` (the daemon) and `netclaw` (the CLI). The daemon does everything. The CLI is a thin client that renders the daemon's output in a terminal. If the daemon isn't running, the CLI can't do anything interesting. +netclaw is a single-process .NET 10 application that runs as an always-on agent. It takes input from Slack, Discord, Mattermost, webhooks, timers and a local TUI, and treats all of them the same way: as messages routed into persistent conversation actors. One daemon owns everything that matters, and the rest of this page maps its parts. + +Two binaries ship together: `netclawd` (the daemon) and `netclaw` (the CLI). The daemon holds the agent logic. The CLI is mostly a thin client that renders the daemon's output in a terminal, with a few commands (`memory`, most of `doctor`) that work straight on files, so they still run when the daemon is down. + +```mermaid +flowchart TB + subgraph IN["Inputs"] + direction LR + CLI["netclaw CLI / TUI"] + CH["Slack, Discord, Mattermost"] + WH["Webhooks"] + RM["Reminders"] + end + subgraph D["netclawd (one process)"] + direction TB + GW["Gateway: auth and policy checks"] + SM["SessionManager"] + SA["LlmSessionActor, one per conversation"] + OBS["Memory observer"] + SUB["Subagents"] + TOOLS["ToolAuthorizer, built-in tools, MCP"] + GW --> SM --> SA + SA --> OBS + SA --> SUB + SA --> TOOLS + end + subgraph EXT["Backing services"] + direction LR + LLM["LLM providers"] + DB[("netclaw.db, SQLite: journal, memory, reminders")] + ONNX["Local ONNX models"] + MCPS["MCP servers"] + end + CLI -->|"SignalR + REST"| GW + CH --> GW + WH --> GW + RM -->|"fires a session"| SM + SA --> LLM + SA --> DB + OBS --> DB + OBS -->|"curation embeds"| ONNX + TOOLS --> MCPS +``` ## Two binaries -### `netclawd` — the daemon - -The daemon is an ASP.NET Core application that owns all agent logic: Akka.NET actor system, LLM sessions, tool execution, channel adapters (Slack/Discord), SignalR hub, REST API, and SQLite persistence. +### `netclawd`: the daemon -It binds to `http://127.0.0.1:5199` (loopback only) by default. A PID file at `~/.netclaw/netclaw.pid` and a lock file enforce single-instance — one daemon per machine. +The daemon is an ASP.NET Core application that owns the [Akka.NET](https://getakka.net/) actor system, LLM sessions, tool execution, the channel adapters (Slack, Discord, Mattermost), the SignalR hub, the REST API, and SQLite persistence. It binds to `http://127.0.0.1:5199` by default, and a PID file at `~/.netclaw/netclaw.pid` plus a lock file enforce one daemon per machine. To listen beyond loopback, see [exposure modes](/deployment/exposure-modes/). -### `netclaw` — the CLI +### `netclaw`: the CLI -A Termina-based TUI. No actor system, no persistence, no tool execution. It connects to the daemon over SignalR and renders what comes back. +A Termina-based TUI with no actor system and no tool execution. It connects to the daemon over SignalR and renders what comes back. It also manages the daemon: `netclaw daemon start` and `netclaw daemon stop` run `netclawd` as a detached process. -The CLI binary includes a `daemon` subcommand (`netclaw daemon start`, `netclaw daemon stop`) for managing the `netclawd` process. Think of `netclaw` as both the management interface and the interactive shell for a daemon that runs independently. +The split matters. Restart the CLI without interrupting conversations. Keep the daemon in Docker and the CLI on your host, or run the CLI inside the container with `docker exec netclaw `, since the image ships both binaries. Talk to the same sessions from Slack instead of a terminal. -The split matters: restart the CLI without interrupting conversations. Run the daemon in Docker and the CLI on your host. Swap the CLI out entirely for the web UI or a channel adapter. +![netclaw status output: overall healthy, the daemon version, the active model, and connector lines for Discord, Mattermost, the demo-utilities MCP server, Slack and the TUI](/screenshots/output/status.png) -![Netclaw status output showing a running daemon](/screenshots/output/status.png) - -`netclaw status` showing daemon state, active connectors, uptime, and current model. +`netclaw status` reports daemon state, connectors, uptime and the current model. ## Communication -The CLI talks to the daemon over two protocols: +The CLI reaches the daemon over two protocols: | Protocol | Path | Purpose | |----------|------|---------| -| **SignalR** | `/hub/session` | Real-time session interaction: create sessions, send messages, receive streaming output | -| **REST** | `/api/*` | Non-session operations: health, stats, device pairing, MCP queries, shutdown | +| **SignalR** | `/hub/session` | Interactive sessions: create or attach to a session, send messages, answer approval prompts, stream output back | +| **REST** | `/api/*` | Everything without real-time delivery: health, stats, device pairing, MCP queries, shutdown | -SignalR carries the interactive traffic. The client sends commands (`CreateSession`, `SendMessage`, `CompactSession`) and receives typed `SessionOutputDto` events back as a stream. REST handles everything that doesn't need real-time delivery. +The hub's methods are `CreateSession`, `EnsureSession`, `AttachSession`, `SendMessage` and `RespondToInteraction`. Output comes back as typed events on a stream. Remote clients authenticate with a device bearer token issued through [pairing](/guides/pairing-remote-devices/). A loopback request counts as the operator's own only when its `Host` is `localhost` or a loopback address and any `Origin` matches. Otherwise it gets a 401. ## Akka.NET internals -[Akka.NET](https://getakka.net/) is the concurrency and persistence layer. Every conversation is an actor. Every channel adapter is an actor. Background jobs, reminders, tool approvals — all actors. +[Akka.NET](https://getakka.net/) is the concurrency and persistence layer. Every conversation is an actor, every channel adapter is an actor, and so are background jobs, reminders and tool approvals. ### Key actors | Actor | Role | |-------|------| -| `LlmSessionActor` | One per conversation thread, keyed by channel + thread identifier. Persistent, with three states: Ready, Processing, Compacting. | -| `SessionManager` | Routes messages to the right session actor via child-per-entity routing (no lookup table). | -| `ReminderManagerActor` | Scheduled tasks, backed by Akka.Reminders + SQLite. | -| `BackgroundJobManagerActor` | Async shell execution. | -| `SubAgentActor` | Child of a session actor. Handles delegated work (research, code analysis, summarization). | +| `LlmSessionActor` | One per conversation thread. Persistent, with five phases: Recovering, Ready, Processing, Compacting, Passivating. See [sessions](/architecture/sessions/). | +| `SessionManager` | Routes messages to the right session actor with child-per-entity routing (no lookup table). | +| `SessionMemoryObserverActor` | A persistent child of each session. Watches the transcript and proposes memories when the session goes quiet. | +| `MemoryCurationActor` | Validates and deduplicates memory proposals. A background worker applies the same logic to the durable checkpoint queue. | +| `SubAgentActor` | Child of a session actor. Runs delegated work (research, code analysis, summarization) in the background while the session carries on. | +| `ReminderManagerActor` | Scheduled tasks, backed by Akka.Reminders and SQLite. | +| `BackgroundJobManagerActor` | Detached shell jobs, up to 5 at a time. | +| `ToolApprovalActor` | Holds approval requests until a human answers. | +| `PairingActor` | Issues pairing codes and device tokens for remote clients. | | `ModelCapabilityActor` | Singleton that caches model capabilities at startup. | -| `ToolApprovalActor` | Approval gates for restricted tools. | +| Channel actors | A gateway, conversation and thread-binding actor for each of Slack, Discord and Mattermost. | -### Why actors? +### Why actors -Each conversation needs its own state, its own message queue, and crash isolation from other conversations. Actors give you that without thread management. A session actor processes one turn at a time, persists events to SQLite, and recovers its full history on restart. If one conversation crashes, the rest keep running. +Each conversation needs its own state, its own message queue and crash isolation from the others. Actors give you that without thread management. A session actor processes one turn at a time, journals its events to SQLite, and recovers its full history on restart. If one conversation crashes, the rest keep running. -Persistence uses [Akka.Persistence.Sql](https://getakka.net/articles/persistence/event-sourcing.html) with SQLite. Events: `TurnRecorded`, `SessionTitleSet`, `SessionCompacted`. Serialization is [Google Protobuf](https://protobuf.dev/) via the `NetclawProtobufSerializer`. +Persistence uses [Akka.Persistence.Sql](https://getakka.net/articles/persistence/event-sourcing.html) with SQLite and [Google Protobuf](https://protobuf.dev/) serialization through `NetclawProtobufSerializer`. The journal records turns, accepted input, tool batches and their calls, approval requests and decisions, compaction, title changes and background job reaping. It's an event log for the session, not just a transcript. ## Three boundaries -Three logical boundaries divide the daemon (from the [runtime spec](https://github.com/netclaw-dev/netclaw/blob/dev/docs/spec/SPEC-001-runtime-boundaries.md)): +Three logical boundaries divide the daemon, from the [runtime spec](https://github.com/netclaw-dev/netclaw/blob/dev/docs/spec/SPEC-001-runtime-boundaries.md). -At the edge, the **gateway** receives transport events (Slack message, webhook POST, CLI input), runs policy checks, and converts them to actor commands. Nothing reaches the session layer before policy runs. +At the edge, the **gateway** receives transport events (a Slack message, a webhook POST, CLI input), runs policy checks, and converts them to actor commands. Nothing reaches the session layer before policy runs. -The **session** layer owns conversation state, turn lifecycle, compaction, and persistence. Sessions communicate outward only by broadcasting messages — they have no idea whether they're talking to Slack, Discord, or the CLI. +The **session** layer owns conversation state, turn lifecycle, compaction and persistence. Sessions only broadcast outward. They don't know whether they're talking to Slack, Discord or the CLI. -**Subscribers** (channel adapters and UI clients) consume those broadcasts via pub/sub. They render output and nothing else. They can't read actor internals or influence model behavior. +**Subscribers** (channel adapters and UI clients) consume those broadcasts. They render output and nothing else, and they can't read actor internals or influence model behavior. -Data flows one direction: gateway sends commands to sessions, sessions broadcast to subscribers. Policy errors close the gate. +Data flows one way: the gateway sends commands to sessions, and sessions broadcast to subscribers. A policy error closes the gate. ## Everything is just input -Every input source — Slack message, webhook POST, timer, CLI chat, web UI — produces a message routed to a session actor with context-specific instructions. The actor doesn't know or care where it came from; turns process identically regardless of source. - -Adding a new channel adapter requires no changes to the session layer. +A Slack message, a webhook POST, a timer, a CLI chat: each produces a message routed to a session actor with context-specific instructions. The actor doesn't know where it came from, and turns run identically regardless of source. Adding a channel adapter requires no changes to the session layer. ## Channel adapters -Slack and Discord adapters run inside the daemon as [hosted services](https://learn.microsoft.com/en-us/aspnet/core/fundamentals/host/hosted-services). They're in-process actors that maintain persistent connections to the platform APIs, not external bridges. +Slack, Discord and Mattermost adapters run inside the daemon as [hosted services](https://learn.microsoft.com/en-us/aspnet/core/fundamentals/host/hosted-services). They're in-process actors holding persistent connections to the platform APIs, not external bridges. | Channel | Connection | Actors | |---------|-----------|--------| | **Slack** | [Socket Mode](https://api.slack.com/apis/socket-mode) via SlackNet | `SlackGatewayActor`, `SlackConversationActor`, `SlackThreadBindingActor` | | **Discord** | WebSocket via Discord.Net | `DiscordGatewayActor`, `DiscordConversationActor`, `DiscordSessionBindingActor` | +| **Mattermost** | WebSocket and REST, with interactive actions posted back to the daemon | `MattermostGatewayActor`, `MattermostConversationActor`, `MattermostSessionBindingActor` | -An [Akka.Streams](https://getakka.net/articles/streams/introduction.html) `SessionPipeline` bridges channel adapters to the actor system. +An [Akka.Streams](https://getakka.net/articles/streams/introduction.html) `SessionPipeline` bridges channel adapters to the actor system. Setup for each is in the [channels section](/channels/slack/). ## LLM providers -Netclaw uses [Microsoft.Extensions.AI](https://learn.microsoft.com/en-us/dotnet/ai/ai-extensions) (`IChatClient`) as the provider abstraction. Supported providers: +netclaw uses [Microsoft.Extensions.AI](https://learn.microsoft.com/en-us/dotnet/ai/ai-extensions) (`IChatClient`) as the provider abstraction. Supported providers: | Provider | Type key | Notes | |----------|----------|-------| -| **Ollama** | `ollama` | Code default. Native integration, queries `/api/show` for capabilities. | -| **OpenRouter** | `openrouter` | Recommended for cloud models. Widest model catalog. | +| **Ollama** | `ollama` | Native integration, queries `/api/show` for capabilities. | +| **OpenRouter** | `openrouter` | Widest model catalog. | | **Anthropic** | `anthropic` | Direct API. | | **OpenAI** | `openai` | Direct API, OAuth or API key. | | **GitHub Copilot** | `github-copilot` | OAuth device flow. Uses your Copilot subscription. | | **Venice.ai** | `veniceai` | Privacy-focused, OpenAI-compatible API. | -| **DeepSeek** | `deepseek` | 1M context window, thinking mode, text-only. | -| **OpenAI-compatible** | `openai-compatible` | Generic self-hosted endpoint. | +| **DeepSeek** | `deepseek` | `deepseek-v4-*` models get a 1M context window and text-only limits from netclaw. | +| **OpenAI-compatible** | `openai-compatible` | Any self-hosted endpoint (llama.cpp, vLLM). The API key is optional. | -At startup, the daemon queries the provider to detect context window size, vision support, and tool use capabilities. If the direct query fails, it falls back to the OpenRouter catalog, then HuggingFace metadata, then assumes text-only. - -See [Models](/configuration/models/) for configuration and [Managed Providers](/configuration/managed-providers/) / [Self-hosted Providers](/configuration/self-hosted-providers/) for provider setup. +At startup the daemon works out each model's context window and input and output modalities. It asks the provider first, then falls back through the OpenRouter catalog and HuggingFace metadata, filling each field from the first source that knows it, and assumes text-only if nothing answers. See [Models](/configuration/models/) for configuration and [managed](/configuration/managed-providers/) or [self-hosted](/configuration/self-hosted-providers/) providers for setup. ## Session lifecycle -Each conversation thread maps to one persistent actor. In Slack, the key is `{channelId}/{threadTs}`; Discord uses a similar channel + thread mapping. - -**States:** Ready → Processing → (threshold check) → Ready or Compacting → Ready. - -When a session's context grows past the compaction threshold, the actor compacts in phases: clear stale tool results (Phase 1), drop old messages via extractive reduction keeping the N most recent (Phase 2), then optionally generate observation notes from the discarded window via a best-effort LLM call (Phase 3). If the LLM call fails, extractive-only results stand. Context shrinks, and the session returns to Ready. +Each conversation thread maps to one persistent actor. The actor cycles between Ready and Processing, compacts when its context passes a threshold, and passivates when idle. [Sessions](/architecture/sessions/) covers the phases, the timeouts, compaction and what a session leaves on disk. ## Tool system -Tools are C# methods registered at compile time via a Roslyn incremental source generator. The generator produces JSON schema and typed deserializers — no runtime reflection. `ToolRegistry` holds the catalog; `IToolExecutor` runs them. +Tools are C# methods registered at compile time through a Roslyn incremental source generator, which produces the JSON schema and typed deserializers with no runtime reflection. `ToolRegistry` holds the catalog and `IToolExecutor` runs them. -Access control is layered: `ToolAccessPolicy` (which tools are available), `ToolPathPolicy` (filesystem boundaries), `ShellCommandPolicy` (allowed shell commands). Feature gates control progressive exposure. +Since 0.27.1 every tool call and every shell call goes through one `ToolAuthorizer`: the same authority and consent loop, so there's no second gate to drift out of sync. The older policy classes (`ToolAccessPolicy` for which tools exist, `ToolPathPolicy` for filesystem boundaries, `ShellCommandPolicy` for shell commands) still do the individual checks inside it. Feature gates hide the tools of a disabled subsystem (memory, search, skills, scheduling, subagents) from every audience. The deployment posture caps the audience of every call, including a sub-agent's, so a Team posture gives `netclaw chat` Team tools. See [the security model](/security/security-model/). + +A session starts with a small core of tools and loads the rest on demand, so the prompt stays small. That's progressive disclosure. `search_tools` ranks its results by how many query words match, so a built-in tool like `list_reminders` turns up ahead of a shell command. ### MCP integration -External tool servers connect via the [Model Context Protocol](https://modelcontextprotocol.io/). `McpClientManager` handles server lifecycle including OAuth 2.1 flows. Sessions see only server summaries until a tool is explicitly requested via `search_tools` (progressive disclosure). See [MCP Servers](/configuration/mcp-servers/) for configuration. +External tool servers connect through the [Model Context Protocol](https://modelcontextprotocol.io/). `McpClientManager` handles the server lifecycle, including OAuth 2.1 flows and, since 0.27.0, pre-registered confidential clients with a stored client secret. Servers connect in parallel at startup. Sessions see server summaries until a tool is requested through `search_tools`. A server's prompts can be invoked as skills, and image, audio and resource results from MCP tools are delivered as files. See [MCP servers](/configuration/mcp-servers/). ## Supporting systems -The daemon keeps a [SQLite](https://www.sqlite.org/docs.html)-backed memory store in `~/.netclaw/netclaw.db`. Facts persist across sessions; a background `MemoryCurationEngine` handles pruning and deduplication. +The daemon keeps memory in `~/.netclaw/netclaw.db`: documents, records, a full-text index and, since 0.27.0, vectors. Two small ONNX models downloaded to `~/.netclaw/models/` embed memories and gate recall, entirely in-process. A background `MemoryCurationEngine` extracts, deduplicates and merges proposals. See the [memory model](/architecture/memory-model/). -**Skills** are markdown files in `~/.netclaw/skills/` that extend what the agent knows how to do. Built-ins ship as embedded resources; extras sync from a CDN feed at startup. +**Skills** are markdown files that extend what the agent knows how to do. The built-in system skills ship inside the daemon binary and are restored on every start. Everything else comes from your own skills directory, operator-configured [skill feeds](/skills/skill-feeds/) and external sources, or from the agent itself through the `skill_manage` tool. -For delegated work — research, code analysis, summarization — a session spawns a **subagent** as a child actor. Definitions live in `~/.netclaw/agents/*.md` with YAML frontmatter (built-ins: research-assistant, code-analyst, summarizer), and each subagent run gets up to 30 tool iterations. See [Custom Subagents](/guides/custom-subagents/). +For delegated work, a session spawns a **subagent** as a child actor. Definitions live in `~/.netclaw/agents/*.md` with YAML frontmatter (built-ins: research-assistant, code-analyst, summarizer), and each run goes on until it gives a final reply, is cancelled or goes quiet for too long, with no fixed cap on tool rounds. `spawn_agent` returns a run ID at once. The run is recorded in the session journal, and its result arrives in a later turn as a notice that names a `result.md` file. See [custom subagents](/skills/custom-subagents/). -Scheduled tasks go through `ReminderManagerActor`. See [Reminders](/configuration/reminders/). +Search runs through Brave, DuckDuckGo or SearXNG backends. Images and attachments are normalized before they reach a model. Scheduled tasks go through `ReminderManagerActor`, covered in [Reminders](/configuration/reminders/). ## Configuration Layered, with later sources winning: -1. `~/.netclaw/config/netclaw.json` — base config -2. `~/.netclaw/config/secrets.json` — encrypted credentials (ASP.NET Core Data Protection) -3. `NETCLAW_*` environment variables — highest priority +1. `~/.netclaw/config/netclaw.json`: base config +2. `~/.netclaw/config/secrets.json`: encrypted credentials (ASP.NET Core Data Protection) +3. `NETCLAW_*` environment variables: highest priority + +`ConfigWatcherService` watches `netclaw.json` with a 500 ms debounce. A valid change triggers a coordinated restart: close ingress, drain active sessions, persist state, restart in-process, then warm the active sessions back up. An invalid change is refused and the running daemon keeps its current configuration. Secrets and environment changes aren't watched, so those need a manual restart, and an OAuth token refresh writes `secrets.json` without causing one. -A `ConfigWatcherService` monitors `netclaw.json` with a 500ms debounce. Valid changes trigger a coordinated restart: close ingress, drain active sessions, persist state, restart, then warm active sessions back up. Active CLI sessions reconnect automatically after the restart completes. +The drain stops each session at a safe point, such as the end of a tool batch, and tells the person in the chat. A turn stopped after its first tool batch resumes after the restart when nothing else is waiting. A running sub-agent parks and resumes too, which can add up to about 18 seconds to a graceful restart. A graceful stop of the whole daemon is still bounded to 30 seconds. ## Deployment -The daemon runs on anything .NET 10 supports, from a Raspberry Pi to a cloud VM. Ships as a self-contained tarball: `netclaw-{version}-{os}-{arch}.tar.gz`. +netclaw runs on Linux (x64 and arm64), Windows x64 with native PowerShell, and macOS on Apple Silicon. The installer downloads two archives per platform, `netclaw` and `netclawd`, as `.tar.gz` (`.zip` on Windows). | Method | Guide | |--------|-------| -| **systemd** (recommended for Linux) | [systemd Service](/deployment/systemd/) | -| **Docker** | [Docker Deployment](/deployment/docker/) | +| **systemd** (recommended for Linux) | [systemd service](/deployment/systemd/) | +| **Docker** | [Docker deployment](/deployment/docker/) | | **Direct** | `netclaw daemon start` (detached process) | -Docker runs only the daemon; the CLI stays on the host. For remote access beyond loopback, see [Exposure Modes](/deployment/exposure-modes/). +The Docker image ships both binaries, with `netclaw` on the PATH, so `docker exec netclaw status` works. For remote access beyond loopback, see [exposure modes](/deployment/exposure-modes/). ## Design principles -None of these are aspirational — they're constraints the codebase enforces. +None of these are aspirational. They're constraints the codebase enforces. -1. **Gall's Law first** — ship a simple working system before building a generalized framework. -2. **Actor transport boundary** — channel adapters consume broadcasts via pub/sub and never drive model internals directly. -3. **Serialization boundary** — Netclaw controls its own persistence schema. Never persist framework-external chat types. -4. **Security boundary** — default deny, explicit allow, fail closed. See [Security Model](/security/security-model/). -5. **Everything is just input** — every source (Slack, webhook, CLI, timer) produces a message routed to a session actor. No special-casing by channel. +1. **Gall's Law first.** Ship a simple working system before building a generalized framework. +2. **Actor transport boundary.** Channel adapters consume broadcasts through pub/sub and never drive model internals directly. +3. **Serialization boundary.** netclaw controls its own persistence schema and never persists framework-external chat types. +4. **Security boundary.** Default deny, explicit allow, fail closed. See the [security model](/security/security-model/). +5. **Everything is just input.** Every source (Slack, webhook, CLI, timer) produces a message routed to a session actor. No special-casing by channel. ## Limitations -No clustering or multi-node distribution. One daemon, one machine, one actor system. - -SQLite is the only persistence backend. Good for single-node; won't scale to multi-instance. - -Security is in Phase 1: all connections are loopback-only with `LocalOperator` trust (full access). Bearer token auth and device pairing for remote access are planned for a future release. - -Binary updates require a daemon restart. No hot code reload. - -## What to read next - -Setting up for the first time? Go to [Models](/configuration/models/) to configure your LLM provider. +- No clustering. One daemon, one machine, one actor system. +- SQLite is the only persistence backend. That's right for a single node and doesn't scale to multiple instances. +- Remote access works through exposure modes and paired devices, not through an identity provider. There's no per-user login. +- Binary updates need a daemon restart. There's no hot code reload. +- Local semantic memory loads about 129 MB of models and makes the daemon roughly twice the size (about 480 MB idle against about 235 MB) it'd be without them. Turn it off if the host can't spare the RAM. -For production deployment, see [systemd Service](/deployment/systemd/) or [Docker Deployment](/deployment/docker/). +## Related pages -For the security model, see [Security Model](/security/security-model/) — covers ACLs, trust context, and audience scoping. +- [Sessions and input model](/architecture/sessions/) - lifecycle, timeouts, compaction and storage +- [Memory model](/architecture/memory-model/) - how the daemon remembers across sessions +- [Context structure](/architecture/context-structure/) - what the model sees each turn +- [Security model](/security/security-model/) - audiences, approvals and the tool authorizer +- [Design philosophy](/architecture/design-philosophy/) - why it's built this way ## Resources -- [Akka.NET documentation](https://getakka.net/) — the actor model framework behind netclaw's concurrency -- [Microsoft.Extensions.AI](https://learn.microsoft.com/en-us/dotnet/ai/ai-extensions) — the `IChatClient` abstraction netclaw uses to talk to LLM providers -- [Model Context Protocol specification](https://modelcontextprotocol.io/) — standard for external tool server integration -- [SignalR documentation](https://learn.microsoft.com/en-us/aspnet/core/signalr/introduction) — real-time communication between CLI and daemon -- [SQLite documentation](https://www.sqlite.org/docs.html) — the persistence and memory backend +- [Akka.NET documentation](https://getakka.net/) - the actor framework behind netclaw's concurrency +- [Microsoft.Extensions.AI](https://learn.microsoft.com/en-us/dotnet/ai/ai-extensions) - the `IChatClient` abstraction for LLM providers +- [Model Context Protocol specification](https://modelcontextprotocol.io/) - the standard for external tool servers +- [SignalR documentation](https://learn.microsoft.com/en-us/aspnet/core/signalr/introduction) - real-time communication between the CLI and the daemon +- [SQLite documentation](https://www.sqlite.org/docs.html) - the persistence and memory backend diff --git a/src/content/docs/architecture/security-model.md b/src/content/docs/architecture/security-model.md index ffede98..0063dd9 100644 --- a/src/content/docs/architecture/security-model.md +++ b/src/content/docs/architecture/security-model.md @@ -1,135 +1,124 @@ --- -title: "Security Architecture" -description: "ACL evaluation, audience scoping, and trust context." +title: "Security architecture" +description: "Why netclaw's security model is shaped the way it is: default-deny, audiences as the trust primitive, one authorizer with four outcomes, and a human in the loop." --- -Netclaw gives an LLM shell access, file access, and the ability to talk to external services. Without strong guardrails, that's a loaded weapon pointed at your infrastructure. The security architecture exists to make the agent useful without making it dangerous. +Netclaw gives an LLM shell access, file access and the ability to talk to external services. Without guardrails, that's a loaded weapon pointed at your infrastructure. The security architecture exists to make the agent useful without making it dangerous, and it's enforced from daemon startup with no extra setup. -All of this is enforced automatically from daemon startup — no additional setup required for the defaults to protect you. +The core bet is that explicit grants are safer than implicit restrictions. Rather than enumerate everything dangerous and block it, netclaw starts from zero permissions and asks you to opt in to each capability. We build up from nothing instead of poking holes in a wall. -The core bet: **explicit grants are safer than implicit restrictions.** Rather than trying to enumerate everything dangerous and blocking it, netclaw starts from zero permissions and requires you to opt in to each capability. This inverts the typical "sandbox" approach — instead of poking holes in a wall, you're building up from nothing. +This page is the design rationale. For what netclaw enforces and the exact lists, see the [Security model](/security/security-model/). For how prompts work, see [How tool approvals work](/security/approvals/). -## Design Decisions +## Design decisions ### Default-deny over default-allow -Most tools start permissive and add restrictions. Netclaw does the opposite. A freshly initialized daemon with no config file binds to loopback, disables shell access, and limits tools to basic file operations in a temporary session directory that's wiped on session end. You build up from there. +Most tools start permissive and add restrictions. Netclaw does the opposite. A fresh daemon with no config file binds loopback, disables shell, and gives channel traffic only basic file tools inside a session directory. Even the local TUI gets only those Public tools until a config sets a deployment posture. You build up from there. -Misconfiguration fails safe. Forget to add a tool grant? The tool is invisible. Typo in a channel ID? That channel gets nothing. Config file corrupted? Daemon won't start. +Misconfiguration fails safe. Forget to add a tool grant and the tool is invisible. Mistype a channel ID and that channel gets nothing. Write a malformed tool list and the daemon refuses to start. ### Audience as the trust primitive -Rather than per-user ACLs (which require identity infrastructure most self-hosted deployments don't have), netclaw classifies trust by **channel type**. A message from the TUI (terminal interface, launched by `netclaw chat`) is from the operator sitting at the machine — high trust. A message from Slack could be anyone in the workspace — lower trust. An unknown channel gets the lowest trust. - -Three audiences, ordered by trust: +Per-user ACLs need identity infrastructure that most self-hosted deployments don't have. Netclaw classifies trust by where a message came from instead. A message from the TUI is from the operator at the machine. A message from Slack could be anyone in the workspace. An unknown source gets the least trust. ``` Personal > Team > Public ``` -![Security posture selection during netclaw init](/screenshots/output/init-02-security-posture.png) - -Posture selection during `netclaw init` — the choice that establishes the baseline trust tier for your deployment. - -This maps to how people actually deploy: you trust yourself on your own machine, partially trust your coworkers in Slack, and don't trust unknown sources at all. During setup, you assign each channel to an audience tier individually: +This maps to how people deploy: you trust yourself on your own machine, partly trust coworkers in a channel, and don't trust unknown sources. Each channel or DM is assigned an audience individually, so one platform can span tiers, with DMs as Personal and a shared ops channel as Team. See [Audiences](/security/security-model/#trust-audiences) for how audiences are assigned. -![Per-channel audience configuration during netclaw init](/screenshots/output/init-04-channel-audiences.png) - -The same platform can span multiple audience tiers — Slack DMs as Personal, a shared ops channel as Team. The exact per-audience permission tables are in the [Security Model reference](/security/security-model/#trust-audiences). +The deployment posture is a ceiling on top of that. The audience that every tool decision uses, for a turn and for any sub-agent or routed skill it starts, is the narrower of the posture's audience and the message's source audience. A Team posture therefore runs the TUI with Team tools, which is the point of choosing it: that choice sets the baseline for what the agent can do wherever the message came from. ### Four layers, not one -No tool executes without passing four independent checks. Each layer can only deny — none can override a denial from another. +Every tool call goes through one authorizer. It applies a single ordered list of rules, and the first rule with an answer wins. Conceptually the rules fall into four layers, and a call has to get through all of them. + +```mermaid +flowchart LR + C["Tool call"] --> A["Admission
does this audience
have the tool?"] + A --> P["Prohibition
hard deny and
protected paths"] + P --> F["Filesystem authority
trusted roots"] + F --> K["Consent
mode, grants,
prompt"] + A -.-> D["Denied"] + P -.-> D + F -.-> D + K --> O["Allowed, requires approval,
requires correction or denied"] +``` -| Layer | Class | What it does | -|-------|-------|-------------| -| **Operation hard deny** | `ShellCommandPolicy` | Blocks unconditionally dangerous commands — `rm -rf /`, `sudo`, fork bombs, daemon self-kill. Structural matching on tokenized verb chains. Not approvable. | -| **Resource hard deny** | `ToolPathPolicy` | Blocks access to protected paths — `secrets.json`, key material, control-plane files. Symlink-aware. Maintains separate deny surfaces for reads, writes, and shell path references. | -| **Tool access grant** | `ToolAccessPolicy` | Audience-scoped allowlist. If a tool isn't granted to the current audience, the model never sees it. Binary — available or invisible. | -| **Approval gate** | `IToolApprovalService` | Human-in-the-loop confirmation for operations that pass the first three checks but are operationally risky. Approvals can be scoped to once, session, or persistent. | +The details those layers cover include the rule that Personal file reads reach as far as writes do, a private temp folder for each session, checks on redirects in compound shell commands, and glob checks that follow symlinked folders. The first three layers can only deny, and nothing the operator answers at a prompt reaches back through them. Consent is the only layer that asks a person, and it has two outcomes beyond allow and deny: it can prompt, or it can hand the agent advice so the agent re-authors the call. Putting prohibition before consent is the point. A saved "Always" grant can never cover a hard-denied command or a protected path, because those rules run first. -The execution order in code doesn't match this conceptual ordering — `ToolAccessPolicy` runs first (fail fast on unavailable tools before evaluating shell commands), then `ShellCommandPolicy`, then `ToolPathPolicy`, then the approval gate. The deny-only semantics are the same regardless of order. See the [reference page](/security/security-model/#four-layer-invocation-stack) for exact deny lists and configuration syntax. +Having one authorizer rather than four independent checks matters for a second reason. Built-in tools, MCP tools, shell, subagents and unattended runs all take the same path, so a rule can't be bypassed by finding the tool that skipped a check. The exact order is in [Layers in order](/security/security-model/#layers-in-order). ### Policy before dispatch -Inbound message ACL checks run in the gateway boundary, before messages reach the session layer. Tool execution policy (shell denies, path checks, approval gates) runs in the session's tool executor, synchronously before any tool fires. The [three-boundary design](/architecture/overview/#three-boundaries) (gateway → session → subscriber) enforces this structurally — the gateway is the only ingress point for messages, and the tool executor is the only execution path for tools. +Inbound message ACL checks run at the gateway, before a message reaches the session layer. Tool policy runs in the session's tool executor, before any tool fires. The [three-boundary design](/architecture/overview/#three-boundaries) (gateway, session, subscriber) makes that structural: the gateway is the only way in for messages, and the tool executor is the only way to run a tool. ### Human-in-the-loop as a layer, not a crutch -Approval gates sit at Layer 4 — after hard denies and access grants have already filtered. The human only sees requests that are structurally permitted but operationally risky. You're not approving every `ls` — you're approving `git push` to a remote. +Approval is the last layer, after hard denies and access grants have already filtered. The human only sees requests that are permitted but risky. You're not approving every `ls`. Reviewed read-only commands never prompt, and an answer of "Always" turns a one-off decision into a saved grant on a phrase, so `git push` is approved once rather than every time. -![Approval gate prompt in Slack — the human sees verb, tool, and directory before choosing](/assets/approval-prompt.png) +Prompts are how you build a policy, not the policy itself. When they keep coming, change a mode or a grant. [Reduce or turn off approval prompts](/guides/reduce-approval-prompts/) says how. -The approval system extracts verb-chain patterns from shell commands. A verb-chain is the leading command tokens (`git push`, `docker compose up`) without paths or flags — so approving `git push` covers `git push origin main` and `git push --force-with-lease`. Compound commands (`cmd1 && cmd2`) are split and each segment is checked independently. +## Signals on every message -Channels that don't support interactive prompts (scheduled reminders, webhooks, headless automation) auto-deny tools that require approval — unless those tools have been persistently pre-approved or appear on the safe-list for non-interactive execution. +Every inbound message carries four signals: -## Trust Context Model +| Signal | Source | Describes | +|--------|--------|-----------| +| Principal classification | Channel adapter | Who is speaking, such as an operator, a trusted internal user, an untrusted external user or verified automation | +| Transport authenticity | Connection type | Whether the sender can be verified: local process, verified, unverified | +| Payload taint | Content origin | Where the data came from: trusted, community, public | +| Audience | Channel type or configuration | What permission set applies | -Every inbound message carries a trust context assembled from: +The audience resolves the permission set. The other signals describe the turn and travel with it. -| Signal | Source | Purpose | -|--------|--------|---------| -| **Principal classification** | Channel adapter | Who is speaking? (operator, team member, external, automation) | -| **Transport authenticity** | Connection type | Can we verify the sender? (local process, verified, unverified) | -| **Payload taint** | Content origin | Where did this data come from? (trusted, community, public) | -| **Audience** | Channel type | What trust level does this channel get? | +## MCP security: two-gate model -The audience resolves the final permission set. The other signals flow through to audit logs for post-hoc review. +External tool servers connecting over the [Model Context Protocol](https://modelcontextprotocol.io/) are gated twice. -## MCP Security: Two-Gate Model +1. **Server gate.** The server must be reachable for the audience: `AllowedMcpServers` for Team and Public, everything for Personal. +2. **Tool gate.** If per-tool grants exist for the server (`McpServerToolGrants`), only the listed tools are visible. With no entry for the server, all its tools are. -External tool servers connecting via the [Model Context Protocol](https://modelcontextprotocol.io/) (MCP) are doubly gated: +Approval modes add a third dimension: each server and tool can be `Auto`, `Approval` or `Deny` per audience. `netclaw mcp add` writes locked-down defaults for Team and Public and an open default for Personal. See [MCP tool permissions](/guides/mcp-tool-permissions/). -1. **Server allowlist** — the MCP server must be explicitly permitted for the audience -2. **Tool allowlist** — if per-tool grants are configured for that server, only listed tools are available +Netclaw doesn't load MCP tools into the model's context until the model searches for them with `search_tools`. Granted but unused tools don't consume context or influence behavior. -When no per-tool grants are configured for a server, all tools on that server are accessible (subject to the server-level gate). Configure `McpServerToolGrants` to restrict individual tools — see [`netclaw mcp permissions`](/cli/mcp-tools/) for the operational details. +## Self-configuration boundaries -Netclaw doesn't load MCP tools into the model's context until explicitly searched via `search_tools`. Granted-but-unused tools don't consume context window or influence behavior — the model doesn't know they exist until it asks. +The agent can change its own personality, instructions, reminders and project registry. It can't change security policy. Writes to the config directory are blocked by the path policy for every tool and every command form, so ACL rules, exposure mode, tool grants and audience mappings are read-only from the agent's side. Reminder tools are scoped to the caller's audience, so a Team session can't manage a Personal reminder. -## Self-Configuration Boundaries +## Threat model scope -The agent can modify its own personality, instructions, scheduled tasks, and project registry — but security policy is off-limits. ACL rules, exposure mode, tool grants, and audience mappings are read-only from the agent's perspective. - -This is enforced at two levels: the `config_write` tool category (one of seven grant categories in the [ACL policy](/security/security-model/#per-audience-permissions)) requires explicit grants, and the resource hard-deny layer blocks direct file access to config paths regardless of tool grants. - -## Threat Model Scope - -What the security model protects against: +What the model protects against: - The LLM deciding to run destructive commands -- Prompt injection via tool output or file contents attempting privilege escalation +- Prompt injection through channel messages attempting privilege escalation - Overly broad tool access from misconfiguration - Credential leakage through command output - Unauthorized network exposure -What it explicitly does **not** protect against: +What it doesn't protect against: -- A malicious operator with direct machine access (they can edit config files) +- A malicious operator with direct machine access +- Anything else the agent can read as the OS user. Agent tools are denied the common credential folders (`~/.ssh`, `~/.aws`, `~/.kube` and similar), but other files your user can read are open to it, and a command you approve still runs with your credentials +- A granted command doing more than its name suggests. Approvals aren't a sandbox - Side-channel attacks against the LLM provider's API -- Denial of service against the daemon process itself -- Vulnerabilities in third-party MCP servers (netclaw gates access, not behavior) - -## Limitations +- Denial of service against the daemon +- Vulnerabilities in third-party MCP servers. Netclaw gates access, not behavior -- Prompt injection detection is regex-based, not semantic — novel phrasings can evade it -- Approval gates require an interactive channel — headless sessions (scheduled jobs, reminders, webhooks with no connected user) auto-deny gated tools unless persistently pre-approved -- No per-user identity within a channel — all Slack users in an allowed channel get the same audience -- Secret redaction catches known patterns only — custom formats need custom path deny rules -- No formal sandboxing (seccomp, namespaces) for shell execution — defense is policy-based, not kernel-based +The second and third items are why the [Security model](/security/security-model/#what-netclaw-doesnt-protect-against) tells you to run netclaw as a separate OS user or in a container when the machine holds other credentials. A kernel sandbox for shell commands is planned and tracked in [netclaw-dev/netclaw#2343](https://github.com/netclaw-dev/netclaw/issues/2343). -## Related +## Related pages -- [Security Model reference](/security/security-model/) — operational details: exact deny lists, configuration syntax, per-audience permissions -- [Hardening](/security/hardening/) — production lockdown beyond the defaults -- [Architecture Overview](/architecture/overview/) — the three-boundary design that enforces policy-before-dispatch -- [`netclaw init`](/cli/init/) — where you first choose your security posture -- [`netclaw doctor`](/cli/doctor/) — verify what's actually being enforced at runtime +- [Security model](/security/security-model/) - audiences, the order of checks, protected paths and the exact lists +- [How tool approvals work](/security/approvals/) - modes, outcomes, grants and prompts +- [Hardening](/security/hardening/) - production lockdown beyond the defaults +- [Architecture overview](/architecture/overview/) - the three-boundary design behind policy-before-dispatch +- [`netclaw doctor`](/cli/doctor/) - verify what's enforced at runtime -## External Resources +## Resources -- [OWASP LLM Top 10](https://genai.owasp.org/llm-top-10/) — the attack taxonomy netclaw's security model is designed against -- [NIST AI Risk Management Framework](https://www.nist.gov/itl/ai-risk-management-framework) — federal guidance on AI system risk management -- [Principle of Least Privilege (POLP)](https://csrc.nist.gov/glossary/term/least_privilege) — the access control philosophy behind default-deny -- [Defense in Depth (NIST)](https://csrc.nist.gov/glossary/term/defense_in_depth) — the layered-security model behind the four-layer stack +- [OWASP LLM Top 10](https://genai.owasp.org/llm-top-10/) - the attack taxonomy the model is designed against +- [NIST AI Risk Management Framework](https://www.nist.gov/itl/ai-risk-management-framework) - federal guidance on AI risk management +- [Principle of least privilege (NIST)](https://csrc.nist.gov/glossary/term/least_privilege) - the philosophy behind default-deny +- [Defense in depth (NIST)](https://csrc.nist.gov/glossary/term/defense_in_depth) - the layered model behind the authorizer's rule order diff --git a/src/content/docs/architecture/sessions.md b/src/content/docs/architecture/sessions.md index de0eefc..9b51d29 100644 --- a/src/content/docs/architecture/sessions.md +++ b/src/content/docs/architecture/sessions.md @@ -1,130 +1,192 @@ --- -title: "Sessions & Input Model" -description: "How netclaw maps conversations to persistent actors, handles multiple clients, and keeps long-running sessions bounded." +title: "Sessions and input model" +description: "How netclaw maps conversations to persistent actors, what ends or stalls a turn, and how long-running sessions stay bounded." --- -Every conversation in netclaw is a session. One Slack thread, one Discord thread, one CLI chat — each maps to a single persistent [actor](https://en.wikipedia.org/wiki/Actor_model) that owns the conversation's state, processes one turn at a time, and survives daemon restarts. +Every conversation in netclaw is a session. One Slack thread, one Discord thread, one Mattermost thread, one CLI chat: each maps to a single persistent [actor](https://en.wikipedia.org/wiki/Actor_model) that owns the conversation's state, processes one turn at a time, and survives daemon restarts. The session doesn't know or care where its messages come from, and that's the foundation everything else builds on. -The session doesn't know or care where its messages come from. That's the foundation everything else builds on. +A session has a few simple obligations. It accepts input, runs a turn, keeps its history from outgrowing the model's context window, and shuts itself down when nobody's talking. The rest of this page is those four things in detail, plus the files a session leaves on disk. ## Session identity -A session's identity is derived from its channel and thread. In Slack, the key is `{channelId}/{threadTs}`. Discord uses a similar channel-plus-thread mapping. The CLI generates its own session ID when you start a chat. +A session's identity comes from its channel and thread. In Slack the key is `{channelId}/{threadTs}`. Discord uses a channel-and-thread mapping, Mattermost binds a channel and a root post, and the CLI generates its own id when you start a chat. -This is a deliberate 1:1 mapping. One thread, one session, one actor. No shared state between conversations, no cross-thread contamination, no global context that leaks between users. If a session actor crashes, the rest keep running — that's what the actor model buys you. +The mapping is 1:1 on purpose. One thread, one session, one actor. There's no shared state between conversations and no global context leaking between users. If a session actor crashes, the rest keep running. -The `SessionManager` routes messages to the right session actor using child-per-entity routing (each session is a child actor created on demand, keyed by session ID). No lookup table, no registry. The actor hierarchy *is* the routing table. +The `SessionManager` routes messages with child-per-entity routing: each session is a child actor created on demand and keyed by session id. There's no lookup table. The actor hierarchy is the routing table. ## Everything is just input -Slack messages, webhook POSTs, scheduled reminders, CLI chat input — they all enter through different channel adapters but produce the same kind of command: a `SendUserMessage` routed to a session actor. The actor processes the turn identically regardless of source. +Slack messages, Discord messages, Mattermost posts, webhook POSTs, scheduled reminders and CLI chat input all enter through different adapters and produce the same command: a `SendUserMessage` routed to a session actor. The turn runs the same way regardless of source. ``` -┌──────────┐ -│ Slack │──┐ -├──────────┤ │ ┌──────────┐ ┌─────────────────┐ -│ Discord │──┼───▸│ Gateway │───▸│ Session Actor │ -├──────────┤ │ └──────────┘ └─────────────────┘ -│ Webhook │──┤ ▲ -├──────────┤ │ │ policy checks -│ CLI/TUI │──┘ │ before dispatch -└──────────┘ +┌───────────────┐ +│ Slack │──┐ +├───────────────┤ │ +│ Discord │──┤ +├───────────────┤ │ ┌──────────┐ ┌─────────────────┐ +│ Mattermost │──┼───▸│ Gateway │───▸│ Session actor │ +├───────────────┤ │ └──────────┘ └─────────────────┘ +│ Webhook │──┤ ▲ +├───────────────┤ │ │ policy checks +│ CLI / TUI │──┘ │ before dispatch +└───────────────┘ ``` -Channel adapters run inside the daemon as hosted services — in-process actors that maintain persistent connections to platform APIs. Slack uses [Socket Mode](https://api.slack.com/apis/socket-mode); Discord uses a WebSocket connection via [Discord.Net](https://discordnet.dev/). - -Adding a new channel means writing a new adapter. No changes to the session layer, the compaction logic, the tool system, or the security model. +Channel adapters run inside the daemon as hosted services: in-process actors holding persistent connections to platform APIs. Adding a channel means writing a new adapter. The session layer, compaction, the tool system and the security model stay untouched. ## Lifecycle -Sessions move through five explicit phases. Transitions are validated at runtime — illegal transitions throw an exception, not a silent fallback. - -``` -Recovering ──▸ Ready ◂──▸ Processing - │ ▴ ▴ │ - │ │ └─────────┘ - ▾ │ - Compacting - │ - ▾ - Passivating ──▸ Ready (on new message) +A session moves through five phases. The actor validates every transition, and an illegal one throws instead of falling back silently. + +```mermaid +stateDiagram-v2 + [*] --> Recovering + Recovering --> Ready: journal replayed + Ready --> Processing: message arrives + Processing --> Ready: turn done + Ready --> Compacting: threshold crossed + Processing --> Compacting: threshold crossed + Compacting --> Ready + Compacting --> Processing: buffered message + Ready --> Passivating: idle + Processing --> Passivating: idle + Passivating --> Ready: new message + Passivating --> [*]: stopped ``` -Ready can also transition directly to Compacting (if a threshold is crossed outside of a turn), and Processing can transition to Compacting mid-turn. +| Phase | What's happening | +|-------|------------------| +| **Recovering** | Journal replay and snapshot recovery. The actor rebuilds state before it accepts a message. | +| **Ready** | Accepting messages, with the idle timer ticking. | +| **Processing** | An LLM call or tool execution is in flight. Incoming messages are buffered, and [who joins the turn](#who-joins-a-turn) depends on who sent them. One turn at a time, always. | +| **Compacting** | The context window is getting full and the session is shrinking its history. Messages are buffered, as in Processing. | +| **Passivating** | The session is winding down: a final memory distillation (5-second grace period), a snapshot, then stop. A new message during passivation aborts it and returns to Ready. | + +## What ends or stalls a turn + +A turn ends when the model finishes, a timeout or watchdog fires, the daemon stops or drains, the [loop guard](#loop-guard) stops it, or a person's message ends an automation turn. A reply cut off at the output limit fails the turn: nothing cut off is posted or stored, and the error category is `ModelBehavior`. A model call that streams only thinking stops after three times `Session.NoProgressTimeoutSeconds`, which is 60 minutes by default. There's no ceiling on tool rounds. The timeouts exist because a hung provider shouldn't hold a session in Processing forever. All of them live under `Session` in `netclaw.json`, in seconds, and the [configuration reference](/configuration/reference/) lists every one. + +| Setting | Default | What it bounds | +|---------|---------|----------------| +| `Session.PrefillTimeoutSeconds` | 1800 | Wait for the first token. Covers queueing and prompt prefill, which can legitimately take minutes on self-hosted backends. | +| `Session.FirstTokenTimeoutSeconds` | 600 | Silence between streaming tokens once the first has arrived. | +| `Session.NoProgressTimeoutSeconds` | 1200 | Time without real output. A backend that sends keepalives but no tokens still gets cut off. | +| `Session.ToolExecutionTimeoutSeconds` | 90 | One tool call. It must produce its first result in this time, and each streamed item resets the clock. | +| `Session.MaxToolIterationsPerTurn` | 60 | Legacy. It still binds, but nothing enforces it, so a turn that keeps doing new work can run past 60 rounds. | +| `Session.SidecarLlmTimeoutSeconds` | 90 | Background calls: title generation, memory distillation, compaction notes. | +| `Session.TurnLlmTimeoutSeconds` | 180 | Fallback. Setting it away from 180 without a `FirstTokenTimeoutSeconds` makes it the silence limit between tokens. | +| `Session.IdleTimeout` | `01:00:00` | Time before an idle session passivates. `00:00:00` disables passivation. | + +A watchdog timeout fails the turn with a timeout error and the session returns to Ready, so the next message works. A slow 27B model behind a busy llama.cpp server is the usual case for raising the first two. An approval prompt isn't on this list because approvals don't time out. + +### Loop guard -**Recovering** — journal replay and snapshot recovery. The actor rebuilds its state from persisted events before accepting any new messages. +A model that repeats itself gets corrected before it burns the turn: -**Ready** — accepting user messages. Idle timeout is ticking. If no one talks to the session for long enough and no subscribers are connected, it passivates. +- **Repeated calls.** After two equal rounds, netclaw refuses the next identical call and returns a correction. The first four refused repeats get a nudge, the fifth a stronger one, and at the sixth the turn ends quietly with the model's text. The person sees no error. +- **Failed calls.** The guard also counts failed calls across different tools since the last batch that held a success. The model gets a nudge naming the repeated cause at a weighted score of 4 and a stronger one at 6. At 8, the next response that asks for tools ends the turn quietly. One success in a batch resets the count. Sub-agent runs use the same guard. -**Processing** — an LLM call or tool execution is in flight. Incoming messages get buffered and processed after the current turn completes. One turn at a time, always. +Only a new turn or new person input resets the counts. -**Compacting** — the context window is getting full. The session runs a [tiered compaction](#compaction) to shrink the history. Messages are buffered during compaction, same as during processing. +### Who joins a turn -**Passivating** — the session is winding down. It extracts any remaining observations to durable memory (with a 5-second grace period), takes a persistence snapshot, and stops the actor. A new message during passivation aborts the shutdown and returns to Ready — passivation is always interruptible. +Each input runs under its own sender's authority, and a turn keeps the authority it started with. + +- A new message from the same person joins the active turn at the next tool-batch boundary. +- A message from another person waits for its own turn. +- An automated input (a reminder, a job result, a webhook or a child notice) waits for its own turn, and it no longer closes an open approval prompt. +- When the operator or a trusted person writes during an automation turn, that turn ends after its current tool batch and the person's turn starts next. Its first model call carries a note that the automated turn stopped unfinished. An interrupted reminder records success and doesn't run its remaining steps. A message from an untrusted external person never ends a turn and waits. ## Multi-subscriber delivery -Multiple clients can attach to the same session simultaneously. This is how you can start a conversation in Slack, then connect to the same session from the CLI to watch tool calls and token usage in real time — each subscriber gets its own filtered view. +Several clients can attach to one session at once. Start a conversation in Slack, then attach from the CLI to watch tool calls and token usage live. Each subscriber gets its own filtered view. -Subscribers join with an `OutputFilter` bitmask that controls what they see: +Subscribers join with an `OutputFilter` bitmask: | Filter | What it includes | |--------|-----------------| | **Text** | Final assembled assistant replies | -| **TextStreaming** | Incremental token-by-token deltas | -| **Thinking** | Reasoning/thinking tokens | +| **TextStreaming** | Incremental token deltas | +| **Thinking** | Reasoning tokens | | **ToolCalls** | Tool call requests and results | -| **Usage** | Token counts and context window consumption | -| **Files** | File attachments produced by the LLM or tools | - -Lifecycle messages — turn completed, session title changes, errors, compaction notifications — are always delivered regardless of the filter. +| **Usage** | Token counts and context window use | +| **Files** | Attachments produced by the model or tools | +| **ProcessingState** | Busy and idle state, for typing indicators | -Adding or removing a subscriber doesn't affect other active subscribers. If a subscriber disconnects (the Slack adapter restarts, a CLI session closes), the actor detects it via a death watch and removes it from the subscriber list. No impact on the conversation. +Lifecycle messages (turn completed, title changes, errors, compaction notices) always arrive regardless of the filter. When a subscriber disconnects, the actor notices through a death watch and drops it. The conversation doesn't notice. ## Compaction -Sessions can run for hours or days. Without compaction, the [context window](https://platform.openai.com/docs/guides/context-window) fills up and the model starts losing earlier context. Netclaw uses a tiered compaction sequence that trades off cost against context preservation. +Sessions can run for days. Without compaction the [context window](https://platform.openai.com/docs/models) fills and the model loses its earlier context. Compaction starts when input tokens pass a fraction of the model's context window: `Session.Tuning.CompactionThreshold`, 0.75 by default. + +It runs in order: + +1. **Clear tool results.** Old tool output is replaced with placeholders, keeping the 3 most recent tool results in full (`KeepRecentToolResults`). No LLM call. +2. **Extractive reduction.** The oldest messages are dropped, keeping the most recent 6 (`KeepRecentMessages`). If what's left is still more than half the context window, the keep count is halved, down to a minimum of 2. +3. **Compaction notes.** One best-effort LLM call, on the Compaction model and failing over to the Fallback model when one is configured, reads the discarded messages and writes notes, which are inserted as a user-role message ahead of the kept ones. If the call fails or returns nothing, the extractive result stands. + +Tool calls stay paired with their results. After compaction the session saves a snapshot, resets its token counter, queues the notes as a [memory](/architecture/memory-model/#compaction-and-memory) checkpoint, and returns to Ready, or to Processing if messages are waiting. The static context layers are rendered again on the next call, as [context structure](/architecture/context-structure/) explains. + +## Storage on disk + +New sessions get a versioned storage envelope under `~/.netclaw/sessions/-/`. Directories appear as they're needed: + +| Path | What it's for | +|------|---------------| +| `workspace/` | The session's working directory (`session_dir` in its context) | +| `attachment-staging/` | Untrusted uploads, before they're copied into the workspace | +| `artifacts/` | Outputs the user or parent session should keep | +| `tmp/` | Windows only. On Linux and macOS the temp folder is `/tmp/netclaw-//p`, outside the envelope and cleaned up by the OS, not when the session ends. netclaw sets `TMPDIR`, `TMP` and `TEMP` for child processes to point at it. Keep anything that must last in `artifacts/`. | +| `worktrees/` | Git worktrees. The agent adds one with `git worktree add`, then calls `set_working_directory`. | +| `logs/session.log` | The log for the run. Tool-call arguments are redacted, and a long string such as a file body shows as a length and hash | +| `subagents//` | The same layout for each subagent run, with its own `artifacts/` and `logs/`. It also holds `task.md` (the agent, requester and brief), `result.md` (written when the run ends) and, for a reply cut off at the output limit, `artifacts/truncated-output.md` | -Compaction triggers when the input token count crosses a threshold — a configurable percentage of the model's context window, set via `CompactionThreshold` in `netclaw.json`. +Sessions that existed before 0.27.0 keep their old paths. netclaw doesn't move or rename their files. -**Phase 1: Tool result clearing.** The cheapest step — no LLM call required. Old tool results get replaced with placeholders while keeping the N most recent tool interactions in full detail. This preserves the reasoning chain ("I called this tool because...") while dropping the bulky output. Often sufficient on its own. +The database stores each session's folder path. If you move or restore the data directory somewhere else, netclaw re-roots the stored path onto the current `sessions/` directory and logs one warning naming the old and new path. -**Phase 2: Memory flush.** An LLM call extracts key facts, decisions, and action items from the conversation and persists them to durable memory storage. This happens *before* the lossy summarization step so that important context survives even if the summary misses it. +:::caution +A pre-0.27 binary can't resume a session that has a versioned storage binding. If you downgrade, sessions created under 0.27 or later won't resume. See [Upgrade netclaw](/guides/upgrading/) before moving backward, and [Data directory layout](/reference/data-directory/) for the full tree. +::: -**Phase 3: Structured summarization.** Another LLM call produces a structured summary of the conversation history covering the primary request, key decisions, pending tasks, and what to do next. The summary replaces the compacted messages. If a prior summary exists, it gets merged iteratively rather than replaced. +## Recovery and restart -Tool call/result pairs stay atomic during compaction. A tool call is never orphaned from its result. The compaction boundary always aligns to a user-message boundary to keep the conversation coherent. +The session journal lives in SQLite through [Akka.Persistence](https://getakka.net/articles/persistence/event-sourcing.html). It records more than turns: accepted input and who sent it, tool batches and their calls, tool approval requests and decisions, sub-agent runs and parked consent requests, compaction, title changes, and background job reaping. A journal or snapshot write that waits on SQLite times out at the connection's `Default Timeout` plus 10 seconds, and a failed save during a turn posts an error that says some actions may already have run. Snapshots are taken every 20 events and after every compaction. -After compaction, the session takes a persistence snapshot, resets its token counter, and returns to Ready (or Processing, if there are buffered messages waiting). +Recovery is automatic. When a session starts, from a restart, a crash or a lazy activation, it replays its journal from the latest snapshot and goes to Ready. Accepted input is journaled before it's acknowledged, so a crash doesn't lose a message you've been told arrived. -## Recovery and warm restart +Restarts come in two kinds. When you edit `netclaw.json`, the daemon drains active sessions, saves a manifest of them, restarts in-process, and warms those sessions back up before normal traffic resumes. A drain stops a tool loop at the next safe point and tells the person ("stopping" for `netclaw daemon stop`, "restarting" otherwise). A turn that stopped after its first tool batch resumes by itself after the restart, as long as no text reached the person, nothing else is waiting and every call has a result. Otherwise the person gets the stop notice. A graceful stop of the whole daemon is bounded to 30 seconds. Sessions that were idle stay cold and recover when someone talks to them. -Sessions persist their state to [SQLite](https://www.sqlite.org/docs.html) via [Akka.Persistence](https://getakka.net/articles/persistence/event-sourcing.html). Events (`TurnRecorded`, `SessionTitleSet`, `SessionCompacted`) are journaled. Snapshots capture the full session state periodically and after every compaction. +A running sub-agent parks and resumes under the original requester's authority. One that can't park in time ends as `owner_restarted`, and its tools don't run again. A graceful restart with an active child can take about 18 seconds longer. -Recovery is automatic. When a session actor starts — whether from a daemon restart, a crash, or a lazy activation — it replays its journal from the latest snapshot forward and transitions to Ready. +## Passivation, idle behavior and background work -For config-triggered restarts (you edited `netclaw.json`, the daemon reloads), the daemon goes further. Before shutting down, it persists a manifest of all active sessions. On startup, those sessions warm proactively — they recover from persistence before normal traffic resumes. The first message after restart includes a transient continuity notice so the model knows it's resuming, not starting fresh. +Passivation keeps the daemon's footprint bounded. After `Session.IdleTimeout` with nothing happening, the session distills memory one last time, takes a snapshot and stops. A new message during the 5-second grace period cancels it. -Sessions that were idle at restart time stay cold. They recover lazily when someone talks to them again. +Three things change the picture: -## Passivation +- **Running background jobs defer it.** A session with a live background job stays resident. Once the job is reaped or finished, the next idle timeout passivates it. +- **Subscribers don't.** A client attached to an idle session doesn't keep it alive. The channel binding stays subscribed until the session commits to stop, then drains its output and shuts down with it. +- **Pending approvals don't.** Approval state is journaled, so an idle session waiting on a person passivates, and the answer rehydrates it and resumes the parked tool batch. +- **Running sub-agents do.** A session with a live or unreported child run stays resident. A child parked on a consent request doesn't hold it, because the request is in the run record and a click wakes the session. -Passivation (idle shutdown) keeps the daemon's memory footprint bounded. Without it, every conversation you've ever had stays resident in memory as a live actor. +[Background jobs](/reference/tools/) (`shell_execute` with background mode) belong to the session that started them, and passivation kills any that are still running. They don't survive a daemon restart either. The next time that session wakes, it sees the job listed as reaped, with its output log still readable. Work that has to outlive the conversation belongs in a [reminder](/configuration/reminders/). -A new message during the passivation window cancels the shutdown and the session resumes normally. The session also defers passivation if subscribers are still connected or if tool interactions are pending — it won't shut down while someone is actively watching or waiting for a decision. +A session can also delegate to a [subagent](/skills/custom-subagents/): a child actor that runs one task in the background and never talks to the user. `spawn_agent` returns a run ID at once, the result arrives in a later turn as a notice that names `result.md`, and `check_agent_run` reads status or cancels. The child can't call `spawn_agent`, `check_agent_run` or `attach_file`, and can't write memory directly. -## Keep reading +## Related pages -- [Architecture Overview](/architecture/overview/) — the three-boundary design and actor internals -- [Context Structure](/architecture/context-structure/) — the layered context each turn assembles, and how it's ordered for caching -- [Design Philosophy](/architecture/design-philosophy/) — why sessions are transport-agnostic -- [Security Architecture](/architecture/security-model/) — how audience dispositions scope what a session can do -- [`netclaw sessions`](/cli/sessions/) — CLI commands for listing and resuming sessions +- [Architecture overview](/architecture/overview/) - the daemon, the actor system and the boundaries between them +- [Context structure](/architecture/context-structure/) - the layers each turn assembles, and how they're ordered for caching +- [Memory model](/architecture/memory-model/) - how a session's transcript becomes memories +- [`netclaw sessions`](/cli/sessions/) - list and resume sessions from the CLI +- [Configuration reference](/configuration/reference/) - every `Session` setting ## Resources -- [Actor model](https://en.wikipedia.org/wiki/Actor_model) — the concurrency model behind session actors -- [Akka.NET Persistence](https://getakka.net/articles/persistence/event-sourcing.html) — journal and snapshot recovery -- [SQLite](https://www.sqlite.org/docs.html) — the persistence backend -- [Slack Socket Mode](https://api.slack.com/apis/socket-mode) — how the Slack adapter connects -- [Discord.Net](https://discordnet.dev/) — the .NET library behind the Discord adapter +- [Actor model](https://en.wikipedia.org/wiki/Actor_model) - the concurrency model behind session actors +- [Akka.NET persistence](https://getakka.net/articles/persistence/event-sourcing.html) - journal and snapshot recovery +- [SQLite](https://www.sqlite.org/docs.html) - the persistence backend +- [Slack Socket Mode](https://api.slack.com/apis/socket-mode) - how the Slack adapter connects +- [Discord.Net](https://discordnet.dev/) - the .NET library behind the Discord adapter diff --git a/src/content/docs/channels/discord.md b/src/content/docs/channels/discord.md index 0e2a175..dd36cfe 100644 --- a/src/content/docs/channels/discord.md +++ b/src/content/docs/channels/discord.md @@ -1,55 +1,70 @@ --- title: "Discord" -description: "Connect Netclaw to your Discord server." +description: "Connect netclaw to your Discord server with a bot token, the Message Content intent and an allow-list." --- -Netclaw connects to Discord over the [Gateway WebSocket API](https://discord.com/developers/docs/events/gateway) — outbound connections only, no public URLs needed. You create a Discord bot, give netclaw one token, and it shows up in your server. +netclaw connects to Discord over the [Gateway WebSocket API](https://docs.discord.com/developers/topics/gateway), which means outbound connections only and no public URL. You create a Discord bot, give netclaw one token, and it shows up in your server and answers in threads. ## Prerequisites -- Netclaw installed and initialized ([`netclaw init`](/cli/init/)) -- A Discord server where you have the "Manage Server" permission -- [Developer Mode](https://support.discord.com/hc/en-us/articles/206346498) enabled in Discord (for copying IDs) +- netclaw installed and initialized ([`netclaw init`](/cli/init/)) +- A Discord server where you have the **Manage Server** permission +- [Developer Mode](https://support.discord.com/hc/en-us/articles/206346498) turned on in Discord, so you can copy IDs ## Create a Discord bot -Head to [discord.com/developers/applications](https://discord.com/developers/applications) and create a new application. +Open the [Discord Developer Portal](https://discord.com/developers/applications) and create an application. ![Discord Applications page](/screenshots/output/discord-setup-applications.png) +The Applications page lists your apps and has the **New Application** button. + ### 1. Create the application -Click **New Application**, give it a name, and hit Create. +Click **New Application**, name it, and click Create. ![New Application dialog](/screenshots/output/discord-setup-create-app.png) +The name you pick here is what members see on the bot. + ### 2. Copy the bot token -Go to **Bot** in the left sidebar. Click **Reset Token** to generate a bot token and copy it immediately — Discord only shows it once. +Open **Bot** in the sidebar, click **Reset Token**, and copy the token right away. Discord shows it once. ![Bot settings page](/screenshots/output/discord-setup-bot-settings.png) -If you lose the token, you can always reset it here, but you'll need to update your netclaw config with the new value. +If you lose it, reset it here and update netclaw with the new value. ### 3. Enable Message Content Intent -Scroll down on the Bot page to **Privileged Gateway Intents** and enable **Message Content**. Without this, netclaw receives message events but can't read their text. +Scroll to **Privileged Gateway Intents** and turn on **Message Content**. It's the only privileged intent netclaw needs, so leave Server Members and Presence off. Without it netclaw receives message events but can't read their text, except in DMs and when the message mentions the bot. ![Privileged Gateway Intents with Message Content enabled](/screenshots/output/discord-setup-bot-intents.png) -### 4. Set up OAuth2 scopes and permissions +The Message Content toggle sits at the bottom of the privileged intents list. + +### 4. Set OAuth2 scopes and permissions -Go to **OAuth2 > URL Generator**. Check the `bot` scope. +Open **OAuth2 > URL Generator** and check the `bot` scope. ![OAuth2 scopes selection](/screenshots/output/discord-setup-oauth-scopes.png) -Then select bot permissions: +Then select the bot permissions. ![Bot permissions checklist](/screenshots/output/discord-setup-bot-permissions.png) -**Required permissions:** Send Messages, Create Public Threads, Send Messages in Threads, Manage Threads, Embed Links, Read Message History, Add Reactions. +The permissions netclaw uses: -Manage Threads is needed so netclaw can rename threads with session titles as conversations progress. +| Permission | Why | +|------------|-----| +| Send Messages | Post replies | +| Create Public Threads | Open a thread on the first reply | +| Send Messages in Threads | Reply inside threads | +| Manage Threads | Rename a thread to the session title. Discord normally lets a thread's creator rename it, so this may be unnecessary. We haven't tested without it. | +| Read Message History | Backfill thread history after a restart | +| Attach Files | Upload files the agent produces | + +The checklist in the screenshot also ticks View Channels, Create Private Threads, Embed Links and Add Reactions. netclaw doesn't call Create Private Threads, Embed Links or Add Reactions, so you can leave those off. Keep View Channels on so the bot can see the channels you add it to. The table is the source of truth for what netclaw needs. ### 5. Install the bot to your server @@ -57,41 +72,37 @@ Copy the generated URL at the bottom of the page. ![Generated OAuth2 URL](/screenshots/output/discord-setup-oauth-url.png) -Open it in your browser. Discord asks which server to add the bot to — pick yours and click **Continue**. +Open it in a browser. Discord asks which server to add the bot to. Pick yours and click **Continue**. ![OAuth install approval dialog](/screenshots/output/discord-setup-install-dialog.png) -Discord then lists the permissions the bot is requesting. Confirm they match the set from step 4 and click **Authorize**. +The server list only offers servers where you have the Manage Server permission. + +Discord then lists the permissions the bot is requesting. Check they match step 4 and click **Authorize**. ![Discord OAuth2 permissions confirmation with the Authorize button](/screenshots/output/discord-setup-authorize-dialog.png) -The bot appears in your server's member list once the daemon is running. +The bot appears in the member list. It shows as online once the daemon is running. ### 6. Set Installation to None -Under **Installation** in the left sidebar, set the install method to **None**. This prevents users from installing the bot to other servers through Discord's app directory. +Under **Installation** in the sidebar, set the install method to **None**. That stops people installing the bot to other servers from the app directory. ![Installation settings set to None](/screenshots/output/discord-setup-installation-none.png) -If you want others to install via a link you control, use **Discord Provided Link** instead. - -![Installation with Discord Provided Link](/screenshots/output/discord-setup-installation-provided-link.png) +If you want others to install through a link you control, choose **Discord Provided Link** instead. ## Configure netclaw -Run [`netclaw config`](/cli/config/), navigate to **Channels**, and enable Discord. Enter your bot token when prompted — netclaw stores it encrypted and saves the config automatically. - -```bash -netclaw config -``` +Run [`netclaw config`](/cli/config/), open **Channels**, enable Discord, and enter the bot token. netclaw stores it encrypted and saves the config. ![The Channels area in netclaw config](/screenshots/output/config-channels-menu.png) -The Channels area in `netclaw config` — enable Discord and enter its bot token here (the adapter list also shows Slack and Mattermost). +The Channels area lists Slack, Discord and Mattermost with a checkbox each. Once an adapter is set up, its row also shows how many channels and users it allows. ### Manual setup -For scripted or headless installs, store the token with [`netclaw secrets`](/cli/secrets/) and edit the config file directly: +For scripted or headless installs, store the token with [`netclaw secrets`](/cli/secrets/): ```bash netclaw secrets set Discord.BotToken your-bot-token @@ -108,19 +119,32 @@ Then enable Discord in `~/.netclaw/config/netclaw.json`: } ``` -Environment variables work too — useful for Docker and CI: +Environment variables work too, with list entries indexed from zero: ```bash export NETCLAW_Discord__BotToken="your-bot-token" export NETCLAW_Discord__Enabled="true" +export NETCLAW_Discord__AllowedChannelIds__0="123456789012345678" ``` +An edit to `netclaw.json` restarts the daemon by itself. A vault or environment change needs a restart you run: + +| Where it runs | Restart | +|---------------|---------| +| Native | `netclaw daemon stop`, then `netclaw daemon start` | +| systemd user unit | `systemctl --user restart netclaw` | +| Docker | `docker restart ` | + +:::caution +Keep each list in one place. Environment entries and `netclaw.json` entries merge by position, so a blank `NETCLAW_Discord__AllowedChannelIds__0` overwrites the first channel in the file. `netclaw config` → Channels writes the lists into `netclaw.json`. +::: + ### How channel IDs are stored -Enter channel names or IDs (comma-separated) in `netclaw config` → Channels. Netclaw resolves each entry against the Discord API to its canonical channel ID before saving. The stored `AllowedChannelIds` field holds IDs, not display names; display names are shown dynamically in the config UI. Names that cannot be resolved remain in `AllowedChannelIds` verbatim and are flagged with a warning — they are inert until the bot can see the channel, at which point a background refresh rewrites them to canonical IDs. Entries added one-at-a-time via the add-channel flow are rejected at entry time if they do not resolve. A genuine auth or network failure blocks the entire save. +Enter channel names or IDs, comma-separated, in `netclaw config` → Channels. netclaw resolves each against the Discord API and saves the canonical ID. `AllowedChannelIds` holds IDs, never display names. A name that can't be resolved stays verbatim with a warning and matches nothing until the bot can see the channel, then a background refresh rewrites it to the ID. A channel added one at a time is rejected on the spot if it doesn't resolve, and a real auth or network failure blocks the whole save. :::caution -**Breaking change (pre-0.24.0 → 0.24.0+):** If you previously placed display names in `AllowedChannelIds`, those values no longer match incoming channel IDs and messages will be silently dropped. Re-enter the channels via `netclaw config` → Channels so they resolve to canonical IDs. `DefaultChannelId` is already a canonical channel ID and is not affected. +Since 0.24.0, channel display names you put in `AllowedChannelIds` by hand no longer match, and messages from those channels are silently dropped. Re-enter them through `netclaw config` → Channels. `DefaultChannelId` is already an ID. ::: ### All config fields @@ -128,16 +152,17 @@ Enter channel names or IDs (comma-separated) in `netclaw config` → Channels. N | Field | Type | Default | Description | |-------|------|---------|-------------| | `Enabled` | bool | `false` | Turn on Discord | -| `BotToken` | string | — | Bot token from the Developer Portal. Store with `netclaw secrets set`. | -| `DefaultChannelId` | string | — | Channel ID for the default channel | -| `MentionOnly` | bool | `true` | Only respond when @-mentioned | +| `BotToken` | string | - | Bot token from the Developer Portal. Store with `netclaw secrets set`. | +| `DefaultChannelId` | string | - | Channel ID that's allowed without being listed in `AllowedChannelIds` | +| `MentionOnly` | bool | `true` | In channels, answer only when @-mentioned | | `AllowDirectMessages` | bool | `false` | Accept DMs | -| `MentionRequiredInDm` | bool | `false` | Require @-mention even in DMs | -| `AllowedChannelIds` | string[] | `[]` | Channel allow-list. Empty + no default = all channel messages denied | -| `AllowedUserIds` | string[] | `[]` | User allow-list. Empty = everyone in allowed channels is accepted | -| `ChannelAudiences` | object | `{}` | Per-channel [audience](/security/security-model/) overrides. Keys are channel IDs or `"dm"`. Values: `"personal"`, `"team"`, `"public"`. | +| `MentionRequiredInDm` | bool | `false` | Require an @-mention even in DMs | +| `MentionRequiredInThreadByChannel` | object | `{}` | Channel ID to bool. `true` makes a thread reply in that channel need an @-mention. See [Mention behavior](#mention-behavior). | +| `AllowedChannelIds` | string[] | `[]` | Channel allow-list. Empty with no default denies every channel message. | +| `AllowedUserIds` | string[] | `[]` | User allow-list. Empty accepts everyone in allowed channels. | +| `ChannelAudiences` | object | `{}` | Per-channel [audience](/security/security-model/) overrides. Keys are channel IDs or `"dm"`. Values are `"personal"`, `"team"` or `"public"`. | -The token lives in `~/.netclaw/config/secrets.json` (encrypted at rest). +The token lives in `~/.netclaw/config/secrets.json`, encrypted at rest. ## Access control @@ -145,11 +170,7 @@ Discord ACL is **default-deny**. Three settings decide who talks to the bot and ### Channels -The bot only responds in channels that pass the allow-list: - -- `DefaultChannelId` allows one channel -- `AllowedChannelIds` allows multiple -- **If both are empty, every channel message is denied.** This is the number one setup mistake. +The bot answers only in channels that pass the allow-list. `DefaultChannelId` allows one, `AllowedChannelIds` allows more, and if both are empty every channel message is denied. That's the number one setup mistake. ```json { @@ -160,40 +181,25 @@ The bot only responds in channels that pass the allow-list: } ``` -Finding channel IDs: right-click a channel in Discord with Developer Mode on, click "Copy Channel ID." +To find a channel ID with Developer Mode on, right-click the channel and choose **Copy Channel ID**. ### Users -`AllowedUserIds` restricts who gets responses: - -- Empty (default) — everyone in allowed channels is accepted -- Non-empty — only listed user IDs get responses, everyone else is silently dropped - -Finding user IDs: right-click a user in Discord with Developer Mode on, click "Copy User ID." +`AllowedUserIds` restricts who gets answers. Empty means everyone in an allowed channel. Non-empty means only the listed users, and everyone else is dropped silently. To copy a user ID, right-click the user and choose **Copy User ID**. -Users in `AllowedUserIds` are treated as `TrustedInternal` by the [security model](/security/security-model/). Everyone else is `UntrustedExternal`. +Listed users are `TrustedInternal` in the [security model](/security/security-model/), and everyone else is `UntrustedExternal`. ### Direct messages -DMs are off by default: - -```json -{ - "Discord": { - "AllowDirectMessages": true - } -} -``` - -With DMs on, users can just type normally — `MentionRequiredInDm` defaults to `false`, so no @-mention needed. Each user gets a single long-running DM session (unlike channels, where each root message starts a new session). +DMs are off by default. Turn them on with `"AllowDirectMessages": true`. Users then write without a mention, because `MentionRequiredInDm` defaults to `false`. Each user gets one long-running DM session, unlike channels, where each root message starts a new session. :::caution -With `AllowDirectMessages: true` and `AllowedUserIds` empty, any server member can DM the bot. Lock down `AllowedUserIds` if that's not what you want. +With DMs on and `AllowedUserIds` empty, any server member can DM the bot. Set `AllowedUserIds` unless that's what you want. ::: ### Audience overrides -Override the default audience per-channel with `ChannelAudiences`: +Audience resolves per message: `ChannelAudiences[channelId]` first, then `ChannelAudiences["dm"]`, then a fallback. The fallback is `Team` for a channel listed in `AllowedChannelIds` or a user listed in `AllowedUserIds`, and `Public` for everything else. A channel allowed only through `DefaultChannelId` is `Public` unless you also list it in `AllowedChannelIds` or set `ChannelAudiences` for it. ```json { @@ -206,105 +212,108 @@ Override the default audience per-channel with `ChannelAudiences`: } ``` -The `"dm"` key is reserved — it matches every direct message rather than a channel ID. A channel-ID entry takes precedence over it. An unrecognized audience value is rejected outright: the message is denied rather than falling back to a default. - -[Security Model](/security/security-model/) has the full breakdown on how audiences map to tools and permissions. +A channel-ID entry beats `"dm"`, and an unrecognized value denies the message instead of falling back. [Security model](/security/security-model/) maps audiences to tools. ## Behavior in Discord ### Threads and sessions -When someone messages the bot in a regular channel, netclaw creates a public thread on its first reply and continues the conversation there. The thread starts as "Netclaw" and netclaw renames it once the LLM generates a session title. - -Thread replies don't need a @-mention — if there's an active session in the thread, the bot responds to everything. +When someone messages the bot in a channel, netclaw creates a public thread on its first reply and continues there. The thread starts as "Netclaw" and is renamed once the model produces a session title. -Idle conversations are freed from memory after 2 hours (individual sessions after 1 hour, unless an approval request is pending). On daemon restart, up to 200 messages of thread history are backfilled so in-progress conversations resume. +A thread is freed from memory when its session goes idle, after `Session.IdleTimeout` (one hour by default), and a later reply wakes it again. After a daemon restart, netclaw backfills up to 200 messages of thread history so conversations resume. ### Mention behavior -`MentionOnly: true` (the default) means the bot ignores messages that don't @-mention it. Two exceptions: +`MentionOnly: true` makes the bot ignore channel messages that don't @-mention it. netclaw strips the mention before the text reaches the model. These rules sit on top: -- **Thread replies** — if a thread already has an active session, the bot responds without needing a mention -- **Daemon restart recovery** — if the daemon restarts and a user continues a thread, the session is re-created from the thread's message history +- **Thread replies:** once a thread has an active session, replies in it don't need a mention, unless `MentionRequiredInThreadByChannel` is `true` for that channel. +- **Restart recovery:** if the daemon restarts and someone continues a thread, netclaw recreates the session from the thread's history. -Netclaw strips the @-mention before passing text to the LLM. +`MentionRequiredInThreadByChannel` maps channel IDs to `true` or `false`: + +```json +{ + "Discord": { + "MentionRequiredInThreadByChannel": { + "123456789012345678": true + } + } +} +``` + +With the rule on, an un-mentioned thread reply is ignored (`ThreadMentionRequired` in the log). The next message that mentions the bot makes netclaw re-read everything posted since its last reply, so it answers with full context. A channel with no entry behaves as `false`. The rule doesn't apply to DMs. + +In `netclaw config` → Channels, **Space** toggles **Require @mention** on the selected channel. A channel you add there starts with the rule on if its audience is `team` or `public`, and off if it's `personal`. Only `true` entries are saved. ### Message formatting -Discord natively renders markdown — bold, italic, code blocks, headers, lists, links. Responses longer than 2,000 characters are split at newline boundaries. +Discord renders markdown natively. netclaw splits replies longer than 2,000 characters at newline boundaries. ### Tool approval -When a tool call needs approval, netclaw posts an interactive button prompt in the thread. The prompt shows the tool name, action, and pattern(s). The full set is five buttons: **Once**, **This chat**, **Always here**, **Always anywhere**, and **Deny** — though **Always here** is omitted when the working directory is a session scratch path or too shallow for a folder-scoped grant. **Always anywhere** grants the command everywhere, so use it sparingly. netclaw shows fewer buttons when some don't apply — a command it can't cleanly parse (shell control flow, or unbalanced quotes) drops to just **Once** and **Deny**. Only the user who triggered the request can approve. - -If posting the button prompt fails, netclaw falls back to a text prompt where you reply with the letter shown next to each option. +When a tool call needs approval, netclaw posts a prompt with buttons in the thread. It shows the tool, the action and the pattern. The choices are **Once**, **This chat**, **Always here**, **Always anywhere**, **Deny**, and **This repository** when it applies. A command netclaw can't parse cleanly gets only **Once** and **Deny**. [How tool approvals work](/security/approvals/) explains what each one grants. Only the user whose request triggered the prompt can answer it. -After a decision, netclaw updates the original message in-place with a checkmark or denied icon. +If posting the buttons fails for a prompt from the main agent, netclaw falls back to a text prompt, and you reply with the letter next to an option. A prompt from a sub-agent has buttons only and no letter fallback. If its buttons can't be posted, netclaw refuses the request and the call doesn't run. After a decision, netclaw edits the original message with a check mark or a denied icon. A prompt that nothing can answer any more is edited to remove the buttons and say why, for example `This approval request was withdrawn because Netclaw restarted before the task finished. Nothing ran.` ### Reminders -Discord reminders deliver to guild text channels only — user DMs are not supported as reminder targets. Accepted formats: +[Reminders](/configuration/reminders/) deliver to a guild text channel or to a user's DM. Accepted targets: -- `<#123456789012345678>` — Discord channel mention -- `channel:123456789012345678` — explicit channel prefix +| Target | Delivers to | +|--------|-------------| +| `<#123456789012345678>` | A channel, by mention | +| `channel:123456789012345678` | A channel, by ID | +| `<@123456789012345678>` or `<@!123456789012345678>` | A user's DM, by mention | +| `dm:123456789012345678` or `@123456789012345678` | A user's DM, by ID | + +A bare ID is rejected, because channel IDs and user IDs have the same shape. DM delivery still goes through ACL, so it needs `AllowDirectMessages: true` and, if you use `AllowedUserIds`, the user on that list. `current_session` reminders post back into the thread that set them. That reminder ends when the session admits the fire, and netclaw doesn't retry a failed post. If the reply post fails for good, the session logs an Error and raises a `reminder.delivery.failed` [alert](/observability/operational-alerts/). ### Proactive messaging -The LLM can start a conversation with the generic `send_channel_message` tool — pass `channel_key: discord` and a `destination` (a guild text channel or a DM) resolved via `lookup_channel_user` or `lookup_channel_destination`. ACL rules still apply. +The model can start a conversation with `send_channel_message`, passing `channel_key: discord` and a destination (a guild text channel or a DM) from `lookup_channel_user` or `lookup_channel_destination`. ACL still applies. ### Ignored messages -The bot drops: empty messages (no text, no attachments), other bots' messages, its own messages, DMs when `AllowDirectMessages` is off, and un-mentioned channel messages when `MentionOnly` is on and there's no active thread. - -Messages over 4,000 characters are truncated. +The bot drops empty messages (no text, no attachments), its own and other bots' messages, DMs when `AllowDirectMessages` is off, and un-mentioned channel messages when `MentionOnly` is on and there's no active thread. ## Verify it works -Restart the daemon and check status: - ```bash -netclaw daemon stop && netclaw daemon start netclaw status ``` -Discord should show `connected`. If it shows `disabled` or `disconnected`, check the [troubleshooting guide](/channels/troubleshooting/). +Discord should show `healthy`. If it shows `disabled` or `disconnected`, see [Channel troubleshooting](/channels/troubleshooting/). -Then @-mention the bot in an allowed channel. If it creates a thread and responds, you're set. +Then @-mention the bot in an allowed channel. If it creates a thread and answers, you're set. :::note -`netclaw doctor` doesn't validate Discord tokens yet. For now, use `netclaw status` and check the daemon logs. +`netclaw doctor` doesn't check Discord yet. Use `netclaw status` and the daemon log. ::: ## Troubleshooting -Common problems and fixes are in [Channel Troubleshooting](/channels/troubleshooting/). The hits: +[Channel troubleshooting](/channels/troubleshooting/) is organized by symptom. The usual ones: -- **Connected but silent** — `AllowedChannelIds` is empty and no `DefaultChannelId` is set, so all traffic gets denied -- **Invalid bot token** — HTTP 401 on startup means the token is wrong or was regenerated in the Developer Portal -- **Gateway disconnected** — the daemon couldn't establish a connection. Usually a network issue or [Discord outage](https://discordstatus.com/). -- **Message Content privileged intent not enabled** — turn it on under Privileged Gateway Intents in the [Developer Portal](https://discord.com/developers/applications) +| Symptom | Likely cause | +|---------|--------------| +| Connected but silent | `AllowedChannelIds` is empty and no `DefaultChannelId` is set. | +| Messages arrive but the text is empty | Message Content intent is off in the Developer Portal. | +| HTTP 401 at startup | The token is wrong or was reset. | +| Startup waits 30 seconds, then `disconnected` | The gateway never sent READY, often because the bot isn't in any server. | +| Stopped answering in a thread | `MentionRequiredInThreadByChannel` is on for the channel. Mention the bot. | -## Next steps +## Related pages -- [Configure audiences and approval gates](/security/security-model/) to control what tools are available in each channel -- [Set up systemd](/deployment/systemd/) so the daemon stays running after reboots -- [Add Slack](/channels/slack/) or [Mattermost](/channels/mattermost/) if your team uses more than one +- [Slack](/channels/slack/) and [Mattermost](/channels/mattermost/) - the other chat channels +- [Channel troubleshooting](/channels/troubleshooting/) - fixes by symptom +- [How tool approvals work](/security/approvals/) - the approval choices +- [Security model](/security/security-model/) - audiences and approval gates +- [`netclaw status`](/cli/status/) - connector health -## Related pages +## Resources -- [`netclaw config`](/cli/config/) — Channels -- [`netclaw secrets`](/cli/secrets/) — token management -- [`netclaw status`](/cli/status/) — primary diagnostic tool for Discord -- [Security Model](/security/security-model/) — audiences and approval gates -- [Channel Troubleshooting](/channels/troubleshooting/) — error codes and debug logging -- [Slack](/channels/slack/) — sibling channel integration -- [Mattermost](/channels/mattermost/) — sibling channel integration - -## External resources - -- [Discord Developer Portal](https://discord.com/developers/applications) — create and manage your bot -- [Discord Developer Docs: Getting Started](https://discord.com/developers/docs/getting-started) — bot setup walkthrough -- [Discord: Privileged Intents](https://discord.com/developers/docs/events/gateway#privileged-intents) — Message Content intent setup -- [Discord: Bot Permissions](https://discord.com/developers/docs/topics/permissions) — permission reference -- [Discord: Enable Developer Mode](https://support.discord.com/hc/en-us/articles/206346498) — how to copy channel and user IDs -- [Discord Status](https://discordstatus.com/) — check for outages +- [Discord Developer Portal](https://discord.com/developers/applications) - create and manage your bot +- [Discord: Gateway and privileged intents](https://docs.discord.com/developers/topics/gateway) - what Message Content does +- [Discord: permissions](https://discord.com/developers/docs/topics/permissions) - permission reference +- [Discord: enable Developer Mode](https://support.discord.com/hc/en-us/articles/206346498) - copy channel and user IDs +- [Discord Status](https://discordstatus.com/) - check for outages diff --git a/src/content/docs/channels/mattermost.md b/src/content/docs/channels/mattermost.md index bc29667..904c290 100644 --- a/src/content/docs/channels/mattermost.md +++ b/src/content/docs/channels/mattermost.md @@ -1,81 +1,97 @@ --- title: "Mattermost" -description: "Connect Netclaw to your self-hosted Mattermost server." +description: "Connect netclaw to a self-hosted Mattermost server with a bot account, an allow-list and text or button approvals." --- -Netclaw connects to Mattermost over a WebSocket for events and the REST API for replies. You run a bot account on your Mattermost server, give netclaw the bot's access token, and it shows up as a bot user — same default-deny ACL model as [Slack](/channels/slack/) and [Discord](/channels/discord/), against infrastructure you host yourself. +netclaw connects to Mattermost over a WebSocket for events and the REST API for replies. You create a bot account on your server, give netclaw the bot's access token, and it answers in threads under the same default-deny ACL as [Slack](/channels/slack/) and [Discord](/channels/discord/). If you want the whole stack in containers, [Run netclaw with Mattermost on Docker Compose](/guides/mattermost-docker-compose/) sets up the server, the bot and netclaw with one sample. ## Prerequisites -- Netclaw installed and initialized ([`netclaw init`](/cli/init/)) -- A self-hosted Mattermost server reachable from the daemon -- System Console access on that server (to enable bot accounts and create the bot) +- netclaw installed and initialized ([`netclaw init`](/cli/init/)) +- A Mattermost server the daemon can reach, with a user who has System Console access +- The server's base URL as netclaw will reach it, such as `https://mm.example.com` or `http://mattermost:8065` on a Compose network ## Create a bot account -Mattermost bots authenticate with a **personal access token**, not OAuth. Two server settings have to be on before you can create one. +Mattermost bots authenticate with a **personal access token**, not OAuth. Two server settings are off by default and both must be on before you can create one. ### 1. Enable bot accounts and access tokens -In the **System Console**, under **Integrations > Bot Accounts**, set **Enable Bot Account Creation** to `true`. Then, under **Integrations > Integration Management**, turn on **Enable Personal Access Tokens**. +In the **System Console**, open **Integrations > Bot Accounts** and set **Enable bot account creation** to `true`. Then open **Integrations > Integration Management** and turn on **Enable personal access tokens**. Both map to `ServiceSettings` keys (`EnableBotAccountCreation` and `EnableUserAccessTokens`) if you'd rather set them in `config.json` or with `MM_SERVICESETTINGS_*` environment variables. -![System Console Bot Accounts settings with Enable Bot Account Creation set to True](/screenshots/output/mattermost-bot-accounts.png) +![System Console Bot Accounts settings with Enable Bot Account Creation set to True and greyed out under a notice that an environment variable set it](/screenshots/output/mattermost-bot-accounts.png) -Both are off by default on a fresh Mattermost install — skipping either is the most common setup snag. +This server was configured through an environment variable, so the setting is locked and the console says so. On your server it's a normal radio button, and skipping this toggle or the token one is the most common setup snag. ### 2. Create the bot and copy its token -Under **Integrations > Bot Accounts**, click **Add Bot Account** and give it a username (e.g. `netclaw`) and display name. Then click **Create New Token**: Mattermost reveals the access token **once**, in a banner — copy it right then. That token (not the Token ID shown in the list) is the `BotToken` netclaw needs. +Under **Integrations > Bot Accounts**, click **Add Bot Account**, give it a username such as `netclaw` and a display name, then click **Create New Token**. Mattermost shows the access token **once**, in a banner. Copy it then. That token, not the Token ID in the list, is the `BotToken` netclaw needs. -![Mattermost Bot Accounts page listing the netclaw bot with its Create New Token control](/screenshots/output/mattermost-create-bot.png) +![Mattermost Integrations > Bot Accounts list with the netclaw bot, its Create New Token, Edit and Disable links, and a token row whose Token ID is masked](/screenshots/output/mattermost-create-bot.png) + +The list is what you see after the bot exists. The **Create New Token** link next to the bot issues a replacement later, and the Token ID in the row, masked here, is not the token netclaw needs. + +The [Compose sample](/guides/mattermost-docker-compose/) provisions this for you, and the bot row ends up looking like this. + +![The netclaw bot row in Mattermost Bot Accounts, managed by admin, with one token described as netclaw and its Token ID masked](/screenshots/output/mattermost-compose-bot-account.png) + +The token description is `netclaw`, and the sample keeps the token itself out of every file in the repository. ### 3. Add the bot to your team and channels -A bot only sees channels it belongs to. Add it to the target team, then invite it to each channel where it should respond (`/invite @netclaw`, or through the channel member menu). +A bot only sees channels it belongs to. Add it to the team, then invite it to each channel where it should answer with `/invite @netclaw` or the channel member menu. ## Configure netclaw :::tip -Run [`netclaw config`](/cli/config/) → Channels. Enable Mattermost, enter your server URL and bot token — netclaw resolves channel names to canonical IDs and saves. You no longer need to look up opaque IDs by hand. +Run [`netclaw config`](/cli/config/) → Channels, enable Mattermost, and enter the server URL and bot token. netclaw resolves channel names to IDs and saves. ::: ![The Channels area in netclaw config](/screenshots/output/config-channels-menu.png) -The Channels area in `netclaw config` — enable Mattermost and enter its server URL and token here (the adapter list also shows Slack and Discord). +The Channels area lists Slack, Discord and Mattermost with a checkbox each. Toggle Mattermost on and press `Enter` to configure it. ### Manual setup -For scripted deployments or Docker installs where a TUI isn't available, store the token with [`netclaw secrets`](/cli/secrets/): +For Docker or scripted installs, store the token with [`netclaw secrets`](/cli/secrets/) and describe the rest in config: ```bash netclaw secrets set Mattermost.BotToken your-bot-token ``` -Then enable Mattermost and point it at your server in `~/.netclaw/config/netclaw.json`: +The token lands in your shell history and the process list this way. In scripts, feed it over stdin instead, as the [Compose guide](/guides/mattermost-docker-compose/) does: `netclaw secrets set Mattermost.BotToken "$(cat)" < .bot-token`. ```json { "Mattermost": { "Enabled": true, "ServerUrl": "https://mm.example.com", - "DefaultChannelId": "abcdefghijklmnopqrstuvwxyz" + "AllowedChannelIds": ["abcdefghijklmnopqrstuvwxyz"], + "AllowedUserIds": ["zyxwvutsrqponmlkjihgfedcba"] } } ``` -Environment variables work too: +Every key also works as an environment variable with the `NETCLAW_` prefix and `__` between segments, with list entries indexed from zero: ```bash -export NETCLAW_Mattermost__BotToken="your-bot-token" -export NETCLAW_Mattermost__Enabled="true" +NETCLAW_Mattermost__Enabled=true +NETCLAW_Mattermost__ServerUrl=http://mattermost:8065 +NETCLAW_Mattermost__AllowedChannelIds__0=abcdefghijklmnopqrstuvwxyz ``` +Restart behavior differs by source. Editing `netclaw.json` makes the daemon restart itself. A change to the vault or to an environment variable needs a restart you do yourself: `systemctl --user restart netclaw`, or `docker restart ` in Docker. + +:::caution +Keep each list in one place. Environment entries and `netclaw.json` entries merge by position, so a blank `NETCLAW_Mattermost__AllowedChannelIds__0` overwrites the first channel in the file and denies it. `netclaw config` → Channels also writes `AllowedChannelIds` and `AllowedUserIds` into `netclaw.json`. +::: + ### How channel IDs are stored -Add channels one at a time in `netclaw config` → Channels. Netclaw resolves each entry against the Mattermost REST API to its canonical channel ID before adding it to the list. The stored `AllowedChannelIds` field holds IDs, not display names; display names are shown dynamically in the config UI. Entries that cannot be resolved are flagged with a warning but saved verbatim in `AllowedChannelIds` — an unresolved name is inert at runtime because the ACL matches against canonical IDs only, so the name grants access to no channel until it resolves. +Add channels one at a time in `netclaw config` → Channels. netclaw resolves each entry against the Mattermost REST API to a canonical channel ID before it adds it. `AllowedChannelIds` holds IDs, never display names. An entry that can't be resolved is flagged and kept verbatim, and it matches no channel until it resolves. When you save, the first channel in the list also becomes `DefaultChannelId`. :::caution -**Breaking change (pre-0.24.0 → 0.24.0+):** If you previously placed display names in `AllowedChannelIds` by hand in `netclaw.json`, those names no longer match any channel ID, so those channels are silently denied. Re-enter them via `netclaw config` → Channels so they resolve to canonical IDs. (`DefaultChannelId` already holds a canonical ID, so it's unaffected.) +Since 0.24.0, display names you typed into `AllowedChannelIds` by hand no longer match, and those channels are silently denied. Re-enter them through `netclaw config` → Channels. `DefaultChannelId` already holds an ID. ::: ### All config fields @@ -83,35 +99,31 @@ Add channels one at a time in `netclaw config` → Channels. Netclaw resolves ea | Field | Type | Default | Description | |-------|------|---------|-------------| | `Enabled` | bool | `false` | Turn on Mattermost | -| `ServerUrl` | string | — | Base URL of your Mattermost server. Required when enabled. | -| `BotToken` | string | — | Bot personal access token. Store with `netclaw secrets set`. | -| `CallbackUrl` | string | — | URL Mattermost can reach for interactive button callbacks. Set it to get approval buttons; leave it unset for text-reply approvals (see [Tool approval](#tool-approval)). | -| `DefaultChannelId` | string | — | A 26-character channel ID. Allows one channel without listing it in `AllowedChannelIds`. | -| `MentionOnly` | bool | `true` | Only respond when @-mentioned in channels | +| `ServerUrl` | string | - | Base URL of your server. Required when enabled. | +| `BotToken` | string | - | Bot access token. Vault only (`netclaw secrets set Mattermost.BotToken`). It isn't a `netclaw.json` key, and the schema rejects it there. | +| `CallbackUrl` | string | - | URL the Mattermost server can reach for approval buttons. Required for sub-agent approvals. Unset means text-reply approvals for the main agent only (see [Tool approval](#tool-approval)). | +| `DefaultChannelId` | string | - | A 26-character channel ID that's allowed without being listed in `AllowedChannelIds` | +| `MentionOnly` | bool | `true` | In channels, answer only when @-mentioned | | `AllowDirectMessages` | bool | `false` | Accept DMs | -| `MentionRequiredInDm` | bool | `false` | Require @-mention even in DMs | -| `AllowedChannelIds` | string[] | `[]` | Channel allow-list. Empty + no default = all channel messages denied | -| `AllowedUserIds` | string[] | `[]` | User allow-list. Empty = everyone in allowed channels is accepted | -| `ChannelAudiences` | object | `{}` | Per-channel [audience](/security/security-model/) overrides. Keys are channel IDs or `"dm"`. Values: `"personal"`, `"team"`, `"public"`. | +| `MentionRequiredInDm` | bool | `false` | Require an @-mention even in DMs | +| `MentionRequiredInThreadByChannel` | object | `{}` | Channel ID to bool. `true` makes a thread reply in that channel need an @-mention. See [Mention behavior](#mention-behavior). | +| `AllowedChannelIds` | string[] | `[]` | Channel allow-list. Empty with no default denies every channel message. | +| `AllowedUserIds` | string[] | `[]` | User allow-list. Empty accepts everyone in allowed channels. | +| `ChannelAudiences` | object | `{}` | Per-channel [audience](/security/security-model/) overrides. Keys are channel IDs or `"dm"`. Values are `"personal"`, `"team"` or `"public"`. | -The token lives in `~/.netclaw/config/secrets.json` (encrypted at rest). Mattermost channel and user IDs are 26-character alphanumeric strings — the System Console lists them, or pull them from the [REST API](https://api.mattermost.com/). +The token lives in `~/.netclaw/config/secrets.json`, encrypted at rest. Channel and user IDs are 26-character strings. You can read them with `mmctl --local user list` and `mmctl --local channel search --team ` inside the server container, or from the [REST API](https://api.mattermost.com/). ## Access control -Mattermost ACL is **default-deny**, evaluated the same way as Slack and Discord. +Mattermost ACL is **default-deny** and works like Slack and Discord. ### Channels -The bot only responds in channels that pass the allow-list: - -- `DefaultChannelId` allows one channel -- `AllowedChannelIds` allows multiple -- **If both are empty, every channel message is denied.** This is the number one setup mistake. +The bot answers only in channels that pass the allow-list. `DefaultChannelId` allows one, `AllowedChannelIds` allows more, and if both are empty every channel message is denied. That's the number one setup mistake. ```json { "Mattermost": { - "DefaultChannelId": "abcdefghijklmnopqrstuvwxyz", "AllowedChannelIds": ["abcdefghijklmnopqrstuvwxyz", "zyxwvutsrqponmlkjihgfedcba"] } } @@ -119,137 +131,182 @@ The bot only responds in channels that pass the allow-list: ### Users -`AllowedUserIds` restricts who gets responses: - -- Empty (default) — everyone in allowed channels is accepted -- Non-empty — only listed user IDs get responses, everyone else is silently dropped - -Users in `AllowedUserIds` are treated as `TrustedInternal` by the [security model](/security/security-model/). Everyone else is `UntrustedExternal`. +`AllowedUserIds` restricts who gets answers. Empty means everyone in an allowed channel. Non-empty means only the listed users, and everyone else is dropped silently, in DMs too. Listed users are `TrustedInternal` in the [security model](/security/security-model/), and everyone else is `UntrustedExternal`. :::caution -Always set `AllowedUserIds` on a production server. With it empty, any user the bot can see can click an approval button in a shared channel and approve a tool call on someone else's behalf. The action-token check still blocks *forged* clicks — but a legitimate click from the wrong person is a real one. +Always set `AllowedUserIds` on a shared server. With it empty, any user in an allowed channel can talk to the bot and start tool calls under that channel's audience. Only the user who triggered a prompt can approve it, but an empty list lets anyone be that user. ::: ### Direct messages -DMs are off by default: +DMs are off by default. Turn them on with `"AllowDirectMessages": true`. Users then write without a mention, because `MentionRequiredInDm` defaults to `false`. With `AllowedUserIds` empty, anyone on the server can DM the bot. + +### Audience overrides + +Audience resolves per message: `ChannelAudiences[channelId]` first, then `ChannelAudiences["dm"]` for direct messages, then a fallback. The fallback is `Team` for a channel listed in `AllowedChannelIds` or a user listed in `AllowedUserIds`, and `Public` for everything else. A DM from someone who isn't on the list is `Public`, and so is a channel allowed only through `DefaultChannelId`, unless you also list it in `AllowedChannelIds` or set `ChannelAudiences` for it. ```json { "Mattermost": { - "AllowDirectMessages": true + "ChannelAudiences": { + "abcdefghijklmnopqrstuvwxyz": "personal", + "dm": "personal" + } } } ``` -With DMs on, users type normally — `MentionRequiredInDm` defaults to `false`, so no @-mention needed. +A channel-ID entry beats `"dm"`, and an unrecognized value denies the message instead of falling back. [Security model](/security/security-model/) maps audiences to tools. For a channel you own, `personal` is what lets the agent request shell commands, still gated by approval. -### Audience overrides +## Behavior in Mattermost -Audience resolves per message: `ChannelAudiences[channelId]` wins first, then `ChannelAudiences["dm"]` for direct messages, then a fallback that lands on `Team` for allow-listed channels and users and `Public` for everything else. Override it explicitly: +### Threads and sessions + +Each thread is its own session, keyed by `{channelId}/{rootPostId}`, and the bot replies in-thread. The thread stays attached to its session until the session goes idle and is freed from memory, after `Session.IdleTimeout` (one hour by default). A later reply in the thread wakes it again. After a daemon restart, thread history is backfilled so conversations continue. + +### Mention behavior + +`MentionOnly: true` makes the bot ignore channel messages that don't @-mention it. netclaw strips the mention before the text reaches the model. Once a thread has an active session, replies in it don't need a mention, unless you turn that rule on for the channel: ```json { "Mattermost": { - "ChannelAudiences": { - "abcdefghijklmnopqrstuvwxyz": "team", - "dm": "personal" + "MentionRequiredInThreadByChannel": { + "abcdefghijklmnopqrstuvwxyz": true } } } ``` -The `"dm"` key is reserved — it matches every direct message rather than a channel ID, and a channel-ID entry takes precedence over it. An unrecognized audience value is rejected outright: the message is denied rather than falling back to a default. +With the rule on, an un-mentioned thread reply is ignored (`ThreadMentionRequired` in the log). The next message that does mention the bot makes netclaw re-read everything posted since its last reply, so it answers with full context. A channel with no entry behaves as `false`. The rule doesn't apply to DMs. -[Security Model](/security/security-model/) has the full breakdown on how audiences map to tools and permissions. +You can also set it in `netclaw config` → Channels. On the channel list, **Space** toggles **Require @mention** for the selected channel. A channel you add there starts with the rule on if its audience is `team` or `public`, and off if it's `personal`. Only `true` entries are saved. -## Behavior in Mattermost +### Typing indicator -### Threads and sessions +While a session works on your message, netclaw sends Mattermost's native typing pulse to the thread every three seconds. It stops when the session goes idle, and it gives up after ten minutes so a stuck session can't leave the thread typing forever. A failed pulse logs a warning and doesn't affect the reply. -Each thread is its own isolated session, keyed by `{channelId}/{rootPostId}`, and the bot replies in-thread. On daemon restart, thread history is backfilled so in-progress conversations pick up where they left off. +### Tool approval -### Mention behavior +When a tool call needs approval, how netclaw asks depends on `CallbackUrl`: -`MentionOnly: true` (the default) means the bot ignores channel messages that don't @-mention it. Thread replies are the exception — once a thread has an active session, the bot responds to everything in it without a mention. Netclaw strips the @-mention before passing text to the LLM. +| `CallbackUrl` | Prompt | How you answer | +|---------------|--------|----------------| +| Unset | A text prompt listing options with letters | Reply with a letter in the thread. No inbound HTTP endpoint is opened. Approval requests from a sub-agent are refused. | +| Set | Buttons | Click. Each button carries a one-time token that expires after 12 hours. | -### Tool approval +The choices are the same on every channel: **Once**, **This chat**, **Always here**, **Always anywhere**, **Deny**, and **This repository** when it applies. A command netclaw can't parse cleanly gets only **Once** and **Deny**. [How tool approvals work](/security/approvals/) explains what each choice grants. Only the user whose request triggered the prompt can answer it. Anyone else gets a warning. -When a tool call needs approval, how netclaw asks depends on `CallbackUrl`: +A prompt from a sub-agent always has buttons only, on every channel, with no lettered fallback. A typed letter doesn't answer it. If two or more prompts are open for you and one is from a sub-agent, a letter answers none and the bot replies `N approval prompts are open. Use the buttons on the prompt you mean.` -- **`CallbackUrl` set** — netclaw posts interactive buttons. The full prompt offers five choices, in order: **Once**, **This chat**, **Always here**, **Always anywhere**, **Deny**. Clicks come back over an inbound HTTP POST to `/api/mattermost/actions`, which the daemon only exposes when a callback URL is configured. Each button carries a one-time, post-bound token that expires after 12 hours. -- **`CallbackUrl` unset** — approvals fall back to text replies: reply with the letter shown next to each option. No inbound HTTP surface is opened. +The lettered prompt, from a real run: -netclaw shows fewer than five when some don't apply — a command it can't cleanly parse (shell control flow, or unbalanced quotes) collapses to just **Once** and **Deny**, and the reply letters track whatever is shown. **Always anywhere** is the broadest grant: it approves the command's verb for every session everywhere and persists to `tool-approvals.json`. Reach for **Always here** or narrower unless you really mean it. +```text +:lock: Tool approval required +Tool: shell_execute +Action: touch /tmp/approval-test +Pattern: touch -The token store is in-memory: buttons posted before a daemon restart return "no longer valid" when clicked afterward. The pending request survives on the session, so ask the agent to re-issue the prompt. +Reply with: +A) Once +B) This chat +C) Always here +D) Always anywhere +E) Deny +``` -:::caution -Reach `/api/mattermost/actions` over TLS. The callback trusts the user ID in the payload as the clicker's identity, so a network attacker who can rewrite headers in plaintext could impersonate another user. -::: +If posting an approval prompt fails, netclaw reports the failure to the model as an unavailable prompt, not as your denial, and the call doesn't run. A prompt that nothing can answer any more is redrawn without buttons and with the reason, such as `This approval request was withdrawn because the task that asked for it was cancelled. Nothing ran.` A click on a prompt that already has an answer gets `That approval prompt already has an answer. This response changed nothing.` + +#### Buttons need a reachable callback + +The Mattermost **server**, not your browser, sends the click to `CallbackUrl`. The URL has to be reachable from the server's network, and it has to reach the daemon's `/api/mattermost/actions` path. The daemon serves that on its one listener at `Daemon.Host:Daemon.Port`. In the default `local` exposure mode that listener is loopback only, and `local` refuses any other bind. A Mattermost server on another host or in another container can't reach it. + +So you pick one of two setups: + +- Leave `CallbackUrl` unset and use text replies. This works for a single-host or Compose stack that doesn't use sub-agents. Any approval request from a sub-agent is refused, the call doesn't run, and the prompt is withdrawn as `could not show it in this channel`. +- Move the daemon to a non-local [exposure mode](/deployment/exposure-modes/) or front the path with a forwarder, then set `CallbackUrl` to the address the Mattermost server uses, for example `https://netclaw.example.com/api/mattermost/actions`. + +Mattermost also blocks integration requests to private and loopback addresses unless the target is in `ServiceSettings.AllowedUntrustedInternalConnections` ([Mattermost docs](https://docs.mattermost.com/administration-guide/configure/environment-configuration-settings.html), **Environment > Developer**). Add the daemon's host there if your callback is a private address. + +Reach the endpoint over TLS or a trusted network. The callback trusts the user ID in the payload as the clicker's identity. The token store lives in memory and holds at most 10,000 unused tokens, oldest evicted first. Buttons posted before a daemon restart answer "no longer valid", so ask the agent to re-issue the prompt. A sub-agent prompt for a run that hadn't parked when the daemon restarted shows the withdrawal text instead. A parked run keeps its prompt, and a click after the restart resumes it. ### Proactive messaging -The LLM can start a conversation with the generic `send_channel_message` tool — pass `channel_key: mattermost` and a `destination` resolved via `lookup_channel_user` or `lookup_channel_destination`. It posts a new thread root and routes replies in that thread back to a live session. ACL rules still apply — DMing a user also requires `AllowDirectMessages: true`. +The model can start a conversation with `send_channel_message`, passing `channel_key: mattermost` and a destination from `lookup_channel_user` or `lookup_channel_destination`. It posts a new thread root and routes replies in that thread to a live session. ACL still applies, and DMing a user needs `AllowDirectMessages: true`. ### Reminders -[Reminders](/configuration/reminders/) reach Mattermost through `channel` delivery with `delivery_transport: "mattermost"`. (In-thread `current_session` delivery isn't available from a Mattermost session yet — `set_reminder` only accepts it for Slack, Discord, the TUI, and SignalR.) Because Mattermost user IDs and channel IDs are both 26-character strings, channel-delivery targets need a prefix: +[Reminders](/configuration/reminders/) reach Mattermost two ways: + +- `current_session` posts back into the thread that set the reminder +- `channel` delivery with `delivery_transport: "mattermost"` posts to a channel or a user's DM + +User IDs and channel IDs have the same shape, so channel delivery needs a prefix: + +| Target | Delivers to | +|--------|-------------| +| `@` | That user's DM | +| `channel:` | That channel | -- `@` — delivers to that user's DM -- `channel:` — delivers to that channel +A bare ID is rejected with a disambiguation error. -A bare ID with no prefix is rejected with a disambiguation error. Unlike Discord, DM delivery works, because a Mattermost DM is itself an addressable channel. +A `current_session` reminder ends when the session admits the fire, and netclaw doesn't retry a failed post. If the reply post fails for good, the session logs an Error and raises a `reminder.delivery.failed` [alert](/observability/operational-alerts/). ## Verify it works -Restart the daemon, then @-mention the bot in an allowed channel: +Check the connector, then mention the bot in an allowed channel: ```bash -netclaw daemon stop && netclaw daemon start +netclaw status ``` -If it replies in a thread, you're set. +```text +connectors: +- mattermost: healthy (enabled) +``` + +If the bot replies in a thread, you're set. + +![Mattermost Assistant channel where alice mentions @netclaw and the netclaw bot answers with a greeting in the thread panel](/screenshots/output/mattermost-conversation.png) -![Netclaw replying to a testuser @-mention in a Mattermost thread](/screenshots/output/mattermost-conversation.png) +The reply lands in a thread under alice's mention, which is also where follow-ups go. -`netclaw status` reports Mattermost health (healthy, degraded, disconnected, or disabled). When something's off, check its status first, then the daemon logs for detail. A bad token, an unreachable server, or any connection failure is contained there — the Mattermost channel degrades on its own and the daemon plus every other channel keeps running. See [Troubleshooting](#troubleshooting) below for the common messages. +`netclaw status` reports `healthy`, `degraded`, `disconnected` or `disabled`. A channel problem stays in the channel. The daemon and your other channels keep running. ## Troubleshooting -Common problems and fixes are in [Channel Troubleshooting](/channels/troubleshooting/). The hits: +[Channel troubleshooting](/channels/troubleshooting/) is organized by symptom. The usual ones: -- **Connected but silent** — `AllowedChannelIds` is empty and no `DefaultChannelId` is set, so all traffic gets denied; or the bot was never added to the channel. -- **`Mattermost is enabled but Mattermost:ServerUrl is not configured`** — add `Mattermost.ServerUrl` to `netclaw.json`. -- **`Mattermost rejected the bot token (HTTP 401)`** — re-issue the bot token and update `secrets.json`. -- **Can't create a bot or token** — Enable Bot Account Creation and Enable Personal Access Tokens are both off by default in the System Console. +| Symptom | Likely cause | +|---------|--------------| +| `disconnected`, `no bot token is configured` | Store the token with `netclaw secrets set`, then restart. | +| `disconnected`, `rejected the bot token (HTTP 401)` or `rejected the request as unauthorized` | The token is wrong or was revoked. Issue a new one and restart. | +| `disconnected`, `ServerUrl is not configured` | Set `Mattermost.ServerUrl`. | +| Connected but silent | The channel or user isn't in the allow-list, or the bot isn't in the channel. | +| Reconnect loop with `outside a clean startup cycle` | A startup race fixed in 0.26.0-beta.5. Upgrade. | +| Can't create a bot or token | One of the two System Console toggles is still off. | + +A bad token or an unreachable server is a fatal connect error: the channel stays offline until you fix the config and restart the daemon. Network blips retry on their own. :::note -Running Mattermost in a container on Apple Silicon? The [`mattermost/mattermost-preview`](https://hub.docker.com/r/mattermost/mattermost-preview) image is amd64-only — start it with `--platform linux/amd64`. It's a testing image with a bundled database; for production, deploy the [`mattermost/mattermost-team-edition`](https://hub.docker.com/r/mattermost/mattermost-team-edition) image and a real database per [Mattermost's Docker install guide](https://docs.mattermost.com/install/install-docker.html). +Testing on Apple Silicon? The [`mattermost/mattermost-preview`](https://hub.docker.com/r/mattermost/mattermost-preview) image is amd64-only and bundles its own database, so use it for trials only. For anything real, run the Team Edition image with Postgres, as the [Compose guide](/guides/mattermost-docker-compose/) does. ::: -## Next steps - -- [Configure audiences and approval gates](/security/security-model/) to control what tools are available in each channel -- [Set up systemd](/deployment/systemd/) so the daemon stays running after reboots -- [Add Slack](/channels/slack/) or [Discord](/channels/discord/) if your team uses more than one - ## Related pages -- [`netclaw config`](/cli/config/) — Channels -- [`netclaw secrets`](/cli/secrets/) — token management -- [`netclaw status`](/cli/status/) — primary diagnostic for Mattermost -- [Security Model](/security/security-model/) — audiences and approval gates -- [Channel Troubleshooting](/channels/troubleshooting/) — error codes and debug logging -- [Slack](/channels/slack/) — sibling channel integration -- [Discord](/channels/discord/) — sibling channel integration - -## External resources - -- [Mattermost](https://mattermost.com/) — the project, if you don't already run a server -- [Mattermost on GitHub](https://github.com/mattermost/mattermost) — source code, issues, and release notes -- [Mattermost: Bot accounts](https://developers.mattermost.com/integrate/reference/bot-accounts/) — creating and managing bots -- [Mattermost: Personal access tokens](https://developers.mattermost.com/integrate/reference/personal-access-token/) — enabling and issuing tokens -- [Mattermost: Enable bot account creation](https://docs.mattermost.com/configure/integrations-configuration-settings.html) — the System Console settings -- [Mattermost: Install with Docker](https://docs.mattermost.com/install/install-docker.html) — container deployment, including the [official images on Docker Hub](https://hub.docker.com/u/mattermost) -- [Mattermost API reference](https://api.mattermost.com/) — the REST endpoints netclaw uses for replies +- [Run netclaw with Mattermost on Docker Compose](/guides/mattermost-docker-compose/) - a tested, hardened stack +- [Channel troubleshooting](/channels/troubleshooting/) - fixes by symptom +- [How tool approvals work](/security/approvals/) - the approval choices +- [Security model](/security/security-model/) - audiences and approval gates +- [Exposure modes](/deployment/exposure-modes/) - reaching the daemon from outside its host +- [Slack](/channels/slack/) and [Discord](/channels/discord/) - the other chat channels + +## Resources + +- [Mattermost](https://mattermost.com/) - the project, if you don't already run a server +- [Mattermost on GitHub](https://github.com/mattermost/mattermost) - source code, issues and release notes +- [Mattermost: bot accounts](https://developers.mattermost.com/integrate/reference/bot-accounts/) - creating and managing bots +- [Mattermost: personal access tokens](https://developers.mattermost.com/integrate/reference/personal-access-token/) - enabling and issuing tokens +- [Mattermost: integrations settings](https://docs.mattermost.com/administration-guide/configure/integrations-configuration-settings.html) - the System Console toggles +- [Mattermost: install with Docker](https://docs.mattermost.com/deploy/server/deploy-containers.html) - the official container deployment +- [Mattermost API reference](https://api.mattermost.com/) - the REST endpoints netclaw calls diff --git a/src/content/docs/channels/slack.md b/src/content/docs/channels/slack.md index 8e449ca..87fd5ab 100644 --- a/src/content/docs/channels/slack.md +++ b/src/content/docs/channels/slack.md @@ -1,24 +1,26 @@ --- title: "Slack" -description: "Connect Netclaw to your Slack workspace." +description: "Connect netclaw to your Slack workspace over Socket Mode with a bot token, an app token and an allow-list." --- -Netclaw talks to Slack over [Socket Mode](https://api.slack.com/apis/socket-mode) — outbound WebSocket connections only. No public URLs, no ingress rules, no reverse proxies. You create a Slack app, give netclaw two tokens, and it shows up in your workspace as a bot. +netclaw talks to Slack over [Socket Mode](https://docs.slack.dev/apis/events-api/using-socket-mode), which means outbound WebSocket connections only. You need no public URL, no ingress rule and no reverse proxy. You create a Slack app, give netclaw two tokens, and it shows up in your workspace as a bot. ## Prerequisites -- Netclaw installed and initialized ([`netclaw init`](/cli/init/)) -- A Slack workspace where you can install apps (some orgs restrict this to workspace admins) +- netclaw installed and initialized ([`netclaw init`](/cli/init/)) +- A Slack workspace where you can install apps (some organizations limit that to admins) -Connecting Slack is a two-part process: create a Slack app to get your tokens, then enter them in [`netclaw config`](/cli/config/) → Channels. The steps below walk through both. +Connecting Slack has two parts. Create a Slack app to get the tokens, then enter them in [`netclaw config`](/cli/config/) → Channels. ## Create a Slack app -The fastest path: create from a manifest. Go to [api.slack.com/apps](https://api.slack.com/apps) and click **Create New App**. +The fastest path is a manifest. Go to [api.slack.com/apps](https://api.slack.com/apps) and click **Create New App**. ![Slack Your Apps page](/screenshots/output/slack-your-apps.png) -Select **From a manifest**, pick your workspace, and paste this: +The page has a filter box for your existing apps and the **Create New App** button at the top right. + +Choose **From a manifest**, pick your workspace, and paste this: ```json { @@ -55,7 +57,8 @@ Select **From a manifest**, pick your workspace, and paste this: "im:write", "mpim:history", "mpim:read", - "users:read" + "users:read", + "users:read.email" ] } }, @@ -79,57 +82,62 @@ Select **From a manifest**, pick your workspace, and paste this: } ``` -Change the `name` and `display_name` to whatever you want your bot to be called. +Change `name` and `display_name` to what you want the bot called. -The manifest enables the App Home **Messages tab** (`messages_tab_enabled`), which is what lets workspace members DM the bot. If you create the app any other way and skip this, Slack shows *"Sending messages to this app has been turned off"* when someone tries to DM it — you can fix that later under **Features → App Home → Messages Tab**, but the manifest sets it correctly from the start. +The manifest turns on the App Home **Messages tab** (`messages_tab_enabled`), which is what lets people DM the bot. If you create the app another way and skip it, Slack tells anyone who tries to DM the bot *"Sending messages to this app has been turned off"*. You can fix that under **Features > App Home > Messages Tab**. -After creating the app, you need two tokens: +After the app exists, you need two tokens. -1. **App-Level Token** — Settings > Basic Information > App-Level Tokens. Click **Generate Token and Scopes**, name it anything, and add the `connections:write` scope. +1. **App-level token.** Open **Settings > Basic Information**, scroll to **App-Level Tokens**, click **Generate Token and Scopes**, name it anything, and add the `connections:write` scope. -![Generate an app-level token dialog](/screenshots/output/slack-generate-app-token.png) + ![Generate an app-level token dialog](/screenshots/output/slack-generate-app-token.png) -Click **Generate** and copy the `xapp-...` token. + The dialog shows the `connections:write` scope that Socket Mode needs. Click **Generate** and copy the `xapp-...` token. -2. **Bot Token** — Go to **Install App** in the sidebar and click **Install to {Your Workspace}**. +2. **Bot token.** Open **Install App** in the sidebar and click **Install to {Your Workspace}**. -![Slack Install App page](/screenshots/output/slack-install-app.png) + ![Slack Install App page](/screenshots/output/slack-install-app.png) -After approving, copy the `xoxb-...` Bot User OAuth Token from the OAuth & Permissions page. + The install page is also where you reinstall after changing scopes. After you approve, copy the `xoxb-...` Bot User OAuth Token from **OAuth & Permissions**. -Then invite the bot to each channel where it should respond: `/invite @YourBotName` +Then invite the bot to each channel where it should answer: `/invite @YourBotName`. ### What the scopes do | Scope | Why | |-------|-----| | `app_mentions:read` | Receive @-mention events | -| `channels:history` | Read message history in public channels | -| `channels:read` | Resolve channel names to IDs, list public channels | -| `chat:write` | Post messages and replies in threads, plus native loading status | -| `chat:write.customize` | Post with custom display name/avatar | +| `channels:history` | Read history in public channels | +| `channels:read` | Resolve channel names to IDs, list public channels and read channel info | +| `chat:write` | Post replies in threads, and set the native loading status | +| `chat:write.customize` | Post under a custom name or icon. Netclaw doesn't do this today. | | `files:read` | Download files shared in conversations | -| `files:write` | Upload files (agent output, attachments) | -| `groups:history` | Read message history in private channels | +| `files:write` | Upload files such as agent output | +| `groups:history` | Read history in private channels | | `groups:read` | List private channels the bot is in | -| `im:history` | Read DM history for thread context | -| `im:read` | List DM conversations | -| `im:write` | Open DM conversations for proactive messaging | -| `mpim:history` | Read group DM history | -| `mpim:read` | List group DM conversations | -| `users:read` | Look up users by name or email for `lookup_slack_user` | +| `im:history` | Receive DM message events and read DM thread replies for context | +| `im:read` | Read DM details, such as the name of a DM listed in `AllowedChannelIds` | +| `im:write` | Open DMs for proactive messages | +| `mpim:history` | Slack requires it for the `message.mpim` event the manifest subscribes to, and for reading group DM thread replies | +| `mpim:read` | Read group DM details, such as the name of a group DM listed in `AllowedChannelIds` | +| `users:read` | Look up users by name for `lookup_slack_user` | +| `users:read.email` | Look up users by email. Slack returns the email field only with this scope. | + +If you installed from an older copy of this manifest, add `users:read.email` under **OAuth & Permissions** and reinstall the app, or email lookups won't match. ## Configure netclaw -Run `netclaw config` → Channels, enable Slack, and paste your tokens. Netclaw resolves channel names to IDs and saves everything — no JSON editing required. +Run `netclaw config` → Channels, enable Slack, and paste your tokens. netclaw resolves channel names to IDs and saves, so you don't edit JSON. ![The Channels area in netclaw config](/screenshots/output/config-channels-menu.png) -The Channels area — enable an adapter and manage its allow-list; here Slack is connected with 2 channels and 1 user. +The Channels area lists Slack, Discord and Mattermost with a checkbox each. Once an adapter is set up, its row also shows how many channels and users it allows. Nothing is set up in this capture. + +When you save from this screen, netclaw writes the first channel as `DefaultChannelId` and removes `DefaultChannelName`. If you set `DefaultChannelName` by hand, the TUI will replace it. ### Manual configuration -For scripted or headless installs, store tokens with [`netclaw secrets`](/cli/secrets/): +For scripted or headless installs, store the tokens with [`netclaw secrets`](/cli/secrets/): ```bash netclaw secrets set Slack.BotToken xoxb-your-bot-token @@ -147,27 +155,39 @@ Then enable Slack and point it at a channel in `~/.netclaw/config/netclaw.json`: } ``` -Environment variables work too: +Environment variables work too, with list entries indexed from zero: ```bash export NETCLAW_Slack__BotToken="xoxb-..." export NETCLAW_Slack__AppToken="xapp-..." +export NETCLAW_Slack__AllowedChannelIds__0="C0123456789" ``` +An edit to `netclaw.json` restarts the daemon by itself. A vault or environment change needs a restart you run: + +| Where it runs | Restart | +|---------------|---------| +| Native | `netclaw daemon stop`, then `netclaw daemon start` | +| systemd user unit | `systemctl --user restart netclaw` | +| Docker | `docker restart ` | + +:::caution +Keep each list in one place. Environment entries and `netclaw.json` entries merge by position, so a blank `NETCLAW_Slack__AllowedChannelIds__0` overwrites the first channel in the file. `netclaw config` → Channels writes the lists into `netclaw.json`. +::: + ### How channel IDs are stored -Enter channel names or IDs (comma-separated) in `netclaw config` → Channels. Netclaw resolves each entry against the Slack API to its canonical channel ID **before saving**. The stored `AllowedChannelIds` field holds IDs, not display names; display names are shown dynamically in the config UI. +Enter channel names or IDs, comma-separated, in `netclaw config` → Channels. netclaw resolves each against the Slack API and saves the canonical ID, never the display name. -Resolution rules: -- **Confirmed ID** — bot finds the channel by ID: kept as-is. -- **Display name** — resolves to its channel ID and saved as the ID. -- **ID-shaped but not enumerable** — kept; a channel ID is the stable ACL key and is never dropped, even if the bot can't list it right now. -- **Unresolvable display name** — dropped and flagged with a Warning in the config UI. Re-open `netclaw config` → Channels after inviting the bot to that channel. +- **Confirmed ID:** kept as is. +- **Display name:** resolved and saved as the ID. +- **ID-shaped but not listable:** kept. A channel ID is the stable ACL key, so it's never dropped, even if the bot can't list the channel right now. +- **Unresolvable display name:** dropped and flagged with a warning. Invite the bot, then reopen the screen. -`DefaultChannelName` is Slack-only and is resolved to a channel ID live at daemon startup, so it is unaffected by the ACL matching rules above. +`DefaultChannelName` is Slack-only and resolves to an ID at daemon startup. :::caution -**Breaking change (pre-0.24.0 → 0.24.0+):** If you previously placed channel display names in `AllowedChannelIds` by hand in `netclaw.json`, those names no longer match incoming channel IDs and messages will be silently dropped. Re-enter the channels via `netclaw config` → Channels so they resolve to canonical IDs. (ID-shaped entries and `DefaultChannelName` are unaffected.) +Since 0.24.0, channel display names you put in `AllowedChannelIds` by hand no longer match, and messages from those channels are silently dropped. Re-enter them through `netclaw config` → Channels. ID-shaped entries and `DefaultChannelName` are unaffected. ::: ### All config fields @@ -175,19 +195,20 @@ Resolution rules: | Field | Type | Default | Description | |-------|------|---------|-------------| | `Enabled` | bool | `false` | Turn on Slack | -| `SocketMode` | bool | `true` | Must be `true`. Only Socket Mode is supported. | -| `BotToken` | string | — | Bot User OAuth Token (`xoxb-...`). Store with `netclaw secrets set`. | -| `AppToken` | string | — | App-Level Token (`xapp-...`). Required for Socket Mode. Store with `netclaw secrets set`. | -| `DefaultChannelName` | string | — | Channel name, resolved to an ID at startup | -| `DefaultChannelId` | string | — | Channel ID directly (use instead of name if you prefer) | -| `MentionOnly` | bool | `true` | Only respond when @-mentioned | +| `SocketMode` | bool | `true` | Must stay `true`. Slack supports Socket Mode only. | +| `BotToken` | string | - | Bot User OAuth Token (`xoxb-...`). Store with `netclaw secrets set`. | +| `AppToken` | string | - | App-level token (`xapp-...`). Store with `netclaw secrets set`. | +| `DefaultChannelName` | string | - | Channel name, resolved to an ID at startup | +| `DefaultChannelId` | string | - | Channel ID, if you'd rather not use a name | +| `MentionOnly` | bool | `true` | In channels, answer only when @-mentioned | | `AllowDirectMessages` | bool | `false` | Accept DMs | -| `MentionRequiredInDm` | bool | `false` | Require @-mention even in DMs | -| `AllowedChannelIds` | string[] | `[]` | Channel allow-list. Empty + no default = all channels denied | -| `AllowedUserIds` | string[] | `[]` | User allow-list. Empty = everyone in allowed channels is accepted | -| `ChannelAudiences` | object | `{}` | Per-channel [audience](/security/security-model/) overrides. Keys are channel IDs or `"dm"`. Values: `"personal"`, `"team"`, `"public"`. | +| `MentionRequiredInDm` | bool | `false` | Require an @-mention even in DMs | +| `MentionRequiredInThreadByChannel` | object | `{}` | Channel ID to bool. `true` makes a thread reply in that channel need an @-mention. See [Mention behavior](#mention-behavior). | +| `AllowedChannelIds` | string[] | `[]` | Channel allow-list. Empty with no default denies every channel message. | +| `AllowedUserIds` | string[] | `[]` | User allow-list. Empty accepts everyone in allowed channels. | +| `ChannelAudiences` | object | `{}` | Per-channel [audience](/security/security-model/) overrides. Keys are channel IDs or `"dm"`. Values are `"personal"`, `"team"` or `"public"`. | -Tokens live in `~/.netclaw/config/secrets.json` (encrypted at rest). +Tokens live in `~/.netclaw/config/secrets.json`, encrypted at rest. ## Access control @@ -195,55 +216,36 @@ Slack ACL is **default-deny**. Three settings decide who talks to the bot and wh ### Channels -The bot only responds in channels that pass the allow-list: - -- `DefaultChannelName` or `DefaultChannelId` allows one channel -- `AllowedChannelIds` allows multiple -- **If all three are empty, every channel message is denied.** This is the number one setup mistake. +The bot answers only in channels that pass the allow-list. `DefaultChannelName` or `DefaultChannelId` allows one, `AllowedChannelIds` allows more, and if all of them are empty every channel message is denied. That's the number one setup mistake. ```json { "Slack": { - "DefaultChannelName": "openclaw", + "DefaultChannelName": "general", "AllowedChannelIds": ["C0123456789", "C9876543210"] } } ``` -Finding channel IDs: right-click a channel name in Slack, "View channel details," scroll to the bottom. [Slack's help article](https://slack.com/help/articles/221769328) has screenshots. +To find a channel ID, right-click the channel name, choose **View channel details**, and scroll to the bottom. [Slack's help article](https://slack.com/help/articles/221769328) has screenshots. ### Users -`AllowedUserIds` restricts who gets responses: - -- Empty (default) — everyone in allowed channels is accepted -- Non-empty — only listed user IDs get responses, everyone else is silently dropped +`AllowedUserIds` restricts who gets answers. Empty means everyone in an allowed channel. Non-empty means only the listed users, and everyone else is dropped silently. To copy a member ID, open the profile, then the three-dot menu, then **Copy member ID** ([Slack's help article](https://slack.com/help/articles/360003534892)). -Finding user IDs: click a user's profile in Slack, open the three-dot menu, "Copy member ID." [Slack's help article](https://slack.com/help/articles/360003534892) has screenshots. - -Users in `AllowedUserIds` are treated as `TrustedInternal` by the [security model](/security/security-model/). Everyone else is `UntrustedExternal`. +Listed users are `TrustedInternal` in the [security model](/security/security-model/), and everyone else is `UntrustedExternal`. ### Direct messages -DMs are off by default: - -```json -{ - "Slack": { - "AllowDirectMessages": true - } -} -``` - -With DMs on, users can just type normally — `MentionRequiredInDm` defaults to `false`, so no @-mention needed. +DMs are off by default. Turn them on with `"AllowDirectMessages": true`. Users then write without a mention, because `MentionRequiredInDm` defaults to `false`. :::caution -With `AllowDirectMessages: true` and `AllowedUserIds` empty, any workspace member can DM the bot. Lock down `AllowedUserIds` if that's not what you want. +With DMs on and `AllowedUserIds` empty, any workspace member can DM the bot. Set `AllowedUserIds` unless that's what you want. ::: ### Audience overrides -Audience is resolved per-message: `Team` for allow-listed channels and allow-listed users; `Public` for everything else — including unvetted DMs (DMs accepted only because `AllowedUserIds` is empty). To treat all DMs as `Team`, set an explicit `ChannelAudiences "dm"` override. Override with `ChannelAudiences`: +Audience resolves per message: `ChannelAudiences[channelId]` first, then `ChannelAudiences["dm"]`, then a fallback. The fallback is `Team` for a channel listed in `AllowedChannelIds` or a user listed in `AllowedUserIds`, and `Public` for everything else. A DM accepted only because `AllowedUserIds` is empty is `Public`. So is a channel allowed only through `DefaultChannelId` or `DefaultChannelName`, which includes the `DefaultChannelName` example above, unless you also list it in `AllowedChannelIds` or set `ChannelAudiences`. ```json { @@ -256,102 +258,121 @@ Audience is resolved per-message: `Team` for allow-listed channels and allow-lis } ``` -The `"dm"` key is reserved — it matches every direct message rather than a channel ID. A channel-ID entry takes precedence over it. An unrecognized audience value is rejected outright: the message is denied rather than falling back to a default. - -[Security Model](/security/security-model/) has the full breakdown on how audiences map to tools and permissions. +A channel-ID entry beats `"dm"`, and an unrecognized value denies the message instead of falling back. [Security model](/security/security-model/) maps audiences to tools. ## Behavior in Slack ### Threads and sessions -Each Slack thread is its own isolated session, and the bot always replies in-thread. Idle threads are freed from memory after 1 hour; the conversation context clears after 2 hours of inactivity. - -On daemon restart, thread history is backfilled so in-progress conversations pick up where they left off. +Each Slack thread is its own session, and the bot always replies in-thread. An idle thread is freed from memory when its session goes idle, after `Session.IdleTimeout` (one hour by default), and a later reply wakes it again. After a daemon restart, thread history is backfilled so conversations continue. ### Mention behavior -`MentionOnly: true` (the default) means the bot ignores messages that don't @-mention it. Two exceptions: +`MentionOnly: true` makes the bot ignore channel messages that don't @-mention it. netclaw strips the mention before the text reaches the model. These rules sit on top: + +- **Thread replies:** once a thread has an active session, replies in it don't need a mention, unless `MentionRequiredInThreadByChannel` is `true` for that channel. +- **File shares:** a message with an attachment follows the same mention rules as any other message. An un-mentioned upload in a channel is ignored. + +`MentionRequiredInThreadByChannel` maps channel IDs to `true` or `false`: + +```json +{ + "Slack": { + "MentionRequiredInThreadByChannel": { + "C0123456789": true + } + } +} +``` -- **Thread replies** — if a thread already has an active session, the bot responds to everything in that thread without needing a mention -- **File shares** — attached files bypass the mention check entirely, with or without an active thread +With the rule on, an un-mentioned thread reply is ignored (`ThreadMentionRequired` in the log). The next message that mentions the bot makes netclaw re-read everything posted since its last reply, so it answers with full context. A channel with no entry behaves as `false`. The rule doesn't apply to DMs. -Netclaw strips the @-mention before passing text to the LLM. +In `netclaw config` → Channels, **Space** toggles **Require @mention** on the selected channel. A channel you add there starts with the rule on if its audience is `team` or `public`, and off if it's `personal`. Only `true` entries are saved. ### Message formatting -Netclaw converts LLM markdown to Slack [Block Kit](https://api.slack.com/block-kit): headers, code blocks, blockquotes, lists, bold, italic, strikethrough, inline code, links. Everything renders natively in Slack. +Netclaw sends each reply as a single Slack Markdown block, and Slack renders it, tables included. Replies over 12,000 characters go out as plain text instead. ### Loading status -While Netclaw is working on a reply, Slack shows the bot's native thread status as ` is thinking...`. This uses Slack's `assistant.threads.setStatus` API with the existing `chat:write` scope, so the manifest above does not need `assistant:write` or the Slack assistant split view. +While netclaw works on a reply, Slack shows the bot's native thread status as ` is thinking...`. That uses `assistant.threads.setStatus` with the existing `chat:write` scope, so the manifest doesn't need `assistant:write`. ### Tool approval -When a tool call needs approval, netclaw posts a Block Kit prompt right in the thread: +When a tool call needs approval, netclaw posts a prompt in the thread with one button per choice. For a prompt from the main agent you can also reply with the letter next to a choice. A prompt from a sub-agent has buttons only, and a typed letter doesn't answer it. -![Tool approval prompt in a Slack thread, with approve and deny buttons](/assets/approval-prompt.png) +![Tool approval prompt in a Mattermost thread: shell_execute, a push to origin main, and six lettered choices from A (Once) to F (Deny)](/assets/approval-prompt.png) -Shows the tool name, the exact command, and the approval buttons. Only the user who triggered the request can approve. System-initiated tool calls (`VerifiedAutomation`) can be approved by anyone in the thread. +The picture shows the same prompt in Mattermost's text-reply form. Slack renders the same six choices as buttons, with the lettered list as a fallback line underneath. A sub-agent prompt has no lettered line. -The full prompt offers five choices: **Once**, **This chat**, **Always here**, **Always anywhere**, and **Deny** — "Always anywhere" is the broadest grant, so reach for it sparingly. netclaw shows fewer when some don't apply — a command it can't cleanly parse (shell control flow, or unbalanced quotes) drops to just **Once** and **Deny**. You can reply with the letter shown next to each option instead of clicking. +The choices are **Once**, **This chat**, **Always here**, **Always anywhere**, **Deny**, and **This repository** when it applies. A command netclaw can't parse cleanly gets only **Once** and **Deny**. [How tool approvals work](/security/approvals/) explains what each one grants. Only the user whose request triggered the prompt can answer it. System-initiated calls (`VerifiedAutomation`) can be answered by anyone in the thread. A prompt that nothing can answer any more is redrawn as a single line with the reason and no buttons, such as `This approval request was withdrawn because the task that asked for it ended. Nothing ran.` ### Proactive messaging -The LLM can initiate conversations through two built-in tools: +The model can start conversations with two tools: | Tool | What it does | -|------|-------------| -| `send_channel_message` | Posts a message to a Slack channel or DM using a resolved destination. Pass `channel_key="slack"` and a `destination` object (with `kind`, `id`) from `lookup_slack_user` or `lookup_channel_destination`. Respects ACL. | -| `lookup_slack_user` | Searches users by name, display name, or email. Returns up to 10 matches. Filtered to `AllowedUserIds` if set. Cached 5 minutes. | +|------|--------------| +| `send_channel_message` | Posts to a Slack channel or DM. Pass `channel_key="slack"` and a `destination` (`kind`, `id`) from `lookup_slack_user` or `lookup_channel_destination`. Respects ACL. | +| `lookup_slack_user` | Searches users by name, display name or email, up to 10 matches, filtered to `AllowedUserIds` when set, cached for five minutes. Email matching needs `users:read.email`. | ### Ignored messages -The bot drops: empty messages (no text, no files), hidden messages, other bots' messages, its own messages, messages with unsupported subtypes (edits, joins, bot subtype messages), DMs when `AllowDirectMessages` is off, and un-mentioned channel messages when `MentionOnly` is on and there's no active thread. +The bot drops empty messages, hidden messages, its own and other bots' messages, subtypes it doesn't handle (edits, joins), DMs when `AllowDirectMessages` is off, and un-mentioned channel messages when `MentionOnly` is on and there's no active thread. -## Verify it works +### Connection supervision -Restart the daemon and check status: +A supervisor checks the Socket Mode connection every 5 seconds and reconnects with exponential backoff up to 5 minutes. You don't restart the daemon for a dropped connection. A bad token or `SocketMode: false` is a fatal connect error: Slack stays offline until you fix it and restart. + +## Verify it works ```bash -netclaw daemon stop && netclaw daemon start netclaw status ``` -Slack should show `connected`. If it doesn't, run `netclaw doctor` — it checks token validity and ACL config. +Look for `slack: healthy (enabled)` under `connectors`. Trimmed output from a daemon that has been on Slack for a few minutes: + +```text +daemon: PID 25721, uptime 11m 33s, endpoint http://127.0.0.1:5199 +... +slack counters: recv=27 routed=27 dropped=26 replied=9 rejected=0 reply_failed=0 +... +connectors: +- discord: disabled (disabled) + Discord connector is disabled in configuration. +... +- slack: healthy (enabled) +``` - +`recv` goes up as Slack delivers events, so a number above zero means the bot is hearing the workspace. `dropped` counts events the bot ignores on purpose, such as a message the ACL denies or a thread reply it isn't part of. In 0.28.2 it also counts the bot's own replies coming back from Slack, so expect it to climb with every answer. `rejected` and `reply_failed` count replies that Slack refused or that failed to post, and they should stay at `0`. If Slack isn't `healthy`, run [`netclaw doctor`](/cli/doctor/), which checks the bot token and the ACL config. -Then @-mention the bot in an allowed channel. If it responds, you're set. +Then @-mention the bot in an allowed channel. If it answers in a thread, you're set. ## Troubleshooting -Common problems and fixes are in [Channel Troubleshooting](/channels/troubleshooting/). The hits: +[Channel troubleshooting](/channels/troubleshooting/) is organized by symptom. The usual ones: -- **Connected but silent** — `AllowedChannelIds` is empty and no default channel is set, so all traffic gets denied -- **Works in some channels, not others** — channel missing from `AllowedChannelIds`, or the bot hasn't been invited -- **Socket Mode keeps disconnecting** — the `xapp-...` App-Level Token may have expired or been revoked +| Symptom | Likely cause | +|---------|--------------| +| Connected but silent | `AllowedChannelIds` is empty and no default channel is set. | +| Works in some channels, not others | The channel isn't in `AllowedChannelIds`, or the bot wasn't invited. | +| Stopped answering in a thread | `MentionRequiredInThreadByChannel` is on for the channel. Mention the bot. | +| Doesn't match users by email | The app lacks `users:read.email`. | +| `Sending messages to this app has been turned off` | The Messages tab is off. | +| Socket Mode won't connect | The `xapp-...` token is wrong or revoked. | -## Next steps +## Related pages -- [Configure audiences and approval gates](/security/security-model/) to control what tools are available in each channel -- [Set up systemd](/deployment/systemd/) so the daemon stays running after reboots -- [Add Discord](/channels/discord/) or [Mattermost](/channels/mattermost/) if your team uses more than one -- [Run `netclaw doctor`](/cli/doctor/) to verify token health and ACL config +- [Mattermost](/channels/mattermost/) and [Discord](/channels/discord/) - the other chat channels +- [Channel troubleshooting](/channels/troubleshooting/) - fixes by symptom +- [How tool approvals work](/security/approvals/) - the approval choices +- [Security model](/security/security-model/) - audiences and approval gates +- [`netclaw doctor`](/cli/doctor/) - Slack auth and ACL checks -## Related pages +## Resources -- [`netclaw config`](/cli/config/) — Channels -- [`netclaw secrets`](/cli/secrets/) — token management -- [`netclaw doctor`](/cli/doctor/) — Slack auth and ACL diagnostics -- [Security Model](/security/security-model/) — audiences and approval gates -- [Channel Troubleshooting](/channels/troubleshooting/) — error codes and debug logging -- [Discord](/channels/discord/) — sibling channel integration -- [Mattermost](/channels/mattermost/) — sibling channel integration - -## External resources - -- [Slack API: Socket Mode](https://api.slack.com/apis/socket-mode) — how Socket Mode connections work -- [Slack API: Bot Token Scopes](https://api.slack.com/scopes) — scope reference -- [Slack: Block Kit](https://api.slack.com/block-kit) — message formatting -- [Slack: Finding IDs](https://slack.com/help/articles/221769328) — channel and user IDs for ACL config -- [Slack: Finding User IDs](https://slack.com/help/articles/360003534892) — step-by-step for copying member IDs +- [Slack: Socket Mode](https://docs.slack.dev/apis/events-api/using-socket-mode) - how Socket Mode connections work +- [Slack: app manifests](https://docs.slack.dev/app-manifests/configuring-apps-with-app-manifests) - the manifest format +- [Slack: scopes](https://docs.slack.dev/reference/scopes) - scope reference +- [Slack: finding channel IDs](https://slack.com/help/articles/221769328) - for the allow-list +- [Slack: finding member IDs](https://slack.com/help/articles/360003534892) - for `AllowedUserIds` diff --git a/src/content/docs/channels/troubleshooting.md b/src/content/docs/channels/troubleshooting.md index 5632d83..7c4dbc3 100644 --- a/src/content/docs/channels/troubleshooting.md +++ b/src/content/docs/channels/troubleshooting.md @@ -1,271 +1,291 @@ --- -title: "Channel Troubleshooting" -description: "Diagnose and fix common Slack, Discord, and Mattermost channel issues." +title: "Channel troubleshooting" +description: "Fix a Slack, Discord or Mattermost bot that won't connect, won't answer, or stopped answering, organized by symptom." --- -Netclaw not responding in Slack, Discord, or Mattermost? Two commands will tell you what's wrong most of the time: +When the bot is silent, two commands and one log tell you most of what you need: ```bash -netclaw status # live connector health (requires running daemon) +netclaw status # live connector health and message counters (needs a running daemon) netclaw doctor # offline config and credential checks ``` -Daemon logs live at `~/.netclaw/logs/`, or `journalctl -u netclaw` if you're running it as a systemd service. +`netclaw status` reports each connector as `healthy`, `degraded`, `disconnected` or `disabled`, with counters such as `recv=2 routed=2 dropped=0 replied=2`. Doctor checks Slack only (auth and ACL). For Discord and Mattermost, use status and the daemon log. -## Symptom Triage +## Where the logs are + +The daemon writes `~/.netclaw/logs/daemon-YYYY-MM-DD.log`. After 10 MiB it continues in `daemon-YYYY-MM-DD.1.log`, `.2.log` and so on, and the active file is the highest-numbered segment of the newest date. Crashes go to `crash-*.log` in the same directory. + +| Where it runs | Read the log | +|---------------|--------------| +| Native | `tail -f ~/.netclaw/logs/daemon-*.log` | +| systemd user unit | `journalctl --user -u netclaw -f`, or the file above | +| Docker | `docker exec sh -c 'tail -f ~/.netclaw/logs/daemon-*.log'`. `docker logs` shows only the entrypoint and a few startup warnings. | + +Restart, when a section below tells you to: + +| Where it runs | Restart | +|---------------|---------| +| Native | `netclaw daemon stop`, then `netclaw daemon start` | +| systemd user unit | `systemctl --user restart netclaw` | +| Docker | `docker restart ` (`netclaw daemon start` does nothing under the image's entrypoint) | + +An edit to `netclaw.json` restarts the daemon on its own. Tokens in the vault and environment variables don't, so restart after changing those. + +## Find your symptom | What you're seeing | Jump to | -|---|---| -| Bot shows "disconnected" in `netclaw status` | [Authentication Errors](#authentication-errors) | -| Bot connected but never replies | [ACL and Permissions](#acl-and-permissions) | -| Bot replies to some messages but not others | [Bot Not Responding to Messages (MentionOnly)](#bot-not-responding-to-messages-mentiononly) | -| Bot was working, then stopped | [Connectivity](#connectivity) | -| Bot processes messages but replies never appear | [Message Delivery](#message-delivery) | +|--------------------|---------| +| `disconnected` or `degraded` in `netclaw status` | [The bot won't connect](#the-bot-wont-connect) | +| It connects, then drops again and again | [The connection keeps dropping](#the-connection-keeps-dropping) | +| Connected, messages arrive, no reply | [The bot is connected but silent](#the-bot-is-connected-but-silent) | +| It answers some messages and not others | [The bot answers only some messages](#the-bot-answers-only-some-messages) | +| It stopped answering in a thread | [The bot stopped answering in a thread](#the-bot-stopped-answering-in-a-thread) | +| It works, then the reply never shows up | [The reply never appears](#the-reply-never-appears) | +| Approval buttons or letters do nothing | [Approvals don't work](#approvals-dont-work) | +| It answers its own messages | [The bot talks to itself](#the-bot-talks-to-itself) | + +![netclaw doctor output: 24 checks, all PASS, ending with doctor exit code: 0](/screenshots/output/doctor.png) -![netclaw doctor running diagnostic checks including Slack Auth and Slack ACL](/screenshots/output/doctor.png) +The Slack ACL and Slack Auth lines are the channel checks. Slack isn't configured in this capture, so both pass trivially. -Doctor only checks Slack right now (Auth and ACL). For Discord and Mattermost, use `netclaw status` for connector health and the daemon logs for details. +## The bot won't connect -## Authentication Errors +A bad token or a missing setting is a fatal connect error. The channel stays offline until you fix the config and restart the daemon. The daemon and your other channels keep running. Network failures are different: those retry on their own. -### Slack: Invalid or Expired Bot Token +### Slack: invalid or expired bot token -`netclaw status` shows Slack as `disconnected`. Doctor's Slack Auth check fails with one of these: +**Symptom:** `netclaw status` shows Slack as `disconnected`, and doctor's Slack Auth check fails with one of these: | Error | Meaning | |-------|---------| -| `invalid_auth` | Bot token is invalid. Check your Slack app's Bot User OAuth Token. | -| `token_revoked` | Token has been revoked. Generate a new one. | -| `token_expired` | Token has expired. Generate a new one. | -| `account_inactive` | Bot account is deactivated. | -| `not_authed` | No authentication token provided. | -| `missing_scope` | Bot token lacks `channels:read` scope. | +| `invalid_auth` | The bot token is invalid. Check your app's Bot User OAuth Token. | +| `token_revoked` | The token was revoked. Generate a new one. | +| `token_expired` | The token expired. Generate a new one. | +| `account_inactive` | The bot account is deactivated. | +| `not_authed` | No token was provided. | +| `missing_scope` | The token lacks `channels:read`, which the startup check needs. Other missing scopes show up when a call needs them, as in [The reply never appears](#the-reply-never-appears). | -If you're not sure which token is bad, regenerate both from the [Slack App Management page](https://api.slack.com/apps): +**Fix:** If you're not sure which token is bad, replace both from the [Slack app page](https://api.slack.com/apps), then restart: ```bash netclaw secrets set Slack.BotToken xoxb-your-new-token netclaw secrets set Slack.AppToken xapp-your-new-token -netclaw daemon stop && netclaw daemon start ``` -### Slack: No Bot Token Found +If doctor says "Slack is enabled but no bot token found", run `netclaw config` → Channels → Slack, or `netclaw secrets set Slack.BotToken xoxb-...`. -Doctor reports "Slack is enabled but no bot token found." +### Slack: Socket Mode is off -Run `netclaw config` → Channels → Slack, or set the token directly: +**Symptom:** the log says `Slack channel currently supports Socket Mode only. Set Slack:SocketMode to true.` -```bash -netclaw secrets set Slack.BotToken xoxb-your-token -``` +**Cause:** `Slack.SocketMode` is `false`. Only Socket Mode is supported. -### Discord: Invalid Bot Token +**Fix:** remove the key or set it to `true`, and make sure the app has Socket Mode on and a valid `xapp-...` token with `connections:write`. -`netclaw status` shows Discord as `disconnected`. Daemon logs show an authentication error at startup. +### Discord: invalid token or no access -| HTTP Code | Meaning | -|-----------|---------| -| 401 | Bot token is invalid. Check your Discord application's bot token. | -| 403 (code 50001) | Bot lacks access. Ensure it has been invited to the server. | -| 403 | Access denied. Check bot permissions. | -| 404 | Resource not found. Check the ID is correct. | +**Symptom:** `netclaw status` shows Discord as `disconnected` and the log shows an authentication error at startup. Discord's API returns HTTP 401 for a wrong or reset token. When the bot can't see a resource, the log shows the Discord error as received. -Discord uses a single bot token (not a `xoxb-`/`xapp-` pair like Slack). Regenerate it from the [Discord Developer Portal](https://discord.com/developers/applications): +**Fix:** reset the token in the [Developer Portal](https://discord.com/developers/applications), then: ```bash netclaw secrets set Discord.BotToken your-new-token -netclaw daemon stop && netclaw daemon start ``` -### Discord: Connection Timeout at Startup +Restart. If the log shows a Discord access error instead of a 401, check that the bot was invited to the server. -Daemon start hangs, then Discord shows `disconnected`. Logs show a 30-second timeout waiting for the gateway READY event. +### Discord: startup waits, then disconnects -The bot token may be valid but the bot hasn't been invited to any server, so Discord has nothing to initialize. Verify the bot has been added to your server. Use Discord's [URL Generator](https://discord.com/developers/docs/topics/oauth2#bot-authorization-flow) to create an invite link with the `bot` scope. +**Symptom:** the daemon starts slowly, and the log shows a 30-second timeout waiting for the gateway READY event. -### Mattermost: Invalid Bot Token +**Cause:** the token is valid, but the bot isn't in any server, so Discord has nothing to initialize. -`netclaw status` shows Mattermost as `disconnected`. The daemon logs show `Mattermost rejected the bot token (HTTP 401)` with more detail. +**Fix:** invite the bot with an OAuth2 URL that has the `bot` scope. See [Discord setup](/channels/discord/#5-install-the-bot-to-your-server). -The bot's personal access token is wrong or was revoked. Re-issue it from the System Console (**Integrations > Bot Accounts**) and update the secret: +### Mattermost: token or URL problems -```bash -netclaw secrets set Mattermost.BotToken your-new-token -netclaw daemon stop && netclaw daemon start -``` +**Symptom:** `netclaw status` shows Mattermost as `disconnected` with one of these reasons: -If startup logs say `Mattermost is enabled but Mattermost:ServerUrl is not configured` or `...no bot token is configured`, the corresponding field is missing from `netclaw.json`/`secrets.json`. See the [Mattermost setup guide](/channels/mattermost/). +| Reason | Fix | +|--------|-----| +| `Mattermost is enabled but Mattermost:ServerUrl is not configured` | Set `Mattermost.ServerUrl`. | +| `Mattermost is enabled but no bot token is configured` | Run `netclaw secrets set Mattermost.BotToken `. | +| `Mattermost rejected the bot token (HTTP 401)` or `Mattermost rejected the request as unauthorized` | Issue a new token under **Integrations > Bot Accounts** and store it. | -## ACL and Permissions +Restart after storing a token. The daemon doesn't watch the vault. -### All Channel Traffic Denied +If you can't create the bot or its token in the first place, two System Console settings are off by default: **Enable bot account creation** and **Enable personal access tokens**. See [Mattermost setup](/channels/mattermost/#create-a-bot-account). -Netclaw receives messages but never responds. `netclaw status` counters show `recv > 0`, `dropped > 0`, `routed = 0`. Logs show `channel_not_allowed`. +## The connection keeps dropping -This is the number one gotcha. When `AllowedChannelIds` is empty and no `DefaultChannelId`/`DefaultChannelName` is set, netclaw denies all channel traffic. +### Slack Socket Mode drops -Set a default channel or list specific channels in `~/.netclaw/config/netclaw.json`: +**Symptom:** Slack goes `disconnected`, then recovers. -```json -{ - "Slack": { - "DefaultChannelId": "C0123456789" - } -} -``` +**Cause and fix:** since 0.27.0 a supervisor checks the connection every 5 seconds and reconnects with backoff up to 5 minutes, so a restart is no longer the first step. If it never recovers, check the [Slack status page](https://status.slack.com/), your network, and that the `xapp-...` token is still valid. -Or allow specific channels: +### Discord gateway drops -```json -{ - "Slack": { - "AllowedChannelIds": ["C0123456789", "C9876543210"] - } -} -``` +**Symptom:** Discord goes `disconnected`. -After editing config, restart the daemon: `netclaw daemon stop && netclaw daemon start` +**Fix:** check [Discord's status page](https://discordstatus.com/) and your network, then confirm the token. Restart if it doesn't come back. -To find Slack channel IDs, right-click the channel name, select "View channel details," and scroll to the bottom. For Discord, enable Developer Mode in Settings > App Settings > Advanced, then right-click any channel and select "Copy Channel ID." See [Slack: Finding IDs](https://slack.com/help/articles/221769328) and [Discord: Finding IDs](https://support.discord.com/hc/en-us/articles/206346498). +### Mattermost reconnect loop -[`netclaw doctor`](/cli/doctor/) catches this one automatically. +**Symptom:** the log repeats this, with the delay growing each time ([netclaw#1517](https://github.com/netclaw-dev/netclaw/issues/1517)): -### User Not Allowed +```text +Connected to Mattermost WebSocket at ws://mattermost:8065/api/v4/websocket +Gateway requested clean reconnect: Mattermost gateway reconnected outside a clean startup cycle; forcing a clean reconnect. +Disconnected from Mattermost WebSocket: Closed by client +``` -Specific users get no response. Logs show `user_not_allowed`. +**Cause:** a startup race in older releases, reported on 0.24.2 and 0.26.0-beta.4. Startup now waits for the WebSocket `OnConnected` event, which shipped in 0.26.0-beta.5. -Add the user's ID to `AllowedUserIds`, or clear the list to allow all users: +**Fix:** upgrade to 0.27.1 or newer. If it still loops, check that `ServerUrl` is an address netclaw can reach, and that any proxy between netclaw and Mattermost forwards WebSocket upgrades to `/api/v4/websocket`. -```json -{ - "Slack": { - "AllowedUserIds": ["U0123456789", "U9876543210"] - } -} -``` +## The bot is connected but silent -### DMs Not Working +**Symptom:** `netclaw status` shows `recv` rising and `routed` at 0. The log has one of these lines: -Direct messages to the bot get no response, but channel messages work fine. Logs show `event_filtered ... reason=routing_policy_ignore ignoreReason=DmNotAllowed`. +```text +event_dropped event=... reason=channel_not_allowed +event_dropped event=... reason=user_not_allowed +event_filtered event=... reason=routing_policy_ignore ignoreReason=DmNotAllowed +``` -`AllowDirectMessages` defaults to `false`. Turn it on: +| Reason | Cause | Fix | +|--------|-------|-----| +| `channel_not_allowed` | The channel isn't in `AllowedChannelIds` and isn't the default channel. This is the number one setup mistake. | Add the channel ID, or set `DefaultChannelId`. | +| `user_not_allowed` | `AllowedUserIds` is set and the sender isn't in it. | Add the user's ID, or clear the list. | +| `DmNotAllowed` | `AllowDirectMessages` is `false`, the default. | Set it to `true`. | ```json { "Slack": { + "AllowedChannelIds": ["C0123456789"], + "AllowedUserIds": ["U0123456789"], "AllowDirectMessages": true } } ``` -Doctor warns if you enable DMs with an empty `AllowedUserIds` list, since any workspace member can then DM the bot. - -### Channel Messages Silently Dropped After Upgrading from Pre-0.24.0 +[`netclaw doctor`](/cli/doctor/) catches an empty allow-list for Slack. To find IDs, see [Slack](/channels/slack/#channels), [Discord](/channels/discord/#channels) or [Mattermost](/channels/mattermost/#how-channel-ids-are-stored). -Bot is connected and the channel is in `AllowedChannelIds`, but every message is dropped. Logs show `channel_not_allowed`. Started after upgrading to 0.24.0 or later. +If no `event_dropped` line appears at all, the event never reached netclaw. On Slack and Mattermost, invite the bot to the channel. On Discord, check the Message Content intent. -**Cause:** Before 0.24.0, `AllowedChannelIds` accepted display names (e.g. `"general"`) or manually entered values that were never resolved to canonical IDs. As of 0.24.0, incoming message channel IDs are matched against stored IDs only — display names no longer match. +### Everything stopped after upgrading from before 0.24.0 -**Fix:** Re-enter the channels via `netclaw config` → Channels so they resolve to canonical IDs and are saved correctly. This is the easiest path — the config UI does the resolution automatically. +**Symptom:** the channel is in `AllowedChannelIds`, but every message logs `channel_not_allowed`. It began after upgrading to 0.24.0 or later. -If you prefer to fix it manually in `~/.netclaw/config/netclaw.json`, replace any display names in `AllowedChannelIds` with the canonical channel ID for each platform: +**Cause:** before 0.24.0, `AllowedChannelIds` accepted display names such as `"general"`. Since 0.24.0, incoming channel IDs match stored IDs only. -- **Slack:** right-click a channel name → "View channel details" → scroll to the bottom for the `C...` ID. See [Slack: Finding IDs](https://slack.com/help/articles/221769328). -- **Discord:** enable Developer Mode in Settings > App Settings > Advanced, then right-click any channel → "Copy Channel ID." -- **Mattermost:** find IDs in the System Console or via the [REST API](https://api.mattermost.com/). +**Fix:** re-enter the channels in `netclaw config` → Channels so they resolve to IDs. Or edit `netclaw.json` and swap each name for its canonical ID. -After editing, restart the daemon: `netclaw daemon stop && netclaw daemon start` +### The allow-list is right but the bot is still silent in Docker or Compose -### Bot Not Responding to Messages (MentionOnly) +**Cause:** an environment variable and `netclaw.json` both set the same list. .NET merges them by position, so a blank `NETCLAW_Mattermost__AllowedChannelIds__0` overwrites the first channel in the file. -Bot is connected and healthy but only responds to some messages. Logs show `event_filtered ... reason=routing_policy_ignore ignoreReason=ChannelMentionRequired`. +**Fix:** keep each list in one place, then check the value the container sees, for example `docker exec printenv NETCLAW_Mattermost__AllowedChannelIds__0`. -`MentionOnly` defaults to `true`, so the bot ignores messages that don't @-mention it. Either @-mention the bot every time, or turn it off: +## The bot answers only some messages -**Slack:** +**Symptom:** the bot is healthy but ignores messages that don't mention it. The log has: -```json -{ - "Slack": { - "MentionOnly": false - } -} +```text +event_filtered event=... reason=routing_policy_ignore ignoreReason=ChannelMentionRequired ``` -**Discord:** +**Cause:** `MentionOnly` defaults to `true`, so in channels the bot answers only messages that @-mention it. + +**Fix:** mention the bot, or turn the rule off for the channel type: ```json { - "Discord": { - "MentionOnly": false, - "MentionRequiredInDm": false - } + "Slack": { "MentionOnly": false }, + "Discord": { "MentionOnly": false }, + "Mattermost": { "MentionOnly": false } } ``` -`MentionRequiredInDm` controls whether DMs also need a mention (defaults to `false`). +`MentionRequiredInDm` separately controls whether DMs need a mention (default `false`). On Slack, a file upload follows the same rule, so an un-mentioned attachment in a channel is ignored. -## Connectivity +## The bot stopped answering in a thread -### Slack Socket Mode Disconnected +**Symptom:** the bot was answering in a thread, then went quiet. The log has: -`netclaw status` shows Slack as `disconnected`. Was working, then stopped. +```text +event_filtered event=... reason=routing_policy_ignore ignoreReason=ThreadMentionRequired +``` -Restart the daemon: +**Cause:** `MentionRequiredInThreadByChannel` is `true` for that channel, so thread replies need a mention. A channel you added through `netclaw config` as `team` or `public` starts with the rule on. -```bash -netclaw daemon stop && netclaw daemon start -``` +**Fix:** mention the bot in the reply. It then re-reads what it missed and answers with full context. To turn the rule off, press **Space** on the channel in `netclaw config` → Channels, or set the channel's entry to `false` in `netclaw.json`. + +If the thread is old, check the idle limit too. A thread is freed from memory when its session goes idle, after `Session.IdleTimeout` (one hour by default), and a later reply wakes it again. A restart backfills history. + +## The reply never appears -If it keeps disconnecting, check the [Slack Status page](https://status.slack.com/) and your network. Make sure the App Token (`xapp-...`) is still valid; Socket Mode requires it. +### Slack: the post fails -Only Socket Mode is supported. Setting `SocketMode: false` leaves Slack unable to connect, so it shows as degraded in `netclaw status` and the daemon logs a connection error. +**Symptom:** the log shows the message was processed, but nothing shows up in Slack, with one of these errors: -### Discord Gateway Disconnected +| Error | Meaning | +|-------|---------| +| `not_in_channel` | The bot isn't in the channel. Invite it with `/invite @botname`. | +| `channel_not_found` | The channel ID is wrong or the channel was deleted. | +| `missing_scope` | The token lacks a scope needed to post. | +| `no_permission` | The bot can't post in this channel. | +| `msg_too_long` | The reply exceeded Slack's size limit. | +| `invalid_blocks` | Slack rejected the formatted payload. | +| `too_many_attachments` | The reply has too many attachments. | +| `rate_limited` | Slack throttled the call. netclaw retries. | -`netclaw status` shows Discord as `disconnected`. +For permission errors, invite the bot and check the scopes under **OAuth & Permissions** in the [Slack API dashboard](https://api.slack.com/apps). Reinstall the app after you change scopes. -Restart the daemon: `netclaw daemon stop && netclaw daemon start`. If it keeps dropping, check the [Discord Status page](https://discordstatus.com/) and verify the bot token. +### Discord: the post fails -### Rate Limiting +**Symptom:** the bot processes the message but no reply shows up. -Replies are delayed or fail intermittently. Logs show `rate_limited` (Slack) or HTTP 429 (Discord). +**Cause:** usually the bot lacks **Send Messages**, **Send Messages in Threads** or **Create Public Threads** in that channel. The log has the HTTP status. A 429 is Discord rate limiting, and netclaw retries. -Netclaw retries on its own. If rate limiting is sustained, check whether another integration shares the same bot token's rate limit budget. +**Fix:** grant the permission in the channel's settings. -## Message Delivery +### Mattermost: the post fails -### Slack: Reply Not Posted +**Symptom:** the log shows a reply error after the message was processed. -Bot processes the message (logs show session activity) but the reply never appears in Slack. +**Cause:** the bot isn't a member of the channel, or it lost access to it. -| Error | Meaning | -|-------|---------| -| `not_in_channel` | Bot hasn't been invited to the channel. Invite it with `/invite @botname`. | -| `channel_not_found` | Channel ID is wrong or the channel was deleted. | -| `missing_scope` | Bot token lacks the required OAuth scope for posting. | -| `no_permission` | Bot doesn't have permission to post in this channel. | -| `msg_too_long` | Response exceeded Slack's message size limit. | -| `invalid_blocks` | Malformed Block Kit payload. Usually a formatting edge case in the response. | -| `too_many_attachments` | Response has too many attachments. | +**Fix:** add the bot to the channel with `/invite @netclaw`. The channel's `reply_failed` counter in `netclaw status` rises while this is broken. -For permission errors: add the bot to the channel and make sure it has `chat:write` scope. Check your app's OAuth scopes in the [Slack API dashboard](https://api.slack.com/apps). +## Approvals don't work -### Discord: Reply Not Posted +| Symptom | Cause | Fix | +|---------|-------|-----| +| Lettered prompt, and your reply is ignored | You aren't the user who triggered the request. The bot posts "Only the requesting user can approve this tool action." | Have the original user reply, or ask again yourself. | +| Mattermost buttons never respond | `CallbackUrl` is set but the Mattermost server can't reach the daemon. | Fix the route, or leave `CallbackUrl` unset if you don't use sub-agents. See [Mattermost](/channels/mattermost/#buttons-need-a-reachable-callback). | +| Mattermost button says "no longer valid" | The daemon restarted. Button tokens live in memory. | Ask the agent to re-issue the prompt. | +| A sub-agent's call is denied with `approval_prompt_unavailable`, and its prompt shows "withdrawn ... could not show it in this channel" | Mattermost has no `CallbackUrl`, and sub-agent prompts need buttons. | Set `CallbackUrl`. See [Mattermost](/channels/mattermost/#buttons-need-a-reachable-callback). | +| A typed letter is ignored and the bot says "N approval prompts are open. Use the buttons on the prompt you mean." | Two or more prompts are open for you and one is from a sub-agent. A letter can't pick one. | Click the buttons on the prompt you mean. | +| "That approval prompt already has an answer" or "Approval response ignored: that approval prompt already has an answer." | A second click or reply on a prompt that was already answered. | Nothing to fix. The first answer stands. | +| A prompt lost its buttons and shows a withdrawal reason | The task that asked was cancelled, ended or stopped, the daemon restarted, or the channel couldn't show the prompt. Nothing ran. | Ask the agent to try again. | +| No prompt appears and the call doesn't run | The channel couldn't post the prompt. netclaw tells the model nobody saw it. | Fix the post error above, then ask again. | -Bot processes the message but no reply appears in Discord. +The choices on each prompt are explained in [How tool approvals work](/security/approvals/). -Check daemon logs for HTTP error codes. Usually the bot is missing `Send Messages` permission in the target channel. Fix it in your Discord server's channel permission settings. +## The bot talks to itself -### Bot Reply Loop +**Symptom:** the bot answers its own messages in a loop. `events.filtered{reason="bot_message"}` and `events.received` are both high. -Bot responds to its own messages, creating an infinite loop. High `events.filtered{reason="bot_message"}` counter alongside high `events.received`. +**Cause:** netclaw filters its own messages, so this is rare. It points to a failed bot-identity lookup at startup. -Netclaw filters its own messages to prevent loops, so this is rare. Restart the daemon. If the loop comes back, bot user ID detection probably failed at startup. Look for errors around the `slack.Auth.Test()` call in the logs. +**Fix:** restart the daemon. If it comes back, look in the log around startup for an authentication error on the channel. -## Debug Logging +## Turn on debug logging -Still stuck? Turn on debug logging in `~/.netclaw/config/netclaw.json`: +Still stuck? Set the log level in `~/.netclaw/config/netclaw.json`: ```json { @@ -280,60 +300,48 @@ Still stuck? Turn on debug logging in `~/.netclaw/config/netclaw.json`: } ``` -Restart the daemon, trigger the failing interaction, and look for these patterns: +The daemon restarts on the edit. Reproduce the problem and read the daemon log. A healthy Slack message flow looks like this: -**Healthy message flow (Slack):** -1. `Routing Slack event {id} to conversation {channelId}` — [SlackGatewayActor, Debug] -2. `turn_routed event={EventId} hasFiles=... textChars=...` — [SlackConversationActor] -3. `turn_received textChars=... fileCount=...` — [SlackThreadBindingActor] -4. `turn_enqueued contentItems=...` — [SlackThreadBindingActor] -5. `Posted Slack reply message` — [SlackThreadBindingActor] +1. `Routing Slack event {id} to conversation {channelId}` (Debug) +2. `turn_routed event=... hasFiles=... textChars=...` +3. `turn_received textChars=... fileCount=...` +4. `turn_enqueued contentItems=...` +5. `Posted Slack reply message` -If step 1 never appears, the message was dropped before routing — check ACL config (`channel_not_allowed`, `user_not_allowed`). If step 1 appears but step 2 does not, the routing policy filtered it (look for `event_filtered reason=routing_policy_ignore`). If step 4 appears but step 5 does not, the reply couldn't be posted — check the error tables above. +If step 1 never appears, the event was dropped before routing, so check the ACL reasons above. If step 1 appears but not step 2, the routing policy filtered it. If step 4 appears but not step 5, the reply couldn't be posted, so check [The reply never appears](#the-reply-never-appears). -**Quick triage commands:** +To see whether sessions are being written at all: ```bash -# Daemon process status -netclaw daemon status - -# Recent crash logs -ls -lt ~/.netclaw/logs/crash-*.log - -# Recent session activity (SQLite) sqlite3 ~/.netclaw/netclaw.db \ "SELECT persistence_id, MAX(created), MAX(sequence_number) \ FROM journal GROUP BY persistence_id \ ORDER BY MAX(created) DESC LIMIT 10;" ``` -## OTLP Diagnostic Patterns +## Read the metrics -If [OpenTelemetry](/observability/opentelemetry/) is enabled, channel metrics tell the story faster than logs: +If [OpenTelemetry](/observability/opentelemetry/) is on, channel metrics are faster than logs. They use the namespaces `netclaw.channel.slack.*`, `netclaw.channel.discord.*` and `netclaw.channel.mattermost.*`. -| Pattern | Metrics | Likely Cause | +| Pattern | Metrics | Likely cause | |---------|---------|--------------| -| Messages received, none routed | `events.received > 0`, `events.routed = 0` | ACL blocking all traffic | -| Messages routed, no replies | `messages.enqueued > 0`, `replies.posted = 0` | Outbound delivery failure | -| High drop rate | `events.dropped` spiking with `reason=channel_not_allowed` | Channel not in allow list | -| Suspected loop | High `events.filtered{reason="bot_message"}` + high `events.received` | Bot message filter working, but volume suggests upstream issue | - -Channel metrics use the namespace `netclaw.channel.slack.*`, `netclaw.channel.discord.*`, and `netclaw.channel.mattermost.*`. - -## Related Pages - -- [`netclaw doctor`](/cli/doctor/) — offline diagnostics -- [`netclaw status`](/cli/status/) — live connector health and message counters -- [`netclaw secrets`](/cli/secrets/) — manage encrypted tokens -- [`netclaw config`](/cli/config/) — Channels -- [OpenTelemetry](/observability/opentelemetry/) — OTLP metrics reference - -## External Resources - -- [Slack API: Bot tokens and permissions](https://api.slack.com/authentication/token-types#bot) — token types and required scopes -- [Slack API: Socket Mode](https://api.slack.com/apis/socket-mode) — how Socket Mode connections work -- [Slack: Finding Channel and User IDs](https://slack.com/help/articles/221769328) — locate IDs for ACL config -- [Discord Developer Docs: Getting Started](https://discord.com/developers/docs/getting-started) — bot setup and permissions -- [Discord: Finding IDs](https://support.discord.com/hc/en-us/articles/206346498) — enable Developer Mode to copy IDs -- [Slack Status](https://status.slack.com/) — check for Slack API outages -- [Discord Status](https://discordstatus.com/) — check for Discord API outages +| Messages received, none routed | `events.received > 0`, `events.routed = 0` | ACL is blocking everything | +| Messages routed, no replies | `messages.enqueued > 0`, `replies.posted = 0` | Outbound delivery is failing | +| Many drops with one reason | `events.dropped` rising with `reason=channel_not_allowed` | Channel isn't in the allow-list | +| Suspected loop | High `events.filtered{reason="bot_message"}` and high `events.received` | The bot-message filter is working, so look upstream | + +## Related pages + +- [Slack](/channels/slack/), [Discord](/channels/discord/) and [Mattermost](/channels/mattermost/) - setup and behavior for each channel +- [`netclaw status`](/cli/status/) - live connector health and counters +- [`netclaw doctor`](/cli/doctor/) - offline diagnostics +- [`netclaw secrets`](/cli/secrets/) - replace a stored token +- [OpenTelemetry](/observability/opentelemetry/) - the metrics reference + +## Resources + +- [Slack API: token types](https://api.slack.com/authentication/token-types#bot) - bot tokens and required scopes +- [Slack: Socket Mode](https://docs.slack.dev/apis/events-api/using-socket-mode) - how Socket Mode connections work +- [Discord: getting started](https://discord.com/developers/docs/getting-started) - bot setup and permissions +- [Mattermost: bot accounts](https://developers.mattermost.com/integrate/reference/bot-accounts/) - tokens and permissions +- [Slack Status](https://status.slack.com/) and [Discord Status](https://discordstatus.com/) - check for outages diff --git a/src/content/docs/cli/approvals.md b/src/content/docs/cli/approvals.md index 130cf43..cc6bf5b 100644 --- a/src/content/docs/cli/approvals.md +++ b/src/content/docs/cli/approvals.md @@ -1,187 +1,212 @@ --- title: "netclaw approvals" -description: "Inspect, revoke, and manage persistent tool approvals." +description: "List, revoke and pre-approve the grants netclaw saves when you answer Always at an approval prompt." --- -When netclaw's [approval gates](/architecture/security-model/#human-in-the-loop-as-a-layer-not-a-crutch) prompt you to allow a tool invocation, your "always allow" choices get saved to `~/.netclaw/config/tool-approvals.json`. This command lets you see what's been approved, revoke entries you no longer want, and add global trust rules. Running `netclaw approvals` with no subcommand opens the [interactive TUI](#tui). +When you answer **Always here**, **This repository** or **Always anywhere** at an approval prompt, netclaw saves a grant to `~/.netclaw/config/tool-approvals.json`. `netclaw approvals` lists those grants, revokes the ones you no longer want, and adds new ones before a prompt ever appears. Run it with no subcommand to open the [interactive TUI](#tui). -![Approval gate prompt in a Slack thread, with approve and deny options](/assets/approval-prompt.png) +This page is the reference for the command and the file. For how grants fit into the authorization model, see [How tool approvals work](/security/approvals/). To stop prompts for a command or a tool, see [Reduce or turn off approval prompts](/guides/reduce-approval-prompts/). -The two "Always" options in the channel prompt — **Always here** (the command's verb, scoped to the current directory) and **Always anywhere** (a global grant) — are what create persistent entries in `tool-approvals.json`. - -Audiences (`personal`, `team`, `public`) scope approvals to who's talking to the agent. An approval granted under `personal` doesn't apply when a team member triggers the same command. See [audience dispositions](/architecture/security-model/#audience-as-the-trust-primitive) for background. - -Don't hand-edit the JSON file. Use this command instead. +Grants belong to one audience (`personal`, `team` or `public`) and one tool. A `personal` grant doesn't apply when a Team user triggers the same command. Don't hand-edit the file. The CLI keeps entries valid. ## Usage ```bash -netclaw approvals # open TUI -netclaw approvals list [options] # print approvals -netclaw approvals revoke [options] # remove an approval -netclaw approvals trust-verb [options] # add a global approval -netclaw approvals tui # open TUI (explicit) -netclaw approvals help # show help +netclaw approvals # open the TUI +netclaw approvals list [options] # print grants +netclaw approvals revoke