diff --git a/.claude/skills/netclaw-docs/SKILL.md b/.claude/skills/netclaw-docs/SKILL.md index 3a22ecb..1719657 100644 --- a/.claude/skills/netclaw-docs/SKILL.md +++ b/.claude/skills/netclaw-docs/SKILL.md @@ -3,64 +3,93 @@ name: netclaw-docs description: Voice and style contract for netclaw.dev docs (Astro + Starlight, dark-mode only). The how-it-should-sound layer on top of CLAUDE.md's authoring process. --- -# netclaw.dev Docs Voice +# netclaw.dev docs voice -Write like you're explaining netclaw to a sharp coworker who knows Linux, Docker, and Slack but has never touched netclaw. Confident, casual-technical, code-first. Open by telling the reader what the thing does and when they'd reach for it, then get out of the way and show commands. The published pages earn trust by being correct and dense, not by being long. CLAUDE.md owns the process (page types, templates, the critique/humanizer pipeline); this skill owns the voice. +Write like you're explaining netclaw to a sharp coworker who knows Linux, Docker, and Slack but has never touched netclaw. Confident, casual-technical, code-first. Open by saying what the thing does and when you'd reach for it, then show commands. The pages earn trust by being correct and dense, not long. CLAUDE.md owns the process (page types, templates, critique and humanizer steps). This skill owns how the page reads. -## Do - -- **Open with the payload.** First sentence states what the feature does and when you'd use it. No "In this section" runway. See `getting-started/installation.md`: "Netclaw has two components..." -- **Use direct imperative voice.** "Run `netclaw status`." "Store tokens with `netclaw secrets`." Never "You can run" or "Users may wish to." -- **Show, don't tell.** A config block, command, or screenshot beats a paragraph. If CLI help already explains a flag, show the output instead of rewriting it. -- **Tables for anything with a shape** - flags, config fields (Field / Type / Default / Description), scopes, error→cause→fix, mode comparisons. Prose is for the one idea a table can't hold. -- **Name the #1 mistake.** The best pages flag the common footgun in bold ("This is the number one setup mistake"). Lead readers away from the trap. -- **Use Starlight callouts** `:::caution` / `:::note` for breaking changes, footguns, and security boundaries. Reserve them for things that bite. -- **Comment real screenshots in, stub honest TODOs out.** `` and `` are the house pattern; never ship fake "coming soon" prose. -- **Close with cross-links.** End with Related pages + Resources (2+ external links to upstream docs/RFCs). +## Relationship to aarons-voice and humanizer -## Don't +For anything under `src/content/docs/`, this skill supersedes `aarons-voice` and both humanizer skills. It already contains the parts of `aarons-voice` that apply to product docs, so don't load it alongside this one. Deliberately not carried over: bare URLs (use markdown links with descriptive text), the no-bold rule (see Bold), first-person "I", exclamation points, and the email phrase list. Humanizer runs on prose only. Where it disagrees with this skill, this skill wins. -- No filler: "It is important to note that", "This section describes", "The following table shows", "In this guide we will". Cut the sentence; keep the table. -- No hedging: "you may want to", "it is recommended that", "it should be noted". State it. -- No rule-of-three padding ("fast, simple, and powerful") or negative parallelism ("not X, but Y") used for rhythm. -- No inflated symbolism, no "seamless / robust / leverage / delve". Plain verbs. -- Don't explain the obvious. If a reader of this audience already knows it, delete it. -- Don't restate code in prose right after showing it. +## Two registers -## Punctuation +- **Reference** (`cli/`, `configuration/`, `channels/`, `deployment/`, `observability/`, most `guides/`): "you" plus imperative. netclaw is the subject of behavior sentences ("netclaw resolves symlinks first"). No "we", no "I". Opinions stop at "use X unless Y". Exact beats charming. +- **Narrative** (design philosophy, comparisons, tutorials, the landing page): "we" is fine for the project's decisions and opinions. State opinions as opinions. A little roughness is fine. Still never "I". -Use the spaced hyphen ` - ` (space-hyphen-space), not the em-dash (`—`), for parenthetical asides and appositives. As of 2026-07-22 the docs drop the em-dash to match the house writing voice, where the em-dash reads as an AI tell and ` - ` is the connector that replaces it. Outbound email and marketing already ban the em-dash, so there's now one rule everywhere - no docs-versus-email split to remember. +A page is one or the other. If a reference page needs a "why", link to the narrative page. -Style notes for ` - `: it joins clauses mid-sentence only, never after a terminal `.`, `?`, or `!`. If the prior sentence ended, start a new one. Don't stack more than one per sentence, and skip it where a period, colon, comma, or parentheses read cleaner. +## Do -Migration: use ` - ` in every new page and in any page you edit. The ~50 legacy pages still contain em-dashes (`security-model.md` alone has ~38) and stay as they are until you edit the page for another reason, or a dedicated sweep converts them. Some channel pages use the double-hyphen ` -- ` instead. Normalize those to ` - ` when you touch them. +- **Open with the payload.** First sentence says what the feature does and when you'd use it. +- **Lead guides with the answer.** Put an "I want / do this" table first where it fits, then explain. +- **Use the words people search for.** Titles and headings say "turn off", "stop asking", "disable", not internal names. +- **Name the thing.** "Slack, Discord, or Mattermost", not "your chat app". +- **Show, don't tell.** A config block, command or screenshot beats a paragraph. Tables for anything with a shape. Prose is for the one idea a table can't hold, and never restates the table. +- **Name the number one mistake**, once per section at most. +- **Use `:::caution` and `:::note`** for breaking changes, footguns and security boundaries only. +- **One concept page, one guide, one reference per topic.** Other pages link instead of re-explaining. +- **Stub honestly.** `` and ``. Never ship "coming soon" prose. -## Per-page-type quick reference +## Don't -- **CLI** (`cli/`): one-line intent → `## Usage` → options table → examples with a screenshot each → Related Commands. -- **Tutorial** (`getting-started/`): what we're building → Prerequisites (with versions) → numbered Steps with exact commands + expected output → Verify It Works → Next Steps. -- **Guide** (`guides/`): the problem in one line → Before You Begin → steps (call out conditional branches) → Troubleshooting (symptom / cause / fix). -- **Concept** (`architecture/`, `security/`): 2–3 paragraph mental model → key-concept sections → Limitations. -- **Configuration** (`configuration/`): what routes through this → minimal JSON block → full fields table (Field/Type/Default) → invalid-config behavior → diagnosing-errors table. -- **Integration** (`channels/`): transport in one line → Prerequisites → third-party setup (app/tokens/scopes, scopes table) → Configure netclaw → Access control → Verify it works → Troubleshooting. -- **Operations** (`deployment/`): when you need a non-default mode → modes table → per-mode config → restart/validation behavior → `netclaw doctor` troubleshooting. +- No filler ("It is important to note", "This page covers", "The following table shows") and no hedging ("you may want to", "it is recommended"). +- No "No X, no Y, no Z" more than once per page. No slogan triads or punchline fragments for rhythm ("That's unrealistic."). +- No screenshot captions shaped like `Label - fragment; fragment`. A caption is one sentence. +- Cut: "surface" as a noun, "just" as a minimiser, "walks through", "the full breakdown", "out of the box" (say "by default"), "first-class", "at a glance", "worth knowing". +- Don't explain what this audience already knows. Don't restate code in prose after showing it. +- Don't sand good sentences flat to chase a clean score. Fix the tell and keep the sentence. A page that passes every check and reads like a compliance form is worse than one with a stray dash. -## Terminology & product facts +## Punctuation -- **netclaw** is lowercase in running prose. CLI is `netclaw`; daemon is `netclawd`. -- Commands and flags go in backticks: `netclaw status`, `--channel beta`. Config keys too: `Search.Backend`, `Daemon.ExposureMode`. -- **`netclaw init`** = focused 5-step first-run setup (provider, identity, security posture, features) that also powers factory reset. Don't pile post-install config into it. -- **`netclaw config`** = the re-entrant, menu-driven dashboard for everything else (channels, search, exposure, webhooks, skills); safe to re-run, autosaves on completion. -- Cross-link slugs: `/cli//`, `/channels//`, `/configuration//`, `/deployment//`, `/security//`, `/getting-started//`. Trailing slash, lowercase. -- Config lives in `~/.netclaw/config/netclaw.json`; secrets in `~/.netclaw/config/secrets.json` (encrypted at rest, written via `netclaw secrets set`). Env override prefix is `NETCLAW_` with `__` path separators. +- **Contractions everywhere:** it's, don't, you'll, can't. Spell out "do not" or "cannot" only inside a `:::caution` or a security boundary statement, where the stiffness is the point. +- **No semicolons in prose.** Write two sentences. +- **No exclamation points.** No emoji, except status glyphs that mirror real CLI output. +- **Sentence-case headings:** `## Related pages`, `## Before you begin`. Proper nouns and command names keep their case. +- **No em dashes at all.** Pick the replacement by position: + - List separators and table cells: ` - ` or a colon. An empty cell is `-`. + - Headings: a colon (`### 401 Unauthorized: secret mismatch`). Heading edits change anchors, so check inbound links. + - Prose: rewrite with a period, colon or comma first. Keep ` - ` for a real aside, about once per paragraph at most, never after a terminal `.`, `?` or `!`. + - Don't search-and-replace. Forty em dashes turned into forty spaced hyphens is a new tic. Normalize ` -- ` the same way. + +## Bold + +Four uses only: UI labels the reader looks for, one warning per section, Symptom / Cause / Fix labels, and a term where it's defined. No bolded thesis sentences. No bold-label bullet lists where a table or subheadings would do. + +## Structure + +- Every page ends with exactly `## Related pages` (three to five internal links) then `## Resources` (two or more external links). No "Next steps", "See also" or "Keep reading". Tutorials may put `## What's next` before them. +- Every screenshot has the instruction before it and a one-sentence caption after it that says something the alt text doesn't. +- Page-type shapes: + - **CLI:** intent, `## Usage`, options table, examples with a screenshot each. + - **Tutorial:** goal, prerequisites with versions, numbered steps with expected output, verify it works. + - **Guide:** problem in one line, before you begin, steps, troubleshooting (Symptom / Cause / Fix). + - **Concept:** two or three paragraphs of mental model, key sections, limitations. + - **Configuration:** what routes through it, minimal JSON, fields table (Field / Type / Default), invalid-config behavior. + - **Integration:** transport in one line, prerequisites, third-party setup, configure netclaw, access control, verify, troubleshooting. + - **Operations:** when you need a non-default mode, modes table, per-mode config, restart behavior, `netclaw doctor`. + +## Terminology and product facts (0.27.1) + +- netclaw is lowercase in running prose. The CLI is `netclaw`, the daemon is `netclawd`. Commands, flags and config keys go in backticks. +- `netclaw init` is first-run bootstrap only: provider, identity, security posture, then health check. That's four steps for Personal and five for Team or Public, which add a feature-selection step. Channels, search, exposure, webhooks and skill sources live in `netclaw config`, the re-entrant menu dashboard. +- Default base directory is `~/.netclaw`, or `$NETCLAW_HOME` if set. Config is `config/netclaw.json`, secrets are `config/secrets.json` (written with `netclaw secrets set`). Env overrides use the `NETCLAW_` prefix with `__` between path segments. +- `Ctrl+Q` quits every TUI. Escape is per screen: it never quits the config, provider, model, init or chat screens (in chat it denies a pending approval), but it does quit from the `netclaw mcp` server list. State keys per screen from the key-handling code, never as a global rule. +- There is no `netclaw daemon restart`. The daemon restarts itself, after draining sessions, when `netclaw.json` changes. Nothing else triggers it, so secrets and environment changes need a manual restart: + - native: `netclaw daemon stop`, then `netclaw daemon start` + - systemd user unit: `systemctl --user restart netclaw` + - Docker: `docker restart ` (the image's entrypoint supervises the daemon) +- Screenshots and command output come from the release container (`ghcr.io/netclaw-dev/netclaw:`), never a host binary. No private hostnames in samples or screenshots. Use `llm.example.internal` or a named public provider. +- Cross-link slugs are lowercase with a trailing slash: `/cli//`, `/channels//`, `/configuration//`. ## Pre-ship checklist -- [ ] Frontmatter `title` + `description` set. -- [ ] Opens with what/when, not runway. No filler or hedge phrases. -- [ ] Reference material is in tables; code/screenshots carry the weight. -- [ ] netclaw lowercase; commands and config keys backticked; cross-links use correct trailing-slash slugs. -- [ ] No placeholder prose - only `` comments for gaps; all referenced screenshots exist. -- [ ] 2+ external resource links. -- [ ] Every runnable snippet was actually run against the current release. -- [ ] `/humanizer` pass done; `npm run build` passes. +- [ ] Frontmatter `title` and `description` set. Headings in sentence case and built from search words. +- [ ] Opens with what and when. Guides lead with the answer. No filler, hedges or cut-list phrases. +- [ ] No em dashes, ` -- `, or semicolons in prose. ` - ` within budget. Contractions used. +- [ ] Reference page has no "we". No page has "I". Bold only in the four permitted uses. +- [ ] Tables carry reference material and aren't restated in prose. +- [ ] Every screenshot has an instruction before it and a one-sentence caption after it, and the file exists. +- [ ] Ends with `## Related pages` then `## Resources` (2+ external links). +- [ ] Every runnable snippet was run against the release container. No private hostnames. +- [ ] No placeholder prose. Only `` comments for gaps. +- [ ] `humanizer:humanizer` pass done on prose only. `npm run build` passes. diff --git a/.github/workflows/deploy.yml b/.github/workflows/deploy.yml index d107a59..a507863 100644 --- a/.github/workflows/deploy.yml +++ b/.github/workflows/deploy.yml @@ -11,7 +11,8 @@ on: jobs: deploy: name: "Deploy to Cloudflare Pages" - runs-on: ubuntu-latest + # Pinned: the Playwright version in package-lock.json has no Chromium build for ubuntu-latest (26.04). + runs-on: ubuntu-24.04 environment: name: production url: https://netclaw.dev @@ -31,6 +32,25 @@ jobs: - name: Install dependencies run: npm ci + - name: Cache Playwright browser + id: playwright-cache + uses: actions/cache@v4 + with: + path: ~/.cache/ms-playwright + key: playwright-${{ runner.os }}-${{ hashFiles('package-lock.json') }} + + # Mermaid diagrams render at build time through a headless browser. Without it + # the build still succeeds but pages with diagrams ship with an empty body + # (scripts/check-rendered-pages.mjs fails the build when that happens). + # The browser revision follows the playwright version pinned in package-lock.json. + - name: Install Playwright browser + run: | + if [ "${{ steps.playwright-cache.outputs.cache-hit }}" = "true" ]; then + npx playwright install-deps chromium-headless-shell + else + npx playwright install --with-deps chromium-headless-shell + fi + - name: Build site run: npm run build env: diff --git a/.github/workflows/pr-validation.yml b/.github/workflows/pr-validation.yml index 3a590a5..7f02316 100644 --- a/.github/workflows/pr-validation.yml +++ b/.github/workflows/pr-validation.yml @@ -8,7 +8,8 @@ on: jobs: build-site: name: "Build Astro" - runs-on: ubuntu-latest + # Pinned: the Playwright version in package-lock.json has no Chromium build for ubuntu-latest (26.04). + runs-on: ubuntu-24.04 steps: - name: Checkout uses: actions/checkout@v4 @@ -22,12 +23,32 @@ jobs: - name: Install dependencies run: npm ci + - name: Cache Playwright browser + id: playwright-cache + uses: actions/cache@v4 + with: + path: ~/.cache/ms-playwright + key: playwright-${{ runner.os }}-${{ hashFiles('package-lock.json') }} + + # Mermaid diagrams render at build time through a headless browser. Without it + # the build still succeeds but pages with diagrams ship with an empty body + # (scripts/check-rendered-pages.mjs fails the build when that happens). + # The browser revision follows the playwright version pinned in package-lock.json. + - name: Install Playwright browser + run: | + if [ "${{ steps.playwright-cache.outputs.cache-hit }}" = "true" ]; then + npx playwright install-deps chromium-headless-shell + else + npx playwright install --with-deps chromium-headless-shell + fi + - name: Build site run: npm run build link-validation: name: "Link and URL Validation" - runs-on: ubuntu-latest + # Pinned: the Playwright version in package-lock.json has no Chromium build for ubuntu-latest (26.04). + runs-on: ubuntu-24.04 steps: - name: Checkout uses: actions/checkout@v4 @@ -41,6 +62,25 @@ jobs: - name: Install dependencies run: npm ci + - name: Cache Playwright browser + id: playwright-cache + uses: actions/cache@v4 + with: + path: ~/.cache/ms-playwright + key: playwright-${{ runner.os }}-${{ hashFiles('package-lock.json') }} + + # Mermaid diagrams render at build time through a headless browser. Without it + # the build still succeeds but pages with diagrams ship with an empty body + # (scripts/check-rendered-pages.mjs fails the build when that happens). + # The browser revision follows the playwright version pinned in package-lock.json. + - name: Install Playwright browser + run: | + if [ "${{ steps.playwright-cache.outputs.cache-hit }}" = "true" ]; then + npx playwright install-deps chromium-headless-shell + else + npx playwright install --with-deps chromium-headless-shell + fi + - name: Build site run: npm run build diff --git a/AGENTS.md b/AGENTS.md index 29b4b45..be1e61b 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -4,7 +4,7 @@ netclaw.dev documentation site — Astro 6.2 + Starlight, dark-mode only, deploy Two guides govern work here. Read both before writing or editing any doc page: -- **[CLAUDE.md](./CLAUDE.md)** — the authoring *process*: page types, per-type templates, the screenshot pipeline, the critique + `/humanizer` review steps, and build/preview commands. +- **[CLAUDE.md](./CLAUDE.md)** — the authoring *process*: page types, per-type templates, the screenshot pipeline, the critique + `humanizer:humanizer` review steps, and build/preview commands. - **[`.claude/skills/netclaw-docs/SKILL.md`](./.claude/skills/netclaw-docs/SKILL.md)** — the *voice and style* contract: the "less is more" house voice, punctuation policy, per-page-type quick reference, terminology, and the pre-ship checklist. CLAUDE.md owns *what* to write and *how to ship it*; the `netclaw-docs` skill owns *how it reads*. Use both. diff --git a/CLAUDE.md b/CLAUDE.md index 51815c4..867ade9 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -26,9 +26,9 @@ Read the target markdown stub in `src/content/docs/` to get the title and descri Before asking the user anything, collect available information: -1. **Netclaw source repo** — Clone `https://github.com/netclaw-dev/netclaw` (or locate a local clone). Key paths relative to that repo root: +1. **Netclaw source repo** — The local clone is at `~/repositories/netclaw-dev/netclaw` (or clone `https://github.com/netclaw-dev/netclaw`). Check out the release tag the docs target (`src/version.json`). Key paths relative to that repo root: - `src/Netclaw.Cli/` — CLI commands, flags, help text - - `src/Netclaw.Configuration/` — Config schemas, defaults, validation + - `src/Netclaw.Configuration/` — Config schemas, defaults, validation. The JSON schema is `src/Netclaw.Configuration/Schemas/netclaw-config.v1.schema.json` (webhook routes: `webhook-route.v1.schema.json` in the same folder) - `src/Netclaw.Daemon/` — Daemon architecture, startup, health checks - `src/Netclaw.Channels.Slack/` and `Netclaw.Channels.Discord/` — Channel integrations - `src/Netclaw.Security/` — Security model, ACLs, approval gates @@ -37,6 +37,7 @@ Before asking the user anything, collect available information: - `docs/adr/` — Architecture decision records (tool schemas, context discovery, progressive tool disclosure) - `docs/runbooks/` — Operational runbooks (background jobs, behavior debugging, daemon upgrades, memory health, subagents, tool approval gates) - `docs/spec/configuration.md` — Configuration specification + - **Warning:** `docs/spec/SPEC-004-cli-contract.md` and `docs/spec/configuration.md` are stale in places. They list commands and keys that were never implemented. When they disagree with the C# source, the JSON schema, or real `--help` output from the release container, the source, schema and `--help` win. - `CLAUDE.md`, `PROJECT_CONTEXT.md`, `CONTRIBUTING.md` — Project context and conventions 2. **Screenshots** — Check `screenshots/output/` for assets matching the page topic. Map them to the page. Screenshots are served at `/screenshots/output/`. 3. **Landing page** — `src/pages/index.astro` contains feature descriptions, architecture overview, and product positioning that inform doc content. @@ -144,9 +145,13 @@ netclaw -## Related Commands +## Related pages -- [`netclaw `](/cli//) — +- [`netclaw `](/cli//) - + +## Resources + +- []() - ``` #### Tutorial Template @@ -180,13 +185,21 @@ description: "" ... -## Verify It Works +## Verify it works -## Next Steps +## What's next + +- []() - + +## Related pages -- []() — +- []() - + +## Resources + +- []() - ``` #### Guide Template @@ -199,7 +212,7 @@ description: "" -## Before You Begin +## Before you begin - @@ -211,7 +224,15 @@ description: "" ### - +**Symptom:** ... **Cause:** ... **Fix:** ... + +## Related pages + +- []() - + +## Resources + +- []() - ``` #### Concept Template @@ -231,6 +252,14 @@ description: "" ## Limitations - + +## Related pages + +- []() - + +## Resources + +- []() - ``` ### Step 5: Screenshot Integration @@ -239,19 +268,20 @@ When including screenshots: - Use `![alt text](/screenshots/output/.png)` for static screenshots - Prefer `.png` over `.gif` unless the animation is essential to understanding -- Write a caption sentence below each screenshot explaining what it shows +- Put the instruction before the screenshot and a one-sentence caption after it that says something the alt text doesn't +- **Captures come from the release container** (`ghcr.io/netclaw-dev/netclaw:`), never a host-installed binary. A host build can be an unreleased prototype. No private hostnames in samples or screenshots. - If a screenshot shows a multi-step flow (like `init`), break it into individual step screenshots: `init-01-provider-list.png`, `init-02-security-posture.png`, etc. -Available screenshot sets: -- `init-*` — Full `netclaw init` wizard flow (10+ steps) -- `status.*` — System status display -- `doctor.*` — Health check diagnostics -- `stats.*` — Usage statistics -- `provider.*` / `provider-manager.png` — Provider management -- `model.*` / `model-manager.png` — Model management -- `mcp-tools.*` / `mcp-tools-*.png` — MCP tool server management -- `webhooks.*` / `webhooks-list.png` — Webhook configuration -- `reminder.*` / `reminder-list.png` — Reminder management +Available screenshot sets (in `screenshots/output/`): +- `init-*` — `netclaw init` wizard steps (provider, identity, security posture, health check, start-over) +- `config-*` — `netclaw config` dashboard and areas (channels, search, exposure, browser, telemetry, webhooks, skills, workspaces) +- `status.*`, `doctor.*`, `stats.*` — system status, health checks, usage statistics +- `provider.*` / `provider-manager.png`, `model.*` / `model-manager.png` — provider and model management +- `mcp-tools.*` / `mcp-tools-*.png` — MCP server management +- `approvals.*` / `approvals-*.png` — tool approvals list and TUI +- `webhooks.*` / `webhooks-list.png`, `reminder.*` / `reminder-list.png` +- `chat-*`, `single-shot-*` — interactive chat and single-shot CLI runs +- `slack-*`, `discord-*`, `mattermost-*` — third-party channel setup and conversations (kept as is) ### Step 6: Critique (3 parallel agents) @@ -280,7 +310,7 @@ After writing the draft, spawn three review agents in parallel. Pass each the fu ### Step 7: Humanizer Pass -Run `/humanizer` on the drafted page. This is mandatory — no page ships without it. The humanizer catches AI writing patterns that slip through manual review: over-explaining, passive voice, hedge words, repetitive structure, and generic filler that makes documentation feel soulless. +Run `humanizer:humanizer` (the newer, rewrite-oriented plugin skill, not the older `humanizer`) on the drafted page. It runs on prose paragraphs only, never on tables, code blocks or frontmatter. Where it disagrees with the `netclaw-docs` skill, the skill wins (for example, ignore suggestions to add first person or tangents on reference pages). This is mandatory — no page ships without it. The humanizer catches AI writing patterns that slip through manual review: over-explaining, passive voice, hedge words, repetitive structure, and generic filler that makes documentation feel soulless. ### Step 8: Revise @@ -304,6 +334,8 @@ Before declaring the page done: - [ ] At least 2 external links to relevant resources - [ ] No robotic phrasing or unnecessary hedging - [ ] `npm run build` passes with no errors +- [ ] Screenshots and command output come from the release container, not a host binary +- [ ] No private hostnames in samples or screenshots - [ ] **Every runnable example actually runs.** Any `docker run`, `docker compose`, or shell snippet a reader is told to execute MUST be run locally against the current release first and verified crash-free — the container reaches `healthy` and `docker logs` shows no fatal error / `Aborted (core dumped)`. Never ship a command you haven't executed. (This is how the stale `Daemon.Host=0.0.0.0` Docker example shipped a guaranteed startup crash.) ## Autonomous Doc Writing Mode @@ -316,13 +348,13 @@ When invoked with a page slug (e.g., `cli/status`), run the full pipeline withou 4. Run `npm run build` to verify 5. Stage and commit the single page with message: `docs: write
/` -The netclaw source repo is expected at `~/repositories/stannardlabs/netclaw/`. If not found, check common locations or skip source-dependent content. +The netclaw source repo is expected at `~/repositories/netclaw-dev/netclaw`. If not found, check common locations or skip source-dependent content. ## Build & Preview ```bash npm run dev -- --host 0.0.0.0 # Dev server with Tailscale access -npm run build # Production build (43 pages) +npm run build # Production build (165 pages built) ``` ## Project Structure diff --git a/astro.config.mjs b/astro.config.mjs index 1e01f94..4d29df7 100644 --- a/astro.config.mjs +++ b/astro.config.mjs @@ -4,8 +4,13 @@ import rehypeMermaid from 'rehype-mermaid'; export default defineConfig({ site: 'https://netclaw.dev', + // Mirrored in public/_redirects (301 on Cloudflare Pages). These also work in `astro preview`. + redirects: { + '/guides/custom-subagents': '/skills/custom-subagents/', + '/guides/connecting-slack': '/channels/slack/', + }, markdown: { - rehypePlugins: [[rehypeMermaid, { strategy: 'inline-svg', dark: true }]], + rehypePlugins: [[rehypeMermaid, { strategy: 'inline-svg', dark: true, mermaidConfig: { htmlLabels: false, flowchart: { htmlLabels: false, wrappingWidth: 280 } } }]], }, vite: { server: { @@ -41,18 +46,24 @@ export default defineConfig({ items: [ { label: 'Installation', slug: 'getting-started/installation' }, { label: 'Quickstart', slug: 'getting-started/quickstart' }, - { label: 'Your First Conversation', slug: 'getting-started/first-conversation' }, + { label: 'Your first conversation', slug: 'getting-started/first-conversation' }, + { label: 'Build your first assistant', slug: 'getting-started/first-assistant' }, ], }, { - label: 'Architecture & Philosophy', + label: 'Guides', items: [ - { label: 'Design Philosophy', slug: 'architecture/design-philosophy' }, - { label: 'Overview', slug: 'architecture/overview' }, - { label: 'Sessions & Input Model', slug: 'architecture/sessions' }, - { label: 'Context Structure', slug: 'architecture/context-structure' }, - { label: 'Memory Model', slug: 'architecture/memory-model' }, - { label: 'Security Model', slug: 'architecture/security-model' }, + { label: 'Reduce or turn off approval prompts', slug: 'guides/reduce-approval-prompts' }, + { label: 'Run unattended', slug: 'guides/unattended-runs' }, + { label: 'Connect an OAuth MCP server', slug: 'guides/oauth-mcp-servers' }, + { label: 'Run netclaw with Mattermost on Docker Compose', slug: 'guides/mattermost-docker-compose' }, + { label: 'Upgrade netclaw', slug: 'guides/upgrading' }, + { label: 'Back up, restore and move netclaw', slug: 'guides/backup-and-restore' }, + { label: 'Use the CLI from another machine', slug: 'guides/pairing-remote-devices' }, + { label: 'MCP tool permissions', slug: 'guides/mcp-tool-permissions' }, + { label: 'Scripting headless sessions', slug: 'guides/scripting-sessions' }, + { label: 'Images, audio and other file types', slug: 'guides/multimodal-files' }, + { label: 'Migrating model configuration', slug: 'guides/migrating-model-config' }, ], }, { @@ -65,42 +76,56 @@ export default defineConfig({ ], }, { - label: 'Skills', + label: 'Skills & Agents', items: [ { label: 'Overview', slug: 'skills/overview' }, - { label: 'Native Manifest & Sync', slug: 'skills/native-manifest' }, - { label: 'Bundling Resources with Skills', slug: 'skills/bundled-resources' }, - { label: 'Skill Server', slug: 'skills/skill-server' }, + { label: 'Native manifest & sync', slug: 'skills/native-manifest' }, + { label: 'Bundling resources with skills', slug: 'skills/bundled-resources' }, + { label: 'Custom subagents', slug: 'skills/custom-subagents' }, + { label: 'Skill feeds', slug: 'skills/skill-feeds' }, + { label: 'External skills', slug: 'skills/external-skills' }, + { label: 'Skill server', slug: 'skills/skill-server' }, { label: 'skillserver CLI', slug: 'skills/skillserver-cli' }, - { label: 'External Skills', slug: 'skills/external-skills' }, - { label: 'Skill Feeds', slug: 'skills/skill-feeds' }, + { label: 'Distributing subagents', slug: 'skills/distributing-subagents' }, + ], + }, + { + label: 'Configuration', + items: [ + { label: 'Managed providers', slug: 'configuration/managed-providers' }, + { label: 'Self-hosted providers', slug: 'configuration/self-hosted-providers' }, + { label: 'Models', slug: 'configuration/models' }, + { label: 'Memory and embeddings', slug: 'configuration/memory' }, + { label: 'Search providers', slug: 'configuration/search-providers' }, + { label: 'MCP servers', slug: 'configuration/mcp-servers' }, + { label: 'Webhooks', slug: 'configuration/webhooks' }, + { label: 'Reminders', slug: 'configuration/reminders' }, + { label: 'Data retention', slug: 'configuration/data-retention' }, + { label: 'Configuration reference', slug: 'configuration/reference' }, ], }, { label: 'Security', items: [ - { label: 'Security Model', slug: 'security/security-model' }, + { label: 'Security model', slug: 'security/security-model' }, + { label: 'How tool approvals work', slug: 'security/approvals' }, { label: 'Hardening', slug: 'security/hardening' }, - { label: 'Secrets Management', slug: 'security/secrets' }, + { label: 'Secrets management', slug: 'security/secrets' }, ], }, { - label: 'Configuration', + label: 'Deployment', items: [ - { label: 'Managed Providers', slug: 'configuration/managed-providers' }, - { label: 'Self-Hosted Providers', slug: 'configuration/self-hosted-providers' }, - { label: 'Models', slug: 'configuration/models' }, - { label: 'Search Providers', slug: 'configuration/search-providers' }, - { label: 'MCP Servers', slug: 'configuration/mcp-servers' }, - { label: 'Webhooks', slug: 'configuration/webhooks' }, - { label: 'Reminders', slug: 'configuration/reminders' }, + { label: 'Docker', slug: 'deployment/docker' }, + { label: 'systemd', slug: 'deployment/systemd' }, + { label: 'Exposure modes', slug: 'deployment/exposure-modes' }, ], }, { label: 'Notifications & Observability', items: [ - { label: 'Operational Alerts', slug: 'observability/operational-alerts' }, - { label: 'Health Checks', slug: 'observability/health-checks' }, + { label: 'Operational alerts', slug: 'observability/operational-alerts' }, + { label: 'Health checks', slug: 'observability/health-checks' }, { label: 'OpenTelemetry', slug: 'observability/opentelemetry' }, ], }, @@ -115,9 +140,12 @@ export default defineConfig({ { label: 'status', slug: 'cli/status' }, { label: 'doctor', slug: 'cli/doctor' }, { label: 'stats', slug: 'cli/stats' }, + { label: 'daemon', slug: 'cli/daemon' }, + { label: 'update', slug: 'cli/update' }, { label: 'provider', slug: 'cli/provider' }, { label: 'model', slug: 'cli/model' }, - { label: 'mcp-tools', slug: 'cli/mcp-tools' }, + { label: 'memory', slug: 'cli/memory' }, + { label: 'mcp', slug: 'cli/mcp-tools' }, { label: 'webhooks', slug: 'cli/webhooks' }, { label: 'reminder', slug: 'cli/reminder' }, { label: 'skill', slug: 'cli/skill' }, @@ -126,27 +154,30 @@ export default defineConfig({ ], }, { - label: 'Deployment', + label: 'Reference', items: [ - { label: 'Docker', slug: 'deployment/docker' }, - { label: 'systemd', slug: 'deployment/systemd' }, - { label: 'Exposure Modes', slug: 'deployment/exposure-modes' }, + { label: 'Slash commands', slug: 'reference/slash-commands' }, + { label: 'Built-in tools', slug: 'reference/tools' }, + { label: 'Data directory layout', slug: 'reference/data-directory' }, + { label: 'Glossary', slug: 'reference/glossary' }, ], }, { - label: 'Guides', + label: 'Architecture & Philosophy', items: [ - { label: 'MCP Tool Permissions', slug: 'guides/mcp-tool-permissions' }, - { label: 'Migrating Model Configuration', slug: 'guides/migrating-model-config' }, - { label: 'Custom Subagents', slug: 'guides/custom-subagents' }, - { label: 'Images, Audio, and Other File Types', slug: 'guides/multimodal-files' }, - { label: 'Pairing Remote Devices', slug: 'guides/pairing-remote-devices' }, - { label: 'Scripting Headless Sessions', slug: 'guides/scripting-sessions' }, + { label: 'Design philosophy', slug: 'architecture/design-philosophy' }, + { label: 'Overview', slug: 'architecture/overview' }, + { label: 'Sessions & input model', slug: 'architecture/sessions' }, + { label: 'Context structure', slug: 'architecture/context-structure' }, + { label: 'Memory model', slug: 'architecture/memory-model' }, + { label: 'Security model', slug: 'architecture/security-model' }, ], }, { label: 'Troubleshooting', items: [ + { label: 'Troubleshooting by symptom', slug: 'troubleshooting/common-problems' }, + { label: 'FAQ', slug: 'troubleshooting/faq' }, { label: 'llama.cpp', slug: 'troubleshooting/llama-cpp' }, { label: 'vLLM', slug: 'troubleshooting/vllm' }, ], @@ -158,6 +189,7 @@ export default defineConfig({ { label: 'vs. Hermes Agent', slug: 'comparisons/hermes-agent' }, ], }, + { label: 'Changelog', link: '/changelog' }, ], }), ], diff --git a/docker-compose.screenshots.yml b/docker-compose.screenshots.yml index e97cd0b..716532e 100644 --- a/docker-compose.screenshots.yml +++ b/docker-compose.screenshots.yml @@ -1,74 +1,66 @@ -services: - ollama: - image: ollama/ollama:latest - restart: unless-stopped - ports: - - "11434:11434" - volumes: - - screenshot-ollama-data:/root/.ollama +# Base of the docs capture / verification stack. Do not run this file directly: +# use screenshots/stack.sh or screenshots/capture.sh, which assemble it with the +# overlays under screenshots/compose/ and set the variables below. +# +# base netns + netclawd (no model config: first-run state) + capture +# screenshots/compose/configured.yml adds Models.Definitions/Roles, the demo MCP server +# screenshots/compose/ollama.yml adds the hermetic Ollama inference backend +# +# Every service shares ONE network namespace (the `netns` container) so the daemon, +# the CLI inside the capture container and any helper all reach each other on +# 127.0.0.1, exactly like a real user's machine. Nothing is published to the host. +# No container_name and no host ports: run as many projects side by side as you like +# (COMPOSE_PROJECT_NAME=dr04 ...). - ollama-init: +x-netclaw-image: &netclaw-image ${NETCLAW_IMAGE:-ghcr.io/netclaw-dev/netclaw:${NETCLAW_VERSION:?set by screenshots/stack.sh}} + +services: + netns: + # Holds the shared network namespace and nothing else. image: curlimages/curl:8.13.0 - entrypoint: ["/bin/sh", "/usr/local/bin/ollama-init.sh"] + entrypoint: ["sleep", "infinity"] + stop_grace_period: 1s + + netclawd: + image: *netclaw-image + network_mode: "service:netns" environment: - OLLAMA_MODEL: ${OLLAMA_MODEL:-qwen2:0.5b} - depends_on: - - ollama + NETCLAW_Daemon__Port: "5199" + NETCLAW_Daemon__DisableSelfUpdate: "true" volumes: - - ./screenshots/scripts/ollama-init.sh:/usr/local/bin/ollama-init.sh:ro - - mcp-demo: - build: ./screenshots/mcp-server - expose: - - "8080" + # The daemon runs as the non-root `netclaw` user; its home is /home/netclaw/.netclaw. + - netclaw-home:/home/netclaw/.netclaw healthcheck: - test: ["CMD", "python", "-c", "import socket; s=socket.create_connection(('127.0.0.1',8080),2); s.close()"] - interval: 5s + test: ["CMD", "curl", "-sf", "http://127.0.0.1:5199/api/health/ready"] + interval: 3s timeout: 5s - retries: 10 + retries: 40 start_period: 5s + stop_grace_period: 10s - netclawd: - # The published image is `netclaw` (not `netclawd`). Pinned to a release tag - # via NETCLAW_VERSION so screenshots track a known build; capture.sh sources - # NETCLAW_VERSION from src/version.json. Override NETCLAW_IMAGE to use :dev. - image: ${NETCLAW_IMAGE:-ghcr.io/netclaw-dev/netclaw:${NETCLAW_VERSION:-0.22.1}} - ports: - - "${NETCLAW_PORT:-5299}:5199" + # VHS + the netclaw CLI from the same release image as the daemon. Only started by + # `docker compose run` (profile `capture`); see screenshots/capture/Dockerfile. + capture: + profiles: ["capture"] + image: netclaw-docs-capture:${NETCLAW_IMAGE_TAG:?set by screenshots/stack.sh} + build: + context: ./screenshots/capture + args: + NETCLAW_IMAGE: *netclaw-image + network_mode: "service:netns" depends_on: - ollama-init: - condition: service_completed_successfully - mcp-demo: + netclawd: condition: service_healthy environment: - # NOTE (Phase 5): 0.22.1's `local` exposure mode binds loopback and aborts on a - # non-loopback Daemon.Host. The old host-reachability model here (Host=0.0.0.0 + - # published port + host-side `curl`/`netclaw` in capture.sh) crashes under 0.22.1. - # The Phase 5 re-shoot must drive the CLI via `docker exec` (loopback-local) or move - # to reverse-proxy mode + Daemon.TrustedProxies, and re-validate end to end. + HOST_UID: ${HOST_UID:-1000} + HOST_GID: ${HOST_GID:-1000} NETCLAW_Daemon__Port: "5199" - NETCLAW_Daemon__DisableSelfUpdate: "true" - NETCLAW_Providers__local-ollama__Type: ollama - NETCLAW_Providers__local-ollama__Endpoint: http://ollama:11434 - NETCLAW_Models__Main__Provider: local-ollama - NETCLAW_Models__Main__ModelId: ${OLLAMA_MODEL:-qwen2:0.5b} - NETCLAW_Models__Fallback__Provider: local-ollama - NETCLAW_Models__Fallback__ModelId: ${OLLAMA_MODEL:-qwen2:0.5b} - NETCLAW_Models__Compaction__Provider: local-ollama - NETCLAW_Models__Compaction__ModelId: ${OLLAMA_MODEL:-qwen2:0.5b} - NETCLAW_McpServers__demo-utilities__Transport: http - NETCLAW_McpServers__demo-utilities__Url: http://mcp-demo:8080/mcp - NETCLAW_McpServers__demo-utilities__Enabled: "true" + NETCLAW_CONTAINER_SUPERVISOR: entrypoint volumes: - # 0.22.1 runs as the non-root `netclaw` user; its home is /home/netclaw, not /root. - - screenshot-netclaw-home:/home/netclaw/.netclaw - healthcheck: - test: ["CMD", "curl", "-sf", "http://127.0.0.1:5199/api/health/ready"] - interval: 5s - timeout: 5s - retries: 30 - start_period: 10s + - netclaw-home:/home/netclaw/.netclaw + - ./screenshots/tapes:/tapes:ro + - ./screenshots/scripts:/work/screenshots/scripts:ro + - ${CAPTURE_OUT_DIR:-./screenshots/output}:/out volumes: - screenshot-ollama-data: - screenshot-netclaw-home: + netclaw-home: diff --git a/examples/mattermost-compose/.env.example b/examples/mattermost-compose/.env.example new file mode 100644 index 0000000..84bbcd5 --- /dev/null +++ b/examples/mattermost-compose/.env.example @@ -0,0 +1,14 @@ +# Copy to .env and fill in. Compose reads .env automatically. + +# Postgres password for the mattermost database user. +POSTGRES_PASSWORD=change-me-to-a-long-random-string + +# Where you open Mattermost in a browser. Use your real https URL later. +MATTERMOST_SITE_URL=http://localhost:8065 + +# Inference: any OpenAI-compatible endpoint (llama.cpp, vLLM, ...). The address works with or without a /v1 suffix. +# configure-netclaw.sh reads these. The compose file doesn't use them. +INFERENCE_BASE_URL=http://llm.example.internal:8080 +INFERENCE_MODEL_ID=your-model-id + +# provision.sh appends MM_CHANNEL_ID and MM_USER_ID here. Don't set them by hand. diff --git a/examples/mattermost-compose/.gitignore b/examples/mattermost-compose/.gitignore new file mode 100644 index 0000000..2cf01c2 --- /dev/null +++ b/examples/mattermost-compose/.gitignore @@ -0,0 +1,2 @@ +.env +.bot-token diff --git a/examples/mattermost-compose/README.md b/examples/mattermost-compose/README.md new file mode 100644 index 0000000..319442c --- /dev/null +++ b/examples/mattermost-compose/README.md @@ -0,0 +1,26 @@ +# netclaw + Mattermost on Docker Compose + +A private assistant stack: netclaw 0.28.2, Mattermost Team Edition 11.11.1 and +Postgres 17.11 on one Compose network. The full walkthrough, with expected +output at every step, is +[Run netclaw with Mattermost on Docker Compose](https://netclaw.dev/guides/mattermost-docker-compose/). + +| File | What it does | +|------|--------------| +| `compose.yaml` | The three services. netclaw publishes no ports and runs read-only with all capabilities dropped. | +| `.env.example` | Copy to `.env`. Passwords and the inference endpoint. | +| `provision.sh` | Creates the admin, a team, a private channel, one member and one bot with a token. Needs bash and curl. | +| `configure-netclaw.sh` | Adds the inference provider, sets the channel audience, stores the bot token in netclaw's vault and restarts. | + +```bash +cp .env.example .env && chmod 600 .env # edit POSTGRES_PASSWORD, INFERENCE_BASE_URL, INFERENCE_MODEL_ID +docker compose up -d --wait +./provision.sh # prints the generated passwords once +./configure-netclaw.sh +``` + +Then sign in at as the member user and mention the bot +in the `assistant` channel. + +Everything lives in named volumes. `docker compose down` keeps your data. +`docker compose down -v` deletes it. diff --git a/examples/mattermost-compose/compose.yaml b/examples/mattermost-compose/compose.yaml new file mode 100644 index 0000000..c9a3e90 --- /dev/null +++ b/examples/mattermost-compose/compose.yaml @@ -0,0 +1,99 @@ +# netclaw + Mattermost (Team Edition) + Postgres on one private network. +# Only Mattermost's web UI is published, and only on the host's loopback. + +services: + postgres: + image: postgres:17.11-alpine + restart: unless-stopped + environment: + POSTGRES_DB: mattermost + POSTGRES_USER: mattermost + POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:?Set POSTGRES_PASSWORD in .env} + volumes: + - postgres-data:/var/lib/postgresql/data + healthcheck: + test: ["CMD-SHELL", "pg_isready -U mattermost -d mattermost"] + interval: 5s + timeout: 5s + retries: 12 + networks: [chat] + + mattermost: + image: mattermost/mattermost-team-edition:11.11.1 + restart: unless-stopped + depends_on: + postgres: + condition: service_healthy + environment: + MM_SQLSETTINGS_DRIVERNAME: postgres + MM_SQLSETTINGS_DATASOURCE: postgres://mattermost:${POSTGRES_PASSWORD}@postgres:5432/mattermost?sslmode=disable&connect_timeout=10 + MM_SERVICESETTINGS_SITEURL: ${MATTERMOST_SITE_URL:-http://localhost:8065} + # Bots and personal access tokens are off by default. netclaw needs both. + MM_SERVICESETTINGS_ENABLEBOTACCOUNTCREATION: "true" + MM_SERVICESETTINGS_ENABLEUSERACCESSTOKENS: "true" + # Nobody can self-register. Accounts come from provision.sh or an admin. + MM_TEAMSETTINGS_ENABLEOPENSERVER: "false" + # Local mode: lets the healthcheck use mmctl over a container-internal socket. + MM_SERVICESETTINGS_ENABLELOCALMODE: "true" + ports: + - "127.0.0.1:8065:8065" # host loopback only + volumes: + - mattermost-config:/mattermost/config + - mattermost-data:/mattermost/data + - mattermost-plugins:/mattermost/plugins + security_opt: + - no-new-privileges:true + healthcheck: + # The image has no shell or curl, so use mmctl. + test: ["CMD", "/mattermost/bin/mmctl", "--local", "system", "status"] + interval: 10s + timeout: 5s + retries: 18 + start_period: 30s + networks: [chat] + + netclaw: + image: ghcr.io/netclaw-dev/netclaw:0.27.1 + restart: unless-stopped + depends_on: + mattermost: + condition: service_healthy + # No ports: the daemon stays in the default `local` exposure mode (loopback). + # You drive it with `docker compose exec`, and Mattermost never calls it. + environment: + # The daemon is a single-file .NET binary that unpacks itself on start. + # The root filesystem is read-only, so unpack into a small exec tmpfs. + DOTNET_BUNDLE_EXTRACT_BASE_DIR: /var/netclaw-bundle + # Mattermost. The bot token is NOT here. It goes in netclaw's vault. + NETCLAW_Mattermost__Enabled: "true" + NETCLAW_Mattermost__ServerUrl: http://mattermost:8065 + NETCLAW_Mattermost__MentionOnly: "true" + NETCLAW_Mattermost__AllowDirectMessages: "true" + # Allow-lists live here and only here. Env entries merge with netclaw.json + # lists by position, so never set the same list in both places. + # Blank until provision.sh writes the IDs to .env. + NETCLAW_Mattermost__AllowedChannelIds__0: ${MM_CHANNEL_ID:-} + NETCLAW_Mattermost__AllowedUserIds__0: ${MM_USER_ID:-} + volumes: + - netclaw-home:/home/netclaw/.netclaw + # Hardening + user: "1654:1654" # start as the netclaw user, so the entrypoint never needs root + read_only: true # root filesystem is immutable + tmpfs: + - /tmp:rw,noexec,nosuid,size=64m # scratch space only + - /var/netclaw-bundle:rw,exec,nosuid,size=128m,uid=1654,gid=1654 # the unpacked daemon (43 MB), rebuilt on every start + cap_drop: [ALL] # no Linux capabilities + security_opt: + - no-new-privileges:true # no setuid escalation + pids_limit: 256 # fork-bomb ceiling + networks: [chat] + +volumes: + postgres-data: + mattermost-config: + mattermost-data: + mattermost-plugins: + netclaw-home: + +networks: + chat: diff --git a/examples/mattermost-compose/configure-netclaw.sh b/examples/mattermost-compose/configure-netclaw.sh new file mode 100755 index 0000000..b2b28b2 --- /dev/null +++ b/examples/mattermost-compose/configure-netclaw.sh @@ -0,0 +1,40 @@ +#!/usr/bin/env bash +# Finish wiring netclaw after provision.sh: apply the IDs from .env, add the +# inference provider, set the channel audience, and store the bot token in +# netclaw's encrypted vault. Run it once. +set -euo pipefail +cd "$(dirname "$0")" + +[ -f .bot-token ] || { echo "Run ./provision.sh first." >&2; exit 1; } +set -a; . ./.env; set +a +: "${MM_CHANNEL_ID:?missing from .env}" "${INFERENCE_BASE_URL:?missing from .env}" "${INFERENCE_MODEL_ID:?missing from .env}" + +nc() { docker compose exec -T netclaw netclaw "$@"; } + +# Recreate netclaw so it picks up MM_CHANNEL_ID and MM_USER_ID from .env. +docker compose up -d --wait netclaw + +nc provider add main openai-compatible --endpoint "$INFERENCE_BASE_URL" +nc model set main main "$INFERENCE_MODEL_ID" + +# Personal posture (what `netclaw init` writes for one operator) and a personal +# audience for the private channel, so the agent can ask to run shell commands. +# Each command still needs your approval. Editing netclaw.json makes the daemon +# restart itself. +docker compose exec -T -e CH="$MM_CHANNEL_ID" netclaw sh -c ' + f=$HOME/.netclaw/config/netclaw.json + jq --arg c "$CH" " + .Security.DeploymentPosture = \"Personal\" + | .Security.ShellExecutionMode = \"HostAllowed\" + | .Security.StrictDefaults = true + | .Tools.ShellMode = \"HostAllowed\" + | .Mattermost.ChannelAudiences[\$c] = \"personal\"" "$f" > /tmp/netclaw.json + cat /tmp/netclaw.json > "$f"; rm -f /tmp/netclaw.json' + +# Token goes in over stdin, so it never appears in a process list or in .env. +docker compose exec -T netclaw sh -c 'netclaw secrets set Mattermost.BotToken "$(cat)"' < .bot-token + +# Secrets changes need a restart. +docker compose restart netclaw +docker compose up -d --wait netclaw +nc status diff --git a/examples/mattermost-compose/provision.sh b/examples/mattermost-compose/provision.sh new file mode 100755 index 0000000..fffb176 --- /dev/null +++ b/examples/mattermost-compose/provision.sh @@ -0,0 +1,105 @@ +#!/usr/bin/env bash +# Provision a fresh Mattermost: admin, team, private channel, one human user, +# one bot account with an access token. Needs only bash and curl. +# +# ./provision.sh # Mattermost at http://localhost:8065 +# MM_URL=https://chat.example.com ./provision.sh +# +# Writes MM_CHANNEL_ID and MM_USER_ID to .env (compose reads them), the bot +# token to .bot-token (mode 600), and prints the generated passwords once. +# Passwords, the admin session token and request bodies reach curl over stdin +# or a private temp file, never on its command line, so `ps` can't show them. +set -euo pipefail +umask 077 # every file this script creates is readable by you only +cd "$(dirname "$0")" + +MM_URL="${MM_URL:-http://localhost:8065}" +TEAM_NAME="${TEAM_NAME:-home}" +CHANNEL_NAME="${CHANNEL_NAME:-assistant}" +BOT_NAME="${BOT_NAME:-netclaw}" +ADMIN_USER="${ADMIN_USER:-admin}" +ADMIN_EMAIL="${ADMIN_EMAIL:-admin@example.com}" +HUMAN_USER="${HUMAN_USER:-alice}" +HUMAN_EMAIL="${HUMAN_EMAIL:-alice@example.com}" + +# The values are pasted into JSON below, so keep them to plain characters. +for v in TEAM_NAME CHANNEL_NAME BOT_NAME ADMIN_USER HUMAN_USER; do + [[ "${!v}" =~ ^[a-z0-9_-]+$ ]] || { echo "$v must be lowercase letters, digits, - or _." >&2; exit 1; } +done +for v in ADMIN_EMAIL HUMAN_EMAIL; do + [[ "${!v}" =~ ^[A-Za-z0-9._+-]+@[A-Za-z0-9.-]+$ ]] || { echo "$v doesn't look like an email address." >&2; exit 1; } +done + +[ -e .bot-token ] && { echo "Refusing to run twice: .bot-token already exists." >&2; exit 1; } + +rand() { od -An -N12 -tx1 /dev/urandom | tr -d ' \n'; } +# First string value for a key in compact JSON. Mattermost returns compact JSON. +field() { sed -n "s/.*\"$1\":\"\([^\"]*\)\".*/\1/p" | head -n1; } +# Stop if an ID came back empty, so a changed response shape fails loudly. +need() { [ -n "$2" ] || { echo "Mattermost returned no $1. Run 'docker compose down -v' and start over." >&2; exit 1; }; } + +ADMIN_PASS="$(rand)" +HUMAN_PASS="$(rand)" + +# The admin token lives in this private file and curl reads it with -H @file. +HDR="$(mktemp)" +trap 'rm -f "$HDR"' EXIT + +api() { # api METHOD PATH [JSON] (sends the admin token once login has run) + local args=(-sS --fail-with-body -X "$1" "$MM_URL/api/v4$2" -H 'Content-Type: application/json') + [ ! -s "$HDR" ] || args+=(-H "@$HDR") + if [ -n "${3:-}" ]; then + printf '%s' "$3" | curl "${args[@]}" --data-binary @- + else + curl "${args[@]}" + fi +} + +echo "Waiting for Mattermost at $MM_URL ..." +until curl -sf "$MM_URL/api/v4/system/ping" >/dev/null; do sleep 2; done + +# The first account on a fresh server becomes the system admin. +api POST /users "{\"email\":\"$ADMIN_EMAIL\",\"username\":\"$ADMIN_USER\",\"password\":\"$ADMIN_PASS\"}" >/dev/null + +# Log in. The session token comes back in the Token response header. +TOKEN="$(printf '{"login_id":"%s","password":"%s"}' "$ADMIN_USER" "$ADMIN_PASS" \ + | curl -sS -D - -o /dev/null -X POST "$MM_URL/api/v4/users/login" \ + -H 'Content-Type: application/json' --data-binary @- \ + | tr -d '\r' | sed -n 's/^[Tt]oken: //p')" +[ -n "$TOKEN" ] || { echo "Login failed." >&2; exit 1; } +printf 'Authorization: Bearer %s\n' "$TOKEN" > "$HDR" + +TEAM_ID="$(api POST /teams "{\"name\":\"$TEAM_NAME\",\"display_name\":\"Home\",\"type\":\"I\"}" | field id)" +need team "$TEAM_ID" +CHANNEL_ID="$(api POST /channels "{\"team_id\":\"$TEAM_ID\",\"name\":\"$CHANNEL_NAME\",\"display_name\":\"Assistant\",\"type\":\"P\"}" | field id)" +need channel "$CHANNEL_ID" + +HUMAN_ID="$(api POST /users "{\"email\":\"$HUMAN_EMAIL\",\"username\":\"$HUMAN_USER\",\"password\":\"$HUMAN_PASS\"}" | field id)" +need user "$HUMAN_ID" +api POST "/teams/$TEAM_ID/members" "{\"team_id\":\"$TEAM_ID\",\"user_id\":\"$HUMAN_ID\"}" >/dev/null +api POST "/channels/$CHANNEL_ID/members" "{\"user_id\":\"$HUMAN_ID\"}" >/dev/null + +BOT_ID="$(api POST /bots "{\"username\":\"$BOT_NAME\",\"display_name\":\"netclaw\"}" | field user_id)" +need bot "$BOT_ID" +api POST "/teams/$TEAM_ID/members" "{\"team_id\":\"$TEAM_ID\",\"user_id\":\"$BOT_ID\"}" >/dev/null +api POST "/channels/$CHANNEL_ID/members" "{\"user_id\":\"$BOT_ID\"}" >/dev/null +BOT_TOKEN="$(api POST "/users/$BOT_ID/tokens" '{"description":"netclaw"}' | field token)" +need "bot token" "$BOT_TOKEN" + +printf '%s' "$BOT_TOKEN" > .bot-token +[ ! -s .env ] || [ -z "$(tail -c1 .env)" ] || echo >> .env # ensure a trailing newline +{ + echo "MM_CHANNEL_ID=$CHANNEL_ID" + echo "MM_USER_ID=$HUMAN_ID" +} >> .env + +cat < - - - - - - - - - diff --git a/public/assets/netclaw-horizontal-dark.svg b/public/assets/netclaw-horizontal-dark.svg deleted file mode 100644 index c48c75e..0000000 --- a/public/assets/netclaw-horizontal-dark.svg +++ /dev/null @@ -1,26 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - \ No newline at end of file diff --git a/public/assets/netclaw-stacked-purple.svg b/public/assets/netclaw-stacked-purple.svg deleted file mode 100644 index e189634..0000000 --- a/public/assets/netclaw-stacked-purple.svg +++ /dev/null @@ -1,26 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - \ No newline at end of file diff --git a/screenshots/.env.example b/screenshots/.env.example new file mode 100644 index 0000000..32a2313 --- /dev/null +++ b/screenshots/.env.example @@ -0,0 +1,12 @@ +# Copy to screenshots/.env (gitignored). Never commit a real hostname or model name. +# +# Real inference without leaking the endpoint (relay mode): set the real https:// base URL and +# the model id it serves. The forwarder (compose/forwarder.yml) presents it inside the stack as +# http://llm.example.internal:8081 with the model id demo-model. +# INFERENCE_UPSTREAM=https://inference.internal.example +# INFERENCE_UPSTREAM_MODEL=the-real-model-id +# +# Or point netclaw straight at an endpoint (external mode; it ends up in netclaw.json, so don't +# capture provider lists this way). Without either, the stack uses a tiny local Ollama model. +# INFERENCE_BASE_URL=http://llm.example.internal:8080 +# INFERENCE_MODEL_ID=your-model-id diff --git a/screenshots/.gitignore b/screenshots/.gitignore new file mode 100644 index 0000000..fcdbee0 --- /dev/null +++ b/screenshots/.gitignore @@ -0,0 +1,3 @@ +.env +.netclaw-screenshot-home/ +mattermost/.bootstrap*.json diff --git a/screenshots/README.md b/screenshots/README.md index 1a2379f..b3bbe5c 100644 --- a/screenshots/README.md +++ b/screenshots/README.md @@ -1,56 +1,169 @@ -# Screenshot capture pipeline +# Screenshot and verification stack -Source of truth for every image under `output/` (served at `/screenshots/output/`). +Everything that produces an image under `output/` (served at `/screenshots/output/`) or proves a command works runs in containers against the **released** Netclaw image. Docker is the only prerequisite. + +**Rule: never use a host `netclaw` binary.** Whatever is on your `PATH` may be a prototype or a dev build. The CLI in every recording and every verification command comes from `ghcr.io/netclaw-dev/netclaw:`. ## Layout | Path | What it is | |------|-----------| -| `capture.sh` | Orchestrator: brings up the Docker stack, runs every VHS tape, tears down | -| `tapes/*.tape` | [VHS](https://github.com/charmbracelet/vhs) recordings — one per CLI command/flow | -| `scripts/` | Helpers (`ollama-init.sh` model pull, `seed-approvals.sh` fixture seeding) | -| `mcp-server/` | Demo MCP server used by the `mcp-tools` capture | -| `output/` | Generated PNGs/GIFs (committed; consumed by the docs) | +| `stack.sh` | Bring up, reset, exec into and tear down a disposable daemon (verification, no recording) | +| `capture.sh` | Record VHS tapes inside the capture container against that daemon | +| `shoot-page.sh` | Screenshot a page of the locally built docs site in the Playwright container | +| `mattermost/run.sh` | Mattermost + daemon + Playwright, all in containers; regenerates `mattermost-*.png` | +| `tapes/*.tape` | [VHS](https://github.com/charmbracelet/vhs) recordings, one per CLI command or flow | +| `capture/` | Capture image: VHS (ttyd, ffmpeg, Chromium) plus the released CLI | +| `playwright/` | Playwright image matching the version locked in `package-lock.json` | +| `compose/` | Compose overlays (`configured.yml`, `ollama.yml`, `forwarder.yml`, `projects.yml`, `skillserver.yml`, `docker-host.yml`); the base is `../docker-compose.screenshots.yml` | +| `scripts/`, `mcp-server/` | Ollama model pull, the seed scripts tapes run out of sight (`seed-*.sh`), a Mattermost API fixture, the third-party redaction script, the demo MCP server | +| `lib.sh`, `.env.example` | Shared settings; copy `.env.example` to `.env` (gitignored) for real inference | + +## Pick the release + +The image tag comes from one place: `documentedVersion` in `src/version.json`. Override per run: + +```bash +NETCLAW_VERSION=0.27.1 ./screenshots/stack.sh up # a specific release +NETCLAW_IMAGE=ghcr.io/netclaw-dev/netclaw:dev ./screenshots/stack.sh up # any image reference +``` + +`src/version.json` may lag the release you're documenting, so pass `NETCLAW_VERSION=0.27.1` until it's bumped. Every command prints the image it used. + +## Verify a command against a clean daemon + +```bash +export COMPOSE_PROJECT_NAME=dr04 # your own name, see "Run several at once" +./screenshots/stack.sh up # clean ~/.netclaw, daemon healthy, provider + model + MCP configured +./screenshots/stack.sh exec netclaw doctor # runs as the netclaw user, like the daemon +./screenshots/stack.sh exec netclaw status +./screenshots/stack.sh daemon-log 100 # the daemon's log file (container stdout is empty) +./screenshots/stack.sh down # removes containers, network and volumes +``` + +Daemon states: + +| State | Command | What you get | +|-------|---------|--------------| +| configured (default) | `stack.sh up` | Provider `inference`, model roles and the demo MCP server `demo-utilities`, added with `netclaw provider add`, `netclaw model set` and `netclaw mcp add`, plus the Security section `netclaw init` writes. This is what a user's `netclaw.json` looks like after init. | +| first run | `stack.sh up --fresh` | Healthy daemon, empty `~/.netclaw`, no model. `netclaw status` says "daemon not configured". Use it for `netclaw init`. | +| bare | `stack.sh up --bare` | Like first run, plus the demo MCP server container. Tapes that seed their own config (`scripts/seed-*.sh`) use it. | + +`stack.sh reset [--fresh]` wipes `~/.netclaw` and restarts the daemon without rebuilding the rest of the stack. Run `./screenshots/stack.sh` with no arguments for the full command list. + +The configured state writes `netclaw.json` instead of setting `NETCLAW_*` env vars because the CLI refuses to run most commands without that file, and the `provider`, `model` and `mcp` TUIs edit it. The 0.25+ env shape still works if you need it for a throwaway daemon: + +```bash +docker run -d --name x -e NETCLAW_Providers__inference__Type=ollama \ + -e NETCLAW_Providers__inference__Endpoint=http://ollama:11434 \ + -e NETCLAW_Models__Definitions__main__Provider=inference \ + -e NETCLAW_Models__Definitions__main__ModelId=qwen2:0.5b \ + -e NETCLAW_Models__Roles__Main=main ghcr.io/netclaw-dev/netclaw:0.27.1 +``` + +The Mattermost flow below uses exactly this env shape. + +## Inference + +| Mode | When | How | +|------|------|-----| +| hermetic (default) | Shots that don't need a real answer | Ollama with `qwen2:0.5b` in the stack. The first `up` of a project pulls the model (about 350 MB), so allow a minute or two. | +| relay | You need real answers in a screenshot | Set `INFERENCE_UPSTREAM` and `INFERENCE_UPSTREAM_MODEL` in `screenshots/.env` (see below). The only mode that keeps the real endpoint and model name out of the config and the frame. | +| external | You need a real reply and nothing is captured that prints the endpoint | Set `INFERENCE_BASE_URL` (an OpenAI-compatible endpoint) and `INFERENCE_MODEL_ID` in the environment or in `screenshots/.env` | + +```bash +INFERENCE_BASE_URL=http://llm.example.internal:8080 INFERENCE_MODEL_ID=my-model \ + ./screenshots/stack.sh up +``` + +External mode writes the endpoint into the daemon's `netclaw.json`, so commands like `netclaw provider list` print it. Capture those screenshots in hermetic mode. + +### Real inference without leaking the endpoint + +Set `INFERENCE_UPSTREAM` (the real `https://` base URL) and `INFERENCE_UPSTREAM_MODEL` (the model id it serves) in the gitignored `screenshots/.env`. `lib.sh` then switches to relay mode and adds `compose/forwarder.yml`: an nginx container in the stack's network namespace relays port 8081 to the upstream and rewrites the upstream's model id to `demo-model` in every response, and `llm.example.internal` resolves to loopback for everything in the namespace. netclaw only ever sees `http://llm.example.internal:8081` and the model `demo-model`, so provider lists, `netclaw init` and chat status bars are safe to capture, and no real model file name reaches a status bar. Check the first frame of every chat tape anyway. + +This is the only relay, and both tape families use it. The seed scripts (`seed-assistant.sh`, `seed-providers.sh`) point their `lab-llama` provider at it, and the configured daemon (`stack.sh up`) registers it as provider `inference`. Ollama stays up alongside it, so `netclaw init` and the provider and model tapes still have a reachable local provider. Without `INFERENCE_UPSTREAM`, tapes that need a real answer (`stats`, `single-shot-cli-execution`, the chat and approval tapes) can't produce one: the 0.5B Ollama model can't call tools. + +## Standard tape settings + +Every terminal image is shown in a docs column about 650 px wide, so a capture wider than about 1000 px shrinks its text below what a reader can read without the lightbox. Use this block, and change only `Height` (crop by terminal height, not by editing the PNG): + +``` +Set Shell "bash" +Set Theme "Dracula" +Set FontSize 16 +Set Width 1000 +Set Height 540 +Set Padding 20 +Set TypingSpeed 40ms +``` -The daemon image and the demo Ollama model come from `../docker-compose.screenshots.yml`. +At `FontSize 16` a character cell is about 10 px wide, so 1000 px holds about 93 columns, and scaled into the column the text lands at about 11 px. Widen a tape only as far as a table needs it (`model.tape` uses 1200 for the long OpenAI-compatible provider label, `doctor.tape` 1700 for its longest check line) and make the page caption say the image can be enlarged. -## Regenerate everything (VHS / TUI shots) +The status-like tapes (`status`, `doctor`, `stats`, `sessions`, `model`, `provider`, `skill-list`, `skill-sync`, `webhooks`, `reminder-list`, `config`, `single-shot-cli-execution`) all start from `scripts/seed-demo-install.sh`, so the images agree on providers, roles, skills, webhook routes, reminders and the MCP server. Add new status-like captures to that family instead of seeding a separate install. + +## Record tapes ```bash -./screenshots/capture.sh +./screenshots/capture.sh status # one tape +./screenshots/capture.sh status doctor # a few +./screenshots/capture.sh # all tapes +OUT_DIR=/tmp/shots ./screenshots/capture.sh status # write elsewhere (default: screenshots/output/) ``` -The daemon image is **pinned to the documented release**: `capture.sh` reads -`src/version.json` and pins `ghcr.io/netclaw-dev/netclaw:`. To -capture against a different build, override before running: +`capture.sh` starts a stack, resets `~/.netclaw` before every tape, runs each tape inside the capture container, copies the results to the output directory owned by you, and always tears the stack down. It exits non-zero if VHS fails or a tape doesn't produce every `Output` and `Screenshot` file it declares. + +How the recording reaches the daemon: the capture container shares the daemon container's network namespace and `~/.netclaw` volume, so `netclaw status` typed in a tape talks to `127.0.0.1:5199` exactly like a real user's CLI. Nothing is published to the host, and no `docker exec` shows up in a recording. + +Tape conventions: + +- Paths are relative to `/work`, so `Output screenshots/output/status.gif` and `Screenshot screenshots/output/status.png` land in the output directory. +- VHS starts bash with a clean `> ` prompt. No hidden preamble is needed unless a tape has to stage state first (see `chat-approval.tape`). +- Put `# stack: fresh` in a tape to run it against the first-run daemon (see `init-wizard.tape`), or `# stack: bare` for the same empty home with the demo MCP server up. +- Setup that isn't the subject of the image runs between `Hide` and `Show`: `bash /work/screenshots/scripts/seed-*.sh` for config, or `Source "/tapes/include/init-personal.tape"` to run the real `netclaw init` wizard (Personal, Ollama). Wait for a seed to finish with `Wait+Screen@4m /seeded: .../` instead of a fixed sleep. +- `# image: 0.27.0` runs one tape against another release (only `update-check.tape` does, to be honestly one version behind). `# host-docker: true` gives a tape the host's Docker socket (only `docker-health.tape`, which filters everything it prints to its own container). +- Only `.png` files are published from a run. A tape still needs an `Output` line for VHS, and its GIF stays in the container. +- VHS applies `Set Width`/`Set Height` once, so size each tape for its tallest screen. `Screenshot` is slightly asynchronous: leave a `Sleep` after every one, and don't issue two in a row. +- Shared setup goes in `tapes/include/*.tape` and is pulled in with `Source "/tapes/include/.tape"`. `capture.sh` only runs top-level tapes. Hide setup with `Hide`/`Show`, and wait for a marker the typed command doesn't contain (`echo SE''EDED`, then `Wait+Screen /SEEDED/`). +- Tapes that need a Git workspace run `scripts/seed-projects.sh` first. It creates `/home/netclaw/projects/website` on a volume the daemon shares (`compose/projects.yml`) and declares it as the workspace root, so shell approval prompts offer "This repository". Skill captures use `scripts/seed-skills.sh`, which publishes demo skills to the stack's skill server (`compose/skillserver.yml`, port 8090) with the `skillserver` CLI baked into the capture image. +- Tapes that drive a real model (`chat-*`, `approvals-*`, `reminder-run`) must make each prompt self-limiting ("then stop, run nothing else") and wait on text the prompt didn't type, such as a tool result line. A model that wanders into an extra tool call leaves a prompt open and the wait times out. +- A trailing `Screenshot` is lost by VHS if nothing follows it, so `capture.sh` appends `Sleep 1s` to the copy it runs. You don't have to. +- In 0.27.1, `Escape` doesn't quit a TUI. Use `Ctrl+Q`. +- The init wizard prints "Step N of 5" until you pick a posture, then "of 4" for Personal. `init-wizard.tape` (Personal) and `init-team.tape` (Team, the fifth step) record that as shown. + +## Mattermost and other browser captures ```bash -NETCLAW_VERSION=0.22.1 ./screenshots/capture.sh # a specific release -NETCLAW_IMAGE=ghcr.io/netclaw-dev/netclaw:dev ./screenshots/capture.sh # bleeding edge -OLLAMA_MODEL=qwen2:0.5b ./screenshots/capture.sh # demo inference model +INFERENCE_BASE_URL=http://llm.example.internal:8080 INFERENCE_MODEL_ID=my-model \ + OUT_DIR=/tmp/shots ./screenshots/mattermost/run.sh ``` -Keep `src/version.json` in step with the release you profiled — the -`docs_version_check` workflow opens an issue when the docs fall behind. +Mattermost, the daemon and Playwright share one compose network, and the daemon reaches Mattermost at `http://mattermost:8065`. The flow is: bootstrap Mattermost (`bootstrap.mjs`), start the daemon, store the bot token with `netclaw secrets set Mattermost.BotToken` (no token in env or compose), enable the channel, post an `@netclaw` message as the test user (`post-message.mjs`) and wait for the reply, then capture the System Console, bot account and thread (`capture.mjs`). `run.sh` sets `Mattermost.CallbackUrl` (through a small forwarder sidecar, because the daemon listens on loopback only) and allow-lists the daemon host in Mattermost (`MM_SERVICESETTINGS_ALLOWEDUNTRUSTEDINTERNALCONNECTIONS`), so tool approvals render as buttons. `SCRIPTED_APPROVAL=1 ./screenshots/mattermost/run.sh` needs no model: `scripted-llm.mjs` answers with one `demo-utilities/http_fetch` MCP call (the demo MCP server, gated to `Approval` for the Personal audience; `SCRIPTED_TOOL=shell` requests a `shell_execute` call instead), `approval.mjs` captures the prompt and `frame-approval.mjs` frames it as `approval-prompt.png` (rounded dark card, purple glow, 2x). The normal flow needs external inference (a 0.5B model won't produce a presentable answer). The bootstrap file with the short-lived bot token is deleted on exit. -## Mattermost capture (web setup + live conversation) +The Compose-sample screenshots (`mattermost-compose-*.png`, plus the recaptured `mattermost-bot-accounts.png`, `mattermost-create-bot.png`, `mattermost-conversation.png` and the landing page's `public/assets/approval-prompt.png`) come from `mattermost/capture-compose.mjs`, which signs in to the stack in `examples/mattermost-compose/` after `provision.sh` and `configure-netclaw.sh` have run. It needs the admin and `alice` passwords that `provision.sh` printed, and a Git repository for the approval prompt (so "This repository" appears). Token IDs are overwritten in the page before each shot. The Slack and Discord redactions are `scripts/redact-third-party.py`. -The Mattermost shots aren't a VHS tape — they're browser screenshots driven by -Playwright. Everything lives in `mattermost/`: +Screenshot a page of the docs site you built locally: -| File | Role | -|------|------| -| `docker-compose.mattermost.yml` | the Mattermost (`mattermost-preview`) container | -| `bootstrap.mjs` | seeds admin / team / bot + token / channel / test user via the REST API → `.bootstrap.json` | -| `run.sh` | orchestrator: compose up → bootstrap → start the pinned netclaw image (inference pointed at an OpenAI-compatible endpoint) → post an `@netclaw` message → run `capture.mjs` → tear down | -| `capture.mjs` | Playwright: the System Console bot-accounts setting, the bot + its token, and the live `@netclaw` thread | +```bash +npm run build +./screenshots/shoot-page.sh cli/status # docs-logs/screenshots/cli-status.png +``` + +`shoot-page.sh` runs `scripts/screenshot-page.mjs` in the same Playwright image and serves `dist/` itself. To shoot a dev server instead, start it with `npm run dev -- --host 0.0.0.0` and set `SCREENSHOT_BASE_URL=http://host.docker.internal:4321`. + +## Run several at once -Regenerate them with: +Every container, network and volume is namespaced by `COMPOSE_PROJECT_NAME`. There are no fixed container names and no published ports, so two agents can run side by side: ```bash -./screenshots/mattermost/run.sh +COMPOSE_PROJECT_NAME=dr04 ./screenshots/stack.sh up +COMPOSE_PROJECT_NAME=dr07 ./screenshots/capture.sh status ``` -Point inference at your own endpoint/model with `SPARK_ENDPOINT` and `SPARK_MODEL_ID` -(the defaults match the local netclaw instance). `.bootstrap.json` holds a short-lived -bot token and is gitignored. +Use your own name every time, and `down` when you're done. The Mattermost flow appends `-mm` to the name, so it doesn't collide with a stack of the same name. Only built images are shared across projects (`netclaw-docs-capture:`, `netclaw-docs-playwright:`, `netclaw-docs-mcp-demo:1`), which is safe because they're deterministic. + +## Design notes + +- **Why the capture image starts from VHS.** The release image is `ubuntu:24.04` (glibc) and the CLI is a self-contained .NET single-file binary. The VHS image is Debian trixie (glibc 2.41) with ICU and libsodium already installed, so the CLI and its self-dropping launcher are copied out of the release image unmodified. The recorded CLI is byte-for-byte the released one. Putting VHS into the release image would have meant installing Chromium, ttyd and ffmpeg on an Ubuntu base that ships Chromium only as a snap. +- **One network namespace.** A tiny `netns` container owns the stack's network namespace, and the daemon, capture container, Ollama and demo MCP server join it. Everything talks over `127.0.0.1`, the daemon stays in its default loopback-only `local` exposure mode, and the `netclaw init` wizard's default Ollama endpoint (`http://localhost:11434`) just works. +- **Ownership.** The capture container runs the tape as the `netclaw` user (uid 1654, same as the daemon) so it can use the shared home, then copies the files out with your uid and gid. diff --git a/screenshots/capture.sh b/screenshots/capture.sh index 91f7238..af3f216 100755 --- a/screenshots/capture.sh +++ b/screenshots/capture.sh @@ -1,58 +1,73 @@ #!/bin/bash -# Orchestrates the screenshot capture pipeline. -# Brings up Docker Compose environment, runs VHS tapes, tears down. -set -eu +# Record VHS tapes inside the capture container against a disposable released daemon. +# Docker is the only prerequisite; the host netclaw is never used. +# +# ./screenshots/capture.sh run every tape +# ./screenshots/capture.sh status run one tape (screenshots/tapes/status.tape) +# ./screenshots/capture.sh status doctor +# +# Output goes to screenshots/output/ (override: OUT_DIR=/some/dir), owned by you. +# A tape starts from a clean ~/.netclaw. A tape whose first line comments contain +# `# stack: fresh` runs against the first-run state (no model config); `# stack: bare` is the +# same empty home plus the demo MCP server container, for tapes that seed their own config; +# all others run against the configured daemon. Exits non-zero if any tape fails; the stack is always +# torn down. Image, project name and inference settings: see stack.sh and README.md. +set -euo pipefail +# shellcheck source=lib.sh +. "$(dirname "${BASH_SOURCE[0]}")/lib.sh" -SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" -PROJECT_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)" -TAPES_DIR="$SCRIPT_DIR/tapes" -OUTPUT_DIR="$SCRIPT_DIR/output" -COMPOSE_FILE="$PROJECT_ROOT/docker-compose.screenshots.yml" +OUT_DIR="${OUT_DIR:-$SHOTS_DIR/output}" +mkdir -p "$OUT_DIR" +export CAPTURE_OUT_DIR="$(cd "$OUT_DIR" && pwd)" -# Use a clean netclaw home for the init wizard so we get first-run screens -export NETCLAW_SCREENSHOT_HOME="$SCRIPT_DIR/.netclaw-screenshot-home" - -mkdir -p "$OUTPUT_DIR" +tapes=("$@") +if [ "${#tapes[@]}" -eq 0 ]; then + for f in "$SHOTS_DIR"/tapes/*.tape; do tapes+=("$(basename "$f" .tape)"); done +fi -NETCLAW_PORT="${NETCLAW_PORT:-5299}" +cleanup() { + echo "==> Tearing down project $COMPOSE_PROJECT_NAME" + "$SHOTS_DIR/stack.sh" down >/dev/null 2>&1 || true +} +trap cleanup EXIT -# Pin the daemon image to the version the docs are written against (src/version.json), -# unless the caller already set NETCLAW_VERSION or NETCLAW_IMAGE. Keeps screenshots -# in lockstep with the documented release. -if [ -z "${NETCLAW_VERSION:-}" ] && [ -f "$PROJECT_ROOT/src/version.json" ]; then - NETCLAW_VERSION="$(node -e "process.stdout.write(require('$PROJECT_ROOT/src/version.json').documentedVersion)" 2>/dev/null || true)" -fi -export NETCLAW_VERSION -echo "==> Capturing against netclaw ${NETCLAW_IMAGE:-ghcr.io/netclaw-dev/netclaw:${NETCLAW_VERSION:-0.22.1}}" +echo "==> Capturing against $NETCLAW_IMAGE ($INFERENCE_MODE inference), output $CAPTURE_OUT_DIR" -echo "==> Starting screenshot environment (building images if needed)..." -docker compose -f "$COMPOSE_FILE" up -d --build --wait +failed=() +for name in "${tapes[@]}"; do + tape="$SHOTS_DIR/tapes/$name.tape" + [ -f "$tape" ] || { echo "capture: no such tape: $name" >&2; exit 2; } + state=configured + if grep -qE '^#[[:space:]]*stack:[[:space:]]*fresh' "$tape"; then state=fresh; fi + if grep -qE '^#[[:space:]]*stack:[[:space:]]*bare' "$tape"; then state=bare; fi -echo "==> Waiting for netclawd health check on port $NETCLAW_PORT..." -for i in $(seq 1 60); do - if curl -sf "http://127.0.0.1:${NETCLAW_PORT}/api/health/ready" >/dev/null 2>&1; then - echo " netclawd is healthy." - break - fi - if [ "$i" -eq 60 ]; then - echo " ERROR: netclawd did not become healthy in time." - docker compose -f "$COMPOSE_FILE" logs netclawd - docker compose -f "$COMPOSE_FILE" down - exit 1 - fi - sleep 2 -done + # `# image: 0.27.0` runs this one tape against another release (update-check.tape needs an + # install that is genuinely one version behind). `# host-docker: true` lets the tape drive the + # host's Docker through its socket (docker-health.tape). + image_version="$(sed -nE 's/^#[[:space:]]*image:[[:space:]]*([0-9A-Za-z._-]+).*/\1/p' "$tape" | head -1)" + host_docker="" + if grep -qE '^#[[:space:]]*host-docker:[[:space:]]*true' "$tape"; then host_docker=1; fi -echo "==> Running VHS tapes..." -for tape in "$TAPES_DIR"/*.tape; do - if [ -f "$tape" ]; then - echo " Running: $(basename "$tape")" - vhs "$tape" || echo " WARNING: $(basename "$tape") failed" + echo "==> $name: clean daemon ($state)" + if ! ( + if [ -n "$image_version" ]; then + unset NETCLAW_IMAGE + export NETCLAW_VERSION="$image_version" + # shellcheck source=lib.sh + . "$SHOTS_DIR/lib.sh" + echo "==> $name: this tape pins image $NETCLAW_IMAGE" + fi + if [ -n "$host_docker" ]; then export CAPTURE_HOST_DOCKER=1; fi + # reset = wipe ~/.netclaw and (re)start the daemon; also creates the stack on first use. + "$SHOTS_DIR/stack.sh" reset $([ "$state" != configured ] && echo "--$state") && + dc "$state" run --rm --no-deps -T --build capture "$name" + ); then + failed+=("$name") fi done -echo "==> Tearing down environment..." -docker compose -f "$COMPOSE_FILE" down - -echo "==> Screenshots saved to: $OUTPUT_DIR" -ls -la "$OUTPUT_DIR"/*.png 2>/dev/null || echo " No PNG files generated." +if [ "${#failed[@]}" -gt 0 ]; then + echo "capture: FAILED tapes: ${failed[*]}" >&2 + exit 1 +fi +echo "==> Done: ${tapes[*]}" diff --git a/screenshots/capture/Dockerfile b/screenshots/capture/Dockerfile new file mode 100644 index 0000000..fce7e0b --- /dev/null +++ b/screenshots/capture/Dockerfile @@ -0,0 +1,48 @@ +# Capture image: VHS (ttyd, ffmpeg, headless chromium) + the released netclaw CLI. +# +# Why this shape: the release image (ubuntu:24.04, glibc) ships the CLI as a +# self-contained .NET single-file binary, and the VHS image (debian trixie, glibc 2.41, +# ICU and libsodium already present) runs it unmodified. So we take the VHS image and +# copy the CLI and its self-dropping launcher out of the release image. The CLI in a +# tape is therefore byte-for-byte the released one, never a host build. +# +# The container shares the daemon's network namespace and ~/.netclaw volume (see +# docker-compose.screenshots.yml), so `netclaw status` typed in a recording reaches the +# daemon over loopback the way a real user's CLI does. +ARG NETCLAW_IMAGE +ARG VHS_IMAGE=ghcr.io/charmbracelet/vhs:v0.12.1 + +FROM ${NETCLAW_IMAGE} AS netclaw + +FROM ${VHS_IMAGE} + +RUN apt-get update \ + && apt-get install -y --no-install-recommends gosu curl ca-certificates git jq \ + && rm -rf /var/lib/apt/lists/* + +# Same uid/gid as the daemon so the shared ~/.netclaw volume is readable and writable. +RUN groupadd --gid 1654 netclaw \ + && useradd --uid 1654 --gid netclaw --create-home netclaw \ + && mkdir -p /home/netclaw/.netclaw /work/screenshots/output \ + && chown -R netclaw:netclaw /home/netclaw /work + +# The skillserver CLI, used by the seed scripts to publish demo skills to the stack's skill server. +ARG SKILLSERVER_VERSION=0.4.1 +RUN arch="$(uname -m)"; case "$arch" in x86_64) rid=x64 ;; aarch64) rid=arm64 ;; esac \ + && curl -fsSL "https://github.com/netclaw-dev/skill-server/releases/download/${SKILLSERVER_VERSION}/skillserver-${SKILLSERVER_VERSION}-linux-${rid}.tar.gz" \ + | tar -xz -C /usr/local/bin \ + && chmod +x /usr/local/bin/skillserver + +COPY --from=netclaw /opt/netclaw/cli/netclaw /opt/netclaw/cli/netclaw +COPY --from=netclaw /usr/local/bin/netclaw /usr/local/bin/netclaw + +COPY entrypoint.sh /usr/local/bin/capture-entrypoint +RUN chmod +x /usr/local/bin/capture-entrypoint + +ENV HOME=/home/netclaw \ + NETCLAW_HOME=/home/netclaw/.netclaw \ + NETCLAW_CONTAINER_SUPERVISOR=entrypoint \ + VHS_NO_SANDBOX=true + +WORKDIR /work +ENTRYPOINT ["/usr/local/bin/capture-entrypoint"] diff --git a/screenshots/capture/entrypoint.sh b/screenshots/capture/entrypoint.sh new file mode 100755 index 0000000..a8d7993 --- /dev/null +++ b/screenshots/capture/entrypoint.sh @@ -0,0 +1,63 @@ +#!/bin/bash +# Runs inside the capture container. Arguments are tape names (without .tape). +# Each tape is recorded as the unprivileged `netclaw` user (same as the daemon), into a +# staging dir, then copied to /out owned by the invoking host user. +# Exits non-zero as soon as any tape fails or does not produce a file it declares. +set -uo pipefail + +STAGE=/work/screenshots/output +OUT=/out +HOST_UID="${HOST_UID:-0}" +HOST_GID="${HOST_GID:-0}" + +if [ "$#" -eq 0 ]; then + echo "capture: no tapes given" >&2 + exit 2 +fi + +# Tapes marked `# host-docker: true` get the host's Docker socket (compose/docker-host.yml). The +# netclaw user needs the socket's group to use it. +if [ -S /var/run/docker.sock ]; then + sock_gid="$(stat -c %g /var/run/docker.sock)" + getent group "$sock_gid" >/dev/null || groupadd --gid "$sock_gid" dockerhost + usermod -aG "$(getent group "$sock_gid" | cut -d: -f1)" netclaw +fi + +for name in "$@"; do + tape="/tapes/${name}.tape" + if [ ! -f "$tape" ]; then + echo "capture: no such tape: $name" >&2 + exit 2 + fi + + echo "==> Recording $name ($(netclaw --version))" + rm -rf "${STAGE:?}"/* + chown netclaw:netclaw "$STAGE" + + # VHS writes a Screenshot asynchronously, so one that is the LAST command in a tape is + # lost when the recording ends. Run a copy with a trailing Sleep so any tape is safe. + run_tape="/tmp/run-${name}.tape" + { cat "$tape"; printf '\nSleep 1s\n'; } > "$run_tape" + chmod 644 "$run_tape" + + if ! gosu netclaw vhs "$run_tape"; then + echo "capture: FAILED: vhs exited non-zero for $name" >&2 + exit 1 + fi + + # Every Output/Screenshot path the tape declares must exist. + missing=0 + while read -r path; do + if [ ! -s "/work/$path" ]; then + echo "capture: FAILED: $name did not produce $path" >&2 + missing=1 + fi + done < <(sed -nE 's/^[[:space:]]*(Output|Screenshot)[[:space:]]+"?([^" ]+)"?.*$/\2/p' "$tape" | grep -v '^#') + [ "$missing" -eq 0 ] || exit 1 + + # Only stills are published. A tape needs an Output line for VHS to run; its GIF stays in staging. + while IFS= read -r -d '' f; do + install -m 0644 -o "$HOST_UID" -g "$HOST_GID" "$f" "$OUT/$(basename "$f")" + echo " wrote $(basename "$f")" + done < <(find "$STAGE" -maxdepth 1 -type f -name '*.png' -print0) +done diff --git a/screenshots/compose/configured.yml b/screenshots/compose/configured.yml new file mode 100644 index 0000000..9303652 --- /dev/null +++ b/screenshots/compose/configured.yml @@ -0,0 +1,29 @@ +# Overlay: the "initialized" daemon. Adds the demo MCP server container. The daemon's +# provider, model roles and MCP entry are NOT env vars: stack.sh writes them with the +# real CLI (`netclaw provider add`, `netclaw model set`, `netclaw mcp add`) so +# ~/.netclaw/config/netclaw.json looks exactly like a user's after `netclaw init`, and +# the CLI's own config preflight and TUIs see it. +# Relative paths resolve against the directory of docker-compose.screenshots.yml. +services: + mcp-demo: + image: netclaw-docs-mcp-demo:1 + build: ./screenshots/mcp-server + network_mode: "service:netns" + depends_on: + - netns + healthcheck: + test: ["CMD", "python", "-c", "import socket; s=socket.create_connection(('127.0.0.1',8080),2); s.close()"] + interval: 3s + timeout: 5s + retries: 20 + start_period: 3s + + netclawd: + depends_on: + mcp-demo: + condition: service_healthy + environment: + # Real inference endpoints can be slow to first token. + NETCLAW_Session__TurnLlmTimeoutSeconds: "600" + NETCLAW_Session__FirstTokenTimeoutSeconds: "600" + NETCLAW_Session__PrefillTimeoutSeconds: "600" diff --git a/screenshots/compose/docker-host.yml b/screenshots/compose/docker-host.yml new file mode 100644 index 0000000..16e3794 --- /dev/null +++ b/screenshots/compose/docker-host.yml @@ -0,0 +1,12 @@ +# Overlay: lets a tape drive the host's Docker (docker-health.tape), by mounting the Docker socket +# and the host's docker client into the capture container. Added by lib.sh only for tapes marked +# `# host-docker: true`. The tape must filter everything it prints to its own container, so nothing +# else on the host shows up in the screenshot. +services: + capture: + environment: + NETCLAW_IMAGE: ${NETCLAW_IMAGE} + COMPOSE_PROJECT_NAME: ${COMPOSE_PROJECT_NAME} + volumes: + - /var/run/docker.sock:/var/run/docker.sock + - ${DOCKER_CLI_PATH:-/usr/bin/docker}:/usr/local/bin/docker:ro diff --git a/screenshots/compose/forwarder.conf.template b/screenshots/compose/forwarder.conf.template new file mode 100644 index 0000000..566f0f2 --- /dev/null +++ b/screenshots/compose/forwarder.conf.template @@ -0,0 +1,23 @@ +# Relays plain HTTP on :8081 to the real inference endpoint, so netclaw is configured with +# http://llm.example.internal:8081 and the real hostname never appears in netclaw.json, +# the TUI or a recording. The upstream's model id is rewritten to a neutral one in every +# response (so /v1/models lists the neutral id and `netclaw init` / `model discover` store it); +# requests keep the neutral id, which llama.cpp style servers accept for any model name. +# Rendered by the nginx image from INFERENCE_UPSTREAM, INFERENCE_UPSTREAM_HOST, +# INFERENCE_UPSTREAM_MODEL and INFERENCE_MODEL_ID (screenshots/.env, gitignored). +server { + listen 8081; + location / { + proxy_pass ${INFERENCE_UPSTREAM}; + proxy_http_version 1.1; + proxy_ssl_server_name on; + proxy_set_header Host ${INFERENCE_UPSTREAM_HOST}; + proxy_set_header Accept-Encoding ""; + proxy_buffering off; + proxy_read_timeout 900s; + proxy_send_timeout 900s; + sub_filter_types application/json text/event-stream; + sub_filter_once off; + sub_filter '${INFERENCE_UPSTREAM_MODEL}' '${INFERENCE_MODEL_ID}'; + } +} diff --git a/screenshots/compose/forwarder.yml b/screenshots/compose/forwarder.yml new file mode 100644 index 0000000..b2e849d --- /dev/null +++ b/screenshots/compose/forwarder.yml @@ -0,0 +1,32 @@ +# Overlay: keeps a real inference endpoint out of every frame. Added by lib.sh when +# INFERENCE_UPSTREAM is set (an https:// URL in screenshots/.env, together with +# INFERENCE_UPSTREAM_MODEL, the model id the upstream really serves). +# +# - `forwarder` joins the stack's network namespace and relays :8081 to the upstream, rewriting +# the upstream's model id to the neutral INFERENCE_MODEL_ID (default demo-model) in responses. +# - `extra_hosts` on the namespace owner maps llm.example.internal to loopback, and every +# container that shares the namespace (daemon, capture, helpers) inherits the entry. +# +# The configured endpoint is http://llm.example.internal:8081 and the model id is demo-model. +# Port 8081 because the demo MCP server owns 8080 in the shared network namespace. +services: + netns: + extra_hosts: + - "llm.example.internal:127.0.0.1" + + forwarder: + image: nginx:1.27-alpine + network_mode: "service:netns" + depends_on: + - netns + environment: + INFERENCE_UPSTREAM: ${INFERENCE_UPSTREAM:?set in screenshots/.env} + INFERENCE_UPSTREAM_HOST: ${INFERENCE_UPSTREAM_HOST:?set in screenshots/.env} + INFERENCE_UPSTREAM_MODEL: ${INFERENCE_UPSTREAM_MODEL:?set in screenshots/.env} + INFERENCE_MODEL_ID: ${INFERENCE_MODEL_ID:?set in screenshots/.env} + volumes: + - ./screenshots/compose/forwarder.conf.template:/etc/nginx/templates/default.conf.template:ro + + netclawd: + depends_on: + - forwarder diff --git a/screenshots/compose/ollama.yml b/screenshots/compose/ollama.yml new file mode 100644 index 0000000..ef2c1fa --- /dev/null +++ b/screenshots/compose/ollama.yml @@ -0,0 +1,31 @@ +# Overlay: hermetic inference. A tiny Ollama model, no external network use after the +# one-time model pull. Shares the stack's network namespace, so the endpoint is +# http://127.0.0.1:11434 (which is also what the `netclaw init` wizard defaults to). +# Relative paths resolve against the directory of docker-compose.screenshots.yml. +services: + ollama: + image: ollama/ollama:0.32.0 + network_mode: "service:netns" + depends_on: + - netns + volumes: + - ollama-data:/root/.ollama + + ollama-init: + image: curlimages/curl:8.13.0 + network_mode: "service:netns" + entrypoint: ["/bin/sh", "/usr/local/bin/ollama-init.sh"] + environment: + OLLAMA_MODEL: ${OLLAMA_MODEL:-qwen2:0.5b} + depends_on: + - ollama + volumes: + - ./screenshots/scripts/ollama-init.sh:/usr/local/bin/ollama-init.sh:ro + + netclawd: + depends_on: + ollama-init: + condition: service_completed_successfully + +volumes: + ollama-data: diff --git a/screenshots/compose/projects.yml b/screenshots/compose/projects.yml new file mode 100644 index 0000000..155ed9f --- /dev/null +++ b/screenshots/compose/projects.yml @@ -0,0 +1,25 @@ +# Overlay: a neutral projects directory shared by the daemon and the capture container, +# so a tape can create a small Git repository at /home/netclaw/projects/website that the +# daemon's shell tool can also see (approval prompts offer "This repository" only inside +# a Git repository). Created and owned by the netclaw uid (1654) by a one-shot init. +services: + projects-init: + image: alpine:3.20 + network_mode: "none" + command: ["sh", "-c", "mkdir -p /projects && chown 1654:1654 /projects"] + volumes: + - projects:/projects + + netclawd: + depends_on: + projects-init: + condition: service_completed_successfully + volumes: + - projects:/home/netclaw/projects + + capture: + volumes: + - projects:/home/netclaw/projects + +volumes: + projects: diff --git a/screenshots/compose/skillserver.yml b/screenshots/compose/skillserver.yml new file mode 100644 index 0000000..abdb677 --- /dev/null +++ b/screenshots/compose/skillserver.yml @@ -0,0 +1,31 @@ +# Overlay: a skill server on the stack's network namespace (port 8090, because the demo MCP +# server owns 8080), for the skill-list and skill-sync captures. It +# is seeded with a throwaway demo key (publishing needs one; reads stay open). Pinned to the release the docs describe. +services: + skillserver-init: + image: ghcr.io/netclaw-dev/skillserver:0.4.1 + user: "0:0" + network_mode: "none" + entrypoint: ["sh", "-c", "chown -R 1654:1654 /data"] + restart: "no" + volumes: + - skill-data:/data + + skillserver: + image: ghcr.io/netclaw-dev/skillserver:0.4.1 + network_mode: "service:netns" + depends_on: + netns: + condition: service_started + skillserver-init: + condition: service_completed_successfully + environment: + SKILLSERVER__DATAPATH: /data + SKILLSERVER__BASEURL: http://127.0.0.1:8090 + SKILLSERVER__APIKEY: demo-publish-key + ASPNETCORE_URLS: http://+:8090 + volumes: + - skill-data:/data + +volumes: + skill-data: diff --git a/screenshots/lib.sh b/screenshots/lib.sh new file mode 100755 index 0000000..5beee83 --- /dev/null +++ b/screenshots/lib.sh @@ -0,0 +1,109 @@ +#!/bin/bash +# Shared helpers for stack.sh and capture.sh. Source it, don't run it. +# +# Resolves, in order: ./screenshots/.env (gitignored), then defaults. +# COMPOSE_PROJECT_NAME isolates containers, network and volumes (default netclaw-shots) +# NETCLAW_VERSION release tag; default = documentedVersion in src/version.json +# NETCLAW_IMAGE full image reference; wins over NETCLAW_VERSION +# INFERENCE_BASE_URL external OpenAI-compatible endpoint (unset = hermetic Ollama) +# INFERENCE_MODEL_ID model id served there (required with INFERENCE_BASE_URL) +# OLLAMA_MODEL hermetic model (default qwen2:0.5b) +# INFERENCE_UPSTREAM base URL of a real OpenAI-compatible server (relay mode). Adds the forwarder +# (compose/forwarder.yml) that answers to http://llm.example.internal:8081 +# inside the stack and rewrites the upstream's model id to `demo-model` in +# every response, so tapes get real answers while neither the real hostname +# nor the real model name reaches netclaw.json or a screenshot. Needs +# INFERENCE_UPSTREAM_MODEL (the id the upstream really serves). Ollama stays +# available alongside it, for the tapes that need a reachable local provider. + +SHOTS_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +PROJECT_ROOT="$(cd "$SHOTS_DIR/.." && pwd)" +BASE_COMPOSE="$PROJECT_ROOT/docker-compose.screenshots.yml" + +if [ -f "$SHOTS_DIR/.env" ]; then + set -a + # shellcheck disable=SC1091 + . "$SHOTS_DIR/.env" + set +a +fi + +export COMPOSE_PROJECT_NAME="${COMPOSE_PROJECT_NAME:-netclaw-shots}" +export COMPOSE_PROGRESS="${COMPOSE_PROGRESS:-quiet}" # set COMPOSE_PROGRESS=plain to see build/pull output +# Compose project names must be lowercase alphanumerics, '-' or '_'. +case "$COMPOSE_PROJECT_NAME" in + *[!a-z0-9_-]* | "") echo "COMPOSE_PROJECT_NAME must match [a-z0-9_-]+ (got '$COMPOSE_PROJECT_NAME')" >&2; exit 2 ;; +esac + +# Version: one source of truth (src/version.json) unless overridden. +if [ -z "${NETCLAW_IMAGE:-}" ]; then + if [ -z "${NETCLAW_VERSION:-}" ]; then + NETCLAW_VERSION="$(sed -n 's/.*"documentedVersion": *"\([^"]*\)".*/\1/p' "$PROJECT_ROOT/src/version.json")" + [ -n "$NETCLAW_VERSION" ] || { echo "could not read documentedVersion from src/version.json" >&2; exit 2; } + fi + NETCLAW_IMAGE="ghcr.io/netclaw-dev/netclaw:${NETCLAW_VERSION}" +fi +export NETCLAW_IMAGE +export NETCLAW_VERSION="${NETCLAW_IMAGE##*:}" +export NETCLAW_IMAGE_TAG="$(printf '%s' "$NETCLAW_VERSION" | tr -c 'A-Za-z0-9_.-' '_')" + +# Inference: three modes. +# relay INFERENCE_UPSTREAM set: real answers through the forwarder, neutral hostname and model +# external INFERENCE_BASE_URL set: a real endpoint written into netclaw.json as is +# hermetic neither: Ollama with a tiny model, no external network use +if [ -n "${INFERENCE_UPSTREAM:-}" ]; then + : "${INFERENCE_UPSTREAM_MODEL:?INFERENCE_UPSTREAM_MODEL is required with INFERENCE_UPSTREAM (the model id the upstream serves)}" + INFERENCE_UPSTREAM_HOST="${INFERENCE_UPSTREAM#*://}"; INFERENCE_UPSTREAM_HOST="${INFERENCE_UPSTREAM_HOST%%/*}" + INFERENCE_BASE_URL="${INFERENCE_BASE_URL:-http://llm.example.internal:8081}" + INFERENCE_MODEL_ID="${INFERENCE_MODEL_ID:-demo-model}" + INFERENCE_MODE=relay + export INFERENCE_UPSTREAM INFERENCE_UPSTREAM_HOST INFERENCE_UPSTREAM_MODEL INFERENCE_MODEL_ID + export NC_INFERENCE_TYPE=openai-compatible + export NC_INFERENCE_ENDPOINT="$INFERENCE_BASE_URL" + export NC_INFERENCE_MODEL="$INFERENCE_MODEL_ID" +elif [ -n "${INFERENCE_BASE_URL:-}" ]; then + : "${INFERENCE_MODEL_ID:?INFERENCE_MODEL_ID is required when INFERENCE_BASE_URL is set}" + INFERENCE_MODE=external + export NC_INFERENCE_TYPE=openai-compatible + export NC_INFERENCE_ENDPOINT="$INFERENCE_BASE_URL" + export NC_INFERENCE_MODEL="$INFERENCE_MODEL_ID" +else + INFERENCE_MODE=hermetic + export NC_INFERENCE_TYPE=ollama + export NC_INFERENCE_ENDPOINT=http://127.0.0.1:11434 + export NC_INFERENCE_MODEL="${OLLAMA_MODEL:-qwen2:0.5b}" +fi + +# Playwright image/module version: the one locked in package-lock.json. +if [ -z "${PLAYWRIGHT_VERSION:-}" ]; then + PLAYWRIGHT_VERSION="$(grep -A1 '"node_modules/playwright":' "$PROJECT_ROOT/package-lock.json" | sed -n 's/.*"version": *"\([^"]*\)".*/\1/p')" +fi +export PLAYWRIGHT_VERSION + +export HOST_UID="${HOST_UID:-$(id -u)}" +export HOST_GID="${HOST_GID:-$(id -g)}" + +# compose_files +# bare: like configured (demo MCP server container up) but stack.sh does not seed any config +# configured: daemon with a provider, model roles and the demo MCP server +# fresh: daemon with no model config at all (first-run state for `netclaw init`) +# all: every overlay, for commands (exec/logs/down) that must see every service +compose_files() { + local state="$1" + local files=(-f "$BASE_COMPOSE") + if [ "$INFERENCE_MODE" != external ]; then files+=(-f "$SHOTS_DIR/compose/ollama.yml"); fi + if [ "$INFERENCE_MODE" = relay ]; then files+=(-f "$SHOTS_DIR/compose/forwarder.yml"); fi + if [ "$state" != fresh ]; then files+=(-f "$SHOTS_DIR/compose/configured.yml" -f "$SHOTS_DIR/compose/skillserver.yml"); fi + files+=(-f "$SHOTS_DIR/compose/projects.yml") + if [ -n "${CAPTURE_HOST_DOCKER:-}" ]; then files+=(-f "$SHOTS_DIR/compose/docker-host.yml"); fi + printf '%s\0' "${files[@]}" +} + +# dc +dc() { + local state="$1"; shift + local files=() + while IFS= read -r -d '' f; do files+=("$f"); done < <(compose_files "$state") + docker compose "${files[@]}" "$@" +} + +home_volume() { printf '%s_netclaw-home' "$COMPOSE_PROJECT_NAME"; } diff --git a/screenshots/mattermost/approval.mjs b/screenshots/mattermost/approval.mjs new file mode 100644 index 0000000..373e5b6 --- /dev/null +++ b/screenshots/mattermost/approval.mjs @@ -0,0 +1,95 @@ +#!/usr/bin/env node +// Captures the landing page's approval prompt (public/assets/approval-prompt.png): the thread +// post that netclaw sends when shell_execute needs approval, with its buttons, framed by +// frame-approval.mjs. Runs in the Playwright container (run.sh with SCRIPTED_APPROVAL=1). +// +// Env: BOOTSTRAP_FILE, OUT_DIR, REPLY_TIMEOUT_SECONDS (default 120) +import { chromium } from 'playwright'; +import { readFileSync, mkdirSync } from 'fs'; +import { dirname, join } from 'path'; +import { fileURLToPath } from 'url'; +import { frame } from './frame-approval.mjs'; + +const __dirname = dirname(fileURLToPath(import.meta.url)); +const boot = JSON.parse(readFileSync(process.env.BOOTSTRAP_FILE || join(__dirname, '.bootstrap.json'), 'utf8')); +const OUT = process.env.OUT_DIR || join(__dirname, '..', 'output'); +const TIMEOUT_MS = Number(process.env.REPLY_TIMEOUT_SECONDS || 120) * 1000; +const BASE = boot.serverUrl; +mkdirSync(OUT, { recursive: true }); + +// Mattermost's Onyx theme. +const ONYX = { + type: 'Onyx', sidebarBg: '#18181a', sidebarText: '#ffffff', sidebarUnreadText: '#ffffff', + sidebarTextHoverBg: '#25262a', sidebarTextActiveBorder: '#afbe5d', sidebarTextActiveColor: '#ffffff', + sidebarHeaderBg: '#1f1f1f', sidebarTeamBarBg: '#121212', sidebarHeaderTextColor: '#ffffff', + onlineIndicator: '#a4c940', awayIndicator: '#dcbd4e', dndIndicator: '#d24b4e', + mentionBg: '#ffffff', mentionColor: '#1f1f1f', centerChannelBg: '#1f1f1f', centerChannelColor: '#dddddd', + newMessageSeparator: '#cc8f00', linkColor: '#a4ffeb', buttonBg: '#a4ffeb', buttonColor: '#1f1f1f', + errorTextColor: '#ff6461', mentionHighlightBg: '#784098', mentionHighlightLink: '#a4ffeb', codeTheme: 'solarized-dark', +}; + +async function api(method, path, body, token) { + const res = await fetch(`${BASE}/api/v4${path}`, { + method, + headers: { 'Content-Type': 'application/json', ...(token ? { Authorization: `Bearer ${token}` } : {}) }, + body: body ? JSON.stringify(body) : undefined, + }); + if (!res.ok) throw new Error(`${method} ${path} -> ${res.status}: ${(await res.text()).slice(0, 200)}`); + return res; +} + +const login = await api('POST', '/users/login', { login_id: boot.testuser.username, password: boot.testuser.password }); +const token = login.headers.get('Token'); +const me = await login.json(); +await api('PUT', `/users/${me.id}/preferences`, [ + { user_id: me.id, category: 'theme', name: '', value: JSON.stringify(ONYX) }, + { user_id: me.id, category: 'tutorial_step', name: me.id, value: '999' }, + { user_id: me.id, category: 'onboarding_task_list', name: 'onboarding_task_list_show', value: 'false' }, + { user_id: me.id, category: 'recommended_next_steps', name: 'hide', value: 'true' }, +], token); + +const root = await (await api('POST', '/posts', { channel_id: boot.channel.id, message: '@netclaw push the website changes' }, token)).json(); +const thread = async () => { + const d = await (await api('GET', `/posts/${root.id}/thread`, null, token)).json(); + return d.order.map(id => d.posts[id]).sort((a, b) => a.create_at - b.create_at); +}; +let prompt; +for (const t0 = Date.now(); Date.now() - t0 < TIMEOUT_MS && !prompt; await new Promise(r => setTimeout(r, 2000))) + prompt = (await thread()).find(p => p.user_id === boot.bot.userId && /Tool approval required/i.test(p.message)); +if (!prompt) { console.error('no approval prompt within the timeout'); process.exit(1); } +const actions = (prompt.props?.attachments?.[0]?.actions || []).map(a => a.name); +console.log('prompt text:\n' + prompt.message + '\nbuttons: ' + JSON.stringify(actions)); +if (!actions.length) { console.error('the prompt has no buttons; CallbackUrl is not taking effect'); process.exit(1); } + +const browser = await chromium.launch(); +const ctx = await browser.newContext({ viewport: { width: 1280, height: 900 }, deviceScaleFactor: 2, colorScheme: 'dark' }); +await ctx.addInitScript(() => { try { localStorage.setItem('__landingPageSeen__', 'true'); localStorage.setItem('onboarding_complete', 'true'); } catch {} }); +const page = await ctx.newPage(); +await page.goto(`${BASE}/login`, { waitUntil: 'networkidle' }); +for (const rx of [/view in browser/i, /continue in browser/i]) { + try { const l = page.getByText(rx).first(); if (await l.isVisible({ timeout: 1500 })) { await l.click(); break; } } catch {} +} +await page.waitForSelector('input[type="password"]', { timeout: 30000 }); +await page.locator('input[type="text"], input[type="email"]').first().fill(boot.testuser.username); +await page.locator('input[type="password"]').first().fill(boot.testuser.password); +await page.getByRole('button', { name: /log in|sign in/i }).first().click(); +await page.waitForURL(u => !/\/login/.test(new URL(u).pathname), { timeout: 20000 }); +await page.waitForLoadState('networkidle', { timeout: 15000 }).catch(() => {}); + +await page.goto(`${BASE}/${boot.team.name}/pl/${root.id}`, { waitUntil: 'networkidle' }); +await page.waitForTimeout(2500); +await page.getByRole('button', { name: /\d+ repl(y|ies)/i }).last().click(); +await page.waitForTimeout(2500); +await page.getByRole('button', { name: /expand/i }).first().click(); // widen the panel so the buttons fit one row +await page.waitForTimeout(1500); +await page.mouse.move(640, 880); +await page.addStyleTag({ content: '.NotificationSeparator{display:none !important}' }); // the "New Messages" line +await page.waitForTimeout(500); +const post = page.locator(`#sidebar-right #post_${prompt.id}, #sidebar-right [id$="${prompt.id}"]`).first(); +await post.waitFor({ timeout: 15000 }); +await page.screenshot({ path: join(OUT, '_approval-full.png') }); +const raw = join(OUT, '_approval-raw.png'); +await post.screenshot({ path: raw }); +await frame(browser, raw, join(OUT, 'approval-prompt.png')); +await browser.close(); +console.log(' ok approval-prompt.png'); diff --git a/screenshots/mattermost/bootstrap.mjs b/screenshots/mattermost/bootstrap.mjs index c6a8c35..4bb79b8 100644 --- a/screenshots/mattermost/bootstrap.mjs +++ b/screenshots/mattermost/bootstrap.mjs @@ -2,16 +2,19 @@ // Bootstraps a fresh mattermost-preview the same way the netclaw repo's Aspire // demo does: first user (auto system-admin), team, bot + access token, channel, // and a non-admin test user. Writes the resulting IDs/token/credentials to -// .bootstrap.json for run.sh (netclaw config) and capture.mjs (Playwright). +// BOOTSTRAP_FILE for run.sh (netclaw config) and capture.mjs (Playwright). // -// Usage: node bootstrap.mjs [serverUrl] (default http://localhost:8065) +// Runs inside the Playwright container (run.sh), on the same compose network as Mattermost. +// Usage: node bootstrap.mjs [serverUrl] (default http://mattermost:8065) +// Env: BOOTSTRAP_FILE where to write the result (default ./.bootstrap.json) import { writeFileSync } from 'fs'; import { dirname, join } from 'path'; import { fileURLToPath } from 'url'; -const SERVER = (process.argv[2] || 'http://localhost:8065').replace(/\/$/, ''); +const SERVER = (process.argv[2] || 'http://mattermost:8065').replace(/\/$/, ''); const __dirname = dirname(fileURLToPath(import.meta.url)); +const BOOTSTRAP_FILE = process.env.BOOTSTRAP_FILE || join(__dirname, '.bootstrap.json'); const ADMIN = { email: 'admin@test.local', username: 'admin', password: 'Admin1234!' }; const USER = { email: 'testuser@test.local', username: 'testuser', password: 'TestUser1234!' }; @@ -97,9 +100,9 @@ async function main() { channel: { id: channel.id, name: channel.name }, bot: { userId: bot.user_id, username: bot.username, token: tok.token }, }; - writeFileSync(join(__dirname, '.bootstrap.json'), JSON.stringify(out, null, 2)); + writeFileSync(BOOTSTRAP_FILE, JSON.stringify(out, null, 2)); console.log(`\n✅ Bootstrap complete. channel=${channel.id} bot=${bot.user_id}`); - console.log(` Wrote ${join(__dirname, '.bootstrap.json')}`); + console.log(` Wrote ${BOOTSTRAP_FILE}`); } main().catch(err => { console.error('❌ Bootstrap failed:', err.message); process.exit(1); }); diff --git a/screenshots/mattermost/capture-compose.mjs b/screenshots/mattermost/capture-compose.mjs new file mode 100644 index 0000000..d181de5 --- /dev/null +++ b/screenshots/mattermost/capture-compose.mjs @@ -0,0 +1,266 @@ +#!/usr/bin/env node +// Captures the Mattermost screenshots for channels/mattermost.md and +// guides/mattermost-docker-compose.md from the stack in examples/mattermost-compose/ +// (provisioned with provision.sh, wired with configure-netclaw.sh). +// +// One consistent look: Chromium, 1280x800 viewport, device scale factor 2, the Onyx +// dark theme, no browser chrome (page screenshots, cropped to the relevant region). +// Token IDs on screen are overwritten in the page before the screenshot is taken. +// +// Usage (Playwright container or any Node with playwright installed): +// MM_URL=http://localhost:8065 ADMIN_PASSWORD=... ALICE_PASSWORD=... \ +// node capture-compose.mjs [step ...] +// Steps (default: all): admin bots thread approval +// Env: TEAM (home), CHANNEL (assistant), OUT_DIR (../output), +// REPO_DIR (directory the approval prompt runs in), REPLY_TIMEOUT_SECONDS (300) + +import { chromium } from 'playwright'; +import { mkdirSync } from 'fs'; +import { dirname, join } from 'path'; +import { fileURLToPath } from 'url'; + +const here = dirname(fileURLToPath(import.meta.url)); +const BASE = (process.env.MM_URL || 'http://localhost:8065').replace(/\/$/, ''); +const TEAM = process.env.TEAM || 'home'; +const CHANNEL = process.env.CHANNEL || 'assistant'; +const OUT = process.env.OUT_DIR || join(here, '..', 'output'); +const REPO_DIR = process.env.REPO_DIR || '~/.netclaw/projects/website'; +const TIMEOUT_MS = Number(process.env.REPLY_TIMEOUT_SECONDS || 300) * 1000; +const ADMIN = { username: 'admin', password: process.env.ADMIN_PASSWORD }; +const ALICE = { username: 'alice', password: process.env.ALICE_PASSWORD }; +if (!ADMIN.password || !ALICE.password) { console.error('ADMIN_PASSWORD and ALICE_PASSWORD are required'); process.exit(2); } +mkdirSync(OUT, { recursive: true }); + +const steps = new Set(process.argv.slice(2).length ? process.argv.slice(2) : ['admin', 'bots', 'thread', 'approval']); +const VIEWPORT = { width: 1280, height: 800 }; +const MASK = 'x'.repeat(26); + +// Mattermost's Onyx theme. +const ONYX = { + type: 'Onyx', sidebarBg: '#18181a', sidebarText: '#ffffff', sidebarUnreadText: '#ffffff', + sidebarTextHoverBg: '#25262a', sidebarTextActiveBorder: '#afbe5d', sidebarTextActiveColor: '#ffffff', + sidebarHeaderBg: '#1f1f1f', sidebarTeamBarBg: '#121212', sidebarHeaderTextColor: '#ffffff', + onlineIndicator: '#a4c940', awayIndicator: '#dcbd4e', dndIndicator: '#d24b4e', + mentionBg: '#ffffff', mentionColor: '#1f1f1f', centerChannelBg: '#1f1f1f', centerChannelColor: '#dddddd', + newMessageSeparator: '#cc8f00', linkColor: '#a4ffeb', buttonBg: '#a4ffeb', buttonColor: '#1f1f1f', + errorTextColor: '#ff6461', mentionHighlightBg: '#784098', mentionHighlightLink: '#a4ffeb', codeTheme: 'solarized-dark', +}; + +async function api(method, path, body, token) { + const res = await fetch(`${BASE}/api/v4${path}`, { + method, + headers: { 'Content-Type': 'application/json', ...(token ? { Authorization: `Bearer ${token}` } : {}) }, + body: body ? JSON.stringify(body) : undefined, + }); + if (!res.ok) throw new Error(`${method} ${path} -> ${res.status}: ${(await res.text()).slice(0, 200)}`); + return res; +} + +async function apiLogin(user) { + const res = await api('POST', '/users/login', { login_id: user.username, password: user.password }); + const token = res.headers.get('Token'); + const me = await res.json(); + user.token = token; user.id = me.id; + await api('PUT', `/users/${me.id}/preferences`, [ + { user_id: me.id, category: 'theme', name: '', value: JSON.stringify(ONYX) }, + { user_id: me.id, category: 'tutorial_step', name: me.id, value: '999' }, + { user_id: me.id, category: 'onboarding_task_list', name: 'onboarding_task_list_show', value: 'false' }, + { user_id: me.id, category: 'recommended_next_steps', name: 'hide', value: 'true' }, + ], token).catch(e => console.warn(' preferences:', e.message)); + return token; +} + +async function newPage(browser, user, viewport = VIEWPORT) { + const ctx = await browser.newContext({ viewport, deviceScaleFactor: 2, colorScheme: 'dark' }); + await ctx.addInitScript(() => { + try { localStorage.setItem('__landingPageSeen__', 'true'); localStorage.setItem('onboarding_complete', 'true'); } catch {} + }); + const page = await ctx.newPage(); + await page.goto(`${BASE}/login`, { waitUntil: 'networkidle' }); + for (const rx of [/view in browser/i, /continue in browser/i]) { + try { const l = page.getByText(rx).first(); if (await l.isVisible({ timeout: 1500 })) { await l.click(); break; } } catch {} + } + await page.waitForSelector('input[type="password"]', { timeout: 30000 }); + await page.locator('input[type="text"], input[type="email"]').first().fill(user.username); + await page.locator('input[type="password"]').first().fill(user.password); + await page.getByRole('button', { name: /log in|sign in/i }).first().click(); + await page.waitForURL(u => !/\/login/.test(new URL(u).pathname), { timeout: 20000 }); + await page.waitForLoadState('networkidle', { timeout: 15000 }).catch(() => {}); + return { ctx, page }; + +} + +async function dismissModals(page) { + for (const rx of [/view in browser/i, /no thanks/i, /skip/i, /got it/i, /not now/i]) { + for (const role of ['link', 'button']) { + try { + const el = page.getByRole(role, { name: rx }).first(); + if (await el.isVisible({ timeout: 300 })) { await el.click({ timeout: 1000 }); await page.waitForTimeout(300); } + } catch {} + } + } +} + +// The "Preview Mode: Email notifications have not been configured" banner is a stack detail. +// Dismiss it the way a person would, with its close button. +async function closeBanner(page) { + try { + const bar = page.locator('.announcement-bar').first(); + if (await bar.isVisible({ timeout: 1500 })) { + await bar.locator('.announcement-bar__close, .close, [aria-label="Close"]').first().click({ timeout: 2000 }); + await page.waitForTimeout(500); + } + } catch {} +} + +const maskTokenIds = page => page.evaluate(mask => { + // Overwrite anything shaped like a 26-character Mattermost ID inside a "Token ID" line. + const walker = document.createTreeWalker(document.body, NodeFilter.SHOW_TEXT); + const hits = []; + while (walker.nextNode()) { + const n = walker.currentNode; + if (/^[a-z0-9]{26}$/.test(n.nodeValue.trim()) && /Token ID/i.test(n.parentElement?.parentElement?.textContent || '')) hits.push(n); + } + for (const n of hits) n.nodeValue = mask; + return hits.length; +}, MASK); + +const shot = async (page, name, clip) => { + await page.screenshot({ path: join(OUT, name), ...(clip ? { clip } : {}) }); + console.log(` ok ${name}`); +}; + +async function waitFor(fn, what) { + const start = Date.now(); + while (Date.now() - start < TIMEOUT_MS) { + const v = await fn(); + if (v) return v; + await new Promise(r => setTimeout(r, 2500)); + } + throw new Error(`timed out waiting for ${what}`); +} + +const browser = await chromium.launch(); +let failures = 0; +const attempt = async (name, fn) => { try { await fn(); } catch (e) { failures++; console.error(` FAIL ${name}: ${e.message}`); } }; + +await apiLogin(ADMIN); +await apiLogin(ALICE); +const channel = (await (await api('GET', `/teams/name/${TEAM}/channels/name/${CHANNEL}`, null, ALICE.token)).json()); + +if (steps.has('admin')) await attempt('admin', async () => { + const { ctx, page } = await newPage(browser, ADMIN); + await page.goto(`${BASE}/admin_console/integrations/bot_accounts`, { waitUntil: 'networkidle' }); + await page.waitForTimeout(2000); + await dismissModals(page); + await closeBanner(page); + await shot(page, 'mattermost-bot-accounts.png', { x: 220, y: 44, width: 1060, height: 376 }); + await ctx.close(); +}); + +if (steps.has('bots')) await attempt('bots', async () => { + const { ctx, page } = await newPage(browser, ADMIN); + await page.goto(`${BASE}/${TEAM}/integrations/bots`, { waitUntil: 'networkidle' }); + await page.waitForTimeout(2000); + await dismissModals(page); + await closeBanner(page); + console.log(` masked ${await maskTokenIds(page)} token id(s)`); + await shot(page, 'mattermost-create-bot.png', { x: 16, y: 76, width: 1240, height: 610 }); + await shot(page, 'mattermost-compose-bot-account.png', { x: 290, y: 372, width: 940, height: 150 }); + await ctx.close(); +}); + +const botUserId = async () => (await (await api('GET', '/users/username/netclaw', null, ALICE.token)).json()).id; + +async function postAsAlice(message) { + const res = await api('POST', '/posts', { channel_id: channel.id, message }, ALICE.token); + return (await res.json()); +} + +// Start each conversation from an empty channel so only that conversation is on screen. +async function clearChannel() { + const data = await (await api('GET', `/channels/${channel.id}/posts?per_page=200`, null, ADMIN.token)).json(); + for (const id of data.order) { + const p = data.posts[id]; + if (p.type === '' && !p.root_id) await api('DELETE', `/posts/${id}`, null, ADMIN.token).catch(() => {}); + } + await new Promise(r => setTimeout(r, 1000)); +} + +async function threadPosts(rootId) { + const data = await (await api('GET', `/posts/${rootId}/thread`, null, ALICE.token)).json(); + return data.order.map(id => data.posts[id]).sort((a, b) => a.create_at - b.create_at); +} + +async function openThread(page, rootId) { + await page.goto(`${BASE}/${TEAM}/pl/${rootId}`, { waitUntil: 'networkidle' }); + await page.waitForTimeout(2500); + await dismissModals(page); + await closeBanner(page); + // The permalink only highlights the post. Open the thread panel like a person would. + await page.getByRole('button', { name: /\d+ repl(y|ies)/i }).last().click(); + await page.waitForTimeout(2500); + await page.mouse.move(640, 790); +} + +if (steps.has('thread')) await attempt('thread', async () => { + const bot = await botUserId(); + await clearChannel(); + const root = await postAsAlice('@netclaw reply with a one-sentence greeting'); + await waitFor(async () => (await threadPosts(root.id)).some(p => p.user_id === bot && p.message), 'the bot reply'); + const { ctx, page } = await newPage(browser, ALICE); + await openThread(page, root.id); + await shot(page, 'mattermost-conversation.png'); + await shot(page, 'mattermost-compose-thread.png', { x: 270, y: 45, width: 965, height: 755 }); + await ctx.close(); +}); + +if (steps.has('approval')) await attempt('approval', async () => { + const bot = await botUserId(); + await clearChannel(); + const root = await postAsAlice(`@netclaw Use the shell to run exactly this, nothing else: git -C ${REPO_DIR} push origin main`); + const prompt = await waitFor(async () => (await threadPosts(root.id)).find(p => p.user_id === bot && /Tool approval required/i.test(p.message)), 'the approval prompt'); + console.log('--- prompt text ---\n' + prompt.message + '\n-------------------'); + const { ctx, page } = await newPage(browser, ALICE, { width: 1280, height: 900 }); + await openThread(page, root.id); + // Widen the thread panel so the command doesn't wrap. + await page.getByRole('button', { name: /expand/i }).first().click(); + await page.waitForTimeout(1500); + await page.mouse.move(640, 880); + const rhs = page.locator('#sidebar-right'); + const rb = await rhs.boundingBox(); + const posts = rhs.locator('.post'); + const fb = await posts.first().boundingBox(); + const lb = await posts.last().boundingBox(); + console.log('rhs', JSON.stringify(rb), 'first', JSON.stringify(fb), 'last', JSON.stringify(lb)); + await page.screenshot({ path: join(OUT, '_pending-full.png') }); + // Landing-page crop (public/assets/approval-prompt.png): just the pending prompt post. + await page.screenshot({ path: join(OUT, 'approval-prompt.png'), clip: { x: rb.x + 2, y: lb.y + 8, width: rb.width - 4, height: 270 } }); + console.log(' ok approval-prompt.png'); + // The guide's image: alice's request, the pending prompt with all six choices, and her + // one-letter reply typed in the thread box. A prompt resolves the instant the reply is + // sent (the post is edited in place), so this is the last frame that still shows the choices. + await page.locator('#reply_textbox').click(); + await page.keyboard.type('A'); + await page.waitForTimeout(400); + const y0 = fb.y - 24; + await page.screenshot({ path: join(OUT, 'mattermost-compose-approval.png'), clip: { x: rb.x + 2, y: y0, width: rb.width - 4, height: rb.y + rb.height - 22 - y0 } }); + console.log(' ok mattermost-compose-approval.png'); + await page.keyboard.press('Enter'); + await waitFor(async () => { + const posts = await threadPosts(root.id); + const last = posts[posts.length - 1]; + return posts.some(p => /Tool approval resolved/i.test(p.message)) && last.user_id === bot && !/Tool approval/i.test(last.message); + }, 'the approval to resolve and the bot to finish'); + await page.waitForTimeout(2500); + await page.mouse.move(640, 880); + const top = (await posts.first().boundingBox()).y - 24; + const boxTop = (await page.locator('#reply_textbox').boundingBox()).y; + await page.screenshot({ path: join(OUT, 'mattermost-compose-approval-resolved.png'), clip: { x: rb.x + 2, y: top, width: rb.width - 4, height: boxTop - 8 - top } }); + console.log(' ok mattermost-compose-approval-resolved.png'); + await ctx.close(); +}); + +await browser.close(); +if (failures) { console.error(`${failures} step(s) failed`); process.exit(1); } +console.log('Done.'); diff --git a/screenshots/mattermost/capture.mjs b/screenshots/mattermost/capture.mjs index f042351..59fd316 100644 --- a/screenshots/mattermost/capture.mjs +++ b/screenshots/mattermost/capture.mjs @@ -1,10 +1,12 @@ #!/usr/bin/env node // Drives Playwright against the bootstrapped Mattermost to capture the // channels/mattermost.md screenshots: the System Console settings, the bot -// account, and a live netclaw conversation. Reads .bootstrap.json. +// account, and a live netclaw conversation. Runs in the Playwright container (run.sh). // -// Each shot is independent (try/catch) so one failure doesn't lose the rest. +// Each shot is independent (try/catch) so one failure doesn't lose the rest; the +// process exits non-zero if any shot failed. // Usage: node capture.mjs +// Env: BOOTSTRAP_FILE (default ./.bootstrap.json), OUT_DIR (default ../output) import { chromium } from 'playwright'; import { readFileSync, mkdirSync } from 'fs'; @@ -12,10 +14,11 @@ import { dirname, join } from 'path'; import { fileURLToPath } from 'url'; const __dirname = dirname(fileURLToPath(import.meta.url)); -const boot = JSON.parse(readFileSync(join(__dirname, '.bootstrap.json'), 'utf8')); -const OUT = join(__dirname, '..', 'output'); +const boot = JSON.parse(readFileSync(process.env.BOOTSTRAP_FILE || join(__dirname, '.bootstrap.json'), 'utf8')); +const OUT = process.env.OUT_DIR || join(__dirname, '..', 'output'); mkdirSync(OUT, { recursive: true }); const BASE = boot.serverUrl; +let failures = 0; const shot = (page, name) => page.screenshot({ path: join(OUT, name) }).then(() => console.log(` ✓ ${name}`)); @@ -86,7 +89,7 @@ try { await dismissModals(page); await shot(page, 'mattermost-create-bot.png'); await ctx.close(); -} catch (e) { console.error(' ✗ admin shots:', e.message); } +} catch (e) { failures++; console.error(' ✗ admin shots:', e.message); } // 3: live conversation as the test user try { @@ -118,7 +121,8 @@ try { await page.waitForTimeout(1000); await shot(page, 'mattermost-conversation.png'); await ctx.close(); -} catch (e) { console.error(' ✗ conversation shot:', e.message); } +} catch (e) { failures++; console.error(' ✗ conversation shot:', e.message); } await browser.close(); +if (failures) { console.error(`${failures} capture step(s) failed`); process.exit(1); } console.log('Done.'); diff --git a/screenshots/mattermost/docker-compose.mattermost.yml b/screenshots/mattermost/docker-compose.mattermost.yml index 2769691..963e5db 100644 --- a/screenshots/mattermost/docker-compose.mattermost.yml +++ b/screenshots/mattermost/docker-compose.mattermost.yml @@ -1,24 +1,106 @@ -# Mattermost server for capturing the channels/mattermost.md screenshots. -# Brought up by run.sh, which then bootstraps it (bootstrap.mjs), starts the -# pinned netclaw image against it, and drives Playwright (capture.mjs). +# Mattermost capture topology. Everything runs in containers on one compose network; +# nothing is published to the host and there are no fixed container names, so projects +# do not collide. Driven by run.sh (do not run directly). # -# mattermost-preview is an all-in-one testing image (bundled Postgres). amd64-only. +# mattermost all-in-one testing image (bundled Postgres, amd64 only) +# netclawd the released daemon; reaches Mattermost at http://mattermost:8065 +# playwright bootstrap + browser captures, reaching Mattermost by the same name +# +# Models use the 0.25+ Models.Definitions / Models.Roles env shape. The bot token is +# NOT an env var: run.sh stores it with `netclaw secrets set Mattermost.BotToken`. +# run.sh sets Mattermost.CallbackUrl to http://netclawd:5299/api/mattermost/actions (a forwarder +# sidecar, because the daemon listens on loopback only) so approvals +# render as buttons. Mattermost refuses integration callbacks to private addresses, so it +# allow-lists the netclawd host below. services: mattermost: - image: mattermost/mattermost-preview:latest - ports: - - "8065:8065" + image: ${MATTERMOST_IMAGE:-mattermost/mattermost-preview:11.7.7} environment: - MM_SERVICESETTINGS_SITEURL: http://localhost:8065 + MM_SERVICESETTINGS_SITEURL: http://mattermost:8065 MM_TEAMSETTINGS_ENABLEOPENSERVER: "true" MM_SERVICESETTINGS_ENABLETESTING: "true" - # NOTE: EnableBotAccountCreation / EnableUserAccessTokens are intentionally - # NOT set here. bootstrap.mjs flips them via the System Console config API so - # the captured screenshot shows them as editable toggles (an env var would - # render them locked: "set through an environment variable"). + # Button callbacks go to the daemon on the private compose network; Mattermost blocks that + # unless the host is listed. + MM_SERVICESETTINGS_ALLOWEDUNTRUSTEDINTERNALCONNECTIONS: netclawd + # EnableBotAccountCreation / EnableUserAccessTokens are intentionally NOT set here. + # bootstrap.mjs flips them through the config API so the captured System Console + # page shows editable toggles (an env var would render them locked). + # No healthcheck: the 11.x preview image ships neither curl nor wget. bootstrap.mjs + # polls /api/v4/system/ping until the server is ready. + + netclawd: + image: ${NETCLAW_IMAGE:?set by run.sh} + depends_on: + - mattermost + environment: + NETCLAW_Daemon__DisableSelfUpdate: "true" + NETCLAW_Providers__inference__Type: ${NC_INFERENCE_TYPE:?set by run.sh} + NETCLAW_Providers__inference__Endpoint: ${NC_INFERENCE_ENDPOINT:?set by run.sh} + NETCLAW_Models__Definitions__main__Provider: inference + NETCLAW_Models__Definitions__main__ModelId: ${NC_INFERENCE_MODEL:?set by run.sh} + NETCLAW_Models__Roles__Main: main + NETCLAW_Models__Roles__Fallback: main + NETCLAW_Models__Roles__Compaction: main + # Real inference endpoints can be slow to first token. + NETCLAW_Session__TurnLlmTimeoutSeconds: "600" + NETCLAW_Session__FirstTokenTimeoutSeconds: "600" + NETCLAW_Session__PrefillTimeoutSeconds: "600" + volumes: + - netclaw-home:/home/netclaw/.netclaw healthcheck: - test: ["CMD", "curl", "-sf", "http://localhost:8065/api/v4/system/ping"] - interval: 5s + test: ["CMD", "curl", "-sf", "http://127.0.0.1:5199/api/health/ready"] + interval: 3s timeout: 5s - retries: 48 - start_period: 20s + retries: 40 + start_period: 5s + + # Shares netclawd's network namespace so it can reach the loopback-only daemon listener and + # re-expose it on :5299 for Mattermost's button callbacks. + callbackfwd: + image: netclaw-docs-playwright:${PLAYWRIGHT_VERSION:?set by run.sh} + network_mode: service:netclawd + user: "${HOST_UID:-1000}:${HOST_GID:-1000}" + command: ["node", "/work/tcp-forward.mjs"] + depends_on: + - netclawd + volumes: + - .:/work:ro + + # Only started by run.sh with SCRIPTED_APPROVAL=1: answers with one scripted shell_execute call (no real model). + scriptedllm: + profiles: ["scripted"] + image: netclaw-docs-playwright:${PLAYWRIGHT_VERSION:?set by run.sh} + user: "${HOST_UID:-1000}:${HOST_GID:-1000}" + command: ["node", "/work/scripted-llm.mjs"] + environment: + SCRIPTED_TOOL: ${SCRIPTED_TOOL:-mcp} + SCRIPTED_COMMAND: ${SCRIPTED_COMMAND:-git push origin main} + volumes: + - .:/work:ro + + # The stack's demo MCP server (screenshots/mcp-server); approval.sh-style runs register it + # with netclawd and gate http_fetch behind approval. + mcp-demo: + profiles: ["scripted"] + image: netclaw-docs-mcp-demo:1 + build: + context: ../mcp-server + + playwright: + profiles: ["tools"] + image: netclaw-docs-playwright:${PLAYWRIGHT_VERSION:?set by run.sh} + build: + context: ../playwright + args: + PLAYWRIGHT_VERSION: ${PLAYWRIGHT_VERSION} + # Run as the invoking user so files written to the mounts are yours, not root's. + user: "${HOST_UID:-1000}:${HOST_GID:-1000}" + environment: + BOOTSTRAP_FILE: /work/.bootstrap.${COMPOSE_PROJECT_NAME}.json + OUT_DIR: /out + volumes: + - .:/work + - ${CAPTURE_OUT_DIR:?set by run.sh}:/out + +volumes: + netclaw-home: diff --git a/screenshots/mattermost/frame-approval.mjs b/screenshots/mattermost/frame-approval.mjs new file mode 100644 index 0000000..2796afc --- /dev/null +++ b/screenshots/mattermost/frame-approval.mjs @@ -0,0 +1,34 @@ +#!/usr/bin/env node +// Frames a raw crop of the approval prompt the way the landing page shows it: a rounded dark +// card with padding and a soft purple glow on a near-black background (colours sampled from the +// original Slack-era image). Output is a PNG at 2x. +// +// Library: import { frame } from './frame-approval.mjs'; await frame(browser, rawPath, outPath, { bg }) +// CLI: node frame-approval.mjs raw.png out.png [cardBackgroundCss] +import { readFileSync } from 'fs'; +import { fileURLToPath } from 'url'; + +export async function frame(browser, rawPath, outPath, { cardBg = '#1f1f1f', dsf = 2 } = {}) { + const b64 = readFileSync(rawPath).toString('base64'); + const ctx = await browser.newContext({ deviceScaleFactor: dsf, viewport: { width: 1400, height: 900 } }); + const page = await ctx.newPage(); + await page.setContent(`
`); + await page.waitForFunction(() => document.getElementById('shot').complete); + // Show the raw capture at its CSS size (it was taken at the same scale factor). + await page.evaluate(d => { const i = document.getElementById('shot'); i.style.width = (i.naturalWidth / d) + 'px'; }, dsf); + await page.locator('#stage').screenshot({ path: outPath }); + await ctx.close(); +} + +if (process.argv[1] === fileURLToPath(import.meta.url)) { + const { chromium } = await import('playwright'); + const browser = await chromium.launch(); + await frame(browser, process.argv[2], process.argv[3], process.argv[4] ? { cardBg: process.argv[4] } : {}); + await browser.close(); +} diff --git a/screenshots/mattermost/post-message.mjs b/screenshots/mattermost/post-message.mjs new file mode 100644 index 0000000..c985103 --- /dev/null +++ b/screenshots/mattermost/post-message.mjs @@ -0,0 +1,44 @@ +#!/usr/bin/env node +// Posts an @netclaw message as the test user, then waits until the bot replies. +// Exits non-zero if no reply arrives in time (the daemon or inference is broken). +// Runs in the Playwright container (run.sh). +// +// Usage: node post-message.mjs ["message text"] +// Env: BOOTSTRAP_FILE (default ./.bootstrap.json), REPLY_TIMEOUT_SECONDS (default 300) + +import { readFileSync } from 'fs'; +import { dirname, join } from 'path'; +import { fileURLToPath } from 'url'; + +const __dirname = dirname(fileURLToPath(import.meta.url)); +const boot = JSON.parse(readFileSync(process.env.BOOTSTRAP_FILE || join(__dirname, '.bootstrap.json'), 'utf8')); +const MESSAGE = process.argv[2] || '@netclaw hey! In one sentence, what are you and what can you help with?'; +const TIMEOUT_MS = Number(process.env.REPLY_TIMEOUT_SECONDS || 300) * 1000; + +async function api(method, path, body, token) { + const res = await fetch(`${boot.serverUrl}${path}`, { + method, + headers: { 'Content-Type': 'application/json', ...(token ? { Authorization: `Bearer ${token}` } : {}) }, + body: body ? JSON.stringify(body) : undefined, + }); + if (!res.ok) throw new Error(`${method} ${path} -> ${res.status}: ${(await res.text()).slice(0, 200)}`); + return res; +} + +const login = await api('POST', '/api/v4/users/login', { login_id: boot.testuser.username, password: boot.testuser.password }); +const userToken = login.headers.get('Token'); +await api('POST', '/api/v4/posts', { channel_id: boot.channel.id, message: MESSAGE }, userToken); +console.log(`posted: ${MESSAGE}`); + +const start = Date.now(); +while (Date.now() - start < TIMEOUT_MS) { + const data = await (await api('GET', `/api/v4/channels/${boot.channel.id}/posts`, null, boot.bot.token)).json(); + const reply = data.order.map(id => data.posts[id]).find(p => p.user_id === boot.bot.userId && p.root_id && p.message); + if (reply) { + console.log(`bot replied after ${Math.round((Date.now() - start) / 1000)}s: ${reply.message.slice(0, 200).replace(/\s+/g, ' ')}`); + process.exit(0); + } + await new Promise(r => setTimeout(r, 3000)); +} +console.error(`no bot reply within ${TIMEOUT_MS / 1000}s`); +process.exit(1); diff --git a/screenshots/mattermost/run.sh b/screenshots/mattermost/run.sh index e7c8d89..3a89804 100755 --- a/screenshots/mattermost/run.sh +++ b/screenshots/mattermost/run.sh @@ -1,83 +1,137 @@ #!/bin/bash -# Orchestrates the Mattermost screenshot capture end to end: -# 1. brings up the Mattermost container -# 2. bootstraps it (admin/team/bot/token/channel/test user) -> .bootstrap.json -# 3. starts the pinned netclaw daemon against it, with inference pointed at -# an OpenAI-compatible endpoint (spark2 by default) -# 4. posts an @netclaw message as the test user so the bot replies -# 5. drives Playwright (capture.mjs) to shoot the System Console + live thread -# 6. tears everything down +# Regenerate the Mattermost screenshots, entirely in containers (Docker is the only +# prerequisite): Mattermost, the released netclaw daemon and Playwright share one +# compose network. Needs real inference so the bot can answer: # -# The daemon uses `--network host` so its loopback control plane is reachable -# from the host, and Mattermost is reached at http://localhost:8065. +# INFERENCE_BASE_URL=http://llm.example.internal:8080 INFERENCE_MODEL_ID= \ +# ./screenshots/mattermost/run.sh # -# Overridable env: -# NETCLAW_IMAGE default ghcr.io/netclaw-dev/netclaw:0.22.1 -# SPARK_ENDPOINT OpenAI-compatible inference endpoint (default spark2) -# SPARK_MODEL_ID model id served there (matches the local netclaw instance) +# Landing-page approval prompt (public/assets/approval-prompt.png) needs no model: set +# SCRIPTED_APPROVAL=1 and a scripted endpoint (scripted-llm.mjs) requests one demo-utilities http_fetch call +# (SCRIPTED_TOOL=shell for a shell_execute call instead); +# the run then captures the prompt with its buttons, framed by frame-approval.mjs, into OUT_DIR. +# +# (or put INFERENCE_UPSTREAM and INFERENCE_UPSTREAM_MODEL in screenshots/.env, gitignored). Other knobs: +# COMPOSE_PROJECT_NAME prefix for this run; "-mm" is appended (default netclaw-shots-mm) +# NETCLAW_VERSION / NETCLAW_IMAGE daemon image (default: src/version.json) +# OUT_DIR where PNGs go (default screenshots/output) +# REPLY_TIMEOUT_SECONDS how long to wait for the bot's reply (default 300) +# +# Steps: compose up Mattermost -> bootstrap (admin, team, bot, channel, test user) -> +# start netclawd -> store the bot token as an encrypted secret and enable the Mattermost +# channel -> post an @netclaw message as the test user and wait for the reply -> +# Playwright captures -> tear everything down. Exits non-zero on any failure. set -euo pipefail -SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" -cd "$SCRIPT_DIR" +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +COMPOSE_PROJECT_NAME="${COMPOSE_PROJECT_NAME:-netclaw-shots}-mm" +export COMPOSE_PROJECT_NAME +# shellcheck source=../lib.sh +. "$HERE/../lib.sh" + +if [ -n "${SCRIPTED_APPROVAL:-}" ]; then + export NC_INFERENCE_TYPE=openai-compatible NC_INFERENCE_ENDPOINT=http://scriptedllm:8080/v1 NC_INFERENCE_MODEL=demo-model +elif [ "$INFERENCE_MODE" = relay ]; then + # This flow runs on its own compose network without the forwarder, and no frame shows the + # daemon's configuration, so the daemon talks to the upstream directly. + export NC_INFERENCE_ENDPOINT="$INFERENCE_UPSTREAM" NC_INFERENCE_MODEL="$INFERENCE_UPSTREAM_MODEL" +elif [ "$INFERENCE_MODE" != external ]; then + echo "mattermost: set INFERENCE_UPSTREAM (or INFERENCE_BASE_URL and INFERENCE_MODEL_ID); the bot needs a real model to answer" >&2 + exit 2 +fi -NETCLAW_IMAGE="${NETCLAW_IMAGE:-ghcr.io/netclaw-dev/netclaw:0.22.1}" -SPARK_ENDPOINT="${SPARK_ENDPOINT:-https://spark2.testlab.petabridge.net}" -SPARK_MODEL_ID="${SPARK_MODEL_ID:-Qwen/Qwen3.6-35B-A3B-FP8}" -SERVER="http://localhost:8065" +OUT_DIR="${OUT_DIR:-$SHOTS_DIR/output}" +mkdir -p "$OUT_DIR" +export CAPTURE_OUT_DIR="$(cd "$OUT_DIR" && pwd)" +BOOTSTRAP=".bootstrap.${COMPOSE_PROJECT_NAME}.json" + +mm() { docker compose -f "$HERE/docker-compose.mattermost.yml" "$@"; } +pw() { mm run --rm --no-deps -T playwright "$@"; } +nc() { mm exec -T -u netclaw netclawd netclaw "$@"; } cleanup() { - echo "==> Tearing down" - docker rm -f netclaw-mm-shot >/dev/null 2>&1 || true - docker compose -f docker-compose.mattermost.yml down -v >/dev/null 2>&1 || true - rm -f .bootstrap.json + if [ -n "${KEEP_STACK:-}" ]; then echo "==> KEEP_STACK set: leaving project $COMPOSE_PROJECT_NAME running"; return; fi + echo "==> Tearing down project $COMPOSE_PROJECT_NAME" + mm --profile scripted --profile tools down -v --remove-orphans >/dev/null 2>&1 || true + rm -f "$HERE/$BOOTSTRAP" } trap cleanup EXIT -echo "==> Bringing up Mattermost" -docker compose -f docker-compose.mattermost.yml up -d +echo "==> Bringing up Mattermost (${MATTERMOST_IMAGE:-mattermost/mattermost-preview:11.7.7}) and building the Playwright image ($PLAYWRIGHT_VERSION)" +mm up -d mattermost +mm build playwright +if [ -n "${SCRIPTED_APPROVAL:-}" ]; then + export SCRIPTED_TOOL="${SCRIPTED_TOOL:-mcp}" # mcp (default) or shell + mm --profile scripted up -d --build scriptedllm mcp-demo +fi echo "==> Bootstrapping Mattermost" -node bootstrap.mjs "$SERVER" +pw node /work/bootstrap.mjs http://mattermost:8065 + +BOT_TOKEN="$(pw node -p "require('/work/$BOOTSTRAP').bot.token")" +CHANNEL_ID="$(pw node -p "require('/work/$BOOTSTRAP').channel.id")" -BOT_TOKEN="$(node -e "process.stdout.write(require('./.bootstrap.json').bot.token)")" -CHANNEL_ID="$(node -e "process.stdout.write(require('./.bootstrap.json').channel.id)")" +echo "==> Starting netclawd ($NETCLAW_IMAGE), models via NETCLAW_Models__Definitions/Roles env" +mm up -d --wait netclawd +mm up -d callbackfwd -echo "==> Starting netclaw ($NETCLAW_IMAGE) against Mattermost + $SPARK_ENDPOINT" -docker rm -f netclaw-mm-shot >/dev/null 2>&1 || true -docker run -d --name netclaw-mm-shot --network host \ - -e NETCLAW_Daemon__Host=127.0.0.1 -e NETCLAW_Daemon__Port=5299 \ - -e NETCLAW_Daemon__ExposureMode=local -e NETCLAW_Daemon__DisableSelfUpdate=true \ - -e NETCLAW_Session__TurnLlmTimeoutSeconds=600 \ - -e NETCLAW_Session__FirstTokenTimeoutSeconds=600 \ - -e NETCLAW_Session__PrefillTimeoutSeconds=600 \ - -e NETCLAW_Mattermost__Enabled=true \ - -e NETCLAW_Mattermost__ServerUrl="$SERVER" \ - -e NETCLAW_Mattermost__BotToken="$BOT_TOKEN" \ - -e NETCLAW_Mattermost__DefaultChannelId="$CHANNEL_ID" \ - -e NETCLAW_Mattermost__MentionOnly=true \ - -e "NETCLAW_Mattermost__ChannelAudiences__${CHANNEL_ID}=team" \ - -e NETCLAW_Providers__spark__Type=openai-compatible \ - -e NETCLAW_Providers__spark__Endpoint="$SPARK_ENDPOINT" \ - -e NETCLAW_Models__Main__Provider=spark -e NETCLAW_Models__Main__ModelId="$SPARK_MODEL_ID" \ - -e NETCLAW_Models__Fallback__Provider=spark -e NETCLAW_Models__Fallback__ModelId="$SPARK_MODEL_ID" \ - -e NETCLAW_Models__Compaction__Provider=spark -e NETCLAW_Models__Compaction__ModelId="$SPARK_MODEL_ID" \ - "$NETCLAW_IMAGE" +echo "==> Storing the bot token as an encrypted secret and enabling the Mattermost channel" +nc secrets set Mattermost.BotToken "$BOT_TOKEN" >/dev/null +# The scripted approval run needs shell_execute, which the "team" audience doesn't get. +AUDIENCE="${CHANNEL_AUDIENCE:-$([ -n "${SCRIPTED_APPROVAL:-}" ] && echo personal || echo team)}" +mm exec -T -u netclaw -e SERVER=http://mattermost:8065 -e CH="$CHANNEL_ID" -e AUD="$AUDIENCE" netclawd sh -c ' + f=/home/netclaw/.netclaw/config/netclaw.json + [ -f "$f" ] || echo "{}" > "$f" + jq --arg s "$SERVER" --arg c "$CH" ".configVersion = 1 + | .Security = {DeploymentPosture: \"Personal\", ShellExecutionMode: \"HostAllowed\", StrictDefaults: true} + | .Mattermost = {Enabled: true, ServerUrl: \$s, CallbackUrl: \"http://netclawd:5299/api/mattermost/actions\", DefaultChannelId: \$c, MentionOnly: true, ChannelAudiences: {(\$c): \$a}}" --arg a "$AUD" \ + "$f" > "$f.tmp" && mv "$f.tmp" "$f"' -echo "==> Waiting for daemon health" -for i in $(seq 1 30); do - curl -sf http://127.0.0.1:5299/api/health/ready >/dev/null 2>&1 && break +echo "==> Waiting for the Mattermost channel to connect" +connected=0 +for _ in $(seq 1 45); do + if mm exec -T -u netclaw netclawd sh -c 'cat /home/netclaw/.netclaw/logs/daemon-*.log' 2>/dev/null \ + | grep -i 'mattermost' | grep -iE 'connected|started' | tail -n 3 | grep -q .; then connected=1; break; fi sleep 2 done +if [ "$connected" -ne 1 ]; then + echo "mattermost: netclawd never reported a Mattermost connection" >&2 + mm exec -T -u netclaw netclawd sh -c 'grep -i mattermost /home/netclaw/.netclaw/logs/daemon-*.log | tail -n 20' >&2 || true + exit 1 +fi +mm exec -T -u netclaw netclawd sh -c 'grep -i mattermost /home/netclaw/.netclaw/logs/daemon-*.log | tail -n 5' + +if [ -n "${SCRIPTED_APPROVAL:-}" ] && [ "$SCRIPTED_TOOL" = mcp ]; then + echo "==> Registering the demo MCP server and gating its tools behind approval" + nc mcp add --transport http demo-utilities http://mcp-demo:8080/mcp >/dev/null + mm exec -T -u netclaw netclawd sh -c ' + f=/home/netclaw/.netclaw/config/netclaw.json + jq ".Tools.AudienceProfiles.Personal.ApprovalPolicy.McpServerDefaults[\"demo-utilities\"] = \"Approval\"" "$f" > "$f.tmp" && mv "$f.tmp" "$f"' + sleep 12 # the daemon restarts itself when netclaw.json changes + for _ in $(seq 1 30); do nc status 2>/dev/null | grep -q 'mcp:demo-utilities: healthy' && break; sleep 2; done +fi +if [ -n "${SCRIPTED_APPROVAL:-}" ]; then + # A Git repository for the shell call to run in, so the prompt looks like real project work. + mm exec -T -u netclaw netclawd sh -c ' + mkdir -p ~/projects/website && cd ~/projects/website && git init -q -b main . \ + && git config user.name alice && git config user.email alice@example.com \ + && echo "# website" > README.md && git add . && git commit -q -m "Initial commit" \ + && git remote add origin git@example.com:acme/website.git' + echo "==> Capturing the approval prompt" + pw node /work/approval.mjs || { + echo "==> approval capture failed; scripted endpoint and daemon logs follow" >&2 + mm logs --tail 20 scriptedllm >&2 || true + mm exec -T -u netclaw netclawd sh -c 'tail -n 40 /home/netclaw/.netclaw/logs/daemon-*.log' >&2 || true + exit 1 + } + echo "==> Done. approval-prompt.png written to $CAPTURE_OUT_DIR" + exit 0 +fi -echo "==> Posting an @netclaw message as the test user" -TOKEN="$(curl -s -i -X POST "$SERVER/api/v4/users/login" -H 'Content-Type: application/json' \ - -d '{"login_id":"testuser","password":"TestUser1234!"}' | grep -i '^token:' | awk '{print $2}' | tr -d '\r')" -curl -s -X POST "$SERVER/api/v4/posts" -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \ - -d "{\"channel_id\":\"$CHANNEL_ID\",\"message\":\"@netclaw hey! In one sentence, what are you and what can you help with?\"}" >/dev/null -echo " posted; waiting for the bot to reply" -sleep 25 +echo "==> Posting an @netclaw message and waiting for the reply" +pw node /work/post-message.mjs echo "==> Capturing screenshots (Playwright)" -node capture.mjs +pw node /work/capture.mjs -echo "==> Done. Shots written to ../output/mattermost-*.png" +echo "==> Done. Shots written to $CAPTURE_OUT_DIR/mattermost-*.png" diff --git a/screenshots/mattermost/scripted-llm.mjs b/screenshots/mattermost/scripted-llm.mjs new file mode 100644 index 0000000..550765d --- /dev/null +++ b/screenshots/mattermost/scripted-llm.mjs @@ -0,0 +1,59 @@ +#!/usr/bin/env node +// A tiny OpenAI-compatible endpoint that answers every first turn with one scripted +// shell_execute tool call, so the approval prompt shows up without a real model. +// Runs in the Playwright image (approval.sh). After the tool result it replies "Done." +// +// Env: PORT (8080), SCRIPTED_TOOL (mcp|shell, default mcp), SCRIPTED_COMMAND (shell command), +// SCRIPTED_MCP_URL (URL the MCP call fetches) +import { createServer } from 'http'; + +const COMMAND = process.env.SCRIPTED_COMMAND || 'git push origin main'; +const KIND = process.env.SCRIPTED_TOOL || 'mcp'; +const MCP_URL = process.env.SCRIPTED_MCP_URL || 'https://status.example.com/api/v1/incidents'; +const PORT = Number(process.env.PORT || 8080); +const id = () => 'chatcmpl-' + Math.random().toString(36).slice(2, 10); + +function completion(body) { + const tools = body.tools || []; + const shell = KIND === 'shell' && tools.find(t => (t.function?.name || t.name) === 'shell_execute'); + const mcp = KIND === 'mcp' && tools.find(t => /http_fetch/.test(t.function?.name || t.name || '')); + const answered = (body.messages || []).some(m => m.role === 'tool'); + const calledMcp = (body.messages || []).some(m => (m.tool_calls || []).some(c => /http_fetch/.test(c.function?.name || ''))); + const loader = KIND === 'mcp' && !mcp && !answered && tools.find(t => (t.function?.name || t.name) === 'load_tool'); + if (loader) { + // MCP tools are deferred: the first turn loads the tool, the next one calls it. + return { message: { role: 'assistant', content: null, tool_calls: [{ id: 'call_ap0', type: 'function', function: { name: 'load_tool', arguments: JSON.stringify({ name: 'demo-utilities/http_fetch', _rationale: 'Load the HTTP fetch tool.' }) } }] }, finish: 'tool_calls' }; + } + if (mcp && !calledMcp) { + const name = mcp.function?.name || mcp.name; + return { message: { role: 'assistant', content: null, tool_calls: [{ id: 'call_ap1', type: 'function', function: { name, arguments: JSON.stringify({ url: MCP_URL, method: 'POST', _rationale: 'Open a status page incident for the website deploy.' }) } }] }, finish: 'tool_calls' }; + } + if (shell && !answered) { + return { message: { role: 'assistant', content: null, tool_calls: [{ id: 'call_ap1', type: 'function', function: { name: 'shell_execute', arguments: JSON.stringify({ command: COMMAND, _rationale: 'Push the pending website changes to origin.' }) } }] }, finish: 'tool_calls' }; + } + return { message: { role: 'assistant', content: 'Done.' }, finish: 'stop' }; +} + +createServer((req, res) => { + let raw = ''; + req.on('data', c => (raw += c)); + req.on('end', () => { + if (req.method === 'GET') { res.setHeader('Content-Type', 'application/json'); return res.end(JSON.stringify({ object: 'list', data: [{ id: 'demo-model', object: 'model' }] })); } + let body = {}; + try { body = JSON.parse(raw || '{}'); } catch {} + const { message, finish } = completion(body); + console.log(`${req.method} ${req.url} stream=${!!body.stream} -> ${finish}`); + const base = { id: id(), object: 'chat.completion', created: Math.floor(Date.now() / 1000), model: 'demo-model' }; + if (body.stream) { + res.writeHead(200, { 'Content-Type': 'text/event-stream' }); + const delta = { ...message }; + if (delta.tool_calls) delta.tool_calls = delta.tool_calls.map((t, index) => ({ index, ...t })); + res.write(`data: ${JSON.stringify({ ...base, object: 'chat.completion.chunk', choices: [{ index: 0, delta, finish_reason: null }] })}\n\n`); + res.write(`data: ${JSON.stringify({ ...base, object: 'chat.completion.chunk', choices: [{ index: 0, delta: {}, finish_reason: finish }], usage: { prompt_tokens: 10, completion_tokens: 10, total_tokens: 20 } })}\n\n`); + res.end('data: [DONE]\n\n'); + } else { + res.setHeader('Content-Type', 'application/json'); + res.end(JSON.stringify({ ...base, choices: [{ index: 0, message, finish_reason: finish }], usage: { prompt_tokens: 10, completion_tokens: 10, total_tokens: 20 } })); + } + }); +}).listen(PORT, () => console.log(`scripted llm on :${PORT}`)); diff --git a/screenshots/mattermost/tcp-forward.mjs b/screenshots/mattermost/tcp-forward.mjs new file mode 100644 index 0000000..491e148 --- /dev/null +++ b/screenshots/mattermost/tcp-forward.mjs @@ -0,0 +1,14 @@ +#!/usr/bin/env node +// Forwards 0.0.0.0:LISTEN to 127.0.0.1:TARGET inside the netclawd network namespace. +// The daemon's default `local` exposure mode binds loopback only, so Mattermost (another +// container) can't reach /api/mattermost/actions directly. This sidecar is the "forwarder" +// the Mattermost channel docs describe; the daemon still sees loopback traffic. +// Env: LISTEN (5299), TARGET (5199) +import net from 'net'; +const LISTEN = Number(process.env.LISTEN || 5299), TARGET = Number(process.env.TARGET || 5199); +net.createServer(c => { + const u = net.connect(TARGET, '127.0.0.1'); + c.pipe(u).pipe(c); + const end = () => { c.destroy(); u.destroy(); }; + c.on('error', end); u.on('error', end); +}).listen(LISTEN, '0.0.0.0', () => console.log(`forwarding :${LISTEN} -> 127.0.0.1:${TARGET}`)); diff --git a/screenshots/mcp-server/Dockerfile b/screenshots/mcp-server/Dockerfile index 76ae002..e60adb3 100644 --- a/screenshots/mcp-server/Dockerfile +++ b/screenshots/mcp-server/Dockerfile @@ -1,6 +1,7 @@ FROM python:3.12-slim WORKDIR /app -RUN pip install --no-cache-dir "mcp[cli]" +# server.py uses the 1.x FastMCP API; mcp 2.x renamed it. +RUN pip install --no-cache-dir "mcp[cli]<2" COPY server.py . EXPOSE 8080 CMD ["python", "server.py"] diff --git a/screenshots/mcp-server/Dockerfile.oauth b/screenshots/mcp-server/Dockerfile.oauth new file mode 100644 index 0000000..60c30f0 --- /dev/null +++ b/screenshots/mcp-server/Dockerfile.oauth @@ -0,0 +1,6 @@ +FROM python:3.12-slim +WORKDIR /app +RUN pip install --no-cache-dir "mcp[cli]<2" +COPY oauth_server.py . +EXPOSE 8080 +CMD ["python", "oauth_server.py"] diff --git a/screenshots/mcp-server/oauth_server.py b/screenshots/mcp-server/oauth_server.py new file mode 100644 index 0000000..64fc900 --- /dev/null +++ b/screenshots/mcp-server/oauth_server.py @@ -0,0 +1,246 @@ +"""OAuth-protected demo MCP server for testing `netclaw mcp auth`. + +One process plays both roles: the MCP resource server at /mcp and a minimal +OAuth 2.1 authorization server (authorization code + PKCE, refresh tokens with +rotation). The authorize endpoint auto-approves, so a script can follow the +redirect with curl instead of a browser. + +MODE (environment variable): + confidential Pre-registered client only. No dynamic registration. The token + endpoint rejects a wrong or missing client secret. + public Dynamic client registration (RFC 7591), public client, PKCE only. + anonymous Like public, but initialize and tools/list work without a token. + Only tools/call returns 401 (models servers that allow anonymous + catalog discovery). + +Other settings: BASE_URL (public URL of this server), CLIENT_ID, CLIENT_SECRET, +TOKEN_TTL (access token lifetime in seconds, default 3600). +""" + +import base64 +import hashlib +import json +import os +import secrets +import time +from urllib.parse import urlencode + +import uvicorn +from mcp.server.fastmcp import FastMCP +from starlette.requests import Request +from starlette.responses import JSONResponse, RedirectResponse, Response + +MODE = os.environ.get("MODE", "confidential") +BASE_URL = os.environ.get("BASE_URL", "http://localhost:8080").rstrip("/") +CLIENT_ID = os.environ.get("CLIENT_ID", "demo-client") +CLIENT_SECRET = os.environ.get("CLIENT_SECRET", "demo-secret") +TOKEN_TTL = int(os.environ.get("TOKEN_TTL", "3600")) + +mcp = FastMCP("oauth-demo", stateless_http=True, host="0.0.0.0", port=8080) + + +@mcp.tool() +def whoami() -> str: + """Return the identity this server associates with the access token.""" + return "demo-user" + + +@mcp.tool() +def generate_uuid() -> str: + """Generate a random UUID v4.""" + import uuid + return str(uuid.uuid4()) + + +@mcp.prompt() +def review_checklist(topic: str) -> str: + """Walk through a short review checklist for a topic.""" + return f"Review {topic}: list the risks, the owner of each risk, and the next action." + + +codes: dict[str, dict] = {} +access_tokens: dict[str, float] = {} +refresh_tokens: dict[str, dict] = {} +dynamic_clients: dict[str, dict] = {} + + +def log(msg: str) -> None: + print(f"[oauth-demo] {msg}", flush=True) + + +def metadata_url() -> str: + return f"{BASE_URL}/.well-known/oauth-protected-resource/mcp" + + +@mcp.custom_route("/.well-known/oauth-protected-resource/mcp", ["GET"]) +@mcp.custom_route("/.well-known/oauth-protected-resource", ["GET"]) +async def protected_resource(_: Request) -> Response: + return JSONResponse({ + "resource": f"{BASE_URL}/mcp", + "authorization_servers": [BASE_URL], + "bearer_methods_supported": ["header"], + }) + + +@mcp.custom_route("/.well-known/oauth-authorization-server", ["GET"]) +async def as_metadata(_: Request) -> Response: + doc = { + "issuer": BASE_URL, + "authorization_endpoint": f"{BASE_URL}/authorize", + "token_endpoint": f"{BASE_URL}/token", + "response_types_supported": ["code"], + "grant_types_supported": ["authorization_code", "refresh_token"], + "code_challenge_methods_supported": ["S256"], + "token_endpoint_auth_methods_supported": + ["client_secret_post", "client_secret_basic"] if MODE == "confidential" else ["none"], + } + if MODE != "confidential": + doc["registration_endpoint"] = f"{BASE_URL}/register" + return JSONResponse(doc) + + +@mcp.custom_route("/register", ["POST"]) +async def register(request: Request) -> Response: + if MODE == "confidential": + return JSONResponse({"error": "registration_not_supported"}, status_code=404) + body = await request.json() + client_id = "dyn-" + secrets.token_hex(6) + dynamic_clients[client_id] = {"redirect_uris": body.get("redirect_uris", [])} + log(f"registered dynamic client {client_id}") + return JSONResponse({ + "client_id": client_id, + "redirect_uris": body.get("redirect_uris", []), + "token_endpoint_auth_method": "none", + "grant_types": ["authorization_code", "refresh_token"], + "response_types": ["code"], + }, status_code=201) + + +def known_client(client_id: str, redirect_uri: str) -> bool: + if MODE == "confidential": + return client_id == CLIENT_ID and redirect_uri.endswith("/api/mcp/oauth/callback") + return client_id in dynamic_clients and redirect_uri in dynamic_clients[client_id]["redirect_uris"] + + +@mcp.custom_route("/authorize", ["GET"]) +async def authorize(request: Request) -> Response: + q = request.query_params + client_id, redirect_uri = q.get("client_id", ""), q.get("redirect_uri", "") + if not known_client(client_id, redirect_uri): + log(f"authorize rejected: unknown client {client_id} or redirect {redirect_uri}") + return JSONResponse({"error": "invalid_request", + "error_description": "unknown client or redirect_uri"}, status_code=400) + if not q.get("code_challenge"): + return JSONResponse({"error": "invalid_request", "error_description": "PKCE required"}, status_code=400) + code = secrets.token_urlsafe(16) + codes[code] = {"client_id": client_id, "redirect_uri": redirect_uri, + "challenge": q["code_challenge"], "scope": q.get("scope", "")} + log(f"authorize ok for {client_id}, scope='{q.get('scope', '')}'") + params = {"code": code} + if q.get("state"): + params["state"] = q["state"] + return RedirectResponse(f"{redirect_uri}?{urlencode(params)}", status_code=302) + + +def client_credentials(request: Request, form: dict) -> tuple[str, str | None]: + auth = request.headers.get("authorization", "") + if auth.lower().startswith("basic "): + raw = base64.b64decode(auth[6:]).decode() + cid, _, sec = raw.partition(":") + return cid, sec + return form.get("client_id", ""), form.get("client_secret") + + +@mcp.custom_route("/token", ["POST"]) +async def token(request: Request) -> Response: + form = dict((await request.form()).items()) + cid, sec = client_credentials(request, form) + grant = form.get("grant_type") + log(f"token request grant={grant} client={cid} secret_presented={sec is not None}") + if MODE == "confidential" and (cid != CLIENT_ID or sec != CLIENT_SECRET): + return JSONResponse({"error": "invalid_client", + "error_description": "client authentication failed"}, status_code=401) + if grant == "authorization_code": + info = codes.pop(form.get("code", ""), None) + if info is None or info["client_id"] != cid: + return JSONResponse({"error": "invalid_grant"}, status_code=400) + verifier = form.get("code_verifier", "") + digest = base64.urlsafe_b64encode(hashlib.sha256(verifier.encode()).digest()).rstrip(b"=").decode() + if digest != info["challenge"]: + return JSONResponse({"error": "invalid_grant", "error_description": "PKCE mismatch"}, status_code=400) + return issue(cid, info["scope"]) + if grant == "refresh_token": + info = refresh_tokens.pop(form.get("refresh_token", ""), None) + if info is None or info["client_id"] != cid: + log("refresh rejected: unknown or already-used refresh token") + return JSONResponse({"error": "invalid_grant"}, status_code=400) + return issue(cid, info["scope"]) + return JSONResponse({"error": "unsupported_grant_type"}, status_code=400) + + +def issue(client_id: str, scope: str) -> Response: + access = "at-" + secrets.token_urlsafe(16) + refresh = "rt-" + secrets.token_urlsafe(16) + access_tokens[access] = time.time() + TOKEN_TTL + refresh_tokens[refresh] = {"client_id": client_id, "scope": scope} + log(f"issued access token (ttl {TOKEN_TTL}s) and refresh token") + return JSONResponse({"access_token": access, "token_type": "Bearer", "expires_in": TOKEN_TTL, + "refresh_token": refresh, "scope": scope}) + + +def valid_bearer(headers: dict) -> bool: + auth = headers.get(b"authorization", b"").decode() + if not auth.lower().startswith("bearer "): + return False + expires = access_tokens.get(auth[7:]) + return expires is not None and expires > time.time() + + +class Gate: + """Return 401 with an RFC 9728 challenge when /mcp has no valid bearer token.""" + + def __init__(self, app): + self.app = app + + async def __call__(self, scope, receive, send): + if scope["type"] != "http" or scope["path"].rstrip("/") != "/mcp": + return await self.app(scope, receive, send) + + headers = dict(scope["headers"]) + body = b"" + while True: + message = await receive() + body += message.get("body", b"") + if not message.get("more_body"): + break + + methods = [] + try: + parsed = json.loads(body) if body else {} + methods = [m.get("method") for m in (parsed if isinstance(parsed, list) else [parsed])] + except ValueError: + pass + + needs_token = MODE != "anonymous" or "tools/call" in methods + ok = valid_bearer(headers) + log(f"/mcp {scope['method']} methods={methods} bearer_valid={ok}") + if needs_token and not ok: + resp = Response(status_code=401, headers={ + "WWW-Authenticate": f'Bearer resource_metadata="{metadata_url()}"'}) + return await resp(scope, receive, send) + + replayed = False + + async def replay(): + nonlocal replayed + if not replayed: + replayed = True + return {"type": "http.request", "body": body, "more_body": False} + return await receive() + + return await self.app(scope, replay, send) + + +if __name__ == "__main__": + log(f"mode={MODE} base_url={BASE_URL}") + uvicorn.run(Gate(mcp.streamable_http_app()), host="0.0.0.0", port=8080, log_level="warning") diff --git a/screenshots/output/approval-prompt-tui.png b/screenshots/output/approval-prompt-tui.png new file mode 100644 index 0000000..ef76f7a Binary files /dev/null and b/screenshots/output/approval-prompt-tui.png differ diff --git a/screenshots/output/approvals-list-json.png b/screenshots/output/approvals-list-json.png deleted file mode 100644 index 353b85f..0000000 Binary files a/screenshots/output/approvals-list-json.png and /dev/null differ diff --git a/screenshots/output/approvals-list.png b/screenshots/output/approvals-list.png index 0550946..b02ab08 100644 Binary files a/screenshots/output/approvals-list.png and b/screenshots/output/approvals-list.png differ diff --git a/screenshots/output/approvals-tui-revoke-confirm.png b/screenshots/output/approvals-tui-revoke-confirm.png index 1939940..8041170 100644 Binary files a/screenshots/output/approvals-tui-revoke-confirm.png and b/screenshots/output/approvals-tui-revoke-confirm.png differ diff --git a/screenshots/output/approvals-tui-revoke-done.png b/screenshots/output/approvals-tui-revoke-done.png deleted file mode 100644 index 578b4f3..0000000 Binary files a/screenshots/output/approvals-tui-revoke-done.png and /dev/null differ diff --git a/screenshots/output/approvals-tui-selected.png b/screenshots/output/approvals-tui-selected.png deleted file mode 100644 index b2c395f..0000000 Binary files a/screenshots/output/approvals-tui-selected.png and /dev/null differ diff --git a/screenshots/output/approvals-tui.png b/screenshots/output/approvals-tui.png index 72251a9..5211720 100644 Binary files a/screenshots/output/approvals-tui.png and b/screenshots/output/approvals-tui.png differ diff --git a/screenshots/output/approvals.gif b/screenshots/output/approvals.gif deleted file mode 100644 index 0ba76a8..0000000 Binary files a/screenshots/output/approvals.gif and /dev/null differ diff --git a/screenshots/output/chat-approval-prompt.png b/screenshots/output/chat-approval-prompt.png new file mode 100644 index 0000000..1595e7d Binary files /dev/null and b/screenshots/output/chat-approval-prompt.png differ diff --git a/screenshots/output/chat-first-greeting.png b/screenshots/output/chat-first-greeting.png new file mode 100644 index 0000000..a1bfa3f Binary files /dev/null and b/screenshots/output/chat-first-greeting.png differ diff --git a/screenshots/output/chat-intro-greeting.png b/screenshots/output/chat-intro-greeting.png deleted file mode 100644 index a4502ad..0000000 Binary files a/screenshots/output/chat-intro-greeting.png and /dev/null differ diff --git a/screenshots/output/chat-profile-questions.png b/screenshots/output/chat-profile-questions.png deleted file mode 100644 index 8023a5c..0000000 Binary files a/screenshots/output/chat-profile-questions.png and /dev/null differ diff --git a/screenshots/output/chat-profile-update-soul.md.png b/screenshots/output/chat-profile-update-soul.md.png deleted file mode 100644 index 456808d..0000000 Binary files a/screenshots/output/chat-profile-update-soul.md.png and /dev/null differ diff --git a/screenshots/output/chat-tui-approval-prompt.png b/screenshots/output/chat-tui-approval-prompt.png new file mode 100644 index 0000000..fb37f65 Binary files /dev/null and b/screenshots/output/chat-tui-approval-prompt.png differ diff --git a/screenshots/output/chat-tui-idle.png b/screenshots/output/chat-tui-idle.png new file mode 100644 index 0000000..5362069 Binary files /dev/null and b/screenshots/output/chat-tui-idle.png differ diff --git a/screenshots/output/config-browser.png b/screenshots/output/config-browser.png deleted file mode 100644 index 4e01817..0000000 Binary files a/screenshots/output/config-browser.png and /dev/null differ diff --git a/screenshots/output/config-channel-audiences.png b/screenshots/output/config-channel-audiences.png new file mode 100644 index 0000000..bac5c60 Binary files /dev/null and b/screenshots/output/config-channel-audiences.png differ diff --git a/screenshots/output/config-channels-menu.png b/screenshots/output/config-channels-menu.png index 1d8cd98..9c65512 100644 Binary files a/screenshots/output/config-channels-menu.png and b/screenshots/output/config-channels-menu.png differ diff --git a/screenshots/output/config-dashboard.png b/screenshots/output/config-dashboard.png index 12a0e6f..9e554a5 100644 Binary files a/screenshots/output/config-dashboard.png and b/screenshots/output/config-dashboard.png differ diff --git a/screenshots/output/config-exposure.png b/screenshots/output/config-exposure.png index fd89799..634f83f 100644 Binary files a/screenshots/output/config-exposure.png and b/screenshots/output/config-exposure.png differ diff --git a/screenshots/output/config-inbound-webhooks.png b/screenshots/output/config-inbound-webhooks.png index b76c2c8..bfc84e2 100644 Binary files a/screenshots/output/config-inbound-webhooks.png and b/screenshots/output/config-inbound-webhooks.png differ diff --git a/screenshots/output/config-retention.png b/screenshots/output/config-retention.png new file mode 100644 index 0000000..15921a0 Binary files /dev/null and b/screenshots/output/config-retention.png differ diff --git a/screenshots/output/config-search.png b/screenshots/output/config-search.png index 1699a37..9034a80 100644 Binary files a/screenshots/output/config-search.png and b/screenshots/output/config-search.png differ diff --git a/screenshots/output/config-skills.png b/screenshots/output/config-skills.png index 3831821..268f782 100644 Binary files a/screenshots/output/config-skills.png and b/screenshots/output/config-skills.png differ diff --git a/screenshots/output/config-telemetry.png b/screenshots/output/config-telemetry.png deleted file mode 100644 index c4b1d77..0000000 Binary files a/screenshots/output/config-telemetry.png and /dev/null differ diff --git a/screenshots/output/config-workspaces.png b/screenshots/output/config-workspaces.png deleted file mode 100644 index 4f4502e..0000000 Binary files a/screenshots/output/config-workspaces.png and /dev/null differ diff --git a/screenshots/output/daemon-pair.png b/screenshots/output/daemon-pair.png new file mode 100644 index 0000000..0286824 Binary files /dev/null and b/screenshots/output/daemon-pair.png differ diff --git a/screenshots/output/discord-setup-applications.png b/screenshots/output/discord-setup-applications.png index f75b187..ac26bc8 100644 Binary files a/screenshots/output/discord-setup-applications.png and b/screenshots/output/discord-setup-applications.png differ diff --git a/screenshots/output/discord-setup-authorize-dialog.png b/screenshots/output/discord-setup-authorize-dialog.png index 5b945a8..fcbffeb 100644 Binary files a/screenshots/output/discord-setup-authorize-dialog.png and b/screenshots/output/discord-setup-authorize-dialog.png differ diff --git a/screenshots/output/discord-setup-installation-provided-link.png b/screenshots/output/discord-setup-installation-provided-link.png deleted file mode 100644 index 3e6c7fb..0000000 Binary files a/screenshots/output/discord-setup-installation-provided-link.png and /dev/null differ diff --git a/screenshots/output/discord-setup-oauth-url.png b/screenshots/output/discord-setup-oauth-url.png index 215d250..9f0ef49 100644 Binary files a/screenshots/output/discord-setup-oauth-url.png and b/screenshots/output/discord-setup-oauth-url.png differ diff --git a/screenshots/output/docker-health.png b/screenshots/output/docker-health.png new file mode 100644 index 0000000..dc4d431 Binary files /dev/null and b/screenshots/output/docker-health.png differ diff --git a/screenshots/output/doctor.gif b/screenshots/output/doctor.gif deleted file mode 100644 index d713b02..0000000 Binary files a/screenshots/output/doctor.gif and /dev/null differ diff --git a/screenshots/output/doctor.png b/screenshots/output/doctor.png index 1206394..3b6aedb 100644 Binary files a/screenshots/output/doctor.png and b/screenshots/output/doctor.png differ diff --git a/screenshots/output/init-01-endpoint.png b/screenshots/output/init-01-endpoint.png index 75288cc..377e45f 100644 Binary files a/screenshots/output/init-01-endpoint.png and b/screenshots/output/init-01-endpoint.png differ diff --git a/screenshots/output/init-01-model-select.png b/screenshots/output/init-01-model-select.png index f90ee23..52f07f5 100644 Binary files a/screenshots/output/init-01-model-select.png and b/screenshots/output/init-01-model-select.png differ diff --git a/screenshots/output/init-01-provider-list.png b/screenshots/output/init-01-provider-list.png index 12d5471..c26b4b3 100644 Binary files a/screenshots/output/init-01-provider-list.png and b/screenshots/output/init-01-provider-list.png differ diff --git a/screenshots/output/init-01-provider-ollama.png b/screenshots/output/init-01-provider-ollama.png index 75288cc..76bd1a1 100644 Binary files a/screenshots/output/init-01-provider-ollama.png and b/screenshots/output/init-01-provider-ollama.png differ diff --git a/screenshots/output/init-02-identity.png b/screenshots/output/init-02-identity.png index 8d39d3d..4a374d5 100644 Binary files a/screenshots/output/init-02-identity.png and b/screenshots/output/init-02-identity.png differ diff --git a/screenshots/output/init-02-security-posture.png b/screenshots/output/init-02-security-posture.png index 1145cab..0b9508f 100644 Binary files a/screenshots/output/init-02-security-posture.png and b/screenshots/output/init-02-security-posture.png differ diff --git a/screenshots/output/init-04-channel-audiences.png b/screenshots/output/init-04-channel-audiences.png deleted file mode 100644 index 2b53e1c..0000000 Binary files a/screenshots/output/init-04-channel-audiences.png and /dev/null differ diff --git a/screenshots/output/init-04-enabled-features.png b/screenshots/output/init-04-enabled-features.png index dfe8e98..32a1096 100644 Binary files a/screenshots/output/init-04-enabled-features.png and b/screenshots/output/init-04-enabled-features.png differ diff --git a/screenshots/output/init-05-health-check.png b/screenshots/output/init-05-health-check.png index aaa6814..f4c92c6 100644 Binary files a/screenshots/output/init-05-health-check.png and b/screenshots/output/init-05-health-check.png differ diff --git a/screenshots/output/init-07-external-skills.png b/screenshots/output/init-07-external-skills.png deleted file mode 100644 index ae7f20b..0000000 Binary files a/screenshots/output/init-07-external-skills.png and /dev/null differ diff --git a/screenshots/output/init-09-exposure.png b/screenshots/output/init-09-exposure.png deleted file mode 100644 index cb749a2..0000000 Binary files a/screenshots/output/init-09-exposure.png and /dev/null differ diff --git a/screenshots/output/init-existing-menu.png b/screenshots/output/init-existing-menu.png index 0c1ddde..def8b06 100644 Binary files a/screenshots/output/init-existing-menu.png and b/screenshots/output/init-existing-menu.png differ diff --git a/screenshots/output/init-start-over-scope.png b/screenshots/output/init-start-over-scope.png index 84c2b82..fe02e71 100644 Binary files a/screenshots/output/init-start-over-scope.png and b/screenshots/output/init-start-over-scope.png differ diff --git a/screenshots/output/mattermost-bot-accounts.png b/screenshots/output/mattermost-bot-accounts.png index 79a09a9..c1a2ea5 100644 Binary files a/screenshots/output/mattermost-bot-accounts.png and b/screenshots/output/mattermost-bot-accounts.png differ diff --git a/screenshots/output/mattermost-compose-approval-resolved.png b/screenshots/output/mattermost-compose-approval-resolved.png new file mode 100644 index 0000000..8cbc22b Binary files /dev/null and b/screenshots/output/mattermost-compose-approval-resolved.png differ diff --git a/screenshots/output/mattermost-compose-approval.png b/screenshots/output/mattermost-compose-approval.png new file mode 100644 index 0000000..54ba2d3 Binary files /dev/null and b/screenshots/output/mattermost-compose-approval.png differ diff --git a/screenshots/output/mattermost-compose-bot-account.png b/screenshots/output/mattermost-compose-bot-account.png new file mode 100644 index 0000000..7d3be6a Binary files /dev/null and b/screenshots/output/mattermost-compose-bot-account.png differ diff --git a/screenshots/output/mattermost-compose-thread.png b/screenshots/output/mattermost-compose-thread.png new file mode 100644 index 0000000..10e59c6 Binary files /dev/null and b/screenshots/output/mattermost-compose-thread.png differ diff --git a/screenshots/output/mattermost-conversation.png b/screenshots/output/mattermost-conversation.png index f3b4a01..91add4a 100644 Binary files a/screenshots/output/mattermost-conversation.png and b/screenshots/output/mattermost-conversation.png differ diff --git a/screenshots/output/mattermost-create-bot.png b/screenshots/output/mattermost-create-bot.png index d6e7843..a1e31ec 100644 Binary files a/screenshots/output/mattermost-create-bot.png and b/screenshots/output/mattermost-create-bot.png differ diff --git a/screenshots/output/mcp-tools-confirm.png b/screenshots/output/mcp-tools-confirm.png new file mode 100644 index 0000000..2ae736a Binary files /dev/null and b/screenshots/output/mcp-tools-confirm.png differ diff --git a/screenshots/output/mcp-tools-personal.png b/screenshots/output/mcp-tools-personal.png index 3e589db..1f49f5c 100644 Binary files a/screenshots/output/mcp-tools-personal.png and b/screenshots/output/mcp-tools-personal.png differ diff --git a/screenshots/output/mcp-tools-public.png b/screenshots/output/mcp-tools-public.png deleted file mode 100644 index 12523d3..0000000 Binary files a/screenshots/output/mcp-tools-public.png and /dev/null differ diff --git a/screenshots/output/mcp-tools-server-list-2.png b/screenshots/output/mcp-tools-server-list-2.png deleted file mode 100644 index 0797d99..0000000 Binary files a/screenshots/output/mcp-tools-server-list-2.png and /dev/null differ diff --git a/screenshots/output/mcp-tools-server-list.png b/screenshots/output/mcp-tools-server-list.png index 3d4869d..8aed8d9 100644 Binary files a/screenshots/output/mcp-tools-server-list.png and b/screenshots/output/mcp-tools-server-list.png differ diff --git a/screenshots/output/mcp-tools-server2-personal.png b/screenshots/output/mcp-tools-server2-personal.png deleted file mode 100644 index 3d95516..0000000 Binary files a/screenshots/output/mcp-tools-server2-personal.png and /dev/null differ diff --git a/screenshots/output/mcp-tools-team.png b/screenshots/output/mcp-tools-team.png index 1c6dbf5..387388b 100644 Binary files a/screenshots/output/mcp-tools-team.png and b/screenshots/output/mcp-tools-team.png differ diff --git a/screenshots/output/mcp-tools.gif b/screenshots/output/mcp-tools.gif deleted file mode 100644 index 2900c5f..0000000 Binary files a/screenshots/output/mcp-tools.gif and /dev/null differ diff --git a/screenshots/output/model-manager.png b/screenshots/output/model-manager.png index eed76ed..df43475 100644 Binary files a/screenshots/output/model-manager.png and b/screenshots/output/model-manager.png differ diff --git a/screenshots/output/model-properties.png b/screenshots/output/model-properties.png new file mode 100644 index 0000000..698442f Binary files /dev/null and b/screenshots/output/model-properties.png differ diff --git a/screenshots/output/model.gif b/screenshots/output/model.gif deleted file mode 100644 index a629cf6..0000000 Binary files a/screenshots/output/model.gif and /dev/null differ diff --git a/screenshots/output/provider-add-openai-compatible.png b/screenshots/output/provider-add-openai-compatible.png new file mode 100644 index 0000000..0feb361 Binary files /dev/null and b/screenshots/output/provider-add-openai-compatible.png differ diff --git a/screenshots/output/provider-manager.png b/screenshots/output/provider-manager.png index 0d717df..2a07f48 100644 Binary files a/screenshots/output/provider-manager.png and b/screenshots/output/provider-manager.png differ diff --git a/screenshots/output/provider.gif b/screenshots/output/provider.gif deleted file mode 100644 index bf71a5f..0000000 Binary files a/screenshots/output/provider.gif and /dev/null differ diff --git a/screenshots/output/reminder-builder-review.png b/screenshots/output/reminder-builder-review.png new file mode 100644 index 0000000..8e165e3 Binary files /dev/null and b/screenshots/output/reminder-builder-review.png differ diff --git a/screenshots/output/reminder-list.png b/screenshots/output/reminder-list.png index 01d1083..73df3ad 100644 Binary files a/screenshots/output/reminder-list.png and b/screenshots/output/reminder-list.png differ diff --git a/screenshots/output/reminder-run-approval.png b/screenshots/output/reminder-run-approval.png new file mode 100644 index 0000000..8756e8f Binary files /dev/null and b/screenshots/output/reminder-run-approval.png differ diff --git a/screenshots/output/reminder.gif b/screenshots/output/reminder.gif index d4fd796..e132899 100644 Binary files a/screenshots/output/reminder.gif and b/screenshots/output/reminder.gif differ diff --git a/screenshots/output/sessions-tui.png b/screenshots/output/sessions-tui.png new file mode 100644 index 0000000..f9b18df Binary files /dev/null and b/screenshots/output/sessions-tui.png differ diff --git a/screenshots/output/single-shot-chat-web-fetch.png b/screenshots/output/single-shot-chat-web-fetch.png deleted file mode 100644 index 7aa7db5..0000000 Binary files a/screenshots/output/single-shot-chat-web-fetch.png and /dev/null differ diff --git a/screenshots/output/single-shot-cli-execution.png b/screenshots/output/single-shot-cli-execution.png index efe878c..17483ac 100644 Binary files a/screenshots/output/single-shot-cli-execution.png and b/screenshots/output/single-shot-cli-execution.png differ diff --git a/screenshots/output/skill-list.png b/screenshots/output/skill-list.png new file mode 100644 index 0000000..90450b1 Binary files /dev/null and b/screenshots/output/skill-list.png differ diff --git a/screenshots/output/skill-sync.png b/screenshots/output/skill-sync.png new file mode 100644 index 0000000..8b17694 Binary files /dev/null and b/screenshots/output/skill-sync.png differ diff --git a/screenshots/output/slack-channel-integration.png b/screenshots/output/slack-channel-integration.png deleted file mode 100644 index c7d2abf..0000000 Binary files a/screenshots/output/slack-channel-integration.png and /dev/null differ diff --git a/screenshots/output/slack-install-app.png b/screenshots/output/slack-install-app.png index f5b1b07..4403239 100644 Binary files a/screenshots/output/slack-install-app.png and b/screenshots/output/slack-install-app.png differ diff --git a/screenshots/output/slash-unknown-command.png b/screenshots/output/slash-unknown-command.png new file mode 100644 index 0000000..1d69694 Binary files /dev/null and b/screenshots/output/slash-unknown-command.png differ diff --git a/screenshots/output/stats.gif b/screenshots/output/stats.gif deleted file mode 100644 index 7c9df47..0000000 Binary files a/screenshots/output/stats.gif and /dev/null differ diff --git a/screenshots/output/stats.png b/screenshots/output/stats.png index ae0a077..e5f38ad 100644 Binary files a/screenshots/output/stats.png and b/screenshots/output/stats.png differ diff --git a/screenshots/output/status.gif b/screenshots/output/status.gif deleted file mode 100644 index 4f7c742..0000000 Binary files a/screenshots/output/status.gif and /dev/null differ diff --git a/screenshots/output/status.png b/screenshots/output/status.png index 51597f8..3b9c8d2 100644 Binary files a/screenshots/output/status.png and b/screenshots/output/status.png differ diff --git a/screenshots/output/update-check.png b/screenshots/output/update-check.png new file mode 100644 index 0000000..f21f1c6 Binary files /dev/null and b/screenshots/output/update-check.png differ diff --git a/screenshots/output/webhooks-list.png b/screenshots/output/webhooks-list.png index e21be6d..3c7b166 100644 Binary files a/screenshots/output/webhooks-list.png and b/screenshots/output/webhooks-list.png differ diff --git a/screenshots/output/webhooks.gif b/screenshots/output/webhooks.gif deleted file mode 100644 index 0e9b8fe..0000000 Binary files a/screenshots/output/webhooks.gif and /dev/null differ diff --git a/screenshots/playwright/Dockerfile b/screenshots/playwright/Dockerfile new file mode 100644 index 0000000..73238e3 --- /dev/null +++ b/screenshots/playwright/Dockerfile @@ -0,0 +1,16 @@ +# Playwright runner for browser captures (Mattermost flow, docs page screenshots). +# Based on Microsoft's Playwright image so Chromium and its OS deps are preinstalled; +# the version is the one locked in package-lock.json (see screenshots/lib.sh), so the +# `playwright` module and the bundled browsers always match. +# +# The module is installed at /node_modules so scripts mounted anywhere under / resolve +# `import ... from 'playwright'` (ESM ignores NODE_PATH). +ARG PLAYWRIGHT_VERSION +FROM mcr.microsoft.com/playwright:v${PLAYWRIGHT_VERSION}-noble +ARG PLAYWRIGHT_VERSION +RUN mkdir /opt/pw && cd /opt/pw \ + && npm install --no-save --no-audit --no-fund --ignore-scripts playwright@${PLAYWRIGHT_VERSION} \ + && ln -s /opt/pw/node_modules /node_modules \ + && node -e "console.log('playwright', require('playwright/package.json').version)" +ENV HOME=/tmp +WORKDIR /work diff --git a/screenshots/scripts/docker-health-down.sh b/screenshots/scripts/docker-health-down.sh new file mode 100755 index 0000000..f003daf --- /dev/null +++ b/screenshots/scripts/docker-health-down.sh @@ -0,0 +1,10 @@ +#!/bin/bash +# Remove the container docker-health-up.sh started (found by its label, so a container that merely +# shares the name is left alone). Safe to run when there is nothing to remove. +set -u +ids="$(docker ps -aq --filter "label=netclaw-docs-capture=docker-health")" +if [ -n "$ids" ]; then + # shellcheck disable=SC2086 + docker rm -fv $ids >/dev/null +fi +echo "docker-health: cleaned up" diff --git a/screenshots/scripts/docker-health-up.sh b/screenshots/scripts/docker-health-up.sh new file mode 100755 index 0000000..c4a7d22 --- /dev/null +++ b/screenshots/scripts/docker-health-up.sh @@ -0,0 +1,33 @@ +#!/bin/bash +# Start a release container for docker-health.tape through the host's Docker and wait until Docker +# reports it healthy, then give it the name the docs use. Runs inside the capture container, which +# has the host's docker socket for this tape only (compose/docker-host.yml). +# +# It never touches a container it didn't start: if something on the host is already called +# `netclaw`, it stops instead of removing it. The tape removes its own container afterwards by label +# (docker-health-down.sh). +set -eu + +IMAGE="${NETCLAW_IMAGE:-ghcr.io/netclaw-dev/netclaw:0.28.2}" +TMP_NAME="${COMPOSE_PROJECT_NAME:-netclaw-shots}-docker-health" +LABEL="netclaw-docs-capture=docker-health" + +if docker inspect netclaw >/dev/null 2>&1; then + echo "docker-health: a container named 'netclaw' already exists on this host. Stop it or remove it first." >&2 + exit 1 +fi + +docker run -d --name "$TMP_NAME" --label "$LABEL" --stop-timeout 60 \ + -e NETCLAW_Daemon__DisableSelfUpdate=true "$IMAGE" >/dev/null + +for _ in $(seq 1 60); do + status="$(docker inspect --format '{{.State.Health.Status}}' "$TMP_NAME")" + if [ "$status" = healthy ]; then + docker rename "$TMP_NAME" netclaw + echo "container is healthy" + exit 0 + fi + sleep 2 +done +echo "docker-health: container never became healthy" >&2 +exit 1 diff --git a/screenshots/scripts/lib-seed.sh b/screenshots/scripts/lib-seed.sh new file mode 100755 index 0000000..2c7e4dc --- /dev/null +++ b/screenshots/scripts/lib-seed.sh @@ -0,0 +1,34 @@ +#!/bin/bash +# Helpers for the seed-*.sh scripts. Source it. These run inside the capture container as the +# `netclaw` user, which shares the daemon's network namespace and ~/.netclaw volume, so the +# plain `netclaw` CLI talks to the daemon exactly as a user's shell would. The capture image has +# python3 but no jq. + +NC_CONFIG="${HOME}/.netclaw/config/netclaw.json" + +# wait_daemon: block until `netclaw status` reports a healthy daemon. The daemon restarts +# itself in-process after every netclaw.json change, so call this after each write. +wait_daemon() { + local i + sleep 3 + for i in $(seq 1 60); do + if netclaw status 2>/dev/null | grep -q '^overall: healthy'; then return 0; fi + sleep 2 + done + echo "seed: daemon did not become healthy" >&2 + return 1 +} + +# json_set
: merge a JSON object into one top-level section of netclaw.json +# (replacing the keys it names, keeping the rest of the section). +json_set() { + python3 - "$NC_CONFIG" "$1" "$2" <<'PY' +import json, sys +path, section, patch = sys.argv[1], sys.argv[2], json.loads(sys.argv[3]) +with open(path) as f: + cfg = json.load(f) +cfg.setdefault(section, {}).update(patch) +with open(path, "w") as f: + json.dump(cfg, f, indent=2) +PY +} diff --git a/screenshots/scripts/mock-mattermost.py b/screenshots/scripts/mock-mattermost.py new file mode 100755 index 0000000..31e5420 --- /dev/null +++ b/screenshots/scripts/mock-mattermost.py @@ -0,0 +1,45 @@ +#!/usr/bin/env python3 +"""Just enough of the Mattermost REST API for `netclaw config` to resolve channel labels. + +The channel permissions screen looks channel IDs up through GET /api/v4/users/me, the bot's teams +and each team's channels. Nothing else is implemented, so this is a fixture for one screenshot +(config-channel-audiences.png), not a Mattermost server. Listens on 127.0.0.1:8065. +""" +import json +from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer + +TEAM = "t4m1d9q2w6x8c3v5b7n1k0j2hz" +CHANNELS = [ + {"id": "c1a5s7i9s3t0a8n2t6b4o1t5k7", "name": "assistant", "display_name": "Assistant", "type": "O"}, + {"id": "c2o8p4s6r1o9o3m5d7e0m2o4k6", "name": "ops-room", "display_name": "Ops Room", "type": "O"}, + {"id": "c3l9u5n1c7h2t4a6b8l0e1d3k5", "name": "lunch", "display_name": "Lunch", "type": "O"}, +] + + +class Handler(BaseHTTPRequestHandler): + def _send(self, body): + data = json.dumps(body).encode() + self.send_response(200) + self.send_header("Content-Type", "application/json") + self.send_header("Content-Length", str(len(data))) + self.end_headers() + self.wfile.write(data) + + def do_GET(self): + path = self.path.split("?")[0] + if path == "/api/v4/users/me": + self._send({"id": "u7b0t2i4d6n8e0t2c4l6a8w0ka", "username": "netclaw-bot"}) + elif path == "/api/v4/users/me/teams": + self._send([{"id": TEAM, "name": "demo"}]) + elif path == f"/api/v4/users/me/teams/{TEAM}/channels": + self._send(CHANNELS) + else: + self.send_response(404) + self.end_headers() + + def log_message(self, *args): + pass + + +if __name__ == "__main__": + ThreadingHTTPServer(("127.0.0.1", 8065), Handler).serve_forever() diff --git a/screenshots/scripts/ollama-init.sh b/screenshots/scripts/ollama-init.sh index 5b3b5a2..c2809cc 100755 --- a/screenshots/scripts/ollama-init.sh +++ b/screenshots/scripts/ollama-init.sh @@ -2,10 +2,12 @@ set -eu MODEL="${OLLAMA_MODEL:-qwen2:0.5b}" +# Runs in the stack's shared network namespace, so Ollama is on loopback. +OLLAMA="http://127.0.0.1:11434" echo "Waiting for Ollama API to become available..." for i in $(seq 1 30); do - if curl -fsS http://ollama:11434/api/tags >/dev/null; then + if curl -fsS "$OLLAMA/api/tags" >/dev/null; then echo "Ollama is ready." break fi @@ -14,9 +16,9 @@ for i in $(seq 1 30); do done echo "Pulling model: $MODEL" -curl -fsS -X POST http://ollama:11434/api/pull -d "{\"name\":\"$MODEL\"}" +curl -fsS -X POST "$OLLAMA/api/pull" -d "{\"name\":\"$MODEL\"}" echo "Verifying model is available..." -curl -fsS http://ollama:11434/api/tags +curl -fsS "$OLLAMA/api/tags" echo "Model init completed." diff --git a/screenshots/scripts/redact-third-party.py b/screenshots/scripts/redact-third-party.py new file mode 100644 index 0000000..33b56ad --- /dev/null +++ b/screenshots/scripts/redact-third-party.py @@ -0,0 +1,65 @@ +#!/usr/bin/env python3 +"""Minimal, declared redactions for the third-party (Slack and Discord) screenshots. + +These images were captured on the maintainer's machine. We keep them, but remove: + + slack-install-app.png app switcher with a truncated real app name (solid box) + discord-setup-applications.png browser toolbar: URL bar, extension icons (crop) + discord-setup-oauth-url.png client_id value (solid box) + discord-setup-authorize-dialog.png the signed-in Discord handle and the debug app's name, + twice (solid boxes; the dialog still shows the + permissions list and the Authorize button) + +Boxes use the colour sampled next to the box, so they read as empty space. No blur. +Each edit checks the source dimensions first, so running the script twice is a no-op. + +Usage (Pillow required): python3 screenshots/scripts/redact-third-party.py +""" +import os +import sys +from PIL import Image + +OUT = os.path.join(os.path.dirname(os.path.abspath(__file__)), "..", "output") + + +def load(name, expected_size): + path = os.path.join(OUT, name) + img = Image.open(path).convert("RGB") + if img.size != expected_size: + print(f"skip {name}: {img.size} is not the original {expected_size} (already edited?)") + return None, path + return img, path + + +def box(img, rect, sample_at): + """Fill rect (x0, y0, x1, y1) with the colour at sample_at.""" + colour = img.getpixel(sample_at) + img.paste(colour, rect) + + +def main(): + img, path = load("slack-install-app.png", (992, 334)) + if img: + box(img, (40, 44, 254, 100), (20, 70)) # app switcher, left rail background + img.save(path); print("ok slack-install-app.png") + + img, path = load("discord-setup-applications.png", (1522, 539)) + if img: + img.crop((0, 41, 1522, 539)).save(path); print("ok discord-setup-applications.png") + + img, path = load("discord-setup-oauth-url.png", (1406, 208)) + if img: + box(img, (383, 156, 542, 180), (1000, 168)) # client_id digits, input background + img.save(path); print("ok discord-setup-oauth-url.png") + + + img, path = load("discord-setup-authorize-dialog.png", (506, 804)) + if img: + box(img, (144, 148, 318, 172), (110, 160)) # app name in the heading, dialog background + box(img, (209, 207, 306, 226), (110, 160)) # handle after "Signed in as", dialog background + box(img, (255, 283, 374, 300), (430, 292)) # app name in the permissions sentence, panel background + img.save(path); print("ok discord-setup-authorize-dialog.png") + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/screenshots/scripts/seed-approvals.sh b/screenshots/scripts/seed-approvals.sh deleted file mode 100755 index 7e7b8f2..0000000 --- a/screenshots/scripts/seed-approvals.sh +++ /dev/null @@ -1,42 +0,0 @@ -#!/bin/bash -# Seeds a realistic tool-approvals.json for screenshot capture. -# Called by the approvals VHS tape before recording. -set -eu - -NETCLAW_HOME="${NETCLAW_HOME:-$HOME/.netclaw}" -CONFIG_DIR="$NETCLAW_HOME/config" -APPROVALS_FILE="$CONFIG_DIR/tool-approvals.json" - -mkdir -p "$CONFIG_DIR" - -# Clean up any existing approvals and quarantine files -rm -f "$APPROVALS_FILE" "$APPROVALS_FILE.v1.bak" "$APPROVALS_FILE.invalid" - -cat > "$APPROVALS_FILE" << 'SEED' -{ - "version": 2, - "audiences": { - "personal": { - "shell_execute": [ - { "verb": "git push", "directory": null }, - { "verb": "git pull", "directory": null }, - { "verb": "git remote", "directory": "/home/user/repos/netclaw/" }, - { "verb": "docker compose", "directory": "/home/user/repos/netclaw/" }, - { "verb": "npm run", "directory": "/home/user/repos/netclaw-website/" }, - { "verb": "ls", "directory": null } - ], - "mcp:demo-utilities:write_file": [ - { "verb": "write_file", "directory": "/home/user/repos/netclaw/" } - ] - }, - "team": { - "shell_execute": [ - { "verb": "git pull", "directory": null }, - { "verb": "git status", "directory": null } - ] - } - } -} -SEED - -echo "Seeded $APPROVALS_FILE" diff --git a/screenshots/scripts/seed-assistant.sh b/screenshots/scripts/seed-assistant.sh new file mode 100755 index 0000000..c51b743 --- /dev/null +++ b/screenshots/scripts/seed-assistant.sh @@ -0,0 +1,37 @@ +#!/bin/bash +# Seed for the tapes that need a working assistant (status, doctor, stats, sessions, chat): one +# OpenAI-compatible provider answered by the stack's forwarder (INFERENCE_UPSTREAM, see +# compose/forwarder.yml), all three model roles on it, and the demo MCP server. +# +# The model ID is the neutral name the forwarder presents (demo-model). It rewrites the upstream's +# real model id to that name in responses, and llama.cpp serves whatever model it has loaded +# regardless of the name in the request, so the real name never reaches netclaw.json or the screen. +# +# Also writes the Security and Identity sections `netclaw init` writes (Personal posture), so the +# config is complete and `netclaw doctor` has no failures. Needs an empty ~/.netclaw +# (`# stack: bare`, which brings up the demo MCP server container). +set -eu +# shellcheck source=lib-seed.sh +. "$(dirname "${BASH_SOURCE[0]}")/lib-seed.sh" + +MODEL_ID="${ASSISTANT_MODEL_ID:-demo-model}" + +netclaw provider add lab-llama openai-compatible --endpoint "${LAB_LLAMA_ENDPOINT:-http://llm.example.internal:8081}" >/dev/null +for role in main fallback compaction; do + netclaw model set "$role" lab-llama "$MODEL_ID" >/dev/null +done +netclaw mcp add --transport http demo-utilities http://127.0.0.1:8080/mcp >/dev/null + +json_set Security '{"DeploymentPosture": "Personal", "ShellExecutionMode": "HostAllowed", "StrictDefaults": true}' +json_set Tools '{"ShellMode": "HostAllowed"}' +json_set Identity '{"AgentName": "Netclaw", "CommunicationStyle": "Concise & casual", "UserTimezone": "Etc/UTC", "UserName": "alice"}' + +for i in $(seq 1 45); do + if netclaw status 2>/dev/null | grep -q 'mcp:demo-utilities: healthy'; then + echo "seeded: provider lab-llama, model $MODEL_ID, MCP demo-utilities" + exit 0 + fi + sleep 2 +done +echo "seed: demo-utilities did not report healthy" >&2 +exit 1 diff --git a/screenshots/scripts/seed-channel-audiences.sh b/screenshots/scripts/seed-channel-audiences.sh new file mode 100755 index 0000000..b1a8135 --- /dev/null +++ b/screenshots/scripts/seed-channel-audiences.sh @@ -0,0 +1,22 @@ +#!/bin/bash +# Seed for the channel audiences screen: enables a Mattermost adapter with three channels and a mix +# of audiences, pointed at the API fixture (mock-mattermost.py, which the tape starts first). The +# adapter can't actually connect, which only affects `netclaw status`, not the config screen. +set -eu +# shellcheck source=lib-seed.sh +. "$(dirname "${BASH_SOURCE[0]}")/lib-seed.sh" + +netclaw secrets set Mattermost.BotToken demo-bot-token-0000 >/dev/null +json_set Mattermost '{ + "Enabled": true, + "ServerUrl": "http://127.0.0.1:8065", + "AllowDirectMessages": true, + "AllowedChannelIds": ["c1a5s7i9s3t0a8n2t6b4o1t5k7", "c2o8p4s6r1o9o3m5d7e0m2o4k6", "c3l9u5n1c7h2t4a6b8l0e1d3k5"], + "ChannelAudiences": { + "c1a5s7i9s3t0a8n2t6b4o1t5k7": "Team", + "c2o8p4s6r1o9o3m5d7e0m2o4k6": "Team", + "c3l9u5n1c7h2t4a6b8l0e1d3k5": "Public" + } +}' +sleep 5 +echo "seeded: Mattermost adapter with three channels" diff --git a/screenshots/scripts/seed-config-demo.sh b/screenshots/scripts/seed-config-demo.sh new file mode 100755 index 0000000..b7d3402 --- /dev/null +++ b/screenshots/scripts/seed-config-demo.sh @@ -0,0 +1,32 @@ +#!/bin/bash +# Seed for the `netclaw config` tape, on top of a completed `netclaw init` (Personal): a +# projects folder for the workspaces picker, one inbound webhook route, and the Claude Code +# skills folder as an external skill source. Neutral names throughout. +set -eu +# shellcheck source=lib-seed.sh +. "$(dirname "${BASH_SOURCE[0]}")/lib-seed.sh" + +mkdir -p "$HOME/projects/website" "$HOME/projects/notes" "$HOME/projects/infra" +json_set Workspaces '{"Directory": "/home/netclaw/projects"}' +wait_daemon + +netclaw webhooks set github-issues \ + --prompt "Triage the new GitHub issue and post a one-paragraph summary." \ + --secret demo-webhook-secret \ + --signature-header X-Hub-Signature-256 --signature-prefix "sha256=" \ + --events issues.opened,issues.closed >/dev/null +json_set Webhooks '{"Enabled": true}' +wait_daemon + +mkdir -p "$HOME/.claude/skills/release-notes" +cat > "$HOME/.claude/skills/release-notes/SKILL.md" <<'SKILL' +--- +name: release-notes +description: Draft release notes from a list of merged pull requests. +--- + +Group the merged pull requests by area and write one sentence per change. +SKILL +netclaw skill source add claude-code --well-known claude-code >/dev/null +wait_daemon +echo "seeded: workspaces, webhook route github-issues, skill source claude-code" diff --git a/screenshots/scripts/seed-demo-install.sh b/screenshots/scripts/seed-demo-install.sh new file mode 100755 index 0000000..4752b0b --- /dev/null +++ b/screenshots/scripts/seed-demo-install.sh @@ -0,0 +1,81 @@ +#!/bin/bash +# ONE coherent demo install for every status-like capture: status, doctor, stats, sessions, model, +# provider, skill list, webhooks list, reminder list, the config dashboard and its sub-screens. +# Because they all start from this script, no two images disagree about what is installed. +# +# providers lab-llama (OpenAI-compatible, answered by the stack's forwarder), my-ollama +# (the stack's local Ollama), my-deepseek (dummy key) +# roles main on lab-llama / demo-model, fallback and compaction on my-ollama / qwen2:0.5b +# MCP demo-utilities (the stack's demo server) +# skills two local, two from the corp-skills feed, one external source (claude-code) +# webhooks github-issues and deploy-notify +# reminders daily-uptime-check and weekly-report +# workspaces /home/netclaw/projects with website (a Git repo), notes and infra +# search DuckDuckGo, set the way the Search screen saves it +# +# Run inside the capture container against an empty ~/.netclaw (`# stack: bare`, which also brings +# up the demo MCP and skill servers). Prints "seeded: demo install" when done. +set -eu +here="$(dirname "${BASH_SOURCE[0]}")" +# shellcheck source=lib-seed.sh +. "$here/lib-seed.sh" + +netclaw provider add lab-llama openai-compatible --endpoint "${LAB_LLAMA_ENDPOINT:-http://llm.example.internal:8081}" >/dev/null +netclaw provider add my-ollama ollama --endpoint http://127.0.0.1:11434 >/dev/null +netclaw provider add my-deepseek deepseek --api-key sk-demo-0000 >/dev/null +netclaw model set main lab-llama "${ASSISTANT_MODEL_ID:-demo-model}" >/dev/null +netclaw model set fallback my-ollama qwen2:0.5b >/dev/null +netclaw model set compaction my-ollama qwen2:0.5b >/dev/null +netclaw mcp add --transport http demo-utilities http://127.0.0.1:8080/mcp >/dev/null + +# The sections `netclaw init` writes for a Personal install, plus a Search backend. +json_set Security '{"DeploymentPosture": "Personal", "ShellExecutionMode": "HostAllowed", "StrictDefaults": true}' +json_set Tools '{"ShellMode": "HostAllowed"}' +json_set Identity '{"AgentName": "Netclaw", "CommunicationStyle": "Concise & casual", "UserTimezone": "Etc/UTC", "UserName": "alice"}' +json_set Search '{"Backend": "duckduckgo"}' +wait_daemon + +# Workspaces: the website repository plus two plain folders for the picker. +bash "$here/seed-projects.sh" +mkdir -p "$HOME/projects/notes" "$HOME/projects/infra" +wait_daemon + +# Skills: local skills, the corp-skills feed, one external source. +bash "$here/seed-skills.sh" +mkdir -p "$HOME/.claude/skills/release-notes" +cat > "$HOME/.claude/skills/release-notes/SKILL.md" <<'SKILL' +--- +name: release-notes +description: Draft release notes from a list of merged pull requests. +--- + +Group the merged pull requests by area and write one sentence per change. +SKILL +netclaw skill source add claude-code --well-known claude-code >/dev/null +wait_daemon +netclaw skill sync >/dev/null + +# Two inbound webhook routes. +netclaw webhooks set github-issues \ + --prompt "Triage the new GitHub issue and post a one-paragraph summary." \ + --secret demo-webhook-secret \ + --signature-header X-Hub-Signature-256 --signature-prefix "sha256=" \ + --events issues.opened,issues.closed --audience personal >/dev/null +netclaw webhooks set deploy-notify \ + --prompt "A deployment finished. Summarize the event payload in one line." \ + --secret demo-webhook-secret --audience personal >/dev/null +json_set Webhooks '{"Enabled": true}' +wait_daemon + +# Two reminders. +bash "$here/seed-reminders.sh" + +for i in $(seq 1 45); do + if netclaw status 2>/dev/null | grep -q 'mcp:demo-utilities: healthy'; then + echo "seeded: demo install" + exit 0 + fi + sleep 2 +done +echo "seed: demo-utilities did not report healthy" >&2 +exit 1 diff --git a/screenshots/scripts/seed-paired-device.sh b/screenshots/scripts/seed-paired-device.sh new file mode 100755 index 0000000..a879635 --- /dev/null +++ b/screenshots/scripts/seed-paired-device.sh @@ -0,0 +1,10 @@ +#!/bin/bash +# Pair one device, "alice-laptop", the way a second machine would: take a code from the daemon host +# and enter it on the device. The device side runs with its own NETCLAW_HOME, so the token and the +# endpoint land there and the daemon host's CLI configuration stays untouched. +set -eu + +code="$(netclaw daemon pair | awk '/Pairing code/ {print $3}')" +mkdir -p /tmp/alice-laptop +printf '%s\nalice-laptop\n' "$code" | NETCLAW_HOME=/tmp/alice-laptop netclaw pair http://127.0.0.1:5199 >/dev/null +echo "seeded: paired device alice-laptop" diff --git a/screenshots/scripts/seed-projects.sh b/screenshots/scripts/seed-projects.sh new file mode 100644 index 0000000..58aa121 --- /dev/null +++ b/screenshots/scripts/seed-projects.sh @@ -0,0 +1,28 @@ +#!/bin/bash +# Creates the neutral demo workspace used by the chat, approvals and reminder tapes: a small +# Git repository at /home/netclaw/projects/website (shared with the daemon by +# screenshots/compose/projects.yml) declared as the workspace root, so shell calls run +# inside the repository and approval prompts offer "This repository". Idempotent. +# Run inside the capture container. +set -eu + +repo="$HOME/projects/website" +mkdir -p "$repo" +cd "$repo" +if [ ! -d .git ]; then + git init -q -b main . + git config user.name alice + git config user.email alice@example.com + printf '# website\n\nStatic marketing site. Edit files under src/.\n' > AGENTS.md + mkdir -p src + printf '

Hello

\n' > src/index.html + git add . + git commit -q -m "Initial commit" +fi + +# Declare the workspace root. The daemon restarts itself when netclaw.json changes. +cfg="$NETCLAW_HOME/config/netclaw.json" +if [ "$(jq -r '.Workspaces.Directory // ""' "$cfg")" != "$HOME/projects" ]; then + jq --arg d "$HOME/projects" '.Workspaces = {Directory: $d}' "$cfg" > "$cfg.tmp" && mv "$cfg.tmp" "$cfg" + sleep 8 +fi diff --git a/screenshots/scripts/seed-providers.sh b/screenshots/scripts/seed-providers.sh new file mode 100755 index 0000000..7678c05 --- /dev/null +++ b/screenshots/scripts/seed-providers.sh @@ -0,0 +1,31 @@ +#!/bin/bash +# Seed for the provider and model manager tapes: three providers with neutral names and all three +# model roles assigned, written with the real CLI the way a user would after init. +# +# my-ollama ollama the stack's local Ollama (reachable) +# lab-llama openai-compatible http://llm.example.internal:8081, answered by the stack's +# forwarder (compose/forwarder.yml) when INFERENCE_UPSTREAM is set, +# unreachable otherwise +# my-deepseek deepseek a dummy key, so the provider can't authenticate +# +# Also writes the Security and Identity sections `netclaw init` writes, so the config is complete. +# Needs an empty ~/.netclaw (`# stack: fresh` or `# stack: bare`). +set -eu +# shellcheck source=lib-seed.sh +. "$(dirname "${BASH_SOURCE[0]}")/lib-seed.sh" + +netclaw provider add my-ollama ollama --endpoint http://127.0.0.1:11434 >/dev/null +netclaw provider add lab-llama openai-compatible --endpoint "${LAB_LLAMA_ENDPOINT:-http://llm.example.internal:8081}" >/dev/null +netclaw provider add my-deepseek deepseek --api-key sk-demo-0000 >/dev/null + +# The OpenAI-compatible type label is long enough to push the model manager's columns out of line, +# so lab-llama stays out of the role table. +netclaw model set main my-deepseek deepseek-chat >/dev/null +netclaw model set fallback my-ollama qwen2:0.5b >/dev/null +netclaw model set compaction my-ollama qwen2:0.5b >/dev/null + +json_set Security '{"DeploymentPosture": "Personal", "ShellExecutionMode": "HostAllowed", "StrictDefaults": true}' +json_set Tools '{"ShellMode": "HostAllowed"}' +json_set Identity '{"AgentName": "Netclaw", "CommunicationStyle": "Concise & casual", "UserTimezone": "Etc/UTC", "UserName": "alice"}' +wait_daemon +echo "seeded: providers my-ollama, lab-llama, my-deepseek; roles main (my-deepseek), fallback and compaction (my-ollama)" diff --git a/screenshots/scripts/seed-reminders.sh b/screenshots/scripts/seed-reminders.sh new file mode 100644 index 0000000..c175b33 --- /dev/null +++ b/screenshots/scripts/seed-reminders.sh @@ -0,0 +1,10 @@ +#!/bin/bash +# Neutral reminders for the reminder tapes: a daily cron reminder and a weekly one. +# Run inside the capture container (talks to the stack's daemon). +set -eu +netclaw reminder create daily-uptime-check cron "0 9 * * *" \ + "Check that the website and API respond, and note anything slower than usual." \ + --name "Daily uptime check" >/dev/null +netclaw reminder create weekly-report cron "CRON_TZ=America/Chicago 0 16 * * FRI" \ + "Summarize this week's completed tasks and open follow-ups in five bullets." \ + --name "Weekly report" >/dev/null diff --git a/screenshots/scripts/seed-skills.sh b/screenshots/scripts/seed-skills.sh new file mode 100644 index 0000000..6c659e6 --- /dev/null +++ b/screenshots/scripts/seed-skills.sh @@ -0,0 +1,91 @@ +#!/bin/bash +# Stages the skills the skill-list and skill-sync captures show: +# - two local skills in ~/.netclaw/skills (hello-greeter with a references/ file, flat-note) +# - two skills and one subagent published to the stack's skill server (port 8090) with the +# skillserver CLI, and the server added as the `corp-skills` feed in netclaw.json. +# Run inside the capture container. With `no-feed` as the first argument, only the local +# skills are created (nothing is published and no feed is configured). +set -eu + +skills="$HOME/.netclaw/skills" +mkdir -p "$skills/hello-greeter/references" "$skills/flat-note" +cat > "$skills/hello-greeter/SKILL.md" <<'SKILL' +--- +name: hello-greeter +description: Greet the user by name and mention one useful thing about their day. +metadata: + version: 1.0.0 +--- + +# Hello greeter + +Greet the user by name. Read `references/greetings.md` for tone before you answer. +SKILL +cat > "$skills/hello-greeter/references/greetings.md" <<'REF' +Keep greetings to one sentence. Warm, never gushing. +REF +cat > "$skills/flat-note/SKILL.md" <<'SKILL' +--- +name: flat-note +description: Capture a quick note as a single markdown bullet. +--- + +# Flat note + +Write the note as one markdown bullet and confirm it back in a sentence. +SKILL + +[ "${1:-}" = "no-feed" ] && exit 0 + +server=http://127.0.0.1:8090 +export SKILLSERVER_API_KEY=demo-publish-key +src="$(mktemp -d)" +agents="$(mktemp -d)" +mkdir -p "$src/k8s-log-triage/references" "$src/incident-summary" +cat > "$src/k8s-log-triage/SKILL.md" <<'SKILL' +--- +name: k8s-log-triage +description: Triage Kubernetes pod logs, group repeated errors and suggest the next check. +metadata: + version: 0.2.0 +--- + +# Kubernetes log triage + +Read `references/patterns.md`, group the errors by pattern and propose one next check. +SKILL +cat > "$src/k8s-log-triage/references/patterns.md" <<'REF' +CrashLoopBackOff: check the previous container's logs first. +ImagePullBackOff: check the registry credentials and the tag. +REF +cat > "$src/incident-summary/SKILL.md" <<'SKILL' +--- +name: incident-summary +description: Summarize an incident timeline in five bullets for a status update. +metadata: + version: 1.1.0 +--- + +# Incident summary + +Turn the timeline into five bullets: impact, cause, fix, follow-ups, owner. +SKILL +cat > "$agents/release-notes-writer.md" <<'AGENT' +--- +name: release-notes-writer +description: Draft release notes from a list of merged changes +modelRole: Compaction +timeoutSeconds: 90 +--- + +You write short release notes. Group the changes by user impact and keep each line under 100 characters. +AGENT + +skillserver publish-all "$src" --server-url "$server" >/dev/null 2>&1 || skillserver publish-all "$src" --server-url "$server" +skillserver publish-subagent "$agents/release-notes-writer.md" --version 1.0.0 --server-url "$server" >/dev/null +rm -rf "$src" "$agents" + +cfg="$NETCLAW_HOME/config/netclaw.json" +jq '.SkillFeeds = {SyncIntervalMinutes: 60, Feeds: [{Name: "corp-skills", Url: "http://127.0.0.1:8090", Enabled: true, TimeoutSeconds: 30}]}' \ + "$cfg" > "$cfg.tmp" && mv "$cfg.tmp" "$cfg" +sleep 15 diff --git a/screenshots/scripts/seed-webhooks.sh b/screenshots/scripts/seed-webhooks.sh new file mode 100644 index 0000000..e9775be --- /dev/null +++ b/screenshots/scripts/seed-webhooks.sh @@ -0,0 +1,16 @@ +#!/bin/bash +# Two neutral webhook routes for the webhooks tape. The secret is a throwaway demo value +# written to a temp file and read once by `set`. Run inside the capture container. +set -eu +cfg="$NETCLAW_HOME/config/netclaw.json" +if [ "$(jq -r '.Webhooks.Enabled // false' "$cfg")" != "true" ]; then + jq '.Webhooks = {Enabled: true}' "$cfg" > "$cfg.tmp" && mv "$cfg.tmp" "$cfg" + sleep 12 +fi +secret="$(mktemp)" +printf 'demo-signing-secret' > "$secret" +netclaw webhooks set github-issues --prompt "Triage this issue: label it and summarize it in two sentences." \ + --secret-file "$secret" --audience personal >/dev/null +netclaw webhooks set deploy-notify --prompt "A deployment finished. Summarize the event payload in one line." \ + --secret-file "$secret" --audience personal >/dev/null +rm -f "$secret" diff --git a/screenshots/shoot-page.sh b/screenshots/shoot-page.sh new file mode 100755 index 0000000..2687f18 --- /dev/null +++ b/screenshots/shoot-page.sh @@ -0,0 +1,30 @@ +#!/bin/bash +# Screenshot a page of the locally built docs site (dist/) in the Playwright container. +# +# npm run build +# ./screenshots/shoot-page.sh cli/status -> docs-logs/screenshots/cli-status.png +# SCREENSHOT_OUT=/some/dir ./screenshots/shoot-page.sh cli/status +# SCREENSHOT_BASE_URL=http://host.docker.internal:4321 ./screenshots/shoot-page.sh cli/status +# (a dev server started with `npm run dev -- --host 0.0.0.0`) +# +# Runs scripts/screenshot-page.mjs with the same pinned Playwright image as the +# Mattermost capture. No host browser or host node needed beyond building the site. +set -euo pipefail +# shellcheck source=lib.sh +. "$(dirname "${BASH_SOURCE[0]}")/lib.sh" + +[ "$#" -ge 1 ] || { echo "usage: shoot-page.sh e.g. cli/status" >&2; exit 2; } + +OUT="${SCREENSHOT_OUT:-$PROJECT_ROOT/docs-logs/screenshots}" +mkdir -p "$OUT" +OUT="$(cd "$OUT" && pwd)" + +IMAGE="netclaw-docs-playwright:$PLAYWRIGHT_VERSION" +docker build -q -t "$IMAGE" --build-arg "PLAYWRIGHT_VERSION=$PLAYWRIGHT_VERSION" "$SHOTS_DIR/playwright" >/dev/null + +env_args=(-e SCREENSHOT_OUT=/out) +[ -z "${SCREENSHOT_BASE_URL:-}" ] || env_args+=(-e "SCREENSHOT_BASE_URL=$SCREENSHOT_BASE_URL") + +docker run --rm --user "$HOST_UID:$HOST_GID" --add-host host.docker.internal:host-gateway \ + "${env_args[@]}" -v "$PROJECT_ROOT:/site:ro" -v "$OUT:/out" \ + "$IMAGE" node /site/scripts/screenshot-page.mjs "$1" | sed "s#^/out#$OUT#" diff --git a/screenshots/stack.sh b/screenshots/stack.sh new file mode 100755 index 0000000..3d938d2 --- /dev/null +++ b/screenshots/stack.sh @@ -0,0 +1,111 @@ +#!/bin/bash +# Bring up a disposable Netclaw daemon (the released image) for verifying commands and +# for tape recordings. Docker is the only prerequisite. Never uses a host netclaw. +# +# ./screenshots/stack.sh up [--fresh|--bare] start the stack and wait until the daemon is healthy +# ./screenshots/stack.sh reset [--fresh|--bare] wipe ~/.netclaw and restart the daemon (inference cache is kept) +# ./screenshots/stack.sh exec run a command in the daemon container as the netclaw user +# ./screenshots/stack.sh daemon-log [N] last N lines (default 200) of the daemon's log file +# ./screenshots/stack.sh logs [args] docker compose logs for the stack (container stdout) +# ./screenshots/stack.sh ps list the stack's containers +# ./screenshots/stack.sh down remove containers, network and volumes +# +# States: +# configured (default) provider, model roles and the demo MCP server are added with the real +# CLI (provider add / model set / mcp add), as a user would after init +# --fresh no model config and an empty ~/.netclaw: first-run state for `netclaw init` +# --bare empty ~/.netclaw like --fresh, but the demo MCP server container is up; the +# tape seeds its own config (see screenshots/scripts/seed-*.sh) +# +# Examples: +# COMPOSE_PROJECT_NAME=dr04 ./screenshots/stack.sh up +# COMPOSE_PROJECT_NAME=dr04 ./screenshots/stack.sh exec netclaw doctor +# COMPOSE_PROJECT_NAME=dr04 ./screenshots/stack.sh down +# +# Image: NETCLAW_VERSION=0.27.1 (default: src/version.json) or NETCLAW_IMAGE=. +# Inference: hermetic Ollama by default; set INFERENCE_BASE_URL + INFERENCE_MODEL_ID +# (env or screenshots/.env) to use an external OpenAI-compatible endpoint. +set -euo pipefail +# shellcheck source=lib.sh +. "$(dirname "${BASH_SOURCE[0]}")/lib.sh" + +state=configured +cmd="${1:-}"; [ "$#" -gt 0 ] && shift +if [ "${1:-}" = "--fresh" ]; then state=fresh; shift; fi +if [ "${1:-}" = "--bare" ]; then state=bare; shift; fi + +wait_healthy() { + local cid status i + cid="$(dc "$state" ps -q netclawd)" + for i in $(seq 1 90); do + status="$(docker inspect -f '{{.State.Health.Status}}' "$cid" 2>/dev/null || echo starting)" + if [ "$status" = healthy ]; then + echo "==> netclawd is healthy ($NETCLAW_IMAGE, project $COMPOSE_PROJECT_NAME, state $state, inference $INFERENCE_MODE)" + return 0 + fi + sleep 2 + done + echo "stack: netclawd did not become healthy" >&2 + dc "$state" logs --tail 80 netclawd >&2 || true + return 1 +} + +# Run the CLI in the daemon container as the netclaw user (no TTY). +nc() { dc all exec -T -u netclaw netclawd netclaw "$@"; } + +# Initialize a configured daemon the way a user would, with the real CLI. The daemon +# restarts itself in-process after each config write, so wait for MCP to connect. +seed() { + local i + nc provider add inference "$NC_INFERENCE_TYPE" --endpoint "$NC_INFERENCE_ENDPOINT" >/dev/null + for role in main fallback compaction; do + nc model set "$role" inference "$NC_INFERENCE_MODEL" >/dev/null + done + nc mcp add --transport http demo-utilities http://127.0.0.1:8080/mcp >/dev/null + # `netclaw init` also writes the Security section (no CLI command for it); add what the + # wizard writes for the Personal posture so `netclaw doctor` passes the Security Policy check. + dc all exec -T -u netclaw netclawd sh -c ' + f=/home/netclaw/.netclaw/config/netclaw.json + jq ".Security = {DeploymentPosture: \"Personal\", ShellExecutionMode: \"HostAllowed\", StrictDefaults: true}" "$f" > "$f.tmp" && mv "$f.tmp" "$f"' + for i in $(seq 1 45); do + if nc status 2>/dev/null | grep -q 'mcp:demo-utilities: healthy'; then + echo "==> seeded: provider inference ($NC_INFERENCE_TYPE), model $NC_INFERENCE_MODEL, MCP demo-utilities" + return 0 + fi + sleep 2 + done + echo "stack: daemon did not report demo-utilities healthy after seeding" >&2 + nc status >&2 || true + return 1 +} + +bring_up() { + dc "$state" up -d --build --remove-orphans + wait_healthy + if [ "$state" = configured ]; then seed; fi +} + +case "$cmd" in + up) + echo "==> Starting $NETCLAW_IMAGE as project $COMPOSE_PROJECT_NAME ($state, $INFERENCE_MODE inference)" + bring_up + ;; + reset) + dc all rm -sf netclawd >/dev/null + docker volume rm "$(home_volume)" >/dev/null 2>&1 || true + bring_up + ;; + daemon-log) dc all exec -T -u netclaw netclawd sh -c 'tail -n "${1:-200}" /home/netclaw/.netclaw/logs/daemon-*.log' _ "$@" ;; + exec) + [ "$#" -gt 0 ] || { echo "usage: stack.sh exec " >&2; exit 2; } + tty_flag=(); [ -t 0 ] || tty_flag=(-T) + dc all exec "${tty_flag[@]}" -u netclaw netclawd "$@" + ;; + logs) dc all logs "$@" ;; + ps) dc all ps "$@" ;; + down) dc all down -v --remove-orphans ;; + *) + sed -n '2,28p' "${BASH_SOURCE[0]}" | sed 's/^# \{0,1\}//' + exit 2 + ;; +esac diff --git a/screenshots/tapes/approvals-list.tape b/screenshots/tapes/approvals-list.tape new file mode 100644 index 0000000..f9a81a2 --- /dev/null +++ b/screenshots/tapes/approvals-list.tape @@ -0,0 +1,18 @@ +# approvals-list.png: `netclaw approvals list` with grants made by trust-verb and by real +# prompt answers (see lib/seed-approvals.tape). +Set Shell "bash" +Set WaitTimeout 240s +Set Theme "Dracula" +Set FontSize 16 +Set Width 1000 +Set Height 380 +Set Padding 20 +Set TypingSpeed 40ms + +Source "/tapes/include/seed-approvals.tape" + +Type "netclaw approvals list" +Enter +Sleep 2s +Screenshot screenshots/output/approvals-list.png +Sleep 2s diff --git a/screenshots/tapes/approvals-tui.tape b/screenshots/tapes/approvals-tui.tape new file mode 100644 index 0000000..834670a --- /dev/null +++ b/screenshots/tapes/approvals-tui.tape @@ -0,0 +1,28 @@ +# approvals-tui.png and approvals-tui-revoke-confirm.png: the approvals manager and its +# revoke confirmation, over the same grants as approvals-list.tape. +Set Shell "bash" +Set WaitTimeout 240s +Set Theme "Dracula" +Set FontSize 16 +Set Width 1000 +Set Height 430 +Set Padding 20 +Set TypingSpeed 40ms + +Source "/tapes/include/seed-approvals.tape" + +Type "netclaw approvals" +Enter +Sleep 3s +Screenshot screenshots/output/approvals-tui.png +Sleep 2s +Down +Sleep 500ms +Type "r" +Sleep 1s +Screenshot screenshots/output/approvals-tui-revoke-confirm.png +Sleep 2s +Escape +Sleep 500ms +Ctrl+Q +Sleep 1s diff --git a/screenshots/tapes/approvals.tape b/screenshots/tapes/approvals.tape deleted file mode 100644 index d19b914..0000000 --- a/screenshots/tapes/approvals.tape +++ /dev/null @@ -1,94 +0,0 @@ -# netclaw approvals — persistent tool approval management -# Seeds a realistic tool-approvals.json, then captures list + TUI + revoke flow -Output screenshots/output/approvals.gif - -Set Shell "bash" -Set Theme "Dracula" -Set Width 1200 -Set Height 700 -Set FontSize 16 -Set Padding 20 -Set TypingSpeed 50ms - -# Seed approvals data (hidden from recording) -Hide -Type "bash screenshots/scripts/seed-approvals.sh" -Enter -Sleep 500ms -Type "clear" -Enter -Sleep 500ms -Show - -# ═══════════════════════════════════════════════ -# SCENE 1: netclaw approvals list -# ═══════════════════════════════════════════════ - -Type "netclaw approvals list" -Enter -Sleep 2s - -Screenshot screenshots/output/approvals-list.png -Sleep 1s - -Type "clear" -Enter -Sleep 500ms - -# ═══════════════════════════════════════════════ -# SCENE 2: netclaw approvals list --json -# ═══════════════════════════════════════════════ - -Type "netclaw approvals list --json" -Enter -Sleep 2s - -Screenshot screenshots/output/approvals-list-json.png -Sleep 1s - -Type "clear" -Enter -Sleep 500ms - -# ═══════════════════════════════════════════════ -# SCENE 3: netclaw approvals TUI — browse and revoke -# ═══════════════════════════════════════════════ - -Type "netclaw approvals" -Enter -Sleep 3s - -# TUI list view — all approvals -Screenshot screenshots/output/approvals-tui.png -Sleep 500ms - -# Navigate down to an entry -Down -Sleep 300ms -Down -Sleep 300ms -Down -Sleep 300ms - -Screenshot screenshots/output/approvals-tui-selected.png -Sleep 500ms - -# Press Delete to initiate revoke -Delete -Sleep 1s - -# Revoke confirmation dialog -Screenshot screenshots/output/approvals-tui-revoke-confirm.png -Sleep 500ms - -# Confirm the revoke -Enter -Sleep 1s - -# Post-revoke state -Screenshot screenshots/output/approvals-tui-revoke-done.png -Sleep 1s - -# Exit TUI -Ctrl+Q -Sleep 1s diff --git a/screenshots/tapes/channel-audiences.tape b/screenshots/tapes/channel-audiences.tape new file mode 100644 index 0000000..3a702b6 --- /dev/null +++ b/screenshots/tapes/channel-audiences.tape @@ -0,0 +1,51 @@ +# netclaw config > Channels > Mattermost > Channels & Permissions: the per-channel audience screen +# stack: fresh +# The old ten-step init wizard had a Channels step. In 0.27.1 the same screen lives in `netclaw config`, once an adapter is enabled and has channels. Runs the +# real `netclaw init` out of sight, then enables a Mattermost adapter whose channels resolve against +# a small API fixture (scripts/mock-mattermost.py) so the rows show names, not raw IDs. +Output screenshots/output/channel-audiences.gif + +Set Shell "bash" +Set Theme "Dracula" +Set FontSize 16 +Set Width 1000 +Set Height 520 +Set Padding 20 +Set TypingSpeed 40ms + +Hide +Source "/tapes/include/init-personal.tape" +Type "nohup python3 /work/screenshots/scripts/mock-mattermost.py >/dev/null 2>&1 &" +Enter +Sleep 2s +Type "bash /work/screenshots/scripts/seed-channel-audiences.sh" +Enter +Sleep 25s +Type "clear" +Enter +Sleep 1s +Show + +Type "netclaw config" +Enter +Sleep 3s + +# 3. Channels, then Mattermost (the third adapter) +Down +Down +Sleep 400ms +Enter +Sleep 2s +Down +Down +Sleep 400ms +Enter +Sleep 3s + +# Manage channels and permissions +Enter +Sleep 4s +Screenshot screenshots/output/config-channel-audiences.png +Sleep 500ms +Ctrl+Q +Sleep 1s diff --git a/screenshots/tapes/chat-approval.tape b/screenshots/tapes/chat-approval.tape new file mode 100644 index 0000000..e3e4efe --- /dev/null +++ b/screenshots/tapes/chat-approval.tape @@ -0,0 +1,61 @@ +# Chat approval prompts: chat-approval-prompt.png (getting-started: home directory, five options), +# chat-tui-approval-prompt.png (cli/chat) and approval-prompt-tui.png (security/approvals). The last +# two run in the demo repository /home/netclaw/projects/website so all six options are offered. +# The session is idle at the prompt when the image is taken, so nothing is answered. +Set Shell "bash" +Set WaitTimeout 240s +Set Theme "Dracula" +Set FontSize 16 +Set Width 1000 +Set Height 540 +Set Padding 20 +Set TypingSpeed 40ms + +# First image: the getting-started prompt, run from the home directory (no repository), so the +# prompt offers five options, exactly as the tutorial's text block says. +Type "netclaw chat" +Enter +Sleep 3s +Type "Use curl to download https://example.com and save it as ~/example.html, then tell me how many bytes it is." +Enter +Wait+Screen@240s /5\. Deny/ +Sleep 1s +Screenshot screenshots/output/chat-approval-prompt.png +Sleep 2s +Ctrl+Q +Sleep 2s + +# The other two run inside the demo repository. +Hide +Type "bash screenshots/scripts/seed-projects.sh && cd ~/projects/website && clear" +Enter +Sleep 4s +Show + +Type "netclaw chat" +Enter +Sleep 3s +Type "Set the working directory to /home/netclaw/projects/website, then use shell_execute to run exactly: npm run build" +Enter +Wait+Screen@240s /6\. Deny/ +Sleep 1s +Screenshot screenshots/output/chat-tui-approval-prompt.png +Sleep 2s +Ctrl+Q +Sleep 2s + +Hide +Type "clear" +Enter +Show +Type "netclaw chat" +Enter +Sleep 3s +Type "Set the working directory to /home/netclaw/projects/website, then use shell_execute to run exactly: git push origin main" +Enter +Wait+Screen@240s /6\. Deny/ +Sleep 1s +Screenshot screenshots/output/approval-prompt-tui.png +Sleep 2s +Ctrl+Q +Sleep 2s diff --git a/screenshots/tapes/chat-first-greeting.tape b/screenshots/tapes/chat-first-greeting.tape new file mode 100644 index 0000000..3d685ad --- /dev/null +++ b/screenshots/tapes/chat-first-greeting.tape @@ -0,0 +1,44 @@ +# stack: fresh +# chat-first-greeting.png: the chat that `netclaw init` hands off to, with the agent's +# greeting and first question. Walks the real wizard (OpenAI-compatible provider, Personal +# posture) against the stack's inference endpoint; the wizard steps are not part of the image. +Set Shell "bash" +Set WaitTimeout 300s +Set Theme "Dracula" +Set FontSize 16 +Set Width 1000 +Set Height 600 +Set Padding 20 +Set TypingSpeed 40ms + +Hide +Type "netclaw init" +Enter +Sleep 4s +Type "6" +Sleep 2s +Backspace 30 +Type "http://llm.example.internal:8081" +Sleep 500ms +Enter +Sleep 2s +Enter +Wait+Screen@60s /Select a model/ +Enter +Sleep 2s +Enter +Sleep 2s +Enter +Sleep 2s +Type "Sam" +Enter +Sleep 2s +Enter +Sleep 2s +Enter +Wait+Screen@240s /Netclaw Chat/ +Wait+Screen@300s /Ready +\| +demo-model/ +Sleep 2s +Show +Screenshot screenshots/output/chat-first-greeting.png +Sleep 2s diff --git a/screenshots/tapes/chat-session.tape b/screenshots/tapes/chat-session.tape new file mode 100644 index 0000000..05a3f15 --- /dev/null +++ b/screenshots/tapes/chat-session.tape @@ -0,0 +1,44 @@ +# chat-tui-idle.png: a resumed session (replayed history, idle status bar) and +# slash-unknown-command.png: the reply to an unknown slash command (no model call). +Set Shell "bash" +Set WaitTimeout 240s +Set Theme "Dracula" +Set FontSize 16 +Set Width 1000 +Set Height 600 +Set Padding 20 +Set TypingSpeed 40ms + +Hide +Type "netclaw chat -p --resume demo-notes 'In two sentences, what is a good structure for weekly meeting notes?' > /dev/null && netclaw chat -p --resume demo-notes 'Turn that into a three-item checklist.' > /dev/null; echo SE''EDED" +Enter +Wait+Screen@240s /SEEDED/ +Type "clear" +Enter +Sleep 1s +Show + +Type "netclaw chat --resume demo-notes" +Enter +Wait+Screen@60s /End of history/ +Sleep 3s +Screenshot screenshots/output/chat-tui-idle.png +Sleep 2s +Ctrl+Q +Sleep 2s + +Hide +Type "clear" +Enter +Show +Type "netclaw chat" +Enter +Sleep 3s +Type "/nope" +Enter +Wait+Screen@30s /Unknown command/ +Sleep 2s +Screenshot screenshots/output/slash-unknown-command.png +Sleep 2s +Ctrl+Q +Sleep 2s diff --git a/screenshots/tapes/config.tape b/screenshots/tapes/config.tape new file mode 100644 index 0000000..eb010da --- /dev/null +++ b/screenshots/tapes/config.tape @@ -0,0 +1,133 @@ +# netclaw config: the settings dashboard and the areas the docs show, on the shared demo install +# stack: bare +# scripts/seed-demo-install.sh builds the install every status-like capture shares. 0.27.1 shows +# thirteen rows of the dashboard, all visible in a 520 px terminal. +# +# Produces: config-dashboard, config-retention, config-channels-menu, config-inbound-webhooks, config-skills, +# config-search, config-exposure. +Output screenshots/output/config.gif + +Set Shell "bash" +Set Theme "Dracula" +Set FontSize 16 +Set Width 1000 +Set Height 520 +Set Padding 20 +Set TypingSpeed 40ms + +Hide +Type "bash /work/screenshots/scripts/seed-demo-install.sh" +Enter +Wait+Screen@8m /seeded: demo install/ +Type "clear" +Enter +Sleep 1s +Show + +Type "netclaw config" +Enter +Sleep 3s +Screenshot screenshots/output/config-dashboard.png +Sleep 500ms + +# 11. Data Retention: a fresh install has no Retention key, so the row shows the default +Down +Down +Down +Down +Down +Down +Down +Down +Down +Down +Sleep 400ms +Enter +Sleep 2s +Screenshot screenshots/output/config-retention.png +Escape +Sleep 1s +Up +Up +Up +Up +Up +Up +Up +Up +Up +Up +Sleep 400ms + +# 3. Channels +Down +Down +Sleep 400ms +Enter +Sleep 2s +Screenshot screenshots/output/config-channels-menu.png +Escape +Sleep 1s + +# 4. Inbound Webhooks +Down +Down +Down +Sleep 400ms +Enter +Sleep 2s +Screenshot screenshots/output/config-inbound-webhooks.png +Escape +Sleep 1s + +# 5. Skill Sources +Down +Down +Down +Down +Sleep 400ms +Enter +Sleep 2s +Screenshot screenshots/output/config-skills.png +Escape +Sleep 1s + +# 6. Search +Down +Down +Down +Down +Down +Sleep 400ms +Enter +Sleep 2s +Screenshot screenshots/output/config-search.png +Escape +Sleep 1s + +# 9. Security & Access > Exposure Mode +Down +Down +Down +Down +Down +Down +Down +Down +Sleep 400ms +Enter +Sleep 1s +Down +Down +Down +Sleep 400ms +Enter +Sleep 2s +Screenshot screenshots/output/config-exposure.png +Escape +Sleep 1s +Escape +Sleep 1s + +Ctrl+Q +Sleep 1s diff --git a/screenshots/tapes/daemon-pair.tape b/screenshots/tapes/daemon-pair.tape new file mode 100644 index 0000000..d64b29f --- /dev/null +++ b/screenshots/tapes/daemon-pair.tape @@ -0,0 +1,32 @@ +# netclaw daemon pair and netclaw daemon devices: a pairing code and the paired-device list +# stack: fresh +# A device named alice-laptop is paired out of sight (scripts/seed-paired-device.sh runs the device +# side with its own NETCLAW_HOME), so the visible commands show a fresh code and a one-device list. +# The config seeded first (scripts/seed-providers.sh) only gives the CLI something to run against. +Output screenshots/output/daemon-pair.gif + +Set Shell "bash" +Set Theme "Dracula" +Set FontSize 16 +Set Width 1000 +Set Height 360 +Set Padding 20 +Set TypingSpeed 40ms + +Hide +Type "bash /work/screenshots/scripts/seed-providers.sh && bash /work/screenshots/scripts/seed-paired-device.sh" +Enter +Wait+Screen@3m /seeded: paired device/ +Type "clear" +Enter +Sleep 1s +Show + +Type "netclaw daemon pair" +Enter +Sleep 3s +Type "netclaw daemon devices" +Enter +Sleep 3s +Screenshot screenshots/output/daemon-pair.png +Sleep 500ms diff --git a/screenshots/tapes/docker-health.tape b/screenshots/tapes/docker-health.tape new file mode 100644 index 0000000..8eeb7a2 --- /dev/null +++ b/screenshots/tapes/docker-health.tape @@ -0,0 +1,44 @@ +# A netclaw container reaching healthy, as the Docker quick start checks it +# stack: fresh +# host-docker: true +# The only tape that talks to the host's Docker (the capture container gets the socket and the host's +# docker client, see compose/docker-host.yml). It starts its own container from the release image, +# gives it the name the docs use, and every command it prints is filtered to that one container, so +# nothing else on the host appears. Names its container -docker-health while it starts and +# renames it to `netclaw` just before the visible part; the end of the tape removes it (by label, so a container that merely shares the name is never touched). +Output screenshots/output/docker-health.gif + +Set Shell "bash" +Set Theme "Dracula" +Set FontSize 16 +Set Width 1600 +Set Height 300 +Set Padding 20 +Set TypingSpeed 40ms + +Hide +Type "bash /work/screenshots/scripts/docker-health-up.sh" +Enter +Wait+Screen@3m /container is healthy/ +Type "clear" +Enter +Sleep 1s +Show + +Type "docker ps --filter name=^netclaw$" +Enter +Sleep 2s +Type "docker inspect --format '{{.State.Health.Status}}' netclaw" +Enter +Sleep 2s +Type "docker exec netclaw netclaw daemon status" +Enter +Sleep 3s +Screenshot screenshots/output/docker-health.png +Sleep 500ms + +Hide +Type "bash /work/screenshots/scripts/docker-health-down.sh" +Enter +Sleep 5s +Show diff --git a/screenshots/tapes/doctor.tape b/screenshots/tapes/doctor.tape index 0fd471c..5618b05 100644 --- a/screenshots/tapes/doctor.tape +++ b/screenshots/tapes/doctor.tape @@ -1,16 +1,30 @@ -# netclaw doctor — configuration diagnostics +# netclaw doctor: all 24 checks and the exit-code line on a typical Personal install +# stack: bare +# Same demo install as status.tape (scripts/seed-demo-install.sh). A normal install has no secrets.json (nothing to put +# in it), so the one WARN is the real one and the exit code is 2. No check fails. The terminal is +# wide enough that no line wraps and tall enough for every check. Output screenshots/output/doctor.gif Set Shell "bash" Set Theme "Dracula" -Set Width 1200 +Set FontSize 16 +Set Width 1700 Set Height 700 -Set FontSize 14 Set Padding 20 -Set TypingSpeed 50ms +Set TypingSpeed 40ms -Type "netclaw doctor" +Hide +Type "bash /work/screenshots/scripts/seed-demo-install.sh" +Enter +Wait+Screen@8m /seeded: demo install/ +Sleep 20s +Type "clear" Enter -Sleep 5s +Sleep 1s +Show +Type "netclaw doctor" +Enter +Sleep 8s Screenshot screenshots/output/doctor.png +Sleep 500ms diff --git a/screenshots/tapes/help.tape b/screenshots/tapes/help.tape index 84f7667..5087f40 100644 --- a/screenshots/tapes/help.tape +++ b/screenshots/tapes/help.tape @@ -3,7 +3,7 @@ Output screenshots/output/help.png Set Shell "bash" Set Theme "Dracula" -Set Width 1200 +Set Width 1000 Set Height 600 Set FontSize 16 Set Padding 20 diff --git a/screenshots/tapes/include/init-personal.tape b/screenshots/tapes/include/init-personal.tape new file mode 100644 index 0000000..c12dfc7 --- /dev/null +++ b/screenshots/tapes/include/init-personal.tape @@ -0,0 +1,31 @@ +# Setup helper, not a tape: Source it between Hide and Show to run the real `netclaw init` wizard +# (Personal posture, Ollama, the stack's one model, identity "Netclaw" / "alice") against the +# first-run daemon. Leaves the shell at a clean prompt with a completed Personal setup, so a +# tape can start from what a reader has right after the quickstart. +# Needs `# stack: fresh` or `# stack: bare` (empty ~/.netclaw, Ollama on localhost:11434). +Type "netclaw init" +Enter +Sleep 3s +Type "4" +Sleep 1500ms +Enter +Sleep 5s +Enter +Sleep 2s +Enter +Sleep 1s +Enter +Sleep 1s +Type "alice" +Enter +Sleep 1s +Enter +Sleep 2s +Enter +# Health check, then the wizard hands over to chat +Sleep 14s +Ctrl+Q +Sleep 2s +Type "clear" +Enter +Sleep 1s diff --git a/screenshots/tapes/include/seed-approvals.tape b/screenshots/tapes/include/seed-approvals.tape new file mode 100644 index 0000000..3cbc357 --- /dev/null +++ b/screenshots/tapes/include/seed-approvals.tape @@ -0,0 +1,43 @@ +# Shared by approvals-list.tape and approvals-tui.tape (VHS `Source`). Everything between +# Hide and Show is not recorded. Builds the grants the approvals images show, through the +# real CLI and real prompt answers: two trust-verb grants, an MCP grant, a team grant, and +# a folder ("Always here") and a repository ("This repository") grant answered in the chat TUI. +Hide +Type "bash screenshots/scripts/seed-projects.sh && cd ~/projects/website && clear" +Enter +Sleep 2s +Type "netclaw approvals trust-verb 'git push' && netclaw approvals trust-verb gh && netclaw approvals trust-verb 'npm run' --audience team && netclaw approvals trust-verb demo-utilities/calculate --tool demo-utilities/calculate; echo SE''EDED" +Enter +Wait+Screen@60s /SEEDED/ +Type "clear" +Enter +Sleep 1s + +Type "netclaw chat" +Enter +Sleep 3s +Type "Call set_working_directory with /home/netclaw/projects/website, then stop. Run nothing else and ask me nothing." +Enter +Wait+Screen@240s /set_working_directory . \/home\/netclaw\/projects/ +Wait+Screen@240s /Ready +\| +demo-model/ +Sleep 1s +Type "Now use shell_execute to run exactly: npm run build" +Enter +Wait+Screen@240s /6\. Deny/ +Sleep 1s +Type "3" +Wait+Screen@240s /Ready +\| +demo-model/ +Sleep 1s +Type "Now use shell_execute to run exactly: make test" +Enter +Wait+Screen@240s /6\. Deny/ +Sleep 1s +Type "4" +Wait+Screen@240s /Ready +\| +demo-model/ +Sleep 1s +Ctrl+Q +Sleep 2s +Type "clear" +Enter +Sleep 1s +Show diff --git a/screenshots/tapes/init-existing.tape b/screenshots/tapes/init-existing.tape new file mode 100644 index 0000000..c59e6e5 --- /dev/null +++ b/screenshots/tapes/init-existing.tape @@ -0,0 +1,35 @@ +# netclaw init on an existing install: the action menu and the start-over scope chooser +# stack: fresh +# Runs the real wizard out of sight first, so the daemon has a completed Personal setup. +# Produces init-existing-menu and init-start-over-scope. +Output screenshots/output/init-existing.gif + +Set Shell "bash" +Set Theme "Dracula" +Set FontSize 16 +Set Width 1100 +Set Height 380 +Set Padding 20 +Set TypingSpeed 40ms + +Hide +Source "/tapes/include/init-personal.tape" +Show + +Type "netclaw init" +Enter +Sleep 3s +Screenshot screenshots/output/init-existing-menu.png +Sleep 500ms + +# Start over from scratch +Down +Sleep 300ms +Down +Sleep 300ms +Enter +Sleep 2s +Screenshot screenshots/output/init-start-over-scope.png +Sleep 500ms +Ctrl+Q +Sleep 1s diff --git a/screenshots/tapes/init-team.tape b/screenshots/tapes/init-team.tape new file mode 100644 index 0000000..781040b --- /dev/null +++ b/screenshots/tapes/init-team.tape @@ -0,0 +1,50 @@ +# netclaw init: the Team posture, which adds a feature-selection step ("Step 4 of 5") +# stack: fresh +# Same first-run state and local Ollama as init-wizard.tape. Produces init-04-enabled-features +# (feature selection, Team posture) (the five-step flow's fourth step). +Output screenshots/output/init-team.gif + +Set Shell "bash" +Set Theme "Dracula" +Set FontSize 16 +Set Width 1000 +Set Height 420 +Set Padding 20 +Set TypingSpeed 40ms + +Type "netclaw init" +Enter +Sleep 3s + +# Ollama, default endpoint, the one model +Type "4" +Sleep 1500ms +Enter +Sleep 5s +Enter +Sleep 2s + +# Identity: name, style, your name, timezone +Enter +Sleep 1s +Enter +Sleep 1s +Type "alice" +Enter +Sleep 1s +Enter +Sleep 2s + +# Security posture: Team +Down +Sleep 500ms +Enter +Sleep 2s + +# Feature selection: all six on for Team +Screenshot screenshots/output/init-04-enabled-features.png +Sleep 500ms +Enter +Sleep 3s +Ctrl+Q +Sleep 1s diff --git a/screenshots/tapes/init-wizard.tape b/screenshots/tapes/init-wizard.tape index 374cf33..d3dcec3 100644 --- a/screenshots/tapes/init-wizard.tape +++ b/screenshots/tapes/init-wizard.tape @@ -1,227 +1,83 @@ -# Netclaw Init Wizard - Full walkthrough (Personal + Ollama + DuckDuckGo) -# Requires: Ollama running on localhost:11434 with qwen2:0.5b -# Captures every wizard screen for documentation. -Output screenshots/output/init-full.gif +# netclaw init: the Personal walkthrough on 0.27.1 (provider, identity, security posture, health check) +# stack: fresh +# Runs against the first-run daemon (empty ~/.netclaw) with the stack's local Ollama on +# localhost:11434 (qwen2:0.5b, the stack's default). Nothing private is involved. +# +# The header reads "Step N of 5" until a posture is chosen, then "Step 4 of 4" for Personal. +# That is what 0.27.1 prints, so the provider, identity and posture images say "of 5" and the +# health check says "of 4". +# +# Produces: init-01-provider-list, init-01-endpoint, init-01-provider-ollama, init-01-model-select, +# init-02-identity, init-02-security-posture, init-05-health-check. +Output screenshots/output/init-wizard.gif Set Shell "bash" Set Theme "Dracula" -Set Width 1200 -Set Height 700 Set FontSize 16 +Set Width 1000 +Set Height 520 Set Padding 20 -Set TypingSpeed 50ms - -# Clean netclaw home for fresh wizard -Hide -Type "export NETCLAW_HOME=/tmp/netclaw-screenshot-home" -Enter -Sleep 300ms -Type "rm -rf /tmp/netclaw-screenshot-home && mkdir -p /tmp/netclaw-screenshot-home" -Enter -Sleep 300ms -Type "clear" -Enter -Sleep 500ms -Show - -# ═══════════════════════════════════════════════ -# STEP 1: LLM Provider -# ═══════════════════════════════════════════════ +Set TypingSpeed 40ms Type "netclaw init" Enter Sleep 3s -# Sub-step 0: Provider list (Anthropic highlighted by default) +# Provider list: eight entries Screenshot screenshots/output/init-01-provider-list.png -Sleep 300ms +Sleep 500ms -# Move to Ollama and capture (0.22 provider list: Anthropic, GitHub Copilot, Ollama) -Down -Sleep 300ms -Down -Sleep 300ms +# 4 = Ollama: its endpoint step +Type "4" +Sleep 1500ms Screenshot screenshots/output/init-01-provider-ollama.png -Sleep 300ms - -# Select Ollama -Enter -Sleep 2s - -# Sub-step 2: Endpoint input (default http://localhost:11434) +Sleep 500ms +Escape +Sleep 800ms +Escape +Sleep 1s + +# 6 = OpenAI-compatible: the endpoint field arrives pre-filled with http://localhost:11434, which +# is the stack's Ollama (it serves the OpenAI-compatible API too), so the wizard carries on with it. +Type "6" +Sleep 1500ms Screenshot screenshots/output/init-01-endpoint.png -Sleep 300ms +Sleep 500ms -# Accept default endpoint +# Accept the endpoint, skip the optional API key, wait for the connectivity check and model discovery +Enter +Sleep 1500ms Enter Sleep 5s - -# Sub-step 3/4: Model discovery + selection (qwen2:0.5b found) Screenshot screenshots/output/init-01-model-select.png -Sleep 300ms - -# Select qwen2:0.5b -Enter -Sleep 2s - -# ═══════════════════════════════════════════════ -# STEP 2: Security Posture -# ═══════════════════════════════════════════════ -Screenshot screenshots/output/init-02-security-posture.png -Sleep 300ms - -# Select Personal (already highlighted) -Enter -Sleep 2s - -# (Feature Selection is SKIPPED for Personal posture) - -# ═══════════════════════════════════════════════ -# STEP 3: Communication Channels (Channel Picker) -# ═══════════════════════════════════════════════ -Screenshot screenshots/output/init-03-channels.png -Sleep 300ms - -# Press 'd' to skip channels (done without selecting any) -Type "d" -Sleep 2s - -# ═══════════════════════════════════════════════ -# STEP 4: Web Search -# ═══════════════════════════════════════════════ -Screenshot screenshots/output/init-04-search.png -Sleep 300ms - -# Select DuckDuckGo (default, first item) -Enter -Sleep 2s - -# ═══════════════════════════════════════════════ -# STEP 5: Browser Automation -# ═══════════════════════════════════════════════ -Screenshot screenshots/output/init-05-browser.png -Sleep 300ms - -# Select "No" (skip browser automation, first item) -Enter -Sleep 2s - -# ═══════════════════════════════════════════════ -# STEP 6: Identity -# ═══════════════════════════════════════════════ - -# Sub-step 0: Agent name -Screenshot screenshots/output/init-06-identity-name.png -Sleep 300ms - -# Accept default name "Netclaw" -Enter -Sleep 2s - -# Sub-step 1: Communication style -Screenshot screenshots/output/init-06-identity-style.png -Sleep 300ms - -# Select "Concise & casual" (first item) -Enter -Sleep 2s - -# Sub-step 2: User name -Screenshot screenshots/output/init-06-identity-user.png -Sleep 300ms - -# Type a name -Type "Aaron" -Sleep 300ms -Enter -Sleep 2s - -# Sub-step 3: Timezone -Screenshot screenshots/output/init-06-identity-timezone.png -Sleep 300ms - -# Accept default timezone -Enter -Sleep 2s - -# Sub-step 4: Workspaces directory -Screenshot screenshots/output/init-06-identity-workspaces.png -Sleep 300ms - -# Accept default -Enter -Sleep 2s - -# Sub-step 5: Webhook URL (optional) -Screenshot screenshots/output/init-06-identity-webhook.png -Sleep 300ms - -# Skip (leave blank) +Sleep 500ms Enter Sleep 2s -# ═══════════════════════════════════════════════ -# STEP 7: External Skills (conditional - Claude Code detected) -# ═══════════════════════════════════════════════ -Screenshot screenshots/output/init-07-external-skills.png -Sleep 300ms - -# Accept detected skills and continue +# Identity: name, style, your name, timezone Enter -Sleep 2s - -# Sub-step 1: Custom skill directory path (optional) -Screenshot screenshots/output/init-07-custom-skills-path.png -Sleep 300ms - -# Skip (leave blank) +Sleep 1500ms +Screenshot screenshots/output/init-02-identity.png +Sleep 500ms Enter -Sleep 2s - -# ═══════════════════════════════════════════════ -# STEP 8: Skill Feeds -# ═══════════════════════════════════════════════ -Screenshot screenshots/output/init-08-skill-feeds.png -Sleep 300ms - -# Select "No — skip" (second item) -Down -Sleep 300ms +Sleep 1s +Type "alice" Enter -Sleep 2s - -# ═══════════════════════════════════════════════ -# STEP 9: Network Exposure -# ═══════════════════════════════════════════════ -Screenshot screenshots/output/init-09-exposure.png -Sleep 300ms - -# Select "Local" (first item, recommended) +Sleep 1s Enter Sleep 2s -# Sub-step: Inbound Webhooks toggle -Screenshot screenshots/output/init-09-webhooks.png -Sleep 300ms - -# Select "No" (second item) -Down -Sleep 300ms +# Security posture: Personal is highlighted +Screenshot screenshots/output/init-02-security-posture.png +Sleep 500ms Enter -Sleep 2s - -# ═══════════════════════════════════════════════ -# STEP 10: Health Check -# ═══════════════════════════════════════════════ -Screenshot screenshots/output/init-10-health-check.png -Sleep 300ms -# Press Enter to start health check -Enter +# Health check starts on its own and hands over to chat within a couple of seconds +# The finished checklist is on screen for about a second and a half before chat takes over, +# so wait for its last line instead of sleeping a fixed time. +Wait+Screen@60s /Configuration written/ +Sleep 200ms +Screenshot screenshots/output/init-05-health-check.png Sleep 8s - -# Capture final health check results -Screenshot screenshots/output/init-10-health-check-complete.png - -Sleep 2s -Ctrl+C +Ctrl+Q +Sleep 1s diff --git a/screenshots/tapes/mcp-tools.tape b/screenshots/tapes/mcp-tools.tape index 0eebabb..54aed89 100644 --- a/screenshots/tapes/mcp-tools.tape +++ b/screenshots/tapes/mcp-tools.tape @@ -1,45 +1,42 @@ -# netclaw mcp tools — MCP tool permissions TUI -# Shows server list, then tool grid with audience toggling -Output screenshots/output/mcp-tools.gif - +# MCP Permissions TUI (`netclaw mcp permissions`): server list, then the tool grid for the +# demo-utilities server under the Personal, Team and Public audiences, plus the Team "Grant all" prompt. Set Shell "bash" +Set WaitTimeout 60s Set Theme "Dracula" -Set Width 1200 -Set Height 700 Set FontSize 16 +Set Width 1000 +Set Height 560 Set Padding 20 -Set TypingSpeed 50ms +Set TypingSpeed 40ms -Type "netclaw mcp tools" +Type "netclaw mcp permissions" Enter -Sleep 3s - -# Server list showing connected MCP servers +Wait+Screen /demo-utilities/ +Sleep 1s Screenshot screenshots/output/mcp-tools-server-list.png -Sleep 300ms +Sleep 1s -# Select first server to see tool grid Enter Sleep 2s - -# Tool grid — Personal audience (default) Screenshot screenshots/output/mcp-tools-personal.png -Sleep 300ms +Sleep 1s -# Switch to Team audience (right arrow) Right Sleep 1s Screenshot screenshots/output/mcp-tools-team.png -Sleep 300ms +Sleep 1s -# Switch to Public audience (right arrow again) -Right +# E on a server that is not enabled for Team asks before granting anything +Type "e" +Sleep 1s +Screenshot screenshots/output/mcp-tools-confirm.png +Sleep 1s +Type "n" Sleep 1s -Screenshot screenshots/output/mcp-tools-public.png -Sleep 300ms -# Return to server list and exit -Escape +Right Sleep 1s -Escape +Screenshot screenshots/output/mcp-tools-public.png Sleep 1s + +Ctrl+Q diff --git a/screenshots/tapes/model-properties.tape b/screenshots/tapes/model-properties.tape new file mode 100644 index 0000000..ea74e9a --- /dev/null +++ b/screenshots/tapes/model-properties.tape @@ -0,0 +1,35 @@ +# model-properties.png: press P on Main in `netclaw model` to open the Model properties panel. +# stack: bare +# Same demo install as model.tape. lab-llama gets ThinkingDialect ReasoningEffort so the thinking +# level field is editable, as it is for a provider that offers it. +Output screenshots/output/model-properties.gif + +Set Shell "bash" +Set Theme "Dracula" +Set FontSize 16 +Set Width 1000 +Set Height 460 +Set Padding 20 +Set TypingSpeed 40ms + +Hide +Type "bash /work/screenshots/scripts/seed-demo-install.sh" +Enter +Wait+Screen@8m /seeded: demo install/ +Type `python3 -c "import json,os;p=os.path.expanduser('~/.netclaw/config/netclaw.json');c=json.load(open(p));c['Providers']['lab-llama'].setdefault('VendorOptions',{})['ThinkingDialect']='ReasoningEffort';json.dump(c,open(p,'w'),indent=2)"` +Enter +Sleep 8s +Type "clear" +Enter +Sleep 1s +Show + +Type "netclaw model" +Enter +Sleep 4s +Type "p" +Sleep 5s +Screenshot screenshots/output/model-properties.png +Sleep 500ms +Ctrl+Q +Sleep 1s diff --git a/screenshots/tapes/model.tape b/screenshots/tapes/model.tape index bfd14a9..79c79e7 100644 --- a/screenshots/tapes/model.tape +++ b/screenshots/tapes/model.tape @@ -1,20 +1,32 @@ -# netclaw model — model management TUI +# netclaw model: the model manager with Main, Fallback and Compaction all assigned +# stack: bare +# Same three neutral providers as provider.tape (scripts/seed-demo-install.sh). The Provider column is capped +# at 40 characters, so the long OpenAI-compatible label ends in an ellipsis at any width. +# +# Produces model-manager. Output screenshots/output/model.gif Set Shell "bash" Set Theme "Dracula" -Set Width 1200 -Set Height 700 Set FontSize 16 +Set Width 1000 +Set Height 340 Set Padding 20 -Set TypingSpeed 50ms +Set TypingSpeed 40ms -Type "netclaw model" +Hide +Type "bash /work/screenshots/scripts/seed-demo-install.sh" +Enter +Wait+Screen@8m /seeded: demo install/ +Type "clear" Enter -Sleep 3s +Sleep 1s +Show +Type "netclaw model" +Enter +Sleep 4s Screenshot screenshots/output/model-manager.png Sleep 500ms - -Escape +Ctrl+Q Sleep 1s diff --git a/screenshots/tapes/provider.tape b/screenshots/tapes/provider.tape index 9cdbaa7..13859ce 100644 --- a/screenshots/tapes/provider.tape +++ b/screenshots/tapes/provider.tape @@ -1,20 +1,63 @@ -# netclaw provider — provider management TUI +# netclaw provider: the provider manager, and its "add a provider" flow for OpenAI-compatible +# stack: bare +# Three providers with neutral names (scripts/seed-demo-install.sh). my-deepseek has a dummy key, so +# it shows the warning glyph. lab-llama is answered by the stack's forwarder when +# INFERENCE_UPSTREAM is set. The Provider column is capped at 40 characters (in builds after +# 0.27.1), so the long OpenAI-compatible label ends in an ellipsis at any width. +# +# Produces provider-manager and provider-add-openai-compatible. Output screenshots/output/provider.gif Set Shell "bash" Set Theme "Dracula" -Set Width 1200 -Set Height 700 Set FontSize 16 +Set Width 1100 +Set Height 500 Set Padding 20 -Set TypingSpeed 50ms +Set TypingSpeed 40ms -Type "netclaw provider" +Hide +Type "bash /work/screenshots/scripts/seed-demo-install.sh" +Enter +Wait+Screen@8m /seeded: demo install/ +Type "clear" Enter -Sleep 3s +Sleep 1s +Show +Type "netclaw provider" +Enter +# Wait for the health probes to settle +Sleep 8s +Up +Up +Sleep 500ms Screenshot screenshots/output/provider-manager.png Sleep 500ms -Escape +# + Add new provider... > OpenAI-compatible > name > the Endpoint step +Down +Down +Down +Down +Down +Down +Down +Down +Sleep 500ms +Enter +Sleep 2s +Down +Down +Down +Down +Down +Sleep 500ms +Enter +Sleep 2s +Enter +Sleep 2s +Screenshot screenshots/output/provider-add-openai-compatible.png +Sleep 500ms +Ctrl+Q Sleep 1s diff --git a/screenshots/tapes/reminder-list.tape b/screenshots/tapes/reminder-list.tape new file mode 100644 index 0000000..2986021 --- /dev/null +++ b/screenshots/tapes/reminder-list.tape @@ -0,0 +1,26 @@ +# reminder-list.png: `netclaw reminder list` over two neutral reminders. +# `reminder list` prints a table with one row per reminder; 1350 px keeps each row on one line. +# stack: bare +Set Shell "bash" +Set WaitTimeout 120s +Set Theme "Dracula" +Set FontSize 16 +Set Width 1350 +Set Height 260 +Set Padding 20 +Set TypingSpeed 40ms + +Hide +Type "bash /work/screenshots/scripts/seed-demo-install.sh" +Enter +Wait+Screen@8m /seeded: demo install/ +Type "clear" +Enter +Sleep 1s +Show + +Type "netclaw reminder list" +Enter +Sleep 2s +Screenshot screenshots/output/reminder-list.png +Sleep 2s diff --git a/screenshots/tapes/reminder-run.tape b/screenshots/tapes/reminder-run.tape new file mode 100644 index 0000000..5c5c5e6 --- /dev/null +++ b/screenshots/tapes/reminder-run.tape @@ -0,0 +1,34 @@ +# reminder-run-approval.png: `netclaw reminder run disk-note` stopped at the approval prompt +# for the reminder's shell call, with "Always here" highlighted. Nothing is answered. +Set Shell "bash" +Set WaitTimeout 240s +Set Theme "Dracula" +Set FontSize 16 +Set Width 1000 +Set Height 700 +Set Padding 20 +Set TypingSpeed 40ms + +Hide +Type "netclaw reminder create disk-note cron '0 8 * * *' 'Use shell_execute to run exactly: touch /tmp/disk-note.txt' --name 'Disk note' > /dev/null; clear" +Enter +Sleep 2s +Show + +Type "netclaw reminder run disk-note" +Enter +Sleep 5s +Wait+Screen@240s /Ready +\| +demo-model/ +Sleep 1s +Type "Run the reminder disk-note now." +Enter +Wait+Screen@240s /5\. Deny/ +Sleep 1s +Down +Sleep 300ms +Down +Sleep 1s +Screenshot screenshots/output/reminder-run-approval.png +Sleep 2s +Ctrl+Q +Sleep 1s diff --git a/screenshots/tapes/reminder.tape b/screenshots/tapes/reminder.tape index c8e9108..bde2cd7 100644 --- a/screenshots/tapes/reminder.tape +++ b/screenshots/tapes/reminder.tape @@ -1,16 +1,46 @@ -# netclaw reminder list — scheduled reminders +# reminder.gif: the Reminder Builder from the first step to Create, about 15 seconds. +# reminder-builder-review.png: the review step, taken just before Create. +# The recording starts inside the builder (the launch is hidden) and ends on the result. Output screenshots/output/reminder.gif Set Shell "bash" +Set WaitTimeout 60s Set Theme "Dracula" -Set Width 1200 -Set Height 600 Set FontSize 16 +Set Width 1000 +Set Height 440 Set Padding 20 -Set TypingSpeed 50ms +Set TypingSpeed 45ms -Type "netclaw reminder list" +Hide +Type "netclaw reminder ui" Enter -Sleep 3s +Wait+Screen /Step 1 of 6/ +Show -Screenshot screenshots/output/reminder-list.png +Sleep 300ms +Type "daily-uptime-check" +Sleep 600ms +Enter +Sleep 1200ms +Down +Sleep 400ms +Down +Sleep 600ms +Enter +Sleep 1200ms +Type "0 9 * * *" +Sleep 600ms +Enter +Sleep 1200ms +Type "Check that the website and API respond." +Sleep 600ms +Enter +Sleep 1200ms +Enter +Wait+Screen /Step 6 of 6/ +Sleep 1500ms +Screenshot screenshots/output/reminder-builder-review.png +Sleep 2s +Enter +Sleep 2500ms diff --git a/screenshots/tapes/sessions.tape b/screenshots/tapes/sessions.tape new file mode 100644 index 0000000..33bf2f3 --- /dev/null +++ b/screenshots/tapes/sessions.tape @@ -0,0 +1,79 @@ +# netclaw sessions: the sessions browser with three titled conversations +# stack: bare +# Needs real inference: set INFERENCE_UPSTREAM (see compose/forwarder.yml), because the stack's +# 0.5B Ollama model can't use tools. seed-demo-install.sh builds the shared demo install, then the tape holds +# three short chats out of sight (each one a session), which takes a few minutes. +# +# Produces sessions-tui. Same three chats as stats.tape, in a shorter window. +Output screenshots/output/sessions.gif + +Set Shell "bash" +Set Theme "Dracula" +Set FontSize 16 +Set Width 1000 +Set Height 320 +Set Padding 20 +Set TypingSpeed 40ms + +Hide +Type "bash /work/screenshots/scripts/seed-demo-install.sh" +Enter +Wait+Screen@8m /seeded: demo install/ +Type "clear" +Enter +Sleep 1s + +# Conversation demo-notes: an interactive chat, so the daemon has time to title the session +Type "netclaw chat" +Enter +Sleep 6s +Type "Give me a three-item checklist for tidying a desk." +Enter +Sleep 6s +Wait+Screen@4m /Ready/ +Sleep 30s +Ctrl+Q +Sleep 2s +Type "clear" +Enter +Sleep 1s + +# Conversation weekly-report: an interactive chat, so the daemon has time to title the session +Type "netclaw chat" +Enter +Sleep 6s +Type "Write a two-sentence status update: the migration is finished and docs work starts Monday." +Enter +Sleep 6s +Wait+Screen@4m /Ready/ +Sleep 30s +Ctrl+Q +Sleep 2s +Type "clear" +Enter +Sleep 1s + +# Conversation trip-planning: an interactive chat, so the daemon has time to title the session +Type "netclaw chat" +Enter +Sleep 6s +Type "Suggest a name for a small weekend hiking club." +Enter +Sleep 6s +Wait+Screen@4m /Ready/ +Sleep 30s +Ctrl+Q +Sleep 2s +Type "clear" +Enter +Sleep 1s + +Show + +Type "netclaw sessions" +Enter +Sleep 3s +Screenshot screenshots/output/sessions-tui.png +Sleep 500ms +Ctrl+Q +Sleep 1s diff --git a/screenshots/tapes/single-shot-cli-execution.tape b/screenshots/tapes/single-shot-cli-execution.tape new file mode 100644 index 0000000..ff63ff5 --- /dev/null +++ b/screenshots/tapes/single-shot-cli-execution.tape @@ -0,0 +1,31 @@ +# netclaw chat -p: a one-shot headless prompt that calls a tool, with usage lines +# stack: bare +# Needs real inference (INFERENCE_UPSTREAM, see compose/forwarder.yml): the tool calls come from +# the model. The prompt is benign and the answer comes from the stack's demo MCP server. The tape +# waits for the shell prompt to come back, because how long the model takes isn't fixed. +Output screenshots/output/single-shot-cli-execution.gif + +Set Shell "bash" +Set Theme "Dracula" +Set FontSize 16 +Set Width 1000 +Set Height 420 +Set Padding 20 +Set TypingSpeed 40ms + +Hide +Type "bash /work/screenshots/scripts/seed-demo-install.sh" +Enter +Wait+Screen@8m /seeded: demo install/ +Type "clear" +Enter +Sleep 1s +Show + +Type `netclaw chat -p "Search your tools for a UUID generator and answer in under fifteen words."` +Enter +Sleep 10s +Wait+Screen@5m /(?m)^> *$/ +Sleep 1s +Screenshot screenshots/output/single-shot-cli-execution.png +Sleep 500ms diff --git a/screenshots/tapes/skill-list.tape b/screenshots/tapes/skill-list.tape new file mode 100644 index 0000000..cd66e1f --- /dev/null +++ b/screenshots/tapes/skill-list.tape @@ -0,0 +1,25 @@ +# skill-list.png: `netclaw skill list` with local skills, the system skills and two feed skills. +# stack: bare +Set Shell "bash" +Set WaitTimeout 120s +Set Theme "Dracula" +Set FontSize 16 +Set Width 1000 +Set Height 520 +Set Padding 20 +Set TypingSpeed 40ms + +Hide +Type "bash /work/screenshots/scripts/seed-demo-install.sh" +Enter +Wait+Screen@8m /seeded: demo install/ +Type "clear" +Enter +Sleep 1s +Show + +Type "netclaw skill list" +Enter +Sleep 3s +Screenshot screenshots/output/skill-list.png +Sleep 2s diff --git a/screenshots/tapes/skill-sync.tape b/screenshots/tapes/skill-sync.tape new file mode 100644 index 0000000..47ac038 --- /dev/null +++ b/screenshots/tapes/skill-sync.tape @@ -0,0 +1,26 @@ +# skill-sync.png: `netclaw skill sync` against the stack's skill server (feed `corp-skills`). +# stack: bare +# The first sync runs hidden so the recorded one reports the steady state. +Set Shell "bash" +Set WaitTimeout 120s +Set Theme "Dracula" +Set FontSize 16 +Set Width 1000 +Set Height 300 +Set Padding 20 +Set TypingSpeed 40ms + +Hide +Type "bash /work/screenshots/scripts/seed-demo-install.sh" +Enter +Wait+Screen@8m /seeded: demo install/ +Type "clear" +Enter +Sleep 1s +Show + +Type "netclaw skill sync" +Enter +Sleep 6s +Screenshot screenshots/output/skill-sync.png +Sleep 2s diff --git a/screenshots/tapes/stats.tape b/screenshots/tapes/stats.tape index 7d34595..fb73bc4 100644 --- a/screenshots/tapes/stats.tape +++ b/screenshots/tapes/stats.tape @@ -1,16 +1,78 @@ -# netclaw stats — usage statistics +# netclaw stats, after a handful of real conversations +# stack: bare +# Needs real inference: set INFERENCE_UPSTREAM (see compose/forwarder.yml), because the stack's +# 0.5B Ollama model can't use tools. seed-demo-install.sh builds the shared demo install, then the tape holds +# three short chats out of sight (each one a session), which takes a few minutes. +# +# Produces stats. sessions.tape holds the same conversations and photographs the sessions browser. Output screenshots/output/stats.gif Set Shell "bash" Set Theme "Dracula" -Set Width 1200 -Set Height 600 Set FontSize 16 +Set Width 1000 +Set Height 640 Set Padding 20 -Set TypingSpeed 50ms +Set TypingSpeed 40ms + +Hide +Type "bash /work/screenshots/scripts/seed-demo-install.sh" +Enter +Wait+Screen@8m /seeded: demo install/ +Type "clear" +Enter +Sleep 1s + +# Conversation demo-notes: an interactive chat, so the daemon has time to title the session +Type "netclaw chat" +Enter +Sleep 6s +Type "Give me a three-item checklist for tidying a desk." +Enter +Sleep 6s +Wait+Screen@4m /Ready/ +Sleep 30s +Ctrl+Q +Sleep 2s +Type "clear" +Enter +Sleep 1s + +# Conversation weekly-report: an interactive chat, so the daemon has time to title the session +Type "netclaw chat" +Enter +Sleep 6s +Type "Write a two-sentence status update: the migration is finished and docs work starts Monday." +Enter +Sleep 6s +Wait+Screen@4m /Ready/ +Sleep 30s +Ctrl+Q +Sleep 2s +Type "clear" +Enter +Sleep 1s + +# Conversation trip-planning: an interactive chat, so the daemon has time to title the session +Type "netclaw chat" +Enter +Sleep 6s +Type "Suggest a name for a small weekend hiking club." +Enter +Sleep 6s +Wait+Screen@4m /Ready/ +Sleep 30s +Ctrl+Q +Sleep 2s +Type "clear" +Enter +Sleep 1s + +Show Type "netclaw stats" Enter Sleep 3s - Screenshot screenshots/output/stats.png +Sleep 500ms + diff --git a/screenshots/tapes/status.tape b/screenshots/tapes/status.tape index 486421a..7bdbd65 100644 --- a/screenshots/tapes/status.tape +++ b/screenshots/tapes/status.tape @@ -1,16 +1,31 @@ -# netclaw status — daemon health and connector status +# netclaw status: a healthy daemon with its connectors, one of them an MCP server +# stack: bare +# seed-demo-install.sh builds the one demo install every status-like capture shares (providers, +# model roles, the demo MCP server, skills, webhooks, reminders), written with the real CLI. The model endpoint is the stack's forwarder +# (INFERENCE_UPSTREAM) when set. Nothing in the output names it. Output screenshots/output/status.gif Set Shell "bash" Set Theme "Dracula" -Set Width 1200 -Set Height 600 Set FontSize 16 +Set Width 1000 +Set Height 600 Set Padding 20 -Set TypingSpeed 50ms +Set TypingSpeed 40ms + +Hide +Type "bash /work/screenshots/scripts/seed-demo-install.sh" +Enter +Wait+Screen@8m /seeded: demo install/ +# Let the daemon run for a minute or two so the uptime reads like a real one +Sleep 75s +Type "clear" +Enter +Sleep 1s +Show Type "netclaw status" Enter Sleep 3s - Screenshot screenshots/output/status.png +Sleep 500ms diff --git a/screenshots/tapes/update-check.tape b/screenshots/tapes/update-check.tape new file mode 100644 index 0000000..6232986 --- /dev/null +++ b/screenshots/tapes/update-check.tape @@ -0,0 +1,24 @@ +# netclaw update --check on an install that is one release behind +# stack: fresh +# image: 0.28.1 +# This is the one tape that does not run the documented release: it runs the previous one (0.28.1), +# which is genuinely a version behind, so the check reports a real update to v0.28.2 instead of a +# staged one. The visible `netclaw --version` makes that plain. +Output screenshots/output/update-check.gif + +Set Shell "bash" +Set Theme "Dracula" +Set FontSize 16 +Set Width 1000 +Set Height 200 +Set Padding 20 +Set TypingSpeed 40ms + +Type "netclaw --version" +Enter +Sleep 2s +Type "netclaw update --check" +Enter +Sleep 5s +Screenshot screenshots/output/update-check.png +Sleep 500ms diff --git a/screenshots/tapes/webhooks.tape b/screenshots/tapes/webhooks.tape index dd011b8..ccb2bd4 100644 --- a/screenshots/tapes/webhooks.tape +++ b/screenshots/tapes/webhooks.tape @@ -1,16 +1,25 @@ -# netclaw webhooks list — inbound webhook routes -Output screenshots/output/webhooks.gif - +# webhooks-list.png: `netclaw webhooks list` with two routes (see scripts/seed-webhooks.sh). +# stack: bare Set Shell "bash" +Set WaitTimeout 120s Set Theme "Dracula" -Set Width 1200 -Set Height 600 Set FontSize 16 +Set Width 1000 +Set Height 300 Set Padding 20 -Set TypingSpeed 50ms +Set TypingSpeed 40ms -Type "netclaw webhooks list" +Hide +Type "bash /work/screenshots/scripts/seed-demo-install.sh" +Enter +Wait+Screen@8m /seeded: demo install/ +Type "clear" Enter -Sleep 3s +Sleep 1s +Show +Type "netclaw webhooks list" +Enter +Sleep 2s Screenshot screenshots/output/webhooks-list.png +Sleep 2s diff --git a/scripts/check-links.mjs b/scripts/check-links.mjs new file mode 100644 index 0000000..d1116c6 --- /dev/null +++ b/scripts/check-links.mjs @@ -0,0 +1,75 @@ +// Crawls dist/ and verifies every internal link, image and #anchor resolves. +// Complements the external link-validator in CI (which does not check anchors). +// Usage: npm run check-links (after npm run build) +import { readFileSync, readdirSync, statSync, existsSync } from 'node:fs'; +import { join, dirname, resolve, relative } from 'node:path'; +import { fileURLToPath } from 'node:url'; + +const ROOT = join(dirname(fileURLToPath(import.meta.url)), '..'); +const DIST = join(ROOT, 'dist'); +if (!existsSync(DIST)) { + console.error('dist/ not found. Run `npm run build` first.'); + process.exit(2); +} + +const walk = (dir) => + readdirSync(dir).flatMap((n) => { + const p = join(dir, n); + return statSync(p).isDirectory() ? walk(p) : [p]; + }); + +const pages = walk(DIST).filter((f) => f.endsWith('.html')); +const ids = new Map(); // abs html path -> Set of ids +const decode = (s) => s.replace(/&/g, '&').replace(/"/g, '"').replace(/'/g, "'"); + +for (const f of pages) { + const html = readFileSync(f, 'utf8'); + const set = new Set(); + for (const m of html.matchAll(/\s(?:id|name)=("([^"]*)"|'([^']*)'|([^\s>]+))/g)) { + set.add(decode(m[2] ?? m[3] ?? m[4])); + } + ids.set(f, set); +} + +function resolveTarget(pathname, fromFile) { + let p = pathname.startsWith('/') ? join(DIST, pathname) : resolve(dirname(fromFile), pathname); + try { + p = decodeURIComponent(p); + } catch {} + return [p, join(p, 'index.html'), p + '.html'].find((c) => existsSync(c) && statSync(c).isFile()); +} + +const problems = []; +let checked = 0; +for (const f of pages) { + const html = readFileSync(f, 'utf8'); + const rel = '/' + relative(DIST, f).replace(/index\.html$/, ''); + for (const m of html.matchAll(/<(a|img|source)\s[^>]*?(href|src)=("([^"]*)"|'([^']*)')/g)) { + const raw = decode(m[4] ?? m[5]); + if (!raw || /^(https?:|mailto:|tel:|data:|javascript:|\/\/)/.test(raw)) continue; + checked++; + const [beforeHash, hash] = raw.split('#'); + const [pathname] = beforeHash.split('?'); + const target = pathname === '' ? f : resolveTarget(pathname, f); + if (!target) { + problems.push(`${rel} missing target: ${raw}`); + continue; + } + if (hash && target.endsWith('.html')) { + let h = hash; + try { + h = decodeURIComponent(hash); + } catch {} + if (!ids.get(target)?.has(h)) problems.push(`${rel} missing anchor: ${raw}`); + } + } +} + +const unique = [...new Set(problems)]; +console.log(`Checked ${checked} internal references across ${pages.length} pages.`); +if (unique.length) { + console.error(`\n${unique.length} broken:`); + for (const p of unique) console.error(' ' + p); + process.exit(1); +} +console.log('No broken internal links or anchors.'); diff --git a/scripts/check-rendered-pages.mjs b/scripts/check-rendered-pages.mjs new file mode 100644 index 0000000..88e2e9b --- /dev/null +++ b/scripts/check-rendered-pages.mjs @@ -0,0 +1,81 @@ +// Post-build guard. Fails the build when a built docs page is empty or missing diagrams. +// +// Why: Mermaid fences are rendered at build time through a headless browser +// (rehype-mermaid, inline-svg). When the browser is not installed the build still +// succeeds, but every page with a diagram ships with an EMPTY body. This script +// catches that, and any other page whose main content collapsed. +// +// Checks, for every page in src/content/docs: +// 1. dist//index.html exists +// 2. main content has at least MIN_WORDS words of prose (SVG, scripts and tags stripped) +// 3. each ```mermaid fence in the source has a rendered +// +// Usage: node scripts/check-rendered-pages.mjs [--verbose] +import { readFileSync, readdirSync, statSync, existsSync } from 'node:fs'; +import { join, dirname, relative } from 'node:path'; +import { fileURLToPath } from 'node:url'; + +const ROOT = join(dirname(fileURLToPath(import.meta.url)), '..'); +const DOCS = join(ROOT, 'src', 'content', 'docs'); +const DIST = join(ROOT, 'dist'); +const MIN_WORDS = 150; +const verbose = process.argv.includes('--verbose'); + +const walk = (dir) => + readdirSync(dir).flatMap((n) => { + const p = join(dir, n); + return statSync(p).isDirectory() ? walk(p) : [p]; + }); + +if (!existsSync(DIST)) { + console.error('dist/ not found. Run `astro build` first.'); + process.exit(2); +} + +const failures = []; +const rows = []; + +for (const file of walk(DOCS).filter((f) => /\.mdx?$/.test(f))) { + const slug = relative(DOCS, file).replace(/\.mdx?$/, '').replace(/(^|\/)index$/, ''); + const src = readFileSync(file, 'utf8'); + const fences = (src.match(/^\s*```mermaid\b/gm) || []).length; + const out = join(DIST, slug, 'index.html'); + if (!existsSync(out)) { + failures.push(`${slug}: no built page at dist/${slug}/index.html`); + continue; + } + const html = readFileSync(out, 'utf8'); + const start = html.indexOf('class="sl-markdown-content"'); + const end = html.indexOf('', start); + const main = start === -1 ? '' : html.slice(start, end === -1 ? undefined : end); + const text = main + .replace(//g, ' ') + .replace(/<(script|style)[\s\S]*?<\/\1>/g, ' ') + .replace(/<[^>]+>/g, ' '); + const words = (text.match(/\S+/g) || []).length; + const svgs = (main.match(/]*\bid="mermaid-/g) || []).length; + + rows.push({ slug, fences, svgs, words }); + if (start === -1) failures.push(`${slug}: main content container not found`); + else if (words < MIN_WORDS) failures.push(`${slug}: main content has only ${words} words (minimum ${MIN_WORDS})`); + if (svgs < fences) failures.push(`${slug}: ${fences} mermaid fence(s) but ${svgs} rendered SVG(s)`); +} + +if (verbose) { + for (const r of rows.filter((r) => r.fences > 0)) { + console.log(` diagram page ${r.slug}: ${r.svgs}/${r.fences} rendered, ${r.words} words`); + } +} + +const diagramPages = rows.filter((r) => r.fences > 0).length; +if (failures.length) { + console.error(`\ncheck-rendered-pages: ${failures.length} problem(s)`); + for (const f of failures) console.error(' ' + f); + console.error( + '\nIf diagrams are missing, the headless browser is probably not installed: npx playwright install chromium-headless-shell. Astro caches rendered markdown, so after fixing it delete node_modules/.astro and rebuild.', + ); + process.exit(1); +} +console.log( + `check-rendered-pages: ${rows.length} pages OK (${diagramPages} with diagrams, min ${Math.min(...rows.map((r) => r.words))} words)`, +); diff --git a/scripts/config-reference-overrides.json b/scripts/config-reference-overrides.json new file mode 100644 index 0000000..92de1b4 --- /dev/null +++ b/scripts/config-reference-overrides.json @@ -0,0 +1,890 @@ +{ + "_readme": [ + "Hand-maintained inputs for scripts/gen-config-reference.mjs. The JSON schema in the product repo is the structural source of truth (keys, types, enums, min/max).", + "The schema omits many defaults and descriptions, so this file supplies them. Every value here was read from the C# options classes of the release named in _verifiedAgainst.", + "Each entry has a 'source' that names the file to re-read at the next release. When a default in C# changes, edit it here and re-run the script.", + "keys: overrides for keys that exist in the schema (default, description, type, allowed). An override for a key that has left the schema fails the run.", + "shared: overrides applied to each prefix in 'prefixes' (the three chat channels). An explicit entry in 'keys' wins.", + "extra: keys the daemon reads that the schema does not list (chat channel secrets, MCP client secret). 'after' pins the row position.", + "sections: one-paragraph orientation per top-level key, rendered above each table. 'notes' renders below it.", + "skip: schema paths that are not rendered as rows." + ], + "_verifiedAgainst": "netclaw 0.28.2 release source (version string 0.28.2)", + "mapKeyNames": { + "Models.Definitions": "name", + "McpServers": "server", + "Providers": "name" + }, + "shared": { + "prefixes": [ + "Slack", + "Discord", + "Mattermost" + ], + "keys": { + "Enabled": { + "default": "`false`", + "description": "Start this channel adapter. Needs the bot token in `secrets.json`.", + "source": "Netclaw.Channels./ChannelOptions.cs" + }, + "DefaultChannelId": { + "default": "unset", + "description": "Default channel for messages the bot starts itself, such as reminder deliveries. It also counts as an allowed channel.", + "source": "SlackChannelOptions.cs, SlackAclPolicy.IsAllowedChannel, ChannelIntegrationRegistrationExtensions.cs" + }, + "MentionOnly": { + "default": "`true`", + "description": "Respond in channels only when the bot is mentioned.", + "source": "SlackChannelOptions.cs, DiscordChannelOptions.cs, MattermostChannelOptions.cs" + }, + "AllowDirectMessages": { + "default": "`false`", + "description": "Accept direct messages. DMs use the audience mapped to the `dm` key in `ChannelAudiences`.", + "source": "SlackChannelOptions.cs, SlackAclPolicy.EvaluateInbound" + }, + "MentionRequiredInDm": { + "default": "`false`", + "description": "Require a mention in DMs too. Applies only when `AllowDirectMessages` is `true`.", + "source": "SlackChannelOptions.cs" + }, + "MentionRequiredInThreadByChannel": { + "default": "`{}`", + "description": "Per-channel switch, keyed by channel ID. `true` requires a mention in thread replies even when the thread already has an active session. A channel with no entry doesn't.", + "source": "SlackChannelOptions.MentionRequiredInThreadFor" + }, + "AllowedChannelIds": { + "default": "`[]`", + "description": "Channel IDs the bot responds in, in addition to `DefaultChannelId`. Messages from any other channel are denied.", + "source": "SlackAclPolicy.IsAllowedChannel" + }, + "AllowedUserIds": { + "default": "`[]`", + "description": "User IDs allowed to talk to the bot. An empty list allows every user in an allowed channel. A listed user is treated as a trusted internal principal.", + "source": "SlackAclPolicy.IsAllowedUser (empty list allows all)" + }, + "ChannelAudiences": { + "default": "`{}`", + "description": "Audience per channel, keyed by channel ID or `dm`. Unmapped channels fall back to the ACL heuristic. The audience picks the `Tools.AudienceProfiles` entry that applies.", + "source": "SlackChannelOptions.cs" + } + } + }, + "keys": { + "Slack.SocketMode": { + "default": "`true`", + "description": "Connect over Slack Socket Mode. Needs the app-level token (`Slack.AppToken`).", + "source": "SlackChannelOptions.cs" + }, + "Slack.DefaultChannelName": { + "default": "unset", + "description": "A channel name the daemon resolves to an ID at startup when `DefaultChannelId` is unset. An unresolved name logs a warning. The `netclaw config` Slack editor deletes it and writes the ID.", + "source": "SlackChannelOptions.cs" + }, + "Logging.LogLevel.Default": { + "default": "`\"Information\"`", + "description": "Minimum level written to `daemon-.log`. An unset or unrecognized value gives `Information`. Other keys under `Logging.LogLevel` (for example `Microsoft`) set per-category levels.", + "source": "Netclaw.Daemon/Configuration/LoggingRegistrationExtensions.ResolveLogLevel" + }, + "Logging.Console.Enabled": { + "default": "`false`", + "description": "Also write single-line log output to the console. Docker and systemd capture it.", + "source": "LoggingRegistrationExtensions.ConfigureNetclawLogging" + }, + "Telemetry.Enabled": { + "default": "`false`", + "description": "Export metrics and logs over OTLP. netclaw has no tracing exporter.", + "source": "Netclaw.Daemon/Configuration/TelemetryOptions.cs" + }, + "Telemetry.Otlp.Endpoint": { + "default": "`\"http://127.0.0.1:4317\"`", + "description": "OTLP collector base URL. Must be an absolute URI. Over HTTP/protobuf netclaw posts to `v1/logs` and `v1/metrics` under it, and reduces a full signal URL to its base. gRPC uses the URL unchanged.", + "source": "TelemetryOptions.cs, TelemetryOptionsValidator.cs" + }, + "Session.MemoryObserverIdleSeconds": { + "default": "`90`", + "description": "Seconds of quiet before the memory observer distills the conversation into memories.", + "source": "SessionConfig.cs" + }, + "Session.IdleTimeout": { + "description": "How long a session sits idle before it saves a snapshot and passivates. Format `HH:mm:ss`. `00:00:00` disables passivation. Background jobs owned by the session are killed when it passivates.", + "source": "SessionConfig.cs", + "default": "`\"01:00:00\"`" + }, + "Session.TurnLlmTimeoutSeconds": { + "default": "`180`", + "description": "Timeout for the main per-turn model call. Also feeds the `FirstTokenTimeoutSeconds` fallback.", + "source": "SessionConfig.RawSessionConfig" + }, + "Session.ToolExecutionTimeoutSeconds": { + "default": "`90`", + "description": "Inactivity budget per tool call. The call must produce its first result within this time and each later streamed item resets it. A stuck tool fails alone without affecting the others in the batch. Background shell jobs aren't bound by it.", + "source": "SessionConfig.RawSessionConfig, docs/runbooks/background-jobs.md" + }, + "Session.SidecarLlmTimeoutSeconds": { + "default": "`90`", + "description": "Timeout for background model calls: title generation, memory distillation and compaction notes. Raise it for slow models.", + "source": "SessionConfig.cs" + }, + "Memory.Curation.NominatorSimilarityThreshold": { + "default": "`0.86`", + "description": "Embedding cosine similarity threshold above which an existing memory is nominated for the curation call to adjudicate.", + "source": "Netclaw.Configuration/Schemas/netclaw-config.v1.schema.json" + }, + "Session.FirstTokenTimeoutSeconds": { + "default": "`600`", + "description": "Maximum silence between streamed deltas once the first one arrives. Subagent runs use it too, since agent definitions no longer carry `timeoutSeconds`. If unset, it uses `TurnLlmTimeoutSeconds` when you changed that, otherwise 600.", + "source": "SessionConfig.BindFromConfiguration" + }, + "Session.PrefillTimeoutSeconds": { + "default": "`1800`", + "description": "Maximum wait for the first streamed delta. Covers queue wait and prompt prefill, which can take minutes on self-hosted backends.", + "source": "SessionConfig.BindFromConfiguration" + }, + "Session.NoProgressTimeoutSeconds": { + "default": "`1200`", + "description": "Hard ceiling on time without a real output token. Keepalive heartbeats don't reset it, so a backend that heartbeats forever without producing a token is killed when it elapses. The value also sets the thinking cap: one model call that streams only thinking, with no answer text and no tool call, stops at three times this value.", + "source": "SessionConfig.BindFromConfiguration" + }, + "Session.MaxToolIterationsPerTurn": { + "default": "`60`", + "description": "Legacy key kept so existing files still load. It is validated (1 to 500) but no longer limits a turn. netclaw refuses a call that repeats the same work, and the watchdog timeouts bound the rest.", + "source": "SessionConfig.cs" + }, + "Session.CompactionThreshold": { + "default": "`0.75`", + "description": "Fraction of the context window that triggers compaction. Same as `Session.Tuning.CompactionThreshold`. The `Tuning` value wins when both are set.", + "source": "SessionTuning.cs, SessionConfig.BindTuning" + }, + "Session.SnapshotInterval": { + "default": "`20`", + "description": "Turns between persistence snapshots. Same as `Session.Tuning.SnapshotInterval`.", + "source": "SessionTuning.cs" + }, + "Session.KeepRecentToolResults": { + "default": "`3`", + "description": "Recent tool call and result pairs kept in full when compaction clears older results. Same as `Session.Tuning.KeepRecentToolResults`.", + "source": "SessionTuning.cs" + }, + "Session.MaxInlineToolResultChars": { + "default": "`12000`", + "description": "Largest tool result placed inline in the conversation. For longer output the last this-many characters stay inline, the rest is dropped from the inline text, and the full output goes to a session file the model reads with `tool_output_read`. `shell_execute` has its own smaller budget of 2,000. Same as `Session.Tuning.MaxInlineToolResultChars`.", + "source": "SessionTuning.cs" + }, + "Session.DiscoveredToolRetentionTurns": { + "default": "`3`", + "description": "User turns a tool loaded with `load_tool` stays available. `0` makes the model load it again every turn. Same as `Session.Tuning.DiscoveredToolRetentionTurns`.", + "source": "SessionTuning.cs" + }, + "Session.DiscoveredToolMaxCount": { + "default": "`12`", + "description": "Most loaded deferred or MCP tools kept across turns. The oldest are evicted first. Same as `Session.Tuning.DiscoveredToolMaxCount`.", + "source": "SessionTuning.cs" + }, + "Session.KeepRecentMessages": { + "default": "`6`", + "description": "Recent messages kept verbatim after compaction summarizes the rest. Counts raw messages, not turn pairs. Same as `Session.Tuning.KeepRecentMessages`.", + "source": "SessionTuning.cs" + }, + "Session.TitleGenerationInterval": { + "default": "`10`", + "description": "Turns between session title refreshes. The title is always generated on turn 1. `0` disables refreshes. Same as `Session.Tuning.TitleGenerationInterval`.", + "source": "SessionTuning.cs" + }, + "Session.DeterministicRetrievalEnabled": { + "default": "`true`", + "description": "Plan automatic memory recall deterministically on each turn. Scheduled for removal, leave it on. Same as `Session.Tuning.DeterministicRetrievalEnabled`.", + "source": "SessionTuning.cs" + }, + "Session.Tuning": { + "default": "-", + "description": "Internal tuning. Every key here has a same-named key directly under `Session`, and the `Tuning` value wins.", + "source": "SessionConfig.BindTuning" + }, + "Session.Tuning.CompactionThreshold": { + "default": "`0.75`", + "description": "See `Session.CompactionThreshold`.", + "source": "SessionTuning.cs" + }, + "Session.Tuning.SnapshotInterval": { + "default": "`20`", + "description": "See `Session.SnapshotInterval`.", + "source": "SessionTuning.cs" + }, + "Session.Tuning.KeepRecentToolResults": { + "default": "`3`", + "description": "See `Session.KeepRecentToolResults`.", + "source": "SessionTuning.cs" + }, + "Session.Tuning.MaxInlineToolResultChars": { + "default": "`12000`", + "description": "See `Session.MaxInlineToolResultChars`.", + "source": "SessionTuning.cs" + }, + "Session.Tuning.DiscoveredToolRetentionTurns": { + "default": "`3`", + "description": "See `Session.DiscoveredToolRetentionTurns`.", + "source": "SessionTuning.cs" + }, + "Session.Tuning.DiscoveredToolMaxCount": { + "default": "`12`", + "description": "See `Session.DiscoveredToolMaxCount`.", + "source": "SessionTuning.cs" + }, + "Session.Tuning.KeepRecentMessages": { + "default": "`6`", + "description": "See `Session.KeepRecentMessages`.", + "source": "SessionTuning.cs" + }, + "Session.Tuning.TitleGenerationInterval": { + "default": "`10`", + "description": "See `Session.TitleGenerationInterval`.", + "source": "SessionTuning.cs" + }, + "Session.Tuning.DeterministicRetrievalEnabled": { + "default": "`true`", + "description": "See `Session.DeterministicRetrievalEnabled`.", + "source": "SessionTuning.cs" + }, + "Session.Tuning.MemoryDistillationTurnInterval": { + "default": "`5`", + "description": "Completed turns between memory distillation passes, regardless of idle state, so memories form during long active sessions. `0` leaves distillation to `MemoryObserverIdleSeconds`.", + "source": "SessionTuning.cs" + }, + "Session.Tuning.MinimumRecallCompositeScore": { + "default": "unset", + "description": "Minimum composite score for a memory to be injected by automatic recall. Unset uses the coordinator's built-in floor. Lower values let weaker matches through, `0` removes the floor.", + "source": "SessionTuning.cs" + }, + "Session.Tuning.MaxRecallInjectedChars": { + "default": "`2000`", + "description": "Character budget for memory text injected by automatic recall in one turn. Whole items are dropped rather than truncated. `0` removes the budget.", + "source": "SessionTuning.cs" + }, + "Security.DeploymentPosture": { + "default": "unset", + "description": "Who talks to this install: `Personal` (you), `Team` or `Public`. `netclaw init` writes it. When it's unset and `StrictDefaults` is `true`, netclaw behaves as `Public` and denies `shell_execute`. It selects the default tool profiles and shell mode, and it narrows the audience of every turn: a channel mapped to `Personal` under a `Team` posture runs as `Team`.", + "source": "TrustContextPolicy.cs SecurityPolicyDefaults.ResolveDeploymentPosture" + }, + "Security.ShellExecutionMode": { + "default": "unset", + "description": "Whether `shell_execute` can run. Unset gives `HostAllowed` for a `Personal` posture and `Off` for the others. `Tools.ShellMode` overrides it. `SandboxOnly` denies every shell call with `shell_requires_sandbox_backend`.", + "source": "SecurityPolicyDefaults.ResolveShellExecutionMode, ToolAccessPolicy.ResolveShellMode" + }, + "Security.StrictDefaults": { + "default": "`true`", + "description": "Fail closed when posture or shell mode is unset. Set `false` only to get `Personal` and `HostAllowed` as the implicit defaults.", + "source": "SecurityPolicyConfig" + }, + "Tools.ShellMode": { + "default": "unset", + "description": "Overrides `Security.ShellExecutionMode` for the shell tools. Unset follows the security setting.", + "source": "ToolConfig.ShellMode, ToolAccessPolicy.ResolveShellMode" + }, + "Tools.MaxOutputChars": { + "default": "`256000`", + "description": "Most output a tool call captures in memory for the spill file. It isn't what the model sees inline, which is `Session.MaxInlineToolResultChars`. Output past the ceiling is discarded.", + "source": "ToolConfig.cs" + }, + "Tools.AudienceProfiles": { + "default": "-", + "description": "Per-audience tool, file and MCP grants. Public, Team and Personal share one shape. A list you set replaces the default list and doesn't merge with it.", + "source": "ToolAudienceProfiles.cs" + }, + "Tools.AudienceProfiles.": { + "default": "-", + "description": "One entry each for `Public`, `Team` and `Personal`.", + "source": "ToolAudienceProfiles.cs" + }, + "Tools.AudienceProfiles..ToolsMode": { + "default": "`Allowlist` (Personal: `All`)", + "description": "`Allowlist` grants only the tools in `AllowedTools`. `All` grants every profile-managed tool.", + "source": "ToolAudienceProfileDefaults.CreatePublic/CreateTeam/CreatePersonal" + }, + "Tools.AudienceProfiles..AllowedTools": { + "default": "Public and Team defaults, see [Built-in tools](/reference/tools/#default-allow-lists)", + "description": "Profile-managed tools this audience can call. Tools outside the profile-managed set are always available. Setting the key replaces the default list, and a value that isn't a list stops startup.", + "source": "ToolAudienceProfileToolCatalog, ToolConfig.BindFromConfiguration" + }, + "Tools.AudienceProfiles..McpServersMode": { + "default": "`Allowlist` (Personal: `All`)", + "description": "`Allowlist` exposes only the servers in `AllowedMcpServers`. `All` exposes every configured server.", + "source": "ToolAudienceProfileDefaults" + }, + "Tools.AudienceProfiles..AllowedMcpServers": { + "default": "`[]`", + "description": "MCP server names this audience can use when `McpServersMode` is `Allowlist`. MCP stays off for Public and Team until you add a server.", + "source": "ToolAudienceProfile" + }, + "Tools.AudienceProfiles..McpServerToolGrants": { + "default": "unset", + "description": "Per-server tool allowlists. A server listed here exposes only the named tools. A server not listed exposes all of its tools. Ignored when `McpServersMode` is `All`.", + "source": "ToolAudienceProfileResolver.IsMcpToolAllowed" + }, + "Tools.AudienceProfiles..ReadFiles.Mode": { + "default": "`Roots` (Personal: `All`)", + "description": "`None` blocks it, `Roots` limits it to `Roots`, `All` allows any path that isn't protected. For the `Personal` audience, `file_read`, `file_list`, `file_search` and `attach_file` reach at least as far as `WriteFiles` does.", + "source": "ToolAudienceProfileDefaults.CreateSessionScopedFilesystemAccess" + }, + "Tools.AudienceProfiles..ReadFiles.Roots": { + "default": "`[\"{session_dir}\"]`", + "description": "Directories used when `Mode` is `Roots`. Tokens: `{session_dir}`, `{skills_dir}`, `{identity_dir}`, `{workspaces_dir}`.", + "source": "ToolAudienceProfileDefaults, ToolAudienceProfileResolver.ResolveToken" + }, + "Tools.AudienceProfiles..WriteFiles.Mode": { + "default": "`Roots` (Personal: `All`)", + "description": "Same values as `ReadFiles.Mode`, for `file_write` and `file_edit`.", + "source": "ToolAudienceProfileDefaults" + }, + "Tools.AudienceProfiles..WriteFiles.Roots": { + "default": "`[\"{session_dir}\"]`", + "description": "Directories used when `Mode` is `Roots`. Same tokens as `ReadFiles.Roots`.", + "source": "ToolAudienceProfileDefaults" + }, + "Tools.AudienceProfiles..AttachFiles.Mode": { + "default": "`Roots` (Personal: `All`)", + "description": "Same values as `ReadFiles.Mode`, for `attach_file`.", + "source": "ToolAudienceProfileDefaults" + }, + "Tools.AudienceProfiles..AttachFiles.Roots": { + "default": "`[\"{session_dir}\"]`", + "description": "Directories used when `Mode` is `Roots`. Same tokens as `ReadFiles.Roots`.", + "source": "ToolAudienceProfileDefaults" + }, + "Tools.AudienceProfiles..ApprovalPolicy": { + "default": "unset (Personal posture: `shell_execute` is `Approval`)", + "description": "Which tools need a human to approve each call. See [How tool approvals work](/security/approvals/).", + "source": "ToolAudienceProfileDefaults.CreateProfilesForPosture" + }, + "Tools.AudienceProfiles..ApprovalPolicy.DefaultMode": { + "default": "`\"Auto\"`", + "description": "Mode for tools with no override.", + "source": "ToolApprovalConfig.cs" + }, + "Tools.AudienceProfiles..ApprovalPolicy.ToolOverrides": { + "default": "`{}`", + "description": "Mode per tool. Keys are tool names (`shell_execute`) or MCP tools as `server/tool` (or `server__tool`), matched without regard to case. If two keys differ only by case, the more restrictive mode wins. A key that contains `:` or has no value is ignored, so the tool keeps the audience default, and `netclaw doctor` reports it. Writes to the config directory are denied before any approval mode is read.", + "source": "ToolApprovalConfig.cs" + }, + "Tools.AudienceProfiles..ApprovalPolicy.McpServerDefaults": { + "default": "`{}`", + "description": "Mode for every tool of one MCP server, keyed by server name and matched without regard to case. An exact `ToolOverrides` entry wins. A key that contains `:` is ignored and `netclaw doctor` reports it.", + "source": "ToolApprovalConfig.TryGetExplicitMode" + }, + "Tools.AudienceProfiles..ChannelAttachments": { + "default": "-", + "description": "Which files from chat messages this audience can send in.", + "source": "ChannelAttachmentPolicy.cs" + }, + "Tools.AudienceProfiles..ChannelAttachments.AllowedCategories": { + "default": "Public: `Image`. Team: `Image`, `Pdf`, `Document`, `Archive`, `Media`. Personal: those plus `Other`.", + "description": "Attachment categories accepted. An empty list rejects every attachment.", + "source": "ToolAudienceProfileDefaults.CreatePublic/Team/PersonalChannelAttachments" + }, + "Tools.AudienceProfiles..ChannelAttachments.MaxFileBytes": { + "default": "`26214400`", + "description": "Largest file accepted, in bytes (25 MiB). Must be above 0 when any category is allowed.", + "source": "ChannelAttachmentPolicy.DefaultMaxFileBytes, ToolAudienceProfiles.ValidateChannelAttachments" + }, + "Tools.AudienceProfiles..ChannelAttachments.MaxFilesPerMessage": { + "default": "`10`", + "description": "Most files accepted on one message. A message over the limit loses all its attachments. Must be above 0 when any category is allowed.", + "source": "ChannelAttachmentPolicy.DefaultMaxFilesPerMessage" + }, + "Tools.AudienceProfiles.GlobalReadRoots": { + "default": "`[\"{skills_dir}\", \"{identity_dir}\", \"{workspaces_dir}\"]`", + "description": "Directories every audience can read, so skill loading, identity files and project discovery work under Team and Public. Setting the key replaces the list.", + "source": "ToolAudienceProfiles.GlobalReadRoots" + }, + "Tools.WebFetch.RequireHttps": { + "default": "`true`", + "description": "Reject plain `http://` URLs in `web_fetch` unless the host is in `HttpAllowList`.", + "source": "WebFetchConfig.cs" + }, + "Tools.WebFetch.HttpAllowList": { + "default": "`[\"localhost\", \"127.0.0.1\", \"::1\", \"[::1]\"]`", + "description": "Hosts that may use plain HTTP when `RequireHttps` is `true`. Setting the key replaces the list, so keep the loopback entries if you still need them.", + "source": "WebFetchConfig.cs, ToolConfig.DefaultedLists" + }, + "Tools.HardDenyPatterns": { + "default": "`[]`", + "description": "Extra shell verb-chain prefixes that are always blocked and can't be approved. They add to the built-in hard-deny list, they don't replace it.", + "source": "ToolConfig.HardDenyPatterns" + }, + "Providers": { + "type": "map of object", + "default": "`{}`", + "description": "LLM provider connections keyed by a name you choose. `Models.Definitions` entries refer to these names.", + "source": "ProviderEntry.cs" + }, + "Models.Definitions": { + "default": "-", + "description": "Model definitions keyed by a name you choose, at least one. Names are case-insensitive and must be unique.", + "source": "NamedModelConfiguration.cs" + }, + "Models.Definitions.": { + "default": "-", + "description": "One model: which provider serves it and how it behaves.", + "source": "ModelReference.cs" + }, + "Models.Definitions..Provider": { + "default": "required", + "description": "Key into `Providers`.", + "source": "ModelReference.cs" + }, + "Models.Definitions..ModelId": { + "default": "required", + "description": "Model identifier sent to the provider.", + "source": "ModelReference.cs" + }, + "Models.Definitions..ContextWindow": { + "default": "detected", + "description": "Context window in tokens. When set, it caps the value the provider reports.", + "source": "ModelReference.cs" + }, + "Models.Definitions..Provenance": { + "default": "unset", + "description": "Where the model data came from: `Live`, `Defaults` or `Manual`. netclaw sets it. You don't need to.", + "source": "ModelReference.cs" + }, + "Models.Definitions..InputModalities": { + "default": "detected", + "description": "Manual override for what the model accepts, comma-separated from `Text`, `Image`, `Audio`, `Video`. Turns off automatic detection.", + "source": "ModelReference.cs", + "allowed": "comma-separated `Text`, `Image`, `Audio`, `Video`" + }, + "Models.Definitions..OutputModalities": { + "default": "detected", + "description": "Manual override for what the model produces. Same format as `InputModalities`.", + "source": "ModelReference.cs", + "allowed": "comma-separated `Text`, `Image`, `Audio`, `Video`" + }, + "Models.Definitions..MaxOutputTokens": { + "default": "unset", + "description": "Output limit in tokens for every call of this model. Thinking tokens count inside it. Must be below `ContextWindow` when that is set. Unset sends 32000, clamped per call to the model maximum and to the context window minus the estimated prompt.", + "source": "ModelReference.cs" + }, + "Models.Definitions..Thinking": { + "default": "unset", + "description": "Provider-neutral thinking level for every call of this model. `Max` is the highest level the provider offers. A level the provider dialect can't send fails startup. Unset uses the model's default.", + "source": "ModelReference.cs" + }, + "Models.Roles": { + "default": "-", + "description": "Which definition plays each role. Values are names from `Definitions`.", + "source": "NamedModelConfiguration.cs" + }, + "Models.Roles.Main": { + "default": "required", + "description": "Model for all interactions.", + "source": "ModelSelection.cs" + }, + "Models.Roles.Fallback": { + "default": "unset", + "description": "Failover model. Falls back to `Main` when unset.", + "source": "ModelSelection.cs" + }, + "Models.Roles.Compaction": { + "default": "unset", + "description": "Cheaper or faster model for compaction. Falls back to `Main` when unset.", + "source": "ModelSelection.cs" + }, + "Memory.Recall.MinCosineSimilarity": { + "default": "unset (follows the model, 0.24 for the default)", + "description": "Cosine floor for automatic recall. Candidates below it are dropped before ranking. The meaning is specific to the active embedding model, so recalibrate before you set it.", + "source": "MemoryConfig.cs MemoryRecallConfig, doctor 'Memory Embeddings' check" + }, + "Memory.Recall.RelevanceGate.Enabled": { + "default": "unset (follows `Memory.Embeddings.Enabled`)", + "description": "Run the cross-encoder relevance gate on recall candidates. Set it explicitly to turn the gate on or off independently of embeddings.", + "source": "MemoryConfig.cs MemoryRelevanceGateConfig" + }, + "Memory.Recall.RelevanceGate.Threshold": { + "default": "unset (model default, 0.02)", + "description": "Score below which the gate drops a candidate. Unset uses the calibrated value for the provisioned relevance model.", + "source": "MemoryConfig.cs, doctor 'Memory Relevance Gate' check" + }, + "ExternalSkills.Sources": { + "default": "`[]`", + "description": "Ordered list of external skill sources. Earlier entries win name collisions and native netclaw skills always win.", + "source": "ExternalSkillsConfig.cs" + }, + "ExternalSkills.Sources[].Name": { + "default": "required", + "description": "Unique name for the source, for example `claude-code`.", + "source": "ExternalSkillsConfig.cs" + }, + "SkillFeeds.Feeds": { + "default": "`[]`", + "description": "Ordered list of skill server feeds. Earlier feeds win name collisions.", + "source": "SkillFeedsConfig.cs" + }, + "SkillFeeds.Feeds[].Name": { + "default": "required", + "description": "Unique feed name, lowercase letters, digits and hyphens. Used as the directory name.", + "source": "SkillFeedsConfig.cs" + }, + "SkillFeeds.Feeds[].Url": { + "default": "required", + "description": "Skill server base URL.", + "source": "SkillFeedsConfig.cs" + }, + "Notifications.Webhooks": { + "default": "`[]`", + "description": "Targets that receive operational alerts by HTTP POST. Empty means alerts only go to the log.", + "source": "NotificationsConfig.cs" + }, + "Notifications.Webhooks[].Url": { + "default": "required", + "description": "HTTP or HTTPS endpoint to POST alerts to.", + "source": "NotificationsConfig.cs WebhookTarget" + }, + "Notifications.Webhooks[].Name": { + "default": "unset", + "description": "Display name used in logs.", + "source": "NotificationsConfig.cs WebhookTarget" + }, + "Notifications.Webhooks[].Headers": { + "default": "unset", + "description": "Static headers sent with every POST, for example `Authorization`.", + "source": "NotificationsConfig.cs WebhookTarget" + }, + "Webhooks.Enabled": { + "default": "`false`", + "description": "Register the inbound webhook endpoint and the `set_webhook`, `list_webhooks` and `delete_webhook` tools.", + "source": "WebhooksConfig.cs, Netclaw.Daemon/Program.cs" + }, + "Webhooks.ExecutionTimeoutSeconds": { + "default": "`300`", + "description": "Longest a webhook-triggered run can take before the daemon marks it failed.", + "source": "WebhooksConfig.cs, WebhookExecutionActor.cs" + }, + "Workspaces.Directory": { + "default": "`~/.netclaw/workspaces`", + "description": "Root for project workspaces. Projects are Git repositories with an `AGENTS.md`, at any depth. A leading `~` expands to your home directory.", + "source": "NetclawPaths.cs" + }, + "McpServers..Transport": { + "default": "`\"stdio\"`", + "description": "How netclaw connects: `stdio` runs `Command`, `sse` and `http` connect to `Url`.", + "source": "McpServerEntry.cs" + }, + "McpServers..Command": { + "default": "unset", + "description": "Executable to run for `stdio`.", + "source": "McpServerEntry.cs" + }, + "McpServers..Arguments": { + "default": "unset", + "description": "Arguments for `Command`.", + "source": "McpServerEntry.cs" + }, + "McpServers..Url": { + "default": "unset", + "description": "Endpoint for `sse` and `http`.", + "source": "McpServerEntry.cs" + }, + "McpServers..EnvironmentVariables": { + "default": "unset", + "description": "Environment variables passed to the `stdio` process. Values stored in `secrets.json` are decrypted first.", + "source": "McpServerEntry.cs" + }, + "McpServers..Headers": { + "default": "unset", + "description": "HTTP headers for `sse` and `http`. Values stored in `secrets.json` are decrypted first.", + "source": "McpServerEntry.cs" + }, + "McpServers..Enabled": { + "default": "`true`", + "description": "Disabled servers are skipped at startup.", + "source": "McpServerEntry.cs" + }, + "McpServers..GrantCategory": { + "default": "`mcp:`", + "description": "Tool metadata category for the server's tools. Authorization doesn't read it: use the audience's `AllowedMcpServers` and `McpServerToolGrants`.", + "source": "McpServerEntry.cs" + }, + "McpServers..OAuthClientId": { + "default": "unset", + "description": "Static OAuth client ID for servers without dynamic client registration.", + "source": "McpServerEntry.cs" + }, + "McpServers..OAuthScope": { + "default": "unset", + "description": "Space-separated OAuth scopes to request, overriding the server's defaults.", + "source": "McpServerEntry.cs" + }, + "Search.BraveApiKey": { + "default": "unset", + "description": "Brave Search API token. Required when `Backend` is `brave`. Lives in `secrets.json`: `netclaw secrets set Search.BraveApiKey `.", + "source": "SearchConfig.cs ([ConfigValue] PersistTo SecretsJson)" + }, + "Search.SearXngEndpoint": { + "default": "unset", + "description": "SearXNG instance base URL, for example `http://searxng.local:8080`. Required when `Backend` is `searxng`.", + "source": "SearchConfig.cs" + }, + "Memory.Embeddings": { + "default": "-", + "description": "Local ONNX embedding model for semantic memory.", + "source": "MemoryConfig.cs MemoryEmbeddingsConfig" + }, + "Memory.Curation": { + "default": "-", + "description": "How new memories are checked against existing ones before they're saved.", + "source": "MemoryConfig.cs MemoryCurationConfig" + }, + "Memory.Recall": { + "default": "-", + "description": "How automatic recall ranks and filters memories, mixing keyword and vector scores.", + "source": "MemoryConfig.cs MemoryRecallConfig" + }, + "Memory.Embeddings.ModelId": { + "default": "`\"snowflake-arctic-embed-m-int8\"`", + "description": "Embedding model to use. Allowed ids are `snowflake-arctic-embed-m-int8`, `snowflake-arctic-embed-m` and `mxbai-embed-large-v1`. An id outside the allowlist is a configuration error. Changing it re-embeds existing memories under the new id.", + "source": "MemoryConfig.cs MemoryEmbeddingsConfig.ModelId" + }, + "SubAgents.PrefillTimeoutSeconds": { + "default": "`1800`", + "description": "Longest wait for a subagent's first streamed token, which covers queue wait and prompt prefill. Same idea as `Session.PrefillTimeoutSeconds`. After the first token the `Session.FirstTokenTimeoutSeconds` budget applies. An agent definition can override it with `prefillTimeoutSeconds` in its frontmatter.", + "source": "SubAgentConfig.cs" + }, + "SubAgents.NoProgressTimeoutSeconds": { + "default": "`1200`", + "description": "Hard ceiling on a subagent run with no real output token. Keepalives don't reset it. Like `Session.NoProgressTimeoutSeconds`, it also sets the thinking cap at three times the value.", + "source": "SubAgentConfig.cs" + }, + "Daemon.UpdateChannel": { + "default": "`\"stable\"`", + "description": "Release channel the update check follows. `stable` only sees stable releases. `beta` also sees prereleases and moves to a stable release once one supersedes the beta.", + "source": "DaemonConfig.cs" + }, + "Notifications.Webhooks[].Format": { + "default": "`\"Generic\"`", + "description": "Payload format: `Generic` is a JSON envelope, `Slack` is Slack Block Kit.", + "source": "NotificationsConfig.cs WebhookFormatDetection" + }, + "Mattermost.ServerUrl": { + "default": "unset", + "description": "Base URL of the Mattermost server, for example `https://mm.example.com`.", + "source": "MattermostChannelOptions.cs" + }, + "Mattermost.CallbackUrl": { + "default": "unset", + "description": "URL Mattermost can reach for interactive button callbacks, for example `http://netclaw-host:5199/api/mattermost/actions`. When unset, approvals fall back to text replies, except for a subagent's approval request, which needs buttons: the binding refuses it and the session withdraws it.", + "source": "MattermostChannelOptions.cs" + }, + "Providers..Type": { + "type": "string", + "default": "`\"ollama\"`", + "description": "Provider type: `anthropic`, `deepseek`, `github-copilot`, `ollama`, `openai`, `openai-compatible`, `openrouter` or `veniceai`.", + "source": "ProviderEntry.cs, IProviderDescriptor.TypeKey implementations" + }, + "Providers..Endpoint": { + "type": "string", + "default": "`\"\"` (the provider type supplies its own default URL)", + "description": "Base URL of the provider API.", + "source": "ProviderEntry.cs" + }, + "Providers..AuthMethod": { + "type": "string", + "default": "`\"None\"`", + "allowed": "`None`, `ApiKey`, `OAuthDevice`, `OAuthPkce`", + "description": "How netclaw authenticates to the provider.", + "source": "ProviderEntry.cs, AuthMethod.cs" + }, + "Providers..ApiKey": { + "type": "string (secret)", + "default": "unset", + "description": "API key. Lives in `secrets.json`: `netclaw secrets set Providers..ApiKey `.", + "source": "ProviderEntry.cs" + }, + "Providers..VendorOptions": { + "type": "object", + "default": "unset", + "description": "Provider-specific non-secret options, for example `GitHubHost` and `GitHubApiBase` for `github-copilot`, `ThinkingDialect` for `openai-compatible`, `DisableThinking` for `ollama`, `ExcludeReasoning` for `openrouter` and `IncludeVeniceSystemPrompt` for `veniceai`. Each provider type defines its own keys.", + "source": "ProviderEntry.VendorOptions" + }, + "Providers.": { + "description": "One provider connection. The name is yours to choose and `Models.Definitions` entries refer to it.", + "source": "ProviderEntry.cs" + }, + "Providers..OAuthAccessToken": { + "type": "string (secret)", + "description": "OAuth access token. netclaw writes it to `secrets.json` when you sign in with an OAuth auth method.", + "source": "ProviderEntry.cs" + }, + "Providers..OAuthRefreshToken": { + "type": "string (secret)", + "description": "OAuth refresh token. netclaw writes it to `secrets.json` when you sign in with an OAuth auth method.", + "source": "ProviderEntry.cs" + }, + "Providers..OAuthAccountId": { + "type": "string (secret)", + "description": "OAuth account identifier. netclaw writes it to `secrets.json` when you sign in with an OAuth auth method.", + "source": "ProviderEntry.cs" + }, + "Providers..OAuthTokenExpiry": { + "type": "string (secrets.json, unencrypted)", + "description": "Expiry time of the OAuth access token, ISO 8601. netclaw writes it to `secrets.json` beside the tokens when you sign in with an OAuth auth method. It is the one value there that isn't encrypted. A key left in `netclaw.json` is retired: `netclaw doctor` reports it and `netclaw doctor --fix` moves it, after a backup.", + "source": "ProviderEntry.cs" + }, + "Session.MemoryDistillationTurnInterval": { + "description": "Same as `Session.Tuning.MemoryDistillationTurnInterval`.", + "source": "SessionTuning.cs, SessionConfig.BindTuning" + }, + "Session.MinimumRecallCompositeScore": { + "description": "Same as `Session.Tuning.MinimumRecallCompositeScore`.", + "source": "SessionTuning.cs, SessionConfig.BindTuning" + }, + "Session.MaxToolDescriptionChars": { + "description": "Same as `Session.Tuning.MaxToolDescriptionChars`.", + "source": "SessionTuning.cs, SessionConfig.BindTuning" + }, + "Session.MaxToolSchemaWarnChars": { + "description": "Same as `Session.Tuning.MaxToolSchemaWarnChars`.", + "source": "SessionTuning.cs, SessionConfig.BindTuning" + }, + "Session.Tuning.MaxToolDescriptionChars": { + "description": "Longest MCP tool description kept at registration. Longer ones are truncated to protect the context window. `0` turns truncation off.", + "source": "schema" + }, + "Session.Tuning.MaxToolSchemaWarnChars": { + "description": "Size of an MCP tool's JSON schema that triggers a warning in the log. The schema itself is never truncated, so this only reports bloated tool definitions. `0` turns the warning off.", + "source": "schema" + }, + "Retention.Logs.Days": { + "default": "`14`", + "description": "Days to keep `daemon-yyyy-MM-dd.log`, its numbered segments `daemon-yyyy-MM-dd.N.log` (the daemon starts a new segment at 10 MiB) and `crash-*.log` files in the logs directory. The daemon deletes older files 30 seconds after it starts and every 12 hours after that. The newest 3 daemon logs, segments counted as files, and the newest 3 crash logs are always kept. `0` keeps them forever. Per-session logs are never touched. See [Data retention](/configuration/data-retention/).", + "source": "RetentionSettings.cs, RetentionPolicy.cs, DaemonLogRetention" + }, + "Session.Tuning.StreamingRetryPolicy.BaseDelay": { + "allowed": "`HH:mm:ss`", + "source": "schema" + }, + "Session.Tuning.StreamingRetryPolicy.MaxDelay": { + "allowed": "`HH:mm:ss`", + "source": "schema" + } + }, + "extra": { + "McpServers..OAuthClientSecret": { + "type": "string (secret)", + "default": "unset", + "description": "Client secret for a pre-registered confidential OAuth client. Vault only: the schema rejects it in `netclaw.json`, so doctor fails the Config Schema check. Set it with `netclaw mcp add --client-id --client-secret ` or `netclaw secrets set`, which store it in `secrets.json`. Needs `OAuthClientId`.", + "after": "McpServers..OAuthClientId", + "source": "McpServerEntry.cs, McpCommand.cs" + }, + "Slack.BotToken": { + "type": "string (secret)", + "default": "unset", + "description": "Bot token, `xoxb-`. Lives in `secrets.json`: `netclaw secrets set Slack.BotToken `.", + "after": "Slack.Enabled", + "source": "SlackChannelOptions.cs, ChannelsEditorModel.cs" + }, + "Slack.AppToken": { + "type": "string (secret)", + "default": "unset", + "description": "App-level token, `xapp-`, for Socket Mode. Lives in `secrets.json`.", + "after": "Slack.BotToken", + "source": "SlackChannelOptions.cs" + }, + "Discord.BotToken": { + "type": "string (secret)", + "default": "unset", + "description": "Bot token. Lives in `secrets.json`: `netclaw secrets set Discord.BotToken `.", + "after": "Discord.Enabled", + "source": "DiscordChannelOptions.cs" + }, + "Mattermost.BotToken": { + "type": "string (secret)", + "default": "unset", + "description": "Bot access token. Lives in `secrets.json`: `netclaw secrets set Mattermost.BotToken `.", + "after": "Mattermost.Enabled", + "source": "MattermostChannelOptions.cs" + }, + "Providers..VendorOptions.ThinkingDialect": { + "type": "string", + "default": "unset", + "allowed": "`ChatTemplateKwargs`, `ReasoningEffort`, `DeepSeek`, `None`", + "description": "`openai-compatible` only. How netclaw sends a thinking level to the server. Without it only the output limit works. `netclaw model set` saves `ReasoningEffort` when it detects llama.cpp. Set it by hand for vLLM, SGLang or a proxy.", + "after": "Providers..VendorOptions", + "source": "OpenAiCompatibleDescriptor.cs, ThinkingDialectMap.cs" + } + }, + "sections": { + "Slack": { + "summary": "The Slack adapter. [Set up Slack](/channels/slack/) walks through the app manifest and tokens." + }, + "Discord": { + "summary": "The Discord adapter. [Set up Discord](/channels/discord/) covers the bot, intents and tokens." + }, + "Mattermost": { + "summary": "The Mattermost adapter. [Set up Mattermost](/channels/mattermost/) covers the bot account and callback URL." + }, + "Logging": { + "summary": "Daemon log level and console output. Logs go to `daemon-.log` under the data directory regardless of these keys, rolling into numbered segments at 10 MiB." + }, + "Telemetry": { + "summary": "OpenTelemetry export. [OpenTelemetry](/observability/opentelemetry/) shows how to wire a collector." + }, + "Session": { + "summary": "Per-conversation timeouts, compaction and memory-capture timing. The timeout keys take whole seconds, `IdleTimeout` takes `HH:mm:ss`. [Sessions](/architecture/sessions/) explains the lifecycle these keys tune.", + "notes": "A value below a key's minimum fails `netclaw doctor` schema validation. The daemon itself enforces a 1 second minimum on `TurnLlmTimeoutSeconds`, `ToolExecutionTimeoutSeconds` and `SidecarLlmTimeoutSeconds`." + }, + "Security": { + "summary": "Deployment posture and shell mode, which choose the default tool profiles. [Security model](/security/security-model/) explains the three postures." + }, + "Tools": { + "summary": "Tool grants per audience, file roots, approval policy and `web_fetch` limits. A configured list replaces the default list, so a list with one tool grants one tool. [Built-in tools](/reference/tools/) lists the tools and defaults, and [How tool approvals work](/security/approvals/) covers `ApprovalPolicy`." + }, + "Providers": { + "summary": "Where each model is served. [Managed providers](/configuration/managed-providers/) and [self-hosted providers](/configuration/self-hosted-providers/) have one example per provider type." + }, + "Models": { + "summary": "Which model plays Main, Fallback and Compaction, and each definition's context window, modalities, output limit and thinking level. [Model configuration](/configuration/models/) has examples. The older inline shape (`Models.Main`, `Models.Fallback`, `Models.Compaction` as objects) still loads. Mixing it with `Definitions` and `Roles` stops startup." + }, + "Memory": { + "summary": "Cross-session memory, embeddings and recall. [Memory and embeddings](/configuration/memory/) explains how the pieces fit together. The embedding model downloads on first start unless `Embeddings.AutoDownload` is `false`.", + "notes": "Set `Memory.Enabled` to `false` and the memory tools and automatic recall disappear for every audience." + }, + "Search": { + "summary": "The `web_search` backend. [Search providers](/configuration/search-providers/) covers Brave and SearXNG setup." + }, + "SkillSync": { + "summary": "Switch for the skill index tools. [Skills overview](/skills/overview/) explains where skills come from." + }, + "ExternalSkills": { + "summary": "Skill directories from other tools, such as Claude Code. [External skills](/skills/external-skills/) has the setup." + }, + "SkillFeeds": { + "summary": "Private skill servers that netclaw syncs from at startup and on a timer. [Skill feeds](/skills/skill-feeds/) has the setup." + }, + "SubAgents": { + "summary": "Switches and watchdogs for `spawn_agent`. [Custom subagents](/skills/custom-subagents/) shows how to define agents." + }, + "Notifications": { + "summary": "Where operational alerts go. [Operational alerts](/observability/operational-alerts/) lists the alert types." + }, + "Webhooks": { + "summary": "The inbound webhook endpoint. The routes themselves aren't in `netclaw.json`: each one is a file under `config/webhooks/`, managed with `netclaw webhooks` or the webhook tools. [Webhooks](/configuration/webhooks/) covers routes and verification." + }, + "Scheduling": { + "summary": "Switch for reminders and scheduled tasks. [Reminders](/configuration/reminders/) covers creating and running them.", + "notes": "Set `Scheduling.Enabled` to `false` and the five reminder tools aren't registered." + }, + "Daemon": { + "summary": "Bind address, port and how the daemon is exposed to the network. Changing any of these in `netclaw.json` restarts the daemon. [Exposure modes](/deployment/exposure-modes/) explains each mode and [Docker](/deployment/docker/) covers the container settings.", + "notes": "The Docker image's health check probes port 5199. A container with a different `Daemon.Port` reports `unhealthy` even though the daemon is up, so override the health check when you change the port.\n\nIn `Local` exposure mode the daemon answers 401 to a request whose `Host` isn't `localhost` or a loopback IP literal (`127.0.0.1`, `[::1]`), or whose `Origin`, when present, isn't `http://` plus the same host. A client that reaches the daemon by another host name, or a browser app on another port, is refused. `Daemon.Host` as a bind value still accepts spellings such as `127.1`." + }, + "Workspaces": { + "summary": "Where netclaw looks for your project repositories. [Context structure](/architecture/context-structure/) explains how projects feed the prompt." + }, + "McpServers": { + "summary": "MCP server profiles keyed by name. Add them with `netclaw mcp add` or edit the file. [MCP servers](/configuration/mcp-servers/) has the walkthrough and [MCP tool permissions](/guides/mcp-tool-permissions/) covers per-audience access." + }, + "Retention": { + "summary": "How long the daemon keeps log files before a scheduled clean-up deletes them. [Data retention](/configuration/data-retention/) explains what is and isn't deleted." + } + }, + "skip": [] +} diff --git a/scripts/gen-config-reference.mjs b/scripts/gen-config-reference.mjs new file mode 100644 index 0000000..c39a23c --- /dev/null +++ b/scripts/gen-config-reference.mjs @@ -0,0 +1,374 @@ +#!/usr/bin/env node +/** + * Generates the option tables in src/content/docs/configuration/reference.md + * from the netclaw JSON schema, plus a hand-maintained overrides file. + * + * Run by hand at each release, from the repo root. Do NOT wire this into + * `npm run build`: the product repo is not available in CI. + * + * node scripts/gen-config-reference.mjs + * node scripts/gen-config-reference.mjs --schema /path/to/netclaw-config.v1.schema.json + * NETCLAW_SCHEMA=/path/to/schema.json node scripts/gen-config-reference.mjs + * node scripts/gen-config-reference.mjs --check # exit 1 if the page is stale + * + * Inputs + * --schema JSON schema (default: $NETCLAW_SCHEMA, then the product clone at + * ~/repositories/netclaw-dev/netclaw/src/Netclaw.Configuration/Schemas/) + * --overrides scripts/config-reference-overrides.json (default). Defaults, + * descriptions and keys that the schema lacks, each with the C# file + * it came from. Re-verify these against the options classes at each release. + * --out the page (default src/content/docs/configuration/reference.md) + * + * The script rewrites only the text between the BEGIN/END GENERATED markers in the + * page. Everything else in the page is hand-written. It exits non-zero when a key + * in the schema still has no default or no description after overrides are applied, + * or when an override names a key that no longer exists, so a new release can't + * silently ship an empty cell. + */ +import { readFileSync, writeFileSync } from 'node:fs'; +import { homedir } from 'node:os'; +import { dirname, join, resolve } from 'node:path'; +import { fileURLToPath } from 'node:url'; + +const here = dirname(fileURLToPath(import.meta.url)); +const root = resolve(here, '..'); + +const args = process.argv.slice(2); +const arg = (name) => { + const i = args.indexOf(name); + return i >= 0 ? args[i + 1] : undefined; +}; +const checkOnly = args.includes('--check'); + +const defaultSchema = join( + homedir(), + 'repositories/netclaw-dev/netclaw/src/Netclaw.Configuration/Schemas/netclaw-config.v1.schema.json', +); +const schemaPath = resolve(arg('--schema') ?? process.env.NETCLAW_SCHEMA ?? defaultSchema); +const overridesPath = resolve(arg('--overrides') ?? join(here, 'config-reference-overrides.json')); +const outPath = resolve(arg('--out') ?? join(root, 'src/content/docs/configuration/reference.md')); + +const BEGIN = ''; +const END = ''; + +const schema = JSON.parse(readFileSync(schemaPath, 'utf8')); +const overrides = JSON.parse(readFileSync(overridesPath, 'utf8')); +const keyOverrides = { ...(overrides.keys ?? {}) }; +for (const prefix of overrides.shared?.prefixes ?? []) { + for (const [key, spec] of Object.entries(overrides.shared?.keys ?? {})) { + const path = `${prefix}.${key}`; + if (!(path in keyOverrides)) keyOverrides[path] = spec; + } +} +const extraKeys = overrides.extra ?? {}; +const sectionMeta = overrides.sections ?? {}; +const mapKeyNames = overrides.mapKeyNames ?? {}; +const skipKeys = new Set(overrides.skip ?? []); + +const problems = []; +const usedOverrides = new Set(); + +// ---- schema helpers ------------------------------------------------------- + +function deref(node) { + while (node && typeof node === 'object' && node.$ref) { + const parts = node.$ref.replace(/^#\//, '').split('/'); + let target = schema; + for (const p of parts) target = target[p]; + const { $ref, ...rest } = node; + node = { ...target, ...rest }; + } + return node; +} + +/** Collapse anyOf [X, null] and a multi-type with null into X plus nullable. */ +function normalize(node) { + node = deref(node); + if (node === false) return { forbidden: true }; + if (node === true || node === undefined) return { type: 'any' }; + let nullable = false; + if (Array.isArray(node.anyOf)) { + const nonNull = node.anyOf.filter((n) => !(n.type === 'null')); + nullable = nonNull.length !== node.anyOf.length; + if (nonNull.length === 1) { + const { anyOf, ...rest } = node; + node = { ...deref(nonNull[0]), ...rest }; + } + } + if (Array.isArray(node.oneOf)) { + // Models: legacy inline roles or named Definitions/Roles. Document the named shape. + const named = node.oneOf.map(deref).find((n) => n.properties && n.properties.Definitions); + if (named) { + const { oneOf, ...rest } = node; + node = { ...named, ...rest }; + } + } + if (Array.isArray(node.type)) { + nullable = nullable || node.type.includes('null'); + const types = node.type.filter((t) => t !== 'null'); + node = { ...node, type: types.length === 1 ? types[0] : types }; + } + return { ...node, nullable }; +} + +function typeLabel(node) { + const n = normalize(node); + if (n.type === 'array') { + const item = normalize(n.items ?? {}); + const inner = item.type === 'object' ? 'object' : (item.enum ? 'string' : (item.type ?? 'any')); + return `array of ${inner}` + (n.nullable ? ' (nullable)' : ''); + } + if (n.type === 'object' && n.additionalProperties && typeof n.additionalProperties === 'object' && !n.properties) { + const v = normalize(n.additionalProperties); + return `map of ${v.type === 'object' ? 'object' : (v.type ?? 'any')}` + (n.nullable ? ' (nullable)' : ''); + } + const t = Array.isArray(n.type) ? n.type.join(' or ') : (n.type ?? 'any'); + return t + (n.nullable ? ' (nullable)' : ''); +} + +const code = (s) => '`' + String(s).replace(/\|/g, '\\|') + '`'; +// Escape pipes that code() has not already escaped, and keep em dashes out of the page. +const cell = (s) => String(s ?? '-').replace(/\r?\n/g, ' ').replace(/(? v !== null).map((v) => code(v)).join(', ')); + else if (n.type === 'array') { + const item = normalize(n.items ?? {}); + if (item.enum) parts.push('items: ' + item.enum.map((v) => code(v)).join(', ')); + } else if (n.type === 'object' && n.additionalProperties && typeof n.additionalProperties === 'object') { + const v = normalize(n.additionalProperties); + if (v.enum) parts.push('values: ' + v.enum.map((x) => code(x)).join(', ')); + } + if (n.minimum !== undefined && n.maximum !== undefined) parts.push(`${n.minimum} to ${n.maximum}`); + else if (n.minimum !== undefined) parts.push(`${n.minimum} or more`); + else if (n.maximum !== undefined) parts.push(`${n.maximum} or less`); + if (n.pattern) parts.push('pattern ' + code(n.pattern)); + return parts.length ? parts.join('; ') : '-'; +} + +// ---- flattening ----------------------------------------------------------- + +const AUDIENCES = ['Public', 'Team', 'Personal']; +const rows = []; // { section, path, type, def, allowed, desc, secret } + +function mapKeyName(path) { + return mapKeyNames[path] ?? 'key'; +} + +function pushRow(section, path, node, extra = {}) { + const ov = keyOverrides[path]; + if (ov) usedOverrides.add(path); + const n = normalize(node); + const schemaDefault = n.default !== undefined ? renderDefault(n.default) : undefined; + let def = ov?.default ?? schemaDefault; + if (def === undefined && n.nullable) def = 'unset'; + const desc = ov?.description ?? n.description; + const row = { + section, + path, + type: ov?.type ?? typeLabel(node), + def, + allowed: ov?.allowed ?? allowed(node), + desc, + ...extra, + }; + if (def === undefined) problems.push(`no default: ${path}`); + if (!desc) problems.push(`no description: ${path}`); + rows.push(row); +} + +function visit(section, path, node, isRoot = false) { + const n = normalize(node); + if (n.forbidden) return; + if (skipKeys.has(path)) return; + + if (n.type === 'object' && n.properties) { + if (!isRoot && (n.description || keyOverrides[path])) { + // A container with its own description gets a row so the section reads completely. + pushContainer(section, path, n); + } + const props = Object.entries(n.properties); + const audienceProps = props.filter(([k]) => AUDIENCES.includes(k)); + const sameShape = + audienceProps.length === 3 && + new Set(audienceProps.map(([, v]) => v.$ref)).size === 1 && + audienceProps[0][1].$ref; + let emittedAudience = false; + for (const [key, child] of props) { + if (sameShape && AUDIENCES.includes(key)) { + if (!emittedAudience) { + emittedAudience = true; + visit(section, `${path}.`, child); + } + continue; + } + visit(section, `${path}.${key}`, child); + } + return; + } + + if (n.type === 'object' && n.additionalProperties && typeof n.additionalProperties === 'object') { + const v = normalize(n.additionalProperties); + if (v.type === 'object' && v.properties) { + if (keyOverrides[path]) pushContainer(section, path, { ...n, type: 'object' }); + pushContainer(section, `${path}.<${mapKeyName(path)}>`, { ...v, description: keyOverrides[`${path}.<${mapKeyName(path)}>`]?.description ?? v.description ?? n.description, type: 'object' }); + for (const [key, child] of Object.entries(v.properties)) { + visit(section, `${path}.<${mapKeyName(path)}>.${key}`, child); + } + return; + } + pushRow(section, path, node); + return; + } + + if (n.type === 'array') { + const item = normalize(n.items ?? {}); + if (item.type === 'object' && item.properties) { + pushRow(section, path, node); + for (const [key, child] of Object.entries(item.properties)) { + visit(section, `${path}[].${key}`, child); + } + return; + } + } + + pushRow(section, path, node); +} + +function pushContainer(section, path, n) { + const ov = keyOverrides[path]; + if (ov) usedOverrides.add(path); + const desc = ov?.description ?? n.description; + if (!desc) problems.push(`no description: ${path}`); + rows.push({ + section, + path, + type: ov?.type ?? 'object', + def: ov?.default ?? '-', + allowed: '-', + desc, + }); +} + +// ---- walk the schema ------------------------------------------------------ + +const sectionNames = Object.keys(schema.properties).filter((k) => k !== 'configVersion'); +const forbiddenSections = []; + +for (const name of sectionNames) { + const node = schema.properties[name]; + if (node === false) { + forbiddenSections.push(name); + continue; + } + visit(name, name, node, true); +} + +// Keys the schema doesn't carry (secrets, provider entries, keys the daemon reads that the schema omits). +for (const [path, spec] of Object.entries(extraKeys)) { + const section = path.split('.')[0]; + if (!sectionNames.includes(section)) { + problems.push(`extra key in unknown section: ${path}`); + continue; + } + rows.push({ + section, + path, + type: spec.type, + def: spec.default ?? 'unset', + allowed: spec.allowed ?? '-', + desc: spec.description, + after: spec.after, + }); + if (!spec.description || !spec.type) problems.push(`incomplete extra key: ${path}`); +} + +for (const path of Object.keys(keyOverrides)) { + if (!usedOverrides.has(path)) problems.push(`override names a key that is not in the schema: ${path}`); +} +for (const name of sectionNames) { + if (!forbiddenSections.includes(name) && !sectionMeta[name]) problems.push(`no section summary: ${name}`); +} + +// ---- render --------------------------------------------------------------- + +function renderRows(list) { + const out = []; + out.push('| Key | Type | Default | Allowed values | Description |'); + out.push('|-----|------|---------|----------------|-------------|'); + for (const r of list) { + out.push(`| ${code(r.path)} | ${cell(r.type)} | ${r.def === undefined ? '-' : cell(r.def)} | ${cell(r.allowed)} | ${cell(r.desc)} |`); + } + return out.join('\n'); +} + +/** Order rows so an extra key with `after` lands right behind the key it names. */ +function orderRows(list) { + const base = list.filter((r) => !r.after); + for (const r of list.filter((x) => x.after)) { + const idx = base.findIndex((b) => b.path === r.after); + if (idx < 0) base.push(r); + else base.splice(idx + 1, 0, r); + } + return base; +} + +const out = []; +for (const name of sectionNames) { + if (forbiddenSections.includes(name)) continue; + const meta = sectionMeta[name] ?? {}; + const list = orderRows(rows.filter((r) => r.section === name)); + out.push(`### ${name}`); + out.push(''); + if (meta.summary) { + out.push(meta.summary); + out.push(''); + } + if (list.length) { + out.push(renderRows(list)); + out.push(''); + } + if (meta.notes) { + out.push(meta.notes); + out.push(''); + } +} + +const generated = out.join('\n').replace(/\n+$/, '\n'); + +// ---- write ---------------------------------------------------------------- + +const page = readFileSync(outPath, 'utf8'); +const start = page.indexOf(BEGIN); +const end = page.indexOf(END); +if (start < 0 || end < 0 || end < start) { + console.error(`Markers not found in ${outPath}. Expected:\n${BEGIN}\n${END}`); + process.exit(2); +} +const next = page.slice(0, start + BEGIN.length) + '\n\n' + generated + '\n' + page.slice(end); + +if (problems.length) { + console.error('Config reference problems:\n ' + problems.join('\n ')); +} + +if (checkOnly) { + if (next !== page) { + console.error(`${outPath} is stale. Run: node scripts/gen-config-reference.mjs`); + process.exit(1); + } +} else if (next !== page) { + writeFileSync(outPath, next); + console.log(`Wrote ${outPath}`); +} else { + console.log(`${outPath} already up to date`); +} +console.log(`${sectionNames.length} sections, ${rows.length} rows, ${forbiddenSections.length} forbidden section(s): ${forbiddenSections.join(', ') || 'none'}`); +if (problems.length) process.exit(1); diff --git a/scripts/gen-tools-reference.mjs b/scripts/gen-tools-reference.mjs new file mode 100644 index 0000000..e47bd51 --- /dev/null +++ b/scripts/gen-tools-reference.mjs @@ -0,0 +1,355 @@ +#!/usr/bin/env node +/** + * Generates the tool tables in src/content/docs/reference/tools.md from the product source. + * + * Run by hand at each release, from the repo root. Do NOT wire this into + * `npm run build`: the product repo is not available in CI. + * + * node scripts/gen-tools-reference.mjs + * node scripts/gen-tools-reference.mjs --src /path/to/netclaw/src + * NETCLAW_SRC=/path/to/netclaw/src node scripts/gen-tools-reference.mjs + * node scripts/gen-tools-reference.mjs --check # exit 1 if the page is stale + * + * What it reads from the product source (re-read these files if the script's output looks wrong): + * - every [NetclawTool(...)] attribute: tool name, grant category, LLM-facing description + * - Netclaw.Daemon/Configuration/Channel*Tools.cs: the three channel tools declared with `Name => "..."` + * - Register / RegisterCore / ReplaceCore calls: whether a tool is loaded up front (core) or on demand (deferred) + * - Netclaw.Configuration/ToolAudienceProfiles.cs: the Public and Team default allow lists, + * which tools are profile-managed, and the Personal profile (ToolsMode All) + * - Netclaw.Actors/Tools/ToolAccessPolicy.cs: shell-coupled tools and the subsystem feature gates + * - each tool's own source file: an `audience == TrustAudience.Public` guard that makes the tool return + * "This tool is not available." for Public even when the allow list would let it through + * + * Hand-written inputs live in scripts/tools-reference-overrides.json: the one-line summary shown for each + * tool, the approval default and the config keys. The run fails when a tool has no summary, when an + * override names a tool that no longer exists, or when a source assertion no longer holds. + * + * The script rewrites only the text between the BEGIN/END GENERATED markers in the page. + */ +import { readdirSync, readFileSync, statSync, writeFileSync } from 'node:fs'; +import { homedir } from 'node:os'; +import { dirname, join, relative, resolve } from 'node:path'; +import { fileURLToPath } from 'node:url'; + +const here = dirname(fileURLToPath(import.meta.url)); +const root = resolve(here, '..'); +const args = process.argv.slice(2); +const arg = (name) => { + const i = args.indexOf(name); + return i >= 0 ? args[i + 1] : undefined; +}; +const checkOnly = args.includes('--check'); + +const srcDir = resolve(arg('--src') ?? process.env.NETCLAW_SRC ?? join(homedir(), 'repositories/netclaw-dev/netclaw/src')); +const overridesPath = resolve(arg('--overrides') ?? join(here, 'tools-reference-overrides.json')); +const outPath = resolve(arg('--out') ?? join(root, 'src/content/docs/reference/tools.md')); + +const BEGIN = ''; +const END = ''; + +const overrides = JSON.parse(readFileSync(overridesPath, 'utf8')); +const toolOverrides = overrides.tools ?? {}; +const groups = overrides.groups ?? []; +const problems = []; + +// ---- read the source tree ------------------------------------------------- + +function* walk(dir) { + for (const entry of readdirSync(dir)) { + if (entry === 'bin' || entry === 'obj' || entry === 'node_modules') continue; + const full = join(dir, entry); + if (statSync(full).isDirectory()) { + if (/Tests$/.test(entry)) continue; + yield* walk(full); + } else if (entry.endsWith('.cs')) { + yield full; + } + } +} + +const files = [...walk(srcDir)].map((path) => ({ path, rel: relative(srcDir, path), text: readFileSync(path, 'utf8') })); +const file = (rel) => { + const f = files.find((x) => x.rel === rel); + if (!f) { + problems.push(`source file not found: ${rel}`); + return { rel, text: '' }; + } + return f; +}; + +/** Returns the balanced-parenthesis argument text that starts at text[open] === '('. */ +function balanced(text, open) { + let depth = 0; + let inString = false; + for (let i = open; i < text.length; i++) { + const c = text[i]; + if (inString) { + if (c === '\\') i++; + else if (c === '"') inString = false; + continue; + } + if (c === '"') inString = true; + else if (c === '(') depth++; + else if (c === ')') { + depth--; + if (depth === 0) return text.slice(open + 1, i); + } + } + return ''; +} + +/** Splits on commas that are not inside a string or nested brackets. */ +function splitArgs(text) { + const parts = []; + let depth = 0; + let inString = false; + let current = ''; + for (let i = 0; i < text.length; i++) { + const c = text[i]; + if (inString) { + current += c; + if (c === '\\') current += text[++i]; + else if (c === '"') inString = false; + continue; + } + if (c === '"') inString = true; + if ('([{'.includes(c)) depth++; + if (')]}'.includes(c)) depth--; + if (c === ',' && depth === 0) { + parts.push(current.trim()); + current = ''; + } else current += c; + } + if (current.trim()) parts.push(current.trim()); + return parts; +} + +const stringLiterals = (expr) => [...expr.matchAll(/"((?:[^"\\]|\\.)*)"/g)].map((m) => m[1].replace(/\\"/g, '"')).join(''); + +// ---- tools declared with [NetclawTool] ----------------------------------- + +const tools = new Map(); // name -> { name, className, grant, description, source } + +for (const f of files) { + let from = 0; + for (;;) { + const at = f.text.indexOf('[NetclawTool(', from); + if (at < 0) break; + from = at + 1; + const open = at + '[NetclawTool'.length; + const argText = balanced(f.text, open); + const rest = f.text.slice(open + argText.length + 2); + const cls = /class\s+(\w+)/.exec(rest)?.[1]; + if (!cls || !argText) continue; + const parts = splitArgs(argText); + if (parts.length < 2) continue; + let name; + if (parts[0].startsWith('"')) name = stringLiterals(parts[0]); + else { + const constName = parts[0].trim(); + name = new RegExp(`const\\s+string\\s+${constName}\\s*=\\s*"([^"]+)"`).exec(f.text)?.[1]; + } + if (!name || !/^[a-z_]+$/.test(name)) continue; // skips doc-comment mentions and non-tool uses + const grant = /Grant\s*=\s*"([^"]+)"/.exec(argText)?.[1] ?? 'default'; + tools.set(name, { name, className: cls, grant, description: stringLiterals(parts[1]), source: f.rel }); + } +} + +// ---- channel tools (IChannelTool implementations) ------------------------- + +const channelRegistration = file('Netclaw.Daemon/Configuration/ChannelToolRegistration.cs').text; +const channelToolsDeferred = /registry\.Register\(tool\)/.test(channelRegistration); +if (!channelToolsDeferred) problems.push('ChannelToolRegistration no longer calls registry.Register(tool): re-check how channel tools load'); + +for (const rel of ['Netclaw.Daemon/Configuration/ChannelSendTools.cs', 'Netclaw.Daemon/Configuration/ChannelLookupTools.cs']) { + const f = file(rel); + for (const m of f.text.matchAll(/class\s+(\w+)[^{]*?\{[\s\S]*?string Name\s*=>\s*"(\w+)";/g)) { + tools.set(m[2], { name: m[2], className: m[1], grant: 'channel', description: '', source: rel, channel: true }); + } +} + +// Channel-specific lookup tools declared with [NetclawTool] already have a grant of "default" and live in the channel projects. + +// ---- loading tier --------------------------------------------------------- + +const tierByClass = new Map(); +for (const f of files) { + for (const m of f.text.matchAll(/\.(RegisterCore|Register|ReplaceCore)\(\s*new\s+(\w+)\(/g)) { + const tier = m[1] === 'Register' ? 'Deferred' : 'Core'; + if (tierByClass.get(m[2]) !== 'Core') tierByClass.set(m[2], tier); + } +} + +// ---- audience defaults ---------------------------------------------------- + +const profilesFile = file('Netclaw.Configuration/ToolAudienceProfiles.cs'); +const catalogText = /static class ToolAudienceProfileToolCatalog[\s\S]*?static class ToolAudienceProfileDefaults/.exec(profilesFile.text)?.[0] ?? ''; +const constants = new Map([...catalogText.matchAll(/public const string (\w+) = "([^"]+)";/g)].map((m) => [m[1], m[2]])); +const lists = new Map(); +for (const m of catalogText.matchAll(/public static IReadOnlyList (\w+) \{ get; \} =\s*\[([\s\S]*?)\];/g)) { + const items = []; + for (const raw of m[2].split(',').map((s) => s.trim()).filter(Boolean)) { + if (raw.startsWith('..')) items.push(...(lists.get(raw.slice(2).trim()) ?? [])); + else if (constants.has(raw)) items.push(constants.get(raw)); + else problems.push(`unresolved catalog item ${raw} in ${m[1]}`); + } + lists.set(m[1], items); +} +const publicDefault = lists.get('PublicDefaultAllowedTools') ?? []; +const teamDefault = lists.get('TeamDefaultAllowedTools') ?? []; +const profileManaged = new Set(lists.get('ProfileManagedTools') ?? []); +if (!publicDefault.length || !teamDefault.length || !profileManaged.size) problems.push('could not read the audience default lists from ToolAudienceProfiles.cs'); + +const personalAll = /CreatePersonal\(\)[\s\S]*?ToolsMode\s*=\s*ToolProfileMode\.All/.test(profilesFile.text); +if (!personalAll) problems.push('CreatePersonal no longer sets ToolsMode = All: the Personal column is wrong'); +const shellApproval = /CreateProfilesForPosture[\s\S]*?\[ToolAudienceProfileToolCatalog\.ShellExecute\]\s*=\s*ToolApprovalMode\.Approval/.test(profilesFile.text); +if (!shellApproval) problems.push('the Personal posture no longer defaults shell_execute to Approval: re-check the approval column'); + +const policy = file('Netclaw.Actors/Tools/ToolAccessPolicy.cs').text; +const shellCoupled = /IsShellCoupledTool\(INetclawTool tool\)\s*=>\s*IsShellTool\(tool\)\s*\|\|\s*string\.Equals\(tool\.Name, CheckBackgroundJobTool\.ToolName/.test(policy); +if (!shellCoupled) problems.push('IsShellCoupledTool changed: re-check which tools need shell execution'); +const shellCoupledTools = ['shell_execute', 'check_background_job']; + +const gateConfig = { + MemoryEnabled: 'Memory.Enabled', + SearchEnabled: 'Search.Enabled', + SkillSyncEnabled: 'SkillSync.Enabled', + SubAgentsEnabled: 'SubAgents.Enabled', + SchedulingEnabled: 'Scheduling.Enabled', +}; +const gateByTool = new Map(); +const gateBlock = /IsFeatureDisabledTool\(string toolName\)[\s\S]*?\n \}/.exec(policy)?.[0] ?? ''; +for (const m of gateBlock.matchAll(/((?:"\w+"\s*(?:or\s*)?)+)\s*=>\s*!_featureGates\.(\w+)/g)) { + for (const t of m[1].matchAll(/"(\w+)"/g)) gateByTool.set(t[1], gateConfig[m[2]] ?? m[2]); +} +if (!gateByTool.size) problems.push('could not read the feature gates from ToolAccessPolicy.IsFeatureDisabledTool'); + +// ---- compute rows --------------------------------------------------------- + +// Tools whose own code refuses Public regardless of the allow lists. +const publicGuard = /^[^/\n]*(?:audience|Audience)\s*==\s*TrustAudience\.Public/m; +const publicBlocked = new Set([...tools.values()].filter((t) => publicGuard.test(files.find((f) => f.rel === t.source)?.text ?? '')).map((t) => t.name)); +for (const name of Object.keys(toolOverrides)) { + if (toolOverrides[name].publicBlocked && !publicBlocked.has(name)) problems.push(`override says ${name} is blocked for Public but the source has no guard`); +} +for (const name of publicBlocked) { + if (!toolOverrides[name]?.publicBlocked) problems.push(`source blocks ${name} for Public: add "publicBlocked": true to its override`); +} + +function availability(tool) { + const n = tool.name; + if (publicBlocked.has(n)) { + return { Public: 'No', Team: profileManaged.has(n) && !teamDefault.includes(n) ? 'No' : 'Yes', Personal: 'Yes', profileManaged: profileManaged.has(n) }; + } + if (!profileManaged.has(n)) return { Public: 'Yes', Team: 'Yes', Personal: 'Yes', profileManaged: false }; + const inPublic = publicDefault.includes(n); + const inTeam = teamDefault.includes(n); + return { + Public: inPublic ? 'Yes' : 'No', + Team: inTeam ? 'Yes' : 'No', + Personal: 'Yes', + profileManaged: true, + }; +} + +const rows = [...tools.values()].map((t) => { + const ov = toolOverrides[t.name]; + if (!ov) problems.push(`no override (summary, group) for tool: ${t.name}`); + const a = availability(t); + const requires = []; + if (shellCoupledTools.includes(t.name)) requires.push('`Tools.ShellMode` or `Security.ShellExecutionMode`, which must resolve to `HostAllowed`'); + if (gateByTool.has(t.name)) requires.push(`\`${gateByTool.get(t.name)}\``); + if (ov?.requires) requires.push(...ov.requires); + const tier = (t.channel || t.source.startsWith('Netclaw.Channels.')) ? 'Deferred' : (tierByClass.get(t.className) ?? 'Deferred'); + if (!t.channel && !t.source.startsWith('Netclaw.Channels.') && !tierByClass.has(t.className)) problems.push(`no Register call found for ${t.className}: loading tier assumed Deferred`); + const shell = shellCoupledTools.includes(t.name); + return { + ...t, + ...a, + Public: shell ? 'No' : a.Public, + Team: shell ? 'No' : a.Team, + tier, + requires, + summary: ov?.summary, + group: ov?.group, + approval: ov?.approval ?? 'Runs without a prompt', + config: ov?.config ?? [], + }; +}); + +for (const name of Object.keys(toolOverrides)) { + if (!tools.has(name)) problems.push(`override names a tool that is not in the source: ${name}`); +} +for (const g of groups) { + if (!rows.some((r) => r.group === g.id)) problems.push(`group with no tools: ${g.id}`); +} +for (const r of rows) { + if (r.group && !groups.some((g) => g.id === r.group)) problems.push(`unknown group ${r.group} for ${r.name}`); +} + +// ---- render --------------------------------------------------------------- + +const cell = (s) => String(s).replace(/(? r.tier === 'Core').length; +out.push(`netclaw 0.27.1 registers ${count} built-in tools. ${coreCount} are loaded into every prompt (core) and the rest are found with \`search_tools\` and loaded with \`load_tool\` (deferred). The table columns say which audiences get each tool by default. A tool outside the profile-managed set is available to every audience when its subsystem is enabled, except that Public never gets ${[...publicBlocked].sort().map((t) => `\`${t}\``).join(', ').replace(/, ([^,]*)$/, ' or $1')}: those tools refuse Public in their own code.`.replace('0.27.1', overrides.version ?? '0.27.1')); +out.push(''); + +for (const g of groups) { + const list = rows.filter((r) => r.group === g.id).sort((x, y) => (toolOverrides[x.name]?.order ?? 0) - (toolOverrides[y.name]?.order ?? 0) || x.name.localeCompare(y.name)); + if (!list.length) continue; + out.push(`### ${g.title}`); + out.push(''); + if (g.intro) { + out.push(g.intro); + out.push(''); + } + out.push('| Tool | What it does | Loaded | Public | Team | Personal | Approval by default | Controlled by |'); + out.push('|------|--------------|--------|--------|------|----------|---------------------|---------------|'); + for (const r of list) { + const needs = [...r.requires, ...r.config.map((c) => `\`${c}\``)]; + out.push(`| \`${r.name}\` | ${cell(r.summary ?? '')} | ${r.tier} | ${r.Public} | ${r.Team} | ${r.Personal} | ${cell(r.approval)} | ${cell(needs.length ? needs.join(', ') : '-')} |`); + } + out.push(''); +} + +out.push('### Default allow lists'); +out.push(''); +out.push('The lists below are the 0.27.1 defaults for `Tools.AudienceProfiles..AllowedTools`. Personal uses `ToolsMode: All` and has no list.'.replace('0.27.1', overrides.version ?? '0.27.1')); +out.push(''); +out.push('| Audience | Default `AllowedTools` |'); +out.push('|----------|------------------------|'); +out.push(`| Public | ${publicDefault.map((t) => `\`${t}\``).join(', ')} |`); +out.push(`| Team | ${teamDefault.map((t) => `\`${t}\``).join(', ')} |`); +out.push(''); + +const generated = out.join('\n'); + +// ---- write ---------------------------------------------------------------- + +const page = readFileSync(outPath, 'utf8'); +const start = page.indexOf(BEGIN); +const end = page.indexOf(END); +if (start < 0 || end < 0 || end < start) { + console.error(`Markers not found in ${outPath}. Expected:\n${BEGIN}\n${END}`); + process.exit(2); +} +const next = page.slice(0, start + BEGIN.length) + '\n\n' + generated + '\n' + page.slice(end); + +if (problems.length) console.error('Tools reference problems:\n ' + problems.join('\n ')); + +if (checkOnly) { + if (next !== page) { + console.error(`${outPath} is stale. Run: node scripts/gen-tools-reference.mjs`); + process.exit(1); + } +} else if (next !== page) { + writeFileSync(outPath, next); + console.log(`Wrote ${outPath}`); +} else { + console.log(`${outPath} already up to date`); +} +console.log(`${count} tools (${coreCount} core), public default ${publicDefault.length}, team default ${teamDefault.length}`); +if (problems.length) process.exit(1); diff --git a/scripts/generate-social-images.mjs b/scripts/generate-social-images.mjs index 06e8ac9..50aaafb 100644 --- a/scripts/generate-social-images.mjs +++ b/scripts/generate-social-images.mjs @@ -24,6 +24,9 @@ const SECTION_MAP = { 'deployment': 'Deployment', 'architecture': 'Architecture', 'guides': 'Guides', + 'reference': 'Reference', + 'troubleshooting': 'Troubleshooting', + 'comparisons': 'Comparisons', }; function loadManifest() { diff --git a/scripts/screenshot-page.mjs b/scripts/screenshot-page.mjs index 9c2cdf3..1418ef1 100755 --- a/scripts/screenshot-page.mjs +++ b/scripts/screenshot-page.mjs @@ -4,16 +4,23 @@ // Usage: node scripts/screenshot-page.mjs // Example: node scripts/screenshot-page.mjs cli/status // -// Requires: npm run build (must be run first) +// Run it in the Playwright container so the browser is the pinned, containerized one: +// ./screenshots/shoot-page.sh cli/status +// +// Serves dist/ itself with a tiny static server (no astro needed), so it runs anywhere +// the `playwright` module resolves. Requires `npm run build` first. // Output: docs-logs/screenshots/.png -// Timeout: 60 seconds total, then kills everything and exits non-zero. +// +// Env: +// SCREENSHOT_OUT output directory (default docs-logs/screenshots) +// SCREENSHOT_BASE_URL screenshot a running server (e.g. `npm run dev`) instead of dist/ +// Timeout: 60 seconds total, then exits non-zero. import { chromium } from 'playwright'; -import { spawn } from 'child_process'; -import { mkdirSync, existsSync } from 'fs'; -import { dirname, join } from 'path'; +import { createServer } from 'http'; +import { mkdirSync, existsSync, statSync, createReadStream } from 'fs'; +import { dirname, join, extname, normalize } from 'path'; import { fileURLToPath } from 'url'; -import { createServer } from 'net'; const TIMEOUT_MS = 60_000; @@ -27,64 +34,67 @@ if (!slug) { process.exit(1); } -const outputDir = join(projectDir, 'docs-logs', 'screenshots'); +const outputDir = process.env.SCREENSHOT_OUT || join(projectDir, 'docs-logs', 'screenshots'); const outputFile = join(outputDir, slug.replace(/\//g, '-') + '.png'); mkdirSync(outputDir, { recursive: true }); const distDir = join(projectDir, 'dist'); -if (!existsSync(distDir)) { - console.error('dist/ not found — run npm run build first'); +const baseUrl = process.env.SCREENSHOT_BASE_URL?.replace(/\/$/, ''); +if (!baseUrl && !existsSync(distDir)) { + console.error('dist/ not found, run npm run build first'); process.exit(1); } -function findFreePort() { - return new Promise((resolve, reject) => { - const srv = createServer(); - srv.listen(0, () => { - const port = srv.address().port; - srv.close(() => resolve(port)); - }); - srv.on('error', reject); - }); +const MIME = { + '.html': 'text/html; charset=utf-8', '.css': 'text/css', '.js': 'text/javascript', '.mjs': 'text/javascript', + '.json': 'application/json', '.svg': 'image/svg+xml', '.png': 'image/png', '.jpg': 'image/jpeg', + '.gif': 'image/gif', '.webp': 'image/webp', '.ico': 'image/x-icon', '.woff': 'font/woff', + '.woff2': 'font/woff2', '.txt': 'text/plain', '.xml': 'application/xml', +}; + +function resolveFile(urlPath) { + const rel = normalize(decodeURIComponent(urlPath.split('?')[0])).replace(/^(\.\.[/\\])+/, ''); + let file = join(distDir, rel); + if (existsSync(file) && statSync(file).isDirectory()) file = join(file, 'index.html'); + else if (!existsSync(file) && existsSync(join(file, 'index.html'))) file = join(file, 'index.html'); + return existsSync(file) && statSync(file).isFile() ? file : null; } -let preview; +let server; const killTimer = setTimeout(() => { console.error('Screenshot timed out after 60s'); - if (preview) preview.kill('SIGKILL'); + server?.close(); process.exit(1); }, TIMEOUT_MS); +let browser; try { - const port = await findFreePort(); - const url = `http://localhost:${port}/${slug}/`; - - preview = spawn('npx', ['astro', 'preview', '--port', String(port)], { - cwd: projectDir, - stdio: ['ignore', 'pipe', 'pipe'], - }); - - const start = Date.now(); - while (Date.now() - start < 15_000) { - try { - const res = await fetch(url); - if (res.ok) break; - } catch {} - await new Promise(r => setTimeout(r, 300)); + let origin = baseUrl; + if (!origin) { + server = createServer((req, res) => { + const file = resolveFile(req.url); + if (!file) { res.writeHead(404); res.end('not found'); return; } + res.writeHead(200, { 'Content-Type': MIME[extname(file)] || 'application/octet-stream' }); + createReadStream(file).pipe(res); + }); + await new Promise(resolve => server.listen(0, '127.0.0.1', resolve)); + origin = `http://127.0.0.1:${server.address().port}`; } + const url = `${origin}/${slug}/`; - const browser = await chromium.launch(); + browser = await chromium.launch(); const page = await browser.newPage({ viewport: { width: 1280, height: 900 } }); - await page.goto(url, { waitUntil: 'networkidle', timeout: 15_000 }); + const response = await page.goto(url, { waitUntil: 'networkidle', timeout: 15_000 }); + if (!response || !response.ok()) throw new Error(`${url} returned ${response?.status()}`); await page.waitForTimeout(500); await page.screenshot({ path: outputFile, fullPage: true }); - await browser.close(); console.log(outputFile); } catch (err) { console.error('Screenshot failed:', err.message); - process.exit(1); + process.exitCode = 1; } finally { clearTimeout(killTimer); - if (preview) preview.kill('SIGTERM'); + await browser?.close(); + server?.close(); } diff --git a/scripts/tools-reference-overrides.json b/scripts/tools-reference-overrides.json new file mode 100644 index 0000000..732b66d --- /dev/null +++ b/scripts/tools-reference-overrides.json @@ -0,0 +1,196 @@ +{ + "_readme": [ + "Hand-maintained inputs for scripts/gen-tools-reference.mjs.", + "The script discovers tool names, loading tier, audience availability, feature gates and shell coupling from the product source. This file adds what source can't say in user terms:", + "a one-line summary, the display group, the default approval behaviour and the config keys a reader would reach for.", + "Approval defaults come from Netclaw.Configuration/ToolAudienceProfiles.cs (CreateProfilesForPosture), Netclaw.Actors/Tools/FilePathApprovalMatcher.cs and docs/runbooks/tool-approval-gates.md. Re-read them at each release.", + "A tool that appears in source but not here fails the run, so a new tool can't ship undocumented." + ], + "version": "0.28.2", + + "groups": [ + { "id": "files", "title": "Files", "intro": "File tools respect the audience's `ReadFiles`, `WriteFiles` and `AttachFiles` roots. By default Public reaches only its own session directory, Team adds `GlobalReadRoots` for reads, and Personal reaches any path that isn't protected." }, + { "id": "shell", "title": "Shell and background jobs" }, + { "id": "web", "title": "Web" }, + { "id": "memory", "title": "Memory" }, + { "id": "skills", "title": "Skills and tool discovery" }, + { "id": "reminders", "title": "Reminders" }, + { "id": "webhooks", "title": "Webhooks" }, + { "id": "agents", "title": "Subagents" }, + { "id": "channels", "title": "Chat channels" } + ], + + "tools": { + "file_read": { + "group": "files", "order": 1, + "summary": "Read a text file. Images load for models that accept them, and PDFs, media and archives return metadata.", + "approval": "Runs without a prompt. Protected files such as `secrets.json` and credential folders such as `~/.ssh` are denied." + }, + "file_list": { + "group": "files", "order": 2, + "summary": "List one directory without recursing." + }, + "file_search": { + "group": "files", "order": 3, + "summary": "Search authorized files by name or UTF-8 text, recursively, without a shell." + }, + "tool_output_read": { + "group": "files", "order": 4, + "summary": "Read the rest of a truncated tool result by call ID. Large output is spilled to a session file and this tool pages through it." + }, + "file_write": { + "group": "files", "order": 5, + "summary": "Create or replace a file, creating parent directories.", + "approval": "Runs without a prompt. Writes under the config directory are always denied." + }, + "file_edit": { + "group": "files", "order": 6, + "summary": "Replace literal text in a file, or write the whole file.", + "approval": "Runs without a prompt. Edits under the config directory are always denied." + }, + "attach_file": { + "group": "files", "order": 7, + "summary": "Send an existing file to the user in the chat." + }, + "set_working_directory": { + "group": "files", "order": 8, + "summary": "Declare the project directory for the session. It scopes path checks and shell approvals and loads the project's `AGENTS.md`." + }, + + "shell_execute": { + "group": "shell", "order": 1, + "summary": "Run a shell command. Pass `_background: true` to run it as a background job.", + "approval": "Prompts on Personal unless a saved grant or the reviewed-safe command list covers it. Never available to Team or Public." + }, + "check_background_job": { + "group": "shell", "order": 2, + "summary": "Check the status and output tail of a background job, or cancel it." + }, + + "web_search": { + "group": "web", "order": 1, + "summary": "Search the web and return titles, URLs and snippets.", + "config": ["Search.Backend"] + }, + "web_fetch": { + "group": "web", "order": 2, + "summary": "Download a URL to a local file and return a preview.", + "config": ["Tools.WebFetch.RequireHttps", "Tools.WebFetch.HttpAllowList"] + }, + + "store_memory": { + "group": "memory", "order": 1, + "summary": "Save knowledge to cross-session memory." + }, + "find_memories": { + "group": "memory", "order": 2, + "summary": "Search memory. Returns IDs, titles, scores and snippets." + }, + "get_memories": { + "group": "memory", "order": 3, + "summary": "Load the full text of memories by ID." + }, + "update_memory": { + "group": "memory", "order": 4, + "summary": "Edit or delete a memory by ID." + }, + + "skill_load": { + "publicBlocked": true, + "group": "skills", "order": 1, + "summary": "Load a skill's instructions and list its reference files." + }, + "skill_read_resource": { + "publicBlocked": true, + "group": "skills", "order": 2, + "summary": "Read a resource file from a skill directory." + }, + "skill_manage": { + "group": "skills", "order": 3, + "summary": "Create, edit, patch or delete skills and their resource files." + }, + "search_tools": { + "group": "skills", "order": 4, + "summary": "Search deferred tools by keyword, or list MCP servers. Doesn't load anything." + }, + "load_tool": { + "group": "skills", "order": 5, + "summary": "Load a deferred tool by exact name so the model can call it.", + "config": ["Session.DiscoveredToolRetentionTurns", "Session.DiscoveredToolMaxCount"] + }, + + "set_reminder": { + "group": "reminders", "order": 1, + "summary": "Create or update a reminder from a relative time, an `every 6h` interval or a cron expression." + }, + "list_reminders": { + "group": "reminders", "order": 2, + "summary": "List reminders with schedules, status and next run." + }, + "cancel_reminder": { + "group": "reminders", "order": 3, + "summary": "Cancel a reminder. A one-shot is deleted and its history stays readable for 12 days. A recurring reminder is disabled and kept." + }, + "get_reminder_history": { + "group": "reminders", "order": 4, + "summary": "Show recent runs of a reminder with outcome, duration and session ID." + }, + "run_reminder": { + "group": "reminders", "order": 5, + "summary": "Run a reminder's prompt in the current chat so you can answer approval prompts and save grants for the unattended run." + }, + + "set_webhook": { + "group": "webhooks", "order": 1, + "summary": "Create or update an inbound webhook route.", + "requires": ["`Webhooks.Enabled`"] + }, + "list_webhooks": { + "group": "webhooks", "order": 2, + "summary": "List webhook routes and whether each one loads.", + "requires": ["`Webhooks.Enabled`"] + }, + "delete_webhook": { + "group": "webhooks", "order": 3, + "summary": "Delete a webhook route by name.", + "requires": ["`Webhooks.Enabled`"] + }, + + "spawn_agent": { + "publicBlocked": true, + "group": "agents", "order": 1, + "summary": "Hand a task to a subagent. It returns a run ID at once, the child works in the background, and the result arrives in a later turn as a notice that names its `result.md`.", + "config": ["SubAgents.PrefillTimeoutSeconds", "SubAgents.NoProgressTimeoutSeconds"] + }, + "check_agent_run": { + "group": "agents", "order": 2, + "summary": "Read the status of a child run you started, or ask it to cancel. Subagents can't call it." + }, + + "send_channel_message": { + "group": "channels", "order": 1, + "summary": "Post to a channel or send a direct message on Slack, Discord or Mattermost, using IDs from the lookup tools.", + "requires": ["a chat channel enabled"] + }, + "lookup_channel_user": { + "group": "channels", "order": 2, + "summary": "Resolve a person to a stable user ID on an enabled channel.", + "requires": ["a chat channel enabled"] + }, + "lookup_channel_destination": { + "group": "channels", "order": 3, + "summary": "Resolve a channel name to a stable ID, or list every destination the bot can reach.", + "requires": ["a chat channel enabled"] + }, + "lookup_slack_user": { + "group": "channels", "order": 4, + "summary": "Find a Slack user by name, display name or email.", + "requires": ["`Slack.Enabled`"] + }, + "lookup_mattermost_user": { + "group": "channels", "order": 5, + "summary": "Find a Mattermost user by username or email.", + "requires": ["`Mattermost.Enabled`"] + } + } +} diff --git a/src/content/docs/cli/approvals.md b/src/content/docs/cli/approvals.md index 130cf43..cc6bf5b 100644 --- a/src/content/docs/cli/approvals.md +++ b/src/content/docs/cli/approvals.md @@ -1,187 +1,212 @@ --- title: "netclaw approvals" -description: "Inspect, revoke, and manage persistent tool approvals." +description: "List, revoke and pre-approve the grants netclaw saves when you answer Always at an approval prompt." --- -When netclaw's [approval gates](/architecture/security-model/#human-in-the-loop-as-a-layer-not-a-crutch) prompt you to allow a tool invocation, your "always allow" choices get saved to `~/.netclaw/config/tool-approvals.json`. This command lets you see what's been approved, revoke entries you no longer want, and add global trust rules. Running `netclaw approvals` with no subcommand opens the [interactive TUI](#tui). +When you answer **Always here**, **This repository** or **Always anywhere** at an approval prompt, netclaw saves a grant to `~/.netclaw/config/tool-approvals.json`. `netclaw approvals` lists those grants, revokes the ones you no longer want, and adds new ones before a prompt ever appears. Run it with no subcommand to open the [interactive TUI](#tui). -![Approval gate prompt in a Slack thread, with approve and deny options](/assets/approval-prompt.png) +This page is the reference for the command and the file. For how grants fit into the authorization model, see [How tool approvals work](/security/approvals/). To stop prompts for a command or a tool, see [Reduce or turn off approval prompts](/guides/reduce-approval-prompts/). -The two "Always" options in the channel prompt — **Always here** (the command's verb, scoped to the current directory) and **Always anywhere** (a global grant) — are what create persistent entries in `tool-approvals.json`. - -Audiences (`personal`, `team`, `public`) scope approvals to who's talking to the agent. An approval granted under `personal` doesn't apply when a team member triggers the same command. See [audience dispositions](/architecture/security-model/#audience-as-the-trust-primitive) for background. - -Don't hand-edit the JSON file. Use this command instead. +Grants belong to one audience (`personal`, `team` or `public`) and one tool. A `personal` grant doesn't apply when a Team user triggers the same command. Don't hand-edit the file. The CLI keeps entries valid. ## Usage ```bash -netclaw approvals # open TUI -netclaw approvals list [options] # print approvals -netclaw approvals revoke [options] # remove an approval -netclaw approvals trust-verb [options] # add a global approval -netclaw approvals tui # open TUI (explicit) -netclaw approvals help # show help +netclaw approvals # open the TUI +netclaw approvals list [options] # print grants +netclaw approvals revoke