From ffbe5f5b0552b8bf986f43f32a78a7175e2db7f2 Mon Sep 17 00:00:00 2001 From: Aaron Stannard Date: Thu, 29 Oct 2026 14:08:13 -0500 Subject: [PATCH] docs: channels, deployment and observability for 0.28 --- src/content/docs/channels/discord.md | 251 +++++------ src/content/docs/channels/mattermost.md | 275 +++++++----- src/content/docs/channels/slack.md | 285 +++++++------ src/content/docs/channels/troubleshooting.md | 396 +++++++++--------- src/content/docs/deployment/docker.md | 358 ++++++++++------ src/content/docs/deployment/exposure-modes.md | 120 +++--- src/content/docs/deployment/systemd.md | 207 +++++---- .../docs/observability/health-checks.md | 203 ++++----- .../docs/observability/opentelemetry.md | 104 +++-- .../docs/observability/operational-alerts.md | 111 +++-- 10 files changed, 1329 insertions(+), 981 deletions(-) diff --git a/src/content/docs/channels/discord.md b/src/content/docs/channels/discord.md index 0e2a175..dd36cfe 100644 --- a/src/content/docs/channels/discord.md +++ b/src/content/docs/channels/discord.md @@ -1,55 +1,70 @@ --- title: "Discord" -description: "Connect Netclaw to your Discord server." +description: "Connect netclaw to your Discord server with a bot token, the Message Content intent and an allow-list." --- -Netclaw connects to Discord over the [Gateway WebSocket API](https://discord.com/developers/docs/events/gateway) — outbound connections only, no public URLs needed. You create a Discord bot, give netclaw one token, and it shows up in your server. +netclaw connects to Discord over the [Gateway WebSocket API](https://docs.discord.com/developers/topics/gateway), which means outbound connections only and no public URL. You create a Discord bot, give netclaw one token, and it shows up in your server and answers in threads. ## Prerequisites -- Netclaw installed and initialized ([`netclaw init`](/cli/init/)) -- A Discord server where you have the "Manage Server" permission -- [Developer Mode](https://support.discord.com/hc/en-us/articles/206346498) enabled in Discord (for copying IDs) +- netclaw installed and initialized ([`netclaw init`](/cli/init/)) +- A Discord server where you have the **Manage Server** permission +- [Developer Mode](https://support.discord.com/hc/en-us/articles/206346498) turned on in Discord, so you can copy IDs ## Create a Discord bot -Head to [discord.com/developers/applications](https://discord.com/developers/applications) and create a new application. +Open the [Discord Developer Portal](https://discord.com/developers/applications) and create an application. ![Discord Applications page](/screenshots/output/discord-setup-applications.png) +The Applications page lists your apps and has the **New Application** button. + ### 1. Create the application -Click **New Application**, give it a name, and hit Create. +Click **New Application**, name it, and click Create. ![New Application dialog](/screenshots/output/discord-setup-create-app.png) +The name you pick here is what members see on the bot. + ### 2. Copy the bot token -Go to **Bot** in the left sidebar. Click **Reset Token** to generate a bot token and copy it immediately — Discord only shows it once. +Open **Bot** in the sidebar, click **Reset Token**, and copy the token right away. Discord shows it once. ![Bot settings page](/screenshots/output/discord-setup-bot-settings.png) -If you lose the token, you can always reset it here, but you'll need to update your netclaw config with the new value. +If you lose it, reset it here and update netclaw with the new value. ### 3. Enable Message Content Intent -Scroll down on the Bot page to **Privileged Gateway Intents** and enable **Message Content**. Without this, netclaw receives message events but can't read their text. +Scroll to **Privileged Gateway Intents** and turn on **Message Content**. It's the only privileged intent netclaw needs, so leave Server Members and Presence off. Without it netclaw receives message events but can't read their text, except in DMs and when the message mentions the bot. ![Privileged Gateway Intents with Message Content enabled](/screenshots/output/discord-setup-bot-intents.png) -### 4. Set up OAuth2 scopes and permissions +The Message Content toggle sits at the bottom of the privileged intents list. + +### 4. Set OAuth2 scopes and permissions -Go to **OAuth2 > URL Generator**. Check the `bot` scope. +Open **OAuth2 > URL Generator** and check the `bot` scope. ![OAuth2 scopes selection](/screenshots/output/discord-setup-oauth-scopes.png) -Then select bot permissions: +Then select the bot permissions. ![Bot permissions checklist](/screenshots/output/discord-setup-bot-permissions.png) -**Required permissions:** Send Messages, Create Public Threads, Send Messages in Threads, Manage Threads, Embed Links, Read Message History, Add Reactions. +The permissions netclaw uses: -Manage Threads is needed so netclaw can rename threads with session titles as conversations progress. +| Permission | Why | +|------------|-----| +| Send Messages | Post replies | +| Create Public Threads | Open a thread on the first reply | +| Send Messages in Threads | Reply inside threads | +| Manage Threads | Rename a thread to the session title. Discord normally lets a thread's creator rename it, so this may be unnecessary. We haven't tested without it. | +| Read Message History | Backfill thread history after a restart | +| Attach Files | Upload files the agent produces | + +The checklist in the screenshot also ticks View Channels, Create Private Threads, Embed Links and Add Reactions. netclaw doesn't call Create Private Threads, Embed Links or Add Reactions, so you can leave those off. Keep View Channels on so the bot can see the channels you add it to. The table is the source of truth for what netclaw needs. ### 5. Install the bot to your server @@ -57,41 +72,37 @@ Copy the generated URL at the bottom of the page. ![Generated OAuth2 URL](/screenshots/output/discord-setup-oauth-url.png) -Open it in your browser. Discord asks which server to add the bot to — pick yours and click **Continue**. +Open it in a browser. Discord asks which server to add the bot to. Pick yours and click **Continue**. ![OAuth install approval dialog](/screenshots/output/discord-setup-install-dialog.png) -Discord then lists the permissions the bot is requesting. Confirm they match the set from step 4 and click **Authorize**. +The server list only offers servers where you have the Manage Server permission. + +Discord then lists the permissions the bot is requesting. Check they match step 4 and click **Authorize**. ![Discord OAuth2 permissions confirmation with the Authorize button](/screenshots/output/discord-setup-authorize-dialog.png) -The bot appears in your server's member list once the daemon is running. +The bot appears in the member list. It shows as online once the daemon is running. ### 6. Set Installation to None -Under **Installation** in the left sidebar, set the install method to **None**. This prevents users from installing the bot to other servers through Discord's app directory. +Under **Installation** in the sidebar, set the install method to **None**. That stops people installing the bot to other servers from the app directory. ![Installation settings set to None](/screenshots/output/discord-setup-installation-none.png) -If you want others to install via a link you control, use **Discord Provided Link** instead. - -![Installation with Discord Provided Link](/screenshots/output/discord-setup-installation-provided-link.png) +If you want others to install through a link you control, choose **Discord Provided Link** instead. ## Configure netclaw -Run [`netclaw config`](/cli/config/), navigate to **Channels**, and enable Discord. Enter your bot token when prompted — netclaw stores it encrypted and saves the config automatically. - -```bash -netclaw config -``` +Run [`netclaw config`](/cli/config/), open **Channels**, enable Discord, and enter the bot token. netclaw stores it encrypted and saves the config. ![The Channels area in netclaw config](/screenshots/output/config-channels-menu.png) -The Channels area in `netclaw config` — enable Discord and enter its bot token here (the adapter list also shows Slack and Mattermost). +The Channels area lists Slack, Discord and Mattermost with a checkbox each. Once an adapter is set up, its row also shows how many channels and users it allows. ### Manual setup -For scripted or headless installs, store the token with [`netclaw secrets`](/cli/secrets/) and edit the config file directly: +For scripted or headless installs, store the token with [`netclaw secrets`](/cli/secrets/): ```bash netclaw secrets set Discord.BotToken your-bot-token @@ -108,19 +119,32 @@ Then enable Discord in `~/.netclaw/config/netclaw.json`: } ``` -Environment variables work too — useful for Docker and CI: +Environment variables work too, with list entries indexed from zero: ```bash export NETCLAW_Discord__BotToken="your-bot-token" export NETCLAW_Discord__Enabled="true" +export NETCLAW_Discord__AllowedChannelIds__0="123456789012345678" ``` +An edit to `netclaw.json` restarts the daemon by itself. A vault or environment change needs a restart you run: + +| Where it runs | Restart | +|---------------|---------| +| Native | `netclaw daemon stop`, then `netclaw daemon start` | +| systemd user unit | `systemctl --user restart netclaw` | +| Docker | `docker restart ` | + +:::caution +Keep each list in one place. Environment entries and `netclaw.json` entries merge by position, so a blank `NETCLAW_Discord__AllowedChannelIds__0` overwrites the first channel in the file. `netclaw config` → Channels writes the lists into `netclaw.json`. +::: + ### How channel IDs are stored -Enter channel names or IDs (comma-separated) in `netclaw config` → Channels. Netclaw resolves each entry against the Discord API to its canonical channel ID before saving. The stored `AllowedChannelIds` field holds IDs, not display names; display names are shown dynamically in the config UI. Names that cannot be resolved remain in `AllowedChannelIds` verbatim and are flagged with a warning — they are inert until the bot can see the channel, at which point a background refresh rewrites them to canonical IDs. Entries added one-at-a-time via the add-channel flow are rejected at entry time if they do not resolve. A genuine auth or network failure blocks the entire save. +Enter channel names or IDs, comma-separated, in `netclaw config` → Channels. netclaw resolves each against the Discord API and saves the canonical ID. `AllowedChannelIds` holds IDs, never display names. A name that can't be resolved stays verbatim with a warning and matches nothing until the bot can see the channel, then a background refresh rewrites it to the ID. A channel added one at a time is rejected on the spot if it doesn't resolve, and a real auth or network failure blocks the whole save. :::caution -**Breaking change (pre-0.24.0 → 0.24.0+):** If you previously placed display names in `AllowedChannelIds`, those values no longer match incoming channel IDs and messages will be silently dropped. Re-enter the channels via `netclaw config` → Channels so they resolve to canonical IDs. `DefaultChannelId` is already a canonical channel ID and is not affected. +Since 0.24.0, channel display names you put in `AllowedChannelIds` by hand no longer match, and messages from those channels are silently dropped. Re-enter them through `netclaw config` → Channels. `DefaultChannelId` is already an ID. ::: ### All config fields @@ -128,16 +152,17 @@ Enter channel names or IDs (comma-separated) in `netclaw config` → Channels. N | Field | Type | Default | Description | |-------|------|---------|-------------| | `Enabled` | bool | `false` | Turn on Discord | -| `BotToken` | string | — | Bot token from the Developer Portal. Store with `netclaw secrets set`. | -| `DefaultChannelId` | string | — | Channel ID for the default channel | -| `MentionOnly` | bool | `true` | Only respond when @-mentioned | +| `BotToken` | string | - | Bot token from the Developer Portal. Store with `netclaw secrets set`. | +| `DefaultChannelId` | string | - | Channel ID that's allowed without being listed in `AllowedChannelIds` | +| `MentionOnly` | bool | `true` | In channels, answer only when @-mentioned | | `AllowDirectMessages` | bool | `false` | Accept DMs | -| `MentionRequiredInDm` | bool | `false` | Require @-mention even in DMs | -| `AllowedChannelIds` | string[] | `[]` | Channel allow-list. Empty + no default = all channel messages denied | -| `AllowedUserIds` | string[] | `[]` | User allow-list. Empty = everyone in allowed channels is accepted | -| `ChannelAudiences` | object | `{}` | Per-channel [audience](/security/security-model/) overrides. Keys are channel IDs or `"dm"`. Values: `"personal"`, `"team"`, `"public"`. | +| `MentionRequiredInDm` | bool | `false` | Require an @-mention even in DMs | +| `MentionRequiredInThreadByChannel` | object | `{}` | Channel ID to bool. `true` makes a thread reply in that channel need an @-mention. See [Mention behavior](#mention-behavior). | +| `AllowedChannelIds` | string[] | `[]` | Channel allow-list. Empty with no default denies every channel message. | +| `AllowedUserIds` | string[] | `[]` | User allow-list. Empty accepts everyone in allowed channels. | +| `ChannelAudiences` | object | `{}` | Per-channel [audience](/security/security-model/) overrides. Keys are channel IDs or `"dm"`. Values are `"personal"`, `"team"` or `"public"`. | -The token lives in `~/.netclaw/config/secrets.json` (encrypted at rest). +The token lives in `~/.netclaw/config/secrets.json`, encrypted at rest. ## Access control @@ -145,11 +170,7 @@ Discord ACL is **default-deny**. Three settings decide who talks to the bot and ### Channels -The bot only responds in channels that pass the allow-list: - -- `DefaultChannelId` allows one channel -- `AllowedChannelIds` allows multiple -- **If both are empty, every channel message is denied.** This is the number one setup mistake. +The bot answers only in channels that pass the allow-list. `DefaultChannelId` allows one, `AllowedChannelIds` allows more, and if both are empty every channel message is denied. That's the number one setup mistake. ```json { @@ -160,40 +181,25 @@ The bot only responds in channels that pass the allow-list: } ``` -Finding channel IDs: right-click a channel in Discord with Developer Mode on, click "Copy Channel ID." +To find a channel ID with Developer Mode on, right-click the channel and choose **Copy Channel ID**. ### Users -`AllowedUserIds` restricts who gets responses: - -- Empty (default) — everyone in allowed channels is accepted -- Non-empty — only listed user IDs get responses, everyone else is silently dropped - -Finding user IDs: right-click a user in Discord with Developer Mode on, click "Copy User ID." +`AllowedUserIds` restricts who gets answers. Empty means everyone in an allowed channel. Non-empty means only the listed users, and everyone else is dropped silently. To copy a user ID, right-click the user and choose **Copy User ID**. -Users in `AllowedUserIds` are treated as `TrustedInternal` by the [security model](/security/security-model/). Everyone else is `UntrustedExternal`. +Listed users are `TrustedInternal` in the [security model](/security/security-model/), and everyone else is `UntrustedExternal`. ### Direct messages -DMs are off by default: - -```json -{ - "Discord": { - "AllowDirectMessages": true - } -} -``` - -With DMs on, users can just type normally — `MentionRequiredInDm` defaults to `false`, so no @-mention needed. Each user gets a single long-running DM session (unlike channels, where each root message starts a new session). +DMs are off by default. Turn them on with `"AllowDirectMessages": true`. Users then write without a mention, because `MentionRequiredInDm` defaults to `false`. Each user gets one long-running DM session, unlike channels, where each root message starts a new session. :::caution -With `AllowDirectMessages: true` and `AllowedUserIds` empty, any server member can DM the bot. Lock down `AllowedUserIds` if that's not what you want. +With DMs on and `AllowedUserIds` empty, any server member can DM the bot. Set `AllowedUserIds` unless that's what you want. ::: ### Audience overrides -Override the default audience per-channel with `ChannelAudiences`: +Audience resolves per message: `ChannelAudiences[channelId]` first, then `ChannelAudiences["dm"]`, then a fallback. The fallback is `Team` for a channel listed in `AllowedChannelIds` or a user listed in `AllowedUserIds`, and `Public` for everything else. A channel allowed only through `DefaultChannelId` is `Public` unless you also list it in `AllowedChannelIds` or set `ChannelAudiences` for it. ```json { @@ -206,105 +212,108 @@ Override the default audience per-channel with `ChannelAudiences`: } ``` -The `"dm"` key is reserved — it matches every direct message rather than a channel ID. A channel-ID entry takes precedence over it. An unrecognized audience value is rejected outright: the message is denied rather than falling back to a default. - -[Security Model](/security/security-model/) has the full breakdown on how audiences map to tools and permissions. +A channel-ID entry beats `"dm"`, and an unrecognized value denies the message instead of falling back. [Security model](/security/security-model/) maps audiences to tools. ## Behavior in Discord ### Threads and sessions -When someone messages the bot in a regular channel, netclaw creates a public thread on its first reply and continues the conversation there. The thread starts as "Netclaw" and netclaw renames it once the LLM generates a session title. - -Thread replies don't need a @-mention — if there's an active session in the thread, the bot responds to everything. +When someone messages the bot in a channel, netclaw creates a public thread on its first reply and continues there. The thread starts as "Netclaw" and is renamed once the model produces a session title. -Idle conversations are freed from memory after 2 hours (individual sessions after 1 hour, unless an approval request is pending). On daemon restart, up to 200 messages of thread history are backfilled so in-progress conversations resume. +A thread is freed from memory when its session goes idle, after `Session.IdleTimeout` (one hour by default), and a later reply wakes it again. After a daemon restart, netclaw backfills up to 200 messages of thread history so conversations resume. ### Mention behavior -`MentionOnly: true` (the default) means the bot ignores messages that don't @-mention it. Two exceptions: +`MentionOnly: true` makes the bot ignore channel messages that don't @-mention it. netclaw strips the mention before the text reaches the model. These rules sit on top: -- **Thread replies** — if a thread already has an active session, the bot responds without needing a mention -- **Daemon restart recovery** — if the daemon restarts and a user continues a thread, the session is re-created from the thread's message history +- **Thread replies:** once a thread has an active session, replies in it don't need a mention, unless `MentionRequiredInThreadByChannel` is `true` for that channel. +- **Restart recovery:** if the daemon restarts and someone continues a thread, netclaw recreates the session from the thread's history. -Netclaw strips the @-mention before passing text to the LLM. +`MentionRequiredInThreadByChannel` maps channel IDs to `true` or `false`: + +```json +{ + "Discord": { + "MentionRequiredInThreadByChannel": { + "123456789012345678": true + } + } +} +``` + +With the rule on, an un-mentioned thread reply is ignored (`ThreadMentionRequired` in the log). The next message that mentions the bot makes netclaw re-read everything posted since its last reply, so it answers with full context. A channel with no entry behaves as `false`. The rule doesn't apply to DMs. + +In `netclaw config` → Channels, **Space** toggles **Require @mention** on the selected channel. A channel you add there starts with the rule on if its audience is `team` or `public`, and off if it's `personal`. Only `true` entries are saved. ### Message formatting -Discord natively renders markdown — bold, italic, code blocks, headers, lists, links. Responses longer than 2,000 characters are split at newline boundaries. +Discord renders markdown natively. netclaw splits replies longer than 2,000 characters at newline boundaries. ### Tool approval -When a tool call needs approval, netclaw posts an interactive button prompt in the thread. The prompt shows the tool name, action, and pattern(s). The full set is five buttons: **Once**, **This chat**, **Always here**, **Always anywhere**, and **Deny** — though **Always here** is omitted when the working directory is a session scratch path or too shallow for a folder-scoped grant. **Always anywhere** grants the command everywhere, so use it sparingly. netclaw shows fewer buttons when some don't apply — a command it can't cleanly parse (shell control flow, or unbalanced quotes) drops to just **Once** and **Deny**. Only the user who triggered the request can approve. - -If posting the button prompt fails, netclaw falls back to a text prompt where you reply with the letter shown next to each option. +When a tool call needs approval, netclaw posts a prompt with buttons in the thread. It shows the tool, the action and the pattern. The choices are **Once**, **This chat**, **Always here**, **Always anywhere**, **Deny**, and **This repository** when it applies. A command netclaw can't parse cleanly gets only **Once** and **Deny**. [How tool approvals work](/security/approvals/) explains what each one grants. Only the user whose request triggered the prompt can answer it. -After a decision, netclaw updates the original message in-place with a checkmark or denied icon. +If posting the buttons fails for a prompt from the main agent, netclaw falls back to a text prompt, and you reply with the letter next to an option. A prompt from a sub-agent has buttons only and no letter fallback. If its buttons can't be posted, netclaw refuses the request and the call doesn't run. After a decision, netclaw edits the original message with a check mark or a denied icon. A prompt that nothing can answer any more is edited to remove the buttons and say why, for example `This approval request was withdrawn because Netclaw restarted before the task finished. Nothing ran.` ### Reminders -Discord reminders deliver to guild text channels only — user DMs are not supported as reminder targets. Accepted formats: +[Reminders](/configuration/reminders/) deliver to a guild text channel or to a user's DM. Accepted targets: -- `<#123456789012345678>` — Discord channel mention -- `channel:123456789012345678` — explicit channel prefix +| Target | Delivers to | +|--------|-------------| +| `<#123456789012345678>` | A channel, by mention | +| `channel:123456789012345678` | A channel, by ID | +| `<@123456789012345678>` or `<@!123456789012345678>` | A user's DM, by mention | +| `dm:123456789012345678` or `@123456789012345678` | A user's DM, by ID | + +A bare ID is rejected, because channel IDs and user IDs have the same shape. DM delivery still goes through ACL, so it needs `AllowDirectMessages: true` and, if you use `AllowedUserIds`, the user on that list. `current_session` reminders post back into the thread that set them. That reminder ends when the session admits the fire, and netclaw doesn't retry a failed post. If the reply post fails for good, the session logs an Error and raises a `reminder.delivery.failed` [alert](/observability/operational-alerts/). ### Proactive messaging -The LLM can start a conversation with the generic `send_channel_message` tool — pass `channel_key: discord` and a `destination` (a guild text channel or a DM) resolved via `lookup_channel_user` or `lookup_channel_destination`. ACL rules still apply. +The model can start a conversation with `send_channel_message`, passing `channel_key: discord` and a destination (a guild text channel or a DM) from `lookup_channel_user` or `lookup_channel_destination`. ACL still applies. ### Ignored messages -The bot drops: empty messages (no text, no attachments), other bots' messages, its own messages, DMs when `AllowDirectMessages` is off, and un-mentioned channel messages when `MentionOnly` is on and there's no active thread. - -Messages over 4,000 characters are truncated. +The bot drops empty messages (no text, no attachments), its own and other bots' messages, DMs when `AllowDirectMessages` is off, and un-mentioned channel messages when `MentionOnly` is on and there's no active thread. ## Verify it works -Restart the daemon and check status: - ```bash -netclaw daemon stop && netclaw daemon start netclaw status ``` -Discord should show `connected`. If it shows `disabled` or `disconnected`, check the [troubleshooting guide](/channels/troubleshooting/). +Discord should show `healthy`. If it shows `disabled` or `disconnected`, see [Channel troubleshooting](/channels/troubleshooting/). -Then @-mention the bot in an allowed channel. If it creates a thread and responds, you're set. +Then @-mention the bot in an allowed channel. If it creates a thread and answers, you're set. :::note -`netclaw doctor` doesn't validate Discord tokens yet. For now, use `netclaw status` and check the daemon logs. +`netclaw doctor` doesn't check Discord yet. Use `netclaw status` and the daemon log. ::: ## Troubleshooting -Common problems and fixes are in [Channel Troubleshooting](/channels/troubleshooting/). The hits: +[Channel troubleshooting](/channels/troubleshooting/) is organized by symptom. The usual ones: -- **Connected but silent** — `AllowedChannelIds` is empty and no `DefaultChannelId` is set, so all traffic gets denied -- **Invalid bot token** — HTTP 401 on startup means the token is wrong or was regenerated in the Developer Portal -- **Gateway disconnected** — the daemon couldn't establish a connection. Usually a network issue or [Discord outage](https://discordstatus.com/). -- **Message Content privileged intent not enabled** — turn it on under Privileged Gateway Intents in the [Developer Portal](https://discord.com/developers/applications) +| Symptom | Likely cause | +|---------|--------------| +| Connected but silent | `AllowedChannelIds` is empty and no `DefaultChannelId` is set. | +| Messages arrive but the text is empty | Message Content intent is off in the Developer Portal. | +| HTTP 401 at startup | The token is wrong or was reset. | +| Startup waits 30 seconds, then `disconnected` | The gateway never sent READY, often because the bot isn't in any server. | +| Stopped answering in a thread | `MentionRequiredInThreadByChannel` is on for the channel. Mention the bot. | -## Next steps +## Related pages -- [Configure audiences and approval gates](/security/security-model/) to control what tools are available in each channel -- [Set up systemd](/deployment/systemd/) so the daemon stays running after reboots -- [Add Slack](/channels/slack/) or [Mattermost](/channels/mattermost/) if your team uses more than one +- [Slack](/channels/slack/) and [Mattermost](/channels/mattermost/) - the other chat channels +- [Channel troubleshooting](/channels/troubleshooting/) - fixes by symptom +- [How tool approvals work](/security/approvals/) - the approval choices +- [Security model](/security/security-model/) - audiences and approval gates +- [`netclaw status`](/cli/status/) - connector health -## Related pages +## Resources -- [`netclaw config`](/cli/config/) — Channels -- [`netclaw secrets`](/cli/secrets/) — token management -- [`netclaw status`](/cli/status/) — primary diagnostic tool for Discord -- [Security Model](/security/security-model/) — audiences and approval gates -- [Channel Troubleshooting](/channels/troubleshooting/) — error codes and debug logging -- [Slack](/channels/slack/) — sibling channel integration -- [Mattermost](/channels/mattermost/) — sibling channel integration - -## External resources - -- [Discord Developer Portal](https://discord.com/developers/applications) — create and manage your bot -- [Discord Developer Docs: Getting Started](https://discord.com/developers/docs/getting-started) — bot setup walkthrough -- [Discord: Privileged Intents](https://discord.com/developers/docs/events/gateway#privileged-intents) — Message Content intent setup -- [Discord: Bot Permissions](https://discord.com/developers/docs/topics/permissions) — permission reference -- [Discord: Enable Developer Mode](https://support.discord.com/hc/en-us/articles/206346498) — how to copy channel and user IDs -- [Discord Status](https://discordstatus.com/) — check for outages +- [Discord Developer Portal](https://discord.com/developers/applications) - create and manage your bot +- [Discord: Gateway and privileged intents](https://docs.discord.com/developers/topics/gateway) - what Message Content does +- [Discord: permissions](https://discord.com/developers/docs/topics/permissions) - permission reference +- [Discord: enable Developer Mode](https://support.discord.com/hc/en-us/articles/206346498) - copy channel and user IDs +- [Discord Status](https://discordstatus.com/) - check for outages diff --git a/src/content/docs/channels/mattermost.md b/src/content/docs/channels/mattermost.md index bc29667..904c290 100644 --- a/src/content/docs/channels/mattermost.md +++ b/src/content/docs/channels/mattermost.md @@ -1,81 +1,97 @@ --- title: "Mattermost" -description: "Connect Netclaw to your self-hosted Mattermost server." +description: "Connect netclaw to a self-hosted Mattermost server with a bot account, an allow-list and text or button approvals." --- -Netclaw connects to Mattermost over a WebSocket for events and the REST API for replies. You run a bot account on your Mattermost server, give netclaw the bot's access token, and it shows up as a bot user — same default-deny ACL model as [Slack](/channels/slack/) and [Discord](/channels/discord/), against infrastructure you host yourself. +netclaw connects to Mattermost over a WebSocket for events and the REST API for replies. You create a bot account on your server, give netclaw the bot's access token, and it answers in threads under the same default-deny ACL as [Slack](/channels/slack/) and [Discord](/channels/discord/). If you want the whole stack in containers, [Run netclaw with Mattermost on Docker Compose](/guides/mattermost-docker-compose/) sets up the server, the bot and netclaw with one sample. ## Prerequisites -- Netclaw installed and initialized ([`netclaw init`](/cli/init/)) -- A self-hosted Mattermost server reachable from the daemon -- System Console access on that server (to enable bot accounts and create the bot) +- netclaw installed and initialized ([`netclaw init`](/cli/init/)) +- A Mattermost server the daemon can reach, with a user who has System Console access +- The server's base URL as netclaw will reach it, such as `https://mm.example.com` or `http://mattermost:8065` on a Compose network ## Create a bot account -Mattermost bots authenticate with a **personal access token**, not OAuth. Two server settings have to be on before you can create one. +Mattermost bots authenticate with a **personal access token**, not OAuth. Two server settings are off by default and both must be on before you can create one. ### 1. Enable bot accounts and access tokens -In the **System Console**, under **Integrations > Bot Accounts**, set **Enable Bot Account Creation** to `true`. Then, under **Integrations > Integration Management**, turn on **Enable Personal Access Tokens**. +In the **System Console**, open **Integrations > Bot Accounts** and set **Enable bot account creation** to `true`. Then open **Integrations > Integration Management** and turn on **Enable personal access tokens**. Both map to `ServiceSettings` keys (`EnableBotAccountCreation` and `EnableUserAccessTokens`) if you'd rather set them in `config.json` or with `MM_SERVICESETTINGS_*` environment variables. -![System Console Bot Accounts settings with Enable Bot Account Creation set to True](/screenshots/output/mattermost-bot-accounts.png) +![System Console Bot Accounts settings with Enable Bot Account Creation set to True and greyed out under a notice that an environment variable set it](/screenshots/output/mattermost-bot-accounts.png) -Both are off by default on a fresh Mattermost install — skipping either is the most common setup snag. +This server was configured through an environment variable, so the setting is locked and the console says so. On your server it's a normal radio button, and skipping this toggle or the token one is the most common setup snag. ### 2. Create the bot and copy its token -Under **Integrations > Bot Accounts**, click **Add Bot Account** and give it a username (e.g. `netclaw`) and display name. Then click **Create New Token**: Mattermost reveals the access token **once**, in a banner — copy it right then. That token (not the Token ID shown in the list) is the `BotToken` netclaw needs. +Under **Integrations > Bot Accounts**, click **Add Bot Account**, give it a username such as `netclaw` and a display name, then click **Create New Token**. Mattermost shows the access token **once**, in a banner. Copy it then. That token, not the Token ID in the list, is the `BotToken` netclaw needs. -![Mattermost Bot Accounts page listing the netclaw bot with its Create New Token control](/screenshots/output/mattermost-create-bot.png) +![Mattermost Integrations > Bot Accounts list with the netclaw bot, its Create New Token, Edit and Disable links, and a token row whose Token ID is masked](/screenshots/output/mattermost-create-bot.png) + +The list is what you see after the bot exists. The **Create New Token** link next to the bot issues a replacement later, and the Token ID in the row, masked here, is not the token netclaw needs. + +The [Compose sample](/guides/mattermost-docker-compose/) provisions this for you, and the bot row ends up looking like this. + +![The netclaw bot row in Mattermost Bot Accounts, managed by admin, with one token described as netclaw and its Token ID masked](/screenshots/output/mattermost-compose-bot-account.png) + +The token description is `netclaw`, and the sample keeps the token itself out of every file in the repository. ### 3. Add the bot to your team and channels -A bot only sees channels it belongs to. Add it to the target team, then invite it to each channel where it should respond (`/invite @netclaw`, or through the channel member menu). +A bot only sees channels it belongs to. Add it to the team, then invite it to each channel where it should answer with `/invite @netclaw` or the channel member menu. ## Configure netclaw :::tip -Run [`netclaw config`](/cli/config/) → Channels. Enable Mattermost, enter your server URL and bot token — netclaw resolves channel names to canonical IDs and saves. You no longer need to look up opaque IDs by hand. +Run [`netclaw config`](/cli/config/) → Channels, enable Mattermost, and enter the server URL and bot token. netclaw resolves channel names to IDs and saves. ::: ![The Channels area in netclaw config](/screenshots/output/config-channels-menu.png) -The Channels area in `netclaw config` — enable Mattermost and enter its server URL and token here (the adapter list also shows Slack and Discord). +The Channels area lists Slack, Discord and Mattermost with a checkbox each. Toggle Mattermost on and press `Enter` to configure it. ### Manual setup -For scripted deployments or Docker installs where a TUI isn't available, store the token with [`netclaw secrets`](/cli/secrets/): +For Docker or scripted installs, store the token with [`netclaw secrets`](/cli/secrets/) and describe the rest in config: ```bash netclaw secrets set Mattermost.BotToken your-bot-token ``` -Then enable Mattermost and point it at your server in `~/.netclaw/config/netclaw.json`: +The token lands in your shell history and the process list this way. In scripts, feed it over stdin instead, as the [Compose guide](/guides/mattermost-docker-compose/) does: `netclaw secrets set Mattermost.BotToken "$(cat)" < .bot-token`. ```json { "Mattermost": { "Enabled": true, "ServerUrl": "https://mm.example.com", - "DefaultChannelId": "abcdefghijklmnopqrstuvwxyz" + "AllowedChannelIds": ["abcdefghijklmnopqrstuvwxyz"], + "AllowedUserIds": ["zyxwvutsrqponmlkjihgfedcba"] } } ``` -Environment variables work too: +Every key also works as an environment variable with the `NETCLAW_` prefix and `__` between segments, with list entries indexed from zero: ```bash -export NETCLAW_Mattermost__BotToken="your-bot-token" -export NETCLAW_Mattermost__Enabled="true" +NETCLAW_Mattermost__Enabled=true +NETCLAW_Mattermost__ServerUrl=http://mattermost:8065 +NETCLAW_Mattermost__AllowedChannelIds__0=abcdefghijklmnopqrstuvwxyz ``` +Restart behavior differs by source. Editing `netclaw.json` makes the daemon restart itself. A change to the vault or to an environment variable needs a restart you do yourself: `systemctl --user restart netclaw`, or `docker restart ` in Docker. + +:::caution +Keep each list in one place. Environment entries and `netclaw.json` entries merge by position, so a blank `NETCLAW_Mattermost__AllowedChannelIds__0` overwrites the first channel in the file and denies it. `netclaw config` → Channels also writes `AllowedChannelIds` and `AllowedUserIds` into `netclaw.json`. +::: + ### How channel IDs are stored -Add channels one at a time in `netclaw config` → Channels. Netclaw resolves each entry against the Mattermost REST API to its canonical channel ID before adding it to the list. The stored `AllowedChannelIds` field holds IDs, not display names; display names are shown dynamically in the config UI. Entries that cannot be resolved are flagged with a warning but saved verbatim in `AllowedChannelIds` — an unresolved name is inert at runtime because the ACL matches against canonical IDs only, so the name grants access to no channel until it resolves. +Add channels one at a time in `netclaw config` → Channels. netclaw resolves each entry against the Mattermost REST API to a canonical channel ID before it adds it. `AllowedChannelIds` holds IDs, never display names. An entry that can't be resolved is flagged and kept verbatim, and it matches no channel until it resolves. When you save, the first channel in the list also becomes `DefaultChannelId`. :::caution -**Breaking change (pre-0.24.0 → 0.24.0+):** If you previously placed display names in `AllowedChannelIds` by hand in `netclaw.json`, those names no longer match any channel ID, so those channels are silently denied. Re-enter them via `netclaw config` → Channels so they resolve to canonical IDs. (`DefaultChannelId` already holds a canonical ID, so it's unaffected.) +Since 0.24.0, display names you typed into `AllowedChannelIds` by hand no longer match, and those channels are silently denied. Re-enter them through `netclaw config` → Channels. `DefaultChannelId` already holds an ID. ::: ### All config fields @@ -83,35 +99,31 @@ Add channels one at a time in `netclaw config` → Channels. Netclaw resolves ea | Field | Type | Default | Description | |-------|------|---------|-------------| | `Enabled` | bool | `false` | Turn on Mattermost | -| `ServerUrl` | string | — | Base URL of your Mattermost server. Required when enabled. | -| `BotToken` | string | — | Bot personal access token. Store with `netclaw secrets set`. | -| `CallbackUrl` | string | — | URL Mattermost can reach for interactive button callbacks. Set it to get approval buttons; leave it unset for text-reply approvals (see [Tool approval](#tool-approval)). | -| `DefaultChannelId` | string | — | A 26-character channel ID. Allows one channel without listing it in `AllowedChannelIds`. | -| `MentionOnly` | bool | `true` | Only respond when @-mentioned in channels | +| `ServerUrl` | string | - | Base URL of your server. Required when enabled. | +| `BotToken` | string | - | Bot access token. Vault only (`netclaw secrets set Mattermost.BotToken`). It isn't a `netclaw.json` key, and the schema rejects it there. | +| `CallbackUrl` | string | - | URL the Mattermost server can reach for approval buttons. Required for sub-agent approvals. Unset means text-reply approvals for the main agent only (see [Tool approval](#tool-approval)). | +| `DefaultChannelId` | string | - | A 26-character channel ID that's allowed without being listed in `AllowedChannelIds` | +| `MentionOnly` | bool | `true` | In channels, answer only when @-mentioned | | `AllowDirectMessages` | bool | `false` | Accept DMs | -| `MentionRequiredInDm` | bool | `false` | Require @-mention even in DMs | -| `AllowedChannelIds` | string[] | `[]` | Channel allow-list. Empty + no default = all channel messages denied | -| `AllowedUserIds` | string[] | `[]` | User allow-list. Empty = everyone in allowed channels is accepted | -| `ChannelAudiences` | object | `{}` | Per-channel [audience](/security/security-model/) overrides. Keys are channel IDs or `"dm"`. Values: `"personal"`, `"team"`, `"public"`. | +| `MentionRequiredInDm` | bool | `false` | Require an @-mention even in DMs | +| `MentionRequiredInThreadByChannel` | object | `{}` | Channel ID to bool. `true` makes a thread reply in that channel need an @-mention. See [Mention behavior](#mention-behavior). | +| `AllowedChannelIds` | string[] | `[]` | Channel allow-list. Empty with no default denies every channel message. | +| `AllowedUserIds` | string[] | `[]` | User allow-list. Empty accepts everyone in allowed channels. | +| `ChannelAudiences` | object | `{}` | Per-channel [audience](/security/security-model/) overrides. Keys are channel IDs or `"dm"`. Values are `"personal"`, `"team"` or `"public"`. | -The token lives in `~/.netclaw/config/secrets.json` (encrypted at rest). Mattermost channel and user IDs are 26-character alphanumeric strings — the System Console lists them, or pull them from the [REST API](https://api.mattermost.com/). +The token lives in `~/.netclaw/config/secrets.json`, encrypted at rest. Channel and user IDs are 26-character strings. You can read them with `mmctl --local user list` and `mmctl --local channel search --team ` inside the server container, or from the [REST API](https://api.mattermost.com/). ## Access control -Mattermost ACL is **default-deny**, evaluated the same way as Slack and Discord. +Mattermost ACL is **default-deny** and works like Slack and Discord. ### Channels -The bot only responds in channels that pass the allow-list: - -- `DefaultChannelId` allows one channel -- `AllowedChannelIds` allows multiple -- **If both are empty, every channel message is denied.** This is the number one setup mistake. +The bot answers only in channels that pass the allow-list. `DefaultChannelId` allows one, `AllowedChannelIds` allows more, and if both are empty every channel message is denied. That's the number one setup mistake. ```json { "Mattermost": { - "DefaultChannelId": "abcdefghijklmnopqrstuvwxyz", "AllowedChannelIds": ["abcdefghijklmnopqrstuvwxyz", "zyxwvutsrqponmlkjihgfedcba"] } } @@ -119,137 +131,182 @@ The bot only responds in channels that pass the allow-list: ### Users -`AllowedUserIds` restricts who gets responses: - -- Empty (default) — everyone in allowed channels is accepted -- Non-empty — only listed user IDs get responses, everyone else is silently dropped - -Users in `AllowedUserIds` are treated as `TrustedInternal` by the [security model](/security/security-model/). Everyone else is `UntrustedExternal`. +`AllowedUserIds` restricts who gets answers. Empty means everyone in an allowed channel. Non-empty means only the listed users, and everyone else is dropped silently, in DMs too. Listed users are `TrustedInternal` in the [security model](/security/security-model/), and everyone else is `UntrustedExternal`. :::caution -Always set `AllowedUserIds` on a production server. With it empty, any user the bot can see can click an approval button in a shared channel and approve a tool call on someone else's behalf. The action-token check still blocks *forged* clicks — but a legitimate click from the wrong person is a real one. +Always set `AllowedUserIds` on a shared server. With it empty, any user in an allowed channel can talk to the bot and start tool calls under that channel's audience. Only the user who triggered a prompt can approve it, but an empty list lets anyone be that user. ::: ### Direct messages -DMs are off by default: +DMs are off by default. Turn them on with `"AllowDirectMessages": true`. Users then write without a mention, because `MentionRequiredInDm` defaults to `false`. With `AllowedUserIds` empty, anyone on the server can DM the bot. + +### Audience overrides + +Audience resolves per message: `ChannelAudiences[channelId]` first, then `ChannelAudiences["dm"]` for direct messages, then a fallback. The fallback is `Team` for a channel listed in `AllowedChannelIds` or a user listed in `AllowedUserIds`, and `Public` for everything else. A DM from someone who isn't on the list is `Public`, and so is a channel allowed only through `DefaultChannelId`, unless you also list it in `AllowedChannelIds` or set `ChannelAudiences` for it. ```json { "Mattermost": { - "AllowDirectMessages": true + "ChannelAudiences": { + "abcdefghijklmnopqrstuvwxyz": "personal", + "dm": "personal" + } } } ``` -With DMs on, users type normally — `MentionRequiredInDm` defaults to `false`, so no @-mention needed. +A channel-ID entry beats `"dm"`, and an unrecognized value denies the message instead of falling back. [Security model](/security/security-model/) maps audiences to tools. For a channel you own, `personal` is what lets the agent request shell commands, still gated by approval. -### Audience overrides +## Behavior in Mattermost -Audience resolves per message: `ChannelAudiences[channelId]` wins first, then `ChannelAudiences["dm"]` for direct messages, then a fallback that lands on `Team` for allow-listed channels and users and `Public` for everything else. Override it explicitly: +### Threads and sessions + +Each thread is its own session, keyed by `{channelId}/{rootPostId}`, and the bot replies in-thread. The thread stays attached to its session until the session goes idle and is freed from memory, after `Session.IdleTimeout` (one hour by default). A later reply in the thread wakes it again. After a daemon restart, thread history is backfilled so conversations continue. + +### Mention behavior + +`MentionOnly: true` makes the bot ignore channel messages that don't @-mention it. netclaw strips the mention before the text reaches the model. Once a thread has an active session, replies in it don't need a mention, unless you turn that rule on for the channel: ```json { "Mattermost": { - "ChannelAudiences": { - "abcdefghijklmnopqrstuvwxyz": "team", - "dm": "personal" + "MentionRequiredInThreadByChannel": { + "abcdefghijklmnopqrstuvwxyz": true } } } ``` -The `"dm"` key is reserved — it matches every direct message rather than a channel ID, and a channel-ID entry takes precedence over it. An unrecognized audience value is rejected outright: the message is denied rather than falling back to a default. +With the rule on, an un-mentioned thread reply is ignored (`ThreadMentionRequired` in the log). The next message that does mention the bot makes netclaw re-read everything posted since its last reply, so it answers with full context. A channel with no entry behaves as `false`. The rule doesn't apply to DMs. -[Security Model](/security/security-model/) has the full breakdown on how audiences map to tools and permissions. +You can also set it in `netclaw config` → Channels. On the channel list, **Space** toggles **Require @mention** for the selected channel. A channel you add there starts with the rule on if its audience is `team` or `public`, and off if it's `personal`. Only `true` entries are saved. -## Behavior in Mattermost +### Typing indicator -### Threads and sessions +While a session works on your message, netclaw sends Mattermost's native typing pulse to the thread every three seconds. It stops when the session goes idle, and it gives up after ten minutes so a stuck session can't leave the thread typing forever. A failed pulse logs a warning and doesn't affect the reply. -Each thread is its own isolated session, keyed by `{channelId}/{rootPostId}`, and the bot replies in-thread. On daemon restart, thread history is backfilled so in-progress conversations pick up where they left off. +### Tool approval -### Mention behavior +When a tool call needs approval, how netclaw asks depends on `CallbackUrl`: -`MentionOnly: true` (the default) means the bot ignores channel messages that don't @-mention it. Thread replies are the exception — once a thread has an active session, the bot responds to everything in it without a mention. Netclaw strips the @-mention before passing text to the LLM. +| `CallbackUrl` | Prompt | How you answer | +|---------------|--------|----------------| +| Unset | A text prompt listing options with letters | Reply with a letter in the thread. No inbound HTTP endpoint is opened. Approval requests from a sub-agent are refused. | +| Set | Buttons | Click. Each button carries a one-time token that expires after 12 hours. | -### Tool approval +The choices are the same on every channel: **Once**, **This chat**, **Always here**, **Always anywhere**, **Deny**, and **This repository** when it applies. A command netclaw can't parse cleanly gets only **Once** and **Deny**. [How tool approvals work](/security/approvals/) explains what each choice grants. Only the user whose request triggered the prompt can answer it. Anyone else gets a warning. -When a tool call needs approval, how netclaw asks depends on `CallbackUrl`: +A prompt from a sub-agent always has buttons only, on every channel, with no lettered fallback. A typed letter doesn't answer it. If two or more prompts are open for you and one is from a sub-agent, a letter answers none and the bot replies `N approval prompts are open. Use the buttons on the prompt you mean.` -- **`CallbackUrl` set** — netclaw posts interactive buttons. The full prompt offers five choices, in order: **Once**, **This chat**, **Always here**, **Always anywhere**, **Deny**. Clicks come back over an inbound HTTP POST to `/api/mattermost/actions`, which the daemon only exposes when a callback URL is configured. Each button carries a one-time, post-bound token that expires after 12 hours. -- **`CallbackUrl` unset** — approvals fall back to text replies: reply with the letter shown next to each option. No inbound HTTP surface is opened. +The lettered prompt, from a real run: -netclaw shows fewer than five when some don't apply — a command it can't cleanly parse (shell control flow, or unbalanced quotes) collapses to just **Once** and **Deny**, and the reply letters track whatever is shown. **Always anywhere** is the broadest grant: it approves the command's verb for every session everywhere and persists to `tool-approvals.json`. Reach for **Always here** or narrower unless you really mean it. +```text +:lock: Tool approval required +Tool: shell_execute +Action: touch /tmp/approval-test +Pattern: touch -The token store is in-memory: buttons posted before a daemon restart return "no longer valid" when clicked afterward. The pending request survives on the session, so ask the agent to re-issue the prompt. +Reply with: +A) Once +B) This chat +C) Always here +D) Always anywhere +E) Deny +``` -:::caution -Reach `/api/mattermost/actions` over TLS. The callback trusts the user ID in the payload as the clicker's identity, so a network attacker who can rewrite headers in plaintext could impersonate another user. -::: +If posting an approval prompt fails, netclaw reports the failure to the model as an unavailable prompt, not as your denial, and the call doesn't run. A prompt that nothing can answer any more is redrawn without buttons and with the reason, such as `This approval request was withdrawn because the task that asked for it was cancelled. Nothing ran.` A click on a prompt that already has an answer gets `That approval prompt already has an answer. This response changed nothing.` + +#### Buttons need a reachable callback + +The Mattermost **server**, not your browser, sends the click to `CallbackUrl`. The URL has to be reachable from the server's network, and it has to reach the daemon's `/api/mattermost/actions` path. The daemon serves that on its one listener at `Daemon.Host:Daemon.Port`. In the default `local` exposure mode that listener is loopback only, and `local` refuses any other bind. A Mattermost server on another host or in another container can't reach it. + +So you pick one of two setups: + +- Leave `CallbackUrl` unset and use text replies. This works for a single-host or Compose stack that doesn't use sub-agents. Any approval request from a sub-agent is refused, the call doesn't run, and the prompt is withdrawn as `could not show it in this channel`. +- Move the daemon to a non-local [exposure mode](/deployment/exposure-modes/) or front the path with a forwarder, then set `CallbackUrl` to the address the Mattermost server uses, for example `https://netclaw.example.com/api/mattermost/actions`. + +Mattermost also blocks integration requests to private and loopback addresses unless the target is in `ServiceSettings.AllowedUntrustedInternalConnections` ([Mattermost docs](https://docs.mattermost.com/administration-guide/configure/environment-configuration-settings.html), **Environment > Developer**). Add the daemon's host there if your callback is a private address. + +Reach the endpoint over TLS or a trusted network. The callback trusts the user ID in the payload as the clicker's identity. The token store lives in memory and holds at most 10,000 unused tokens, oldest evicted first. Buttons posted before a daemon restart answer "no longer valid", so ask the agent to re-issue the prompt. A sub-agent prompt for a run that hadn't parked when the daemon restarted shows the withdrawal text instead. A parked run keeps its prompt, and a click after the restart resumes it. ### Proactive messaging -The LLM can start a conversation with the generic `send_channel_message` tool — pass `channel_key: mattermost` and a `destination` resolved via `lookup_channel_user` or `lookup_channel_destination`. It posts a new thread root and routes replies in that thread back to a live session. ACL rules still apply — DMing a user also requires `AllowDirectMessages: true`. +The model can start a conversation with `send_channel_message`, passing `channel_key: mattermost` and a destination from `lookup_channel_user` or `lookup_channel_destination`. It posts a new thread root and routes replies in that thread to a live session. ACL still applies, and DMing a user needs `AllowDirectMessages: true`. ### Reminders -[Reminders](/configuration/reminders/) reach Mattermost through `channel` delivery with `delivery_transport: "mattermost"`. (In-thread `current_session` delivery isn't available from a Mattermost session yet — `set_reminder` only accepts it for Slack, Discord, the TUI, and SignalR.) Because Mattermost user IDs and channel IDs are both 26-character strings, channel-delivery targets need a prefix: +[Reminders](/configuration/reminders/) reach Mattermost two ways: + +- `current_session` posts back into the thread that set the reminder +- `channel` delivery with `delivery_transport: "mattermost"` posts to a channel or a user's DM + +User IDs and channel IDs have the same shape, so channel delivery needs a prefix: + +| Target | Delivers to | +|--------|-------------| +| `@` | That user's DM | +| `channel:` | That channel | -- `@` — delivers to that user's DM -- `channel:` — delivers to that channel +A bare ID is rejected with a disambiguation error. -A bare ID with no prefix is rejected with a disambiguation error. Unlike Discord, DM delivery works, because a Mattermost DM is itself an addressable channel. +A `current_session` reminder ends when the session admits the fire, and netclaw doesn't retry a failed post. If the reply post fails for good, the session logs an Error and raises a `reminder.delivery.failed` [alert](/observability/operational-alerts/). ## Verify it works -Restart the daemon, then @-mention the bot in an allowed channel: +Check the connector, then mention the bot in an allowed channel: ```bash -netclaw daemon stop && netclaw daemon start +netclaw status ``` -If it replies in a thread, you're set. +```text +connectors: +- mattermost: healthy (enabled) +``` + +If the bot replies in a thread, you're set. + +![Mattermost Assistant channel where alice mentions @netclaw and the netclaw bot answers with a greeting in the thread panel](/screenshots/output/mattermost-conversation.png) -![Netclaw replying to a testuser @-mention in a Mattermost thread](/screenshots/output/mattermost-conversation.png) +The reply lands in a thread under alice's mention, which is also where follow-ups go. -`netclaw status` reports Mattermost health (healthy, degraded, disconnected, or disabled). When something's off, check its status first, then the daemon logs for detail. A bad token, an unreachable server, or any connection failure is contained there — the Mattermost channel degrades on its own and the daemon plus every other channel keeps running. See [Troubleshooting](#troubleshooting) below for the common messages. +`netclaw status` reports `healthy`, `degraded`, `disconnected` or `disabled`. A channel problem stays in the channel. The daemon and your other channels keep running. ## Troubleshooting -Common problems and fixes are in [Channel Troubleshooting](/channels/troubleshooting/). The hits: +[Channel troubleshooting](/channels/troubleshooting/) is organized by symptom. The usual ones: -- **Connected but silent** — `AllowedChannelIds` is empty and no `DefaultChannelId` is set, so all traffic gets denied; or the bot was never added to the channel. -- **`Mattermost is enabled but Mattermost:ServerUrl is not configured`** — add `Mattermost.ServerUrl` to `netclaw.json`. -- **`Mattermost rejected the bot token (HTTP 401)`** — re-issue the bot token and update `secrets.json`. -- **Can't create a bot or token** — Enable Bot Account Creation and Enable Personal Access Tokens are both off by default in the System Console. +| Symptom | Likely cause | +|---------|--------------| +| `disconnected`, `no bot token is configured` | Store the token with `netclaw secrets set`, then restart. | +| `disconnected`, `rejected the bot token (HTTP 401)` or `rejected the request as unauthorized` | The token is wrong or was revoked. Issue a new one and restart. | +| `disconnected`, `ServerUrl is not configured` | Set `Mattermost.ServerUrl`. | +| Connected but silent | The channel or user isn't in the allow-list, or the bot isn't in the channel. | +| Reconnect loop with `outside a clean startup cycle` | A startup race fixed in 0.26.0-beta.5. Upgrade. | +| Can't create a bot or token | One of the two System Console toggles is still off. | + +A bad token or an unreachable server is a fatal connect error: the channel stays offline until you fix the config and restart the daemon. Network blips retry on their own. :::note -Running Mattermost in a container on Apple Silicon? The [`mattermost/mattermost-preview`](https://hub.docker.com/r/mattermost/mattermost-preview) image is amd64-only — start it with `--platform linux/amd64`. It's a testing image with a bundled database; for production, deploy the [`mattermost/mattermost-team-edition`](https://hub.docker.com/r/mattermost/mattermost-team-edition) image and a real database per [Mattermost's Docker install guide](https://docs.mattermost.com/install/install-docker.html). +Testing on Apple Silicon? The [`mattermost/mattermost-preview`](https://hub.docker.com/r/mattermost/mattermost-preview) image is amd64-only and bundles its own database, so use it for trials only. For anything real, run the Team Edition image with Postgres, as the [Compose guide](/guides/mattermost-docker-compose/) does. ::: -## Next steps - -- [Configure audiences and approval gates](/security/security-model/) to control what tools are available in each channel -- [Set up systemd](/deployment/systemd/) so the daemon stays running after reboots -- [Add Slack](/channels/slack/) or [Discord](/channels/discord/) if your team uses more than one - ## Related pages -- [`netclaw config`](/cli/config/) — Channels -- [`netclaw secrets`](/cli/secrets/) — token management -- [`netclaw status`](/cli/status/) — primary diagnostic for Mattermost -- [Security Model](/security/security-model/) — audiences and approval gates -- [Channel Troubleshooting](/channels/troubleshooting/) — error codes and debug logging -- [Slack](/channels/slack/) — sibling channel integration -- [Discord](/channels/discord/) — sibling channel integration - -## External resources - -- [Mattermost](https://mattermost.com/) — the project, if you don't already run a server -- [Mattermost on GitHub](https://github.com/mattermost/mattermost) — source code, issues, and release notes -- [Mattermost: Bot accounts](https://developers.mattermost.com/integrate/reference/bot-accounts/) — creating and managing bots -- [Mattermost: Personal access tokens](https://developers.mattermost.com/integrate/reference/personal-access-token/) — enabling and issuing tokens -- [Mattermost: Enable bot account creation](https://docs.mattermost.com/configure/integrations-configuration-settings.html) — the System Console settings -- [Mattermost: Install with Docker](https://docs.mattermost.com/install/install-docker.html) — container deployment, including the [official images on Docker Hub](https://hub.docker.com/u/mattermost) -- [Mattermost API reference](https://api.mattermost.com/) — the REST endpoints netclaw uses for replies +- [Run netclaw with Mattermost on Docker Compose](/guides/mattermost-docker-compose/) - a tested, hardened stack +- [Channel troubleshooting](/channels/troubleshooting/) - fixes by symptom +- [How tool approvals work](/security/approvals/) - the approval choices +- [Security model](/security/security-model/) - audiences and approval gates +- [Exposure modes](/deployment/exposure-modes/) - reaching the daemon from outside its host +- [Slack](/channels/slack/) and [Discord](/channels/discord/) - the other chat channels + +## Resources + +- [Mattermost](https://mattermost.com/) - the project, if you don't already run a server +- [Mattermost on GitHub](https://github.com/mattermost/mattermost) - source code, issues and release notes +- [Mattermost: bot accounts](https://developers.mattermost.com/integrate/reference/bot-accounts/) - creating and managing bots +- [Mattermost: personal access tokens](https://developers.mattermost.com/integrate/reference/personal-access-token/) - enabling and issuing tokens +- [Mattermost: integrations settings](https://docs.mattermost.com/administration-guide/configure/integrations-configuration-settings.html) - the System Console toggles +- [Mattermost: install with Docker](https://docs.mattermost.com/deploy/server/deploy-containers.html) - the official container deployment +- [Mattermost API reference](https://api.mattermost.com/) - the REST endpoints netclaw calls diff --git a/src/content/docs/channels/slack.md b/src/content/docs/channels/slack.md index 8e449ca..87fd5ab 100644 --- a/src/content/docs/channels/slack.md +++ b/src/content/docs/channels/slack.md @@ -1,24 +1,26 @@ --- title: "Slack" -description: "Connect Netclaw to your Slack workspace." +description: "Connect netclaw to your Slack workspace over Socket Mode with a bot token, an app token and an allow-list." --- -Netclaw talks to Slack over [Socket Mode](https://api.slack.com/apis/socket-mode) — outbound WebSocket connections only. No public URLs, no ingress rules, no reverse proxies. You create a Slack app, give netclaw two tokens, and it shows up in your workspace as a bot. +netclaw talks to Slack over [Socket Mode](https://docs.slack.dev/apis/events-api/using-socket-mode), which means outbound WebSocket connections only. You need no public URL, no ingress rule and no reverse proxy. You create a Slack app, give netclaw two tokens, and it shows up in your workspace as a bot. ## Prerequisites -- Netclaw installed and initialized ([`netclaw init`](/cli/init/)) -- A Slack workspace where you can install apps (some orgs restrict this to workspace admins) +- netclaw installed and initialized ([`netclaw init`](/cli/init/)) +- A Slack workspace where you can install apps (some organizations limit that to admins) -Connecting Slack is a two-part process: create a Slack app to get your tokens, then enter them in [`netclaw config`](/cli/config/) → Channels. The steps below walk through both. +Connecting Slack has two parts. Create a Slack app to get the tokens, then enter them in [`netclaw config`](/cli/config/) → Channels. ## Create a Slack app -The fastest path: create from a manifest. Go to [api.slack.com/apps](https://api.slack.com/apps) and click **Create New App**. +The fastest path is a manifest. Go to [api.slack.com/apps](https://api.slack.com/apps) and click **Create New App**. ![Slack Your Apps page](/screenshots/output/slack-your-apps.png) -Select **From a manifest**, pick your workspace, and paste this: +The page has a filter box for your existing apps and the **Create New App** button at the top right. + +Choose **From a manifest**, pick your workspace, and paste this: ```json { @@ -55,7 +57,8 @@ Select **From a manifest**, pick your workspace, and paste this: "im:write", "mpim:history", "mpim:read", - "users:read" + "users:read", + "users:read.email" ] } }, @@ -79,57 +82,62 @@ Select **From a manifest**, pick your workspace, and paste this: } ``` -Change the `name` and `display_name` to whatever you want your bot to be called. +Change `name` and `display_name` to what you want the bot called. -The manifest enables the App Home **Messages tab** (`messages_tab_enabled`), which is what lets workspace members DM the bot. If you create the app any other way and skip this, Slack shows *"Sending messages to this app has been turned off"* when someone tries to DM it — you can fix that later under **Features → App Home → Messages Tab**, but the manifest sets it correctly from the start. +The manifest turns on the App Home **Messages tab** (`messages_tab_enabled`), which is what lets people DM the bot. If you create the app another way and skip it, Slack tells anyone who tries to DM the bot *"Sending messages to this app has been turned off"*. You can fix that under **Features > App Home > Messages Tab**. -After creating the app, you need two tokens: +After the app exists, you need two tokens. -1. **App-Level Token** — Settings > Basic Information > App-Level Tokens. Click **Generate Token and Scopes**, name it anything, and add the `connections:write` scope. +1. **App-level token.** Open **Settings > Basic Information**, scroll to **App-Level Tokens**, click **Generate Token and Scopes**, name it anything, and add the `connections:write` scope. -![Generate an app-level token dialog](/screenshots/output/slack-generate-app-token.png) + ![Generate an app-level token dialog](/screenshots/output/slack-generate-app-token.png) -Click **Generate** and copy the `xapp-...` token. + The dialog shows the `connections:write` scope that Socket Mode needs. Click **Generate** and copy the `xapp-...` token. -2. **Bot Token** — Go to **Install App** in the sidebar and click **Install to {Your Workspace}**. +2. **Bot token.** Open **Install App** in the sidebar and click **Install to {Your Workspace}**. -![Slack Install App page](/screenshots/output/slack-install-app.png) + ![Slack Install App page](/screenshots/output/slack-install-app.png) -After approving, copy the `xoxb-...` Bot User OAuth Token from the OAuth & Permissions page. + The install page is also where you reinstall after changing scopes. After you approve, copy the `xoxb-...` Bot User OAuth Token from **OAuth & Permissions**. -Then invite the bot to each channel where it should respond: `/invite @YourBotName` +Then invite the bot to each channel where it should answer: `/invite @YourBotName`. ### What the scopes do | Scope | Why | |-------|-----| | `app_mentions:read` | Receive @-mention events | -| `channels:history` | Read message history in public channels | -| `channels:read` | Resolve channel names to IDs, list public channels | -| `chat:write` | Post messages and replies in threads, plus native loading status | -| `chat:write.customize` | Post with custom display name/avatar | +| `channels:history` | Read history in public channels | +| `channels:read` | Resolve channel names to IDs, list public channels and read channel info | +| `chat:write` | Post replies in threads, and set the native loading status | +| `chat:write.customize` | Post under a custom name or icon. Netclaw doesn't do this today. | | `files:read` | Download files shared in conversations | -| `files:write` | Upload files (agent output, attachments) | -| `groups:history` | Read message history in private channels | +| `files:write` | Upload files such as agent output | +| `groups:history` | Read history in private channels | | `groups:read` | List private channels the bot is in | -| `im:history` | Read DM history for thread context | -| `im:read` | List DM conversations | -| `im:write` | Open DM conversations for proactive messaging | -| `mpim:history` | Read group DM history | -| `mpim:read` | List group DM conversations | -| `users:read` | Look up users by name or email for `lookup_slack_user` | +| `im:history` | Receive DM message events and read DM thread replies for context | +| `im:read` | Read DM details, such as the name of a DM listed in `AllowedChannelIds` | +| `im:write` | Open DMs for proactive messages | +| `mpim:history` | Slack requires it for the `message.mpim` event the manifest subscribes to, and for reading group DM thread replies | +| `mpim:read` | Read group DM details, such as the name of a group DM listed in `AllowedChannelIds` | +| `users:read` | Look up users by name for `lookup_slack_user` | +| `users:read.email` | Look up users by email. Slack returns the email field only with this scope. | + +If you installed from an older copy of this manifest, add `users:read.email` under **OAuth & Permissions** and reinstall the app, or email lookups won't match. ## Configure netclaw -Run `netclaw config` → Channels, enable Slack, and paste your tokens. Netclaw resolves channel names to IDs and saves everything — no JSON editing required. +Run `netclaw config` → Channels, enable Slack, and paste your tokens. netclaw resolves channel names to IDs and saves, so you don't edit JSON. ![The Channels area in netclaw config](/screenshots/output/config-channels-menu.png) -The Channels area — enable an adapter and manage its allow-list; here Slack is connected with 2 channels and 1 user. +The Channels area lists Slack, Discord and Mattermost with a checkbox each. Once an adapter is set up, its row also shows how many channels and users it allows. Nothing is set up in this capture. + +When you save from this screen, netclaw writes the first channel as `DefaultChannelId` and removes `DefaultChannelName`. If you set `DefaultChannelName` by hand, the TUI will replace it. ### Manual configuration -For scripted or headless installs, store tokens with [`netclaw secrets`](/cli/secrets/): +For scripted or headless installs, store the tokens with [`netclaw secrets`](/cli/secrets/): ```bash netclaw secrets set Slack.BotToken xoxb-your-bot-token @@ -147,27 +155,39 @@ Then enable Slack and point it at a channel in `~/.netclaw/config/netclaw.json`: } ``` -Environment variables work too: +Environment variables work too, with list entries indexed from zero: ```bash export NETCLAW_Slack__BotToken="xoxb-..." export NETCLAW_Slack__AppToken="xapp-..." +export NETCLAW_Slack__AllowedChannelIds__0="C0123456789" ``` +An edit to `netclaw.json` restarts the daemon by itself. A vault or environment change needs a restart you run: + +| Where it runs | Restart | +|---------------|---------| +| Native | `netclaw daemon stop`, then `netclaw daemon start` | +| systemd user unit | `systemctl --user restart netclaw` | +| Docker | `docker restart ` | + +:::caution +Keep each list in one place. Environment entries and `netclaw.json` entries merge by position, so a blank `NETCLAW_Slack__AllowedChannelIds__0` overwrites the first channel in the file. `netclaw config` → Channels writes the lists into `netclaw.json`. +::: + ### How channel IDs are stored -Enter channel names or IDs (comma-separated) in `netclaw config` → Channels. Netclaw resolves each entry against the Slack API to its canonical channel ID **before saving**. The stored `AllowedChannelIds` field holds IDs, not display names; display names are shown dynamically in the config UI. +Enter channel names or IDs, comma-separated, in `netclaw config` → Channels. netclaw resolves each against the Slack API and saves the canonical ID, never the display name. -Resolution rules: -- **Confirmed ID** — bot finds the channel by ID: kept as-is. -- **Display name** — resolves to its channel ID and saved as the ID. -- **ID-shaped but not enumerable** — kept; a channel ID is the stable ACL key and is never dropped, even if the bot can't list it right now. -- **Unresolvable display name** — dropped and flagged with a Warning in the config UI. Re-open `netclaw config` → Channels after inviting the bot to that channel. +- **Confirmed ID:** kept as is. +- **Display name:** resolved and saved as the ID. +- **ID-shaped but not listable:** kept. A channel ID is the stable ACL key, so it's never dropped, even if the bot can't list the channel right now. +- **Unresolvable display name:** dropped and flagged with a warning. Invite the bot, then reopen the screen. -`DefaultChannelName` is Slack-only and is resolved to a channel ID live at daemon startup, so it is unaffected by the ACL matching rules above. +`DefaultChannelName` is Slack-only and resolves to an ID at daemon startup. :::caution -**Breaking change (pre-0.24.0 → 0.24.0+):** If you previously placed channel display names in `AllowedChannelIds` by hand in `netclaw.json`, those names no longer match incoming channel IDs and messages will be silently dropped. Re-enter the channels via `netclaw config` → Channels so they resolve to canonical IDs. (ID-shaped entries and `DefaultChannelName` are unaffected.) +Since 0.24.0, channel display names you put in `AllowedChannelIds` by hand no longer match, and messages from those channels are silently dropped. Re-enter them through `netclaw config` → Channels. ID-shaped entries and `DefaultChannelName` are unaffected. ::: ### All config fields @@ -175,19 +195,20 @@ Resolution rules: | Field | Type | Default | Description | |-------|------|---------|-------------| | `Enabled` | bool | `false` | Turn on Slack | -| `SocketMode` | bool | `true` | Must be `true`. Only Socket Mode is supported. | -| `BotToken` | string | — | Bot User OAuth Token (`xoxb-...`). Store with `netclaw secrets set`. | -| `AppToken` | string | — | App-Level Token (`xapp-...`). Required for Socket Mode. Store with `netclaw secrets set`. | -| `DefaultChannelName` | string | — | Channel name, resolved to an ID at startup | -| `DefaultChannelId` | string | — | Channel ID directly (use instead of name if you prefer) | -| `MentionOnly` | bool | `true` | Only respond when @-mentioned | +| `SocketMode` | bool | `true` | Must stay `true`. Slack supports Socket Mode only. | +| `BotToken` | string | - | Bot User OAuth Token (`xoxb-...`). Store with `netclaw secrets set`. | +| `AppToken` | string | - | App-level token (`xapp-...`). Store with `netclaw secrets set`. | +| `DefaultChannelName` | string | - | Channel name, resolved to an ID at startup | +| `DefaultChannelId` | string | - | Channel ID, if you'd rather not use a name | +| `MentionOnly` | bool | `true` | In channels, answer only when @-mentioned | | `AllowDirectMessages` | bool | `false` | Accept DMs | -| `MentionRequiredInDm` | bool | `false` | Require @-mention even in DMs | -| `AllowedChannelIds` | string[] | `[]` | Channel allow-list. Empty + no default = all channels denied | -| `AllowedUserIds` | string[] | `[]` | User allow-list. Empty = everyone in allowed channels is accepted | -| `ChannelAudiences` | object | `{}` | Per-channel [audience](/security/security-model/) overrides. Keys are channel IDs or `"dm"`. Values: `"personal"`, `"team"`, `"public"`. | +| `MentionRequiredInDm` | bool | `false` | Require an @-mention even in DMs | +| `MentionRequiredInThreadByChannel` | object | `{}` | Channel ID to bool. `true` makes a thread reply in that channel need an @-mention. See [Mention behavior](#mention-behavior). | +| `AllowedChannelIds` | string[] | `[]` | Channel allow-list. Empty with no default denies every channel message. | +| `AllowedUserIds` | string[] | `[]` | User allow-list. Empty accepts everyone in allowed channels. | +| `ChannelAudiences` | object | `{}` | Per-channel [audience](/security/security-model/) overrides. Keys are channel IDs or `"dm"`. Values are `"personal"`, `"team"` or `"public"`. | -Tokens live in `~/.netclaw/config/secrets.json` (encrypted at rest). +Tokens live in `~/.netclaw/config/secrets.json`, encrypted at rest. ## Access control @@ -195,55 +216,36 @@ Slack ACL is **default-deny**. Three settings decide who talks to the bot and wh ### Channels -The bot only responds in channels that pass the allow-list: - -- `DefaultChannelName` or `DefaultChannelId` allows one channel -- `AllowedChannelIds` allows multiple -- **If all three are empty, every channel message is denied.** This is the number one setup mistake. +The bot answers only in channels that pass the allow-list. `DefaultChannelName` or `DefaultChannelId` allows one, `AllowedChannelIds` allows more, and if all of them are empty every channel message is denied. That's the number one setup mistake. ```json { "Slack": { - "DefaultChannelName": "openclaw", + "DefaultChannelName": "general", "AllowedChannelIds": ["C0123456789", "C9876543210"] } } ``` -Finding channel IDs: right-click a channel name in Slack, "View channel details," scroll to the bottom. [Slack's help article](https://slack.com/help/articles/221769328) has screenshots. +To find a channel ID, right-click the channel name, choose **View channel details**, and scroll to the bottom. [Slack's help article](https://slack.com/help/articles/221769328) has screenshots. ### Users -`AllowedUserIds` restricts who gets responses: - -- Empty (default) — everyone in allowed channels is accepted -- Non-empty — only listed user IDs get responses, everyone else is silently dropped +`AllowedUserIds` restricts who gets answers. Empty means everyone in an allowed channel. Non-empty means only the listed users, and everyone else is dropped silently. To copy a member ID, open the profile, then the three-dot menu, then **Copy member ID** ([Slack's help article](https://slack.com/help/articles/360003534892)). -Finding user IDs: click a user's profile in Slack, open the three-dot menu, "Copy member ID." [Slack's help article](https://slack.com/help/articles/360003534892) has screenshots. - -Users in `AllowedUserIds` are treated as `TrustedInternal` by the [security model](/security/security-model/). Everyone else is `UntrustedExternal`. +Listed users are `TrustedInternal` in the [security model](/security/security-model/), and everyone else is `UntrustedExternal`. ### Direct messages -DMs are off by default: - -```json -{ - "Slack": { - "AllowDirectMessages": true - } -} -``` - -With DMs on, users can just type normally — `MentionRequiredInDm` defaults to `false`, so no @-mention needed. +DMs are off by default. Turn them on with `"AllowDirectMessages": true`. Users then write without a mention, because `MentionRequiredInDm` defaults to `false`. :::caution -With `AllowDirectMessages: true` and `AllowedUserIds` empty, any workspace member can DM the bot. Lock down `AllowedUserIds` if that's not what you want. +With DMs on and `AllowedUserIds` empty, any workspace member can DM the bot. Set `AllowedUserIds` unless that's what you want. ::: ### Audience overrides -Audience is resolved per-message: `Team` for allow-listed channels and allow-listed users; `Public` for everything else — including unvetted DMs (DMs accepted only because `AllowedUserIds` is empty). To treat all DMs as `Team`, set an explicit `ChannelAudiences "dm"` override. Override with `ChannelAudiences`: +Audience resolves per message: `ChannelAudiences[channelId]` first, then `ChannelAudiences["dm"]`, then a fallback. The fallback is `Team` for a channel listed in `AllowedChannelIds` or a user listed in `AllowedUserIds`, and `Public` for everything else. A DM accepted only because `AllowedUserIds` is empty is `Public`. So is a channel allowed only through `DefaultChannelId` or `DefaultChannelName`, which includes the `DefaultChannelName` example above, unless you also list it in `AllowedChannelIds` or set `ChannelAudiences`. ```json { @@ -256,102 +258,121 @@ Audience is resolved per-message: `Team` for allow-listed channels and allow-lis } ``` -The `"dm"` key is reserved — it matches every direct message rather than a channel ID. A channel-ID entry takes precedence over it. An unrecognized audience value is rejected outright: the message is denied rather than falling back to a default. - -[Security Model](/security/security-model/) has the full breakdown on how audiences map to tools and permissions. +A channel-ID entry beats `"dm"`, and an unrecognized value denies the message instead of falling back. [Security model](/security/security-model/) maps audiences to tools. ## Behavior in Slack ### Threads and sessions -Each Slack thread is its own isolated session, and the bot always replies in-thread. Idle threads are freed from memory after 1 hour; the conversation context clears after 2 hours of inactivity. - -On daemon restart, thread history is backfilled so in-progress conversations pick up where they left off. +Each Slack thread is its own session, and the bot always replies in-thread. An idle thread is freed from memory when its session goes idle, after `Session.IdleTimeout` (one hour by default), and a later reply wakes it again. After a daemon restart, thread history is backfilled so conversations continue. ### Mention behavior -`MentionOnly: true` (the default) means the bot ignores messages that don't @-mention it. Two exceptions: +`MentionOnly: true` makes the bot ignore channel messages that don't @-mention it. netclaw strips the mention before the text reaches the model. These rules sit on top: + +- **Thread replies:** once a thread has an active session, replies in it don't need a mention, unless `MentionRequiredInThreadByChannel` is `true` for that channel. +- **File shares:** a message with an attachment follows the same mention rules as any other message. An un-mentioned upload in a channel is ignored. + +`MentionRequiredInThreadByChannel` maps channel IDs to `true` or `false`: + +```json +{ + "Slack": { + "MentionRequiredInThreadByChannel": { + "C0123456789": true + } + } +} +``` -- **Thread replies** — if a thread already has an active session, the bot responds to everything in that thread without needing a mention -- **File shares** — attached files bypass the mention check entirely, with or without an active thread +With the rule on, an un-mentioned thread reply is ignored (`ThreadMentionRequired` in the log). The next message that mentions the bot makes netclaw re-read everything posted since its last reply, so it answers with full context. A channel with no entry behaves as `false`. The rule doesn't apply to DMs. -Netclaw strips the @-mention before passing text to the LLM. +In `netclaw config` → Channels, **Space** toggles **Require @mention** on the selected channel. A channel you add there starts with the rule on if its audience is `team` or `public`, and off if it's `personal`. Only `true` entries are saved. ### Message formatting -Netclaw converts LLM markdown to Slack [Block Kit](https://api.slack.com/block-kit): headers, code blocks, blockquotes, lists, bold, italic, strikethrough, inline code, links. Everything renders natively in Slack. +Netclaw sends each reply as a single Slack Markdown block, and Slack renders it, tables included. Replies over 12,000 characters go out as plain text instead. ### Loading status -While Netclaw is working on a reply, Slack shows the bot's native thread status as ` is thinking...`. This uses Slack's `assistant.threads.setStatus` API with the existing `chat:write` scope, so the manifest above does not need `assistant:write` or the Slack assistant split view. +While netclaw works on a reply, Slack shows the bot's native thread status as ` is thinking...`. That uses `assistant.threads.setStatus` with the existing `chat:write` scope, so the manifest doesn't need `assistant:write`. ### Tool approval -When a tool call needs approval, netclaw posts a Block Kit prompt right in the thread: +When a tool call needs approval, netclaw posts a prompt in the thread with one button per choice. For a prompt from the main agent you can also reply with the letter next to a choice. A prompt from a sub-agent has buttons only, and a typed letter doesn't answer it. -![Tool approval prompt in a Slack thread, with approve and deny buttons](/assets/approval-prompt.png) +![Tool approval prompt in a Mattermost thread: shell_execute, a push to origin main, and six lettered choices from A (Once) to F (Deny)](/assets/approval-prompt.png) -Shows the tool name, the exact command, and the approval buttons. Only the user who triggered the request can approve. System-initiated tool calls (`VerifiedAutomation`) can be approved by anyone in the thread. +The picture shows the same prompt in Mattermost's text-reply form. Slack renders the same six choices as buttons, with the lettered list as a fallback line underneath. A sub-agent prompt has no lettered line. -The full prompt offers five choices: **Once**, **This chat**, **Always here**, **Always anywhere**, and **Deny** — "Always anywhere" is the broadest grant, so reach for it sparingly. netclaw shows fewer when some don't apply — a command it can't cleanly parse (shell control flow, or unbalanced quotes) drops to just **Once** and **Deny**. You can reply with the letter shown next to each option instead of clicking. +The choices are **Once**, **This chat**, **Always here**, **Always anywhere**, **Deny**, and **This repository** when it applies. A command netclaw can't parse cleanly gets only **Once** and **Deny**. [How tool approvals work](/security/approvals/) explains what each one grants. Only the user whose request triggered the prompt can answer it. System-initiated calls (`VerifiedAutomation`) can be answered by anyone in the thread. A prompt that nothing can answer any more is redrawn as a single line with the reason and no buttons, such as `This approval request was withdrawn because the task that asked for it ended. Nothing ran.` ### Proactive messaging -The LLM can initiate conversations through two built-in tools: +The model can start conversations with two tools: | Tool | What it does | -|------|-------------| -| `send_channel_message` | Posts a message to a Slack channel or DM using a resolved destination. Pass `channel_key="slack"` and a `destination` object (with `kind`, `id`) from `lookup_slack_user` or `lookup_channel_destination`. Respects ACL. | -| `lookup_slack_user` | Searches users by name, display name, or email. Returns up to 10 matches. Filtered to `AllowedUserIds` if set. Cached 5 minutes. | +|------|--------------| +| `send_channel_message` | Posts to a Slack channel or DM. Pass `channel_key="slack"` and a `destination` (`kind`, `id`) from `lookup_slack_user` or `lookup_channel_destination`. Respects ACL. | +| `lookup_slack_user` | Searches users by name, display name or email, up to 10 matches, filtered to `AllowedUserIds` when set, cached for five minutes. Email matching needs `users:read.email`. | ### Ignored messages -The bot drops: empty messages (no text, no files), hidden messages, other bots' messages, its own messages, messages with unsupported subtypes (edits, joins, bot subtype messages), DMs when `AllowDirectMessages` is off, and un-mentioned channel messages when `MentionOnly` is on and there's no active thread. +The bot drops empty messages, hidden messages, its own and other bots' messages, subtypes it doesn't handle (edits, joins), DMs when `AllowDirectMessages` is off, and un-mentioned channel messages when `MentionOnly` is on and there's no active thread. -## Verify it works +### Connection supervision -Restart the daemon and check status: +A supervisor checks the Socket Mode connection every 5 seconds and reconnects with exponential backoff up to 5 minutes. You don't restart the daemon for a dropped connection. A bad token or `SocketMode: false` is a fatal connect error: Slack stays offline until you fix it and restart. + +## Verify it works ```bash -netclaw daemon stop && netclaw daemon start netclaw status ``` -Slack should show `connected`. If it doesn't, run `netclaw doctor` — it checks token validity and ACL config. +Look for `slack: healthy (enabled)` under `connectors`. Trimmed output from a daemon that has been on Slack for a few minutes: + +```text +daemon: PID 25721, uptime 11m 33s, endpoint http://127.0.0.1:5199 +... +slack counters: recv=27 routed=27 dropped=26 replied=9 rejected=0 reply_failed=0 +... +connectors: +- discord: disabled (disabled) + Discord connector is disabled in configuration. +... +- slack: healthy (enabled) +``` - +`recv` goes up as Slack delivers events, so a number above zero means the bot is hearing the workspace. `dropped` counts events the bot ignores on purpose, such as a message the ACL denies or a thread reply it isn't part of. In 0.28.2 it also counts the bot's own replies coming back from Slack, so expect it to climb with every answer. `rejected` and `reply_failed` count replies that Slack refused or that failed to post, and they should stay at `0`. If Slack isn't `healthy`, run [`netclaw doctor`](/cli/doctor/), which checks the bot token and the ACL config. -Then @-mention the bot in an allowed channel. If it responds, you're set. +Then @-mention the bot in an allowed channel. If it answers in a thread, you're set. ## Troubleshooting -Common problems and fixes are in [Channel Troubleshooting](/channels/troubleshooting/). The hits: +[Channel troubleshooting](/channels/troubleshooting/) is organized by symptom. The usual ones: -- **Connected but silent** — `AllowedChannelIds` is empty and no default channel is set, so all traffic gets denied -- **Works in some channels, not others** — channel missing from `AllowedChannelIds`, or the bot hasn't been invited -- **Socket Mode keeps disconnecting** — the `xapp-...` App-Level Token may have expired or been revoked +| Symptom | Likely cause | +|---------|--------------| +| Connected but silent | `AllowedChannelIds` is empty and no default channel is set. | +| Works in some channels, not others | The channel isn't in `AllowedChannelIds`, or the bot wasn't invited. | +| Stopped answering in a thread | `MentionRequiredInThreadByChannel` is on for the channel. Mention the bot. | +| Doesn't match users by email | The app lacks `users:read.email`. | +| `Sending messages to this app has been turned off` | The Messages tab is off. | +| Socket Mode won't connect | The `xapp-...` token is wrong or revoked. | -## Next steps +## Related pages -- [Configure audiences and approval gates](/security/security-model/) to control what tools are available in each channel -- [Set up systemd](/deployment/systemd/) so the daemon stays running after reboots -- [Add Discord](/channels/discord/) or [Mattermost](/channels/mattermost/) if your team uses more than one -- [Run `netclaw doctor`](/cli/doctor/) to verify token health and ACL config +- [Mattermost](/channels/mattermost/) and [Discord](/channels/discord/) - the other chat channels +- [Channel troubleshooting](/channels/troubleshooting/) - fixes by symptom +- [How tool approvals work](/security/approvals/) - the approval choices +- [Security model](/security/security-model/) - audiences and approval gates +- [`netclaw doctor`](/cli/doctor/) - Slack auth and ACL checks -## Related pages +## Resources -- [`netclaw config`](/cli/config/) — Channels -- [`netclaw secrets`](/cli/secrets/) — token management -- [`netclaw doctor`](/cli/doctor/) — Slack auth and ACL diagnostics -- [Security Model](/security/security-model/) — audiences and approval gates -- [Channel Troubleshooting](/channels/troubleshooting/) — error codes and debug logging -- [Discord](/channels/discord/) — sibling channel integration -- [Mattermost](/channels/mattermost/) — sibling channel integration - -## External resources - -- [Slack API: Socket Mode](https://api.slack.com/apis/socket-mode) — how Socket Mode connections work -- [Slack API: Bot Token Scopes](https://api.slack.com/scopes) — scope reference -- [Slack: Block Kit](https://api.slack.com/block-kit) — message formatting -- [Slack: Finding IDs](https://slack.com/help/articles/221769328) — channel and user IDs for ACL config -- [Slack: Finding User IDs](https://slack.com/help/articles/360003534892) — step-by-step for copying member IDs +- [Slack: Socket Mode](https://docs.slack.dev/apis/events-api/using-socket-mode) - how Socket Mode connections work +- [Slack: app manifests](https://docs.slack.dev/app-manifests/configuring-apps-with-app-manifests) - the manifest format +- [Slack: scopes](https://docs.slack.dev/reference/scopes) - scope reference +- [Slack: finding channel IDs](https://slack.com/help/articles/221769328) - for the allow-list +- [Slack: finding member IDs](https://slack.com/help/articles/360003534892) - for `AllowedUserIds` diff --git a/src/content/docs/channels/troubleshooting.md b/src/content/docs/channels/troubleshooting.md index 5632d83..7c4dbc3 100644 --- a/src/content/docs/channels/troubleshooting.md +++ b/src/content/docs/channels/troubleshooting.md @@ -1,271 +1,291 @@ --- -title: "Channel Troubleshooting" -description: "Diagnose and fix common Slack, Discord, and Mattermost channel issues." +title: "Channel troubleshooting" +description: "Fix a Slack, Discord or Mattermost bot that won't connect, won't answer, or stopped answering, organized by symptom." --- -Netclaw not responding in Slack, Discord, or Mattermost? Two commands will tell you what's wrong most of the time: +When the bot is silent, two commands and one log tell you most of what you need: ```bash -netclaw status # live connector health (requires running daemon) +netclaw status # live connector health and message counters (needs a running daemon) netclaw doctor # offline config and credential checks ``` -Daemon logs live at `~/.netclaw/logs/`, or `journalctl -u netclaw` if you're running it as a systemd service. +`netclaw status` reports each connector as `healthy`, `degraded`, `disconnected` or `disabled`, with counters such as `recv=2 routed=2 dropped=0 replied=2`. Doctor checks Slack only (auth and ACL). For Discord and Mattermost, use status and the daemon log. -## Symptom Triage +## Where the logs are + +The daemon writes `~/.netclaw/logs/daemon-YYYY-MM-DD.log`. After 10 MiB it continues in `daemon-YYYY-MM-DD.1.log`, `.2.log` and so on, and the active file is the highest-numbered segment of the newest date. Crashes go to `crash-*.log` in the same directory. + +| Where it runs | Read the log | +|---------------|--------------| +| Native | `tail -f ~/.netclaw/logs/daemon-*.log` | +| systemd user unit | `journalctl --user -u netclaw -f`, or the file above | +| Docker | `docker exec sh -c 'tail -f ~/.netclaw/logs/daemon-*.log'`. `docker logs` shows only the entrypoint and a few startup warnings. | + +Restart, when a section below tells you to: + +| Where it runs | Restart | +|---------------|---------| +| Native | `netclaw daemon stop`, then `netclaw daemon start` | +| systemd user unit | `systemctl --user restart netclaw` | +| Docker | `docker restart ` (`netclaw daemon start` does nothing under the image's entrypoint) | + +An edit to `netclaw.json` restarts the daemon on its own. Tokens in the vault and environment variables don't, so restart after changing those. + +## Find your symptom | What you're seeing | Jump to | -|---|---| -| Bot shows "disconnected" in `netclaw status` | [Authentication Errors](#authentication-errors) | -| Bot connected but never replies | [ACL and Permissions](#acl-and-permissions) | -| Bot replies to some messages but not others | [Bot Not Responding to Messages (MentionOnly)](#bot-not-responding-to-messages-mentiononly) | -| Bot was working, then stopped | [Connectivity](#connectivity) | -| Bot processes messages but replies never appear | [Message Delivery](#message-delivery) | +|--------------------|---------| +| `disconnected` or `degraded` in `netclaw status` | [The bot won't connect](#the-bot-wont-connect) | +| It connects, then drops again and again | [The connection keeps dropping](#the-connection-keeps-dropping) | +| Connected, messages arrive, no reply | [The bot is connected but silent](#the-bot-is-connected-but-silent) | +| It answers some messages and not others | [The bot answers only some messages](#the-bot-answers-only-some-messages) | +| It stopped answering in a thread | [The bot stopped answering in a thread](#the-bot-stopped-answering-in-a-thread) | +| It works, then the reply never shows up | [The reply never appears](#the-reply-never-appears) | +| Approval buttons or letters do nothing | [Approvals don't work](#approvals-dont-work) | +| It answers its own messages | [The bot talks to itself](#the-bot-talks-to-itself) | + +![netclaw doctor output: 24 checks, all PASS, ending with doctor exit code: 0](/screenshots/output/doctor.png) -![netclaw doctor running diagnostic checks including Slack Auth and Slack ACL](/screenshots/output/doctor.png) +The Slack ACL and Slack Auth lines are the channel checks. Slack isn't configured in this capture, so both pass trivially. -Doctor only checks Slack right now (Auth and ACL). For Discord and Mattermost, use `netclaw status` for connector health and the daemon logs for details. +## The bot won't connect -## Authentication Errors +A bad token or a missing setting is a fatal connect error. The channel stays offline until you fix the config and restart the daemon. The daemon and your other channels keep running. Network failures are different: those retry on their own. -### Slack: Invalid or Expired Bot Token +### Slack: invalid or expired bot token -`netclaw status` shows Slack as `disconnected`. Doctor's Slack Auth check fails with one of these: +**Symptom:** `netclaw status` shows Slack as `disconnected`, and doctor's Slack Auth check fails with one of these: | Error | Meaning | |-------|---------| -| `invalid_auth` | Bot token is invalid. Check your Slack app's Bot User OAuth Token. | -| `token_revoked` | Token has been revoked. Generate a new one. | -| `token_expired` | Token has expired. Generate a new one. | -| `account_inactive` | Bot account is deactivated. | -| `not_authed` | No authentication token provided. | -| `missing_scope` | Bot token lacks `channels:read` scope. | +| `invalid_auth` | The bot token is invalid. Check your app's Bot User OAuth Token. | +| `token_revoked` | The token was revoked. Generate a new one. | +| `token_expired` | The token expired. Generate a new one. | +| `account_inactive` | The bot account is deactivated. | +| `not_authed` | No token was provided. | +| `missing_scope` | The token lacks `channels:read`, which the startup check needs. Other missing scopes show up when a call needs them, as in [The reply never appears](#the-reply-never-appears). | -If you're not sure which token is bad, regenerate both from the [Slack App Management page](https://api.slack.com/apps): +**Fix:** If you're not sure which token is bad, replace both from the [Slack app page](https://api.slack.com/apps), then restart: ```bash netclaw secrets set Slack.BotToken xoxb-your-new-token netclaw secrets set Slack.AppToken xapp-your-new-token -netclaw daemon stop && netclaw daemon start ``` -### Slack: No Bot Token Found +If doctor says "Slack is enabled but no bot token found", run `netclaw config` → Channels → Slack, or `netclaw secrets set Slack.BotToken xoxb-...`. -Doctor reports "Slack is enabled but no bot token found." +### Slack: Socket Mode is off -Run `netclaw config` → Channels → Slack, or set the token directly: +**Symptom:** the log says `Slack channel currently supports Socket Mode only. Set Slack:SocketMode to true.` -```bash -netclaw secrets set Slack.BotToken xoxb-your-token -``` +**Cause:** `Slack.SocketMode` is `false`. Only Socket Mode is supported. -### Discord: Invalid Bot Token +**Fix:** remove the key or set it to `true`, and make sure the app has Socket Mode on and a valid `xapp-...` token with `connections:write`. -`netclaw status` shows Discord as `disconnected`. Daemon logs show an authentication error at startup. +### Discord: invalid token or no access -| HTTP Code | Meaning | -|-----------|---------| -| 401 | Bot token is invalid. Check your Discord application's bot token. | -| 403 (code 50001) | Bot lacks access. Ensure it has been invited to the server. | -| 403 | Access denied. Check bot permissions. | -| 404 | Resource not found. Check the ID is correct. | +**Symptom:** `netclaw status` shows Discord as `disconnected` and the log shows an authentication error at startup. Discord's API returns HTTP 401 for a wrong or reset token. When the bot can't see a resource, the log shows the Discord error as received. -Discord uses a single bot token (not a `xoxb-`/`xapp-` pair like Slack). Regenerate it from the [Discord Developer Portal](https://discord.com/developers/applications): +**Fix:** reset the token in the [Developer Portal](https://discord.com/developers/applications), then: ```bash netclaw secrets set Discord.BotToken your-new-token -netclaw daemon stop && netclaw daemon start ``` -### Discord: Connection Timeout at Startup +Restart. If the log shows a Discord access error instead of a 401, check that the bot was invited to the server. -Daemon start hangs, then Discord shows `disconnected`. Logs show a 30-second timeout waiting for the gateway READY event. +### Discord: startup waits, then disconnects -The bot token may be valid but the bot hasn't been invited to any server, so Discord has nothing to initialize. Verify the bot has been added to your server. Use Discord's [URL Generator](https://discord.com/developers/docs/topics/oauth2#bot-authorization-flow) to create an invite link with the `bot` scope. +**Symptom:** the daemon starts slowly, and the log shows a 30-second timeout waiting for the gateway READY event. -### Mattermost: Invalid Bot Token +**Cause:** the token is valid, but the bot isn't in any server, so Discord has nothing to initialize. -`netclaw status` shows Mattermost as `disconnected`. The daemon logs show `Mattermost rejected the bot token (HTTP 401)` with more detail. +**Fix:** invite the bot with an OAuth2 URL that has the `bot` scope. See [Discord setup](/channels/discord/#5-install-the-bot-to-your-server). -The bot's personal access token is wrong or was revoked. Re-issue it from the System Console (**Integrations > Bot Accounts**) and update the secret: +### Mattermost: token or URL problems -```bash -netclaw secrets set Mattermost.BotToken your-new-token -netclaw daemon stop && netclaw daemon start -``` +**Symptom:** `netclaw status` shows Mattermost as `disconnected` with one of these reasons: -If startup logs say `Mattermost is enabled but Mattermost:ServerUrl is not configured` or `...no bot token is configured`, the corresponding field is missing from `netclaw.json`/`secrets.json`. See the [Mattermost setup guide](/channels/mattermost/). +| Reason | Fix | +|--------|-----| +| `Mattermost is enabled but Mattermost:ServerUrl is not configured` | Set `Mattermost.ServerUrl`. | +| `Mattermost is enabled but no bot token is configured` | Run `netclaw secrets set Mattermost.BotToken `. | +| `Mattermost rejected the bot token (HTTP 401)` or `Mattermost rejected the request as unauthorized` | Issue a new token under **Integrations > Bot Accounts** and store it. | -## ACL and Permissions +Restart after storing a token. The daemon doesn't watch the vault. -### All Channel Traffic Denied +If you can't create the bot or its token in the first place, two System Console settings are off by default: **Enable bot account creation** and **Enable personal access tokens**. See [Mattermost setup](/channels/mattermost/#create-a-bot-account). -Netclaw receives messages but never responds. `netclaw status` counters show `recv > 0`, `dropped > 0`, `routed = 0`. Logs show `channel_not_allowed`. +## The connection keeps dropping -This is the number one gotcha. When `AllowedChannelIds` is empty and no `DefaultChannelId`/`DefaultChannelName` is set, netclaw denies all channel traffic. +### Slack Socket Mode drops -Set a default channel or list specific channels in `~/.netclaw/config/netclaw.json`: +**Symptom:** Slack goes `disconnected`, then recovers. -```json -{ - "Slack": { - "DefaultChannelId": "C0123456789" - } -} -``` +**Cause and fix:** since 0.27.0 a supervisor checks the connection every 5 seconds and reconnects with backoff up to 5 minutes, so a restart is no longer the first step. If it never recovers, check the [Slack status page](https://status.slack.com/), your network, and that the `xapp-...` token is still valid. -Or allow specific channels: +### Discord gateway drops -```json -{ - "Slack": { - "AllowedChannelIds": ["C0123456789", "C9876543210"] - } -} -``` +**Symptom:** Discord goes `disconnected`. -After editing config, restart the daemon: `netclaw daemon stop && netclaw daemon start` +**Fix:** check [Discord's status page](https://discordstatus.com/) and your network, then confirm the token. Restart if it doesn't come back. -To find Slack channel IDs, right-click the channel name, select "View channel details," and scroll to the bottom. For Discord, enable Developer Mode in Settings > App Settings > Advanced, then right-click any channel and select "Copy Channel ID." See [Slack: Finding IDs](https://slack.com/help/articles/221769328) and [Discord: Finding IDs](https://support.discord.com/hc/en-us/articles/206346498). +### Mattermost reconnect loop -[`netclaw doctor`](/cli/doctor/) catches this one automatically. +**Symptom:** the log repeats this, with the delay growing each time ([netclaw#1517](https://github.com/netclaw-dev/netclaw/issues/1517)): -### User Not Allowed +```text +Connected to Mattermost WebSocket at ws://mattermost:8065/api/v4/websocket +Gateway requested clean reconnect: Mattermost gateway reconnected outside a clean startup cycle; forcing a clean reconnect. +Disconnected from Mattermost WebSocket: Closed by client +``` -Specific users get no response. Logs show `user_not_allowed`. +**Cause:** a startup race in older releases, reported on 0.24.2 and 0.26.0-beta.4. Startup now waits for the WebSocket `OnConnected` event, which shipped in 0.26.0-beta.5. -Add the user's ID to `AllowedUserIds`, or clear the list to allow all users: +**Fix:** upgrade to 0.27.1 or newer. If it still loops, check that `ServerUrl` is an address netclaw can reach, and that any proxy between netclaw and Mattermost forwards WebSocket upgrades to `/api/v4/websocket`. -```json -{ - "Slack": { - "AllowedUserIds": ["U0123456789", "U9876543210"] - } -} -``` +## The bot is connected but silent -### DMs Not Working +**Symptom:** `netclaw status` shows `recv` rising and `routed` at 0. The log has one of these lines: -Direct messages to the bot get no response, but channel messages work fine. Logs show `event_filtered ... reason=routing_policy_ignore ignoreReason=DmNotAllowed`. +```text +event_dropped event=... reason=channel_not_allowed +event_dropped event=... reason=user_not_allowed +event_filtered event=... reason=routing_policy_ignore ignoreReason=DmNotAllowed +``` -`AllowDirectMessages` defaults to `false`. Turn it on: +| Reason | Cause | Fix | +|--------|-------|-----| +| `channel_not_allowed` | The channel isn't in `AllowedChannelIds` and isn't the default channel. This is the number one setup mistake. | Add the channel ID, or set `DefaultChannelId`. | +| `user_not_allowed` | `AllowedUserIds` is set and the sender isn't in it. | Add the user's ID, or clear the list. | +| `DmNotAllowed` | `AllowDirectMessages` is `false`, the default. | Set it to `true`. | ```json { "Slack": { + "AllowedChannelIds": ["C0123456789"], + "AllowedUserIds": ["U0123456789"], "AllowDirectMessages": true } } ``` -Doctor warns if you enable DMs with an empty `AllowedUserIds` list, since any workspace member can then DM the bot. - -### Channel Messages Silently Dropped After Upgrading from Pre-0.24.0 +[`netclaw doctor`](/cli/doctor/) catches an empty allow-list for Slack. To find IDs, see [Slack](/channels/slack/#channels), [Discord](/channels/discord/#channels) or [Mattermost](/channels/mattermost/#how-channel-ids-are-stored). -Bot is connected and the channel is in `AllowedChannelIds`, but every message is dropped. Logs show `channel_not_allowed`. Started after upgrading to 0.24.0 or later. +If no `event_dropped` line appears at all, the event never reached netclaw. On Slack and Mattermost, invite the bot to the channel. On Discord, check the Message Content intent. -**Cause:** Before 0.24.0, `AllowedChannelIds` accepted display names (e.g. `"general"`) or manually entered values that were never resolved to canonical IDs. As of 0.24.0, incoming message channel IDs are matched against stored IDs only — display names no longer match. +### Everything stopped after upgrading from before 0.24.0 -**Fix:** Re-enter the channels via `netclaw config` → Channels so they resolve to canonical IDs and are saved correctly. This is the easiest path — the config UI does the resolution automatically. +**Symptom:** the channel is in `AllowedChannelIds`, but every message logs `channel_not_allowed`. It began after upgrading to 0.24.0 or later. -If you prefer to fix it manually in `~/.netclaw/config/netclaw.json`, replace any display names in `AllowedChannelIds` with the canonical channel ID for each platform: +**Cause:** before 0.24.0, `AllowedChannelIds` accepted display names such as `"general"`. Since 0.24.0, incoming channel IDs match stored IDs only. -- **Slack:** right-click a channel name → "View channel details" → scroll to the bottom for the `C...` ID. See [Slack: Finding IDs](https://slack.com/help/articles/221769328). -- **Discord:** enable Developer Mode in Settings > App Settings > Advanced, then right-click any channel → "Copy Channel ID." -- **Mattermost:** find IDs in the System Console or via the [REST API](https://api.mattermost.com/). +**Fix:** re-enter the channels in `netclaw config` → Channels so they resolve to IDs. Or edit `netclaw.json` and swap each name for its canonical ID. -After editing, restart the daemon: `netclaw daemon stop && netclaw daemon start` +### The allow-list is right but the bot is still silent in Docker or Compose -### Bot Not Responding to Messages (MentionOnly) +**Cause:** an environment variable and `netclaw.json` both set the same list. .NET merges them by position, so a blank `NETCLAW_Mattermost__AllowedChannelIds__0` overwrites the first channel in the file. -Bot is connected and healthy but only responds to some messages. Logs show `event_filtered ... reason=routing_policy_ignore ignoreReason=ChannelMentionRequired`. +**Fix:** keep each list in one place, then check the value the container sees, for example `docker exec printenv NETCLAW_Mattermost__AllowedChannelIds__0`. -`MentionOnly` defaults to `true`, so the bot ignores messages that don't @-mention it. Either @-mention the bot every time, or turn it off: +## The bot answers only some messages -**Slack:** +**Symptom:** the bot is healthy but ignores messages that don't mention it. The log has: -```json -{ - "Slack": { - "MentionOnly": false - } -} +```text +event_filtered event=... reason=routing_policy_ignore ignoreReason=ChannelMentionRequired ``` -**Discord:** +**Cause:** `MentionOnly` defaults to `true`, so in channels the bot answers only messages that @-mention it. + +**Fix:** mention the bot, or turn the rule off for the channel type: ```json { - "Discord": { - "MentionOnly": false, - "MentionRequiredInDm": false - } + "Slack": { "MentionOnly": false }, + "Discord": { "MentionOnly": false }, + "Mattermost": { "MentionOnly": false } } ``` -`MentionRequiredInDm` controls whether DMs also need a mention (defaults to `false`). +`MentionRequiredInDm` separately controls whether DMs need a mention (default `false`). On Slack, a file upload follows the same rule, so an un-mentioned attachment in a channel is ignored. -## Connectivity +## The bot stopped answering in a thread -### Slack Socket Mode Disconnected +**Symptom:** the bot was answering in a thread, then went quiet. The log has: -`netclaw status` shows Slack as `disconnected`. Was working, then stopped. +```text +event_filtered event=... reason=routing_policy_ignore ignoreReason=ThreadMentionRequired +``` -Restart the daemon: +**Cause:** `MentionRequiredInThreadByChannel` is `true` for that channel, so thread replies need a mention. A channel you added through `netclaw config` as `team` or `public` starts with the rule on. -```bash -netclaw daemon stop && netclaw daemon start -``` +**Fix:** mention the bot in the reply. It then re-reads what it missed and answers with full context. To turn the rule off, press **Space** on the channel in `netclaw config` → Channels, or set the channel's entry to `false` in `netclaw.json`. + +If the thread is old, check the idle limit too. A thread is freed from memory when its session goes idle, after `Session.IdleTimeout` (one hour by default), and a later reply wakes it again. A restart backfills history. + +## The reply never appears -If it keeps disconnecting, check the [Slack Status page](https://status.slack.com/) and your network. Make sure the App Token (`xapp-...`) is still valid; Socket Mode requires it. +### Slack: the post fails -Only Socket Mode is supported. Setting `SocketMode: false` leaves Slack unable to connect, so it shows as degraded in `netclaw status` and the daemon logs a connection error. +**Symptom:** the log shows the message was processed, but nothing shows up in Slack, with one of these errors: -### Discord Gateway Disconnected +| Error | Meaning | +|-------|---------| +| `not_in_channel` | The bot isn't in the channel. Invite it with `/invite @botname`. | +| `channel_not_found` | The channel ID is wrong or the channel was deleted. | +| `missing_scope` | The token lacks a scope needed to post. | +| `no_permission` | The bot can't post in this channel. | +| `msg_too_long` | The reply exceeded Slack's size limit. | +| `invalid_blocks` | Slack rejected the formatted payload. | +| `too_many_attachments` | The reply has too many attachments. | +| `rate_limited` | Slack throttled the call. netclaw retries. | -`netclaw status` shows Discord as `disconnected`. +For permission errors, invite the bot and check the scopes under **OAuth & Permissions** in the [Slack API dashboard](https://api.slack.com/apps). Reinstall the app after you change scopes. -Restart the daemon: `netclaw daemon stop && netclaw daemon start`. If it keeps dropping, check the [Discord Status page](https://discordstatus.com/) and verify the bot token. +### Discord: the post fails -### Rate Limiting +**Symptom:** the bot processes the message but no reply shows up. -Replies are delayed or fail intermittently. Logs show `rate_limited` (Slack) or HTTP 429 (Discord). +**Cause:** usually the bot lacks **Send Messages**, **Send Messages in Threads** or **Create Public Threads** in that channel. The log has the HTTP status. A 429 is Discord rate limiting, and netclaw retries. -Netclaw retries on its own. If rate limiting is sustained, check whether another integration shares the same bot token's rate limit budget. +**Fix:** grant the permission in the channel's settings. -## Message Delivery +### Mattermost: the post fails -### Slack: Reply Not Posted +**Symptom:** the log shows a reply error after the message was processed. -Bot processes the message (logs show session activity) but the reply never appears in Slack. +**Cause:** the bot isn't a member of the channel, or it lost access to it. -| Error | Meaning | -|-------|---------| -| `not_in_channel` | Bot hasn't been invited to the channel. Invite it with `/invite @botname`. | -| `channel_not_found` | Channel ID is wrong or the channel was deleted. | -| `missing_scope` | Bot token lacks the required OAuth scope for posting. | -| `no_permission` | Bot doesn't have permission to post in this channel. | -| `msg_too_long` | Response exceeded Slack's message size limit. | -| `invalid_blocks` | Malformed Block Kit payload. Usually a formatting edge case in the response. | -| `too_many_attachments` | Response has too many attachments. | +**Fix:** add the bot to the channel with `/invite @netclaw`. The channel's `reply_failed` counter in `netclaw status` rises while this is broken. -For permission errors: add the bot to the channel and make sure it has `chat:write` scope. Check your app's OAuth scopes in the [Slack API dashboard](https://api.slack.com/apps). +## Approvals don't work -### Discord: Reply Not Posted +| Symptom | Cause | Fix | +|---------|-------|-----| +| Lettered prompt, and your reply is ignored | You aren't the user who triggered the request. The bot posts "Only the requesting user can approve this tool action." | Have the original user reply, or ask again yourself. | +| Mattermost buttons never respond | `CallbackUrl` is set but the Mattermost server can't reach the daemon. | Fix the route, or leave `CallbackUrl` unset if you don't use sub-agents. See [Mattermost](/channels/mattermost/#buttons-need-a-reachable-callback). | +| Mattermost button says "no longer valid" | The daemon restarted. Button tokens live in memory. | Ask the agent to re-issue the prompt. | +| A sub-agent's call is denied with `approval_prompt_unavailable`, and its prompt shows "withdrawn ... could not show it in this channel" | Mattermost has no `CallbackUrl`, and sub-agent prompts need buttons. | Set `CallbackUrl`. See [Mattermost](/channels/mattermost/#buttons-need-a-reachable-callback). | +| A typed letter is ignored and the bot says "N approval prompts are open. Use the buttons on the prompt you mean." | Two or more prompts are open for you and one is from a sub-agent. A letter can't pick one. | Click the buttons on the prompt you mean. | +| "That approval prompt already has an answer" or "Approval response ignored: that approval prompt already has an answer." | A second click or reply on a prompt that was already answered. | Nothing to fix. The first answer stands. | +| A prompt lost its buttons and shows a withdrawal reason | The task that asked was cancelled, ended or stopped, the daemon restarted, or the channel couldn't show the prompt. Nothing ran. | Ask the agent to try again. | +| No prompt appears and the call doesn't run | The channel couldn't post the prompt. netclaw tells the model nobody saw it. | Fix the post error above, then ask again. | -Bot processes the message but no reply appears in Discord. +The choices on each prompt are explained in [How tool approvals work](/security/approvals/). -Check daemon logs for HTTP error codes. Usually the bot is missing `Send Messages` permission in the target channel. Fix it in your Discord server's channel permission settings. +## The bot talks to itself -### Bot Reply Loop +**Symptom:** the bot answers its own messages in a loop. `events.filtered{reason="bot_message"}` and `events.received` are both high. -Bot responds to its own messages, creating an infinite loop. High `events.filtered{reason="bot_message"}` counter alongside high `events.received`. +**Cause:** netclaw filters its own messages, so this is rare. It points to a failed bot-identity lookup at startup. -Netclaw filters its own messages to prevent loops, so this is rare. Restart the daemon. If the loop comes back, bot user ID detection probably failed at startup. Look for errors around the `slack.Auth.Test()` call in the logs. +**Fix:** restart the daemon. If it comes back, look in the log around startup for an authentication error on the channel. -## Debug Logging +## Turn on debug logging -Still stuck? Turn on debug logging in `~/.netclaw/config/netclaw.json`: +Still stuck? Set the log level in `~/.netclaw/config/netclaw.json`: ```json { @@ -280,60 +300,48 @@ Still stuck? Turn on debug logging in `~/.netclaw/config/netclaw.json`: } ``` -Restart the daemon, trigger the failing interaction, and look for these patterns: +The daemon restarts on the edit. Reproduce the problem and read the daemon log. A healthy Slack message flow looks like this: -**Healthy message flow (Slack):** -1. `Routing Slack event {id} to conversation {channelId}` — [SlackGatewayActor, Debug] -2. `turn_routed event={EventId} hasFiles=... textChars=...` — [SlackConversationActor] -3. `turn_received textChars=... fileCount=...` — [SlackThreadBindingActor] -4. `turn_enqueued contentItems=...` — [SlackThreadBindingActor] -5. `Posted Slack reply message` — [SlackThreadBindingActor] +1. `Routing Slack event {id} to conversation {channelId}` (Debug) +2. `turn_routed event=... hasFiles=... textChars=...` +3. `turn_received textChars=... fileCount=...` +4. `turn_enqueued contentItems=...` +5. `Posted Slack reply message` -If step 1 never appears, the message was dropped before routing — check ACL config (`channel_not_allowed`, `user_not_allowed`). If step 1 appears but step 2 does not, the routing policy filtered it (look for `event_filtered reason=routing_policy_ignore`). If step 4 appears but step 5 does not, the reply couldn't be posted — check the error tables above. +If step 1 never appears, the event was dropped before routing, so check the ACL reasons above. If step 1 appears but not step 2, the routing policy filtered it. If step 4 appears but not step 5, the reply couldn't be posted, so check [The reply never appears](#the-reply-never-appears). -**Quick triage commands:** +To see whether sessions are being written at all: ```bash -# Daemon process status -netclaw daemon status - -# Recent crash logs -ls -lt ~/.netclaw/logs/crash-*.log - -# Recent session activity (SQLite) sqlite3 ~/.netclaw/netclaw.db \ "SELECT persistence_id, MAX(created), MAX(sequence_number) \ FROM journal GROUP BY persistence_id \ ORDER BY MAX(created) DESC LIMIT 10;" ``` -## OTLP Diagnostic Patterns +## Read the metrics -If [OpenTelemetry](/observability/opentelemetry/) is enabled, channel metrics tell the story faster than logs: +If [OpenTelemetry](/observability/opentelemetry/) is on, channel metrics are faster than logs. They use the namespaces `netclaw.channel.slack.*`, `netclaw.channel.discord.*` and `netclaw.channel.mattermost.*`. -| Pattern | Metrics | Likely Cause | +| Pattern | Metrics | Likely cause | |---------|---------|--------------| -| Messages received, none routed | `events.received > 0`, `events.routed = 0` | ACL blocking all traffic | -| Messages routed, no replies | `messages.enqueued > 0`, `replies.posted = 0` | Outbound delivery failure | -| High drop rate | `events.dropped` spiking with `reason=channel_not_allowed` | Channel not in allow list | -| Suspected loop | High `events.filtered{reason="bot_message"}` + high `events.received` | Bot message filter working, but volume suggests upstream issue | - -Channel metrics use the namespace `netclaw.channel.slack.*`, `netclaw.channel.discord.*`, and `netclaw.channel.mattermost.*`. - -## Related Pages - -- [`netclaw doctor`](/cli/doctor/) — offline diagnostics -- [`netclaw status`](/cli/status/) — live connector health and message counters -- [`netclaw secrets`](/cli/secrets/) — manage encrypted tokens -- [`netclaw config`](/cli/config/) — Channels -- [OpenTelemetry](/observability/opentelemetry/) — OTLP metrics reference - -## External Resources - -- [Slack API: Bot tokens and permissions](https://api.slack.com/authentication/token-types#bot) — token types and required scopes -- [Slack API: Socket Mode](https://api.slack.com/apis/socket-mode) — how Socket Mode connections work -- [Slack: Finding Channel and User IDs](https://slack.com/help/articles/221769328) — locate IDs for ACL config -- [Discord Developer Docs: Getting Started](https://discord.com/developers/docs/getting-started) — bot setup and permissions -- [Discord: Finding IDs](https://support.discord.com/hc/en-us/articles/206346498) — enable Developer Mode to copy IDs -- [Slack Status](https://status.slack.com/) — check for Slack API outages -- [Discord Status](https://discordstatus.com/) — check for Discord API outages +| Messages received, none routed | `events.received > 0`, `events.routed = 0` | ACL is blocking everything | +| Messages routed, no replies | `messages.enqueued > 0`, `replies.posted = 0` | Outbound delivery is failing | +| Many drops with one reason | `events.dropped` rising with `reason=channel_not_allowed` | Channel isn't in the allow-list | +| Suspected loop | High `events.filtered{reason="bot_message"}` and high `events.received` | The bot-message filter is working, so look upstream | + +## Related pages + +- [Slack](/channels/slack/), [Discord](/channels/discord/) and [Mattermost](/channels/mattermost/) - setup and behavior for each channel +- [`netclaw status`](/cli/status/) - live connector health and counters +- [`netclaw doctor`](/cli/doctor/) - offline diagnostics +- [`netclaw secrets`](/cli/secrets/) - replace a stored token +- [OpenTelemetry](/observability/opentelemetry/) - the metrics reference + +## Resources + +- [Slack API: token types](https://api.slack.com/authentication/token-types#bot) - bot tokens and required scopes +- [Slack: Socket Mode](https://docs.slack.dev/apis/events-api/using-socket-mode) - how Socket Mode connections work +- [Discord: getting started](https://discord.com/developers/docs/getting-started) - bot setup and permissions +- [Mattermost: bot accounts](https://developers.mattermost.com/integrate/reference/bot-accounts/) - tokens and permissions +- [Slack Status](https://status.slack.com/) and [Discord Status](https://discordstatus.com/) - check for outages diff --git a/src/content/docs/deployment/docker.md b/src/content/docs/deployment/docker.md index a8b496f..9180200 100644 --- a/src/content/docs/deployment/docker.md +++ b/src/content/docs/deployment/docker.md @@ -1,88 +1,117 @@ --- -title: "Docker Deployment" -description: "Run the netclaw daemon in a Docker container with persistent config, health checks, and zero-downtime upgrades." +title: "Run netclaw in Docker" +description: "Run the netclaw daemon in a container with a persistent volume, health checks, a clean restart path, and an optional hardened profile." --- -The netclaw Docker image runs the daemon (`netclawd`) in a supervised container as the non-root user `netclaw` (UID 1654). The daemon binds loopback inside the container, so you drive it with `docker exec`. Reaching it from the host or the network is a non-local exposure mode — see [Exposure Modes](/deployment/exposure-modes/). +The netclaw image runs the daemon (`netclawd`) as the non-root user `netclaw` (UID 1654). PID 1 is [`tini`](https://github.com/krallin/tini), which starts an entrypoint script that supervises the daemon and restarts it when it exits. The daemon binds loopback inside the container, so you drive it with `docker exec`. To reach it from the host or another container, use a non-local mode from [Exposure modes](/deployment/exposure-modes/). ## Before you begin -- [Docker Engine](https://docs.docker.com/engine/install/) 20.10+ or Docker Desktop -- A provider API key (OpenRouter, Anthropic, OpenAI, etc.) or a reachable [Ollama](https://ollama.com/) instance -- Either provider/model config to pass as `NETCLAW_*` env vars, or an initialized config you run with [`netclaw init`](/cli/init/) inside the container +- [Docker Engine](https://docs.docker.com/engine/install/) with Compose v2. These steps were run on Docker 29. +- At least 1 GiB of memory for the container. See [Memory and first start](#memory-and-first-start). +- Outbound HTTPS to `huggingface.co` on first start, unless you [turn embeddings off](#memory-and-first-start). +- A model provider: an API key for a hosted one, or a reachable [Ollama](https://ollama.com/) or other OpenAI-compatible endpoint. :::caution -Don't add `-e NETCLAW_Daemon__Host=0.0.0.0`. The default `local` exposure mode binds loopback and **rejects** a non-loopback host — the daemon aborts on startup if you force `0.0.0.0` without switching exposure modes. To reach the daemon from outside the container, use a non-local exposure mode (see [Exposure Modes](/deployment/exposure-modes/)), not a raw bind override. +Don't set `NETCLAW_Daemon__Host=0.0.0.0` to "open up" the container. The default `local` mode rejects a non-loopback host and the daemon aborts at startup. Pick an exposure mode instead, which also sets up authentication for remote callers. ::: ## Quick start -Use a named volume — Docker creates it with the right ownership for the non-root `netclaw` user. (Bind-mounting a host directory works too, but you must `chown` it to UID 1654 first, or the daemon can't write to it.) +One named volume holds all state. Docker creates it owned by UID 1654, so there's nothing to `chown`. ```bash docker run -d \ --name netclaw \ + --stop-timeout 60 \ -v netclaw-home:/home/netclaw/.netclaw \ -e NETCLAW_Providers__openrouter__Type=openrouter \ - -e NETCLAW_Providers__openrouter__ApiKey=sk-or-v1-... \ + -e NETCLAW_Providers__openrouter__ApiKey="$OPENROUTER_API_KEY" \ -e NETCLAW_Models__Definitions__claude__Provider=openrouter \ -e NETCLAW_Models__Definitions__claude__ModelId=anthropic/claude-sonnet-4 \ -e NETCLAW_Models__Roles__Main=claude \ - ghcr.io/netclaw-dev/netclaw + ghcr.io/netclaw-dev/netclaw:0.28.2 ``` -The volume persists identity, config, credentials, session state, and logs across restarts. Self-update is disabled in the image — the image tag is the version, though update-availability checks still run so you'll know when a new release exists. - -**Tags:** `:latest` tracks the most recent release. Pin to a version tag (e.g., `:1.2.3`) in production. - -Verify the daemon is healthy: +The container reaches `healthy` in about ten seconds. Check it: ```bash -docker exec netclaw netclaw status +docker inspect --format '{{.State.Health.Status}}' netclaw +docker exec netclaw netclaw daemon status ``` -The CLI inside the container talks to the daemon over loopback, so no endpoint config is needed. To run the CLI from your host instead, point a daemon at a non-local exposure mode — see [Exposure Modes](/deployment/exposure-modes/). +![Terminal showing docker ps for the netclaw container Up and healthy on 5199/tcp, docker inspect printing healthy, and netclaw daemon status reporting the daemon running](/screenshots/output/docker-health.png) -### First run with Docker only +A few seconds after start, Docker reports `(healthy)` and the daemon answers from inside the container. -If you don't have `netclaw init` on the host, run it inside the container: +`docker logs` shows only the entrypoint's own lines. The daemon writes to `logs/daemon-YYYY-MM-DD.log` inside the volume, and continues in `daemon-YYYY-MM-DD.1.log`, `.2.log` and so on once a file reaches 10 MiB. `tail -f ~/.netclaw/logs/daemon-*.log` follows all of them. To stream daemon logs to `docker logs` too, add `-e NETCLAW_Logging__Console__Enabled=true`. -```bash -docker exec -it netclaw netclaw init +Env vars override both `netclaw.json` and `secrets.json`, so a value set with `-e` can't be changed later with `netclaw config`. Double underscores separate path segments, following the [.NET configuration convention](https://learn.microsoft.com/en-us/dotnet/core/extensions/configuration-providers#environment-variable-configuration-provider). The `ModelId` must match an identifier the provider serves. See [Models](/configuration/models/). + +**Tags:** `0.28.2` is one exact release. `0.28` floats to the newest 0.28 patch, `latest` to the newest stable release, and `beta` to the newest prerelease. Pin an exact tag in production and read the [upgrade guide](/guides/upgrading/) before you move it. + +### Use the CLI: put the config in a file + +:::caution +A container configured only with `NETCLAW_*` variables has no `config/netclaw.json`. The daemon runs fine, but `netclaw status` and `netclaw chat -p` print `daemon not configured - please run netclaw init` and exit, because the CLI checks for the file. +::: + +Write the config as a file instead. Create the directory on the host with a `config/netclaw.json`: + +```json +{ + "configVersion": 1, + "Security": { + "DeploymentPosture": "Personal", + "ShellExecutionMode": "HostAllowed", + "StrictDefaults": true + }, + "Tools": { "ShellMode": "HostAllowed" }, + "Providers": { + "ollama": { "Type": "ollama", "Endpoint": "http://llm.example.internal:11434" } + }, + "Models": { + "Definitions": { "main": { "Provider": "ollama", "ModelId": "qwen3:30b" } }, + "Roles": { "Main": "main" } + } +} ``` -The interactive wizard works the same way over `docker exec -it`. +Point `Endpoint` at your own server. Mount the directory as the data directory. The entrypoint changes its ownership to UID 1654 on start: -If you choose a non-local exposure mode during first-run setup, netclaw now seeds a one-shot local bootstrap credential before the first successful non-local daemon start. That prevents first boot from getting stuck waiting on pairing. +```bash +docker run -d --name netclaw --stop-timeout 60 \ + -v "$PWD/netclaw-data:/home/netclaw/.netclaw" \ + ghcr.io/netclaw-dev/netclaw:0.28.2 -## Configuration via environment variables +docker exec netclaw netclaw status +docker exec netclaw netclaw chat -p "Reply with the single word pong." +``` -Pass provider credentials and model config as `NETCLAW_`-prefixed environment variables. Double underscores separate path segments, following the [.NET configuration convention](https://learn.microsoft.com/en-us/dotnet/core/extensions/configuration-providers#environment-variable-configuration-provider). Env vars take highest priority, overriding both `netclaw.json` and `secrets.json`. +For a scripted setup, `docker exec netclaw netclaw provider add` and `netclaw model set` create the file without the wizard. Or run the wizard in a container that's already up: ```bash -docker run -d \ - --name netclaw \ - -v netclaw-home:/home/netclaw/.netclaw \ - -e NETCLAW_Providers__openrouter__Type=openrouter \ - -e NETCLAW_Providers__openrouter__ApiKey=sk-or-v1-... \ - -e NETCLAW_Models__Definitions__claude__Provider=openrouter \ - -e NETCLAW_Models__Definitions__claude__ModelId=anthropic/claude-sonnet-4 \ - -e NETCLAW_Models__Roles__Main=claude \ - ghcr.io/netclaw-dev/netclaw +docker exec -it netclaw netclaw init ``` -Keep secrets out of config files — inject them at runtime. The `ModelId` value must match a valid identifier from the provider API; see [Models](/configuration/models/) for the full schema and available model IDs. +Without `Security.DeploymentPosture` the daemon uses the strict Public posture, and `shell_execute` is denied outright. If you stay env-only and want the Personal posture `init` writes, add `NETCLAW_Security__DeploymentPosture=Personal` and `NETCLAW_Security__ShellExecutionMode=HostAllowed`. See [Hardening](/security/hardening/). + +Env list entries merge with file lists by position, so set a list such as `Tools.AudienceProfiles.Team.AllowedTools` in the file or in env vars, never both. An empty `NETCLAW_*` list variable next to a file value aborts startup: + +```text +Configuration key 'Tools.AudienceProfiles.Team.AllowedTools' has list items and also a value. One source sets items and another source sets a value. Remove one of them, for example the NETCLAW_* variable. +``` ## Docker Compose -For anything beyond quick testing, use Compose. This example pairs netclaw with a local Ollama instance on a private Compose network. The daemon stays in `local` mode (loopback); you reach it with `docker exec`, so there's no `Host` override and no published control-plane port: +Use Compose for anything you plan to keep. This file runs netclaw next to Ollama on a private network. There's no `Host` override and no published port, because you reach the daemon with `docker exec`: ```yaml services: netclaw: - image: ghcr.io/netclaw-dev/netclaw - container_name: netclaw + image: ghcr.io/netclaw-dev/netclaw:0.28.2 restart: unless-stopped + stop_grace_period: 60s depends_on: - ollama volumes: @@ -96,7 +125,7 @@ services: ollama: image: ollama/ollama:latest - container_name: ollama + restart: unless-stopped volumes: - ollama-data:/root/.ollama @@ -107,113 +136,138 @@ volumes: ```bash docker compose up -d +docker compose exec ollama ollama pull qwen3:30b ``` -The daemon comes up reporting `healthy`. Drive it with `docker exec netclaw netclaw status` (or `docker exec -it netclaw netclaw chat`). +The daemon reports `healthy` without the model, but it can't answer until Ollama has pulled it. `qwen3:30b` is a large download. Pick a smaller tag for a first test. -Pull the model into Ollama before netclaw can use it: +For a hosted provider, keep one service and swap the `environment` block for the `openrouter` variables from the quick start, reading the key from your shell with `${OPENROUTER_API_KEY}`. -```bash -docker exec ollama ollama pull qwen3:30b -``` +## Hardened profile -Netclaw references Ollama by service name (`http://ollama:11434`) since Compose puts both containers on the same network. +The image runs as non-root already. This profile adds a read-only root filesystem, no Linux capabilities, `no-new-privileges`, and a process limit: -`qwen3:30b` is a large model. For first-time testing on modest hardware, pick a smaller Ollama model. +```yaml +services: + netclaw: + image: ghcr.io/netclaw-dev/netclaw:0.28.2 + restart: unless-stopped + stop_grace_period: 60s + user: "1654:1654" + read_only: true + cap_drop: [ALL] + security_opt: + - no-new-privileges:true + pids_limit: 512 + tmpfs: + - /tmp:rw,noexec,nosuid,size=64m + - /var/netclaw-bundle:rw,exec,nosuid,size=128m,uid=1654,gid=1654 + volumes: + - netclaw-home:/home/netclaw/.netclaw + environment: + # The root filesystem is read-only, so unpack into the exec tmpfs above. + DOTNET_BUNDLE_EXTRACT_BASE_DIR: /var/netclaw-bundle + NETCLAW_Providers__openrouter__Type: openrouter + NETCLAW_Providers__openrouter__ApiKey: ${OPENROUTER_API_KEY} + NETCLAW_Models__Definitions__claude__Provider: openrouter + NETCLAW_Models__Definitions__claude__ModelId: anthropic/claude-sonnet-4 + NETCLAW_Models__Roles__Main: claude -### Docker socket access +volumes: + netclaw-home: +``` -If you want the agent to manage Docker containers as part of its tool use, mount the socket: +Two settings make it start, and each one fails without it. The Mattermost [Compose guide](/guides/mattermost-docker-compose/) uses the same recipe. -```bash --v /var/run/docker.sock:/var/run/docker.sock -``` +| Setting | Why | +|---------|-----| +| `user: "1654:1654"` | The entrypoint starts as root to repair ownership, then drops privileges. With `cap_drop: [ALL]` root can't write into a directory it doesn't own, so the entrypoint exits with `failed to create /home/netclaw/.netclaw`. Starting as 1654 skips that step. | +| `DOTNET_BUNDLE_EXTRACT_BASE_DIR` and the `/var/netclaw-bundle` tmpfs | The single-file daemon unpacks its native libraries at startup, about 43 MB under `/var/netclaw-bundle`. With a read-only root filesystem and no override it exits with `Failed to determine location for extracting embedded files`. The mount needs `exec`: a `noexec` mount fails with `Unable to load shared library 'e_sqlite3'` and `failed to map segment from shared object`. | -Add this to the `volumes` section of the netclaw service in your Compose file or to the `docker run` command. +The `/tmp` tmpfs isn't optional if the agent uses a shell or writes files. On Linux the daemon creates each run's private temp folder at `/tmp/netclaw-//p` (mode 0700) and doesn't fall back to another directory. With a read-only root and no writable `/tmp`, the container still starts and reports `healthy`, but every shell call fails with `Error preparing managed temporary directory: Read-only file system : '/tmp/netclaw-1654'`. The tmpfs is `noexec`, so a program the agent writes there can't run, and `size=64m` caps the folder. Don't point the extraction directory at it. -Treat this as host-level access. A process that can talk to the Docker socket can usually control the host. +We keep the unpack in its own small tmpfs because it's rebuilt on every start anyway. Nothing executable lands in the data volume, so backups stay clean, and nothing you'd miss lives in RAM. The 128m size leaves about three times the 43 MB it uses. This setup passed a health check and a chat turn with a shell call, which ran in `/tmp/netclaw-1654//p`. The same file without the `/tmp` line failed that shell call. -## Volume layout +Two other layouts also run: -Everything the daemon persists lives under `/home/netclaw/.netclaw`: +- Mount the whole home as an exec tmpfs, such as `/home/netclaw:exec,uid=1654,gid=1654,size=256m`, with the volume mounted inside it. Home becomes writable, so `~/.gitconfig` and `~/.local/bin` work until a restart wipes them. It's a larger mount, and it holds more than the unpack. +- Point `DOTNET_BUNDLE_EXTRACT_BASE_DIR` into the data volume, such as `/home/netclaw/.netclaw/.bundle`. It survives restarts, but it puts 43 MB of executables in the volume, which a backup picks up. The folder is safe to delete. -``` -/home/netclaw/.netclaw/ -├── client/config.json # CLI endpoint state -├── config/ -│ ├── netclaw.json # Daemon settings -│ └── secrets.json # Credentials -├── identity/ # Agent personality (SOUL.md, AGENTS.md, TOOLING.md) -├── sessions/ # Conversation history -├── keys/ # Key material -├── workspaces/ # Agent project workspaces (git repos with AGENTS.md) -├── environment/ -├── schedules/ -└── logs/ # crash-*.log, session logs -``` +The volume mounts at `/home/netclaw/.netclaw` and stays writable. Everything else under `/home/netclaw` is read-only, so a tool that writes to `~/.cache`, `~/.gitconfig` or `~/.local` fails. With a bind mount instead of a named volume, `chown 1654:1654` the host directory first, because nothing in this profile runs as root to do it. -Back up this volume before upgrades. `config/` and `identity/` are the critical directories; everything else can be regenerated. If you use a bind mount, copy `~/.netclaw` directly. If you use a named volume, export it with your normal Docker volume backup process before upgrades. +If you'd rather keep the entrypoint's ownership repair, leave out `user:` and add `CHOWN`, `DAC_OVERRIDE`, `FOWNER`, `SETUID`, `SETGID` and `KILL` back with `cap_add`. Without `KILL`, root can't signal the daemon after it drops to UID 1654. `docker stop` then exits `1` with `Unexpected error when forwarding signal` and the daemon never gets to drain. -## Health checks +## Reaching the daemon -The image has a built-in health check that polls every 15 seconds with a 30-second startup grace period: +`docker exec netclaw netclaw ` is the default path. The CLI inside the container talks to the daemon over loopback, and a root `exec` re-runs itself as `netclaw`. +A published port won't work in `local` mode, because the daemon only listens on the container's loopback. From another container, `curl http://netclaw:5199/api/health/ready` gets `connection refused`. Inside the container, call the daemon as `http://127.0.0.1:5199` or `http://localhost:5199`. In `local` mode the daemon answers `/api/health/status` with `401` when the `Host` header is any other name, such as `netclaw:5199`, or when an `Origin` header doesn't match it. `/api/health/ready` stays anonymous either way. For a host CLI, a sibling container, or a reverse proxy, set an exposure mode. A same-network reverse proxy needs these variables, where `TrustedProxies` is your proxy's address or the Docker network's subnet: + +```bash +-e NETCLAW_Daemon__ExposureMode=reverse-proxy \ +-e NETCLAW_Daemon__Host=0.0.0.0 \ +-e NETCLAW_Daemon__TrustedProxies__0=172.16.0.0/12 ``` -HEALTHCHECK --interval=15s --timeout=5s --start-period=30s --retries=3 - CMD curl -sf http://127.0.0.1:5199/api/health/ready || exit 1 -``` -| Endpoint | Auth | Purpose | -|----------|------|---------| -| `GET /api/health/ready` | None | Returns `200 OK` when the daemon is ready. Use this for orchestrators and load balancers. | -| `GET /api/health/status` | Required | Returns detailed runtime status. | +That container reached `healthy`, answered `/api/health/ready` to another container, and returned `401` for `/api/health/status` without a device token. Tunnel modes (`tailscale-serve`, `cloudflare-tunnel`) need the tunnel agent in the same network namespace and `NETCLAW_Daemon__SkipTunnelProcessCheck=true`. See [Exposure modes](/deployment/exposure-modes/). + +## Memory and first start + +Embeddings are on by default since 0.27.0. On first start the daemon downloads two ONNX models from `huggingface.co` (about 129 MB) into `models/` inside the volume, then loads them. + +| Setup | Observed | +|-------|----------| +| Embeddings on, idle | About 485 MiB resident, flat | +| 512 MiB limit | OOM-killed over and over. In three minutes the entrypoint restarted the daemon six times | +| 768 MiB limit | Stable at about 555 MiB | +| `Memory.Embeddings.Enabled=false` | About 300 MiB for the first minute and a half, then 220 to 225 MiB. Nothing downloaded, and `/api/health/status` reports embeddings `disabled`. A 512 MiB limit is stable | -Check container health from the host: +An OOM kill doesn't stop the container. The entrypoint restarts the daemon, so `docker ps` keeps showing the container as running while the health check flaps. Set a 1 GiB limit or more. To turn embeddings off: ```bash -docker inspect --format='{{.State.Health.Status}}' netclaw +-e NETCLAW_Memory__Embeddings__Enabled=false ``` +Without egress, copy the model files into `models/` and set `NETCLAW_Memory__Embeddings__AutoDownload=false`, or turn embeddings off. See [Memory and embeddings](/configuration/memory/). The daemon starts either way. It raises a `memory.embedding_model.unavailable` [alert](/observability/operational-alerts/) and falls back to lexical recall. + ## Process supervision -The entrypoint script is a PID 1 supervisor. If the daemon exits (config-update restart, crash, `netclaw init` wizard completion), the entrypoint waits 2 seconds and restarts it. The container stays alive, so `docker exec` sessions survive daemon restarts. +The entrypoint restarts the daemon every time it exits, so a config-driven restart never stops the container. A daemon that dies within 30 seconds of starting is restarted after 2, 4, 8, 16, 32, then 60 seconds. A daemon that runs longer than 30 seconds resets the delay. -That also helps with pairing: if the daemon is running in a container and you need a daemon-host bootstrap or recovery path, `docker exec netclaw netclaw daemon pair` works the same way as running it on a bare-metal daemon host. -`docker stop` sends SIGTERM, which the entrypoint forwards to the daemon for a clean shutdown. Docker's default 10-second stop timeout is plenty. +- Valid edits to `netclaw.json` restart the daemon in-process, including `Daemon.Host`, `Daemon.Port` and `Daemon.ExposureMode`. +- Changes to `NETCLAW_*` variables, `secrets.json`, or the image need `docker restart netclaw`. +- Inside the image `netclaw daemon start` never launches a second daemon. With the daemon up it prints `Daemon managed by container supervisor (PID n).` and exits `0`. With it down it says the supervisor owns startup and exits `1`. `netclaw daemon stop` prints `The container supervisor will restart the daemon.` and the entrypoint brings it back two seconds later. There is no `netclaw daemon restart`, so use `docker restart`. +- A restart or `docker stop` stops a session's tool loop at a safe point and posts one notice in its channel ("restarting", or "stopping" for `netclaw daemon stop`). A running sub-agent is parked and resumes after the restart, which can add about 18 seconds. The `daemon.stopping` [alert](/observability/operational-alerts/) names sessions that didn't acknowledge in `timedOutSessionIds`. +- `docker stop` forwards SIGTERM. The daemon needs up to 30 seconds to drain sessions and webhook alerts, so set `--stop-timeout 60` or `stop_grace_period: 60s`. Docker's 10 second default can kill a daemon with a turn in flight. An idle container stops in under a second. -## Upgrading +## Health checks -Self-update is disabled in the image (`NETCLAW_Daemon__DisableSelfUpdate=true`), so upgrades mean pulling a new image. Schema migrations are forward-only with no automatic rollback. +The image defines this check, and Docker reports it as `healthy`, `unhealthy` or `starting`: -Model configuration is the one exception. On 0.25.0, the first model write converts `Models` to named definitions and roles and keeps a restorable snapshot — see [Migrating model configuration](/guides/migrating-model-config/), especially if you set models through `NETCLAW_Models__*` variables. +```dockerfile +HEALTHCHECK --interval=15s --timeout=5s --start-period=30s --retries=3 \ + CMD netclaw daemon status >/dev/null || exit 1 +``` -```bash -# Pull the new version -docker pull ghcr.io/netclaw-dev/netclaw:latest +`netclaw daemon status` exits `1` when the daemon isn't running, when `/api/health/ready` doesn't answer, or when the PID in `netclaw.pid` isn't shown to serve this home. The image bakes `NETCLAW_HOME`, so the home check passes. It finds the endpoint from `netclaw.json`, `NETCLAW_*` variables or the default, so a container with a custom `Daemon.Port` stays `healthy`. If it can't read the config, it won't guess an address. See [Health checks](/observability/health-checks/) for the endpoints and response fields. -# Stop the old container (volume stays) -docker stop netclaw && docker rm netclaw +A container with no provider configured still reports `healthy`, because the daemon is up. `netclaw status` shows `degraded` and chat turns return a configuration banner. A container whose `Models` or provider config is invalid doesn't report `healthy`: the daemon exits at startup. A running daemon whose `netclaw.json` becomes unreadable keeps serving its old config, but the check prints `The local daemon address is unknown. No readiness probe was sent. Repair the local configuration and retry.` and exits `1`, so the container turns `unhealthy` after three failed checks. See [the troubleshooting entry](#container-keeps-restarting-with-error-modelsroles). -# Start with the new image (same flags you launched it with) -docker run -d \ - --name netclaw \ - -v netclaw-home:/home/netclaw/.netclaw \ - ghcr.io/netclaw-dev/netclaw:latest +## Volume layout and backups -# Wait for readiness -until [ "$(docker inspect --format='{{.State.Health.Status}}' netclaw)" = "healthy" ]; do sleep 2; done -echo "Daemon is ready" -``` +State lives in `/home/netclaw/.netclaw`. See [Data directory layout](/reference/data-directory/) for every folder. The daemon deletes its own daemon and crash logs, numbered segments included, after 14 days, and [Data retention](/configuration/data-retention/) covers what else grows and how to change that. Stop the container, copy the whole directory (including `keys/`, without which `secrets.json` can't be read), then start it again. [Back up, restore and move netclaw](/guides/backup-and-restore/) has the commands and a tested round trip. + +## Upgrading -With Compose: +Self-update is off in the image (`NETCLAW_Daemon__DisableSelfUpdate=true`). The tag is the version. When a newer release exists, `netclaw update` inside a container reports it and exits `1` with `Pull a newer container image to upgrade.` On the current release it prints `Netclaw is up to date` and exits `0`. ```bash docker compose pull docker compose up -d ``` -To rollback, stop the container and start with the previous image tag. If the new version already ran a schema migration, restore the volume from backup to roll back. +Back up first and read [Upgrade netclaw](/guides/upgrading/). A newer image migrates the database on start and can't be undone by switching the tag back. ## Image details @@ -224,49 +278,99 @@ To rollback, stop the container and start with the previous image tag. If the ne | Architectures | `linux/amd64`, `linux/arm64` | | Port | `5199` | | Volume | `/home/netclaw/.netclaw` | +| Baked environment | `NETCLAW_HOME`, `NETCLAW_DAEMON_PATH`, `NETCLAW_Daemon__DisableSelfUpdate=true`, `NETCLAW_CONTAINER_SUPERVISOR=entrypoint` | | License | Apache-2.0 | -Built on Ubuntu 24.04 (not a minimal runtime), the image ships with `git`, `jq`, `sqlite3`, `python3`, `curl`, `wget`, `gh`, and more. Operators can `apt-get install` additional tools if the agent needs them. +The image ships `git`, `gh`, `jq`, `sqlite3`, `python3`, `curl` and `wget`. The daemon runs as a normal user, so the agent can't `apt-get install`. It can install into `~/.local/bin`, `~/.dotnet` and `~/.dotnet/tools`, which are on `PATH`, and a mounted `/tools/bin` is on `PATH` too. For system packages, run `docker exec -u root netclaw apt-get install ...` or build a derived image. + +### Docker socket access + +To let the agent manage containers, mount the socket with `-v /var/run/docker.sock:/var/run/docker.sock`. Anything that can talk to the socket can usually control the host, so treat this as host-level access. + +### Kubernetes + +The image needs one writable volume at `/home/netclaw/.netclaw` and one replica, because the daemon holds a lock file on the data directory. Kubernetes ignores the image's `HEALTHCHECK`. An `httpGet` probe fails in `local` mode, because the kubelet calls the pod IP and the daemon only listens on loopback. Use an `exec` probe that runs the image's own check, `netclaw daemon status`, which follows `Daemon.Port`. In `reverse-proxy` mode with `Daemon.Host` set to `0.0.0.0`, `httpGet` works. These probes haven't been run on a cluster, so test them before you rely on them. ## Troubleshooting -### CLI can't connect +### `daemon not configured - please run netclaw init` -Run the CLI inside the container, where it reaches the daemon over loopback: +**Symptom:** `docker exec netclaw netclaw status` or `chat -p` prints this, but the container is healthy. -```bash -docker exec netclaw netclaw status -``` +**Cause:** The container is configured with `NETCLAW_*` variables only, so there's no `config/netclaw.json`. + +**Fix:** [Put the config in a file](#use-the-cli-put-the-config-in-a-file) or run `docker exec -it netclaw netclaw init`. + +### Container is `unhealthy` and `docker logs` shows `Aborted (core dumped)` + +**Symptom:** The container flaps between `starting` and `unhealthy`. `docker logs` shows a `Fatal error` line with a crash log path, the exception, then `Aborted (core dumped)` and `restarting in 60s`. -A host CLI (or `curl`) hitting a published port won't reach a `local`-mode daemon — it binds loopback inside the container, and `local` mode refuses to bind `0.0.0.0`. For host or network access, configure a non-local exposure mode (see [Exposure Modes](/deployment/exposure-modes/)); reverse-proxy mode additionally needs `Daemon.TrustedProxies` set to the proxy's address or CIDR. +**Cause:** The daemon failed during startup and the entrypoint keeps restarting it. The abort is how the daemon reports a fatal configuration error. The restart delay climbs to 60 seconds. -### Container keeps restarting (crash loop) +**Fix:** Read the first lines of `docker logs netclaw` for the exception, or the newest `logs/crash-*.log` in the volume. Common causes: -The entrypoint restarts the daemon on every exit by design, so a misconfiguration shows up as a restart loop. Check `docker logs netclaw` for the fatal error. The most common one: +| Message | Cause | +|---------|-------| +| `Invalid local topology: Daemon.Host '0.0.0.0' is not a loopback address.` | A host override in `local` mode. | +| `Tunnel prerequisite not met: ExposureMode='tailscale-serve' requires 'tailscaled'` | A tunnel mode without the tunnel process. Set `SkipTunnelProcessCheck` for a sidecar. | +| `The key {...} was not found in the key ring` | `secrets.json` restored without its `keys/` directory. See [Back up, restore and move netclaw](/guides/backup-and-restore/). | +| `has list items and also a value` | A list set in both a variable and the file. | +### Container keeps restarting with `error: Models:Roles...` + +**Symptom:** `docker logs` repeats one line, then `netclawd exited (code=1) after 0s, restarting in 2s...` with a delay that doubles to 60 seconds. There's no stack trace, no `Aborted (core dumped)` and no crash log. + +```text +error: Models:Roles:Main references unknown definition 'nope'. Defined: stub-stub-model. Fix the name, or run `netclaw model set main `. ``` -Invalid local topology: Daemon.Host '0.0.0.0' is not a loopback address. + +**Cause:** `Models` or a provider entry is invalid, and the daemon refuses to start with it. A running daemon behaves differently. It keeps its previous configuration and shows `config on disk not applied` in [`netclaw status`](/cli/status/). + +**Fix:** Correct the `Models` section in `netclaw.json`, or the `NETCLAW_Models__*` variable the message names. The next restart picks it up. + +### Container keeps restarting with `error: Couldn't read netclaw.json` + +**Symptom:** The same loop as above, with this line: + +```text +error: Couldn't read netclaw.json: A duplicate key 'configVersion' was found. ``` -That means `NETCLAW_Daemon__Host=0.0.0.0` was set while the exposure mode is still `local` — remove the host override (the default loopback bind is correct) or switch to a non-local exposure mode. Other culprits: missing provider config, an invalid API key, or a required `netclaw.json` field. +**Cause:** `netclaw.json` or `secrets.json` is empty, cut off, or has a duplicate key. The container turns `unhealthy` after a minute. `docker exec` still works, but `netclaw status` and `netclaw chat` print the same line. If the daemon was already running when the file broke, it keeps its old config and `netclaw daemon status` exits `1` with `The local daemon address is unknown`. + +**Fix:** Repair the file in the volume, or `docker cp` a good copy in. The next restart picks it up. See [`Couldn't read netclaw.json`](/troubleshooting/common-problems/#couldnt-read-netclawjson). + +### Exit code 137, `Killed` or a restarting daemon with a healthy-looking container + +**Cause:** The memory limit is too low for the embedding models. `docker inspect --format '{{.State.OOMKilled}}' netclaw` prints `true`, and the container stays `running` because the entrypoint restarts the daemon. + +**Fix:** Raise the limit to 1 GiB or more, or turn embeddings off. + +### Permission denied on a bind mount + +**Cause:** The entrypoint couldn't take ownership. The mount is read-only, or the container runs with `user: "1654:1654"` against a directory owned by someone else. + +**Fix:** Make the mount writable, or `chown -R 1654:1654` the host directory. + +### Host CLI or `curl` can't connect -### Health check failing +**Cause:** `local` mode listens on the container's loopback only. -The 30-second start period gives the daemon time to initialize. If it's still unhealthy after that, the daemon isn't starting. Check logs and run `netclaw doctor` from the host, or `docker exec netclaw netclaw doctor` inside the container. +**Fix:** Run the CLI with `docker exec`, or set an [exposure mode](/deployment/exposure-modes/). ## Related pages -- [Models](/configuration/models/) — model slot configuration -- [Exposure Modes](/deployment/exposure-modes/) — remote access via reverse proxy, Tailscale, or Cloudflare Tunnel -- [systemd Service](/deployment/systemd/) — bare-metal Linux alternative -- [OpenTelemetry](/observability/opentelemetry/) — daemon metrics and log export +- [Upgrade netclaw](/guides/upgrading/) - what changes between releases and how to roll back +- [Back up, restore and move netclaw](/guides/backup-and-restore/) - copy the volume and prove it restores +- [Exposure modes](/deployment/exposure-modes/) - reach the daemon from outside the container +- [Health checks](/observability/health-checks/) - endpoints and response fields +- [Data directory layout](/reference/data-directory/) - what's in the volume ## Resources -- [Docker Engine installation guide](https://docs.docker.com/engine/install/) — platform-specific install instructions -- [Docker Compose documentation](https://docs.docker.com/compose/) — multi-container orchestration -- [Docker security guide](https://docs.docker.com/engine/security/) — baseline guidance for socket mounts and daemon exposure -- [GitHub Container Registry (GHCR)](https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-container-registry) — pulling and authenticating with ghcr.io -- [.NET environment variable configuration](https://learn.microsoft.com/en-us/dotnet/core/extensions/configuration-providers#environment-variable-configuration-provider) — the double-underscore nesting convention -- [Docker exec reference](https://docs.docker.com/reference/cli/docker/container/exec/) — run CLI commands inside a live container -- [Ollama model library](https://ollama.com/library) — browse available models for local inference +- [Docker Engine installation guide](https://docs.docker.com/engine/install/) - platform-specific install instructions +- [Docker Compose documentation](https://docs.docker.com/compose/) - multi-container orchestration +- [Docker security guide](https://docs.docker.com/engine/security/) - capabilities, `no-new-privileges` and socket exposure +- [`docker run` reference](https://docs.docker.com/reference/cli/docker/container/run/) - `--read-only`, `--tmpfs`, `--cap-drop` and `--stop-timeout` +- [.NET environment variable configuration](https://learn.microsoft.com/en-us/dotnet/core/extensions/configuration-providers#environment-variable-configuration-provider) - the double-underscore nesting convention +- [Ollama model library](https://ollama.com/library) - available models for local inference diff --git a/src/content/docs/deployment/exposure-modes.md b/src/content/docs/deployment/exposure-modes.md index 05b6867..40835b9 100644 --- a/src/content/docs/deployment/exposure-modes.md +++ b/src/content/docs/deployment/exposure-modes.md @@ -1,9 +1,9 @@ --- -title: Exposure Modes -description: Configure how netclaw is exposed to the network — local, reverse proxy, Tailscale Serve, Tailscale Funnel, or Cloudflare Tunnel. +title: Exposure modes +description: "Choose how the netclaw daemon is reachable over the network: local, reverse proxy, Tailscale Serve, Tailscale Funnel or Cloudflare Tunnel." --- -Exposure mode controls how the daemon is reachable over the network. Most setups use local mode. You only need a non-local mode when something external (GitHub webhooks, CI, a second machine, or a reverse proxy) needs to reach the daemon. +Exposure mode controls how the daemon is reachable over the network. Most setups use local mode. You only need a non-local mode when something external needs to reach the daemon: GitHub webhooks, CI, a second machine, a reverse proxy, or another container. ## Before you begin @@ -16,24 +16,34 @@ Exposure mode controls how the daemon is reachable over the network. Most setups | Mode | Config value | Required process | Reachability | Risk | |------|-------------|-----------------|-------------|------| -| **Local** | `local` | None | Loopback only | Lowest | -| **Reverse Proxy** | `reverse-proxy` | Your reverse proxy | Whatever the proxy exposes | Medium | -| **Tailscale Serve** | `tailscale-serve` | `tailscaled` | Your Tailscale network (called a [tailnet](https://tailscale.com/kb/1136/tailnet)), HTTPS | Low | -| **Tailscale Funnel** | `tailscale-funnel` | `tailscaled` | Public internet via Tailscale | High | -| **Cloudflare Tunnel** | `cloudflare-tunnel` | `cloudflared` | Public internet via Cloudflare | High | +| Local | `local` | None | Loopback only | Lowest | +| Reverse Proxy | `reverse-proxy` | Your reverse proxy | Whatever the proxy exposes | Medium | +| Tailscale Serve | `tailscale-serve` | `tailscaled` | Your Tailscale network (called a [tailnet](https://tailscale.com/kb/1136/tailnet)), HTTPS | Low | +| Tailscale Funnel | `tailscale-funnel` | `tailscaled` | Public internet via Tailscale | High | +| Cloudflare Tunnel | `cloudflare-tunnel` | `cloudflared` | Public internet via Cloudflare | High | -Configure the exposure mode via `netclaw config` — navigate to **Security & Access → Exposure Mode** — or set `Daemon.ExposureMode` directly in `netclaw.json`. +Configure the exposure mode via `netclaw config` - navigate to **Security & Access → Exposure Mode** - or set `Daemon.ExposureMode` directly in `netclaw.json`. -![Exposure Mode selection](/screenshots/output/config-exposure.png) +![The Exposure Mode screen with five options, Local checked and marked recommended, and a warning triangle on Tailscale Funnel and Cloudflare Tunnel](/screenshots/output/config-exposure.png) -The Exposure Mode picker in `netclaw config → Security & Access` — Local, Reverse Proxy, Tailscale Serve/Funnel, or Cloudflare Tunnel. +The Exposure Mode picker in `netclaw config → Security & Access`. The triangle marks the two modes that reach the public internet. Options marked with a warning triangle expose the daemon to the public internet. Tailscale Serve is the recommended remote mode: tailnet-only access, no public exposure. :::caution -In every non-local mode (reverse proxy and the three tunnel modes), a loopback connection is **not** trusted as the local operator — the loopback peer is the proxy or tunnel forwarding remote traffic, not a same-host process. You must pair a device or authenticate remotely, even from the daemon's own host, and pairing codes can't be minted from a loopback connection. Only `local` mode auto-trusts loopback. (This closes the SEC-005 tunnel-loopback auth bypass.) +In every non-local mode (reverse proxy and the three tunnel modes), a loopback connection is **not** trusted as the local operator. The loopback peer is the proxy or tunnel forwarding remote traffic, not a same-host process. Only `local` mode auto-trusts loopback, so in a non-local mode `curl http://127.0.0.1:5199/api/health/status` on the daemon host returns `401` until it carries a device token. `netclaw daemon pair` still works from the daemon host in every mode, because it proves it's local with a key-ring proof instead of a loopback connection. ::: +In `local` mode loopback is trusted only when the request looks local too. The `Host` header must be `localhost` or a loopback IP literal such as `127.0.0.1`, and an `Origin` header, if there is one, must be `http://` plus that same host. Anything else gets `401`: + +| Request | Result | +|---------|--------| +| `http://127.0.0.1:5199/api/health/status` or `http://localhost:5199/...` | `200` | +| Same call with `Host: netclaw:5199`, or any DNS name that resolves to loopback, `host.docker.internal` included | `401` | +| A browser page on another local port, such as `http://localhost:3000`, calling the daemon | `401` (the `Origin` doesn't match) | + +A web app should call the daemon from its server process, or pair with `netclaw daemon pair` and send the device token. Anonymous routes (`/api/health/ready`, webhooks, the Mattermost callback) answer regardless. + ### Reverse Proxy ```json @@ -50,10 +60,22 @@ Use this when nginx, Caddy, Traefik, HAProxy, or another reverse proxy is the pu In this mode: -- `Daemon.Host` must be a non-loopback internal address. `127.0.0.1`, `::1`, and `localhost` are rejected in `reverse-proxy` mode. +- `Daemon.Host` must be a non-loopback address: an internal IP, or `0.0.0.0` to listen on every interface. `127.0.0.1`, `::1` and `localhost` are rejected in `reverse-proxy` mode. - `TrustedProxies` must list the proxy's source IP or CIDR. Forwarded headers are only honored from those peers. -If the proxy runs on the same machine, the final hop into netclaw still needs to target a non-loopback internal IP. A same-host reverse proxy is fine, but the final hop into netclaw still cannot use loopback. +If the proxy runs on the same machine, the final hop into netclaw still needs to target a non-loopback address. A same-host reverse proxy is fine, but that hop can't use loopback. The local CLI maps `0.0.0.0` back to `127.0.0.1` on its own. + +In Docker the container's address changes on every start, so bind `0.0.0.0` and trust the network that holds your proxy: + +```bash +docker run -d --name netclaw --network proxy-net \ + -e NETCLAW_Daemon__ExposureMode=reverse-proxy \ + -e NETCLAW_Daemon__Host=0.0.0.0 \ + -e NETCLAW_Daemon__TrustedProxies__0=172.16.0.0/12 \ + ghcr.io/netclaw-dev/netclaw:0.28.2 +``` + +From another container on that network, `/api/health/ready` returns `200` with no credentials, and `/api/health/status` returns `401` without a device token. ## Configuration @@ -77,7 +99,7 @@ No `Daemon` section needed. The daemon binds to `127.0.0.1:5199` and is only rea } ``` -Tailscale Serve creates an HTTPS endpoint on your tailnet that proxies to the daemon's local port. Only devices on your tailnet can reach it. Then run [`tailscale serve`](https://tailscale.com/kb/1242/tailscale-serve) to proxy your tailnet hostname to `127.0.0.1:5199`. +Tailscale Serve creates an HTTPS endpoint on your tailnet that proxies to the daemon's local port. Only devices on your tailnet can reach it. Then run [`tailscale serve`](https://tailscale.com/kb/1242/tailscale-serve) to proxy your tailnet hostname to `127.0.0.1:5199`. The daemon refuses to start unless `tailscaled` is running or `SkipTunnelProcessCheck` is set. ### Tailscale Funnel @@ -89,7 +111,7 @@ Tailscale Serve creates an HTTPS endpoint on your tailnet that proxies to the da } ``` -Funnel extends Serve to the public internet. Anyone with the URL can reach the daemon, though netclaw's device authentication still applies. Configure it with [`tailscale funnel`](https://tailscale.com/kb/1223/funnel). Only use Funnel when you need public internet access — Tailscale Serve covers everything else. +Funnel extends Serve to the public internet. Anyone with the URL can reach the daemon, though netclaw's device authentication still applies. Configure it with [`tailscale funnel`](https://tailscale.com/kb/1223/funnel). Only use Funnel when you need public internet access - Tailscale Serve covers everything else. ### Cloudflare Tunnel @@ -105,7 +127,9 @@ Routes traffic through Cloudflare's network to the daemon. Set up `cloudflared` If `cloudflared` runs outside the daemon's process namespace, set `Daemon.SkipTunnelProcessCheck` to `true` so startup validation skips the local process probe without skipping the rest of the auth checks. - +:::caution +A hostname behind a Cloudflare Access login also blocks callers that can't log in. Inbound [webhooks](/configuration/webhooks/), Mattermost button callbacks sent to a `CallbackUrl` on the tunnel hostname and a [paired remote CLI](/cli/daemon/) all get Cloudflare's login page instead of the daemon, because netclaw doesn't send Cloudflare Access credentials. If you need any of them, plan for that in your Access setup. +::: ### Full Daemon section @@ -127,9 +151,9 @@ If `cloudflared` runs outside the daemon's process namespace, set `Daemon.SkipTu } ``` -Case-insensitive — `reverse-proxy`, `ReverseProxy`, and `REVERSE-PROXY` all work, same as the tunnel modes. +Case-insensitive - `reverse-proxy`, `ReverseProxy`, and `REVERSE-PROXY` all work, same as the tunnel modes. -**Docker users:** if you switch to a tunnel mode, update your container's port binding to match the `Host` and `Port` values here. See [Docker Deployment](/deployment/docker/) for details. +**Docker users:** a published port doesn't help in a tunnel mode, because the daemon still binds loopback inside the container. Run the tunnel agent in the same network namespace (Compose `network_mode: "service:netclaw"`, or a sidecar in the same pod) and set `Daemon.SkipTunnelProcessCheck` to `true`, since the agent isn't in the daemon's process namespace. A container started this way reaches `healthy` and listens on `127.0.0.1:5199`. The image's health check runs `netclaw daemon status`, which follows `Daemon.Port`, so a custom port stays `healthy`. For a reverse proxy, use the `0.0.0.0` form above. See [Run netclaw in Docker](/deployment/docker/#reaching-the-daemon). ### Environment variables @@ -144,9 +168,9 @@ NETCLAW_Daemon__TrustedProxies__0=10.0.0.10 Double underscores separate path segments, following the [.NET configuration convention](https://learn.microsoft.com/en-us/dotnet/core/extensions/configuration-providers#environment-variable-configuration-provider). -### Restart required +### Restarts -`Host`, `Port`, and `ExposureMode` require a daemon restart — they aren't hot-reloaded. Other config changes trigger an automatic restart; the daemon drains active sessions and restarts itself. +Editing `Daemon.Host`, `Daemon.Port` or `Daemon.ExposureMode` in `netclaw.json` restarts the daemon in-process like any other config change: it drains active sessions, rebinds and comes back. Restart it yourself only when the value comes from an environment variable or the unit file. ```bash # systemd @@ -158,9 +182,7 @@ docker restart netclaw ## Inbound webhooks -Tunnel modes make [inbound webhooks](/configuration/webhooks/) possible. External services like GitHub or CI systems can trigger autonomous runs via HTTP POST. - -They do nothing in local mode. +[Inbound webhooks](/configuration/webhooks/) let external services like GitHub or CI trigger autonomous runs with an HTTP POST. Routes are served whenever `Webhooks.Enabled` is `true`, in every mode. In `local` mode only processes on the daemon host can reach `/api/webhooks/`, so a service on the internet needs a tunnel or a reverse proxy in front of it. ## Device pairing @@ -176,9 +198,11 @@ If the daemon is already running and you need to add another device, pair from t netclaw daemon pair ``` +The command prints a code. In a non-local mode the next line reads `netclaw pair `: replace the placeholder with the HTTPS address your proxy or tunnel publishes. + ## Startup validation -If the mode requires `tailscaled` or `cloudflared` and that process isn't running, you'll see this in the daemon logs (`~/.netclaw/logs/daemon.log` or `journalctl --user -u netclaw` for systemd): +If the mode requires `tailscaled` or `cloudflared` and that process isn't running, the daemon logs this to `~/.netclaw/logs/daemon-*.log`. `docker logs` shows the same message as an exception, starting `Unhandled exception. System.InvalidOperationException:` and without the `Daemon startup aborted:` prefix or the `Remediation:` label, just before `Aborted (core dumped)` and a restart: ``` Daemon startup aborted: Tunnel prerequisite not met: ExposureMode='tailscale-serve' requires 'tailscaled' to be running unless Daemon.SkipTunnelProcessCheck=true is explicitly set for a sidecar or host-managed tunnel topology. Remediation: Start 'tailscaled' locally, or set Daemon.SkipTunnelProcessCheck=true only when the tunnel is intentionally managed outside the Netclaw process namespace. @@ -190,7 +214,7 @@ If no paired devices exist and no alternative auth scheme is configured: Daemon startup aborted: No remote authentication available: ExposureMode='tailscale-serve' requires either at least one paired device or an alternative remote auth scheme. Remediation: Run 'netclaw daemon pair' to pair a device, or check your auth configuration before exposing the daemon remotely. ``` -Both are fatal. The daemon won't start until you fix the underlying issue. +Both are fatal. The daemon won't start until you fix the underlying issue. Under systemd it restarts every five seconds. In Docker the entrypoint retries with a delay that grows to 60 seconds. In `reverse-proxy` mode, startup also fails if the daemon is still bound to loopback or if `TrustedProxies` is empty or malformed. @@ -198,13 +222,15 @@ In `reverse-proxy` mode, startup also fails if the daemon is still bound to loop ### Start here: `netclaw doctor` -Run [`netclaw doctor`](/cli/doctor/) first — it has a dedicated `exposure-mode` check. +Run [`netclaw doctor`](/cli/doctor/) first. It has a dedicated `exposure-mode` check, and it's one of several checks beyond config validity. + +![netclaw doctor output: 24 checks, all PASS, ending with doctor exit code: 0](/screenshots/output/doctor.png) -![Netclaw doctor output showing health check diagnostics](/screenshots/output/doctor.png) +The `exposure-mode` line near the bottom reports `local` and the address it's bound to. ### Tunnel process not running -Startup aborted with "required tunnel process is not running." +**Symptom:** Startup aborted with `Tunnel prerequisite not met`. Start the required process first: @@ -228,7 +254,7 @@ Or switch to local mode if you don't need remote access: ### No paired devices -Startup aborted with "no paired devices exist and no alternative remote authentication scheme is configured." +**Symptom:** Startup aborted with `No remote authentication available`. If this is a brand-new non-local install, check whether the daemon actually completed its first successful start. The one-shot bootstrap only happens before that first successful non-local startup. @@ -240,13 +266,13 @@ netclaw daemon pair If the device store was lost after the first successful non-local start and the daemon will not start anymore, temporarily switch `Daemon.ExposureMode` to `local` or restore `devices.json` and `secrets.json` from backup. Then start the daemon, run `netclaw daemon pair`, and switch back. -You can also run `netclaw init` on an existing install — it shows an action menu with options to open the configuration editor, redo identity, or start over. +You can also run `netclaw init` on an existing install - it shows an action menu with options to open the configuration editor, redo identity, or start over. ### Reverse proxy bound to loopback -Startup aborted because `Daemon.ExposureMode` is `reverse-proxy` but `Daemon.Host` is `127.0.0.1`, `::1`, or `localhost`. +**Symptom:** Startup aborted because `Daemon.ExposureMode` is `reverse-proxy` but `Daemon.Host` is `127.0.0.1`, `::1` or `localhost`. -Bind netclaw to a non-loopback internal IP and point the proxy at that address instead of loopback: +Bind netclaw to a non-loopback address and point the proxy at it instead of loopback: ```json { @@ -260,9 +286,9 @@ Bind netclaw to a non-loopback internal IP and point the proxy at that address i ### Non-loopback bind in local mode -`netclaw doctor` reports a warning: "ExposureMode is 'local' but bind address is not loopback." +**Symptom:** `netclaw doctor` fails the `exposure-mode` check with `Invalid local topology: Daemon.Host '0.0.0.0' is not a loopback address.` and exits `1`. The daemon itself refuses to start in this combination. -The daemon is reachable beyond localhost without tunnel protection. Either bind to loopback: +Either bind to loopback: ```json { @@ -285,18 +311,18 @@ Or switch to the exposure mode that reflects how the daemon is actually reachabl ## Related pages -- [Docker Deployment](/deployment/docker/) — containerized daemon with loopback port binding -- [systemd Service](/deployment/systemd/) — bare-metal Linux service management -- [Webhooks](/configuration/webhooks/) — inbound webhook route configuration -- [Models](/configuration/models/) — model slot configuration -- [`netclaw doctor`](/cli/doctor/) — built-in health check diagnostics +- [Run netclaw in Docker](/deployment/docker/) - the containerized daemon and how to reach it +- [systemd service](/deployment/systemd/) - bare-metal Linux service management +- [Pairing remote devices](/guides/pairing-remote-devices/) - authenticate a second machine +- [Webhooks](/configuration/webhooks/) - inbound webhook route configuration +- [`netclaw doctor`](/cli/doctor/) - built-in health check diagnostics ## Resources -- [Tailscale Serve documentation](https://tailscale.com/kb/1242/tailscale-serve) — set up HTTPS endpoints on your tailnet -- [Tailscale Funnel documentation](https://tailscale.com/kb/1223/funnel) — expose tailnet services to the public internet -- [Cloudflare Tunnel documentation](https://developers.cloudflare.com/cloudflare-one/connections/connect-networks/) — route traffic through Cloudflare to your origin -- [Cloudflare Access documentation](https://developers.cloudflare.com/cloudflare-one/policies/access/) — identity-aware access policies for tunneled services -- [nginx reverse proxy guide](https://docs.nginx.com/nginx/admin-guide/web-server/reverse-proxy/) — baseline reverse-proxy behavior and forwarding model -- [Cloudflare Tunnel service docs](https://developers.cloudflare.com/cloudflare-one/connections/connect-networks/configure-tunnels/local-management/as-a-service/) — service startup and lifecycle details for cloudflared -- [Tailscale download](https://tailscale.com/download) — install Tailscale on your platform +- [Tailscale Serve documentation](https://tailscale.com/kb/1242/tailscale-serve) - set up HTTPS endpoints on your tailnet +- [Tailscale Funnel documentation](https://tailscale.com/kb/1223/funnel) - expose tailnet services to the public internet +- [Cloudflare Tunnel documentation](https://developers.cloudflare.com/cloudflare-one/connections/connect-networks/) - route traffic through Cloudflare to your origin +- [Cloudflare Access documentation](https://developers.cloudflare.com/cloudflare-one/policies/access/) - identity-aware access policies for tunneled services +- [nginx reverse proxy guide](https://docs.nginx.com/nginx/admin-guide/web-server/reverse-proxy/) - baseline reverse-proxy behavior and forwarding model +- [Cloudflare Tunnel service docs](https://developers.cloudflare.com/cloudflare-one/connections/connect-networks/configure-tunnels/local-management/as-a-service/) - service startup and lifecycle details for cloudflared +- [Tailscale download](https://tailscale.com/download) - install Tailscale on your platform diff --git a/src/content/docs/deployment/systemd.md b/src/content/docs/deployment/systemd.md index f8fb5f6..30c698d 100644 --- a/src/content/docs/deployment/systemd.md +++ b/src/content/docs/deployment/systemd.md @@ -1,27 +1,28 @@ --- title: "systemd Service" -description: "Run Netclaw as a systemd service on Linux." +description: "Run the netclaw daemon as a systemd user service on Linux: install it, read its logs, update it and troubleshoot it." --- -The netclaw daemon (`netclawd`) runs as a systemd user service — no `sudo`, no root, just your own account. One CLI command sets everything up. +The netclaw daemon (`netclawd`) runs as a systemd user service. No `sudo`, no root, just your own account. One CLI command sets everything up. Linux only: macOS and Windows have no background service, so see [Not on Linux](#not-on-linux). ## Why user-level systemd? Netclaw is a personal agent. It operates on files in your home directory, signs in to services using your credentials, and connects to tools you've authorized. Running the daemon under your own user account matches the privilege envelope it actually needs. -Choosing `systemd --user` over `systemd --system` is **security-neutral relative to running the daemon directly from your shell**. The same files are reachable, the same credentials are accessible, the same network endpoints are in scope. systemd gives you auto-restart on crash, lingering across logout, and journal integration. None of those require elevated privileges. +Choosing `systemd --user` over `systemd --system` is security-neutral relative to running the daemon directly from your shell. The same files are reachable, the same credentials are accessible, the same network endpoints are in scope. systemd gives you auto-restart on crash, lingering across logout, and journal integration. None of those require elevated privileges. -The contrast that matters is user-level versus system-level. A system-level service would run as root, or force you to provision a dedicated `netclaw` user. Netclaw's threat model has no use for either. Root would put a personal agent in the same privilege class as the kernel; a dedicated service user would split your config and credentials across two filesystem owners for no benefit. +The contrast that matters is user-level versus system-level. A system-level service would run as root, or force you to provision a dedicated `netclaw` user. Netclaw's threat model has no use for either. Root would put a personal agent in the same privilege class as the kernel. A dedicated service user would split your config and credentials across two filesystem owners for no benefit. For stricter isolation, layer hardening directives onto the unit file: `ProtectHome=`, `NoNewPrivileges=true`, seccomp filters. Those add real sandboxing. User-level alone doesn't claim to. ## Before you begin -- Linux with systemd **236+** (Ubuntu 20.04+, Debian 11+, Fedora 36+, RHEL 8+, etc.) — check with `systemctl --version` -- `dbus-user-session` installed — required for `systemctl --user` on headless/minimal servers (e.g., `sudo apt install dbus-user-session` on Debian/Ubuntu, then re-login) -- Netclaw installed — see [Installation](/getting-started/installation/) if you don't have it yet -- An initialized `~/.netclaw` directory — run `netclaw init` first -- A configured provider and model — see [Models](/configuration/models/) +- Linux with systemd user services. These steps were run on Ubuntu 24.04 with systemd 255. Check yours with `systemctl --version`. +- `dbus-user-session` installed, which `systemctl --user` needs on headless and minimal servers (`sudo apt install dbus-user-session` on Debian and Ubuntu, then log in again) +- About 1 GiB of free memory. Local embeddings are on by default and download about 129 MB of models on first start. See [Memory and first start](/deployment/docker/#memory-and-first-start). +- Netclaw installed. See [Installation](/getting-started/installation/) if you don't have it yet. +- An initialized `~/.netclaw` directory. Run `netclaw init` first. +- A configured provider and model. See [Models](/configuration/models/). ## Install the service @@ -29,21 +30,31 @@ For stricter isolation, layer hardening directives onto the unit file: `ProtectH netclaw daemon install ``` -This writes a unit file to `~/.config/systemd/user/netclaw.service`, enables it, and runs `loginctl enable-linger $USER` so the service survives logout. - -Start it: +This writes a unit file to `~/.config/systemd/user/netclaw.service` and `~/.netclaw/config/daemon.env`, enables the unit, and runs `loginctl enable-linger $USER` so the service survives logout. It doesn't start the service. Start it: ```bash systemctl --user start netclaw ``` +If the command ends with `Service installed but linger failed: Could not enable linger: Access denied`, the unit is installed but your session can't enable lingering. The command exits `1`. Run `sudo loginctl enable-linger $USER`. + +On a host without user systemd, such as a minimal container or a session with no `XDG_RUNTIME_DIR`, `install` writes nothing and exits `1`: + +```text +Cannot install the service: systemd user services are not available on this host (Failed to connect to bus: No medium found). Run the daemon under a container supervisor, or start it with `netclaw daemon start` (it will not restart after a crash or reboot). +``` + +See [Failed to connect to bus](#failed-to-connect-to-bus-when-running-systemctl) for the fix. + Verify: ```bash netclaw status ``` -![Netclaw status output showing a running daemon](/screenshots/output/status.png) +![netclaw status output: overall healthy, version 0.28.2, the active model, and connector lines for Discord, Mattermost, the demo-utilities MCP server, Slack and the TUI](/screenshots/output/status.png) + +The `uptime` on the daemon line is how you tell a unit that restarted from one that never stopped. ## The unit file @@ -56,24 +67,25 @@ After=network.target [Service] Type=simple -ExecStart=/path/to/netclawd -ExecStop=/path/to/netclaw daemon stop +ExecStart=/home/you/.netclaw/bin/netclawd +ExecStop=/home/you/.netclaw/bin/netclaw daemon stop +TimeoutStopSec=60 Restart=always RestartSec=5 Environment=DOTNET_ENVIRONMENT=Production -Environment=PATH=/path/to/installdir:/home/you/.local/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin +EnvironmentFile=-/home/you/.netclaw/config/daemon.env [Install] WantedBy=default.target ``` -The `/path/to/` placeholders are resolved to actual binary locations by `netclaw daemon install` — the literal string never gets written. `ExecStop` uses the CLI's `daemon stop` command instead of a raw signal, which lets the daemon fire shutdown webhooks and drain active sessions before exiting. After `ExecStop` completes, systemd sends SIGTERM with a 10-second grace period. If the process is still alive after that, SIGKILL finishes it. In practice the daemon shuts down well within that window. +The binary paths are the real locations of `netclawd` and `netclaw`, resolved when you run `netclaw daemon install`. `ExecStop` uses the CLI's `daemon stop` command instead of a raw signal, which lets the daemon fire shutdown webhooks and drain active sessions before exiting. `TimeoutStopSec=60` gives that drain room: the daemon's graceful shutdown budget is 30 seconds, and `daemon stop` waits up to 45 before it force-kills. Existing installs pick up the 60-second value after you re-run `netclaw daemon install`. -The `Environment=PATH=` directive is there because systemd `--user` services start with a sanitized default PATH that does not include `~/.local/bin` or any custom install directory. Without it, the agent's shell tool can't resolve `netclaw` (or other user-installed binaries) when it shells out from inside the daemon. The generated PATH puts your install directory first, then `~/.local/bin`, then the standard system path. If you upgrade from a Netclaw version that predates this directive, see the [Troubleshooting](#shell-tool-cant-find-netclaw-or-other-user-installed-binaries) section below. +`EnvironmentFile` points at `~/.netclaw/config/daemon.env`, which holds a `PATH=` line. systemd `--user` services start with a sanitized default PATH that does not include `~/.local/bin` or any custom install directory, so `netclaw daemon install` captures the PATH of the shell you ran it from and writes it there. Without it, the agent's shell tool can't resolve `netclaw` (or other user-installed binaries) when it shells out from inside the daemon. The leading `-` makes systemd tolerate a missing file. `netclaw daemon install` and `netclaw doctor --fix` replace only the first `PATH` line and keep every other line in the file as it was, so `daemon.env` is also where your own settings go (see [Configuration](#configuration)). If you upgrade from a Netclaw version that predates this, see the [Troubleshooting](#shell-tool-cant-find-netclaw-or-other-user-installed-binaries) section below. ### Manual installation -If you prefer to create the service file yourself, adjust the `ExecStart`, `ExecStop`, and `PATH` entries to point at wherever your binaries and user-bin directory actually live: +To write the service file yourself, point `ExecStart`, `ExecStop` and `PATH` at wherever your binaries and user-bin directory live. `%h` expands to your home directory: ```bash mkdir -p ~/.config/systemd/user @@ -85,12 +97,13 @@ After=network.target [Service] Type=simple -ExecStart=/path/to/netclawd -ExecStop=/path/to/netclaw daemon stop +ExecStart=%h/.netclaw/bin/netclawd +ExecStop=%h/.netclaw/bin/netclaw daemon stop +TimeoutStopSec=60 Restart=always RestartSec=5 Environment=DOTNET_ENVIRONMENT=Production -Environment=PATH=/path/to/installdir:/home/you/.local/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin +Environment=PATH=%h/.netclaw/bin:%h/.local/bin:/usr/local/bin:/usr/bin:/bin [Install] WantedBy=default.target @@ -102,8 +115,6 @@ loginctl enable-linger $USER systemctl --user start netclaw ``` -The `%h` specifier (expands to your home directory) also works in ExecStart/ExecStop paths if you prefer relative-to-home references. - ## Service management | Action | Command | @@ -112,11 +123,12 @@ The `%h` specifier (expands to your home directory) also works in ExecStart/Exec | Stop | `systemctl --user stop netclaw` | | Restart | `systemctl --user restart netclaw` | | Status | `systemctl --user status netclaw` | -| Logs (journal) | `journalctl --user -u netclaw` | +| Lifecycle events | `journalctl --user -u netclaw` | +| Daemon log | `tail -f ~/.netclaw/logs/daemon-*.log` | | Enable on boot | `systemctl --user enable netclaw` | | Disable on boot | `systemctl --user disable netclaw` | -The CLI commands `netclaw daemon start`, `netclaw daemon stop`, and `netclaw daemon status` also work, regardless of whether systemd is managing the process. +`netclaw daemon status` works whoever started the daemon, as long as the PID in `netclaw.pid` is shown to serve this home. If it isn't, nothing is signalled. See [`Nothing was stopped`](#nothing-was-stopped-pid-is-not-shown-to-serve-the-home). `netclaw daemon stop` and `netclaw daemon start` use the unit when it is running this home's daemon, so they're interchangeable with `systemctl --user stop` and `start`, and a daemon stopped with `netclaw daemon stop` stays stopped. There's no `netclaw daemon restart`: use `systemctl --user restart netclaw`. See [`netclaw daemon`](/cli/daemon/). ### Lingering @@ -134,24 +146,32 @@ If your daemon stops every time you disconnect SSH, lingering isn't enabled. ## Configuration -Config lives at `~/.netclaw/config/netclaw.json`. To relocate the base directory, set `NETCLAW_HOME` — useful when you want data on a separate volume or need to run multiple instances with isolated state. Add it to the `[Service]` section of the unit file: +Config lives at `~/.netclaw/config/netclaw.json`. To relocate the base directory, set `NETCLAW_HOME`. That helps when you want data on a separate volume or need to run multiple instances with isolated state. Put it in the new home's `daemon.env`, which `netclaw daemon install` and `netclaw doctor --fix` leave alone: -```ini -[Service] -Environment=NETCLAW_HOME=/data/netclaw +```bash +export NETCLAW_HOME=/data/netclaw +mkdir -p "$NETCLAW_HOME/config" +echo "NETCLAW_HOME=$NETCLAW_HOME" >> "$NETCLAW_HOME/config/daemon.env" ``` +The file has to sit under the new home. With `NETCLAW_HOME` exported, `netclaw daemon install` reads and writes `/config/daemon.env`, so a line left in `~/.netclaw/config/daemon.env` never reaches the unit and the daemon runs on `~/.netclaw`. Use an absolute path. Keep the same value exported in the shell where you run `netclaw`, because `netclaw daemon stop`, `start` and `status` refuse to act when the PID file's process isn't shown to serve the CLI's home. A value with `~`, `$HOME` or `%USERPROFILE%` gives the CLI no proof on Linux. If a line holds a credential, run `chmod 600` on `daemon.env`. + +The install script honors `NETCLAW_HOME` too: config goes to `/config`, and a fresh install puts `bin/` and `env` under it. `INSTALL_DIR` still wins, and an existing `~/.netclaw/bin` install is upgraded in place. `CONFIG_DIR` and `NETCLAW_CONFIG_DIR` no longer do anything. + ### Key paths | Path | Contents | |------|----------| | `~/.netclaw/config/netclaw.json` | Daemon settings | | `~/.netclaw/config/secrets.json` | Provider API keys and credentials | -| `~/.netclaw/netclaw.db` | SQLite database (sessions, stats) | -| `~/.netclaw/logs/daemon.log` | Rolling log file | +| `~/.netclaw/config/daemon.env` | The `PATH` the unit gives the daemon's shell tool, plus any settings of your own | +| `~/.netclaw/netclaw.db` | SQLite database: history, reminders, stats, memory | +| `~/.netclaw/logs/daemon-YYYY-MM-DD.log` | Daily log file, continued in `daemon-YYYY-MM-DD.1.log`, `.2.log` and so on at 10 MiB | | `~/.netclaw/netclaw.pid` | PID file | | `~/.netclaw/netclaw.lock` | Singleton lock (OS-backed exclusive lock) | +[Data directory layout](/reference/data-directory/) lists everything else. + ### Network binding Default binding is `127.0.0.1:5199`, loopback only. @@ -184,7 +204,9 @@ For remote access via Tailscale or Cloudflare Tunnel, see [Exposure Modes](/depl ### Config reload -A file watcher on `netclaw.json` picks up changes and triggers a graceful drain-and-restart automatically. One exception: `Daemon.Host`, `Daemon.Port`, and `Daemon.ExposureMode` require a manual `systemctl --user restart netclaw` — the file watcher ignores these fields because changing the listener binding mid-flight isn't safe. +A file watcher on `netclaw.json` picks up valid changes and triggers a graceful drain-and-restart automatically (tool loops stop at a safe point, and a running sub-agent parks and resumes, which can add about 18 seconds), including `Daemon.Host`, `Daemon.Port`, and `Daemon.ExposureMode`. It doesn't watch `secrets.json` or environment variables (including the unit file's `Environment=` lines), so run `systemctl --user restart netclaw` after changing those. + +An invalid `Models` or provider section doesn't restart anything. The running daemon keeps its previous configuration, and [`netclaw status`](/cli/status/) reports `degraded` with `config on disk not applied: ` until you fix the file. A daemon that restarts with that file exits `1` at startup with the same one-line reason, and systemd restarts it every five seconds. ## Health checks @@ -193,7 +215,7 @@ The daemon exposes two HTTP endpoints: | Endpoint | Auth | Purpose | |----------|------|---------| | `GET /api/health/ready` | None | Returns `200 OK` when the daemon is ready | -| `GET /api/health/status` | Required | Detailed runtime status | +| `GET /api/health/status` | Required, except from loopback in `local` mode | Detailed runtime status. See [Health checks](/observability/health-checks/) | Smoke test: @@ -205,60 +227,73 @@ To block systemd from reporting the service as "started" until the daemon is act ```ini ExecStartPost=/bin/sh -c 'until curl -sf http://127.0.0.1:5199/api/health/ready; do sleep 2; done' +TimeoutStartSec=300 ``` +`TimeoutStartSec` covers `ExecStartPost`, and its default is 90 seconds. The first start downloads about 130 MB of embedding models and can take longer than that. Without the longer timeout, systemd kills the unit and `Restart=always` loops. + For richer metrics and log export, see [OpenTelemetry](/observability/opentelemetry/). ## Logging -Logs go to `~/.netclaw/logs/daemon.log` as a rolling file — `journalctl --user -u netclaw` captures stdout/stderr from the process, but the structured application logs live in the file. Change the log level via `Logging:LogLevel:Default` in `netclaw.json`: +The daemon writes to `~/.netclaw/logs/daemon-YYYY-MM-DD.log`, one series per UTC day. At 10 MiB it continues in `daemon-YYYY-MM-DD.1.log`, then `.2.log`, and a restart appends to the first segment below the limit. The journal shows only what systemd sees: start, stop and the `ExecStop` output. The daemon deletes daily, numbered and crash logs older than 14 days (the newest three of each are always kept), starting 30 seconds after it starts and again every 12 hours. [Data retention](/configuration/data-retention/) covers the setting. + +```bash +tail -f ~/.netclaw/logs/daemon-*.log +``` + +To see daemon logs in the journal too, turn on console logging in a drop-in: + +```bash +mkdir -p ~/.config/systemd/user/netclaw.service.d +cat > ~/.config/systemd/user/netclaw.service.d/logging.conf << 'EOF' +[Service] +Environment=NETCLAW_Logging__Console__Enabled=true +EOF +systemctl --user daemon-reload +systemctl --user restart netclaw +journalctl --user -u netclaw -f +``` + +The default level is `Information`. Change it with `Logging:LogLevel:Default` in `netclaw.json`: ```json { "Logging": { "LogLevel": { - "Default": "Information" + "Default": "Warning" } } } ``` -Valid levels: `Trace`, `Debug`, `Information`, `Warning`, `Error`, `Critical`. +Valid levels: `Trace`, `Debug`, `Information`, `Warning`, `Error`, `Critical`. Crash reports land in `logs/crash-*.log`, and [`netclaw doctor`](/cli/doctor/) flags recent ones. + +## Updating -Follow the log in real time: +The daemon checks for a new release at startup and every 24 hours, logs a warning and raises an `update.available` [alert](/observability/operational-alerts/). It never downloads or applies one. You do that: ```bash -tail -f ~/.netclaw/logs/daemon.log +netclaw update --check +netclaw update ``` -## Upgrading +`netclaw update` stops the service with `systemctl --user`, swaps the two binaries, and starts it again, so the unit file doesn't change. See [`netclaw update`](/cli/update/). -Schema migrations are forward-only with no automatic rollback, so back up before upgrading. +[Back up first](/guides/backup-and-restore/) and read [Upgrade netclaw](/guides/upgrading/). Each release can migrate `netclaw.db`, and an older binary can't read everything a newer one wrote. To go back, stop the service and reinstall an older release over the binaries with the install script and `NETCLAW_VERSION`. That's only safe within the same minor series. -Model configuration is the one exception. On 0.25.0, the first model write converts `Models` to named definitions and roles and keeps a restorable snapshot — see [Migrating model configuration](/guides/migrating-model-config/), especially if your unit sets models through `NETCLAW_Models__*` variables. - -Unlike the Docker deployment, bare-metal installs can self-update — the daemon periodically checks for new releases and applies them. After an update, the process exits and systemd's `Restart=always` brings it back on the new version. For manual upgrades: +If you install by hand instead: ```bash -# 1. Stop the daemon systemctl --user stop netclaw - -# 2. Back up the database -cp ~/.netclaw/netclaw.db ~/.netclaw/netclaw.db.bak.$(date +%s) - -# 3. Replace binaries (method depends on how you installed) -# For manual installs, download and extract the new release. -# For package manager installs, update the package. - -# 4. Start the daemon +curl -sSL https://releases.netclaw.dev/install.sh | NETCLAW_VERSION=0.28.2 bash systemctl --user start netclaw - -# 5. Verify until curl -sf http://127.0.0.1:5199/api/health/ready; do sleep 2; done -echo "Daemon is ready" ``` -To rollback: stop the daemon, restore the database backup, put the old binaries back, and start again. +## Not on Linux + +macOS and Windows have no background service. `netclaw daemon install` refuses with `Service installation is not yet supported on macOS` or `Service installation is currently Linux-only (systemd)`. On those systems start the daemon yourself with `netclaw daemon start`. It runs detached from the terminal and stops with `netclaw daemon stop`. If the daemon doesn't answer, `daemon stop` can't prove the PID is yours and refuses, printing the `kill ` command to use (Task Manager on Windows). It doesn't start again after a reboot, so launch it from a login script if you want that. The macOS service request is tracked in [netclaw issue 1015](https://github.com/netclaw-dev/netclaw/issues/1015). The unit file above doesn't apply there. ## Uninstalling @@ -266,7 +301,7 @@ To rollback: stop the daemon, restore the database backup, put the old binaries netclaw daemon uninstall ``` -Your data in `~/.netclaw` stays untouched. +This stops and disables the service and removes the unit file and `daemon.env`, including any settings of your own in it. To refresh the unit and keep those lines, run `netclaw daemon install` again instead. Your data in `~/.netclaw` stays untouched. To remove manually: @@ -283,7 +318,9 @@ systemctl --user daemon-reload Before diving into specific symptoms, run [`netclaw doctor`](/cli/doctor/). It checks provider connectivity, config validity, daemon health, and common misconfigurations in one pass. -![Netclaw doctor output showing health check diagnostics](/screenshots/output/doctor.png) +![netclaw doctor output: 24 checks, all PASS, ending with doctor exit code: 0](/screenshots/output/doctor.png) + +On a fresh install with no stored credentials, the usual lone warning is a missing `secrets.json`. ### Daemon stops after SSH disconnect @@ -309,14 +346,14 @@ ss -tlnp | grep 5199 If nothing shows, check the daemon log: ```bash -tail -20 ~/.netclaw/logs/daemon.log +ls -t ~/.netclaw/logs/daemon-*.log | head -1 | xargs tail -20 ``` Common causes: port conflict (another process on 5199), JSON syntax error in `netclaw.json`, or missing provider configuration. ### "Daemon already running" when starting -The lock file at `~/.netclaw/netclaw.lock` is held by another process — either a daemon is already running or a previous instance didn't exit cleanly. Check for it: +The lock file at `~/.netclaw/netclaw.lock` is held by another process. Either a daemon is already running or a previous instance didn't exit cleanly. Check for it: ```bash pgrep -a netclawd @@ -332,9 +369,9 @@ systemctl --user start netclaw ### Shell tool can't find `netclaw` (or other user-installed binaries) -If your agent reports `command not found` when shelling out to `netclaw stats`, `gh`, or any other binary in `~/.local/bin`, the systemd unit's PATH is missing the directory where those binaries live. This was the default before the `Environment=PATH=` directive was baked into the install template, so older units don't have it at all. +If your agent reports `command not found` when shelling out to `netclaw stats`, `gh`, or any other binary in `~/.local/bin`, the systemd unit's PATH is missing the directory where those binaries live. Older units were generated without a PATH source (an inline `Environment=PATH=` line, later the `EnvironmentFile`), so they don't have it at all. -Run `netclaw doctor` to confirm — the **Systemd Unit PATH** check will report a Warning if the unit lacks a PATH directive or doesn't include the daemon's install directory. +Run `netclaw doctor` to confirm. The **Systemd Unit PATH** check will report a Warning if the unit lacks a PATH source or the PATH doesn't include the daemon's install directory. Refresh the unit file: @@ -344,27 +381,45 @@ netclaw daemon install systemctl --user restart netclaw ``` -Verify the daemon picked up the new PATH: +`netclaw doctor --fix` replaces the `PATH` line in `daemon.env` and keeps your other lines, without rewriting the unit. Verify the daemon picked up the new PATH: ```bash cat /proc/$(pgrep -u "$USER" netclawd)/environ | tr '\0' '\n' | grep PATH ``` -### Service fails on older systemd versions +### `Could not determine whether systemd owns the daemon lifecycle` + +**Symptom:** `netclaw daemon stop` or `start` exits `1` and says the user session bus isn't reachable. + +**Cause:** A unit is installed, but your shell has no `XDG_RUNTIME_DIR`, so the CLI can't ask systemd who owns the daemon. It refuses to guess, because a guess could leave `Restart=always` bringing the daemon straight back. + +**Fix:** Run the command from a login session, or export `XDG_RUNTIME_DIR` as shown above. `systemctl --user` works from the same shell. + +### `Nothing was stopped. PID is not shown to serve the home` + +**Symptom:** `netclaw daemon stop`, `start` or `status` exits `1` with two lines on stderr, the first naming `PID is not shown to serve `. + +**Cause:** The CLI signals the PID in `netclaw.pid` only when it can show that process serves this home. The usual reasons are a stale PID file, or a different `NETCLAW_HOME` in your shell than in `daemon.env`. `daemon stop` only ever stops the daemon of its own home. + +**Fix:** If no daemon of this home runs, delete `~/.netclaw/netclaw.pid`. If the PID is one of this home's daemons, check it, then `kill `. Never use `pkill netclawd`, which would also hit daemons of other homes. + +### `Systemd Unit PATH` warning from `netclaw doctor` + +**Cause:** `config/daemon.env` is missing, usually after restoring a backup onto the machine. -User-level services require systemd 236+. Check with `systemctl --version`. On older systems, skip systemd and run `netclaw daemon start` directly — it detaches as a background process. +**Fix:** Run `netclaw doctor --fix`, then `systemctl --user restart netclaw`. ## Related pages -- [Docker Deployment](/deployment/docker/) — containerized alternative -- [Exposure Modes](/deployment/exposure-modes/) — remote access via Tailscale or Cloudflare Tunnel -- [Models](/configuration/models/) — model slot configuration -- [OpenTelemetry](/observability/opentelemetry/) — metrics and log export -- [`netclaw doctor`](/cli/doctor/) — built-in health check diagnostics +- [`netclaw daemon`](/cli/daemon/) - start, stop and install the daemon +- [`netclaw update`](/cli/update/) - install a new release +- [Upgrade netclaw](/guides/upgrading/) - what changes between releases +- [Run netclaw in Docker](/deployment/docker/) - the containerized alternative +- [Exposure modes](/deployment/exposure-modes/) - remote access via Tailscale or Cloudflare Tunnel ## Resources -- [systemd user services documentation](https://www.freedesktop.org/software/systemd/man/latest/systemd.service.html) — full unit file reference -- [systemd user services on ArchWiki](https://wiki.archlinux.org/title/Systemd/User) — best practical guide for user-level systemd -- [loginctl enable-linger](https://www.freedesktop.org/software/systemd/man/latest/loginctl.html) — why lingering matters for headless services -- [.NET environment variable configuration](https://learn.microsoft.com/en-us/dotnet/core/extensions/configuration-providers#environment-variable-configuration-provider) — the double-underscore nesting convention used by `NETCLAW_` env vars +- [systemd user services documentation](https://www.freedesktop.org/software/systemd/man/latest/systemd.service.html) - full unit file reference +- [systemd user services on ArchWiki](https://wiki.archlinux.org/title/Systemd/User) - best practical guide for user-level systemd +- [loginctl enable-linger](https://www.freedesktop.org/software/systemd/man/latest/loginctl.html) - why lingering matters for headless services +- [.NET environment variable configuration](https://learn.microsoft.com/en-us/dotnet/core/extensions/configuration-providers#environment-variable-configuration-provider) - the double-underscore nesting convention used by `NETCLAW_` env vars diff --git a/src/content/docs/observability/health-checks.md b/src/content/docs/observability/health-checks.md index fcc471c..66554c3 100644 --- a/src/content/docs/observability/health-checks.md +++ b/src/content/docs/observability/health-checks.md @@ -1,56 +1,55 @@ --- -title: "Health Checks" -description: "HTTP health endpoints for monitoring and orchestration." +title: "Health checks" +description: "The daemon's HTTP health endpoints for Docker, Kubernetes, load balancers and monitoring, and how to read the status response." --- -The daemon exposes two HTTP health endpoints for container orchestration, load balancers, and monitoring systems. +The daemon serves two HTTP health endpoints on its port (default `5199`). Use `/api/health/ready` for container and load balancer probes, and `/api/health/status` when you want to know what's wrong. ## Endpoints | Endpoint | Auth | Purpose | |----------|------|---------| -| `GET /api/health/ready` | None | Liveness/readiness probe | -| `GET /api/health/status` | Required | Full subsystem status | - -Both listen on the daemon's configured port (default `5199`). +| `GET /api/health/ready` | None | Liveness and readiness probe | +| `GET /api/health/status` | Required, except from a loopback call in `local` mode | Full subsystem status | ## Readiness probe ```bash -curl -sf http://127.0.0.1:5199/api/health/ready +curl -si http://127.0.0.1:5199/api/health/ready ``` -Returns `200 OK` with body `healthy` when the daemon is accepting requests. No authentication, no JSON — just a string. Use this for Docker HEALTHCHECK, Kubernetes liveness probes, and load balancer health checks. +```text +HTTP/1.1 200 OK +Content-Type: application/json; charset=utf-8 +X-Netclaw-Generation: 1 + +"healthy" +``` -Any non-200 response (or connection refused) means the daemon isn't ready. +The body is the JSON string `"healthy"`, quotes included. Any other response, or a refused connection, means the daemon isn't accepting requests. `X-Netclaw-Generation` counts in-process restarts since the process started, so it goes up each time a config edit restarts the daemon. It's how the init wizard confirms it's talking to the daemon that picked up the change. -### Docker HEALTHCHECK +The probe reports that the process is up. It stays `200` when a connector is down or no model is configured. Read `/api/health/status` for those. -The official container image uses this: +### Docker + +The official image uses this check: ```dockerfile HEALTHCHECK --interval=15s --timeout=5s --start-period=30s --retries=3 \ - CMD curl -sf http://127.0.0.1:5199/api/health/ready || exit 1 + CMD netclaw daemon status >/dev/null || exit 1 ``` -The 30-second start period gives the daemon time to initialize providers and connect channels before the first probe fires. +`netclaw daemon status` exits `0` when the process is up and `/api/health/ready` answers. It exits `1` when the daemon isn't running, the probe gets no answer, the PID in `netclaw.pid` isn't shown to serve this home, or `netclaw.json` or `secrets.json` can't be read (`The local daemon address is unknown. No readiness probe was sent. Repair the local configuration and retry.`). It reads the endpoint from `netclaw.json`, `NETCLAW_*` variables or the default, so a custom `Daemon.Port` doesn't break the check. + +The 30-second start period covers provider setup and the first-start model download. Check the result with `docker inspect --format '{{.State.Health.Status}}' `. ### Kubernetes -```yaml -livenessProbe: - httpGet: - path: /api/health/ready - port: 5199 - initialDelaySeconds: 30 - periodSeconds: 15 - timeoutSeconds: 5 - failureThreshold: 3 -``` +Kubernetes ignores the image's `HEALTHCHECK`, so define a probe yourself. In `local` mode the daemon only listens on the pod's loopback, and an `httpGet` probe comes from the kubelet to the pod IP, so it's refused. Use an `exec` probe that runs the image's own command, `netclaw daemon status`. In `reverse-proxy` mode with `Daemon.Host` set to `0.0.0.0`, the endpoint is reachable and anonymous, so `httpGet` on port `5199` works. These probes haven't been run on a cluster, so test them before you rely on them. See [Run netclaw in Docker](/deployment/docker/#kubernetes) for the other requirements. -### Systemd +### systemd -For systemd-managed daemons, use a `ExecStartPost` check or a watchdog timer: +To hold the service in "activating" until the daemon accepts requests, add a drop-in with an `ExecStartPost` check: ```ini [Service] @@ -60,146 +59,150 @@ ExecStartPost=/bin/sh -c 'until curl -sf http://127.0.0.1:5199/api/health/ready; ## Status endpoint ```bash -curl -s http://127.0.0.1:5199/api/health/status \ - -H "Authorization: Bearer $(netclaw auth token)" +curl -s http://127.0.0.1:5199/api/health/status ``` -Returns a JSON object with the state of every subsystem. This is the same data that [`netclaw status`](/cli/status/) displays — the CLI just formats it as a table. +It returns the state of every subsystem. [`netclaw status`](/cli/status/) shows the same data as a table. Once you're authorized, the HTTP status is always `200`, so parse `overall`. A fresh container with no model shows `overall: degraded` and `degradedReason: no main model configured (Models:Main missing)`. -Requires authentication (bearer token or loopback origin). Returns `401` without valid credentials. +On the daemon host in `local` mode, loopback requests need no credentials, as long as they look local. The `Host` must be `localhost` or a loopback IP such as `127.0.0.1`, and an `Origin` header, if present, must be `http://` plus that same host. A request through another host name, or from a browser page on another local port, gets `401`. Don't send an `Authorization` header from loopback, because an invalid token returns `401`. In every other mode, loopback isn't trusted and callers need a paired device's token as `Authorization: Bearer `. A remote monitor that only needs liveness should use `/api/health/ready`. ### Response structure +This is the response from a daemon with one local model, embeddings on and the other connectors disabled: + ```json { "overall": "healthy", + "configNotApplied": null, "build": { - "version": "0.4.2", - "commitHash": "a1b2c3d", - "buildTimestamp": "2026-05-01T12:00:00Z" + "version": "0.28.2", + "commitHash": "7419d84", + "buildTimestamp": "2026-10-29T00:10:12Z" }, "process": { - "pid": 1234, - "startedAtUtc": "2026-05-05T08:00:00Z", - "uptimeSeconds": 3600 + "pid": 33, + "startedAtUtc": "2026-10-07T15:28:19.0289154+00:00", + "uptimeSeconds": 284 }, "connectors": [ { "key": "slack", "displayName": "Slack", - "enabled": true, - "status": "healthy", - "message": null + "enabled": false, + "status": "disabled", + "message": "Slack connector is disabled in configuration." }, { - "key": "discord", - "displayName": "Discord", - "enabled": true, - "status": "disconnected", - "message": "Gateway timeout" - }, - { - "key": "mcp:github", - "displayName": "GitHub MCP", + "key": "tui", + "displayName": "TUI", "enabled": true, "status": "healthy", "message": null } ], + "persistence": { + "provider": "Sqlite" + }, + "telemetry": { + "enabled": false, + "otlpEndpoint": "http://127.0.0.1:4317", + "channels": [] + }, "model": { - "modelId": "anthropic/claude-sonnet-4", - "displayName": "Claude Sonnet 4", - "provider": "openrouter", - "inputModalities": ["text", "image"], - "outputModalities": ["text"], - "contextWindow": 200000 + "modelId": "/models/example-model.gguf", + "displayName": "/models/example-model", + "provider": "ollama", + "inputModalities": "Text, Image", + "outputModalities": "Text", + "contextWindow": 131072, + "degraded": false, + "degradedReason": null }, - "persistence": { - "provider": "sqlite" + "update": { + "available": false, + "selfUpdateDisabled": true, + "state": "up-to-date", + "currentVersion": "0.28.2", + "latestVersion": null, + "releaseNotesUrl": null, + "errorDetail": null }, "memory": { "provider": "sqlite", "status": "healthy", - "databasePath": "/home/netclaw/.netclaw/memory/netclaw-memory.db", - "pendingCheckpoints": 0 + "databasePath": "/home/netclaw/.netclaw/netclaw.db", + "pendingCheckpoints": 0, + "embeddings": { + "status": "ok", + "modelId": "snowflake-arctic-embed-m-int8", + "degradedReason": null + } }, "reminders": { - "scheduledCount": 3, + "scheduledCount": 1, "activeExecutions": 0, "failedCount": 0 - }, - "telemetry": { - "enabled": true, - "otlpEndpoint": "http://localhost:4317" - }, - "update": { - "state": "up-to-date", - "available": false, - "currentVersion": "0.4.2", - "latestVersion": "0.4.2" } } ``` -### Overall status +The `connectors` list also holds `discord`, `mattermost`, one `mcp:` entry per configured MCP server, and a `provider:` entry (display name `Provider/`) while an OAuth provider's refresh token is rejected. Its status is `auth-failed`, so `overall` turns `degraded`, and its message says to run `netclaw provider`. `inputModalities` and `outputModalities` are comma-separated strings, not arrays. `telemetry.channels` carries the per-channel counters that `netclaw status` prints. `selfUpdateDisabled` is `true` in the Docker image. -The `overall` field is computed from connector states: +### Overall status | Overall | Condition | |---------|-----------| -| `healthy` | All enabled connectors are healthy | -| `degraded` | Any enabled connector is `disconnected`, `degraded`, `auth-required`, or `auth-failed` | +| `degraded` | The daemon has no usable chat model (`model.degraded` is `true`), or an enabled connector is `disconnected`, `degraded`, `auth-required` or `auth-failed`, or the daemon refused a change to `netclaw.json` (`configNotApplied` holds the reason) | +| `healthy` | Everything else | + +`configNotApplied` is `null` when the file on disk is the configuration the daemon runs. It carries a one-line reason, such as `Models:Roles:Main references unknown definition 'nope'. Defined: stub-stub-model. ...`, after a hand edit left `Models` or a provider invalid. The daemon keeps its previous configuration and applies nothing from the file until you fix it. [`netclaw status`](/cli/status/) prints it as `config on disk not applied: `. A restart with that file exits `1` with the same line. -The HTTP status code is always `200` — parse the `overall` field to determine health. This avoids false-positive container restarts when a single channel has a transient disconnect. +The HTTP status stays `200` either way. That avoids restarting a container because one channel had a transient disconnect. ### Connector statuses -Each connector (Slack, Discord, MCP servers) reports one of: +Each connector (Slack, Discord, Mattermost, the TUI and MCP servers) reports one of: | Status | Meaning | |--------|---------| | `healthy` | Connected and operational | -| `degraded` | Partially functional (e.g., reconnecting) | +| `degraded` | Partially functional, for example reconnecting | | `disconnected` | Connection lost | -| `auth-required` | Needs OAuth flow (MCP servers) | -| `auth-failed` | Credentials rejected | +| `auth-required` | Needs an OAuth flow (MCP servers) | +| `auth-failed` | Credentials rejected, including an OAuth provider that needs a new sign-in | | `disabled` | Turned off in config | +| `unknown` | The connector hasn't reported a state | ### Memory status -| Status | Meaning | -|--------|---------| -| `healthy` | Database accessible, checkpoint backlog ≤ 25 | -| `degraded` | Checkpoint backlog growing (> 25 pending) | -| `unavailable` | Database unreachable | +| Field | Values | +|-------|--------| +| `memory.status` | `healthy` when the database answers, `degraded` when the check throws, `unavailable` when there's no store | +| `memory.embeddings.status` | `ok` (models loaded), `degraded` (provisioning or loading failed, so recall is lexical only), `disabled` (`Memory.Embeddings.Enabled` is `false`) | +| `update.state` | `up-to-date`, `update-available` or `unknown` (the check failed or hasn't run) | -## Monitoring integration +A backlog of memory checkpoints isn't part of this status. [`netclaw doctor`](/cli/doctor/) has a check for it. -### Prometheus / Grafana +## Monitoring integration -Poll `/api/health/status` on an interval and extract metrics: +For a simple availability check, poll the readiness probe: ```bash -# Simple availability check (no auth needed) curl -sf http://127.0.0.1:5199/api/health/ready && echo 1 || echo 0 ``` -For richer metrics, use the [OpenTelemetry](/observability/opentelemetry/) integration which exports to OTLP directly. - -### Operational alerts - -When a connector transitions to `disconnected` or `auth-failed`, the daemon fires an [operational alert](/observability/operational-alerts/) to configured webhook targets. You don't need to poll the status endpoint to detect failures — alerts push to you. +For metrics, use the [OpenTelemetry](/observability/opentelemetry/) export. When a connector drops, an [operational alert](/observability/operational-alerts/) pushes to your webhooks, so you don't need to poll the status endpoint to catch it. ## Related pages -- [`netclaw status`](/cli/status/) — CLI that formats this endpoint's response as a table -- [`netclaw doctor`](/cli/doctor/) — offline diagnostics for configuration issues -- [Operational Alerts](/observability/operational-alerts/) — push notifications on health state changes -- [OpenTelemetry](/observability/opentelemetry/) — metrics and traces export -- [Docker Deployment](/deployment/docker/) — container health check configuration +- [`netclaw status`](/cli/status/) - the CLI view of this endpoint +- [`netclaw doctor`](/cli/doctor/) - offline diagnostics for configuration issues +- [Operational alerts](/observability/operational-alerts/) - push notifications on health changes +- [OpenTelemetry](/observability/opentelemetry/) - metrics and logs export +- [Run netclaw in Docker](/deployment/docker/) - container health and restarts ## Resources -- [ASP.NET Health Checks](https://learn.microsoft.com/en-us/aspnet/core/host-and-deploy/health-checks) — the framework behind the endpoints -- [Kubernetes Probes](https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/) — configuring liveness and readiness probes -- [Docker HEALTHCHECK](https://docs.docker.com/reference/dockerfile/#healthcheck) — container health check instruction reference +- [Docker HEALTHCHECK](https://docs.docker.com/reference/dockerfile/#healthcheck) - the instruction the image uses +- [Kubernetes probes](https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/) - liveness, readiness and startup probes +- [systemd.service ExecStartPost](https://www.freedesktop.org/software/systemd/man/latest/systemd.service.html#ExecStartPost=) - run a command after the service starts diff --git a/src/content/docs/observability/opentelemetry.md b/src/content/docs/observability/opentelemetry.md index eb2ed01..1def6bd 100644 --- a/src/content/docs/observability/opentelemetry.md +++ b/src/content/docs/observability/opentelemetry.md @@ -1,6 +1,6 @@ --- title: OpenTelemetry -description: Export metrics and logs from netclaw via OTLP for monitoring token usage, session activity, and system health. +description: "Export netclaw metrics and logs over OTLP to watch token usage, session activity and daemon health." --- Netclaw can push metrics and structured logs to any [OpenTelemetry](https://opentelemetry.io/) Protocol (OTLP) compatible collector. Flip it on, point it at your collector, and your existing backend (Grafana, Datadog, Honeycomb, whatever speaks OTLP) gets per-channel message flow metrics, token consumption counters, and full daemon logs. @@ -23,24 +23,27 @@ Merge a `Telemetry` block into your existing `~/.netclaw/config/netclaw.json`: | Field | Type | Default | Description | |-------|------|---------|-------------| | `Enabled` | bool | `false` | Turns on the OTLP export pipeline | -| `Otlp:Endpoint` | string | `http://127.0.0.1:4317` | OTLP collector endpoint (gRPC) | +| `Otlp:Endpoint` | string | `http://127.0.0.1:4317` | Base URL of the OTLP collector | -Netclaw uses gRPC OTLP on port 4317, not HTTP/Protobuf (4318). If your collector only accepts HTTP OTLP, you'll get silent failures. - -Environment variable overrides follow the [.NET double-underscore convention](https://learn.microsoft.com/en-us/dotnet/core/extensions/configuration-providers#environment-variable-configuration-provider): +`Otlp:Endpoint` is a base URL. Netclaw speaks gRPC by default, normally on port 4317, and uses the endpoint as written. To use HTTP/protobuf on port 4318, set `OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf` and leave the endpoint at the base. The exporters then post to `/v1/logs` and `/v1/metrics`: ```bash export NETCLAW_Telemetry__Enabled="true" -export NETCLAW_Telemetry__Otlp__Endpoint="http://127.0.0.1:4317" +export NETCLAW_Telemetry__Otlp__Endpoint="http://otel-collector:4318" +export OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf" ``` -Telemetry config changes need a daemon restart. Run [`netclaw doctor`](/cli/doctor/) first to catch config errors before you bounce the daemon: +A full signal URL such as `http://otel-collector:4318/v1/logs` is reduced to its base, so an endpoint copied from an older setup keeps working. The daemon log shows `POST http://otel-collector:4318/v1/logs` and a `200`. + +Environment variable overrides follow the [.NET double-underscore convention](https://learn.microsoft.com/en-us/dotnet/core/extensions/configuration-providers#environment-variable-configuration-provider): ```bash -netclaw doctor -netclaw daemon stop && netclaw daemon start +export NETCLAW_Telemetry__Enabled="true" +export NETCLAW_Telemetry__Otlp__Endpoint="http://127.0.0.1:4317" ``` +The daemon restarts itself when you edit telemetry settings in `netclaw.json`. Environment-variable settings, including `OTEL_*` variables, need a manual restart: `systemctl --user restart netclaw` under systemd, `docker restart ` in Docker, or `netclaw daemon stop` then `netclaw daemon start` for a daemon you started by hand. Run [`netclaw doctor`](/cli/doctor/) first to catch config errors. + ### Validation If `Otlp:Endpoint` isn't a valid absolute URI, the daemon refuses to start, even with telemetry disabled: @@ -53,15 +56,15 @@ Telemetry:Otlp:Endpoint must be an absolute URI. ## What gets exported -Every signal carries an OpenTelemetry resource identifying the service. The name defaults to `netclawd`, but it — and the rest of the `service.*` attributes — are yours to set; see [Service identity](#service-identity). +Every signal carries an OpenTelemetry resource identifying the service. The name defaults to `netclawd`, but you can set it, and the rest of the `service.*` attributes, yourself. See [Service identity](#service-identity). -Every log line the daemon produces goes to your collector, with full formatting and scope data (`IncludeFormattedMessage`, `IncludeScopes`, `ParseStateValues` all enabled). Two meters cover metrics: one for session-level token usage, one for per-channel message flow. Full reference below. +Every log line the daemon produces goes to your collector, with full formatting and scope data (`IncludeFormattedMessage`, `IncludeScopes`, `ParseStateValues` all enabled). That includes the daemon's own HTTP request logs and one Info line per tool call (`Invoking tool [...]`, with secrets redacted and long strings replaced by a length and hash), so exported log volume rises by about one record per tool call. Two meters cover metrics: one for session-level token usage, one for per-channel message flow. Full reference below. -Distributed tracing is off for now. The cross-actor model produces disconnected spans with no meaningful causality chain, so it's more noise than signal. +There's no tracing exporter. The cross-actor model produces disconnected spans with no meaningful causality chain, so it's more noise than signal. Only logs and metrics leave the daemon. ## Service identity -Service identity comes from the standard OpenTelemetry environment variables — netclaw keeps no identity config of its own. Set them however you launch the daemon: +Service identity comes from the standard OpenTelemetry environment variables. netclaw keeps no identity config of its own. Set them however you launch the daemon: ```bash export OTEL_SERVICE_NAME="netclaw-prod" @@ -80,10 +83,10 @@ Four attributes end up on every signal and on each [operational alert](/observab netclaw logs the resolved identity once at startup, so you can confirm the env vars took effect: ``` -OpenTelemetry service identity resolved: service.name=netclaw-prod, service.namespace=ops, service.instance.id=claw-prod-01, service.version=0.22.1 +OpenTelemetry service identity resolved: service.name=netclaw-prod, service.namespace=ops, service.instance.id=claw-prod-01, service.version=0.28.2 ``` -Give each instance a distinct name (or namespace / instance ID) when several report to the same collector or alert webhook — that's what lets you tell them apart. +Give each instance a distinct name (or namespace and instance ID) when several report to the same collector or alert webhook. That's what lets you tell them apart. :::note The short-lived `Telemetry:ServiceName` property from 0.18.2 is gone. Since 0.19.0, service identity is sourced only from the OpenTelemetry environment variables above. @@ -105,7 +108,7 @@ These are aggregate totals across all models and providers, with no per-model or ### Channel metrics (`Netclaw.Channels`) -Per-channel message pipeline metrics. Each metric name is prefixed with `netclaw.channel.{channel-type}` where channel type is one of: `slack`, `tui`, `headless`, `signalr`, `reminder`, `webhook`, `discord`. +Per-channel message pipeline metrics. Each metric name is prefixed with `netclaw.channel.{channel-type}` where channel type is one of: `slack`, `tui`, `headless`, `signalr`, `reminder`, `webhook`, `discord`, `mattermost`. | Metric suffix | Type | Attributes | Description | |---------------|------|------------|-------------| @@ -113,13 +116,13 @@ Per-channel message pipeline metrics. Each metric name is prefixed with `netclaw | `.events.dropped` | Counter | `reason` | Events dropped before processing | | `.events.filtered` | Counter | `reason` | Events filtered by policy | | `.events.routed` | Counter | `kind` | Events that reached conversation routing | -| `.messages.enqueued` | Counter | — | Messages accepted into session queue | -| `.replies.posted` | Counter | — | Successful replies sent | +| `.messages.enqueued` | Counter | - | Messages accepted into session queue | +| `.replies.posted` | Counter | - | Successful replies sent | | `.replies.rejected` | Counter | `error_code` | Rejected reply attempts | -| `.replies.failed` | Counter | — | Failed reply attempts | -| `.reply.duration.ms` | Histogram | — | Reply post latency in milliseconds (includes both successful and failed attempts) | +| `.replies.failed` | Counter | - | Failed reply attempts | +| `.reply.duration.ms` | Histogram | - | Reply post latency in milliseconds (includes both successful and failed attempts) | -> `Netclaw.Webhooks` (the meter behind [`netclaw stats`](/cli/stats/)) is not wired into OTLP. The `netclaw.channel.webhook.*` metrics above cover message flow through the webhook channel; per-route delivery stats only show up in `netclaw stats`. +> `Netclaw.Webhooks` (the meter behind [`netclaw stats`](/cli/stats/)) is not wired into OTLP. The `netclaw.channel.webhook.*` metrics above cover message flow through the webhook channel. Per-route delivery stats only show up in `netclaw stats`. ## Diagnostic queries @@ -131,14 +134,48 @@ Per-channel message pipeline metrics. Each metric name is prefixed with `netclaw ## Collector setup -Point the endpoint at any [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/) that accepts gRPC OTLP. A minimal local setup with Docker: +Point the endpoint at any [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/) that accepts OTLP. This config receives gRPC on 4317 and HTTP on 4318 and prints what arrives, so you can see the pipeline work before you wire up a backend: + +```yaml +receivers: + otlp: + protocols: + grpc: + endpoint: 0.0.0.0:4317 + http: + endpoint: 0.0.0.0:4318 + +exporters: + debug: + verbosity: detailed + +service: + pipelines: + logs: + receivers: [otlp] + exporters: [debug] + metrics: + receivers: [otlp] + exporters: [debug] +``` + +Save it as `otel.yaml`, run the collector on the same Docker network as netclaw, and point netclaw at it over gRPC or HTTP: ```bash -docker run -d --name otel-collector \ - -p 4317:4317 \ - otel/opentelemetry-collector-contrib:latest +docker network create netclaw-net +docker run -d --name otel-collector --network netclaw-net \ + -v "$PWD/otel.yaml:/etc/otelcol/config.yaml:ro" \ + otel/opentelemetry-collector:latest +docker run -d --name netclaw --network netclaw-net \ + -e NETCLAW_Telemetry__Enabled=true \ + -e NETCLAW_Telemetry__Otlp__Endpoint=http://otel-collector:4317 \ + ghcr.io/netclaw-dev/netclaw:0.28.2 ``` +For HTTP/protobuf, use `http://otel-collector:4318` and add `-e OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf`. + +`docker logs otel-collector` shows a `Logs` line within seconds, with the number of log records, and a `Metrics` line after the first export interval (about a minute) once a chat turn has run. The detailed exporter prints each record, your `service.name`, and the `netclaw.session.*` metrics. The collector's output includes the daemon's own HTTP request logs, including the image's health check calling `/api/health/ready` every 15 seconds. + Route to your backend (Prometheus, Grafana Cloud, Datadog, etc.) via the collector's [exporter configuration](https://opentelemetry.io/docs/collector/configuration/#exporters). For a quick local stack, [Grafana OTel-LGTM](https://github.com/grafana/docker-otel-lgtm) bundles the collector, Prometheus, Loki, and Grafana in one container. Good for kicking the tires before committing to a production backend. @@ -155,14 +192,15 @@ Once data is flowing, build dashboards around the [diagnostic queries](#diagnost ## Related pages -- [Operational Alerts](/observability/operational-alerts/) for outbound webhook notifications -- [`netclaw status`](/cli/status/) shows the telemetry row and OTLP endpoint -- [`netclaw doctor`](/cli/doctor/) validates OTLP endpoint configuration -- [`netclaw stats`](/cli/stats/) has in-process counters, including webhook metrics that aren't in OTLP +- [Operational alerts](/observability/operational-alerts/) - outbound webhook notifications +- [Health checks](/observability/health-checks/) - the status endpoint, including the telemetry fields +- [`netclaw status`](/cli/status/) - the telemetry row and OTLP endpoint +- [`netclaw doctor`](/cli/doctor/) - validates OTLP endpoint configuration +- [`netclaw stats`](/cli/stats/) - in-process counters, including webhook metrics that aren't in OTLP ## Resources -- [OpenTelemetry Collector documentation](https://opentelemetry.io/docs/collector/) covers setup, configuration, and deployment -- [OTLP specification](https://opentelemetry.io/docs/specs/otlp/) describes the wire protocol netclaw uses -- [Grafana OTel-LGTM Docker image](https://github.com/grafana/docker-otel-lgtm) is an all-in-one local observability stack -- [.NET environment variable configuration](https://learn.microsoft.com/en-us/dotnet/core/extensions/configuration-providers#environment-variable-configuration-provider) explains the double-underscore nesting convention +- [OpenTelemetry Collector documentation](https://opentelemetry.io/docs/collector/) - setup, configuration and deployment +- [OTLP specification](https://opentelemetry.io/docs/specs/otlp/) - the wire protocol netclaw uses +- [Grafana OTel-LGTM Docker image](https://github.com/grafana/docker-otel-lgtm) - an all-in-one local observability stack +- [.NET environment variable configuration](https://learn.microsoft.com/en-us/dotnet/core/extensions/configuration-providers#environment-variable-configuration-provider) - the double-underscore nesting convention diff --git a/src/content/docs/observability/operational-alerts.md b/src/content/docs/observability/operational-alerts.md index adf44dd..9ae30ad 100644 --- a/src/content/docs/observability/operational-alerts.md +++ b/src/content/docs/observability/operational-alerts.md @@ -1,47 +1,62 @@ --- title: Operational Alerts -description: Configure outbound webhook notifications for daemon lifecycle events, provider failures, channel disconnects, and more. +description: "Send outbound webhook notifications for daemon lifecycle events, provider failures, channel disconnects, memory model problems and more." --- -Netclaw emits operational alerts when something happens that you or your ops tooling should know about — a provider going down, a channel disconnecting, the daemon crashing. Netclaw delivers these as outbound webhook POSTs to URLs you configure, with native Slack Block Kit formatting or a generic JSON envelope. +Netclaw emits operational alerts when something happens that you or your ops tooling should know about: a provider going down, a channel disconnecting, the daemon crashing. Netclaw delivers these as outbound webhook POSTs to URLs you configure, with native Slack Block Kit formatting or a generic JSON envelope. -## Quick Start +## Quick start 1. Run [`netclaw config`](/cli/config/) → **Telemetry & Alerting**. 2. Add a webhook URL and choose `Slack` or `Generic` format. -3. Restart the daemon — you'll get a `daemon.started` alert confirming delivery works. +3. Save. The daemon restarts itself on the config change, and you'll get a `daemon.started` alert confirming delivery works. Repeats of the same type and source inside the 300-second deduplication window collapse into one alert. That's it for most installs. For scripted deployments or headless servers, see [Manual configuration](#manual-configuration) below. -## Alert Types +## Alert types | Type String | Severity | What Happened | |-------------|----------|---------------| | `daemon.started` | Info | Daemon finished startup. Includes PID. | | `daemon.stopping` | Info | Graceful shutdown initiated. Includes reason. | -| `daemon.crashing` | Critical | Unhandled exception — the process is going down. | +| `daemon.crashing` | Critical | Unhandled exception - the process is going down. | | `update.available` | Info | A newer netclaw binary exists in the release feed. | | `provider.failover` | Warning | Primary LLM provider failed; traffic moved to fallback. | | `provider.unreachable` | Critical | All configured LLM providers are unavailable. | | `channel.disconnected` | Warning | Slack, Discord, or Mattermost connection lost. | +| `channel.reconnected` | Info | The Slack connection came back after a drop. Slack only. | +| `provider.auth.expired` | Warning | A provider's OAuth credentials couldn't be refreshed. The summary says to run `netclaw provider`, or `netclaw provider add --auth oauth-device` to sign in again. A rejected OpenAI refresh raises it too. | | `mcp.auth.expired` | Warning | MCP OAuth token expired and refresh was rejected. | | `mcp.server.disconnected` | Warning | Connection to an MCP server dropped. | | `mcp.server.reconnected` | Info | MCP server reconnected successfully after a previous disconnect. | | `webhook.received` | Info | A valid inbound webhook delivery was accepted and queued. | | `webhook.route.invalid` | Warning | A webhook route file is missing or invalid. | -| `reminder.execution.failed` | Warning | A scheduled reminder failed to execute. | +| `reminder.execution.failed` | Warning | A scheduled reminder failed to execute. A failed reply post on a `current_session` reminder is `reminder.delivery.failed` instead. | +| `reminder.delivery.failed` | Warning | A reminder ran and its session admitted it, but the reply post failed for good. Not retried. Context: `reminderId`, `sessionId`, `channel`, `failureKind`, `error`. | +| `reminder.status.query_failed` | Warning | The 12-hour sweep or startup reconciliation couldn't read the status of a past one-shot reminder. It is left as is and retried on the next pass. | +| `reminder.tombstone.invalid` | Warning | The end record of a finished one-shot reminder under `schedules/reminders/ended/` can't be read, so its history is unreadable and it doesn't run. | +| `reminder.schedule.failed` | Warning | A reminder couldn't be scheduled for its next run. Repeated failures auto-disable it. | +| `reminder.settlement.failed` | Warning | A reminder ran, but recording its outcome failed. | +| `reminder.tool_denied` | Warning | A reminder ran, but one of its tool calls was denied (policy, a hard deny, or an approval with nobody to answer). The run is recorded as `denied`. One alert per reminder per distinct reason, until a run succeeds. | | `reminder.auto_disabled` | Critical | Reminder disabled after repeated consecutive failures. | | `reminder.schema.invalid_dropped` | Warning | Invalid reminder definitions were dropped at startup. | | `reminder.schema.legacy_rejected` | Warning | Legacy reminder definitions missing trust fields were rejected at startup. | | `background-job.schema.legacy_rejected` | Warning | Legacy background job definitions missing trust fields were rejected at startup. | +| `memory.embedding_model.unavailable` | Warning | The local embedding model couldn't be downloaded or loaded. Recall falls back to lexical search. Fires once per daemon run. | +| `memory.relevance_model.unavailable` | Warning | The relevance-gate model couldn't be downloaded or loaded, so recall isn't filtered by the cross-encoder. Fires once per daemon run. | -`provider.auth.expired` is defined but not currently emitted. +All configured destinations receive all alert types - there's no per-destination filtering. -All configured destinations receive all alert types — there's no per-destination filtering. +## Configure alert destinations -## Configuring Alert Destinations +Use `netclaw config` → **Telemetry & Alerting** to add, edit, or remove webhook targets at any time. In Docker you can set the same fields as environment variables, which need a `docker restart` to take effect: -Use `netclaw config` → **Telemetry & Alerting** to add, edit, or remove webhook targets at any time. +```bash +-e NETCLAW_Notifications__Webhooks__0__Url=https://alerts.example.com/netclaw \ +-e NETCLAW_Notifications__Webhooks__0__Name=ops \ +-e NETCLAW_Notifications__Webhooks__0__Format=Generic \ +-e NETCLAW_Notifications__Webhooks__0__Headers__X-Api-Key=your-key +``` ### Manual configuration @@ -72,14 +87,14 @@ For scripted or headless installs, add outbound webhook targets directly in `~/. } ``` -Restart the daemon after editing `netclaw.json` for changes to take effect. +The daemon restarts itself after a valid edit to `netclaw.json`. Each webhook target has: | Field | Required | Description | |-------|----------|-------------| | `Url` | Yes | Endpoint to POST alerts to | -| `Name` | No | Human-readable label for logs (falls back to the URL if omitted) | +| `Name` | No | Human-readable label for logs. Logs show `(unnamed webhook)` if omitted, never the URL, because URLs can hold secrets | | `Format` | No | `Slack` or `Generic` (default). URLs containing `hooks.slack.com` auto-detect as Slack. | | `Headers` | No | Custom HTTP headers (auth tokens, API keys) | @@ -91,65 +106,77 @@ Top-level notification settings: | `MaxRetries` | 2 | Retry attempts for failed deliveries | | `TimeoutSeconds` | 10 | HTTP timeout per delivery attempt | -## Payload Formats +## Payload formats ### Generic JSON -Every alert arrives as a JSON POST with this envelope: +Every alert arrives as a JSON POST with `Content-Type: application/json` and a `User-Agent` of `Netclaw/ (+https://netclaw.dev; sha=)` and `X-Netclaw-Component: webhook`, plus any custom headers you set. This is the `daemon.stopping` alert a Docker container sent on `docker stop`: ```json { - "alertId": "550e8400-e29b-41d4-a716-446655440000", - "type": "provider.unreachable", - "severity": "critical", - "summary": "All LLM providers are unreachable", - "timestamp": "2026-05-02T14:30:00Z", + "alertId": "ad0d316727bf", + "type": "daemon.stopping", + "severity": "info", + "summary": "Netclaw daemon stopping: daemon-stop", + "timestamp": "2026-10-07T15:34:12.323848+00:00", "source": "netclaw", - "hostname": "claw-prod-01", + "hostname": "a9fdada82d02", "service": { - "name": "netclaw-prod", - "namespace": "ops", - "instanceId": "claw-prod-01:12345", - "version": "0.22.1" + "name": "netclawd", + "instanceId": "a9fdada82d02:33", + "version": "0.28.2" }, "context": { - "lastProvider": "anthropic", - "errorCount": "5" + "pid": "33", + "reason": "daemon-stop", + "drainOutcome": "drained", + "activeSessions": "0", + "drainedSessions": "0", + "timedOutSessions": "0", + "restartReminders": "0" } } ``` -The `service` object is the [service identity](/observability/opentelemetry/#service-identity) sourced from the OpenTelemetry environment variables. `instanceId` always has a value (it defaults to `{hostname}:{pid}`); `namespace` appears only when you set it. When several netclaw instances post to the same endpoint, that's how you tell which one fired. The `context` object varies by alert type, carrying whatever extra detail is relevant to that event. +`alertId` is a short hex string, and every `context` value is a string. When a session doesn't acknowledge a stop or restart drain, `daemon.stopping` also carries `timedOutSessionIds`, a comma-separated list of those session IDs, and `drainOutcome` is `timeout`. The `type` and `severity` values come from the table above, with `severity` in lowercase. + +The `service` object is the [service identity](/observability/opentelemetry/#service-identity) sourced from the OpenTelemetry environment variables. `instanceId` always has a value (it defaults to `{hostname}:{pid}`), and `namespace` appears only when you set it. When several netclaw instances post to the same endpoint, that's how you tell which one fired. The `context` object varies by alert type, carrying whatever extra detail is relevant to that event. ### Slack Block Kit When `Format` is `Slack`, netclaw sends [Block Kit](https://api.slack.com/block-kit) messages with severity-colored headers and structured fields: -- 🔴 Critical alerts -- ⚠️ Warnings -- ℹ️ Info events +- Critical alerts get `:red_circle:` +- Warnings get `:warning:` +- Info events get `:information_source:` Each message includes a `text` fallback for notification previews. The `blocks` payload has the header, summary, metadata (severity, type, timestamp, hostname), and alert-specific context. -## Delivery Behavior +## Delivery behavior -**Deduplication** — Identical alerts (same type and source) within the deduplication window are suppressed. This prevents notification storms when a provider flaps or a connection drops and reconnects rapidly. +**Deduplication** - Identical alerts (same type and source) within the deduplication window are suppressed. This prevents notification storms when a provider flaps or a connection drops and reconnects rapidly. -**Retries with backoff** — Failed deliveries retry with exponential backoff and jitter. With the default `MaxRetries` of 2: +**Retries with backoff** - Failed deliveries retry with exponential backoff and jitter. With the default `MaxRetries` of 2: | Attempt | Base Delay | Range (with ±25% jitter) | |---------|-----------|---------------------| | 1 | 2s | 1.5s – 2.5s | | 2 | 4s | 3s – 5s | -Backoff caps at 30 seconds for higher retry counts. Netclaw doesn't retry client errors (4xx) — only server errors (5xx) and timeouts trigger retries. +Backoff caps at 30 seconds for higher retry counts. Netclaw doesn't retry client errors (4xx) - only server errors (5xx) and timeouts trigger retries. + +**Bounded queue** - Netclaw buffers alerts in a 256-slot in-memory queue. When the queue fills, the oldest buffered alert is dropped to make room for the new one, so the daemon always accepts new alerts without blocking. + +## Related pages -**Bounded queue** — Netclaw buffers alerts in a 256-slot in-memory queue. When the queue fills, the oldest buffered alert is dropped to make room for the new one, so the daemon always accepts new alerts without blocking. +- [Health checks](/observability/health-checks/) - poll daemon state instead of waiting for a push +- [OpenTelemetry](/observability/opentelemetry/) - export metrics and logs for dashboards and long-term monitoring +- [`netclaw status`](/cli/status/) - check current system health interactively +- [`netclaw config`](/cli/config/) - the **Telemetry & Alerting** screen +- [Memory and embeddings](/configuration/memory/) - what the memory model alerts mean -## Further Reading +## Resources -- [Slack Incoming Webhooks setup guide](https://api.slack.com/messaging/webhooks) — Create the webhook URL you'll paste into the `Url` field -- [Slack Block Kit documentation](https://api.slack.com/block-kit) — Reference for the structured message format netclaw uses for Slack alerts -- [PagerDuty Events API v2](https://developer.pagerduty.com/docs/events-api-v2/overview/) — Set up a PagerDuty service to receive Generic-format alerts -- [OpenTelemetry](/observability/opentelemetry/) — Export metrics via OpenTelemetry Protocol (OTLP) for dashboards and long-term monitoring -- [netclaw status](/cli/status/) — Check current system health interactively +- [Slack Incoming Webhooks setup guide](https://api.slack.com/messaging/webhooks) - create the webhook URL you'll paste into the `Url` field +- [Slack Block Kit documentation](https://api.slack.com/block-kit) - the structured message format netclaw uses for Slack alerts +- [PagerDuty Events API v2](https://developer.pagerduty.com/docs/events-api-v2/overview/) - set up a PagerDuty service to receive Generic-format alerts