Optional monitor integration: alert receiver, silences, monitor metrics, cometduty export

docs/COMMANDS.md

@@ -398,6 +398,7 @@ cometcli genesis install --package mychain-1-packages/acme --home ~/acme-node
   - Files with key material are refused, so a mnemonic or `priv_validator_key.json` can't leave this way.
 - **Carrying the image to an offline machine:** `host prepare --offline_ready --save_image /tmp/img.tar` on a connected machine, then `--image_tar` on the offline one.
 - **Removing a test network:** `cometcli network remove <chain-id> --profiles a,b`. For each profile it stops the container and deletes the node home, but only when that home's genesis is for that chain. It also deletes the chain's genesis builder files on those hosts and the profiles, and reports free disk.
+- **Monitoring a network:** `cometcli network export <chain-id> --format cometduty` writes a cometduty sidecar per node and the chain-role instance; `--deploy` starts them on the hosts. See [With a monitor](#with-a-monitor-optional).
 
 ## Remote nodes over SSH
 
@@ -626,6 +627,65 @@ in `watch.alerts.telegram_token`. Only presses from that chat, and from
 `telegram_users` if set, count. Slack and Discord get reports but can't answer
 approvals.
 
+## With a monitor (optional)
+
+cometcli works without a monitor, and a monitor works without cometcli. If you run
+one — [cometduty](https://github.com/abhijitkrm/cometduty), or Prometheus with
+Alertmanager — three optional links make them work together. Each is off until you
+set it up, and each works with either monitor.
+
+```bash
+# cometduty for every node of a network: config + compose per host, the
+# chain-role instance beside one node, every alert labeled profile=<name>
+cometcli network export mychain-1 --format cometduty                 # files only
+cometcli network export mychain-1 --format cometduty --silence-api --link --deploy
+cometcli network export mychain-1 --format cometduty --webhook-url https://ops.example:9180/alerts
+cometcli network export mychain-1 --format cometduty --telegram     # alerts to watch's Telegram chat too
+# one cometduty for the whole network instead of one per node: every
+# validator's signing, the active set, chain health, and the nodes whose RPC
+# it can reach from the --chain-on host (--rpc adds a public endpoint).
+# No disk, signer-state or log alerts — those need the sidecar layout.
+cometcli network export mychain-1 --format cometduty --layout standalone --chain-on val01 --deploy
+
+# work a monitor's alerts the moment they fire
+cometcli config set-key WATCH_WEBHOOK_SECRET      # cometduty's webhook.secret
+cometcli watch --listen 127.0.0.1:9180            # POST /alerts
+```
+
+| link | what it does | set up with |
+|---|---|---|
+| alerts → `watch` | `watch --listen` receives cometduty's webhook (verified HMAC signature, replays refused) or Alertmanager's (bearer token `WATCH_WEBHOOK_TOKEN`). The node an alert names is triaged at once: a case triage finds is worked like a sweep's; when triage finds nothing, the alert itself becomes the incident (`monitor:<kind>`), cleared by the monitor's resolve. Alerts about no single node (a stalled chain) run a fleet sweep. Sweeps keep running. | `watch.listen` or `--listen` |
+| silences during planned work | before `node service stop/restart`, `node recreate`, `node provision --reconfigure` (15 min) and `upgrade switch` (30 min, the whole chain) cometcli silences the node's alerts — matcher `profile=<name>`, or `chain_id=<id>` — through the Alertmanager v2 silence API. A monitor that can't be reached never blocks the work; the tool says it couldn't silence. | `endpoints.monitor` on the profile and/or `monitor.silence`, token `MONITOR_API_TOKEN` |
+| `watch` leaves planned work alone | an issue on a node under planned work — recorded by the commands above (in `~/.cometcli/planned.json`, so this works with no monitor), or an active silence of the node or its chain at its monitor — is posted once as 🔕 and not worked. If it's still there when the work ends, it's handled as new; if it clears, nothing is posted. | nothing (monitor silences: as above) |
+| one Telegram chat | `--telegram` sets cometduty's own Telegram notifier to the bot and chat `watch` uses (`watch.alerts`, `TELEGRAM_BOT_TOKEN`; `--telegram-chat` for another), so the monitor's alerts and watch's diagnosis and approvals land together. In Telegram each issue is **one card** edited as the work goes on — bold headline, a plain reason (the case's `plain:`), the raw evidence and the agent's report collapsed, a status line — plus a reply when it resolves; approvals are separate messages with Approve / Deny. cometduty does the same for its alerts (one card per node). | `--telegram` |
+| monitor metrics in triage | triage reads the node's monitor `/metrics` (signing streak, slashing-window %, signer height, disk, open alerts, silences) as `monitor.*` signals. Cases: `monitor-unreachable` (nobody may be paged) and `monitor-alerts-silenced`. | `endpoints.monitor` |
+
+```yaml
+# ~/.cometcli/config.yaml — what --link writes
+profiles:
+  val01:
+    endpoints:
+      monitor: http://127.0.0.1:28686    # on the node's host, reached through ssh
+monitor:
+  silence:
+    - url: http://127.0.0.1:28687        # the chain-role instance
+      via: val01                         # reached through this profile's ssh
+watch:
+  listen: 127.0.0.1:9180
+```
+
+The chain-role instance sees the chain through the RPC of the node it runs beside
+(`--chain-on`, default the first profile): node RPCs listen on their host's loopback,
+so that is the one it can reach. When that node is down it reports `no-nodes` — it
+can't see the chain — not a halt; pick a node you restart rarely.
+
+Only `kind`, `target`, the labels and the severity of an alert steer what cometcli does.
+Its message text — which can quote node logs — never reaches the agent.
+
+For Alertmanager, give your alerts a `profile` label (or a `node` or `instance` on a
+profile's host) so cometcli knows which node they are about, and point a webhook
+receiver at `/alerts` with `http_config.authorization.credentials` set to the token.
+
 ## Incident drills
 
 Check how well cometcli handles real faults, on a throwaway network:

internal/cli/export.go

@@ -0,0 +1,260 @@
+package cli
+
+import (
+	"fmt"
+	"os"
+	"path/filepath"
+	"sort"
+	"strings"
+
+	"github.com/spf13/cobra"
+
+	"github.com/abhijitkrm/cometcli/internal/config"
+	"github.com/abhijitkrm/cometcli/internal/netspec"
+	"github.com/abhijitkrm/cometcli/internal/silence"
+	"github.com/abhijitkrm/cometcli/internal/toolkit"
+	"github.com/abhijitkrm/cometcli/internal/tools/networktool"
+)
+
+func networkExportCmd() *cobra.Command {
+	var format, layout, rpcs, profiles, out, webhookURL, chainOn, image, bind, tgChat string
+	var silenceAPI, deploy, link, telegram bool
+	cmd := &cobra.Command{
+		Use:   "export <chain-id> --format cometduty",
+		Short: "Write monitoring config for a network's nodes (cometduty), optionally deploy it",
+		Long: `Writes cometduty monitoring for the nodes of <chain-id>, from what the
+profiles already know (RPC, EVM and metrics ports, node home, container,
+operator address). Every alert carries the label profile=<name>.
+
+  --layout sidecar     (default) a cometduty beside each node — everything,
+                       including disk, signer-state and log alerts that need
+                       the node's machine — plus a chain-role instance
+  --layout standalone  one cometduty for the whole network on the --chain-on
+                       host: every validator's signing, the active set, chain
+                       health, and each node whose RPC it can reach; no disk,
+                       signer-state or log alerts. --rpc adds endpoints it
+                       can reach (a public RPC)
+
+  --webhook-url   send alerts to a receiver, e.g. cometcli watch --listen;
+                  signed with a secret saved as WATCH_WEBHOOK_SECRET
+  --telegram      send every alert to Telegram too, with the bot and chat
+                  watch already uses (watch.alerts, TELEGRAM_BOT_TOKEN);
+                  --telegram-chat picks another chat
+  --silence-api   turn on cometduty's silence API (token saved as
+                  MONITOR_API_TOKEN) so cometcli silences a node's alerts
+                  during planned work
+  --deploy        copy each node's files to its host and start cometduty
+                  there (docker compose), after you confirm
+  --link          record each node's monitor (endpoints.monitor) and the
+                  chain instance (monitor.silence) in cometcli's config
+
+Without --deploy nothing leaves this machine: the files are in --out.
+cometduty doesn't need cometcli, and cometcli doesn't need cometduty.`,
+		Args: cobra.ExactArgs(1),
+		RunE: func(cmd *cobra.Command, args []string) error {
+			w := cmd.OutOrStdout()
+			chain := args[0]
+			if format != "cometduty" {
+				return fmt.Errorf("--format: only cometduty is supported")
+			}
+			cfg, err := config.Load()
+			if err != nil {
+				return err
+			}
+			var list []*config.Profile
+			want := splitCSV(profiles)
+			var names []string
+			for n := range cfg.Profiles {
+				names = append(names, n)
+			}
+			sort.Strings(names)
+			for _, n := range names {
+				p := cfg.Profiles[n]
+				if (len(want) > 0 && !contains(want, n)) || (len(want) == 0 && p.ChainID != chain) {
+					continue
+				}
+				p.Name = n
+				list = append(list, p)
+			}
+			for _, n := range want {
+				if cfg.Profiles[n] == nil {
+					return fmt.Errorf("no profile %s", n)
+				}
+			}
+			if len(list) == 0 {
+				return fmt.Errorf("no profiles on %s", chain)
+			}
+			o := networktool.MonitorOpts{Image: image, WebhookURL: webhookURL, ChainOn: chainOn, MetricsBind: bind}
+			if webhookURL != "" {
+				if o.Secret = credential("WATCH_WEBHOOK_SECRET"); o.Secret == "" {
+					o.Secret = networktool.RandomSecret()
+					if err := config.SetCredential("WATCH_WEBHOOK_SECRET", o.Secret); err != nil {
+						return err
+					}
+					fmt.Fprintln(w, "✓ new webhook secret saved as the WATCH_WEBHOOK_SECRET credential")
+				}
+			}
+			if silenceAPI {
+				if o.APIToken = credential(silence.TokenCredential); o.APIToken == "" {
+					o.APIToken = networktool.RandomSecret()
+					if err := config.SetCredential(silence.TokenCredential, o.APIToken); err != nil {
+						return err
+					}
+					fmt.Fprintf(w, "✓ new silence API token saved as the %s credential\n", silence.TokenCredential)
+				}
+			}
+			if telegram || tgChat != "" {
+				o.TelegramToken, o.TelegramChat = cfg.Watch.Alerts.TelegramToken, cfg.Watch.Alerts.TelegramChatID
+				if o.TelegramToken == "" {
+					o.TelegramToken = credential("TELEGRAM_BOT_TOKEN")
+				}
+				if tgChat != "" {
+					o.TelegramChat = tgChat
+				}
+				if o.TelegramToken == "" || o.TelegramChat == "" {
+					return fmt.Errorf("--telegram needs a bot token (watch.alerts.telegram_token or the TELEGRAM_BOT_TOKEN credential) and a chat (watch.alerts.telegram_chat_id or --telegram-chat)")
+				}
+			}
+			var sets []networktool.MonitorFiles
+			switch layout {
+			case "sidecar":
+				if rpcs != "" {
+					return fmt.Errorf("--rpc is for --layout standalone")
+				}
+				if sets, err = networktool.CometdutyFiles(chain, list, o); err != nil {
+					return err
+				}
+			case "standalone":
+				f, skipped, err := networktool.StandaloneFiles(chain, list, o, splitCSV(rpcs))
+				if err != nil {
+					return err
+				}
+				sets = []networktool.MonitorFiles{f}
+				if len(skipped) > 0 {
+					fmt.Fprintf(w, "note: %s not watched as nodes — their RPC is loopback-only on another host (signing is still watched for validators)\n", strings.Join(skipped, ", "))
+				}
+			default:
+				return fmt.Errorf("--layout: sidecar or standalone")
+			}
+			if out == "" {
+				if out, err = config.Path(filepath.Join("monitor", netspec.SafeName(chain))); err != nil {
+					return err
+				}
+			}
+			chainHost := networktool.ChainHost(list, chainOn)
+			for _, f := range sets {
+				dir := filepath.Join(out, f.Dir)
+				if err := os.MkdirAll(dir, 0o700); err != nil {
+					return err
+				}
+				for name, content := range f.Files {
+					if err := os.WriteFile(filepath.Join(dir, name), content, 0o600); err != nil {
+						return err
+					}
+				}
+				on := f.Profile
+				if on == "" {
+					on = chainHost.Name + " (" + map[string]string{"sidecar": "chain-role", "standalone": "standalone"}[layout] + " instance)"
+				}
+				fmt.Fprintf(w, "  %-28s → %s  (metrics :%d)\n", on, dir, f.Port)
+			}
+			if layout == "standalone" {
+				fmt.Fprintf(w, "✓ cometduty standalone config for %s (%d profiles) in %s\n", chain, len(list), out)
+			} else {
+				fmt.Fprintf(w, "✓ cometduty files for %d nodes of %s in %s\n", len(list), chain, out)
+			}
+
+			if deploy {
+				if !flagYes && !confirm(cmd, fmt.Sprintf("start cometduty on the hosts of %d nodes (docker compose, as each node home's owner)?", len(list))) {
+					return fmt.Errorf("cancelled — the files are in %s", out)
+				}
+				for _, f := range sets {
+					p := chainHost
+					if f.Profile != "" {
+						p = cfg.Profiles[f.Profile]
+					}
+					c := &toolkit.Context{Context: cmd.Context(), Profile: p, Cfg: cfg,
+						Approver: func(*toolkit.Context, string, toolkit.Tier, map[string]any) (bool, error) { return true, nil }}
+					msg, err := networktool.DeployMonitor(c, chain, f, p.Home)
+					c.Close()
+					if err != nil {
+						return fmt.Errorf("%s: %w", p.Name, err)
+					}
+					who := f.Profile
+					if who == "" {
+						who = layout + " instance on " + p.Name
+						if layout == "sidecar" {
+							who = "chain instance on " + p.Name
+						}
+					}
+					fmt.Fprintf(w, "✓ %s: %s\n", who, msg)
+				}
+			}
+			if link {
+				for _, f := range sets {
+					if f.Profile != "" {
+						cfg.Profiles[f.Profile].Endpoints.Monitor = fmt.Sprintf("http://127.0.0.1:%d", f.Port)
+					} else {
+						u := fmt.Sprintf("http://127.0.0.1:%d", f.Port)
+						var kept []config.SilenceAPI
+						for _, s := range cfg.Monitor.Silence {
+							if s.Via != chainHost.Name || s.URL != u {
+								kept = append(kept, s)
+							}
+						}
+						cfg.Monitor.Silence = append(kept, config.SilenceAPI{URL: u, Via: chainHost.Name})
+					}
+				}
+				if err := cfg.Save(); err != nil {
+					return err
+				}
+				if layout == "standalone" {
+					fmt.Fprintln(w, "✓ linked: the standalone instance under monitor.silence (planned work silences it)")
+				} else {
+					fmt.Fprintln(w, "✓ linked: each profile's endpoints.monitor and the chain instance under monitor.silence")
+				}
+			}
+			var next []string
+			if !deploy {
+				if layout == "standalone" {
+					next = append(next, "copy the folder to any machine that reaches those RPCs and run `docker compose up -d` there — or rerun with --deploy")
+				} else {
+					next = append(next, "copy each folder to its node's host and run `USER_ID=$(stat -c %u:%g <node home>) docker compose up -d` there — or rerun with --deploy")
+				}
+			}
+			if webhookURL != "" {
+				next = append(next, "receive the alerts: cometcli watch --listen <addr> (reachable at "+webhookURL+")")
+			}
+			if !link {
+				if layout == "standalone" {
+					next = append(next, "rerun with --link so cometcli silences it during planned work")
+				} else {
+					next = append(next, "rerun with --link so triage reads each node's monitor and cometcli silences it during planned work")
+				}
+			}
+			for i, n := range next {
+				fmt.Fprintf(w, "next %d: %s\n", i+1, n)
+			}
+			if !silenceAPI && link {
+				fmt.Fprintln(w, "note: without --silence-api cometduty serves no silence API; restarts and upgrades will page as usual")
+			}
+			return nil
+		},
+	}
+	f := cmd.Flags()
+	f.StringVar(&format, "format", "cometduty", "what to write (cometduty)")
+	f.StringVar(&layout, "layout", "sidecar", "sidecar (one per node + chain instance) or standalone (one for the network)")
+	f.StringVar(&rpcs, "rpc", "", "standalone: extra RPC endpoints it can reach, comma-separated (data sources, not paged)")
+	f.StringVar(&profiles, "profiles", "", "nodes to include (default: every profile on <chain-id>)")
+	f.StringVar(&out, "out", "", "where to write (default ~/.cometcli/monitor/<chain-id>)")
+	f.StringVar(&webhookURL, "webhook-url", "", "send alerts here, signed (e.g. where cometcli watch --listen is reachable)")
+	f.BoolVar(&telegram, "telegram", false, "send alerts to Telegram too (watch's bot and chat)")
+	f.StringVar(&tgChat, "telegram-chat", "", "Telegram chat id for alerts (implies --telegram)")
+	f.BoolVar(&silenceAPI, "silence-api", false, "enable cometduty's silence API (token in the MONITOR_API_TOKEN credential)")
+	f.StringVar(&chainOn, "chain-on", "", "profile whose host runs the chain-role (or standalone) instance (default: the first)")
+	f.StringVar(&image, "image", "", "cometduty image (default "+networktool.CometdutyImage+")")
+	f.StringVar(&bind, "metrics-bind", "127.0.0.1", "where cometduty's /metrics and silence API listen (0.0.0.0 for a remote Prometheus)")
+	f.BoolVar(&deploy, "deploy", false, "copy the files to each host and start cometduty there")
+	f.BoolVar(&link, "link", false, "record the monitors in cometcli's config (endpoints.monitor, monitor.silence)")
+	return cmd
+}

internal/cli/root.go

@@ -75,7 +75,7 @@ status, cometcli doctor, cometcli tx unjail) and a tool the agent calls.`,
 
 	for _, c := range toolGroupCommands(reg) {
 		if c.Use == "network" {
-			c.AddCommand(networkImportCmd(), networkShowCmd(), networkRemoveCmd())
+			c.AddCommand(networkImportCmd(), networkShowCmd(), networkRemoveCmd(), networkExportCmd())
 		}
 		if c.Use == "val" {
 			c.AddCommand(operatorKeyCmd())
@@ -183,7 +183,8 @@ func argsFromFlags(cmd *cobra.Command) toolkit.Args {
 // hand-written commands (doctor, agent) can drive tools too.
 func RunTool(cmd *cobra.Command, t toolkit.Tool, args toolkit.Args) error {
 	// keys.* and network.* work without an active profile
-	c, err := NewCtx(cmd, !strings.HasPrefix(t.Name(), "keys.") && !strings.HasPrefix(t.Name(), "network."))
+	// keys, network and fleet commands work on no single node
+	c, err := NewCtx(cmd, !strings.HasPrefix(t.Name(), "keys.") && !strings.HasPrefix(t.Name(), "network.") && !strings.HasPrefix(t.Name(), "fleet."))
 	if err != nil {
 		return err
 	}