[abhijitkrm]

cometcli works with a monitor (cometduty, or Prometheus + Alertmanager) and without one. Everything here is off until configured, and each link works with either monitor.

What's new

• watch --listen receives cometduty's webhook or Alertmanager's.
  • cometduty alerts are checked against their HMAC signature, and replays and stale timestamps are refused. Alertmanager alerts need the bearer token.
  • The node an alert names is triaged at once. When triage finds nothing, the alert itself becomes the incident (monitor:<kind>), and the monitor's resolve clears it.
  • An alert about no single node runs a fleet sweep.
  • Only kind, target, labels and severity steer the work. Alert text, which can quote node logs, never reaches the agent.
• Silences during planned work: node service stop/restart, node recreate and provision --reconfigure silence the node's alerts for 15 minutes; upgrade switch silences the whole chain's for 30.
  • They use the Alertmanager v2 API at the node's endpoints.monitor and at monitor.silence, going through ssh when the API is on the node's loopback.
  • The chain instance's no-nodes/stalled are silenced too when its source node is the one being worked on.
  • A monitor that can't be reached never blocks the work.
• endpoints.monitor: triage reads the monitor's /metrics as monitor.* signals. New cases: monitor-unreachable and monitor-alerts-silenced.
• network export <chain> --format cometduty writes a sidecar per node plus the chain-role instance from the profiles.
  • Metrics and the API listen on loopback by default.
  • --deploy starts them on the hosts (run as the node home's owner).
  • --link records them in the config.
  • --webhook-url and --silence-api generate secrets and save them as credentials.
• Smaller fixes: fleet commands no longer need an active profile; watch summarizes issues left from an earlier run in one line.

Live test (primium-6, 4 hosts, 2026-10-14 IST)

• Deploy: built the cometduty branch image on the hosts. network export --silence-api --webhook-url … --link --deploy started 5 instances in 47s.
• Triage: reads every node's monitor through ssh (signing streak, window %, signer height, disk, alerts, silences).
• Unplanned outage (archive container stopped directly): the chain instance's alert reached watch in seconds and a fleet sweep found the archive down. On recovery, watch re-triaged the archive and reported it resolved.
• Planned stop (node service stop): silenced at the sidecar and the chain instance; no node-down reached watch in 4+ minutes. Found and fixed the chain instance's no-nodes leak.
• Four cometduty bugs found along the way, fixed in Webhook payload v1, signed webhooks, silence API, generic sidecar kit + 4 live-found fixes cometduty#2: evm-lag false positive, node-down never firing after catch-up, signer-stalled on non-signers, chains.d permission.
• Tests: 427 pass. TestCometdutyAcceptsExport validates every generated config with the real cometduty binary (set COMETDUTY_BIN).

Depends on

• network export defaults to ghcr.io/abhijitkrm/cometduty:0.3.0, which needs Webhook payload v1, signed webhooks, silence API, generic sidecar kit + 4 live-found fixes cometduty#2 released as v0.3.0. Until then, pass --image.

Telegram and live fixes (2026-10-14)

• network export --telegram (or --telegram-chat) points cometduty's Telegram notifier at the bot and chat watch uses, so both tools post to one chat.
• One card per issue: in Telegram, watch posts each issue as one card and edits it as the work goes on. The card has a bold headline, a plain reason (new optional plain: field on KB cases; the 54 critical/high cases have one), the raw evidence and the agent's report as collapsed quotes, and a status line. A reply marks the resolve. Notifiers that can't edit get the card when it opens and when it's done. Approvals have a bold headline and plain Approve / Deny buttons. No emoji.
• Planned work: watch leaves planned work alone. Planned operations are recorded locally in planned.json (so this works with no monitor) and checked against active silences at the monitors. An issue under planned work is posted once and not worked; if it outlasts the work it's handled then; if it clears, nothing is posted.
• Queued alerts: alerts that pile up during long work collapse to each condition's latest state. Live, an already-resolved evm-down had started an agent run.
• Cooldown: an issue that returns within the cooldown is now reported ("back again — not acting automatically"). Before, it was silent.
• Redeploy: a redeploy restarts cometduty when its config changed. compose up -d alone kept the old config.
• fleet tools: fleet status/exec/shell now set a call deadline that covers --timeout. A fixed 90s had cut long commands short.
• Live run: archive stopped on primium-6 → cometduty cards → watch card → approval in Telegram → restart → resolved. About 7 messages, against 15+ before.

Standalone layout

• network export --layout standalone writes one cometduty for the whole network, run on the --chain-on host. It covers every validator's signing, the active set, chain health, and each node whose RPC it can reach. --rpc adds endpoints such as a public RPC; they serve as data sources and don't page.
• It has no disk, signer-state or log alerts; those need the sidecar layout (the default). Its default metrics port is 28696, so it can run next to sidecars.
• Fix: a validator profile that saved only its account address had no valoper, so neither layout watched its signing. The valoper is now derived from the account address.
• Live: one standalone instance on primium-6's archive host saw all 3 validators bonded and signing, through that host's RPC alone.

🤖 Generated with Claude Code