[abhijitkrm]

Disk failure handling, from a live disk-full drill on primium-6 (2026-10-15). Goes with cometduty feat/disk-alerts (0.4.0); each tool still works alone.

Triage (works without a monitor)

• New host signals over ssh:
  • host.fs_readonly: findmnt on the node home.
  • host.kernel_disk_errors: kernel journal, last hour; sudo -n first, then the user's own view.
  • host.disk_health_failed: smartctl -H, when installed.
  • pvs.corrupt: priv_validator_state.json that doesn't parse.
• New cases, each with plain-language text: host-fs-readonly, host-disk-io-errors, host-disk-failing, val-signer-state-corrupt.
• host-disk-full / -high / -free-low playbooks: they now look at the whole filesystem (du -x of the mount, files over 1G), not just the node's data directory. In the drill the 148 GiB filler sat outside the node home and the old step showed only 8.3 GiB.
• host-disk-full fix steps: restart the node once space is free. A node that hits a full disk freezes and stays frozen after space returns.

Docker logs

• Triage reads docker logs -t --tail 20000 and cuts the time window itself. --since made dockerd read the node's whole json-file log on every sweep.
• New node compose files cap Docker logs at 5 × 100 MB (logging:). The live nodes had 2.1–4.7 GB logs; existing nodes pick it up on node.recreate.

Safety

• node.service start|restart and node.recreate refuse a validator whose priv_validator_state.json is corrupt. It can't tell what it last signed and could double-sign.

network export

• Sidecars get host mode: the host's /proc/1/mounts and /var/log read-only, plus the group that reads kern.log. At deploy, cometcli checks what the host has and drops what it doesn't. A journald-only host such as Amazon Linux has no kern.log; triage reads the kernel journal there.
• Inode and fill-rate alerts turned on (5%, 12 h).
• --rpc now works with --layout sidecar: it gives the chain instance a second endpoint. Without one, export says a stall there is reported on that node.
• CometdutyImage → cometduty:0.4.0: cometduty reads its config strictly, so the new keys need it. Merge and release cometduty 0.4.0 first.

Live checks (primium-6)

• Triage on the archive after a labelled drill line in the kernel journal: kernel_disk_errors=1, and host-disk-io-errors (critical) matched. Ubuntu and Amazon Linux both report fs_readonly and kernel_disk_errors.
• Export detected each host's views at deploy (mount table everywhere; kern.log only on the Ubuntu hosts).

Tests

go test ./... passes. New tests: signer-state parsing, the host fitting of sidecar files, and the chain instance's extra RPCs. The KB self-match test covers the new cases, and the healthy baseline includes the new signals.

🤖 Generated with Claude Code