diff --git a/docs/COMMANDS.md b/docs/COMMANDS.md index 364a766..ae65b0d 100644 --- a/docs/COMMANDS.md +++ b/docs/COMMANDS.md @@ -240,11 +240,11 @@ Every node of the chain is checked against it according to its role. ```bash cometcli network import run-genesis/network-config.env run-validator/network-config.env \ - run-archive/network-config.env --image-naming 'primium-{tag}' # from node-setup's env files -cometcli network show primium-1 -cometcli network check --network primium-1 # spec sanity, chain drift, every node of the chain -cometcli network check --network primium-1 --prod # plus public-network hardening -cometcli network check --network primium-1 --nodes val1,archive1 + run-archive/network-config.env --image-naming 'mychain-{tag}' # from KEY=value env files +cometcli network show mychain-1 +cometcli network check --network mychain-1 # spec sanity, chain drift, every node of the chain +cometcli network check --network mychain-1 --prod # plus public-network hardening +cometcli network check --network mychain-1 --nodes val1,archive1 ``` `network check` reports at three levels: @@ -266,14 +266,14 @@ cometcli network check --network primium-1 --nodes val1,archive1 ```bash cometcli network import run-genesis/network-config.env --chain-id mychain-1 # the new network's spec -cometcli profile add v1 --ssh-host 203.0.113.1 --service docker --unit primium-validator --role validator +cometcli profile add v1 --ssh-host 203.0.113.1 --service docker --unit mychain-1-validator --role validator # … one profile per genesis validator cometcli --profile v1 host prepare # a fresh host: docker, compose, git, jq (asks first) -cometcli --profile v1 upgrade build --tag v0.7.2 --image 'primium-{tag}' # the image, on each host +cometcli --profile v1 upgrade build --tag v0.7.2 --image 'mychain-{tag}' # the image, on each host cometcli genesis create mychain-1 --validators v1,v2,v3,v4 --keyring file --accounts treasury.txt ``` -`genesis create` runs node-setup's distributed genesis workflow, with cometcli as the +`genesis create` runs a distributed genesis workflow, with cometcli as the coordinator: 1. **On each validator's own host:** `init` in the image, so the node and consensus keys @@ -293,8 +293,8 @@ coordinator: `host prepare` installs only what's missing (Ubuntu/Debian or Amazon Linux/RHEL, passwordless sudo); without passwordless sudo it prints the commands to run instead. `upgrade build` on a -host with no node yet, or no `Dockerfile.primium`, builds with cometcli's built-in Dockerfile -(the one node-setup uses), and remembers the Dockerfile each host used. Before anything is +host with no node yet, or no Dockerfile beside the node's compose project, builds with cometcli's built-in Dockerfile +and remembers the Dockerfile each host used. Before anything is created, `genesis create` checks that every node's host ports are free and names the container holding them. @@ -307,7 +307,7 @@ Back up each validator's `priv_validator_key.json` off-machine afterwards. ```bash cometcli network import --from-node val1 # spec + genesis.json from a running node -cometcli profile add arch1 --ssh-host 203.0.113.9 --service docker --unit primium-archive --role archive +cometcli profile add arch1 --ssh-host 203.0.113.9 --service docker --unit mychain-1-archive --role archive cometcli --profile arch1 node provision --peers_from val1,val2 # init, genesis, configs, compose, start cometcli --profile arch1 node provision --reconfigure # re-render configs of an existing node ``` @@ -338,6 +338,67 @@ For a validator, once it's synced: 3. Run `cometcli --profile

val create --amount `. The consensus key and moniker come from the node, and commission is encoded the way the chain's cosmos-sdk version expects. +### A validator joining a running network + +```bash +cometcli --profile val3 node provision --role validator --network mychain-1 --peers_from val1,val2 +cometcli --profile val3 wait until --condition synced +cometcli --profile val3 val operator-key # key in the node's own keyring; mnemonic → 0600 file on that host +cometcli --profile val1 tx send --to --amount +cometcli --profile val3 val create --amount # its consensus key is read from the node +``` + +While a node is down, chain queries (jail status, upgrade plans, params) go to another +node of the same chain from your profiles, through that node's own connection. + +### Validators you don't run (Workflow B) and offline keys + +`genesis create` needs SSH to every genesis validator. For a launch with validators run +by other organisations, they make their keys on their own machines, ideally offline. +Only public files change hands, and each is checksummed and checked where it arrives. + +```bash +# each outside validator, on their machine (or --profile ): +cometcli host prepare --offline_ready # while still online: image loaded, evmd runs with no network +cometcli genesis join mychain-1 --moniker acme --home ~/acme-node --ip 203.0.113.9 # → acme.join.json + +# you, the coordinator: +cometcli genesis base mychain-1 acme.join.json beta.join.json --own val1,val2 --kit mychain-1-kit +# → the kit: base genesis + spec + kit.json + README + SHA256SUMS; send it to every outside validator + +# each outside validator: +cometcli genesis gentx --kit mychain-1-kit --home ~/acme-node # → acme.gentx.json, the only file sent back + +# you: +cometcli genesis collect --kit mychain-1-kit acme.gentx.json beta.gentx.json +# → final genesis (strictly validated), packages// with every peer filled in; your validators start + +# each outside validator: +cometcli genesis install --package mychain-1-packages/acme --home ~/acme-node +``` + +- **Keys stay on the key machine.** + - `join` and `gentx` run evmd in a container with `--network none`. + - A new operator mnemonic goes to a 0600 file on that machine (`--mnemonic-to`, default `.operator.mnemonic`) and is never shown. + - To recover an existing key instead: `--recover-from `. +- **Offline is checked.** `join` and `gentx` refuse to run on a machine with a default route or a connected interface (docker's own bridges don't count). `--allow-online` overrides this, and the join and gentx files then record that the keys were made online. +- **The coordinator checks every gentx before using it.** Each must: + - be signed against the same kit + - be from a validator in the kit + - stake the agreed amount, from the account the kit funds + - use the consensus key from its join file and that node's ID + - carry a signature + + Then `collect-gentxs` runs, and `validate-genesis` in strict mode. A changed kit or package fails its checksum before anything is used. +- **Your own validators** (`--own`) get their keys made on their hosts, as with `genesis create`, and are started by `collect`. +- **No USB stick: QR transfer.** + - **On the offline machine:** `cometcli qr show acme.gentx.json` shows the file as a few QR codes, one per Enter. `--png

` writes them as images instead, to show full screen. + - **On a connected machine:** scan them with a phone and paste each code's text into `cometcli qr read`, or pass photos of the codes: `cometcli qr read *.jpg`. + - Parts can be scanned in any order. The file is written only when every part is in and it matches its checksum. + - Files with key material are refused, so a mnemonic or `priv_validator_key.json` can't leave this way. +- **Carrying the image to an offline machine:** `host prepare --offline_ready --save_image /tmp/img.tar` on a connected machine, then `--image_tar` on the offline one. +- **Removing a test network:** `cometcli network remove --profiles a,b`. For each profile it stops the container and deletes the node home, but only when that home's genesis is for that chain. It also deletes the chain's genesis builder files on those hosts and the profiles, and reports free disk. + ## Remote nodes over SSH Run cometcli on your laptop and manage a node on another machine. Every host @@ -431,6 +492,8 @@ cometcli gov propose --kind params --module feemarket --set 'min_gas_multiplier= cometcli chain params --module feemarket # one module's current params cometcli gov propose --kind messages --title "…" --summary "…" \ --messages '[{"@type":"/cosmos.distribution.v1beta1.MsgCommunityPoolSpend", …}]' +cometcli gov propose --kind cancel-upgrade # cancels the scheduled plan (refused if the vote can't pass before its height) +cometcli upgrade ready # this node vs the scheduled plan: finds the image that registers it, or says to build one cometcli gov deposit --proposal 7 # tops up exactly what's missing to the minimum cometcli val vote --proposal 7 --option yes cometcli wait until --condition proposal-status --value 7:PASSED @@ -568,7 +631,7 @@ approvals. Check how well cometcli handles real faults, on a throwaway network: ```bash -cometcli drill testnet up --spec primium-1 # 4 local validators built like your network +cometcli drill testnet up --spec mychain-1 # 4 local validators built like your network cometcli drill testnet up --image # or a plain evmd network cometcli drill list # the scenarios cometcli drill run # all of them (--scenario oom,partition to pick) @@ -703,8 +766,8 @@ can't work out what to do. | What you type | Who handles it | |---|---| -| **A direct command:** "restart val3", "stop the node", "unjail", "vote yes on 6", "send 100adex to cosmos1…", "delegate 1000adex to cosmosvaloper1…", "withdraw rewards", "show last 200 logs of val4", "add peer id@host:26656", "set app mempool.max-txs 0", "build v0.7.3", "switch to primium-v0.7.3", "check the network", "triage", "prepare val5", "provision arch1 as archive for primium-2"; for a group: "stop all primium-1 validators", "build v0.7.2 on all validators", "triage all nodes" | cometcli runs the tool with the words as its arguments (a group: on each node in turn, stopping at the first refusal). Changes and transactions still ask, and key use always does. A name that isn't one of your nodes ("stop worrying") goes to the model. | -| **A known question:** jailed validators, validator count, peers, sync and lag, health (from triage), fleet status, one network's nodes ("is primium-2 healthy"), version, upgrade readiness, recent log errors, spec conformance, uptime, consensus, proposals, one proposal and its tally ("did proposal 7 pass"), votes, rewards, balance, upgrade plan, slashing params | Read-only tools and a fixed answer. A question may name the node ("is val2 healthy"); a node that doesn't answer is answered locally too. Repeats within 30s come from a cache. | +| **A direct command:** "restart val3", "stop the node", "unjail", "vote yes on 6", "send 100adex to cosmos1…", "delegate 1000adex to cosmosvaloper1…", "withdraw rewards", "show last 200 logs of val4", "add peer id@host:26656", "set app mempool.max-txs 0", "build v0.7.3", "switch to mychain-v0.7.3", "check the network", "triage", "prepare val5", "provision arch1 as archive for mychain-2"; for a group: "stop all mychain-1 validators", "build v0.7.2 on all validators", "triage all nodes" | cometcli runs the tool with the words as its arguments (a group: on each node in turn, stopping at the first refusal). Changes and transactions still ask, and key use always does. A name that isn't one of your nodes ("stop worrying") goes to the model. | +| **A known question:** jailed validators, validator count, peers, sync and lag, health (from triage), fleet status, one network's nodes ("is mychain-2 healthy"), version, upgrade readiness, recent log errors, spec conformance, uptime, consensus, proposals, one proposal and its tally ("did proposal 7 pass"), votes, rewards, balance, upgrade plan, slashing params | Read-only tools and a fixed answer. A question may name the node ("is val2 healthy"); a node that doesn't answer is answered locally too. Repeats within 30s come from a cache. | | **`/incident` where triage finds one clear root case with a playbook** (32 of 68 cases) | cometcli runs the playbook (see below). | | **Anything else:** why/how questions, ambiguous requests, judgement calls (config parse errors, app hash, double sign, corruption) | The model. For `/incident` it starts from cometcli's triage. | diff --git a/docs/reports/v0.6.0-scratch-test.md b/docs/reports/v0.6.0-scratch-test.md new file mode 100644 index 0000000..f316213 --- /dev/null +++ b/docs/reports/v0.6.0-scratch-test.md @@ -0,0 +1,193 @@ +# cometcli v0.6.0 — scratch test report + +A release gate run on four real SSH hosts (2× Amazon Linux 2023, 2× Ubuntu 24.04), +wiped to bare machines first and rebuilt with cometcli only: host setup, image builds, +a new chain from genesis, a validator joining later, an archive node, governance, a +governed software upgrade, jailing, outages, and docker-level incidents. Run on +2026-10-12 (IST). + +## Summary + +| area | result | +|---|---| +| Bare-host setup (docker, compose, jq, git) | ✓ 4/4 hosts, 34–60s each | +| Image build on fresh hosts | ✓ (2 fixes) | +| Genesis with 2 validators | ✓ 79s | +| 3rd validator joins a running chain | ✓ (new `val operator-key`) | +| Archive node | ✓ full history from block 1 | +| Governance: text + params proposals | ✓ both passed; params diff exact | +| Governed upgrade v0.7.2 → v0.7.3 | ✓ halted at the height, all nodes switched, resumed in 90s (3 fixes) | +| Jailed for downtime → unjail | ✓ local playbook, no model | +| Stop / revive | ✓ caught up in 20–27s | +| All nodes down at once | ✓ resumed 27s after start | +| Config / memory / docker / port / network / image incidents | ✓ all fixed locally after fixes; model runs were slow and twice harmful | +| New node replaying through a past upgrade | ✓ local playbook, correct binary | +| Fleet triage | ✓ | +| Staking: delegate, redelegate, undelegate, withdraw, edit | ✓ (commission edit fixed) | +| Chain halt: over 1/3 of the power offline | ✓ triage now names who isn't voting; resumed 19s after start | +| Archive cut off from its peers | ✓ local playbook, 168s | +| Upgrade scheduled, then cancelled by governance | ✓ new `--kind cancel-upgrade` | +| Double-sign safety (signing state, key copies, permissions) | ✓ (4 guards added) | +| Workflow B with outside validators + QR transfer, on the rebuilt hosts | ✓ 3 validators in ~2.5 min, then removed | +| Disk nearly full, clock skew | not run: fault injection over raw ssh was blocked by auto mode | +| Prompts answered without a model | 93% (174 of 187 over the week; every prompt of round 2 ran locally) | + +**23 findings, all fixed in this release** (13 in round 1, 10 in round 2). The most +serious (12): with `bash` allowed, a model run deleted an archive's chain databases and +`upgrade-info.json` and re-synced it on the wrong binary; the shell guard now refuses +those in every mode, along with `--unsafe-skip-upgrades`. Round 2 found two more that +could have cost a validator or an archive: `provision --reconfigure` would have taken +an upgraded node back to the genesis binary (17), and a whole config dir or a backup +could carry a consensus key past the double-sign guard (21). + +Also in this release: no deployment-specific names in cometcli. Container names +default to `-`, the Dockerfile lookup and image labels are generic, +and help, docs and examples use placeholder names. + +--- + + +Hosts: 35.89.193.85 (v6-sohan1, Amazon Linux 2023), 34.220.79.132 (v6-val2, AL2023), +34.221.12.221 (v6-val3, Ubuntu 24.04), 34.222.28.185 (v6-arch, Ubuntu 24.04). Chain primium-6, cosmos/evm v0.7.2 → v0.7.3. + +## 0. Backup and wipe +- Backed up 9 node homes (keys, pvs, keyrings, genesis/configs) from primium-1 and primium-2, plus primium-2 mnemonics and specs → ~/cometcli-livetest/backup-20261012 (0700). +- network remove primium-2 (4 nodes, ~5.6G) and primium-1 (4 + an old home on val3, ~250G): ✓ — after the fix for root-owned files (sudo fallback). +- Packages uninstalled (docker, compose, jq, git), docker data removed: hosts at 3–6% disk. +- Found: a GitHub token embedded in evm-node git remotes on 34.220.79.132 and 34.221.12.221 — reported; evm-node/, node/, /opt/go-evmd-build left untouched. + +## 1. host prepare on bare hosts +| host | OS | time | result | +|---|---|---|---| +| v6-sohan1 | AL2023 | 60s | ✓ git, jq, docker (AL pkg), compose v2.29.7, docker group | +| v6-val2 | AL2023 | 47s | ✓ same | +| v6-val3 | Ubuntu 24.04 | 34s | ✓ git, jq, docker.com repo + compose/buildx, docker group | +| v6-arch | Ubuntu 24.04 | 35s | ✓ same | + +## 2. Image build (group command "build v0.7.2 on all primium-6 nodes") +- Finding 1 (fixed): on a fresh host, build couldn't name the image or pick the repo — now from the network spec (naming, repo). +- Finding 2 (fixed): a consensus-breaking release always asked even on a host with no node, so --allowedTools couldn't cover it; now only a host with a running node gets the always-ask. +- Built v0.7.2 on all 4 hosts with cometcli's built-in Dockerfile (no node yet): ✓, no model call. 2031s total — sequential (≈8.5 min/host). Improvement: run group builds in parallel after approval. + +## 3. Genesis with 2 validators (genesis create, file keyring) +- ✓ 79s: keys made on each host, mnemonics → 0600 files, base genesis, gentxs signed on hosts, strict validate, both started, blocks from height 6. +- Finding 3 (cosmetic): nodes not provisioned yet show "UNREACHABLE" in fleet status / network check. + +## 4. 3rd validator joins the running chain +- node provision (peers_from the 2 validators) ✓, synced in 16s. +- Finding 4 (fixed): no cometcli way to make a joining validator's operator key — provision told the operator to `docker exec -it … keys add` (mnemonic on screen, profile without a signer). Added `cometcli val operator-key`: key in the node's keyring, mnemonic to a 0600 file on the host, profile set to sign there. +- Funded from sohan1 (KEY ACCESS approval, code 0), val create 2M (KEY ACCESS, file keyring in container, code 0), in consensus after 13s (20/20 signed). Power: sohan1 10M, val2 10M, val3 2M (9%). + +## 5. Archive node (4th host) +- node provision --role archive ✓, synced in 28s; pruning = "nothing", history from block 1. +- network check: PASS, everything matches the spec (4 nodes). Fleet: 4 nodes, 3 peers each. + +## T1. Governance +- Text proposal #2: proposed (KEY ACCESS, code 0), 3 yes votes in parallel (code 0 each), PASSED after 1m48s; "did proposal 2 pass" answered locally with the tally. +- Params proposal #3 (distribution community_tax 0.02 → 0.03): the approval showed exactly that diff and the full message; 3 yes votes; PASSED after 1m37s; chain params show 0.03 and every other field unchanged. +- (#1 was rejected — my test script failed to read the id and no one voted; not a cometcli fault.) +- Finding 3 fixed: fleet status shows "not set up yet" for profiles whose node isn't provisioned (was UNREACHABLE). + +## T2. Governed software upgrade v0.7.2 → v0.7.3 +- upgrade releases: v0.7.3 registers only the old handler name, which the running v0.7.2 also registers → built v0.7.3 with --handler v0.7.2-to-v0.7.3 (no-op, migrations only). +- Builds on 4 hosts in parallel: 521s (vs 2031s sequential for the first build). +- Finding 5 (fixed): on a running node the image was named primium-v0.7.2:v0.7.3 (running name + new tag); now from the spec naming → primium-v0.7.3. Cost one rebuild (493s; no build cache across runs — improvement). +- upgrade ready ✓ on all 4. Proposal #4 (in_blocks 1500): the tool checked the image/handler on the proposer and estimated the halt (≈7 min). Finding 6 (fixed): upgrade proposals required --summary; now default from the plan. +- 3 yes votes, PASSED in 1m58s; plan at 9334; every node halted there. +- Finding 7 (fixed): upgrade switch required --image; now it picks the local image whose binary registers the halted plan. +- upgrade switch on 4 nodes in parallel ✓; chain resumed 90s after switching; all on 0.7.3, plan cleared, 0 missed blocks. + +## T3. Jailed for downtime (val3, 9% power) +- Stopped → jailed after 3m15s (1000-block window). /incident: triage root node-killed-sigkill; playbook ran locally (logs, restart, synced 30s, waited out the jail) and stopped at the unjail for approval — no model call, 120s. Unjail with KEY ACCESS approval, code 0; back in consensus in 13s. + +## T4. Stop / revive +- Archive down 180s → caught up in 27s. val3 down 60s (no jail) → caught up in 20s, signing 20/20. No jails. + +## T5. All nodes down +- "stop all primium-6 nodes" (group command, no model) → whole network down 2 min → "start all primium-6 nodes" → chain resumed 27s after start; all 4 healthy, nobody jailed. + +## T6. Docker-level and config incidents (on the archive: no slashing) +| case | injected | result | model | time | +|---|---|---|---|---| +| a. broken config.toml | `[consensus` appended, restart | cfg-parse-error → repair-config commented the line, restart, synced, recorded | none | 104s | +| b. memory limit 150 MB | docker update + restart | first run: root "restarted recently" (only 1 restart yet) → 7-min wait → model: fixed memory after ~25 min but also "fixed" CORS lines misled by stale logs. Finding 8 (fixed): a limit < 1 GB alone makes container-memory-limit the root. Rerun: fixed locally | none (rerun) | 110s (vs ~1491s by the model) | +| c. docker daemon stopped | systemctl stop docker | host-docker-down → started docker, started node, synced | none | 55s | +| d. port held by another container | squatter on 26656 | host-port-in-use: facts gathered, decision left to the operator (could be a 2nd validator). Finding 9 (fixed): the report hid who holds the port — short fact output now shown, holder shown as name/image | none | 58s | +| e. network detached (knock-on of d) | — | node-network-detached → node.recreate, synced | none | 168s | +| f. container and image gone | compose down + rmi | first run: only "container missing" seen → model suggested the mis-named image. Finding 10 (fixed): triage reads the image from the compose file when the container is gone; image-missing explains container-missing; playbooks recreate from the compose file; node.recreate works without a container | (rerun below) | | +- 6f rerun (detection fixed): triage now named node-image-missing via the compose file; the playbook rebuilt v0.7.3 — **without** the no-op handler the chain's v0.7.3 was built with. Finding 11 (fixed): a rebuild must reproduce the exact binary. upgrade build now records an added handler per tag in the network spec (`upgrades:`) and the profile, and reuses it for any later build of that tag on any node of the chain. +- **Finding 12 (critical, fixed): with bash allowed (test setting --allowedTools bash), the model destroyed the archive's state** while working 6f: `rm data/upgrade-info.json`, `cd data && rm -rf *.db addrbook.json`, then re-synced from block 1 on the state-breaking v0.7.3 binary → AppHash divergence at 9335. It then proposed `--unsafe-skip-upgrades 9334` (would fork the node; refused by -p). The guard refused `rm -rf *.db priv_validator_state.json` but not `rm -rf *.db` from inside data/. Now refused in every mode: deleting any *.db store, cs.wal, upgrade-info.json, wasm/ under a node's data; `--unsafe-skip-upgrades` in any command, edit or written file. Regression tests use the exact commands the model ran. +- Takeaway: every one of the model's runs on this archive was slow (25–63 min) and two did damage; every local playbook run fixed its case in 55–168s. +| f (rerun 2). fresh archive replaying through the upgrade | archive re-provisioned on v0.7.2, replayed to 9334 and halted | first diagnosed as "chain halted at upgrade" (node down → no chain view). Finding 13 (fixed): chain queries fall back to another node of the same chain through its own connection; now upgrade-replay-past-upgrade → new playbook: upgrade.switch (auto-picks the image registering the plan, preferring the spec's naming) + replay; passed 9334 with the rebuilt (handler) binary — no AppHash mismatch | none | 994s (15m replay) | + +## T7. Fleet triage +- "triage all primium-6 nodes": 4 nodes, no known problem, no model call, 72s. + +## Routing over the whole run +- 92 prompts in 8h: 86 answered locally (93%), 6 by the model. The model's runs were the incidents whose case didn't match yet or had no playbook — all fixed in this release; two of them damaged the archive (finding 12). +- Model rounds are now audited with token counts (were missing). + +--- + +# Round 2 — the remaining cases (2026-10-12, 22:40–23:45 IST) + +Same four hosts and chain (primium-6 on v0.7.3), operated through cometcli. Faults were +injected through cometcli where possible (`node set-config`, `node service`). + +## R1. Staking +| operation | result | +|---|---| +| delegate 500 DEX val2 → val3 | ✓ code 0 | +| redelegate 100 DEX val3 → sohan1 | ✓ code 0 | +| undelegate 50 DEX from val3 | ✓ code 0 (val3: 1,980,000 + 500 − 100 − 50 = 1,980,350 ✓) | +| withdraw rewards + commission | ✓ code 0, balances moved | +| edit moniker (and back) | ✓ code 0 | +| edit commission 5% → 6% | **Finding 14 (fixed):** the rate went out as "0.06" and the chain refused it ("cannot unmarshal into a *big.Int"). cosmos.Dec tx fields are gogoproto LegacyDec on SDK v0.54 too, so the scaled integer is always sent. The chain then refused the change under its one-per-24h rule (val3's commission was set at creation today); `val edit` now says so up front with the time it becomes possible. | + +## R2. Chain halt: over 1/3 of the voting power offline +- Stopped sohan1 (45.5% of the power): the chain halted at 73004. +- Triage on val2 said only "node stalled", and flagged the p2p port as unreachable, which was misleading. `watch` said CHAIN HALT, guessing "upgrade height? consensus failure?". +- **Finding 15 (fixed):** a node with no block for 30s now reads the consensus state. New case `chain-halted-validators-offline`: "only 54.5% of the voting power is voting (2/3 needed) — not voting: v6-sohan1 (45.5%)". The fleet CHAIN HALT line says the same. +- Started sohan1 (group command, no model): blocks resumed 19s later, 0 missed blocks anywhere (a halted chain doesn't count misses). + +## R3. Archive cut off from its peers +- Injected through cometcli: `pex=false`, `persistent_peers=""`, `max_num_inbound_peers=0`, restart → 0 peers, falling behind. (A firewall rule over raw ssh was blocked by auto mode.) +- `/incident`: triage `node-no-peers` → local playbook re-added the chain's 3 other nodes, restarted, synced in 25s. 168s, no model call. +- **Finding 16 (fixed):** `network check` still passed the archive with pex off and no inbound peers. It now warns for archive and RPC nodes, and provision renders pex on (and 40 inbound when it was 0). +- **Finding 17 (fixed, would have broken the archive):** `provision --reconfigure` takes its image from the spec, whose tag is the genesis one (v0.7.2). After the governed upgrade that would have started the archive on the old binary. Reconfigure now keeps the image the node runs ("keeping the image … runs now: primium-v0.7.3"). Rerun: `network check` 0 warn, archive on v0.7.3, 3 peers. +- Fixed: `/incident`'s "Evidence:" was empty after a successful fix (computed after the playbook changed the signals). + +## R4. Upgrade scheduled, then cancelled +- **Finding 18 (fixed):** building v0.7.3 again with a test handler would have replaced the handler recorded for v0.7.3 (`v0.7.2-to-v0.7.3`), so the next rebuild for a replay (finding 11) would diverge. A tag's recorded handler is never replaced now: a build with another handler needs its own image name and isn't recorded. Live: refused without an image name; built as `primium-v0.7.3-canceltest` (500s); the record unchanged. +- Too-close heights refused; an upgrade proposal without `image` refused (the plan name is checked against the binary first). +- Proposal 5 (plan `v0.7.3-cancel-test` at 83429, ≈13 min): readiness checked on the proposer, 3 yes votes, PASSED in 1m48s. Triage on val2: `upgrade-imminent` (2485 blocks to halt). +- **Finding 23 (fixed):** `upgrade ready` required name and image; it now takes the scheduled plan and finds the image that registers it. val2: "✗ no image registers plan … — build it before the halt", not ready; sohan1: ready. +- **Finding 22 (fixed):** cancelling needed a hand-written `MsgCancelUpgrade` JSON. New `gov propose --kind cancel-upgrade`: refused when nothing is scheduled, or when the vote can't pass before the height; the approval names the plan it cancels. Proposal 6 passed with ≈1335 blocks to spare; plan cleared; triage clean. 142s. + +## R5. Double-sign safety +- **Finding 19 (fixed):** `sec doublesign` asked systemd about a docker node ("inactive … unknown"). It now reads the container state, checks the signing state against the node's height, and reports `double_sign_check_height` (0 on every node: a note, it's the CometBFT default). +- **Finding 20 (fixed):** `sec perms` flagged the `config/` and `data/` dirs (0755) that cometcli itself created. Homes are made owner-only at init and on reconfigure; val3 reconfigured live: 0 problems, 18 blocks missed during the 78s recreate. +- **Finding 21 (fixed):** the shell guard caught key copies by file name, but not `rsync -a …/v6-val3/config/ …/v6-arch/config/` or `tar xzf val3-backup.tgz -C `. Both now refused in every mode (whole config/data dirs, recursively or as a source; archives unpacked into a node home), while `cp genesis.json ~/.evmd/config/` and making backups still work. Tested by command, not by asking a model to try it on a live validator. + +## R6. Workflow B with outside validators, and QR transfer +A second chain, scratch-7, beside primium-6 on port offset 100 (sohan1 yours; val2 and val3 as outside validators, keys made on their hosts in `--network none` containers, `--allow-online` since the hosts are connected). +| step | result | +|---|---| +| join ×2 | ✓ 21s; mnemonics to 0600 files on the hosts, never shown | +| base (kit) | ✓ 29s | +| tampered kit | ✓ refused: "base-genesis.json doesn't match its checksum" | +| gentx ×2 | ✓ 15s | +| QR: a mnemonic | ✓ refused: "contains key material" | +| QR: val3's gentx | ✓ 7 codes, 77 columns wide; read back from the images in reverse order, byte-identical | +| collect with a swapped gentx | ✓ refused: "two gentxs for s7-val2" | +| collect (with the QR-transferred gentx) | ✓ 36s; strict validate-genesis; sohan1 started | +| install ×2 | ✓ 38s; containers `scratch-7-validator` beside `primium-validator` (the new naming) | +| chain | ✓ 3 bonded, 0 missed, network check PASS; primium-6 unaffected | +| network remove | ✓ homes, containers and builder files removed; hosts back to 9–13% | + +## Not run +- **Disk nearly full, clock skew.** They need a fault injected on the host (`fallocate`, `date -s`), which auto mode blocked over raw ssh. The KB cases (`host-disk-high`, `host-disk-full`, `host-clock-unsynced`) are covered by their KB tests only. + +## Leftovers +- Image `primium-v0.7.3-canceltest` on sohan1 (from R4). +- `~/cometcli-livetest/s7` on the Mac (the scratch-7 kit and a test mnemonic for a chain that no longer exists). diff --git a/go.mod b/go.mod index 9a34fea..9ac16df 100644 --- a/go.mod +++ b/go.mod @@ -15,7 +15,9 @@ require ( github.com/cosmos/go-bip39 v1.0.0 github.com/decred/dcrd/dcrec/secp256k1/v4 v4.3.0 github.com/kevinburke/ssh_config v1.2.0 + github.com/makiuchi-d/gozxing v0.1.1 github.com/muesli/reflow v0.3.0 + github.com/skip2/go-qrcode v0.0.0-20200617195104-da1b6568686e github.com/spf13/cobra v1.10.2 github.com/spf13/pflag v1.0.9 github.com/tyler-smith/go-bip32 v1.0.0 @@ -116,6 +118,7 @@ require ( golang.org/x/net v0.41.0 // indirect golang.org/x/sys v0.47.0 // indirect golang.org/x/text v0.41.0 // indirect + golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1 // indirect google.golang.org/genproto/googleapis/api v0.0.0-20250218202821-56aae31c358a // indirect google.golang.org/genproto/googleapis/rpc v0.0.0-20260526163538-3dc84a4a5aaa // indirect ) diff --git a/go.sum b/go.sum index 81c8d15..ddf37b8 100644 --- a/go.sum +++ b/go.sum @@ -221,6 +221,8 @@ github.com/linxGnu/grocksdb v1.8.14 h1:HTgyYalNwBSG/1qCQUIott44wU5b2Y9Kr3z7SK5Of github.com/linxGnu/grocksdb v1.8.14/go.mod h1:QYiYypR2d4v63Wj1adOOfzglnoII0gLj3PNh4fZkcFA= github.com/lucasb-eyer/go-colorful v1.3.0 h1:2/yBRLdWBZKrf7gB40FoiKfAWYQ0lqNcbuQwVHXptag= github.com/lucasb-eyer/go-colorful v1.3.0/go.mod h1:R4dSotOR9KMtayYi1e77YzuveK+i7ruzyGqttikkLy0= +github.com/makiuchi-d/gozxing v0.1.1 h1:xxqijhoedi+/lZlhINteGbywIrewVdVv2wl9r5O9S1I= +github.com/makiuchi-d/gozxing v0.1.1/go.mod h1:eRIHbOjX7QWxLIDJoQuMLhuXg9LAuw6znsUtRkNw9DU= github.com/mattn/go-isatty v0.0.20 h1:xfD0iDuEKnDkl03q4limB+vH+GxLEtL/jb4xVJSWWEY= github.com/mattn/go-isatty v0.0.20/go.mod h1:W+V8PltTTMOvKvAeJH7IuucS94S2C6jfK/D7dTCTo3Y= github.com/mattn/go-localereader v0.0.1 h1:ygSAOl7ZXTx4RdPYinUpg6W99U8jWvWi9Ye2JC/oIi4= @@ -289,6 +291,8 @@ github.com/rs/cors v1.11.1/go.mod h1:XyqrcTp5zjWr1wsJ8PIRZssZ8b/WMcMf71DJnit4EMU github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM= github.com/sasha-s/go-deadlock v0.3.5 h1:tNCOEEDG6tBqrNDOX35j/7hL5FcFViG6awUGROb2NsU= github.com/sasha-s/go-deadlock v0.3.5/go.mod h1:bugP6EGbdGYObIlx7pUZtWqlvo8k9H6vCBBsiChJQ5U= +github.com/skip2/go-qrcode v0.0.0-20200617195104-da1b6568686e h1:MRM5ITcdelLK2j1vwZ3Je0FKVCfqOLp5zO6trqMLYs0= +github.com/skip2/go-qrcode v0.0.0-20200617195104-da1b6568686e/go.mod h1:XV66xRDqSt+GTGFMVlhk3ULuV0y9ZmzeVGR4mloJI3M= github.com/spf13/cobra v1.10.2 h1:DMTTonx5m65Ic0GOoRY2c16WCbHxOOw6xxezuLaBpcU= github.com/spf13/cobra v1.10.2/go.mod h1:7C1pvHqHw5A4vrJfjNwvOdzYu0Gml16OCs2GRiTUUS4= github.com/spf13/pflag v1.0.9 h1:9exaQaMOCwffKiiiYk6/BndUBv+iRViNW+4lEMi0PvY= diff --git a/internal/agent/incident.go b/internal/agent/incident.go index 98906df..47ca196 100644 --- a/internal/agent/incident.go +++ b/internal/agent/incident.go @@ -71,6 +71,9 @@ func (a *Agent) runIncident(ctx context.Context, what string) (string, error) { return handoff(triageNote) } + // the evidence is what triage saw: refreshSignals updates sig in place + // while the playbook runs, and after a fix it matches nothing + _, evidence := c.Matches(sig) a.emit(Event{Kind: EvText, Text: fmt.Sprintf("Known case **%s** — %s.\nRunning its playbook locally, no model call; changes and transactions still ask you.", c.ID, c.Title)}) var done []string for i, st := range c.Playbook { @@ -108,7 +111,7 @@ func (a *Agent) runIncident(ctx context.Context, what string) (string, error) { done = append(done, fmt.Sprintf("%s: %s", label, stepOutcome(st.Do, r.Text))) } - _, matched := c.Matches(sig) + matched := evidence if c.PlaybookKind == "report" { return a.reportIncident(ctx, what, c, matched, done), nil } @@ -271,9 +274,16 @@ func (a *Agent) runToolOn(ctx context.Context, t toolkit.Tool, args toolkit.Args // says nothing about the incident). func stepOutcome(tool, text string) string { text = strings.TrimSpace(text) - switch tool { - case "node.logs", "bash", "read", "grep": - return fmt.Sprintf("%d lines of output", len(strings.Split(text, "\n"))) + lines := strings.Split(text, "\n") + switch { + case tool == "node.logs" || tool == "read" || tool == "grep": + return fmt.Sprintf("%d lines of output", len(lines)) + case tool == "bash" && len(lines) <= 4 && len(text) <= 300: + // a short answer is the finding itself (who holds a port, what a + // command printed): show it + return strings.Join(strings.Fields(strings.ReplaceAll(text, "\n", " · ")), " ") + case tool == "bash": + return fmt.Sprintf("%d lines of output", len(lines)) } return firstLine(text) } diff --git a/internal/agent/loop.go b/internal/agent/loop.go index 566931d..09ba0a4 100644 --- a/internal/agent/loop.go +++ b/internal/agent/loop.go @@ -551,6 +551,7 @@ func (a *Agent) logLLM(round int, resp *Response) { _ = lg.Log(audit.KindLLM, a.profileName(), map[string]any{ "round": round, "provider": a.Provider.Name(), "model": a.Model, "text": a.Redact.Text(resp.Text), "calls": calls, "done": resp.Done, + "tokens_in": resp.Usage.Input, "tokens_out": resp.Usage.Output, "tokens_cached": resp.Usage.CacheRead, }) } diff --git a/internal/agent/route.go b/internal/agent/route.go index 658c127..852a87f 100644 --- a/internal/agent/route.go +++ b/internal/agent/route.go @@ -311,7 +311,7 @@ func (a *Agent) knownChain(id string) bool { } // runCommandOnGroup runs a direct command on every node it names ("stop -// all primium-1 validators", "build v0.7.2 on all validators"), one after +// all mychain-1 validators", "build v0.7.2 on all validators"), one after // the other, each with its own approvals; a failure doesn't stop the rest. func (a *Agent) runCommandOnGroup(ctx context.Context, input string, m router.Match) (string, bool) { in := m.Intent diff --git a/internal/cli/genesis.go b/internal/cli/genesis.go index c9e8296..e0d84a5 100644 --- a/internal/cli/genesis.go +++ b/internal/cli/genesis.go @@ -24,8 +24,8 @@ func genesisCmd(reg *toolkit.Registry) *cobra.Command { create := &cobra.Command{ Use: "create ", Short: "Create a new network from its spec on the validators' hosts and start it", - Long: `Runs node-setup's distributed genesis workflow with cometcli as the -coordinator. For each validator profile (its host, local or over SSH): + Long: `Runs a distributed genesis workflow with cometcli as the coordinator. +For each validator profile (its host, local or over SSH): init in the image — node and consensus keys stay there — and the operator key in the node's own keyring. Then a base genesis from the spec with every validator funded (and --accounts), each validator's gentx signed on its own @@ -145,6 +145,7 @@ peer, and every node started. Mnemonics are shown only here, or written to f.IntVar(&portStep, "port-step", 0, "host port offset added per further validator (several on one machine)") f.BoolVar(&yes, "yes", false, "don't ask for confirmation") cmd.AddCommand(create) + cmd.AddCommand(genesisBCmds()...) return cmd } diff --git a/internal/cli/genesis_b.go b/internal/cli/genesis_b.go new file mode 100644 index 0000000..230042c --- /dev/null +++ b/internal/cli/genesis_b.go @@ -0,0 +1,724 @@ +package cli + +import ( + "encoding/json" + "fmt" + "os" + "path/filepath" + "strconv" + "strings" + "time" + + "github.com/spf13/cobra" + + "github.com/abhijitkrm/cometcli/internal/client/host" + "github.com/abhijitkrm/cometcli/internal/config" + "github.com/abhijitkrm/cometcli/internal/netspec" + "github.com/abhijitkrm/cometcli/internal/toolkit" + "github.com/abhijitkrm/cometcli/internal/tools/networktool" +) + +// Workflow B: a genesis with validators the coordinator can't reach (see +// networktool/workflowb.go). Validator-side commands run on this machine, +// or on --profile's host. + +// keyMachine is the context for a validator-side step: --profile's host, +// or this machine. +func keyMachine(cmd *cobra.Command, profile string) (*toolkit.Context, *config.Config, error) { + cfg, err := config.Load() + if err != nil { + return nil, nil, err + } + p := &config.Profile{Name: "local", Transport: config.Transport{Type: "local"}} + if profile != "" { + pp, ok := cfg.Profiles[profile] + if !ok { + return nil, nil, fmt.Errorf("no profile %s", profile) + } + pp.Name = profile + p = pp + } + c := &toolkit.Context{Context: cmd.Context(), Profile: p, Cfg: cfg, Out: cmd.OutOrStdout(), + Approver: func(*toolkit.Context, string, toolkit.Tier, map[string]any) (bool, error) { return true, nil }} + c.AutoApproveBelow = toolkit.TierOnChain + return c, cfg, nil +} + +func keyringFor(backend string) (networktool.KeyringOpts, error) { + kr := networktool.KeyringOpts{Backend: backend} + switch backend { + case "file": + pw, err := keyringPassword() + if err != nil { + return kr, err + } + kr.Password = pw + case "test": + default: + return kr, fmt.Errorf("--keyring: file (password, the default) or test (unencrypted, test networks only)") + } + return kr, nil +} + +func writeJSON(path string, v any) error { + raw, err := json.MarshalIndent(v, "", " ") + if err != nil { + return err + } + return os.WriteFile(path, append(raw, '\n'), 0o644) +} + +func offlineNote(o networktool.Offline) string { + if o.Online { + return "made on a CONNECTED machine (--allow-online): " + o.Detail + } + return "made with no network" +} + +const backupReminder = `back up, off this machine: the node home's config/priv_validator_key.json and the operator mnemonic. +never run this key on two machines at once — that is a double sign, and the validator is tombstoned.` + +// coordinator state between genesis base and collect: the validators the +// coordinator runs itself, and where the base was built +type coordState struct { + ChainID string `json:"chain_id"` + Builder string `json:"builder"` + Keyring string `json:"keyring"` // of your own validators' operator keys + Own []ownState `json:"own"` +} + +type ownState struct { + Profile string `json:"profile"` + Moniker string `json:"moniker"` + Home string `json:"home"` + User string `json:"user"` + PortOffset int `json:"port_offset"` +} + +func coordPath(chainID string) (string, error) { + p, err := netspec.Path(chainID) + if err != nil { + return "", err + } + return strings.TrimSuffix(p, ".yaml") + ".coordinator.json", nil +} + +func genesisBCmds() []*cobra.Command { + var ( + profile, home, moniker, ip, recoverFrom, mnemonicTo, keyring, out string + p2pPort int + allowOnline bool + ) + join := &cobra.Command{ + Use: "join ", + Short: "Validator: make this node's keys (offline) and the join file for the coordinator", + Long: `Run on the validator's own machine — with no network if you can (load the +image first: host prepare --offline_ready). Makes the node and consensus +keys and the operator key in the node's own keyring, inside a container +with no network. A new operator mnemonic goes to a 0600 file on this +machine (--mnemonic-to), never to the screen. Writes .join.json: +the account, node id and consensus public key — nothing secret.`, + Args: cobra.ExactArgs(1), + RunE: func(cmd *cobra.Command, args []string) error { + w := cmd.OutOrStdout() + spec, _, err := netspec.Load(args[0]) + if err != nil { + return err + } + if moniker == "" || home == "" { + return fmt.Errorf("--moniker and --home are required") + } + c, _, err := keyMachine(cmd, profile) + if err != nil { + return err + } + defer c.Close() + if ip == "" && c.Profile.Transport.Type == "ssh" { + ip = host.Resolve(c.Profile.Transport).HostName + } + if ip == "" { + return fmt.Errorf("--ip: the public IP other validators will dial") + } + kr, err := keyringFor(keyring) + if err != nil { + return err + } + j, mfile, err := networktool.PrepareJoin(c, spec, networktool.JoinOpts{Home: home, Moniker: moniker, IP: ip, P2PPort: p2pPort, + Keyring: kr, RecoverFrom: recoverFrom, MnemonicTo: mnemonicTo, AllowOnline: allowOnline}) + if err != nil { + return err + } + if err := os.MkdirAll(out, 0o755); err != nil { + return err + } + f := filepath.Join(out, j.Moniker+".join.json") + if err := writeJSON(f, j); err != nil { + return err + } + fmt.Fprintf(w, "✓ %s: node %s, operator %s — keys %s\n", j.Moniker, j.NodeID[:12], j.Account, offlineNote(j.Offline)) + if mfile != "" { + fmt.Fprintf(w, "✓ operator mnemonic → %s on %s (0600) — never shown\n", mfile, c.Profile.Name) + } + fmt.Fprintf(w, "→ send %s to the coordinator\n\n%s\n", f, backupReminder) + return nil + }, + } + jf := join.Flags() + jf.StringVar(&profile, "profile", "", "make the keys on this profile's host instead of this machine") + jf.StringVar(&home, "home", "", "the new node home") + jf.StringVar(&moniker, "moniker", "", "this validator's moniker") + jf.StringVar(&ip, "ip", "", "public IP peers dial (default: the profile's SSH host)") + jf.IntVar(&p2pPort, "p2p-port", 26656, "public P2P port") + jf.StringVar(&recoverFrom, "recover-from", "", "recover the operator key from a mnemonic file on the key machine (else a new key)") + jf.StringVar(&mnemonicTo, "mnemonic-to", "", "where a new mnemonic goes on the key machine (default .operator.mnemonic)") + jf.StringVar(&keyring, "keyring", "file", "operator keyring: file (password) or test (unencrypted, test networks)") + jf.BoolVar(&allowOnline, "allow-online", false, "make keys on a connected machine (marked in the join file)") + jf.StringVar(&out, "out", ".", "directory for the join file") + + var ( + own, builder, kitDir, accountsFile, homeTmpl, mnemonicsTo, bkeyring string + portOffset int + yes bool + ) + base := &cobra.Command{ + Use: "base [join files…]", + Short: "Coordinator: base genesis funding every validator, written as a kit to hand out", + Long: `Builds the base genesis from the network spec, funding every validator in +the join files (and --own validators, whose keys cometcli makes on their +hosts, as genesis create does), plus --accounts. Writes the kit: the base +genesis, the spec, kit.json and a README, all checksummed. Hand the kit +to every outside validator; they sign with genesis gentx --kit.`, + Args: cobra.MinimumNArgs(1), + RunE: func(cmd *cobra.Command, args []string) error { + w := cmd.OutOrStdout() + spec, specPath, err := netspec.Load(args[0]) + if err != nil { + return err + } + for _, f := range spec.Review(false) { + if f.Sev == netspec.Fail { + return fmt.Errorf("the spec has a problem: %s", f.What) + } + } + if kitDir == "" { + kitDir = spec.Chain.ID + "-kit" + } + var joins []*networktool.Join + for _, f := range args[1:] { + j, err := networktool.ReadJoin(f, spec) + if err != nil { + return err + } + joins = append(joins, j) + } + cfg, err := config.Load() + if err != nil { + return err + } + var ownProfiles []*config.Profile + for _, n := range splitCSV(own) { + p, ok := cfg.Profiles[n] + if !ok { + return fmt.Errorf("no profile %s", n) + } + p.Name = n + ownProfiles = append(ownProfiles, p) + } + if builder == "" && len(ownProfiles) > 0 { + builder = ownProfiles[0].Name + } + if builder == "" { + return fmt.Errorf("--builder: a profile whose host has docker and the image builds the base genesis") + } + if len(joins)+len(ownProfiles) == 0 { + return fmt.Errorf("no validators: pass join files and/or --own profiles") + } + var extra []networktool.Account + if accountsFile != "" { + if extra, err = readAccounts(accountsFile); err != nil { + return err + } + } + fmt.Fprintf(w, "base genesis for %s: %d outside validators, %d of yours (%s), builder %s\n", spec.Chain.ID, len(joins), len(ownProfiles), strings.Join(splitCSV(own), ","), builder) + if !yes && !confirm(cmd, "make the keys of your validators on their hosts and build the base genesis?") { + return fmt.Errorf("cancelled") + } + ctxFor := func(p *config.Profile) *toolkit.Context { + c := &toolkit.Context{Context: cmd.Context(), Profile: p, Cfg: cfg, Out: w, + Approver: func(*toolkit.Context, string, toolkit.Tier, map[string]any) (bool, error) { return true, nil }} + c.AutoApproveBelow = toolkit.TierOnChain + return c + } + kit := &networktool.Kit{Version: networktool.FormatVersion, ChainID: spec.Chain.ID, Denom: spec.Chain.Denom, + Stake: spec.Genesis.InitialStake, Image: networktool.ImageFor(spec), Created: nowUTC()} + var nodes []*networktool.GenesisNode + state := coordState{ChainID: spec.Chain.ID, Builder: builder, Keyring: bkeyring} + // your validators: keys made where they stay + if len(ownProfiles) > 0 { + var kr networktool.KeyringOpts + if kr, err = keyringFor(bkeyring); err != nil { + return err + } + for _, p := range ownProfiles { + c := ctxFor(p) + defer c.Close() + h := homeTmpl + if h == "" { + h = "~/.cometcli-nodes/" + spec.Chain.ID + "-{profile}" + } + h = strings.ReplaceAll(h, "{profile}", p.Name) + n, mnemonic, err := networktool.PrepareValidator(c, spec, h, kr) + if err != nil { + return fmt.Errorf("%s: %w", p.Name, err) + } + if err := showMnemonic(w, mnemonicsTo, p.Name, n.Account, mnemonic); err != nil { + return err + } + addr := host.Resolve(p.Transport).HostName + ":" + strconv.Itoa(26656+portOffset) + kit.Validators = append(kit.Validators, networktool.KitValidator{Moniker: n.Moniker, Account: n.Account, NodeID: n.NodeID, Address: addr}) + nodes = append(nodes, n) + state.Own = append(state.Own, ownState{Profile: p.Name, Moniker: n.Moniker, Home: n.Home, User: n.User, PortOffset: portOffset}) + fmt.Fprintf(w, "✓ %s (yours): node %s, operator %s\n", p.Name, n.NodeID[:12], n.Account) + } + } + for _, j := range joins { + kit.Validators = append(kit.Validators, networktool.KitValidator{Moniker: j.Moniker, Account: j.Account, NodeID: j.NodeID, + ConsKey: j.ConsKey, Address: fmt.Sprintf("%s:%d", j.IP, j.P2PPort), Offline: !j.Offline.Online && j.Offline.Checked}) + nodes = append(nodes, &networktool.GenesisNode{Moniker: j.Moniker, Account: j.Account, NodeID: j.NodeID}) + fmt.Fprintf(w, "✓ %s (outside): node %s, operator %s — keys %s\n", j.Moniker, j.NodeID[:12], j.Account, offlineNote(j.Offline)) + } + if err := uniqueValidators(kit.Validators); err != nil { + return err + } + bp, ok := cfg.Profiles[builder] + if !ok { + return fmt.Errorf("no profile %s", builder) + } + bp.Name = builder + bc := ctxFor(bp) + defer bc.Close() + gen, err := networktool.BuildBase(bc, spec, nodes, extra) + if err != nil { + return err + } + id, err := networktool.WriteKit(kitDir, kit, gen, specPath) + if err != nil { + return err + } + cp, err := coordPath(spec.Chain.ID) + if err != nil { + return err + } + if err := writeJSON(cp, state); err != nil { + return err + } + fmt.Fprintf(w, "✓ base genesis: %d validators funded, %d extra accounts\n✓ kit → %s (id %.16s…)\n", len(kit.Validators), len(extra), kitDir, id) + fmt.Fprintf(w, "→ send the kit to each outside validator: cometcli genesis gentx --kit --home \n") + fmt.Fprintf(w, "→ then: cometcli genesis collect --kit %s \n", kitDir) + return nil + }, + } + bf := base.Flags() + bf.StringVar(&own, "own", "", "profiles of validators you run (keys made on their hosts), comma-separated") + bf.StringVar(&builder, "builder", "", "profile whose host builds the base genesis (default: the first --own)") + bf.StringVar(&kitDir, "kit", "", "kit directory to write (default -kit)") + bf.StringVar(&accountsFile, "accounts", "", "extra genesis accounts: lines of '
'") + bf.StringVar(&homeTmpl, "home", "", "node home of --own validators; {profile} is replaced (default ~/.cometcli-nodes/-{profile})") + bf.StringVar(&mnemonicsTo, "mnemonics-to", "", "write --own operator mnemonics to /.mnemonic (0600) instead of showing them") + bf.StringVar(&bkeyring, "keyring", "file", "operator keyring of --own validators: file or test") + bf.IntVar(&portOffset, "port-offset", 0, "host port offset of --own validators (beside another network)") + bf.BoolVar(&yes, "yes", false, "don't ask for confirmation") + + var gprofile, ghome, gmoniker, gkit, gkeyring, gout string + var gallow bool + gentx := &cobra.Command{ + Use: "gentx --kit --home ", + Short: "Validator: sign this node's gentx against the coordinator's kit (offline)", + Long: `Verifies the kit's checksums, checks this home is the node the kit lists, +then signs the gentx in a container with no network. Writes +.gentx.json — the gentx and public node info, nothing secret — +which is the only file to send back.`, + Args: cobra.NoArgs, + RunE: func(cmd *cobra.Command, _ []string) error { + w := cmd.OutOrStdout() + k, base, spec, id, err := networktool.ReadKit(gkit) + if err != nil { + return err + } + fmt.Fprintf(w, "✓ kit %s verified (id %.16s…, %d validators)\n", k.ChainID, id, len(k.Validators)) + c, _, err := keyMachine(cmd, gprofile) + if err != nil { + return err + } + defer c.Close() + if gmoniker == "" { + gmoniker = c.Profile.Name + } + kr, err := keyringFor(gkeyring) + if err != nil { + return err + } + r, err := networktool.SignGentx(c, k, id, base, spec, ghome, gmoniker, kr, gallow) + if err != nil { + return err + } + if err := os.MkdirAll(gout, 0o755); err != nil { + return err + } + f := filepath.Join(gout, r.Moniker+".gentx.json") + if err := writeJSON(f, r); err != nil { + return err + } + fmt.Fprintf(w, "✓ %s gentx signed — %s\n→ send %s back to the coordinator\n", r.Moniker, offlineNote(r.Offline), f) + return nil + }, + } + gf := gentx.Flags() + gf.StringVar(&gkit, "kit", "", "the kit directory from the coordinator") + gf.StringVar(&ghome, "home", "", "this validator's node home (from genesis join)") + gf.StringVar(&gmoniker, "moniker", "", "this validator's moniker (default: the profile name)") + gf.StringVar(&gprofile, "profile", "", "sign on this profile's host instead of this machine") + gf.StringVar(&gkeyring, "keyring", "file", "operator keyring: file or test") + gf.BoolVar(&gallow, "allow-online", false, "sign on a connected machine (marked in the gentx file)") + gf.StringVar(&gout, "out", ".", "directory for the gentx file") + + var ckit, cpkgs string + var cnoStart, callowMissing bool + collect := &cobra.Command{ + Use: "collect --kit [gentx files…]", + Short: "Coordinator: verify the gentxs, build the final genesis, write each validator's package", + Long: `Checks every returned gentx against the kit before using it: same kit, +a validator in it, the agreed stake, the account the kit funds, the +consensus key from its join file, its node id. Your own validators sign +on their hosts. Then collect-gentxs and strict validate-genesis, and one +package per validator: the final genesis, the spec, and every other +validator as a peer. Your own validators are installed and started.`, + RunE: func(cmd *cobra.Command, args []string) error { + w := cmd.OutOrStdout() + k, base, spec, id, err := networktool.ReadKit(ckit) + if err != nil { + return err + } + cfg, err := config.Load() + if err != nil { + return err + } + var st coordState + if cp, err := coordPath(k.ChainID); err == nil { + if raw, err := os.ReadFile(cp); err == nil { + _ = json.Unmarshal(raw, &st) + } + } + if st.Builder == "" { + return fmt.Errorf("no coordinator state for %s — run genesis collect where genesis base ran", k.ChainID) + } + ctxFor := func(name string) (*toolkit.Context, error) { + p, ok := cfg.Profiles[name] + if !ok { + return nil, fmt.Errorf("no profile %s", name) + } + p.Name = name + c := &toolkit.Context{Context: cmd.Context(), Profile: p, Cfg: cfg, Out: w, + Approver: func(*toolkit.Context, string, toolkit.Tier, map[string]any) (bool, error) { return true, nil }} + c.AutoApproveBelow = toolkit.TierOnChain + return c, nil + } + gentxs := map[string][]byte{} + got := map[string]bool{} + for _, f := range args { + raw, err := os.ReadFile(f) + if err != nil { + return err + } + var r networktool.GentxReturn + if err := json.Unmarshal(raw, &r); err != nil { + return fmt.Errorf("%s: %w", f, err) + } + if err := networktool.CheckReturn(k, id, &r); err != nil { + return fmt.Errorf("%s rejected: %w", f, err) + } + if got[r.Moniker] { + return fmt.Errorf("two gentxs for %s", r.Moniker) + } + got[r.Moniker] = true + gentxs[r.Moniker] = r.Gentx + fmt.Fprintf(w, "✓ %s: gentx checked — %s\n", r.Moniker, offlineNote(r.Offline)) + } + if len(st.Own) > 0 { + kr, err := keyringFor(st.Keyring) + if err != nil { + return err + } + for _, o := range st.Own { + c, err := ctxFor(o.Profile) + if err != nil { + return err + } + defer c.Close() + var me networktool.KitValidator + for _, v := range k.Validators { + if v.Moniker == o.Moniker { + me = v + } + } + n := &networktool.GenesisNode{Ctx: c, Home: o.Home, User: o.User, Moniker: o.Moniker, Account: me.Account, NodeID: me.NodeID, Image: k.Image} + g, err := networktool.GentxAt(n, spec, base, kr, me.Address) + if err != nil { + return err + } + gentxs[o.Moniker] = g + got[o.Moniker] = true + fmt.Fprintf(w, "✓ %s (yours): gentx signed on its host\n", o.Moniker) + } + } + var missing []string + for _, v := range k.Validators { + if !got[v.Moniker] { + missing = append(missing, v.Moniker) + } + } + if len(missing) > 0 && !callowMissing { + return fmt.Errorf("no gentx yet from %s — collect when all are in, or --allow-missing to launch without them (their accounts stay funded)", strings.Join(missing, ", ")) + } + bc, err := ctxFor(st.Builder) + if err != nil { + return err + } + defer bc.Close() + final, err := networktool.CollectAt(bc, spec, base, gentxs) + if err != nil { + return err + } + gp, err := networktool.GenesisPath(k.ChainID) + if err != nil { + return err + } + if err := os.WriteFile(gp, final, 0o600); err != nil { + return err + } + // the spec beside it, so provision and network check know the network + if sp, err := netspec.Path(k.ChainID); err == nil { + if _, err := os.Stat(sp); err != nil { + raw, _ := os.ReadFile(filepath.Join(ckit, "spec.yaml")) + _ = os.WriteFile(sp, raw, 0o644) + } + } + fmt.Fprintf(w, "✓ final genesis valid (%d bytes, %d gentxs) → %s\n", len(final), len(gentxs), gp) + if cpkgs == "" { + cpkgs = k.ChainID + "-packages" + } + pk, err := networktool.WritePackages(cpkgs, k, final, filepath.Join(ckit, "spec.yaml")) + if err != nil { + return err + } + for _, o := range st.Own { + if cnoStart { + break + } + c, err := ctxFor(o.Profile) + if err != nil { + return err + } + defer c.Close() + p, gen, sp, err := networktool.ReadPackage(pk[o.Moniker]) + if err != nil { + return err + } + if _, err := networktool.InstallPackage(c, p, gen, sp, o.Home, o.PortOffset, ""); err != nil { + return fmt.Errorf("%s: %w", o.Moniker, err) + } + fmt.Fprintf(w, "✓ %s (yours) started\n", o.Moniker) + } + fmt.Fprintf(w, "packages → %s// — send each outside validator theirs: cometcli genesis install --package --home \n", cpkgs) + return nil + }, + } + cf := collect.Flags() + cf.StringVar(&ckit, "kit", "", "the kit genesis base wrote") + cf.StringVar(&cpkgs, "packages", "", "directory for the packages (default -packages)") + cf.BoolVar(&cnoStart, "no-start", false, "don't install and start your own validators") + cf.BoolVar(&callowMissing, "allow-missing", false, "build the genesis without validators that haven't sent a gentx") + + var iprofile, ihome, ipkg, iip string + var ioffset int + install := &cobra.Command{ + Use: "install --package --home ", + Short: "Validator: install the final genesis from your package and start the node", + Args: cobra.NoArgs, + RunE: func(cmd *cobra.Command, _ []string) error { + w := cmd.OutOrStdout() + p, gen, sp, err := networktool.ReadPackage(ipkg) + if err != nil { + return err + } + fmt.Fprintf(w, "✓ package for %s on %s verified (genesis %.16s…, %d peers)\n", p.Moniker, p.ChainID, p.Genesis, len(p.Peers)) + c, _, err := keyMachine(cmd, iprofile) + if err != nil { + return err + } + defer c.Close() + if c.Profile.Service.Unit == "" { + c.Profile.Service.Unit = netspec.ContainerName(p.ChainID, netspec.Validator) + } + c.Progress = func(s string) { fmt.Fprintln(w, s) } + res, err := networktool.InstallPackage(c, p, gen, sp, ihome, ioffset, iip) + if err != nil { + return err + } + _ = res + fmt.Fprintf(w, "✓ %s started on %s\n", p.Moniker, p.ChainID) + return nil + }, + } + inf := install.Flags() + inf.StringVar(&ipkg, "package", "", "your package directory from the coordinator") + inf.StringVar(&ihome, "home", "", "this validator's node home (from genesis join)") + inf.StringVar(&iprofile, "profile", "", "install on this profile's host instead of this machine") + inf.StringVar(&iip, "ip", "", "public IP to advertise (default: the profile's SSH host, or detected)") + inf.IntVar(&ioffset, "port-offset", 0, "host port offset (beside another network)") + + return []*cobra.Command{join, base, gentx, collect, install} +} + +func uniqueValidators(vs []networktool.KitValidator) error { + seen := map[string]string{} + for _, v := range vs { + for kind, val := range map[string]string{"moniker": v.Moniker, "account": v.Account, "node id": v.NodeID, "consensus key": v.ConsKey} { + if val == "" { + continue + } + k := kind + "=" + val + if who, ok := seen[k]; ok { + return fmt.Errorf("%s and %s share a %s (%s) — every validator needs its own", who, v.Moniker, kind, val) + } + seen[k] = v.Moniker + } + } + return nil +} + +func nowUTC() string { return time.Now().UTC().Format(time.RFC3339) } + +// networkRemoveCmd stops a network's nodes and deletes their data. +func networkRemoveCmd() *cobra.Command { + var profiles string + var keepProfiles, yes bool + cmd := &cobra.Command{ + Use: "remove --profiles a,b", + Short: "Stop a network's nodes and delete their data (test networks): homes, builder files, profiles", + Long: `For each profile: stops its container and deletes its node home — only +when that home's genesis is for , so another network's data is +never touched. Also deletes the chain's genesis builder files on those +hosts, and the profiles (unless --keep-profiles). Reports the space freed.`, + Args: cobra.ExactArgs(1), + RunE: func(cmd *cobra.Command, args []string) error { + w := cmd.OutOrStdout() + chain := args[0] + cfg, err := config.Load() + if err != nil { + return err + } + names := splitCSV(profiles) + if len(names) == 0 { + return fmt.Errorf("--profiles: the nodes of %s to remove", chain) + } + for _, n := range names { + p, ok := cfg.Profiles[n] + if !ok { + return fmt.Errorf("no profile %s", n) + } + if p.ChainID != chain { + return fmt.Errorf("profile %s is on %s, not %s — not removed", n, p.ChainID, chain) + } + fmt.Fprintf(w, " %s: %s on %s (container %s)\n", n, orDash(p.Home), hostOf(p), orDash(p.Service.Unit)) + } + if !yes && !confirm(cmd, fmt.Sprintf("stop these %s nodes and DELETE their homes (keys and data)?", chain)) { + return fmt.Errorf("cancelled") + } + hosts := map[string]bool{} + for _, n := range names { + p := cfg.Profiles[n] + p.Name = n + c := &toolkit.Context{Context: cmd.Context(), Profile: p, Cfg: cfg} + msg, err := networktool.RemoveNode(c, chain, p.Home, p.Service.Unit) + if err != nil { + c.Close() + return fmt.Errorf("%s: %w", n, err) + } + fmt.Fprintf(w, "✓ %s: %s\n", n, msg) + if hk := hostOf(p); !hosts[hk] { + hosts[hk] = true + if b := networktool.RemoveBuilder(c, chain); b != "" { + fmt.Fprintf(w, "✓ %s: %s\n", n, b) + } + if h, err := c.Host(); err == nil { + if r, _, _ := h.Run(c, `df -h "$HOME" | tail -1 | awk '{print $4 " free (" $5 " used)"}'`); r != "" { + fmt.Fprintf(w, " %s disk: %s", hk, r) + } + } + } + c.Close() + if !keepProfiles { + delete(cfg.Profiles, n) + } + } + if !keepProfiles { + if err := cfg.Save(); err != nil { + return err + } + fmt.Fprintf(w, "✓ profiles removed: %s\n", strings.Join(names, ", ")) + } + return nil + }, + } + f := cmd.Flags() + f.StringVar(&profiles, "profiles", "", "the network's node profiles, comma-separated") + f.BoolVar(&keepProfiles, "keep-profiles", false, "keep the profiles") + f.BoolVar(&yes, "yes", false, "don't ask for confirmation") + return cmd +} + +// operatorKeyCmd makes a joining validator's operator key in its node's +// own keyring. +func operatorKeyCmd() *cobra.Command { + var keyring, mnemonicTo, name string + cmd := &cobra.Command{ + Use: "operator-key", + Short: "Make this validator's operator key in its node's own keyring (validators joining after genesis)", + Long: `For a validator joining a running network (node provision, then this, +then fund the address and val create): creates the operator key inside +the node container's keyring. The mnemonic goes to a 0600 file on the +node's host (default .operator.mnemonic) and is never shown; the +profile is set to sign there.`, + Args: cobra.NoArgs, + RunE: func(cmd *cobra.Command, _ []string) error { + w := cmd.OutOrStdout() + c, _, err := keyMachine(cmd, flagProfile) + if err != nil { + return err + } + defer c.Close() + if flagProfile == "" { + return fmt.Errorf("--profile: the validator's profile") + } + if name == "" { + name = c.Profile.Name + } + kr, err := keyringFor(keyring) + if err != nil { + return err + } + acc, mfile, err := networktool.OperatorKey(c, name, kr, mnemonicTo) + if err != nil { + return err + } + fmt.Fprintf(w, "✓ operator key %s → %s, in %s's keyring\n✓ mnemonic → %s on the node's host (0600) — never shown\n", name, acc, c.Profile.Service.Unit, mfile) + fmt.Fprintf(w, "→ fund %s, then: cometcli --profile %s val create --amount \n\n%s\n", acc, c.Profile.Name, backupReminder) + return nil + }, + } + cmd.Flags().StringVar(&keyring, "keyring", "file", "file (password) or test (unencrypted, test networks)") + cmd.Flags().StringVar(&mnemonicTo, "mnemonic-to", "", "where the mnemonic goes on the node's host (0600)") + cmd.Flags().StringVar(&name, "name", "", "key name (default: the profile name)") + return cmd +} diff --git a/internal/cli/network.go b/internal/cli/network.go index 87f68c8..4fd33ea 100644 --- a/internal/cli/network.go +++ b/internal/cli/network.go @@ -16,9 +16,9 @@ func networkImportCmd() *cobra.Command { var force bool cmd := &cobra.Command{ Use: "import [more.env …]", - Short: "Create a network spec from node-setup network-config.env files", - Long: `Reads node-setup's network-config.env (the run-genesis one is the most -complete; run-validator / run-archive ones fill in the rest) and writes + Short: "Create a network spec from network-config.env files (KEY=value)", + Long: `Reads network-config.env files of KEY=value lines (CHAIN_ID, DENOM, +TIMEOUT_COMMIT, …; give the most complete first, later files fill in the rest) and writes ~/.cometcli/networks/.yaml — the spec network check compares the chain and every node against. Keys with no spec field are kept under extra.`, Args: cobra.ArbitraryArgs, @@ -93,7 +93,7 @@ chain and every node against. Keys with no spec field are kept under extra.`, return nil }, } - cmd.Flags().StringVar(&naming, "image-naming", "", "how images built from releases are named, e.g. primium-{tag}") + cmd.Flags().StringVar(&naming, "image-naming", "", "how images built from releases are named, e.g. mychain-{tag}") cmd.Flags().StringVar(&repo, "repo", "", "official source repo (default cosmos/evm)") cmd.Flags().BoolVar(&force, "force", false, "replace an existing spec") cmd.Flags().StringVar(&chainID, "chain-id", "", "use this chain id (a new network with these settings)") diff --git a/internal/cli/profile.go b/internal/cli/profile.go index 788de9b..938c0a0 100644 --- a/internal/cli/profile.go +++ b/internal/cli/profile.go @@ -244,7 +244,7 @@ func profileAddCmd() *cobra.Command { f.StringVar(&p.Signer.Backend, "signer-backend", "", "os | file") f.StringVar(&p.Signer.Mode, "signer-mode", "", "local (cometcli keyring) | container (the node's own keyring); empty asks when both work") f.StringVar(&p.Signer.ContainerHome, "container-home", "", "node home inside the container, for signing there (e.g. /data/node0/evmd)") - f.StringVar(&imageName, "image-name", "", "how images built from releases are named; {tag} = release tag, e.g. primium-{tag} → primium-v0.7.2") + f.StringVar(&imageName, "image-name", "", "how images built from releases are named; {tag} = release tag, e.g. mychain-{tag} → mychain-v0.7.2") f.StringVar(&p.Signer.ContainerKey, "container-key", "", "key name (or address) in the container's keyring (default: the operator account)") f.StringVar(&p.Signer.ContainerKeyring, "container-keyring", "", "keyring backend inside the container: test | file (detected when empty)") f.StringVar(&feeDenom, "fee-denom", "", "fee denom (e.g. atest)") diff --git a/internal/cli/qr.go b/internal/cli/qr.go new file mode 100644 index 0000000..0270a01 --- /dev/null +++ b/internal/cli/qr.go @@ -0,0 +1,148 @@ +package cli + +import ( + "bufio" + "fmt" + "os" + "path/filepath" + "strings" + + "github.com/spf13/cobra" + + "github.com/abhijitkrm/cometcli/internal/qrxfer" +) + +// qrCmd moves a small public file (a join or gentx file) off an +// air-gapped machine as QR codes, with nothing plugged into it. +func qrCmd() *cobra.Command { + cmd := &cobra.Command{ + Use: "qr", + Short: "Move a small public file off an offline machine as QR codes (join and gentx files)", + } + var pngDir string + var all bool + show := &cobra.Command{ + Use: "show ", + Short: "Offline machine: show a file as QR codes to scan, one at a time", + Long: `Compresses the file and shows it as a few QR codes, each carrying the +file's checksum. Scan them with a phone or camera on a connected machine +and give the result to cometcli qr read there. Files with key material +are refused: only public files (join, gentx) may leave this way.`, + Args: cobra.ExactArgs(1), + RunE: func(cmd *cobra.Command, args []string) error { + w := cmd.OutOrStdout() + data, err := os.ReadFile(args[0]) + if err != nil { + return err + } + parts, err := qrxfer.Encode(filepath.Base(args[0]), data) + if err != nil { + return err + } + if pngDir != "" { + if err := os.MkdirAll(pngDir, 0o755); err != nil { + return err + } + for i, p := range parts { + f := filepath.Join(pngDir, fmt.Sprintf("%s.qr%02d.png", filepath.Base(args[0]), i+1)) + if err := qrxfer.PNG(p, f); err != nil { + return err + } + } + fmt.Fprintf(w, "%d QR images → %s\n", len(parts), pngDir) + return nil + } + in := bufio.NewReader(cmd.InOrStdin()) + for i, p := range parts { + s, err := qrxfer.Terminal(p) + if err != nil { + return err + } + fmt.Fprintf(w, "\n%s — part %d of %d (%s, %d bytes)\n%s", filepath.Base(args[0]), i+1, len(parts), filepath.Base(args[0]), len(data), s) + if !all && i < len(parts)-1 { + fmt.Fprint(w, "scan it, then press Enter for the next part ") + if _, err := in.ReadString('\n'); err != nil { + return nil + } + } + } + fmt.Fprintf(w, "\nall %d parts shown. On the connected machine: cometcli qr read\n", len(parts)) + return nil + }, + } + show.Flags().StringVar(&pngDir, "png", "", "write the codes as PNG images into this directory instead (to show full screen)") + show.Flags().BoolVar(&all, "all", false, "print every code at once instead of one per Enter") + + var outDir string + read := &cobra.Command{ + Use: "read [images…]", + Short: "Connected machine: reassemble a file from scanned QR codes (pasted text or photos)", + Long: `Paste each scanned code's text, one per line, in any order (a phone's QR +scanner gives the text), or pass photos/screenshots of the codes. The file +is written only when every part is in and it matches its checksum.`, + RunE: func(cmd *cobra.Command, args []string) error { + w := cmd.OutOrStdout() + var a qrxfer.Assembler + add := func(text, from string) error { + isNew, err := a.Add(text) + if err != nil { + return fmt.Errorf("%s: %w", from, err) + } + got, n, name := a.Progress() + if isNew { + fmt.Fprintf(w, "✓ %s part — %d of %d\n", name, got, n) + } + return nil + } + for _, img := range args { + text, err := qrxfer.DecodeImage(img) + if err != nil { + return err + } + if err := add(text, img); err != nil { + return err + } + } + if len(args) == 0 { + fmt.Fprintln(w, "paste each scanned code (one per line), in any order; an empty line or EOF ends") + sc := bufio.NewScanner(cmd.InOrStdin()) + sc.Buffer(make([]byte, 64<<10), 1<<20) + for sc.Scan() { + line := strings.TrimSpace(sc.Text()) + if line == "" { + if _, n, _ := a.Progress(); n > 0 && len(a.Missing()) == 0 { + break + } + continue + } + if err := add(line, "pasted line"); err != nil { + fmt.Fprintf(w, "! %v — skipped\n", err) + continue + } + if _, n, _ := a.Progress(); n > 0 && len(a.Missing()) == 0 { + break + } + } + } + name, data, err := a.File() + if err != nil { + return err + } + if err := os.MkdirAll(outDir, 0o755); err != nil { + return err + } + p := filepath.Join(outDir, name) + if _, err := os.Stat(p); err == nil { + return fmt.Errorf("%s exists — not overwritten (use --out)", p) + } + if err := os.WriteFile(p, data, 0o644); err != nil { + return err + } + fmt.Fprintf(w, "✓ %s reassembled and verified (%d bytes) → %s\n", name, len(data), p) + return nil + }, + } + read.Flags().StringVar(&outDir, "out", ".", "directory to write the file into") + cmd.AddCommand(show, read) + return cmd +} diff --git a/internal/cli/root.go b/internal/cli/root.go index e9f6045..8750c01 100644 --- a/internal/cli/root.go +++ b/internal/cli/root.go @@ -75,11 +75,14 @@ status, cometcli doctor, cometcli tx unjail) and a tool the agent calls.`, for _, c := range toolGroupCommands(reg) { if c.Use == "network" { - c.AddCommand(networkImportCmd(), networkShowCmd()) + c.AddCommand(networkImportCmd(), networkShowCmd(), networkRemoveCmd()) + } + if c.Use == "val" { + c.AddCommand(operatorKeyCmd()) } root.AddCommand(c) } - root.AddCommand(profileCmd(), auditCmd(), versionCmd(), initCmd(), sessionsCmd(), oneCmd(reg), configCmd(), trustCmd(), watchCmd(reg), drillCmd(reg), sshCmd(), genesisCmd(reg), routeCmd()) + root.AddCommand(profileCmd(), auditCmd(), versionCmd(), initCmd(), sessionsCmd(), oneCmd(reg), configCmd(), trustCmd(), watchCmd(reg), drillCmd(reg), sshCmd(), genesisCmd(reg), routeCmd(), qrCmd()) root.AddCommand(extra...) return root } diff --git a/internal/kb/cases/host.yaml b/internal/kb/cases/host.yaml index c97a0e7..4ae9edb 100644 --- a/internal/kb/cases/host.yaml +++ b/internal/kb/cases/host.yaml @@ -148,7 +148,7 @@ - two instances of a validator = double-sign risk; stop one immediately playbook_kind: report playbook: - - {do: bash, args: {command: "docker ps --filter publish={sig:proc.port_taken}", description: "the container holding the port"}, when: "proc.port_taken > 0", note: "who holds the port (docker refused to start the node)"} + - {do: bash, args: {command: "docker ps --filter publish={sig:proc.port_taken} --format '{{.Names}} (image {{.Image}}, up {{.RunningFor}})'", description: "the container holding the port"}, when: "proc.port_taken > 0", note: "who holds the port (docker refused to start the node)"} - {do: bash, args: {command: "ss -ltnp 2>/dev/null | grep -E ':(26656|26657|26660|1317|9090|8545|8546) ' || true", description: "what listens on the node's ports"}, note: "what listens on the node's ports"} test: signals: {logs.port_in_use: 2} @@ -160,7 +160,7 @@ symptoms: restarts with no error in the logs; docker OOMKilled may be false because the node runs under a wrapper shell match: all: ["proc.mem_limit_mb < 4096"] - any: ["proc.oom_kills > 0", "proc.mem_used_pct_of_limit >= 90", "proc.restarts >= 3"] + any: ["proc.oom_kills > 0", "proc.mem_used_pct_of_limit >= 90", "proc.restarts >= 3", "proc.mem_limit_mb < 1024"] confirm: - "docker inspect -f '{{.HostConfig.Memory}}' (bytes; 0 = unlimited) and docker stats --no-stream" - "a node needs several GB under load — a limit of a few hundred MB will always be killed" @@ -236,7 +236,7 @@ - id: node-image-missing title: The node's docker image isn't on the host - explains: [node-process-down, node-rpc-down, node-crash-loop, node-stalled, node-catching-up, node-killed-sigkill, node-restarted-recently, node-blocksync-slow, evm-rpc-down, val-in-set-while-syncing, node-behind-but-not-reporting, val-jailed-downtime, val-jailed-waiting-period, val-not-signing, val-jail-imminent, val-signing-degraded, node-no-peers, node-low-peers] + explains: [node-process-down, node-rpc-down, node-crash-loop, node-stalled, node-catching-up, node-killed-sigkill, node-restarted-recently, node-blocksync-slow, evm-rpc-down, val-in-set-while-syncing, node-behind-but-not-reporting, val-jailed-downtime, val-jailed-waiting-period, val-not-signing, val-jail-imminent, val-signing-degraded, node-no-peers, node-low-peers, node-container-missing] severity: high match: all: ["proc.image_present == false"] @@ -248,12 +248,12 @@ - "[change] upgrade.build the release the image was built from, under the same name; start the node" playbook: - {do: upgrade.build, args: {tag: "{sig:proc.image_version}", image: "{sig:proc.image}"}, note: "rebuild the image from its official release (asks you; several minutes)"} - - {do: node.service, args: {action: start}, note: "start the node"} + - {do: node.recreate, note: "start the node from its compose file (recreates the container if it's gone)"} - {do: wait.until, args: {condition: synced, timeout: 1800}, note: "wait until it has caught up"} - {do: wait.until, args: {condition: unjailable, timeout: 1800}, when: "val.jailed == true", note: "wait out the jail period"} - {do: val.unjail, when: "val.jailed == true", note: "unjail (signed by the operator key — asks you)"} test: - signals: {proc.image_present: false, proc.running: false, proc.image: "primium-v0.7.2", proc.image_version: "v0.7.2"} + signals: {proc.image_present: false, proc.running: false, proc.image: "mychain-v0.7.2", proc.image_version: "v0.7.2"} - id: node-container-missing title: The node's container doesn't exist @@ -267,6 +267,11 @@ - docker compose down / docker rm, or the profile names the wrong container fix: - "[change] docker compose up -d from the node's compose file (node.provision reconfigure=true re-renders it from the spec)" + playbook: + - {do: node.recreate, note: "create the container again from the node's compose file (same data dir)"} + - {do: wait.until, args: {condition: synced, timeout: 1800}, note: "wait until it has caught up"} + - {do: wait.until, args: {condition: unjailable, timeout: 1800}, when: "val.jailed == true", note: "wait out the jail period"} + - {do: val.unjail, when: "val.jailed == true", note: "unjail (signed by the operator key — asks you)"} test: signals: {proc.container_missing: true, proc.running: false} diff --git a/internal/kb/cases/sync.yaml b/internal/kb/cases/sync.yaml index df670b2..3e511fc 100644 --- a/internal/kb/cases/sync.yaml +++ b/internal/kb/cases/sync.yaml @@ -85,6 +85,27 @@ test: signals: {node.catching_up: true, node.block_age_s: 7200} +- id: chain-halted-validators-offline + title: Chain halted — validators holding over 1/3 of the voting power aren't voting + explains: [node-stalled, node-low-peers, node-p2p-port-exposure, val-jailed-downtime, val-jailed-waiting-period, val-not-signing, val-jail-imminent, val-signing-degraded] + severity: critical + symptoms: no new blocks on any node; the consensus state shows under 2/3 of the power voting + match: + all: ["node.stalled == true", "consensus.voted_pct < 66.67", "node.peers > 0", "consensus.absent_n > 0"] + confirm: + - "node.consensus — prevotes/precommits per round; consensus.absent names who hasn't voted" + - "fleet.status — are the absent validators' nodes yours, and are they down?" + causes: + - the absent validators' nodes are stopped, crashed or cut off (a third of the power or more) + - "an upgrade height: check upgrade.blocks_to_halt — then every node waits for the new binary, not for votes" + fix: + - "[read] the absent validators are in consensus.absent, with their share of the power" + - "[change] start or reconnect their nodes (one of yours: \"start \"); others: reach their operators — blocks resume once over 2/3 vote" + verify: + - "node.block_age_s < 30" + test: + signals: {node.stalled: true, node.block_age_s: 120, node.catching_up: false, node.peers: 2, consensus.voted_pct: 54.5, consensus.absent_n: 1, consensus.absent: "val1 (45.5%)"} + - id: node-stalled title: Node stalled — not catching up but no new blocks explains: [val-jailed-downtime, val-jailed-waiting-period, val-not-signing, val-jail-imminent, val-signing-degraded] diff --git a/internal/kb/cases/upgrade.yaml b/internal/kb/cases/upgrade.yaml index a69af14..86fc022 100644 --- a/internal/kb/cases/upgrade.yaml +++ b/internal/kb/cases/upgrade.yaml @@ -105,5 +105,8 @@ - "[change] or avoid replay entirely: snap.statesync / a recent trusted snapshot" verify: - "node.height passes the upgrade height; wait.until synced" + playbook: + - {do: upgrade.switch, note: "switch to the image built for this plan — the one whose binary registers it (asks you)"} + - {do: wait.until, args: {condition: synced, timeout: 3600}, note: "replay on to the chain's height"} test: signals: {logs.upgrade_needed: 2, upgrade.pending: false, node.catching_up: true} diff --git a/internal/netspec/netspec_test.go b/internal/netspec/netspec_test.go index 7266359..4235dc8 100644 --- a/internal/netspec/netspec_test.go +++ b/internal/netspec/netspec_test.go @@ -144,3 +144,39 @@ func TestCmdFlag(t *testing.T) { t.Fatal("flag parsing") } } + +func TestUpgradeHandlersAreRecordedPerTag(t *testing.T) { + s := &Spec{} + if !s.RecordHandler("v0.7.3", "v0.7.2-to-v0.7.3") || s.RecordHandler("v0.7.3", "v0.7.2-to-v0.7.3") { + t.Fatal("record once") + } + if s.HandlerFor("v0.7.3") != "v0.7.2-to-v0.7.3" || s.HandlerFor("v0.8.0") != "" { + t.Fatalf("lookup: %+v", s.Upgrades) + } +} + +func TestContainerNameFromChain(t *testing.T) { + for _, c := range []struct { + id string + role Role + want string + }{ + {"testnet-1", Validator, "testnet-1-validator"}, + {"evmos_9001-2", Archive, "evmos_9001-2-archive"}, + {"weird/chain id", RPC, "weird-chain-id-rpc"}, + {"", "", "evmd-validator"}, + } { + if got := ContainerName(c.id, c.role); got != c.want { + t.Errorf("ContainerName(%q, %q) = %q, want %q", c.id, c.role, got, c.want) + } + } +} + +func TestHostHomeFromAnyDeploymentPrefix(t *testing.T) { + if s := FromEnv(map[string]string{"CHAIN_ID": "x-1", "ACME_HOST_HOME": "/srv/acme"}); s.HostHome != "/srv/acme" { + t.Errorf("_HOST_HOME not read: %q", s.HostHome) + } + if s := FromEnv(map[string]string{"CHAIN_ID": "x-1", "HOST_HOME": "/srv/x"}); s.HostHome != "/srv/x" || s.Extra["HOST_HOME"] != "" { + t.Errorf("HOST_HOME: %q extra %v", s.HostHome, s.Extra) + } +} diff --git a/internal/netspec/render.go b/internal/netspec/render.go index b7ad7be..9dda856 100644 --- a/internal/netspec/render.go +++ b/internal/netspec/render.go @@ -22,6 +22,9 @@ type NodeParams struct { ExternalAddress string // ip:port advertised to peers ("" = leave unset) PersistentPeers []string Prod bool // public-network hardening + // InboundClosed: the node's config takes no inbound peers now (an + // archive or RPC node gets CometBFT's default back; other values stay) + InboundClosed bool } func q(s string) string { return strconv.Quote(s) } @@ -59,6 +62,13 @@ func (s *Spec) Patches(role Role, version string, n NodeParams) []Patch { if len(n.PersistentPeers) > 0 { set("config.toml", "p2p", "persistent_peers", q(strings.Join(n.PersistentPeers, ","))) } + if role != Validator { + // it serves other nodes: as network check expects + set("config.toml", "p2p", "pex", "true") + if n.InboundClosed { + set("config.toml", "p2p", "max_num_inbound_peers", "40") + } + } if AppMempool(version) { set("config.toml", "mempool", "type", q("app")) } @@ -77,7 +87,7 @@ func (s *Spec) Patches(role Role, version string, n NodeParams) []Patch { } } } - open := !n.Prod && role != Validator // test networks keep archive/rpc open like node-setup does + open := !n.Prod && role != Validator // test networks keep archive/rpc open if n.Prod || role == Validator { set("config.toml", "rpc", "cors_allowed_origins", "[]") } else if open { diff --git a/internal/netspec/roles.go b/internal/netspec/roles.go index e1eb280..99fd3c8 100644 --- a/internal/netspec/roles.go +++ b/internal/netspec/roles.go @@ -27,6 +27,22 @@ func RoleOf(profileRole string) Role { return Validator } +var containerUnsafe = regexp.MustCompile(`[^a-zA-Z0-9_.-]+`) + +// ContainerName is the default container for a node of a chain when its +// profile names none: -, so two chains on one host never +// share a name. +func ContainerName(chainID string, role Role) string { + id := strings.Trim(containerUnsafe.ReplaceAllString(chainID, "-"), "-._") + if id == "" { + id = "evmd" + } + if role == "" { + role = Validator + } + return id + "-" + string(role) +} + // Expect is one setting a node of a role should have. type Expect struct { File string // config.toml | app.toml | cmd (the container command) @@ -162,6 +178,7 @@ func (s *Spec) Expectations(role Role, version string, prod bool) []Expect { Want: "without debug, personal", Why: "debug and personal namespaces don't belong on a validator"}) } case Archive: + servesPeers(add, "an archive") add(Expect{File: "app.toml", Key: []string{"pruning"}, Sev: Fail, Check: eq("nothing"), Want: `"nothing"`, Why: "an archive keeps all historical state"}) add(Expect{File: "config.toml", Key: []string{"tx_index", "indexer"}, Sev: Fail, Check: eq("kv"), Want: `"kv"`, @@ -170,6 +187,7 @@ func (s *Spec) Expectations(role Role, version string, prod bool) []Expect { hardenRPC(add, "archive") } case RPC: + servesPeers(add, "an RPC node") add(Expect{File: "config.toml", Key: []string{"tx_index", "indexer"}, Sev: Warn, Check: eq("kv"), Want: `"kv"`, Why: "tx queries need the indexer"}) if prod { @@ -179,6 +197,16 @@ func (s *Spec) Expectations(role Role, version string, prod bool) []Expect { return e } +// servesPeers: a node that exists to serve others takes inbound peers and +// exchanges addresses (a validator behind sentries may close both). +func servesPeers(add func(Expect), who string) { + add(Expect{File: "config.toml", Key: []string{"p2p", "pex"}, Sev: Warn, Check: eq(true), Want: "true", + Why: who + " shares peer addresses so the network finds its nodes"}) + add(Expect{File: "config.toml", Key: []string{"p2p", "max_num_inbound_peers"}, Sev: Warn, + Check: func(g any) bool { n, ok := g.(int64); return g == nil || (ok && n > 0) }, Want: "> 0", + Why: who + " serves other nodes — with no inbound peers nobody can sync from it"}) +} + // hardenRPC is the public-network CORS/unlock hardening every role needs. func hardenRPC(add func(Expect), who string) { add(Expect{File: "config.toml", Key: []string{"rpc", "cors_allowed_origins"}, Sev: Warn, Check: emptyList, Want: "[]", diff --git a/internal/netspec/spec.go b/internal/netspec/spec.go index 43eb7d5..8706758 100644 --- a/internal/netspec/spec.go +++ b/internal/netspec/spec.go @@ -36,6 +36,9 @@ type Spec struct { FeeDenom string `yaml:"fee_denom,omitempty"` DBBackend string `yaml:"db_backend"` // goleveldb | rocksdb HostHome string `yaml:"host_home,omitempty"` + // Upgrades records releases built with an added upgrade handler, so a + // node that loses its image (or a new node) rebuilds the same binary. + Upgrades []UpgradeBuild `yaml:"upgrades,omitempty"` // Extra keeps imported settings with no field yet, so nothing is lost. Extra map[string]string `yaml:"extra,omitempty"` } @@ -49,10 +52,10 @@ type Chain struct { } type Image struct { - Name string `yaml:"name"` // e.g. primium-evm + Name string `yaml:"name"` // e.g. mychain-evm Tag string `yaml:"tag"` // e.g. v0.7.2 Repo string `yaml:"repo,omitempty"` // official source, e.g. cosmos/evm - Naming string `yaml:"naming,omitempty"` // e.g. primium-{tag} + Naming string `yaml:"naming,omitempty"` // e.g. mychain-{tag} } type Genesis struct { @@ -238,7 +241,7 @@ func ParseEnv(raw string) map[string]string { var envKeyRe = regexp.MustCompile(`^[A-Z_][A-Z0-9_]*$`) -// FromEnv builds a spec from node-setup's network-config.env keys. Keys it +// FromEnv builds a spec from network-config.env keys (KEY=value). Keys it // doesn't map are kept in Extra. func FromEnv(env map[string]string) *Spec { used := map[string]bool{} @@ -297,7 +300,13 @@ func FromEnv(env map[string]string) *Spec { } s.MinGasPrice = str("MIN_GAS_PRICE") s.DBBackend = str("DB_BACKEND") - s.HostHome = str("PRIMIUM_HOST_HOME") + // HOST_HOME, or a deployment's own _HOST_HOME + s.HostHome = str("HOST_HOME") + for k := range env { + if s.HostHome == "" && strings.HasSuffix(k, "_HOST_HOME") { + s.HostHome = str(k) + } + } for k, v := range env { if !used[k] { if s.Extra == nil { @@ -448,3 +457,42 @@ func (s *Spec) GasDenom() string { } return s.Chain.Denom } + +// Bech32 is the chain's account address prefix (default cosmos). +func (s *Spec) Bech32() string { + if s.Chain.Bech32Prefix != "" { + return s.Chain.Bech32Prefix + } + return "cosmos" +} + +// UpgradeBuild is a release tag and the handler added when building it. +type UpgradeBuild struct { + Tag string `yaml:"tag"` + Handler string `yaml:"handler"` +} + +// HandlerFor is the handler recorded for a release tag ("" if none). +func (s *Spec) HandlerFor(tag string) string { + for _, u := range s.Upgrades { + if u.Tag == tag { + return u.Handler + } + } + return "" +} + +// RecordHandler notes that tag was built with handler (once per tag). +func (s *Spec) RecordHandler(tag, handler string) bool { + for i, u := range s.Upgrades { + if u.Tag == tag { + if u.Handler == handler { + return false + } + s.Upgrades[i].Handler = handler + return true + } + } + s.Upgrades = append(s.Upgrades, UpgradeBuild{Tag: tag, Handler: handler}) + return true +} diff --git a/internal/qrxfer/qrxfer.go b/internal/qrxfer/qrxfer.go new file mode 100644 index 0000000..600c819 --- /dev/null +++ b/internal/qrxfer/qrxfer.go @@ -0,0 +1,201 @@ +// Package qrxfer moves a small public file off an air-gapped machine as a +// series of QR codes: shown on its screen, scanned by a phone or camera on +// a connected machine, and reassembled there. Nothing is ever plugged into +// the offline machine. +// +// Each code is one line of text: +// +// CQR1||||| +// +// so parts can be scanned in any order, repeats are harmless, a part of +// another file is refused, and the file is written only when it hashes to +// the sum every part carries. +package qrxfer + +import ( + "bytes" + "compress/gzip" + "crypto/sha256" + "encoding/base64" + "encoding/hex" + "fmt" + "image" + _ "image/jpeg" // photos of the screen + _ "image/png" + "io" + "os" + "path/filepath" + "regexp" + "strconv" + "strings" + + "github.com/makiuchi-d/gozxing" + gzqr "github.com/makiuchi-d/gozxing/qrcode" + qrcode "github.com/skip2/go-qrcode" + + "github.com/abhijitkrm/cometcli/internal/egress" +) + +const ( + magic = "CQR1" + // ChunkSize is the compressed bytes per code: small enough that a code + // fits an 80-column terminal and scans from a phone. + ChunkSize = 160 + // MaxFile bounds what QR transfer is for: join and gentx files, a few KB. + MaxFile = 64 << 10 +) + +var nameRe = regexp.MustCompile(`^[A-Za-z0-9._-]{1,100}$`) + +// Encode splits a file into QR payloads. It refuses anything that carries +// key material: QR transfer is for public files only. +func Encode(name string, data []byte) ([]string, error) { + name = filepath.Base(name) + if !nameRe.MatchString(name) { + return nil, fmt.Errorf("file name %q: letters, digits, . _ - only", name) + } + if len(data) > MaxFile { + return nil, fmt.Errorf("%s is %d bytes — QR transfer is for small files (join, gentx: a few KB)", name, len(data)) + } + if f := egress.Scan(string(data)); len(f) > 0 { + return nil, fmt.Errorf("%s contains key material (%s) — it must never leave the machine, by QR or otherwise", name, egress.Kinds(f)) + } + sum := sha256.Sum256(data) + var z bytes.Buffer + zw, _ := gzip.NewWriterLevel(&z, gzip.BestCompression) + _, _ = zw.Write(data) + _ = zw.Close() + packed := z.Bytes() + n := (len(packed) + ChunkSize - 1) / ChunkSize + if n == 0 { + n = 1 + } + parts := make([]string, 0, n) + for i := 0; i < n; i++ { + lo, hi := i*ChunkSize, min((i+1)*ChunkSize, len(packed)) + parts = append(parts, strings.Join([]string{magic, hex.EncodeToString(sum[:]), strconv.Itoa(i + 1), strconv.Itoa(n), name, + base64.RawURLEncoding.EncodeToString(packed[lo:hi])}, "|")) + } + return parts, nil +} + +// Terminal renders a payload as a QR code with half-block characters. +func Terminal(payload string) (string, error) { + q, err := qrcode.New(payload, qrcode.Medium) + if err != nil { + return "", err + } + return q.ToSmallString(false), nil +} + +// PNG writes a payload as a QR code image. +func PNG(payload, path string) error { + return qrcode.WriteFile(payload, qrcode.Medium, 640, path) +} + +// Assembler collects scanned parts until the file is complete. +type Assembler struct { + sum string + name string + n int + parts map[int][]byte +} + +// Add takes one scanned line. It reports whether the part was new. +func (a *Assembler) Add(line string) (bool, error) { + f := strings.Split(strings.TrimSpace(line), "|") + if len(f) != 6 || f[0] != magic { + return false, fmt.Errorf("not a cometcli QR part") + } + i, err1 := strconv.Atoi(f[2]) + n, err2 := strconv.Atoi(f[3]) + if err1 != nil || err2 != nil || n < 1 || n > 1000 || i < 1 || i > n || len(f[1]) != 64 || !nameRe.MatchString(f[4]) { + return false, fmt.Errorf("damaged QR part") + } + if a.sum == "" { + a.sum, a.name, a.n, a.parts = f[1], f[4], n, map[int][]byte{} + } + if f[1] != a.sum || n != a.n { + return false, fmt.Errorf("this part belongs to another file (%s), not %s — scan one file at a time", f[4], a.name) + } + chunk, err := base64.RawURLEncoding.DecodeString(f[5]) + if err != nil { + return false, fmt.Errorf("damaged QR part %d", i) + } + if old, ok := a.parts[i]; ok { + if !bytes.Equal(old, chunk) { + return false, fmt.Errorf("part %d scanned twice with different content", i) + } + return false, nil + } + a.parts[i] = chunk + return true, nil +} + +// Progress is (parts held, parts needed, file name). +func (a *Assembler) Progress() (int, int, string) { return len(a.parts), a.n, a.name } + +// Missing lists the parts not scanned yet. +func (a *Assembler) Missing() []int { + var m []int + for i := 1; i <= a.n; i++ { + if _, ok := a.parts[i]; !ok { + m = append(m, i) + } + } + return m +} + +// File reassembles and verifies the file. +func (a *Assembler) File() (string, []byte, error) { + if a.n == 0 || len(a.parts) < a.n { + return "", nil, fmt.Errorf("missing parts %v", a.Missing()) + } + var packed []byte + for i := 1; i <= a.n; i++ { + packed = append(packed, a.parts[i]...) + } + zr, err := gzip.NewReader(bytes.NewReader(packed)) + if err != nil { + return "", nil, fmt.Errorf("the parts don't form a file: %w", err) + } + data, err := io.ReadAll(io.LimitReader(zr, MaxFile+1)) + if err != nil || len(data) > MaxFile { + return "", nil, fmt.Errorf("the parts don't form a file") + } + sum := sha256.Sum256(data) + if hex.EncodeToString(sum[:]) != a.sum { + return "", nil, fmt.Errorf("the reassembled file doesn't match its checksum — rescan") + } + return a.name, data, nil +} + +// DecodeImage reads every cometcli QR code in an image file (a photo or +// screenshot of the offline machine's screen). +func DecodeImage(path string) (string, error) { + f, err := os.Open(path) + if err != nil { + return "", err + } + defer f.Close() + img, _, err := image.Decode(f) + if err != nil { + return "", fmt.Errorf("%s: %w", path, err) + } + bmp, err := gozxing.NewBinaryBitmapFromImage(img) + if err != nil { + return "", err + } + // a photo needs the detector; a clean screenshot sometimes decodes + // only as a pure code + for _, pure := range []bool{false, true} { + hints := map[gozxing.DecodeHintType]interface{}{gozxing.DecodeHintType_TRY_HARDER: true} + if pure { + hints[gozxing.DecodeHintType_PURE_BARCODE] = true + } + if res, err := gzqr.NewQRCodeReader().Decode(bmp, hints); err == nil { + return res.GetText(), nil + } + } + return "", fmt.Errorf("%s: no QR code found — retake it straight on, the whole code in frame", path) +} diff --git a/internal/qrxfer/qrxfer_test.go b/internal/qrxfer/qrxfer_test.go new file mode 100644 index 0000000..2eb2742 --- /dev/null +++ b/internal/qrxfer/qrxfer_test.go @@ -0,0 +1,120 @@ +package qrxfer + +import ( + "math/rand" + "path/filepath" + "strings" + "testing" + + "github.com/cosmos/go-bip39" +) + +func gentxLike() []byte { + var b strings.Builder + b.WriteString(`{"version":1,"chain_id":"primium-3","moniker":"p3-val3","gentx":{"body":{"messages":[`) + r := rand.New(rand.NewSource(1)) + for i := 0; i < 40; i++ { + b.WriteString(`{"k":"`) + for j := 0; j < 40; j++ { + b.WriteByte("0123456789abcdef"[r.Intn(16)]) + } + b.WriteString(`"},`) + } + b.WriteString(`{}]}}}`) + return []byte(b.String()) +} + +func TestRoundTripAnyOrderWithRepeats(t *testing.T) { + data := gentxLike() + parts, err := Encode("p3-val3.gentx.json", data) + if err != nil { + t.Fatal(err) + } + if len(parts) < 2 { + t.Fatalf("want several parts for %d bytes, got %d", len(data), len(parts)) + } + var a Assembler + for i := len(parts) - 1; i >= 0; i-- { // reversed, with a repeat + if _, err := a.Add(parts[i]); err != nil { + t.Fatal(err) + } + if i == 1 { + if isNew, err := a.Add(parts[i]); err != nil || isNew { + t.Fatalf("a repeated part should be ignored: %v %v", isNew, err) + } + } + } + name, got, err := a.File() + if err != nil || name != "p3-val3.gentx.json" || string(got) != string(data) { + t.Fatalf("roundtrip: %v %q", err, name) + } +} + +func TestIncompleteForeignAndDamaged(t *testing.T) { + p1, _ := Encode("a.json", gentxLike()) + p2, _ := Encode("b.json", []byte(`{"other":true}`)) + var a Assembler + _, _ = a.Add(p1[0]) + if _, _, err := a.File(); err == nil || !strings.Contains(err.Error(), "missing parts") { + t.Fatalf("incomplete: %v", err) + } + if _, err := a.Add(p2[0]); err == nil || !strings.Contains(err.Error(), "another file") { + t.Fatalf("foreign part accepted: %v", err) + } + bad := p1[1][:len(p1[1])-6] + "AAAAAA" + for _, x := range p1[1:] { + if x == p1[1] { + _, _ = a.Add(bad) + continue + } + _, _ = a.Add(x) + } + if _, _, err := a.File(); err == nil { + t.Fatal("a damaged part must not produce a file") + } + if _, err := a.Add("hello world"); err == nil { + t.Fatal("arbitrary text accepted as a part") + } +} + +func TestRefusesKeyMaterial(t *testing.T) { + e, _ := bip39.NewEntropy(256) + m, _ := bip39.NewMnemonic(e) + if _, err := Encode("op.mnemonic", []byte(m)); err == nil || !strings.Contains(err.Error(), "key material") { + t.Fatalf("a mnemonic must never be QR-encoded: %v", err) + } + if _, err := Encode("k.json", []byte(`{"priv_key":{"type":"tendermint/PrivKeyEd25519","value":"q2Ej1vT0Yd8kbm0GhX9wJ0Yv6p7c9W3qkH1sXy8bVwQ0c2Vt9tO3y7mJx3qT8f3h4Lr2Vb6ZpN9w0Qn1aYk5sA=="}}`)); err == nil { + t.Fatal("a consensus key must never be QR-encoded") + } +} + +// What a phone or webcam reads off the screen: render each part as an +// image and decode it back. +func TestImagesDecode(t *testing.T) { + data := gentxLike() + parts, _ := Encode("g.json", data) + var a Assembler + for i, p := range parts { + f := filepath.Join(t.TempDir(), "part.png") + if err := PNG(p, f); err != nil { + t.Fatal(err) + } + txt, err := DecodeImage(f) + if err != nil { + t.Fatalf("part %d: %v", i+1, err) + } + if _, err := a.Add(txt); err != nil { + t.Fatal(err) + } + } + if _, got, err := a.File(); err != nil || string(got) != string(data) { + t.Fatalf("decoded images: %v", err) + } + s, err := Terminal(parts[0]) + if err != nil || len(strings.Split(s, "\n")) < 20 { + t.Fatalf("terminal rendering: %v", err) + } + if w := len([]rune(strings.Split(s, "\n")[0])); w > 80 { + t.Errorf("a code is %d columns wide; it must fit an 80-column terminal", w) + } +} diff --git a/internal/router/intents/health.yaml b/internal/router/intents/health.yaml index a7802c7..588f487 100644 --- a/internal/router/intents/health.yaml +++ b/internal/router/intents/health.yaml @@ -37,8 +37,8 @@ patterns: - '^(is |how is |hows |status of |health of )?(the )?(?P[a-z0-9][a-z0-9._]*-[a-z0-9._-]+)( network| chain)?( healthy| ok| okay| up| doing| status)?$' examples: - - is primium-2 healthy - - status of primium-1 + - is mychain-2 healthy + - status of testnet-1 steps: - {as: f, tool: fleet.status, args: {chain: "{chain}"}} answer: | diff --git a/internal/toolkit/context.go b/internal/toolkit/context.go index 350d50b..5512d39 100644 --- a/internal/toolkit/context.go +++ b/internal/toolkit/context.go @@ -4,6 +4,7 @@ import ( "context" "fmt" "io" + "sort" "sync" "time" @@ -68,6 +69,10 @@ type Context struct { txb *tx.Builder txErr error signerB *tx.Builder // TxSigner's choice, reused for the context's life + // sibling is another node of the chain whose gRPC stands in while this + // one is down; noSibling stops a sibling looking for siblings itself + sibling *Context + noSibling bool } var errNoProfile = fmt.Errorf("no active profile — run `cometcli profile add`") @@ -135,10 +140,45 @@ func (c *Context) GRPC() (*grpcclient.Conn, error) { conn, err, c.GRPCFallback = fb, nil, true } } + if err != nil && c.Profile.Endpoints.FallbackGRPC == "" && !c.noSibling { + // the node is down: ask another node of the same chain you run + // (through its own connection) — jail status, upgrade plans and + // params are the chain's, not this node's + if fb, sib := c.siblingGRPCLocked(); fb != nil { + conn, err, c.GRPCFallback, c.sibling = fb, nil, true, sib + } + } c.grpc = conn return conn, err } +// siblingGRPCLocked dials the gRPC of another profile on the same chain. +func (c *Context) siblingGRPCLocked() (*grpcclient.Conn, *Context) { + if c.Cfg == nil || c.Profile.ChainID == "" { + return nil, nil + } + var names []string + for n, p := range c.Cfg.Profiles { + if n != c.Profile.Name && p.ChainID == c.Profile.ChainID && p.Endpoints.GRPC != "" { + names = append(names, n) + } + } + sort.Strings(names) + for i, n := range names { + if i == 2 { + break + } + p := *c.Cfg.Profiles[n] + p.Name = n + sub := &Context{Context: c.Context, Profile: &p, Cfg: c.Cfg, noSibling: true} + if conn, err := sub.GRPC(); err == nil { + return conn, sub + } + sub.Close() + } + return nil, nil +} + // FallbackGRPC switches chain queries to endpoints.fallback_grpc — for a // node that dialed fine but can't answer yet (just restarted, still // loading state). It returns false when there is no fallback. @@ -359,6 +399,10 @@ func (c *Context) Check(g Gate) error { // it per-call would kill auditing for the rest of the session. The owner // that opened the logger closes it at shutdown. func (c *Context) Close() { + if c.sibling != nil { + c.sibling.Close() // closes the stand-in gRPC with its connection + c.grpc, c.sibling = nil, nil + } if c.grpc != nil { c.grpc.Close() } diff --git a/internal/tools/common/common.go b/internal/tools/common/common.go index e5c55e1..e2d0e62 100644 --- a/internal/tools/common/common.go +++ b/internal/tools/common/common.go @@ -7,6 +7,7 @@ import ( query "cosmossdk.io/api/cosmos/base/query/v1beta1" cmtservice "cosmossdk.io/api/cosmos/base/tendermint/v1beta1" "crypto/sha256" + "encoding/hex" "encoding/json" "fmt" "math/big" @@ -118,6 +119,40 @@ func ValoperFromNode(c *toolkit.Context) (string, error) { return "", fmt.Errorf("no validator on chain uses this node's consensus key — it isn't a validator (yet), or runs a different key than the one registered") } +// MonikersByConsAddr maps each bonded validator's consensus address +// (upper-case hex, as CometBFT reports it) to its moniker. +func MonikersByConsAddr(c *toolkit.Context) (map[string]string, error) { + g, err := c.GRPC() + if err != nil { + return nil, err + } + out := map[string]string{} + var key []byte + for { + res, err := g.Staking.Validators(c, &stakingv1beta1.QueryValidatorsRequest{Status: "BOND_STATUS_BONDED", Pagination: &query.PageRequest{Key: key, Limit: 200}}) + if err != nil { + return nil, err + } + for _, v := range res.Validators { + if v.ConsensusPubkey == nil { + continue + } + if pub := protowireFieldBytes(v.ConsensusPubkey.Value, 1); pub != nil { + sum := sha256.Sum256(pub) + name := v.OperatorAddress + if v.Description != nil && v.Description.Moniker != "" { + name = v.Description.Moniker + } + out[strings.ToUpper(hex.EncodeToString(sum[:20]))] = name + } + } + if res.Pagination == nil || len(res.Pagination.NextKey) == 0 { + return out, nil + } + key = res.Pagination.NextKey + } +} + // Account resolves the bech32 address of the key that will sign — the // same choice BroadcastMsgs uses, so messages and signature always agree. func Account(c *toolkit.Context) (string, error) { @@ -473,21 +508,19 @@ func SDKVersion(c *toolkit.Context) string { return "" } -var sdkMinorRe = regexp.MustCompile(`^v0\.(\d+)\.`) - -// TxDec encodes a decimal (e.g. "0.05") for a transaction field typed -// cosmos.Dec: cosmos-sdk v0.54+ decodes txs through the protobuf v2 API, -// where it's the plain decimal; earlier versions take the legacy scaled -// integer (0.05 → 50000000000000000). Sending the wrong form makes 5% -// read as 5×10^16. -func TxDec(c *toolkit.Context, s string) (string, error) { - if m := sdkMinorRe.FindStringSubmatch(SDKVersion(c)); m != nil { - if minor, _ := strconv.Atoi(m[1]); minor >= 54 { - if _, ok := new(big.Rat).SetString(s); !ok { - return "", fmt.Errorf("%q isn't a decimal", s) - } - return s, nil - } +// TxDec encodes a fraction (e.g. "0.05", or "1" for 100%) for a transaction +// field typed cosmos.Dec. Such fields are gogoproto LegacyDec on every SDK +// version (v0.54 too, checked live: a plain "0.06" fails with "cannot +// unmarshal into a *big.Int"), so the wire form is always the scaled +// integer: 0.05 → 50000000000000000. +func TxDec(_ *toolkit.Context, s string) (string, error) { + r, ok := new(big.Rat).SetString(s) + if !ok || strings.ContainsAny(s, "/eE") { + return "", fmt.Errorf("%q isn't a decimal", s) } - return DecScaled(s) + scaled := new(big.Rat).Mul(r, big.NewRat(1000000000000000000, 1)) + if !scaled.IsInt() { + return "", fmt.Errorf("%q has more than 18 decimal places", s) + } + return scaled.Num().String(), nil } diff --git a/internal/tools/common/txdec_test.go b/internal/tools/common/txdec_test.go index c163357..8195dde 100644 --- a/internal/tools/common/txdec_test.go +++ b/internal/tools/common/txdec_test.go @@ -1,25 +1,21 @@ package common import ( - "context" "testing" - - "github.com/abhijitkrm/cometcli/internal/config" - "github.com/abhijitkrm/cometcli/internal/toolkit" ) -// cosmos-sdk v0.54 decodes cosmos.Dec tx fields as plain decimals; older -// versions as the legacy scaled integer. 5% must be sent in the right form. -func TestTxDecFollowsTheSDK(t *testing.T) { - for sdk, want := range map[string]string{"v0.53.4": "50000000000000000", "v0.54.3": "0.05", "v0.55.0": "0.05"} { - c := &toolkit.Context{Context: context.Background(), Profile: &config.Profile{Metadata: map[string]string{"sdk_version": sdk}}} - got, err := TxDec(c, "0.05") - if err != nil || got != want { - t.Errorf("sdk %s: TxDec(0.05) = %q %v, want %q", sdk, got, err, want) +// cosmos.Dec tx fields are gogoproto LegacyDec on every SDK version, v0.54 +// included: the wire form is the scaled integer, and the input is always a +// fraction (1 is 100%, not an already-scaled 10^-18). +func TestTxDecIsAlwaysScaled(t *testing.T) { + for in, want := range map[string]string{"0.05": "50000000000000000", "0.06": "60000000000000000", "1": "1000000000000000000", "0": "0"} { + if got, err := TxDec(nil, in); err != nil || got != want { + t.Errorf("TxDec(%q) = %q %v, want %q", in, got, err, want) } } - c := &toolkit.Context{Context: context.Background(), Profile: &config.Profile{Metadata: map[string]string{"sdk_version": "v0.54.3"}}} - if _, err := TxDec(c, "five"); err == nil { - t.Error("a non-decimal must be refused") + for _, bad := range []string{"five", "1/20", "0.0000000000000000001"} { + if _, err := TxDec(nil, bad); err == nil { + t.Errorf("TxDec(%q) must be refused", bad) + } } } diff --git a/internal/tools/fleet/fleet.go b/internal/tools/fleet/fleet.go index 2550bbe..ad27322 100644 --- a/internal/tools/fleet/fleet.go +++ b/internal/tools/fleet/fleet.go @@ -84,7 +84,11 @@ func (statusTool) Run(c *toolkit.Context, a toolkit.Args) (*toolkit.Result, erro signing = fmt.Sprintf("%d/%d missed", s.Missed, s.Window) } status := "ok" + pp := c.Cfg.Profiles[r.name] switch { + case !s.Reachable && pp != nil && pp.Home == "" && pp.Endpoints.Comet == "": + // a profile made ahead of its node (node provision fills these) + status = "not set up yet" case !s.Reachable: status = "UNREACHABLE" case s.Tombstoned: diff --git a/internal/tools/fs/fs.go b/internal/tools/fs/fs.go index d56a51b..c4eedf0 100644 --- a/internal/tools/fs/fs.go +++ b/internal/tools/fs/fs.go @@ -84,6 +84,15 @@ func (t *target) inRoot(c *toolkit.Context) bool { var protectedRe = regexp.MustCompile(`priv_validator_key\.json$|node_key\.json$|(^|/)\.?mnemonics?(\.txt)?(/|$)|mnemonic|(^|/)keyring-(file|test|os)(/|$)|/\.ssh/id_|/\.cometcli/keys(/|$)|/\.gnupg/|(^|/)keystore(/|$)|(^|/)utc--[0-9]|(^|/)key_seed\.json$`) +// forbiddenContent explains why text may never be written by the agent: +// a node told to skip an upgrade the network applied forks from it. +func forbiddenContent(s string) string { + if strings.Contains(s, "unsafe-skip-upgrades") { + return "--unsafe-skip-upgrades makes a node skip an upgrade the network applied — its state forks from the chain's; the operator decides that" + } + return "" +} + // forbidden explains why a path may never pass through the agent. func forbidden(p string, write bool) string { if m := protectedRe.FindString(strings.ToLower(p)); m != "" { @@ -236,6 +245,9 @@ func (Write) Run(c *toolkit.Context, a toolkit.Args) (*toolkit.Result, error) { return nil, err } content, _ := a["content"].(string) + if f := forbiddenContent(content); f != "" { + return nil, fmt.Errorf("refused: %s", f) + } detail := map[string]any{"path": t.path, "host": t.h.String()} verb := "create" if _, ok := t.exists(c); ok { @@ -296,6 +308,9 @@ func (Edit) Run(c *toolkit.Context, a toolkit.Args) (*toolkit.Result, error) { } oldS, _ := a["old_string"].(string) newS, _ := a["new_string"].(string) + if f := forbiddenContent(newS); f != "" { + return nil, fmt.Errorf("refused: %s", f) + } if oldS == "" { return nil, fmt.Errorf("old_string is empty — use write to create a file") } diff --git a/internal/tools/govtool/gov.go b/internal/tools/govtool/gov.go index 4b69ad8..07da591 100644 --- a/internal/tools/govtool/gov.go +++ b/internal/tools/govtool/gov.go @@ -56,12 +56,12 @@ type proposeTool struct{} func (proposeTool) Name() string { return "gov.propose" } func (proposeTool) Desc() string { return "Submit a governance proposal: kind=text (title, summary), kind=upgrade (name = the release's upgrade handler name, height or in_blocks; " + - "refused when the vote can't finish before the height), kind=params (module + set: built from the chain's current params, only the named fields changed — title/summary default to the diff), or kind=messages (JSON array of {\"@type\":…} msgs, authority = the gov module). " + + "refused when the vote can't finish before the height), kind=params (module + set: built from the chain's current params, only the named fields changed — title/summary default to the diff), kind=cancel-upgrade (cancels the scheduled plan; refused when none is, or when the vote can't pass before its height), or kind=messages (JSON array of {\"@type\":…} msgs, authority = the gov module). " + "Deposit defaults to the chain's minimum. Returns the proposal id." } func (proposeTool) Schema() map[string]any { return toolkit.ObjSchema(common.WithTx(map[string]any{ - "kind": toolkit.Enum("proposal kind", "text", "upgrade", "params", "messages"), + "kind": toolkit.Enum("proposal kind", "text", "upgrade", "cancel-upgrade", "params", "messages"), "module": toolkit.Str(`params: the module — evm, feemarket, erc20, staking, slashing, gov, mint, distribution, … (any with MsgUpdateParams on this chain)`), "set": toolkit.Str(`params: the changes, ';'-separated — field=value, list+=items, list-=items (nested: a.b=value). e.g. "active_static_precompiles+=ics02" (precompiles by name) or "base_fee_change_denominator=16; min_gas_price=0.5"`), "dry_run": toolkit.Bool("build and show the proposal (the diff for params) without signing or sending it"), @@ -122,6 +122,36 @@ func Build(c *toolkit.Context, a toolkit.Args, proposer string) (*Plan, error) { summary = "Changes " + pp.Module + " params (all others unchanged):\n" + strings.Join(pp.Diff, "\n") } } + if a.String("kind", "") == "upgrade" && a.String("name", "") != "" { + if title == "" { + title = "Software upgrade " + a.String("name", "") + } + if summary == "" { + summary = "Upgrade to " + a.String("name", "") + if img := a.String("image", ""); img != "" { + summary += " (image " + img + ")" + } + summary += ". Validators: build or stage the new binary before the halt height, then switch after the halt." + } + } + // cancelling: there must be a plan, and its name makes the defaults + var cancelPlan *upgradev1beta1.Plan + if a.String("kind", "") == "cancel-upgrade" { + res, err := g.Upgrade.CurrentPlan(c, &upgradev1beta1.QueryCurrentPlanRequest{}) + if err != nil { + return nil, fmt.Errorf("current upgrade plan: %w", err) + } + if res.Plan == nil { + return nil, fmt.Errorf("no upgrade is scheduled — nothing to cancel") + } + cancelPlan = res.Plan + if title == "" { + title = "Cancel software upgrade " + cancelPlan.Name + } + if summary == "" { + summary = fmt.Sprintf("Cancels the scheduled upgrade %s at height %d. Nodes keep running the current binary.", cancelPlan.Name, cancelPlan.Height) + } + } if title == "" || summary == "" { return nil, fmt.Errorf("title and summary are required") } @@ -231,6 +261,34 @@ func Build(c *toolkit.Context, a toolkit.Args, proposer string) (*Plan, error) { } msg.Messages = []*anypb.Any{any} p.Detail["upgrade"] = fmt.Sprintf("%s at %d", name, height) + case "cancel-upgrade": + cur, blockTime, err := chainClock(c) + if err != nil { + return nil, err + } + // the cancel must pass before the chain halts at the plan's height + left := cancelPlan.Height - cur + need := int64(votingPeriod/blockTime) + 1 + if left <= need { + hint := "" + if !expedited && params.ExpeditedVotingPeriod != nil { + hint = fmt.Sprintf(" — an expedited vote takes %s (expedited=true)", params.ExpeditedVotingPeriod.AsDuration()) + } + return nil, fmt.Errorf("the plan %s halts the chain in %d blocks (≈ %s), but this vote takes %s ≈ %d blocks: it can't pass in time%s", + cancelPlan.Name, left, (time.Duration(left) * blockTime).Round(time.Second), votingPeriod, need, hint) + } + p.Notes = append(p.Notes, fmt.Sprintf("cancels %s at height %d (≈ %s from now); the vote ends ≈ %d blocks before it", + cancelPlan.Name, cancelPlan.Height, (time.Duration(left)*blockTime).Round(time.Minute), left-need)) + gov, err := ModuleAddress("gov", accountPrefix(proposer)) + if err != nil { + return nil, err + } + any, err := cosmosAny(&upgradev1beta1.MsgCancelUpgrade{Authority: gov}) + if err != nil { + return nil, err + } + msg.Messages = []*anypb.Any{any} + p.Detail["cancels"] = fmt.Sprintf("%s at %d", cancelPlan.Name, cancelPlan.Height) case "messages": anys, err := parseMessages(a.String("messages", "")) if err != nil { diff --git a/internal/tools/govtool/gov_test.go b/internal/tools/govtool/gov_test.go index 7f4ef90..3d7074f 100644 --- a/internal/tools/govtool/gov_test.go +++ b/internal/tools/govtool/gov_test.go @@ -50,3 +50,15 @@ func TestParseMessages(t *testing.T) { t.Fatal("bad coin accepted") } } + +func TestCancelUpgradeIsAKind(t *testing.T) { + var p proposeTool + kinds := p.Schema()["properties"].(map[string]any)["kind"].(map[string]any)["enum"] + found := false + for _, k := range kinds.([]string) { + found = found || k == "cancel-upgrade" + } + if !found { + t.Fatalf("kinds %v: cancel-upgrade missing", kinds) + } +} diff --git a/internal/tools/networktool/create.go b/internal/tools/networktool/create.go index 55befe4..60050e4 100644 --- a/internal/tools/networktool/create.go +++ b/internal/tools/networktool/create.go @@ -67,7 +67,7 @@ func CreateNetwork(ctxFor func(*config.Profile) *toolkit.Context, spec *netspec. } unit := p.Service.Unit if unit == "" { - unit = "primium-validator" + unit = netspec.ContainerName(spec.Chain.ID, netspec.Validator) } ports := netspec.Compose{Services: spec.Services, PortOffset: o.PortBase + i*o.PortStep}.HostPorts() if err := portsFree(c, h, ports, unit); err != nil { @@ -138,7 +138,7 @@ func CreateNetwork(ctxFor func(*config.Profile) *toolkit.Context, spec *netspec. if u := profiles[i].Service.Unit; u != "" { return u } - return "primium-validator" + return netspec.ContainerName(spec.Chain.ID, netspec.Validator) } if p := profiles[i]; p.Transport.Type == "ssh" && p.Transport.Host != "" { return p.Transport.Host diff --git a/internal/tools/networktool/genesis.go b/internal/tools/networktool/genesis.go index c3a1264..506748a 100644 --- a/internal/tools/networktool/genesis.go +++ b/internal/tools/networktool/genesis.go @@ -94,6 +94,7 @@ func PrepareValidator(c *toolkit.Context, spec *netspec.Spec, home string, kr Ke if out, code, err := run(c, h, fmt.Sprintf("%s init %s --chain-id %s --home %s >/dev/null 2>&1 && echo ok", ev, common.ShellQ(n.Moniker), common.ShellQ(spec.Chain.ID), containerHome), 3*time.Minute); err != nil || code != 0 || !strings.Contains(out, "ok") { return nil, "", fmt.Errorf("init on %s failed: %v %s", c.Profile.Name, err, out) } + lockHome(c, h, abs) // the operator key, created in the node's keyring; --output json // carries the mnemonic, which only the caller sees res, err := host.ExecIn(c, h, fmt.Sprintf("%s keys add %s --keyring-backend %s --home %s --output json 2>/dev/null", ev, common.ShellQ(n.Moniker), kr.Backend, containerHome), kr.stdin(2), 1<<20) @@ -188,6 +189,18 @@ func BuildBase(c *toolkit.Context, spec *netspec.Spec, nodes []*GenesisNode, ext // Gentx signs a validator's genesis transaction on its own host, against // the base genesis; only the gentx comes back. func Gentx(n *GenesisNode, spec *netspec.Spec, base []byte, kr KeyringOpts) ([]byte, error) { + return gentxIn(n, evmdIn(n.User, n.Home, n.Image), spec, base, kr, "") +} + +// GentxAt is Gentx with the address peers dial written into the memo. +func GentxAt(n *GenesisNode, spec *netspec.Spec, base []byte, kr KeyringOpts, addr string) ([]byte, error) { + return gentxIn(n, evmdIn(n.User, n.Home, n.Image), spec, base, kr, addr) +} + +// gentxIn signs with ev, the evmd command for the node's home. addr is +// the ip:port peers dial, written into the gentx memo; "" lets evmd find +// the machine's own IP (it can't without a network). +func gentxIn(n *GenesisNode, ev string, spec *netspec.Spec, base []byte, kr KeyringOpts, addr string) ([]byte, error) { c := n.Ctx h, err := c.Host() if err != nil { @@ -204,7 +217,9 @@ func Gentx(n *GenesisNode, spec *netspec.Spec, base []byte, kr KeyringOpts) ([]b if gp, ok := new(big.Int).SetString(spec.MinGasPrice, 10); ok { fee = "--fees " + new(big.Int).Mul(gp, big.NewInt(200000)).String() + spec.GasDenom() } - ev := evmdIn(n.User, n.Home, n.Image) + if ip, port, ok := strings.Cut(addr, ":"); ok && ip != "" { + fee += " --ip " + common.ShellQ(ip) + " --p2p-port " + common.ShellQ(port) + " --node-id " + common.ShellQ(n.NodeID) + } script := fmt.Sprintf("rm -rf %[1]s/config/gentx && %[2]s genesis gentx %[3]s %[4]s%[5]s --keyring-backend %[6]s --chain-id %[7]s %[8]s --home %[9]s >/dev/null 2>&1 && cat %[1]s/config/gentx/*.json", common.ShellQ(n.Home), ev, common.ShellQ(n.Moniker), stake, spec.Chain.Denom, kr.Backend, common.ShellQ(spec.Chain.ID), fee, containerHome) res, err := host.ExecIn(c, h, script, kr.stdin(1), 1<<20) diff --git a/internal/tools/networktool/hostprep.go b/internal/tools/networktool/hostprep.go index 674d121..2b81035 100644 --- a/internal/tools/networktool/hostprep.go +++ b/internal/tools/networktool/hostprep.go @@ -6,6 +6,7 @@ import ( "time" "github.com/abhijitkrm/cometcli/internal/client/host" + "github.com/abhijitkrm/cometcli/internal/netspec" "github.com/abhijitkrm/cometcli/internal/toolkit" ) @@ -19,8 +20,15 @@ func (hostPrepareTool) Desc() string { "the compose plugin, git, jq and curl (Ubuntu/Debian or Amazon Linux/RHEL, with passwordless sudo), and lets the " + "user run docker. Checks first and changes nothing that's already there; shows the plan and asks." } -func (hostPrepareTool) Schema() map[string]any { return toolkit.ObjSchema(nil) } -func (hostPrepareTool) Tier() toolkit.Tier { return toolkit.TierLocalChange } +func (hostPrepareTool) Schema() map[string]any { + return toolkit.ObjSchema(map[string]any{ + "offline_ready": toolkit.Bool("also get it ready to make validator keys with no network: the node image loaded and evmd checked with networking off (run while still online)"), + "image": toolkit.Str("node image for offline use (default: the profile's network spec image)"), + "image_tar": toolkit.Str("load the image from this tar on the host (docker save output carried over) when it isn't there"), + "save_image": toolkit.Str("also write the image to this tar on the host, to carry to another offline machine"), + }) +} +func (hostPrepareTool) Tier() toolkit.Tier { return toolkit.TierLocalChange } func (hostPrepareTool) Timeout(toolkit.Args) time.Duration { return 20 * time.Minute } @@ -122,7 +130,66 @@ sudo -n chmod +x /usr/local/lib/docker/cli-plugins/docker-compose`, composeVersi func shellQ(s string) string { return "'" + strings.ReplaceAll(s, "'", `'\''`) + "'" } -func (hostPrepareTool) Run(c *toolkit.Context, _ toolkit.Args) (*toolkit.Result, error) { +func (t hostPrepareTool) Run(c *toolkit.Context, a toolkit.Args) (*toolkit.Result, error) { + res, err := t.prepare(c) + if err != nil || !a.Bool("offline_ready", false) { + return res, err + } + off, err := offlineReady(c, a) + if err != nil { + return nil, err + } + res.Text += "\n" + off + return res, nil +} + +// offlineReady checks (and makes) what making keys with no network needs: +// the image on the machine and evmd running with networking off. +func offlineReady(c *toolkit.Context, a toolkit.Args) (string, error) { + h, err := c.Host() + if err != nil { + return "", err + } + image := a.String("image", "") + if image == "" { + spec, _, err := netspec.Load(c.Profile.ChainID) + if err != nil { + return "", fmt.Errorf("offline_ready: which image? pass image, or give the profile a chain id with a network spec") + } + image = ImageFor(spec) + } + var b strings.Builder + if _, code, _ := run(c, h, "docker image inspect "+shellQ(image)+" >/dev/null 2>&1", time.Minute); code != 0 { + tar := a.String("image_tar", "") + if tar == "" { + return "", fmt.Errorf("image %s isn't on %s — build it (upgrade build) or carry it over: docker save on a machine that has it, then image_tar=", image, c.Profile.Name) + } + if out, code, _ := run(c, h, "docker load -i "+shellHome(tar)+" 2>&1 | tail -1", 10*time.Minute); code != 0 { + return "", fmt.Errorf("docker load %s: %s", tar, strings.TrimSpace(out)) + } + fmt.Fprintf(&b, "✓ loaded %s from %s\n", image, tar) + } + out, code, _ := run(c, h, "docker run --rm --network none --entrypoint evmd "+shellQ(image)+" version 2>&1 | tail -1", 2*time.Minute) + if code != 0 { + return "", fmt.Errorf("evmd doesn't run from %s with networking off: %s", image, strings.TrimSpace(out)) + } + fmt.Fprintf(&b, "✓ evmd %s runs from %s with no network\n", strings.TrimSpace(out), image) + if tar := a.String("save_image", ""); tar != "" { + out, code, _ := run(c, h, "docker save -o "+shellHome(tar)+" "+shellQ(image)+" && chmod 644 "+shellHome(tar)+" && du -h "+shellHome(tar)+" | cut -f1", 10*time.Minute) + if code != 0 { + return "", fmt.Errorf("docker save: %s", strings.TrimSpace(out)) + } + fmt.Fprintf(&b, "✓ image saved to %s (%s) — carry it to the offline machine and run host prepare offline_ready image_tar= there\n", tar, strings.TrimSpace(out)) + } + if o := CheckOffline(c, h); o.Online { + fmt.Fprintf(&b, "· %s is online now (%s): disconnect it before genesis join/gentx — they refuse to make keys on a connected machine unless --allow-online\n", c.Profile.Name, o.Detail) + } else { + b.WriteString("✓ no network on this machine: ready for genesis join and gentx\n") + } + return strings.TrimSpace(b.String()), nil +} + +func (hostPrepareTool) prepare(c *toolkit.Context) (*toolkit.Result, error) { if c.Profile == nil { return nil, fmt.Errorf("host.prepare needs a profile for the host (cometcli profile add --ssh-host …)") } diff --git a/internal/tools/networktool/network.go b/internal/tools/networktool/network.go index 07e54fe..06d8f99 100644 --- a/internal/tools/networktool/network.go +++ b/internal/tools/networktool/network.go @@ -219,7 +219,7 @@ func containerInfo(c *toolkit.Context, h host.Host) (version string, cmd []strin _ = json.Unmarshal([]byte(parts[0]), &cmd) version = parts[1] if version == "" || version == "" { - // primium-v0.7.2 or primium-evm:v0.7.2 + // mychain-v0.7.2 or mychain-evm:v0.7.2 if _, tag, ok := strings.Cut(parts[2], ":"); ok { version = tag } else if i := strings.LastIndex(parts[2], "-v"); i >= 0 { diff --git a/internal/tools/networktool/provision.go b/internal/tools/networktool/provision.go index 59e7107..0336fbd 100644 --- a/internal/tools/networktool/provision.go +++ b/internal/tools/networktool/provision.go @@ -85,7 +85,7 @@ func (provisionTool) Run(c *toolkit.Context, a toolkit.Args) (*toolkit.Result, e moniker := a.String("moniker", p.Name) unit := p.Service.Unit if unit == "" { - unit = "primium-" + string(role) + unit = netspec.ContainerName(spec.Chain.ID, role) } image := a.String("image", "") if image == "" { @@ -107,10 +107,6 @@ func (provisionTool) Run(c *toolkit.Context, a toolkit.Args) (*toolkit.Result, e } } - // the image must be on the host already (build it: upgrade build) - if _, code, _ := run(c, h, "docker image inspect "+common.ShellQ(image)+" >/dev/null 2>&1", time.Minute); code != 0 { - return nil, fmt.Errorf("image %s isn't on %s — build it there first: cometcli --profile %s upgrade build --tag %s", image, p.Name, p.Name, spec.Image.Tag) - } // the home: absolute, writable, and never someone's existing validator home := a.String("home", spec.HostHome) if home == "" { @@ -131,6 +127,18 @@ func (provisionTool) Run(c *toolkit.Context, a toolkit.Args) (*toolkit.Result, e if exists && !reconfigure { return nil, fmt.Errorf("%s already holds a node (config/priv_validator_key.json) — re-initialising would replace its keys and state (double-sign risk). To re-render its configs only: reconfigure=true", home) } + if exists && a.String("image", "") == "" { + // a node past an upgrade runs a newer image than the spec's + // (genesis) tag: re-rendering must not take it back to that binary + if cur := currentImage(c, h, unit, home); cur != "" && cur != image { + step("· keeping the image %s runs now: %s (the spec's is %s, for nodes starting from genesis)", unit, cur, image) + image = cur + } + } + // the image must be on the host already (build it: upgrade build) + if _, code, _ := run(c, h, "docker image inspect "+common.ShellQ(image)+" >/dev/null 2>&1", time.Minute); code != 0 { + return nil, fmt.Errorf("image %s isn't on %s — build it there first: cometcli --profile %s upgrade build --tag %s", image, p.Name, p.Name, spec.Image.Tag) + } // peers and the advertised address peers := splitList(a.String("peers", "")) @@ -199,12 +207,22 @@ func (provisionTool) Run(c *toolkit.Context, a toolkit.Args) (*toolkit.Result, e } } + lockHome(c, h, home) // new, or one cometcli made before it did this + // configs, rendered for the role from the spec version := spec.Image.Tag if m := regexp.MustCompile(`v\d+\.\d+\.\d+`).FindString(image); m != "" { version = m } - patches := spec.Patches(role, version, netspec.NodeParams{Moniker: moniker, ExternalAddress: extAddr, PersistentPeers: peers, Prod: a.Bool("prod", false)}) + inboundClosed := false + if raw, err := h.ReadFile(c, path.Join(home, "config", "config.toml")); err == nil { + var cur map[string]any + if _, err := toml.Decode(string(raw), &cur); err == nil { + n, ok := netspec.Lookup(cur, []string{"p2p", "max_num_inbound_peers"}).(int64) + inboundClosed = ok && n == 0 + } + } + patches := spec.Patches(role, version, netspec.NodeParams{Moniker: moniker, ExternalAddress: extAddr, PersistentPeers: peers, Prod: a.Bool("prod", false), InboundClosed: inboundClosed}) for _, file := range []string{"config.toml", "app.toml", "client.toml"} { fp := path.Join(home, "config", file) raw, err := h.ReadFile(c, fp) @@ -441,3 +459,21 @@ func portsFree(c *toolkit.Context, h host.Host, ports []int, unit string) error } return nil } + +// currentImage is the image a node runs: its container's, else the one +// its compose file names. +func currentImage(c *toolkit.Context, h host.Host, unit, home string) string { + if out, code, _ := run(c, h, "docker inspect -f '{{.Config.Image}}' "+common.ShellQ(unit)+" 2>/dev/null", time.Minute); code == 0 { + if img := strings.TrimSpace(out); img != "" { + return img + } + } + out, _, _ := run(c, h, "sed -n 's/^ *image: *//p' "+common.ShellQ(path.Join(home, "docker-compose.yml"))+" 2>/dev/null | head -1", time.Minute) + return strings.Trim(strings.TrimSpace(out), `"'`) +} + +// lockHome makes a node's config and data dirs owner-only, as sec perms +// expects: they hold the consensus key and the signing state. +func lockHome(c *toolkit.Context, h host.Host, home string) { + _, _, _ = run(c, h, "chmod 700 "+common.ShellQ(path.Join(home, "config"))+" "+common.ShellQ(path.Join(home, "data"))+" 2>/dev/null", time.Minute) +} diff --git a/internal/tools/networktool/workflowb.go b/internal/tools/networktool/workflowb.go new file mode 100644 index 0000000..949187f --- /dev/null +++ b/internal/tools/networktool/workflowb.go @@ -0,0 +1,709 @@ +package networktool + +// Workflow B: a genesis with validators the coordinator can't reach. +// Outside validators make their keys on their own (ideally offline) +// machines and exchange only public files with the coordinator: +// +// validator: genesis join → .join.json (account, node id, consensus pubkey) +// coordinator: genesis base → kit/ (base genesis funding every account) +// validator: genesis gentx → .gentx.json (gentx signed against the kit) +// coordinator: genesis collect → packages// (final genesis + every peer) +// validator: genesis install → node started +// +// Each file carries checksums and is verified where it arrives. Keys and +// mnemonics never leave the key machine: they're made inside a container +// with no network, and a new mnemonic goes to a 0600 file on that machine. + +import ( + "crypto/sha256" + "encoding/hex" + "encoding/json" + "fmt" + "os" + "path" + "path/filepath" + "regexp" + "sort" + "strings" + "time" + + "github.com/abhijitkrm/cometcli/internal/client/host" + "github.com/abhijitkrm/cometcli/internal/keys" + "github.com/abhijitkrm/cometcli/internal/netspec" + "github.com/abhijitkrm/cometcli/internal/toolkit" + "github.com/abhijitkrm/cometcli/internal/tools/common" +) + +// FormatVersion is the version of the join, kit, gentx and package files. +const FormatVersion = 1 + +// Offline records whether the key machine had a network when keys were +// made or a gentx signed. +type Offline struct { + Checked bool `json:"checked"` + Online bool `json:"online"` + Detail string `json:"detail,omitempty"` +} + +// CheckOffline looks for a default route or an up interface with a global +// address on the machine. Docker's own bridges don't count: every machine +// with docker has them, connected or not. +func CheckOffline(c *toolkit.Context, h host.Host) Offline { + out, _, _ := run(c, h, `{ ip route show default 2>/dev/null; ip -o addr show up scope global 2>/dev/null | awk '$2 !~ /^(docker|br-|veth|virbr|cni|flannel|lo)/ {print "if " $2 " " $4}'; } || true +if ! command -v ip >/dev/null 2>&1; then route -n get default 2>/dev/null | awk '/interface:/{print "route via " $2}'; fi`, 30*time.Second) + out = strings.TrimSpace(out) + if out == "" { + return Offline{Checked: true} + } + lines := strings.Split(out, "\n") + return Offline{Checked: true, Online: true, Detail: strings.Join(lines[:min(len(lines), 3)], "; ")} +} + +// offlineGate refuses a connected key machine unless allowed, in which +// case the result is marked as made online. +func offlineGate(c *toolkit.Context, h host.Host, allowOnline bool) (Offline, error) { + o := CheckOffline(c, h) + if o.Online && !allowOnline { + return o, fmt.Errorf("this machine is online (%s) — validator keys should be made with no network: disconnect it and run again, or pass --allow-online (the result is marked as made online)", o.Detail) + } + return o, nil +} + +// evmdOffline runs evmd in the image with no network at all. +func evmdOffline(user, home, image string) string { + return strings.Replace(evmdIn(user, home, image), "docker run --rm -i", "docker run --rm -i --network none", 1) +} + +// --- join ------------------------------------------------------------------ + +// Join is what a validator sends before the base genesis exists: public +// facts only. +type Join struct { + Version int `json:"version"` + ChainID string `json:"chain_id"` + Moniker string `json:"moniker"` + Account string `json:"account"` + NodeID string `json:"node_id"` + ConsKey string `json:"consensus_pubkey"` + IP string `json:"ip"` + P2PPort int `json:"p2p_port"` + Offline Offline `json:"offline"` +} + +// JoinOpts are a validator's choices for genesis join. +type JoinOpts struct { + Home string + Moniker string + IP string + P2PPort int + Keyring KeyringOpts + RecoverFrom string // a mnemonic file on the key machine (else a new key) + MnemonicTo string // where a new mnemonic goes on the key machine (0600) + AllowOnline bool +} + +var ( + nodeIDRe = regexp.MustCompile(`^[0-9a-f]{40}$`) + monikerRe = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9._-]{0,63}$`) +) + +// PrepareJoin makes a validator's keys on the machine c reaches: node and +// consensus keys from init, the operator key in the node's own keyring +// (new, or recovered from a mnemonic file there). Nothing secret comes +// back. +func PrepareJoin(c *toolkit.Context, spec *netspec.Spec, o JoinOpts) (*Join, string, error) { + if !monikerRe.MatchString(o.Moniker) { + return nil, "", fmt.Errorf("moniker %q: letters, digits, . _ - (up to 64)", o.Moniker) + } + h, err := c.Host() + if err != nil { + return nil, "", err + } + off, err := offlineGate(c, h, o.AllowOnline) + if err != nil { + return nil, "", err + } + image := ImageFor(spec) + if _, code, _ := run(c, h, "docker image inspect "+common.ShellQ(image)+" >/dev/null 2>&1", time.Minute); code != 0 { + return nil, "", fmt.Errorf("image %s isn't on this machine — load it while still online: cometcli host prepare --offline-ready --image %s", image, image) + } + abs, user, hasKey, err := prepareHome(c, h, o.Home) + if err != nil { + return nil, "", err + } + if hasKey { + return nil, "", fmt.Errorf("%s already holds a node (config/priv_validator_key.json) — use a fresh home", abs) + } + ev := evmdOffline(user, abs, image) + // a join that fails part-way removes the home it made (it held no node + // before), so it can simply be run again + done := false + defer func() { + if !done { + _, _, _ = run(c, h, "rm -rf "+common.ShellQ(abs), time.Minute) + } + }() + if out, code, err := run(c, h, fmt.Sprintf("%s init %s --chain-id %s --home %s >/dev/null 2>&1 && echo ok", ev, common.ShellQ(o.Moniker), common.ShellQ(spec.Chain.ID), containerHome), 3*time.Minute); err != nil || code != 0 || !strings.Contains(out, "ok") { + return nil, "", fmt.Errorf("init failed: %v %s", err, out) + } + lockHome(c, h, abs) + j := &Join{Version: FormatVersion, ChainID: spec.Chain.ID, Moniker: o.Moniker, IP: o.IP, P2PPort: o.P2PPort, Offline: off} + if j.P2PPort == 0 { + j.P2PPort = 26656 + } + + // the operator key: the mnemonic stays in a file on this machine — the + // script returns only the address + mnemonicTo := o.MnemonicTo + if mnemonicTo == "" { + mnemonicTo = abs + ".operator.mnemonic" + } + var script string + if o.RecoverFrom != "" { + script = fmt.Sprintf(`set -e; src=%[1]s; case "$src" in "~/"*) src="$HOME${src#\~}";; esac +test -f "$src" || { echo "no mnemonic file $src" >&2; exit 3; } +umask 077; tr -s ' \t\r\n' ' ' < "$src" | sed 's/^ //; s/ $//' | tr -d '\n' > %[2]s/.import.mnemonic +trap 'rm -f %[2]s/.import.mnemonic' EXIT +out=$(%[3]s keys add %[4]s --recover --source %[5]s/.import.mnemonic --keyring-backend %[6]s --home %[5]s --output json 2>%[2]s/.import.err) || { grep -v -i -E 'mnemonic|passphrase' %[2]s/.import.err | tail -3 >&2; rm -f %[2]s/.import.err; exit 4; } +rm -f %[2]s/.import.err; echo "$out" | sed -n 's/.*"address":"\([^"]*\)".*/\1/p'`, + shellHome(o.RecoverFrom), common.ShellQ(abs), ev, common.ShellQ(o.Moniker), containerHome, o.Keyring.Backend) + } else { + script = fmt.Sprintf(`set -e; umask 077; out=%[1]s/.newkey.json; trap 'rm -f "$out"' EXIT +%[2]s keys add %[3]s --keyring-backend %[4]s --home %[5]s --output json > "$out" 2>/dev/null +m=%[6]s; case "$m" in "~/"*) m="$HOME${m#\~}";; esac +sed -n 's/.*"mnemonic":"\([^"]*\)".*/\1/p' "$out" > "$m"; chmod 600 "$m"; test -s "$m" +sed -n 's/.*"address":"\([^"]*\)".*/\1/p' "$out"`, + common.ShellQ(abs), ev, common.ShellQ(o.Moniker), o.Keyring.Backend, containerHome, shellHome(mnemonicTo)) + } + res, err := host.ExecIn(c, h, script, o.Keyring.stdin(2), 1<<16) + if err != nil || res.Code != 0 { + return nil, "", fmt.Errorf("creating the operator key failed (exit %d): %s", res.Code, strings.TrimSpace(res.Output)) + } + j.Account = strings.TrimSpace(lastLine(res.Output)) + if !strings.HasPrefix(j.Account, spec.Bech32()+"1") { + return nil, "", fmt.Errorf("unexpected operator address %q", j.Account) + } + id, _, _ := run(c, h, ev+" comet show-node-id --home "+containerHome+" 2>/dev/null | tail -1", time.Minute) + j.NodeID = strings.TrimSpace(id) + pk, _, _ := run(c, h, ev+" comet show-validator --home "+containerHome+" 2>/dev/null | tail -1", time.Minute) + var cons struct { + Key string `json:"key"` + } + _ = json.Unmarshal([]byte(strings.TrimSpace(pk)), &cons) + j.ConsKey = cons.Key + if !nodeIDRe.MatchString(j.NodeID) || j.ConsKey == "" { + return nil, "", fmt.Errorf("reading the node id and consensus key failed") + } + where := mnemonicTo + if o.RecoverFrom != "" { + where = "" + } + done = true + return j, where, nil +} + +func lastLine(s string) string { + l := strings.Split(strings.TrimSpace(s), "\n") + return l[len(l)-1] +} + +// ReadJoin reads and checks a join file. +func ReadJoin(path string, spec *netspec.Spec) (*Join, error) { + raw, err := os.ReadFile(path) + if err != nil { + return nil, err + } + var j Join + if err := json.Unmarshal(raw, &j); err != nil { + return nil, fmt.Errorf("%s: %w", path, err) + } + switch { + case j.Version != FormatVersion: + return nil, fmt.Errorf("%s: format version %d, want %d", path, j.Version, FormatVersion) + case j.ChainID != spec.Chain.ID: + return nil, fmt.Errorf("%s: chain %q, not %s", path, j.ChainID, spec.Chain.ID) + case !monikerRe.MatchString(j.Moniker): + return nil, fmt.Errorf("%s: bad moniker %q", path, j.Moniker) + case !strings.HasPrefix(j.Account, spec.Bech32()+"1"): + return nil, fmt.Errorf("%s: account %q isn't a %s address", path, j.Account, spec.Bech32()) + case !nodeIDRe.MatchString(j.NodeID): + return nil, fmt.Errorf("%s: bad node id %q", path, j.NodeID) + case j.ConsKey == "": + return nil, fmt.Errorf("%s: no consensus pubkey", path) + } + if _, err := keys.ValAddress(j.Account); err != nil { + return nil, fmt.Errorf("%s: account %q: %w", path, j.Account, err) + } + return &j, nil +} + +// --- kit ------------------------------------------------------------------- + +// Kit is what the coordinator hands every validator after the base +// genesis: the base genesis, the spec, and who's in it. +type Kit struct { + Version int `json:"version"` + ChainID string `json:"chain_id"` + Denom string `json:"denom"` + Stake string `json:"stake"` + Image string `json:"image"` + Validators []KitValidator `json:"validators"` + Created string `json:"created"` +} + +// KitValidator is a genesis validator as everyone sees it. +type KitValidator struct { + Moniker string `json:"moniker"` + Account string `json:"account"` + NodeID string `json:"node_id"` + ConsKey string `json:"consensus_pubkey,omitempty"` + Address string `json:"address"` // ip:port peers dial + Offline bool `json:"keys_made_offline"` +} + +// writeManifest writes SHA256SUMS over files in dir and returns its hash +// (the id the next step checks). +func writeManifest(dir string, files []string) (string, error) { + sort.Strings(files) + var b strings.Builder + for _, f := range files { + raw, err := os.ReadFile(filepath.Join(dir, f)) + if err != nil { + return "", err + } + sum := sha256.Sum256(raw) + fmt.Fprintf(&b, "%s %s\n", hex.EncodeToString(sum[:]), f) + } + if err := os.WriteFile(filepath.Join(dir, "SHA256SUMS"), []byte(b.String()), 0o644); err != nil { + return "", err + } + id := sha256.Sum256([]byte(b.String())) + return hex.EncodeToString(id[:]), nil +} + +// verifyManifest checks every file SHA256SUMS lists and returns its hash. +func verifyManifest(dir string) (string, error) { + raw, err := os.ReadFile(filepath.Join(dir, "SHA256SUMS")) + if err != nil { + return "", fmt.Errorf("%s has no SHA256SUMS — not a cometcli kit or package", dir) + } + for _, l := range strings.Split(strings.TrimSpace(string(raw)), "\n") { + f := strings.Fields(l) + if len(f) != 2 || strings.ContainsAny(f[1], `/\`) { + return "", fmt.Errorf("SHA256SUMS: bad line %q", l) + } + b, err := os.ReadFile(filepath.Join(dir, f[1])) + if err != nil { + return "", fmt.Errorf("%s is missing", f[1]) + } + sum := sha256.Sum256(b) + if hex.EncodeToString(sum[:]) != f[0] { + return "", fmt.Errorf("%s doesn't match its checksum — the kit was changed or damaged in transit", f[1]) + } + } + id := sha256.Sum256(raw) + return hex.EncodeToString(id[:]), nil +} + +const kitReadme = `Genesis kit for %[1]s + +Check it first: every file is listed with its sha256 in SHA256SUMS, and +cometcli verifies them before using anything. + +On your validator machine — offline if you can (the image must be loaded: +cometcli host prepare --offline-ready, while still online): + + cometcli genesis gentx --kit --home + +Send back the one file it writes (.gentx.json). It holds your +signed gentx and public node info — nothing secret. Keep your node home's +config/priv_validator_key.json and your operator mnemonic backed up off +the machine, and never run the same key on two machines (double-sign). +` + +// WriteKit writes the kit to dir and returns its id. +func WriteKit(dir string, k *Kit, base []byte, specPath string) (string, error) { + if err := os.MkdirAll(dir, 0o755); err != nil { + return "", err + } + spec, err := os.ReadFile(specPath) + if err != nil { + return "", err + } + kj, _ := json.MarshalIndent(k, "", " ") + files := map[string][]byte{ + "base-genesis.json": base, "spec.yaml": spec, "kit.json": kj, + "README.txt": []byte(fmt.Sprintf(kitReadme, k.ChainID)), + } + var names []string + for n, b := range files { + if err := os.WriteFile(filepath.Join(dir, n), b, 0o644); err != nil { + return "", err + } + names = append(names, n) + } + return writeManifest(dir, names) +} + +// ReadKit verifies a kit and returns it with its base genesis, its spec +// and its id. +func ReadKit(dir string) (*Kit, []byte, *netspec.Spec, string, error) { + id, err := verifyManifest(dir) + if err != nil { + return nil, nil, nil, "", err + } + var k Kit + raw, _ := os.ReadFile(filepath.Join(dir, "kit.json")) + if err := json.Unmarshal(raw, &k); err != nil || k.Version != FormatVersion { + return nil, nil, nil, "", fmt.Errorf("kit.json: unreadable or format version %d", k.Version) + } + base, _ := os.ReadFile(filepath.Join(dir, "base-genesis.json")) + spec, _, err := netspec.Load(filepath.Join(dir, "spec.yaml")) + if err != nil { + return nil, nil, nil, "", err + } + if spec.Chain.ID != k.ChainID { + return nil, nil, nil, "", fmt.Errorf("the kit's spec is for %s, the kit for %s", spec.Chain.ID, k.ChainID) + } + var g struct { + ChainID string `json:"chain_id"` + } + if json.Unmarshal(base, &g) != nil || g.ChainID != k.ChainID { + return nil, nil, nil, "", fmt.Errorf("the base genesis isn't for %s", k.ChainID) + } + return &k, base, spec, id, nil +} + +// --- gentx ----------------------------------------------------------------- + +// GentxReturn is the one file a validator sends back. +type GentxReturn struct { + Version int `json:"version"` + ChainID string `json:"chain_id"` + KitID string `json:"kit_id"` + Moniker string `json:"moniker"` + Account string `json:"account"` + NodeID string `json:"node_id"` + Gentx json.RawMessage `json:"gentx"` + Offline Offline `json:"offline"` +} + +// SignGentx signs this machine's gentx against the kit's base genesis, in +// a container with no network, after checking the node home is the one +// the kit lists. +func SignGentx(c *toolkit.Context, k *Kit, kitID string, base []byte, spec *netspec.Spec, homeDir, moniker string, kr KeyringOpts, allowOnline bool) (*GentxReturn, error) { + h, err := c.Host() + if err != nil { + return nil, err + } + off, err := offlineGate(c, h, allowOnline) + if err != nil { + return nil, err + } + var me *KitValidator + for i := range k.Validators { + if k.Validators[i].Moniker == moniker { + me = &k.Validators[i] + } + } + if me == nil { + return nil, fmt.Errorf("%s isn't a validator in this kit", moniker) + } + abs, user, hasKey, err := prepareHome(c, h, homeDir) + if err != nil { + return nil, err + } + if !hasKey { + return nil, fmt.Errorf("%s has no node keys — this must be the home genesis join made", abs) + } + ev := evmdOffline(user, abs, k.Image) + id, _, _ := run(c, h, ev+" comet show-node-id --home "+containerHome+" 2>/dev/null | tail -1", time.Minute) + if strings.TrimSpace(id) != me.NodeID { + return nil, fmt.Errorf("this home's node id %s isn't %s's in the kit (%s) — wrong home?", strings.TrimSpace(id), moniker, me.NodeID) + } + n := &GenesisNode{Ctx: c, Home: abs, User: user, Moniker: moniker, Account: me.Account, NodeID: me.NodeID, Image: k.Image} + g, err := gentxIn(n, ev, spec, base, kr, me.Address) + if err != nil { + return nil, err + } + return &GentxReturn{Version: FormatVersion, ChainID: k.ChainID, KitID: kitID, Moniker: moniker, Account: me.Account, NodeID: me.NodeID, Gentx: json.RawMessage(strings.TrimSpace(string(g))), Offline: off}, nil +} + +// --- collect --------------------------------------------------------------- + +// gentxDoc is the part of a gentx collect checks. +type gentxDoc struct { + Body struct { + Messages []struct { + Type string `json:"@type"` + Description struct { + Moniker string `json:"moniker"` + } `json:"description"` + Delegator string `json:"delegator_address"` + Validator string `json:"validator_address"` + Pubkey struct { + Key string `json:"key"` + } `json:"pubkey"` + Value struct { + Denom string `json:"denom"` + Amount string `json:"amount"` + } `json:"value"` + } `json:"messages"` + Memo string `json:"memo"` + } `json:"body"` + Signatures []string `json:"signatures"` +} + +// CheckReturn verifies a returned gentx against the kit before anything +// is built from it. The signature itself is checked by validate-genesis. +func CheckReturn(k *Kit, kitID string, r *GentxReturn) error { + if r.Version != FormatVersion { + return fmt.Errorf("%s: format version %d", r.Moniker, r.Version) + } + if r.ChainID != k.ChainID || r.KitID != kitID { + return fmt.Errorf("%s: signed against a different kit (chain %s, kit %.12s…, this kit %.12s…) — send them the current kit", r.Moniker, r.ChainID, r.KitID, kitID) + } + var me *KitValidator + for i := range k.Validators { + if k.Validators[i].Moniker == r.Moniker { + me = &k.Validators[i] + } + } + if me == nil { + return fmt.Errorf("%s isn't in the kit", r.Moniker) + } + var g gentxDoc + if err := json.Unmarshal(r.Gentx, &g); err != nil || len(g.Body.Messages) != 1 { + return fmt.Errorf("%s: the gentx isn't one create-validator message", r.Moniker) + } + m := g.Body.Messages[0] + valoper, _ := keys.ValAddress(me.Account) + switch { + case m.Type != "/cosmos.staking.v1beta1.MsgCreateValidator": + return fmt.Errorf("%s: gentx message is %s", r.Moniker, m.Type) + case m.Description.Moniker != r.Moniker: + return fmt.Errorf("%s: gentx moniker is %q", r.Moniker, m.Description.Moniker) + case m.Validator != valoper || (m.Delegator != "" && m.Delegator != me.Account): + return fmt.Errorf("%s: gentx is for %s, the kit funds %s", r.Moniker, m.Validator, valoper) + case m.Value.Denom != k.Denom || m.Value.Amount != k.Stake: + return fmt.Errorf("%s: stakes %s%s, the agreed amount is %s%s", r.Moniker, m.Value.Amount, m.Value.Denom, k.Stake, k.Denom) + case me.ConsKey != "" && m.Pubkey.Key != me.ConsKey: + return fmt.Errorf("%s: gentx consensus key isn't the one in the join file — a different home signed it", r.Moniker) + case !strings.HasPrefix(g.Body.Memo, me.NodeID+"@"): + return fmt.Errorf("%s: gentx memo %q isn't this node (%s)", r.Moniker, g.Body.Memo, me.NodeID) + case len(g.Signatures) != 1 || g.Signatures[0] == "": + return fmt.Errorf("%s: gentx isn't signed", r.Moniker) + } + return nil +} + +// CollectAt assembles and strictly validates the final genesis on the +// builder's host from the kit's base genesis and the gentxs. +func CollectAt(c *toolkit.Context, spec *netspec.Spec, base []byte, gentxs map[string][]byte) ([]byte, error) { + h, err := c.Host() + if err != nil { + return nil, err + } + dir, _, _, err := prepareHome(c, h, "~/.cometcli-build/genesis-"+spec.Chain.ID) + if err != nil { + return nil, err + } + if err := h.WriteFile(c, path.Join(dir, "config", "genesis.json"), base, 0o644); err != nil { + return nil, fmt.Errorf("the builder home on %s has no config dir — run genesis base first: %w", c.Profile.Name, err) + } + return Collect(c, spec, gentxs) +} + +// --- packages -------------------------------------------------------------- + +// Package is what a validator installs: the final genesis and its peers. +type Package struct { + Version int `json:"version"` + ChainID string `json:"chain_id"` + Moniker string `json:"moniker"` + NodeID string `json:"node_id"` + Genesis string `json:"genesis_sha256"` + Peers []string `json:"peers"` +} + +// WritePackages writes one package per validator under dir: the final +// genesis, the spec, and every other validator as a peer. +func WritePackages(dir string, k *Kit, final []byte, specPath string) (map[string]string, error) { + sum := sha256.Sum256(final) + spec, err := os.ReadFile(specPath) + if err != nil { + return nil, err + } + out := map[string]string{} + for _, v := range k.Validators { + var peers []string + for _, o := range k.Validators { + if o.Moniker != v.Moniker && o.Address != "" { + peers = append(peers, o.NodeID+"@"+o.Address) + } + } + pd := filepath.Join(dir, v.Moniker) + if err := os.MkdirAll(pd, 0o755); err != nil { + return nil, err + } + pk := Package{Version: FormatVersion, ChainID: k.ChainID, Moniker: v.Moniker, NodeID: v.NodeID, Genesis: hex.EncodeToString(sum[:]), Peers: peers} + pj, _ := json.MarshalIndent(pk, "", " ") + for n, b := range map[string][]byte{"genesis.json": final, "spec.yaml": spec, "package.json": pj} { + if err := os.WriteFile(filepath.Join(pd, n), b, 0o644); err != nil { + return nil, err + } + } + if _, err := writeManifest(pd, []string{"genesis.json", "spec.yaml", "package.json"}); err != nil { + return nil, err + } + out[v.Moniker] = pd + } + return out, nil +} + +// ReadPackage verifies a package. +func ReadPackage(dir string) (*Package, []byte, string, error) { + if _, err := verifyManifest(dir); err != nil { + return nil, nil, "", err + } + var p Package + raw, _ := os.ReadFile(filepath.Join(dir, "package.json")) + if err := json.Unmarshal(raw, &p); err != nil || p.Version != FormatVersion { + return nil, nil, "", fmt.Errorf("package.json: unreadable or format version %d", p.Version) + } + gen, _ := os.ReadFile(filepath.Join(dir, "genesis.json")) + sum := sha256.Sum256(gen) + if hex.EncodeToString(sum[:]) != p.Genesis { + return nil, nil, "", fmt.Errorf("genesis.json isn't the one the package names") + } + return &p, gen, filepath.Join(dir, "spec.yaml"), nil +} + +// InstallPackage puts the final genesis into the validator's home (made +// by genesis join) and starts the node with the package's peers. +func InstallPackage(c *toolkit.Context, p *Package, gen []byte, specPath, homeDir string, portOffset int, extIP string) (*toolkit.Result, error) { + h, err := c.Host() + if err != nil { + return nil, err + } + abs, user, hasKey, err := prepareHome(c, h, homeDir) + if err != nil { + return nil, err + } + if !hasKey { + return nil, fmt.Errorf("%s has no node keys — install into the home genesis join made", abs) + } + spec, _, err := netspec.Load(specPath) + if err != nil { + return nil, err + } + id, _, _ := run(c, h, evmdOffline(user, abs, ImageFor(spec))+" comet show-node-id --home "+containerHome+" 2>/dev/null | tail -1", time.Minute) + if strings.TrimSpace(id) != p.NodeID { + return nil, fmt.Errorf("this home is node %s, the package is for %s (%s) — wrong home or wrong package", strings.TrimSpace(id), p.Moniker, p.NodeID) + } + if err := h.WriteFile(c, path.Join(abs, "config", "genesis.json"), gen, 0o644); err != nil { + return nil, err + } + args := toolkit.Args{"role": "validator", "network": specPath, "home": abs, "moniker": p.Moniker, "reconfigure": true, + "peers": strings.Join(p.Peers, ","), "port_offset": float64(portOffset)} + if extIP != "" { + args["external_ip"] = extIP + } + return (provisionTool{}).Run(c, args) +} + +// --- removal --------------------------------------------------------------- + +// RemoveNode stops a node and deletes its home — only when the home's +// genesis is the named chain's, so another network's data is never +// touched. It returns how much space was freed. +func RemoveNode(c *toolkit.Context, chainID, homeDir, unit string) (string, error) { + h, err := c.Host() + if err != nil { + return "", err + } + out, _, _ := run(c, h, fmt.Sprintf(`h=%s; case "$h" in "~"|"~/"*) h="$HOME${h#\~}";; esac; test -d "$h" || { echo none; exit 0; } +c=$( { cat "$h/config/genesis.json" 2>/dev/null || sudo -n cat "$h/config/genesis.json" 2>/dev/null; } | grep -m1 -o '"chain_id": *"[^"]*"' | sed 's/.*"\([^"]*\)"$/\1/'); echo "$h ${c:--} $( { du -sh "$h" 2>/dev/null || sudo -n du -sh "$h"; } | cut -f1)"`, shellHome(homeDir)), time.Minute) + f := strings.Fields(out) + if len(f) == 1 && f[0] == "none" { + return "nothing at " + homeDir, nil + } + if len(f) < 3 { + return "", fmt.Errorf("reading %s: %s", homeDir, out) + } + abs, chain, size := f[0], f[1], f[2] + if chain != chainID { + return "", fmt.Errorf("%s holds chain %q, not %s — not removed", abs, chain, chainID) + } + if abs == "/" || strings.Count(strings.Trim(abs, "/"), "/") < 1 { + return "", fmt.Errorf("refusing to remove %s", abs) + } + script := fmt.Sprintf(`cd %[1]s 2>/dev/null && { docker compose -f docker-compose.yml down -t 20 >/dev/null 2>&1 || true; } +docker rm -f %[2]s >/dev/null 2>&1 || true +# a container that ran as root leaves root-owned files: sudo, without a password, for those +{ rm -rf %[1]s %[1]s.operator.mnemonic 2>/dev/null || sudo -n rm -rf %[1]s %[1]s.operator.mnemonic; } && ! test -e %[1]s && echo removed`, common.ShellQ(abs), common.ShellQ(unit)) + if out, _, err := run(c, h, script, 3*time.Minute); err != nil || !strings.Contains(out, "removed") { + return "", fmt.Errorf("removing %s: %v %s", abs, err, out) + } + return fmt.Sprintf("removed %s (%s, chain %s)", abs, size, chain), nil +} + +// RemoveBuilder deletes the chain's genesis builder home on a host. +func RemoveBuilder(c *toolkit.Context, chainID string) string { + h, err := c.Host() + if err != nil { + return "" + } + dir := "~/.cometcli-build/genesis-" + chainID + out, _, _ := run(c, h, fmt.Sprintf(`d="$HOME/.cometcli-build/genesis-%s"; test -d "$d" && { s=$(du -sh "$d" | cut -f1); rm -rf "$d" && echo "removed %s ($s)"; } || true`, chainID, dir), time.Minute) + return strings.TrimSpace(out) +} + +// OperatorKey creates the validator's operator key in its running node's +// own keyring (a validator joining after genesis). The mnemonic goes to a +// 0600 file on the node's host — never back over the connection — and the +// profile is set to sign with it there. +func OperatorKey(c *toolkit.Context, name string, kr KeyringOpts, mnemonicTo string) (string, string, error) { + p := c.Profile + if p.Service.Type != "docker" || p.Service.Unit == "" || p.Home == "" { + return "", "", fmt.Errorf("%s isn't a provisioned docker node (node provision first)", p.Name) + } + h, err := c.Host() + if err != nil { + return "", "", err + } + if mnemonicTo == "" { + mnemonicTo = p.Home + ".operator.mnemonic" + } + ev := "docker exec -i " + common.ShellQ(p.Service.Unit) + " evmd" + if out, _, _ := run(c, h, fmt.Sprintf("%s keys show %s --keyring-backend %s --home %s -a 2>/dev/null "$out" 2>/dev/null +m=%[5]s; case "$m" in "~/"*) m="$HOME${m#\~}";; esac +test -e "$m" && { echo "refusing to overwrite $m" >&2; exit 5; } +sed -n 's/.*"mnemonic":"\([^"]*\)".*/\1/p' "$out" > "$m"; chmod 600 "$m"; test -s "$m" +sed -n 's/.*"address":"\([^"]*\)".*/\1/p' "$out"`, ev, common.ShellQ(name), kr.Backend, containerHome, shellHome(mnemonicTo)) + res, err := host.ExecIn(c, h, script, kr.stdin(2), 1<<16) + if err != nil || res.Code != 0 { + return "", "", fmt.Errorf("creating the operator key failed (exit %d): %s", res.Code, strings.TrimSpace(res.Output)) + } + acc := strings.TrimSpace(lastLine(res.Output)) + if !strings.Contains(acc, "1") { + return "", "", fmt.Errorf("unexpected operator address %q", acc) + } + np := *p + if np.Metadata == nil { + np.Metadata = map[string]string{} + } + np.Metadata["account"] = acc + np.Signer.Mode, np.Signer.ContainerKey, np.Signer.ContainerKeyring, np.Signer.ContainerHome = "container", name, kr.Backend, containerHome + if c.Cfg != nil { + if err := c.Cfg.UpsertProfile(&np); err != nil { + return "", "", err + } + *p = np + } + return acc, mnemonicTo, nil +} diff --git a/internal/tools/networktool/workflowb_test.go b/internal/tools/networktool/workflowb_test.go new file mode 100644 index 0000000..d73ade6 --- /dev/null +++ b/internal/tools/networktool/workflowb_test.go @@ -0,0 +1,156 @@ +package networktool + +import ( + "encoding/json" + "os" + "path/filepath" + "strings" + "testing" + + "github.com/abhijitkrm/cometcli/internal/keys" + "github.com/abhijitkrm/cometcli/internal/netspec" +) + +const testSpec = `chain: + id: wb-1 + denom: adex + bech32_prefix: cosmos +image: + name: primium-evm + tag: v0.7.2 + naming: primium-{tag} +genesis: + initial_stake: '1000' + initial_balance: '2000' +` + +const ( + accA = "cosmos1gk5aka9mc8rw2uxm3xmzrtne526vj9t2z8rzay" + idA = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa" + idB = "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb" +) + +func kitFixture(t *testing.T) (dir string, k *Kit, id string) { + t.Helper() + d := t.TempDir() + sp := filepath.Join(d, "spec-src.yaml") + _ = os.WriteFile(sp, []byte(testSpec), 0o644) + k = &Kit{Version: FormatVersion, ChainID: "wb-1", Denom: "adex", Stake: "1000", Image: "primium-v0.7.2", + Validators: []KitValidator{ + {Moniker: "val-a", Account: accA, NodeID: idA, ConsKey: "Q09OU0E=", Address: "10.0.0.1:26656"}, + {Moniker: "val-b", Account: "cosmos1xtk0ehth60ls99rvnayxkwhqcsv608md0px450", NodeID: idB, Address: "10.0.0.2:26756"}, + }} + dir = filepath.Join(d, "kit") + id, err := WriteKit(dir, k, []byte(`{"chain_id":"wb-1"}`), sp) + if err != nil { + t.Fatal(err) + } + return dir, k, id +} + +func TestKitRoundTripAndTamper(t *testing.T) { + dir, _, id := kitFixture(t) + k, base, spec, id2, err := ReadKit(dir) + if err != nil || id2 != id || k.ChainID != "wb-1" || spec.Chain.ID != "wb-1" || !strings.Contains(string(base), "wb-1") { + t.Fatalf("read kit: %v %v", err, k) + } + // a changed base genesis is caught before anything uses it + _ = os.WriteFile(filepath.Join(dir, "base-genesis.json"), []byte(`{"chain_id":"wb-1","x":1}`), 0o644) + if _, _, _, _, err := ReadKit(dir); err == nil || !strings.Contains(err.Error(), "checksum") { + t.Fatalf("tampered kit accepted: %v", err) + } +} + +func gentxFor(t *testing.T, moniker, account, consKey, stake, memo string, signed bool) json.RawMessage { + t.Helper() + valoper, err := keys.ValAddress(account) + if err != nil { + t.Fatal(err) + } + sigs := []string{} + if signed { + sigs = []string{"c2ln"} + } + g := map[string]any{"body": map[string]any{"memo": memo, "messages": []any{map[string]any{ + "@type": "/cosmos.staking.v1beta1.MsgCreateValidator", "description": map[string]any{"moniker": moniker}, + "delegator_address": "", "validator_address": valoper, "pubkey": map[string]any{"key": consKey}, + "value": map[string]any{"denom": "adex", "amount": stake}}}}, "signatures": sigs} + raw, _ := json.Marshal(g) + return raw +} + +func TestCollectChecksEveryGentx(t *testing.T) { + _, k, id := kitFixture(t) + good := func() *GentxReturn { + return &GentxReturn{Version: FormatVersion, ChainID: "wb-1", KitID: id, Moniker: "val-a", Account: accA, NodeID: idA, + Gentx: gentxFor(t, "val-a", accA, "Q09OU0E=", "1000", idA+"@10.0.0.1:26656", true)} + } + if err := CheckReturn(k, id, good()); err != nil { + t.Fatalf("a good gentx was rejected: %v", err) + } + for name, mutate := range map[string]func(r *GentxReturn){ + "other kit": func(r *GentxReturn) { r.KitID = "deadbeef" }, + "not in kit": func(r *GentxReturn) { r.Moniker = "val-z" }, + "wrong stake": func(r *GentxReturn) { r.Gentx = gentxFor(t, "val-a", accA, "Q09OU0E=", "999999", idA+"@x:1", true) }, + "other account": func(r *GentxReturn) { + r.Gentx = gentxFor(t, "val-a", k.Validators[1].Account, "Q09OU0E=", "1000", idA+"@x:1", true) + }, + "other cons key": func(r *GentxReturn) { r.Gentx = gentxFor(t, "val-a", accA, "T1RIRVI=", "1000", idA+"@x:1", true) }, + "other node": func(r *GentxReturn) { r.Gentx = gentxFor(t, "val-a", accA, "Q09OU0E=", "1000", idB+"@x:1", true) }, + "moniker swapped": func(r *GentxReturn) { r.Gentx = gentxFor(t, "val-b", accA, "Q09OU0E=", "1000", idA+"@x:1", true) }, + "unsigned": func(r *GentxReturn) { r.Gentx = gentxFor(t, "val-a", accA, "Q09OU0E=", "1000", idA+"@x:1", false) }, + "garbage": func(r *GentxReturn) { r.Gentx = json.RawMessage(`{"body":{}}`) }, + } { + r := good() + mutate(r) + if err := CheckReturn(k, id, r); err == nil { + t.Errorf("%s: accepted", name) + } + } +} + +func TestReadJoinValidates(t *testing.T) { + d := t.TempDir() + spec := &netspec.Spec{Chain: netspec.Chain{ID: "wb-1", Bech32Prefix: "cosmos"}} + write := func(j Join) string { + p := filepath.Join(d, "j.json") + raw, _ := json.Marshal(j) + _ = os.WriteFile(p, raw, 0o644) + return p + } + ok := Join{Version: FormatVersion, ChainID: "wb-1", Moniker: "val-a", Account: accA, NodeID: idA, ConsKey: "Q09OU0E=", IP: "1.2.3.4", P2PPort: 26656} + if _, err := ReadJoin(write(ok), spec); err != nil { + t.Fatal(err) + } + for name, j := range map[string]Join{ + "chain": func() Join { x := ok; x.ChainID = "other-1"; return x }(), + "moniker": func() Join { x := ok; x.Moniker = "bad name!"; return x }(), + "account": func() Join { x := ok; x.Account = "osmo1abc"; return x }(), + "node id": func() Join { x := ok; x.NodeID = "xyz"; return x }(), + "conskey": func() Join { x := ok; x.ConsKey = ""; return x }(), + } { + if _, err := ReadJoin(write(j), spec); err == nil { + t.Errorf("join with a bad %s accepted", name) + } + } +} + +func TestPackagesPeerWithEveryoneElse(t *testing.T) { + dir, k, _ := kitFixture(t) + out := filepath.Join(t.TempDir(), "pkgs") + pk, err := WritePackages(out, k, []byte(`{"chain_id":"wb-1","final":true}`), filepath.Join(dir, "spec.yaml")) + if err != nil { + t.Fatal(err) + } + p, gen, _, err := ReadPackage(pk["val-a"]) + if err != nil || !strings.Contains(string(gen), "final") { + t.Fatal(err) + } + if len(p.Peers) != 1 || p.Peers[0] != idB+"@10.0.0.2:26756" { + t.Fatalf("val-a peers = %v", p.Peers) + } + _ = os.WriteFile(filepath.Join(pk["val-b"], "genesis.json"), []byte(`{}`), 0o644) + if _, _, _, err := ReadPackage(pk["val-b"]); err == nil { + t.Fatal("a tampered package was accepted") + } +} diff --git a/internal/tools/node/edit.go b/internal/tools/node/edit.go index 114de6f..66d3652 100644 --- a/internal/tools/node/edit.go +++ b/internal/tools/node/edit.go @@ -3,6 +3,7 @@ package node import ( "errors" "fmt" + "path" "regexp" "strconv" "strings" @@ -118,15 +119,25 @@ func (recreateTool) Run(c *toolkit.Context, _ toolkit.Args) (*toolkit.Result, er } out, code, err := h.Run(c, "docker inspect -f '{{index .Config.Labels \"com.docker.compose.project.config_files\"}}|{{index .Config.Labels \"com.docker.compose.project.working_dir\"}}|{{index .Config.Labels \"com.docker.compose.service\"}}' "+common.ShellQ(p.Service.Unit)) parts := strings.SplitN(strings.TrimSpace(out), "|", 3) - if err != nil || code != 0 || len(parts) < 3 || parts[0] == "" || parts[0] == "" { + var file, dir, svc string + switch { + case err == nil && code == 0 && len(parts) == 3 && parts[0] != "" && parts[0] != "": + file, dir, svc = strings.Split(parts[0], ",")[0], parts[1], parts[2] + case strings.Contains(strings.ToLower(out), "no such") && p.Home != "": + // the container is gone: the compose file cometcli wrote in the + // node's home creates it again, on the same data dir + dir, file = p.Home, path.Join(p.Home, "docker-compose.yml") + if _, c2, _ := h.Run(c, "test -f "+common.ShellQ(file)); c2 != 0 { + return nil, fmt.Errorf("%s is gone and %s has no docker-compose.yml to create it from — node.provision reconfigure=true renders one", p.Service.Unit, p.Home) + } + default: return nil, fmt.Errorf("%s wasn't created by docker compose — no compose file to recreate it from", p.Service.Unit) } - file, dir, svc := strings.Split(parts[0], ",")[0], parts[1], parts[2] if err := c.Approve(fmt.Sprintf("recreate %s (docker compose up -d --force-recreate %s, %s)", p.Service.Unit, svc, file), toolkit.TierLocalChange, map[string]any{"compose": file, "service": svc}); err != nil { return nil, err } - res, err := host.Exec(c, h, fmt.Sprintf("cd %s && docker compose -f %s up -d --force-recreate %s 2>&1 | tail -3", common.ShellQ(dir), common.ShellQ(file), common.ShellQ(svc)), 64<<10) + res, err := host.Exec(c, h, fmt.Sprintf("cd %s && docker compose -f %s up -d --force-recreate %s 2>&1 | tail -3", common.ShellQ(dir), common.ShellQ(file), svcArg(svc)), 64<<10) if err != nil || res.Code != 0 { return nil, fmt.Errorf("recreate failed: %v %s", err, strings.TrimSpace(res.Output)) } @@ -240,3 +251,10 @@ func (repairConfigTool) Run(c *toolkit.Context, a toolkit.Args) (*toolkit.Result } return &toolkit.Result{Text: text, Data: map[string]any{"changed": true, "disabled": disabled}}, nil } + +func svcArg(svc string) string { + if svc == "" { + return "" // every service in the file + } + return common.ShellQ(svc) +} diff --git a/internal/tools/sectool/sec.go b/internal/tools/sectool/sec.go index 1f0a037..3f666e8 100644 --- a/internal/tools/sectool/sec.go +++ b/internal/tools/sectool/sec.go @@ -5,9 +5,13 @@ package sectool import ( "encoding/json" "fmt" + "strconv" "strings" + "github.com/BurntSushi/toml" + "github.com/abhijitkrm/cometcli/internal/toolkit" + "github.com/abhijitkrm/cometcli/internal/tools/common" ) // Register adds all sec.* tools. @@ -261,13 +265,55 @@ func (doubleSignTool) Run(c *toolkit.Context, _ toolkit.Args) (*toolkit.Result, var b strings.Builder fmt.Fprintf(&b, "last signed: height=%s round=%d step=%d\n", st.Height, st.Round, st.Step) fmt.Fprintf(&b, "signature present: %v\n", st.Signature != "") - // Check for a second signer risk: is the service running while we might migrate? - if c.Profile.Service.Unit != "" { - out, code, _ := h.Run(c, fmt.Sprintf("systemctl is-active %s 2>/dev/null || echo unknown", c.Profile.Service.Unit)) - c.LogShell("is-active "+c.Profile.Service.Unit, code) - fmt.Fprintf(&b, "service %s: %s\n", c.Profile.Service.Unit, strings.TrimSpace(out)) + data := map[string]any{"height": st.Height, "round": st.Round, "step": st.Step} + // is this signer running now? (a second copy of the key must not be) + if unit := c.Profile.Service.Unit; unit != "" { + cmd := "systemctl is-active " + common.ShellQ(unit) + " 2>/dev/null || echo unknown" + if c.Profile.Service.Type == "docker" { + cmd = "docker inspect -f '{{.State.Status}}' " + common.ShellQ(unit) + " 2>/dev/null || echo 'no container'" + } + out, code, _ := h.Run(c, cmd) + c.LogShell("service state "+unit, code) + fmt.Fprintf(&b, "service %s (%s): %s\n", unit, orDefault(c.Profile.Service.Type, "systemd"), strings.TrimSpace(out)) + } + // the signing state must not be ahead of the chain: a key restored with + // an older state (or a copy started elsewhere) signs twice + if cc, err := c.Comet(); err == nil { + if ns, err := cc.Status(c); err == nil { + h0, _ := strconv.ParseInt(st.Height, 10, 64) + top := ns.SyncInfo.LatestBlockHeight + switch { + case h0 > top+1: + fmt.Fprintf(&b, "✗ signing state is %d blocks ahead of the node (%d) — don't start another signer; let the node catch up\n", h0-top, top) + default: + fmt.Fprintf(&b, "✓ signing state matches the node (height %d)\n", top) + } + data["node_height"] = top + } + } + // CometBFT's own guard: refuse to sign if the key signed in the last N blocks + if raw, err := h.ReadFile(c, c.Profile.Home+"/config/config.toml"); err == nil { + var cfg struct { + Consensus struct { + DoubleSignCheckHeight int64 `toml:"double_sign_check_height"` + } `toml:"consensus"` + } + if _, err := toml.Decode(string(raw), &cfg); err == nil { + n := cfg.Consensus.DoubleSignCheckHeight + data["double_sign_check_height"] = n + if n == 0 { + b.WriteString("! double_sign_check_height = 0 — the node starts signing without checking that its key signed nothing recently (10 or so is a safety net when moving a validator)\n") + } else { + fmt.Fprintf(&b, "✓ double_sign_check_height = %d\n", n) + } + } + } + return &toolkit.Result{Text: b.String(), Data: data}, nil +} + +func orDefault(s, d string) string { + if s == "" { + return d } - return &toolkit.Result{Text: b.String(), Data: map[string]any{ - "height": st.Height, "round": st.Round, "step": st.Step, - }}, nil + return s } diff --git a/internal/tools/shell/classify.go b/internal/tools/shell/classify.go index b6071ce..4ceadd9 100644 --- a/internal/tools/shell/classify.go +++ b/internal/tools/shell/classify.go @@ -41,9 +41,32 @@ var ( copiers = map[string]bool{"cp": true, "install": true, "ln": true, "rsync": true, "scp": true} interpreters = map[string]bool{"python": true, "python3": true, "python2": true, "perl": true, "node": true, "ruby": true, "php": true, "jq": false} // a node's data dir (or what's in it), by name or by the stores it holds - nodeDataRe = regexp.MustCompile(`(^|/)data/?(\*)?$|(^|/)data/(\*|application\.db|blockstore\.db|state\.db|tx_index\.db|evidence\.db|cs\.wal|snapshots)(/|$)|(application|blockstore|state|evidence)\.db/?$|(^|/)cs\.wal(/|$)`) + nodeDataRe = regexp.MustCompile(`(^|/)data/?(\*)?$|(^|/)data/(\*|[^/]*\.db|cs\.wal|snapshots)(/|$)|\.db/?$|(^|/)\*\.db$|(^|/)cs\.wal(/|$)|(^|/)upgrade-info\.json$|(^|/)wasm(/|$)`) ) +var ( + // a node home's config or data dir, as a copy operand + nodeDirRe = regexp.MustCompile(`(^|/)(config|data)/?\*?$`) + // a node's home, or its config/data dirs + nodeHomeRe = regexp.MustCompile(`(^|/)\.cometcli-nodes(/|$)|(^|/)\.[a-z0-9]+d(/|$)|(^|/)(config|data)/?$`) +) + +// tarExtracts: tar's mode is extract (x, --extract, --get). +func tarExtracts(args []string) bool { + for i, x := range args { + switch { + case x == "--extract" || x == "--get": + return true + case strings.HasPrefix(x, "--"): + case strings.HasPrefix(x, "-") || i == 0: + if strings.Contains(x, "x") { + return true + } + } + } + return false +} + func (a *acc) raise(t toolkit.Tier, reason string) { if t > a.v.Tier || a.v.Reason == "" && t == a.v.Tier && t >= toolkit.TierLocalChange { a.v.Tier, a.v.Reason = t, reason @@ -478,6 +501,39 @@ func (a *acc) simple(words []string) { a.forbid("overwriting priv_validator_state.json risks double-signing") } } + // a whole node config or data dir moved between homes carries the + // consensus key or the signing state with it, under no file name + if copiers[name] || (name == "docker" && len(args) > 0 && args[0] == "cp") { + ops := nonFlags(args) + if name == "docker" { + ops = nonFlags(args[1:]) + } + recursive := name == "rsync" + for _, x := range args { + if x == "--recursive" || x == "--archive" || strings.HasPrefix(x, "-") && !strings.HasPrefix(x, "--") && strings.ContainsAny(x, "rRa") { + recursive = true + } + } + for i, x := range ops { + // from a config/data dir, or recursively into one (a single + // file such as genesis.json into config/ is fine) + if nodeDirRe.MatchString(x) && (i < len(ops)-1 || recursive) { + a.forbid("copying a node's whole " + path.Base(strings.TrimRight(x, "/*")) + " dir carries its consensus key or signing state (double-sign risk) — copy the files you need by name") + break + } + } + } + // unpacking an archive into a node's home can restore a key with an + // older signing state + if (name == "tar" && tarExtracts(args)) || name == "unzip" { + for i, x := range args { + if (x == "-C" || x == "--directory" || x == "-d") && i+1 < len(args) && nodeHomeRe.MatchString(args[i+1]) || + strings.HasPrefix(x, "--directory=") && nodeHomeRe.MatchString(strings.TrimPrefix(x, "--directory=")) { + a.forbid("unpacking an archive into a node's home can restore its consensus key with an older signing state (double-sign risk) — the operator restores backups") + break + } + } + } if name == "dd" && containsWord(args, "of=") && containsWord(args, "priv_validator_state.json") { a.forbid("overwriting priv_validator_state.json risks double-signing") } @@ -491,6 +547,9 @@ func (a *acc) simple(words []string) { } } } + if strings.Contains(strings.Join(words, " "), "unsafe-skip-upgrades") { + a.forbid("--unsafe-skip-upgrades makes a node skip an upgrade the network applied — its state forks from the chain's; the operator decides that") + } if interpreters[name] && a.cmdText != "" && strings.Contains(a.cmdText, "priv_validator_state") { a.forbid("a script that touches priv_validator_state.json risks double-signing — the operator handles that file") } diff --git a/internal/tools/shell/classify_test.go b/internal/tools/shell/classify_test.go index bbdaf26..8878b59 100644 --- a/internal/tools/shell/classify_test.go +++ b/internal/tools/shell/classify_test.go @@ -197,3 +197,58 @@ func TestKeySeedIsProtected(t *testing.T) { } } } + +// What a model ran on the scratch-test archive with bash allowed: each one +// corrupts or forks a node's state and must be refused in every mode. +func TestStateDestroyingCommandsFromTheScratchTest(t *testing.T) { + for _, cmd := range []string{ + "cd /home/ubuntu/.cometcli-nodes/v6-arch/data && rm -rf *.db addrbook.json", + "cd data && rm -rf *.db", + "rm -rf ~/.evmd/data/*.db", + "rm /home/ubuntu/.cometcli-nodes/v6-arch/data/upgrade-info.json", + "rm -f data/upgrade-info.json", + "docker run -d --name x primium-v0.7.3 start --unsafe-skip-upgrades 9334", + "evmd start --unsafe-skip-upgrades 9334", + } { + if v := Classify(cmd, Opts{Binary: "evmd"}); v.Forbidden == "" { + t.Errorf("%q must be refused (got %s)", cmd, v.Reason) + } + } + for _, cmd := range []string{"rm -f /tmp/notes.db.bak.txt", "ls data/*.db", "du -sh data/application.db"} { + if v := Classify(cmd, Opts{Binary: "evmd"}); v.Forbidden != "" { + t.Errorf("%q should be allowed: %s", cmd, v.Forbidden) + } + } +} + +// A consensus key moved without its file name: a whole config dir, or a +// backup unpacked into a node's home. Both are refused in every mode; +// single files into config/ and creating backups are not. +func TestKeyMovedWithoutItsName(t *testing.T) { + for _, cmd := range []string{ + "rsync -a val3:/home/ubuntu/.cometcli-nodes/v6-val3/config/ ~/.cometcli-nodes/v6-arch/config/", + "cp -r /backup/config ~/.evmd/", + "cp -a old/data ~/.evmd/", + "scp -r host:~/.evmd/config .", + "docker cp evmd-validator:/root/.evmd/config ./", + "cp -r /tmp/restore/* ~/.evmd/config/", + "tar xzf val3-backup.tgz -C ~/.cometcli-nodes/v6-arch", + "tar -xf b.tar --directory=/home/u/.evmd", + "unzip backup.zip -d ~/.evmd/config", + } { + if v := Classify(cmd, Opts{}); v.Forbidden == "" { + t.Errorf("not refused: %s", cmd) + } + } + for _, cmd := range []string{ + "cp genesis.json ~/.evmd/config/", + "cp config/app.toml config/app.toml.bak", + "tar czf backup.tgz -C ~/.evmd config", + "tar xzf go1.23.linux-amd64.tar.gz -C /usr/local", + "cp -r ~/.config/foo /tmp/", + } { + if v := Classify(cmd, Opts{}); v.Forbidden != "" { + t.Errorf("refused: %s (%s)", cmd, v.Forbidden) + } + } +} diff --git a/internal/tools/triage/collect.go b/internal/tools/triage/collect.go index 27be533..6adcfc7 100644 --- a/internal/tools/triage/collect.go +++ b/internal/tools/triage/collect.go @@ -9,6 +9,7 @@ import ( "encoding/json" "fmt" "math/big" + "path" "regexp" "sort" "strconv" @@ -214,6 +215,26 @@ func collectComet(c *toolkit.Context, r *Report, set func(string, any)) error { if ni, err := cc.NetInfo(c); err == nil { set("node.peers", float64(ni.NPeers)) } + // no new block for a while: who isn't voting? Under 2/3 of the power + // voting is a chain halt, whoever's node this is + if !st.SyncInfo.CatchingUp && time.Since(st.SyncInfo.LatestBlockTime) > 30*time.Second { + if cs, err := cc.DumpConsensusState(c); err == nil { + if v := ConsensusVotes(cs.RoundState); v != nil { + set("consensus.voted_pct", v.VotedPct) + names, _ := common.MonikersByConsAddr(c) + var absent []string + for _, a := range v.Absent { + n := names[a.Addr] + if n == "" { + n = a.Addr + } + absent = append(absent, fmt.Sprintf("%s (%.1f%%)", n, a.Pct)) + } + set("consensus.absent", strings.Join(absent, ", ")) + set("consensus.absent_n", float64(len(absent))) + } + } + } // average block time over the last 100 blocks if h := st.SyncInfo.LatestBlockHeight; h > 101 { back := h - 100 @@ -226,6 +247,72 @@ func collectComet(c *toolkit.Context, r *Report, set func(string, any)) error { return nil } +// Absentee is a validator with no vote at the current height. +type Absentee struct { + Addr string // consensus address, upper-case hex + Pct float64 // share of the voting power +} + +// Votes is who has voted at the height a node is stuck on. +type Votes struct { + VotedPct float64 // power that prevoted or precommitted in any round + Absent []Absentee +} + +var bitArrayRe = regexp.MustCompile(`BA\{\d+:([x_]+)\}`) + +// ConsensusVotes reads a dump_consensus_state round state: a validator has +// voted if any round of the height has its prevote or precommit. +func ConsensusVotes(raw []byte) *Votes { + var rs struct { + Validators struct { + Validators []struct { + Address string `json:"address"` + VotingPower string `json:"voting_power"` + } `json:"validators"` + } `json:"validators"` + Votes []struct { + Prevotes string `json:"prevotes_bit_array"` + Precommits string `json:"precommits_bit_array"` + } `json:"votes"` + } + if json.Unmarshal(raw, &rs) != nil || len(rs.Validators.Validators) == 0 || len(rs.Votes) == 0 { + return nil + } + vals := rs.Validators.Validators + voted := make([]bool, len(vals)) + for _, r := range rs.Votes { + for _, ba := range []string{r.Prevotes, r.Precommits} { + if m := bitArrayRe.FindStringSubmatch(ba); m != nil { + for i, ch := range m[1] { + if ch == 'x' && i < len(voted) { + voted[i] = true + } + } + } + } + } + var total, in float64 + power := make([]float64, len(vals)) + for i, v := range vals { + power[i], _ = strconv.ParseFloat(v.VotingPower, 64) + total += power[i] + if voted[i] { + in += power[i] + } + } + if total == 0 { + return nil + } + out := &Votes{VotedPct: round2(in / total * 100)} + for i, v := range vals { + if !voted[i] { + out.Absent = append(out.Absent, Absentee{Addr: strings.ToUpper(v.Address), Pct: round2(power[i] / total * 100)}) + } + } + return out +} + func round2(f float64) float64 { return float64(int64(f*100+0.5)) / 100 } func collectChain(c *toolkit.Context, r *Report, set func(string, any)) error { @@ -447,6 +534,7 @@ func collectProcess(c *toolkit.Context, r *Report, set func(string, any)) error case noContainerRe.MatchString(why): set("proc.container_missing", true) set("proc.running", false) + composeImage(c, h, set) return nil } return fmt.Errorf("docker inspect %s failed: %v", svc.Unit, err) @@ -1090,6 +1178,33 @@ var ( noImageRe = regexp.MustCompile(`(?i)no such image|pull access denied|manifest unknown|repository does not exist`) ) +var ( + composeImageRe = regexp.MustCompile(`(?m)^\s*image:\s*["']?([^"'\s]+)`) + versionRe = regexp.MustCompile(`v\d+\.\d+\.\d+`) +) + +// composeImage reads, for a node whose container is gone, the image its +// compose file runs and whether that image is still on the host. +func composeImage(c *toolkit.Context, h host.Host, set func(string, any)) { + if c.Profile == nil || c.Profile.Home == "" { + return + } + raw, err := h.ReadFile(c, path.Join(c.Profile.Home, "docker-compose.yml")) + if err != nil { + return + } + m := composeImageRe.FindStringSubmatch(string(raw)) + if m == nil { + return + } + set("proc.image", m[1]) + _, present := runOK(c, h, "docker image inspect "+common.ShellQ(m[1])+" >/dev/null 2>&1 && echo ok") + set("proc.image_present", present) + if v := versionRe.FindString(m[1]); v != "" { + set("proc.image_version", v) + } +} + // dockerStartError reads why a stopped container didn't start: docker's // own error (ports, image) is in its state, not in the node's logs. func dockerStartError(c *toolkit.Context, h host.Host, unit string, set func(string, any)) { diff --git a/internal/tools/triage/consensus_test.go b/internal/tools/triage/consensus_test.go new file mode 100644 index 0000000..cd72925 --- /dev/null +++ b/internal/tools/triage/consensus_test.go @@ -0,0 +1,22 @@ +package triage + +import "testing" + +// The round state a node stuck at a height reports when the biggest +// validator is down: its bit is unset in every round. +func TestConsensusVotesNamesTheAbsent(t *testing.T) { + raw := []byte(`{"height":"73005","validators":{"validators":[ + {"address":"0fb5d14dca750c9e64b91bb747a437d49191c647","voting_power":"10000100"}, + {"address":"AAAA","voting_power":"10000000"}, + {"address":"BBBB","voting_power":"1980350"}]}, + "votes":[ + {"round":0,"prevotes_bit_array":"BA{3:_xx} 11980350/21980450 = 0.55","precommits_bit_array":"BA{3:___} 0/21980450 = 0.00"}, + {"round":1,"prevotes_bit_array":"BA{3:___} 0/21980450 = 0.00","precommits_bit_array":"BA{3:___} 0/21980450 = 0.00"}]}`) + v := ConsensusVotes(raw) + if v == nil || v.VotedPct != 54.5 || len(v.Absent) != 1 || v.Absent[0].Addr != "0FB5D14DCA750C9E64B91BB747A437D49191C647" || v.Absent[0].Pct != 45.5 { + t.Fatalf("got %+v", v) + } + if ConsensusVotes([]byte(`{"votes":[]}`)) != nil { + t.Fatal("no validators: no verdict") + } +} diff --git a/internal/tools/triage/fleet.go b/internal/tools/triage/fleet.go index aeb2747..3afe179 100644 --- a/internal/tools/triage/fleet.go +++ b/internal/tools/triage/fleet.go @@ -134,7 +134,15 @@ func FleetFindings(nodes []NodeReport) []string { } } if reachable >= 2 && stalled == reachable { - out = append(out, fmt.Sprintf("CHAIN HALT on %s: all %d reachable nodes are stalled near height %d — a chain-wide problem (upgrade height? consensus failure?), not one node", orNone(ch), reachable, int64(top))) + why := "a chain-wide problem (upgrade height? consensus failure?), not one node" + for _, n := range group { + if a, _ := n.Report.Signals["consensus.absent"].(string); a != "" { + pct, _ := f(n, "consensus.voted_pct") + why = fmt.Sprintf("only %.1f%% of the voting power is voting (2/3 needed) — not voting: %s", pct, a) + break + } + } + out = append(out, fmt.Sprintf("CHAIN HALT on %s: all %d reachable nodes are stalled near height %d — %s", orNone(ch), reachable, int64(top), why)) } for _, n := range group { if n.Report.Signals["node.reachable"] != true { diff --git a/internal/tools/triage/triage_test.go b/internal/tools/triage/triage_test.go index 30c7838..a048401 100644 --- a/internal/tools/triage/triage_test.go +++ b/internal/tools/triage/triage_test.go @@ -292,3 +292,17 @@ func TestMempoolMismatchByVersion(t *testing.T) { } } } + +// A memory limit far below what a node needs is the root cause even before +// the container has crashed often enough to look like a crash loop. +func TestTinyMemoryLimitIsTheRootBeforeRestartsPileUp(t *testing.T) { + base := kb.Load("", "") + hits := base.Match(kb.Signals{"proc.mem_limit_mb": 150.0, "proc.restarts": 1.0, "proc.uptime_s": 20.0, "proc.running": true}, "cosmos-evm") + if len(hits) == 0 || hits[0].Case.ID != "container-memory-limit" { + var ids []string + for _, h := range hits { + ids = append(ids, h.Case.ID) + } + t.Fatalf("top case %v, want container-memory-limit first", ids) + } +} diff --git a/internal/tools/upgradetool/flow.go b/internal/tools/upgradetool/flow.go index e722d3d..2d7c300 100644 --- a/internal/tools/upgradetool/flow.go +++ b/internal/tools/upgradetool/flow.go @@ -28,8 +28,11 @@ import ( "strings" "time" + upgradev1beta1 "cosmossdk.io/api/cosmos/upgrade/v1beta1" + "github.com/abhijitkrm/cometcli/internal/client/host" "github.com/abhijitkrm/cometcli/internal/config" + "github.com/abhijitkrm/cometcli/internal/netspec" "github.com/abhijitkrm/cometcli/internal/toolkit" "github.com/abhijitkrm/cometcli/internal/tools/common" ) @@ -284,8 +287,8 @@ func (buildTool) Desc() string { func (buildTool) Schema() map[string]any { return toolkit.ObjSchema(map[string]any{ "tag": toolkit.Str("release tag to build, e.g. v0.6.1"), - "image": toolkit.Str("image to produce; {tag} is the release tag, {version} the tag without v — e.g. primium-{tag} → primium-v0.7.2 (default: the profile's metadata.image, else the running image's name with the new tag)"), - "dockerfile": toolkit.Str("Dockerfile on the node (default: the one the last build used, Dockerfile.primium beside the compose project or under the home dir, else cometcli's built-in one; \"builtin\" forces that)"), + "image": toolkit.Str("image to produce; {tag} is the release tag, {version} the tag without v — e.g. mychain-{tag} → mychain-v0.7.2 (default: the profile's metadata.image, else the running image's name with the new tag)"), + "dockerfile": toolkit.Str("Dockerfile on the node (default: the one the last build used, a Dockerfile beside the node's compose project, else cometcli's built-in one; \"builtin\" forces that)"), "handler": toolkit.Str("forks only: add a no-op upgrade handler with this plan name (official releases ship their own — leave empty)"), "repo": toolkit.Str("github owner/repo (default: the running image's source)"), }, "tag") @@ -296,6 +299,7 @@ func (buildTool) Timeout(toolkit.Args) time.Duration { return 50 * time.Minute } var tagRe = regexp.MustCompile(`^[A-Za-z0-9._-]+$`) func (buildTool) Run(c *toolkit.Context, a toolkit.Args) (*toolkit.Result, error) { + asked := a.String("image", "") // before the spec's naming fills it in r, err := inspectRunning(c) fresh := false if err != nil { @@ -311,16 +315,35 @@ func (buildTool) Run(c *toolkit.Context, a toolkit.Args) (*toolkit.Result, error if repo := c.Profile.Metadata["repo"]; repo != "" { r.Repo = repo } + specDefaults(c.Profile, r, a) + } else if a.String("image", "") == "" && c.Profile.Metadata["image"] == "" { + // the spec's naming beats deriving from the running image: with + // mychain-{tag}, the version is in the name, not the tag + specImage(c.Profile, a) } tag := a.String("tag", "") handler := a.String("handler", "") + if handler == "" && tag != "" { + // a rebuild must reproduce the image the chain ran: if this tag was + // built with an added handler, add it again + handler = c.Profile.Metadata["handler."+tag] + if handler == "" && c.Profile.ChainID != "" { + if spec, _, err := netspec.Load(c.Profile.ChainID); err == nil { + handler = spec.HandlerFor(tag) + } + } + } + record, err := handlerRecord(c, asked, r.Image, tag, handler) + if err != nil { + return nil, err + } repo := a.String("repo", r.Repo) if !tagRe.MatchString(tag) || (handler != "" && !tagRe.MatchString(handler)) || repo == "" { return nil, fmt.Errorf("tag, handler and repo must be plain names (letters, digits, . _ -)") } image := ImageFor(c, a.String("image", ""), r.Image, tag) if fresh && strings.HasPrefix(image, ":") { - return nil, fmt.Errorf("%s has no node container yet to take the image name from — pass image, e.g. primium-{tag}", c.Profile.Name) + return nil, fmt.Errorf("%s has no node container yet to take the image name from — pass image, e.g. mychain-{tag}", c.Profile.Name) } if image == r.Image { return nil, fmt.Errorf("%s is the image running now — build a different tag", image) @@ -337,17 +360,9 @@ func (buildTool) Run(c *toolkit.Context, a toolkit.Args) (*toolkit.Result, error } } if dockerfile == "" && r.ComposeDir != "" { - for _, d := range []string{r.ComposeDir, path.Dir(r.ComposeDir)} { - p := path.Join(d, "Dockerfile.primium") - if _, code, _ := sh(c, "test -f "+common.ShellQ(p), time.Minute); code == 0 { - dockerfile = p - break - } - } - } - if dockerfile == "" { - // deployments differ per host: look under the user's home - out, _, _ := sh(c, "find \"$HOME\" -maxdepth 7 -name Dockerfile.primium -not -path '*/.cometcli-build/*' 2>/dev/null | head -10", 2*time.Minute) + // the Dockerfile beside the node's compose project (or one level up) + out, _, _ := sh(c, "for d in "+common.ShellQ(r.ComposeDir)+" "+common.ShellQ(path.Dir(r.ComposeDir))+ + "; do find \"$d\" -maxdepth 1 -type f -name 'Dockerfile*' ! -name '.cometcli.Dockerfile' 2>/dev/null; done | sort -u | head -10", time.Minute) var found []string for _, l := range strings.Split(strings.TrimSpace(out), "\n") { if l = strings.TrimSpace(l); l != "" { @@ -355,14 +370,16 @@ func (buildTool) Run(c *toolkit.Context, a toolkit.Args) (*toolkit.Result, error } } switch len(found) { + case 0: case 1: dockerfile = found[0] - case 0: - dockerfile = "builtin" default: - return nil, fmt.Errorf("several Dockerfile.primium on this host — pass dockerfile with the one that built %s (or \"builtin\" for cometcli's):\n %s", r.Image, strings.Join(found, "\n ")) + return nil, fmt.Errorf("several Dockerfiles beside %s — pass dockerfile with the one that built %s (or \"builtin\" for cometcli's):\n %s", r.ComposeDir, r.Image, strings.Join(found, "\n ")) } } + if dockerfile == "" { + dockerfile = "builtin" + } builtinNote := "" if dockerfile == "builtin" { // none on this host: cometcli's own, written beside its builds @@ -371,14 +388,14 @@ func (buildTool) Run(c *toolkit.Context, a toolkit.Args) (*toolkit.Result, error if err != nil { return nil, err } - dockerfile = path.Join(strings.TrimSpace(home), ".cometcli-build", "Dockerfile.primium") + dockerfile = path.Join(strings.TrimSpace(home), ".cometcli-build", "Dockerfile.evmd") if _, code, _ := sh(c, "mkdir -p "+common.ShellQ(path.Dir(dockerfile)), time.Minute); code != 0 { return nil, fmt.Errorf("can't create %s on %s", path.Dir(dockerfile), c.Profile.Name) } if err := h.WriteFile(c, dockerfile, []byte(builtinDockerfile), 0o644); err != nil { return nil, err } - why := "no Dockerfile.primium on this host" + why := "no Dockerfile beside the node's compose project" if fresh { why = "no node on this host yet" } @@ -412,7 +429,7 @@ if [ -n "$need" ] && [ -n "$have" ] && [ "$(printf '%%s\n%%s\n' "$have" "$need" echo "note: $(basename %s) builds with Go $have; $(git describe --tags 2>/dev/null) needs Go $need — built with golang:$need (your Dockerfile is unchanged)" fi `, common.ShellQ(dockerfile), common.ShellQ(dockerfile)) - fmt.Fprintf(&script, "docker build -q -f \"$df\" -t %s --label name=primium --label org.opencontainers.image.version=%s --label org.opencontainers.image.source=https://github.com/%s . >/dev/null\n", + fmt.Fprintf(&script, "docker build -q -f \"$df\" -t %s --label org.opencontainers.image.version=%s --label org.opencontainers.image.source=https://github.com/%s . >/dev/null\n", common.ShellQ(image), common.ShellQ(tag), repo) fmt.Fprintf(&script, "echo \"version: $(docker run --rm --entrypoint %s %s version 2>&1 | head -1)\"\n", r.Binary, common.ShellQ(image)) detail := map[string]any{"image": image, "tag": tag, "repo": repo, "dockerfile": dockerfile, "source": src} @@ -424,8 +441,9 @@ fi detail["builder"] = fmt.Sprintf("Go %s → %s (%s needs it; the build uses a copy, your Dockerfile is unchanged)", have, need, tag) } prompt := fmt.Sprintf("build %s on %s from %s@%s (takes several minutes; the node keeps running)", image, c.Profile.Name, repo, tag) - if why := breaking(c, repo, tag); why != "" { - // consensus-breaking: always asked, whatever the auto-approval + if why := breaking(c, repo, tag); why != "" && !fresh { + // a running node could be switched onto it too early: the warning + // matters, so it's always asked — a new host has nothing to protect detail["consensus"] = why + " — every validator must switch together at an upgrade height (gov proposal or halt-height); never restart a node onto it early" if err := toolkit.RequireApproval(c, "CONSENSUS-BREAKING release — "+prompt, toolkit.TierLocalChange, detail); err != nil { return nil, err @@ -451,18 +469,90 @@ fi fmt.Fprintf(&b, "\nhandler %s: in the new binary ✓", handler) data["handler"] = handler } - // remember the Dockerfile for the next build on this node - if c.Cfg != nil && c.Profile.Name != "" && c.Profile.Metadata["dockerfile"] != dockerfile { + // remember the Dockerfile, and any handler added to this tag, for the + // next build of it on this node (a rebuild after the image was lost + // must be the same binary, or the node can't replay the upgrade) + if c.Cfg != nil && c.Profile.Name != "" { _ = c.Cfg.UpdateProfile(c.Profile.Name, func(p *config.Profile) { if p.Metadata == nil { p.Metadata = map[string]string{} } p.Metadata["dockerfile"] = dockerfile + if record { + p.Metadata["handler."+tag] = handler + } }) + if record && c.Profile.ChainID != "" { + if spec, _, err := netspec.Load(c.Profile.ChainID); err == nil && spec.RecordHandler(tag, handler) { + _, _ = spec.Save() + } + } + if record { + if c.Profile.Metadata == nil { + c.Profile.Metadata = map[string]string{} + } + c.Profile.Metadata["handler."+tag] = handler + } } return &toolkit.Result{Text: b.String(), Data: data}, nil } +// handlerRecord says whether a build of tag with handler is recorded for +// later rebuilds. The handler a tag was built with is what the chain ran: +// a build with another one is a different binary, so it needs its own +// image name and never replaces the record. +func handlerRecord(c *toolkit.Context, asked, running, tag, handler string) (bool, error) { + if handler == "" { + return false, nil + } + recorded := c.Profile.Metadata["handler."+tag] + if recorded == "" && c.Profile.ChainID != "" { + if spec, _, err := netspec.Load(c.Profile.ChainID); err == nil { + recorded = spec.HandlerFor(tag) + } + } + if recorded == "" || recorded == handler { + return true, nil + } + named := toolkit.Args{} + specImage(c.Profile, named) + img := ImageFor(c, asked, running, tag) + if asked == "" || img == ImageFor(c, "", running, tag) || img == ImageFor(c, named.String("image", ""), running, tag) { + return false, fmt.Errorf("%s was built for this chain with handler %s — a rebuild of it must add the same one, or nodes replaying the upgrade diverge. To build %s with %s anyway, give it its own image name (image=…); it won't replace the record", tag, recorded, tag, handler) + } + return false, nil +} + +// imagesFor lists the local images (other than the running one) whose +// binary registers plan; with several, those named the network's way win. +func imagesFor(c *toolkit.Context, r *running, plan string) []string { + out, _, _ := sh(c, "docker images --format '{{.Repository}}:{{.Tag}}' | grep -v ''", time.Minute) + var found []string + for _, img := range strings.Fields(out) { + if img == r.Image { + continue + } + if ok, err := imageHas(c, img, r.Binary, plan); err == nil && ok { + found = append(found, img) + } + } + if len(found) > 1 { + if spec, _, err := netspec.Load(c.Profile.ChainID); err == nil && spec.Image.Naming != "" { + re := regexp.MustCompile("^" + strings.ReplaceAll(regexp.QuoteMeta(spec.Image.Naming), regexp.QuoteMeta("{tag}"), `v\d+\.\d+\.\d+`) + "(:latest)?$") + var named []string + for _, f := range found { + if re.MatchString(f) { + named = append(named, f) + } + } + if len(named) == 1 { + found = named + } + } + } + return found +} + func tail(s string, n int) string { l := strings.Split(strings.TrimSpace(s), "\n") if len(l) > n { @@ -508,14 +598,44 @@ func (readyTool) Desc() string { } func (readyTool) Schema() map[string]any { return toolkit.ObjSchema(map[string]any{ - "name": toolkit.Str("upgrade plan name"), - "image": toolkit.Str("the image to switch to after the halt"), - }, "name", "image") + "name": toolkit.Str("upgrade plan name (default: the plan scheduled on chain)"), + "image": toolkit.Str("the image to switch to after the halt (default: the local image whose binary registers the plan)"), + }) } func (readyTool) Tier() toolkit.Tier { return toolkit.TierObserve } func (readyTool) Run(c *toolkit.Context, a toolkit.Args) (*toolkit.Result, error) { - ok, text, data, err := Ready(c, a.String("name", ""), a.String("image", "")) + name, image := a.String("name", ""), a.String("image", "") + if name == "" { + g, err := c.GRPC() + if err != nil { + return nil, fmt.Errorf("pass name — can't read the scheduled plan: %w", err) + } + res, err := g.Upgrade.CurrentPlan(c, &upgradev1beta1.QueryCurrentPlanRequest{}) + if err != nil { + return nil, err + } + if res.Plan == nil { + return nil, fmt.Errorf("no upgrade is scheduled on chain — pass name (and image) to check ahead of a proposal") + } + name = res.Plan.Name + } + if image == "" { + r, err := inspectRunning(c) + if err != nil { + return nil, err + } + switch found := imagesFor(c, r, name); len(found) { + case 0: + text := fmt.Sprintf("✗ no image on %s registers plan %q — build it before the halt: upgrade build --tag (with --handler %s if the release has none)\nnot ready\n", c.Profile.Name, name, name) + return &toolkit.Result{Text: text, Data: map[string]any{"ready": false, "name": name}}, nil + case 1: + image = found[0] + default: + return nil, fmt.Errorf("several images register %q (%s) — pass image", name, strings.Join(found, ", ")) + } + } + ok, text, data, err := Ready(c, name, image) if err != nil { return nil, err } @@ -567,8 +687,8 @@ func (switchTool) Desc() string { } func (switchTool) Schema() map[string]any { return toolkit.ObjSchema(map[string]any{ - "image": toolkit.Str("the image to run now, e.g. primium-evm:v0.6.1"), - }, "image") + "image": toolkit.Str("the image to run now, e.g. mychain-v0.7.3 (default: the local image whose binary registers the halted plan)"), + }) } func (switchTool) Tier() toolkit.Tier { return toolkit.TierLocalChange } func (switchTool) Timeout(toolkit.Args) time.Duration { return 10 * time.Minute } @@ -581,12 +701,29 @@ func (switchTool) Run(c *toolkit.Context, a toolkit.Args) (*toolkit.Result, erro return nil, err } image := a.String("image", "") - if image == "" || strings.ContainsAny(image, "'\"|\\ ") { - return nil, fmt.Errorf("image is required (e.g. primium-evm:v0.6.1)") + if strings.ContainsAny(image, "'\"|\\ ") { + return nil, fmt.Errorf("image %q: a plain image name", image) } if r.ComposeFile == "" { return nil, fmt.Errorf("%s isn't managed by docker compose — no deployed compose file to update", c.Profile.Service.Unit) } + if image == "" { + // the image built for the halted plan: the one whose binary registers it + logs, _, _ := sh(c, "docker logs --tail 300 "+common.ShellQ(c.Profile.Service.Unit)+" 2>&1 | grep -a 'UPGRADE \"' | tail -1", time.Minute) + m := upgradeNeededRe.FindStringSubmatch(logs) + if m == nil { + return nil, fmt.Errorf("%s hasn't halted for an upgrade (no `UPGRADE \"…\" NEEDED` in its log) — nothing to switch to yet", c.Profile.Service.Unit) + } + found := imagesFor(c, r, m[1]) + switch len(found) { + case 0: + return nil, fmt.Errorf("no image on %s registers the halted plan %q — build it: upgrade build --tag (and --handler %s if the release has none)", c.Profile.Name, m[1], m[1]) + case 1: + image = found[0] + default: + return nil, fmt.Errorf("several images register %q (%s) — pass image", m[1], strings.Join(found, ", ")) + } + } if r.Image == image { return nil, fmt.Errorf("%s already runs %s", c.Profile.Service.Unit, image) } @@ -699,3 +836,39 @@ func fetch(c *toolkit.Context, url string) string { raw, _ := io.ReadAll(io.LimitReader(resp.Body, 1<<20)) return string(raw) } + +// specDefaults fills what a new host can't tell from a running node — the +// release's source repo and the image naming — from the network's spec. +func specDefaults(p *config.Profile, r *running, a toolkit.Args) { + if p == nil || p.ChainID == "" { + return + } + spec, _, err := netspec.Load(p.ChainID) + if err != nil { + return + } + if spec.Image.Repo != "" && p.Metadata["repo"] == "" { + r.Repo = spec.Image.Repo + } + if a.String("image", "") == "" && p.Metadata["image"] == "" { + specImage(p, a) + } +} + +// specImage sets the image naming from the network's spec, when it has one. +func specImage(p *config.Profile, a toolkit.Args) { + if p == nil || p.ChainID == "" { + return + } + spec, _, err := netspec.Load(p.ChainID) + if err != nil { + return + } + naming := spec.Image.Naming + if naming == "" && spec.Image.Name != "" { + naming = spec.Image.Name + ":{tag}" + } + if naming != "" { + a["image"] = naming + } +} diff --git a/internal/tools/upgradetool/flow_test.go b/internal/tools/upgradetool/flow_test.go index 2e08df2..c0d748b 100644 --- a/internal/tools/upgradetool/flow_test.go +++ b/internal/tools/upgradetool/flow_test.go @@ -1,10 +1,12 @@ package upgradetool import ( - "github.com/abhijitkrm/cometcli/internal/config" - "github.com/abhijitkrm/cometcli/internal/toolkit" "sort" "testing" + + "github.com/abhijitkrm/cometcli/internal/config" + "github.com/abhijitkrm/cometcli/internal/netspec" + "github.com/abhijitkrm/cometcli/internal/toolkit" ) func TestSemverOrder(t *testing.T) { @@ -81,3 +83,64 @@ func TestBreakingNotes(t *testing.T) { t.Fatalf("from line: %v", m) } } + +func TestFreshHostTakesImageAndRepoFromTheSpec(t *testing.T) { + t.Setenv("COMETCLI_HOME", t.TempDir()) + spec := &netspec.Spec{Chain: netspec.Chain{ID: "fresh-1", Denom: "adex"}, Image: netspec.Image{Name: "primium-evm", Tag: "v0.7.2", Repo: "cosmos/evm", Naming: "primium-{tag}"}} + if _, err := spec.Save(); err != nil { + t.Fatal(err) + } + r := &running{Repo: "someone/else"} + a := toolkit.Args{} + specDefaults(&config.Profile{Name: "n1", ChainID: "fresh-1"}, r, a) + if r.Repo != "cosmos/evm" || a.String("image", "") != "primium-{tag}" { + t.Fatalf("repo %q image %q", r.Repo, a.String("image", "")) + } + // an explicit image or the profile's own settings win + r, a = &running{Repo: "x/y"}, toolkit.Args{"image": "mine:{tag}"} + specDefaults(&config.Profile{Name: "n1", ChainID: "fresh-1", Metadata: map[string]string{"repo": "my/fork"}}, r, a) + if r.Repo != "x/y" || a.String("image", "") != "mine:{tag}" { + t.Fatalf("overrides lost: repo %q image %q", r.Repo, a.String("image", "")) + } +} + +func TestRunningNodeUsesTheSpecNaming(t *testing.T) { + t.Setenv("COMETCLI_HOME", t.TempDir()) + spec := &netspec.Spec{Chain: netspec.Chain{ID: "named-1"}, Image: netspec.Image{Name: "primium-evm", Tag: "v0.7.2", Naming: "primium-{tag}"}} + if _, err := spec.Save(); err != nil { + t.Fatal(err) + } + a := toolkit.Args{} + specImage(&config.Profile{ChainID: "named-1"}, a) + c := &toolkit.Context{Profile: &config.Profile{}} + // running primium-v0.7.2, building v0.7.3: primium-v0.7.3, not primium-v0.7.2:v0.7.3 + if got := ImageFor(c, a.String("image", ""), "primium-v0.7.2", "v0.7.3"); got != "primium-v0.7.3" { + t.Fatalf("image %q", got) + } +} + +// A tag's recorded handler is what the chain ran: a build with another +// handler must not take the tag's image name, and is never recorded. +func TestOtherHandlerNeverReplacesTheRecord(t *testing.T) { + t.Setenv("COMETCLI_HOME", t.TempDir()) + spec := &netspec.Spec{Chain: netspec.Chain{ID: "rec-1"}, Image: netspec.Image{Name: "x-evm", Tag: "v0.7.2", Naming: "x-{tag}"}, + Upgrades: []netspec.UpgradeBuild{{Tag: "v0.7.3", Handler: "v0.7.2-to-v0.7.3"}}} + if _, err := spec.Save(); err != nil { + t.Fatal(err) + } + c := &toolkit.Context{Profile: &config.Profile{ChainID: "rec-1"}} + for _, asked := range []string{"", "x-{tag}", "x-v0.7.3"} { + if _, err := handlerRecord(c, asked, "x-v0.7.3", "v0.7.3", "test-plan"); err == nil { + t.Errorf("image %q: a different handler under the tag's own image name must be refused", asked) + } + } + if rec, err := handlerRecord(c, "x-v0.7.3-test", "x-v0.7.3", "v0.7.3", "test-plan"); err != nil || rec { + t.Errorf("own image name: allowed but not recorded, got %v %v", rec, err) + } + if rec, err := handlerRecord(c, "", "x-v0.7.3", "v0.7.3", "v0.7.2-to-v0.7.3"); err != nil || !rec { + t.Errorf("same handler: %v %v", rec, err) + } + if rec, err := handlerRecord(c, "", "x-v0.7.3", "v0.8.0", "v0.7.3-to-v0.8.0"); err != nil || !rec { + t.Errorf("new tag: %v %v", rec, err) + } +} diff --git a/internal/tools/val/val.go b/internal/tools/val/val.go index 48c81ea..1fb5c7a 100644 --- a/internal/tools/val/val.go +++ b/internal/tools/val/val.go @@ -332,6 +332,13 @@ func (editTool) Run(c *toolkit.Context, a toolkit.Args) (*toolkit.Result, error) if g, err := c.GRPC(); err == nil { if res, err := g.Staking.Validator(c, &stakingv1beta1.QueryValidatorRequest{ValidatorAddr: valoper}); err == nil && res.Validator != nil { msg.Description = res.Validator.Description + // the chain allows one commission change per 24h (creation counts) + if cm := res.Validator.Commission; a.String("commission-rate", "") != "" && cm != nil && cm.UpdateTime != nil { + if next := cm.UpdateTime.AsTime().Add(24 * time.Hour); time.Now().Before(next) { + return nil, fmt.Errorf("commission was last set %s; the chain allows one change per 24h — next possible after %s (in %s)", + cm.UpdateTime.AsTime().Local().Format("2006-01-02 15:04 MST"), next.Local().Format("2006-01-02 15:04 MST"), time.Until(next).Round(time.Minute)) + } + } } } if m := a.String("moniker", ""); m != "" {