Repository navigation
Conversation
Without msf-1's launch outputs: inventory.ini (terraform writes it), files/shadowfork_block.json and files/prestate/ (jochemnet-specific). VPC 10.33.0.0/16 (msf-1 has 10.32). Secrets: the metrics/logs usernames and xatu ingest address are glamsterdam-msf-2, edited with sops --set.
…), bridge mainnet txs Images instead of jochemnet: a canonical mainnet snapshot advanced by one geth pre-run (EIP-8282 contracts, funding, 370k receivers/variant), replayed by every other client, so there is no prestate import and no shadowfork_head_height. shadowfork_height and the genesis time stay TODO(msf-2) until the fresh build exists; 0 fails the restore. Layouts read from the published archives: geth keeps the geth/ prefix (out_dir /data/geth), reth and erigon are plain snapshot.tar.zst, nethermind is not FlatDb. ethrex has no image and snap-syncs from the restored ELs (sepsf-2). Static enodes reset. shadowfork.yaml: shadowfork_snapshot_base_url picks the snapshot host (play vars beat group_vars). mempool-bridge.yaml: mainnet devp2p -> our ELs on the bootnode, host network so the SF-P2P DOCKER-USER rules do not drop it.
Tested from a fra1 droplet: devp2p handshakes with mainnet work, but every reachable mainnet node is full (27 of 34 dials: too many peers, including ethpandaops' lighthouse-geth-001 and the 14 nodes it dials out to), and the bridge's node key is random per process, so it cannot be made a trusted peer. rpc mode polls txpool_content on a mainnet node with the txpool API and sends to one validator EL per client (resolved: prysm-besu-1, grandine-erigon-1, grandine-ethrex-1, prysm-geth-1, prysm-nethermind-1, prysm-reth-1). The config holds basic-auth URLs: 0600, no_log.
The images now cross Amsterdam in the pre-run and fill 325k receivers per variant at 24 KiB and 64 KiB (Amsterdam's test_account_access needs both; 64 KiB is illegal before EIP-7954). So the CL starts on Gloas (gloas_fork_epoch 0, 200M gas from genesis, builders from genesis) and there is no live transition on msf-2. - normalize-shadowfork-genesis.py takes an optional amsterdamTime; shadowfork_amsterdam_time (TODO until the build) sets it to the image's own fork time. egg would put it at genesis, after the image's Amsterdam blocks, and the ELs would rewind them. - pregenesis check: eth_config compares (next // current).activationTime, since a post-Amsterdam head has no next fork; and checks code at both EIP-8282 addresses, failing when the head is past Amsterdam. - the deploy-eip8282-contracts assertoor startup test is dropped: it refuses after Gloas, and the image holds both contracts.
![redpandabot[bot]](https://avatars.githubusercontent.com/in/3749543?s=60&v=4){kind=small}
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
…dabot) - mempool-bridge.yaml: the target URLs are built with a loop. regex_replace's '\1' arrived as U+0001 (Jinja unescapes string literals), so every URL pointed at one bogus host. Rendered against the msf-1 inventory with dummy credentials: one rpc-<host> URL per client. - snapshot_download.sh.j2: also removes geth/nodekey and geth/nodes. An archive packed with the geth/ prefix keeps them one level down, out of reach of the top-level cleanup. (Our images exclude both at pack time; the restore no longer depends on that.)
This comment has been minimized.
This comment has been minimized.
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
Every EL but nimbus-el restores shadowfork-images.ethpandaops.io/jochemnet/<client>/<head>/; the client group_vars are msf-1's jochemnet layouts again (reth snapshot-v2, erigon snapshot-pruned, nethermind FlatDb, ethrex restored to chain-1 with full sync), except geth and the bootnode, which keep out_dir /data/geth for the export's geth/ prefix. SHADOW_FORK_FILE and amsterdamTime are read from the bucket next to the geth snapshot, and the pregenesis gate fetches the same block when the inventory has no file. Drops the mempool-bridge play: the release head is not on mainnet.
qu0b
changed the title
This comment has been minimized.
This comment has been minimized.
The snapshot export now packs geth's instance dir unprefixed, the snapshots.ethpandaops.io convention.
There was a problem hiding this comment.
Summary
msf-2 is a new mainnet shadowfork inventory restored from the re-exported jochemnet release, with Gloas at genesis. The last commit changed the geth/bootnode snapshot restore target to /data/geth/geth, which contradicts the PR description's statement that the msf-2 export packs the geth/ prefix; the earlier review's open findings are otherwise resolved (mempool-bridge.yaml is gone, and the geth key/peer cleanup now matches geth/nodekey and geth/nodes).
Issues
- 🟡
ansible/inventories/msf-2/group_vars/geth.yaml:11— geth snapshot extracts one level too deep for the geth/-prefixed msf-2 export — see the thread on that line
Reviewed @ 9eb5dee7
"Plans are worthless, but planning is everything." — Eisenhower
| # role: ethpandaops.general.snapshot_fetcher: a geth archive is geth's instance dir, unprefixed | ||
| # (snapshots.ethpandaops.io and benchmarkoor snapshot export alike), so it goes under <datadir>/geth. | ||
| # Extracted flat, geth opens chaindata/ but misses triedb/merkle.journal and rewinds its head. | ||
| snapshot_fetcher_out_dir: /data/geth/geth |
There was a problem hiding this comment.
🟡 geth snapshot extracts one level too deep for the geth/-prefixed msf-2 export
The PR description says the msf-2 (shadowfork-images) export packs the geth/ prefix, unlike the old snapshots.ethpandaops.io jochemnet tarball, and the pre-9eb5dee code set out_dir: /data/geth for that reason. The fetch container mounts snapshot_fetcher_out_dir at /data and untars with -C /data, and geth's instance dir is <geth_datadir>/geth = /data/geth/geth, so a prefixed archive must be extracted into /data/geth; extracting into /data/geth/geth would put chaindata at /data/geth/geth/geth/chaindata while geth looks at /data/geth/geth/chaindata. ansible/inventories/msf-2/group_vars/bootnode.yaml:105 has the same value. I could not fetch the bucket to inspect the archive (Cloudflare refused), so if the re-export is in fact unprefixed this is moot — but it contradicts the PR's stated layout and the pre-9eb5dee code.
… a reduced fleet can pass its own total
There was a problem hiding this comment.
msf-2 is a copy of msf-1 that boots from the cleaned jochemnet release snapshots with Gloas at genesis; the latest commit only routes NUMBER_OF_VALIDATORS through a new overridable ethereum_genesis_validator_count (default 36000, matching the 36x1000 validator ranges). The previously open mempool-bridge findings are moot (the file was dropped) and the geth snapshot-depth finding is resolved: geth now restores to /data/geth/geth exactly as the merged msf-1 inventory, and the snapshot cleanup handles both prefixed and unprefixed key/nodes paths. I found no new real issues in the changed scope.
Reviewed 45 changed file(s) @ 27ef4f16 — no blocking issues found.
"Plans are worthless, but planning is everything." — Eisenhower
This comment has been minimized.
This comment has been minimized.
…r teku's 32-byte cap Found on the 2026-10-11 mock run: teku-nethermind-1's beacon restart-looped on "Invalid value for option '--validators-graffiti': 'glamsterdam-msf-2-teku-nethermind-1' converts to 35 bytes". buildoor-teku-nethermind-1 (44 bytes) would have hit it too.
There was a problem hiding this comment.
Summary
The one new commit adds --Merge.TerminalTotalDifficulty=0 to the msf-2 Nethermind args so the restored replay headers are treated as post-merge; the flag exists in Nethermind 2.1.0's MergeConfig and is applied via PoSSwitcher, so it looks correct. Of the earlier open findings, the mempool-bridge ones are moot (file removed) and the geth layout/cleanup ones are addressed, but the Lighthouse graffiti issue is still present: msf-2's Lighthouse group_vars still passes the long machine hostname, which Lighthouse's VC rejects.
Issues
- 🔴
ansible/inventories/msf-2/group_vars/lighthouse.yaml:50— Lighthouse validator rejects the >32-byte machine hostname as graffiti — see the thread on that line
Reviewed @ 70f42085
"Plans are worthless, but planning is everything." — Eisenhower
| - "{{ eth_testnet_config_dir }}:/network-config:ro" | ||
| lighthouse_validator_container_command_extra_args: | ||
| - --testnet-dir=/network-config | ||
| - --graffiti={{ ansible_hostname }} |
There was a problem hiding this comment.
🔴 the teku graffiti fix is incomplete: Lighthouse rejects the long hostname too
This still passes --graffiti={{ ansible_hostname }} to the Lighthouse validator client. Terraform names each droplet <network>-<node> (e.g. glamsterdam-msf-2-lighthouse-geth-1, 34 bytes) and fact caching (gathering=smart, fact_caching_timeout=0 in ansible/ansible.cfg) keeps the pre-bootstrap value, so it is the same over-32-byte string the teku fix just worked around. Lighthouse v8.3.0-rc.0 validator_client/src/config.rs returns Your graffiti is too long! 32 bytes maximum! and exits, so every Lighthouse VC restart-loops; the same ansible_hostname is still used in nimbus.yaml:53, lodestar.yaml:53, grandine.yaml:49 and prysm.yaml:63.
This branch has not been deployed
1 participant
msf-2 is a mainnet shadowfork for benchmarking. It uses the same fleet as msf-1, restored from the cleaned jochemnet release snapshots. That release is:
1769856769;shadowfork-images.ethpandaops.io/jochemnet/<client>/<head>/, in the snapshots.ethpandaops.io layout.The snapshot is already on Amsterdam, so msf-2 starts on Gloas at genesis: there is no live transition, and builders run from genesis.
Draft:
shadowfork_heightis 24452538, the common published head. Setethereum_genesis_timestampfor the real launch; it remains 0 in this branch. The repaired Nethermind archive is published and its final offline CI-helper check passed.What it does
<base>/jochemnet/<el>/<shadowfork_height>/<filename>.shadowfork.yamltakes the host fromshadowfork_snapshot_base_url, because play vars beat group_vars, so setting the role'ssnapshot_fetcher_base_urlin the inventory would be overridden. nimbus-el still snap-syncs from the restored ELs.out_dir /data/geth/geth, as msf-1: a geth archive is geth's instance dir with no prefix (snapshots.ethpandaops.io andbenchmarkoor snapshot exportalike). Extracted flat, geth openschaindata/but missestriedb/merkle.journaland rewinds its head (found by the e2e prototype).snapshot-v2.tar.zst. This is v2 storage;snapshot.tar.zstis the old v1 format.snapshot-pruned.tar.zst. Frozen up to 24401000; the ~43k release blocks above that are in the DB, so retire runs soon after genesis.--FlatDb.Enabled=trueand--Merge.TerminalTotalDifficulty=0, matching the snapshot replay. The TTD override is required to reopen synthetic headers correctly./data/ethrex/chain-1(ethrex hardcodes<datadir>/chain-<id>), with--syncmode=fullso it follows the CL.GAS_LIMIT_SCHEDULEsets 200M at epoch 0.SHADOW_FORK_FILEandshadowfork_amsterdam_timeare read withlookup('url')from the geth snapshot's_snapshot_eth_getBlockByNumber.jsonand_snapshot_metadata.json(.shadowfork.amsterdam_time). They are not committed files.amsterdamTime. egg would set it to genesis, after the release's Amsterdam blocks, and every EL would rewind them..result.next // .result.current, since Amsterdam is already current.files/shadowfork_block.json, it fetches the same block JSON from the bucket.snapshot_download.sh.j2also removesgeth/nodekeyandgeth/nodes.Validated
ansible-playbook --syntax-checkpasses forplaybook.yamlandshadowfork.yaml, run against msf-1's inventory.ini renamed to msf-2. msf-2 has no inventory.ini until terraform runs.shadowfork.yaml's play vars at a stand-in head of 24443820:…/jochemnet/geth/24443820/snapshot.tar.zst → /data/geth/geth…/jochemnet/besu/24443820/snapshot.tar.zst → /data/besu…/jochemnet/nethermind/24443820/snapshot.tar.zst → /data/nethermind/nethermind_db…/jochemnet/reth/24443820/snapshot-v2.tar.zst → /data/reth…/jochemnet/erigon/24443820/snapshot-pruned.tar.zst → /data/erigon…/jochemnet/ethrex/24443820/snapshot.tar.zst → /data/ethrex/chain-1amsterdam_time=1769856769, the block hash,GLOAS_FORK_EPOCH="0"and the 200M schedule. At the unpublished default height (0) they fail on a 404 that names the URL.blockAccessListHashandslotNumber.gloasgenesis state withGLOAS_FORK_EPOCH: 0andGAS_LIMIT_SCHEDULEat 200M.latest_block_hashequals an independent RLP+keccak of the full Amsterdam header, includingblockAccessListHashandslotNumber. Hashing without those fields gives a different value.normalize-shadowfork-genesis.py … 1769856769setsamsterdamTimeand keeps mainnet's osaka/bpo1/bpo2 times.Open before launch
deposit_contract_block_hash == block hashusing the real published head and Amsterdam header.snapshot-v2.tar.zstand erigonsnapshot-pruned.tar.zstboth restored successfully in the mock. The image-domain WAF now allows both names.dffrom restore on, and consider a larger host for nethermind if the run is meant to last more than a few days.2026-10-11 mock and Nethermind restart fix
The 8-node mock (one validator node per EL, 6,000 validators) reached finalized epoch 3 on every node, with identical sampled EL block hash/state root. Funding dry run passed; all 33 mock resources were destroyed.
Nethermind needs
--Merge.TerminalTotalDifficulty=0, matching the replay: stored synthetic headers can have total difficulty below mainnet TTD. The repaired snapshot reopened offline at block 24452538 with the correct hash and state root using this override; the same consumer config without it rejected the final block. The replacement archive is published (1,325,681,692,842 bytes), passed public range/hash verification, and passed the final offline CI helper at 22:48 UTC. All six public archives and head/metadata objects were audited successfully.