feat(test-cli): Add support for testing block building via simulator

[fselmo]

🗒️ Description

Adds a hive-based simulator that verifies EL clients build correct blocks via testing_buildBlockV1

For each payload in a blockchain_test_engine fixture, the test sends the expected transactions and block attributes to the client's block building endpoint. The resulting block is validated against the fixture's expected output (state_root, receipts_root, etc). The fixture block is then imported via engine_newPayloadVX to advance the chain for subsequent blocks.

gas_limit is validated a bit different for now, since testing_buildBlockV1 doesn't accept a gas limit parameter. The client picks its own within the EIP-1559 adjustment range so we validate it is within that range until we can actually account for the gas limit. For this reason, block_hash is also excluded from comparison as it depends on gas_limit.

Also looking for recs on the naming (uv run build-block). We could go with build by itself but it feels vague to me. That said, I can be easily convinced here 😄.

🔗 Related Issues or PRs

closes #2560

✅ Checklist

• All: Ran fast static checks to avoid unnecessary CI fails, see also Code Standards and Enabling Pre-commit Checks:

  just static

• All: PR title adheres to the repo standard - it will be used as the squash commit message and should start type(scope):.
• All: Considered updating the online docs in the ./docs/ directory.
• All: Set appropriate labels for the changes (only maintainers can apply labels).

Cute Animal Picture

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 90.01%. Comparing base (6672cfe) to head (a884759).
⚠️ Report is 57 commits behind head on forks/amsterdam.

Additional details and impacted files

@@                 Coverage Diff                 @@
##           forks/amsterdam    #2679      +/-   ##
===================================================
+ Coverage            86.25%   90.01%   +3.75%     
===================================================
  Files                  599      539      -60     
  Lines                37032    32618    -4414     
  Branches              3795     3030     -765     
===================================================
- Hits                 31943    29361    -2582     
+ Misses                4525     2699    -1826     
+ Partials               564      558       -6     

Flag	Coverage Δ
unittests	90.01% <ø> (+3.75%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[marioevz]

Looks great! Just two comments to try to improve maintainability.

[fselmo]

@marioevz lmk if the last commit here is what you meant. I re-ran against geth bal-devnet-3 with this cherry-picked there with some minor updates that are needed (slotnum, etc) and it's still running smooth 👍🏼

[LouisTsai-Csie]

Thank you @fselmo for this feature! I leave some comment but they are all style-related issue.

For test_via_build.py, i think test_blockchain_via_build is too large as a single function and has too much nested logic. Could we split it into smaller helpers?

My proposal:

test_blockchain_via_build
-> bootstrap_engine_at_genesis: handle genesis block FCU, verify genesis hash
-> Extract valid payload and calls build_validate_and_advance for each payload

build_validate_and_advance
-> build block
-> validate block response and gas limit
-> proceed the chain with newPayload
-> update the chain with FCU

---

edit: two more things for the new command

• Could we add some docs for the new command?
• Should we PR to Hive for the new simulator?

[danceratopz]

Really nice!! 🔥

Nice review by @LouisTsai-Csie. In addition, I added one comment below about adding execution request verification for forks >= Prague.

About the naming, imo, I don't think this merits a new top-level command namespace and it that this rather belongs under consume, please see the comment below.

Docs

Also, could you please add some docs for this? Assuming you rename under consume, we could add it to:

• docs/running_tests/running.md
• docs/consume/simulators.md?

If we don't rename, then it will be trickier to find a spot to document this, but I think this demonstrates that it belongs to the consume family? But it could go in a new section docs/running_tests/build_block/index.md. But I'm strongly in favor of not adding a new top-level entry-point for this functionality.

System Test

Last request, could you add an e2e test in https://github.com/ethereum/execution-specs/blob/c9d8be0101c6649e060950317e14fb4f515f70e9/.github/workflows/hive-consume.yaml ? It should be pretty quick!

[fselmo]

But I'm strongly in favor of not adding a new top-level entry-point for this functionality.

@danceratopz I gave my reasoning in your first comment but I'm curious why the strong favor here. Imo it muddies what consume means. build is also quite generic and build for filling tests with state seems odd to me. I think we need to have some team discussion about it tbh. Could use some more heads on it. Will raise it in the next testing call.

[LouisTsai-Csie]

The recent 2 commits fix all my review comment, but i think we still need the e2e tests: https://github.com/ethereum/execution-specs/blob/c9d8be0101c6649e060950317e14fb4f515f70e9/.github/workflows/hive-consume.yaml

This could be done after we decided whether it should be an individual command or under consume family

[danceratopz]

@danceratopz I gave my reasoning in your first comment but I'm curious why the strong favor here. Imo it muddies what consume means.

Totally fair and I agree it's a different breed of simulator! My consume rationale stems from the fact that this simulator loads a BlockchainEngineFixture and the client consumes the fixture's transactions in order to build a block. I mentally attach "consume" to "client consumes definitions from fixtures". Through that lens, to me, the lack of consume prefix doesn't adequately indicate that fixtures are the input.

You have a different interpretation of "consume" (send blocks/payloads then validate against post state), which is also legitimate.

I'm happy to reconcile my understanding of consume if others have similar interpretations, but I think it'd still be worth a quick discussion in the call tomorrow!

The following is out of scope for this PR, but it'd be great to have names that are intuitive,... in that spirit, I'll just float these ideas while we're on the topic... 😆

Perhaps the name consume itself is too overloaded? I think if we named it again (and perhaps we even can?), I would prefer to call the simulators (in hive path language; not CLI names) as:

• simulators/ethereum/eels/consensus/engine,
• simulators/ethereum/eels/consensus/rlp,
• simulators/ethereum/eels/consensus/sync ( I think that works, still consensus).

I think we discussed this naming once, @spencer-tb. This would break everyone's CIs, but if we did it when (if) we make breaking changes to the release layout, then I think it'd be reasonable if it helps everyone's understanding.

Then (as is currently in this PR) we could have:

• simulators/ethereum/eels/build-block

Or if we think we'll have other build-block simulators, then

• simulators/ethereum/eels/build-block/engine

When it comes to the CLI command, a unified top-level entry point such as uv run hivesim build-block, uv run hivesim execute, uv run hivesim consensus-engine (albeit a bit long) would also give more clarity about these commands and all simulators could be readily discovered via uv run hivesim --help. Or use hive instead of hivesim but I think it's more ambiguous.

[spencer-tb]

I honestly would like to think on naming conventions a bit harder. And align it with what ever "test-el" becomes in the future (fast engine test / blockchain test cli runners). For example:

The hive versions:

• uv run hive enginetest <- consume engine
• uv run hive blocktest <- consume rlp
• uv run hive synctest <- consume sync
• uv run hive buildtest <- new block builder

Then the native cli version, replace el with whatever, maybe evm or native:

• uv run el enginetest
• uv run el blocktest
• uv run el statetest
• uv run el buildtest <- cli version

That being said, I'd lean on keeping what we have here and pushing for a naming overhaul/alignment once the fast cli runners are complete and actively in use!

[danceratopz]

@fselmo happy to go with build-block! Would you like to add the end-to-end test here in this PR or as a follow-up? I think it'd be ok to go in consume-hive.yaml for now? We could consider renaming to hive.yaml in a different PR?

[fselmo]

Would you like to add the end-to-end test here in this PR or as a follow-up?

Hmm I'm not sure I understand what you and @LouisTsai-Csie mean here with e2e test because hive doesn't have this simulator yet. Is this what you mean or something else? We will either need to PR to hive and merge it so that hive-consume.yaml can use it or we can do in a followup PR. Lmk @danceratopz 👀

Either way I need both of you to remove your Requested Changes and approve before this can be merged 😅

[danceratopz]

Would you like to add the end-to-end test here in this PR or as a follow-up?

Hmm I'm not sure I understand what you and @LouisTsai-Csie mean here with e2e test because hive doesn't have this simulator yet. Is this what you mean or something else? We will either need to PR to hive and merge it so that hive-consume.yaml can use it or we can do in a followup PR. Lmk @danceratopz 👀

I won't manage a review until tomorrow, so if you want to add it in the meantime, you could add a new dev entry here?
https://github.com/fselmo/execution-specs/blob/ff283695f8b4a646f9049aa0ea682ca178f15465/.github/workflows/hive-consume.yaml#L109-L112

Either way I need both of you to remove your Requested Changes and approve before this can be merged 😅

Will do! :)

[danceratopz]

This is looking great. I took it for a spin locally and generally got it working well out the box for Geth/v1.17.4-unstable-da34eb59-20260513.

I did see some fails for tests that I'd filled (with -m eels_base_coverage --from Paris) on this branch, but I don't think it's worth reporting them in detail here. Ship it. I think these can be investigated and fixed in follow-ups.

[danceratopz]

I did wonder if there was a better test set to use here, something with multiple blocks? test_clz_stack_underflow is a state test, therefore single block.

Suggested change

	test_filter: "Osaka and test_clz_stack_underflow"
	test_filter: "Osaka and test_block_hashes_history"

e2e test added!