Skip to content

feat(test-benchmark): add eth transfer cases for repricing#2837

Merged
LouisTsai-Csie merged 11 commits into
ethereum:forks/amsterdamfrom
jochem-brouwer:eth-transfer-bench-repricing
May 14, 2026
Merged

feat(test-benchmark): add eth transfer cases for repricing#2837
LouisTsai-Csie merged 11 commits into
ethereum:forks/amsterdamfrom
jochem-brouwer:eth-transfer-bench-repricing

Conversation

@jochem-brouwer
Copy link
Copy Markdown
Member

@jochem-brouwer jochem-brouwer commented May 12, 2026

🗒️ Description

We have ether transfer benchmarks, but they're incomplete. We're missing:

  • Non-existing EOA
  • Existing EOA (balance-only, zero nonce, no code)
  • Contract account

For the latter two cases, we want to use pre-existing accounts from perf-devnet-3, mainnet, or jochemnet snapshots instead of creating them during pre-allocation. This keeps the accounts outside the client cache and prevents benchmark skewing.

Verification on gas-benchmarks with 300M: https://github.com/NethermindEth/gas-benchmarks/actions/runs/25720366344/job/75519841684

🔗 Related Issues or PRs

N/A.

✅ Checklist

  • All: Ran fast static checks to avoid unnecessary CI fails, see also Code Standards and Enabling Pre-commit Checks:
    just static
  • All: PR title adheres to the repo standard - it will be used as the squash commit message and should start type(scope):.
  • All: Considered updating the online docs in the ./docs/ directory.
  • All: Set appropriate labels for the changes (only maintainers can apply labels).
  • Tests: Ran mkdocs serve locally and verified the auto-generated docs for new tests in the Test Case Reference are correctly formatted.
  • Tests: For PRs implementing a missed test case, update the post-mortem document to add an entry the list.
  • Ported Tests: All converted JSON/YML tests from ethereum/tests or tests/static have been assigned @ported_from marker.

Cute Animal Picture

Put a link to a cute animal picture inside the parenthesis-->

@LouisTsai-Csie
Copy link
Copy Markdown
Collaborator

LouisTsai-Csie commented May 12, 2026

Changes:

I split the original ether transfer test into two: test_ether_transfer (the existing one) and test_ether_transfer_on_receiver (new). This avoids skipping test parametrization. In addition, the latest implementation now accounts for the execution cost of Bittrex-created contracts.

Question:

After review, i notice the scenario of diff_to_nonexistent in test_ether_transfers_onchain_receivers is the same as empty_account x diff_acc_to_diff_acc parametrization in test_ether_transfers benchmark

@codecov
Copy link
Copy Markdown

codecov Bot commented May 12, 2026

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 86.94%. Comparing base (d61f7ac) to head (6d1dd69).

Additional details and impacted files
@@               Coverage Diff                @@
##           forks/amsterdam    #2837   +/-   ##
================================================
  Coverage            86.94%   86.94%           
================================================
  Files                  586      586           
  Lines                35767    35767           
  Branches              3362     3362           
================================================
  Hits                 31098    31098           
  Misses                4010     4010           
  Partials               659      659           
Flag Coverage Δ
unittests 86.94% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@LouisTsai-Csie LouisTsai-Csie marked this pull request as ready for review May 12, 2026 06:05
Copy link
Copy Markdown
Contributor

@fselmo fselmo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not really sure how to properly test this (I can't locally) but I left some things I had questions on and some nits. @LouisTsai-Csie can you give a proper sanity check on the actual test run here? If so, can you give a final approve / review here?

Comment thread tests/benchmark/stateful/bloatnet/test_transaction_types.py Outdated
Comment thread tests/benchmark/stateful/bloatnet/test_transaction_types.py Outdated
Comment thread tests/benchmark/stateful/bloatnet/test_transaction_types.py
Comment thread tests/benchmark/stateful/bloatnet/test_transaction_types.py Outdated
Comment thread tests/benchmark/stateful/bloatnet/test_transaction_types.py Outdated
@LouisTsai-Csie
Copy link
Copy Markdown
Collaborator

LouisTsai-Csie commented May 13, 2026

Nice review and thanks @fselmo , we could not test it out without snapshot, i will test it in gas-benchmarks, review and merge.

Edit: this is the latest benchmark payload generation with receipt check, it is a successful run: https://github.com/NethermindEth/gas-benchmarks/actions/runs/25806153732/job/75809369217

@jochem-brouwer
Copy link
Copy Markdown
Member Author

Claude and I added a distinct EOA sender with

uv run python -c "
from tests.benchmark.stateful.bloatnet.test_transaction_types import (SENDER_BASE_KEY)
from execution_testing import EOA

out_path = 'tests/benchmark/stateful/bloatnet/fixtures/sender_addresses.txt'
with open(out_path, 'w') as f:
    for i in range(SENDER_COUNT):
        f.write(EOA(key=15000 + i).hex() + '\n')
print(f'wrote {SENDER_COUNT} addresses to {out_path}')
"

Gas benchmark target: NethermindEth/gas-benchmarks#144

@LouisTsai-Csie
Copy link
Copy Markdown
Collaborator

LouisTsai-Csie commented May 14, 2026

@jochem-brouwer i think the account generated in gas-benchmark does not align with the one in EELS.

Based on your code section, the pk range starts from 15000

uv run python -c "
from tests.benchmark.stateful.bloatnet.test_transaction_types import (SENDER_BASE_KEY)
from execution_testing import EOA

out_path = 'tests/benchmark/stateful/bloatnet/fixtures/sender_addresses.txt'
with open(out_path, 'w') as f:
    for i in range(SENDER_COUNT):
        f.write(EOA(key=15000 + i).hex() + '\n') # the pk range from 15,000 to 15,000+sender_count
print(f'wrote {SENDER_COUNT} addresses to {out_path}')

But in the EELS code, the sender pk starts from SENDER_BASE_KEY (0x1111111111111111111111111111111111111111111111111111111111111111)

I will change EELS implementation in this PR to align with gas-benchmarks address.


Edit:

I notice that the first address in sender_addresses.txt does not match the one calculate from EOA(key=15000), but EOA(key=0x1111111111111111111111111111111111111111111111111111111111111111). The implementation in EELS and gas-benchmarks match but the generation code is wrong. Nothing needs changed here.

@LouisTsai-Csie LouisTsai-Csie force-pushed the eth-transfer-bench-repricing branch from 147b123 to 6d1dd69 Compare May 14, 2026 07:46
@LouisTsai-Csie LouisTsai-Csie self-assigned this May 14, 2026
@LouisTsai-Csie LouisTsai-Csie changed the title feat(benchmarks): add eth transfer cases for repricing feat(test-benchmark): add eth transfer cases for repricing May 14, 2026
@LouisTsai-Csie LouisTsai-Csie added A-test-benchmark Area: execution_testing.benchmark and tests/benchmark C-feat Category: an improvement or new feature labels May 14, 2026
@LouisTsai-Csie LouisTsai-Csie merged commit c8117f2 into ethereum:forks/amsterdam May 14, 2026
18 checks passed
LouisTsai-Csie added a commit to LouisTsai-Csie/execution-specs that referenced this pull request May 14, 2026
…2837)

* (Claude): add eth transfer cases for repricing

* refactor: split test based on pre-alloc

* refactor: bump execution cost for contract ether reception

* chore: remove unnecessary parametrization

* refactor: move new benchmark under stateful folder

* fix: apply suggested changes

* feat: add receipt check

* (Claude): add distinct senders

* (Claude): do not limit distinct senders

* (Claude): add uniq jumpdest contract test

* refactor unique contract code receiver case

---------

Co-authored-by: LouisTsai <q1030176@gmail.com>
@jochem-brouwer
Copy link
Copy Markdown
Member Author

Hi Louis, sorry, I copied an old version of the pkey generated code, and removed a constant without checking if the generated code was correct.

This is the code to generate the file

uv run python -c "
from tests.benchmark.stateful.bloatnet.test_transaction_types import (
    SENDER_BASE_KEY, SENDER_COUNT,
)
from execution_testing import EOA

out_path = 'tests/benchmark/stateful/bloatnet/fixtures/sender_addresses.txt'
with open(out_path, 'w') as f:
    for i in range(SENDER_COUNT):
        f.write(EOA(key=SENDER_BASE_KEY + i).hex() + '\n')
print(f'wrote {SENDER_COUNT} addresses to {out_path}')
"

SENDER_COUNT = 15000 in a previous version of the PR.

(I edited SENDER_BASE_KEY instead of SENDER_COUNT it seems 🤦 )

Sorry about that, this gave extra work which was not necessary, will check code next time I comment it. Thanks for checking and catching this!! 😄 👍 (it should thus not be a problem, and you noted this, the posted code is wrong but the code in the PR is right and there is thus no problem)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-test-benchmark Area: execution_testing.benchmark and tests/benchmark C-feat Category: an improvement or new feature

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants