-
Notifications
You must be signed in to change notification settings - Fork 885
[codex] add evm-only executor load test harness #3658
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
9dae839
15ce1e5
af98434
7c27a0a
4c643d0
fd0b615
3fe649f
69f2b54
7c73081
98bd357
5d3ac20
599e184
7fa0b6d
1390018
4d23269
1f29569
bbe827e
4f2763f
74905ee
0d63fb4
1d6fe15
a8d2d95
e5b513a
f27ad40
181ab3c
e20d332
902daa5
c0ebb10
bcfba73
3c58557
6dd1f60
cf656de
4bc9274
201f2e2
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,204 @@ | ||
| # evmonly-loadtest | ||
|
|
||
| `evmonly-loadtest` is a standalone executable for feeding synthetic blocks to | ||
| the EVM-only executor without Cosmos SDK state, mempool, RPC, or chain | ||
| persistence. | ||
|
|
||
| The synthetic workload defaults to local EVM chain ID `1337`; override it with | ||
| `--chain-id` when testing another signing domain. | ||
|
|
||
| It currently generates pure EVM legacy transfer transactions, ERC20 transfer | ||
| transactions using `sei-load`'s compiled contract runtime, and a contract-call | ||
| workload that exercises nested StateDB | ||
| snapshot/revert behavior. By default, each generated sender account has one | ||
| nonce-0 transaction and is funded in the command's in-memory genesis state | ||
| before its block is queued. Recipients are unique by default so the transfer | ||
| workloads exercise the optimistic no-overlap case. Pass | ||
| `--recipient-conflict-rate=<0..1>` to pair that fraction of each block's | ||
| transactions onto shared recipients, or pass `--recipient=0x...` to force all | ||
| transactions to a single recipient. Pass `--same-sender` to use one sender per | ||
| block with sequential transaction nonces. | ||
|
|
||
| Run a bounded prebuilt test: | ||
|
|
||
| ```bash | ||
| go run ./giga/evmonly/cmd/evmonly-loadtest --blocks=1000 --txs-per-block=1000 | ||
| ``` | ||
|
|
||
| Example local saturation run: | ||
|
|
||
| ```bash | ||
| go run ./giga/evmonly/cmd/evmonly-loadtest \ | ||
| --metrics-addr= \ | ||
| --report-interval=5s \ | ||
| --blocks=2000 \ | ||
| --txs-per-block=1000 \ | ||
| --builders=16 \ | ||
| --workers=1 \ | ||
| --executor-workers=12 \ | ||
| --gas-price-wei=0 \ | ||
| --min-gas-price-wei=0 \ | ||
| --queue-size=512 | ||
| ``` | ||
|
|
||
| The three executor OCC scenarios have direct command-line forms: | ||
|
|
||
| ```bash | ||
| # Conflict-free: unique senders and recipients. | ||
| go run ./giga/evmonly/cmd/evmonly-loadtest \ | ||
| --blocks=400 \ | ||
| --gas-price-wei=0 --min-gas-price-wei=0 | ||
|
|
||
| # Hot recipient: unique senders all credit one account. | ||
| go run ./giga/evmonly/cmd/evmonly-loadtest \ | ||
| --blocks=400 \ | ||
| --gas-price-wei=0 --min-gas-price-wei=0 \ | ||
| --recipient=0x00000000000000000000000000000000000000f1 | ||
|
|
||
| # Same-sender nonce chain: one sender and sequential nonces within each block. | ||
| go run ./giga/evmonly/cmd/evmonly-loadtest \ | ||
| --blocks=400 \ | ||
| --gas-price-wei=0 --min-gas-price-wei=0 \ | ||
| --same-sender | ||
| ``` | ||
|
|
||
| Example conflict run: | ||
|
|
||
| ```bash | ||
| go run ./giga/evmonly/cmd/evmonly-loadtest \ | ||
| --metrics-addr= \ | ||
| --report-interval=5s \ | ||
| --blocks=400 \ | ||
| --txs-per-block=5000 \ | ||
| --builders=16 \ | ||
| --workers=1 \ | ||
| --executor-workers=12 \ | ||
| --gas-price-wei=0 \ | ||
| --min-gas-price-wei=0 \ | ||
| --recipient-conflict-rate=0.10 | ||
| ``` | ||
|
|
||
| Example snapshot/revert contract-call run: | ||
|
|
||
| ```bash | ||
| go run ./giga/evmonly/cmd/evmonly-loadtest \ | ||
| --metrics-addr= \ | ||
| --report-interval=5s \ | ||
| --workload=snapshot-revert \ | ||
| --blocks=400 \ | ||
| --txs-per-block=5000 \ | ||
| --builders=16 \ | ||
| --workers=1 \ | ||
| --executor-workers=12 \ | ||
| --gas-price-wei=0 \ | ||
| --min-gas-price-wei=0 | ||
| ``` | ||
|
|
||
| The command prebuilds every bounded run before starting the prepare/recover and | ||
| executor workers, which isolates executor throughput from block generation: | ||
|
|
||
| ```bash | ||
| go run ./giga/evmonly/cmd/evmonly-loadtest \ | ||
| --metrics-addr= \ | ||
| --report-interval=5s \ | ||
| --blocks=400 \ | ||
| --txs-per-block=5000 \ | ||
| --builders=48 \ | ||
| --workers=1 \ | ||
| --executor-workers=24 \ | ||
| --gas-price-wei=0 \ | ||
| --min-gas-price-wei=0 \ | ||
| --queue-size=512 | ||
| ``` | ||
|
|
||
| Prebuilding requires `--blocks > 0` and stores every raw block in memory. Sender | ||
| recovery still runs in the measured phase, but it is pipelined ahead of | ||
| execution through `--prepare-workers`. | ||
|
|
||
| The zero gas price/min-gas settings keep the conflict-free transfer workload | ||
| focused on the optimistic no-overlap case. Non-zero fees make every transaction | ||
| update the same coinbase balance, which is a real intra-block conflict. | ||
|
|
||
| Useful knobs: | ||
|
|
||
| - `--blocks`: number of blocks to prebuild and execute. This is required and | ||
| must be greater than `0`. | ||
| - `--workers`: parallel executor workers. The default is `1`. Prepared blocks | ||
| are forwarded to workers in block-number order, but `--workers > 1` can still | ||
| finish execution out of order; this is safe for the harness because generated | ||
| state is frozen for prebuilt runs and executor changesets are not applied back | ||
| into the input state. | ||
| - `--executor-workers`: parallel OCC workers inside each executor. The default | ||
| is `min(12, GOMAXPROCS)`, following the `sei-v3` OCC worker default. | ||
| - `--prepare-workers`: parallel stateless preparation workers used for | ||
| transaction RLP decode and sender recovery. The default is `GOMAXPROCS`. | ||
| - `--parse-workers`: parallel transaction decode/sender recovery workers inside | ||
| each prepared block. The default `0` uses `1` when `--prepare-workers > 1` to | ||
| avoid multiplying block-level and intra-block parser pools, otherwise it uses | ||
| `GOMAXPROCS`. | ||
| - `--builders`: parallel block builders used to keep the input queue full. The | ||
| default is `GOMAXPROCS`. | ||
| - `--queue-size`: buffered raw and prepared blocks. The default is `64`. | ||
| - `--result-pool-size`: reusable executor result slots. The default `0` | ||
| sizes the pool for in-flight executor and async sink results; negative | ||
| disables result pooling. | ||
| - `--result-sink`: executor output sink, either `discard` or `file`. The | ||
| default is `discard`. | ||
| - `--persist-dir`: directory used by `--result-sink=file` for temporary | ||
| append-only changeset and receipt files. Files are removed on shutdown or | ||
| interrupt cleanup. | ||
| - `--persist-buffer-size`: buffered writer size for `--result-sink=file`. | ||
| - `--persist-queue-size`: async file-sink record queue size. The default `0` | ||
| uses `2 * --queue-size`. | ||
|
Comment on lines
+148
to
+152
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🟡 The 'Useful knobs' list in the loadtest README omits Extended reasoning...The bug: fs.BoolVar(&cfg.persistSync, "persist-sync", false, "fsync persistent result files from the async sink writer")
Where the gap is:
Why nothing else catches it: This is a pure documentation gap — the flag itself works correctly in code and is tested. There's no validation or generation step tying the README's knob list to the actual flag set in Step-by-step proof:
Fix: Add a line for Severity: This is scoped to |
||
| - `--metrics-addr`: Prometheus endpoint. The default is | ||
| `127.0.0.1:9698`; set it to empty to disable HTTP metrics. | ||
| - `--report-interval`: stdout rate reporting interval. The default is `5s`. | ||
| - `--gas-price-wei`, `--min-gas-price-wei`, `--sender-balance-wei`, | ||
| `--transfer-value-wei`: transaction economics for the generated accounts. | ||
| - `--recipient-conflict-rate`: fraction of each block's transactions that are | ||
| paired onto shared recipients; `0` keeps recipients unique and `1` pairs all | ||
| possible transactions. | ||
| - `--same-sender`: use one sender per native-transfer block and assign | ||
| transaction nonces in block order. | ||
| - `--workload`: workload type, either `transfer`, `erc20-transfer`, or | ||
| `snapshot-revert`. | ||
| - `--snapshot-revert-contract`, `--snapshot-revert-helper`: generated contract | ||
| addresses used by `--workload=snapshot-revert`. | ||
|
|
||
| The command reports these saturation signals on stdout and at `/metrics`: | ||
|
|
||
| - block input throughput | ||
| - block preparation throughput | ||
| - prepared transactions per second | ||
| - block finishing throughput | ||
| - finished, successful, and failed transactions per second | ||
| - total gas consumed per second | ||
| - total OCC transaction rerun attempts | ||
| - prepared blocks queued for execution and cumulative totals | ||
| - result-sink records queued, enqueued, written, bytes written, enqueue wait, | ||
| and write time | ||
| - result-pool capacity, available slots, and overflow allocations | ||
|
|
||
| The default executor output path intentionally discards results through mocks: | ||
|
|
||
| - `generatedState` implements `evmonly.StateReader` and supplies generated | ||
| genesis balances, nonces, code, and storage. | ||
| - `discardResultSink` applies the executor `StateChangeSet` to | ||
| `discardStateWriter` and discards Ethereum receipts. | ||
|
|
||
| With `--result-sink=file`, the loadtest harness hands pooled | ||
| `evmonly.BlockResult` values to an async writer through the executor's | ||
| `evmonly.ResultSink` interface. The writer appends changesets to | ||
| `changesets.rlp` and receipts to `receipts.rlp` under `--persist-dir`; each | ||
| record is framed as an 8-byte big-endian block height, an 8-byte big-endian RLP | ||
| payload length, and the RLP payload. The files are temporary calibration | ||
| artifacts and are removed when the process exits normally or handles | ||
| `SIGINT`/`SIGTERM`. `sink_enqueue_wait` is the primary backpressure signal: a | ||
| non-zero value means executor workers waited for async sink queue capacity. | ||
|
|
||
| Native and ERC20 transaction construction and genesis seeding come from | ||
| `github.com/sei-protocol/sei-load/generator/offline`. Future reusable EVM | ||
| workloads should add an offline scenario there and a block/recipient adapter in | ||
| this command. Executor-specific workloads can remain beside | ||
| `TransferWorkload`, `ERC20TransferWorkload`, and `SnapshotRevertWorkload` and | ||
| reuse the same block producer/prepare/executor/metrics pipeline. | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[nit]
--persist-syncis missing from this knob list even though it materially changes file-sink cost (flush +fsyncper record) and is exercised inmain_test.go. Worth a line next to--persist-buffer-sizeso the durability/throughput tradeoff is discoverable.