Skip to content

Feat/proof backfill batching#21602

Draft
dhyaniarun1993 wants to merge 7 commits into
developfrom
feat/proof-backfill-batching
Draft

Feat/proof backfill batching#21602
dhyaniarun1993 wants to merge 7 commits into
developfrom
feat/proof-backfill-batching

Conversation

@dhyaniarun1993

@dhyaniarun1993 dhyaniarun1993 commented Jun 30, 2026

Copy link
Copy Markdown
Contributor

Description

op-proofs backfill writes one MDBX transaction per block. On a 1.2 M-block extension that's 1.2 M commit/fsync round-trips. This PR batches K blocks into a single transaction.

  • New flag: --proofs-history.backfill-batch-size (range 1..=100, default 50).
  • Backfill on op-mainnet improvements: 8.3 → 22.5 blk/s (~2.7×) on a 1.2 M-block run.
  • See Caveats before assuming the same multiplier on your host.

Failure semantics
A crash or validation error mid-batch drops the open RW transaction; up to K blocks of progress are lost. earliest in the proofs DB is unchanged, so the next run resumes from the same anchor and redoes them. Atomicity is preserved at batch granularity, not block granularity.

Caveats

  • Hardware-sensitive - The 2.7× figure is from a 64 GB server. MDBX is memory-mapped and relies on the OS page cache for hot pages — at 64 GB, most of the trie working set stays resident, so the dominant per-block cost shifts to commit/fsync (which batching attacks directly). On a smaller host, per-block I/O would dominates instead, and the commit-reduction win shrinks.
  • Cache-warmup ramp - When the proofs DB already has populated history tables (continuing a paused backfill, or re-running on an existing snapshot), the first several minutes run noticeably below the steady-state rate while the OS page cache reloads the hot trie/history pages. Throughput climbs as residency grows.

Tests

Additional context

Metadata

@dhyaniarun1993 dhyaniarun1993 self-assigned this Jun 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant