Skip to content

Compact Headers & Compact Blocks - BIP 152 Implementation#266

Merged
blondfrogs merged 2 commits intomasterfrom
compactheaders
Apr 21, 2026
Merged

Compact Headers & Compact Blocks - BIP 152 Implementation#266
blondfrogs merged 2 commits intomasterfrom
compactheaders

Conversation

@blondfrogs
Copy link
Copy Markdown
Member

@blondfrogs blondfrogs commented Oct 31, 2025

Compact Headers & Compact Blocks - BIP 152 Implementation

Implements BIP 152 style bandwidth optimization for both header sync and block relay.

Features

  1. Compact Headers (cmpheaders):
  • New message for efficient header sync of checkpointed blocks
  • Omits Equihash nSolution (~1344 bytes) from POW block headers
  • ~80% bandwidth reduction (344KB vs 1.7MB per 2000 headers)
  • PON blocks send full data (already compact)
  1. Compact Blocks (cmpctblock - BIP 152):
  • Send blocks with short transaction IDs instead of full transactions
  • Reconstruct blocks from mempool + short IDs
  • Request missing transactions via getblocktxn/blocktxn messages
  • High-bandwidth mode for low-latency block relay
  • Ring buffer of 100 recent transactions for better reconstruction
  1. Headers-First Sync:
  • Download all headers before requesting blocks
  • Multi-peer parallel block downloads after header sync
  • CanDirectFetch() gates block downloads until synced
  • Periodic disk flush (every 250k headers) prevents data loss

Network Protocol

  • Protocol version 170021 (SENDCMPCT_VERSION/CMPHEADERS_VERSION)
  • Messages: cmpheaders, sendcmpct, cmpctblock, getblocktxn, blocktxn
  • Checkpoint validation for security (compact headers only below checkpoint)
  • Initialize pindexBestKnownBlock from peer nStartingHeight for multi-peer sync

Database

  • Stores actual block hash for compact headers (can't compute without nSolution)
  • Backwards compatible serialization
  • Old nodes ↔ new nodes: fully compatible

Implementation

New files:

  • blockencodings.cpp/h - BIP 152 compact block structures
  • crypto/siphash.cpp/h - Short transaction ID generation

Modified:

  • main.cpp - Protocol handlers, sync logic
  • chain.h - CDiskBlockIndex with hashBlock storage
  • version.h, main.h, net.cpp/h, and 6 others

Testing

  • ✅ Synced to 2M+ blocks at ~45 blocks/sec
  • ✅ Backwards compatible with protocol < 170021
  • ✅ Database compatibility verified

@blondfrogs blondfrogs changed the title Compactheaders Compact Headers & Compact Blocks - BIP 152 Implementation Oct 31, 2025
@blondfrogs blondfrogs merged commit 93fab07 into master Apr 21, 2026
4 checks passed
@blondfrogs blondfrogs mentioned this pull request Apr 21, 2026
Merged
4 tasks
blondfrogs added a commit that referenced this pull request Apr 22, 2026
@blondfrogs
Perf/ibd speedups (#277)
279c37f 
* Silence per-block LogPrintf calls on sync hot path

Four unconditional LogPrintf calls were firing on every PON header and
every block connect/disconnect during IBD, dominating debug.log volume
and measurably slowing sync. Convert them to LogPrint with a category
so they only fire under -debug=pon or -debug=fluxnode.

- src/pon/pon.cpp: PON difficulty adjustment traces (2x per PON header)
- src/fluxnode/fluxnode.cpp: FluxnodeCache::LogDebugData, plus an
  early-return so the per-block string concatenation over
  mapStartTxTracker / mapStartTxDOSTracker is skipped when the fluxnode
  category isn't active.

* Adopt Bitcoin Core's dbcache split to favor the in-memory UTXO set

Fluxd used to allocate 3/4 of -dbcache to the block tree DB whenever
-insightexplorer was enabled, which left only ~76 MiB of in-memory UTXO
cache at the default -dbcache=450. During IBD this caused frequent
full-cache flushes (multi-second stalls every few thousand blocks).

Switch to Bitcoin Core's allocation: block tree DB gets 1/8 of the
total, capped at 2 MiB without txindex or 1024 MiB with txindex /
insightexplorer. Coin DB cache is capped at 8 MiB. The remainder goes
to the in-memory UTXO set.

At -dbcache=450 with all indexes on, the UTXO cache grows from ~76 MiB
to ~386 MiB; at -dbcache=2000 it reaches ~1.7 GiB, which eliminates
most flush stalls during IBD.

* Implement BIP 152 low-bandwidth mode (peer-initiated compact blocks)

PR #266 shipped high-bandwidth compact block relay (unsolicited
cmpctblock messages to up to three peers). Low-bandwidth mode lets a
peer request a compact block on demand via getdata, saving bandwidth
for edge peers (light clients, metered connections) that haven't opted
into high-bandwidth announcements.

Adds MSG_CMPCT_BLOCK as a new inv type, handled by ProcessGetData.
When a block is within MAX_CMPCTBLOCK_DEPTH (5 blocks) of the tip, the
server responds with a cmpctblock message; older blocks fall back to
sending a full block because the requester is unlikely to have the
mempool state needed to reconstruct it.

Type number 11 is used instead of the BIP 152 canonical value of 4
because type 4 is already allocated to "spork" on the Flux network.
IsFluxnodeType() is tightened to an explicit [4, 10] range so the new
type isn't misrouted.

* Early-return CheckForExpired/UndoExpiredStartTx when tracker is empty

CheckForExpiredStartTx and CheckForUndoExpiredStartTx are called on
every ConnectBlock/DisconnectBlock. For most of chain history the
relevant trackers (mapStartTxTracker, mapStartTxDOSTracker) are empty,
so the iterate-and-conditionally-copy loops do no work but still pay
for the cs_main-adjacent lock acquisition and the subsequent log
lines. Skip the body entirely when there is nothing to do.

The emptiness check runs under the lock, so it races correctly with
concurrent modifications.

* Cap debug.log size and enforce it at runtime

Previously ShrinkDebugFile() only ran at startup and only when -debug
was not set (the -shrinkdebugfile default was !fDebug). With debug=1
the log could grow without bound.

Changes:
- -maxdebugfilesize upper cap is now 10 GiB when -debug is enabled
  (so a long debug session has room to breathe) and 2 GiB otherwise.
  The 500 MB default is unchanged.
- Default -shrinkdebugfile to true so the cap is enforced even when
  debug logging is on.
- Make ShrinkDebugFile() safe to call at runtime by writing the kept
  tail to a temp file and atomically renaming; concurrent LogPrintStr
  writes land on the unlinked inode and fReopenDebugLog is set so the
  next log write reopens on the new file.
- Schedule ShrinkDebugFile() every 5 minutes so a long-running node
  with heavy debug output doesn't fill the disk between restarts.

* Harden BIP 152 compact block handling

Two fixes identified by an audit of fluxd's BIP 152 implementation
against Bitcoin Core:

1. Clean up per-peer compact-block in-flight state on disconnect.
   FinalizeNode previously iterated mapPartialBlocks but did nothing
   (a TODO comment acknowledged the gap). Peer-keyed cleanup is
   actually available via listCompactBlocksInFlight, which stores
   {NodeId, block hash} markers. Walk it, drop entries for the
   disconnecting peer, and drop the corresponding mapPartialBlocks /
   mapPartialBlocksTime entries only when no other peer is still
   working on the same block. Prevents a slow memory leak when peers
   churn mid-reconstruction.

2. Unify the compact block depth constant as MAX_CMPCTBLOCK_DEPTH,
   shared by the cmpctblock reception check and the new low-bandwidth
   MSG_CMPCT_BLOCK getdata response. Value is 100, sized to roughly
   Bitcoin Core's ~50-minute wall-clock window at fluxd's 30-second
   PON block spacing (Bitcoin uses 5 at 10-minute blocks). The prior
   reception value of 10 covered only ~5 minutes on PON, which is too
   narrow for typical mempool turnover.
@blondfrogs blondfrogs mentioned this pull request Apr 22, 2026
Open
9 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@blondfrogs