Feat/blockstore base and db info earliest#31
Open
dntjd1097 wants to merge 10 commits into
Open
Conversation
After pruning, set the blockstore base to the actual lowest block meta
("H:") height that still exists, instead of blindly using pruneHeight+1.
This ensures cometbft's LoadBaseMeta(base) finds a real block so /status
reports earliest_block_* instead of empty values (which occur when base
is 0 or points at a deleted block meta).
Scans the "H:" prefix for the minimum height (heights are ascii without
zero-padding, so they don't sort numerically). Falls back to
pruneHeight+1 if the scan finds nothing or errors. Shared by all chains
using pruneBlockAndStateStore; behavior is unchanged where the meta at
pruneHeight+1 exists, just verified.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
SnapshotAndRestoreApp previously removed application.db immediately after creating the snapshot, then restored in place — a failed or wrong restore (e.g. an incompatible hashing stack) left no original to fall back to. With --verify-after-prune (default on), restore now happens into a temporary DB on the same filesystem, the restored per-store roots are verified against the pre-snapshot commit info, and only on success is the original removed and the verified DB atomically renamed into place. Any failure before the swap leaves the original application.db intact; a mid-swap rename failure preserves the verified DB at a logged path for manual recovery. With --verify-after-prune=false the previous destructive in-place restore is kept (lower peak disk, no fallback). The safe path costs transient disk equal to the original plus the restored DB simultaneously. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Setting the base from the lowest surviving block meta could land on a stale orphan height: range deletion uses an exclusive upper bound, so the all-nines digit boundaries (9999999, 999999, ...) are skipped and survive, making the scan report base=9999999 even though real blocks are far higher. That misrepresents retention. Drop the base override entirely (both the lowest-surviving scan added earlier and the prior pruneHeight+1). cometbft maintains its own base during block pruning; leave it to the node. Removes findLowestBlockHeight. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Delete SetBlockStoreStateBase and its blockStoreKey var (now uncalled after the base override was removed), the cmtstore import they were the only user of, and the long-unused appKeyInfos var. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Document standard vs celestia-dedicated builds (make build / build-celestia, Docker, raw go build), the full current flag set including --verify-after-prune and --iavl-disable-fastnode, the two application pruning strategies (PruneAppState vs SnapshotAndRestoreApp), the verify/verify-before-destroy guard, per-chain config, and the celestia rationale. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
After offline pruning, CometBFT's /status returned empty earliest_block_* values (base=0/"") because the blockstore base either pointed at a deleted block meta or was never updated. LoadBaseMeta(base) then returns nil and all earliest_* fields stay zero. Set the blockstore base to the retain floor (pruneHeight+1) at the end of pruneBlockAndStateStore. This differs from the previously reverted approach (a7aca84) in three ways that avoid its failure modes: - Use the retain floor instead of the global-minimum surviving "H:" key. deleteHeightRange leaves digit-boundary all-nines metas (e.g. H:9999999) as orphans because the per-digit range iterator end bound is exclusive; the old global-min scan picked those orphans and set base=9999999, misreporting retention. - Verify the meta actually exists (Has "H:<height>"), scanning upward from the retain floor, so LoadBaseMeta(base) is guaranteed a real block. - Write with SetSync (fsync) and log a read-back, so the base survives process exit instead of being lost by an async Set. Failure is non-fatal: a wrong/absent base only affects /status earliest_*, not node operation, so we log and continue. Also add basefix/, a standalone diagnose/repair tool for nodes whose base is already corrupted from an earlier prune, so the base can be fixed without a full re-prune. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
DbState only read state.db (latest state + the genesis initial_height), so
db-info never showed the post-prune base block. initial_height stays 1 because
it is the chain's genesis height, not the retained base.
Read the blockstore base (BlockStoreState.Base) and load its block meta
("H:<base>") to populate earliest_block_height / earliest_block_hash /
earliest_app_hash / earliest_block_time, mirroring CometBFT /status. The lookup
is best-effort and non-fatal: when the base is unset or its meta was pruned it
leaves the fields empty/zero, the same condition under which /status reports
empty earliest_* values.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- "Block & state stores" now documents that base is rewritten to the retain floor (with meta-existence check, fsync, and the orphan-avoidance rationale) instead of being left to the node. - db-info JSON example and notes now cover the earliest_block_* fields and clarify that initial_height is the genesis height, not the post-prune base. - Add a basefix section documenting the standalone diagnose/repair tool and the hard limit that pruned blocks are unrecoverable. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The blockstore base is now set automatically at the end of pruning (setBlockStoreBase in generic_chain.go), so the standalone basefix tool is redundant: an already-corrupted base is repaired simply by re-running prune. Drop the tool and its README section to keep one code path. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.