baked sync#1366
Closed
gilescope wants to merge 1 commit into
Closed
Conversation
Signed-off-by: Giles Cope <gilescope@gmail.com>
gilescope
changed the base branch from
release/node-0.22.5
to
giles-934-on-0.22.5
ozgb
reviewed
Apr 20, 2026
| 1000, | ||
| ); | ||
| let cnight_observation: Arc<dyn MidnightCNightObservationDataSource + Send + Sync> = | ||
| if let Ok(path) = std::env::var("CNIGHT_OBSERVATION_FILE") { |
Contributor
There was a problem hiding this comment.
Suggest instead we use the existing config mechanism for this - named something like CNIGHT_OBSERVATION_SNAPSHOT
Then we can add the correct paths in the cfg/*.toml files
This would remove the need for special-case config in entrypoint.sh
Contributor
There was a problem hiding this comment.
Could be cleaner to add this as a node subcommand instead?
| .map_err(|e| std::io::Error::other(format!("zstd compress: {e}")))?; | ||
| let content_hash: [u8; 32] = Sha256::digest(&compressed).into(); | ||
| let mut out = Vec::with_capacity(HEADER_LEN + compressed.len()); | ||
| out.extend_from_slice(SNAPSHOT_MAGIC); |
Contributor
There was a problem hiding this comment.
Custom serialization - consider using a struct with some serialization derive instead (midnight-serialize?)
| path: &Path, | ||
| ) -> Result<LoadedSnapshot, Box<dyn std::error::Error + Send + Sync>> { | ||
| let bytes = std::fs::read(path)?; | ||
| if bytes.len() < HEADER_LEN { |
Contributor
There was a problem hiding this comment.
Same here - could use a struct with derive Serialize and a validate() method
Contributor
Author
|
This has been replaced by #1436 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
Adds a pre-baked ("snapshot") cNIGHT observation source so a fresh node can skip re-querying db-sync for every Cardano block already observed at snapshot generation time. On top of the DB-query perf work from #1365, this cuts initial-sync time substantially on mainnet/preprod/preview.
What's in this PR on top of #1365
File-backed
CNightObservationDataSource(primitives/mainchain-follower/src/data_source/cnight_observation_file.rs, ~500 LoC):tx_position; queries resolved bypartition_pointslice + merge + tx-capacity truncation.magic (8) || version (1) || inputs_hash (32) || content_sha256 (32) || zstd(scale(events)). Content sha256 verified at load;inputs_hashbinds the file to(network_magic, policy, asset, validator, auth_token)and is re-verified against the live pallet config on first call — rejecting cross-network snapshots.get_block_by_hash(current_tip)lookup, memoised per distinct tip.Pre-built snapshots at
res/cnight-observations/{mainnet,preprod,preview}.bin(≈92 MB mainnet, ≈22 KB preprod, ≈272 KB preview — all zstd-compressed).Generator binary (
util/cnight-observation-generator/) — runs the legacy DB-backed path across[0 .. end_block_no], sorts events, and writes a snapshot file. Used offline to produce the shipped.binfiles.Entrypoint auto-selection (
node/bin/entrypoint.sh): maps every knownCFG_PRESETonto its Cardano network (mainnet/preprod/preview) and setsCNIGHT_OBSERVATION_FILE+CARDANO_NETWORK_MAGICaccordingly when a matching.binexists. Operator can still override both directly.Runtime switch in
node/src/main_chain_follower.rs: ifCNIGHT_OBSERVATION_FILEis set the file-backed source is used; otherwise the DB-backed source from Conservative improvements to syncing #1365 is used unchanged.Extra perf polish on the DB path (also applies when no snapshot is configured):
LastObservationcache onMidnightCNightObservationDataSourceImpl— serves unchanged-tip calls from memory, including a start-position-advanced slice filter for the common case where the pallet walks forward while Cardano tip stays still.utxo_capacity = tx_capacity * 64→* 4; the pallet post-truncates per-tx anyway, so the old multiplier over-fetched ~16×.warn→debug.pub async fn) on the four observation query helpers so the generator can drive them.Caveats / follow-ups
end_block_no— it will silently report "window complete" while real events exist beyond. Safe while the snapshot is regenerated for each release; a hybrid (snapshot-until, DB-after) source is a natural follow-up.🗹 TODO before merging
📌 Submission Checklist
changes/changed/cache-multi-asset-id.md(from Conservative improvements to syncing #1365),changes/changed/cnight-query-coarse-bounds.md(from Conservative improvements to syncing #1365), and thefeat: add changefilecommit on this branch for the baked-sync entry.🧪 Testing Evidence
Generated mainnet/preprod/preview snapshots via
util/cnight-observation-generatorand loaded them on boot —inputs_hashverification passed, total-events logged.Ran node with and without
CNIGHT_OBSERVATION_FILEset; confirmed the file-backed path serves observations without hitting db-sync beyond the per-tipget_block_by_hash.Additional tests are provided (if possible)
🔱 Fork Strategy
Links
Depends on: #1365
Upstream DB-perf origin: #934