Sync Perf Part 2#1436
Open
gilescope wants to merge 17 commits into
Open
Conversation
Signed-off-by: Giles Cope <gilescope@gmail.com>
…ns (#1434) * fix(node): tune autovacuum on db-sync hot tables for stable query plans Lower autovacuum_analyze_scale_factor from the postgres default of 0.1 to 0.01 on the cardano-db-sync tables midnight-node queries (block, tx, tx_out, tx_in, ma_tx_out, datum). The default 10% growth threshold means autoanalyze never fires for big append-heavy tables, leaving the planner on stale statistics and producing extreme worst-case plans (observed >400s queries on otherwise idle preview/preprod DBs) for the cnight-observation lookups. Applied alongside the existing index creation in create_cnight_observation_indexes, idempotent. Refs: #1298 Signed-off-by: Scott Buckel <scott.buckel@shielded.io> * change file Signed-off-by: Justin Frevert <justinfrevert@gmail.com> --------- Signed-off-by: Scott Buckel <scott.buckel@shielded.io> Signed-off-by: Justin Frevert <justinfrevert@gmail.com> Co-authored-by: Justin Frevert <justinfrevert@gmail.com>
Bumps the npm_and_yarn group with 1 update in the /util/toolkit-js directory: [postcss](https://github.com/postcss/postcss). Updates `postcss` from 8.5.8 to 8.5.10 - [Release notes](https://github.com/postcss/postcss/releases) - [Changelog](https://github.com/postcss/postcss/blob/main/CHANGELOG.md) - [Commits](postcss/postcss@8.5.8...8.5.10) --- updated-dependencies: - dependency-name: postcss dependency-version: 8.5.10 dependency-type: indirect dependency-group: npm_and_yarn ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…memory while syncing. Signed-off-by: Giles Cope <gilescope@gmail.com>
Signed-off-by: Giles Cope <gilescope@gmail.com>
Signed-off-by: Giles Cope <gilescope@gmail.com>
Signed-off-by: Giles Cope <gilescope@gmail.com>
Signed-off-by: Giles Cope <gilescope@gmail.com>
gilescope
commented
Apr 29, 2026
Signed-off-by: Squirrel <giles.cope@shielded.io>
Signed-off-by: Giles Cope <gilescope@gmail.com>
Signed-off-by: Giles Cope <gilescope@gmail.com>
Signed-off-by: Giles Cope <gilescope@gmail.com>
ozgb
reviewed
Apr 29, 2026
| midnight_metrics_opt.clone(), | ||
| 1000, | ||
| ); | ||
| let cnight_observation: Arc<dyn MidnightCNightObservationDataSource + Send + Sync> = { |
Contributor
There was a problem hiding this comment.
Would be neater to put this block into a new func
| // Keep the in-flight ledger state alive across consecutive | ||
| // apply_transaction calls so the substrate storage cache does not | ||
| // evict arena subtrees we'll need for the next tx in this block. | ||
| if let Ok(mut guard) = INTRA_BLOCK_LEDGER_PIN.lock() { |
Contributor
There was a problem hiding this comment.
Not fork-aware - we may be able to enhance this once #1443 is in - 1443 tracks persistence, so we could make this a pin of many ledger states where each state == a fork
| /// Returns immediately. Single-flight: concurrent triggers are no-ops. | ||
| fn maybe_kick_refresh(&self, target_end: u32) { | ||
| { | ||
| let mut g = match self.refresh_in_flight.lock() { |
Contributor
There was a problem hiding this comment.
Replace this block with:
if self.refresh_in_flight.try_lock().is_err() {
return;
}The mutex can just be Mutex<()>
| BulkCachedCNightObservationDataSource, bulk_pull, | ||
| }; | ||
|
|
||
| // Bulk-read is gated on `chainspec_cnight_genesis` — without it we |
Contributor
There was a problem hiding this comment.
You should be able to get these values from the chainspec rather than the cnight-genesis file
| ); | ||
| let t0 = std::time::Instant::now(); | ||
| let events = | ||
| bulk_pull(&cnight_observation_pool, &cnight_addresses, 0, bulk_end_block) |
Contributor
There was a problem hiding this comment.
This is a lot of data to pull into memory - could we store some on disk? Ideally in the parity-db backend
Alternatively - don't pull in all the data here, and rely on the refresh window instead
|
|
||
| observed_async_trait!( | ||
| impl MidnightCNightObservationDataSource for BulkCachedCNightObservationDataSource { | ||
| async fn get_utxos_up_to_capacity( |
Contributor
There was a problem hiding this comment.
It would be good to run equivalence tests vs the standard CNightObservationDataSource here - CI tests against a db-sync snapshot/internal db-sync instance? Could be a mini-snapshot of a few hundred cardano blocks
Signed-off-by: Giles Cope <gilescope@gmail.com>
Signed-off-by: Giles Cope <gilescope@gmail.com>
Signed-off-by: Giles Cope <gilescope@gmail.com>
Signed-off-by: Giles Cope <gilescope@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview
feat: syncs to tip of mainnet from genesis in 3h 19m (572k blocks, 48bps). (needs tidying)
🗹 TODO before merging
giles-pre-verifierbranch, so far the juice doesn't warrant the squeeze)📌 Submission Checklist
git commit -s) for the DCO🧪 Testing Evidence
🔱 Fork Strategy
Links