Hashing takes a long time

[Slach]

Discussed in #1445

^{Originally posted by ruslanen July 1, 2026}
@Slach hi,

We use ClickHouse (different versions) + clickhouse-backup in production, on setups with a lot of
tables (for example: ~50K MergeTree + ~500K Join, ~954 GiB).

I noticed that hashing takes a really long time during backup.
Looks like fetching hash_of_all_files from system.parts runs together with FREEZE and they contend on some mutex inside ClickHouse - FREEZE jumps from ~100ms to ~5000ms per table because of it.

I tried just skipping that lookup and the difference is huge:

• with hashing: FREEZE ~5029ms, create ~9.5h
• without: FREEZE ~116ms, create ~26min

I know hash_of_all_files isn't useless - incremental create --diff-from-remote uses it to
dedup parts against the previous full backup, so skipping it on full backups would quietly hurt
future increments.

That's why I only did it behind an opt-in flag, not automatically.
Concretely, what I did:

• added a config field clickhouse.skip_hash_of_all_files (env CLICKHOUSE_SKIP_HASH_OF_ALL_FILES);
• added a --skip-hash-of-all-files CLI flag to create / create_remote / watch;
• gated the existing lookup on it in AddTableToLocalBackup - one extra condition, so the default
  behavior is unchanged:

// before
useHashOfAllFiles := version >= 19011000
// after
useHashOfAllFiles := version >= 19011000 && !b.cfg.ClickHouse.SkipHashOfAllFiles

When the flag is off (default) everything works exactly as today; when it's on, the per-part
hash_of_all_files read from system.parts is skipped and the FREEZE contention goes away.

My two main questions:

1. Would you accept a PR with this config/flag as-is? If yes, which shape do you prefer:
   • just keep the opt-in flag and document the diff-from-remote caveat,
   • or keep the flag but warn/block when it's used with --diff-from-remote?
2. Or is there a better way to speed up the hashing itself - e.g. batch/defer the system.parts
   lookup so it doesn't fight with FREEZE - so no flag would be needed at all?

[Slach]

@ruslanen could you share
SELECT count(), quantiles(query_duration_ms)(0.5,0.99) AS t FROM system.query_log WHERE query_duration_ms > 0 AND query LIKE '%hash_of_all_files%' ?

[Slach]

FREEZE is not depents on hash_of_all_files
but maybe fetching hash_of_all_files could took time, but this happens per table

[ruslanen]

@ruslanen could you share SELECT count(), quantiles(query_duration_ms)(0.5,0.99) AS t FROM system.query_log WHERE query_duration_ms > 0 AND query LIKE '%hash_of_all_files%' ?

Sorry, I have already removed VM with large data
I can return later again

[Slach]

@ruslanen ok. please we need data for reproducing behavior

need to understand what exactly increase time for freeze and moving data parts hadlinks into backup folder

[Slach]

@ruslanen any news from your side?