Repository navigation
fix(sync): tolerate non-numeric shard and replica names in stale replica cleanup - #354
Merged
GrigoryPervakov merged 2 commits intoSep 30, 2026
Conversation
…ica cleanup listStaleDatabaseReplicasQuery cast database_shard_name and database_replica_name from system.clusters with toInt32. That table lists every cluster the server knows about: the static clusters from remote_servers carry empty names and a Replicated database created by a user carries whatever names the user chose. Neither parses as an integer, the cast throws, CleanupDatabaseReplicas fails, and reconciliation loops on SchemaInSync=ReplicasNotCleanedUp without starting new pods. Cast with toInt32OrNull and drop rows where either name is NULL. Rows the operator did not number were never something the cleanup could act on, so the remaining logic sees exactly the rows it saw before. The database_replica_name != '' guard folds into the NULL filter. The commander integration suite gets a spec that creates a Replicated database with shard-a/replica-b names and runs the cleanup; it fails on the cast without this change. Fixes ClickHouse#349
GrigoryPervakov
approved these changes
Sep 30, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
listStaleDatabaseReplicasQuerycastsdatabase_shard_nameanddatabase_replica_namefromsystem.clusterswithtoInt32. That table lists every cluster the server knows about, not only the Replicated databases the operator numbered: the static clusters fromremote_serverscarry empty names, and a Replicated database created by a user with its own shard and replica names carries whatever the user chose. Neither parses as an integer, so the cast throws,CleanupDatabaseReplicasfails,SchemaInSyncstaysReplicasNotCleanedUp, and reconciliation loops on that step without starting new pods. #349 reports it withCode: 32on the empty string.What
Use
toInt32OrNullfor both casts and drop the rows where either isNULL. Names the operator did not choose can no longer break the query, and the rest of the cleanup logic sees exactly the rows it saw before, since a row with a non-numeric name was never something it could act on.The
database_replica_name != ''guard is folded into theNULLfilter: an empty name givesNULLtoo.Testing
The commander integration suite gets a spec that creates a Replicated database with
shard-a/replica-bnames on one replica and runsCleanupDatabaseReplicasagainst the cluster. Without the fix that fails on the cast; with the fix it succeeds and leaves the database alone.On the exact
Code: 32from the report: I could not make a staticdefaultcluster trigger it on 25.8.32.4 or 26.7.5.10, single node or three nodes, remote execution forced, short-circuit evaluation disabled or not; thedatabase_replica_name != ''filter was applied before the cast every time. The report does not say which version hit it, so the test uses the non-numeric case, which is the same defect and reproduces on every version. The fix covers both because the cast is now total.Related Issues
Fixes #349