Skip to content

fix(sync): tolerate non-numeric shard and replica names in stale replica cleanup - #354

Merged
GrigoryPervakov merged 2 commits into
ClickHouse:mainfrom
melancholictheory:fix/stale-replica-query-cast
Sep 30, 2026
Merged

GrigoryPervakov merged 2 commits into
ClickHouse:mainfrom
melancholictheory:fix/stale-replica-query-cast

Conversation

@melancholictheory

Copy link
Copy Markdown
Contributor

Why

listStaleDatabaseReplicasQuery casts database_shard_name and database_replica_name from system.clusters with toInt32. That table lists every cluster the server knows about, not only the Replicated databases the operator numbered: the static clusters from remote_servers carry empty names, and a Replicated database created by a user with its own shard and replica names carries whatever the user chose. Neither parses as an integer, so the cast throws, CleanupDatabaseReplicas fails, SchemaInSync stays ReplicasNotCleanedUp, and reconciliation loops on that step without starting new pods. #349 reports it with Code: 32 on the empty string.

What

Use toInt32OrNull for both casts and drop the rows where either is NULL. Names the operator did not choose can no longer break the query, and the rest of the cleanup logic sees exactly the rows it saw before, since a row with a non-numeric name was never something it could act on.

The database_replica_name != '' guard is folded into the NULL filter: an empty name gives NULL too.

Testing

The commander integration suite gets a spec that creates a Replicated database with shard-a / replica-b names on one replica and runs CleanupDatabaseReplicas against the cluster. Without the fix that fails on the cast; with the fix it succeeds and leaves the database alone.

On the exact Code: 32 from the report: I could not make a static default cluster trigger it on 25.8.32.4 or 26.7.5.10, single node or three nodes, remote execution forced, short-circuit evaluation disabled or not; the database_replica_name != '' filter was applied before the cast every time. The report does not say which version hit it, so the test uses the non-numeric case, which is the same defect and reproduces on every version. The fix covers both because the cast is now total.

Related Issues

Fixes #349

…ica cleanup

listStaleDatabaseReplicasQuery cast database_shard_name and
database_replica_name from system.clusters with toInt32. That table lists
every cluster the server knows about: the static clusters from
remote_servers carry empty names and a Replicated database created by a
user carries whatever names the user chose. Neither parses as an integer,
the cast throws, CleanupDatabaseReplicas fails, and reconciliation loops
on SchemaInSync=ReplicasNotCleanedUp without starting new pods.

Cast with toInt32OrNull and drop rows where either name is NULL. Rows the
operator did not number were never something the cleanup could act on, so
the remaining logic sees exactly the rows it saw before. The
database_replica_name != '' guard folds into the NULL filter.

The commander integration suite gets a spec that creates a Replicated
database with shard-a/replica-b names and runs the cleanup; it fails on the
cast without this change.

Fixes ClickHouse#349
@GrigoryPervakov
GrigoryPervakov merged commit 146f699 into ClickHouse:main Sep 30, 2026
27 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CleanupDatabaseReplicas fails with Code 32 (Cannot parse Int32 from String) on empty database_shard_name

2 participants