Skip to content

CleanupDatabaseReplicas fails with Code 32 (Cannot parse Int32 from String) on empty database_shard_name #349

Closed
#354
Closed
CleanupDatabaseReplicas fails with Code 32 (Cannot parse Int32 from String) on empty database_shard_name#349
#354
Labels
potential bugTo be reviewed by developers and confirmed/rejected.

Description

@TheDJVG
TheDJVG
opened on Sep 26, 2026

Company or project name

No response

Describe what's wrong

With spec.settings.enableDatabaseSync: true (default), cluster reconciliation gets stuck in a loop on CleanupDatabaseReplicas:

type: SchemaInSync
status: "False"
reason: ReplicasNotCleanedUp
message: "Some stale replicas are not cleaned up"

Because the reconcile loop keeps failing and requeuing on this step, new pods never start.

Setting enableDatabaseSync: false bypasses the issue and lets the pods start immediately.

Does it reproduce on the most recent release?

Yes

How to reproduce

  1. Deploy a standard KeeperCluster and ClickHouseCluster with default settings (enableDatabaseSync left enabled).
  2. Scale up replicas (or wait for reconciliation against an existing pod).
  3. The operator runs CleanupDatabaseReplicas and ClickHouse throws Code 32.

Expected behavior

Reconciliation succeeds and allows new pods to start

Error message and/or stacktrace

 <Error> executeQuery: Code: 32. DB::Exception: Attempt to read after eof: Cannot parse Int32 from String, because value is too short: while executing 'FUNCTION toInt32(__table1.
  database_shard_name :: 0) -> toInt32(__table1.database_shard_name) Int32 : 16'. (ATTEMPT_TO_READ_AFTER_EOF) (in query: SELECT `__table1`.`cluster` AS `database`, toInt32(`__table1`.
  `database_shard_name`) AS `shard_id`, toInt32(`__table1`.`database_replica_name`) AS `replica_id`, `__table1`.`is_active` AS `is_active` FROM `system`.`clusters` AS `__table1` WHERE
  (`__table1`.`database_replica_name` != '') AND ((toInt32(`__table1`.`database_shard_name`) >= 1) OR (toInt32(`__table1`.`database_replica_name`) >= 1)))

Additional context

In internal/controller/clickhouse/commands.go, listStaleDatabaseReplicasQuery runs:

toInt32(database_shard_name) AS shard_id,
toInt32(database_replica_name) AS replica_id,

system.clusters contains static clusters from remote_servers (such as default), where database_shard_name is "". ClickHouse's filter transform evaluates toInt32("") on those rows before filtering, which throws Code: 32.

Activity

  1. TheDJVG
    added
    potential bugTo be reviewed by developers and confirmed/rejected.
    on Sep 26, 2026

  2. GrigoryPervakov commented on Sep 29, 2026

    @GrigoryPervakov
    GrigoryPervakov
    on Sep 29, 2026
    Member

    @TheDJVG the log messages you sent suggest that you run ClickHouse version <24.11, am I right?

  3. melancholictheory commented on Sep 29, 2026

    @melancholictheory
    melancholictheory
    on Sep 29, 2026
    Contributor

    Thanks for the precise pointer, the cast is the problem. Fix in #354: toInt32OrNull on both names and a NULL filter, so rows the operator did not number are skipped instead of blowing up the query.

    One honest note: I could not reproduce the Code: 32 on the empty string from a static default cluster on 25.8.32.4 or 26.7.5.10, in any combination of single or three nodes, forced remote execution, and short-circuit evaluation on or off; the != '' filter always ran first for me. Which ClickHouse version are you on? The regression test in the PR uses a Replicated database with non-numeric shard and replica names instead, which fails the same way on every version, and the fix covers both cases.

  4. TheDJVG commented on Sep 30, 2026

    @TheDJVG
    TheDJVG
    on Sep 30, 2026
    Author

    Yes, this single test instance is running 24.3.18.7. I did not notice this with any of the production clusters that were upgraded to 26.7 already.

    To be honest this instance should not have been on 24.3 and that happened because of something unrelated and that's why I'm not sure if the operator even supports 24.3, maybe not. That one is on me.

  5. GrigoryPervakov
    closed this as completedin 146f699on Sep 30, 2026
  6. pull
    added a commit that references this issue on Sep 30, 2026

    fix(sync): tolerate non-numeric shard and replica names in stale repl…

    146f699
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    potential bugTo be reviewed by developers and confirmed/rejected.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions