Skip to content

Cassandra HLC sequence can regress at counter overflow under a forward clamp #465

Closed

Description

@LeeroyHannigan

HybridClock::generate_with_created_at in crates/storage-cassandra/src/stream_util.rs handles counter overflow (logical_counter > 999_999) by sleeping one millisecond and overwriting last_timestamp_ms with a fresh clock read. When the clock had been clamped forward by an earlier backwards step, the overwrite discards the clamp and the next sequence number is smaller than the previous one by the clamp width.

Reproduced in a unit test with last_timestamp_ms set 5 s ahead and the counter driven to the limit: the sequence before overflow had prefix 1792766145271, the one after had prefix 1792766140272. The per-record property created_at_ms <= hlc_ms still holds on the overflow record; the sequence order within the shard does not.

Reaching it needs a multi-second backwards wall-clock step followed by about a million writes inside the clamp window on one instance. The fix is to take the maximum of the re-read and the previous last_timestamp_ms in the overflow branch, with a unit test for overflow under a forward clamp.

Found while working on #463, which only moved the branch into the new method and did not change it.

Metadata

Metadata

Labels

backend:cassandraRelated to the Cassandra backend specificallybugSomething isn't working

Type

No type

Projects

No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions