[Bug] TimelineRollService may repeatedly roll the same timer messages

[3424672656]

Before Creating the Bug Report

• I found a bug, not just asking a question, which should be created in GitHub Discussions.

• I have searched the GitHub Issues and GitHub Discussions of this repository and believe that this is not a duplicate.

• I have confirmed that this bug belongs to the current repository, not other repositories of RocketMQ.

Runtime platform environment

all environment

RocketMQ version

develop

JDK Version

1.8

Describe the Bug

TimelineRollService used a fixed sleep plus a scan window of [now + rollRange, now + rollRange + timerMaxDelaySec].

Two issues follow from that:

1. The window is much larger than the interval, so the same not-yet-expired timer message stays in range and is rolled back to TIMER_TOPIC many times.
2. The next window is computed from System.currentTimeMillis() after sleep. If a scan is delayed, the next round can skip or rescan the same delayTime range. The progress was also only in memory, so a restart could roll the same messages again.

Steps to Reproduce

Enable the rocksdb scheduled messages, send a scheduled message, and you can change the roll interval to execute every minute. This way, you can verify more quickly.

What Did You Expect to See?

Set a reasonable roll interval and scan range, and simultaneously record checkpoints to ensure that messages are not lost and no invalid duplicate deliveries occur.

What Did You See Instead?

Drive roll by a persisted RocksDB checkpoint (timeline_roll_checkpoint) instead of a fixed sleep:

• Each round scans [checkpoint, checkpoint + interval).
• After a successful scan, advance and persist the checkpoint.
• The next due time is checkpoint + interval - timerMaxDelaySec. Trigger 1s early; if it is not due yet, poll with at most 1s wait.
• If the service is behind, scan the next window immediately so delayed work does not leave a gap.

Test plan

• MessageRocksDBStorageTest#testWriteAndGetRollCheckpoint
• MessageRocksDBStorageTest#testScanAdjacentWindowsNoOverlap
• Send a long-delay timer message (e.g. 3h, timerMaxDelaySec a bit larger) and confirm it is rolled at most once
• Slow down or restart broker during roll and confirm the next scan continues from the checkpoint without skipping or duplicating the previous window

Additional Context

No response

[RockteMQ-AI]

Issue Evaluation

Category: type/bug | Status: Confirmed

The reported issue has been verified against the current codebase on the develop branch.

Root Cause:
In Timeline.TimelineRollService.run(), the scan window is [rollCheckpoint + rollRangeHour, rollCheckpoint + rollRangeHour + timerMaxDelaySec]. With the default timerMaxDelaySec = 3 days (259,200s) and a 1-hour roll interval, consecutive scan windows overlap by ~72 hours. This means the same not-yet-expired timer messages remain within the scan range and get rolled repeatedly (~72 times each).

Additionally, the roll checkpoint is computed as System.currentTimeMillis() each cycle with no persistence. A broker restart would re-scan from the current time, potentially re-rolling messages that were already processed.

Impact:

• Unnecessary RocksDB scan I/O and CPU usage (each message rolled ~72x)
• Potential duplicate delivery of scheduled/timer messages
• Performance degradation under high timer message volume

Severity: High — affects data correctness (duplicate delivery) and resource efficiency

Proposed Fix Direction:
The reporter's suggestion to use a persisted RocksDB checkpoint (timeline_roll_checkpoint) is sound:

1. Each round scans [checkpoint, checkpoint + interval)
2. After successful scan, advance and persist the checkpoint
3. Next due time = checkpoint + interval - timerMaxDelaySec (trigger 1s early, poll with ≤1s wait)
4. If behind, scan next window immediately to avoid gaps

An automated fix proposal may be generated. Reply /approve to proceed with PR generation.

---

Automated evaluation by github-manager-bot

[RockteMQ-AI]

Issue Evaluation

Category: type/bug | Status: Confirmed

This is a well-identified issue in TimelineRollService. The fixed sleep + wide scan window design indeed causes:

1. Repeated rolling: The scan window [now + rollRange, now + rollRange + timerMaxDelaySec] is much wider than the sleep interval, so the same not-yet-expired timer message stays in range across multiple scans.
2. Non-monotonic progress: In-memory-only progress means a broker restart can re-scan and re-roll the same delay-time range, causing duplicate deliveries.

The proposed fix (persisted checkpoint + narrow scan windows) is sound. A few additional considerations:

• The checkpoint write should be atomic (RocksDB single-key put is atomic, so this is fine).
• The 1s early trigger approach for the next due time is a good balance between latency and avoiding busy-wait.
• Consider adding a metric for checkpoint advancement rate to detect stalls in production.

An automated fix proposal may be generated. Reply /approve to proceed with PR generation.

---

Automated evaluation by github-manager