Repository navigation
fix(schedule): each due time fires once, on the leader, up to 5 minutes late - #671
Merged
Merged
Conversation
…ing, within 5 minutes A tick that comes late, after a takeover or a restart, still fires the due time it missed, up to 5 minutes late; when it missed several, only the latest starts a run. The old matcher's cron cases move into DueTimesTest.
…st_due V31 adds last_due to scheduled_test_runs and disruption_schedules. ScheduleFirings.claim moves it forward to a due time in one UPDATE ... WHERE last_due IS NULL OR last_due < :due, and says whether this call moved it: two members that both believe they lead fire a due time once. JPA never writes last_due, so a PUT that loaded a schedule before a claim and commits after it cannot write the old value back. It is not in API responses.
…es late; the scheduler lease is gone Both evaluators matched the cron against the current minute under a 55 s PostgreSQL lease held by a per-process id. A leader killed at 10:36:52 left its lease to block the new leader's next tick, and the 10:37 run never started (s12 of the failure study); ticks 60 s apart could also straddle a minute and skip it. The evaluators now tick every 10 s. For each enabled schedule DueTimes picks the latest due time since last_due (or since the schedule was created), up to 5 minutes late, and ScheduleFirings claims it; only the member whose claim moved last_due starts the run. A refused start still counts as the firing. A test run carries the Idempotency-Key schedule:<id>:<due>. A PUT that turns a schedule on or changes its cron passes over the due times before it, so re-enabling does not fire at once. SchedulerLeaseService and its callers are removed; scheduler_leases stays one release, for pods of the previous version during a rolling upgrade.
The Scheduling section says each due time of a test or disruption schedule fires once, on the leader, within 10 s or up to 5 minutes late after a restart, a takeover or a drain, that only the latest missed one starts a run, that a PUT turning a schedule on passes over earlier due times, and that a test run carries schedule:<id>:<due> as its Idempotency-Key. The drain note no longer says a due time inside a drain is skipped. Three MCP caveat refs follow the scheduler lines that moved.
…UT race, and upgrades - The disruption evaluator ticks with SKIP. A tick runs each plan it fires to its end; a tick started meanwhile fired a schedule due the same minute into the target the first plan held, and spent its due time on a REJECTED report. - A test-schedule PUT that turns a schedule on or changes its cron passes over the earlier due times before it saves, so a tick between the two no longer fires one of them. - Due times are counted after the latest of created_at, last_due and last_run_at. A row from before V31 has no last_due, so the first tick after an upgrade re-fired the due time the previous version had just fired, up to 5 minutes back; last_run_at also covers firings by the previous version during a rolling upgrade.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
J4c of the resilient-quorum plan (
plans/j4-resilient-quorum.md§4.7). It fixes F3 from the failure study: a schedule due time was skipped at takeover.The bug
Both evaluators matched the cron against the current minute every 60 s, under a 55 s PostgreSQL lease held by a per-process id. In s12, an every-minute schedule fired once a minute except at 10:37, right after the leader was killed at 10:36:52: the dead process's lease blocked the new leader's next tick, and nothing caught up. Ticks 60 s apart could also straddle a minute and skip it with no failure at all.
What changes
last_duetoscheduled_test_runsanddisruption_schedules.ScheduleFirings.claimis oneUPDATE … SET last_due = :due WHERE id = :id AND (last_due IS NULL OR last_due < :due). Only the member whose update changed a row fires that due time, so a paused leader and the one that took over fire it once between them.DueTimes.toFirepicks the latest due time after the latest ofcreated_at,last_dueandlast_run_at, up to 5 minutes late. When several were missed, only the latest starts a run.last_run_atcovers rows from before V31, and firings by the previous version during a rolling upgrade.schedule:<id>:<due>, findable withGET /api/tests?clientKey=.PUTthat does either moveslast_dueto now, before it saves, so re-enabling a schedule does not fire a due time it missed while off. A rename that re-sendsenabled: truepasses over nothing.last_due(insertable = false, updatable = false), so a PUT that loaded a schedule before a claim cannot write the old value back. It is not in API responses.SchedulerLeaseService, its two callers and its IT are removed. Thescheduler_leasestable stays one release, for pods of the previous version during a rolling upgrade.Proof
DueTimesTest, which also takes over the old matcher's cron cases;ScheduleFiringsITon PostgreSQL: 8 concurrent claims of one due time win once, for both tables; a stale save keeps the claim;TestSchedulerFiringTest: one run across ticks; a refused start counts; a missed minute fires after a takeover; re-enabling waits for the next due time; a tick during the PUT fires nothing; a due time the previous version fired is not fired again;DisruptionSchedulerTickTest../mvnw verify(1922 unit, 104 IT, Spotless);go test ./...incli;scripts/check-book-style.sh;scripts/native-smoke-test.sh.Known limits
last_run_atcloses most of that window, but not a firing still starting when the new version ticks.nowand the enabled list it started with. This predates the change.