Skip to content

fix(schedule): each due time fires once, on the leader, up to 5 minutes late - #671

Merged
bmscomp merged 5 commits into
mainfrom
fix/schedules-fire-once
Nov 15, 2026
Merged

bmscomp merged 5 commits into
mainfrom
fix/schedules-fire-once

Conversation

@bmscomp

@bmscomp bmscomp commented Nov 15, 2026

Copy link
Copy Markdown
Owner

J4c of the resilient-quorum plan (plans/j4-resilient-quorum.md §4.7). It fixes F3 from the failure study: a schedule due time was skipped at takeover.

The bug

Both evaluators matched the cron against the current minute every 60 s, under a 55 s PostgreSQL lease held by a per-process id. In s12, an every-minute schedule fired once a minute except at 10:37, right after the leader was killed at 10:36:52: the dead process's lease blocked the new leader's next tick, and nothing caught up. Ticks 60 s apart could also straddle a minute and skip it with no failure at all.

What changes

  • Each due time is claimed in PostgreSQL.
    • V31 adds last_due to scheduled_test_runs and disruption_schedules.
    • ScheduleFirings.claim is one UPDATE … SET last_due = :due WHERE id = :id AND (last_due IS NULL OR last_due < :due). Only the member whose update changed a row fires that due time, so a paused leader and the one that took over fire it once between them.
    • A database error fails closed: the due time does not fire.
  • A late tick still fires.
    • Both evaluators tick every 10 s.
    • DueTimes.toFire picks the latest due time after the latest of created_at, last_due and last_run_at, up to 5 minutes late. When several were missed, only the latest starts a run.
    • last_run_at covers rows from before V31, and firings by the previous version during a rolling upgrade.
  • A refused start still counts as the firing, logged with its reason as before.
  • Test runs are traceable. A run a test schedule starts carries the Idempotency-Key schedule:<id>:<due>, findable with GET /api/tests?clientKey=.
  • Turning a schedule on or changing its cron passes over the earlier due times. A PUT that does either moves last_due to now, before it saves, so re-enabling a schedule does not fire a due time it missed while off. A rename that re-sends enabled: true passes over nothing.
  • JPA never writes last_due (insertable = false, updatable = false), so a PUT that loaded a schedule before a claim cannot write the old value back. It is not in API responses.
  • The disruption evaluator ticks with SKIP. A tick runs each plan it fires to its end. Without SKIP, a tick started meanwhile fired a schedule due the same minute into the target the first plan held, and spent its due time on a REJECTED report.
  • The lease is gone. SchedulerLeaseService, its two callers and its IT are removed. The scheduler_leases table stays one release, for pods of the previous version during a rolling upgrade.
  • Docs. The API reference's Scheduling section and its drain note follow, and three MCP caveat refs follow the scheduler lines that moved.

Proof

  • On Kind (three members, image built from this branch):
    • s12, an every-minute schedule with the leader killed twice, started exactly one run for each of the 7 due times between the schedule's creation and its deletion, each within 9 s. That includes the minutes right after both kills.
    • s09b, s08b, s15, s02, s10 and s11 give the same results as the J4b image.
  • New tests:
    • DueTimesTest, which also takes over the old matcher's cron cases;
    • ScheduleFiringsIT on PostgreSQL: 8 concurrent claims of one due time win once, for both tables; a stale save keeps the claim;
    • TestSchedulerFiringTest: one run across ticks; a refused start counts; a missed minute fires after a takeover; re-enabling waits for the next due time; a tick during the PUT fires nothing; a due time the previous version fired is not fired again;
    • DisruptionSchedulerTickTest.
    • Each new test was watched failing before its code.
  • Local mirror of CI:
    • ./mvnw verify (1922 unit, 104 IT, Spotless);
    • go test ./... in cli;
    • scripts/check-book-style.sh;
    • the native image build with scripts/native-smoke-test.sh.

Known limits

  • During the rolling upgrade from the lease version, an old-version leader and a new-version leader can each fire the same minute once. last_run_at closes most of that window, but not a firing still starting when the new version ticks.
  • A crash between the claim and the start loses that one firing.
  • A long disruption tick evaluates later schedules with the now and the enabled list it started with. This predates the change.

…ing, within 5 minutes

A tick that comes late, after a takeover or a restart, still fires the
due time it missed, up to 5 minutes late; when it missed several, only
the latest starts a run. The old matcher's cron cases move into
DueTimesTest.
…st_due

V31 adds last_due to scheduled_test_runs and disruption_schedules.
ScheduleFirings.claim moves it forward to a due time in one UPDATE ...
WHERE last_due IS NULL OR last_due < :due, and says whether this call
moved it: two members that both believe they lead fire a due time once.

JPA never writes last_due, so a PUT that loaded a schedule before a
claim and commits after it cannot write the old value back. It is not
in API responses.
…es late; the scheduler lease is gone

Both evaluators matched the cron against the current minute under a 55 s
PostgreSQL lease held by a per-process id. A leader killed at 10:36:52
left its lease to block the new leader's next tick, and the 10:37 run
never started (s12 of the failure study); ticks 60 s apart could also
straddle a minute and skip it.

The evaluators now tick every 10 s. For each enabled schedule DueTimes
picks the latest due time since last_due (or since the schedule was
created), up to 5 minutes late, and ScheduleFirings claims it; only the
member whose claim moved last_due starts the run. A refused start still
counts as the firing. A test run carries the Idempotency-Key
schedule:<id>:<due>.

A PUT that turns a schedule on or changes its cron passes over the due
times before it, so re-enabling does not fire at once. SchedulerLeaseService
and its callers are removed; scheduler_leases stays one release, for pods
of the previous version during a rolling upgrade.
The Scheduling section says each due time of a test or disruption
schedule fires once, on the leader, within 10 s or up to 5 minutes late
after a restart, a takeover or a drain, that only the latest missed one
starts a run, that a PUT turning a schedule on passes over earlier due
times, and that a test run carries schedule:<id>:<due> as its
Idempotency-Key. The drain note no longer says a due time inside a
drain is skipped. Three MCP caveat refs follow the scheduler lines
that moved.
…UT race, and upgrades

- The disruption evaluator ticks with SKIP. A tick runs each plan it
  fires to its end; a tick started meanwhile fired a schedule due the
  same minute into the target the first plan held, and spent its due
  time on a REJECTED report.
- A test-schedule PUT that turns a schedule on or changes its cron
  passes over the earlier due times before it saves, so a tick between
  the two no longer fires one of them.
- Due times are counted after the latest of created_at, last_due and
  last_run_at. A row from before V31 has no last_due, so the first tick
  after an upgrade re-fired the due time the previous version had just
  fired, up to 5 minutes back; last_run_at also covers firings by the
  previous version during a rolling upgrade.
@bmscomp
bmscomp merged commit 4089e47 into main Nov 15, 2026
71 of 73 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant