Skip to content

fix(faucet): take redelivery floors and positions from GitHub's times, not faucet's clock - #39

Merged
ThomasK33 merged 18 commits into
mainfrom
fix/redelivery-clock-26
Nov 16, 2026
Merged

ThomasK33 merged 18 commits into
mainfrom
fix/redelivery-clock-26

Conversation

@ThomasK33

@ThomasK33 ThomasK33 commented Nov 16, 2026 •

Copy link
Copy Markdown
Member

Why

The redelivery sweep promises that every failed delivery in GitHub's list ends recovered, given up, or inside a counted, logged gap. On main, faucet's clock sets the retention floor and the first-run lookback, and an empty list moves the position to GitHub's Date. Each of these can drop failures silently when the two clocks differ (#26). This PR gives each time its own source: attempt times for where a read stops, a checked page Date for where GitHub's retention ends, and the stored Date of the last head read for where a gap after a silence starts. Four review rounds by five models shaped the design and the code.

What changed

  • faucet/redelivertime.go adds a type per time role. A Date cannot become a read bound, and the scan uses no gap or bound from faucet's clock.
  • The store keeps a new Clear (the Date of the last head read). Migration 000008_redelivery_clear moves the old position into it, renames the position column, and raises the epoch, so a run of an older build fails loudly instead of writing a Date back.
  • A used Date that is missing, or more than 1 hour behind an attempt, fails the read. The README states the trust window (6 hours ahead, 1 hour behind) and the two risks outside it.

Scope

Tradeoffs

  • A missing Date fails the run instead of falling back to faucet's clock. The fallback would bring limit 1 back.
  • The migration resets the position instead of lowering it. Lowering it lost 6 to 30 failures once the old Date lead passed 6 hours. The reset costs one re-read of up to 24 hours and a few extra gaps.

Blast Radius

Every deployment with redelivery on runs migration 000008. An instance of an older build still running after it gets a failing /cron/redeliver (500) until it is replaced. A RedeliveryStore of your own must now keep Clear, and a run fails with a named error when it does not.

Verification

  • go test -race ./... with Postgres, golangci-lint (0 issues), sqlc and go mod tidy with no diff, and e2e in verify/ (234 of 234) all pass.
  • 23 new witness tests assert that no failure is lost silently. 7 of them lose failures silently on main.
  • A reviewer's random search, kept out of the tree, found no silent loss in 7,000 fake-store cases and 300 pgstore cases.

Closes #26


Generated with xum • Model: anthropic:claude-opus-5-5 • Thinking: high

…rsor

The redesign of the redelivery clocks (#26) takes its gap floors from
GitHub's Date header. Its tests need a GitHub whose Date leads, lags or
is missing, whose pages take time to arrive, and that refuses the cursor
of an attempt it has deleted. The fake GitHub had none of these.

- `fakeGitHub.date` gives the Date of each list response, by call and
  cursor. A zero time is a response without a Date.
- `redeliveryEnv.slowPages` moves the clock forward inside every list
  call, before GitHub reads its list.
- `fakeGitHub.refusesStaleCursors` answers 422 to a cursor whose attempt
  left GitHub's retention.
- `simSpec.setup` changes the env before the first delivery, and the
  simulation now follows GitHub's clock, so a test can set a skew.

No production code changes, and no existing test changes. One new test
pins what the three knobs do, because the tests of #26 lean on them.

Refs #26
…clear

A redelivery run kept one time, the top, for two jobs: the next read
bound and the old end of the next gap. Those jobs need opposite bounds
(#26). The top also held GitHub's Date after an empty list, and a Date
can run ahead of GitHub's list. This commit gives the stores a second
time, the clear, so that the scan can keep attempt times and Dates apart.
The scan starts to use it in the next commit.

- `RedeliveryScan.Clear` and `ScanStep.Clear`. A step replaces the stored
  clear, also with an earlier time: the store keeps the latest Date, not
  the highest, so one Date that ran ahead does not stay.
- Migration `000008_redelivery_clear` adds `clear_at`, moves the old top
  into it and resets the top. No row says whether its top is an attempt
  time or a Date, so the next run is a first run that knows when
  tracking began. The down migration gives an older build its position
  back.
- A changed hook resets the clear with the top, and the gaps of its open
  ranges start at `OlderAt - RedeliveryReadOverlap`, where the read of
  the range would have stopped (#36 did this for the scan's own gaps).
- `storetest.RunRedelivery` checks all of it, and the fake store and
  pgstore pass. A pgstore test runs the migration both ways.

Refs #26
…ot faucet's clock

The redelivery sweep promises that every failed delivery that GitHub lists
ends recovered, given up, or inside a counted gap. On main it broke that
promise in two ways (#26). A gap ended at faucet's clock less 72 hours, so
a clock behind GitHub's left a slice of deleted failures outside the gap,
and a slow page widened the slice. The first run's gap started on faucet's
clock, so a clock ahead of GitHub's left the oldest listed attempts unread
and uncounted. And an empty list moved the position to GitHub's Date, which
can run ahead of GitHub's list and put the next failures below the bound.

One time served two roles that need opposite bounds. The scan now keeps
three kinds of time apart by type (`faucet/redelivertime.go`), so a Date
cannot become a read bound:

- An attempt time (`delivered_at`, the top, a range's ends) is the only
  source of a read bound. An empty list leaves the top as it is.
- A page's Date gives where GitHub's retention has ended, and only after a
  check: a missing Date, or one more than `RedeliveryDateSlack` (1h) behind
  the newest attempt known, fails the read. The scan checks it on the first
  page and on a page that ends the list, and nowhere else. A gap ends at
  `Date + 1s - 72h + slack`.
- The clear, the stored Date of the last head read, gives gap starts: the
  head's gap starts at the later of `top - overlap` and
  `clear - RedeliveryDateMargin (6h) - overlap`.

A first run reads 24 hours below the newest attempt and records
`[newest - 72h, newest - 24h]` as its gap. After an empty first run or the
migration it also reads back to `clear - margin`, and its gap reaches up to
what expired since. A range whose new end is below
`min(head Date, clear) - 72h` closes unread as a gap, so one Date far ahead
closes nothing, and a range whose page fails ends this way.

Tests. Each case of #26 and of the three design reviews is a test in
`redeliver_gaps_test.go` or `redeliver_sim_test.go` that asserts no silent
loss: a clock that lags or leads, a Date that leads, a slow page, one Date
far ahead (the clear, and open ranges), an empty first run or the migration
followed by an outage of 30 and 80 hours, a migrated top that was a Date
ahead, range pages without a Date, a whole-second Date as far behind as the
slack, a refused cursor, a changed hook, a quiet hook over 10 days, and the
outage after which a quiet hook logs a gap (64h55m).

Four tests pinned main's first run and empty list. An empty list now
records no gap, the lookback counts from the newest attempt, and a gap ends
at the oldest attempt that GitHub still lists. The cron tests' fake hook
sends a Date, and the e2e lookback test moves its old delivery below the
new bound.

Refs #26
…and the migration

The README still described main: a position that is GitHub's Date after
an empty list, a first reply of `gaps: 1` for a new database, and two
limits that came from faucet's clock and from a Date that leads.

- README, "Redeliver failed webhooks". A new paragraph says what the
  position is and what a run takes from the Date. The `gap` entry says
  where a gap ends and when a quiet hook logs one (64h55m). The limits
  are now the two risks that remain outside the window in which
  redelivery trusts a Date (more than 6h5m ahead before an outage of 72
  hours, more than 1h behind on a quiet hook), each with its condition
  first. The burst limit stays and is tracked separately.
- README, "Upgrading". The migration costs one re-read of the last 24
  hours and one more `before tracking began` gap, and a `RedeliveryStore`
  of your own must keep the clear. The rollback paragraph names migration
  `000008_redelivery_clear`.
- `faucet/readme_test.go` ties the new numbers to the constants.
- AGENTS.md: the "known exception on main" is gone. The line now says
  what holds: no faucet clock in the scan, read bounds from attempt times
  by type, and M and S as the trust window.
- The verify skill's redelivery page says the same about the position
  and the first reply.

Refs #26
`classify` and `newestOn` say what they do in their names. Two other
comments in `redelivertime.go` say the same in plainer words, and the
one on `firstRunBound` now gives both reasons why a clear may take part
in that bound.

Refs #26
…not say

Delete comments that narrate unexported helpers, restate test names or
explain test arithmetic. Keep exported API docs, the HTTP Date resolution
note and GitHub's deletion order. The migration's reason becomes a pointer
to #26.
…d types

The comment cleanup left code that only its comments had explained. This
changes no behavior.

- A range end is no longer made into an attempt time. `RedeliveryRange.bound`
  gives the bound of a range, and `readBound.rangeEnd` and
  `RedeliveryRange.olderEnd` are gone. An `attemptTime` is now always a time
  that GitHub stamped.
- `readBound.loweredTo` is the one way for a bare time into a read bound, and
  it can only lower the bound. `firstRunBound` uses it for the clear.
- `clearAnchor` and its `headDate` are now `storedClear` and `clearAt`, so
  the clear has one name.
- `expiredByBoth(date, clear)` replaces `pageDate.expiredBelow`. Its name
  says that a range closes unread only below what both Dates call expired.
- `storedClear.firstRunGapEnd` holds the rule that a first run with a clear
  also counts what expired since tracking began.
- A missing attempt and a missing gap start are explicit: `attemptTime.set`,
  an `orEarlier` that treats both sides alike, and a `headGapStart` that
  says whether there is a start. `gapBetween` no longer reads a zero time as
  no gap.
- `checkDate`, `readRanges` and `readRange` call their parameter
  `newestKnown`, because it is the stored top raised by the head of this run.

Refs #26
…k their premises

The numbers in these tests had their meaning in comments that are gone.
Each one now has a name. Where a scenario depends on what GitHub lists, or
on what faucet's clock says, the test asserts that before the run.

- The migration simulation takes its two rates from `steadyCapacity`. At
  half the capacity the sweep reads the last day again and recovers every
  failure. At 95 percent of it the unread part ends in gaps. Before, 42 and
  70 an hour stood for that.
- The fake store's `migrate` is now `applyMigration000008`, the cron test
  names the attempt that starts tracking, and the e2e test writes the age of
  its second delivery from the lookback.
- New test: a range whose last page is empty ends as a gap that stops at the
  oldest attempt read above it. The README promises that, and no test failed
  when `orEarlier` dropped the new end of the range.

Refs #26
"The scan reads no clock of faucet's" claimed too much. Each step still
gives the store faucet's clock as its time of record (`ScanStep.Now`). What
holds is that no read bound and no gap comes from that clock.

"A read never stops at a Date" claimed too much as well. The first run that
reads an attempt, after an empty list or the migration, reads down to the
stored Date less the margin. What holds is that the position is never a
Date, and that a Date can lower where a read stops and never raise it.

The doc of `Redeliver`, the README, AGENTS.md, the feature note and the doc
of `RedeliveryScan.Top` now say this.

Refs #26
… a clear

`RedeliveryScan.Clear` and `ScanStep.Clear` are new struct fields, so a
`RedeliveryStore` that was written before them compiles unchanged and drops
the clear. With such a store the scan broke its promise without a word. A
fake store that ignores `Clear` fails 7 of 46 redelivery tests, and in two
of them a failure ends with no final state: no range closes unread without
a stored clear, so the ranges of a burst stay open for good, and an empty
first run leaves no start of tracking.

A store that keeps `Clear` and still holds a top from before it has the
other half of the problem. That top can be a Date, which is what migration
`000008_redelivery_clear` exists for.

Both cases show as a top without a clear. A store that follows the contract
never returns one: a head step writes both, and the migration and a changed
hook reset the top. `topAndClearOf` now rejects it, and the run fails before
it reads a page. It asks for nothing, and the settle and the purge still
run, as after a failed first page. The README tells the owner of such a
store what to do.

`headGapStart` loses its case of a top without a clear, which can no longer
reach it.

This check is not in the rules of design v4. It is the last commit of the
branch, so it can be dropped on its own.

Refs #26
…gainst an attempt

An HTTP Date drops the fraction of its second. checkDate compared it with
the newest attempt less the slack, so a Date whose clock was 59m59.5s
behind an attempt later in the same second failed the check on every
run, and the run asked for nothing. The check now accepts a Date when
Date + 1s is not before the newest attempt less the slack, as the gap
floor already allows for.
…uild before it

A run of the build before the migration that held the lease during it,
or that an instance not yet replaced by a rolling deploy started after
it, still wrote top_at with main's SQL. On a first run that left a top
without a clear, and every later run failed; otherwise it put a Date
back into the position, and failures below it were lost.

The up migration now moves the old position into clear_at, renames top_at
to top_attempt_at, raises the epoch and ends the lease. Every redelivery
statement of the old build on the scan row reads or writes top_at, so its
runs fail with a column error and write no position, and its other routes
keep working. The down migration renames the column back. A test runs the
old build's statements, as sqlc generated them at e8382ff, against a
migrated database, both in flight and started after the migration, and
then shows that this build tracks every failure.
On a hook change the stores logged each open range as the gap
[OlderAt - O, max(NewerAt, OlderAt)], but kept the row with its own
older_at and newer_at: without the overlap, and inverted when NewerAt
lay inside the overlap below OlderAt.

RedeliveryRange.UnreadGap now gives the gap of a range that closes
unread, for the run's expired ranges and for both stores. pgstore deletes
the open ranges of the old hook and records each gap with the statement
that a scan step uses, and the fake store does the same. storetest gains
GapRecords: a store that lists the gaps it keeps is checked to keep each
one as its step gave it, or as the store reported it when the hook
changed. The pgstore and fake suites implement it.
…d above it

Since #36 the gap of a range that closes unread starts at OlderAt - O,
but it still ended at max(NewerAt, OlderAt). A range whose NewerAt lay
inside the overlap below OlderAt then logged a gap up to 3 minutes above
the oldest attempt that a run had read. UnreadGap now ends at
max(NewerAt, OlderAt - O), rule 8's [OlderAt - O, NewerAt], for the
expired ranges and the ranges of a changed hook.

The #36 test "the range expires unread" closes its range by reading it,
because its one run after the outage still has the Date from before it. A
new test closes a range in the run after that, without a page for it, and
checks both ends of its gap.
…rollback do what the README says

Four README claims did not hold:
- "A run that finds the list empty records no gap": it records one when
  the last run that read the list was about 64h55m before it, or more.
- "a gap can hold up to an hour in which GitHub deleted nothing": it can
  reach past the end of GitHub's retention by the slack, a second and the
  lead of the Date, up to about 7 hours inside the trust window.
- the migration "loses no failure silently": the first limit, a Date far
  ahead and then an outage, applies through it.
- the rollback "gets its old position back": only when no run has read
  the head of the list since the migration. After runs that read only an
  empty list, the old build gets the Date of the last of them.

The README now says each, condition first. A test pins how far a gap
reaches past GitHub's retention, the migration test checks the position
that the rollback leaves after an empty list, and the README test pins
the two new numbers to the constants. Item 3 gets a placeholder for its
tracking issue.
The burst limit moves to #37, and the stretch that a replaced hook leaves
unread is #38.

---
_Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-opus-5-5` • Thinking: `high`_
Remove the comments that the round-4 fixes added on the migration 000008
fence, the pre-migration statements and build in the migration test, and
the pgstore gap reader. The keep-list doc comments on UnreadGap,
RedeliveryScan.ClosedGaps and storetest.GapRecords stay.
…8 for that build

---
_Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-opus-5-5` • Thinking: `high`_
@ThomasK33
ThomasK33 merged commit 2372453 into main Nov 16, 2026
10 checks passed
@ThomasK33
ThomasK33 deleted the fix/redelivery-clock-26 branch November 16, 2026 14:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

redelivery: take retention floors and empty-list moves from GitHub's clock without silent drops

1 participant