Repository navigation
fix(faucet): take redelivery floors and positions from GitHub's times, not faucet's clock - #39
Merged
Merged
Conversation
…rsor The redesign of the redelivery clocks (#26) takes its gap floors from GitHub's Date header. Its tests need a GitHub whose Date leads, lags or is missing, whose pages take time to arrive, and that refuses the cursor of an attempt it has deleted. The fake GitHub had none of these. - `fakeGitHub.date` gives the Date of each list response, by call and cursor. A zero time is a response without a Date. - `redeliveryEnv.slowPages` moves the clock forward inside every list call, before GitHub reads its list. - `fakeGitHub.refusesStaleCursors` answers 422 to a cursor whose attempt left GitHub's retention. - `simSpec.setup` changes the env before the first delivery, and the simulation now follows GitHub's clock, so a test can set a skew. No production code changes, and no existing test changes. One new test pins what the three knobs do, because the tests of #26 lean on them. Refs #26
…clear A redelivery run kept one time, the top, for two jobs: the next read bound and the old end of the next gap. Those jobs need opposite bounds (#26). The top also held GitHub's Date after an empty list, and a Date can run ahead of GitHub's list. This commit gives the stores a second time, the clear, so that the scan can keep attempt times and Dates apart. The scan starts to use it in the next commit. - `RedeliveryScan.Clear` and `ScanStep.Clear`. A step replaces the stored clear, also with an earlier time: the store keeps the latest Date, not the highest, so one Date that ran ahead does not stay. - Migration `000008_redelivery_clear` adds `clear_at`, moves the old top into it and resets the top. No row says whether its top is an attempt time or a Date, so the next run is a first run that knows when tracking began. The down migration gives an older build its position back. - A changed hook resets the clear with the top, and the gaps of its open ranges start at `OlderAt - RedeliveryReadOverlap`, where the read of the range would have stopped (#36 did this for the scan's own gaps). - `storetest.RunRedelivery` checks all of it, and the fake store and pgstore pass. A pgstore test runs the migration both ways. Refs #26
…ot faucet's clock The redelivery sweep promises that every failed delivery that GitHub lists ends recovered, given up, or inside a counted gap. On main it broke that promise in two ways (#26). A gap ended at faucet's clock less 72 hours, so a clock behind GitHub's left a slice of deleted failures outside the gap, and a slow page widened the slice. The first run's gap started on faucet's clock, so a clock ahead of GitHub's left the oldest listed attempts unread and uncounted. And an empty list moved the position to GitHub's Date, which can run ahead of GitHub's list and put the next failures below the bound. One time served two roles that need opposite bounds. The scan now keeps three kinds of time apart by type (`faucet/redelivertime.go`), so a Date cannot become a read bound: - An attempt time (`delivered_at`, the top, a range's ends) is the only source of a read bound. An empty list leaves the top as it is. - A page's Date gives where GitHub's retention has ended, and only after a check: a missing Date, or one more than `RedeliveryDateSlack` (1h) behind the newest attempt known, fails the read. The scan checks it on the first page and on a page that ends the list, and nowhere else. A gap ends at `Date + 1s - 72h + slack`. - The clear, the stored Date of the last head read, gives gap starts: the head's gap starts at the later of `top - overlap` and `clear - RedeliveryDateMargin (6h) - overlap`. A first run reads 24 hours below the newest attempt and records `[newest - 72h, newest - 24h]` as its gap. After an empty first run or the migration it also reads back to `clear - margin`, and its gap reaches up to what expired since. A range whose new end is below `min(head Date, clear) - 72h` closes unread as a gap, so one Date far ahead closes nothing, and a range whose page fails ends this way. Tests. Each case of #26 and of the three design reviews is a test in `redeliver_gaps_test.go` or `redeliver_sim_test.go` that asserts no silent loss: a clock that lags or leads, a Date that leads, a slow page, one Date far ahead (the clear, and open ranges), an empty first run or the migration followed by an outage of 30 and 80 hours, a migrated top that was a Date ahead, range pages without a Date, a whole-second Date as far behind as the slack, a refused cursor, a changed hook, a quiet hook over 10 days, and the outage after which a quiet hook logs a gap (64h55m). Four tests pinned main's first run and empty list. An empty list now records no gap, the lookback counts from the newest attempt, and a gap ends at the oldest attempt that GitHub still lists. The cron tests' fake hook sends a Date, and the e2e lookback test moves its old delivery below the new bound. Refs #26
…and the migration The README still described main: a position that is GitHub's Date after an empty list, a first reply of `gaps: 1` for a new database, and two limits that came from faucet's clock and from a Date that leads. - README, "Redeliver failed webhooks". A new paragraph says what the position is and what a run takes from the Date. The `gap` entry says where a gap ends and when a quiet hook logs one (64h55m). The limits are now the two risks that remain outside the window in which redelivery trusts a Date (more than 6h5m ahead before an outage of 72 hours, more than 1h behind on a quiet hook), each with its condition first. The burst limit stays and is tracked separately. - README, "Upgrading". The migration costs one re-read of the last 24 hours and one more `before tracking began` gap, and a `RedeliveryStore` of your own must keep the clear. The rollback paragraph names migration `000008_redelivery_clear`. - `faucet/readme_test.go` ties the new numbers to the constants. - AGENTS.md: the "known exception on main" is gone. The line now says what holds: no faucet clock in the scan, read bounds from attempt times by type, and M and S as the trust window. - The verify skill's redelivery page says the same about the position and the first reply. Refs #26
`classify` and `newestOn` say what they do in their names. Two other comments in `redelivertime.go` say the same in plainer words, and the one on `firstRunBound` now gives both reasons why a clear may take part in that bound. Refs #26
…not say Delete comments that narrate unexported helpers, restate test names or explain test arithmetic. Keep exported API docs, the HTTP Date resolution note and GitHub's deletion order. The migration's reason becomes a pointer to #26.
…d types The comment cleanup left code that only its comments had explained. This changes no behavior. - A range end is no longer made into an attempt time. `RedeliveryRange.bound` gives the bound of a range, and `readBound.rangeEnd` and `RedeliveryRange.olderEnd` are gone. An `attemptTime` is now always a time that GitHub stamped. - `readBound.loweredTo` is the one way for a bare time into a read bound, and it can only lower the bound. `firstRunBound` uses it for the clear. - `clearAnchor` and its `headDate` are now `storedClear` and `clearAt`, so the clear has one name. - `expiredByBoth(date, clear)` replaces `pageDate.expiredBelow`. Its name says that a range closes unread only below what both Dates call expired. - `storedClear.firstRunGapEnd` holds the rule that a first run with a clear also counts what expired since tracking began. - A missing attempt and a missing gap start are explicit: `attemptTime.set`, an `orEarlier` that treats both sides alike, and a `headGapStart` that says whether there is a start. `gapBetween` no longer reads a zero time as no gap. - `checkDate`, `readRanges` and `readRange` call their parameter `newestKnown`, because it is the stored top raised by the head of this run. Refs #26
…k their premises The numbers in these tests had their meaning in comments that are gone. Each one now has a name. Where a scenario depends on what GitHub lists, or on what faucet's clock says, the test asserts that before the run. - The migration simulation takes its two rates from `steadyCapacity`. At half the capacity the sweep reads the last day again and recovers every failure. At 95 percent of it the unread part ends in gaps. Before, 42 and 70 an hour stood for that. - The fake store's `migrate` is now `applyMigration000008`, the cron test names the attempt that starts tracking, and the e2e test writes the age of its second delivery from the lookback. - New test: a range whose last page is empty ends as a gap that stops at the oldest attempt read above it. The README promises that, and no test failed when `orEarlier` dropped the new end of the range. Refs #26
"The scan reads no clock of faucet's" claimed too much. Each step still gives the store faucet's clock as its time of record (`ScanStep.Now`). What holds is that no read bound and no gap comes from that clock. "A read never stops at a Date" claimed too much as well. The first run that reads an attempt, after an empty list or the migration, reads down to the stored Date less the margin. What holds is that the position is never a Date, and that a Date can lower where a read stops and never raise it. The doc of `Redeliver`, the README, AGENTS.md, the feature note and the doc of `RedeliveryScan.Top` now say this. Refs #26
… a clear `RedeliveryScan.Clear` and `ScanStep.Clear` are new struct fields, so a `RedeliveryStore` that was written before them compiles unchanged and drops the clear. With such a store the scan broke its promise without a word. A fake store that ignores `Clear` fails 7 of 46 redelivery tests, and in two of them a failure ends with no final state: no range closes unread without a stored clear, so the ranges of a burst stay open for good, and an empty first run leaves no start of tracking. A store that keeps `Clear` and still holds a top from before it has the other half of the problem. That top can be a Date, which is what migration `000008_redelivery_clear` exists for. Both cases show as a top without a clear. A store that follows the contract never returns one: a head step writes both, and the migration and a changed hook reset the top. `topAndClearOf` now rejects it, and the run fails before it reads a page. It asks for nothing, and the settle and the purge still run, as after a failed first page. The README tells the owner of such a store what to do. `headGapStart` loses its case of a top without a clear, which can no longer reach it. This check is not in the rules of design v4. It is the last commit of the branch, so it can be dropped on its own. Refs #26
…gainst an attempt An HTTP Date drops the fraction of its second. checkDate compared it with the newest attempt less the slack, so a Date whose clock was 59m59.5s behind an attempt later in the same second failed the check on every run, and the run asked for nothing. The check now accepts a Date when Date + 1s is not before the newest attempt less the slack, as the gap floor already allows for.
…uild before it A run of the build before the migration that held the lease during it, or that an instance not yet replaced by a rolling deploy started after it, still wrote top_at with main's SQL. On a first run that left a top without a clear, and every later run failed; otherwise it put a Date back into the position, and failures below it were lost. The up migration now moves the old position into clear_at, renames top_at to top_attempt_at, raises the epoch and ends the lease. Every redelivery statement of the old build on the scan row reads or writes top_at, so its runs fail with a column error and write no position, and its other routes keep working. The down migration renames the column back. A test runs the old build's statements, as sqlc generated them at e8382ff, against a migrated database, both in flight and started after the migration, and then shows that this build tracks every failure.
On a hook change the stores logged each open range as the gap [OlderAt - O, max(NewerAt, OlderAt)], but kept the row with its own older_at and newer_at: without the overlap, and inverted when NewerAt lay inside the overlap below OlderAt. RedeliveryRange.UnreadGap now gives the gap of a range that closes unread, for the run's expired ranges and for both stores. pgstore deletes the open ranges of the old hook and records each gap with the statement that a scan step uses, and the fake store does the same. storetest gains GapRecords: a store that lists the gaps it keeps is checked to keep each one as its step gave it, or as the store reported it when the hook changed. The pgstore and fake suites implement it.
…d above it Since #36 the gap of a range that closes unread starts at OlderAt - O, but it still ended at max(NewerAt, OlderAt). A range whose NewerAt lay inside the overlap below OlderAt then logged a gap up to 3 minutes above the oldest attempt that a run had read. UnreadGap now ends at max(NewerAt, OlderAt - O), rule 8's [OlderAt - O, NewerAt], for the expired ranges and the ranges of a changed hook. The #36 test "the range expires unread" closes its range by reading it, because its one run after the outage still has the Date from before it. A new test closes a range in the run after that, without a page for it, and checks both ends of its gap.
…rollback do what the README says Four README claims did not hold: - "A run that finds the list empty records no gap": it records one when the last run that read the list was about 64h55m before it, or more. - "a gap can hold up to an hour in which GitHub deleted nothing": it can reach past the end of GitHub's retention by the slack, a second and the lead of the Date, up to about 7 hours inside the trust window. - the migration "loses no failure silently": the first limit, a Date far ahead and then an outage, applies through it. - the rollback "gets its old position back": only when no run has read the head of the list since the migration. After runs that read only an empty list, the old build gets the Date of the last of them. The README now says each, condition first. A test pins how far a gap reaches past GitHub's retention, the migration test checks the position that the rollback leaves after an empty list, and the README test pins the two new numbers to the constants. Item 3 gets a placeholder for its tracking issue.
The burst limit moves to #37, and the stretch that a replaced hook leaves unread is #38. --- _Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-opus-5-5` • Thinking: `high`_
Remove the comments that the round-4 fixes added on the migration 000008 fence, the pre-migration statements and build in the migration test, and the pgstore gap reader. The keep-list doc comments on UnreadGap, RedeliveryScan.ClosedGaps and storetest.GapRecords stay.
…8 for that build --- _Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-opus-5-5` • Thinking: `high`_
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The redelivery sweep promises that every failed delivery in GitHub's list ends recovered, given up, or inside a counted, logged gap. On main, faucet's clock sets the retention floor and the first-run lookback, and an empty list moves the position to GitHub's
Date. Each of these can drop failures silently when the two clocks differ (#26). This PR gives each time its own source: attempt times for where a read stops, a checked pageDatefor where GitHub's retention ends, and the storedDateof the last head read for where a gap after a silence starts. Four review rounds by five models shaped the design and the code.What changed
faucet/redelivertime.goadds a type per time role. ADatecannot become a read bound, and the scan uses no gap or bound from faucet's clock.Clear(theDateof the last head read). Migration000008_redelivery_clearmoves the old position into it, renames the position column, and raises the epoch, so a run of an older build fails loudly instead of writing aDateback.Datethat is missing, or more than 1 hour behind an attempt, fails the read. The README states the trust window (6 hours ahead, 1 hour behind) and the two risks outside it.Scope
Tradeoffs
Datefails the run instead of falling back to faucet's clock. The fallback would bring limit 1 back.Datelead passed 6 hours. The reset costs one re-read of up to 24 hours and a few extra gaps.Blast Radius
Every deployment with redelivery on runs migration
000008. An instance of an older build still running after it gets a failing/cron/redeliver(500) until it is replaced. ARedeliveryStoreof your own must now keepClear, and a run fails with a named error when it does not.Verification
go test -race ./...with Postgres,golangci-lint(0 issues), sqlc andgo mod tidywith no diff, and e2e inverify/(234 of 234) all pass.Closes #26
Generated with
xum• Model:anthropic:claude-opus-5-5• Thinking:high