Repository navigation
fix(githttp): wait for the sweep restore of a missing registered pack - #893
Conversation
The orphan sweep can delete the bytes of a pack that a writer registered a moment before, and it restores them within the same pass. A hydration that started inside that window listed the packs 8 times with no wait and failed with ErrPackMissing, so a cold read answered 500 until the restore landed. The hydration now waits between two listings that find the same pack missing (50 ms, doubling to 1 s). A pack that a fold replaced is not listed again, so a fold race still retries at once. The doc comment of Sweep no longer says that the retry absorbs the window when it did not. Signed-off-by: Thomas Kosiewski <tk@coder.com> --- _Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-sonnet-5-5` • Thinking: `xhigh`_ <!-- xum-attribution: model=anthropic:claude-sonnet-5-5 thinking=xhigh --> Change-Id: I85a778564e1f0af05b2b857d80b007798270a1d2
Signed-off-by: Thomas Kosiewski <tk@coder.com> --- _Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-sonnet-5-5` • Thinking: `xhigh`_ <!-- xum-attribution: model=anthropic:claude-sonnet-5-5 thinking=xhigh --> Change-Id: Icbc8515a2fc19d1ff22e458790fd38a0dc635abc
|
@codex review |
|
@codex security review |
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c49a78b39d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
🛡️ Codex Security ReviewSecurity review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
…or the clock The sweeper test waits until the hydration has read the missing pack before it releases the restore. The wait test starts its timer at the first listing. A new test checks that a fold race retries at once, and the sweeper test file keeps only the red test, which saves 5 seconds. Signed-off-by: Thomas Kosiewski <tk@coder.com> --- _Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-sonnet-5-5` • Thinking: `xhigh`_ <!-- xum-attribution: model=anthropic:claude-sonnet-5-5 thinking=xhigh --> Change-Id: Id8298a2ea63ab49164ba2bc51c22aaa0452b91cc
|
@codex review |
|
@codex security review |
|
Codex Review: Didn't find any major issues. 👍 Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
🛡️ Codex Security ReviewSecurity review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
Signed-off-by: Thomas Kosiewski <tk@coder.com> --- _Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-sonnet-5-5` • Thinking: `xhigh`_ <!-- xum-attribution: model=anthropic:claude-sonnet-5-5 thinking=xhigh --> Change-Id: Ic507e861904466eeaee9b1164dffadb4f579ae19
|
@codex review |
|
@codex security review |
|
Codex Review: Didn't find any major issues. 🚀 Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
🛡️ Codex Security ReviewSecurity review completed. No security issues were found in this pull request. Reviewed commit: Only the user who started this review can view the report in Codex. ℹ️ About Codex security reviews in GitHubThis is an experimental Codex feature. Security reviews are triggered when:
Once complete, Codex will leave suggestions, or a comment if no findings are found. |
Why
The orphan sweep can delete the bytes of a pack that a writer registered a moment before, and it restores them within the same pass (
internal/jobs/sweep.go, the recheck and restore aftersafeDelete). A repository whose registry lists a pack with no bytes cannot hydrate. A cold read that started inside that window listed the packs 8 times with no wait between the listings and failed withErrPackMissing, so the client read500until the restore landed. The doc comment ofSweepsaid that hydration's bounded retry absorbs this. It did not: 8 listings take a few milliseconds, and the restore is a registry read plus a server-side copy.I reproduced the window with decorators that park the sweeper at chosen calls (sqlite registry, file blob store, real git packs). The registry lists the pack and the blob is not found from the delete to the restore: 185 to 589 µs without a fault, until the next sweep's heal after a failed restore or a dead sweeper. No order of events leaves a committed ref's pack without its bytes for good, so this is a transient read failure and not data loss.
What changed
hydratePacksBywaits between two listings that find the same pack missing: 50 ms, doubling to 1 s, and it stops when the context ends. A pack that a fold replaced is not listed again, so the fold race still retries at once. The last attempt does not wait. A pack that is truly gone now fails after about 2.5 s instead of a few milliseconds. The waits add up to 2.55 s, so a restore that takes longer than that (a very large pack, or a failed restore) still fails the cold read, and the next sweep's heal pass is the backstop. A longer wait would hold the repository's hydrate lock longer: concurrent cold reads of a repository whose pack is gone for good queue behind it, and the k-th read fails after about k × 2.5 s (live node, 4 reads: 2.8, 5.3, 7.9 and 10.5 s).Sweepsays what is true.TestSweepWindow_ColdHydrateInWindowWaitsForRestorestarts a cold hydrate 200 ms before the restore (the restore waits until the hydration has read the missing pack) and fails onmain.TestHydratePacks_WaitsForAPackThatStaysMissingcovers the wait and the context, andTestHydratePacks_FoldRaceRetriesWithoutWaitingchecks that a fold race retries at once.hydratePacksByis a covered declaration. The model does not describe the wait.Scope
No behavior change for a hydration that finds every pack. I did not change the restore: a failed restore still waits for the next sweep's heal (every hour by default).
Verification
TestHydratePacks_WaitsForAPackThatStaysMissingandTestSweepWindow_ColdHydrateInWindowWaitsForRestore. A mutant that waits for every miss fails the fold test.go testofinternal/jobs,internal/gitreadand the hydrate tests ofinternal/githttp(local backend),gofmt, golangci-lint 2.15.0 (0 issues), the doc-link gate and both fingerprint gates pass.Generated with
xum• Model:anthropic:claude-sonnet-5-5• Thinking:xhigh