Skip to content

fix(githttp): wait for the sweep restore of a missing registered pack - #893

Merged
ThomasK33 merged 4 commits into
mainfrom
hydrate-backoff-for-a-restoring-sweep
Nov 16, 2026
Merged

ThomasK33 merged 4 commits into
mainfrom
hydrate-backoff-for-a-restoring-sweep

Conversation

@ThomasK33

@ThomasK33 ThomasK33 commented Nov 16, 2026 •

Copy link
Copy Markdown
Member

Why

The orphan sweep can delete the bytes of a pack that a writer registered a moment before, and it restores them within the same pass (internal/jobs/sweep.go, the recheck and restore after safeDelete). A repository whose registry lists a pack with no bytes cannot hydrate. A cold read that started inside that window listed the packs 8 times with no wait between the listings and failed with ErrPackMissing, so the client read 500 until the restore landed. The doc comment of Sweep said that hydration's bounded retry absorbs this. It did not: 8 listings take a few milliseconds, and the restore is a registry read plus a server-side copy.

I reproduced the window with decorators that park the sweeper at chosen calls (sqlite registry, file blob store, real git packs). The registry lists the pack and the blob is not found from the delete to the restore: 185 to 589 µs without a fault, until the next sweep's heal after a failed restore or a dead sweeper. No order of events leaves a committed ref's pack without its bytes for good, so this is a transient read failure and not data loss.

What changed

  • hydratePacksBy waits between two listings that find the same pack missing: 50 ms, doubling to 1 s, and it stops when the context ends. A pack that a fold replaced is not listed again, so the fold race still retries at once. The last attempt does not wait. A pack that is truly gone now fails after about 2.5 s instead of a few milliseconds. The waits add up to 2.55 s, so a restore that takes longer than that (a very large pack, or a failed restore) still fails the cold read, and the next sweep's heal pass is the backstop. A longer wait would hold the repository's hydrate lock longer: concurrent cold reads of a repository whose pack is gone for good queue behind it, and the k-th read fails after about k × 2.5 s (live node, 4 reads: 2.8, 5.3, 7.9 and 10.5 s).
  • The comment of Sweep says what is true.
  • TestSweepWindow_ColdHydrateInWindowWaitsForRestore starts a cold hydrate 200 ms before the restore (the restore waits until the hydration has read the missing pack) and fails on main. TestHydratePacks_WaitsForAPackThatStaysMissing covers the wait and the context, and TestHydratePacks_FoldRaceRetriesWithoutWaiting checks that a fold race retries at once.
  • The lane-schedule fingerprints are re-recorded, because hydratePacksBy is a covered declaration. The model does not describe the wait.

Scope

No behavior change for a hydration that finds every pack. I did not change the restore: a failed restore still waits for the next sweep's heal (every hour by default).

Verification

  • A mutant without the wait fails TestHydratePacks_WaitsForAPackThatStaysMissing and TestSweepWindow_ColdHydrateInWindowWaitsForRestore. A mutant that waits for every miss fails the fold test.
  • go test of internal/jobs, internal/gitread and the hydrate tests of internal/githttp (local backend), gofmt, golangci-lint 2.15.0 (0 issues), the doc-link gate and both fingerprint gates pass.

Generated with xum • Model: anthropic:claude-sonnet-5-5 • Thinking: xhigh

The orphan sweep can delete the bytes of a pack that a writer registered a
moment before, and it restores them within the same pass. A hydration that
started inside that window listed the packs 8 times with no wait and failed
with ErrPackMissing, so a cold read answered 500 until the restore landed.
The hydration now waits between two listings that find the same pack
missing (50 ms, doubling to 1 s). A pack that a fold replaced is not
listed again, so a fold race still retries at once. The doc comment of
Sweep no longer says that the retry absorbs the window when it did not.

Signed-off-by: Thomas Kosiewski <tk@coder.com>

---
_Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-sonnet-5-5` • Thinking: `xhigh`_
<!-- xum-attribution: model=anthropic:claude-sonnet-5-5 thinking=xhigh -->

Change-Id: I85a778564e1f0af05b2b857d80b007798270a1d2
Signed-off-by: Thomas Kosiewski <tk@coder.com>

---
_Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-sonnet-5-5` • Thinking: `xhigh`_
<!-- xum-attribution: model=anthropic:claude-sonnet-5-5 thinking=xhigh -->

Change-Id: Icbc8515a2fc19d1ff22e458790fd38a0dc635abc
@ThomasK33

Copy link
Copy Markdown
Member Author

@codex review

@ThomasK33

Copy link
Copy Markdown
Member Author

@codex security review

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-16T16:40:12.057397Z 351742a Manual request
🔒 Security Review ✅ Completed 2026-11-16T16:41:20.613835Z 351742a Manual request
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c49a78b39d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread internal/githttp/hydrate.go Resolved
Comment thread internal/jobs/sweep_window_test.go Resolved
@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review

Security review completed. No security issues were found in this pull request.

Reviewed commit: c49a78b39d

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

…or the clock

The sweeper test waits until the hydration has read the missing pack
before it releases the restore. The wait test starts its timer at the
first listing. A new test checks that a fold race retries at once, and the
sweeper test file keeps only the red test, which saves 5 seconds.

Signed-off-by: Thomas Kosiewski <tk@coder.com>

---
_Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-sonnet-5-5` • Thinking: `xhigh`_
<!-- xum-attribution: model=anthropic:claude-sonnet-5-5 thinking=xhigh -->

Change-Id: Id8298a2ea63ab49164ba2bc51c22aaa0452b91cc
@ThomasK33

Copy link
Copy Markdown
Member Author

@codex review

@ThomasK33

Copy link
Copy Markdown
Member Author

@codex security review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. 👍

Reviewed commit: 491e92a4c1

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review

Security review completed. No security issues were found in this pull request.

Reviewed commit: 491e92a4c1

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

Signed-off-by: Thomas Kosiewski <tk@coder.com>

---
_Generated with [`xum`](https://github.com/coder/xum) • Model: `anthropic:claude-sonnet-5-5` • Thinking: `xhigh`_
<!-- xum-attribution: model=anthropic:claude-sonnet-5-5 thinking=xhigh -->

Change-Id: Ic507e861904466eeaee9b1164dffadb4f579ae19
@ThomasK33

Copy link
Copy Markdown
Member Author

@codex review

@ThomasK33

Copy link
Copy Markdown
Member Author

@codex security review

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. 🚀

Reviewed commit: 351742a987

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@chatgpt-codex-connector

Copy link
Copy Markdown

🛡️ Codex Security Review

Security review completed. No security issues were found in this pull request.

Reviewed commit: 351742a987

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

  • You comment "@codex security review"
  • A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

@ThomasK33
ThomasK33 added this pull request to the merge queue Nov 16, 2026
Merged via the queue into main with commit 044adc3 Nov 16, 2026
15 checks passed
@ThomasK33
ThomasK33 deleted the hydrate-backoff-for-a-restoring-sweep branch November 16, 2026 17:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant