Skip to content

fix(queue-storage): make an explicit progress flush wait for the attempt lock - #726

Merged
hardbyte merged 4 commits into
release/0.6.6from
claude/beautiful-dijkstra-vqdyqy
Nov 12, 2026
Merged

hardbyte merged 4 commits into
release/0.6.6from
claude/beautiful-dijkstra-vqdyqy

Conversation

@hardbyte

Copy link
Copy Markdown
Owner

An explicit flush_progress() on a receipt attempt could write nothing and still return success. Found while forward-porting #647 to main (#725). Present in 0.6.16 and 0.6.17.

Cause

#647 made the heartbeat batch leave out receipt attempts whose lock another transaction holds, through upsert_attempt_state_progress_from_receipts_tx. flush_progress calls the same helper, so a flush that overlapped a heartbeat or a completion of the same attempt was filtered out, the helper returned zero rows, flush_progress ignored the count, and the worker treated the progress as persisted.

Change

  • The helper takes skip_locked. Heartbeat batches pass true and behave as before; flush_progress passes false and waits for the attempt's lock.
  • test_queue_storage_flush_progress_waits_for_a_locked_receipt_attempt holds the attempt lock, checks that the flush waits, releases the lock and reads the progress back. It fails with the flush on the skip path.

Not in this PR

Three other issues the same review found on main also exist on 0.6 and are fixed in #725: the final progress update lost when an attempt parks for a callback, the accelerated heartbeat cutoff drifting later while rescue waits for a connection, and younger claim slots waiting behind the oldest when rescue passes do not re-arm.


Generated by Claude Code

…mpt lock

A heartbeat batch leaves out receipt attempts whose lock another transaction holds. flush_progress shared that helper, so a flush that overlapped a heartbeat or a completion of the same attempt wrote nothing and still returned Ok. The flush now waits for the lock; only heartbeat batches skip.
@hardbyte hardbyte added the full-ci Run the full CI matrix (Python build+test, E2E) on this PR label Nov 12, 2026 — with Claude
@coderabbitai

coderabbitai Bot commented Nov 12, 2026 •

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 9f4a2c62-4dd9-4880-8db8-8c3484c8236b

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-12T08:03:48.649514Z 324cea5 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

…klog test

The test waited for the retried jobs by counting completed rows in storage. Queue prune drops a sealed slot's completed rows once the ring wraps past it, so a recovery slower than one revolution could never reach the count. The handler now counts the recovery runs, and the test then waits for the queue to hold no running, retryable or available job.

Copy link
Copy Markdown
Owner Author

Rust tests (pg17, rest) failed rescue_drains_an_abandoned_backlog_without_rescuing_live_claims with "timed out after 360s waiting for the retried jobs to complete". The same failure occurred on release/0.6.6's branch run for #723's merge commit, before this PR's change, which only touches flush_progress.

Cause: the test counted completed rows in storage while the recovery client runs with eager ring rotation, so queue prune could drop a sealed slot's completed rows before the count reached its target. The fix for this on main (in #725) is now ported here: the handler counts recovery runs, and the test then waits for no running, retryable or available job.


Generated by Claude Code

…it exists

Another test's stale-observation cleanup can delete the row between reports,
so the poll reads it with fetch_optional instead of failing on RowNotFound.

Copy link
Copy Markdown
Owner Author

Rust tests (pg17, workspace) failed in runtime_control::tests::held_session_observes_and_reports_without_pool_or_new_connections with RowNotFound: the test polled its observation row with fetch_one, and the row can be missing between reports, for example after another test's stale-observation cleanup. It isn't related to the flush change. b38b360 polls with fetch_optional so it waits for the next report.


Generated by Claude Code

@hardbyte
hardbyte merged commit eae662f into release/0.6.6 Nov 12, 2026
34 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

full-ci Run the full CI matrix (Python build+test, E2E) on this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant