Skip to content

fix(queue-storage): keep a late heartbeat from replacing parked progress - #727

Merged
hardbyte merged 3 commits into
mainfrom
claude/forward-port-0.6.17-park-progress
Nov 12, 2026
Merged

hardbyte merged 3 commits into
mainfrom
claude/forward-port-0.6.17-park-progress

Conversation

@hardbyte

Copy link
Copy Markdown
Owner

Follow-up to #725, from a review finding there.

Problem

#725 made parking for a callback store the handler's final progress. A heartbeat that read an older progress snapshot before the park could still write it after the park committed: mark_finalizing stops new snapshots but not one already being written, the park did not take the attempt lock, and the receipt progress upsert matched the parked attempt's still-open claim.

Change

  • Parking a receipt attempt takes the attempt lock before changing the lease, as heartbeat progress writes do. A heartbeat holding the lock finishes first; one that tries during the park skips the attempt.
  • The receipt progress upsert leaves out attempts whose lease is waiting_external, so a heartbeat that gets the lock after the park writes nothing.
  • Lease-mode heartbeats already require state = 'running' and lock the lease row, so they are unchanged.

Tests

  • test_queue_storage_heartbeat_does_not_overwrite_parked_progress parks a receipt attempt with progress, sends a heartbeat with an older snapshot, and checks it writes nothing and the parked progress survives. It fails without the upsert change.
  • test_queue_storage_flush_progress_waits_for_a_locked_receipt_attempt now waits until the flush is seen waiting on a lock before releasing it, instead of sleeping 200 ms.
  • Locally on PostgreSQL 18: the callback, progress and heartbeat runtime tests, 28 of 28.

0.6 is unaffected: there the park stores no progress at all.


Generated by Claude Code

A heartbeat that read a progress snapshot before its attempt parked for a callback could write that snapshot after the park committed, replacing the final progress the park stored. Parking a receipt attempt now takes the attempt lock, which heartbeat writes take too, and the receipt progress upsert leaves out attempts that are waiting for a callback.

The explicit flush test now waits until the flush is seen waiting on a lock before releasing it.
@hardbyte hardbyte added the full-ci Run the full CI matrix (Python build+test, E2E) on this PR label Nov 12, 2026 — with Claude
@coderabbitai

coderabbitai Bot commented Nov 12, 2026 •

Copy link
Copy Markdown

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 3 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 4ece0661-671f-4055-aaba-3b9f70f3c399
Commits

Reviewing files that changed from the base of the PR and between e5d690b and 0a0475a.

Files selected for processing: 3

  • .github/workflows/ci.yml
  • awa-model/src/queue_storage.rs
  • awa/tests/queue_storage_runtime_test.rs
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-12T08:42:10.319790Z 95b0c99 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 95b0c99287

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread awa-model/src/queue_storage.rs Resolved
…alizers

External completion, failure and retry deleted the lease row and then waited
for the attempt lock, the reverse of the order every other receipt path and
the callback park use, so a fast callback could deadlock with a parking
worker. They now take the attempt locks for the callback's leases first.
psycopg 3.3.9 has no Python 3.15 binary wheel yet, so installing psycopg
and psycopg-binary separately paired psycopg 3.3.9 with psycopg-binary
3.3.8, which fails to import. The extra pins matching versions.
@hardbyte
hardbyte merged commit 456c805 into main Nov 12, 2026
52 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

full-ci Run the full CI matrix (Python build+test, E2E) on this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant