Skip to content

fix(queue-storage): preserve moved-attempt reads and snapshots (0.6) - #652

Draft
hardbyte wants to merge 2 commits into
release/0.6.6from
brian/fix-live-attempt-reads-0.6
Draft

hardbyte wants to merge 2 commits into
release/0.6.6from
brian/fix-live-attempt-reads-0.6

Conversation

@hardbyte

@hardbyte hardbyte commented Nov 6, 2026 •

Copy link
Copy Markdown
Owner

HELD OUT of 0.6.14 per review: measure the added per-read relation-lock footprint and prune Blocked rate under load before reconsidering release inclusion. Full CI does not clear this hold.

After a priority move, the old tombstoned ready row and the live row can share a lane sequence. Active lease hydration then duplicates the running job; compact point reads can return the old payload instead of the aged metadata. Exclude the full tombstone identity from active joins and require the compact item's exact prior run lease.

Point reads lock the ready, lease and claim families before their first data query establishes a repeatable-read snapshot. This protects running evidence from claim/lease truncation and also closes the first-query window: a SELECT can take its snapshot before acquiring relation locks, so a read waiting behind queue prune could miss a failed job carried to the live slot. Ready locks are acquired before the snapshot; queue prune takes its ready child NOWAIT before terminal children, and claim-family lock order is preserved. No migration, configuration or public API change.

Validation:

  • Original a41c85b5: ten focused read regressions pass on PostgreSQL 17 and 18. Matched old-source controls expose duplicate materialized hydration, stale compact payload, real claim-prune and lease-TRUNCATE races; the row-local control passes before and after.
  • The new first-query interleave uses an actual retained failed job and real queue prune. It observes the pruner's failed-row carry and reader's ready-relation wait by exact backend identity, releases all locks and joins both tasks before assertions, then checks carried1 and exact Failed/id/runlease/arguments/metadata/finalization time against a fresh read.
  • The unchanged new fixture fails with the original ready-lock omission on both PostgreSQL 17 and 18: the retained failed job disappears. With the prelock, it passes on both majors. These are specific logical failures and passes, with unchanged bounds.
  • Both Rust workspaces pass formatting, offline all-target/all-feature Clippy with warnings denied, and workspace builds.
  • Current 9b05fe2b automatic full CI 42288944227 passes all 29 jobs. Earlier parent full-CI success is separate.

This remains a draft; mixed-load release acceptance is separate from the focused read proofs.

@hardbyte hardbyte added the full-ci Run the full CI matrix (Python build+test, E2E) on this PR label Nov 6, 2026 — with ChatGPT Codex Connector
@coderabbitai

coderabbitai Bot commented Nov 6, 2026

Copy link
Copy Markdown

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

hardbyte commented Nov 6, 2026

Copy link
Copy Markdown
Owner Author

Holding this one out of 0.6.14. We're looking into how Postgres's lock manager behaves under heavy queue-storage workloads, where ring prune repeatedly requests ACCESS EXCLUSIVE and TRUNCATEs partitions. Until we understand that better, the patch release should take only changes that don't add to lock traffic. This PR does add to it:

  • load_job now takes ACCESS SHARE on every partition of the lease and claim ring parents on each call, and list paths call it once per candidate. That's a much larger relation-lock footprint per read: more locks overflow the fast-path slots, and prune is blocked, and retries its ACCESS EXCLUSIVE, more often.
  • I'd like the per-read lock count and the prune Blocked rate measured under load first, ideally against a design where a list page shares one snapshot transaction.

Before this ships, it also needs the same follow-up as main (#600, ac26f88f): put ready_entries in the upfront LOCK TABLE. A statement takes its snapshot before it locks the relations it reads, so a queue prune can truncate a ready, done or receipt-completion partition between the snapshot and the first query's lock.

The tombstone exclusion on the attempt joins looks good and carries no lock cost. If we want it sooner, it could ship separately.


Generated by Claude Code

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

full-ci Run the full CI matrix (Python build+test, E2E) on this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant