Skip to content

Prevent stale workers from acknowledging newer message claims #22

Description

@polyrand

Context

LiteQueue can retry a locked message, but a claim has no identity beyond message_id. done(), mark_failed(), and retry() update any matching row without checking its current state.

Current code:

  • pop() changes READY to LOCKED and returns the message.
  • retry() changes any matching message to READY and leaves its old lock_time in place.
  • done() and mark_failed() accept acknowledgements from any worker that knows the message ID.

A focused reproduction on commit 2b82ede showed this race:

  1. Worker A pops a message.
  2. Recovery retries the message.
  3. Worker B pops the same message.
  4. Worker A calls done(message_id).
  5. LiteQueue marks worker B's active claim as done.

The stale done() returned True, and the newer claim ended in DONE.

Why implement this

Retry is meant to recover work after a worker stops responding. Without claim ownership, a slow worker can acknowledge a newer delivery. If the newer worker then fails, LiteQueue has already removed the task from active work. This breaks reliable redelivery and can lose work.

Proposed direction

  • Add a claim token or monotonically increasing attempt number to each successful pop.
  • Include the claim identity in the returned Message or a dedicated claim type.
  • Require message_id, claim identity, and LOCKED status in done() and mark_failed() updates.
  • Restrict retry to documented source states.
  • Clear obsolete lock_time and done_time fields when returning work to READY.
  • Provide one atomic operation for retrying expired claims instead of requiring list_locked() followed by separate retry() calls.
  • Document and migrate the schema and acknowledgement API.

Acceptance criteria

  • A stale worker cannot complete or fail a newer claim.
  • Invalid transitions return a clear failure without changing the row.
  • Retrying a claim clears obsolete timestamps.
  • Expired-claim recovery is atomic under concurrent workers.
  • Tests reproduce the stale-worker race for both pop implementations.
  • make test passes.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions