Skip to content

Add batch queue operations #26

Description

@polyrand

Context

Every put(), pop(), and acknowledgement currently runs as a separate SQLite operation or transaction. Workloads with many small messages pay repeated Python, SQL, locking, and commit overhead.

The backlog proposed focused batch operations after lifecycle and indexing correctness were stable. FIFO indexing is now fixed, while safe acknowledgement should build on claim ownership rather than the current message-ID-only lifecycle.

Why implement this

SQLite performs best when related writes share a transaction. Batch APIs can improve throughput without adding runtime dependencies or expanding LiteQueue into a scheduler.

Proposed direction

  • Add put_many() with one transaction and a documented all-or-nothing policy.
  • Add pop_many(limit) that atomically claims the oldest ready messages.
  • Preserve deterministic FIFO result order; do not rely on unspecified RETURNING row order.
  • Add batch completion and failure only after claim tokens or attempt IDs exist.
  • Define input and output types and behavior for zero, negative, and oversized limits.
  • Keep the single-message methods as the basic API.

Acceptance criteria

  • Batch claims cannot overlap across consumers.
  • Batch order matches repeated single-message FIFO order.
  • Capacity and transaction failure behavior are documented.
  • Partial database changes cannot leak from an all-or-nothing batch.
  • Benchmarks compare batch throughput with repeated single operations.
  • make test passes.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions