Unclear error on passing connection with autocommit=True on creation

[abcdenis]

Hi,

what I do:

• create connection with autocommit=True and pass it to LiteQueue constructor

what I get:

  File ".../lib/python3.12/site-packages/litequeue.py", line 235, in __init__
    self.conn.execute("PRAGMA synchronous = NORMAL;")
sqlite3.OperationalError: Safety level may not be changed inside a transaction

what I expected to get:
error that connection has enabled autocommit field and it is unsupported by library

Thank you.

litequeue version 0.9

[polyrand]

Thanks for reporting! I was actually working on removing the option to pass a custom sqlite3 connection object, and instead forcing the caller to use the internal connection settings done by LiteQueue.

Using autocommit=True may interfere with the assumptions from LiteQueue. I'm also trying to make it more thread-safe, and one of the things I was doing is splitting a singe "write" connection from read-only connections, to avoid other threads from reading uncommitted data started in a transaction by another thread.

This would be incompatible with passing a custom sqlite3.Connection object.

I'm also removing the option to host more than one queue in a single DB file, and assume that LiteQueue would be used with isolated SQLite database files, one file per queue (instead of multiple queues in the same file). Mostly so that the code can be simplified, and I can work on polishing the code and making it more robust.

I haven't released any new version yet, but I'm interested in knowing your use case to make sure it stays useful.

[abcdenis]

Thank you!
I've noticed new "one queue per file" rule and split queues to dedicated files in my pet project today :)

I don't have much feedback because I use litequeue for a week.

[polyrand]

Good, if one-queue per file works for you, I'll continue with the current changes. I'll probably release a 0.11 version in a few days. I hadn't updated this project in a while, but now I'm reviewing some parts and noticed that it was a bit brittle in some ways.

It supported multiple queues per file, but I wasn't pleased with the implementation and even had some bugs. I'm now making it just "a persistent queue backed by SQLite", and simplifying everything around that, even if I have to let some features go, or disallow certain behaviours.

[abcdenis]

I plan to use litequeue as persistent queue for pet project: one process put data in queue and another process read data from it.
Will it be possible in future version? There were some announce like "only one writing thread and many reading threads" and it confuses me.

[polyrand]

I plan to use litequeue as persistent queue for pet project: one process put data in queue and another process read data from it.
Will it be possible in future version?

Yes, that use case will be possible, and it is one of LiteQueue’s intended uses.

Each process should create its own LiteQueue instance using the same queue name and folder, so both instances open the same SQLite queue file:

queue = LiteQueue(name="tasks", folder=queue_folder)

Please don’t create and pass a custom sqlite3.Connection. Let LiteQueue create and manage its own connections.

here were some announce like "only one writing thread and many reading threads" and it confuses me.

The “one writer connection” wording describes LiteQueue’s internal connection management; it does not mean that only one application process can produce or consume messages. Each process has its own connections, while SQLite coordinates access to the shared database file and serializes writes when necessary. Claiming a message with pop() is performed in a transaction, so two consumers should not receive the same message.

The connection changes are primarily about thread safety and transaction isolation. If multiple threads share one SQLite connection, a read performed by one thread can become part of another thread’s active transaction. It may then see uncommitted changes that could later be rolled back. Similarly, another thread’s write could accidentally become part of that transaction and be committed or rolled back with it.

By managing the database file itself, LiteQueue can use a dedicated write connection and separate query-only read connections. The thread that owns a transaction can see its own uncommitted changes, while other threads use separate connections and continue seeing only committed data. This makes transaction boundaries much clearer and safer.

So your producer-process/consumer-process setup is supported: point both LiteQueue instances at the same queue file and let LiteQueue manage the SQLite connections.

The tl;dr is that this is all for better thread safety.

Also notice that all this information is for an unreleased version of LiteQueue, which has breaking changes: mainly just allowing a single queue per database. (I'll release a new version tomorrow probably).

[polyrand]

I just released version 0.11, which has all the recent changes and supports a single queue per file only. See the README for usage examples. Also I added an examples folder.

[abcdenis]

queue = LiteQueue(name="tasks", folder=queue_folder)

So one queue will occupy not a file but entire folder?
I'll rather prefer specify filename (and queue name) and let litequeue manage it. Now it looks like "litequeue could change contract in the future and store something more in this folder".

[abcdenis]

I forgot to mention uuid7 added in python 3.14!
https://docs.python.org/3/library/uuid.html#uuid.uuid7

[polyrand]

I forgot to mention uuid7 added in python 3.14!
https://docs.python.org/3/library/uuid.html#uuid.uuid7

I know, but I want to keep this backwards compatible up to Python 3.12, the uuidv7 function is actually copied from https://github.com/python/cpython/blob/f429fb36a16134a2a0e839d0f385eb8fee280de4/Lib/uuid.py

[polyrand]

So one queue will occupy not a file but entire folder?

No, it just lets you specific where to create the SQLite file, and the sqlite file is based on the queue name.

I'll rather prefer specify filename (and queue name) and let litequeue manage it. Now it looks like "litequeue could change contract in the future and store something more in this folder".

I think you're right, I may be over-complicating the API. I was over-focusing on queue_name. Now only the filename should be enough, since now each file can only have a single queue anyway, so file = queue. Just specifying the sqlite file location should be enough. I'll update this.

[polyrand]

Versio 0.12 now uses only a filename as input, the filename is the queue name, since a sqlite file can now only contain a single queue. The README, examples and tests are also updated.

[abcdenis]

Thank you!