Skip to content

Open packaged SQLite DBs in read-only immutable mode to avoid locking errors under SLURM#294

Open
wwood wants to merge 1 commit intomainfrom
codex/github-mention-sqlite-disk-i/o-error-when-running-under-s
Open

Open packaged SQLite DBs in read-only immutable mode to avoid locking errors under SLURM#294
wwood wants to merge 1 commit intomainfrom
codex/github-mention-sqlite-disk-i/o-error-when-running-under-s

Conversation

@wwood
Copy link
Copy Markdown
Owner

@wwood wwood commented Mar 31, 2026

Motivation

  • Triggered by reproducible SQLite disk I/O error failures observed when running under SLURM on some HPC/shared filesystems, likely due to file locking/journaling interactions.
  • Packaged SingleM SQLite DBs are read-only at query time, so opening them in an immutable read-only mode should avoid lock/journal activity and be more robust on parallel/network filesystems.

Description

  • Added _acquire_readonly_engine() helpers to SequenceDatabase and MetapackageReadNameStore that construct a quoted absolute path and open the DB with a SQLite URI including mode=ro, immutable=1, and uri=true.
  • Replaced direct create_engine("sqlite:///...") calls for read/query access with the new helper so packaged DBs (e.g. otus.sqlite3 and read_taxonomies.sqlite3) are opened read-only immutable when acquired.
  • Kept DB creation/import paths unchanged (only query-time acquire paths are switched to immutable/read-only), and used urllib.parse.quote + os.path.abspath to safely build file URIs.
  • Added test/test_sqlite_readonly_mode.py unit tests that assert the acquire paths construct the expected read-only immutable SQLite URIs.

Testing

  • Ran the project test suite with pixi run -e dev pytest test as required and the full suite completed with 225 passed, 24 skipped.
  • The new test_sqlite_readonly_mode.py tests pass as part of that run and validate the read-only immutable engine construction.

Codex Task

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant