diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 0000000..a792d5f --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,15 @@ +# Changelog + +## [0.1.0] - 2026-06-08 + +Initial release of biofuse. + +- Read-only FUSE views of VCF Zarr data in standard bioinformatics formats, + generated on demand using vcztools. +- `mount-plink` — PLINK 1.9 binary view (`.bed`/`.bim`/`.fam`), diploid input. +- `mount-bgen` — BGEN view (`.bgen`/`.sample`/`.bgen.bgi`) using zlib + level 0 fixed-size blocks for O(1) random access; haploid supported, + mixed ploidy not supported by the mount. +- Inherits vcztools view filter / backend / log options on both commands, plus + `--basename` and `--access-log`. +- Requires vcztools >= 0.2. diff --git a/README.md b/README.md index e5e3749..4f5e5db 100644 --- a/README.md +++ b/README.md @@ -1,3 +1,6 @@ +[![CI](https://github.com/sgkit-dev/biofuse/actions/workflows/ci.yml/badge.svg?branch=main)](https://github.com/sgkit-dev/biofuse/actions/workflows/ci.yml) +[![PyPI Downloads](https://static.pepy.tech/badge/biofuse)](https://pepy.tech/projects/biofuse) + # biofuse Read-only views of VCF Zarr (VCZ) data in standard bioinformatics file formats @@ -6,37 +9,92 @@ via a FUSE filesystem. Currently supported views: - **PLINK 1.9 binary** (`.bed` / `.bim` / `.fam`) — via `mount-plink`. - **Oxford BGEN** (`.bgen` / `.sample` / `.bgen.bgi`) — via `mount-bgen`. -## Status - -Pre-release. The mount serves the streaming file (`.bed` / `.bgen`) on -demand via a worker subprocess running the matching -[`vcztools`](https://github.com/sgkit-dev/vcztools) encoder -(`BedEncoder` / `BgenEncoder`); the static sidecars are computed once -at mount time and held in the worker's memory. - -The mounted PLINK view supports the access patterns of `plink1.9` and -`plink2` for typical analysis commands (`--freq`, `--missing`, -`--hardy`, etc.) — see `tests/test_plink_apps.py` for the verified set. -The mounted BGEN view always uses zlib level 0 (stored, fixed-size -blocks) for O(1) random access; `bgenix` / `qctool` parity checks live -in `tests/test_bgen_apps.py`. +The streaming file (`.bed` / `.bgen`) is generated on demand using the +matching [`vcztools`](https://github.com/sgkit-dev/vcztools) encoder; the +static sidecars are computed once at mount time. + +## Stability and correctness + +A core design principle of biofuse is that **the mount must never become +unresponsive**. All the work of decoding VCF Zarr and encoding it into PLINK +or BGEN bytes is delegated to +[vcztools](https://github.com/sgkit-dev/vcztools); biofuse itself does one +thing — present that data as a correct, dependable read-only filesystem. +Keeping the two responsibilities separate keeps the surface biofuse has to get +exactly right small. + +- **The filesystem stays responsive under load.** Encoding runs off the + filesystem's request-handling path, and every read and open is bounded by a + timeout: a slow or stuck encode returns a normal I/O error (`EIO` / + `EAGAIN`) rather than blocking. One wedged file handle cannot freeze the + others, and unmount never hangs. +- **Failures are contained.** An error inside the encoder surfaces to the + caller as an I/O error, not a crash — the mount keeps serving every other + file. +- **The view is read-only and immutable.** Writes, truncation and appends are + rejected with `EROFS`; the sidecars are computed once when the mount starts + and served unchanged for its lifetime. +- **POSIX behaviour is tested.** A dedicated filesystem test + harness (`fs_tests/`) exercises syscall semantics (`read` / `pread` / + `lseek`, `stat`, `mmap`, directory listing, write rejection), cross-checks + the served bytes against a reference, and runs read-stress and liveness + probes that confirm the mount stays responsive while the streaming file is + saturated. + +## Performance and access patterns + +biofuse is optimised for **linear, sequential reads** — the access pattern +used by the majority of downstream tools, which stream variants start-to-end. +The streaming `.bed` / `.bgen` file is encoded on demand as the consumer reads +forward, and bytes already produced are buffered, so reading straight through +the file does no redundant work. The mounts are verified against `plink1.9` +and `plink2` (`--bfile`, `--freq`, `--missing`, `--hardy`, …) for PLINK, and +`bgenix`, `qctool`, REGENIE, SAIGE, BOLT-LMM and `plink2 --bgen` for BGEN. + +Random and backward access still work, but are slower: seeking backwards or +skipping far ahead can make biofuse re-encode from an earlier point in the +file. The kernel page cache holds bytes that have already been served, so +re-reading a region — and multi-pass tools that scan the file more than once +(e.g. flashpca) — stays cheap once the data is warm. + +For BGEN, the `.bgen` payload uses zlib level 0 (stored, fixed-size variant +blocks) together with the `.bgen.bgi` index, so a tool can fetch an individual +variant by byte range without decompressing or re-encoding the rest of the +file — variant-targeted access (e.g. `bgenix -v`) is efficient as well as +whole-file scans. + +The sidecar files (`.bim` / `.fam` / `.sample` / `.bgen.bgi`) are computed +once when the mount starts, so reads of them are always fast regardless of +access order. These can be suppressed individually where not needed +(e.g., the .bgen.bgi can be large and is not needed for many workloads). + +Because the streaming file is produced on demand, a read that stalls beyond an +internal timeout surfaces as `EIO` rather than blocking indefinitely; in +practice this only appears under pathological random-access load. ## Install -biofuse depends on libfuse 3 system headers when building from source: +biofuse depends on libfuse 3 system headers (`pyfuse3` builds from source): ```bash sudo apt-get install -y fuse3 libfuse3-dev pkg-config ``` -Then with [`uv`](https://docs.astral.sh/uv/): +Then: ```bash -uv sync --group test +python -m pip install biofuse # or: uv pip install biofuse ``` -vcztools is currently consumed as a sibling-directory path dependency -(`../vcztools`); see `pyproject.toml`. +### Remote and zipped stores + +The `vcz_url` argument and the inherited `--backend-storage` / +`--storage-option` options accept cloud, fsspec, and HTTP stores, plus +`.vcz.zip` files. biofuse depends on bare `vcztools`; to mount cloud-backed +stores install the matching vcztools extra, e.g. +`pip install 'vcztools[obstore]'` or `pip install 'vcztools[icechunk]'`. See +the [vcztools documentation](https://sgkit-dev.github.io/vcztools/) for the +available storage backends. ## Usage @@ -66,6 +124,11 @@ Example: ```bash mkdir /tmp/plink-mnt biofuse mount-plink ./sample.vcz /tmp/plink-mnt & +# The mount runs in the foreground, so it is backgrounded with `&`. It is +# not ready the instant the process starts — it first opens the VCZ and +# builds the sidecars — so wait for the mounted file to appear before +# running the consumer tool. +until [ -e /tmp/plink-mnt/sample.bed ]; do sleep 0.1; done plink1.9 --bfile /tmp/plink-mnt/sample --freq --out ./out fusermount3 -u /tmp/plink-mnt ``` @@ -81,8 +144,7 @@ Mounts a read-only directory at `/mount/dir` containing uses zlib level 0 (stored, fixed-size variant blocks) so byte-range random access is O(1); downstream tools (bgenix, qctool, REGENIE, SAIGE, BOLT-LMM, plink2 `--bgen`) consume the mount unchanged. The -`.bgen.bgi` SQLite sidecar is generated once at mount time and held in -the worker's memory alongside `.sample`. +`.bgen.bgi` SQLite sidecar and `.sample` are generated once at mount time. Options mirror `mount-plink`: `--basename`, `--access-log`, and the shared bcftools-style filter / backend / log set inherited from @@ -94,6 +156,8 @@ Example: ```bash mkdir /tmp/bgen-mnt biofuse mount-bgen ./sample.vcz /tmp/bgen-mnt & +# Wait for the mount to come up before reading from it (see mount-plink above). +until [ -e /tmp/bgen-mnt/sample.bgen ]; do sleep 0.1; done bgenix -g /tmp/bgen-mnt/sample.bgen -list fusermount3 -u /tmp/bgen-mnt ``` @@ -129,6 +193,3 @@ uv run prek install # install git pre-commit hook (one-off) uv run --only-group=lint prek -c prek.toml run --all-files ``` -## Licence - -Apache 2.0. See `LICENSE`. diff --git a/pyproject.toml b/pyproject.toml index 4eb47c1..aabe4f9 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -18,7 +18,7 @@ dependencies = [ ] requires-python = ">=3.11" classifiers = [ - "Development Status :: 3 - Alpha", + "Development Status :: 4 - Beta", "Operating System :: POSIX :: Linux", "Operating System :: MacOS :: MacOS X", "Intended Audience :: Science/Research",