From 2de9247cac3d04cbfe9eb25e50ccd166b77c6377 Mon Sep 17 00:00:00 2001 From: Jerome Kelleher Date: Mon, 8 Jun 2026 13:54:41 +0100 Subject: [PATCH 1/3] Prepare for 0.1.0 release Add CHANGELOG, switch README to PyPI install with badges, drop pre-release/path-dependency notes, and bump the development status classifier to Beta. --- CHANGELOG.md | 15 +++++++++++++++ README.md | 33 ++++++++++++++++++++------------- pyproject.toml | 2 +- 3 files changed, 36 insertions(+), 14 deletions(-) create mode 100644 CHANGELOG.md diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 0000000..81f5167 --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,15 @@ +# Changelog + +## [0.1.0] - 2026-06-08 + +Initial release of biofuse. + +- Read-only FUSE views of VCF Zarr data in standard bioinformatics formats, + generated on demand using vcztools. +- `mount-plink` — PLINK 1.9 binary view (`.bed`/`.bim`/`.fam`), diploid input. +- `mount-bgen` — Oxford BGEN view (`.bgen`/`.sample`/`.bgen.bgi`) using zlib + level 0 fixed-size blocks for O(1) random access; haploid supported, + mixed ploidy not supported by the mount. +- Inherits vcztools view filter / backend / log options on both commands, plus + `--basename` and `--access-log`. +- Requires vcztools >= 0.2. diff --git a/README.md b/README.md index e5e3749..f7b7799 100644 --- a/README.md +++ b/README.md @@ -1,3 +1,6 @@ +[![CI](https://github.com/sgkit-dev/biofuse/actions/workflows/ci.yml/badge.svg?branch=main)](https://github.com/sgkit-dev/biofuse/actions/workflows/ci.yml) +[![PyPI Downloads](https://static.pepy.tech/badge/biofuse)](https://pepy.tech/projects/biofuse) + # biofuse Read-only views of VCF Zarr (VCZ) data in standard bioinformatics file formats @@ -6,13 +9,11 @@ via a FUSE filesystem. Currently supported views: - **PLINK 1.9 binary** (`.bed` / `.bim` / `.fam`) — via `mount-plink`. - **Oxford BGEN** (`.bgen` / `.sample` / `.bgen.bgi`) — via `mount-bgen`. -## Status +The streaming file (`.bed` / `.bgen`) is generated on demand using the +matching [`vcztools`](https://github.com/sgkit-dev/vcztools) encoder; the +static sidecars are computed once at mount time. -Pre-release. The mount serves the streaming file (`.bed` / `.bgen`) on -demand via a worker subprocess running the matching -[`vcztools`](https://github.com/sgkit-dev/vcztools) encoder -(`BedEncoder` / `BgenEncoder`); the static sidecars are computed once -at mount time and held in the worker's memory. +## Supported access patterns The mounted PLINK view supports the access patterns of `plink1.9` and `plink2` for typical analysis commands (`--freq`, `--missing`, @@ -23,20 +24,27 @@ in `tests/test_bgen_apps.py`. ## Install -biofuse depends on libfuse 3 system headers when building from source: +biofuse depends on libfuse 3 system headers (`pyfuse3` builds from source): ```bash sudo apt-get install -y fuse3 libfuse3-dev pkg-config ``` -Then with [`uv`](https://docs.astral.sh/uv/): +Then: ```bash -uv sync --group test +python -m pip install biofuse # or: uv pip install biofuse ``` -vcztools is currently consumed as a sibling-directory path dependency -(`../vcztools`); see `pyproject.toml`. +### Remote and zipped stores + +The `vcz_url` argument and the inherited `--backend-storage` / +`--storage-option` options accept cloud, fsspec, and HTTP stores, plus +`.vcz.zip` files. biofuse depends on bare `vcztools`; to mount cloud-backed +stores install the matching vcztools extra, e.g. +`pip install 'vcztools[obstore]'` or `pip install 'vcztools[icechunk]'`. See +the [vcztools documentation](https://sgkit-dev.github.io/vcztools/) for the +available storage backends. ## Usage @@ -81,8 +89,7 @@ Mounts a read-only directory at `/mount/dir` containing uses zlib level 0 (stored, fixed-size variant blocks) so byte-range random access is O(1); downstream tools (bgenix, qctool, REGENIE, SAIGE, BOLT-LMM, plink2 `--bgen`) consume the mount unchanged. The -`.bgen.bgi` SQLite sidecar is generated once at mount time and held in -the worker's memory alongside `.sample`. +`.bgen.bgi` SQLite sidecar and `.sample` are generated once at mount time. Options mirror `mount-plink`: `--basename`, `--access-log`, and the shared bcftools-style filter / backend / log set inherited from diff --git a/pyproject.toml b/pyproject.toml index 4eb47c1..aabe4f9 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -18,7 +18,7 @@ dependencies = [ ] requires-python = ">=3.11" classifiers = [ - "Development Status :: 3 - Alpha", + "Development Status :: 4 - Beta", "Operating System :: POSIX :: Linux", "Operating System :: MacOS :: MacOS X", "Intended Audience :: Science/Research", From 9bdd4ca415bc4f51aaa5a7b5288cf2b45cc5c439 Mon Sep 17 00:00:00 2001 From: Jerome Kelleher Date: Mon, 8 Jun 2026 14:06:18 +0100 Subject: [PATCH 2/3] Rewrite README access-patterns section for users Lead with the linear-sequential-read optimisation, describe random/ backward access cost and page-cache behaviour, and drop test-file references in favour of user-relevant properties. --- README.md | 37 +++++++++++++++++++++++++++++-------- 1 file changed, 29 insertions(+), 8 deletions(-) diff --git a/README.md b/README.md index f7b7799..e7a1441 100644 --- a/README.md +++ b/README.md @@ -13,14 +13,35 @@ The streaming file (`.bed` / `.bgen`) is generated on demand using the matching [`vcztools`](https://github.com/sgkit-dev/vcztools) encoder; the static sidecars are computed once at mount time. -## Supported access patterns - -The mounted PLINK view supports the access patterns of `plink1.9` and -`plink2` for typical analysis commands (`--freq`, `--missing`, -`--hardy`, etc.) — see `tests/test_plink_apps.py` for the verified set. -The mounted BGEN view always uses zlib level 0 (stored, fixed-size -blocks) for O(1) random access; `bgenix` / `qctool` parity checks live -in `tests/test_bgen_apps.py`. +## Performance and access patterns + +biofuse is optimised for **linear, sequential reads** — the access pattern +used by the majority of downstream tools, which stream variants front-to-back. +The streaming `.bed` / `.bgen` file is encoded on demand as the consumer reads +forward, and bytes already produced are buffered, so reading straight through +the file does no redundant work. The mounts are verified against `plink1.9` +and `plink2` (`--bfile`, `--freq`, `--missing`, `--hardy`, …) for PLINK, and +`bgenix`, `qctool`, REGENIE, SAIGE, BOLT-LMM and `plink2 --bgen` for BGEN. + +Random and backward access still work, but are slower: seeking backwards or +skipping far ahead can make biofuse re-encode from an earlier point in the +file. The kernel page cache holds bytes that have already been served, so +re-reading a region — and multi-pass tools that scan the file more than once +(e.g. flashpca) — stays cheap once the data is warm. + +For BGEN, the `.bgen` payload uses zlib level 0 (stored, fixed-size variant +blocks) together with the `.bgen.bgi` index, so a tool can fetch an individual +variant by byte range without decompressing or re-encoding the rest of the +file — variant-targeted access (e.g. `bgenix -v`) is efficient as well as +whole-file scans. + +The sidecar files (`.bim` / `.fam` / `.sample` / `.bgen.bgi`) are computed +once when the mount starts, so reads of them are always fast regardless of +access order. + +Because the streaming file is produced on demand, a read that stalls beyond an +internal timeout surfaces as `EIO` rather than blocking indefinitely; in +practice this only appears under pathological random-access load. ## Install From 6654f4aa5dbfb4a4376fc0807687037df514b6ee Mon Sep 17 00:00:00 2001 From: Jerome Kelleher Date: Mon, 8 Jun 2026 14:52:44 +0100 Subject: [PATCH 3/3] Add stability section and harden README examples Add a headline Stability and correctness section, and make the usage examples wait for the FUSE mount to come up before reading from it. --- CHANGELOG.md | 2 +- README.md | 43 ++++++++++++++++++++++++++++++++++++++----- 2 files changed, 39 insertions(+), 6 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 81f5167..a792d5f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,7 +7,7 @@ Initial release of biofuse. - Read-only FUSE views of VCF Zarr data in standard bioinformatics formats, generated on demand using vcztools. - `mount-plink` — PLINK 1.9 binary view (`.bed`/`.bim`/`.fam`), diploid input. -- `mount-bgen` — Oxford BGEN view (`.bgen`/`.sample`/`.bgen.bgi`) using zlib +- `mount-bgen` — BGEN view (`.bgen`/`.sample`/`.bgen.bgi`) using zlib level 0 fixed-size blocks for O(1) random access; haploid supported, mixed ploidy not supported by the mount. - Inherits vcztools view filter / backend / log options on both commands, plus diff --git a/README.md b/README.md index e7a1441..4f5e5db 100644 --- a/README.md +++ b/README.md @@ -13,10 +13,38 @@ The streaming file (`.bed` / `.bgen`) is generated on demand using the matching [`vcztools`](https://github.com/sgkit-dev/vcztools) encoder; the static sidecars are computed once at mount time. +## Stability and correctness + +A core design principle of biofuse is that **the mount must never become +unresponsive**. All the work of decoding VCF Zarr and encoding it into PLINK +or BGEN bytes is delegated to +[vcztools](https://github.com/sgkit-dev/vcztools); biofuse itself does one +thing — present that data as a correct, dependable read-only filesystem. +Keeping the two responsibilities separate keeps the surface biofuse has to get +exactly right small. + +- **The filesystem stays responsive under load.** Encoding runs off the + filesystem's request-handling path, and every read and open is bounded by a + timeout: a slow or stuck encode returns a normal I/O error (`EIO` / + `EAGAIN`) rather than blocking. One wedged file handle cannot freeze the + others, and unmount never hangs. +- **Failures are contained.** An error inside the encoder surfaces to the + caller as an I/O error, not a crash — the mount keeps serving every other + file. +- **The view is read-only and immutable.** Writes, truncation and appends are + rejected with `EROFS`; the sidecars are computed once when the mount starts + and served unchanged for its lifetime. +- **POSIX behaviour is tested.** A dedicated filesystem test + harness (`fs_tests/`) exercises syscall semantics (`read` / `pread` / + `lseek`, `stat`, `mmap`, directory listing, write rejection), cross-checks + the served bytes against a reference, and runs read-stress and liveness + probes that confirm the mount stays responsive while the streaming file is + saturated. + ## Performance and access patterns biofuse is optimised for **linear, sequential reads** — the access pattern -used by the majority of downstream tools, which stream variants front-to-back. +used by the majority of downstream tools, which stream variants start-to-end. The streaming `.bed` / `.bgen` file is encoded on demand as the consumer reads forward, and bytes already produced are buffered, so reading straight through the file does no redundant work. The mounts are verified against `plink1.9` @@ -37,7 +65,8 @@ whole-file scans. The sidecar files (`.bim` / `.fam` / `.sample` / `.bgen.bgi`) are computed once when the mount starts, so reads of them are always fast regardless of -access order. +access order. These can be suppressed individually where not needed +(e.g., the .bgen.bgi can be large and is not needed for many workloads). Because the streaming file is produced on demand, a read that stalls beyond an internal timeout surfaces as `EIO` rather than blocking indefinitely; in @@ -95,6 +124,11 @@ Example: ```bash mkdir /tmp/plink-mnt biofuse mount-plink ./sample.vcz /tmp/plink-mnt & +# The mount runs in the foreground, so it is backgrounded with `&`. It is +# not ready the instant the process starts — it first opens the VCZ and +# builds the sidecars — so wait for the mounted file to appear before +# running the consumer tool. +until [ -e /tmp/plink-mnt/sample.bed ]; do sleep 0.1; done plink1.9 --bfile /tmp/plink-mnt/sample --freq --out ./out fusermount3 -u /tmp/plink-mnt ``` @@ -122,6 +156,8 @@ Example: ```bash mkdir /tmp/bgen-mnt biofuse mount-bgen ./sample.vcz /tmp/bgen-mnt & +# Wait for the mount to come up before reading from it (see mount-plink above). +until [ -e /tmp/bgen-mnt/sample.bgen ]; do sleep 0.1; done bgenix -g /tmp/bgen-mnt/sample.bgen -list fusermount3 -u /tmp/bgen-mnt ``` @@ -157,6 +193,3 @@ uv run prek install # install git pre-commit hook (one-off) uv run --only-group=lint prek -c prek.toml run --all-files ``` -## Licence - -Apache 2.0. See `LICENSE`.