rapidgzip-rust

rapidgzip-rust is a Rust 2024, decoder-only implementation of the rapidgzip approach to parallel DEFLATE decompression. It uses the same marker/window strategy for gzip, zlib, and raw-DEFLATE streams, with zlib-rs as the inflate backend, and adds direct parallel paths for BGZF, stored gzip, and dense multi-member gzip archives.

The project provides:

• the rapidgzip-core library crate;
• the rapidgzip-rust binary, distributed by the rapidgzip-rust-cli package;
• verified decoding of single-member gzip, concatenated/multi-member gzip, BGZF, and zlib, plus structurally validated raw DEFLATE;
• both a push API over std::io::Write and an owned std::io::Read + Send stream suitable for parsers such as paraseq;
• opt-in random-access index construction, interoperable index formats, and a decoded-output Read + Seek adapter;
• opt-in newline counting, line-annotated indexes, and indexed seeking by zero-based line number;
• bounded structural analysis of container framing, every DEFLATE block, dynamic Huffman alphabets, symbol composition, and predecessor-window use;
• decoding of non-seekable compressed input such as standard input, a FIFO, a process substitution, or a socket.

Non-seekable input is decoded sequentially by the same zlib-rs path the parallel decoders fall back to, so it receives the same format-specific validation as a regular file but is not decoded in parallel. See Non-seekable input.

The project intentionally does not provide compression. Its random-access API is decoder-only and leaves ordinary decode operations unchanged.

Library installation

Add the decoder to a Rust project with:

cargo add rapidgzip-core

The package name contains a hyphen and its Rust crate name is rapidgzip_core. Rust 1.87 or newer is required.

Container formats

Strict gzip is the compatibility-preserving default. Select zlib or raw DEFLATE explicitly, or opt into detection between gzip and zlib:

use rapidgzip_core::{Decoder, Format};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let _zlib = Decoder::builder().format(Format::Zlib).build()?;
    let _raw = Decoder::builder()
        .format(Format::RawDeflate)
        .expected_uncompressed_size(Some(1_000_000))
        .build()?;
    let _detected = Decoder::builder().auto_detect_format().build()?;
    Ok(())
}

Auto-detection performs an exact, non-consuming two-byte check and never guesses raw DEFLATE, which has no identifying header. Zlib CMF/FLG, its declared history window, and its Adler-32 trailer are checked. Raw DEFLATE has no container checksum or stored size: successful decoding establishes structural validity and exact input consumption, while DecoderBuilder::expected_uncompressed_size can add an exact size contract. DecodeReport::format always contains the concrete detected or selected format and the report remains Copy.

Streaming reader

Decoder::open owns the compressed file and returns a movable DecoderReader implementing Read + Send:

use rapidgzip_core::Decoder;
use std::io::{self, Read};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let decoder = Decoder::builder().decoder_threads(8).build()?;
    let mut reader = decoder.open("reads.fastq.gz")?;

    // A parser can own this as Box<dyn Read + Send> instead.
    io::copy(&mut reader, &mut io::sink())?;
    let report = reader.finish()?;
    assert!(report.member_count >= 1);
    Ok(())
}

Reaching EOF verifies every member footer. Calling finish before EOF discards unread decoded bytes, verifies the remainder, and returns the final report. Dropping the reader early cancels its background work and does not claim that the unread remainder was verified.

Runtime telemetry and worker control

DecoderReader::handle returns a cloneable DecoderHandle. Retain it before moving the reader into paraseq or a Box<dyn Read + Send>:

use rapidgzip_core::{Decoder, DecoderPressure};
use std::io::Read;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let decoder = Decoder::builder().decoder_threads(32).build()?;
    let reader = decoder.open("reads.fastq.gz")?;
    let control = reader.handle();

    // The parser may now take ownership of the reader.
    let mut parser_input: Box<dyn Read + Send> = Box::new(reader);

    // A process-wide scheduler can reduce or restore the decoder's ceiling.
    control.set_worker_limit(8)?;
    let stats = control.stats();
    if matches!(stats.pressure, DecoderPressure::ConsumerBound { .. }) {
        control.set_worker_limit(2)?;
    }

    std::io::copy(&mut parser_input, &mut std::io::sink())?;
    Ok(())
}

The configured worker count is an immutable maximum, not an eager allocation. Workers are created lazily as the empirical controller finds useful parallel work. Lowering the runtime ceiling is nonblocking: in-flight tasks finish, excess workers stop accepting new tasks, and retire after publishing completed results they already own. They can be recreated if the ceiling and measured demand later increase. Sustained backpressure at the final reader handoff automatically reduces the active target and actual decode activity to one. A worker already holding a completed result can remain live, normally parked between bounded handoff retries, until the consumer advances or the decode is cancelled. It remains visible in spawned_workers but not busy_workers.

DecoderStats distinguishes the configured maximum, current application ceiling, adaptive active target, workers executing decode tasks, and live OS threads. It also reports the selected decode path, verified members, produced and consumed bytes, average rates, and a high-level pressure classification. Snapshots use relaxed atomic loads, are deliberately approximate, and describe rapidgzip task activity rather than operating-system CPU utilization.

For ordinary gzip, zlib, and raw-DEFLATE files, telemetry may briefly report DecoderPath::MarkerAdmission. This bounded input-aware screen compares exact zlib-rs work with a useful-width speculative marker wave after applying the configured budget, runtime ceiling, visible processors, and available task count. Its terminal path is Sequential or MarkerWindow; BGZF, stored, and dense-member inputs retain their specialized routes.

Push decoding

Decoder::decode avoids the final reader copy and calls a Write value only from the calling thread. Decoder::decode_path is its filesystem-path counterpart and automatically applies the same regular/non-regular routing as Decoder::open. The writer therefore does not need to implement Send:

use rapidgzip_core::Decoder;
use std::io;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let decoder = Decoder::default();
    let report = decoder.decode_path("reads.fastq.gz", &mut io::sink())?;
    println!("verified {} gzip members", report.member_count);
    Ok(())
}

Parallel input is represented by the ReadAt trait. Implementations are provided for files on Unix and Windows, byte slices, Vec<u8>, Arc<T>, and Box<T>. A custom source must support concurrent positional reads and keep its length and contents stable for the complete decode. Input that cannot satisfy that contract is handled by the entry points below instead.

Non-seekable input

Decoder::stream_reader and Decoder::decode_stream accept any std::io::Read, so compressed input arriving on standard input, a FIFO, a process substitution, or a socket can be decoded without a second decompressor. They mirror Decoder::reader and Decoder::decode, and stream_reader returns the same DecoderReader:

use rapidgzip_core::Decoder;
use std::io::{self, Read};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let decoder = Decoder::default();
    let reader = decoder.stream_reader(io::stdin())?;

    // Still Read + Send, so a parser can own it.
    let mut parser_input: Box<dyn Read + Send> = Box::new(reader);
    io::copy(&mut parser_input, &mut io::sink())?;
    Ok(())
}

Decoder::open does this routing itself, so a program whose input is a path that may or may not be a regular file needs no special case:

use rapidgzip_core::Decoder;
use std::io;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let decoder = Decoder::builder().decoder_threads(8).build()?;
    // A regular file decodes in parallel; a FIFO decodes sequentially.
    let mut reader = decoder.open("reads.fastq.gz")?;
    io::copy(&mut reader, &mut io::sink())?;
    Ok(())
}

What works exactly as it does for a regular file: every supported format, including concatenated and empty gzip members, BGZF's 28-byte EOF member, and fully stored streams. Gzip members require matching CRC32 and ISIZE; zlib requires a matching Adler-32; raw DEFLATE requires a complete final block and exact source end. Truncation, invalid DEFLATE, framing/checksum mismatches, and trailing bytes are errors, and DecoderBuilder::output_limit still fails before emitting bytes past the limit. Reaching EOF, or an Ok report, still means the complete input was verified.

What differs: the four parallel decode paths all need positional reads, so a non-seekable source always uses the sequential path. Telemetry preserves the builder contract: DecoderStats::configured_workers and DecodeReport::decoder_threads remain the requested maximum budget, while DecoderStats::active_workers is one and both spawned_workers and auxiliary_threads are zero. Nothing is spooled to memory or disk: input memory is one DecoderBuilder::input_page_size window, raised to two bytes only when configured smaller so detection can retain its prefix. stream_reader advances its resumable inflater only inside the consumer's Read::read call, so a slow consumer naturally stops reading the producer. Dropping it immediately drops the source; there is no streaming coordinator thread to block or detach.

Structural analysis

Decoder::analyze verifies the complete input while returning structured container, block, alphabet, symbol, and predecessor-window facts. Analysis is an explicit operation, so ordinary decoding and its small Copy DecodeReport are unchanged:

use rapidgzip_core::{AnalyzeOptions, Decoder};
use std::fs::File;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let source = File::open("reads.fastq.gz")?;
    let options = AnalyzeOptions::default()
        .maximum_blocks(250_000)
        .maximum_retained_backreferences(10_000);
    let analysis = Decoder::default().analyze_with_options(&source, options)?;

    println!("{} members", analysis.streams.len());
    for (kind, count) in analysis.block_type_counts() {
        println!("{kind:?}: {count}");
    }
    assert_eq!(analysis.compressed_size_in_bytes, source.metadata()?.len());
    Ok(())
}

The default result retains up to 100,000 streams, 100,000 blocks, and 1 MiB of optional gzip-header metadata across the input. Individual predecessor-window references are omitted by default. Their counts, length histogram, farthest reach, deterministic interval-union count, and window coverage remain exact; each block says how many details were omitted. AnalyzeOptions makes every retention limit explicit, and exceeding a structural limit returns a typed AnalysisErrorKind through DecodeError::Analysis rather than allocating without bound.

The walk is intentionally single-threaded and causal. It keeps one 32 KiB history ring and verifies gzip CRC32/ISIZE or zlib Adler-32 without retaining decoded output. Concatenated and empty gzip members, BGZF (including its EOF member), zlib, raw DEFLATE, format detection, output limits, exact-size contracts, trailing-data rules, positional ReadAt sources, and streaming Read sources are supported. Decoder::analyze_stream is the forward-only counterpart. Timings and rapidgzip-specific text formatting remain CLI presentation data and are not part of the deterministic Analysis value.

Random-access indexes and seeking

Index construction is explicit per decode operation. This keeps the existing DecodeReport small and Copy, and keeps checkpoint-window work out of calls that do not request it:

use rapidgzip_core::{Decoder, DeflateIndex, IndexOptions, IndexedReader};
use std::fs::File;
use std::io::{self, Read, Seek, SeekFrom};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let decoder = Decoder::builder().decoder_threads(8).build()?;
    let source = File::open("reads.fastq.gz")?;
    let indexed = decoder.decode_with_index(
        &source,
        &mut io::sink(),
        IndexOptions::default(),
    )?;

    let mut serialized = File::create("reads.fastq.gz.rgzidx")?;
    indexed.index.write_native(&mut serialized)?;

    let mut serialized = File::open("reads.fastq.gz.rgzidx")?;
    let index = DeflateIndex::read_native(&mut serialized)?;
    let mut reader = IndexedReader::new(File::open("reads.fastq.gz")?, index)?;
    reader.seek(SeekFrom::Start(4_000_000))?;
    let mut buffer = [0; 4096];
    reader.read_exact(&mut buffer)?;
    Ok(())
}

The pull API is Decoder::reader_with_index; its IndexingDecoderReader remains Read + Send, exposes the same telemetry and worker controls as DecoderReader, and returns an IndexedDecodeReport from finish. decode_stream_with_index and stream_reader_with_index can also collect a coarse member-boundary index while consuming forward-only input. The result can be used later only with a stable positional copy of the same compressed bytes.

An existing index can also drive a complete parallel decode. The push API borrows it; the Read + Send API takes an Arc so its stored windows are not cloned into the background coordinator:

use rapidgzip_core::{Decoder, DeflateIndex};
use std::fs::File;
use std::io::{self, Read};
use std::sync::Arc;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let mut serialized = File::open("reads.fastq.gz.rgzidx")?;
    let index = Arc::new(DeflateIndex::read_native(&mut serialized)?);
    let decoder = Decoder::builder().decoder_threads(16).build()?;
    let mut reader = decoder.reader_from_index(
        File::open("reads.fastq.gz")?,
        Arc::clone(&index),
    )?;

    let control = reader.handle();
    control.set_worker_limit(8)?;
    io::copy(&mut reader, &mut io::sink())?;
    let report = reader.finish()?;
    assert!(report.member_count >= 1);
    Ok(())
}

decode_from_index is the corresponding lower-overhead Write operation. Both APIs are strict: the index must match the source, selected container, compressed bit boundaries, and decompressed offsets. They never hide a bad index by falling back to ordinary decoding. Gzip CRC32/ISIZE and zlib Adler-32 remain fully verified because the operation decodes every indexed span from source origin. Concatenated and empty gzip members are preserved, and an imported BGZF .gzi works even though that format omits the final decompressed size. Telemetry identifies DecoderPath::IndexedParallel and exposes the same dynamic worker ceiling as other parallel readers.

DeflateIndex records gzip, BGZF, zlib, or raw-DEFLATE provenance and reads and writes:

• the native versioned format, which preserves all rapidgzip-rust metadata;
• indexed_gzip GZIDX versions 0/1 (writing version 1);
• htslib BGZF .gzi indexes; and
• gztool version 0/1 indexes.

Format parsers apply explicit checkpoint and window-allocation limits through IndexReadOptions. The native format represents every container. .gzi export requires an index proven to come from BGZF; GZIDX and gztool export require gzip-family provenance; gztool line-aware export requires real line counters and never invents them. The CLI detects formats from a bounded prefix, streams index parsing, rejects trailing bytes, and writes exports through a same-directory temporary file so failed conversions cannot truncate an existing index.

Line metadata is collected only when requested. Enabling DecoderBuilder::count_lines adds DecodeReport::line_count; combining it with an explicit indexing operation also annotates every retained checkpoint:

use rapidgzip_core::{Decoder, IndexOptions, IndexedReader};
use std::fs::File;
use std::io;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let decoder = Decoder::builder()
        .decoder_threads(8)
        .count_lines(true)
        .build()?;
    let source = File::open("reads.fastq.gz")?;
    let indexed = decoder.decode_with_index(
        &source,
        &mut io::sink(),
        IndexOptions::default(),
    )?;
    assert_eq!(indexed.decode.line_count, indexed.index.total_line_count());

    let mut reader = IndexedReader::new(source, indexed.index)?;
    let byte_offset = reader.seek_to_line(1_000_000)?;
    println!("line 1000000 begins at decoded byte {byte_offset}");
    Ok(())
}

A line offset is the number of b'\n' bytes preceding a position. Line zero begins at decoded byte zero. A final unterminated line does not increase line_count, while seek_to_line can still reach it after scanning from the nearest checkpoint. Counting happens once on final ordered bytes, after marker resolution, and is disabled by default. DecodeReport remains Copy because the result is an optional scalar. A line-aware index is published only when every retained checkpoint received an exact count; partial metadata is never presented as complete. Strict full-stream decoding with line counting enabled also recomputes imported per-checkpoint and total line counts, rejecting structurally valid but incorrect navigation metadata. Without line counting, imported line offsets remain explicitly trusted navigation data.

IndexedReader validates the index and any recorded source size before use. Resuming at a gzip member or zlib-header checkpoint verifies that complete framing unit, including bytes discarded during a seek. A raw stream has no checksum to verify. An interior checkpoint cannot authenticate bytes skipped before it because persisted indexes do not carry prefix checksum state.

Command-line installation and use

Install the CLI package with:

cargo install rapidgzip-rust-cli

The installed executable is named rapidgzip-rust:

# Decompress to stdout.
rapidgzip-rust -P 16 reads.fastq.gz > reads.fastq

# Verify every member and discard decoded output.
rapidgzip-rust -P 16 --test reads.fastq.gz

# Count decoded bytes or newline bytes.
rapidgzip-rust --count reads.fastq.gz
rapidgzip-rust --count-lines reads.fastq.gz

# Decode and count in one pass. Counts use stderr when payload uses stdout.
rapidgzip-rust -c --count --count-lines reads.fastq.gz > reads.fastq

# Build a native index, then use it for strict full-stream parallel decoding.
rapidgzip-rust --test --export-index reads.rgzidx \
  --index-format native reads.fastq.gz
rapidgzip-rust --import-index reads.rgzidx -c reads.fastq.gz > reads.fastq

# Extract byte and zero-based line ranges in the requested order.
rapidgzip-rust --ranges '1KiB@4MiB,10L@1000L' -c reads.fastq.gz

# Authenticate the complete source before an imported random-access read.
rapidgzip-rust --import-index reads.rgzidx --verify \
  --ranges '10L@1000L' -c reads.fastq.gz

# Export gztool version 1 with real line counters.
rapidgzip-rust --count-lines --export-index reads.gzi \
  --index-format gztool-with-lines reads.fastq.gz

# Refuse to overwrite an existing output file.
rapidgzip-rust -P 16 --output reads.fastq reads.fastq.gz

# Read standard input, decoded sequentially and verified the same way.
cat reads.fastq.gz | rapidgzip-rust - > reads.fastq

# Print rapidgzip-compatible framing and DEFLATE block analysis.
rapidgzip-rust --analyze reads.fastq.gz

# Retain bounded per-reference detail; aggregate summaries are always exact.
rapidgzip-rust --analyze --verbose \
  --analysis-reference-limit 10000 reads.fastq.gz

# Streaming analysis uses the same bounded forward walk.
cat reads.fastq.gz | rapidgzip-rust --analyze -

# A FIFO or process substitution given as a path is routed the same way.
rapidgzip-rust <(some_producer) > reads.fastq

The CLI auto-detects gzip or zlib by default; raw DEFLATE requires --format raw-deflate. --chunk-size controls the decoded handoff size in KiB. Output defaults to stdout when redirected; at a terminal, a regular input derives a safe output filename using case-insensitive compression suffixes. Existing files require --force. Index export defaults to the native, format-neutral representation; gzip-specific interoperable formats must be selected explicitly.

-P/--decoder-parallelism (--threads is an alias) is a maximum decoder-worker budget. Parallel paths bootstrap from the smaller of the affinity-visible processors and this requested budget, then create more workers only while measurements justify them. They may retain fewer active workers when the input exposes less parallel work, the consumer is backpressured, or additional concurrency reduces throughput.

Imported indexes are never advisory. Full-stream operations use decode_from_index, and malformed, incomplete, or source-mismatched indexes fail without falling back. Range extraction uses IndexedReader; a line range requires complete line metadata in an imported index or builds a line-aware index first. A seek from an interior DEFLATE checkpoint cannot authenticate the skipped prefix. For that reason, --verify on an imported range performs a complete strict indexed decode before extraction. Decoder options that would otherwise be ignored by an unverified imported range are rejected and explain that --verify is required. --no-verify, --sparse-windows, and the sequential and locked-read I/O methods are rejected because the current implementation cannot honor their semantics. Outside imported ranges, complete decode paths already verify their selected framing. --no-sparse-windows, -d, and -k remain compatibility aliases. This is a deliberately compatible subset, not a claim that every rapidgzip CLI option is implemented.

Correctness and resource behavior

• Every accepted gzip member is terminated by an actual final DEFLATE block and checked against its CRC32 and modulo-2^32 uncompressed size.
• Every accepted zlib stream has a valid CMF/FLG header, respects its declared window, ends exactly once, and matches its Adler-32 trailer.
• Raw DEFLATE must end exactly after its final block; it is structurally validated but not described as checksum-authenticated.
• Concatenated members, empty members, optional gzip headers, BGZF data, and the conventional BGZF EOF member are supported.
• Bytes resembling a gzip header inside DEFLATE data are never trusted as a boundary without independent inflation, footer authentication, and exact adjacency to the preceding verified member.
• Trailing non-gzip data, truncated input, invalid DEFLATE, and footer mismatches are errors. Output written before an error is not rolled back.
• DecoderBuilder::output_limit bounds total decoded output. The decoder fails before emitting bytes beyond the configured limit.
• DecoderBuilder::expected_uncompressed_size optionally requires one exact total for any format, rejecting overruns before handoff and underruns at end.
• Work queues and reader handoff are bounded. Memory still scales with active workers and configured chunk sizes; the defaults are intended for throughput on general-purpose machines rather than minimum memory use.
• Structural analysis retains one output-history window plus explicitly bounded stream, block, optional-header, alphabet, and detailed-reference results. Checked counter or allocation failure is reported as a typed analysis error; exact summaries do not require detailed references.
• All of the above hold identically for non-seekable input, because it runs the same sequential decoder that the parallel paths already use as their authoritative fallback. It is not decoded in parallel, and the telemetry says so rather than reporting an unused thread budget.

There is no unsafe public API. Private unsafe code is limited to the zlib-rs ABI, checked SIMD operations, proven initialized-buffer operations, and the audited Send implementation for exclusively owned resumable inflate state; each site has a local safety argument. See SAFETY.md for the complete audit.

Performance status

The current implementation clears its zlib-ng-backed C++ rapidgzip parity gate on the public FASTQ workload and on synthetic single-member, concatenated, and BGZF corpora. On the public FASTQ workload it also exceeds the ISA-L-enabled C++ build at multi-worker budgets; the remaining measured ISA-L gap is the one-worker case.

Performance is workload- and machine-dependent. Reproduce the published measurements rather than treating these results as a universal speed claim:

• ARCHITECTURE.md describes the algorithm and scheduling paths.
• BENCHMARKING.md records corpora, commands, versions, thread counts, hashes, throughput, and memory measurements.
• PERFORMANCE_AUDIT.md records the ISA-L comparison and optimization audit.
• CHANGELOG.md summarizes each published release.

The release runner can generate self-verified single-member, multi-member, true-BGZF, stored, zlib, and raw-DEFLATE controls without downloading data:

benchmarks/run-fair.sh --generate --cpus 0-43

ISA-L, zlib-ng, and gzippy competitors are configured through explicit paths; the runner records their identities and versions, requires decoded SHA-256 parity before timing, rotates measured order, and writes raw observations plus deterministic summaries under target/bench-results. See BENCHMARKING.md for the complete interface and release-host requirements.

Platform and compatibility policy

• Rust edition: 2024
• Minimum supported Rust version (MSRV): 1.87
• First-class positional file sources: Unix and Windows
• SIMD: runtime-dispatched x86-64 AVX2/SSE4.1 where applicable, baseline NEON on AArch64, and scalar fallbacks
• Inflate backend: zlib-rs through libz-rs-sys

Pre-1.0 releases should be treated as an evolving initial API. Correct format-specific decoding, multi-member gzip verification, and the Read + Send contract are foundational; configuration details may be refined as additional workloads are measured.

Development

cargo fmt --all --check
cargo clippy --workspace --all-targets -- -D warnings
cargo test --workspace --all-targets
RUSTDOCFLAGS="-D warnings" cargo doc --workspace --no-deps

The integration suite covers gzip, zlib, raw DEFLATE, multi-member streams, BGZF, corruption, format detection across short/interrupted reads, false header candidates, output limits and exact sizes, index construction, indexed full-stream parallel decode, seeking, cancellation, one-byte consumer buffers, line counting and seeking, CLI index/range workflows, and direct paraseq consumption. Generated benchmark corpora and large sequencing files are deliberately not stored in the repository.

Releasing

Maintainers can validate a prospective release without retaining version-file changes:

scripts/bump_and_publish.sh --dry-run 0.2.0

The dry run accepts any clean branch, which lets a release-preparation pull request exercise the exact package contents before merge. Omitting --dry-run requires main, updates the workspace version, repeats formatting, lint, test, documentation, and package checks, creates and pushes a release commit and annotated tag, publishes the crates to crates.io, and creates a GitHub release from that version's changelog section. A real release requires a clean main branch plus Cargo and GitHub authentication, and asks for confirmation before external changes; --yes is available for an intentional unattended release.

License

The repository is distributed under the combined terms of BSD-3-Clause and MIT. See LICENSE-BSD-3-CLAUSE and LICENSE-MIT.