Harden WASM decoders against corrupt headers, pin numcodecs interop, add docs

AGENTS.md

@@ -0,0 +1,142 @@
+# AGENTS.md — rumcodecs
+
+## Overview
+
+rumcodecs is a pnpm workspace providing **buffer compression codecs for
+JavaScript** — a Rust/WASM reimplementation of
+[numcodecs.js](https://github.com/manzt/numcodecs.js) built for Zarr. Five
+codec classes are exported from one npm package,
+`@fideus-labs/rumcodecs`:
+
+- **`Blosc`** — the Blosc meta-compressor via the pure-Rust
+  [blusc](https://crates.io/crates/blusc) crate, compiled to WASM (SIMD128).
+- **`LZ4`** — [lz4_flex](https://crates.io/crates/lz4_flex) via WASM, using
+  numcodecs framing (4-byte little-endian original size + LZ4 block).
+- **`Zstd`** — the [zstd](https://crates.io/crates/zstd) crate via WASM,
+  standard Zstd frames.
+- **`GZip`** / **`Zlib`** — pure JS via
+  [fflate](https://github.com/101arrowz/fflate) (a peer dependency); no WASM
+  involved.
+
+The contract is **drop-in compatibility**: identical API surface to
+numcodecs.js and identical byte formats to Python numcodecs, so chunks are
+interchangeable across ecosystems. Three layers hold that up:
+[`test/compat.test.ts`](./test/compat.test.ts) asserts the API surface,
+[`test/interop.test.ts`](./test/interop.test.ts) decodes buffers actually
+written by Python `numcodecs`, and the Rust unit tests in
+[`lib.rs`](./crates/rumcodecs-wasm/src/lib.rs) pin the framing we emit. Note
+what the per-codec suites do _not_ prove: they only show encode/decode symmetry,
+which would survive both directions drifting off the format together. The
+interop fixtures are the ones that fail when that happens.
+
+## Modules
+
+| Module                                         | Read                                                                     |
+| ---------------------------------------------- | ------------------------------------------------------------------------ |
+| TypeScript codec classes                       | [./src/](./src/)                                                         |
+| Shared `Codec` / `CodecConstructor` interfaces | [./src/types.ts](./src/types.ts)                                         |
+| Rust WASM bindings (blusc, lz4_flex, zstd)     | [./crates/rumcodecs-wasm/src/lib.rs](./crates/rumcodecs-wasm/src/lib.rs) |
+| Test suites (round-trip, compat, interop)      | [./test/](./test/)                                                       |
+| Benchmark package (vs numcodecs.js)            | [./benchmark/](./benchmark/)                                             |
+| CI / release workflows                         | [./.github/workflows/](./.github/workflows/)                             |
+
+## Context to load on demand
+
+| Task                              | Read                                                                                                                                                           |
+| --------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| Add or change a codec class       | [./src/types.ts](./src/types.ts), the sibling codec in [./src/](./src/), its test in [./test/](./test/)                                                        |
+| numcodecs API-compatibility rules | [./test/compat.test.ts](./test/compat.test.ts)                                                                                                                 |
+| Byte-format / interop questions   | [./test/interop.test.ts](./test/interop.test.ts), [./test/fixtures/README.md](./test/fixtures/README.md)                                                       |
+| Change or add WASM exports        | [./crates/rumcodecs-wasm/src/lib.rs](./crates/rumcodecs-wasm/src/lib.rs), the `build:wasm*` scripts in [./package.json](./package.json)                        |
+| Run or extend benchmarks          | [./benchmark/src/run.ts](./benchmark/src/run.ts), [./benchmark/src/data.ts](./benchmark/src/data.ts), [./benchmark/src/results.ts](./benchmark/src/results.ts) |
+| CI matrix and gates               | [./.github/workflows/ci.yml](./.github/workflows/ci.yml), [./.github/workflows/rust-ci.yml](./.github/workflows/rust-ci.yml)                                   |
+| Release / publish flow            | [./.github/workflows/release.yml](./.github/workflows/release.yml)                                                                                             |
+
+## Conventions
+
+- Conventional Commits with optional scopes: `feat: …`, `fix(blosc): …`,
+  `ci: …`, `build: …` — match the existing `git log` style.
+- Tooling is [vite-plus](https://www.npmjs.com/package/vite-plus) (`vp`):
+  `pnpm build` / `pnpm test` / `pnpm check` wrap `vp build` / `vp test` /
+  `vp check` (oxlint + oxfmt + typecheck). Keep `pnpm check` clean before
+  committing — CI runs it as a gate.
+- **Every codec class is a drop-in for numcodecs.js**: static `codecId`,
+  static `fromConfig(config)`, the same constructor signature, and the same
+  static members (`Blosc.SHUFFLE`, `LZ4.max_buffer_size`,
+  `Zstd.MAX_CLEVEL`, …). Any new surface must be asserted in
+  [./test/compat.test.ts](./test/compat.test.ts).
+- **Byte formats are Python-numcodecs compatible** and must never change
+  silently: LZ4 uses a 4-byte little-endian original-size header before the
+  block; Zstd emits standard frames; Blosc emits Blosc1-compatible frames
+  (`BLOSC_FORWARD_COMPAT_SPLIT`). A new codec or option combination needs a
+  fixture in [./test/fixtures/](./test/fixtures/) — regenerate with its
+  `generate.py`, and do not refresh existing fixtures to make a failure go
+  away, since that discards the evidence of the break.
+- WASM-backed codecs load the module lazily through a cached dynamic import
+  (the `getModule()` pattern in [./src/blosc.ts](./src/blosc.ts)); new
+  WASM-backed codecs must reuse it so pure-JS codecs never pay the WASM load.
+- `GZip` and `Zlib` stay pure-JS on fflate, and fflate stays a
+  `peerDependency` — do not move it to `dependencies` or reimplement it in
+  the WASM crate.
+- The Rust functions return an empty `Vec` on failure; the TypeScript layer
+  converts that into a thrown `Error`. Keep error handling on that boundary —
+  **no panics across `wasm_bindgen`**, because a panic traps the WASM instance
+  and every later call fails with it, so one corrupt chunk breaks every
+  subsequent read.
+- **An empty `Vec` alone is not a sufficient error signal**, because a frame
+  that legitimately holds zero bytes decodes to exactly the same thing. Each
+  wrapper compares the result against the length the frame itself declares:
+  `blosc_declared_size`, the 4-byte LE header `src/lz4.ts` reads directly, and
+  `zstd_declared_content_size`. Only a mismatch is an error. Getting this wrong
+  is silent: with `out` supplied, a "failed" decode returns the caller's own
+  untouched buffer, which reads as data rather than as an error. Note the limit
+  of this check — it catches truncation and a short decode, but a Blosc1 frame
+  carries no checksum over its payload, so corruption that still parses can
+  decode to the declared length and be returned as valid. That is pinned in
+  [./test/compat.test.ts](./test/compat.test.ts) so it stays a known fact.
+- **Allocations sized from a header must be fallible.** The plausibility guards
+  below bound what a header may claim, not what the allocator can supply; an
+  infallible `vec![0; n]` turns a shortfall into `handle_alloc_error`, which
+  aborts the module exactly like a panic. Reserve through `try_reserve_exact`
+  (see `try_zeroed`) so it degrades to an ordinary decode failure.
+- **Sizes read out of a compressed header are untrusted input.** Every decoder
+  we wrap trusts them to index or to size an allocation up front — `blusc`
+  indexes the source with them, `lz4_flex` zero-fills the declared size before
+  reading a byte, `zstd` reserves the declared content size — so a corrupt
+  header turns into an out-of-bounds panic or a multi-gigabyte allocation that
+  aborts the module. Validate before handing anything over:
+  `blosc_header_is_sane`, `lz4_declared_size_is_plausible`,
+  `zstd_declared_size_is_plausible`. The expansion bounds come from each
+  format's own ceiling (LZ4 255:1, zstd 32768:1) and are pinned by tests that
+  compress a constant buffer, which sits right on that ceiling — tighten them
+  and real data starts failing to decode.
+- `decode(data, out?)` on the WASM-backed codecs must honor the optional `out`
+  buffer (write into it and return it) — zarr consumers rely on this. `GZip`
+  and `Zlib` ignore `out` and return fflate's own buffer, matching numcodecs.js
+  exactly; both behaviors are pinned in
+  [./test/compat.test.ts](./test/compat.test.ts).
+- WASM builds require `RUSTFLAGS="-C target-feature=+simd128"` and build both
+  `web` and `nodejs` targets into `pkg/` (see the `build:wasm*` scripts);
+  `pkg/` is generated output, never hand-edited.
+- A new codec touches five places: `src/<codec>.ts`, `src/index.ts`, the export
+  map in `package.json` (subpath export), the `build.lib.entry` list in
+  [./vite.config.ts](./vite.config.ts), and a `test/<codec>.test.ts` — plus the
+  codec table in [./README.md](./README.md). Miss the vite entry and the
+  advertised subpath resolves to a file that was never built;
+  [./test/index.test.ts](./test/index.test.ts) checks the three lists agree.
+- Node ≥ 18 is supported (`engines`); CI tests Node 22 and 24.
+
+## Key dependencies
+
+| Dependency                | Version                     | Notes                                                                                                      |
+| ------------------------- | --------------------------- | ---------------------------------------------------------------------------------------------------------- |
+| `blusc`                   | 0.0.6                       | Pure-Rust Blosc (Blosc2 API, Blosc1-compatible output) — the `Blosc` codec                                 |
+| `lz4_flex`                | ~0.12                       | Pure-Rust LZ4 block compression — the `LZ4` codec                                                          |
+| `zstd`                    | ~0.13 (no default features) | Zstd — the `Zstd` codec                                                                                    |
+| `wasm-bindgen` / `js-sys` | 0.2 / 0.3                   | WASM boundary                                                                                              |
+| `fflate`                  | ^0.8 (peer)                 | Pure-JS gzip/zlib — the `GZip` and `Zlib` codecs                                                           |
+| `vite-plus` (`vp`)        | 0.2                         | Build / test / check driver                                                                                |
+| `vitest`                  | 4                           | Test runner                                                                                                |
+| `tinybench`               | 6                           | Benchmark timing (benchmark package)                                                                       |
+| `numcodecs` (npm)         | ^0.3.2 (dev)                | **numcodecs.js** — the head-to-head baseline in [./benchmark/](./benchmark/)                               |
+| `numcodecs` (PyPI)        | 0.16.5                      | **Python numcodecs** — wrote [./test/fixtures/](./test/fixtures/); never installed, `uv` fetches it pinned |

CODE_OF_CONDUCT.md

@@ -0,0 +1,40 @@
+# Builder's Code v1.0
+
+A Code of Conduct for people who build things.
+
+---
+
+## The Rule
+
+**Stay professional. Stay technical.**
+
+## Expected
+
+- Contribute constructively.
+- Respect others' time and work.
+- Focus on the work and its technical merit.
+
+## Not Welcome
+
+- Harassment, name-calling, or personal attacks.
+- Trolling, spamming, or derailing discussions.
+- Discussions about contributors rather than their contributions.
+
+## Enforcement
+
+Violations result in:
+
+1. **Warning** - First offense.
+2. **Temporary suspension** - Repeated or serious violations.
+3. **Permanent ban** - Continued violations.
+
+Maintainers can remove, block, or ban anyone who disrupts the project.
+
+---
+
+## License
+
+This Code of Conduct — the text of this file — is dedicated to the public domain
+under [CC0 1.0 Universal](https://creativecommons.org/publicdomain/zero/1.0/), so
+other projects may reuse it freely. It does not change the license of the rest of
+the repository, which is [MIT](LICENSE.txt).

README.md

@@ -0,0 +1,147 @@
+<p align="center">
+  <picture>
+    <source media="(prefers-color-scheme: dark)" srcset="docs/assets/rumcodecs-logo-dark.svg" />
+    <img src="docs/assets/rumcodecs-logo.svg" alt="rumcodecs" width="160" />
+  </picture>
+</p>
+
+<h1 align="center">rumcodecs</h1>
+
+<p align="center">
+  <a href="https://github.com/fideus-labs/rumcodecs/actions/workflows/ci.yml"><img src="https://github.com/fideus-labs/rumcodecs/actions/workflows/ci.yml/badge.svg" alt="CI" /></a>
+  <a href="https://github.com/fideus-labs/rumcodecs/actions/workflows/rust-ci.yml"><img src="https://github.com/fideus-labs/rumcodecs/actions/workflows/rust-ci.yml/badge.svg" alt="Rust CI" /></a>
+  <a href="https://www.npmjs.com/package/@fideus-labs/rumcodecs"><img src="https://img.shields.io/npm/v/%40fideus-labs%2Frumcodecs" alt="npm" /></a>
+  <a href="LICENSE.txt"><img src="https://img.shields.io/badge/license-MIT-blue.svg" alt="MIT License" /></a>
+</p>
+
+<p align="center">
+  <strong>Buffer compression codecs for the browser and Node.js — numcodecs, reimplemented in Rust and WebAssembly.</strong>
+</p>
+
+<p align="center">
+  A drop-in replacement for
+  <a href="https://github.com/manzt/numcodecs.js">numcodecs.js</a>: the same
+  codec classes with the same API and the same byte formats, backed by Rust
+  compiled to WASM (SIMD128) via
+  <a href="https://crates.io/crates/blusc">blusc</a>. Built for
+  <a href="https://zarr.dev/">Zarr</a> — chunks written by Python
+  <code>numcodecs</code> decode in the browser, pinned by fixtures Python
+  actually wrote.
+</p>
+
+## 📦 The codecs
+
+| Codec       | Backend                                                     | Format                                                                                                              |
+| ----------- | ----------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------- |
+| **`Blosc`** | [blusc](https://crates.io/crates/blusc) (Rust → WASM)       | Blosc meta-compressor: `blosclz` / `lz4` / `lz4hc` / `snappy` / `zlib` / `zstd` with shuffle and bitshuffle filters |
+| **`LZ4`**   | [lz4_flex](https://crates.io/crates/lz4_flex) (Rust → WASM) | numcodecs framing — 4-byte little-endian original size + LZ4 block                                                  |
+| **`Zstd`**  | [zstd](https://crates.io/crates/zstd) (Rust → WASM)         | Standard Zstd frames                                                                                                |
+| **`GZip`**  | [fflate](https://github.com/101arrowz/fflate) (pure JS)     | gzip                                                                                                                |
+| **`Zlib`**  | [fflate](https://github.com/101arrowz/fflate) (pure JS)     | zlib                                                                                                                |
+
+Every class carries the numcodecs.js surface — static `codecId`, static
+`fromConfig`, the same constructor signatures and static members
+(`Blosc.SHUFFLE`, `Zstd.MAX_CLEVEL`, …) — and
+[`test/compat.test.ts`](test/compat.test.ts) asserts it stays that way.
+Compatibility with the _Python_ side is checked against real chunks: the
+buffers in [`test/fixtures/`](test/fixtures/) were written by Python
+`numcodecs`, and [`test/interop.test.ts`](test/interop.test.ts) decodes every
+one of them. That pins the Python → JavaScript direction; the reverse is the
+same shared byte format but is not exercised here, since asserting it would
+mean running Python in CI. The WASM module loads lazily on first
+encode/decode, so the pure-JS codecs never pay for it.
+
+## 🚀 Quick start
+
+```sh
+npm install @fideus-labs/rumcodecs
+# fflate is a peer dependency, needed only for GZip and Zlib
+npm install fflate
+```
+
+```ts
+import { Blosc } from "@fideus-labs/rumcodecs";
+
+const codec = new Blosc(5, "zstd", Blosc.SHUFFLE);
+const compressed = await codec.encode(data);
+const restored = await codec.decode(compressed);
+```
+
+Each codec is also a subpath export, so bundlers ship only what you use:
+
+```ts
+import Blosc from "@fideus-labs/rumcodecs/blosc";
+
+// The numcodecs config shape works unchanged — e.g. straight from Zarr metadata.
+const codec = Blosc.fromConfig({ id: "blosc", cname: "lz4", clevel: 5, shuffle: 1 });
+```
+
+## 📊 Benchmarks
+
+The [`benchmark/`](benchmark/) workspace package runs rumcodecs head-to-head
+against numcodecs.js on the same generated arrays
+([tinybench](https://github.com/tinylibs/tinybench), multiple sizes and
+dtypes):
+
+```sh
+pnpm bench          # console table with speedups
+pnpm bench:report   # write JSON + markdown report
+```
+
+## 🛠️ Development
+
+### Prerequisites
+
+- Rust 1.91+ with [wasm-pack](https://rustwasm.github.io/wasm-pack/)
+- Node 18+ (CI tests 22 and 24) and [pnpm](https://pnpm.io/)
+
+### Setup
+
+```sh
+git clone https://github.com/fideus-labs/rumcodecs
+cd rumcodecs
+pnpm install          # runs prepare: builds WASM (web + node) and the bundle
+pnpm test
+```
+
+### Repository structure
+
+```text
+rumcodecs/
+├── src/                    # TypeScript codec classes (Blosc, GZip, Zlib, LZ4, Zstd)
+├── crates/rumcodecs-wasm/  # Rust WASM bindings: blusc, lz4_flex, zstd
+├── pkg/                    # wasm-pack output (generated)
+├── test/                   # vitest suites: round-trip, numcodecs API, interop fixtures
+├── benchmark/              # head-to-head benchmark vs numcodecs.js
+└── .github/workflows/      # ci.yml, rust-ci.yml, release.yml
+```
+
+### Commands
+
+| Command                                    | Purpose                                                                                     |
+| ------------------------------------------ | ------------------------------------------------------------------------------------------- |
+| `pnpm test`                                | Run all vitest suites (round-trip, per-codec, compat, interop)                              |
+| `pnpm check`                               | Lint, format, and typecheck (oxlint / oxfmt / tsc via `vp`)                                 |
+| `pnpm build`                               | Bundle with [vite-plus](https://www.npmjs.com/package/vite-plus) and emit type declarations |
+| `pnpm build:wasm` / `pnpm build:wasm:node` | Rebuild the WASM package for web / Node (SIMD128)                                           |
+| `pnpm bench`                               | Benchmark against numcodecs.js                                                              |
+| `cargo test -- --test-threads=1`           | Rust unit tests for the WASM crate                                                          |
+
+## 🤝 Contributing
+
+Start with [AGENTS.md](AGENTS.md) for the repository map and conventions, and
+the [open issues](https://github.com/fideus-labs/rumcodecs/issues) for what to
+pick up next. All participation is governed by our
+[Code of Conduct](CODE_OF_CONDUCT.md).
+
+## 📄 License
+
+MIT — see [LICENSE.txt](LICENSE.txt). Copyright (c) Fideus Labs LLC.
+
+rumcodecs reimplements the API of
+[numcodecs.js](https://github.com/manzt/numcodecs.js) (MIT, © Trevor Manz) so
+existing Zarr tooling works unchanged. Blosc support comes from the pure-Rust
+[blusc](https://crates.io/crates/blusc) implementation of the
+[Blosc](https://www.blosc.org/) meta-compressor; LZ4 from
+[lz4_flex](https://crates.io/crates/lz4_flex); Zstd from the
+[zstd](https://crates.io/crates/zstd) crate.