[gspark-etri]

Background

Switching a node from decode B2 to decode B8 sent 28.3 MB with onggi against 6.1 MB with Xet-style chunking (#51), because onggi compressed each ~4 KiB sub-chunk alone (#52). A probe on the lab store measured what grouping would give: new sub-chunks compressed per chunk would send 5.8 MB, and in frames of up to 256 KiB across chunks, 3.3 MB.

Changes

• Compression frames (pkg/store/frame.go). In a sub-chunked file, push collects consecutive new chunks and compresses them together, up to Config.Frame raw bytes (new stores: 255 KiB, --frame; 0 turns it off).
  • A frame is kept only if it is smaller than its chunks compressed alone. Weights compress about as well chunk by chunk with byte planes and Huffman.
  • Frames stay below 256 KiB so their byte planes fit one Huffman block.
  • Each chunk keeps its own index entry. Its Loc names the frame (Offset, Stored, Codec), the frame's raw length (Frame) and its place in it (Inner), so dedup and the per-chunk hash check are unchanged.
• Format.
  • Index v3 (ONGGIIX3, 56-byte entries) is written only for packs with framed chunks; v1 and v2 are still read and written as before.
  • Locator node kind 2.
  • Config frame.
  • Old stores keep working, and new stores need a reader with this change.
• Readers.
  • Pull, ReadStored and repack decode each frame once while chunks come in pack order, and coalesce chunks of one frame into one read.
  • Node caches and peers receive framed chunks re-encoded one by one.
  • Repack copies live chunks of a frame on their own.
  • GC counts a chunk's share of its frame (Loc.share).
• Docs: docs/evaluation.md, README.

Verification

• Unit tests.
  • Round trip with and without frames; variant dedup kept; frames halve the packed bytes of a variant whose change only frames can compress.
  • Cache fill from frames (no framed entries in the cache, second pull reads 0 B from the store), the framed store as a peer, and GC repack after half the chunks die (pull identical, index consistent).
  • Index v3 round trip, a v2 index when nothing is framed, and a chunk outside its frame rejected.
  • Full suite green on the node.
• ATOM Llama-3.1-8B set (263.9 GB), lab node:

	before	frames, always	frames, when smaller (this PR)
store (du)	16.97 GB	18.12 GB	16.71 GB
push	1.3 min	1.0 min	1.5 min
transfer decode B2 → B8	28.3 MB	3.57 MB	3.57 MB (Xet-style 6.1 MB)
delta apply B8 from a B2 copy	5.7 s, 28.3 MB read	7.1 s, 3.8 MB	4.8 s, 3.6 MB
pull decode B8 / prefill C128	identical	identical (sha256)	identical (sha256)

Compatibility and limits

• The format is extended (index v3, locator kind 2). Older binaries cannot read framed packs; stores without frames are unchanged.
• Reading one chunk of a frame decodes the frame (≤ 255 KiB). Pulls group a frame's chunks, so a whole-artifact pull decodes each frame once.

Related Issue

Closes #52

🤖 Generated with Claude Code