[gspark-etri]

Background

Two-level chunking (#15) was switched on per file by name: store.DefaultSubKinds / push --sub-kinds defaulted
to atom-prefill,atom-decode,atom-other, which pkg/layout derives from .rbln and the words prefill/decode. A
TensorRT-LLM engine or a furiosa-llm bundle never got sub-chunks, and a renamed ATOM file lost them. The design (a
variant coordinate plus three byte-level weight-placement classes) and ADR-0009 say storage decisions come from the
bytes, not from vendor names.

Changes

• A1: size policy. New stores sub-chunk the new chunks of every file of at least --sub-min-file (default
  1 MiB), recorded in the store config as sub_min_file. Store.subChunks(rel, size): stores that recorded
  sub_kinds keep deciding by kind; neither field means every file, as before. --sub-kinds LIST|all still selects the
  old behaviour; kinds and a size exclude each other at creation. onggi ls shows the policy. pkg/layout stays for
  reports (dedup-report rows, inspect) and as the existing pairing hint in diff.
• A2: placement classes. onggi dedup-report --classify [--reference PATH]... reports per file
  referenced / embedded-rearranged / embedded-shifted / new from chunk shares alone: the share in the
  references (coarse), and the share found elsewhere at coarse 64 KiB and at fine 4 KiB chunks. Thresholds are flags
  with documented defaults (0.95 / gap 0.25 / 0.5). JSON adds files[].class, files[].shares and classify. No
  format is parsed.
• Docs: docs/sub-chunk-policy.md (candidates, measurements, choice, compatibility, placement results), raw data
  docs/data/sub-chunk-policy-20261030.json, README (en/ko), artifact-formats, evaluation.

Verification

Measurement (lab node with the ATOM set, HF checkpoint and RNGD artifacts; nice -n 19, 16 threads, shared node;
n = 1 per configuration, sizes deterministic for a given push order; binaries ec9bc0f and main 05c9760):

	kind (old default)	size ≥ 1 MiB	probe (experiment)
ATOM set 263.9 GB → store	16.70 GB, 110 s	16.78 GB, 120 s	26.92 GB
HF checkpoint 16.07 GB → store	10.73 GB, 38 s	11.28 GB (+0.53 GB metadata), 122 s	10.74 GB
RNGD 4 artifacts 32.4 GB → store	10.79 GB	11.53 GB	10.78 GB
ATOM decode B2 → B8 (_lm), fetched	3.57 MB	3.57 MB	3.58 MB
RNGD pfbatch → preset, fetched	4.33 MB	0.65 MB	2.59 MB

The binary from main stores the ATOM set in 16.70 GB, identical to kind. The probe (sub-chunk a file's first 64 new
chunks, continue if ≥ 10 % found) was rejected: the first file of a family finds nothing stored, so later rearranged
files cannot match it at 4 KiB, and ATOM loses 10.2 GB. Bytes alone cannot tell that first copy from a checkpoint, so
the size policy trades a bounded metadata cost on new checkpoints (+3.3 %, ~3× push time for that content) for never
losing dedup. Details and the placement classes of both sets: docs/sub-chunk-policy.md.

Tests: TestSubMinFileIgnoresNames (a rank0.engine and renamed.bin get sub-chunks; the decision depends only
on size), TestSubPolicyIsStoredWithTheStore (an old config with sub_kinds keeps its behaviour; a new store keeps
sub_min_file; both together are rejected), TestPlacementClasses, TestPlacementReferenced (reference through a
symlink, as in an HF snapshot), TestClassRulesAndThresholds, TestDedupReportClassify. go test -race ./...,
go vet ./..., gofmt -l . clean locally (Go 1.24.13 in golang:1.24-bookworm).

Compatibility and limits

• Existing stores are unchanged (sub_kinds kept). Manifests (including files[].kind), chunk lists, locators,
  indexes and composite tables are unchanged. The binary from main pulled an artifact from a store created by this
  branch, byte-identical; this branch reads and pushes to a store created by main.
• An old binary pushing to a new store ignores sub_min_file and sub-chunks every file, small ones included. The
  result is valid for all versions; it only costs metadata for files under 1 MiB.
• Cost of the size policy: referenced checkpoints are sub-chunked when first pushed, so every artifact that references
  them carries a larger locator (RNGD preset: +227 MB vs +12 MB). A follow-up could let locators reference composites.
• TensorRT-LLM engines live on the lab A100 host and were not re-measured; by size they now get sub-chunks.
• Expected rebase conflict with Integrate verified delta comparison and compatible artifact preparation #72 (integration/onggi-serving): both touch pkg/store/push.go. This PR changes
  one line there (s.subChunks(files[i].rel, files[i].Size)), and Integrate verified delta comparison and compatible artifact preparation #72 adds Profile around it; the merge is mechanical.
  cmd/onggi/store.go (flags) is touched by both too.

Related Issue

Closes #74

🤖 Generated with Claude Code