[gspark-etri]

Background

The paper needs baselines that bench/baselines.sh (#51) does not cover. It also needs repeated runs of the headline numbers, which are single runs in docs/evaluation.md (#73).

Changes

• bench/tensor_dedup.py is a ZipLLM-style tensor-level dedup baseline:
  • It reads only the safetensors JSON header and keys each tensor by sha256.
  • Every other file is one opaque object (whole-file sha256).
  • Each distinct object is compressed with zstd -3.
  • The stored total includes headers and metadata.
  • .rbln and other vendor files are not parsed (ADR-0009).
• bench/switch_baselines.sh measures rsync (--no-whole-file, with and without zstd) and xdelta3 (2 GiB source window) for one variant switch:
  • bytes sent
  • wall time
  • peak RSS
  • a sha256 check of the result
• bench/repeat.sh repeat|ablation covers:
  • n interleaved rounds of: push of the set, the A→B transfer (onggi and --sub 0), plain pull, and delta apply (pull --base);
  • one ablation run: default, --sub 0, --frame 0 and --compress none.

Verification

Runs on etri-001, 2026-10-30, ATOM Llama-3.1-8B set (19 files, 263,893,021,566 B). Setup:

• Binary built on the node from main 05c9760, sha256 811d770a….
• nice -n 19, GOMAXPROCS=32.
• Warm page cache (it cannot be dropped without root).
• Pair: B2→B8 = decode_S256_B2_lm → decode_S256_B8_lm.

Repeated onggi runs (n = 5), median [IQR]:

measurement	bytes	seconds
push of the set: stored (incl. indexes)	16.709 GB [0.24 MB] (15.79×)	84.5 [1.0]
B2→B8 transfer, onggi	3,567,290 B (every run)	23.9 [0.6]
B2→B8 transfer, --sub 0 (Xet-style)	6,129,786 B (every run)	16.5 [0.3]
B8 plain pull (no cache)	9.91 GB read	28.5 [1.5]
B8 delta apply from a B2 copy	3.57 MB read, 15.01 GB left in place	5.62 [0.25]

All 10 pulls were byte-identical (sha256).

Ablation (n = 1):

setting	stored	B2→B8 transfer
default	16.71 GB	3.57 MB
--sub 0	29.50 GB	6.13 MB
--frame 0	16.97 GB	28.3 MB
--compress none	25.93 GB	149.1 MB

Tensor-level dedup:

input	tensor-level dedup	onggi
ATOM set	199.16 GB (1.325×)	16.71 GB
HF + ATOM	224.07 GB	28.37 GB

On ATOM the tensor-level baseline equals whole-file dedup with zstd: no .rbln exposes a tensor table, and no two files are identical.

rsync / xdelta3:

switch	method	bytes sent	time	peak RSS	identical
B2→B8	rsync	218.9 MB	32.4 s	8 MB	yes
B2→B8	rsync -z zstd	2.0 MB (+0.99 MB checksums back)	35.1 s	14 MB	yes
B2→B8	xdelta3 -B 2 GiB	3.62 MB delta	encode 16.6 s, decode 17.3 s	4.3 GB	yes
B2→C128	rsync	9.61 GB	210.5 s	11 MB	yes
B2→C128	rsync -z zstd	7.29 GB	222.5 s	15 MB	yes
B2→C128	xdelta3 -B 2 GiB	3.59 MB delta	encode 36.3 s, decode 17.5 s	4.3 GB	yes
B2→C128	onggi (n = 1, delta pull)	1.92 GB	38.2 s	2.4 GB	yes

xdelta3's result depends on the source window:

• With the default 64 MiB window, B2→C128 needs 3.72 GB and takes 761 s to encode.
• With 4 GiB, B2→B8 gives 3.62 MB.
• With 16 GiB, xdelta3 3.1.0 crashes with SIGSEGV.

The scripts also passed a small synthetic smoke test on the node.

Compatibility and limits

• Scripts only; no change to onggi. The raw data (JSON/TSV with every sample, commands, binary sha256) is kept with the paper, outside this repo.
• Network:
  • rsync ran as a local transfer, so "Total bytes sent" stands for the bytes on the network. The receiver's block checksums also cross it (0.99 MB here).
  • Neither rsync nor xdelta3 needs a CDC store. xdelta3 needs both variants at the sender and one delta per ordered pair.
• bsdiff was not run: it needs about 17× the file size in memory (~260 GB for 15 GB files) and a suffix sort over 15 GB.
• The S3 full download was not run: there is no S3 endpoint or credentials on the node.
• Copy-from-base (Delta apply: copy bytes the base holds at other offsets from the base file (TensorRT-LLM batch change reads 10.8 GB) #58) has no flag of its own. Only --base on or off is measured.

Related Issue

Refs #73 (paper measurements; not closing until the paper tables are written)

🤖 Generated with Claude Code