[gspark-etri]

Background

Delta apply (#41) only keeps bytes that sit at the same offset in the base. When a variant holds the same bytes at other offsets, the pull fetched them although the node had them (#58):

• A TensorRT-LLM batch change shifts the weights by 17,952 B, so a b2 → b8 pull read 10.8 GB (GPU generality: TensorRT-LLM engine variants on the A100, measured with onggi #56).
• ATOM prefill holds decode's weights in another order: 79 % of its bytes are elsewhere in a decode copy.

Changes

• chooseBase also scores each offered base by the bytes it holds anywhere: moved, through a by-digest index of its extents.
• Consume path, unchanged. It is used when at least MinSame of the bytes stay at the same offsets.
• New local path. Otherwise, if same plus moved reach MinSame, fromBase reads every still-needed stored chunk the base holds from the base's files where they are:
  • one reader per worker, front to back per file;
  • each chunk is hash-checked before it is written;
  • the base is not claimed or changed;
  • a chunk that cannot be read or fails its check goes to the cache, peers or store as before.
• Stats local_base, local_bytes. onggi pull --base reports which path was used. The CSI plugin's offer gets the local path as well.
• Docs: evaluation (GPU section) and the CSI README.

Verification

• New test, shifted content (a 17,952-byte header in front of the base's bytes, as in TensorRT):
  • no claim;
  • 8.38 MB of 8.4 MB read from the base, 22 KB from the store;
  • output identical, base untouched.
• The existing rebase, frame, cache and GC tests pass.
• Lab node, ATOM store, decode B2 restored, then prefill C128 pulled:

prefill C128 (14.0 GB)	store read	from the base	time	sha256
plain pull	9.47 GB	0	48.3 s	—
with --base B2 (0 % same offset, 79 % elsewhere)	1.93 GB	11.1 GB	40.2 s	identical; base untouched

The store here is on the same disk, so the time gain is small. Against a network store, the 80 % fewer store reads are the gain.

Compatibility and limits

• No format change. Without bases, nothing changes.
• The base must stay readable during the pull. If it disappears, the remaining chunks come from the other sources.

Related Issue

Closes #58

🤖 Generated with Claude Code