Skip to content

feat(encoding): compress v2.3 mini-block offsets - #8331

Draft
Xuanwo wants to merge 1 commit into
xuanwo/generic-block-v5-7-offset-selectionfrom
xuanwo/generic-block-v5-8-offset-adoption
Draft

feat(encoding): compress v2.3 mini-block offsets#8331
Xuanwo wants to merge 1 commit into
xuanwo/generic-block-v5-7-offset-selectionfrom
xuanwo/generic-block-v5-8-offset-adoption

Conversation

@Xuanwo

@Xuanwo Xuanwo commented Aug 6, 2026

Copy link
Copy Markdown
Collaborator

Stack 8 of 10 for Lance generic block sequence compression.

This makes Lance 2.3 variable-width mini-block offsets the first non-sparse adopter of generic block compression. The writer emits the generic representation only when its complete serialized form is strictly smaller; otherwise it keeps the legacy interleaved flat layout. Chunk-local scan, range, and take scheduling remain intact.

Lance 2.0–2.2 writers and readers keep their existing bytes and selection behavior. SparseLayout keeps its existing wire format and explicitly vetoes the generic path.

Benchmark

Representative 262,144-row UTF-8 workloads compare equivalent v2.2 and v2.3 mini-block encoding at ae0a49ba2 with the release-with-debug profile. The descendant measurement commit adds only documentation and the benchmark harness after this PR's runtime code.

Workload Encoded bytes Encode median Scan median Cold take median
Range, v2.2 → v2.3 5,283,840 → 4,218,880 (-20.2%) 1.512 → 6.824 ms 1.330 → 1.170 ms 249.2 → 202.3 µs
Delta, v2.2 → v2.3 10,436,608 → 9,355,264 (-10.4%) 2.650 → 10.014 ms 2.136 → 2.259 ms 232.4 → 246.3 µs

Stack navigation

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-encoding Encoding, IO, file reader/writer enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant