Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions docs/src/format/file/encoding.md
Original file line number Diff line number Diff line change
Expand Up @@ -587,6 +587,7 @@ on a per-value basis. We use ☑️ to mark a technique that is applied on a per
| Constant | ✅ (2.1) | ❓ | ❓ |
| Range | ✅ (2.3) | ❌ | ❌ |
| Delta | ✅ (2.3) | ❌ | ❌ |
| Dictionary | ✅ (2.3) | ❌ | ❌ |
| Bitpacking | ✅ (2.1) | ❓ | ✅ (2.1) |
| Fsst | ❓ | ✅ (2.1) | ✅ (2.1) |
| Rle | ✅ (2.2) | ❌ | ✅ (2.1) |
Expand Down Expand Up @@ -639,6 +640,21 @@ has one, and readers reconstruct values with checked prefix sums.
%%% proto.message.Delta %%%
```

### Block Dictionary

A block dictionary represents unsigned `u32` or `u64` values with a `u32` indices child and a dictionary-items
child. Both children are concrete block codecs. When either child has a payload, the outer payload is framed as:

```text
u64 indices_payload_bytes
u64 items_payload_bytes
indices payload
dictionary-items payload
```

A metadata-only child has a zero framed length. If both children are metadata-only, the dictionary itself has no
payload. Readers validate the frame boundaries, child widths and cardinalities, and every dictionary index.

### Bitpacking

Bitpacking is a compression technique that removes the unused bits from a set of values. For example, if we have
Expand Down
1 change: 1 addition & 0 deletions rust/lance-encoding/src/encodings/physical.rs
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ pub mod block;
pub mod byte_stream_split;
pub mod constant;
pub mod delta;
pub mod dictionary;
pub mod fsst;
pub mod general;
pub mod packed;
Expand Down
Loading
Loading