drtc

Lossless text compression that shrinks the token count of LLM context — reconstructs your input exactly, optimized for GPT-4o's o200k_base tokenizer.

Not yet published to crates.io — add Crates.io / Docs.rs badges after the first cargo publish.

Overview

Large language models bill and bound you by tokens, and repetitive context — logs, code, transcripts, reference docs — wastes them. General-purpose compressors (gzip, zstd, brotli) don't help: their binary output, re-encoded as prompt text, tokenizes to more tokens than the original.

drtc is a fully lossless compressor whose output is paste-able UTF-8 text engineered to tokenize to fewer tokens under o200k_base, then decompress back to the exact original bytes (including invalid UTF-8, which is escaped and preserved). It optimizes token count, not just bytes — and reports both ratios so you can see the difference.

Highlights

• Lossless — verifiable roundtrip (drtc --test); property-tested over arbitrary bytes.
• Token-aware — optimizes for o200k_base (GPT-4o); reports both token and byte ratios.
• Three-tier pipeline — FastCDC + xxHash dedup → dictionary/RLE codes → token-level LZ77.
• Binary-safe — arbitrary input; invalid UTF-8 escaped as \xNN and round-tripped.
• Paste-hardened — survives BOM insertion, CRLF conversion, trailing-newline stripping.
• Tunable — --fast / --best presets, plus a library API for embedding.

Installation

Not yet on crates.io. Build from source:

git clone https://github.com/Portfoligno/drtc
cd drtc
cargo install --locked --path .

This installs a drtc binary. (After the first crates.io release: cargo install --locked drtc.)

Usage

drtc reads a file or stdin and writes a file or stdout. The operation is chosen by a flag — there are no subcommands. With no flag it compresses.

# Compress a file to stdout
drtc input.txt

# Compress to a file
drtc input.txt output.drtc

# Pipe usage (stdin -> stdout)
cat context.txt | drtc > context.drtc

# Decompress
drtc -d context.drtc original.txt

# Verify a lossless roundtrip (writes nothing; exits 1 on mismatch)
drtc -t input.txt

# Show statistics (printed to stderr)
drtc -s input.txt > out.drtc

# Ratio-optimized vs speed-optimized
drtc --best input.txt
drtc --fast input.txt

# Benchmark (20 iterations)
drtc -b -n 20 input.txt

Line wrapping defaults to width 120; use -w <N> to change it or --no-wrap to disable. Run drtc --help for the full option list.

drtc shines on large, redundant text. Very small inputs may expand in token count (ratio < 1×) due to fixed marker overhead.

How it works

drtc runs a three-tier lossless pipeline (all tiers run by default; --fast disables Tier 3):

1. Deduplication — content-defined chunking (FastCDC) with xxHash; duplicate chunks become compact references.
2. Dictionary + RLE — frequent token n-grams become single-token codes; runs of repeated characters collapse via run-length encoding.
3. Token-level LZ77 — back-references over the token stream. Offsets and lengths are expressed in characters, not tokens, because BPE is context-sensitive (tokenize(A+B) != tokenize(A)+tokenize(B)), so token offsets wouldn't decode reliably.

Output begins with a self-describing header (===DRTC:<version>===) and an optional registry of chunk/dictionary entries. See docs/property-test-strategy.md for the testing approach.

Library usage

drtc is also a library exposing the Compressor trait, DrtcConfig (default(), speed_optimized(), ratio_optimized()), and CompressionStats.

use drtc::pipeline::DrtcCompressor;
use drtc::config::DrtcConfig;
use drtc::Compressor;

let compressor = DrtcCompressor::new(DrtcConfig::default());
let (compressed, _stats) = compressor.compress(input_bytes, false)?;
let restored = compressor.decompress(&compressed)?;
assert_eq!(restored, input_bytes);

Development

cargo test                          # run the suite
cargo test --features thorough-tests # more property-test cases
cargo bench                         # criterion benchmarks (full_pipeline)
cargo run --example token_check     # inspect o200k_base token costs
cargo fmt                           # required before commits

Property-test regression files (*.proptest-regressions) are committed and must not be deleted — they re-test previously found bugs.

License

Licensed under either of

• Apache License, Version 2.0 (LICENSE-APACHE or http://www.apache.org/licenses/LICENSE-2.0)
• MIT license (LICENSE-MIT or http://opensource.org/licenses/MIT)

at your option.

Contribution

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in the work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.