VCDIFF Kotlin Decoder

A Kotlin Multiplatform implementation of a VCDIFF (RFC 3284) decoder library for efficient binary differencing and compression.

Overview

This library provides a VCDIFF decoder that can decode delta files created according to RFC 3284 - The VCDIFF Generic Differencing and Compression Data Format. VCDIFF is a format for expressing one data stream as a variant of another data stream, commonly used for binary differencing, compression, and patch applications.

The library provides one-shot decoding, reusable decoder instances, a streaming decoder for incremental processing, and a structural inspection API.

Features

• Kotlin Library: RFC 3284 compliant VCDIFF decoding with clean, idiomatic Kotlin API
• Streaming Decoder: Incremental decoding for processing delta bytes as they arrive
• Structural Inspection: Parse and inspect VCDIFF delta structure without full decoding
• Comprehensive Validation: Support for all VCDIFF instruction types (ADD, COPY, RUN)
• Address Caching: Efficient decoding with proper address cache implementation
• Checksum Validation: Full Adler-32 checksum validation support
• Robust Error Handling: Typed exception hierarchy for precise error handling
• Extensive Testing: 162 test cases including fuzz testing

Limitations

• Application Headers: This implementation does not handle application header information
• Secondary Compression: This decoder does not support secondary compression (eg, gzip, bzip2)
• Custom Code Tables: This decoder does not support custom instruction code tables
• Compatibility: Works with VCDIFF deltas created using xdelta3 -e -S -A (no secondary compression, no application header)

Checksum Support

• VCD_ADLER32: This implementation detects and parses the VCD_ADLER32 extension (bit 0x04 in window indicator)
• Non-standard Extension: The Adler-32 checksum is not part of RFC 3284 but is supported by some implementations
• Validation: Full Adler-32 checksum validation is implemented and performed during decoding

Installation

Gradle (Kotlin DSL)

dependencies {
    implementation("com.ably.vcdiff:vcdiff:0.1.0")
}

Gradle (Groovy DSL)

dependencies {
    implementation 'com.ably.vcdiff:vcdiff:0.1.0'
}

Cloning with Test Suite

This repository includes the VCDIFF test suite as a git submodule. To clone the repository with all test cases:

git clone --recursive https://github.com/ably/vcdiff-kotlin.git

If you've already cloned the repository without the submodule, initialize it:

git submodule update --init --recursive

To update the test suite submodule to the latest version:

git submodule update --remote

Quick Start

One-Shot Decoding

import com.ably.vcdiff.decode
import java.io.File

fun main() {
    val source = File("original.txt").readBytes()
    val delta = File("changes.vcdiff").readBytes()

    val result = decode(source, delta)
    File("result.txt").writeBytes(result)
}

Reusable Decoder

For decoding multiple deltas against the same source:

import com.ably.vcdiff.VcdiffDecoder
import java.io.File

fun main() {
    val source = File("original.txt").readBytes()
    val decoder = VcdiffDecoder(source)

    val target1 = decoder.decode(File("delta1.vcdiff").readBytes())
    val target2 = decoder.decode(File("delta2.vcdiff").readBytes())
}

Streaming Decoder

For processing delta bytes incrementally as they arrive:

import com.ably.vcdiff.VcdiffStreamingDecoder
import java.io.ByteArrayOutputStream

fun main() {
    val source = File("original.txt").readBytes()
    val decoder = VcdiffStreamingDecoder(source)

    val output = ByteArrayOutputStream()

    // Feed chunks as they arrive
    output.write(decoder.append(chunk1))
    output.write(decoder.append(chunk2))
    output.write(decoder.append(chunk3))

    // Signal end of stream
    output.write(decoder.finish())

    val result = output.toByteArray()
}

Error Handling

The decoder provides specific exception types for different error conditions:

import com.ably.vcdiff.*

fun main() {
    try {
        val target = decode(source, delta)
    } catch (e: InvalidMagicException) {
        println("Invalid VCDIFF file: $e")
    } catch (e: UnsupportedVersionException) {
        println("Unsupported VCDIFF version: $e")
    } catch (e: ChecksumMismatchException) {
        println("Checksum failed: expected ${e.expected}, got ${e.actual}")
    } catch (e: InvalidFormatException) {
        println("Malformed delta: $e")
    } catch (e: VcdiffException) {
        println("VCDIFF error: $e")
    }
}

Structural Inspection

Parse and inspect the structure of a VCDIFF delta without full decoding:

import com.ably.vcdiff.parseDelta
import java.io.File

fun main() {
    val delta = File("changes.vcdiff").readBytes()
    val parsed = parseDelta(delta)

    println("Version: ${parsed.header.version}")
    println("Windows: ${parsed.windows.size}")

    for (window in parsed.windows) {
        println("  Target length: ${window.targetWindowLength}")
        println("  Instructions: ${window.instructions.size}")
    }
}

API Reference

Core Functions

decode(source: ByteArray, delta: ByteArray): ByteArray

Decodes a VCDIFF delta file using the provided source data and returns the reconstructed target data.

Parameters:

• source: The original source data (may be empty for deltas that don't reference source)
• delta: The VCDIFF delta file data

Returns:

• Decoded target data as ByteArray

Throws:

• VcdiffException if decoding fails (malformed delta, checksum validation failure, etc.)

parseDelta(delta: ByteArray): ParsedDelta

Parses a VCDIFF delta into its structural components without performing full decoding. Useful for debugging and tooling.

Parameters:

• delta: The VCDIFF delta file data

Returns:

• A ParsedDelta containing the header and list of windows with their instructions

Classes

VcdiffDecoder(source: ByteArray)

Creates a new decoder instance with the specified source data. Useful for decoding multiple deltas against the same source.

Methods:

• decode(delta: ByteArray): ByteArray - Decodes a single VCDIFF delta using the decoder's source data

VcdiffStreamingDecoder(source: ByteArray)

Creates a streaming decoder that accepts delta bytes incrementally. Emits reconstructed target bytes as soon as complete windows are decoded.

Methods:

• append(data: ByteArray): ByteArray - Feed delta bytes; returns any decoded output available
• finish(): ByteArray - Signal end of stream; returns any remaining decoded output

Exception Types

• VcdiffException: Base exception for all VCDIFF errors
• InvalidMagicException: Invalid VCDIFF magic bytes
• UnsupportedVersionException: Unsupported VCDIFF version
• InvalidFormatException: Malformed delta structure (truncated data, bad lengths, etc.)
• ChecksumMismatchException: Adler-32 checksum validation failure (includes expected and actual fields)

Data Types

ParsedDelta

• header: Header - VCDIFF file header
• windows: List<Window> - List of delta windows

Header

• version: Byte - VCDIFF version (always 0)
• indicator: HeaderIndicator - Header flags

Window

• sourceSegmentSize: Long - Size of source segment referenced
• sourceSegmentPosition: Long - Offset into source data
• targetWindowLength: Long - Expected target window size
• hasChecksum: Boolean - Whether Adler-32 checksum is present
• checksum: Long - Adler-32 checksum value
• instructions: List<Instruction> - Decoded instructions

Instruction (sealed class)

• Instruction.Add(size: Long, data: ByteArray) - Add literal bytes
• Instruction.Copy(size: Long, mode: Int, address: Long) - Copy from source or target
• Instruction.Run(size: Long, byte: Byte) - Repeat a single byte

Testing

Running Tests

To run all tests:

gradle test

To run with verbose output:

gradle test --info

To run only the integration tests:

gradle test --tests "com.ably.vcdiff.VcdiffTest"

To run only the fuzz tests:

gradle test --tests "com.ably.vcdiff.FuzzTest"

Test Suite

The test suite includes:

• 52 targeted positive tests: Specific feature validation (basic operations, varint boundaries, code table entries)
• 33 targeted negative tests: Invalid VCDIFF files with strict exception type assertions
• 20 general positive tests: Valid VCDIFF files covering various data patterns
• 32 streaming decoder tests: Incremental decoding of all positive test cases
• 5 structural inspection tests: parseDelta API validation
• 20 fuzz tests: Random byte generation, mutation-based, structure-aware, and boundary condition fuzzing
• Total: 162 test cases

Test Results

• Positive tests: 72/72 passed
• Negative tests: 33/33 passed
• Streaming tests: 32/32 passed
• Inspection tests: 5/5 passed
• Fuzz tests: 20/20 passed

Creating VCDIFF Deltas

This is a decoder-only library. To create compatible VCDIFF delta files, use tools such as xdelta3:

# Create a VCDIFF delta (compatible with this decoder)
xdelta3 -e -S -A -s original.txt modified.txt delta.vcdiff

Requirements

• Kotlin: 1.9+
• JVM: Java 8 or higher (when targeting JVM/Android)
• Dependencies: None (zero runtime dependencies)

Contributing

Contributions are welcomed. Please follow these guidelines.

Getting Started

1. Fork the repository
2. Clone your fork with submodules: git clone --recursive <your-fork-url>
3. Create a feature branch: git checkout -b feature/your-feature-name
4. Make your changes
5. Test your changes thoroughly
6. Submit a pull request

Development Guidelines

• Code Style: Follow Kotlin coding conventions
• Testing: All new features must include tests
• Documentation: Update documentation for any API changes
• Commits: Use clear, descriptive commit messages

Before Submitting

Ensure your contribution passes all checks:

# Run all tests
gradle test

# Check for compilation errors
gradle compileKotlin

Reporting Issues

When reporting bugs, please include:

• JDK version
• Kotlin version
• Operating system
• Minimal reproduction case
• Expected vs actual behavior
• Sample VCDIFF files (if applicable)

Feature Requests

For new features, please:

• Check existing issues first
• Describe the use case
• Provide RFC 3284 references if applicable
• Consider backwards compatibility

License

This project is licensed under the Apache License 2.0. See the LICENSE file for details.

References

• RFC 3284: The VCDIFF Generic Differencing and Compression Data Format
• xdelta3: VCDIFF binary diff tool
• VCDIFF Test Suite