DETAILED_DOCUMENTATION.txt

================================================================================
TURBOLOADER COMPREHENSIVE TECHNICAL DOCUMENTATION
Version 1.7.1
================================================================================

This document provides a detailed, senior-engineer-level breakdown of the
TurboLoader codebase architecture, implementation details, and complete
step-by-step instructions for building, testing, and running all components.

Written by: Senior Software Engineer/Researcher
Target Audience: Engineers and researchers seeking deep technical understanding


TABLE OF CONTENTS
================================================================================
1. PROJECT OVERVIEW & ARCHITECTURE
2. CORE DESIGN PATTERNS & PHILOSOPHY
3. DETAILED CODEBASE STRUCTURE
4. BUILD SYSTEM COMPREHENSIVE GUIDE
5. STEP-BY-STEP INSTALLATION
6. RUNNING THE TEST SUITE
7. BENCHMARKING FRAMEWORK
8. PYTHON API & BINDINGS
9. ADVANCED FEATURES & OPTIMIZATION
10. TROUBLESHOOTING & DEBUGGING


================================================================================
SECTION 1: PROJECT OVERVIEW & ARCHITECTURE
================================================================================

1.1 WHAT IS TURBOLOADER?
------------------------------------------------------------------------------

TurboLoader is a high-performance machine learning data loading library written
in C++20 with Python bindings. The primary design goal is to eliminate data
loading as the bottleneck in ML training pipelines by achieving throughputs
that significantly exceed standard frameworks like PyTorch DataLoader.

Key Performance Numbers:
- Peak Throughput: 21,035 images/second (16 workers, batch_size=64)
- 12x faster than PyTorch DataLoader
- 1.3x faster than TensorFlow's tf.data
- 52+ Gbps local file I/O throughput
- TBL v2 Format: 4,875 img/s TAR→TBL conversion throughput
- TBL v2 Compression: 40-60% space savings with LZ4
- Smart Batching: 15-25% throughput improvement with size-aware grouping

The library achieves these performance numbers through several key techniques:
1. Lock-free SPSC (Single Producer Single Consumer) ring buffers
2. Memory-mapped I/O for zero-copy file access
3. 19 SIMD-accelerated transforms (AVX2/NEON) including AutoAugment policies
4. Per-worker thread-local decoders to eliminate contention
5. GPU-accelerated JPEG decoding via nvJPEG (optional)
6. Intelligent prefetching and batching strategies
7. TBL v2 Format with LZ4 streaming compression (40-60% smaller than TAR)
8. Smart Batching for size-aware sample grouping
9. Distributed Training with deterministic sharding for multi-node setups


1.2 ARCHITECTURAL PRINCIPLES
------------------------------------------------------------------------------

The architecture follows several critical design principles:

PRINCIPLE 1: ZERO-COPY PHILOSOPHY
Wherever possible, data is accessed via memory-mapped files and passed by
reference or moved (C++ move semantics) rather than copied. This eliminates
redundant memory allocations and reduces cache pressure.

PRINCIPLE 2: LOCK-FREE CONCURRENCY
Traditional locks introduce contention and unpredictable latency. TurboLoader
uses atomic operations and lock-free data structures for all hot paths. Mutexes
are only used in cold paths like initialization.

PRINCIPLE 3: SIMD-FIRST DESIGN
All performance-critical operations (image transforms, format conversions, etc.)
are implemented with SIMD intrinsics. The library automatically detects CPU
capabilities and falls back to NEON on ARM or scalar code if SIMD unavailable.

PRINCIPLE 4: PER-WORKER RESOURCE ISOLATION
Each worker thread maintains its own decoder instances, buffer pools, and state.
This eliminates false sharing and cache line bouncing between CPU cores.

PRINCIPLE 5: CONDITIONAL COMPILATION FOR OPTIONAL FEATURES
GPU acceleration (nvJPEG), async I/O (io_uring), and other platform-specific
features are compiled conditionally. The library provides graceful fallbacks
ensuring it works on all platforms (Linux, macOS, Windows).


1.3 HIGH-LEVEL DATA FLOW
------------------------------------------------------------------------------

The data flow through TurboLoader follows this pipeline:

Step 1: SOURCE READING
    TAR files are memory-mapped into the process address space. The TarReader
    parses TAR headers in-memory without system calls for reading.

    Location: src/readers/tar_reader.hpp
    Key Method: TarReader::load_samples()

Step 2: WORKER DISPATCH
    The UnifiedPipeline maintains a pool of Worker threads. Each worker pulls
    TAR entry metadata from a thread-safe queue and processes samples
    independently.

    Location: src/pipeline/pipeline.hpp
    Key Classes: UnifiedPipeline, TarWorker

Step 3: DECODING
    Workers decode JPEG data using per-worker JPEGDecoder instances (libjpeg-
    turbo with SIMD). On systems with nvJPEG, GPU decoding can be used for 10x
    faster JPEG decompression.

    Location: src/decode/jpeg_decoder.hpp, src/decode/nvjpeg_decoder.hpp

Step 4: TRANSFORMATION
    Decoded RGB images pass through a transform pipeline. Transforms are
    composable and use SIMD intrinsics for operations like resize, normalize,
    color jitter, etc.

    Location: src/transforms/*.hpp
    Key Transforms: resize_transform.hpp, normalize_transform.hpp

Step 5: BATCHING
    Transformed samples are collected into batches. Smart batching groups
    similar-sized samples to reduce padding overhead.

    Location: src/pipeline/smart_batching.hpp

Step 6: TENSOR CONVERSION
    Batches are converted to PyTorch/TensorFlow/JAX tensor format (CHW or HWC
    layout) and returned to the Python layer via pybind11.

    Location: src/transforms/tensor_conversion.hpp


================================================================================
SECTION 2: CORE DESIGN PATTERNS & PHILOSOPHY
================================================================================

2.1 CONCURRENT QUEUE DESIGN
------------------------------------------------------------------------------

The pipeline uses lock-free SPSC (Single Producer Single Consumer) ring buffers
for passing data between pipeline stages. This is critical for performance.

IMPLEMENTATION DETAILS:

File: src/core/spsc_ring_buffer.hpp

The ring buffer maintains two atomic indices:
- write_idx_: Modified only by producer thread
- read_idx_: Modified only by consumer thread

This separation ensures each thread writes to its own cache line, preventing
false sharing. The buffer size is always a power of 2, allowing fast modulo
operations via bitwise AND:

    index & (capacity - 1)  // Fast modulo

Memory ordering is carefully chosen:
- push() uses memory_order_release on write_idx_
- pop() uses memory_order_acquire on read_idx_

This establishes a happens-before relationship ensuring all writes to the data
buffer are visible to the consumer before it reads write_idx_.


2.2 MEMORY POOLING STRATEGY
------------------------------------------------------------------------------

Creating and destroying std::vector<uint8_t> for every image is expensive due
to allocator overhead. TurboLoader uses object pooling to reuse allocations.

File: src/core/object_pool.hpp

The ObjectPool maintains a free list of pre-allocated objects. When a worker
needs a buffer:

1. Try to pop from free list (lock-free)
2. If empty, allocate new object
3. When done, return object to pool

The pool is thread-safe and grows dynamically. Objects are never freed until
program termination, eliminating deallocation overhead entirely.


2.3 SIMD ABSTRACTION LAYER
------------------------------------------------------------------------------

File: src/transforms/simd_utils.hpp

TurboLoader provides a thin abstraction over SIMD intrinsics to support
multiple ISAs:

#ifdef __AVX2__
    // Use 256-bit AVX2 intrinsics
    __m256i vec = _mm256_load_si256(...)
#elif defined(__ARM_NEON)
    // Use 128-bit NEON intrinsics
    uint8x16_t vec = vld1q_u8(...)
#else
    // Scalar fallback
    for (size_t i = 0; i < size; ++i) { ... }
#endif

Key SIMD Operations Implemented:
- cvt_u8_to_f32_normalized: Convert uint8 [0,255] to float [0.0,1.0]
- mul_u8_scalar: Multiply uint8 pixels by scalar (brightness)
- horizontal_flip: Flip image horizontally using SIMD loads/stores
- bilinear_interpolation: SIMD-accelerated image resizing

The SIMD functions process 32 bytes (AVX2) or 16 bytes (NEON) per iteration,
achieving 8-16x speedup over scalar code.


================================================================================
SECTION 3: DETAILED CODEBASE STRUCTURE
================================================================================

3.1 DIRECTORY LAYOUT
------------------------------------------------------------------------------

/Users/arnavjain/turboloader/
├── CMakeLists.txt                # Root build configuration
├── pyproject.toml                # Python package metadata
├── setup.py                      # Python build script
├── README.md                     # User-facing documentation
├── CHANGELOG.md                  # Version history
├── ARCHITECTURE.md               # Architecture documentation
│
├── src/                          # C++ source code
│   ├── core/                     # Core data structures
│   │   ├── object_pool.hpp       # Thread-safe object pool
│   │   ├── sample.hpp            # Sample data structure
│   │   └── spsc_ring_buffer.hpp  # Lock-free queue
│   │
│   ├── decode/                   # Image/video decoders
│   │   ├── jpeg_decoder.hpp      # libjpeg-turbo decoder
│   │   ├── png_decoder.hpp       # libpng decoder
│   │   ├── webp_decoder.hpp      # libwebp decoder
│   │   ├── nvjpeg_decoder.hpp    # GPU JPEG decoder (CUDA)
│   │   ├── image_decoder.hpp     # Multi-format dispatcher
│   │   ├── video_decoder.hpp     # FFmpeg video decoder
│   │   ├── csv_decoder.hpp       # CSV parser
│   │   └── parquet_decoder.hpp   # Apache Parquet reader
│   │
│   ├── formats/                  # Custom binary formats
│   │   ├── tbl_format.hpp        # TBL v2 binary format spec
│   │   ├── tbl_reader_v2.hpp     # TBL v2 reader with LZ4
│   │   └── tbl_writer_v2.hpp     # TBL v2 streaming writer
│   │
│   ├── gpu/                      # GPU-specific code
│   │   ├── multi_gpu_pipeline.hpp    # Multi-GPU pipeline
│   │   └── multi_gpu_pipeline.cpp    # Implementation
│   │
│   ├── io/                       # I/O abstractions
│   │   └── io_uring_reader.hpp   # Linux async I/O (io_uring)
│   │
│   ├── pipeline/                 # Core pipeline logic
│   │   ├── pipeline.hpp          # Main UnifiedPipeline class (includes distributed training)
│   │   ├── prefetch_pipeline.hpp # Double-buffer prefetching
│   │   └── smart_batching.hpp    # Size-aware batching (NEW in v1.7.0)
│   │
│   ├── python/                   # Python bindings
│   │   └── turboloader_bindings.cpp  # pybind11 bindings
│   │
│   ├── readers/                  # Data source readers
│   │   ├── tar_reader.hpp        # TAR archive reader (mmap)
│   │   ├── tbl_reader.hpp        # TBL format reader
│   │   ├── http_reader.hpp       # HTTP/HTTPS remote loading
│   │   ├── s3_reader.hpp         # AWS S3 reader
│   │   ├── gcs_reader.hpp        # Google Cloud Storage
│   │   └── reader_orchestrator.hpp   # Auto source detection
│   │
│   ├── transforms/               # 19 SIMD-accelerated transforms
│   │   ├── simd_utils.hpp        # SIMD abstraction layer (AVX2/NEON)
│   │   ├── transform_base.hpp    # Base transform class
│   │   ├── transforms.hpp        # All transforms header
│   │   ├── resize_transform.hpp  # Bilinear/bicubic/Lanczos resize
│   │   ├── normalize_transform.hpp   # Mean/std normalization
│   │   ├── crop_transform.hpp    # Center/random crop
│   │   ├── flip_transform.hpp    # Horizontal/vertical flip
│   │   ├── color_jitter_transform.hpp  # Brightness/contrast/saturation
│   │   ├── rotation_transform.hpp      # Arbitrary rotation
│   │   ├── affine_transform.hpp        # Affine transformations
│   │   ├── blur_transform.hpp          # Gaussian blur
│   │   ├── pad_transform.hpp           # Padding
│   │   ├── erasing_transform.hpp       # Random erasing (Cutout)
│   │   ├── grayscale_transform.hpp     # Color to grayscale
│   │   ├── perspective_transform.hpp   # Perspective warp (NEW in v1.5.1)
│   │   ├── posterize_transform.hpp     # Bit depth reduction (NEW in v1.5.1)
│   │   ├── solarize_transform.hpp      # Threshold inversion (NEW in v1.5.1)
│   │   ├── autoaugment_transform.hpp   # AutoAugment policies (NEW in v1.5.1)
│   │   └── tensor_conversion.hpp       # PyTorch/TF/JAX conversion
│   │
│   └── writers/                  # Data format writers
│       ├── tbl_writer.hpp        # Legacy TBL writer
│       └── tbl_writer_v2.hpp     # TBL v2 streaming writer (NEW in v1.5.0)
│
├── tests/                        # C++ test suite
│   ├── CMakeLists.txt            # Test build configuration
│   ├── test_tar_reader.cpp       # TAR reader tests
│   ├── test_image_decoder.cpp    # Decoder tests
│   ├── test_http_reader.cpp      # HTTP reader tests
│   ├── test_unified_pipeline.cpp # Pipeline integration tests
│   ├── test_transforms.cpp       # Transform tests
│   ├── test_nvjpeg_decoder.cpp   # GPU decode tests
│   ├── test_pipeline_gpu_decode.cpp  # GPU pipeline tests
│   ├── test_smart_batching.cpp   # Batching tests
│   ├── test_avx512_simd.cpp      # AVX-512 SIMD tests
│   ├── test_tbl_format.cpp       # TBL format tests
│   └── ... (20+ test files)
│
├── benchmarks/                   # Performance benchmarks
│   ├── 01_pil_baseline.py        # PIL baseline benchmark
│   ├── 02_pytorch_naive.py       # PyTorch DataLoader
│   ├── 03_pytorch_optimized.py   # Optimized PyTorch
│   ├── 05_turboloader.py         # TurboLoader benchmark
│   ├── 08_tensorflow.py          # TensorFlow tf.data
│   └── run_all_benchmarks.py     # Benchmark suite runner
│
├── examples/                     # Usage examples
│   ├── transform_example.py      # Basic transform usage
│   ├── avx512_performance.py     # AVX-512 demo
│   ├── tbl_conversion.py         # TAR to TBL conversion
│   └── complete_v110_workflow.py # Complete pipeline example
│
├── python/                       # Pure Python utilities
│   ├── webdataset_loader.py      # WebDataset compatibility
│   ├── tensorflow_dataloader.py  # TensorFlow integration
│   └── jax_dataloader.py         # JAX integration
│
└── turboloader/                  # Python package
    └── __init__.py               # Package entry point


3.2 CRITICAL SOURCE FILES - DEEP DIVE
------------------------------------------------------------------------------

FILE: src/pipeline/pipeline.hpp (500+ lines)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

This is the heart of TurboLoader. It implements the UnifiedPipeline class which
orchestrates the entire data loading process.

Key Classes:

1. UnifiedPipelineConfig
   Configuration structure containing all pipeline parameters:
   - num_workers: Number of worker threads (default: 4)
   - batch_size: Samples per batch (default: 32)
   - queue_size: Buffer size for each worker (default: 128)
   - shuffle: Whether to shuffle samples
   - prefetch: Enable double-buffering
   - use_gpu_decode: Use nvJPEG for GPU JPEG decoding
   - smart_batching: Enable size-aware batching (NEW in v1.7.0)

   ===== Distributed Training (NEW in v1.7.1) =====
   - enable_distributed: Enable multi-node data loading (default: false)
   - world_rank: Rank of this process, 0 to world_size-1 (default: 0)
   - world_size: Total number of processes (default: 1)
   - drop_last: Drop incomplete batches at end (default: false)
   - distributed_seed: Seed for shuffling, same across all ranks (default: 42)

2. TarWorker
   Each worker thread runs an instance of this class. Worker lifecycle:

   a) INITIALIZATION PHASE:
      - Create thread-local JPEGDecoder (or NvJpegDecoder if GPU enabled)
      - Initialize memory pools for decoded images
      - Set CPU affinity to reduce migration overhead

   b) PROCESSING LOOP:
      while (running) {
          1. Pull TarEntryMetadata from shared queue (lock-free pop)
          2. Memory-map the JPEG region (zero-copy via TarReader)
          3. Decode JPEG to RGB888:
             - If use_gpu_decode && nvJPEG available: GPU decode
             - Else: libjpeg-turbo SIMD decode
          4. Apply transform pipeline (resize, normalize, augment)
          5. Push decoded Sample to output queue
      }

   c) SHUTDOWN:
      - Drain remaining samples from queue
      - Clean up decoder resources
      - Join thread

3. UnifiedPipeline
   The main pipeline class. Usage pattern:

   // Create pipeline
   UnifiedPipelineConfig config;
   config.num_workers = 8;
   config.batch_size = 64;
   config.use_gpu_decode = true;

   UnifiedPipeline pipeline("/path/to/dataset.tar", config);

   // Iterate over batches
   for (size_t epoch = 0; epoch < num_epochs; ++epoch) {
       pipeline.reset();  // Reset for new epoch

       while (auto batch = pipeline.next_batch()) {
           // batch contains 64 samples
           // Process batch (forward/backward pass)
       }
   }

   CRITICAL IMPLEMENTATION DETAILS:

   Memory Management:
   - Workers allocate Sample objects from thread-local pools
   - Samples are moved (not copied) through the pipeline
   - RGB data uses std::vector with reserve() to avoid reallocations

   Thread Synchronization:
   - Worker startup uses condition variable to ensure all threads ready
   - Sample queues use lock-free SPSC buffers (one queue per worker)
   - Shutdown uses atomic flag checked in hot loop

   GPU Decode Path (lines 379-409):
   When use_gpu_decode=true and nvJPEG available:

   #ifdef HAVE_NVJPEG
   if (use_gpu_ && nvjpeg_decoder_) {
       NvJpegResult gpu_result;
       // Decode on GPU - this is ~10x faster than CPU
       nvjpeg_decoder_->decode(jpeg_data, size, gpu_result);

       // Result is in GPU memory, copy to CPU
       sample.image_data = std::move(gpu_result.rgb_data);
   }
   #endif

   If GPU unavailable or disabled, falls back to CPU:
   decoder_->decode_sample(sample);  // libjpeg-turbo


FILE: src/decode/jpeg_decoder.hpp (300+ lines)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Implements JPEG decoding using libjpeg-turbo, which uses SIMD intrinsics
(SSE2, AVX2, NEON) for fast decompression.

Key Implementation Details:

1. Thread-Local Decoder State:
   Each JPEGDecoder instance is used by exactly one thread. This eliminates
   lock contention and allows reusing the jpeg_decompress_struct across
   multiple images.

   struct jpeg_decompress_struct cinfo_;
   struct jpeg_error_mgr jerr_;
   bool initialized_;

2. Decode Process:
   void decode_sample(Sample& sample) {
       // Point libjpeg at input JPEG bytes (no copy)
       jpeg_mem_src(&cinfo_, jpeg_data, jpeg_size);

       // Read JPEG header
       jpeg_read_header(&cinfo_, TRUE);

       // Configure output: RGB888, no color space conversion
       cinfo_.out_color_space = JCS_RGB;
       cinfo_.dct_method = JDCT_IFAST;  // Fast integer DCT

       // Start decompression
       jpeg_start_decompress(&cinfo_);

       // Allocate output buffer
       size_t row_stride = cinfo_.output_width * 3;
       sample.decoded_rgb.resize(cinfo_.output_height * row_stride);

       // Read scanlines using SIMD-accelerated path
       uint8_t* row_ptr = sample.decoded_rgb.data();
       while (cinfo_.output_scanline < cinfo_.output_height) {
           jpeg_read_scanlines(&cinfo_, &row_ptr, 1);
           row_ptr += row_stride;
       }

       // Cleanup
       jpeg_finish_decompress(&cinfo_);
   }

3. Error Handling:
   Uses setjmp/longjmp for error recovery. If JPEG is corrupted:
   - Log error message
   - Skip sample (don't crash entire training run)
   - Continue processing next sample

4. Performance Optimizations:
   - JDCT_IFAST: Fast integer DCT (trades accuracy for speed)
   - No color space conversion (YCbCr->RGB done by libjpeg-turbo)
   - Reuse decompressor struct (avoid re-initialization overhead)


FILE: src/transforms/resize_transform.hpp (400+ lines)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Implements bilinear and bicubic image resizing using SIMD intrinsics.

Bilinear Interpolation Algorithm:

For each output pixel (x_out, y_out):
1. Map to source coordinates:
   x_src = x_out * (src_width / dst_width)
   y_src = y_out * (src_height / dst_height)

2. Find four surrounding pixels:
   (x0, y0) = floor(x_src, y_src)
   (x1, y1) = (x0 + 1, y0 + 1)

3. Compute interpolation weights:
   wx = x_src - x0
   wy = y_src - y0

4. Bilinear interpolation:
   top = (1-wx) * src[y0,x0] + wx * src[y0,x1]
   bot = (1-wx) * src[y1,x0] + wx * src[y1,x1]
   result = (1-wy) * top + wy * bot

SIMD Optimization:

The inner loop processes 8 pixels simultaneously using AVX2:

__m256 weight_x = _mm256_set1_ps(wx);
__m256 weight_y = _mm256_set1_ps(wy);

// Load 8 pixels from each corner
__m256 top_left = _mm256_load_ps(src + y0*stride + x0);
__m256 top_right = _mm256_load_ps(src + y0*stride + x1);
__m256 bot_left = _mm256_load_ps(src + y1*stride + x0);
__m256 bot_right = _mm256_load_ps(src + y1*stride + x1);

// Horizontal interpolation
__m256 top = _mm256_add_ps(
    _mm256_mul_ps(_mm256_sub_ps(one, weight_x), top_left),
    _mm256_mul_ps(weight_x, top_right)
);
__m256 bot = _mm256_add_ps(
    _mm256_mul_ps(_mm256_sub_ps(one, weight_x), bot_left),
    _mm256_mul_ps(weight_x, bot_right)
);

// Vertical interpolation
__m256 result = _mm256_add_ps(
    _mm256_mul_ps(_mm256_sub_ps(one, weight_y), top),
    _mm256_mul_ps(weight_y, bot)
);

_mm256_store_ps(dst + y_out*out_stride + x_out, result);

This achieves ~8x speedup over scalar code on AVX2 processors.


FILE: src/readers/tar_reader.hpp (250+ lines)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Implements memory-mapped TAR file reading with zero-copy access.

TAR Format Background:
TAR files consist of a sequence of:
- 512-byte header (filename, size, permissions, etc.)
- File data (padded to 512-byte boundary)
- Repeat

TarReader Implementation:

1. Memory Mapping:
   int fd = open(tar_path, O_RDONLY);
   struct stat st;
   fstat(fd, &st);

   // Map entire TAR file into process address space
   void* mapped = mmap(NULL, st.st_size, PROT_READ,
                       MAP_PRIVATE | MAP_POPULATE, fd, 0);

   // Advise kernel about access pattern
   madvise(mapped, st.st_size, MADV_SEQUENTIAL);

   Benefits:
   - No read() system calls (kernel handles page faults)
   - Zero-copy: JPEG data accessed directly from mapped memory
   - Kernel automatically manages page cache
   - MAP_POPULATE pre-faults pages for predictable latency

2. TAR Header Parsing:
   struct TarHeader {
       char name[100];
       char mode[8];
       char uid[8];
       char gid[8];
       char size[12];    // Octal string!
       char mtime[12];
       char checksum[8];
       char typeflag;
       // ... more fields
   };

   The size field is in OCTAL (legacy Unix format):
   size_t parse_octal(const char* str, size_t len) {
       size_t result = 0;
       for (size_t i = 0; i < len && str[i]; ++i) {
           if (str[i] < '0' || str[i] > '7') break;
           result = result * 8 + (str[i] - '0');
       }
       return result;
   }

3. Entry Indexing:
   On initialization, TarReader scans entire TAR file and builds an index:

   std::vector<TarEntryMetadata> entries_;

   struct TarEntryMetadata {
       std::string filename;
       size_t offset;        // Byte offset in mapped region
       size_t size;          // File size in bytes
       const uint8_t* data;  // Pointer into mmap'd region (zero-copy!)
   };

   This index allows O(1) access to any file in the TAR.

4. Shuffling Support:
   If shuffling enabled:
   std::random_device rd;
   std::mt19937 g(rd());
   std::shuffle(entries_.begin(), entries_.end(), g);

   This shuffles the entry vector, not the file data itself (efficient).


================================================================================
SECTION 4: BUILD SYSTEM COMPREHENSIVE GUIDE
================================================================================

4.1 CMAKE CONFIGURATION
------------------------------------------------------------------------------

TurboLoader uses CMake 3.15+ as its build system. The build is configured to
detect available dependencies and enable features conditionally.

Root CMakeLists.txt Structure:

1. Project Setup:
   cmake_minimum_required(VERSION 3.15)
   project(TurboLoader VERSION 1.3.0 LANGUAGES CXX)
   set(CMAKE_CXX_STANDARD 20)
   set(CMAKE_CXX_STANDARD_REQUIRED ON)

   C++20 is required for:
   - std::span (zero-copy array views)
   - Concepts (for template constraints)
   - <bit> header (bit_cast for type punning)

2. Compiler Flags:
   if (CMAKE_CXX_COMPILER_ID MATCHES "GNU|Clang")
       add_compile_options(
           -O3                    # Aggressive optimization
           -march=native          # Use all available CPU instructions
           -ffast-math            # Fast floating-point (trade precision)
           -Wall -Wextra          # Enable warnings
           -Wno-unused-parameter  # Suppress unused param warnings
       )
   endif()

   -march=native is critical: enables AVX2/AVX-512 on Intel, NEON on ARM.

3. Dependency Detection:
   # libjpeg-turbo (required)
   find_package(JPEG REQUIRED)

   # libpng (required)
   find_package(PNG REQUIRED)

   # libwebp (required)
   find_package(WebP REQUIRED)

   # CURL (required for HTTP/S3/GCS readers)
   find_package(CURL REQUIRED)

   # nvJPEG (optional - GPU JPEG decode)
   check_language(CUDA)
   if (CMAKE_CUDA_COMPILER)
       enable_language(CUDA)
       find_library(NVJPEG_LIBRARY nvjpeg)
       if (NVJPEG_LIBRARY)
           add_definitions(-DHAVE_NVJPEG)
       endif()
   endif()

   # io_uring (optional - Linux async I/O)
   if (CMAKE_SYSTEM_NAME STREQUAL "Linux")
       find_library(URING_LIBRARY uring)
       if (URING_LIBRARY)
           add_definitions(-DHAVE_IOURING)
       endif()
   endif()

4. Library Target:
   add_library(turboloader INTERFACE)
   target_include_directories(turboloader INTERFACE src)
   target_link_libraries(turboloader INTERFACE
       ${JPEG_LIBRARIES}
       ${PNG_LIBRARIES}
       ${WEBP_LIBRARIES}
       CURL::libcurl
   )

   INTERFACE library means turboloader is header-only for C++ users.
   Python bindings compile to a shared library (_turboloader.so).

5. Python Bindings:
   find_package(pybind11 REQUIRED)

   pybind11_add_module(_turboloader
       src/python/turboloader_bindings.cpp
   )
   target_link_libraries(_turboloader PRIVATE turboloader)

   This creates _turboloader.cpython-313-darwin.so (on macOS) or
   _turboloader.cpython-313-x86_64-linux-gnu.so (on Linux).


4.2 DEPENDENCY MANAGEMENT
------------------------------------------------------------------------------

REQUIRED DEPENDENCIES:

1. libjpeg-turbo
   Purpose: SIMD-accelerated JPEG decoding
   Install: brew install jpeg-turbo (macOS)
           apt install libjpeg-turbo8-dev (Ubuntu)
   Version: 2.0+

   Why libjpeg-turbo vs libjpeg?
   - 2-6x faster decoding via SIMD intrinsics
   - Compatible with libjpeg API
   - Widely used in production (Chrome, Android)

2. libpng
   Purpose: PNG image decoding
   Install: brew install libpng (macOS)
           apt install libpng-dev (Ubuntu)
   Version: 1.6+

3. libwebp
   Purpose: WebP image decoding (modern format, better compression)
   Install: brew install webp (macOS)
           apt install libwebp-dev (Ubuntu)
   Version: 1.0+

4. libcurl
   Purpose: HTTP/HTTPS support for remote data loading
   Install: brew install curl (macOS)
           apt install libcurl4-openssl-dev (Ubuntu)
   Version: 7.50+

5. pybind11
   Purpose: C++ <-> Python bindings
   Install: pip install pybind11
   Version: 2.10+

OPTIONAL DEPENDENCIES:

6. NVIDIA CUDA + nvJPEG
   Purpose: GPU-accelerated JPEG decoding (10x speedup)
   Install: CUDA Toolkit 11.0+
   Platforms: Linux, Windows (not macOS)

   Detection: CMake checks for CUDA compiler and nvjpeg library
   If found: Compiles with -DHAVE_NVJPEG
   If not found: Falls back to CPU decode (graceful degradation)

7. liburing
   Purpose: Linux io_uring for async I/O
   Install: apt install liburing-dev (Ubuntu 20.04+)
   Kernel: Linux 5.1+

   Detection: CMake checks for liburing
   If found: Compiles with -DHAVE_IOURING
   If not found: Uses standard I/O

8. FFmpeg
   Purpose: Video decoding support
   Install: brew install ffmpeg (macOS)
           apt install libavcodec-dev libavformat-dev (Ubuntu)
   Optional: Only needed if loading video datasets

9. Apache Arrow
   Purpose: Parquet file support (columnar data)
   Install: brew install apache-arrow (macOS)
           apt install libarrow-dev libparquet-dev (Ubuntu)
   Optional: Only needed for Parquet datasets


================================================================================
SECTION 5: STEP-BY-STEP INSTALLATION
================================================================================

5.1 FULL INSTALLATION FROM SOURCE (macOS)
------------------------------------------------------------------------------

This section provides a complete, step-by-step guide for building TurboLoader
from source on macOS. We'll build both the C++ library and Python bindings.

STEP 1: Install Homebrew (if not already installed)

    /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

STEP 2: Install System Dependencies

    # Core dependencies
    brew install cmake           # Build system (3.15+)
    brew install python@3.13     # Python runtime
    brew install jpeg-turbo      # SIMD JPEG decoding
    brew install libpng          # PNG decoding
    brew install webp            # WebP decoding
    brew install curl            # HTTP support

    # Optional dependencies
    brew install ffmpeg          # Video decoding (optional)
    brew install apache-arrow    # Parquet support (optional)

    Verify installations:

    cmake --version              # Should show 3.15+
    python3.13 --version         # Should show 3.13.x
    brew list jpeg-turbo         # Confirm jpeg-turbo installed
    brew list libpng             # Confirm libpng installed
    brew list webp               # Confirm webp installed

STEP 3: Install Python Dependencies

    # Create virtual environment (recommended)
    python3.13 -m venv ~/turboloader-env
    source ~/turboloader-env/bin/activate

    # Install build dependencies
    pip install --upgrade pip
    pip install pybind11>=2.10.0
    pip install cmake>=3.15
    pip install setuptools>=45
    pip install wheel
    pip install build

    # Install runtime dependencies
    pip install numpy>=1.19.0
    pip install torch>=1.8.0     # PyTorch integration

    # Optional: Install benchmarking/testing tools
    pip install Pillow>=8.0.0
    pip install tqdm>=4.60.0
    pip install matplotlib>=3.3.0
    pip install psutil>=5.8.0

STEP 4: Clone Repository

    cd ~/
    git clone https://github.com/ALJainProjects/TurboLoader.git
    cd TurboLoader

    Verify directory structure:

    ls -la src/           # Should see core/, decode/, transforms/, etc.
    ls -la tests/         # Should see test_*.cpp files
    ls -la CMakeLists.txt # Root CMake config should exist

STEP 5: Build C++ Library and Tests

    # Create build directory
    mkdir -p build
    cd build

    # Configure with CMake
    cmake -DCMAKE_BUILD_TYPE=Release \
          -DCMAKE_CXX_COMPILER=clang++ \
          -DCMAKE_CXX_FLAGS="-O3 -march=native" \
          ..

    # Compile (use all CPU cores)
    make -j$(sysctl -n hw.ncpu)

    Expected output:

    -- Detecting dependencies...
    -- libjpeg-turbo: /opt/homebrew/opt/jpeg-turbo/include
    -- libpng: /opt/homebrew/opt/libpng/include
    -- libwebp: /opt/homebrew/opt/webp/include
    -- libcurl: /Library/Developer/CommandLineTools/SDKs/MacOSX15.sdk/usr/include
    ...
    [100%] Built target turboloader

    This creates:
    - build/tests/test_*  (C++ test executables)
    - Compiled object files and libraries

STEP 6: Run C++ Test Suite

    # Run all tests via CTest
    cd /Users/arnavjain/turboloader/build
    ctest --output-on-failure -j8

    Expected output:

    Test project /Users/arnavjain/turboloader/build
          Start  1: tar_reader
     1/17 Test  #1: tar_reader .......................   Passed    0.05 sec
          Start  2: image_decoder
     2/17 Test  #2: image_decoder ....................   Passed    0.12 sec
          Start  3: http_reader
     3/17 Test  #3: http_reader ......................   Passed    1.23 sec
    ...
    100% tests passed, 0 tests failed out of 17

    If any tests fail, check the specific test output for details.

STEP 7: Build Python Package

    # Return to root directory
    cd /Users/arnavjain/turboloader

    # Build wheel
    python3.13 -m build

    Expected output:

    * Creating isolated environment: venv+pip...
    * Installing packages in isolated environment:
      - cmake>=3.15
      - pybind11>=2.10.0
      - setuptools>=45
      - wheel
    ...
    Successfully built turboloader-1.7.1.tar.gz and
    turboloader-1.7.1-cp313-cp313-macosx_15_0_arm64.whl

    This creates:
    - dist/turboloader-1.7.1.tar.gz (source distribution)
    - dist/turboloader-1.7.1-cp313-cp313-macosx_15_0_arm64.whl (wheel)

STEP 8: Install Python Package

    # Install from built wheel
    pip install dist/turboloader-1.7.1-cp313-cp313-macosx_15_0_arm64.whl

    Or install in development mode (editable):

    pip install -e .

    Verify installation:

    python3.13 -c "import turboloader; print(turboloader.__version__)"
    # Should output: 1.7.1

STEP 9: Verify Installation

    # Test Python import
    python3.13 -c "
    import turboloader
    import numpy as np
    print(f'TurboLoader version: {turboloader.__version__}')
    print('Successfully imported TurboLoader!')
    "

    Expected output:

    TurboLoader version: 1.7.1
    Successfully imported TurboLoader!


5.2 FULL INSTALLATION FROM SOURCE (Ubuntu/Linux)
------------------------------------------------------------------------------

STEP 1: Install System Dependencies

    # Update package list
    sudo apt update

    # Core dependencies
    sudo apt install -y build-essential cmake git
    sudo apt install -y python3-dev python3-pip python3-venv
    sudo apt install -y libjpeg-turbo8-dev
    sudo apt install -y libpng-dev
    sudo apt install -y libwebp-dev
    sudo apt install -y libcurl4-openssl-dev

    # Optional: io_uring support (Linux 5.1+ kernel)
    sudo apt install -y liburing-dev

    # Optional: FFmpeg video support
    sudo apt install -y libavcodec-dev libavformat-dev libavutil-dev libswscale-dev

    # Optional: Parquet support
    sudo apt install -y libarrow-dev libparquet-dev

STEP 2-9: Follow same steps as macOS section above

    The remaining steps (Python dependencies, clone repo, build, test, install)
    are identical to the macOS instructions.


5.3 INSTALLATION VIA PIP (Pre-built)
------------------------------------------------------------------------------

For users who want pre-built binaries without compiling from source:

STEP 1: Install from PyPI

    pip install turboloader==1.7.1

STEP 2: Verify Installation

    python -c "import turboloader; print(turboloader.__version__)"

NOTE: Pre-built wheels are currently available for:
- macOS ARM64 (M1/M2/M3/M4 chips)
- Linux x86_64 (glibc 2.17+)
- Python 3.8, 3.9, 3.10, 3.11, 3.12, 3.13, 3.14

If your platform is not supported, you must build from source (see above).


================================================================================
SECTION 6: RUNNING THE TEST SUITE
================================================================================

6.1 C++ TEST SUITE OVERVIEW
------------------------------------------------------------------------------

TurboLoader includes a comprehensive C++ test suite using Google Test framework.
The tests validate correctness of all core components.

Test Categories:

1. Unit Tests (test individual components)
2. Integration Tests (test pipeline end-to-end)
3. Performance Tests (validate throughput claims)

Total Test Count: 17 test executables, 100+ individual test cases


6.2 RUNNING ALL TESTS
------------------------------------------------------------------------------

METHOD 1: Using CTest (Recommended)

    cd /Users/arnavjain/turboloader/build
    ctest --output-on-failure -j8

    Flags explained:
    --output-on-failure: Only show output for failing tests
    -j8: Run 8 tests in parallel (faster)

METHOD 2: Running Individual Tests

    # Run single test
    ./tests/test_tar_reader

    # Run with verbose output
    ./tests/test_tar_reader --gtest_verbose

    # Run specific test case
    ./tests/test_tar_reader --gtest_filter=TarReaderTest.LoadSamples


6.3 DETAILED TEST DESCRIPTIONS
------------------------------------------------------------------------------

TEST: test_tar_reader.cpp
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Purpose: Validates TAR file parsing, memory mapping, and entry extraction

Test Cases:
1. ParseTarHeader: Verify TAR header parsing (octal size conversion, etc.)
2. MemoryMapping: Validate mmap() succeeds and MAP_POPULATE works
3. EntryExtraction: Extract files and verify contents match original
4. LargeFile: Test with TAR > 2GB to verify offset handling
5. CorruptedTar: Ensure graceful handling of malformed TAR files

How to Run:
    ./tests/test_tar_reader

Expected Output:
    [==========] Running 5 tests from 1 test suite.
    [----------] Global test environment set-up.
    [----------] 5 tests from TarReaderTest
    [ RUN      ] TarReaderTest.ParseTarHeader
    [       OK ] TarReaderTest.ParseTarHeader (1 ms)
    [ RUN      ] TarReaderTest.MemoryMapping
    [       OK ] TarReaderTest.MemoryMapping (5 ms)
    ...
    [==========] 5 tests from 1 test suite ran. (23 ms total)
    [  PASSED  ] 5 tests.


TEST: test_image_decoder.cpp
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Purpose: Validates JPEG, PNG, WebP, BMP, TIFF decoding

Test Cases:
1. DecodeJPEG: Decode JPEG and verify dimensions, pixel values
2. DecodePNG: Decode PNG with alpha channel
3. DecodeWebP: Decode WebP (lossy and lossless)
4. DecodeBMP: Decode BMP (uncompressed format)
5. DecodeTIFF: Decode TIFF with various compressions

How to Run:
    ./tests/test_image_decoder

Performance Validation:
    The test measures decoding throughput:
    - JPEG: Should achieve 200+ images/sec
    - PNG: Should achieve 150+ images/sec
    - WebP: Should achieve 180+ images/sec


TEST: test_unified_pipeline.cpp
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Purpose: End-to-end pipeline integration test

Test Cases:
1. BasicPipeline: Load TAR, decode, batch, verify output
2. MultiWorker: Test with 1, 2, 4, 8 workers, verify throughput scales
3. Shuffling: Verify samples are shuffled between epochs
4. Transforms: Apply resize, normalize, verify output correctness
5. ErrorHandling: Inject corrupted images, verify pipeline continues

How to Run:
    ./tests/test_unified_pipeline

Expected Behavior:
    - All workers start successfully
    - Batches have correct size (last batch may be smaller)
    - Throughput increases with worker count
    - Pipeline handles errors gracefully (logs warning, skips sample)


TEST: test_transforms.cpp (Google Test)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Purpose: Validates all 19 image transforms

Test Cases:
1. ResizeTest: Bilinear/bicubic resize, verify pixel interpolation
2. NormalizeTest: Mean/std normalization, verify output range
3. CropTest: Center/random crop, verify output dimensions
4. FlipTest: Horizontal/vertical flip, verify pixel ordering
5. ColorJitterTest: Brightness/contrast/saturation, verify ranges
6. RotationTest: Arbitrary angle rotation, verify no artifacts
7. ... (19 total transform tests)

How to Run:
    ./tests/test_transforms

SIMD Validation:
    Each test runs twice:
    1. With SIMD enabled (AVX2 or NEON)
    2. With SIMD disabled (scalar fallback)

    Results are compared to ensure SIMD implementation matches scalar.


TEST: test_pipeline_gpu_decode.cpp
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Purpose: Validates GPU-accelerated JPEG decoding via nvJPEG

Test Cases:
1. GPUDecodeEnabled: Test with use_gpu_decode=true
2. GPUDecodeDisabled: Test with use_gpu_decode=false (CPU path)
3. GPUCPUConsistency: Decode same image on GPU and CPU, compare results
4. MultiWorkerGPU: Test 4 workers each with own GPU decoder
5. GPUPerformance: Benchmark GPU vs CPU decode speed

How to Run:
    ./tests/test_pipeline_gpu_decode

NOTE: This test requires CUDA and nvJPEG. If not available, test will
      use CPU fallback and log a warning (not a failure).

Expected Performance (on NVIDIA GPU):
    - GPU decode: 1,000+ images/sec
    - CPU decode: 100-200 images/sec
    - GPU is 10x faster

Consistency Validation:
    GPU and CPU decoders should produce identical output (pixel-perfect).
    The test allows <2 difference per channel due to rounding.


6.4 TEST DATA GENERATION
------------------------------------------------------------------------------

Many tests require sample data (images, TAR files). TurboLoader includes
scripts to generate test datasets.

GENERATING A TEST TAR FILE:

    python3.13 - <<'EOF'
    from PIL import Image
    import tarfile
    from pathlib import Path
    import random

    # Create 100 random images
    Path("/tmp/test_images").mkdir(exist_ok=True)

    for i in range(100):
        img = Image.new('RGB', (256, 256))
        pixels = img.load()
        for x in range(256):
            for y in range(256):
                pixels[x, y] = (
                    random.randint(0, 255),
                    random.randint(0, 255),
                    random.randint(0, 255)
                )
        img.save(f"/tmp/test_images/{i:04d}.jpg", quality=90)

    # Create TAR archive
    with tarfile.open("/tmp/test_dataset.tar", "w") as tar:
        for i in range(100):
            tar.add(f"/tmp/test_images/{i:04d}.jpg",
                   arcname=f"{i:04d}.jpg")

    print("Created /tmp/test_dataset.tar with 100 images")
    EOF

Now you can use /tmp/test_dataset.tar for testing:

    ./tests/test_unified_pipeline /tmp/test_dataset.tar


================================================================================
SECTION 7: BENCHMARKING FRAMEWORK
================================================================================

7.1 BENCHMARKING OVERVIEW
------------------------------------------------------------------------------

TurboLoader includes a comprehensive benchmarking suite to validate performance
claims against competing frameworks: PyTorch DataLoader, TensorFlow tf.data,
FFCV, DALI, etc.

Benchmark Categories:
1. Baseline: Pure Python/PIL (slowest, for reference)
2. PyTorch: Standard DataLoader and optimized variants
3. TensorFlow: tf.data API
4. TurboLoader: Our implementation
5. Specialized: FFCV, NVIDIA DALI

All benchmarks measure:
- Throughput (images/second)
- CPU utilization
- Memory usage
- Latency (time to first batch)


7.2 RUNNING BENCHMARKS
------------------------------------------------------------------------------

PREREQUISITE: Create benchmark dataset

    The benchmarks need a TAR file with 1000+ images for accurate measurement.

    # Generate 1000-image dataset (~250MB)
    python3.13 - <<'EOF'
    from PIL import Image
    import tarfile
    from pathlib import Path
    import random

    Path("/tmp/benchmark_dataset").mkdir(exist_ok=True)

    print("Generating 1000 images...")
    for i in range(1000):
        img = Image.new('RGB', (256, 256))
        pixels = img.load()
        for x in range(256):
            for y in range(256):
                pixels[x, y] = (random.randint(0, 255),
                               random.randint(0, 255),
                               random.randint(0, 255))
        img.save(f"/tmp/benchmark_dataset/{i:04d}.jpg", quality=90)
        if (i + 1) % 100 == 0:
            print(f"  {i + 1}/1000 images generated")

    print("Creating TAR archive...")
    with tarfile.open("/tmp/benchmark_dataset.tar", "w") as tar:
        for i in range(1000):
            tar.add(f"/tmp/benchmark_dataset/{i:04d}.jpg",
                   arcname=f"{i:04d}.jpg")

    print("Done! Created /tmp/benchmark_dataset.tar")
    EOF

RUNNING INDIVIDUAL BENCHMARKS:

1. PIL Baseline:
    python benchmarks/01_pil_baseline.py /tmp/benchmark_dataset.tar

    Expected throughput: ~50-100 images/sec (single-threaded)

2. PyTorch Naive:
    python benchmarks/02_pytorch_naive.py /tmp/benchmark_dataset.tar

    Expected throughput: ~500-800 images/sec (4 workers)

3. PyTorch Optimized:
    python benchmarks/03_pytorch_optimized.py /tmp/benchmark_dataset.tar

    Expected throughput: ~1,200-1,500 images/sec (4 workers, prefetch)

4. TurboLoader:
    python benchmarks/05_turboloader.py /tmp/benchmark_dataset.tar

    Expected throughput: ~10,000-15,000 images/sec (4 workers)

5. TensorFlow:
    python benchmarks/08_tensorflow.py /tmp/benchmark_dataset.tar

    Expected throughput: ~7,000-9,000 images/sec (4 workers)

RUNNING ALL BENCHMARKS:
    python benchmarks/run_all_benchmarks.py /tmp/benchmark_dataset.tar

    This runs all benchmarks sequentially and generates a comparison report.


7.3 BENCHMARK DEEP DIVE: 05_turboloader.py
------------------------------------------------------------------------------

Let's examine the TurboLoader benchmark in detail to understand how to use
the API effectively.

FILE: benchmarks/05_turboloader.py

    #!/usr/bin/env python3
    import turboloader
    import time
    import sys

    def benchmark_turboloader(tar_path, num_workers=4, batch_size=32):
        """
        Benchmark TurboLoader pipeline with specified configuration.

        Args:
            tar_path: Path to TAR file containing images
            num_workers: Number of worker threads (default: 4)
            batch_size: Samples per batch (default: 32)
        """
        print(f"TurboLoader Benchmark")
        print(f"Workers: {num_workers}, Batch Size: {batch_size}")
        print(f"Dataset: {tar_path}")
        print("-" * 60)

        # Create pipeline configuration
        # This struct controls all pipeline behavior
        config = {
            'num_workers': num_workers,    # Worker thread count
            'batch_size': batch_size,      # Samples per batch
            'shuffle': True,               # Shuffle samples between epochs
            'prefetch': True,              # Enable double-buffering
            'use_gpu_decode': False,       # GPU decode (if available)
            'queue_size': 128,             # Per-worker queue depth
        }

        # Create pipeline instance
        # This initializes:
        # - Memory maps the TAR file
        # - Spawns worker threads
        # - Allocates queues and buffers
        pipeline = turboloader.UnifiedPipeline(tar_path, config)

        # Warm-up phase
        # The first batch is slower due to cold caches and thread startup
        # We discard warm-up results from throughput calculation
        print("Warming up...")
        warmup_batch = pipeline.next_batch()
        if warmup_batch is None:
            print("ERROR: Failed to load warm-up batch")
            return
        print(f"Warm-up complete. Batch shape: {warmup_batch.shape}")

        # Benchmark phase
        # We measure throughput over multiple batches to get stable numbers
        print("\nBenchmarking...")
        total_samples = 0
        total_time = 0.0
        batch_count = 0

        # Reset pipeline to start from beginning
        pipeline.reset()

        # Start timer
        start_time = time.perf_counter()

        # Iterate over all batches
        # pipeline.next_batch() returns None when dataset exhausted
        while True:
            batch = pipeline.next_batch()
            if batch is None:
                break  # End of dataset

            # Process batch (in real training, this would be forward/backward pass)
            # For benchmarking, we just count samples
            total_samples += batch.shape[0]
            batch_count += 1

        # End timer
        end_time = time.perf_counter()
        total_time = end_time - start_time

        # Calculate metrics
        throughput = total_samples / total_time
        avg_batch_time = (total_time / batch_count) * 1000  # milliseconds

        # Print results
        print(f"\nResults:")
        print(f"  Total Samples: {total_samples}")
        print(f"  Total Time: {total_time:.2f} seconds")
        print(f"  Throughput: {throughput:.0f} images/second")
        print(f"  Avg Batch Time: {avg_batch_time:.2f} ms")
        print(f"  Batches Processed: {batch_count}")

        return {
            'throughput': throughput,
            'total_time': total_time,
            'total_samples': total_samples,
            'batch_count': batch_count,
            'avg_batch_time': avg_batch_time,
        }

    if __name__ == '__main__':
        if len(sys.argv) < 2:
            print("Usage: python 05_turboloader.py <tar_path> [num_workers] [batch_size]")
            sys.exit(1)

        tar_path = sys.argv[1]
        num_workers = int(sys.argv[2]) if len(sys.argv) > 2 else 4
        batch_size = int(sys.argv[3]) if len(sys.argv) > 3 else 32

        benchmark_turboloader(tar_path, num_workers, batch_size)

RUNNING THE BENCHMARK:

    python benchmarks/05_turboloader.py /tmp/benchmark_dataset.tar 4 32

Expected output:

    TurboLoader Benchmark
    Workers: 4, Batch Size: 32
    Dataset: /tmp/benchmark_dataset.tar
    ------------------------------------------------------------
    Warming up...
    Warm-up complete. Batch shape: (32, 256, 256, 3)

    Benchmarking...

    Results:
      Total Samples: 1000
      Total Time: 0.08 seconds
      Throughput: 12,500 images/second
      Avg Batch Time: 2.56 ms
      Batches Processed: 32


7.4 SCALABILITY BENCHMARKS
------------------------------------------------------------------------------

A key claim is that TurboLoader scales efficiently with worker count. Let's
validate this with a scalability benchmark.

SCRIPT: benchmarks/scalability_test.py

    #!/usr/bin/env python3
    import turboloader
    import time
    import sys

    def test_scalability(tar_path, max_workers=16):
        """Test throughput scaling with worker count"""
        results = []

        for num_workers in [1, 2, 4, 8, 16]:
            if num_workers > max_workers:
                break

            print(f"\n{'='*60}")
            print(f"Testing with {num_workers} workers")
            print(f"{'='*60}")

            config = {
                'num_workers': num_workers,
                'batch_size': 64,
                'shuffle': True,
                'prefetch': True,
                'use_gpu_decode': False,
                'queue_size': 128,
            }

            pipeline = turboloader.UnifiedPipeline(tar_path, config)

            # Warm-up
            _ = pipeline.next_batch()
            pipeline.reset()

            # Benchmark
            start = time.perf_counter()
            total = 0
            while True:
                batch = pipeline.next_batch()
                if batch is None:
                    break
                total += batch.shape[0]
            elapsed = time.perf_counter() - start

            throughput = total / elapsed
            efficiency = (throughput / num_workers) / (results[0]['throughput'] if results else throughput)

            result = {
                'workers': num_workers,
                'throughput': throughput,
                'efficiency': efficiency,
                'speedup': throughput / results[0]['throughput'] if results else 1.0,
            }
            results.append(result)

            print(f"Throughput: {throughput:.0f} img/s")
            print(f"Speedup: {result['speedup']:.2f}x")
            print(f"Efficiency: {efficiency*100:.1f}%")

        print(f"\n{'='*60}")
        print("Scalability Summary")
        print(f"{'='*60}")
        print(f"{'Workers':<10} {'Throughput':<15} {'Speedup':<10} {'Efficiency':<10}")
        print("-" * 60)
        for r in results:
            print(f"{r['workers']:<10} {r['throughput']:>10.0f} img/s  {r['speedup']:>8.2f}x  {r['efficiency']*100:>8.1f}%")

    if __name__ == '__main__':
        if len(sys.argv) < 2:
            print("Usage: python scalability_test.py <tar_path>")
            sys.exit(1)
        test_scalability(sys.argv[1])

RUNNING SCALABILITY TEST:

    python benchmarks/scalability_test.py /tmp/benchmark_dataset.tar

Expected output (Apple M4 Max, 1000 images):

    ============================================================
    Scalability Summary
    ============================================================
    Workers    Throughput      Speedup    Efficiency
    ------------------------------------------------------------
    1           2,180 img/s      1.00x      100.0%
    2           4,020 img/s      1.84x       92.0%
    4           6,755 img/s      3.10x       77.5%
    8           6,973 img/s      3.20x       40.0%
    16         21,036 img/s      9.65x       60.3%

Analysis:
- Linear scaling up to 4 workers (77.5% efficiency)
- Efficiency drops at 8 workers (likely memory bandwidth saturation)
- Efficiency recovers at 16 workers (prefetching hides latency)


================================================================================
SECTION 8: PYTHON API & BINDINGS
================================================================================

8.1 PYTHON API OVERVIEW
------------------------------------------------------------------------------

TurboLoader provides a Pythonic API that feels natural to PyTorch/TensorFlow
users while exposing the full power of the underlying C++ implementation.

API Design Principles:
1. Minimal boilerplate (no complex setup)
2. Numpy-compatible output (easy integration)
3. PyTorch-like iterator interface
4. Framework-agnostic (works with PyTorch, TF, JAX)


8.2 BASIC USAGE
------------------------------------------------------------------------------

EXAMPLE 1: Simple Data Loading

    import turboloader
    import numpy as np

    # Create pipeline with default config
    # Defaults: 4 workers, batch_size=32, shuffle=True
    pipeline = turboloader.UnifiedPipeline(
        tar_path="/path/to/dataset.tar"
    )

    # Iterate over batches
    for batch in pipeline:
        # batch is a numpy array: shape (batch_size, H, W, 3), dtype uint8
        print(f"Batch shape: {batch.shape}")

        # Process batch (your training code here)
        # ...

EXAMPLE 2: Custom Configuration

    import turboloader

    # Create custom config
    config = {
        'num_workers': 8,           # Use 8 worker threads
        'batch_size': 64,           # 64 samples per batch
        'shuffle': True,            # Shuffle between epochs
        'prefetch': True,           # Enable double-buffering
        'use_gpu_decode': True,     # Use GPU JPEG decode (if available)
        'queue_size': 256,          # Larger queue for smoother delivery
    }

    pipeline = turboloader.UnifiedPipeline(
        tar_path="/path/to/imagenet.tar",
        config=config
    )

    # Training loop
    for epoch in range(num_epochs):
        pipeline.reset()  # Reset for new epoch (re-shuffles)

        for batch_idx, batch in enumerate(pipeline):
            # Your training code
            loss = model(batch)
            loss.backward()
            optimizer.step()

EXAMPLE 3: With Transforms

    import turboloader

    # Define transform pipeline
    transforms = [
        turboloader.Resize(224, 224, interpolation='bilinear'),
        turboloader.RandomHorizontalFlip(p=0.5),
        turboloader.ColorJitter(brightness=0.2, contrast=0.2),
        turboloader.Normalize(mean=[0.485, 0.456, 0.406],
                             std=[0.229, 0.224, 0.225]),
        turboloader.ToTensor(layout='CHW'),  # Convert to PyTorch format
    ]

    config = {
        'num_workers': 4,
        'batch_size': 32,
        'transforms': transforms,  # Apply transforms in C++
    }

    pipeline = turboloader.UnifiedPipeline("/path/to/data.tar", config)

    for batch in pipeline:
        # batch is now: shape (batch_size, 3, 224, 224), dtype float32
        # Range: [0.0, 1.0], normalized with ImageNet mean/std
        pass


8.3 PYTORCH INTEGRATION
------------------------------------------------------------------------------

TurboLoader can be used as a drop-in replacement for PyTorch DataLoader.

EXAMPLE: PyTorch Training Loop

    import torch
    import torch.nn as nn
    import turboloader

    # Define model
    model = torchvision.models.resnet50(pretrained=True)
    model = model.cuda()

    # Define loss and optimizer
    criterion = nn.CrossEntropyLoss()
    optimizer = torch.optim.SGD(model.parameters(), lr=0.1)

    # Create TurboLoader pipeline
    config = {
        'num_workers': 8,
        'batch_size': 64,
        'shuffle': True,
        'use_gpu_decode': True,  # Offload JPEG decode to GPU
    }

    loader = turboloader.UnifiedPipeline("/data/imagenet.tar", config)

    # Training loop
    for epoch in range(num_epochs):
        model.train()
        loader.reset()

        for batch_idx, images in enumerate(loader):
            # Convert numpy to PyTorch tensor
            # TurboLoader outputs numpy, PyTorch expects torch.Tensor
            images = torch.from_numpy(images).cuda()

            # Assuming labels are in TAR filename or separate file
            # (label loading logic would go here)
            labels = get_labels(batch_idx)  # Your label extraction

            # Forward pass
            optimizer.zero_grad()
            outputs = model(images)
            loss = criterion(outputs, labels)

            # Backward pass
            loss.backward()
            optimizer.step()

            if batch_idx % 100 == 0:
                print(f"Epoch {epoch}, Batch {batch_idx}, Loss: {loss.item():.4f}")

ADVANCED: Zero-Copy GPU Transfer

    For maximum performance, you can use PyTorch's CUDA streams to overlap
    data transfer with computation:

    import torch
    import turboloader

    # Create CUDA stream for async transfers
    stream = torch.cuda.Stream()

    loader = turboloader.UnifiedPipeline("/data/train.tar", config)

    # Prefetch first batch
    images_cpu = next(iter(loader))
    images_gpu = torch.from_numpy(images_cpu).cuda(non_blocking=True)

    for next_images_cpu in loader:
        # While GPU computes on current batch, prefetch next batch
        with torch.cuda.stream(stream):
            next_images_gpu = torch.from_numpy(next_images_cpu).cuda(non_blocking=True)

        # Compute on current batch
        outputs = model(images_gpu)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()

        # Swap batches
        torch.cuda.current_stream().wait_stream(stream)
        images_gpu = next_images_gpu


8.4 TENSORFLOW INTEGRATION
------------------------------------------------------------------------------

TurboLoader integrates seamlessly with TensorFlow via tf.data.Dataset.

EXAMPLE: TensorFlow Training

    import tensorflow as tf
    import turboloader
    import numpy as np

    # Create TurboLoader pipeline
    config = {
        'num_workers': 4,
        'batch_size': 32,
        'shuffle': True,
    }

    loader = turboloader.UnifiedPipeline("/data/train.tar", config)

    # Wrap in tf.data.Dataset for TensorFlow compatibility
    def turboloader_generator():
        """Generator that yields batches from TurboLoader"""
        loader.reset()
        for batch in loader:
            # batch is numpy array: (batch_size, H, W, 3), uint8
            # Convert to float32 and normalize
            batch = batch.astype(np.float32) / 255.0
            yield batch

    # Create tf.data.Dataset
    dataset = tf.data.Dataset.from_generator(
        turboloader_generator,
        output_signature=tf.TensorSpec(shape=(None, 256, 256, 3), dtype=tf.float32)
    )

    # Define model
    model = tf.keras.applications.ResNet50(weights='imagenet')
    model.compile(
        optimizer='adam',
        loss='categorical_crossentropy',
        metrics=['accuracy']
    )

    # Train
    model.fit(dataset, epochs=10)


8.5 JAX INTEGRATION
------------------------------------------------------------------------------

For JAX users, TurboLoader provides numpy arrays that can be directly used
with jax.numpy functions.

EXAMPLE: JAX Training

    import jax
    import jax.numpy as jnp
    import turboloader

    # Create TurboLoader
    config = {'num_workers': 4, 'batch_size': 64}
    loader = turboloader.UnifiedPipeline("/data/train.tar", config)

    # Define JAX model
    def model(params, x):
        # Your JAX model here
        ...
        return logits

    # Training loop
    for epoch in range(num_epochs):
        loader.reset()

        for batch_np in loader:
            # Convert numpy to JAX array (zero-copy on CPU, copy on GPU)
            batch_jax = jnp.array(batch_np)

            # Compute loss and gradients
            loss, grads = jax.value_and_grad(loss_fn)(params, batch_jax)

            # Update parameters
            params = optimizer.update(grads, params)


================================================================================
SECTION 9: ADVANCED FEATURES & OPTIMIZATION
================================================================================

9.1 SMART BATCHING
------------------------------------------------------------------------------

Smart batching groups similar-sized samples together to reduce padding overhead.

Traditional batching pads all samples to max size in batch:
- Sample 1: 224x224 -> padded to 512x512 (wasted computation)
- Sample 2: 512x512 -> no padding
- Result: 50% wasted computation on sample 1

Smart batching groups by size:
- Batch 1: All 224x224 images
- Batch 2: All 512x512 images
- Result: Minimal padding, 15-25% throughput improvement

USAGE:

    config = {
        'num_workers': 4,
        'batch_size': 32,
        'smart_batching': True,      # Enable smart batching
        'size_tolerance': 0.1,       # Group samples within 10% size difference
    }

    pipeline = turboloader.UnifiedPipeline("/data/mixed_size.tar", config)

File: src/pipeline/smart_batching.hpp


9.2 GPU-ACCELERATED JPEG DECODING
------------------------------------------------------------------------------

nvJPEG provides 10x faster JPEG decoding by using NVIDIA GPUs.

REQUIREMENTS:
- NVIDIA GPU with CUDA Compute Capability 3.5+
- CUDA Toolkit 11.0+
- nvJPEG library (included in CUDA Toolkit)

USAGE:

    config = {
        'num_workers': 4,
        'batch_size': 64,
        'use_gpu_decode': True,  # Enable GPU decode
    }

    pipeline = turboloader.UnifiedPipeline("/data/imagenet.tar", config)

BEHAVIOR:
- If GPU available: nvJPEG decodes on GPU, ~10x speedup
- If GPU unavailable: Falls back to CPU (libjpeg-turbo), no error
- Per-worker GPU decoders eliminate contention

File: src/decode/nvjpeg_decoder.hpp


9.3 LINUX IO_URING ASYNC I/O
------------------------------------------------------------------------------

io_uring provides 2-3x faster disk I/O on NVMe SSDs via kernel-level async I/O.

REQUIREMENTS:
- Linux kernel 5.1+
- liburing library

USAGE:

    config = {
        'num_workers': 4,
        'batch_size': 32,
        'use_iouring': True,  # Enable io_uring
    }

    pipeline = turboloader.UnifiedPipeline("/data/train.tar", config)

BEHAVIOR:
- On Linux 5.1+ with liburing: Uses io_uring for async I/O
- Otherwise: Falls back to memory-mapped I/O (mmap)

File: src/io/io_uring_reader.hpp


9.4 DISTRIBUTED TRAINING (NEW in v1.7.1)
------------------------------------------------------------------------------

TurboLoader supports multi-node distributed training with deterministic sharding
integrated directly into UnifiedPipeline. Each rank receives a contiguous,
non-overlapping shard of the dataset.

IMPLEMENTATION DETAILS:

Distributed training is built into UnifiedPipeline via 5 configuration parameters:
- enable_distributed: Enable multi-node data loading (default: false)
- world_rank: Rank of this process, 0 to world_size-1 (default: 0)
- world_size: Total number of processes (default: 1)
- drop_last: Drop incomplete batches at end (default: false)
- distributed_seed: Seed for shuffling, same across all ranks (default: 42)

SHARDING STRATEGY:

Each rank gets a contiguous shard of samples:
- Rank 0: samples [0, N/world_size)
- Rank 1: samples [N/world_size, 2N/world_size)
- ...
- Last rank: gets all remaining samples if not using drop_last

Workers within each rank use round-robin scheduling on their shard.

USAGE WITH PYTORCH DDP:

    import torch
    import torch.distributed as dist
    import turboloader

    # Initialize process group
    dist.init_process_group(backend='nccl')

    rank = dist.get_rank()
    world_size = dist.get_world_size()

    # Create DataLoader with distributed config
    loader = turboloader.DataLoader(
        data_path="/data/imagenet.tar",
        batch_size=64,
        num_workers=4,
        shuffle=True,
        enable_distributed=True,      # Enable distributed training
        world_rank=rank,               # This process's rank
        world_size=world_size,         # Total number of processes
        drop_last=True,                # Drop incomplete batches
        distributed_seed=42            # Same seed across all ranks
    )

    # Each rank automatically gets its shard - no manual partitioning needed
    for batch in loader:
        # Your training code
        outputs = model(batch)
        loss = criterion(outputs, labels)
        loss.backward()
        optimizer.step()

USAGE WITH HOROVOD:

    import horovod.torch as hvd
    import turboloader

    hvd.init()

    loader = turboloader.DataLoader(
        data_path="/data/train.tar",
        batch_size=64,
        num_workers=4,
        shuffle=True,
        enable_distributed=True,
        world_rank=hvd.rank(),
        world_size=hvd.size(),
        drop_last=True,
        distributed_seed=42
    )

    for batch in loader:
        # Your training code
        ...

USAGE WITH DEEPSPEED:

    import deepspeed
    import turboloader

    # DeepSpeed initializes distributed training
    model_engine, optimizer, _, _ = deepspeed.initialize(
        model=model,
        model_parameters=model.parameters(),
        config=deepspeed_config
    )

    rank = dist.get_rank()
    world_size = dist.get_world_size()

    loader = turboloader.DataLoader(
        data_path="/data/train.tar",
        batch_size=64,
        num_workers=4,
        enable_distributed=True,
        world_rank=rank,
        world_size=world_size,
        drop_last=True
    )

    for batch in loader:
        loss = model_engine(batch)
        model_engine.backward(loss)
        model_engine.step()

KEY FEATURES:

1. Deterministic Sharding: Each rank gets the same shard across runs (same seed)
2. No Duplication: Ranks receive disjoint subsets of data
3. Load Balancing: Workers within each rank use round-robin for even distribution
4. Drop Last Support: Ensures all ranks have same batch count (required for DDP)
5. Integrated Design: No separate distributed pipeline class - built into UnifiedPipeline

File: src/pipeline/pipeline.hpp (lines 180-185 for config, lines 400-450 for implementation)


9.5 CUSTOM BINARY FORMAT - TBL v2 (NEW in v1.5.0)
------------------------------------------------------------------------------

TurboLoader defines a custom binary format (TBL v2) optimized for ML data loading
with LZ4 streaming compression for 40-60% space savings over TAR.

TBL v2 ADVANTAGES OVER TAR:
- 40-60% smaller file size (LZ4 compression)
- 4,875 img/s TAR→TBL conversion throughput
- O(1) random access via memory-mapped index
- Streaming writer with constant memory usage
- Data integrity validation (CRC32/CRC16 checksums)
- Cached image dimensions for fast filtered loading
- Rich metadata support (JSON, Protobuf, MessagePack)

TBL v2 FORMAT STRUCTURE:

    [Header: 64 bytes]
    - Magic: "TBLV2\0\0\0" (8 bytes)
    - Version: uint32_t
    - Num samples: uint64_t
    - Index offset: uint64_t
    - Compression: uint8_t (0=none, 1=LZ4)
    - Metadata format: uint8_t

    [Compressed Samples]
    - LZ4-compressed image data
    - Each sample stored with:
      * Compressed size (uint32_t)
      * Uncompressed size (uint32_t)
      * CRC32 checksum (uint32_t)
      * Compressed data bytes

    [Index Table - Memory Mapped]
    - Sample offsets (uint64_t each)
    - Image dimensions (uint32_t width, uint32_t height)
    - Format metadata

CONVERTING TAR TO TBL v2:

    # Python API
    import turboloader

    # Create TBL v2 writer
    writer = turboloader.TblWriterV2(
        output_path="/data/imagenet.tbl",
        compression=True,           # Enable LZ4 compression
        metadata_format=turboloader.MetadataType.JSON
    )

    # Read TAR and convert
    reader = turboloader.TarReader("/data/imagenet.tar")
    for sample in reader:
        writer.add_sample(
            data=sample.data,
            format=sample.format,      # JPEG, PNG, etc.
            metadata={"label": sample.label}
        )

    # Finalize (writes index table)
    writer.finalize()

    # Example script
    python examples/tbl_conversion.py /data/imagenet.tar

USING TBL v2 FILES:

    # TurboLoader auto-detects TBL v2 format
    loader = turboloader.DataLoader(
        data_path="/data/imagenet.tbl",  # .tbl extension
        batch_size=64,
        num_workers=4
    )

    # Or explicitly use TBL v2 reader
    reader = turboloader.TblReaderV2("/data/imagenet.tbl")

    # Memory-mapped access (zero-copy)
    for i in range(reader.num_samples()):
        sample = reader.get_sample(i)
        # sample.data contains decompressed image bytes
        # sample.width, sample.height cached for fast access

PERFORMANCE CHARACTERISTICS:

- Conversion: 4,875 img/s TAR→TBL throughput
- Reading: Same speed as TAR (memory-mapped, zero-copy)
- Compression: 40-60% space savings (LZ4 is very fast)
- Random access: O(1) via memory-mapped index
- Memory usage: Constant (streaming writer/reader)

COMPRESSION LEVELS:

TBL v2 uses LZ4 compression which provides:
- Compression ratio: 40-60% for typical images
- Compression speed: 500+ MB/s
- Decompression speed: 2+ GB/s (negligible overhead)

COMPARISON TO OTHER FORMATS:

    Format          Size    Random Access    Compression Speed
    ────────────────────────────────────────────────────────────
    TAR             100%    Sequential       N/A (uncompressed)
    TAR.GZ          35%     Sequential       50 MB/s (slow)
    WebDataset TAR  100%    Sequential       N/A
    TBL v1          88%     O(1)             N/A (uncompressed)
    TBL v2          45%     O(1)             500+ MB/s (fast!)

File: src/formats/tbl_format.hpp
      src/formats/tbl_reader_v2.hpp
      src/formats/tbl_writer_v2.hpp


================================================================================
SECTION 10: TROUBLESHOOTING & DEBUGGING
================================================================================

10.1 COMMON BUILD ERRORS
------------------------------------------------------------------------------

ERROR: "jpeg.h: No such file or directory"

    CAUSE: libjpeg-turbo not installed or not found by CMake

    FIX:
    # macOS
    brew install jpeg-turbo

    # Ubuntu
    sudo apt install libjpeg-turbo8-dev

    # Then rebuild
    cd build && cmake .. && make

ERROR: "undefined reference to `jpeg_CreateDecompress'"

    CAUSE: Linking against system libjpeg instead of libjpeg-turbo

    FIX: Explicitly specify libjpeg-turbo path

    cmake -DCMAKE_PREFIX_PATH=/opt/homebrew/opt/jpeg-turbo ..

ERROR: "Python.h: No such file or directory"

    CAUSE: Python development headers not installed

    FIX:
    # macOS
    brew install python@3.13

    # Ubuntu
    sudo apt install python3-dev

ERROR: Compilation fails with "unknown type name '__m256i'"

    CAUSE: AVX2 intrinsics not available (old CPU or disabled)

    FIX: TurboLoader should auto-detect and fall back to NEON/scalar.
          If not, force scalar mode:

    cmake -DFORCE_SCALAR_SIMD=ON ..


10.2 COMMON RUNTIME ERRORS
------------------------------------------------------------------------------

ERROR: "Failed to open TAR file: Permission denied"

    CAUSE: Insufficient permissions to read TAR file

    FIX: Check file permissions
    ls -l /path/to/file.tar

    If needed:
    chmod 644 /path/to/file.tar

ERROR: "Corrupted TAR header at offset XXXX"

    CAUSE: TAR file is malformed or corrupted

    FIX: Verify TAR file integrity
    tar -tzf /path/to/file.tar > /dev/null

    If corrupted, re-download or regenerate TAR file.

ERROR: "Worker thread crashed: SIGSEGV"

    CAUSE: Memory corruption (usually in SIMD code or decoder)

    DEBUG:
    1. Run with AddressSanitizer:
       cmake -DCMAKE_CXX_FLAGS="-fsanitize=address -g" ..
       make
       ./tests/test_unified_pipeline

    2. Check output for memory errors

    3. Report bug with stack trace

ERROR: "No samples loaded (empty TAR file?)"

    CAUSE: TAR file contains no valid JPEG files

    DEBUG:
    # List TAR contents
    tar -tvf /path/to/file.tar | head -20

    # Verify files are JPEG
    tar -xOf /path/to/file.tar 0001.jpg | file -
    # Should output: JPEG image data


10.3 PERFORMANCE DEBUGGING
------------------------------------------------------------------------------

PROBLEM: Throughput lower than expected

    DIAGNOSIS STEPS:

    1. Check worker count:
       # Too few workers = underutilized CPU
       # Too many workers = memory bandwidth saturation
       # Optimal: num_workers = num_physical_cores

    2. Check batch size:
       # Too small = overhead dominates
       # Too large = memory pressure
       # Optimal: 32-64 for images

    3. Profile CPU usage:
       # While benchmark running, in another terminal:
       top -pid $(pgrep -f turboloader)

       # Should see 100% CPU utilization across all cores
       # If not, increase num_workers

    4. Profile I/O:
       iostat -x 1

       # Check disk utilization (%util column)
       # Should be 80-100% if I/O bound
       # If low, increase prefetching

    5. Check SIMD usage:
       # Verify AVX2/NEON instructions used
       objdump -d _turboloader.so | grep -E "vpmov|vmul|vadd"

       # Should see many SIMD instructions
       # If not, SIMD disabled (performance loss)

PROBLEM: High memory usage

    DIAGNOSIS:

    1. Reduce queue_size:
       config['queue_size'] = 64  # Default: 128

    2. Reduce num_workers:
       # Each worker holds decoded images in memory

    3. Use smart_batching:
       # Reduces padding, saves memory

PROBLEM: Slow first batch (long latency)

    DIAGNOSIS:

    This is normal due to:
    - Thread startup overhead
    - Cold caches (CPU and disk)
    - mmap page faults

    FIX: Use warm-up phase (discard first batch from timing)

    # Warm-up
    _ = pipeline.next_batch()

    # Now measure
    start = time.time()
    for batch in pipeline:
        # Your code
        ...


10.4 ENABLING DEBUG LOGGING
------------------------------------------------------------------------------

TurboLoader has built-in debug logging (disabled by default for performance).

ENABLE DEBUG LOGS:

    # C++ logs (compile time)
    cmake -DDEBUG_LOGGING=ON ..
    make

    # Python logs (runtime)
    import turboloader
    import logging

    logging.basicConfig(level=logging.DEBUG)
    turboloader.set_log_level("DEBUG")

    pipeline = turboloader.UnifiedPipeline("/data/train.tar")

DEBUG OUTPUT:

    [TurboLoader] [DEBUG] Opened TAR file: /data/train.tar (size: 1.2 GB)
    [TurboLoader] [DEBUG] Found 5000 entries in TAR
    [TurboLoader] [DEBUG] Starting 4 worker threads
    [TurboLoader] [DEBUG] Worker 0 started (CPU 0)
    [TurboLoader] [DEBUG] Worker 1 started (CPU 1)
    [TurboLoader] [DEBUG] Worker 2 started (CPU 2)
    [TurboLoader] [DEBUG] Worker 3 started (CPU 3)
    [TurboLoader] [DEBUG] All workers ready
    [TurboLoader] [DEBUG] Decoded sample 0 (256x256 JPEG) in 1.2ms
    [TurboLoader] [DEBUG] Decoded sample 1 (512x512 JPEG) in 3.4ms
    ...


================================================================================
CONCLUSION
================================================================================

This document provided a comprehensive, senior-engineer-level deep dive into
the TurboLoader codebase. You should now understand:

1. The architectural principles and design patterns
2. How each major component works internally
3. How to build, test, and benchmark the system
4. How to integrate TurboLoader into ML training pipelines
5. Advanced optimization techniques
6. How to debug and troubleshoot issues

For further information:
- README.md: User-facing documentation
- ARCHITECTURE.md: High-level architecture overview
- CHANGELOG.md: Version history and features
- Source code comments: Inline documentation

The TurboLoader project is actively maintained. For bugs, feature requests,
or questions, please file an issue at:
https://github.com/ALJainProjects/TurboLoader/issues

Happy fast data loading!

================================================================================
END OF DOCUMENTATION
================================================================================