Release 1.0.0

Normalized Semantic Chunker - Docker Image Published

This release publishes a single Docker image with CUDA support that works on both GPU and CPU machines.

Pull the image

docker pull ghcr.io/smart-models/normalized-semantic-chunker:1.0.0
# or
docker pull ghcr.io/smart-models/normalized-semantic-chunker:latest

Run with GPU (recommended)

docker run --gpus all -p 8000:8000 ghcr.io/smart-models/normalized-semantic-chunker:1.0.0

Run on CPU (fallback)

docker run -p 8000:8000 ghcr.io/smart-models/normalized-semantic-chunker:1.0.0

Docker Compose

cd docker

# With GPU
docker compose --profile gpu up -d

# CPU only
docker compose --profile cpu up -d

Verify installation

curl http://localhost:8000/

For more information, see the README.

Full Changelog: v0.7.2...v1.0.0

Release v0.7.2

🚀 Normalized Semantic Chunker - Docker Images Published

This release includes Docker images for both CPU and GPU variants of the Normalized Semantic Chunker:

CPU Image

docker pull ghcr.io/smart-models/Normalized-Semantic-Chunker:v0.7.2-cpu
docker pull ghcr.io/smart-models/Normalized-Semantic-Chunker:latest-cpu

GPU Image (CUDA 12.1)

docker pull ghcr.io/smart-models/Normalized-Semantic-Chunker:v0.7.2-gpu
docker pull ghcr.io/smart-models/Normalized-Semantic-Chunker:latest-gpu

Docker Compose

# CPU deployment
cd docker
docker compose --profile cpu up -d

# GPU deployment (requires NVIDIA GPU and drivers)
cd docker
docker compose --profile gpu up -d

Quick Start

# Run CPU version
docker run -p 8080:8080 ghcr.io/smart-models/Normalized-Semantic-Chunker:v0.7.2-cpu

# Run GPU version (requires nvidia-docker)
docker run --gpus all -p 8080:8080 ghcr.io/smart-models/Normalized-Semantic-Chunker:v0.7.2-gpu

Features

• Intelligent text normalization and semantic chunking
• Support for multiple languages and document formats
• GPU acceleration for improved performance
• RESTful API with comprehensive documentation
• Docker containerization for easy deployment

For more information, see the README.

v0.7.1

Normalized Semantic Chunker v0.7.1

✨ New Features
JSON File Support: Processing of JSON files with format {"chunks": [{"text": "..."}, ...]}
Dynamic Memory Management: Smart worker allocation based on system resources
Verbosity Controls: Configurable logging for debugging and production
Configurable Parameters: Control via environment variables
File Validation: Input size and format checks

🚀 Performance Improvements
Batch Processing: Prevents OOM errors for large documents (>20K sentences)
Model Caching: Cache system with automatic expiration (1h default)
Adaptive Workers: Scalability based on document size
Memory Cleanup: Optimized GPU memory management
Adaptive Step Size: Optimization based on document size

🛡️ Robustness
Error Handling: Smart fallback mechanisms for tiktoken errors
Automatic Recovery: Recovery mechanisms from processing failures
Improved Logging: Detailed and configurable logging system
Input Validation: Comprehensive checks on file size, format, and content

📈 Improvement Metrics
⬇️ Memory Usage: -30–40% for large documents
⚡ Speed: +15–25% for documents >10K sentences
🛠️ Reliability: +95% reduction in processing errors
🔧 Configurability: Full control via environment variables

Changelog

• Torch library to 2.6.0
• Docker compose fix

v0.7.0

Normalized Semantic Chunker v0.7.0

✨ New Features
JSON File Support: Processing of JSON files with format {"chunks": [{"text": "..."}, ...]}
Dynamic Memory Management: Smart worker allocation based on system resources
Verbosity Controls: Configurable logging for debugging and production
Configurable Parameters: Control via environment variables
File Validation: Input size and format checks

🚀 Performance Improvements
Batch Processing: Prevents OOM errors for large documents (>20K sentences)
Model Caching: Cache system with automatic expiration (1h default)
Adaptive Workers: Scalability based on document size
Memory Cleanup: Optimized GPU memory management
Adaptive Step Size: Optimization based on document size

🛡️ Robustness
Error Handling: Smart fallback mechanisms for tiktoken errors
Automatic Recovery: Recovery mechanisms from processing failures
Improved Logging: Detailed and configurable logging system
Input Validation: Comprehensive checks on file size, format, and content

📈 Improvement Metrics
⬇️ Memory Usage: -30–40% for large documents
⚡ Speed: +15–25% for documents >10K sentences
🛠️ Reliability: +95% reduction in processing errors
🔧 Configurability: Full control via environment variables

v0.5.0

Normalized Semantic Chunker v0.5.0

A cutting-edge tool that processes text documents and splits them into semantically coherent segments while ensuring optimal chunk size for downstream NLP tasks. Ideal for retrieval-augmented generation (RAG) and other token-sensitive applications.

Key Features

• Adaptive semantic chunking with precise token limit control
• Parallel multi-percentile optimization and GPU acceleration
• Intelligent handling of small and oversized chunks
• REST API with FastAPI

Prerequisites

• Docker and Docker Compose (for Docker deployment)
• NVIDIA GPU with CUDA support (recommended)
• NVIDIA Container Toolkit (for GPU passthrough in Docker)
• Python 3.10-3.12 (Python 3.11 recommended, Python 3.13 not supported due to dependency compatibility issues)