semantic-chunking

A fast, lightweight and easy-to-use Python library for splitting text into semantically meaningful chunks.

Fully neural approach for text chunking

🍱 Semantically create chunks from large document for passing to LLM workflows

RAG boilerplate with semantic/propositional chunking, hybrid search (BM25 + dense), LLM reranking, query enhancement agents, CrewAI orchestration, Qdrant vector search, Redis/Mongo sessioning, Celery ingestion pipeline, Gradio UI, and an evaluation suite (Hit-Rate, MRR, hybrid configs).

Turn speech into semantic paragraphs in real time, on a CPU (no GPU) — single-pass streaming chunker + Moonshine STT + a live mic demo. Plus a transcribe/chunk/summarize API.

Rust CLI implementing the Recursive Language Model (RLM) pattern for Claude Code. Process documents 100x larger than context windows through intelligent chunking, SQLite persistence, and recursive sub-LLM orchestration.

Search knowledge by what documents mean and how they look — not one or the other.

🍶 llm-distillery ⇢ use LLMs to run map-reduce summarization tasks on large documents until a target token size is met.

Token-aware, LangChain-compatible semantic chunker with PDF, markdown, and layout support

Semantic Chunking is a Python library for segmenting text into meaningful chunks using embeddings from Sentence Transformers.

Advanced semantic text chunking with custom structural markers, whole-text coherence preservation, and flexible token management. Features async processing, LangChain integration, and dynamic drift detection. Ideal for RAG systems, augmented text processing, and domain-specific document analysis.

A research framework to evaluate how document parsing quality determines downstream RAG performance.

Advanced local-first RAG system powered by Ollama and LangGraph. Optimized for high-performance sLLM orchestration featuring adaptive intent routing, semantic chunking, intelligent hybrid search (FAISS + BM25), and real-time thought streaming. Includes integrated PDF analysis and secure vector caching.

A practical guide to 6 document chunking strategies for RAG and LLM applications — Document, Fixed-Size, Recursive, Sentence, Semantic, and Agentic chunking with working code and plain-English explanations.

Local-first desktop application for converting documents into AI training datasets. Generate instruction, conversation, and fine-tuning datasets from PDFs, DOCX, Markdown, HTML, and text using Ollama or cloud LLMs.

AI-native data annotation pipeline using Dynamic In-Context Learning (ICL) routing. Leverages RAG to retrieve semantically relevant few-shot examples for long-context document chunking and targeted LLM prompting.

Sementic chunking algorithm in (mostly) Go

High-performance, zero-dependency Markdown parser and semantic chunking tree for RAG and LLM agent contexts.

A high-performance Retrieval-Augmented Generation pipeline for technical Q&A workloads. Combines hybrid retrieval (dense + BM25), query expansion, Reciprocal Rank Fusion (RRF), and cross-encoder re-ranking to improve retrieval precision and answer grounding. Evaluated with Ragas, showing measurable gains in context recall and faithfulness.

Cutting-edge semantic text processing system that uses hierarchical clustering and advanced language models to automatically organize and summarize large volumes of text.