Fully neural approach for text chunking

A fast, lightweight and easy-to-use Python library for splitting text into semantically meaningful chunks.

🍱 semantic-chunking ⇢ semantically create chunks from large document for passing to LLM workflows

🍶 llm-distillery ⇢ use LLMs to run map-reduce summarization tasks on large documents until a target token size is met.

Semantic Chunking is a Python library for segmenting text into meaningful chunks using embeddings from Sentence Transformers.

🤖 Automated Q&A Dataset Generation Pipeline powered by LLMs. Multi-stage pipeline that searches, filters, extracts and transforms web content into high-quality question-answer datasets for LLM training. Supports multiple LLM providers (Groq, Mistral, Ollama) and search engines.

Retrieval-Augmented Generation (RAG) Fundamentals and Semantic Chunking

Token-aware, LangChain-compatible semantic chunker with PDF, markdown, and layout support