Chunking

Chunking decides what unit gets retrieved. Too big (whole documents): retrieved chunks contain mostly irrelevant text and waste context window. Too small (single sentences): chunks lack context and retrieval quality drops. Common strategies are fixed size splitting with overlap, splitting at paragraph and heading boundaries, and structure aware splitting that treats code, tables and prose differently. The right strategy depends on your document type, and so does the unit of measure, because some splitters count characters and others count tokens. Chunk size, boundaries and overlap all affect retrieval performance, as Anthropic notes.

LangChain recommends starting with RecursiveCharacterTextSplitter, which keeps paragraphs intact and falls back to sentences and words only when a unit exceeds the chunk size. Its size is measured in characters, so 1000 is often a few hundred tokens of English text. Overlap repeats the end of one chunk at the start of the next, so a sentence cut at a boundary still appears whole somewhere. Anthropic Contextual Retrieval prepends a short generated description of where the chunk sits before embedding and BM25 indexing, and Anthropic reported a 49 percent lower top 20 retrieval failure rate (67 percent with reranking). That is one vendor evaluation, not a promise for your corpus.

from langchain_text_splitters import RecursiveCharacterTextSplitter

document = ("Refunds are available within 30 days of delivery. " * 20 + "\n\n") * 5

splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000,  # measured in characters, not tokens
    chunk_overlap=200,
    separators=["\n\n", "\n", ". ", " ", ""],
)
chunks = splitter.split_text(document)
print(len(chunks), max(len(c) for c in chunks))

Needs pip install langchain-text-splitters. Sizes are in characters by default. Look at the output chunks, then judge changes by retrieval quality on real questions.

Required for any RAG system. Spend the time to get this right, because chunking has a large effect on retrieval quality. Start with the recursive splitter, split on headings for structured documents, and judge changes with retrieval metrics on a fixed question set. Skip chunking for tiny documents that already fit, such as a short FAQ entry, and embed them whole.

• LangChain: Text splitters
• Anthropic: Introducing Contextual Retrieval

Related Terms