Chunking for High-Quality Retrieval

Why Chunking Matters for Retrieval

Chunking improves retrieval quality by letting the system match the passage that answers a query instead of returning an entire document. That matters because retrieval quality is often limited by context quality. Even the best language model cannot produce accurate answers if retrieval returns whole documents, irrelevant sections, duplicate information, or too much context.

The goal is not to split documents into smaller pieces; it is to deliver only the passages that deserve space in the model's context window.

Chunking also affects how much context is preserved around a passage, how many embeddings must be stored, and how much irrelevant information reaches the model. Smaller chunks improve retrieval precision and reduce irrelevant context, but they increase the number of embeddings and can lose surrounding context. Larger chunks preserve more context, but they make it easier to retrieve irrelevant text and waste tokens in the LLM context window.

This is where most retrieval systems begin making architectural tradeoffs. Every RAG system eventually runs into the same architectural decision: should each chunk be stored as its own document, or should multiple chunks remain grouped within the parent document? That choice has real consequences for retrieval quality, storage efficiency, ranking, and scalability.

The Hard Part is Ranking, Not Storage

Every RAG system has to store chunks somewhere. A few years ago the hard question was where those chunks lived. Should each chunk become its own document, or stay inside its parent?

That question is mostly settled. Most vector databases now store many vectors inside one record, so multi-vector storage no longer separates one system from another.

The hard part moved to query time. Which passages actually earn space in the model's context? That is a ranking problem, and it is where architectures still diverge.

Two storage models still shape how systems answer that question:

Keeping chunks in the parent is the better foundation. On its own, though, it returns every chunk. The missing piece is ranking that reaches inside the document and picks the few passages that matter.

How Chunking Works in Vespa

Vespa keeps chunks inside the parent document and ranks down to the chunk. Documents remain whole. Chunks stay inside the document. Retrieval finds the best documents. Layered ranking then scores the chunks inside those documents and returns only the most relevant passages.

The result is passage-level retrieval with document-level context, shared metadata, efficient storage, and efficient network transfer.

How it works:

• Documents become chunks: During indexing, Vespa splits documents into passages. Teams can use fixed-length chunking, sentence chunking, custom chunkers, or pre-chunked input. Each chunk gets its own embedding and stays part of its parent document.
• Chunks stay with the document: Unlike systems that create millions of standalone chunk documents, Vespa stores chunks inside the original document. Each document holds its original content, chunk text, chunk embeddings, and metadata, with metadata stored once rather than duplicated across every chunk.
• Retrieval runs across chunks in one query: At query time Vespa matches on several signals at once: lexical matching over the chunk text, vector search over the chunk embeddings, and structured filters over metadata. Any matching chunk makes its parent document a candidate. Retrieval works at passage level while keeping document context.
• Layered ranking selects the best passages: This is the step that resolves the chunking tradeoff. Retrieval finds the best documents. Layered ranking then scores every chunk inside them and returns only the highest-scoring passages. Instead of dozens of passages from one document, Vespa returns the few that earned their place. That cuts network traffic, context size, and irrelevant passages while keeping document-level relevance.

Chunking vs. layered ranking

Chunking alone does not determine retrieval quality. You can split documents into chunks in many different ways, but if the system still returns every chunk from a matched document, the model’s context window will fill with irrelevant text. Vespa uses layered ranking to avoid that outcome.

Chunking creates the retrieval units. It determines how documents are split into passages, how those passages are embedded, and how they are stored.

Layered ranking decides which of those passages are actually returned. It scores the chunks inside each winning document and selects only the ones most relevant to the query.

Chunk-level vs. Token-Level Multi-Vector

You have already seen chunk-level multi-vector at work. When Vespa embeds each chunk and stores the vectors inside the parent document, that array of embeddings is the chunk-level representation. It is the standard approach, and it is what most people mean when they call a system multi-vector.

The word gets used a second way, and the two are easy to confuse. Token-level multi-vector comes from late-interaction models like ColBERT and ColPali. Instead of one vector per chunk, these models produce one vector per token, then score a passage with MaxSim, matching every query token against every passage token.

The two work at different granularities and solve different problems. Chunking decides how a document is divided into passages. Token-level representation decides how finely each passage is matched. They compose rather than compete. You chunk a document into passages, retrieve the best passages with chunk-level vectors, then, where precision matters most, score those passages token by token to catch fine-grained matches a single pooled vector would blur.

Vespa supports both in the same platform and the same query. Store chunk embeddings, token-level tensors, or both, and express late-interaction scoring as a ranking function. You can start with chunk-level retrieval and layer token-level precision onto the passages that survive ranking, without adding a second system or rebuilding your schema.

Configuring Chunking in Vespa

Treating each chunk as an independent document makes retrieval straightforward, simplifies vector search, and ensures that only relevant chunks are returned. But it also duplicates metadata across every chunk, dramatically increases document count, breaks the relationship between chunks and their parent document, and makes it harder to use document-level signals during ranking.

Keeping many chunks inside a single parent document solves those issues by preserving document context, storing metadata once, and reducing the number of documents that need to be indexed. The downside is that once a document matches, every chunk in that document may be returned, which can fill the model’s context window with irrelevant passages and waste bandwidth on large documents.

• Chunking method and size: Choose how documents are split into passages using built-in fixed-length or sentence chunkers, a custom chunker component, or pre-split input if you already chunk upstream.
• Embeddings: Embed each chunk with a built-in or custom embedder, and choose how those embeddings are stored. Quantization options like int8 or binary help control the memory footprint at scale.
• Cross-chunk proximity: Use element-gap sets to control whether text matches spanning adjacent chunks contribute to text relevance. This can help preserve relevance across chunk boundaries.
• Chunk selection: Use layered ranking and select-elements-by to control how many chunks to return and how those chunks are scored.
• Hybrid signals: Combine lexical matching over chunk text with nearest-neighbor search over chunk embeddings, sparse retrieval, and structured filters in a single query.

Chunking is Only one Part of Retrieval

Chunking is the foundation of modern retrieval, but retrieval quality depends on more than how documents are split. Once chunks exist, Vespa combines chunk-aware storage, hybrid retrieval, document-aware ranking, and layered ranking to ensure that AI applications receive the most relevant passages while preserving full document context.

That is the key difference between chunking as a preprocessing step and chunking as part of a retrieval architecture.

Most systems make you choose between passage-level precision and document-level context. Vespa removes that compromise by combining both in one system.