Does chunking really matter more than the model or the vector database?

In most projects we see, yes. The model can only answer from the context retrieval gives it, and the database just stores and searches vectors you created during chunking. If chunking splits a fact in half or strips a heading, retrieval hands the model the wrong material and even the best model fails. Fixing chunking usually moves answer quality more than swapping models or databases.

What's the difference between semantic and structure-aware chunking?

Structure-aware chunking cuts on the document's own structure (headings, sections, tables) and is cheap and reliable for most real documents. Semantic chunking embeds sentences and cuts where the meaning shifts, producing single-topic chunks even in messy text, but it costs extra embedding calls and needs tuning. Start structure-aware, and move to semantic only when topic-bleed is your measured failure mode.

How do I handle tables and code in RAG chunking?

Keep them whole. Naive splitters slice a table away from its header row or cut a function mid-body, which destroys their meaning. Detect tables and code blocks during chunking and treat each as a single unit, even if it bends your size limit. For wide tables, it also helps to repeat the header context or add a short text summary alongside the raw rows so the chunk embeds on what it actually contains.

Why use chunk overlap, and how much is enough?

Overlap repeats a bit of text between adjacent chunks so a fact sitting on a boundary isn't lost from both. Without it, a definition that spans the end of one chunk and the start of the next can fail to retrieve cleanly in either. Around 10 to 15 percent of your chunk size is a sensible default. Too much overlap inflates your index and returns near-duplicate results, so increase it only if your evaluation shows boundary facts are being missed.

All guides

RAG8 min read

By Leeor MeirovitzLast updated: 14 July 2026

Chunking strategies that make or break your RAG system

Diagram of a document being split into overlapping text chunks before embedding

TL;DR

Chunking has more impact on RAG answer quality than your vector database and often more than your model. If retrieval hands the model the wrong context, no amount of prompt engineering saves you.
There's no universal chunk size. Start with structure-aware splitting at around 500 to 800 tokens with light overlap, then measure on real questions before you touch anything else.
Tables, code, and headings break naive splitters. Preserve structure and attach metadata, because the chunk that retrieves well is rarely the one that reads well in isolation.

Why chunking decides your RAG quality before the model ever runs

Here's the uncomfortable truth most teams learn the hard way: your RAG system is only as good as the chunk it retrieves. You can swap in a smarter model, pay for a faster vector database, and rewrite your prompt five times, but if the retrieval step hands the model a fragment that's missing the answer, the model will either hallucinate or tell you it doesn't know. Both are failures the user blames on the AI.

Chunking is the step where you take a document (a PDF, a help article, a contract, a wiki page) and split it into smaller pieces before you embed them. Each piece becomes a vector. At query time, you retrieve the closest pieces and stuff them into the prompt. That's it. The whole pipeline rests on one assumption: the chunk you retrieve actually contains the answer, in enough context for the model to use it.

The pattern we see across client projects is that nearly every disappointing RAG demo traces back to chunking, not the model. A clause gets split across two chunks so neither retrieves cleanly. A table loses its header row and becomes a grid of meaningless numbers. A heading that gave a paragraph its meaning sits three chunks away. The model never had a chance, because retrieval never gave it the right material.

Too small, and chunks lose the context that makes them answerable on their own.
Too large, and you dilute the embedding so the relevant sentence gets drowned out by surrounding noise.
Split in the wrong place, and a single fact gets cut in half across two vectors that each retrieve poorly.
Strip the structure, and the model loses the headings, labels, and order that told it what the text meant.

Fixed-size chunking: the default everyone starts with (and why it's a trap)

Fixed-size chunking splits text every N tokens or characters, regardless of what the text says. Every tutorial starts here because it's trivial to implement: count to 512 tokens, cut, repeat. It works well enough for a quick prototype, and that's exactly why it's dangerous. It demos fine on clean prose, then quietly falls apart on real documents.

The problem is that documents don't think in token counts. A 512-token cut lands in the middle of a sentence, halfway through a list, or right between a question and its answer. The embedding for that broken chunk represents a thought that was never finished, so it retrieves for the wrong queries or doesn't retrieve at all. You won't notice in testing because your test questions probably map to the chunks that happened to split cleanly.

We're not telling you to never use fixed-size chunking. For uniform, flat text (transcripts, chat logs, plain notes) it's perfectly reasonable and fast. Just don't reach for it on structured documents and assume you're done. It's a starting line, not a finish line.

Pro: dead simple, fast, predictable token budgets, no parsing required.
Con: cuts mid-sentence and mid-idea, which produces chunks that embed poorly.
Con: ignores headings, lists, and tables entirely, so structure is lost.
Best fit: flat, uniform text where natural boundaries don't carry much meaning.

Sentence and paragraph chunking: respecting where ideas actually end

The first real upgrade is to split on natural language boundaries instead of arbitrary counts. Sentence-aware and paragraph-aware chunking use the document's own punctuation and line breaks to decide where to cut, then group those units up to a target size. A paragraph is usually one coherent idea, which is exactly what you want a chunk to be.

In practice you don't chunk one sentence at a time, because a lone sentence rarely carries enough context to answer anything. Instead you pack sentences or paragraphs together until you hit your size budget, then start a new chunk at the next clean boundary. This gets you chunks that read like complete thoughts, which embed far better than mid-sentence fragments.

This approach handles the vast majority of prose-heavy content (knowledge bases, blog posts, documentation, policy text) and it's cheap to run. If you're moving off fixed-size and you don't yet know your documents deeply, this is the safe, sensible default to land on while you measure.

Split on sentence boundaries, then group sentences up to your target token count.
Prefer paragraph breaks as chunk edges whenever the paragraph fits the budget.
Never let a chunk end mid-sentence; carry the trailing sentence into the next chunk.
Watch out for documents with poor punctuation (OCR output, scraped HTML) where sentence detection gets noisy.

Semantic chunking: splitting where the meaning shifts

Semantic chunking takes the next step: instead of trusting punctuation, it groups sentences by meaning. You embed sentences (or small windows of them), measure how similar each is to the next, and cut where the similarity drops, which signals the topic just changed. The result is chunks that each cover one subject, even when the original document rambled across several within a single paragraph.

When it works, it's the cleanest mapping between a chunk and a single idea. A long support article that drifts from billing to shipping to returns gets cut at exactly those topic seams, so each chunk retrieves only for queries about its own subject. That precision can noticeably lift answer quality on messy, mixed-topic content.

The honest trade-off is cost and complexity. Semantic chunking runs embeddings during the chunking step, not just at query time, so it's slower and pricier to build your index. It also has knobs (the similarity threshold) that need tuning per corpus. The pattern we see is that teams reach for it too early. Get structure-aware and paragraph chunking working first, measure, and only move to semantic chunking when you've proven your remaining errors are topic-bleed problems it can actually fix.

Embed sentences, compare neighbors, and cut where semantic similarity drops sharply.
Produces single-topic chunks even when the source mixes topics within a paragraph.
Costs extra embedding calls and adds a threshold you'll need to tune per document type.
Worth it when topic-bleed is your measured failure mode, not as a reflex first choice.

Structure-aware chunking: the one that wins for most real documents

If we had to pick a single default for the documents companies actually feed into RAG, it's structure-aware chunking. The idea is simple: use the document's own structure (Markdown headings, HTML tags, PDF sections, slide titles) as the primary cut points, then fall back to paragraph and sentence splitting inside each section. You're letting the author's organization do the heavy lifting, because they already grouped related ideas under headings for a human reader.

This matters because structure carries meaning that flat text throws away. A section titled 'Refund eligibility' tells you what every paragraph beneath it is about. If you split by structure, that heading stays attached to its content and you can even prepend it to each chunk so the embedding knows the subject. Split by raw token count and that same heading gets orphaned, and the paragraph below it becomes ambiguous.

Structure-aware splitting also gives you natural homes for the hard cases. A table lives under its own heading and can be kept whole. A code block stays intact instead of being sliced mid-function. Because you respect the document's skeleton, you get chunks that are both retrievable and readable, which is the combination that actually produces good answers.

Use headings, sections, and tags as primary boundaries; paragraph-split within each section.
Prepend the section heading (or full heading path) to each chunk so context travels with it.
Keep tables and code blocks whole instead of letting a size limit cut through them.
Falls back gracefully: a section that's too big just gets paragraph-chunked underneath its heading.

Chunk size, overlap, and the metadata that quietly saves you

Two numbers shape every chunking strategy: size and overlap. Size is how big each chunk is; overlap is how much text you repeat from the end of one chunk at the start of the next. Overlap exists to stop a fact that sits on a boundary from being lost. If a definition spans the last line of chunk A and the first line of chunk B, a small overlap means at least one chunk holds the whole thing.

For size, a useful starting band is 500 to 800 tokens for most prose, with 10 to 15 percent overlap. Smaller chunks (around 200 to 300 tokens) give sharper, more precise retrieval and suit fact-lookup use cases like FAQs. Larger chunks (1,000-plus tokens) give the model more surrounding context and suit reasoning over longer passages, at the cost of more noise per retrieval. There's no universal answer; the right size depends on your documents and the questions people ask.

The piece teams underrate most is metadata. Every chunk should carry more than its text: the source document, the section heading, the page or URL, a date, and any tags that matter (product, region, version). This pays off twice. It lets you filter before you search (only this product's docs, only the current version), which sharpens retrieval cheaply, and it lets you cite sources in the answer, which is what turns a demo into something a business will trust.

Start at 500 to 800 tokens with 10 to 15 percent overlap, then adjust based on measured results.
Use smaller chunks for precise fact lookup, larger chunks for reasoning over context.
Add overlap to protect facts that land on a chunk boundary, but don't overdo it (it bloats your index).
Attach metadata (source, heading, date, version, tags) to every chunk for filtering and citations.

A practical playbook: start simple, measure, then iterate

Here's the framework we actually use, and it's deliberately boring because boring is what ships. Don't start by choosing the cleverest chunking method. Start by building an evaluation set, then let the numbers tell you where to spend effort. Most teams do this backwards: they spend a week perfecting semantic chunking and zero time measuring whether it helped.

Step one, write 30 to 50 real questions your users would ask, with the correct source passage noted for each. Step two, ship the simplest reasonable chunker (structure-aware with 600-token chunks and 10 percent overlap) and measure retrieval: for each question, did the right passage land in the top results? That single metric, retrieval hit rate, tells you more than any vibe-check of the final answers. Step three, look at the misses and find the pattern. Tables not retrieving? Fix table handling. Answers split across chunks? Increase overlap or size. Topic-bleed? Now semantic chunking earns its place.

The reason this works is that it stops you optimizing blind. Each change gets measured against the same question set, so you keep what helps and discard what doesn't. The pattern we see is that one or two targeted fixes (usually around tables, headings, or overlap) close most of the gap, and the exotic techniques people obsess over move the needle far less than disciplined measurement does.

Build a labeled question set first (30 to 50 real questions with their correct source passage).
Measure retrieval hit rate, not just final-answer quality, so you isolate the chunking problem.
Ship the simplest structure-aware chunker, then fix only the failure patterns your data shows.
Re-run the same eval after every change so you keep what helps and drop what doesn't.

Want this built for your business?

We map the highest-leverage place to start and ship a first live system within two weeks.

Book a strategy call

Common questions

There isn't a single best size. A solid starting point is 500 to 800 tokens with 10 to 15 percent overlap for general prose. Use smaller chunks (200 to 300 tokens) when you need precise fact lookup like FAQs, and larger chunks (1,000-plus) when the model needs more surrounding context to reason. Pick a starting size, measure retrieval hit rate on real questions, and adjust from there.

Ask AI about X18 Global

“What does X18 Global (x18global.com) do for enterprise AI and automation - and can you summarise their guide "Chunking strategies that make or break your RAG system"?”

ChatGPT Claude Perplexity Google AI Grok

Keep reading

AI Agents