RAG architecture patterns: eight designs and when to use each

Why the pattern matters more than the model

When a RAG system gives bad answers, the instinct is to reach for a bigger model. It is almost always the wrong move. The quality of a retrieval-augmented system is set by how it retrieves, not by the model that writes the final sentence. Pick the wrong architecture and no model will save you; pick the right one and a modest model will shine.

The good news is that RAG architectures form a clear ladder. Each rung adds capability and cost. The skill is not knowing the fanciest pattern; it is knowing the simplest one that solves your actual problem, and being able to tell when you have outgrown it.

1. Naive RAG (start here)

Naive RAG is the textbook version: chunk your documents, embed them, store the vectors, and at query time retrieve the top matches and hand them to the model. It is unglamorous and it is the right place to start for almost everyone.

Start naive because it tells you the truth about your data and your problem before you spend on complexity:

• It is fast to build and cheap to run.
• It exposes your real failure modes, which tell you what to fix next.
• For clean, well-structured content and straightforward questions, it is often enough.

2. Re-ranked RAG (the highest-value upgrade)

The single most cost-effective improvement to a RAG system is re-ranking. You retrieve more candidates than you need, then use a re-ranking model to score them for relevance and keep only the best before sending them to the model. It fixes the most common RAG complaint, the right answer was retrieved but buried, more reliably than almost anything else.

If naive RAG is returning roughly-relevant-but-not-quite results, this is your next step, not a bigger model.

3. Hybrid RAG (when keywords still matter)

Pure semantic search is great at meaning and bad at exactness. Ask for a specific product code, error number, or proper noun and semantic search can drift. Hybrid RAG combines semantic similarity with traditional keyword search, so you get the best of both.

Reach for hybrid when:

• Your content is full of exact terms: SKUs, error codes, names, clause numbers.
• Users search with precise language that must match precisely.
• Semantic-only retrieval keeps missing obvious keyword hits.

4. Query-rewriting RAG (when users ask badly)

Real questions are messy: vague, multi-part, full of pronouns referring to earlier context. Query-rewriting RAG uses a model to clean up or expand the user's question before retrieval, turning a fuzzy ask into one or several precise queries.

This pattern earns its keep in conversational interfaces, where a question like and what about the second one only makes sense in the context of the chat. Rewrite first, retrieve second, and your hit rate jumps without touching the rest of the pipeline.

5. Agentic RAG (when retrieval needs to reason)

Sometimes one retrieval is not enough. A complex question needs the system to retrieve, reason about what it found, decide it needs more, and retrieve again. Agentic RAG wraps retrieval in a reasoning loop, letting the system run multiple searches and synthesise across them.

It is powerful and it is expensive, so use it deliberately:

• Multi-hop questions that require connecting facts from different documents.
• Open-ended research where the right next query depends on the last result.
• Cases where a single retrieval genuinely cannot contain the answer.

6, 7, 8. Graph, multi-index, and hierarchical RAG

At the top of the ladder are the specialist patterns. Graph RAG stores relationships between entities, not just text chunks, which shines when the answer depends on connections (who reports to whom, what depends on what). Multi-index RAG routes a query to the right knowledge base among several. Hierarchical RAG retrieves summaries first, then drills into detail, which helps across very large corpora.

These solve real problems, but they are where teams most often over-build. Do not reach for graph RAG because it sounds impressive; reach for it because your questions are fundamentally about relationships and the simpler patterns demonstrably fail.

How to actually choose

Ignore the temptation to pick the most advanced pattern. Choose by failure mode, the specific way your current system is wrong, and climb the ladder one rung at a time.

• Building your first system? Naive RAG. Measure before anything else.
• Right content retrieved but ranked poorly? Add re-ranking.
• Missing exact terms and codes? Go hybrid.
• Users ask vague or conversational questions? Add query rewriting.
• Questions need multiple linked lookups? Agentic RAG.
• Answers are fundamentally about relationships, or you span many distinct corpora? Graph or multi-index, carefully.