Fine-tuning vs RAG vs prompting: a decision framework

The three levers, and what each actually changes

Almost every request to make an AI model better comes down to three levers, and the single biggest source of wasted money in enterprise AI is pulling the wrong one. They are not interchangeable, and they do not compete so much as solve different problems.

Get this distinction right and most decisions make themselves:

• Prompting changes what you ask. You shape the instructions, examples, and structure you give the model at request time. Cheapest, fastest, most flexible.
• RAG (retrieval-augmented generation) changes what the model knows. It fetches relevant passages from your own data and feeds them in as context, so answers are grounded in your business.
• Fine-tuning changes how the model behaves. You train it on examples so it adopts a consistent style, format, or task pattern without being told every time.

Start with prompting (almost always)

Prompting gets dismissed as the beginner option, which is exactly why teams skip past it and overspend. A surprising share of problems people bring to us as fine-tuning projects dissolve once the prompt is written properly: clear instructions, a few worked examples, a defined output structure, and constraints on what not to do.

Prompting wins on iteration speed. You can test a change in seconds, with no training run, no data pipeline, and nothing to redeploy. It is also the easiest to govern, because the behaviour lives in text you can read and version. The limits are real, though: prompts get unwieldy past a point, they cost tokens every call, and they cannot teach the model facts it was never trained on. That last limit is where RAG comes in.

Reach for RAG when the model needs your data

The clearest signal you need RAG is simple: the model keeps confidently inventing answers about your business. It does not know your refund policy, your contracts, or last quarter's numbers, so it guesses. RAG closes that gap by retrieving the relevant material at the moment of the question and grounding the answer in it, with sources attached.

RAG is the right tool when:

• Answers must reflect your own documents, data, or policies.
• The underlying information changes often, so it cannot be baked into the model.
• You need traceability, every answer pointing back to a source.
• The knowledge base is large enough that you cannot just paste it into the prompt.

The mistake almost everyone makes

Here is the pattern we see most often, and it is an expensive one. A team wants the model to know their internal knowledge, so they decide to fine-tune it on their documents. It feels intuitive, train the model on our stuff so it learns our stuff, and it is almost always wrong.

Fine-tuning is poor at injecting knowledge. It nudges behaviour and style, but it does not reliably memorise facts, and the moment your documents change, the fine-tuned model is out of date with no easy fix. RAG handles knowledge cleanly: update the source, and the next answer reflects it. As a rule, fine-tune for form, retrieve for facts.

When fine-tuning genuinely earns its cost

None of this means fine-tuning is useless. It is the right lever in specific situations, usually once you are operating at scale and prompting has hit a ceiling. The tell is that you keep writing longer and longer prompts to force consistent behaviour, and it still drifts.

Fine-tuning makes sense when:

• You need a consistent style, tone, or output format across very high volume, and prompting it every time is costly or unreliable.
• You have a narrow, repeatable task and a good set of labelled examples to learn from.
• You want to shrink prompts (and cost) by baking the instructions into the model.
• Latency matters and you want a smaller, specialised model to match a larger one on your specific task.

They combine better than they compete

In production, the strongest systems rarely pick just one. A common, durable pattern is a fine-tuned smaller model that has learned your format and tone, wrapped in a RAG pipeline that feeds it current facts, with prompting on top to handle the specifics of each request.

That layering is the point. Fine-tuning handles how it sounds, RAG handles what it knows, prompting handles what you want right now. Treating them as a stack instead of a choice is what separates teams that ship reliable AI from teams stuck tuning a single dial.

A simple decision path

When a new use case lands on the table, this is the order we work through it, and it keeps cost and complexity proportional to the problem.

• Can a well-written prompt do it? Try that first. Most things stop here.
• Does it need your data or current facts? Add RAG.
• Does it need consistent behaviour or format at scale that prompting cannot hold? Then, and only then, consider fine-tuning.
• Re-evaluate as volume grows. The right answer at a prototype is often the wrong one at scale, and the reverse is just as true.