RAG vs. fine-tuning: pick in 30 seconds

// the clean split

the WHAT

RAG

Fresh, changing, or private facts
"Chat over my docs"
Update by changing the documents — no retraining
Cost: retrieval infra + context-window budget; quality depends on chunking

the HOW

fine-tuning

A consistent format / tone / style
A narrow skill or task pattern
Specialize a smaller, cheaper, faster model
Cost: a labeled dataset + a training run + redeploy

Why not just fine-tune the facts in?

Because it backfires. Fine-tuning is a poor, unreliable way to add knowledge — and training a model on facts it doesn't already know teaches it a worse habit: to assert things it isn't sure of.

A 2024 study (Gekhman et al., EMNLP) found that as a model finally learns new facts via fine-tuning, it linearly increases its tendency to hallucinate. A separate study (Ovadia et al.) found RAG consistently beats fine-tuning for knowledge injection — for both old and brand-new knowledge.

Not "never," just "wrong tool." Fine-tuning can encode facts, slowly and at a hallucination cost. For knowledge you want retrieval. For behavior you want fine-tuning.

They're not rivals

The best setups use both: fine-tune the how (a model that always answers in your format and voice), and use RAG for the what (current, citeable facts). That hybrid pattern (e.g. RAFT) beats either alone.

Pick in 30 seconds: need current/proprietary facts? RAG. Need consistent format, tone, or a narrow skill? Fine-tune. Need both? Do both — and start with prompting before you reach for either.

Sources: Microsoft — RAG vs fine-tuning · Gekhman et al. — fine-tuning new knowledge & hallucination · Ovadia et al. — fine-tuning vs retrieval · AWS/Anthropic — fine-tuning best practices

One concept a week. Free.

The deeper, copy-paste version of each ToolCall short — in your inbox.

// total: 0.00 · spam: void · unsubscribe: one click