RAG vs Fine-Tuning: How to Choose for Your LLM Application
A practical decision guide for engineers: what RAG and fine-tuning each fix, what they cost, how they fail, and why most production systems end up combining them.
Every team that ships an LLM feature hits the same wall. The base model is fluent, but it does not know your product, your policies or last week's release notes, and it does not answer in the shape your application needs. Two tools come up immediately: retrieval-augmented generation (RAG) and fine-tuning. They are often pitched as rivals. They are not. They fix different problems, and picking the wrong one costs weeks of work and a system that still gets the answer wrong.
This guide walks through what each technique actually changes, where each one breaks, and a decision process you can apply to your own use case.
The short version: RAG changes what the model knows at answer time. Fine-tuning changes how the model behaves. If your problem is missing or changing knowledge, start with RAG. If your problem is format, tone, or a narrow task the model keeps doing badly, consider fine-tuning. Many production systems need both.
Start with the right question: knowledge gap or behaviour gap?
Before choosing a technique, write down what is going wrong in concrete terms. Look at twenty failed outputs and sort them into two buckets.
Symptom | Gap | Usual fix |
|---|---|---|
Answers with outdated pricing or a deprecated API | Knowledge | RAG |
Makes up an internal policy it has never seen | Knowledge | RAG |
Cannot cite where an answer came from | Knowledge | RAG |
Ignores the JSON schema one call in fifty | Behaviour | Better prompting, structured output, then fine-tuning |
Tone is wrong for your brand, no matter the prompt | Behaviour | Fine-tuning |
A narrow classification task needs a 2,000-token prompt to work | Behaviour | Fine-tuning a smaller model |
If most failures land in the knowledge bucket, fine-tuning will disappoint you. If most land in the behaviour bucket, adding a vector database will not help.
How RAG works
RAG keeps the model frozen and gives it the relevant facts inside the prompt, fetched at query time. A typical pipeline has two halves.
Indexing (offline)
Ingest your sources: docs, tickets, PDFs, database rows.
Chunk them into passages small enough to be specific but large enough to stand alone. A few hundred tokens with some overlap is a common starting point; split on headings and paragraphs rather than fixed character counts where you can.
Embed each chunk with an embedding model, turning text into a vector.
Store the vectors with their text and metadata (source, date, access level) in a vector index. Postgres with pgvector is often enough; you do not need a dedicated database on day one.
Answering (online)
Embed the question with the same model.
Retrieve the nearest chunks. Combining vector search with keyword search (hybrid retrieval) catches exact terms like error codes and product names that embeddings blur.
Rerank the candidates with a cross-encoder or an LLM, keeping the best handful.
Generate an answer from those chunks, with instructions to cite them and to say so when the answer is not in the context.
def answer(question, user):
candidates = hybrid_search(question, k=30, filters={"access": user.access})
context = rerank(question, candidates)[:5]
prompt = (
"Answer using only the sources below. "
"Cite sources as [n]. If the sources do not contain the answer, say so.\n\n"
+ format_sources(context)
+ "\n\nQuestion: " + question
)
return llm.generate(prompt)
Where RAG shines
Fresh knowledge. Update a document, re-embed it, and the next answer uses it. No training run.
Citations. Every answer can point at the passages it used, which makes it auditable and lets users verify it.
Access control. Filter retrieval by the caller's permissions, so a user never sees content from a document they cannot open. Fine-tuned weights cannot forget selectively.
Debuggability. When an answer is wrong you can inspect what was retrieved and see whether retrieval or generation failed.
Large, long-tail corpora. Thousands of documents, each rarely asked about, are exactly what retrieval is built for.
Where RAG struggles
Retrieval fails silently. If the right chunk is not retrieved, the model answers from whatever it got, often confidently. Most bad RAG answers are retrieval bugs, not model bugs.
Chunking loses context. A chunk that says "this limit is 10 per minute" is useless if the heading that names the API was in the previous chunk. Prepending titles and section paths to each chunk helps a lot.
Multi-hop questions. "Which of our enterprise customers are affected by the change in last month's release?" needs facts from several places joined together. Single-shot retrieval rarely gets all of them; you need query decomposition or an agentic loop.
Latency and token cost. Every call carries thousands of context tokens plus a retrieval round trip.
It does not teach behaviour. Putting five examples of your output format in the context helps, but it is not the same as a model that reliably produces that format.
How fine-tuning works
Fine-tuning continues training a pretrained model on your own examples so its weights shift towards the behaviour you want. The common forms:
Supervised fine-tuning (SFT): pairs of input and ideal output. This is what most teams mean by fine-tuning.
Parameter-efficient fine-tuning (PEFT), such as LoRA: freeze the base model and train small adapter matrices. Far cheaper in memory and storage, and you can keep several adapters for one base model.
Preference tuning, such as DPO: pairs of a better and a worse answer to the same prompt, used to push the model towards the preferred style.
Training data for SFT is usually chat-formatted JSONL, one example per line:
{"messages": [
{"role": "system", "content": "Classify the support ticket. Reply with one label."},
{"role": "user", "content": "I was charged twice for my March invoice."},
{"role": "assistant", "content": "billing_duplicate_charge"}
]}
Where fine-tuning shines
Consistent format and style. Strict output schemas, a house writing voice, a specific way of structuring code reviews.
Narrow, repeated tasks. Classification, extraction, routing and tagging, where you have many labelled examples.
Shorter prompts. Behaviour baked into the weights means you can drop long instructions and few-shot examples, cutting latency and cost on every call.
Smaller models. A fine-tuned small model can match a much larger general model on one narrow task, at a fraction of the cost. This is often the biggest win.
Where fine-tuning struggles
It is an unreliable way to add facts. Models trained on new facts tend to learn the style of the answers more readily than the facts themselves, and they can hallucinate in that confident new style. Facts seen only a few times in training are recalled poorly.
Knowledge goes stale. Every change to the underlying information means a new dataset and a new training run.
No citations, no access control. Whatever went into the weights is available to every user, and the model cannot tell you where an answer came from.
Data is the real cost. Hundreds to thousands of high-quality examples, reviewed by someone who knows the domain. Bad examples train bad behaviour very efficiently.
Regressions. Tuning hard on one task can degrade general abilities. You need an eval suite that checks more than the target task.
Side by side
Dimension | RAG | Fine-tuning |
|---|---|---|
What it changes | The context the model sees | The model's weights |
Best for | Facts, documents, changing data | Format, style, narrow tasks |
Updating knowledge | Re-index a document, minutes | New dataset and training run |
Citations | Yes | No |
Per-user access control | Yes, via retrieval filters | No |
Upfront effort | Pipeline, chunking, retrieval tuning | Labelled data, training, evals |
Per-request cost | Higher: long prompts plus retrieval | Lower: short prompts, possibly a smaller model |
Typical failure | Wrong or missing chunks retrieved | Confident answers that drifted from the truth |
A decision process that works
Build an eval set first. Fifty to a hundred real questions with known good answers. Without it, every later step is a guess.
Push prompting as far as it goes. Clear instructions, a few examples, structured output modes. This is cheap, fast to iterate, and often enough. Measure it against your eval set.
Diagnose the remaining failures into knowledge and behaviour gaps, as in the table at the top.
Knowledge gaps: add RAG. Measure retrieval on its own before judging the answers.
Behaviour gaps that survive good prompting: fine-tune. Use outputs that passed your evals as a starting point for training data, then have a human review them.
Both: combine them as described below.
Combining the two
The strongest systems use each technique for what it is good at:
Fine-tune the generator to use retrieved context. Train on examples that include retrieved passages, some of them irrelevant, with answers that cite the right ones and ignore the noise. The model gets better at grounding and at saying "not in the sources".
Fine-tune the retrieval side. An embedding model or reranker tuned on your own query and document pairs often improves RAG more than changing the LLM, especially in jargon-heavy domains.
Fine-tune small helper models. Query rewriting, intent routing and metadata extraction are narrow tasks where a small tuned model is fast and cheap, while a larger model writes the final answer.
How to evaluate either approach
Measure retrieval and generation separately, or you will not know which half to fix.
Retrieval: for each eval question, label which chunks contain the answer. Track recall@k (is a correct chunk in the top k?) and MRR (how high does the first correct chunk rank?).
Generation: faithfulness (is every claim supported by the provided context?), answer relevance (does it answer the question asked?) and correctness against your reference answer.
LLM-as-judge scales these checks, but calibrate it: grade a sample by hand and confirm the judge agrees with you before trusting its numbers.
Run the full suite on every change: new chunking, a new embedding model, a new prompt, a new fine-tuned checkpoint.
Common mistakes
Fine-tuning on a product manual and expecting the model to answer questions about it accurately. Use RAG for this.
Blaming the LLM for bad RAG answers without checking what was retrieved.
Fixed-size chunking that cuts tables and code blocks in half.
Skipping keyword search, then wondering why exact error codes are never found.
Fine-tuning before trying a well-structured prompt and structured outputs.
Shipping without an eval set, so nobody can tell whether a change helped.
Worked example: a support assistant for a SaaS product
Say you are building an assistant that answers customer questions from your docs, changelog and help-centre articles, and drafts replies for the support team.
The docs change every release and answers must link to the right article. That is a knowledge problem, so the core is RAG with hybrid search, reranking and citations, filtered by the customer's plan so they only see features they have.
Support wants every drafted reply in a fixed structure with a specific tone, and prompting gets it right most of the time but not reliably. That is a behaviour problem. After collecting a few hundred approved replies, a LoRA fine-tune of the generator on retrieval-augmented examples locks in the format.
Routing tickets to the right team is a narrow classification task with thousands of historical labels, so a small fine-tuned classifier handles it in milliseconds.
Three techniques, each doing the job it is suited to, measured against one shared eval set.
Conclusion
RAG and fine-tuning answer different questions. RAG answers "what should the model know right now?", and fine-tuning answers "how should the model behave every time?". Diagnose your failures before choosing, start with prompting and a solid eval set, reach for RAG when knowledge is the gap and fine-tuning when behaviour is, and combine them when you need both. The teams that ship reliable LLM features are not the ones with the fanciest technique; they are the ones who can measure which part of their system is failing.
