RAG vs fine-tuning in 2026: a decision ladder that starts with prompting

RAG gives a model fresh, private knowledge at query time without changing its weights; fine-tuning changes the weights to teach behaviour, format or style. Start with prompt engineering, add RAG when the model needs your data, and fine-tune only when behaviour still falls short. Many production systems combine both.

Most explainers on this topic stop at "RAG for knowledge, fine-tuning for behaviour". That is right, but it leaves out three things that matter in 2026: long-context models can now take small knowledge bases straight into the prompt, OpenAI is winding down its fine-tuning platform, and retrieval opens a new route for prompt injection. This guide adds all three and turns the choice into a five-rung ladder, with a trigger, cost driver and evaluation method for each rung. Platform facts were checked on 23 October 2026.

How do RAG, fine-tuning and prompt engineering compare?

IBM sums up the RAG side well: "RAG plugs an LLM into stores of current, private data that would otherwise be inaccessible to it." And Red Hat makes the point most teams learn the hard way: "Often, you can customize a model by using both fine-tuning and RAG architecture."

How does RAG work?

The idea comes from Lewis et al. (2020), who described "models which combine pre-trained parametric and non-parametric memory for language generation" (arXiv 2005.11401). In their paper, the non-parametric memory was "a dense vector index of Wikipedia". In a business system, it is your own documents.

A typical RAG pipeline has five steps:

1. Ingest: collect documents (policies, product data, tickets, contracts) and keep their permissions.
2. Chunk: split them into passages small enough to retrieve precisely.
3. Embed and index: turn each chunk into a vector and store it, usually alongside a keyword index.
4. Retrieve and rerank: for each question, fetch the most relevant chunks, then reorder them with a reranking model.
5. Generate: send the question plus the retrieved chunks to the model, with instructions to answer only from them and cite their sources.

Retrieval quality decides answer quality. Anthropic's contextual retrieval work (19 September 2024) adds a short, document-aware description to each chunk before indexing. Anthropic reported: "This method can reduce the number of failed retrievals by 49% and, when combined with reranking, by 67%." Those are Anthropic's results on its own test sets, but the lesson travels: invest in chunking, hybrid search and reranking before you blame the model.

How does fine-tuning work?

Fine-tuning continues training a pre-trained model on your own examples, so the weights shift towards the behaviour you want. The common methods:

• Supervised fine-tuning (SFT): train on pairs of inputs and ideal outputs. Good for formats, tone, classification and domain-specific phrasing.
• Preference tuning, such as DPO (direct preference optimisation): train on pairs of better and worse answers so the model learns which it should prefer.
• Parameter-efficient methods, such as LoRA: instead of updating every weight, LoRA "freezes the pre-trained model weights and injects trainable rank decomposition matrices into each layer". Its authors reported that, against full fine-tuning of GPT-3 175B, "LoRA can reduce the number of trainable parameters by 10,000 times and the GPU memory requirement by 3 times" (Hu et al., arXiv 2106.09685).

What fine-tuning does well: consistent structure, a house style, a narrow task done cheaply by a smaller model, or behaviour that is hard to describe in a prompt. What it does badly: keeping facts current. A fine-tuned model knows what it was trained on, and nothing that changed afterwards.

What is the 2026 decision ladder?

Climb one rung at a time. Move up only when an evaluation on the current rung shows a gap you can't close.

Two rules keep this honest. First, build the evaluation set before you build anything else; without it, you can't tell whether a rung helped. Second, write down the trigger that sent you up a rung, so you can revisit it when models improve.

Do long-context models make RAG obsolete?

For small knowledge bases, sometimes. Anthropic's guidance is direct: if your knowledge base is "smaller than 200,000 tokens (about 500 pages of material)", you can include it in the prompt and skip retrieval. With prompt caching, which Anthropic says can reduce latency "by > 2x and costs by up to 90%", sending the same large context repeatedly becomes far more affordable.

RAG still wins when:

• The corpus is large: thousands of documents won't fit in any context window.
• Data changes daily: re-indexing a few documents beats rebuilding a giant prompt.
• Permissions differ by user: retrieval can filter by what each user is allowed to see; a shared prompt can't.
• You need citations: retrieval tells you which passage each answer came from.
• Latency and cost per call matter at scale: a few relevant chunks are cheaper than a full manual on every request.

So long context is a rung on the ladder, not a replacement for it.

Can you still fine-tune OpenAI models?

Only for existing customers, and not for long. On 23 October 2026, OpenAI's model optimization guide said: "OpenAI is winding down the fine-tuning platform." It adds that the platform "is no longer accessible to new users" and that "All fine-tuned models will remain available for inference until their base models are deprecated." OpenAI's deprecations page sets out the timeline:

What this means in practice:

• If you rely on an OpenAI fine-tuned model, plan its replacement now. It will stop working when its base model is retired, and you won't be able to retrain it after 6 January 2027.
• Keep your training data portable. Your curated examples are the real asset; store them in a neutral format you can reuse with another provider or model.
• Consider open-weight models with LoRA if fine-tuning is essential and you want control over the model's lifecycle. You take on hosting, but nobody can switch the platform off.
• Re-test the lower rungs first. Newer models with better prompting, long context or RAG often close the gap that fine-tuning was meant to fix.

Provider policies change quickly, so check each vendor's current documentation before you commit to a tuning plan.

What drives cost, and how do you evaluate each approach?

We don't publish token prices here because they change often and differ by provider and model. The cost drivers are stable, though:

• Prompting: input and output tokens per request, multiplied by volume. Long system prompts are paid on every call unless cached.
• RAG: one-off ingestion and embedding, ongoing index hosting and re-indexing, retrieval and reranking calls, plus the extra input tokens from retrieved chunks. The engineering time for chunking, permissions and evaluation is usually the largest line.
• Fine-tuning: collecting and cleaning training examples (often the biggest cost), training runs, evaluation, and hosting or serving the tuned model. Retraining whenever the base model or your requirements change.

Evaluation should be designed before any of it:

1. Build a test set of real questions with agreed good answers, including hard and adversarial cases.
2. Measure retrieval separately for RAG: did the right passage come back in the top results?
3. Measure answers for correctness, faithfulness to the sources, format and refusal behaviour.
4. Track cost and latency per request alongside quality, so you can see what each rung buys.
5. Re-run the suite whenever you change the model, prompt, index or training data.

How do you secure RAG against prompt injection?

RAG adds a new attack surface: the documents themselves. OWASP's top LLM risk, LLM01:2025 Prompt Injection, defines it as a vulnerability that "occurs when user prompts alter the LLM's behavior or output in unintended ways", and notes that "Indirect prompt injections occur when an LLM accepts input from external sources, such as websites or files." OWASP also says RAG doesn't fully mitigate prompt injection, and one of its scenarios describes a modified document carrying malicious instructions into a RAG application.

Practical defences:

• Treat retrieved text as data, never as instructions. Separate it clearly in the prompt and tell the model not to follow instructions found inside it.
• Control what gets indexed. Ingest only trusted sources, log who added what, and review external content before it enters the index.
• Enforce permissions at retrieval time, so users only get chunks they are allowed to see.
• Limit what the model can do. If the system can call tools or take actions, give it the least privilege it needs and require human approval for anything consequential.
• Filter and monitor outputs for leaked data, unexpected links and policy breaks, and test with adversarial documents before launch.

How can Elgrith help?

Elgrith has delivered 1,000+ projects for 250+ clients since 2015, from our New Delhi headquarters and client centers in New York, London and Shanghai. Our generative AI development team builds LLM products end to end, our LLM integration team adds RAG, retrieval evaluation and model APIs to existing systems, and our AI agent development team builds tool-using agents with guardrails. For a wider view of AI work, see AI development. We start with the lowest rung that works and build the evaluation set first. You get a fixed written quote after a free call. Book a call or start a project.