Fine-Tuning or Prompting? A Decision Guide

Prompting, RAG, or fine-tuning - three different fixes for three different problems. Picking the wrong one wastes time and doesn't fix what's actually broken.

Three tools, three different problems

This entry assumes you know what fine-tuning is (see that entry for mechanics) - here we decide which of three approaches fits: prompting, RAG, or fine-tuning. Each targets a different gap between what the model does and what you need.

Start with prompting, always

If the model has the knowledge but isn't using it the way you want - wrong format, missing a step, wrong tone - that's a prompting problem: clearer instructions, better examples, structured output. Cheapest, fastest fix, resolving most gaps alone.

Reach for RAG when it's a knowledge gap

If the model doesn't know something - internal docs, today's price list, weekly-changing content - no prompting fixes that, since the information isn't in the training data or your prompt. RAG retrieves the missing facts at request time, staying current with no retraining.

Reach for fine-tuning last, and rarely

Fine-tuning earns its cost when you need a specific format applied with extreme consistency at huge volume, or far more examples than fit in a prompt. It's the most expensive to iterate on, and a poor tool for new factual knowledge - that's what RAG is for.

The deciding questions

How often does the knowledge change (favors RAG if often)? How much data do you have (fine-tuning needs a real dataset)? What's your tolerance for maintenance (prompting iterates instantly; fine-tuning means retraining whenever behavior shifts)?

EXAMPLE

Three gaps, three different fixes: Gap: "The model writes summaries that are too long and don't follow our house style." -> Prompting fix: add explicit length limits, house-style examples (few-shot), maybe a structured output template. Gap: "The model doesn't know about the product we launched last month, or our current pricing." -> RAG fix: index your current docs/pricing pages, retrieve relevant chunks at request time. No retraining needed when pricing changes again next quarter. Gap: "We need the model to always output a very specific, unusual JSON-like internal format, with 100% consistency, across 50,000 requests a day, and prompting keeps drifting slightly under load." -> Fine-tuning fix (maybe): if you have hundreds+ of real labeled examples and the format truly can't be enforced reliably enough by prompting/structured-output alone, fine-tuning bakes the format in more durably than repeating instructions every call. Quick self-check table: | Question | If yes, points toward | |---|---| | Would pasting the missing info into the prompt fix it right now? | RAG | | Is it about instructions/format/tone the model currently ignores or misapplies? | Prompting | | Do you have hundreds+ of examples and does it NOT change often? | Fine-tuning (maybe) | | Not sure? | Prompting first - it's the cheapest experiment |

🛠️ EXERCISE — TRY IT YOURSELF

Classify three real or hypothetical model-behavior gaps from your own work using the decision tree.

Write down three separate things you wish a model did better in a project you're working on.
For each, ask: would pasting the missing information directly into the prompt fix it right now?
For each remaining one, ask: is this really a formatting/tone/instruction-following issue rather than a knowledge gap?
Classify each of the three as prompting, RAG, or fine-tuning (or a combination).
For any you classified as fine-tuning, check: do you actually have a real dataset of examples, or just a vague sense of desired behavior?

✅ SELF-CHECK

☐ For the gap you classified as a knowledge gap, could you point to the specific missing information that a retrieval step would need to supply?
☐ For anything you initially leaned toward fine-tuning, did the decision tree talk you out of it in favor of a cheaper option first?

QUICK QUIZ

Why is fine-tuning generally a poor fix for a model 'not knowing' a specific, frequently-changing fact?