Fine-tuning vs RAG: when you actually need to fine-tune
Continuing a model's training on examples of the behaviour you want, so that behaviour becomes its default instead of something you ask for on every request. It changes how a model works, not what it knows.
There are only three ways to change what a model does for you, and they are not interchangeable. Picking the wrong one is expensive in a way that only shows up months later.
| Lever | What it changes | When it costs you |
|---|---|---|
| Prompting | What the model does this time | Every request, in tokens — and long instructions get followed less reliably as the conversation grows |
| Retrieval | What the model knows right now | Per request, in retrieval work. Facts stay current because they live outside the model |
| Fine-tuning | What the model does by default | Once, up front. Free at request time — the behaviour is baked in |
What fine-tuning actually is
You take a base model and continue training it on examples of the behaviour you want: pairs of input and the answer a good practitioner in your field would have given. The model's defaults shift — which words it reaches for, what shape an answer takes, what it asks before it answers.
It is not a database. Nothing is stored and looked up. The examples move the model's tendencies; they do not become facts it can recite.
In practice it is an adapter, not a new model
Retraining every weight is expensive and rarely necessary. The normal approach — LoRA, or low-rank adaptation — trains a small set of extra parameters that sit on top of a frozen base model. The result is a file of tens of megabytes rather than tens of gigabytes.
That has a consequence worth knowing: one base model can serve many adapters. A single machine can run the same base weights for a law firm, a clinic and a logistics operator, swapping a small adapter per tenant. Fine-tuning per industry does not mean a whole model per industry.
When you actually need it
The register matters and a prompt cannot hold it. Japanese business honorifics, clinical phrasing, the way an underwriter hedges. You can describe register in a prompt; under a long conversation the model drifts back to its defaults. Fine-tuning moves the default.
Format compliance at volume. If every output must be a particular structure and "usually correct" is not good enough, examples teach it more reliably than instructions do.
The instructions have outgrown the answer. When the system prompt is longer than what it produces, you are paying for those tokens on every single request. Fine-tuning is how that cost is moved off the per-request path.
A small model in a narrow domain. A well-tuned small model often beats a much larger general one inside one trade — which is a cost decision as much as a quality one. See bring your own model.
The base model is weak in your language or field. Some languages and specialisms are thin in public training data. Examples are how you fix that.
When you do not need it
Facts do not belong in a fine-tune. Prices, policies, availability, case status — anything that can change on a Tuesday belongs in a knowledge base, where changing it is an upload. A fine-tuned model does not update when your price list does, and it will keep answering confidently with what it learned.
- You have not yet tried a better prompt. Most "we need a custom model" problems are prompt problems. Fine-tuning is the expensive answer; earn it first.
- You have a handful of examples. A few dozen will not move a model. Hundreds of good ones might; quality beats volume, and bad examples train bad habits faithfully.
- The thing you want changes often. Anything you would want to edit weekly should not be baked in.
What it takes to do properly
Examples. Hundreds to low thousands, drawn from real work and checked by someone who knows the field. This is the part customers underestimate — the data collection is usually longer than the training.
A held-out evaluation set. Examples the model never saw, scored before and after. Without this you cannot tell whether the fine-tune helped, hurt, or did nothing — and "it feels better" is not a measurement. This is the step most often skipped and the one that decides whether the project was worth doing.
A plan for the base model changing. Adapters are tied to the base they were trained on. When you move to a newer base model, the tuning is redone. Budget for it as maintenance, not as a one-off.
The failure to design against
A model tuned too hard on one narrow style gets worse at everything else — it answers in your house voice even when the question needed a different register, and it becomes confidently fluent about things it was never taught. This is why the evaluation set matters, and why the sensible target is usually a modest shift in defaults rather than a model that can only do one thing.