Loading…
Loading…
LLM fine tuning explained: full tuning vs LoRA and QLoRA, supervised and preference tuning, the data you need, what it costs and when it is worth doing.
LLM fine tuning changes how a model behaves, not what it knows. Use it for format, tone, vocabulary and narrow tasks.
Most fine tuning now trains a small add on, not the whole model. LoRA and QLoRA put large open models within reach of a single GPU.
Supervised fine tuning teaches by example. Preference tuning, through RLHF or DPO, teaches by comparison. The original ChatGPT used both.
Data decides the outcome, and preparing it costs more than the training run.
Build the evaluation set before you train. Without a baseline, nobody can show the tuned model is better.
A fine tuned open model can run entirely on your own servers, which matters under India's DPDP Act and the GDPR.
Search for LLM fine tuning and most results are code tutorials that begin at the training script. They skip the questions that decide whether the project pays off. Is fine tuning the right tool at all? How will you measure success? What happens when a better base model arrives next year?
At Aptibit, we fine tune language models for clients, so we have an interest in the technique. That is why this guide spends as long on when not to fine tune as on how to do it.
LLM fine tuning is the process of taking a large language model that has already been trained on general text and training it further on a smaller set of examples from one task or domain. The model keeps its general language ability and learns a specific behaviour, such as a format, a tone or a narrow job.
NIST's glossary defines it in the same terms: a training step that starts from a pretrained model and adds task or domain specific information. NIST adds that it is often a supervised learning task, which is the form most businesses use.
Pretraining, where a model learns language itself from enormous amounts of text, is beyond all but a handful of organisations. Fine tuning is the far cheaper step that follows.
An LLM fine tuning example: an insurer wants every claim note turned into a summary with the same seven fields, in house style. A general model gets this right most of the time. A model tuned on a few hundred approved summaries gets it right consistently, often on a smaller, cheaper model.
The first choice is how much of the model to change. The three main approaches differ mostly in cost and hardware.
Full fine tuning updates every parameter, the internal settings a model learns, of which large models have billions. It offers the most freedom, needs the most memory, and produces a complete new copy of the model for every task.
LoRA, short for low rank adaptation, freezes the original model and trains a small set of extra settings alongside it. In the 2021 LoRA paper, Microsoft researchers reported that, compared with fully fine tuning the 175 billion parameter GPT-3, LoRA cut the number of trainable parameters by 10,000 times and the GPU memory needed by three times. Quality matched or beat full fine tuning on the models tested, with no added delay when the model answers.
QLoRA goes further. It compresses the frozen model to 4 bit precision and then trains a LoRA add on over it. The 2023 QLoRA paper from University of Washington researchers tuned a 65 billion parameter model on a single 48GB GPU while preserving the task performance of full 16 bit fine tuning. Serious fine tuning no longer needs a data centre.
Together, these methods are known as parameter efficient fine tuning, or PEFT. Hugging Face, whose PEFT library implements them, describes the aim: tune a small number of extra parameters, cut computing and storage costs, and perform comparably to a fully tuned model. Adapters, small extra layers inserted into the model, belong to the same family.
For most business work, LoRA or QLoRA on an open model is the sensible default. Full fine tuning earns its cost only when the task is far from anything the base model has seen. LoRA also lets one base model serve several tasks, each with its own small add on.
Supervised fine tuning trains a model on pairs of inputs and ideal outputs, written or approved by people. The model learns by imitation: given this, produce that. OpenAI's documentation describes it as providing examples of correct responses to guide the model's behaviour, and it is the most common form of fine tuning in business.
Instruction tuning is supervised fine tuning on a broad mix of tasks, each phrased as a plain instruction, such as summarise this letter. It turns a raw pretrained model, which only continues text, into one that follows requests. The instruct or chat versions of open models have already been through it.
So the two terms describe one technique at two scales. The model builder does instruction tuning once, broadly. You do supervised fine tuning narrowly, on your task, on top of a model that is already instruction tuned.
Supervised tuning shows the model a right answer. Preference tuning shows it two answers and says which is better. It suits qualities that are easier to judge than to write down: helpfulness, tone, when to decline, how much detail to give.
Reinforcement learning from human feedback, or RLHF, is the original method. People rank the model's answers, a separate reward model learns those rankings, and the language model is trained to score well against it. It works, but it is complex and can be unstable to train.
Direct preference optimisation, or DPO, published by Stanford researchers in 2023, removes the separate reward model and trains directly on pairs of preferred and rejected answers. The authors showed it matching or beating RLHF on the tasks they tested while being substantially simpler to run. For most business preference tuning, DPO is now the practical starting point.
In two stages: supervised fine tuning, then reinforcement learning from human feedback. OpenAI set out the recipe in its 2022 InstructGPT paper, and the original ChatGPT, launched later that year, was trained with the same methods on conversation data.
People wrote demonstrations of good answers, and GPT-3 was fine tuned on them. People then ranked the model's answers, and those rankings drove the reinforcement learning stage. The result was striking: people preferred answers from a 1.3 billion parameter InstructGPT model over those from the 175 billion parameter GPT-3, a model more than 100 times larger.
The lesson for business: how a model is tuned can matter more than how big it is.
LLM fine tuning is strong wherever the problem is how the model responds, rather than what it knows.
It is good at format: returning the same structure and the same fields every time, in data your systems can read. It is good at style and tone, and at the vocabulary of your field, used correctly. It is good at narrow classification and extraction, such as routing tickets or pulling clauses from contracts. And it can make a small model match a large one on a single task, cutting running costs.
It is poor at adding facts, especially facts that change. Training on your price list does not make the model reliably recall prices. It may blend old and new, it cannot cite where an answer came from, and every change means training again. It does not add reasoning ability the model lacks. And it can erode general abilities the base model had, a problem researchers call catastrophic forgetting, which is why testing must reach beyond the target task.
The rule: if the problem is a missing fact, give the model the fact when it answers. If it is a behaviour, fine tune.
Use retrieval augmented generation (RAG) when the model needs knowledge, and fine tuning when it needs a behaviour. RAG looks up the relevant passages in your documents at the moment of each question and has the model answer from them, with sources, so it handles facts that change and can show its evidence. Fine tuning handles format, tone and narrow skills. Many strong systems use both: RAG for what the model should know, fine tuning for how it should answer. Our companion guide, RAG vs fine tuning, covers the full decision with costs and examples, and our enterprise search guide covers the retrieval side.
Less than most people expect, and of higher quality than most people prepare.
OpenAI's fine tuning documentation sets a minimum of ten examples, reports improvements from 50 to 100, and recommends starting with 50 well crafted demonstrations, adding more only if those help. If 50 good examples change nothing, it advises rethinking the task or the prompt before collecting more data. The QLoRA authors reached a similar conclusion from another direction: tuning on a small, high quality dataset produced state of the art results.
In our experience, a few hundred to a thousand carefully checked examples make a real difference on a focused business task. They must be consistent, so no two teach opposite answers; cover the real range of inputs, including rare ones; be checked by someone who knows the domain; and be free of personal data the task does not need, because a model can repeat what it was trained on.
Set aside part of the data and never train on it. That held out set shows whether the tuned model is genuinely better, not merely better at repeating its homework.
1. Define the task and the measure. Write down what a good output looks like and how you will score it.
2. Build the evaluation set first. Collect real inputs with approved answers, including cases the model should decline. OpenAI's guidance is blunt on this point: set up evaluations before investing in fine tuning.
3. Try prompting and retrieval first. Many projects stop here, and that is a good outcome.
4. Choose the base model. Compare candidate models on your evaluation set, along with licence, speed, cost and where they can run.
5. Prepare the training data. Collect, clean, check and format the examples. This is usually the longest step.
6. Train. Start with LoRA or QLoRA and modest settings, and watch the held out score as training runs, so you stop before the model starts memorising.
7. Evaluate before and after. Compare the tuned model with the base model on the same evaluation set, on your task and on the general abilities you still need, and test deliberate attempts to misuse it.
8. Deploy and monitor. Serve the model where your data rules require, review a sample of outputs regularly, and feed failures into the next round of training data.
There is no single best LLM for fine tuning. The best base model is the smallest one that reaches your quality bar on your own evaluation set, carries a licence that permits your use, and can run where your data must stay.
Open models, whose weights you can download, give the most control: you keep the tuned weights, run them on your own servers and decide when to upgrade. Leading open families from Meta, Mistral and Google come in many sizes. Licences differ, and some restrict certain commercial uses, so read the licence before the benchmark.
Closed models, available only through a vendor's service, can also be fine tuned on some platforms. Setup is simpler, but your training data goes to the vendor, the tuned model lives on the vendor's platform, and when the vendor retires the base model, your tuned version usually goes with it.
Start small. A model with a few billion parameters, tuned well, often handles a narrow task as well as a much larger general model. Move up only when your evaluation set shows you must.
The training run is usually the smallest part of the cost. With LoRA or QLoRA, a focused task can often be tuned on a single GPU: the QLoRA authors trained their best chatbot model in 24 hours on one. The larger costs sit around the run.
Data preparation is usually the largest cost, because experts must write, check or correct the examples. Budget for it first.
Evaluation is the cost most often skipped. Building a test set and running it after every change takes steady effort, and it makes the rest of the spend trustworthy.
Compute covers the training runs. Expect several, because the first settings are rarely the best.
Hosting is a running cost for as long as the model is in use: your own GPU servers, rented cloud GPUs, or a vendor's usage charge, which on some platforms is higher for a tuned model than for its base.
Retuning is the cost nobody budgets. When a better base model arrives or your task changes, you train again. A clean, versioned dataset and automated evaluation make that routine.
In short: modest for compute, significant for people, and ongoing for hosting and upkeep.
Fine tuning turns your data into part of the model, so every legal team should ask where the training data goes and where the tuned model runs. With a closed model, both sit with the vendor. With an open model, both can stay on servers you control, including networks with no internet connection at all.
For organisations handling personal data, that difference is material. India's Digital Personal Data Protection Act and the EU's GDPR both expect personal data to be processed for a clear purpose and kept no longer than needed. A model can reproduce fragments of its training data, so the safest rule is simple: remove personal data the task does not need before training, and keep the model wherever that data would be allowed to live.
At Aptibit, we fine tune language models for organisations that need consistent expertise in their field, predictable costs at volume, complete data privacy, or answers in a specific style and format. We work with your team to gather and prepare the material the model will learn from, closing gaps and removing noise. We compare leading open models on fit for your domain, speed, cost and licensing. We train with the most efficient approach for your budget and timeline, then test against your real scenarios and deliberate attempts to misuse the model, round after round, until it meets your targets.
We deploy with monitoring and records in place, in the cloud or on your own servers, including fully air gapped networks, with safeguards that keep the model on topic and protect personal data. A focused fine tune typically takes two to three weeks. You meet the engineers who will do the work before you sign, and the models, code and data pipelines belong to you, under an IP assignment signed before work begins. All of it is senior engineering at India cost levels, well below US and Western European rates.
And if your evaluation shows that a better prompt or RAG solves the problem, we will tell you so before you spend the budget.
LLM fine tuning earns its place when the problem is behaviour: a fixed output format, a house style, the vocabulary of a specialised field, or a narrow classification or extraction task run at high volume. It shines when a small tuned model can replace a large general one, and when data rules mean the model must live inside your network.
It disappoints when the real need is knowledge, especially facts that change, which suit retrieval; when answers must cite their sources; when there is no evaluation set, so nobody can show the tuned model is better; when the training data is inconsistent, unrepresentative or unchecked; and when nobody plans for retuning as base models and business needs move on.
Fine tuning is further training of a large language model that has already learned from general text, using a smaller set of examples from one task or domain. The model keeps its general language ability and learns a specific behaviour, such as a fixed format, a house style, specialist vocabulary or a narrow classification job. It costs far less than training a model from scratch.
Supervised fine tuning trains a language model on pairs of inputs and ideal outputs, written or approved by people. The model learns by imitation to produce answers like the examples. It is the most common form of LLM fine tuning for business tasks, used to teach formats, tone, domain vocabulary and narrow jobs such as extraction or routing. The quality and consistency of the examples matter more than their number.
Instruction tuning is supervised fine tuning on a broad mix of tasks, each written as a plain instruction, such as summarise this letter or list the risks in this contract. It turns a raw pretrained model, which only continues text, into one that follows requests. Model builders do it once. The instruct or chat versions of open models have already been instruction tuned, so businesses rarely need to repeat it.
The original ChatGPT was fine tuned in two stages, following the method OpenAI published in its 2022 InstructGPT paper. First came supervised fine tuning on demonstrations of good answers written by people. Then came reinforcement learning from human feedback: people ranked the model's answers, a reward model learned from those rankings, and that reward model guided further training. ChatGPT applied the same methods to conversation data.
No single model is best. The right base model is the smallest one that reaches your quality bar on your own test set, with a licence that permits your use and the ability to run where your data must stay. Open models give the most control, because you keep the tuned weights and can host them yourself. Start small, and move up only if testing shows you must.
The training compute is usually the smallest part. With efficient methods such as LoRA or QLoRA, a focused task can often be tuned on a single GPU. The larger costs are preparing and checking the training data, building and running evaluations, hosting the tuned model for as long as you use it, and retuning when the base model or your task changes. Budget for people before compute.
LoRA, short for low rank adaptation, fine tunes a language model by freezing its original settings and training a small set of extra ones alongside them. The 2021 LoRA paper reported 10,000 times fewer trainable parameters and a third of the GPU memory compared with fully fine tuning GPT-3, with matching or better quality. QLoRA adds compression, so large models can be tuned on a single GPU.
Use retrieval augmented generation when the model needs knowledge, especially facts that change or answers that must cite sources. Use fine tuning when the model needs a behaviour, such as a fixed format, a tone, a vocabulary or a narrow skill. Many strong systems combine the two: retrieval supplies the facts, and fine tuning shapes how the model answers. Try prompting and retrieval first, then fine tune what they cannot fix.
Yes. Aptibit prepares training data with your team, compares leading open models on fit for your domain, speed, cost and licensing, fine tunes with the most efficient approach for your budget, and tests against your real scenarios before deploying in the cloud or on your own servers, including air gapped networks. A focused fine tune typically takes two to three weeks, and the models, code and data pipelines belong to you.