📝 Fine-Tuning

Description

< What is it? >

• Fine-tuning continues training from a pretrained checkpoint instead of starting with random weights. The original model supplies broad representations and capabilities; a smaller, targeted dataset adapts those capabilities to a task, domain, format, or behavior.

• Transfer learning: fine-tuning reuses learned representations and adapts them to a new task or domain. Starting from pretrained parameters $\theta_0$, supervised fine-tuning (SFT) solves:

  $$\theta^* = \arg\min_\theta \frac{1}{N}\sum_{i=1}^{N}\mathcal{L}\bigl(f_\theta(x_i), y_i\bigr) + \lambda\,R(\theta,\theta_0)$$

  The first term learns from the new examples. The optional regularizer keeps the new parameters near the useful pretrained solution, reducing overfitting and catastrophic forgetting.

• Fine-tuning changes model parameters, unlike prompt engineering, which changes the instructions, or RAG, which supplies knowledge at inference time. See Google’s overview of fine-tuning, distillation, and prompt engineering.

Key points

< Post-training, objectives, and parameter updates >

• Training stage: post-training adapts a model after its initial pretraining, commonly to improve instruction following, task performance, or alignment with preferences. Fine-tuning from an existing checkpoint is used within this stage. Continued pretraining retains a language-modeling objective and is often described separately as domain adaptation or mid-training. These stage boundaries vary between training pipelines.

• Two separate choices: the training objective specifies the signal to learn from, such as example answers, response preferences, or rewards. The parameter-update method specifies which parameters can change, such as all model weights or only LoRA matrices. SFT can therefore use either full fine-tuning or LoRA; Direct Preference Optimization (DPO) and reinforcement learning can also use parameter-efficient updates.

• LoRA within post-training: using Low-Rank Adaptation (LoRA) to adapt a pretrained model is a form of post-training. LoRA defines how weight updates are represented; it does not determine the labels, loss, or desired behavior. See the LoRA conceptual guide.

• Example — combining objectives and update methods: one possible training sequence is:

  Pretrained checkpoint
      → SFT with QLoRA on question–answer examples
      → Optional DPO with LoRA updates on preferred/rejected answers
      → Evaluate and deploy

  The DPO stage starts from the SFT-adapted model. Quantized Low-Rank Adaptation (QLoRA) uses a quantized base with LoRA updates, so the sequence need not change the adapter architecture between stages. The objectives and stages are selected for the task; this sequence is not mandatory. See Post-Training for the broader pipeline.

< LoRA in one equation >

• Low-Rank Adaptation (LoRA) freezes a weight matrix $W_0$ and learns a low-rank update instead of another full matrix:

  $$W_{\text{effective}} = W_0 + \Delta W = W_0 + \frac{\alpha}{r}BA$$

  where $A \in \mathbb{R}^{r\times d_{in}}$, $B \in \mathbb{R}^{d_{out}\times r}$, and the rank $r$ is much smaller than either model dimension. Only $A$ and $B$ receive gradients.

• The rank controls adapter capacity; $\alpha$ controls update scale. Target modules determine where the update is inserted. These choices must be tuned together rather than copied blindly from a different model.

• At inference, the adapter can remain separate for quick swapping or be merged into the base weights for a standalone checkpoint. QLoRA keeps the frozen base quantized during training while the adapter computations remain at higher precision.

< End-to-end workflow >

• Write the contract. Define the inputs, expected outputs, unacceptable behavior, latency budget, and primary metric before collecting data.
• Build a baseline. Evaluate the unmodified model with a strong prompt. Compare the adapted model against this baseline.
• Curate and split data. Deduplicate first, then split by user, source, document, or time so near-identical examples cannot leak across train and test sets.
• Choose the checkpoint and method. Match the model license, tokenizer, context length, architecture, and serving environment. Choose the objective from available supervision and the update method from measured constraints.
• Train conservatively. Use a small learning rate, few epochs, regular evaluation, and checkpoints. Sweep one or two important knobs instead of trusting a single run.
• Evaluate broadly. Measure the target task, unrelated capabilities, edge cases, and production constraints against both the base and prompt-only baselines.
• Package the inference contract. Version the base checkpoint, adapter or weights, tokenizer, chat template, preprocessing, decoding settings, and evaluation report together.
• Deploy gradually. Canary the new version, monitor live distributions and regressions, and keep an immediate rollback path.

< Dataset design >

• SFT example: one conceptual prompt–response training record looks like this; the exact schema depends on the framework:

  {
    "messages": [
      {"role": "system", "content": "Answer as a concise support specialist."},
      {"role": "user", "content": "Can I change my delivery address?"},
      {"role": "assistant", "content": "You can change it before the order ships. Open the order and select Edit address."}
    ]
  }

• Match production. Training examples should use the same input shape, role markers, special tokens, preprocessing, and output contract that inference will use. A template mismatch is data-distribution shift.

• Loss masking: response-only or assistant-only loss focuses instruction SFT on the desired answers. Some recipes also train on prompt tokens. Configure masking deliberately and verify that the intended target tokens contribute to the loss. See the SFT trainer documentation.

• Quality beats raw count. Remove contradictions, boilerplate, duplicates, impossible examples, and label errors. Consistent examples teach a clear decision boundary; inconsistent examples set a ceiling on model quality.

• Cover boundaries, not only happy paths. Include rare classes, ambiguous inputs, valid refusals, malformed inputs, and examples that look similar but require different answers.

• Protect the test set. Keep it untouched until model choices are made. If repeated decisions use the test score, it has silently become another validation set.

• Track data provenance, consent, licensing, and sensitive information. Fine-tuned models can memorize rare or repeated strings even when aggregate evaluation looks healthy.

< Training recipe >

• Begin with the smallest sensible learning rate and sweep logarithmically. Fine-tuning usually needs much smaller steps than training from scratch because the starting checkpoint is already useful.
• Small behavioral datasets often need only a few epochs. Select the checkpoint at the validation optimum; a falling training loss with a rising validation loss is ordinary overfitting, not a reason to train longer.
• Keep effective batch size explicit: micro-batch × gradient accumulation × number of devices. Changing any one of them can change optimization even when the code still runs.
• Set sequence length from the real data distribution. Truncation can remove the answer or label, while excessive padding wastes memory. Packing short independent samples improves utilization but must not let attention cross sample boundaries.
• Use warmup, gradient clipping, checkpointing, and a validation schedule appropriate to dataset size. Tune learning rate, batch size, epochs, rank, and dropout through Hyperparameter Tuning, not intuition alone.
• Read model outputs from every run. For SFT, token-prediction loss alone does not establish whether an answer is truthful, safe, concise, or useful; other objectives require their own task-level checks.

< Evaluation >

• Evaluation layers:

  Layer	What to measure	Example
  Target task	The capability being adapted	Accuracy, F1, exact match, pass rate, or pairwise win rate
  Regression suite	Capabilities that should remain unchanged	General reasoning, language quality, calibration, or prior product tasks
  Robustness and safety	Behavior near the data boundary	Paraphrases, adversarial inputs, refusal precision, privacy probes
  Production	Operational quality	Latency, memory, throughput, cost, and output-validity rate

• Compare at least three systems: the original model, the original model with the best prompt, and the fine-tuned model with its production prompt. Report uncertainty or repeated-run variance when differences are small.

• Use human review for subjective behavior. Automated or model-based judges are useful for scale, but calibrate them against humans and check for length, style, and self-preference biases.

< Deployment checklist >

• Pin the exact base checkpoint and tokenizer revision.
• Store the preprocessing or chat template with the model artifact.
• Record training-data version, random seed, optimizer, scheduler, precision, and all adapter settings.
• Confirm the serving runtime supports the chosen quantization and adapter targets.
• Run the same evaluation suite on the packaged artifact, not only the in-memory training checkpoint.
• Canary against live traffic, monitor by data slice, and retain the previous model for rollback.

Comparison

< Training objectives and supervision >

• Training signals and purposes: these approaches describe different aspects of training and can be combined. The table focuses on language-model adaptation.

  Technique	Training data or signal	What the model learns	Example
  Supervised Fine-Tuning (SFT)	Inputs paired with desired outputs	Reproduce target responses, commonly using token-level cross-entropy	Answer support questions in a specified format
  Direct Preference Optimization (DPO)	A prompt with a preferred and rejected response	Optimize response preferences directly, without a separately trained reward model	Prefer a correct, relevant answer over an incorrect or unhelpful one
  Reinforcement Learning from Human Feedback (RLHF)	Human preferences, commonly used to fit a reward model	Improve generated responses by optimizing the learned reward	Improve helpfulness using a reward model and Proximal Policy Optimization (PPO)
  Reinforcement Learning with Verifiable Rewards (RLVR)	Automatically checkable outcomes	Increase reward from verifiers or task environments	Solve math problems or produce code that passes tests
  Knowledge distillation	Teacher responses, labels, or probability distributions	Transfer a teacher's behavior to a student, often a smaller model	Train a compact assistant from a stronger model's answers
  Continued pretraining / domain-adaptive pretraining	Additional raw domain text, with prediction targets derived from the text	Adapt language representations to a domain using a self-supervised objective	Continue next-token training on a technical corpus before instruction tuning

  Sources: SFT, DPO, the classic RLHF pipeline, GRPO and reward functions, distillation, and domain-adaptive pretraining.

• Feedback sources and algorithms: RLHF identifies human feedback as the source of supervision, while RLVR identifies verifiable rewards. PPO and Group Relative Policy Optimization (GRPO) are optimization algorithms. GRPO can use verifiable rewards or other reward functions; it is not synonymous with RLVR. DPO uses a direct preference loss rather than the classic reward-model-plus-RL loop.

• Overlapping categories: training on a teacher's generated answers can be both SFT and distillation. Likewise, preference labels can come from human reviewers or AI evaluators. The data source, training objective, and parameter-update method should be specified separately when describing a recipe.

< Parameter-update methods >

• Trainable parameters and tradeoffs: the following choices determine the scope of adaptation, independently of the training signal above.

  Method	Parameters updated	Main advantage	Main tradeoff
  Head-only / linear probing	A task-specific output head; backbone frozen	Low training cost and unchanged backbone representations	Limited adaptation of the feature representation
  Partial fine-tuning	Selected layers or blocks	Adapts more of the model than head-only training	Layer selection becomes another hyperparameter
  Full fine-tuning	All model parameters	Broad freedom to adapt	Substantial gradient, optimizer-state, and checkpoint storage costs
  Low-Rank Adaptation (LoRA)	Small low-rank update matrices; base weights frozen	Fewer trainable parameters and compact task-specific checkpoints	Rank and target modules constrain the update; serving must load or merge adapters
  QLoRA	LoRA parameters over a frozen, typically 4-bit quantized base	Further reduces base-model memory during training	Requires compatible quantization and training kernels
  Adapter tuning	Small modules inserted into the network; base weights frozen	Reuses one base model across compact task-specific modules	Added modules may increase inference work
  Prompt tuning / prefix tuning	Learned continuous prompt or prefix vectors; base weights frozen	Trains a small set of task-specific parameters	Restricted adaptation capacity; learned vectors must be included during inference

• Parameter-efficient fine-tuning: LoRA, adapter tuning, and soft-prompt methods belong to Parameter-Efficient Fine-Tuning (PEFT). Prompt tuning learns numerical vectors through training; ordinary prompt engineering edits text instructions without gradient updates. See the PEFT documentation, QLoRA paper, and soft-prompt guide.

• Selecting a method: evaluate the least costly method that can express the required change. Head-only training can be sufficient when pretrained features already separate the target classes. LoRA is a practical candidate for large-model adaptation; full fine-tuning may be useful when it produces a measured quality improvement that justifies its resource cost.

< Fine-tuning vs. other adaptation approaches >

• Match the approach to the need:

  Need	First choice	Why
  Better instructions, wording, or output format	Prompt engineering	Fast, reversible, and requires no training
  Current, private, or frequently changing facts	RAG	Keeps knowledge outside the weights and makes sources retrievable
  Stable behavior repeated across many requests	Fine-tuning	Moves the behavior into the model instead of every prompt
  A classifier or predictor for a labeled task	Fine-tuning	Reuses pretrained features and learns a task-specific decision boundary
  Deeper fluency in a specialized corpus	Continued pretraining, optionally followed by SFT	Learns domain patterns before teaching instruction behavior
  Lower inference cost at similar quality	Distillation	Trains a smaller student to imitate a stronger teacher

• Establish a baseline: evaluate prompt-only behavior before adding training. Fine-tuning requires representative examples, a held-out evaluation set, and a stable definition of success.

Troubleshoot

< Common failure modes >

• Diagnosis and remedies:

  Symptom	Likely cause	Fix
  Training improves while validation worsens	Dataset too small, too many epochs, or update too large	Stop earlier, lower the learning rate, add data or regularization
  Target task improves but general ability drops	Catastrophic forgetting	Use PEFT, lower update size, or mix representative general data
  Correct format only for familiar wording	Training templates are too uniform	Vary prompts and evaluate paraphrases
  Great offline score, poor production quality	Leakage or train–serve skew	Rebuild grouped splits and reproduce the inference pipeline in evaluation
  Rare cases fail consistently	Class or scenario imbalance	Resample, reweight, and add boundary examples
  Model repeats private or exact training text	Memorization and duplication	Deduplicate, remove sensitive data, reduce epochs, and run extraction probes
  Adapter loads but quality collapses	Wrong base checkpoint, tokenizer, template, or target modules	Version and validate the complete model bundle
  Bigger dataset makes the model worse	Low-quality synthetic or contradictory examples	Filter, verify, diversify, and measure data slices separately

Video Tutorial

How LLM fine-tuning actually works

Related ideas

• Post-training for Agentic RAG applies these techniques to retrieval tools and grounded answer generation.
• Post-Training explains instruction tuning, preference optimization, and reinforcement-learning stages.
• Parameter-Efficient Fine-Tuning (PEFT) covers methods that train a small subset of parameters.
• Low-Rank Adaptation (LoRA) explains low-rank updates in detail.
• Hugging Face TRL and PEFT covers training tools for objectives and parameter-efficient updates.

Reference

• Google Machine Learning Crash Course: Fine-tuning, distillation, and prompt engineering
• Hugging Face TRL: SFT Trainer
• Hugging Face TRL: DPO Trainer
• Hugging Face TRL: GRPO Trainer
• Hugging Face TRL: Distillation Trainer
• Training language models to follow instructions with human feedback
• Don't Stop Pretraining: Adapt Language Models to Domains and Tasks
• How transferable are features in deep neural networks? — layer transferability (Yosinski et al., 2014)
• Universal Language Model Fine-tuning for Text Classification — ULMFiT (Howard & Ruder, 2018)
• Parameter-Efficient Transfer Learning for NLP — adapter modules (Houlsby et al., 2019)
• LoRA: Low-Rank Adaptation of Large Language Models (Hu et al., 2021)
• QLoRA: Efficient Finetuning of Quantized LLMs (Dettmers et al., 2023)
• Hugging Face PEFT documentation — parameter-efficient fine-tuning methods and tooling
• Hugging Face fine-tuning guide — trainer-based workflow