📝 Fine-Tuning

Description

< What is it? >

• Fine-tuning continues training from a pretrained checkpoint instead of starting with random weights. The original model supplies broad representations and capabilities; a smaller, targeted dataset adapts those capabilities to a task, domain, format, or behavior.

• This is transfer learning: reuse what the model already learned, then make smaller updates for a narrower goal. Starting from pretrained parameters $\theta_0$, supervised fine-tuning solves:

  $$\theta^* = \arg\min_\theta \frac{1}{N}\sum_{i=1}^{N}\mathcal{L}\bigl(f_\theta(x_i), y_i\bigr) + \lambda\,R(\theta,\theta_0)$$

  The first term learns from the new examples. The optional regularizer keeps the new parameters near the useful pretrained solution, reducing overfitting and catastrophic forgetting.

• Fine-tuning changes model parameters, unlike prompt engineering, which changes the instructions, or RAG, which supplies knowledge at inference time.

< Fine-tuning vs post-training >

• Fine-tuning is a training mechanism: update all weights, selected layers, or a small set of adapter parameters from an existing checkpoint.
• Post-training is a broader stage after pretraining. It can contain supervised fine-tuning (SFT), preference optimization, reinforcement learning, distillation, and safety training. See Post-Training for the full pipeline.
• Continued pretraining uses more unlabeled next-token data to deepen domain knowledge. SFT uses labeled input→output examples to teach a task or behavior. They may use the same optimizer, but their datasets and goals are different.

< When should you use it? >

Need	First choice	Why
Better instructions, wording, or output format	Prompt engineering	Fast, reversible, and requires no training
Current, private, or frequently changing facts	RAG	Keeps knowledge outside the weights and makes sources retrievable
Stable behavior repeated across many requests	Fine-tuning	Moves the behavior into the model instead of every prompt
A classifier or predictor for a labeled task	Fine-tuning	Reuses pretrained features and learns a task-specific decision boundary
Deeper fluency in a specialized corpus	Continued pretraining, then SFT	Learns domain patterns before teaching instruction behavior
Lower inference cost at similar quality	Distillation	Trains a smaller student to imitate a stronger teacher

• Fine-tune only after a prompt-only baseline is not good enough. You should also have representative examples, a held-out evaluation set, and a stable definition of success. Otherwise training makes the failure more expensive without making it more measurable.

Key points

< What can be trained? >

Method	Parameters updated	Main advantage	Main tradeoff
Head-only / linear probing	A new output head	Cheapest and least likely to forget	Limited adaptation
Partial fine-tuning	Last layers or selected blocks	More capacity than a new head	Layer choice becomes another hyperparameter
Full fine-tuning	Every model weight	Maximum freedom to adapt	Highest memory, storage, and forgetting risk
Adapters / LoRA	Small trainable modules	One frozen base can support many small variants	Adds adapter configuration and serving complexity
QLoRA	LoRA adapters over a quantized frozen base	Greatly reduces training memory	Quantization and kernels add implementation constraints

• Start with the least expensive method that can express the change. Head-only training is often enough for standard classification; LoRA is a strong starting point for adapting a large language model; full fine-tuning is justified when adapters leave a measured quality gap and the infrastructure can support it.

< LoRA in one equation >

• Low-Rank Adaptation freezes a weight matrix $W_0$ and learns a low-rank update instead of another full matrix:

  $$W_{\text{effective}} = W_0 + \Delta W = W_0 + \frac{\alpha}{r}BA$$

  where $A \in \mathbb{R}^{r\times d_{in}}$, $B \in \mathbb{R}^{d_{out}\times r}$, and the rank $r$ is much smaller than either model dimension. Only $A$ and $B$ receive gradients.

• The rank controls adapter capacity; $\alpha$ controls update scale. Target modules determine where the update is inserted. These choices must be tuned together rather than copied blindly from a different model.

• At inference, the adapter can remain separate for quick swapping or be merged into the base weights for a standalone checkpoint. QLoRA keeps the frozen base quantized during training while the adapter computations remain at higher precision.

< End-to-end workflow >

1. Write the contract. Define the inputs, expected outputs, unacceptable behavior, latency budget, and primary metric before collecting data.
2. Build a baseline. Evaluate the unmodified model with a strong prompt. Fine-tuning must beat this baseline, not an intentionally weak one.
3. Curate and split data. Deduplicate first, then split by user, source, document, or time so near-identical examples cannot leak across train and test sets.
4. Choose the checkpoint and method. Match the model license, tokenizer, context length, architecture, and serving environment. Choose full, partial, or parameter-efficient tuning from measured constraints.
5. Train conservatively. Use a small learning rate, few epochs, regular evaluation, and checkpoints. Sweep one or two important knobs instead of trusting a single run.
6. Evaluate broadly. Measure the target task, unrelated capabilities, edge cases, and production constraints against both the base and prompt-only baselines.
7. Package the inference contract. Version the base checkpoint, adapter or weights, tokenizer, chat template, preprocessing, decoding settings, and evaluation report together.
8. Deploy gradually. Canary the new version, monitor live distributions and regressions, and keep an immediate rollback path.

< Dataset design >

• For an instruction-tuned language model, one conceptual training record looks like this (the exact schema depends on the framework):

  {
    "messages": [
      {"role": "system", "content": "Answer as a concise support specialist."},
      {"role": "user", "content": "Can I change my delivery address?"},
      {"role": "assistant", "content": "You can change it before the order ships. Open the order and select Edit address."}
    ]
  }

• Match production. Training examples should use the same input shape, role markers, special tokens, preprocessing, and output contract that inference will use. A template mismatch is data-distribution shift.

• Mask the right loss. In causal-LM SFT, the prompt provides context and the response is normally the target. If prompt tokens contribute to the loss, the model spends capacity predicting user text instead of learning how to respond.

• Quality beats raw count. Remove contradictions, boilerplate, duplicates, impossible examples, and label errors. Consistent examples teach a clear decision boundary; inconsistent examples set a ceiling on model quality.

• Cover boundaries, not only happy paths. Include rare classes, ambiguous inputs, valid refusals, malformed inputs, and examples that look similar but require different answers.

• Protect the test set. Keep it untouched until model choices are made. If repeated decisions use the test score, it has silently become another validation set.

• Track data provenance, consent, licensing, and sensitive information. Fine-tuned models can memorize rare or repeated strings even when aggregate evaluation looks healthy.

< Training recipe >

• Begin with the smallest sensible learning rate and sweep logarithmically. Fine-tuning usually needs much smaller steps than training from scratch because the starting checkpoint is already useful.
• Small behavioral datasets often need only a few epochs. Select the checkpoint at the validation optimum; a falling training loss with a rising validation loss is ordinary overfitting, not a reason to train longer.
• Keep effective batch size explicit: micro-batch × gradient accumulation × number of devices. Changing any one of them can change optimization even when the code still runs.
• Set sequence length from the real data distribution. Truncation can remove the answer or label, while excessive padding wastes memory. Packing short independent samples improves utilization but must not let attention cross sample boundaries.
• Use warmup, gradient clipping, checkpointing, and a validation schedule appropriate to dataset size. Tune learning rate, batch size, epochs, rank, and dropout through Hyperparameter Tuning, not intuition alone.
• Read model outputs from every run. Loss measures token prediction, not whether an answer is truthful, safe, concise, or useful.

< Evaluation >

Layer	What to measure	Example
Target task	The capability being adapted	Accuracy, F1, exact match, pass rate, or pairwise win rate
Regression suite	Capabilities that should remain unchanged	General reasoning, language quality, calibration, or prior product tasks
Robustness and safety	Behavior near the data boundary	Paraphrases, adversarial inputs, refusal precision, privacy probes
Production	Operational quality	Latency, memory, throughput, cost, and output-validity rate

• Compare at least three systems: the original model, the original model with the best prompt, and the fine-tuned model with its production prompt. Report uncertainty or repeated-run variance when differences are small.
• Use human review for subjective behavior. Automated or model-based judges are useful for scale, but calibrate them against humans and check for length, style, and self-preference biases.

< Common failure modes >

Symptom	Likely cause	Fix
Training improves while validation worsens	Dataset too small, too many epochs, or update too large	Stop earlier, lower the learning rate, add data or regularization
Target task improves but general ability drops	Catastrophic forgetting	Use PEFT, lower update size, or mix representative general data
Correct format only for familiar wording	Training templates are too uniform	Vary prompts and evaluate paraphrases
Great offline score, poor production quality	Leakage or train–serve skew	Rebuild grouped splits and reproduce the inference pipeline in evaluation
Rare cases fail consistently	Class or scenario imbalance	Resample, reweight, and add boundary examples
Model repeats private or exact training text	Memorization and duplication	Deduplicate, remove sensitive data, reduce epochs, and run extraction probes
Adapter loads but quality collapses	Wrong base checkpoint, tokenizer, template, or target modules	Version and validate the complete model bundle
Bigger dataset makes the model worse	Low-quality synthetic or contradictory examples	Filter, verify, diversify, and measure data slices separately

< Deployment checklist >

• Pin the exact base checkpoint and tokenizer revision.
• Store the preprocessing or chat template with the model artifact.
• Record training-data version, random seed, optimizer, scheduler, precision, and all adapter settings.
• Confirm the serving runtime supports the chosen quantization and adapter targets.
• Run the same evaluation suite on the packaged artifact, not only the in-memory training checkpoint.
• Canary against live traffic, monitor by data slice, and retain the previous model for rollback.

Crash course

How LLM fine-tuning actually works (LoRA, serverless, one-click deploy)

Reference

• How transferable are features in deep neural networks? — layer transferability (Yosinski et al., 2014)
• Universal Language Model Fine-tuning for Text Classification — ULMFiT (Howard & Ruder, 2018)
• Parameter-Efficient Transfer Learning for NLP — adapter modules (Houlsby et al., 2019)
• LoRA: Low-Rank Adaptation of Large Language Models (Hu et al., 2021)
• QLoRA: Efficient Finetuning of Quantized LLMs (Dettmers et al., 2023)
• Hugging Face PEFT documentation — parameter-efficient fine-tuning methods and tooling
• Hugging Face fine-tuning guide — trainer-based workflow