📝 Post-Training

Description

< What is it? >

• [The idea]

  • Pre-training produces a base model → builds KNOWLEDGE
    It makes the model a next-token predictor that has absorbed an enormous text corpus. It can continue any document, but it does not answer questions, follow instructions, hold a format, reason step by step, or decline anything — those behaviours were never the objective.
  • Post-training produces an instruct model → teaches BEHAVIOR
    It is everything done to that base model afterwards to turn it into a usable assistant. Same architecture, same weights being updated by the same gradient descent — only the data and the objective change.
    • Historically post-training used a tiny fraction of pretraining's compute. That is no longer safe to assume: RL-heavy reasoning training has grown into a major cost centre of its own.
  • Fine-tuning creates a specialist model → creates EXPERTISE
    

• [Base vs instruct, concretely]

  prompt:  "What is the capital of France?"
  
  base model     →  "  What is the capital of Germany?
                       What is the capital of Italy?  …"      (it continues the document)
  
  instruct model →  "Paris."                                  (it answers)

  • The instruct model is not more knowledgeable. It has been taught which behaviour the prompt is asking for.

• [The stages]

  Stage	Data	Objective	What it buys
  Pretraining	Trillions of tokens, unlabelled	Next-token prediction	Knowledge, grammar, latent skills
  Mid-training	Curated domain / long-context data	Next-token prediction	Domain depth, longer context, code
  SFT	10³–10⁶ prompt→response pairs	Next-token prediction on responses	Instruction following, format, tone
  Preference optimization	Human/AI comparisons	RLHF, DPO and relatives	Helpfulness, style, refusal behaviour
  RLVR	Problems with checkable answers	RL against a verifier	Reasoning, math, code correctness
  Distillation	Teacher outputs	Match the teacher	A small model that behaves like a large one

  • The stages are cumulative and ordered — preference optimization on a model that has not been through SFT has almost nothing sensible to rank.

< Supervised fine-tuning (SFT) >

• Ordinary supervised learning: prompt→response pairs, cross-entropy loss, backprop. The one twist is loss masking — the loss is computed on the response tokens only. Training the model to predict the user's own prompt teaches it to imagine questions, not to answer them.
• Data quality dominates quantity. A few thousand carefully written, consistently formatted examples routinely beat hundreds of thousands of scraped ones; the LIMA result is the canonical demonstration. SFT is teaching behaviour, and behaviour is learned from consistency.
• The chat template is part of the model. Special tokens and role markers used in training must be reproduced byte-for-byte at inference. A mismatched template is the same class of bug as a mismatched scaler in data normalization — the model silently receives something it never saw in training.
• Catastrophic forgetting is real: narrow fine-tuning data degrades unrelated capabilities. Mix in general instruction data, keep learning rates small, and keep epochs few.

< Learning from preferences >

• [Why preferences and not more SFT]

  • For open-ended questions there is no single correct response to imitate, and SFT can only imitate. Humans are also far more reliable at comparing two answers than at writing the ideal one. So the signal is collected as pairs: given a prompt, which response is better?

• [RLHF — the classic three-step pipeline]

  1. Collect comparisons. Sample two or more responses per prompt; annotators pick the winner y_w over the loser y_l.

  2. Fit a reward model. A copy of the model with a scalar head, trained under the Bradley–Terry assumption that the probability of preferring one response is a sigmoid of the reward difference:

     $$P(y_w \succ y_l \mid x)=\sigma\!\left(r_\phi(x,y_w)-r_\phi(x,y_l)\right)$$

  3. Optimize the policy against that reward with PPO, held near the SFT model by a KL penalty:

     $$\max_{\pi_\theta}\; \mathbb{E}_{x\sim D,\; y\sim\pi_\theta(\cdot\mid x)}\left[r_\phi(x,y)\right] -\beta\,\mathrm{KL}\!\left[\pi_\theta(y\mid x)\,\|\,\pi_{\mathrm{ref}}(y\mid x)\right]$$
  • The KL term is the whole safety net. Without it the policy walks off to whatever gibberish maximizes an imperfect reward model — reward hacking, the defining failure of RLHF. β trades alignment against staying coherent, and is exactly a regularization strength: it pulls the policy back toward a reference rather than toward zero.

• [DPO — the same thing without the reward model]

  • Direct Preference Optimization rewrites the constrained RL problem in closed form, so the reward model and the RL loop both disappear and preference pairs become a single classification loss:

    $$\mathcal{L}_{\mathrm{DPO}}=-\,\mathbb{E}_{(x,y_w,y_l)\sim D} \left[\log\sigma\!\left( \beta\log\frac{\pi_\theta(y_w\mid x)}{\pi_{\mathrm{ref}}(y_w\mid x)} -\beta\log\frac{\pi_\theta(y_l\mid x)}{\pi_{\mathrm{ref}}(y_l\mid x)} \right)\right]$$

  • Far simpler and cheaper — two models in memory instead of four, no sampling loop. The cost is that it learns only from the fixed offline dataset, whereas online RL keeps generating fresh responses to be judged.

< RL with verifiable rewards (RLVR) >

• When the answer can be checked — a unit test passes, a math result matches, a program compiles — the reward needs no human and no learned reward model, so there is nothing to hack. That removes the ceiling on how long the model can be trained.
• This is the mechanism behind reasoning models: rewarded for reaching correct answers, the model learns to spend more tokens thinking before committing, and long chains of thought emerge from the incentive rather than from imitation.
• GRPO is the workhorse algorithm here — it samples a group of responses per prompt and uses their mean reward as the baseline, dropping PPO's separate value network.
• The limit is scope: it only applies where a verifier exists. Taste, tone and safety still need human or AI preferences.

Key points

< Full fine-tuning vs PEFT >

Method	Trainable params	Memory	Use when
Full fine-tuning	100%	Highest — weights + gradients + optimizer state	You have the compute and want maximum quality
LoRA	~0.1–1% (low-rank adapters)	Much lower; base weights frozen	The default for most teams. Adapters swap at serve time
QLoRA	Same as LoRA	Lowest — 4-bit frozen base	Single-GPU fine-tuning of a large model
Prompt / prefix tuning	Tiny	Minimal	Light task steering; weaker than LoRA in practice

• LoRA's real operational win is many variants, one base: dozens of adapters can be served against a single set of frozen weights.

< Prompt vs RAG vs post-train >

Need	Reach for	Why
Different wording, format, or a role	Prompt engineering	Instant, free, reversible. Always try first
The model lacks facts	RAG	Knowledge should be retrieved, not baked in — it changes
The model lacks a behaviour or a house style	SFT	Behaviour is what fine-tuning actually teaches
Rankable quality with no single right answer	Preference optimization	Comparisons carry signal that imitation cannot
Checkable correctness at high volume	RLVR	A verifier gives unlimited, unhackable reward
Lower latency/cost at similar quality	Distillation	Move the behaviour into a smaller model

• Fine-tuning to inject facts is the most common expensive mistake: it is a poor way to store knowledge, and the knowledge goes stale in the weights where you cannot update or cite it.

< Data is the product >

• Annotator agreement is the ceiling on reward-model quality. If two humans disagree on which response is better, no model can learn the difference — write the guidelines first.
• Synthetic data works, but only with filtering. Generate broadly, then verify, deduplicate and rank. Unfiltered model-generated data narrows diversity with every round.
• Check for contamination: benchmark items leaking into training data produce a model that scores well and behaves no better.

< Evaluating a post-trained model >

• Pairwise win rate against the previous checkpoint is the standard headline metric — usually via an LLM judge, which is fast and cheap but biased toward longer, more confident, and self-authored answers. Calibrate against human judgement periodically; never let a judge be the only signal.
• Track a capability suite alongside it. Alignment work can quietly cost accuracy elsewhere — the alignment tax — and a win-rate-only dashboard will not show it.
• Evaluate refusals in both directions. Over-refusal on benign requests is a real failure, not a safe default, and it is the predictable overshoot of harmlessness training.

< Failure modes >

Symptom	Cause	Fix
Sycophancy — agrees with whatever the user asserts	Annotators preferred agreeable answers	Preference data that rewards correct disagreement
Verbosity — long, padded, list-everything answers	Length correlates with human and judge preference	Length-control terms; penalize in the reward
Mode collapse — every answer sounds identical	Over-optimized against the reward; KL too weak	Raise β, fewer RL steps, more diverse prompts
Reward hacking — high reward, useless output	Reward model exploited off-distribution	Stronger KL, fresh preference data, cap training steps
Catastrophic forgetting	Narrow SFT data	Mix in general data, lower LR, fewer epochs
Format brittleness — breaks on a slightly different prompt	Templates too uniform in training	Vary phrasing and system prompts in the SFT set
Over-refusal	Harmlessness signal overshot	Add benign-but-adjacent examples that should be answered

< Practical hyperparameters >

• Learning rates are far smaller than pretraining: roughly 1e-5–2e-5 for full SFT, 1e-4–3e-4 for LoRA. Too large and the model forgets; too small and nothing moves.
• 1–3 epochs. SFT sets are small and memorize fast — a rising validation loss with a still-falling training loss is ordinary overfitting, and the response is early stopping, not a bigger run.
• AdamW with cosine decay and a short warmup is the standard recipe (see Gradient Descent).
• Read samples by hand every run. Post-training failures are behavioural, and loss curves do not show sycophancy.

Crash course

Pre-training, Post-training & Fine Tuning

What Pretraining, Post-Training, and Fine-Tuning Actually Mean

How LLM Post-Training Works

Reference

• Training language models to follow instructions with human feedback — InstructGPT, the SFT + RLHF recipe (Ouyang et al., 2022)
• Deep Reinforcement Learning from Human Preferences (Christiano et al., 2017)
• Constitutional AI: Harmlessness from AI Feedback — RLAIF (Bai et al., 2022)
• Direct Preference Optimization (Rafailov et al., 2023)
• LIMA: Less Is More for Alignment (Zhou et al., 2023)
• LoRA: Low-Rank Adaptation of Large Language Models (Hu et al., 2021) · QLoRA (Dettmers et al., 2023)
• DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via RL — GRPO and verifiable rewards (2025)
• Tulu 3: Pushing Frontiers in Open Language Model Post-Training — a fully documented open recipe (AI2, 2024)
• TRL — Transformer Reinforcement Learning — SFT, DPO, GRPO trainers (Hugging Face)