torchtune is a native-PyTorch library for authoring, fine-tuning, and experimenting with large language models.
Hackable training recipes for SFT, knowledge distillation, DPO, PPO, GRPO, and quantization-aware training.
It provides modular model components, configurable training recipes, and YAML configuration files for workflows such
as full fine-tuning, LoRA, and QLoRA.
Familiar PyTorch: training logic uses standard PyTorch, making it easier to read, debug, and customize.
Reusable recipes: ready-made pipelines reduce boilerplate while modular components can be replaced when needed.
Efficient training: supports techniques such as LoRA, QLoRA, reduced precision, activation checkpointing, gradient
accumulation, and distributed training with FSDP2.
Configurable runs: YAML files separate model, dataset, checkpoint, and hyperparameter choices from recipe code.
Ecosystem integration: includes checkpoint-conversion utilities and integrations with Hugging Face Datasets and
EleutherAI's LM Evaluation Harness.
Correctness focus: components and recipes are tested for numerical and benchmark parity with reference
implementations.