📝 Training, Validation & Test Sets
Description
- A dataset is split into separate, non-overlapping subsets so the model can be trained, tuned, and evaluated without measuring it on the same examples used to make decisions.
| Dataset | Purpose | What it affects |
|---|---|---|
| Training set | Fit the model | Tunes learnable parameters, such as weights and biases |
| Validation set | Compare configurations | Tunes hyperparameters, such as learning rate, regularization, model depth, and training duration |
| Test set | Final evaluation | Estimates generalization; it must not tune parameters or hyperparameters |
Training set → tune model parameters
Validation set → tune hyperparameters and select the model
Test set → evaluate the finished model only
Key points
- Parameters are learned directly from training examples. Hyperparameters are choices made around training, such as optimizer settings, architecture, thresholds, and early-stopping rules.
- Reusing test results to choose a model, prompt, threshold, or hyperparameter turns the test set into another validation set. Keep it hidden until all decisions are locked.
- Fit preprocessing steps—including normalization, imputation, vocabulary selection, and feature selection—on the training set only, then apply the fitted transformation to validation and test data. This prevents data leakage.
- Use stratified, grouped, or time-based splits when random splitting would break class balance, place related examples in different subsets, or leak future information into the past.
- With limited data, use cross-validation inside the training portion for tuning, while preserving a separate test set for the final evaluation.
Reference
- Cross-validation: evaluating estimator performance (scikit-learn)
- Common pitfalls and recommended practices (scikit-learn)