Train, Test and ValidationSummary
One page to revise from.
- A model's score on the data it learned from is too optimistic. A model that fits its training data perfectly but fails on new data is overfitting; one that cannot even fit the pattern is underfitting.
- The training set is where the model learns its parameters (slope, intercept, , ). It is the biggest share, usually 60–80 %.
- The validation set is used to choose hyperparameters — settings you pick before training, such as the learning rate, the threshold or the degree of a curve.
- The test set is used once, at the very end, for an honest score. Training or tuning on it is a data leak.
- Shuffle before splitting, or each set may see a different part of the world.
- Training error keeps falling as a model gets more flexible; validation error falls and then rises. Ship the model at the bottom of the validation curve.
Training
learn the parameters
60–80 %
Validation
choose the hyperparameters
10–20 %
Test
one final, honest score
10–20 %