Testing Your AISummary
One page to revise from.
- A score from a single validation split depends on the luck of which rows were held back, and those rows never help the model learn.
- K-fold cross-validation cuts the data into k folds. Each fold is held out once while the model trains on the rest; the average of the k scores is the cross-validation score.
- Every row is used for training k − 1 times and for validation once. The price is k trainings instead of one.
- Leave-one-out (LOOCV) is k-fold with : one row per fold, n models. Fine for small data, far too slow for big data.
- Small k gives pessimistic scores (each model sees little data); large k is costly. k = 5 or 10 is the usual choice.
- Cross-validation is how hyperparameters such as the degree of a curve are chosen. The test set still stays locked until the very end.
k-fold
k models; each trains on of the data
score = average of the k fold scores
Leave-one-out
: n models
each validation fold is one row