Underfitting and OverfittingSummary
One page to revise from.
- Underfitting: the model is too simple to follow the pattern — high bias. It does badly even on its training data.
- Overfitting: the model is so flexible that it learns the noise of its training data — high variance. It does well on training data and badly on new data.
- Training the same kind of model on different samples is like throwing darts: bias is how far the average lands from the truth, variance is how much the results scatter.
- The expected error on new data is bias² + variance + noise. Making a model more flexible lowers bias and raises variance; the best model balances them.
- Many hyperparameters are bias–variance dials: the degree of a curve, the span of LOESS, the k of k-nearest neighbours.
- LOESS builds a smooth curve from many weighted local straight lines; a wide span underfits, a narrow span overfits.
- KNN predicts by a majority vote of the k nearest examples; k = 1 overfits, a very large k underfits.
- Judge the balance on held-out or cross-validated data, never on the training data.
Bias–variance decomposition