Train, Test and ValidationThe Validation Set

Some numbers a model learns by itself from the training set. Others you must choose before training starts: these are called hyperparameters. You have already met two — the learning rate in Linear Regression and the threshold in Logistic Regression. Which of these is not a hyperparameter?

The slope is learned from the training data. Hyperparameters are chosen — and to choose them fairly we need data the model did not train on. That is the validation set's job.

Choose the Degree

A farmer wants to know how much fertiliser (bags per hectare) gives the best wheat harvest (tonnes per hectare): too little starves the crop, too much burns its roots. We fit a polynomial of degree 1 to 7 to the training points only, and score each curve on the validation points it never saw. Try different degrees, then ship the best one.

Training error always falls as the curve is allowed to bend more — a degree-7 curve almost touches every training point. Validation error falls at first and then rises: the bendy curves are chasing noise. The bottom of that U is the model to ship. A curve that is too simple is said to underfit; one that is too bendy overfits.