Train, Test and ValidationWhy Do We Split Data?
Board exams are three months away. Kabir's coaching centre hands out one "guaranteed" sample paper, and Kabir learns all 30 of its answers by heart. In the mock test — the very same paper — he scores 100 %.
Does that 100 % tell you how Kabir will do in the real board exam?
To find out whether Kabir understands, you have to test him on questions he has never seen. Machine-learning models are exactly the same — and in this chapter you will see why.
The Memoriser and the Learner
Two models were trained on the same eight students: hours studied in a week, and marks in that week's quiz out of 10 (the blue dots). Model A is a bendy curve that passes exactly through every one of them. Model B is a plain straight line. Look at both, then bring in eight new students that neither model has seen.
Model A had zero error on the students it learned from — yet for new students it predicts marks below 0 and above 10. It memorised every wobble of its eight students, including the luck. This is called overfitting. Model B made small errors everywhere, and equally small errors on new students: it learned the pattern, so it generalises.
Hold Some Data Back
So the rule is: never judge a model on the data it learned from. Before training, we split our data into three parts.
- The training set is where the model learns — the practice questions.
- The validation set is used to choose between models and settings — the mock tests.
- The test set is locked away and used once, at the very end, for an honest score — the board exam.
A model's score on its own training data is usually