Train, Test and ValidationThe Test Set
With the hyperparameters chosen, it is finally time to open the test set — once.
Kabir is now an intern at a start-up. He trains 50 versions of a model, checks each one's test score, and keeps the best: 94 %. A month later, on real customers, the model scores only 81 %. What went wrong?
The real customers were unusual
He chose the model using the test set, so it was no longer unseen
50 versions were too few
Every time you look at a test score and change something because of it, a little information about the test set leaks into your choices. After 50 peeks, the test set has become just another validation set, and its score is too good to be true. This is called a data leak.
Tip
Three rules for the test set: never train on it, never tune on it, and look at it once — at the very end.
Which Set Does the Job?
Sort each job into the set that does it.
The final score in the report
Gradient descent adjusts w and b on it
Choosing the learning rate
Usually the biggest share of the data
Opened only once, at the very end
Comparing degree 2 with degree 5
Choosing the pass/fail threshold
The slope and intercept are learned from it
Training
Validation
Test
Speed Round: Honest or Leak?
Get five right to move on.