Testing Your AIOur Previous Approach
In Train, Test and Validation we split the data into three parts: a training set to learn from, a validation set to choose settings with, and a test set that stays locked away for the final score. Harpreet keeps a separate test set locked away too; the 20 farms are her training and validation data.
One Split, One Score
Below, a random 15 farms are the training farms and the other 5 are the validation farms. A degree-2 curve is fitted to the training farms and scored on the validation farms. Make new random splits and complete both missions.
Nothing changed except which five farms were held back — yet one split says the model is excellent and another says it is poor. The big flaw of a single validation split is
The luck of the draw
A split can hand the validation set the easiest farms, or the strangest ones. And the five validation farms are farms the model never learns from — with only 20, that hurts.