Underfitting and OverfittingIntroduction
Neha's family grows tea in Munnar. Every week she writes down how much leaf the pickers bring in, and she wants a model that describes the harvest through the year.
You have met the two ways such a model can go wrong. In Train, Test and Validation, a bendy curve memorised eight students — perfect on them, useless on new ones. That is overfitting. A model that is too simple to follow the pattern at all is underfitting.
Think of two kinds of people. One jumps to a conclusion from too little — "it rained on Monday, so it will rain all week". The other overthinks every tiny detail and misses the big picture. A model that jumps to conclusions has high bias; a model that overreacts to every detail has high variance. A model that is too simple will
In this chapter you will see bias and variance, and turn the dials that trade one for the other.
Three Curves, One Year
Here are Neha's 12 readings from one year, and the dashed curve — the estate's real harvest pattern, which in real life nobody knows exactly. Fit all three curves and compare each with the dashed one.
The straight line is too simple: it misses the rise and fall of the season. That is underfitting. The very bendy curve passes close to every reading but swings wildly between them — it learned the luck of this one year. That is overfitting. The middle curve follows the real pattern.
Bias and Variance on a Dartboard
Imagine training the same kind of model many times, each time on a different sample of data. Every trained model is one dart, and the bull's-eye is the truth. Bias is how far the middle of the darts lands from the bull's-eye; variance is how much the darts scatter. Use the two sliders to create each player.
Bias:
Variance:
A straight line fitted to a harvest that rises and falls is like the steady player: whichever year you train it on, it misses in the same way. That is high bias. A wildly bendy curve is the shaky player: right on average, but each version lands somewhere different.