Underfitting and OverfittingLOESS Regression
Settings that you choose before training, such as the degree of a curve, are hyperparameters. Many of them are dials between bias and variance. LOESS (locally estimated scatterplot smoothing) has a very clear one.
LOESS draws a smooth curve through scattered data without choosing a formula. To find the curve's height at one week, it looks only at the nearby weeks — the neighbourhood — gives the closest ones the most weight, and fits a short straight line through them. Doing this at every week and joining the results gives a smooth curve. The dial is the span: the fraction of all the data in each neighbourhood.
A wide span
With a span near 1, every neighbourhood holds almost all the data, so every local line is nearly the same line. The curve ignores the ups and downs of the year: high
A narrow span
With a tiny span, each neighbourhood holds only 3 or 4 weeks, so the curve swings to follow every unusual week: high variance, or overfitting.
Dial in the Span
Here are Neha's 40 weekly readings, and 12 more weeks held back to check the curve. Tea has two big harvests: the first flush in spring and the monsoon flush. The amber window shows the neighbourhood and the local line for one week — tap the chart to move it. Drag the span and complete the missions.
Span:
The error on the held-out weeks is high at both ends of the dial and lowest in between — underfitting on one side, overfitting on the other.