Decision TreesThe Problem of Perturbations
Decision trees are quick to train, easy to read and not easily upset by odd values. But they have one big weakness: they are unstable. A tiny change in the training data — a perturbation — can produce a completely different tree.
A week later Meena discovers that her scale was faulty. One Alphonso she had written down as 250 g really weighs 235 g. Surely 15 grams cannot matter?
Drag the ringed mango left and right and watch the tree's first question.
Moving one mango by 15 g changed the root question from length to weight, and with it every question below. Nothing about mangoes changed — only one number in one row of the data. A model whose shape changes this much when its data changes a little is said to have high variance.
Why Is Instability A Problem?
Left alone, a tree is too eager to please. It keeps slicing until every leaf is pure, so it grows deep and learns the noise in its training data along with the pattern. A deep tree overfits, and its exact shape depends on luck: which mangoes happened to be measured, and how carefully.
How can we stop a tree from growing into a huge, overfitted mess?
Pruning
Pruning a tree means setting rules that stop it early, such as a maximum depth ("no more than 2 questions") or a minimum leaf size ("do not split a box with fewer than 5 mangoes"). Pruning tames overfitting. But even a pruned tree can change its shape when one data point moves — that weakness belongs to single trees.