Random ForestVariance in Composition
In the last chapter a single tree's high variance — its habit of changing shape when the data changes a little — was its weakness. In a forest it becomes a strength. Leo Breiman, who invented random forests in 2001, showed that a forest is strongest when its trees are as different as possible, so that their mistakes do not line up.
The coloured map shows which kind the model predicts at every weight and length, and the hollow rings are Meena's 10 new mangoes. The chart under the map keeps the score of every model you grow: single trees and forests of 25. Switch between 1 tree and 25 trees, and re-grow each a few times.
With every re-grow, the single tree's map jumps: around the mangoes, about a quarter of it changes colour, and its score on the new mangoes swings up and down. The forest's map moves far less — usually only a few per cent near the mangoes — and its score swings much less. Averaging many different trees cancels out much of their variance.
Which Is More Reliable?
Meena re-grows both models many times. Which one can she trust more on mangoes it has never seen?
Embracing diversity
Each tree in a forest is weak and wobbly on its own. Together they make a steady, strong
Variance in Predictions
Here 9 trees from one forest vote on the 10 new mangoes. Each row is a tree and each column a mango; a cell shows that tree's vote, in amber for Alphonso and green for Totapuri, with a red ring when the vote is wrong. The last row is the forest's majority. Tap the columns and complete both missions.
The trees' mistakes are scattered: one tree slips on one mango, another tree on another. As long as most trees are right about a mango, the vote is right. But when most trees make the same mistake, voting cannot save the forest. Fixing exactly those mangoes inspired a different family of methods, boosting, in which trees are trained one after another and each new tree concentrates on the mistakes of the trees before it.