Random ForestSummary
One page to revise from — the ideas first, then the formulas.
- An ensemble combines many models: a majority vote for a kind, an average for a number. A random forest is an ensemble of decision trees.
- A majority of many voters that are each right more than half of the time, and that decide independently, is right far more often than any one of them (Condorcet's jury theorem). Voters who copy each other gain nothing.
- Bagging: every tree learns from its own bootstrap sample, drawn with replacement from the data. A bag has repeats and misses about a third of the rows — its out-of-bag rows.
- Random feature choice: at every split, a tree may look at only a random handful of features. Together with bagging, this makes every tree different.
- A single deep tree has high variance: its shape jumps when the data changes a little. Averaging many different trees cancels most of that variance, so a forest is far steadier.
- When most trees make the same mistake, voting cannot fix it; boosting trains trees one after another on the mistakes of the trees before.
The majority of 3 voters is right
0.648 when
A row is left out of a bag
0.35 for 10 rows, about 0.37 for big data