Linear RegressionFinding the Best Line
With the slider you found the best slope by trying values one after another. A real model may have thousands of numbers to learn, and it cannot try every combination. It needs a smarter way.
Walking Down the Error Valley
Imagine standing blindfolded on a hillside, trying to reach the bottom of the valley. You feel which way the ground slopes under your feet and take one step downhill. Then you feel again, and step again — until the ground is flat. This is gradient descent, the method that trains almost every machine-learning model.
For Riya's line, the ground is the error for each slope — the purple valley below — and the ball is the current slope. Each step follows one rule:
The number (alpha) is the learning rate: how big a step to take. Pick a learning rate and take steps.
Note
The "slope of the valley" is the derivative from Class 12 calculus. For Riya's data the valley is , so : it is negative left of the bottom (step right), positive right of it (step left) and zero at . A real model has many numbers to learn; the list of all their slopes together is called the gradient, which gives the method its name.
The Formula Shortcut
For a straight line there is also a shortcut. Setting the derivatives to zero gives a formula that lands on the bottom of the valley in one jump:
Let's try it on Aman's study log: hours studied and marks in four weekly tests — (1, 50), (2, 55), (3, 65) and (4, 70). First the means:
| product | |||||
|---|---|---|---|---|---|
| 1 | 50 | −1.5 | −10 | 15 | 2.25 |
| 2 | 55 | −0.5 | −5 | 2.5 | 0.25 |
| 3 | 65 | 0.5 | 5 | 2.5 | 0.25 |
| 4 | 70 | 1.5 | 10 | 15 | 2.25 |
| Sum | 5 |
So
Aman's model: marks ≈ 7 × hours + 42.5. Each extra hour of study is worth about 7 marks.
Play With the Formula
Here the computer runs that formula every time you tap. Tap the grid to add points and watch the purple best line jump to its new place.
Did you notice how a single point far from the others can swing the whole line? Such points are called outliers. Always look at your data before you trust the line.
If the formula finds the answer in one jump, why use step-by-step gradient descent at all? With many features, the shortcut needs a big matrix inverse (, from the Matrices chapter). For a dataset with millions of rows and thousands of features, the shortcut becomes
That is why gradient descent is the engine inside modern machine learning, from this little line all the way to the neural networks behind chatbots.