Popular Courses
Beginner – Linear Regression
Introduction
Beginner – Decision Trees
Introduction
Beginner – Neural Networks
Introduction

Sign in to Curvingo

or

New Account     Reset Password     

Plans

Details
Neural Networks
Introduction
Computational Graphs
Activation Functions
Backpropagation: How Networks Learn
See for yourself
The Deep Learning Universe
Code with Python
Summary
Curvingo · Made in India

Share

Send us Feedback

Please let us know if you have any feedback and suggestions, or if you find any errors and bugs in our content.

Sorry, your message couldn’t be submitted. Please try again!

Thanks for your feedback!

Reset Progress

This will delete your progress and chat data for all chapters in this course, and cannot be undone!

Glossary

Select one of the keywords on the left…

Neural NetworksBackpropagation: How Networks Learn

Reading time: ~10 min

A new network starts with random weights, so its first answers are poor. Training improves the weights one small step at a time, with the same gradient descent you used in Linear Regression:

  1. Forward pass: feed in an example and work out the prediction \hat{y}.
  2. Loss: measure how wrong it is, L = (\hat{y} - y)^2.
  3. Backward pass: for every weight, work out how much the loss changes when that weight changes a little — its gradient. The gradients are worked out backwards through the graph, from the loss to the first layer: this is backpropagation.
  4. Update: move every weight a small step against its gradient, w_{\text{new}} = w - \alpha \times \text{gradient}.

Here is a tiny network: one input, two hidden neurons with the sigmoid activation, and one output, \hat{y} = v_1 h_1 + v_2 h_2. Its only training example is x = 1 with the target y = 1, and the learning rate is \alpha = 0.5. The red line under the network records the loss after every forward pass. Run forward and backward passes and complete both missions.

Three rounds of forward and backward took the loss from 0.25 to below 0.001. A real network does exactly the same — for millions of weights and millions of examples, again and again.

The Chain Rule at Work

Let's do the first backward pass by hand. Before it, \hat{y} = 0.5, h_1 = \sigma(1) = 0.731 and v_1 = 0.5. First, how does the loss change with the prediction?

\dfrac{dL}{d\hat{y}} = 2(\hat{y} - y) = 2(0.5 - 1)

So dL/d\hat{y} = .

The output is \hat{y} = v_1 h_1 + v_2 h_2, so when v_1 grows by a little, \hat{y} grows by h_1 times as much. By the chain rule, the two rates multiply:

\dfrac{dL}{dv_1} = \dfrac{dL}{d\hat{y}} \times h_1 = -1 \times 0.731 = -0.731

v_1^{\text{new}} = 0.5 - 0.5 \times (-0.731)

So the new weight is v_1 = — the widget showed it rounded, as 0.87, after its first backward pass.

Note

For a weight further back, the chain is longer: w_1 changes h_1, h_1 changes \hat{y}, and \hat{y} changes L. The computer multiplies the rate of every link in the chain, just as you did for v_1 — that gave the widget's gradient of −0.098 for w_1.

Backpropagation is the chain rule from Class 12 calculus, applied again and again from the loss back to the first layer, reusing every piece it has already worked out.

Sina