Neural NetworksSummary
One page to revise from — the ideas first, then the formulas.
- A neuron multiplies its inputs by weights, adds a bias, and passes the total through an activation function. Without the activation it is just linear regression's straight line.
- A neural network joins neurons in layers: an input layer, one or more hidden layers, and an output layer. Many hidden layers make a deep network.
- A computational graph breaks the calculation into small nodes. Working through it from the inputs to the output is the forward pass.
- Activation functions — step, sigmoid, tanh, ReLU — bend the output. Without them, any number of layers can only draw straight lines.
- Training lowers a loss by gradient descent. Backpropagation works out every weight's gradient with the chain rule, from the loss back to the first layer.
- More hidden neurons allow more complicated boundaries, but training can get stuck: then reset the weights, or change the network's shape or learning rate.
- CNNs, RNNs, GANs and transformers are networks of different shapes, built from the same parts.
One neuron
Activations
Loss
Gradient descent step