Popular Courses
Beginner – Linear Regression
Introduction
Beginner – Decision Trees
Introduction
Beginner – Neural Networks
Introduction

Sign in to Curvingo

or

New Account     Reset Password     

Plans

Details
Neural Networks
Introduction
Computational Graphs
Activation Functions
Backpropagation: How Networks Learn
See for yourself
The Deep Learning Universe
Code with Python
Summary
Curvingo · Made in India

Share

Send us Feedback

Please let us know if you have any feedback and suggestions, or if you find any errors and bugs in our content.

Sorry, your message couldn’t be submitted. Please try again!

Thanks for your feedback!

Reset Progress

This will delete your progress and chat data for all chapters in this course, and cannot be undone!

Glossary

Select one of the keywords on the left…

Neural NetworksActivation Functions

Reading time: ~10 min

After a neuron adds up its inputs, the activation function decides what comes out. Three famous ones:

  • Step: 1 when z \geq 0, otherwise 0 — a plain on–off switch. The first artificial neurons, in the 1950s, used it.
  • Sigmoid: \sigma(z) = 1/(1 + e^{-z}), a smooth S from 0 to 1, like a dimmer switch. It is handy when the answer is a chance. Its cousin tanh is the same S stretched to run from −1 to 1.
  • ReLU (rectified linear unit): \text{ReLU}(z) = \max(0, z). Negative inputs become 0 and positive inputs pass through unchanged. It is quick to compute, and it is the most common activation inside deep networks today.

Pick an activation and drag the purple knob along the curve. Complete all three missions.

Why Not Leave It Out?

Why bother with an activation at all? Take a tiny network with one input x, two hidden neurons and one output:

h_1 = \text{act}(x)

h_2 = \text{act}(-x)

y = v_1 h_1 + v_2 h_2

With no activation, y = v_1 x - v_2 x = (v_1 - v_2)\,x — a straight line through the origin, however you choose the weights. Try to make the purple output pass through the amber flags.

Weight v₁ = ${v1}

Weight v₂ = ${v2}

Two straight-line layers on top of each other make just another straight line. So a network without activations is no smarter than linear regression, however many layers it has. ReLU puts a bend into each neuron's output, and many bends together can follow almost any curve. That is what people mean when they say activation functions add non-linearity.

Choosing an Activation

What would happen to a neural network with no activation functions at all?

It could only learn straight-line patterns, however deep it was
It would train a hundred times faster and better
It would memorise the whole dataset at once

No one-size-fits-all

ReLU is the usual choice inside deep networks, sigmoid at the output when the answer is a chance, and tanh in some older networks. There is no perfect choice: picking the right function depends on the problem you want to solve.

Speed Round: What Comes Out?

Get five right to move on.

Sina