Neural NetworksSee for yourself
Riya has noticed something odd. On a few evenings her kulfis sell out within an hour; on others she goes home with a full box. Cool evenings are slow, but so are scorching ones, and so are very humid, sticky ones. Her notebook of 42 evenings shows a sweet spot: warm, but not too hot, and neither too dry nor too humid.
On the chart, green dots are evenings that sold out and blue dots are slow ones. A straight line cannot fence off a patch in the middle — but a network can learn to. This network has two inputs (temperature and humidity, each scaled to lie roughly between −1 and 1), one hidden layer of tanh neurons, and a sigmoid output. The background shows its answer everywhere: green for "sells out". Each press of Train runs 100 rounds of gradient descent on all 42 evenings. Complete both missions.
Hidden neurons:
One hidden neuron can draw only one straight boundary, and no straight line can fence off a patch in the middle — so it gives up and calls every evening slow, which is right for 26 of the 42 (62 %). With several hidden neurons, their straight edges join up into a curved fence around the sweet spot. More neurons give more bends, and more bends can follow more complicated patterns.
When Training Gets Stuck
Training a network does not always go smoothly. Sometimes the accuracy climbs for a while and then stops, stuck at the same number however long you keep training.
If your network's accuracy stops improving and stays stuck, what should you try?
More art than science
Training neural networks is hard. A network can fail because of unlucky starting weights, a learning rate that is too big or too small, or a shape that does not suit the problem. Getting a network to learn well takes care, patience and plenty of