Gradient descent and local minima, for people who never took calculus

Black unisex t-shirt, with a happy orange and white cat sliding belly-up down a curving teal slope, above the text GRADIENT DESCENT

The cat on our Gradient Descent shirt is lying on its back, paws up, eyes shut, sliding down a curve. As pictures of machine learning go, it is more accurate than most diagrams, and it takes a few minutes and no calculus to see why.

Start on a hillside in thick fog. You need to get down, but you cannot see the valley, only feel the ground under your feet. So you feel which way slopes down most steeply, take a step that way, and feel again. That is gradient descent, and it is how most AI models learn.

The hill is made of mistakes

A model is a very large collection of adjustable numbers, called parameters. Small models have thousands, big language models have billions. At the start they are set more or less at random, and the model’s answers are rubbish.

To improve it, you show it examples and measure how wrong it is. That measure is called the loss. Lower is better. Now picture every possible setting of all those numbers as a landscape, where the height at each point is the loss. Training means finding a low point.

The gradient is the slope under your feet: for each parameter, which direction would make the loss go down, and how steeply. Gradient descent nudges every parameter a little in its downhill direction, then measures again. Repeat many times.

Two details matter. The size of each step is called the learning rate. Too large and you leap clean over the valley and land on the opposite slope; too small and you will still be walking next year. And in practice the slope is estimated from a small batch of examples at a time, not the whole dataset, so each step is a slightly wobbly guess. That version is called stochastic gradient descent, and the wobble turns out to be useful.

The local minimum

Walk downhill in fog for long enough and you will reach a spot where every direction goes up. You stop. You might be at the bottom of the valley. Or you might be in a small hollow halfway down the mountain, with a much deeper valley a kilometre away that you will never see. That hollow is a local minimum.

One caveat, to be fair to the field. For the huge models trained today, getting stuck in a bad local minimum turns out to be less of a problem than people once feared. With millions of directions to move in, it is rare for all of them to point uphill at once. The more common trouble is flat ground, where the slope is so gentle that progress crawls, and saddle points, which slope up one way and down another. The wobble from small batches helps shake training loose from both.

Why the cat works

Gradient descent has no ambition. It does not know where the best valley is. It knows only that this way is down, and it goes, and when there is no more down, it stops, perfectly content.

Hence the cat, sliding with its eyes shut. People who train models spend weeks watching a loss curve do exactly this, and they recognise the expression.

Gradient Descent Cat is an orange and white cat on its back, paws up, going down a teal curve, €26. Gradient Descent Ski Cat is a vintage ski poster with the cat on skis, heading for a plaque that reads LOCAL MINIMUM. Both are in the Cats and AI humour collection and the machine learning gift guide, next to reward hacking.

Shopping Cart
Scroll to Top