How Gradient Descent Actually Works

From a tiny logistic regression to a language model with hundreds of billions of parameters, nearly every neural network in use today was trained by the same simple idea: gradient descent. The name sounds technical, but the intuition behind it is thoroughly everyday. In this post we'll build that intuition step by step, make it concrete with a few lines of code, and then dwell on the single most important dial — the learning rate. You don't need a background in calculus; the only concept we'll lean on is the feeling of a slope.

Walking downhill in the fog

Picture your model as a hilly landscape you're walking across. Let the height at each point represent the current loss — how wrong the model is right now. Your goal is to reach the lowest valley, the point of least error. The trouble is that you can't see the whole map from above; you're in fog, able to feel only the ground beneath your feet. Even that, however, is enough: at wherever you stand you can sense which way the slope descends, take a small step in that direction, and repeat. After enough steps, no matter where you started, you find yourself at the bottom of a valley.

That sense of "which way is downhill" is what mathematics calls the gradient: the derivative of the loss with respect to each parameter. The gradient points in the direction of steepest increase, so since we want the loss to decrease, we move the opposite way. The update rule fits on a single line:

Here $\theta$ denotes the model's parameters (its weights), $\nabla_\theta \mathcal{L}$ is the gradient, and $\eta$ is the learning rate — how far we travel on each step. The minus sign in front of the gradient does one job: it turns this uphill-pointing vector into a downhill move.

Learning a line in twelve lines

The best way to see that the abstract rule really works is to run it by hand. In the example below we fit a single-parameter model to a handful of data points that follow the relationship y = 2x. The model starts at w = 0, measures its error, computes the gradient, and edges closer to the right answer on every step:

import numpy as np
 
x = np.array([1, 2, 3, 4], dtype=float)
y = np.array([2, 4, 6, 8], dtype=float)   # true relationship: y = 2x
w = 0.0
lr = 0.01
 
for step in range(200):
    pred = w * x
    loss = ((pred - y) ** 2).mean()       # mean squared error
    grad = (2 * x * (pred - y)).mean()    # d loss / d w
    w -= lr * grad                        # the update step
print(round(w, 3))                        # ≈ 2.0

Three things happen on each pass of the loop: the model makes a prediction (pred), a loss measures how far that prediction strays from the true values, and the derivative of the loss with respect to $w$ tells us which way to go. The line w -= lr * grad then takes a small step in that direction. After two hundred steps the model arrives at w ≈ 2 entirely on its own. A real neural network does exactly this; its only differences are that it runs the same procedure for millions of parameters at once and computes the gradients automatically, via backpropagation, rather than by hand.

The decisive role of a single number

The success of gradient descent hinges largely on one hyperparameter, the learning rate, because it sets how far you travel on each step. Choose it too small and training crawls hopelessly — you inch toward the valley and never reach the bottom in any reasonable time. Choose it too large and the opposite failure appears: your steps overshoot to the far wall of the valley, the loss oscillates up and down, and it may even diverge. In the "just right" band between these extremes, the descent is both stable and fast.

In practice, a startling share of problems described as "my model won't train" trace back to this one number. If the training curve isn't falling the way you expect, the first intervention to try is usually to lower the learning rate gradually and watch the loss curve closely.

Pulling it together

Gradient descent reduces a daunting goal to a remarkably simple loop: predict, measure how wrong you are, take a small step in the direction that lowers the error, and repeat. What lets this loop carry all of deep learning is backpropagation, which computes the gradients efficiently for large networks. In the next post we'll see how this learning mechanism meets the structure at the heart of modern language models — attention.