Gradient Descent, Visualized
Watch the optimizer follow the slope of the loss landscape to a minimum — animated with Manim.
The core idea
Training a neural network means finding parameters θ that minimize a loss function L(θ). Gradient descent does this by asking: which direction is downhill? Then it takes a small step that way — repeatedly.
The update rule is just:
θ ← θ − α · ∇L(θ)
where α is the learning rate — the step size.
Why not just find the minimum analytically?
In a real neural network, L(θ) lives in millions of dimensions and has no closed-form solution. Gradient descent works at any scale — it only needs the local slope, not the global picture.
The learning rate matters
Too large → the step overshoots the minimum and the loss bounces or diverges. Too small → training takes forever. Modern optimizers (Adam, AdaGrad) adapt α per-parameter so you rarely need to hand-tune it.