Learn AI /How Models Improve /Gradient descent
Gradient descent
- Intermediate
- How Models Improve
- Technical readers
Also called: stochastic gradient descent, SGD.
Back to the milestone you came from
In five seconds
What it is
A method for reducing a model’s loss by repeatedly adjusting its internal settings in the direction opposite the loss gradient.
In plain words
It is a repeated process for making a model less wrong: measure error, see which way to adjust the settings, and take a small step.
An everyday example
Imagine trying to reach the bottom of a misty hillside. You cannot see the whole landscape, but you can feel which way the ground slopes at your current position. You step in the downhill direction, check the slope again and repeat. The slope is like the gradient, your position is like the model’s current parameters, and the length of each step is like the learning rate. Very small steps make progress slow. Very large steps can carry you past the lower ground or make you bounce between slopes.
In the story, people can inspect the mistakes and change rules in a fairly direct, meaningful way. In a real neural network, the parameters are usually many numerical values with no simple human interpretation, and the gradient is computed mathematically rather than by staff reasoning through category labels.
Remember thisGradient descent repeatedly adjusts model parameters in the direction opposite the loss gradient. The learning rate sets the step size, and lower training loss does not by itself show better performance on unseen data.
Understand it
Understand it
Why it matters
If you use or procure AI systems, gradient descent helps explain why training takes many iterations, why settings such as learning rate matter, and why better results come from careful optimisation rather than one-time tuning.
How repeated small updates lower loss
After using this explainer, a reader should be able to say that gradient descent repeatedly measures loss, computes a downhill direction, and takes controlled steps to reduce error.
An interactive loss landscape shows a marker for the model’s current position. Users can step through training, change the learning rate, and switch between full-batch and stochastic updates to see smoother or noisier paths toward lower loss.
Users can press Next Step or Auto Run, drag a learning-rate slider, and toggle between full-batch gradient descent and stochastic gradient descent; the simulation updates the path and the loss-over-time chart.
Describe this without the diagram
Screen-reader mode presents the simulation as a narrated sequence. It explains that the model starts at a point on a loss curve with high loss, measures the local slope, then updates parameters in the downhill direction. It reports the effect of different learning rates: small steps reduce loss slowly, moderate steps reduce loss steadily, and very large steps can jump past the low point and fail to settle. It also contrasts full-batch updates, which follow a smoother path, with stochastic updates, which are noisier but often faster per step.
With reduced motion: Readers who prefer less motion see three static side-by-side states: start, one downhill update, and several updates later. A separate static comparison shows paths for small, medium, and too-large learning rates, plus a smooth path for full-batch and a jagged path for stochastic updates.
Gradient descent is a common way to improve a machine-learning model during training. The model starts with many settings, often called parameters, that control its outputs. We give the model examples, compare its outputs with the target answers, and calculate a loss: a number that summarises how wrong it was.
Next, the training process calculates how the loss changes as each parameter changes. Together, these rates form the gradient, which points towards a local increase in loss. Gradient descent updates the parameters in the opposite direction, which is intended to reduce the loss.
Then the model takes a small step. The size of that step is controlled by the learning rate. This repeated cycle of measure, calculate, and nudge tends to make the model less wrong on the training data, although not every individual step must lower loss.
Stochastic gradient descent, or SGD, uses one example or a small subset of examples per update instead of the whole dataset. That usually makes each update cheaper and noisier. Gradient descent is closely related to backpropagation, but they are not the same: backpropagation computes gradients, while gradient descent uses those gradients to update parameters.
How “Walking downhill in mist” maps onto the real thing
| In the story | In the AI system |
|---|---|
| adjustable scoring rules in the spreadsheet | model parameters or weights |
| checking a small batch of comments | using a batch in stochastic gradient descent |
| seeing which topic scores were too high or too low | calculating the gradient from the loss |
| slightly changing the scores and trying again | updating parameters with a learning rate |
A worked example
To reproduce one entry in the training log, the internal auditor examines the weight attached to the word “bus” when the classifier predicts the category “transport access”.
Starting from: Current weight for “bus” = 0.40. Learning rate = 0.10. For one mini-batch, the gradient of the loss with respect to this weight is -0.30.
- The classifier processes the mini-batch, compares its predictions with the human-coded categories and calculates a loss.
- Training calculates a gradient of -0.30 for the “bus” weight. Locally, this indicates that increasing the weight should reduce the loss.
- Gradient descent subtracts the learning rate multiplied by the gradient: 0.40 - (0.10 × -0.30) = 0.43.
- The next mini-batch uses the updated weight. Repeating this process aims to reduce training loss, while separate validation data tests whether performance also improves on unseen answers.
Result: The weight increases from 0.40 to 0.43, giving “bus” slightly more influence when the classifier predicts “transport access”. The calculation verifies one update, not that the classifier has improved overall.
In a United Nations workplace
At a regional bureau, an internal auditor is tracing how a classifier coded open-ended answers from a needs assessment. Before a note for the file goes to the chief of section at close of business, the auditor compares the training log with the coding protocol and validation results.
The log records how gradient descent adjusted the classifier’s parameters after each mini-batch of human-coded answers. Reviewing the loss, gradients and learning rate helps the auditor distinguish model training from later manual corrections and assess whether the claimed improvement is supported by validation results.
Before you try thisNeeds-assessment answers may contain personal or protection-sensitive details. Raw answers must not be entered into an external tool that has not been approved for such data, because respondents could be identified or vulnerabilities exposed. Training requires authorised data, an approved environment, restricted access, data minimisation and human validation.
Where you may hear it
- A vendor demonstration to an ICT governance board, when the learning-rate setting appears on screen
- A technical evaluation meeting where a procurement unit is testing a supplier’s training claims
- A model review in an evaluation office after validation loss starts rising
- A briefing to the chief of section before a coding method is cleared for a monitoring report
Avoid confusion
Avoid confusion
Often confused with
Optimisation
The process of adjusting a model’s internal settings to reduce loss, plus the algorithms and rules used to choose each adjustment.
Backpropagation
A method that propagates derivatives backwards through a model to compute how changes to each weight would affect the loss.
Chain rule
A calculus rule for tracing how a small change in one setting affects the final error through a chain of intermediate calculations.
Local and global minimum
A local minimum is the lowest loss in a nearby region, while a global minimum is the lowest loss anywhere in the whole set of possible model settings.
Common misconceptions
Common misconception: Gradient descent and backpropagation are the same thing.
Correction: They work together but do different jobs. Backpropagation computes how each parameter affected the loss, while gradient descent uses that information to update the parameters.
Common misconception: It always finds the best possible solution.
Correction: It often finds a good solution, but not always the global minimum. The loss surface can have flat regions, local minima, or unstable areas, and results depend on initialisation and settings.
Common misconception: A larger learning rate always trains faster.
Correction: Bigger steps can speed progress at first, but if they are too large the updates can overshoot low-loss regions and make training unstable or fail to converge.
Common misconception: Stochastic gradient descent is just a worse, noisy version of full-batch training.
Correction: It is noisier, but that noise can make training practical on large datasets and can sometimes help the process move through difficult parts of the loss landscape.
Quick check
What does stochastic gradient descent change compared with basic full-batch gradient descent?
SGD uses one example or a small batch for each update rather than the whole dataset. It does not require a larger learning rate, wait for a full epoch before updating, or select only the hardest examples.
Go deeper
Go deeper
The simple version
It is a repeated process for making a model less wrong: measure error, see which way to adjust the settings, and take a small step.
The technical version
Given parameters and a differentiable loss function, the method computes the gradient of loss with respect to the parameters and updates them by moving in the negative gradient direction, often using mini-batches in SGD and a learning-rate hyperparameter to scale each step.
Reviewed external resources
EncyclopaediaGradient descent (opens in a new tab)Wikipedia
This article gives a broad reference-style overview of definitions, variants, and related concepts, and suits readers who want a more formal summary.
Intermediatelink checked 2026-07-31
CourseReducing Loss (opens in a new tab)Google
This course page focuses on reducing loss during training with practical explanations and suits beginners and workplace learners.
Intermediatelink checked 2026-07-31
VideoGradient descent, how neural networks learn (opens in a new tab)3Blue1Brown
This video gives a strong visual intuition for downhill movement on a loss surface and suits visual learners who want the big picture first.
Intermediatelink checked 2026-07-31
VideoNeural networks, explained visually (opens in a new tab)3Blue1Brown
This video places gradient descent inside the wider training process for neural networks and suits readers who want context before details.
Intermediatelink checked 2026-07-31
VideoVideo explanations: Gradient descent (opens in a new tab)YouTube search
This search results page helps readers browse multiple video explanations and compare teaching styles; it suits people still looking for a presentation that clicks for them.
search results, not one pagelink checked 2026-07-31
DiscussionCommunity discussion: Gradient descent (opens in a new tab)Reddit search
This search results page lets readers see community questions and explanations about gradient descent; it suits readers who want informal discussion and common points of confusion.
search results, not one pagelink checked 2026-07-31
Where this came from
Drafted with a language model through an offline pipeline, checked against this site's schema, and every external link above fetched to see whether it answers. Treat it as a starting point and check anything that matters against the primary sources. How this site is built and what that means.
The one thing to remember
Gradient descent repeatedly adjusts model parameters in the direction opposite the loss gradient. The learning rate sets the step size, and lower training loss does not by itself show better performance on unseen data.