Momentum

Deeper
How Models Improve
Technical readers

Detail

In five seconds

What it is

A training method that carries part of the previous update into the next one so parameter changes keep moving in useful directions and wobble less.

In plain words

Plain gradient descent listens only to the latest training signal. Momentum also remembers some recent movement, so updates tend to wobble less and keep going in directions that have been helping for several steps.

An everyday example

Amina and Luis are moving a loaded trolley through a corridor with small bumps in the floor. If they react only to each bump, they keep jerking the trolley left and right and make slow progress. Instead, they keep some of the trolley’s existing motion while still correcting its direction when a new bump pushes it off course. The trolley then wiggles less from side to side and keeps rolling forward more steadily. If they push too hard for too long, however, the trolley can overshoot the doorway they wanted.

Where the story stops working

A real trolley has physical mass and keeps moving because of mechanics. A model does not have literal inertia. Momentum in optimisation is just a mathematical rule for combining past and current gradients. Also, the loss surface can have thousands or millions of dimensions, so there is no single corridor or doorway in the real process.

Remember thisMomentum retains part of the previous parameter update, helping training move steadily through consistent gradients and resist brief noisy reversals. Poor settings can still cause overshooting.

Understand it

Why it matters

If you compare training runs, choose optimisers, or diagnose unstable learning, momentum matters because it changes how fast and how smoothly a model moves toward lower loss. A run that looks noisy or slow under plain gradient descent may improve simply because updates are bouncing around. Momentum can reduce that bounce and often reach useful performance sooner, which affects compute cost, training time and model quality. It also matters for troubleshooting: if momentum is too low, training may crawl; if too high, updates may overshoot or oscillate. For non-specialists overseeing AI work, this helps in reading training reports and asking better questions about why one run converged and another did not.

Plain gradient descent vs momentum on a wavy slope

Show how momentum uses both the current gradient and a stored update to reduce zigzagging and make steadier progress toward lower loss.

01
Step 1: Start on the slope
Both methods begin from the same point on a curved loss surface. The local gradient points downhill but also swings side to side because the valley is narrow.
Draw two identical contour valleys side by side. Mark the same starting point near the upper left of each valley. Add a short arrow labelled current gradient pointing diagonally downhill.
02
Step 2: Plain update
Without momentum, the parameter update uses only the current gradient. The next step responds sharply to local changes, so the path can bounce across the valley.
In the left panel, draw a path of 4 to 5 arrows that alternate left-right while moving slowly downward along the valley. Label each arrow uses current gradient only.
03
Step 3: Momentum update
With momentum, the new update combines the current gradient with part of the previous update. This carries useful direction forward while damping rapid sideways changes.
In the right panel, show a small faded previous arrow and a current gradient arrow combining into a darker resulting arrow. Add a compact formula label: velocity = beta times old velocity minus learning rate times gradient.
04
Step 4: Compare the paths
Across several steps, momentum usually reduces oscillation across the valley and can move faster along the long shallow direction, though too much can overshoot.
Complete the right-panel path with 4 to 5 arrows that cross the valley less often and travel farther along it. Add a caution icon near the end with the label too much momentum can overshoot.

A side-by-side comparison with the same wavy valley in both panels. The left panel shows plain gradient descent taking short alternating steps across the valley. The right panel shows momentum carrying part of the previous step forward, so the path oscillates less across the valley and moves farther along it. Read top to bottom through the four labelled steps, then compare left and right at each step.

Hover or tap each step label to highlight the matching arrows in both panels; a toggle switches formula labels on and off.

Describe this without the diagram

This comparison diagram explains why momentum changes training behaviour. There are two panels showing the same narrow, curved valley in a loss surface. In the left panel, plain gradient descent starts at the upper side of the valley and follows only the current gradient each time. The path zigzags from one side of the valley to the other, so it makes progress but wastes motion sideways. In the right panel, momentum starts at the same point but each new update combines the current gradient with part of the previous update, shown as a stored velocity. Because recent movement is partly carried forward, the path swings less across the valley and travels farther along the valley toward lower loss. A final note says that if the learning rate or momentum is too high, the method can overshoot the target region.

With reduced motion: Show the same two panels as a static comparison. Use numbered arrows and shaded overlays instead of animation to indicate how previous velocity and current gradient combine in the momentum panel.

When a model learns, training computes a loss, which is a number showing how wrong the current output is. It then computes a gradient: a set of values describing how the loss changes as parameters change. Moving against the gradient is expected to reduce the loss. Basic gradient descent uses only the current gradient at each step.

Momentum adds memory. Instead of using just today’s gradient, it keeps a running update vector, often called velocity. Each new update combines part of the previous velocity with the current gradient, then uses that combined value to change the model’s parameters.

What goes in: the current gradient, the previous velocity, a learning rate, and a momentum coefficient such as 0.9. What happens: the old velocity is shrunk a little, the new gradient is added, and the parameters are moved using the result. What comes out: a smoother sequence of parameter updates.

This helps for two common reasons. First, gradients from one mini-batch to the next can be noisy, so plain gradient descent may zigzag. Momentum averages over recent steps, which can damp that wobble. Second, if many gradients point in a similar direction over time, momentum can build speed along that direction and move faster through shallow regions.

Momentum does not guarantee a better solution, and it is not the same as adaptive methods such as Adam. It is a rule for updating parameters during optimisation, and its behaviour depends on other choices such as learning rate, batch size and the shape of the loss surface.

How “Pushing a loaded trolley down a bumpy corridor” maps onto the real thing

In the story	In the AI system
the trolley’s existing motion	the stored velocity from previous parameter updates
small floor bumps	noisy gradients from different mini-batches
keeping some forward roll while correcting course	combining previous velocity with the current gradient
overshooting the doorway	stepping past a good minimum if momentum or learning rate is too high

A worked example

For one weight in the proposed situation-report classifier, the technical evaluation panel checks the vendor’s explanation of an update using momentum.

Starting from: Current weight: 5.00. Previous velocity: -0.30. Current gradient: 0.80. Learning rate: 0.1. Momentum coefficient: 0.9.

Compute the new velocity: 0.9 × (-0.30) - 0.1 × 0.80 = -0.27 - 0.08 = -0.35.
Update the weight using the velocity: 5.00 + (-0.35) = 4.65.
For the next mini-batch, suppose noise causes the gradient to reverse slightly to -0.10.
Compute the next velocity: 0.9 × (-0.35) - 0.1 × (-0.10) = -0.315 + 0.01 = -0.305. The update retains the earlier direction rather than reversing immediately.
When gradients usually point in a consistent direction, the velocity can accumulate. Small, brief reversals then have less influence than they would under plain gradient descent.

Result: The weight moves from 5.00 to 4.65 and then to 4.345. Momentum smooths the effect of the noisy second mini-batch, although excessive momentum or an unsuitable learning rate could carry the update too far.

In a United Nations workplace

In a procurement unit, a protection officer serving on the technical evaluation panel reviews a vendor’s proposed classifier for tagging a draft situation report. The demonstration must finish before close of business so the reporting officer can finalise the regional bulletin.

When the vendor says the model was trained with momentum, this means that each parameter update retained part of the preceding update. Consistent gradients therefore built speed, while small reversals from noisy mini-batches had less immediate effect. It does not establish that the classifier is accurate, safe or suitable for deployment.

Before you try thisA draft situation report may contain personal, protection-related or security-sensitive material. It must not be provided to a bidder or entered into a tool that has not been approved for such data. Testing should use synthetic or properly authorised, minimised data in a controlled environment. Momentum offers no privacy or confidentiality protection.

Where you may hear it

A vendor demo during a procurement evaluation
A technical review of a model-training report
A meeting to clear an internally developed classifier
A discussion with developers after training loss begins to oscillate

Avoid confusion

Often confused with

Common misconceptions

Common misconception: Momentum means using a larger learning rate.
Correction: They are related but different. Learning rate scales how large each update is right now. Momentum adds a memory of previous updates. You can have a small learning rate with high momentum, or a larger learning rate with no momentum, and the behaviour will differ.
Common misconception: Momentum always makes training faster and better.
Correction: It often helps, but not always. On some problems it can overshoot, interact badly with a poor learning rate, or offer little benefit compared with another optimiser. The effect depends on the loss landscape, batch noise and the chosen hyperparameters.
Common misconception: Momentum and Adam are the same thing.
Correction: Adam includes momentum-like moving averages, but it also rescales updates using per-parameter estimates of gradient magnitude. Standard momentum keeps a running update direction; Adam adds adaptive step sizing on top of related ideas.
Common misconception: Momentum removes the need for backpropagation or gradients.
Correction: It does not. Backpropagation is still used to compute gradients, and momentum uses those gradients to build the next update. It changes the update rule after gradients are available.

Quick check

What is the main effect of adding momentum to gradient descent?

Go deeper

The simple version

The technical version

In first-order optimisation, momentum augments gradient descent with a state variable often called velocity. A common form is v_t = beta v_{t-1} - alpha grad L(theta_t), followed by theta_{t+1} = theta_t + v_t, where alpha is the learning rate and beta controls how much past update direction is retained. This acts like an exponential moving average of past gradients, expressed through the update vector. In directions where gradients keep similar sign across steps, the velocity accumulates and effective progress increases. In directions where gradients alternate sign, as in narrow ravines or noisy mini-batch training, the averaging can cancel part of the oscillation. The result is often faster movement along shallow, consistent directions and less side-to-side zigzagging across steep directions. Different conventions write the equations with signs arranged differently, but the core mechanism is the same: combine current gradient information with decayed update history. Nesterov momentum is a related variant that evaluates the gradient after a look-ahead step, so it should not be treated as identical to standard momentum.

Reviewed external resources

EncyclopaediaStochastic gradient descent (opens in a new tab)Wikipedia
This link gives a broad reference overview of stochastic gradient descent and related update ideas, which suits readers who want definitions and context.
Intermediatelink checked 2026-07-31
CourseReducing Loss (opens in a new tab)Google
This link gives a structured course-style explanation of reducing loss during training, which suits readers who want guided learning.
Intermediatelink checked 2026-07-31
VideoNeural networks, explained visually (opens in a new tab)3Blue1Brown
This link gives a visual intuition for neural network training dynamics, which suits readers who learn best from geometric explanations.
Intermediatelink checked 2026-07-31
VideoVideo explanations: Momentum (opens in a new tab)YouTube search
This link is a YouTube search results page for momentum explanations, so it suits readers who want to compare several video styles rather than rely on one source.
search results, not one pagelink checked 2026-07-31
DiscussionCommunity discussion: Momentum (opens in a new tab)Reddit search
This link is a Reddit search results page for momentum discussions, so it suits readers who want to see common questions, informal explanations and practical confusion points.
search results, not one pagelink checked 2026-07-31

Where this came from

Drafted with a language model through an offline pipeline, checked against this site's schema, and every external link above fetched to see whether it answers. Treat it as a starting point and check anything that matters against the primary sources. How this site is built and what that means.

The one thing to remember

Momentum retains part of the previous parameter update, helping training move steadily through consistent gradients and resist brief noisy reversals. Poor settings can still cause overshooting.