Learn AI /How Models Improve /Vanishing and exploding gradients

Vanishing and exploding gradients

Deeper
How Models Improve
Technical readers

Also called: vanishing gradient, exploding gradient.

Detail

In five seconds

What it is

Training can stall or blow up when error signals shrink toward zero across layers or time steps, or grow so large that updates become unstable.

In plain words

During training, the model sends a correction signal backwards. If that signal fades away, earlier parts barely learn. If it grows out of control, updates become unstable.

An everyday example

A team is reviewing survey responses from many offices. The lead reviewer writes a note saying which issue labels need correcting, then each person in a long line passes a shorter summary of that note to the next person. If each handover trims the message a little, the people at the far end receive almost nothing and cannot improve their labels. If each handover exaggerates the message, the far end receives a dramatic instruction and over-corrects many labels. The review process then either stalls quietly or becomes chaotic.

Where the story stops working

People can choose to ignore, rewrite or clarify a note, but gradients are numerical values produced by fixed computations during backpropagation. Also, real models can have many paths for the signal, not just one line of people, and modern methods such as gating, residual connections or gradient clipping can change the behaviour in ways this story does not show.

Remember thisGood training needs gradients that stay informative across depth or time, not signals that fade to almost zero or blow up to unstable size.

Understand it

Why it matters

For a non-specialist, this term matters because a model can fail during training even when the data, hardware and objective all look reasonable. If gradients vanish, teams may waste time assuming the model only needs more epochs, when the real issue is that early layers are barely learning. If gradients explode, training may appear erratic, expensive or impossible to reproduce. Understanding this helps you ask better questions about architecture choice, stability checks, monitoring and whether reported improvements are robust. It also helps distinguish an optimisation problem from other issues such as poor labels, overfitting or a badly chosen loss function.

Backward signal: fading versus blowing up

Show why repeated backward multiplications can make early layers receive either almost no learning signal or an unstable one.

01
Start from output error
The backward pass begins with the model’s output error and sends a gradient to the previous layer.
Draw five boxes in a row labelled Layer 1 to Layer 5. Place Output/Error at the far right. Add arrows pointing left from Output/Error through the layers on both sides of the comparison.
02
Vanishing case
If each backward step multiplies by factors smaller than 1, the signal shrinks quickly.
On the left side, place numbers above the arrows such as 1.0, 0.4, 0.16, 0.06, 0.02. Make arrow thickness and colour intensity fade as they move left. Add a caption: 'Early layers get tiny updates'.
03
Exploding case
If each backward step multiplies by factors larger than 1, the signal grows quickly.
On the right side, place numbers above the arrows such as 1.0, 2.5, 6.3, 15.8, 39.5. Make arrows thicker and more saturated as they move left. Add a caption: 'Updates become unstable'.
04
Training consequence
Useful learning needs gradients that are informative and numerically manageable, not near zero and not extreme.
Below both sides, add a middle band labelled 'healthy range' and place small markers showing that both extremes fall outside it. Add note chips: 'slow or stalled learning' on left, 'jumping loss / NaN risk' on right.

Two side-by-side chains of layers read from right to left, because the error signal travels backwards from output to earlier layers. The left chain shows gradient values shrinking at each step; the right chain shows them growing. A short caption under each side states the training effect on early layers.

Hover or tap each arrow to reveal the multiplication factor at that step and a short note explaining how it changes the signal.

Describe this without the diagram

This is a side-by-side comparison of backward error signals in a deep model. On both sides, the signal starts at the output and moves left through five layers. In the left panel, the numbers above the arrows decrease from 1.0 to 0.02, and the arrows fade, showing vanishing gradients. The caption says early layers get tiny updates, so learning is slow or stalled. In the right panel, the numbers increase from 1.0 to 39.5, and the arrows thicken, showing exploding gradients. The caption says updates become unstable, with jumping loss or invalid numbers. A final note says useful training needs gradients in a manageable middle range.

With reduced motion: A static two-column diagram replaces hover effects. Each column shows five left-pointing arrows with fixed labels for the gradient size at each step, plus captions explaining the training outcome.

In many models, training works by sending an error signal backwards through the model so each parameter gets a hint about how to change. That hint is called a gradient. A gradient tells optimisation which direction should reduce the loss, meaning the measured mismatch between the model’s output and the target.

Vanishing gradients and exploding gradients are two related failure modes in that backward pass. In the vanishing case, the signal gets smaller and smaller as it moves back through many layers or many time steps. Early parts of the model then receive almost no useful update. In the exploding case, the signal grows larger and larger, so updates become erratic, oversized or numerically unstable.

They are related because both come from repeated multiplication during backpropagation, often through the chain rule. If many factors in that product are less than 1, the gradient can shrink rapidly. If many factors are greater than 1, it can grow rapidly. This is especially common in deep networks and recurrent neural networks, but the exact risk depends on the architecture, activation functions, parameter initialisation and optimiser settings.

What goes in is an input, a target and the model’s current parameters. What happens is a forward pass to compute output and loss, then a backward pass to compute gradients. What comes out should be parameter updates that gradually improve the model. With vanishing gradients, learning becomes very slow or stops in earlier layers. With exploding gradients, the loss may jump around, become huge or turn into invalid numbers such as NaN. These problems are not the same as overfitting, and they are not simply “bad data”; they are mainly about how the training signal behaves inside the model.

How “Passing correction notes down a long line” maps onto the real thing

In the story	In the AI system
the correction note moving down the line	the gradient moving backwards through layers or time steps
each person shortening the note	repeated multiplication shrinking the gradient
each person exaggerating the note	repeated multiplication enlarging the gradient
far-end reviewers barely changing labels or changing too much	early layers learning too slowly or updating unstably

A worked example

A small recurrent model predicts the next word in short text sequences, but the sequence length is 20 time steps.

Starting from: At one training step, the loss at the output is 0.8 and the backward path repeatedly multiplies by an approximate factor from the recurrent weights and activation derivative.

Vanishing case: if the typical factor is about 0.5 per time step, the gradient reaching 10 steps back is roughly 0.8 × 0.5^10 ≈ 0.00078.
By 20 steps back, it is roughly 0.8 × 0.5^20 ≈ 0.00000076, so parameters affecting early words get almost no update.
Exploding case: if the typical factor is about 1.5 per time step, the gradient 10 steps back is roughly 0.8 × 1.5^10 ≈ 46.
By 20 steps back, it is roughly 0.8 × 1.5^20 ≈ 2660, so one update can become wildly oversized unless training controls are added.

Result: In the first run, the model struggles to learn long-range patterns. In the second, the loss may swing sharply or produce unstable values. Both runs use the same data; the difference is how the backward signal behaves.

In a United Nations workplace

During a staff town hall at a Resident Coordinator's Office, a procurement officer is asked why a vendor's approved pilot for classifying non-sensitive service-desk comments learned almost nothing in one run and became unstable in another. The technical evaluation note is due for clearance by close of business.

Across many layers or time steps, backpropagation multiplies local derivatives. If their magnitudes are generally below 1, the signal reaching earlier parameters can become negligible. If they are generally above 1, it can become extremely large. The evaluation should therefore seek evidence that the vendor identified and controlled both behaviours, rather than treating a flat or erratic loss curve as a data issue.

Before you try thisBid material, vendor submissions, pricing, evaluator comments and personal data must not be uploaded to any tool not approved for procurement-sensitive material. Doing so could breach confidentiality and compromise the integrity of the process. Diagnostic tests should use approved systems and suitably controlled, non-sensitive data.

Where you may hear it

A technical clarification with a vendor before an evaluation note goes for clearance
An ICT governance board reviewing evidence from a model-training pilot
A progress review where the loss curve remains flat or suddenly spikes
A staff town hall when an unexpectedly specific question reaches the procurement officer

Avoid confusion

Often confused with

Common misconceptions

Common misconception: Vanishing gradients just mean the model has converged successfully.
Correction: Not necessarily. Convergence means training has approached a stable solution for the chosen objective. Vanishing gradients can happen far from a good solution, leaving some layers almost frozen because the backward signal became too small to guide useful updates.
Common misconception: Exploding gradients only happen when the learning rate is too high.
Correction: A high learning rate can make instability worse, but exploding gradients can arise even before the optimiser step size is applied. The core problem is that the computed gradient itself becomes very large through repeated multiplication in backpropagation.
Common misconception: These problems only affect very old models and are no longer relevant.
Correction: Modern design choices reduce the risk, but they do not remove it in every setting. Long sequences, very deep networks, poor initialisation or unsuitable activations can still produce vanishing or exploding gradients, so practitioners still monitor for them.

Quick check

Which statement best captures the difference between vanishing and exploding gradients?

Go deeper

The simple version

During training, the model sends a correction signal backwards. If that signal fades away, earlier parts barely learn. If it grows out of control, updates become unstable.

The technical version

Let the loss be L and consider gradients computed by backpropagation. For a parameter in an early layer or early time step, the derivative often becomes a product of many Jacobian terms from the chain rule. When the relevant norms of those terms are typically below 1, the product can decay exponentially with depth or sequence length, causing vanishing gradients. When the norms are typically above 1, the product can grow exponentially, causing exploding gradients. The exact behaviour depends on weight matrices, activation derivatives, normalisation, residual paths, gating and numerical precision. In practice, symptoms include tiny parameter updates in early layers, very slow learning of long-range dependencies, unstable loss spikes, gradient norms that become extreme and occasional NaN values. Common mitigations include careful initialisation, activation choices, residual connections, gated recurrent units, normalisation methods, gradient clipping and adaptive optimisers, but each changes training dynamics rather than magically solving every case.

Reviewed external resources

EncyclopaediaVanishing gradient problem (opens in a new tab)Wikipedia
A broad overview of the problem, its causes and common mitigation ideas, suited to readers who want a reference-style starting point.
Intermediatelink checked 2026-07-31
CourseReducing Loss (opens in a new tab)Google
A practical course-style explanation of reducing loss that gives context for why stable gradients matter during training, suited to readers connecting theory to training behaviour.
Intermediatelink checked 2026-07-31
VideoNeural networks, explained visually (opens in a new tab)3Blue1Brown
A visual introduction to neural networks and backpropagation that helps readers build intuition before tackling this specific failure mode, suited to visual learners.
Intermediatelink checked 2026-07-31
VideoVideo explanations: Vanishing and exploding gradients (opens in a new tab)YouTube search
A search results page collecting multiple video explanations of vanishing and exploding gradients, suited to readers who want to compare teaching styles rather than rely on one source.
search results, not one pagelink checked 2026-07-31
DiscussionCommunity discussion: Vanishing and exploding gradients (opens in a new tab)Reddit search
A search results page collecting community discussions and troubleshooting threads on these issues, suited to readers looking for practitioner questions, examples and informal explanations.
search results, not one pagelink checked 2026-07-31

Where this came from

Drafted with a language model through an offline pipeline, checked against this site's schema, and every external link above fetched to see whether it answers. Treat it as a starting point and check anything that matters against the primary sources. How this site is built and what that means.

The one thing to remember

Good training needs gradients that stay informative across depth or time, not signals that fade to almost zero or blow up to unstable size.