Learn AI /How Models Improve /Adam optimiser

Adam optimiser

Deeper
How Models Improve
Technical readers

Also called: Adam, Adam optimizer.

Detail

In five seconds

What it is

A training method that adjusts each model setting using both recent gradient direction and the recent size of past gradients.

In plain words

Adam is a way to train a model that remembers recent update directions and also remembers which settings have been getting large or small gradients, so it can move each setting by a different amount.

An everyday example

A team in Arusha, Vienna and Copenhagen is refining a shared summary of a long meeting. After each review round, they mark phrases that seem too vague or too strong. The editor does not react only to the latest comments: she remembers the overall direction of recent feedback and also notices which sections keep attracting very large corrections. For sections with repeated large swings, she makes smaller edits; for sections with lighter, steadier feedback, she is willing to change more.

Where the story stops working

A human editor can understand the meaning of the notes and choose changes deliberately. Adam does not inspect meaning in that way. It only applies numerical update rules to parameters using gradients. Also, in the story the editor may choose to ignore some comments, while Adam follows a fixed mathematical procedure unless its hyperparameters are changed.

Remember thisAdam combines a moving average of recent gradients with a moving average of their squares, corrects their early bias, and uses both to scale each parameter update.

Understand it

Why it matters

If you train or commission models, the optimiser affects cost, speed and reliability. Adam often reaches a useful result quickly without as much manual tuning as simpler methods, so teams use it a lot in practice. But it is not automatically best for every task. The choice can change whether training is stable, whether progress stalls, and how well the final model performs on new data rather than just the training set. For non-specialists, this matters because a model that looks good in a demo may depend heavily on training choices such as the optimiser and its hyperparameters.

How Adam turns gradients into parameter updates

Show why Adam uses two moving averages and how that leads to per-parameter update sizes rather than one identical step for all parameters.

01
1. Read the gradient
For each parameter, training computes a gradient from the current loss. The sign shows direction; the size shows how strongly the loss changes nearby.
Draw a column of three example parameters with gradient values such as +0.8, -0.1, +0.02. Use arrows to indicate positive and negative direction.
02
2. Update the direction memory
Adam updates a moving average of recent gradients. This smooths noisy changes and carries information about consistent direction over time.
Show a small memory box labelled 'first moment' receiving the new gradient and a faded arrow from the previous step's memory value.
03
3. Update the size memory
Adam also updates a moving average of squared gradients. This tracks how large recent gradients have been, without cancelling positive and negative values.
Show a second memory box labelled 'second moment' with squared example values, using larger circles for larger magnitudes.
04
4. Correct and scale the update
Early in training, the moving averages are biased toward zero because they started at zero, so Adam corrects that. It then divides the direction estimate by the square root of the size estimate, giving each parameter its own effective step size before applying the learning rate.
Display a formula-like layout: corrected direction over square root of corrected size, then an arrow to updated parameter values. Show one parameter getting a smaller step than another.

A left-to-right step diagram for one training iteration. Start with the current gradient for each parameter, then update two running averages, apply bias correction, and produce a scaled parameter update. Read from step 1 to step 4; a small note under each step explains the role of that stage.

Hover or tap each parameter row to highlight how the same raw learning rate leads to different final step sizes because the second-moment estimate differs.

Describe this without the diagram

This diagram explains one Adam update in four steps. First, each parameter has a gradient from the current loss, which provides a direction and magnitude. Second, Adam updates a running average of recent gradients, called the first moment, to keep a smoothed sense of direction. Third, it updates a running average of squared gradients, called the second moment, to track recent gradient size for each parameter. Fourth, because both averages started at zero, Adam applies a correction early in training, then scales the update by dividing the corrected direction estimate by the square root of the corrected size estimate and multiplying by the learning rate. The result is that different parameters can move by different amounts in the same training step.

With reduced motion: A static four-panel diagram shows the same sequence without animation: gradients in panel one, first-moment memory in panel two, second-moment memory in panel three, and the corrected, scaled parameter update in panel four.

When a model is trained, it produces an error value called a loss. From that loss, training calculates gradients: numbers that show which way each setting, or parameter, should move to reduce the loss. An optimiser is the rule for turning those gradients into actual parameter updates.

Adam combines two ideas. First, it keeps a running average of past gradients, which acts a bit like momentum and helps updates keep moving in a useful direction instead of zig-zagging. Second, it keeps a running average of squared gradients, which tracks how large recent gradients have been for each parameter.

Using those two running averages, Adam gives each parameter its own effective step size. If a parameter has had large gradients, Adam tends to scale its update down. If a parameter has had smaller gradients, Adam may allow a relatively larger update. This often helps training move faster and more steadily, especially when different parameters behave very differently.

What goes in are the current gradients, a base learning rate, and Adam's saved moving averages from earlier training steps. What happens is that Adam updates those averages, corrects for their early-step bias, and computes a new update for each parameter. What comes out is a new set of parameter values, hopefully giving lower loss on the next pass through the data.

How “Editing meeting notes with memory of past corrections” maps onto the real thing

In the story	In the AI system
recent overall direction of feedback	running average of past gradients
sections with repeated large corrections	parameters with large recent squared gradients
smaller edits in unstable sections	scaling down updates for parameters with high gradient magnitude
each section edited by a different amount	per-parameter adaptive step sizes

A worked example

For the translation-request classifier, consider one model parameter at training step 2. Its current value is 0.50, the new gradient is 0.80, and the stored Adam values after step 1 are m = 0.04 and v = 0.00016.

Starting from: Use beta1 = 0.9, beta2 = 0.999 and a learning rate of 0.001. Ignore epsilon for this numerical illustration.

Update the first moment: m = 0.9 × 0.04 + 0.1 × 0.80 = 0.116.
Update the second moment: v = 0.999 × 0.00016 + 0.001 × 0.80² = 0.00079984.
Correct the initial zero bias at step 2: m-hat = 0.116 / (1 - 0.9²) ≈ 0.6105, and v-hat = 0.00079984 / (1 - 0.999²) ≈ 0.4001.
Compute the Adam update: 0.001 × 0.6105 / sqrt(0.4001) ≈ 0.000965.
Subtract the update from the parameter: 0.50 becomes approximately 0.499035.

Result: Adam changes this parameter by about 0.000965. Plain gradient descent using only the current gradient would change it by 0.000800. Adam is not inherently more aggressive: the difference comes from its bias-corrected memories of recent direction and gradient size.

In a United Nations workplace

A reporting officer supporting a UN Country Team working group is testing a small classifier to flag translation requests as same-day or routine after a backlog threatens clearance of the weekly progress report. Training uses approved, non-sensitive request metadata, such as document type, length and stated deadline, rather than the documents themselves.

During training, some model settings may receive large or erratic gradients while others change more steadily. Adam tracks recent gradient direction and squared gradient size for each setting, allowing it to scale updates separately rather than applying the same-sized adjustment throughout the model.

Before you try thisDo not upload document text, names, contact details or other personal or sensitive material to a tool not approved for it. Minimise the training data, use an approved environment and retain human review of routing decisions, since a misclassification could delay clearance or expose protected information.

Where you may hear it

A vendor demo to an ICT governance board
A technical review before an internal model pilot
A procurement meeting comparing model-training approaches
A troubleshooting call after a training run stops improving

Avoid confusion

Often confused with

Common misconceptions

Common misconception: Adam always gives better final results than basic gradient descent.
Correction: Adam often trains faster at the start and can be easier to tune, but 'better' depends on the task, model and data. In some settings another optimiser may generalise as well or better on unseen data, even if Adam reduces training loss quickly.
Common misconception: Adam chooses the perfect learning rate automatically, so tuning is unnecessary.
Correction: Adam adapts update sizes per parameter, but it still uses a base learning rate and other hyperparameters. Poor settings can still make training unstable, too slow, or lead to a disappointing result.
Common misconception: Adam is just momentum with a different name.
Correction: Adam includes a momentum-like moving average of gradients, but it also tracks a moving average of squared gradients and uses that to scale updates per parameter. That extra mechanism is a major difference.

Quick check

What makes Adam different from plain gradient descent?

Go deeper

The simple version

Adam is a way to train a model that remembers recent update directions and also remembers which settings have been getting large or small gradients, so it can move each setting by a different amount.

The technical version

At training step t, Adam receives the gradient g_t for each parameter. It updates two exponential moving averages: m_t = beta1 m_(t-1) + (1 - beta1) g_t and v_t = beta2 v_(t-1) + (1 - beta2) g_t^2, where the square is element-wise. Because these estimates start at zero, Adam applies bias correction, typically m_hat_t = m_t / (1 - beta1^t) and v_hat_t = v_t / (1 - beta2^t). The parameter update is then theta_t = theta_(t-1) - alpha m_hat_t / (sqrt(v_hat_t) + epsilon), with division, square root and epsilon applied element-wise. Intuitively, m_hat_t provides a smoothed direction and v_hat_t normalises by recent gradient magnitude, producing adaptive per-parameter step sizes.

Reviewed external resources

EncyclopaediaWikipedia search: Adam optimiser (opens in a new tab)Wikipedia
A search results page on Wikipedia that helps readers find encyclopedia-style background on the term, suited to readers who want a starting point rather than one guaranteed article.
search results, not one pagelink checked 2026-07-31
CourseReducing Loss (opens in a new tab)Google
A course page about reducing loss that suits readers who want a structured introduction to how training updates improve model performance.
Intermediatelink checked 2026-07-31
VideoNeural networks, explained visually (opens in a new tab)3Blue1Brown
A visual video explanation of neural networks that is useful for readers who want intuition about gradients and training before tackling optimiser details.
Intermediatelink checked 2026-07-31
VideoVideo explanations: Adam optimiser (opens in a new tab)YouTube search
A YouTube search results page that helps readers compare different video explanations of Adam, suited to readers who prefer to choose a teaching style that fits them.
search results, not one pagelink checked 2026-07-31
DiscussionCommunity discussion: Adam optimiser (opens in a new tab)Reddit search
A Reddit search results page that shows community discussions and practical questions about Adam, suited to readers who want informal troubleshooting and differing viewpoints.
search results, not one pagelink checked 2026-07-31

Where this came from

Drafted with a language model through an offline pipeline, checked against this site's schema, and every external link above fetched to see whether it answers. Treat it as a starting point and check anything that matters against the primary sources. How this site is built and what that means.

The one thing to remember

Adam combines a moving average of recent gradients with a moving average of their squares, corrects their early bias, and uses both to scale each parameter update.