Learn AI /How Models Improve /Learning rate
Learning rate
- Intermediate
- How Models Improve
- Technical readers
Also called: step size.
Back to the milestone you came from
In five seconds
What it is
A training setting that controls how far model parameters move after each error-based update.
In plain words
It is the dial that controls how big each training adjustment should be after the model makes mistakes.
An everyday example
A team is sorting public feedback letters into themes such as transport, health and education. After checking a batch, they notice many letters were put in the wrong pile, so they decide to adjust their sorting guide. One manager says to change the guide only a little each time, while another suggests rewriting large parts at once. Small edits make progress slow; huge edits make the guide swing from one mistake pattern to another.
A human team can reason about why a guide should change, while a model update usually follows numerical gradients without human-style judgement. Also, real training often changes thousands or millions of parameters at once, not one guide document.
Remember thisThe learning rate scales each error-based update. Too low can make training impractically slow; too high can overshoot useful parameter values and prevent stable convergence.
Understand it
Understand it
Why it matters
Learning rate strongly affects cost, speed and reliability. If it is poorly chosen, a project may waste compute, miss deadlines or produce a model that appears to train but never becomes stable enough for trustworthy use.
Small, balanced and oversized updates
Show how the same gradient leads to slow, stable or unstable training depending on update size.
01
Same starting point
All three panels begin at the same position on the same loss curve, so only the update size changes.
Draw three identical U-shaped curves with a dot near the upper left in each panel.
02
Small learning rate
Tiny updates follow the downhill direction but need many steps to get close to the minimum.
In the left panel, show 5 to 6 short arrows moving gradually towards the bottom.
03
Balanced learning rate
Medium updates still go downhill but reach a low-loss region in fewer steps.
In the middle panel, show 3 to 4 medium arrows landing near the bottom without crossing it wildly.
04
Too-large learning rate
Oversized updates overshoot the low point, so loss can oscillate or diverge instead of settling.
In the right panel, show long arrows crossing the bottom repeatedly from left to right.
Three side-by-side panels show a ball moving down the same loss curve. Left: tiny steps inch towards the bottom. Middle: moderate steps reach the bottom efficiently. Right: oversized steps jump past the bottom and bounce between sides. Each jump is labelled as gradient multiplied by learning rate.
On hover or tap, each panel reveals a short caption: slow progress, efficient progress, or overshooting. An optional slider can increase step size and update the arrow lengths in all panels.
Describe this without the diagram
The diagram compares three training runs on the same bowl-shaped loss curve. In all cases, the optimiser starts from the same point and follows the same downhill direction. With a small learning rate, the parameter updates are very short, so the model approaches lower loss slowly. With a moderate learning rate, the updates are large enough to make good progress and reach a low-loss region efficiently. With a too-large learning rate, the updates jump past the low point and keep crossing it, which can make training unstable.
With reduced motion: A static three-panel comparison replaces animation. Each panel shows the full path as fixed arrows: many short arrows for a small learning rate, a few medium arrows for a balanced learning rate, and long crossing arrows for a too-large learning rate.
When a model is trained, it produces an output. A loss function turns how well that output meets a training target or objective into a number the training process can reduce.
Next, the training method calculates a gradient. A gradient shows how the loss changes as each parameter changes. A gradient-based optimiser usually moves against it to reduce the loss. The learning rate scales that movement.
If the learning rate is too small, training can be very slow. The model may improve, but only after many updates. If it is too large, the updates can overshoot the better region and make loss bounce around or even grow.
So the input is the gradient and the current parameter values, the learning rate scales the update, and the output is a new set of parameter values. The aim is steady progress towards lower loss, not the biggest jump on any single step.
How “Adjusting labels in a feedback-sorting office” maps onto the real thing
| In the story | In the AI system |
|---|---|
| the sorting guide being edited | model parameters being updated |
| checking the wrongly sorted letters | measuring loss from prediction errors |
| how much the guide is changed after each review | learning rate scaling each update |
| swinging from one mistake pattern to another | overshooting and unstable training |
A worked example
In an approved test environment, a data specialist trains a simple classifier to route synthetic application summaries for manual eligibility review.
Starting from: One parameter has a current value of 0.80 and a gradient of 0.50. The update rule is: new value = old value minus learning rate × gradient.
- At a learning rate of 0.01, the update is 0.005, giving a new parameter value of 0.795. Progress may be steady but slow.
- At a learning rate of 0.1, the update is 0.05, giving a new value of 0.75. This is a larger step in the gradient direction.
- At a learning rate of 1.2, the update is 0.60, giving a new value of 0.20. Such a large step may pass a lower-loss region.
- If later gradients reverse direction after repeated overshooting, the loss may oscillate instead of converging steadily.
Result: The gradient is unchanged, but the learning rate determines the size of the update. A suitable value supports stable convergence; one that is too small slows training, while one that is too large can make it unstable.
In a United Nations workplace
A budget officer in a specialised agency technical division is supporting the review of a long list of consultant applications against an approved roster profile. A data specialist tests several learning rates while retraining an internal classifier on synthetic examples before the shortlist is sent to the chief of section.
If the learning rate is too low, training may not converge in time to support the review. If it is too high, the loss may jump or oscillate, leaving performance unstable. The setting affects training behaviour, not whether the resulting classifier is fair, accurate or suitable for screening.
Before you try thisReal applications contain personal and HR-sensitive information. They must not be entered into a public or otherwise unapproved AI tool for tuning or testing. Any approved classifier should support documented human review, not determine eligibility or shortlisting on its own.
Where you may hear it
- A vendor demo to an ICT governance board
- A technical review before an internal model is cleared for testing
- A progress meeting with the data specialist retraining a classifier
- A procurement unit's discussion of model-training requirements
Avoid confusion
Avoid confusion
Often confused with
Momentum
A training method that carries part of the previous update into the next one so parameter changes keep moving in useful directions and wobble less.
Hyperparameter
A hyperparameter is a training setting chosen before or during learning that shapes how a model is fitted, rather than a value the model learns from data.
Common misconceptions
Common misconception: A higher learning rate always means faster and better training.
Correction: A larger value can reduce loss quickly at first, but if it is too large it can overshoot good regions and make training unstable or worse overall.
Common misconception: The learning rate tells the model how much data to learn from.
Correction: It does not control dataset size or model capacity. It only scales the size of each parameter update during optimisation.
Common misconception: If loss is going down, the learning rate must be correct.
Correction: Loss can decrease while training is still inefficient. A very small value may work but take far longer than necessary, and a slightly too-large value may look fine before later instability appears.
Common misconception: Learning rate and momentum are the same setting.
Correction: Learning rate controls update size. Momentum changes how past gradients influence the current update direction and can smooth noisy steps.
Quick check
What is the main role of the learning rate during training?
The learning rate scales the update applied after computing the gradient. Dataset size, loss-function choice and accuracy measurement are separate parts of training.
Go deeper
Go deeper
The simple version
It is the dial that controls how big each training adjustment should be after the model makes mistakes.
The technical version
In iterative optimisation, the learning rate is a hyperparameter that scales the gradient-based parameter update, such as θ_new = θ_old − η∇L, where η sets the step size and strongly affects convergence behaviour.
Reviewed external resources
EncyclopaediaLearning rate (opens in a new tab)Wikipedia
This page gives a broad reference-style overview of the concept and related terms, which suits readers who want definitions and background.
Intermediatelink checked 2026-07-31
CourseReducing Loss (opens in a new tab)Google
This course material focuses on reducing loss during training, which suits readers who want a practical introduction to how tuning affects improvement.
Intermediatelink checked 2026-07-31
VideoNeural networks, explained visually (opens in a new tab)3Blue1Brown
This video gives an intuitive visual sense of gradient-based training, which suits readers who learn best from geometric explanations.
Intermediatelink checked 2026-07-31
VideoVideo explanations: Learning rate (opens in a new tab)YouTube search
This search results page helps readers find multiple video explanations of learning rate from different teachers, which suits people comparing presentation styles rather than seeking one authoritative source.
search results, not one pagelink checked 2026-07-31
DiscussionCommunity discussion: Learning rate (opens in a new tab)Reddit search
This search results page shows community discussions and troubleshooting examples about learning rate, which suits readers looking for practical questions and informal experiences.
search results, not one pagelink checked 2026-07-31
Where this came from
Drafted with a language model through an offline pipeline, checked against this site's schema, and every external link above fetched to see whether it answers. Treat it as a starting point and check anything that matters against the primary sources. How this site is built and what that means.
The one thing to remember
The learning rate scales each error-based update. Too low can make training impractically slow; too high can overshoot useful parameter values and prevent stable convergence.