Learn AI /How Models Improve /Optimisation
Optimisation
- Intermediate
- How Models Improve
- Technical readers
Also called: optimizer, optimiser, optimization.
Back to the milestone you came from
In five seconds
What it is
The process of adjusting a model’s internal settings to reduce loss, plus the algorithms and rules used to choose each adjustment.
In plain words
Optimisation is the repeated process of checking how wrong a model is and adjusting its settings to make it less wrong.
An everyday example
A team handles incoming messages and uses a checklist to decide which ones are urgent. At first, the checklist mislabels too many messages, so the team reviews yesterday’s mistakes. They notice, for example, that the rule about time-sensitive wording is too weak and the rule about formal tone is too strong, so they adjust both a little. They repeat this review-and-adjust cycle each day until the checklist makes fewer mistakes overall.
A human team can explain why it changed a checklist rule in plain language. In many models, especially neural networks, the parameters are numerous and not naturally interpretable one by one. Also, real optimisation often updates thousands or millions of parameters from small batches of data using mathematics, not a single daily review of obvious mistakes.
Remember thisOptimisation is the process of adjusting model parameters to reduce an objective, while the optimiser is the specific update rule used at each step.
Understand it
Understand it
Why it matters
If you hear that a model was trained with a certain optimiser, learning rate or number of steps, that is not minor engineering detail. These choices affect cost, speed, stability and what kind of errors remain. A model can fail because the data are poor, but it can also fail because the optimisation process is unstable, too slow, or trapped in a poor solution. For non-specialists, this matters when assessing whether a system was developed carefully, whether more training is likely to help, and whether claims about performance are robust or just the result of lucky tuning on one setup.
How optimisation reduces loss over repeated updates
After reading the diagram, the learner should be able to describe optimisation as a repeated loop: measure loss, compute update directions, change parameters, and check whether improvement continues.
01
1. Start with current parameters
The model begins a training step with its present weights and a batch of training examples.
Show a box labelled 'Current model parameters' with a smaller note underneath: 'weights at this step'.
02
2. Measure error as loss
The model makes predictions for the batch and the loss function converts the mistakes into a number to reduce.
Show a box labelled 'Predictions + loss' with an icon of input data entering and a numeric loss value leaving.
03
3. Work out which way to change
Using backpropagation, the system estimates a gradient for each parameter: which changes would likely raise or lower the loss.
Show a box labelled 'Gradients' with small up and down arrows next to parameter symbols.
04
4. Apply an optimiser
The optimiser uses the gradients and its own rules, such as learning rate or momentum, to choose the next parameter update.
Show a box labelled 'Update rule' with a formula-style caption: 'uses gradients + settings'.
05
5. Repeat and watch progress
The new parameters are used on the next batch. If loss keeps dropping and then stabilises, training may be converging.
Show a box labelled 'New parameters' with a smaller trend line sloping down, then add a curved arrow back to step 2.
A left-to-right step diagram showing one training loop. Each box names one stage and a small output from that stage. Arrows connect the stages and then return to the start to show repetition. Read from left to right, then follow the final arrow back to the first box.
On hover or tap, each step reveals one sentence about what goes in and what comes out of that stage; the return arrow highlights to emphasise repetition.
Describe this without the diagram
This diagram explains optimisation as a repeated five-step loop. Step 1: begin with the model’s current parameters and a batch of training data. Step 2: run the model and compute a loss value that measures error. Step 3: calculate gradients, which estimate how changing each parameter would affect the loss. Step 4: an optimiser combines those gradients with settings such as learning rate, and sometimes momentum, to choose an update. Step 5: the model now has new parameters, and the loop repeats on more data. Over many cycles, the loss often decreases and the updates may become smaller, which can indicate convergence.
With reduced motion: A static five-box sequence with arrows from step 1 to step 5 and a final curved arrow returning to step 2, plus a small downward trend line beside the last box to indicate repeated improvement over training.
In machine learning, optimisation means trying to make a model less wrong by changing its internal settings, often called parameters or weights. You first need a way to measure how wrong the model is. That measure is usually called a loss function.
What goes in? A model, training data, a loss value, and a method for updating the parameters. That method may be called an optimiser or optimizer; the spelling differs by region, but the idea is the same. Optimisation is the overall process. An optimiser is the specific update rule, such as stochastic gradient descent or Adam.
What happens? The system computes the loss on some training examples, estimates how small parameter changes would affect it, and then nudges the parameters in a direction expected to lower it. These estimated sensitivities form a gradient, which describes the loss change for a small change in each parameter.
What comes out? Usually, a new set of parameter values expected to lower loss on the training data. Repeating this many times may lead to convergence, meaning the parameters or loss approach stable values under a chosen stopping rule. Lower training loss does not guarantee better performance on new data, so optimisation is only one part of building a useful model.
How “Sorting urgent and non-urgent messages” maps onto the real thing
| In the story | In the AI system |
|---|---|
| the checklist rules | the model’s parameters or weights |
| reviewing yesterday’s mistakes | measuring loss on training examples |
| seeing which rule was too weak or too strong | estimating gradients for different parameters |
| adjusting the rules a little each day | an optimiser updating parameters step by step |
A worked example
A simplified linear model predicts whether a message is urgent from two signals: deadline wording and a routine greeting.
Starting from: Prediction = 0.8 × deadline signal + 0.6 × greeting signal. One labelled example says "Please send by 16:00", giving signals 1 and 0, with target urgency 1. For this illustration, loss is half the squared prediction error.
- The model predicts 0.8. Its loss is 0.5 × (0.8 − 1)² = 0.02.
- The gradient for the deadline weight is (0.8 − 1) × 1 = −0.2. The gradient for the greeting weight is zero because that signal is absent.
- With a learning rate of 0.1, gradient descent updates the deadline weight to 0.8 − 0.1 × (−0.2) = 0.82. The greeting weight remains 0.6.
- The new prediction is 0.82 and the loss is 0.5 × (0.82 − 1)² = 0.0162, which is lower.
- Repeating this process across batches of labelled examples adjusts both weights to reduce average loss. A different optimiser or learning rate could produce different update paths.
Result: Optimisation moved the parameters in a direction that reduced the stated loss. The gradient indicated the direction, while the optimiser and learning rate determined the actual update.
In a United Nations workplace
At close of business before rotating out of a regional bureau, a desk officer finalises a handover note on a pilot that classifies incoming requests as requiring action or background only. During training, the model processes batches of reviewed examples, calculates how its errors affect its parameters and uses an optimiser to update them. The handover records the optimiser and learning rate used, rather than merely saying that the model was trained.
The incoming colleague needs those details to reproduce the training run and understand why another run may learn faster, become unstable or reach a different result. Optimisation covers the process of reducing the objective; the optimiser is the rule used for each parameter update.
Before you try thisA handover note or mailbox may contain names, internal correspondence and sensitive assessments. Do not upload either to a tool that has not been approved for that material. Training examples must be authorised, minimised and appropriately protected, with human review retained because a lower training loss does not prevent consequential misrouting.
Where you may hear it
- A vendor demo to an ICT governance board, when the training slide names the optimiser and learning rate
- A procurement unit's technical evaluation, when bidders explain how their training run converged
- A model review at Headquarters after the loss begins oscillating instead of falling
- A regional bureau handover briefing on whether a pilot can be reproduced
Avoid confusion
Avoid confusion
Often confused with
Gradient descent
A method for reducing a model’s loss by repeatedly adjusting its internal settings in the direction opposite the loss gradient.
Backpropagation
A method that propagates derivatives backwards through a model to compute how changes to each weight would affect the loss.
Loss function
A loss function turns a model’s output and the correct target into a single number that shows how wrong the model was for that example or batch.
Gradient
A gradient tells you how much a model’s error would change if each setting were nudged slightly up or down.
Common misconceptions
Common misconception: Optimisation and gradient descent are the same thing.
Correction: Gradient descent is one family of methods used in optimisation. Optimisation is the broader goal and process of reducing loss. Many optimisers exist, including variants with momentum, adaptive step sizes, or constraints.
Common misconception: The optimiser is the model.
Correction: The model produces predictions from its current parameters. The optimiser does not make predictions; it updates those parameters during training. After training, the optimiser may no longer be needed for inference.
Common misconception: If optimisation lowers training loss, the model is necessarily getting better.
Correction: Lower training loss means the model fits the training data better under the chosen loss function. It may still perform poorly on new data if it overfits, if the loss misses important real-world goals, or if the data are unrepresentative.
Common misconception: There is one best optimiser for all tasks.
Correction: Different optimisers behave differently depending on model architecture, data scale, batch size and tuning choices. An optimiser that works well for one task may be unstable or inefficient for another.
Quick check
Which statement best distinguishes optimisation from an optimiser?
Optimisation is the broad task and repeated procedure of improving model parameters to reduce an objective. An optimiser is the rule that chooses parameter updates, often using gradients. The loss function defines what to reduce, gradients describe local changes, and backpropagation is one way to compute them.
Go deeper
Go deeper
The simple version
Optimisation is the repeated process of checking how wrong a model is and adjusting its settings to make it less wrong.
The technical version
Formally, optimisation in machine learning seeks parameter values that minimise an objective, usually a loss function over training data, sometimes plus regularisation terms. A common process iterates over batches of examples, computes gradients of the objective with respect to parameters, often via backpropagation and the chain rule, and applies an update rule. That update rule is the optimiser: for example, stochastic gradient descent, SGD with momentum, or Adam. The learning rate controls update size; momentum and adaptive methods modify how past or per-parameter gradient information affects the step. In non-convex problems such as deep neural networks, optimisation does not guarantee reaching a global minimum, and practical concerns include convergence speed, numerical stability, sensitivity to hyperparameters, and issues such as vanishing or exploding gradients. The term 'optimization' is the same concept in another spelling.
Reviewed external resources
EncyclopaediaMathematical optimization (opens in a new tab)Wikipedia
This link gives a broad mathematical view of optimisation and suits readers who want formal background and terminology beyond machine learning.
Intermediatelink checked 2026-07-31
CourseReducing Loss (opens in a new tab)Google
This course page focuses on reducing loss during training and suits beginners to intermediate readers who want a practical walkthrough.
Intermediatelink checked 2026-07-31
VideoNeural networks, explained visually (opens in a new tab)3Blue1Brown
This video gives an intuitive visual picture of how neural network training changes parameters and suits visual learners.
Intermediatelink checked 2026-07-31
VideoVideo explanations: Optimisation (opens in a new tab)YouTube search
This search results page offers many video explanations of optimisation from different teachers and suits readers comparing styles and difficulty levels.
search results, not one pagelink checked 2026-07-31
DiscussionCommunity discussion: Optimisation (opens in a new tab)Reddit search
This search results page surfaces community discussions about optimisation and suits readers who want informal explanations, examples and common confusions.
search results, not one pagelink checked 2026-07-31
Where this came from
Drafted with a language model through an offline pipeline, checked against this site's schema, and every external link above fetched to see whether it answers. Treat it as a starting point and check anything that matters against the primary sources. How this site is built and what that means.
The one thing to remember
Optimisation is the process of adjusting model parameters to reduce an objective, while the optimiser is the specific update rule used at each step.