Learn AI /How Models Improve /Regularisation
Regularisation
- Deeper
- How Models Improve
- Technical readers
Also called: regularization, weight decay.
Back to the milestone you came from
In five seconds
What it is
Regularisation changes a model's training objective or procedure to reduce overfitting and improve performance beyond the training data.
In plain words
Regularisation changes training to reduce overfitting. It may add a penalty or use another method so the model usually works better on new data.
An everyday example
Imagine adjusting every slider on an equaliser until one song sounds perfect. Extreme settings may correct tiny quirks in that recording but make the next song sound worse. Adding a cost for extreme slider positions encourages a more restrained setting. The first song may sound marginally less perfect, but a wider range of songs sounds better. Regularisation similarly discourages a model from making large or numerous adjustments merely to match its training examples.
Real regularisation does not literally judge whether a pattern is suspicious or faulty. It changes a mathematical objective during training. Also, not all regularisation works by discouraging large weights: dropout, early stopping, and data augmentation act differently, so the 'cost on adjustments' story fits some methods better than others.
Remember thisRegularisation alters training to discourage overfitting. Its strength is tuned on held-out data, since too little may not help and too much can cause underfitting.
This term is contested
The broad idea is widely accepted, but terminology can vary. Some people use 'regularisation' narrowly for penalty terms such as L1 or L2, while others use it more broadly to include methods like dropout, early stopping, and data augmentation. 'Weight decay' is also often treated as equivalent to L2 regularisation, although in some optimiser implementations the update rule is not exactly the same.
Understand it
Understand it
Why it matters
If you only look at how well a model matches its training data, you can choose a system that looks excellent during training but performs poorly in real use. Regularisation matters because it helps teams prefer models that are more dependable on new cases, not just impressive on old ones. For non-specialists, this affects procurement, evaluation, risk, and trust: a slightly less perfect training score may be the safer operational choice if it means fewer surprises later.
Training without and with regularisation
Show how adding a penalty changes the training objective and can improve performance on new data even if training fit becomes slightly worse.
01
Same starting point
Both columns start with the same training data and the same model architecture.
Draw two equal columns labelled 'No regularisation' and 'With regularisation'. At the top of both, show the same dataset icon feeding into the same model icon.
02
Different objective
The left column uses prediction loss only. The right column uses prediction loss plus a penalty that discourages large or overly flexible parameter values.
In the left column, show a box labelled 'optimise prediction loss'. In the right column, show a box labelled 'optimise prediction loss + penalty'. Add a small side note: 'penalty strength = hyperparameter'.
03
Parameter updates
Optimisation updates the weights in both cases, but the regularised path pushes against very large weights.
Show arrows from each objective box to weight sliders. In the right column, add a counter-force arrow or spring icon opposing extreme slider positions.
04
Result on training data
The unregularised model often reaches lower training loss because it can fit more detail, including noise.
At the bottom left, show a training-loss badge reading 'very low'. At the bottom right, show a training-loss badge reading 'slightly higher'.
05
Result on new data
The regularised model often performs better on unseen examples because it relies less on accidental patterns in the training set.
Below both columns, add a test-data row. Show the left model with a 'less stable on new data' badge and the right model with a 'more stable on new data' badge.
A side-by-side diagram compares two training paths. The left path minimises prediction loss alone. The right path minimises prediction loss plus a regularisation penalty. Read from top to bottom in each column, then compare the final models and their behaviour on training versus new data.
Hovering or tapping each column highlights the objective used and swaps a small caption between 'fits training data closely' and 'generalises better to new data'.
Describe this without the diagram
This diagram compares two training processes side by side. Both begin with the same dataset and the same model. On the left, training minimises prediction loss only. On the right, training minimises prediction loss plus a penalty that discourages extreme parameter values. The left path ends with very low training loss but weaker behaviour on new data. The right path ends with slightly higher training loss but better performance on unseen data. The main point is that regularisation changes the objective during training so the model is less likely to fit noise.
With reduced motion: A static two-column comparison replaces any highlighting. Each column shows the same starting dataset and model, a different objective box, then a final note: left column, lower training loss; right column, better generalisation on new data.
When a model is trained, optimisation reduces a loss: a number that measures how wrong its predictions are on the training data. Regularisation changes the objective or another part of the training procedure to reduce overfitting.
What goes in is the usual training data, the model, and one or more regularisation settings. A weight is a number inside the model that affects its output. A common method adds a penalty to the loss when weights become large. Another regularisation method, dropout, switches off some units temporarily during training, but it is a different technique from weight decay.
With penalty-based regularisation, optimisation balances two aims: fit the data and avoid overly complex settings. The combined loss might be written as prediction loss plus a penalty term. If the penalty is too weak, the model may still fit noise. If it is too strong, the model can become too simple and miss real patterns.
What comes out is usually a model that performs a little less perfectly on the training set but better on new data. In other words, regularisation often trades some training accuracy for better generalisation. It does not guarantee improvement, and regularisation settings are hyperparameters that need testing.
How “Adjusting an equaliser for more than one song” maps onto the real thing
| In the story | In the AI system |
|---|---|
| large adjustments carry a cost | a penalty term discourages large weights or overly complex settings |
| matching every odd jump in last month's data | overfitting to noise in the training data |
| the rule that is more reliable next month | better generalisation to unseen data |
A worked example
A programme officer in a regional bureau is testing a classifier that assigns themes to cleared project summaries before the first reading of a progress report.
Starting from: The same classifier is trained twice on labelled summaries, with separate validation and test sets. One version has no penalty; the other uses L1 regularisation.
- The first version minimises classification loss alone.
- The second minimises classification loss plus λ||w||₁, which penalises the absolute size of its weights and can drive less useful weights to zero.
- Several values of λ are compared on the validation set. The choice is based on performance beyond the training data, not on which version fits the training examples most closely.
- The selected regularised model is then checked once on the unseen test set.
Result: The regularised classifier may fit the labelled summaries slightly less closely while assigning themes more reliably to new summaries. If the penalty is too strong, however, it will underfit, so λ still requires validation.
In a United Nations workplace
A reporting officer in a Headquarters department is testing a model to code open-ended answers from a needs assessment into themes before the monitoring report goes for clearance.
Without regularisation, the model may fit incidental wording in the earlier answers rather than patterns that transfer to new responses. Adding a penalty during training can reduce that overfitting, with the strength selected against held-out data.
Before you try thisNeeds-assessment answers may contain personal or protection-sensitive details. Raw responses must not be entered into a tool that has not been approved for that material. Regularisation does not prevent memorisation, provide anonymisation or make an unsuitable service safe.
Where you may hear it
- During a validation-results review before model sign-off
- At an ICT governance board discussion of a proposed model
- In a vendor demo when weight decay settings appear on screen
- During a technical first reading of the training configuration
Avoid confusion
Avoid confusion
Often confused with
Common misconceptions
Common misconception: Regularisation simply makes a model smaller.
Correction: Sometimes it leads to smaller weights, but the main point is to change training so the model avoids overly specific solutions. A model can keep the same architecture and still be regularised through penalties, dropout, or other methods.
Common misconception: If regularisation raises training loss, it must be making the model worse.
Correction: Not necessarily. Training loss measures fit to examples already seen. Regularisation often accepts a small rise there to improve performance on unseen data, which is usually the real goal.
Common misconception: Weight decay, L2 regularisation, and dropout are all identical.
Correction: They are related but not identical. L2 regularisation adds a penalty on weight size to the objective. Weight decay is often used as a close synonym, though in some optimisers the exact update differs. Dropout instead randomly removes units or connections during training, so it works through a different mechanism.
Quick check
What is the main purpose of regularisation during model training?
Regularisation is mainly used to improve generalisation by reducing overfitting. Depending on the method, it may add a penalty, alter training examples, or change the training procedure. It does not guarantee a global minimum or require collecting more data.
Go deeper
Go deeper
The simple version
Regularisation changes training to reduce overfitting. It may add a penalty or use another method so the model usually works better on new data.
The technical version
Regularisation modifies the optimisation objective or training procedure to reduce overfitting and improve generalisation. In the common penalty-based form, the objective becomes empirical loss plus a term such as λ||w||² for L2 or λ||w||₁ for L1, where w are parameters and λ controls strength. This biases training toward solutions with smaller or sparser weights. In deep learning, 'weight decay' is often used as a near-synonym for L2 regularisation, but the equivalence depends on the optimiser implementation; for example, decoupled weight decay in Adam is not identical to simply adding an L2 penalty to the loss. Broader uses of the word regularisation also include methods such as dropout, early stopping, label smoothing, and data augmentation, which reduce overfitting through different mechanisms rather than the same penalty term.
Reviewed external resources
EncyclopaediaRegularization (opens in a new tab)Wikipedia
This link gives a broad reference overview of regularisation methods and terminology, and it suits readers who want definitions and examples in one place.
Intermediatelink checked 2026-07-31
CourseReducing Loss (opens in a new tab)Google
This course material gives a practical introduction to reducing loss and improving model behaviour, and it suits readers who want applied training guidance.
Intermediatelink checked 2026-07-31
VideoNeural networks, explained visually (opens in a new tab)3Blue1Brown
This video gives visual intuition for how neural network training behaves, and it suits readers who learn best from geometric and animated explanations.
Intermediatelink checked 2026-07-31
VideoVideo explanations: Regularisation (opens in a new tab)YouTube search
This search results page gives a range of video explanations of regularisation from different creators, and it suits readers who want to compare teaching styles before choosing one.
search results, not one pagelink checked 2026-07-31
DiscussionCommunity discussion: Regularisation (opens in a new tab)Reddit search
This search results page gives community discussions and informal explanations of regularisation, and it suits readers who want practical questions, caveats, and varied viewpoints.
search results, not one pagelink checked 2026-07-31
Where this came from
Drafted with a language model through an offline pipeline, checked against this site's schema, and every external link above fetched to see whether it answers. Treat it as a starting point and check anything that matters against the primary sources. How this site is built and what that means.
The one thing to remember
Regularisation alters training to discourage overfitting. Its strength is tuned on held-out data, since too little may not help and too much can cause underfitting.