Learn AI /Training and Model Adaptation /LoRA

LoRALoRA

Also called: low-rank adaptation, adapter.

Detail

In five seconds

What it is

A method for adapting a large model by training a small set of added weights instead of changing most of the original model.

In plain words

LoRA lets you customise a large model by adding and training a small attachment instead of editing the whole model.

An everyday example

A transport planner in Santiago has a large printed map of a city. The base map already shows roads, bridges and districts, so they do not redraw it. For a festival weekend, they place a few transparent overlays on top that highlight temporary bus lanes, road closures and preferred detours. The original map stays the same, but the overlays change how someone uses it for this specific situation.

Where the story stops working

A map overlay is manually designed and easy to inspect, while LoRA weights are learned numerically through optimisation and are not human-readable route notes. Also, real LoRA changes happen inside matrix operations in multiple layers, not as a separate visible sheet sitting on top of the model.

Remember thisLoRA freezes the base model and learns small, low-rank updates in selected layers, reducing the parameters and storage needed for task-specific adaptation.

This term is contested

People often use "adapter" loosely to include LoRA, but in technical discussions "adapter" can also refer to other architectures, so the labels are related but not always identical.

Understand it

Why it matters

LoRA matters because it changes the cost and governance of model adaptation. If an organisation wants a model tailored to a narrow task, full fine-tuning may require more computing, storage and model management. LoRA can make experimentation cheaper and faster, allow several task-specific adapters to share one base model, and make rollback easier because the base checkpoint stays unchanged. For non-specialists, this affects procurement, deployment choices, security review, version control and whether a system can be updated for local needs without maintaining many full model copies. It is not a guarantee of quality or safety, but it is an important practical option.

Full fine-tuning versus LoRA adaptation

Show why LoRA changes a model by training a small added component while leaving the original model weights unchanged.

01
Start from the same base model
Both approaches begin with an existing pre-trained checkpoint.
Draw two equal columns. At the top of each, place a large rounded rectangle labelled 'Base model checkpoint'. Use the same colour and size in both columns.
02
Choose what can change
Full fine-tuning updates most or all weights. LoRA freezes the base weights and adds small trainable matrices to selected layers.
In the left column, show the model block with many small highlighted internal cells labelled 'all weights trainable'. In the right column, grey out the main model labelled 'frozen' and attach small coloured side blocks to two or three layers labelled 'LoRA A/B matrices'.
03
Train on task data
Both methods use training examples, but the optimiser updates very different numbers of parameters.
Below each model, draw an input dataset icon feeding into a training loop arrow. On the left, the arrow returns to the full model. On the right, the arrow returns only to the coloured LoRA side blocks.
04
Save the result
Full fine-tuning often produces another large checkpoint. LoRA usually produces a much smaller adapter file, which can later be merged or loaded with the base model.
At the bottom, draw an output box for the left labelled 'new full checkpoint: large'. Draw an output box for the right labelled 'LoRA adapter: small' plus a note 'use with base model or merge later'.

A side-by-side diagram with two columns. The left column shows full fine-tuning updating the whole model. The right column shows the same base model frozen, with small LoRA modules attached to selected layers and only those modules updated. Read from top to bottom in each column, then compare storage and training effort at the bottom.

Hover or tap each highlighted area to reveal a short note: what stays frozen, what gets updated, and what file is saved afterwards.

Describe this without the diagram

This diagram compares two ways to adapt a model. In the left column, full fine-tuning starts from a base model checkpoint, makes most or all of its weights trainable, updates the whole model during training, and saves a large new checkpoint. In the right column, LoRA starts from the same base model checkpoint, freezes the original weights, attaches small trainable LoRA matrices to selected layers, updates only those added parts during training, and saves a small adapter file that is used with the base model or merged into it later. The key comparison is that both methods aim to adapt the model, but LoRA changes far fewer parameters and usually needs less storage.

With reduced motion: A static two-column comparison remains visible at all times, with labels showing which weights are trainable, which are frozen, and what output file each approach produces.

LoRA is a way to adjust a pre-trained model for a new task without rewriting the whole model. You start with an existing model checkpoint, keep its original weights frozen, and add small trainable parts to selected layers. In machine learning, weights are the numbers that shape how the model produces outputs.

During training, the model still processes the input in the usual way, but the added LoRA parts slightly shift the calculations. These added parts are designed with low rank, which means they represent changes in a compressed form using far fewer numbers than a full weight update. The aim is to capture the most useful adjustment directions without storing a complete second copy of the model.

What goes in is usually a base model, a training dataset, and choices about where to attach LoRA modules, such as attention layers. What happens is optimisation: the system repeatedly compares outputs with target examples and updates only the small added weights. What comes out is either a compact LoRA adapter file or a merged model where those learned changes are folded into the base weights.

LoRA is often confused with fine-tuning in general. It is a type of parameter-efficient fine-tuning, not a separate stage of learning like pretraining. It is also often called an adapter, although people use "adapter" more broadly for several related methods, so the terms are not always exact synonyms.

How “Adding route overlays to a city map” maps onto the real thing

In the story	In the AI system
the printed base map	the frozen pre-trained model
transparent overlays	the added trainable LoRA weights
festival-specific route changes	task-specific adaptation learned from training data
using overlays without redrawing every street	adapting behaviour without updating all model parameters

A worked example

An evaluation office needs an approved 7-billion-parameter model to classify recommendations in monitoring reports by theme before an annual synthesis is circulated. It uses LoRA rather than creating a fully fine-tuned copy of the model.

Starting from: Base model: 7 billion parameters. LoRA rank: 16 on selected attention projections. Training set: 50,000 approved, labelled report passages.

Load the base checkpoint and freeze its original weight matrices, so optimisation does not change them.
Insert trainable factor matrices A and B into selected attention projections. With shared dimension 16, the update BA has rank at most 16.
Train on the labelled passages. Gradients update only A and B, reducing the trainable parameter count and associated optimiser state.
Save the learned adapter separately. At deployment, load it with the matching base checkpoint or merge the update into a validated deployment copy.

Result: The evaluation office receives a comparatively small task-specific adapter rather than another full 7-billion-parameter checkpoint. The original base model remains available for other tasks, while the adapter can be versioned and tested separately.

In a United Nations workplace

A communications officer supporting a United Nations Country Team working group is finalising a handover note for an incoming colleague before close of business. An approved internal model uses a LoRA adapter to flag departures from agreed terminology in draft joint talking points.

The adapter can learn this narrow drafting pattern while the base model remains frozen. It is smaller to store, version and replace than a fully fine-tuned model, which is useful when agreed terminology changes but the approved base model does not.

Before you try thisUse only approved systems and cleared training material. Do not train or test the adapter on unpublished positions, personal data, confidential negotiations or protection-sensitive material unless both the system and dataset are explicitly approved for that purpose. A small adapter can still encode or reproduce sensitive information.

Where you may hear it

During a vendor demo to a technology governance board
In the technical review of an internal model adaptation plan
At a procurement meeting on model hosting and storage
During a regional bureau handover on model serving

Avoid confusion

Often confused with

Common misconceptions

Common misconception: LoRA means the model learns without training the model at all.
Correction: Training still happens. The difference is that optimisation updates only the added LoRA weights while the original base weights stay frozen. That is why LoRA is usually cheaper than full fine-tuning, not free or automatic.
Common misconception: LoRA and fine-tuning are different categories, so choosing one excludes the other.
Correction: LoRA is a way of doing fine-tuning more efficiently. More precisely, it is one method within parameter-efficient fine-tuning. Full fine-tuning updates many or all model weights; LoRA fine-tuning updates a small added set.
Common misconception: A LoRA adapter contains all the knowledge of the full model, so the base model is no longer needed.
Correction: Usually the adapter stores only the learned changes, not the full model. In practice you normally need the original base model plus the adapter, unless you merge the adapter into a copy of the base checkpoint.
Common misconception: Low rank means lower quality by definition.
Correction: Low rank means the update is represented in a compressed form. A rank that is too small can limit adaptation, but many tasks work well because the needed changes are structured enough to be captured with relatively few parameters.

Quick check

Which statement best describes LoRA?

Go deeper

The simple version

LoRA lets you customise a large model by adding and training a small attachment instead of editing the whole model.

The technical version

LoRA is a parameter-efficient fine-tuning method that freezes a pre-trained weight matrix W and learns a low-rank update, often written W + BA. A and B are trainable factor matrices with a small shared dimension r, so BA has rank at most r. These updates are inserted into selected layers, commonly attention projections, and only A and B receive gradient updates during optimisation. This reduces trainable parameters, optimiser state and often storage for task-specific adaptations. At inference, the adapter can remain separate or be merged into the base weights. Exact memory and speed benefits depend on implementation, target modules, precision, batch size and whether the adapter is merged.

Reviewed external resources

EncyclopaediaLoRa (opens in a new tab)Wikipedia
This link gives a general overview of LoRA and suits readers who want a broad starting point before going into implementation details.
Intermediatelink checked 2026-07-31
CourseLLM Course (opens in a new tab)Hugging Face
This course material suits readers who want practical explanations of how LoRA fits into modern large-model training and adaptation workflows.
Deeperlink checked 2026-07-31
CourseShort Courses (opens in a new tab)DeepLearning.AI
This course suits readers who prefer guided, hands-on lessons that show how LoRA is used in applied machine learning practice.
Intermediatelink checked 2026-07-31
VideoVideo explanations: LoRA (opens in a new tab)YouTube search
This search results page helps readers find different video explanations of LoRA and suits those who learn best by comparing multiple presenters.
search results, not one pagelink checked 2026-07-31
DiscussionCommunity discussion: LoRA (opens in a new tab)Reddit search
This search results page helps readers see community questions and experiences about LoRA and suits readers who want informal discussion alongside more structured sources.
search results, not one pagelink checked 2026-07-31

Where this came from

Drafted with a language model through an offline pipeline, checked against this site's schema, and every external link above fetched to see whether it answers. Treat it as a starting point and check anything that matters against the primary sources. How this site is built and what that means.

The one thing to remember

LoRA freezes the base model and learns small, low-rank updates in selected layers, reducing the parameters and storage needed for task-specific adaptation.