presentsUNAIVERSE
The UN AI timeline

    Learn AI /Training and Model Adaptation /Parameter-efficient fine-tuning

    Parameter-efficient fine-tuningPEFT

    Also called: PEFT.

    Detail

    In five seconds

    What it is

    A way to adapt a large model to a new task by training only a small set of added or selected weights instead of updating the whole model.

    In plain words

    It is often a cheaper way to customise a big model. Instead of updating the whole model, you train only a small add-on or part of it.

    An everyday example

    A media club uses one camera for several projects. Instead of rebuilding the camera for each assignment, it adjusts a small clip-on lens that changes how the camera handles a particular kind of shot. The camera body stays unchanged, and each project keeps only its own compact attachment. Similarly, parameter-efficient fine-tuning keeps the large base model fixed while training and storing a much smaller set of task-specific weights.

    Where the story stops working

    A keyboard overlay changes fixed shortcuts in a simple, visible way. PEFT changes numerical weights inside a model through optimisation on data, so its effects are distributed and less predictable. Also, some PEFT methods do not literally sit on top of the model as a separate visible layer; that is just a helpful picture.

    Remember thisParameter-efficient fine-tuning freezes most or all of the base model and updates only a small designated set of weights, reducing training memory, optimiser state and storage needs.

    This term is contested

    The broad idea is stable, but the boundary of what counts as PEFT is not perfectly fixed. Some authors use the term narrowly for adapter-style methods, while others include prompt tuning, selective unfreezing or other low-parameter adaptation strategies.

    Understand it

    Understand it

    Why it matters

    For a non-specialist, PEFT matters because it changes the cost and governance of model adaptation. It can reduce hardware needs, shorten training time and lower storage demands. That makes it easier to test a specialised model for a narrow task without committing to a full retraining project. It also supports cleaner version management: one base model can serve many use cases through separate small adapters. But PEFT is not automatically safer or better. You still need good data, evaluation, access control, documentation and human oversight. In procurement or project planning, the key question is often not only 'can we fine-tune?' but 'do we need full fine-tuning, or will PEFT achieve the goal at lower cost and lower operational burden?'

    Full fine-tuning versus PEFT

    Show why PEFT adapts a model by changing a small set of parameters while leaving most base weights unchanged.

    1. 01

      Same starting point

      Both approaches begin with one pretrained base model and a task dataset.

      Draw one large rectangle labelled 'Base model checkpoint' above both columns, with arrows down into the left and right paths.

    2. 02

      Full fine-tuning path

      In full fine-tuning, training updates many or all model weights.

      Left column: show a large model block filled with many small highlighted dots across the whole block; label 'many weights updated'.

    3. 03

      PEFT path

      In PEFT, the base model stays frozen and only a small added or selected set of parameters is trained.

      Right column: show a large pale model block labelled 'frozen base weights' plus a small bright side block labelled 'trainable adapter' or 'small parameter set'.

    4. 04

      What gets saved

      Full fine-tuning often saves a full specialised model, while PEFT often saves only the compact adapter package.

      Under left column, draw a large storage icon labelled 'full specialised checkpoint'. Under right column, draw a small storage icon labelled 'adapter files only'.

    5. 05

      Using the result

      At inference time, the PEFT adapter is attached to the same base model to produce task-specific behaviour.

      Show on the right a final combined box made from the base model plus the small adapter, with an arrow to 'task output'.

    A side-by-side diagram compares two adaptation paths starting from the same base model. Read left to right. The left path shows many weights updated and a full new model copy saved. The right path shows the base model frozen, a small trainable module updated, and a compact adapter saved for later attachment to the same base model.

    Hover or tap each column to highlight which weights change and to display approximate relative storage sizes, such as 'large' versus 'small', without claiming a fixed ratio.

    Describe this without the diagram

    The diagram compares two ways to adapt the same pretrained model. Both start from one base model and a task dataset. In the full fine-tuning column, training changes many or all weights in the whole model, and the result is typically saved as a full specialised checkpoint. In the PEFT column, the base model remains frozen, shown in a muted colour, while only a small adapter or selected parameter set is trainable, shown in a bright colour. The saved result is a much smaller adapter file. To use it, the adapter is attached back to the base model, producing task-specific output without storing a full separate model copy.

    With reduced motion: A static two-column comparison remains visible at all times, with coloured labels marking updated weights in the full fine-tuning path and the small trainable adapter in the PEFT path.

    Large language models contain many weights, which are the numeric settings that shape their outputs. Traditional fine-tuning changes many or all of those weights for a new task. Parameter-efficient fine-tuning changes only a small fraction of them, often by adding small trainable components while keeping the original model mostly frozen, meaning not updated.

    What goes in is a pre-existing model, a target task, and training examples. During training, the model processes the examples as usual, but the learning step updates only the small PEFT components or a selected subset of weights. The main model weights stay fixed or nearly fixed.

    What comes out is an adapted model behaviour without storing or retraining a full new copy of the entire model. In practice, teams often keep the original checkpoint and save only the small PEFT package. That package can later be attached to the base model to produce the specialised version.

    This matters because full fine-tuning can be expensive in memory, storage, time and energy. PEFT often makes adaptation practical on smaller budgets and lets one organisation maintain several task-specific variants of the same base model. It is commonly used for domain adaptation, style control, instruction following, or task-specific performance improvements.

    PEFT is a family of methods, not one single method. LoRA is one well-known example, but there are others. It is also different from in-context learning, where you steer behaviour by prompt examples without updating any weights at all.

    How “Adjusting a clip-on camera lens” maps onto the real thing

    In the storyIn the AI system
    standard keyboard hardwarebase model with its original weights
    thin removable overlaysmall trainable PEFT components
    swapping overlays for different desks or projectsloading different task-specific adapters onto the same base model
    storing overlays instead of whole keyboardssaving far fewer trained parameters and less storage

    A worked example

    At a regional hub, an information management officer prepares a pilot to classify cleared monitoring report passages against the labels in a results framework before the next reporting round.

    Starting from: An approved pretrained model, a cleaned set of labelled passages containing no personal or confidential material, and lightweight low-rank update matrices designated for training.

    1. The officer prepares training and validation examples using the same labels and classification instructions that will apply during reporting.
    2. The pretrained backbone is frozen. During backpropagation, gradients pass through the model, but the optimiser updates only the low-rank matrices.
    3. The adapted model is tested on held-back passages, including ambiguous cases that must be referred for human judgement rather than assigned automatically.
    4. The low-rank weights are saved separately and loaded alongside the unchanged base model when the classification pilot runs.

    Result: If validation is satisfactory, the pilot gains a task-specific adaptation while requiring substantially less training memory and storage than updating and saving another full model. The classifications remain proposed labels for review, not amendments to the results framework.

    In a United Nations workplace

    Before close of business, a political affairs officer in a Resident Coordinator's Office adds to a handover note the design of a pilot for drafting talking points from cleared, public source material. The pilot keeps the approved base model fixed and trains a small adapter on previously cleared examples.

    The adapter can learn recurring structure and drafting conventions without creating and maintaining a separately fine-tuned copy of the full model. The handover note should identify the unchanged base model, the adapter version and the validation results, since all three matter when the pilot is resumed.

    Before you try thisConfidential political reporting, source identities, personal data and security-sensitive material must not be used for training or entered into an environment not approved for them. Adaptation does not make outputs factual, appropriately caveated or cleared. Every draft still requires substantive review.

    Where you may hear it

    • A vendor demo to an ICT governance board, when deployment options reach adapters and low-rank updates
    • A procurement unit's technical evaluation of model customisation and hosting requirements
    • A regional bureau planning how several task-specific adaptations will share one approved base model
    • A model review before adapter files and validation records are uploaded to the portal

    Avoid confusion

    Avoid confusion

    Often confused with

    Common misconceptions

    • PEFT means no training happens.

      Training still happens. The difference is that only a small subset of parameters is updated, rather than the entire model. You still need data, optimisation, validation and testing.

    • PEFT and LoRA are the same thing.

      LoRA is one popular PEFT method, not the whole category. PEFT includes several approaches that reduce how many parameters are trained or stored.

    • Because fewer parameters change, PEFT always matches full fine-tuning.

      Sometimes it comes close, but not always. Performance depends on the task, data quality, base model and PEFT method. For some tasks, updating the whole model may still work better.

    • PEFT makes a model smaller in all respects.

      Usually it reduces the number of trainable and stored task-specific parameters, not necessarily the size of the base model used at inference time. You often still need the original base model loaded.

    Quick check

    Which statement best captures parameter-efficient fine-tuning?

    Go deeper

    Go deeper

    The simple version

    It is often a cheaper way to customise a big model. Instead of updating the whole model, you train only a small add-on or part of it.

    The technical version

    PEFT refers to adaptation methods that restrict training to a small number of parameters relative to the full model. A common pattern is to freeze the pretrained backbone and optimise lightweight task-specific modules, such as low-rank update matrices, adapters inserted into layers, soft prompts, or other selective parameter subsets. During backpropagation, gradients may be computed through the network, but only the designated trainable parameters are updated. This cuts optimiser state, memory and checkpoint size for task adaptation. The resulting artefact is often an adapter that must be combined with the original checkpoint at inference time. PEFT is therefore about adaptation efficiency, not a single architecture, and it is distinct from prompt-only steering because weights do change.

    Reviewed external resources

    Where this came from

    Drafted with a language model through an offline pipeline, checked against this site's schema, and every external link above fetched to see whether it answers. Treat it as a starting point and check anything that matters against the primary sources. How this site is built and what that means.

    The one thing to remember

    Parameter-efficient fine-tuning freezes most or all of the base model and updates only a small designated set of weights, reducing training memory, optimiser state and storage needs.