Learn AI /Training and Model Adaptation /Preference learning
Preference learning
- Deeper
- Training and Model Adaptation
- Technical readers
Also called: DPO, preference data.
Back to the milestone you came from
In five seconds
What it is
A method for adjusting a model so it more often produces outputs people rank as better, safer or more useful than alternatives.
In plain words
Instead of showing the model one perfect answer each time, you show it several answers and tell it which one people preferred. The model is then adjusted to produce preferred kinds of answers more often.
An everyday example
Someone prepares two possible lunches and asks which you would rather take. You choose the one with fruit instead of sweets. The next day, you choose between two more, preferring the option with less packaging. After many such choices, the person becomes more likely to pack a lunch matching the patterns in your selections. You never supplied one perfect recipe or a numerical score. The learning came from repeated comparisons, and it may still miss something you did not consider, such as an allergy.
A recommendation list is usually a simple ranking problem with clear items to compare. Language model outputs are open-ended text, so preferences can be noisier, more subjective and easier to game. Also, a reviewer choosing one answer over another does not mean the chosen answer is factually correct; it only means it better matched the selected criteria in that comparison.
Remember thisPreference learning makes outputs matching reviewers' choices more likely. It transfers their criteria and blind spots, not an independent guarantee of truth or safety.
This term is contested
The umbrella term is used inconsistently. Some people use 'preference learning' broadly for any training from comparative judgements, while others use it more narrowly for specific objectives such as DPO or distinguish it from reward-model-based RLHF.
Understand it
Understand it
Why it matters
Preference learning affects how a model behaves in practice, so it matters for procurement, risk review and quality control. A system may look much better after preference tuning because it is more polite, concise or policy-aligned, yet those gains do not automatically mean it is more accurate or less biased. If you are comparing vendors or internal model versions, you need to ask what preferences were optimised, who supplied the ratings, what data was excluded, and how trade-offs were tested. This is also important for governance: preference learning can encode organisational norms and safety constraints, but it can also reproduce reviewer bias or over-optimise for outputs that merely look good to raters.
How preference learning changes a model
After viewing, the reader should be able to explain that preference learning uses comparisons between outputs to push a model towards preferred behaviour.
01
1. Start with a base or instruction-tuned model
A model already exists and can produce several plausible responses to the same prompt.
Draw a model box on the left with a small label: 'existing checkpoint'. Show one prompt entering and two response cards coming out, labelled A and B.
02
2. Collect preferences
A human rater or another trusted process compares A and B and marks which better fits the chosen criteria.
Place a comparison box after the responses. Show a simple toggle or badge: 'A preferred' or 'B preferred'. Add a caption: 'criteria may include usefulness, safety, tone or task success'.
03
3. Convert comparisons into a training signal
The learning algorithm increases the model’s tendency to produce preferred responses and reduces the tendency for rejected ones.
Show arrows from the preference label into a training box. Inside the box, include two compact lines: 'raise score/probability of preferred output' and 'lower score/probability of rejected output'.
04
4. Save an updated checkpoint
The result is a new model version that more often matches the selected preferences on similar prompts.
Draw a final model box labelled 'updated checkpoint'. Add a small note below: 'DPO does this directly; some methods use a separate reward model first'.
A left-to-right flow with four boxes: prompt, multiple candidate responses, preference signal, and updated model checkpoint. Read from left to right to see how comparisons become parameter updates. Small notes under the boxes distinguish direct preference optimisation from approaches that first train a reward model.
Hovering or tapping each step reveals a short note with example criteria and a sentence on how DPO differs from reward-model-based approaches.
Describe this without the diagram
The diagram is a four-step flow. First, an existing model receives one prompt and produces two candidate responses, A and B. Second, a reviewer compares A and B and marks which response is preferred according to criteria such as usefulness, safety, tone or task success. Third, the comparison becomes a training signal: the method pushes the model to make the preferred response more likely and the rejected response less likely. Fourth, the updated parameters are saved as a new checkpoint. A note explains that DPO applies this update directly from preference pairs, while some related methods first train a separate reward model.
With reduced motion: A static left-to-right flowchart with four labelled boxes and arrows: existing model, candidate responses, preference comparison, and updated checkpoint. Notes below the final two boxes explain that DPO uses direct preference updates, while some alternatives insert a reward model stage.
Preference learning uses preference data, often comparisons, instead of only example answers. People, or another trusted process, look at two or more model outputs for the same prompt and indicate which one they prefer. That preference data becomes the training signal.
What goes in is usually a prompt plus candidate responses and a label such as “A is better than B”. Sometimes the preference is about helpfulness, harmlessness, style or task success. The data does not say the preferred answer is perfect; it only says it is better than the alternative under the chosen criteria.
What happens next depends on the method. In direct preference optimisation, often shortened to DPO, the model is updated to favour the preferred response over the rejected response more strongly than a reference model does. Other approaches may first train a reward model, meaning a separate model that scores outputs, and then optimise against that score. That is why preference learning is related to, but not identical to, reinforcement learning from human feedback.
When used for model adaptation, preference learning produces a new checkpoint that tends to follow the chosen preferences more often. This can improve usefulness or safety, but it can also narrow behaviour, amplify the biases in the raters or make the model sound polished without being more correct. Preference learning changes behaviour after pretraining; it does not add reliable new factual knowledge in the same way people often imagine when they say “train it on our data”.
How “Choosing between two packed lunches” maps onto the real thing
| In the story | In the AI system |
|---|---|
| Two ranked lists compared for the same search | Two candidate model outputs for the same prompt |
| Reviewer marks the better list | Preference label such as preferred versus rejected |
| Adjusting the ranking system after many comparisons | Updating model parameters to favour preferred outputs |
| Criteria like relevance and recency | Preference dimensions such as usefulness, safety or style |
A worked example
At 16:30, a reporting officer in a regional bureau tests a model that turns synthetic extracts resembling a situation report into the five bullets required for an 18:00 briefing note.
Starting from: Prompt: "Produce five bullets covering the main development, agreed action, responsible function, deadline and outstanding caveat." Candidate A follows the structure and stays within the supplied facts. Candidate B is shorter but omits the caveat and assigns the action to the wrong function.
- Using a written rubric, reviewers rank A above B because it follows the requested structure without changing the underlying facts.
- The training record stores the same prompt with A as the preferred response, y+, and B as the rejected response, y-.
- In direct preference optimisation, the update increases the model's preference margin for A over B relative to a reference model. It does not require a separate reward model or reinforcement-learning stage.
- The resulting checkpoint is tested on held-out prompts for factual retention, omissions and consistency, not merely for whether its bullets look like the preferred examples.
Result: The adapted model becomes more likely to produce the structure reviewers preferred. It may also reproduce weaknesses in their rankings, so the reporting officer still checks every briefing note against the cleared source.
In a United Nations workplace
At the review meeting closing a drafting-assistant pilot, an internal auditor supporting a UN Country Team working group examines how programme officers ranked alternative summaries for a progress report before the findings go to the Resident Coordinator's Office.
The rankings can be used as comparative supervision: the model is adjusted to favour drafts that staff consistently judged clearer, more complete and appropriately cautious. The exercise also reveals whether the ranking criteria rewarded brevity at the expense of decisions, caveats or follow-up actions.
Before you try thisDo not use actual audit evidence, confidential staff comments, personal data or sensitive material as ranking examples in a tool not approved for them. Use an approved environment and minimised or synthetic examples. A strong preference score reflects the reviewers' criteria, including any blind spots, and is not evidence that a draft is accurate.
Where you may hear it
- At an ICT governance board review of controls for a drafting-assistant pilot
- During a vendor debrief on how reviewer rankings will be used after initial training
- At a checkpoint evaluation before the next pilot phase is cleared
- In a lessons-learned session comparing reviewer rubrics and model outputs
Avoid confusion
Avoid confusion
Often confused with
Reinforcement learning from human feedback
A method for adjusting a model so its outputs better match human preferences by learning from ranked examples and reward signals.
Fine-tuning
Updating a pre-trained model's parameters with additional examples to improve performance on a narrower task, style or domain.
Instruction tuning
A post-pretraining adaptation step that teaches a general language model to respond to prompts in the requested format, style and task pattern.
Common misconceptions
Common misconception: Preference learning simply teaches the model the right answer.
Correction: Usually it teaches relative preference, not absolute truth. The signal often says one response is better than another under chosen criteria, such as clarity or safety. A preferred answer can still contain errors, omissions or bias.
Common misconception: DPO and reinforcement learning from human feedback are the same thing.
Correction: They are closely related but not identical. DPO learns directly from preference pairs, while classic RLHF often trains a separate reward model and then uses reinforcement learning to optimise against that reward. People sometimes use the terms loosely, but the training mechanics differ.
Common misconception: If we use our staff to rate outputs, the model will absorb all of our internal knowledge.
Correction: Preference data mainly shapes behaviour and style. It can improve how the model responds to prompts similar to the rated examples, but it is not a reliable way to inject a large knowledge base or guarantee up-to-date facts.
Quick check
Which statement best distinguishes DPO from a classic reward-model-based RLHF pipeline?
The key distinction is the training path. DPO uses preference pairs directly in the optimisation objective. In a classic RLHF setup, preference data often trains a reward model first, and reinforcement learning then updates the policy to score well under that reward. Neither approach guarantees factual correctness.
Go deeper
Go deeper
The simple version
Instead of showing the model one perfect answer each time, you show it several answers and tell it which one people preferred. The model is then adjusted to produce preferred kinds of answers more often.
The technical version
Preference learning is a post-training approach that uses comparative judgements over candidate outputs as supervision. Given a prompt x and responses y+ and y-, the objective encourages the model to favour y+ over y-. In DPO, this is done by increasing that preference margin relative to a reference policy, avoiding a separate reward-model-plus-RL stage. In reward-model-based pipelines, pairwise comparisons are first used to fit a scalar reward model, and a policy is then optimised to maximise expected reward. In both cases, the training signal is ordinal or comparative rather than a single ground-truth target, which makes label quality, rater consistency, preference specification and distribution shift central concerns.
Reviewed external resources
EncyclopaediaPreference learning (opens in a new tab)Wikipedia
This link gives a broad overview of the term and basic definitions, which suits readers who want a quick orientation before going deeper.
Intermediatelink checked 2026-07-31
CourseLLM Course (opens in a new tab)Hugging Face
This link gives course-style explanations of post-training methods in the wider LLM pipeline, which suits readers who want structured technical context.
Deeperlink checked 2026-07-31
CourseShort Courses (opens in a new tab)DeepLearning.AI
This link gives short lesson-based introductions to practical modern AI methods, which suits readers who prefer guided video teaching over a long textbook.
Intermediatelink checked 2026-07-31
VideoVideo explanations: Preference learning (opens in a new tab)YouTube search
This search page gives multiple video explanations from different creators, which suits readers who want to compare teaching styles and should remember it is a search results page, not one vetted lesson.
search results, not one pagelink checked 2026-07-31
DiscussionCommunity discussion: Preference learning (opens in a new tab)Reddit search
This search page gives community discussions and questions from many users, which suits readers who want informal explanations and troubleshooting ideas and should remember it is a search results page, not one authoritative source.
search results, not one pagelink checked 2026-07-31
Where this came from
Drafted with a language model through an offline pipeline, checked against this site's schema, and every external link above fetched to see whether it answers. Treat it as a starting point and check anything that matters against the primary sources. How this site is built and what that means.
The one thing to remember
Preference learning makes outputs matching reviewers' choices more likely. It transfers their criteria and blind spots, not an independent guarantee of truth or safety.