presentsUNAIVERSE
The UN AI timeline

    Learn AI /Training and Model Adaptation /Reinforcement learning from human feedback

    Reinforcement learning from human feedbackRLHF

    Also called: RLHF.

    Detail

    In five seconds

    What it is

    A method for adjusting a model so its outputs better match human preferences by learning from ranked examples and reward signals.

    In plain words

    People look at several answers from a model, say which ones they prefer, and those preferences are used to train the model to answer in more preferred ways.

    An everyday example

    A playlist generator makes several mixes for the same request. Listeners compare pairs and choose the mix they prefer. A separate scoring system learns to predict those choices. The generator is then adjusted to create mixes that receive higher predicted scores, without changing so abruptly that it loses what it already did well. The result may suit those listeners better, but it does not reveal an objectively correct playlist, and different ranking rules could produce different preferences.

    Where the story stops working

    A search tool returns documents that already exist, while a language model generates new text token by token. Also, in RLHF the reward signal is only an approximation of human preference, not a direct measure of truth, legality, or policy compliance. The analogy also hides technical issues such as instability during optimisation and the risk of rewarding style over substance.

    Remember thisRLHF trains towards patterns that reviewers rewarded. It can improve usefulness and tone, but preference is not proof of truth, safety or institutional clearance.

    This term is contested

    The term is widely used, but people do not always mean exactly the same pipeline by it. Some use RLHF narrowly for the reinforcement-learning stage after a reward model is trained, while others use it more broadly for the whole process of collecting human preferences, training the reward model and optimising the model.

    Understand it

    Understand it

    Why it matters

    For non-specialists, RLHF matters because it helps explain why a model can feel more helpful after development without becoming fully reliable. If you procure, govern, evaluate or use AI systems, this term tells you what kind of improvement was made: the model was shaped to match human preferences under a chosen feedback process. That affects quality, safety, tone, bias, labour needs, auditability and cost. It also helps you ask better questions, such as whose preferences were collected, what instructions reviewers used, whether truthfulness was measured separately, and how the updated model was tested before deployment.

    How RLHF changes a model after initial training

    Show the ordered stages of RLHF and explain that human preferences are converted into a reward signal, then used to update the model.

    1. 01

      1. Generate candidate answers

      A partly trained model receives prompts and produces several possible answers for each one.

      Draw a prompt card feeding into a model box, with three short answer cards coming out side by side.

    2. 02

      2. Collect human preferences

      People compare the answers and mark which one is better for the task, based on guidance.

      Show the three answer cards under a bracket, with a human reviewer icon selecting or ranking them.

    3. 03

      3. Train a reward model

      A separate model learns to predict those human preferences and outputs a score for new answers.

      Convert the ranked answers into a smaller scoring box labelled reward model, with output scores such as low, medium, high.

    4. 04

      4. Optimise the main model

      The main model is updated so it tends to produce answers that the reward model scores more highly, within training constraints.

      Show the reward scores feeding back into the main model, ending in a new box labelled updated model version.

    A left-to-right step diagram with four boxes connected by arrows. Read from raw prompt handling to human comparisons, then reward modelling, then model optimisation, ending with a revised model version. Each box names the input and output of that stage.

    Hover or tap each step to reveal one sentence on what goes in and what comes out; arrows highlight in sequence from step 1 to step 4.

    Describe this without the diagram

    The diagram presents RLHF as a four-step process. First, a model takes a prompt and generates several candidate answers. Second, human reviewers compare those answers and indicate which is better. Third, those comparisons are used to train a reward model, which is a scoring system that predicts which answers humans would prefer. Fourth, the main model is updated using reinforcement learning so it is more likely to produce answers that receive higher reward scores. The final output is an updated model version, not a guarantee of truth or safety.

    With reduced motion: A static four-box diagram is shown with numbered arrows from left to right: candidate answers, human comparisons, reward model, updated model version.

    After a model has already learned general patterns from large amounts of text, developers often want it to behave in more useful ways. They may want it to follow instructions better, avoid unsafe wording, or give answers people prefer. Reinforcement learning from human feedback is one way to do that.

    The input is usually a set of model answers to the same prompt, plus human judgements about which answer is better. Those judgements can come from rankings, ratings, or comparisons. A separate reward model is then trained to predict those human preferences. A reward model is a system that scores outputs based on patterns in the feedback.

    Next, the main model is updated so it tends to produce outputs that get higher reward scores. This part uses reinforcement learning, a training method where a system is adjusted using signals about better and worse outcomes rather than only exact correct answers. In practice, many descriptions of RLHF bundle together several stages: collecting comparisons, training the reward model, and then optimising the main model.

    What comes out is a new model version that often feels more helpful, more polite, or more on-task to users. But it does not become inherently truthful or reliable. It has learned to match the kinds of responses the feedback process rewarded, and that can reflect both good judgement and the limits or biases of the people, prompts, and scoring rules used.

    How “Ranking mixes from a playlist generator” maps onto the real thing

    In the storyIn the AI system
    staff ranking several result listshumans comparing multiple model outputs
    scoring rule learned from past rankingsreward model trained on human feedback
    tuning the search tool to get higher scoresoptimising the model with reinforcement learning
    more useful ranked resultsoutputs that better match preferred behaviour

    A worked example

    A public information officer at Headquarters is testing a model that drafts plain-language website answers before a scheduled content release.

    Starting from: Prompt: “How does a General Assembly committee consider draft text?” The model produces several candidate answers from the same prompt.

    1. Reviewers rank pairs of answers using written criteria covering clarity, relevance, factual restraint and appropriate tone.
    2. The rankings are used to train a reward model that estimates which answer reviewers would prefer for a given prompt.
    3. The drafting model is optimised to increase the reward model’s score, while a constraint limits how far it moves from the earlier model.
    4. On held-out prompts, reviewers compare the updated model with the earlier version and record both preferences and factual errors.

    Result: The updated model may produce answers closer to the reviewers’ stated preferences. It can still be confidently wrong, omit an important qualification or reproduce weaknesses in the ranking guidelines, so publication clearance remains unchanged.

    In a United Nations workplace

    A partnerships officer in a regional bureau is completing the technical evaluation matrix for three bids before close of business. One bidder says its drafting assistant was improved through reinforcement learning from human feedback.

    The officer checks whether this means people ranked candidate outputs under written criteria, those rankings trained a reward model, and the main model was then optimised against that signal. A feedback button or supervised fine-tuning alone would not substantiate the claim.

    Before you try thisBid contents, evaluators’ comments, pricing and other procurement-sensitive material must not be entered into any tool not approved for that information. Preference-based training also does not establish factual accuracy, security or fitness for purpose, so the claim still requires technical evidence and independent testing.

    Where you may hear it

    • A technical evaluation meeting for an AI procurement
    • A vendor demonstration to an information technology governance board
    • A model-card review before clearance of a pilot
    • A safety briefing on how an assistant was adapted after initial training

    Avoid confusion

    Avoid confusion

    Often confused with

    Common misconceptions

    • RLHF means the model learned directly from users chatting with it.

      Not necessarily. The feedback is often collected in structured annotation tasks where reviewers compare prepared outputs. Some systems may later use user signals, but RLHF usually refers to a specific training pipeline rather than any informal feedback from use.

    • If a model used RLHF, its answers are therefore truthful and safe.

      RLHF trains a model to match rewarded preferences, not to guarantee facts or compliance. If reviewers reward answers that sound confident or polite, the model may learn those traits more strongly than accuracy unless truthfulness and safety are separately measured and reinforced.

    • RLHF is the same as fine-tuning on correct answers.

      Supervised fine-tuning learns from target examples, such as prompt-answer pairs. RLHF usually adds a later stage where humans compare outputs, a reward model predicts those preferences, and reinforcement learning updates the model using that score signal.

    • RLHF is the same as alignment.

      Alignment is a broader and contested term about making AI systems behave in intended or acceptable ways. RLHF is one concrete method within that wider area. A model can use RLHF and still have important alignment problems.

    Quick check

    Which description best captures what RLHF adds after a model already produces text?

    Go deeper

    Go deeper

    The simple version

    People look at several answers from a model, say which ones they prefer, and those preferences are used to train the model to answer in more preferred ways.

    The technical version

    RLHF typically follows earlier training stages. A base or instruction-tuned model is sampled on prompts to produce candidate completions. Human annotators then provide preference data, often pairwise rankings between completions under written guidelines. That dataset is used to train a reward model that estimates the relative desirability of outputs for a prompt. The main policy model is then optimised with a reinforcement-learning objective to increase expected reward while usually constraining it not to drift too far from the starting model. In large language model practice, this optimisation has often used methods such as PPO, though newer variants and related preference-optimisation methods also exist. The result is a new checkpoint that better matches the measured preference signal, but the reward model can be imperfect, vulnerable to shortcut learning, and sensitive to annotator bias, prompt distribution and reward hacking.

    Reviewed external resources

    Where this came from

    Drafted with a language model through an offline pipeline, checked against this site's schema, and every external link above fetched to see whether it answers. Treat it as a starting point and check anything that matters against the primary sources. How this site is built and what that means.

    The one thing to remember

    RLHF trains towards patterns that reviewers rewarded. It can improve usefulness and tone, but preference is not proof of truth, safety or institutional clearance.