RLHF (Reinforcement Learning from Human Feedback)
Training method that uses human preference rankings to align model outputs with human values.
Last updated: October 17, 2026
What Is RLHF (Reinforcement Learning from Human Feedback)?
RLHF is the alignment technique behind instruction following chat models. The standard recipe, described in the InstructGPT paper (Ouyang et al., 2022), has three steps. First, supervised fine-tuning on demonstrations written by labelers. Second, training a reward model on human rankings of model outputs. Third, optimizing the language model against that reward model with the PPO reinforcement learning algorithm. In human evaluations in that paper, outputs from a 1.3B parameter InstructGPT model were preferred over outputs from the 175B GPT-3.
The idea predates chat models: Christiano et al. (2017) trained agents from human preferences between pairs of behaviors. Newer methods simplify the pipeline. Direct Preference Optimization (Rafailov et al., 2023) uses a classification loss and no sampling from the model during training, and Constitutional AI (Bai et al., 2022) takes preference labels from an AI model guided by written principles, called RLAIF. RLHF has side effects too. Sharma et al. found sycophancy in five assistants and linked it in part to human preference judgments, and Kirk et al. found that RLHF significantly reduces output diversity compared with supervised fine-tuning. Test for these tendencies.
Code Example
import math
def preference_loss(reward_chosen: float, reward_rejected: float) -> float:
# Pairwise loss: small when the chosen reply scores higher than the rejected one
return -math.log(1 / (1 + math.exp(-(reward_chosen - reward_rejected))))
print(round(preference_loss(2.0, 0.5), 3)) # 0.201
print(round(preference_loss(0.5, 2.0), 3)) # 1.701The pairwise idea behind reward model training, in plain Python. The model is pushed to score the preferred reply above the rejected one. Real training computes this over batches of model scores.
When Should You Use RLHF (Reinforcement Learning from Human Feedback)?
You rarely run RLHF yourself, because it happens inside the lab that trains the model. It still matters: it explains behavior such as polite refusals and a pull toward agreeing with the user, so you can write evals for it.
Common Questions
What is the difference between RLHF and fine-tuning?
RLHF is a kind of fine-tuning. Supervised fine-tuning learns from example outputs. RLHF learns from comparisons: a reward model is trained on which of two outputs people prefer, and the language model is then optimized to score well under it.
Sources
Related Terms
Building with RLHF (Reinforcement Learning from Human Feedback)?
I've shipped this pattern in real production systems. If you want a second pair of eyes on your architecture, that's what I do.