What Is RLHF? How Human Feedback Data Works And What It Costs

Meta Description: What is RLHF? Learn how reinforcement learning from human feedback works, what the training data looks like, who produces it and what it costs to buy.

RLHF, or reinforcement learning from human feedback, is a training method that teaches an AI model to produce the kind of answers people prefer by learning from human judgments instead of a fixed scoring rule. In practice, people compare or rate a model’s outputs, a second model learns to predict those judgments, and the original model is then tuned to score well against that learned predictor. The technique is the main reason modern chat assistants follow instructions, decline harmful requests and write in a usable tone rather than simply continuing text the way a raw pretrained model does.

RLHF At A Glance

Question	Short Answer
What is RLHF?	A post-training method that tunes a model toward outputs humans prefer, using human comparisons or ratings as the training signal
Main stages	Supervised fine-tuning, preference data collection, reward model training, policy optimization
Typical optimizer	PPO in the original recipe; DPO and GRPO are common alternatives
Core data type	Pairwise comparisons or rankings of model responses to the same prompt
Other data types	Demonstrations, ratings, rewrites, rubrics, red teaming attacks
Who produces the data	In-house teams, expert networks, crowd platforms and managed annotation vendors
Main quality risks	Low annotator agreement, reward hacking, sycophancy, missing domain expertise
How it is priced	Usually per hour of contributor time or per task, plus platform or management fees
Best known early example	OpenAI’s InstructGPT (2022)

What Is RLHF In Plain Language?

A model trained only on internet text learns to predict the next word, which makes it fluent but not necessarily helpful. Writing a mathematical rule for “a good answer” is close to impossible, because good depends on accuracy, tone, completeness, safety and what the user meant.

RLHF solves this by using people as the scoring rule. Instead of defining quality in code, you show human reviewers two or more answers to the same prompt and ask which is better. Collect enough of those judgments and you can train a model that predicts them, then use that predictor to steer the main model.

The idea predates chatbots. In Deep Reinforcement Learning From Human Preferences (2017), Paul Christiano and colleagues at OpenAI and DeepMind showed that agents could learn Atari games and simulated robot movement from non-expert comparisons of short video clips, with human feedback on less than 1% of the agent’s interactions. OpenAI later applied the approach to text in Learning To Summarize From Human Feedback (2020), where models trained on human comparisons of summaries outperformed human-written reference summaries on a Reddit dataset.

How RLHF Works Step By Step

The reference recipe most teams still describe is the one OpenAI published in Training Language Models To Follow Instructions With Human Feedback (March 2022), the InstructGPT paper. It has three stages after pretraining.

Step 1: Supervised Fine-Tuning (SFT)

Human writers produce demonstrations: a prompt plus the ideal response. The pretrained model is fine-tuned on these examples so it learns the basic shape of a helpful answer. In InstructGPT, the SFT dataset contained about 13,000 training prompts. SFT alone gets a model most of the way to following instructions, but demonstrations are slow to write and only show one good answer per prompt.

Step 2: Collecting Preference Or Comparison Data

The SFT model generates several responses to each prompt, and human labelers rank them. InstructGPT labelers ranked between four and nine responses per prompt, and every pair inside a ranking becomes a training comparison. Ranking is faster than writing and captures finer distinctions, such as which of two correct answers is clearer or safer. The InstructGPT reward model dataset used about 33,000 prompts.

Step 3: Training A Reward Model

A separate model, usually initialized from the same base model, is trained to take a prompt and a response and output a single score. It learns by being rewarded when it gives the human-preferred response a higher score than the rejected one. Once trained, it acts as a fast, automated stand-in for the human labelers. Some labs train more than one: Meta’s Llama 2 paper (2023) describes separate reward models for helpfulness and for safety, trained on 1,418,091 binary comparisons Meta collected itself.

Step 4: Policy Optimization With PPO

Finally, the SFT model (now called the policy) generates responses, the reward model scores them, and a reinforcement learning algorithm updates the policy to earn higher scores. InstructGPT used Proximal Policy Optimization (PPO), with a penalty that keeps the policy from drifting too far from the SFT model. Without that penalty, the policy tends to find odd outputs the reward model overrates. The result was striking: in human evaluations, outputs from the 1.3 billion parameter InstructGPT model were preferred over those of the 175 billion parameter GPT-3, despite being about 100 times smaller.

Anthropic described a closely related pipeline in Training A Helpful And Harmless Assistant With RLHF (April 2022), adding an online loop in which preference models and policies were retrained weekly on fresh human feedback.

Newer Variants And Alternatives To RLHF

Classic PPO-based RLHF is expensive to run and sensitive to tuning. Since 2022, several methods have changed either how the model is optimized or where the feedback comes from.

Method	Source	What Changes	Human Data Still Needed
DPO	Rafailov et al., Stanford, 2023	Removes the separate reward model and RL loop	Yes, pairwise preferences
Constitutional AI / RLAIF	Anthropic, 2022	An AI model judges responses against written principles	Principles plus some human data
RLVR	Ai2, Tulu 3, 2024	Reward comes from an automatic correctness check	Verifiable problems and answers
GRPO with rule-based rewards	DeepSeek, R1, 2025	Drops PPO’s value model; scores groups of outputs	Little for reasoning tasks
Rubrics as Rewards	Scale AI researchers, 2025	Structured checklists score open-ended answers	Expert reference answers or rubrics

Direct Preference Optimization (DPO)

Direct Preference Optimization: Your Language Model Is Secretly A Reward Model, by Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher Manning and Chelsea Finn (May 2023), showed that the standard RLHF objective can be solved with a simple classification loss directly on preference pairs. There is no separate reward model and no reinforcement learning loop. The authors reported that DPO matched or beat PPO-based RLHF on sentiment control, summarization and single-turn dialogue, while being more stable and lighter on compute. DPO still needs the same kind of human preference data, which is why it changes engineering cost more than data cost.

RLAIF And Constitutional AI

Anthropic’s Constitutional AI: Harmlessness From AI Feedback (December 2022) replaced most human harmlessness labels with AI judgments. Humans write a short list of principles, the “constitution.” The model critiques and revises its own responses against those principles, then an AI model compares pairs of responses to generate preference labels for a preference model. Anthropic called this reinforcement learning from AI feedback (RLAIF). A later Google study, RLAIF Vs. RLHF (2023), reported that AI-generated preference labels achieved performance comparable to human labels on summarization and dialogue tasks. RLAIF reduces labeling volume, but humans still define the principles and check that the AI judge agrees with expert judgment.

Reinforcement Learning With Verifiable Rewards (RLVR)

For tasks with checkable answers, a learned reward model is unnecessary. The Allen Institute for AI (Ai2) introduced the term in Tulu 3: Pushing Frontiers In Open Language Model Post-Training (November 2024), describing RLVR as RL without a reward model, used to boost specific skills. DeepSeek’s DeepSeek-R1 paper (January 2025, later published in Nature) trained reasoning with rule-based accuracy rewards (answer matching for math, compiler test cases for code) and format rewards, using an algorithm called Group Relative Policy Optimization (GRPO). The authors said they avoided neural reward models for reasoning because those models are susceptible to reward hacking at scale. For data buyers, RLVR shifts demand from comparisons toward hard, well-specified problems with verified solutions, often written by domain experts.

Rubric-Based Rewards

Most valuable work, such as medical advice, legal analysis or research summaries, cannot be checked by a compiler. In Rubrics As Rewards: Reinforcement Learning Beyond Verifiable Domains (July 2025), researchers at Scale AI used structured, checklist-style rubrics as the reward signal. Each criterion states a fact, reasoning step or pitfall, weighted by importance. The best variant improved over Likert-scale LLM-as-judge rewards by up to 31% relative on HealthBench and 7% on GPQA-Diamond. Scale’s write-up notes that rubrics guided by expert reference answers produced better models than rubrics generated by AI alone, which keeps experts in the loop.

What Human Feedback Data Looks Like

“RLHF data” covers several distinct deliverables. Most projects mix them.

Data Type	What The Contributor Does	Typical Use
Demonstrations	Writes an ideal response to a prompt	SFT, the first stage of RLHF
Pairwise comparisons	Picks the better of two responses, often with a strength rating	Reward models, DPO
Rankings	Orders three or more responses	Reward models (each pair becomes a comparison)
Ratings	Scores a single response on scales such as accuracy, helpfulness, safety	Evaluation, filtering, reward shaping
Rewrites	Edits a model response into a better version	SFT data, chosen and rejected pairs
Rubrics and verified problems	Writes grading criteria or problems with checkable answers	RLVR, rubric-based rewards, benchmarks
Red teaming	Tries to make the model produce harmful or policy-breaking output	Safety training and evaluation

Pairwise Comparisons And Rankings

This is the core RLHF format. A contributor sees one prompt and two or more responses, picks a preference and often adds a short justification. Guidelines define what “better” means: factual accuracy first, then instruction following, then style, for example. Comparisons are relatively fast, which is why they dominate preference datasets. Ratings are noisier because people anchor scales differently, while rewrites and demonstrations are slower and increasingly require subject matter experts.

Red Teaming

Red teamers deliberately attack the model. Anthropic’s Red Teaming Language Models To Reduce Harms (2022) released 38,961 red team attacks and found that RLHF-trained models became harder to red team as they grew larger. Red teaming work ranges from general adversarial prompting to specialist probing in areas such as cybersecurity or chemical safety. It overlaps with trust and safety work, and teams that already run content moderation services often supply it.

Who Produces RLHF Data

In-House Teams

Frontier labs keep some feedback work internal, especially policy-sensitive labeling and data used to calibrate outside vendors. They give the tightest control but are hard to scale with release cycles.

Expert Networks

As models improved, generalist feedback stopped being enough. Grading a model’s answer on tax law or oncology requires someone who knows more than the model. Expert networks recruit credentialed professionals for hourly contract work. Mercor says on its homepage that it works with more than 5 million experts and lists an average contracted rate of $93 per hour. Surge AI focuses on high-skill human data for frontier labs; Reuters reported in July 2025 that it passed $1 billion in revenue the previous year and counted Google, OpenAI and Anthropic as customers.

Crowd Platforms And Managed Vendors

Crowd and research platforms recruit large, diverse contributor pools, useful for broad preference collection, multilingual work and evaluation. Prolific is a research participant platform that also sells AI data collection. Toloka offers preference data, demonstrations, agent trajectories and red teaming, and says on its site that more than 70% of its contributors hold advanced degrees. Scale AI is the best known managed vendor; it supplied some of the roughly 40 InstructGPT contractors, alongside Upwork. In June 2025, Meta invested in Scale AI in a deal Scale said valued it at $29 billion; Reuters reported the investment as $14.3 billion for a 49% stake, and founder Alexandr Wang left to join Meta. Reuters later reported that Google and OpenAI planned to move work away from Scale after the deal, a reminder that buyers weigh vendor ownership and neutrality alongside quality.

For a vendor-by-vendor comparison, see our list of the top RLHF and LLM training data companies. For image, video and general labeling, see the broader list of top data annotation and labeling companies.

Quality Issues In RLHF Data

Annotator Agreement

Human preferences are noisy. In the InstructGPT study, training labelers agreed with each other about 73% of the time, and held-out labelers about 77%. That means roughly one in four comparisons could flip depending on who labeled it. Agreement drops further on subjective tasks such as tone or creativity. Good vendors measure inter-annotator agreement, use overlapping assignments on a sample of tasks and run calibration sessions so that disagreement reflects genuine ambiguity rather than unclear guidelines.

Reward Hacking And Overoptimization

A reward model is an imperfect proxy for human judgment, and optimizing too hard against it can make the real output worse. OpenAI researchers Leo Gao, John Schulman and Jacob Hilton measured this in Scaling Laws For Reward Model Overoptimization (2022), describing it as an instance of Goodhart’s law. Common symptoms include longer answers, excessive hedging or confident formatting that the reward model likes but users do not.

Sycophancy

Preference data can reward flattery. Anthropic’s Towards Understanding Sycophancy In Language Models (2023) found that responses matching a user’s stated views were more likely to be preferred, and that both humans and preference models sometimes chose convincing sycophantic answers over correct ones. In April 2025, OpenAI rolled back a GPT-4o update that had become overly flattering, saying it had leaned too heavily on short-term thumbs-up and thumbs-down feedback.

Missing Expertise

A generalist cannot reliably tell whether a model’s drug interaction answer or contract clause is correct, so they may reward the response that sounds more authoritative. This is the main reason pricing has moved upmarket: frontier work now leans on physicians, lawyers, engineers and PhD researchers. Teams training models for health care should also look at specialist medical data annotation companies, where clinical credentials and HIPAA handling are standard requirements.

Contributor Welfare

Red teaming and harmful content labeling expose workers to disturbing material. TIME reported in January 2023 that Kenyan workers employed by Sama to label toxic text for OpenAI took home roughly $1.32 to $2 per hour, while the contract rate was $12.50 per hour, which Sama said covered infrastructure, benefits and quality assurance staff. Buyers increasingly ask vendors about wellness support, pay floors and opt-out policies.

What RLHF Data Costs

Few vendors publish buyer prices for RLHF work; most quote per project. What is public are contributor pay rates and some platform fees, which set a floor on what buyers pay.

Published Rates And Reported Figures

Source	Figure	What It Covers
Mercor homepage	$93 per hour average contracted rate; oncologist role listed at $350 to $450 per hour	Expert contractor pay
DataAnnotation homepage	Up to $50 per hour generalist; up to $125+ professional; up to $150+ coding	Contributor pay ceilings
Prolific pricing page	42.8% platform fee for corporate customers; $8 per hour minimum participant pay, $12 recommended	Self-serve platform cost
TIME (January 2023)	$12.50 per hour contract rate billed to OpenAI by Sama	Managed toxic content labeling in Kenya
Business Insider (October 2025)	Mercor CEO Brendan Foody said the company paid over $1.5 million a day to contractors	Market scale indicator

DataAnnotation lists its rates as “up to” figures, so they are ceilings rather than averages. These are amounts paid to contributors. A managed vendor adds recruiting, quality review, project management, tooling and margin on top, so the buyer’s effective cost per hour is higher than the contributor rate.

What Drives The Price

• Expertise level. Generalist comparisons cost a fraction of what board-certified physicians or senior engineers cost per hour. Published rates above span roughly an order of magnitude.
• Task length and complexity. A quick A/B preference on a short answer takes seconds; grading a multi-step agent trajectory or writing a verified math problem can take an hour or more.
• Quality controls. Overlap (multiple labelers per item), expert review layers and gold-standard checks multiply labor hours.
• Language and locale. Low-resource languages and native-speaker requirements narrow the pool.
• Turnaround and volume. Rush timelines and sudden scale-ups cost more; committed volume usually earns discounts.
• Security and exclusivity. Unreleased model access, on-premise work, background checks and exclusive data rights raise prices.

Estimating A Budget

Work backward from hours: multiply items by average minutes per item, add overlap and review time, then apply the contributor rate for the required expertise plus the vendor’s fee or markup. Data costs are only one line of a model budget; our guide to AI development cost covers compute, engineering and deployment.

How To Buy RLHF Data

Define The Signal You Need

Decide whether you need demonstrations, comparisons, rubrics, verified problems or red teaming, and which training method will consume them. DPO and reward models need pairwise preferences; RLVR needs checkable problems; safety work needs adversarial prompts and policy labels.

Write Guidelines And A Gold Set First

Draft clear labeling guidelines with worked examples of edge cases, then label a gold set internally. The gold set becomes the benchmark for vendor pilots and ongoing quality checks.

Run A Paid Pilot With Two Or Three Vendors

Give each vendor the same guidelines and a few hundred tasks. Compare agreement with your gold set, inter-annotator agreement, justification quality, throughput and how they handle questions. Ask who exactly did the work and what credentials they hold.

Check Security, Ownership And Welfare Terms

Confirm data ownership, confidentiality, model access controls, certifications such as SOC 2 where relevant, and the vendor’s policies on contributor pay and exposure to harmful content.

Monitor And Iterate

Preference data goes stale as the model improves, so plan recurring batches and watch trained models for reward hacking or sycophancy.

Frequently Asked Questions

What Is RLHF In Simple Terms?

RLHF is a way of training an AI model using human judgments as the scoring system. People compare or rate model answers, a reward model learns those preferences, and the AI is then tuned to produce answers the reward model scores highly.

Is RLHF Still Used, Or Has DPO Replaced It?

Both are used. DPO removes the reward model and RL loop but still relies on human preference pairs, so many teams treat it as a form of preference tuning within the broader RLHF family. Labs also combine preference methods with RLVR for math and code and with AI feedback for safety.

What Is The Difference Between RLHF And RLAIF?

In RLHF, humans provide the preference labels. In RLAIF, an AI model provides them, usually guided by written principles, as in Anthropic’s Constitutional AI. RLAIF lowers labeling volume, but humans still set the principles and validate the AI judge.

How Much Human Feedback Data Does A Model Need?

It varies widely. InstructGPT used about 33,000 prompts for reward modeling, with several ranked responses each, while Meta collected over 1.4 million binary comparisons for Llama 2. Smaller domain-specific fine-tunes can work with far fewer, high-quality examples.

Why Is RLHF Data So Expensive?

Because the most useful feedback now comes from experts who know more than the model, and their time is costly. Published expert rates reach hundreds of dollars per hour, and quality controls such as overlapping labels and expert review add further hours.

Conclusion

RLHF turned fluent but unreliable language models into assistants people can use, by replacing hand-written scoring rules with human judgment. The basic recipe of demonstrations, comparisons, a reward model and policy optimization still underpins most post-training, even as DPO simplifies the optimization, RLAIF scales feedback with AI judges, and verifiable and rubric-based rewards move the human role toward writing hard problems and grading criteria.

For buyers, the practical lesson is that the data matters more than the algorithm. Agreement rates, expert credentials, clear guidelines and protection against reward hacking determine whether feedback improves a model. Pricing follows expertise, so define the signal you need, pilot vendors against a gold set and budget from hours, not item counts.