RLHF annotation quality: why preference labels disagree and what to fix

Annotation for reinforcement learning from human feedback (RLHF) looks inconsistent because preference data asks raters for a judgment, and OpenAI’s 2022 InstructGPT paper reports labelers agreeing only 72.6% to 77.3% of the time. Good RLHF annotation quality depends on separating the disagreement that comes with the task from the inconsistency that points to a broken rubric, interface, or rater. You’ll get a benchmark to plan around, a way to sort expected splits from defects, and fixes built for preference data.

Why is RLHF annotation quality inconsistent?

RLHF annotation quality is inconsistent because preference data often has no single right answer. Raters choose which of two model responses is better, and two qualified raters can disagree. In classification-style data annotation, such as tagging an entity in a contract, a correct label exists, so disagreement means someone made a mistake.

When both responses are accurate and differ only in length, tone, or structure, raters can split without either one being wrong. OpenAI’s InstructGPT and summarization papers show how far below 100% preference data sits.

Plan around the low to high 70s for open-ended helpfulness, and expect higher numbers on narrow, well-specified criteria.

How can you tell real disagreement from inconsistent human feedback?

Real disagreement tracks taste and close calls, while inconsistent human feedback tracks something your team controls, like the rubric, the interface, or a single rater. Splits spread across style choices are expected, and splits concentrated in one criterion, rater, or response position are defects.

A 2025 study by Zhang and colleagues found that most splits in two preference datasets came from underspecified tasks and response style, so treating every split as error throws away information about users. Meta’s 2023 Llama 2 paper adds that annotators agreed more often on clearly different pairs than on similar ones. Each signal below is either expected or fixable.

The expected rows set the floor for agreement, and the fixable rows are where agreement gains come from.

How do you improve preference data quality in RLHF?

You improve preference data quality in RLHF by fixing the defects behind inconsistent labels while keeping real disagreement as data. Guidelines, gold sets, and RLHF annotator calibration still apply (our guide to data labeling best practices covers them), but preference data needs six fixes of its own.

1. Rate the margin as well as the winner. Meta’s Llama 2 annotators rated each choice on a four-point scale from significantly better to negligibly better or unsure, a setup we cover in RLHF vs supervised fine-tuning (SFT) vs model evaluation, so near-ties stop counting as clear wins.
2. Judge helpfulness and safety separately. The 2023 BeaverTails team at Peking University found that a model trained on separate helpfulness and harmlessness rankings outperformed one trained on a single combined ranking, and traced the gap to the ambiguity of folding two judgments into one label.
3. Write tie-break rules for style. Decide in the rubric how to weigh length, formatting, and refusal wording, the issues a 2025 study of diverging preferences by Zhang and colleagues linked to many splits.
4. Randomize which response appears first, then check whether choices still follow position.
5. Require a short rationale tied to the rubric. In Athyna Intelligence’s finance RLHF workstream, every comparison carried a two-to-four-sentence rationale citing specific numbers or tool calls, and generic preference statements were rejected.
6. Keep real splits as data. The Zhang and colleagues study also showed that majority voting trains reward models to pick a winner even when people disagree.

Should you use domain experts or generalist annotators for RLHF?

Use generalist annotators for style and general helpfulness, and PhDs and domain experts when picking the better response depends on specialist knowledge. Only an expert can tell when a confident answer rests on the wrong reasoning, and in specialized domains those comparisons decide RLHF annotation quality.

The rater pool moves agreement on its own. Anthropic’s 2022 helpful and harmless paper reported about 63% agreement between its researchers and lightly vetted crowdworkers, which the authors called poor next to OpenAI’s results. Expert judgment also holds up at production volume: in our finance RLHF and RLVR (reinforcement learning with verifiable rewards) case study, up to 20 experts in US Generally Accepted Accounting Principles (GAAP) handled about 1,500 paired comparisons a week. For an AI lab, that means rubric-grounded expert judgments don’t have to come at the cost of throughput. A task needs domain experts when:

• The right answer depends on a professional standard, such as US GAAP or a clinical guideline.
• A plausible final number can rest on reasoning only a specialist would catch.
• Rationales must cite domain specifics before a reviewer can audit them.

What good human feedback data quality looks like in RLHF

Good human feedback data quality in RLHF means agreement near the 72% to 77% benchmark for open-ended tasks, defects traced and fixed, and genuine disagreement kept as data. Chasing 100% wastes budget while erasing how people differ. To improve RLHF annotation quality, audit where your splits cluster and match each fix to its cause: the rubric, the interface, or a rater pool that lacks the expertise.

Athyna Intelligence matches AI labs with vetted PhDs and domain experts, mainly from LATAM, with most work happening within US working hours. Candidates are vetted and presented within 4 days, at 40% to 60% below US rates. Talk to Athyna Intelligence about expert raters for your RLHF workstream.

‍