BLOG
Case Study
RLHF vs SFT vs Model Evaluation: What's the Difference?
October 11, 2026
TL;DR
RLHF vs SFT vs model evaluation comes down to what the person doing the work is handing over. In SFT, they write the ideal answer. In RLHF, they decide which of the model's answers is better, and in evaluation they score the model against a rubric. Each output lands somewhere different, which is why each one calls for a different kind of expert, something we see every week running expert RLHF workstreams at Athyna Intelligence.
SFT produces demonstrations, RLHF produces preference data, and model evaluation produces measurements. The first two go straight into training, and the third tells the team whether the model is ready and what to train next.
OpenAI's InstructGPT paper shows all three side by side, and it's the method OpenAI later used to train ChatGPT. Labelers wrote demonstrations for supervised fine-tuning, ranked model outputs for RLHF, and then rated the finished models in human evaluations. The payoff was big. Raters preferred answers from the 1.3-billion-parameter model trained this way over the 175-billion-parameter GPT-3, a model 100 times larger.
Labs rarely run these as tidy, one-time stages. Meta's Llama 2 team collected preference data in weekly batches, trained five successive RLHF versions, and evaluated each one before the next round began. Post-training is a loop with a lot of laps.
Each output changes something different about the model.
The middle column is where most of the confusion lives, because ranking a model's outputs looks a lot like evaluating them.
Supervised fine-tuning trains a model on pairs of prompts and ideal responses, so it learns to answer the way a skilled person would. The human writes, or carefully curates, the exact response the model should give. These pairs are usually the first training a base model gets after pretraining, and one of several kinds of labeled data a language model depends on.
Small, sharp datasets go a long way in SFT. LIMA, a study from Meta AI and university partners, fine-tuned a model on just 1,000 curated examples with no RLHF at all. People rated its answers as good as or better than GPT-4's in 43% of comparisons.
The catch is the ceiling. Llama 2's researchers wrote that an SFT model's performance is capped by the writing ability of its most skilled annotators, and that the model also picks up the tail of poorly written examples. Learning from recipe cards works the same way, since you'll never cook better than the cards. For a medical or legal model, whoever writes the examples has to be the expert you'd want answering the question yourself.
RLHF, or reinforcement learning from human feedback, turns rankings into behavior in two moves. People compare a model's outputs and pick the better one, and those choices train a reward model. The reward model then scores new outputs while the main model learns to chase higher scores.
RLHF depends on human experts because a reward model has no independent sense of quality, so it learns whatever the raters prefer, blind spots and all. OpenAI's ChatGPT launch post describes training with RLHF using InstructGPT's methods, and it traces the model's wordiness partly to trainers who preferred longer answers that looked more comprehensive. One small rater habit, repeated thousands of times, became the model's personality.
Close calls are where expertise earns its keep. Llama 2's helpfulness reward model agreed with human raters 80.7% of the time when they had marked one response "significantly better," and only 54.7% of the time when they'd called it "negligibly better," barely above a coin flip. Near-ties like these are where a domain expert spots a difference a generalist would miss.
Newer methods like Direct Preference Optimization (DPO) and AI-generated feedback change the plumbing, and both still run on, or get checked against, human judgment.
One comparison at a time. A rater reads a prompt and two or more model responses, then decides against a written rubric which one is better, and each of those decisions becomes a row of preference data. A typical comparison runs through four steps.
At Athyna Intelligence, our finance RLHF workstream ran about 1,500 paired comparisons and 300 to 400 senior expert hours a week, across a bench of up to 20 experts. Every comparison came with a ranking and a two-to-four-sentence rationale citing specific numbers or tool calls from the transcripts. Vague rationales were rejected outright.
Volume like that only holds up with quality checks built in. Every expert completed calibration before touching live work, pre-scored gold tasks were seeded invisibly into batches, and experts double-rated overlapping work so we could measure how often they agreed.
Every round of post-training ends with a verdict, and model evaluation delivers it. Evaluators score a model against a rubric or a fixed test set, and those scores decide which version moves forward, whether it ships, and which weaknesses the next round of training should target.
Rankings and evaluation scores end up in different places. A ranking feeds the reward model and changes the model, while an evaluation score stays out of training so it can measure the model fairly.
The same logic is why training and evaluation work should run as separate workstreams, even when the same people do both. Meta's Llama 2 team made a similar call, citing Goodhart's law when it declined to treat its own reward model as the only judge.
Professionals averaging 14 years of experience built the 1,320 tasks in GDPval, OpenAI's benchmark of real-world professional tasks, and experts from the same 44 occupations graded model outputs blind. OpenAI said its automated grader isn't yet reliable enough to replace them, a sign that evaluation needs experts as much as training does.
Helpfulness sits firmly in the eye of the beholder. In Llama 2's human evaluations, rater agreement on helpfulness scored between 0.37 and 0.55, compared with 0.70 to 0.95 on safety. When raters split that widely, the people writing the rubric matter as much as the people applying it.
If a person can write down the right answer and the model mainly needs format, task structure, or domain style, supervised fine-tuning is enough. RLHF earns its extra cost once quality turns into a judgment call that people recognize more easily than they can write.
Meta hit that turning point while building Llama 2. After checking 180 examples by hand, the team found its SFT model's answers were often as good as the ones annotators had written, so it shifted annotation effort toward preference data.
Some answers can be checked by a program, like code that runs or a figure that matches a ledger. For those, teams can use reinforcement learning with verifiable rewards (RLVR), where automated checks score each output. Experts still decide what the checks should be.
The right method depends on what a person can produce most reliably.
Different outputs call for different strengths. Writing a demonstration takes the skill of a strong practitioner, while ranking takes a sharp eye for small differences and the discipline to judge them the same way every time. Evaluation leans hardest on rubric design.
SFT, RLHF, and model evaluation each break in a different way when the wrong person does the work.
Evaluation can call for a wide range of expertise. For Llama 2's red-teaming, Meta drew on more than 350 people, including domain experts in cybersecurity, election fraud, law, and civil rights.
Job titles blur all of this, which keeps hiring managers on their toes. An AI trainer might write examples on Monday, rank responses on Tuesday, and review domain-specific outputs by Friday, so the skills behind the work matter more than the title.
The quickest way to sort RLHF vs SFT vs model evaluation is to ask what each person hands over and where it lands. Get that question right, and the staffing decisions that follow mostly sort themselves out.
If you're building a bench for expert demonstrations, preference ranking, or evaluation, Athyna Intelligence matches AI teams with vetted PhDs and domain experts for this kind of work, at lightning speed. For US teams, hiring AI trainers from Latin America adds real-time overlap with your working hours, so feedback loops close the same day.
Athyna's content specialists covering global hiring and AI training trends for growing teams.
.svg)
The difference lies in what the person produces. SFT workers write ideal answers for the model to imitate, RLHF workers rank the model's own answers to train a reward model, and evaluators score the model against a rubric. The first two feed training directly, and evaluation results decide what ships and what gets trained next.
RLHF, short for reinforcement learning from human feedback, trains a model on people's judgments about which of its responses is better. It needs experts because the reward model copies whatever raters prefer, mistakes included, and the hardest comparisons are near-ties that only someone who knows the subject can call reliably.
Use SFT when a person can write the correct answer and the model mainly needs format, structure, or domain style. Move to RLHF when quality becomes easier to recognize than to write, as with tone or helpfulness. Many teams start with SFT and add preference training once the model's answers rival what annotators can write.
Model evaluation runs alongside training. Evaluators produce scores that measure the model, and those scores stay out of the training data so they remain trustworthy. The results still steer training, since they decide which model version moves forward and which weaknesses the next round of fine-tuning should target.
A person working on RLHF reads a prompt and two or more model responses, picks the better one against a rubric, and often rates how much better it is. Many projects also ask for a short written rationale. Each judgment becomes one row of preference data that trains the reward model.
