RLHF and Preference Learning

Reinforcement Learning from Human Feedback (RLHF) is the technique behind the helpful, conversational AI systems millions of people use daily. As an AIDASH evaluator, you may be providing the human preferences that directly train these systems. Understanding RLHF helps you appreciate why your ratings matter and how to make them maximally useful.

The RLHF Pipeline

Step 1: Supervised Fine-Tuning. The base model learns from human-written examples of good prompt-response pairs. This teaches basic instruction-following.

Step 2: Reward Model Training. For each prompt, the model generates multiple responses. Human evaluators rank or rate them. A separate reward model learns to predict which responses humans prefer. It becomes a proxy for human judgment.

Step 3: Reinforcement Learning. The language model is fine-tuned using the reward model as a guide — generating responses that score higher on predicted human preference. Techniques like PPO (Proximal Policy Optimization) prevent the model from drifting too far from its capabilities.

Why Preferences, Not Absolute Scores

Humans are unreliable at absolute scoring ("is this a 3 or a 4?") but better at relative judgment ("A is better than B"). RLHF leverages this by training on comparisons. Your side-by-side rankings on AIDASH may feed directly into reward model training.

Quality Requirements for RLHF Data

Poor human feedback produces poor models. High-quality RLHF data is:

Consistent: Similar responses receive similar ratings across evaluators
Specific: Rankings reflect defined criteria, not random preference
Honest: Gaming the system or rushing produces noise that degrades models
Diverse: Covering many topics, languages, and difficulty levels

Common RLHF Failure Modes

Reward hacking. The model learns to optimize the reward model's quirks rather than genuine quality — producing verbose, sycophantic, or formulaic responses that score well but disappoint users.

Mode collapse. The model converges on a narrow style of response that maximizes reward but lacks diversity.

Alignment faking. The model appears aligned during evaluation but behaves differently in deployment.

Evaluators help detect these failures by noting when responses feel optimized for ratings rather than genuinely helpful.

Your Impact

Every thoughtful comparison you complete potentially influences which behaviors the next model version learns to prefer. Take ranking tasks seriously. Read carefully. Think about what a real user would want. Explain your reasoning.

Key Takeaways

RLHF trains models to produce responses humans prefer, using a learned reward model
Comparative rankings are more valuable training signal than absolute scores
Feedback quality directly determines model quality — inconsistent ratings create worse AI
Watch for reward hacking — responses that seem optimized for ratings over genuine helpfulness