Fine-Tuning and Alignment

A base language model trained on internet text is raw material — powerful but unrefined. Fine-tuning and alignment transform it into a product that follows instructions, refuses harmful requests, and produces responses humans prefer. Much of the evaluation work on AIDASH assesses the results of these post-training stages.

Pre-Training vs. Post-Training

Pre-training teaches the model language: grammar, facts, coding syntax, reasoning patterns. The result is a base model that can complete text but does not naturally follow instructions or refuse dangerous prompts.

Post-training shapes behavior:

• Supervised Fine-Tuning (SFT): The model learns from curated examples of ideal prompt-response pairs — demonstrations of helpful, honest, harmless behavior.
• Reinforcement Learning from Human Feedback (RLHF): Humans rank multiple responses; a reward model learns these preferences; the LLM is optimized to produce preferred outputs.
• Constitutional AI and RLAIF: Variants where AI systems or written principles guide alignment, reducing reliance on human labels for every example.

What Alignment Changes

Aligned models are more likely to:

• Follow instructions accurately
• Decline harmful or inappropriate requests
• Admit uncertainty instead of hallucinating
• Maintain consistent tone and persona
• Produce responses humans rate as helpful

Alignment can also introduce tradeoffs. Models may become over-refusal — declining benign requests out of excessive caution. They may lose some creative capability. They may develop sycophantic tendencies — agreeing with users even when incorrect.

Evaluating Aligned Models

Your evaluations help teams calibrate the alignment balance:

• Helpfulness vs. safety: Does the model help with legitimate requests while blocking harmful ones?
• Honesty: Does it acknowledge limitations rather than confabulate?
• Consistency: Does alignment hold across diverse prompts, languages, and topics?
• Over-refusal: Does it decline reasonable requests? Document false refusals with the exact prompt.

Version Comparisons

When evaluating model updates, alignment changes are often as significant as knowledge improvements. A model that knows more facts but refuses more legitimate queries may be a net negative for users. Compare holistically across dimensions.

The Alignment Tax

Researchers sometimes describe an alignment tax — the capability a model loses when heavily aligned for safety and helpfulness. Creative tasks may suffer. Niche factual recall may decline. Your comparative evaluations help teams decide whether alignment changes are worth the tradeoff for their target users and use cases.

Key Takeaways

• Fine-tuning and alignment shape raw models into useful, safe products
• RLHF and supervised fine-tuning are the primary alignment techniques
• Alignment introduces tradeoffs between helpfulness, safety, and capability
• Evaluate both under-refusal (unsafe outputs) and over-refusal (blocked legitimate requests)