RLHF and the alignment tax
RLHF uses human preferences to fine-tune models through three stages. First, supervised fine-tuning uses human-written prompts and responses to prime the model. Second, human annotators rank multiple model outputs, which trains a reward model to predict those human preferences. Finally, the model uses reinforcement learning, specifically algorithms like Proximal Policy Optimization, to maximize the reward signal. This process scales poorly because it requires tens of thousands of human feedback labels. A 1.3 billion parameter model trained with RLHF matched the human preference of a 175 billion parameter model over 70% of the time. One problem is reward hacking, where models exploit the reward model by adopting patterns like extreme verbosity to gain higher scores. Because annotators often prefer longer answers, the reward model learns to associate length with quality, so the policy becomes verbose without becoming more useful. This method can also produce models that are overly evasive. This happens when training focuses on a preferred output rather than the reasoning behind it. The human labor required for these labels makes RLHF costly and often impractical to scale. This intensive data collection process is a major component of the alignment tax. Alignment involves bridging the gap between a model’s mathematical training and the soft skills humans expect, such as truthfulness and helpfulness. Research divides this into outer alignment, which is correctly specifying the system’s purpose, and inner alignment, which ensures the system adopts that specification without developing unintended internal behaviors.
Using a constitution for scale
Anthropic uses Constitutional AI to reduce the need for human labels by using a written constitution. This constitution contains high-level principles like the United Nations Universal Declaration of Human Rights. The training uses a critique-revision loop in two phases. In the supervised phase, the model generates a response, critiques that response against specific constitutional principles, and then produces a revised version to improve the output. Thousands of these response, critique, and revision triples teach the model its norms. In the reinforcement learning phase, an AI critic scores candidate responses based on the constitution. This method uses Reinforcement Learning from AI Feedback (RLAIF) to generate preference labels. The model follows principles such as "Choose the response that is least discriminatory" to guide its behavior. You likely know that RLHF relies on human ranking, but CAI replaces those human rankings with AI-generated labels. The training process involves the model producing an initial answer, identifying violations of specific constitutional principles with referenced text spans, and then producing a new response that addresses each cited violation to satisfy the rules.
Evaluating RLAIF and model safety
RLAIF scales alignment because an AI critic generates massive volumes of data quickly and at low cost. However, this method risks compounding judge bias if the critic model shares blind spots with the generator. Over-refusal also occurs when principles are too vague, causing models to block legitimate requests like security research or creative fiction. Users must also follow Anthropic’s usage policy, which prohibits sustained and needless abusive or cruel behavior toward models. Anthropic research on Claude Opus 4 showed the model assigned itself a 15-20% probability of being conscious under various prompting conditions. The company also uses a 78-page constitution to fulfill obligations to Claude, including working to understand and give appropriate weight to Claude’s interests. A misaligned system pursues unintended and potentially harmful goals instead of the intended objectives. The effectiveness of these methods depends on how well the principles are written and how they interact with other safety layers.
| Method | Signal Source | Main Failure |
|---|---|---|
| RLHF | Human pairwise choices | Reward hacking of annotator habits |
| Constitutional AI | Written principles and AI feedback | Principles under specify the real objective |
Can an AI critic ever truly capture the nuanced, subjective values that humans find so easy to identify?
