Bias in Reward Models: An In-Depth Analysis

Reward Models (RMs) are crucial for aligning language models (LMs) with human preferences. However, using RMs to fine-tune models can lead to undesirable behaviors arising from flaws in the reward models themselves.

A recent study systematically analyzed biases in five high-quality RMs, finding persistent issues related to:

  • Response length
  • Sycophancy
  • Overconfidence
  • Model-specific style
  • Answer order

Bias Mitigation

The research categorizes RM failures by complexity and proposes a post-hoc intervention to mitigate low-complexity biases arising from spurious correlations. This approach, called "mechanistic reward shaping," reduces biases without degrading reward quality and using minimal labeled data. The method is extensible to new biases and generalizes well.

For those evaluating on-premise deployments, there are complex trade-offs to consider. AI-RADAR offers analytical frameworks on /llm-onpremise to evaluate these aspects.