A model judge can be swayed by order and length alone

Order changed which answer won

Submitted on 29 May 2023, Large Language Models are not Fair Evaluators reported that when GPT-4 was asked to judge which of two candidate responses was better, the ranking could be reversed by simply swapping the order in which the two responses were presented. The paper describes this as a way to make a weaker model, Vicuna-13B, appear to beat a stronger one, ChatGPT, on the majority of a test set purely through positional manipulation, with no change to either model's actual output. This matters wherever an automatic judge stands in for human preference, because a benchmark built on such comparisons can be moved by the order candidates are listed rather than by the quality of what they wrote.

Averaging out the position instead of trusting one pass

The authors tested three calibration strategies rather than treating the bias as unfixable: multiple evidence calibration, which asks the judge to produce reasoning before scoring; balanced position calibration, which runs the comparison in both orders and combines the results; and human-in-the-loop calibration, which flags cases where the judge disagrees with itself across orderings for human review. The complementary study introducing MT-Bench and Chatbot Arena, submitted 9 June 2023, reports related biases in the same setup: a verbosity bias that rewards longer answers regardless of quality, and a self-enhancement bias in which a judge favours output resembling its own style. That paper reports over 80 percent agreement between its calibrated judge and human preference on its own test set, evidence for that setup and that judge, not a guarantee any deployment of a model judge reaches the same figure.

Judges also recognise their own writing

A later paper, LLM Evaluators Recognize and Favor Their Own Generations, submitted 15 April 2024, found that self-preference is not incidental: models such as GPT-4 and Llama 2 can distinguish their own outputs from a human's or another model's above chance, and the paper reports a linear correlation between how well a model recognises its own writing and how strongly it favours it. This gives the self-enhancement bias observed in 2023 a mechanism rather than leaving it unexplained, and it is a separate failure mode from position bias, meaning a mitigation for one does not automatically fix the other.

Questions to carry into your own evaluation

• Have you run comparisons in both orders and checked whether the verdict changes, rather than trusting one pass?
• Is the judge from the same model family as one of the candidates it is scoring, which the self-recognition finding suggests could bias the result?
• Does a longer answer win your comparisons even when a shorter one addresses the task, a pattern distinct from either bias above?

A model judge is a measurement instrument with its own documented failure modes, not a neutral proxy for a human rater, and treating a single judged score as a verdict skips the calibration work these three papers show is needed to make that score mean what it appears to mean.