Chatbot Arena traded a fixed answer key for human votes

Two different ways to avoid a fixed answer key

Both papers respond to a shared problem: benchmarks with a single correct answer cannot easily score open-ended chat. The Judging LLM-as-a-Judge paper, submitted to arXiv on 9 June 2023, introduces two things at once: MT-Bench, a set of multi-turn questions graded by a strong model acting as judge, and Chatbot Arena, a crowdsourced site where people submit a prompt, see two anonymous model responses, and vote for the better one. A later paper, Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference, submitted 7 March 2024, describes the platform after it had collected more than 240,000 votes and sets out the statistical ranking methods used to turn pairwise votes into a leaderboard.

Where judge scores tracked human votes, and where they did not

The judge paper's central measurement is agreement: it reports that a strong judge model matched both controlled and crowdsourced human preference judgements more than 80 percent of the time, comparable to the rate at which two human judges agree with each other. That is a specific, bounded claim about agreement rate under the paper's own test conditions, not a claim that a model judge is equivalent to a human rater in general. The same paper documents named failure modes: position bias, where the order responses are shown in shifts the verdict; verbosity bias, where a longer answer is favoured regardless of quality; and self-enhancement bias, where a model judge rates its own outputs more highly. It also notes that reasoning-heavy comparisons are where a judge's own limited reasoning ability is most likely to produce the wrong call.

What an Elo-style rating leaves out

The Arena paper builds its leaderboard from pairwise votes using statistical ranking methods rather than a fixed test set, which is what lets it rank models people had never been asked to compare directly. That design has a cost: a rating reflects the mix of prompts users happened to submit and vote on, not a fixed task distribution, so a change in who is using the site or what they ask about can move a ranking without any model changing. A single Elo-style number also collapses many kinds of quality, tone and safety preference into one ordering, unlike a benchmark that reports accuracy on a defined task. The successor platform, now at arena.ai, where lmarena.ai now redirects, keeps the same battle-and-leaderboard format, retrieved 16 September 2026.

Questions to carry into your own evaluation

• Is a reported score a judge-model verdict, a human vote count, or a combined ranking, and which of the documented biases could apply to it?
• Does the comparison being judged resemble the open-ended chat both papers test, or a task, such as multi-step arithmetic, where judge reasoning is weaker?
• What population of prompts produced a cited Arena rating, and does it resemble the questions your own use case will ask?

Both methods were built to solve a real problem, grading open-ended answers at scale, and both papers are candid about the biases their approach introduces alongside the coverage it buys.