Human comparison-based reward models predate chat by years

Learning a reward function from comparisons, not a score

On 12 June 2017, researchers from OpenAI and DeepMind posted Deep Reinforcement Learning from Human Preferences. The method it demonstrates trains a reward model not from a hand-written reward function, but from human comparisons between pairs of short trajectory segments: a person is shown two clips of an agent's behaviour and picks the better one, and a model learns to predict which of two segments a human would prefer. A reinforcement-learning agent is then trained against that learned reward model rather than against a designed one.

What the paper tested and how much feedback it needed

The paper reports testing this approach on Atari games and on simulated robot locomotion tasks, describing the resulting behaviours as considerably more complex than reinforcement learning from human feedback had previously achieved. It states that useful reward models could be learned from feedback on less than one percent of the agent's interactions with its environment, and that some behaviours were trained using about an hour of total human labelling time. The point being demonstrated is that comparison-based feedback, rather than a numeric score for every action, can supply enough signal to train a working policy on tasks where a reward function is hard to specify by hand.

What carried over five years later

The same comparison-based reward-modelling structure reappears in OpenAI's 2022 InstructGPT paper, which trains a reward model on human rankings of language-model outputs and then fine-tunes the language model against it. The 2017 paper never mentions language models, chat interfaces or instruction-following; its tested domains are Atari and simulated robotics, and its measured outcome is task performance on those specific benchmarks, not any property of text generation. What transferred was the general mechanism, comparison-based reward learning with limited human labelling, not any specific finding about what makes a good chat response, which the later paper had to establish separately with its own labellers and its own tasks.

Questions to carry into your own evaluation

• Is a claim about reinforcement learning from human feedback citing the mechanism this 2017 paper demonstrated, or a specific later result on a different domain?
• How much human labelling did a given reward model actually require, and does that figure come from the cited paper or a different one?
• Does a comparison-based reward model transfer its behaviour across domains, or does each new domain need its own labelled comparisons?

The 2017 paper's contribution was a general mechanism, tested on games and robotics, for training a reward model from human comparisons rather than a hand-designed score. Its later reuse for language models is a separate, later-demonstrated application of that mechanism, not evidence the 2017 paper itself provided about chat behaviour.