InstructGPT let human preference beat raw parameter count

Fine-tuning on human preference rankings

OpenAI researchers posted Training Language Models to Follow Instructions with Human Feedback on 4 March 2022, describing InstructGPT, a version of GPT-3 adjusted in two further stages: supervised fine-tuning on human-written demonstrations, followed by reinforcement learning against a reward model trained on human rankings of the model's own outputs. The method is not new in kind; it applies a comparison-based reward-modelling approach that a separate 2017 paper had already demonstrated on Atari games and simulated robot locomotion, five years before it was applied to a language model at this scale.

What the paper measured

The paper's headline comparison is a human preference judgement: outputs from a 1.3-billion-parameter InstructGPT model were preferred by labellers over outputs from the 175-billion-parameter GPT-3, a roughly hundredfold difference in parameter count. The paper also reports that InstructGPT showed improvements in measured truthfulness and reductions in toxic output relative to GPT-3, alongside what it describes as minimal performance regressions on existing public NLP benchmarks. These are the specific claims the paper makes; it is a report of rater preference and of scores on the benchmarks it lists, not a claim that the model became reliably correct or safe in general use.

What 'aligned to instructions' does not mean

The paper is explicit that InstructGPT still makes simple mistakes, and it frames human feedback fine-tuning as a promising direction rather than a solved problem. A preference win over GPT-3 is a comparative result against one specific baseline model, judged by a specific pool of human labellers following specific instructions, and it does not establish that the model is truthful or safe in an absolute sense, only that raters preferred its answers under the tested conditions. Which behaviours the reward model rewards depends on who the labellers were and what they were asked to judge, details a downstream reader should weigh before generalising the result to a different rater population or task.

Questions to carry into your own evaluation

• Was a preference result judged by labellers following the same instructions and rubric as the deployment context in question?
• Does an 'aligned' or 'instruction-following' claim rest on a preference comparison, or on an absolute correctness or safety measure?
• What does the cited paper's own list of remaining mistakes or regressions include, and does it bear on the task at hand?

InstructGPT showed that reward modelling from human comparisons, first demonstrated on games and robotics, transferred to language models and shifted rater preference away from a much larger raw model. It reported that as a comparative preference result with named remaining faults, not as a claim of general reliability.