Constitutional AI used model feedback to reduce human labelling

Two stages, and a name for the second

Anthropic's paper, submitted 15 December 2022, describes a two-stage method for training what it calls 'a harmless but non-evasive AI assistant'. In the first stage, a model generates a response, then critiques and revises that response against a written set of principles, and the model is fine-tuned on its own revised outputs. In the second stage, a model compares pairs of sampled responses against the same principles to build a preference dataset, which trains a reward model exactly as human comparison data would in standard reinforcement learning from human feedback, except the comparisons come from a model rather than a person. The paper names this second stage 'RL from AI Feedback', RLAIF, deliberately mirroring the RLHF terminology used in OpenAI's InstructGPT paper, which supplied the human-feedback method this one is built to partly replace.

What the paper reports the method achieves

Anthropic's summary of the paper states the resulting assistant 'engages with harmful queries by explaining its objections to them' rather than refusing outright or evading the question, which the authors present as an improvement over models that become 'evasive' when trained purely to avoid harmful outputs. The paper also reports that adding chain-of-thought reasoning to the AI-feedback step improved both the transparency of the model's judgments and, in the authors' human evaluations, the resulting harmlessness. Both claims are the paper's own reported findings from its own evaluation process, using evaluators and criteria the authors selected, and are not independently reproduced results.

What the method leaves unresolved

The method replaces human labelling of individual harmful examples, but it does not remove human judgement from the process: a person still writes the constitution of principles the model critiques itself against, and that document's content, scope and omissions set the boundaries of what the trained model will treat as harmful. Neither source opened for this entry publishes the constitution's specific wording or explains how its principles were chosen, which means a reader can evaluate the method's mechanism but not, from these documents alone, the specific values it encoded in this instance. Whether a written set of principles generalises more predictably than a large set of labelled examples is also a claim about consistency, not one this paper measures directly against a labelled-data baseline trained to the same scale.

Questions to carry into your own evaluation

• Who authored the constitution or principle set behind a given Constitutional-AI-trained model, and is that document available to inspect?
• Does a claimed harmlessness improvement come from the authors' own human evaluation, or from a separate, independently run comparison?
• Where a model is trained on its own AI-generated feedback, what independent check exists that the feedback model is applying the stated principles consistently?

Constitutional AI is a method for shifting labelling effort from individual examples to a written set of principles, evaluated in the paper's own tests as improving non-evasive harmlessness. It is not, on the evidence in these documents, a method that removes human judgement from the system, only one that relocates where that judgement is applied.