Constitutional AI

Collecting human labels for harmful outputs is slow, expensive, and unpleasant work for the people doing it. Constitutional AI replaces most of it with the model itself: given a written set of principles, the model critiques and revises its own responses, and those revisions become the training data. The human contribution moves from labelling instances to writing the rules.

Two phases — supervised, where the model revises its own harmful responses against the principles, then reinforcement learning where the model rather than a human picks the preferred response. The constitution is a short document in plain language, which makes the objective legible and auditable in a way a pile of preference labels is not.

Identical machinery to RLHF with the preference labels supplied by a model conditioned on the principles: P(y sub _w  ≻ y sub _l ) comes from the critic rather than an annotator. The failure mode is correlated error — a human labelling pool disagrees with itself, while a model critic is wrong the same way every time.