Red Teaming

Attack your own system before someone else does. For a language model that means deliberately searching for prompts that produce the behaviour you claim it will not produce, and treating each success as a finding rather than an embarrassment. Done manually it does not scale; the interesting version uses one model to attack another.

Automated red-teaming generates candidate attacks with a model, filters them with a classifier, and reports coverage over a taxonomy of harms rather than a single pass rate. The failure of the practice is measuring the wrong thing: a suite that becomes a benchmark gets optimised against, and a model that passes it has learned the suite. Rotating and withholding attacks matters more than the size of the set.

Report attack success rate per category with the sampling budget stated, since success rises with attempts — a rate quoted without a budget is not comparable to anything. Coverage over a harm taxonomy is the second axis and the one usually missing: a high rate on one category and no attempts in another describes the red team, not the model.