Alignment
14 concepts, 14 reviewed and merged.
Behaviour
Evaluation
- Dangerous Capability EvaluationBefore deploying a model, test whether it can do the specific things that would be harmful if it could — not whether…verified
- Model-Written EvaluationWriting evaluations by hand is slow, and the behaviours worth measuring keep multiplying.verified
- Safety Benchmark ValidityA benchmark labelled "safety" is only measuring safety if a model can score well on it without simply being more…verified
Foundations
Preference
- DPORLHF trains a reward model and then optimises against it with reinforcement learning — two models, two stages, and a…verified
- PPOPolicy gradient methods improve a policy by pushing up the probability of actions that scored well.verified
- Reward ModelNobody can write down a loss function for 'a helpful answer'.verified
- RLHFInstruction tuning teaches a model to imitate good answers.verified
Regimes
Research
- Deceptive AlignmentA model that behaves well while it is being tested and differently when it is not.verified
- Scalable OversightEvery method that trains a model on human judgement assumes a human who can judge.verified
- Weak-to-Strong GeneralizationFine-tune a strong model on labels produced by a weak one.verified