Context
8 concepts, 8 reviewed and merged.
Behaviour
Evaluation
Foundations
Methods
- Context CompressionIf the window is expensive and much of what goes into it is redundant, spend some compute shortening the context…verified
- Long-Context TrainingA model does not get a long context by having its positions rescaled; it gets one by then being trained on long…verified
- Position InterpolationRather than asking a model to handle positions it has never seen, squeeze the new positions into the range it already…verified