Weights
8 concepts, 8 reviewed and merged.
Foundations
- Merge InterferenceMerging two models degrades both a little.verified
- Merge vs EnsembleBoth take several models and combine them; the difference is when.verified
- Model MergingTake two models fine-tuned from the same base for different things, average their weights, and get one model that…verified
- Permutation SymmetrySwap two neurons in a hidden layer, and swap the corresponding weights that read from them, and the network computes…verified
Methods
- Fisher-Weighted MergingA plain average treats every weight as equally important to every model, which is obviously false — some weights a…verified
- Model SoupFine-tune the same base many times with different hyperparameters, then average the weights of the ones that came out…verified
- Task ArithmeticSubtract a base model's weights from its fine-tune and you get a task vector — a direction in weight space that means…verified
- Weight AveragingAverage the weights a single run passes through, rather than the weights of separate runs.verified