Low-Rank Factorization

Replace a big weight matrix with the product of two thin ones. If the original matrix is close to low rank, almost nothing is lost and the parameter count falls sharply. Whether that condition holds is an empirical question about the layer, and the answer differs between attention projections and feed-forward matrices.

This is the same machinery as LoRA used for a different purpose: LoRA adds a low-rank update to a frozen matrix, factorisation replaces the matrix outright. Compression by factorisation is only a speedup when the two thin matrices together do less work than the original, which needs rank below roughly half the smaller dimension — above that it costs more than it saves.

Approximate W ∈ ℝ super ^m×n  by AB with A ∈ ℝ super ^m×r , B ∈ ℝ super ^r×n , cost falling from mn to r(m+n). The optimal such approximation is the truncated SVD, with error given by the discarded singular values — σ sub _r+1  in spectral norm, the root-sum-square of the tail in Frobenius — so how well it works is read directly off the spectrum, and a flat spectrum means it cannot work at all.