Prompt Tuning

Instead of changing a model to fit a task, prompt tuning changes what you put in front of it — but learns those tokens by gradient descent rather than writing them. A handful of trained vectors are prepended to the input and everything else stays frozen. One base model serves every task, each carrying a few thousand parameters of its own.

Learn continuous embeddings prepended to the input sequence; the vectors live in embedding space and correspond to no real token. It closes on full fine-tuning as the base model grows, which is the result the paper is named for — cheap adaptation gets better, not worse, with scale. Prefix tuning is the more general form, prepending to every layer rather than the input alone.

Optimise P ∈ ℝ super ^{k×d sub _model}   for k soft tokens with all model weights frozen, so trainable parameters are k·d sub _model  against the full N. At k = 20 on a 4096-wide model that is 82k parameters, independent of depth — the saving grows with the model rather than shrinking.