Vector Quantization

Instead of a continuous latent, keep a fixed dictionary of vectors and replace each encoding with whichever entry is nearest. The representation becomes a list of integers — which is what lets an image be modelled the way a sentence is, by predicting the next index.

A codebook of K entries, nearest-neighbour lookup in latent space, and a straight-through estimator because the lookup has no gradient of its own. The characteristic failure is codebook collapse: a few entries take all the traffic while the rest are never selected and therefore never updated, so the effective vocabulary is far smaller than K. Commitment losses, EMA updates and dead-entry restarts all exist for that one problem.

z sub _q  is the codebook entry minimising ‖z sub _e  − e sub _k ‖, with the gradient copied straight through from z sub _q  back to z sub _e . The loss adds ‖sg[z sub _e ] − e‖² to pull entries toward their assignments and β‖z sub _e  − sg[e]‖² to stop the encoder outrunning them. The size that matters is the perplexity of the usage distribution rather than K itself.