Code-Switching

People switch languages mid-sentence, and they do it by rules rather than at random. Most multilingual systems are built as though each input has one language — a language identifier at the front, a tokenizer tuned for it — and that assumption breaks on a great deal of ordinary speech and writing.

The failure is architectural rather than statistical: a pipeline that identifies a language and then routes has already lost, because there is no single answer to route on. Models trained on monolingual data handle switching poorly and there is little natural code-switched text to train on, so the data problem and the modelling problem reinforce each other.

Switching happens at syntactically licensed points rather than uniformly, so a model that treats it as noise learns the wrong distribution. Evaluation needs matrix-language and embedded-language accuracy reported separately, since an average hides that a system may handle the dominant language and drop the other entirely.