Inductive Bias

The assumptions a model makes before it sees any data. Every architecture prefers some explanations to others, and that preference is what lets it generalise from finitely many examples at all — a model with no preferences could fit the training set in endlessly many ways and would have no reason to choose the one that works on the next example.

The practical question is whether yours matches the problem, and the trade is unusually legible. A convolution assumes that position does not change meaning and that nearby pixels belong together, which is enormously right for images and lets it learn from far less data. A transformer assumes almost nothing of the kind, which makes it worse than a convolutional network on small image datasets and better on large ones. The assumption you did not build in has to be paid for in examples.

The trade is between how large the hypothesis space is and how much data is needed to pick within it. A stronger bias means a smaller space, so fewer examples pin the answer down — and a larger floor on the error if the truth is not in that space. The crossover with dataset size is therefore real rather than a matter of taste: below it the bias saves more than its floor costs, above it the floor dominates and the weaker assumption wins, and no amount of data moves a floor.