Training-Serving Skew

A feature computed one way when the model was trained and another way when it is served. The model is fine, the data is fine, and the predictions are quietly wrong — because the number arriving at inference does not mean what the number in training meant.

It happens because the two paths are usually written twice, in different languages, by different people, months apart. A batch job computes a seven-day average over a table; a serving path computes it over a live stream with a different window boundary and a different null policy. Neither is a bug on its own. The fix that works is not care but sharing — one definition, executed by both paths, which is the argument a feature store makes.

It is invisible to every check that looks at one side. Training metrics are computed on training features and look correct; serving latency and error rates look correct; only a comparison of the two distributions catches it, and that comparison is a thing someone has to build. So the expected time to detection is set by whether anyone is diffing the two, and where nobody is, the answer is bounded by how long until a person notices the product is worse.