Online Experiment

Split live traffic between the current model and the new one, and measure which does better on the thing you actually care about. It is the only measurement that is not a proxy — and it costs real traffic, real time, and exposes real users to whichever arm turns out to be worse.

Two mistakes account for most of the wasted experiments. The first is stopping when the result looks good, which turns a fixed-sample test into a procedure that finds significance eventually whatever is true. The second is measuring a proxy that is easy to move — clicks, session length — instead of the outcome that matters, and then shipping a model that moved the proxy in the wrong direction for the outcome. Both are decided before the experiment starts, which is when the thinking has to happen.

Sample size scales as the variance over the square of the effect you want to detect, so halving the detectable effect quadruples the traffic. That is the whole economics: a two-percent improvement is cheap to establish and a half-percent one may cost more traffic than the product has, which is why mature systems accumulate many small unmeasurable gains and cannot tell you which of them worked.