Inference / Adaptive
verifiedEarly Exit
Stop partway up the stack when the answer is already clear. Attach a small classifier to intermediate layers, and if one of them is confident enough, return from there rather than running the remaining layers. Easy inputs leave early; hard ones go all the way.
Training the exits is the part that decides whether it works. Train them jointly and they compete with the main objective; train them afterwards on a frozen backbone and they are honest but weaker. Either way the confidence they report is the thing being thresholded, so a poorly calibrated exit head does not merely lose accuracy — it loses it invisibly, by leaving early on inputs it had no business being confident about.
The saving is the layers skipped and the risk is that the skipped layers would have changed the answer. Those two are set by one threshold, so this is Selective Prediction's trade with depth in place of abstention: raise the bar and fewer inputs exit early, lower it and more do while the ones that should not exit start to. There is no setting that gets the saving without the risk, and the useful report is the pair rather than the speedup.
436 of 1000 flagged. 75% of them were right and 107 were false alarms; 82% of what should have been caught was, leaving 71 missed.
Inputs an early exit would answer correctly against ones it would get wrong, by the exit head's confidence. Drag the threshold up to watch the wrong exits fall away and the saving fall with them — one dial sets both, which is why a speedup alone says nothing.
Reviewed by opendroid · 2026-08-18
- arXiv:2004.12993 — DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference
- arXiv:1709.01686 — BranchyNet: Fast Inference via Early Exiting from Deep Neural Networks