What a Weight Update Buys: Leverage Across Training Stages
arXiv cs.LG2d4 min read
arXiv:2610.37719v1 Announce Type: new Abstract: Post-training moves a language model's weights by about a thousandth of their norm and changes its behavior sharply; pretraining moves them by a third and buys a slow drift. We introduce leverage, an exact measure of what a unit of weight change buys: the Fisher quadratic form of an update, computed with one forward-mode pass per direction, divided by the same form for random directions of the same size at the same checkpoint. Leverage orders released Olmo-3, T\"ulu-3, Zephyr, and OLMo-2 checkpoints over four orders of magnitude: pretraining wind