Fora: From Weight-Space to Function-Space Protection in Capability-Preserving Fine-Tuning
arXiv cs.LG1mo4 min read
arXiv:2606.31092v2 Announce Type: new Abstract: Full fine-tuning adapts large language models to new tasks but can erode capabilities they already possess. Existing remedies protect through proxies such as parameter distances, importance penalties, output matching, or dominant singular directions of the weights, but none directly asks which activation directions the preserved capability relies on. We argue that a capability is characterized more faithfully by the activation subspace it induces than by the singular geometry of the weight matrix, and develop function-space protection, instantiat
