Parameter-free Adaptive Sparse Attention via Compression-Based Content Selection

arXiv cs.LG5d4 min read

arXiv:2607.21752v1 Announce Type: new Abstract: Data-adaptive sparse attention masks substantially outperform fixed patterns (e.g., BigBird and Longformer) and can even exceed dense attention on long sequences. Existing adaptive approaches---including SBM-Transformer, Dynamic Mask Attention, and NSA---typically require additional learnable parameters, custom gradient estimators, or specialized CUDA kernels. We show that classical data compression provides an effective masking signal with \textbf{no additional parameters}. By computing per-block gzip compression ratios, we identify non-redundan

Read the full story at arXiv cs.LG →

Parameter-free Adaptive Sparse Attention via Compression-Based Content Selection

More in Machine Learning

Reducing Human Annotation with ML Active Learning

Learning What Matters: Supervising Sparse Attention Routing with Causal Evidence Sets

On the Depth Scalability of Logic Gate Networks

Pretraining EHR Foundation Models with Patient-Aware Sampling

From Perturbation Correction to Geometry-Aware Sampling: Sharpness-Guided Equilibrium Sampling for Balanced Flat Minima in Long-Tailed Learning

Cross-Domain Off-Policy Evaluation and Learning for Contextual Bandits