Ghost in the Kernel: In-Context Learning with Efficient Transformers via Domain Generalization
arXiv stat.ML1mo4 min read
arXiv:2607.00479v1 Announce Type: cross Abstract: Transformer-based large models have demonstrated remarkable generalization abilities across different tasks by leveraging a context-aware attention module for in-context learning. With richer context, transformers adapt more effectively to the current use case without any parameter updates. However, the quadratic computational and memory complexity with respect to context length significantly slows data processing in softmax transformers. Linear transformers were proposed to address this issue by reducing the complexity to linear dependence on
