Competitive-to-cooperative attention dynamics
A reduced dynamical model explains how attention heads move from competition over important positions to specialized sparse patterns in successive stages.
Incremental Learning of Sparse Attention Patterns in Transformers
2025-10-22
Incremental Learning of Sparse Attention Patterns in Transformers
October 2025
Research context
Open in graphAddresses inferred
Derived by ontology inference rules from authored assertions.
Produces
Research objectConvergence guarantee
Stage-wise convergence guarantee for sparse-attention learning
The simplified dynamical system provably converges through a sequence of increasingly expressive attention patterns under the stated assumptions.
Research objectForward invariance
Forward invariance of the dominant attention pattern
In the reduced gradient-flow system, once the statistically strongest feature and attention position lead their alternatives, the trajectory preserves that ordering through the corresponding competitive phase.
Versions
- International Conference on Machine Learning 2026 · canonical
- Workshop on Principles of Generative Modeling 2025