Competitive-to-cooperative attention dynamics
Produced by Incremental Learning of Sparse Attention Patterns in Transformers
Sparse attention pattern
An attention structure in which substantial interaction weight is concentrated on a restricted subset of token positions.
An attention structure in which substantial interaction weight is concentrated on a restricted subset of token positions.
The selected positions may be fixed by architecture or learned from the statistical structure of the task.
Connections
Characterized by
Forward invariance of the dominant attention pattern
Produced by Incremental Learning of Sparse Attention Patterns in Transformers
Stage-wise convergence guarantee for sparse-attention learning
Produced by Incremental Learning of Sparse Attention Patterns in Transformers