Forward invariance of the dominant attention pattern
Produced by Incremental Learning of Sparse Attention Patterns in Transformers
Invariance
A property under which a specified transformation or evolution preserves a mathematical object, relation, set, or system behavior.
A property under which a specified transformation or evolution preserves a mathematical object, relation, set, or system behavior.
An invariance is meaningful only relative to what is transformed and what is preserved. Different forms include invariance under dynamics, relabeling, changes of coordinates, or perturbations.
Connections
Has specialization
Label-permutation invariance under gradient flow
Produced by Induction Heads Interpolate N-Grams