Forward invariance of the dominant attention pattern
Produced by Incremental Learning of Sparse Attention Patterns in Transformers
Forward invariance
A property of a subset of a dynamical system’s state space such that every trajectory beginning in the subset remains there at all later times for which it is defined.
A property of a subset of a dynamical system's state space such that every trajectory beginning in the subset remains there at all later times for which it is defined.
Forward-invariant regions can encode qualitative facts about training trajectories, such as preserving the ordering between a dominant component and its competitors.