Label-permutation invariance under gradient flow
Produced by Induction Heads Interpolate N-Grams
Disentangled Transformer
An attention-only Transformer variant that removes multilayer perceptrons and replaces additive residual connections with concatenation so that successive computations remain explicit.
An attention-only Transformer variant that removes multilayer perceptrons and replaces additive residual connections with concatenation so that successive computations remain explicit.
The architecture is designed as a more directly analyzable representation of the computations available to standard attention-only Transformers.