Transformer

A family of neural sequence architectures built around attention-based interactions between positions.

A family of neural sequence architectures built around attention-based interactions between positions.

is aModel All entries Open in graph

Its components combine token representations, attention, nonlinear transformations, and residual computation.

Connections

has part

Induction head Self-attention

has specialization

Disentangled Transformer

related to

Sparse attention pattern

specialization of

Neural network

studied by

Mechanistic interpretability Training dynamics

Has applicable result

Model

Competitive-to-cooperative attention dynamics

Produced by Incremental Learning of Sparse Attention Patterns in Transformers

Related publications

studied by

Incremental Learning of Sparse Attention Patterns in Transformers

2026