Induction head

An attention-head mechanism that uses earlier context matches to promote a corresponding continuation token.

An attention-head mechanism that uses earlier context matches to promote a corresponding continuation token.

is aResearch object All entries Open in graph

By retrieving earlier matching contexts and promoting their continuations, an induction head can implement in-context associative recall without updating model parameters.

Connections

enables

In-context learning

part of

Transformer

studied by

Mechanistic interpretability

Characterized by

Research object

Induction heads implement smoothed n-gram estimators

Produced by Induction Heads Interpolate N-Grams

Related publications

studied by

Induction Heads Interpolate N-Grams

2026