Pretraining
Training performed before a model is adapted to a target task or used in its target setting, typically on broad data with a general or self-supervised objective.
Training performed before a model is adapted to a target task or used in its target setting, typically on broad data with a general or self-supervised objective.
For language models, next-token prediction is a standard pretraining objective. What a model learns during pretraining depends jointly on the data distribution, architecture, objective, and optimization process.