Attention Sinks in LLMs: An In-Depth Analysis

Large Language Models (LLMs) often exhibit a peculiar behavior: they allocate a disproportionate amount of attention to specific tokens, a phenomenon known as 'attention sinks'. While these sinks are generally considered detrimental, a notable exception has been identified: the model's consistent emphasis on the first token of the input sequence.

A recent study analyzed the mechanisms underlying the formation of these 'attention sinks', focusing in particular on the first input token. The researchers identified a simple mechanism, referred to as the 'P0 Sink Circuit', which allows the model to recognize the token at position zero and induce an attention sink within two transformer blocks, without relying on semantic information.

The Role of the 'P0 Sink Circuit'

This mechanism serves as the basis for the attention sink on position zero. By analyzing training traces from a 30 billion parameter A3B MoE model trained from scratch, the researchers found that this mechanism emerges early in training and becomes increasingly concentrated in the first two layers. This suggests a possible signal for monitoring the convergence states of pre-training.

Understanding these internal mechanisms is crucial for optimizing the performance of LLMs and mitigating potential negative effects resulting from inefficient attention allocation.