What is the attention mechanism in neural networks?

The attention mechanism allows a model to focus on the most relevant parts of the input when producing each output token, rather than compressing the entire input into a fixed-size vector. Given a query and a set of key-value pairs, attention computes a weighted sum of values, where weights are determined by the similarity between the query and each key. Self-attention computes attention between elements in the same sequence, enabling each position to attend to all other positions. Attention is the core building block of the Transformer architecture.