Self-Attention is a mathematical process where every word in a sequence "looks" at every other word to determine which ones provide the most relevant context for its own meaning. It has 3 main components, Query, Key and Value.
$$ \begin{aligned}\text{Query (Q)} &: \text{The specific information a token is "looking for"} \\\text{Key (K)} &: \text{The label a token offers to help others "find it"} \\\text{Value (V)} &: \text{The actual content/meaning a token "gives" once matched} \\\text{Attention} &: \text{The degree of relevance between a Query and a Key}\end{aligned} $$
To make this work, the model creates three distinct versions of every token vector using learned linear transformations. We use something called Scaled Dot-Product Attention.
$$ \text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V $$
$$ \begin{aligned} Q, K^T &: \text{Search Query \& Transposed Index Keys} \\ d_k &: \text{Scaling factor (prevents gradient vanishing)} \\ \text{softmax} &: \text{Score normalization (0 to 1 range)} \\ V &: \text{Value matrix (weighted content extracted)} \end{aligned} $$
Input: The vectors from the previous layer (Embeddings + Positional Encodings).
$$ The → [0.2, 0.5, 0.1, 0.8]\\ cat → [0.7, 0.1, 0.9, 0.2]\\ sat → [0.3, 0.8, 0.2, 0.4]\\ $$
The Process (Scaled Dot-Product Attention):
The Match ($Query \cdot Key$ ): We calculate the dot product of every Query with every Key. This produces a "Compatibility Score." High score = high relevance.
$$ Q_sat · K_The \\ Q_sat · K_cat \\ Q_sat · K_sat $$
The Scaling: We divide the scores by the square root of the dimension size to keep numbers stable.