#Attention Is All You Need: The Genesis of the Transformer

Prior to the Transformer, attention was used primarily as an augmentation for RNNs, helping the model "focus" on specific parts of an input sequence during decoding. The Transformer elevated attention from a supporting component to the primary architectural primitive. By eliminating sequential processing, the architecture allowed for unprecedented parallelization during training, enabling models to ingest massive datasets with a constant computational depth. This shift effectively solved the "vanishing gradient" problem inherent in long RNN sequences, as every token in a Transformer has a direct path to every other token, regardless of their distance in the sequence.

At the core of the Transformer is the Scaled Dot-Product Attention mechanism. It computes a relationship between a Query ($Q$), a Key ($K$), and a Value ($V$) by calculating the dot product of $Q$ and $K$, scaling the result, and applying a softmax function to weight the values in $V$. The critical innovation here is the scaling factor of $1/\sqrt{d_k}$.

The authors observed that as the dimensionality of the keys ($d_k$) increases, the magnitude of the dot products grows, pushing the softmax function into regions where the gradient is extremely small. By dividing the dot product by the square root of the dimension, the model preserves a unit variance, ensuring that gradients remain stable during backpropagation. This mathematical stabilizer is what allows Transformers to scale to the massive hidden dimensions seen in modern architectures.

Rather than performing a single attention operation across the entire hidden dimension, the Transformer splits the model's representation into multiple "heads." Each head performs an independent attention operation in a unique subspace. This allows the model to "jointly attend" to information from different perspectives simultaneously. One head might learn to identify syntactic relationships (e.g., subject-verb agreement), while another focuses on semantic resolution (e.g., pronoun antecedents). By concatenating the outputs of these heads and projecting them back into the model dimension, the Transformer achieves a high-density reasoning capability that a single attention head could not replicate.

The Transformer architecture is composed of an Encoder and a Decoder, each consisting of a stack of $N=6$ identical layers. The Encoder generates a continuous representation of the input, while the Decoder utilizes that representation to generate an output sequence one token at a time. Every sub-layer within these blocks - whether it is an attention mechanism or a feed-forward network - is wrapped in a residual connection followed by Layer Normalization. This "Add & Norm" pattern is vital for deep scaling, as it allows gradients to flow through the network without degradation, maintaining the structural integrity of the representations as they pass through dozens of layers.

Each layer in the Transformer stack contains a fully connected Feed-Forward Network (FFN) that is applied to each position separately and identically. This FFN consists of two linear transformations with a ReLU activation in between. While the attention mechanism is responsible for moving information between positions, the FFN is where the "heavy lifting" of data transformation occurs at each individual position. In the original model, the inner dimension of the FFN was four times larger than the model dimension ($2048$ vs $512$), providing the necessary computational capacity for the model to refine and process the relational data captured by the attention heads.

Because the Transformer contains no recurrence or convolution, it is inherently permutation-invariant - it treats the input as a "bag of tokens" with no sense of order. To inject positional information, the authors introduced Sinusoidal Positional Encodings. These are added to the input embeddings and use a series of sine and cosine functions of different frequencies to encode the absolute position of each token. The choice of sinusoids was deliberate: the authors hypothesized that it would allow the model to learn to attend by relative positions, as any fixed offset can be represented as a linear function of the current position. This also allows the model to theoretically generalize to sequence lengths longer than those encountered during training.

The training of the Transformer employs a technique called Teacher Forcing, where the Decoder is provided with the ground-truth previous tokens to predict the next one. To prevent the model from "cheating" by looking ahead at the target it is trying to predict, the Decoder uses Masked Multi-Head Attention. This masking sets the attention scores for future positions to $-\infty$ before the softmax step, ensuring that the prediction for a specific token can only depend on the tokens that preceded it. This causal constraint is what allows Transformers to function as effective autoregressive generators.

The Transformer did more than just improve translation scores; it provided the blueprint for the unification of machine learning. The same architectural primitives now power vision models (ViT), audio systems, and even robotics. By proving that self-attention is a universal sequence processor, the paper set the stage for the convergence of AI research, where the focus shifted from hand-engineered architectural priors to the pure scaling of attention-based blocks.