โ† Wiki Index / Computation & Algorithms / Transformer (Machine Learning)
๐Ÿ’ป Computation & Algorithms

Transformer (Machine Learning)

The transformer is a deep neural network architecture that relies on stacked self-attention mechanisms and position-wise feedforward networks to process sequential data in parallel without recurrent connections.

Transformer (Machine Learning) is a deep neural network architecture introduced by Vaswani and colleagues in 2017 that processes sequential and structured data through stacked self-attention mechanisms rather than recurrence or convolution. Prior to its introduction, sequence modeling in machine learning relied predominantly on architectures such as the Recurrent Neural Network, including long short-term memory networks and gated recurrent units, or specialized configurations of the Convolutional Neural Network. These recurrent models maintain an internal hidden state updated sequentially token by token, which enforces sequential computation during training and impedes parallelization across distributed compute clusters. The transformer architecture dispenses with recurrent transitions altogether, relying on Self-Attention to compute pairwise interactions across all positions in a sequence concurrently.

By parallelizing sequence processing across modern graphics processing units and tensor accelerators, the transformer established a scalable foundation for modern deep learning. The architecture underlies large language models, vision transformers, multimodal representation frameworks, and scientific modeling systems such as AlphaFold. As explored in the CRV Science Labs Neural Networks Synthesis Review, the transformer represents a paradigm shift where inductive biases regarding sequential locality are replaced by dynamic, data-dependent routing operating over continuous vector representations.

Architectural Foundations and Attention Mechanisms

Multi-Head Self-Attention Mechanics

The foundational building block of the transformer is the scaled dot-product attention mechanism. Given an input sequence transformed into a matrix of vector embeddings, the network projects each token representation into three continuous vector spaces: queries (Q), keys (K), and values (V). For an input matrix X with sequence length N and model dimension dmodel, linear projection matrices WQ, WK, and WV generate representations Q = X WQ, K = X WK, and V = X WV. The attention score between token i and token j is computed as the inner product of query qi and key kj, scaled by the square root of the key dimension dk:

Attention(Q, K, V) = softmax((Q KT) / โˆšdk) V

Scaling by 1/โˆšdk counteracts the growth of inner products in high-dimensional vector spaces, where large vector magnitudes push the softmax activation function into regions characterized by vanishingly small gradients. Rather than computing a single attention distribution, multi-head attention projects queries, keys, and values h times into lower-dimensional subspaces of dimension dk = dmodel / h. Each attention head computes distinct contextual relationships in parallel. One head may focus on immediate syntactic associations, while another captures long-range coreference links. The outputs of all h heads are concatenated and linearly projected via output matrix WO to restore the original dimensionality dmodel.

Feedforward Networks and Residual Normalization

Each transformer block couples multi-head attention with a position-wise feedforward network (FFN). The FFN operates identically and independently on each token representation:

FFN(x) = max(0, x W1 + b1) W2 + b2

The intermediate hidden dimension of the feedforward layer typically expands to 4 ร— dmodel before projecting back to dmodel, often replacing the rectified linear unit (ReLU) with smooth non-linearities such as Gaussian Error Linear Units (GELU) or gated linear units (SwiGLU).

To stabilize optimization throughout deep stacks containing dozens or hundreds of layers, transformer blocks incorporate identity shortcut connections adopted from the Residual Neural Network. Each sub-layer output is added to its input:

LayerOutput = LayerNorm(x + SubLayer(x))

In original post-layer-normalization (Post-LN) transformers, normalization was applied following the residual addition, which caused gradient magnitudes to vary sharply across layer depths. Modern implementations predominantly employ pre-layer-normalization (Pre-LN) or root-mean-square normalization (RMSNorm), where inputs are normalized prior to entering the multi-head attention and feedforward sub-layers. Pre-LN preserves an unimpeded residual stream that permits unhindered gradient propagation during Backpropagation and stable training with Gradient Descent.

Positional Representation and Sequence Processing

Overcoming the Lack of Inherent Recurrence

Unlike recurrent networks, standard dot-product self-attention possesses permutation equivariance: permuting the rows of an input matrix permutes the rows of the output matrix by the identical permutation. Because language, genetic sequences, and temporal time series depend on sequential ordering, the transformer must receive explicit positional information. Without positional information, a sentence like "the dog bit the cat" would produce identical unordered self-attention representations to "the cat bit the dog".

Sinusoidal, Learned, and Rotary Positional Encodings

In the original transformer architecture, positional information was injected by adding fixed sinusoidal encodings to the token embeddings before the first layer:

PE(pos, 2i) = sin(pos / 100002i / dmodel)

PE(pos, 2i+1) = cos(pos / 100002i / dmodel)

This formulation assigns each position a geometric progression of wave frequencies from 2ฯ€ to 10000 ร— 2ฯ€, enabling the network to learn relative positional relationships through linear transformations. Subsequent architectures explored learned absolute positional embeddings, where a distinct embedding vector is assigned to each sequential index up to a fixed context limit.

More recent architectures utilize relative positional encodings and Rotary Position Embedding (RoPE). Rather than adding vector offsets to token embeddings, RoPE applies a coordinate rotation to query and key vectors in the complex plane:

qm = Rm WQ xm

Because the inner product between rotated vectors depends on their relative distance m โˆ’ n, RoPE naturally encodes token separation and generalizes to context lengths beyond those observed during training.

Computational Complexity and Scaling Dynamics

Quadratic Memory and Compute Constraints

A primary operational constraint of standard self-attention is its computational complexity with respect to sequence length N. Computing the full attention score matrix Q KT requires O(N2 ร— dmodel) arithmetic operations and O(N2) memory allocation per attention head. For short sequences of a few hundred tokens, this overhead is modest. However, as context lengths scale to tens of thousands or millions of tokens, quadratic complexity becomes the principal hardware bottleneck.

To mitigate this constraint, linear attention approximations, sparse attention patterns, and hardware-aware exact attention algorithms have been developed. Notable among these is FlashAttention, which reorganizes attention computation into tiled blocks sized for on-chip static random-access memory (SRAM), eliminating redundant reads and writes to high-bandwidth memory (HBM) without changing the mathematical output.

Loss Surfaces and Overparameterized Optimization

Transformers exhibit benign scaling behavior described by empirical neural scaling laws. When parameter count, dataset size, and compute budgets increase in balanced proportions, cross-entropy loss on the underlying Loss Function decays smoothly as a power law over multiple orders of magnitude.

Deep transformer models routinely operate in heavily overparameterized regimes where parameter counts surpass training token counts. Rather than succumbing to catastrophic overfitting predicted by classical learning theory, transformers traverse the interpolation boundary and exhibit Double Descent. Within this regime, overparameterized models discover smooth interpolating surfaces across the underlying Neural Manifold. The combination of residual streams, normalization layers, and high-dimensional attention routing yields a loss landscape relatively free of pathological local minima, enabling consistent convergence under adaptive optimizers such as AdamW.

Applications Across Scientific Disciplines

The flexibility of self-attention has enabled the transformer to supersede domain-specific architectures across numerous scientific and engineering fields:

  • Natural Language Processing: Encoder-only models such as BERT learn bidirectional contextual representations for classification and extraction, while decoder-only autoregressive models such as GPT generate coherent long-form text by masking future tokens.
  • Computer Vision: The Vision Transformer (ViT) treats non-overlapping image patches as token embeddings, achieving state-of-the-art visual classification and generation without relying on convolutional sliding filters.
  • Structural Biology and Chemistry: In molecular biology, AlphaFold incorporates specialized transformer blocks termed the Evoformer, which process evolutionary multiple sequence alignments and spatial residue pair representations to predict three-dimensional protein structures with atomic accuracy.
  • Multimodal Scientific Discovery: Modern multi-agent systems and foundation models unify proteomic sequences, astronomical spectroscopy, and climate sensor grids into shared latent vector spaces, as detailed in the CRV Science Labs Neural Networks Synthesis Review.

References

  1. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017). Attention Is All You Need. Advances in Neural Information Processing Systems, 30. https://arxiv.org/abs/1706.03762
  2. Bahdanau, D., Cho, K., & Bengio, Y. (2015). Neural Machine Translation by Jointly Learning to Align and Translate. International Conference on Learning Representations. https://arxiv.org/abs/1409.0473
  3. He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep Residual Learning for Image Recognition. IEEE Conference on Computer Vision and Pattern Recognition, 770โ€“778. https://doi.org/10.1109/cvpr.2016.90
  4. Belkin, M., Hsu, D., Ma, S., & Mandal, S. (2019). Reconciling modern machine-learning practice and the classical biasโ€“variance trade-off. Proceedings of the National Academy of Sciences, 116(32), 15849โ€“15854. https://doi.org/10.1073/pnas.1903070116
  5. LeCun, Y., Bengio, Y., & Hinton, G. (2015). Deep learning. Nature, 521(7553), 436โ€“444. https://doi.org/10.1038/nature14539
๐Ÿงฌ Support Open Science
Explore interactive science games on LowKey Games or browse scientific apparel & accessories in the CRV Science Shop.
๐Ÿ’ก Spot an inaccuracy or have primary literature to expand this article?
Submit a citation or revision note using your shared CRV Science / LowKey Games profile ID.
Suggest Revision