โ† Wiki Index / Computation & Algorithms / Activation Function
๐Ÿ’ป Computation & Algorithms

Activation Function

An activation function is a mathematical transformation applied to the net input of an artificial neuron that introduces non-linearity into a neural network, enabling the system to approximate complex non-linear functions.

An activation function is a mathematical operator applied to the net input of an artificial neuron that transforms aggregated affine sums into output activations, governing the representation capacity and gradient dynamics of deep learning models. By interposing non-linear operations between successive linear matrix transformations, activation functions enable multilayer networks to escape the algebraic constraints of linear composition. Without non-linear activations, an artificial neural network of arbitrary depth would collapse algebraically into a single linear map, incapable of modeling complex geometric decision manifolds.

The mathematical selection and historical evolution of neural non-linearities are analyzed in the CRV Science Labs Neural Networks Synthesis Review. Modern connectionist models employ diverse activation familiesโ€”ranging from historical step and sigmoidal functions to rectified linear operators and continuous gating mechanismsโ€”each introducing distinct inductive biases, numerical stability characteristics, and optimization trade-offs during network training via backpropagation.

Functional Role and Non-Linearity in Deep Learning

The primary theoretical role of an activation function is to introduce curvature into the coordinate transformations executed across layered architectures.

The Necessity of Non-Linear Transformations

In a feedforward network where each layer computes an affine mapping z[l] = W[l]a[lโˆ’1] + b[l], setting the activation function to the identity map a = f(z) = z reduces the composite forward evaluation of an L-layer network to:

a[L] = W[L](W[Lโˆ’1] ยทยทยท (W[1]x + b[1]) ยทยทยท + b[Lโˆ’1]) + b[L]

Because the product of multiple matrices remains a matrix, there exists a single matrix Weff = โˆl=1L W[l] and bias vector beff such that a[L] = Weffx + beff. Consequently, a deep linear network possesses no greater expressive power than a single-layer linear model. Introducing a scalar non-linear activation ฯƒ between layers breaks this linear closure, permitting each layer to bend, fold, and re-orient the representation space.

Universal Approximation and Representation Capacity

The presence of non-linear activations underpins the Universal Approximation Theorem. This foundational result establishes that a feedforward network with a single hidden layer containing a finite number of non-linear neurons can approximate any continuous function defined over compact Euclidean subsets to arbitrary precision.

Initially proved for continuous sigmoidal curves, the universality property holds for virtually any continuous, non-polynomial activation function. However, the geometric form of the non-linearity governs how efficiently a network constructs functional approximations: while smooth sigmoidal functions partition space through superpositions of soft half-spaces, piecewise linear functions divide the input domain into polyhedral regions, conferring differing inductive biases on generalization.

Classical Saturating Activation Functions

Early connectionist systems adopted saturating activation functions designed to mimic biological threshold firing behaviors and provide squashing transformations into bounded intervals.

The Heaviside Step Function and Threshold Logic

The earliest mathematical artificial neuron, introduced in the McCulloch-Pitts neuron and Frank Rosenblatt's perceptron, utilized the Heaviside step function:

f(z) = 1 if z โ‰ฅ 0, and f(z) = 0 if z < 0

The Heaviside step maps real numbers to binary states {0, 1}. While well-suited for discrete boolean logic, the step function exhibits zero derivative at all non-zero values and is discontinuous at the origin. Because the derivative vanishes almost everywhere, error gradients cannot propagate backward through hidden layers via the chain rule, preventing gradient-based parameter calibration in multilayer configurations.

Logistic Sigmoid and Hyperbolic Tangent Formulations

To enable gradient descent optimization, researchers replaced the Heaviside step with smooth, differentiable S-shaped curves. The standard logistic sigmoid function squashes inputs into the open interval (0, 1):

ฯƒ(z) = 1 / (1 + exp(โˆ’z))

Its first derivative is expressed neatly in terms of the function itself: ฯƒโ€ฒ(z) = ฯƒ(z)(1 โˆ’ ฯƒ(z)), attaining a maximum slope of 0.25 at z = 0. Closely related is the hyperbolic tangent function (tanh), which maps inputs into the symmetric range (โˆ’1, 1):

tanh(z) = (exp(z) โˆ’ exp(โˆ’z)) / (exp(z) + exp(โˆ’z))

The tanh function offers the advantage of zero-centered outputs, which prevents systematic directional bias in weight updates across successive layers. However, both sigmoid and tanh suffer from severe gradient saturation: when |z| grows moderately large, the derivative approaches zero, causing backpropagated error signals to attenuate and impeding learning in deep networks.

Modern Non-Saturating and Rectified Activations

The breakthrough enabling the optimization of very deep architectures came with the transition from saturating sigmoidal curves to non-saturating rectified linear functions.

Rectified Linear Units (ReLU) and Biological Plausibility

The rectified linear unit (ReLU), popularized by Vinod Nair and Geoffrey Hinton, evaluates the positive part of its argument:

f(z) = max(0, z)

For positive inputs (z > 0), the derivative is unity (fโ€ฒ(z) = 1), preventing gradient saturation regardless of activation magnitude. For negative inputs (z < 0), the activation and derivative are zero, creating sparse representations where only an active subset of neurons fires for any given input pattern. This sparse coding mirrors energy-efficient computational dynamics observed in biological cortical networks.

Furthermore, computing ReLU requires only a simple threshold comparison without evaluating transcendental exp(z) operations, substantially reducing computational overhead on modern hardware.

Leaky ReLU, Parametric ReLU, and Smooth Variants

Despite its advantages, standard ReLU can cause neurons to become permanently inactive during training if their parameters receive updates that keep z < 0 across all training samplesโ€”a condition termed the dying ReLU problem. To address this issue, researchers introduced variants with non-zero negative slopes.

Leaky ReLU assigns a small, fixed positive slope ฮฑ (such as ฮฑ = 0.01) to negative inputs: f(z) = z if z โ‰ฅ 0, and f(z) = ฮฑz if z < 0. Parametric ReLU (PReLU) treats ฮฑ as a learnable parameter updated alongside network weights. Smooth approximations, such as the Exponential Linear Unit (ELU) and Gaussian Error Linear Unit (GELU)โ€”which weighs inputs by the cumulative distribution function of the standard normal distributionโ€”provide smooth curvature around zero, enhancing representation learning in advanced transformer architectures.

Specialized, Adaptive, and Output Normalization Functions

Activation functions also serve specialized architectural roles at network boundaries and within dynamic gating pathways.

Softmax and Multiclass Probabilistic Categorization

At the output stage of categorical classification networks, raw unnormalized pre-activations, termed logits, must be converted into a valid probability distribution over K mutually disjoint classes. The softmax function normalizes logits through exponentiation:

ฯƒ(z)i = exp(zi) / (โˆ‘j=1K exp(zj))

Softmax ensures that each output value lies strictly within the interval (0, 1) and that all class probabilities sum to unity. Pairing softmax outputs with a categorical cross-entropy loss function simplifies gradient calculation during gradient descent, as the composite derivative yields a clean linear error term: โˆ‚J / โˆ‚zi = pi โˆ’ yi.

Self-Gated and Learnable Activation Dynamics

Beyond static scalar operators, modern architectures explore self-gated and data-dependent activation functions. The Swish activation, defined as f(z) = z ยท ฯƒ(ฮฒz) (where ฯƒ is the logistic sigmoid and ฮฒ is a constant or learnable parameter), provides a smooth non-monotonic curve with negative curvature for small negative values.

Similarly, gated linear units (GLUs) split hidden state vectors into two parallel branches, using the output of one branch to element-wise gate the linear activation of the second. By integrating continuous multiplicative gating directly into intermediate activation stages, modern neural networks dynamically route information and calibrate representation depth on a sample-by-sample basis.

References

  1. Nair, V., & Hinton, G. E. (2010). Rectified linear units improve restricted Boltzmann machines. Proceedings of the 27th International Conference on Machine Learning (ICML), 807โ€“814. https://dl.acm.org/doi/10.5555/3104322.3104425
  2. Mohamed, A. R., & Hinton, G. (2010). Phone recognition using Restricted Boltzmann Machines. ICASSP, 5066โ€“5069. https://doi.org/10.1109/icassp.2010.5495651
  3. Cybenko, G. (1989). Approximation by superpositions of a sigmoidal function. Mathematics of Control, Signals and Systems, 2(4), 303โ€“314. https://doi.org/10.1007/BF02551274
  4. LeCun, Y., Bengio, Y., & Hinton, G. (2015). Deep learning. Nature, 521(7553), 436โ€“444. https://doi.org/10.1038/nature14539
๐Ÿงฌ Support Open Science
Explore interactive science games on LowKey Games or browse scientific apparel & accessories in the CRV Science Shop.
๐Ÿ’ก Spot an inaccuracy or have primary literature to expand this article?
Submit a citation or revision note using your shared CRV Science / LowKey Games profile ID.
Suggest Revision