Akaike Information Criterion

The Akaike Information Criterion (AIC) is an asymptotic, information-theoretic estimator of relative out-of-sample prediction error used to compare and rank competing statistical models fitted to the same empirical dataset. Formulated by the Japanese statistician Hirotugu Akaike in a landmark 1971 symposium paper and published in IEEE Transactions on Automatic Control in 1974 (originally designated "An Information Criterion"), the AIC bridges Ronald Fisher's method of maximum likelihood estimation with Solomon Kullback and Richard Leibler's information divergence (D_KL). For a statistical model with k independently estimated parameters and maximized likelihood value L̂, the Akaike Information Criterion is defined as:

Within applied statistics, econometrics, ecology, and computational biology—most notably when selecting nucleotide, amino acid, and codon substitution models in molecular evolution evaluated via the Felsenstein pruning algorithm—AIC operationalizes Occam's razor quantitatively. Because adding extra free parameters (k) to a family of models always increases or maintains the in-sample maximized log-likelihood ln(L̂) even when those parameters fit random noise (overfitting), AIC penalizes each additional parameter by +2 log-likelihood units, selecting the candidate model that minimizes AIC and thereby minimizes expected information loss when predicting future independent observations.

Expected Kullback–Leibler Information Loss D_KL(f ‖ g_θ)

Suppose an empirical sample of n observations y = (y₁, ..., y_n) is generated by an unknown, high-dimensional true data-generating probability distribution f(y), and an investigator approximates f(y) using a candidate parametric model family g(y | θ) indexed by a k-dimensional parameter vector θ ∈ Θ ⊂ R^k. In information theory, the information lost when approximating the true distribution f with the model g(· | θ) is measured by the Kullback–Leibler (KL) divergence (relative Shannon entropy):

Because the first expectation E_f[ln f(Z)] depends solely on the unknown truth f and is an identical constant across all candidate models g₁, g₂, ..., g_R, minimizing the Kullback–Leibler divergence D_KL(f ‖ g(· | θ)) is mathematically equivalent to maximizing the expected out-of-sample log-likelihood E_f[ ln g(Z | θ) ] evaluated on a new independent validation sample Z ~ f.

Why In-Sample Log-Likelihood Overfits by k Units (Akaike's Theorem)

In practice, neither the truth f nor the true parameter vector θ₀ is known; instead, the investigator estimates the maximum likelihood estimator (MLE) θ̂(y) by maximizing the in-sample log-likelihood ℓ(θ | y) = Σ_i=1ⁿ ln g(y_i | θ) on the observed dataset y. Because the same data y are used both to estimate θ̂(y) and to evaluate ln L̂ = ℓ(θ̂(y) | y), the in-sample maximized log-likelihood is an upwardly biased (optimistic) estimator of the expected out-of-sample predictive log-likelihood E_YE_Z[ℓ(θ̂(Y) | Z)].

Expanding ℓ(θ) in a second-order Taylor series around the KL-minimizing parameter θ₀ using the Fisher information matrix I(θ₀) and the observed Hessian matrix J(θ₀), Hirotugu Akaike (1974) proved that under standard regularity conditions as n → ∞ (for a correctly specified or nearly well-specified model where Tr(I J⁻¹) → k), the asymptotic optimism bias equals the parameter dimension k:

Subtracting this asymptotic bias k from ln L̂ gives the unbiased estimator of expected relative KL log-likelihood (ln L̂ − k); multiplying by −2 (on the historical deviance scale so that differences match χ² likelihood-ratio statistics) yields AIC = −2 ln L̂ + 2*k*. For ordinary least-squares (OLS) regression with n observations and normally distributed errors of residual sum of squares RSS = Σ(y_i − ŷ_i)² (where σ̂²_MLE = RSS / n), the AIC reduces (up to an additive sample-size constant) to AIC = n ln(RSS / n) + 2*k* (where k includes the intercept, slopes, and error variance σ²).

Hurvich and Tsai's Second-Order Corrected Criterion (AIC_c)

When the sample size n is small relative to the number of estimated parameters k (specifically when the ratio n / k < 40), the first-order asymptotic bias approximation k underestimates the true overfitting penalty, causing standard AIC to select overparameterized models. Building on Nariaki Sugiura (1978), Clifford Hurvich and Chih-Ling Tsai (1989) derived the exact finite-sample bias for linear regression and autoregressive models, yielding the second-order bias-corrected Akaike Information Criterion (AIC_c):

Because the extra penalty term 2*k*(k + 1)/(n − k − 1) vanishes automatically as n → ∞ (so AIC_c → AIC in large samples), Kenneth Burnham and David Anderson recommend using AIC_c routinely across empirical studies.

Relative Differences (Δ_i), Akaike Weights (w_i), and Multi-Model Inference

Because the unknown truth constant E_f[ln f(Z)] is omitted from AIC, individual raw AIC values (which can be positive or negative) carry no absolute meaning on their own; only differences in AIC (Δ_i) between candidate models fitted to the exact same response data y are interpretable. Given a set of R candidate models with scores AIC₁, ..., AIC_R and minimum score AIC_min = min_r AIC_r:

By Burnham and Anderson's empirical calibration rules of thumb:

• Δ_i = 0: The top-ranked (KL-best) model in the candidate set.
• 0 ≤ Δ_i ≤ 2: Substantial empirical support; models within 2 AIC units of AIC_min are statistically competitive.
• 4 ≤ Δ_i ≤ 7: Considerably less empirical support.
• Δ_i > 10: Essentially zero empirical support relative to the top model.

Transforming Δ_i back onto the relative likelihood scale via the exponential Boltzmann-like factor L(M_i | y) ∝ exp( −(1/2) Δ_i ) and normalizing across all R candidate models yields the Akaike weights (w_i):

where Σ_i=1^R w_i = 1. The Akaike weight w_i represents the weight of evidence (approximate probability) that model M_i is the Kullback–Leibler best model among the R candidates, and the evidence ratio between any two models is w_i / w_j = exp( (1/2)(Δ_j − Δ_i) ). Rather than relying on a single winner-take-all model when several models have Δ_i < 4, investigators compute model-averaged parameter estimates θ̂̄ = Σ_i=1^R w_i θ̂_i across the entire model ensemble.

Asymptotic Efficiency (AIC / Cross-Validation) vs. Consistency (Schwarz BIC)

A recurring theoretical comparison in statistics contrasts AIC (2*k* − 2 ln L̂) with Gideon Schwarz's (1978) Bayesian Information Criterion (BIC = k ln(n) − 2 ln L̂). Although their formulas appear algebraically similar, they optimize fundamentally different mathematical objectives:

1. BIC (Bayes Factor Approximation & Consistency): Derived as a Laplace asymptotic approximation to −2 ln P(y | M_i) under uniform model priors, BIC replaces the constant penalty 2 with ln(n) (which exceeds 2 for any n ≥ 8; e.g., ln(1,000) ≈ 6.91). If the true data-generating model f has a finite dimension k₀ and is explicitly included in the candidate set, BIC is model-selection consistent (selecting k₀ with probability → 1 as n → ∞), whereas AIC maintains a non-zero asymptotic probability (~15.7% for 1 extra parameter) of selecting a slightly larger model.
2. AIC (Minimax Prediction Efficiency & Leave-One-Out Cross-Validation): As proved by Mervyn Stone (1977), selecting models by AIC is asymptotically equivalent to leave-one-out cross-validation (LOOCV). Furthermore, Yuhong Yang (2005) proved that when the true biological or physical process f is infinite-dimensional (tapering truths where no finite candidate model is literally exact), AIC is minimax-rate optimal (asymptotically efficient) for minimizing out-of-sample squared prediction error.

Substitution Model Selection in Phylogenetics (jModelTest, ModelFinder, PAML)

Unlike classical Likelihood Ratio Tests (LRTs) (where 2Δℓ ~ χ²_df requires the competing models to be strictly nested subsets of one another), AIC and BIC can directly compare non-nested models alongside nested ones in a single pass. In molecular evolution and cladistics, automated model-selection tools such as Modeltest / jModelTest (David Posada and Keith Crandall) and ModelFinder in IQ-TREE evaluate 88 to 286 candidate nucleotide substitution models (from JC69, K80, and HKY85 to GTR+F+I+Γ₄ and FreeRate +R models) and 61 × 61 codon substitution models for positive selection, using AIC, AIC_c (with alignment length or site/branch count as n), and BIC to select the substitution matrix that avoids both under-parameterization bias (long-branch attraction) and over-parameterization variance.

Can I compare AIC values between two models that use different transformations of the dependent variable (e.g., y vs. ln y) or different sample sizes?

No—not directly. Because the maximized log-likelihood ln(L̂) is the sum of log probability densities over the n data points, dropping even a single outlier observation (n − 1 vs. n) changes the scale of ln(L̂) and invalidates AIC comparison. Similarly, comparing a model fitted to raw y against a model fitted to transformed ln(y) (or √*y*) violates the shared baseline density f(y) unless you explicitly add the Jacobian determinant of the variable transformation (−2 Σ_i=1ⁿ ln |*d*(g(y_i))/dy_i|, which equals +2 Σ ln(y_i) for a log transformation) to the transformed model's AIC.

Why does adding one useless parameter to a model never increase its AIC by more than +2.0?

Suppose Model A has k parameters and Model B adds one extra parameter (k + 1) that has zero true effect in the population. Because Model A is nested inside Model B, maximizing the likelihood of Model B over a larger parameter space guarantees that ln(L̂_B) ≥ ln(L̂_A) (the in-sample fit can never get worse). Under the null hypothesis, twice the log-likelihood increase 2(ln L̂_B − ln L̂_A) follows a χ²₁ distribution with mean 1.0 (and minimum 0.0). Therefore, AIC_B − AIC_A = 2(1) − 2(ln L̂_B − ln L̂_A) ≤ +2.0. When two nested models differ by ΔAIC ≈ 1.5 to 2.0, the extra parameter in the larger model is almost certainly an uninformative pretender variable fitting noise.

When should I use AIC (or AIC_c) instead of BIC?

Use AIC (and AIC_c whenever n/k < 40) when your primary scientific goal is predictive accuracy and when you expect the underlying real-world system (such as an ecosystem, clinical cohort, or evolving protein sequence) to be highly complex with many small tapering effects not fully captured by any simple 3-parameter equation. Use BIC when n is large and your goal is structural system identification—isolating the few dominant causal variables with minimal false positives under the assumption that a simple true model exists in your candidate set.

1. Akaike, H. (1974). "A new look at the statistical model identification." IEEE Transactions on Automatic Control, 19(6), 716–723. https://doi.org/10.1109/TAC.1974.1100705
2. Stone, M. (1977). "An asymptotic equivalence of choice of model by cross-validation and Akaike's criterion." Journal of the Royal Statistical Society: Series B (Methodological), 39(1), 44–47. https://doi.org/10.1111/j.2517-6161.1977.tb01603.x
3. Hurvich, C. M., & Tsai, C.-L. (1989). "Regression and time series model selection in small samples." Biometrika, 76(2), 297–307. https://doi.org/10.1093/biomet/76.2.297
4. Burnham, K. P., & Anderson, D. R. (2004). "Multimodel inference: understanding AIC and BIC in model selection." Sociological Methods & Research, 33(2), 261–304. https://doi.org/10.1177/0049124104268644