Incomplete Lineage Sorting

Incomplete lineage sorting (ILS)—also termed deep coalescence or ancestral polymorphism retention—is a stochastic phenomenon in molecular phylogenetics and population genetics in which an ancestral genetic polymorphism persists across two or more successive speciation events, such that the genealogical coalescence of sampled alleles occurs deeper in time than the ancestral species split, frequently producing a gene tree topology that is discordant with the underlying species tree topology. First analyzed mathematically by Fumio Tajima (1983), Pekka Pamilo and Masatoshi Nei (1988), and unified under the Multispecies Coalescent (MSC) framework by Wayne Maddison (1997) and Bruce Rannala and Ziheng Yang (2003), incomplete lineage sorting explains why different genomic loci within the same set of organisms routinely yield conflicting phylogenetic trees.

When an ancestral species with a large effective population size (N_e) undergoes rapid cladogenesis—where the internal time interval (t generations) between consecutive speciation events is short relative to the average coalescent time of 2*N*_e generations—lineages sampled from sister species often fail to coalesce inside their immediate common ancestor. Entering the deeper ancestral population as three or more distinct lineages, they coalesce at random with equal probability, generating symmetrical phylogenetic discordance and hemiplasy (the false appearance of convergent evolution or reversal on a species tree caused by a single mutation on a discordant gene tree, coined by John Avise and Terence Robinson in 2008).

🧬 Interactive Phylogenetics Simulation: Explore cladograms, character-state evolution, and phylogenetic tree reconstruction in PhyloClade.

Mathematical Derivation for a Rooted Three-Taxon Species Tree

Consider a rooted three-species phylogeny with true species tree topology ((A, B), C), where sister species A and B diverged at time T₁ in the past, and their common ancestral population AB diverged from outgroup species C at an older time T₂ > T₁. Let t = T₂ − T₁ denote the duration (in generations) of the internal ancestral branch separating the two speciation events, and let N_e denote the diploid effective population size of that ancestral AB population. In coalescent theory, time along the internal branch is measured in coalescent units (τ):

τ = t / (2 N_e) (for diploid autosomal loci; or t / [½ N_e] for haploid cytoplasmic / mitochondrial loci)

Suppose we sample one haploid chromosome (allele) from each of the three extant species (a ∈ A, b ∈ B, and c ∈ C) and trace their genealogical lineages backward in time using Kingman's coalescent:

1. Coalescence Within the Ancestral AB Branch (Concordant Lineage Sorting): Entering the ancestral population AB at time T₁, lineages a and b can coalesce with each other following an exponential distribution with hazard rate 1 per coalescent unit (1 / [2*N*_e] per generation) before reaching the deeper node T₂. The probability that a and b coalesce during interval τ is 1 − exp(−*τ*). Whenever this occurs, the merged ab ancestor later coalesces with lineage c above T₂, producing the concordant gene tree ((a, b), c).
2. Failure to Coalesce Before T₂ (Deep Coalescence): With exponential survival probability exp(−*τ*) = exp[ − t / (2*N*_e) ], lineages a and b fail to coalesce inside the internal branch AB. Consequently, all three distinct lineages (a, b, and c) enter the deep common ancestral population ABC above T₂ simultaneously. Because all three lineages are exchangeable under neutral panmixia above T₂, any pair of the three lineages—(a, b), (a, c), or (b, c)—is equally likely to coalesce first, with exact probability ⅓ each.

Combining both scenarios, Pamilo and Nei (1988) derived the exact probabilities for the three possible rooted gene tree topologies given internal branch length τ = t / (2*N*_e):

• Concordant Gene Tree ((a, b), c):

*P*[ ((a, b), c) ] = [ 1 − exp(−*τ*) ] + (⅓) exp(−*τ*) = 1 − (⅔) exp(−*τ*)

• First Discordant Gene Tree ((a, c), b):

*P*[ ((a, c), b) ] = (⅓) exp(−*τ*)

• Second Discordant Gene Tree ((b, c), a):

*P*[ ((b, c), a) ] = (⅓) exp(−*τ*)

Two immediate quantitative rules emerge from these equations. First, for a three-taxon rooted tree under pure ILS, the concordant gene tree ((a, b), c) is always strictly more probable than either discordant tree (1 − (⅔)e^−*τ* > (⅓)e^−*τ* for all τ > 0). Second, the two discordant gene trees ((a, c), b) and ((b, c), a) occur at identical expected frequencies, a symmetry principle that forms the statistical null hypothesis for detecting post-speciation hybridization.

Degnan and Rosenberg's (2006) "Anomalous Gene Trees"

Because for three species the concordant gene tree is always the single most frequent topology, early phylogenomics assumed that taking a democratic "majority vote" across thousands of independent loci (or concatenating all gene alignments into a single supermatrix) would always converge on the true species tree as the number of sequenced genes approached infinity. In 2006, James H. Degnan and Noah A. Rosenberg overturned this assumption by discovering the anomaly zone for species trees containing four or more taxa (n ≥ 4).

Consider an asymmetric ("pectinate" or caterpillar) four-taxon species tree (((A, B), C), D) with two consecutive short internal branches τ₁ (ancestral to AB) and τ₂ (ancestral to ABC). If both τ₁ and τ₂ are sufficiently short (for instance, τ₁, τ₂ < 0.15 coalescent units), all four lineages a, b, c, and d frequently enter the root ancestor ABCD uncoalesced. Why does this create a discordant majority tree? A symmetric gene tree topology ((a, b), (c, d)) can be generated by two different temporal orderings of coalescent events in the root (either a + b coalesce first followed by c + d, OR c + d coalesce first followed by a + b), whereas the asymmetric true species topology (((a, b), c), d) requires a single specific strict sequence (a + b must coalesce first, then ab + c must coalesce second). Consequently, inside the anomaly zone, symmetric discordant anomalous gene trees (AGTs) have a higher mathematical probability than the true matching species tree, causing supermatrix concatenation methods to converge with 100% bootstrap support onto the wrong species tree.

Coalescent-Aware Species Tree Reconstruction (ASTRAL, BEST, StarBEAST)

To avoid concatenation bias in the anomaly zone, modern phylogenomics employs Multispecies Coalescent (MSC) algorithms that explicitly model the distribution of gene trees inside the species tree:

• Quartet-Based Summary Methods (e.g., ASTRAL, Mirarab et al., 2014): Take advantage of the mathematical proof that for unrooted four-taxon quartets, there are no anomalous gene trees—the true unrooted quartet ((A,B),(C,D)) always has probability 1 − (⅔)e^−*τ* > ⅓e^−*τ*. ASTRAL decomposes thousands of individual maximum-likelihood gene trees into unrooted quartets to find the statistically consistent species tree across genomes.
• Site-Based and Full-Bayesian Coalescent Methods (e.g., SVDquartets, StarBEAST3, BPP): Jointly co-estimate individual gene genealogies, ancestral population sizes (θ = 4*N*_eμ), and species divergence times (τ) directly from multi-locus sequence alignments.

The textbook example of incomplete lineage sorting occurs in the great ape clade (Homininae) comprising humans (Homo sapiens), chimpanzees (Pan troglodytes), and gorillas (Gorilla gorilla). Fossil and molecular clock calibrations show that the Gorilla lineage diverged from the Human-Chimpanzee ancestor at T₂ ≈ 8 to 10 million years ago, while Human and Chimpanzee diverged at T₁ ≈ 6 to 7 million years ago—leaving an internal branch of t ≈ 2 to 3 million years (~100,000 generations), while the ancestral effective population size was large (N_e ≈ 50,000 to 90,000, so τ = t / [2*N*_e] ≈ 0.55 to 0.80).

When Scally et al. (2012) and Mailund et al. sequenced and aligned the completegorilla, chimpanzee, and human autosomal genomes in 500-bp windows, they found that only ~70% of the human autosomal genome follows the true species tree ((Human, Chimpanzee), Gorilla); a full ~15% of our genes group ((Human, Gorilla), Chimpanzee) and another ~15% group ((Chimpanzee, Gorilla), Human) due to incomplete lineage sorting! Notably, on the X chromosome, the percentage of incomplete lineage sorting drops by roughly half (to ~16% total discordance) because the X chromosome has a smaller effective population size (N_e,X = ¾ N_e,auto, inflating τ_X = (4/3)*τ*_auto) and experiences stronger linked selection during speciation.

How does Patterson's D-statistic (the ABBA-BABA test) distinguish incomplete lineage sorting from hybridization (introgression)?

Given a four-taxon tree (((P₁, P₂), P₃), O) with outgroup O carrying ancestral allele A and derived mutant allele B, discordant character patterns where P₃ shares derived allele B with P₂ (ABBA) or with P₁ (BABA) can arise from either ILS or gene flow. Because incomplete lineage sorting in the ancestor of (P₁, P₂, P₃) is completely symmetric, pure ILS produces equal numbers of ABBA and BABA sites across the genome (D = [N_ABBA − N_BABA] / [N_ABBA + N_BABA] = 0). A statistically significant non-zero D (Z-score > 3) proves asymmetric post-speciation adaptive introgression or hybridization between P₃ and either P₂ (D > 0) or P₁ (D < 0).

Why does mitochondrial DNA (mtDNA) sort to reciprocal monophyly four times faster than autosomal nuclear DNA?

In a dioecious diploid species with an equal sex ratio, mitochondrial DNA is haploid (1 copy per gamete vs. 2 for autosomes) and maternally inherited through females only (½ of the population), giving mtDNA an effective population size equal to one-fourth that of an autosomal locus (N_e,mt = ¼ N_e,auto). Consequently, for any physical time interval t, the coalescent branch length for mtDNA (τ_mt = t / N_e,mt = 4 τ_auto) is four times longer, causing mtDNA lineages to coalesce and achieve reciprocal monophyly four times faster than nuclear genes.

What is the difference between homoplasy and hemiplasy?

Homoplasy is true independent evolutionary convergence, parallelism, or character reversal—where the same mutation or trait arises two or more separate times on different branches of a tree. Hemiplasy (Avise and Robinson, 2008) occurs when a trait mutates only once in history along an internal branch of a discordant gene tree (such as a ((Human, Gorilla), Chimp) locus), but when that character is mapped onto the true species tree ((Human, Chimp), Gorilla), it creates the illusion of two independent convergent origins or an origin followed by a secondary loss.

1. Pamilo, P., & Nei, M. (1988). "Relationships between gene trees and species trees." Molecular Biology and Evolution, 5(5), 568–583. doi:10.1093/oxfordjournals.molbev.a040517
2. Maddison, W. P. (1997). "Gene trees in species trees." Systematic Biology, 46(3), 523–536. doi:10.1093/sysbio/46.3.523
3. Degnan, J. H., & Rosenberg, N. A. (2006). "Discordance of species trees with their most likely gene trees." PLoS Genetics, 2(5), e68. doi:10.1371/journal.pgen.0020068
4. Scally, A., Dutheil, J. Y., Hillier, L. W., Jordan, G. E., Goodhead, I., Herrero, J., ... & Durbin, R. (2012). "Insights into hominid evolution from the gorilla genome sequence." Nature, 483(7388), 169–175. doi:10.1038/nature10842