Modern Coexistence Theory

Modern Coexistence Theory (MCT) is the mathematical framework in community ecology—developed and synthesized by Peter L. Chesson across a series of foundational papers (1990, 1994, 2000, 2018)—that explains how competing species stably coexist within an ecosystem or biosphere without driving one another to competitive exclusion. Rather than treating diverse ecological hypotheses (such as resource partitioning, specialist predators under the Janzen–Connell hypothesis, or temporal weather fluctuations) as separate qualitative explanations, Modern Coexistence Theory translates any mechanistic population model into two orthogonal quantitative axes: (1) Stabilizing mechanisms (niche differences, 1 − ρ), which cause species to limit themselves (intraspecific competition) more strongly than they limit competitors (interspecific competition), generating negative frequency dependence that rescues rare species; and (2) Equalizing mechanisms, which reduce average fitness differences (κ_j / κ_i) among competitors so that a smaller stabilizing niche difference is sufficient to maintain coexistence (Chesson, 2000).

Recovering from Low Density as the Benchmark of Stable Coexistence

Because real biological populations experience demographic stochasticity and environmental disturbances that periodically depress their abundance, a fixed equilibrium point on paper does not guarantee persistence in nature if a species cannot recover after being perturbed to low density. Modern Coexistence Theory therefore defines stable coexistence via the mutual invasibility criterion (Turelli, 1978; Chesson, 2000): two or more competing species stably coexist if and only if every species i, when reduced to an arbitrarily low density (N_i → 0, the invader state), exhibits a positive long-term average per-capita growth rate (invasion growth rate, r̄_i > 0) in the presence of the remaining resident species fluctuating at their stationary equilibrium or attractor.

In discrete time (where N_i(t + 1) = λ_i(t) N_i(t)), the long-term invasion growth rate of rare species i is the expected logarithmic per-capita growth rate across time:

r̄_i = E[ ln λ_i(t) ]_Ni→0 > 0

Because resident species at stationary equilibrium neither grow without bound nor decline to zero over the long term, the long-term average growth rate of every resident species r equals zero by definition (r̄_r = 0). Subtracting a weighted average of the residents' zero long-term growth rates from the invader's growth rate (r̄_i = r̄_i − q_ir r̄_r, where q_ir is a scaling comparison quotient that equates the two species' sensitivities to shared limiting factors) allows Chesson's framework to partition r̄_i into additive fitness and stabilizing components.

Niche Overlap (ρ) and Fitness Ratio (κ_j / κ_i)

In a constant environment governed by the two-species Competitive Lotka-Volterra Equations or derived directly from MacArthur's consumer–resource model and Tilman's Resource-Ratio Hypothesis, the per-capita growth rate of species i is written in Chesson's standardized form as:

(1 / N_i) dN_i/dt = r_0,i · ( 1 − α_ii N_i − α_ij N_j )

where r_0,i is the intrinsic growth rate, α_ii is the intraspecific competition coefficient (the per-capita self-limitation of species i, equal to 1/K_i), and α_ij is the interspecific competition coefficient (the per-capita competitive effect of species j on species i).

Chesson (2000, 2012) showed that all four competition coefficients (α₁₁, α₁₂, α₂₁, α₂₂) combine into two dimensionless quantities that govern community assembly:

1. Niche Overlap (ρ) and Stabilizing Niche Difference (1 − ρ):

ρ = √[ (α₁₂ · α₂₁) / (α₁₁ · α₂₂) ] Niche overlap ρ is the geometric mean of interspecific competition relative to intraspecific competition. When two species use identical resources in identical ways at the same time and place, ρ = 1 and niche difference (1 − ρ) = 0. As species diverge in resource use, root depth, phenology, or specialist natural enemies, intraspecific competition (α₁₁, α₂₂) exceeds interspecific competition (α₁₂, α₂₁), driving ρ → 0 and stabilizing niche difference (1 − ρ) → 1.

1. Average Fitness Ratio (κ₂ / κ₁):

κ₂ / κ₁ = (k₂ / k₁) · √[ (α₁₁ · α₁₂) / (α₂₂ · α₂₁) ] (where k_i scales intrinsic demographic performance). The fitness ratio measures the degree to which one species outcompetes the other in the absence of niche differentiation—combining intrinsic reproductive output with overall sensitivity to resource depletion.

Stable mutual coexistence (both r̄₁ > 0 and r̄₂ > 0) occurs if and only if the stabilizing niche overlap ρ is small enough to bound the fitness ratio between the two species:

ρ < ( κ₁ / κ₂ ) < ( 1 / ρ ) ⟺ ρ < min( κ₁/κ₂, κ₂/κ₁ )

The Four Additive Terms of the Invasion Growth Rate

In variable environments where environmental conditions E_i(t, x) and competitive intensity C_i(t, x) fluctuate across time t and space x, Chesson (1994, 2000) proved that the invasion growth rate r̄_i of any rare species i decomposes into four additive terms:

r̄_i = ΔE_i − ΔC_i + ΔN_i + ΔI_i

1. Fluctuation-Independent Average Fitness Difference (ΔE_i − ΔC_i baseline): Reflects baseline differences in mean demographic performance (ΔE_i) and equilibrium resource partitioning or specialist predation under constant conditions (ΔC_i).
2. Relative Nonlinearity of Competition (ΔN_i): Emerges when competing species differ in the curvature (second derivative ∂²r_i/∂C²) of their growth response to a shared fluctuating resource or competitive factor C(t)—such as a saturating gleaner vs. a linear opportunist—and each resident modifies the variance Var(C) to favor its rare competitor.
3. The Temporal Storage Effect (ΔI_i, temporal): Emerges when species have differential responses to temporal weather variation (E_i(t)), positive covariance between environmental favorability and competition when abundant (Cov(E_r, C_r) > 0), and a buffered persistent life stage (such as a dormant soil seed bank, diapausing lake resting eggs, or long-lived adult trees; ∂²ln λ_i / ∂E_i∂C_i < 0).
4. Spatial Storage Effect and Fitness–Density Covariance (ΔI_i, spatial + Δκ_i): Across spatially heterogeneous landscapes or metacommunities, spatial covariance between local environmental favorability and local competition (spatial storage effect) and the spatial aggregation of individuals in favorable patches (fitness–density covariance) provide additional spatial stabilization (Chesson, 2000; Sears & Chesson, 2007).

Over the past two decades, experimental and observational ecologists have applied Modern Coexistence Theory to measure niche differences (1 − ρ) and fitness ratios (κ_j/κ_i) directly in the field:

• Serpentine Annual Grasslands (Levine & HilleRisLambers, 2009): By sowing density gradients of competing California annual plants (Lasthenia californica, Plantago erecta, Vulpia microstachys) into natural field plots and neutralizing niche differences via experimental neighborhood manipulations, Janneke HilleRisLambers and Jonathan Levine proved that observed multi-species coexistence requires strong stabilizing niche differences (1 − ρ ≈ 0.3–0.6) to overcome large fitness inequalities (κ_j/κ_i > 2).
• Biological Invasions and Exotic Dominance (MacDougall et al., 2009): Modern Coexistence Theory resolved a long-standing paradox in invasion biology: does an exotic invader succeed because it occupies an "empty niche" (high 1 − ρ) or because it has a superior competitive fitness advantage (high κ_invader/κ_native)? Field parameterizations showed that if an exotic species differs in niche (high 1 − ρ) without a large fitness advantage, it merely coexists peacefully with native flora; invasive displacement and monoculture formation occur when the exotic possesses a large average fitness advantage (often conferred by the enemy release hypothesis or novel weapons hypothesis) that exceeds the community's stabilizing niche differences.
• Soil Microbial Feedbacks and Phylogeny (Godoy et al., 2014; Kandlikar et al., 2019): Integrating Bever's plant–soil feedback framework into MCT demonstrated that host-specific rhizosphere pathogens act as a powerful stabilizing mechanism (increasing α_ii relative to α_ij), whereas arbuscular mycorrhizal fungi and asymmetric light competition often amplify fitness differences (κ_j/κ_i).

Can equalizing mechanisms alone maintain species coexistence?

No. Equalizing mechanisms only reduce the average fitness difference between competitors (κ_j/κ_i → 1); they do not create negative frequency dependence. If stabilizing niche differences are strictly zero (1 − ρ = 0), even a tiny fitness difference or demographic random walk (ecological drift) will eventually cause one species to exclude the other. Stable coexistence always requires a non-zero stabilizing mechanism (1 − ρ > 0) sufficient to exceed the remaining fitness difference.

How does Hubbell's Neutral Theory fit into Modern Coexistence Theory?

Stephen Hubbell's Neutral Theory of Biodiversity (2001), which assumes all individuals of all species are demographically identical per capita, corresponds to the single mathematical origin point of Chesson's coexistence plane where stabilizing niche differences equal zero (1 − ρ = 0) and fitness differences equal zero (κ_j/κ_i = 1).

Why can two species with very different niches still fail to coexist?

Even if two species have substantial niche differentiation (for example, 1 − ρ = 0.4, so ρ = 0.6), if one species has an overwhelming demographic or competitive fitness advantage over the other (for example, κ₂/κ₁ = 3.0 > 1/0.6 = 1.67), the superior competitor will still exclude the inferior species. Coexistence depends on the ratio between niche differences and fitness differences, not on niche differences in isolation.

1. Chesson, P. (1994). "Multispecies competition in variable environments." Theoretical Population Biology, 45(3), 227–276. https://doi.org/10.1006/tpbi.1994.1013
2. Chesson, P. (2000). "Mechanisms of maintenance of species diversity." Annual Review of Ecology and Systematics, 31, 343–366. https://doi.org/10.1146/annurev.ecolsys.31.1.343
3. Adler, P. B., HilleRisLambers, J., & Levine, J. M. (2007). "A niche for neutrality." Ecology Letters, 10(2), 95–104. https://doi.org/10.1111/j.1461-0248.2006.00996.x
4. Levine, J. M., & HilleRisLambers, J. (2009). "The importance of niches for the maintenance of species diversity." Nature, 461(7261), 254–257. https://doi.org/10.1038/nature08251
5. MacDougall, A. S., Gilbert, B., & Levine, J. M. (2009). "Plant invasions and the niche." Journal of Ecology, 97(4), 609–615. https://doi.org/10.1111/j.1365-2745.2009.01514.x
6. Chesson, P. (2018). "Updates on mechanisms of maintenance of species diversity." Journal of Ecology, 106(5), 1773–1794. https://doi.org/10.1111/1365-2745.13035