Negative Frequency-Dependent Selection

Negative frequency-dependent selection (NFDS) (also termed rare-allele advantage or apostatic selection in predator-prey contexts) is a mode of natural selection in which the relative Darwinian fitness of a genotype or phenotype decreases monotonically as its relative frequency in the breeding population increases (dw_i / dp_i < 0). Because any allele or morph experiences its highest reproductive success when rare and suffers reduced fitness when abundant, negative frequency-dependent selection acts as a self-correcting homeostatic restoring force in population genetics.

Alongside heterozygote advantage, negative frequency-dependent selection is the most potent mechanism of balancing selection in nature. Unlike overdominance—which struggles to maintain more than a handful of alleles simultaneously and imposes a heavy segregation load at equilibrium—negative frequency-dependent selection can stably maintain dozens to hundreds of functional alleles at a single locus (such as plant self-incompatibility S-loci and vertebrate Major Histocompatibility Complex genes) with zero segregational genetic load at the internal equilibrium.

Interactive Simulation: Explore how frequency-dependent fitness regimes protect rare alleles from stochastic loss in the Allele Drift Simulator.

Haploid and Clonal Rare-Type Advantage

Consider a population with two competing types, A₁ and A₂, occurring at frequencies p and q = 1 − p. Under linear negative frequency-dependent selection, the relative fitness of each variant declines in direct proportion to its own frequency in the population, governed by selection sensitivity coefficients s₁ > 0 and s₂ > 0:

• Fitness of A₁: w₁(p) = 1 − s₁ · p
• Fitness of A₂: w₂(q) = 1 − s₂ · q = 1 − s₂ · (1 − p)

The mean fitness of the population is w̄ = p · w₁(p) + q · w₂(q) = 1 − s₁p² − s₂q², and the single-generation change in the allele frequency of A₁ is:

Δp = [ pq · (w₁(p) − w₂(q)) ] / w̄ = [ pq · (s₂q − s₁p) ] / (1 − s₁p² − s₂q²)

Setting Δp = 0 yields a unique internal polymorphic equilibrium where the fitnesses of both types become exactly equal (w₁(p̂) = w₂(q̂) ⇒ s₁p̂ = s₂q̂):

p̂ = s₂ / (s₁ + s₂) and q̂ = s₁ / (s₁ + s₂)

In the symmetric case where s₁ = s₂ = s, the equilibrium lies at p̂ = q̂ = 0.50. Whenever p < p̂, w₁(p) > w₂(q) and Δp > 0; whenever p > p̂, w₁(p) < w₂(q) and Δp < 0.

Diploid Genotypic Formulation and Comparison with Overdominance

In a diploid Mendelian population with genotypes A₁A₁ (frequency P = p²), A₁A₂ (frequency H = 2pq), and A₂A₂ (frequency Q = q²), linear genotypic negative frequency dependence (Clarke & O'Donald, 1964; Wright, 1969) assigns fitnesses inversely proportional to each genotype's frequency:

• w₁₁ = 1 − t · p²
• w₁₂ = 1 − t · (2pq)
• w₂₂ = 1 − t · q²

Although the algebraic form of Δp under negative frequency-dependent selection resembles that of heterozygote advantage, three profound mathematical and biological differences separate the two regimes:

1. Zero Segregation Variance at Equilibrium: Under heterozygote advantage, homozygotes are permanently inferior to heterozygotes (w₁₁ < w₁₂ > w₂₂) at all allele frequencies, generating a standing segregation load at equilibrium. Under negative frequency-dependent selection, when the population reaches its equilibrium (p = q = 0.50 in the allelic model, or when genotype fitnesses equalize), all morphs or alleles have identical relative fitness (w₁ = w₂), eliminating relative fitness variance and differential mortality at equilibrium.
2. Violation of Mean Fitness Maximization: Under constant fitness coefficients, Fisher's fundamental theorem of natural selection guarantees that Δp moves the population toward a local maximum of mean fitness w̄. Under frequency-dependent selection, because the fitness landscape w_i(p) shifts as p changes, natural selection equalizes marginal fitnesses rather than maximizing w̄—in fact, in competitive resource-use models, selection can drive the population to an evolutionary stable strategy (ESS) that minimizes w̄.
3. Maintenance of Hyper-Allelic Diversity (k ≫ 2): In a multiallelic system with k alleles where w_i = 1 − s · p_i, any newly arising genetic mutation starts at p_new = 1 / (2N_e) ≈ 0 and automatically enjoys the maximum possible fitness w_new ≈ 1 > w̄ = 1 − s/k, guaranteeing invasion and stable maintenance of dozens of alleles at equal frequencies p̂_i = 1 / k.

Apostatic Selection and Predator Search Images (Clarke 1962)

In 1962, Bryan C. Clarke coined the term apostatic selection to describe negative frequency-dependent predation on visually polymorphic prey—such as the European grove snail (Cepaea nemoralis and Cepaea hortensis), whose shells vary in ground color (yellow, pink, brown) and dark spiral banding (0 to 5 bands). Visual vertebrate predators such as song thrushes (Turdus philomelos) and blue jays form a cognitive neurological search image (Nikolaas Tinbergen, 1960) tuned to the most frequently encountered cryptic prey morph in a local habitat patch. Because predators disproportionately detect and consume the abundant morph while overlooking rare atypical morphs ("apostates"), rare shell patterns enjoy higher survival until they become common enough to trigger a switch in the predator's search image.

Elderflower Orchids, Scale-Eating Cichlids, and Behavioural Game Theory

• Deceptive Pollination in Elderflower Orchids (Dactylorhiza sambucina): Across European meadows, the nectarless food-deceptive orchid Dactylorhiza sambucina blooms in two discrete flower-color morphs: yellow and magenta-purple. Naive bumblebee queens (Bombus spp.) emerging in spring visit the most common flower color first; upon discovering that the flowers contain zero nectar reward, the bees learn to avoid that color and switch to visiting the rarer color morph. In a classic field manipulation, Luc Gigord et al. (2001) constructed experimental arrays varying yellow-to-purple ratios from 1:9 to 9:1 and proved that rare morphs achieved vastly higher pollinia removal and seed set, stabilizing the polymorphism near a 50:50 ratio.
• Jaw Handedness in Scale-Eating Cichlids (Perissodus microlepis): In Lake Tanganyika, the scale-eating cichlid fish Perissodus microlepis exhibits an asymmetrical mouth opening twisted either to the left ("sinistral", attacking the right flank of prey fish) or to the right ("dextral", attacking the left flank of prey). Michio Hori (1993) demonstrated over an 11-year time series that whenever sinistral cichlids become more abundant (>55%), prey fish guard their right flanks more vigilantly, giving rare dextral cichlids higher foraging success and driving a damped 5-year oscillation around the 50:50 equilibrium.
• Alternative Male Mating Strategies (Rock-Paper-Scissors and Sneaker Males): In side-blotched lizards (Uta stansburiana), Barry Sinervo and Curt Lively (1996) documented a three-morph negative frequency-dependent cycle: aggressive orange-throated males defeat mate-guarding blue-throated males, yellow-throated "sneaker" (female-mimic) males cuckold large territories of orange-throated males, and blue-throated males detect and exclude yellow sneakers, driving a 4- to 5-year cyclic turnover of allele frequencies.

Gametophytic and Sporophytic Self-Incompatibility (S-Loci)

The strongest molecular negative frequency-dependent selection known in biology occurs at the self-incompatibility (S)-locus of flowering plants, analyzed mathematically by Sewall Wright in 1939. In gametophytic self-incompatibility (found in Solanaceae, Rosaceae, and Papaveraceae), a haploid pollen grain carrying allele S_i is destroyed by stylar RNases (or S-receptor kinase cascades in Brassicaceae) if the diploid maternal plant carries S_i as either of its two alleles (e.g., S_iS_j).

Consequently, common S-alleles frequently land on incompatible stigmas and fail to fertilize ovules, whereas a rare or newly mutated allele S_k is compatible with virtually 100% of maternal plants in the population. Under Wright's infinite-population model, k alleles are maintained at exact uniform frequencies p̂ = 1 / k (provided k ≥ 3), and in finite populations of size N_e with mutation rate μ, mutation-selection-drift balance maintains 30 to >100 deeply diverged alleles per species. Similarly, in fungi such as the split-gill mushroom Schizophyllum commune, two unlinked mating-type loci (A and B) governed by negative frequency-dependent mate compatibility maintain over 20,000 distinct functional mating types.

What is the difference between negative and positive frequency-dependent selection?

Negative frequency-dependent selection favors rare phenotypes (dw_i / dp_i < 0), acting as a form of balancing selection that preserves genetic polymorphism within a population. Positive frequency-dependent selection favors common phenotypes (dw_i / dp_i > 0) and penalizes rare oddities. For example, in Müllerian aposematic mimicry among toxic Heliconius butterflies, local predators learn to avoid the most abundant warning-color wing pattern; rare mutant wing patterns are not recognized as toxic by predators and are quickly eliminated, driving local fixation of a single warning pattern within each geographic region.

Why does a time lag in negative frequency-dependent selection cause oscillations instead of a static equilibrium?

When the fitness of a genotype depends instantaneously on its current frequency in the same generation—such as pollen compatibility at plant S-loci or Fisherian sex-ratio selection—allele frequencies converge smoothly to a static stable equilibrium (p̂). However, when frequency dependence is mediated by a second coevolving population (such as a parasite population under the Red Queen hypothesis or a predator population's numerical response), there is an ecological and evolutionary time lag (delay τ) between when a host allele becomes common and when the matching parasite reaches peak abundance. This delayed negative feedback overshoots the equilibrium point in both directions, generating sustained limit-cycle oscillations.

How does Fisher's principle of 1:1 sex ratios operate via negative frequency-dependent selection?

Formulated by Ronald Fisher (1930) and D. A. Düsing (1884), Fisher's principle explains why most dioecious species maintain a 1:1 primary sex ratio (equal parental investment in sons and daughters). In any diploid species, total male reproductive output must equal total female reproductive output (every offspring has one mother and one father). If males become rare in the population, each surviving son achieves a higher average number of matings than each daughter; consequently, parents genetically predisposed to produce the rarer sex leave more grandchildren, driving allele frequencies back to the 1:1 equilibrium.

1. Clarke, B. (1962). "Balanced polymorphism and the diversity of sympatric species." Systematics Association Publication, 4, 47–70. https://scholar.google.com/scholar?q=Clarke+1962+Balanced+polymorphism+and+the+diversity+of+sympatric+species
2. Wright, S. (1939). "The distribution of self-sterility alleles in populations." Genetics, 24(4), 538–552. doi:10.1093/genetics/24.4.538
3. Hori, M. (1993). "Frequency-dependent natural selection in the handedness of scale-eating cichlid fish." Science, 260(5105), 216–219. doi:10.1126/science.260.5105.216
4. Gigord, L. D. B., Macnair, M. R., & Smithson, A. (2001). "Negative frequency-dependent selection maintains a dramatic flower color polymorphism in the rewardless orchid Dactylorhiza sambucina (L.) Soò." Proceedings of the National Academy of Sciences, 98(11), 6253–6255. doi:10.1073/pnas.111162598