Reinforcement (Speciation)

Reinforcement is an evolutionary mechanism of speciation in which natural selection directly increases prezygotic reproductive isolation—specifically positive assortative mating or gametic discrimination—between two incompletely isolated populations in sympatry or across a parapatric hybrid zone as a consequence of postzygotic selection against low-fitness hybrids (w_hybrid < 1). First anticipated in an early form by Alfred Russel Wallace in his 1889 book Darwinism (historically termed the Wallace effect, though Wallace invoked group-level selection), the modern individual-selection model was formulated by Theodosius Dobzhansky in his 1937 and 1940 syntheses Genetics and the Origin of Species and named "reinforcement" by W. Frank Blair in 1955.

In Dobzhansky's framework, when two populations diverge in allopatry, they accumulate incidental genetic differences—including Bateson–Dobzhansky–Muller incompatibilities or ecological mismatches—that render interpopulation hybrids partially sterile, inviable, or ecologically maladapted. If the two populations subsequently come into secondary contact, individuals that mate heterospecifically waste gametes and parental investment on unfit hybrid offspring, whereas individuals carrying alleles that confer stronger conspecific mate discrimination leave a higher proportion of viable, fertile purebred progeny. Consequently, natural selection favors alleles that accentuate divergence in courtship signals, floral coloration, pheromonal profiles, or female preference exclusively within the zone of geographic overlap.

The primary empirical diagnostic signature of reinforcement is reproductive character displacement (RCD): a biogeographic pattern in which prezygotic mating barriers, courtship signals, or mate preferences exhibit substantially greater divergence between two taxa in sympatric populations than in allopatric populations where the two species do not co-occur. Unlike ordinary allopatric speciation, where reproductive isolation arises strictly as an incidental pleiotropic byproduct of drift and adaptation, reinforcement represents the direct adaptive evolution of reproductive isolation by natural selection.

Two-Allele vs. One-Allele Models and Recombination Breakdown

Despite the intuitive appeal of Dobzhansky's verbal argument, Joseph Felsenstein demonstrated in a foundational 1981 mathematical analysis that reinforcement faces a severe population genetic obstacle in sexually reproducing diploids: recombination. To understand Felsenstein's objection, consider a three-locus haploid or diploid model in which two hybridizing populations differ at disruptive postzygotic incompatibility loci B and C (such that B₁C₁ and B₂C₂ have fitness 1, whereas recombinant hybrids B₁C₂ and B₂C₁ have reduced fitness 1 − s), while locus A controls assortative mating.

Felsenstein distinguished between two genetic architectures for the mating locus A:

1. One-allele mechanism: A single allele A₁ increases the propensity for assortative mating in both nascent species (for example, an allele causing females to mate with whichever male matches their own phenotype or natal host patch, or an allele shifting flowering time conditionally). Because the same A₁ allele is favored in both genetic backgrounds (B₁C₁ and B₂C₂), recombination does not oppose its spread, and reinforcement proceeds readily even under high gene flow.
2. Two-allele mechanism: Most biological courtship systems require fixing different alleles in the two species—for instance, fixing preference/signal allele A₁ in population 1 (B₁C₁) and allele A₂ in population 2 (B₂C₂). For selection against hybrids at loci B and C to drive divergence at locus A, non-random statistical associations—linkage disequilibrium (D_AB)—must be maintained between the assortative mating allele A₁ and the adaptive background B₁C₁.

However, whenever hybridization produces viable and even partially fertile F₁ heterozygotes (A₁B₁C₁ / A₂B₂C₂, with fitness 1 − s > 0), meiotic recombination at rate r between the mating locus A and the selected loci B, C breaks down linkage disequilibrium at each generation according to:

*D*'_AB ≈ (1 − r) D_AB + Δ*D*_selection

If the recombination rate r and the migration rate m exceed the strength of indirect selection transmitted from the incompatibility loci (proportional to s D_AB), recombination decouples the mate-choice allele A₁ from the species-1 genome, allowing A₁ to introgress into both populations and homogenizing the assortative mating system instead of reinforcing speciation.

Evolutionary Solutions Overcoming Felsenstein's Objection

Subsequent theoretical work by Maria R. Servedio, Mark Kirkpatrick, Nick H. Barton, and Mohamed A. F. Noor (Servedio & Noor 2003; Kirkpatrick & Ravigné 2002) identified four biological mechanisms that enable two-allele reinforcement to succeed despite Felsenstein's recombination barrier:

• Chromosomal inversions and tight physical linkage (r → 0): If the prezygotic preference/signal loci and postzygotic hybrid sterility loci reside inside the same paracentric or pericentric chromosomal inversion, crossover suppression in heterokaryotypes reduces effective recombination r to near zero, preserving maximal linkage disequilibrium D_AB across the hybrid zone (Noor et al. 2001).
• Direct natural or sexual selection on preference/signal loci: If the mate-recognition trait also experiences direct ecological selection in the local habitat ("magic traits," Gavrilets 2004) or strong asymmetric migration where a continent population swamps an island population (Servedio & Kirkpatrick 1997), direct selection supplements indirect reinforcement.
• Direct female fitness costs of heterospecific mating: Even when F₁ hybrids are sterile (s = 1, preventing any backcross recombination), females that mate with heterospecific males suffer direct reproductive loss proportional to C_female = s f_het (where f_het is the frequency of heterospecific encounters), creating strong direct selection on female preference loci without requiring recombinant hybrids.

Coyne and Orr's Drosophila Comparative Meta-Analysis

For decades following Mayr (1963) and Paterson (1985), skeptics argued that reinforcement was rare in nature and that sympatric species differed in mating signals only because pairs that already possessed strong prezygotic isolation before secondary contact were the only ones able to coexist without fusing. In 1989 and 1997, Jerry A. Coyne and H. Allen Orr tested reinforcement quantitatively across 119 and later 171 species pairs of Drosophila by plotting both prezygotic behavioral isolation and postzygotic isolation (hybrid sterility and inviability) against Nei's allozyme genetic distance (D, a molecular clock proxy for divergence time).

Coyne and Orr discovered three diagnostic patterns across the genus:

1. Postzygotic isolation accumulates at the same rate with genetic distance D in both allopatric and sympatric species pairs.
2. Among young species pairs (D < 0.50), sympatric pairs exhibit more than twice the level of prezygotic behavioral isolation (mean isolation index 0.76 vs. 0.36) compared to allopatric pairs of identical evolutionary age.
3. Concordant with Haldane's rule, enhanced prezygotic isolation in sympatry is strongest specifically in young species pairs that have already evolved unisexual F₁ hybrid male sterility, confirming that postzygotic maladaptation drives the rapid sympatric acceleration of premating isolation.

Floral Color Shift in Phlox drummondii (Hopkins & Rausher 2011)

The annual Texas wildflower Phlox drummondii provides a fully resolved molecular and field demonstration of reinforcement. Throughout its western allopatric range in central Texas, P. drummondii produces light-blue corollas identical in hue to its eastern congener Phlox cuspidata. Where the two species overlap in a sympatric zone across eastern central Texas, interspecific pollinations mediated by pipevine swallowtail butterflies (Battus philenor) yield F₁ hybrids with 65%–90% ovule and pollen sterility, consuming maternal ovules that would otherwise produce viable purebred seeds.

Strictly within the sympatric contact zone with P. cuspidata, P. drummondii populations have evolved dark-red corollas. Field transplant arrays conducted by Donald A. Levin (1985) and Robin Hopkins and Mark D. Rausher (2011, 2012) demonstrated that foraging B. philenor butterflies exhibit strong floral constancy by color, reducing interspecific pollen transfer between blue P. cuspidata and dark-red P. drummondii by up to 68% and conferring a net 35%–50% maternal fitness advantage (s ≈ 0.35) to dark-red P. drummondii plants when P. cuspidata is present. Molecular dissection revealed that the shift from light blue to dark red is controlled by two cis-regulatory mutations: a downregulation of the flavonoid 3',5'-hydroxylase gene (F3'5'H, shifting anthocyanin synthesis from blue delphinidin to red cyanidin pigment) and an upregulation of an R2R3-Myb transcription factor increasing total pigment intensity.

Plumage Divergence in Ficedula Flycatchers and Inversions in Drosophila

In central Europe and the Baltic islands of Gotland and Öland, the collared flycatcher (Ficedula albicollis) and the pied flycatcher (Ficedula hypoleuca) breed in sympatric deciduous woodlands. In allopatry (such as Spain, Britain, and northern Scandinavia), male pied flycatchers display jet-black dorsal plumage and white forehead patches closely resembling male collared flycatchers. In sympatry, where interspecific pairings produce completely sterile F₁ hybrid females (ZW Haldane's rule) and reduced-fitness hybrid males, male pied flycatchers have evolved dull-brown, female-like dorsal plumage while female collared flycatchers exhibit heightened mate choice selectivity. Genomic mapping by Sæther et al. (2007) demonstrated that male plumage traits, female species recognition, and hybrid sterility loci are all physically linked on the non-recombining Z sex chromosome, shielding the co-adapted gene complex from recombination.

Similarly, in the Pacific Northwest of North America, Drosophila pseudoobscura overlaps sympatrically with its sibling species D. persimilis. Sympatric D. pseudoobscura females discriminate strongly against D. persimilis males compared to allopatric D. pseudoobscura females from Utah or Bogota. Quantitative trait locus (QTL) mapping by Mohamed A. F. Noor and colleagues (2001) revealed that every QTL governing enhanced sympatric female mate discrimination maps directly inside fixed paracentric inversions on chromosome 2 and the X chromosome that also house the primary Bateson–Dobzhansky–Muller hybrid male sterility genes.

What is the difference between reinforcement and reproductive character displacement?

Reinforcement is the evolutionary process whereby natural selection strengthens prezygotic isolation between hybridizing populations to prevent the production of low-fitness hybrids. Reproductive character displacement (RCD) is the empirical biogeographic pattern—greater divergence of mating signals or preferences in sympatry than in allopatry—which can result either from reinforcement during incomplete speciation or from ecological/reproductive interference between already fully isolated species.

Why did Joseph Felsenstein argue that recombination opposes reinforcement?

In two-allele reinforcement models, one species must fix mating allele A₁ while the other fixes A₂, driven by postzygotic incompatibilities at separate loci B and C. Felsenstein (1981) showed mathematically that whenever F₁ hybrids are partially fertile, meiotic recombination breaks down the linkage disequilibrium (D_AB) between the assortative mating locus and the hybrid incompatibility loci, allowing a single mating allele to sweep through both species unless suppressed by chromosomal inversions or tight linkage.

Why does reinforcement often cause asymmetric mate discrimination in only one of two hybridizing species?

Asymmetric reinforcement commonly arises when one species is locally rarer in the sympatric zone or pays a higher maternal fitness cost per heterospecific mating. Individuals of the rarer species encounter heterospecific mates far more frequently than conspecifics, subjecting the rarer population (or the sex with higher parental investment, typically females) to much stronger selection for enhanced mate discrimination.

1. Dobzhansky, T. (1940). "Speciation as a stage in evolutionary divergence." The American Naturalist, 74(753), 312–321. https://doi.org/10.1086/280899
2. Felsenstein, J. (1981). "Skepticism towards Santa Rosalia, or why are there so few kinds of animals?" Evolution, 35(1), 124–138. https://doi.org/10.1111/j.1558-5646.1981.tb04864.x
3. Coyne, J. A., & Orr, H. A. (1989). "Patterns of speciation in Drosophila." Evolution, 43(2), 362–381. https://doi.org/10.1111/j.1558-5646.1989.tb04233.x
4. Hopkins, R., & Rausher, M. D. (2011). "Identification of two genes causing reinforcement in the Texas wildflower Phlox drummondii." Nature, 469(7330), 411–414. https://doi.org/10.1038/nature09641