Evolution under Stochastic Transmission: Mutation-Rate Modifiers

Evolutionary analyses of large populations commonly incorporate stochasticity through temporal variation in selection while treating genetic transmission as fixed. Much less attention has been given to stochasticity in transmission itself. We study a selected locus with alleles $A$ and $a$ under constant selection, linked to a neutral modifier locus whose alleles $M_1$ and $M_2$ control the mutation rate from $A$ to $a$. Under constant transmission, the Reduction Principle applies: near a mutation--selection balance where $M_1$ is fixed with mutation rate $u_1$, a rare allele $M_2$ invades if its associated rate $u_2$ is smaller than $u_1$, but cannot invade if $u_2$ is larger than $u_1$. This result holds for both haploid and diploid populations and is independent of recombination, which affects only the rate, not the direction, of evolutionary change. We extend this framework by allowing the mutation rate associated with the invading modifier to fluctuate randomly across generations. In this stochastic setting, invasion is no longer determined by mean mutation rates alone. Instead, it depends on the temporal distribution of mutation rates, the strength of selection at the selected locus, and the recombination rate between modifier and target. Stochastic transmission and recombination therefore do not merely rescale deterministic predictions based on the Reduction Principle; they can alter the direction of selection on modifier alleles.