Evolutionary Game Theory and Evolutionarily Stable Strategies

How game theory explains behavior that evolves over generations rather than being chosen deliberately by a reasoning player.

Most of this module treats players as people who sit down, think through their options, and choose a strategy on purpose. Evolutionary game theory asks a different question: what happens when the “players” are organisms - or even genes, firms, or habits - that don’t reason at all, but simply survive or don’t, and pass their traits on to future generations if they do? It turns out many of the same mathematical tools apply, just interpreted differently.

Strategies that survive instead of strategies that are chosen

In the classical game theory covered earlier in this module, a player picks a strategy because it’s the best response to what others are doing. In evolutionary game theory, a strategy isn’t picked at all - it’s simply a trait some fraction of a population happens to have, like aggressive or passive behavior in a species of animal. Individuals with more successful traits tend to reproduce more, so those traits become more common over generations, while less successful traits fade out. The “game” is played not once by reasoning individuals, but repeatedly across a population, generation after generation.

The evolutionarily stable strategy

An evolutionarily stable strategy, or ESS, is a strategy that, once it dominates a population, can’t be successfully invaded by a rare alternative strategy. If nearly everyone in a population plays the ESS, any small group of individuals trying something different will do worse on average and eventually die out, so the population stays put at that strategy. This is a close cousin of the Nash equilibrium covered earlier in this module, but it’s a stronger condition, since it also has to hold up against invasion by rare mutant strategies, not just against a single rational opponent’s best response.

Hawks and doves

Imagine a population of animals competing for food, where each individual is either a "hawk," who always fights over a resource, or a "dove," who backs down rather than fight. If the whole population were doves, a single hawk would win every contest and thrive, so doves alone aren't stable. If the whole population were hawks, they'd constantly injure each other in fights, and a dove that avoided fighting altogether would actually do better by staying safe. The stable outcome, biologists find, is usually a mix of hawks and doves in specific proportions - a population that settles into that mix resists invasion by either pure strategy.

Replicator dynamics: how populations get there

Replicator dynamics describes the process by which a population moves toward an evolutionarily stable strategy over time: strategies that currently perform above the population’s average payoff grow as a share of the population, while those performing below average shrink. This isn’t a single individual reasoning its way to the right answer - it’s an aggregate process, similar to how a market share can shift toward better products over time even though no single consumer is tracking the whole market.

Assuming this framework only applies to animals

Evolutionary game theory was developed by biologists studying animal behavior, but economists now use it widely to study human institutions too - how business practices, social norms, or trading strategies spread or die out based on which ones perform better, even when no individual firm or person is consciously optimizing the system as a whole. A habit or convention can be evolutionarily stable in a market the same way a behavior can be evolutionarily stable in a species.

Why this matters beyond biology

This framework helps explain persistence: why certain business practices, social conventions, or even seemingly irrational-looking behaviors stick around even when no one is deliberately choosing them, as long as they’re hard for an alternative to successfully displace once established. It also helps explain change: a strategy that looked stable can become unstable if the environment shifts, opening the door for a new strategy to take hold.

Key takeaways

Evolutionary game theory studies strategies that spread or fade through survival and reproduction, not deliberate choice.
An evolutionarily stable strategy resists invasion by any small group of individuals trying a different strategy.
An ESS is a stronger condition than an ordinary Nash equilibrium, since it must also resist rare mutant strategies.
The hawk-dove example shows a stable population often mixes strategies rather than settling on just one.
Replicator dynamics describes populations shifting toward above-average-performing strategies over time.
Economists apply this framework to business practices and social norms, not just animal behavior.