Ross Cressman, Josef Hofbauer and W.Gordon S. Hines:

Evolutionary stability in strategic models of single-locus frequency-dependent viability selection

J. Math. Biol. 34 (1996), 707-733.

Abstract. The dynamic stability of an evolutionarily stable strategy (ESS) is analyzed for a diploid species under individual viability selection. An individual's viability depends on the genotypic frequencies at a single autosomal locus through a payoff matrix determined by phenotypic behaviours (i.e. strategies). It is shown that an ESS of this payoff matrix is dynamically stable if there are at most three alleles. The proof makes heavy use of center manifold and normal form calculations.

The case of two strategies was treated earlier by Maynard Smith, Eshel and Lessard. The general case is still open.

reprint from Springer Link.