ABSTRACT
FST, F ′ST, ΦST, Φ ′ ST, and Dest are the primary metrics utilized for empirically estimating and test-
ing the magnitude of genetic divergence among populations. There is currently active discussion in the literature about which of these metrics are most appropriate for empirical surveys of genetic differentiation. Here we compare the performance of each metric in 80 simulated population comparisons with an a priori known level of genetic differentiation that ranges from 0 to 100%. In these simulations, we manipulate population characteristics such as the genetic distance between haplotypes and the diversity of haplotypes that are shared among populations, as well as those that are unique to specific populations, illustrating key features and differences among the metrics, with an eye toward separating biological signal from statistical noise. FST is the best choice for datasets consisting of neutral unlinked single nucleotide polymorphism (SNP) datasets involving two alleles per locus. Dest and F ′ST tend to be the best metrics for analyses with more than two alleles, at least where the genetic distance among the alleles is not important, but the use of F ′ST and, to a greater extent, Dest is currently limited to relatively simple datasets, due to a dearth of computer software. If genetic distance among alleles is an important consideration, then ΦST is the better metric, but we demonstrate that ΦST and Φ′ST can accentuate either noise or signal, depending upon the characteristics of the populations and the hypotheses being tested. In many cases, it can be informative to apply both distance-based and allele/haplotype-based metrics, or both fixation and genetic differentiation indices. All of these measures are highly sensitive to the diversity of alleles shared between populations, with common alleles dominating the behavior of all of these metrics. ΦST is shown to be relatively unaffected by the phenomenon of high allelic diversity driving down estimates of genetic differentiation that plague FST. Overall, there is no single metric that best captures population genetic differentiation, and we recommend that researchers report both a fixation index (FST or ΦST) and an index of genetic differentiation (F ′ST or Dest) for their datasets because they represent different properties of population partitioning. When indices of fixation and genetic differentiation are in agreement, one can be sure of the conclusion. When the two methods yield differing results, the pattern and direction of discord can be diagnostic of a particular phenomenon, and we provide a range of simulations across parameter space to illustrate both points.
