Heterozygosity

A population bottleneck can result in a loss of heterozygosity. In this hypothetical population, an allele has become fixed after the population repeatedly dropped from 10 to 3.

One of the results of population structure is a reduction in heterozygosity. When populations split, alleles have a higher chance of reaching fixation within subpopulations, especially if the subpopulations are small or have been isolated for long periods. This reduction in heterozygosity can be thought of as an extension of inbreeding, with individuals in subpopulations being more likely to share a recent common ancestor.^[7] The scale is important — an individual with both parents born in the United Kingdom is not inbred relative to that country's population, but is more inbred than two humans selected from the entire world. This motivates the derivation of Wright's F-statistics (also called "fixation indices"), which measure inbreeding through observed versus expected heterozygosity.^[8] For example, ${\displaystyle F_{IS}}$ measures the inbreeding coefficient at a single locus for an individual ${\displaystyle I}$ relative to some subpopulation ${\displaystyle S}$:^[9]

${\displaystyle F_{IS}=1-{\frac {H_{I}}{H_{S}}}}$

Here, ${\displaystyle H_{I}}$ is the fraction of individuals in subpopulation ${\displaystyle S}$ that are heterozygous. Assuming there are two alleles, ${\displaystyle A_{1},A_{2}}$ that occur at respective frequencies ${\displaystyle p_{S},q_{S}}$, it is expected that under random mating the subpopulation ${\displaystyle S}$ will have a heterozygosity rate of ${\displaystyle H_{S}=2p_{S}(1-p_{S})=2p_{S}q_{S}}$. Then:

${\displaystyle F_{IS}=1-{\frac {H_{I}}{2p_{S}q_{S}}}}$

Similarly, for the total population ${\displaystyle T}$, we can define ${\displaystyle H_{T}=2p_{T}q_{T}}$ allowing us to compute the expected heterozygosity of subpopulation ${\displaystyle S}$ and the value ${\displaystyle F_{ST}}$ as:^[9]

${\displaystyle F_{ST}=1-{\frac {H_{S}}{H_{T}}}=1-{\frac {2p_{S}q_{S}}{2p_{T}q_{T}}}}$

If F is 0, then the allele frequencies between populations are identical, suggesting no structure. The theoretical maximum value of 1 is attained when an allele reaches total fixation, but most observed maximum values are far lower.^[7] F_ST is one of the most common measures of population structure and there are several different formulations depending on the number of populations and the alleles of interest. Although it is sometimes used as a genetic distance between populations, it does not always satisfy the triangle inequality and thus is not a metric.^[10] It also depends on within-population diversity, which makes interpretation and comparison difficult.^[4]

Admixture inference

An individual's genotype can be modelled as an admixture between K discrete clusters of populations.^[9] Each cluster is defined by the frequencies of its genotypes, and the contribution of a cluster to an individual's genotypes is measured via an estimator. In 2000, Jonathan K. Pritchard introduced the STRUCTURE algorithm to estimate these proportions via Markov chain Monte Carlo, modelling allele frequencies at each locus with a Dirichlet distribution.^[11] Since then, algorithms (such as ADMIXTURE) have been developed using other estimation techniques.^[12][13] Estimated proportions can be visualized using bar plots — each bar represents an individual, and is subdivided to represent the proportion of an individual's genetic ancestry from one of the K populations.^[9]

Varying K can illustrate different scales of population structure; using a small K for the entire human population will subdivide people roughly by continent, while using large K will partition populations into finer subgroups.^[9] Though clustering methods are popular, they are open to misinterpretation: for non-simulated data, there is never a "true" value of K, but rather an approximation considered useful for a given question.^[3] They are sensitive to sampling strategies, sample size, and close relatives in data sets; there may be no discrete populations at all; and there may be hierarchical structure where subpopulations are nested.^[3] Clusters may be admixed themselves,^[9] and may not have a useful interpretation as source populations.^[14]

A study of population structure of humans in Northern Africa and neighboring populations modelled using ADMIXTURE and assuming K=2,4,6,8 populations (Figure B, top to bottom). Varying K changes the scale of clustering. At K=2, 80% of the inferred ancestry for most North Africans is assigned to cluster that is common to Basque, Tuscan, and Qatari Arab individuals (in purple). At K=4, clines of North African ancestry appear (in light blue). At K=6, opposite clines of Near Eastern (Qatari) ancestry appear (in green). At K=8, Tunisian Berbers appear as a cluster (in dark blue).^[15]

Dimensionality reduction

A map of the locations of genetic samples of several African populations (left) and principal components 1 and 2 of the data superimposed on the map (right). The principal coordinate plane has been rotated 16.11° to align with the map. It corresponds to the east–west and north–south distributions of the populations.^[16]

Genetic data are high dimensional and dimensionality reduction techniques can capture population structure. Principal component analysis (PCA) was first applied in population genetics in 1978 by Cavalli-Sforza and colleagues and resurged with high-throughput sequencing.^[9][17] Initially PCA was used on allele frequencies at known genetic markers for populations, though later it was found that by coding SNPs as integers (for example, as the number of non-reference alleles) and normalizing the values, PCA could be applied at the level of individuals.^[13][18] One formulation considers ${\displaystyle N}$ individuals and ${\displaystyle S}$ bi-allelic SNPs. For each individual ${\displaystyle i}$, the value at locus ${\displaystyle l}$ is ${\displaystyle g_{i,l}}$ is the number of non-reference alleles (one of ${\displaystyle 0,1,2}$). If the allele frequency at ${\displaystyle l}$ is ${\displaystyle p_{l}}$, then the resulting ${\displaystyle N\times S}$ matrix of normalized genotypes has entries:^[9]

${\displaystyle {\frac {g_{i,l}-2p_{l}}{\sqrt {2p_{l}(1-p_{l})}}}}$

PCA transforms data to maximize variance; given enough data, when each individual is visualized as point on a plot, discrete clusters can form.^[13] Individuals with admixed ancestries will tend to fall between clusters, and when there is homogenous isolation by distance in the data, the top PC vectors will reflect geographic variation.^[19][13] The eigenvectors generated by PCA can be explicitly written in terms of the mean coalescent times for pairs of individuals, making PCA useful for inference about the population histories of groups in a given sample. PCA cannot, however, distinguish between different processes that lead to the same mean coalescent times.^[20]

Multidimensional scaling and discriminant analysis have been used to study differentiation, population assignment, and to analyze genetic distances.^[21] Neighborhood graph approaches like t-distributed stochastic neighbor embedding (t-SNE) and uniform manifold approximation and projection (UMAP) can visualize continental and subcontinental structure in human data.^[22][23] With larger datasets, UMAP better captures multiple scales of population structure; fine-scale patterns can be hidden or split with other methods, and these are of interest when the range of populations is diverse, when there are admixed populations, or when examining relationships between genotypes, phenotypes, and/or geography.^[23][24] Variational autoencoders can generate artificial genotypes with structure representative of the input data, though they do not recreate linkage disequilibrium patterns.^[25]