Definition

The p-value is the probability under the null hypothesis of obtaining a real-valued test statistic at least as extreme as the one obtained. Consider an observed test-statistic ${\displaystyle t}$ from unknown distribution ${\displaystyle T}$. Then the p-value ${\displaystyle p}$ is what the prior probability would be of observing a test-statistic value at least as "extreme" as ${\displaystyle t}$ if null hypothesis ${\displaystyle H_{0}}$ were true. That is:

• ${\displaystyle p=\Pr(T\geq t\mid H_{0})}$ for a one-sided right-tail test-statistic distribution,
• ${\displaystyle p=\Pr(T\leq t\mid H_{0})}$ for a one-sided left-tail test-statistic distribution,
• ${\displaystyle p=2\min\{\Pr(T\geq t\mid H_{0}),\Pr(T\leq t\mid H_{0})\}}$ for a two-sided test-statistic distribution. If the distribution of ${\displaystyle T}$ is symmetric about zero, then ${\displaystyle p=\Pr(|T|\geq |t|\mid H_{0})}$

Interpretations

The error that a practising statistician would consider the more important to avoid (which is a subjective judgment) is called the error of the first kind. The first demand of the mathematical theory is to deduce such test criteria as would ensure that the probability of committing an error of the first kind would equal (or approximately equal, or not exceed) a preassigned number α, such as α = 0.05 or 0.01, etc. This number is called the level of significance.

— Jerzy Neyman, "The Emergence of Mathematical Statistics"^[8]

In a significance test, the null hypothesis ${\displaystyle H_{0}}$ is rejected if the p-value is less than or equal to a predefined threshold value ${\displaystyle \alpha }$, which is referred to as the alpha level or significance level. ${\displaystyle \alpha }$ is not derived from the data, but rather is set by the researcher before examining the data. ${\displaystyle \alpha }$ is commonly set to 0.05, though lower alpha levels are sometimes used. In 2018, a group of statisticians led by Daniel Benjamin proposed the adoption of the 0.005 value as standard value for statistical significance worldwide.^[9]

Different p-values based on independent sets of data can be combined, for instance using Fisher's combined probability test.

Distribution

The p-value is a function of the chosen test statistic ${\displaystyle T}$ and is therefore a random variable. If the null hypothesis fixes the probability distribution of ${\displaystyle T}$ precisely (e.g. ${\displaystyle H_{0}:\theta =\theta _{0}}$ where ${\displaystyle \theta }$ is the only parameter), and if that distribution is continuous, then when the null-hypothesis is true the p-value is uniformly distributed between 0 and 1. Regardless of the truth of the ${\displaystyle H_{0}}$, the p-value is not fixed; if the same test is repeated independently with fresh data one will typically obtain a different p-value in each iteration.

Usually only a single p-value relating to a hypothesis is observed, so the p-value is interpreted by a significance test and no effort is made to estimate the distribution it was drawn from. When a collection of p-values are available (e.g. when considering a group of studies on the same subject), the distribution of p-values is sometimes called a p-curve.^[10] A p-curve can be used to assess the reliability of scientific literature, such as by detecting publication bias or p-hacking. ^[10][11]

Distribution for composite hypothesis

In parametric hypothesis testing problems, a simple or point hypothesis refers to a hypothesis where the parameter's value is assumed to be a single number. In contrast, in a composite hypothesis the parameter's value is given by a set of numbers. When the null-hypothesis is composite (or the distribution of the statistic is discrete), then when the null-hypothesis is true the probability of obtaining a p-value less than or equal to any number between 0 and 1 is still less than or equal to that number. In other words, it remains the case that very small values are relatively unlikely if the null-hypothesis is true, and that a significance test at level ${\displaystyle \alpha }$ is obtained by rejecting the null-hypothesis if the p-value is less than or equal to ${\displaystyle \alpha }$.^[12][13]

For example, when testing the null hypothesis that a distribution is normal with a mean less than or equal to zero against the alternative that the mean is greater than zero (${\displaystyle H_{0}:\mu \leq 0}$, variance known), the null hypothesis does not specify the exact probability distribution of the appropriate test statistic. In this example that would be the Z-statistic belonging to the one-sided one-sample Z-test. For each possible value of the theoretical mean, the Z-test statistic has a different probability distribution. In these circumstances the p-value is defined by taking the least favorable null-hypothesis case, which is typically on the border between null and alternative. This definition ensures the complementarity of p-values and alpha-levels: ${\displaystyle \alpha =0.05}$ means one only rejects the null hypothesis if the p-value is less than or equal to ${\displaystyle 0.05}$, and the hypothesis test will indeed have a maximum type-1 error rate of ${\displaystyle 0.05}$.

Misuse

According to the ASA, there is widespread agreement that p-values are often misused and misinterpreted.^[3] One practice that has been particularly criticized is accepting the alternative hypothesis for any p-value nominally less than .05 without other supporting evidence. Although p-values are helpful in assessing how incompatible the data are with a specified statistical model, contextual factors must also be considered, such as "the design of a study, the quality of the measurements, the external evidence for the phenomenon under study, and the validity of assumptions that underlie the data analysis".^[3] Another concern is that the p-value is often misunderstood as being the probability that the null hypothesis is true.^[3][15]

Some statisticians have proposed abandoning p-values and focusing more on other inferential statistics,^[3] such as confidence intervals,^[16][17] likelihood ratios,^[18][19] or Bayes factors,^[20][21][22] but there is heated debate on the feasibility of these alternatives.^[23][24] Others have suggested to remove fixed significance thresholds and to interpret p-values as continuous indices of the strength of evidence against the null hypothesis.^[25][26] Yet others suggested to report alongside p-values the prior probability of a real effect that would be required to obtain a false positive risk (i.e. the probability that there is no real effect) below a pre-specified threshold (e.g. 5%).^[27]

That said, in 2019 a task force by ASA had convened to consider the use of statistical methods in scientific studies, specifically hypothesis tests and p-values, and their connection to replicability.^[7] It states that "Different measures of uncertainty can complement one another; no single measure serves all purposes.", citing p-value as one of these measures. They also stress that p-values can provide valuable information when considering the specific value as well as when compared to some threshold. In general, it stresses that "p-values and significance tests, when properly applied and interpreted, increase the rigor of the conclusions drawn from data."

Testing the fairness of a coin

As an example of a statistical test, an experiment is performed to determine whether a coin flip is fair (equal chance of landing heads or tails) or unfairly biased (one outcome being more likely than the other).

Suppose that the experimental results show the coin turning up heads 14 times out of 20 total flips. The full data ${\displaystyle X}$ would be a sequence of twenty times the symbol "H" or "T." The statistic on which one might focus could be the total number ${\displaystyle T}$ of heads. The null hypothesis is that the coin is fair, and coin tosses are independent of one another. If a right-tailed test is considered, which would be the case if one is actually interested in the possibility that the coin is biased towards falling heads, then the p-value of this result is the chance of a fair coin landing on heads at least 14 times out of 20 flips. That probability can be computed from binomial coefficients as

${\displaystyle {\begin{aligned}&\Pr(14{\text{ heads}})+\Pr(15{\text{ heads}})+\cdots +\Pr(20{\text{ heads}})\\&={\frac {1}{2^{20}}}\left[{\binom {20}{14}}+{\binom {20}{15}}+\cdots +{\binom {20}{20}}\right]={\frac {60,\!460}{1,\!048,\!576}}\approx 0.058\end{aligned}}}$

This probability is the p-value, considering only extreme results that favor heads. This is called a one-tailed test. However, one might be interested in deviations in either direction, favoring either heads or tails. The two-tailed p-value, which considers deviations favoring either heads or tails, may instead be calculated. As the binomial distribution is symmetrical for a fair coin, the two-sided p-value is simply twice the above calculated single-sided p-value: the two-sided p-value is 0.115.

In the above example:

• Null hypothesis (H₀): The coin is fair, with Pr(heads) = 0.5
• Test statistic: Number of heads
• Alpha level (designated threshold of significance): 0.05
• Observation O: 14 heads out of 20 flips; and
• Two-tailed p-value of observation O given H₀ = 2 × min(Pr(no. of heads ≥ 14 heads), Pr(no. of heads ≤ 14 heads)) = 2 × min(0.058, 0.978) = 2*0.058 = 0.115.

The Pr (no. of heads ≤ 14 heads) = 1 - Pr(no. of heads ≥ 14 heads) + Pr (no. of head = 14) = 1 - 0.058 + 0.036 = 0.978; however, the symmetry of this binomial distribution makes it an unnecessary computation to find the smaller of the two probabilities. Here, the calculated p-value exceeds .05, meaning that the data falls within the range of what would happen 95% of the time, if the coin were fair. Hence, the null hypothesis is not rejected at the .05 level.

However, had one more head been obtained, the resulting p-value (two-tailed) would have been 0.0414 (4.14%), in which case the null hypothesis would be rejected at the 0.05 level.