Problem

As a simple example, we can use a logistic regression with one explanatory variable and two categories to answer the following question:

A group of 20 students spends between 0 and 6 hours studying for an exam. How does the number of hours spent studying affect the probability of the student passing the exam?

The reason for using logistic regression for this problem is that the values of the dependent variable, pass and fail, while represented by "1" and "0", are not cardinal numbers. If the problem was changed so that pass/fail was replaced with the grade 0–100 (cardinal numbers), then simple regression analysis could be used.

The table shows the number of hours each student spent studying, and whether they passed (1) or failed (0).

We wish to fit a logistic function to the data consisting of the hours studied (x_k) and the outcome of the test (y_k =1 for pass, 0 for fail). The data points are indexed by the subscript k which runs from ${\displaystyle k=1}$ to ${\displaystyle k=K=20}$. The x variable is called the "explanatory variable", and the y variable is called the "categorical variable" consisting of two categories: "pass" or "fail" corresponding to the categorical values 1 and 0 respectively.

Model

Graph of a logistic regression curve fitted to the (x_m,y_m) data. The curve shows the probability of passing an exam versus hours studying.

The logistic function is of the form:

${\displaystyle p(x)={\frac {1}{1+e^{-(x-\mu )/s}}}}$

where μ is a location parameter (the midpoint of the curve, where ${\displaystyle p(\mu )=1/2}$) and s is a scale parameter. This expression may be rewritten as:

${\displaystyle p(x)={\frac {1}{1+e^{-(\beta _{0}+\beta _{1}x)}}}}$

where ${\displaystyle \beta _{0}=-\mu /s}$ and is known as the intercept (it is the vertical intercept or y-intercept of the line ${\displaystyle y=\beta _{0}+\beta _{1}x}$), and ${\displaystyle \beta _{1}=1/s}$ (inverse scale parameter or rate parameter): these are the y-intercept and slope of the log-odds as a function of x. Conversely, ${\displaystyle \mu =-\beta _{0}/\beta _{1}}$ and ${\displaystyle s=1/\beta _{1}}$.

Fit

The usual measure of goodness of fit for a logistic regression uses logistic loss (or log loss), the negative log-likelihood. For a given x_k and y_k, write ${\displaystyle p_{k}=p(x_{k})}$. The ${\displaystyle p_{k}}$ are the probabilities that the corresponding ${\displaystyle y_{k}}$ will equal one and ${\displaystyle 1-p_{k}}$ are the probabilities that they will be zero (see Bernoulli distribution). We wish to find the values of ${\displaystyle \beta _{0}}$ and ${\displaystyle \beta _{1}}$ which give the "best fit" to the data. In the case of linear regression, the sum of the squared deviations of the fit from the data points (y_k), the squared error loss, is taken as a measure of the goodness of fit, and the best fit is obtained when that function is minimized.

The log loss for the k-th point ${\displaystyle \ell _{k}}$ is:

${\displaystyle \ell _{k}={\begin{cases}-\ln p_{k}&{\text{ if }}y_{k}=1,\\-\ln(1-p_{k})&{\text{ if }}y_{k}=0.\end{cases}}}$

The log loss can be interpreted as the "surprisal" of the actual outcome ${\displaystyle y_{k}}$ relative to the prediction ${\displaystyle p_{k}}$, and is a measure of information content. Log loss is always greater than or equal to 0, equals 0 only in case of a perfect prediction (i.e., when ${\displaystyle p_{k}=1}$ and ${\displaystyle y_{k}=1}$, or ${\displaystyle p_{k}=0}$ and ${\displaystyle y_{k}=0}$), and approaches infinity as the prediction gets worse (i.e., when ${\displaystyle y_{k}=1}$ and ${\displaystyle p_{k}\to 0}$ or ${\displaystyle y_{k}=0}$ and ${\displaystyle p_{k}\to 1}$), meaning the actual outcome is "more surprising". Since the value of the logistic function is always strictly between zero and one, the log loss is always greater than zero and less than infinity. Unlike in a linear regression, where the model can have zero loss at a point by passing through a data point (and zero loss overall if all points are on a line), in a logistic regression it is not possible to have zero loss at any points, since ${\displaystyle y_{k}}$ is either 0 or 1, but ${\displaystyle 0<p_{k}<1}$.

These can be combined into a single expression:

${\displaystyle \ell _{k}=-y_{k}\ln p_{k}-(1-y_{k})\ln(1-p_{k}).}$

This expression is more formally known as the cross-entropy of the predicted distribution ${\displaystyle {\big (}p_{k},(1-p_{k}){\big )}}$ from the actual distribution ${\displaystyle {\big (}y_{k},(1-y_{k}){\big )}}$, as probability distributions on the two-element space of (pass, fail).

The sum of these, the total loss, is the overall negative log-likelihood ${\displaystyle -\ell }$, and the best fit is obtained for those choices of ${\displaystyle \beta _{0}}$ and ${\displaystyle \beta _{1}}$ for which ${\displaystyle -\ell }$ is minimized.

Alternatively, instead of minimizing the loss, one can maximize its inverse, the (positive) log-likelihood:

${\displaystyle \ell =\sum _{k:y_{k}=1}\ln(p_{k})+\sum _{k:y_{k}=0}\ln(1-p_{k})=\sum _{k=1}^{K}\left(\,y_{k}\ln(p_{k})+(1-y_{k})\ln(1-p_{k})\right)}$

or equivalently maximize the likelihood function itself, which is the probability that the given data set is produced by a particular logistic function:

${\displaystyle L=\prod _{k:y_{k}=1}p_{k}\,\prod _{k:y_{k}=0}(1-p_{k})}$

This method is known as maximum likelihood estimation.

Parameter estimation

Since ℓ is nonlinear in ${\displaystyle \beta _{0}}$ and ${\displaystyle \beta _{1}}$, determining their optimum values will require numerical methods. One method of maximizing ℓ is to require the derivatives of ℓ with respect to ${\displaystyle \beta _{0}}$ and ${\displaystyle \beta _{1}}$ to be zero:

${\displaystyle 0={\frac {\partial \ell }{\partial \beta _{0}}}=\sum _{k=1}^{K}(y_{k}-p_{k})}$

${\displaystyle 0={\frac {\partial \ell }{\partial \beta _{1}}}=\sum _{k=1}^{K}(y_{k}-p_{k})x_{k}}$

and the maximization procedure can be accomplished by solving the above two equations for ${\displaystyle \beta _{0}}$ and ${\displaystyle \beta _{1}}$, which, again, will generally require the use of numerical methods.

The values of ${\displaystyle \beta _{0}}$ and ${\displaystyle \beta _{1}}$ which maximize ℓ and L using the above data are found to be:

${\displaystyle \beta _{0}\approx -4.1}$

${\displaystyle \beta _{1}\approx 1.5}$

which yields a value for μ and s of:

${\displaystyle \mu =-\beta _{0}/\beta _{1}\approx 2.7}$

${\displaystyle s=1/\beta _{1}\approx 0.67}$

Predictions

The ${\displaystyle \beta _{0}}$ and ${\displaystyle \beta _{1}}$ coefficients may be entered into the logistic regression equation to estimate the probability of passing the exam.

For example, for a student who studies 2 hours, entering the value ${\displaystyle x=2}$ into the equation gives the estimated probability of passing the exam of 0.25:

${\displaystyle t=\beta _{0}+2\beta _{1}\approx -4.1+2\cdot 1.5=-1.1}$

${\displaystyle p={\frac {1}{1+e^{-t}}}\approx 0.25={\text{Probability of passing exam}}}$

Similarly, for a student who studies 4 hours, the estimated probability of passing the exam is 0.87:

${\displaystyle t=\beta _{0}+4\beta _{1}\approx -4.1+4\cdot 1.5=1.9}$

${\displaystyle p={\frac {1}{1+e^{-t}}}\approx 0.87={\text{Probability of passing exam}}}$

This table shows the estimated probability of passing the exam for several values of hours studying.

Hours of study (x)	Passing exam
	Log-odds (t)	Odds (et)	Probability (p)
1	−2.57	0.076 ≈ 1:13.1	0.07
2	−1.07	0.34 ≈ 1:2.91	0.26
${\displaystyle \mu \approx 2.7}$	0	1	${\displaystyle {\tfrac {1}{2}}}$ = 0.50
3	0.44	1.55	0.61
4	1.94	6.96	0.87
5	3.45	31.4	0.97

Model evaluation

The logistic regression analysis gives the following output.

By the Wald test, the output indicates that hours studying is significantly associated with the probability of passing the exam (${\displaystyle p=0.017}$). Rather than the Wald method, the recommended method^[17] to calculate the p-value for logistic regression is the likelihood-ratio test (LRT), which for these data give ${\displaystyle p\approx 0.00064}$ (see § Deviance and likelihood ratio tests below).

Generalizations

This simple model is an example of binary logistic regression, and has one explanatory variable and a binary categorical variable which can assume one of two categorical values. Multinomial logistic regression is the generalization of binary logistic regression to include any number of explanatory variables and any number of categories.

Definition of the logistic function

An explanation of logistic regression can begin with an explanation of the standard logistic function. The logistic function is a sigmoid function, which takes any real input ${\displaystyle t}$, and outputs a value between zero and one.^[2] For the logit, this is interpreted as taking input log-odds and having output probability. The standard logistic function ${\displaystyle \sigma$ :\mathbb {R} \rightarrow (0,1)} is defined as follows:

${\displaystyle \sigma (t)={\frac {e^{t}}{e^{t}+1}}={\frac {1}{1+e^{-t}}}}$

A graph of the logistic function on the t-interval (−6,6) is shown in Figure 1.

Let us assume that ${\displaystyle t}$ is a linear function of a single explanatory variable ${\displaystyle x}$ (the case where ${\displaystyle t}$ is a linear combination of multiple explanatory variables is treated similarly). We can then express ${\displaystyle t}$ as follows:

${\displaystyle t=\beta _{0}+\beta _{1}x}$

And the general logistic function ${\displaystyle p:\mathbb {R} \rightarrow (0,1)}$ can now be written as:

${\displaystyle p(x)=\sigma (t)={\frac {1}{1+e^{-(\beta _{0}+\beta _{1}x)}}}}$

In the logistic model, ${\displaystyle p(x)}$ is interpreted as the probability of the dependent variable ${\displaystyle Y}$ equaling a success/case rather than a failure/non-case. It is clear that the response variables ${\displaystyle Y_{i}}$ are not identically distributed: ${\displaystyle P(Y_{i}=1\mid X)}$ differs from one data point ${\displaystyle X_{i}}$ to another, though they are independent given design matrix ${\displaystyle X}$ and shared parameters ${\displaystyle \beta }$.^[11]

Definition of the inverse of the logistic function

We can now define the logit (log odds) function as the inverse ${\displaystyle g=\sigma ^{-1}}$ of the standard logistic function. It is easy to see that it satisfies:

${\displaystyle g(p(x))=\sigma ^{-1}(p(x))=\operatorname {logit} p(x)=\ln \left({\frac {p(x)}{1-p(x)}}\right)=\beta _{0}+\beta _{1}x,}$

and equivalently, after exponentiating both sides we have the odds:

${\displaystyle {\frac {p(x)}{1-p(x)}}=e^{\beta _{0}+\beta _{1}x}.}$

Interpretation of these terms

In the above equations, the terms are as follows:

• ${\displaystyle g}$ is the logit function. The equation for ${\displaystyle g(p(x))}$ illustrates that the logit (i.e., log-odds or natural logarithm of the odds) is equivalent to the linear regression expression.
• ${\displaystyle \ln }$ denotes the natural logarithm.
• ${\displaystyle p(x)}$ is the probability that the dependent variable equals a case, given some linear combination of the predictors. The formula for ${\displaystyle p(x)}$ illustrates that the probability of the dependent variable equaling a case is equal to the value of the logistic function of the linear regression expression. This is important in that it shows that the value of the linear regression expression can vary from negative to positive infinity and yet, after transformation, the resulting expression for the probability ${\displaystyle p(x)}$ ranges between 0 and 1.
• ${\displaystyle \beta _{0}}$ is the intercept from the linear regression equation (the value of the criterion when the predictor is equal to zero).
• ${\displaystyle \beta _{1}x}$ is the regression coefficient multiplied by some value of the predictor.
• base ${\displaystyle e}$ denotes the exponential function.

Definition of the odds

The odds of the dependent variable equaling a case (given some linear combination ${\displaystyle x}$ of the predictors) is equivalent to the exponential function of the linear regression expression. This illustrates how the logit serves as a link function between the probability and the linear regression expression. Given that the logit ranges between negative and positive infinity, it provides an adequate criterion upon which to conduct linear regression and the logit is easily converted back into the odds.^[2]

So we define odds of the dependent variable equaling a case (given some linear combination ${\displaystyle x}$ of the predictors) as follows:

${\displaystyle {\text{odds}}=e^{\beta _{0}+\beta _{1}x}.}$

The odds ratio

For a continuous independent variable the odds ratio can be defined as:

The image represents an outline of what an odds ratio looks like in writing, through a template in addition to the test score example in the "Example" section of the contents. In simple terms, if we hypothetically get an odds ratio of 2 to 1, we can say... "For every one-unit increase in hours studied, the odds of passing (group 1) or failing (group 0) are (expectedly) 2 to 1 (Denis, 2019).

${\displaystyle \mathrm {OR} ={\frac {\operatorname {odds} (x+1)}{\operatorname {odds} (x)}}={\frac {\left({\frac {p(x+1)}{1-p(x+1)}}\right)}{\left({\frac {p(x)}{1-p(x)}}\right)}}={\frac {e^{\beta _{0}+\beta _{1}(x+1)}}{e^{\beta _{0}+\beta _{1}x}}}=e^{\beta _{1}}}$

This exponential relationship provides an interpretation for ${\displaystyle \beta _{1}}$: The odds multiply by ${\displaystyle e^{\beta _{1}}}$ for every 1-unit increase in x.^[18]

For a binary independent variable the odds ratio is defined as ${\displaystyle {\frac {ad}{bc}}}$ where a, b, c and d are cells in a 2×2 contingency table.^[19]

Multiple explanatory variables

If there are multiple explanatory variables, the above expression ${\displaystyle \beta _{0}+\beta _{1}x}$ can be revised to ${\displaystyle \beta _{0}+\beta _{1}x_{1}+\beta _{2}x_{2}+\cdots +\beta _{m}x_{m}=\beta _{0}+\sum _{i=1}^{m}\beta _{i}x_{i}}$. Then when this is used in the equation relating the log odds of a success to the values of the predictors, the linear regression will be a multiple regression with m explanators; the parameters ${\displaystyle \beta _{j}}$ for all ${\displaystyle j=0,1,2,\dots ,m}$ are all estimated.

Again, the more traditional equations are:

${\displaystyle \log {\frac {p}{1-p}}=\beta _{0}+\beta _{1}x_{1}+\beta _{2}x_{2}+\cdots +\beta _{m}x_{m}}$

and

${\displaystyle p={\frac {1}{1+b^{-(\beta _{0}+\beta _{1}x_{1}+\beta _{2}x_{2}+\cdots +\beta _{m}x_{m})}}}}$

where usually ${\displaystyle b=e}$.

Many explanatory variables, two categories

The above example of binary logistic regression on one explanatory variable can be generalized to binary logistic regression on any number of explanatory variables x₁, x₂,... and any number of categorical values ${\displaystyle y=0,1,2,\dots }$.

To begin with, we may consider a logistic model with M explanatory variables, x₁, x₂ ... x_M and, as in the example above, two categorical values (y = 0 and 1). For the simple binary logistic regression model, we assumed a linear relationship between the predictor variable and the log-odds (also called logit) of the event that ${\displaystyle y=1}$. This linear relationship may be extended to the case of M explanatory variables:

${\displaystyle t=\log _{b}{\frac {p}{1-p}}=\beta _{0}+\beta _{1}x_{1}+\beta _{2}x_{2}+\cdots +\beta _{M}x_{M}}$

where t is the log-odds and ${\displaystyle \beta _{i}}$ are parameters of the model. An additional generalization has been introduced in which the base of the model (b) is not restricted to the Euler number e. In most applications, the base ${\displaystyle b}$ of the logarithm is usually taken to be e. However, in some cases it can be easier to communicate results by working in base 2 or base 10.

For a more compact notation, we will specify the explanatory variables and the β coefficients as ${\displaystyle (M+1)}$-dimensional vectors:

${\displaystyle {\boldsymbol {x}}=\{x_{0},x_{1},x_{2},\dots ,x_{M}\}}$

${\displaystyle {\boldsymbol {\beta }}=\{\beta _{0},\beta _{1},\beta _{2},\dots ,\beta _{M}\}}$

with an added explanatory variable x₀ =1. The logit may now be written as:

${\displaystyle t=\sum _{m=0}^{M}\beta _{m}x_{m}={\boldsymbol {\beta }}\cdot x}$

Solving for the probability p that ${\displaystyle y=1}$ yields:

${\displaystyle p({\boldsymbol {x}})={\frac {b^{{\boldsymbol {\beta }}\cdot {\boldsymbol {x}}}}{1+b^{{\boldsymbol {\beta }}\cdot {\boldsymbol {x}}}}}={\frac {1}{1+b^{-{\boldsymbol {\beta }}\cdot {\boldsymbol {x}}}}}=S_{b}(t)}$,

where ${\displaystyle S_{b}}$ is the sigmoid function with base ${\displaystyle b}$. The above formula shows that once the ${\displaystyle \beta _{m}}$ are fixed, we can easily compute either the log-odds that ${\displaystyle y=1}$ for a given observation, or the probability that ${\displaystyle y=1}$ for a given observation. The main use-case of a logistic model is to be given an observation ${\displaystyle {\boldsymbol {x}}}$, and estimate the probability ${\displaystyle p({\boldsymbol {x}})}$ that ${\displaystyle y=1}$. The optimum beta coefficients may again be found by maximizing the log-likelihood. For K measurements, defining ${\displaystyle {\boldsymbol {x}}_{k}}$ as the explanatory vector of the k-th measurement, and ${\displaystyle y_{k}}$ as the categorical outcome of that measurement, the log likelihood may be written in a form very similar to the simple ${\displaystyle M=1}$ case above:

${\displaystyle \ell =\sum _{k=1}^{K}y_{k}\log _{b}(p({\boldsymbol {x_{k}}}))+\sum _{k=1}^{K}(1-y_{k})\log _{b}(1-p({\boldsymbol {x_{k}}}))}$

As in the simple example above, finding the optimum β parameters will require numerical methods. One useful technique is to equate the derivatives of the log likelihood with respect to each of the β parameters to zero yielding a set of equations which will hold at the maximum of the log likelihood:

${\displaystyle {\frac {\partial \ell }{\partial \beta _{m}}}=0=\sum _{k=1}^{K}y_{k}x_{mk}-\sum _{k=1}^{K}p({\boldsymbol {x}}_{k})x_{mk}}$

where x_mk is the value of the x_m explanatory variable from the k-th measurement.

Consider an example with ${\displaystyle M=2}$ explanatory variables, ${\displaystyle b=10}$, and coefficients ${\displaystyle \beta _{0}=-3}$, ${\displaystyle \beta _{1}=1}$, and ${\displaystyle \beta _{2}=2}$ which have been determined by the above method. To be concrete, the model is:

${\displaystyle t=\log _{10}{\frac {p}{1-p}}=-3+x_{1}+2x_{2}}$

${\displaystyle p={\frac {b^{{\boldsymbol {\beta }}\cdot {\boldsymbol {x}}}}{1+b^{{\boldsymbol {\beta }}\cdot x}}}={\frac {b^{\beta _{0}+\beta _{1}x_{1}+\beta _{2}x_{2}}}{1+b^{\beta _{0}+\beta _{1}x_{1}+\beta _{2}x_{2}}}}={\frac {1}{1+b^{-(\beta _{0}+\beta _{1}x_{1}+\beta _{2}x_{2})}}}}$,

where p is the probability of the event that ${\displaystyle y=1}$. This can be interpreted as follows:

• ${\displaystyle \beta _{0}=-3}$ is the y-intercept. It is the log-odds of the event that ${\displaystyle y=1}$, when the predictors ${\displaystyle x_{1}=x_{2}=0}$. By exponentiating, we can see that when ${\displaystyle x_{1}=x_{2}=0}$ the odds of the event that ${\displaystyle y=1}$ are 1-to-1000, or ${\displaystyle 10^{-3}}$. Similarly, the probability of the event that ${\displaystyle y=1}$ when ${\displaystyle x_{1}=x_{2}=0}$ can be computed as ${\displaystyle 1/(1000+1)=1/1001.}$
• ${\displaystyle \beta _{1}=1}$ means that increasing ${\displaystyle x_{1}}$ by 1 increases the log-odds by ${\displaystyle 1}$. So if ${\displaystyle x_{1}}$ increases by 1, the odds that ${\displaystyle y=1}$ increase by a factor of ${\displaystyle 10^{1}}$. The probability of ${\displaystyle y=1}$ has also increased, but it has not increased by as much as the odds have increased.
• ${\displaystyle \beta _{2}=2}$ means that increasing ${\displaystyle x_{2}}$ by 1 increases the log-odds by ${\displaystyle 2}$. So if ${\displaystyle x_{2}}$ increases by 1, the odds that ${\displaystyle y=1}$ increase by a factor of ${\displaystyle 10^{2}.}$ Note how the effect of ${\displaystyle x_{2}}$ on the log-odds is twice as great as the effect of ${\displaystyle x_{1}}$, but the effect on the odds is 10 times greater. But the effect on the probability of ${\displaystyle y=1}$ is not as much as 10 times greater, it's only the effect on the odds that is 10 times greater.

Multinomial logistic regression: Many explanatory variables and many categories

In the above cases of two categories (binomial logistic regression), the categories were indexed by "0" and "1", and we had two probabilities: The probability that the outcome was in category 1 was given by ${\displaystyle p({\boldsymbol {x}})}$and the probability that the outcome was in category 0 was given by ${\displaystyle 1-p({\boldsymbol {x}})}$. The sum of these probabilities equals 1, which must be true, since "0" and "1" are the only possible categories in this setup.

In general, if we have ${\displaystyle M+1}$ explanatory variables (including x₀) and ${\displaystyle N+1}$ categories, we will need ${\displaystyle N+1}$ separate probabilities, one for each category, indexed by n, which describe the probability that the categorical outcome y will be in category y=n, conditional on the vector of covariates x. The sum of these probabilities over all categories must equal 1. Using the mathematically convenient base e, these probabilities are:

${\displaystyle p_{n}({\boldsymbol {x}})={\frac {e^{{\boldsymbol {\beta }}_{n}\cdot {\boldsymbol {x}}}}{1+\sum _{u=1}^{N}e^{{\boldsymbol {\beta }}_{u}\cdot {\boldsymbol {x}}}}}}$ for ${\displaystyle n=1,2,\dots ,N}$

${\displaystyle p_{0}({\boldsymbol {x}})=1-\sum _{n=1}^{N}p_{n}({\boldsymbol {x}})={\frac {1}{1+\sum _{u=1}^{N}e^{{\boldsymbol {\beta }}_{u}\cdot {\boldsymbol {x}}}}}}$

Each of the probabilities except ${\displaystyle p_{0}({\boldsymbol {x}})}$ will have their own set of regression coefficients ${\displaystyle {\boldsymbol {\beta }}_{n}}$. It can be seen that, as required, the sum of the ${\displaystyle p_{n}({\boldsymbol {x}})}$ over all categories n is 1. The selection of ${\displaystyle p_{0}({\boldsymbol {x}})}$ to be defined in terms of the other probabilities is artificial. Any of the probabilities could have been selected to be so defined. This special value of n is termed the "pivot index", and the log-odds (t_n) are expressed in terms of the pivot probability and are again expressed as a linear combination of the explanatory variables:

${\displaystyle t_{n}=\ln \left({\frac {p_{n}({\boldsymbol {x}})}{p_{0}({\boldsymbol {x}})}}\right)={\boldsymbol {\beta }}_{n}\cdot {\boldsymbol {x}}}$

Note also that for the simple case of ${\displaystyle N=1}$, the two-category case is recovered, with ${\displaystyle p({\boldsymbol {x}})=p_{1}({\boldsymbol {x}})}$ and ${\displaystyle p_{0}({\boldsymbol {x}})=1-p_{1}({\boldsymbol {x}})}$.

The log-likelihood that a particular set of K measurements or data points will be generated by the above probabilities can now be calculated. Indexing each measurement by k, let the k-th set of measured explanatory variables be denoted by ${\displaystyle {\boldsymbol {x}}_{k}}$ and their categorical outcomes be denoted by ${\displaystyle y_{k}}$ which can be equal to any integer in [0,N]. The log-likelihood is then:

${\displaystyle \ell =\sum _{k=1}^{K}\sum _{n=0}^{N}\Delta (n,y_{k})\,\ln(p_{n}({\boldsymbol {x}}_{k}))}$

where ${\displaystyle \Delta (n,y_{k})}$ is an indicator function which equals 1 if y_k = n and zero otherwise. In the case of two explanatory variables, this indicator function was defined as y_k when n = 1 and 1-y_k when n = 0. This was convenient, but not necessary.^[20] Again, the optimum beta coefficients may be found by maximizing the log-likelihood function generally using numerical methods. A possible method of solution is to set the derivatives of the log-likelihood with respect to each beta coefficient equal to zero and solve for the beta coefficients:

${\displaystyle {\frac {\partial \ell }{\partial \beta _{nm}}}=0=\sum _{k=1}^{K}\Delta (n,y_{k})x_{mk}-\sum _{k=1}^{K}p_{n}({\boldsymbol {x}}_{k})x_{mk}}$

where ${\displaystyle \beta _{nm}}$ is the m-th coefficient of the ${\displaystyle {\boldsymbol {\beta }}_{n}}$ vector and ${\displaystyle x_{mk}}$ is the m-th explanatory variable of the k-th measurement. Once the beta coefficients have been estimated from the data, we will be able to estimate the probability that any subsequent set of explanatory variables will result in any of the possible outcome categories.

As a generalized linear model

The particular model used by logistic regression, which distinguishes it from standard linear regression and from other types of regression analysis used for binary-valued outcomes, is the way the probability of a particular outcome is linked to the linear predictor function:

${\displaystyle \operatorname {logit} (\operatorname {\mathbb {E} } [Y_{i}\mid x_{1,i},\ldots ,x_{m,i}])=\operatorname {logit} (p_{i})=\ln \left({\frac {p_{i}}{1-p_{i}}}\right)=\beta _{0}+\beta _{1}x_{1,i}+\cdots +\beta _{m}x_{m,i}}$

Written using the more compact notation described above, this is:

${\displaystyle \operatorname {logit} (\operatorname {\mathbb {E} } [Y_{i}\mid \mathbf {X} _{i}])=\operatorname {logit} (p_{i})=\ln \left({\frac {p_{i}}{1-p_{i}}}\right)={\boldsymbol {\beta }}\cdot \mathbf {X} _{i}}$

This formulation expresses logistic regression as a type of generalized linear model, which predicts variables with various types of probability distributions by fitting a linear predictor function of the above form to some sort of arbitrary transformation of the expected value of the variable.

The intuition for transforming using the logit function (the natural log of the odds) was explained above. It also has the practical effect of converting the probability (which is bounded to be between 0 and 1) to a variable that ranges over ${\displaystyle (-\infty ,+\infty )}$ — thereby matching the potential range of the linear prediction function on the right side of the equation.

Both the probabilities p_i and the regression coefficients are unobserved, and the means of determining them is not part of the model itself. They are typically determined by some sort of optimization procedure, e.g. maximum likelihood estimation, that finds values that best fit the observed data (i.e. that give the most accurate predictions for the data already observed), usually subject to regularization conditions that seek to exclude unlikely values, e.g. extremely large values for any of the regression coefficients. The use of a regularization condition is equivalent to doing maximum a posteriori (MAP) estimation, an extension of maximum likelihood. (Regularization is most commonly done using a squared regularizing function, which is equivalent to placing a zero-mean Gaussian prior distribution on the coefficients, but other regularizers are also possible.) Whether or not regularization is used, it is usually not possible to find a closed-form solution; instead, an iterative numerical method must be used, such as iteratively reweighted least squares (IRLS) or, more commonly these days, a quasi-Newton method such as the L-BFGS method.^[21]

The interpretation of the β_j parameter estimates is as the additive effect on the log of the odds for a unit change in the j the explanatory variable. In the case of a dichotomous explanatory variable, for instance, gender ${\displaystyle e^{\beta }}$ is the estimate of the odds of having the outcome for, say, males compared with females.

An equivalent formula uses the inverse of the logit function, which is the logistic function, i.e.:

${\displaystyle \operatorname {\mathbb {E} } [Y_{i}\mid \mathbf {X} _{i}]=p_{i}=\operatorname {logit} ^{-1}({\boldsymbol {\beta }}\cdot \mathbf {X} _{i})={\frac {1}{1+e^{-{\boldsymbol {\beta }}\cdot \mathbf {X} _{i}}}}}$

The formula can also be written as a probability distribution (specifically, using a probability mass function):

${\displaystyle \Pr(Y_{i}=y\mid \mathbf {X} _{i})={p_{i}}^{y}(1-p_{i})^{1-y}=\left({\frac {e^{{\boldsymbol {\beta }}\cdot \mathbf {X} _{i}}}{1+e^{{\boldsymbol {\beta }}\cdot \mathbf {X} _{i}}}}\right)^{y}\left(1-{\frac {e^{{\boldsymbol {\beta }}\cdot \mathbf {X} _{i}}}{1+e^{{\boldsymbol {\beta }}\cdot \mathbf {X} _{i}}}}\right)^{1-y}={\frac {e^{{\boldsymbol {\beta }}\cdot \mathbf {X} _{i}\cdot y}}{1+e^{{\boldsymbol {\beta }}\cdot \mathbf {X} _{i}}}}}$

As a latent-variable model

The logistic model has an equivalent formulation as a latent-variable model. This formulation is common in the theory of discrete choice models and makes it easier to extend to certain more complicated models with multiple, correlated choices, as well as to compare logistic regression to the closely related probit model.

Imagine that, for each trial i, there is a continuous latent variable Y_i^* (i.e. an unobserved random variable) that is distributed as follows:

${\displaystyle Y_{i}^{\ast }={\boldsymbol {\beta }}\cdot \mathbf {X} _{i}+\varepsilon _{i}\,}$

where

${\displaystyle \varepsilon _{i}\sim \operatorname {Logistic} (0,1)\,}$

i.e. the latent variable can be written directly in terms of the linear predictor function and an additive random error variable that is distributed according to a standard logistic distribution.

Then Y_i can be viewed as an indicator for whether this latent variable is positive:

${\displaystyle Y_{i}={\begin{cases}1&{\text{if }}Y_{i}^{\ast }>0\ {\text{ i.e. }}-\varepsilon _{i}<{\boldsymbol {\beta }}\cdot \mathbf {X} _{i},\\0&{\text{otherwise.}}\end{cases}}}$

The choice of modeling the error variable specifically with a standard logistic distribution, rather than a general logistic distribution with the location and scale set to arbitrary values, seems restrictive, but in fact, it is not. It must be kept in mind that we can choose the regression coefficients ourselves, and very often can use them to offset changes in the parameters of the error variable's distribution. For example, a logistic error-variable distribution with a non-zero location parameter μ (which sets the mean) is equivalent to a distribution with a zero location parameter, where μ has been added to the intercept coefficient. Both situations produce the same value for Y_i^* regardless of settings of explanatory variables. Similarly, an arbitrary scale parameter s is equivalent to setting the scale parameter to 1 and then dividing all regression coefficients by s. In the latter case, the resulting value of Y_i^* will be smaller by a factor of s than in the former case, for all sets of explanatory variables — but critically, it will always remain on the same side of 0, and hence lead to the same Y_i choice.

(This predicts that the irrelevancy of the scale parameter may not carry over into more complex models where more than two choices are available.)

It turns out that this formulation is exactly equivalent to the preceding one, phrased in terms of the generalized linear model and without any latent variables. This can be shown as follows, using the fact that the cumulative distribution function (CDF) of the standard logistic distribution is the logistic function, which is the inverse of the logit function, i.e.

${\displaystyle \Pr(\varepsilon _{i}<x)=\operatorname {logit} ^{-1}(x)}$

Then:

${\displaystyle {\begin{aligned}\Pr(Y_{i}=1\mid \mathbf {X} _{i})&=\Pr(Y_{i}^{\ast }>0\mid \mathbf {X} _{i})\\[5pt]&=\Pr({\boldsymbol {\beta }}\cdot \mathbf {X} _{i}+\varepsilon _{i}>0)\\[5pt]&=\Pr(\varepsilon _{i}>-{\boldsymbol {\beta }}\cdot \mathbf {X} _{i})\\[5pt]&=\Pr(\varepsilon _{i}<{\boldsymbol {\beta }}\cdot \mathbf {X} _{i})&&{\text{(because the logistic distribution is symmetric)}}\\[5pt]&=\operatorname {logit} ^{-1}({\boldsymbol {\beta }}\cdot \mathbf {X} _{i})&\\[5pt]&=p_{i}&&{\text{(see above)}}\end{aligned}}}$

This formulation—which is standard in discrete choice models—makes clear the relationship between logistic regression (the "logit model") and the probit model, which uses an error variable distributed according to a standard normal distribution instead of a standard logistic distribution. Both the logistic and normal distributions are symmetric with a basic unimodal, "bell curve" shape. The only difference is that the logistic distribution has somewhat heavier tails, which means that it is less sensitive to outlying data (and hence somewhat more robust to model mis-specifications or erroneous data).

Two-way latent-variable model

Yet another formulation uses two separate latent variables:

${\displaystyle {\begin{aligned}Y_{i}^{0\ast }&={\boldsymbol {\beta }}_{0}\cdot \mathbf {X} _{i}+\varepsilon _{0}\,\\Y_{i}^{1\ast }&={\boldsymbol {\beta }}_{1}\cdot \mathbf {X} _{i}+\varepsilon _{1}\,\end{aligned}}}$

where

${\displaystyle {\begin{aligned}\varepsilon _{0}&\sim \operatorname {EV} _{1}(0,1)\\\varepsilon _{1}&\sim \operatorname {EV} _{1}(0,1)\end{aligned}}}$

where EV₁(0,1) is a standard type-1 extreme value distribution: i.e.

${\displaystyle \Pr(\varepsilon _{0}=x)=\Pr(\varepsilon _{1}=x)=e^{-x}e^{-e^{-x}}}$

Then

${\displaystyle Y_{i}={\begin{cases}1&{\text{if }}Y_{i}^{1\ast }>Y_{i}^{0\ast },\\0&{\text{otherwise.}}\end{cases}}}$

This model has a separate latent variable and a separate set of regression coefficients for each possible outcome of the dependent variable. The reason for this separation is that it makes it easy to extend logistic regression to multi-outcome categorical variables, as in the multinomial logit model. In such a model, it is natural to model each possible outcome using a different set of regression coefficients. It is also possible to motivate each of the separate latent variables as the theoretical utility associated with making the associated choice, and thus motivate logistic regression in terms of utility theory. (In terms of utility theory, a rational actor always chooses the choice with the greatest associated utility.) This is the approach taken by economists when formulating discrete choice models, because it both provides a theoretically strong foundation and facilitates intuitions about the model, which in turn makes it easy to consider various sorts of extensions. (See the example below.)

The choice of the type-1 extreme value distribution seems fairly arbitrary, but it makes the mathematics work out, and it may be possible to justify its use through rational choice theory.

It turns out that this model is equivalent to the previous model, although this seems non-obvious, since there are now two sets of regression coefficients and error variables, and the error variables have a different distribution. In fact, this model reduces directly to the previous one with the following substitutions:

${\displaystyle {\boldsymbol {\beta }}={\boldsymbol {\beta }}_{1}-{\boldsymbol {\beta }}_{0}}$

${\displaystyle \varepsilon =\varepsilon _{1}-\varepsilon _{0}}$

An intuition for this comes from the fact that, since we choose based on the maximum of two values, only their difference matters, not the exact values — and this effectively removes one degree of freedom. Another critical fact is that the difference of two type-1 extreme-value-distributed variables is a logistic distribution, i.e. ${\displaystyle \varepsilon =\varepsilon _{1}-\varepsilon _{0}\sim \operatorname {Logistic} (0,1).}$ We can demonstrate the equivalent as follows:

${\displaystyle {\begin{aligned}\Pr(Y_{i}=1\mid \mathbf {X} _{i})={}&\Pr \left(Y_{i}^{1\ast }>Y_{i}^{0\ast }\mid \mathbf {X} _{i}\right)&\\[5pt]={}&\Pr \left(Y_{i}^{1\ast }-Y_{i}^{0\ast }>0\mid \mathbf {X} _{i}\right)&\\[5pt]={}&\Pr \left({\boldsymbol {\beta }}_{1}\cdot \mathbf {X} _{i}+\varepsilon _{1}-\left({\boldsymbol {\beta }}_{0}\cdot \mathbf {X} _{i}+\varepsilon _{0}\right)>0\right)&\\[5pt]={}&\Pr \left(({\boldsymbol {\beta }}_{1}\cdot \mathbf {X} _{i}-{\boldsymbol {\beta }}_{0}\cdot \mathbf {X} _{i})+(\varepsilon _{1}-\varepsilon _{0})>0\right)&\\[5pt]={}&\Pr(({\boldsymbol {\beta }}_{1}-{\boldsymbol {\beta }}_{0})\cdot \mathbf {X} _{i}+(\varepsilon _{1}-\varepsilon _{0})>0)&\\[5pt]={}&\Pr(({\boldsymbol {\beta }}_{1}-{\boldsymbol {\beta }}_{0})\cdot \mathbf {X} _{i}+\varepsilon >0)&&{\text{(substitute }}\varepsilon {\text{ as above)}}\\[5pt]={}&\Pr({\boldsymbol {\beta }}\cdot \mathbf {X} _{i}+\varepsilon >0)&&{\text{(substitute }}{\boldsymbol {\beta }}{\text{ as above)}}\\[5pt]={}&\Pr(\varepsilon >-{\boldsymbol {\beta }}\cdot \mathbf {X} _{i})&&{\text{(now, same as above model)}}\\[5pt]={}&\Pr(\varepsilon <{\boldsymbol {\beta }}\cdot \mathbf {X} _{i})&\\[5pt]={}&\operatorname {logit} ^{-1}({\boldsymbol {\beta }}\cdot \mathbf {X} _{i})\\[5pt]={}&p_{i}\end{aligned}}}$

As a "log-linear" model

Yet another formulation combines the two-way latent variable formulation above with the original formulation higher up without latent variables, and in the process provides a link to one of the standard formulations of the multinomial logit.

Here, instead of writing the logit of the probabilities p_i as a linear predictor, we separate the linear predictor into two, one for each of the two outcomes:

${\displaystyle {\begin{aligned}\ln \Pr(Y_{i}=0)&={\boldsymbol {\beta }}_{0}\cdot \mathbf {X} _{i}-\ln Z\\\ln \Pr(Y_{i}=1)&={\boldsymbol {\beta }}_{1}\cdot \mathbf {X} _{i}-\ln Z\end{aligned}}}$

Two separate sets of regression coefficients have been introduced, just as in the two-way latent variable model, and the two equations appear a form that writes the logarithm of the associated probability as a linear predictor, with an extra term ${\displaystyle -\ln Z}$ at the end. This term, as it turns out, serves as the normalizing factor ensuring that the result is a distribution. This can be seen by exponentiating both sides:

${\displaystyle {\begin{aligned}\Pr(Y_{i}=0)&={\frac {1}{Z}}e^{{\boldsymbol {\beta }}_{0}\cdot \mathbf {X} _{i}}\\[5pt]\Pr(Y_{i}=1)&={\frac {1}{Z}}e^{{\boldsymbol {\beta }}_{1}\cdot \mathbf {X} _{i}}\end{aligned}}}$

In this form it is clear that the purpose of Z is to ensure that the resulting distribution over Y_i is in fact a probability distribution, i.e. it sums to 1. This means that Z is simply the sum of all un-normalized probabilities, and by dividing each probability by Z, the probabilities become "normalized". That is:

${\displaystyle Z=e^{{\boldsymbol {\beta }}_{0}\cdot \mathbf {X} _{i}}+e^{{\boldsymbol {\beta }}_{1}\cdot \mathbf {X} _{i}}}$

and the resulting equations are

${\displaystyle {\begin{aligned}\Pr(Y_{i}=0)&={\frac {e^{{\boldsymbol {\beta }}_{0}\cdot \mathbf {X} _{i}}}{e^{{\boldsymbol {\beta }}_{0}\cdot \mathbf {X} _{i}}+e^{{\boldsymbol {\beta }}_{1}\cdot \mathbf {X} _{i}}}}\\[5pt]\Pr(Y_{i}=1)&={\frac {e^{{\boldsymbol {\beta }}_{1}\cdot \mathbf {X} _{i}}}{e^{{\boldsymbol {\beta }}_{0}\cdot \mathbf {X} _{i}}+e^{{\boldsymbol {\beta }}_{1}\cdot \mathbf {X} _{i}}}}.\end{aligned}}}$

Or generally:

${\displaystyle \Pr(Y_{i}=c)={\frac {e^{{\boldsymbol {\beta }}_{c}\cdot \mathbf {X} _{i}}}{\sum _{h}e^{{\boldsymbol {\beta }}_{h}\cdot \mathbf {X} _{i}}}}}$

This shows clearly how to generalize this formulation to more than two outcomes, as in multinomial logit. This general formulation is exactly the softmax function as in

${\displaystyle \Pr(Y_{i}=c)=\operatorname {softmax} (c,{\boldsymbol {\beta }}_{0}\cdot \mathbf {X} _{i},{\boldsymbol {\beta }}_{1}\cdot \mathbf {X} _{i},\dots ).}$

In order to prove that this is equivalent to the previous model, the above model is overspecified, in that ${\displaystyle \Pr(Y_{i}=0)}$ and ${\displaystyle \Pr(Y_{i}=1)}$ cannot be independently specified: rather ${\displaystyle \Pr(Y_{i}=0)+\Pr(Y_{i}=1)=1}$ so knowing one automatically determines the other. As a result, the model is nonidentifiable, in that multiple combinations of β₀ and β₁ will produce the same probabilities for all possible explanatory variables. In fact, it can be seen that adding any constant vector to both of them will produce the same probabilities:

${\displaystyle {\begin{aligned}\Pr(Y_{i}=1)&={\frac {e^{({\boldsymbol {\beta }}_{1}+\mathbf {C} )\cdot \mathbf {X} _{i}}}{e^{({\boldsymbol {\beta }}_{0}+\mathbf {C} )\cdot \mathbf {X} _{i}}+e^{({\boldsymbol {\beta }}_{1}+\mathbf {C} )\cdot \mathbf {X} _{i}}}}\\[5pt]&={\frac {e^{{\boldsymbol {\beta }}_{1}\cdot \mathbf {X} _{i}}e^{\mathbf {C} \cdot \mathbf {X} _{i}}}{e^{{\boldsymbol {\beta }}_{0}\cdot \mathbf {X} _{i}}e^{\mathbf {C} \cdot \mathbf {X} _{i}}+e^{{\boldsymbol {\beta }}_{1}\cdot \mathbf {X} _{i}}e^{\mathbf {C} \cdot \mathbf {X} _{i}}}}\\[5pt]&={\frac {e^{\mathbf {C} \cdot \mathbf {X} _{i}}e^{{\boldsymbol {\beta }}_{1}\cdot \mathbf {X} _{i}}}{e^{\mathbf {C} \cdot \mathbf {X} _{i}}(e^{{\boldsymbol {\beta }}_{0}\cdot \mathbf {X} _{i}}+e^{{\boldsymbol {\beta }}_{1}\cdot \mathbf {X} _{i}})}}\\[5pt]&={\frac {e^{{\boldsymbol {\beta }}_{1}\cdot \mathbf {X} _{i}}}{e^{{\boldsymbol {\beta }}_{0}\cdot \mathbf {X} _{i}}+e^{{\boldsymbol {\beta }}_{1}\cdot \mathbf {X} _{i}}}}.\end{aligned}}}$

As a result, we can simplify matters, and restore identifiability, by picking an arbitrary value for one of the two vectors. We choose to set ${\displaystyle {\boldsymbol {\beta }}_{0}=\mathbf {0} .}$ Then,

${\displaystyle e^{{\boldsymbol {\beta }}_{0}\cdot \mathbf {X} _{i}}=e^{\mathbf {0} \cdot \mathbf {X} _{i}}=1}$

and so

${\displaystyle \Pr(Y_{i}=1)={\frac {e^{{\boldsymbol {\beta }}_{1}\cdot \mathbf {X} _{i}}}{1+e^{{\boldsymbol {\beta }}_{1}\cdot \mathbf {X} _{i}}}}={\frac {1}{1+e^{-{\boldsymbol {\beta }}_{1}\cdot \mathbf {X} _{i}}}}=p_{i}}$

which shows that this formulation is indeed equivalent to the previous formulation. (As in the two-way latent variable formulation, any settings where ${\displaystyle {\boldsymbol {\beta }}={\boldsymbol {\beta }}_{1}-{\boldsymbol {\beta }}_{0}}$ will produce equivalent results.)

Most treatments of the multinomial logit model start out either by extending the "log-linear" formulation presented here or the two-way latent variable formulation presented above, since both clearly show the way that the model could be extended to multi-way outcomes. In general, the presentation with latent variables is more common in econometrics and political science, where discrete choice models and utility theory reign, while the "log-linear" formulation here is more common in computer science, e.g. machine learning and natural language processing.

As a single-layer perceptron

The model has an equivalent formulation

${\displaystyle p_{i}={\frac {1}{1+e^{-(\beta _{0}+\beta _{1}x_{1,i}+\cdots +\beta _{k}x_{k,i})}}}.\,}$

This functional form is commonly called a single-layer perceptron or single-layer artificial neural network. A single-layer neural network computes a continuous output instead of a step function. The derivative of p_i with respect to X = (x₁, ..., x_k) is computed from the general form:

${\displaystyle y={\frac {1}{1+e^{-f(X)}}}}$

where f(X) is an analytic function in X. With this choice, the single-layer neural network is identical to the logistic regression model. This function has a continuous derivative, which allows it to be used in backpropagation. This function is also preferred because its derivative is easily calculated:

${\displaystyle {\frac {\mathrm {d} y}{\mathrm {d} X}}=y(1-y){\frac {\mathrm {d} f}{\mathrm {d} X}}.\,}$

In terms of binomial data

A closely related model assumes that each i is associated not with a single Bernoulli trial but with n_i independent identically distributed trials, where the observation Y_i is the number of successes observed (the sum of the individual Bernoulli-distributed random variables), and hence follows a binomial distribution:

${\displaystyle Y_{i}\,\sim \operatorname {Bin} (n_{i},p_{i}),{\text{ for }}i=1,\dots ,n}$

An example of this distribution is the fraction of seeds (p_i) that germinate after n_i are planted.

In terms of expected values, this model is expressed as follows:

${\displaystyle p_{i}=\operatorname {\mathbb {E} } \left[\left.{\frac {Y_{i}}{n_{i}}}\,\right|\,\mathbf {X} _{i}\right]\,,}$

so that

${\displaystyle \operatorname {logit} \left(\operatorname {\mathbb {E} } \left[\left.{\frac {Y_{i}}{n_{i}}}\,\right|\,\mathbf {X} _{i}\right]\right)=\operatorname {logit} (p_{i})=\ln \left({\frac {p_{i}}{1-p_{i}}}\right)={\boldsymbol {\beta }}\cdot \mathbf {X} _{i}\,,}$