Binary Logistic Regression

January 3, 2026

On this page

Logistic Regression: A Fundamental Classification Method

For a binary response, we want to model a probability between zero and one. A linear predictor does not enforce these bounds. Logistic regression instead models the log-odds as a linear function and transforms the result into a probability. We derive its likelihood, gradient, and Hessian, then examine Bayesian inference and coefficient interpretation.

Recap: The chain rule for a linear score

For a fixed predictor vector $\mathbf x$ and differentiable scalar function $g$ , the vector-calculus rule used below is

\nabla_{\boldsymbol\beta}\,g(\mathbf x^\top\boldsymbol\beta) =g'(\mathbf x^\top\boldsymbol\beta)\,\mathbf x.

Thus each observation contributes its predictor vector times a scalar derivative. The Bernoulli likelihood will determine that scalar.

Key Logistic Regression Framework

Let $Y \in \{0,1\}$ be a binary outcome with predictors $x_1, x_2, \dots, x_p$ . For observation $i$ , we use the column vector $\mathbf{x}_i = (1, x_{i1}, x_{i2}, \dots, x_{ip})^T$ , whose first entry supplies the intercept. We write $\pi_i = \Pr(Y_i=1\mid\mathbf{x}_i)$ and model its log-odds as a linear function:

Component Form:

\text{logit}\bigl(\Pr(Y_i = 1 \mid \mathbf{x}_i)\bigr) = \beta_0 + \beta_1 x_{i1} + \dots + \beta_p x_{ip}

Vectorized Form:

\text{logit}(\pi_i) = \mathbf{x}_i^T \boldsymbol{\beta}, \quad \text{where } \boldsymbol{\beta} = (\beta_0, \beta_1, \dots, \beta_p)^T

The logit function $\text{logit}(p) = \ln\!\bigl(\frac{p}{1-p}\bigr)$ maps probabilities strictly between zero and one to the real line. The inverse of this link is the sigmoid function, which satisfies $\sigma(\text{logit}(p)) = p$ and $\text{logit}\bigl(\sigma(z)\bigr) = z$ .

Component Form:

\Pr(Y_i = 1 \mid \mathbf{x}_i) = \frac{1}{1 + \exp\!\bigl(-(\beta_0 + \beta_1 x_{i1} + \dots + \beta_p x_{ip})\bigr)}

Vectorized Form:

\pi_i = \sigma(\mathbf{x}_i^T \boldsymbol{\beta}) = \frac{1}{1 + \exp(-\mathbf{x}_i^T \boldsymbol{\beta})}

For the likelihood, we keep the predictor matrix $\mathbf X$ fixed and assume independent draws $Y_i\mid\mathbf X,\boldsymbol\beta\sim\operatorname{Bernoulli}(\pi_i)$ . In the frequentist calculation, the parameters stay fixed and $y_i$ denotes an observed outcome. Thus $\mathbb E[Y_i\mid\mathbf X,\boldsymbol\beta]=\pi_i$ and $\operatorname{Var}(Y_i\mid\mathbf X,\boldsymbol\beta)=\pi_i(1-\pi_i)$ .

The Bayesian section uses the same conditional model for the outcomes but adds a prior on $\boldsymbol\beta$ . Posterior averages then vary $\boldsymbol\beta$ while keeping the observed $(\mathbf X,\mathbf y)$ fixed.

1. Frequentist Approach

In the frequentist approach, we treat $\boldsymbol{\beta} = (\beta_0, \beta_1, \dots, \beta_p)^T$ as fixed but unknown and estimate it by maximizing the likelihood of the observed outcomes, conditional on the predictors.

Likelihood and MLE

Given $n$ independent observations $\{(\mathbf{x}_i, y_i)\}_{i=1}^n$ , with $y_i \in \{0,1\}$ , the probability of $y_i = 1$ is $\pi_i = \sigma(\mathbf{x}_i^T\boldsymbol{\beta})$ . The likelihood is:

Component Form:

L(\boldsymbol{\beta}) = \prod_{i=1}^{n} \pi_i^{y_i}\,(1 - \pi_i)^{(1 - y_i)}

Log-Likelihood:

\ell(\boldsymbol{\beta}) = \sum_{i=1}^{n} \Bigl[y_i \ln(\pi_i) + (1 - y_i)\ln(1-\pi_i)\Bigr]

Derivative of the Log-Likelihood

To maximize $\ell(\boldsymbol{\beta})$ , we need the gradient $\frac{\partial \ell}{\partial \boldsymbol{\beta}}$ . We derive it one observation at a time, then collect the terms in matrix form.

Express the likelihood through the coefficients.

Substituting the logistic probabilities into the log-likelihood gives:

\ell(\boldsymbol{\beta}) = \sum_{i=1}^n \Bigl[ y_i \,\ln\bigl(\sigma(\mathbf{x}_i^\top \boldsymbol{\beta})\bigr) + (1 - y_i)\,\ln\bigl(1 - \sigma(\mathbf{x}_i^\top \boldsymbol{\beta})\bigr) \Bigr]

Differentiate the probability map.

The derivative of $\sigma(z)$ is:

\frac{d\,\sigma(z)}{dz} = \sigma(z)\,\bigl(1 - \sigma(z)\bigr).

We use this identity when differentiating each logarithm below.

Apply the chain rule to one observation.

Write the contribution from observation $i$ as:

L_i(\boldsymbol{\beta}) = y_i \,\ln\bigl(\sigma(\mathbf{x}_i^\top \boldsymbol{\beta})\bigr) + (1 - y_i)\,\ln\bigl(1 - \sigma(\mathbf{x}_i^\top \boldsymbol{\beta})\bigr).

Let $z_i = \mathbf{x}_i^\top \boldsymbol{\beta}$ . The chain rule gives the two derivatives we need:

For $\ln(\sigma(z_i))$ ,

\frac{\partial}{\partial \boldsymbol{\beta}} \bigl[\ln(\sigma(z_i))\bigr] = \frac{1}{\sigma(z_i)} \cdot \sigma(z_i)\bigl(1 - \sigma(z_i)\bigr) \cdot \mathbf{x}_i = \bigl(1 - \sigma(z_i)\bigr)\mathbf{x}_i.

and for $\ln(1-\sigma(z_i))$ ,

\frac{\partial}{\partial \boldsymbol{\beta}} \bigl[\ln\bigl(1 - \sigma(z_i)\bigr)\bigr] = \frac{1}{1 - \sigma(z_i)} \cdot \bigl[-\sigma(z_i)\bigl(1 - \sigma(z_i)\bigr)\bigr] \cdot \mathbf{x}_i = -\sigma(z_i)\,\mathbf{x}_i.

Combining the two contributions gives:

\frac{\partial L_i}{\partial \boldsymbol{\beta}} = y_i \,\bigl(1 - \sigma(z_i)\bigr)\mathbf{x}_i \;-\; (1 - y_i)\,\sigma(z_i)\,\mathbf{x}_i.

After collecting terms, the gradient is the predictor vector multiplied by the observed outcome minus its predicted probability:

\frac{\partial L_i}{\partial \boldsymbol{\beta}} = \mathbf{x}_i\,\bigl[y_i - \sigma(z_i)\bigr].

Combine the observation-level derivatives.

The gradient of the full log-likelihood is:

\nabla_{\boldsymbol{\beta}} \ell(\boldsymbol{\beta}) = \sum_{i=1}^n \frac{\partial L_i}{\partial \boldsymbol{\beta}} = \sum_{i=1}^n \mathbf{x}_i \,\bigl[y_i - \sigma(\mathbf{x}_i^\top \boldsymbol{\beta})\bigr].

Let $\mathbf{X}$ be the $n\times(p+1)$ matrix with row $i$ equal to $\mathbf{x}_i^\top$ , and let $\mathbf{y}$ be the outcome vector. Defining $\mathbf{p}=\sigma(\mathbf{X}\boldsymbol{\beta})$ , with the sigmoid applied to each entry, gives:

\nabla_{\boldsymbol{\beta}} \ell(\boldsymbol{\beta}) = \mathbf{X}^\top \bigl(\mathbf{y} - \mathbf{p}\bigr).

Equivalently, we can write the derivative for one coefficient or for the full vector:

Component form for coefficient $\beta_j$ :

$\frac{\partial \ell}{\partial \beta_j} = \sum_{i=1}^n \bigl[y_i - \sigma(\mathbf{x}_i^\top \boldsymbol{\beta})\bigr]\,x_{ij}.$
Vector form:

$\nabla \ell(\boldsymbol{\beta}) = \mathbf{X}^\top \,\bigl(\mathbf{y} - \sigma(\mathbf{X}\boldsymbol{\beta})\bigr) = \mathbf{X}^\top \bigl(\mathbf{y} - \mathbf{p}\bigr).$

Hessian Matrix

Newton–Raphson also uses the Hessian $\mathbf{H}$ , the matrix of second derivatives of the log-likelihood. Differentiating the gradient gives:

Component Form
For coefficients $\beta_j$ and $\beta_k$ , the second partial derivative is:

$\frac{\partial^2 \ell}{\partial \beta_j \partial \beta_k} = -\sum_{i=1}^n x_{ij}\,x_{ik}\,\pi_i\bigl(1 - \pi_i\bigr),$

where $\pi_i = \sigma\bigl(\mathbf{x}_i^\top \boldsymbol{\beta}\bigr)$ .
Vectorized Form
Define the diagonal matrix $\mathbf{W} = \mathrm{diag}\!\bigl(\pi_i\bigl(1 - \pi_i\bigr)\bigr)$ . Then the Hessian matrix is:

$\mathbf{H} = \frac{\partial^2 \ell(\boldsymbol{\beta})}{\partial \boldsymbol{\beta}\,\partial \boldsymbol{\beta}^\top} = -\mathbf{X}^\top \mathbf{W}\,\mathbf{X}.$

Why No Closed-Form Solution?

Setting $\nabla \ell(\boldsymbol{\beta}) = 0$ gives:

\mathbf{X}^T (\mathbf{y} - \mathbf{p}) = 0

Because $\mathbf{p}$ depends nonlinearly on $\boldsymbol{\beta}$ through $\sigma(\cdot)$ , these equations do not generally have a closed-form solution. When a finite maximizer exists, we typically compute it numerically:

Newton-Raphson: Iteratively update $\boldsymbol{\beta}$ using:
$\boldsymbol{\beta}^{(t+1)} = \boldsymbol{\beta}^{(t)} - \bigl[\mathbf{H}(\boldsymbol{\beta}^{(t)})\bigr]^{-1} \nabla \ell(\boldsymbol{\beta}^{(t)})$
Gradient ascent: Update via $\boldsymbol{\beta}^{(t+1)} = \boldsymbol{\beta}^{(t)} + \eta \nabla \ell(\boldsymbol{\beta}^{(t)})$ with a suitable positive step size $\eta$ . The plus sign reflects maximization; minimizing the negative log-likelihood instead uses gradient descent.

2. Bayesian Approach

Bayesian logistic regression uses the same likelihood and describes uncertainty about $\boldsymbol{\beta}$ through a prior $p(\boldsymbol{\beta})$ . For example, we may use a Gaussian prior with precision $\alpha>0$ :

\boldsymbol{\beta} \sim \mathcal{N}(\mathbf{0},\, \alpha^{-1}\mathbf{I})

Posterior Inference

The posterior is proportional to the likelihood times the prior:

p(\boldsymbol{\beta} \mid \mathbf{y}, \mathbf{X}) \propto p(\mathbf{y} \mid \mathbf{X}, \boldsymbol{\beta}) \, p(\boldsymbol{\beta})

Log-Posterior:

\ln p(\boldsymbol{\beta} \mid \mathbf{y}, \mathbf{X}) = \ell(\boldsymbol{\beta}) - \frac{\alpha}{2} \boldsymbol{\beta}^T \boldsymbol{\beta} + \text{constant}

Why No Closed-Form Posterior?

Combining a Gaussian prior with the logistic likelihood does not produce a Gaussian posterior. In other words, this prior is not conjugate to the likelihood. We generally cannot compute the posterior's normalizing constant in closed form, so we use numerical methods such as:

Markov Chain Monte Carlo (MCMC): Sample from the posterior using algorithms like Metropolis-Hastings or Hamiltonian Monte Carlo.
Variational inference: Approximate the posterior within a chosen family, such as Gaussian distributions, by maximizing the evidence lower bound (ELBO).

Estimating Parameters Step by Step

Frequentist (MLE) Derivation

The score equation does not generally let us solve directly for the coefficients. Instead, we use the derivatives above in an iterative procedure: calculate the current probabilities, evaluate the gradient and Hessian, and update the coefficients. The expressions below collect the quantities needed for these updates.

Define the Likelihood
For each observation $(\mathbf{x}_i, y_i)$ , where $y_i \in \{0,1\}$ , the probability of the observed outcome is:

$\Pr(Y_i = y_i \mid \mathbf{x}_i) = \bigl[\sigma(\mathbf{x}_i^\top \boldsymbol{\beta})\bigr]^{y_i} \,\bigl[1 - \sigma(\mathbf{x}_i^\top \boldsymbol{\beta})\bigr]^{1 - y_i},$

where $\sigma(z) = \tfrac{1}{1 + e^{-z}}$ is the sigmoid function.
Log-Likelihood
Taking logs turns products into sums:

$\ell(\boldsymbol{\beta}) = \sum_{i=1}^n \Bigl[y_i\,(\mathbf{x}_i^\top \boldsymbol{\beta}) \;-\;\ln\bigl(1 + \exp(\mathbf{x}_i^\top \boldsymbol{\beta})\bigr)\Bigr].$

This expression is more convenient for differentiation and for numerical optimization.
Gradient and Hessian
Let $\mathbf{X}$ be the $n \times (p+1)$ design matrix, $\mathbf{y}$ the $n \times 1$ outcome vector, and $\mathbf{p} = \sigma(\mathbf{X}\boldsymbol{\beta})$ (applied row-by-row). Then:
- Gradient:
  $\nabla \ell(\boldsymbol{\beta}) = \mathbf{X}^\top \bigl(\mathbf{y} - \mathbf{p}\bigr).$
- Hessian:
  $\mathbf{H} = \frac{\partial^2 \ell}{\partial \boldsymbol{\beta} \partial \boldsymbol{\beta}^\top} = -\mathbf{X}^\top \mathbf{W} \,\mathbf{X},$
  where $\mathbf{W} = \mathrm{diag}\bigl(p_i(1 - p_i)\bigr)$ .
Newton-Raphson Update
- Initialization: Choose a starting guess $\boldsymbol{\beta}^{(0)}$ .
- Iterate until convergence:
  $\boldsymbol{\beta}^{(t+1)} = \boldsymbol{\beta}^{(t)} + \bigl[\mathbf{X}^\top \mathbf{W}^{(t)} \mathbf{X}\bigr]^{-1} \, \mathbf{X}^\top \bigl(\mathbf{y} - \mathbf{p}^{(t)}\bigr).$
We recompute the Hessian $\mathbf{H}$ and gradient $\nabla \ell(\boldsymbol{\beta})$ at each iteration. The inverse notation assumes a nonsingular Hessian; in computation, we solve the corresponding linear system. Damping or a line search may be needed to control the step.
Gradient Ascent

If computing the Hessian is costly, we can use gradient ascent, which requires only first derivatives:

\boldsymbol{\beta}^{(t+1)} = \boldsymbol{\beta}^{(t)} + \eta \,\nabla \ell(\boldsymbol{\beta}^{(t)}),

where $\eta$ is the step size or learning rate.

Gradient ascent avoids solving the Newton system, although it may require more iterations. Stochastic and mini-batch versions replace the full-data gradient with an estimate from a subset of observations; their step sizes must be chosen accordingly.

Interpreting Logistic Regression Results

1. Statistical Interpretation

In the additive model specified above, a coefficient $\beta_j$ describes the change in log-odds associated with a one-unit increase in $x_j$ , holding the other predictors fixed:

Log-Odds Interpretation:
- $\beta_j$ is the change in $\log\!\Bigl(\frac{\pi}{1-\pi}\Bigr)$ (log-odds of the event) for a one-unit increase in $x_j$ , holding all other predictors constant.
Odds Ratio:
- $\exp(\beta_j)$ is the odds ratio (OR) associated with a one-unit increase in $x_j$ .
- If $\exp(\beta_j) > 1$ , the odds of $Y=1$ increase as $x_j$ increases.
- If $\exp(\beta_j) < 1$ , the odds of $Y=1$ decrease as $x_j$ increases.
Confidence Intervals:
- We usually report an estimated odds ratio $\exp(\hat\beta_j)$ with a confidence interval. An interval containing 1 does not reject the null odds ratio of 1 at the corresponding two-sided significance level; it does not establish that there is no association.

2. Layman Interpretation

For a non-technical audience, predicted probabilities are often easier to interpret than log-odds. The important distinction is that an odds ratio multiplies odds, not probabilities. For example:

“Chance” vs. “Odds”:
- A one-unit increase in $x_j$ multiplies the odds of $Y=1$ by $\exp(\beta_j)$ , with the other predictors held fixed. If the original probability is $p$ and the odds ratio is $r$ , the new probability is $rp/(1-p+rp)$ . Doubling the odds therefore does not generally double the probability.
Example Phrase:
- If an additional study hour has an estimated odds ratio of 2, a baseline passing probability of 0.5 becomes about 0.67, not 1.0. This describes an association in the fitted model, not necessarily the causal effect of studying an extra hour.
Intuitive Summaries:
- Report predicted probabilities for clearly specified predictor values. This makes the size of the association visible without confusing a percentage change in odds with a percentage-point change in probability.

Model Assumptions & Diagnostics

Linearity in the Log-Odds

What It Means: Continuous predictors should relate linearly to the log-odds of the outcome, not directly to the outcome itself.

How to Check:

Box–Tidwell Check: For a strictly positive continuous predictor, terms of the form $x_j \ln(x_j)$ can be used to assess departure from linearity on the log-odds scale. This is evidence about the chosen functional form, not proof that an unflagged model is correct.
Residual or Partial Residual Plots: Plot deviance or Pearson residuals vs. the predictor. A systematic pattern suggests nonlinearity.
Spline Terms: Including spline (e.g., cubic) expansions of predictors can reveal or capture nonlinearity.

Mathematical Note: If $\mathrm{logit}(\pi_i) = \beta_0 + \sum_j \beta_j x_{ij}$ does not adequately describe the relationship, consider higher-order or interaction terms.

No Perfect Separation

What It Means: Complete or quasi-complete separation can prevent a finite maximum likelihood estimate from existing. Along a separating direction, some coefficient estimates diverge toward $\pm \infty$ .

How to Check:

Software Warnings: Common packages (R, Python, etc.) often warn about divergence or extremely large coefficient estimates.
Cross-Tabulations: Inspect whether specific categories of predictors contain only 0’s or only 1’s for the outcome.

Remedies: Use Firth’s bias-reduced logistic regression or add a penalty (e.g., ridge or lasso).
Mathematical Note: For Bernoulli observations, $\ell(\boldsymbol{\beta})\leq 0$ . Under complete separation, the log-likelihood approaches its supremum, $0$ , along a separating direction, but does not attain it at finite coefficients. It is the coefficients—not the log-likelihood—that become unbounded. See the Stan discussion of separation.

Independence of Observations

What It Means: Each data point (observation) is assumed to be independent from the others.

How to Check:

Study Design: Ensure the data-collection process didn’t violate independence (e.g., repeated measures, clustered data).
Time or Cluster Structure: Examine residual patterns over time or within groups, using diagnostics appropriate to a binary-response model.
Intraclass Correlation (ICC): For grouped data, a high ICC indicates dependence within groups.

Remedies: Use Generalized Estimating Equations (GEE) or Mixed-Effects Logistic Regression if observations are correlated.

Absence of High Multicollinearity

What It Means: Predictors should not be excessively correlated with each other, as it inflates variance of the parameter estimates.

How to Check:

Variance Inflation Factor (VIF): $\text{VIF} > 5$ (or 10) often indicates problematic collinearity.
Correlation Matrix: Look for predictor pairs with correlations near 1 or -1.
Eigenvalues & Condition Indices: A condition index above 30 is suspicious.

Remedies: Remove or combine redundant predictors, or apply regularization (ridge or lasso).

Correct Link Function

What It Means: Logistic regression assumes a logit link. If the true relationship is closer to a probit or complementary log-log, the model may be misspecified.

How to Check:

Compare Fit: Fit alternative links (e.g., probit) and compare information criteria (AIC/BIC).
Residual Plots: Look for systematic deviation that might indicate a different link is needed.

Mathematical Note: The logit link is $\ln\!\bigl(\frac{\pi}{1-\pi}\bigr)$ . Alternative links just change the transformation of $\pi$ .

No Overdispersion (for Aggregated Binomial Data)

What It Means: When using grouped/binomial data, logistic regression assumes binomial variance. Overdispersion occurs if the variance exceeds what's expected under the binomial assumption.

How to Check:

Deviance / Degrees of Freedom: If the ratio is noticeably greater than 1, consider overdispersion.
Pearson Chi-Square: Compare to degrees of freedom for an overdispersion test.

Remedies: Use a quasi-binomial or beta-binomial approach if overdispersion is detected.

Adequate Sample Size

What It Means: Logistic regression requires enough data relative to the number of parameters. Small sample sizes can lead to instability or bias in estimates.

How to Check:

Events and Non-Events: Compare the available outcomes with the number of fitted parameters, including nonlinear terms and interactions. A fixed events-per-variable threshold is not a guarantee of adequate estimation.
Simulation or Power Analysis: Evaluate how sample size affects parameter estimation confidence.

Remedies: Collect more data, reduce the number of predictors, or apply penalized methods (ridge, lasso, or Firth).

Conclusion

The inverse logit turns the linear predictor into a valid probability. Differentiating the Bernoulli log-likelihood then gives observed outcomes minus predicted probabilities, weighted by the predictor vectors. The Hessian supplies the curvature for Newton–Raphson; adding a prior instead changes the objective to posterior inference without changing the likelihood.

Interpretation requires a separate step: coefficients describe conditional log-odds associations, and odds ratios are not probability ratios. Checking the functional form, separation, dependence, and the amount of available information helps us judge whether those fitted associations are useful for the intended question.