Surrogate Loss Functions and Fisher Consistency in Binary Classification

April 3, 2026

On this page

In binary classification, we want to make as few incorrect predictions as possible. The 0–1 loss records each mistake: it is 1 for an incorrect prediction and 0 for a correct one. With true label $y \in \{-1,+1\}$ and prediction $\hat{y}$ , we define it as

L_{\text{0–1}}(y, \hat{y}) \;=\; \begin{cases} 0, & \text{if } y = \hat{y},\\[1mm] 1, & \text{if } y \neq \hat{y}. \end{cases}

In indicator notation, we can write this as

L_{\text{0–1}}(y,\hat{y}) \;=\; 1\{y \neq \hat{y}\}.

We take $\hat{y} = \mathrm{sign}\bigl(f(x)\bigr)$ , where $f$ is a measurable score function: it maps the input features $x$ to a real number whose sign determines the predicted class. Throughout this note, we predict $+1$ when the score is zero, so $\mathrm{sign}(t)=+1$ for $t\ge 0$ and $-1$ otherwise.

For a nonzero score, a positive product $y\,f(x)$ indicates a correct prediction and a negative product indicates a mistake. At a zero score, correctness depends on the label and our tie-breaking rule. Defining the margin $z = y\,f(x)$ , we therefore have the bound

L_{\text{0–1}}(y,\hat{y}) \;\le\; 1\{y\,f(x)\le 0\} = 1\{z\le 0\}.

The inequality becomes equality whenever the score is nonzero. The margin-based indicator counts every zero score as an error; the actual classification loss uses the fixed tie-breaking rule above.

The 0–1 loss measures classification error, but changing a score without changing its sign leaves the loss unchanged. This makes direct optimization difficult. A surrogate loss replaces it with a criterion that changes more usefully with the score. We must then check whether minimizing its population risk, the average loss over the data distribution, preserves the desired classification decision.

The Role of Surrogate Loss Functions

Surrogate losses replace the 0–1 loss with a criterion that is easier to optimize. The examples below are convex in the margin, although not all are smooth: the hinge loss has a corner. We first compare their forms, then examine when minimizing their expected risks preserves the desired classification rule.

Hinge Loss: $\phi_{\text{hinge}}(z) = \max\{0,\,1 - z\}$ , used in support vector machines.
Exponential Loss: $\phi_{\text{exp}}(z) = \exp(-z)$ , used in AdaBoost.
Logistic Loss: $\phi_{\text{log}}(z) = \ln\bigl(1 + \exp(-z)\bigr)$ , used in logistic regression.
Truncated Quadratic Loss: $\phi_{\text{tquad}}(z)=\begin{cases}(1-z)^2,&z<1,\\0,&z\ge1.\end{cases}$ , which is quadratic below its threshold and zero above it.

In these expressions, $z = y\,f(x)$ is the margin: the product of the true label $y$ and the score $f(x)$ . Its sign records whether the score agrees with the label.

Why Convexity Matters

Convexity helps explain the computational appeal of these losses: a local minimum of a convex function is also a global minimum. We check convexity in the scalar margin below. Whether the full fitting problem is convex also depends on how the score function is parameterized.

1. Hinge Loss

Defined as

\phi_{\text{hinge}}(z) = \max\{0,\,1 - z\},

the hinge loss is the pointwise maximum of the linear function $1-z$ and the constant function 0. We can break this down further:

\phi_{\text{hinge}}(z) = \begin{cases} 1 - z, & \text{if } z < 1,\\[1mm] 0, & \text{if } z \ge 1. \end{cases}

Why Convex?

Piecewise Definition:
For $z < 1$ , we have $\phi_{\text{hinge}}(z) = 1 - z$ , and for $z \ge 1$ , $\phi_{\text{hinge}}(z) = 0$ .
Convexity Criterion:
A function that is the maximum of two convex functions is itself convex. Here, the constant function 0 is convex, and $1-z$ is an affine (linear) function, which is convex. Thus, their pointwise maximum, $\phi_{\text{hinge}}(z)$ , is convex.
Conclusion:
Hence, $\phi_{\text{hinge}}(z) = \max\{0,\,1 - z\}$ is convex.

2. Exponential Loss

With the form

\phi_{\text{exp}}(z) = e^{-z},

the exponential loss is smooth. Its second derivative also establishes strict convexity:

First Derivative:
$\frac{d}{dz} \phi_{\text{exp}}(z) = \frac{d}{dz} \bigl(e^{-z}\bigr) = -\,e^{-z}.$
Second Derivative:
$\frac{d^2}{dz^2} \phi_{\text{exp}}(z) = \frac{d}{dz} \bigl(-\,e^{-z}\bigr) = e^{-z}.$
Positivity of Second Derivative:
Since $e^{-z} > 0$ for all real $z$ , it follows that

$\frac{d^2}{dz^2} \phi_{\text{exp}}(z) > 0,$

which implies that $\phi_{\text{exp}}(z)$ is strictly convex.

3. Logistic Loss

The logistic loss is given by

\phi_{\text{log}}(z) = \ln\bigl(1 + e^{-z}\bigr).

As with the exponential loss, we can establish strict convexity by differentiating twice:

First Derivative:
Let $\alpha(z)=e^{-z}$ . Then,

$\phi_{\text{log}}'(z) = \frac{d}{dz} \ln\bigl(1 + \alpha(z)\bigr) = \frac{1}{1 + \alpha(z)} \cdot \frac{d}{dz}\bigl(1 + \alpha(z)\bigr) = \frac{-\,\alpha(z)}{1 + \alpha(z)}.$

Hence,

$\phi_{\text{log}}'(z) = -\,\frac{e^{-z}}{1 + e^{-z}}.$
Second Derivative:
Differentiating again,

$\phi_{\text{log}}''(z) = \frac{d}{dz} \Bigl(-\,\frac{e^{-z}}{1 + e^{-z}}\Bigr).$

A systematic approach is to set $\alpha(z)=e^{-z}$ (so that $\alpha'(z)=-e^{-z}=-\alpha(z)$ ) and apply the quotient rule. This yields:

$\phi_{\text{log}}''(z) = \frac{e^{-z}}{(1 + e^{-z})^2}.$
Positivity of Second Derivative:
Since both $e^{-z}$ and $(1 + e^{-z})^2$ are positive for all $z$ , we have:

$\phi_{\text{log}}''(z) > 0,$

confirming that $\phi_{\text{log}}(z)$ is strictly convex.

The standard logistic loss evaluates to $\ln 2 \approx 0.693$ at $z=0$ , below the value 1 of the margin-based 0–1 loss. Multiplying it by the positive constant $1/\ln 2$ makes its value equal to 1 at $z=0$ without changing the minimizer of the surrogate risk. This scaling makes the logistic loss an upper bound on the 0–1 loss; preserving the minimizer and providing a pointwise upper bound are distinct properties.

4. Truncated Quadratic Loss

Often defined as

\phi_{\text{tquad}}(z) = \begin{cases} (1 - z)^2, & \text{if } z < 1,\\[1mm] 0, & \text{if } z \ge 1, \end{cases}

this loss uses a quadratic penalty for $z < 1$ and assigns zero loss when $z \ge 1$ . Its convexity follows from its slope: the derivative of $(1-z)^2$ increases to zero at the joining point, then remains zero on the flat part. Equivalently, it is the squared hinge loss, $[\max\{0,1-z\}]^2$ .

Fisher Consistency, Bridging Surrogate Losses and the 0,1 Loss

Convexity answers an optimization question, not whether the resulting decisions are correct. Fisher consistency asks whether a population surrogate-risk minimizer gives a Bayes-optimal classification rule. It does not by itself show that a finite-sample fit reaches that population minimum.

Let $(X,Y)\sim P$ , with $Y\in\{-1,+1\}$ , and write $P_X$ for the distribution of the features alone. Population risks average over the joint distribution, with the candidate score $f$ held fixed. At a fixed $x$ , we average only over $Y$ , with $\Pr_P(Y=+1\mid X=x)=\eta(x)$ and $\Pr_P(Y=-1\mid X=x)=1-\eta(x)$ . We write these expectations as $\mathbb E_P$ , or $\mathbb E$ once the distribution is clear. Under the 0,1 loss, the Bayes classifier chooses the more likely label:

h^*(x) = \begin{cases} +1, & \text{if } \eta(x) \ge 0.5, \\[1mm] -1, & \text{otherwise.} \end{cases}

When we define the surrogate risk as

R_\phi(f) = \mathbb{E}_{(X,Y)\sim P}\bigl[\phi(Y \, f(X))\bigr],

We minimize over all measurable real-valued score functions, rather than a restricted model class. Whenever a minimizer $f^*$ of $R_\phi(f)$ exists, Fisher consistency requires that it satisfies, almost surely,

\operatorname{sign}\bigl(f^*(x)\bigr) \;=\; \begin{cases} +1, & \text{if } \eta(x) > 0.5, \\ -1, & \text{if } \eta(x) < 0.5. \end{cases}

Thus, a population surrogate-risk minimizer makes the Bayes-optimal decision wherever one class is more likely than the other. To obtain a corresponding result for a learned classifier, we would also need to control estimation, approximation, and optimization errors.

Surrogate Losses as Upper Bounds

Some surrogate losses also upper-bound the 0–1 loss pointwise; the logistic loss does so after the scaling described above. We distinguish that upper-bound property from the stronger question of how excess surrogate risk controls excess classification risk. Recall the margin-based 0–1 loss

\ell(z) = \mathbf{1}\{z \le 0\},

and the surrogate risk for a classifier $f$ is given by

R_\phi(f) = \mathbb E_{(X,Y)\sim P}[\phi(Y\,f(X))]\;=\; \int \phi\bigl(y\,f(x)\bigr)\,dP(x,y),

where $\phi$ is a nonnegative surrogate loss, finite at zero. The actual classification risk is bounded above by the expectation of $\ell(Yf(X))$ , including at score ties. Convexity is useful for optimization but is not part of the definition of calibration.

Classification Calibration:

Classification calibration requires a positive conditional-risk gap between the best score and any score with the wrong sign. A consequence is that, for every $\epsilon>0$ , there is a $\delta(\epsilon)>0$ such that every measurable score function $f$ satisfies

R_{0\text{-}1}(f) - R_{0\text{-}1}^* \ge \epsilon \quad \Longrightarrow \quad R_\phi(f) - R_\phi^* \ge \delta(\epsilon),

where

R_{0\text{-}1}(f) = \mathbb{E}\bigl[\mathbf{1}\{\mathrm{sign}(f(X)) \neq Y\}\bigr]

is the misclassification error, $R_{0\text{-}1}^*$ is the Bayes risk, and $R_\phi(f)$ is the surrogate risk. Here $R_\phi^*=\inf_f R_\phi(f)$ , with the infimum over all measurable scores. The conditional-gap definition and its excess-risk consequence are developed by Bartlett, Jordan, and McAuliffe, Section 2.

To examine the corresponding population decisions at a given $x$ , define the conditional surrogate risk as

C_\phi(f(x)) = \eta(x)\,\phi(f(x)) + (1-\eta(x))\,\phi(-f(x)),

with $\eta(x)=P(Y=1|X=x)$ . For a classification-calibrated loss $\phi$ , an attained conditional-risk minimum must give the correct sign whenever the classes are not equally likely. In particular, for every $x$ with $\eta(x) \neq 1/2$ , any minimizer $f^*(x)$ of the conditional risk

C_\phi(f(x)) = \mathbb E_{Y\sim P_{Y\mid X=x}}[\phi(Y\,f(x))]

satisfies

\mathrm{sign}(f^*(x)) = \mathrm{sign}(2\eta(x)-1).

More precisely, for a generic conditional class probability $\eta\in[0,1]$ and score $\alpha\in\mathbb R$ , define

\begin{aligned} C_\eta(\alpha)&=\eta\phi(\alpha)+(1-\eta)\phi(-\alpha),\\ H_\phi(\eta)&=\inf_{\alpha\in\mathbb R}C_\eta(\alpha),\\ H_\phi^-(\eta)&=\inf_{\alpha(2\eta-1)\le 0}C_\eta(\alpha). \end{aligned}

The calibration condition is $H_\phi^-(\eta)>H_\phi(\eta)$ whenever $\eta\ne 1/2$ : a wrong-sign or zero score must have strictly larger conditional risk than the best achievable risk. This definition also covers losses whose risks approach their infimum without reaching it at a finite score.

The excess-risk inequality explains how calibration controls classifiers whose surrogate risks approach the optimum:

The inequality

R_{0\text{-}1}(f) - R_{0\text{-}1}^* \ge \epsilon \quad \Longrightarrow \quad R_\phi(f) - R_\phi^* \ge \delta(\epsilon)

tells us that a misclassification error at least $\epsilon$ above the Bayes risk forces the surrogate risk to remain at least $\delta(\epsilon)$ above its optimum. Because $\delta(\epsilon)$ is positive for every $\epsilon > 0$ , sufficiently small excess surrogate risk rules out any fixed positive gap in classification risk. In particular, when $R_\phi(f)$ converges to $R_\phi^*$ , the excess classification risk $R_{0\text{-}1}(f) - R_{0\text{-}1}^*$ must converge to zero. This does not mean that each individual decrease in surrogate risk produces a decrease in classification error.

Fisher consistency therefore concerns the decisions made by population risk minimizers: their signs agree with $\mathrm{sign}(2\eta(x)-1)$ when one class is more likely. The displayed calibration inequality also controls near-optimal surrogate risks. Both statements concern population risks; neither alone guarantees that a numerical fitting procedure reaches those risks.

Recap: Why conditioning reduces the risk calculation

The conditional-expectation note gives the tower property. For a fixed score $f$ and an integrable loss under the joint law $(X,Y)\sim P$ ,

\mathbb E_{(X,Y)\sim P}[\phi(Yf(X))] =\mathbb E_{X\sim P_X}\!\left[ \mathbb E_{Y\sim P(\cdot\mid X)}[\phi(Yf(X))\mid X] \right].

At a fixed feature value, only the label is averaged and the candidate score is a scalar. This is why the hinge and logistic calculations below minimize a conditional risk before identifying the population decision.

Mathematical Demonstration for Hinge Loss

The hinge loss is defined as

\phi_{\text{hinge}}(z) = \max\{0,\,1 - z\},

which can be written in piecewise form as

\phi_{\text{hinge}}(z) = \begin{cases} 1 - z, & \text{if } z < 1,\\[1mm] 0, & \text{if } z \ge 1. \end{cases}

For a fixed $x$ , define the conditional risk associated with a surrogate loss $\phi$ as

C_\eta(\alpha) = \eta \, \phi(\alpha) + (1-\eta) \, \phi(-\alpha),

where $\alpha = f(x)$ and $\eta = \eta(x)$ .

First consider scores outside $[-1,1]$ . The conditional risk is $\eta(1-\alpha)$ for $\alpha<-1$ and $(1-\eta)(1+\alpha)$ for $\alpha>1$ . Moving toward the nearest endpoint never increases either expression, so a minimizer can always be chosen inside $[-1,1]$ . On that interval,

$\phi_{\text{hinge}}(\alpha) = 1 - \alpha$ , because $\alpha\le 1$ ; and
$\phi_{\text{hinge}}(-\alpha) = 1 + \alpha$ , because $\alpha\ge -1$ .

Then the conditional risk becomes:

\begin{aligned} C_\eta(\alpha) &= \eta \, (1 - \alpha) + (1-\eta) \, (1 + \alpha)\\[1mm] &= \eta + (1-\eta) - \eta \, \alpha + (1-\eta) \, \alpha\\[1mm] &= 1 + \alpha \, \bigl[(1-\eta) - \eta\bigr]\\[1mm] &= 1 + \alpha \, (1 - 2\eta). \end{aligned}

On this interval, the coefficient of the score determines which endpoint minimizes $C_\eta(\alpha)$ :

If $\eta > 0.5$ , then $1 - 2\eta < 0$ , so $C_\eta(\alpha)$ is a decreasing function of $\alpha$ . Its minimum over $\alpha \in [-1,1]$ is achieved at $\alpha = 1$ .
If $\eta < 0.5$ , then $1 - 2\eta > 0$ , so $C_\eta(\alpha)$ is an increasing function of $\alpha$ . Its minimum is achieved at $\alpha = -1$ .
If $\eta = 0.5$ , then $C_{0.5}(\alpha) = 1$ for every $\alpha \in [-1,1]$ , so every score in that interval minimizes the risk.

Thus, when the classes are not equally likely, one valid choice of minimizer $\alpha^*(\eta)$ is

\alpha^*(\eta) = \begin{cases} 1, & \text{if } \eta > 0.5,\\[1mm] -1, & \text{if } \eta < 0.5. \end{cases}

Taking the sign for $\eta\ne 1/2$ , we obtain

\text{sign}\bigl(\alpha^*(\eta)\bigr) = \text{sign}(2\eta - 1),

which is the Bayes-optimal decision rule. At $\eta=1/2$ , either predicted class is Bayes optimal. The conditional minimum is $H_\phi(\eta)=2\min\{\eta,1-\eta\}$ , whereas the best wrong-sign or zero score has risk $H_\phi^-(\eta)=1$ . Their gap is $|2\eta-1|>0$ whenever $\eta\ne 1/2$ , verifying calibration as well as the sign of a minimizer.

Mathematical Demonstration for Logistic Loss

The logistic loss is defined as

\phi_{\text{log}}(z) = \ln(1 + e^{-z}).

For a fixed $x$ , define the conditional risk associated with a surrogate loss $\phi$ as

C_\eta(\alpha) = \eta \, \phi(\alpha) + (1-\eta) \, \phi(-\alpha),

where $\alpha = f(x)$ and $\eta = \eta(x) = P(Y = +1 \mid X = x)$ .

For the logistic loss, this becomes

C_\eta(\alpha) = \eta \ln(1 + e^{-\alpha}) + (1-\eta) \ln(1 + e^{\alpha}).

For a conditional class probability strictly between zero and one, the conditional risk is strictly convex. We therefore find its unique finite minimizer $\alpha^*(\eta)$ by differentiating $C_\eta(\alpha)$ with respect to $\alpha$ and setting the derivative equal to zero.

Differentiate the first term:
The derivative of $\ln(1 + e^{-\alpha})$ with respect to $\alpha$ is

$\frac{d}{d\alpha} \ln(1 + e^{-\alpha}) = -\frac{e^{-\alpha}}{1 + e^{-\alpha}}.$
Differentiate the second term:
Similarly, the derivative of $\ln(1 + e^{\alpha})$ with respect to $\alpha$ is

$\frac{d}{d\alpha} \ln(1 + e^{\alpha}) = \frac{e^{\alpha}}{1 + e^{\alpha}}.$

Thus, the derivative of the conditional risk is

C_\eta'(\alpha) = \eta \left(-\frac{e^{-\alpha}}{1 + e^{-\alpha}}\right) + (1-\eta) \left(\frac{e^{\alpha}}{1 + e^{\alpha}}\right).

Setting $C_\eta'(\alpha) = 0$ , we obtain

-\eta \frac{e^{-\alpha}}{1 + e^{-\alpha}} + (1-\eta) \frac{e^{\alpha}}{1 + e^{\alpha}} = 0.

Writing both fractions with a common denominator and solving for the score gives

\alpha^*(\eta) = \ln \frac{\eta}{1-\eta}.

Now, observe the sign of $\alpha^*(\eta)$ :

If $\eta > 0.5$ , then $\frac{\eta}{1-\eta} > 1$ , so $\alpha^*(\eta) > 0$ .
If $\eta < 0.5$ , then $\frac{\eta}{1-\eta} < 1$ , so $\alpha^*(\eta) < 0$ .
If $\eta = 0.5$ , then $\alpha^*(\eta) = \ln 1 = 0$ .

Thus,

\text{sign}\bigl(\alpha^*(\eta)\bigr) = \text{sign}\left(\ln \frac{\eta}{1-\eta}\right) = \text{sign}(2\eta-1),

which gives a Bayes-optimal decision, using our positive-label tie convention at $\eta=1/2$ .

The minimizing logistic score has the Bayes-optimal sign whenever the two classes have different probabilities. At conditional class probabilities zero or one, the same decision is approached as the score tends to the corresponding infinite limit; no finite score attains the infimum.

Visual Interpretation

The plot above compares the surrogate curves with the 0–1 step function, which is discontinuous at $z=0$ :

Hinge Loss: Defined as $\max\{0,\,1-z\}$ , this curve decreases linearly as the margin increases through $z<1$ and is 0 for $z \ge 1$ .
Exponential Loss: Given by $e^{-z}$ , this smooth curve decays exponentially as $z$ increases, thereby penalizing misclassified examples with a low margin quite heavily.
Standard Logistic Loss: This is expressed as $\ln(1+e^{-z})$ . At $z=0$ , it evaluates to $\ln 2 \approx 0.693$ , below the 0–1 loss value of 1. Multiplication by a positive constant preserves its risk minimizer; this follows from the scaling itself, rather than from classification calibration.
Scaled Logistic Loss: To obtain a curve that upper-bounds the 0–1 loss, we scale the logistic loss by $1/\ln 2$ . The resulting loss, $\frac{1}{\ln 2}\ln(1+e^{-z}),$ attains the value 1 at $z=0$ and is at least the 0–1 loss for $z \le 0$ . Its pointwise upper bound also yields an upper bound on the misclassification error ( $R_{0\text{-}1}(f) = \mathbb{E}\bigl[\mathbf{1}\{\mathrm{sign}(f(X)) \neq Y\}\bigr]$ ).
Truncated Quadratic Loss: Defined as $(1-z)^2$ for $z < 1$ and 0 for $z \ge 1$ , this loss combines a quadratic penalty with a truncation to avoid excessive punishment for high-margin points.

The standard logistic loss lies below the 0–1 loss at $z=0$ , yet its population minimizer still gives the desired classification decision. Scaling adds a pointwise upper bound without changing that minimizer. The distinction matters: an upper bound compares the values of two losses, whereas consistency and calibration concern the decisions obtained by minimizing risk.

Key Takeaways:

The optimization problem: The 0–1 loss directly measures classification mistakes, but its discontinuity and nonconvexity make it difficult to optimize.
The surrogate choice: Hinge, exponential, logistic, and truncated quadratic losses are convex in the margin. Smoothness is a separate property; in particular, the hinge loss is not differentiable at its corner.
The population target: Fisher consistency asks whether a population surrogate-risk minimizer gives a Bayes-optimal decision. Calibration additionally controls excess classification risk through excess surrogate risk.
The upper bound: Scaling can provide a pointwise bound, as when multiplying the logistic loss by $1/\ln(2)$ . That bound alone does not establish consistency or calibration.
The fitted classifier: A population guarantee does not remove the need to control estimation, approximation, and numerical optimization errors when learning from data.

Choosing a surrogate involves two questions: can we optimize it, and does its population objective preserve the decision we want? Convexity helps with the first question; consistency and calibration address the second. Keeping these roles separate lets us use a convenient loss without mistaking computational convenience for a statistical guarantee.