Analysis of loss functions for fast single-class classification

Abstract

We consider neural network training, in applications in which there are many possible classes, but at test time, the task is a binary classification task of determining whether the given example belongs to a specific class. We define the single logit classification (SLC) task: training the network so that at test time, it would be possible to accurately identify whether the example belongs to a given class in a computationally efficient manner, based only on the output logit for this class. We propose a natural principle, the Principle of Logit Separation, as a guideline for choosing and designing losses suitable for the SLC task. We show that the cross-entropy loss function is not aligned with the Principle of Logit Separation. In contrast, there are known loss functions, as well as novel batch loss functions that we propose, which are aligned with this principle. Our experiments show that indeed in almost all cases, losses that are aligned with the Principle of Logit Separation obtain at least 20% relative accuracy improvement in the SLC task compared to losses that are not aligned with it, and sometimes considerably more. Furthermore, we show that fast SLC does not cause any drop in binary classification accuracy, compared to standard classification in which all logits are computed, and yields a speedup which grows with the number of classes.

Appendices

Appendix

Omitted proofs: alignment with the Principle of Logit Separation

All the considered losses are a function of the output logits and the example labels. For a network model $\theta$, denote the vector of logits it assigns to example x by $z^\theta (x) = (z^\theta _1(x),\ldots ,z^\theta _k(x))$. When $\theta$ and x are clear from context, we write $z_j$ instead of $z_j^\theta (x)$. Denote the logit output of the sample by $S_\theta = ((z^\theta (x_1),y_1),\ldots ,(z^\theta (x_n),y_n))$. A loss function $\ell :\cup _{n=1}^\infty ({\mathbb {R}}^k \times [k])^n \rightarrow {\mathbb {R}}_+$ assigns a loss to a training sample based on the output logits of the model and on the labels of the training examples. The goal of training is to find a model $\theta$ which minimizes $\ell (S_\theta ) \equiv \ell (S,\theta )$. In almost all the losses we study, the loss on the training sample is the sum over all examples of a loss defined on a single example: $\ell (S_\theta ) \equiv \sum _{i=1}^n \ell (z^\theta (x_i),y_i)$, thus we only define $\ell (z,y)$. We explicitly define $\ell (S_\theta )$ below only when this is not the case.

1.1 Self-normalization

We prove that the self-normalization loss satisfies the PoLS: let a training sample S and a neural network model $\theta$, and consider an example $(x,y) \in S$. We consider the two terms of the loss in order. First, consider $-\log (p_y)$. From the definition of $p_y$ (Eq. 1) we have that

$$\begin{aligned} - \log (p_y) = \log \left( \sum _{j=1}^k e^{z_j-z_y}\right) = \log \left( 1+ \sum _{j\ne y} e^{z_j-z_y}\right) . \end{aligned}$$

Set $\epsilon _0 := \log (1+e^{-2})$. Then, if $-\log (p_y) < \epsilon _0$, we have $\sum _{j\ne y} e^{z_j-z_y} \le e^{-2}$, which implies that (a) $\forall j \ne y, z_j \le z_y - 2$ and (b) $e^{z_y} \ge \sum _{j=1}^k e^{z_j}/(1+e^{-2}) \ge \frac{1}{2} \sum _{j=1}^ke^{z_j}.$ Second, consider the second term. There is an $\epsilon _1 > 0$ such that if $\log ^2 (\sum _{j=1}^k e^{z_j}) < \epsilon _1$ then (c) $2e^{-1}< \sum _{j=1}^k e^{z_j} < e$, which implies $e^{z_y} < e$ and hence (d) $z_y < 1$. Now, let $\theta$ such that $\ell (S_\theta ) \le \epsilon := \min (\epsilon _0, \epsilon _1)$. Then $\forall (x,y) \in S$, $\ell (z^\theta (x),y) \le \epsilon$. From (b) and (c), $e^{-1}< \frac{1}{2} \sum _{j=1}^k e^{z_j} < e^{z^y}$, hence $z_y > -1$. Combining with (d), we get $-1< z_y < 1$. Combined with (a), we get that for $j \ne y$, $z_j < -1$. To summarize, $\forall (x,y),(x',y') \in S$ and $\forall y'' \ne y'$, we have that $z_y^\theta (x)> -1 > z_{y''}^\theta (x')$, implying PoLS alignment.

1.2 Noise-contrastive estimation

Recall the definition of the NCE loss from Eq. (8):

$$\begin{aligned} \ell (z,y) = -\log g_y - t\cdot {\mathbb {E}}_{j \sim q}\left[ \log (1-g_j)\right] \end{aligned}$$

where $g_j := (1 + t\cdot q(j)\cdot e^{-z_j})^{-1}.$ We prove that the NCE loss satisfies the PoLS: $g_j$ is monotonic increasing in $z_j$. Hence, if the loss is small, $g_y$ is large and $g_j$ for $j \ne y$, is small. Formally, fix t, and let a training sample S. There is an $\epsilon _0 > 0$ such that if $-\log g_j \le \epsilon _0$, then $z_j > 0$. Also, there is an $\epsilon _1 > 0$ (which depends on q) such that if $- {\mathbb {E}}_{j \sim q}\left[ \log (1-g_j)\right] \le \epsilon _1$ then $\forall j \ne y$, $\log (1-g_j)$ must be small enough so that $z_j < 0$. Now, consider $\theta$ such that $\ell (S_\theta ) \le \epsilon := \min (\epsilon _0, \epsilon _1)$. Then for every $(x,y) \in S$, $\ell (z^\theta (x),y) \le \epsilon$. This implies that for every $(x,y),(x',y') \in S$ and $y'' \ne y'$, we have that $z^\theta _y(x)> 0 > z^\theta _{y''}(x')$, thus this loss is aligned with the PoLS.

1.3 Binary cross-entropy

This loss is similar in form to the NCE loss: for $g_j$ as in Eq. (8), $g_j = \sigma (z_j - \ln (t \cdot q(j)))$. Since $\sigma (z_j)$ is monotonic, the proof method for NCE carries over and thus the binary cross-entropy loss satisfies the PoLS as well.

1.4 Batch losses

Recall that the batch losses are defined as $\ell (S_\theta ) := {\mathbb {E}}_B[L(B_\theta )]$, where $B_\theta$ is a random batch out of $S_\theta$ and L is $L_c$ for the cross-entropy (Definition 2), and $L_m$ is the max-margin loss (Definition 3). If true logits are greater than false logits in every batch separately when using, then the PoLS is satisfied on the whole sample, since every pair of examples appears together in some batch. The following lemma formalizes this:

Lemma 1

If L is aligned with the PoLS, and $\ell$ is defined by $\ell (S_\theta ) := {\mathbb {E}}_B[L(B_\theta )]$, then $\ell$ is also aligned with the PoLS.

Proof

Assume a training sample S and a neural network model $\theta$. Since L is aligned with the PoLS, there is some $\epsilon ' > 0$ such if $L(B_\theta ) < \epsilon '$, then for each $(x,y),(x',y') \in B$ and $y'' \ne y'$ we have that $z_y^\theta (x) > z_{y''}^\theta (x')$. Let $\epsilon = \epsilon '/\left( {\begin{array}{c}n\\ m\end{array}}\right)$, and assume $\ell (S_\theta ) < \epsilon$. Since there are $\left( {\begin{array}{c}n\\ m\end{array}}\right)$ batches of size m in S, this implies that for every batch B of size m, $L(B_\theta ) \le \epsilon '$. For any $(x,y),(x',y') \in S$, there is a batch B that includes both examples. Thus, for $y'' \ne y'$, $z_y^\theta (x) > z_{y''}^\theta (x')$. Since this holds for any two examples in S, $\ell$ is also PoLS-aligned. $\square$

(1) Batch cross-entropy To show that the batch cross-entropy satisfies the PoLS, we show that $L_c$ does, which by Lemma 1 implies this for $\ell$. By the continuity of $\mathrm {KL}$, and since for discrete distributions, $\mathrm {KL}(P||Q) =0 \iff P \equiv Q$, there is an $\epsilon > 0$ such that if $L(B_\theta ) \equiv \mathrm {KL}(P_B || Q^\theta _B)] < \epsilon$, then for all i, j, $|P_B(i,j) - Q^\theta _B(i,j)| \le \frac{1}{2m}$. Therefore, for each example $(x,y) \in B$,

$$\begin{aligned} \frac{e^{z_y^\theta (x)}}{Z(B)} > \frac{1}{2m}, \qquad \text { and }\qquad \forall j \ne y, \quad \frac{e^{z_j^\theta (x)}}{Z(B)} < \frac{1}{2m}. \end{aligned}$$

It follows that for any two examples $(x,y),(x',y') \in B$, if $y \ne y'$, then $z_y^\theta (x)> \frac{1}{2m} > z_{y'}^\theta (x')$. Therefore L satisfies the PoLS, which completes the proof.

(2) Batch max-margin To show that the batch max-margin loss satisfies the PoLS, we show this for $L_m$ and invoke Lemma 1. Set $\epsilon = \gamma /m$. If $L(B_\theta ) < \epsilon$, then $\gamma -z_{+}^B + z_{-}^B < \gamma$, implying $z_{+}^B > z_{-}^B$. Hence, any $(x,y),(x',y') \in B$ such that $y \ne y'$ satisfy $z^\theta _y(x) \ge z_{+}^B > z_{-}^B \ge z^\theta _y(x')$. Thus L is aligned with the PoLS, implying the same for $\ell$.