Efficient set-valued prediction in multi-class classification

Abstract

In cases of uncertainty, a multi-class classifier preferably returns a set of candidate classes instead of predicting a single class label with little guarantee. More precisely, the classifier should strive for an optimal balance between the correctness (the true class is among the candidates) and the precision (the candidates are not too many) of its prediction. We formalize this problem within a general decision-theoretic framework that unifies most of the existing work in this area. In this framework, uncertainty is quantified in terms of conditional class probabilities, and the quality of a predicted set is measured in terms of a utility function. We then address the problem of finding the Bayes-optimal prediction, i.e., the subset of class labels with the highest expected utility. For this problem, which is computationally challenging as there are exponentially (in the number of classes) many predictions to choose from, we propose efficient algorithms that can be applied to a broad family of utility functions. Our theoretical results are complemented by experimental studies, in which we analyze the proposed algorithms in terms of predictive accuracy and runtime efficiency.

Appendices

Regret bounds for the utility functions

In this part we present a short theoretical analysis that relates the Bayes optimal solution for the set-based utility functions to the solution obtained on the probabilities given by a trained model. The goal is to upper bound the regret of the set-based utility functions by the $L_1$ error of the class probability estimates. The analysis is performed on the level of a single ${\varvec{x}}$.

Let ${\hat{P}}({\varvec{x}})$ be the estimate of the true underlying distribution $P({\varvec{x}})$. Let $U^*(P, u)$ denote the optimal utility for P obtained by the optimal solution $\hat{Y}^*$ (this solution does not have to be unique). Now, let $\hat{Y}$ denote the optimal solution with respect to ${\hat{P}}({\varvec{x}})$. We define the regret of $\hat{Y}$ as:

$$\begin{aligned} \mathrm {reg}_u(\hat{Y})= & {} U^*(P, u) - U(\hat{Y}, P, u) \\= & {} \sum _{c \in {\mathcal {Y}}} u(c,\hat{Y}^*) P(c\,|\,{\varvec{x}}) - \sum _{c \in {\mathcal {Y}}} u(c,\hat{Y}) P(c\,|\,{\varvec{x}}) \\= & {} \sum _{c \in {\mathcal {Y}}} \left( u(c,\hat{Y}^*) - u(c,\hat{Y}) \right) P(c\,|\,{\varvec{x}}) \end{aligned}$$

We bound $\mathrm {reg}_u({\hat{P}}({\varvec{x}}))$ in terms of the $L_1$-estimation error, i.e.:

$$\begin{aligned} \sum _{c \in {\mathcal {Y}}} | P(c\,|\,{\varvec{x}}) - {\hat{P}}(c\,|\,{\varvec{x}}) | \end{aligned}$$

Note that if $\hat{Y}^* = \hat{Y}$ the regret is 0. Otherwise, we need to have

$$\begin{aligned} U(\hat{Y}, {\hat{P}}, u) \ge U(\hat{Y}^*, {\hat{P}}, u) \end{aligned}$$

Thus, we can write

$$\begin{aligned} \mathrm {reg}_u(\hat{Y})\le & {} U^*(P, u) - U(\hat{Y}, P, u) + U(\hat{Y}, {\hat{P}}, u) - U(\hat{Y}^*, {\hat{P}}, u) \nonumber \\= & {} \sum _{c \in {\mathcal {Y}}} \left( u(c,\hat{Y}^*) - u(c,\hat{Y}) \right) P(c\,|\,{\varvec{x}}) + \sum _{c \in {\mathcal {Y}}} \left( u(c,\hat{Y}) - u(c,\hat{Y}^*) \right) {\hat{P}}(c\,|\,{\varvec{x}}) \nonumber \\= & {} \sum _{c \in {\mathcal {Y}}} u(c,\hat{Y}^*) \left( P(c\,|\,{\varvec{x}}) - {\hat{P}}(c\,|\,{\varvec{x}}) \right) + \sum _{c \in {\mathcal {Y}}} u(c,\hat{Y}) \left( {\hat{P}}(c\,|\,{\varvec{x}}) - P(c\,|\,{\varvec{x}}) \right) \nonumber \\\le & {} \sum _{c \in {\mathcal {Y}}} u(c,\hat{Y}^*) \left| P(c\,|\,{\varvec{x}}) - {\hat{P}}(c\,|\,{\varvec{x}}) \right| + \sum _{c \in {\mathcal {Y}}} u(c,\hat{Y}) \left| {\hat{P}}(c\,|\,{\varvec{x}}) - P(c\,|\,{\varvec{x}}) \right| \end{aligned}$$

(14)

$$\begin{aligned}= & {} \sum _{c \in {\mathcal {Y}}} \left( u(c,\hat{Y}^*) + u(c,\hat{Y}) \right) \left| P(c\,|\,{\varvec{x}}) - {\hat{P}}(c\,|\,{\varvec{x}}) \right| \nonumber \\\le & {} 2 \sum _{c \in {\mathcal {Y}}} \left| P(c\,|\,{\varvec{x}}) - {\hat{P}}(c\,|\,{\varvec{x}}) \right| \end{aligned}$$

(15)

The inequality in (14) follows from the properties of the absolute function, $a \le |a|$, while the one in (15) holds because the utility functions are from the bounded interval, $u(\cdot ,\cdot ) \in [0,1]$. We clearly see that the regret is upper bounded by the quality of the estimated probability distribution.

Fig. 3

A visualization of $g_{\alpha ,\beta }$ in function of different values of $|\hat{Y}|$ and K

Generalized reject option utility and parameter bounds

In this part we analyze which values $\alpha$ and $\beta$ can take so that the $g_{\alpha ,\beta }$ family is lower bounded by precision. This family is visualized in Fig. 3. For a given K, the following inequality must hold $\forall s\in \{1,\ldots ,K\}$, such that $g_{\alpha ,\beta }(s)$ is lower bounded by precision:

$$\begin{aligned} g_{\alpha , \beta }(s) \ge g_P(s)\,, \end{aligned}$$

with utilities:

$$\begin{aligned} g_{\alpha , \beta }(s) = 1-\alpha \Big (\frac{s-1}{K-1}\Big )^\beta \,,\quad g_P(s) = \frac{1}{s}\,. \end{aligned}$$

When looking at the boundary cases (i.e., $s=1, s=K$), we find that:

$$\begin{aligned} \alpha \le \frac{K-1}{K}\,. \end{aligned}$$

By fixing $\alpha =\frac{K-1}{K}$, the above inequality can be rewritten, $\forall s\in \{2,\ldots ,K-1\}$, as:

$$\begin{aligned}&1-\Big (\frac{K-1}{K}\Big )\Big (\frac{s-1}{K-1}\Big )^{\beta } \ge \frac{1}{s} \\&\quad \Leftrightarrow \Big (\frac{s-1}{K-1}\Big )^{\beta } \le \frac{K}{s}\Big (\frac{s-1}{K-1}\Big ) \\&\quad \Leftrightarrow \beta \ge \log _{\frac{s-1}{K-1}}\frac{K}{s}+1 \\&\quad \Rightarrow \beta \ge \log _{\frac{1}{K-1}}\frac{K}{2}+1\\ \end{aligned}$$

Note that in the limit, when $K\rightarrow \infty$, we obtain the following upper and lower bound for $\alpha$ and $\beta$, respectively:

$$\begin{aligned} \lim _{K\rightarrow \infty } \frac{K-1}{K} = 1\,,\quad \lim _{K\rightarrow \infty } \log _{\frac{1}{K-1}}\frac{K}{2}+1 = 0\,. \end{aligned}$$

Experimental setup

For all image datasets, except ALOI.BIN, we use hidden representations obtained by convolutional neural networks, whereas for the text datasets (bottom part of Table 2) tf-idf representations are used. The dimensionality of the representations are denoted by D. For the MNIST dataset we use a convolutional neural network with three consecutive convolutional, batchnorm and max-pool layers, followed by a fully connected dense layer with 32 hidden units. We use ReLU activation functions and optimize the categorical cross-entropy loss by means of Adam optimization with learning rate $\eta = 1e-3$. For the VOC 2006,³ VOC 2007,${}^{3}$ Caltech-101 and Caltech-256, the hidden representations are obtained by resizing images to 224x224 pixels and passing them through the convolutional part of an entire VGG16 architecture, including a max-pooling operation (Simonyan and Zisserman 2014). The weights are set to those obtained by training the network on ImageNet. For all convolutional neural networks, the number of epochs are set to 100 and early stopping is applied with a patience of five iterations. For ALOI.BIN, we use the ALOI⁴ dataset with precalculated random binning features (Rahimi and Recht 2008). Training is performed end-to-end on a GPU, by using the PyTorch library (Paszke et al. 2017) and infrastructure with the following specifications:

• CPU: i7-6800K 3.4 GHz (3.8 GHz Turbo Boost) – 6 cores / 12 threads.

• GPU: 2x Nvidia GTX 1080 Ti 11GB + 1x Nvidia Tesla K40c 11GB.

• RAM: 64GB DDR4-2666.

For the bacteria dataset, tf-idf representations are calculated by using 3-, 4-, and 5-grams extracted from each 16S rRNA sequence in the dataset (Fiannaca et al. 2018). For the proteins dataset, we consider 3-grams in order to calculate the tf-idf representation for each protein sequence. To comply with literature, we concatenate the tf-idf representations with functional domain encoding vectors, which provide distinct functional and evolutional information about the protein sequence. For more information about the functional domain encodings, we refer the reader to (Li et al. 2018). Precalculated tf-idf representations were provided with the DMOZ and LSHTC1 dataset.⁵

Finally, we use the learned hidden representations for the image datasets and calculate tf-idf representations for the text datasets to train the probabilistic models using a dual L2-regularized logistic regression model. For the DMOZ and LSHTC1 dataset we enforce sparsity by clipping all the learned weights less than a threshold $\eta = 0.1$ to zero (Babbar and Schölkopf 2017). We implemented all SVBOP algorithms in C++ using the LIBLINEAR library (Fan et al. 2008) and H-NSW implementation from NMSLIB (Naidan and Boytsov 2015). All experiments were conducted on Intel Xeon E5-2697 v3 2.60GHz (14 cores) with 64GB RAM. We include detail information about selection of hyperparameters for all the models in the next section.

Hyperparameters

Table 5 Values of hyperparameters used for SVBOP-Full, SVBOP-HSG, and SVBOP-HF algorithms for different datasets

For the LIBLINEAR library, used for the implementations of all SVBOP algorithms, as well as the baselines, we tuned two parameters: C – inverse of the regularization strength and $\epsilon _{l}$ –  tolerance of termination criterion. For SVBOP-HSG and the underlying H-NSW indexing method, we tuned four parameters: M – the maximum number of neighbors in the layers of H-NSW index, $\textit{ef}_c$ – size of the dynamic candidate list during H-NSW index construction, and k – initial size of the query to H-NSW index and $\textit{ef}_s$ – size of the dynamic candidate list during H-NSW index query, that both were always set to the same value. For balanced 2-means tree building, we tuned two parameters: l – the maximum number of leaves on the last level of a tree and $\epsilon _{c}$ – §tolerance of termination criterion of the 2-means algorithm. We list all the hyperparameters we used to obtained all the results presented in Sects. 5.2 and 5.3 in Table 5.