Abstract
We propose a new Reject Option Classification technique to identify and remove regions of uncertainty in the decision space for a given neural classifier and dataset. Such existing formulations employ a learned rejection (remove)/selection (keep) function and require either a known cost for rejecting examples or strong constraints on the accuracy or coverage of the selected examples. We consider an alternative formulation by instead analyzing the complementary reject region and employing a validation set to learn per-class softmax thresholds. The goal is to maximize the accuracy of the selected examples subject to a natural randomness allowance on the rejected examples (rejecting more incorrect than correct predictions). We provide results showing the benefits of the proposed method over naïvely thresholding calibrated/uncalibrated softmax scores with 2-D points, imagery, and text classification datasets using state-of-the-art pretrained models. Source code is available at https://github.com/osu-cvl/learning-idk.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Agresti, A., Coull, B.A.: Approximate is better than “exact" for interval estimation of binomial proportions. Am. Stat. 52(2) (1998)
Bao, H., Dong, L., Piao, S., Wei, F.: BEiT: BERT pre-training of image transformers. In: ICLR (2022)
Barbieri, F., Camacho-Collados, J., Neves, L., Espinosa-Anke, L.: TweetEval: unified benchmark and comparative evaluation for tweet classification. In: EMNLP (2020)
Brown, L.D., Cai, T.T., DasGupta, A.: Interval estimation for a binomial proportion. Stat. Sci. 16(2) (2001)
Chow, C.K.: On optimum recognition error and reject tradeoff. IEEE Trans. Inf. Theory 16(1), 41–46 (1970)
Clopper, C.J., Pearson, E.S.: The use of confidence or fiducial limits illustrated in the case of the binomial. Biometrika 26(4) (1934)
Darlow, L.N., Crowley, E.J., Antoniou, A., Storkey, A.J.: CINIC-10 is not imagenet or CIFAR-10. arXiv preprint arxiv:1810.03505 (2018)
Davis, J., Frank, L.: Revisiting Batch Normalization. In: ECCV (2022)
Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: ImageNet: a large-scale hierarchical image database. In: CVPR (2009)
Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. In: Human Language Technologies (2019)
El-Yaniv, R., Wiener, Y.: On the foundations of noise-free selective classification. J. Mach. Learn. Res. 11 (2010)
Franc, V., Prusa, D., Voracek, V.: Optimal Strategies for Reject Option Classifiers. arXiv preprint arxiv:2101.12523 (2021)
Geifman, Y., El-Yaniv, R.: Selective classification for deep neural networks. In: NIPS (2017)
Geifman, Y., El-Yaniv, R.: SelectiveNet: a deep neural network with an integrated reject option. In: ICML (2019)
Guo, C., Pleiss, G., Sun, Y., Weinberger, K.Q.: On calibration of modern neural networks. In: ICML (2017)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR (2016)
Krizhevsky, A., Hinton, G., et al.: Learning multiple layers of features from tiny images (2009)
Lu, Z., Sreekumar, G., Goodman, E., Banzhaf, W., Deb, K., Boddeti, V.: Neural Architecture Transfer. IEEE Trans. Pattern Anal. Mach. Intell. 43(09) (2021)
Maas, A.L., Daly, R.E., Pham, P.T., Huang, D., Ng, A.Y., Potts, C.: Learning word vectors for sentiment analysis. In: Human Language Technologies (2011)
Maji, S., Chicago, T., Rahtu, E., Kannala, J., Blaschkó, M., Vedaldi, A.: Fine-Grained Visual Classification of Aircraft. arXiv preprint arxiv:1306.5151 (2013)
Pietraszek, T.: Optimizing abstaining classifiers using ROC analysis. In: ICML (2005)
Tortorella, F.: An optimal reject rule for binary classifiers. In: Advances in Pattern Recognition (2000)
Zhang, X., Zhao, J., LeCun, Y.: Character-level convolutional networks for text classification. In: NIPS (2015)
Zhao, Y., Chen, J., Oymak, S.: On the Role of Dataset Quality and Heterogeneity in Model Confidence. arXiv preprint arxiv:2002.09831 (2020)
Acknowledgements
This research was supported by the U.S. Air Force Research Laboratory under Contract #GRT00054740 (Release #AFRL-2022-3339).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Kashani Motlagh, N., Davis, J., Anderson, T., Gwinnup, J. (2022). Learning When to Say “I Don’t Know". In: Bebis, G., et al. Advances in Visual Computing. ISVC 2022. Lecture Notes in Computer Science, vol 13598. Springer, Cham. https://doi.org/10.1007/978-3-031-20713-6_15
Download citation
DOI: https://doi.org/10.1007/978-3-031-20713-6_15
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-20712-9
Online ISBN: 978-3-031-20713-6
eBook Packages: Computer ScienceComputer Science (R0)