Abstract
In this paper, we propose a novel multiclass classifier for the open-set recognition scenario. This scenario is the one in which there are no a priori training samples for some classes that might appear during testing. Usually, many applications are inherently open set. Consequently, successful closed-set solutions in the literature are not always suitable for real-world recognition problems. The proposed open-set classifier extends upon the Nearest-Neighbor (NN) classifier. Nearest neighbors are simple, parameter independent, multiclass, and widely used for closed-set problems. The proposed Open-Set NN (OSNN) method incorporates the ability of recognizing samples belonging to classes that are unknown at training time, being suitable for open-set recognition. In addition, we explore evaluation measures for open-set problems, properly measuring the resilience of methods to unknown classes during testing. For validation, we consider large freely-available benchmarks with different open-set recognition regimes and demonstrate that the proposed OSNN significantly outperforms their counterparts in the literature.
References
Bartlett, P. L., & Wegkamp, M. H. (2008). Classification with a reject option using a hinge loss. Journal of Machine Learning Research, 9, 1823–1840.
Bishop, C. M. (2006). Pattern recognition and machine learning (1st ed.). New York: Information Science and Statistics, Springer.
Boureau, Y. L., Bach, F., LeCun, Y., & Ponce, J. (2010). Learning mid-level features for recognition. In International conference on computer vision and pattern recognition (pp. 2559–2566). San Francisco, CA: IEEE Press.
Cevikalp, H., & Triggs, B. (2012). Efficient object detection using cascades of nearest convex model classifiers. In International conference on computer vision and pattern recognition (pp. 3138–3145). Providence, RI: IEEE Press.
Chang, C. C., & Lin, C. J. (2011). LIBSVM: A library for support vector machines. Transactions on Intelligent Systems and Technology, 2(3), 27:1–27:27.
Chen, P. H., Lin, C. J., & Schölkopf, B. (2005). A tutorial on nu-support vector machines. Applied Stochastic Models in Business and Industry, 21(2), 111–136.
Chew, S. W., Lucey, S., Lucey, P., Sridharan, S., & Cohn, J. F. (2012). Improved facial expression recognition via uni-hyperplane classification. In International conference on computer vision and pattern recognition (pp. 2554–2561). Providence, RI: IEEE Press.
Chow, C. K. (1970). On optimum recognition error and reject tradeoff. Transactions on Information Theory, 16(1), 41–46.
Costa, F.O., Eckmann, M., Scheirer, W. J., & Rocha, A. (2012). Open set source camera attribution. In Conference on graphics, patterns, and images (pp. 71–78). Ouro Preto: IEEE Press.
Costa, F. O., Silva, E., Eckmann, M., Scheirer, W. J., & Rocha, A. (2014). Open set source camera attribution and device linking. Pattern Recognition Letters, 39, 92–101.
Demšar, J. (2006). Statistical comparisons of classifiers over multiple data sets. Journal of Machine Learning Research, 7, 1–30.
de Haan, L., & Ferreira, A. (2007). Extreme value theory: An introduction (1st ed.), Springer Series in Operations Research and Financial Engineering. New York: Springer.
Dubuisson, B., & Masson, M. (1993). A statistical decision rule with incomplete knowledge about classes. Pattern Recognition, 26(1), 155–165.
Frey, P. W., & Slate, D. J. (1991). Letter recognition using Holland-style adaptive classifiers. Machine Learning, 6(2), 161–182.
Fukunaga, K. (1990). Hypothesis testing. In Introduction to statistical pattern recognition, Chapter 3, Computer Science and Scientific Computing Series, 2nd ed. (pp. 51–123). London: Academic Press.
Geusebroek, J. M., Burghouts, G. J., & Smeulders, A. W. M. (2005). The Amsterdam library of object images. International Journal of Computer Vision, 61(1), 103–112.
Griffin, G., Holub, A., & Perona, P. (2007). Caltech-256 object category dataset. Tech. rep., California Institute of Technology.
Heflin, B., Scheirer, W. J., & Boult, T. E. (2012). Detecting and classifying scars, marks, and tattoos found in the wild. In International conference on biometrics: Theory, applications and systems (pp. 31–38). Arlington, VA: IEEE Press.
Hodge, V. J., & Austin, J. (2004). A survey of outlier detection methodologies. Artificial Intelligence Review, 22(2), 85–126.
Jayadeva, K. R., & Chandra, S. (2007). Twin support vector machines for pattern classification. Transactions on Pattern Analysis and Machine Intelligence, 29(5), 905–910.
Jin, H., Liu, Q., & Lu, H. (2004). Face detection using one-class-based support vectors. In International conference on automatic face and gesture recognition (pp. 457–462). Seoul: IEEE Press.
Kadous, M. W. (2002). Temporal classification: Extending the classification paradigm to multivariate time series. PhD Thesis, The University of New South Wales, New South Wales, Australia.
Kuncheva, L. I., & Hadjitodorov, S. T. (2004) Using diversity in cluster ensembles. In International conference on systems, man and cybernetics (Vol. 2, pp. 1214–1219). The Hague: IEEE Press.
Lazebnik, S., Schmid, C., & Ponce, J. (2006). Beyond bags of features: Spatial pyramid matching for recognizing natural scene categories. In International conference on computer vision and pattern recognition (Vol. 2, pp. 2169–2178). New York, NY: IEEE Press.
Lowe, D. (2004). Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision, 60(2), 91–110.
Malisiewicz, T., Gupta, A., & Efros, A. A. (2011). Ensemble of exemplar-svms for object detection and beyond. In International Conference on Computer Vision (pp. 89–96). Barcelona: IEEE Press.
Manevitz, L. M., & Yousef, M. (2002). One-class svms for document classification. Journal of Machine Learning Research, 2, 139–154.
Michie, D., Spiegelhalter, D. J., & Taylor, C. C. (1994). Machine learning, neural and statistical classification, Ellis Horwood Series in Artificial Intelligence. Upper Saddle River, NJ: Prentice Hall.
Muzzolini, R., Yang, Y. H., & Pierson, R. (1998). Classifier design with incomplete knowledge. Pattern Recognition, 31(4), 345–369.
Nist, D., & Stew, H. (2006). Scalable recognition with a vocabulary tree. In International conference on computer vision and pattern recognition (Vol. 2, pp. 2162–2168). New York, NY: IEEE Press.
Papa, J. P., Falcão, A. X., Miranda, P. A. V., Suzuki, C. T. N., & Mascarenhas, N. D. A. (2007). Design of robust pattern classifiers based on optimum-path forests. In International symposium on mathematical morphology, MCT/INPE, Rio de Janeiro (Vol. 1, pp. 337–348).
Papa, J. P., Falcão, A. X., & Suzuki, C. T. N. (2009). Supervised pattern classification based on optimum-path forest. International Journal of Imaging Systems and Technology, 19(2), 120–131.
Phillips, P. J., Grother, P., & Micheals, R. (2011). Evaluation methods in face recognition. In S. Z. Li & A. K. Jain (Eds.), Handbook of face recognition (pp. 329–348). New York: Springer.
Pietraszek, T. (2005). Optimizing abstaining classifiers using ROC analysis. In International conference on machine learning (pp. 665–672). Bonn: ACM Press.
Pritsos, D. A., & Stamatatos, E. (2013). Open-set classification for automated genre identification. In P. Serdyukov, P. Braslavski, S. O. Kuznetsov, J. Kamps, S. Rüger, E. Agichtein, I. Segalovich, & E. Yilmaz (Eds.), Advances in information retrieval, Lecture Notes in Computer Science (Vol. 7814, pp. 207–217). Berlin: Springer.
Rocha, A., & Goldenstein, S. (2009). Multi-class from binary: Divide to conquer. In International conference on computer vision theory and applications (pp. 1–8). Lisboa: Springer.
Rocha, A., & Goldenstein, S. (2014). Multiclass from binary: Expanding one-vs-all, one-vs-one and ECOC-based approaches. Transactions on Neural Networks and Learning Systems, 25(2), 289–302.
Scheirer, W. J., Rocha, A., Micheals, R. J., & Boult, T. E. (2011). Meta-recognition: The theory and practice of recognition score analysis. Transactions on Pattern Analysis and Machine Intelligence, 33(8), 1689–1695.
Scheirer, W. J., Rocha, A. R., Parris, J., & Boult, T. E. (2012). Learning for meta-recognition. Transactions on Information Forensics and Security, 7(4), 1214–1224.
Scheirer, W. J., Rocha, A. R., Sapkota, A., & Boult, T. E. (2013). Towards open set recognition. Transactions on Pattern Analysis and Machine Intelligence, 35(7), 1757–1772.
Scheirer, W. J., Jain, L. P., & Boult, T. E. (2014). Probability models for open set recognition. Transactions on Pattern Analysis and Machine Intelligence, 36(11), 2317–2324.
Schölkopf, B., Platt, J. C., Shawe-Taylor, J., Smola, A. J., & Williamson, R. C. (1999). Estimating the support of a high-dimensional distribution. Tech. rep., Microsoft Research, Redmond, WA.
Sokolova, M., & Lapalme, G. (2009). A systematic analysis of performance measures for classification tasks. Information Processing and Management, 45(4), 427–437.
Stehling, R. O., Nascimento, M. A., & Falcão, A. X. (2002). A compact and efficient image retrieval approach based on border/interior pixel classification. International conference on information and knowledge management (pp. 102–109). McLean, VA: ACM Press.
van Gemert, J. C., Veenman, C. J., Smeulders, A. W. M., & Geusebroek, J. M. (2010). Visual word ambiguity. Transactions on Pattern Analysis and Machine Intelligence, 32(7), 1271–1283.
Wu, M., & Ye, J. (2009). A small sphere and large margin approach for novelty detection using training data with outliers. Transactions on Pattern Analysis and Machine Intelligence, 31(11), 2088–2092.
Zhou, X. S., & Huang, T. S. (2003). Relevance feedback in image retrieval: A comprehensive review. Multimedia Systems, 8(6), 536–544.
Author information
Authors and Affiliations
Corresponding author
Additional information
Editor: Hendrik Blockeel.
Rights and permissions
About this article
Cite this article
Mendes Júnior, P.R., de Souza, R.M., Werneck, R.d.O. et al. Nearest neighbors distance ratio open-set classifier. Mach Learn 106, 359–386 (2017). https://doi.org/10.1007/s10994-016-5610-8
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10994-016-5610-8
Keywords
- Open-set recognition
- Nearest neighbor classifier
- Open-set nearest-neighbor classifier
- Nearest neighbors distance ratio
- Open-set evaluation measures