Machine Learning

, Volume 106, Issue 3, pp 359–386 | Cite as

Nearest neighbors distance ratio open-set classifier

  • Pedro R. Mendes JúniorEmail author
  • Roberto M. de Souza
  • Rafael de O. Werneck
  • Bernardo V. Stein
  • Daniel V. Pazinato
  • Waldir R. de Almeida
  • Otávio A. B. Penatti
  • Ricardo da S. Torres
  • Anderson Rocha


In this paper, we propose a novel multiclass classifier for the open-set recognition scenario. This scenario is the one in which there are no a priori training samples for some classes that might appear during testing. Usually, many applications are inherently open set. Consequently, successful closed-set solutions in the literature are not always suitable for real-world recognition problems. The proposed open-set classifier extends upon the Nearest-Neighbor (NN) classifier. Nearest neighbors are simple, parameter independent, multiclass, and widely used for closed-set problems. The proposed Open-Set NN (OSNN) method incorporates the ability of recognizing samples belonging to classes that are unknown at training time, being suitable for open-set recognition. In addition, we explore evaluation measures for open-set problems, properly measuring the resilience of methods to unknown classes during testing. For validation, we consider large freely-available benchmarks with different open-set recognition regimes and demonstrate that the proposed OSNN significantly outperforms their counterparts in the literature.


Open-set recognition Nearest neighbor classifier Open-set nearest-neighbor classifier Nearest neighbors distance ratio Open-set evaluation measures 


  1. Bartlett, P. L., & Wegkamp, M. H. (2008). Classification with a reject option using a hinge loss. Journal of Machine Learning Research, 9, 1823–1840.MathSciNetzbMATHGoogle Scholar
  2. Bishop, C. M. (2006). Pattern recognition and machine learning (1st ed.). New York: Information Science and Statistics, Springer.zbMATHGoogle Scholar
  3. Boureau, Y. L., Bach, F., LeCun, Y., & Ponce, J. (2010). Learning mid-level features for recognition. In International conference on computer vision and pattern recognition (pp. 2559–2566). San Francisco, CA: IEEE Press.Google Scholar
  4. Cevikalp, H., & Triggs, B. (2012). Efficient object detection using cascades of nearest convex model classifiers. In International conference on computer vision and pattern recognition (pp. 3138–3145). Providence, RI: IEEE Press.Google Scholar
  5. Chang, C. C., & Lin, C. J. (2011). LIBSVM: A library for support vector machines. Transactions on Intelligent Systems and Technology, 2(3), 27:1–27:27.Google Scholar
  6. Chen, P. H., Lin, C. J., & Schölkopf, B. (2005). A tutorial on nu-support vector machines. Applied Stochastic Models in Business and Industry, 21(2), 111–136.MathSciNetCrossRefzbMATHGoogle Scholar
  7. Chew, S. W., Lucey, S., Lucey, P., Sridharan, S., & Cohn, J. F. (2012). Improved facial expression recognition via uni-hyperplane classification. In International conference on computer vision and pattern recognition (pp. 2554–2561). Providence, RI: IEEE Press.Google Scholar
  8. Chow, C. K. (1970). On optimum recognition error and reject tradeoff. Transactions on Information Theory, 16(1), 41–46.CrossRefzbMATHGoogle Scholar
  9. Costa, F.O., Eckmann, M., Scheirer, W. J., & Rocha, A. (2012). Open set source camera attribution. In Conference on graphics, patterns, and images (pp. 71–78). Ouro Preto: IEEE Press.Google Scholar
  10. Costa, F. O., Silva, E., Eckmann, M., Scheirer, W. J., & Rocha, A. (2014). Open set source camera attribution and device linking. Pattern Recognition Letters, 39, 92–101.CrossRefGoogle Scholar
  11. Demšar, J. (2006). Statistical comparisons of classifiers over multiple data sets. Journal of Machine Learning Research, 7, 1–30.MathSciNetzbMATHGoogle Scholar
  12. de Haan, L., & Ferreira, A. (2007). Extreme value theory: An introduction (1st ed.), Springer Series in Operations Research and Financial Engineering. New York: Springer.Google Scholar
  13. Dubuisson, B., & Masson, M. (1993). A statistical decision rule with incomplete knowledge about classes. Pattern Recognition, 26(1), 155–165.CrossRefGoogle Scholar
  14. Frey, P. W., & Slate, D. J. (1991). Letter recognition using Holland-style adaptive classifiers. Machine Learning, 6(2), 161–182.Google Scholar
  15. Fukunaga, K. (1990). Hypothesis testing. In Introduction to statistical pattern recognition, Chapter 3, Computer Science and Scientific Computing Series, 2nd ed. (pp. 51–123). London: Academic Press.Google Scholar
  16. Geusebroek, J. M., Burghouts, G. J., & Smeulders, A. W. M. (2005). The Amsterdam library of object images. International Journal of Computer Vision, 61(1), 103–112.CrossRefGoogle Scholar
  17. Griffin, G., Holub, A., & Perona, P. (2007). Caltech-256 object category dataset. Tech. rep., California Institute of Technology.Google Scholar
  18. Heflin, B., Scheirer, W. J., & Boult, T. E. (2012). Detecting and classifying scars, marks, and tattoos found in the wild. In International conference on biometrics: Theory, applications and systems (pp. 31–38). Arlington, VA: IEEE Press.Google Scholar
  19. Hodge, V. J., & Austin, J. (2004). A survey of outlier detection methodologies. Artificial Intelligence Review, 22(2), 85–126.CrossRefzbMATHGoogle Scholar
  20. Jayadeva, K. R., & Chandra, S. (2007). Twin support vector machines for pattern classification. Transactions on Pattern Analysis and Machine Intelligence, 29(5), 905–910.CrossRefzbMATHGoogle Scholar
  21. Jin, H., Liu, Q., & Lu, H. (2004). Face detection using one-class-based support vectors. In International conference on automatic face and gesture recognition (pp. 457–462). Seoul: IEEE Press.Google Scholar
  22. Kadous, M. W. (2002). Temporal classification: Extending the classification paradigm to multivariate time series. PhD Thesis, The University of New South Wales, New South Wales, Australia.Google Scholar
  23. Kuncheva, L. I., & Hadjitodorov, S. T. (2004) Using diversity in cluster ensembles. In International conference on systems, man and cybernetics (Vol. 2, pp. 1214–1219). The Hague: IEEE Press.Google Scholar
  24. Lazebnik, S., Schmid, C., & Ponce, J. (2006). Beyond bags of features: Spatial pyramid matching for recognizing natural scene categories. In International conference on computer vision and pattern recognition (Vol. 2, pp. 2169–2178). New York, NY: IEEE Press.Google Scholar
  25. Lowe, D. (2004). Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision, 60(2), 91–110.CrossRefGoogle Scholar
  26. Malisiewicz, T., Gupta, A., & Efros, A. A. (2011). Ensemble of exemplar-svms for object detection and beyond. In International Conference on Computer Vision (pp. 89–96). Barcelona: IEEE Press.Google Scholar
  27. Manevitz, L. M., & Yousef, M. (2002). One-class svms for document classification. Journal of Machine Learning Research, 2, 139–154.zbMATHGoogle Scholar
  28. Michie, D., Spiegelhalter, D. J., & Taylor, C. C. (1994). Machine learning, neural and statistical classification, Ellis Horwood Series in Artificial Intelligence. Upper Saddle River, NJ: Prentice Hall.Google Scholar
  29. Muzzolini, R., Yang, Y. H., & Pierson, R. (1998). Classifier design with incomplete knowledge. Pattern Recognition, 31(4), 345–369.CrossRefGoogle Scholar
  30. Nist, D., & Stew, H. (2006). Scalable recognition with a vocabulary tree. In International conference on computer vision and pattern recognition (Vol. 2, pp. 2162–2168). New York, NY: IEEE Press.Google Scholar
  31. Papa, J. P., Falcão, A. X., Miranda, P. A. V., Suzuki, C. T. N., & Mascarenhas, N. D. A. (2007). Design of robust pattern classifiers based on optimum-path forests. In International symposium on mathematical morphology, MCT/INPE, Rio de Janeiro (Vol. 1, pp. 337–348).Google Scholar
  32. Papa, J. P., Falcão, A. X., & Suzuki, C. T. N. (2009). Supervised pattern classification based on optimum-path forest. International Journal of Imaging Systems and Technology, 19(2), 120–131.CrossRefGoogle Scholar
  33. Phillips, P. J., Grother, P., & Micheals, R. (2011). Evaluation methods in face recognition. In S. Z. Li & A. K. Jain (Eds.), Handbook of face recognition (pp. 329–348). New York: Springer.Google Scholar
  34. Pietraszek, T. (2005). Optimizing abstaining classifiers using ROC analysis. In International conference on machine learning (pp. 665–672). Bonn: ACM Press.Google Scholar
  35. Pritsos, D. A., & Stamatatos, E. (2013). Open-set classification for automated genre identification. In P. Serdyukov, P. Braslavski, S. O. Kuznetsov, J. Kamps, S. Rüger, E. Agichtein, I. Segalovich, & E. Yilmaz (Eds.), Advances in information retrieval, Lecture Notes in Computer Science (Vol. 7814, pp. 207–217). Berlin: Springer.CrossRefGoogle Scholar
  36. Rocha, A., & Goldenstein, S. (2009). Multi-class from binary: Divide to conquer. In International conference on computer vision theory and applications (pp. 1–8). Lisboa: Springer.Google Scholar
  37. Rocha, A., & Goldenstein, S. (2014). Multiclass from binary: Expanding one-vs-all, one-vs-one and ECOC-based approaches. Transactions on Neural Networks and Learning Systems, 25(2), 289–302.CrossRefGoogle Scholar
  38. Scheirer, W. J., Rocha, A., Micheals, R. J., & Boult, T. E. (2011). Meta-recognition: The theory and practice of recognition score analysis. Transactions on Pattern Analysis and Machine Intelligence, 33(8), 1689–1695.CrossRefGoogle Scholar
  39. Scheirer, W. J., Rocha, A. R., Parris, J., & Boult, T. E. (2012). Learning for meta-recognition. Transactions on Information Forensics and Security, 7(4), 1214–1224.CrossRefGoogle Scholar
  40. Scheirer, W. J., Rocha, A. R., Sapkota, A., & Boult, T. E. (2013). Towards open set recognition. Transactions on Pattern Analysis and Machine Intelligence, 35(7), 1757–1772.CrossRefGoogle Scholar
  41. Scheirer, W. J., Jain, L. P., & Boult, T. E. (2014). Probability models for open set recognition. Transactions on Pattern Analysis and Machine Intelligence, 36(11), 2317–2324.CrossRefGoogle Scholar
  42. Schölkopf, B., Platt, J. C., Shawe-Taylor, J., Smola, A. J., & Williamson, R. C. (1999). Estimating the support of a high-dimensional distribution. Tech. rep., Microsoft Research, Redmond, WA.Google Scholar
  43. Sokolova, M., & Lapalme, G. (2009). A systematic analysis of performance measures for classification tasks. Information Processing and Management, 45(4), 427–437.CrossRefGoogle Scholar
  44. Stehling, R. O., Nascimento, M. A., & Falcão, A. X. (2002). A compact and efficient image retrieval approach based on border/interior pixel classification. International conference on information and knowledge management (pp. 102–109). McLean, VA: ACM Press.Google Scholar
  45. van Gemert, J. C., Veenman, C. J., Smeulders, A. W. M., & Geusebroek, J. M. (2010). Visual word ambiguity. Transactions on Pattern Analysis and Machine Intelligence, 32(7), 1271–1283.CrossRefGoogle Scholar
  46. Wu, M., & Ye, J. (2009). A small sphere and large margin approach for novelty detection using training data with outliers. Transactions on Pattern Analysis and Machine Intelligence, 31(11), 2088–2092.CrossRefGoogle Scholar
  47. Zhou, X. S., & Huang, T. S. (2003). Relevance feedback in image retrieval: A comprehensive review. Multimedia Systems, 8(6), 536–544.CrossRefGoogle Scholar

Copyright information

© The Author(s) 2016

Authors and Affiliations

  1. 1.RECOD Lab., Institute of Computing (IC)University of Campinas (UNICAMP)CampinasBrazil
  2. 2.Faculty of Electrical Engineering and Computing (FEEC)University of Campinas (UNICAMP)CampinasBrazil
  3. 3.SAMSUNG Research InstituteAdvanced Technologies GroupCampinasBrazil

Personalised recommendations