Nearest neighbors distance ratio open-set classifier


In this paper, we propose a novel multiclass classifier for the open-set recognition scenario. This scenario is the one in which there are no a priori training samples for some classes that might appear during testing. Usually, many applications are inherently open set. Consequently, successful closed-set solutions in the literature are not always suitable for real-world recognition problems. The proposed open-set classifier extends upon the Nearest-Neighbor (NN) classifier. Nearest neighbors are simple, parameter independent, multiclass, and widely used for closed-set problems. The proposed Open-Set NN (OSNN) method incorporates the ability of recognizing samples belonging to classes that are unknown at training time, being suitable for open-set recognition. In addition, we explore evaluation measures for open-set problems, properly measuring the resilience of methods to unknown classes during testing. For validation, we consider large freely-available benchmarks with different open-set recognition regimes and demonstrate that the proposed OSNN significantly outperforms their counterparts in the literature.


  1. Bartlett, P. L., & Wegkamp, M. H. (2008). Classification with a reject option using a hinge loss. Journal of Machine Learning Research, 9, 1823–1840.

    MathSciNet  MATH  Google Scholar 

  2. Bishop, C. M. (2006). Pattern recognition and machine learning (1st ed.). New York: Information Science and Statistics, Springer.

    Google Scholar 

  3. Boureau, Y. L., Bach, F., LeCun, Y., & Ponce, J. (2010). Learning mid-level features for recognition. In International conference on computer vision and pattern recognition (pp. 2559–2566). San Francisco, CA: IEEE Press.

  4. Cevikalp, H., & Triggs, B. (2012). Efficient object detection using cascades of nearest convex model classifiers. In International conference on computer vision and pattern recognition (pp. 3138–3145). Providence, RI: IEEE Press.

  5. Chang, C. C., & Lin, C. J. (2011). LIBSVM: A library for support vector machines. Transactions on Intelligent Systems and Technology, 2(3), 27:1–27:27.

    Google Scholar 

  6. Chen, P. H., Lin, C. J., & Schölkopf, B. (2005). A tutorial on nu-support vector machines. Applied Stochastic Models in Business and Industry, 21(2), 111–136.

    MathSciNet  Article  MATH  Google Scholar 

  7. Chew, S. W., Lucey, S., Lucey, P., Sridharan, S., & Cohn, J. F. (2012). Improved facial expression recognition via uni-hyperplane classification. In International conference on computer vision and pattern recognition (pp. 2554–2561). Providence, RI: IEEE Press.

  8. Chow, C. K. (1970). On optimum recognition error and reject tradeoff. Transactions on Information Theory, 16(1), 41–46.

    Article  MATH  Google Scholar 

  9. Costa, F.O., Eckmann, M., Scheirer, W. J., & Rocha, A. (2012). Open set source camera attribution. In Conference on graphics, patterns, and images (pp. 71–78). Ouro Preto: IEEE Press.

  10. Costa, F. O., Silva, E., Eckmann, M., Scheirer, W. J., & Rocha, A. (2014). Open set source camera attribution and device linking. Pattern Recognition Letters, 39, 92–101.

    Article  Google Scholar 

  11. Demšar, J. (2006). Statistical comparisons of classifiers over multiple data sets. Journal of Machine Learning Research, 7, 1–30.

    MathSciNet  MATH  Google Scholar 

  12. de Haan, L., & Ferreira, A. (2007). Extreme value theory: An introduction (1st ed.), Springer Series in Operations Research and Financial Engineering. New York: Springer.

  13. Dubuisson, B., & Masson, M. (1993). A statistical decision rule with incomplete knowledge about classes. Pattern Recognition, 26(1), 155–165.

    Article  Google Scholar 

  14. Frey, P. W., & Slate, D. J. (1991). Letter recognition using Holland-style adaptive classifiers. Machine Learning, 6(2), 161–182.

    Google Scholar 

  15. Fukunaga, K. (1990). Hypothesis testing. In Introduction to statistical pattern recognition, Chapter 3, Computer Science and Scientific Computing Series, 2nd ed. (pp. 51–123). London: Academic Press.

  16. Geusebroek, J. M., Burghouts, G. J., & Smeulders, A. W. M. (2005). The Amsterdam library of object images. International Journal of Computer Vision, 61(1), 103–112.

    Article  Google Scholar 

  17. Griffin, G., Holub, A., & Perona, P. (2007). Caltech-256 object category dataset. Tech. rep., California Institute of Technology.

  18. Heflin, B., Scheirer, W. J., & Boult, T. E. (2012). Detecting and classifying scars, marks, and tattoos found in the wild. In International conference on biometrics: Theory, applications and systems (pp. 31–38). Arlington, VA: IEEE Press.

  19. Hodge, V. J., & Austin, J. (2004). A survey of outlier detection methodologies. Artificial Intelligence Review, 22(2), 85–126.

    Article  MATH  Google Scholar 

  20. Jayadeva, K. R., & Chandra, S. (2007). Twin support vector machines for pattern classification. Transactions on Pattern Analysis and Machine Intelligence, 29(5), 905–910.

    Article  MATH  Google Scholar 

  21. Jin, H., Liu, Q., & Lu, H. (2004). Face detection using one-class-based support vectors. In International conference on automatic face and gesture recognition (pp. 457–462). Seoul: IEEE Press.

  22. Kadous, M. W. (2002). Temporal classification: Extending the classification paradigm to multivariate time series. PhD Thesis, The University of New South Wales, New South Wales, Australia.

  23. Kuncheva, L. I., & Hadjitodorov, S. T. (2004) Using diversity in cluster ensembles. In International conference on systems, man and cybernetics (Vol. 2, pp. 1214–1219). The Hague: IEEE Press.

  24. Lazebnik, S., Schmid, C., & Ponce, J. (2006). Beyond bags of features: Spatial pyramid matching for recognizing natural scene categories. In International conference on computer vision and pattern recognition (Vol. 2, pp. 2169–2178). New York, NY: IEEE Press.

  25. Lowe, D. (2004). Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision, 60(2), 91–110.

    Article  Google Scholar 

  26. Malisiewicz, T., Gupta, A., & Efros, A. A. (2011). Ensemble of exemplar-svms for object detection and beyond. In International Conference on Computer Vision (pp. 89–96). Barcelona: IEEE Press.

  27. Manevitz, L. M., & Yousef, M. (2002). One-class svms for document classification. Journal of Machine Learning Research, 2, 139–154.

    MATH  Google Scholar 

  28. Michie, D., Spiegelhalter, D. J., & Taylor, C. C. (1994). Machine learning, neural and statistical classification, Ellis Horwood Series in Artificial Intelligence. Upper Saddle River, NJ: Prentice Hall.

    Google Scholar 

  29. Muzzolini, R., Yang, Y. H., & Pierson, R. (1998). Classifier design with incomplete knowledge. Pattern Recognition, 31(4), 345–369.

    Article  Google Scholar 

  30. Nist, D., & Stew, H. (2006). Scalable recognition with a vocabulary tree. In International conference on computer vision and pattern recognition (Vol. 2, pp. 2162–2168). New York, NY: IEEE Press.

  31. Papa, J. P., Falcão, A. X., Miranda, P. A. V., Suzuki, C. T. N., & Mascarenhas, N. D. A. (2007). Design of robust pattern classifiers based on optimum-path forests. In International symposium on mathematical morphology, MCT/INPE, Rio de Janeiro (Vol. 1, pp. 337–348).

  32. Papa, J. P., Falcão, A. X., & Suzuki, C. T. N. (2009). Supervised pattern classification based on optimum-path forest. International Journal of Imaging Systems and Technology, 19(2), 120–131.

    Article  Google Scholar 

  33. Phillips, P. J., Grother, P., & Micheals, R. (2011). Evaluation methods in face recognition. In S. Z. Li & A. K. Jain (Eds.), Handbook of face recognition (pp. 329–348). New York: Springer.

    Google Scholar 

  34. Pietraszek, T. (2005). Optimizing abstaining classifiers using ROC analysis. In International conference on machine learning (pp. 665–672). Bonn: ACM Press.

  35. Pritsos, D. A., & Stamatatos, E. (2013). Open-set classification for automated genre identification. In P. Serdyukov, P. Braslavski, S. O. Kuznetsov, J. Kamps, S. Rüger, E. Agichtein, I. Segalovich, & E. Yilmaz (Eds.), Advances in information retrieval, Lecture Notes in Computer Science (Vol. 7814, pp. 207–217). Berlin: Springer.

    Google Scholar 

  36. Rocha, A., & Goldenstein, S. (2009). Multi-class from binary: Divide to conquer. In International conference on computer vision theory and applications (pp. 1–8). Lisboa: Springer.

  37. Rocha, A., & Goldenstein, S. (2014). Multiclass from binary: Expanding one-vs-all, one-vs-one and ECOC-based approaches. Transactions on Neural Networks and Learning Systems, 25(2), 289–302.

    Article  Google Scholar 

  38. Scheirer, W. J., Rocha, A., Micheals, R. J., & Boult, T. E. (2011). Meta-recognition: The theory and practice of recognition score analysis. Transactions on Pattern Analysis and Machine Intelligence, 33(8), 1689–1695.

    Article  Google Scholar 

  39. Scheirer, W. J., Rocha, A. R., Parris, J., & Boult, T. E. (2012). Learning for meta-recognition. Transactions on Information Forensics and Security, 7(4), 1214–1224.

    Article  Google Scholar 

  40. Scheirer, W. J., Rocha, A. R., Sapkota, A., & Boult, T. E. (2013). Towards open set recognition. Transactions on Pattern Analysis and Machine Intelligence, 35(7), 1757–1772.

    Article  Google Scholar 

  41. Scheirer, W. J., Jain, L. P., & Boult, T. E. (2014). Probability models for open set recognition. Transactions on Pattern Analysis and Machine Intelligence, 36(11), 2317–2324.

    Article  Google Scholar 

  42. Schölkopf, B., Platt, J. C., Shawe-Taylor, J., Smola, A. J., & Williamson, R. C. (1999). Estimating the support of a high-dimensional distribution. Tech. rep., Microsoft Research, Redmond, WA.

  43. Sokolova, M., & Lapalme, G. (2009). A systematic analysis of performance measures for classification tasks. Information Processing and Management, 45(4), 427–437.

    Article  Google Scholar 

  44. Stehling, R. O., Nascimento, M. A., & Falcão, A. X. (2002). A compact and efficient image retrieval approach based on border/interior pixel classification. International conference on information and knowledge management (pp. 102–109). McLean, VA: ACM Press.

    Google Scholar 

  45. van Gemert, J. C., Veenman, C. J., Smeulders, A. W. M., & Geusebroek, J. M. (2010). Visual word ambiguity. Transactions on Pattern Analysis and Machine Intelligence, 32(7), 1271–1283.

    Article  Google Scholar 

  46. Wu, M., & Ye, J. (2009). A small sphere and large margin approach for novelty detection using training data with outliers. Transactions on Pattern Analysis and Machine Intelligence, 31(11), 2088–2092.

    Article  Google Scholar 

  47. Zhou, X. S., & Huang, T. S. (2003). Relevance feedback in image retrieval: A comprehensive review. Multimedia Systems, 8(6), 536–544.

    Article  Google Scholar 

Download references

Author information



Corresponding author

Correspondence to Pedro R. Mendes Júnior.

Additional information

Editor: Hendrik Blockeel.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Mendes Júnior, P.R., de Souza, R.M., Werneck, R.d.O. et al. Nearest neighbors distance ratio open-set classifier. Mach Learn 106, 359–386 (2017).

Download citation


  • Open-set recognition
  • Nearest neighbor classifier
  • Open-set nearest-neighbor classifier
  • Nearest neighbors distance ratio
  • Open-set evaluation measures