Criteria of efficiency for set-valued classification

  • Vladimir Vovk
  • Ilia Nouretdinov
  • Valentina Fedorova
  • Ivan Petej
  • Alex Gammerman
Open Access


We study optimal conformity measures for various criteria of efficiency of set-valued classification in an idealised setting. This leads to an important class of criteria of efficiency that we call probabilistic and argue for; it turns out that the most standard criteria of efficiency used in literature on conformal prediction are not probabilistic unless the problem of classification is binary. We consider both unconditional and label-conditional conformal prediction.


Conformal prediction Label-conditional conformal prediction Predictive efficiency Informational efficiency 

Mathematics Subject Classification (2010)

68T05 68Q32 62G15 



We are grateful to the reviewers of the conference and journal versions of this paper for their helpful comments.


  1. 1.
    Balasubramanian, V.N., Ho, S.S., Vovk, V. (eds.): Conformal prediction for reliable machine learning: theory, adaptations, and applications. Elsevier, Amsterdam (2014)Google Scholar
  2. 2.
    Dawid, A.P.: Probability Forecasting. In: Kotz, S., Balakrishnan, N., Read, C. B., Vidakovic, B., Johnson, N.L. (eds.) Encyclopedia of Statistical Sciences. 2nd edn., vol. 10, pp 6445–6452. Wiley, Hoboken (2006)Google Scholar
  3. 3.
    Fedorova, V., Gammerman, A., Nouretdinov, I., Vovk, V.: Hypergraphical conformal predictors. Int. J. Artif. Intell. Tools 24(6), 1560,003 (2015). COPA 2013 Special IssueCrossRefzbMATHGoogle Scholar
  4. 4.
    Gneiting, T., Raftery, A.E.: Strictly proper scoring rules, prediction, and estimation. J. Am. Stat. Assoc. 102, 359–378 (2007)MathSciNetCrossRefzbMATHGoogle Scholar
  5. 5.
    Johansson, U., König, R., Löfström, T., Boström, H.: Evolved Decision Trees as Conformal Predictors. In: de la Fraga, L.G. (ed.) Proceedings of the 2013 IEEE Conference on Evolutionary Computation, vol. 1, pp 1794–1801. Cancun, Mexico (2013)Google Scholar
  6. 6.
    Karp, R.M.: Reducibility among Combinatorial Problems. In: Miller, R.E., Thatcher, J.W. (eds.) Complexity of Computer Computations, pp 85–103. Plenum Press, New York (1972)CrossRefGoogle Scholar
  7. 7.
    Le Cun, Y., Boser, B.E., Denker, J.S., Henderson, D., Howard, R.E., Hubbard, W.E., Jackel, L.D.: Handwritten Digit Recognition with a Back-Propagation Network. In: Touretzky, D.S. (ed.) Advances in Neural Information Processing Systems 2, pp 396–404. Morgan Kaufmann, San Francisco (1990)Google Scholar
  8. 8.
    Lehmann, E.L.: Testing Statistical Hypotheses, 2nd edn. Springer, New York (1986)CrossRefzbMATHGoogle Scholar
  9. 9.
    Lei, J.: Classification with confidence. Biometrika 101, 755–769 (2014)MathSciNetCrossRefzbMATHGoogle Scholar
  10. 10.
    Lei, J., Robins, J., Wasserman, L.: Distribution free prediction sets. J. Am. Stat. Assoc. 108, 278–287 (2013)MathSciNetCrossRefzbMATHGoogle Scholar
  11. 11.
    Lei, J., Wasserman, L.: Distribution free prediction bands for nonparametric regression. J. R. Stat. Soc. B 76, 71–96 (2014)MathSciNetCrossRefGoogle Scholar
  12. 12.
    Martello, S., Toth, P.: Knapsack Problems: Algorithms and Computer Implementations. Wiley, Chichester (1990)zbMATHGoogle Scholar
  13. 13.
    Melluish, T., Saunders, C., Nouretdinov, I., Vovk, V.: Comparing the Bayes and Typicalness Frameworks. In: De Raedt, L., Flach, P.A. (eds.) Proceedings of the Twelfth European Conference on Machine Learning, Lecture Notes in Computer Science, vol. 2167, pp 360–371. Springer, Heidelberg (2001)Google Scholar
  14. 14.
    Papadopoulos, H., Gammerman, A., Vovk, V. (eds.): Special Issue of the Annals of Mathematics and Artificial Intelligence on Conformal Prediction and its Applications, vol. 74(1–2). Springer (2015)Google Scholar
  15. 15.
    Sadinle, M., Lei, J., Wasserman, L.: Least ambiguous set-valued classifiers with bounded error levels. Tech. Rep. arXiv:1609.00451v1 [stat.ME] e-Print archive (2016)
  16. 16.
    Saunders, C., Gammerman, A., Vovk, V.: Transduction with Confidence and Credibility. In: Dean, T. (ed.) Proceedings of the Sixteenth International Joint Conference on Artificial Intelligence, vol. 2, pp 722–726. Morgan Kaufmann, San Francisco (1999)Google Scholar
  17. 17.
    Smith, J., Nouretdinov, I., Craddock, R., Offer, C., Gammerman, A.: Anomaly Detection of Trajectories with Kernel Density Estimation by Conformal Prediction. In: Iliadis, L., Maglogiannis, I., Papadopoulos, H., Sioutas, S., Makris, C. (eds.) AIAI Workshops, COPA 2014, IFIP Advances in Information and Communication Technology, vol. 437, pp 271–280 (2014)Google Scholar
  18. 18.
    Vovk, V., Fedorova, V., Nouretdinov, I., Gammerman, A.: Criteria of Efficiency for Conformal Prediction. In: Gammerman, A., Luo, Z., Vega, J., Vovk, V. (eds.) Proceedings of the Fifth International Symposium on Conformal and Probabilistic Prediction with Applications (COPA 2016), Lecture Notes in Artificial Intelligence, vol. 9653, pp 23–39. Springer, Switzerland (2016)Google Scholar
  19. 19.
    Vovk, V., Gammerman, A., Shafer, G.: Algorithmic Learning in a Random World. Springer, New York (2005)zbMATHGoogle Scholar
  20. 20.
    Vovk, V., Petej, I., Fedorova, V.: From Conformal to Probabilistic Prediction. In: Iliadis, L. , Maglogiannis, I., Papadopoulos, H., Sioutas, S., Makrism, C. (eds.) AIAI Workshops, COPA 2014, IFIP Advances in Information and Communication Technology, vol. 437, pp 221–230 (2014)Google Scholar

Copyright information

© The Author(s) 2017

Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (, which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Authors and Affiliations

  1. 1.Computer Learning Research Centre, Department of Computer ScienceRoyal Holloway, University of LondonEghamUK
  2. 2.YandexRussia

Personalised recommendations