Criteria of efficiency for set-valued classification


We study optimal conformity measures for various criteria of efficiency of set-valued classification in an idealised setting. This leads to an important class of criteria of efficiency that we call probabilistic and argue for; it turns out that the most standard criteria of efficiency used in literature on conformal prediction are not probabilistic unless the problem of classification is binary. We consider both unconditional and label-conditional conformal prediction.


  1. 1.

    Balasubramanian, V.N., Ho, S.S., Vovk, V. (eds.): Conformal prediction for reliable machine learning: theory, adaptations, and applications. Elsevier, Amsterdam (2014)

  2. 2.

    Dawid, A.P.: Probability Forecasting. In: Kotz, S., Balakrishnan, N., Read, C. B., Vidakovic, B., Johnson, N.L. (eds.) Encyclopedia of Statistical Sciences. 2nd edn., vol. 10, pp 6445–6452. Wiley, Hoboken (2006)

  3. 3.

    Fedorova, V., Gammerman, A., Nouretdinov, I., Vovk, V.: Hypergraphical conformal predictors. Int. J. Artif. Intell. Tools 24(6), 1560,003 (2015). COPA 2013 Special Issue

    Article  MATH  Google Scholar 

  4. 4.

    Gneiting, T., Raftery, A.E.: Strictly proper scoring rules, prediction, and estimation. J. Am. Stat. Assoc. 102, 359–378 (2007)

    MathSciNet  Article  MATH  Google Scholar 

  5. 5.

    Johansson, U., König, R., Löfström, T., Boström, H.: Evolved Decision Trees as Conformal Predictors. In: de la Fraga, L.G. (ed.) Proceedings of the 2013 IEEE Conference on Evolutionary Computation, vol. 1, pp 1794–1801. Cancun, Mexico (2013)

  6. 6.

    Karp, R.M.: Reducibility among Combinatorial Problems. In: Miller, R.E., Thatcher, J.W. (eds.) Complexity of Computer Computations, pp 85–103. Plenum Press, New York (1972)

    Google Scholar 

  7. 7.

    Le Cun, Y., Boser, B.E., Denker, J.S., Henderson, D., Howard, R.E., Hubbard, W.E., Jackel, L.D.: Handwritten Digit Recognition with a Back-Propagation Network. In: Touretzky, D.S. (ed.) Advances in Neural Information Processing Systems 2, pp 396–404. Morgan Kaufmann, San Francisco (1990)

    Google Scholar 

  8. 8.

    Lehmann, E.L.: Testing Statistical Hypotheses, 2nd edn. Springer, New York (1986)

    Book  MATH  Google Scholar 

  9. 9.

    Lei, J.: Classification with confidence. Biometrika 101, 755–769 (2014)

    MathSciNet  Article  MATH  Google Scholar 

  10. 10.

    Lei, J., Robins, J., Wasserman, L.: Distribution free prediction sets. J. Am. Stat. Assoc. 108, 278–287 (2013)

    MathSciNet  Article  MATH  Google Scholar 

  11. 11.

    Lei, J., Wasserman, L.: Distribution free prediction bands for nonparametric regression. J. R. Stat. Soc. B 76, 71–96 (2014)

    MathSciNet  Article  Google Scholar 

  12. 12.

    Martello, S., Toth, P.: Knapsack Problems: Algorithms and Computer Implementations. Wiley, Chichester (1990)

    MATH  Google Scholar 

  13. 13.

    Melluish, T., Saunders, C., Nouretdinov, I., Vovk, V.: Comparing the Bayes and Typicalness Frameworks. In: De Raedt, L., Flach, P.A. (eds.) Proceedings of the Twelfth European Conference on Machine Learning, Lecture Notes in Computer Science, vol. 2167, pp 360–371. Springer, Heidelberg (2001)

  14. 14.

    Papadopoulos, H., Gammerman, A., Vovk, V. (eds.): Special Issue of the Annals of Mathematics and Artificial Intelligence on Conformal Prediction and its Applications, vol. 74(1–2). Springer (2015)

  15. 15.

    Sadinle, M., Lei, J., Wasserman, L.: Least ambiguous set-valued classifiers with bounded error levels. Tech. Rep. arXiv:1609.00451v1 [stat.ME] e-Print archive (2016)

  16. 16.

    Saunders, C., Gammerman, A., Vovk, V.: Transduction with Confidence and Credibility. In: Dean, T. (ed.) Proceedings of the Sixteenth International Joint Conference on Artificial Intelligence, vol. 2, pp 722–726. Morgan Kaufmann, San Francisco (1999)

  17. 17.

    Smith, J., Nouretdinov, I., Craddock, R., Offer, C., Gammerman, A.: Anomaly Detection of Trajectories with Kernel Density Estimation by Conformal Prediction. In: Iliadis, L., Maglogiannis, I., Papadopoulos, H., Sioutas, S., Makris, C. (eds.) AIAI Workshops, COPA 2014, IFIP Advances in Information and Communication Technology, vol. 437, pp 271–280 (2014)

  18. 18.

    Vovk, V., Fedorova, V., Nouretdinov, I., Gammerman, A.: Criteria of Efficiency for Conformal Prediction. In: Gammerman, A., Luo, Z., Vega, J., Vovk, V. (eds.) Proceedings of the Fifth International Symposium on Conformal and Probabilistic Prediction with Applications (COPA 2016), Lecture Notes in Artificial Intelligence, vol. 9653, pp 23–39. Springer, Switzerland (2016)

  19. 19.

    Vovk, V., Gammerman, A., Shafer, G.: Algorithmic Learning in a Random World. Springer, New York (2005)

    MATH  Google Scholar 

  20. 20.

    Vovk, V., Petej, I., Fedorova, V.: From Conformal to Probabilistic Prediction. In: Iliadis, L. , Maglogiannis, I., Papadopoulos, H., Sioutas, S., Makrism, C. (eds.) AIAI Workshops, COPA 2014, IFIP Advances in Information and Communication Technology, vol. 437, pp 221–230 (2014)

Download references


We are grateful to the reviewers of the conference and journal versions of this paper for their helpful comments.

Author information



Corresponding author

Correspondence to Vladimir Vovk.

Additional information

A preliminary version of this paper was published as Working Paper 11 of the On-line Compression Modelling project (New Series),, in April 2014. Its conference version [18] was published in the Proceedings of the Fifth Symposium on Conformal and Probabilistic Prediction and Their Applications (COPA 2016, Madrid, April 2016) under the title “Criteria of efficiency for conformal prediction”. This journal version also incorporates (in Section ??) some material of our paper [20] in COPA 2014. This work was partially supported by EPSRC (grant EP/K033344/1), the Air Force Office of Scientific Research (grant “Semantic Completions”), and the EU Horizon 2020 Research and Innovation programme (grant 671555).

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (, which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Vovk, V., Nouretdinov, I., Fedorova, V. et al. Criteria of efficiency for set-valued classification. Ann Math Artif Intell 81, 21–46 (2017).

Download citation


  • Conformal prediction
  • Label-conditional conformal prediction
  • Predictive efficiency
  • Informational efficiency

Mathematics Subject Classification (2010)

  • 68T05
  • 68Q32
  • 62G15