Criteria of Efficiency for Conformal Prediction

  • Vladimir Vovk
  • Valentina Fedorova
  • Ilia Nouretdinov
  • Alexander Gammerman
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9653)


We study optimal conformity measures for various criteria of efficiency in an idealised setting. This leads to an important class of criteria of efficiency that we call probabilistic; it turns out that the most standard criteria of efficiency used in literature on conformal prediction are not probabilistic.


Conformal prediction Predictive efficiency Informational efficiency 



We are grateful to the reviewers for helpful comments. This work was partially supported by EPSRC (grant EP/K033344/1), the Air Force Office of Scientific Research (grant “Semantic Completions”), and the EU Horizon 2020 Research and Innovation programme (grant 671555).


  1. 1.
    Balasubramanian, V.N., Ho, S.S., Vovk, V. (eds.): Conformal Prediction for Reliable Machine Learning: Theory, Adaptations, and Applications. Elsevier, Amsterdam (2014)Google Scholar
  2. 2.
    Dawid, A.P.: Probability forecasting. In: Kotz, S., Balakrishnan, N., Read, C.B., Vidakovic, B., Johnson, N.L. (eds.) Encyclopedia of Statistical Sciences, vol. 10, 2nd edn, pp. 6445–6452. Wiley, Hoboken, NJ (2006)Google Scholar
  3. 3.
    Fedorova, V., Gammerman, A., Nouretdinov, I., Vovk, V.: Conformal prediction under hypergraphical models. In: Papadopoulos, H., Andreou, A.S., Iliadis, L., Maglogiannis, I. (eds.) AIAI 2013. IFIP AICT, vol. 412, pp. 371–383. Springer, Heidelberg (2013)CrossRefGoogle Scholar
  4. 4.
    Gneiting, T., Raftery, A.E.: Strictly proper scoring rules, prediction, and estimation. J. Am. Stat. Assoc. 102, 359–378 (2007)MathSciNetCrossRefzbMATHGoogle Scholar
  5. 5.
    Johansson, U., König, R., Löfström, T., Boström, H.: Evolved decision trees as conformal predictors. In: de la Fraga, L.G. (ed.) Proceedings of the 2013 IEEE Conference on Evolutionary Computation, vol. 1, pp. 794–1801. Cancun, Mexico (2013)Google Scholar
  6. 6.
    Le Cun, Y., Boser, B.E., Denker, J.S., Henderson, D., Howard, R.E., Hubbard, W.E., Jackel, L.D.: Handwritten digit recognition with a back-propagation network. In: Touretzky, D.S. (ed.) Advances in Neural Information Processing Systems 2, pp. 396–404. Morgan Kaufmann, San Francisco, CA (1990)Google Scholar
  7. 7.
    Lehmann, E.L.: Testing Statistical Hypotheses, 2nd edn. Springer, New York (1986)CrossRefzbMATHGoogle Scholar
  8. 8.
    Lei, J., Robins, J., Wasserman, L.: Distribution free prediction sets. J. Am. Stat. Assoc. 108, 278–287 (2013)MathSciNetCrossRefzbMATHGoogle Scholar
  9. 9.
    Lei, J., Wasserman, L.: Distribution free prediction bands for nonparametric regression. J. Roy. Stat. Soc. B 76, 71–96 (2014)MathSciNetCrossRefGoogle Scholar
  10. 10.
    Melluish, T., Saunders, C., Nouretdinov, I., Vovk, V.: Comparing the Bayes and typicalness frameworks. In: Flach, P.A., De Raedt, L. (eds.) ECML 2001. LNCS (LNAI), vol. 2167, pp. 360–371. Springer, Heidelberg (2001)CrossRefGoogle Scholar
  11. 11.
    Papadopoulos, H., Gammerman, A., Vovk, V.: Special issue of the conformal prediction and its applications. Ann. Math. Artif. Intell. 74(1–2), 1–7 (2015). SpringerMathSciNetCrossRefzbMATHGoogle Scholar
  12. 12.
    Saunders, C., Gammerman, A., Vovk, V.: Transduction with confidence and credibility. In: Dean, T. (ed.) Proceedings of the Sixteenth International Joint Conference on Artificial Intelligence, vol. 2, pp. 722–726. Morgan Kaufmann, San Francisco, CA (1999)Google Scholar
  13. 13.
    Smith, J., Nouretdinov, I., Craddock, R., Offer, C., Gammerman, A.: Anomaly detection of trajectories with kernel density estimation by conformal prediction. In: Iliadis, L., Maglogiannis, I., Papadopoulos, H., Sioutas, S., Makris, C. (eds.) Artificial Intelligence Applications and Innovations. IFIP AICT, vol. 437, pp. 271–280. Springer, Heidelberg (2014)Google Scholar
  14. 14.
    Vovk, V., Gammerman, A., Shafer, G.: Algorithmic Learning in a Random World. Springer, New York (2005)zbMATHGoogle Scholar
  15. 15.
    Vovk, V., Petej, I., Fedorova, V.: From conformal to probabilistic prediction. In: Iliadis, L., Maglogiannis, I., Papadopoulos, H., Sioutas, S., Makris, C. (eds.) Artificial Intelligence Applications and Innovations. IFIP AICT, vol. 437, pp. 221–230. Springer, Heidelberg (2014)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2016

Authors and Affiliations

  • Vladimir Vovk
    • 1
  • Valentina Fedorova
    • 2
  • Ilia Nouretdinov
    • 1
  • Alexander Gammerman
    • 1
  1. 1.Computer Learning Research CentreRoyal Holloway, University of LondonEghamUK
  2. 2.YandexMoscowRussia

Personalised recommendations