Abstract
We study optimal conformity measures for various criteria of efficiency of set-valued classification in an idealised setting. This leads to an important class of criteria of efficiency that we call probabilistic and argue for; it turns out that the most standard criteria of efficiency used in literature on conformal prediction are not probabilistic unless the problem of classification is binary. We consider both unconditional and label-conditional conformal prediction.
References
Balasubramanian, V.N., Ho, S.S., Vovk, V. (eds.): Conformal prediction for reliable machine learning: theory, adaptations, and applications. Elsevier, Amsterdam (2014)
Dawid, A.P.: Probability Forecasting. In: Kotz, S., Balakrishnan, N., Read, C. B., Vidakovic, B., Johnson, N.L. (eds.) Encyclopedia of Statistical Sciences. 2nd edn., vol. 10, pp 6445–6452. Wiley, Hoboken (2006)
Fedorova, V., Gammerman, A., Nouretdinov, I., Vovk, V.: Hypergraphical conformal predictors. Int. J. Artif. Intell. Tools 24(6), 1560,003 (2015). COPA 2013 Special Issue
Gneiting, T., Raftery, A.E.: Strictly proper scoring rules, prediction, and estimation. J. Am. Stat. Assoc. 102, 359–378 (2007)
Johansson, U., König, R., Löfström, T., Boström, H.: Evolved Decision Trees as Conformal Predictors. In: de la Fraga, L.G. (ed.) Proceedings of the 2013 IEEE Conference on Evolutionary Computation, vol. 1, pp 1794–1801. Cancun, Mexico (2013)
Karp, R.M.: Reducibility among Combinatorial Problems. In: Miller, R.E., Thatcher, J.W. (eds.) Complexity of Computer Computations, pp 85–103. Plenum Press, New York (1972)
Le Cun, Y., Boser, B.E., Denker, J.S., Henderson, D., Howard, R.E., Hubbard, W.E., Jackel, L.D.: Handwritten Digit Recognition with a Back-Propagation Network. In: Touretzky, D.S. (ed.) Advances in Neural Information Processing Systems 2, pp 396–404. Morgan Kaufmann, San Francisco (1990)
Lehmann, E.L.: Testing Statistical Hypotheses, 2nd edn. Springer, New York (1986)
Lei, J.: Classification with confidence. Biometrika 101, 755–769 (2014)
Lei, J., Robins, J., Wasserman, L.: Distribution free prediction sets. J. Am. Stat. Assoc. 108, 278–287 (2013)
Lei, J., Wasserman, L.: Distribution free prediction bands for nonparametric regression. J. R. Stat. Soc. B 76, 71–96 (2014)
Martello, S., Toth, P.: Knapsack Problems: Algorithms and Computer Implementations. Wiley, Chichester (1990)
Melluish, T., Saunders, C., Nouretdinov, I., Vovk, V.: Comparing the Bayes and Typicalness Frameworks. In: De Raedt, L., Flach, P.A. (eds.) Proceedings of the Twelfth European Conference on Machine Learning, Lecture Notes in Computer Science, vol. 2167, pp 360–371. Springer, Heidelberg (2001)
Papadopoulos, H., Gammerman, A., Vovk, V. (eds.): Special Issue of the Annals of Mathematics and Artificial Intelligence on Conformal Prediction and its Applications, vol. 74(1–2). Springer (2015)
Sadinle, M., Lei, J., Wasserman, L.: Least ambiguous set-valued classifiers with bounded error levels. Tech. Rep. arXiv:1609.00451v1 [stat.ME] arXiv.org e-Print archive (2016)
Saunders, C., Gammerman, A., Vovk, V.: Transduction with Confidence and Credibility. In: Dean, T. (ed.) Proceedings of the Sixteenth International Joint Conference on Artificial Intelligence, vol. 2, pp 722–726. Morgan Kaufmann, San Francisco (1999)
Smith, J., Nouretdinov, I., Craddock, R., Offer, C., Gammerman, A.: Anomaly Detection of Trajectories with Kernel Density Estimation by Conformal Prediction. In: Iliadis, L., Maglogiannis, I., Papadopoulos, H., Sioutas, S., Makris, C. (eds.) AIAI Workshops, COPA 2014, IFIP Advances in Information and Communication Technology, vol. 437, pp 271–280 (2014)
Vovk, V., Fedorova, V., Nouretdinov, I., Gammerman, A.: Criteria of Efficiency for Conformal Prediction. In: Gammerman, A., Luo, Z., Vega, J., Vovk, V. (eds.) Proceedings of the Fifth International Symposium on Conformal and Probabilistic Prediction with Applications (COPA 2016), Lecture Notes in Artificial Intelligence, vol. 9653, pp 23–39. Springer, Switzerland (2016)
Vovk, V., Gammerman, A., Shafer, G.: Algorithmic Learning in a Random World. Springer, New York (2005)
Vovk, V., Petej, I., Fedorova, V.: From Conformal to Probabilistic Prediction. In: Iliadis, L. , Maglogiannis, I., Papadopoulos, H., Sioutas, S., Makrism, C. (eds.) AIAI Workshops, COPA 2014, IFIP Advances in Information and Communication Technology, vol. 437, pp 221–230 (2014)
Acknowledgments
We are grateful to the reviewers of the conference and journal versions of this paper for their helpful comments.
Author information
Authors and Affiliations
Corresponding author
Additional information
A preliminary version of this paper was published as Working Paper 11 of the On-line Compression Modelling project (New Series), http://alrw.net, in April 2014. Its conference version [18] was published in the Proceedings of the Fifth Symposium on Conformal and Probabilistic Prediction and Their Applications (COPA 2016, Madrid, April 2016) under the title “Criteria of efficiency for conformal prediction”. This journal version also incorporates (in Section ??) some material of our paper [20] in COPA 2014. This work was partially supported by EPSRC (grant EP/K033344/1), the Air Force Office of Scientific Research (grant “Semantic Completions”), and the EU Horizon 2020 Research and Innovation programme (grant 671555).
Rights and permissions
Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.
About this article
Cite this article
Vovk, V., Nouretdinov, I., Fedorova, V. et al. Criteria of efficiency for set-valued classification. Ann Math Artif Intell 81, 21–46 (2017). https://doi.org/10.1007/s10472-017-9540-3
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10472-017-9540-3
Keywords
- Conformal prediction
- Label-conditional conformal prediction
- Predictive efficiency
- Informational efficiency
Mathematics Subject Classification (2010)
- 68T05
- 68Q32
- 62G15