Conformal predictions for information fusion

A comparative study of p-value combination methods
  • Vineeth N. BalasubramanianEmail author
  • Shayok Chakraborty
  • Sethuraman Panchanathan


The increased availability of a wide range of sensing technologies over the last few decades has resulted in an equivalent increased need for reliable information fusion methods in machine learning applications. While existing theories such as the Dempster-Shafer theory and the possibility theory have been used for several years now, they do not provide guarantees of error calibration in information fusion settings. The Conformal Predictions (CP) framework is a new game-theoretic approach to reliable machine learning, which provides a methodology to obtain error calibration under classification and regression settings. In this work, we present a methodology to extend the Conformal Predictions framework to both classification and regression-based information fusion settings. This methodology is based on applying the CP framework to each data source as an independent hypothesis test, and subsequently using p-value combination methods as a test statistic for the combined hypothesis after fusion. The proposed methodology was studied in classification and regression settings within two real-world application contexts: person recognition using multiple modalities (classification), and head pose estimation using multiple image features (regression). Our experimental results showed that quantile methods of combining p-values (such as the Standard Normal Function and the Non-conformity Aggregation methods) provided the most statistically valid calibration results, and can be considered to extend the CP framework for information fusion settings.


Conformal predictors Information fusion Multiple hypothesis testing Face processing applications 

Mathematics Subject Classification

68T10: Pattern recognition Speech recognition 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Ali, H., Antenreiter, M., Auer, P., Csurka, G., de Campos, T., Hussain, Z., Laaksonen, J., Ortner, R., Pasupa, K., Perronnin, F., Saunders, C., Shawe-Taylor, J., Viitaniemi, V.: Description, analysis and evaluation of confidence estimation procedures for sub-categorization. Tech. Rep. D6.2.1, Xerox Research Center Europe (2009)Google Scholar
  2. 2.
    Bailly, K., Milgram, M.: 2009 special issue: boosting feature selection for neural network based regression. Neural Netw. 22(5–6), 748–756 (2009)CrossRefGoogle Scholar
  3. 3.
    Balasubramanian, V., Chakraborty, S., Panchanathan, S.: Generalized query by transduction for online active learning. In: Proceedings of the International Conference on Computer Vision (ICCV 2009) Workshop on Online Learning for Computer Vision (2009)Google Scholar
  4. 4.
    Balasubramanian, V., Gouripeddi, R., Panchanathan, S., Vermillion, J., Bhaskaran, A., Siegel, R.: Support vector machine based conformal predictors for risk of complications following a coronary drug eluting stent procedure. In: Computers in Cardiology, pp. 5–8 (2009)Google Scholar
  5. 5.
    Balasubramanian, V., Panchanathan, S., Chakraborty, S.: Multiple cue integration in transductive confidence machines for head pose classification. In: Computer Vision and Pattern Recognition Workshops, CVPRW ’08. IEEE Computer Society Conference, pp. 1–8 (2008)Google Scholar
  6. 6.
    Balasubramanian, V., Panchanathan, S., Chakraborty, S.: Multiple cue integration in transductive confidence machines for head pose classification. In: Computer Vision and Pattern Recognition Workshops, CVPRW ’08. IEEE Computer Society Conference, pp. 1–8 (2008). doi: 10.1109/CVPRW.2008.4563070
  7. 7.
    Balasubramanian, V., Ye, J., Panchanathan, S.: Biased manifold embedding: a framework for person-independent head pose estimation. In: Computer Vision and Pattern Recognition, CVPR ’07. IEEE Conference, pp. 1–7 (2007). doi: 10.1109/CVPR.2007.383280
  8. 8.
    Beiraghi, S., Ahmadi, M., Ahmed, M.S., Shridhar, M.: Application of fuzzy integrals in fusion of classifiers for low error rate handwritten numerals recognition. Int. Conf. Pattern Recog. (ICPR) 2, 2487 (2000)Google Scholar
  9. 9.
    Ben-Yacoub, S., Abdeljaoued, Y., Mayoraz, E.: Fusion of face and speech data for person identity verification. IEEE Trans. Neural Netw. 10(5), 1065–1074 (1999). doi: 10.1109/72.788647 Google Scholar
  10. 10.
    Bolme, D., Beveridge, J., Howe, A.: Person identification using text and image data. In: Biometrics: Theory, Applications, and Systems, BTAS 2007. First IEEE International Conference, pp. 1–6 (2007)Google Scholar
  11. 11.
    Brown, L.M., Tian, Y.L.: Comparative study of coarse head pose estimation. In: IEEE Workshop on Motion and Video Computing, Orlando, Florida, pp. 125–130 (2002)Google Scholar
  12. 12.
    Brunelli, R., Falavigna, D.: Person identification using multiple cues. IEEE Trans. Pattern. Anal. Mach. Intell. 17, 955–966 (1995)CrossRefGoogle Scholar
  13. 13.
    Buchanan, B.G., Shortliffe, E.H.: Rule Based Expert Systems: The Mycin Experiments of the Stanford Heuristic Programming Project (The Addison-Wesley series in artificial intelligence). Addison-Wesley Longman Publishing Co., Inc. (1984)Google Scholar
  14. 14.
    Carrasco, M., Pizarro, L., Mery, D.: Bimodal biometric person identification system under perturbations. In: Advances in Image and Video Technology, pp. 114–127 (2007)Google Scholar
  15. 15.
    Dasarathy, B.V.: Decision Fusion. IEEE Computer Society Press, Los Alamitos CA (1994)Google Scholar
  16. 16.
    Dashevskiy, M., Luo, Z.: Network traffic demand prediction with confidence. In: IEEE Global Telecommunications Conference IEEE GLOBECOM 2008, pp. 1–5 (2008).
  17. 17.
    Dempster, A.: A generalization of bayesian inference. In: Classic Works of the Dempster-Shafer Theory of Belief Functions, pp. 73–104 (2008)Google Scholar
  18. 18.
    Dubois, D., Prade, H.: Possibility theory and its applications: a retrospective and prospective view. In: Fuzzy Systems, FUZZ ’03. The 12th IEEE International Conference, vol. 1, pp. 5–11 (2003)Google Scholar
  19. 19.
    Duc, B., Bign, E.S., Bign, J., Matre, G., Fischer, S.: Fusion of audio and video information for multi modal person authentication. Pattern Recognit. Lett. 18, 835–843 (1997)CrossRefGoogle Scholar
  20. 20.
    Edgington, E.S.: An AdditiveMethod for Combining Probability Values from Independent Experiments. J. Psychol. 80(2), 351–363 (1972). doi: 10.1080/00223980.1972.9924813 Google Scholar
  21. 21.
    Ekenel, H.K., Fischer, M., Jin, Q., Stiefelhagen, R.: Multi-modal person identification in a smart environment. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 1–8 (2007)Google Scholar
  22. 22.
    Ekenel, H.K., Jin, Q., Fischer, M., Stiefelhagen, R.: ISL person identification systems in the CLEAR 2007 evaluations. In: Stiefelhagen, R., Bowers, R., Fiscus, J. (Eds.) Multimodal Technologies for Perception of Humans, vol. 4625, pp. 256–265. Springer, Berlin/Heidelberg (2008). doi: 10.1007/978-3-540-68585-2_24
  23. 23.
    Eugene, T.H., Weinstein, E., Kabir, R., Park, A.: Multi-modal face and speaker identification on a handheld device. In: Proceedings Workshop Multimodal User Authentication, pp. 120–132 (2003)Google Scholar
  24. 24.
    Fisher, S.R.A.: Statistical methods for research workers, vol. 14, pp. 140–142. Edinburgh, Oliver and Boyd (1970)Google Scholar
  25. 25.
    Smarandache, F., Dezert, J. (eds.): Advances and Applications of DSmT for Information Fusion (Collected works), 2nd volume, Am. Res Press (2006)Google Scholar
  26. 26.
    Fox, N., Gross, R., Cohn, J., Reilly, R.: Robust biometric person identification using automatic classifier fusion of speech, mouth, and face experts. IEEE Trans. Multimed. 9(4), 701–714 (2007)CrossRefGoogle Scholar
  27. 27.
    Shafer, G.: Perspectives on the theory and practice of belief functions. Int. J. Approx. Reason. 4(5–6), 323–362 (1990)zbMATHMathSciNetCrossRefGoogle Scholar
  28. 28.
    Good, I.J.: On the weighted combination of significance tests. J. R. Stat. Soc. Ser. B Methodol. 17(2), 264–265 (1955)zbMATHMathSciNetGoogle Scholar
  29. 29.
    Hu, R., Damper, R.: Fusion of two classifiers for speaker identification: removing and not removing silence. In: Information Fusion, 2005 8th International Conference, vol. 1, p. 8 (2005)Google Scholar
  30. 30.
    Jain, A., Nandakumar, K., Ross, A.: Score normalization in multimodal biometric systems. Pattern Recog 38(12), 2285, 2270 (2005)CrossRefGoogle Scholar
  31. 31.
    Jost, L.: Combining significance levels from multiple experiments or analyses. (2009). Accessed 18 June 2009
  32. 32.
    Kostas, H.P., Proedrou, K., Vovk, V., Gammerman, A., Ex, S.T.: Inductive confidence machines for regression. In: Elomaa T., Mannila, H., Toivonen, H. (eds.) Proceedings of the 13th European Conference on Machine Learning, vol. 2430, pp. 345–356 (2002)Google Scholar
  33. 33.
    Lambrou, A., Papadopoulos, H., Gammerman, A.: Reliable confidence measures for medical diagnosis with evolutionary algorithms. IEEE transactions on information technology in biomedicine: a publication of the IEEE Engineering. Med Biol Soc 15(1), 93–99 (2011). (PMID: 21062682) doi: 10.1109/TITB.2010.2091144 CrossRefGoogle Scholar
  34. 34.
    Lancaster, H.O.: The combination of probabilities: an application of orthonormal functions. Aust. N. Z. J. Stat. 3(1), 20–33 (1961)zbMATHMathSciNetCrossRefGoogle Scholar
  35. 35.
    Li, M., Vitányi, P.: An introduction to Kolmogorov complexity and its applications. 2nd edn. Springer, Secaucus (1997)Google Scholar
  36. 36.
    Liptak, T.: On the combination of independent tests. Magyar Tud Akad Mat Kutato Int Kozl 3, 171–197 (1958)Google Scholar
  37. 37.
    Little, G., Krishna, S., Black, J., Panchanathan, S.: A methodology for evaluating robustness of face recognition algorithms with respect to variations in pose and illumination angle. In: Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 89–92. Philadelphia. (2005)Google Scholar
  38. 38.
    Loughin, T.M.: A systematic comparison of methods for combining p-values from independent tests. Comput. Stat. Data Anal. 47(3), 467–485 (2004)zbMATHMathSciNetCrossRefGoogle Scholar
  39. 39.
    Luque, J., Morros, R., Garde, A., Anguita, J., Farrus, M., Macho, D., Marqus, F., Martnez, C., Vilaplana, V., Hernando, J.: Audio, video and multimodal person identification in a smart room. In: Multimodal Technologies for Perception of Humans, pp. 258–269 (2007)Google Scholar
  40. 40.
    Marcel, S., McCool, C., Chakraborty, S., Balasubramanian, V., Panchanathan, S., Nolazco, J., Garcia, L., Aceves, R., et al.: Mobile biometry (mobio) face and speaker verification evaluation. In: Proceedings of the 20th International Conference on Pattern Recognition (ICPR2010) (2010)Google Scholar
  41. 41.
    Michaelsen, E., Jaeger, K.: Evidence fusion using the GESTALT-system. In: Information Fusion, 2008 11th International Conference on, pp. 1–7. IEEE (2008)Google Scholar
  42. 42.
    Mosteller, F., Bush, R.: Selected quantitative techniques. Handbook of Social Psychology (1954)Google Scholar
  43. 43.
    Mudholkar, G.S., George, E.O.: The logit method for combining probabilities. In: Symposium on Optimizing Methods in Statistics, pp. 345–366. Academic Press, New York (1979)Google Scholar
  44. 44.
    Murphy-Chutorian, E., Trivedi, M.: Head pose estimation in computer vision: a survey. IEEE Trans. Pattern Anal. Mach. Intell. 31(4), 607–626 (2009)CrossRefGoogle Scholar
  45. 45.
    Nolazco-Flores, J., Garcia-Perera, P.: Enhancing acoustic models for robust speaker verification. In: Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (2008)Google Scholar
  46. 46.
    Nouretdinov, I., Melluish, T., Vovk, V.: Ridge regression confidence machine. In: Proceedings of the 18th International Conference on Machine Learning, pp. 385—392 (2001)Google Scholar
  47. 47.
    Palanivel, S., Yegnanarayana, B.: Multimodal person authentication using speech, face and visual speech. Comput. Vis. Image Underst. 109(1), 44–55 (2008)CrossRefGoogle Scholar
  48. 48.
    Papadopoulos, H.: Inductive conformal prediction: theory and application to neural networks. In: Tools in Artificial Intelligence, pp. 315–329 (2008)Google Scholar
  49. 49.
    Papadopoulos, H., Vovk, V., Gammerman, A.: Regression conformal prediction with nearest neighbours. J. Artif. Int. Res 40(1), 815–840 (2011). zbMATHMathSciNetGoogle Scholar
  50. 50.
    Pesarin, F.: Multivariate permutation tests: with applications in biostatistics, vol. 240. Wiley, Chichester (2001)Google Scholar
  51. 51.
    Poh, N., Korczak, J.: Hybrid biometric person authentication using face and voice features. In: Proceedings AVBPA, pp. 348–353 (2001)Google Scholar
  52. 52.
    Proedrou, K.: Rigorous measures of confidence for pattern recognition and regression. PhD thesis, Royal Holloway College, University of London, Advisor-Alex Gammerman (2003)Google Scholar
  53. 53.
    Rogova, G.L., Nimier, V.: Reliability in information fusion: literature survey. In: Svensson, P.,Schubert, J. (eds.) Proceedings of the 7th International Conference on Information Fusion. International Society of Information Fusion, vol. II, pp. 1158–1165. Mountain View, CA (2004)Google Scholar
  54. 54.
    Sanderson, C.: Biometric person recognition: Face, speech and fusion. VDM Publishing (2008)Google Scholar
  55. 55.
    Shafer, G.: A mathematical theory of evidence. Princeton University Press, Princeton (1976)zbMATHGoogle Scholar
  56. 56.
    Shafer, G., Vovk, V.: A tutorial on conformal prediction. J. Mach. Learn. Res 9, 371–421 (2008)zbMATHMathSciNetGoogle Scholar
  57. 57.
    Shaffer, J.P.: Multiple hypothesis testing. Annual Review of Psychology 46(1), 561–584 (1995). doi: 10.1146/ CrossRefGoogle Scholar
  58. 58.
    Tippett, L.H.C.: The methods of statistics. 4th edn. Dover, New York (1963)Google Scholar
  59. 59.
    Vovk, V.: On-line confidence machines are well-calibrated. In: FOCS ’02: 43rd Symposium on Foundations of Computer Science. IEEE Computer Society, pp. 187–196, Washington, DC. (2002)Google Scholar
  60. 60.
    Vovk, V., Gammerman, A., Shafer, G.: Algorithmic Learning in a Random World. Springer, Secaucus (2005)zbMATHGoogle Scholar
  61. 61.
    Wilkinson, B.: A statistical consideration in psychological research. Psychol. Bull. 48(3), 156–8 (1951)MathSciNetCrossRefGoogle Scholar
  62. 62.
    Yang, F., Wang, H.-Z., Mi, H., de Lin, C., wen Cai, W.: Using random forest for reliable classification and cost-sensitive learning for medical diagnosis. BMC Bioinforma. 10(1), S22 (2009)CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media Dordrecht 2014

Authors and Affiliations

  • Vineeth N. Balasubramanian
    • 1
    Email author
  • Shayok Chakraborty
    • 1
  • Sethuraman Panchanathan
    • 1
  1. 1.Indian Institute of TechnologyHyderabadIndia

Personalised recommendations