Reliability Assessment of Ensemble Classifiers: Application in Mammography

  • Maciej A. Mazurowski
  • Jacek M. Zurada
  • Georgia D. Tourassi
Part of the Lecture Notes in Computer Science book series (LNCS, volume 5116)


In classifier ensembles predictions of different classifiers regarding a query are combined into one final decision. It was previously shown that using ensemble techniques can significantly improve classification performance. In this study we build upon this result and propose to use variability in the predictions of classifiers contributing to the final decision as an indicator of its reliability. The study hypothesis is tested with respect to previously proposed information-theoretic computer-aided decision (IT-CAD) system for detection of masses in mammograms. A database of 1820 regions of interest (ROIs) extracted from digital database of screening mammography (DDSM) is used. Experimental results show that the proposed reliability assessment successfully identifies decisions that can not be trusted. Further, a low correlation between reliability and the classifier output is noted. This opens a possibility of combining reliability and ensemble output into one improved decision.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Kittler, J., Hatef, M., Duin, R.P., Matas, J.: On combining classifiers. IEEE Transactions on Pattern Analysis and Machine Intelligence 20, 226–239 (1998)CrossRefGoogle Scholar
  2. 2.
    Kuncheva, L.I.: Combining Pattern Classifiers. Willey-Interscience (2004)Google Scholar
  3. 3.
    Ranawana, R., Palade, V.: Multi-classifier systems: Review and a roadmap for developers. International Journal of Hybrid Intelligent Systems 3, 35–61 (2006)zbMATHGoogle Scholar
  4. 4.
    Bloch, I.: Some aspects of dempster-shafer evidence theory for classification of multi-modality medical images taking partial volume effect into account. Pattern Recognition Letters 17, 905–919 (1996)CrossRefGoogle Scholar
  5. 5.
    Kuncheva, L.I.: Switching between selection and fusion in combining classifiers: An experiment. IEEE Transactions on Systems, Man, and Cybernetics – Part B: Cybernetics 32, 146–156 (2002)CrossRefGoogle Scholar
  6. 6.
    Zhou, Z.-H., Jiang, Y.: Medical diagnosis with c4.5 rule preceded by artificial neural network ensemble. IEEE Transaction on Information Technology in Biomedicine 7, 37–42 (2003)CrossRefGoogle Scholar
  7. 7.
    Greene, D., Tsymbal, A., Bolshakova, N., Cunningham, P.: Ensemble clustering in medical diagnostics. In: Proceedings of 17th IEEE Symposium on Computer-Based Medical Systems (CBMS 2004), pp. 575–581 (2004)Google Scholar
  8. 8.
    West, D., Mangiameli, P., Rampal, R., West, V.: Ensemble strategies for a medical diagnostic decision support system: A breast cancer diagnosis application. European Journal of Operational Research 162, 532–551 (2005)zbMATHCrossRefGoogle Scholar
  9. 9.
    Raza, M., Gondal, I., David Green, R.L.C.: Classifier fusion using Dempster-Shafer theory of evidence to predict breast cancer tumors. In: 2006 IEEE Region 10 Conference (TENCON 2006), pp. 1–4 (2006)Google Scholar
  10. 10.
    Mazurowski, M.A., Zurada, J.M., Tourassi, G.D.: Database decomposition of a knowledge base cad system in mammography; an ensemble approach to improve detection performance. In: Proc. SPIE Medical Imaging 2008 (in press) (2008)Google Scholar
  11. 11.
    Habas, P.A., Zurada, J.M., Elmaghraby, A.S., Tourassi, G.D.: Reliability analysis framework for computer-assisted medical decision systems. Med. Phys. 34, 763–772 (2007)CrossRefGoogle Scholar
  12. 12.
    Habas, P.A., Zurada, J.M., Elmaghraby, A.S., Tourassi, G.D.: Probabilistic framework for reliability analysis of information-theoretic cad systems in mammography. In: Proceedings of the 28th IEEE EMBS Annual International Conference, New York City, USA, August 30-September 3, 2006, pp. 6113–6116 (2006)Google Scholar
  13. 13.
    Tourassi, G.D., Vargas-Voracek, R., Catarious Jr., D.M., Floyd Jr., C.E.: Computer-assisted detection of mammographic masses: A template matching scheme based on mutual information. Medical Physics 30, 2123–2130 (2003)CrossRefGoogle Scholar
  14. 14.
    Tourassi, G.D., Haarawood, B., Singh, S., Lo, J.Y., Floyd, C.E.: Evaluation of information-theoretic similarity measures for content-based retrieval and detection of masses in mammograms. Medical Physics 34, 140–150 (2007)CrossRefGoogle Scholar
  15. 15.
    Jiang, Y.: Uncertainty in the output of artificial neural networks. IEEE Trans. Med. Imaging 22, 913–921 (2003)CrossRefGoogle Scholar
  16. 16.
    Heath, M., et al.: Current status of the digital database for screening mammography, ch. In: Digital Mammography. Kluwer Academic, Dordrecht (1998)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2008

Authors and Affiliations

  • Maciej A. Mazurowski
    • 1
  • Jacek M. Zurada
    • 1
  • Georgia D. Tourassi
    • 2
  1. 1.Computational Intelligence Laboratory Department of Electrical and Computer EngineeringUniversity of LouisvilleLouisville 
  2. 2.Duke Advanced Imaging Laboratories, Department of RadiologyDuke University Medical CenterDurham 

Personalised recommendations