Multiclassifier Systems: Back to the Future

  • Joydeep Ghosh
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 2364)


While a variety of multiple classifier systems have been studied since at least the late 1950’s, this area came alive in the 90’s with significant theoretical advances as well as numerous successful practical applications. This article argues that our current understanding of ensemble-type multiclassifier systems is now quite mature and exhorts the reader to consider a broader set of models and situations for further progress. Some of these scenarios have already been considered in classical pattern recognition literature, but revisiting them often leads to new insights and progress. As an example, we consider how to integrate multiple clusterings, a problem central to several emerging distributed data mining applications. We also revisit output space decomposition to show how this can lead to extraction of valuable domain knowledge in addition to improved classification accuracy.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Selfridge, O.G.: Pandemonium: a paradigm for learning. Proc. of Symp. held at the National Physical Lab. (1958) 513–526Google Scholar
  2. 2.
    Nilsson, N.J.: Learning Machines: Foundations of Trainable Pattern-Classifying Systems. McGraw Hill, NY (1965)MATHGoogle Scholar
  3. 3.
    Kanal, L.: Patterns in pattern recognition. IEEE Trans. Information Theory IT-20 (1974) 697–722CrossRefMathSciNetGoogle Scholar
  4. 4.
    Minsky, M.: Logical versus analogical or symbolic versus connectionist or neat versus scruffy. AI Magazine 12 (1991) 34–51Google Scholar
  5. 5.
    Granger, C.W.J.: Combining forecasts-twenty years later. Journal of Forecasting 8 (1989) 167–173CrossRefGoogle Scholar
  6. 6.
    French, S.: Group consensus probability distributions: A critical survey. In Bernardo et al., J.M., ed.: Bayesian Statistics 2. North Holland, New York (1985)Google Scholar
  7. 7.
    Benediktsson, J.A., Swain, P.H.: Consensus theoretic classification methods. IEEE Transactions on Systems, Man, and Cybernetics 22 (1992) 688–704MATHCrossRefGoogle Scholar
  8. 8.
    Luo, R.C., Kay, M.G.: Multisensor integration and fusion in intelligent systems. IEEE Transactions on Systems, Man, and Cybernetics 19 (1989) 901–931CrossRefGoogle Scholar
  9. 9.
    Narendra, K., Balakrishnan, J., Ciliz, K.: Adaptation and learning using multiple models, switching and tuning. IEEE Control Systems Magazine (1995) 37–51Google Scholar
  10. 10.
    Murray-Smith, R., Johansen, T.A.: Multiple Model Approaches to Modelling and Control. Taylor and Francis, UK (1997)Google Scholar
  11. 11.
    Ghosh, J., Deuser, L., Beck, S.: A neural network based hybrid system for detection, characterization and classification of short-duration oceanic signals. IEEE Jl. of Ocean Engineering 17 (1992) 351–363CrossRefGoogle Scholar
  12. 12.
    Sharkey, A.: On combining artificial neural networks. Connection Science 8 (1996) 299–314CrossRefGoogle Scholar
  13. 13.
    Kittler, J., Hatef, M., Duin, R., Matas, J.: On combining classifiers. IEEE Trans. Pattern Analysis and Machine Intelligence 20 (1998) 226–239CrossRefGoogle Scholar
  14. 14.
    Tumer, K.: Linear and order statistics combiners for reliable pattern classification. PhD thesis, Dept. of ECE, Univ. of Texas at Austin, Dec. (1996)Google Scholar
  15. 15.
    Perrone, M.P.: Improving Regression Estimation: Averaging Methods for Variance Reduction with Extensions to General Convex Measure Optimization. PhD thesis, Brown University (1993)Google Scholar
  16. 16.
    Krogh, A., Vedelsby, J.: Neural network ensembles, cross validation and active learning. In G. Tesauro, D.T., Leen, T., eds.: Advances in Neural Information Processing Systems-7. (1995) 231–238Google Scholar
  17. 17.
    Hansen, L.K., Salamon, P.: Neural network ensembles. IEEE Transactions on Pattern Analysis and Machine Intelligence 12 (1990) 993–1000CrossRefGoogle Scholar
  18. 18.
    Tumer, K., Ghosh, J.: Analysis of decision boundaries in linearly combined neural classifiers. Pattern Recognition 29 (1996) 341–348CrossRefGoogle Scholar
  19. 19.
    Tumer, K., Ghosh, J.: Ensemble correlation and error reduction in ensemble classifiers. Connection Science 8 (1996) 385–404CrossRefGoogle Scholar
  20. 20.
    Dietterich, T.G.: Ensemble methods in machine learning. In Kittler, J., Roli, F. eds.: Multiple Classifier Systems. LNCS Vol. 1857, Springer (2001) 1–15CrossRefGoogle Scholar
  21. 21.
    Friedman, J., Hastie, T., Tibshirani, R.: Additive logistic regression: a statistical view of boosting. Technical Report Dept. of Statistics, Stanford University. (1998)Google Scholar
  22. 22.
    Ghosh, J., Beck, S., Chu, C.: Evidence combination techniques for robust classification of short-duration oceanic signals. In: SPIE Conf. on Adaptive and Learning Systems, SPIE Proc. Vol. 1706. (1992) 266–276Google Scholar
  23. 23.
    Xu, L., Krzyzak, A., Suen, C.Y.: Methods of combining multiple classifiers and their applications to handwriting recognition. IEEE Transactions on Systems, Man and Cybernetics 22 (1992) 418–435CrossRefGoogle Scholar
  24. 24.
    Ho, T.K., Hull, J.J., Srihari, S.N.: Decision combination in multiple classifier systems. IEEE Transactions on Pattern Analysis and Machine Intelligence 16 (1994) 66–76CrossRefGoogle Scholar
  25. 25.
    Kittler, J., Roli (Eds.), F.: Multiple Classifier Systems. LNCS Vol. 1857, Springer (2001)MATHGoogle Scholar
  26. 26.
    Jordan, M., Jacobs, R.: Hierarchical mixture of experts and the EM algorithm. Neural Computation 6 (1994) 181–214CrossRefGoogle Scholar
  27. 27.
    Holmstrom, L., Koistinen, P., Laaksonen, J., Oja, E.: Neural and statistical classifiers-taxonomy and two case studies. IEEE Transactions on Neural Networks 8 (1997) 5–17CrossRefGoogle Scholar
  28. 28.
    Ramamurti, V., Ghosh, J.: Structurally adaptive modular networks for non-stationary environments. IEEE Transactions on Neural Networks 10 (1999) 152–60CrossRefGoogle Scholar
  29. 29.
    Woods, K., Kegelmeyer, W.P., Bowyer, K.: Combination of multiple classifiers using local accuracy estimates. IEEE Transactions on Pattern Analysis and Machine Intelligence 19 (1997) 405–410CrossRefGoogle Scholar
  30. 30.
    Jain, A.K., Dubes, R.C.: Algorithms for Clustering Data. Prentice Hall, New Jersey (1988)MATHGoogle Scholar
  31. 31.
    Barthelemy, J.P., Laclerc, B., Monjardet, B.: On the use of ordered sets in problems of comparison and consensus of classifications. Jl. of Classification 3 (1986) 225–256CrossRefGoogle Scholar
  32. 32.
    Neumann, D.A., Norton, V.T.: Clustering and isolation in the consensus problem for partitions. Journal of Classification 3 (1986) 281–298MATHCrossRefMathSciNetGoogle Scholar
  33. 33.
    Kim, J., Warnow, T.: Tutorial on phylogenetic tree estimation. In: Intelligent Systems for Molecular Biology, Heidelberg. (1999)Google Scholar
  34. 34.
    Johnson, E., Kargupta, H.: Collective, hierarchical clustering from distributed, heterogeneous data. In Zaki, M., Ho, C., eds.: Large-Scale Parallel KDD Systems. Volume 1759 of LNCS., Springer-Verlag (1999) 221–244CrossRefGoogle Scholar
  35. 35.
    Fred, A.L.N., Jain, A.K.: Data clustering using evidence accumulation. In: Proc. ICPR. (2002) to appearGoogle Scholar
  36. 36.
    Bollacker, K.D., Ghosh, J.: Effective supra-classifiers for knowledge base construction. Pattern Recognition Letters 20(11–13) (1999) 1347–52CrossRefGoogle Scholar
  37. 37.
    Kargupta, H., Chan, P., eds.: Advances in Distributed and Parallel Knowledge Discovery. AAAI/MIT Press, Cambridge, MA (2000)Google Scholar
  38. 38.
    Strehl, A., Ghosh, J.: Cluster ensembles — a knowledge reuse framework for combining partitionings. In: Proceedings of AAAI 2002, Edmonton, Canada, AAAI (2002) in press.Google Scholar
  39. 39.
    Strehl, A., Ghosh, J., Mooney, R.: Impact of similarity measures on web-page clustering. In: Proc. AAAI Workshop on AI for Web Search (AAAI 2000), Austin, AAAI (2000) 58–64Google Scholar
  40. 40.
    Kumar, S.: Modular learning through output space decomposition. PhD thesis, Dept. of ECE, Univ. of Texas at Austin, Dec. (2000)Google Scholar
  41. 41.
    Anand, R., Methrotra, K., Mohan, C.K., Ranka, S.: Efficient classification for multiclass problems using modular neural networks. IEEE Transactions on Neural Networks 6 (1995) 117–125CrossRefGoogle Scholar
  42. 42.
    Berenstein, C., Kanal, L.N., Lavine, D.: Consensus rules. In Kanal, L.N., Lemmer, J.F., eds.: Uncertainty in Artificial Intelligence. North Holland, New York (1986)Google Scholar
  43. 43.
    Hastie, T., Tibshirani, R.: Classification by pairwise coupling. In Jordan, M.I., Kearns, M.J., Solla, S.A., eds.: Advances in Neural Information Processing Systems. Volume 10., The MIT Press (1998)Google Scholar
  44. 44.
    Kumar, S., Crawford, M.M., Ghosh, J.: A versatile framework for labelling imagery with a large number of classes. In: Proc. IJCNN. (1999)Google Scholar
  45. 45.
    Dietterich, T.G., Bakiri, G.: Solving multiclass learning problems via errorcorrecting output codes. Jl. of Artificial Intelligence Research 2 (1995) 263–286MATHGoogle Scholar
  46. 46.
    Ricci, F., Aha, D.: Extending local learners with errorcorrecting output codes. Technical report, Technical Report 9701-08, IRST (1997)Google Scholar
  47. 47.
    Chakrabarti, S., Dom, B., Agrawal, R., Raghavan, P.: Scalable feature selection, classication and signature generation for organizing large text databases into hierarchical topic taxonomies. VLDB Journal 7 (1998) 163–178CrossRefGoogle Scholar
  48. 48.
    Kumar, S., Ghosh, J., Crawford, M.M.: Hierarchical fusion of multiple classifiers for hyperspectral data analysis”, Pattern Analysis and Applications, spl. Issue on Fusion of Multiple Classifiers 5 (2002) In PressGoogle Scholar
  49. 49.
    Thrun, S., Pratt, L.: Learning To Learn. Kluwer Academic, Norwell, MA (1997)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2002

Authors and Affiliations

  • Joydeep Ghosh
    • 1
  1. 1.Department of Electrical and Computer EngineeringUniversity of TexasAustin

Personalised recommendations