Bayesian Linear Combination of Neural Networks

  • Battista Biggio
  • Giorgio Fumera
  • Fabio Roli
Part of the Studies in Computational Intelligence book series (SCI, volume 247)


Classifier ensembles have been one of the main topics of interest in the neural networks, machine learning and pattern recognition communities during the past fifteen years [21,28,16,17,26,36,27,23,11]. They are currently one of the state of the art techniques available for the design of classification systems and an effective option to the traditional approach based on the design of a single, monolithic classifier in many applications. Broadly speaking, two main choices have to be made in the design of a classifier ensemble: how to generate individual classifiers and how to combine them. Two main approaches have emerged to deal with these design steps: coverage optimisation, focused on generating an ensemble of classifiers as much complementary as possible, which are then fused with simple combining rules, and decision optimisation, focused on finding the most effective combining rule to exploit at best a given classifier ensemble [21]. One of the most studied and widely used combining rules, especially in the former approach, is the linear combination of classifier outputs. Linear combiners are often used for neural network ensembles, given that neural networks provide continuous outputs. The simplicity of linear combiners and their continuous nature favoured the development of analytical models for the analysis of the performance of ensembles of predictors, both for the case of regression problems and for the relatively more complex case of classification problems.


Optimal Weight Ensemble Size Average Rule Added Error Ideal Boundary 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Asuncion, A., Newman, D.J.: UCI Machine Learning Repository (2007),
  2. 2.
    Battiti, R., Colla, A.M.: Democracy in neural nets: Voting schemes for classification. Neural Networks 7, 691–707 (1994)CrossRefGoogle Scholar
  3. 3.
    Benediktsson, J.A., Sveinsson, J.R., Ersoy, O.K., Swain, P.H.: Parallel Consensual Neural Networks. IEEE Transactions on Neural Networks 8(1), 54–64 (1997)CrossRefGoogle Scholar
  4. 4.
    Biggio, B., Fumera, G., Roli, F.: Bayesian Analysis of Linear Combiners. In: [11], pp. 292–301Google Scholar
  5. 5.
    Bishop, C.M.: Neural Networks for Pattern Recognition. Clarendon Press, Oxford (1995)Google Scholar
  6. 6.
    Brown, G., Wyatt, J.L., Tino, P.: Managing Diversity in Regression Ensembles. Journal of Machine Learning Research 6, 1621–1650 (2005)MathSciNetGoogle Scholar
  7. 7.
    Eckhardt, D.E., Lee, L.D.: A theoretical basis for the analysis of multiversion software subject to coincident errors. IEEE Transactions on Software Engineering 11(12), 1511–1517 (1985)CrossRefGoogle Scholar
  8. 8.
    Fumera, G., Roli, F.: A Theoretical and Experimental Analysis of Linear Combiners for Multiple Classifier Systems. IEEE Transactions on Pattern Analysis and Machine Intelligence 27, 942–956 (2005)CrossRefGoogle Scholar
  9. 9.
    Fumera, G., Roli, F., Serrau, A.: A Theoretical Analysis of Bagging as a Linear Combination of Classifiers. IEEE Transactions on Pattern Analysis and Machine Intelligence 30(7), 1293–1299 (2008)CrossRefGoogle Scholar
  10. 10.
    Geman, S., Bienenstock, E., Doursat, R.: Neural Networks and the bias/variance dilemma. Neural Computation 4, 1–58 (1992)CrossRefGoogle Scholar
  11. 11.
    Haindl, M., Kittler, J., Roli, F. (eds.): MCS 2007. LNCS, vol. 4472. Springer, Heidelberg (2007)zbMATHGoogle Scholar
  12. 12.
    Hansen, L.K., Salamon, P.: Neural Network Ensembles. IEEE Transactions on Pattern Analysis and Machine Intelligence 12, 993–1001 (1990)CrossRefGoogle Scholar
  13. 13.
    Hashem, S.: Optimal Linear Combination of Neural Networks. Neural Networks 10, 599–614 (1997)CrossRefGoogle Scholar
  14. 14.
    Hashem, S., Schmeiser, B.: Improving Model Accuracy Using Optimal Linear Combinations of Trained Neural Networks. IEEE Transactions on Neural Networks 6, 792–794 (1995)CrossRefGoogle Scholar
  15. 15.
    Kittler, J., Hatef, M., Duin, R.P.W., Matas, J.: On combining classifiers. IEEE Transactions on Pattern Analysis and Machine Intelligence 20, 226–239 (1998)CrossRefGoogle Scholar
  16. 16.
    Kittler, J., Roli, F. (eds.): MCS 2000. LNCS, vol. 1857. Springer, Heidelberg (2000)Google Scholar
  17. 17.
    Kittler, J., Roli, F. (eds.): MCS 2001. LNCS, vol. 2096. Springer, Heidelberg (2001)zbMATHGoogle Scholar
  18. 18.
    Kittler, J., Alkoot, F.M.: Sum versus Vote Fusion in Multiple Classifier Systems. IEEE Transactions on Pattern Analysis and Machine Intelligence 25, 110–115 (2003)CrossRefGoogle Scholar
  19. 19.
    Krogh, A., Vedelsby, J.: Neural network ensembles, cross validation, and active learning. In: Advances in Neural Information Processing Systems, pp. 231–238. MIT Press, Cambridge (1995)Google Scholar
  20. 20.
    Kuncheva, L.I.: A theoretical study on six classifier fusion strategies. IEEE Transactions on Pattern Analysis and Machine Intelligence 24, 281–286 (2002)CrossRefGoogle Scholar
  21. 21.
    Kuncheva, L.I.: Combining Pattern Classifiers: Methods and Algorithms. Wiley, Hoboken (2004)zbMATHCrossRefGoogle Scholar
  22. 22.
    Liu, Y.: Negative Correlation Learning and Evolutionary Neural Network Ensembles. PhD thesis, University College, The University of New South Wales, Australian Defence Force Academy, Canberra, Australia (1998)Google Scholar
  23. 23.
    Oza, N.C., Polikar, R., Kittler, J., Roli, F. (eds.): MCS 2005. LNCS, vol. 3541. Springer, Heidelberg (2005)zbMATHGoogle Scholar
  24. 24.
    Perrone, M.P., Cooper, L.N.: When Networks Disagree: Ensemble Methods for Hybrid Neural Networks. In: Mammone, R.J. (ed.) Neural Networks for Speech and Vision, pp. 126–142. Chapman-Hall, New York (1993)Google Scholar
  25. 25.
    Rogova, G.: Combining the results of several neural network classifiers. Neural Networks 7, 777–781 (1994)CrossRefGoogle Scholar
  26. 26.
    Roli, F., Kittler, J. (eds.): MCS 2002. LNCS, vol. 2364. Springer, Heidelberg (2002)zbMATHGoogle Scholar
  27. 27.
    Roli, F., Kittler, J., Windeatt, T. (eds.): MCS 2004. LNCS, vol. 3077. Springer, Heidelberg (2004)zbMATHGoogle Scholar
  28. 28.
    Sharkey, A.J.C. (ed.): Combining Artificial Neural Nets. Springer, London (1999)zbMATHGoogle Scholar
  29. 29.
    Tresp, V., Taniguchi, M.: Combining estimators using non-constant weighting functions. In: Tesauro, G., Touretzky, D.S., Leen, T.K. (eds.) Advances in Neural Information Processing Systems. MIT Press, Cambridge (1995)Google Scholar
  30. 30.
    Tumer, K.: Linear and order statistics combiners for reliable pattern classification. PhD thesis, The University of Texas, Austin (1996)Google Scholar
  31. 31.
    Tumer, K., Ghosh, J.: Analysis of Decision Boundaries in Linearly Combined Neural Classifiers. Pattern Recognition 29, 341–348 (1996)CrossRefGoogle Scholar
  32. 32.
    Tumer, K., Ghosh, J.: Linear and order statistics combiners for pattern classification. In: [28], pp. 127–155Google Scholar
  33. 33.
    Ueda, N.: Optimal Linear Combination of Neural Networks for Improving Classification Performance. IEEE Transactions on Pattern Analysis and Machine Intelligence 22, 207–215 (2000)CrossRefGoogle Scholar
  34. 34.
    Verikas, A., Lipnickas, A., Malmqvist, K., Bacauskiene, M., Gelzinis, A.: Soft Combination of Neural Classifiers: A Comparative Study. Pattern Recognition Letters 20, 429–444 (1999)CrossRefGoogle Scholar
  35. 35.
    Wolpert, D.H.: Stacked generalization. Neural Networks 5, 241–259 (1992)CrossRefGoogle Scholar
  36. 36.
    Windeatt, T., Roli, F. (eds.): MCS 2003. LNCS, vol. 2709. Springer, Heidelberg (2003)zbMATHGoogle Scholar
  37. 37.
    Zanda, M., Brown, G., Fumera, G., Roli, F.: Ensemble Learning in Linearly Combined Classifiers Via Negative Correlation. In: [11], pp. 440–449Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2009

Authors and Affiliations

  • Battista Biggio
    • 1
  • Giorgio Fumera
    • 1
  • Fabio Roli
    • 1
  1. 1.Department of Electrical and Electronic EngineeringUniversity of CagliariCagliariItaly

Personalised recommendations