Bias-Variance Analysis of ECOC and Bagging Using Neural Nets

Part of the Studies in Computational Intelligence book series (SCI, volume 373)


One of the methods used to evaluate the performance of ensemble classifiers is bias and variance analysis. In this chapter, we analyse bootstrap aggregating (Bagging) and Error Correcting Output Coding (ECOC) ensembles using a biasvariance framework; and make comparisons with single classifiers, while having Neural Networks (NNs) as base classifiers. As the performance of the ensembles depends on the individual base classifiers, it is important to understand the overall trends when the parameters of the base classifiers – nodes and epochs for NNs –, are changed. We show experimentally on 5 artificial and 4 UCI MLR datasets that there are some clear trends in the analysis that should be taken into consideration while designing NN classifier systems.


Prediction Error Loss Function Ensemble Method Machine Learn Research Error Correct Output Code 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Allwein, E., Schapire, R., Singer, Y.: Reducing multiclass to binary: A unifying approach for margin classifiers. J. Machine Learning Research 1, 113–141 (2002)MathSciNetGoogle Scholar
  2. 2.
    Asuncion, A., Newman, D.J.: UCI machine learning repository, School of Information and Computer Science. University of California, Irvine (2007)Google Scholar
  3. 3.
    Breiman, L.: Arcing classifiers. The Annals of Stat. 26, 801–849 (1998)MathSciNetCrossRefzbMATHGoogle Scholar
  4. 4.
    Dietterich, T.G., Bakiri, G.: Solving multi-class learning problems via error-correcting output codes. J. Artif. Intell. Research 2, 263–286 (1995)zbMATHGoogle Scholar
  5. 5.
    Domingos, P.: Why does bagging work? A Bayesian account and its implications. In: Heckerman, D., Mannila, H., Pregibon, D. (eds.) Proc. the 3rd Int. Conf. Knowledge Discovery and Data Mining, Newport Beach, CA, pp. 155–158. AAAI Press, Menlo Park (1997)Google Scholar
  6. 6.
    Domingos, P.: A unified bias-variance decomposition for zero-one and squared loss. In: Proc. the 17th Natl. Conf. Artif. Intell., Austin, TX, pp. 564–569. MIT Press, Cambridge (2000)Google Scholar
  7. 7.
    Escalera, S., Tax, D.M.J., Pujol, O., Radeva, P., Duin, R.P.W.: Subclass problem-dependent design for error-correcting output codes. IEEE Trans. Pattern Analysis and Machine Intell. 30, 1041–1054 (2008)CrossRefGoogle Scholar
  8. 8.
    Freund, Y., Schapire, R.E.: Experiments with a new boosting algorithm. In: Saitta, L. (ed.) Proc. the 13th Int. Conf. Machine Learning, Bari, Italy, pp. 148–156. Morgan Kaufmann, San Francisco (1996)Google Scholar
  9. 9.
    Friedman, J.H.: On bias, variance, 0/1 loss and the curse of dimensionality. Data Mining and Knowledge Discovery 1, 55–77 (1997)CrossRefGoogle Scholar
  10. 10.
    Geman, S., Bienenstock, E., Doursat, R.: Neural networks and the bias/variance dilemma. Neural Comp. 4, 1–58 (1992)CrossRefGoogle Scholar
  11. 11.
    Heskes, T.: Bias/variance decomposition for likelihood-based estimators. Neural Comp. 10, 1425–1433 (1998)CrossRefGoogle Scholar
  12. 12.
    James, G.M.: Majority vote classifiers: Theory and applications. PhD Thesis, Department of Statistics, University of Standford (1998)Google Scholar
  13. 13.
    James, G.: Variance and bias for general loss functions. Machine Learning 51, 115–135 (2003)CrossRefzbMATHGoogle Scholar
  14. 14.
    James, G.M., Hastie, T.: The error coding method and PICT’s. Comp. and Graph. Stat. 7, 377–387 (1998)MathSciNetCrossRefGoogle Scholar
  15. 15.
    Kohavi, R., Wolpert, D.H.: Bias plus variance decomposition for zero-one loss functions. In: Saitta, L. (ed.) Proc. the 13th Int. Conf. Machine Learning, Bari, Italy, pp. 275–283. Morgan Kaufmann, San Francisco (1996)Google Scholar
  16. 16.
    Kong, E.B., Dietterich, T.G.: Error-correcting output coding corrects bias and variance. In: Prieditis, A., Russell, S.J. (eds.) Proc. the 12th Int. Conf. Machine Learning, Tahoe City, CA, pp. 313–321. ACM Press, New York (1995)Google Scholar
  17. 17.
    Schapire, R.E., Freund, Y., Bartlett, P., Lee, W.S.: Boosting the margin: a new explanation for the effectiveness of voting methods. The Annals of Stat. 26, 1651–1686 (1998)MathSciNetCrossRefzbMATHGoogle Scholar
  18. 18.
    Smith, R.S., Windeatt, T.: The bias variance trade-off in bootstrapped error correcting output code ensembles. In: Benediktsson, J.A., Kittler, J., Roli, F. (eds.) MCS 2009. LNCS, vol. 5519, pp. 1–10. Springer, Heidelberg (2009)CrossRefGoogle Scholar
  19. 19.
    Tibshirani, R.: Bias, variance and prediction error for classification rules. Technical Report, University of Toronto, Toronto, Canada (1996)Google Scholar
  20. 20.
    Tumer, K., Ghosh, J.: Analysis of decision boundaries in linearly combined neural classifiers. Pattern Recogn. 29, 341–348 (1996)CrossRefGoogle Scholar
  21. 21.
    Tumer, K., Ghosh, J.: Error correlation and error reduction in ensemble classifiers. Connection Science 8, 385–403 (1996)CrossRefGoogle Scholar
  22. 22.
    Valentini, G., Dietterich, T.: Bias-variance analysis of support vector machines for the development of SVM-based ensemble methods. J. Machine Learning Research 5, 725–775 (2004)MathSciNetGoogle Scholar
  23. 23.
    Webb, G.I., Conilione, P.: Estimating bias and variance from data. School of Comp. Science and Software Engineering, Monash University (2005)Google Scholar
  24. 24.
    Windeatt, T., Ghaderi, R.: Coding and decoding strategies for multi-class learning problems. Inf. Fusion 4, 11–21 (2003)CrossRefGoogle Scholar
  25. 25.
    Wolpert, D.H.: On bias plus variance. Neural Comp. 9, 1211–1244 (1996)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2011

Authors and Affiliations

  1. 1.Centre for Vision, Speech and Signal ProcessingUniversity of SurreyUK
  2. 2.Sabanci UniversityTuzlaTurkey

Personalised recommendations