Advertisement

Bias-Variance Analysis of Local Classification Methods

  • Julia Schiffner
  • Bernd Bischl
  • Claus Weihs
Conference paper
Part of the Studies in Classification, Data Analysis, and Knowledge Organization book series (STUDIES CLASS)

Abstract

In recent years an increasing amount of so called local classification methods has been developed. Local approaches to classification are not new. Well-known examples are the k nearest neighbors method and classification trees (e.g. CART). However, the term ‘local’ is usually used without further explanation of its particular meaning, we neither know which properties local methods have nor for which types of classification problems they may be beneficial. In order to address these problems we conduct a benchmark study. Based on 26 artificial and real-world data sets selected local and global classification methods are analyzed in terms of the bias-variance decomposition of the misclassification rate. The results support our intuition that local methods exhibit lower bias compared to global counterparts. This reduction comes at the price of an only slightly increased variance such that the error rate in total may be improved.

References

  1. Allwein EL, Shapire RE, Singer Y (2000) Reducing multiclass to binary: A unifying approach for margin classifiers. J Mach Learn Res 1:113–141Google Scholar
  2. Atkeson CG, Moore AW, Schaal S (1997) Locally weighted learning. Artif Intell Rev 11(1-5): 11–73CrossRefGoogle Scholar
  3. Bischl B (2010) mlr: Machine learning in R. URL http://mlr.r-forge.r-project.org
  4. Bishop CM (2006) Pattern recognition and machine learning. Springer, New YorkMATHGoogle Scholar
  5. Breiman L (1996) Bias, variance, and arcing classifiers. Tech. Rep. 460, Statistics Department, University of California at Berkeley, Berkeley, CA, URL www.stat.berkeley.edu
  6. Czogiel I, Luebke K, Zentgraf M, Weihs C (2007) Localized linear discriminant analysis. In: Decker R, Lenz HJ (eds) Advances in data analysis, Springer, Berlin Heidelberg, Studies in classification, data analysis, and knowledge organization, vol 33, pp 133–140Google Scholar
  7. Eugster MJA, Hothorn T, Leisch F (2008) Exploratory and inferential analysis of benchmark experiments. Tech. Rep. 30, Institut für Statistik, Ludwig-Maximilians-Universität München, Germany, URL http://epub.ub.uni-muenchen.de/4134/
  8. Frank A, Asuncion A (2010) UCI machine learning repository. University of California, Irvine, School of Information and Computer Sciences, URL http://archive.ics.uci.edu/ml
  9. Hand DJ, Vinciotti V (2003) Local versus global models for classification problems: Fitting models where it matters. American Statistician 57(2):124–131MathSciNetCrossRefGoogle Scholar
  10. Hastie T, Tibshirani R (1996) Discriminant analysis by Gaussian mixtures. J Royal Stat Soc B 58(1):155–176MathSciNetMATHGoogle Scholar
  11. Hastie T, Tibshirani R, Friedman J (2009) The elements of statistical learning: data mining, inference, and prediction, 2nd edn. Springer, New YorkMATHGoogle Scholar
  12. James GM (2003) Variance and bias for general loss functions. Mach Learn 51(2):115–135MATHCrossRefGoogle Scholar
  13. Leisch F, Dimitriadou E (2010) mlbench: Machine learning benchmark problems. R package version 2.0-0Google Scholar
  14. R Development Core Team (2009) R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria, URL http://www.R-project.org
  15. Szepannek G, Schiffner J, Wilson J, Weihs C (2008) Local modelling in classification. In: Perner P (ed) Advances in data mining. Medical applications, e-commerce, marketing, and theoretical aspects, Springer, Berlin Heidelberg, LNCS, vol 5077, pp 153–164Google Scholar
  16. Venables WN, Ripley BD (2002) Modern applied statistics with S, 4th edn. Springer, New York, URL http://www.stats.ox.ac.uk/pub/MASS4

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  1. 1.Department of StatisticsTU Dortmund UniversityDortmundGermany

Personalised recommendations