Data Mining and Knowledge Discovery

, Volume 1, Issue 1, pp 55–77

On Bias, Variance, 0/1—Loss, and the Curse-of-Dimensionality

  • Jerome H. Friedman
Article

DOI: 10.1023/A:1009778005914

Cite this article as:
Friedman, J.H. Data Mining and Knowledge Discovery (1997) 1: 55. doi:10.1023/A:1009778005914

Abstract

The classification problem is considered in which an outputvariable y assumes discrete values with respectiveprobabilities that depend upon the simultaneous values of a set of input variablesx = {x_1,....,x_n}. At issue is how error in the estimates of theseprobabilities affects classification error when the estimates are used ina classification rule. These effects are seen to be somewhat counterintuitive in both their strength and nature. In particular the bias andvariance components of the estimation error combine to influenceclassification in a very different way than with squared error on theprobabilities themselves. Certain types of (very high) bias can becanceled by low variance to produce accurate classification. This candramatically mitigate the effect of the bias associated with some simpleestimators like “naive” Bayes, and the bias induced by thecurse-of-dimensionality on nearest-neighbor procedures. This helps explainwhy such simple methods are often competitive with and sometimes superiorto more sophisticated ones for classification, and why“bagging/aggregating” classifiers can often improveaccuracy. These results also suggest simple modifications to theseprocedures that can (sometimes dramatically) further improve theirclassification performance.

classification bias variance curse-of-dimensionality bagging naive Bayes nearest-neighbors 

Copyright information

© Kluwer Academic Publishers 1997

Authors and Affiliations

  • Jerome H. Friedman
    • 1
  1. 1.Department of Statistics and Stanford Linear Accelerator CenterStanford UniversityUSA

Personalised recommendations