Machine Learning

, Volume 82, Issue 3, pp 475–481

An experimental test of Occam’s razor in classification

Technical Note

DOI: 10.1007/s10994-010-5227-2

Cite this article as:
Zahálka, J. & Železný, F. Mach Learn (2011) 82: 475. doi:10.1007/s10994-010-5227-2

Abstract

A widely persisting interpretation of Occam’s razor is that given two classifiers with the same training error, the simpler classifier is more likely to generalize better. Within a long-lasting debate in the machine learning community over Occam’s razor, Domingos (Data Min. Knowl. Discov. 3:409–425, 1999) rejects this interpretation and proposes that model complexity is only a confounding factor usually correlated with the number of models from which the learner selects. It is thus hypothesized that the risk of overfitting (poor generalization) follows only from the number of model tests rather than the complexity of the selected model. We test this hypothesis on 30 UCI data sets using polynomial classification models. The results confirm Domingos’ hypothesis on the 0.05 significance level and thus refutes the above interpretation of Occam’s razor. Our experiments however also illustrate that decoupling the two factors (model complexity and number of model tests) is problematic.

Keywords

Model complexity Generalization Empirical evaluation 
Download to read the full article text

Copyright information

© The Author(s) 2010

Authors and Affiliations

  1. 1.Faculty of Electrical EngineeringCzech Technical University in PraguePrahaCzech Republic

Personalised recommendations