Abstract
In this paper I compare parametric and nonparametric regression models with the help of a simulated data set. Doing so, I have two main objectives. The first one is to differentiate five concepts of simplicity and assess their respective importance. The second one is to show that the scope of the existing philosophical literature on simplicity and model selection is too narrow because it does not take the nonparametric approach into account (Sober in Philosophy of Science 69(S3), S112–S123, 2002; Forster and Sober in The British Journal for the Philosophy of Science 45(1), 1–35, 1994; Forster, 2001, in Philosophy of Science 74(5), 588–600, 2007; Hitchcock and Sober in The British Journal for the Philosophy of Science 55(1), 1–34, 2004; Mikkelson in Philosophy of Science 73(4), 440–447, 2006; Baker 2013). More precisely, I point out that a measure of simplicity in terms of the number of adjustable parameters is inadequate to characterise nonparametric models and to compare them with parametric models. This allows me to weed out false claims about what makes a model simpler than another. Furthermore, I show that the importance of simplicity in model selection cannot be captured by the notion of parametric simplicity. ‘Simplicity’ is an umbrella term. While parametric simplicity can be ignored, there are other notions of simplicity that need to be taken into consideration when we choose a model. Such notions are not discussed in the previously mentioned literature. The latter therefore portrays an incomplete picture of why simplicity matters when we choose a model. Overall I support a pluralist view according to which we cannot give a general and interesting (epistemic or pragmatic) justification for the importance of simplicity in science.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
Under the assumptions made in Section 2.1, least squares estimates and maximum likelihood estimates are the same.
I am not suggesting here that the previously mentioned philosophers are not aware that the picture is incomplete and that more work needs to be done. My intention is to bring the debates forward.
Principal component analysis (PCA) is a technique that can “summarise” the variance of a data set into a lower dimension.
I understand that Sober is referring to scientific theories in biology. The latter are more complex than a set of conditions that must be met for a kernel regression to be adequately applied. Nonetheless, the point is that the dimensionality of the data and the nature of the model that we use (a posteriori and a priori considerations) will determine the importance of dimensional simplicity.
References
Baker, A. (2013). Simplicity In Zalta, E.N. (Ed.), , The Stanford encyclopedia of philosophy (Fall 2013 ed.)
Forster, M., & Sober, E. (1994). How to tell when simpler, more unified, or less ad hoc theories will provide more accurate predictions. The British Journal for the Philosophy of Science, 45(1), 1– 35.
Forster, M.R. (2001). The new science of simplicity. In Simplicity, inference and modelling: keeping it sophisticatedly simple (pp. 83–119). Cambridge University Press.
Forster, M.R. (2007). A philosopher’s guide to empirical success. Philosophy of Science, 74(5), 588–600.
Friedman, J., Hastie, T., & Tibshirani, R. (2001). The elements of statistical learning volume 1. Berlin: Springer series in statistics Springer.
Friend, M., Goethe, N.B., & Harizanov, V.S. (2007). Induction, algorithmic learning theory, and philosophy volume 9. Springer Science & Business Media.
Hitchcock, C., & Sober, E. (2004). Prediction versus accommodation and the risk of overfitting. The British Journal for the Philosophy of Science, 55(1), 1–34.
Hurvich, C.M., Simonoff, J.S., & Tsai, C. -L. (1998). Smoothing parameter selection in nonparametric regression using an improved akaike information criterion. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 60(2), 271–293.
James, G., Witten, D., Hastie, T., & Tibshirani, R. (2013). An introduction to statistical learning. Springer.
Kelly, K.T. (2007a). How simplicity helps you find the truth without pointing at it. In Induction, algorithmic learning theory, and philosophy (pp. 111–143). Springer.
Kelly, K.T. (2007b). Ockhams razor, empirical complexity, and Truth-Finding efficiency. Theoretical Computer Science, 383(2), 270–289.
Lurz, R.W. (2009). The philosophy of animal minds. Cambridge University Press.
Mikkelson, G.M. (2006). Realism versus instrumentalism in a new statistical framework. Philosophy of Science, 73(4), 440–447.
Sober, E. (1994). From a biological point of view: essays in evolutionary philosophy. Cambridge University Press.
Sober, E. (2002). Instrumentalism, parsimony, and the akaike framework. Philosophy of Science, 69(S3), S112–S123.
Sober, E. (2009). Parsimony and models of animal minds. In The philosophy of animal minds (pp. 237–257). Cambridge University Press.
Zellner, A., Keuzenkamp, H.A., & McAleer, M. (2001). Simplicity, inference and modelling: Keeping it sophisticatedly simple. Cambridge University Press.
Acknowledgments
I would like to thank the anonymous referees for their detailed and useful feedback. I presented an earlier version of this paper at Sun Yat-Sen University, Fudan University, Beijing Normal University, and National Yang-Ming University. I am very grateful to all the participants. I would also like to thank Dr Liu Mo, Professor Xiang Huang, and my students at Fudan University.
Author information
Authors and Affiliations
Corresponding author
Appendix
Appendix
Rights and permissions
About this article
Cite this article
Rochefort-Maranda, G. Simplicity and model selection. Euro Jnl Phil Sci 6, 261–279 (2016). https://doi.org/10.1007/s13194-016-0137-1
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s13194-016-0137-1