Selecting credit rating models: a cross-validation-based comparison of discriminatory power

Abstract

In commercial banking, various statistical models for corporate credit rating have been theoretically promoted and applied to bank-specific credit portfolios. In this paper, we empirically compare and test the performance of a wide range of parametric and nonparametric credit rating model approaches in a statistically coherent way, based on a ‘real-world’ data set. We repetitively (k times) split a large sample of industrial firms’ default data into disjoint training and validation subsamples. For all model types, we estimate k out-of-sample discriminatory power measures, allowing us to compare the models coherently. We observe that more complex and nonparametric approaches, such as random forest, neural networks, and generalized additive models, perform best in-sample. However, comparing k out-of-sample cross-validation results, these models overfit and lose some of their predictive power. Rather than improving discriminatory power, we perceive their major contribution to be their usefulness as diagnostic tools for the selection of rating factors and the development of simpler, parametric models.