Assessing the Error Probability of the Model Selection Test

Abstract

The asymptotic error probability of Linhart's model selection test isevaluated, and compared with the nominal significance level. We examine thecase where the expected discrepancies of the candidate models from the truemodel are asymptotically equal. The local alternatives method is employed inthe limiting operation of the asymptotic evaluation. Although the errorprobability under the null hypothesis is actually shown to be equal to orless than the level for most situations, intolerable violations of the errorcontrol are observed for nested models: It is often erroneously concludedthat the smaller model is significantly better than the larger model. Toprevent this violation, a modification of Linhart's test statistic isproposed. The effectiveness of the proposed test is confirmed throughtheoretical analysis and numerical simulations.