Accurate prediction of exceedance probabilities is important in many applications. For example, in process planning and control, engineers should anticipate the risk that a product fails to meet its specification limits. Statistical comparison between candidate probability prediction methods is commonly performed using scoring rules, like the continuous ranked probability score (CRPS) and the logarithm score (LogS). In this work, a new scoring rule, the exceedance probability score, is proposed. The experiments in simulated and real industrial data show that the new scoring rule is useful in comparing and testing differences in the predictive accuracy of competitive probabilistic predictions in regression setting. The proposed scoring rule have some similarities with CRPS and LogS, but is more directly connected to the accuracy in the prediction of exceedance probabilities.
AMS Subject Classification
62J99 62H15 62P30
CRPS Density forecast logarithm score Probability prediction Scoring rules
This is a preview of subscription content, log in to check access.
Cannon, A. J. 2011. Quantile regression neural networks: Implementation in r and application to precipitation downscaling. Comput. Geosci., 37(9), 1277–1284.CrossRefGoogle Scholar
Diebold, F., and R. Mariano. 1995. Comparing predictive accuracy. J. Business Econ. Stat., 13, 253–263.Google Scholar
Diks, C., V. Panchenko, and D. van Dijk. 2011. Likelihood-based scoring rules for comparing density forecasts in tails. J. Econometrics, 163(2), 215–230.MathSciNetCrossRefGoogle Scholar
Gneiting, T., and A. Raftery. 2007. Strictly proper scoring rules, prediction, and estimation. J. Am. Stat. Assoc., 102(477), 359–378.MathSciNetCrossRefGoogle Scholar
Gneiting, T., and A. E. Raftery. 2005. Weather forecasting with ensemble methods. Science, 310(5746), 248–249.CrossRefGoogle Scholar
Gneiting, T., and R. Ranjan. 2011. Comparing density forecasts using threshold- and quantile-weighted scoring rules. J. Business Econ. Stat., 29(3), 411–422.MathSciNetCrossRefGoogle Scholar
Gneiting, T., L. Stanberry, E. Grimit, L. Held, and N. Johnson. 2008. Assessing probabilistic forecasts of multivariate quantities, with an application to ensemble predictions of surface winds. TEST Off. J. Span. Soc. Stat. Operations Res., 17(2), 211–235.MathSciNetzbMATHGoogle Scholar
Hall, P., R. Wolff, and Q. Yao. 1999. Methods for estimating a conditional distribution function. J. Am. Stat. Assoc., 94, 154–163.MathSciNetCrossRefGoogle Scholar