Abstract
The development of new tests and methods used in the evaluation of time series forecasts and forecasting models remains as important today as it has been for the last 50 years. Paraphrasing what Sir Clive W.J. Granger (arguably the father of modern day time series forecasting) said in the 1990s at a conference in Svinkloev, Denmark, “OK, the model looks like an interesting extension, but can it forecast better than existing models?” Indeed, the forecast evaluation literature continues to expand, with interesting new tests and methods being developed at a rapid pace. In this chapter, we discuss a selected group of predictive accuracy tests and model selection methods that have been developed in recent years, and that are now widely used in the forecasting literature. We begin by reviewing several tests for comparing the relative forecast accuracy of different models, in the case of point forecasts. We then broaden the scope of our discussion by introducing density-based predictive accuracy tests. We conclude by noting that predictive accuracy is typically assessed in terms of a given loss function, such as mean squared forecast error or mean absolute forecast error. Most tests, including those discussed here, are consequently loss function dependent, and the relative forecast superiority of predictive models is therefore also dependent on specification of a loss function. In light of this fact, we conclude this chapter by discussing loss function robust predictive density accuracy tests that have recently been developed using principles of stochastic dominance.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
We require that for j = 1, …, p i, \(E(\nabla _{\theta }F_i(u|Z^t, \theta _i^{\dagger }))_j \ge D_t(u)\), with \(\sup _t \sup _{u\in \Re }E(D_t(u)^{2r})<\infty \).
References
Andrews, D. W., & Soares, G. (2010). Inference for parameters defined by moment inequalities using generalized moment selection. Econometrica, 78(1), 119–157.
Andrews, D. W. K. (2002). Higher-order improvements of a computationally attractive “k”-step bootstrap for extremum estimators. Econometrica, 70(1), 119–162.
Andrews, D. W. K. (2004). The block–block bootstrap: Improved asymptotic refinements. Econometrica, 72(3), 673–700.
Bierens, H. J. (1990). A consistent conditional moment test of functional form. Econometrica, 58, 1443–1458.
Bierens, H. J., & Ploberger, W. (1997). Asymptotic theory of integrated conditional moment tests. Econometrica, 65, 1129–1151.
Chang, Y., Gomes, J. F., & Schorfheide, F. (2002). Learning-by-doing as a propagation mechanism. American Economic Review, 92(5), 1498–1520.
Chao, J., Corradi, V., & Swanson, N. R. (2001). Out-of-sample tests for granger causality. Macroeconomic Dynamics, 5(4), 598–620.
Clark, T. E., & McCracken, M. W. (2001). Tests of equal forecast accuracy and encompassing for nested models. Journal of Econometrics, 105(1), 85–110.
Clark, T. E., & McCracken, M. W. (2003). Evaluating long horizon forecasts. Working Paper, University of Missouri-Columbia.
Corradi, V., & Distaso, W. (2011). Multiple forecast model evaluation. In The handbook of economic forecasting (pp. 391–414). Oxford: Oxford University Press.
Corradi, V., & Swanson, N. R. (2002). A consistent test for out of sample nonlinear predictive ability. Journal of Econometrics, 110, 353–381.
Corradi, V., & Swanson, N. R. (2005). A test for comparing multiple misspecified conditional interval models. Econometric Theory, 21(5), 991–1016.
Corradi, V., & Swanson, N. R. (2006a). Predictive density and conditional confidence interval accuracy tests. Journal of Econometrics, 135(1), 187–228.
Corradi, V., & Swanson, N. R. (2006b). Predictive density evaluation. Handbook of Economic Forecasting, 1, 197–284.
Corradi, V., & Swanson, N. R. (2007). Nonparametric bootstrap procedures for predictive inference based on recursive estimation schemes. International Economic Review, 48(1), 67–109.
Corradi, V., Swanson, N. R., & Olivetti, C. (2001). Predictive ability with cointegrated variables. Journal of Econometrics, 104(2), 315–358.
De Jong, R. M. (1996). The bierens test under data dependence. Journal of Econometrics, 72(1), 1–32.
Diebold, F. X., & Mariano, R. S. (2002). Comparing predictive accuracy. Journal of Business & Economic Statistics, 20(1), 134–144.
Fernández-Villaverde, J., & Rubio-RamÍrez, J. F. (2004). Comparing dynamic equilibrium models to data: A Bayesian approach. Journal of Econometrics, 123(1), 153–187.
Gianni, A., & Giacomini, R. (2007). Comparing density forecasts via weighted likelihood ratio tests. Journal of Business & Economic Statistics, 25(2), 177–190.
Granger, C. W. J. (1999). Outline of forecast theory using generalized cost function. Spanish Economic Review, 1, 161–173.
Granger, C. W. J. (1993). On the limitations of comparing mean square forecast errors: A comment. Journal of Forecasting, 12(8), 651–652.
Hall, P., & Horowitz, J. L. (1996). Bootstrap critical values for tests based on generalized-method-of-moments estimators. Econometrica, 64(4), 891–916.
Hansen, B. E. (1996a). Inference when a nuisance parameter is not identified under the null hypothesis. Econometrica, 64, 413–430.
Hansen, B. E. (1996b). Stochastic equicontinuity for unbounded dependent heterogeneous arrays. Econometric Theory, 12, 347–359.
Hansen, R. P. (2005). A test for superior predictive ability. Journal of Business & Economic Statistics, 23(4), 365–380.
Holm, S. (1979). A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics, 6, 65–70.
Inoue, A., & Shintani, M. (2006). Bootstrapping GMM estimators for time series. Journal of Econometrics, 133(2), 531–555.
Jin, S., Corradi, V., & Swanson, N. R. (2017). Robust forecast comparison. Econometric Theory, 33(6), 1306–1351.
Kilian, L. (1999). Exchange rates and monetary fundamentals: What do we learn from long-horizon regressions? Journal of Applied Econometrics, 14(5), 491–510.
Kitamura, Y. (2002). Econometric comparisons of conditional models. Working Paper, University of Pennsylvania.
Lee, T. H., White, H., & Granger, C. W. J. (1993). Testing for neglected nonlinearity in time series models: A comparison of neural network methods and alternative tests. Journal of Econometrics, 56(3), 269–290.
Linton, O. B., Maasoumi, E., & Whang, Y. J. (2002). Consistent testing for stochastic dominance: A subsampling approach. Social Science Electronic Publishing, 72(3), 735–765.
Linton, O., Maassoumi, E., & Whang, Y. J. (2005). Consistent testing for stochastic dominance: A subsampling approach. Review of Economic Studies, 72, 735–765.
McCracken, M. W. (2000). Robust out-of-sample inference. Journal of Econometrics, 99, 195–223.
Meese, R. A., & Rogoff, K. (1983). Empirical exchange rate models of the seventies: Do they fit out-of-sample? Journal of International Economics, 14, 3–24.
Politis, D. N., Romano, J. P., & Wolf, M. (1999). Subsampling. Springer Series in Statistics. New York: Springer.
Romano, J. P., & Wolf, M. (2005). Stepwise multiple testing as formalized data snooping. Econometrica, 73(4), 1237–1282.
Rossi, B. (2005). Testing long-horizon predictive ability with high persistence, and the meese–rogoff puzzle. International Economic Review, 46(1), 61–92.
Schorfheide, F. (2010). Loss function-based evaluation of DSGE models. Journal of Applied Econometrics, 15(6), 645–670.
Stinchcombe, M. B., & White, H. (1998). Consistent specification testing with nuisance parameters present only under the alternative. Econometric Theory, 14(3), 295–325.
Vuong, Q. H. (1989). Likelihood ratio tests for model selection and non-nested hypotheses. Econometrica, 57(2), 307–333.
Weiss, A. (1996). Estimating time series models using the relevant cost function. Journal of Applied Econometrics, 11(5), 539–560.
West, K. D. (1996). Asymptotic inference about predictive ability. Econometrica, 64, 1067–1084.
White, H. (1982). Maximum likelihood estimation of misspecified models. Econometrica, 50(1), 1–25.
White, H. (2000). A reality check for data snooping. Econometrica, 68(5), 1097–1126.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this chapter
Cite this chapter
Cheng, M., Swanson, N.R., Yao, C. (2020). Forecast Evaluation. In: Fuleky, P. (eds) Macroeconomic Forecasting in the Era of Big Data. Advanced Studies in Theoretical and Applied Econometrics, vol 52. Springer, Cham. https://doi.org/10.1007/978-3-030-31150-6_16
Download citation
DOI: https://doi.org/10.1007/978-3-030-31150-6_16
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-31149-0
Online ISBN: 978-3-030-31150-6
eBook Packages: Economics and FinanceEconomics and Finance (R0)