Abstract
In recent years, an impressive body of research on predictive accuracy testing and model comparison has been published in the econometrics discipline. Key contributions to this literature include the paper by Diebold and Mariano (J Bus Econ Stat 13:253–263, 1995) which sets the groundwork for much of the subsequent work in the area, West (Econometrica 64:1067–1084, 1996) who considers a variant of the DM test that allows for parameter estimation error in certain contexts, and White (Econometrica 68:1097–1126, 2000) who develops testing methodology suitable for comparing many models. In this chapter, we begin by reviewing various key testing results in the extant literature, both under vanishing and non-vanishing parameter estimation error, with focus on the construction of valid bootstrap critical values in the case of non-vanishing parameter estimation error, under recursive estimation schemes, drawing on Corradi and Swanson (Int Econ Rev 48:67–109, 2007a). We then review recent extensions to the evaluation of multiple confidence intervals and predictive densities, for both the case of a known conditional distribution Corradi and Swanson (J Econ 135:187–228, 2006a; Handbook of economic forecasting Elsevier, Amsterdam, pp 197–284) and of an unknown conditional distribution. Finally, we introduce a novel approach in which forecast combinations are evaluated via the examination of the quantiles of the expected loss distribution. More precisely, we compare models looking at cumulative distribution functions (CDFs) of prediction errors, for a given loss function, via the principle of stochastic dominance, and we choose the model whose CDF is stochastically dominated, over some given range of interest.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Notes
- 1.
- 2.
Here, we use a recursive estimation scheme, where data up to time \(t\ge R\) are used in estimation. West and McCracken (1998) also consider a rolling estimation scheme, in which a rolling windows of \(R\) observations is used for estimation.
- 3.
If we instead use a rolling estimation scheme, then
$$ \widetilde{{\theta }}_{k,t}=\arg \max _{{\theta }_{k}}\left\{ \frac{1}{R }\sum _{j=t-R+1}^{t}q_{k,j}\left( { X}_{k,t},{\theta }_{k}\right) \right\} R\le t\le T. $$ - 4.
- 5.
In the sequel, for ease of notation, the version of the DM test that we discuss will be \(\widehat{S}_{P}(0,k),\) with \(k=1.\)
- 6.
With a slight abuse of notation, in this section the subscript \(0\) denotes the “true” conditional distribution model, rather than the benchmark model; and the subscript 1 thus now denotes the benchmark model.
- 7.
The basic difference between subsampling and “m out of n” bootstrap is that in the latter case we resample overlapping blocks.
References
Amisano, G. and R. Giacomini, 2007, Comparing Density Forecasts via Weighted Likelihood Ratio Tests. Journal of Business and Economic Statistics 25, 177–190.
Andrews, D.W.K., 2002, Higher-Order Improvements of a Computationally Attractive \(k-\)step Bootstrap for Extremum Estimators, Econometrica 70, 119–162.
Andrews, D.W.K. and G. Soares, 2010, Inference for Parameters Defined by Moment Inequalities Using Generalized Moment Selection, Econometrica 78, 119–158.
Bierens, H.J., 2007, Econometric Analysis of Linearized Singular Dynamic Stochastic General Equilibrium Models, Journal of Econometrics, 136, 595–627.
Bierens, H.J., and N.R. Swanson, 2000, The Econometric Consequences of the Ceteris Paribus Condition in Theoretical Economics, Journal of Econometrics, 95, 223–253.
Capistran, C. and A. Timmermann, 2009, Disagreement and Biases in Inflation Expectations, Journal of Money, Credit and Banking 41, 365–396.
Chang, Y.S., J.F. Gomes, and F. Schorfheide, 2002, Learning-by-Doing as a Propagation Mechanism, American Economic Review 92, 1498–1520.
Christiano, L.J., 1988, Why Does Inventory Investment Fluctuate So Much, Journal of Monetary Economics 21, 247–280.
Christoffersen, P., 1998, Evaluating Interval Forecasts, International Economic Review 39, 841–862.
Christoffersen, P. and F.X. Diebold, 1996, Further Results on Forecasting and Model Selection under Asymmetric Loss, Journal of Applied Econometrics, 11, 561–572.
Christoffersen, P. and F.X. Diebold, 1997, Optimal Prediction Under Asymmetric Loss, Econometric Theory 13, 808–817.
Christoffersen, P. and F.X. Diebold, 2000, How Relevant is Volatility Forecasting for Financial Risk Management?, Review of Economics and Statistics 82, 12–22.
Clark, T.E. and M.W. McCracken, (2001), Tests of Equal Forecast Accuracy and Encompassing for Nested Models, Journal of Econometrics 105, 85–110.
Cogley, T., and J.M. Nason, 1995, Output Dynamics for Real Business Cycles Models, American Economic Review 85, 492–511.
Corradi, V. and W. Distaso, 2011, Multiple Forecast Evaluation, Oxford Handbook of Economic Forecasting, eds. D.F. Hendry and M.P. Clements, Oxford University Press, Oxford.
Corradi, V. and N.R. Swanson, 2005, A test for comparing multiple misspecified conditional intervals. Econometric Theory 21, 991–1016.
Corradi, V. and N.R. Swanson, 2006a, Predictive density and conditional confidence interval accuracy tests. Journal of Econometrics 135, 187–228.
Corradi, V. and N.R. Swanson, 2006b, Predictive density evaluation, in: C.W.J. Granger, G. Elliot and A. Timmermann, (Eds.), Handbook of economic forecasting Elsevier, Amsterdam, pp. 197–284.
Corradi, V. and N.R. Swanson, 2007a, Nonparametric bootstrap procedures for predictive inference based on recursive estimation schemes. International Economic Review 48, 67–109.
Corradi, V. and N.R. Swanson, 2007b, Evaluation of dynamic stochastic general equilibrium models based on distributional comparison of simulated and historical data. Journal of Econometrics 136, 699–723.
Diebold, F.X. and R.S. Mariano, 1995, Comparing predictive accuracy. Journal of Business and Economic Statistics 13, 253–263.
Diebold, F.X., L.E. Ohanian, and J. Berkowitz, 1998, Dynamic Equilibrium Economies: A Framework for Comparing Models and Data, Review of Economic Studies 65, 433–451.
Duffie, D. and J. Pan, 1997, An Overview of Value at Risk, Journal of Derivatives 4, 7–49.
Elliott, G. and A. Timmermann, 2004, Optimal Forecast Combinations Under General Loss Functions and Forecast Error Distributions, Journal of Econometrics 122, 47–79.
Elliott, G. and A. Timmermann, 2005, Optimal Forecast Combination Under Regime Switching, International Economic Review 46, 1081–1102.
Elliott, G., Komunjer I., and A. Timmermann, 2008, Biases in Macroeconomic Forecasts: Irrationality or Asymmetric Loss? 2008, Journal of the European Economic Association, 6, 122–157.
Fernandez-Villaverde, J. and J.F. Rubio-Ramirez, 2004, Comparing Dynamic Equilibrium Models to Data, Journal of Econometrics 123, 153–180.
Giacomini, R. and I. Komunjer, 2005, Evaluation and Combination of Conditional Quantile Forecasts, Journal of Business and Economic Statistics 23, 416–431.
Giacomini, R., and H. White, 2006, Conditional Tests for Predictive Ability, Econometrica 74, 1545–1578.
Goncalves, S., and H. White, 2004, Maximum Likelihood and the Bootstrap for Nonlinear Dynamic Models, Journal of Econometrics, 119, 199–219.
Hall, P., and J.L. Horowitz, 1996, Bootstrap Critical Values for Tests Based on Generalized Method of Moments Estimators. Econometrica 64, 891–916.
Hansen, P. R., 2005, A Test for Superior Predictive Ability, Journal of Business and Economic Statistics 23, 365–380.
Hansen, P.R., A. Lunde and J.M. Nason, 2011, The Model Confidence Set, Econometrica, 79, 453–497.
Inoue, A., and M. Shintani, 2006, Bootstrapping GMM Estimators for Time Series, Journal of Econometrics 133, 531–555.
King, R.G., C.I. Plosser, and S.T. Rebelo, 1988a, Production, Growth and Business Cycles 1: The Basic Neoclassical Model, Journal of Monetary Economics 21, 195–232.
King, R.G., C.I. Plosser, and S.T. Rebelo, 1988b, Production, Growth and Business Cycles 2: New Directions, Journal of Monetary Economics 21, 309–341.
Kitamura, Y., 2004, Econometric Comparisons of Conditional Models, Working Paper, Yale University.
Kunsch H.R., 1989, The Jackknife and the Bootstrap for General Stationary Observations. Annals of Statistics 17, 1217–1241.
Kydland, F.E., and E.C. Prescott, 1982, Time to Build and Aggregate Fluctuations, Econometrica 50, 1345–1370.
Linton, O., E. Maasoumi and Y.J. Whang, 2005, Consistent Testing for Stochastic Dominance Under General Sampling Schemes, Review of Economic Studies 72, 735–765.
Long, J.B. and C.I. Plosser, 1983, Real Business Cycles, Journal of Political Economy 91, 39–69.
Romano, J.P. and M. Wolf, 2005, Stepwise Multiple Testing as Formalized Data Snooping, Econometrica 73, 1237–1282.
Schorfheide, F., 2000, Loss Function Based Evaluation of DSGE Models, Journal of Applied Econometrics 15, 645–670.
Schmitt-Grohe, S., 2000, Endogenous Business Cycles and the Dynamics of Output, Hours and Consumption, American Economic Review 90, 1136–1159.
Smith, J. and K.F. Wallis, 2009, A Simple Explanation of the Forecast Combination Puzzle, Oxford Bullettin of Economics and Statistics 71, 331–355.
Sullivan R., A. Timmermann, and H. White, 1999, Data-Snooping, Technical Trading Rule Performance, and the Bootstrap, Journal of Finance 54, 1647–1691.
Sullivan, R., A. Timmermann, and H. White, 2001, Dangers of Data-Driven Inference: The Case of Calendar Effects in Stock Returns, Journal of Econometrics 105, 249–286.
Vuong, Q., 1989, Likelihood Ratio Tests for Model Selection and Non-Nested Hypotheses, Econometrica 57, 307–333.
Wallis, K.F., 2005, Combining Interval and Density Forecast: A Modest Proposal, Oxford Bullettin of Economics and Statistics 67, 983–994.
Watson, M.W., 1993, Measure of Fit for Calibrated Models, Journal of Political Economy 101, 1011–1041.
West, K.F., 1996, Asymptotic Inference About Predictive Ability, Econometrica 64, 1067–1084.
West, K.F., and M.W. McCracken, 1998, Regression Based Tests for Predictive Ability, International Economic Review 39, 817–840.
White, H., 1982, Maximum Likelihood Estimation of Misspecified Models, Econometrica 50, 1–25.
White, H., 2000, A reality check for Data Snooping, Econometrica 68, 1097–1126.
Zellner, A., 1986, Bayesian Estimation and Prediction Using Asymmetric Loss Function, Journal of the American Statistical Association 81, 446–451.
Acknowledgments
This chapter has been prepared for the Festschrift in honor of Halbert L. White in the event of the conference celebrating his sixtieth birthday, entitled “Causality, Prediction, and Specification Analysis: Recent Advances and Future Directions”, and held at the University of California, San Diego on May 6–7, 2011. Swanson thanks the Rutgers University Research Council for financial support.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2013 Springer Science+Business Media New York
About this chapter
Cite this chapter
Corradi, V., Swanson, N.R. (2013). A Survey of Recent Advances in Forecast Accuracy Comparison Testing, with an Extension to Stochastic Dominance. In: Chen, X., Swanson, N. (eds) Recent Advances and Future Directions in Causality, Prediction, and Specification Analysis. Springer, New York, NY. https://doi.org/10.1007/978-1-4614-1653-1_5
Download citation
DOI: https://doi.org/10.1007/978-1-4614-1653-1_5
Published:
Publisher Name: Springer, New York, NY
Print ISBN: 978-1-4614-1652-4
Online ISBN: 978-1-4614-1653-1
eBook Packages: Business and EconomicsEconomics and Finance (R0)