Abstract
Model selection is a complex task widely examined in the literature due to the major gains in forecasting accuracy when performed successfully. To do so, many approaches have been proposed exploiting the available historical data in different ways. In-sample testing is the most common approach but is highly affected by the data and parameter estimation uncertainty. Out-of-sample tests, which use part of the data to evaluate the performance of the forecasting methods, are also well-known alternatives which usually lead to improvements. However, these are still vulnerable to data uncertainty such as noise and outliers. On the other hand, resampling techniques can be used to produce multiple clones of a time series with the same characteristics but a different component of randomness. In this paper, a model selection technique is proposed which takes advantage of the bootstrapping process to mitigate the effect of noise in the original data and then applies out-of-sample tests to the generated series to evaluate the forecasting performance of different methods. The approach is assessed across a large dataset of diverse time series and benchmarked versus other traditional approaches leading to promising results.
Similar content being viewed by others
References
Akaike H (1970) Statistical predictor identification. Ann Inst Stat Math 21:243–247
Akaike H (1974) A new look at the statistical model identification. IEEE Trans Autom Control 19(6):716–723
Assimakopoulos V, Nikolopoulos K (2000) The theta model: a decomposition approach to forecasting. Int J Forecast 16:521–530
Atsalakis G (2014) New technology product demand forecasting using a fuzzy inference system. Oper Res Int J 14(2):225–236
Austin PC, Tu JV (2004) Bootstrap methods for developing predictive models. Am Stat 58(2):131–137
Bergmeir C, Hyndman RJ, Benitez J (2016) Bagging exponential smoothing methods using STL decomposition and Box–Cox transformation. Int J Forecast 32:303–312
Bork L, Møller SV (2015) Forecasting house prices in the 50 states using dynamic model averaging and dynamic model selection. Int J Forecast 31:63–78
Box GEP, Cox DR (1964) An analysis of transformations. J R Stat Soc B 26:211–252
Buhlmann P (1997) Sieve bootstrap for time series. Bernoulli 3:123–148
Carlstein E (1992) Resampling techniques for stationary time series: some recent developments. New directions in time series analysis. Part I. Holden-Day, Springer, San Francisco, New York
Claeskens G, Magnus JR, Vasnev AL, Wang W (2016) The forecast combination puzzle: a simple theoretical explanation. Int J Forecast 32:754–762
Cleveland RB, Cleveland WS, McRae J, Terpenning I (1990) A seasonal-trend decomposition procedure based on loess. J Off Stat 6:3–73
Cleveland WS, Grosse E, Shyu WM (1992) Local regression models. Statistical models in S. Chapman & Hall/CRC, London (Chapter 8)
Efron B (1979) Bootstrap methods: another look at the jackknife. Ann Stat 7:1–26
Efron B (2003) Second thoughs on the bootstrap. Stat Sci 18(2):135–140
Fildes R (1989) Evaluation of aggregate and individual forecast method selection rules. Manag Sci 39:1056–1065
Fildes R (2001) Beyond forecasting competitions. Int J Forecast 17:556–560
Fildes R, Petropoulos F (2015) Simple versus complex selection rules for forecasting many time series. J Bus Res 68:1692–1701
Gaglianone WP, Marins JTM (2017) Evaluation of exchange rate point and density forecasts: an application to Brazil. Int J Forecast 33:707–728
Gardner ES (1985) Exponential smoothing: the state of the art. J Forecast 4(1):1–28
Guerrero V (1993) Time-series analysis supported by power transformations. J Forecast 12:37–48
Hall P, Horowitz JL, Ying BY (1995) Blocking rules for the bootstrap with dependent data. Biometrika 82:561–574
Hyndman RJ, Athanasopoulos G (2014) Forecasting: principles and practice. OTexts, Clayton
Kunsch HR (1989) The jackknife and the bootstrap for general stationary observations. Ann Stat 17:1217–1241
Lahiri SN (1999) Theoritical comparisons of block bootstrap methods. Ann Stat 27:386–404
Ledolter J (1989) The effect of additive outliers on the forecasts from ARIMA models. Int J Forecast 5(2):231–240
Makridakis S, Hibon M (2000) The M3-competition: results, conclusions and implications. Int J Forecast 16:451–476
Makridakis S, Winkler RL (1989) Sampling distributions of post-sample forecasting errors. Appl Stat J R Stat Soc Ser C 38:331–342
Makridakis S, Wheelwright SC, Hyndman RJ (1998) Forecasting: methods and applications, 3rd edn. Wiley, New York
Ord K, Fildes R (2013) Principles of business forecasting. South-Western Cengage Learning, Boston
Papageorgiou M, Kotsialos A, Poulimenos A (2001) Long term sales forecasting for industrial supply chain management. Oper Res Int J 1(3):241–261
Politis DN, Romano JP (1992) A circular block resampling procedure for stationary data. In exploring the limits of bootstrap. Wiley, New York
Politis DN, Romano JP (1994) The stationary bootstrap. J Am Stat Assoc 89:1303–1313
Shah C (1997) Model selection in univariate time series forecasting using discriminant analysis. Int J Forecast 13:489–500
Singh K (1981) On the asymptotic accuracy of Efron’s bootstrap. Ann Stat 9:1187–1195
Tashman LJ (2000) Out-of-sample tests of forecasting accuracy: an analysis and review. Int J Forecast 16:437–450
Tian W, Song J, Li Z, Wilde P (2014) Bootstrap techniques for sensitivity analysis and model selection in building thermal performance analysis. Appl Energy 135:320–328
Voulgaridou D, Kirytopoulos K, Leopoulos V (2009) An analytic network process approach for sales forecasting. Oper Res Int J 9(1):35–53
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Sarris, D., Spiliotis, E. & Assimakopoulos, V. Exploiting resampling techniques for model selection in forecasting: an empirical evaluation using out-of-sample tests. Oper Res Int J 20, 701–721 (2020). https://doi.org/10.1007/s12351-017-0347-0
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s12351-017-0347-0