Survey-Based Forecasting: To Average or Not to Average

Abstract

Forecasting inflation rate is of tremendous importance for firms, consumers, as well as monetary policy makers. Besides macroeconomic indicators, professional surveys deliver experts’ expectation and perception of the future movements of the price level. This research studies survey-based inflation forecast in an extended recent sample covering the Great Recession and its aftermath. Traditional methods extract the central tendency in mean or median and use it as a predictor in a simple linear model. Among the three widely cited surveys, we confirm the superior forecasting capability of the Survey of Professional Forecasters (SPF). While each survey consists of many individual experts, we utilize machine learning methods to aggregate the individual information. In addition to the off-the-shelf machine leaning algorithms such as the Lasso, the random forest and the gradient boosting machine (GBM), we tailor the standard Lasso by differentiating the penalty level according to an expert’s experience, in order to handle for participants’ frequent entries and exits in surveys. The tailored Lasso delivers strong empirical results in the SPF and beats all other methods except for the overall best performer, GBM. Combining forecasts of the tailored Lasso model and GBM further achieves the most accurate inflation forecast in both the SPF and the Livingston Survey, which beyonds the reach of a single machine learning algorithm. We conclude that combination of machine learning forecasts is a useful technique to predict inflation, and averaging should be exercised in a new generation of algorithms capable of digesting disaggregated information.

Appendix: Cross Validation and Tuning of the Hyperparameters

Hyperparameters in models of Sect. 2.1 were tuned within sets of designed grids through a series of nested cross-validation procedure. The nested cross-validation are described in Fig. 7 while the tuning grids of the hyperparameters are displayed in Table 6.

Fig. 7

The Nested Cross Validation Procedure $\bullet$ Notes This graph is for illustration purpose only and the number of data points involved are different from the real data

The nested cross-validation is conducted by both the inner loops and the outer loops, as shown Fig. 7. Given that the outer loop is built by a series of rolling windows to train for optimal parameters, each of the training set is then divided into training subsets for tuning hyperparameters. Optimal model is then selected according to the RMSE. In this study, lengths of training subsets were set to be 80% of the training length. According to Varma and Simon [24] (2006), this nested cross-validation approach estimates the true error with nearly zero bias.

Table 6 lists the grids of the tuning hyperparameters on which we evaluate the machine learning methods in Sect. 2.1. For Lasso, we took exponential for the lambda values $\lambda$ and $\lambda _{2}$, in order to be compatible with the standard way to compare the forecasting errors with values of $\ln \left( \lambda \right)$, instead of $\lambda$.

Table 6 Tuning Grids of Hyperparameters in models of Sect. 2.1

For our tailored Lasso, the adaptive penalty level $\lambda _{1}$ is designed to be $\ln \left( x\right)$. Based on our design, the penalty factor of the model, $\lambda _{1}\exp (\lambda _{2}/s_{it})$, is more sensitive to an increase in the accumulated number of forecasts $s_{it}$ when $s_{it}$ is at a relatively low level. In other words, while forecasters who have accumulated 2 forecasts are penalized much lighter than those who have made merely 1 forecast, the difference between the penalties imposed on forecasters who have accumulated 20 forecasts and 21 forecasts are much smaller. It is intuitive that professional skills improve with experience in a decreasing rate. On the other hand, as SPF consist of quarterly data and the Livingston Survey is of half-yearly data, we set $\lambda _{1}=\ln (10)$ for the SPF while $\lambda _{1}=\ln (5)$ for the Livingston Survey. Optimal design of the adaptive factor $\lambda _{1}$ requires further tuning in the future.

Furthermore, since more trees is in general beneficial to the performance of the random forest algorithm, we skipped its tuning procedure and take the default value of 500. On the other hand, to avoid potential overfitting caused by applying an excessive number of trees on GBM algorithm, we tuned for the number of trees with values from 100 to 500 for a fair comparison between GBM and the random forest.

Since data on individual response is not available for the Survey of Consumers, models in Sect. 2.1 would be applied on the datasets of the Survey of Profession Forecasters and the Livingston Survey, but not the Survey of Consumers. Besides, parameters in models of Sect. 2.1 were tuned through a series of nested cross-validation procedure. This nested cross-validation approach could estimate the true error with nearly zero bias (Varma and Simon [24]).