Assessing distributional properties of forecast errors for fan-chart modelling

Abstract

This paper considers the problem of assessing the distributional properties (normality and symmetry) of macroeconomic forecast errors of G7 countries for the purpose of fan-chart modelling. Our results indicate that the assumption of symmetry of the marginal distribution of forecast errors is reasonable, whereas the assumption of normality is not, making symmetric prediction intervals clearly preferable.

Appendices

Figures and Tables

See Fig. 2 and Tables 1, 2, 3, and  4.

Fig. 2

Aggregate CE forecast errors (final data)

Table 1 P-values of normality test $\mathcal {T}_N$: GDP

Table 2 P-values of normality test $\mathcal {T}_N$: CPI

Table 3 P-values of symmetry test $\mathcal {T}_S$: GDP

Table 4 P-values of symmetry test $\mathcal {T}_S$: CPI

Simulation study

In this section, we present and discuss the results of a simulation study examining the small-sample properties of the normality and symmetry tests under different patterns of dependence by considering artificial data generated according to the following ARMA models

• M1: $X_{t}=0.5X_{t-1}+\varepsilon _{t}$,

• M2: $X_{t}=0.6X_{t-1}-0.5X_{t-2}+\varepsilon _{t}$,

• M3: $X_{t}=0.6X_{t-1}+0.3\varepsilon _{t-1}+\varepsilon _{t}$.

Here, and throughout this section, $\{\varepsilon _{t}\}$ are i.i.d. random variables, the common distribution of which is either standard normal (labelled N) or generalized lambda with quantile function $Q(w)=\lambda _{1}+(1/\lambda _{2})\{w^{\lambda _{3}}-(1-w)^{\lambda _{4}}\}$, $0<w<1$, standardized to have zero mean and unit variance (see Ramberg and Schmeiser 1974). The parameter values of the generalized lambda distribution used in the experiments are taken from Bai and Ng (2005) and can be found in Table 5, along with the corresponding coefficients of skewness and kurtosis; the distributions N, S1, S2 are symmetric, whereas A1, A2, A3 are asymmetric.

For each design point, 1000 independent realizations of $\{X_{t}\}$ of length $n + 100$, with $n \in \{100,200\}$ (as representative samples for macroeconomic applications), are generated.¹² The first 100 data points of each realization are then discarded in order to eliminate start-up effects, and the remaining n data points are used to compute the value of the $\mathcal {T}_N$ and $\mathcal {T}_S$ test statistics. In the case of bootstrap tests, the order of the autoregressive sieve is determined by minimizing the AIC in the range $1 \le p < 5\log _{10}(n)$, while the number of bootstrap replications is $B = 499$. We note that using a larger number of bootstrap replications did not change the results substantially (see Davison and Hinkley 1997, pp. 155–156, for an explanation).

Table 5 Innovation distributions

Table 6 Empirical rejection frequencies of tests: AIC

Table 7 Empirical rejection frequencies of tests: BIC

The Monte Carlo rejection frequencies of the test statistics at 5% significance level are reported in Table 6. The null rejection probabilities of the tests are generally insignificantly different from the nominal level across all relevant DGPs. Their rejection frequencies improve with both the sample size and non-normality in the distribution of innovations, although not uniformly (compare the results for A1 and A2). To assess the sensitivity of results with respect to the method used to determine the order of the autoregressive sieve, we consider selecting the latter by minimizing BIC in addition to AIC. The rejection frequencies are reported in Table 7. It is clear that there is little to choose between AIC and BIC, the rejection frequencies not being notably different across the two criteria for any given combination of noise distribution and the sample size n. It is worth noting that results from experiments based on artificial time series confirm the robustness of the properties of the test procedure with respect to the choice of order selection criterion.