A toolbox of permutation tests for structural change

Abstract

The sup$LM$ test for structural change is embedded into a permutation test framework for a simple location model. The resulting conditional permutation distribution is compared to the usual (unconditional) asymptotic distribution, showing that the power of the test can be clearly improved in small samples. Furthermore, the permutation test is embedded into a general framework that encompasses tools for binary and multivariate dependent variables as well as model-based permutation testing for structural change. It is also demonstrated that the methods can not only be employed for analyzing structural changes in time series data but also for recursive partitioning of cross-section data. The procedures suggested are illustrated using both artificial data and empirical applications (number of youth homicides, employment discrimination data, carbon flux in tropical forests, stock returns, and demand for economics journals).

Appendix

1.1 Proofs

The (asymptotic) distribution of the multivariate statistic $Z = (Z_{\pi _1}, \dots , Z_{\pi _m})^\top$ is derived by embedding the statistic into the framework of Strasser and Weber (1999), as discussed in Hothorn et al. (2006a). More precisely, the test statistic $\max _{\pi \in \Pi } Z_\pi$ considered above is the maximum-type statistic $c_{\max }$ of Hothorn et al. (2006a) if the influence function $h(Y) = Y$ is used for the observations $Y_i$ and the transformation $g(t) = (\mathbf{1 }_{[0, \pi _1]}(t), \dots , \mathbf{1 }_{[0, \pi _m]}(t))^\top$ is used for the associated timings $t_i$. The transformation $g$ used the indicator function $\mathbf{1 }_I$ of the interval $I$ and thus corresponds to a vector of indicators for the time up to the timings $\pi _j$ ($j = 1, \dots , k$).

Using these transformations $h(\cdot )$ and $g(\cdot )$, the unstandardized test statistic $T$ is in the notation of Hothorn et al. (2006a)

$$\begin{aligned} T \, = \, \text{ vec} \left( \sum _{i = 1}^n g(t_i) h(Y_i)^\top \right) \, = \, \left( n_{1, \pi _1} \bar{Y}_{1, \pi _1}, \dots , n_{1, \pi _m} \bar{Y}_{1, \pi _m} \right)^\top . \end{aligned}$$

(8)

Under $H_0$, given all permutations $\sigma \in S$ of the observations $Y_1, \dots , Y_n$, the unstandardized statistic has expectation

$$\begin{aligned} \text{ E}_\sigma [T] \, = \, \text{ vec} \left( \left( \sum _{i = 1}^n g(t_i) \right) n^{-1} \sum _{i = 1}^n h(Y_i)^\top \right) \, = \, \left( n_{1, \pi _1}, \dots , n_{1, \pi _m} \right)^\top \bar{Y} \end{aligned}$$

(9)

and each unstandardized statistic has variance

$$\begin{aligned} \text{ Var}_\sigma [T_\pi ] \, = \, \left( n_{1, \pi } - \frac{n_{1, \pi }^2}{n} \right) \frac{RSS_{0}}{n-1} \, = \, \frac{n_{1, \pi } n_{2, \pi }}{n} \frac{RSS_{0}}{n-1}, \end{aligned}$$

(10)

where the residual sum of squares is $RSS_0 = \sum _{i = 1}^n (Y_i - \bar{Y})^2$. The equations directly follow from Eq. 7 in Strasser and Weber (1999).

Standardizing the vector of raw statistics $T = (T_{\pi _1}, \dots , T_{\pi _m})^\top$ by their respective mean and standard deviation yields the vector of statistics $Z = (Z_{\pi _1}, \dots , Z_{\pi _m})^\top$:

$$\begin{aligned} Z_\pi&= \frac{T_\pi - \text{ E}_\sigma [T_\pi ]}{\sqrt{\text{ Var}_\sigma [T_\pi ]}} \\&= \frac{n_{1, \pi } \bar{Y}_{1, \pi } - n_{1, \pi } \bar{Y}}{\sqrt{\frac{n_{1, \pi } n_{2, \pi }}{n} \frac{RSS_{0}}{n-1}}} \\&= \sqrt{ \frac{n_{1, \pi } n_{2, \pi }}{n} } \frac{\bar{Y}_{1, \pi } - \bar{Y}_{2, \pi }}{\sqrt{ RSS_0 / (n-1) }}, \end{aligned}$$

because of the following simple relationship between $\bar{Y}_{1, \pi }$, $\bar{Y}_{2, \pi }$ and $\bar{Y}$:

$$\begin{aligned} \bar{Y} \, = \, \frac{n_{1, \pi } \bar{Y}_{1, \pi } + n_{2, \pi } \bar{Y}_{2, \pi }}{n}. \end{aligned}$$

Consequently, $Z$ has zero mean and unit variance given all permutations $\sigma \in S$. Similarly, the covariance between two elements of $Z$, $Z_\pi$ and $Z_\tau$ say, is

$$\begin{aligned}&\frac{RSS_0}{n-1} \left( \sum _{i = 1}^n \mathbf{1 }_{[0, \pi ]}(t_i) \mathbf{1 }_{[0, \tau ]}(t_i) \right) - \frac{1}{n} \frac{RSS_0}{n-1} \left( \sum _{i = 1}^n \mathbf{1 }_{[0, \pi ]}(t_i) \right) \left( \sum _{i = 1}^n \mathbf{1 }_{[0, \tau ]}(t_i) \right) \\&\qquad \quad = \frac{n \min (n_{1, \pi }, n_{1, \tau }) - n_{1, \pi } n_{1, \tau }}{n} \frac{RSS_0}{n-1}. \end{aligned}$$

Assuming that $\pi < \tau$ and using the variance computed above, the correlation is thus

$$\begin{aligned} \frac{n_{1, \pi } n_{2, \tau }}{\sqrt{n_{1, \pi } n_{2, \pi } n_{1, \tau } n_{2, \tau }}}. \end{aligned}$$

Given that we derived the first two moments of $Z$ by embedding the statistic into the framework of Strasser and Weber (1999), the asymptotic normality of $Z$ follows by application of their Theorem 2.3.

1.2 Power and size for autocorrelated series

To study the performance of the tests with dependent data, we use a simulation setup as in Sect. 2.3. The only difference is that the errors are now autocorrelated with $\varrho = 0, 0.1, 0.2, 0.3, 0.5, 0.9$. The length of the time series considered is either very short ($n = 10$) or moderate ($n = 50$) and either there is no change ($\delta = 0$) or a large shift in the mean ($\delta = 15$). Five different versions of the tests are assessed: the unconditional and conditional test ($\mathcal D _{\infty }$ and $\mathcal D _{\sigma |Y}$, respectively) on the original data (as in Sect. 2.3), the unconditional test computed with a robust HAC covariance estimate and the unconditional and conditional test computed on the residuals of an AR(1) model (fitted by OLS).

Using the uncorrected tests on the original data ($\mathcal D _{\infty }$ and $\mathcal D _{\sigma |Y}$, respectively), it can be seen that size distortions occur for $\varrho > 0$ even if there is no change ($\delta = 0$). For $\varrho$ up to $0.2$ these are still moderate but become very large afterwards and are even more pronounced for the conditional version of the test. However, in moderately large time series ($n = 50$), the problem can be remedied by either using a HAC correction or applying the tests to the AR(1) residuals. These three tests keep their size and have reasonable power with the conditional test having the highest power. However, none of the tests is able to distinguish between autocorrelation and a shift in the mean (with $\delta = 15$) if the time series is very short ($n = 10$) or if the autocorrelation is very high ($\varrho \ge 0.7$) and the length only moderate ($n = 50$).