A hybrid segmentation method for multivariate time series based on the dynamic factor model

Abstract

There have been a slew of ready-made methods for the segmentation of univariate time series, but in contrast, there are fewer segmentation methods to satisfy the demand for multivariate time series analysis. It has become a common practice to develop more segmentation methods for multivariate time series by extending segmentation methods of univariate time series. But on the contrary, this paper tries to reduce multivariate time series to a univariate common factor sequence to adapt to the methods for segmentation of univariate time series. First, a common factor sequence is extracted from the multivariate time series as a composite index by a dynamic factor model. Then, three typical search methods including binary segmentation, segment neighborhoods and the pruned exact linear time are applied to the common factor sequence to detect the change points and the segmentation result is considered as the final segmentation result of multivariate time series. The case studies show the applicability and robustness of the proposed approach in hydrometeorological time series segmentation.

Appendices

Appendix 1

Let for all t,

$$s_{t}=\left( \begin{array}{c} f_{t} \\ f_{t-1} \\ \vdots \\ f_{t-p+1} \\ \end{array} \right) _{p\times 1},$$

(18)

so Eqs. (7)–(8) can be transformed into the state space form:

$$\begin{aligned} s_{t}=\, & {} Bs_{t-1}+ D\xi _{t}, \end{aligned}$$

(19)

$$\begin{aligned} x_{t} =\, & {} Gs_{t}, \, +\, v_t \end{aligned}$$

(20)

where

$$\begin{aligned} B= & {} \left( \begin{array}{ccccc} {\phi }_{1}, &{} {\phi }_{2}, &{} {\ldots }, &{} {\phi }_{p-1}, &{} {\phi }_{p} \\ 1, &{} 0, &{} {\ldots }, &{} 0, &{} 0 \\ 0, &{} 1, &{} {\ldots }, &{} 0, &{} 0 \\ \vdots &{} \vdots &{} \ddots &{} \vdots &{} \vdots \\ 0, &{} 0, &{} {\ldots }, &{} 1, &{} 0 \\ \end{array} \right) _{p\times p}, \end{aligned}$$

(21)

$$\begin{aligned} D=\, & {} \left[ \left( \begin{array}{cccccccc} 1, &{} 0, {\ldots}, &{} 0 \\ \end{array} \right) _{1\times p}\right] ', \end{aligned}$$

(22)

$$\begin{aligned} \xi _{t}=\, & {} w_{t}, \end{aligned}$$

(23)

$$\begin{aligned} G=\, & {} \left( \begin{array}{cccc} \Lambda _{n\times 1}, &{} {\mathbf{0}}_{n\times (p-1)} \\ \end{array} \right). \end{aligned}$$

(24)

Define

$$\begin{aligned} A=\, & {} \left( \begin{array}{cccc} 1, &{} 0, {\ldots }, &{}0 \\ \end{array} \right) _{1\times p}, \end{aligned}$$

(25)

$$\begin{aligned} B^{*}=\, & {} \left( \begin{array}{cccc} \phi _{1}, &{} \phi _{2}, {\ldots }, &{}\phi _{p} \\ \end{array} \right) _{1\times p}, \end{aligned}$$

(26)

so the following expression holds:

$$As_{t}=f_{t}=\sum _{i=1}^{p}\phi _{i}f_{t-i}+w_{t}=B^{*}s_{t-1}+w_{t}.$$

(27)

Assuming $v_{t}\sim \mathrm {N}(0, \Sigma _{v})$ and the initial state vector $f_{0}$ is distributed $\mathrm {N}(\delta ,\Omega )$, we can express the complete-data log-likelihood function as

$$\begin{array}{ll} \log (L(\Theta ))&{} = -\frac{1}{2}\log (|\Omega |)-\frac{1}{2}(f_{0}-\delta )'\Omega ^{-1}(f_{0}-\delta )\\ &{} -\frac{T}{2}\log (|\Sigma _{w}|)-\frac{1}{2}\sum _{t=1}^{T}(As_{t}-B^{*}s_{t-1})'\Sigma _{w}^{-1}(As_{t}-B^{*}s_{t-1})\\ &{} -\frac{T}{2}\log (|\Sigma _{v}|)-\frac{1}{2}\sum _{t=1}^{T}(x_{t}-\Lambda As_{t})'\Sigma _{w}^{-1}(x_{t}-\Lambda As_{t}){,} \end{array}$$

(28)

where $\Theta =(\mathrm {vec}(\Lambda )',\mathrm {vec}(B^{*})',\mathrm {vech}(\Sigma _{w})',\mathrm {vech}(\Sigma _{v})')$ is the vector containing all the unknown parameters and $\mathrm {vec}(\cdot )$ denotes the vectorization of a matrix column-wise from left to right, and $\mathrm {vech}(\cdot )$ denotes the vectorization of the lower triangular part of a matrix column-wise from left to right. Let

$$X_{T}=:\{x_{t},1\le t\le T\},$$

(29)

and define $Q(\Theta )$ as the expectation of ${\log (L(\Theta ))}$ conditional on $X_{T}$, namely,

$$Q(\Theta )=\mathrm {E}(\log (L(\Theta ))|X_{T}).$$

(30)

We get the iteration formula by calculating the partial differential of Eq. (30) regarding unknown parameters:

$$\begin{aligned} \Sigma _{w}=\, & {} \frac{1}{T}\sum _{t=1}^{T}(AM_{00}A'-B^{*}), \end{aligned}$$

(31)

$$\begin{aligned} \Sigma _{v}=\, & {} \frac{1}{T}\sum _{t=1}^{T}(x_{t}x_{t}'-\Lambda As_{t}^{T}x_{t}'), \end{aligned}$$

(32)

$$\begin{aligned} B^{*}=\, & {} AM_{01}M_{11}^{-1}, \end{aligned}$$

(33)

$$\begin{aligned} \Lambda \;=\, & {} x_{t}(As_{t}^{T})'(AM_{01}A')^{-1}, \end{aligned}$$

(34)

where

$$M_{jk}=\sum _{t=1}^{T}\mathrm {E}(s_{t-j}s_{t-k}|X_{T})=\sum _{t=1}^{T}(P^{T}_{t-j,t-k}+s^{T}_{t-j}(s^{T}_{t-k})'),\; j,k=0,1.$$

(35)

In addition, define

$$s_{r}^{t}:=\mathrm {E}(s_{r}|X_{t})$$

(36)

as the conditional expectation based on $X_{t}$. The conditional variance and covariance based on $X_{t}$ are respectively denoted by

$$P_{r}^{t}:=\mathrm {cov}(s_{r},s_{r}|X_{t})$$

(37)

and

$$P_{r,u}^{t}:=\mathrm {cov}(s_{r},s_{u}|X_{t}),$$

(38)

which can be estimated by the updating and smooth equations of the Kalman filter (Durbin and Koopman 2012). The EM estimation procedure is performed in the following steps (Shumway and Stoffer 2010; Seong et al. 2013):

• (1) Given the initial values $\Theta ^{0},\delta$ and $\Omega$. (In general, the initial values of $\Sigma _{w}$ and $\Sigma _{v}$ are set as the identity matrices with associated dimensions and the initial values of $B^{*}$ and $\Lambda$ are set as the zero matrices with associated dimensions. Moreover, we set $\delta =0$ and $\Omega =\kappa I$, where I is an identity matrix and $\kappa$ is 1 for stationary process and is a large value such as $10^6$ for nonstationary process). On iteration $j, \mathrm {for} \; j = 1,2,\ldots$:

• (2) Compute the negative log-likelihood ${-\log (L_{X}(\Theta ^{j-1}))}$.

• (3) Perform the E-Step of EM algorithm. Obtain smoothed values $s^{T}_{t},P^{T}_{t},P^{T}_{t,t-1}$ for $t=1,\ldots ,T$ by the Kalman filter based on $\Theta ^{j-1}$ and then calculate $M_{ij}$ for $i,j=0,1$ according to Eq. (35).

• (4) Perform the M-Step of EM algorithm. Update the estimates $\Theta ^{j}$ according to Eqs. (31)–(34).

• (5) Repeat Steps (2)–(4) until the likelihood values converge.

Appendix 2

Let for all t,

$$s_{t}=\left( \begin{array}{c} f_{t} \\ f_{t-1} \\ \vdots \\ f_{t-p+1} \\ v_{t} \\ v_{t-1} \\ \vdots \\ v_{t-q+1} \\ \end{array} \right) _{(p+nq)\times 1} ,$$

(39)

so Eqs. (7)–(9) can be transformed into the state space form:

$$\begin{aligned} s_{t}=\, & {} Bs_{t-1}+ D\xi _{t}, \end{aligned}$$

(40)

$$\begin{aligned} x_{t}=\, & {} Gs_{t}, \end{aligned}$$

(41)

where

$$B=\left( \begin{array}{cc} B_{1}, &{} {\mathbf{0}}_{p\times nq} \\ {\mathbf{0}}_{nq\times p}, &{} B_{2} \\ \end{array} \right) ,$$

(42)

with

$$\begin{aligned} B_{1}=\left( \begin{array}{ccccc} {\phi }_{1}, &{} {\phi }_{2}, &{} {\ldots }, &{} {\phi }_{p-1}, &{} {\phi }_{p} \\ 1, &{} 0, &{} {\ldots }, &{} 0, &{} 0 \\ 0, &{} 1, &{} {\ldots }, &{} 0, &{} 0 \\ \vdots &{} \vdots &{} \ddots &{} \vdots &{} \vdots \\ 0, &{} 0, &{} {\ldots }, &{} 1, &{} 0 \\ \end{array} \right) _{p\times p}, \end{aligned}$$

(43)

$$\begin{aligned} B_{2}=\left( \begin{array}{ccccc} {\Psi }_{1}, &{} {\Psi }_{2}, &{} {\ldots }, &{} {\Psi }_{q-1}, &{} {\Psi }_{q} \\ I_{n\times n}, &{} \mathbf {0}_{n\times n}, &{} {\ldots }, &{} \mathbf {0}_{n\times n}, &{} \mathbf {0}_{n\times n} \\ \mathbf {0}_{n\times n}, &{} I_{n\times n}, &{} {\ldots }, &{} \mathbf {0}_{n\times n}, &{} \mathbf {0}_{n\times n} \\ \vdots &{} \vdots &{} \ddots &{} \vdots &{} \vdots \\ \mathbf {0}_{n\times n}, &{} \mathbf {0}_{n\times n}, &{} {\ldots }, &{} I_{n\times n}, &{} \mathbf {0}_{n\times n} \\ \end{array} \right) _{nq \times nq}, \end{aligned}$$

(44)

$$\begin{aligned} D=\left[ \begin{array}{c} \left( \begin{array}{cccccccc} 1, &{} 0, &{} {\ldots }, &{} 0, &{} \mathbf {0}_{1\times n}, &{} \mathbf {0}_{1\times n}, &{} {\ldots }, &{} \mathbf {0}_{1\times n} \\ \mathbf {0}_{n\times 1}, &{} \mathbf {0}_{n\times 1}, &{} {\ldots }, &{} \mathbf {0}_{n\times 1}, &{} I_{n\times n}, &{} \mathbf {0}_{n\times n}, &{} {\ldots }, &{} \mathbf {0}_{n\times n} \\ \end{array} \right) _{(1+n)\times (p+nq)} \\ \end{array} \right] ', \end{aligned}$$

(45)

$$\xi =\left( \begin{array}{c} w_{t} \\ e_{t} \\ \end{array} \right) _{(1+n)\times 1},$$

(46)

$$\begin{aligned} G=\left( \begin{array}{cccc} \Lambda , &{} \mathbf {0}_{n\times (p-1)}, &{} I_{n\times n}, &{} \mathbf {0}_{n\times n(q-1)} \\ \end{array} \right) , \end{aligned}$$

(47)

in which $I_{i\times j}$ and ${\mathbf{0}}_{i\times j}$ stand for an i-by-j identity matrix and an i-by-j zero matrix respectively.

In this case, let $\eta _{t}=x_t-\hat{x}_t$ denote innovations and its variances are signified as $F_t$. The Kalman filter allows the computation of the Gaussian log-likelihood function via the prediction error decomposition (Engle and Watson 1981; Koopman et al. 1999). Assuming

$$x_{t}|x_{1},{\ldots },x_{t-1}\sim \mathrm {N}\left( x_{t|t-1},F_{t}\right) ,$$

(48)

the log-likelihood function is given by

$$\begin{aligned} \log (L\left( \Theta \right) )= & {} \log (p\left( x_1,\ldots ,x_T;\Theta \right) )=\sum _{t=1}^{T}\log (p\left( x_t|x_1,\ldots ,x_{t-1};\Theta \right) ) \\= & {} \sum _{t=1}^{T}-\left( \frac{n}{2}\log (2\pi )+\log (\det (F_{t}))+\eta _{t}'F_{t}^{-1}\eta _{t}\right) =\sum _{t=1}^{T}L_{t} \\= & {} -\frac{nT}{2}\log (2\pi )-\frac{1}{2}\sum _{t=1}^{T}\left[ \log (\det (F_{t}))+\eta _{t}'F_{t}^{-1}\eta _{t}\right] , \end{aligned}$$

(49)

where $\Theta$ is the vector of parameters for a specific statistical model represented in the state space form. The iterative procedure given by Eq. (50) involves finding $H^{k}$, the information matrix evaluated at $\Theta ^{k}$; and $\alpha ^{k}$ is a scalar step length to obtain new estimates $\Theta ^{k+1}$ based upon estimates from the k-th iteration:

$$\Theta ^{k+1}=\Theta ^{k}+\alpha ^{k}\left( H^{k}\right) ^{-1}\frac{\partial L}{\partial \Theta }|_{\Theta ^{k}}.$$

(50)

For a symmetric matrix B, the following expressions are satisfied:

$$\begin{aligned} \frac{\partial |B|}{\partial x}=\, & {} |B|\mathrm {tr}\left( B^{-1}\frac{\partial B}{\partial x}\right) , \end{aligned}$$

(51)

$$\begin{aligned} \frac{\partial B^{-1}}{\partial x}= & {} -B^{-1}\frac{\partial B}{\partial x}B^{-1}. \end{aligned}$$

(52)

Differentiate $L_{t}$ in Eq. (49) with respect to the parameter $\Theta _{i}$ according to Eqs. (51) and (52), we get the following expressions (Engle and Watson 1981):

$$\begin{aligned} \frac{\partial L_{t}}{\partial \Theta _{i}}= & {} -\frac{1}{2}\mathrm {tr}\left( F_{t}^{-1}\frac{\partial F_{t}}{\partial \Theta _{i}}\right) -\left( \frac{\partial \eta _{t}}{\partial \Theta _{i}}\right) 'F_{t}^{-1}\eta _{t} + \frac{1}{2}\eta _{t}'F_{t}^{-1}\frac{\partial F_{t}}{\partial \Theta _{i}}F_{t}^{-1}\eta _{t} \end{aligned}$$

(53)

$$\begin{aligned} \qquad = & {} -\frac{1}{2}\mathrm {tr}\left( F_{t}^{-1}\frac{\partial F_{t}}{\partial \Theta _{i}}\right) \left( I-F_{t}^{-1}\eta _{t}\eta _{t}'\right) -\left( \frac{\partial \eta _{t}}{\partial \Theta _{i}}\right) 'F_{t}^{-1}\eta _{t} \end{aligned}$$

(54)

$$\qquad = L_{1_{t}}+L_{2_{t}}.$$

(55)

To get the second derivative matrix of the log-likelihood, first calculate

$$\begin{aligned} \frac{\partial L_{1_t}}{\partial \Theta _{j}}&= -\frac{1}{2}\mathrm {tr}\left[ \partial \left( F_{t}^{-1}\frac{\partial F_{t}}{\partial \Theta _{i}}\right) /\partial \Theta _{j}\right] \times \left( I-F_{t}^{-1}\eta _{t}\eta _{t}'\right) \\&\quad -\frac{1}{2}\mathrm {tr}\left[ \left( F_{t}^{-1}\frac{\partial F_{t}}{\partial \Theta _{i}}\right) F_{t}^{-1}\frac{\partial F_{t}}{\partial \Theta _{j}}\eta _{t}\eta _{t}'\right] \\&\quad +\frac{1}{2}\mathrm {tr}\left\{ F_{t}^{-1}\frac{\partial F_{t}}{\partial \Theta _{i}}F_{t}^{-1}\times \left[ \frac{\partial \eta _{t}}{\partial \Theta _{j}}\eta _{t}'+\eta _{t}\left( \frac{\partial \eta _{t}}{\partial \Theta _{j}}\right) '\right] \right\} , \end{aligned}$$

(56)

and the only random variables in this expression are the $\eta _{t}$. Hence, taking the expected value of Eq. (56), we have

$$E\left( \frac{\partial L_{1_{t}}}{\partial \Theta _{j}}\right) =-\frac{1}{2}\mathrm {tr}\left( F_{t}^{-1}\frac{\partial F_{t}}{\partial \Theta _{i}}F_{t}^{-1}\frac{\partial F_{t}}{\partial \Theta _{j}}\right) .$$

(57)

Similarly, differentiate $L_{2_{t}}$ with respect to $\Theta _{j}$ to obtain

$$\frac{\partial L_{2_{t}}}{\partial \Theta _{j}}=\frac{\partial ^{2}\eta _{t}}{\partial \Theta _{i}\partial \Theta _{j}}F_{t}^{-1}\eta _{t} -\left( \frac{\partial \eta _{t}}{\partial \Theta _{i}}\right) '\frac{\partial F_{t}^{-1}}{\partial \Theta _{j}}\eta _{t} -\left( \frac{\partial \eta _{t}}{\partial \Theta _{i}}\right) 'F_{t}^{-1}\frac{\partial \eta _{t}}{\partial \Theta _{j}}.$$

(58)

Take expected values of Eq. (58),

$$E\left( \frac{\partial L_{2_{t}}}{\partial \Theta _{j}}\right) =-\left( \frac{\partial \eta _{t}}{\partial \Theta _{i}}\right) 'F_{t}^{-1}\frac{\partial \eta _{t}}{\partial \Theta _{j}}.$$

(59)

The ij-th element of the information matrix is the negative of the sum of Eq. (57) and Eq. (59) summed over all time periods. Thus

$$H_{ij}=\sum _{t}\mathrm {tr}\left[ F_{t}^{-1}\frac{\partial F_{t}}{\partial \Theta _{i}}F_{t}^{-1}\frac{\partial F_{t}}{\partial \Theta _{j}}\right] +\sum _{t}\mathrm {tr}\left( \frac{\partial \eta _{t}}{\partial \Theta _{i}}\right) 'F_{t}^{-1}\frac{\partial \eta _{t}}{\partial \Theta _{j}}.$$

(60)

The expression Eq. (60) requires $\eta _{t}$ and its variance $F_{t}$, which can be calculated numerically by the smoothing equations of the Kalman filter (Durbin and Koopman 2012). In turn, the updated estimate of $\Theta$ will be employed in the equations of the Kalman filter. Further, the iteration process will achieve the goal of estimating the DFM.