hhsmm: an R package for hidden hybrid Markov/semi-Markov models

Abstract

This paper introduces the hhsmm R package, which involves functions for initializing, fitting, and predication of hidden hybrid Markov/semi-Markov models. These models are flexible models with both Markovian and semi-Markovian states, which are applied to situations where the model involves absorbing or macro-states. The left-to-right models and the models with series/parallel networks of states are two models with Markovian and semi-Markovian states. The hhsmm also includes Markov/semi-Markov switching regression model as well as the auto-regressive HHSMM, the nonparametric estimation of the emission distribution using penalized B-splines, prediction of future states and the residual useful lifetime estimation in the predict function. The commercial modular aero-propulsion system simulation (C-MAPSS) data-set is also included in the package, which is used for illustration of the application of the package features. The application of the hhsmm package to the analysis and prediction of the Spain’s energy demand is also presented.

Appendix

1.1 Forward-backward algorithm for the HHSMM model

Denote the sequence $\{Y_{t},\ldots ,Y_{s}\}$ by $Y_{t:s}$ and suppose that the observation sequence $X_{0:\tau -1}$ with the corresponding hidden state sequence $S_{0:\tau -1}$ is observed. The forward-backward algorithm is an algorithm to compute the probabilities

$$\begin{aligned}L_j(t)=P(S_t=j|X_{0:{\tau -1}}=x_{0:{\tau -1}})\end{aligned}$$

within the E-step of the EM algorithm. The above probabilities are computed in the backward recursion of the forward-backward algorithm.

For a semi-Markovian state j, and $t=0,\ldots ,\tau -2$, the forward recursion computes the following probabilities (Guédon 2005),

$$\begin{aligned} F_j(t)= & P(S_{t+1}\ne j,S_t = j|X_{0:t} = x_{0:t})\nonumber \\= & \frac{f_j(x_t)}{N_t}\Biggl [\sum _{{u=1}}^{t}\Bigg \{\prod _{\nu =1}^{u-1}\frac{f_j(x_{t-\nu })}{N_{t-\nu }}\Bigg \}d_j(u)\sum _{i\ne j}p_{ij}F_i(t-u)\nonumber \\ &\qquad+\Bigg \{\prod _{\nu =1}^{t}\frac{f_j(x_{t-\nu })}{N_{t-\nu }}\Bigg \}d_j(t+1)\pi _j\Biggl ], \end{aligned}$$

(A.21)

and

$$\begin{aligned} F_j(\tau -1)= & P(S_{\tau -1} = j|X_{0:{\tau -1}} =x_{0:{\tau -1}})\nonumber \\= & \frac{f_j(x_{\tau -1})}{N_{\tau -1}}\Biggl [\sum _{u=1}^{\tau -1}\Bigg \{\prod _{\nu =1}^{u-1}\frac{f_j(x_{\tau -1-\nu })}{N_{\tau -1-\nu }}\Bigg \}D_j(u)\sum _{i\ne j}p_{ij}F_i(\tau -1-u)\nonumber \\ &\qquad+\Bigg \{\prod _{\nu =1}^{\tau -1}\frac{f_j(x_{\tau -1-\nu })}{N_{\tau -1-\nu }}\Bigg \}D_j(\tau )\pi _j\Biggl ]. \end{aligned}$$

(A.22)

where the normalizing factor $N_t$ is computed as follows

$$\begin{aligned} N_t = P(X_t = x_t|X_{0:{t-1}} = x_{0:{t-1}}) =\sum _{j}P(S_t = j,X_t = x_t|X_{0:{t-1}} = x_{0:{t-1}}) \end{aligned}$$

For a Markovian state j, and for $t = 0$,

$$\begin{aligned} {\tilde{F}}_j(0) = P(S_0 = j|X_0 = x_0)=\frac{f_j(x_0)}{N_0}\pi _j\end{aligned}$$

and for $t = 1,\ldots ,\tau -1$

$$\begin{aligned} {\tilde{F}}_j(t) = P(S_t = j|X_{0:t} = x_{0:t}) =\frac{f_j(x_t)}{N_t}\sum _{i}{\tilde{p}}_{ij}{\tilde{F}}_i(t-1) \end{aligned}$$

(A.23)

The log-likelihood of the model is then

$$\begin{aligned}\log P(X_{0:{\tau -1}} = x_{0:{\tau -1}};\theta ) = \sum _{t=0}^{\tau -1}\log N_t,\end{aligned}$$

which is used as a criteria for convergence of the EM algorithm and the evaluate the quality of the model. The backward recursion is initialized by

$$\begin{aligned}L_j(\tau -1) = P(S_{\tau -1} = j|X_{0:{\tau -1}} = x_{0:{\tau -1}}) = F_j(\tau -1) = {\tilde{F}}_j(\tau -1).\end{aligned}$$

For a semi-Markovian state j, we have

$$\begin{aligned}L_j(\tau -1) = P(S_{\tau -1} = j|X_{0:{\tau -1}} = x_{0:{\tau -1}}) = F_j(\tau -1), \end{aligned}$$

and for $t = \tau -2,\ldots ,0$,

$$\begin{aligned} L_j(t)= & {} L1_j(t) + L_j(t+1) - G_j(t+1)\sum _{i\ne j}{p}_{ij}{\tilde{F}}_i(t), \end{aligned}$$

(A.24)

where

$$\begin{aligned} L1_j(t)= & {} \sum _{k\ne j}G_k(t+1)p_{jk}F_j(t), \end{aligned}$$

(A.25)

and

$$\begin{aligned} G_j(t+1)= & {} \sum _{u=1}^{\tau -1-t}G_j(t+1,u) \end{aligned}$$

with

$$\begin{aligned} G_j(t+1,u)=\left\{ \prod _{\nu =0}^{\tau -2-t}\frac{f_j(x_{\tau -1-\nu })}{N_{\tau -1-\nu }}\right\} D_j(\tau -1-t). \end{aligned}$$

For a Markovian state j and for $t = \tau -2,\ldots ,0$

$$\begin{aligned} L_j(t)= & {} L1_j(t) \end{aligned}$$

(A.26)

Furthermore, if a mixture of multivariate normal distributions with probability density function

$$\begin{aligned} f_j(x) = \sum _{k=1}^{K_j}\lambda _{kj} {{\mathcal {N}}}_p(x; \mu _{kj}, \Sigma _{kj}),\quad j=1,\ldots ,J, \end{aligned}$$

(A.27)

is considered as the emission distribution, then the following probabilities of the mixture components are computed in the E-step of the $(s+1)$th iteration of the EM algorithm

$$\begin{aligned} \gamma _{kj}^{(s+1)}(t) = \frac{\lambda _{kj}^{(s)}{{\mathcal {N}}}_p(x_t; \mu _{kj}^{(s)}, \Sigma _{kj}^{(s)})}{f_j(x_t) }, \end{aligned}$$

(A.28)

where $\lambda _{kj}^{(s)}$, $\mu _{kj}^{(s)}$ and $\Sigma _{kj}^{(s)}$ are the sth updates of the emission parameters in the M-step of the sth iteration of the EM algorithm.

1.2 The M-step of the EM algorithm

In the M-step of the EM algorithm, the initial probabilities are updated as follows

$$\begin{aligned}\pi _j^{(s+1)} = P(S_0 = j|X_{0:{\tau -1}} = x_{0:{\tau -1}};\theta ^{(s)}) = L_j(0), \end{aligned}$$

For a semi-Markovian state i, the transition probabilities are updated as follows

$$\begin{aligned} p_{ij}^{(s+1)}= & {} \frac{\sum _{t=0}^{\tau -2}G_j(t+1)p_{ij}^{(s)}F_i(t)}{\sum _{t=0}^{\tau -2}L1_i(t)} \end{aligned}$$

(A.29)

and for a Markovian state i

$$\begin{aligned} {\tilde{p}}_{ij}^{(s+1)}= & {} \frac{\sum _{t=0}^{\tau -2}{\tilde{G}}_j(t+1){\tilde{p}}_{ij}^{(s)}{\tilde{F}}_i(t)}{\sum _{t=0}^{\tau -2}L_i(t)} \end{aligned}$$

(A.30)

If we consider the mixture of multivariate normals, with the probability density function (A.27), as the emission distribution, then its parameters are updated as follows

$$\begin{aligned}&\lambda _{kj}^{(s+1)} = \frac{\sum _{t=0}^{\tau -1}\gamma _{kj}^{(s)}(t) L_j^{(s)}(t)}{\sum _{m=1}^{K_j}\sum _{t=0}^{\tau -1}\gamma _{mj}^{(s)}(t) L_j^{(s)}(t)} = \frac{\sum _{t=0}^{\tau -1}\gamma _{kj}^{(s)}(t) L_j^{(s)}(t)}{\sum _{t=0}^{\tau -1}L_j^{(s)}(t)},\\&\mu _{kj}^{(s+1)} = \frac{\sum _{t=0}^{\tau -1}\gamma _{kj}^{(s)}(t) L_j^{(s)}(t)x_t}{\sum _{t=0}^{\tau -1}\gamma _{kj}^{(s)}(t) L_j^{(s)}(t)},\\&\Sigma _{kj}^{(s+1)} = \frac{\sum _{t=0}^{\tau -1}\gamma _{kj}^{(s)}(t) L_j^{(s)}(t)(x_t-\mu _{kj}^{(s+1)})(x_t-\mu _{kj}^{(s+1)})^T}{\sum _{t=0}^{\tau -1}\gamma _{kj}^{(s)}(t) L_j^{(s)}(t)} \end{aligned}$$

Also, the parameters of the sojourn time distribution are updated by maximization of the following quasi-log-likelihood function

$$\begin{aligned}{\tilde{Q}}_d(\{d_j(u)\}|\theta ^{(s)}) = \sum _{u=1}^{M_j}{\tilde{\eta }}_{j,u}^{(s)}\log d_j(u), \end{aligned}$$

where

$$\begin{aligned}&{\tilde{\eta }}_{j,u}=\sum _{t=0}^{\tau -2} G_j(t+1,u)\sum _{i\ne j} p_{ij} F_i(t) + A(u) d_j(u)\pi _j \prod _{v=1}^u \frac{f_j(x_{u_{\tau } - v)}}{N_{u_{\tau }-v}},\\&A(u) = \left\{ \begin{array}{lr}\frac{L_{1j}(u-1)}{F_j(u-1)},&{} u\le \tau -1\\ 1,&{} u>\tau -1,\end{array}\right. \end{aligned}$$

and $u_{\tau } = \min (u,\tau )$.

1.3 Viterbi algorithm and smoothing for the HHSMM model

The Viterbi algorithm (Viterbi 1967) is an algorithm to obtain the most likely state sequence, given the observations and the estimated parameters of the model.

For a semi-Markovian state j, and for $t=0,\ldots ,\tau -2$, the probability of the most probable state sequence is obtained by the Viterbi recursion as follows

$$\begin{aligned} \alpha _j(t)= & {} \max _{s_0,\ldots ,s_{\tau -1}}P(S_{t+1}\ne j,S_t=j,S_{0:{t-1}}=s_{0:{t-1}},X_{0:t}=x_{0:t})\nonumber \\= & {} f_j(x_t)\max \Bigg [\max _{1\le u\le t}\Bigg [\Bigg \{\prod _{\nu =1}^{u-1}f_j(x_{t-\nu })\Bigg \}d_j(u)\max _{i\ne j}\{p_{ij}\alpha _i(t-u)\}\Bigg ],\nonumber \\&\times \Bigg \{\prod _{\nu =1}^{t}f_j(x_{t-\nu })\Bigg \}d_j(t+1)\pi _j\Bigg ], \end{aligned}$$

(A.31)

and

$$\begin{aligned} \alpha _j(\tau -1)= & {} \max _{s_0,\ldots ,s_{\tau -2}} P(S_{\tau -1} =j,S_{0:{\tau -2}} =s_{0:{\tau -2}},X_{0:{\tau -1}} =x_{0:{\tau -1}})\nonumber \\= & {} f_j(x_{\tau -1})\max \Bigg [\max _{1\le u\le \tau -1}\Bigg [\Bigg \{\prod _{\nu =1}^{u-1}f_j(x_{\tau -1-\nu })\Bigg \}D_j(u)\max _{i\ne j}\{p_{ij}\alpha _i(\tau -1-u)\}\Bigg ],\nonumber \\&\times \Bigg \{\prod _{\nu }^{\tau -1}f_j(x_{\tau -1-\nu })\Bigg \}D_j(\tau )\pi _j\Bigg ], \end{aligned}$$

(A.32)

For a Markovian state j, the Viterbi recursion is initialized by

$$\begin{aligned}{\tilde{\alpha }}_j(0) = P(S_0 = j,X_0 = x_0) = f_j(x_0)\pi _j\end{aligned}$$

and for $t = 1,\ldots .\tau -1$,

$$\begin{aligned} {\tilde{\alpha }}_j(t)= & \max _{s_0,\ldots ,s_{t-1}}P(S_t = j,S_0^{t-1} = s_0^{t-1},X_{0:t} = x_{0:t})\end{aligned}$$

(A.33)

$$\begin{aligned}= & f_j(x_t)\max _i\{{\tilde{p}}_{ij}{\tilde{\alpha }}_i(t-1)\}. \end{aligned}$$

(A.34)

After obtaining the probability of the most probable state sequence, the current most likely state is obtained as ${\hat{s}}_t^* =\arg \max _{1\le j\le J} \alpha _j(t)$.

Another approach for obtaining the state sequence is the smoothing method, which uses the backward probabilities $L_j(t)$ instead of $\alpha _j(t)$.