Identifying patterns in financial markets: extending the statistical jump model for regime identification

Abstract

Regime-driven models are popular for addressing temporal patterns in both financial market performance and underlying stylized factors, wherein a regime describes periods with relatively homogeneous behavior. Recently, statistical jump models have been proposed to learn regimes with high persistence, based on clustering temporal features while explicitly penalizing jumps across regimes. In this article, we extend the jump model by generalizing the discrete hidden state variable into a probability vector over all regimes. This allows us to estimate the probability of being in each regime, providing valuable information for downstream tasks such as regime-aware portfolio models and risk management. Our model’s smooth transition from one regime to another enhances robustness over the original discrete model. We provide a probabilistic interpretation of our continuous model and demonstrate its advantages through simulations and real-world data experiments. The interpretation motivates a novel penalty term, called mode loss, which pushes the probability estimates to the vertices of the probability simplex thereby improving the model’s ability to identify regimes. We demonstrate through a series of empirical and real world tests that the approach outperforms traditional regime-switching models. This outperformance is pronounced when the regimes are imbalanced and historical data is limited, both common in financial markets.

A Appendix: Proofs

Proof of Proposition 1

From the probabilistic assumptions, the joint likelihood is

$$\begin{aligned} p({\varvec{Y}},{\varvec{S}}|\Theta )&=p({\varvec{Y}}|{\varvec{S}}, \Theta )p({\varvec{S}}) \end{aligned}$$

(33)

$$\begin{aligned}&=\prod _{t=0}^{T-1}p({\varvec{y}}_t|\Theta , {\varvec{s}}_t)\times \pi ({\varvec{s}}_0)\times \prod _{t=1}^{T-1}K\left( {\varvec{s}}_{t-1}, {\varvec{s}}_t\right) \,. \end{aligned}$$

(34)

Thus

$$\begin{aligned} \log p({\varvec{Y}},{\varvec{S}}|\Theta )=\sum _{t=0}^{T-1}\log p({\varvec{y}}_t|\Theta , {\varvec{s}}_t)+\log \pi ({\varvec{s}}_0)+\sum _{t=1}^{T-1}\log K\left( {\varvec{s}}_{t-1}, {\varvec{s}}_t\right) \,. \end{aligned}$$

(35)

For the first term, by Jensen’s inequality we have

$$\begin{aligned} \log p({\varvec{y}}_t|\Theta , {\varvec{s}}_t)&=\log \left( \sum _{k=0}^{K-1}s_{tk}p({\varvec{y}}_t|{\varvec{\theta }}_k)\right) \end{aligned}$$

(36)

$$\begin{aligned}&\ge \sum _{k=0}^{K-1}s_{tk}\log p({\varvec{y}}_t|{\varvec{\theta }}_k) \,. \end{aligned}$$

(37)

Inserting (37) into (35), we obtain

$$\begin{aligned} \log p({\varvec{Y}},{\varvec{S}}|\Theta ) \ge&\sum _{t, k}s_{tk}\log p({\varvec{y}}_t|{\varvec{\theta }}_k) +\log \pi ({\varvec{s}}_0)+\sum _{t=1}^{T-1}\log K\left( {\varvec{s}}_{t-1}, {\varvec{s}}_t\right) \,. \end{aligned}$$

(38)

From equation (17), it follows that the above right hand side is precisely the negation of the objective function J. Therefore minimizing J is equivalent to maximizing a lower bound of the joint log-likelihood function. $\square$

Proof of Proposition 2

From the previous proof, we know that

$$\begin{aligned} \log p({\varvec{Y}},{\varvec{S}}|\Theta )&=\sum _{t=0}^{T-1}\log p({\varvec{y}}_t|\Theta , {\varvec{s}}_t)+\log p({\varvec{S}}) \end{aligned}$$

(39)

$$\begin{aligned}&\ge \sum _{t,k}s_{tk}\log p({\varvec{y}}_t|{\varvec{\theta }}_k)+\log p({\varvec{S}}) \end{aligned}$$

(40)

Inserting (18) into (40), we obtain

$$\begin{aligned} \sum _{t, k}s_{t,k}\left( -l({\varvec{y}}_t,{\varvec{\theta }}_k)-\log \nu \right) -{{\mathcal {L}}}({\varvec{S}})-\log \eta =&-\left( \sum _{t, k}s_{t,k}l({\varvec{y}}_t,{\varvec{\theta }}_k) + {{\mathcal {L}}}({\varvec{S}})\right) +\text {const} \end{aligned}$$

(41)

$$\begin{aligned} =&-J+\text {const} \,. \end{aligned}$$

(42)

Thus, minimizing J is equivalent to maximizing a lower bound of the log-likelihood $p({\varvec{Y}},{\varvec{S}}|\Theta )$. $\square$