Rate of Mutual Information Between Coarse-Grained Non-Markovian Variables

Abstract

The problem of calculating the rate of mutual information between two coarse-grained variables that together specify a continuous time Markov process is addressed. As a main obstacle, the coarse-grained variables are in general non-Markovian, therefore, an expression for their Shannon entropy rates in terms of the stationary probability distribution is not known. A numerical method to estimate the Shannon entropy rate of continuous time hidden-Markov processes from a single time series is developed. With this method the rate of mutual information can be determined numerically. Moreover, an analytical upper bound on the rate of mutual information is calculated for a class of Markov processes for which the transition rates have a bipartite character. Our general results are illustrated with explicit calculations for four-state networks.

Appendix: Detailed Derivation of the Analytical Upper Bound

The first upper bound H(Y ₂|Y ₁) can be easily calculated by using the conditional probability

$$ P(Y_2|Y_1)= \frac{\sum_{X_1}P(Y_2,Y_1,X_1)}{P(Y_1)}= \frac{\sum_\alpha P_i^{\alpha}w^{\alpha}_{ij}\tau}{P_i}, $$

(68)

where Y ₂≠Y ₁. We here performed the substitutions X ₁→α, Y ₁→i, and Y ₂→j. Using this formula in (22) we obtain

$$\begin{aligned} H(Y_2|Y_1)=-\sum_{i,\alpha}P_i^\alpha \sum_{j\neq i}w_{ij}^\alpha \biggl(\ln \tau+\ln\frac{\sum_\beta P_i^\beta w_{ij}^\beta}{P_i}-1 \biggr)+\textrm{O}(\tau). \end{aligned}$$

(69)

Moreover, H(Y _N+1|Y _N ,…,Y ₁) up to order τ is given by the above formula for any finite N. In order to demonstrate this we first rewrite (22) as

$$\begin{aligned} & H(Y_{N+1}|Y_{N},\ldots,Y_1) \\ &\quad = -\frac{1}{\tau}\sum_{Y_{N+1}\neq Y_N}\sum _{Y_N\ldots Y_1}P(Y_{N+1},Y_N,\ldots,Y_1) \ln P(Y_{N+1}|Y_N,\ldots,Y_1) \\ &\qquad{} -\frac{1}{\tau}\sum_{Y_N\ldots Y_1}P(Y_{N},Y_N, \ldots,Y_1)\ln P(Y_{N}|Y_N, \ldots,Y_1), \end{aligned}$$

(70)

where P(Y _N ,Y _N ,…,Y ₁) denotes the probability of having a sequence for which Y _N+1=Y _N . For Y _N+1≠Y _N , the expression of the conditional probability P(Y _N+1|Y _N ,…,Y ₁) has at least one transition probability term of order τ. Therefore, as P(Y _N+1|Y _N ,…,Y ₁) is at least a term of order τ, it is convenient to further rewrite the above expression as

$$\begin{aligned} & H(Y_{N+1}|Y_{N},\ldots,Y_1) \\ &\quad =- \frac{1}{\tau}\sum_{Y_{N+1}\neq Y_N}\sum _{Y_N}P(Y_{N+1},Y_N)\ln \tau \\ &\qquad{} -\frac{1}{\tau}\sum_{Y_{N+1}\neq Y_N}\sum _{Y_N\ldots Y_1}P(Y_{N+1},Y_N,\ldots,Y_1) \ln \frac{P(Y_{N+1}|Y_N,\ldots,Y_1)}{\tau} \\ &\qquad{} -\frac{1}{\tau}\sum_{Y_N\ldots Y_1}P(Y_{N},Y_N, \ldots,Y_1)\ln P(Y_{N}|Y_N, \ldots,Y_1), \end{aligned}$$

(71)

where in the first line we summed over the variables Y ₁,…,Y _N−1. The three following relations are important for the subsequent derivation. First, for Y _N+1≠Y _N ,

$$ P(Y_{N+1}, Y_N,\ldots,Y_1) = \left \{ \begin{array}{l@{\quad}l} P(Y_{N+1}, Y_N) + \textrm{O}(\tau^2) & \mathrm{if}\ Y_N= Y_{N-1}=\cdots=Y_1\\ \textrm{O}(\tau^2) & \mathrm{otherwise}. \end{array} \right . $$

(72)

Moreover,

$$ P(Y_N,\ldots,Y_1) = \left \{ \begin{array}{l@{\quad}l} P(Y_N) + \textrm{O}(\tau) & \mathrm{if}\ Y_N= Y_{N-1}=\cdots=Y_1\\ A\tau^\eta+ \textrm{O}(\tau^{\eta+1}) & \mathrm{otherwise}, \end{array} \right . $$

(73)

where η≥1 is an integer and A is a constant independent of τ. Finally, the conditional probability distribution fulfills

$$ P(Y_{N+1}| Y_N,\ldots,Y_1) = \left \{ \begin{array}{l@{\quad}l} P(Y_{N+1}|Y_N) + \textrm{O}(\tau^2) & \mathrm{if}\ Y_N= Y_{N-1}=\cdots=Y_1\\ B\tau^\nu+ \textrm{O}(\tau^{\nu+1})& \mathrm{otherwise}, \end{array} \right . $$

(74)

where ν≥1 is an integer and B is a constant independent of τ. With these three relations, the term in the second line in Eq. (71) becomes

$$\begin{aligned} & \frac{1}{\tau}\sum_{Y_{N+1}\neq Y_N}\sum _{Y_N\ldots Y_1}P(Y_{N+1},Y_N,\ldots,Y_1) \ln \frac{P(Y_{N+1}|Y_N,\ldots,Y_1)}{\tau} \\ &\quad = \frac{1}{\tau}\sum_{Y_{N+1}\neq Y_N}\sum _{Y_N} P(Y_{N+1},Y_N)\ln \frac{P(Y_{N+1}|Y_N)}{\tau}+\textrm{O}(\tau), \end{aligned}$$

(75)

where we used τ ^ν+η−1lnτ ^ν−1∈O(τ). For the term in the third line in Eq. (71) we need the relations,

$$ P(Y_{N}|Y_N,\ldots,Y_1)= 1-\sum _{Y_{N+1}\neq Y_{N}} P(Y_{N+1}|Y_N, \ldots,Y_1) $$

(76)

and

$$ P(Y_{N},Y_N,\ldots,Y_1)= P(Y_N, \ldots,Y_1) \biggl(1-\sum_{Y_{N+1}\neq Y_{N}} P(Y_{N+1}|Y_N,\ldots,Y_1) \biggr) $$

(77)

which lead to

$$ \frac{1}{\tau}\sum_{Y_N\ldots Y_1}P(Y_{N},Y_N, \ldots,Y_1)\ln P(Y_{N}|Y_N, \ldots,Y_1)= \frac{1}{\tau}\sum_{Y_{N+1}\neq Y_N} P(Y_{N+1},Y_{N})+\textrm{O}(\tau). $$

(78)

Inserting (75) and (78) in (71) we obtain

$$ H(Y_{N+1}|Y_{N},\ldots,Y_1)= H(Y_{N+1}|Y_{N})+ \textrm{O}(\tau). $$

(79)

Therefore, since the Y process is stationary, from (69), we obtain for any finite N

$$\begin{aligned} H(Y_{N+1}|Y_{N},\ldots,Y_1)=-\sum _{i,\alpha}P_i^\alpha\sum _{j\neq i}w_{ij}^\alpha \biggl(\ln \tau+\ln \frac{\sum_\beta P_i^\beta w_{ij}^\beta}{P_i}-1 \biggr)+\textrm{O}(\tau). \end{aligned}$$

(80)

Applying the same method to the X process we get,

$$\begin{aligned} H(X_{N+1}|X_{N},\ldots,X_1)=-\sum _{i,\alpha}P_i^\alpha\sum _{\beta\neq \alpha}w_{i}^{\alpha\beta} \biggl(\ln \tau+\ln \frac{\sum_j P_j^\alpha w_{j}^{\alpha\beta}}{P^\alpha}-1 \biggr)+\textrm{O}(\tau). \end{aligned}$$

(81)