A Graph-Algorithmic Approach for the Study of Metastability in Markov Chains

Abstract

Large continuous-time Markov chains with exponentially small transition rates arise in modeling complex systems in physics, chemistry, and biology. We propose a constructive graph-algorithmic approach to determine the sequence of critical timescales at which the qualitative behavior of a given Markov chain changes, and give an effective description of the dynamics on each of them. This approach is valid for both time-reversible and time-irreversible Markov processes, with or without symmetry. Central to this approach are two graph algorithms, Algorithm 1 and Algorithm 2, for obtaining the sequences of the critical timescales and the hierarchies of Typical Transition Graphs or T-graphs indicating the most likely transitions in the system without and with symmetry, respectively. The sequence of critical timescales includes the subsequence of the reciprocals of the real parts of eigenvalues. Under a certain assumption, we prove sharp asymptotic estimates for eigenvalues (including pre-factors) and show how one can extract them from the output of Algorithm 1. We discuss the relationship between Algorithms 1 and 2 and explain how one needs to interpret the output of Algorithm 1 if it is applied in the case with symmetry instead of Algorithm 2. Finally, we analyze an example motivated by R. D. Astumian’s model of the dynamics of kinesin, a molecular motor, by means of Algorithm 2.

Appendices

Appendix 1: Proof of Theorem 2.4

The following notations will be used throughout the proof.

• ${\mathcal {G}}(n-l)$ is the set of all W-graphs with $n-l$ sinks and l arcs for the graph $G({\mathcal {S}},{\mathcal {A}},{\mathcal {U}})$. The set of subgraphs of G with exactly l arcs emanated from distinct vertices will be denoted by ${\mathcal {H}}(l)$. Note that graphs in ${\mathcal {H}}(l)$ might contain directed cycles. Therefore, ${\mathcal {G}}(n-l) \subsetneq {\mathcal {H}}(l)$.

• ${\mathcal {S}}^l$ is the set of all ordered selections of l vertices out of n (in the combinatorial sense):

  $$\begin{aligned} {\mathcal {S}}^l: = \{ i_1 i_2 \cdots i_l ~|~ i_m \in \{ 1, 2 , \cdots , n-1\}, 1 \le m \le l , {\text { distinct}} \}. \end{aligned}$$

  Note that $|{\mathcal {S}}^l| = n(n-1)\ldots (n-l+1)$.

• ${\mathcal {O}}^l$ is the set of combinations of l vertices out of n (in the combinatorial sense). Each combination in ${\mathcal {O}}^l$ is ordered so that $i_1<i_2<\ldots <i_l$, i.e.,

  $$\begin{aligned} {\mathcal {O}}^l := \{ i_1 i_2 \cdots i_l \in {\mathcal {S}}^l ~|~ i_1< i_2< \cdots < i_l \}. \end{aligned}$$

  Note that $|{\mathcal {O}}^l| = \left( \begin{array}{c}n\\ l\end{array}\right)$.

• Every sequence $j_1,\ldots j_l$ in ${\mathcal {S}}^l$ can be permuted to an ordered sequence $\{i_1,\ldots , i_l\}$ in ${\mathcal {O}}^l$. This defines the permutation map $\sigma$: $\sigma (j_1,\ldots ,j_l) = (i_1,\ldots ,i_l)$. Note that the map $\sigma$ is onto but not one to one.

• We will call sequences in ${\mathcal {S}}^l$ equivalent if and only if they are mapped to the same sequence in ${\mathcal {O}}^l$. Therefore, ${\mathcal {O}}^l={\mathcal {S}}^l/\sim$, meaning that ${\mathcal {O}}^l$ is the set of equivalence classes in ${\mathcal {S}}^l$.

• For any $i_1 \cdots i_l \in {\mathcal {O}}^l$, we will denote by $L_{ i_1 \cdots i_l }$ the $l\times l$ submatrix of L consisting of the intersection of $i_1, \cdots , i_l$ rows and columns of L.

Proof

The proof of Theorem 2.4 consists of the following three steps.

Step 1: :

The characteristic polynomial of the generator matrix L is

$$\begin{aligned} P_L(t) := {\text {det}} ( t I - L) = t^n + \sum _{l = 1}^{n-1} C_l t^{n-l} \end{aligned}$$

whose coefficients $C_l$ are given by

$$\begin{aligned} C_l&= \sum _{i_1 \cdots i_l \in {\mathcal {O}}^l} {\text {det}} ( - L_{i_1 \cdots i_l} )\nonumber \\&= (-1)^l \sum _{i_1 \cdots i_l \in {\mathcal {O}}^l} ~\sum _{ \begin{array}{c} j_1 \cdots j_l \in {\mathcal {S}}^l\\ j_1 \cdots j_l\sim i_1 \cdots i_l \end{array}} (-1)^{| \sigma (j_1, \cdots , j_l) |} L_{i_1 j_1} \cdots L_{i_l j_l}, \end{aligned}$$

(23)

where $| \sigma (j_1, \cdots , j_l) |$ is the number of inversions in $\sigma (j_1, \cdots , j_l)$.

Step 2: :

Using the zero row sum property of the generator matrix L, Eq. (23) can be further simplified resulting at the following more compact expression:

$$\begin{aligned} C_l = \sum _{g \in {\mathcal {G}}(n-l)} \Pi (g), \quad \mathrm{where}\quad \Pi (g) = \prod _{(i\rightarrow j)\in g}L_{ij}. \end{aligned}$$

(24)

The key idea of the derivation of Eq. (24) is to show that

$$\begin{aligned} C_l = (-1)^l \sum _{g \in {\mathcal {H}}(l) } a_g \Pi (g), \quad \mathrm{where}\quad a_g = {\left\{ \begin{array}{ll} (-1)^l, &{} {\text { if }} g \in {\mathcal {G}}(n-l),\\ 0, &{} {\text { if }} g \in {\mathcal {H}}(l) {\setminus } {\mathcal {G}}(n-l). \end{array}\right. } \end{aligned}$$

(25)

Step 3: :

Comparing the coefficient of the characteristic polynomial

$$\begin{aligned} P_L(t) = {\text {det}} ( t I - L ) = t^n + \sum _{l = 1}^{n-1} C_l t^{n-l} = t (t + \lambda _1) \cdots (t + \lambda _{n-1}), \end{aligned}$$

we obtain the following estimates for eigenvalues:

$$\begin{aligned} \lambda _m = \frac{ \Pi (g^{*}_{m}) }{ \Pi (g^{*}_{m+1}) } (1 + o(1) ). \end{aligned}$$

(26)

Eq. (26) is equivalent to Eq. (7).

Now we elaborate each step.

Step 1:

Consider the following polynomial in n variables $t_1, t_2, \cdots , t_n$:

$$\begin{aligned} P_L( t_1, t_2, \cdots , t_n)&:= {\text {det}} ( {\text {diag}} \{ t_1, t_2, \cdots , t_n\} - L)\nonumber \\&= t_1 \cdots t_n + \sum _{l = 1}^{n - 1} ~\sum _{i_1 \cdots i_l \in {\mathcal {O}}^l} C_{i_1 \cdots i_l} t_{i_{l+1}} \cdots t_{i_n} \end{aligned}$$

(27)

where $\{ i_{l+1}, \cdots , i_n \} = \{ 1, \cdots , n\} {\setminus } \{ i_1, \cdots , i_l \}$. Replacing all $t_i$ by t, we recover the characteristic polynomial $P_L(t)$ where

$$\begin{aligned} C_l = \sum _{i_1 \cdots i_l \in {\mathcal {O}}^l} C_{ i_1 \cdots i_{l} }. \end{aligned}$$

(28)

The term $C_{i_1 \cdots i_l} t_{i_{l+1}} \cdots t_{i_n}$ in Eq. (27) is obtained by picking $t_{i_{l+1}}$, ..., $t_{i_n}$ from the diagonal entries in rows of the matrix ${\text {diag}} \{ t_1, t_2, \cdots , t_n\} - L$ and multiplying them by the determinant of $-{L_{i_1\ldots i_l}}$. Hence,

$$\begin{aligned} C_{ i_1 \cdots i_l } = {\text {det}} \left( -L_{ i_1 \cdots i_l } \right) . \end{aligned}$$

(29)

Combining Eqs. (28) and (29) and applying the Leibniz formula for determinants, we obtain Eq. (23).

Step 2:

Consider the product terms $L_{i_1 j_1} \cdots L_{i_l j_l}$, $i_1 \cdots i_l \in {\mathcal {O}}^l, i_1 \cdots i_l \sim j_1 \cdots j_l$ in Eq. (23). Suppose the sequences $i_1 \cdots i_l$ and $j_1 \cdots j_l$ agree on exactly s entries and differ at $l-s$ entries, i.e.,

$$\begin{aligned} i_{m_1} =&j_{m_1}, \cdots , i_{m_s} = j_{m_s}, \end{aligned}$$

(30)

$$\begin{aligned} i_{m_{s+1}} \ne&j_{m_{s+1}}, \cdots , i_{m_l} \ne j_{m_l}, {\text { where }} i_{m_{s+1}} \cdots i_{m_l} \sim j_{m_{s+1}} \cdots j_{m_l} \in {\mathcal {S}}^{l-s}. \end{aligned}$$

(31)

Note that s can be any number between 0 and l except for $l-1$. Using the zero sum property of L, we obtain

$$\begin{aligned}&L_{i_1 j_1} \cdots L_{i_l j_l} = L_{ i_{m_1} i_{m_1} } \cdots L_{ i_{m_s} i_{m_s} } L_{ i_{m_{s+1}} j_{m_{s+1}} } \cdots L_{ i_{m_l} j_{m_l} }\nonumber \\&= \left( - \sum _{d_1 \ne i_{m_1} } L_{i_{m_1} d_1} \right) \cdots \left( - \sum _{d_s \ne i_{m_s} } L_{i_{m_s} d_s} \right) L_{ i_{m_{s+1}} j_{m_{s+1}} } \cdots L_{ i_{m_l} j_{m_l} } \nonumber \\&= (-1)^s \sum _{d_1 \ne i_{m_1}, \cdots , d_s \ne i_{m_s} } L_{ i_{m_1} d_1 } \cdots L_{ i_{m_s} d_s } L_{ i_{m_{s+1}} j_{m_{s+1}} } \cdots L_{ i_{m_l} j_{m_l} }. \end{aligned}$$

(32)

It is helpful to consider the collection of graphs g with n vertices and sets of arcs

$$\begin{aligned} \{ (i_{m_1} \rightarrow d_1) , \cdots , (i_{m_s} \rightarrow d_s), (i_{m_{s+1} } \rightarrow j_{m_{s+1}}), \cdots , ( i_{m_l} \rightarrow j_{m_l} ) \} \end{aligned}$$

corresponding to the products

$$\begin{aligned} L_{ i_{m_1} d_1 } \cdots L_{ i_{m_s} d_s } L_{ i_{m_{s+1}} j_{m_{s+1}} } \cdots L_{ i_{m_l} j_{m_l} } \end{aligned}$$

in Eq. (32). Each of the vertices $i_1,\ldots ,i_l$ of g has exactly one outgoing arc, while the other vertices have no outgoing arcs. Hence, the graphs g belong to the set of graphs ${\mathcal {H}}(l)$. If $s \le l-2$, each of the vertices $\{ i_{m_{s+1}}, \cdots , i_{m_l} \}$ of the graph g has exactly one outgoing arc heading to $\{ i_{m_{s+1}}, \cdots , i_{m_l} \}$, and exactly one incoming arc from $\{ i_{m_{s+1}}, \cdots , i_{m_l} \}$. Therefore, the arcs $\{(i_{m_{s+1} } \rightarrow j_{m_{s+1}}), \cdots , ( i_{m_l} \rightarrow j_{m_l} ) \}$ necessarily form cycles in g. The arcs $\{ (i_{m_1} \rightarrow d_1) , \cdots , (i_{m_s} \rightarrow d_s)\}$ do not necessarily form cycles. Therefore, Eq. (23) can be rewritten using Eq. (32) and the graphs ${\mathcal {H}}(l)$ as

$$\begin{aligned} C_l&= \sum _{i_1 \cdots i_l \in {\mathcal {O}}^l} {\text {det}} ( - L_{i_1 \cdots i_l} ) = (-1)^l \sum _{i_1 \cdots i_l \in {\mathcal {O}}^l} ~\sum _{ \begin{array}{c} j_1 \cdots j_l \in {\mathcal {S}}^l\\ j_1 \cdots j_l\sim i_1 \cdots i_l \end{array}} (-1)^{| \sigma (j_1, \cdots , j_l) |} L_{i_1 j_1} \cdots L_{i_l j_l} \nonumber \\&= (-1)^l \sum _{i_1 \cdots i_l \in {\mathcal {O}}^l} ~\sum _{ \begin{array}{c} j_1 \cdots j_l \in {\mathcal {S}}^l\\ j_1 \cdots j_l\sim i_1 \cdots i_l \end{array}} (-1)^{| \sigma (j_1, \cdots , j_l) |} (-1)^s L_{ i_{m_{s+1}} j_{m_{s+1}} } \nonumber \\&\qquad \cdots L_{ i_{m_l} j_{m_l} } \sum _{ \begin{array}{c} d_1\ne i_{m_1}\\ \cdots \\ d_s \ne i_{m_s} \end{array}} L_{ i_{m_1} d_1 } \cdots L_{ i_{m_s} d_s } \nonumber \\&=(-1)^l \sum _{g\in {\mathcal {H}}(l)}a_g\Pi (g), \end{aligned}$$

(33)

where the factors $a_g$ will be determined below.

First we assume that the graph g with the set of arcs $\{ (i_1 \rightarrow j_1), \cdots , (i_l \rightarrow j_l) \}$ contains no cycles, i.e., $g \in {\mathcal {G}} (n-l)\subset {\mathcal {H}}(l)$. This can happen only if $s=l$ in Eq. (33), i.e., $i_1=j_1$, ...,$i_l=j_l$. Therefore, $(-1)^{| \sigma (j_1, \cdots , j_l) |} =(-1)^0 = 1$, and

$$\begin{aligned} L_{i_1 i_1} \cdots L_{i_l i_l}= & {} \left( - \sum _{d_1 \ne i_1} L_{i_1 d_1} \right) \cdots \left( - \sum _{d_l \ne i_l} L_{i_l d_l} \right) \\= & {} (-1)^l \sum _{d_1 \ne i_1, \cdots , d_l \ne i_l} L_{i_1 d_1} \cdots L_{i_l d_l}. \end{aligned}$$

Hence, the product $L_{i_1 d_1} \cdots L_{i_l d_l}$ corresponding to the graph g enters Eq. (33) only once. Therefore, $a_{g} = (-1)^l$ for all $g \in {\mathcal {G}} (n-l)$.

Now we assume that the set of arcs $\{ (i_1 \rightarrow x_1), \cdots , (i_l \rightarrow x_l) \}$ of g contains $N\ge 1$ cycles, i.e., $g \in {\mathcal {H}}(l) {\setminus } {\mathcal {G}}(n-l)$. In this case, the product $L_{i_1 x_1} \cdots L_{i_l x_l}$, $i_p\ne x_p$, $1\le p\le l$, enters Eq. (33) $2^N$ times either with the plus or minus sign. The number $2^N$ comes from the fact that each cycle in g can be formed in two ways: (i) by arcs corresponding to off-diagonal factors $L_{i_pj_p}$, $i_p\ne j_p$, in Eq. (33), or (ii) by arcs originating from the replacement of diagonal factors $L_{i_pi_p}$ in Eq. (33) with $-\sum _{d_p\ne i_p}L_{i_pd_p}$. We will prove that $L_{i_1 x_1} \cdots L_{i_l x_l}$, $i_p\ne x_p$, $1\le p\le l$, enters Eq. (33) with sign plus the same number of times as it does with sign minus. This will imply that $a_g=0$. To do so, we show that for each entry of $L_{i_1 x_1} \cdots L_{i_l x_l}$, one can uniquely define another entry with an opposite sign. Let c be a cycle in g. Consider two terms in Eq. (33) containing the product $L_{i_1x_1}\ldots L_{i_lx_l}$ that correspond to possibilities (i) and (ii) for the origin of a selected cycle c in g, while all other factors corresponding to the arcs not in c originate in the same way. Let $\sigma _1(j^1_1,\ldots j^1_l)$ and $\sigma _2(j^2_1,\ldots ,j^2_l)$ be the permutations in Eq. (33) corresponding to (i) and (ii), respectively, and $s_1$ and $s_2$ be the corresponding numbers of fixed entries in $\sigma _1$ and $\sigma _2$, respectively. If the cycle c has length |c|, then

$$\begin{aligned} (-1)^{|\sigma _1|} = (-1)^{|\sigma _2|}(-1)^{|c|-1},\quad (-1)^{s_1} = (-1)^{s_2+|c|}. \end{aligned}$$

(34)

Here we have used the known combinatorial fact that the parity of a permutation consisting of cycles $c_1$, ..., $c_k$ is $(-1)^{\sum _{j=1}^k|c_j|-1}$. Therefore, the signs $(-1)^l(-1)^{|\sigma _1|}(-1)^{s_1}$ and $(-1)^l(-1)^{|\sigma _2|}(-1)^{s_2}$ preceding the corresponding products in Eq. (33) are opposite. This implies that all products corresponding to any graph $g\in {\mathcal {H}}(l) {\setminus } {\mathcal {G}}(n-l)$ cancel out, i.e., $a_g=0$.

Therefore, $C_l = \sum _{g\in {\mathcal {G}}(n-l)}\Pi (g)$, i.e., Eq. (24) holds. An example illustrating this cancellation is given at the end of “Appendix 1.”

Step 3:

Let us write the characteristic polynomial $P_L(t)$ in the form

$$\begin{aligned} P_L(t) = {\text {det}} ( tI -L ) = t ( t + \lambda _1 ) \cdots ( t + \lambda _{n-1} ), \end{aligned}$$

(35)

where $|\lambda _1|<|\lambda _2|<\ldots <|\lambda _{n-1}|$. A simple calculation gives

$$\begin{aligned} C_l = \sum _{i_1 \cdots i_r \in {\mathcal {O}}^l } \lambda _{i_1} \cdots \lambda _{i_l}. \end{aligned}$$

(36)

Comparing Eqs. (24) and (36), we obtain

$$\begin{aligned} \frac{C_l}{ C_{l-1}} =\frac{\sum _{i_1 \cdots i_l \in {\mathcal {O}}^l } \lambda _{i_1} \ldots \lambda _{i_l}}{\sum _{i_1 \cdots i_{l-1} \in {\mathcal {O}}^{l-1} } \lambda _{i_1} \ldots \lambda _{i_{l-1}}} =\frac{\sum _{g\in {\mathcal {G}}(n-l)}\Pi (g)}{\sum _{g\in {\mathcal {G}}(n-l+1)}\Pi (g)},\quad l\le l\le n-1. \end{aligned}$$

(37)

Since $L_{ij}$ are of the form $\kappa _{ij}\exp (-U_{ij}/\varepsilon )$, the sums in the enumerators and the denominators are dominated by their largest summands in Eq. (37). According to Assumption 3, all optimal W-graphs are unique. Therefore,

$$\begin{aligned} \frac{C_l}{ C_{l-1}} = \lambda _{n-l}(1+o(1)) = \frac{\Pi (g^{*}_{n-l}) }{\Pi (g^{*}_{n-l+1}) } (1+o(1)), \end{aligned}$$

(38)

and Eq. (26) immediately follows.

Example 9.1

Let us illustrate the cancellation of terms in Eq. (33). Let $n=4$, $l=3$, and $\{i_1,i_2,i_3\}=\{1,2,3\}$. Then the inner sum in Eq. (33) becomes

$$\begin{aligned} \begin{array}{lll} &{} &{}\sum \limits _{(j_1,j_2,j_3)\sim (1,2,3)}(-1)^{|\sigma (j_1,j_2,j_3)|}L_{1j_1}L_{2j_2}L_{3j_3} = \\ &{}(j_1,j_2,j_3) = (1,2,3):~\sigma = 0,~s = 3:\qquad &{}-(L_{12}+L_{13}+L_{14})(L_{21}+L_{23}+L_{24})\\ &{} &{} \qquad (L_{31}+L_{32}+L_{34})\\ &{}(j_1,j_2,j_3) = (1,3,2):~\sigma =1,~s = 1:\qquad &{}+(L_{12}+L_{13}+L_{14})L_{23}L_{32} \\ &{}(j_1,j_2,j_3) = (2,1,3):~\sigma =1,~s = 1:\qquad &{}+L_{12}L_{21}(L_{31}+L_{32}+L_{34}) \\ &{}(j_1,j_2,j_3) = (2,3,1):~\sigma =2,~s = 0:\qquad &{}+L_{12}L_{23}L_{31} \\ &{}(j_1,j_2,j_3) = (3,2,1):~\sigma =1,~s = 1:\qquad &{}+L_{13}(L_{21}+L_{23}+L_{24})L_{31} \\ &{}(j_1,j_2,j_3) = (3,1,2):~\sigma =2,~s = 0:\qquad &{}+L_{13}L_{21}L_{32}. \end{array} \end{aligned}$$

For each product term of the form $L_{ix_1}L_{2x_2}L_{3x_3}$, $x_q\in \{1,2,3,4\}$, one can draw a graph with the set of vertices ${\mathcal {S}} = \{1,2,3,4\}$ and the set of arcs ${\mathcal {A}} = \{1\rightarrow x_1,2\rightarrow x_2,3\rightarrow x_3,4\rightarrow x_4\}$. One can check that all terms corresponding to graphs with no cycles are encountered just once and only in the terms originating from $(j_1,j_2,j_3) = (1,2,3)$. All of them are preceded by the sign “-.” This corresponds to $a_g = (-1)^l = (-1)^3$. On the contrary, each term corresponding to a graph with cycles (in this example, there can be at most $N=1$ cycle) is encountered exactly twice ($2^1 = 2$): once it comes from the product corresponding to $(j_1,j_2,j_3) = (1,2,3)$ with sign “-,” and once it comes from some nonidentical permutation with sign “+.” Hence, all of such term cancel out.

Appendix 2: Proof of Theorem 6.1

Prior to start proving Theorem 6.1, we prove some auxiliary facts and introduce some use useful definitions. The proof of Theorem 6.1 will exploit the following lemmas.

Lemma 10.1

Suppose the function FindTgraphs is run on a graph $G({\mathcal {S}},{\mathcal {A}},{\mathcal {U}})$ satisfying Assumptions 1 and 2: FindTgraphs $(r,k,G,\Gamma ,{\mathcal {B}}')$. Let c be a cycle detected in the graph $\Gamma$ at step k. Suppose the weights of all outgoing arcs with tails in c and heads not in c are modified according to the update rule

$$\begin{aligned} U^{new}_{ij} = U_{ij} -U_{\mu (i)} +\gamma _k. \end{aligned}$$

Then $U_{ij}^{new} \ge \gamma _k$ for all $i\in c$, $j \notin c$, and $U_{ij}^{new} = \gamma _k$ if and only if $U_{ij}=U_{\mu (i)}$.

Proof

The fact that $U_{ij}^{new} \ge \gamma _k$ for all $i\in c$ and $j \notin c$ follows from the fact that $U_{ij}\ge U_{\mu (i)}$. The equality takes place if and only if $U_{\mu (i)} = U_{ij}$, i.e., if $i\rightarrow j$ is another min-arc from i.

Corollary 10.1

Suppose the function FindSymTgraphs is run on a graph $G({\mathcal {S}},{\mathcal {A}},{\mathcal {U}})$ satisfying Assumptions 1 and 2: FindSymTgraphs $(r,p,G,T,{\mathcal {B}})$. Let C be a closed communicating class detected in the graph T at step p. Suppose the weights of all outgoing arcs with tails in C and heads not in C are modified according to the update rule

$$\begin{aligned} U^{new}_{ij} = U_{ij} - U_{\min }(i) +\theta _p. \end{aligned}$$

Then $U_{ij}^{new} > \theta _p$ for all $i\in C$, $j \notin C$.

We will denote by ${\mathcal {S}}(v_{c})$ the subset of vertices of the original graph $G({\mathcal {S}},{\mathcal {A}},{\mathcal {U}})$ contracted into the super-vertex $v_{c}$.

Corollary 10.2

Suppose the function FindTgraphs is run on a graph $G({\mathcal {S}},{\mathcal {A}},{\mathcal {U}})$ satisfying Assumptions 1 and 2: FindTgraphs $(r,k,G,\Gamma ,{\mathcal {B}}')$. Let $c_1$, ..., $c_N$ be a sequence of cycles created after the addition of arcs of weights $\gamma _1 \le \gamma _2 \le \ldots \le \gamma _N$, respectively, such that ${\mathcal {S}}(v_{c_N})\supset {\mathcal {S}}(v_{c_l})$ for all $l < N$. Let i and j be vertices of G such that $(i\rightarrow j)\in {\mathcal {A}}$ and $i\in {\mathcal {S}}(v_{c_l})$, $l < N$, and $j\notin {\mathcal {S}}(v_{c_N})$. Let $U^{(l)}$ the (possibly modified) weight of the arc $i\rightarrow j$ after the creation of the cycle $c_l$. Then $U^{(N)}_{ij} \ge \gamma _N$ and $U^{(N)}_{ij} = \gamma _N$ if and only if $(i\rightarrow j)$ is a min-arc from i.

Corollary 10.3

Suppose the function FindTgraphs is run on a graph $G({\mathcal {S}},{\mathcal {A}},{\mathcal {U}})$ satisfying Assumptions 1 and 2: FindTgraphs $(r,k,G,\Gamma ,{\mathcal {B}}')$. Let $c_1$, ..., $c_N$ be a sequence of nested cycles created after the addition of arcs of weights $\gamma _1 \le \gamma _2 \le \ldots \le \gamma _N$, respectively, i.e., ${\mathcal {S}}(v_{c_1})\subset {\mathcal {S}}(v_{c_2})\subset \ldots \subset {\mathcal {S}}(v_{c_N})$. Suppose each set of vertices ${\mathcal {S}}(v_{c_l})$ contains a vertex x with an outgoing arc $x\rightarrow y$ such that $U_{xy} = U_{\min }(x)$, but $y\notin {\mathcal {S}}(v_{c_l})$, $l = 1,\ldots ,N-1$. Suppose there is an arc $(i\rightarrow j)\in {\mathcal {A}}$ such that $i\in {\mathcal {S}}(v_{c_1})$, $j\notin {\mathcal {S}}(v_{c_N})$. Then $U^{(N)}_{ij} = U_{ij}-U_{\min }(i) + \gamma _{N}$.

Proof

For all $1\le l\le N-1$, we have $U_{\min }(v_{c_l}) = \gamma _l$. Therefore,

$$\begin{aligned} U^{(1)}_{ij}&= U_{ij}-U_{\min }(i) + \gamma _1,\\ U^{(2)}_{ij}&= U_{ij}^{(1)} - U_{\min }(v_{c_1}) + \gamma _2 = U_{ij}-U_{\min }(i) + \gamma _2,\\&\ldots \\ U^{(N)}_{ij}&= U_{ij}^{(N-1)} - U_{\min }(v_{c_{N-1}}) + \gamma _N = U_{ij}-U_{\min }(i) + \gamma _N. \end{aligned}$$

Lemma 10.2

Let ${\mathcal {S}}'\subset {\mathcal {S}}$ be a subset of vertices of a directed graph $G({\mathcal {S}},{\mathcal {A}})$. Suppose every vertex in ${\mathcal {S}}'$ has at least one outgoing arc. Then

• (i) if all arcs with tails in ${\mathcal {S}}'$ have heads also in ${\mathcal {S}}'$, then there is at least one directed cycle formed by the arcs with tails in ${\mathcal {S}}'$;

• (ii) if the arcs with tails in ${\mathcal {S}}'$ form no directed cycles, then at least one of them must head in ${\mathcal {S}}\backslash {\mathcal {S}}'$.

Proof

Let us select one outgoing arc for each vertex in ${\mathcal {S}}'$ and denote the set of the selected arcs by ${\mathcal {A}}'$. If all arcs in ${\mathcal {A}}'$ head in ${\mathcal {S}}'$, then $|{\mathcal {A}}'| = |{\mathcal {S}}'|$ in the graph $G':=G'({\mathcal {S}}',{\mathcal {A}}')$. Hence, $G'$ cannot be a directed forest. Hence, it contains at least one cycle which proves Statement (i). Statement (ii) is the negation of (i).

Proof

(Theorem 6.1.) We start with Statement 4 because its proof is the shortest. A vertex i is an absorbing state of $T_p$ if and only if the weight of min-arcs from i in the original graph G is greater than $\theta _p$. In turn, this happens if and only if i has no outgoing arc in $\Gamma _{K_p}$, i.e., i is absorbing in $\Gamma _{K_p}$.

Auxiliary Statement: At the end of steps $K_p$ and p of Algorithms 1 and 2, respectively, for all $p\ge 0$ we have: ${\mathcal {B}}'\subseteq {\mathcal {B}}$ and the sets of distinct values in ${\mathcal {B}}'$ and ${\mathcal {B}}$ coincide.

Statements 1, 2, and 3 and the auxiliary statement will be proven by induction in the recursion level r of Algorithm 2.

Basis. The initial graphs $T_0 = T_0({\mathcal {S}},\emptyset ,\emptyset )$ and $\Gamma _0=\Gamma _0({\mathcal {S}},\emptyset ,\emptyset )$ in Algorithms 1 and 2 coincide. Furthermore, as the initializations in Algorithms 1 and 2 are complete, ${\mathcal {B}}'\subseteq {\mathcal {B}}$, and the sets of distinct arc weights in the buckets ${\mathcal {B}}'$ and ${\mathcal {B}}$ are the same. This gives us the induction basis.

Induction Assumption. Assume that at step $p_r$ of Algorithm 2 and the corresponding step $K_{p_r}$ of Algorithm 1, we have:

• ${\mathcal {B}}'\subseteq {\mathcal {B}}$;

• the set of distinct arc weights in ${\mathcal {B}}$ and ${\mathcal {B}}'$ are the same;

• all closed communicating classes are contracted into single super-vertices in $T^{(r)}_{p_r}$ and $\Gamma ^{(r')}_{K_{p_r}}$, where $r'$ is the recursion level in Algorithm 1 at the end of step $K_{p_r}$; furthermore, the set of vertices of $T^{(r)}_{p_r}$ and $\Gamma ^{(r')}_{K_{p_r}}$ coincide, and each super-vertex v of $T^{(r)}_{p_r}$ has the corresponding super-vertex $v'$ of $\Gamma ^{(r')}_{K_{p_r}}$ such that ${\mathcal {S}}(v) = {\mathcal {S}}(v')$;

• $\Gamma _{K_{p_r}}$ is a subgraph of $T_{p_r}$;

• the sets of distinct values in $\{\theta _p\}_{p=1}^{p_r}$ and $\{\gamma _{k}\}_{k=1}^{ K_{p_r}}$ coincide.

To prove the induction step, we need to show that Statements 1, 2, 3, and the auxiliary statement hold up to $p=p_{r+1}$.

Induction Step.

• 1. The induction assumptions imply that all graphs $\Gamma _{k}$, $K_{p-1}<k\le K_p$, are subgraphs of $T_p$ for all p such that the recursion levels remain $r'$ and r in Algorithms 1 and 2, respectively. For such p, the graphs $T_p^{(r)}$ and $\Gamma _{K_p}^{(r')}$ are built by adding arcs from buckets ${\mathcal {B}}$ and ${\mathcal {B}}'$, respectively, while no new arcs are added to these buckets and no arc weights are modified. Hence, Statements 1, 2, 3, and the auxiliary statement hold for all such p. Therefore, if no cycles are encountered by Algorithm 1 at steps $K_{p_r}<k\le K_{p_{r+1}-1}$, Statements 1, 2, 3, and the auxiliary statement hold for $K_{p_r}<k\le K_{p_{r+1}-1}$.

• 2. Now we show that Statements 1, 2, 3, and the auxiliary statement hold independent of whether or not cycles were encountered in Algorithm 1 at some $K_{p_r}<k\le K_{p_{r+1}-1}$. Note that a cycle can be formed in Algorithm 1 at some step $K_p<k\le K_{p+1}$ where $p_r<p<p_{r+1}$ only if an open communicating class is formed at step p of Algorithm 2.

  Since no arcs are added to the bucket ${\mathcal {B}}$ by Algorithm 2 at steps $p_r<p<p_{r+1}$, the graphs $T_p^{(r)}$ consist of all min-arcs of the graph $G^{(r)}$ of weights $\le \theta _{p_{r+1}-1}$. Now we consider Algorithm 1 for steps $K_{p_r}<k\le K_{p_{r+1}-1}$ and prove Statements 1, 2, 3, and the auxiliary statement by induction in the number of cycles.

  Let $c_1$ be the first cycle created in $\Gamma ^{(r')}$ at step $k_1$ after the addition of an arc of weight $\theta _p$. Since $\Gamma _{k_1}^{(r')}$ is a subgraph of $T_{p}^{(r)}$, the cycle $c_1$ must be a subclass of an open communicating class C created in $T^{(r)}$ after the addition of a set of arcs of weight $\theta _p$. Therefore, $c_1$ is an open communicating class in $T_p^{(r)}$. This means the set of min-arcs with tails in $c_1$ and heads not in $c_1$ in not empty. All these min-arcs are in $T_p^{(r)}$, and none of them is in $\Gamma _{k_1}^{(r')}$. By Lemma 10.1, the weights of these min-arcs become $\theta _p$ after the modification. One of them is picked and added to the bucket ${\mathcal {B}}'$. Hence, this arc will be added to $\Gamma ^{(r'+1)}$ at some $k_1<k\le K_p$. This allows us to conclude that at least one more arc of weight $\theta _p$ will be removed from ${\mathcal {B}}'$ after the cycle $c_1$ is formed. Hence, $c_1$ is not a closed communicating class in $\Gamma _{K_p}^{(r')}$. Therefore, Statements 1, 2, 3 and the auxiliary statement hold for $K_{p_r}<k\le \min \{k_2,K_{p_{r+1}-1}\}$.

  Assume that Statements 1, 2, 3, and the auxiliary statement hold up to step $k_{N} < K_{p_{r+1} -1}$ of Algorithm 1. Let cycle $c_N$ be encountered at step $k_N$ in Algorithm 1 after the addition of an arc of weight $\theta _p$. The set of vertices ${\mathcal {S}}^{(r')}(v_{c_N})$ is a subclass of an open communicating class C of the graph $T_p$ by the induction assumption. Hence, the set of the min-arcs with tails in ${\mathcal {S}}^{(r')}(v_{c_N})$ and heads not in ${\mathcal {S}}^{(r')}(v_{c_N})$ is not empty. All of these arcs are in $T_p^{(r)}$ but not in $\Gamma _{k_N}^{(r')}$. By Corollary 10.2, their weights will become $\theta _p$ during step $k_N$ of Algorithm 1. One of these min-arcs will be added to the bucket ${\mathcal {B}}'$ and then removed from it at some step $k_N<k\le K_{p}$. Hence, the cycle $c_N$ cannot be a closed communicating class in $T_p^{(r)}$. This proves Statements 1, 2, 3, and the auxiliary statement for all $K_{p_r}<k\le K_{p_{r+1}-1}$.

• 3. Now we show that Statements 1, 2, 3, and the auxiliary statement hold for $K_{p_{r+1}-1}+1\le k\le K_{p_{r+1}}$. Let C be the closed communicating class formed in the graph $T_{p_{r+1}}^{(r)}$ after the addition of the set of min-arcs of weight $\theta _{p_{r+1}}$. Therefore, all min-arcs from the vertices in C head in C. After contracting C into a single super-vertex $v_C$, the weight of min-arcs from it will be

  $$\begin{aligned} \min _{i\in C,~j\notin C}[U_{ij}-U_{\min }(i) +\theta _{p_{r+1}}] >\theta _{p_{r+1}}. \end{aligned}$$

  (39)

(Here $U_{\min }(i)$ is the weight of min-arcs from i in the graph $G^{(r)}$.) Let the recursion level at the end of step $K_{p_{r+1}-1}$ in Algorithm 1 be $r''$, and $C'$ be the set of vertices in $\Gamma ^{(r'')}$ corresponding to C. If no cycles were formed in Algorithm 1 with vertices in C then $C'=C$. Otherwise, some of the vertices of C are contacted into super-vertices in $\Gamma ^{(r'')}$. In this case, the subset of vertices of C contracted into super-vertices is an open communicating subclass of C. Let us show that $C'$ is a closed communicating class of $\Gamma _{K_{p_{r+1}}}^{(r'')}$.

Lemma 10.2 implies that the min-arcs from $C'$ all heading in $C'$ form at least one cycle c. Consider two cases.

Case 1: cycle c includes whole $C'$. Then by Corollary 10.3, the weight of min-arcs from the vertex $v_c$ in Algorithm 1 will be given by Eq. (39), i.e., the same as it is in Algorithm 2, and one of those min-arcs will be added to ${\mathcal {B}}'$. Hence, Statements 1, 2, 3, and the auxiliary statement hold at $p=p_{r+1}$ and $K_{p_{r+1}-1}<k\le K_{p_{r+1}}$.

Case 2: the cycle c does not include all vertices from $C'$. Since C does not contain closed communicating subclasses, the set of min-arcs in $G^{(r)}$ with tails in c and heads not in c is not empty. By Corollary 10.3, these min-arcs will be the min-arcs from $v_c$, and their weights will be $\theta _p$. One of these min-arcs will be added to ${\mathcal {B}}'$ and then removed. Hence, the min-arcs from $(C\backslash {\mathcal {S}}^{(r)}(v_c))\cup \{v_c\}$ head to $(C\backslash {\mathcal {S}}^{(r)}(v_c))\cup \{v_c\}$. By Lemma 10.2, they form at least one cycle c. Again, there are two options: either c includes all vertices of C or not. In the former case, using the argument from Case 1, we prove the induction step. In the latter case, we use the argument from Case 2. Repeating this argument at most $|C|-1$ number of times (as each new cycle includes at least one more vertex of $C'$ in comparison with the previous one), we obtain a cycle including all vertices of C.

Repeating this argument for all closed communicating classes formed in $T^{(r)}$ at step $p_{r+1}$, we conclude that all closed communicating classes encountered in Algorithm 2 at steps $p_{r+1}$ will be contracted into single super-vertices by both Algorithms 1 and 2, and arcs of the same weight will be added to ${\mathcal {B}}$ and ${\mathcal {B}}'$. Hence, the induction step is proven. This completes the proof of Theorem 6.1.