# Markov chains on \({{\mathbb {Z}}^+}\): analysis of stationary measure via harmonic functions approach

- 238 Downloads

## Abstract

We suggest a method for constructing a positive harmonic function for a wide class of transition kernels on \({{\mathbb {Z}}^+}\). We also find natural conditions under which this harmonic function has a positive finite limit at infinity. Further, we apply our results on harmonic functions to asymptotically homogeneous Markov chains on \({{\mathbb {Z}}^+}\) with asymptotically negative drift which arise in various queueing models. More precisely, assuming that the Markov chain satisfies Cramér’s condition, we study the tail asymptotics of its stationary distribution. In particular, we clarify the impact of the rate of convergence of chain jumps towards the limiting distribution.

## Keywords

Transition kernel Harmonic function Markov chain Stationary distribution Renewal function Exponential change of measure Queues## Mathematics Subject Classification

60J10 60J45 60K25 60F10 31C05## 1 Introduction

*P*(

*j*,

*i*) is the transition probability from

*j*to

*i*. In terms of infinite matrices, \(\pi \) is a positive left eigenvector of

*P*corresponding to the eigenvalue 1. We are interested in the asymptotic behaviour of \(\pi (i)\) for large values of

*i*. Markov chains on \({{\mathbb {Z}}^+}\) and their stationary probabilities naturally arise in various queueing models. A classical example is given by the queue length process at service completion epochs in the M/G/1 queue, which goes back to Kendall [10]; see Boxma and Lotov [3] for further analysis. A natural modification of this process is a system where the service rate depends on the current state of the system.

*i*, that is,

*asymptotically homogeneous in space*, that is,

*Lindley recursion*[11]:

*Q*(

*i*,

*j*), is asymptotically homogeneous in space, that is, \(Q(i,i+j)\rightarrow e^{\beta j}{{\mathbb {P}}}\{\xi =j\}\) as \(i\rightarrow \infty \), where the limit corresponds to the jump distribution of a random walk with positive mean \({{\mathbb {E}}}\xi e^{\beta \xi }\).

So, we need to alter this transition kernel a little in order to get a transition probability which makes it possible to analyse the corresponding stochastic object, which is a Markov chain with asymptotically positive drift. We perform such an alteration via Doob’s *h*-transform with a suitably chosen harmonic function. To make this approach possible, we first need to understand how to construct harmonic functions for transition kernels and how to analyse the harmonic functions constructed.

*Q*be a nonnegative finite transition kernel on \({{\mathbb {Z}}^+}\), that is, \(Q(i,j)\ge 0\) and

*i*and

*j*, there exists

*n*such that \(Q^n(i,j)>0\).

*u*(

*i*) is called

*harmonic*if \(Qu=u\), which means

*u*is a right eigenvector of

*Q*corresponding to the eigenvalue 1. In this paper, we only consider

*nonnegative*harmonic functions. Pruitt [14] has found sufficient and necessary conditions for the existence of such a function, but these conditions are quite hard to verify. Furthermore, his results do not give any information on the limiting behaviour of harmonic functions as \(i\rightarrow \infty \). Since this information is important for the study of asymptotic properties of Markov chains (see, for example, Foley and McDonald [8]), we are interested in a constructive approach to harmonic functions which would allow us to determine their asymptotics. This is the first problem we solve here; see Sect. 2.

As mentioned above, our primary motivation for studying harmonic functions of transition kernels comes from asymptotic tail analysis of stationary measures of Markov chains. The standard tool for studying large deviations is an exponential change of measure (Cramér transform). If we follow this approach in the context of Markov chains, then we usually get a positive transition kernel which is not stochastic, in general. In Sect. 5, we show how the results on asymptotics for harmonic functions obtained in Sect. 3 can be helpful in the study of stationary measures of asymptotically homogeneous Markov chains.

If the jumps of a positive recurrent Markov chain are bounded, then the equation for its invariant measure can be considered as a system of linear difference equations. The asymptotics of a fundamental solution to these equations can be found using refinements of the Poincaré–Perron theorem; see Elaydi [6] for details. However, one can not apply these results directly as we are only interested in a positive solution.

## 2 Construction of harmonic function

*i*, \(j\in {{\mathbb {Z}}^+}\), that

*underlying*Markov chain.

*f*is harmonic for the kernel

*Q*. Indeed, it follows by conditioning on \(X^Q_1\) that

*f*(

*i*) originates from what we observe in the following two particular cases: The first simple case is provided by a stochastic kernel

*Q*where we have the harmonic function \(f(i)\equiv 1\) which is the unique (up to a constant factor) bounded harmonic function for recurrent Markov kernels; see Meyn and Tweedie ([12], Theorem 17.1.5).

*Q*which is obtained from some stochastic kernel

*P*of a Markov chain \(Y_n\) by killing it in some set \(B\subset {{\mathbb {Z}}^+}\), that is,

*i*where \(P(i,{{\mathbb {Z}}^+}\setminus B)>0\). In this case, \(Q(i,{{\mathbb {Z}}^+})={{\mathbb {P}}}_i\{Y_1\not \in B\}\) and (3) reads as

*P*is transient and

*B*is finite, then the probability of not returning to

*B*is a harmonic function for this Markov chain killed in

*B*. It was proved by Doney [5, Theorem 1] that there is a unique harmonic function for a transient (towards \(\infty \)) random walk on \({{\mathbb {Z}}}\) killed at leaving \({{\mathbb {Z}}^+}\). This harmonic function equals the renewal function generated by descending ladder heights, which in turn is equal to \({{\mathbb {P}}}_i\{\tau _B=\infty \}\) with \(B=\{-1,-2,\ldots \}\); see Sect. 4.

*f*(

*i*) may be also defined as

*local time*at state

*j*,

## 3 On the limiting behaviour of the harmonic function for a transient kernel

In this section, we answer, in particular, the following question: What are natural conditions sufficient for (2) in the case when *Q* is transient? These sufficient conditions are presented in Proposition 3, and they guarantee that \(\limsup _{i\rightarrow \infty }f(i)\le 1\). Another question is what conditions guarantee that *f*(*i*) is a positive function and, moreover, \(\liminf _{i\rightarrow \infty }f(i)\ge 1\). This is answered in Proposition 2. Combining these two statements, we find sufficient conditions for the existence of a harmonic function satisfying \(f(i)\rightarrow 1\), which is our main result in this section, given in the following theorem.

### Theorem 1

*f*is harmonic and \(f(i)\rightarrow 1\) as \(i\rightarrow \infty \).

We split the proof of this result into two parts: the lower and upper bounds, given in Propositions 2 and 3, respectively.

*i*and

*j*) radius of convergence

*R*bigger than 1. It seems to be quite difficult to compare this assumption with our condition (8). Clearly, the condition (8) is ready for verification in particular cases because the total masses and the embedded Markov chain are factorised there. Also, our condition (7) is weaker than the ‘closeness’ condition in [8].

### 3.1 Lower bound for the harmonic function *f*

We start with a solidarity property for the kernel *Q* related to positivity of the function *f*, which may be considered as a generalisation of the Harnack inequality well known for sub-stochastic matrices.

### Proposition 1

If \(f(i)>0\) for some \(i\in {{\mathbb {Z}}^+}\), then \(f(j)>0\) for all \(j\in {{\mathbb {Z}}^+}\).

### Proof

*Q*implies that

### Proposition 2

### Proof

*Q*, transience of \(X_n^Q\) is equivalent to the following: for any fixed

*N*,

*N*. By the Markov property, for every \(j\le N\),

### 3.2 Upper bound for the harmonic function *f*

### Proposition 3

### Proof

*f*(

*i*) in terms of exponential moments of local times, by (9).

*N*and rewrite (17) as follows:

*N*,

*f*(

*i*). \(\square \)

### 3.3 On exponential moments for local times

### Example 1

*Q*be a local perturbation at the origin of the transition kernel of a simple random walk on \({{\mathbb {Z}}^+}\) as follows:

*q*/

*p*, that is,

*f*(

*i*) for

*Q*is a solution to the system of equations

*f*(0) as in (19), we conclude that the expressions in (20) and (21) are equal for all \(\alpha <p/q\). Further, for every \(\alpha >p/q\) and every \(f(0)>0\), the function

*f*(

*i*) from (21) becomes negative for

*i*large enough. Therefore, there is no positive harmonic function for \(\alpha >p/q\). Finally, in the critical case \(\alpha =p/q\), we have \(f(i)=f(0)(q/p)^i\rightarrow 0\). \(\Box \)

### Example 2

*f*with \(f(i)\rightarrow 1\) as \(i\rightarrow \infty \) if and only if \(p/q>\alpha _0\ldots \alpha _{N-1}\). This is equivalent to

Next let us give simple sufficient conditions that guarantee finiteness of some exponential moment of local times for a Markov chain \(X_n\) valued in \({{\mathbb {Z}}^+}\).

### Proposition 4

*N*. Then

### Proof

As follows from the proof, (23) holds with any \(\gamma <\log \displaystyle \frac{1}{1-p_\eta }\) provided the chain \(X_n\) satisfies the condition (22) with \(N=0\).

### Proposition 5

### Proof

The last proposition helps us to deduce finiteness of exponential moments of local times for a Markov chain with everywhere positive drift.

### Proposition 6

### Proof

*T*such that

*h*satisfies

*j*between

*i*and \(i+T\),

*g*(

*x*) is linear on \([-T,j-i+M]\). Therefore, for \(i\le j\le i+T\),

*g*is decreasing,

## 4 Random walk with negative drift conditioned to stay nonnegative

*h*-transform over \(S_n\) killed at leaving \({{\mathbb {Z}}^+}\), that is, a Markov chain on \({{\mathbb {Z}}^+}\) with transition probabilities

*f*is a positive harmonic function for the killed random walk, that is,

*u*(

*j*) stands for the mass function of the renewal process of strict descending ladder heights of \(S_n\).

*f*(

*i*). Start with the following transition kernel on \({{\mathbb {Z}}^+}\):

*Q*. Hence, the function

*x*, \(j>0\). Therefore,

The random walk conditioned to stay nonnegative is the simplest Markov chain where the general scheme of construction of a harmonic function helps. In the next section, we follow almost the same techniques in our study of tail behaviour for asymptotically homogeneous in space Markov chains with negative drift under Cramér-type assumptions. Although the scheme is the same in the main aspects, some additional arguments are required.

## 5 Positive recurrent Markov chains: asymptotic behaviour of stationary distribution

In this section, we consider an asymptotically homogeneous in space Markov chain \(X_n\) with jumps \(\xi (i)\), that is, \(\xi (i)\Rightarrow \xi \) as \(i\rightarrow \infty \). Our next result describes the case when the convergence of jumps is so fast that the stationary measure \(\pi \) of \(X_n\) is asymptotically proportional to that of the random walk \(W_n\) delayed at the origin.

### Theorem 2

*c*in front of \(e^{-\beta i}\) is positive they introduced the following condition:

### Example 3

### Proof of Theorem 2

*h*(

*i*) be a harmonic function for \(X_n\) killed at entering [0,

*N*], that is,

*h*-transform on \(X_n\) killed at entering [0,

*N*] and define a new Markov chain \({\widehat{X}}_n\) on \({{\mathbb {Z}}^+}\) with the following transition kernel:

*h*is harmonic, then we also have

*h*(

*i*) is such that the jumps \({\widehat{\xi }}(i)\) of the chain \({\widehat{X}}_n\) satisfy the following conditions:

*N*and to construct a harmonic function

*h*(

*i*) for \(X_n\) killed at entering [0,

*N*] such that

*h*satisfies the conditions (49)–(51). The intuition behind our construction of the function

*h*is simple. Since we consider asymptotically homogeneous Markov chain, the chain behaves similarly to the random walk with jumps like \(\xi \). We assume that the limiting jump satisfies Cramér’s condition; hence, it should be such that \(h(i)\sim e^{\beta i}\) as \(i\rightarrow \infty \).

*i*. Let us find a level

*N*such that the kernel \(Q^{(\beta )}_N\) satisfies the conditions of Theorem 1.

*N*such that condition (8) of Theorem 1 holds; here \(\ell ^{(\beta )}_N(i)\) is the local time at state

*i*of the underlying chain \(X^{(\beta )}_{N,n}\), \(n\ge 0\), of the kernel \(Q^{(\beta )}_N\).

*N*, a minorant with positive mean for the jumps \(\xi ^{(\beta )}_N(i)\) of the chain \(X^{(\beta )}_{N,n}\). The asymptotic homogeneity of the Markov chain \(X_n\) implies that

*N*and a random variable \(\eta \) with positive mean, \({{\mathbb {E}}}\eta >0\), such that

*N*,

*N*, the kernel \(Q^{(\beta )}_N\) satisfies all the conditions of Theorem 1. Therefore, there exists a positive harmonic function

*f*for this kernel such that \(f(i)\rightarrow 1\) as \(i\rightarrow \infty \).

*h*(

*i*) is a harmonic function for the Markov chain \(X_n\) killed at entering [0,

*N*]. Let us check that

*h*produces \({\widehat{X}}_n\) satisfying the conditions (49)–(51). First, the condition (49) holds because, for any \(j\ge N-i\),

We now turn our attention to the case where \({{\mathbb {E}}}e^{\beta \xi (i)}\) converges to 1 in a nonsummable way. The next result describes the behaviour of \(\pi \) in terms of a nonuniform exponential change of measure.

### Theorem 3

*o*(1/

*x*). Then, for some \(c>0\),

It should be noticed that Theorem 2 can be seen as a special case of Theorem 3 with \(\beta (x)\equiv \beta \). We have decided to split these two statements because of the milder moment condition (44) in Theorem 2 and since the proof of Theorem 3 is simpler via a reduction to the case of summable rate of convergence, which has been considered in Theorem 2.

### Proof of Theorem 3

*N*] in the same way as in the proof of Theorem 2 and to deduce that \(\pi (i)\sim c/g(i)\) as \(i\rightarrow \infty \), which completes the proof. \(\square \)

Since the function \(\beta (x)\) is stated implicitly in Theorem 3, it calls for specification of some cases where \(\beta (x)\) can be expressed in terms of the difference \({{\mathbb {E}}}e^{\beta \xi (i)}-1\).

### Corollary 1

### Proof

Notice that, since \(r\in (-2,-3/2)\), \(\alpha (x)\) is regularly varying at infinity with index \(r+1\in (-1,-1/2)\), \(A(x)\rightarrow \infty \), \(A(x)=o(x)\) as \(x\rightarrow \infty \) and \(\sum _{i=1}^\infty \alpha ^2(i)<\infty \).

Notice that the key condition on the rate of convergence of \({{\mathbb {E}}}e^{\beta \xi (i)}\) to 1 that implies the asymptotics (60) in the last corollary is that the sequence \(\alpha ^2(i)\) is summable. If it is not so, that is, if the index \(r+1\) of regular variation in the function \(\alpha (x)\) is between \(-1/2\) and 0, then the asymptotic behaviour of \(\pi (i)\) is different from (60). This is specified in the following corollary.

### Corollary 2

*M*, there exist constants \(D_{k,j}\) such that

### Proof

*z*equals \(1+m_1R_1\). Thus, \(R_1=-\,1/m_1\). Further, the coefficient of \(z^2\) is \(D_{1,1}R_1+m_1R_2+m_2R_1^2/2\) and, consequently,

*x*, but (61) and (62) remain valid, then one has, by the same arguments,

### Corollary 3

### Proof

*w*(

*z*) satisfying \(F(z,w(z))=0\). Since \(F(0,0)=0\) and \(\frac{\partial }{\partial w}F(0,0)=m_1>0\), we can apply Theorem B.4 from Flajolet and Sedgewick [7] which says that

*w*(

*z*) is analytic in a vicinity of zero, that is, there exists a \(\rho >0\) such that

*i*such that \(|\alpha (i)|<\rho \).

We conclude with the following remark. In the proof of Corollary 3, we have adapted the derivation of the Cramér series in large deviations for sums of independent random variables; see, for example, Petrov [13]. There is just one difference: we need analyticity of an implicit function instead of analyticity of an inverse function.

## Notes

### Acknowledgements

We are thankful to the referee whose comments helped to clarify some conditions of Theorem 1.

## References

- 1.Bertoin, J., Doney, R.: On conditioning a random walk to stay nonnegative. Ann. Probab.
**22**, 2152–2167 (1994)CrossRefGoogle Scholar - 2.Borovkov, A.A., Korshunov, D.: Large-deviation probabilities for one-dimensional Markov chains. Part 1: stationary distributions. Theory Probab. Appl.
**41**, 1–24 (1997)CrossRefGoogle Scholar - 3.Boxma, O.J., Lotov, V.I.: On a class of one-dimensional random walks. Markov Process. Relat. Fields
**2**, 349–362 (1996)Google Scholar - 4.Denisov, D.E.: On the existence of a regularly varying majorant of an integrable monotone function. Math. Notes
**76**, 129–133 (2006)CrossRefGoogle Scholar - 5.Doney, R.: The Martin boundary and ratio limit theorems for killed random walks. J. Lond. Math. Soc.
**58**, 761–768 (1998)CrossRefGoogle Scholar - 6.Elaydi, S.: An introduction to difference equations. 3rd Ed. Undergraduate Texts in Mathematics, Springer, New York (2005)Google Scholar
- 7.Flajolet, P., Sedgewick, R.: Analytic Combinatorics. Cambridge University Press, Cambridge (2009)CrossRefGoogle Scholar
- 8.Foley, R.D., McDonald, D.R.: Constructing a harmonic function for an irreducible nonnegative matrix with convergence parameter \(R>1\). Bull. Lond. Math. Soc.
**44**, 533–544 (2012)CrossRefGoogle Scholar - 9.Korshunov, D.: The key renewal theorem for a transient Markov chain. J. Theoret. Probab.
**21**, 234–245 (2008)CrossRefGoogle Scholar - 10.Kendall, D.G.: Stochastic processes occurring in the theory of queues and their analysis by the method of the imbedded Markov chain. Ann. Math. Stat.
**24**, 338–354 (1953)CrossRefGoogle Scholar - 11.Lindley, D.V.: The theory of queues with a single server. Proc. Camb. Philos. Soc.
**48**, 277–289 (1952)CrossRefGoogle Scholar - 12.Meyn, S., Tweedie, R.: Markov Chains and Stochastic Stability. Wiley, New York (1993)CrossRefGoogle Scholar
- 13.Petrov, V.V.: Sums of Independent Random Variables. Springer, Berlin (1975)CrossRefGoogle Scholar
- 14.Pruitt, W.E.: Eigenvalues of non-negative matrices. Ann. Math. Stat.
**35**, 1797–1800 (1964)CrossRefGoogle Scholar

## Copyright information

**Open
Access**This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.