Variance reduction techniques for the simulation of Markov process

Abstract

Let X _n, n≧0 be an irreducible, aperiodic, Markov chain with finite state space E, transition matrix P, and stationary distribution π. Let f be a real valued function on E and define r = πf. A method of reducing the variance of simulation estimates for r is presented. The method combines the techniques of numerical analysis and simulation by partially solving an appropriate system of linear equations using some matrix iterative procedure and then estimating the difference between the true and partial solutions via simulation. After k iterations of the iterative procedure, functions f _ν ν = 0, ..., k are defined so that r = πf _gn for each ν. Let $\hat x_v (N) = \sum\limits_{n = 0}^N {f_v (x_n )/(N + 1)}$ and $\hat x_\beta (N) = \sum\limits_{v = 0}^k {\beta (v)\hat x_v (N)}$ where $\sum\limits_{v = 0}^k {\beta (v) = 1}$. Then $\hat x_\beta (N) \to r$ a.s. as N → ∞ and β is chosen to minimize the asymptotic variance of $\hat x_\beta (N)$. Numerical results for a simple queueing model are presented.