Abstract
The aim of this paper is to solve the basic stochastic shortest-path problem (SSPP) for Markov chains (MCs) with countable state space and then apply the results to a class of nearest-neighbor MCs on the lattice state space \(\mathbb Z \times \mathbb Z \) whose only moves are one step up, down, to the right or to the left. The objective is to control the MC, by suppressing certain moves, so as to minimize the expected time to reach a certain given target state. We characterize the optimal policies for SSPPs for general MCs with countably infinite state space, the main tool being a verification theorem for the value function, and give an algorithmic construction. Then we apply the results to a large class of examples: nearest-neighbor MCs for which the state space \(\mathbb Z \times \mathbb Z \) is split by a vertical line into two regions inside which the transition probabilities are the same for every state. We give a necessary and sufficient condition for the so-called distance-diminishing policy to be optimal. For the general case in which this condition does not hold we develop an explicit finite construction of an optimal policy.
Similar content being viewed by others
References
Bertsekas DP (2001) Dynamic programming and optimal control, vol II, 2nd edn. Athena Scientific, Belmont
Bertsekas DP, Tsitsiklis JN (1991) An analysis of stochastic shortest path problems. Math Oper Res 16:580–595
Bonet B (2007) On the speed of convergence of value iteration on stochastic shortest-path problems. Math Oper Res 32:365–373
Briggs AJ, Detweiler C, Scharstein D, Vandenberg-Rodes A (2004) Expected shortest paths for landmark-based robot navigation. Int J Rob Res 23:717–728
Eaton JH, Zadeh LA (1962) Optimal pursuit strategies in discrete state probabilistic systems. Trans ASME Ser D J Basic Eng 84:23–29
Puterman ML (1994) Markovian decision problems: discrete stochastic dynamic programming. Wiley, New York
Saerens M, Achbany Y, Fouss F, Yen L (2009) Randomized shortest-path problems: two related models. Neural Comput 21:2363–2404
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Lücking, D., Stadje, W. The stochastic shortest-path problem for Markov chains with infinite state space with applications to nearest-neighbor lattice chains. Math Meth Oper Res 77, 239–264 (2013). https://doi.org/10.1007/s00186-013-0427-8
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00186-013-0427-8