Abstract
Recursive methods have been proposed for the numerical solution of equations arising in the analysis and control of Markov processes. Two examples are (i) solving for the equilibrium probabilities, and (ii) solving for the minimal expected cost or time to reach state zero, in a Markov process with left-skip-free transition structure. We discuss conditions under which recursive techniques are numerically stable and efficient, giving applications to descriptive and control models for queues.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
T. Altiok, Approximation of general distributions by phase-type distributions, Technical Report No. 82-4, Department of Industrial Engineering, N.C. State University, Raleigh (1982).
W. Bux and U. Herzog, The phase concept: Approximation of measured data and performance analysis, in:Computer Performance, ed. K. Chandy and M. Reiser (North-Holland, Amsterdam, 1977).
W. Cao and W. Stewart, Queueing models, block upper Hessenberg matrices, and the method of Neuts (1986), this volume.
E. Cinlar,Introduction to Stochastic Processes (Prentice-Hall, Englewood Cliffs, NJ, 1975).
D. Cox, A use of complex probabilities in the theory of stochastic processes, Proc. Camb. Phil. Soc. 51(1955)313.
G. Golub and C. van Loan,Matrix Computations (Johns Hopkins University Press, Baltimore, 1983).
W. Grassmann, Rounding errors in some recursive methods used in computational probability, Technical Report No. 73 (NSF ECS 80-17867), Department of Operations Research, Stanford University (1983).
W. Grassmann and M. Chaudhry, A new method to solve steady-state queueing equations Nav. Res. Log. Quart. 29(1982)461.
D. Gross and C. Harris,Fundamentals of Queueing Theory (Wiley, New York, 1974).
D. Heyman and M. Sobel,Stochastic Models in Operations Research, Vol. I (McGraw-Hill, New York, 1982).
J. Keilson, The use of Green's functions in the study of random walks, with applications to queueing theory, J. Math. and Phys. 41(1962)42.
F. Kelly,Reversibility in Stochastic Networks (Wiley, New York, 1979).
R. Marie, Calculating equilibrium probabilities for λ(n)/C(k)/1/N queues, Proc. of Performance 80(1980)117.
M. Neuts, Queues solvable without Rouch's theorem, Oper. Res. 27(1979)767.
M. Neuts,Matrix-Geometric Solutions in Stochastic Models (Johns Hopkins University Press, Baltimore, 1981).
M. Neuts and R. Nadarajan, A multiserver queue with thresholds for the acceptance of customers into service, Oper. Res. 30(1982)948.
M. Schäl, Conditions for optimality in dynamic programming and for the limit ofn-stage optimal policies to be optimal, Z. Wahrscheinlichkeitstheorie und Verw. Geb. 32(1975)179.
R. Schassberger,Warteschlangen (Springer-Verlag, New York, 1973).
S. Stidham, Regenerative processes in the theory of queues, with applications to the alternating-priority queue, Adv. Appl. Prob. 4(1972)542.
S. Stidham and R. Weber, Monotone and insensitive optimal policies for the control of queues with undiscounted costs, Program in Operations Research, N.C. State University, Raleigh (1983).
M. van Hoorn,Algorithms and Approximations for Queueing Systems (Mathematical Centre, Amsterdam, 1983).
J. Wijngaard and S. Stidham, Forward recursion for Markov decision processes with skip-free-to-the-right transitions, Part I: Theory and Algorithms, Math. Oper. Res. 11(1986)295.
Author information
Authors and Affiliations
Additional information
Research partially supported by the U.S. Army Research Office, Contract No. DAAG29-82-K-0152, at North Carolina State University, Raleigh, North Carolina.
Rights and permissions
About this article
Cite this article
Stidham, S. Stable recursive procedures for numerical computations in markov models. Ann Oper Res 8, 27–40 (1987). https://doi.org/10.1007/BF02187080
Issue Date:
DOI: https://doi.org/10.1007/BF02187080