Abstract
The application of Markov decision theory to the control of queueing systems often leads to models with enormous state spaces. Hence, direct computation of optimal policies with standard techniques and algorithms is almost impossible for most practical models. A convenient technique to overcome this issue is to use one-step policy improvement. For this technique to work, one needs to have a good understanding of the queueing system under study, and its (approximate) value function under policies that decompose the system into less complicated systems. This warrants the research on the relative value functions of simple queueing models, that can be used in the control of more complex queueing systems. In this chapter we provide a survey of value functions of basic queueing models and show how they can be applied to the control of more complex queueing systems.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsReferences
E. Altman, Applications of Markov decision processes in communication networks: a survey, in Handbook of Markov Decision Processes, edited by E.A. Feinberg, A. Shwartz (Kluwer, Dordrecht, 2002)
S. Asmussen, O. Nerman, M. Olsson, Fitting phase type distributions via the EM algorithm. Scand. J. Stat. 23, 419–441 (1996)
J.S. Baras, D.-J. Ma, A.M. Makowski, k competing queues with geometric service requirements and linear costs: the μ c rule is always optimal. Syst. Control Lett. 6, 173–180 (1985)
R. Bellman, Adaptive Control Processes: A Guided Tour (Princeton University Press, Princeton, NJ, 1961)
S. Bhulai, On the value function of the M/Cox(r)/1 queue. J. Appl. Probab. 43 (2), 363–376 (2006)
S. Bhulai, G.M. Koole, On the structure of value functions for threshold policies in queueing models. J. Appl. Probab. 40, 613–622 (2003)
S. Bhulai, F.M. Spieksma, On the uniqueness of solutions to the Poisson equations for average cost Markov chains with unbounded cost functions. Math. Meth. Oper. Res. 58 (2), 221–236 (2003)
S. Bhulai, A.C. Brooms, F.M. Spieksma, On structural properties of the value function for an unbounded jump Markov process with an application to a processor-sharing retrial queue. Queueing Syst. 76 (4), 425–446 (2014)
P. Chevalier, N. Tabordon, R. Shumsky, Routing and staffing in large call centers with specialized and fully flexible servers. Working paper (2004)
E. Hyytiä, J. Virtamo, Dynamic routing and wavelength assignment using first policy iteration. Technical Report COST 257, Helsinki University of Technology (2000)
G.M. Koole, Stochastic Scheduling and Dynamic Programming. CWI Tract, vol. 113 (CWI, Amsterdam, 1995)
S.A. Lippman, Applying a new device in the optimization of exponential queueing systems. Oper. Res. 23, 687–710 (1975)
R.A. Marie, Calculating equilibrium probabilities for λ(m)∕C k ∕1∕N queues, in Proceedings of the International Symposium on Computer Performance Modeling (1980)
R.E. Mickens, Difference Equations: Theory and Applications (Chapman & Hall, London, 1990)
J.M. Norman, Heuristic Procedures in Dynamic Programming (Manchester University Press, Manchester, 1972)
E.L. Örmeci, Dynamic admission control in a call center with one shared and two dedicated service facilities. IEEE Trans. Autom. Control 49, 1157–1161 (2004)
T.J. Ott, K.R. Krishnan, Separable routing: a scheme for state-dependent routing of circuit switched telephone traffic. Ann. Oper. Res. 35, 43–68 (1992)
M.L. Puterman, Markov Decision Processes: Discrete Stochastic Dynamic Programming (Wiley, New York, 1994)
H. Rummukainen, J. Virtamo, Polynomial cost approximations in markov decision theory based call admission control. IEEE/ACM Trans. Netw. 9 (6), 769–779 (2001)
S.A.E. Sassen, H.C. Tijms, R.D. Nobel, A heuristic rule for routing customers to parallel servers. Statistica Neerlandica 51, 107–121 (1997)
R.F. Serfozo, An equivalence between continuous and discrete time Markov decision processes. Oper. Res. 27, 616–620 (1979)
F.M. Spieksma, Geometrically ergodic Markov chains and the optimal control of queues. Ph.D. thesis, Leiden University (1990)
S. Stidham Jr., R.R. Weber, A survey of Markov decision models for control of networks of queues. Queueing Syst. 13, 291–314 (1993)
H.C. Tijms, Stochastic Models: An Algorithmic Approach (Wiley, New York, 1994)
D. Towsley, R.H. Hwang, J.F. Kurose, MDP routing for multi-rate loss networks. Comput. Netw. ISDN 34, 241–261 (2000)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2017 Springer International Publishing AG
About this chapter
Cite this chapter
Bhulai, S. (2017). Value Function Approximation in Complex Queueing Systems. In: Boucherie, R., van Dijk, N. (eds) Markov Decision Processes in Practice. International Series in Operations Research & Management Science, vol 248. Springer, Cham. https://doi.org/10.1007/978-3-319-47766-4_2
Download citation
DOI: https://doi.org/10.1007/978-3-319-47766-4_2
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-47764-0
Online ISBN: 978-3-319-47766-4
eBook Packages: Business and ManagementBusiness and Management (R0)