Queueing Systems

, Volume 21, Issue 3–4, pp 337–389 | Cite as

The achievable region method in the optimal control of queueing systems; formulations, bounds and policies

  • Dimitris Bertsimas


We survey a new approach that the author and his co-workers have developed to formulate stochastic control problems (predominantly queueing systems) asmathematical programming problems. The central idea is to characterize the region of achievable performance in a stochastic control problem, i.e., find linear or nonlinear constraints on the performance vectors that all policies satisfy. We present linear and nonlinear relaxations of the performance space for the following problems: Indexable systems (multiclass single station queues and multiarmed bandit problems), restless bandit problems, polling systems, multiclass queueing and loss networks. These relaxations lead to bounds on the performance of an optimal policy. Using information from the relaxations we construct heuristic nearly optimal policies. The theme in the paper is the thesis that better formulations lead to deeper understanding and better solution methods. Overall the proposed approach for stochastic control problems parallels efforts of the mathematical programming community in the last twenty years to develop sharper formulations (polyhedral combinatorics and more recently nonlinear relaxations) and leads to new insights ranging from a complete characterization and new algorithms for indexable systems to tight lower bounds and nearly optimal algorithms for restless bandit problems, polling systems, multiclass queueing and loss networks.


Queueing networks loss networks multiarmed bandits bounds policies optimization 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. [1]
    E. Anderson and P. Nash,Linear Programming in Infinite-Dimensional Spaces;Theory and Applications (Wiley, 1987).Google Scholar
  2. [2]
    F. Avram, D. Bertsimas and M. Ricard, Optimization of multiclass fluid queueing networks: a linear control approach, to appear inStochastic Networks;Proc. IMA (1995), eds. F. Kelly and R. Williams, pp. 199–234.Google Scholar
  3. [3]
    D. Atkins and H. Chen, Dynamic scheduling control for a network of queues, to appear.Google Scholar
  4. [4]
    R.E. Bellman,Dynamic Programming (Princeton University Press, Princeton, 1957).Google Scholar
  5. [5]
    D. Bertsimas and T. Cryssikou, Bounds and policies for loss networks, in preparation.Google Scholar
  6. [6]
    D. Bertsimas and J. Nino-Mora, Conservation laws, extended polymatroids and multiarmed bandit problems; a unified approach to indexable systems, to appear in Math. Oper. Res.Google Scholar
  7. [7]
    D. Bertsimas and J. Nino-Mora, Restless bandits, linear programming relaxations and a primaldual heuristic, submitted for publication.Google Scholar
  8. [8]
    D. Bertsimas, I. Paschalidis and J. Tsitsiklis, Optimization of multiclass queueing networks: polyhedral and nonlinear characterizations of achievable performance, Ann. Appl. Prob. 4 (1994) 43–75.Google Scholar
  9. [9]
    D. Bertsimas, I. Paschalidis and J. Tsitsiklis, Branching bandits and Klimov's problem: achievable region and side constraints, to appear in IEEE Trans. Autom. Contr.Google Scholar
  10. [10]
    D. Bertsimas and H. Xu, Optimization of polling systems and dynamic vehicle routing problems on networks, submitted for publication.Google Scholar
  11. [11]
    D. Bertsimas and R. Vohra, Linear programming relaxations, approximation algorithms and randomization: a unified approach to covering problems, submitted for publication.Google Scholar
  12. [12]
    P.P. Bhattacharya, L. Georgiadis and P. Tsoucas, Extended polymatroids: Properties and optimization,Proc. Int. Conf. on Integer Programming and Combinatorial Optimization, Carnegie Mellon University (Mathematical Programming Society, 1991) pp 349–361.Google Scholar
  13. [13]
    O.J. Boxma, H. Levy and J.A. Weststrate, Optimization of polling systems, in:Performance '90, eds. P. King, I. Mitrani and R. Pooley (North-Holland, Amsterdam 1990) pp. 349–361.Google Scholar
  14. [14]
    H. Chen and D. Yao, Optimal scheduling control of a multiclass fluid network, to appear in Oper. Res.Google Scholar
  15. [15]
    E. Coffman and I. Mitrani, A characterization of waiting time performance realizable by single server queues, Oper. Res. 28 (1980) 810–821.Google Scholar
  16. [16]
    D.R. Cox and W.L. Smith,Queues (Methuen, London/Wiley, New York, 1961).Google Scholar
  17. [17]
    J. Edmonds, Submodular functions, matroids and certain polyhedra, in: Combinatorial Structures and Their Applications, eds. R. Guy et al. (Gordon and Breach, New York, 1970) pp. 69–87.Google Scholar
  18. [18]
    A. Federgruen and H. Groenevelt, Characterization and optimization of achievable performance in general queueing systems. Oper. Res. 36 (1988) 733–741.Google Scholar
  19. [19]
    A. Federgruen and H. Groenevelt,M/G/c queueing systems with multiple customer classes: Characterization and control of achievable performance under nonpreemptive priority rules, Manag. Sci. 34 (1988) 1121–1138.Google Scholar
  20. [20]
    E. Gelenbe and I. Mitrani,Analysis and Synthesis of Computer Systems (Academic Press, New York, 1980).Google Scholar
  21. [21]
    J.C. Gittins and D.M. Jones, A dynamic allocation index for the sequential design of experiments, eds. J. Gani, K. Sarkadi and I. Vince in:Progress in Statistics European Meeting of Statisticians 1972, Vol. 1 (North-Holland, Amsterdam, 1974) pp. 241–266.Google Scholar
  22. [22]
    J.C. Gittins, Bandit processes and dynamic allocation indices, J. Roy. Stat. Soc. Series, B14 (1979) 148–177.Google Scholar
  23. [23]
    J.C. Gittins,Bandit Processes and Dynamic Allocation Indices (Wiley, 1989).Google Scholar
  24. [24]
    K.D. Glazebrook, Stochastic scheduling with order constraints, Int. J. Syst. Sci. (1976) 657–666.Google Scholar
  25. [25]
    K.D. Glazebrook, Sensitivity analysis for stochastic scheduling problems, Math. Oper. Res. 12 (1987) 205–223.Google Scholar
  26. [26]
    J.M. Harrison, A priority queue with discounted linear costs, Oper. Res. 23 (1975) 260–269.Google Scholar
  27. [27]
    J.M. Harrison, Dynamic scheduling of a multiclass queue: discount optimality, Oper. Res. 23 (1975) 270–282.Google Scholar
  28. [28]
    J.M. Harrison and L.M. Wein, Scheduling network of queues: Heavy traffic analysis of a simple open network', Queueing Systems 5 (1989) 265–280.Google Scholar
  29. [29]
    D.P. Heyman and M.J. Sobel,Stochastic Models in Operations Research, Vol. II: Stochastic Optimization (McGraw-Hill, New York, 1984).Google Scholar
  30. [30]
    M. Hofri and K.W. Ross, On the optimal control of two queues with server set up times and its analysis, SIAM J. Comp. 16 (1988) 399–419.Google Scholar
  31. [31]
    W.A. Horn, Single-machine job sequencing with treelike precedence ordering and linear delay penalties, SIAM J. Appl. Math. 23 (1972) 189–202.Google Scholar
  32. [32]
    F. Kelly, Loss networks, Ann. Appl. Prob. 1 (1991) 319–378.Google Scholar
  33. [33]
    F. Kelly, Bounds on the performance of dynamic routing schemes for highly connected networks, Math. Oper. Res. 19 (1994) 1–20.Google Scholar
  34. [34]
    L. Kleinrock and H. Levy, The analysis of random polling systems, Oper. Res. 36 (1988) 716–732.Google Scholar
  35. [35]
    G.P. Klimov, Time sharing service systems I, Theor, Prob. Appl. 19 (1974) 532–551.Google Scholar
  36. [36]
    S. Kumar and P.R. Kumar, Performance bounds for queueing networks and scheduling policies, IEEE Trans. Autom. Contr. 39 (1994) 1600–1611.Google Scholar
  37. [37]
    H. Levy and M. Sidi, Polling systems: applications, modelling and optimization, IEEE Trans. Commun. 38 (1990) 1750–1760.Google Scholar
  38. [38]
    H. Levy, M. Sidi and O.J. Boxma, Dominance relations in polling systems, Queueing Systems 6 (1990) 155–171.Google Scholar
  39. [39]
    L. Lovász and A. Schrijver, Cones of matrices and set functions and 0–1 optimization, SIAM J. Opt. (1990) 166–190.Google Scholar
  40. [40]
    I. Meilijson and G. Weiss, Multiple feedback at a single-server station, Stoch. Process. Appl. 5 (1977) 195–205.Google Scholar
  41. [41]
    K.G. Murty,Linear Programming (Wiley, New York, 1983).Google Scholar
  42. [42]
    Z. Ou and L. Wein, Performance bounds for scheduling queueing networks, Ann. Appl. Prob. 2 (1992) 460–480.Google Scholar
  43. [43]
    M. Queyranne, Structure of a simple scheduling polyhedron, Math. Program. 58 (1993) 263–285.Google Scholar
  44. [44]
    C. Papadimitriou,Computational Complexity (Addison-Wesley, 1994).Google Scholar
  45. [45]
    C. Papadimitriou and J. Tsitsiklis, Complexity of queueing network problems, Extended Abstract (1993).Google Scholar
  46. [46]
    M. Reiman and L. Wein, Dynamic scheduling of a two-class queue with setups, submitted for publication.Google Scholar
  47. [47]
    K.W. Ross and D.D. Yao, Optimal dynamic scheduling in Jackson Networks, IEEE Trans. Autom. Contr. 34 (1989) 47–53.Google Scholar
  48. [48]
    M.H. Rothkopf, Scheduling independent tasks on parallel processors, Manag. Sci. 12 (1966) 437–447.Google Scholar
  49. [49]
    M.H. Rothkopf, Scheduling with random service times, Manag. Sci. 12 (1966) 707–713.Google Scholar
  50. [50]
    J.G. Shantikumar and D.D. Yao, Multiclass queueing systems: Polymatroidal structure and optimal scheduling control, Oper. Res. 40, Suppl. 2 (1992) S293–299.Google Scholar
  51. [51]
    W.E. Smith, Various optimizers for single-stage production, Naval Res. Logist. Quart. 3 (1956) 59–66.Google Scholar
  52. [52]
    H. Takagi,Analysis of Polling Systems (MIT Press, 1986).Google Scholar
  53. [53]
    H. Takagi, Queueing analysis of polling systems, ACM Comput. Surveys 20 (1988) 5–28.Google Scholar
  54. [54]
    D.W. Tcha and S.R. Pliska, Optimal control of single-server queueing networks and multi-classM/G/1 queues with feedback, Oper. Res. 25 (1977) 248–258.Google Scholar
  55. [55]
    J.N. Tsitsiklis, A lemma on the multiarmed bandit problem, IEEE Trans. Autom. Contr. 31 (1986) 576–577.Google Scholar
  56. [56]
    J.N. Tsitsiklis, A short proof of the Gittins index theorem, Ann. Appl. Prob, to appear.Google Scholar
  57. [57]
    P. Tsoucas, The region of achievable performance in a model of Klimov, Technical Report RC16543, IBM T.J. Watson Research Center (1991).Google Scholar
  58. [58]
    P.P. Varaiya, J.C. Walrand and C. Buyukkoc, Extensions of the multiarmed bandit problem: The discounted case, IEEE Trans. Autom. Contr. 30 (1985) 426–439.Google Scholar
  59. [59]
    R. Weber, On the Gittins index for multiarmed bandits, Ann. Appl. Prob. 2 (1992) 1024–1033.Google Scholar
  60. [60]
    G. Weiss, Branching bandit processes, Prob. Eng. Inf. Sci. 2 (1988) 269–278.Google Scholar
  61. [61]
    P. Whittle, Multiarmed bandits and the Gittins index, J. Roy. Stat. Soc. 42 (1980) 143–149.Google Scholar
  62. [62]
    P. Whittle, Restless bandits: activity allocation in changing world,Celebration of Applied Probability, ed. J. Gani, J. Appl. Prob. A25 (1988) 287–298.Google Scholar

Copyright information

© J.C. Baltzer AG, Science Publishers 1995

Authors and Affiliations

  • Dimitris Bertsimas
    • 1
  1. 1.Sloan School of Management and Operations Research CenterMassachusetts Institute of TechnologyCambridgeUSA

Personalised recommendations