The Role of Information in System Stability with Partially Observable Servers
- 5 Downloads
We present a methodology for analyzing the role of information on system stability. For this we consider a simple discrete-time controlled queueing system, where the controller has a choice of which server to use at each time slot and server performance varies according to a Markov modulated random environment. At the extreme cases of information availability, that is when there is either full information or no information, stability regions and maximally stabilizing policies are trivial. But in the more realistic cases where only the environment state of the selected server is observed, only the service successes are observed or only queue length is observed, finding throughput maximizing control laws is a challenge. To handle these situations, we devise a Partially Observable Markov Decision Process (POMDP) formulation of the problem and illustrate properties of its solution. We further model the system under given decision rules, using Quasi-Birth-and-Death (QBD) structure to find a matrix analytic expression for the stability bound. We use this formulation to illustrate how the stability region grows as the number of controller belief states increases. The example that we consider in this paper is a case of two servers where the environment of each is modulated like a Gilbert-Elliot channel. As simple as this case seems, there appear to be no closed form descriptions of the stability region under the various regimes considered. However, the numerical approximations to the POMDP Bellman equations together with the numerical solutions of the QBDs, both of which are in agreement, hint at a variety of structural results.
KeywordsSystem stability POMDP Control Queueing systems Optimal Bellman equations QBD Information Markov models
Mathematics Subject Classification 201060J28 49L20
Unable to display preview. Download preview PDF.
Azam Asanjarani’s research is supported by the Australian Research Council Centre of Excellence for the Mathematical and Statistical Frontiers (ACEMS). Yoni Nazarathy’s research is supported by ARC grant DP180101602.
- Asanjarani A (2016) QBD Modelling Of a finite state controller for queueing systems with unobservable Markovian environments. In: Proceedings of the 11th International Conference on Queueing Theory and Network Applications, ACM, p 20Google Scholar
- Bäuerle N., Rieder U (2011) Markov Decision Processes with Applications to Finance. Springer Science & Business MediaGoogle Scholar
- Bramson M (2008) Stability of Queueing Networks. SpringerGoogle Scholar
- Hernández-Lerma O, Lasserre JB (2012) Discrete-time Markov Control Processes: Basic Optimality Criteria. SpringerGoogle Scholar
- Kuhn J, Nazarathy Y (2015) Wireless Channel Selection with Reward-Observing Restless Multi-armed Bandits. draft book chapter submittedGoogle Scholar
- Larrañaga M., Assaad M, Destounis A, Paschos GS (2016) Asymptotically optimal pilot allocation over Markovian fading channels. arXiv:1608.08413
- Latouche G, Ramaswami V (1999) Introduction to matrix analytic methods in stochastic modeling. SIAMGoogle Scholar
- Leng B, Krishnamachari B, Guo X, Niu Z (2016) Optimal operation of a green server with bursty traffic, In: proceeding of Global Communications Conference (GLOBECOM), IEEE, pp 1–6Google Scholar
- Meyn S (2008) Control techniques for complex networks. Cambridge University PressGoogle Scholar
- Meshram R, Manjunath D, Gopalan A (2016) On the Whittle index for restless multiarmed hidden Markov bandits. arXiv:1603.04739
- Mészáros A, Telek M (2014) Markov decision process and linear programming based control of MAP/MAP/n queues. In: European Workshop on Performance Engineering, 179–193 SpringerGoogle Scholar
- Nazarathy Y, Taimre T, Asanjarani A, Kuhn J, Patch B, Vuorinen A (2015) The challenge of stabilizing control for queueing systems with unobservable server states. In: 5th Australian Control Conference (AUCC), IEEE, pp 342–347Google Scholar
- Puterman ML (2014) Markov Decision Processes: Discrete Stochastic Dynamic Programming. John Wiley & SonsGoogle Scholar
- Whittle P (1982) Optimization Over Time. John Wiley Sons Inc.Google Scholar