Skip to main content
Book cover

International Conference on Machine Learning for Networking

MLN 2018: Machine Learning for Networking pp 117–132Cite as

Common Structures in Resource Management as Driver for Reinforcement Learning: A Survey and Research Tracks

  • 1196 Accesses

Part of the Lecture Notes in Computer Science book series (LNISA,volume 11407)

Abstract

In the era of growing digitalization, dynamic resource management becomes one of the critical problems in many application fields where, due to the permanently evolving environment, the trade-off between cost and system performance needs to be continuously adapted. While traditional approaches based on prior system specification or model learning are challenged by the complexity and the dynamicity of these systems, a new paradigm of learning in interaction brings a strong promise - based on the toolset of model-free Reinforcement Learning (RL) and its great success stories in various domains. However, current RL methods still struggle to learn rapidly in incremental, online settings, which is a barrier to deal with many practical problems. To address the slow convergence issue, one approach consists in exploiting the system’s structural properties, instead of acting in full model-free mode. In this paper, we review the existing resource management systems and unveil their common structural properties. We propose a meta-model and discuss the tracks on how these properties can enhance general purpose RL algorithms.

Keywords

  • Resource management
  • RL
  • Capacity management
  • Learning through interactions

This is a preview of subscription content, access via your institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/978-3-030-19945-6_8
  • Chapter length: 16 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
eBook
USD   79.99
Price excludes VAT (USA)
  • ISBN: 978-3-030-19945-6
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
Softcover Book
USD   99.99
Price excludes VAT (USA)
Learn about institutional subscriptions
Fig. 1.
Fig. 2.
Fig. 3.
Fig. 4.

References

  1. Sutton, R.S., Barto, A.G.: RL: An Introduction, 2nd edn. The MIT Press, Cambridge, London (2017)

    Google Scholar 

  2. Clark, J. This Preschool is for Robots. Bloomberg (2015)

    Google Scholar 

  3. Gu, S., Holly, E., et al.: Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates. In: IEEE International Conference on Robotics and Automation (ICRA), Singapore (2017)

    Google Scholar 

  4. Pit.ai. https://www.pit.ai/

  5. Mnih, V., Kavukcuoglu, K., Silver, D., et al.: Human-level control through deep Reinforcement Learning. Nature 518, 529–533 (2015)

    CrossRef  Google Scholar 

  6. Silver, D., Hassabis, D.: AlphaGo: mastering the ancient game of Go with Machine Learning. Google Research Blog (2016)

    Google Scholar 

  7. Jin, Y., Bouzid, M., Kostadinov, D., Aghasaryan, A.: Model-free resource management of cloud-based applications using RL. In: International Workshop on Network Intelligence (NI/ICIN2018), Paris, France (2018)

    Google Scholar 

  8. Liu, Y., Watt, W.: Stabilizing customer abandonment in many-server queues with time-varying arrivals. Oper. Res. 60(6), 1551–1564 (2012)

    CrossRef  MathSciNet  Google Scholar 

  9. Fu, M.C., Marcus, S.I., Wang, I.: Monotone optimal policies for a transient queueing staffing problem. Oper. Res. 48(2), 327–331 (2000)

    CrossRef  Google Scholar 

  10. Bassamboo, A., Harrison, J.M., Zeevi, A.: Design and control of a large call center: asymptotic analysis of an LP-based method. Oper. Res. 54(3), 419–435 (2006)

    CrossRef  MathSciNet  Google Scholar 

  11. Defraeye, M., Van Nieuwenhuyse, I.: Staffing and scheduling under nonstationary demand for service: a literature review. Omega 58, 4–25 (2016)

    CrossRef  Google Scholar 

  12. Gans, N., Koole, G., Mandelbaum, A.: Telephone call centers: tutorial, review, and research prospects. Manuf. Serv. Oper. Manage. 5(2), 79–141 (2003)

    CrossRef  Google Scholar 

  13. Tan, T., Alp, O.: An integrated approach to inventory and flexible capacity management subject to fixed costs and non-stationary stochastic demand. OR Spectrum 31(2), 337–360 (2009)

    CrossRef  MathSciNet  Google Scholar 

  14. Buyukkaramikli, N.C., van Ooijen, H.P., Bertrand, J.W.: Integrating inventory control and capacity management at a maintenance service provider. Ann. Oper. Res. 231(1), 185–206 (2015)

    CrossRef  MathSciNet  Google Scholar 

  15. Bradley, J.R., Glynn, P.W.: Managing capacity and inventory jointly in manufacturing systems. Manage. Sci. 48(2), 273–288 (2002)

    CrossRef  Google Scholar 

  16. Snyder, L.V., Atan, Z., Peng, P., Rong, Y., Schmitt, A.J., Sinsoysal, B.: OR/MS models for supply chain disruptions: a review. IIE Trans. 48(2), 89–109 (2015)

    CrossRef  Google Scholar 

  17. Parikh, S., Patel, N., Prajapati, H.: Resource management in cloud computing: classification and taxonomy. CoRR (2017)

    Google Scholar 

  18. Jennings, B., Stadler, R.: Resource management in clouds: survey and research challenges. J. Netw. Syst. Manage. 23, 567–619 (2015)

    CrossRef  Google Scholar 

  19. Mann, Z.A.: Allocation of virtual machines in cloud data centers - a survey of problem models and optimization algorithms. ACM Comput. Surv. 48(1), 11 (2015)

    CrossRef  Google Scholar 

  20. Amazon: AWS Auto Scaling. https://aws.amazon.com/autoscaling/

  21. Jacobson, D., Yuan, D., Joshi, N.: Scryer: Netflix’s Predictive Auto Scaling Engine. Netflix Technology Blog (2013)

    Google Scholar 

  22. Roy, N., Dubey, A., Gokhale, A.: Efficient autoscaling in the cloud using predictive models for workload forecasting. In: IEEE CLOUD 2011, Washington, pp. 500–507 (2011)

    Google Scholar 

  23. Li, H., Venugopal, S.: Using RL for controlling an elastic web application hosting platform. In: International Conference on Automatic Computing, pp. 205–208 (2011)

    Google Scholar 

  24. Rao, J., Bu, X., Xu, C.-Z., Wang, K.: A distributed self-learning approach for elastic provisioning of virtualized cloud resources. In: 19th Annual IEEE International Symposium on Modelling, Analysis, and Simulation of Computer and Telecommunication Systems, pp. 45–54 (2011)

    Google Scholar 

  25. Manvi, S.S., Shyam, G.K.: Resource management for Infrastructure as a Service (IaaS) in cloud computing: a survey. J. Netw. Comput. Appl. 41, 424–440 (2014)

    CrossRef  Google Scholar 

  26. SON: Self-Organizing Networks. https://www.3gpp.org/technologies/keywords-acronyms/105-son

  27. Hämäläinen, S., Sanneck, H., Sartori, C.: LTE self-organising networks (SON): Network Management Automation for Operational Efficiency. Wiley, Chichester (2012)

    Google Scholar 

  28. Sesia, S., Toufik, I., Baker, M.: LTE - The UMTS Long Term Evolution: From Theory to Practice, 2nd edn. Wiley, Chichester (2011)

    CrossRef  Google Scholar 

  29. Rodriguez, J.: Fundamentals of 5G Mobile Networks. Wiley, Chichester (2015)

    CrossRef  Google Scholar 

  30. Network Functions Virtualisation – Update White Paper. ETSI (2013)

    Google Scholar 

  31. Evolution of the cloud-native mobile core, Nokia White Paper (2017)

    Google Scholar 

  32. Evolving Mobile Core to Being Cloud Native. Cisco White Paper (2017)

    Google Scholar 

  33. Project Clearwater - IMS in the Cloud. http://www.projectclearwater.org/

  34. Pearl, J.: Causality: Models, Reasoning and Inference, 2nd edn. Cambridge University Press, New York (2009)

    CrossRef  Google Scholar 

  35. Yoo, J.: Queueing models for staffing service operations. Ph.D. dissertation. University of Maryland, College Park, MD (1996)

    Google Scholar 

  36. Djonin, D.V., Krishnamurthy, V.: Q-learning algorithms for constrained markov decision processes with randomized monotone policies: application to MIMO transmission control. IEEE Trans. Signal Process. 55(5), 2170–2181 (2007)

    CrossRef  MathSciNet  Google Scholar 

  37. Djonin, D.V., Krishnamurthy, V.: MIMO transmission control in fading channels—a constrained markov decision process formulation with monotone randomized policies. IEEE Trans. Signal Process. 55(10), 5069–5083 (2007)

    CrossRef  MathSciNet  Google Scholar 

  38. Krishnamurthy, V.: Structural Results for Partially Observed Markov Decision Processes (2015). arXiv:1512.03873. https://arxiv.org/abs/1512.03873

  39. Rosenbaum, P.: Design of Observational Studies. Springer, New York (2010). https://doi.org/10.1007/978-1-4419-1213-8

    CrossRef  MATH  Google Scholar 

  40. Shanmugam, K., Kocaoglu, M., Dimakis, A., Vishwanath, S.: Learning causal graphs with small interventions. In: NIPS 2015, Cambridge, MA, USA, pp. 3195–3203 (2015)

    Google Scholar 

  41. Le, T., Hoang, T., Li, J., Liu, L., Liu, H.: A fast PC algorithm for high dimensional causal discovery with multi-core PCs. In: IEEE/ACM Transactions on Computational Biology and Bioinformatics (2015). https://doi.org/10.1109/tcbb.2016.2591526

  42. Spirtes, P., Glymour, C., Scheines, R.: Causation, Prediction, and Search, 2nd edn. MIT Press, Cambridge (2000)

    MATH  Google Scholar 

  43. Ruder, S.: Transfer Learning - Machine Learning’s Next Frontier. Blog post (2017). http://ruder.io/transfer-learning/

  44. Bingel, J., Søgaard, A.: Identifying beneficial task relations for multi-task learning in deep neural networks. In: EACL, pp. 164–169 (2017)

    Google Scholar 

  45. Glorot, X., Bordes, A., Bengio, Y.: Domain adaptation for large-scale sentiment classification: a deep learning approach. In: 28th International Conference on Machine Learning, pp. 513–520 (2011)

    Google Scholar 

  46. Taylor, M., Stone, P.: Transfer learning for reinforcement learning domains: a survey. J. Mach. Learn. Res. 10, 1633–1685 (2009)

    MathSciNet  MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Nokia Bell Labs, Paris-Saclay, Nozay, France

    Yue Jin, Dimitre Kostadinov, Makram Bouzid & Armen Aghasaryan

Authors
  1. Yue Jin
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Dimitre Kostadinov
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Makram Bouzid
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Armen Aghasaryan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yue Jin .

Editor information

Editors and Affiliations

  1. Télécom SudParis, Évry, France

    Éric Renault

  2. Inria, Paris, France

    Paul Mühlethaler

  3. CNAM/CEDRIC, Paris, France

    Selma Boumerdassi

Rights and permissions

Reprints and Permissions

Copyright information

© 2019 Springer Nature Switzerland AG

About this paper

Verify currency and authenticity via CrossMark

Cite this paper

Jin, Y., Kostadinov, D., Bouzid, M., Aghasaryan, A. (2019). Common Structures in Resource Management as Driver for Reinforcement Learning: A Survey and Research Tracks. In: Renault, É., Mühlethaler, P., Boumerdassi, S. (eds) Machine Learning for Networking. MLN 2018. Lecture Notes in Computer Science(), vol 11407. Springer, Cham. https://doi.org/10.1007/978-3-030-19945-6_8

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-19945-6_8

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-19944-9

  • Online ISBN: 978-3-030-19945-6

  • eBook Packages: Computer ScienceComputer Science (R0)