Resource management of cloud-enabled systems using model-free reinforcement learning

Abstract

The digital system of the future will face the growing challenge of controlling the system behavior in complex dynamically evolving environments. In this paper, we examine the applicability of a new management paradigm based on a reinforcement learning approach, where no preliminary specification of the system model is required. The learning agent identifies the most adequate control policies in live interaction with a partially observed system and provides it with autonomous management capabilities. We present the results of experimentation with cloud-based applications and discuss the technical challenges that need to be addressed in this field. Furthermore, we present the results of experimentation on a 5G network slice that hosts a cloud-based application in a multi-agent reinforcement learning setting, and demonstrate the value of information exchange between learning agents.

This is a preview of subscription content, access via your institution.

Fig 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17

References

  1. 1.

    Sutton RS, Barto AG (1992) Reinforcement learning: an introduction, 2nd edn. The MIT Press, Cambridge, Massachusetts; London, England

    Google Scholar 

  2. 2.

    Clark J (2015) This preschool is for robots. Bloomberg

  3. 3.

    Gu S, Holly E, Lillicrap T, Levine S (2017) Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates. In: IEEE international conference on robotics and automation (ICRA), Singapore

  4. 4.

    Pit.ai, https://www.pit.ai/

  5. 5.

    Mnih V, Kavukcuoglu K, Silver D, Rusu A, Veness J et al (2015) Human-level control through deep reinforcement learning. Nature 518:529–533

    Article  Google Scholar 

  6. 6.

    Silver D, Hassabis D (2016) AlphaGo: mastering the ancient game of go with machine learning. Google Research Blog

  7. 7.

    Jamshidi P, Sharifloo A, Pahl C, Metzger A, Estrada G (2015) Self-learning cloud controllers: fuzzy q-learning for knowledge evolution. In: International conference on cloud and autonomic computing, pp 208–211

    Google Scholar 

  8. 8.

    OpenStack Cloud Operating System, http://www.openstack.org/

  9. 9.

    Anicas M, An introduction to HAProxy and load balancing concepts. https://www.digitalocean.com/community/tutorials/an-introduction-to-haproxy-and-load-balancing-concepts

  10. 10.

    Erinle B (2015) Performance testing with JMeter, 2nd edn. Packt Publishing

  11. 11.

    Luong D-H, Thieu H-T, Outtagarts A, Ghamri-Doudane Y (2018) Predictive autoscaling orchestration for cloud-native telecom microservices. In: 2018 IEEE 5G world forum (5GWF). IEEE, pp 153–158

  12. 12.

    Auto scaling in Amazon web services, https://aws.amazon.com/autoscaling/

  13. 13.

    Jacobson D, Yuan D, Joshi N (2013) Scryer: Netflix’s predictive auto scaling engine. Netflix Technology Blog

  14. 14.

    Roy N, Dubey A, Gokhale A (2011) Efficient autoscaling in the cloud using predictive models for workload forecasting. In: Proceedings of the 2011 IEEE 4th international conference on CLOUD computing (CLOUD '11), Washington, DC, USA, pp 500–507

  15. 15.

    Li H and Venugopal S (2011) Using reinforcement learning for controlling an elastic web application hosting platform. International conference on automatic computing. pp 205–208

  16. 16.

    Rao J, Bu X, Xu C-Z and Wang K (2011) A distributed self-learning approach for elastic provisioning of virtualized cloud resources. 19th annual IEEE international symposium on modelling, analysis, and simulation of computer and telecommunication systems. pp 45–54

Download references

Author information

Affiliations

Authors

Corresponding author

Correspondence to Yue Jin.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Jin, Y., Bouzid, M., Kostadinov, D. et al. Resource management of cloud-enabled systems using model-free reinforcement learning. Ann. Telecommun. 74, 625–636 (2019). https://doi.org/10.1007/s12243-019-00720-y

Download citation

Keywords

  • Reinforcement learning
  • Cloud computing
  • Application scaling
  • Resource optimization