Skip to main content
SpringerLink
Log in

Resource management of cloud-enabled systems using model-free reinforcement learning

  • Published:
Annals of Telecommunications Aims and scope Submit manuscript

Abstract

The digital system of the future will face the growing challenge of controlling the system behavior in complex dynamically evolving environments. In this paper, we examine the applicability of a new management paradigm based on a reinforcement learning approach, where no preliminary specification of the system model is required. The learning agent identifies the most adequate control policies in live interaction with a partially observed system and provides it with autonomous management capabilities. We present the results of experimentation with cloud-based applications and discuss the technical challenges that need to be addressed in this field. Furthermore, we present the results of experimentation on a 5G network slice that hosts a cloud-based application in a multi-agent reinforcement learning setting, and demonstrate the value of information exchange between learning agents.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17

Similar content being viewed by others

References

  1. Sutton RS, Barto AG (1992) Reinforcement learning: an introduction, 2nd edn. The MIT Press, Cambridge, Massachusetts; London, England

    Book  MATH  Google Scholar 

  2. Clark J (2015) This preschool is for robots. Bloomberg

  3. Gu S, Holly E, Lillicrap T, Levine S (2017) Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates. In: IEEE international conference on robotics and automation (ICRA), Singapore

  4. Pit.ai, https://www.pit.ai/

  5. Mnih V, Kavukcuoglu K, Silver D, Rusu A, Veness J et al (2015) Human-level control through deep reinforcement learning. Nature 518:529–533

    Article  Google Scholar 

  6. Silver D, Hassabis D (2016) AlphaGo: mastering the ancient game of go with machine learning. Google Research Blog

  7. Jamshidi P, Sharifloo A, Pahl C, Metzger A, Estrada G (2015) Self-learning cloud controllers: fuzzy q-learning for knowledge evolution. In: International conference on cloud and autonomic computing, pp 208–211

    Google Scholar 

  8. OpenStack Cloud Operating System, http://www.openstack.org/

  9. Anicas M, An introduction to HAProxy and load balancing concepts. https://www.digitalocean.com/community/tutorials/an-introduction-to-haproxy-and-load-balancing-concepts

  10. Erinle B (2015) Performance testing with JMeter, 2nd edn. Packt Publishing

  11. Luong D-H, Thieu H-T, Outtagarts A, Ghamri-Doudane Y (2018) Predictive autoscaling orchestration for cloud-native telecom microservices. In: 2018 IEEE 5G world forum (5GWF). IEEE, pp 153–158

  12. Auto scaling in Amazon web services, https://aws.amazon.com/autoscaling/

  13. Jacobson D, Yuan D, Joshi N (2013) Scryer: Netflix’s predictive auto scaling engine. Netflix Technology Blog

  14. Roy N, Dubey A, Gokhale A (2011) Efficient autoscaling in the cloud using predictive models for workload forecasting. In: Proceedings of the 2011 IEEE 4th international conference on CLOUD computing (CLOUD '11), Washington, DC, USA, pp 500–507

  15. Li H and Venugopal S (2011) Using reinforcement learning for controlling an elastic web application hosting platform. International conference on automatic computing. pp 205–208

  16. Rao J, Bu X, Xu C-Z and Wang K (2011) A distributed self-learning approach for elastic provisioning of virtualized cloud resources. 19th annual IEEE international symposium on modelling, analysis, and simulation of computer and telecommunication systems. pp 45–54

Download references

Author information

Authors and Affiliations

  1. Nokia Bell Labs, Nokia, Paris, France

    Yue Jin, Makram Bouzid, Dimitre Kostadinov & Armen Aghasaryan

Authors
  1. Yue Jin
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Makram Bouzid
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Dimitre Kostadinov
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Armen Aghasaryan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yue Jin.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Jin, Y., Bouzid, M., Kostadinov, D. et al. Resource management of cloud-enabled systems using model-free reinforcement learning. Ann. Telecommun. 74, 625–636 (2019). https://doi.org/10.1007/s12243-019-00720-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12243-019-00720-y

Keywords

Search

Navigation