Using Reinforcement Learning to Handle the Runtime Uncertainties in Self-adaptive Software

  • Tong Wu
  • Qingshan LiEmail author
  • Lu Wang
  • Liu He
  • Yujie Li
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 11176)


The growth of scale and complexity of software as well as the complex environment with high dynamic lead to the uncertainties in self-adaptive software’s decision making at run time. Self-adaptive software needs the ability to avoid negative effects of uncertainties to the quality attributes of the software. However, existing planning methods cannot handle the two types of runtime uncertainties caused by complexity of system and running environment. This paper proposes a planning method to handle these two types of runtime uncertainties based on reinforcement learning. To handle the uncertainty from the system, the proposed method can exchange ineffective self-adaptive strategies to effective ones according to the iterations of execution effects at run time. It can plan dynamically to handle uncertainty from environment by learning knowledge of relationship between system states and actions. This method can also generate new strategies to deal with unknown situations. Finally, we use a complex distributed e-commerce system, Bookstore, to validate the ability of proposed method.


Self-adaptive software Runtime uncertainty Reinforcement learning Self-adaptive planning method 



This work is supported by the Projects (61672401) supported by the National Natural Science Foundation of China; Projects (315***10101, 315**0102) supported by the Pre-Research Project of the “Thirteenth Five-Year-Plan” of China.


  1. 1.
    Krupitzer, C., Roth, F.M., Vansyckel, S., et al.: A survey on engineering approaches for self-adaptive systems. Pervasive Mob. Comput. 17(PB), 184–206 (2015)CrossRefGoogle Scholar
  2. 2.
    Cheng, S.W., Garlan, D.: Handling uncertainty in autonomic systems. In: International Workshop on Living with Uncertainties (2007)Google Scholar
  3. 3.
    Esfahani, N., Kouroshfar, E., Malek, S., et al.: Taming uncertainty in self-adaptive software. In: 13th European conference on Foundations of Software Engineering, pp. 234–244. ACM (2011)Google Scholar
  4. 4.
    Elkhodary, A., Esfahani, N., Malek, S., et al.: FUSION: a framework for engineering self-tuning self-adaptive software systems. In: 18th ACM SIGSOFT International Symposium on Foundations of Software Engineering, pp. 7–16. ACM (2010)Google Scholar
  5. 5.
    Mao, X., Dong, M., Liu, L., et al.: An integrated approach to developing self-adaptive software. J. Inf. Sci. Eng. 30(4), 1071–1085 (2014)Google Scholar
  6. 6.
    Amoui, M., Salehie, M., Mirarab, S., et al.: Adaptive action selection in autonomic software using reinforcement learning. In: International Conference on Autonomic and Autonomous Systems, pp. 175–181. IEEE Computer Society (2008)Google Scholar

Copyright information

© Springer Nature Switzerland AG 2018

Authors and Affiliations

  • Tong Wu
    • 1
  • Qingshan Li
    • 1
    Email author
  • Lu Wang
    • 1
  • Liu He
    • 1
  • Yujie Li
    • 1
  1. 1.Software Engineering InstituteXidian UniversityXi’anChina

Personalised recommendations