Using Reinforcement Learning to Handle the Runtime Uncertainties in Self-adaptive Software
The growth of scale and complexity of software as well as the complex environment with high dynamic lead to the uncertainties in self-adaptive software’s decision making at run time. Self-adaptive software needs the ability to avoid negative effects of uncertainties to the quality attributes of the software. However, existing planning methods cannot handle the two types of runtime uncertainties caused by complexity of system and running environment. This paper proposes a planning method to handle these two types of runtime uncertainties based on reinforcement learning. To handle the uncertainty from the system, the proposed method can exchange ineffective self-adaptive strategies to effective ones according to the iterations of execution effects at run time. It can plan dynamically to handle uncertainty from environment by learning knowledge of relationship between system states and actions. This method can also generate new strategies to deal with unknown situations. Finally, we use a complex distributed e-commerce system, Bookstore, to validate the ability of proposed method.
KeywordsSelf-adaptive software Runtime uncertainty Reinforcement learning Self-adaptive planning method
This work is supported by the Projects (61672401) supported by the National Natural Science Foundation of China; Projects (315***10101, 315**0102) supported by the Pre-Research Project of the “Thirteenth Five-Year-Plan” of China.
- 2.Cheng, S.W., Garlan, D.: Handling uncertainty in autonomic systems. In: International Workshop on Living with Uncertainties (2007)Google Scholar
- 3.Esfahani, N., Kouroshfar, E., Malek, S., et al.: Taming uncertainty in self-adaptive software. In: 13th European conference on Foundations of Software Engineering, pp. 234–244. ACM (2011)Google Scholar
- 4.Elkhodary, A., Esfahani, N., Malek, S., et al.: FUSION: a framework for engineering self-tuning self-adaptive software systems. In: 18th ACM SIGSOFT International Symposium on Foundations of Software Engineering, pp. 7–16. ACM (2010)Google Scholar
- 5.Mao, X., Dong, M., Liu, L., et al.: An integrated approach to developing self-adaptive software. J. Inf. Sci. Eng. 30(4), 1071–1085 (2014)Google Scholar
- 6.Amoui, M., Salehie, M., Mirarab, S., et al.: Adaptive action selection in autonomic software using reinforcement learning. In: International Conference on Autonomic and Autonomous Systems, pp. 175–181. IEEE Computer Society (2008)Google Scholar