Abstract
To deal with situations not specifically designed for (unknown-unknowns), self-adaptive systems need to learn the best – or at least good enough – action to perform in each context faced during operation. An established solution for doing so is through the use of online learning. The complexity of online learning however increases in the presence of context shifts – which are typical in self-adaptive systems. In this paper, we (i) propose a new metric, convergence inertia, to assess the robustness of reinforcement learning policies against context shifts, and (ii) use it to assess the robustness of different policies within the family of multi-armed bandits (MAB) to context shifts. Through an experiment with a self-adaptation exemplar of a web server, we demonstrate that inertia and the accompanying interpretation of the unknown-unknowns problem is a viable way to inform the selection of online learning policies for self-adaptive systems, since it brings the influence of context shifts to the forefront. In our experiment, we found that non-stationary MAB policies are better suited to handling context shifts in terms of inertia, although stationary policies tend to perform well in terms of overall convergence.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Alberts, E.G.: Adapting with Regret: Using Multi-armed Bandits with Self-adaptive Systems. Master’s thesis, University of Amsterdam (2022). https://scripties.uba.uva.nl/search?id=727497
Auer, P., Cesa-Bianchi, N., Fischer, P.: Finite-time analysis of the multiarmed bandit problem. Mach. Learn. 47(2), 235–256 (2002). https://doi.org/10.1023/A:1013689704352
Auer, P., Cesa-Bianchi, N., Freund, Y., Schapire, R.E.: The nonstochastic multiarmed bandit problem. SIAM J. Comput. 32(1), 48–77 (2002). https://doi.org/10.1137/S0097539701398375
Bubeck, S.: Regret analysis of stochastic and nonstochastic multi-armed bandit problems. Found. Trends
Mach. Learn. 5(1), 1–122 (2012). https://doi.org/10.1561/2200000024Cabri, G., Capodieci, N.: Applying multi-armed bandit strategies to change of collaboration patterns at runtime. In: 2013 1st International Conference on Artificial Intelligence, Modelling and Simulation, pp. 151–156. IEEE, Kota Kinabalu (2013). https://doi.org/10.1109/AIMS.2013.31
Cardozo, N., Dusparic, I.: Auto-cop: adaptation generation in context-oriented programming using reinforcement learning options. CoRR abs/2103.06757 (2021)
Gheibi, O., Weyns, D., Quin, F.: Applying machine learning in self-adaptive systems: a systematic literature review. ACM Trans. Auton. Adapt. Syst. 15(3), 1–37 (2021). https://doi.org/10.1145/3469440
Kephart, J., Chess, D.: The vision of autonomic computing. Computer 36(1), 41–50 (2003)
Kim, D., Park, S.: Reinforcement learning-based dynamic adaptation planning method for architecture-based self-managed software. In: 2009 ICSE Workshop on Software Engineering for Adaptive and Self-Managing Systems, pp. 76–85 (2009). https://doi.org/10.1109/SEAMS.2009.5069076, iSSN: 2157–2321
Kinneer, C., Coker, Z., Wang, J., Garlan, D., Goues, C.L.: Managing uncertainty in self-adaptive systems with plan reuse and stochastic search. In: Proceedings of the 13th International Conference on Software Engineering for Adaptive and Self-Managing Systems, pp. 40–50. ACM, Gothenburg (2018). https://doi.org/10.1145/3194133.3194145
Kivinen, J., Szepesvári, C., Ukkonen, E., Zeugmann, T. (eds.): ALT 2011. LNCS (LNAI), vol. 6925. Springer, Heidelberg (2011). https://doi.org/10.1007/978-3-642-24412-4
Krupitzer, C., Roth, F.M., VanSyckel, S., Schiele, G., Becker, C.: A survey on engineering approaches for self-adaptive systems. Perv. Mob. Comput. 17, 184–206 (2015). https://doi.org/10.1016/j.pmcj.2014.09.009. Feb
Lattimore, T., Szepesvári, C.: Bandit Algorithms, 1st edn. Cambridge University Press, Cambridge (2020). https://doi.org/10.1017/9781108571401
de Lemos, R., et al.: Software engineering for self-adaptive systems: a second research roadmap. In: de Lemos, R., Giese, H., Müller, H.A., Shaw, M. (eds.) Software Engineering for Self-Adaptive Systems II. LNCS, vol. 7475, pp. 1–32. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-35813-5_1
Li, L., Chu, W., Langford, J., Schapire, R.E.: A contextual-bandit approach to personalized news article recommendation. In: Proceedings of the 19th International Conference on World Wide Web - WWW 2010, p. 661 (2010). https://doi.org/10.1145/1772690.1772758, arXiv: 1003.0146
Metzger, A., Kley, T., Palm, A.: Triggering proactive business process adaptations via online reinforcement learning. In: Fahland, D., Ghidini, C., Becker, J., Dumas, M. (eds.) BPM 2020. LNCS, vol. 12168, pp. 273–290. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58666-9_16
Metzger, A., Quinton, C., Mann, Z.A., Baresi, L., Pohl, K.: Feature-model-guided online learning for self-adaptive systems, vol. 12571, pp. 269–286 (2020). https://doi.org/10.1007/978-3-030-65310-1_20, arXiv: 1907.09158
Moreno, G.A., Schmerl, B., Garlan, D.: SWIM: an exemplar for evaluation and comparison of self-adaptation approaches for web applications. In: Proceedings of the 13th International Conference on Software Engineering for Adaptive and Self-Managing Systems, pp. 137–143. ACM, Gothenburg (2018). https://doi.org/10.1145/3194133.3194163
Palm, A., Metzger, A., Pohl, K.: Online reinforcement learning for self-adaptive information systems. In: Dustdar, S., Yu, E., Salinesi, C., Rieu, D., Pant, V. (eds.) CAiSE 2020. LNCS, vol. 12127, pp. 169–184. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-49435-3_11
Porter, B., Rodrigues Filho, R.: Distributed emergent software: assembling, perceiving and learning systems at scale. In: 2019 IEEE 13th International Conference on Self-Adaptive and Self-Organizing Systems (SASO), pp. 127–136 (2019). https://doi.org/10.1109/SASO.2019.00024, iSSN: 1949–3681
Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. MIT press, Cambridge (2018)
Mach. Learn. 5(1), 1–122 (2012). Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Alberts, E., Gerostathopoulos, I. (2022). Measuring Convergence Inertia: Online Learning in Self-adaptive Systems with Context Shifts. In: Margaria, T., Steffen, B. (eds) Leveraging Applications of Formal Methods, Verification and Validation. Adaptation and Learning. ISoLA 2022. Lecture Notes in Computer Science, vol 13703. Springer, Cham. https://doi.org/10.1007/978-3-031-19759-8_15
Download citation
DOI: https://doi.org/10.1007/978-3-031-19759-8_15
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-19758-1
Online ISBN: 978-3-031-19759-8
eBook Packages: Computer ScienceComputer Science (R0)