An Iterative Decision-Making Scheme for Markov Decision Processes and Its Application to Self-adaptive Systems

  • Guoxin Su
  • Taolue Chen
  • Yuan Feng
  • David S. Rosenblum
  • P. S. Thiagarajan
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9633)


Software is often governed by and thus adapts to phenomena that occur at runtime. Unlike traditional decision problems, where a decision-making model is determined for reasoning, the adaptation logic of such software is concerned with empirical data and is subject to practical constraints. We present an Iterative Decision-Making Scheme (IDMS) that infers both point and interval estimates for the undetermined transition probabilities in a Markov Decision Process (MDP) based on sampled data, and iteratively computes a confidently optimal scheduler from a given finite subset of schedulers. The most important feature of IDMS is the flexibility for adjusting the criterion of confident optimality and the sample size within the iteration, leading to a tradeoff between accuracy, data usage and computational overhead. We apply IDMS to an existing self-adaptation framework Rainbow and conduct a case study using a Rainbow system to demonstrate the flexibility of IDMS.


Goal State Markov Decision Process Computational Overhead Cumulative Cost Integer Matrice 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Baier, C., Katoen, J.-P.: Principles of Model Checking. The MIT Press, Cambridge (2008)zbMATHGoogle Scholar
  2. 2.
    Bencomo, N., Belaggoun, A., Issarny, V.: Dynamic decision networks for decision-making in self-adaptive systems: A case study. In: Proceedings of the 8th International Symposium on Software Engineering for Adaptive and Self-Managing Systems, SEAMS 2013, pp. 113–122. IEEE Press, Piscataway, NJ, USA (2013)Google Scholar
  3. 3.
    Calinescu, R., Ghezzi, C., Johnson, K., Pezzé, M., Rafiq, Y., Tamburrelli, G.: Formal verification with confidence intervals: A new approach to establishing the quality-of-service properties of software systems. IEEE Trans. Reliab. 99, 1–19 (2015)Google Scholar
  4. 4.
    Calinescu, R., Grunske, L., Kwiatkowska, M., Mirandola, R., Tamburrelli, G.: Dynamic QoS management and optimization in service-based systems. IEEE Trans. Softw. Eng. 37(3), 387–409 (2011)CrossRefGoogle Scholar
  5. 5.
    Camára, J., de Lemos, R.: Evaluation of resilience in self-adaptive systems using probabilistic model-checking. In: 2012 ICSE Workshop on Software Engineering for Adaptive and Self-Managing Systems (SEAMS), pp. 53–62, June 2012Google Scholar
  6. 6.
    Cámara, J., Moreno, G.A., Garlan, D.: Stochastic game analysis and latency awareness for proactive self-adaptation. In: Proceedings of the 9th International Symposium on Software Engineering for Adaptive and Self-Managing Systems, SEAMS, pp. 155–164. ACM, New York, NY, USA (2014)Google Scholar
  7. 7.
    Celiku, O., Garlan, D., Schmerl, B.: Augmenting architectural modeling to cope with uncertainty. In: Proceedings of the International Workshop on Living with Uncertainty (IWLU 2007), Atlanta, Georgia, USA (2007)Google Scholar
  8. 8.
    Chen, T., Forejt, V., Kwiatkowska, M.Z., Parker, D., Simaitis, A.: Automatic verification of competitive stochastic systems. Formal Method Syst. Des. 43(1), 61–92 (2013)CrossRefzbMATHGoogle Scholar
  9. 9.
    Cheng, S.-W.: Rainbow: Cost-Effective Software Architecture-based Self Adaptation. Ph.D. thesis, Carnegie Mellon University (2008)Google Scholar
  10. 10.
    Cheng, S.-W., Garlan, D.: Handling uncertainty in autonomic systems. In: Proceedings of the International Workshop on Living with Uncertainty (IWLU 2007), Atlanta, Georgia, USA (2007)Google Scholar
  11. 11.
    Cheng, S.-W., Garlan, D., Schmerl, B.: Architecture-based self-adaptation in the presence of multiple objectives. In: ICSE Workshop on Software Engineering for Adaptive and Self-Managing Systems (SEAMS 2006), Shanghai, China (2006)Google Scholar
  12. 12.
    Cooray, D., Malek, S., Roshandel, R., Kilgore, D.: RESISTing reliability degradation through proactive reconfiguration. In: Proceedings of the IEEE/ACM International Conference on Automated Software Engineering, ASE 2010, pp. 83–92. ACM, New York, NY, USA (2010)Google Scholar
  13. 13.
    Elkhodary, A., Esfahani, N., Malek, S.: FUSION: A framework for engineering self-tuning self-adaptive software systems. In: Proceedings of the Eighteenth ACM SIGSOFT International Symposium on Foundations of Software Engineering, FSE 2010, pp. 7–16. ACM, New York, NY, USA (2010)Google Scholar
  14. 14.
    Epifani, I., Ghezzi, C., Mirandola, R., Tamburrelli, G.: Model evolution by run-time parameter adaptation. In: Proceedings of the 31st International Conference on Software Engineering, ICSE 2009, pp. 111–121. IEEE Computer Society, Washington, DC, USA (2009)Google Scholar
  15. 15.
    Esfahani, N., Kouroshfar, E., Malek, S.: Taming uncertainty in self-adaptive software. In: Proceedings of the 19th ACM SIGSOFT Symposium and the 13th European Conference on Foundations of Software Engineering, ESEC/FSE 2011, pp. 234–244. ACM, New York, NY, USA (2011)Google Scholar
  16. 16.
    Esfahani, N., Malek, S.: Uncertainty in self-adaptive software systems. In: de Lemos, R., Giese, H., Müller, H.A., Shaw, M. (eds.) Software Engineering for Self-Adaptive Systems. LNCS, vol. 7475, pp. 214–238. Springer, Heidelberg (2013)CrossRefGoogle Scholar
  17. 17.
    Filieri, A., Hoffmann, H., Maggio, M.: Automated multi-objective control for self-adaptive software design. In: Proceedings of the 10th Joint Meeting on Foundations of Software Engineering, ESEC/FSE 2015, pp. 13–24 (2015)Google Scholar
  18. 18.
    Filieri, A., Tamburrelli, G., Ghezzi, C.: Supporting self-adaptation via quantitative verification and sensitivity analysis at run time. IEEE Trans. Softw. Eng. 42, 75–99 (2015)CrossRefGoogle Scholar
  19. 19.
    Forejt, V., Kwiatkowska, M., Norman, G., Parker, D.: Automated verification techniques for probabilistic systems. In: Bernardo, M., Issarny, V. (eds.) SFM 2011. LNCS, vol. 6659, pp. 53–113. Springer, Heidelberg (2011)CrossRefGoogle Scholar
  20. 20.
    Ghezzi, C., Pinto, L.S., Spoletini, P., Tamburrelli, G.: Managing non-functional uncertainty via model-driven adaptivity. In: Proceedings of the International Conference on Software Engineering, ICSE 2013, pp. 33–42. IEEE Press (2013)Google Scholar
  21. 21.
    Givan, R., Leach, S.M., Dean, T.L.: Bounded-parameter Markov Decision Processes. J. Artif. Intell. 122(1–2), 71–109 (2000)MathSciNetCrossRefzbMATHGoogle Scholar
  22. 22.
    Huebscher, M.C., McCann, J.A.: A survey of autonomic computing-degrees, models, and applications. ACM Comput. Surv. 40(3), 7: 1–7: 28 (2008)CrossRefGoogle Scholar
  23. 23.
    Moreno, G.A., Cámara, J., Garlan, D., Schmerl, B.: Proactive self-adaptation under uncertainty: A probabilistic model checking approach. In: Proceedings of the 10th Joint Meeting on Foundations of Software Engineering, ESEC/FSE 2015, pp. 1–12. ACM, New York, NY, USA (2015)Google Scholar
  24. 24.
    Nilim, A., Ghaoui, L.E.: Robust control of Markov Decision Processes with uncertain transition matrices. Oper. Res. 53(5), 780–798 (2005)MathSciNetCrossRefzbMATHGoogle Scholar
  25. 25.
    Poladian, V., Garlan, D., Shaw, M., Satyanarayanan, M., Schmerl, B., Sousat, J.: Leveraging resource prediction for anticipatory dynamic configuration. In: First International Conference on Self-Adaptive and Self-Organizing Systems, SASO 2007, pp. 214–223 (2007)Google Scholar
  26. 26.
    Puggelli, A., Li, W., Sangiovanni-Vincentelli, A.L., Seshia, S.A.: Polynomial-time verification of PCTL properties of MDPs with convex uncertainties. In: Sharygina, N., Veith, H. (eds.) CAV 2013. LNCS, vol. 8044, pp. 527–542. Springer, Heidelberg (2013)CrossRefGoogle Scholar
  27. 27.
    Puterman, M.L.: Markov decision processes. Handb. Oper. Res. Manage. Sci. 2, 331–434 (1990)MathSciNetCrossRefzbMATHGoogle Scholar
  28. 28.
    Su, G., Feng, Y., Chen, T., Rosenblum, D.S.: Asymptotic perturbation bounds for probabilistic model checking with empirically determined probability parameters. IEEE Trans. Softw. Eng. 99, 1–19 (2015)Google Scholar
  29. 29.
    Su, G., Rosenblum, D.S., Tamburrelli, G.: Reliability of run-time quality-of-service evaluation using parametirc model checking. In: Proceedings of the 38th International Conference on Software Engineering, ICSE 2016. ACM, New York, NY, USA (2016)Google Scholar
  30. 30.
    Tewari, A., Bartlett, P.L.: Bounded parameter markov decision processes with average reward criterion. In: Bshouty, N.H., Gentile, C. (eds.) COLT. LNCS (LNAI), vol. 4539, pp. 263–277. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  31. 31.
    Varga, R.S.: Matrix Iterative Analysis. Springer Series in Computational Mathematics. Springer, Heidelberg (2009)zbMATHGoogle Scholar
  32. 32.
    Wiesemann, W., Kuhn, D., Rustem, B.: Robust markov decision processes. Math. Oper. Res. 38(1), 153–183 (2013)MathSciNetCrossRefzbMATHGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2016

Authors and Affiliations

  • Guoxin Su
    • 1
  • Taolue Chen
    • 2
  • Yuan Feng
    • 3
  • David S. Rosenblum
    • 1
  • P. S. Thiagarajan
    • 4
  1. 1.School of ComputingNational University of SingaporeSingaporeSingapore
  2. 2.Department of Computer ScienceMiddlesex University LondonLondonUK
  3. 3.Centre for Quantum Computation and Intelligent SystemsUniversity of Technology SydneySydneyAustralia
  4. 4.Laboratory of Systems PharmacologyHarvard Medical SchoolBostonUSA

Personalised recommendations