Modelling and Analysis of Markov Reward Automata

  • Dennis Guck
  • Mark Timmer
  • Hassan Hatefi
  • Enno Ruijters
  • Mariëlle Stoelinga
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8837)


Costs and rewards are important ingredients for many types of systems, modelling critical aspects like energy consumption, task completion, repair costs, and memory usage. This paper introduces Markov reward automata, an extension of Markov automata that allows the modelling of systems incorporating rewards (or costs) in addition to nondeterminism, discrete probabilistic choice and continuous stochastic timing. Rewards come in two flavours: action rewards, acquired instantaneously when taking a transition; and state rewards, acquired while residing in a state. We present algorithms to optimise three reward functions: the expected cumulative reward until a goal is reached, the expected cumulative reward until a certain time bound, and the long-run average reward. We have implemented these algorithms in the SCOOP/IMCA tool chain and show their feasibility via several case studies.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Andova, S., Hermanns, H., Katoen, J.-P.: Discrete-time rewards model-checked. In: Larsen, K.G., Niebert, P. (eds.) FORMATS 2003. LNCS, vol. 2791, pp. 88–104. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  2. 2.
    Bamberg, R.: Non-deterministic generalised stochastic Petri nets modelling and analysis. Master’s thesis, University of Twente (2012)Google Scholar
  3. 3.
    Bernardo, M.: An algebra-based method to associate rewards with EMPA terms. In: Degano, P., Gorrieri, R., Marchetti-Spaccamela, A. (eds.) ICALP 1997. LNCS, vol. 1256, pp. 358–368. Springer, Heidelberg (1997)CrossRefGoogle Scholar
  4. 4.
    Boudali, H., Crouzen, P., Stoelinga, M.I.A.: A rigorous, compositional, and extensible framework for dynamic fault tree analysis. IEEE Transactions on Dependable and Secure Computing 7(2), 128–143 (2010)CrossRefGoogle Scholar
  5. 5.
    Bozzano, M., Cimatti, A., Katoen, J.-P., Nguyen, V.Y., Noll, T., Roveri, M.: Safety, dependability and performance analysis of extended AADL models. The Computer Journal 54(5), 754–775 (2011)CrossRefGoogle Scholar
  6. 6.
    Braitling, B., Fioriti, L.M.F., Hatefi, H., Wimmer, R., Becker, B., Hermanns, H.: MeGARA: Menu-based game abstraction and abstraction refinement of Markov automata. In: QAPL. EPTCS, vol. 154, pp. 48–63 (2014)Google Scholar
  7. 7.
    Brazdil, T., Brozek, V., Chatterjee, K., Forejt, V., Kucera, A.: Two views on multiple mean-payoff objectives in Markov decision processes. In: LICS, pp. 33–42. IEEE (2011)Google Scholar
  8. 8.
    Chatterjee, K., Henzinger, M.: Faster and dynamic algorithms for maximal end-component decomposition and related graph problems in probabilistic verification. In: SODA, pp. 1318–1336. SIAM (2011)Google Scholar
  9. 9.
    Clark, G.: Formalising the specification of rewards with PEPA. In: PAPM, pp. 139–160 (1996)Google Scholar
  10. 10.
    de Alfaro, L.: Formal Verification of Probabilistic Systems. PhD thesis, Stanford University (1997)Google Scholar
  11. 11.
    Deng, Y., Hennessy, M.: Compositional reasoning for weighted Markov decision processes. Science of Computer Programming 78(12), 2537–2579 (2013), Special Section on International Software Product Line Conference 2010 and Fundamentals of Software Engineering (selected papers of FSEN 2011)Google Scholar
  12. 12.
    Deng, Y., Hennessy, M.: On the semantics of Markov automata. Information and Computation 222, 139–168 (2013)MathSciNetCrossRefzbMATHGoogle Scholar
  13. 13.
    Eisentraut, C., Hermanns, H., Katoen, J.-P., Zhang, L.: A semantics for every GSPN. In: Colom, J.-M., Desel, J. (eds.) PETRI NETS 2013. LNCS, vol. 7927, pp. 90–109. Springer, Heidelberg (2013)CrossRefGoogle Scholar
  14. 14.
    Eisentraut, C., Hermanns, H., Zhang, L.: Concurrency and composition in a stochastic world. In: Gastin, P., Laroussinie, F. (eds.) CONCUR 2010. LNCS, vol. 6269, pp. 21–39. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  15. 15.
    Eisentraut, C., Hermanns, H., Zhang, L.: On probabilistic automata in continuous time. In: LICS, pp. 342–351. IEEE (2010)Google Scholar
  16. 16.
    Groote, J.F., Ponse, A.: The syntax and semantics of μCRL. In: ACP, Workshops in Computing, pp. 26–62. Springer (1995)Google Scholar
  17. 17.
    Guck, D., Han, T., Katoen, J.-P., Neuhäußer, M.R.: Quantitative timed analysis of interactive Markov chains. In: Goodloe, A.E., Person, S. (eds.) NFM 2012. LNCS, vol. 7226, pp. 8–23. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  18. 18.
    Guck, D., Hatefi, H., Hermanns, H., Katoen, J.-P., Timmer, M.: Modelling, reduction and analysis of Markov automata. In: Joshi, K., Siegle, M., Stoelinga, M., D’Argenio, P.R. (eds.) QEST 2013. LNCS, vol. 8054, pp. 55–71. Springer, Heidelberg (2013)CrossRefGoogle Scholar
  19. 19.
    Guck, D., Timmer, M., Hatefi, H., Ruijters, E.J.J., Stoelinga, M.I.A.: Modelling and analysis of Markov reward automata (extended version). Technical Report TR-CTIT-14-06, CTIT, University of Twente, Enschede (2014)Google Scholar
  20. 20.
    Hatefi, H., Hermanns, H.: Model checking algorithms for Markov automata. Electronic Communications of the EASST 53 (2012)Google Scholar
  21. 21.
    Haverkort, B.R., Cloth, L., Hermanns, H., Katoen, J.-P., Baier, C.: Model checking performability properties. In: DSN, pp. 103–112. IEEE (2002)Google Scholar
  22. 22.
    Haverkort, B.R., Hermanns, H., Katoen, J.-P.: On the use of model checking techniques for dependability evaluation. In: SRDS, pp. 228–237. IEEE (2000)Google Scholar
  23. 23.
    Hermanns, H. (ed.): Interactive Markov Chains. LNCS, vol. 2428. Springer, Heidelberg (2002)zbMATHGoogle Scholar
  24. 24.
    Katoen, J.-P., Zapreev, I.S., Hahn, E.M., Hermanns, H., Jansen, D.N.: The ins and outs of the probabilistic model checker MRMC. Performance Evaluation 68(2), 90–104 (2011)CrossRefGoogle Scholar
  25. 25.
    Neuhäußer, M.R.: Model Checking Nondeterministic and Randomly Timed Systems. PhD thesis, University of Twente (2010)Google Scholar
  26. 26.
    Neuhäußer, M.R., Stoelinga, M.I.A., Katoen, J.-P.: Delayed nondeterminism in continuous-time Markov decision processes. In: de Alfaro, L. (ed.) FOSSACS 2009. LNCS, vol. 5504, pp. 364–379. Springer, Heidelberg (2009)CrossRefGoogle Scholar
  27. 27.
    Segala, R.: Modeling and Verification of Randomized Distributed Real-Time Systems. PhD thesis, Massachusetts Institute of Technology (1995)Google Scholar
  28. 28.
    Song, L., Zhang, L., Godskesen, J.C.: Late weak bisimulation for Markov automata. Technical report, ArXiv e-prints (2012)Google Scholar
  29. 29.
    Srinivasan, M.M.: Nondeterministic polling systems. Management Science 37(6), 667–681 (1991)CrossRefzbMATHGoogle Scholar
  30. 30.
    Timmer, M.: SCOOP: A tool for symbolic optimisations of probabilistic processes. In: QEST, pp. 149–150. IEEE (2011)Google Scholar
  31. 31.
    Timmer, M.: Efficient Modelling, Generation and Analysis of Markov Automata. PhD thesis, University of Twente (2013)Google Scholar
  32. 32.
    Timmer, M., Katoen, J.-P., van de Pol, J.C., Stoelinga, M.I.A.: Efficient modelling and generation of Markov automata. In: Koutny, M., Ulidowski, I. (eds.) CONCUR 2012. LNCS, vol. 7454, pp. 364–379. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  33. 33.
    Timmer, M., van de Pol, J., Stoelinga, M.I.A.: Confluence reduction for Markov automata. In: Braberman, V., Fribourg, L. (eds.) FORMATS 2013. LNCS, vol. 8053, pp. 243–257. Springer, Heidelberg (2013)CrossRefGoogle Scholar
  34. 34.
    van de Pol, J., Timmer, M.: State space reduction of linear processes using control flow reconstruction. In: Liu, Z., Ravn, A.P. (eds.) ATVA 2009. LNCS, vol. 5799, pp. 54–68. Springer, Heidelberg (2009)CrossRefGoogle Scholar
  35. 35.
    Wunderling, R.: Paralleler und objektorientierter Simplex-Algorithmus. PhD thesis, Technische Universität Berlin (1996)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Dennis Guck
    • 1
  • Mark Timmer
    • 1
  • Hassan Hatefi
    • 2
  • Enno Ruijters
    • 1
  • Mariëlle Stoelinga
    • 1
  1. 1.Formal Methods and ToolsUniversity of TwenteThe Netherlands
  2. 2.Dependable Systems and SoftwareSaarland UniversityGermany

Personalised recommendations