Polynomial-Time Alternating Probabilistic Bisimulation for Interval MDPs

  • Vahid HashemiEmail author
  • Andrea Turrini
  • Ernst Moritz Hahn
  • Holger Hermanns
  • Khaled Elbassioni
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10606)


Interval Markov decision processes (IMDPs) extend classical MDPs by allowing intervals to be used as transition probabilities. They provide a powerful modelling tool for probabilistic systems with an additional variation or uncertainty that relaxes the need of knowing the exact transition probabilities, which are usually difficult to get from real systems. In this paper, we discuss a notion of alternating probabilistic bisimulation to reduce the size of the IMDPs while preserving the probabilistic CTL properties it satisfies from both computational complexity and compositional reasoning perspectives. Our alternating probabilistic bisimulation stands on the competitive way of resolving the IMDP nondeterminism which in turn finds applications in the settings of the controller (parameter) synthesis for uncertain (parallel) probabilistic systems. By using the theory of linear programming, we improve the complexity of computing the bisimulation from the previously known EXPTIME to PTIME. Moreover, we show that the bisimulation for IMDPs is a congruence with respect to two facets of parallelism, namely synchronous product and interleaving. We finally demonstrate the practical effectiveness of our proposed approaches by applying them on several case studies using a prototypical tool.


Interval Markov Decision Processes (IMDPs) Exact Transition Probabilities Compositional Reasoning Probabilistic Computation Tree Logic (PCTL) Interval Markov Chains (IMCs) 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Alur, R., Henzinger, T.A., Kupferman, O., Vardi, M.Y.: Alternating refinement relations. In: Sangiorgi, D., de Simone, R. (eds.) CONCUR 1998. LNCS, vol. 1466, pp. 163–178. Springer, Heidelberg (1998). doi: 10.1007/BFb0055622 CrossRefGoogle Scholar
  2. 2.
    Baier, C., Katoen, J.-P.: Principles of Model Checking. The MIT Press, Cambridge (2008)zbMATHGoogle Scholar
  3. 3.
    Ben-Or, M.: Another advantage of free choice: completely asynchronous agreement protocols (extended abstract). In: PODC, pp. 27–30 (1983)Google Scholar
  4. 4.
    Benedikt, M., Lenhardt, R., Worrell, J.: LTL model checking of interval Markov chains. In: Piterman, N., Smolka, S.A. (eds.) TACAS 2013. LNCS, vol. 7795, pp. 32–46. Springer, Heidelberg (2013). doi: 10.1007/978-3-642-36742-7_3 CrossRefGoogle Scholar
  5. 5.
    Böde, E., Herbstritt, M., Hermanns, H., Johr, S., Peikenkamp, T., Pulungan, R., Rakow, J., Wimmer, R., Becker, B.: Compositional dependability evaluation for STATEMATE. ITSE 35(2), 274–292 (2009)CrossRefGoogle Scholar
  6. 6.
    Cattani, S., Segala, R.: Decision algorithms for probabilistic bisimulation. In: Brim, L., Křetínský, M., Kučera, A., Jančar, P. (eds.) CONCUR 2002. LNCS, vol. 2421, pp. 371–386. Springer, Heidelberg (2002). doi: 10.1007/3-540-45694-5_25 CrossRefGoogle Scholar
  7. 7.
    Chatterjee, K., Sen, K., Henzinger, T.A.: Model-checking \(\omega \)-regular properties of interval Markov chains. In: Amadio, R. (ed.) FoSSaCS 2008. LNCS, vol. 4962, pp. 302–317. Springer, Heidelberg (2008). doi: 10.1007/978-3-540-78499-9_22 CrossRefGoogle Scholar
  8. 8.
    Chehaibar, G., Garavel, H., Mounier, L., Tawbi, N., Zulian, F.: Specification, verification of the PowerScale® bus arbitration protocol: An industrial experiment with LOTOS. In: FORTE, pp. 435–450 (1996)Google Scholar
  9. 9.
    Coste, N., Hermanns, H., Lantreibecq, E., Serwe, W.: Towards performance prediction of compositional models in industrial GALS designs. In: Bouajjani, A., Maler, O. (eds.) CAV 2009. LNCS, vol. 5643, pp. 204–218. Springer, Heidelberg (2009). doi: 10.1007/978-3-642-02658-4_18 CrossRefGoogle Scholar
  10. 10.
    Delahaye, B., Katoen, J.-P., Larsen, K.G., Legay, A., Pedersen, M.L., Sher, F., Wąsowski, A.: Abstract probabilistic automata. In: Jhala, R., Schmidt, D. (eds.) VMCAI 2011. LNCS, vol. 6538, pp. 324–339. Springer, Heidelberg (2011). doi: 10.1007/978-3-642-18275-4_23 CrossRefGoogle Scholar
  11. 11.
    Delahaye, B., Katoen, J.-P., Larsen, K.G., Legay, A., Pedersen, M.L., Sher, F., Wasowski, A.: New results on abstract probabilistic automata. In: ACSD, pp. 118–127 (2011)Google Scholar
  12. 12.
    Delahaye, B., Larsen, K.G., Legay, A., Pedersen, M.L., Wąsowski, A.: Decision problems for interval Markov Chains. In: Dediu, A.-H., Inenaga, S., Martín-Vide, C. (eds.) LATA 2011. LNCS, vol. 6638, pp. 274–285. Springer, Heidelberg (2011). doi: 10.1007/978-3-642-21254-3_21 CrossRefGoogle Scholar
  13. 13.
    Fecher, H., Leucker, M., Wolf, V.: Don’t Know in probabilistic systems. In: Valmari, A. (ed.) SPIN 2006. LNCS, vol. 3925, pp. 71–88. Springer, Heidelberg (2006). doi: 10.1007/11691617_5 CrossRefGoogle Scholar
  14. 14.
    Fischer, M.J., Lynch, N.A., Paterson, M.S.: Impossibility of distributed consensus with one faulty process. J. ACM 32(2), 374–382 (1985)MathSciNetCrossRefzbMATHGoogle Scholar
  15. 15.
    Gebler, D., Hashemi, V., Turrini, A.: Computing behavioral relations for probabilistic concurrent systems. In: Remke, A., Stoelinga, M. (eds.) Stochastic Model Checking. Rigorous Dependability Analysis Using Model Checking Techniques for Stochastic Systems. LNCS, vol. 8453, pp. 117–155. Springer, Heidelberg (2014). doi: 10.1007/978-3-662-45489-3_5 CrossRefGoogle Scholar
  16. 16.
    Givan, R., Leach, S.M., Dean, T.L.: Bounded-parameter Markov decision processes. Artif. Intell. 122(1–2), 71–109 (2000)MathSciNetCrossRefzbMATHGoogle Scholar
  17. 17.
    Hahn, E.M., Han, T., Zhang, L.: Synthesis for PCTL in parametric markov decision processes. In: Bobaru, M., Havelund, K., Holzmann, G.J., Joshi, R. (eds.) NFM 2011. LNCS, vol. 6617, pp. 146–161. Springer, Heidelberg (2011). doi: 10.1007/978-3-642-20398-5_12 CrossRefGoogle Scholar
  18. 18.
    Hahn, E.M., Hashemi, V., Hermanns, H., Turrini, A.: Exploiting robust optimization for interval probabilistic bisimulation. In: Agha, G., Van Houdt, B. (eds.) QEST 2016. LNCS, vol. 9826, pp. 55–71. Springer, Cham (2016). doi: 10.1007/978-3-319-43425-4_4 CrossRefGoogle Scholar
  19. 19.
    Hansson, H., Jonsson, B.: A logic for reasoning about time and reliability. Formal Asp. Comput. 6(5), 512–535 (1994)CrossRefzbMATHGoogle Scholar
  20. 20.
    Hashemi, V., Hatefi, H., Krčál, J.: Probabilistic bisimulations for PCTL model checking of interval MDPs. In: SynCoP, pp. 19–33. EPTCS (2014)Google Scholar
  21. 21.
    Hashemi, V., Hermanns, H., Song, L., Subramani, K., Turrini, A., Wojciechowski, P.: Compositional bisimulation minimization for interval Markov decision processes. In: Dediu, A.-H., Janoušek, J., Martín-Vide, C., Truthe, B. (eds.) LATA 2016. LNCS, vol. 9618, pp. 114–126. Springer, Cham (2016). doi: 10.1007/978-3-319-30000-9_9 CrossRefGoogle Scholar
  22. 22.
    Hashemi, V., Hermanns, H., Turrini, A.: Compositional reasoning for interval Markov decision processes.
  23. 23.
    Hermanns, H., Katoen, J.-P.: Automated compositional Markov chain generation for a plain-old telephone system. SCP 36(1), 97–127 (2000)zbMATHGoogle Scholar
  24. 24.
    Iyengar, G.N.: Robust dynamic programming. Math. Oper. Res. 30(2), 257–280 (2005)MathSciNetCrossRefzbMATHGoogle Scholar
  25. 25.
    Jonsson, B., Larsen, K.G.: Specification and refinement of probabilistic processes. In: LICS, pp. 266–277 (1991)Google Scholar
  26. 26.
    Kanellakis, P.C., Smolka, S.A.: CCS expressions, finite state processes, and three problems of equivalence. I&C, pp. 43–68 (1990)Google Scholar
  27. 27.
    Katoen, J.-P., Kemna, T., Zapreev, I., Jansen, D.N.: Bisimulation minimisation mostly speeds up probabilistic model checking. In: Grumberg, O., Huth, M. (eds.) TACAS 2007. LNCS, vol. 4424, pp. 87–101. Springer, Heidelberg (2007). doi: 10.1007/978-3-540-71209-1_9 CrossRefGoogle Scholar
  28. 28.
    Kozine, I., Utkin, L.V.: Interval-valued finite Markov chains. Reliable Comput. 8(2), 97–113 (2002)MathSciNetCrossRefzbMATHGoogle Scholar
  29. 29.
    Kwiatkowska, M., Norman, G., Parker, D.: PRISM 4.0: verification of probabilistic real-time systems. In: Gopalakrishnan, G., Qadeer, S. (eds.) CAV 2011. LNCS, vol. 6806, pp. 585–591. Springer, Heidelberg (2011). doi: 10.1007/978-3-642-22110-1_47 CrossRefGoogle Scholar
  30. 30.
    Lahijanian, M., Andersson, S.B., Belta, C.: Formal verification and synthesis for discrete-time stochastic systems. IEEE Tr. Autom. Contr. 60(8), 2031–2045 (2015)MathSciNetCrossRefzbMATHGoogle Scholar
  31. 31.
    Larsen, K.G., Skou, A.: Bisimulation through probabilistic testing (preliminary report). In: POPL, pp. 344–352 (1989)Google Scholar
  32. 32.
    Luna, R., Lahijanian, M., Moll, M., Kavraki, L.E.: Asymptotically optimal stochastic motion planning with temporal goals. In: Akin, H.L., Amato, N.M., Isler, V., van der Stappen, A.F. (eds.) Algorithmic Foundations of Robotics XI. STAR, vol. 107, pp. 335–352. Springer, Cham (2015). doi: 10.1007/978-3-319-16595-0_20 Google Scholar
  33. 33.
    Luna, R., Lahijanian, M., Moll, M., Kavraki, L.E.: Fast stochastic motion planning with optimality guarantees using local policy reconfiguration. In: ICRA, pp. 3013–3019 (2014)Google Scholar
  34. 34.
    Luna, R., Lahijanian, M., Moll, M., Kavraki, L.E.: Optimal and efficient stochastic motion planning in partially-known environments. In: AAAI, pp. 2549–2555 (2014)Google Scholar
  35. 35.
    Paige, R., Tarjan, R.E.: Three partition refinement algorithms. SIAM J. Comput. 16(6), 973–989 (1987)MathSciNetCrossRefzbMATHGoogle Scholar
  36. 36.
    PRISM model checker.
  37. 37.
    Puggelli, A.: Formal Techniques for the Verification and Optimal Control of Probabilistic Systems in the Presence of Modeling Uncertainties. Ph.D. thesis, EECS Department, University of California, Berkeley (2014)Google Scholar
  38. 38.
    Puggelli, A., Li, W., Sangiovanni-Vincentelli, A.L., Seshia, S.A.: Polynomial-time verification of PCTL properties of MDPs with convex uncertainties. In: Sharygina, N., Veith, H. (eds.) CAV 2013. LNCS, vol. 8044, pp. 527–542. Springer, Heidelberg (2013). doi: 10.1007/978-3-642-39799-8_35 CrossRefGoogle Scholar
  39. 39.
    Schrijver, A.: Theory of Linear and Integer Programming. Wiley, New York (1998)zbMATHGoogle Scholar
  40. 40.
    Segala, R.: Modeling and Verification of Randomized Distributed Real-Time Systems. Ph.D. thesis, MIT (1995)Google Scholar
  41. 41.
    Segala, R.: Probability and nondeterminism in operational models of concurrency. In: Baier, C., Hermanns, H. (eds.) CONCUR 2006. LNCS, vol. 4137, pp. 64–78. Springer, Heidelberg (2006). doi: 10.1007/11817949_5 CrossRefGoogle Scholar
  42. 42.
    Sen, K., Viswanathan, M., Agha, G.: Model-checking markov chains in the presence of uncertainties. In: Hermanns, H., Palsberg, J. (eds.) TACAS 2006. LNCS, vol. 3920, pp. 394–410. Springer, Heidelberg (2006). doi: 10.1007/11691372_26 CrossRefGoogle Scholar
  43. 43.
    Wolff, E.M., Topcu, U., Murray, R.M.: Robust control of uncertain Markov decision processes with temporal logic specifications. In: CDC, pp. 3372–3379 (2012)Google Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  • Vahid Hashemi
    • 1
    Email author
  • Andrea Turrini
    • 2
  • Ernst Moritz Hahn
    • 1
    • 2
  • Holger Hermanns
    • 1
  • Khaled Elbassioni
    • 3
  1. 1.Saarland UniversitySaarbrückenGermany
  2. 2.State Key Laboratory of Computer Science, ISCASBeijingChina
  3. 3.Masdar Institute of Science and TechnologyAbu DhabiUAE

Personalised recommendations