Advertisement

How Fast and Fat Is Your Probabilistic Model Checker? An Experimental Performance Comparison

  • David N. Jansen
  • Joost-Pieter Katoen
  • Marcel Oldenkamp
  • Mariëlle Stoelinga
  • Ivan Zapreev
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4899)

Abstract

This paper studies the efficiency of several probabilistic model checkers by comparing verification times and peak memory usage for a set of standard case studies. The study considers the model checkers ETMCC, MRMC, PRISM (sparse and hybrid mode), YMER and VESTA, and focuses on fully probabilistic systems. Several of our experiments show significantly different run times and memory consumptions between the tools—up to various orders of magnitude—without, however, indicating a clearly dominating tool. For statistical model checking YMER clearly prevails whereas for the numerical tools MRMC and PRISM (sparse) are rather close.

Keywords

Model Check Memory Usage Memory Consumption Statistical Model Check Reachability Property 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Agha, G., Meseguer, J., Sen, K.: PMaude: Rewrite-based specification language for probabilistic object systems. ENTCS 153(2), 213–239 (2006)Google Scholar
  2. 2.
    Andova, S., Hermanns, H., Katoen, J.-P.: Discrete-time rewards model-checked. In: Larsen, K.G., Niebert, P. (eds.) FORMATS 2003. LNCS, vol. 2791, pp. 88–104. Springer, Berlin (2004)Google Scholar
  3. 3.
    Aziz, A., Sanwal, K., Singhal, V., Brayton, R.K.: Verifying continuous time Markov chains. In: Alur, R., Henzinger, T.A. (eds.) CAV 1996. LNCS, vol. 1102, pp. 269–276. Springer, Heidelberg (1996)Google Scholar
  4. 4.
    Baier, C., Ciesinski, F., Größer, M.: ProbMela and verification of Markov decision processes. SIGMETRICS Perform. Eval. Rev. 32(4), 22–27 (2005)CrossRefGoogle Scholar
  5. 5.
    Baier, C., Haverkort, B., Hermanns, H., Katoen, J.-P.: Model-checking algorithms for continuous-time Markov chains. IEEE Trans. on Softw. Eng. 29(6), 524–541 (2003)CrossRefGoogle Scholar
  6. 6.
    Bode, E., Herbstritt, M., Hermanns, H., Johr, S., Peikenkamp, T., Pulungan, R., Wimmer, R., Becker, B.: Compositional Performability Evaluation for STATEMATE. In: Quantitative Evaluation of Systems: QEST, pp. 167–178. IEEE CS, Los Alamitos (2006)Google Scholar
  7. 7.
    Changuion, B., Davies, I., Nelte, M.: DaNAMiCS: A Petri net editor (2007), http://www.cs.uct.ac.za/Research/DNA/microweb/danamics/DNAFrameH.html
  8. 8.
    Cox, D.R.: A use of complex probabilities in the theory of stochastic processes. In: Proc. Cambridge Philosophical Society, vol. 51, pp. 313–319 (1955)Google Scholar
  9. 9.
    D’Aprile, D., Donatelli, S., Sproston, J.: CSL model checking for the GreatSPN tool. In: Aykanat, C., Dayar, T., Körpeoğlu, İ. (eds.) ISCIS 2004. LNCS, vol. 3280, pp. 543–552. Springer, Heidelberg (2004)Google Scholar
  10. 10.
    Dijkstra, E.W.: Hierarchical ordering of sequential processes. Acta Informatica 1, 115–138 (1971)CrossRefMathSciNetGoogle Scholar
  11. 11.
    Fokkink, W., Pang, J.: Simplifying Itai-Rodeh leader election for anonymous rings. ENTCS 128(6), 53–68 (2005)Google Scholar
  12. 12.
    Gupta, R., Smolka, S.A., Bhaskar, S.: On randomization in sequential and distributed algorithms. ACM Comput. Surv. 26(1), 7–86 (1994)CrossRefGoogle Scholar
  13. 13.
    Hansson, H., Jonsson, B.: A logic for reasoning about time and reliability. Formal Aspects of Computing 6(5), 512–535 (1994)zbMATHCrossRefGoogle Scholar
  14. 14.
    Hartonas-Garmhausen, V., Campos, S., Clarke, E.M.: ProbVerus: probabilistic symbolic model checking. In: Katoen, J.-P. (ed.) AMAST-ARTS 1999, ARTS 1999, and AMAST-WS 1999. LNCS, vol. 1601, pp. 96–110. Springer, Heidelberg (1999)CrossRefGoogle Scholar
  15. 15.
    Hermanns, H., Herzog, U., Klehmet, U., Mertsiotakis, V., Siegle, M.: Compositional performance modelling with the TIPPtool. Performance Evaluation 39(1–4), 5–35 (2000)zbMATHCrossRefGoogle Scholar
  16. 16.
    Hermanns, H., Katoen, J.-P., Meyer-Kayser, J., Siegle, M.: A Markov chain model checker. In: Schwartzbach, M.I., Graf, S. (eds.) ETAPS 2000 and TACAS 2000, LNCS, vol. 1785, pp. 347–362. Springer, Heidelberg (2000)CrossRefGoogle Scholar
  17. 17.
    Hermanns, H., Meyer-Kayser, J., Siegle, M.: Multi-terminal binary decision diagrams to represent and analyse continuous-time Markov chains. In: Plateau, B., Stewart, W.J., Silva, M. (eds.) Num. Sol. of Markov Chains, Zaragoza, Prensas Universitarias, pp. 188–207 (1999)Google Scholar
  18. 18.
    Hillston, J.: A Compositional Approach to Performance Modelling. Cambridge University Press, New York (1996)Google Scholar
  19. 19.
    Hogg, R.V., Craig, A.T.: Introduction to Mathematical Statistics, 4th edn. Macmillan, New York (1978)Google Scholar
  20. 20.
    Ibe, O.C., Trivedi, K.S.: Stochastic Petri net models of polling systems. IEEE J. on Sel. Areas in Comm. 8(9), 1649–1657 (1990)CrossRefGoogle Scholar
  21. 21.
    Itai, A., Rodeh, M.: Symmetry breaking in distributed networks. Inf. and Comp. 88(1), 60–87 (1990)zbMATHCrossRefMathSciNetGoogle Scholar
  22. 22.
    Karlin, S., McGregor, J.L.: The differential equations of birth-and-death processes, and the Stieltjes moment problem. Trans. of the AMS 85(2), 489–546 (1957)zbMATHCrossRefMathSciNetGoogle Scholar
  23. 23.
    Katoen, J.-P., Khattri, M., Zapreev, I.S.: A Markov reward model checker. In: Quantitative Evaluation of Systems, pp. 243–244. IEEE CS, Los Alamitos (2005)Google Scholar
  24. 24.
    Katoen, J.-P., Zapreev, I.S.: Safe on-the-fly steady-state detection for time-bounded reachability. In: Quantitative Evaluation of Systems: QEST, pp. 301–310. IEEE CS, Los Alamitos (2006)Google Scholar
  25. 25.
    Kwiatkowska, M., Norman, G., Parker, D.: PRISM: Probabilistic symbolic model checker. In: Field, T., et al. (eds.) TOOLS 2002. LNCS, vol. 2324, pp. 200–204. Springer, Heidelberg (2002)CrossRefGoogle Scholar
  26. 26.
    Kwiatkowska, M., Norman, G., Parker, D.: Stochastic model checking. In: Bernardo, M., Hillston, J. (eds.) Formal methods for performance evaluation. LNCS, vol. 4486, pp. 220–270. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  27. 27.
    Lassaigne, R., Peyronnet, S.: Approximate verification of probabilistic systems. In: Hermanns, H., Segala, R. (eds.) PROBMIV 2002, PAPM-PROBMIV 2002, and PAPM 2002. LNCS, vol. 2399, pp. 213–214. Springer, Heidelberg (2002)CrossRefGoogle Scholar
  28. 28.
    Lecca, P., Priami, C.: Cell cycle control in eukaryotes: A BioSpi model. Technical Report DIT-03-045, Informatica e Telecommunicazioni: University of Trento (2003)Google Scholar
  29. 29.
    Mohanty, S.G., Montazer-Haghighi, A., Trueblood, R.: On the transient behavior of a finite birth-death process with an application. Computers and Operations Research 20(3), 239–248 (1993)zbMATHCrossRefMathSciNetGoogle Scholar
  30. 30.
    Norman, G., Shmatikov, V.: Analysis of probabilistic contract signing. J. of Computer Security 14(6), 561–589 (2006)Google Scholar
  31. 31.
    Oldenkamp, M.: Probabilistic model checking: A comparison of tools. MSc thesis, Univ. of Twente, Netherlands (2007), http://www.cs.utwente.nl/~oldenkampha/
  32. 32.
    Penna, G.D., Intrigila, B., Melatti, I., Tronci, E., Zilli, M.V.: Finite horizon analysis of Markov chains with the Murphi verifier. STTT 8(4-5), 397–409 (2006)CrossRefGoogle Scholar
  33. 33.
    Pnueli, A., Zuck, L.D.: Verification of multiprocess probabilistic protocols. Distributed Comput. 1(1), 53–72 (1986)zbMATHCrossRefGoogle Scholar
  34. 34.
    PRISM: Probabilistic symbolic model checker (2006), http://www.cs.bham.ac.uk/~dxp/prism/
  35. 35.
    Sen, K., Viswanathan, M., Agha, G.: Statistical model checking of black-box probabilistic systems. In: Alur, R., Peled, D.A. (eds.) CAV 2004. LNCS, vol. 3114, pp. 202–215. Springer, Heidelberg (2004)Google Scholar
  36. 36.
    Sen, K., Viswanathan, M., Agha, G.: On statistical model checking of stochastic systems. In: Etessami, K., Rajamani, S.K. (eds.) CAV 2005. LNCS, vol. 3576, pp. 266–280. Springer, Heidelberg (2005)Google Scholar
  37. 37.
    Shedler, G.S.: Regenerative Stochastic Simulation. Academic Pr, London (1993)zbMATHGoogle Scholar
  38. 38.
    Somenzi, F.: CUDD: CU decision diagram package. Public software (1997), http://vlsi.colorado.edu/~fabio/CUDD/
  39. 39.
    Younes, H.L.S.: Ymer: A statistical model checker. In: Etessami, K., Rajamani, S.K. (eds.) CAV 2005. LNCS, vol. 3576, pp. 429–433. Springer, Heidelberg (2005)Google Scholar
  40. 40.
    Younes, H.L.S., Kwiatkowska, M., Norman, G., Parker, D.: Numerical vs. statistical probabilistic model checking. STTT 8(3), 216–228 (2006)CrossRefGoogle Scholar
  41. 41.
    Younes, H.L.S., Simmons, R.G.: Probabilistic verification of discrete event systems using acceptance sampling. In: Brinksma, E., Larsen, K.G. (eds.) CAV 2002. LNCS, vol. 2404, pp. 223–235. Springer, Heidelberg (2002)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2008

Authors and Affiliations

  • David N. Jansen
    • 1
    • 3
  • Joost-Pieter Katoen
    • 1
    • 2
  • Marcel Oldenkamp
    • 2
  • Mariëlle Stoelinga
    • 2
  • Ivan Zapreev
    • 1
    • 2
  1. 1.MOVES Group, RWTH, AachenGermany
  2. 2.FMT GroupUniversity of TwenteEnschedeThe Netherlands
  3. 3.ICISRadboud UniversityNijmegenThe Netherlands

Personalised recommendations