Skip to main content
SpringerLink
Log in

Sound statistical model checking for MDP using partial order and confluence reduction

  • SMC
  • Published:
International Journal on Software Tools for Technology Transfer Aims and scope Submit manuscript

Abstract

Statistical model checking (SMC) is an analysis method that circumvents the state space explosion problem in model-based verification by combining probabilistic simulation with statistical methods that provide clear error bounds. As a simulation-based technique, it can in general only provide sound results if the underlying model is a stochastic process. In verification, however, models are very often variations of nondeterministic transition systems. In classical exhaustive model checking, partial order reduction and confluence reduction allow the removal of spurious nondeterministic choices. In this paper, we show that both can be adapted to detect and discard such choices on-the-fly during simulation, thus extending the applicability of SMC to a subclass of Markov decision processes. We prove their correctness in a uniform way and study their effectiveness and efficiency using an implementation in the modes simulator for the Modest modelling language. The examples we use highlight the different strengths and limitations of the two approaches. We find that runtime may be affected by unnecessary recomputations, and thus also investigate the feasibility of caching results to speed up simulation at the cost of increased memory usage.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Notes

  1. Definitions 11, 12, 13, and 14 are all based on [5].

  2. We omit the \(\tau \)-loops that need to be added to deadlock states for brevity from now on.

    Fig. 3
    figure 3

    VMDP modelling hosts (\(H\)) that send on a shared medium (\(M\))

    Full size image

  3. We mostly cite [3] in the remainder of this section as it nicely summarises both linear-time approaches presented before [4, 12] in addition to introducing an extension to \(\text {PCTL*}_{\setminus \text {X}}\).

  4. By abuse of language, we use the word “transition” when we actually mean “equivalence class of transitions under \(\equiv \)”.

  5. Our implementation in modes therefore uses large default values for \(k_{\mathrm{max}}\) and \(l\) so the user usually need not worry about these parameters. If SMC aborts, the cause and its location is reported, including how it was detected, which may be that \(k_{\mathrm{max}}\) or \(l\) was exceeded.

  6. For the CSMA/CD models, the state space sizes reported are for the digital clocks semantics of the PTA created by mcpta and thus not directly comparable to the state spaces modes works on.

References

  1. Agresti, A., Coull, B.A.: Approximate is better than “exact” for interval estimation of binomial proportions. Am. Stat. 52(2), 119–126 (1998)

    MathSciNet  Google Scholar 

  2. Andel, T.R., Yasinsac, A.: On the credibility of MANET simulations. IEEE Comput. 39(7), 48–54 (2006)

    Article  Google Scholar 

  3. Baier, C., D’Argenio, P.R., Größer, M.: Partial order reduction for probabilistic branching time. Electron. Notes Theor. Comput. Sci. 153(2), 97–116 (2006)

    Article  Google Scholar 

  4. Baier, C., Größer, M., Ciesinski, F.: Partial order reduction for probabilistic systems. In: QEST, pp. 230–239. IEEE Computer Society, New York (2004)

  5. Baier, C., Katoen, J.-P.: Principles of Model Checking. MIT Press, Cambridge (2008)

    MATH  Google Scholar 

  6. Blom, S.: Partial \(\tau \)-confluence for efficient state space generation. Technical Report SEN-R0123, CWI (2001)

  7. Blom, S., van de Pol, J.: State space reduction by proving confluence. In: Brinksma, E., Larsen, K.G. (eds.) CAV. Lecture Notes in Computer Science, vol. 2404, pp. 596–609. Springer, Berlin (2002)

  8. Bogdoll, J., Fioriti, L.M.F., Hartmanns, A., Hermanns, H.: Partial order methods for statistical model checking and simulation. In: Bruni, R., Dingel, J. (eds.) FMOODS/FORTE. Lecture Notes in Computer Science, vol. 6722, pp. 59–74. Springer, Berlin (2011)

  9. Bogdoll, J., Hartmanns, A., Hermanns, H.: Simulation and statistical model checking for modestly nondeterministic models. In: Schmitt, J.B. (ed.) MMB/DFT. Lecture Notes in Computer Science, vol. 7201, pp. 249–252. Springer, Berlin (2012)

  10. Brázdil, T., Chatterjee, K., Chmelik, M., Forejt, V., Kretínský, J., Kwiatkowska, M.Z., Parker, D., Ujma, M.: Verification of Markov decision processes using learning algorithms. CoRR, abs/1402.2967 (2014)

  11. Chaum, D.: The dining cryptographers problem: unconditional sender and recipient untraceability. J. Cryptol. 1(1), 65–75 (1988)

    Article  MathSciNet  MATH  Google Scholar 

  12. D’Argenio, P.R., Niebert, P.: Partial order reduction on concurrent probabilistic programs. In: QEST, pp. 240–249. IEEE Computer Society, New York (2004)

  13. Evangelista, S., Pajault, C.: Solving the ignoring problem for partial order reduction. STTT 12(2), 155–170 (2010)

    Article  Google Scholar 

  14. Forejt, V., Kwiatkowska, M.Z., Norman, G., Parker, D.: Automated verification techniques for probabilistic systems. In: Bernardo, M., Issarny, V. (eds.) SFM. Lecture Notes in Computer Science, vol. 6659, pp. 53–113. Springer, Berlin (2011)

  15. Giro, S., D’Argenio, P.R., Fioriti, L.M.F.: Partial order reduction for probabilistic systems: a revision for distributed schedulers. In: Bravetti, M., Zavattaro, G. (eds.) CONCUR. Lecture Notes in Computer Science, vol. 5710, pp. 338–353. Springer, Berlin (2009)

  16. Godefroid, P.: Partial-order methods for the verification of concurrent systems—an approach to the state-explosion problem. Lecture Notes in Computer Science, vol. 1032. Springer, Berlin (1996)

  17. Groote, J.F., van de Pol, J.: State space reduction using partial tau-confluence. In: Nielsen, M., Rovan, B. (eds.) MFCS. Lecture Notes in Computer Science, vol. 1893, pp. 383–393. Springer, Berlin (2000)

  18. Hansen, H., Kwiatkowska, M.Z., Qu, H.: Partial order reduction for model checking Markov decision processes under unconditional fairness. In: QEST, pp. 203–212. IEEE Computer Society, New York (2011)

  19. Hansen, H., Timmer, M.: A comparison of confluence and ample sets in probabilistic and non-probabilistic branching time. Theor. Comput. Sci. 538C, 103–123 (2014)

    Article  MathSciNet  Google Scholar 

  20. Hartmanns, A., Hermanns, H.: A modest approach to checking probabilistic timed automata. In: QEST, pp. 187–196. IEEE Computer Society, New York (2009)

  21. Hartmanns, A., Hermanns, H.: The modest toolset: an integrated environment for quantitative modelling and verification. In: Ábrahám, E., Havelund, K. (eds.) TACAS. Lecture Notes in Computer Science, vol. 8413, pp. 593–598. Springer, Berlin (2014)

  22. Hartmanns, A., Timmer, M.: On-the-fly confluence detection for statistical model checking. In: Brat, G., Rungta, N., Venet, A. (eds.) NASA Formal Methods. Lecture Notes in Computer Science, vol. 7871, pp. 337–351. Springer, Berlin (2013)

  23. Henriques, D., Martins, J., Zuliani, P., Platzer, A., Clarke, E.M.: Statistical model checking for Markov decision processes. In: QEST, pp. 84–93. IEEE Computer Society, New York (2012)

  24. Hérault, T., Lassaigne, R., Magniette, F., Peyronnet, S.: Approximate probabilistic model checking. In: Steffen, B., Levi, G. (eds.) VMCAI. Lecture Notes in Computer Science, vol. 2937, pp. 73–84. Springer, Berlin (2004)

  25. Hoeffding, W.: Probability inequalities for sums of bounded random variables. J. Am. Stat. Assoc. 58(301), 13–30 (1963)

    Article  MathSciNet  MATH  Google Scholar 

  26. Kwiatkowska, M.Z., Norman, G., Parker, D.: PRISM 4.0: Verification of probabilistic real-time systems. In: Gopalakrishnan, G., Qadeer, S. (eds.) CAV. LNCS, vol. 6806, pp. 585–591. Springer, Berlin (2011)

  27. Lassaigne, R., Peyronnet, S.: Approximate planning and verification for large Markov decision processes. In: Ossowski, S., Lecca, P. (eds.) SAC, pp. 1314–1319. ACM, New York (2012)

  28. Legay, A., Delahaye, B., Bensalem, S.: Statistical model checking: an overview. In: Barringer, H., Falcone, Y., Finkbeiner, B., Havelund, K., Lee, I., Pace, G.J., Rosu, G., Sokolsky, O., Tillmann, N. (eds.) RV. Lecture Notes in Computer Science, vol. 6418, pp. 122–135. Springer, Berlin (2010)

  29. Legay, A., Sedwards, S.: Lightweight Monte Carlo algorithm for Markov decision processes. CoRR, abs/1310.3609 (2013)

  30. Mateescu, R., Wijs, A.: Sequential and distributed on-the-fly computation of weak tau-confluence. Sci. Comput. Program. 77(10–11), 1075–1094 (2012)

    Article  MATH  Google Scholar 

  31. Nimal, V.: Statistical approaches for probabilistic model checking. Master’s thesis, Oxford University (2010)

  32. Pace, G.J., Lang, F., Mateescu, R.: Calculating-confluence compositionally. In: Hunt Jr., W.A., Somenzi, F. (eds.) CAV. Lecture Notes in Computer Science, vol. 2725, pp. 446–459. Springer, Berlin (2003)

  33. Peled, D.: Combining partial order reductions with on-the-fly model-checking. In: Dill, D.L. (ed.) CAV. Lecture Notes in Computer Science, vol. 818, pp. 377–390. Springer, Berlin (1994)

  34. Peled, D.: Combining partial order reductions with on-the-fly model-checking. Formal Methods Syst. Des. 8(1), 39–64 (1996)

    Article  Google Scholar 

  35. Ross, S.M.: Simulation, 4th edn. Elsevier Academic Press, Amsterdam (2006)

    MATH  Google Scholar 

  36. Timmer, M.: Efficient Modelling, Generation and Analysis of Markov Automata. PhD thesis, University of Twente, The Netherlands (2013)

  37. Timmer, M., Stoelinga, M., van de Pol, J.: Confluence reduction for probabilistic systems. In: Abdulla, P.A., Rustan, K., Leino, M. (eds.) TACAS. Lecture Notes in Computer Science, vol. 6605, pp. 311–325. Springer, Berlin (2011)

  38. Timmer, M., van de Pol, J., Stoelinga, M.: Confluence reduction for Markov automata. In: Braberman, V.A., Fribourg, L. (eds.) FORMATS. Lecture Notes in Computer Science, vol. 8053, pp. 243–257. Springer, Berlin (2013)

  39. Valmari, A.: A stubborn attack on state explosion. In: Clarke, E.M., Kurshan, R.P. (eds.) CAV. Lecture Notes in Computer Science, vol. 531, pp. 156–165. Springer, Berlin (1990)

  40. Younes, H.L.S., Kwiatkowska, M.Z., Norman, G., Parker, D.: Numerical vs. statistical probabilistic model checking. STTT 8(3), 216–228 (2006)

    Article  Google Scholar 

  41. Younes, H.L.S., Simmons, R.G.: Probabilistic verification of discrete event systems using acceptance sampling. In: Brinksma, E., Larsen, K.G. (eds.) CAV. Lecture Notes in Computer Science, vol. 2404, pp. 223–235. Springer, Berlin (2002)

Download references

Acknowledgments

This work was supported by the Transregional Collaborative Research Centre SFB/TR 14 AVACS, by the NWO-DFG bilateral project ROCKS, by the 7th EU Framework Programme under grant agreements 295261 (MEALS) and 318490 (SENSATION), by NWO under grant 612.063.817 (SYRUP), and by the CAS/SAFEA International Partnership Program for Creative Research Teams.

Author information

Authors and Affiliations

  1. Saarland University, Saarbrücken, Germany

    Arnd Hartmanns

  2. University of Twente, Enschede, The Netherlands

    Mark Timmer

Authors
  1. Arnd Hartmanns
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mark Timmer
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Arnd Hartmanns.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Hartmanns, A., Timmer, M. Sound statistical model checking for MDP using partial order and confluence reduction. Int J Softw Tools Technol Transfer 17, 429–456 (2015). https://doi.org/10.1007/s10009-014-0349-7

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10009-014-0349-7

Keywords

Search

Navigation