Skip to main content
SpringerLink
Log in

On Correctness, Precision, and Performance in Quantitative Verification

QComp 2020 Competition Report

  • Conference paper
  • First Online:
Leveraging Applications of Formal Methods, Verification and Validation: Tools and Trends (ISoLA 2020)

Part of the book series: Lecture Notes in Computer Science ((LNTCS,volume 12479))

Included in the following conference series:

Abstract

Quantitative verification tools compute probabilities, expected rewards, or steady-state values for formal models of stochastic and timed systems. Exact results often cannot be obtained efficiently, so most tools use floating-point arithmetic in iterative algorithms that approximate the quantity of interest. Correctness is thus defined by the desired precision and determines performance. In this paper, we report on the experimental evaluation of these trade-offs performed in QComp 2020: the second friendly competition of tools for the analysis of quantitative formal models. We survey the precision guarantees—ranging from exact rational results to statistical confidence statements—offered by the nine participating tools. They gave rise to a performance evaluation using five tracks with varying correctness criteria, of which we present the results.

The authors are listed alphabetically. This work was supported by DFG grant 389792660 as part of TRR 248 (CPEC), DFG grant 383882557 (SUV), ERC Advanced Grant 787914 (FRAPPANT), ERC Advanced Grant 834115 (FUN2MODEL), ERC Advanced Grant 695614 (POWVER), the Guangdong Science and Technology Department (grant no. 2018B010107004), the National Natural Science Foundation of China (grant nos. 61761136011, 61532019, 61836005), National Science Foundation grant CCF-1856733, NWO project 15474 (SEQUOIA), and NWO VENI grant no. 639.021.754.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Data Availability.

The tools used and data generated in the performance evaluation are archived at qcomp.org and DOI 10.5281/zenodo.3965313 [57].

Notes

  1. 1.

    Storm, on the other hand, is not compared with Storm-static, thus its “wins n” numbers, marked *, are not part of the same sum as those of the other tools.

References

  1. Abate, A., et al.: ARCH-COMP19 category report: stochastic modelling. In: ARCH. EPiC Series in Computing, vol. 61, pp. 62–102. EasyChair (2019). https://doi.org/10.29007/f2vb

  2. Agha, G., Palmskog, K.: A survey of statistical model checking. ACM Trans. Model. Comput. Simul. 28(1), 6:1–6:39 (2018). https://doi.org/10.1145/3158668

  3. Alur, R., Dill, D.L.: A theory of timed automata. Theor. Comput. Sci. 126(2), 183–235 (1994). https://doi.org/10.1016/0304-3975(94)90010-8

    Article  MathSciNet  MATH  Google Scholar 

  4. Amparore, E.G., Balbo, G., Beccuti, M., Donatelli, S., Franceschinis, G.: 30 years of GreatSPN. In: Fiondella, L., Puliafito, A. (eds.) Principles of Performance and Reliability Modeling and Evaluation, pp. 227–254. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-30599-8_9

  5. Arnold, F., Belinfante, A., van der Berg, F., Guck, D., Stoelinga, M.: DFTCalc: a tool for efficient fault tree analysis. In: SAFECOMP. LNCS, vol. 8153, pp. 293–301. Springer (2013). https://doi.org/10.1007/978-3-642-40793-2_27

    Chapter  Google Scholar 

  6. Ashok, P., Butkova, Y., Hermanns, H., Kretínský, J.: Continuous-time Markov decisions based on partial exploration. In: ATVA. LNCS, vol. 11138, pp. 317–334. Springer (2018). https://doi.org/10.1007/978-3-030-01090-4_19

    Chapter  Google Scholar 

  7. Ashok, P., Chatterjee, K., Daca, P., Kretínský, J., Meggendorfer, T.: Value iteration for long-run average reward in Markov decision processes. In: CAV. LNCS, vol. 10426, pp. 201–221. Springer (2017). https://doi.org/10.1007/978-3-319-63387-9_10

    Chapter  Google Scholar 

  8. Ashok, P., Kretínský, J., Weininger, M.: PAC statistical model checking for Markov decision processes and stochastic games. In: CAV. LNCS, vol. 11561, pp. 497–519. Springer (2019). https://doi.org/10.1007/978-3-030-25540-4_29

    Chapter  Google Scholar 

  9. Baier, C., de Alfaro, L., Forejt, V., Kwiatkowska, M.: Model checking probabilistic systems. In: Handbook of Model Checking, pp. 963–999. Springer (2018). https://doi.org/10.1007/978-3-319-10575-8_28

    Chapter  MATH  Google Scholar 

  10. Baier, C., Katoen, J.P., Hermanns, H.: Approximate symbolic model checking of continuous-time Markov chains. In: CONCUR. LNCS, vol. 1664, pp. 146–161. Springer (1999). https://doi.org/10.1007/3-540-48320-9_12

    Chapter  Google Scholar 

  11. Baier, C., Klein, J., Leuschner, L., Parker, D., Wunderlich, S.: Ensuring the reliability of your model checker: interval iteration for MDPs. In: CAV. LNCS, vol. 10426, pp. 160–180. Springer (2017). https://doi.org/10.1007/978-3-319-63387-9_8

    Chapter  Google Scholar 

  12. Bauer, M.S., Mathur, U., Chadha, R., Sistla, A.P., Viswanathan, M.: Exact quantitative probabilistic model checking through rational search. In: FMCAD, pp. 92–99. IEEE (2017). https://doi.org/10.23919/FMCAD.2017.8102246

  13. Behrmann, G., et al.: UPPAAL 4.0. In: QEST, pp. 125–126. IEEE Computer Society (2006). https://doi.org/10.1109/QEST.2006.59

  14. Bonet, B., Geffner, H.: Labeled RTDP: improving the convergence of real-time dynamic programming. In: ICAPS, pp. 12–21. AAAI Press (2003)

    Google Scholar 

  15. Brázdil, T., Chatterjee, K., Chmelik, M., Forejt, V., Kretínský, J., Kwiatkowska, M.Z., Parker, D., Ujma, M.: Verification of Markov decision processes using learning algorithms. In: ATVA. LNCS, vol. 8837, pp. 98–114. Springer (2014). https://doi.org/10.1007/978-3-319-11936-6_8

    Chapter  Google Scholar 

  16. Budde, C.E., D’Argenio, P.R., Hartmanns, A.: Better automated importance splitting for transient rare events. In: SETTA. LNCS, vol. 10606, pp. 42–58. Springer (2017). https://doi.org/10.1007/978-3-319-69483-2_3

    Chapter  Google Scholar 

  17. Budde, C.E., D’Argenio, P.R., Hartmanns, A.: Automated compositional importance splitting. Sci. Comput. Program. 174, 90–108 (2019). DOI: 10.1016/j.scico.2019.01.006

    Article  Google Scholar 

  18. Budde, C.E., D’Argenio, P.R., Hartmanns, A., Sedwards, S.: An efficient statistical model checker for nondeterminism and rare events. STTT (2020, to appear)

    Google Scholar 

  19. Budde, C.E., Dehnert, C., Hahn, E.M., Hartmanns, A., Junges, S., Turrini, A.: JANI: Quantitative model and tool interaction. TACAS. LNCS 10206, 151–168 (2017). https://doi.org/10.1007/978-3-662-54580-5_9

    Chapter  Google Scholar 

  20. Butkova, Y., Fox, G.: Optimal time-bounded reachability analysis for concurrent systems. In: TACAS. LNCS, vol. 11428, pp. 191–208. Springer (2019). https://doi.org/10.1007/978-3-030-17465-1_11

    Chapter  Google Scholar 

  21. Butkova, Y., Hartmanns, A., Hermanns, H.: A Modest approach to modelling and checking Markov automata. In: QEST. LNCS, vol. 11785, pp. 52–69. Springer (2019). https://doi.org/10.1007/978-3-030-30281-8_4

    Chapter  Google Scholar 

  22. Butkova, Y., Hatefi, H., Hermanns, H., Krcál, J.: Optimal continuous time Markov decisions. In: ATVA. LNCS, vol. 9364, pp. 166–182. Springer (2015). https://doi.org/10.1007/978-3-319-24953-7_12

    Chapter  Google Scholar 

  23. Butkova, Y., Wimmer, R., Hermanns, H.: Long-run rewards for Markov automata. TACAS. LNCS 10206, 188–203 (2017). https://doi.org/10.1007/978-3-662-54580-5_11

    Chapter  MATH  Google Scholar 

  24. Ceska, M., Hensel, C., Junges, S., Katoen, J.P.: Counterexample-driven synthesis for probabilistic program sketches. In: FM. LNCS, vol. 11800, pp. 101–120. Springer (2019). https://doi.org/10.1007/978-3-030-30942-8_8

    Chapter  Google Scholar 

  25. Chen, T., Forejt, V., Kwiatkowska, M.Z., Parker, D., Simaitis, A.: Automatic verification of competitive stochastic systems. Formal Methods Syst. Des. 43(1), 61–92 (2013). DOI: 10.1007/s10703-013-0183-7

    Article  MATH  Google Scholar 

  26. Courtney, T., Gaonkar, S., Keefe, K., Rozier, E., Sanders, W.H.: Möbius 2.3: an extensible tool for dependability, security, and performance evaluation of large and complex system models. In: DSN, pp. 353–358. IEEE Computer Society (2009). https://doi.org/10.1109/DSN.2009.5270318

  27. D’Argenio, P.R., Hartmanns, A., Legay, A., Sedwards, S.: Statistical approximation of optimal schedulers for probabilistic timed automata. In: iFM. LNCS, vol. 9681, pp. 99–114. Springer (2016). https://doi.org/10.1007/978-3-319-33693-0_7

    Chapter  Google Scholar 

  28. D’Argenio, P.R., Hartmanns, A., Sedwards, S.: Lightweight statistical model checking in nondeterministic continuous time. In: ISoLA. LNCS, vol. 11245, pp. 336–353. Springer (2018). https://doi.org/10.1007/978-3-030-03421-4_22

    Chapter  Google Scholar 

  29. D’Argenio, P.R., Jeannet, B., Jensen, H.E., Larsen, K.G.: Reduction and refinement strategies for probabilistic analysis. In: PAPM-PROBMIV. LNCS, vol. 2399, pp. 57–76. Springer (2002). https://doi.org/10.1007/3-540-45605-8_5

    Chapter  MATH  Google Scholar 

  30. Dehnert, C., Jansen, N., Wimmer, R., Ábrahám, E., Katoen, J.P.: Fast debugging of PRISM models. In: ATVA. LNCS, vol. 8837, pp. 146–162. Springer (2014). https://doi.org/10.1007/978-3-319-11936-6_11

    Chapter  Google Scholar 

  31. Dehnert, C., et al.: PROPhESY: A PRObabilistic ParamEter SYnthesis tool. In: Kroening, D., Păsăreanu, C.S. (eds.) CAV 2015. LNCS, vol. 9206, pp. 214–231. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-21690-4_13

    Chapter  Google Scholar 

  32. Dehnert, C., Junges, S., Katoen, J.P., Volk, M.: A Storm is coming: a modern probabilistic model checker. In: CAV. LNCS, vol. 10427, pp. 592–600. Springer (2017). https://doi.org/10.1007/978-3-319-63390-9_31

  33. Delgrange, F., Katoen, J.P., Quatmann, T., Randour, M.: Simple strategies in multi-objective MDPs. In: TACAS. LNCS, vol. 12078, pp. 346–364. Springer (2020). https://doi.org/10.1007/978-3-030-45190-5_19

    Chapter  Google Scholar 

  34. van Dijk, T., Hahn, E.M., Jansen, D.N., Li, Y., Neele, T., Stoelinga, M., Turrini, A., Zhang, L.: A comparative study of BDD packages for probabilistic symbolic model checking. In: SETTA. LNCS, vol. 9409, pp. 35–51. Springer (2015). https://doi.org/10.1007/978-3-319-25942-0_3

    Chapter  Google Scholar 

  35. Eisentraut, C., Hermanns, H., Zhang, L.: On probabilistic automata in continuous time. In: LICS, pp. 342–351. IEEE Computer Society (2010). https://doi.org/10.1109/LICS.2010.41

  36. Etessami, K., Kwiatkowska, M.Z., Vardi, M.Y., Yannakakis, M.: Multi-objective model checking of Markov decision processes. Logic. Methods Comput. Sci. 4(4) (2008). https://doi.org/10.2168/LMCS-4(4:8)2008

  37. Feng, Y., Hahn, E.M., Turrini, A., Ying, S.: Model checking omega-regular properties for quantum Markov chains. In: CONCUR. LIPIcs, vol. 85, pp. 35:1–35:16. Schloss Dagstuhl - Leibniz-Zentrum für Informatik (2017). https://doi.org/10.4230/LIPIcs.CONCUR.2017.35

  38. Fu, C., Turrini, A., Huang, X., Song, L., Feng, Y., Zhang, L.: Model checking probabilistic epistemic logic for probabilistic multiagent systems. In: IJCAI, pp. 4757–4763. ijcai.org (2018). https://doi.org/10.24963/ijcai.2018/661

  39. Gainer, P., Hahn, E.M., Schewe, S.: Accelerated model checking of parametric Markov chains. In: ATVA. LNCS, vol. 11138, pp. 300–316. Springer (2018). https://doi.org/10.1007/978-3-030-01090-4_18

    Chapter  Google Scholar 

  40. Gordon, A.D., Henzinger, T.A., Nori, A.V., Rajamani, S.K.: Probabilistic programming. In: FOSE, pp. 167–181. ACM (2014). https://doi.org/10.1145/2593882.2593900

  41. Gros, T.P.: Markov automata taken by Storm. Master’s thesis, Saarland University, Germany (2018)

    Google Scholar 

  42. Guck, D., Hatefi, H., Hermanns, H., Katoen, J.P., Timmer, M.: Modelling, reduction and analysis of Markov automata. In: QEST. LNCS, vol. 8054, pp. 55–71. Springer (2013). https://doi.org/10.1007/978-3-642-40196-1_5

    Chapter  Google Scholar 

  43. Haddad, S., Monmege, B.: Reachability in MDPs: Refining convergence of value iteration. In: RP. LNCS, vol. 8762, pp. 125–137. Springer (2014). https://doi.org/10.1007/978-3-319-11439-2_10

    Chapter  MATH  Google Scholar 

  44. Haddad, S., Monmege, B.: Interval iteration algorithm for MDPs and IMDPs. Theor. Comput. Sci. 735, 111–131 (2018). https://doi.org/10.1016/j.tcs.2016.12.003

    Article  MathSciNet  MATH  Google Scholar 

  45. Hahn, E.M., Hartmanns, A.: A comparison of time- and reward-bounded probabilistic model checking techniques. SETTA. LNCS 9984, 85–100 (2016). https://doi.org/10.1007/978-3-319-47677-3_6

    Chapter  MATH  Google Scholar 

  46. Hahn, E.M., Hartmanns, A., Hensel, C., Klauck, M., Klein, J., Kretínský, J., Parker, D., Quatmann, T., Ruijters, E., Steinmetz, M.: The 2019 comparison of tools for the analysis of quantitative formal models (QComp 2019 competition report). In: TACAS: TOOLympics. LNCS, vol. 11429, pp. 69–92. Springer (2019). https://doi.org/10.1007/978-3-030-17502-3_5

    Chapter  Google Scholar 

  47. Hahn, E.M., Hartmanns, A., Hermanns, H., Katoen, J.P.: A compositional modelling and analysis framework for stochastic hybrid systems. Formal Methods Syst. Des. 43(2), 191–232 (2013). DOI: 10.1007/s10703-012-0167-z

    Article  MATH  Google Scholar 

  48. Hahn, E.M., Hashemi, V., Hermanns, H., Lahijanian, M., Turrini, A.: Multi-objective robust strategy synthesis for interval MDPs. In: QEST. LNCS, vol. 10503, pp. 207–223. Springer (2017). https://doi.org/10.1007/978-3-319-66335-7_13

    Chapter  Google Scholar 

  49. Hahn, Ernst Moritz, Hashemi, Vahid, Hermanns, Holger, Turrini, Andrea: Exploiting Robust Optimization for Interval Probabilistic Bisimulation. In: Agha, Gul, Van Houdt, Benny (eds.) QEST 2016. LNCS, vol. 9826, pp. 55–71. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-43425-4_4

    Chapter  Google Scholar 

  50. Hahn, E.M., Li, G., Schewe, S., Zhang, L.: Lazy determinisation for quantitative model checking. CoRR abs/1311.2928 (2013). arxiv.org/abs/1311.2928

  51. Hahn, E.M., Li, Y., Schewe, S., Turrini, A., Zhang, L.: iscasMc: a web-based probabilistic model checker. In: FM. LNCS, vol. 8442, pp. 312–317. Springer (2014). https://doi.org/10.1007/978-3-319-06410-9_22

    Chapter  Google Scholar 

  52. Hahn, E.M., Schewe, S., Turrini, A., Zhang, L.: A simple algorithm for solving qualitative probabilistic parity games. In: CAV. LNCS, vol. 9780, pp. 291–311. Springer (2016). https://doi.org/10.1007/978-3-319-41540-6_16

    Chapter  Google Scholar 

  53. Hartmanns, A., Hermanns, H.: The modest toolset: an integrated environment for quantitative modelling and verification. In: Ábrahám, E., Havelund, K. (eds.) TACAS 2014. LNCS, vol. 8413, pp. 593–598. Springer, Heidelberg (2014). https://doi.org/10.1007/978-3-642-54862-8_51

    Chapter  Google Scholar 

  54. Hartmanns, A., Hermanns, H.: Explicit model checking of very large MDP using partitioning and secondary storage. In: ATVA. LNCS, vol. 9364, pp. 131–147. Springer (2015). https://doi.org/10.1007/978-3-319-24953-7_10

    Chapter  MATH  Google Scholar 

  55. Hartmanns, A., Junges, S., Katoen, J.P., Quatmann, T.: Multi-cost bounded reachability in MDP. In: TACAS. LNCS, vol. 10806, pp. 320–339. Springer (2018). https://doi.org/10.1007/978-3-319-89963-3_19

    Chapter  MATH  Google Scholar 

  56. Hartmanns, A., Kaminski, B.L.: Optimistic value iteration. In: CAV. LNCS, vol. 12225, pp. 488–511. Springer (2020). https://doi.org/10.1007/978-3-030-53291-8_26

    Chapter  Google Scholar 

  57. Hartmanns, A., Klauck, M.: The 2020 Comparison of Tools for the Analysis of Quantitative Formal Models: Results and Reproduction. Zenodo (2020). https://doi.org/10.5281/zenodo.3965313

    Article  Google Scholar 

  58. Hartmanns, A., Klauck, M., Parker, D., Quatmann, T., Ruijters, E.: The quantitative verification benchmark set. In: TACAS. LNCS, vol. 11427, pp. 344–350. Springer (2019). https://doi.org/10.1007/978-3-030-17462-0_20

    Chapter  Google Scholar 

  59. Hartmanns, A., Sedwards, S., D’Argenio, P.R.: Efficient simulation-based verification of probabilistic timed automata. In: Winter Simulation Conference, pp. 1419–1430. IEEE (2017). https://doi.org/10.1109/WSC.2017.8247885

  60. Hensel, C., Junges, S., Katoen, J.P., Quatmann, T., Volk, M.: The probabilistic model checker Storm. CoRR abs/2002.07080 (2020). arxiv.org/abs/2002.07080

  61. Jansen, D.N.: Understanding Fox and Glynn’s “Computing Poisson probabilities”. CTIT technical report series (2011)

    Google Scholar 

  62. Junges, S., et al.: Parameter synthesis for Markov models. CoRR abs/1903.07993 (2019). arxiv.org/abs/1903.07993

  63. Kelmendi, E., Krämer, J., Kretínský, J., Weininger, M.: Value iteration for simple stochastic games: Stopping criterion and learning algorithm. In: CAV. LNCS, vol. 10981, pp. 623–642. Springer (2018). https://doi.org/10.1007/978-3-319-96145-3_36

    Chapter  Google Scholar 

  64. Klauck, M., Steinmetz, M., Hoffmann, J., Hermanns, H.: Compiling probabilistic model checking into prob. planning. In: ICAPS, pp. 150–154. AAAI Press (2018)

    Google Scholar 

  65. Klauck, M., Steinmetz, M., Hoffmann, J., Hermanns, H.: Bridging the gap between probabilistic model checking and probabilistic planning: Survey, compilations, and empirical comparison. J. Artif. Intell. Res. 68, 247–310 (2020). https://doi.org/10.1613/jair.1.11595

    Article  MathSciNet  MATH  Google Scholar 

  66. Kolobov, A., Mausam, Weld, D.S., Geffner, H.: Heuristic search for generalized stochastic shortest path MDPs. In: ICAPS. AAAI Press (2011)

    Google Scholar 

  67. Kwiatkowska, M.Z., Norman, G., Parker, D.: Stochastic games for verification of probabilistic timed automata. In: FORMATS. LNCS, vol. 5813, pp. 212–227. Springer (2009). https://doi.org/10.1007/978-3-642-04368-0_17

    Chapter  MATH  Google Scholar 

  68. Kwiatkowska, M.Z., Norman, G., Parker, D.: PRISM 4.0: Verification of probabilistic real-time systems. In: CAV. LNCS, vol. 6806, pp. 585–591. Springer (2011). https://doi.org/10.1007/978-3-642-22110-1_47

    Chapter  Google Scholar 

  69. Kwiatkowska, M.Z., Norman, G., Parker, D.: The PRISM benchmark suite. In: QEST, pp. 203–204. IEEE Computer Society (2012). https://doi.org/10.1109/QEST.2012.14

  70. Kwiatkowska, M.Z., Norman, G., Parker, D., Sproston, J.: Performance analysis of probabilistic timed automata using digital clocks. Formal Methods Syst. Des. 29(1), 33–78 (2006). DOI: 10.1007/s10703-006-0005-2

    Article  MATH  Google Scholar 

  71. Kwiatkowska, M.Z., Norman, G., Segala, R., Sproston, J.: Automatic verification of real-time systems with discrete probability distributions. Theor. Comput. Sci. 282(1), 101–150 (2002). https://doi.org/10.1016/S0304-3975(01)00046-9

    Article  MathSciNet  MATH  Google Scholar 

  72. Legay, A., Sedwards, S., Traonouez, L.M.: Scalable verification of Markov decision processes. In: WS-FMDS at SEFM. LNCS, vol. 8938, pp. 350–362. Springer (2014). https://doi.org/10.1007/978-3-319-15201-1_23

    Chapter  Google Scholar 

  73. Lewis, E., Böhm, F.: Monte Carlo simulation of Markov unreliability models. Nucl. Eng. Design 77(1), 49–62 (1984). https://doi.org/10.1016/0029-5493(84)90060-8

    Article  Google Scholar 

  74. Li, Y., Liu, W., Turrini, A., Hahn, E.M., Zhang, L.: An efficient synthesis algorithm for parametric Markov chains against linear time properties. CoRR abs/1605.04400 (2016)

    Google Scholar 

  75. de Moura, L.M., Bjørner, N.: Z3: An efficient SMT solver. In: TACAS. LNCS, vol. 4963, pp. 337–340. Springer (2008). https://doi.org/10.1007/978-3-540-78800-3_24

    Chapter  Google Scholar 

  76. Neupane, T., Myers, C.J., Madsen, C., Zheng, H., Zhang, Z.: STAMINA: stochastic approximate model-checker for infinite-state analysis. In: Dillig, I., Tasiran, S. (eds.) CAV 2019. LNCS, vol. 11561, pp. 540–549. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-25540-4_31

    Chapter  Google Scholar 

  77. Neupane, T., Zhang, Z., Madsen, C., Zheng, H., Myers, C.J.: Approximation techniques for stochastic analysis of biological systems. In: Automated Reasoning for Systems Biology and Medicine, Computational Biology, vol. 30, pp. 327–348. Springer (2019). https://doi.org/10.1007/978-3-030-17297-8_12

    Chapter  Google Scholar 

  78. Puterman, M.L.: Markov Decision Processes: Discrete Stochastic Dynamic Programming. Wiley Series in Probability and Statistics, Wiley (1994). https://doi.org/10.1002/9780470316887

  79. Quatmann, T., Junges, S., Katoen, J.P.: Markov automata with multiple objectives. In: CAV. LNCS, vol. 10426, pp. 140–159. Springer (2017). https://doi.org/10.1007/978-3-319-63387-9_7

    Chapter  Google Scholar 

  80. Quatmann, T., Katoen, J.P.: Sound value iteration. In: CAV. LNCS, vol. 10981, pp. 643–661. Springer (2018). https://doi.org/10.1007/978-3-319-96145-3_37

    Chapter  Google Scholar 

  81. Reijsbergen, D., de Boer, P.T., Scheinhardt, W.R.W., Juneja, S.: Path-ZVA: general, efficient, and automated importance sampling for highly reliable Markovian systems. ACM Trans. Model. Comput. Simul. 28(3), 22:1–22:25 (2018). https://doi.org/10.1145/3161569

  82. Ruijters, E., et al.: FFORT: a benchmark suite for fault tree analysis. In: ESREL (2019). https://doi.org/10.3850/978-981-11-2724-3_0641-cd

  83. Ruijters, E., Reijsbergen, D., de Boer, P.T., Stoelinga, M.: Rare event simulation for dynamic fault trees. Reliab. Eng. Syst. Saf. 186, 220–231 (2019). DOI: 10.1016/j.ress.2019.02.004

    Article  Google Scholar 

  84. Spel, J., Junges, S., Katoen, J.P.: Are parametric Markov chains monotonic? In: ATVA. LNCS, vol. 11781, pp. 479–496. Springer (2019). https://doi.org/10.1007/978-3-030-31784-3_28

    Chapter  Google Scholar 

  85. Steinmetz, M., Hoffmann, J., Buffet, O.: Goal probability analysis in probabilistic planning: Exploring and enhancing the state of the art. J. Artif. Intell. Res. 57, 229–271 (2016). https://doi.org/10.1613/jair.5153

    Article  MathSciNet  MATH  Google Scholar 

  86. Sullivan, K.J., Dugan, J.B., Coppit, D.: The Galileo fault tree analysis tool. In: FTCS, pp. 232–235. IEEE Computer Society (1999). https://doi.org/10.1109/FTCS.1999.781056

  87. Volk, M., Junges, S., Katoen, J.P.: Fast dynamic fault tree analysis by model checking techniques. IEEE Trans. Ind. Informatics 14(1), 370–379 (2018). DOI: 10.1109/TII.2017.2710316

    Article  Google Scholar 

  88. Younes, H.L.S., Kwiatkowska, M.Z., Norman, G., Parker, D.: Numerical vs. statistical probabilistic model checking. Int. J. Softw. Tools Technol. Transf. 8(3), 216–228 (2006). https://doi.org/10.1007/s10009-005-0187-8

  89. Younes, H.L.S., Littman, M.L., Weissman, D., Asmuth, J.: The first probabilistic track of the International Planning Competition. J. Artif. Intell. Res. 24, 851–887 (2005). DOI: 10.1613/jair.1880

    Article  MATH  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. University of Twente, Enschede, The Netherlands

    Carlos E. Budde & Arnd Hartmanns

  2. Saarland University, Saarland Informatics Campus, Saarbrücken, Germany

    Michaela Klauck

  3. Technical University of Munich, Munich, Germany

    Jan Křetínský

  4. University of Birmingham, Birmingham, UK

    David Parker

  5. RWTH Aachen University, Aachen, Germany

    Tim Quatmann

  6. State Key Laboratory of Computer Science, Institute of Software, Chinese Academy of Sciences, Beijing, China

    Andrea Turrini

  7. Institute of Intelligent Software, Guangzhou, Guangzhou, China

    Andrea Turrini

  8. Utah State University, Logan, UT, USA

    Zhen Zhang

Authors
  1. Carlos E. Budde
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Arnd Hartmanns
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Michaela Klauck
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Jan Křetínský
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. David Parker
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Tim Quatmann
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Andrea Turrini
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Zhen Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Arnd Hartmanns .

Editor information

Editors and Affiliations

  1. University of Limerick and Lero, Limerick, Ireland

    Tiziana Margaria

  2. TU Dortmund, Dortmund, Germany

    Bernhard Steffen

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Budde, C.E. et al. (2021). On Correctness, Precision, and Performance in Quantitative Verification. In: Margaria, T., Steffen, B. (eds) Leveraging Applications of Formal Methods, Verification and Validation: Tools and Trends. ISoLA 2020. Lecture Notes in Computer Science(), vol 12479. Springer, Cham. https://doi.org/10.1007/978-3-030-83723-5_15

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-83723-5_15

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-83722-8

  • Online ISBN: 978-3-030-83723-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation