On hypothesis testing for statistical model checking

  • Daniël Reijsbergen
  • Pieter-Tjerk de Boer
  • Werner Scheinhardt
  • Boudewijn Haverkort


Hypothesis testing is an important part of statistical model checking (SMC). It is typically used to verify statements of the form \(p > p_0\) or \(p < p_0\), where \(p\) is an unknown probability intrinsic to the system model and \(p_0\) is a given threshold value. Many techniques for this have been introduced in the SMC literature. We give a comprehensive overview and comparison of these techniques, starting by introducing a framework in which they can all be described. We distinguish between three classes of techniques, differing in what type of output correctness guarantees they give when the true \(p\) is very close to the threshold \(p_0\). For each technique, we show how to parametrise it in terms of quantities that are meaningful to the user. Having parametrised them consistently, we graphically compare the boundaries of their decision thresholds, and numerically compare the correctness, power and efficiency of the tests. A companion website allows users to get more insight in the properties of the tests by interactively manipulating the parameters.


Statistical model checking Hypothesis testing Probabilistic verification Survey 



This work is partially supported by the Netherlands Organisation for Scientific Research (NWO), project number 612.064.812, and by the EU project QUANTICOL, 600708.


  1. 1.
  2. 2.
    Aziz, A., Sanwal, K., Singhal, V., Brayton, R.: Model-checking continuous-time Markov chains. ACM TOCL 1(1), 162–170 (2000)MathSciNetCrossRefGoogle Scholar
  3. 3.
    Aziz, A., Sanwal, K., Singhal, V., Brayton, R.K.: Verifying continuous-time Markov chains. Lect Notes Comput Sci 1102, 269–276 (1996)CrossRefGoogle Scholar
  4. 4.
    Baier, C., Haverkort, B.R., Hermanns, H., Katoen, J.P.: On the logical characterisation of performability properties. In Automata, Languages and Programming, pages 780–792. LNCS Volume 1853, Springer, 2000Google Scholar
  5. 5.
    Baier, C., Haverkort, B.R., Hermanns, H., Katoen, J.P.: Model-checking algorithms for continuous-time Markov chains. IEEE Trans. Softw. Eng. 29(6), 524–541 (2003)CrossRefGoogle Scholar
  6. 6.
    Baier, C., Katoen, J.P.: Principles of Model Checking. MIT press, Cambridge (2008)zbMATHGoogle Scholar
  7. 7.
    Ballarini, P., Djafri, H., Duflot, M., Haddad, S., Pekergin, N.: COSMOS: a statistical model checker for the hybrid automata stochastic logic. In: Proceedings of the Eighth International Conference on the Quantitative Evaluation of Systems (QEST), pp. 143–144. IEEE (2011)Google Scholar
  8. 8.
    Bengtsson, J., Larsen, K., Larsson, F., Pettersson, P., Yi, W.: UPPAAL—a tool suite for automatic verification of real-time systems. Hybrid Syst. III, 232–243 (1996)Google Scholar
  9. 9.
    Chong, E., Żak, S.: An Introduction to Optimization. Wiley, Hoboken (2004)Google Scholar
  10. 10.
    Chow, Y.S., Robbins, H.: On the asymptotic theory of fixed-width sequential confidence intervals for the mean. Ann. Math. Stat. 36(2), 457–462 (1965)MathSciNetCrossRefzbMATHGoogle Scholar
  11. 11.
    Darling, D.A., Robbins, H.: Some nonparametric sequential tests with power one. Proc. Nat. Acad. Sci. USA 61(3), 804–809 (1968)MathSciNetCrossRefzbMATHGoogle Scholar
  12. 12.
    El Rabih, D., Pekergin, N.: Statistical model checking using perfect simulation. In: Automated Technology for Verification and Analysis, pp. 120–134. LNCS Volume 5799, Springer (2009)Google Scholar
  13. 13.
    Fishman, G.S.: Discrete-Event Simulation: Modeling, Programming, and Analysis. Springer, Berlin (2001)CrossRefGoogle Scholar
  14. 14.
    Glynn, P.W.: A GSMP formalism for discrete event systems. Proc. IEEE 77(1), 14–23 (1989)CrossRefGoogle Scholar
  15. 15.
    Grubbs, F.E.: On designing single sampling inspection plans. The Ann. Math. Stat., pp. 242–256 (1949)Google Scholar
  16. 16.
    Haas, P.J., Shedler, G.S.: Stochastic Petri net representation of discrete event simulations. IEEE Trans. Softw. Eng. 15(4), 381–393 (1989)MathSciNetCrossRefzbMATHGoogle Scholar
  17. 17.
    Hansson, H., Jonsson, B.: A logic for reasoning about time and reliability. Formal aspects of computing 6(5), 512–535 (1994)CrossRefzbMATHGoogle Scholar
  18. 18.
    Hérault, T., Lassaigne, R., Magniette, F., Peyronnet, S.: Approximate probabilistic model checking. Lect. Notes Comput. Sci. 2937, 307–329 (2004)Google Scholar
  19. 19.
    Hérault, T., Lassaigne, R., Peyronnet, S.: APMC 3.0: Approximate verification of discrete and continuous time markov chains. In: Proceedings of the Third International Conference on the Quantitative Evaluation of Systems (QEST), pp. 129–130. IEEE (2006)Google Scholar
  20. 20.
    Hoeffding, W.: Probability inequalities for sums of bounded random variables. J. Am. Stat. Assoc. pp. 13–30 (1963)Google Scholar
  21. 21.
    Jeffreys, H.: Theory of Probability. Oxford University Press, Oxford (1961)zbMATHGoogle Scholar
  22. 22.
    Jegourel, C., Legay, A., Sedwards, S.: A platform for high performance statistical model checking—PLASMA. Tools and Algorithms for the Construction and Analysis of Systems, pp. 498–503 (2012)Google Scholar
  23. 23.
    Jha, S., Clarke, E., Langmead, C., Legay, A., Platzer, A., Zuliani, P.: A Bayesian approach to model checking biological systems. In: Computational Methods in Systems Biology, pp. 218–234. Springer (2009)Google Scholar
  24. 24.
    Katoen, J. P., Khattri, M., Zapreev, I.S.: A Markov reward model checker. In: Second International Conference on the Quantitative Evaluation of Systems (QEST), pp. 243–244. IEEE (2005)Google Scholar
  25. 25.
    Kwiatkowska, M., Norman, G., Parker, D.: PRISM: Probabilistic symbolic model checker. In: Computer Performance Evaluation: Modelling Techniques and Tools, pp. 113–140. LNCS Volume 2324, Springer (2002)Google Scholar
  26. 26.
    Lai, T.L.: Nearly optimal sequential tests of composite hypotheses. Ann. Stat. pp. 856–886 (1988)Google Scholar
  27. 27.
    Legay, A., Delahaye, B., Bensalem, S.: Statistical model checking: an overview. In: Runtime Verification, pp. 122–135. Springer (2010)Google Scholar
  28. 28.
    Matthes, K.: Zur Theorie der Bedienungsprozesse. In: Proceedings of the Third Prague Conference on Information Theory, Statistical Decision Functions and Random Processes, pp. 513–528. Publishing House of the Czechoslovak Academy of Sciences (1962)Google Scholar
  29. 29.
    Reijsbergen, D.: Efficient simulation techniques for stochastic model checking. PhD thesis, University of Twente, Enschede (2013)Google Scholar
  30. 30.
    Reijsbergen, D., de Boer, P.T., Scheinhardt, W.: A sequential hypothesis test based on a generalized azuma inequality. ForthcomingGoogle Scholar
  31. 31.
    Ripley, B.D.: Stochastic Simulation. Wiley, Hoboken (1987)CrossRefzbMATHGoogle Scholar
  32. 32.
    Ross, S.M.: Stochastic Processes. Wiley, Hoboken (1996)zbMATHGoogle Scholar
  33. 33.
    Sebastio, S., Vandin, A.: MultiVeStA: Statistical model checking for discrete event simulators. In: Proceedings of the 7th International Conference on Performance Evaluation Methodologies and Tools (VALUETOOLS) (2013)Google Scholar
  34. 34.
    Sen, K., Viswanathan, M., Agha, G.: Statistical model checking of black-box probabilistic systems. In: Computer Aided Verification, pp. 202–215. LNCS Volume 3114, Springer (2004)Google Scholar
  35. 35.
    Sen, K., Viswanathan, M., Agha, G.: On statistical model checking of stochastic systems. In: Computer Aided Verification, pp. 266–280. LNCS Volume 3576, Springer (2005)Google Scholar
  36. 36.
    Sen, K., Viswanathan, M., Agha, G.: VeStA: A statistical model-checker and analyzer for probabilistic systems. In: Proceedings of the Second International Conference on the Quantitative Evaluation of Systems (QEST), pp. 251–252. IEEE (2005)Google Scholar
  37. 37.
    Wald, A.: Sequential tests of statistical hypotheses. Ann. Math. Stat. 16(2), 117–186 (1945)MathSciNetCrossRefzbMATHGoogle Scholar
  38. 38.
    Younes, H.L.S.: Verification and planning for stochastic processes with asynchronous events. PhD thesis, Carnegie Mellon (2005)Google Scholar
  39. 39.
    Younes, H.L.S.: Error control for probabilistic model checking. In Verification, Model Checking, and Abstract Interpretation, pp. 142–156. Springer (2006)Google Scholar
  40. 40.
    Younes, H.L.S., Clarke, E., Zuliani, P.: Statistical verification of probabilistic properties with unbounded until. Formal Methods: Foundations and Applications, pp. 144–160 (2011)Google Scholar
  41. 41.
    Younes, H.L.S., Kwiatkowska, M., Norman, G., Parker, D.: Numerical vs. statistical probabilistic model checking. Int. J. Softw. Tools Technol. Transf. (STTT) 8(3), 216–228 (2006)CrossRefGoogle Scholar
  42. 42.
    Younes, H.L.S., Simmons, R.G.: Probabilistic verification of discrete event systems using acceptance sampling. In: Computer Aided Verification, pp. 223–235. LNCS Volume 2404, Springer (2002)Google Scholar
  43. 43.
    Younes, H.L.S., Simmons, R.G.: Statistical probabilistic model checking with a focus on time-bounded properties. Inform. Comput. 204(9), 1368–1409 (2006)MathSciNetCrossRefzbMATHGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2014

Authors and Affiliations

  • Daniël Reijsbergen
    • 1
  • Pieter-Tjerk de Boer
    • 2
  • Werner Scheinhardt
    • 2
  • Boudewijn Haverkort
    • 2
  1. 1.University of EdinburghEdinburghUK
  2. 2.University of TwenteEnschedeThe Netherlands

Personalised recommendations