Sequential Schemes for Frequentist Estimation of Properties in Statistical Model Checking

  • Cyrille Jegourel
  • Jun Sun
  • Jin Song Dong
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10503)


Statistical Model Checking (SMC) is an approximate verification method that overcomes the state space explosion problem for probabilistic systems by Monte Carlo simulations. Simulations might be however costly if many samples are required. It is thus necessary to implement efficient algorithms to reduce the sample size while preserving precision and accuracy. In the literature, some sequential schemes have been provided for the estimation of property occurrence based on predefined confidence and absolute or relative error. Nevertheless, these algorithms remain conservative and may result in huge sample sizes if the required precision standards are demanding. In this article, we compare some useful bounds and some sequential methods based on frequentist estimations. We propose outperforming and rigorous alternative schemes, based on Massart bounds and robust confidence intervals. Our theoretical and empirical analysis show that our proposal reduces the sample size while providing guarantees on error bounds.



Cyrille Jegourel and Jun Sun are partially supported by NRF grant GNRF1501 and Jin Song Dong by the project: Reliable Prototyping Framework for Daily Living Assistance of Frail Ageing People (RELIANCE).


  1. 1.
    Angluin, D., Valiant, L.: Fast probabilistic algorithms for Hamiltonian circuits and matchings. J. Comput. Syst. Sci. 18(2), 155–193 (1979)MathSciNetCrossRefzbMATHGoogle Scholar
  2. 2.
    Brown, L., Cai, T., DasGupta, A.: Interval estimation for a binomial proportion. Stat. Sci. 16(2), 101–133 (2001)MathSciNetzbMATHGoogle Scholar
  3. 3.
    Chen, J.: Properties of a new adaptive sampling method with applications to scalable learning. In: WI, pp. 9–15, Atlanta (2013)Google Scholar
  4. 4.
    Chernoff, H.: A measure of asymptotic efficiency for tests of a hypothesis based on the sum of observations. Ann. Math. Stat. 23(4), 493–507 (1952)MathSciNetCrossRefzbMATHGoogle Scholar
  5. 5.
    Clarke, E.M., Zuliani, P.: Statistical model checking for cyber-physical systems. In: Bultan, T., Hsiung, P.-A. (eds.) ATVA 2011. LNCS, vol. 6996, pp. 1–12. Springer, Heidelberg (2011). doi: 10.1007/978-3-642-24372-1_1 CrossRefGoogle Scholar
  6. 6.
    Clopper, C.J., Pearson, E.S.: The use of confidence or fiducial limits illustrated in the case of the binomial. Biometrika 26, 404–413 (1934)CrossRefzbMATHGoogle Scholar
  7. 7.
    Dagum, P., Karp, R., Luby, M., Ross, S.: An optimal algorithm for Monte Carlo estimation. SIAM J. Comput. 29(5), 1484–1496 (2000)MathSciNetCrossRefzbMATHGoogle Scholar
  8. 8.
    David, A., Larsen, K.G., Legay, A., Mikucionis, M., Poulsen, D.B.: Uppaal SMC tutorial. STTT 17(4), 397–415 (2015)CrossRefGoogle Scholar
  9. 9.
    Frey, J.: Fixed-width sequential confidence intervals for a proportion. Am. Stat. 64(3), 242–249 (2010)MathSciNetCrossRefGoogle Scholar
  10. 10.
    Grosu, R., Peled, D., Ramakrishnan, C.R., Smolka, S.A., Stoller, S.D., Yang, J.: Using statistical model checking for measuring systems. In: Margaria, T., Steffen, B. (eds.) ISoLA 2014. LNCS, vol. 8803, pp. 223–238. Springer, Heidelberg (2014). doi: 10.1007/978-3-662-45231-8_16 Google Scholar
  11. 11.
    Hérault, T., Lassaigne, R., Magniette, F., Peyronnet, S.: Approximate probabilistic model checking. In: Steffen, B., Levi, G. (eds.) VMCAI 2004. LNCS, vol. 2937, pp. 73–84. Springer, Heidelberg (2004). doi: 10.1007/978-3-540-24622-0_8 CrossRefGoogle Scholar
  12. 12.
    Hérault, T., Lassaigne, R., Peyronnet, S.: APMC 3.0: approximate verification of discrete and continuous time Markov chains. In: QEST, pp. 129–130 (2006)Google Scholar
  13. 13.
    Hoeffding, W.: Probability inequalities for sums of bounded random variables. J. Am. Stat. Assoc. 58(301), 13–30 (1963)MathSciNetCrossRefzbMATHGoogle Scholar
  14. 14.
    Jegourel, C., Legay, A., Sedwards, S.: A platform for high performance statistical model checking – PLASMA. In: Flanagan, C., König, B. (eds.) TACAS 2012. LNCS, vol. 7214, pp. 498–503. Springer, Heidelberg (2012). doi: 10.1007/978-3-642-28756-5_37 CrossRefGoogle Scholar
  15. 15.
    Jegourel, C., Legay, A., Sedwards, S.: Importance splitting for statistical model checking rare properties. In: Sharygina, N., Veith, H. (eds.) CAV 2013. LNCS, vol. 8044, pp. 576–591. Springer, Heidelberg (2013). doi: 10.1007/978-3-642-39799-8_38 CrossRefGoogle Scholar
  16. 16.
    Kwiatkowska, M.Z., Norman, G., Parker, D.: PRISM 2.0: a tool for probabilistic model checking. In: QEST, pp. 322–323. IEEE (2004)Google Scholar
  17. 17.
    Massart, P.: The tight constant in the Dvoretzky-Kiefer-Wolfowitz inequality. Ann. Prob. 18, 1269–1283 (1990)MathSciNetCrossRefzbMATHGoogle Scholar
  18. 18.
    Metropolis, N., Ulam, S.: The Monte Carlo method. J. Am. Stat. Assoc. 44(247), 335–341 (1949)CrossRefzbMATHGoogle Scholar
  19. 19.
    Okamoto, M.: Some inequalities relating to the partial sum of binomial probabilities. Ann. Inst. Statis. Math. 10, 29–35 (1958)MathSciNetCrossRefzbMATHGoogle Scholar
  20. 20.
    Wald, A.: Sequential tests of statistical hypotheses. Ann. Math. Stat. 16(2), 117–186 (1945)MathSciNetCrossRefzbMATHGoogle Scholar
  21. 21.
    Watanabe, O.: Sequential sampling techniques for algorithmic learning theory. Theoret. Comput. Sci. 348, 3–14 (2005)MathSciNetCrossRefzbMATHGoogle Scholar
  22. 22.
    Younes, H.: Verification and planning for stochastic processes with asynchronous events. Ph.D. thesis, Carnegie Mellon University (2004)Google Scholar
  23. 23.
    Younes, H.L.S., Kwiatkowska, M., Norman, G., Parker, D.: Numerical vs. statistical probabilistic model checking: an empirical study. In: Jensen, K., Podelski, A. (eds.) TACAS 2004. LNCS, vol. 2988, pp. 46–60. Springer, Heidelberg (2004). doi: 10.1007/978-3-540-24730-2_4 CrossRefGoogle Scholar
  24. 24.
    Zuliani, P., Platzer, A., Clarke, E.M.: Bayesian statistical model checking with application to stateflow/simulink verification. FMSD 43(2), 338–367 (2013)zbMATHGoogle Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  1. 1.Singapore University of Technology and DesignSingaporeSingapore
  2. 2.Griffith UniversityBrisbaneAustralia

Personalised recommendations