Test Coverage Estimation Using Threshold Accepting

  • Thao Dang
  • Noa Shalev
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8837)


This paper is concerned with model-based testing of hybrid systems. In our previous work [6], we proposed a test generation algorithm, called gRRT, guided by a coverage measure defined using the star discrepancy notion. An important ingredient in this algorithm is a procedure for dynamically estimating the coverage, which is done based on a box partition of the continuous state space. The goal of this estimation is to identify the areas in the state space which have not been sufficiently visited. A drawback of this guiding method is that its complexity depends on the number of the boxes in the partition, which needs to be fine enough to guarantee a good coverage estimate. Thus in high dimensions the method can become very costly. To enhance the scalability of the algorithm gRRT we propose in this paper a new guiding method, motivated by the observation that trying to optimize the coverage in each exploration step is, on one hand, computationally costly, and on the other hand, not always a good choice since this may make the system try to expand in the directions which are not reachable (due to the controllability of the system). Instead of considering all the boxes in the partition, we propose to use a randomized search to quickly find a region that yields a high local discrepancy value. This randomized search is based on threshold accepting, a well-known integer optimization heuristic. We also present some experimental results obtained on a challenging circuit benchmark and a number of randomly generated examples, which shows that the new guiding method allows achieving better time and coverage efficiency.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Abbas, H., Fainekos, G.: Convergence Proofs for Simulated Annealing Falsification of Safety Properties. In: Proc. of 50th Annual Allerton Conference on Communication, Control, and Computing, Monticello, IL (October 2012)Google Scholar
  2. 2.
    Alur, R., Courcoubetis, C., Halbwachs, N., Henzinger, T.A., Ho, P.-H., Nicollin, X., Olivero, A., Sifakis, J., Yovine, S.: The algorithmic analysis of hybrid systems. Theoretical Computer Science 138(1), 3–34 (1995)MathSciNetCrossRefzbMATHGoogle Scholar
  3. 3.
    Beck, J., Chen, W.W.L.: Irregularities of distribution. In: Acta Arithmetica, UK, Cambridge University Press (1997)Google Scholar
  4. 4.
    Branicky, M., Curtiss, M., Levine, J., Morgan, S.: Sampling-based reachability algorithms for control and verification of complex systems. In: Thirteenth Yale Workshop on Adaptive and Learning Systems (2005)Google Scholar
  5. 5.
    Dang, T., Shalev, N.: State estimation and property-guided exploration for hybrid systems testing. In: Nielsen, B., Weise, C. (eds.) ICTSS 2012. LNCS, vol. 7641, pp. 152–167. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  6. 6.
    Dang, T., Nahhal, T.: Coverage-guided test generation for continuous and hybrid systems. Formal Methods in System Design 34(2), 183–213 (2009)CrossRefzbMATHGoogle Scholar
  7. 7.
    Dueck, G., Scheuer, T.: Threshold accepting: A general purpose optimization algorithm appearing superior to simulated annealing. Journal of Computational Physics 90(1), 161–175 (1990)MathSciNetCrossRefzbMATHGoogle Scholar
  8. 8.
    Dobkin, D., Eppstein, D.: Computing the discrepancy. In: SCG 1993: Proceedings of the Ninth Annual Symposium on Computational Geometry, pp. 47–52. ACM Press, New York (1993)Google Scholar
  9. 9.
    Kim, J., Esposito, J., Kumar, V.: Sampling-based algorithm for testing and validating robot controllers. Int. J. Rob. Res. 25(12), 1257–1272 (2006)CrossRefzbMATHGoogle Scholar
  10. 10.
    Esposito, J., Kim, J.W., Kumar, V.: Adaptive RRTs for validating hybrid robotic control systems. In: Proceedings Workshop on Algorithmic Foundations of Robotics, Zeist, The Netherlands (July 2004)Google Scholar
  11. 11.
    Bhatia, A., Frazzoli, E.: Incremental search methods for reachability analysis of continuous and hybrid systems. In: Alur, R., Pappas, G.J. (eds.) HSCC 2004. LNCS, vol. 2993, pp. 142–156. Springer, Heidelberg (2004)Google Scholar
  12. 12.
    Gnewuch, M., Srivastav, A., Winzen, C.: Finding optimal volume subintervals with k points and calculating the star discrepancy are NP-hard problems. J. Complexity 25(2), 115–127 (2009)MathSciNetCrossRefzbMATHGoogle Scholar
  13. 13.
    Gnewuch, M., Wahlström, M., Winzen, C.: SIAM Journal of Numerical Analysis 50, 781–807 (2012)Google Scholar
  14. 14.
    Julius, A., Fainekos, G.E., Anand, M., Lee, I., Pappas, G.J.: Robust test generation and coverage for hybrid systems. In: Bemporad, A., Bicchi, A., Buttazzo, G. (eds.) HSCC 2007. LNCS, vol. 4416, pp. 329–342. Springer, Heidelberg (2007)Google Scholar
  15. 15.
    Kuffner, J., LaValle, S.: RRT-connect: An efficient approach to single-query path planning. In: Proc. IEEE Int’l Conf. on Robotics and Automation (ICRA 2000), San Francisco, CA (April 2000)Google Scholar
  16. 16.
    LaValle, S., Kuffner, J.: Rapidly-exploring random trees: Progress and prospects. In: Workshop on the Algorithmic Foundations of Robotics (2000)Google Scholar
  17. 17.
    Mcinn, P.: Search-based Software Test Data Generation: A Survey: Research Articles. Softw. Test. Verif. Reliab. 14(2), 105–156Google Scholar
  18. 18.
    Plaku, E., Kavraki, L.E., Vardi, M.Y.: Hybrid systems: From verification to falsification. In: Damm, W., Hermanns, H. (eds.) CAV 2007. LNCS, vol. 4590, pp. 463–476. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  19. 19.
    Plaku, E., Kavraki, L.E., Vardi, M.Y.: Hybrid Systems: From Verification to Falsification by Combining Motion Planning and Discrete Search. Formal Methods in System Design 34(2), 157–182 (2009)CrossRefzbMATHGoogle Scholar
  20. 20.
    Sankaranarayanan, S., Fainekos, G.: Falsification of Temporal Properties of Hybrid Systems Using the Cross-Entropy Method. In: Proceedings of the ACM International Conference on Hybrid Systems: Computation and Control, Beijing, China (April 2012)Google Scholar
  21. 21.
    Thiémard, E.: An algorithm to compute bounds for the star discrepancy. J. Complexity 17(4), 850–880 (2001)MathSciNetCrossRefzbMATHGoogle Scholar
  22. 22.
    Winker, P., Fang, K.: Application of Threshold-Accepting to the Evaluation of the Discrepancy of a Set of Points. SIAM Journal on Numerical Analysis 34, 2028–2042 (1997)MathSciNetCrossRefzbMATHGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Thao Dang
    • 1
  • Noa Shalev
    • 1
  1. 1.Centre EquationVERIMAG/CNRSGièresFrance

Personalised recommendations