Coverage Metrics for Saturation-Based and Search-Based Testing of Concurrent Software

  • Bohuslav Křena
  • Zdeněk Letko
  • Tomáš Vojnar
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7186)

Abstract

Coverage metrics play a crucial role in testing. They allow one to estimate how well a program has been tested and/or to control the testing process. Several concurrency-related coverage metrics have been proposed, but most of them do not reflect concurrent behaviour accurately enough. In this paper, we propose several new metrics that are suitable primarily for saturation-based or search-based testing of concurrent software. Their distinguishing feature is that they are derived from various dynamic analyses designed for detecting synchronisation errors in concurrent software. In fact, the way these metrics are obtained is generic, and further metrics can be obtained in a similar way from other analyses. The underlying motivation is that, within such analyses, behavioural aspects crucial for occurrence of various bugs are identified, and hence it makes sense to track how well the occurrence of such phenomena is covered by testing. Next, coverage tasks of the proposed as well as some existing metrics are combined with an abstract identification of the threads participating in generation of the phenomena captured in the concerned tasks. This way, further, more precise metrics are obtained. Finally, an empirical evaluation of the proposed metrics, which confirms that several of them are indeed more suitable for saturation-based and search-based testing than the previously known metrics, is presented.

Keywords

Concurrent Program Program Location Test Execution Coverage Metrics Data Race 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Bensalem, S., Havelund, K.: Dynamic Deadlock Analysis of Multi-threaded Programs. In: Ur, S., Bin, E., Wolfsthal, Y. (eds.) HVC 2005. LNCS, vol. 3875, pp. 208–223. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  2. 2.
    Bron, A., Farchi, E., Magid, Y., Nir, Y., Ur, S.: Applications of Synchronization Coverage. In: Proc. of PPoPP 2005. ACM Press (2005)Google Scholar
  3. 3.
    Clarke, E., Grumberg, O., Peled, D.: Model Checking. MIT Press (1999)Google Scholar
  4. 4.
    Edelstein, O., Farchi, E., Goldin, E., Nir, Y., Ratsaby, G., Ur, S.: Framework for Testing Multi-threaded Java Programs. Concurrency and Computation: Practice and Experience 15(3-5), 485–499 (2003)MATHCrossRefGoogle Scholar
  5. 5.
    Elmas, T., Qadeer, S., Tasiran, S.: Goldilocks: A Race and Transaction-aware Java Runtime. In: Proc. of PLDI 2007. ACM Press (2007)Google Scholar
  6. 6.
    Hovemeyer, D., Pugh, W.: Finding Concurrency Bugs in Java. In: Proc. of PODC 2004. ACM Press (2004)Google Scholar
  7. 7.
    Joshi, P., Park, C.-S., Sen, K., Naik, M.: A Randomized Dynamic Program Analysis Technique for Detecting Real Deadlocks. In: Proc. of PLDI 2009. ACM Press (2009)Google Scholar
  8. 8.
    Kahlon, V., Yang, Y., Sankaranarayanan, S., Gupta, A.: Fast and Accurate Static Data-Race Detection for Concurrent Programs. In: Damm, W., Hermanns, H. (eds.) CAV 2007. LNCS, vol. 4590, pp. 226–239. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  9. 9.
    Křena, B., Letko, Z., Vojnar, T., Ur, S.: A Platform for Search-based Testing of Concurrent Software. In: Proc. of PADTAD 2010. ACM Press (2010)Google Scholar
  10. 10.
    Lu, S., Tucek, J., Qin, F., Zhou, Y.: Avio: Detecting Atomicity Violations via Access Interleaving Invariants. In: Proc. of ASPLOS 2006. ACM Press (2006)Google Scholar
  11. 11.
    Manson, J., Pugh, W., Adve, S.V.: The Java Memory Model. In: Proc. of POPL 2005. ACM Press (2005)Google Scholar
  12. 12.
    McMinn, P.: Search-based Software Test Data Generation: A Survey: Research Articles. Software Testing, Verification, and Reliability 14(2), 105–156 (2004)CrossRefGoogle Scholar
  13. 13.
    Musuvathi, M., Qadeer, S., Ball, T., Basler, G., Nainar, P.A., Neamtiu, I.: Finding and Reproducing Heisenbugs in Concurrent Programs. In: Proc. of OSDI 2008. USENIX Association (2008)Google Scholar
  14. 14.
    Pozniansky, E., Schuster, A.: Efficient On-the-fly Data Race Detection in Multithreaded C++ Programs. In: Proc. of PPoPP 2003. ACM Press (2003)Google Scholar
  15. 15.
    Savage, S., Burrows, M., Nelson, G., Sobalvarro, P., Anderson, T.: Eraser: A Dynamic Data Race Detector for Multi-threaded Programs. In: Proc. of SOSP 1997. ACM Press (1997)Google Scholar
  16. 16.
    Sherman, E., Dwyer, M.B., Elbaum, S.: Saturation-based Testing of Concurrent Programs. In: Proc. of ESEC/FSE 2009. ACM Press (2009)Google Scholar
  17. 17.
    Soriano, J., Jimenez, M., Cantera, J.M., Hierro, J.J.: Delivering Mobile Enterprise Services on Morfeo’s MC Open Source Platform. In: Proc. of MDM 2006. IEEE CS (2006)Google Scholar
  18. 18.
    Trainin, E., Nir-Buchbinder, Y., Tzoref-Brill, R., Zlotnick, A., Ur, S., Farchi, E.: Forcing Small Models of Conditions on Program Interleaving for Detection of Concurrent Bugs. In: Proc. of PADTAD 2009. ACM Press (2009)Google Scholar
  19. 19.
    von Praun, C., Gross, T.R.: Object Race Detection. In: Proc. of OOPSLA 2001. ACM Press (2001)Google Scholar
  20. 20.
    Williams, A., Thies, W., Ernst, M.D.: Static Deadlock Detection for Java Libraries. In: Gao, X.-X. (ed.) ECOOP 2005. LNCS, vol. 3586, pp. 602–629. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  21. 21.
    Yang, C.-S.D., Souter, A.L., Pollock, L.L.: All-du-path Coverage for Parallel Programs. In: Proc. of ISSTA 1998. ACM Press (1998)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Bohuslav Křena
    • 1
  • Zdeněk Letko
    • 1
  • Tomáš Vojnar
    • 1
  1. 1.FITBrno University of TechnologyCzech Republic

Personalised recommendations