International School on Formal Methods for the Design of Computer, Communication and Software Systems

SFM 2014: Formal Methods for Executable Software Models pp 310-342 | Cite as

Model-Based Testing

  • Malte Lochau
  • Sven Peldszus
  • Matthias Kowal
  • Ina Schaefer
Part of the Lecture Notes in Computer Science book series (LNCS, volume 8483)

Abstract

Software more and more pervades our everyday lives. Hence, we have high requirements towards the trustworthiness of the software. Software testing greatly contributes to the quality assurance of modern software systems. However, as today’s software system get more and more complex and exist in many different variants, we need rigorous and systematic approaches towards software testing. In this tutorial, we, first, present model-based testing as an approach for systematic test case generation, test execution and test result evaluation for single system testing. The central idea of model-based testing is to base all testing activities on an executable model-based test specification. Second, we consider model-based testing for variant-rich software systems and review two model-based software product line testing techniques. Sample-based testing generates a set of representative variants for testing, and variability-aware product line testing uses a family-based test model which contains the model-based specification of all considered product variants.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Alur, R., Henzinger, T.A., Kupferman, O., Vardi, M.Y.: Alternating Refinement Relations. In: Sangiorgi, D., de Simone, R. (eds.) CONCUR 1998. LNCS, vol. 1466, pp. 163–178. Springer, Heidelberg (1998)CrossRefGoogle Scholar
  2. 2.
    Asirelli, P., ter Beek, M.H., Fantechi, A., Gnesi, S.: A Model-Checking Tool for Families of Services. In: Bruni, R., Dingel, J. (eds.) FMOODS/FORTE 2011. LNCS, vol. 6722, pp. 44–58. Springer, Heidelberg (2011)Google Scholar
  3. 3.
    Asirelli, P., ter Beek, M.H., Gnesi, S., Fantechi, A.: Formal Description of Variability in Product Families. In: SPLC 2011, pp. 130–139 (2011)Google Scholar
  4. 4.
    Asirelli, P., ter Beek, M.H., Fantechi, A., Gnesi, S.: A Logical Framework to Deal with Variability. In: Méry, D., Merz, S. (eds.) IFM 2010. LNCS, vol. 6396, pp. 43–58. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  5. 5.
    Bauer, S.S., Hennicker, R., Janisch, S.: Interface Theories for (A)synchronously Communicating Modal I/O-Transition Systems. Electronic Proceedings in Theoretical Computer Science 46, 1–8 (2011)CrossRefGoogle Scholar
  6. 6.
    Beizer, B.: Software Testing Techniques, 2nd edn. Van Nostrand Reinhold Co., New York (1990)Google Scholar
  7. 7.
    Bernot, G.: Testing against Formal Specifications: A Theoretical View. In: Abramsky, S. (ed.) TAPSOFT 1991, CCPSD 1991, and ADC-Talks 1991. LNCS, vol. 494, pp. 99–119. Springer, Heidelberg (1991)Google Scholar
  8. 8.
    van der Bijl, M., Rensink, A., Tretmans, J.: Compositional testing with ioco. In: Petrenko, A., Ulrich, A. (eds.) FATES 2003. LNCS, vol. 2931, pp. 86–100. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  9. 9.
    Broy, M. (ed.): Model-Based Testing of Reactive Systems: Advanced Lectures, 1st edn. Springer, Heidelberg (2005)MATHGoogle Scholar
  10. 10.
    Chvatal, V.: A Greedy Heuristic For The Set-Covering Problem. Mathematics of Operations Research (1979)Google Scholar
  11. 11.
    Cichos, H., Oster, S., Lochau, M., Schürr, A.: Model-Based Coverage-Driven Test Suite Generation for Software Product Lines. In: Whittle, J., Clark, T., Kühne, T. (eds.) MODELS 2011. LNCS, vol. 6981, pp. 425–439. Springer, Heidelberg (2011)CrossRefGoogle Scholar
  12. 12.
    Clements, P.C., Northrop, L.: Software Product Lines: Practices and Patterns. SEI Series in Software Eng. Addison-Wesley (2001)Google Scholar
  13. 13.
    Cohen, M.B., Dwyer, M.B., Shi, J.: Coverage and Adequacy in Software Product Line Testing. In: ISSTA, pp. 53–63. ACM (2006)Google Scholar
  14. 14.
    Czarnecki, K., Antkiewicz, M.: Mapping Features to Models: A Template Approach Based on Superimposed Variants. In: Glück, R., Lowry, M. (eds.) GPCE 2005. LNCS, vol. 3676, pp. 422–437. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  15. 15.
    Czarnecki, K., Eisenecker, U.W.: Generative Programming: Methods, Tools, and Applications. ACM Press/Addison-Wesley (2000)Google Scholar
  16. 16.
    De Nicola, R.: Extensional Equivalence for Transition Systems. Acta Informatica 24, 211–237 (1987), http://portal.acm.org/citation.cfm?id=25067.25074MathSciNetCrossRefMATHGoogle Scholar
  17. 17.
    of Electrical, I., Engineers, E.: IEEE Standard Glossary of Software Engineering Technology 610.121990 (1990)Google Scholar
  18. 18.
    of Electrical, I., Engineers, E.: IEEE Standard for Software Test Documentation IEEE Std 829-1998 (1998)Google Scholar
  19. 19.
    Engström, E.: Exploring Regression Testing and Software Product Line Testing - Research and State of Practice. Lic dissertation, Lund University (May 2010)Google Scholar
  20. 20.
    Engström, E., Runeson, P.: Software Product Line Testing – A Systematic Mapping Study. Information and Software Technology 53(1), 2–13 (2011)CrossRefGoogle Scholar
  21. 21.
    Ferber, S., Haag, J., Savolainen, J.: Feature Interaction and Dependencies: Modeling Features for Reengineering a Legacy Product Line. In: Chastek, G.J. (ed.) SPLC 2002. LNCS, vol. 2379, p. 235. Springer, Heidelberg (2002)CrossRefGoogle Scholar
  22. 22.
    Fischbein, D., Uchitel, S., Braberman, V.A.: A Foundation for Behavioural Conformance in Software Product Line Architectures. In: Hierons, R.M., Muccini, H. (eds.) ISSTA 2006, pp. 39–48. ACM (2006)Google Scholar
  23. 23.
    Gustafsson, T.: An Approach for Selecting Software Product Line Instances for Testing. In: SPLiT (2007)Google Scholar
  24. 24.
    Henard, C., Papadakis, M., Perrouin, G., Klein, J., Traon, Y.L.: Multi-objective Test Generation for Software Product Lines. In: SPLC (2013)Google Scholar
  25. 25.
    Johansen, M.F., Haugen, O., Fleurey, F.: An Algorithm for Generating t-wise Covering Arrays from Large Feature Models. In: SPLC, pp. 46–55. ACM (2012)Google Scholar
  26. 26.
    Johansen, M.F., Haugen, Ø., Fleurey, F.: Properties of Realistic Feature Models Make Combinatorial Testing of Product Lines Feasible. In: Whittle, J., Clark, T., Kühne, T. (eds.) MODELS 2011. LNCS, vol. 6981, pp. 638–652. Springer, Heidelberg (2011)CrossRefGoogle Scholar
  27. 27.
    Kang, K.C., Cohen, S.G., Hess, J.A., Novak, W.E., Peterson, A.S.: Feature-Oriented Domain Analysis (FODA) Feasibility Study. Tech. rep., Carnegie-Mellon University Software Engineering Institute (November 1990)Google Scholar
  28. 28.
    Kim, C.H.P., Batory, D., Khurshid, S.: Reducing Combinatorics in Testing Product Lines. In: AOSD, pp. 57–68. ACM (2011)Google Scholar
  29. 29.
    Kowal, M., Schulze, S., Schaefer, I.: Towards Efficient SPL Testing by Variant Reduction. In: VariComp, pp. 1–6. ACM (2013)Google Scholar
  30. 30.
    Kuhn, D.R., Wallace, D.R., Gallo, J. A.M.: Software fault interactions and implications for software testing. IEEE Trans. Softw. Eng. 30(6), 418–421 (2004)CrossRefGoogle Scholar
  31. 31.
    Larsen, K.G., Nyman, U., Wąsowski, A.: Modal I/O Automata for Interface and Product Line Theories. In: De Nicola, R. (ed.) ESOP 2007. LNCS, vol. 4421, pp. 64–79. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  32. 32.
    Larsen, K.G., Thomsen, B.: A Modal Process Logic. In: LICS, pp. 203–210 (1988)Google Scholar
  33. 33.
    Lochau, M., Oster, S., Goltz, U., Schürr, A.: Model-based Pairwise Testing for Feature Interaction Coverage in Software Product Line Engineering. Software Quality Journal, 1–38 (2011)Google Scholar
  34. 34.
    Lochau, M., Schaefer, I., Kamischke, J., Lity, S.: Incremental Model-Based Testing of Delta-Oriented Software Product Lines. In: Brucker, A.D., Julliand, J. (eds.) TAP 2012. LNCS, vol. 7305, pp. 67–82. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  35. 35.
    Lochau, M., Goltz, U.: Feature Interaction Aware Test Case Generation for Embedded Control Systems. Electron. Notes Theor. Comput. Sci. 264, 37–52 (2010)CrossRefGoogle Scholar
  36. 36.
    Lüttgen, G., Vogler, W.: Modal Interface Automata. In: Baeten, J.C.M., Ball, T., de Boer, F.S. (eds.) TCS 2012. LNCS, vol. 7604, pp. 265–279. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  37. 37.
    McGregor, J.D.: Testing a Software Product Line. Tech. Rep. CMU/SEI-2001-TR-022, Carnegie Mellon, Software Engineering Inst. (2001)Google Scholar
  38. 38.
    McGregor, J.D., Sodhani, P., Madhavapeddi, S.: Testing Variability in a Software Product Line. In: Proceedings of the Software Product Line Testing Workshop (SPLiT), pp. 45–50. Avaya Labs, Boston (2004)Google Scholar
  39. 39.
    Mendonca, M., Wąsowski, A., Czarnecki, K.: SAT-based Analysis of Feature Models is Easy. In: Proc. Int’l Software Product Line Conference, pp. 231–240 (2009)Google Scholar
  40. 40.
    da, M.S., Neto, P.A., Carmo Machado, I.D., McGregor, J.D., de Almeida, E.S., de Lemos Meira, S.R.: A Systematic Mapping Study of Software Product Lines Testing. Inf. Softw. Technol. 53, 407–423 (2011)CrossRefGoogle Scholar
  41. 41.
    Myers, G.J.: The Art of Software Testing. Wiley, New York (1979)Google Scholar
  42. 42.
    Olimpiew, E.M.: Model-Based Testing for Software Product Lines. Ph.D. thesis, George Mason University (2008)Google Scholar
  43. 43.
    van Osch, M.: Hybrid input-output conformance and test generation. In: Havelund, K., Núñez, M., Roşu, G., Wolff, B. (eds.) FATES 2006 and RV 2006. LNCS, vol. 4262, pp. 70–84. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  44. 44.
    Oster, S., Lochau, M., Zink, M., Grechanik, M.: Pairwise Feature-Interaction Testing for SPLs: Potentials and Limitations. In: 3rd International Workshop on Feature-Oriented Software Development (FOSD) (2011)Google Scholar
  45. 45.
    Oster, S., Zorcic, I., Markert, F., Lochau, M.: MoSo-PoLiTe - Tool Support for Pairwise and Model-Based Software Product Line Testing. In: VaMoS (2011)Google Scholar
  46. 46.
    Oster, S., Wübbeke, A., Engels, G., Schürr, A.: Model-Based Software Product Lines Testing Survey. In: Model-based Testing for Embedded Systems. CRC Press Taylor & Francis (2010) (to appear)Google Scholar
  47. 47.
    Perrouin, G., Sen, S., Klein, J., Le Traon, B.: Automated and Scalable T-wise Test Case Generation Strategies for Software Product Lines. In: ICST, pp. 459–468 (2010)Google Scholar
  48. 48.
    Raclet, J.B., Badouel, E., Benveniste, A., Caillaud, B., Legay, A., Passerone, R.: Modal Interfaces: Unifying Interface Automata and Modal Specifications. In: EMSOFT 2009, pp. 87–96. ACM (2009)Google Scholar
  49. 49.
    Reuys, A., Kamsties, E., Pohl, K., Reis, S.: Model-Based System Testing of Software Product Families. In: Pastor, Ó., Falcão e Cunha, J. (eds.) CAiSE 2005. LNCS, vol. 3520, pp. 519–534. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  50. 50.
    Schaefer, I., Rabiser, R., Clarke, D., Bettini, L., Benavides, D., Botterweck, G., Pathak, A., Trujillo, S., Villela, K.: Software diversity: state of the art and perspectives. STTT 14(5), 477–495 (2012)CrossRefGoogle Scholar
  51. 51.
    Scheidemann, K.: Verifying Families of System Configurations. Ph.D. thesis, TU Munich (2007)Google Scholar
  52. 52.
    Schmaltz, J., Tretmans, J.: On Conformance Testing for Timed Systems. In: Cassez, F., Jard, C. (eds.) FORMATS 2008. LNCS, vol. 5215, pp. 250–264. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  53. 53.
    Tevanlinna, A., Taina, J., Kauppinen, R.: Product Family Testing: A Survey. ACM SIGSOFT Software Engineering Notes 29, 12–18 (2004)CrossRefGoogle Scholar
  54. 54.
    Tretmans, J.: Testing concurrent systems: A formal approach. In: Baeten, J.C.M., Mauw, S. (eds.) CONCUR 1999. LNCS, vol. 1664, p. 46. Springer, Heidelberg (1999)CrossRefGoogle Scholar
  55. 55.
    Utting, M., Legeard, B.: Practical Model-Based Testing: A Tools Approach. Morgan Kaufmann, San Francisco (2007)Google Scholar
  56. 56.
    van Veenendaal, E. (ed.): ISTQB-Glossary-of-Testing-Terms-2-0. Glossary Working Party (2007)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2014

Authors and Affiliations

  • Malte Lochau
    • 1
  • Sven Peldszus
    • 1
  • Matthias Kowal
    • 2
  • Ina Schaefer
    • 2
  1. 1.TU DarmstadtGermany
  2. 2.TU BraunschweigGermany

Personalised recommendations