Advertisement

Mutation Testing for RoboChart

Chapter
  • 141 Downloads

Abstract

This chapter describes a test-generation approach that takes as input a model S of the expected behavior of a robotic system and seeds faults into S, leading to a set of mutants of S. Given a mutant M of S, we check whether M is a valid implementation of S, and, if it is not, we find a test case that demonstrates this: a test case that reveals the seeded fault. In order to automate this approach, we used the Wodel tool to seed faults and a combination of two tools, RoboTool and FDR, to generate tests that detect the seeded faults. The result is an overall test-generation technique that can be automated and that derives test cases that are guaranteed to find certain faults.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

Notes

Acknowledgements

This work has been supported by EPSRC grant EP/R025134/2 RoboTest: Systematic Model-Based Testing and Simulation of Mobile Autonomous Robots, Spanish MINECO-FEDER grant FAME RTI2018-093608-B-C31 and the Comunidad de Madrid project FORTE-CM S2018/TCS-4314.

References

  1. 1.
    H. Agrawal, R.A. DeMillo, B. Hathaway, W. Hsu, W. Hsu, E.W. Krauser, R.J. Martin, A.P. Mathur, E. Spafford, Design of mutant operators for the C programming language. Technical report, Purdue University, 1989Google Scholar
  2. 2.
    P. Ammann, J. Offutt, Introduction to Software Testing (Cambridge University Press, Cambridge, 2008)CrossRefGoogle Scholar
  3. 3.
    P. Ammann, M.E. Delamaro, J. Offutt, Establishing theoretical minimal sets of mutants, in Verification and Validation 2014 IEEE Seventh International Conference on Software Testing (2014), pp. 21–30 ISSN: 2159–4848Google Scholar
  4. 4.
    J.H. Andrews, L.C. Briand, Y. Labiche, Is mutation an appropriate tool for testing experiments? in 27th International Conference on Software Engineering (ICSE 2005), 15–21 May 2005, St. Louis, Missouri, ed. by G.-C. Roman, W.G. Griswold, B. Nuseibeh (ACM, New York, 2005), pp. 402–411Google Scholar
  5. 5.
    J.S. Bradbury, J.R. Cordy, J. Dingel, Mutation operators for concurrent Java (J2SE 5.0), in Second Workshop on Mutation Analysis (Mutation 2006 - ISSRE Workshops 2006) (2006), pp. 83–92Google Scholar
  6. 6.
    T.A. Budd, D. Angluin, Two notions of correctness and their relation to testing. Acta Inform. 18(1), 31–45 (1982)MathSciNetCrossRefGoogle Scholar
  7. 7.
    P.C. Cañizares, A. Núñez, M.G. Merayo, Mutomvo: mutation testing framework for simulated cloud and HPC environments. J. Syst. Softw. 143, 187–207 (2018)CrossRefGoogle Scholar
  8. 8.
    A. Cavalcanti, J. Baxter, R.M. Hierons, R. Lefticaru, Testing robots using CSP, in Proceedings of Tests and Proofs - 13th International Conference, TAP 2019, Held as Part of the Third World Congress on Formal Methods 2019, Porto, 9–11 Oct 2019, ed. by D. Beyer, C. Keller. Lecture Notes in Computer Science, vol. 11823 (Springer, Berlin, 2019), pp. 21–38Google Scholar
  9. 9.
    T.T. Chekam, M. Papadakis, Y.L. Traon, M. Harman, An empirical study on mutation, statement and branch coverage fault revelation that avoids the unreliable clean program assumption, in Proceedings of the 39th International Conference on Software Engineering, ICSE 2017, Buenos Aires, 20–28 May 2017, ed. by S. Uchitel, A. Orso, M.P. Robillard (IEEE/ACM, New York, 2017), pp. 597–608Google Scholar
  10. 10.
    J.A. Clark, H. Dan, R.M. Hierons, Semantic mutation testing. Sci. Comput. Program. 78(4), 345–363 (2013)CrossRefGoogle Scholar
  11. 11.
    B. Darolti, Software engineering for robotics: an autonomous robotic vacuum cleaner for solar panels. Master’s thesis, University of York, 2019Google Scholar
  12. 12.
    M.E. Delamaro, J.C. Maldonado, Proteum-a tool for the assessment of test adequacy for C programs, in Proceedings of the Conference on Performability in Computing Systems (PCS’96), New Brunswick, NJ, July 1996, pp. 79–95Google Scholar
  13. 13.
    M.E. Delamaro, J. Offutt, P. Ammann, Designing deletion mutation operators, in Verification and Validation 2014 IEEE Seventh International Conference on Software Testing (2014), pp. 11–20Google Scholar
  14. 14.
    P. Delgado-Pérez, I. Medina-Bulo, J.J. Domínguez-Jiménez, A García-Domínguez, F. Palomo-Lozano, Class mutation operators for C++ object-oriented systems. Ann. Télécomm. 70(3–4), 137–148 (2015)Google Scholar
  15. 15.
    P. Delgado-Pérez, S. Segura, I. Medina-Bulo, Assessment of C++ object-oriented mutation operators: a selective mutation approach. Softw. Test. Verif. Reliab. 27(4–5), e1630 (2017)Google Scholar
  16. 16.
    R.A. DeMillo, A.J. Offutt, Constraint-based automatic test data generation. IEEE Trans. Softw. Eng. 17(9), 900–910 (1991)CrossRefGoogle Scholar
  17. 17.
    R.A. DeMillo, R.J. Lipton, F.G. Sayward, Hints on test data selection: help for the practicing programmer. Computer 11(4), 34–41 (1978)CrossRefGoogle Scholar
  18. 18.
    R.A. DeMillo, D.S. Guindi, W.M. McCracken, A.J. Offutt, K.N. King, An extended overview of the Mothra software testing environment, in [1988] Proceedings. Second Workshop on Software Testing, Verification, and Analysis, July 1988, pp. 142–151Google Scholar
  19. 19.
    L. Deng, J. Offutt, N. Li, Empirical evaluation of the statement deletion mutation operator, in Sixth IEEE International Conference on Software Testing, Verification and Validation, ICST 2013, Luxembourg, 18–22 March 2013 (IEEE Computer Society, Washington, 2013), pp. 84–93Google Scholar
  20. 20.
    A. Estero-Botaro, F. Palomo-Lozano, I. Medina-Bulo, Quantitative evaluation of mutation operators for WS-BPEL compositions, in Proceedings of the 3rd International Conference on Software Testing, Verification, and Validation Workshops (ICSTW’10) (2010), pp. 142–150Google Scholar
  21. 21.
    S.C.P.F. Fabbri, M.E. Delamaro, J.C. Maldonado, P.C. Masiero, Mutation analysis testing for finite state machines, in Proceedings of the 5th International Symposium on Software Reliability Engineering, Monterey, CA, 6–9 Nov 1994, pp. 220–229Google Scholar
  22. 22.
    S.C.P.F. Fabbri, J.C. Maldonado, P.C. Masiero, M.E. Delamaro, W.E. Wong, Mutation testing applied to validate specifications based on Petri Nets, in Proceedings of the IFIP TC6 8th International Conference on Formal Description Techniques VIII (1995), pp. 329–337Google Scholar
  23. 23.
    S.C.P.F. Fabbri, J.C. Maldonado, T. Sugeta, P.C. Masiero, Mutation testing applied to validate specifications based on Statecharts, in Proceedings of the 10th International Symposium on Software Reliability Engineering (ISSRE’99), Boca Raton, FL, 1–4 Nov 1999, pp. 210Google Scholar
  24. 24.
    G. Fraser, A. Zeller, Mutation-driven generation of unit tests and oracles. IEEE Trans. Softw. Eng. 38(2), 278–292 (2012)CrossRefGoogle Scholar
  25. 25.
    T. Gibson-Robinson, P. Armstrong, A. Boulgakov, A.W. Roscoe, FDR3 - a modern refinement checker for CSP, in Tools and Algorithms for the Construction and Analysis of Systems (2014), pp. 187–201Google Scholar
  26. 26.
    M. Gligoric, L. Zhang, C. Pereira, G. Pokam, Selective mutation testing for concurrent code, in Proceedings of the 2013 International Symposium on Software Testing and Analysis, ISSTA 2013 (Association for Computing Machinery, New York, 2013), pp. 224–234CrossRefGoogle Scholar
  27. 27.
    P. Gómez-Abajo, E. Guerra, J. de Lara, A domain-specific language for model mutation and its application to the automated generation of exercises. Comput. Lang. Syst. Struct. 49, 152–173 (2017)Google Scholar
  28. 28.
    P. Gómez-Abajo, E. Guerra, J. de Lara, M.G. Merayo, A tool for domain-independent model mutation. Sci. Comput. Program. 163, 85–92 (2018)CrossRefGoogle Scholar
  29. 29.
    R. Gopinath, C. Jensen, A. Groce, Mutations: How close are they to real faults? in 25th IEEE International Symposium on Software Reliability Engineering, ISSRE 2014, Naples, 3–6 Nov 2014 (IEEE Computer Society, Washington, 2014), pp. 189–200Google Scholar
  30. 30.
    R. Gopinath, A. Alipour, I. Ahmed, C. Jensen, A. Groce, How hard does mutation analysis have to be, anyway? in 2015 IEEE 26th International Symposium on Software Reliability Engineering (ISSRE) (2015)Google Scholar
  31. 31.
    M. Harman, Y. Jia, W.B. Langdon, Strong higher order mutation-based test data generation, in Proceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on Foundations of software engineering (ESEC/FSE’11) (Association for Computing Machinery, New York, 2011), pp. 212–222Google Scholar
  32. 32.
    M. Harman, S.A. Mansouri, Y. Zhang, Search-based software engineering: trends, techniques and applications. ACM Comput. Surv. 45(1), 11:1–11:61 (2012)Google Scholar
  33. 33.
    Y. Jia, M. Harman, Constructing subtle faults using higher order mutation testing, in Eighth IEEE International Working Conference on Source Code Analysis and Manipulation (SCAM 2008), Beijing, 28–29 Sept 2008, pp. 249–258Google Scholar
  34. 34.
    Y. Jia, M. Harman, Milu: a customizable, runtime-optimized higher order mutation testing tool for the full C language, in Proceedings of the 3rd Testing: Academic and Industrial Conference Practice and Research Techniques (TAIC PART’08), Windsor, 29–31 Aug 2008, pp. 94–98Google Scholar
  35. 35.
    S.-W. Kim, J.A. Clark, J.A. McDermid, Investigating the effectiveness of object-oriented testing strategies using the mutation method. Softw. Test. Verif. Reliab. 11(3), 207–225 (2001)CrossRefGoogle Scholar
  36. 36.
    K.N. King, A.J. Offutt, A Fortran language system for mutation-based software testing. Softw. Pract. Exper. 21(7), 685–718 (1991)CrossRefGoogle Scholar
  37. 37.
    M. Kintis, M. Papadakis, Y. Jia, N. Malevris, Y. Le Traon, M. Harman, Detecting trivial mutant equivalences via compiler optimisations. IEEE Trans. Softw. Eng. 44(4), 308–333 (2018)CrossRefGoogle Scholar
  38. 38.
    E.W. Krauser, A.P. Mathur, V. Rego, High performance software testing on SIMD machines. IEEE Trans. Softw. Eng. 17(5), 403–423 (1991)CrossRefGoogle Scholar
  39. 39.
    W. Krenn, R. Schlick, S. Tiran, B. Aichernig, E. Jobstl, H. Brandl, MoMut::UML model-based mutation testing for UML, in 2015 IEEE 8th International Conference on Software Testing, Verification and Validation (ICST), April 2015, pp. 1–8Google Scholar
  40. 40.
    M. Kusano, C. Wang, Ccmutator: a mutation generator for concurrency constructs in multithreaded C/C++ applications, in Proceedings of the 28th IEEE/ACM International Conference on Automated Software Engineering, ASE’13 (IEEE Press, Piscataway, 2013), pp. 722–725Google Scholar
  41. 41.
    M.Z. Kwiatkowska, G. Norman, D. Parker, PRISM 4.0: Verification of probabilistic real-time systems, in Proceedings of Computer Aided Verification - 23rd International Conference, CAV 2011, Snowbird, UT, USA, July 14–20, 2011, ed. by G. Gopalakrishnan, S. Qadeer. Lecture Notes in Computer Science, vol. 6806 (Springer, Berlin, 2011), pp. 585–591Google Scholar
  42. 42.
    S.C. Lee, J. Offutt, Generating test cases for XML-based web component interactions using mutation analysis, in 12th International Symposium on Software Reliability Engineering (ISSRE 2001), Hong Kong, 27–30 Nov 2001, pp. 200–209Google Scholar
  43. 43.
    M. Linares-Vásquez, G. Bavota, M. Tufano, K. Moran, M. Di Penta, C. Vendome, C. Bernal-Cárdenas, D. Poshyvanyk, Enabling mutation testing for Android apps, in Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering, ESEC/FSE 2017 (Association for Computing Machinery, New York, 2017), pp. 233–244CrossRefGoogle Scholar
  44. 44.
    T. Loise, X. Devroey, G. Perrouin, M. Papadakis, P. Heymans, Towards security-aware mutation testing, in Proceedings of the 10th International Conference on Software Testing, Verification and Validation Workshops (ICSTW’17) (2017), pp. 97–102Google Scholar
  45. 45.
    Y. Lou, D. Hao, L. Zhang, Mutation-based test-case prioritization in software evolution, in 2015 IEEE 26th International Symposium on Software Reliability Engineering (ISSRE) (2015), pp. 46–57Google Scholar
  46. 46.
    Y.-S. Ma, Y.R. Kwon, J. Offutt, Inter-class mutation operators for Java, in 13th International Symposium on Software Reliability Engineering (ISSRE 2002), Annapolis, MD, 12–15 Nov 2002, pp. 352–366Google Scholar
  47. 47.
    Y.-S. Ma, A.J. Offutt, Y.-R. Kwon, Mujava: a mutation system for Java, in Proceedings of the 28th international Conference on Software Engineering (ICSE ’06), Shanghai, 20–28 May 2006, pp. 827–830Google Scholar
  48. 48.
    L. Madeyski, W. Orzeszyna, R. Torkar, M. Józala, Overcoming the equivalent mutant problem: a systematic literature review and a comparative experiment of second order mutation. IEEE Trans. Softw. Eng. 40(1), 23–42 (2014)CrossRefGoogle Scholar
  49. 49.
    P.R. Mateo, M.P. Usaola, J.L.F. Alemán, Validating second-order mutation at system level. IEEE Trans. Softw. Eng. 39(4), 570–587 (2013)CrossRefGoogle Scholar
  50. 50.
    S. Mirshokraie, A. Mesbah, K. Pattabiraman, Guided mutation testing for JavaScript Web applications. IEEE Trans. Softw. Eng. 41(5), 429–444 (2015)CrossRefGoogle Scholar
  51. 51.
    A. Miyazawa, P. Ribeiro, W. Li, A. Cavalcanti, J. Timmis, J. Woodcock, RoboChart: modelling and verification of the functional behaviour of robotic applications. Softw. Syst. Model. 18(5), 3097–3149 (2019)CrossRefGoogle Scholar
  52. 52.
    S. Moon, Y. Kim, M. Kim, S. Yoo, Ask the mutants: mutating faulty programs for fault localization, in Verification and Validation 2014 IEEE Seventh International Conference on Software Testing (2014), pp. 153–162Google Scholar
  53. 53.
    T. Mouelhi, Y. Le Traon, B. Baudry, Mutation analysis for security tests qualification, in Testing: Academic and Industrial Conference Practice and Research Techniques - MUTATION (TAICPART-MUTATION 2007), Sept 2007, pp. 233–242Google Scholar
  54. 54.
    A.S. Namin, J.H. Andrews, D.J. Murdoch, Sufficient mutation operators for measuring test effectiveness, in Proceedings of the 30th International Conference on Software Engineering, ICSE ’08 (Association for Computing Machinery, New York, 2008), pp. 351–360Google Scholar
  55. 55.
    A. Offutt, The coupling effect: fact or fiction. ACM SIGSOFT Softw. Eng. Notes 14(8), 131–140 (1989)CrossRefGoogle Scholar
  56. 56.
    J. Offutt, Investigations of the software testing coupling effect. ACM Trans. Softw. Eng. Methodol. 1(1), 5–20 (1992)CrossRefGoogle Scholar
  57. 57.
    A.J. Offutt, A. Lee, G. Rothermel, R.H. Untch, C. Zapf, An experimental determination of sufficient mutant operators. ACM Trans. Softw. Eng. Methodol. 5(2), 99–118 (1996)CrossRefGoogle Scholar
  58. 58.
    A.J. Offutt, J. Voas, J. Payne, Mutation operators for Ada. Technical report, Information and Software Systems Engineering, George Mason University, 1996Google Scholar
  59. 59.
    M. Papadakis, Y. Le Traon, Using mutants to locate “Unknown” faults, in Verification and Validation 2012 IEEE Fifth International Conference on Software Testing (2012), pp. 691–700Google Scholar
  60. 60.
    M. Papadakis, Y. Le Traon, Metallaxis-FL: mutation-based fault localization. Softw. Test. Verif. Reliab. 25(5–7), 605–628 (2015)CrossRefGoogle Scholar
  61. 61.
    M. Papadakis, N. Malevris, Automatic mutation test case generation via dynamic symbolic execution, in 2010 IEEE 21st International Symposium on Software Reliability Engineering (2010), pp. 121–130Google Scholar
  62. 62.
    M. Papadakis, N. Malevris, Mutation based test case generation via a path selection strategy. Inf. Softw. Technol. 54(9), 915–932 (2012)CrossRefGoogle Scholar
  63. 63.
    A.V. Pizzoleto, F.C. Ferrari, J. Offutt, L. Fernandes, M. Ribeiro, A systematic literature review of techniques and metrics to reduce the cost of mutation testing. J. Syst. Softw. 157, 110388 (2019)CrossRefGoogle Scholar
  64. 64.
    M. Polo, M. Piattini, I. García-Rodríguez, Decreasing the cost of mutation testing with second-order mutants. Softw. Test. Verif. Reliab. 19(2), 111–131 (2009)CrossRefGoogle Scholar
  65. 65.
    R. Ramler, T. Wetzlmaier, C. Klammer, An empirical study on the application of mutation testing for a safety-critical industrial software system, in Proceedings of the Symposium on Applied Computing, SAC 2017, Marrakech, April 3–7, 2017, ed. by A. Seffah, B. Penzenstadler, C. Alves, X. Peng (ACM, New York, 2017), pp. 1401–1408Google Scholar
  66. 66.
    A.W. Roscoe, Understanding Concurrent Systems. Texts in Computer Science (Springer, Berlin, 2011)Google Scholar
  67. 67.
    G. Rothermel, R.H. Untch, C. Chu, M.J. Harrold, Prioritizing test cases for regression testing. IEEE Trans. Softw. Eng. 27(10), 929–948 (2001)CrossRefGoogle Scholar
  68. 68.
    D. Schuler, A. Zeller, Javalanche: efficient mutation testing for Java, in Proceedings of the 7th joint meeting of the European Software Engineering Conference and the International Symposium on Foundations of Software Engineering, Amsterdam, 24–28 Aug 2009, pp. 297–298Google Scholar
  69. 69.
    H. Shahriar, M. Zulkernine, Mutation-based testing of buffer overflow vulnerabilities, in Proceedings of the 32nd Annual IEEE International Computer Software and Applications Conference, COMPSAC 2008, Turku, 28 July–1 Aug 2008, pp. 979–984Google Scholar
  70. 70.
    D. Shin, S. Yoo, D.-H. Bae, A theoretical and empirical study of diversity-aware mutation adequacy criterion. IEEE Trans. Softw. Eng. 44(10), 914–931 (2018)CrossRefGoogle Scholar
  71. 71.
    D. Shin, S. Yoo, M. Papadakis, D.-H. Bae, Empirical evaluation of mutation-based test case prioritization techniques. Softw. Test. Verif. Reliab. 29(1–2), e1695 (2019)Google Scholar
  72. 72.
    F.C. Souza, M. Papadakis, V. Durelli, M. Delamaro, Test data generation techniques for mutation testing: a systematic mapping (2014)Google Scholar
  73. 73.
    T. Srivatanakul, J.A. Clark, S. Stepney, F. Polack, Challenging formal specifications by mutation: a CSP security example, in Proceedings of the 10th Asia-Pacific Software Engineering Conference (APSEC’03) (2003), pp. 340–350Google Scholar
  74. 74.
    M. Trakhtenbrot, New mutations for evaluation of specification and implementation levels of adequacy in testing of Statecharts models, in Proceedings of the 3rd Workshop on Mutation Analysis (MUTATION’07), Windsor, 10–14 Sept 2007, pp. 151–160Google Scholar
  75. 75.
    J. Tuya, M.J. Suarez Cabal, C. de la Riva, SQLmutation: a tool to generate mutants of SQL database queries, in Proceedings of the 2nd Workshop on Mutation Analysis (MUTATION’06), Raleigh, NC, Nov 2006, p. 1Google Scholar
  76. 76.
    J. Tuya, M.J. Suarez Cabal, C. de la Riva, Mutating database queries. Inf. Softw. Technol. 49(4), 398–417 (2007)CrossRefGoogle Scholar
  77. 77.
    L. Zhang, S.-S. Hou, J.-J. Hu, T. Xie, H. Mei, Is operator-based mutant selection superior to random mutant selection? in Proceedings of the 32nd ACM/IEEE International Conference on Software Engineering - Volume 1, ICSE ’10 (Association for Computing Machinery, New York, 2010), pp. 435–444Google Scholar
  78. 78.
    L. Zhang, M. Gligoric, D. Marinov, S. Khurshid, Operator-based and random mutant selection: better together, in 2013 28th IEEE/ACM International Conference on Automated Software Engineering (ASE) (2013), pp. 92–102Google Scholar

Copyright information

© Springer Nature Switzerland AG 2021

Authors and Affiliations

  1. 1.The University of SheffieldSheffieldUK
  2. 2.Universidad Autónoma de MadridMadridSpain
  3. 3.University of BradfordBradfordUK
  4. 4.Universidad Complutense de MadridMadridSpain

Personalised recommendations