Advertisement

A survey on formal active and passive testing with applications to the cloud

  • Ana R. Cavalli
  • Teruo Higashino
  • Manuel Núñez
Article

Introduction

Testing has become an integral part of innovation, production, and operation of systems. The activity of testing is already a flourishing area with the active participation of a large community of researchers and experts who are highly aware of its importance and impact for the future deployment and use of software and systems. Formal methods have proved to be very promising for developing automated and generic testing methods. Actually, the combination of formal methods and testing is currently well understood and tools to automate testing activities are widely available.

(Formal) testing is the assessment, by means of experiments, that a product conforms to its (formal) requirements. Test cases are designed to test particular aspects of the system, called test objectives. In order to formally obtain testing objectives, it is necessary to provide mathematical models for the semantic of the studied system, formal frameworks for testing, languages to describe the expected...

Keywords

Virtual Machine Session Initiation Protocol Finite State Machine Execution Trace Symbolic Execution 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.

Notes

Acknowledgements

This research has been partially supported by the Spanish MEC project ESTuDIo (TIN2012-36812-C02), the Comunidad de Madrid project SICOMORo-CM (S2013/ICE-3006), the Japanese Grants-in-Aid for Scientific Research(S) (26220001), the UCM-Santander program to fund research groups (group 910606), and the INTER-TRUST european project (FP7 program).

References

  1. 1.
    Alcalde B, Cavalli A, Chen D, Khuu D, Lee D (2004) Network protocol system passive testing for fault management: A backward checking approach. In: Proceedings of 24th IFIP Int. Conf. on Formal Techniques for Networked and Distributed Systems (FORTE’04), Lecture Notes in Computer Science, vol. 3235. Springer, Berlin, pp 150–166Google Scholar
  2. 2.
    Alur R, Dill D (1994) A theory of timed automata. Theor Comput Sci 126:183–235CrossRefMATHMathSciNetGoogle Scholar
  3. 3.
    Andrés C, Cambronero M, Nuñez M (2010) Formal passive testing of service-oriented systems. In: Proceedings of 2010 IEEE Int. Conf. on Services Computing (SCC 2010), pp 610-613Google Scholar
  4. 4.
    Andrés C, Merayo M, Núñez M (2008) Passive testing of timed systems. In: Proceedings of 6th Int. Symp. on Automated Technology for Verification and Analysis (ATVA 2008), Lecture Notes in Computer Science, vol.5311. Springer, Berlin, pp 418–427Google Scholar
  5. 5.
    Andrés C, Merayo MG, Núñez M (2012) Formal passive testing of timed systems: Theory and tools. Softw Test Verification Reliab 22(6):365–405CrossRefGoogle Scholar
  6. 6.
    Arnedo JA, Cavalli A, Núñez M (2003) Fast testing of critical properties through passive testing. In: Proceedings of 15th IFIP Int. Conf. on Testing of Communicating Systems (TestCom 2003), Lecture Notes in Computer Science, vol. 2644. Springer, Berlin, pp 295–310Google Scholar
  7. 7.
    Banzai T, Koizumi H, Kanbayashi R, Imada T, Hanawa T, Sato M (2010) D-cloud: Design of a software testing environment for reliable distributed systems using cloud computing technology. In: Proceedings of 10th IEEE/ACM Int. Conf.on Cluster,Cloud and Grid Computing (CCGrid’10), pp 631–636Google Scholar
  8. 8.
    Bayse E, Cavalli A, Nez MN, Zaïdi F (2005) A passive testing approach based on invariants: Application to the wap. Comput Netw 48(2):247–266CrossRefMATHGoogle Scholar
  9. 9.
    Bellard F (2005) Qemu, a fast and portable dynamic translator, a fast and portable dynamic translator. In: Proceedings of 2005 USENIX Annual Technical Conference (ATEC ’05), pp 41 41. USENIX AssociationGoogle Scholar
  10. 10.
    Benharref A, Dssouli R, Serhani M, En-Nouaary A, Glitho R (2007) New approach for efsm-based passive testing of web services. In: Proceedings of 19th IFIP Int. Conf. on Testing of Software and Communicating Systems (TestCom 2007),Lecture Notes in Computer Science, vol. 4581. Springer, Berlin, pp 13–27Google Scholar
  11. 11.
  12. 12.
    Buyya R, Ranjan R, Calheiros RN (2009) Modeling and simulation of scalable cloud computing environments and the CloudSim toolkit: Challenges and opportunities. In: Proceedings of 7th IEEE Int. Conf. High Performance Computing and Simulation (HPCS’09), pp 1–11Google Scholar
  13. 13.
    Cacciari L, Rafiq O (1999) Controllability and observability in distributed testing. Inf Softw Technol 41(11–12):767–780CrossRefGoogle Scholar
  14. 14.
    Casanova H, Legrand A, Quinson M (2008) SimGrid: a generic framework for large-scale distributed experiments. In: Proceedings of 10th Int. Conf. on Computer Modeling and Simulation (UKSIM’08), pp 126–131Google Scholar
  15. 15.
    Cavalli A, Gervy C, Prokopenko S (2003) New approaches for passive testing using an extended finite state machine specification. Inf Softw Technol 45(12):837–852CrossRefGoogle Scholar
  16. 16.
    Cavalli A, Grepet C, Maag S, Tortajada V (2004) A validation model for the dsr protocol. In: Proceedings of IEEE Int. Workshop on Wireless Ad Hoc Networking (WWAN’04), pp 768–773Google Scholar
  17. 17.
    Chan W, Mei L, Zhang Z (2009) Modeling and testing of cloud applications. In: Proceedings of 4th IEEE Int. Conf. on Asia-Pacific Services Computing (APSCC’09), pp 111–118Google Scholar
  18. 18.
    Chow TS (1978) Test design modeled by finite-state machines. IEEE Trans Softw Eng 4(3):178–187CrossRefMATHGoogle Scholar
  19. 19.
    Corwin EH (2011) Deep packet inspection: Shaping the internet and the implications on privacy and security. Inf Secur J A Glob Perspect 20(6):311–316CrossRefGoogle Scholar
  20. 20.
    Dorofeeva M, El-Fakih K, Yevtushenko N (2005) An improved conformance testing method. In: Proceedings of 25th IFIP Int. Conf. on Formal Techniques for Networked and Distributed Systems (FORTE’05). Springer, Berlin, pp 363–378Google Scholar
  21. 21.
    Dorofeeva R, El-Fakih K, Maag S, Cavalli AR, Yevtushenko N (2010) Fsm-based conformance testing methods: A survey annotated with experimental evaluation. Inf Softw Technol 52:1286–1297CrossRefGoogle Scholar
  22. 22.
    Duale AY, Uyar MU (2004) A method enabling feasible conformance test sequence generation for efsm models. IEEE Trans Comput 53(5):614–627CrossRefGoogle Scholar
  23. 23.
    El-Fakih K, Yevtushenko Nv, Bochmann G (2004) Fsm-based incremental conformance testing methods. IEEE Trans Softw Eng 30(7):425–436CrossRefGoogle Scholar
  24. 24.
    En-Nouaary A, Dssoul R, Khendek F, Elqortobi A (1998) Timed test cases generation based on state characterization technique. In: Proceedings of 19th IEEE Real-Time System Sympsium (RTSS’98), pp 220–229, IEEEGoogle Scholar
  25. 25.
    Fujiwara S v, Bochmann G, Khendek F, Amalou M, Ghedamsi A (1991) Test selection based on finite state models. IEEE Trans Softw Eng 17(6):591–603CrossRefGoogle Scholar
  26. 26.
    García-Guirado A, Fernández-Pascual R, García JM (2009) Virtual-GEMS: an infrastructure to simulate virtual machines. In: Proceedings of 5th annual workshop on modeling, benchmarking and simulation (MoBS’09), pp 1–10Google Scholar
  27. 27.
  28. 28.
    Gonenc G (1970) A method for the design of fault detection experiments. IEEE Trans Comput C-19(6):551–558CrossRefGoogle Scholar
  29. 29.
    Hallé S, Villemaire R (2009) Runtime monitoring of web service choreographies using streaming xml. In: 2009 ACM Symposium on Applied Computing,SAC’09, pp 2118–2125Google Scholar
  30. 30.
    Hierons R, Merayo M, Núñez M (2013) Passive testing with asynchronous communications. In: Proceedings of 33rd IFIP Int. Conf. on Formal Techniques for Distributed Systems (FORTE’13), Lecture Notes in Computer Science, vol. 7892. Springer, Berlin, pp 99–113Google Scholar
  31. 31.
    Hierons RM (2012) Oracles for distributed testing. IEEE Trans Softw Eng 38(3):629–641CrossRefGoogle Scholar
  32. 32.
    Hierons RM, Merayo MG, Núñez M (2012) Implementation relations and test generation for systems with distributed interfaces. Distrib Comput 25(1):35–62CrossRefMATHGoogle Scholar
  33. 33.
    Higashino Tv, Bochmann G (1994) Automatic analysis and test case derivation for a restricted class of lotos expressions with data parameters. IEEE Trans Softw Eng 20(1):29–42CrossRefGoogle Scholar
  34. 34.
    Higashino T, Nakata A, Taniguchi K, Cavalli AR (1999) Generating test cases for a timed i/o automaton model. In: Proceedings of 12th IFIP Int. Workshop on Testing of Communicating Systems (IWTCS’99). Springer, Berlin, pp 197–214Google Scholar
  35. 35.
    Hiromori A, Umedu T, Yamaguchi H, Higashino T (2012) Protocol testing and performance evaluation for manets with non-uniform node density distribution. In: Proceedings of 24th IFIP Int. Conf. on Testing Software and Systems (ICTSS 2012). Springer, Berlin, pp 231–246Google Scholar
  36. 36.
    Jard C, Jéron T, Kahlouche H, Viho C (1998) Towards automatic distribution of testers for distributed conformance testing. In: Proceedings of Int. Conf. on Formal Description Techniques and Protocol Specification, Testing and Verification (FORTE’98/PSTV’98). Kluwer Academic Publishers, Norwell, pp 353–368Google Scholar
  37. 37.
    Joshi P, Gunawi H, Sen K (2011) PREFAIL: a programmable tool for multiple-failure injection. ACM SIGPLAN Not 46(10):171–188CrossRefGoogle Scholar
  38. 38.
    Kliazovich D, amd SU, Khan PB (2012) GreenCloud: a packet-level simulator of energy-aware cloud computing data centers. J Supercomput 62(3):1263–1283CrossRefGoogle Scholar
  39. 39.
    Ladani B, Alcalde B, Cavalli A (2005) Passive testing: A constrained invariant checking approach. In: Proceedings of 17th IFIP Int. Conf. on Testing of Communicating Systems (TestCom 2005), Lecture Notes in Computer Science, vol.3502. Springer, Berlin, pp 9–22Google Scholar
  40. 40.
    Lai R (2002) A survey of communication protocol testing. J Syst Softw 62:21–46CrossRefGoogle Scholar
  41. 41.
    Lalanne F, Maag S (2013) A formal data-centric approach for passive testing of communication protocols. IEEE/ACM Trans Netw 21(3):788–801CrossRefGoogle Scholar
  42. 42.
    Lee D, Chen D, Hao R, Miller R, Wu J, Yin X (2002) A formal approach for passive testing of protocol data portions. In: Proceedings of 10th IEEE Int. Conf. on Network Protocols (ICNP 2002), pp 122–131Google Scholar
  43. 43.
    Lee D, Chen D, Hao R, Miller R, Wu J, Yin X (2006) Network protocol system monitoring-a formal approach with passive testing. IEEE/ACM Trans Netw 14(2):424–437CrossRefGoogle Scholar
  44. 44.
    Lee D, Netravali A, Sabnani K, Sugla B, John A (1997) Passive testing and applications to network management. In: Proceedings of 5th IEEE Int. Conf. on Network Protocols (ICNP’97), pp 113–122Google Scholar
  45. 45.
    Lee D, Yannakakis M (1996) Principles and methods of testing finite state machines-a survey. Proc IEEE 84(8):1090–1123CrossRefGoogle Scholar
  46. 46.
    Leucker M, Schallhart C (2009) A brief account of runtime verification. J Logic Algebraic Program 78(5):293–303CrossRefMATHGoogle Scholar
  47. 47.
    Li X, Higashino T, Higuchi M, Taniguchi K (1994) Automatic generation of extended uio sequences for communication protocols in an efsm model. In: Proceedings of 7th IFIP Int. Workshop on Protocol Test Systems (IWPTS’94). Springer, Berlin, pp 225–240Google Scholar
  48. 48.
    Luo G, Dssouli R, Bochmann Gv (1993) Generating synchronizable test sequences based on finite state machine with distributed ports. In: Proceedings of 6th IFIP Int. Workshop on Protocol Test Systems (IWPTS’93), pp 139–153. North-HollandGoogle Scholar
  49. 49.
    Maag S, Zaidi F (2006) Testing methodology for an ad hoc routing protocol. In: Proceedings of 2006 ACM Int. Workshop on Performance Monitoring,Measurement, and Evaluation of Heterogeneous Wireless and Wired Networks, pp 48–55Google Scholar
  50. 50.
    Mammar A, Cavalli A, Jimenez W, Mallouli W, Montes E (2011) Using testing techniques for vulnerability detection in c programs. In: Proceedings of 23rd IFIP Int. Conf. on Testing Software and Systems (ICTSS’11), Lecture Notes in Computer Science, vol. 7019. Springer, Berlin, pp 80–96Google Scholar
  51. 51.
    Mehrotra N (2010) Cloud-testing vs. testing a cloud. In: Proceedings 10th Int. Software Testing Conf., pp 1–7Google Scholar
  52. 52.
    Mell P, Scarfone K (2007) Guide to intrusion detection and prevention systems (IDPS). http://csrc.nist.gov/publications/nistpubs/800-94/SP800-94.pdf
  53. 53.
    Miller R (1998) Passive testing of networks using a cfsm specification. In: Proceedings of 1998 IEEE Int. Conf. on Performance Computing and Communications (IPCCC ’98), pp 111–116Google Scholar
  54. 54.
    Miller R, Arisha K (2000) On fault location in networks by passive testing. In: Proceedings of 2000 IEEE Int. Conf. on Performance, Computing, Communications (IPCCC ’00), pp 281–287Google Scholar
  55. 55.
    Miller R, Arisha K (2001) Fault identification in networks by passive testing. In: Proceedings of 34th Simulation Symp., pp 277–284Google Scholar
  56. 56.
    Morales G, Maag S, Cavalli A, Mallouli W, Montes E, Wehbi B (2010) Timed extended invariants for the passive testing of web services. In: Proceedings of 2010 IEEE Int. Conf. on Web Services (ICWS 2010), pp 592–599Google Scholar
  57. 57.
    Mouttappa P, Maag S, Cavalli A (2012) An iosts based passive testing approach for the validation of data-centric protocols. In: Proceedings of 12th Int. Conf. on Quality Software (QSIC 2012), pp 49–58Google Scholar
  58. 58.
    Mouttappa P, Maag S, Cavalli A (2013) Monitoring based on iosts for testing functional and security properties: Application to an automotive case study. In: Proceedings of 37th IEEE Int. Conf. on Computer Software and Applications (COMPSAC ’13), pp 1–10Google Scholar
  59. 59.
    Mouttappa P, Maag S, Cavalli A (2013) Using passive testing based on symbolic execution and slicing techniques: Application to the validation of communication protocols. Comput Netw 57(15):2992–3008CrossRefGoogle Scholar
  60. 60.
    Nguyen H, Poizat P, Zaidi F (2012) Online verification of value-passing choreographies through property-oriented passive testing. In: Proceedings of 14th IEEE Int. Symp. on High-Assurance Systems Engineering (HASE 2012), pp 106–113Google Scholar
  61. 61.
    Núñez A, Vazquez-Polettí JL, Caminero AC, Castañé GG, Carretero J, Llorente IM (2012) iCanCloud: a flexible and scalable cloud infrastructure simulator. J Grid Comput 10(1):185–209CrossRefGoogle Scholar
  62. 62.
    Petrenko A, Yevtushenko N, Lebedev A, Das A (1993) Nondeterministic state machines in protocol conformance testing. In: Proceedings of 6th IFIP Int. Workshop on Protocol Test Systems (IWPTS’93), pp 363–378.North-HollandGoogle Scholar
  63. 63.
    Rafiq O, Cacciari L (2003) Coordination algorithm for distributed testing. J Supercomput 24(2):203–211CrossRefMATHGoogle Scholar
  64. 64.
    Richters M, Gogolla M (2003) Aspect-oriented monitoring of UML and OCL constraints. In: 4th Int. Workshop on Aspect-Oriented Modeling with UML on 6th Int. Conf. on the Unified Modeling Language (UML’03)Google Scholar
  65. 65.
    Rumbaugh J, Blaha M, Premerlani W, Eddy F, Lorensen WE, et al. (1991) Object-oriented modeling and design. Prentice-Hall Englewood Cliffs, NJGoogle Scholar
  66. 66.
    Cheung SC, STC, Xu Z (2001) Toward generic timing tests for distributed multimedia software systems. In: Proceedings of 12th, I E E E Int. Symp. on Software Reliability Engineering (ISSRE’01), pp 210–220Google Scholar
  67. 67.
    Sabnani K, Dahbura A (1988) A protocol test generation procedure. Comput Netw ISDN Syst 15(4):285–297CrossRefGoogle Scholar
  68. 68.
    Shin KS, Jung JH, Cheon JY, Choi SB (2007) Real-time network monitoring scheme based on SNMP for dynamic information. J Netw Comput Appl 30(1):331–353CrossRefGoogle Scholar
  69. 69.
    Sidhu DP, Leung TK (1989) Formal methods for protocol testing: A detailed study. IEEE Trans Softw Eng 15(4):413–426CrossRefGoogle Scholar
  70. 70.
    Simmonds J (2011) Dynamic analysis of web services.Ph.D. thesis, University of TorontoGoogle Scholar
  71. 71.
  72. 72.
    Springintveld J, Vaandrager F, D’Argenio PR (2001) Testing timed automata. Theor Comput Sci 254(1-2):225–257CrossRefMATHMathSciNetGoogle Scholar
  73. 73.
    Sun T, Yasumoto K, Mori M, Higashino T (2003) Qos functional testing for multi-media systems. In: Proceedings of 23rd IFIP Int. Conf. on Formal Techniques for Networked and Distributed Systems (FORTE’03). Springer, Berlin, pp 319–334Google Scholar
  74. 74.
    Tabourier M, Cavalli A (1999) Passive testing and application to the gsm-map protocol. Inf Softw Technol 41(11-12):813–821CrossRefGoogle Scholar
  75. 75.
    Ural H, Williams C (2006) Constructing checking sequences for distributed testing. Form Asp Comput 18(1):84–101CrossRefMATHGoogle Scholar
  76. 76.
    Ural H, Xu Z (2007) An efsm-based passive fault detection approach. In: Proceedings of 19th IFIP Int. Conf. on Testing of Software and Communicating Systems (TestCom2007),Lecture Notes in Computer Science, vol 4581. Springer, Berlin, pp 335–350Google Scholar
  77. 77.
    Ural H, Xu Z, Zhang F (2007) An improved approach to passive testing of fsm-based systems. In: Proceedings of 2nd IEEE Int. Workshop on Automation of Software Test (AST ’07), p 6Google Scholar
  78. 78.
    Misic V, STC, Cheung SC (1998) Towards a framework for testing distributed multimedia software systems. In: Proceedings of 1998 IEEE Int. Symp. on Software Engineering for Parallel and Distributed Systems (PDSE’98), pp 72 81Google Scholar
  79. 79.
  80. 80.
    Vuong ST, Chan W, Ito M (1989) The uiov-method for protocol test sequence generation. In: Proceedings of 2nd IFIP Int. Workshop on Protocol Test Systems (IWPTS’89), pp 161–175. North-HollandGoogle Scholar
  81. 81.
    Wang CJ, Liu MT (1992) Axiomatic test sequence generation for extended finite state machines. In: Proceedings of 12th IEEE Int. Conf. on Distributed Computing Systems (ICDCS-12), pp 252–259Google Scholar
  82. 82.
    Wehbi B, Montes E, Bourdelles M (2012) Events-based security monitoring using mmt tool. In: Proceedings of 5th IEEE Int. Conf. on Software Testing,Verification and Validation (ICST2012), pp 860–863Google Scholar
  83. 83.
    Wu J, Zhao Y, Yin X (2002) 22nd IFIP Int. Conf. on Formal Techniques for Networked and Distributed Systems (FORTE 2002). Springer, Berlin, pp 101–116Google Scholar
  84. 84.
    Xiaoping C, Lalanne F, Maag S (2012) A logic-based passive testing approach for the validation of communicating protocols. In: Proceedings of 7th Int. Conf. on Evaluation of Novel Approaches to Software Engineering (ENASE’12), pp 53–64Google Scholar
  85. 85.
    Zeng W, Wang Y (2009) Design and implementation of server monitoring system based on snmp. In: Proceedings of 2009 Conf. on Artificial Intelligence (JCAI ’09), pp 680–682Google Scholar
  86. 86.
    Zhao Y, Yin X, Wu J (2003) Problems in the information dissemination of the internet routing. J Comput Sci Technol 18(2):139–152CrossRefMATHMathSciNetGoogle Scholar

Copyright information

© Institut Mines-Télécom and Springer-Verlag France 2015

Authors and Affiliations

  • Ana R. Cavalli
    • 1
  • Teruo Higashino
    • 2
  • Manuel Núñez
    • 3
  1. 1.Télécom SudParisEvryFrance
  2. 2.Graduate School of Information Science and TechnologyOsaka UniversityOsakaJapan
  3. 3.Departamento de Sistemas Informáticos y ComputaciónUniversidad Complutense de MadridMadridSpain

Personalised recommendations