Advertisement

Automated Software Engineering

, Volume 21, Issue 1, pp 65–105 | Cite as

GUITAR: an innovative tool for automated testing of GUI-driven software

  • Bao N. Nguyen
  • Bryan Robbins
  • Ishan Banerjee
  • Atif Memon
Article

Abstract

Most of today’s software applications feature a graphical user interface (GUI) front-end. System testing of these applications requires that test cases, modeled as sequences of GUI events, be generated and executed on the software. We term GUI testing as the process of testing a software application through its GUI. Researchers and practitioners agree that one must employ a variety of techniques (e.g., model-based, capture/replay, manually scripted) for effective GUI testing. Yet, the tools available today for GUI testing are limited in the techniques they support. In this paper, we describe an innovative tool called GUITAR that supports a wide variety of GUI testing techniques. The innovation lies in the architecture of GUITAR, which uses plug-ins to support flexibility and extensibility. Software developers and quality assurance engineers may use this architecture to create new toolchains, new workflows based on the toolchains, and plug in a variety of measurement tools to conduct GUI testing. We demonstrate these features of GUITAR via several carefully crafted case studies.

Keywords

GUI testing Test automation Test generation 

References

  1. Alfaro, L.D., Henzinger, T.A.: Interface theories for component-based design. In: Proceedings of the First International Workshop on Embedded Software (EMSOFT ’01), pp. 148–165. Springer, London (2001) Google Scholar
  2. Amalfitano, D., Fasolino, A.R., Tramontana, P.: A GUI crawling-based technique for Android mobile application testing. In: Proceedings of the IEEE Fourth International Conference on Software Testing, Verification and Validation Workshops (ICSTW ’11), pp. 252–261. IEEE Comput. Soc., Los Alamitos (2011) CrossRefGoogle Scholar
  3. Artzi, S., Dolby, J., Jensen, S.H., Møller, A., Tip, F.: A framework for automated testing of javascript web applications. In: Proceedings of the 33rd International Conference on Software Engineering (ICSE ’11), pp. 571–580. ACM, New York (2011) Google Scholar
  4. Baresi, L., Young, M.: Test oracles. Technical report CIS-TR-01-02, University of Oregon, Dept. of Computer and Information Science, Eugene, Oregon, USA (2001) Google Scholar
  5. Belli, F.: Finite-state testing and analysis of graphical user interfaces. In: Proceedings of the 12th International Symposium on Software Reliability Engineering (ISSR ’01), p. 34. IEEE Comput. Soc., Washington (2001) CrossRefGoogle Scholar
  6. Brooks, P.A., Memon, A.M.: Automated GUI testing guided by usage profiles. In: Proceedings of the Twenty-Second IEEE/ACM International Conference on Automated Software Engineering (ASE ’07), pp. 333–342. ACM, New York (2007) CrossRefGoogle Scholar
  7. Brooks, P., Robinson, B., Memon, A.M.: An initial characterization of industrial graphical user interface systems. In: Proceedings of the 2nd IEEE International Conference on Software Testing, Verification and Validation (ICST 2009). IEEE Computer Society, Washington (2009) Google Scholar
  8. Cadar, C., Dunbar, D., Engler, D.: KLEE: unassisted and automatic generation of high-coverage tests for complex systems programs. In: Proceedings of the 8th USENIX Conference on Operating Systems Design and Implementation (OSDI ’08), pp. 209–224. USENIX Association, Berkeley (2008) Google Scholar
  9. Chang, T.H., Yeh, T., Miller, R.C.: GUI testing using computer vision. In: Conference on Human Factors in Computing Systems, pp. 1535–1544 (2010) Google Scholar
  10. Chen, J., Subramaniam, S.: A GUI environment to manipulate FSMs for testing GUI-based applications in Java. In: Conference on System Sciences, vol. 9, p. 9061 (2001) Google Scholar
  11. Chen, W.K., Tsai, T.H., Chao, H.H.: Integration of specification-based and CR-based approaches for GUI testing. In: Conference on Advanced Information Networking and Applications, pp. 967–972 (2005) Google Scholar
  12. Cunha, M., Paiva, A., Ferreira, H., Abreu, R.: PETTool: a pattern-based GUI testing tool. In: International Conference on Software Technology and Engineering, pp. 202–206 (2010) Google Scholar
  13. Draheim, D., Lutteroth, C., Weber, G.: A source code independent reverse engineering tool for dynamic web sites. In: Proceedings of the Ninth European Conference on Software Maintenance and Reengineering (CSMR ’05), pp. 168–177. IEEE Comput. Soc., Washington (2005) CrossRefGoogle Scholar
  14. Ganov, S., Kilmar, C., Khurshid, S., Perry, D.: Test generation for graphical user interfaces based on symbolic execution. In: Proceedings of the International Workshop on Automation of Software Test (2008) Google Scholar
  15. Grechanik, M., Xie, Q., Fu, C.: Creating GUI testing tools using accessibility technologies. In: Conference on Software Testing, Verification, and Validation, pp. 243–250 (2009) Google Scholar
  16. Hellmann, T.D., Hosseini Khayat, A., Maurer, F.: Supporting test-driven development of graphical user interfaces using agile interaction design. In: Proceedings of the 2010 Third International Conference on Software Testing, Verification, and Validation Workshops, pp. 444–447 (2010) CrossRefGoogle Scholar
  17. Huang, S., Cohen, M.B., Memon, A.M.: Repairing GUI test suites using a genetic algorithm. In: Proceedings of the 2010 Third International Conference on Software Testing, Verification and Validation (ICST ’10), pp. 245–254. IEEE Comput. Soc., Washington (2010) CrossRefGoogle Scholar
  18. Jääskeläinen, A., Katara, M., Kervinen, A., Maunumaa, M., Pääkkönen, T., Takala, T., Virtanen, H.: Automatic GUI test generation for smartphone applications—an evaluation. In: Proceedings of the Software Engineering in Practice Track of the 31st International Conference on Software Engineering (ICSE 2009), pp. 112–122 (companion volume). IEEE Computer Society, Los Alamitos (2009) Google Scholar
  19. McMaster, S., Memon, A.M.: An extensible heuristic-based framework for GUI test case maintenance. In: Proceedings of the First International Workshop on TESTing Techniques & Experimentation Benchmarks for Event-Driven Software (TESTBEDS ’09). IEEE Computer Society, Washington (2009) Google Scholar
  20. Memon, A.M.: An event-flow model of GUI-based applications for testing. Softw. Test. Verif. Reliab. 17, 137–157 (2007) CrossRefGoogle Scholar
  21. Memon, A.M., Nguyen, B.N.: Advances in automated model-based system testing of software applications with a GUI front-end. In: Zelkowitz, M.V. (ed.) Advances in Computers, vol. 80, pp. 121–162. Academic Press, San Diego (2010) Google Scholar
  22. Memon, A.M., Xie, Q.: Studying the fault-detection effectiveness of GUI test cases for rapidly evolving software. IEEE Trans. Softw. Eng. 31(10), 884–896 (2005) CrossRefGoogle Scholar
  23. Memon, A.M., Banerjee, I., Nagarajan, A.: GUI ripping: reverse engineering of graphical user interfaces for testing. In: Proceedings of the 10th Working Conference on Reverse Engineering (WCRE ’03), p. 260. IEEE Computer Society, Washington (2003) Google Scholar
  24. Memon, A.M., Pollack, M.E., Soffa, M.L.: Hierarchical GUI test case generation using automated planning. IEEE Trans. Softw. Eng. 27(2), 144–155 (2001a) CrossRefGoogle Scholar
  25. Memon, A.M., Soffa, M.L., Pollack, M.E.: Coverage criteria for GUI testing. In: Proceedings of the 8th European Software Engineering Conference Held Jointly with 9th ACM SIGSOFT International Symposium on Foundations of Software Engineering, vol. 26, pp. 256–267. ACM, New York (2001b) Google Scholar
  26. Mesbah, A., van Deursen, A.: Invariant-based automatic testing of AJAX user interfaces. In: Proceedings of the 31st International Conference on Software Engineering (ICSE ’09), pp. 210–220. IEEE Computer Society, Washington (2009) Google Scholar
  27. Myers, B.A.: User interface software tools. ACM Trans. Comput.-Hum. Interact. 2(1), 64–103 (1995) CrossRefGoogle Scholar
  28. Nguyen, D.H., Strooper, P., Suess, J.G.: Model-based testing of multiple GUI variants using the GUI test generator. In: Workshop on Automation of Software Test, pp. 24–30 (2010) CrossRefGoogle Scholar
  29. Paiva, A.C.R., Faria, J.C.P., Mendes, P.M.C.: Reverse engineered formal models for GUI testing. In: Proc. of Conf. on Formal Methods for Industrial Critical Systems, 4916(1), pp. 218–233 (2008) CrossRefGoogle Scholar
  30. Robinson, B., Francis, P., Ekdahl, F.: A defect-driven process for software quality improvement. In: Proceedings of the Second ACM-IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM ’08), pp. 333–335. ACM, New York (2008). doi: 10.1145/1414004.1414072 CrossRefGoogle Scholar
  31. Ruiz, A., Price, Y.W.: GUI testing made easy. In: Proceedings of the Testing: Academic & Industrial Conference—Practice and Research Techniques, pp. 99–103. IEEE Computer Society, Washington (2008) Google Scholar
  32. Shneiderman, B., Plaisant, C., Cohen, M., Jacobs, S.: Designing the User Interface: Strategies for Effective Human–Computer Interaction, 5th edn. Addison-Wesley, Reading (2009) Google Scholar
  33. Silva, J.L., Campos, J.C., Paiva, A.C.R.: Model-based user interface testing with spec explorer and ConcurTaskTrees. Electron. Notes Theor. Comput. Sci. 208, 77–93 (2008) CrossRefGoogle Scholar
  34. Silva, J.C., Saraiva, J., Campos, J.C.: A generic library for GUI reasoning and testing. In: ACM Symposium on Applied Computing, pp. 121–128 (2009) Google Scholar
  35. Staiger, S.: Static analysis of programs with graphical user interface. In: 11th European Conference on Software Maintenance and Reengineering (CSMR ’07), pp. 252–264 (2007) CrossRefGoogle Scholar
  36. Ural, H., Yang, B.: A test sequence selection method for protocol testing. IEEE Trans. Commun. 39(4), 514–523 (1991) CrossRefGoogle Scholar
  37. Veanes, M., Campbell, C., Grieskamp, W., Schulte, W., Tillmann, N., Nachmanson, L.: Model-based testing of object-oriented reactive systems with Spec Explorer. In: Hierons, R.M., Bowen, J.P., Harman, M. (eds.) Formal Methods and Testing, Chap.: Model-Based Testing of Object-Oriented Reactive Systems with Spec Explorer, pp. 39–76. Springer, Berlin (2008) CrossRefGoogle Scholar
  38. Vieira, M., Leduc, J., Hasling, B., Subramanyan, R., Kazmeier, J.: Automation of GUI testing using a model-driven approach. In: Conference on Software Engineering, pp. 9–14 (2006) Google Scholar
  39. Xie, Q., Memon, A.M.M.: Designing and comparing automated test oracles for GUI-based software applications. ACM Trans. Softw. Eng. Methodol. 16(1), 4 (2007) CrossRefGoogle Scholar
  40. Xie, Q., Memon, A.M.: Using a pilot study to derive a GUI model for automated testing. ACM Trans. on Softw. Eng. and Method. (2008) Google Scholar
  41. Yuan, X., Memon, A.M.: Generating event sequence-based test cases using GUI runtime state feedback. IEEE Trans. Softw. Eng. 36, 81–95 (2010) CrossRefGoogle Scholar
  42. Yuan, X., Cohen, M.B., Memon, A.M.: GUI interaction testing: incorporating event context. IEEE Trans. Softw. Eng. 37(4), 559–574 (2011) CrossRefGoogle Scholar
  43. Zhang, S., Saff, D., Bu, Y., Ernst, M.D.: Combined static and dynamic automated test generation. In: Proceedings of the 2011 International Symposium on Software Testing and Analysis (ISSTA ’11), pp. 353–363. ACM, New York (2011) CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2013

Authors and Affiliations

  • Bao N. Nguyen
    • 1
  • Bryan Robbins
    • 1
  • Ishan Banerjee
    • 1
  • Atif Memon
    • 1
  1. 1.Department of Computer ScienceUniversity of MarylandCollege ParkUSA

Personalised recommendations