Software Quality Journal

, Volume 19, Issue 4, pp 801–839 | Cite as

Automated GUI performance testing

  • Andrea Adamoli
  • Dmitrijs Zaparanuks
  • Milan Jovic
  • Matthias Hauswirth
Article

Abstract

A significant body of prior work has devised approaches for automating the functional testing of interactive applications. However, little work exists for automatically testing their performance. Performance testing imposes additional requirements upon GUI test automation tools: the tools have to be able to replay complex interactive sessions, and they have to avoid perturbing the application’s performance. We study the feasibility of using five Java GUI capture and replay tools for GUI performance test automation. Besides confirming the severity of the previously known GUI element identification problem, we also describe a related problem, the temporal synchronization problem, which is of increasing importance for GUI applications that use timer-driven activity. We find that most of the tools we study have severe limitations when used for recording and replaying realistic sessions of real-world Java applications and that all of them suffer from the temporal synchronization problem. However, we find that the most reliable tool, Pounder, causes only limited perturbation and thus can be used to automate performance testing. Based on an investigation of Pounder’s approach, we further improve its robustness and reduce its perturbation. Finally, we demonstrate in a set of case studies that the conclusions about perceptible performance drawn from manual tests still hold when using automated tests driven by Pounder. Besides the significance of our findings to GUI performance testing, the results are also relevant to capture and replay-based functional GUI test automation approaches.

Keywords

Performance testing Graphical user interfaces Test automation Perfomance analysis 

References

  1. Adamoli, A., & Hauswirth, M. (2010). Trevis: A context tree visualization & analysis framework and its use for classifying performance failure reports. In SoftVis ’10: Proceedings of the ACM symposium on software visualization.Google Scholar
  2. Alsmadi, I. (2008). The utilization of user sessions in testing. In ICIS ’08: Proceedings of the seventh IEEE/ACIS international conference on computer and information science (icis 2008) (pp. 581–585). Washington, DC, USA: IEEE Computer Society.Google Scholar
  3. Belli, F. (2001). Finite-state testing and analysis of graphical user interfaces. Software reliability engineering, international symposium on, 0:34.Google Scholar
  4. Blackburn, S. M., Garner, R., Hoffmann, C., Khang, A. M., McKinley, K. S., Bentzur, R., et al. (2006). The dacapo benchmarks: Java benchmarking development and analysis. In OOPSLA ’06: Proceedings of the 21st annual ACM SIGPLAN conference on object-oriented programming systems, languages, and applications (pp. 169–190). New York, NY, USA: ACM.Google Scholar
  5. Brooks, P. A., & Memon, A. M. (2007). Automated gui testing guided by usage profiles. In ASE ’07: Proceedings of the twenty-second IEEE/ACM international conference on Automated software engineering (pp. 333–342). New York, NY, USA: ACM.Google Scholar
  6. Brooks, P. A., Robinson, B. P., & Memon, A. M. (2009). An initial characterization of industrial graphical user interface systems. Software testing, verification, and validation, 2008 international conference on, 0:11–20.Google Scholar
  7. Chang, T.-H., Yeh, T., & Miller, R. C. (2010). Gui testing using computer vision. In Proceedings of the 28th international conference on human factors in computing systems, CHI ’10 (pp. 1535–1544). New York, NY, USA: ACM.Google Scholar
  8. Chinnapongse, V., Lee, I., Sokolsky, O., Wang, S., & Jones, P. L. (2009). Model-based testing of gui-driven applications. In Proceedings of the 7th IFIP WG 10.2 international workshop on software technologies for embedded and ubiquitous systems, SEUS ’09 (pp. 203–214). Berlin, Heidelberg: Springer-Verlag.Google Scholar
  9. de Oliveira, D. A. S., Crandall, J. R., Wassermann, G., Felix Wu, S., Su, Z., & Chong, F. T. (2006). Execrecorder: Vm-based full-system replay for attack analysis and system recovery. In ASID ’06: Proceedings of the 1st workshop on architectural and system support for improving software dependability (pp. 66–71). New York, NY, USA: ACM.Google Scholar
  10. Deursen, A.,, & Mesbah, A. (2010). Research issues in the automated testing of ajax applications. In Proceedings of the 36th conference on current trends in theory and practice of computer science, SOFSEM ’10 (pp. 16–28). Berlin, Heidelberg: Springer-Verlag.Google Scholar
  11. El Ariss, O., Xu, D., Dandey, S., Vender, B., McClean, P., & Slator, B. (2010). A systematic capture and replay strategy for testing complex gui based java applications. In Proceedings of the 2010 seventh international conference on information technology: New generations, ITNG ’10 (pp. 1038–1043). Washington, DC, USA: IEEE Computer Society.Google Scholar
  12. Elbaum, S., Karre, S., & Rothermel, G. (2003). Improving web application testing with user session data. In Proceedings of the 25th international conference on software engineering, ICSE ’03 (pp. 49–59). Washington, DC, USA: IEEE Computer Society.Google Scholar
  13. Elbaum, S., Rothermel, G., Karre, S., & Fisher, M., II. (2005). Leveraging user-session data to support web application testing. IEEE Transactions on Software Engineering, 31, 187–202.CrossRefGoogle Scholar
  14. Georges, A., Buytaert, D., & Eeckhout, L. (2007). Statistically rigorous java performance evaluation. In OOPSLA ’07: Proceedings of the 22nd annual ACM SIGPLAN conference on object-oriented programming systems and applications (pp. 57–76). New York, NY, USA: ACM.Google Scholar
  15. Grechanik, M., Xie, Q., Fu, C. (2009). Maintaining and evolving gui-directed test scripts. In ICSE ’09: Proceedings of the 2009 IEEE 31st international conference on software engineering (pp. 408–418). Washington, DC, USA: IEEE Computer Society.Google Scholar
  16. Hackner, D. R., & Memon, A. M. (2008). Test case generator for guitar. In Companion of the 30th international conference on software engineering, ICSE Companion ’08 (pp. 959–960). New York, NY, USA: ACM.Google Scholar
  17. Jovic, M., Adamoli, A., Zaparanuks, D., & Hauswirth, M. (2010) Automating performance testing of interactive java applications. In: AST ’10: Proceedings of the 5th Workshop on Automation of Software Test, pp. 8–15, New York, NY, USA, 2010. ACM.Google Scholar
  18. Jovic, M., & Hauswirth, M. (2008). Measuring the performance of interactive applications with listener latency profiling. In: PPPJ ’08: Proceedings of the 6th international symposium on principles and practice of programming in java (pp. 137–146). New York, NY, USA: ACM.Google Scholar
  19. Kasik, D. J., & George, H. G. (1996). Toward automatic generation of novice user test scripts. In Proceedings of the SIGCHI conference on human factors in computing systems: Common ground, CHI ’96 (pp. 244–251). New York, NY, USA: ACM.Google Scholar
  20. Li, P., Huynh, T., Reformat, M., & Miller, J. (2007). A practical approach to testing gui systems. Empirical Software Engineering, 12, 331–357.CrossRefGoogle Scholar
  21. Li, K., Wu, M. (2004). Effective GUI testing automation: Developing an automated GUI testing tool. Alameda, CA, USA: SYBEX Inc.Google Scholar
  22. Lindvall, M., Rus, I., Donzelli, P., Memon, A., Zelkowitz, M., Betin-Can, A, et al. (2007). Experimenting with software testbeds for evaluating new technologies. Empirical Software Engineering: An International Journal, 12(4), 417–444.CrossRefGoogle Scholar
  23. Liu, H., Jin, H., Liao, X., Hu, L., & Yu, C. (2009). Live migration of virtual machine based on full system trace and replay. In HPDC ’09: Proceedings of the 18th ACM international symposium on high performance distributed computing (pp. 101–110). New York, NY, USA: ACM.Google Scholar
  24. Liu, C.-H., Kung, D. C., Hsia, P., Hsu, C.-T. (2000a). Object-based data flow testing of web applications. In Proceedings of the the first Asia-Pacific conference on quality software (APAQS’00), APAQS ’00 (pp. 7–16). Washington, DC, USA: IEEE Computer Society.Google Scholar
  25. Liu, C.-H., Kung, D. C., Hsia, P., & Hsu, C.-T. (2000b). Structural testing of web applications. In Proceedings of the 11th international symposium on software reliability engineering (pp. 84–96). Washington, DC, USA: IEEE Computer Society.Google Scholar
  26. Lowell, C., & Stell-Smith, J. (2003) Successful automation of gui driven acceptance testing. In Proceedings of the 4th international conference on extreme programming and agile processes in software engineering, XP’03 (pp. 331–333). Berlin, Heidelberg: Springer-Verlag.Google Scholar
  27. Lucca, G. D., Fasolino, A., & Faralli, F. (2002). Testing web applications. In Proceedings of the international conference on software maintenance (ICSM’02) (pp. 310–319). Washington, DC, USA: IEEE Computer Society.Google Scholar
  28. Marchetto, A., Ricca, F., & Tonella, P. (2008a). A case study-based comparison of web testing techniques applied to ajax web applications. International Journal of Software Tools and Technology Transactions, 10, 477–492.CrossRefGoogle Scholar
  29. Marchetto, A., Tonella, P., & Ricca, F. (2008b). State-based testing of ajax web applications. In ICST (pp. 121–130). IEEE Computer Society.Google Scholar
  30. McMaster, S., & Memon, A. (2008). Call-stack coverage for gui test suite reduction. ACM Transactions of Software Engineering, 34, 99–115.CrossRefGoogle Scholar
  31. McMaster, S., & Memon, A. M. (2009). An extensible heuristic-based framework for gui test case maintenance. In TESTBEDS ’09: Proceedings of the first international workshop on TESTing techniques & experimentation benchmarks for event-driven software.Google Scholar
  32. Memon, A. M. (2008). Automatically repairing event sequence-based gui test suites for regression testing. ACM Transactions of Software Engineering Methodology, 18(2), 1–36.CrossRefGoogle Scholar
  33. Memon, A. M., Pollack, M. E., & Soffa, M. L. (2001) Hierarchical gui test case generation using automated planning. IEEE Transactions of Software Engineering, 27, 144–155.CrossRefGoogle Scholar
  34. Memon, A., Nagarajan, A., & Xie, Q. (2005). Automating regression testing for evolving gui software. Journal of Software Maintenance, 17, 27–64.CrossRefGoogle Scholar
  35. Memon, A. M., & Xie, Q. (2005). Studying the fault-detection effectiveness of gui test cases for rapidly evolving software. IEEE Transactions of Software Engineering, 31, 884–896.CrossRefGoogle Scholar
  36. Mesbah, A. , & van Deursen, A. (2009). Invariant-based automatic testing of ajax user interfaces. In Proceedings of the 31st international conference on software engineering, ICSE ’09 (pp. 210–220). Washington, DC, USA: IEEE Computer Society.Google Scholar
  37. Meszaros, G. (2003). Agile regression testing using record & playback. In Companion of the 18th annual ACM SIGPLAN conference on object-oriented programming, systems, languages, and applications, OOPSLA ’03 (pp. 353–360). New York, NY, USA: ACM.Google Scholar
  38. Mitchell, A., & James F. Power. An approach to quantifying the run-time behaviour of java gui applications. In: WISICT ’04: Proceedings of the winter international symposium on Information and communication technologies, pp. 1–6. Trinity College Dublin, 2004.Google Scholar
  39. Mu, B, Zhan, M., & Hu, L. (2009). Design and implementation of gui automated testing framework based on xml. In Proceedings of the 2009 WRI world congress on software engineering—Vol. 04, WCSE ’09 (pp. 194–199). Washington, DC, USA: IEEE Computer Society.Google Scholar
  40. Mytkowicz, T., Diwan, A., Hauswirth, M., & Sweeney, P. F. (2009). Producing wrong data without doing anything obviously wrong! In: ASPLOS ’09: Proceeding of the 14th international conference on architectural support for programming languages and operating systems (pp. 265–276). New York, NY, USA: ACM.Google Scholar
  41. Mytkowicz, T., Diwan, A., Hauswirth, M., & Sweeney, P. F. (2010). Evaluating the accuracy of java profilers. In PLDI ’10: Proceedings of the 2010 ACM SIGPLAN conference on programming language design and implementation (pp. 187–197). New York, NY, USA: ACM.Google Scholar
  42. Narayanasamy, S., Pokam, G., & Calder, B. (2005). Bugnet: Continuously recording program execution for deterministic replay debugging. In ISCA ’05: Proceedings of the 32nd annual international symposium on computer architecture (pp. 284–295). Washington, DC, USA: IEEE Computer Society.Google Scholar
  43. Nguyen, D. H., Strooper, P., & Suess, J. G. (2010). Model-based testing of multiple gui variants using the gui test generator. In Proceedings of the 5th workshop on automation of software test, AST ’10 (pp. 24–30). New York, NY, USA: ACM.Google Scholar
  44. Ricca, F., & Tonella, P. (2001). Analysis and testing of web applications. In Proceedings of the 23rd international conference on software engineering, ICSE ’01 (pp. 25–34). Washington, DC, USA: IEEE Computer Society.Google Scholar
  45. Ronsse, M., & Bosschere, K. D. (1999). Recplay: a fully integrated practical record/replay system. ACM Transactions on Computer System, 17(2), 133–152.CrossRefGoogle Scholar
  46. Ronsse, M., De Bosschere, K., Christiaens, M., de Kergommeaux, J. C., & Kranzlmüller, D. (2003). Record/replay for nondeterministic program executions. Commun. ACM, 46(9):62–67.CrossRefGoogle Scholar
  47. Ruiz, A., & Price, Y. W. (2007). Test-driven gui development with testng and abbot. IEEE Software, 24, 51–57.CrossRefGoogle Scholar
  48. Ruiz, A., & Price, Y. W. (2008). Gui testing made easy. In Proceedings of the testing: Academic & industrial conference—practice and research techniques (pp. 99–103). Washington, DC, USA: IEEE Computer Society.Google Scholar
  49. Sampath, S. (2004). Towards defining and exploiting similarities in web application use cases through user session analysis. In Proceedings of the second international workshop on dynamic analysis.Google Scholar
  50. Shehady, R. K., & Siewiorek, D. P. (1997). A method to automate user interface testing using variable finite state machines. In Proceedings of the 27th international symposium on fault-tolerant computing (FTCS ’97), FTCS ’97 (p. 80). Washington, DC, USA: IEEE Computer Society.Google Scholar
  51. Steven, J., Chandra, P., Fleck, B., & Podgurski, A. (2000). jRapture: A Capture/Replay tool for observation-based testing. SIGSOFT Software Engineering Notes, 25(5), 158–167.CrossRefGoogle Scholar
  52. Strecker, J., & Memon, A. M. (2008). Relationships between test suites, faults, and fault detection in gui testing. In ICST ’08: Proceedings of the first international conference on software testing, verification, and validation. Washington, DC, USA: IEEE Computer Society.Google Scholar
  53. Sun, Y., & Jones, E. L. (2004). Specification-driven automated testing of gui-based java programs. In Proceedings of the 42nd annual Southeast regional conference, ACM-SE 42 (pp. 140–145). New York, NY, USA: ACM.Google Scholar
  54. Sun Microsystems. (2004). Java Virtual Machine Tool Interface (JVMTI), http://www.java.sun.com/j2se/1.5.0/docs/guide/jvmti.
  55. Silva, J. C., Saraiva, J., & Campos, J. C. (2009). A generic library for gui reasoning and testing. In Proceedings of the 2009 ACM symposium on applied computing, SAC ’09 (pp. 121–128). New York, NY, USA: ACM.Google Scholar
  56. White, L., & Almezen, H. (2000). Generating test cases for gui responsibilities using complete interaction sequences. Software reliability engineering, international symposium on, 0:110.Google Scholar
  57. Xie, Q. (2006). Developing cost-effective model-based techniques for gui testing. In Proceedings of the 28th international conference on software engineering, ICSE ’06 (pp. 997–1000), New York, NY, USA: ACM.Google Scholar
  58. Xie, Q., & Memon, A. M. (2007). Designing and comparing automated test oracles for gui-based software applications. ACM Transactions of Software Engineering Methodology.Google Scholar
  59. Xie, Q., & Memon, A. M. (2008) Using a pilot study to derive a gui model for automated testing. ACM Transactions of Software Engineering Methodology, 18, 7:1–7:35.Google Scholar
  60. Yuan, X., Cohen, M. B., & Memon, A. M. (2009). Towards dynamic adaptive automated test generation for graphical user interfaces. In Proceedings of the IEEE international conference on software testing, verification, and validation workshops (pp. 263–266). Washington, DC, USA, 2009. IEEE Computer Society.Google Scholar
  61. Yang, J.-T., Huang, J.-L., Wang, F.-J., & Chu, W. C. (1999). An object-oriented architecture supporting web application testing. In 23rd international computer software and applications conference, COMPSAC ’99 (pp. 122–127). Washington, DC, USA: IEEE Computer Society.Google Scholar
  62. Yuan, X., & Memon, A. M. (2007). Using GUI run-time state as feedback to generate test cases. In ICSE ’07: Proceedings of the 29th international conference on software engineering (pp. 396–405). Washington, DC, USA: IEEE Computer Society.Google Scholar
  63. Yuan, X., & Memon, A. M. (2010). Generating event sequence-based test cases using gui runtime state feedback. IEEE Transactions on Software Engineering, 36, 81–95.CrossRefGoogle Scholar
  64. Zaparanuks, D., & Hauswirth, M. (2010). Characterizing the design and performance of interactive java applications. In ISPASS (pp. 23–32). IEEE Computer Society.Google Scholar

Copyright information

© Springer Science+Business Media, LLC 2011

Authors and Affiliations

  • Andrea Adamoli
    • 1
  • Dmitrijs Zaparanuks
    • 1
  • Milan Jovic
    • 1
  • Matthias Hauswirth
    • 1
  1. 1.LuganoSwitzerland

Personalised recommendations