Software Quality Journal

, Volume 14, Issue 4, pp 359–377 | Cite as

Employing user profiles to test a new version of a GUI component in its context of use

  • Atif M. MemonEmail author


Advancement in reusable component technology has had a significant impact on the development of complex graphical user interfaces (GUIs), which are front-ends to most of today’s software. Software developers can, with very little effort, integrate components into their software’s GUI. Problems, however, arise when new versions of GUI components replace their predecessors in an implementation. Often, the inclusion of a new version of a component breaks some part of the software, i.e., tasks that end-users were able to perform before modifications were made can no longer be performed. Software developers (who also happen to be component users) are unable to perform adequate regression testing in this context because of several factors, including lack of source code, environmental side-effects on GUI rendering, event-driven nature of GUIs, and large number of possible permutations of events. This paper presents a new technique that end-users’ quality assurance (QA) teams can employ to test the new version of a component in its application context by using the existing version as a baseline. The technique combines lightweight event-level dynamic profiling to collect user profiles in a transparent manner, GUI reverse engineering to extract the structure of the component’s GUI, test case execution to replay the collected profiles on the new version, and GUI oracles that collect properties from the existing version. Empirical studies demonstrate the practicality, usefulness, and limitations of the technique.


GUI testing User profiles Software components Test oracles User session data Automated testing 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. Abbot Java GUI Test Framework, 2003.
  2. Bertolino, A., Polini, A. 2003. A framework for component deployment testing. In Proceedings of the 25th International Conference on Software Engineering, pp. 221–231, IEEE Computer Society.Google Scholar
  3. Beydeda, S., Gruhn, V. 2001. An integrated testing technique for component-based software.Google Scholar
  4. Capture-Replay Tool, 2003.
  5. Dawson, S., Jahanian, F., Mitton, T., Tung, T.-L. 1996. Testing of fault-tolerant and real-time distributed systems via protocol fault injektion. In Symposium on Fault-Tolerant Computing, pp. 404–414.Google Scholar
  6. Dillon, L.K., Ramakrishna, Y.S. 1996. Generating oracles from your favorite temporal logic specifications. In Proceedings of the Fourth ACM SIGSOFT Symposium on the Foundations of Software Engineering (New York), acknowledgementsol. 21 of ACM Software Engineering Notes, pp. 106–117, ACM Press.Google Scholar
  7. Dillon, L.K., Yu, Q. 1994. Oracles for checking temporal properties of concurrent systems. In Proceedings of the ACM SIGSOFT ’94 Symposium on the Foundations of Software Engineering, pp. 140–153.Google Scholar
  8. Finsterwalder, M. 2001. Automating acceptance tests for gui applications in an extreme programming environment. In Proceedings of the 2nd International Conference on eXtreme Programming and Flexible Processes in Software Engineering, pp. 114–117.Google Scholar
  9. Hamlet, D., Mason, D., Woit, D. 2001. Theory of software reliability based on components. In Proceedings of the 23rd International Conference on Software Engineering, pp. 361–370, IEEE Computer Society.Google Scholar
  10. Haraty, R.A., Mansour, N., Daou, B. 2001. Regression testing of database applications. In Proceedings of the 2001 ACM Symposium on Applied Computing, pp. 285–289, ACM Press.Google Scholar
  11. Harrold, M.J. 2000. Testing: a roadmap. In Proceedings of the Conference on The Future Software Engineering, pp. 61–72, ACM Press.Google Scholar
  12. Hicinbothom, J.H., Zachary, W.W. 1993. A tool for automatically generating transcripts of human-computer interaction. In Proceedings of the Human Factors and Ergonomics Society 37th Annual Meeting, vol. 2 of Special Sessions: Demonstrations, p. 1042.Google Scholar
  13. Huang, Y.-W., Huang, S.-K., Lin, T.-P., Tsai, C.-H. 2003. Web application security assessment by fault injection and behavior monitoring. In Proceedings of the Twelfth International Conference on World Wide Web, pp. 148–159, ACM Press.Google Scholar
  14. Jeon, T., Seung, H.W., Lee, S. 2002. Embedding built-in tests in hot spots of an object-oriented framework. SIGPLAN Not. 37(8):25–34.CrossRefGoogle Scholar
  15. JUnit, Testing Resources for Extreme Programming.
  16. Korel, B. 1999. Black-box understanding of COTS components. In Proceedings: Seventh International Workshop on Program Comprehension, pp. 92–99, IEEE Computer Society Press.Google Scholar
  17. Kranitis, N., Paschalis, A., Gizopoulos, D., Zorian, Y. 2002. Effective software self-test methodology for processor cores. In Proceedings of the Conference on Design, Automation and Test in Europe, p. 592, IEEE Computer Society.Google Scholar
  18. Krstic, A., Lai, W.C., Cheng, K.T., Chen, L., Dey, S. 2002. Embedded software-based self-testing for soc design. In Proceedings of the 39th Conference on Design Automation, pp. 355–360, ACM Press.Google Scholar
  19. Kung, D.C., Gao, J., Hsia, P., Toyoshima, Y., Chen, C. 1996. On regression testing of object-oriented programs. The Journal of Systems and Software 32(1):21–31.CrossRefGoogle Scholar
  20. Lee White, H.A., Alzeidi, N. 2001. User-based testing of gui sequences and their interactions. In Proceedings of the 12th International Symposium Software Reliability Engineering, pp. 54–63.Google Scholar
  21. Lu, G. 1991. Automatic testing for the applications with direct manipulation interfaces. In Proceedings of the Fourth International Conference on Human-Computer Interaction, vol. 1 of Congress II: Design and Implementation of Interactive Systems: SOFTWARE ENGINEERING—METHODS, TECHNIQUES AND TOOLS; Applications of Techniques and Tools, pp. 602–606.Google Scholar
  22. Marick, B. 2002. Bypassing the GUI. Software Testing and Quality Engineering Magazine: 41–47.Google Scholar
  23. Memon, A., Nagarajan, A., Xie, Q. 2005. Automating regression testing for evolving GUI software. Journal of Software Maintenance and Evolution: Research and Practice 17(1):27–64.CrossRefGoogle Scholar
  24. Memon, A.M. 2001. A Comprehensive Framework for Testing Graphical User Interfaces. Ph.D. thesis, Department of Computer Science, University of Pittsburgh.Google Scholar
  25. Memon, A.M. 2002. GUI testing: Pitfalls and process. IEEE Computer 35(8)90–91.Google Scholar
  26. Memon, A.M., Banerjee, I., Nagarajan, A. 2003. GUI ripping: Reverse engineering of graphical user interfaces for testing. In Proceedings of the 10th Working Conference on Reverse Engineering.Google Scholar
  27. Memon, A.M., Pollack, M.E., Soffa, M.L. 1999. Using a goal-driven approach to generate test cases for GUIs. In Proceedings of the 21st International Conference on Software Engineering, pp. 257–266, ACM Press.Google Scholar
  28. Memon, A.M., Pollack, M.E., Soffa, M.L. 2000a. Automated test oracles for GUIs. In Proceedings of the ACM SIGSOFT 8th International Symposium on the Foundations of Software Engineering (FSE-8) (NY), pp. 30–39.Google Scholar
  29. Memon, A.M., Pollack, M.E., Soffa, M.L. 2000b. Plan generation for GUI testing. In Proceedings of the Fifth International Conference on Artificial Intelligence Planning and Scheduling, pp. 226–235, AAAI Press.Google Scholar
  30. Memon, A.M., Pollack, M.E., Soffa, M.L. 2001a. Hierarchical GUI test case generation using automated planning. IEEE Transactions on Software Engineering 27(2):144–155.CrossRefGoogle Scholar
  31. Memon, A.M., Soffa, M.L. 2003. Regression testing of guis. In Proceedings of the 9th European Software Engineering Conference held jointly with 10th ACM SIGSOFT International Symposium on Foundations of Software Engineering, pp. 118–127, ACM Press.Google Scholar
  32. Memon, A.M., Soffa, M.L., Pollack, M.E. 2001b. Coverage criteria for GUI testing. In Proceedings of the 8th European Software Engineering Conference (ESEC) and 9th ACM SIGSOFT International Symposium on the Foundations of Software Engineering (FSE-9), pp. 256–267.Google Scholar
  33. Memon, A.M., Xie, Q. 2004. Using transient/persistent errors to develop automated test oracles for event-driven software. In Proceedings of The International Conference on Automated Software Engineering 2004 (ASE’04), pp. 186–195.Google Scholar
  34. Memon A.M., Xie, Q. 2005. Studying the fault-detection effectiveness of GUI test cases for rapidly evolving software. IEEE Transactions on Software Engineering, 31(10):884–896.CrossRefGoogle Scholar
  35. Mercury Interactive WinRunner, 2003.
  36. Morris, J., Lam, P., Lee, G., Parker, K., Bundell, G.A. 2002. Determining component reliability using a testing index. In Proceedings of the Twenty-fifth Australasian Conference on Computer Science, pp. 167–176, Australian Computer Society, Inc.Google Scholar
  37. Myers, B.A. 1993. Why are human-computer interfaces difficult to design and implement? Technical Report CS-93-183, Carnegie Mellon University, School of Computer Science.Google Scholar
  38. Nagarajan, A., Memon, A.M. 2003. Refactoring using event-based profiling of GUI applications. In Proceedings of the First International Workshop on REFactoring: Achievements, Challenges, Effects 2003.Google Scholar
  39. Onoma, A.K., Tsai, W.-T., Poonawala, M., Suganuma, H. 1998. Regression testing in an industrial environment. Commun. ACM 41(5)81–86.CrossRefGoogle Scholar
  40. Orso, A., Harrold, M.J., Rosenblum, D.S. 2001a. Component metadata for software engineering tasks. In Revised Papers from the Second International Workshop on Engineering Distributed Objects, pp. 129–144, Springer-Verlag.Google Scholar
  41. Orso, A., Harrold, M.J., Rosenblum, D.S., Rothermel, G., Soffa, M.L., Do, H. 2001b. Using component metadata to support the regression testing of component -based software. In Proceedings of the IEEE International Conference on Software Maintenance.Google Scholar
  42. Richardson, D.J., Leif-Aha, S., OMalley, T.O. 1992. Specification-based Test Oracles for Reactive Systems. In Proceedings of the 14th International Conference on Software Engineering, pp. 105–118.Google Scholar
  43. Sparling, M. 2000. Lessons learned through six years of component-based development. Communication ACM 43(10):47–53.CrossRefGoogle Scholar
  44. Ulrich, A., Chrobok-Diening, G. 1999. International workshop on testing distributed component-based systems. SIGSOFT Software Engineering Notes 24(4):43–46.Google Scholar
  45. Vitharana, P. 2003. Risks and challenges of component-based software development. Communication ACM 46(8):67–72.CrossRefGoogle Scholar
  46. Voas, J., Charron, F., McGraw, G., Miller, K., Friedman, M. 1997. Predicting how badly “good” software can behave. IEEE Software 14(4):73–83.CrossRefGoogle Scholar
  47. Voas, J., Payne, J. 1998. Cots software failures: Can anything be done? In Proceedings of the First IEEE Workshop on Application Specific Software Engineering and Technology (ASSET’98), pp. 140–145, IEEE Press.Google Scholar
  48. White, L. 1996. Regression testing of GUI event interactions. In Proceedings of the International Conference on Software Maintenance (Washington, Nov.4–8), pp. 350–358.Google Scholar
  49. Wittenberg, C.H. 2000. Progress in testing component-based software (abstract only). In Proceedings of the International Symposium on Software Testing and Analysis, p. 178, ACM Press.Google Scholar
  50. Xie, Q., Memon, A.M. 2005. Rapid crash testing for continuously evolving GUI-based software applications. In Proceedings of The International Conference on Software Maintenance 2005 (ICSM’05) (Budapest, Hungary).Google Scholar
  51. Xie, Q., Memon, A.M. Designing and comparing automated test oracles for GUI-based software applications. ACM Transactions on Software Testing and Methodology (to appear).Google Scholar
  52. Zhao, W., Papachristou, C. 1998. Testing dsp cores based on self-test programs. In Proceedings of the Conference on Design, Automation and Test in Europe, pp. 166–172, IEEE Computer Society.Google Scholar

Copyright information

© Springer Science + Business Media, LLC 2006

Authors and Affiliations

  1. 1.Department of Computer ScienceUniversity of MarylandCollege ParkUSA

Personalised recommendations