Extracting Widget Descriptions from GUIs
Graphical User Interfaces (GUIs) are typically designed to simplify data entering, data processing and visualization of results. However, GUIs can also be exploited for other purposes. For instance, automatic tools can analyze GUIs to retrieve information about the data that can be processed by an application. This information can serve many purposes such as ease application integration, augment test case generation, and support reverse engineering techniques.
In the last years, the scientific community provided an increasing attention to the automatic extraction of information from interfaces. For instance, in the domain of Web applications, learning techniques have been used to extract information from Web forms. The knowledge about the data that can be processed by an application is not only relevant for the Web, but it is also extremely useful to support the same techniques when applied to desktop applications.
In this paper we present a technique for the automatic extraction of descriptive information about the data that can be handled by widgets in GUI-based desktop applications. The technique is grounded on mature standards and best practices about the design of GUIs, and exploits the presence of textual descriptions in the GUIs to automatically obtain descriptive data for data widgets. The early empirical results with three desktop applications show that the presented algorithm can extract data with high precision and recall, and can be used to improve generation of GUI test cases.
Keywordsprogram analysis graphical user interface testing GUI applications
- 1.Buddi, http://buddi.digitalcave.ca/
- 3.Java look and feel design guidelines, http://java.sun.com/products/jlf/ed2/book/
- 6.ISO 9241-12:1998 Ergonomic requirements for office work with visual display terminals (VDTs) - Part 12: Presentation of information (1998)Google Scholar
- 9.Fu, C., Grechanik, M., Xie, Q.: Inferring types of references to gui objects in test scripts. In: Proceedings of the International Conference on Software Testing Verification and Validation (2009)Google Scholar
- 10.He, B., Chang, K.C.-C.: Statistical schema matching across web query interfaces. In: Proceedings of the International Conference on Management of Data (2003)Google Scholar
- 11.Lo, R., Webby, R., Jeffery, R.: Sizing and estimating the coding and unit testing effort for gui systems. In: Proceedings of the 3rd International Symposium on Software Metrics: From Measurement to Empirical Results (1996)Google Scholar
- 12.Mariani, L., Pezzè, M., Riganelli, O., Santoro, M.: Autoblacktest: a tool for automatic black-box testing. In: Proceeding of the International Conference on Software Engineering (2011)Google Scholar
- 13.Nguyen, H., Nguyen, T., Freire, J.: Learning to extract form labels. In: Proceedings of the VLDB Endowment, 1 (August 2008)Google Scholar
- 14.Preece, J., Rogers, Y., Sharp, H., Benyon, D., Holland, S., Carey, T.: Human-Computer Interaction. Addison Wesley (1994)Google Scholar
- 15.Sánchez Ramón, O., Sánchez Cuadrado, J., García Molina, J.: Model-driven reverse engineering of legacy graphical user interfaces. In: Proceedings of the International Conference on Automated Software Engineering (2010)Google Scholar
- 16.Shehady, R.K., Siewiorek, D.P.: A method to automate user interface testing using variable finite state machines. In: Proceedings of the International Symposium on Fault-Tolerant Computing (1997)Google Scholar
- 17.Tichy, W.F., Koerner, S.J.: Text to software: developing tools to close the gaps in software engineering. In: Proceedings of the FSE/SDP Workshop on Future of Software Engineering Research (2010)Google Scholar
- 18.Vieira, M., Leduc, J., Hasling, B., Subramanyan, R., Kazmeier, J.: Automation of gui testing using a model-driven approach. In: Proceedings of the 2006 International Workshop on Automation of Software Test (2006)Google Scholar
- 19.Wu, W., Yu, C., Doan, A., Meng, W.: An interactive clustering-based approach to integrating source query interfaces on the deep Web. In: Proceedings of the International Conference on Management of Data (2004)Google Scholar