Extracting Widget Descriptions from GUIs

  • Giovanni Becce
  • Leonardo Mariani
  • Oliviero Riganelli
  • Mauro Santoro
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7212)


Graphical User Interfaces (GUIs) are typically designed to simplify data entering, data processing and visualization of results. However, GUIs can also be exploited for other purposes. For instance, automatic tools can analyze GUIs to retrieve information about the data that can be processed by an application. This information can serve many purposes such as ease application integration, augment test case generation, and support reverse engineering techniques.

In the last years, the scientific community provided an increasing attention to the automatic extraction of information from interfaces. For instance, in the domain of Web applications, learning techniques have been used to extract information from Web forms. The knowledge about the data that can be processed by an application is not only relevant for the Web, but it is also extremely useful to support the same techniques when applied to desktop applications.

In this paper we present a technique for the automatic extraction of descriptive information about the data that can be handled by widgets in GUI-based desktop applications. The technique is grounded on mature standards and best practices about the design of GUIs, and exploits the presence of textual descriptions in the GUIs to automatically obtain descriptive data for data widgets. The early empirical results with three desktop applications show that the presented algorithm can extract data with high precision and recall, and can be used to improve generation of GUI test cases.


program analysis graphical user interface testing GUI applications 


  1. 1.
  2. 2.
  3. 3.
    Java look and feel design guidelines,
  4. 4.
  5. 5.
  6. 6.
    ISO 9241-12:1998 Ergonomic requirements for office work with visual display terminals (VDTs) - Part 12: Presentation of information (1998)Google Scholar
  7. 7.
    Bertolino, A., Gnesi, S.: Use case-based testing of product lines. SIGSOFT Softw. Eng. Notes 28, 355–358 (2003)CrossRefGoogle Scholar
  8. 8.
    Fröhlich, P., Link, J.: Automated Test Case Generation from Dynamic Models. In: Bertino, E. (ed.) ECOOP 2000. LNCS, vol. 1850, pp. 472–491. Springer, Heidelberg (2000)CrossRefGoogle Scholar
  9. 9.
    Fu, C., Grechanik, M., Xie, Q.: Inferring types of references to gui objects in test scripts. In: Proceedings of the International Conference on Software Testing Verification and Validation (2009)Google Scholar
  10. 10.
    He, B., Chang, K.C.-C.: Statistical schema matching across web query interfaces. In: Proceedings of the International Conference on Management of Data (2003)Google Scholar
  11. 11.
    Lo, R., Webby, R., Jeffery, R.: Sizing and estimating the coding and unit testing effort for gui systems. In: Proceedings of the 3rd International Symposium on Software Metrics: From Measurement to Empirical Results (1996)Google Scholar
  12. 12.
    Mariani, L., Pezzè, M., Riganelli, O., Santoro, M.: Autoblacktest: a tool for automatic black-box testing. In: Proceeding of the International Conference on Software Engineering (2011)Google Scholar
  13. 13.
    Nguyen, H., Nguyen, T., Freire, J.: Learning to extract form labels. In: Proceedings of the VLDB Endowment, 1 (August 2008)Google Scholar
  14. 14.
    Preece, J., Rogers, Y., Sharp, H., Benyon, D., Holland, S., Carey, T.: Human-Computer Interaction. Addison Wesley (1994)Google Scholar
  15. 15.
    Sánchez Ramón, O., Sánchez Cuadrado, J., García Molina, J.: Model-driven reverse engineering of legacy graphical user interfaces. In: Proceedings of the International Conference on Automated Software Engineering (2010)Google Scholar
  16. 16.
    Shehady, R.K., Siewiorek, D.P.: A method to automate user interface testing using variable finite state machines. In: Proceedings of the International Symposium on Fault-Tolerant Computing (1997)Google Scholar
  17. 17.
    Tichy, W.F., Koerner, S.J.: Text to software: developing tools to close the gaps in software engineering. In: Proceedings of the FSE/SDP Workshop on Future of Software Engineering Research (2010)Google Scholar
  18. 18.
    Vieira, M., Leduc, J., Hasling, B., Subramanyan, R., Kazmeier, J.: Automation of gui testing using a model-driven approach. In: Proceedings of the 2006 International Workshop on Automation of Software Test (2006)Google Scholar
  19. 19.
    Wu, W., Yu, C., Doan, A., Meng, W.: An interactive clustering-based approach to integrating source query interfaces on the deep Web. In: Proceedings of the International Conference on Management of Data (2004)Google Scholar
  20. 20.
    Yuan, X., Memon, A.M.: Generating event sequence-based test cases using GUI run-time state feedback. IEEE Transactions on Software Engineering 36(1), 81–95 (2010)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Giovanni Becce
    • 1
  • Leonardo Mariani
    • 1
  • Oliviero Riganelli
    • 1
  • Mauro Santoro
    • 1
  1. 1.Department of Informatics, Systems and CommunicationsUniversity of Milano BicoccaMilanoItaly

Personalised recommendations