Skip to main content

Automated Retrieval of Graphical User Interface Prototypes from Natural Language Requirements

  • 744 Accesses

Part of the Lecture Notes in Computer Science book series (LNISA,volume 12801)


High-fidelity Graphical User Interface (GUI) prototyping represents a suitable approach for allowing to clarify and refine requirements elicitated from customers. In particular, GUI prototypes can facilitate to mitigate and reduce misunderstandings between customers and developers, which may occur due to the ambiguity and vagueness of informal Natural Language (NL). However, employing high-fidelity GUI prototypes is more time-consuming and expensive compared to other simpler GUI prototyping methods. In this work, we propose a system that automatically processes Natural Language Requirements (NLR) and retrieves fitting GUI prototypes from a semi-automatically created large-scale GUI repository for mobile applications. We extract several text segments from the GUI hierarchy data to obtain textual representations for the GUIs. To achieve ad-hoc GUI retrieval from NLR, we adopt multiple Information Retrieval (IR) approaches and Automatic Query Expansion (AQE) techniques. We provide an extensive and systematic evaluation of the applied IR and AQE approaches for their effectiveness in terms of GUI retrieval relevance on a manually annotated dataset of NLR in the form of search queries and User Stories (US). We found that our GUI retrieval performs well in the conducted experiments and discuss the results.


  • Automatic Prototyping of Graphical User Interfaces (GUIs)
  • GUI retrieval
  • GUI prototypes from natural language requirements

This is a preview of subscription content, access via your institution.

Buying options

USD   29.95
Price excludes VAT (USA)
  • DOI: 10.1007/978-3-030-80599-9_33
  • Chapter length: 9 pages
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
USD   79.99
Price excludes VAT (USA)
  • ISBN: 978-3-030-80599-9
  • Instant PDF download
  • Readable on all devices
  • Own it forever
  • Exclusive offer for individuals only
  • Tax calculation will be finalised during checkout
Softcover Book
USD   99.99
Price excludes VAT (USA)
Fig. 1.


  1. Azad, H.K., Deepak, A.: Query expansion techniques for information retrieval: a survey. Inf. Process. Manag. 56(5), 1698–1735 (2019)

    Google Scholar 

  2. Behrang, F., Reiss, S.P., Orso, A.: Guifetch: supporting app design and development through gui search. In: Proceedings of the 5th International Conference on Mobile Software Engineering and Systems, pp. 236–246 (2018)

    Google Scholar 

  3. Bernal-Cárdenas, C., et al.: Guigle: a gui search engine for android apps. In: 2019 IEEE/ACM 41st International Conference on Software Engineering: Companion Proceedings (ICSE-Companion), pp. 71–74. IEEE (2019)

    Google Scholar 

  4. Berry, D.M., Kamsties, E.:Ambiguity in requirements specification. In: do Prado Leite, J.C.S., Doorn, J.H. (eds) Perspectives on Software Requirements. The Springer International Series in Engineering and Computer Science, vol. 753. Springer, Boston, MA (2004).

  5. Carpineto, C., De Mori, R., Romano, G., Bigi, B.: Brigitte: an information-theoretic approach to automatic query expansion. ACM Trans. Inf. Syst. (TOIS) 19(1), 1–27 (2001)

    Google Scholar 

  6. Dalpiaz, F.: Requirements data sets (user stories). Mendeley Data, vol. 1 (2018)

    Google Scholar 

  7. Deka, B., et al.: Rico: a mobile app dataset for building data-driven design applications. In: Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology, pp. 845–854 (2017)

    Google Scholar 

  8. Deka, B., Huang, Z., Kumar, R.: Erica: interaction mining mobile apps. In: Proceedings of the 29th Annual Symposium on User Interface Software and Technology, pp. 767–776 (2016)

    Google Scholar 

  9. Galke, L., Saleh, A., Scherp, A.: Word embeddings for practical information retrieval. INFORMATIK 2017 (2017)

    Google Scholar 

  10. Huang, F., Canny, J.F., Nichols, J.: Swire: sketch-based user interface retrieval. In: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, pp. 1–10 (2019)

    Google Scholar 

  11. Kolthoff, K.: Automatic generation of graphical user interface prototypes from unrestricted natural language requirements. In: 2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE), pp. 1234–1237. IEEE (2019)

    Google Scholar 

  12. Kolthoff, K., Bartelt, C., Ponzetto, S.P.: GUI2WiRe: rapid wireframing with a mined and large-scale GUI repository using natural language requirements. In: 35th IEEE/ACM International Conference on Automated Software Engineering (ASE 2020). ACM (2020)

    Google Scholar 

  13. Manning, C.D., Raghavan, P., Schütze, H.: Introduction to information retrieval. Cambridge University Press, Cambridge (2008)

    Google Scholar 

  14. Moran, K., Bernal-Cárdenas, C., Curcio, M., Bonett, R., Poshyvanyk, D.: Machine learning-based prototyping of graphical user interfaces for mobile apps. arXiv preprint arXiv:1802.02312 (2018)

  15. Robertson, S.E., et al.: Okapi at TREC-3. Nist Special Publication Sp, 109:109 (1995)

    Google Scholar 

  16. Rudd, J., Stern, K., Isensee, S.: Low vs. high-fidelity prototyping debate. Interactions 3(1), 76–85 (1996)

    Google Scholar 

  17. Nakatani, S.: Language detection library for java. Retrieved 7 July 2016 (2010)

    Google Scholar 

  18. Zowghi D., Coulin C.: Requirements elicitation: a survey of techniques, approaches, and tools. In: Aurum, A., Wohlin, C. (eds) Engineering and Managing Software Requirements. Springer, Heidelberg. (2005).

Download references


This work is supported by the German Federal Ministry of Education and Research (BMBF).

Author information

Authors and Affiliations


Corresponding author

Correspondence to Kristian Kolthoff .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and Permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Verify currency and authenticity via CrossMark

Cite this paper

Kolthoff, K., Bartelt, C., Ponzetto, S.P. (2021). Automated Retrieval of Graphical User Interface Prototypes from Natural Language Requirements. In: Métais, E., Meziane, F., Horacek, H., Kapetanios, E. (eds) Natural Language Processing and Information Systems. NLDB 2021. Lecture Notes in Computer Science(), vol 12801. Springer, Cham.

Download citation

  • DOI:

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-80598-2

  • Online ISBN: 978-3-030-80599-9

  • eBook Packages: Computer ScienceComputer Science (R0)