Skip to main content

GUI Element Detection from Mobile UI Images Using YOLOv5

  • Conference paper
  • First Online:
Mobile Web and Intelligent Information Systems (MobiWIS 2022)

Abstract

In mobile application development, building a consistent user interface (UI) might be a costly and time-consuming process. This is especially the case if an organization has a separate team for each mobile platform such as iOS and Android. In this regard, the companies that choose the native mobile app development path end up going through do-overs as the UI work done on one platform needs to be repeated for other platforms too. One of the tedious parts of UI design tasks is creating a graphical user interface (GUI). There are numerous tools and prototypes in the literature that aim to create feasible GUI automation solutions to speed up this process and reduce the labor workload. However, as the technologies evolve and improve new versions of existing algorithms are created and offered. Accordingly, this study aims to employ the latest version of YOLO, which is YOLOv5, to create a custom object detection model that recognizes GUI elements in a given UI image. In order to benchmark the newly trained YOLOv5 GUI element detection model, existing work from the literature and their data set is considered and used for comparison purposes. Therefore, this study makes use of 450 UI samples of the VINS dataset for testing, a similar amount for validation and the rest for model training. Then the findings of this work are compared with another study that has used the SSD algorithm and VINS dataset to train, validate and test its model, which showed that proposed algorithm outperformed SSD’s mean average precision (mAP) by 15.69%.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 59.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 79.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Puiu, T.: Your smartphone is millions of times more powerful than the Apollo 11 guidance computers. ZME Science (2021). https://www.zmescience.com/science/news-science/smartphone-power-compared-to-apollo-432/. Accessed 15 Jan 2022

  2. Ceci, L.: Number of apps available in leading app stores as of 1st quarter 2021. Statista (2022). https://www.statista.com/statistics/276623/number-of-apps-available-in-leading-app-stores/. Accessed 30 Jan 2022

  3. Statista Research Department: Revenue of mobile apps worldwide 2017–2025, by segment. Statista (2021). https://www.statista.com/statistics/269025/worldwide-mobile-app-revenue-forecast/. Accessed 02 Feb 2022

  4. Chen, J., et al.: Object detection for graphical user interface: old fashioned or deep learning or a combination? In: Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pp. 1202–1214. ACM (2020)

    Google Scholar 

  5. Xie, M., et al.: UIED: a hybrid tool for GUI element detection. In: Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pp. 1655–1659. ACM (2020)

    Google Scholar 

  6. Bunian, S., et al.: VINS: visual search for mobile user interface design. In: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, pp. 1–14. ACM (2021)

    Google Scholar 

  7. Nguyen, T.A., Csallner, C.: Reverse engineering mobile application user interfaces with REMAUI (T). In: Proceedings of the 30th IEEE/ACM International Conference on Automated Software Engineering, pp. 248–259. ACM (2015)

    Google Scholar 

  8. Chen, S., et al.: Automated cross-platform GUI code generation for mobile apps. In: IEEE 1st International Workshop on Artificial Intelligence for Mobile, pp. 13–16. IEEE (2019)

    Google Scholar 

  9. OpenCV About. https://opencv.org/about/. Accessed 13 Feb 2022

  10. Apple Developer Vision. https://developer.apple.com/documentation/vision. Accessed 16 Feb 2022

  11. Google Colab. https://colab.research.google.com. Accessed 1 Jan 2022

  12. Zhiqiang, W., Jun, L.: A review of object detection based on convolutional neural network. In: Proceedings of the 36th Chinese Control Conference, pp. 85–112. IEEE (2017)

    Google Scholar 

  13. Girshick, R.: Fast R-CNN. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1440–1448. IEEE (2015)

    Google Scholar 

  14. Redmon, J., et al.: You only look once: unified, real-time object detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 779–788. IEEE (2016)

    Google Scholar 

  15. Deka, B., et al.: Rico: a mobile app dataset for building data-driven design applications. In: Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology, pp. 845–854. ACM (2017)

    Google Scholar 

  16. What is Uplabs? https://www.uplabs.com/faq. Accessed 18 Feb 2022

  17. Lin, T..-Y.., et al.: Microsoft coco: common objects in context. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8693, pp. 740–755. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10602-1_48

    Chapter  Google Scholar 

  18. Unel, F.O., et al.: The power of tiling for small object detection. In: Proceeding of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 582–591. IEEE (2019)

    Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding authors

Correspondence to Mehmet Dogan Altinbas or Tacha Serif .

Editor information

Editors and Affiliations

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Altinbas, M.D., Serif, T. (2022). GUI Element Detection from Mobile UI Images Using YOLOv5. In: Awan, I., Younas, M., Poniszewska-Marańda, A. (eds) Mobile Web and Intelligent Information Systems. MobiWIS 2022. Lecture Notes in Computer Science, vol 13475. Springer, Cham. https://doi.org/10.1007/978-3-031-14391-5_3

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-14391-5_3

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-14390-8

  • Online ISBN: 978-3-031-14391-5

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics