Skip to main content
SpringerLink
Log in

An Early Rico Retrospective: Three Years of Uses for a Mobile App Dataset

  • Chapter
  • First Online:
Artificial Intelligence for Human Computer Interaction: A Modern Approach

Part of the book series: Human–Computer Interaction Series ((HCIS))

Abstract

The Rico dataset, containing design data from more than 9.7 k Android apps spanning \(27\) categories, was released in 2017. It exposes visual, textual, structural, and interactive design properties of more than 72 k unique UI screens. Over the years since its release, the original paper has been cited nearly 100 times according to Google Scholar and the dataset has been used as the basis for numerous research projects. In this chapter, we describe the creation of Rico using a system that combined crowdsourcing and automation to scalably mine design and interaction data from Android apps at runtime. We then describe two projects that we conducted using the dataset: the training of an autoencoder to identify similarity between UI designs, and an exploration of the use of Google’s Material Design within the dataset using machine learned models. We conclude with an overview of other work that has used Rico to understand our mobile UI world and build data-driven models that assist users, designers, and developers.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 149.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 199.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 199.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    Rico—a Spanish word meaning “rich”.

  2. 2.

    Material Design https://material.io/guidelines/.

References

  1. Android Activities (2016). https://developer.android.com/guide/components/activities.html

  2. Database of Android Apps on Kaggle (2016). https://www.kaggle.com/orgesleka/android-apps

  3. UI Overview (2016). https://developer.android.com/guide/topics/ui/overview.html

  4. A L (2014) Why and how to avoid hamburger menus

    Google Scholar 

  5. Abadi M, Agarwal A, Barham P, Brevdo E, Chen Z, Citro C, Corrado GS, Davis A, Dean J, Devin M, Ghemawat S, Goodfellow I, Harp A, Irving G, Isard M, Jia Y, Jozefowicz R, Kaiser L, Kudlur M, Levenberg J, Mané D, Monga R, Moore S, Murray D, Olah C, Schuster M, Shlens J, Steiner B, Sutskever I, Talwar K, Tucker P, Vanhoucke V, Vasudevan V, Viégas F, Vinyals O, Warden P, Wattenberg M, Wicke M, Yu Y, Zheng X (2015) TensorFlow: large-scale machine learning on heterogeneous systems. Software available from tensorflow.org

    Google Scholar 

  6. Alexander C (1977) A pattern language: towns, buildings, construction. Oxford University Press, Oxford

    Google Scholar 

  7. Alharbi K, Yeh T (2015) Collect, decompile, extract, stats, and diff: mining design pattern changes in Android apps. In: Proceedings of MobileHCI

    Google Scholar 

  8. Amini S (2014) Analyzing mobile app privacy using computation and crowdsourcing. PhD thesis, Carnegie Mellon University

    Google Scholar 

  9. Avgeriou P, Papasalouros A, Retalis S, Skordalakis M (2003) Towards a pattern language for learning management systems. Educ Technol & Soc 6(2):11–24

    MATH  Google Scholar 

  10. Azim T, Neamtiu I (2013) Targeted and depth-first exploration for systematic testing of android apps. In: ACM SIGPLAN Notices

    Google Scholar 

  11. Babich N (2017) Floating action button in ux design

    Google Scholar 

  12. Bengio Y (2009) Learning deep architectures for ai. Found Trends Mach Learn 2:1

    Article  Google Scholar 

  13. Bhoraskar R, Han S, Jeon J, Azim T, Chen S, Jung J, Nath S, Wang R, Wetherall D (2014) Brahmastra: driving apps to test the security of third-party components. In: Proceeding of the SEC

    Google Scholar 

  14. Borchers JO (2000) A pattern approach to interaction design. In: Proceedings of the 3rd conference on designing interactive systems: processes, practices, methods, and techniques, DIS ’00, ACM, New York, NY, USA, pp 369–378

    Google Scholar 

  15. Chen J, Chen C, Xing Z, Xia X, Zhu L, Grundy J, Wang J (2020) Wireframe-based ui design search through image autoencoder. ACM Trans Softw Eng Methodol 29:3

    Google Scholar 

  16. Chen J, Xie M, Xing Z, Chen C, Xu X, Zhu L, Li G (2020) Object detection for graphical user interface: old fashioned or deep learning or a combination? New York, NY, USA, Association for Computing Machinery, pp 1202–1214

    Google Scholar 

  17. Chen Y, Pandey M, Song JY, Lasecki WS, Oney S (2020) Improving crowd-supported gui testing with structural guidance. In: Proceedings of the 2020 CHI conference on human factors in computing systems, CHI ’20, Association for Computing Machinery, New York, NY, USA, pp 1–13

    Google Scholar 

  18. Chung ES, Hong JI, Lin J, Prabaker MK, Landay JA, Liu AL (2004) Development and evaluation of emerging design patterns for ubiquitous computing. In: Proceedings of the 5th conference on designing interactive systems: processes, practices, methods, and techniques, ACM, pp 233–242

    Google Scholar 

  19. Constine J (2014) Kill the hamburger button

    Google Scholar 

  20. Dearden A, Finlay J (2006) Pattern languages in hci: a critical review. Human-Comput Inter 21(1):49–102

    Article  Google Scholar 

  21. Deka B, Huang Z, Franzen C, Hibschman J, Afergan D, Li Y, Nichols J, Kumar R (2017) Rico: a mobile app dataset for building data-driven design applications. In: 30th annual symposium on user interface software and technology, UIST ’17, ACM, New York, NY, USA

    Google Scholar 

  22. Deka B, Huang Z, Kumar R (2016) ERICA: Interaction mining mobile apps. In: Proceedings of the UIST

    Google Scholar 

  23. Doosti B, Dong T, Deka B, Nichols J (2018) A computational method for evaluating UI patterns. ArXiv e-prints

    Google Scholar 

  24. Frank M, Dong B, Felt AP, Song D (2012) Mining permission request patterns from android and facebook applications. In: Proceeding of the ICDM

    Google Scholar 

  25. Fu B, Lin J, Li L, Faloutsos C, Hong J, Sadeh N (2013) Why people hate your app: making sense of user feedback in a mobile app store. In: Proceedings of the KDD

    Google Scholar 

  26. Gupta K, Achille A, Lazarow J, Davis L, Mahadevan V, Shrivastava A (2020) Layout generation and completion with self-attention

    Google Scholar 

  27. Huang F, Canny JF, Nichols J (2019) Swire: sketch-based user interface retrieval. In: Proceedings of the 2019 CHI conference on human factors in computing systems, CHI ’19, Association for Computing Machinery, New York, NY, USA, pp 1–10

    Google Scholar 

  28. Jager T (2017) Is the floating action button bad ux design?

    Google Scholar 

  29. Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In: Pereira F, Burges CJC, Bottou L, Weinberger KQ (eds) Advances in neural information processing systems 25. Curran Associates Inc, Stateline, NV, USA, pp 1097–1105

    Google Scholar 

  30. Landay JA, Borriello G (2003) Design patterns for ubiquitous computing. Computer 36(8):93–95

    Google Scholar 

  31. Lee C, Kim S, Han D, Yang H, Park Y-W, Kwon BC, Ko S (2020) Guicomp: a gui design assistant with real-time, multi-faceted feedback. In: Proceedings of the 2020 CHI conference on human factors in computing systems, CHI ’20, Association for Computing Machinery, New York, NY, USA, pp 1–13

    Google Scholar 

  32. Lee H-Y, Yang W, Jiang L, Le M, Essa I, Gong H, Yang M-H (2020) Neural design network: graphic layout generation with constraints. In: Proceedings of European conference on computer vision (ECCV)

    Google Scholar 

  33. Lee K, Flinn J, Giuli T, Noble B, Peplin C (2013) Amc: verifying user interface properties for vehicular applications. In: Proceeding of the Mobisys

    Google Scholar 

  34. Li Y, He J, Zhou X, Zhang Y, Baldridge J (2020) Mapping natural language instructions to mobile UI action sequences. In: Proceedings of the 58th annual meeting of the association for computational linguistics, Association for Computational Linguistics, pp 8198–8210

    Google Scholar 

  35. Li Y, Li G, He L, Zheng J, Li H, Guan Z (2020) Widget captioning: generating natural language description for mobile user interface elements. In: Proceedings of the 2020 conference on empirical methods in natural language processing (EMNLP), Association for Computational Linguistics, pp 5495–5510

    Google Scholar 

  36. Li Y, Yang Z, Guo Y, Chen X (2019) Humanoid: a deep learning-based approach to automated black-box android app testing. In: 2019 34th IEEE/ACM international conference on automated software engineering (ASE), pp 1070–1073

    Google Scholar 

  37. Liu TF, Craft M, Situ J, Yumer E, Mech R, Kumar R (2018) Learning design semantics for mobile apps. In: Proceedings of the 31st annual ACM symposium on user interface software and technology, UIST ’18, Association for Computing Machinery, New York, NY, USA, pp 569–579

    Google Scholar 

  38. Luis AL, Asutosh Hota AO (2020) Enrico: a high-quality dataset for topic modeling of mobile ui designs. In: Proceedings of MobileHCI extended abstracts

    Google Scholar 

  39. Micallef N, Adi E, Misra G (2018) Investigating login features in smartphone apps. In: Proceedings of the 2018 ACM international joint conference and 2018 international symposium on pervasive and ubiquitous computing and wearable computers, UbiComp ’18, Association for Computing Machinery, New York, NY, USA, pp 842–851

    Google Scholar 

  40. Moran K, Bernal-Cárdenas C, Curcio M, Bonett R, Poshyvanyk D (2020) Machine learning-based prototyping of graphical user interfaces for mobile apps. IEEE Trans Softw Eng 46(2):196–221

    Google Scholar 

  41. Nair V, Hinton GE (2010) Rectified linear units improve restricted boltzmann machines. In: Proceeding of the ICML, pp 807–814

    Google Scholar 

  42. Neil T (2014) Mobile design pattern gallery: UI patterns for smartphone apps. O’Reilly Media, Inc., Sebastopol

    Google Scholar 

  43. Pernice K, Budiu R (2016) Hamburger menus and hidden navigation hurt ux metrics

    Google Scholar 

  44. Ross AS, Zhang X, Fogarty J, Wobbrock JO (2018) Examining image-based button labeling for accessibility in android apps through large-scale analysis. In: Proceedings of the 20th international ACM SIGACCESS conference on computers and accessibility, ASSETS ’18, Association for Computing Machinery, New York, NY, USA, pp 119–130

    Google Scholar 

  45. Sahami Shirazi A, Henze N, Schmidt A, Goldberg R, Schmidt B, Schmauder H (2013) Insights into layout patterns of mobile user interfaces by an automatic analysis of Android apps. In: Proceeding of the EICS

    Google Scholar 

  46. Saponas TS, Prabaker MK, Abowd GD, Landay JA () The impact of pre-patterns on the design of digital home applications. In: Proceedings of the 6th conference on designing interactive systems, ACM (2006), pp 189–198

    Google Scholar 

  47. Sereshkeh AR, Leung G, Perumal K, Phillips C, Zhang M, Fazly A, Mohomed I (2020) Vasta: a vision and language-assisted smartphone task automation system. In: Proceedings of the 25th international conference on intelligent user interfaces, IUI ’20, Association for Computing Machinery, New York, NY, USA, pp 22–32

    Google Scholar 

  48. Siang TY (2015) Material design: why the floating action button is bad ux design. https://medium.com/tech-in-asia/material-design-why-the-floating-action-button-is-bad-ux-design-acd5b32c5ef, https://medium.com/tech-in-asia/material-design-why-, https://medium.com/tech-in-asia/material-design-why-the-floating-action-button-is-bad-ux-design-acd5b32c5ef-the-floating-action-button-is-bad-ux-design-acd5b32c5ef

  49. Swearngin A, Li Y (2019) Modeling mobile interface tappability using crowdsourcing and deep learning. In: Proceedings of the 2019 CHI conference on human factors in computing systems, CHI ’19, Association for Computing Machinery, New York, NY, USA, pp 1–11

    Google Scholar 

  50. Szydlowski M, Egele M, Kruegel C, Vigna G (2012) Challenges for dynamic analysis of iOS applications. In: Open problems in network security. Springer, pp 65–77

    Google Scholar 

  51. Tidwell J (2010) Designing interfaces: patterns for effective interaction design. O’Reilly Media, Inc., Sebastopol

    Google Scholar 

  52. Viennot N, Garcia E, Nieh J (2014) A measurement study of google play. In: ACM SIGMETRICS performance evaluation review, vol. 42, ACM, pp 221–233

    Google Scholar 

  53. Wania CE, Atwood ME (2009) Pattern languages in the wild: exploring pattern languages in the laboratory and in the real world. In: Proceedings of the 4th international conference on design science research in information systems and technology, ACM, p 12

    Google Scholar 

  54. Wu Z, Jiang Y, Liu Y, Ma X (2020) Predicting and diagnosing user engagement with mobile ui animation via a data-driven approach. In: Proceedings of the 2020 CHI conference on human factors in computing systems, CHI ’20, Association for Computing Machinery, New York, NY, USA, pp 1–13

    Google Scholar 

  55. Zhang T, Liu Y, Gao J, Gao LP, Cheng J (2020) Deep learning-based mobile application isomorphic gui identification for automated robotic testing. IEEE Softw 37(4):67–74

    Google Scholar 

Download references

Acknowledgements

We thank the reviewers for their helpful comments and suggestions and the crowd workers who helped build the Rico dataset. This work was supported in part by a Google Faculty Research Award.

Author information

Authors and Affiliations

  1. McKinsey, Chicago, IL, USA

    Biplab Deka

  2. Google, Inc., Mountain View, CA, USA

    Bardia Doosti, Chad Franzen, Daniel Afergan, Yang Li & Tao Dong

  3. University of California, Berkeley, CA, USA

    Forrest Huang

  4. Northwestern University, Evanston, IL, USA

    Joshua Hibschman

  5. University of Illinois at Urbana-Champaign, Champaign, IL, USA

    Ranjitha Kumar

  6. Apple, Inc., Seattle, WA, USA

    Jeffrey Nichols

Authors
  1. Biplab Deka
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Bardia Doosti
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Forrest Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Chad Franzen
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Joshua Hibschman
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Daniel Afergan
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Yang Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Ranjitha Kumar
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Tao Dong
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Jeffrey Nichols
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jeffrey Nichols .

Editor information

Editors and Affiliations

  1. Google Research (United States), Mountain View, CA, USA

    Yang Li

  2. Advanced Interactive Technologies Lab, ETH Zurich, Zurich, Switzerland

    Otmar Hilliges

Rights and permissions

Reprints and permissions

Copyright information

© 2021 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Deka, B. et al. (2021). An Early Rico Retrospective: Three Years of Uses for a Mobile App Dataset. In: Li, Y., Hilliges, O. (eds) Artificial Intelligence for Human Computer Interaction: A Modern Approach. Human–Computer Interaction Series. Springer, Cham. https://doi.org/10.1007/978-3-030-82681-9_8

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-82681-9_8

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-82680-2

  • Online ISBN: 978-3-030-82681-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation