Skip to main content
SpringerLink
Log in

Modeling Mobile Interface Tappability Using Crowdsourcing and Deep Learning

  • Chapter
  • First Online:
Artificial Intelligence for Human Computer Interaction: A Modern Approach

Part of the book series: Human–Computer Interaction Series ((HCIS))

Abstract

Tapping is an immensely important gesture in mobile touchscreen interfaces, yet people still frequently are required to learn which elements are tappable through trial and error. Predicting human behavior for this everyday gesture can help mobile app designers understand an important aspect of the usability of their apps without having to run a user study. In this chapter, we present an approach for modeling tappability of mobile interfaces at scale. We conducted large-scale data collection of interface tappability over a rich set of mobile apps using crowdsourcing and computationally investigated a variety of signifiers that people use to distinguish tappable versus not tappable elements. Based on the dataset, we developed and trained a deep neural network that predicts how likely a user will perceive an interface element as tappable versus not tappable. To demonstrate the capability of the trained tappability model, we developed TapShoe, a tool that automatically diagnoses mismatches between the tappability of each element as perceived by a human user—predicted by our model, and the intended or actual tappable state of the element specified by the developer or designer. Our model achieved reasonable accuracy: mean precision 90.2% and recall 87.0%, in matching human perception on identifying tappable UI elements. The tappability model and TapShoe were well received by designers via an informal evaluation with 7 professional interaction designers.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 149.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 199.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 199.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Abadi M, Barham P, Chen J, Chen Z, Davis A, Dean J, Devin M, Ghemawat S, Irving G, Isard M et al (2016) Tensorflow: a system for large-scale machine learning. In: OSDI, vol 16, pp 265–283

    Google Scholar 

  2. Affordances and Design (2015). https://www.interaction-design.org/literature/article/affordances-and-design

  3. Beyond Blue Links: Making Clickable Elements Recognizable (2015). https://www.nngroup.com/articles/clickable-elements/

  4. Borji A (2019) Saliency prediction in the deep learning era: successes and limitations. IEEE transactions on pattern analysis and machine intelligence

    Google Scholar 

  5. Bylinskii Z, Kim NW, O’Donovan P, Alsheikh S, Madan S, Pfister H, Durand F, Russell B, Hertzmann A (2017) Learning visual importance for graphic designs and data visualizations. In: Proceedings of the 30th annual ACM symposium on user interface software and technology, UIST’17. ACM, New York, NY, USA, pp 57–69. https://doi.org/10.1145/3126594.3126653

  6. Deka B, Huang Z, Franzen C, Hibschman J, Afergan D, Li Y, Nichols J, Kumar R (2017) Rico: a mobile app dataset for building data-driven design applications. In: Proceedings of the 30th annual ACM symposium on user interface software and technology, UIST’17. ACM, New York, NY, USA, pp 845–854. https://doi.org/10.1145/3126594.3126651

  7. Deka B, Huang Z, Franzen C, Nichols J, Li Y, Kumar R (2017) Zipt: zero-integration performance testing of mobile app designs. In: Proceedings of the 30th annual ACM symposium on user interface software and technology. ACM, pp 727–736. https://doi.org/10.1145/3126594.3126647

  8. Deka B, Huang Z, Kumar R (2016) Erica: interaction mining mobile apps. In: Proceedings of the 29th annual symposium on user interface software and technology, UIST’16. ACM, New York, NY, USA, pp. 767–776. https://doi.org/10.1145/2984511.2984581

  9. Duan P, Wierzynski C, Nachman L (2020) Optimizing user interface layouts via gradient descent. In: Proceedings of the 2020 CHI conference on human factors in computing systems, CHI’20. Association for Computing Machinery, New York, NY, USA, pp 1–12. https://doi.org/10.1145/3313831.3376589

  10. Flat UI Elements Attract Less Attention and Cause Uncertainty (2017). https://www.nngroup.com/articles/flat-ui-less-attention-cause-uncertainty/

  11. Fleiss JL (1971) Measuring nominal scale agreement among many raters. Psychol Bull 76(5):378

    Google Scholar 

  12. Gaver WW (1991) Technology affordances. In: Proceedings of the SIGCHI conference on human factors in computing systems, CHI’91. ACM, New York, NY, USA, pp 79–84. https://doi.org/10.1145/108844.108856

  13. Gibson JJ (1978) The ecological approach to the visual perception of pictures. Leonardo 11(3):227–235

    Article  Google Scholar 

  14. Greenberg MD, Easterday MW, Gerber EM (2015) Critiki: a scaffolded approach to gathering design feedback from paid crowd workers. In: Proceedings of the 2015 ACM SIGCHI conference on creativity and cognition’, C&C ’15. ACM, New York, NY, USA, pp 235–244. https://doi.org/10.1145/2757226.2757249

  15. Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9(8):1735–1780

    Article  Google Scholar 

  16. Kittur A, Chi EH, Suh B (2008) Crowdsourcing user studies with mechanical turk. In: Proceedings of the SIGCHI conference on human factors in computing systems, CHI’08. ACM, New York, NY, USA, pp 453–456. https://doi.org/10.1145/1357054.1357127

  17. Komarov S, Reinecke K, Gajos KZ (2013) Crowdsourcing performance evaluations of user interfaces. In: Proceedings of the SIGCHI conference on human factors in computing systems, CHI’13. ACM, New York, NY, USA, pp 207–216. https://doi.org/10.1145/2470654.2470684

  18. Landis JR, Koch GG (1977) An application of hierarchical kappa-type statistics in the assessment of majority agreement among multiple observers. Biometrics 363–374

    Google Scholar 

  19. LeCun Y, Bengio Y, Hinton G (2015) Deep learning. Nature 521(7553):436

    Article  Google Scholar 

  20. Li Y, Bengio S, Bailly G (2018) Predicting human performance in vertical menu selection using deep learning. In: Proceedings of the 2018 CHI conference on human factors in computing systems, pp 1–7

    Google Scholar 

  21. Luther K, Tolentino J-L, Wu W, Pavel A, Bailey BP, Agrawala M, Hartmann B, Dow SP (2015) Structuring, aggregating, and evaluating crowdsourced design critique. In: Proceedings of the 18th ACM conference on computer supported cooperative work & social computing, CSCW’15. ACM, New York, NY, USA, pp 473–485. https://doi.org/10.1145/2675133.2675283

  22. Material Design Guidelines (2018). https://material.io/design/

  23. Nair V, Hinton GE (2010) Rectified linear units improve restricted boltzmann machines. In: Proceedings of the 27th international conference on machine learning (ICML), pp 807–814

    Google Scholar 

  24. Nebeling M, Speicher M, Norrie MC (2013) Crowdstudy: general toolkit for crowdsourced evaluation of web interfaces. In: Proceedings of the SIGCHI symposium on engineering interactive computing systems. ACM, pp 255–264. https://doi.org/10.1145/2494603.2480303

  25. Norman D (2013) The design of everyday things: revised and expanded edition, Constellation

    Google Scholar 

  26. Norman DA (1999) Affordance, conventions, and design, Interactions 6(3):38–43. https://doi.org/10.1145/301153.301168

  27. Norman DA (2008) The way i see it: signifiers, not affordances. Interactions 15(6):18–19. https://doi.org/10.1145/1409040.1409044

  28. Pennington J, Socher R, Manning C (2014) Glove: global vectors for word representation. In: Proceedings of the 2014 conference on empirical methods in natural language processing, EMLNP’14, pp 1532–1543

    Google Scholar 

  29. Pfeuffer K, Li Y (2018) Analysis and modeling of grid performance on touchscreen mobile devices. In: Proceedings of the SIGCHI conference on human factors in computing systems, CHI’18. ACM, New York, NY, USA, pp 288:1–288:12. https://doi.org/10.1145/3173574.3173862

  30. Schneider H, Frison K, Wagner J, Butz A (2016) Crowdux: a case for using widespread and lightweight tools in the quest for ux. In: Proceedings of the 2016 ACM conference on designing interactive systems, DIS’16. ACM, New York, NY, USA, pp 415–426. https://doi.org/10.1145/2901790.2901814

  31. Swearngin A, Li Y (2019) Modeling mobile interface tappability using crowdsourcing and deep learning. In: Proceedings of the 2019 CHI conference on human factors in computing systems, CHI ’19. Association for Computing Machinery, New York, NY, USA, pp 1–11. https://doi.org/10.1145/3290605.3300305

  32. Tidwell J (2010) Designing interfaces: patterns for effective interaction design. O’Reilly Media, Inc

    Google Scholar 

  33. Vaswani A, Shazeer N, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser L, Polosukhin I (2017) Attention is all you need. In: Guyon I, Luxburg UV, Bengio S, Wallach H, Fergus R, Vishwanathan S, Garnett R (eds) Advances in neural information processing systems, vol 30. Curran Associates, Inc. https://proceedings.neurips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf

  34. Visual Affordance Testing (2018). http://practicaluxmethods.com/product/visual-affordance-testing/

  35. Wobbrock JO, Aung HH, Rothrock B, Myers BA (2005) Maximizing the guessability of symbolic input. In: Proceedings of the SIGCHI conference on human factors in computing systems extended abstracts. ACM, pp 1869–1872. https://doi.org/10.1145/1056808.1057043

  36. Wu Z, Jiang Y, Liu Y, Ma X (2020) Predicting and diagnosing user engagement with mobile ui animation via a data-driven approach. In: Proceedings of the 2020 CHI conference on human factors in computing systems’, CHI’20. Association for computing machinery, New York, NY, USA, pp 1–13. https://doi.org/10.1145/3313831.3376324

  37. Xu A, Huang S-W, Bailey B (2014) Voyant: generating structured feedback on visual designs using a crowd of non-experts. In: Proceedings of the 17th ACM conference on computer supported cooperative work & social computing. ACM, pp 1433–1444. https://doi.org/10.1145/2531602.2531604

  38. Yuan A, Li Y (2020) Modeling human visual search performance on realistic webpages using analytical and deep learning methods. In: Proceedings of the 2020 CHI conference on human factors in computing systems, CHI’20. Association for Computing Machinery, New York, NY, USA, pp 1–12. https://doi.org/10.1145/3313831.3376870

  39. Zheng Q, Jiao J, Cao Y, Lau RW (2018) Task-driven webpage saliency. In: Proceedings of the European conference on computer vision (ECCV)

    Google Scholar 

  40. Zhou J, Tang Z, Zhao M, Ge X, Zhuang F, Zhou M, Zou L, Yang C, Xiong H (2020) Intelligent exploration for user interface modules of mobile app with collective learning. In: Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, KDD’20. Association for Computing Machinery, New York, NY, USA, pp 3346–3355. https://doi.org/10.1145/3394486.3403387

Download references

Author information

Authors and Affiliations

  1. University of Washington, Seattle, WA, USA

    Amanda Swearngin

  2. Work conducted during an internship at Google Research, Mountain View, USA

    Amanda Swearngin

  3. Google Research, Mountain View, CA, USA

    Yang Li

Authors
  1. Amanda Swearngin
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yang Li
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Amanda Swearngin .

Editor information

Editors and Affiliations

  1. Google Research (United States), Mountain View, CA, USA

    Yang Li

  2. Advanced Interactive Technologies Lab, ETH Zurich, Zurich, Switzerland

    Otmar Hilliges

Rights and permissions

Reprints and permissions

Copyright information

© 2021 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Swearngin, A., Li, Y. (2021). Modeling Mobile Interface Tappability Using Crowdsourcing and Deep Learning. In: Li, Y., Hilliges, O. (eds) Artificial Intelligence for Human Computer Interaction: A Modern Approach. Human–Computer Interaction Series. Springer, Cham. https://doi.org/10.1007/978-3-030-82681-9_3

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-82681-9_3

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-82680-2

  • Online ISBN: 978-3-030-82681-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation