Skip to main content
Log in

What can a swiped word tell us more? Demographic and behavioral correlates from shape-writing text entry

  • Original Article
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

Shape-writing (aka gesture typing or swiping) is a word-based text entry method for touchscreen keyboards. It works by landing the finger on (or close to) the first character of the desired word and then sliding over all the other character keys without lifting the finger until the last word character is reached. This generates a trajectory of swiped characters on the keyboard layout which can be translated to a meaningful word by a statistical decoder. We hypothesize that swiping carries rich information about the user, such as demographic (e.g., age or gender) and behavioral (e.g., swiping familiarity or input finger) information. To test our hypothesis, we trained several sequence classifiers using different recurrent neural network architectures to predict demographic and behavioral correlates of users from swipe trajectories. We show that our sequence classifiers are always performing better than a random classifier; therefore, we conclude that cognitive and motor control mechanisms are embodied and reflected in swipe trajectories, validating thus our research hypothesis. Taken together, our results have implications for user privacy. Currently swiping is supported by all mobile vendors and has millions of users, so people may be inadvertently profiled at an unprecedented granularity. Future work should consider new ways of addressing these issues without impacting the user’s swiping experience.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

Availability of data and materials

The swiping dataset we used is publicly available at https://osf.io/sj67f/.

Notes

  1. https://github.com/first20hours/google-10000-english.

  2. https://www.forbes.com/global2000/.

References

  1. Reyal S, Zhai S, Kristensson PO (2015) Performance and user experience of touchscreen and gesture keyboards in a lab setting and in the wild. In: Proceedings SIGCHI conference on human factors in computing systems (CHI), pp. 679–688

  2. Palin K, Feit AM, Kim S, Kristensson PO, Oulasvirta A (2019) How do people type on mobile devices? observations from a study with 37,000 volunteers. In: Proceedings international conference on human-computer interaction with mobile devices and services (MobileHCI), pp. 1–12

  3. Quinn P, Zhai S (2018) Modeling gesture-typing movements. Hum Comput Interact 33(3):234–280

    Article  Google Scholar 

  4. Leiva LA, Kim S, Cui W, Bi X, Oulasvirta A (2021) How we swipe: a large-scale shape-writing dataset and empirical findings. In: Proceedings international conference on human-computer interaction with mobile devices and services (MobileHCI)

  5. Buschek D, Bisinger B, Alt F (2018) ResearchIME: a mobile keyboard application for studying free typing behaviour in the wild. In: Proceedings SIGCHI conference on human factors in computing systems (CHI), pp. 1–14

  6. Henze N, Rukzio E, Boll S (2012) Observational and experimental investigation of typing behaviour using virtual keyboards for mobile devices. In: Proceedings SIGCHI conference on human factors in computing systems (CHI), pp. 2659–2668

  7. Dhakal V, Feit AM, Kristensson PO, Oulasvirta A (2018) Observations on typing from 136 million keystrokes. In: Proceedings SIGCHI conference on human factors in computing systems (CHI), pp. 1–12

  8. Zhai S, Kristensson PO (2003) Shorthand writing on stylus keyboard. In: Proceedings SIGCHI conference on human factors in computing systems (CHI), pp. 97–104

  9. Markussen A, Jakobsen MR, Hornbæk K (2014) Vulture: a mid-air word-gesture keyboard. In: Proceedings SIGCHI conference on human factors in computing systems (CHI), pp. 1073–1082

  10. Zhu S, Zheng J, Zhai S, Bi X (2019) i’sFree: eyes-free gesture typing via a touch-enabled remote control. In: Proceedings SIGCHI conference on human factors in computing systems (CHI), pp. 1–12

  11. Gupta A, Ji C, Yeo H-S, Quigley A, Vogel D (2019) RotoSwype: word-gesture typing using a ring. In: Proceedings SIGCHI conference on human factors in computing systems (CHI), pp. 1–12

  12. Yeo H-S, Phang X-S, Castellucci SJ, Kristensson PO, Quigley A (2017) Investigating tilt-based gesture keyboard entry for single-handed text entry on large devices. In: Proceedings SIGCHI conference on human factors in computing systems (CHI), pp. 4194–4202

  13. Kristensson PO, Zhai S (2007) Command strokes with and without preview: using pen gestures on keyboard for command selection. In: Proceedings SIGCHI conference on human factors in computing systems (CHI), pp. 1137–1146

  14. Alvina J, Griggio CF, Bi X, Mackay WE (2017) CommandBoard: creating a general-purpose command gesture input space for soft keyboard. In: Proceedings ACM symposium on user interface software technology (UIST), pp. 17–28

  15. Cui W, Zheng J, Lewis B, Vogel D, Bi X (2019) HotStrokes: word-gesture shortcuts on a trackpad. In: Proceedings SIGCHI conference on human factors in computing systems (CHI), pp. 1–13

  16. Starov O, Gill P, Nikiforakis N (2016) Are you sure you want to contact us? quantifying the leakage of PII via website contact forms. In: Proceedings PoPETs

  17. Leung C, Ren J, Choffnes D, Wilson C (2016) Should you use the app for that? comparing the privacy implications of app- and web-based online services. In: Proceedings internet measurements conference (IMC)

  18. Leiva LA, Diaz M, Ferrer MA, Plamondon R (2020) Human or machine? it is not what you write, but how you write it. In: Proceedings of the International conference on pattern recognition (ICPR)

  19. Leiva LA, Arapakis I, Iordanou C (2021) My mouse, my rules: privacy issues of behavioral user profiling via mouse tracking. In: Proceedings of ACM SIGIR conference on human information interaction and retrieval (CHIIR)

  20. White RW, Doraiswamy PM, Horvitz E (2018) Detecting neurodegenerative disorders from web search signals. npj Digital Med 1(8)

  21. Gajos KZ, Reinecke K, Donovan M, Stephen CD, Hung AY, Schmahmann JD, Gupta AS (2020) Computer mouse use captures ataxia and parkinsonism, enabling accurate measurement and detection. Mov Disord 35(2)

  22. Chen MC, Anderson JR, Sohn MH (2001) What can a mouse cursor tell us more? correlation of eye/mouse movements on web browsing. In: Proceedings extended abstracts on human factors in computing systems (CHI EA)

  23. Mueller F, Lockerd A (2001) Cheese: tracking mouse movement activity on websites, a tool for user modeling. In: Proceedings extended abstracts on human factors in computing systems (CHIEA)

  24. Huang J, White RW, Buscher G, Wang K (2012) Improving searcher models using mouse cursor activity. In: Proceedings International ACM SIGIR conference on research and development in information retrieval (SIGIR)

  25. Huang J, White R, Buscher G (2012) User see, user point: Gaze and cursor alignment in web search. In: Proceedings SIGCHI conference on human factors in computing systems (CHI)

  26. Navalpakkam V, Jentzsch L, Sayres R, Ravi S, Ahmed A, Smola A (2013) Measurement and modeling of eye-mouse behavior in the presence of nonlinear page layouts. In: Proceedings the web conference (WWW)

  27. Arapakis I, Leiva LA (2016) Predicting user engagement with direct displays using mouse cursor information. In: Proceedings International ACM SIGIR conference on research and development in information retrieval (SIGIR)

  28. Chen Y, Liu Y, Zhang M, Ma S (2017) User satisfaction prediction with mouse movement information in heterogeneous search environment. IEEE Trans Knowl Data Eng 29(11)

  29. Liu Y, Chen Y, Tang J, Sun J, Zhang M, Ma S, Zhu X (2015) Different users, different opinions: predicting search satisfaction with mouse movement information. In: Proceedings International ACM SIGIR conference on research and development in information retrieval (SIGIR)

  30. Arroyo E, Selker T, Wei W (2006) Usability tool for analysis of web designs using mouse tracks. In: Proceedings extended abstracts on human factors in computing systems (CHIEA)

  31. Atterer R, Wnuk M, Schmidt A (2006) Knowing the user’s every move: user activity tracking for website usability evaluation and implicit interaction. In: Proceedings the web conference (WWW)

  32. Leiva LA (2011) Restyling website design via touch-based interactions. In: Proceedings international conference on human-computer interaction with mobile devices and services (MobileHCI)

  33. Krátky P, Chudá D (2018) Recognition of web users with the aid of biometric user model. J Intell Inf Syst 51(3)

  34. Lu H, Rose J, Liu Y, Awad A, Hou L (2017) Combining mouse and eye movement biometrics for user authentication. In: Traoré I, Awad A, Woungang I (eds) Information security practices. Springer

    Google Scholar 

  35. Claypool M, Le P, Wased M, Brown D (2001) Implicit interest indicators. In: Proceedings intelligent user interfaces (IUI)

  36. Shapira B, Taieb-Maimon M, Moskowitz A (2006) Study of the usefulness of known and new implicit indicators and their optimal combination for accurate inference of users interests. In: Proceedings symposium on applied computing (SAC)

  37. Guo Q, Agichtein E (2008) Exploring mouse movements for inferring query intent. In: Proceedings International ACM SIGIR conference on research and development in information retrieval (SIGIR)

  38. Guo Q, Agichtein E (2010) Ready to buy or just browsing? detecting web searcher goals from interaction data. In: Proceedings international ACM SIGIR conference on research and development in information retrieval (SIGIR)

  39. Guo Q, Lagun D, Agichtein E (2012) Predicting web search success with fine-grained interaction data. In: Proceedings international conference on information and knowledge management (CIKM)

  40. Huang J, White RW, Dumais S (2011) No clicks, no problem: using cursor movements to understand and improve search. In: Proceedings SIGCHI conference on human factors in computing systems (CHI)

  41. Guo Q, Agichtein E (2012) Beyond dwell time: estimating document relevance from cursor movements and other post-click searcher behavior. In: Proceedings the web conference (WWW)

  42. Speicher M, Both A, Gaedke M (2013) TellMyRelevance! predicting the relevance of web search results from cursor interactions. In: Proceedings International conference on information and knowledge management (CIKM)

  43. Diriye A, White R, Buscher G, Dumais S (2012) Leaving so soon? understanding and predicting web search abandonment rationales. In: Proceedings international conference on information and knowledge management (CIKM)

  44. Arapakis I, Lalmas M, Cambazoglu BB, Marcos M-C, Jose JM (2014) User engagement in online news: under the scope of sentiment, interest, affect, and gaze. J Assoc Inf Sci Technol 65(10)

  45. Arapakis I, Lalmas M, Valkanas G (2014) Understanding within-content engagement through pattern analysis of mouse gestures. In: Proceedings international conference on information and knowledge management (CIKM)

  46. Hauger D, Paramythis A, Weibelzahl S (2011) Using browser interaction data to determine page reading behavior. In: Proceedings UMAP

  47. Lagun D, Ageev M, Guo Q, Agichtein E (2014) Discovering common motifs in cursor movement data for improving web search. In: Proceedings ACM conference on web search and data mining (WSDM)

  48. Boi P, Fenu G, Spano LD, Vargiu V (2016) Reconstructing user’s attention on the web through mouse movements and perception-based content identification. ACM Trans Appl Percept 13(3)

  49. Arapakis I, Penta A, Joho H, Leiva LA (2020) A price-per-attention auction scheme using mouse cursor information. ACM Trans Inf Syst 38(2)

  50. Accot J, Zhai S (1997) Beyond fitts’ law: models for trajectory-based HCI tasks. In: Proceedings SIGCHI conference on human factors in computing systems (CHI)

  51. Card SK, English WK, Burr BJ (1987) Evaluation of mouse, rate-controlled isometric joystick, step keys, and text keys, for text selection on a CRT. In: Baecker RM, Buxton WAS (eds) Human-computer interaction. Taylor & Francis

    Google Scholar 

  52. Zimmermann P, Guttormsen S, Danuser B, Gomez P (2003) Affective computing - a rationale for measuring mood with mouse and keyboard. Int J Occup Saf Ergon 9

  53. Kaklauskas A, Krutinis M, Seniut M (2009) Biometric mouse intelligent system for student’s emotional and examination process analysis. In: Proceedings ICALT

  54. Azcarraga J, Suarez MT (2012) Predicting academic emotions based on brainwaves, mouse behaviour and personality profile. In: Proceedings PRICAI

  55. Yamauchi T (2013) Mouse trajectories and state anxiety: feature selection with random forest. In: Proceedings ACII

  56. Kapoor A, Burleson W, Picard RW (2007) Automatic prediction of frustration. Int J Hum Comput Stud 65(8)

  57. Yamauchi T, Bowman C (2014) Mining cursor motions to find the gender, experience, and feelings of computer users. In: Proceedings ICDMW

  58. Kratky P, Chuda D (2016) Estimating gender and age of web page visitors from the way they use their mouse. In: Proceedings WWW companion

  59. Pentel A (2017) Predicting age and gender by keystroke dynamics and mouse patterns. In: Adj proceedings UMAP

  60. Vertanen K, Kristensson PO (2011) A versatile dataset for text entry evaluations based on genuine mobile emails. In: Proceedings international conference on human-computer interaction with mobile devices and services (MobileHCI), pp. 295–298

  61. Sutskever I, Vinyals O, Le QV (2014) Sequence to sequence learning with neural networks. In: Proceedings NeurIPS

  62. Du W, Wang Y, Qiao Y (2018) Recurrent spatial-temporal attention network for action recognition in videos. IEEE Trans Image Process 27(3)

  63. Mao J, Xu W, Yang Y, Wang J, Huang Z, Yuille A (2015) Deep captioning with multimodal recurrent neural networks (m-RNN). In: Proceedings ICLR

  64. Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9(8)

  65. Cho K, van Merrienboer B, Gulcehre C, Bougares F, Schwenk H, Bengio Y (2014) Learning phrase representations using RNN encoder-decoder for statistical machine translation. In: Proceedings EMNLP

  66. Jozefowicz R, Zaremba W, Sutskever I (2015) An empirical exploration of recurrent network architectures. In: Proceedings ICML

  67. Chung J, Gulcehre C, Cho K, Bengio Y (2014) Empirical evaluation of gated recurrent neural networks on sequence modeling. In: NeurIPS workshops

  68. Dey R, Salemt FM (2017) Gate-variants of gated recurrent unit (GRU) neural networks. In: Proceedings MWSCAS

  69. Kingma DP, Ba JL (2015) Adam: a method for stochastic optimization. In: Proceedings ICLR

  70. Ranganath R, Gerrish S, Blei DM (2014) Black box variational inference. In: Proceedings AISTATS

  71. Powers DMW (2011) Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlation. J Mach Learn Technol 2(1)

  72. Smith MW, Sharit J, Czaja SJ (1999) Aging, motor control, and the performance of computer mouse tasks. Hum Factors 41(3)

  73. Bahdanau D, Cho K, Bengio Y (2015) Neural machine translation by jointly learning to align and translate. In: Proceedings ICLR

  74. Luong M-T, Pham H, Manning CD (2015) Effective approaches to attention-based neural machine translation. In: Proceedings EMNLP

Download references

Acknowledgements

The experiments presented in this paper were carried out using the HPC facilities of the University of Luxembourg: http://hpc.uni.lu.

Funding

This work was supported by the Horizon 2020 FET program of the European Union through the ERA-NET Cofund funding grant CHIST-ERA-20-BCI-001 and the European Innovation Council Pathfinder program (SYMBIOTIK project, Grant 101071147).

Author information

Authors and Affiliations

Authors

Contributions

DCAL was involved in software, writing — original draft. BAY contributed to methodology, writing — original draft. LAL was involved in conceptualization, methodology, writing — reviewing and editing.

Corresponding authors

Correspondence to Désirée C. A. Lemarquis or Luis A. Leiva.

Ethics declarations

Conflict of interest

The authors have no competing interests to declare that are relevant to the content of this article.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Lemarquis, D.C.A., Yilma, B.A. & Leiva, L.A. What can a swiped word tell us more? Demographic and behavioral correlates from shape-writing text entry. Neural Comput & Applic 35, 15531–15548 (2023). https://doi.org/10.1007/s00521-023-08559-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-023-08559-0

Keywords

Navigation