Capturing You Watching You: Characterizing Visual-Motor Dynamics in Touchscreen Interactions

  • Leslie M. BlahaEmail author
  • Joseph W. Houpt
  • Mary E. Frame
  • Jacob A. Kern
Conference paper
Part of the Mathematics and Visualization book series (MATHVISUAL)


The relationship between where people look and where people reach has been studied since the dawn of experimental psychology. This relationship has implications for the designs of interactive visualizations, particularly for applications involving touchscreens. We present a new visual-motor analytics dashboard for the joint study of eye movement and hand/finger movement dynamics. Our modular approach combines real-time playback of gaze and finger-dragging behavior together with statistical models quantifying the dynamics of both modalities. To aid in visualization and inference with these data, we apply Gaussian process regression models which capture the similarities and differences between eye and finger movements, while providing a statistical model of the observed functional data. Smooth estimates of the dynamics are included in the dashboard to enable visual-analytic exploration of visual-motor behaviors on touchscreen interfaces.


Radial Basis Function Gaussian Process Finger Movement Gaussian Process Regression Finger Position 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.



The authors thank three anonymous reviewers for their feedback on this chapter. The views expressed in this paper are those of the authors and do not reflect the official policy or position of the Department of Defense or the U.S. Government. This work was supported in part by AFOSR LRIR to L.M.B. and AFOSR grant FA9550-13-1-0087 to J.W.H. Distribution A: Approved for public release; distribution unlimited. 88ABW Cleared 08/26/2015; 88ABW-2015-4098.


  1. 1.
    Abrams, R.A., Meyer, D.E., Kornblum, S.: Speed and accuracy of saccadic eye movements: characteristics of impulse variability in the oculomotor system. J. Exp. Psychol. Hum. Percept. Perform. 15 (3), 529–543 (1989)CrossRefGoogle Scholar
  2. 2.
    Abrams, R.A., Meyer, D.E., Kornblum, S.: Eye-hand coordination: oculomotor control in rapid aimed limb movements. J. Exp. Psychol. Hum. Percept. Perform. 16 (2):248–267 (1990)CrossRefGoogle Scholar
  3. 3.
    Blaha, L., Schill, M.T.: Modeling touch interactions on very large touchscreens. In: Proceedings of the 36th Annual Meeting of the Cognitive Science Society, Quebec City (2014)Google Scholar
  4. 4.
    Binsted, G., Chua, R., Helsen, W., Elliott, D.: Eye-hand coordination in goal-directed aiming. Hum. Mov. Sci. 20 (4), 563–585 (2001)CrossRefGoogle Scholar
  5. 5.
    Bizzi, E., Kalil, R.E., Tagliasco, V.: Eye-head coordination in monkeys: evidence for centrally patterned organization. Science 173 (3995), 452–454 (1971)CrossRefGoogle Scholar
  6. 6.
    Bock, O.: Contribution of retinal versus extraretinal signals towards visual localization in goal-directed movements. Exp. Brain Res. 64 (3), 476–482 (1986)CrossRefGoogle Scholar
  7. 7.
    Bostock, M., Ogievetsky, V., Heer, J.: D3: data driven documents. IEEE Trans. Visual. Comput. Graph. 17 (12), 2301–2309 (2011)CrossRefGoogle Scholar
  8. 8.
    Cöltekin, A., Demsar, U., Brychtová, A., Vandrol, J.: Eye-hand coordination during visual search on geographic displays. In: Proceedings of the 2nd International Workshop on Eye Tracking for Spatial Research (ET4S 2014). ACM, New York (2014)Google Scholar
  9. 9.
    De Leeuw, J.R.: jsPsych: a JavaScript library for creating behavioral experiments in a web browser. Behav. Res. Methods 47 (1), 1–12 (2015)Google Scholar
  10. 10.
    Demšar, U., Çöltekin, A.: Quantifying the interactions between eye and mouse movements on spatial visual interfaces through trajectory visualisations. In: Workshop on analysis of movement data at GIScience, Vienna, pp. 23–26 (2014)Google Scholar
  11. 11.
    Epelboim, J., Steinman, R.M., Kowler, E., Edwards, M., Pizlo, Z., Erkelens, C.J., Collewijn, H.: The function of visual search and memory in sequential looking tasks. Vis. Res. 35 (23), 3401–3422 (1995)CrossRefGoogle Scholar
  12. 12.
    Fitts, P.M.: The information capacity of the human motor system in controlling amplitude of movement. J. Exp. Psycholo. 47, 381–391 (1954)CrossRefGoogle Scholar
  13. 13.
    Franco-Watkins, A.M., Johnson, J.G.: Applying the decision moving window to risky choice: comparison of eye-tracking and mouse-tracing methods. Judgm. Decis. Making 6 (8), 740–749 (2011)Google Scholar
  14. 14.
    Freeman, J.B., Ambady, N.: MouseTracker: software for studying real-time mental processing using a computer mouse-tracking method. Behav. Res. Methods 42 (1), 226–241 (2010)CrossRefGoogle Scholar
  15. 15.
    Han, J.Y.: Low-cost multi-touch sensing through frustrated total internal reflection. In: Proceedings of the 18th Annual ACM Symposium on User Interface Software and Technology, pp. 115–118. ACM, Seattle (2005)Google Scholar
  16. 16.
    Hayhoe, M., Ballard, D.: Eye movements in natural behavior. Trends Cogn. Sci. 9 (4), 188–194 (2005)CrossRefGoogle Scholar
  17. 17.
    ISO9241-400: Ergonomics of Human-System Interaction – Part 400: Principles and Requirements for Physical Input Devices, Geneva (2007)Google Scholar
  18. 18.
    Jagacinski, R.J., Flach, J.M.: Control Theory for Humans: Quantitative Approaches to Modeling Performance. CRC Press, Mahwah (2003)Google Scholar
  19. 19.
    Land, M.F., Hayhoe, M.: In what ways do eye movements contribute to everyday activities? Vis. Res. 41 (25), 3559–3565 (2001)CrossRefGoogle Scholar
  20. 20.
    Li, X., Cöltekin, A., Kraak, M.-J.: Visual exploration of eye movement data using the space-time-cube. In: Geographic Information Science, pp. 295–309. Springer, Berlin (2010)Google Scholar
  21. 21.
    McKinstry, C., Dale, R., Spivey, M.J.: Action dynamics reveal parallel competition in decision making. Psychol. Sci. 19 (1), 22–24 (2008)CrossRefGoogle Scholar
  22. 22.
    Patterson, R.E., Blaha, L.M., Grinstein, G.G., Liggett, K.K., Kaveney, D.E., Sheldon, K.C., Havig, P.R., Moore, J.A.: A human cognition framework for information visualization. Comput. Graph. 42, 42–58 (2014)CrossRefGoogle Scholar
  23. 23.
    Pelz, J., Hayhoe, M., Loeber, R.: The coordination of eye, head, and hand movements in a natural task. Exp. Brain Res. 139 (3), 266–277 (2001)CrossRefGoogle Scholar
  24. 24.
    Prablanc, C., Desmurget, M., Gréa, H.: Neural control of on-line guidance of hand reaching movements. Prog. Brain Res. 142, 155–170 (2003)CrossRefGoogle Scholar
  25. 25.
    Prablanc, C., Echallier, J., Jeannerod, M., Komilis, E.: Optimal response of eye and hand motor systems in pointing at a visual target. I. Spatio-temporal characteristics of eye and hand movements and their relationship when varying the amount of visual information. Biol. Cybern. 35 (3), 113–124 (1978)Google Scholar
  26. 26.
    R Development Core Team: R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna (2011)Google Scholar
  27. 27.
    Rasmussen, C.E., Williams, C.K.I.: Gaussian Processes for Machine Learning. The MIT Press, Cambridge (2006)zbMATHGoogle Scholar
  28. 28.
    Richardson, D.C., Dale, R.: Looking to understand: the coupling between speakers’ and listeners’ eye movements and its relationship to discourse comprehension. Cogn. Sci. 29 (6), 1045–1060 (2005)CrossRefGoogle Scholar
  29. 29.
    Song, J.-H., Nakayama, K.: Hidden cognitive states revealed in choice reaching tasks. Trends Cogn. Sci. 13 (8), 360–366 (2009)CrossRefGoogle Scholar
  30. 30.
    Soukoreff, R.W., MacKenzie, I.S.: Towards a standard for pointing device evaluation: perspectives on 27 years of Fitts’ law research in HCI. Int. J. Hum. Comput. Stud. 61 (6), 751–789 (2004)Google Scholar
  31. 31.
    Spivey, M.J., Dale, R.: Continuous dynamics in real-time cognition. Curr. Dir. Psychol. Sci. 15 (5), 207–211 (2006)CrossRefGoogle Scholar
  32. 32.
    Vanhatalo, J., Riihimäki, J., Hartikainen, J., Jylänki, P., Tolvanen, V., Vehtari, A.: GPstuff: Bayesian modeling with gaussian processes. J. Mach. Learn. Res. 14 (1), 1175–1179 (2013)MathSciNetzbMATHGoogle Scholar
  33. 33.
    Woodworth, R.S.: Accuracy of voluntary movement. Psychol. Rev. Monogr. Suppl. 3 (3), i (1899)Google Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  • Leslie M. Blaha
    • 1
    Email author
  • Joseph W. Houpt
    • 2
  • Mary E. Frame
    • 3
  • Jacob A. Kern
    • 2
  1. 1.Air Force Research LaboratoryDaytonUSA
  2. 2.Wright State UniversityDaytonUSA
  3. 3.Miami UniversityOxfordUSA

Personalised recommendations