Should I Stay or Should I Go? Selecting Between Touch and Mid-Air Gestures for Large-Display Interaction

  • Mikkel R. Jakobsen
  • Yvonne Jansen
  • Sebastian Boring
  • Kasper Hornbæk
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9298)


Users can interact with large displays in many ways, including touch and mid-air gestures. However, it remains unclear how these ways compare and when users choose one over the other. In a first experiment, we compare touch and mid-air gestures to identify their relative performance for target acquisition. In a second experiment, participants choose freely between touch and mid-air gestures and we artificially require movement to simulate situations where mid-air is considered beneficial. Results from the first experiment show mid-air to be overall slower than touch depending on the task; in the second experiment, participants mostly chose touch in particular for selecting small targets and they rarely switched between mid-air and touch. Results also show that when faced with an increasing cost of using touch in the form of movement, participants chose mid-air over touch; touch remains as fast as mid-air on average.


Large display Mid-air Touch Freehand gestures User study 



This work has been supported in part by the Danish Council for Strategic Research under grant 10-092316.


  1. 1.
    American Psychological Association. The Publication manual of the American psychological association (6th edn.). Washington, DC (2010)Google Scholar
  2. 2.
    Ball, R., North, C., Bowman, D.A.: Move to improve: promoting physical navigation to increase user performance with large displays. In: Proceedings of CHI. ACM, pp. 191–200 (2007)Google Scholar
  3. 3.
    Banerjee, A., Burstyn, J., Girouard, A., Vertegaal, R.: Pointable: an in-air pointing technique to manipulate out-of-reach targets on tabletops. In: Proceedings of ITS. ACM, pp. 11–20 (2011)Google Scholar
  4. 4.
    Baudisch, P., Cutrell, E., Robbins, D., et al.: Drag-and-pop and drag-and-pick: techniques for accessing remote screen content on touch-and pen-operated systems. In: Proceedings of INTERACT, pp. 57–64 (2003)Google Scholar
  5. 5.
    Bezerianos, A., Isenberg, P.: Perception of visual variables on tiled wall-sized displays for information visualization applications. IEEE Trans. Vis. Comput. Graph. (Proc. InfoVis.) 18(12), 2516–2525 (2012)CrossRefGoogle Scholar
  6. 6.
    Boring, S., Baur, D., Butz, A., Gustafson, S., Baudisch, P.: Touch projector: mobile interaction through video. In: Proceedings of CHI, pp. 2287–2296. ACM (2010)Google Scholar
  7. 7.
    Bowman, D.A., McMahan, R.P., Ragan, E.D.: Questioning naturalism in 3D user interfaces. Commun. ACM 55(9), 78–88 (2012)CrossRefGoogle Scholar
  8. 8.
    Casiez, G., Roussel, N., Vogel, D.: 1 € Filter: a simple speed-based low-pass filter for noisy input in interactive systems. In: Proceedings of CHI, pp. 2527–2530 (2012)Google Scholar
  9. 9.
    Cockburn, A., Quinn, P., Gutwin, C., Ramos, G., Looser, J.: Air pointing: design and evaluation of spatial target acquisition with and without visual feedback. Int. J. Hum. Comput. Stud. 69(6), 401–414 (2011)CrossRefGoogle Scholar
  10. 10.
    Cumming, G.: The new statistics: why and how. Psychol. Sci. 25(1), 7–29 (2014)CrossRefGoogle Scholar
  11. 11.
    Douglas, S.A., Kirkpatrick, A.E., MacKenzie, I.S.: Testing pointing device performance and user assessment with the ISO 9241, Part 9 Standard. In: Proceedings of CHI, pp. 215–222. ACM (1999)Google Scholar
  12. 12.
    Grossman, T., Balakrishnan, R.: The bubble cursor: enhancing target acquisition by dynamic resizing of the cursor’s activation area. In: Proceedings of CHI, pp. 281–290. ACM (2005)Google Scholar
  13. 13.
    Hilliges, O., Izadi, S., Wilson, A.D., Hodges, S., Garcia-Mendoza, A., Butz, A.: Interactions in the air: adding further depth to interactive tabletops. In: Proceedings of UIST, pp. 139–148. ACM (2009)Google Scholar
  14. 14.
    Hinckley, K., Wigdor, D.: Input technologies and techniques. In: Sears, A., Jacko, J.A. (eds.) The Human-Computer Interaction Handbook: Fundamentals, Evolving Technologies and Emerging Applications. CRC Press, Boca Raton (2011)Google Scholar
  15. 15.
    Hornbæk, K., Law, E.L.-C.: Meta-analysis of correlations among usability measures. In: Proceedings of CHI, pp. 617–626. ACM Press (2007)Google Scholar
  16. 16.
    Jakobsen, M.R., Hornbæk, K.: Up close and personal: collaborative work on a high-resolution multitouch wall display. ACM Trans. Comput. Hum. Interact. 21(2), 11:1–11:34 (2014)CrossRefGoogle Scholar
  17. 17.
    Jakobsen, M.R., Hornbæk, K.: Is moving improving? Some effects of locomotion in wall-display interaction. In: Proceedings of CHI, pp. 4169–4178. ACM (2015)Google Scholar
  18. 18.
    Jansen, Y., Dragicevic, P., Fekete, J.-D.: Tangible remote controllers for wall-size displays. In: Proceedings of CHI, pp. 2865–2874. ACM (2012)Google Scholar
  19. 19.
    Khan, A., Fitzmaurice, G., Almeida, D., Burtnyk, N., Kurtenbach, G.: A remote control interface for large displays. In: Proceedings of UIST, pp. 127–136. ACM (2004)Google Scholar
  20. 20.
    Kurdyukova, E., Obaid, M., André, E.: Direct, bodily or mobile interaction?: comparing interaction techniques for personalized public displays. In: Proceedings of MUM, pp. 44:1–44:9. ACM (2012)Google Scholar
  21. 21.
    Mahyar, N., Sarvghad, A., Tory, M.: A closer look at note taking in the co-located collaborative visual analytics process. In: IEEE VAST, pp. 171–178 (2010)Google Scholar
  22. 22.
    Markussen, A., Jakobsen, M.R., Hornbæk, K.: Vulture: a mid-air word-gesture keyboard. In: Proceedings of CHI, pp. 1073–1082. ACM (2014)Google Scholar
  23. 23.
    Marquardt, N., Ballendat, T., Boring, S., Greenberg, S., Hinckley, K.: Gradual engagement: facilitating information exchange between digital devices as a function of proximity. In: Proceedings of ITS, pp. 31–40. ACM (2012)Google Scholar
  24. 24.
    Marquardt, N., Jota, R., Greenberg, S., Jorge, J.A.: The continuous interaction space: interaction techniques unifying touch and gesture on and above a digital surface. In: Campos, P., Graham, N., Jorge, J., Nunes, N., Palanque, P., Winckler, M. (eds.) INTERACT 2011, Part III. LNCS, vol. 6948, pp. 461–476. Springer, Heidelberg (2011)CrossRefGoogle Scholar
  25. 25.
    McCallum, D.C., Irani, P.: ARC-Pad: absolute + relative cursor positioning for large displays with a mobile touchscreen. In: Proceedings of UIST, pp. 153–156. ACM (2009)Google Scholar
  26. 26.
    Müller, J., Bailly, G., Bossuyt, T., Hillgren, N.: MirrorTouch: combining touch and mid-air gestures for public displays. In: Proceedings of MobileHCI, pp. 319–328. ACM (2014)Google Scholar
  27. 27.
    Myers, B.A., Bhatnagar, R., Nichols, J., et al.: Interacting at a distance: measuring the performance of laser pointers and other devices. In: Proceedings of CHI, pp. 33–40. ACM (2002)Google Scholar
  28. 28.
    Nacenta, M.A., Gutwin, C., Aliakseyeu, D., Subramanian, S.: There and back again: cross-display object movement in multi-display environments. Hum. Comput. Interact. 24(1–2), 170–229 (2009)CrossRefGoogle Scholar
  29. 29.
    Nancel, M., Chapuis, O., Pietriga, E., Yang, X.-D., Irani, P.P., Beaudouin-Lafon, M.: High-precision pointing on large wall displays using small handheld devices. In: Proceedings of CHI, pp. 831–840. ACM (2013)Google Scholar
  30. 30.
    Nancel, M., Wagner, J., Pietriga, E., Chapuis, O., Mackay, W.: Mid-air pan-and-zoom on wall-sized displays. In: Proceedings of CHI, pp. 177–186 (2011)Google Scholar
  31. 31.
    Nielsen, J., Levy, J.: Measuring usability: preference vs. performance. Commun. ACM 37(4), 66–75 (1994)CrossRefGoogle Scholar
  32. 32.
    Olsen, D.R., Jr., Nielsen, T.: Laser pointer interaction. In: Proceedings of CHI, pp. 17–22. ACM (2001)Google Scholar
  33. 33.
    Pedersen, E.W., Hornbæk, K.: An experimental comparison of touch interaction on vertical and horizontal surfaces. In: Proceedings of NordiCHI, pp. 370–379. ACM (2012)Google Scholar
  34. 34.
    Pierce, J.S., Forsberg, A.S., Conway, M.J., Hong, S., Zeleznik, R.C., Mine, M.R.: Image plane interaction techniques in 3D immersive environments. In: Proceedings of I3D, p. 39–ff. ACM (1997)Google Scholar
  35. 35.
    Sasangohar, F., MacKenzie, I.S., Scott, S.D.: Evaluation of mouse and touch input for a tabletop display using fitts’ reciprocal tapping task. In: Proceedings of HFES, vol. 53, no. 12, pp. 839–843 (2009)Google Scholar
  36. 36.
    Sauro, J., Lewis, J.R.: Average task times in usability tests: what to report?. In: Proceedings of CHI, pp. 2347–2350. ACM (2010)Google Scholar
  37. 37.
    Schick, A., van de Camp, F., Ijsselmuiden, J., Stiefelhagen, R.: Extending touch: towards interaction with large-scale surfaces. In: Proceedings of ITS, pp. 117–124. ACM (2009)Google Scholar
  38. 38.
    Swaminathan, K., Sato, S.: Interaction design for large displays. Interactions 4(1), 15–24 (1997)CrossRefGoogle Scholar
  39. 39.
    Vogel, D., Balakrishnan, R.: Interactive public ambient displays: transitioning from implicit to explicit, public to personal, interaction with multiple users. In: UIST 2004: Proceedings of UIST, pp. 137–146. ACM (2004)Google Scholar
  40. 40.
    Vogel, D., Balakrishnan, R.: Distant freehand pointing and clicking on very large, high resolution displays. In: Proceedings of UIST, pp. 33–42. ACM (2005)Google Scholar
  41. 41.
    Vogel, D., Baudisch, P.: Shift: a technique for operating pen-based interfaces using touch. In: Proceedings of CHI, pp. 657–666 (2007)Google Scholar
  42. 42.
    Wilson, A.D.: Robust computer vision-based detection of pinching for one and two-handed gesture input. In: Proceedings of UIST, pp. 255–258. ACM (2006)Google Scholar
  43. 43.
    Wilson, A., Shafer, S.: XWand: UI for intelligent spaces. In: Proceedings of CHI, pp. 545–552. ACM (2003)Google Scholar

Copyright information

© IFIP International Federation for Information Processing 2015

Authors and Affiliations

  • Mikkel R. Jakobsen
    • 1
  • Yvonne Jansen
    • 1
  • Sebastian Boring
    • 1
  • Kasper Hornbæk
    • 1
  1. 1.Department of Computer ScienceUniversity of CopenhagenCopenhagenDenmark

Personalised recommendations