Human-Computer Interaction

INTERACT 2015: Human-Computer Interaction – INTERACT 2015 pp 455-473 | Cite as

Should I Stay or Should I Go? Selecting Between Touch and Mid-Air Gestures for Large-Display Interaction

  • Mikkel R. Jakobsen
  • Yvonne Jansen
  • Sebastian Boring
  • Kasper Hornbæk
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9298)


Users can interact with large displays in many ways, including touch and mid-air gestures. However, it remains unclear how these ways compare and when users choose one over the other. In a first experiment, we compare touch and mid-air gestures to identify their relative performance for target acquisition. In a second experiment, participants choose freely between touch and mid-air gestures and we artificially require movement to simulate situations where mid-air is considered beneficial. Results from the first experiment show mid-air to be overall slower than touch depending on the task; in the second experiment, participants mostly chose touch in particular for selecting small targets and they rarely switched between mid-air and touch. Results also show that when faced with an increasing cost of using touch in the form of movement, participants chose mid-air over touch; touch remains as fast as mid-air on average.


Large display Mid-air Touch Freehand gestures User study 


  1. 1.
    American Psychological Association. The Publication manual of the American psychological association (6th edn.). Washington, DC (2010)Google Scholar
  2. 2.
    Ball, R., North, C., Bowman, D.A.: Move to improve: promoting physical navigation to increase user performance with large displays. In: Proceedings of CHI. ACM, pp. 191–200 (2007)Google Scholar
  3. 3.
    Banerjee, A., Burstyn, J., Girouard, A., Vertegaal, R.: Pointable: an in-air pointing technique to manipulate out-of-reach targets on tabletops. In: Proceedings of ITS. ACM, pp. 11–20 (2011)Google Scholar
  4. 4.
    Baudisch, P., Cutrell, E., Robbins, D., et al.: Drag-and-pop and drag-and-pick: techniques for accessing remote screen content on touch-and pen-operated systems. In: Proceedings of INTERACT, pp. 57–64 (2003)Google Scholar
  5. 5.
    Bezerianos, A., Isenberg, P.: Perception of visual variables on tiled wall-sized displays for information visualization applications. IEEE Trans. Vis. Comput. Graph. (Proc. InfoVis.) 18(12), 2516–2525 (2012)CrossRefGoogle Scholar
  6. 6.
    Boring, S., Baur, D., Butz, A., Gustafson, S., Baudisch, P.: Touch projector: mobile interaction through video. In: Proceedings of CHI, pp. 2287–2296. ACM (2010)Google Scholar
  7. 7.
    Bowman, D.A., McMahan, R.P., Ragan, E.D.: Questioning naturalism in 3D user interfaces. Commun. ACM 55(9), 78–88 (2012)CrossRefGoogle Scholar
  8. 8.
    Casiez, G., Roussel, N., Vogel, D.: 1 € Filter: a simple speed-based low-pass filter for noisy input in interactive systems. In: Proceedings of CHI, pp. 2527–2530 (2012)Google Scholar
  9. 9.
    Cockburn, A., Quinn, P., Gutwin, C., Ramos, G., Looser, J.: Air pointing: design and evaluation of spatial target acquisition with and without visual feedback. Int. J. Hum. Comput. Stud. 69(6), 401–414 (2011)CrossRefGoogle Scholar
  10. 10.
    Cumming, G.: The new statistics: why and how. Psychol. Sci. 25(1), 7–29 (2014)CrossRefGoogle Scholar
  11. 11.
    Douglas, S.A., Kirkpatrick, A.E., MacKenzie, I.S.: Testing pointing device performance and user assessment with the ISO 9241, Part 9 Standard. In: Proceedings of CHI, pp. 215–222. ACM (1999)Google Scholar
  12. 12.
    Grossman, T., Balakrishnan, R.: The bubble cursor: enhancing target acquisition by dynamic resizing of the cursor’s activation area. In: Proceedings of CHI, pp. 281–290. ACM (2005)Google Scholar
  13. 13.
    Hilliges, O., Izadi, S., Wilson, A.D., Hodges, S., Garcia-Mendoza, A., Butz, A.: Interactions in the air: adding further depth to interactive tabletops. In: Proceedings of UIST, pp. 139–148. ACM (2009)Google Scholar
  14. 14.
    Hinckley, K., Wigdor, D.: Input technologies and techniques. In: Sears, A., Jacko, J.A. (eds.) The Human-Computer Interaction Handbook: Fundamentals, Evolving Technologies and Emerging Applications. CRC Press, Boca Raton (2011)Google Scholar
  15. 15.
    Hornbæk, K., Law, E.L.-C.: Meta-analysis of correlations among usability measures. In: Proceedings of CHI, pp. 617–626. ACM Press (2007)Google Scholar
  16. 16.
    Jakobsen, M.R., Hornbæk, K.: Up close and personal: collaborative work on a high-resolution multitouch wall display. ACM Trans. Comput. Hum. Interact. 21(2), 11:1–11:34 (2014)CrossRefGoogle Scholar
  17. 17.
    Jakobsen, M.R., Hornbæk, K.: Is moving improving? Some effects of locomotion in wall-display interaction. In: Proceedings of CHI, pp. 4169–4178. ACM (2015)Google Scholar
  18. 18.
    Jansen, Y., Dragicevic, P., Fekete, J.-D.: Tangible remote controllers for wall-size displays. In: Proceedings of CHI, pp. 2865–2874. ACM (2012)Google Scholar
  19. 19.
    Khan, A., Fitzmaurice, G., Almeida, D., Burtnyk, N., Kurtenbach, G.: A remote control interface for large displays. In: Proceedings of UIST, pp. 127–136. ACM (2004)Google Scholar
  20. 20.
    Kurdyukova, E., Obaid, M., André, E.: Direct, bodily or mobile interaction?: comparing interaction techniques for personalized public displays. In: Proceedings of MUM, pp. 44:1–44:9. ACM (2012)Google Scholar
  21. 21.
    Mahyar, N., Sarvghad, A., Tory, M.: A closer look at note taking in the co-located collaborative visual analytics process. In: IEEE VAST, pp. 171–178 (2010)Google Scholar
  22. 22.
    Markussen, A., Jakobsen, M.R., Hornbæk, K.: Vulture: a mid-air word-gesture keyboard. In: Proceedings of CHI, pp. 1073–1082. ACM (2014)Google Scholar
  23. 23.
    Marquardt, N., Ballendat, T., Boring, S., Greenberg, S., Hinckley, K.: Gradual engagement: facilitating information exchange between digital devices as a function of proximity. In: Proceedings of ITS, pp. 31–40. ACM (2012)Google Scholar
  24. 24.
    Marquardt, N., Jota, R., Greenberg, S., Jorge, J.A.: The continuous interaction space: interaction techniques unifying touch and gesture on and above a digital surface. In: Campos, P., Graham, N., Jorge, J., Nunes, N., Palanque, P., Winckler, M. (eds.) INTERACT 2011, Part III. LNCS, vol. 6948, pp. 461–476. Springer, Heidelberg (2011)CrossRefGoogle Scholar
  25. 25.
    McCallum, D.C., Irani, P.: ARC-Pad: absolute + relative cursor positioning for large displays with a mobile touchscreen. In: Proceedings of UIST, pp. 153–156. ACM (2009)Google Scholar
  26. 26.
    Müller, J., Bailly, G., Bossuyt, T., Hillgren, N.: MirrorTouch: combining touch and mid-air gestures for public displays. In: Proceedings of MobileHCI, pp. 319–328. ACM (2014)Google Scholar
  27. 27.
    Myers, B.A., Bhatnagar, R., Nichols, J., et al.: Interacting at a distance: measuring the performance of laser pointers and other devices. In: Proceedings of CHI, pp. 33–40. ACM (2002)Google Scholar
  28. 28.
    Nacenta, M.A., Gutwin, C., Aliakseyeu, D., Subramanian, S.: There and back again: cross-display object movement in multi-display environments. Hum. Comput. Interact. 24(1–2), 170–229 (2009)CrossRefGoogle Scholar
  29. 29.
    Nancel, M., Chapuis, O., Pietriga, E., Yang, X.-D., Irani, P.P., Beaudouin-Lafon, M.: High-precision pointing on large wall displays using small handheld devices. In: Proceedings of CHI, pp. 831–840. ACM (2013)Google Scholar
  30. 30.
    Nancel, M., Wagner, J., Pietriga, E., Chapuis, O., Mackay, W.: Mid-air pan-and-zoom on wall-sized displays. In: Proceedings of CHI, pp. 177–186 (2011)Google Scholar
  31. 31.
    Nielsen, J., Levy, J.: Measuring usability: preference vs. performance. Commun. ACM 37(4), 66–75 (1994)CrossRefGoogle Scholar
  32. 32.
    Olsen, D.R., Jr., Nielsen, T.: Laser pointer interaction. In: Proceedings of CHI, pp. 17–22. ACM (2001)Google Scholar
  33. 33.
    Pedersen, E.W., Hornbæk, K.: An experimental comparison of touch interaction on vertical and horizontal surfaces. In: Proceedings of NordiCHI, pp. 370–379. ACM (2012)Google Scholar
  34. 34.
    Pierce, J.S., Forsberg, A.S., Conway, M.J., Hong, S., Zeleznik, R.C., Mine, M.R.: Image plane interaction techniques in 3D immersive environments. In: Proceedings of I3D, p. 39–ff. ACM (1997)Google Scholar
  35. 35.
    Sasangohar, F., MacKenzie, I.S., Scott, S.D.: Evaluation of mouse and touch input for a tabletop display using fitts’ reciprocal tapping task. In: Proceedings of HFES, vol. 53, no. 12, pp. 839–843 (2009)Google Scholar
  36. 36.
    Sauro, J., Lewis, J.R.: Average task times in usability tests: what to report?. In: Proceedings of CHI, pp. 2347–2350. ACM (2010)Google Scholar
  37. 37.
    Schick, A., van de Camp, F., Ijsselmuiden, J., Stiefelhagen, R.: Extending touch: towards interaction with large-scale surfaces. In: Proceedings of ITS, pp. 117–124. ACM (2009)Google Scholar
  38. 38.
    Swaminathan, K., Sato, S.: Interaction design for large displays. Interactions 4(1), 15–24 (1997)CrossRefGoogle Scholar
  39. 39.
    Vogel, D., Balakrishnan, R.: Interactive public ambient displays: transitioning from implicit to explicit, public to personal, interaction with multiple users. In: UIST 2004: Proceedings of UIST, pp. 137–146. ACM (2004)Google Scholar
  40. 40.
    Vogel, D., Balakrishnan, R.: Distant freehand pointing and clicking on very large, high resolution displays. In: Proceedings of UIST, pp. 33–42. ACM (2005)Google Scholar
  41. 41.
    Vogel, D., Baudisch, P.: Shift: a technique for operating pen-based interfaces using touch. In: Proceedings of CHI, pp. 657–666 (2007)Google Scholar
  42. 42.
    Wilson, A.D.: Robust computer vision-based detection of pinching for one and two-handed gesture input. In: Proceedings of UIST, pp. 255–258. ACM (2006)Google Scholar
  43. 43.
    Wilson, A., Shafer, S.: XWand: UI for intelligent spaces. In: Proceedings of CHI, pp. 545–552. ACM (2003)Google Scholar

Copyright information

© IFIP International Federation for Information Processing 2015

Authors and Affiliations

  • Mikkel R. Jakobsen
    • 1
  • Yvonne Jansen
    • 1
  • Sebastian Boring
    • 1
  • Kasper Hornbæk
    • 1
  1. 1.Department of Computer ScienceUniversity of CopenhagenCopenhagenDenmark

Personalised recommendations