The Continuous Interaction Space: Interaction Techniques Unifying Touch and Gesture on and above a Digital Surface

  • Nicolai Marquardt
  • Ricardo Jota
  • Saul Greenberg
  • Joaquim A. Jorge
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6948)


The rising popularity of digital table surfaces has spawned considerable interest in new interaction techniques. Most interactions fall into one of two modalities: 1) direct touch and multi-touch (by hand and by tangibles) directly on the surface, and 2) hand gestures above the surface. The limitation is that these two modalities ignore the rich interaction space between them. To move beyond this limitation, we first contribute a unification of these discrete interaction modalities called the continuous interaction space. The idea is that many interaction techniques can be developed that go beyond these two modalities, where they can leverage the space between them. That is, we believe that the underlying system should treat the space on and above the surface as a continuum, where a person can use touch, gestures, and tangibles anywhere in the space and naturally move between them. Our second contribution illustrates this, where we introduce a variety of interaction categories that exploit the space between these modalities. For example, with our Extended Continuous Gestures category, a person can start an interaction with a direct touch and drag, then naturally lift off the surface and continue their drag with a hand gesture over the surface. For each interaction category, we implement an example (or use prior work) that illustrates how that technique can be applied. In summary, our primary contribution is to broaden the design space of interaction techniques for digital surfaces, where we populate the continuous interaction space both with concepts and examples that emerge from considering this space as a continuum.


Touch gestures surfaces interactive tabletops 3D interaction tangibles portable devices continuous interaction space 


  1. 1.
    Bartindale, T., Harrison, C.: Stacks on the surface: resolving physical order using fiducial markers with structured transparency. In: Proc. of ITS 2009. ACM, New York (2009)Google Scholar
  2. 2.
    Baudisch, P., Becker, T., Rudeck, F.: Lumino: tangible blocks for tabletop computers based on glass fiber bundles. In: Proc. of CHI 2010, pp. 1165–1174. ACM, New York (2010)Google Scholar
  3. 3.
    Benko, H., Ishak, E.W., Feiner, S.: Cross-dimensional gestural interaction techniques for hybrid immersive environments. In: Proc. of IEEE VR 2005, pp. 209–216 (2005)Google Scholar
  4. 4.
    Benko, H., Saponas, T.S., Morris, D., Tan, D.: Enhancing input on and above the interactive surface with muscle sensing. In: Proc. of ITS 2009, pp. 93–100. ACM, New York (2009)Google Scholar
  5. 5.
    Bier, E.A., Stone, M.C., Pier, K., Buxton, W., DeRose, T.D.: Toolglass and magic lenses: the see-through interface. In: Proc. of SIGGRAPH 1993, pp. 73–80. ACM, New York (1993)CrossRefGoogle Scholar
  6. 6.
    Buxton, W.: A three-state model of graphical input. In: Proc. of INTERACT 1990, pp. 449–456. North-Holland Publishing Co, Amsterdam (1990)Google Scholar
  7. 7.
    Cao, X., Wilson, A., Balakrishnan, R., Hinckley, K., Hudson, S.: ShapeTouch: Leveraging Contact Shape on Interactive Surfaces. In: Proc. of TABLETOP 2008. IEEE, Los Alamitos (2008)Google Scholar
  8. 8.
    Cockburn, A., Bryant, A.: Leogo: An Equal Opportunity User Interface for Programming. Journal of Visual Languages & Computing 8(5-6), 601–619 (1997)CrossRefGoogle Scholar
  9. 9.
    Echtler, F., Huber, M., Klinker, G.: Shadow tracking on multi-touch tables. In: Proc. of AVI 2008, pp. 388–391. ACM, New York (2008)CrossRefGoogle Scholar
  10. 10.
    Epps, J., Lichman, S., Wu, M.: A study of hand shape use in tabletop gesture interaction. In: CHI 2006 Extended Abstracts, pp. 748–753. ACM, New York (2006)CrossRefGoogle Scholar
  11. 11.
    Grossman, T., Wigdor, D.: Going Deeper: a Taxonomy of 3D on the Tabletop. In: Proc. of TABLETOP 2007, pp. 137–144. IEEE, Los Alamitos (2007)Google Scholar
  12. 12.
    Grossman, T., Balakrishnan, R.: The design and evaluation of selection techniques for 3D volumetric displays. In: Proc. of UIST 2006, pp. 3–12. ACM, New York (2006)CrossRefGoogle Scholar
  13. 13.
    Hancock, M., Carpendale, S., Cockburn, A.: Shallow-depth 3d interaction: design and evaluation of one-, two- and three-touch techniques. In: Proc. of CHI 2007, pp. 1147–1156. ACM, New York (2007)Google Scholar
  14. 14.
    Hancock, M., ten Cate, T., Carpendale, S., Isenberg, T.: Supporting sandtray therapy on an interactive tabletop. In: Proc. of CHI 2010, pp. 2133–2142. ACM, New York (2010)Google Scholar
  15. 15.
    Hancock, M., ten Cate, T., Carpendale, S.: Sticky tools: full 6DOF force-based interaction for multi-touch tables. In: Proc. of ITS 2009, pp. 133–140. ACM, New York (2009)Google Scholar
  16. 16.
    Hilliges, O., Izadi, S., Wilson, A.D., Hodges, S., Garcia-Mendoza, A., Butz, A.: Interactions in the air: adding further depth to interactive tabletops. In: Proc. of UIST 2009, pp. 139–148. ACM, New York (2009)CrossRefGoogle Scholar
  17. 17.
    Izadi, S., Hodges, S., Taylor, S., et al.: Going beyond the display: a surface technology with an electronically switchable diffuser. In: Proc. of UIST 2008, pp. 269–278. ACM, New York (2008)CrossRefGoogle Scholar
  18. 18.
    Kattinakere, R.S., Grossman, T., Subramanian, S.: Modeling steering within above-the-surface interaction layers. In: Proc. of CHI 2007, pp. 317–326. ACM, New York (2007)Google Scholar
  19. 19.
    Lee, J., Lee, J., Kim, H., Kim, J.: Gesture-Based Interactions on Multiple Large Displays with a Tabletop Interface. In: Universal Access in Human-Computer Interaction. Ambient Interaction, pp. 936–942 (2007)Google Scholar
  20. 20.
    Lucero, A., Aliakseyeu, D., Martens, J.: Augmenting Mood Boards: Flexible and Intuitive Interaction in the Context of the Design Studio. In: Proc. of TABLETOP 2007, pp. 147–154. IEEE, Los Alamitos (2007)Google Scholar
  21. 21.
    Malik, S., Ranjan, A., Balakrishnan, R.: Interacting with large displays from a distance with vision-tracked multi-finger gestural input. In: Proc. of UIST 2005, pp. 43–52. ACM, New York (2005)CrossRefGoogle Scholar
  22. 22.
    Marquardt, N., Kiemer, J., Greenberg, S.: What Caused That Touch? Expressive Interaction with a Surface through Fiduciary-Tagged Gloves. In: Proc. of ITS 2010, pp. 139–142. ACM, New York (2010)Google Scholar
  23. 23.
    Parker, J.K., Mandryk, R.L., Inkpen, K.M.: TractorBeam: seamless integration of local and remote pointing for tabletop displays. In: Proc. of Graphics Interface - GI 2005, pp. 33–40. Canadian Information Processing Society (2005)Google Scholar
  24. 24.
    Parker, J.K., Mandryk, R.L., Inkpen, K.M.: Integrating Point and Touch for Interaction with Digital Tabletop Displays. IEEE Comput. Graph. Appl. 26(5), 28–35 (2006)CrossRefGoogle Scholar
  25. 25.
    Pinelle, D., Nacenta, M., Gutwin, C., Stach, T.: The effects of co-present embodiments on awareness and collaboration in tabletop groupware. In: Proc. of Graphics Interface - GI 2008, pp. 1–8. Canadian Information Processing Society (2008)Google Scholar
  26. 26.
    Shoemaker, G., Tsukitani, T., Kitamura, Y., Booth, K.S.: Body-centric interaction techniques for very large wall displays. In: Proc. of NordiCHI 2010, pp. 463–472. ACM, New York (2010)Google Scholar
  27. 27.
    Spindler, M., Stellmach, S., Dachselt, R.: PaperLens: advanced magic lens interaction above the tabletop. In: Proc. of ITS 2009, pp. 69–76. ACM, New York (2009)Google Scholar
  28. 28.
    Starner, T., Leibe, B., Minnen, D., Westeyn, T., Hurst, A., Weeks, J.: The perceptive workbench: Computer-vision-based gesture tracking, object tracking, and 3D reconstruction for augmented desks. Machine Vision and Applications 14(1) (2003)Google Scholar
  29. 29.
    Sturman, D.J., Zeltzer, D.: A Survey of Glove-based Input. IEEE Comput. Graph. Appl. 14(1), 30–39 (1994)CrossRefGoogle Scholar
  30. 30.
    Subramanian, S., Aliakseyeu, D., Lucero, A.: Multi-layer interaction for digital tables. In: Proc. UIST 2006, pp. 269–272. ACM, New York (2006)CrossRefGoogle Scholar
  31. 31.
    Ullmer, B., Ishii, H.: The metaDESK: models and prototypes for tangible user interfaces. In: Proc. of UIST 1997, pp. 223–232. ACM, New York (1997)CrossRefGoogle Scholar
  32. 32.
    Underkoffler, J., Ishii, H.: Urp: a luminous-tangible workbench for urban planning and design. In: Proc. of CHI 1999, pp. 386–393. ACM, New York (1999)Google Scholar
  33. 33.
    Vogel, D., Balakrishnan, R.: Distant freehand pointing and clicking on very large, high resolution displays. In: Proc. of UIST 2005, pp. 33–42. ACM, New York (2005)CrossRefGoogle Scholar
  34. 34.
    Voida, S., Tobiasz, M., Stromer, J., Isenberg, P., Carpendale, S.: Getting practical with interactive tabletop displays: designing for dense data, “fat fingers,” diverse interactions, and face-to-face collaboration. In: Proc. of ITS 2009, pp. 109–116. ACM, New York (2009)Google Scholar
  35. 35.
    Weiss, M., Schwarz, F., Jakubowski, S., Borchers, J.: Madgets: actuating widgets on interactive tabletops. In: Proc. of UIST 2010, pp. 293–302. ACM, New York (2010)Google Scholar
  36. 36.
    Wilson, A.: Depth-Sensing Video Cameras for 3D Tangible Tabletop Interaction. In: Proc. of TABLETOP 2007, pp. 201–204. IEEE, Los Alamitos (2007)Google Scholar
  37. 37.
    Wilson, A.D., Benko, H.: Combining multiple depth cameras and projectors for interactions on, above and between surfaces. In: Proc. of UIST 2010, pp. 273–282. ACM, New York (2010)Google Scholar
  38. 38.
    Wilson, A.D., Izadi, S., Hilliges, O., Garcia-Mendoza, A., Kirk, D.: Bringing physics to the surface. In: Proc. of UIST 2008, pp. 67–76. ACM, New York (2008)CrossRefGoogle Scholar
  39. 39.
    Wilson, A.D.: TouchLight: an imaging touch screen and display for gesture-based interaction. In: Proc. of ICMI, pp. 69–76. ACM Press, New York (2004)CrossRefGoogle Scholar
  40. 40.
    Wu, M., Balakrishnan, R.: Multi-finger and whole hand gestural interaction techniques for multi-user tabletop displays. In: Proc. of UIST 2003, pp. 193–202. ACM, New York (2003)CrossRefGoogle Scholar
  41. 41.
    Yilmaz, A., Javed, O., Shah, M.: Object tracking: A survey. ACM Computing Surveys (CSUR) 38 (2006)Google Scholar

Copyright information

© IFIP International Federation for Information Processing 2011

Authors and Affiliations

  • Nicolai Marquardt
    • 1
  • Ricardo Jota
    • 2
  • Saul Greenberg
    • 1
  • Joaquim A. Jorge
    • 2
  1. 1.Dept. Computer ScienceUniversity of CalgaryCalgaryCanada
  2. 2.VIMMI Group / INESC-ID IST Technical University of LisbonPortugal

Personalised recommendations