Personal and Ubiquitous Computing

, Volume 18, Issue 5, pp 1213–1225 | Cite as

Tangible displays for the masses: spatial interaction with handheld displays by using consumer depth cameras

  • Martin Spindler
  • Wolfgang Büschel
  • Charlotte Winkler
  • Raimund Dachselt
Original Article

Abstract

Spatially aware handheld displays are a promising approach to interact with complex information spaces in a more natural way by extending the interaction space from the 2D surface to the 3D physical space around them. This is achieved by utilizing their spatial position and orientation for interaction purposes. Technical solutions for spatially tracked displays already exist in research laboratories, e.g., embedded in a tabletop environment. Along with a large stationary screen, such multi-display systems provide a rich design space with a variety of benefits to users, e.g., the explicit support of co-located parallel work and collaboration. As we see a great future in the underlying interaction principles, the question is how the technology can be made accessible to the public. With our work, we want to address this issue. In the long term, we envision a low-cost tangible display ecosystem that is suitable for everyday usage and supports both active displays (e.g., the iPad) and passive projection media (e.g., paper screens and everyday objects such as a mug). The two major contributions of this article are a presentation of an exciting design space and a requirement analysis regarding its technical realization with special focus on a broad adoption by the public. In addition, we present a proof of concept system that addresses one technical aspect of this ecosystem: the spatial tracking of tangible displays with a consumer depth camera (Kinect).

Keywords

Tangible magic lenses PaperLens Tangible display ecosystem Spatial interaction Multi-display environment Interaction above the tabletop Kinect 

References

  1. 1.
    OpenKinect—imaging information. http://openkinect.org/wiki/Imaging_Information
  2. 2.
    Akaoka E, Ginn T, Vertegaal R (2010) DisplayObjects: prototyping functional physical interfaces on 3d styrofoam, paper or cardboard models. In: Proceedings of the fourth international conference on Tangible, embedded, and embodied interaction (TEI), ACM, pp 49–56Google Scholar
  3. 3.
    Bandyopadhyay D, Raskar R, Fuchs H (2001) Dynamic shader lamps: painting on movable objects. In: Proceedings of the International Symposium on augmented reality (ISAR), IEEE, pp 207–216Google Scholar
  4. 4.
    Bier EA, Stone MC, Pier K, Buxton W, DeRose TD (1993) Toolglass and magic lenses: the see-through interface. In: Proceedings of SIGGRAPH, ACM, pp 445–446Google Scholar
  5. 5.
    Butler DA, Izadi S, Hilliges O, Molyneaux D, Hodges S, Kim D (2012) Shake’n’sense: reducing interference for overlapping structured light depth cameras. In: Proceedings of the 2012 ACM annual conference on human factors in computing systems (CHI), ACM, pp 1933–1936Google Scholar
  6. 6.
    Clark A, Piumsomboon T (2011) A realistic augmented reality racing game using a depth-sensing camera. In: Proceedings of the International Conference on virtual reality continuum and its applications in industry (VRCAI), ACM, pp 499–502Google Scholar
  7. 7.
    Costanza E, Giaccone M, Kueng O, Shelley S, Huang J (2010) Ubicomp to the masses: a large-scale study of two tangible interfaces for download. In: Proceedings of the ACM international conference on Ubiquitous computing (Ubicomp), ACM (2010), pp 173–182Google Scholar
  8. 8.
    Fischler MA, Bolles RC (1981) Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Comm ACM 24:381–395CrossRefMathSciNetGoogle Scholar
  9. 9.
    Fitzmaurice GW (1993) Situated information spaces and spatially aware palmtop computers. Comm ACM Special Issue Comput Augment Environ Back Real World 36(7):39–49Google Scholar
  10. 10.
    Haller M, Leitner J, Seifried T, Wallace JR, Scott SD, Richter C, Brandl P, Gokcezade A, Hunter S (2010) The NiCE discussion room: integrating paper and digital media to support co-located group meetings. In: Proceedings of the International ACM Conference on human factors in computing systems (CHI), ACM, pp 609–618Google Scholar
  11. 11.
    Han JY (2005) Low-cost multi-touch sensing through frustrated total internal reflection. In: Proceedings of the 18th annual ACM symposium on user interface software and technology (UIST), ACM, pp 115–118Google Scholar
  12. 12.
    Harrison C, Dey AK (2008) Lean and zoom: proximity-aware user interface and content magnification. In: Proceedings of the International ACM Conference on Human Factors in Computing Systems (CHI), ACM, pp 507–510Google Scholar
  13. 13.
    Harrison C, Benko H, Wilson AD (2011) OmniTouch: wearable multitouch interaction everywhere. In: Proceedings of the International ACM Symposium on user interface software and technology (UIST), ACM, pp 441–450Google Scholar
  14. 14.
    Holman D, Vertegaal R, Altosaar M, Troje N, Johns D (2005) Paper windows: interaction techniques for digital paper. In: Proceedings of the International ACM conference on human factors in computing systems (CHI), ACM, pp 591–599Google Scholar
  15. 15.
    Huber J, Steimle J, Liao C, Liu Q, Mühlhäuser M (2012) LightBeam: interacting with augmented real-world objects in pico projections. In: Proceedings of the International Conference on mobile and ubiquitous multimedia (MUM). ACM, Article 16Google Scholar
  16. 16.
    Ishii H, Ullmer B (1997) Tangible bits. Towards seamless interfaces between people, bits and atoms. In: Proceedings of the International ACM Conference on human factors in computing systems (CHI), ACM, pp 234–241Google Scholar
  17. 17.
    Izadi S, Hodges S, Taylor S, Rosenfeld D, Villar N, Butler A, Westhues J (2008) Going beyond the display: a surface technology with an electronically switchable diffuser. In: Proceedings of the 21st annual ACM symposium on user interface software and technology (UIST), ACM, pp 269–278Google Scholar
  18. 18.
    Junuzovic S, Inkpen Quinn K, Blank T, Gupta A (2012) IllumiShare: sharing any surface. In: Proceedings of the International ACM Conference on human factors in computing systems (CHI), ACM, pp 1919–1928Google Scholar
  19. 19.
    Khalilbeigi M, Lissermann R, Mühlhäuser M, Steimle J (2011) Xpaaand: interaction techniques for rollable displays. In: Proceedings of the International ACM Conference on human factors in computing systems (CHI), ACM, pp 2729–2732Google Scholar
  20. 20.
    Khalilbeigi M, Lissermann R, Kleine W, Steimle J (2012) FoldMe: interacting with double-sided foldable displays. In: Proceedings of the Sixth International Conference on tangible, embedded and embodied interaction (TEI), ACM, pp 33–40Google Scholar
  21. 21.
    Kirsh D (1995) The intelligent use of space. Artificial intelligence—special volume on computational research on interaction and agency, part 2, 73(1–2):31–68Google Scholar
  22. 22.
    Kray C, Rohs M, Hook J, Kratz S (2008) Group coordination and negotiation through spatial proximity regions around mobile devices on augmented tabletops. In: Proceedings of the 3rd International Workshop on horizontal interactive human computer systems (TABLETOP), IEEE, pp 1–8Google Scholar
  23. 23.
    Lee JC, Hudson SE, Summet JW, Dietz PH (2005) Moveable interactive projected displays using projector based tracking. In: Proceedings of the International ACM Symposium on user interface software and technology (UIST), ACM, pp 63–72Google Scholar
  24. 24.
    Lee JC, Hudson SE, Tse E (2008) Foldable interactive displays. In: Proceedings of the ACM Symposium on user interface software and technology (UIST), ACM, pp 287–290Google Scholar
  25. 25.
    Linder N, Maes P (2010) LuminAR: portable robotic augmented reality interface design and prototype. In: Adjunct Proceedings of the International ACM Symposium on user interface software and technology (UIST), ACM, pp 395–396Google Scholar
  26. 26.
    Lissermann R, Olberding S, Petry B, Mühlhäuser M, Steimle J (2012) PaperVideo: interacting with videos on multiple paper-like displays. In: Proceedings of the ACM International Conference on Multimedia (MM), ACM, pp 129–138Google Scholar
  27. 27.
    Merrill D, Kalanithi J, Maes P (2007) Siftables: towards sensor network user interfaces. In: Proceedings of the 1st international conference on tangible and embedded interaction (TEI), ACM, pp 75–78Google Scholar
  28. 28.
    Newcombe RA, Izadi S, Hilliges O, Molyneaux D, Kim D, Davison AJ, Kohli P, Shotton J, Hodges S, Fitzgibbon A (2011) KinectFusion: real-time dense surface mapping and tracking. In: Proceedings of the International Symposium on mixed and augmented reality (ISMAR), IEEE, pp 127–136Google Scholar
  29. 29.
    Rekimoto J (1998) A multiple device approach for supporting whiteboard-based interactions. In: Proceedings of the International ACM Conference on human factors in computing systems (CHI), ACM, pp 344–351Google Scholar
  30. 30.
    Spindler M, Stellmach S, Dachselt R (2009). PaperLens: advanced magic lens interaction above the tabletop. In: Proceedings of the International ACM Conference on interactive tabletops and surfaces (ITS), ACM, pp 69–76Google Scholar
  31. 31.
    Spindler M, Dachselt R (2009) Towards pen-based annotation techniques for tangible magic lenses above a tabletop. In: Accompanying DVD of the ACM International Conference on interactive tabletops and surfaces (ITS), ACMGoogle Scholar
  32. 32.
    Spindler M, Tominski C, Schumann H, Dachselt R (2010) Tangible views for information visualization. In: Proceedings of the International ACM Conference on interactive tabletops and surfaces (ITS), ACM, pp 157–166Google Scholar
  33. 33.
    Spindler M, Martsch M, Dachselt R (2012) Going beyond the surface: studying multi-layer interaction above the tabletop. In: Proceedings of the International ACM Conference on human factors in computing systems (CHI), ACM, pp 1277–1286Google Scholar
  34. 34.
    Spindler M, Büschel W, Dachselt R (2012). Towards spatially aware tangible displays for the masses. In: Proceedings of Workshop on designing collaborative interactive spaces for e-Creativity, e-Science and e-Learning (DCIS) at AVIGoogle Scholar
  35. 35.
    Spindler M, Büschel W, Dachselt R (2012) Use your head: tangible windows for 3D information spaces in a tabletop environment. In: Proceedings of the International ACM Conference on interactive tabletops and surfaces (ITS), ACM, pp 245–254Google Scholar
  36. 36.
    Sellen AJ, Harper RHR (2001) The myth of the paperless office. The MIT Press, Cambridge, MAGoogle Scholar
  37. 37.
    Tarun AP, Wang P, Girouard A, Strohmeier P, Reilly D, Vertegaal R (2013) PaperTab: an electronic paper computer with multiple large flexible electrophoretic displays. In: CHI extended abstracts on human factors in computing systems (CHI EA), ACM, pp 3131–3134Google Scholar
  38. 38.
    Tsang M, Fitzmaurice GW, Kurtenbach G, Khan A, Buxton B (2002) Boom chameleon: simultaneous capture of 3D viewpoint, voice and gesture annotations on a spatially-aware display. In: Proceedings of the International ACM Symposium on user interface software and technology (UIST), ACM, pp 111–120Google Scholar
  39. 39.
    Ullmer B, Ishii H (1997) The metaDESK: models and prototypes for tangible user interfaces. In: Proceedings of the International ACM Symposium on user interface software and technology (UIST), ACM, pp 223–232Google Scholar
  40. 40.
    Wallace JR, Scott SD, Stutz T, Enns T, Inkpen K (2009) Investigating teamwork and taskwork in single- and multi-display groupware systems. Pers Ubiquitous Comput 13(8):569–581CrossRefGoogle Scholar
  41. 41.
    Weiser M (1991) The computer for the 21st century. Sci Am 265(3):66–75CrossRefGoogle Scholar
  42. 42.
    Wilson D (2010) Using a depth camera as a touch sensor. In: Proceedings of the International ACM Conference on interactive tabletops and surfaces (ITS), ACM, pp 69–72Google Scholar
  43. 43.
    Yee K (2003) Peephole displays: pen interaction on spatially aware handheld computers. In: Proceedings of the International ACM Conference on human factors in computing systems (CHI), ACM, pp 1–8Google Scholar
  44. 44.
    Zhang Z, Wu Y, Shan Y, Shafer S (2001) Visual panel: virtual mouse, keyboard and 3D controller with an ordinary piece of paper. In: Proceedings of the Workshop on perceptive user interfaces (PUI), ACM, pp 1–8Google Scholar

Copyright information

© Springer-Verlag London 2013

Authors and Affiliations

  • Martin Spindler
    • 1
  • Wolfgang Büschel
    • 2
  • Charlotte Winkler
    • 1
  • Raimund Dachselt
    • 2
  1. 1.University of MagdeburgMagdeburgGermany
  2. 2.Technische Universität DresdenDresdenGermany

Personalised recommendations