Interactive Environment-Aware Handheld Projectors for Pervasive Computing Spaces

  • David Molyneaux
  • Shahram Izadi
  • David Kim
  • Otmar Hilliges
  • Steve Hodges
  • Xiang Cao
  • Alex Butler
  • Hans Gellersen
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7319)


This paper presents two novel handheld projector systems for indoor pervasive computing spaces. These projection-based devices are “aware” of their environment in ways not demonstrated previously. They offer both spatial awareness, where the system infers location and orientation of the device in 3D space, and geometry awareness, where the system constructs the 3D structure of the world around it, which can encompass the user as well as other physical objects, such as furniture and walls. Previous work in this area has predominantly focused on infrastructure-based spatial-aware handheld projection and interaction. Our prototypes offer greater levels of environment awareness, but achieve this using two opposing approaches; the first infrastructure-based and the other infrastructure-less sensing. We highlight a series of interactions including direct touch, as well as in-air gestures, which leverage the shadow of the user for interaction. We describe the technical challenges in realizing these novel systems; and compare them directly by quantifying their location tracking and input sensing capabilities.


Handheld projection geometry and spatial awareness interaction 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Beardsley, P., Baar, J.V., Raskar, R., Forlines, C.: Interaction using a handheld projector. IEEE Computer Graphics and Applications 25(1), 39–43 (2005)CrossRefGoogle Scholar
  2. 2.
    Brumitt, B., Meyers, B., Krumm, J., Kern, A., Shafer, S.: EasyLiving: Technologies for Intelligent Environments. In: Thomas, P., Gellersen, H.-W. (eds.) HUC 2000. LNCS, vol. 1927, pp. 12–27. Springer, Heidelberg (2000)CrossRefGoogle Scholar
  3. 3.
    Cao, X., Balakrishnan, R.: Interacting with dynamically defined information spaces using a handheld projector and pen. In: Proc. of ACM UIST 2006, pp. 225–234 (2006)Google Scholar
  4. 4.
    Cao, X., Forlines, C., Balakrishnan, R.: Multi-user interaction using handheld projectors. In: Proc. ACM UIST 2007, pp. 43–52 (2007)Google Scholar
  5. 5.
    Cowan, L., Li, K.: ShadowPuppets: collocated interaction with mobile projector phones using hand shadows. In: Proc: ACM CHI 2011, pp. 2707–2716 (2011)Google Scholar
  6. 6.
    Du, H., et al.: Interactive 3D modeling of indoor environments with a consumer depth camera. In: Proc. Ubiquitous Computing (UbiComp 2011), pp. 75–84. ACM (2011)Google Scholar
  7. 7.
    Ehnes, J., Hirota, K., Hirose, M.: Projected augmentation - AR using rotatable video projectors. In: Proc: 3rd IEEE and ACM ISMAR, pp. 26–35 (2004)Google Scholar
  8. 8.
    Harrison, C., Benko, H., Wilson, A.D.: OmniTouch: Wearable Multitouch Interaction Everywhere. In: Proc. ACM UIST 2011 (2011)Google Scholar
  9. 9.
    Hartley, R., Zisserman, A.: Multiple View Geometry in Computer Vision. Cambridge University Press (2003)Google Scholar
  10. 10.
    Izadi, S., Kim, D., Hilliges, O., Molyneaux, D., et al.: KinectFusion: RealTime Interactions with Dynamic 3D Surface Reconstructions. In: Proc. ACM UIST 2011 (2011)Google Scholar
  11. 11.
    Kazhdan, M., Bolitho, M., Hoppe, H.: Poisson surface reconstruction. In: Proc. Geometry Processing (SGP 2006), Switzerland, pp. 61–70 (2006)Google Scholar
  12. 12.
    Kidd, C.D., et al.: The Aware Home: A Living Laboratory for Ubiquitous Computing Research. In: Yuan, F., Hartkopf, V. (eds.) CoBuild 1999. LNCS, vol. 1670, pp. 191–198. Springer, Heidelberg (1999)CrossRefGoogle Scholar
  13. 13.
    Krueger, M.: Artificial Reality 2. Addison-Wesley Professional (1991)Google Scholar
  14. 14.
    Mistry, P., Maes, P., Chang, L.: WUW - Wear Ur World - A Wearable Gestural Interface. In: ACM CHI 2009 Extended Abstracts (2009)Google Scholar
  15. 15.
    Molyneaux, D., Gellersen, H., Kortuem, G., Schiele, B.: Cooperative Augmentation of Smart Objects with Projector-Camera Systems. In: Krumm, J., Abowd, G.D., Seneviratne, A., Strang, T. (eds.) UbiComp 2007. LNCS, vol. 4717, pp. 501–518. Springer, Heidelberg (2007)CrossRefGoogle Scholar
  16. 16.
    Naemura, T., Nitta, T., Mimura, A., Harashima, H.: Virtual Shadows- Enhanced Interaction in Mixed Reality Environment. In: IEEE VR 2002, p. 293 (2002)Google Scholar
  17. 17.
    Raskar, R., et al.: The office of the future: a unified approach to image-based modeling and spatially immersive displays. In: Proc. SIGGRAPH 1998, pp. 179–188 (1998)Google Scholar
  18. 18.
    Raskar, R., VanBaar, J., Beardsley, P., et al.: iLamps: geometrically aware and self-configuring projectors. ACM Trans. Graph. 22(3), 809–818 (2003)CrossRefGoogle Scholar
  19. 19.
    Raskar, R., et al.: RFIG Lamps: interacting with a self-describing world via photosensing wireless tags and projectors. ACM ToG 23(3), 406–415Google Scholar
  20. 20.
    Pinhanez, C.: The Everywhere Displays Projector: A Device to Create Ubiquitous Graphical Interfaces. In: Abowd, G.D., Brumitt, B., Shafer, S. (eds.) UbiComp 2001. LNCS, vol. 2201, pp. 315–331. Springer, Heidelberg (2001)CrossRefGoogle Scholar
  21. 21.
    Sato, Y., et al.: Fast tracking of hands and fingertips in infrared images for augmented desk interface. In: Proc: AFGR. IEEE (2000)Google Scholar
  22. 22.
    Shotton, J., et al.: Real-time human pose recognition in parts from single depth images. In: Proc. CVPR 2011, pp. 1297–1304 (2011)Google Scholar
  23. 23.
    Sugimoto, M., Miyahara, K., Inoue, H., Tsunesada, Y.: Hotaru: Intuitive Manipulation Techniques for Projected Displays of Mobile Devices. In: Costabile, M.F., Paternó, F. (eds.) INTERACT 2005. LNCS, vol. 3585, pp. 57–68. Springer, Heidelberg (2005)CrossRefGoogle Scholar
  24. 24.
    Teller, S., Chen, J., Balakrishnan, H.: Pervasive pose-aware applications and Infrastructure. IEEE CG&A 23(4), 14–18 (2003)Google Scholar
  25. 25.
    Weiser, M.: The Computer for the Twenty-First Century. Scientific American 265(3), 94–100 (1991)CrossRefGoogle Scholar
  26. 26.
    Willis, K.D.D., Poupyrev, I., Shiratori, T.: Motionbeam: Character interaction with handheld projectors. In: Proc: ACM CHI 2011, pp. 1031–1040 (2011)Google Scholar
  27. 27.
    Willis, K.D.D., Poupyrev, I., et al.: SideBySide: ad-hoc multi-user interaction with handheld projectors. In: Proc. ACM UIST 2011, pp. 431–440 (2011)Google Scholar
  28. 28.
    Wilson, A., Izadi, D., Hilliges, O., et al.: Bringing physics to the surface. In: Proc: ACM UIST 2008, pp. 67–76 (2008)Google Scholar
  29. 29.
    Wilson, A., Benko, H.: Combining multiple depth cameras and projectors for interactions on, above and between surfaces. In: Proc: ACM UIST 2010, pp. 273–282 (2010)Google Scholar
  30. 30.
    Xu, H., Iwai, D., Hiura, S., Sato, K.: User Interface by Virtual Shadow Projection. In: Proc: SICE-ICASE, pp. 4817–4818 (2006)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • David Molyneaux
    • 1
    • 2
  • Shahram Izadi
    • 1
  • David Kim
    • 1
    • 3
  • Otmar Hilliges
    • 1
  • Steve Hodges
    • 1
  • Xiang Cao
    • 1
  • Alex Butler
    • 1
  • Hans Gellersen
    • 2
  1. 1.Microsoft ResearchCambridgeUK
  2. 2.School of Computing and CommunicationsLancaster UniversityUK
  3. 3.School of Computer ScienceNewcastle UniversityUK

Personalised recommendations