Image-Based Wearable Tangible Interface

  • Jiung-Yao Huang
  • Yong-Zeng Yeo
  • Lin Huei
  • Chung-Hsien Tsai
Part of the Smart Innovation, Systems and Technologies book series (SIST, volume 21)


This paper presents a novel technique for the mobile TUI system which consists of a pico projector and a camera only. The proposed system can transform an arbitrary flat surface into a touch panel. It allows user to interact with a computer by hand gestures on any flat surface in anytime and anyplace. The contributions of the proposed system include the extraction of display screen from the live captured image by a RGB-based camera, shadow-based fingertips detection approach and a fast yet reliable FSM grammar to determine user’s finger gesture. At the end, a prototype system is built to validate the proposed techniques and an experiment is shown in the last.


Tangible User Interface Computer Vision Finger Tracking Camera Projector 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Jeffries, R., Miller, J.R., Wharton, C., Uyeda, K.: User interface evaluation in the real world: a comparison of four techniques. In: SIGCHI 1991, pp. 4–11 (1991)Google Scholar
  2. 2.
    Kay, A.: User Interface: a personal view, pp. 121–131 (1990)Google Scholar
  3. 3.
  4. 4.
    Ishii, H.: The tangible user interface and its evolution. Communications of the ACM - Organic User Interfaces 51(6), 32–36 (2008)Google Scholar
  5. 5.
    Licsar, A., Sziranyi, T.: Hand Gesture Recognition in Camera-Projector System. In: International Workshop on Human-Computer Interaction, pp. 83–93 (2004)Google Scholar
  6. 6.
    Mistry, P., Maes, P., Chang, L.: WUW - Wear Ur World - A Wearable Gestural Interface. In: ACM CHI, pp. 4111–4116 (2009)Google Scholar
  7. 7.
    Wilson, A.D.: PlayAnywhere: a compact interactive tabletop projection-vision system. In: ACM UIST, pp. 83–92 (2005)Google Scholar
  8. 8.
    Yoshida, T., Hirobe, Y., Nii, H., Kawakami, N., Tachi, S.: Twinkle: Interacting with Physical Surfaces Using Handheld Projector. In: IEEE VR, pp. 87–90 (2010)Google Scholar
  9. 9.
    Gonzalez, R.C., Woods, R.E.: Digital Image Processing, 2nd edn. (2002)Google Scholar
  10. 10.
    Harrison, C., Benko, H., Wilson, A.D.: OmniTouch: Wearable Multitouch Interaction Everywhere. In: ACM UIST, pp. 441–450 (2011)Google Scholar
  11. 11.
    Zhang, Z.: A Flexible New Technique for Camera Calibration. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1330–1334 (2000)Google Scholar
  12. 12.
    Falcao, G., Hurtos, N., Massich, J., Fofi, D.: Projector-Camera Calibration Toolbox (2009),
  13. 13.
    Mitra, S., Acharya, T.: Gesture Recognition: A Survey. IEEE Transactions on Systems, Man, and Cybernetics, 311–324 (2007)Google Scholar
  14. 14.
    OS X Lion, OS X Lion: About Multi-Touch gestures,

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  • Jiung-Yao Huang
    • 1
  • Yong-Zeng Yeo
    • 1
  • Lin Huei
    • 2
  • Chung-Hsien Tsai
    • 3
  1. 1.Department of Computer Science and Information EngineeringNTPUTaipeiTaiwan
  2. 2.Department of Computer Science and Information EngineeringTKUTaipeiTaiwan
  3. 3.Department of Computer Science & Information EngineeringNDUHsinchuTaiwan

Personalised recommendations