INTAIRACT: Joint Hand Gesture and Fingertip Classification for Touchless Interaction

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7585)


In this demo we present intAIRact, an online hand-based touchless interaction system. Interactions are based on easy-to-learn hand gestures, that combined with translations and rotations render a user friendly and highly configurable system. The main advantage with respect to existing approaches is that we are able to robustly locate and identify fingertips. Hence, we are able to employ a simple but powerful alphabet of gestures not only by determining the number of visible fingers in a gesture, but also which fingers are being observed. To achieve such a system we propose a novel method that jointly infers hand gestures and fingertip locations using a single depth image from a consumer depth camera. Our approach is based on a novel descriptor for depth data, the Oriented Radial Distribution (ORD) [1]. On the one hand, we exploit the ORD for robust classification of hand gestures by means of efficient k-NN retrieval. On the other hand, maxima of the ORD are used to perform structured inference of fingertip locations. The proposed method outperforms other state-of-the-art approaches both in gesture recognition and fingertip localization. An implementation of the ORD extraction on a GPU yields a real-time demo running at approximately 17fps on a single laptop.


Random Forest Gesture Recognition Hand Gesture Depth Data Structure Inference 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    Suau, X., Ruiz-Hidalgo, J., Casas, J.R.: Oriented Radial Distribution on Depth Data: Appication to the Detection of End-Effectors. In: ICASSP (2012)Google Scholar
  2. 2.
    Apple Inc.: Magic Trackpad (2012)Google Scholar
  3. 3.
    Suau, X., Ruiz-Hidalgo, J., Casas, J.R.: Real-Time Head and Hand Tracking based on 2.5D data. Transactions on Multimedia, 1 (2012)Google Scholar
  4. 4.
    Keskin, C., Kırac, F., Kara, Y.E., Akarun, L.: Real Time Hand Pose Estimation using Depth Sensors. In: ICCV-CDC4CV, pp. 1228–1234 (2011)Google Scholar
  5. 5.
    Minnen, D., Zafrulla, Z.: Towards robust cross-user hand tracking and shape recognition. In: ICCV-CDC4CV, Oblong Industries, Los Angeles, CA, USA (2011)Google Scholar
  6. 6.
    Frome, A., Huber, D., Kolluri, R., Bülow, T., Malik, J.: Recognizing Objects in Range Data Using Regional Point Descriptors. In: Pajdla, T., Matas, J. (eds.) ECCV 2004, Part III. LNCS, vol. 3023, pp. 224–237. Springer, Heidelberg (2004)CrossRefGoogle Scholar
  7. 7.
    Rusu, R., Bradski, G., Thibaux, R., Hsu, J.: Fast 3D recognition and pose using the viewpoint feature histogram. In: IROS, pp. 2155–2162 (2010)Google Scholar
  8. 8.
    Tombari, F., Salti, S., Di Stefano, L.: Unique Signatures of Histograms for Local Surface Description. In: Daniilidis, K., Maragos, P., Paragios, N. (eds.) ECCV 2010, Part III. LNCS, vol. 6313, pp. 356–369. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  9. 9.
    Shotton, J., Fitzgibbon, A., Cook, M., Sharp, T., Finocchio, M., Moore, R., Kipman, A., Blake, A.: Real-Time Human Pose Recognition in Parts from Single Depth Images. In: CVPR, pp. 1297–1304 (2011)Google Scholar
  10. 10.
    Uebersax, D., Gall, J., Van den Bergh, M., Van Gool, L.: Real-time Sign Language Letter and Word Recognition from Depth Data. In: ICCV-HCI, pp. 1–8 (2011)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  1. 1.Universitat Politècnica de CatalunyaSpain

Personalised recommendations