INTAIRACT: Joint Hand Gesture and Fingertip Classification for Touchless Interaction
In this demo we present intAIRact, an online hand-based touchless interaction system. Interactions are based on easy-to-learn hand gestures, that combined with translations and rotations render a user friendly and highly configurable system. The main advantage with respect to existing approaches is that we are able to robustly locate and identify fingertips. Hence, we are able to employ a simple but powerful alphabet of gestures not only by determining the number of visible fingers in a gesture, but also which fingers are being observed. To achieve such a system we propose a novel method that jointly infers hand gestures and fingertip locations using a single depth image from a consumer depth camera. Our approach is based on a novel descriptor for depth data, the Oriented Radial Distribution (ORD) . On the one hand, we exploit the ORD for robust classification of hand gestures by means of efficient k-NN retrieval. On the other hand, maxima of the ORD are used to perform structured inference of fingertip locations. The proposed method outperforms other state-of-the-art approaches both in gesture recognition and fingertip localization. An implementation of the ORD extraction on a GPU yields a real-time demo running at approximately 17fps on a single laptop.
KeywordsRandom Forest Gesture Recognition Hand Gesture Depth Data Structure Inference
Unable to display preview. Download preview PDF.
- 1.Suau, X., Ruiz-Hidalgo, J., Casas, J.R.: Oriented Radial Distribution on Depth Data: Appication to the Detection of End-Effectors. In: ICASSP (2012)Google Scholar
- 2.Apple Inc.: Magic Trackpad (2012)Google Scholar
- 3.Suau, X., Ruiz-Hidalgo, J., Casas, J.R.: Real-Time Head and Hand Tracking based on 2.5D data. Transactions on Multimedia, 1 (2012)Google Scholar
- 4.Keskin, C., Kırac, F., Kara, Y.E., Akarun, L.: Real Time Hand Pose Estimation using Depth Sensors. In: ICCV-CDC4CV, pp. 1228–1234 (2011)Google Scholar
- 5.Minnen, D., Zafrulla, Z.: Towards robust cross-user hand tracking and shape recognition. In: ICCV-CDC4CV, Oblong Industries, Los Angeles, CA, USA (2011)Google Scholar
- 7.Rusu, R., Bradski, G., Thibaux, R., Hsu, J.: Fast 3D recognition and pose using the viewpoint feature histogram. In: IROS, pp. 2155–2162 (2010)Google Scholar
- 9.Shotton, J., Fitzgibbon, A., Cook, M., Sharp, T., Finocchio, M., Moore, R., Kipman, A., Blake, A.: Real-Time Human Pose Recognition in Parts from Single Depth Images. In: CVPR, pp. 1297–1304 (2011)Google Scholar
- 10.Uebersax, D., Gall, J., Van den Bergh, M., Van Gool, L.: Real-time Sign Language Letter and Word Recognition from Depth Data. In: ICCV-HCI, pp. 1–8 (2011)Google Scholar