Go with the Flow: Hand Trajectories in 3D via Clustered Scene Flow
Tracking hands and estimating their trajectories is useful in a number of tasks, including sign language recognition and human computer interaction. Hands are extremely difficult objects to track, their deformability, frequent self occlusions and motion blur cause appearance variations too great for most standard object trackers to deal with robustly.
In this paper, the 3D motion field of a scene (known as the Scene Flow, in contrast to Optical Flow, which is it’s projection onto the image plane) is estimated using a recently proposed algorithm, inspired by particle filtering. Unlike previous techniques, this scene flow algorithm does not introduce blurring across discontinuities, making it far more suitable for object segmentation and tracking. Additionally the algorithm operates several orders of magnitude faster than previous scene flow estimation systems, enabling the use of Scene Flow in real-time, and near real-time applications.
A novel approach to trajectory estimation is then introduced, based on clustering the estimated scene flow field in both space and velocity dimensions. This allows estimation of object motions in the true 3D scene, rather than the traditional approach of estimating 2D image plane motions. By working in the scene space rather than the image plane, the constant velocity assumption, commonly used in the prediction stage of trackers, is far more valid, and the resulting motion estimate is richer, providing information on out of plane motions. To evaluate the performance of the system, 3D trajectories are estimated on a multi-view sign-language dataset, and compared to a traditional high accuracy 2D system, with excellent results.
KeywordsOptical Flow Sign Language Hand Trajectory Hand Tracking Trajectory Estimation
Unable to display preview. Download preview PDF.
- 1.Awad, G., Han, J., Sutherland, A.: A unified system for segmentation and tracking of face and hands in sign language recognition. In: ICPR, Hong Kong, China (August 2006)Google Scholar
- 2.Basha, T., Moses, Y., Kiryati, N.: Multi-view scene flow estimation: A view centered variational approach. In: CVPR (2010)Google Scholar
- 4.Buehler, P., Zisserman, A., Everingham, M.: Learning sign language by watching tv (using weakly aligned subtitles). In: CVPR, Miami, FL, USA, June 20-26 (2009)Google Scholar
- 6.Furukawa, Y., Ponce, J.: Dense 3d motion capture from synchronized video streams. In: CVPR (2008)Google Scholar
- 7.Hadfield, S., Bowden, R.: Kinecting the dots: Particle based scene flow from depth sensors. In: ICCV, Barcelona, Spain, November 6-13 (2011)Google Scholar
- 8.Han, J., Awad, G., Sutherland, A.: Automatic skin segmentation and tracking in sign language recognition. IET Computer Vision (2009)Google Scholar
- 9.Huguet, F., Devernay, F.: A variational method for scene flow estimation from stereo sequences. In: ICCV (2007)Google Scholar
- 10.Imagawa, K., Lu, S., Igi, S.: Color-based hands tracking system for sign language recognition. In: FGR, Nara, Japan, April 14-16 (1998)Google Scholar
- 11.Kadir, T., Bowden, R., Ong, E., Zisserman, A.: Minimal training, large lexicon, unconstrained sign language recognition. In: BMVC (2004)Google Scholar
- 12.Kim, J.H., Thang, N.D., Kim, T.S.: 3-d hand motion tracking and gesture recognition using a data glove. In: ISIE (2009)Google Scholar
- 13.Li, R., Sclaroff, S.: Multi-scale 3d scene flow from binocular stereo sequences. In: Workshop on Motion and Video Computing (2005)Google Scholar
- 14.Neumann, J., Aloimonos, Y.: Spatio-temporal stereo using multi-resolution subdivision surfaces. IJCV (2002)Google Scholar
- 15.Pitsikalis, V., Theodorakis, S., Vogler, C., Athena, R., Maragos, P.: Advances in phonetics-based sub-unit modeling for transcription alignment and sign language recognition. In: Workshop on Gesture Recognition (2011)Google Scholar
- 16.Pons, J., Keriven, R., Faugeras, O.: Multiview stereo reconstruction and scene flow estimation with a global image-based match score. IJCV (2007)Google Scholar
- 18.Scharstein, D., Szeliski, R.: High-accuracy stereo depth maps using structured light. In: CVPR (2003)Google Scholar
- 19.Sidenbladh, H.: Probabilistic Tracking and Reconstruction of 3D Human Motion in Monocular Video Sequence. Ph.D. thesis, Stockholm Royal Institute of Technology (2001)Google Scholar