Facilitated Gesture Recognition Based Interfaces for People with Upper Extremity Physical Impairments

  • Hairong Jiang
  • Juan P. Wachs
  • Bradley S. Duerstock
Part of the Lecture Notes in Computer Science book series (LNCS, volume 7441)


A gesture recognition based interface was developed to facilitate people with upper extremity physical impairments as an alternative way to perform laboratory experiments that require ‘physical’ manipulation of components. A color, depth and spatial information based particle filter framework was constructed with unique descriptive features for face and hands representation. The same feature encoding policy was subsequently used to detect, track and recognize users’ hands. Motion models were created employing dynamic time warping (DTW) method for better observation encoding. Finally, the hand trajectories were classified into different classes (commands) by applying the CONDENSATION method and, in turn, an interface was designed for robot control, with a recognition accuracy of 97.5%. To assess the gesture recognition and control policies, a validation experiment consisting in controlling a mobile service robot and a robotic arm in a laboratory environment was conducted.


Gesture recognition particle filter dynamic time warping (DTW) CONDENSATION 


  1. 1.
    Jacko, J.A.: Human-Computer Interaction Design and Development Approaches. In: 14th HCI International Conference, pp. 169–180 (2011)Google Scholar
  2. 2.
    Moon, I., Lee, M., Ryu, J., Mun, M.: Intelligent Robotic Wheelchair with EMG-, Gesture-, and Voice-based Interfaces. In: International Conference on Intelligent Robots and Systems, pp. 3453–3458. IEEE Press (2003)Google Scholar
  3. 3.
    Reale, M., Liu, P.: Yin. L.J.: Using eye gaze, head pose and facial expression for personalized non-player character interaction. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, pp. 13–18. IEEE Press (2011)Google Scholar
  4. 4.
    Soriano, M., Martinkauppi, B., Huovinen, S., Laaksonen, M.: Skin detection in video under changing illumination conditions. In: 15th International Conference on Pattern Recognition, vol. 1, pp. 839–842 (2000)Google Scholar
  5. 5.
    Bradski, G.R.: Computer vision face tracking as a component of a perceptual user interface. In: Workshop on Applications of Computer Vision, Princeton, NJ, pp. 214–219 (1998)Google Scholar
  6. 6.
    Isard, M., Black, A.: CONDENSATION: Conditional density propagation for visual tracking. International Journal of Computer Vision 29, 5–28 (1998)CrossRefGoogle Scholar
  7. 7.
    Pérez, P., Hue, C., Vermaak, J., Gangnet, M.: Color-Based Probabilistic Tracking. In: Heyden, A., Sparr, G., Nielsen, M., Johansen, P. (eds.) ECCV 2002, Part I. LNCS, vol. 2350, pp. 661–675. Springer, Heidelberg (2002)CrossRefGoogle Scholar
  8. 8.
    Bilal, S., Akmeliawati, R., Shafie, A.A., Salami, M.J.E.: Hidden Markov Model for human to computer interaction: a study on human hand gesture recognition. In: Artificial Intelligence (2011)Google Scholar
  9. 9.
    Black, M.J., Jepson, A.D.: A Probabilistic Framework for Matching Temporal Trajectories: CONDENSATION-Based Recognition of Gestures and Expressions. In: Burkhardt, H.-J., Neumann, B. (eds.) ECCV 1998, Part I. LNCS, vol. 1406, pp. 909–924. Springer, Heidelberg (1998)Google Scholar
  10. 10.
    Viola, P., Jones, M.: Rapid object detection using a boosted cascade of simple features. In: International Conference on Computer Vision and Pattern Recognition, pp. 511–518 (2001)Google Scholar
  11. 11.
    Jones, M.J., Rehg, J.M.: Statistical color models with application to skin detection. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol. 46, pp. 81–96 (2002)Google Scholar
  12. 12.
    Hess, R., Fern, A.: Discriminatively Trained Particle Filters for Complex Multi-Object Tracking. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 240–247 (2009)Google Scholar
  13. 13.
    Aach, J., Church, G.M.: Alignment gene expression time series with time warping algorithms. J. Bioinformatics 17(6), 495–508 (2001)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2012

Authors and Affiliations

  • Hairong Jiang
    • 1
  • Juan P. Wachs
    • 1
  • Bradley S. Duerstock
    • 1
    • 2
  1. 1.School of Industrial EngineeringPurdue UniversityWest LafayetteUSA
  2. 2.Weldon School of Biomedical EngineeringPurdue UniversityWest LafayetteUSA

Personalised recommendations