Skip to main content
SpringerLink
Log in

Combining hidden Markov model and fuzzy neural network for continuous recognition of complex dynamic gestures

  • Original Article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

In the design of gesture-based user interfaces, continuously recognizing complex dynamic gestures is a challenging task, because of the high-dimensional information of gestures, ambiguous semantic meanings of gestures, and the presence of unpredictable non-gesture body motions. In this paper, we propose a hybrid model that can leverage the time-series modeling ability of hidden Markov model and the fuzzy inference ability of fuzzy neural network. First, a complex dynamic gesture is decomposed and fed into the hybrid model. The likelihood probability of an observation sequence estimated by the hidden Markov model is used as fuzzy membership degree of the corresponding fuzzy class variable in fuzzy neural network. Next, fuzzy rule modeling and fuzzy inference are performed by fuzzy neural network for gesture classification. To spot key gestures accurately, a threshold model is introduced to calculate the likelihood threshold of an input pattern and provide a reliability measure of whether to accept the pattern as a gesture. Finally, the proposed method is applied to recognize ten user-defined dynamic gestures for controlling interactive digital television in a smart room. Results of our experiment show that the proposed method performed better in terms of spotting reliability and recognition accuracy than conventional gesture recognition methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

References

  1. Feng, Z.Q., Zhang, M.M., Pan, Z.G., Yang, B., Xu, T., Tang, H.K., Li, Y.: 3D-freehand-pose initialization based on operator’s cognitive behavioral models. Vis. Comput. 26(6–8), 607–617 (2010)

    Article  Google Scholar 

  2. Kulshreshth, A., LaViola Jr., J.J.: Exploring the usefulness of finger-based 3D gesture menu selection. In: Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI’14), pp. 1093–1102 (2014)

  3. Pfeil, K.P., Koh, S.L., LaViola Jr. J.J.: Exploring 3D gesture metaphors for interaction with unmanned aerial vehicles. In: Proceedings of the 18th International Conference on Intelligent User Interfaces (IUI’13), pp. 257–266 (2013)

  4. LaViola, J.J.: An introduction to 3D gestural interfaces. In: Proceedings of SIGGRAPH’14, ACM SIGGRAPH 2014 Course, Vancouver, British Columbia, Canada. doi:10.1145/2614028.2615424

  5. Mitra, S., Acharya, T.: Gesture recognition: a survey. IEEE Trans. Syst. Man Cybern. Part C: Appl. Rev. 37(3), 311–324 (2007)

  6. Rautaray, S.S., Agrawal, A.: Vision based hand gesture recognition for human computer interaction: a survey. Artif. Intell. Rev. (2012). doi:10.1007/s10462-012-9356-9

  7. Kölsch, M., Turk, M., Höllerer, T.: Vision-based interfaces for mobility. In: Proceedings of IEEE International Conference on Mobile and Ubiquitous Systems (Mobiquitous’04), pp. 86–94 (2004)

  8. Ren, Z., Yuan, J.S., Meng, J.J., Zhang, Z.Y.: Robust part-based hand gesture recognition using Kinect sensor. IEEE Trans. Multimed. 15(5), 1110–1120 (2013)

    Article  Google Scholar 

  9. Stenger, B., Torr, P.H.S., Cipolla, R.: Model-based hand tracking using a hierarchical Bayesian filter. IEEE Trans. Pattern Anal. Mach. Intell. 28(9), 1372–1384 (2006)

    Article  Google Scholar 

  10. Stergiopoulou, E., Papamarkos, N.: Hand gesture recognition using a neural network shape fitting technique. Eng. Appl. Artif. Intell. 22(8), 1141–1158 (2009)

    Article  Google Scholar 

  11. Bobick, A.F., Davis, J.W.: The recognition of human movement using temporal templates. IEEE Trans. Pattern Anal. Mach. Intell. 23(3), 257–267 (2001)

    Article  Google Scholar 

  12. Kim, I.C., Chien, S.I.: Analysis of 3D hand trajectory gestures using stroke-based composite hidden Markov models. Appl. Intell. 15, 131–143 (2001)

    Article  MATH  Google Scholar 

  13. Lee, H., Kim, J.: An HMM-based threshold model approach for gesture recognition. IEEE Trans. Pattern Anal. Mach. Intell. 21(10), 961–973 (1999)

    Article  Google Scholar 

  14. Oz, C., Leu, M.C.: American sign language word recognition with a sensory glove using artificial neural networks. Eng. Appl. Artif. Intell. 24, 1204–1213 (2011)

    Article  Google Scholar 

  15. Seo, H.J., Milanfar, P.: Action recognition from one example. IEEE Trans. Pattern Anal. Mach. Intell. 33(5), 867–882 (2011)

    Article  Google Scholar 

  16. Sohn, M.K., Lee, S.H., Kim, D.J., Kim, B., Kim, H.: 3D hand gesture recognition from one example. In: IEEE International Conference on Consumer Electronics, pp. 171–172 (2013)

  17. Yang, H.D., Park, A.Y., Lee, S.W.: Gesture spotting and recognition for human–robot interaction. IEEE Trans. Robot. 23(2), 256–270 (2007)

    Article  Google Scholar 

  18. Jacob, R.J.K.: Eye Movement-Based Human–Computer Interaction Techniques: Toward Non-Command Interfaces. In: Advances in Human–Computer Interaction, vol. 4. Ablex Publishing Co., Norwood, pp. 151–190 (1993)

  19. Peng, B., Qian, G.: Online gesture spotting from visual hull data. IEEE Trans. Pattern Anal. Mach. Intell. 33(6), 1175–1188 (2011)

    Article  Google Scholar 

  20. Bilal, S., Akmeliawati, R., Shafie, A.A., Salami, M.J.E.: Hidden Markov model for human to computer interaction: a study on human hand gesture recognition. Artif. Intell. Rev. 40(4), 495–516 (2013)

    Article  Google Scholar 

  21. Wang, X.Y., Dai, G.Z. : A Novel Method to Recognize Complex Dynamic Gesture by Combing HMM and FNN Models. In: Proceedings of the 2007 IEEE Symposium on Computational Intelligence in Image and Signal Processing (CIIP’07), pp. 13–18 (2007)

  22. Pedersoli, F., Benini, S., Adami, N., Leonardi, R.: XKin: an open source framework for hand pose and gesture recognition using Kinect. Vis. Comput. 30(10), 1107–1122 (2014)

    Article  Google Scholar 

  23. Zhu, J.Y., Wang, X.Y., Wang, W.X., Dai, G.Z.: Hand gesture recognition based on structure analysis. Chin. J. Comput. 29(12), 2130–2137 (2006)

    Google Scholar 

  24. Wang, X.Y., Zhang, X.W., Dai, G.Z.: Continuous Input for Gesture-based Interaction. In: Proceedings of the 12th International Conference on Intelligent User Interfaces (IUI’07), pp. 235–242 (2007)

  25. Coupe, S.: Machine learning of projected 3D shape. Ph.D. Thesis, University of Manchester, Manchester (2009)

  26. Linde, Y., Buzo, A., Gray, R.M.: An algorithm for vector quantizer design. IEEE Trans. Commun. 28(1), 84–95 (1980)

    Article  Google Scholar 

  27. Rabiner, L.R.: A tutorial on hidden Markov models and selected applications in speech recognition. Proc. IEEE 77(2), 257–286 (1989)

    Article  Google Scholar 

  28. Viterbi, A.J.: Error bounds for convolution codes and an asymptotically optimum decoding algorithm. IEEE Trans. Inf. Theory 13, 260–269 (1967)

    Article  MATH  Google Scholar 

  29. Yang, H.D., Sclaroff, S., Lee, S.W.: Sign language spotting with a threshold model based on conditional random fields. IEEE Trans. Pattern Anal. Mach. Intell. 31(7), 1264–1277 (2009)

    Article  Google Scholar 

  30. Wu, H.Y., Wang, J.M., Zhang, X.L.: User-centered gesture development in TV viewing environment. Multimed. Tools Appl. (2014). doi:10.1007/s11042-014-2323-5

  31. Colaco, A., Kirmani, A., Yang, H.S., Gong, N.W., Schmandt, C., Goyal, V.K.: Mime: compact, low-power 3D gesture sensing for interaction with head-mounted displays. In: Proceedings of the ACM Symposium of User Interface Software and Technology (UIST’ 13), pp. 227–236 (2013)

  32. Mo, Z.Y., Lewis, J.P., Neumann, U.: SmartCanvas: a gesture-driven intelligent drawing desk. In: Proceedings of the ACM Symposium of User Interface Software and Technology (UIST’05), pp. 239-243 (2005)

  33. Wilson, A.D.: Robust computer vision-based detection of pinching for one and two-handed gesture input. In: Proceedings of the ACM Symposium of User Interface Software and Technology (UIST’ 06), pp. 255–258 (2006)

  34. Hilliges, O., Izadi, S., Wilson, A.D., Hodges, S., Mendoza, A.G., Butz, A.: Interactions in the air: adding further depth to interactive tabletops. In: Proceedings of the ACM Symposium of User Interface Software and Technology (UIST’ 09). pp. 139–148 (2009)

  35. Song, P., Goh, W.B., Hutama, W., Fu, C.W., Liu, X.P.: A handle bar metaphor for virtual object manipulation with mid-air interaction. In: Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI’12), pp. 1297–1306 (2012)

  36. Wobbrock, J.O., Morris, M.R., Wilson, A.D.: User-defined gestures for surface computing. In: Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI’ 09), pp. 1083–1092 (2009)

Download references

Acknowledgments

We thank the financial support from the National Natural Science Foundation of China, No. 61202344; the Fundamental Research Funds for the Central Universities, Sun Yat-Sen University, No. 1209119; Special Project on the Integration of Industry, Education and Research of Guangdong Province, No. 2012B091000062; the Fundamental Research Funds for the Central Universities, Tongji University, Nos. 0600219052, 0600219053. We would like to express our great appreciation to editor and reviewers.

Author information

Authors and Affiliations

  1. The School of Communication and Design, Sun Yat-Sen University, Guangzhou, China

    Huiyue Wu

  2. User Experience Lab, College of Arts and Media, Tongji University, Shanghai, China

    Jianmin Wang

  3. Pennsylvania State University, University Park, PA, USA

    Xiaolong Zhang

  4. Taiyuan University of Technology, Taiyuan, Shanxi, China

    Xiaolong Zhang

Authors
  1. Huiyue Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jianmin Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xiaolong Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Huiyue Wu or Xiaolong Zhang.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wu, H., Wang, J. & Zhang, X. Combining hidden Markov model and fuzzy neural network for continuous recognition of complex dynamic gestures. Vis Comput 33, 1265–1278 (2017). https://doi.org/10.1007/s00371-015-1147-2

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-015-1147-2

Keywords

Search

Navigation