Skip to main content
SpringerLink
Log in

Jointly network: a network based on CNN and RBM for gesture recognition

  • Machine Learning Applications for Self-Organized Wireless Networks
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

Hand belongs to non-rigid objects and is rich in variety, making gesture recognition more difficult. The essence of dynamic gesture recognition is the classification and recognition of single-frame still images. Therefore, this paper mainly focuses on static gesture recognition. At present, there are some problems in gesture recognition, such as accuracy, real-time or poor robustness. To solve the above problems, in this paper, the Kinect sensor is used to obtain the color and depth gesture samples, and the gesture samples are processed. On this basis, a jointly network of CNN and RBM is proposed for gesture recognition. It mainly uses superposed network of multiple RBMs to carry out unsupervised feature extraction and combined with supervised feature extraction of CNN. Finally, these two features are combined to classify them. The simulation results show that the proposed jointly network has a better performance in identifying simple background gesture samples and the recognition capability of gesture samples in complex background needs to be improved.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19
Fig. 20
Fig. 21

Similar content being viewed by others

References

  1. Traver VJ, Latorre-Carmon Luzanin AP, Salvador-Balaguer E, Filiberto P, Bahram J (2017) Three-dimensional integral imaging for gesture recognition under occlusions. IEEE Signal Process Lett 24(2):171–175

    Article  Google Scholar 

  2. Oyedotun OK, Khashman A (2017) Deep learning in vision-based static hand gesture recognition. Neural Comput Appl 28(12):3941–3951

    Article  Google Scholar 

  3. Nasri S, Behrad A, Razzazi F (2015) Spatio-temporal 3D surface matching for hand gesture recognition using ICP algorithm. SIViP 9(5):1205–1220

    Article  MATH  Google Scholar 

  4. Li GF, Tang H, Sun Y, Kong JY, Jiang GZ, Jiang D, Tao B, Xu S, Liu HH (2017) Hand gesture recognition based on convolution neural network. Cluster Comput. https://doi.org/10.1007/s10586-017-1435-x

    Article  Google Scholar 

  5. He Y, Li GF, Liao YJ, Sun Y, Kong JY, Jiang GZ, Jiang D, Tao B, Xu S, Liu HH (2017) Gesture recognition based on an improved local sparse representation classification algorithm. Cluster Comput. https://doi.org/10.1007/s10586-017-1237-1

    Article  Google Scholar 

  6. Ding WL, Li GF, Jiang GZ, Fang YF, Ju ZJ, Liu HH (2015) Intelligent computation in grasping control of dexterous robot hand. J Comput Theor Nanosci 12(12):6096–6099

    Article  Google Scholar 

  7. Li B, Sun Y, Li GF, Kong JY, Jiang GZ, Jiang D, Tao B, Xu S, Liu HH (2017) Gesture recognition based on modified adaptive orthogonal matching pursuit algorithm. Cluster Comput. https://doi.org/10.1007/s10586-017-1231-7

    Article  Google Scholar 

  8. Ordóñez FJ, Roggen D (2016) Deep convolutional and lstm recurrent neural networks for multimodal wearable activity recognition. Sensors 16(1):115

    Article  Google Scholar 

  9. Jiang D, Zheng ZJ, Li GF, Sun Y, Kong JY, Jiang GZ, Xiong HG, Tao B, Xu S, Yu H, Liu HH, Ju ZJ (2018) Gesture recognition based on binocular vision. Cluster Comput. https://doi.org/10.1007/s10586-018-1844-5

    Article  Google Scholar 

  10. Xiong HG, Fan HL, Li GF, Jiang GZ (2015) Research on steady-state simulation in dynamic job shop scheduling problem. Adv Mech Eng 7(9):1–11

    Article  Google Scholar 

  11. Barros P, Maciel-Junior NT, Fernandes BJ, Bezerra BL, Fernandes SM (2017) A dynamic gesture recognition and prediction system using the convexity approach. Comput Vis Image Underst 155:139–149

    Article  Google Scholar 

  12. Escalante HJ, Guyon I, Athitsos V, Jangyodsuk P, Wan J (2017) Principal motion components for one-shot gesture recognition. Pattern Anal Appl 20(1):167–182

    Article  MathSciNet  Google Scholar 

  13. Boughrara H, Chtourou M, Amar CB, Chen L (2016) Facial expression recognition based on a mlp neural network using constructive training algorithm. Multimed Tools Appl 75(2):709–731

    Article  Google Scholar 

  14. Li GF, Gu YS, Kong JY, Jiang GZ, Xie LX, Wu ZH, Li Z, He Y, Gao P (2013) Intelligent control of air compressor production process. Appl Math Inf Sci 7(3):1051–1058

    Article  Google Scholar 

  15. Li GF, Qu PX, Kong JY, Jiang GZ, Xie LX, Gao P, Wu ZH, He Y (2013) Coke oven intelligent integrated control system. Appl Math Inf Sci 7(3):1043–1050

    Article  Google Scholar 

  16. Rautaray SS, Agrawal A (2015) Vision based hand gesture recognition for human computer interaction: a survey. Artif Intell Rev 43(1):1–54

    Article  Google Scholar 

  17. Chakravarthi MK, Tiwari RK, Handa S (2015) Accelerometer based static gesture recognition and mobile monitoring system using neural networks. Proc Comput Sci 70:683–687

    Article  Google Scholar 

  18. Luzanin O, Plancak M (2014) Hand gesture recognition using low-budget data glove and cluster-trained probabilistic neural network. Assembly Autom 34(1):94–105

    Article  Google Scholar 

  19. Pisharady PK, Saerbeck M (2015) Recent methods and databases in vision-based hand gesture recognition: a review. Comput Vis Image Underst 141:152–165

    Article  Google Scholar 

  20. Kılıboz NÇ, Güdükbay U (2015) A hand gesture recognition technique for human–computer interaction. J Vis Commun Image Represent 28:97–104

    Article  Google Scholar 

  21. Hinton G, Sejnowski T (1983) Optimal perceptual inference. In IEEE conference on computer vision and pattern recognition

  22. Varghese B, Buyya R (2018) Next generation cloud computing: new trends and research directionsJ]. Future Gener Comput Syst 79:849–861

    Article  Google Scholar 

  23. Sun Y, Hu JB, Li GF, Jiang GZ, Xiong HG, Tao B, Zheng ZJ, Jiang D (2018) Gear reducer optimal design based on computer multimedia simulation. J Supercomput. https://doi.org/10.1007/s11227-018-2255-3

    Article  Google Scholar 

  24. Tieleman T (2008) Training restricted boltzmann machines using approximations to the likelihood gradient. In: International conference on machine learning (IC-ML) 2008

  25. Li GF, Liu Z, Jiang GZ, Xiong HG, Liu HH (2015) Numerical simulation of the influence factors for rotary kiln in temperature field and stress field and the structure optimization. Adv Mech Eng 7(6):1687814015589667

    Article  Google Scholar 

  26. Nguyen-Dinh LV, Calatroni A, Tröster G (2017) Supporting one-time point annotations for gesture recognition. IEEE Trans Pattern Anal Mach Intell 39(11):2270–2283

    Article  Google Scholar 

  27. Deng L, He XD, Gao JF (2013) Deep stacking network for information retrieval. In: 2013 IEEE international conference on acoustics, speech, and signal processing (ICASSP)

  28. Li Z, Li GF, Jiang GZ, Fang YF, Ju ZJ, Liu HH (2015) Intelligent computation of grasping and manipulation for multi-fingered robotic hands. J Comput Theor Nanosci 12(12):6192–6197

    Article  Google Scholar 

  29. Li GF, Liu J, Jiang GZ, Liu HH (2015) Numerical simulation of temperature field and thermal stress field in the new type of ladle with the nanometer adiabatic material. Adv Mech Eng 7(4):1687814015575988

    Google Scholar 

  30. Xiong HG, Fan HL, Jiang GZ, Li GF (2017) A simulation -based study of dispatching rules in a dynamic job shop scheduling problem with batch release and extended technical precedence constraints. Eur J Oper Res 257(1):13–24

    Article  MathSciNet  MATH  Google Scholar 

  31. Goh JEE, Goh MLI, Estrada JS, Lindog NC, Tabulog JCM, Talavera NEC (2017) Presentation-aid Armband with IMU, EMG sensor and bluetooth for free-hand writing and hand gesture recognition. Int J Comput Sci Res 1(3):54–66

    Google Scholar 

  32. Li GF, Qu PX, Kong JY, Jiang GZ, Xie LX, Wu ZH, Gao P, He Y (2013) Influence of working lining parameters on temperature and stress field of ladle. Appl Math Inf Sci 7(2):439–448

    Article  Google Scholar 

  33. Ohn-Bar E, Trivedi MM (2014) Hand gesture recognition in real time for automotive interfaces: a multimodal vision-based approach and evaluations. IEEE Trans Intell Transp Syst 15(6):2368–2377

    Article  Google Scholar 

  34. Chen DS, Li GF, Sun Y, Kong JY, Jiang GZ, Tang H, Ju ZJ, Yu H, Liu HH (2017) An interactive image segmentation method in hand gesture recognition. Sensors 17(2):253

    Article  Google Scholar 

  35. Liao YJ, Sun Y, Li GF, Kong JY, Jiang GZ, Jiang D, Cai HB, Ju ZJ, Yu H, Liu HH (2017) Simultaneous calibration: a jointly optimization approach for multiple kinect and external cameras. Sensors 17(7):1491

    Article  Google Scholar 

  36. Miao W, Li GF, Jiang GZ, Fang YF, Ju ZJ, Liu HH (2015) Optimal grasp planning of multi-fingered robotic hands: a review. Appl Comput Math 14(3):238–247

    MathSciNet  MATH  Google Scholar 

  37. Chen DS, Li GF, Sun Y, Jiang GZ, Kong JY, Liu HH (2017) Fusion hand gesture segmentation and extraction based on CMOS sensor and 3D sensor. Int J Wirel Mobile Comput 12(3):305–312

    Article  Google Scholar 

  38. Sun Y, Li CQ, Li GF, Jiang GZ, Jiang D, Liu HH, Zheng ZJ, Shu WN (2018) Gesture recognition based on Kinect and sEMG signal fusion. Mobile Netw Appl 23(4):797–805

    Article  Google Scholar 

  39. Fang YF, Liu HH, Li GF, Zhu XY (2015) A multichannel surface EMG system for hand motion recognition. Int J Humanoid Rob 12(2):1550011

    Article  Google Scholar 

  40. Li Z, Li GF, Sun Y, Jiang GZ, Kong JY, Liu HH (2017) Development of articulated robot trajectory planning. Int J Comput Sci Math 8(1):52–60

    Article  MathSciNet  Google Scholar 

  41. Miao W, Li GF, Sun Y, Jiang GZ, Kong JY, Liu HH (2016) Gesture recognition based on sparse representation. Int J Wirel Mobile Comput 11(4):348–356

    Article  Google Scholar 

  42. Yin Q, Li GF, Zhang JG (2015) Research on the method of step feature extraction for EOD robot based on 2d laser radar. Discrete Contin Dyn Syst-Ser S 8(6):1415–1421

    Article  MathSciNet  MATH  Google Scholar 

  43. Ding WL, Li GF, Sun Y, Jiang GZ, Kong JY, Liu HH (2017) D-S evidential theory on sEMG signal recognition. Int J Comput Sci Math 8(2):138–145

    Article  MathSciNet  Google Scholar 

  44. Jadooki S, Mohamad D, Saba T et al (2017) Fused features mining for depth-based hand gesture recognition to classify blind human communication. Neural Comput Appl 28(11):3285–3294

    Article  Google Scholar 

  45. Du F, Sun Y, Li GF, Li Z, Kong JY, Jiang GZ, Jiang D (2017) Adaptive fuzzy sliding mode control for 2-DOF articulated robot. J Wuhan Univ Sci Technol 40(6):446–450

    Google Scholar 

  46. Núñez JC, Cabido R, Pantrigo JJ, Montemayor AS, Vélez JF (2018) Convolutional neural networks and long short-term memory for skeleton-based human activity and hand gesture recognition. Pattern Recogn 76:80–94

    Article  Google Scholar 

  47. Li GF, Miao W, Jiang GZ, Fang YF, Ju ZJ, Liu HH (2015) Intelligent control model and its simulation of flue temperature in coke oven. Discrete Contin Dyn Syst Ser S (DCDS-S) 8(6):1223–1237

    Article  MathSciNet  MATH  Google Scholar 

  48. Poularakis S, Katsavounidis I (2016) Low-complexity hand gesture recognition system for continuous streams of digits and letters. IEEE Trans Cybern 46(9):2094–2108

    Article  Google Scholar 

  49. Chang Wenjun, Li Gongfa, Kong Jianyi, Sun Ying, Jiang Guozhang, Liu Honghai (2018) Thermal mechanical stress analysis of ladle lining with integral brick joint. Arch Metall Mater 63(2):659–666

    Google Scholar 

  50. Misra S, Singha J, Laskar RH (2017) Vision-based hand gesture recognition of alphabets, numbers, arithmetic operators and ASCII characters in order to develop a virtual text-entry interface system. Neural Comput Appl 29(8):117–135

    Article  Google Scholar 

  51. Li GF, Kong JY, Jiang GZ, Xie LX, Jiang ZG, Zhao G (2012) Air-fuel ratio intelligent control in coke oven combustion process. In Int Interdiscip J 15(11):4487–4494

    Google Scholar 

  52. Baraldi L, Paci F, Serra G, Benini L, Cucchiara R (2015) Gesture recognition using wearable vision sensors to enhance visitors’ museum experiences. IEEE Sens J 15(5):2705–2714

    Google Scholar 

  53. Li Gongfa, Zhang Leilei, Sun Ying, Kong Jianyi (2018) Internet of things sensors and haptic feedback for sEMG based hands. Multimed Tools Appl. https://doi.org/10.1007/s11042-018-6293-x

    Article  Google Scholar 

  54. Gravina R, Ma C, Pace P, Aloi G, Russo W, Li W, Fortino G (2017) Cloud-based activity-aaService cyber–physical framework for human activity monitoring in mobility. Future Gener Comput Syst 75:158–171

    Article  Google Scholar 

  55. Singha J, Roy A, Laskar RH (2018) Dynamic hand gesture recognition using vision-based approach for human–computer interaction. Neural Comput Appl 29(4):1129–1141

    Article  Google Scholar 

Download references

Acknowledgements

This work was supported by Grants of the National Natural Science Foundation of China (Grant Nos. 51575407, 51575338, 51575412, 61733011, 51505349) and Grants of the National Defense Pre-Research Foundation of Wuhan University of Science and Technology (GF201705). This paper is funded by Wuhan University of Science and Technology graduate students’ short-term study abroad special funds.

Author information

Authors and Affiliations

  1. Key Laboratory of Metallurgical Equipment and Control Technology of Ministry of Education, Wuhan University of Science and Technology, Wuhan, 430081, China

    Wentao Cheng, Ying Sun & Gongfa Li

  2. Hubei Key Laboratory of Mechanical Transmission and Manufacturing Engineering, Wuhan University of Science and Technology, Wuhan, 430081, China

    Ying Sun & Guozhang Jiang

  3. Research Center of Biologic Manipulator and Intelligent Measurement and Control, Wuhan University of Science and Technology, Wuhan, 430081, China

    Gongfa Li

  4. 3D Printing and Intelligent Manufacturing Engineering Institute, Wuhan University of Science and Technology, Wuhan, 430081, China

    Guozhang Jiang

  5. School of Computing, University of Portsmouth, Portsmouth, PO1 3HE, UK

    Honghai Liu

Authors
  1. Wentao Cheng
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ying Sun
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Gongfa Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Guozhang Jiang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Honghai Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Gongfa Li.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Cheng, W., Sun, Y., Li, G. et al. Jointly network: a network based on CNN and RBM for gesture recognition. Neural Comput & Applic 31 (Suppl 1), 309–323 (2019). https://doi.org/10.1007/s00521-018-3775-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-018-3775-8

Keywords

Search

Navigation