Neural Computing and Applications

, Volume 31, Supplement 1, pp 309–323 | Cite as

Jointly network: a network based on CNN and RBM for gesture recognition

  • Wentao Cheng
  • Ying Sun
  • Gongfa LiEmail author
  • Guozhang Jiang
  • Honghai Liu
Machine Learning Applications for Self-Organized Wireless Networks


Hand belongs to non-rigid objects and is rich in variety, making gesture recognition more difficult. The essence of dynamic gesture recognition is the classification and recognition of single-frame still images. Therefore, this paper mainly focuses on static gesture recognition. At present, there are some problems in gesture recognition, such as accuracy, real-time or poor robustness. To solve the above problems, in this paper, the Kinect sensor is used to obtain the color and depth gesture samples, and the gesture samples are processed. On this basis, a jointly network of CNN and RBM is proposed for gesture recognition. It mainly uses superposed network of multiple RBMs to carry out unsupervised feature extraction and combined with supervised feature extraction of CNN. Finally, these two features are combined to classify them. The simulation results show that the proposed jointly network has a better performance in identifying simple background gesture samples and the recognition capability of gesture samples in complex background needs to be improved.


Kinect sensor Jointly network Static gesture recognition Simulation experiment 



This work was supported by Grants of the National Natural Science Foundation of China (Grant Nos. 51575407, 51575338, 51575412, 61733011, 51505349) and Grants of the National Defense Pre-Research Foundation of Wuhan University of Science and Technology (GF201705). This paper is funded by Wuhan University of Science and Technology graduate students’ short-term study abroad special funds.


  1. 1.
    Traver VJ, Latorre-Carmon Luzanin AP, Salvador-Balaguer E, Filiberto P, Bahram J (2017) Three-dimensional integral imaging for gesture recognition under occlusions. IEEE Signal Process Lett 24(2):171–175Google Scholar
  2. 2.
    Oyedotun OK, Khashman A (2017) Deep learning in vision-based static hand gesture recognition. Neural Comput Appl 28(12):3941–3951Google Scholar
  3. 3.
    Nasri S, Behrad A, Razzazi F (2015) Spatio-temporal 3D surface matching for hand gesture recognition using ICP algorithm. SIViP 9(5):1205–1220zbMATHGoogle Scholar
  4. 4.
    Li GF, Tang H, Sun Y, Kong JY, Jiang GZ, Jiang D, Tao B, Xu S, Liu HH (2017) Hand gesture recognition based on convolution neural network. Cluster Comput. Google Scholar
  5. 5.
    He Y, Li GF, Liao YJ, Sun Y, Kong JY, Jiang GZ, Jiang D, Tao B, Xu S, Liu HH (2017) Gesture recognition based on an improved local sparse representation classification algorithm. Cluster Comput. Google Scholar
  6. 6.
    Ding WL, Li GF, Jiang GZ, Fang YF, Ju ZJ, Liu HH (2015) Intelligent computation in grasping control of dexterous robot hand. J Comput Theor Nanosci 12(12):6096–6099Google Scholar
  7. 7.
    Li B, Sun Y, Li GF, Kong JY, Jiang GZ, Jiang D, Tao B, Xu S, Liu HH (2017) Gesture recognition based on modified adaptive orthogonal matching pursuit algorithm. Cluster Comput. Google Scholar
  8. 8.
    Ordóñez FJ, Roggen D (2016) Deep convolutional and lstm recurrent neural networks for multimodal wearable activity recognition. Sensors 16(1):115Google Scholar
  9. 9.
    Jiang D, Zheng ZJ, Li GF, Sun Y, Kong JY, Jiang GZ, Xiong HG, Tao B, Xu S, Yu H, Liu HH, Ju ZJ (2018) Gesture recognition based on binocular vision. Cluster Comput. Google Scholar
  10. 10.
    Xiong HG, Fan HL, Li GF, Jiang GZ (2015) Research on steady-state simulation in dynamic job shop scheduling problem. Adv Mech Eng 7(9):1–11Google Scholar
  11. 11.
    Barros P, Maciel-Junior NT, Fernandes BJ, Bezerra BL, Fernandes SM (2017) A dynamic gesture recognition and prediction system using the convexity approach. Comput Vis Image Underst 155:139–149Google Scholar
  12. 12.
    Escalante HJ, Guyon I, Athitsos V, Jangyodsuk P, Wan J (2017) Principal motion components for one-shot gesture recognition. Pattern Anal Appl 20(1):167–182MathSciNetGoogle Scholar
  13. 13.
    Boughrara H, Chtourou M, Amar CB, Chen L (2016) Facial expression recognition based on a mlp neural network using constructive training algorithm. Multimed Tools Appl 75(2):709–731Google Scholar
  14. 14.
    Li GF, Gu YS, Kong JY, Jiang GZ, Xie LX, Wu ZH, Li Z, He Y, Gao P (2013) Intelligent control of air compressor production process. Appl Math Inf Sci 7(3):1051–1058Google Scholar
  15. 15.
    Li GF, Qu PX, Kong JY, Jiang GZ, Xie LX, Gao P, Wu ZH, He Y (2013) Coke oven intelligent integrated control system. Appl Math Inf Sci 7(3):1043–1050Google Scholar
  16. 16.
    Rautaray SS, Agrawal A (2015) Vision based hand gesture recognition for human computer interaction: a survey. Artif Intell Rev 43(1):1–54Google Scholar
  17. 17.
    Chakravarthi MK, Tiwari RK, Handa S (2015) Accelerometer based static gesture recognition and mobile monitoring system using neural networks. Proc Comput Sci 70:683–687Google Scholar
  18. 18.
    Luzanin O, Plancak M (2014) Hand gesture recognition using low-budget data glove and cluster-trained probabilistic neural network. Assembly Autom 34(1):94–105Google Scholar
  19. 19.
    Pisharady PK, Saerbeck M (2015) Recent methods and databases in vision-based hand gesture recognition: a review. Comput Vis Image Underst 141:152–165Google Scholar
  20. 20.
    Kılıboz NÇ, Güdükbay U (2015) A hand gesture recognition technique for human–computer interaction. J Vis Commun Image Represent 28:97–104Google Scholar
  21. 21.
    Hinton G, Sejnowski T (1983) Optimal perceptual inference. In IEEE conference on computer vision and pattern recognitionGoogle Scholar
  22. 22.
    Varghese B, Buyya R (2018) Next generation cloud computing: new trends and research directionsJ]. Future Gener Comput Syst 79:849–861Google Scholar
  23. 23.
    Sun Y, Hu JB, Li GF, Jiang GZ, Xiong HG, Tao B, Zheng ZJ, Jiang D (2018) Gear reducer optimal design based on computer multimedia simulation. J Supercomput. Google Scholar
  24. 24.
    Tieleman T (2008) Training restricted boltzmann machines using approximations to the likelihood gradient. In: International conference on machine learning (IC-ML) 2008Google Scholar
  25. 25.
    Li GF, Liu Z, Jiang GZ, Xiong HG, Liu HH (2015) Numerical simulation of the influence factors for rotary kiln in temperature field and stress field and the structure optimization. Adv Mech Eng 7(6):1687814015589667Google Scholar
  26. 26.
    Nguyen-Dinh LV, Calatroni A, Tröster G (2017) Supporting one-time point annotations for gesture recognition. IEEE Trans Pattern Anal Mach Intell 39(11):2270–2283Google Scholar
  27. 27.
    Deng L, He XD, Gao JF (2013) Deep stacking network for information retrieval. In: 2013 IEEE international conference on acoustics, speech, and signal processing (ICASSP)Google Scholar
  28. 28.
    Li Z, Li GF, Jiang GZ, Fang YF, Ju ZJ, Liu HH (2015) Intelligent computation of grasping and manipulation for multi-fingered robotic hands. J Comput Theor Nanosci 12(12):6192–6197Google Scholar
  29. 29.
    Li GF, Liu J, Jiang GZ, Liu HH (2015) Numerical simulation of temperature field and thermal stress field in the new type of ladle with the nanometer adiabatic material. Adv Mech Eng 7(4):1687814015575988Google Scholar
  30. 30.
    Xiong HG, Fan HL, Jiang GZ, Li GF (2017) A simulation -based study of dispatching rules in a dynamic job shop scheduling problem with batch release and extended technical precedence constraints. Eur J Oper Res 257(1):13–24MathSciNetzbMATHGoogle Scholar
  31. 31.
    Goh JEE, Goh MLI, Estrada JS, Lindog NC, Tabulog JCM, Talavera NEC (2017) Presentation-aid Armband with IMU, EMG sensor and bluetooth for free-hand writing and hand gesture recognition. Int J Comput Sci Res 1(3):54–66Google Scholar
  32. 32.
    Li GF, Qu PX, Kong JY, Jiang GZ, Xie LX, Wu ZH, Gao P, He Y (2013) Influence of working lining parameters on temperature and stress field of ladle. Appl Math Inf Sci 7(2):439–448Google Scholar
  33. 33.
    Ohn-Bar E, Trivedi MM (2014) Hand gesture recognition in real time for automotive interfaces: a multimodal vision-based approach and evaluations. IEEE Trans Intell Transp Syst 15(6):2368–2377Google Scholar
  34. 34.
    Chen DS, Li GF, Sun Y, Kong JY, Jiang GZ, Tang H, Ju ZJ, Yu H, Liu HH (2017) An interactive image segmentation method in hand gesture recognition. Sensors 17(2):253Google Scholar
  35. 35.
    Liao YJ, Sun Y, Li GF, Kong JY, Jiang GZ, Jiang D, Cai HB, Ju ZJ, Yu H, Liu HH (2017) Simultaneous calibration: a jointly optimization approach for multiple kinect and external cameras. Sensors 17(7):1491Google Scholar
  36. 36.
    Miao W, Li GF, Jiang GZ, Fang YF, Ju ZJ, Liu HH (2015) Optimal grasp planning of multi-fingered robotic hands: a review. Appl Comput Math 14(3):238–247MathSciNetzbMATHGoogle Scholar
  37. 37.
    Chen DS, Li GF, Sun Y, Jiang GZ, Kong JY, Liu HH (2017) Fusion hand gesture segmentation and extraction based on CMOS sensor and 3D sensor. Int J Wirel Mobile Comput 12(3):305–312Google Scholar
  38. 38.
    Sun Y, Li CQ, Li GF, Jiang GZ, Jiang D, Liu HH, Zheng ZJ, Shu WN (2018) Gesture recognition based on Kinect and sEMG signal fusion. Mobile Netw Appl 23(4):797–805Google Scholar
  39. 39.
    Fang YF, Liu HH, Li GF, Zhu XY (2015) A multichannel surface EMG system for hand motion recognition. Int J Humanoid Rob 12(2):1550011Google Scholar
  40. 40.
    Li Z, Li GF, Sun Y, Jiang GZ, Kong JY, Liu HH (2017) Development of articulated robot trajectory planning. Int J Comput Sci Math 8(1):52–60MathSciNetGoogle Scholar
  41. 41.
    Miao W, Li GF, Sun Y, Jiang GZ, Kong JY, Liu HH (2016) Gesture recognition based on sparse representation. Int J Wirel Mobile Comput 11(4):348–356Google Scholar
  42. 42.
    Yin Q, Li GF, Zhang JG (2015) Research on the method of step feature extraction for EOD robot based on 2d laser radar. Discrete Contin Dyn Syst-Ser S 8(6):1415–1421MathSciNetzbMATHGoogle Scholar
  43. 43.
    Ding WL, Li GF, Sun Y, Jiang GZ, Kong JY, Liu HH (2017) D-S evidential theory on sEMG signal recognition. Int J Comput Sci Math 8(2):138–145MathSciNetGoogle Scholar
  44. 44.
    Jadooki S, Mohamad D, Saba T et al (2017) Fused features mining for depth-based hand gesture recognition to classify blind human communication. Neural Comput Appl 28(11):3285–3294Google Scholar
  45. 45.
    Du F, Sun Y, Li GF, Li Z, Kong JY, Jiang GZ, Jiang D (2017) Adaptive fuzzy sliding mode control for 2-DOF articulated robot. J Wuhan Univ Sci Technol 40(6):446–450Google Scholar
  46. 46.
    Núñez JC, Cabido R, Pantrigo JJ, Montemayor AS, Vélez JF (2018) Convolutional neural networks and long short-term memory for skeleton-based human activity and hand gesture recognition. Pattern Recogn 76:80–94Google Scholar
  47. 47.
    Li GF, Miao W, Jiang GZ, Fang YF, Ju ZJ, Liu HH (2015) Intelligent control model and its simulation of flue temperature in coke oven. Discrete Contin Dyn Syst Ser S (DCDS-S) 8(6):1223–1237MathSciNetzbMATHGoogle Scholar
  48. 48.
    Poularakis S, Katsavounidis I (2016) Low-complexity hand gesture recognition system for continuous streams of digits and letters. IEEE Trans Cybern 46(9):2094–2108Google Scholar
  49. 49.
    Chang Wenjun, Li Gongfa, Kong Jianyi, Sun Ying, Jiang Guozhang, Liu Honghai (2018) Thermal mechanical stress analysis of ladle lining with integral brick joint. Arch Metall Mater 63(2):659–666Google Scholar
  50. 50.
    Misra S, Singha J, Laskar RH (2017) Vision-based hand gesture recognition of alphabets, numbers, arithmetic operators and ASCII characters in order to develop a virtual text-entry interface system. Neural Comput Appl 29(8):117–135Google Scholar
  51. 51.
    Li GF, Kong JY, Jiang GZ, Xie LX, Jiang ZG, Zhao G (2012) Air-fuel ratio intelligent control in coke oven combustion process. In Int Interdiscip J 15(11):4487–4494Google Scholar
  52. 52.
    Baraldi L, Paci F, Serra G, Benini L, Cucchiara R (2015) Gesture recognition using wearable vision sensors to enhance visitors’ museum experiences. IEEE Sens J 15(5):2705–2714Google Scholar
  53. 53.
    Li Gongfa, Zhang Leilei, Sun Ying, Kong Jianyi (2018) Internet of things sensors and haptic feedback for sEMG based hands. Multimed Tools Appl. Google Scholar
  54. 54.
    Gravina R, Ma C, Pace P, Aloi G, Russo W, Li W, Fortino G (2017) Cloud-based activity-aaService cyber–physical framework for human activity monitoring in mobility. Future Gener Comput Syst 75:158–171Google Scholar
  55. 55.
    Singha J, Roy A, Laskar RH (2018) Dynamic hand gesture recognition using vision-based approach for human–computer interaction. Neural Comput Appl 29(4):1129–1141Google Scholar

Copyright information

© The Natural Computing Applications Forum 2018

Authors and Affiliations

  • Wentao Cheng
    • 1
  • Ying Sun
    • 1
    • 2
  • Gongfa Li
    • 1
    • 3
    Email author
  • Guozhang Jiang
    • 2
    • 4
  • Honghai Liu
    • 5
  1. 1.Key Laboratory of Metallurgical Equipment and Control Technology of Ministry of EducationWuhan University of Science and TechnologyWuhanChina
  2. 2.Hubei Key Laboratory of Mechanical Transmission and Manufacturing EngineeringWuhan University of Science and TechnologyWuhanChina
  3. 3.Research Center of Biologic Manipulator and Intelligent Measurement and ControlWuhan University of Science and TechnologyWuhanChina
  4. 4.3D Printing and Intelligent Manufacturing Engineering InstituteWuhan University of Science and TechnologyWuhanChina
  5. 5.School of ComputingUniversity of PortsmouthPortsmouthUK

Personalised recommendations