Gesture recognition based on binocular vision

  • Du JiangEmail author
  • Zujia Zheng
  • Gongfa Li
  • Ying Sun
  • Jianyi Kong
  • Guozhang Jiang
  • Hegen Xiong
  • Bo Tao
  • Shuang Xu
  • Hui Yu
  • Honghai Liu
  • Zhaojie Ju


A convenient and effective binocular vision system is set up. Gesture information can be accurately extract from the complex environment with the system. The template calibration method is used to calibrate the binocular camera and the parameters of the camera are accurately obtained. In the phase of stereo matching, the BM algorithm is used to quickly and accurately match the images of the left and right cameras to get the parallax of the measured gesture. Combined with triangulation principle, resulting in a more dense depth map. Finally, the depth information is remapped to the original color image to realize three-dimensional reconstruction and three-dimensional cloud image generation. According to the cloud image information, it can be judged that the binocular vision system can effectively segment the gesture from the complex background.


Binocular vision Gesture recognition Gesture segmentation Template calibration method 



This work was supported by grants of National Natural Science Foundation of China (Grant Nos. 51575407, 51575338, 51575412, 61733011) and the Grants of National Defense Pre-Research Foundation of Wuhan University of Science and Technology (GF201705).


  1. 1.
    Al-Helali, B.M., Mahmoud, S.A.: Arabic online handwriting recognition (AOHR): a survey. ACM Comput. Surv. (CSUR). 50(3), 33 (2017)CrossRefGoogle Scholar
  2. 2.
    Sturman, D.J., David Z., Steve P.: Hands-on interaction with virtual environments. Proceedings of the 2nd Annual ACM SIGGRAPH Symposium on User Interface Software and Technology, pp. 19–24. ACM (1989)Google Scholar
  3. 3.
    In-Cheol, K., Chien, S.-I.: Analysis of 3d hand trajectory gestures using stroke-based composite hidden markov models. Appl. Intell. 15(2), 131–143 (2001)CrossRefzbMATHGoogle Scholar
  4. 4.
    Noor, T., Shanableh, T., Assaleh, K.: Glove-based continuous Arabic sign language recognition in user-dependent mode. IEEE Trans. Hum.-Mach. Syst. 45(4), 526–533 (2015)CrossRefGoogle Scholar
  5. 5.
    Fang, Y.L., Liu, H.L., Gongfa, Z.X.: A multichannel surface emg system for hand motion recognition. Int. J. Hum. Robot. 12(02), 1550011 (2015). CrossRefGoogle Scholar
  6. 6.
    Frederic, K., Puhl, M., Krüger, A.: User-independent real-time hand gesture recognition based on surface electromyography. Proceedings of the 19th International Conference on Human-Computer Interaction with Mobile Devices and Services, p. 36. ACM (2017)Google Scholar
  7. 7.
    Tobely, T.E., Yoshiki, Y., Tsuda, R., Tsuruta, N., Amamiy, M.: Dynamic hand gesture recognition based on randomized self-organizing map algorithm. International Conference on Algorithmic Learning Theory, pp. 252–263. Springer, Berlin (2000)Google Scholar
  8. 8.
    Bhuyan, M.K., MacDorman, K.F., Kar, M.K., Neog, D.R., Lovell, B.C., Gadde, P.: Hand pose recognition from monocular images by geometrical and texture analysis. J. Vis. Lang. Comput. 28, 39–55 (2015)CrossRefGoogle Scholar
  9. 9.
    Yin, Q., Li, G., Zhu, J.: Research on the method of step feature extraction for EOD robot based on 2d laser radar. Discret. Contin. Dyn. Syst. Ser. 8(6), 1415–1421 (2015)MathSciNetCrossRefzbMATHGoogle Scholar
  10. 10.
    Li, Z., Li, G., Jiang, G., Fang, Y., Zhaojie, J., Liu, H.: Intelligent computation of grasping and manipulation for multi-fingered robotic hands. J. Comput. Theor. Nanosci. 12(12), 6192–6197 (2015)CrossRefGoogle Scholar
  11. 11.
    Li, Z., Li, G., Sun, Y., Jiang, G., Kong, J., Liu, H.: Development of articulated robot trajectory planning. Int. J. Comput. Sci. Math. 8(1), 52–60 (2017)MathSciNetCrossRefGoogle Scholar
  12. 12.
    Ding, W., Li, G., Sun, Y., Jiang, G., Kong, J., Liu, H.: D-S evidential theory on semg signal recognition. Int. J. Comput. Sci. Math. 8(2), 138–145 (2017)MathSciNetCrossRefGoogle Scholar
  13. 13.
    He, Y.L., Gongfa, L.Y., Sun, Y.K., Jianyi, J., Guozhang, J., Du, T., Bo, X., Shuang, L.H.: Gesture recognition based on an improved local sparse representation classification algorithm. Clust. Comput. (2017).
  14. 14.
    Li, B.S., Ying, L.G., Kong, J.J., Guozhang, J.D., Tao, B.X., Shuang, L.H.: Gesture recognition based on modified adaptive orthogonal matching pursuit algorithm. Clust. Comput. (2017).
  15. 15.
    Chen, D., Li, G., Sun, Y., Kong, J., Jiang, G., Tang, H., Zhaojie, J., Hui, Y., Liu, H.: An interactive image segmentation method in hand gesture recognition. Sensors 17(2), 253 (2017)CrossRefGoogle Scholar
  16. 16.
    Chen, D., Li, G., Sun, Y., Kong, J., Jiang, G., Li, J., Liu, H.: Fusion hand gesture segmentation and extraction based on CMOS sensor and 3D sensor. Int. J. Wirel. Mob. Comput. 12(3), 305–312 (2017)CrossRefGoogle Scholar
  17. 17.
    Li, G., Miao, W., Jiang, G., Fang, Y., Zhaojie, J., Liu, H.: Intelligent control model and its simulation of flue temperature in coke oven. Discret. Contin. Dyn. Syst. Ser. 8(6), 1223–1237 (2015)MathSciNetCrossRefzbMATHGoogle Scholar
  18. 18.
    Ding, W., Li, G., Jiang, G., Fang, Y., Zhaojie, J., Liu, H.: Intelligent computation in grasping control of dexterous robot hand. J. Comput. Theor. Nanosci. 12(12), 6096–6099 (2015)CrossRefGoogle Scholar
  19. 19.
    Wei, M., Li, G., Jiang, G., Fang, Y., Zhaojie, J., Liu, H.: Optimal grasp planning of multi-fingered robotic hands: a review. Appl. Comput. Math. 14(3), 238–247 (2015)MathSciNetzbMATHGoogle Scholar
  20. 20.
    Feng, L.B., Sheng, D.M., Liu, Y.: A gesture recognition method based on binocular vision system. International Conference on Computer Vision Systems, pp. 257-267. Springer, Cham (2017)Google Scholar
  21. 21.
    Jadooki, S., Mohamad, D., Saba, T., Almazyad, A.S., Rehman, Amjad: Fused features mining for depth-based hand gesture recognition to classify blind human communication. Neural Comput. Appl. 28(11), 3285–3294 (2017)CrossRefGoogle Scholar
  22. 22.
    Stroppa, L., Cristalli, C.: Stereo vision system for accurate 3D measurements of connector pins’ positions in production lines. Exp. Tech. 41(1), 69–78 (2017)CrossRefGoogle Scholar
  23. 23.
    Miao, W., Li, G., Sun, Y., Jiang, G., Kong, J., Liu, H.: Gesture recognition based on sparse representation. Int. J. Wirel. Mob. Comput. 11(4), 348–356 (2016)CrossRefGoogle Scholar
  24. 24.
    Mehryar, E., Evans, A.: Nasal patches and curves for expression-robust 3D face recognition. IEEE Trans. Pattern Anal. Mach. Intell. 39(5), 995–1007 (2017)CrossRefGoogle Scholar
  25. 25.
    Zhaojie, Ju, Ji, Xiaofei, Li, Jing, Liu, Honghai: An integrative framework of human hand gesture segmentation for human-robot interaction. IEEE Syst. J. 99, 1–11 (2015)Google Scholar
  26. 26.
    Li, G., Kong, J., Jiang, G., Xie, L., Jiang, Z., Zhao, G.: Air-fuel ratio intelligent control in coke oven combustion process. Inform. Int. Interdiscip. J. 15(11), 4487–4494 (2012)Google Scholar
  27. 27.
    Pop, D.O., Rogozan, A., Nashashibi, F., Bensrhair, A.: Fusion of stereo vision for pedestrian recognition using convolutional neural networks. ESANN 2017-25th European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning. (2017)Google Scholar
  28. 28.
    Chen, D., Li, G., Jiang, G., Fang, Y., Zhaojie, J., Liu, H.: Intelligent computational control of multi-fingered dexterous robotic hand. J. Comput. Theor. Nanosci. 12(12), 6126–6132 (2015)CrossRefGoogle Scholar
  29. 29.
    Li, G., Liu, J., Jiang, G., Liu, H.: Numerical simulation of temperature field and thermal stress field in the new type of ladle with the nanometer adiabatic material. Adv. Mech. Eng. 7(4), 1–13 (2015)Google Scholar
  30. 30.
    Starr, J.W., Lattimer, B.: Evidential sensor fusion of long-wavelength infrared stereo vision and 3D-LIDAR for rangefinding in fire environments. Fire Technol. 53, 1961–1983 (2017)CrossRefGoogle Scholar
  31. 31.
    Li, G., Yuesheng, G., Kong, J., Jiang, G., Xie, L., Zehao, W., Li, Z., He, Y., Gao, P.: Intelligent control of air compressor production process. Appl. Math. Inform. Sci. 7(3), 1051–1058 (2013)CrossRefGoogle Scholar
  32. 32.
    Scharstein, D., Szeliski, R.: High-accuracy stereo depth maps using structured light computer vision and pattern recognition, 2003. Proceedings. 2003 IEEE Computer Society Conference on. IEEE. 1: I-I (2003)Google Scholar
  33. 33.
    Mei, Q., Gao, J., Lin, H., Chen, Y., Yunbo, H., Wang, W., Zhang, G., Chen, X.: Structure light telecentric stereoscopic vision 3D measurement system based on Scheimpflug condition. Opt. Lasers Eng. 86, 83–91 (2016)CrossRefGoogle Scholar
  34. 34.
    Aguilar, J.J., Torres, F., Lope, M.A.: Stereo vision for 3D measurement: accuracy analysis, calibration and industrial applications. Measurement 18(4), 193–200 (1996)CrossRefGoogle Scholar
  35. 35.
    Li, G., Tang, H., Sun, Y., Kong, J., Guozhang J., Du, J., Bo T., Shuang, X., Liu, H.: Hand gesture recognition based on convolution neural network. Clust. Comput. (2017).
  36. 36.
    Zhang, Y., Wang, Z., Zou, L., Fang, H.: Event-based finite-time filtering for multi-rate systems with fading measurements. IEEE Trans. Aerosp. Electron. Syst. 53(3), 1431–1441 (2017)CrossRefGoogle Scholar
  37. 37.
    Abdel-Aziz, Y., Karara, H.M.: Direct linear transformation into object space coordinates in close range photogrametry. Urbana-Champaign. pp. 1–18 (1971)Google Scholar
  38. 38.
    Zhang, Y., Wang, Z., Ma, L.: Variance-constrained state estimation for networked multi-rate systems with measurement quantization and probabilistic sensor failures. Int. J. Robust Nonlinear Control 26(16), 3507–3523 (2016)MathSciNetCrossRefzbMATHGoogle Scholar
  39. 39.
    Tsai, R.: A versatile camera calibration technique for high-accuracy 3D machine vision metrology using off-the-shelf TV cameras and lenses. IEEE J. Robot. Autom. 3(4), 323–344 (1987)CrossRefGoogle Scholar
  40. 40.
    Moons, T., Van Gool, L., Proesmans, M., Pauwels, E.: Affine reconstruction from perspective image pairs with a relative object-camera translation in between. IEEE Trans. Pattern Anal. Mach. Intell. 18(1), 77–83 (1996)CrossRefGoogle Scholar
  41. 41.
    Triggs, B.: Autocalibration and the absolute quadric. Computer Vision and Pattern Recognition, 1997. Proceedings., 1997 IEEE Computer Society Conference on. pp. 609–614. IEEE (1997)Google Scholar
  42. 42.
    Liao, Y., Sun, Y., Li, G., Kong, J., Guozhang Jiang, Du, Jiang, H.C., Zhaojie, J., Hui, Y., Liu, H.: Simultaneous calibration: a joint optimization approach for multiple kinect and external cameras. Sensors 17(7), 1491 (2017). CrossRefGoogle Scholar
  43. 43.
    Tian, Z.: Face recognition from a single image per person using deep architecture neural networks. Clust. Comput. 19(1), 73–77 (2016)MathSciNetCrossRefGoogle Scholar
  44. 44.
    Sun, Y., Li, C., Li, G., Jiang, G., Jiang, D., Liu, H., Zheng, Z., Shu, W.: Gesture recognition based on kinect and sEMG signal fusion. Mob Netw Appl. (2018).
  45. 44.
    Li, G., Peixin, Q., Kong, J., Jiang, G., Xie, L., Gao, P., Zehao, W., He, Y.: Coke oven intelligent integrated control system. Appl. Math. Inform. Sci. 7(3), 1043–1050 (2013)CrossRefGoogle Scholar
  46. 45.
    Li, G., Peixin, Q., Kong, J., Jiang, G., Xie, L., Zehao, W., Gao, P., He, Y.: Influence of working lining parameters on temperature and stress field of ladle. Appl. Math. Inform. Sci. 7(2), 439–448 (2013)CrossRefGoogle Scholar
  47. 46.
    Li, G., Liu, Z., Jiang, G., Xiong, H., Liu, H.: Numerical simulation of the influence factors for rotary kiln in temperature field and stress field and the structure optimization. Adv. Mech. Eng. 7(6), 1–15 (2015)CrossRefGoogle Scholar
  48. 47.
    Wang, F., Jia, K., Feng, J.: The real-time depth map obtainment based on stereo matching. The Euro-China Conference on Intelligent Data Analysis and Applications, pp. 138–144, Springer International Publishing (2016)Google Scholar
  49. 48.
    Yang, Q.: Stereo matching using tree filtering. IEEE Trans. Pattern Anal. Mach. Intell. 37(4), 834–846 (2015)CrossRefGoogle Scholar
  50. 49.
    Mozerov, M.G., van de Weijer, J.: Accurate stereo matching by two-step energy minimization. IEEE Trans. Image Process. 24(3), 1153–1163 (2015)MathSciNetCrossRefGoogle Scholar
  51. 50.
    Hirschmuller, H., Scharstein, D.: Evaluation of stereo matching costs on images with radiometric differences. IEEE Trans. Pattern Anal. Mach. Intell. 31(9), 1582–1599 (2009)CrossRefGoogle Scholar
  52. 51.
    Li, J., Huang, W., Shao, L., Allinson, N.: Building recognition in urban environments: a survey of state-of-the-art and future challenges. Inform. Sci. 277(2), 406–420 (2014)CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Key Laboratory of Metallurgical Equipment and Control Technology of Ministry of EducationWuhan University of Science and TechnologyWuhanChina
  2. 2.Hubei Key Laboratory of Mechanical Transmission and Manufacturing EngineeringWuhan University of Science and TechnologyWuhanChina
  3. 3.Research Center of Biologic Manipulator and Intelligent Measurement and ControlWuhan University of Science and TechnologyWuhanChina
  4. 4.School of ComputingUniversity of PortsmouthPortsmouthUK

Personalised recommendations