Abstract
Recently, the concept of human–robot collaboration has raised many research interests. Instead of robots replacing human operators in workplaces, human–robot collaboration is the direction that allows human operators and robots to work together. Although the communication channels between human operators and robots are still limited, gesture recognition has been effectively applied as the interface between humans and computers for a long time. Covering some of the most important technologies and algorithms of gesture recognition, this chapter is intended to provide an overview of the gesture recognition research and explore the possibility to apply gesture recognition in human–robot collaboration. In this chapter, an overall model of gesture recognition for human–robot collaboration is also proposed. There are four essential technical components in the model of gesture recognition for human–robot collaboration: sensor technologies, gesture identification, gesture tracking and gesture classification. Reviewed approaches are classified according to the four essential technical components. After the reviewed technical components, an example of gesture recognition for human–robot collaboration is provided. In the last part of the chapter, future research trends are outlined.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
J. Krüger, T.K. Lien, A. Verl, Cooperation of human and machines in assembly lines, CIRP Ann. Manuf. Technol.58(2), 628–646 (2009)
S. Green, X. Chen, M. Billinnghurst, J.G. Chase, Human Robot collaboration: an augmented reality approach a literature review and analysis. Mechatronics 5(1), 1–10 (2007)
P.R. Cohen, H.J. Levesque, Teamwork, in Nous (1991) pp. 487–512
L.S. Vygotsky, Mind in Society: The Development of Higher Psychological Processes (Harvard university press, 1980)
P.R. Cohen, H.J. Levesque, Persistence, intention, and commitment, in Reason. About Actions Plans (1990), pp. 297–340
C. Breazeal, A. Brooks, J. Gray, G. Hoffman, C. Kidd, H. Lee, J. Lieberman, A. Lockerd, D. Mulanda, Humanoid robots as cooperative partners for people. Int. J. Humanoid Robot. 1(2), 1–34 (2004)
A. Bauer, D. Wollherr, M. Buss, Human–robot collaboration: a survey. Int. J. Humanoid Robot. 5(01), 47–66 (2008)
S. Mitra, T. Acharya, Gesture recognition: a survey. IEEE Trans. Syst. Man, Cybern. Part C Appl. Rev. 37(3), 311–324 (2007)
R. Parasuraman, T.B. Sheridan, C.D. Wickens, A model for types and levels of human interaction with automation. IEEE Trans. Syst. Man Cybern. Part A Syst. Humans30(3), 286–297 (2000)
T.E. Starner, Visual recognition of american sign language using hidden markov models (1995)
T. Starner, J. Weaver, A. Pentland, Real-time american sign language recognition using desk and wearable computer based video. IEEE Trans Pattern Anal Mach Intell 20(12), 1371–1375 (1998)
N.R. Howe, M.E. Leventon, W.T. Freeman, Bayesian reconstruction of 3d human motion from single-camera video. NIPS 99, 820–826 (1999)
Y. Katsuki, Y. Yamakawa, M. Ishikawa, High-speed human/Robot hand interaction system, in Proceedings of the Tenth Annual ACM/IEEE International Conference on Human–Robot Interaction Extended Abstracts (ACM, 2015), pp. 117–118
M. Elmezain, A. Al-Hamadi, J. Appenrodt, B. Michaelis, A hidden markov model-based continuous gesture recognition system for hand motion trajectory, in 19th International Conference on Pattern Recognition, 2008. ICPR 2008 (IEEE, 2008), pp. 1–4
Y. Matsumoto, A. Zelinsky, An algorithm for real-time stereo vision implementation of head pose and gaze direction measurement, in Fourth IEEE International Conference on Automatic Face and Gesture Recognition, 2000. Proceedings (IEEE, 2000), pp. 499–504
J.P. Wachs, M. Kölsch, H. Stern, Y. Edan, Vision-based hand-gesture applications. Commun. ACM 54(2), 60–71 (2011)
J. Suarez, R.R. Murphy, Hand gesture recognition with depth images: a review, in RO-MAN, 2012 IEEE (IEEE, 2012), pp. 411–417
P. Doliotis, A. Stefan, C. McMurrough, D. Eckhard, V. Athitsos, Comparing gesture recognition accuracy using color and depth information, in Proceedings of the 4th International Conference on Pervasive Technologies Related to Assistive Environments (ACM, 2011), p. 20
M. Hansard, S. Lee, O. Choi, R.P. Horaud, Time-of-Flight Cameras: Principles, Methods and Applications (Springer Science & Business Media, 2012)
T. Kapuściński, M. Oszust, M. Wysocki, Hand gesture recognition using time-of-flight camera and viewpoint feature histogram, in Intelligent Systems in Technical and Medical Diagnostics (Springer, 2014), pp. 403–414
S.B. Gokturk, H. Yalcin, C. Bamji, A time-of-flight depth sensor-system description, issues and solutions, in Conference on Computer Vision and Pattern Recognition Workshop, 2004. CVPRW’04 (IEEE, 2004), p. 35
J.D. Arango Paredes, B. Munoz, W. Agredo, Y. Ariza-Araujo, J.L. Orozco, A. Navarro, A reliability assessment software using Kinect to complement the clinical evaluation of Parkinson’s disease, in 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) (IEEE, 2015), pp. 6860–6863
C. Wang, Z. Liu, S.-C. Chan, Superpixel-based hand gesture recognition with Kinect depth camera. IEEE Trans Multimed 17(1), 29–39 (2015)
H. Liu, Y. Wang, W. Ji, L. Wang, A context-aware safety system for human–robot collaboration. Procedia Manuf. 17, 238–245 (2018)
A. Erol, G. Bebis, M. Nicolescu, R.D. Boyle, X. Twombly, Vision-based hand pose estimation: a review. Comput. Vis. Image Underst. 108(1), 52–73 (2007)
T. Labs, Myo (2015). https://www.myo.com/
Y. Zhang, C. Harrison, Tomo: wearable, low-cost electrical impedance tomography for hand gesture recognition, in Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology (ACM, 2015), pp. 167–173
N. Haroon, A.N. Malik, Multiple hand gesture recognition using surface EMG signals. J. Biomed. Eng. Med. Imaging 3(1), 1 (2016)
S. Roy, S. Ghosh, A. Barat, M. Chattopadhyay, and D. Chowdhury, Real-time implementation of electromyography for hand gesture detection using micro accelerometer, in Artificial Intelligence and Evolutionary Computations in Engineering Systems (Springer, 2016), pp. 357–364
Google, Project soli (2015). https://www.google.com/atap/project-soli/
J. Smith, T. White, C. Dodge, J. Paradiso, N. Gershenfeld, D. Allport, Electric field sensing for graphical interfaces. IEEE Comput. Graph. Appl. 18(3), 54–60 (1998)
F. Adib, C.-Y. Hsu, H. Mao, D. Katabi, F. Durand, Capturing the human figure through a wall. ACM Trans. Graph. 34(6), 219 (2015)
F. Adib, D. Katabi, See Through Walls with WIFI!, vol. 43, no. 4 (ACM, 2013)
F. Adib, Z. Kabelac, D. Katabi, R.C. Miller, 3d tracking via body radio reflections, in Usenix NSDI, vol. 14 (2014)
J. Letessier, F. Bérard, Visual tracking of bare fingers for interactive surfaces, in Proceedings of the 17th Annual ACM Symposium on User Interface Software and Technology (ACM, 2004), pp. 119–122
D. Weinland, R. Ronfard, E. Boyer, A survey of vision-based methods for action representation, segmentation and recognition. Comput. Vis. Image Underst. 115(2), 224–241 (2011)
D.G. Lowe, Object recognition from local scale-invariant features, in The Proceedings of the Seventh IEEE International Conference on Computer Vision, 1999, vol. 2 (IEEE, 1999), pp. 1150–1157
H. Bay, T. Tuytelaars, L. Van Gool, Surf: speeded up robust features, in Computer Vision–ECCV 2006 (Springer, 2006), pp. 404–417
E. Rublee, V. Rabaud, K. Konolige, G. Bradski, ORB: an efficient alternative to SIFT or SURF, in 2011 IEEE International Conference on Computer Vision (ICCV) (IEEE, 2011), pp. 2564–2571
S. Belongie, J. Malik, J. Puzicha, Shape matching and object recognition using shape contexts. IEEE Trans. Pattern Anal. Mach. Intell. 24(4), 509–522 (2002)
B. Allen, B. Curless, Z. Popović, Articulated body deformation from range scan data. ACM Trans. Graph. (TOG)21(3), 612–619 (ACM, 2002)
R. Cutler, M. Turk, View-based interpretation of real-time optical flow for gesture recognition, fg (IEEE, 1998), p. 416
J.L. Barron, D.J. Fleet, S.S. Beauchemin, Performance of optical flow techniques. Int. J. Comput. Vis. 12(1), 43–77 (1994)
C. Thurau, V. Hlaváč, Pose primitive based human action recognition in videos or still images, in IEEE Conference on Computer Vision and Pattern Recognition, 2008. CVPR 2008 (IEEE, 2008), pp. 1–8
Q. Pu, S. Gupta, S. Gollakota, S. Patel, Whole-home gesture recognition using wireless signals, in Proceedings of the 19th annual international conference on Mobile computing & networking (ACM, 2013), pp. 27–38
R. Ronfard, C. Schmid, B. Triggs, Learning to parse pictures of people, in Computer Vision—ECCV 2002 (Springer, 2002), pp. 700–714
S.-J. Lee, C.-S. Ouyang, S.-H. Du, A neuro-fuzzy approach for segmentation of human objects in image sequences. IEEE Trans. Syst. Man, Cybern. Part B Cybern33(3), 420–437 (2003)
D. Tang, H.J. Chang, A. Tejani, T.-K. Kim, Latent regression forest: structured estimation of 3D articulated hand posture, in 2014 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (IEEE, 2014), pp. 3786–3793
J. Han, L. Shao, D. Xu, J. Shotton, Enhanced computer vision with microsoft kinect sensor: a review. IEEE Trans Cybern. 43(5), 1318–1334 (2013)
I. Oikonomidis, N. Kyriazis, A.A. Argyros, Efficient model-based 3D tracking of hand articulations using Kinect. BMVC 1(2), 3 (2011)
J. Shotton, A. Fitzgibbon, M. Cook, T. Sharp, M. Finocchio, R. Moore, A. Kipman, A. Blake, Real-time human pose recognition in parts from single depth images. Commun. ACM 56(1), 116–124 (2013)
M. Ye, X. Wang, R. Yang, L. Ren, M. Pollefeys, Accurate 3D pose estimation from a single depth image, in IEEE International Conference on Computer Vision (ICCV), 2011 (IEEE, 2011), pp. 731–738
Y. Li, Hand gesture recognition using Kinect, in 2012 IEEE 3rd International Conference on Software Engineering and Service Science (ICSESS) (IEEE, 2012), pp. 196–199
H. Liu, T. Fang, T. Zhou, Y. Wang, L. Wang, Deep learning-based multimodal control interface for human–robot collaboration. Procedia CIRP 72, 3–8 (2018)
D. Comaniciu, V. Ramesh, P. Meer, Real-time tracking of non-rigid objects using mean shift, in IEEE Conference on Computer Vision and Pattern Recognition, 2000. Proceedings, vol 2 (IEEE, 2000), pp. 142–149
R.E. Kalman, A new approach to linear filtering and prediction problems. J. Fluids Eng. 82(1), 35–45 (1960)
S. Haykin, Kalman Filtering and Neural Networks, vol. 47 (Wiley, 2004)
R. Kandepu, B. Foss, L. Imsland, Applying the unscented Kalman filter for nonlinear state estimation. J. Process Control 18(7), 753–768 (2008)
R. Gao, L. Wang, R. Teti, D. Dornfeld, S. Kumara, M. Mori, M. Helu, Cloud-enabled prognosis for manufacturing. CIRP Ann. Technol. 64(2), 749–772 (2015)
J. Krüger, T.K. Lien, A. Verl, Cooperation of human and machines in assembly lines. CIRP Ann. Technol. 58(2), 628–646 (2009)
K. Okuma, A. Taleghani, N. De Freitas, J.J. Little, D.G. Lowe, A boosted particle filter: multitarget detection and tracking, in Computer Vision-ECCV 2004 (Springer, 2004), pp. 28–39
S. Oron, A. Bar-Hillel, D. Levi, S. Avidan, Locally orderless tracking, in 2012 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (IEEE, 2012), pp. 1940–1947
J. Kwon, K.M. Lee, Tracking by sampling trackers, in 2011 IEEE International Conference on Computer Vision (ICCV) (IEEE, 2011), pp. 1195–1202
T. Li, S. Sun, T.P. Sattar, J.M. Corchado, Fight sample degeneracy and impoverishment in particle filters: a review of intelligent approaches. Expert Syst. Appl. 41(8), 3944–3954 (2014)
T. Li, T.P. Sattar, S. Sun, Deterministic resampling: unbiased sampling to avoid sample impoverishment in particle filters. Signal Process 92(7), 1637–1645 (2012)
J.M. Del Rincón, D. Makris, C.O. Uruňuela, J.-C. Nebel, Tracking human position and lower body parts using Kalman and particle filters constrained by human biomechanics. IEEE Trans. Syst. Man, Cybern. Part B Cybern.41(1), 26–37 (2011)
D.A. Ross, J. Lim, R.-S. Lin, M.-H. Yang, Incremental learning for robust visual tracking. Int. J. Comput. Vis. 77(1–3), 125–141 (2008)
J. Kwon, K.M. Lee, F.C. Park, Visual tracking via geometric particle filtering on the affine group with optimal importance functions, in IEEE Conference on Computer Vision and Pattern Recognition, 2009. CVPR 2009 (IEEE, 2009), pp. 991–998
Z. Kalal, J. Matas, K. Mikolajczyk, P-N learning: bootstrapping binary classifiers by structural constraints, in 2010 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (IEEE, 2010), pp. 49–56
B. Babenko, M.-H. Yang, S. Belongie, Visual tracking with online multiple instance learning, in IEEE Conference on Computer Vision and Pattern Recognition, 2009. CVPR 2009 (IEEE, 2009), pp. 983–990
A.W.M. Smeulders, D.M. Chu, R. Cucchiara, S. Calderara, A. Dehghan, M. Shah, Visual tracking: an experimental survey. IEEE Trans. Pattern Anal. Mach. Intell. 36(7), 1442–1468 (2014)
A.D. Wilson, A.F. Bobick, Parametric hidden markov models for gesture recognition. IEEE Trans. Pattern Anal. Mach. Intell. 21(9), 884–900 (1999)
S. Lu, J. Picone, S. Kong, Fingerspelling alphabet recognition using a two-level hidden markov model, in Proceedings of the International Conference on Image Processing, Computer Vision, and Pattern Recognition (IPCV). The Steering Committee of The World Congress in Computer Science, Computer Engineering and Applied Computing (WorldComp) (2013), p. 1
J. McCormick, K. Vincs, S. Nahavandi, D. Creighton, S. Hutchison, Teaching a digital performing agent: artificial neural network and hidden Markov model for recognising and performing dance movement, in Proceedings of the 2014 International Workshop on Movement and Computing (ACM, 2014), p. 70
S.-Z. Yu, Hidden semi-Markov models. Artif. Intell. 174(2), 215–243 (2010)
H. Liu, L. Wang, Human motion prediction for human–robot collaboration. J. Manuf. Syst. 44, 287–294 (2017)
L.R. Rabiner, A tutorial on hidden Markov models and selected applications in speech recognition. IEEE Proc. 77(2), 257–286 (1989)
M.A. Hearst, S.T. Dumais, E. Osman, J. Platt, B. Scholkopf, Support vector machines. IEEE Intell. Syst. Their Appl. 13(4), 18–28 (1998)
M.E. Tipping, Sparse Bayesian learning and the relevance vector machine. J. Mach. Learn. Res. 1, 211–244 (2001)
B. Schiilkopf, The kernel trick for distances, in Advances in Neural Information Processing Systems 13: Proceedings of the 2000 Conference, vol. 13 (MIT Press, 2001), p. 301
A. Cenedese, G.A. Susto, G. Belgioioso, G.I. Cirillo, F. Fraccaroli, Home automation oriented gesture classification from inertial measurements. IEEE Trans. Autom. Sci. Eng. 12(4), 1200–1210 (2015)
K. Feng, F. Yuan, Static hand gesture recognition based on HOG characters and support vector machines, in 2013 2nd International Symposium on Instrumentation and Measurement, Sensor Network and Automation (IMSNA) (IEEE, 2013), pp. 936–938
D. Ghimire, J. Lee, Geometric feature-based facial expression recognition in image sequences using multi-class adaboost and support vector machines. Sensors 13(6), 7714–7734 (2013)
O. Patsadu, C. Nukoolkit, B. Watanapa, Human gesture recognition using Kinect camera, in 2012 International Joint Conference on Computer Science and Software Engineering (JCSSE) (IEEE, 2012), pp. 28–32
A. Krizhevsky, I. Sutskever, G.E. Hinton, Imagenet classification with deep convolutional neural networks, in Advances in Neural Information Processing Systems (2012), pp. 1097–1105
Y. LeCun, L. Bottou, Y. Bengio, P. Haffner, Gradient-based learning applied to document recognition. IEEE Proc. 86(11), 2278–2323 (1998)
H. Liu, L. Wang, Remote human–robot collaboration: a cyber–physical system application for hazard manufacturing environment. J. Manuf. Syst. 54, 24–34 (2020)
H. Liu, T. Fang, T. Zhou, L. Wang, Towards robust human–robot collaborative manufacturing: multimodal fusion. IEEE Access 6, 74762–74771 (2018)
P. Wang, H. Liu, L. Wang, R.X. Gao, Deep learning-based human motion recognition for predictive context-aware human–robot collaboration. CIRP Ann. (2018)
S. Hochreiter, J. Urgen Schmidhuber, Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)
L. Wang, R.X. Gao, J. Váncza, J. Krüger, X.V. Wang, S. Makris, G. Chryssolouris, Symbiotic human–robot collaborative assembly. CIRP Ann. Manuf. Technol.68(2), 701–726 (2019)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this chapter
Cite this chapter
Liu, H., Wang, L. (2021). Latest Developments of Gesture Recognition for Human–Robot Collaboration. In: Wang, L., Wang, X.V., Váncza, J., Kemény, Z. (eds) Advanced Human-Robot Collaboration in Manufacturing. Springer, Cham. https://doi.org/10.1007/978-3-030-69178-3_2
Download citation
DOI: https://doi.org/10.1007/978-3-030-69178-3_2
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-69177-6
Online ISBN: 978-3-030-69178-3
eBook Packages: EngineeringEngineering (R0)