Abstract
In human-robot collaborative manufacturing, industrial robots would work alongside the human workers who jointly perform the assigned tasks. Recent research work revealed that recognised human motions could be used as input for industrial robots control. However, the human-robot collaboration team still cannot work symbiotically. In response to the requirement, this chapter explores the potential of establishing context awareness between a human worker and an industrial robot for human-robot collaborative assembly . The context awareness between the human worker and the industrial robot is established by applying gesture recognition , human motion recognition and Augmented Reality (AR) based worker instruction technologies. Such a system works in a cyber-physical environment and is demonstrated by case studies.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
J. Krüger, T.K. Lien, A. Verl, Cooperation of human and machines in assembly lines. CIRP Ann. Technol. 58, 628–646 (2009)
S.A. Green, M. Billinghurst, X. Chen, G.J. Chase, Human-robot collaboration: a literature review and augmented reality approach in design. Int. J. Adv. Robot. Syst. 1–18 (2008)
P.R. Cohen, H.J. Levesque, Teamwork. Nous 487–512 (1991)
L.S. Vygotsky, Mind in society: the development of higher psychological processes (Harvard University Press, 1980)
P.R. Cohen, H.J. Levesque, persistence, intention, and commitment. Reason. About Actions Plans 297–340 (1990)
C. Breazeal et al., Humanoid robots as cooperative partners for people. Int. J. Humanoid Robot. 1, 1–34 (2004)
Z.M. Bi, L. Wang, Advances in 3D data acquisition and processing for industrial applications. Robot. Comput. Integr. Manuf. 26, 403–413 (2010)
B. Schmidt, L. Wang, Depth camera based collision avoidance via active robot control. J. Manuf. Syst. 33, 711–718 (2014)
H. Liu, L. Wang, Gesture recognition for human-robot collaboration: a review (J. Ind. Ergon, Int, 2017). doi:10.1016/j.ergon.2017.02.004
A. Bauer, D. Wollherr, M. Buss, Human–robot collaboration: a survey. Int. J. Humanoid Robot. 5, 47–66 (2008)
S. Mitra, T. Acharya, Gesture recognition: a survey. IEEE Trans. Syst. Man, Cybern. Part C Appl. Rev. 37, 311–324 (2007)
R. Parasuraman, T.B. Sheridan, C.D. Wickens, A model for types and levels of human interaction with automation. IEEE Trans. Syst. Man Cybern. Part A Syst. Humans. 30, 286–297 (2000)
T.E. Starner, Visual Recognition of American Sign Language Using Hidden Markov Models (1995)
T. Starner, J. Weaver, A. Pentland, Real-time american sign language recognition using desk and wearable computer based video, in IEEE Trans. on Pattern Analysis and Machine Intelligence, vol. 20 (1998), pp. 1371–1375
N.R. Howe, M.E. Leventon, W.T. Freeman, Bayesian reconstruction of 3D human motion from single-camera video. NIPS 99, 820–826 (1999)
Y. Katsuki, Y. Yamakawa, M. Ishikawa, High-speed human/robot hand interaction system, in Proceedings of the Tenth Annual ACM/IEEE International Conference on Human-Robot Interaction Extended Abstracts (2015), pp. 117–118
M. Elmezain, A. Al-Hamadi, J. Appenrodt, B. Michaelis, A hidden markov model-based continuous gesture recognition system for hand motion trajectory, in 19th International Conference on Pattern Recognition (2008), pp. 1–4
Y. Matsumoto, A. Zelinsky, An algorithm for real-time stereo vision implementation of head pose and gaze direction measurement, in Proceedings Fourth IEEE International Conference on Automatic Face and Gesture Recognition (2000), pp. 499–504
J.P. Wachs, M. Kölsch, H. Stern, Y. Edan, Vision-based hand-gesture applications. Commun. ACM 54, 60–71 (2011)
J. Suarez, R.R. Murphy, Hand gesture recognition with depth images: a review. IEEE RO-MAN 411–417 (2012)
P. Doliotis, A. Stefan, C. McMurrough, D. Eckhard, V. Athitsos, Comparing gesture recognition accuracy using color and depth information, in Proceedings of the 4th International Conference on Pervasive Technologies Related to Assistive Environments (2011), p. 20
T. Sharp et al., Accurate, robust, and flexible real-time hand tracking, in Proceeding CHI (2015), p. 8
A. Erol, G. Bebis, M. Nicolescu, R.D. Boyle, X. Twombly, Vision-based hand pose estimation: a review. Comput. Vis. Image Underst. 108, 52–73 (2007)
T. Labs, Myo (2015) https://www.myo.com/
Y. Zhang, C. Harrison, Tomo: wearable, low-cost electrical impedance tomography for hand gesture recognition, in Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology (2015), pp. 167–173
N. Haroon, A.N. Malik, Multiple hand gesture recognition using surface EMG signals. J. Biomed. Eng. Med. Imaging 3, 1 (2016)
S. Roy, S. Ghosh, A. Barat, M. Chattopadhyay, D. Chowdhury, Artif. Intell. Evol. Comput. Engin. Syst. 357–364 (2016)
Google, Project soli (2015) https://www.google.com/atap/project-soli/
J. Smith et al., Electric field sensing for graphical interfaces. Comput. Graph. Appl. IEEE 18, 54–60 (1998)
F. Adib, C.-Y. Hsu, H. Mao, D. Katabi, F. Durand, Capturing the human figure through a wall. ACM Trans. Graph. 34, 219 (2015)
F. Adib, D. Katabi, See through walls with WiFi! ACM. 43 (2013)
F. Adib, Z. Kabelac, D. Katabi, R.C. Miller, 3D tracking via body radio reflections. Usenix NSDI 14 (2014)
J. Letessier, F. Bérard, Visual tracking of bare fingers for interactive surfaces, in Proceedings of the 17th annual ACM symposium on User interface software and technology (2004), pp. 119–122
D. Weinland, R. Ronfard, E. Boyer, A survey of vision-based methods for action representation, segmentation and recognition. Comput. Vis. Image Underst. 115, 224–241 (2011)
D.G. Lowe, Object recognition from local scale-invariant features, in Proceedings of 7th IEEE International Conference on Computer Vision, vol. 2 (1999), pp. 1150–1157
H. Bay, T. Tuytelaars, L. Van Gool, Computer vision—ECCV (2006), pp. 404–417
E. Rublee, V. Rabaud, K. Konolige, G. Bradski, ORB: an efficient alternative to SIFT or SURF, in IEEE International Conference on Computer Vision (ICCV) (2011), pp. 2564–2571
S. Belongie, J. Malik, J. Puzicha, Shape matching and object recognition using shape contexts. Pattern Anal. Mach. Intell. IEEE Trans. 24, 509–522 (2002)
B. Allen, B. Curless, Z. Popović, Articulated body deformation from range scan data. ACM Trans. Graph. 21, 612–619 (2002)
I. Oikonomidis, N. Kyriazis, A.A. Argyros, Efficient model-based 3D tracking of hand articulations using Kinect. BMVC 1, 3 (2011)
R. Cutler, M. Turk, View-Based Interpretation of Real-Time Optical Flow for Gesture Recognition (1998), p. 416
J.L. Barron, D.J. Fleet, S.S. Beauchemin, Performance of optical flow techniques. Int. J. Comput. Vis. 12, 43–77 (1994)
C. Thurau, V. Hlaváč, Pose primitive based human action recognition in videos or still images, in IEEE Conference on Computer Vision and Pattern Recognition (2008), pp. 1–8
Q. Pu, S. Gupta, S. Gollakota, S. Patel, Whole-home gesture recognition using wireless signals, in Proceedings of the 19th Annual International Conference on Mobile Computing & Networking (2013), pp. 27–38
R. Ronfard, C. Schmid, B. Triggs, Computer Vision (2002), pp. 700–714
S.-J. Lee, C.-S. Ouyang, S.-H. Du, A neuro-fuzzy approach for segmentation of human objects in image sequences. Syst. Man Cybern. Part B Cybern. IEEE Trans. 33, 420–437 (2003)
D. Tang, H.J. Chang, A. Tejani, T.-K. Kim, Latent regression forest: structured estimation of 3D articulated hand posture, in IEEE Conference on Computer Vision and Pattern Recognition (2014), pp. 3786–3793
J. Taylor, J. Shotton, T. Sharp, A. Fitzgibbon, The vitruvian manifold: inferring dense correspondences for one-shot human pose estimation, in IEEE Conference on Computer Vision and Pattern Recognition (2012), pp. 103–110
J. Han, L. Shao, D. Xu, J. Shotton, Enhanced computer vision with microsoft kinect sensor: a review. Cybern. IEEE Trans. 43, 1318–1334 (2013)
Y. Li, Hand gesture recognition using Kinect, in IEEE 3rd International Conference on Software Engineering and Service Science (2012), pp. 196–199
D. Comaniciu, V. Ramesh, P. Meer, Real-time tracking of non-rigid objects using mean shift. IEEE Conf. Comput. Vis. Pattern Recognit. 2, 142–149 (2000)
S. Thrun, W. Burgard, D. Fox, Probabilistic Robotics (MIT Press, 2005)
R.E. Kalman, A new approach to linear filtering and prediction problems. J. Fluids Eng. 82, 35–45 (1960)
S. Haykin, Kalman Filtering and Neural Networks, vol. 47 (Wiley, 2004)
E. Wan, R. Van Der Merwe, The unscented Kalman filter for nonlinear estimation, in IEEE Adaptive Systems for Signal Processing, Communications, and Control Symposium (2000), pp. 153–158
K. Okuma, A. Taleghani, N. De Freitas, J.J. Little, D.G. Lowe, Computer Vision (Springer, 2004), pp. 28–39
S. Oron, A. Bar-Hillel, D. Levi, S. Avidan, Locally orderless tracking, in IEEE Conference on Computer Vision and Pattern Recognition 1940–1947 (2012)
J. Kwon, K.M. Lee, Tracking by sampling trackers, in IEEE International Conference on Computer Vision (2011), pp. 1195–1202
J. Kwon, K.M. Lee, F.C. Park, Visual tracking via geometric particle filtering on the affine group with optimal importance functions, in IEEE Conference on Computer Vision and Pattern Recognition (2009), pp. 991–998
R. Gao, L. Wang, R. Teti, D. Dornfeld, S. Kumara, M. Mori, M. Helu, Cloud-enabled prognosis for manufacturing. CIRP Ann. Technol. 64(2), 749–772 (2015)
T. Li, S. Sun, T.P. Sattar, J.M. Corchado, Fight sample degeneracy and impoverishment in particle filters: a review of intelligent approaches. Expert Syst. Appl. 41, 3944–3954 (2014)
T. Li, T.P. Sattar, S. Sun, Deterministic resampling: unbiased sampling to avoid sample impoverishment in particle filters. Sig. Process. 92, 1637–1645 (2012)
Rincón J.M. Del, D. Makris, C.O. Uruňuela, J.-C. Nebel, Tracking human position and lower body parts using Kalman and particle filters constrained by human biomechanics. Syst. Man. Cybern. Part B Cybern. IEEE Trans. 41, 26–37 (2011)
D.A. Ross, J. Lim, R.-S. Lin, M.-H. Yang, Incremental learning for robust visual tracking. Int. J. Comput. Vis. 77, 125–141 (2008)
Z. Kalal, J. Matas, K. Mikolajczyk, Pn learning: bootstrapping binary classifiers by structural constraints, in IEEE Conference on Computer Vision and Pattern Recognition (2010), pp. 49–56
B. Babenko, M.-H. Yang, S. Belongie, Visual tracking with online multiple instance learning, in IEEE Conference on Computer Vision and Pattern Recognition (2009), pp. 983–990
A.W.M. Smeulders et al., Visual tracking: an experimental survey. Pattern Anal. Mach. Intell. IEEE Trans. 36, 1442–1468 (2014)
L.E. Peterson, K-nearest neighbor. Scholarpedia 4, 1883 (2009)
A.D. Wilson, A.F. Bobick, Parametric hidden markov models for gesture recognition. Pattern Anal. Mach. Intell. IEEE Trans. 21, 884–900 (1999)
S. Lu, J. Picone, S. Kong, Fingerspelling Alphabet Recognition Using A Two-level Hidden Markov Modeli in Proceedings of the International Conference on Image Processing, Computer Vision, and Pattern Recognition (2013), p. 1
J. McCormick, K. Vincs, S. Nahavandi, D. Creighton, S. Hutchison, Teaching a digital performing agent: artificial neural network and hidden Markov model for recognising and performing dance movement, in Proceedings of the 2014 International Workshop on Movement and Computing (2014), p. 70
S.-Z. Yu, Hidden semi-Markov models. Artif. Intell. 174, 215–243 (2010)
M.A. Hearst, S.T. Dumais, E. Osman, J. Platt, B. Scholkopf, Support vector machines. IEEE Intell. Syst. their Appl. 13, 18–28 (1998)
M.E. Tipping, Sparse Bayesian learning and the relevance vector machine. J. Mach. Learn. Res. 1, 211–244 (2001)
B. Schiilkopf, The kernel trick for distances, in Proceedings of the 2000 Conference on Advances in Neural Information Processing Systems, vol. 13 (2001), p. 301
A. Cenedese, G.A. Susto, G. Belgioioso, G.I. Cirillo, F. Fraccaroli, Home automation oriented gesture classification from inertial measurements. Autom. Sci. Eng. IEEE Trans. 12, 1200–1210 (2015)
K. Feng, F. Yuan, Static hand gesture recognition based on HOG characters and support vector machines, in 2nd International Symposium on Instrumentation and Measurement, Sensor Network and Automation (2013), pp. 936–938
D. Ghimire, J. Lee, Geometric feature-based facial expression recognition in image sequences using multi-class adaboost and support vector machines. Sensors 13, 7714–7734 (2013)
O. Patsadu, C. Nukoolkit, B. Watanapa, Human gesture recognition using Kinect camera, in International Joint Conference on Computer Science and Software Engineering (2012), pp. 28–32
R.E. Schapire, Nonlinear estimation and classification (Springer, 2003), pp. 149–171
Y. Freund, R.E. Schapire, A decision-theoretic generalization of on-line learning and an application to boosting. J. Comput. Syst. Sci. 55, 119–139 (1997)
S. Celebi, A.S. Aydin, T.T. Temiz, T. Arici, Gesture recognition using skeleton data with weighted dynamic time warping. VISAPP 1, 620–625 (2013)
E.J. Keogh, M.J. Pazzani, Derivative dynamic time warping. SDM 1, 5–7 (2001)
S.S. Haykin, Neural Networks and Learning Machines, vol. 3 (Pearson Education Upper Saddle River, 2009)
T.H.H. Maung, Real-time hand tracking and gesture recognition system using neural networks. World Acad. Sci. Eng. Technol. 50, 466–470 (2009)
H. Hasan, S. Abdul-Kareem, Static hand gesture recognition using neural networks. Artif. Intell. Rev. 41, 147–181 (2014)
T. D’Orazio, G. Attolico, G. Cicirelli, C. Guaragnella, A neural network approach for human gesture recognition with a kinect sensor. ICPRAM 741–746 (2014)
A.H. El-Baz, A.S. Tolba, An efficient algorithm for 3D hand gesture recognition using combined neural classifiers. Neural Comput. Appl. 22, 1477–1484 (2013)
K. Subramanian, S. Suresh, Human action recognition using meta-cognitive neuro-fuzzy inference system. Int. J. Neural Syst. 22, 1250028 (2012)
Z.-H. Zhou, J. Wu, W. Tang, Ensembling neural networks: many could be better than all. Artif. Intell. 137, 239–263 (2002)
Y. LeCun, Y. Bengio, G. Hinton, Deep learning. Nature 521, 436–444 (2015)
J. Schmidhuber, Deep learning in neural networks: an overview. Neural Netw 61, 85–117 (2015)
J. Tompson, M. Stein, Y. Lecun, K. Perlin, Real-time continuous pose recovery of human hands using convolutional networks. ACM Trans. Graph 33, 169 (2014)
K. Simonyan, A. Zisserman, Two-stream convolutional networks for action recognition in videos. Adv. Neural Inf. Process. Syst. 568–576 (2014)
J. Nagi et al., Max-pooling convolutional neural networks for vision-based hand gesture recognition, in IEEE International Conference on Signal and Image Processing Applications (2011), pp. 342–347
A. Jain, J. Tompson, Y. LeCun, C. Bregler, Computer Vision (2014), pp. 302–315
K. Li, Y. Fu, Prediction of human activity by discovering temporal sequence patterns. IEEE Trans. Pattern Anal. Mach. Intell. 36, 1644–1657 (2014)
M.S. Ryoo, Human activity prediction: early recognition of ongoing activities from streaming videos, in IEEE International Conference on Computer Vision (2011), pp. 1036–1043
W. Ding, K. Liu, F. Cheng, J. Zhang, Learning hierarchical spatio-temporal pattern for human activity prediction. J. Vis. Commun. Image Represent. 35, 103–111 (2016)
L.R. Rabiner, A tutorial on hidden Markov models and selected applications in speech recognition. IEEE Proc. 77, 257–286 (1989)
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
Copyright information
© 2018 Springer International Publishing AG
About this chapter
Cite this chapter
Wang, L., Wang, X.V. (2018). Context-Aware Human-Robot Collaborative Assembly. In: Cloud-Based Cyber-Physical Systems in Manufacturing . Springer, Cham. https://doi.org/10.1007/978-3-319-67693-7_11
Download citation
DOI: https://doi.org/10.1007/978-3-319-67693-7_11
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-67692-0
Online ISBN: 978-3-319-67693-7
eBook Packages: EngineeringEngineering (R0)