Abstract
Human–robot collaboration is increasingly present not only in research environments, but also in industry and many contemporary day-to-day activities. There is a need for the automation of tasks ranging from the simplest to the most complex ones. The insertion of robotic arms provides a considerable step useful in achieving this goal. In this context, safety remains a concern, however. Among the most frequent issue in this collaboration context is human–robot collision. While focus has been on automation efficiency of the of the activities, there is a growing need to reduce or even prevent damage to the involved agents. As part of this goal, a new mechanism for detecting human–robot collisions is proposed in this article. It has been tested in a well-controlled scenario using equipment commonly present in collaborative scenarios for maintenance on a radio base station. The robot used is a UR5 robotic arm in addition to three 2D cameras and a network rack. In our experimental scenario, a person interacts with the network devices installed within the rack while conducting basic collaborative activities inserted in this context. For collision detection, deep learning models were used and evaluated. These were trained to detect overlap between humans and robots considering the view and perspectives from three different cameras. Finally, a new ensemble learning system is proposed in order to establish whether or not a collision took place. It receives as input the result of overlap detection through deep learning models. Results suggest that the proposed system is capable of detecting collisions with an average global accuracy of 89.81% of correctness in a well-controlled scenario. The effectiveness of the proposed ensemble is exposed in comparison with the use of only one of the cameras for decision-making. Finally, the proposed system is shown to detect collisions in real time and to achieve a low response time.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., Isard, M. & Kudlur, M. (2016). Tensorflow: a system for large-scale machine learning. In 12th USENIX symposium on operating systems design and implementation (OSDI 16), (pp. 265–283)
Ahuett-Garza, H., & Kurfess, T. (2018). A brief discussion on the trends of habilitating technologies for industry 4.0 and smart manufacturing. Manufacturing Letters, 15, 60–63.
Anvaripour, M., & Saif, M. (2019). Collision detection for human-robot interaction in an industrial setting using force myography and a deep learning approach. In 2019 IEEE international conference on systems, man and cybernetics (SMC), (pp. 2149–2154). IEEE.
Bauer, A., Wollherr, D., & Buss, M. (2008). Human-robot collaboration: a survey. International Journal of Humanoid Robotics, 5(01), 47–66.
e Souza, Ad. M., & Stemmer, M. R. (2018). Extraction and classification of human body parameters for gait analysis. Journal of Control, Automation and Electrical Systems, 29(5), 586–604.
Ebert, D.M., & Henrich, D.D. (2002). Safe human-robot-cooperation: image-based collision detection for industrial robots. In IEEE/RSJ international conference on intelligent robots and systems, (vol 2, pp. 1826–1831), Sep. 10.1109/IRDS.2002.1044021.
Fernandes, F., da Silveira Bueno, Rd. L., Cavalcanti, P. D., & Admasu, P. D. (2020). Generating stochastic processes through convolutional neural networks. Journal of Control, Automation and Electrical Systems, 31(2), 294–303.
Gecks, T., & Henrich, D.(2006). Multi-camera collision detection allowing for object occlusions. In 37th international symposium on robotics (ISR 2006)/4th German conference on robotics (Robotik 2006).
Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep learning. MIT Press.
Henrich, D., & Gecks, T. (2008). Multi-camera collision detection between known and unknown objects. In 2008 Second ACM/IEEE international conference on distributed smart cameras, (pp. 1–10). IEEE.
Kingma, D.P., & Ba, J.(2015). Adam: a method for stochastic optimization. In 3rd international conference on learning representations, ICLR 2015.
Kittler, J., Hatef, M., Duin, R. P., & Matas, J. (1998). On combining classifiers. IEEE Transactions on Pattern Analysis and Machine Intelligence, 20(3), 226–239.
LeCun, Y., & Ranzato, M. (2013). Deep learning tutorial. In Tutorials in international conference on machine learning (ICML’13), (pp. 1–29). Citeseer.
LeCun, Y., Bengio, Y., & Hinton, G. (2015). Deep learning. Nature, 521(7553), 436–444.
Microsoft. (2019). microsoft dynamics 365 manufacturing trends report.
Min, E., Guo, X., Liu, Q., Zhang, G., Cui, J., & Long, J. (2018). A survey of clustering with deep learning: from the perspective of network architecture. IEEE Access, 6, 39501–39514.
Mohammed, A., Schmidt, B., & Wang, L. (2017). Active collision avoidance for human-robot collaboration driven by vision sensors. International Journal of Computer Integrated Manufacturing, 30(9), 970–980.
Reis, G., Dantas, M., Bezerra, D., Nunes, G., Dreyer, P., Ledebour, C., Kelner, J., Sadok, D., Souza, R., Lins, S., et al. (2021). Gripper design for radio base station autonomous maintenance system. International Journal of Automation and Computing, 18, 1–9.
Sharkawy, A.-N., & Aspragathos, N. (2018). Human-robot collision detection based on neural networks. Int. J. Mech. Eng. Robot. Res, 7(2), 150–157.
Sharkawy, A.-N., Koustoumpardis, P. N., & Aspragathos,N. A. (2018). Manipulator collision detection and collided link identification based on neural networks. In International conference on robotics in Alpe-Adria Danube region, (pp. 3–12). Springer.
Sharkawy, A.-N., Koustoumpardis, P. N., & Aspragathos, N. (2020). Human-robot collisions detection for safe human-robot interaction using one multi-input-output neural network. Soft Computing, 24(9), 6687–6719.
Silva, I. J., Junior, C. O. V., Costa, A. H. R., & Bianchi, R. A. (2021). Toward robotic cognition by means of decision tree of deep neural networks applied in a humanoid robot. Journal of Control, Automation and Electrical Systems, 1, 1–11.
Silva, I. R., Barbosa, G. B., Ledebour, C. C., Oliveira Filho, A. T., Kelner, J., Sadok, D., Lins, S., & Souza, R. (2020). Assessing deep learning models for human-robot collaboration collision detection in industrial environments. In Brazilian conference on intelligent systems, (pp. 240–255). Springer.
Simonyan, K., & Zisserman, A. (2015). Very deep convolutional networks for large-scale image recognition. In 3rd international conference on learning representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, conference track proceedings.
Takiguchi, T., Lee, J. H., & Okamoto, S. (2018). Collision avoidance algorithm using deep learning type artificial intelligence for a mobile robot. In Proceedings of the international multiconference of engineers and computer scientists, vol 1.
Van Nguyen, T., Thai, N. H., Pham, H. T., Phan, T. A., Nguyen, L., Le, H. X., & Nguyen, H. D. (2019). Adaptive neural network-based backstepping sliding mode control approach for dual-arm robots. Journal of Control, Automation and Electrical Systems, 30(4), 512–521.
Vasic, M., & Billard, A. (2013). Safety issues in human-robot interactions. In 2013 ieee international conference on robotics and automation, (pp. 197–204). IEEE.
Wang, Y., Ye, X., Yang, Y., & Zhang, W. (2017). Hand movement prediction based collision-free human-robot interaction. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, (pp. 20–21).
Zou, Q., Xie, S., Lin, Z., Wu, M., & Ju, Y. (2016). Finding the best classification threshold in imbalanced classification. Big Data Research, 5, 2–8.
Acknowledgements
This work was financed in part by the Conselho Nacional de Desenvolvimento Científico e Tecnológico (CNPq); the Fundação de Amparo a Ciência e Tecnologia de Pernambuco (FACEPE); the Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (CAPES); and the Research, Development and Innovation Center, Ericsson Telecommunications Inc., Brazil.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that they have no conflict of interest.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Rodrigues, I.R., Barbosa, G., Oliveira Filho, A. et al. A New Mechanism for Collision Detection in Human–Robot Collaboration using Deep Learning Techniques. J Control Autom Electr Syst 33, 406–418 (2022). https://doi.org/10.1007/s40313-021-00829-3
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s40313-021-00829-3