Skip to main content

Advertisement

SpringerLink
Log in

Dynamic Speed and Separation Monitoring Based on Scene Semantic Information

  • Short Paper
  • Published:
Journal of Intelligent & Robotic Systems Aims and scope Submit manuscript

Abstract

Human-robot collaboration (HRC) based on speed and separation monitoring should consider the difference of risk factors in the scene; otherwise, the sudden invasion of non-operators or routine operation of the operator may stop the robot system. In this paper, we propose a sensing network based on the fusion of multi-information to obtain scene semantic information and employ it to realize risk assessment. However, due to the influence of light on the image information sensed by RGB cameras, it is not easy to obtain accurate scene semantic information. We apply a depth camera and a thermal imager to obtain depth and infrared information to enhance the RGB images. We build a risk information database and use it to quantify the obtained scene semantic information into risk factors. The dynamic change of risk factors judges whether the distance between humans and robots is safe. The experimental results verify that the algorithm of intelligent human-robot monitoring can realize the analysis of dangerous situations and control the robot system, thereby reducing the number of false shutdowns and improving safety.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Ajoudani, A., Zanchettin, A.M., Ivaldi, S., Albu-Schäffer, A., Kosuge, K., Khatib, O.: Progress and prospects of the human---robot collaboration. Auton. Robot. 42(5), 957–975 (2018)

    Article  Google Scholar 

  2. Bdiwi, M.: Integrated sensors system for human safety during cooperating with industrial robots for handing-over and assembling tasks - ScienceDirect. Proc. CIRP. 23, 65–70 (2014)

    Article  Google Scholar 

  3. Matthias, B., Reisinger, T.: Example Application of ISO/TS 15066 to a Collaborative Assembly Scenario. In: Isr: International Symposium on Robotics (2016)

    Google Scholar 

  4. Marvel, J.A.: Performance metrics of speed and separation monitoring in shared workspaces. IEEE Trans. Autom. Sci. Eng. 10(2), 405–414 (2013)

    Article  Google Scholar 

  5. Zanchettin, A.M., Ceriani, N.M., Rocco, P., Ding, H., Matthias, B.: Safety in human-robot collaborative manufacturing environments: metrics and control. IEEE Trans. Autom. Sci. Eng. 13(2), 882–893 (2015)

    Article  Google Scholar 

  6. Shin, H., K. Seo, and S. Rhim. Allowable Maximum Safe Velocity Control Based on Human-Robot Distance for Collaborative Robot. 2018

    Book  Google Scholar 

  7. Byner, C., Matthias, B., Ding, H.: Dynamic speed and separation monitoring for collaborative robot applications - Concepts and performance. Robot. Comput. Integr. Manuf. 58(AUG.), 239–252 (2019)

    Article  Google Scholar 

  8. Cai, K., Wang, C., Song, S., Chen, H., Meng, M.Q.H.: Risk-aware path planning under uncertainty in dynamic environments. J. Intell. Robot. Syst. 101(3), 47 (2021)

    Article  Google Scholar 

  9. Tarbouriech, S., Suleiman, W.: Bi-objective motion planning approach for safe motions: application to a collaborative robot. J. Intell. Robot. Syst. 99(1), 45–63 (2020)

    Article  Google Scholar 

  10. Chen, J.H., Song, K.T.: Collision-free motion planning for human-robot collaborative safety under Cartesian constraint. In: 2018 IEEE international conference on robotics and automation (ICRA) (2018)

    Google Scholar 

  11. Wei, Q., Zha, D., Jie, Z.: An effective approach for causal variables analysis in diesel engine production by using mutual information and network deconvolution. J. Intell. Manuf. 9, 1–11 (2018)

    Google Scholar 

  12. Marvel, J.A., Falco, J., Marstio, I.: Characterizing task-based human–robot collaboration safety in manufacturing. IEEE Trans. Syst. Man Cybern. Syst. 45(2), 260–275 (2015)

    Article  Google Scholar 

  13. Lucci, N., Lacevic, B., Zanchettin, A.M., Rocco, P.: Combining speed and separation monitoring with power and force limiting for safe collaborative robotics applications. IEEE Robot. Autom. Lett. 5(4), 6121–6128 (2020)

    Article  Google Scholar 

  14. Kim, E., Kirschner, R., Yamada, Y., Okamoto, S.: Estimating probability of human hand intrusion for speed and separation monitoring using interference theory. Robot. Comput. Integr. Manuf. 61, 101819.1–101819.7 (2020)

    Article  Google Scholar 

  15. Kumar, S., Arora, S., Sahin, F.: Speed and Separation Monitoring using on-robot Time--of--Flight laser--ranging sensor arrays. In: 2019 IEEE 15th International Conference on Automation Science and Engineering (CASE) (2019)

    Google Scholar 

  16. Mazhar, O., Navarro, B., Ramdani, S., Passama, R., Cherubini, A.: A real-time human-robot interaction framework with robust background invariant hand gesture detection. Robot. Comput. Integr. Manuf. 60, 34–48 (2019)

    Article  Google Scholar 

  17. Aliev, K., Antonelli, D.: Proposal of a monitoring system for collaborative robots to predict outages and to assess reliability factors exploiting machine learning. Appl. Sci. 11(4), 1621 (2021)

    Article  Google Scholar 

  18. Wu, Q., Ding, K., Huang, B.: Approach for fault prognosis using recurrent neural network. J. Intell. Manuf. 31(7), 1621–1633 (2020)

    Article  Google Scholar 

  19. Sun, Y., Zuo, W., Liu, M.: RTFNet: RGB-thermal fusion network for semantic segmentation of urban scenes. IEEE Robot. Autom. Lett. 4(3), 2576–2583 (2019)

    Article  Google Scholar 

  20. Qi, C.R., et al.: PointNet: deep learning on point sets for 3D classification and segmentation. In: 2017 IEEE conference on computer vision and pattern recognition (CVPR) (2017)

    Google Scholar 

  21. Lecun, Y., Bengio, Y., Hinton, G.: Deep learning. Nature. 521(7553), 436–444 (2015)

    Article  Google Scholar 

  22. Haddadin, S., Luca, A., Albu-Schäffer, A.: Robot collisions: a survey on detection, isolation, and identification. IEEE Trans. Robot. 33(6), 1292–1312 (2017)

    Article  Google Scholar 

  23. Albini, A., Cannata, G.: Pressure distribution classification and segmentation of human hands in contact with the robot body. Int. J. Robot. Res. 39(6), 668–687 (2020)

    Article  Google Scholar 

  24. ArkinJacob, et al.: Multimodal estimation and communication of latent semantic knowledge for robust execution of robot instructions. Int. J. Robot. Res. 39(10–11), 1279–1304 (2020)

    Google Scholar 

  25. Zhou, T., Wachs, J.P.: Spiking neural networks for early prediction in human–robot collaboration. Int. J. Robot. Res. 38(14), 1619–1643 (2019)

    Article  Google Scholar 

  26. Bouvrie, J.: Notes on Convolutional Neural Networks. neural nets (2006)

    Google Scholar 

  27. He, K., et al.: Mask R-CNN. In: 2017 IEEE International Conference on Computer Vision (ICCV) (2017)

    Google Scholar 

  28. Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: towards real-time object detection with region proposal networks. IEEE Trans. Pattern Anal. Mach. Intell. 39(6), 1137–1149 (2017)

    Article  Google Scholar 

  29. The American Society of Safety Engineers: ANSI Z10, Occupational health and safety management systems. The American National Standards Institute (2012)

Download references

Acknowledgement

Research was supported by the National Key Research and Development Program of China under Grant 2019YFB1310200.

Code Availability

All data and materials as well as software application or custom code support our published claims and comply with field standards.

Funding

Partial financial support was received from the National Key Research and Development Program of China under Grant 2019YFB1310200.

Author information

Author notes

  1. Botao Yang and ShuXin Xie contributed equally to this work.

Authors and Affiliations

  1. Jiangsu Provincial Key Laboratory of Advanced Robotics, School of Mechanical and Electrical Engineering, Soochow University, Suzhou, 215123, China

    Botao Yang, ShuXin Xie, Guodong Chen, Zihao Ding & Zhenhua Wang

Authors
  1. Botao Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. ShuXin Xie
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Guodong Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Zihao Ding
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Zhenhua Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

All authors contributed to the study conception and design. Material preparation, data collection and analysis were performed by Botao Yang, Shuxin Xie, Guodong Chen, Zihao Ding, and Zhenhua Wang. The first draft of the manuscript was written by Botao Yang and all authors commented on previous versions of the manuscript. All authors read and approved the final manuscript. Botao Yang and Shuxin Xie are contributes equally to this work.

Corresponding author

Correspondence to Guodong Chen.

Ethics declarations

Conflict of Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Ethics Approval

Approval was obtained from the ethics committee of Soochow University.

Consent to Participate

Informed consent was obtained from all individual participants included in the study.

Consent for Publication

The participant has consented to the submission of the research manuscript to the journ.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yang, B., Xie, S., Chen, G. et al. Dynamic Speed and Separation Monitoring Based on Scene Semantic Information. J Intell Robot Syst 106, 35 (2022). https://doi.org/10.1007/s10846-022-01607-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10846-022-01607-2

Keywords

Search

Navigation