Skip to main content
SpringerLink
Log in

Performance Evaluation of Human Detection Systems for Robot Safety

  • Published:
Journal of Intelligent & Robotic Systems Aims and scope Submit manuscript

Abstract

Detecting and tracking people is becoming more important in robotic applications because of the increasing demand for collaborative work in which people interact closely with and in the same workspace as robots. New safety standards allow people to work next to robots, but require that they be protected from harm while they do so. Sensors that detect and track people are a natural way of implementing the necessary safety monitoring, and have the added advantage that the information about where the people are and where they are going can be fed back into the application and used to give the robot greater situational awareness for performing tasks. The results should help users determine if such a system will provide sufficient protection for people to be able to work safely in collaborative applications with industrial robots.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. ISO TC 184: ISO 10218-2:2011 Robots and robotic devices – Safety requirements for industrial robots – Part 2: Robot systems and integration. In. (2011)

  2. ISO TC 184: ISO 10218-1:2011 Robots and robotic devices – Safety requirements for industrial robots – Part 1: Robots. In. (2011)

  3. American National Standards Institute, Robotics Industries Association: ANSI/RIA R15.06-2012, Industrial Robots and Robot Systems - Safety Requirements. In. (2012)

  4. Shneier, M., Hong, T., Cheok, G., Saidi, K., Shackleford, W.: Performance evaluation methods for human detection and tracking systems for robotic applications. In., vol. NISTIR 8045. National Institute of Standards and Technology, Gaithersburg (2015)

  5. Ogale, N.A.: A survey of techniques for human detection from video. University of Maryland (2006)

  6. Ferryman, J., Crowley, J.L.: IEEE International Workshop on Performance Evaluation of Tracking and Surveillance. In: Ferryman, J., Crowley, J.L. (eds.) (2013)

  7. Nascimento, J.C., Marques, J.S.: Performance evaluation of object detection algorithms for video surveillance. IEEE Trans. Multimedia 8(4), 761–774 (2006). doi:10.1109/TMM.2006.876287

    Article  Google Scholar 

  8. Stiefelhagen, R., Garofolo, J.: Multimodal Technologies for Perception of Humans, First International Evaluation Workshop on Classification of Events, Activities and Relationships, CLEAR 2006, Lecture Notes in Computer Science, vol. 4122. Springer, Southampton (2007)

    Google Scholar 

  9. Nghiem, A.T., Bremond, F., Thonnat, M., Valentin, V.: ETISEO, performance evaluation for video surveillance systems. In: IEEE Conference on Advanced Video and Signal Based Surveillance, 2007. AVSS 2007, pp. 476–481 (2007)

  10. Home Office Centre for Applied Science and Technology: Imagery Library for Intelligent Detection Systems (i-LIDS): The i-LIDS User Guide. In. (2011)

  11. British Home Office: Imagery Library for Intelligent Detection Systems. https://www.gov.uk/imagery-library-for-intelligent-detection-systems (2013). Accessed 30 May 2014

  12. Brown, L.M., Senior, A.W., Tian, Y., Connell, J., Hampapur, A., Shu, C., Merkl, H., Lu, M.: Performance evaluation of surveillance systems under varying conditions. Paper presented at the IEEE international workshop on performance evaluation of tracking and surveillance, Colorado

  13. Yin, F., Makris, D., Velastin, S.: Performance evaluation of object tracking algorithms. Paper presented at the 10th IEEE international workshop on performance evaluation of tracking and surveillance (PETS 2007), Rio de Janeiro

  14. Lazarevic-McManus, N., Renno, J.R., Makris, D., Jones, G.A.: An object-based comparative methodology for motion detection based on the F-Measure. Comput. Vis. Image Underst. 111(1), 74–85 (2008). doi:10.1016/j.cviu.2007.07.007

    Article  Google Scholar 

  15. Bashir, F., Porikli, F.: Performance evaluation of object detection and tracking systems. Paper presented at the 9th IEEE international workshop on performance evaluation of tracking and surveillance (PETS 2006), New York

  16. Black, J., Ellis, T., Rosin, P.: A novel method for video tracking performance evaluation. Paper presented at the joint IEEE international workshop on visual surveillance and performance evaluation of tracking and surveillance, Nice, 2003

  17. Davis, J., Goadrich, M.: The relationship between Precision-Recall and ROC curves. Paper presented at the Proceedings of the 23rd international conference on machine learning, Pittsburgh, 2006

  18. Kalal, Z., Matas, J., Mikolajczyk, K.: Online learning of robust object detectors during unstable tracking. In: 2009 IEEE 12th International Conference on Computer Vision Workshops (ICCV Workshops), pp. 1417–1424 (2009)

  19. Popoola, J., Amer, A.: Performance evaluation for tracking algorithms using object labels. In: IEEE International Conference on Acoustics, Speech and Signal Processing, 2008. ICASSP 2008, pp. 733–736 (2008)

  20. Bernardin, K., Elbs, A., Stiefelhagen, R.: Multiple object tracking performance metrics and evaluation in a smart room environment. Paper presented at the The 6th IEEE international workshop on visual surveillance, VS 2006, Graz

  21. Moeslund, T.B., Hilton, A., Krüger, V.: A survey of advances in vision-based human motion capture and analysis. Comput. Vis. Image Underst. 104(2), 90–126 (2006). doi:10.1016/j.cviu.2006.08.002

    Article  Google Scholar 

  22. Baumann, A., Bolz, M., Ebling, J., Koenig, M., Loos, H.S., Merkel, M., Niem, W., KarlWarzelhan, J., Yu, J.: A review and comparison of measures for automatic video surveillance systems. EURASIP Journal on Image and Video Processing 2008 (2008). doi:10.1155/2008/824726

  23. Stiefelhagen, R., Bowers, R., Fiscus, J.G.: Multimodal Technologies for Perception of Humans, International Evaluation Workshops CLEAR 2007 and RT 2007, Baltimore, MD, USA, May 8–11, 2007, Revised Selected Papers. Lecture Notes in Computer Science. Springer, Berlin Heidelberg (2008)

  24. Bodt, B., Camden, R., Scott, H., Jacoff, A., Hong, T., Chang, T., Norcross, R., Downs, T., Virts, A.: Performance measurements for evaluating static and dynamic multiple human detection and tracking systems in unstructured environments. Paper presented at the Proceedings of the 9th workshop on performance metrics for intelligent systems, Gaithersburg, Maryland, 2009

  25. Godil, A., Bostelman, R., Saidi, K., Shackleford, W., Cheok, G., Shneier, M., Hong, T.: 3D ground-truth systems for object/human recognition and tracking. In: 2013 IEEE Conference on 2013 Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 719–726. IEEE

  26. Nikon Corporation: iGPS/iSpace. http://www.nikon.com/products/instruments/lineup/industrial/metrology_assisted_production/large_volume/igps/index.htm. Accessed 2 June 2014

  27. Schmitt, R., Nisch, S., Schönberg, A., Demeester, F., Renders, S.: Performance evaluation of iGPS for industrial applications. In: 2010 International Conference on Indoor Positioning and Indoor Navigation (IPIN), pp. 1–8 (2010)

  28. Mosqueira, G., Apetz, J., Santos, K.M., Villani, E., Suterio, R., Trabasso, L.G.: Analysis of the indoor GPS system as feedback for the robotic alignment of fuselages using laser radar measurements as comparison. Robot. Comput.-Integr. Manuf. 28(6), 700–709 (2012). doi:10.1016/j.rcim.2012.03.004

    Article  Google Scholar 

  29. Wang, Z., Mastrogiacomo, L., Franceschini, F., Maropoulos, P.G.: Experimental comparison of dynamic tracking performance of iGPS and laser tracker. Int. J. Adv. Manuf. Technol. 56(1–4), 205–213 (2011). doi:10.1007/s00170-011-3166-0

    Article  Google Scholar 

  30. Depenthal, C.: Path tracking with IGPS. In: International Conference on Indoor Positioning and Indoor Navigation (IPIN), pp. 1–6 (2010)

  31. Chambers, D.R., Flannigan, C., Wheeler, B.: High-accuracy real-time pedestrian detection system using 2D and 3D features. SPIE Proceedings Three-Dimensional Imaging, Visualization, and Display, vol. 83840G, pp. 83840G-83841–83840G-83811 (2012)

  32. Dalal, N.: The INRIA Person Dataset. http://pascal.inrialpes.fr/data/human/ (2005)

  33. Papageorgiou, C.P., Oren, M., Poggio, T.: A general framework for object detection. In: 6th International Conference on Computer Vision, pp. 555–562 (1998)

  34. Freund, Y., Schapire, R.E.: A short introduction to boosting. Journal of Japanese Society for Artificial Intelligence 14(5), 771–780 (1999)

    Google Scholar 

  35. Cortes, C., Vapnik, V.: Support-vector networks. Mach. Learn. 20(3), 273–297 (1995). doi:10.1007/BF00994018

    MATH  Google Scholar 

  36. Dalal, N., Triggs, B.: Histograms of oriented gradients for human detection. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition, CVPR, pp. 886–893 (2005)

  37. Shackleford, W.: Position tracking performance metrics. https://github.com/usnistgov/PTPM (2014)

  38. ISO TC 199: ISO 13855:2010 Safety of machinery – Positioning of safeguards with respect to the approach speeds of parts of the human body. In: International Organization for Standardization (ed.) (2010)

  39. Bodt, B., Hong, T.: UGV safe operations capstone experiment. Paper presented at the Army science conference, Orlando, 2010

Download references

Author information

Authors and Affiliations

  1. Intelligent Systems Division, National Institute of Standards and Technology, Gaithersburg, MD, USA

    William Shackleford, Geraldine Cheok, Tsai Hong, Kamel Saidi & Michael Shneier

Authors
  1. William Shackleford
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Geraldine Cheok
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Tsai Hong
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Kamel Saidi
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Michael Shneier
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Michael Shneier.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Shackleford, W., Cheok, G., Hong, T. et al. Performance Evaluation of Human Detection Systems for Robot Safety. J Intell Robot Syst 83, 85–103 (2016). https://doi.org/10.1007/s10846-016-0334-3

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10846-016-0334-3

Keywords

Search

Navigation