Journal of Intelligent & Robotic Systems

, Volume 83, Issue 1, pp 85–103 | Cite as

Performance Evaluation of Human Detection Systems for Robot Safety

  • William Shackleford
  • Geraldine Cheok
  • Tsai Hong
  • Kamel Saidi
  • Michael ShneierEmail author


Detecting and tracking people is becoming more important in robotic applications because of the increasing demand for collaborative work in which people interact closely with and in the same workspace as robots. New safety standards allow people to work next to robots, but require that they be protected from harm while they do so. Sensors that detect and track people are a natural way of implementing the necessary safety monitoring, and have the added advantage that the information about where the people are and where they are going can be fed back into the application and used to give the robot greater situational awareness for performing tasks. The results should help users determine if such a system will provide sufficient protection for people to be able to work safely in collaborative applications with industrial robots.


Human detection Human-robot collaboration Human tracking Performance evaluation Performance metrics Robot safety 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    ISO TC 184: ISO 10218-2:2011 Robots and robotic devices – Safety requirements for industrial robots – Part 2: Robot systems and integration. In. (2011)Google Scholar
  2. 2.
    ISO TC 184: ISO 10218-1:2011 Robots and robotic devices – Safety requirements for industrial robots – Part 1: Robots. In. (2011)Google Scholar
  3. 3.
    American National Standards Institute, Robotics Industries Association: ANSI/RIA R15.06-2012, Industrial Robots and Robot Systems - Safety Requirements. In. (2012)Google Scholar
  4. 4.
    Shneier, M., Hong, T., Cheok, G., Saidi, K., Shackleford, W.: Performance evaluation methods for human detection and tracking systems for robotic applications. In., vol. NISTIR 8045. National Institute of Standards and Technology, Gaithersburg (2015)Google Scholar
  5. 5.
    Ogale, N.A.: A survey of techniques for human detection from video. University of Maryland (2006)Google Scholar
  6. 6.
    Ferryman, J., Crowley, J.L.: IEEE International Workshop on Performance Evaluation of Tracking and Surveillance. In: Ferryman, J., Crowley, J.L. (eds.) (2013)Google Scholar
  7. 7.
    Nascimento, J.C., Marques, J.S.: Performance evaluation of object detection algorithms for video surveillance. IEEE Trans. Multimedia 8(4), 761–774 (2006). doi: 10.1109/TMM.2006.876287 CrossRefGoogle Scholar
  8. 8.
    Stiefelhagen, R., Garofolo, J.: Multimodal Technologies for Perception of Humans, First International Evaluation Workshop on Classification of Events, Activities and Relationships, CLEAR 2006, Lecture Notes in Computer Science, vol. 4122. Springer, Southampton (2007)Google Scholar
  9. 9.
    Nghiem, A.T., Bremond, F., Thonnat, M., Valentin, V.: ETISEO, performance evaluation for video surveillance systems. In: IEEE Conference on Advanced Video and Signal Based Surveillance, 2007. AVSS 2007, pp. 476–481 (2007)Google Scholar
  10. 10.
    Home Office Centre for Applied Science and Technology: Imagery Library for Intelligent Detection Systems (i-LIDS): The i-LIDS User Guide. In. (2011)Google Scholar
  11. 11.
    British Home Office: Imagery Library for Intelligent Detection Systems. (2013). Accessed 30 May 2014
  12. 12.
    Brown, L.M., Senior, A.W., Tian, Y., Connell, J., Hampapur, A., Shu, C., Merkl, H., Lu, M.: Performance evaluation of surveillance systems under varying conditions. Paper presented at the IEEE international workshop on performance evaluation of tracking and surveillance, ColoradoGoogle Scholar
  13. 13.
    Yin, F., Makris, D., Velastin, S.: Performance evaluation of object tracking algorithms. Paper presented at the 10th IEEE international workshop on performance evaluation of tracking and surveillance (PETS 2007), Rio de JaneiroGoogle Scholar
  14. 14.
    Lazarevic-McManus, N., Renno, J.R., Makris, D., Jones, G.A.: An object-based comparative methodology for motion detection based on the F-Measure. Comput. Vis. Image Underst. 111(1), 74–85 (2008). doi: 10.1016/j.cviu.2007.07.007 CrossRefGoogle Scholar
  15. 15.
    Bashir, F., Porikli, F.: Performance evaluation of object detection and tracking systems. Paper presented at the 9th IEEE international workshop on performance evaluation of tracking and surveillance (PETS 2006), New YorkGoogle Scholar
  16. 16.
    Black, J., Ellis, T., Rosin, P.: A novel method for video tracking performance evaluation. Paper presented at the joint IEEE international workshop on visual surveillance and performance evaluation of tracking and surveillance, Nice, 2003Google Scholar
  17. 17.
    Davis, J., Goadrich, M.: The relationship between Precision-Recall and ROC curves. Paper presented at the Proceedings of the 23rd international conference on machine learning, Pittsburgh, 2006Google Scholar
  18. 18.
    Kalal, Z., Matas, J., Mikolajczyk, K.: Online learning of robust object detectors during unstable tracking. In: 2009 IEEE 12th International Conference on Computer Vision Workshops (ICCV Workshops), pp. 1417–1424 (2009)Google Scholar
  19. 19.
    Popoola, J., Amer, A.: Performance evaluation for tracking algorithms using object labels. In: IEEE International Conference on Acoustics, Speech and Signal Processing, 2008. ICASSP 2008, pp. 733–736 (2008)Google Scholar
  20. 20.
    Bernardin, K., Elbs, A., Stiefelhagen, R.: Multiple object tracking performance metrics and evaluation in a smart room environment. Paper presented at the The 6th IEEE international workshop on visual surveillance, VS 2006, GrazGoogle Scholar
  21. 21.
    Moeslund, T.B., Hilton, A., Krüger, V.: A survey of advances in vision-based human motion capture and analysis. Comput. Vis. Image Underst. 104(2), 90–126 (2006). doi: 10.1016/j.cviu.2006.08.002 CrossRefGoogle Scholar
  22. 22.
    Baumann, A., Bolz, M., Ebling, J., Koenig, M., Loos, H.S., Merkel, M., Niem, W., KarlWarzelhan, J., Yu, J.: A review and comparison of measures for automatic video surveillance systems. EURASIP Journal on Image and Video Processing 2008 (2008). doi: 10.1155/2008/824726
  23. 23.
    Stiefelhagen, R., Bowers, R., Fiscus, J.G.: Multimodal Technologies for Perception of Humans, International Evaluation Workshops CLEAR 2007 and RT 2007, Baltimore, MD, USA, May 8–11, 2007, Revised Selected Papers. Lecture Notes in Computer Science. Springer, Berlin Heidelberg (2008)Google Scholar
  24. 24.
    Bodt, B., Camden, R., Scott, H., Jacoff, A., Hong, T., Chang, T., Norcross, R., Downs, T., Virts, A.: Performance measurements for evaluating static and dynamic multiple human detection and tracking systems in unstructured environments. Paper presented at the Proceedings of the 9th workshop on performance metrics for intelligent systems, Gaithersburg, Maryland, 2009Google Scholar
  25. 25.
    Godil, A., Bostelman, R., Saidi, K., Shackleford, W., Cheok, G., Shneier, M., Hong, T.: 3D ground-truth systems for object/human recognition and tracking. In: 2013 IEEE Conference on 2013 Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 719–726. IEEEGoogle Scholar
  26. 26.
  27. 27.
    Schmitt, R., Nisch, S., Schönberg, A., Demeester, F., Renders, S.: Performance evaluation of iGPS for industrial applications. In: 2010 International Conference on Indoor Positioning and Indoor Navigation (IPIN), pp. 1–8 (2010)Google Scholar
  28. 28.
    Mosqueira, G., Apetz, J., Santos, K.M., Villani, E., Suterio, R., Trabasso, L.G.: Analysis of the indoor GPS system as feedback for the robotic alignment of fuselages using laser radar measurements as comparison. Robot. Comput.-Integr. Manuf. 28(6), 700–709 (2012). doi: 10.1016/j.rcim.2012.03.004 CrossRefGoogle Scholar
  29. 29.
    Wang, Z., Mastrogiacomo, L., Franceschini, F., Maropoulos, P.G.: Experimental comparison of dynamic tracking performance of iGPS and laser tracker. Int. J. Adv. Manuf. Technol. 56(1–4), 205–213 (2011). doi: 10.1007/s00170-011-3166-0 CrossRefGoogle Scholar
  30. 30.
    Depenthal, C.: Path tracking with IGPS. In: International Conference on Indoor Positioning and Indoor Navigation (IPIN), pp. 1–6 (2010)Google Scholar
  31. 31.
    Chambers, D.R., Flannigan, C., Wheeler, B.: High-accuracy real-time pedestrian detection system using 2D and 3D features. SPIE Proceedings Three-Dimensional Imaging, Visualization, and Display, vol. 83840G, pp. 83840G-83841–83840G-83811 (2012)Google Scholar
  32. 32.
    Dalal, N.: The INRIA Person Dataset. (2005)
  33. 33.
    Papageorgiou, C.P., Oren, M., Poggio, T.: A general framework for object detection. In: 6th International Conference on Computer Vision, pp. 555–562 (1998)Google Scholar
  34. 34.
    Freund, Y., Schapire, R.E.: A short introduction to boosting. Journal of Japanese Society for Artificial Intelligence 14(5), 771–780 (1999)Google Scholar
  35. 35.
    Cortes, C., Vapnik, V.: Support-vector networks. Mach. Learn. 20(3), 273–297 (1995). doi: 10.1007/BF00994018 zbMATHGoogle Scholar
  36. 36.
    Dalal, N., Triggs, B.: Histograms of oriented gradients for human detection. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition, CVPR, pp. 886–893 (2005)Google Scholar
  37. 37.
    Shackleford, W.: Position tracking performance metrics. (2014)
  38. 38.
    ISO TC 199: ISO 13855:2010 Safety of machinery – Positioning of safeguards with respect to the approach speeds of parts of the human body. In: International Organization for Standardization (ed.) (2010)Google Scholar
  39. 39.
    Bodt, B., Hong, T.: UGV safe operations capstone experiment. Paper presented at the Army science conference, Orlando, 2010Google Scholar

Copyright information

© Springer Science+Business Media Dordrecht (outside the USA) 2016

Authors and Affiliations

  1. 1.Intelligent Systems DivisionNational Institute of Standards and TechnologyGaithersburgUSA

Personalised recommendations