Skip to main content
SpringerLink
Log in

Human Detection and Identification by Robots Using Thermal and Visual Information in Domestic Environments

  • Published:
Journal of Intelligent & Robotic Systems Aims and scope Submit manuscript

Abstract

In this paper a robust system for enabling robots to detect and identify humans in domestic environments is proposed. Robust human detection is achieved through the use of thermal and visual information sources that are integrated to detect human-candidate objects, which are further processed in order to verify the presence of humans and their identity using face information in the thermal and visual spectrums. Face detection is used to verify the presence of humans, and face recognition to identify them. Active vision mechanisms are employed in order to improve the relative pose of a candidate object/person in case direct identification is not possible. The response of the different modules is characterized, and the proposed system is validated using image databases of real domestic environments, and human detection and identification benchmarks of the RoboCup@Home research community.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Similar content being viewed by others

References

  1. Sinha, P., Balas, B., Ostrovsky, Y., Russell, R.: Face recognition by humans: 19 results all computer vision researchers should know about. Proc. of the IEEE. 94(11), 1948–1962 (2006)

    Article  Google Scholar 

  2. Hermosilla, G., Loncomilla, P., Ruiz-del-Solar, J.: Thermal face recognition using local interest points and descriptors for HRI applications. Lect. Notes Comput. Sci. (RoboCup Symposium 2010) (2010, in press)

  3. Hermosilla, G., Ruiz-del-Solar, J., Verschae, R., Correa, M.: Face recognition using thermal infrared images for human-robot interaction applications: a comparative study. In: 6th IEEE Latin American Robotics Symposium – LARS 2009, Valparaíso, Chile (CD Proceedings), 29–30 Oct. 2009

  4. Ruiz-del-Solar, J., Verschae, R., Correa, M.: Recognition of faces in unconstrained environments: a comparative study. EURASIP Journal on Advances in Signal Processing (Recent Advances in Biometric Systems: A Signal Processing Perspective), vol. 2009, Article ID 184617, p. 19 (2009)

  5. Kong, S., Heo, J., Abidi, B., Paik, J., Abidi, M.: Recent advances in visual and infrared face recognition - a review. J. Comput. Vis. Image Understanding 97(1), 103–135 (2005)

    Article  Google Scholar 

  6. Meis, U., Oberlander, M., Ritter, W.: Reinforcing the reliability of pedestrian detection in far-infrared sensing. 2004 IEEE Intelligent Vehicles Symposium, pp. 779–783, 14–17 June 2004

  7. Binelli, E., Broggi, A., Fascioli, A., Ghidoni, S., Grisleri, P., Graf, T., Meinecke, M.: A modular tracking system for far infrared pedestrian recognition. 2005 IEEE Intelligent Vehicles Symposium, pp. 759–764, 6–8 June 2005

  8. Mudaly, S.S.: Novel computer-based infrared pedestrian data-acquisition system. Electron. Lett. 15(13), 371–372 (1979)

    Article  Google Scholar 

  9. Nanda, H., Davis, L.: Probabilistic template based pedestrian detection in infrared videos. IEEE Intell. Veh. Symposium 1, 15–20 (2002)

    Google Scholar 

  10. Bertozzi, M., Broggi, A., Fascioli, A., Graf, T., Meinecke, M.-M.: Pedestrian detection for driver assistance using multiresolution infrared vision. IEEE Trans. Veh. Technol. 53(6), 1666–1678 (2004)

    Article  Google Scholar 

  11. Wu, S.-Q, Song, W., Jiang, L.-J., Xie, S.-L., Pan, F., Yau, W.-Y., Ranganath, S.: Infrared face recognition by using blood perfusion data. Lect. Notes Comput. Sci. 3546, 527–531 (2005)

    Google Scholar 

  12. Wilder, J., Phillips, P.J., Jiang, C., Wiener, S.: Comparison of visible and infra-red imagery for face recognition. In: Proc. of the 2nd Int. Conf. on Automatic Face and Gesture Recognition, pp.182–187, 14–16 Oct 1996

  13. Li, S., Chu, R., Liao, Sh., Zhang, L.: Illumination invariant face recognition using near-infrared images. IEEE Trans. Pattern Anal. Mach. Intell. 29(4), 627–639 (2007)

    Article  Google Scholar 

  14. Li, S., Chu, R., Ao, M., Zhang, L., He, R.: Highly accurate and fast face recognition using near infrared images. Lect. Notes Comput. Sci. 3832, 151–158 (2005)

    Article  Google Scholar 

  15. Verschae, R., Ruiz-del-Solar, J., Correa, M.: A unified learning framework for object detection and classification using nested cascades of boosted classifiers. Mach. Vis. Appl. 19(2), 85–103 (2008)

    Article  Google Scholar 

  16. Correa, M., Ruiz-del-Solar, J., Bernuy, F.: Face recognition for human-robot interaction applications: a comparative study. Lect. Notes Comput. Sci.,(RoboCup Symposium 2008) 5399, 473–484 (2009)

    Article  Google Scholar 

  17. Zhao, W., Chellappa, R., Rosenfeld, A., Phillips, P.J.: Face recognition: a literature survey. ACM Comput. Surv. 35(4), 399–458 (2003)

    Article  Google Scholar 

  18. Tan, X., Chen, S., Zhou, Z.-H., Zhang, F.: Face recognition from a single image per person: a survey. Pattern Recogn. 39, 1725–1745 (2006)

    Article  MATH  Google Scholar 

  19. Chellappa, R., Wilson, C.L., Sirohey, S.: Human and machine recognition of faces: a survey. Proc. IEEE 83(5), 705–740 (1995)

    Article  Google Scholar 

  20. Face Recognition Homepage. Available in January 2008. http://www.face-rec.org/

  21. Turk, M., Pentland, A.: Eigenfaces for recognition. J. Cognitive Neurosicence 3(1), 71–86 (1991)

    Article  Google Scholar 

  22. Mendez, H., San Martin, C., Kittler, J., Plasencia, Y., Garcia, E.: Face recognition with LWIR imagery using local binary patterns. In: Proceedings ICB2009 (2009)

  23. Ruiz-del-Solar, J., Verschae, R.: Robust skin segmentation using neighborhood information. In: The Eleventh International Conference on Image Processing (ICIP 2004), 24–27 October 2004, pp. 207–210. IEEE Press, Singapore (2004)

    Google Scholar 

  24. Viola, P., Jones, M.: Rapid object detection using a boosted cascade of simple features. Proc. Conf. Comput. Vis. Patt. Recogn. 1, 511–518 (2001)

    Google Scholar 

  25. Jones, M., Rehg, J.M.: Statistical color models with application to skin detection. Int. J. Comput. Vis. 46(1), 81–96 (2002)

    Article  MATH  Google Scholar 

  26. Hjelmas, E., Kee Low, B.: Face detection: a survey. Comput. Vis. Image Understanding, 83(3), 236–274 (2001)

    Article  MATH  Google Scholar 

  27. Satake, J., Miura, J.: Robust Stereo-based Person Detecting and Tracking for a Person Following Robot. In: Proc. ICRA 2009 workshop on person detection and tracking. Kobe, Japan (2009)

    Google Scholar 

  28. Wilhelm, T., Böhme, H.-J., Gross, H.-M.: A multi-modal system for tracking and analyzing faces on a mobile robot. Robot. Auton. Syst. 48(1), 31–40, (2004), European Conference on Mobile Robots (ECMR ’03)

    Article  Google Scholar 

  29. Medionia, G., R.J. Françoisa A., Siddiquia, M., Kima, K., Yoonb, H.: Robust real-time vision for a personal service robot. Comput. Vis. Image Understanding 108(1–2), 196–203, Special Issue on Vision for Human-Computer Interaction, October–November 2007

    Article  Google Scholar 

  30. Li, L., Koh, Y.T., Ge, S.S., Huang, W.: Stereo-based human detection for mobile service robots. Control, Automation, Robotics and Vision Conference, 2004, ICARCV, vol. 1, pp. 74–79 (2004)

  31. Bellotto, N., Hu, H.: Multisensor-based human detection and tracking for mobile service robots systems, man, and cybernetics, Part B: cybernetics. IEEE Trans. 39(1), 167–181 (2009)

    Google Scholar 

  32. Böhme, H.J., Wilhelma, T., Keya, J., Schauera, C., Schrötera, C., Großa, H-M., Hempelb, T.: An approach to multi-modal human–machine interaction for intelligent service robots. Robot. Auton. Syst. 44(1), 83–96 (2003)

    Article  Google Scholar 

  33. Ahonen, T., Hadid, A., Pietikainen, M.: Face description with local binary patterns: application to face recognition. IEEE Trans. Pattern Anal. Mach. Intell. 28(12), 2037–2041 (2006)

    Article  Google Scholar 

  34. Wisspeintner, T., van der Zant, T., Iocchi, L., Schiffer, S.: RoboCup Home: scientific competition and benchmarking for domestic service robots. Interaction Studies 10(3), 392–426(35) (2009)

    Article  Google Scholar 

  35. iRobot Official Website. Available on Dec. 2010. http://store.irobot.com/home/index.jsp

  36. RoboCup Home Official Website. Available on Dec. 2010. http://www.ai.rug.nl/robocupathome/

  37. FLIR TAU 320 thermal camera. Information available on Dec. 2010. http://www.flir.com/cvs/cores/uncooled/products/tau/

  38. PMD Technologies website: http://www.pmdtec.com/. Accessed Dec 2010

  39. Carmen Robot Navigation Toolkit website: http://carmen.sourceforge.net/. Accessed Dec 2010

  40. Ruiz-del-Solar, J., Correa, M., Lee-Ferng, J., Hevia-Koch, P., Parra, I., Mascaró, M.: UChile HomeBreakers 2010 Team Description Paper. RoboCup Symposium 2010, 19-25 June 2010. Singapore (CD Proceedings)

  41. Munder, S., Gavrila, D.M.: An experimental study on pedestrian classification. IEEE Trans. Pattern Anal. Mach. Intell. 28, 1863–1868 (2006)

    Article  Google Scholar 

  42. Gavrila, D.M., Munder, S.: Multi-cue pedestrian detection and tracking from a moving vehicle. Int. J. Comput. Vis. 73, 41–59 (2007)

    Article  Google Scholar 

  43. Enzweiler, M., Gavrila, D.M.: Monocular pedestrian detection: survey and experiments. Pattern Anal. Mach. Intell., IEEE Trans. 31(12), 2179–2195 (2009)

    Article  Google Scholar 

  44. Felzenszwalb, P.F., Girshick, R.B., Mcallester, D.: Cascade object detection with deformable part models. In: Proc. of IEEE Int’l Conference on Computer Vision and Pattern Recognition (2010)

  45. Dalal, N., Triggs, B.: Histograms of oriented gradients for human detection. In: CVPR, vol. 1, pp. 886–893 (2005)

  46. Viola, P.A., Jones, M.J., Snow, D.: Detecting pedestrians using patterns of motion and appearance. IJCV, 63(2), 153–161 (2005)

    Article  Google Scholar 

  47. Tuzel, O., Porikli, F., Meer, P.: Pedestrian detection via classification on riemannian manifolds. IEEE T-PAMI, 30(10), 1713–1727 (2008)

    Article  Google Scholar 

  48. Paisitkriangkrai, S., Shen, C., Zhang, J.: Fast pedestrian detection using a cascade of boosted covariance features. Circuits Syst. Video Technol., IEEE Trans. 18(8), 1140–1151 (2008)

    Article  Google Scholar 

  49. Zhu, Q., Yeh, M.-C., Cheng, K.-T., Avidan, S.: Fast human detection using a cascade of histograms of oriented gradients. In: Computer Vision and Pattern Recognition, IEEE Computer Society Conference, vol. 2, pp. 1491–1498 (2006)

  50. Wojek, C., Schiele, B.: A performance evaluation of single and multi-feature people detection. In: DAGM Symp. on Patt Rec, pp. 82–91 (2008)

  51. Tao, J., Odobez, J.-M.: Fast human detection from videos using covariance features. In: Workshop on VS at ECCV (2008)

  52. Dalal, N., Triggs, B., Schmid C.: Human detection using oriented histograms of flow and appearance. In: ECCV (2), pp. 428–441 (2006)

  53. Dollar, P., Wojek, C., Schiele, B., Perona, P.: Pedestrian detection: a benchmark. In: CVPR, pp. 304–311 (2009)

Download references

Author information

Authors and Affiliations

  1. Department of Electrical Engineering, Universidad de Chile, Av. Tupper 2007, Santiago, Chile

    Mauricio Correa, Gabriel Hermosilla, Rodrigo Verschae & Javier Ruiz-del-Solar

  2. Advanced Mining Technology Center, Universidad de Chile, Av. Tupper 2007, Santiago, Chile

    Mauricio Correa, Gabriel Hermosilla, Rodrigo Verschae & Javier Ruiz-del-Solar

Authors
  1. Mauricio Correa
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Gabriel Hermosilla
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Rodrigo Verschae
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Javier Ruiz-del-Solar
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Rodrigo Verschae.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Correa, M., Hermosilla, G., Verschae, R. et al. Human Detection and Identification by Robots Using Thermal and Visual Information in Domestic Environments. J Intell Robot Syst 66, 223–243 (2012). https://doi.org/10.1007/s10846-011-9612-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10846-011-9612-2

Keywords

Search

Navigation