Journal of Intelligent & Robotic Systems

, Volume 66, Issue 1–2, pp 223–243 | Cite as

Human Detection and Identification by Robots Using Thermal and Visual Information in Domestic Environments

  • Mauricio Correa
  • Gabriel Hermosilla
  • Rodrigo VerschaeEmail author
  • Javier Ruiz-del-Solar


In this paper a robust system for enabling robots to detect and identify humans in domestic environments is proposed. Robust human detection is achieved through the use of thermal and visual information sources that are integrated to detect human-candidate objects, which are further processed in order to verify the presence of humans and their identity using face information in the thermal and visual spectrums. Face detection is used to verify the presence of humans, and face recognition to identify them. Active vision mechanisms are employed in order to improve the relative pose of a candidate object/person in case direct identification is not possible. The response of the different modules is characterized, and the proposed system is validated using image databases of real domestic environments, and human detection and identification benchmarks of the RoboCup@Home research community.


Human detection Human identification Service robot Thermal image 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Sinha, P., Balas, B., Ostrovsky, Y., Russell, R.: Face recognition by humans: 19 results all computer vision researchers should know about. Proc. of the IEEE. 94(11), 1948–1962 (2006)CrossRefGoogle Scholar
  2. 2.
    Hermosilla, G., Loncomilla, P., Ruiz-del-Solar, J.: Thermal face recognition using local interest points and descriptors for HRI applications. Lect. Notes Comput. Sci. (RoboCup Symposium 2010) (2010, in press)Google Scholar
  3. 3.
    Hermosilla, G., Ruiz-del-Solar, J., Verschae, R., Correa, M.: Face recognition using thermal infrared images for human-robot interaction applications: a comparative study. In: 6th IEEE Latin American Robotics Symposium – LARS 2009, Valparaíso, Chile (CD Proceedings), 29–30 Oct. 2009Google Scholar
  4. 4.
    Ruiz-del-Solar, J., Verschae, R., Correa, M.: Recognition of faces in unconstrained environments: a comparative study. EURASIP Journal on Advances in Signal Processing (Recent Advances in Biometric Systems: A Signal Processing Perspective), vol. 2009, Article ID 184617, p. 19 (2009)Google Scholar
  5. 5.
    Kong, S., Heo, J., Abidi, B., Paik, J., Abidi, M.: Recent advances in visual and infrared face recognition - a review. J. Comput. Vis. Image Understanding 97(1), 103–135 (2005)CrossRefGoogle Scholar
  6. 6.
    Meis, U., Oberlander, M., Ritter, W.: Reinforcing the reliability of pedestrian detection in far-infrared sensing. 2004 IEEE Intelligent Vehicles Symposium, pp. 779–783, 14–17 June 2004Google Scholar
  7. 7.
    Binelli, E., Broggi, A., Fascioli, A., Ghidoni, S., Grisleri, P., Graf, T., Meinecke, M.: A modular tracking system for far infrared pedestrian recognition. 2005 IEEE Intelligent Vehicles Symposium, pp. 759–764, 6–8 June 2005Google Scholar
  8. 8.
    Mudaly, S.S.: Novel computer-based infrared pedestrian data-acquisition system. Electron. Lett. 15(13), 371–372 (1979)CrossRefGoogle Scholar
  9. 9.
    Nanda, H., Davis, L.: Probabilistic template based pedestrian detection in infrared videos. IEEE Intell. Veh. Symposium 1, 15–20 (2002)Google Scholar
  10. 10.
    Bertozzi, M., Broggi, A., Fascioli, A., Graf, T., Meinecke, M.-M.: Pedestrian detection for driver assistance using multiresolution infrared vision. IEEE Trans. Veh. Technol. 53(6), 1666–1678 (2004)CrossRefGoogle Scholar
  11. 11.
    Wu, S.-Q, Song, W., Jiang, L.-J., Xie, S.-L., Pan, F., Yau, W.-Y., Ranganath, S.: Infrared face recognition by using blood perfusion data. Lect. Notes Comput. Sci. 3546, 527–531 (2005)Google Scholar
  12. 12.
    Wilder, J., Phillips, P.J., Jiang, C., Wiener, S.: Comparison of visible and infra-red imagery for face recognition. In: Proc. of the 2nd Int. Conf. on Automatic Face and Gesture Recognition, pp.182–187, 14–16 Oct 1996Google Scholar
  13. 13.
    Li, S., Chu, R., Liao, Sh., Zhang, L.: Illumination invariant face recognition using near-infrared images. IEEE Trans. Pattern Anal. Mach. Intell. 29(4), 627–639 (2007)CrossRefGoogle Scholar
  14. 14.
    Li, S., Chu, R., Ao, M., Zhang, L., He, R.: Highly accurate and fast face recognition using near infrared images. Lect. Notes Comput. Sci. 3832, 151–158 (2005)CrossRefGoogle Scholar
  15. 15.
    Verschae, R., Ruiz-del-Solar, J., Correa, M.: A unified learning framework for object detection and classification using nested cascades of boosted classifiers. Mach. Vis. Appl. 19(2), 85–103 (2008)CrossRefGoogle Scholar
  16. 16.
    Correa, M., Ruiz-del-Solar, J., Bernuy, F.: Face recognition for human-robot interaction applications: a comparative study. Lect. Notes Comput. Sci.,(RoboCup Symposium 2008) 5399, 473–484 (2009)CrossRefGoogle Scholar
  17. 17.
    Zhao, W., Chellappa, R., Rosenfeld, A., Phillips, P.J.: Face recognition: a literature survey. ACM Comput. Surv. 35(4), 399–458 (2003)CrossRefGoogle Scholar
  18. 18.
    Tan, X., Chen, S., Zhou, Z.-H., Zhang, F.: Face recognition from a single image per person: a survey. Pattern Recogn. 39, 1725–1745 (2006)zbMATHCrossRefGoogle Scholar
  19. 19.
    Chellappa, R., Wilson, C.L., Sirohey, S.: Human and machine recognition of faces: a survey. Proc. IEEE 83(5), 705–740 (1995)CrossRefGoogle Scholar
  20. 20.
    Face Recognition Homepage. Available in January 2008.
  21. 21.
    Turk, M., Pentland, A.: Eigenfaces for recognition. J. Cognitive Neurosicence 3(1), 71–86 (1991)CrossRefGoogle Scholar
  22. 22.
    Mendez, H., San Martin, C., Kittler, J., Plasencia, Y., Garcia, E.: Face recognition with LWIR imagery using local binary patterns. In: Proceedings ICB2009 (2009)Google Scholar
  23. 23.
    Ruiz-del-Solar, J., Verschae, R.: Robust skin segmentation using neighborhood information. In: The Eleventh International Conference on Image Processing (ICIP 2004), 24–27 October 2004, pp. 207–210. IEEE Press, Singapore (2004)Google Scholar
  24. 24.
    Viola, P., Jones, M.: Rapid object detection using a boosted cascade of simple features. Proc. Conf. Comput. Vis. Patt. Recogn. 1, 511–518 (2001)Google Scholar
  25. 25.
    Jones, M., Rehg, J.M.: Statistical color models with application to skin detection. Int. J. Comput. Vis. 46(1), 81–96 (2002)zbMATHCrossRefGoogle Scholar
  26. 26.
    Hjelmas, E., Kee Low, B.: Face detection: a survey. Comput. Vis. Image Understanding, 83(3), 236–274 (2001)zbMATHCrossRefGoogle Scholar
  27. 27.
    Satake, J., Miura, J.: Robust Stereo-based Person Detecting and Tracking for a Person Following Robot. In: Proc. ICRA 2009 workshop on person detection and tracking. Kobe, Japan (2009)Google Scholar
  28. 28.
    Wilhelm, T., Böhme, H.-J., Gross, H.-M.: A multi-modal system for tracking and analyzing faces on a mobile robot. Robot. Auton. Syst. 48(1), 31–40, (2004), European Conference on Mobile Robots (ECMR ’03)CrossRefGoogle Scholar
  29. 29.
    Medionia, G., R.J. Françoisa A., Siddiquia, M., Kima, K., Yoonb, H.: Robust real-time vision for a personal service robot. Comput. Vis. Image Understanding 108(1–2), 196–203, Special Issue on Vision for Human-Computer Interaction, October–November 2007CrossRefGoogle Scholar
  30. 30.
    Li, L., Koh, Y.T., Ge, S.S., Huang, W.: Stereo-based human detection for mobile service robots. Control, Automation, Robotics and Vision Conference, 2004, ICARCV, vol. 1, pp. 74–79 (2004)Google Scholar
  31. 31.
    Bellotto, N., Hu, H.: Multisensor-based human detection and tracking for mobile service robots systems, man, and cybernetics, Part B: cybernetics. IEEE Trans. 39(1), 167–181 (2009)Google Scholar
  32. 32.
    Böhme, H.J., Wilhelma, T., Keya, J., Schauera, C., Schrötera, C., Großa, H-M., Hempelb, T.: An approach to multi-modal human–machine interaction for intelligent service robots. Robot. Auton. Syst. 44(1), 83–96 (2003)CrossRefGoogle Scholar
  33. 33.
    Ahonen, T., Hadid, A., Pietikainen, M.: Face description with local binary patterns: application to face recognition. IEEE Trans. Pattern Anal. Mach. Intell. 28(12), 2037–2041 (2006)CrossRefGoogle Scholar
  34. 34.
    Wisspeintner, T., van der Zant, T., Iocchi, L., Schiffer, S.: RoboCup Home: scientific competition and benchmarking for domestic service robots. Interaction Studies 10(3), 392–426(35) (2009)CrossRefGoogle Scholar
  35. 35.
    iRobot Official Website. Available on Dec. 2010.
  36. 36.
    RoboCup Home Official Website. Available on Dec. 2010.
  37. 37.
    FLIR TAU 320 thermal camera. Information available on Dec. 2010.
  38. 38.
    PMD Technologies website: Accessed Dec 2010
  39. 39.
    Carmen Robot Navigation Toolkit website: Accessed Dec 2010
  40. 40.
    Ruiz-del-Solar, J., Correa, M., Lee-Ferng, J., Hevia-Koch, P., Parra, I., Mascaró, M.: UChile HomeBreakers 2010 Team Description Paper. RoboCup Symposium 2010, 19-25 June 2010. Singapore (CD Proceedings)Google Scholar
  41. 41.
    Munder, S., Gavrila, D.M.: An experimental study on pedestrian classification. IEEE Trans. Pattern Anal. Mach. Intell. 28, 1863–1868 (2006)CrossRefGoogle Scholar
  42. 42.
    Gavrila, D.M., Munder, S.: Multi-cue pedestrian detection and tracking from a moving vehicle. Int. J. Comput. Vis. 73, 41–59 (2007)CrossRefGoogle Scholar
  43. 43.
    Enzweiler, M., Gavrila, D.M.: Monocular pedestrian detection: survey and experiments. Pattern Anal. Mach. Intell., IEEE Trans. 31(12), 2179–2195 (2009)CrossRefGoogle Scholar
  44. 44.
    Felzenszwalb, P.F., Girshick, R.B., Mcallester, D.: Cascade object detection with deformable part models. In: Proc. of IEEE Int’l Conference on Computer Vision and Pattern Recognition (2010)Google Scholar
  45. 45.
    Dalal, N., Triggs, B.: Histograms of oriented gradients for human detection. In: CVPR, vol. 1, pp. 886–893 (2005)Google Scholar
  46. 46.
    Viola, P.A., Jones, M.J., Snow, D.: Detecting pedestrians using patterns of motion and appearance. IJCV, 63(2), 153–161 (2005)CrossRefGoogle Scholar
  47. 47.
    Tuzel, O., Porikli, F., Meer, P.: Pedestrian detection via classification on riemannian manifolds. IEEE T-PAMI, 30(10), 1713–1727 (2008)CrossRefGoogle Scholar
  48. 48.
    Paisitkriangkrai, S., Shen, C., Zhang, J.: Fast pedestrian detection using a cascade of boosted covariance features. Circuits Syst. Video Technol., IEEE Trans. 18(8), 1140–1151 (2008)CrossRefGoogle Scholar
  49. 49.
    Zhu, Q., Yeh, M.-C., Cheng, K.-T., Avidan, S.: Fast human detection using a cascade of histograms of oriented gradients. In: Computer Vision and Pattern Recognition, IEEE Computer Society Conference, vol. 2, pp. 1491–1498 (2006)Google Scholar
  50. 50.
    Wojek, C., Schiele, B.: A performance evaluation of single and multi-feature people detection. In: DAGM Symp. on Patt Rec, pp. 82–91 (2008)Google Scholar
  51. 51.
    Tao, J., Odobez, J.-M.: Fast human detection from videos using covariance features. In: Workshop on VS at ECCV (2008)Google Scholar
  52. 52.
    Dalal, N., Triggs, B., Schmid C.: Human detection using oriented histograms of flow and appearance. In: ECCV (2), pp. 428–441 (2006)Google Scholar
  53. 53.
    Dollar, P., Wojek, C., Schiele, B., Perona, P.: Pedestrian detection: a benchmark. In: CVPR, pp. 304–311 (2009)Google Scholar

Copyright information

© Springer Science+Business Media B.V. 2011

Authors and Affiliations

  • Mauricio Correa
    • 1
    • 2
  • Gabriel Hermosilla
    • 1
    • 2
  • Rodrigo Verschae
    • 1
    • 2
    Email author
  • Javier Ruiz-del-Solar
    • 1
    • 2
  1. 1.Department of Electrical EngineeringUniversidad de ChileSantiagoChile
  2. 2.Advanced Mining Technology CenterUniversidad de ChileSantiagoChile

Personalised recommendations