Detecting Humans in 2D Thermal Images by Generating 3D Models

  • Stefan Markov
  • Andreas Birk
Part of the Lecture Notes in Computer Science book series (LNCS, volume 4667)


There are two significant challenges to standard approaches to detect humans through computer vision. First, scenarios when the poses and postures of the humans are completely unpredictable. Second, situations when there are many occlusions, i.e., only parts of the body are visible. Here a novel approach to perception is presented where a complete 3D scene model is learned on the fly to represent a 2D snapshot. In doing so, an evolutionary algorithm generates pieces of 3D code that are rendered and the resulting images are compared to the current camera picture via an image similarity function. Based on the feedback of this fitness function, a crude but very fast online evolution generates an approximate 3D model of the environment where non-human objects are represented by boxes. The key point is that 3D models of humans are available as code sniplets to the EA, which can use them to represent human shapes or portions of them if they are in the image. Results from experiments with real world data from a search and rescue application using a thermal camera are presented.


Hill Climbing Thermal Camera Rescue Robot Roulette Selection Robocup Rescue 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. Abouaf, J.: Trial by fire: teleoperated robot targets chernobyl. Computer Graphics and Applications 18(4), 10–14 (1998)CrossRefGoogle Scholar
  2. Aoyama, H., Himoto, A., Fuchiwaki, O., Misaki, D., Sumrall, T.: Micro hopping robot with ir sensor for disaster survivor detection. In: SSRR. IEEE International Workshop on Safety, Security and Rescue Robotics, pp. 189–194. IEEE Computer Society Press, Los Alamitos (2005)Google Scholar
  3. Agarwal, A., Triggs, B.: Learning to track 3d human motion from silhouettes. In: Twenty-first international conference on Machine learning, ACM Press, New York (2004)Google Scholar
  4. Birk, A., Carpin, S.: Rescue robotics - a crucial milestone on the road to autonomous systems. Advanced Robotics Journal 20(5), 595–695 (2006)CrossRefGoogle Scholar
  5. Bahadori, S., Iocchi, L., Nardi, D., Settembre, G.P.: Stereo vision based human body detection from a localized mobile robot. In: IEEE Conference on Advanced Video and Signal Based Surveillance, pp. 499–504. IEEE Computer Society Press, Los Alamitos (2005)CrossRefGoogle Scholar
  6. Birk, A.: Learning geometric concepts with an evolutionary algorithm. In: Proc. of The Fifth Annual Conference on Evolutionary Programming, The MIT Press, Cambridge (1996)Google Scholar
  7. Birk, A., Paul, W.J.: Schemas and genetic programming. In: Intern. Conf. on the Integration of Elementary Functions into Complex Behavior (1994)Google Scholar
  8. Birk, A., Markov, S., Delchev, I., Pathak, K.: Autonomous rescue operations on the iub rugbot. In: SSRR. IEEE International Workshop on Safety, Security, and Rescue Robotics, IEEE Press, Los Alamitos (2006)Google Scholar
  9. Birk, A., Pathak, K., Schwertfeger, S., Chonnaparamutt, W.: The iub rugbot: an intelligent, rugged mobile robot for search and rescue operations. In: SSRR. IEEE International Workshop on Safety, Security, and Rescue Robotics, IEEE Press, Los Alamitos (2006)Google Scholar
  10. Burion, S.: Human detection for robotic urban search and rescue masters thesis. Technical report, Institute de Production Robotique (IPR) (1998)Google Scholar
  11. Cucchiara, R., Grana, C., Prati, A., Vezzani, R.: Computer vision system for in-house video surveillance. Vision, Image and Signal Processing, IEE Proceedings- 152(2), 242–249 (2005)CrossRefGoogle Scholar
  12. Davids, A.: Urban search and rescue robots: from tragedy to technology. Intelligent Systems 17(2), 81–83 (2002)Google Scholar
  13. Hao, Q., Brady, D.J., Guenther, B.D., Burchett, J., Shankar, M., Feller, S.: Human tracking with wireless distributed pyroelectric sensors. IEEE Sensors Journal 6(6), 1683–1696 (2006)CrossRefGoogle Scholar
  14. Hogg, D.: Model-based vision: A program to see a walking person. Image and Vision Computing 1(1), 5–20 (1983)CrossRefGoogle Scholar
  15. Jacoff, A., Messina, E., Weiss, B., Tadokoro, S., Nakagawa, Y.: Test arenas and performance metrics for urban search and rescue robots. In: IROS. Proceedings of the Intelligent and Robotic Systems Conference (2003) Google Scholar
  16. Jacoff, A., Weiss, B., Messina, E.: Evolution of a performance metric for urban search and rescue. In: PERMIS. Performance Metrics for Intelligent Systems, Gaithersburg, MD (2003)Google Scholar
  17. Kakadiaris, L., Metaxas, D.: Model-based estimation of 3d human motion. IEEE Transactions on Pattern Analysis and Machine Intelligence 22, 1453–1459 (2000)CrossRefGoogle Scholar
  18. Kitano, H., Tadokoro, S.: Robocup rescue. a grand challenge for multiagent and intelligent systems. AI Magazine 22(1), 39–52 (2001)Google Scholar
  19. Murphy, R., Casper, J., Micire, M.: Potential tasks and research issues for mobile robots in robocup rescue. In: Stone, P., Balch, T., Kraetzschmar, G.K. (eds.) RoboCup 2000. LNCS (LNAI), vol. 2019, pp. 339–334. Springer, Heidelberg (2001)Google Scholar
  20. Mohan, A., Papageorgiou, C., Poggio, T.: Example-based object detection in images by components. Pattern Analysis and Machine Intelligence, IEEE Transactions on 23(4), 349–361 (2001)CrossRefGoogle Scholar
  21. Murphy, R.R.: Trial by fire. IEEE Robotics and Automation Magazine 11(3), 50–61 (2004)CrossRefGoogle Scholar
  22. Oren, M., Papageorgiou, C., Sinhaand, P., Osuna, E., Poggio, T.: Pedestrian detection using wavelet templates. In: Proc. Computer Vision and Pattern Recognition, June 1997, pp. 193–199 (1997)Google Scholar
  23. Park, S.H., Jung, K., Hea, J.K., Kim, H.J.: Vision-based traffic surveillance system on the internet. In: Computational Intelligence and Multimedia Applications. In: ICCIMA. Third International Conference on, pp. 201–205 (1999)Google Scholar
  24. Papageorgiou, C., Poggio, T.: A trainable system for object detection. Int’l J. Computer Vision 38(1), 15–33 (2000)zbMATHCrossRefGoogle Scholar
  25. Paul, W.J., Solomonoff, R.: Autonomous theory building systems. Annals of Operations Research 55(1), 179–193 (1995)zbMATHCrossRefGoogle Scholar
  26. Paletta, L., Wiesenhofer, S., Brandle, N., Sidla, O., Lypetskyy, Y.: Visual surveillance system for monitoring of passenger flows at public transportation junctions. Intelligent Transportation Systems, 862–867 (2005) Google Scholar
  27. Shah, B., Choset, H.: Survey on urban search and rescue robots. Journal of the Robotics Society of Japan (JRSJ) 22(5), 40–44 (2004)Google Scholar
  28. Forrest, S.: Genetic algorithms - principles of natural selection applied to computation. Science 261, 872–878 (1993)CrossRefGoogle Scholar
  29. Sinha, P.: Object recognition via image invariants: A case study. Investigative Ophthalmology and Visual Science 35, 1735–1740 (1994)Google Scholar
  30. Snyder, R.G.: Robots assist in search and rescue efforts at wtc. IEEE Robotics and Automation Magazine 8(4), 26–28 (2001)Google Scholar
  31. Sparacino, F.: In: Inter-face body boundaries, issue editor emanuele quinz, anomalie, n.2, paris, france, anomos (2001),
  32. Scholtz, J., Young, J., Drury, J., Yanco, H.: Evaluation of human-robot interaction awareness in search and rescue. In: Proceedings of the International Conference on Robotics and Automation, ICRA’2004, pp. 2327–2332. IEEE Press, Los Alamitos (2004)Google Scholar
  33. Thrun, S., Bennewitz, M., Burgard, W., Cremers, A.B., Dellaert, F., Fox, D., Hahnel, D., Rosenberg, C., Roy, N., Schulte, J., Schulz, D.: Minerva: A second-generation museum tour-guide robot. In: Proceedings of the International Conference on Robotics and Automation (ICRA) (1999) Google Scholar
  34. Teo, A.W., Garg, H.K., Puthusserypady, S.: Detection of humans buried in rubble: an electronic nose to detect human body odor. In: Engineering in Medicine and Biology, 24th Annual Conference and the Annual Fall Meeting of the Biomedical Engineering Society, EMBS/BMES, vol. 3, pp. 1811–1812 (2002)Google Scholar
  35. Valentine, D., Abdi, H., O’Toole, A.J., Cottrell, G.W.: Connectionist models of face processing: a survey. Pattern Recognition 27 (1994)Google Scholar
  36. Wren, C.R., Azarbayejani, A., Darrell, T., Pentland, A.: Pfinder: Real-time tracking of the human body. IEEE Transactions on Pattern Analysis and Machine Intelligence 19(7), 780–785 (1997)CrossRefGoogle Scholar
  37. Yow, K., Cipolla, R.: Feature-based human face detection. Image and Vision Computing 15(9), 713–735 (1997)CrossRefGoogle Scholar
  38. Yang, M.-H., Kriegman, D.J., Ahuja, N.: Detecting faces in images: a survey. Pattern Analysis and Machine Intelligence, IEEE Transactions on 24(1), 34–58 (2002)CrossRefGoogle Scholar
  39. Yang, J., Stiefelhagen, R., Meier, U., Waibel, A.: Real-time face and facial feature tracking and applications. In: Proceedings of Auditory-Visual Speech Processing Conference, pp. 79–84. Terrigal, South Wales, Australia (1998)Google Scholar
  40. Yuille, A.: Deformable templates for face recognition. J. Cognitive Neuroscience 3(1), 59–70 (1991)CrossRefGoogle Scholar
  41. Zhao, W., Chellappa, R., Phillips, P.J., Rosenfeld, A.: Face recognition: A literature survey. ACM Comput. Surv. 35(4), 399–458 (2003)CrossRefGoogle Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2007

Authors and Affiliations

  • Stefan Markov
    • 1
  • Andreas Birk
    • 1
  1. 1.School of Engineering and Science, Jacobs University Bremen, Campus Ring 1, D-28759 BremenGermany

Personalised recommendations