Fire Technology

, Volume 51, Issue 4, pp 823–845 | Cite as

Firefighting Robot Stereo Infrared Vision and Radar Sensor Fusion for Imaging through Smoke

  • Jong-Hwan KimEmail author
  • Joseph W. Starr
  • Brian Y. Lattimer


Firefighting robots are actively being researched to reduce firefighter injuries and deaths as well as increase their effectiveness on performing tasks. There has been difficulty in making firefighting robots autonomous because the commonly used sensors for autonomous robot navigation do not perform well in fire smoke-filled environments where low visibility and high temperature are present. In order to overcome these limitations, a multi-spectral vision system was developed that uses sensor fusion between stereo thermal infrared (IR) vision and frequency modulated-continuous wave (FMCW) radar to locate objects through zero visibility smoke in real-time. In this system, the stereo IR vision was used to obtain 3-D information about the scene while the radar provided more accurate distances of objects in the field of view. Through globally matching radar objects with those in the 3-D image, the accuracy of the stereo IR vision map was updated removing the far-field inaccuracy of the stereo IR as well as ghost objects created due to stereo mismatch. The system was sufficiently fast to provide real-time matching of objects in the scene allowing for dynamic reaction object tracking and locating. Through large-scale fire experiments with and without smoke in the field of view, the distance error for the stereo IR vision was reduced from 1% to 19.0% to 1% to 2% due to sensor fusion of the stereo IR with FMCW radar.


Firefighting robot Object detection Stereo IR vision FMCW radar Multisensor fusion Dense smoke Fire environment 



This work was sponsored by the Office of Naval Research Grant No. N00014-11-1-0074 scientific office Dr. Thomas McKenna. The authors would also like to thank Rosana K. Lee for helping and supporting this research.


  1. 1.
    NFPA (2013) Deadliest Fires in the U.S. with 5 or more Firefighter Deaths at the Fire Grounds, 1977–2012: Fire Incident Data OrganizationGoogle Scholar
  2. 2.
    Karter MJ, Molis JL (2012) US Firefighter Injuries—2011: National Fire Protection Association. Fire Analysis and Research DivisionGoogle Scholar
  3. 3.
    Fahy RF, LeBlanc PR, Molis JL (2012) Firefighter Fatalities in the United States-2011, NFPAGoogle Scholar
  4. 4.
    Karter MJ (2011) Fire loss in the United States during 2010: National Fire Protection Association Quincy, MAGoogle Scholar
  5. 5.
    Karter MJ (2005) Selected special analyses of firefighter fatalities. National Fire Protection Association Quincy, MAGoogle Scholar
  6. 6.
    Ronchi E, Gwynne S, Purser D, Colonna P (2013) Representation of the impact of smoke on agent walking speeds in evacuation models. Fire Technol 49:411–431CrossRefGoogle Scholar
  7. 7.
    Bryan JL (2002) Behavioral response to fire and smoke. In: DiNenno PJ (ed) SFPE handbook of fire protection engineering, vol 2. Society of Fire Protection Engineers, BethesdaGoogle Scholar
  8. 8.
    Wright M, Cook G, Webber G (2001) The effects of smoke on people’s walking Speeds using overhead lighting and Wayguidance provision. In: Proceedings of the 2nd international symposium on human behaviour in fire. MIT, Boston, pp 275–284Google Scholar
  9. 9.
    Jin T (1976) Visibility through fire smoke (No. 42): Report of Fire Research Institute of JapanGoogle Scholar
  10. 10.
    Bertram C, Evans MH, Javaid M, Stafford T, Prescott T (2013) Sensory augmentation with distal touch: the tactile helmet project. In: Lepora AF, Mura A, Krapp HG, Verschure PFM, Prescott TJ (eds) Biomimetic and biohybrid systems. Springer, Berlin, pp 24–35Google Scholar
  11. 11.
    Rutherford P (1997) Auditory navigation and the escape from smoke filled buildings. In: Junge R (ed) CAAD futures 1997. Springer, Berlin, pp 299–304Google Scholar
  12. 12.
    Lahr D, Orekhov V, Lee B, Hong D (2013) Early Developments of a Parallelly Actuated Humanoid, SAFFiR. In: ASME 2013 international design engineering technical conferences and computers and information in engineering conference, pp V06BT07A054–V06BT07A054Google Scholar
  13. 13.
    Kim J-H, Keller B, Lattimer BY (2013) Sensor fusion based seek-and-find fire algorithm for intelligent firefighting robot. In: IEEE/ASME international conference on advanced intelligent mechatronics (AIM), pp 1482–1486Google Scholar
  14. 14.
    McNeil JG, Starr J, Lattimer BY (2013) Autonomous fire suppression using multispectral sensors. Presented at the IEEE/ASME international conference on advanced intelligent mechatronics (AIM 2013). Wollongong, AustraliaGoogle Scholar
  15. 15.
    Starr JW, Lattimer BY (2013) Application of thermal infrared stereo vision in fire environments. Presented at the IEEE/ASME international conference on advanced intelligent mechatronics (AIM 2013). Wollongong, AustraliaGoogle Scholar
  16. 16.
    Starr J, Lattimer BY (2013) Evaluation of navigation sensors in fire smoke environments. Fire Technol. doi: 10.1007/s10694-013-0356-3
  17. 17.
    Starr JW, Lattimer B (2012) A comparison of IR stereo vision and LIDAR for use in fire environments. In IEEE Sensors, 2012, pp 1–4Google Scholar
  18. 18.
    Khoon TN, Sebastian P, Saman ABS (2012) Autonomous fire fighting mobile platform. Proc Eng 41:1145–1153CrossRefGoogle Scholar
  19. 19.
    Liljeback P, Stavdahl O, Beitnes A (2006) SnakeFighter-development of a water hydraulic fire fighting snake robot. In: 9th International conference on control, automation, robotics and vision, ICARCV’06, pp 1–6Google Scholar
  20. 20.
    Hong JH, Min B-C, Taylor JM, Raskin V, Matson ET (2012) NL-based communication with firefighting robots. In: IEEE international conference on systems, man, and cybernetics (SMC), pp 1461–1466Google Scholar
  21. 21.
    Tan CF, Alkahari M, Rahman A (2011) Development of ground vehicle for fire fighting purpose, pp 78–81Google Scholar
  22. 22.
    Miyazawa K (2002) Fire robots developed by the Tokyo Fire Department. Adv Robot 16:553–556MathSciNetCrossRefGoogle Scholar
  23. 23.
    Chang PH, Kang YH, Cho GR, Kim JH, Jin M, Lee J et al (2006) Control architecture design for a fire searching robot using task oriented design methodology. In: International Joint Conference SICE-ICASE, 2006, pp 3126–3131Google Scholar
  24. 24.
    Kim YD, Kim YG, Lee SH, Kang JH, An J (2009) Portable fire evacuation guide robot system. In: International conference on intelligent robots and systems, IROS 2009. IEEE/RSJ, pp 2789–2794Google Scholar
  25. 25.
    Longo D, Muscato G (2005) CLAWAR WP3 Applications: natural/outdoor and underwater robots. In: Climbing and walking robots. Springer, pp 1159–1170Google Scholar
  26. 26.
    Penders J, Alboul L, Witkowski U, Naghsh A, Saez-Pons J, Herbrechtsmeier S et al (2011) A robot swarm assisting a human fire-fighter. Adv Robot 25:93–117CrossRefGoogle Scholar
  27. 27.
    Kleindienst S, Dimmeler A, Clement D (1995) Obscuration effects of smoke clouds. In: Society of photo-optical instrumentation engineers (SPIE) conference series, pp 440–451Google Scholar
  28. 28.
    Koylu U, Faeth G (1996) Spectral extinction coefficients of soot aggregates from turbulent diffusion flames. J Heat Transf 118:415–421CrossRefGoogle Scholar
  29. 29.
    Dalzell W, Sarofim A (1969) Optical constants of soot and their application to heat-flux calculations. J Heat Transf 91:100–104CrossRefGoogle Scholar
  30. 30.
    Jin T (2008) Chapter 2–4 Visibility and human behavior in fire smoke. In: SFPE handbook of fire protection engineering, 4th edn. The Society of Fire Protection Engineers, BostonGoogle Scholar
  31. 31.
    Felicetti R (2013) Assessment methods of fire damages in concrete tunnel linings. Fire Technol 49:509–529CrossRefGoogle Scholar
  32. 32.
    Hajpál M (2002) Changes in sandstones of historical monuments exposed to fire or high temperature. Fire Technol 38:373–382CrossRefGoogle Scholar
  33. 33.
    Cider L (1993) Cleaning and reliability of smoke-contaminated electronics. Fire Technol 29:226–245CrossRefGoogle Scholar
  34. 34.
    Amon F, Ducharme A (2009) Image frequency analysis for testing of fire service thermal imaging cameras. Fire Technol 45:313–322CrossRefGoogle Scholar
  35. 35.
    Maxwell FD (1971) A portable IR system for observing fire thru smoke. Fire Technol 7:321–331CrossRefGoogle Scholar
  36. 36.
    Amon F, Hamins A, Bryner N, Rowe J (2008) Meaningful performance evaluation conditions for fire service thermal imaging cameras. Fire Saf J 43:541–550CrossRefGoogle Scholar
  37. 37.
    Amon F, Hamins A, Rowe J (2006) First responder thermal imaging cameras: establishment of representative performance testing conditions. In: Defense and security symposium, pp 62050Y–62050Y-12Google Scholar
  38. 38.
    Amon F, Bryner N, Hamins A (2004) Evaluation of thermal imaging cameras used in fire fighting applications. In: Defense and Security, pp 44–53Google Scholar
  39. 39.
    Amon F, Benetis V, Kim J, Hamins A (2004) Development of a performance evaluation facility for fire fighting thermal imagers. In: Holst GC (ed) Proceedings SPIE, Infrared imaging systems: design, analysis, modeling, and testing XV, vol 5407. Orlando, FL. doi: 10.1117/12.566779
  40. 40.
    Sakai M, Aoki Y (2006) Human and object detection in smoke-filled space using millimeter-wave radar based measurement system. In: 18th international conference on pattern recognition, ICPR 2006, pp 750–750Google Scholar
  41. 41.
    Sakai M, Aoki Y, Takagi M (2005) The support system of the firefighter’s activity by detecting objects in smoke space. In: Optomechatronic technologies 2005, pp 60510L–60510L-11Google Scholar
  42. 42.
    Omine Y, Sakai M, Aoki Y, Takagi M (2004) Basic examination of a technique to visualize space filled with dense smoke using millimeter-wave radar. In: Optics east, pp 272–281Google Scholar
  43. 43.
    Suchong H, Zhengou Z, Yue W, Siyong Z (1996) Multi-target detection in FMCW radar. In: CIE International conference of radar proceedings, pp 367–370Google Scholar
  44. 44.
    Stove AG (1992) Linear FMCW radar techniques. In: IEE proceedings of radar and signal processing, pp 343–350Google Scholar
  45. 45.
    Rohling H (1983) Radar CFAR thresholding in clutter and multiple target situations. IEEE transactions on aerospace and electronic systems, pp 608–621Google Scholar
  46. 46.
    Brooker G, Hennessey R, Lobsey C, Bishop M, Widzyk‐Capehart E (2007) Seeing through dust and water vapor: millimeter wave radar sensors for mining applications. J Field Robot 24:527–557CrossRefGoogle Scholar
  47. 47.
    Olver A, Cuthbert L (1988) FMCW radar for hidden object detection. In: IEE proceedings of communications, radar and signal processing, pp 354–361Google Scholar
  48. 48.
    Dey D, Geyer C, Singh S, Digioia M (2011) A cascaded method to detect aircraft in video imagery. Int J Robot Res 30:1527–1540CrossRefGoogle Scholar
  49. 49.
    Fischler MA, Bolles RC (1981) Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun ACM 24:381–395MathSciNetCrossRefGoogle Scholar
  50. 50.
    Babrauskas V, Grayson S (1990) Heat release in fires. Taylor & Francis, Boca RatonGoogle Scholar

Copyright information

© Springer Science+Business Media New York 2014

Authors and Affiliations

  • Jong-Hwan Kim
    • 1
    Email author
  • Joseph W. Starr
    • 1
  • Brian Y. Lattimer
    • 1
  1. 1.Department of Mechanical EngineeringVirginia TechBlacksburgUSA

Personalised recommendations