Detection and Tracking of Floating Objects Using a UAV with Thermal Camera

  • Håkon Hagen Helgesen
  • Frederik Stendahl Leira
  • Tor Arne Johansen
  • Thor I. Fossen
Part of the Lecture Notes in Control and Information Sciences book series (LNCIS, volume 474)


This paper develops a vision-based tracking system in unmanned aerial vehicles based on thermal images . The tracking system are tailored toward objects at sea and consists of three main modules that are independent. The first module is an object detection algorithm that uses image analysis techniques to detect marine vessels in thermal images and extract the center of each object. Moreover, as long as the size of the vessel is known or computed in an image where the whole vessel is visible, the center can be identified in situations where only a part of the object is visible. The pixel position of the center is used in a nonlinear state estimator to estimate the position and velocity in a world-fixed coordinate frame. This is called the filtering part of the tracking system. The state estimator is nonlinear because only two coordinates in the world-frame can be computed with the pixel coordinates. This originates from the fact that cameras are bearing-only sensors that are unable to measure range. The last module in the tracking system is data association , which is used to relate new measurements with existing tracks. The tracking system is evaluated in two different case studies. The first case study investigates three different measures for data association in a Monte Carlo simulation. The second case study concerns tracking of a single object in a field experiment, where the object detection algorithm and the filtering part of the tracking system are evaluated. The results show that the modules in the tracking system are reliable with high precision.


Kalman Filter Tracking System Mahalanobis Distance Thermal Image Visual Sensor 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.



The authors would like to thank Lars Semb and Krzysztof Cisek for their technical support and flawless execution of the practical aspects of the field experiments. They would also like to thank Laboratório de Sistemas e Tecnologia Subaquática (LSTS) at the University of Porto, João Sousa and Kanna Rajan for inviting us to participate in their yearly Rapid Environment Picture (REP) exercise in the Azores. This work was partly supported by the Norwegian Research Council (grant numbers 221666 and 223254) through the Center of Autonomous Marine Operations and Systems at Norwegian University of Science and Technology (NTNU AMOS).


  1. 1.
    Hosen, J., Helgesen, H.H., Fusini, L., Fossen, T.I., Johansen, T.A.: Vision-aided nonlinear observer for fixed-wing unmanned aerial vehicle navigation. J. Guid. Control Dyn. 39(8), 1777–1789 (2016)CrossRefGoogle Scholar
  2. 2.
    Fusini, L., Hosen, J., Helgesen, H.H., Johansen, T.A., Fossen, T.I.: Experimental validation of a uniformly semi-globally exponentially stable non-linear observer for gnss- and camera-aided inertial navigation for fixed-wing uavs. In: Proceedings of the International Conference on Unmanned Aircraft Systems, pp. 851–860 (2015)Google Scholar
  3. 3.
    Yu, X., Zhang, Y.: Sense and avoid technologies with applications to unmanned aircraft systems: review and prospects. Progr. Aerosp. Sci. 74, 152–166 (2015)CrossRefGoogle Scholar
  4. 4.
    Yu, H., Beard, R.: A vision-based collision avoidance technique for micro air vehicles using local-level frame mapping and path planning. Auton. Robots 34(1–2), 93–109 (2013)CrossRefGoogle Scholar
  5. 5.
    Ortiz, A., Bonnin-Pascual, F., Garcia-Fidalgo, E.: Vessel inspection: a micro-aerial vehicle-based approach. J. Intell. Robot. Syst. 76(1), 151–167 (2014)CrossRefGoogle Scholar
  6. 6.
    Helgesen, H.H., Leira, F.S., Johansen, T.A., Fossen, T.I.: Tracking of marine surface objects from unmanned aerial vehicles with a pan/tilt unit using a thermal camera and optical flow. In: Proceedings of the International Conference on Unmanned Aircraft Systems, pp. 107–117 (2016)Google Scholar
  7. 7.
    Leira, F.S., Johansen, T.A., Fossen, T.I.: Automatic detection, classification and tracking of objects in the ocean surface from uavs using a thermal camera. In: Proceedings of the IEEE Aerospace Conference, Big Sky, US (2015)Google Scholar
  8. 8.
    Leira, F.S., Trnka, K., Fossen, T.I., Johansen, T.A.: A ligth-weight thermal camera payload with georeferencing capabilities for small fixed-wing UAVs. In: Proceedings of the International Conference on Unmanned Aircraft Systems, pp. 485–494 (2015)Google Scholar
  9. 9.
    Barber, D.B., Redding, J.D., McLain, T.W., Beard, R.W., Taylor, C.N.: Vision-based target geo-location using a fixed-wing miniature air vehicle. J. Intell. Robot. Syst. 47(4), 361–382 (2006)CrossRefGoogle Scholar
  10. 10.
    Kim, J.-H., Sukkarieh, S.: Airborne simultaneous localisation and map building. In: Proceedings of the IEEE International Conference on Robotics and Automation, vol. 1 (2003)Google Scholar
  11. 11.
    Bryson, M., Sukkarieh, S.: Building a robust implementation of bearing-only inertial slam for a uav. J. Field Robot. 24(1–2), 113–143 (2007)CrossRefGoogle Scholar
  12. 12.
    Haugen, J., Imsland, L., Løset, S., Skjetne, R.: Ice observer system for ice management operations. In: Proceedings of the International Offshore and Polar Engineering Conference, pp. 1120–1127 (2011)Google Scholar
  13. 13.
    Johansen, T.A., Perez, T.: Unmanned aerial surveillance system for hazard collision avoidance in autonomous shipping. In: Proceedings of the International Conference on Unmanned Aircraft Systems (2016)Google Scholar
  14. 14.
    Hu, W., Tan, T., Wang, L., Maybank, S.: A survey on visual surveillance of object motion and behaviors. IEEE Trans. Syst. Man Cybern. Part C (Appl. Rev.) 34(3), 334–352 (2004)Google Scholar
  15. 15.
    Yilmaz, A., Javed, O., Shah, M.: Object tracking: a survey. ACM Comput. Surv. 38(4) (2006)Google Scholar
  16. 16.
    Kumar, R., Sawhney, H., Samarasekera, S., Hsu, S., Tao, H., Guo, Y., Hanna, K., Pope, A., Wildes, R., Hirvonen, D., Hansen, M., Burt, P.: Aerial video surveillance and exploitation. Proc. IEEE 89(10), 1518–1539 (2001)CrossRefGoogle Scholar
  17. 17.
    Yang, H., Shao, L., Zheng, F., Wang, L., Song, Z.: Recent advances and trends in visual tracking: a review. Neurocomputing 74(18), 3823–3831 (2011)CrossRefGoogle Scholar
  18. 18.
    Lowe, D.: Object recognition from local scale-invariant features. In: Proceedings of the International Conference on Computer Vision, pp. 1150–1157 (1999)Google Scholar
  19. 19.
    Bay, H., Ess, A., Tuytelaars, T., Van Gool, L.: Speeded-up robust features (surf). Comput. Vis. Image Underst. 110(3), 346–359 (2008)CrossRefGoogle Scholar
  20. 20.
    Shi, J., Tomasi, C.: Good features to track. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 593–600 (1994)Google Scholar
  21. 21.
    Qadir, A., Neubert, J., Semke, W.: On-board visual tracking with unmanned aircraft system (UAs). In: AIAA Infotech at Aerospace Conference and Exhibit 2011 (2011)Google Scholar
  22. 22.
    Teulire, C., Eck, L., Marchand, E.: Chasing a moving target from a flying UAV. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 4929–4934 (2011)Google Scholar
  23. 23.
    Bar-Shalom, Y., Li, X.R., Kirubarajan, T.: Estimation with Applications to Tracking and Navigation: Theory Algorithms and Software. Wiley, New Jersey (2004)Google Scholar
  24. 24.
    Martínez, C., Mondragón, I.F., Campoy, P., Sánchez-López, J.L., Olivares-Méndez, M.A.: A hierarchical tracking strategy for vision-based applications on-board UAVs. J. Intell. Robot. Syst. 72(3), 517–539 (2013)CrossRefGoogle Scholar
  25. 25.
    Kadyrov, A., Yu, H., Liu, H.: Ship detection and segmentation using image correlation. In: Proceedings of the IEEE International Conference on Systems, Man, and Cybernetics, pp. 3119–3126 (2013)Google Scholar
  26. 26.
    Prevost, C.G., Desbiens, A., Gagnon, E.: Extended kalman filter for state estimation and trajectory prediction of a moving object detected by an unmanned aerial vehicle. In: 2007 American Control Conference, pp. 1805–1810 (2007)Google Scholar
  27. 27.
    Niedfeldt, P.C., Beard, R.W.: Multiple target tracking using recursive RANSAC. In: 2014 American Control Conference, pp. 3393–3398 (2014)Google Scholar
  28. 28.
    Bishop, C.: Pattern Recognition and Machine Learning. Springer, Heidelberg (2006)zbMATHGoogle Scholar
  29. 29.
    Niedfeldt, P.C., Beard, R.W.: Recursive RANSAC: multiple signal estimation with outliers. IFAC Proc. 46(23), 430–435 (2013)CrossRefGoogle Scholar
  30. 30.
    Konstantinova, P., Udvarev, A., Semerdjiev, T.: A study of a target tracking algorithm using global nearest neighbor approach. In: Proceedings of the International Conference on Computer Systems and Technologies (CompSysTech03), pp. 290–295 (2003)Google Scholar
  31. 31.
    Soille, P.: Morphological Image Analysis: Principles and Applications, 2nd edn. Springer Inc, New York, Secaucus, NJ, USA (2003)zbMATHGoogle Scholar
  32. 32.
    Suzuki, S., Abe, K.: Topological structural analysis of digitized binary images by border following. Comput. Vis. Graph. Image Process. 30(1), 32–46 (1985)CrossRefzbMATHGoogle Scholar
  33. 33.
    Hemerly, E.M.: Automatic georeferencing of images acquired by UAV’s. Int. J. Autom. Comput. 11(4), 347–352 (2014)CrossRefGoogle Scholar
  34. 34.
    Hu, M.-K.: Visual pattern recognition by moment invariants. IRE Trans. Inf. Theory 8(2), 179–187 (1962)CrossRefzbMATHMathSciNetGoogle Scholar
  35. 35.
    Wilthil, E.F., Brekke, E.F.: Compensation of navigation uncertainty for target tracking on a moving platform. In: Proceedings of the 19th International Conference on Information Fusion, pp. 1616–1621 (2016)Google Scholar
  36. 36.
    Bryson, M., Sukkarieh, S.: Bearing-only slam for an airborne vehicle. In: Proceedings of the Australasian Conference on Robotics and Automation, vol. 4 (2005)Google Scholar
  37. 37.
    Li, X.R., Jilkov, V.P.: Survey of maneuvering target tracking: dynamic models. Proc. SPIE 4048, 212–235 (2000)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  • Håkon Hagen Helgesen
    • 1
  • Frederik Stendahl Leira
    • 1
  • Tor Arne Johansen
    • 1
  • Thor I. Fossen
    • 1
  1. 1.NTNU AMOS, Department of Engineering CyberneticsNorwegian University of Science and TechnologyTrondheimNorway

Personalised recommendations