Skip to main content
SpringerLink
Log in

Detection and Tracking of Floating Objects Using a UAV with Thermal Camera

  • Chapter
  • First Online:
Sensing and Control for Autonomous Vehicles

Part of the book series: Lecture Notes in Control and Information Sciences ((LNCIS,volume 474))

Abstract

This paper develops a vision-based tracking system in unmanned aerial vehicles based on thermal images . The tracking system are tailored toward objects at sea and consists of three main modules that are independent. The first module is an object detection algorithm that uses image analysis techniques to detect marine vessels in thermal images and extract the center of each object. Moreover, as long as the size of the vessel is known or computed in an image where the whole vessel is visible, the center can be identified in situations where only a part of the object is visible. The pixel position of the center is used in a nonlinear state estimator to estimate the position and velocity in a world-fixed coordinate frame. This is called the filtering part of the tracking system. The state estimator is nonlinear because only two coordinates in the world-frame can be computed with the pixel coordinates. This originates from the fact that cameras are bearing-only sensors that are unable to measure range. The last module in the tracking system is data association , which is used to relate new measurements with existing tracks. The tracking system is evaluated in two different case studies. The first case study investigates three different measures for data association in a Monte Carlo simulation. The second case study concerns tracking of a single object in a field experiment, where the object detection algorithm and the filtering part of the tracking system are evaluated. The results show that the modules in the tracking system are reliable with high precision.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 169.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 219.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD 219.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Hosen, J., Helgesen, H.H., Fusini, L., Fossen, T.I., Johansen, T.A.: Vision-aided nonlinear observer for fixed-wing unmanned aerial vehicle navigation. J. Guid. Control Dyn. 39(8), 1777–1789 (2016)

    Article  Google Scholar 

  2. Fusini, L., Hosen, J., Helgesen, H.H., Johansen, T.A., Fossen, T.I.: Experimental validation of a uniformly semi-globally exponentially stable non-linear observer for gnss- and camera-aided inertial navigation for fixed-wing uavs. In: Proceedings of the International Conference on Unmanned Aircraft Systems, pp. 851–860 (2015)

    Google Scholar 

  3. Yu, X., Zhang, Y.: Sense and avoid technologies with applications to unmanned aircraft systems: review and prospects. Progr. Aerosp. Sci. 74, 152–166 (2015)

    Article  Google Scholar 

  4. Yu, H., Beard, R.: A vision-based collision avoidance technique for micro air vehicles using local-level frame mapping and path planning. Auton. Robots 34(1–2), 93–109 (2013)

    Article  Google Scholar 

  5. Ortiz, A., Bonnin-Pascual, F., Garcia-Fidalgo, E.: Vessel inspection: a micro-aerial vehicle-based approach. J. Intell. Robot. Syst. 76(1), 151–167 (2014)

    Article  Google Scholar 

  6. Helgesen, H.H., Leira, F.S., Johansen, T.A., Fossen, T.I.: Tracking of marine surface objects from unmanned aerial vehicles with a pan/tilt unit using a thermal camera and optical flow. In: Proceedings of the International Conference on Unmanned Aircraft Systems, pp. 107–117 (2016)

    Google Scholar 

  7. Leira, F.S., Johansen, T.A., Fossen, T.I.: Automatic detection, classification and tracking of objects in the ocean surface from uavs using a thermal camera. In: Proceedings of the IEEE Aerospace Conference, Big Sky, US (2015)

    Google Scholar 

  8. Leira, F.S., Trnka, K., Fossen, T.I., Johansen, T.A.: A ligth-weight thermal camera payload with georeferencing capabilities for small fixed-wing UAVs. In: Proceedings of the International Conference on Unmanned Aircraft Systems, pp. 485–494 (2015)

    Google Scholar 

  9. Barber, D.B., Redding, J.D., McLain, T.W., Beard, R.W., Taylor, C.N.: Vision-based target geo-location using a fixed-wing miniature air vehicle. J. Intell. Robot. Syst. 47(4), 361–382 (2006)

    Article  Google Scholar 

  10. Kim, J.-H., Sukkarieh, S.: Airborne simultaneous localisation and map building. In: Proceedings of the IEEE International Conference on Robotics and Automation, vol. 1 (2003)

    Google Scholar 

  11. Bryson, M., Sukkarieh, S.: Building a robust implementation of bearing-only inertial slam for a uav. J. Field Robot. 24(1–2), 113–143 (2007)

    Article  Google Scholar 

  12. Haugen, J., Imsland, L., Løset, S., Skjetne, R.: Ice observer system for ice management operations. In: Proceedings of the International Offshore and Polar Engineering Conference, pp. 1120–1127 (2011)

    Google Scholar 

  13. Johansen, T.A., Perez, T.: Unmanned aerial surveillance system for hazard collision avoidance in autonomous shipping. In: Proceedings of the International Conference on Unmanned Aircraft Systems (2016)

    Google Scholar 

  14. Hu, W., Tan, T., Wang, L., Maybank, S.: A survey on visual surveillance of object motion and behaviors. IEEE Trans. Syst. Man Cybern. Part C (Appl. Rev.) 34(3), 334–352 (2004)

    Google Scholar 

  15. Yilmaz, A., Javed, O., Shah, M.: Object tracking: a survey. ACM Comput. Surv. 38(4) (2006)

    Google Scholar 

  16. Kumar, R., Sawhney, H., Samarasekera, S., Hsu, S., Tao, H., Guo, Y., Hanna, K., Pope, A., Wildes, R., Hirvonen, D., Hansen, M., Burt, P.: Aerial video surveillance and exploitation. Proc. IEEE 89(10), 1518–1539 (2001)

    Article  Google Scholar 

  17. Yang, H., Shao, L., Zheng, F., Wang, L., Song, Z.: Recent advances and trends in visual tracking: a review. Neurocomputing 74(18), 3823–3831 (2011)

    Article  Google Scholar 

  18. Lowe, D.: Object recognition from local scale-invariant features. In: Proceedings of the International Conference on Computer Vision, pp. 1150–1157 (1999)

    Google Scholar 

  19. Bay, H., Ess, A., Tuytelaars, T., Van Gool, L.: Speeded-up robust features (surf). Comput. Vis. Image Underst. 110(3), 346–359 (2008)

    Article  Google Scholar 

  20. Shi, J., Tomasi, C.: Good features to track. In: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 593–600 (1994)

    Google Scholar 

  21. Qadir, A., Neubert, J., Semke, W.: On-board visual tracking with unmanned aircraft system (UAs). In: AIAA Infotech at Aerospace Conference and Exhibit 2011 (2011)

    Google Scholar 

  22. Teulire, C., Eck, L., Marchand, E.: Chasing a moving target from a flying UAV. In: IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 4929–4934 (2011)

    Google Scholar 

  23. Bar-Shalom, Y., Li, X.R., Kirubarajan, T.: Estimation with Applications to Tracking and Navigation: Theory Algorithms and Software. Wiley, New Jersey (2004)

    Google Scholar 

  24. Martínez, C., Mondragón, I.F., Campoy, P., Sánchez-López, J.L., Olivares-Méndez, M.A.: A hierarchical tracking strategy for vision-based applications on-board UAVs. J. Intell. Robot. Syst. 72(3), 517–539 (2013)

    Article  Google Scholar 

  25. Kadyrov, A., Yu, H., Liu, H.: Ship detection and segmentation using image correlation. In: Proceedings of the IEEE International Conference on Systems, Man, and Cybernetics, pp. 3119–3126 (2013)

    Google Scholar 

  26. Prevost, C.G., Desbiens, A., Gagnon, E.: Extended kalman filter for state estimation and trajectory prediction of a moving object detected by an unmanned aerial vehicle. In: 2007 American Control Conference, pp. 1805–1810 (2007)

    Google Scholar 

  27. Niedfeldt, P.C., Beard, R.W.: Multiple target tracking using recursive RANSAC. In: 2014 American Control Conference, pp. 3393–3398 (2014)

    Google Scholar 

  28. Bishop, C.: Pattern Recognition and Machine Learning. Springer, Heidelberg (2006)

    MATH  Google Scholar 

  29. Niedfeldt, P.C., Beard, R.W.: Recursive RANSAC: multiple signal estimation with outliers. IFAC Proc. 46(23), 430–435 (2013)

    Article  Google Scholar 

  30. Konstantinova, P., Udvarev, A., Semerdjiev, T.: A study of a target tracking algorithm using global nearest neighbor approach. In: Proceedings of the International Conference on Computer Systems and Technologies (CompSysTech03), pp. 290–295 (2003)

    Google Scholar 

  31. Soille, P.: Morphological Image Analysis: Principles and Applications, 2nd edn. Springer Inc, New York, Secaucus, NJ, USA (2003)

    MATH  Google Scholar 

  32. Suzuki, S., Abe, K.: Topological structural analysis of digitized binary images by border following. Comput. Vis. Graph. Image Process. 30(1), 32–46 (1985)

    Article  MATH  Google Scholar 

  33. Hemerly, E.M.: Automatic georeferencing of images acquired by UAV’s. Int. J. Autom. Comput. 11(4), 347–352 (2014)

    Article  Google Scholar 

  34. Hu, M.-K.: Visual pattern recognition by moment invariants. IRE Trans. Inf. Theory 8(2), 179–187 (1962)

    Article  MATH  MathSciNet  Google Scholar 

  35. Wilthil, E.F., Brekke, E.F.: Compensation of navigation uncertainty for target tracking on a moving platform. In: Proceedings of the 19th International Conference on Information Fusion, pp. 1616–1621 (2016)

    Google Scholar 

  36. Bryson, M., Sukkarieh, S.: Bearing-only slam for an airborne vehicle. In: Proceedings of the Australasian Conference on Robotics and Automation, vol. 4 (2005)

    Google Scholar 

  37. Li, X.R., Jilkov, V.P.: Survey of maneuvering target tracking: dynamic models. Proc. SPIE 4048, 212–235 (2000)

    Article  Google Scholar 

Download references

Acknowledgements

The authors would like to thank Lars Semb and Krzysztof Cisek for their technical support and flawless execution of the practical aspects of the field experiments. They would also like to thank Laboratório de Sistemas e Tecnologia Subaquática (LSTS) at the University of Porto, João Sousa and Kanna Rajan for inviting us to participate in their yearly Rapid Environment Picture (REP) exercise in the Azores. This work was partly supported by the Norwegian Research Council (grant numbers 221666 and 223254) through the Center of Autonomous Marine Operations and Systems at Norwegian University of Science and Technology (NTNU AMOS).

Author information

Authors and Affiliations

  1. NTNU AMOS, Department of Engineering Cybernetics, Norwegian University of Science and Technology, O.S. Bragstads plass 2D, 7491, Trondheim, Norway

    Håkon Hagen Helgesen, Frederik Stendahl Leira, Tor Arne Johansen & Thor I. Fossen

Authors
  1. Håkon Hagen Helgesen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Frederik Stendahl Leira
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Tor Arne Johansen
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Thor I. Fossen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Håkon Hagen Helgesen .

Editor information

Editors and Affiliations

  1. Norwegian University of Science and Technology (NTNU), Trondheim, Norway

    Thor I. Fossen

  2. Norwegian University of Science and Technology (NTNU), Trondheim, Norway

    Kristin Y. Pettersen

  3. Eindhoven University of Technology (TuE), Eindhoven, The Netherlands

    Henk Nijmeijer

Rights and permissions

Reprints and permissions

Copyright information

© 2017 Springer International Publishing AG

About this chapter

Cite this chapter

Helgesen, H.H., Leira, F.S., Johansen, T.A., Fossen, T.I. (2017). Detection and Tracking of Floating Objects Using a UAV with Thermal Camera. In: Fossen, T., Pettersen, K., Nijmeijer, H. (eds) Sensing and Control for Autonomous Vehicles. Lecture Notes in Control and Information Sciences, vol 474. Springer, Cham. https://doi.org/10.1007/978-3-319-55372-6_14

Download citation

  • DOI: https://doi.org/10.1007/978-3-319-55372-6_14

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-319-55371-9

  • Online ISBN: 978-3-319-55372-6

  • eBook Packages: EngineeringEngineering (R0)

Publish with us

Policies and ethics

Search

Navigation