Tracking of Ocean Surface Objects from Unmanned Aerial Vehicles with a Pan/Tilt Unit using a Thermal Camera

  • Håkon Hagen Helgesen
  • Frederik Stendahl Leira
  • Thor I. Fossen
  • Tor Arne Johansen
Article
  • 58 Downloads

Abstract

This paper presents four vision-based tracking system architectures for marine surface objects using a fixed-wing unmanned aerial vehicle (UAV) with a thermal camera mounted in a pan/tilt gimbal. The tracking systems estimate the position and velocity of an object in the North-East (NE) plane, and differ in how the measurement models are defined. The first tracking system measures the position and velocity of the target with georeferencing and optical flow. The states are estimated in a Kalman filter. A Kalman filter is also utilized in the second architecture, but only the georeferenced position is used as a measurement. A bearing-only measurement model is the basis for the third tracking system, and because the measurement model is nonlinear, an extended Kalman filter is used for state estimation. The fourth tracking system extends the bearing-only tracking system to let navigation uncertainty in the UAV position affect the target estimates in a Schmidt-Kalman filter. All tracking architectures are evaluated on data gathered at a flight experiment near the Azores islands outside of Portugal. The results show that various marine vessels can be tracked quite accurately.

Keywords

Unmanned aerial vehicle Machine vision Target tracking Optical flow Georeferencing 

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    COLREGs - convention on the international regulations for preventing collisions at sea, International Maritime Organization (IMO) (1972)Google Scholar
  2. 2.
    Bar-Shalom, Y., Li, X.R., Kirubarajan, T.: Estimation with Applications to Tracking and Navigation: Theory Algorithms and Software. Wiley, Hoboken (2004)Google Scholar
  3. 3.
    Barber, D.B., Redding, J.D., McLain, T.W., Beard, R.W., Taylor, C.N.: Vision-based target geo-location using a fixed-wing miniature air vehicle. J. Intell. Robot. Syst. 47(4), 361–382 (2006)CrossRefGoogle Scholar
  4. 4.
    Horn, B.K.P., Schunk, B.G.: Determining optical flow. Artif. Intell. 17, 185–204 (1981)CrossRefGoogle Scholar
  5. 5.
    Bradski, G.: The opencv library. Dr Dobb’s Journal of Software Tools (2000)Google Scholar
  6. 6.
    Bryson, M., Sukkarieh, S.: Bearing-Only Slam for an Airborne Vehicle. In: Proceedings of the Australasian Conference on Robotics and Automation, vol. 4 (2005)Google Scholar
  7. 7.
    Bryson, M., Sukkarieh, S.: Building a robust implementation of bearing-only inertial slam for a uav. J. Field Rob. 24(1-2), 113–143 (2007)CrossRefGoogle Scholar
  8. 8.
    Doherty, P., Rudol, P.: A uav search and rescue scenario with human body detection and geolocalization. Lect. Notes Comput. Sci (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 4830, 1–13 (2007)Google Scholar
  9. 9.
    Dumble, S.J., Gibbens, P.W.: Efficient terrain-aided visual horizon based attitude estimation and localization. J. Intell. Robot. Syst. 78(2), 205–221 (2015)CrossRefGoogle Scholar
  10. 10.
    Egeland, O., Gravdahl, J.T.: Modeling and Simulation for Automatic Control. Marine Cybernetics Trondheim, Norway (2002)Google Scholar
  11. 11.
    Elkins, L., Sellers, D., Monach, W.R.: The autonomous maritime navigation (amn) project: Field tests, autonomous and cooperative behaviors, data fusion, sensors, and vehicles. J. Field Rob. 27(6), 790–818 (2010)CrossRefGoogle Scholar
  12. 12.
    Fossen, T.: Handbook of Marine Craft Hydrodynamics and Motion Control. Wiley, Hoboken (2011)CrossRefGoogle Scholar
  13. 13.
    Fusini, L., Hosen, J., Helgesen, H.H., Johansen, T.A., Fossen, T.I.: Experimental validation of a uniformly semi-globally exponentially stable non-linear observer for gnss- and camera-aided inertial navigation for fixed-wing Uavs. In: Proceedings of the International Conference on Unmanned Aircraft Systems, pp. 851–860 (2015)Google Scholar
  14. 14.
    Gomez-Balderas, J.E., Flores, G., García Carrillo, L.R., Lozano, R.: Tracking a ground moving target with a quadrotor using switching control. J. Intell. Robot. Syst. 70(1), 65–78 (2013)CrossRefGoogle Scholar
  15. 15.
    Hansen, J., Fossen, T., Johansen, T.: Nonlinear observer for Ins aided by time-delayed Gnss measurements: implementation and Uav experiments. In: Proceedings of the International Conference on Unmanned Aircraft Systems, pp. 157–166 (2015)Google Scholar
  16. 16.
    Helgesen, H.H., Leira, F.S., Johansen, T.A., Fossen, T.I.: Tracking of marine surface objects from unmanned aerial vehicles with a pan/tilt unit using a thermal camera and optical flow. In: Proceedings of the International Conference on Unmanned Aircraft Systems, pp. 107–117 (2016)Google Scholar
  17. 17.
    Helgesen, H.H., Leira, F.S., Johansen, T.A., Fossen, T.I.: Detection and tracking of floating objects using a Uav with thermal camera. In: Fossen, T.I., Pettersen, K.Y., Nijmeijer, H. (eds.) Sensing and Control for Autonomous Vehicles: Applications to Land, Water and Air Vehicles, pp. 289–316. Springer International Publishing (2017)Google Scholar
  18. 18.
    Hemerly, E.M.: Automatic georeferencing of images acquired by uav’s. Int. J. Autom. Comput. 11(4), 347–352 (2014)CrossRefGoogle Scholar
  19. 19.
    Hiba, A., Zsedrovits, T., Bauer, P., Zarandy, A.: Fast horizon detection for airborne visual systems. In: Proceedings of the International Conference on Unmanned Aircraft Systems, pp. 886–891 (2016). https://doi.org/10.1109/ICUAS.2016.7502661
  20. 20.
    Hosen, J., Helgesen, H.H., Fusini, L., Fossen, T.I., Johansen, T.A.: A vision-aided nonlinear observer for fixed-wing Uav navigation. In: Proceedings of the AIAA Guidance, Navigation, and Control Conference (2016)Google Scholar
  21. 21.
    Hosen, J., Helgesen, H.H., Fusini, L., Fossen, T.I., Johansen, T.A.: Vision-aided nonlinear observer for fixed-wing unmanned aerial vehicle navigation. J. Guid. Control. Dyn. 39(8), 1777–1789 (2016)CrossRefGoogle Scholar
  22. 22.
    Huntsberger, T., Aghazarian, H., Howard, A., Trotz, D.C.: Stereo vision–based navigation for autonomous surface vessels. J. Field Rob. 28(1), 3–18 (2011)CrossRefGoogle Scholar
  23. 23.
    Hutchinson, S., Hager, G., Corke, P.: A tutorial on visual servo control. IEEE Trans. Robot. Autom. 12(5), 651–670 (1996)CrossRefGoogle Scholar
  24. 24.
    Johansen, T., Fossen, T.: Nonlinear filtering with exogenous Kalman filter and double Kalman filter. In: Proceedings of the European Control Conference, Aalborg (2016)Google Scholar
  25. 25.
    Johansen, T.A., Perez, T.: Unmanned aerial surveillance system for hazard collision avoidance in autonomous shipping. In: Proceedings of the International Conference on Unmanned Aircraft Systems (2016)Google Scholar
  26. 26.
    Kadyrov, A., Yu, H., Liu, H.: Ship detection and segmentation using image correlation. In: Proceedings of the IEEE International Conference on Systems, Man, and Cybernetics, pp. 3119–3126 (2013)Google Scholar
  27. 27.
    Kim, J.H., Sukkarieh, S.: Airborne simultaneous localisation and map building. In: Proceedings of the IEEE International Conference on Robotics and Automation, vol. 1 (2003)Google Scholar
  28. 28.
    Leira, F.S., Johansen, T.A., Fossen, T.I.: Automatic detection, classification and tracking of objects in the ocean surface from Uavs using a thermal camera. In: Proceedings of the IEEE Aerospace Conference, Big Sky, US (2015)Google Scholar
  29. 29.
    Leira, F.S., Trnka, K., Fossen, T.I., Johansen, T.A.: A ligth-weight thermal camera payload with georeferencing capabilities for small fixed-wing Uavs. In: Proceedings of the International Conference on Unmanned Aircraft Systems, pp. 485–494 (2015)Google Scholar
  30. 30.
    Li, X.R., Jilkov, V.P.: Survey of maneuvering target tracking: dynamic models. In: Proceedings of SPIE, vol. 4048, pp. 212–235 (2000)Google Scholar
  31. 31.
    Lowe, D.: Object recognition from local scale-invariant features. In: Proceedings of the International Conference on Computer Vision, pp. 1150–1157 (1999)Google Scholar
  32. 32.
    Mammarella, M., Campa, G., Fravolini, M.L., Napolitano, M.R.: Comparing optical flow algorithms using 6-dof motion of real-world rigid objects. IEEE Trans. Syst. Man Cybern. 42(6), 1752–1762 (2012)CrossRefGoogle Scholar
  33. 33.
    Martínez, C., Mondragón, I.F., Campoy, P., Sánchez-López, J.L., Olivares-Méndez, M.A.: A hierarchical tracking strategy for vision-based applications on-board uavs. J. Intell. Robot. Syst. 72(3), 517–539 (2013)CrossRefGoogle Scholar
  34. 34.
    Muja, M., Lowe, D.G.: Flann, fast library for approximate nearest neighbors. In: Proceedings of the International Conference on Computer Vision Theory and Applications (2009)Google Scholar
  35. 35.
    Novoselov, R.Y., Herman, S.M., Gadaleta, S.M., Poore, A.B.: Mitigating the effects of residual biases with Schmidt-Kalman filtering. In: Proceedings of the 8Th International Conference on Information Fusion, vol. 1 (2005)Google Scholar
  36. 36.
    Ortiz, A., Bonnin-Pascual, F., Garcia-Fidalgo, E.: Vessel inspection: a micro-aerial vehicle-based approach. J. Intell. Robot. Syst. 76(1), 151–167 (2014)CrossRefGoogle Scholar
  37. 37.
    Rudol, P., Doherty, P.: Human body detection and geolocalization for Uav search and rescue missions using color and thermal imagery. In: Proceedings of the IEEE Aerospace Conference (2008)Google Scholar
  38. 38.
    Wilthil, E.F., Brekke, E.F.: Compensation of navigation uncertainty for target tracking on a moving platform. In: Proceedings of the 19Th International Conference on Information Fusion, pp. 1616–1621 (2016)Google Scholar
  39. 39.
    Wolf, M.T., Assad, C., Kuwata, Y., Howard, A., Aghazarian, H., Zhu, D., Lu, T., Trebi-Ollennu, A., Huntsberger, T.: 360-degree visual detection and target tracking on an autonomous surface vehicle. J. Field Rob. 27(6), 819–833 (2010)CrossRefGoogle Scholar
  40. 40.
    Yang, C., Blasch, E., Douville, P.: Design of Schmidt-Kalman filter for target tracking with navigation errors. In: Proceedings of the IEEE Aerospace Conference (2010)Google Scholar
  41. 41.
    Yilmaz, A., Javed, O., Shah, M.: Object tracking: a survey. ACM Comput. Surv. 38(13), 1–45 (2006)Google Scholar
  42. 42.
    Yu, X., Zhang, Y.: Sense and avoid technologies with applications to unmanned aircraft systems: review and prospects. Prog. Aerosp. Sci. 74, 152–166 (2015)CrossRefGoogle Scholar
  43. 43.
    Zhao, S., Lin, F., Peng, K., Dong, X., Chen, B.M., Lee, T.H.: Vision-aided estimation of attitude, velocity, and inertial measurement bias for uav stabilization. J. Intell. Robot. Syst. 81(3), 531–549 (2016)CrossRefGoogle Scholar
  44. 44.
    Zhou, G., Li, C., Cheng, P.: Unmanned Aerial Vehicle (Uav) real-time video registration for forest fire monitoring. In: Proceedings of IEEE International Geoscience and Remote Sensing Symposium, vol. 3, pp. 1803–1806 (2005)Google Scholar

Copyright information

© Springer Science+Business Media B.V. 2017

Authors and Affiliations

  1. 1.NTNU Center for Autonomous Marine Operations and Systems, Department of Engineering CyberneticsNorwegian University of Science and TechnologyTrondheimNorway

Personalised recommendations