Skip to main content
SpringerLink
Log in

A Probabilistic Underwater Localisation based on Cross-view and Cross-domain Acoustic and Aerial Images

  • Regular paper
  • Published:
Journal of Intelligent & Robotic Systems Aims and scope Submit manuscript

Abstract

This paper presents a cross-domain and cross-view framework for underwater robot localisation, which does not require any Global Positioning System (GPS) information. The proposed localisation method uses colour aerial images and underwater acoustic images to estimate the robot’s position. The method identifies the correlation among images from distinct domains, given by the matching of images acquired in partially structured environments with shared features. The validation of the proposed method is done using a real dataset, which was acquired by an underwater vehicle in a Marina. Besides, it was compared to Dead Reckoning and a learning-based particle filter method. The experimental results present the feasibility of the proposed method and its advances in relation to state-of-the-art algorithms.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

Data Availability

The code that support the findings of this study are available from the corresponding author upon reasonable request.

References

  1. Concha, A., Drews-Jr, P., Campos, M., Civera, J.: Real-time localization and dense mapping in underwater environments from a monocular sequence. In: OCEANS 2015 - Genova, pp. 1–5 (2015). 10.1109/OCEANS-Genova.2015.7271476

  2. Kuhn, V.N., Drews-Jr, P.L.J., Gomes, S.C.P., Cunha, M.A.B., Botelho, S.S.d.C.: Automatic control of a rov for inspection of underwater structures using a low-cost sensing. Journal of the Brazilian Society of Mechanical Sciences and Engineering 37(1), 361–374 (2015)

  3. Petillot, Y.R., Antonelli, G., Casalino, G., Ferreira, F.: Underwater robots: From remotely operated vehicles to intervention-autonomous underwater vehicles. IEEE RA Magazine 26(2), 94–101 (2019). https://doi.org/10.1109/MRA.2019.2908063

    Article  Google Scholar 

  4. Siddall, R., Kovač, M.: Launching the aquamav: bioinspired design for aerial-aquatic robotic platforms. Bioinspiration & Biomimetics 9(3), 031001 (2014)

    Article  Google Scholar 

  5. Sathappan, N., Tokhi, M.O., Penaluna, L., Zhao, Z., Duan, F., Shirkoohi, G., Kaur, A.: A literature review on data transmission using electromagnetic waves under different aquatic environments. Marine Technology Society Journal 55(5), 138–149 (2021)

    Article  Google Scholar 

  6. Drews-Jr, P., Nascimento, E.R., Campos, M.F.M., Elfes, A.: Automatic restoration of underwater monocular sequences of images. In: 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 1058–1064 (2015). https://doi.org/10.1109/IROS.2015.7353501

  7. Paull, L., Saeedi, S., Seto, M., Li, H.: Auv navigation and localization: A review. IEEE J. Oceanic Eng. 39(1), 131–149 (2013)

    Article  Google Scholar 

  8. C. S. Ribeiro, P.O., M. dos Santos, M., L. J. Drews-Jr, P., S. C. Botelho, S.: Forward looking sonar scene matching using deep learning. In: 2017 16th IEEE International Conference on Machine Learning and Applications (ICMLA), pp. 574–579 (2017). https://doi.org/10.1109/ICMLA.2017.00-99

  9. Remmas, W., Chemori, A., Kruusmaa, M.: Diver tracking in open waters: A low-cost approach based on visual and acoustic sensor fusion. J. Field Robotics 38(3), 494–508 (2021)

    Article  Google Scholar 

  10. Hurtos, N., Nagappa, S., Cufi, X., Petillot, Y., Salvi, J.: Evaluation of registration methods on two-dimensional forward-looking sonar imagery. In: OCEANS - Bergen, 2013 MTS/IEEE, pp. 1–8 (2013). https://doi.org/10.1109/OCEANS-Bergen.2013.6608124

  11. Wu, Y.: Coordinated path planning for an unmanned aerial-aquatic vehicle (UAAV) and an autonomous underwater vehicle (AUV) in an underwater target strike mission. Ocean Engineering 182, 162–173 (2019). https://doi.org/10.1016/j.oceaneng.2019.04.062

    Article  Google Scholar 

  12. Leung, K.Y.K., Clark, C.M., Huissoon, J.P.: Localization in urban environments by matching ground level video images with an aerial image. In: IEEE ICRA 2008, pp. 551–556 (2008). https://doi.org/10.1109/ROBOT.2008.4543264

  13. Viswanathan, A., Pires, B.R., Huber, D.: Vision based robot localization by ground to satellite matching in gps-denied situations. In: 2014 IEEE/RSJ IROS, pp. 192–198 (2014). https://doi.org/10.1109/IROS.2014.6942560

  14. Wolff, M., Collins, R.T., Liu, Y.: Regularity-driven building facade matching between aerial and street views. In: IEEE CVPR, pp. 1591–1600 (2016). https://doi.org/10.1109/CVPR.2016.176

  15. Hu, S., Feng, M., Nguyen, R.M.H., Hee Lee, G.: CVM-Net: Cross-view matching network for image-based ground-to-aerial geo-localization. In: IEEE CVPR, pp. 7258–7267 (2018)

  16. Zhuo, X., Koch, T., Kurz, F., Fraundorfer, F., Reinartz, P.: Automatic uav image geo-registration by matching uav images to georeferenced image data. Remote Sensing 9(4), 376 (2017)

    Article  Google Scholar 

  17. Drews-Jr, P., Neto, A., Campos, M.: Hybrid unmanned aerial underwater vehicle: Modeling and simulation. In: IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 4637–4642 (2014). https://doi.org/10.1109/IROS.2014.6943220

  18. Horn, A.C., Pinheiro, P.M., Grando, R.B., da Silva, C.B., Neto, A.A., Drews-Jr, P.L.J.: A novel concept for hybrid unmanned aerial underwater vehicles focused on aquatic performance. In: IEEE Latin American Robotics Symposium (LARS) and Brazilian Symposium on Robotics (SBR), pp. 1–6 (2020). https://doi.org/10.1109/LARS/SBR/WRE51543.2020.9307110

  19. Dos Santos, M.M., De Giacomo, G.G., Drews-Jr, P.L.J., Botelho, S.S.C.: Satellite and underwater sonar image matching using deep learning. In: 2019 Latin American Robotics Symposium (LARS), 2019 Brazilian Symposium on Robotics (SBR) and 2019 Workshop on Robotics in Education (WRE), pp. 109–114 (2019). https://doi.org/10.1109/LARS-SBR-WRE48964.2019.00027

  20. Dos Santos, M.M., De Giacomo, G.G., Drews, P.L.J., Botelho, S.S.C.: Matching color aerial images and underwater sonar images using deep learning for underwater localization. IEEE Robotics and Automation Letters 5(4), 6365–6370 (2020)

    Article  Google Scholar 

  21. Chopra, S., Hadsell, R., LeCun, Y.: Learning a similarity metric discriminatively, with application to face verification. In: IEEE CVPR, vol. 1, pp. 539–546 (2005)

  22. De Giacomo, G.G., dos Santos, M.M., Drews-Jr, P.L., Botelho, S.S.: Cooperative training of triplet networks for cross-domain matching. In: 2020 Latin American Robotics Symposium (LARS), 2020 Brazilian Symposium on Robotics (SBR) and 2020 Workshop on Robotics in Education (WRE), pp. 1–6 (2020)

  23. Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial nets. In: Advances in Neural Information Processing Systems, pp. 2672–2680 (2014)

  24. Hoffer, E., Ailon, N.: Deep metric learning using triplet network. In: International Workshop on Similarity-Based Pattern Recognition, pp. 84–92 (2015). Springer

  25. Thrun, S.: Probabilistic robotics. Communications of the ACM 45(3), 52–57 (2002)

    Article  Google Scholar 

  26. Dos Santos, M.M., De Giacomo, G.G., Drews-Jr, P.L., Botelho, S.S., Mello, C.D.: A framework for underwater vehicle localization based on cross-view and cross-domain acoustic and aerial images. In: 2021 Latin American Robotics Symposium (LARS), 2021 Brazilian Symposium on Robotics (SBR), and 2021 Workshop on Robotics in Education (WRE), pp. 204–209 (2021). IEEE

  27. Dos Santos, M.M., De Giacomo, G.G., Drews-Jr, P.L., Botelho, S.S.: Cross-view and cross-domain underwater localization based on optical aerial and acoustic underwater images. IEEE Robotics and Automation Letters 7(2), 4969–4974 (2022)

    Article  Google Scholar 

  28. Djuric, P.M., Kotecha, J.H., Zhang, J., Huang, Y., Ghirmai, T., Bugallo, M.F., Miguez, J.: Particle filtering. IEEE Signal Processing Magazine 20(5), 19–38 (2003)

    Article  Google Scholar 

  29. Gao, X., Shen, S., Hu, Z., Wang, Z.: Ground and aerial meta-data integration for localization and reconstruction: A review. Pattern Recognition Letters 127, 202–214 (2018)

    Article  Google Scholar 

  30. Bansal, M., Daniilidis, K., Sawhney, H.: Ultrawide baseline facade matching for geo-localization. In: Large-Scale Visual Geo-Localization, pp. 77–98. Springer, (2016)

  31. Bansal, M., Sawhney, H.S., Cheng, H., Daniilidis, K.: Geo-localization of street views with aerial image databases. In: Proceedings of the 19th ACM International Conference on Multimedia. MM ’11, pp. 1125–1128. ACM, New York, NY, USA (2011). 10.1145/2072298.2071954

  32. Li, A., Morariu, V.I., Davis, L.S.: Planar structure matching under projective uncertainty for geolocation. In: ECCV, pp. 265–280 (2014)

  33. Majdik, A.L., Albers-Schoenberg, Y., Scaramuzza, D.: MAV urban localization from google street view data. In: IEEE/RSJ IROS, pp. 3979–3986 (2013)

  34. Noda, M., Takahashi, T., Deguchi, D., Ide, I., Murase, H., Kojima, Y., Naito, T.: Vehicle ego-localization by matching in-vehicle camera images to an aerial image. In: ACCV, pp. 163–173 (2010)

  35. Lin, T.-Y., Belongie, S., Hays, J.: Cross-view image geolocalization. In: IEEE CVPR, pp. 891–898 (2013)

  36. Castaldo, F., Zamir, A., Angst, R., Palmieri, F., Savarese, S.: Semantic cross-view matching. In: IEEE ICCVw, pp. 9–17 (2015)

  37. Lin, T.-Y., Cui, Y., Belongie, S., Hays, J.: Learning deep representations for ground-to-aerial geolocalization. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2015)

  38. Workman, S., Jacobs, N.: On the location dependence of convolutional neural network features. In: IEEE CVPRw, pp. 70–78 (2015)

  39. Workman, S., Souvenir, R., Jacobs, N.: Wide-area image geolocalization with aerial reference imagery. In: The IEEE International Conference on Computer Vision (ICCV) (2015)

  40. Tian, Y., Chen, C., Shah, M.: Cross-view image matching for geo-localization in urban environments. In: IEEE CVPR, pp. 3608–3616 (2017)

  41. Ren, S., He, K., Girshick, R., Sun, J.: Faster r-cnn: Towards real-time object detection with region proposal networks. In: Cortes, C., Lawrence, N., Lee, D., Sugiyama, M., Garnett, R. (eds.) Advances in Neural Information Processing Systems, vol. 28. Curran Associates, Inc., (2015). https://proceedings.neurips.cc/paper/2015/file/14bfa6bb14875e45bba028a21ed38046-Paper.pdf

  42. Matas, J., Galambos, C., Kittler, J.: Robust detection of lines using the progressive probabilistic hough transform. CVIU 78(1), 119–137 (2000)

    Google Scholar 

  43. Bay, H., Ess, A., Tuytelaars, T., Gool, L.V.: Speeded-up robust features (surf). CVIU 110(3), 346–359 (2008). https://doi.org/10.1016/j.cviu.2007.09.014

    Article  Google Scholar 

  44. Drews-Jr, P., Longui, J., Rosa, V.: Real-time depth estimation for underwater inspection using dual laser and camera. In: Symposium on Computing and Automation for Offshore Shipbuilding (NAVCOMP), pp. 52–56 (2013). https://doi.org/10.1109/NAVCOMP.2013.16

  45. Armstrong, R.A., Pizarro, O., Roman, C.: Underwater robotic technology for imaging mesophotic coral ecosystems. In: Mesophotic Coral Ecosystems, pp. 973–988. Springer, (2019)

  46. Greenaway, S.F., Sullivan, K.D., Umfress, S.H., Beittel, A.B., Wagner, K.D.: Revised depth of the challenger deep from submersible transects; including a general method for precise, pressure-derived depths in the ocean. Deep Sea Research Part I: Oceanographic Research Papers 178, 103644 (2021)

    Article  Google Scholar 

  47. dos Santos, M.M., De Giacomo, G.G., Drews, P.L.J., Botelho, S.S.C.: Semantic segmentation of static and dynamic structures of marina satellite images using deep learning. In: 2019 8th Brazilian Conference on Intelligent Systems (BRACIS), pp. 711–716 (2019). https://doi.org/10.1109/BRACIS.2019.00129

  48. Irwin, F., et al.: An isotropic 3x3 image gradient operator. Presentation at Stanford AI Project 2014(02) (1968)

  49. Quigley, M., Conley, K., Gerkey, B., Faust, J., Foote, T., Leibs, J., Wheeler, R., Ng, A.Y.: Ros: an open-source robot operating system. In: ICRA Workshop on Open Source Software, vol. 3, p. 5 (2009). Kobe, Japan

  50. Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomedical image segmentation. In: MICCAI, pp. 234–241 (2015)

Download references

Acknowledgements

This research is partly supported by CNPq, CAPES and FAPERGS. We also would like to thank the colleagues from NAUTEC-FURG for helping with the experimental data and for productive discussions and meetings. Finally, we would like to thank NVIDIA for donating high-performance graphics cards. All authors are with NAUTEC, Intelligent Robotics and Automation Group, Federal University of Rio Grande - FURG, Rio Grande - Brazil.

Author information

Authors and Affiliations

  1. Federal University of Rio Grande - Intelligent Robotics and Automation Group, Itália Km 8, Rio Grande, 96203-900, RS, Brazil

    Matheus Machado dos Santos, Giovanni G. de Giacomo, Paulo Lilles Jorge Drews-Jr & Silvia Silva da Costa Botelho

  2. Federal University of Pelotas - Intelligent Systems and Control Group, Pelotas, 96010-450, RS, Brazil

    Paulo Jefferson Dias de Oliveira Evald

Authors
  1. Matheus Machado dos Santos
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Paulo Jefferson Dias de Oliveira Evald
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Giovanni G. de Giacomo
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Paulo Lilles Jorge Drews-Jr
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Silvia Silva da Costa Botelho
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Matheus M. dos Santos: Conceptualisation; Methodology; Software; Validation; Investigation; Writing - Original Draft.Paulo J. D. O. Evald: Writing - Review & Editing. Giovanni G. De Giacomo: Data Curation; Formal analysis. Paulo L. J. Drews-Jr: Visualisation; Project administration; Review & Editing. Silvia S. C. Botelho: Supervision; Resources.

Corresponding author

Correspondence to Matheus Machado dos Santos.

Ethics declarations

Ethics Approval

Not applicable.

Consent to Participate

Not applicable.

Consent to Publish

Not applicable.

Conflicts of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

dos Santos, M.M., de Oliveira Evald, P.J.D., de Giacomo, G.G. et al. A Probabilistic Underwater Localisation based on Cross-view and Cross-domain Acoustic and Aerial Images. J Intell Robot Syst 108, 34 (2023). https://doi.org/10.1007/s10846-023-01837-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s10846-023-01837-y

Keywords

Search

Navigation