Skip to main content
SpringerLink
Log in

Semantic Segmentation of Marine Species in an Unconstrained Underwater Environment

  • Conference paper
  • First Online:
Robotics, Computer Vision and Intelligent Systems (ROBOVIS 2020, ROBOVIS 2021)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 1667))

  • 277 Accesses

Abstract

A non-invasive Underwater Fish Observatory (UFO) was developed and deployed on the seafloor to perform continuous recording of stereo video and sonar data as well as various oceanic parameters at a high temporal sampling rate. The acquired image data is processed to automatically detect, classify and measure the size of passing aquatic organisms. An important subtask in this processing chain is the semantic segmentation of the previously detected animals. Within this publication, a former segmentation system, that only considered a binary classification of fish and background, is extended to a multi-class segmentation system by including an additional species. Since the images usually contain a lot of background, the semantic segmentation is a problem with a high class imbalance, which demands special care in the choice of loss functions and evaluation metrics. Therefore, three different loss functions, namely Dice loss, Focal loss and Lovasz loss, which are well suited for class-imbalance problems, are investigated and their effect on the final mean intersection-over-union (IoU) on a separate test set is explored. For the given dataset, the model trained with a Focal loss performed best achieving an average, class specific IoU of 0.982 for the background class, 0.828 for the Aurelia aurita and 0.678 for the Gadus morhua.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 64.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 84.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Geomar helmholtz centre for ocean research kiel. https://www.geomar.de/, Accessed 28 Mar 2022

  2. Macartney germany gmbh. https://www.macartney.de/, Accessed 28 Mar 2022

  3. rosemann software gmbh. https://www.camiq.net, Accessed 28 Mar 2022

  4. Thuenen institute. https://www.thuenen.de, Accessed 28 Mar 2022

  5. Abdeldaim, A.M., Houssein, E.H., Hassanien, A.E.: Color image segmentation of fishes with complex background in water. In: Hassanien, A.E., Tolba, M.F., Elhoseny, M., Mostafa, M. (eds.) AMLTA 2018. AISC, vol. 723, pp. 634–643. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-74690-6_62

    Chapter  Google Scholar 

  6. Badrinarayanan, V., Kendall, A., Cipolla, R.: Segnet: a deep convolutional encoder-decoder architecture for image segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 39(12), 2481–2495 (2017)

    Article  Google Scholar 

  7. Berman, M., Triki, A.R., Blaschko, M.B.: The lovász-softmax loss: a tractable surrogate for the optimization of the intersection-over-union measure in neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4413–4421 (2018)

    Google Scholar 

  8. Biewald, L.: Experiment tracking with weights and biases (2020). https://www.wandb.com/software available from wandb.com

  9. Böer., G., Veeramalli., R., Schramm., H.: Segmentation of fish in realistic underwater scenes using lightweight deep learning models. In: Proceedings of the 2nd International Conference on Robotics, Computer Vision and Intelligent Systems - ROBOVIS, pp. 158–164. INSTICC, SciTePress (2021). https://doi.org/10.5220/0010712700003061

  10. Chen, L.C., Papandreou, G., Schroff, F., Adam, H.: Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv:1706.05587 (2017)

  11. Dearden, P., Theberge, M., Yasué, M.: Using underwater cameras to assess the effects of snorkeler and scuba diver presence on coral reef fish abundance, family richness, and species composition. Environ. Monit. Assess. 163(1), 531–538 (2010)

    Article  Google Scholar 

  12. Dice, L.R.: Measures of the amount of ecologic association between species. Ecology 26(3), 297–302 (1945)

    Article  Google Scholar 

  13. Fernandes, A.F.: Deep learning image segmentation for extraction of fish body measurements and prediction of body weight and carcass traits in nile tilapia. Comput. Electron. Agric. 170, 105274 (2020)

    Article  Google Scholar 

  14. Garcia, R.: Automatic segmentation of fish using deep learning with application to fish size measurement. ICES J. Marine Sci. 77(4), 1354–1366 (2020)

    Article  Google Scholar 

  15. Güler, R.A., Neverova, N., Kokkinos, I.: Densepose: dense human pose estimation in the wild. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7297–7306 (2018)

    Google Scholar 

  16. Harvey, E.S., Santana-Garcon, J., Goetze, J., Saunders, B.J., Cappo, M.: The use of stationary underwater video for sampling sharks. In: Shark Research: Emerging Technologies and Applications for the Field and Laboratory, pp. 111–132 (2018)

    Google Scholar 

  17. He, K., Gkioxari, G., Dollár, P., Girshick, R.B.: Mask R-CNN. CoRR abs/1703.06870 (2017). http://arxiv.org/abs/1703.06870

  18. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)

    Google Scholar 

  19. Isensee, F., Jaeger, P.F., Kohl, S.A., Petersen, J., Maier-Hein, K.H.: nnu-net: a self-configuring method for deep learning-based biomedical image segmentation. Nat. Methods 18(2), 203–211 (2021)

    Article  Google Scholar 

  20. Jaccard, P.: The distribution of the flora in the alpine zone. 1. New Phytologist 11(2), 37–50 (1912)

    Google Scholar 

  21. Kawabata, K., et al.: Underwater image gathering by utilizing stationary and movable sensor nodes: towards observation of symbiosis system in the coral reef of okinawa. Int. J. Distrib. Sensor Netw. 10(7), 835642 (2014)

    Article  Google Scholar 

  22. Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)

  23. Konovalov, D.A., Saleh, A., Efremova, D.B., Domingos, J.A., Jerry, D.R.: Automatic weight estimation of harvested fish from images. In: 2019 Digital Image Computing: Techniques and Applications (DICTA), pp. 1–7. IEEE (2019)

    Google Scholar 

  24. Längkvist, M., Kiselev, A., Alirezaie, M., Loutfi, A.: Classification and segmentation of satellite orthoimagery using convolutional neural networks. Remote Sens. 8(4), 329 (2016)

    Article  Google Scholar 

  25. Letessier, T.B., Juhel, J.B., Vigliola, L., Meeuwig, J.J.: Low-cost small action cameras in stereo generates accurate underwater measurements of fish. J. Exp. Marine Biol. Ecol. 466, 120–126 (2015)

    Article  Google Scholar 

  26. Li, L., Dong, B., Rigall, E., Zhou, T., Donga, J., Chen, G.: Marine animal segmentation. IEEE Trans. Circ. Syst. Video Technol. 32, 2303–2314 (2021)

    Article  Google Scholar 

  27. Lin, T.Y., Goyal, P., Girshick, R., He, K., Dollár, P.: Focal loss for dense object detection. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2980–2988 (2017)

    Google Scholar 

  28. Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3431–3440 (2015)

    Google Scholar 

  29. Marchesan, M., Spoto, M., Verginella, L., Ferrero, E.A.: Behavioural effects of artificial light on fish species of commercial interest. Fisher. Res. 73(1–2), 171–185 (2005)

    Article  Google Scholar 

  30. Milletari, F., Navab, N., Ahmadi, S.A.: V-net: fully convolutional neural networks for volumetric medical image segmentation. In: 2016 Fourth International Conference on 3D Vision (3DV), pp. 565–571. IEEE (2016)

    Google Scholar 

  31. Qin, H., Peng, Y., Li, X.: Foreground extraction of underwater videos via sparse and low-rank matrix decomposition. In: 2014 ICPR Workshop on Computer Vision for Analysis of Underwater Imagery, pp. 65–72. IEEE (2014)

    Google Scholar 

  32. Ronneberger, O., Fischer, P., Brox, T.: U-net: convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) MICCAI 2015. LNCS, vol. 9351, pp. 234–241. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-24574-4_28

    Chapter  Google Scholar 

  33. Rosen, S., Holst, J.C.: Deepvision in-trawl imaging: sampling the water column in four dimensions. Fisher. Res. 148, 64–73 (2013)

    Article  Google Scholar 

  34. Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., Chen, L.C.: Mobilenetv 2: inverted residuals and linear bottlenecks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4510–4520 (2018)

    Google Scholar 

  35. Sekachev, B., et al.: opencv/cvat: v1.1.0 (2020). https://doi.org/10.5281/zenodo.4009388

  36. Sorensen, T.A.: A method of establishing groups of equal amplitude in plant sociology based on similarity of species content and its application to analyses of the vegetation on danish commons. Biol. Skar. 5, 1–34 (1948)

    Google Scholar 

  37. Spampinato, C., Giordano, D., Di Salvo, R., Chen-Burger, Y.H.J., Fisher, R.B., Nadarajan, G.: Automatic fish classification for underwater species behavior understanding. In: Proceedings of the First ACM International Workshop on Analysis and Retrieval of Tracked Events and Motion in Imagery Streams, pp. 45–50 (2010)

    Google Scholar 

  38. Treml, M., et al.: Speeding up semantic segmentation for autonomous driving (2016)

    Google Scholar 

  39. Wilhelms, I., et al.: Atlas of length-weight relationships of 93 fish and crustacean species from the north sea and the north-east atlantic. Technical report, Johann Heinrich von Thünen Institute, Federal Research Institute for Rural \(\ldots \) (2013)

    Google Scholar 

  40. Williams, K., Lauffenburger, N., Chuang, M.C., Hwang, J.N., Towler, R.: Automated measurements of fish within a trawl using stereo images from a camera-trawl device (camtrawl). Methods Oceanogr. 17, 138–152 (2016)

    Article  Google Scholar 

  41. Yadan, O.: Hydra - a framework for elegantly configuring complex applications. Github (2019). https://github.com/facebookresearch/hydra

  42. Yakubovskiy, P.: Segmentation models pytorch (2020). https://github.com/qubvel/segmentation_models.pytorch

  43. Yu, C., Wang, J., Peng, C., Gao, C., Yu, G., Sang, N.: Learning a discriminative feature network for semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1857–1866 (2018)

    Google Scholar 

  44. Yu, C., et al.: Segmentation and measurement scheme for fish morphological features based on mask r-cnn. Inf. Process. Agric. 7(4), 523–534 (2020)

    Google Scholar 

  45. Zarco-Perello, S., Enríquez, S.: Remote underwater video reveals higher fish diversity and abundance in seagrass meadows, and habitat differences in trophic interactions. Sci. Rep. 9(1), 1–11 (2019)

    Article  Google Scholar 

  46. Zhao, H., Shi, J., Qi, X., Wang, X., Jia, J.: Pyramid scene parsing network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2881–2890 (2017)

    Google Scholar 

Download references

Acknowledgements

This work was in part financially supported by the German Federal Ministry of Food and Agriculture (BMEL), grant number 2819111618. The planning and construction, the sensor integration and the operation of the UFO has been realized by the MacArtney Germany GmbH [2]. All species identifications of imaged underwater organisms, as well as the biological accompanying studies, were carried out by the Helmholtz Centre for Ocean Research Kiel [1]. The administrative project coordination and validation of biophysical data was done by the Thuenen institute [4]. The recording of the camera data was realised using the CamIQ software, which has been supplied by the rosemann software GmbH [3].

Author information

Authors and Affiliations

  1. Institute of Applied Computer Science, Kiel University of Applied Sciences, Kiel, Germany

    Gordon Böer & Hauke Schramm

  2. Department of Computer Science, Faculty of Engineering, Kiel University, Kiel, Germany

    Hauke Schramm

Authors
  1. Gordon Böer
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hauke Schramm
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Gordon Böer .

Editor information

Editors and Affiliations

  1. Óbuda University, Budapest, Hungary

    Péter Galambos

  2. Aarhus University, Aarhus, Denmark

    Erdal Kayacan

  3. University of Paris-EST Créteil (UPEC), Créteil, France

    Kurosh Madani

Rights and permissions

Reprints and permissions

Copyright information

© 2022 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Böer, G., Schramm, H. (2022). Semantic Segmentation of Marine Species in an Unconstrained Underwater Environment. In: Galambos, P., Kayacan, E., Madani, K. (eds) Robotics, Computer Vision and Intelligent Systems. ROBOVIS ROBOVIS 2020 2021. Communications in Computer and Information Science, vol 1667. Springer, Cham. https://doi.org/10.1007/978-3-031-19650-8_7

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-19650-8_7

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-19649-2

  • Online ISBN: 978-3-031-19650-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation