Saliency Subtraction Inspired Automated Event Detection in Underwater Environments

  • Nitin Kumar
  • H. K. SardanaEmail author
  • S. N. Shome
  • Neerja Mittal


Unmanned underwater exploration in unconstrained environments is a challenging problem. Analysis of the large volumes of images/videos captured by underwater stations/vehicles manually is a major bottleneck for further research. Existing computer vision methods either do not target unconstrained underwater environments or they only aim to detect static or moving entities. In this paper, we present a novel method for analyzing underwater videos and detecting events. Entry/exit of an object in scene is treated as an event independent of the other objects present therein. The method is applied on underwater videos with no prior knowledge, thus aiding in automated underwater exploration. The method is inspired by the fact that saliency of objects in the scene is invariant of the surrounding environment. The proposed method is composed of three main steps: Local Patch Saliency, Adaptive Saliency Subtraction, and event generation for analyzing underwater imagery from the videos. The method is aimed at detecting overlapping events containing man-made as well as natural objects including those containing multiple objects in the unconstrained underwater conditions. The performance of the method is evaluated on publicly available videos obtained from Ocean Networks Canada and Fish4Knowledge datasets. Ground truth for Ocean Networks Canada videos is not available; hence, a method for generating the same for varied sources is also presented. The algorithm achieves a precision of 98% for event detection with 20% misclassification rate. The results show the robustness of the method that performs even in complex and varying underwater conditions.


Adaptive Saliency Subtraction Event detection Ground truth generation Local Patch Saliency 



The authors are grateful to Dr. Maia Hoeberechts and team for providing the Ocean Networks Canada Dataset. We are also thankful to Akanksha Pathania, Parminder Kaur, Gifty Aggarwal, and Neha for assisting us in generating the ground truth for the underwater videos.

Funding Information

Nitin Kumar is thankful to the Council of Scientific and Industrial Research - Central Scientific Instruments Organisation (CSIR-CSIO), Chandigarh for providing the funding and opportunity to carry out this work at CSIR-CSIO.

Compliance with Ethical Standards

Conflict of interest

The authors declare that they have no conflict of interest.

Ethical approval

This article does not contain any studies with human participants or animals performed by any of the authors.

Informed Consent

Informed consent is not necessary for the present study.


  1. 1.
    Yuh J. Design and control of autonomous underwater robots: a survey. Auton Robots 2000;8(1):7–24.CrossRefGoogle Scholar
  2. 2.
    Griffiths G (ed.) Technology and applications of autonomous underwater vehicles. CRC Press; 2002 Nov 28.Google Scholar
  3. 3.
    Tu Z, Abel A, Zhang L, Luo B, Hussain A. A new spatio-temporal saliency-based video object segmentation. Cogn Comput 2016;8(4):629–47.CrossRefGoogle Scholar
  4. 4.
    Aboudib A, Gripon V, Coppin G. A biologically inspired framework for visual information processing and an application on modeling bottom-up visual attention. Cogn Comput 2016;8(6):1007–26.CrossRefGoogle Scholar
  5. 5.
    Jia X, Li X, Jin Y, Miao J. Region-enhanced multi-layer extreme learning machine. Cogn Comput 2019;11(1):101–9.CrossRefGoogle Scholar
  6. 6.
    Wang H, Xu L, Wang X, Luo B. Learning optimal seeds for ranking saliency. Cogn Comput 2018; 10(2):347–58.CrossRefGoogle Scholar
  7. 7.
    Zheng A, Xu M, Luo B, Zhou Z, Li C. CLASS collaborative low-rank and sparse separation for moving object detection. Cognit Comput 2017;9(2):180–93.CrossRefGoogle Scholar
  8. 8.
    Olmos A, Trucco E. 2002. Detecting man-made objects in unconstrained subsea videos. In: BMVC. p. 1–10.Google Scholar
  9. 9.
    Edgington DR, Salamy KA, Risi M, Sherlock RE, Walther D, Koch C. Automated event detection in underwater video. Oceans 2003. Celebrating the past teaming toward the future (IEEE Cat. No. 03CH37492). IEEE; 2003. p. P2749–53.Google Scholar
  10. 10.
    Walther D, Edgington DR, Koch C. Detection and tracking of objects in underwater video. Proceedings of the 2004 IEEE computer society conference on computer vision and pattern recognition, 2004. CVPR 2004. IEEE; 2004. p. I–I.Google Scholar
  11. 11.
    Kabatek M, Azimi-Sadjadi MR, Tucker JD. An underwater target detection system for electro-optical imagery data. OCEANS. IEEE; 2009. p. 1–8.Google Scholar
  12. 12.
    Cucchiara R, Grana C, Piccardi M, Prati A. Detecting moving objects, ghosts, and shadows in video streams. IEEE Trans Pattern Anal Mach Intell 2003;25(10):1337–42.CrossRefGoogle Scholar
  13. 13.
    Lipton AJ, Fujiyoshi H, Patil RS. Moving target classification and tracking from real-time video. Proceedings fourth IEEE workshop on applications of computer vision. WACV’98 (Cat. No. 98EX201). IEEE; 1998. p. 8–14.Google Scholar
  14. 14.
    Piccardi M. Background subtraction techniques: a review. 2004 IEEE International conference on systems, man and cybernetics (IEEE Cat. No. 04CH37583). IEEE; 2004. p. 3099–104.Google Scholar
  15. 15.
    Spampinato C, Palazzo S. Enhancing object detection performance by integrating motion objectness and perceptual organization. Proceedings of the 21st international conference on pattern recognition (ICPR2012). IEEE; 2012. p. 3640–3.Google Scholar
  16. 16.
    Palazzo S, Kavasidis I, Spampinato C. Covariance based modeling of underwater scenes for fish detection. 2013 IEEE International conference on image processing. IEEE; 2013. p. 1481–5.Google Scholar
  17. 17.
    Spampinato C, Palazzo S, Kavasidis I. A texton-based kernel density estimation approach for background modeling under extreme conditions. Comput Vis Image Understand 2014;122:74–83.CrossRefGoogle Scholar
  18. 18.
    Heikkilä M, Pietikäinen M, Schmid C. Description of interest regions with local binary patterns. Pattern Recogn 2009;42(3):425–36.CrossRefGoogle Scholar
  19. 19.
    Liao S, Zhao G, Kellokumpu V, Pietikäinen M, Li SZ. Modeling pixel process with scale invariant local patterns for background subtraction in complex scenes. 2010 IEEE Computer society conference on computer vision and pattern recognition. IEEE; 2010. p. 1301–6.Google Scholar
  20. 20.
    Spampinato C, Chen-Burger YH, Nadarajan G, Fisher RB. Detecting, tracking and counting fish in low quality unconstrained underwater videos. VISAPP (2) 2008;2008(514–9):1.Google Scholar
  21. 21.
    Spampinato C, Beauxis-Aussalet E, Palazzo S, Beyan C, van Ossenbruggen J, He J, Boom B, Huang X. A rule-based event detection system for real-life underwater domain. Mach Vis Appl 2014;25(1): 99–117.CrossRefGoogle Scholar
  22. 22.
    Akilan T, Wu QJ, Yang Y. Fusion-based foreground enhancement for background subtraction using multivariate multi-model Gaussian distribution. Inform Sci 2018;430:414–31.CrossRefGoogle Scholar
  23. 23.
    Minematsu T, Shimada A, Uchiyama H, Taniguchi RI. Analytics of deep neural network-based background subtraction. J Imag 2018;4(6):78.CrossRefGoogle Scholar
  24. 24.
    Rova A, Mori G, Dill LM. One fish, two fish, butterfish, trumpeter: recognizing fish in underwater video. InMVA. 2007; 404–7.Google Scholar
  25. 25.
    Shortis MR, Ravanbakhsh M, Shafait F, Mian A. Progress in the automated identification, measurement, and counting of fish in underwater image sequences. Marine Technol Soc J 2016;50(1):4–16.CrossRefGoogle Scholar
  26. 26.
    Vasamsetti S, Mittal N, Neelapu BC, Sardana HK. 3D local spatio-temporal ternary patterns for moving object detection in complex scenes. Cogn Comput 2019;11(1):18–30.CrossRefGoogle Scholar
  27. 27.
    Oliver K, Hou W, Wang S. 2010. Image feature detection and matching in underwater conditions, Vol. 7678: International Society for Optics and Photonics.Google Scholar
  28. 28.
    Kavasidis I, Palazzo S. Quantitative performance analysis of object detection algorithms on underwater video footage. Proceedings of the 1st ACM international workshop on multimedia analysis for ecological data. ACM; 2012. p. 57–60.Google Scholar
  29. 29.
    Han KM, Choi HT. Shape context based object recognition and tracking in structured underwater environment. 2011 IEEE International geoscience and remote sensing symposium. IEEE; 2011. p. 617–20.Google Scholar
  30. 30.
    Kim D, Lee D, Myung H, Choi HT. Object detection and tracking for autonomous underwater robots using weighted template matching. 2012 Oceans-Yeosu. IEEE; 2012. p. 1–5.Google Scholar
  31. 31.
    Leonard I, Arnold-Bos A, Alfalou A. Interest of correlation-based automatic target recognition in underwater optical images: theoretical justification and first results. Ocean sensing and monitoring II. International Society for Optics and Photonics; 2010. p. 76780O.Google Scholar
  32. 32.
    Barat C, Rendas MJ. A robust visual attention system for detecting manufactured objects in underwater video. OCEANS 2006. IEEE; 2006. p. 1–6.Google Scholar
  33. 33.
    Barat C, Phlypo R. A fully automated method to detect and segment a manufactured object in an underwater color image. EURASIP J Adv Signal Process 2010;1:10.Google Scholar
  34. 34.
    Itti L, Koch C, Niebur E. A model of saliency-based visual attention for rapid scene analysis. IEEE Trans Pattern Anal Mach Intell 1998;1(11):1254–9.CrossRefGoogle Scholar
  35. 35.
    Wang HB, Dong X, Shen J, Wu XW, Chen Z. Saliency-based adaptive object extraction for color underwater images. Applied mechanics and materials. Trans Tech Publications; 2013. p. 3964–70.Google Scholar
  36. 36.
    Bazeille S, Quidu I, Jaulin L. Identification of underwater man-made object using a colour criterion. Conference on detection and classification of underwater targets; 2007. p. xx.Google Scholar
  37. 37.
    Maldonado-Ramírez A, Torres-Méndez LA. 2016. Robotic visual tracking of relevant cues in underwater environments with poor visibility conditions. J Sensors.Google Scholar
  38. 38.
    Gebali A, Albu AB, Hoeberechts M. Detection of salient events in large datasets of underwater video. IEEE; 2012. p. 14.Google Scholar
  39. 39.
    Oliva A, Torralba A, Castelhano MS, Henderson JM. Top-down control of visual attention in object detection. Proceedings 2003 international conference on image processing (Cat. No. 03CH37429). IEEE; 2003. p. I-253.Google Scholar
  40. 40.
    Rutishauser U, Walther D, Koch C, Perona P. Is bottom-up attention useful for object recognition? Proceedings of the 2004 IEEE computer society conference on computer vision and pattern recognition, 2004. CVPR 2004. IEEE; 2004. p. II-II.Google Scholar
  41. 41.
    Bosch A, Zisserman A, Muñoz X. Scene classification using a hybrid generative/discriminative approach. IEEE Trans Pattern Anal Mach Intell 2008;30(4):712–27.CrossRefGoogle Scholar
  42. 42.
    Kinchla RA, Wolfe JM. The order of visual processing:”Top-down,””bottom-up,” or ”middle-out”. Percept Psychophys 1979;25(3):225–31.CrossRefGoogle Scholar
  43. 43.
    Zelnik-Manor L, Irani M. Event-based analysis of video. InCVPR (2) 2001;8:123–30.Google Scholar
  44. 44.
    Ke Y, Sukthankar R, Hebert M. Event detection in crowded videos. 2007 IEEE 11th international conference on computer vision. IEEE; 2007. p. 1–8.Google Scholar
  45. 45.
    Borji A, Itti L. State-of-the-art in visual attention modeling. IEEE Trans Pattern Anal Mach Intell 2013; 35(1):185–207.CrossRefGoogle Scholar
  46. 46.
    Li H, Ngan KN. A co-saliency model of image pairs. IEEE Trans Image Process 2011;20(12):3365–75.CrossRefGoogle Scholar
  47. 47.
    Meng F, Li H, Liu G, Ngan KN. Object co-segmentation based on shortest path algorithm and saliency model. IEEE Trans Multimed 2012;14(5):1429–41.CrossRefGoogle Scholar
  48. 48.
    Borji A, Itti L. Exploiting local and global patch rarities for saliency detection. 2012 IEEE conference on computer vision and pattern recognition. IEEE; 2012. p. 478–85.Google Scholar
  49. 49.
    Borji A, Sihite DN, Itti L. Quantitative analysis of human-model agreement in visual saliency modeling: a comparative study. IEEE Trans Image Process 2013;22(1):55–69.CrossRefGoogle Scholar
  50. 50.
    Lin Y, Tong Y, Cao Y, Zhou Y, Wang S. Visual-attention-based background modeling for detecting infrequently moving objects. IEEE Trans Circ Syst Video Technol 2017;27(6):1208–21.CrossRefGoogle Scholar
  51. 51.
    Sezgin M, Sankur B. Survey over image thresholding techniques and quantitative performance evaluation. J Electron Imag 2004;13(1):146–66.CrossRefGoogle Scholar
  52. 52.
    Kumar N, Sardana HK, Shome SN. Saliency based shape extraction of objects in unconstrained underwater environment. Multimed Tools Appl 2019;78(11):15121–39.CrossRefGoogle Scholar
  53. 53.
    Kavasidis I, Palazzo S, Di Salvo R, Giordano D, Spampinato C. An innovative web-based collaborative platform for video annotation. Multimed Tools Appl 2014;70(1):413–32.CrossRefGoogle Scholar
  54. 54.
    Barnes CR, Best MM, Bornhold BD, Juniper SK, Pirenne B, Phibbs P. The NEPTUNE project-a cabled ocean observatory in the NE Pacific: overview, challenges and scientific objectives for the installation and operation of Stage I in Canadian waters. 2007 Symposium on underwater technology and workshop on scientific use of submarine cables and related technologies. IEEE; 2007. p. 308–13.Google Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2019

Authors and Affiliations

  1. 1.Academy of Scientific and Innovative Research (AcSIR)New DelhiIndia
  2. 2.CSIR-Central Scientific Instruments Organisation (CSIR-CSIO)ChandigarhIndia
  3. 3.CSIR-Central Mechanical Engineering Research Institute (CSIR-CMERI)DurgapurIndia

Personalised recommendations