Machine Vision and Applications

, Volume 25, Issue 1, pp 99–117 | Cite as

A rule-based event detection system for real-life underwater domain

  • Concetto Spampinato
  • Emmanuelle Beauxis-Aussalet
  • Simone Palazzo
  • Cigdem Beyan
  • Jacco van Ossenbruggen
  • Jiyin He
  • Bas Boom
  • Xuan Huang
Special Issue Paper


Understanding and analyzing fish behaviour is a fundamental task for biologists that study marine ecosystems because the changes in animal behaviour reflect environmental conditions such as pollution and climate change. To support investigators in addressing these complex questions, underwater cameras have been recently used. They can continuously monitor marine life while having almost no influence on the environment under observation, which is not the case with observations made by divers for instance. However, the huge quantity of recorded data make the manual video analysis practically impossible. Thus machine vision approaches are needed to distill the information to be investigated. In this paper, we propose an automatic event detection system able to identify solitary and pairing behaviours of the most common fish species of the Taiwanese coral reef. More specifically, the proposed system employs robust low-level processing modules for fish detection, tracking and recognition that extract the raw data used in the event detection process. Then each fish trajectory is modeled and classified using hidden Markov models. The events of interest are detected by integrating end-user rules, specified through an ad hoc user interface, and the analysis of fish trajectories. The system was tested on 499 events of interest, divided into solitary and pairing events for each fish species. It achieved an average accuracy of 0.105, expressed in terms of normalized detection cost. The obtained results are promising, especially given the difficulties occurring in underwater environments. And moreover, it allows marine biologists to speed up the behaviour analysis process, and to reliably carry on their investigations.


Event detection Fish detection  Fish recognition  Trajectory classification  Behaviour understanding 


  1. 1.
    Gkalelis, N., Mezaris, V., Kompatsiaris, I.: High-level event detection in video exploiting discriminant concepts. In: 9th International Workshop on Content-Based Multimedia Indexing (CBMI 2011). Madrid, Spain, 06/2011 (2011)Google Scholar
  2. 2.
    Liao, M.-Y., Chen, D.-Y., Sua, C.-W., Tyan, H.-R.: Real-time event detection and its application to surveillance systems. In: Proceedings of 2006 IEEE International Symposium on Circuits and Systems, 2006. ISCAS 2006, vol. 4, p. 512 (2006)Google Scholar
  3. 3.
    Ballan, L., Bertini, M., Bimbo, A.D., Seidenari, L., Serra, G.: Event detection and recognition for semantic annotation of video. Multimedia Tools Appl. 51, 279–302 (2011)CrossRefGoogle Scholar
  4. 4.
    Spampinato, C., Chen-Burger, Y.-H., Nadarajan, G., Fisher, R.: Detecting, tracking and counting fish in low quality unconstrained underwater videos. In: Proceedings of 3rd International Conference on Computer Vision Theory and Applications (VISAPP), vol. 2, pp. 514–519 (2008)Google Scholar
  5. 5.
    Spampinato, C., Giordano, D., Di Salvo, R., Chen-Burger, Y.-H.J., Fisher, R.B., Nadarajan, G.: Automatic fish classification for underwater species behavior understanding. In: Proceedings of the first ACM international workshop on analysis and retrieval of tracked events and motion in imagery streams, pp. 45–50. ARTEMIS ’10, ACM, New York, NY, USA (2010)Google Scholar
  6. 6.
    Spampinato, C., Palazzo, S., Giordano, D., Kavasidis, I., Lin, F.-P., Lin, Y.-T.: Covariance based fish tracking in real-life underwater environment. In: VISAPP (2), pp. 409–414 (2012)Google Scholar
  7. 7.
    Rijnsdorp, A.D., Peck, M.A., Engelhard, G.H., Mšllmann, C., Pinnegar, J.K.: Resolving the effect of climate change on fish populations. ICES Journal of Marine Science: Journal du Conseil 66(7), 1570–1583 (2009)CrossRefGoogle Scholar
  8. 8.
    Scott, G.R., Sloman, K.A.: The effects of environmental pollutants on complex fish behaviour: integrating behavioural and physiological indicators of toxicity. Aquatic Toxicol 68(4), 369–392 (2004)CrossRefGoogle Scholar
  9. 9.
    Spampinato, C., Palazzo, S., Boom, B., van Ossenbruggen, J., Kavasidis, I., Di Salvo, R., Lin, F.-P., Giordano, D., Hardman, L., Fisher, R.: Understanding fish behavior during typhoon events in real-life underwater environments. Multimedia Tools Appl. pp. 1–38 (2012). doi: 10.1007/s11042-012-1101-5
  10. 10.
    Cupillard, F., Avanzi, A., Bremond, F., Thonnat, M.: Video understanding for metro surveillance. In: IEEE International Conference on Networking Sensing and Control, vol. 1, pp. 186–191, IEEE (2004)Google Scholar
  11. 11.
    Ke, Y., Sukthankar, R., Hebert, M.: Event detection in crowded videos. In: IEEE 11th International Conference on Computer Vision, vol. 23, pp. 1–8 (2007)Google Scholar
  12. 12.
    Zhang, Z., Huang, K., Tan, T., Wang, L.: Trajectory series analysis based event rule induction for visual surveillance. In: IEEE Conference on Computer Vision and, Pattern Recognition, pp. 1–8 (2007)Google Scholar
  13. 13.
    Haering, N., Qian, R.J., Sezan, M,I.: A semantic event-detection approach and its application to detecting hunts in wildlife video (2000)Google Scholar
  14. 14.
    Liao, M.-Y., Chen, D.-Y., Sua, C.-W., Tyan, H.-R.: Real-time event detection and its application to surveillance systems. In: Proceedings of the IEEE International Symposium on Circuits and Systems (2006)Google Scholar
  15. 15.
    Li, B., Ibrahim Sezan, M.: Event detection and summarization in sports video. In: Proceedings IEEE Workshop on Content-Based Access of Image and Video Libraries CBAIVL 2001, pp. 132–138 (2001)Google Scholar
  16. 16.
    Sadlier, D.A., O’Connor, N.E.: Event detection in field sports video using audio-visual features and a support vector Machine (2005)Google Scholar
  17. 17.
    Medioni, G., Cohen, I., Bremond, F., Hongeng, S., Nevatia, R.: Event detection and analysis from video streams. IEEE Trans. Pattern Anal. Mach. Intell. 23(8), 873–889 (2001)CrossRefGoogle Scholar
  18. 18.
    Assfalg, J., Bertini, M., Colombo, C., Bimbo, A.D., Nunziati, W.: Highlight extraction in soccer videos (2003)Google Scholar
  19. 19.
    Suzuki, N., Hirasawa, K., Tanaka, K., Kobayashi, Y., Sato, Y., Fujino, Y.: Learning motion patterns and anomaly detection by Human trajectory analysis. In: IEEE International Conference on Systems, Man and, Cybernetics, pp. 498–503 (2007)Google Scholar
  20. 20.
    Porikli, F., Haga, T.: Event detection by eigenvector decomposition using object and frame features. In: Conference on Computer Vision and Pattern Recognition, Workshop (2004)Google Scholar
  21. 21.
    Huang, C.-L., Shih, H.-C., Chao, C.-Y.: Semantic analysis of soccer video using dynamic Bayesian network (2006)Google Scholar
  22. 22.
    Piciarelli, C., Foresti, G.L., Snidaro, L.: Trajectory clustering and its applications for video surveillance. In: IEEE Conference on Advanced Video and Signal Based Surveillance (2005)Google Scholar
  23. 23.
    Zhan, B., Monekosso, D.N., Remagnino, P., Velastin, S.A., Xu, L.-Q.: Crowd analysis: a survey. Mach. Vision Appl. 19(5–6), 345–357 (2008)Google Scholar
  24. 24.
    Andrade, E.L., Blunsden, S., Fisher, R.B.: Modelling crowd scenes for event detection. In: 18th International Conference on Pattern Recognition, vol. 1, pp. 175–178 (2006)Google Scholar
  25. 25.
    Soori, U., Arshad, M.: Underwater crowd flow detection using Lagrangian dynamics. Indian J. Marine Sci. 38, 359–364 (2009)Google Scholar
  26. 26.
    Meila, M., Shi, J.: A random walks view of spectral segmentation. In: AISTATS, pp. 8–11. AISTATS (2001)Google Scholar
  27. 27.
    Rissanen, J.: Stochastic Complexity in Statistical Inquiry, Series in Computer Science, vol. 15. World Scientific, Singapore (1989)Google Scholar
  28. 28.
    Wang, F., Jiang, Y.-G., Ngo, C.-W.: Video event detection using motion relativity and visual relatedness. In: Proceedings of ACM multimedia (2008)Google Scholar
  29. 29.
    Branson, K., Robie, A.A., Bender, J., Perona, P., Dickinson, M.H.: High-throughput ethomics in large groups of Drosophila. Nat. Methods 6, 451–457 (2009)CrossRefGoogle Scholar
  30. 30.
    Palmer, T., Tamte, M., Halje, P., Enqvist, O., Petersson, P.: A system for automated tracking of motor components in neurophysiological research. J. Neurosci. Methods 205, 334–344 (2012)CrossRefGoogle Scholar
  31. 31.
    Poppe, R.: A survey on vision-based human action recognition. Image Vision Comput. 28, 976–990 (2010)CrossRefGoogle Scholar
  32. 32.
    Burgos-Artizzu, X., Dollár, P., Lin, D., Anderson, D., Perona, P.: Social behavior recognition in continuous videos. In: CVPR (2012)Google Scholar
  33. 33.
    Haritaoglu, I., Harwood, D., Davis, L.S.: W4: Who? when? where? what? a real time system for detecting and tracking people. In: Proceedings of the International Conference on Automatic Face and Gesture Recognition, vol. 1, (Nara, Japan), pp. 222–227 (2008)Google Scholar
  34. 34.
    Faro, A., Giordano, D., Spampinato, C.: Adaptive background modeling integrated with luminosity sensors and occlusion processing for reliable vehicle detection. IEEE Trans. Intell. Transportation Syst. 12, 1398–1412 (2011)CrossRefGoogle Scholar
  35. 35.
    Porikli, F.: Achieving real-time object detection and tracking under extreme conditions. J. Real-Time Image Process. 1(1), 33–40 (2006)CrossRefGoogle Scholar
  36. 36.
    Stauffer, C., Grimson, W.E.L.: Adaptive background mixture models for real-time tracking. In: Proceedings 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Cat No PR00149, 2(c), 246–252 (1999)Google Scholar
  37. 37.
    Faro, A., Giordano, D., Spampinato, C.: Integrating location tracking, traffic monitoring and semantics in a layered its architecture. IET Intell. Transport Syst. 5(3), 197–206 (2011)CrossRefGoogle Scholar
  38. 38.
    Porikli, F., Wren, C.: Change detection by frequency decomposition: Wave-back. In: Proceedings of Workshop on Image Analysis for Multimedia Interactive Services (2005)Google Scholar
  39. 39.
    Porikli, F.: Multiplicative background-foreground estimation under uncontrolled illumination using intrinsic images. In: Proceedings of IEEE Motion Multi-Workshop (2005)Google Scholar
  40. 40.
    Barnich, O., Van Droogenbroeck, M.: ViBe: a universal background subtraction algorithm for video sequences. IEEE Trans. Image Process. 20, 1709–1724 (2011)CrossRefMathSciNetGoogle Scholar
  41. 41.
    Porikli, F.: Change detection by frequency decomposition: Wave-back. In: Proceedings of Workshop on Image Analysis for Multimedia Interactive Services (2005)Google Scholar
  42. 42.
    Schapire, R.E., Singer, Y.: Improved boosting algorithms using confidence-rated predictions. Mach. Learn. 37, 297–336 (1999)CrossRefzbMATHGoogle Scholar
  43. 43.
    Kim, K., Chalidabhongse, T., Harwood, D., Davis, L.: Background modeling and subtraction by codebook construction. In: International Conference on Image Processing, 2004. ICIP ’04. 2004, vol. 5, pp. 3061–3064 (2004)Google Scholar
  44. 44.
    Spampinato, C., Palazzo, S.: Enhancing object detection performance by integrating motion objectness and perceptual organization. In: Proceedings of IEEE International Conference on, Pattern Recognition, pp. 3640–3643 (2012)Google Scholar
  45. 45.
    Alexe, B., Deselaers, T., Ferrari, V.: Measuring the objectness of image windows. In: IEEE Transactions on PAMI, vol. 99, PrePrints (2012)Google Scholar
  46. 46.
    Cheng, C., Koschan, A., Chen, C.-H., Page, D.L., Abidi, M.A.: Outdoor scene image segmentation based on background recognition and perceptual organization. IEEE Trans. Image Process. 21(3), 1007–1019 (2012)CrossRefMathSciNetGoogle Scholar
  47. 47.
    Rother, C., Kolmogorov, V., Blake, A.: GrabCut: interactive foreground extraction using iterated graph cuts. ACM Trans. Graphics (TOG), pp. 309–314 (2004) Google Scholar
  48. 48.
    He, X.C., Yung, N.H.C.: Curvature scale space corner detector with adaptive threshold and dynamic region of support. In: International Conference on Pattern Recognition, vol. 2, pp. 791–794. IEEE Computer Society, Los Alamitos, CA, USA (2004)Google Scholar
  49. 49.
    Mokhtarian, F., Suomela, R.: Robust image corner detection through curvature scale space. IEEE Trans. Pattern Anal. Mach. Intell. 20(12), 1376–1381 (1998)CrossRefGoogle Scholar
  50. 50.
    Spampinato, C., Giordano, D., Salvo, R.D., Chen-Burger, Y.H., Fisher, R.B., Nadarajan, G.: Automatic fish classification for underwater species behavior understanding. In: Proceedings of the first ACM international workshop on analysis and retrieval of tracked events and motion in imagery streams, New York, NY, USA, pp. 45–50 (2010)Google Scholar
  51. 51.
    Chih-Chung, C., Chih-Jen, L.: LIBSVM: a library for support vector machines. ACM Trans. Intell. Syst. Technol. 2(3), 1–27 (2011)CrossRefGoogle Scholar
  52. 52.
    Porikli, F., Tuzel, O., Meer, P.: Covariance tracking using model update based on lie algebra. In: Proceedings IEEE Conference on Computer Vision and Pattern Recognition (2005)Google Scholar
  53. 53.
    Rabiner, L.R.: A tutorial on hidden Markov models and selected applications in speech recognition. Proc. IEEE 77(2), 257–286 (1989)CrossRefGoogle Scholar
  54. 54.
    Kavasidis, I., Palazzo, S., Di Salvo, R., Giordano, D., Spampinato, C.: A semi-automatic tool for detection and tracking ground truth generation in videos. In: VIGTA ’12: Proceedings of the 1st International Workshop on Visual Interfaces for Ground Truth Collection in Computer Vision Applications, pp. 1–5, ACM (2012)Google Scholar
  55. 55.
    Isard, M., Blake, A.: Condensation–conditional density propagation for visual tracking. Int. J. Comput. Vision 29(1), 5–28 (1998)CrossRefGoogle Scholar
  56. 56.
    Lazarevic-McManus, N., Renno, J., Jones, G.A.: Performance evaluation in visual surveillance using the f-measure. In: Proceedings of the 4th ACM international workshop on Video surveillance and sensor networks, VSSN ’06, pp. 45–52, ACM, New York, NY, USA (2006)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  • Concetto Spampinato
    • 1
  • Emmanuelle Beauxis-Aussalet
    • 2
  • Simone Palazzo
    • 1
  • Cigdem Beyan
    • 3
  • Jacco van Ossenbruggen
    • 2
  • Jiyin He
    • 2
  • Bas Boom
    • 3
  • Xuan Huang
    • 3
  1. 1.Department of Electrical, Electronics and Computer EngineeringUniversity of CataniaCataniaItaly
  2. 2.Centrum voor Wiskunde en Informatica (CWI)AmsterdamNetherlands
  3. 3.School of InformaticsUniversity of EdinburghEdinburghUK

Personalised recommendations