Machine Vision and Applications

, Volume 26, Issue 1, pp 89–102 | Cite as

Hierarchical classification with reject option for live fish recognition

  • Phoenix X. Huang
  • Bastiaan J. Boom
  • Robert B. Fisher
Original Paper


A live fish recognition system is needed in application scenarios where manual annotation is too expensive, i.e. too many underwater videos. We present a novel balance-enforced optimized tree with reject option (BEOTR) for live fish recognition. It recognizes the top 15 common species of fish and detects new species in an unrestricted natural environment recorded by underwater cameras. The three main contributions of the paper are: (1) a novel hierarchical classification method suited for greatly unbalanced classes, (2) a novel classification-rejection method to clear up decisions and reject unknown classes, (3) an application of the classification method to free swimming fish. This system assists ecological surveillance research, e.g. fish population statistics in the open sea. BEOTR is automatically constructed based on inter-class similarities. Afterwards, trajectory voting is used to eliminate accumulated errors during hierarchical classification and, therefore, achieves better performance. We apply a Gaussian mixture model and Bayes rule as a reject option after the hierarchical classification to evaluate the posterior probability of being a certain species to filter less confident decisions. The proposed BEOTR-based hierarchical classification method achieves significant improvements compared to state-of-the-art techniques on a live fish image dataset of 24,150 manually labelled images from South Taiwan Sea.


BEOTR Live fish recognition Hierarchical classification Reject option GMM 

Supplementary material

138_2014_641_MOESM1_ESM.pdf (244 kb)
ESM 1 (PDF 244 kb)


  1. 1.
    Lee, D., Schoenberger, R.B., Shiozawa, D., Xu, X.Q., Zhan, P.C.: Contour matching for a fish recognition and migration-monitoring system. Proc. SPIE 5606(1), 37–48 (2004)CrossRefGoogle Scholar
  2. 2.
    Ruff, B.P., Marchant, J.A., Frost, A.R.: Fish sizing and monitoring using a stereo image analysis system applied to fish farming. Aquac. Eng. 14(2), 155–173 (1995)CrossRefGoogle Scholar
  3. 3.
    Strachan, N.J.C., Nesvadba, P., Allen, A.R.: Fish species recognition by shape analysis of images. Pattern Recognit. 23(5), 539–544 (1990)CrossRefGoogle Scholar
  4. 4.
    Okamoto, M., Morita, S., Sato, T.: Fundamental study to estimate fish biomass around coral reef using 3-dimensional underwater video system. In: Proceedings of the OCEANS 2000 MTS/IEEE Conference and Exhibition, vol. 2, pp. 1389–1392 (2000)Google Scholar
  5. 5.
    Walther, D., Edgington, D.R., Koch, C.: Detection and tracking of objects in underwater video. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol. 1, pp. 544–549 (2004)Google Scholar
  6. 6.
    Rova, A., Mori, G., Dill, L.M.: One fish, two fish, butterfish, trumpeter: recognizing fish in underwater video. In: IAPR Conference on Machine Vision Applications, pp. 404–407 (2007)Google Scholar
  7. 7.
    Zion, B., Shklyar, A., Karplus, I.: In-vivo fish sorting by computer vision. Aquac. Eng. 22, 165–179 (June 2000)Google Scholar
  8. 8.
    Heithaus, M.R., Dill, L.M.: Food availability and tiger shark predation risk influence bottlenose dolphin habitat use. Ecology 83(2), 480–491 (2002)CrossRefGoogle Scholar
  9. 9.
    Strachan, N.J.C.: Length measurement of fish by computer vision. Comput. Electron. Agric. 8(2), 93–104 (1993)CrossRefGoogle Scholar
  10. 10.
    Toh, Y.H., Ng, T.M., Liew, B.K.: Automated fish counting using image processing. In: Proceedings of the International Conference on Computational Intelligence and Software Engineering, pp. 1–5 (2009)Google Scholar
  11. 11.
    Schettini, R., Corchs, S.: Underwater image processing: state of the art of restoration and image enhancement methods. EURASIP J. Adv. Signal Process 2010, 1–7 (Jan. 2010)Google Scholar
  12. 12.
    Strachan, N.J.C.: Recognition of fish species by colour and shape. Image Vis. Comput. 11, 2–10 (Jan. 1993)Google Scholar
  13. 13.
    Spampinato, C., Giordano, D., Salvo, R.D., Chen-Burger, Y.H., Fisher, R.B., Nadarajan, G.: Automatic fish classification for underwater species behavior understanding. In: Proceedings of the First ACM International Workshop on Analysis And Retrieval of Tracked Events and Motion in Imagery Streams, pp. 45–50 (2010)Google Scholar
  14. 14.
    Larsen, R., Ólafsdóttir, H., Ersbøll, B.: Shape and texture based classification of fish species. In: Proceedings of SCIA, pp. 745–749 (2009)Google Scholar
  15. 15.
    Caley, M.J., Carr, M.H., Hixon, M.A., Hughes, T.P., Jones, G.P., Menge, B.A.: Recruitment and the local dynamics of open marine populations. Annu. Rev. Ecol. Syst. 27, 477–500 (Jan. 1996)Google Scholar
  16. 16.
    Brehmer, P., Chi, T.D., Mouillot, D.: Amphidromous fish school migration revealed by combining fixed sonar monitoring (horizontal beaming) with fishing data. J. Exp. Mar. Biol. Ecol. 334, 139–150 (June 2006)Google Scholar
  17. 17.
    Nadarajan, G., Chen-Burger, Y., Fisher, R., Spampinato, C.: A flexible system for automated composition of intelligent video analysis. In: Proceedings of ISPA, pp. 259–264 (2011)Google Scholar
  18. 18.
    Chih-Chung, C., Chih-Jen, L.: LIBSVM: a library for support vector machines. ACM Trans. Intell. Syst. Technol. 2(3), 1–27 (2011)CrossRefGoogle Scholar
  19. 19.
    Cortes, C., Vapnik, V.: Support-vector networks. Mach. Learn. 20, 273–297 (Sept. 1995)Google Scholar
  20. 20.
    Duan, K.-B., Keerthi, S.S.: Which is the best multiclass SVM method? An empirical study. In: Proceedings of the 6th International Conference on Multiple Classifier Systems (MCS’05), pp. 278–285. Springer, New York (2005)Google Scholar
  21. 21.
    Carlos, S., Alex, F.: A survey of hierarchical classification across different application domains. Data Min. Knowl. Discov. 22(1–2), 31–72 (2010)Google Scholar
  22. 22.
    Deng, J., Berg, A., Li, K., Fei-Fei, L.: What does classifying more than 10,000 image categories tell us? ECCV 6315, 71–84 (2010)Google Scholar
  23. 23.
    Gordon, A.D.: A review of hierarchical classification. J. R. Stat. Soc. 150(2), 119–137 (1987)zbMATHGoogle Scholar
  24. 24.
    Mathis, C.: Classification using a hierarchical bayesian approach. In: Proceedings of ICPR, vol. 4, pp. 103–106 (2002)Google Scholar
  25. 25.
    Deng, J., Dong, W., Socher, R., Li, L., Li, K., Fei-Fei, L.: ImageNet: a large-scale hierarchical image database. In: Proceedings of CVPR, pp. 248–255 (2009)Google Scholar
  26. 26.
    Platt, J.C.: Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. In: Advances in Large Margin Classifiers, pp. 61–74. MIT Press, USA (1999)Google Scholar
  27. 27.
    Wang, Y.-C.F., Casasent, D.: A support vector hierarchical method for multi-class classification and rejection. In: Proceedings of the International Joint Conference on Neural Networks (IJCNN’09), pp. 3281–3288 (2009)Google Scholar
  28. 28.
    Rother, C., Kolmogorov, V., Blake, A.: GrabCut: interactive foreground extraction using iterated graph cuts. ACM Trans. Graph. (SIGGRAPH), 23, 309–314 (2004)Google Scholar
  29. 29.
    Mokhtarian, F., Suomela, R.: Robust image corner detection through curvature scale space. IEEE Trans. PAMI 20(12), 1376–1381 (1998)CrossRefGoogle Scholar
  30. 30.
    He, X.C., Yung, N.H.C.: Curvature scale space corner detector with adaptive threshold and dynamic region of support. In: Proceedings of the International Conference on Pattern Recognition, vol. 2, pp. 791–794. IEEE Computer Society (2004)Google Scholar
  31. 31.
    Saeys, Y., Inza, In, Larrañaga, P.: A review of feature selection techniques in bioinformatics. Bioinformatics 23(19), 2507–2517 (2007)CrossRefGoogle Scholar
  32. 32.
    Flusser, J., Zitova, B., Suk, T.: Moments and moment invariants in pattern recognition. Wiley (2009)Google Scholar
  33. 33.
    Flusser, J., Suk, T., Zitova, B.: Affine moment invariants. In: Moments and Moment Invariants in Pattern Recognition, p. 49C112, Wiley, New York (2009)Google Scholar
  34. 34.
    Bosch, A., Zisserman, A., Munoz, X.: Representing shape with a spatial pyramid kernel. In: Proceedings of the 6th ACM International Conference on Image and Video Retrieval (CIVR’07), p. 401C408. ACM, New York (2007)Google Scholar
  35. 35.
    Bishop, C.M.: Neural Networks for Pattern Recognition. Oxford University Press Inc, Oxford (1995)Google Scholar
  36. 36.
    Mckenna, S.J., Gong, S., Raja, Y.: Modelling facial colour and identity with Gaussian mixtures. Pattern Recognit. 31, 1883–1892 (1998)CrossRefGoogle Scholar
  37. 37.
    Shental, N., Bar-hillel, A., Hertz, T., Weinshall, D.: Computing Gaussian mixture models with EM using equivalence constraints. In: Advances in Neural Information Processing Systems, vol. 16, MIT Press, USA (2003)Google Scholar
  38. 38.
    Figueiredo, M.A.T., Jain, A.: Unsupervised learning of finite mixture models. IEEE Trans. Pattern Anal. Mach. Intell. 24(3), 381–396 (2002)CrossRefGoogle Scholar
  39. 39.
    Zhao, Y., Karypis, G., Du, D.-Z.: Criterion functions for document clustering. University of Minnesota (2005)Google Scholar
  40. 40.
    Rissanen, J.: Stochastic complexity and modeling. Ann. Stat. 14, 1080–1100 (1986)Google Scholar
  41. 41.
    Boom, B., Huang, P., He, J., Fisher, R.: Supporting ground-truth annotation of image datasets using clustering. In: Proceedings of the 21st International Conference on Pattern Recognition (ICPR), pp. 1542–1545 (2012)Google Scholar
  42. 42.
    Hastie, T., Tibshirani, R., Friedman, J.J.H.: The Elements of Statistical Learning, vol. 1. Springer, New York (2001)CrossRefzbMATHGoogle Scholar
  43. 43.
    Chib, S.: Marginal likelihood from the Gibbs output. J. Am. Stat. Assoc. 90(432), 1313–1321 (1995)CrossRefzbMATHMathSciNetGoogle Scholar
  44. 44.
    Huang, P.X., Boom, B.J., Fisher, R.B.: Underwater live fish recognition using a balance-guaranteed optimized tree. In: Computer Vision-ACCV 2012, pp. 422–433 (2013)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2014

Authors and Affiliations

  • Phoenix X. Huang
    • 1
  • Bastiaan J. Boom
    • 1
  • Robert B. Fisher
    • 2
  1. 1.EdinburghUK
  2. 2.EdinburghUK

Personalised recommendations