Behavior Research Methods

, Volume 47, Issue 4, pp 1020–1031 | Cite as

Classification of collective behavior: a comparison of tracking and machine learning methods to study the effect of ambient light on fish shoaling

  • Sachit Butail
  • Philip Salerno
  • Erik M. Bollt
  • Maurizio Porfiri


Traditional approaches for the analysis of collective behavior entail digitizing the position of each individual, followed by evaluation of pertinent group observables, such as cohesion and polarization. Machine learning may enable considerable advancements in this area by affording the classification of these observables directly from images. While such methods have been successfully implemented in the classification of individual behavior, their potential in the study collective behavior is largely untested. In this paper, we compare three methods for the analysis of collective behavior: simple tracking (ST) without resolving occlusions, machine learning with real data (MLR), and machine learning with synthetic data (MLS). These methods are evaluated on videos recorded from an experiment studying the effect of ambient light on the shoaling tendency of Giant danios. In particular, we compute average nearest-neighbor distance (ANND) and polarization using the three methods and compare the values with manually-verified ground-truth data. To further assess possible dependence on sampling rate for computing ANND, the comparison is also performed at a low frame rate. Results show that while ST is the most accurate at higher frame rate for both ANND and polarization, at low frame rate for ANND there is no significant difference in accuracy between the three methods. In terms of computational speed, MLR and MLS take significantly less time to process an image, with MLS better addressing constraints related to generation of training data. Finally, all methods are able to successfully detect a significant difference in ANND as the ambient light intensity is varied irrespective of the direction of intensity change.


Giant danio Group observable Isomap Social behavior 



This research was supported by the National Science Foundation under grants nos. CMMI-0745753, CMMI-1129820, and CMMI-1129859. The authors would like to thank Fabrizio Ladu and Tiziana Bartolini for performing verification of the ground-truth data.


  1. Abaid, N., Bollt, E., Porfiri, M. (2012). Topological analysis of complexity in multiagent systems. Physical Review E, 85(4), 041907. doi: 10.1103/PhysRevE.85.041907 CrossRefGoogle Scholar
  2. Baek, J., Cosman, P., Feng, Z., Silver, J., Schafer, W.R. (2002). Using machine vision to analyze and classify Caenorhabditis elegans behavioral phenotypes quantitatively. Journal of Neuroscience Methods, 118(1), 9–21. doi: 10.1016/S0165-0270(02)00117-6 CrossRefPubMedGoogle Scholar
  3. Bar-Shalom, Y. (1987). Tracking and data association. San Diego: Academic Press Professional Inc.Google Scholar
  4. Bishop, C. (2006). Pattern recognition and machine learning. Springer.Google Scholar
  5. Bobick, A., & Davis, J. (2001). The recognition of human movement using temporal templates. IEEE Transactions on Pattern Analysis and Machine Intelligence, 23(3), 257–267. doi: 10.1109/34.910878 CrossRefGoogle Scholar
  6. Bohil, C.J., Alicea, B., Biocca, F.A. (2011). Virtual reality in neuroscience research and therapy. Nature Reviews Neuroscience, 12(12), 752–762. doi: 10.1038/nrn3122 PubMedGoogle Scholar
  7. Brunelli, R., & Poggio, T. (1993). Face recognition: Features versus templates. IEEE Transactions on Pattern Analysis and Machine Intelligence, 15(10), 1042–1052. doi: 10.1109/34.254061 CrossRefGoogle Scholar
  8. Buhmann, M. (2000). Radial basis functions. Acta Numerica, 1–38.Google Scholar
  9. Butail, S., Bartolini, T., Porfiri, M. (2013). Collective response of zebrafish shoals to a free-swimming robotic fish. PLoS One, 8(10), e76123. doi: 10.1371/journal.pone.0076123 PubMedCentralCrossRefPubMedGoogle Scholar
  10. Butail, S., Bollt, E.M., Porfiri, M. (2013). Analysis and classification of collective behavior using generative modeling and nonlinear manifold learning. Journal of Theoretical Biology, 336(7), 185–199. doi: 10.1016/j.jtbi.2013.07.029 CrossRefPubMedGoogle Scholar
  11. Butail, S., Chicoli, A., Paley, D. A. (2012). Putting the fish in the fish tank: Immersive VR for animal behavior experiments. In Proceedings of the IEEE international conference on robotics and automation (icra) (pp. 5018–5023). Minneapolis. doi: 10.1109/ICRA.2012.6225102
  12. Cox, I.J. (1993). A review of statistical data association for motion correspondence. International Journal of Computer Vision, 10(1), 53–66. doi: 10.1007/BF01440847 CrossRefGoogle Scholar
  13. Dadda, M., Koolhaas, W.H., Domenici, P. (2010). Behavioural asymmetry affects escape performance in a teleost fish. Biology Letters, 6(3), 414–417. doi: 10.1098/rsbl.2009.0904 PubMedCentralCrossRefPubMedGoogle Scholar
  14. Delcourt, J., Becco, C., Vandewalle, N., Poncin, P. (2009). A video multitracking system for quantification of individual behavior in a large fish shoal: Advantages and limits. Behavior Research Methods, 41(1), 228–235. doi: 10.3758/BRM.41.1.228 CrossRefPubMedGoogle Scholar
  15. Delcourt, J., Denoël, M., Ylieff, M., Poncin, P. (2013). Video multitracking of fish behaviour: A synthesis and future perspectives. Fish and Fisheries, 14, 186–204. doi: 10.1111/j.1467-2979.2012.00462.x CrossRefGoogle Scholar
  16. DeLellis, P., Polverino, G., Ustuner, G., Abaid, N., Macrì, S., Bollt, E.M., Porfiri, M. (2014). Collective behaviour across animal species. Scientific Reports, 4, 3723. doi: 10.1038/srep03723 PubMedCentralCrossRefPubMedGoogle Scholar
  17. DeLellis, P., Porfiri, M., Bollt, E. (2013). Topological analysis of group fragmentation in multi-agent systems. Physical Review E, 87(2), 022818. doi: 10.1103/PhysRevE.87.022818 CrossRefGoogle Scholar
  18. Diehl, S. (1988). Foraging efficiency of three freshwater fishes: Effects of structural complexity and light. Oikos, 53(2), 207–214. doi: 10.2307/3566064 CrossRefGoogle Scholar
  19. Eagle, N., & Pentland, A.S. (2009). Eigenbehaviors: Identifying structure in routine. Behavioral Ecology and Sociobiology, 63(7), 1057–1066. doi: 10.1007/s00265-009-0739-0 CrossRefGoogle Scholar
  20. Elgammal, A., & Lee, C. (2007). Nonlinear manifold learning for dynamic shape and dynamic appearance. Computer Vision and Image Understanding, 106(1), 31–46. doi: 10.1016/j.cviu.2005.09.010 CrossRefGoogle Scholar
  21. Fernandez-Juricic, E., & Kowalski, V. (2011). Where does a flock end from an information perspective? A comparative experiment with live and robotic birds. Behavioral Ecology, 22(6), 1304–1311. doi: 10.1093/beheco/arr132
  22. Fröhlich, H., Hoenselaar, A., Eichner, J., Rosenbrock, H., Birk, G., Zell, A. (2008). Automated classification of the behavior of rats in the forced swimming test with support vector machines. Neural Networks, 21(1), 92–101. doi: 10.1016/j.neunet.2007.09.019 CrossRefPubMedGoogle Scholar
  23. Fry, S., Rohrseitz, N., Straw, A., Dickinson, M. (2008). TrackFly: Virtual reality for a behavioral system analysis in free-flying fruit flies. Journal of Neuroscience Methods, 171(1), 110–117. doi: 10.1016/j.jneumeth.2008.02.016 CrossRefPubMedGoogle Scholar
  24. Halloy, J., Sempo, G., Caprari, G., Rivault, C., Asadpour, M., Tâche, F., Deneubourg, J.L. (2007). Social integration of robots into groups of cockroaches to control self-organized choices. Science, 318(5853), 1155–1158. doi: 10.1126/science.1144259 CrossRefPubMedGoogle Scholar
  25. He, L., Chao, Y., Suzuki, K., Wu, K. (2009). Fast connected-component labeling. Pattern Recognition, 42(1), 1977–1987. doi: 10.1016/j.patcog.2008.10.013 CrossRefGoogle Scholar
  26. Higgs, D.M., & Fuiman, L.A. (1996). Light intensity and schooling behaviour in larval gulf menhaden. Journal of Fish Biology, 48(5), 979–991.CrossRefGoogle Scholar
  27. Hoare, D. J., & Krause, J. (2003). Social organisation, shoal structure and information transfer. Fish and Fisheries, 4(3), 269–279. doi: 10.1046/j.1467-2979.2003.00130.x CrossRefGoogle Scholar
  28. Hunter, J.R. (1968). Effects of light on schooling and feeding of Jack Mackerel, Trachurus symmetricus. Journal of the Fisheries Research Board of Canada, 25(2), 393–407. doi: 10.1139/f68-031 CrossRefGoogle Scholar
  29. Kabra, M., Robie, A.A., Rivera-Alba, M., Branson, S., Branson, K. (2013). JAABA: Interactive machine learning for automatic annotation of animal behavior. Nature Methods, 10(1), 64–7. doi: 10.1038/nmeth.2281 CrossRefPubMedGoogle Scholar
  30. Kalueff, A.V., Gebhardt, M., Stewart, A.M., Cachat, J.M., Brimmer, M., Chawla, J.S., Schneider, H. (2013). Towards a comprehensive catalog of zebrafish behavior 1.0 and beyond. Zebrafish, 10(1), 70–86. doi: 10.1089/zeb.2012.0861 PubMedCentralCrossRefPubMedGoogle Scholar
  31. Kirby, M. (2001). Geometric data analysis: An empirical approach to dimensionality reduction and the study of patterns. New York: Wiley.Google Scholar
  32. Kopman, V., Laut, J., Polverino, G., Porfiri, M. (2013). Closed-loop control of zebrafish response using a bioinspired robotic-fish in a preference test. Journal of the Royal Society Interface, 20120540(78). doi: 10.1098/rsif.2012.0540
  33. Krause, J., Winfield, A.F.T., Deneubourg, J. (2011). Interactive robots in experimental biology. Trends in Ecology and Evolution, 26(7), 369–375. doi: 10.1016/j.tree.2011.03.015 CrossRefPubMedGoogle Scholar
  34. Ladu, F., Butail, S., Macrì, S., Porfiri, M (2014). Sociality modulates the effects of ethanol in zebrafish. Alcoholism, Clinical and Experimental Research. doi: 10.1111/acer.12432
  35. Lewis, J. (1995). Fast normalized cross-correlation. Vision Interface, 120–123. doi:
  36. Lowe, D.G. (2004). Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision, 60(2), 91–110. doi: 10.1023/B:VISI.0000029664.99615.94 CrossRefGoogle Scholar
  37. Miller, N., Garnier, S., Hartnett, A.T., Couzin, I. D. (2013). Both information and social cohesion determine collective decisions in animal groups. Proceedings of the National Academy of Sciences of the United States of America, 110(13), 5263–5268. doi: 10.1073/pnas.1217513110 PubMedCentralCrossRefPubMedGoogle Scholar
  38. Miller, N., & Gerlai, R. (2012). From schooling to shoaling: Patterns of collective motion in zebrafish (Danio rerio). PLoS One, 7(11), e48865. doi: 10.1371/journal.pone.0048865 PubMedCentralCrossRefPubMedGoogle Scholar
  39. Miller, N.Y., & Gerlai, R. (2008). Oscillations in shoal cohesion in zebrafish (Danio rerio). Behavioural Brain Research, 193(1), 148–51. doi: 10.1016/j.bbr.2008.05.004 PubMedCentralCrossRefPubMedGoogle Scholar
  40. Munkres, J. (1957). Algorithms for the assignment and transportation problems. Journal of Society of Industrial and Applied Mathematics, 5(1), 32–38.CrossRefGoogle Scholar
  41. Noldus, L.P.J.J., Spink, A.J., Tegelenbosch, R.A.J. (2001). EthoVision: A versatile video tracking system for automation of behavioral experiments [Proceedings Paper]. Behavior Research Methods. Instruments, & Computers, 33(3), 398–414. doi: 10.3758/BF03195394 CrossRefGoogle Scholar
  42. Parker, M.O., Ife, D., Ma, J., Pancholi, M., Smeraldi, F., Straw, C., Brennan, C.H. (2013). Development and automation of a test of impulse control in zebrafish. Frontiers in Systems Neuroscience, 7, 65. doi: 10.3389/fnsys.2013.00065 PubMedCentralCrossRefPubMedGoogle Scholar
  43. Parrish, J.K., & Hammer, W.M. (1997). Animal groups in three dimensions. Cambridge University Press.Google Scholar
  44. Penney, G.P., Weese, J., Little, J.A., Desmedt, P., Hill, D.L., Hawkes, D.J. (1998). A comparison of similarity measures for use in 2-D-3-D medical image registration. IEEE Transactions on Medical Imaging, 17(4), 586–595. doi: 10.1109/42.730403 CrossRefPubMedGoogle Scholar
  45. Pietropaolo, S., Branchi, I., Cirulli, F., Chiarotti, F., Aloe, L., Alleva, E. (2004). Long-term effects of the periadolescent environment on exploratory activity and aggressive behaviour in mice: Social versus physical enrichment. Physiology & Behavior, 81(3), 443–53. doi: 10.1016/j.physbeh.2004.02.022 CrossRefGoogle Scholar
  46. Pless, R. (2003). Image spaces and video trajectories: Using Isomap to explore video sequences. In Proceedings of the IEEE international conference on computer vision (iccv) (pp. 1433–1440). IEEE. doi: 10.1109/ICCV.2003.1238658
  47. Poppe, R. (2010). A survey on vision-based human action recognition. Image and Vision Computing, 28(6), 976–990. doi: 10.1016/j.imavis.2009.11.014 CrossRefGoogle Scholar
  48. Quera, V., Beltran, F. S., Givoni, I. E., Dolado, R. (2013). Determining shoal membership using affinity propagation. Behavioural Brain Research, 241(1), 38–49. doi: 10.1016/j.bbr.2012.11.031 CrossRefPubMedGoogle Scholar
  49. Saverino, C., & Gerlai, R. (2008). The social zebrafish: Behavioral responses to conspecific, heterospecific, and computer animated fish. Behavioural Brain Research, 191(1), 77–87. doi: 10.1016/j.bbr.2008.03.013 PubMedCentralCrossRefPubMedGoogle Scholar
  50. Shotton, J., Sharp, T., Kipman, A., Fitzgibbon, A., Finocchio, M., Blake, A., Moore, R. (2013). Real-time human pose recognition in parts from single depth images. Communications of the ACM, 56(1), 116. doi: 10.1145/2398356.2398381 CrossRefGoogle Scholar
  51. Souvenir, R., & Pless, R. (2007). Image distance functions for manifold learning. Image and Vision Computing, 25(3), 365–373. doi: 10.1016/j.imavis.2006.01.016 CrossRefGoogle Scholar
  52. Swain, D.T., Couzin, I.D., Leonard, N.E. (2011). Real-time feedback-controlled robotic rish for behavioral experiments with fish schools. Proceedings of the IEEE, 100(1), 150–163. doi: 10.1109/JPROC.2011.2165449 CrossRefGoogle Scholar
  53. Tarca, A.L., Carey, V.J., Chen, X., Romero, R., Drghici, S. (2007). Machine learning and its applications to biology . PLoS Computational Biology, 3(6), e116. doi: doi: 10.1371/journal.pcbi.0030116 doi: 10.1371/journal.pcbi.0030116 PubMedCentralCrossRefPubMedGoogle Scholar
  54. Tenenbaum, J.B, de Silva, V., Langford, J.C. (2000). A global geometric framework for nonlinear dimensionality reduction. Science, 290(5500), 2319–23. doi: doi: 10.1126/science.290.5500.2319 doi: 10.1126/science.290.5500.2319 CrossRefPubMedGoogle Scholar
  55. Tien, J.H., Levin, S.A., Rubenstein, D.I. (2004). Dynamics of fish shoals: Identifying key decision rules. Evolutionary Ecology Research, 6(4), 555–565.Google Scholar
  56. Torisawa, S., Takagi, T., Fukuda, H., Ishibashi, Y., Sawada, Y., Okada, T., Yamane, T. (2007). Schooling behaviour and retinomotor response of juvenile Pacific bluefin tuna Thunnus orientalis under different light intensities. Journal of Fish Biology, 71(2), 411–420. doi: 10.1111/j.1095-8649.2007.01498.x CrossRefGoogle Scholar
  57. Vand der Maaten, L.J.P., Postma, E.O., Van den Herik, H.J. (2009). Dimensionality reduction. A comparative review (Tech. Rep.). Tilburg University.Google Scholar
  58. Vicsek, T., Czirók, A., Ben-Jacob, E., Cohen, I., Shochet, O. (1995). Novel type of phase transition in a system of self-driven particles. Physical Review Letters, 75(6), 1226–1229. doi: doi: 10.1103/PhysRevLett.75.1226 doi: 10.1103/PhysRevLett.75.1226 CrossRefPubMedGoogle Scholar
  59. Webster, M.M., Goldsmith, J., Ward, A.J.W., Hart, P.J.B. (2007). Habitat-specific chemical cues influence association preferences and shoal cohesion in fish. Behavioral Ecology and Sociobiology, 62(2), 273–280. doi: 10.1007/s00265-007-0462-7 CrossRefGoogle Scholar
  60. Weissbrod, A., Shapiro, A., Vasserman, G., Edry, L., Dayan, M., Yitzhaky, A., Kimchi, T. (2013). Automated long-term tracking and social behavioural phenotyping of animal colonies within a semi-natural environment. Nature Communications, 4(2018), 2018. doi: 10.1038/ncomms3018 PubMedGoogle Scholar
  61. Whitney, R.R. (1969). Schooling of Fishes Relative to Available Light. Transactions of the American Fisheries Society, 98(3), 497–504. doi: 10.1577/1548-8659(1969)98[497:SOFRTA]2.0.CO;2 CrossRefGoogle Scholar
  62. Yang, H., Shao, L., Zheng, F., Wang, L., Song, Z. (2011). Recent advances and trends in visual tracking: A review. Neurocomputing, 74(18), 3823–3831. doi: 10.1016/j.neucom.2011.07.024 CrossRefGoogle Scholar
  63. Yilmaz, A., Javed, O., Shah, M. (2006). Object tracking: A survey. ACM Computing Surveys (CSUR), 38(4), 1–45. doi: 10.1145/1177352.1177355 CrossRefGoogle Scholar
  64. Zabala, F., Polidoro, P., Robie, A., Branson, K., Perona, P., Dickinson, M.H. (2012). A simple strategy for detecting moving objects during locomotion revealed by animal-robot interactions. Current Biology, 22(14), 1344–1350. doi: 10.1016/j.cub.2012.05.024 CrossRefPubMedGoogle Scholar
  65. Zhao, F., Huang, Q., Gao, W. (2006). Image matching by normalized cross-correlation. In Proceedings of the IEEE international conference on acoustics speed and signal processing proceedings (Vol. 2, pp. 729–732). IEEE. doi: 10.1109/ICASSP.2006.1660446

Copyright information

© Psychonomic Society, Inc. 2014

Authors and Affiliations

  • Sachit Butail
    • 1
  • Philip Salerno
    • 2
  • Erik M. Bollt
    • 3
  • Maurizio Porfiri
    • 2
  1. 1.Indraprastha Institute of Information Technology Delhi (IIITD)New DelhiIndia
  2. 2.Department of Mechanical and Aerospace EngineeringNew York University Polytechnic School of EngineeringBrooklynUSA
  3. 3.Department of MathematicsClarkson UniversityPotsdamUSA

Personalised recommendations