Skip to main content
Log in

Beyond tracking: using deep learning to discover novel interactions in biological swarms

  • Original Article
  • Published:
Artificial Life and Robotics Aims and scope Submit manuscript


Most deep-learning frameworks for understanding biological swarms are designed to fit perceptive models of group behavior to individual-level data (e.g., spatial coordinates of identified features of individuals) that have been separately gathered from video observations. Despite considerable advances in automated tracking, these methods are still very expensive or unreliable when tracking large numbers of animals simultaneously. Moreover, this approach assumes that the human-chosen features include sufficient features to explain important patterns in collective behavior. To address these issues, we propose training deep network models to predict system-level states directly from generic graphical features from the entire view, which can be relatively inexpensive to gather in a completely automated fashion. Because the resulting predictive models are not based on human-understood predictors, we use explanatory modules (e.g., Grad-CAM) that combine information hidden in the latent variables of the deep-network model with the video data itself to communicate to a human observer which aspects of observed individual behaviors are most informative in predicting group behavior. This represents an example of augmented intelligence in behavioral ecology—knowledge co-creation in a human–AI team. As proof of concept, we utilize a 20-day video recording of a colony of over 50 Harpegnathos saltator ants to showcase that, without any individual annotations provided, a trained model can generate an “importance map” across the video frames to highlight regions of important behaviors, such as dueling (which the AI has no a priori knowledge of), that play a role in the resolution of reproductive-hierarchy re-formation. Based on the empirical results, we also discuss the potential use and current challenges to further develop the proposed framework as a tool to discover behaviors that have not yet been considered crucial to understand complex social dynamics within biological collectives.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or Ebook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others





  1. Bozek K, Hebert L, Mikheyev AS, Stephens GJ (2018) Towards dense object tracking in a 2D honeybee hive. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp 4185–4193

  2. Choi T, Pavlic TP, Richa AW (2017) Automated synthesis of scalable algorithms for inferring non-local properties to assist in multi-robot teaming. In: 13th IEEE Conference on Automation Science and Engineering (CASE 2017), pp 1522–1527

  3. Choi T, Kang S, Pavlic TP (2020) Learning local behavioral sequences to better infer non-local properties in real multi-robot systems. In: 2020 IEEE International Conference Robotics and Automation (ICRA 2020), pp 2138–2144

  4. Choi T, Pyenson B, Liebig J, Pavlic TP (2021) Identification of abnormal states in videos of ants undergoing social phase change. In: Proceedings of the AAAI Conference on Artificial Intelligencec (AAAI 2021), vol 35. pp 15286–15292

  5. Chollet F (2020) Grad-CAM class activation visualization.

  6. Couzin ID, Krause J, James R, Ruxton GD, Franks NR (2002) Collective memory and spatial sorting in animal groups. J Theor Biol 218(1):1–11

    Article  MathSciNet  Google Scholar 

  7. Goodfellow I, Bengio Y, Courville A (2016) Deep learning. MIT Press, Cambridge

    MATH  Google Scholar 

  8. Graving JM, Chae D, Naik H, Li L, Koger B, Costelloe BR, Couzin ID (2019) DeepPoseKit, a software toolkit for fast and robust animal pose estimation using deep learning. eLife 8:e47994

    Article  Google Scholar 

  9. He T, Guo J, Chen N, Xu X, Wang Z, Fu K, Liu L, Yi Z (2019) MediMLP: using Grad-CAM to extract crucial variables for lung cancer postoperative complication prediction. IEEE J Biomed Health Inform 24(6):1762–1771

  10. Lertvittayakumjorn P, Toni F (2019) Human-grounded evaluations of explanation methods for text classification. arXiv preprint arXiv:1908.11355

  11. Li CL, Sohn K, Yoon J, Pfister T (2021) CutPaste: self-supervised learning for anomaly detection and localization. In: EEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2021), pp 9664–9674

  12. Liebig J, Peeters C, Hölldobler B (1999) Worker policing limits the number of reproductives in a ponerine ant. Proc R Soc Lond Ser B Biol Sci 266(1431):1865–1870

    Article  Google Scholar 

  13. Mehran R, Oyama A, Shah M (2009) Abnormal crowd behavior detection using social force model. In: Conference on Computer Vision and Pattern Recognition (CVPR 2009), pp 935–942

  14. Nath T, Mathis A, Chen AC, Patel A, Bethge M, Mathis MW (2019) Using deeplabcut for 3D markerless pose estimation across species and behaviors. Nat Protoc 14(7):2152–2176

    Article  Google Scholar 

  15. Panwar H, Gupta P, Siddiqui MK, Morales-Menendez R, Bhardwaj P, Singh V (2020) A deep learning and Grad-CAM based color visualization approach for fast detection of COVID-19 cases using chest X-ray and CT-scan images. Chaos Solitons Fractals 140

  16. Peeters C, Crewe R (1985) Worker reproduction in the ponerine ant Ophthalmopone berthoudi: an alternative form of eusocial organization. Behav Ecol Sociobiol 18(1):29–37

  17. Pratt SC, Mallon EB, Sumpter DJ, Franks NR (2002) Quorum sensing, recruitment, and collective decision-making during colony emigration by the ant Leptothorax albipennis. Behav Ecol Sociobiol 52(2):117–127

  18. Rajagopal D, Balachandran V, Hovy E, Tsvetkov Y (2021) SelfExplain: a self-explaining architecture for neural text classifiers. arXiv preprint arXiv:2103.12279

  19. Redmon J, Farhadi A (2018) YOLOv3: an incremental improvement. arXiv preprint arXiv:1804.02767

  20. Reid CR, Lutz MJ, Powell S, Kao AB, Couzin ID, Garnier S (2015) Army ants dynamically adjust living bridges in response to a cost-benefit trade-off. Proc Natl Acad Sci 112(49):15113–15118

  21. Romero-Ferrero F, Bergomi MG, Hinz RC, Heras FJ, de Polavieja GG (2019) tracking all individuals in small or large collectives of unmarked animals. Nat Methods 16(2):179–182

  22. Sasaki T, Penick CA, Shaffer Z, Haight KL, Pratt SC, Liebig J (2016) A simple behavioral model predicts the emergence of complex animal hierarchies. Am Nat 187(6):765–775

  23. Selvaraju RR, Cogswell M, Das A, Vedantam R, Parikh D, Batra D (2017) Grad-CAM: visual explanations from deep networks via gradient-based localization. In: IEEE International Conference on Computer Vision (ICCV 2017), pp 618–626

  24. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556

  25. Wang L, Xiong Y, Wang Z, Qiao Y, Lin D, Tang X, Van Gool L (2016) Temporal segment networks: towards good practices for deep action recognition. In: Leibe B, Matas J, Sebe N, Welling M (eds) Computer Vision – ECCV 2016. ECCV 2016. Lecture Notes in Computer Science, vol 9912. Springer, Cham, pp 20–36

Download references

Author information

Authors and Affiliations


Corresponding author

Correspondence to Theodore P. Pavlic.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This work was presented in part at the joint symposium with the 15th International Symposium on Distributed Autonomous Robotic Systems 2021 and the 4th International Symposium on Swarm Behavior and Bio-Inspired Robotics 2021 (Online, June 1–4, 2021).

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Choi, T., Pyenson, B., Liebig, J. et al. Beyond tracking: using deep learning to discover novel interactions in biological swarms. Artif Life Robotics 27, 393–400 (2022).

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: