Skip to main content
SpringerLink
Log in

Noninvasive Tracking of Every Individual in Unmarked Mouse Groups Using Multi-Camera Fusion and Deep Learning

  • Original Article
  • Published:
Neuroscience Bulletin Aims and scope Submit manuscript

Abstract

Accurate and efficient methods for identifying and tracking each animal in a group are needed to study complex behaviors and social interactions. Traditional tracking methods (e.g., marking each animal with dye or surgically implanting microchips) can be invasive and may have an impact on the social behavior being measured. To overcome these shortcomings, video-based methods for tracking unmarked animals, such as fruit flies and zebrafish, have been developed. However, tracking individual mice in a group remains a challenging problem because of their flexible body and complicated interaction patterns. In this study, we report the development of a multi-object tracker for mice that uses the Faster region-based convolutional neural network (R-CNN) deep learning algorithm with geometric transformations in combination with multi-camera/multi-image fusion technology. The system successfully tracked every individual in groups of unmarked mice and was applied to investigate chasing behavior. The proposed system constitutes a step forward in the noninvasive tracking of individual mice engaged in social behavior.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

References

  1. Dunbar RI, Shultz S. Evolution in the social brain. Science 2007, 317: 1344–1347.

    Article  CAS  PubMed  Google Scholar 

  2. Sallet J, Mars RB, Noonan M, Andersson JL, O’Reilly J, Jbabdi S. Social network size affects neural circuits in macaques. Science 2011, 334: 697–700.

    Article  CAS  PubMed  Google Scholar 

  3. Ben-Ami Bartal I, Decety J, Mason P. Empathy and pro-social behavior in rats. Science 2011, 334: 1427–1430.

    Article  PubMed  Google Scholar 

  4. Keller M, Vandenberg LN, Charlier TD. The parental brain and behavior: A target for endocrine disruption. Front Neuroendocrinol 2019, 54: 100765.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  5. Bridges RS. Neuroendocrine regulation of maternal behavior. Front Neuroendocrinol 2015, 36: 178–196.

    Article  CAS  PubMed  Google Scholar 

  6. Jager A, Maas DA, Fricke K, de Vries RB, Poelmans G, Glennon JC. Aggressive behavior in transgenic animal models: A systematic review. Neurosci Biobehav Rev 2018, 91: 198–217.

    Article  PubMed  Google Scholar 

  7. Manchia M, Carpiniello B, Valtorta F, Comai S. Serotonin dysfunction, aggressive behavior, and mental illness: Exploring the link using a dimensional approach. ACS Chem Neurosci 2017, 8: 961–972.

    Article  CAS  PubMed  Google Scholar 

  8. Safran RJ, Levin II, Fosdick BK, McDermott MT, Semenov GA, Hund AK, et al. Using networks to connect individual-level reproductive behavior to population patterns. Trends Ecol Evol 2019, 34: 497–501.

    Article  PubMed  Google Scholar 

  9. Port M, Schülke O, Ostner J. Reproductive tolerance in male primates: Old paradigms and new evidence. Evol Anthropol 2018, 27: 107–120.

    Article  PubMed  Google Scholar 

  10. Bass C, Halligan PW. Illness related deception: Social or psychiatric problem? J R Soc Med 2007, 100: 81–84.

    Article  PubMed  PubMed Central  Google Scholar 

  11. Greene RW, Biederman J, Zerwas S, Monuteaux MC, Goring JC, Faraone SV. Psychiatric comorbidity, family dysfunction, and social impairment in referred youth with oppositional defiant disorder. Am J Psychiatry 2002, 159: 1214–1224.

    Article  PubMed  Google Scholar 

  12. Shultz S, Klin A, Jones W. Neonatal transitions in social behavior and their implications for autism. Trends Cogn Sci 2018, 22: 452–469.

    Article  PubMed  PubMed Central  Google Scholar 

  13. Yu B, Yuan B, Dai JK, Cheng TL, Xia SN, He LJ, et al. Reversal of social recognition deficit in adult mice with MECP2 duplication via normalization of MeCP2 in the medial prefrontal cortex. Neurosci Bull 2020, 36: 570–584.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  14. Green MF, Horan WP, Lee J, McCleery A, Reddy LF, Wynn JK. Social disconnection in schizophrenia and the general community. Schizophr Bull 2018, 44: 242–249.

    Article  PubMed  Google Scholar 

  15. Fernández-Theoduloz G, Paz V, Nicolaisen-Sobesky E, Pérez A, Buunk AP, Cabana Á, et al. Social avoidance in depression: A study using a social decision-making task. J Abnorm Psychol 2019, 128: 234–244.

    Article  PubMed  Google Scholar 

  16. Lemyre A, Gauthier-Légaré A, Bélanger RE. Shyness, social anxiety, social anxiety disorder, and substance use among normative adolescent populations: A systematic review. Am J Drug Alcohol Abuse 2019, 45: 230–247.

    Article  PubMed  Google Scholar 

  17. Romano M, Moscovitch DA, Ma RF, Huppert JD. Social problem solving in social anxiety disorder. J Anxiety Disord 2019, 68: 102152.

    Article  PubMed  Google Scholar 

  18. Rein B, Ma KJ, Yan Z. A standardized social preference protocol for measuring social deficits in mouse models of autism. Nat Protoc 2020, 15: 3464–3477.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  19. Ariyasiri K, Choi TI, Kim OH, Hong TI, Gerlai R, Kim CH. Pharmacological (ethanol) and mutation (Sam2 KO) induced impairment of novelty preference in zebrafish quantified using a new three-chamber social choice task. Prog Neuropsychopharmacol Biol Psychiatry 2019, 88: 53–65.

    Article  CAS  PubMed  Google Scholar 

  20. Zheng J, Tian Y, Xu H, Gu L, Xu H. A standardized protocol for the induction of specific social fear in mice. Neurosci Bull 2021, 37: 1708–1712.

    Article  PubMed  PubMed Central  Google Scholar 

  21. Fan Z, Zhu H, Zhou T, Wang S, Wu Y, Hu H. Using the tube test to measure social hierarchy in mice. Nat Protoc 2019, 14: 819–831.

    Article  CAS  PubMed  Google Scholar 

  22. Zhou T, Zhu H, Fan Z, Wang F, Chen Y, Liang H, et al. History of winning remodels thalamo-PFC circuit to reinforce social dominance. Science 2017, 357: 162–168.

    Article  CAS  PubMed  Google Scholar 

  23. Gomez-Marin A, Paton JJ, Kampff AR, Costa RM, Mainen ZF. Big behavioral data: Psychology, ethology and the foundations of neuroscience. Nat Neurosci 2014, 17: 1455–1462.

    Article  CAS  PubMed  Google Scholar 

  24. Crabbe JC, Morris RG. Festina lente: Late-night thoughts on high-throughput screening of mouse behavior. Nat Neurosci 2004, 7: 1175–1179.

    Article  CAS  PubMed  Google Scholar 

  25. Romero-Ferrero F, Bergomi MG, Hinz RC, Heras FJH, de Polavieja GG. Idtracker ai Tracking all individuals in small or large collectives of unmarked animals. Nat Methods 2019, 16: 179–182.

    Article  CAS  PubMed  Google Scholar 

  26. Bai YX, Zhang SH, Fan Z, Liu XY, Zhao X, Feng XZ, et al. Automatic multiple zebrafish tracking based on improved HOG features. Sci Rep 2018, 8: 10884.

    Article  PubMed  PubMed Central  Google Scholar 

  27. Orger MB, de Polavieja GG. Zebrafish behavior: Opportunities and challenges. Annu Rev Neurosci 2017, 40: 125–147.

    Article  CAS  PubMed  Google Scholar 

  28. Egnor SE, Branson K. Computational analysis of behavior. Annu Rev Neurosci 2016, 39: 217–236.

    Article  CAS  PubMed  Google Scholar 

  29. Hong W, Kennedy A, Burgos-Artizzu XP, Zelikowsky M, Navonne SG, Perona P, et al. Automated measurement of mouse social behaviors using depth sensing, video tracking, and machine learning. Proc Natl Acad Sci U S A 2015, 112: E5351–E5360.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  30. Pérez-Escudero A, Vicente-Page J, Hinz RC, Arganda S, de Polavieja GG. idTracker: Tracking individuals in a group by automatic identification of unmarked animals. Nat Methods 2014, 11: 743–748.

    Article  PubMed  Google Scholar 

  31. Dolado R, Gimeno E, Beltran FS, Quera V, Pertusa JF. A method for resolving occlusions when multitracking individuals in a shoal. Behav Res Methods 2015, 47: 1032–1043.

    Article  PubMed  Google Scholar 

  32. Pereira TD, Aldarondo DE, Willmore L, Kislin M, Wang SS, Murthy M, et al. Fast animal pose estimation using deep neural networks. Nat Methods 2019, 16: 117–125.

    Article  CAS  PubMed  Google Scholar 

  33. Forkosh O, Karamihalev S, Roeh S, Alon U, Anpilov S, Touma C, et al. Identity domains capture individual differences from across the behavioral repertoire. Nat Neurosci 2019, 22: 2023–2028.

    Article  CAS  PubMed  Google Scholar 

  34. Noldus LP, Spink AJ, Tegelenbosch RA. EthoVision: A versatile video tracking system for automation of behavioral experiments. Behav Res Methods Instrum Comput 2001, 33: 398–414.

    Article  CAS  PubMed  Google Scholar 

  35. Spink A, Tegelenbosch R, Buma M, Noldus L. The EthoVision video tracking system—a tool for behavioral phenotyping of transgenic mice. Physiol Behav 2001, 73: 731–744.

    Article  CAS  PubMed  Google Scholar 

  36. Ohayon S, Avni O, Taylor AL, Perona P, Egnor SR. Automated multi-day tracking of marked mice for the analysis of social behaviour. J Neurosci Methods 2013, 219: 10–19.

    Article  PubMed  PubMed Central  Google Scholar 

  37. Shemesh Y, Sztainberg Y, Forkosh O, Shlapobersky T, Chen A, Schneidman E. High-order social interactions in groups of mice. eLife 2013, 2: e00759.

    Article  PubMed  PubMed Central  Google Scholar 

  38. Howerton CL, Garner JP, Mench JA. A system utilizing radio frequency identification (RFID) technology to monitor individual rodent behavior in complex social settings. J Neurosci Methods 2012, 209: 74–78.

    Article  PubMed  Google Scholar 

  39. Freund J, Brandmaier AM, Lewejohann L, Kirste I, Kritzler M, Krüger A, et al. Emergence of individuality in genetically identical mice. Science 2013, 340: 756–759.

    Article  CAS  PubMed  Google Scholar 

  40. Weissbrod A, Shapiro A, Vasserman G, Edry L, Dayan M, Yitzhaky A, et al. Automated long-term tracking and social behavioural phenotyping of animal colonies within a semi-natural environment. Nat Commun 2018, 2013: 4.

    Google Scholar 

  41. Pavković Ž, Potrebić M, Kanazir S, Pešić V. Motivation, risk-taking and sensation seeking behavior in propofol anesthesia exposed peripubertal rats. Prog Neuro Psychopharmacol Biol Psychiatry 2020, 96: 109733.

    Article  Google Scholar 

  42. Jin X, Ji L, Chen Q, Sheng R, Ji F, Yang J. Anesthesia plus surgery in neonatal period impairs preference for social novelty in mice at the juvenile age. Biochem Biophys Res Commun 2020, 530: 603–608.

    Article  CAS  PubMed  Google Scholar 

  43. Stratmann G, Lee J, Sall JW, Lee BH, Alvi RS, Shih J, et al. Effect of general anesthesia in infancy on long-term recognition memory in humans and rats. Neuropsychopharmacology 2014, 39: 2275–2287.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  44. Branson K, Robie AA, Bender J, Perona P, Dickinson MH. High-throughput ethomics in large groups of Drosophila. Nat Methods 2009, 6: 451–457.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  45. de Chaumont F, Coura RD, Serreau P, Cressant A, Chabout J, Granon S, et al. Computerized video analysis of social interactions in mice. Nat Methods 2012, 9: 410–417.

    Article  PubMed  Google Scholar 

  46. Mathis A, Mamidanna P, Cury KM, Abe T, Murthy VN, Mathis MW, et al. DeepLabCut: Markerless pose estimation of user-defined body parts with deep learning. Nat Neurosci 2018, 21: 1281–1289.

    Article  CAS  PubMed  Google Scholar 

  47. Bala PC, Eisenreich BR, Yoo SBM, Hayden BY, Park HS, Zimmermann J. Automated markerless pose estimation in freely moving macaques with OpenMonkeyStudio. Nat Commun 2020, 11: 4560.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  48. Pereira TD, Tabris N, Matsliah A, Turner DM, Li J, Ravindranath S, et al. SLEAP: A deep learning system for multi-animal pose tracking. Nat Methods 2022, 19: 486–495.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  49. Marshall JD, Aldarondo DE, Dunn TW, Wang WL, Berman GJ, Ölveczky BP. Continuous whole-body 3D kinematic recordings across the rodent behavioral repertoire. Neuron 2021, 109: 420-437.e8.

    Article  CAS  PubMed  Google Scholar 

  50. Huang K, Han Y, Chen K, Pan H, Zhao G, Yi W, et al. A hierarchical 3D-motion learning framework for animal spontaneous behavior mapping. Nat Commun 2021, 12: 2784.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  51. Han Y, Huang K, Chen K, Pan H, Ju F, Long Y, et al. MouseVenue3D: A markerless three-dimension behavioral tracking system for matching two-photon brain imaging in free-moving mice. Neurosci Bull 2022, 38: 303–317.

    Article  CAS  PubMed  Google Scholar 

  52. Schofield D, Nagrani A, Zisserman A, Hayashi M, Matsuzawa T, Biro D, et al. Chimpanzee face recognition from videos in the wild using deep learning. Sci Adv 2019, 5: 0736.

    Article  Google Scholar 

  53. Liu WQ, Camps O, Sznaier M. Multi-camera multi-object tracking. 2017: arXiv: 1709.07065. doi: https://doi.org/10.48550/arXiv.1709.07065.

  54. Wang X. Intelligent multi-camera video surveillance: A review. Pattern Recognit Lett 2013, 34: 3–19.

    Article  Google Scholar 

  55. Thanos PK, Restif C, O’Rourke JR, Lam CY, Metaxas D. Mouse Social Interaction Test (MoST): A quantitative computer automated analysis of behavior. J Neural Transm 2017, 124: 3–11.

    Article  PubMed  Google Scholar 

  56. Peters SM, Pinter IJ, Pothuizen HH, de Heer RC, van der Harst JE, Spruijt BM. Novel approach to automatically classify rat social behavior using a video tracking system. J Neurosci Methods 2016, 268: 163–170.

    Article  PubMed  Google Scholar 

  57. Ren S, He K, Girshick R, Sun J. Faster R-CNN: Towards real-time object detection with region proposal networks. IEEE Trans Pattern Anal Mach Intell 2017, 39: 1137–1149.

    Article  PubMed  Google Scholar 

  58. Blackman S, Populi R (1999) Design and Analysis of Modern Tracking Systems. Artech House Publishers, Norwood.

    Google Scholar 

  59. Zhu H, Liu D, Zhang S, Zhu Y, Teng L, Teng S. Solving the Many to Many assignment problem by improving the Kuhn-Munkres algorithm with backtracking. Theor Comput Sci 2016, 618: 30–41.

    Article  Google Scholar 

  60. He K, Zhang X, Ren S, Sun J. Deep residual learning for image recognition. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2016, 2016: 770–778.

    Google Scholar 

  61. Szegedy C, Ioffe S, Vanhoucke V, Alemi A. Inception-v4, inception-ResNet and the impact of residual connections on learning. Proc AAAI Conf Artif Intell 2017, https://doi.org/10.48550/arXiv.1602.07261.

    Article  Google Scholar 

  62. Su F, Sun Y, Hu Y, Yuan P, Wang X, Wang Q, et al. Development and validation of a deep learning system for ascites cytopathology interpretation. Gastric Cancer 2020, 23: 1041–1050.

    Article  PubMed  Google Scholar 

  63. Lauer J, Zhou M, Ye S, Menegas W, Schneider S, Nath T, et al. Multi-animal pose estimation, identification and tracking with DeepLabCut. Nat Methods 2022, 19: 496–504.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  64. Agezo S, Berman GJ. Tracking together: Estimating social poses. Nat Methods 2022, 19: 410–411.

    Article  CAS  PubMed  Google Scholar 

  65. Jia Y, Li S, Guo X, Lei B, Hu J, Xu XH, et al. Selfee, self-supervised features extraction of animal behaviors. eLife 2022, 11: e76218.

Download references

Acknowledgments

This work was supported by grants from the National Key R&D Program of China (2017YFA0105201); the National Natural Science Foundation of China (81925011, 92149304,31900698, 32170954, and 32100763; the Key-Area Research and Development Program of Guangdong Province (2019B030335001); The Youth Beijing Scholars Program (015), Support Project of High-level Teachers in Beijing Municipal Universities (CIT&TCD20190334); Beijing Advanced Innovation Center for Big Data-based Precision Medicine, Capital Medical University, Beijing, China (PXM2021_014226_000026).

Author information

Author notes

  1. Feng Su, Yangzhen Wang, Mengping Wei, and Chong Wang contributed equally to this work.

Authors and Affiliations

  1. Department of Neurobiology, School of Basic Medical Sciences, Beijing Key Laboratory of Neural Regeneration and Repair, Capital Medical University, Beijing, 100069, China

    Feng Su, Mengping Wei, Lei Yang & Chen Zhang

  2. Chinese Institute for Brain Research, Beijing, 102206, China

    Feng Su & Chen Zhang

  3. State Key Laboratory of Translational Medicine and Innovative Drug Development, Nanjing, 210000, China

    Feng Su & Chen Zhang

  4. Peking-Tsinghua Center for Life Sciences, Academy for Advanced Interdisciplinary Studies, Peking University, Beijing, 100871, China

    Feng Su & Dong-Gen Luo

  5. School of Life Sciences, Tsinghua University, Beijing, 100084, China

    Yangzhen Wang

  6. School of Biological Science and Medical Engineering, Beihang University, Beijing, 100191, China

    Chong Wang

  7. The Key Laboratory of Developmental Genes and Human Disease, Institute of Life Sciences, Southeast University, Nanjing, 210096, Jiangsu, China

    Shaoli Wang

  8. Institute for Artificial Intelligence, the State Key Laboratory of Intelligence Technology and Systems, Beijing National Research Center for Information Science and Technology, Department of Computer Science and Technology, Tsinghua University, Beijing, 100084, China

    Jianmin Li

  9. School of Mechanical Engineering and Automation, Beihang University, Beijing, 100191, China

    Peijiang Yuan

Authors
  1. Feng Su
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Yangzhen Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mengping Wei
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Chong Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Shaoli Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Lei Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Jianmin Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Peijiang Yuan
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Dong-Gen Luo
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Chen Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Peijiang Yuan, Dong-Gen Luo or Chen Zhang.

Ethics declarations

Conflict of interest

The authors declare that there are no conflicts of interest.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file1 (PDF 1125 kb)

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Su, F., Wang, Y., Wei, M. et al. Noninvasive Tracking of Every Individual in Unmarked Mouse Groups Using Multi-Camera Fusion and Deep Learning. Neurosci. Bull. 39, 893–910 (2023). https://doi.org/10.1007/s12264-022-00988-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12264-022-00988-6

Keywords

Search

Navigation