Skip to main content
Log in

MouseVenue3D: A Markerless Three-Dimension Behavioral Tracking System for Matching Two-Photon Brain Imaging in Free-Moving Mice

  • Method
  • Published:
Neuroscience Bulletin Aims and scope Submit manuscript

Abstract

Understanding the connection between brain and behavior in animals requires precise monitoring of their behaviors in three-dimensional (3-D) space. However, there is no available three-dimensional behavior capture system that focuses on rodents. Here, we present MouseVenue3D, an automated and low-cost system for the efficient capture of 3-D skeleton trajectories in markerless rodents. We improved the most time-consuming step in 3-D behavior capturing by developing an automatic calibration module. Then, we validated this process in behavior recognition tasks, and showed that 3-D behavioral data achieved higher accuracy than 2-D data. Subsequently, MouseVenue3D was combined with fast high-resolution miniature two-photon microscopy for synchronous neural recording and behavioral tracking in the freely-moving mouse. Finally, we successfully decoded spontaneous neuronal activity from the 3-D behavior of mice. Our findings reveal that subtle, spontaneous behavior modules are strongly correlated with spontaneous neuronal activity patterns.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

References

  1. Datta SR, Anderson DJ, Branson K, Perona P, Leifer A. Computational neuroethology: A call to action. Neuron 2019, 104: 11–24.

    CAS  PubMed  PubMed Central  Google Scholar 

  2. Mathis MW, Mathis A. Deep learning tools for the measurement of animal behavior in neuroscience. Curr Opin Neurobiol 2020, 60: 1–11.

    CAS  PubMed  Google Scholar 

  3. Bala PC, Eisenreich BR, Yoo SBM, Hayden BY, Park HS, Zimmermann J. Automated markerless pose estimation in freely moving macaques with OpenMonkeyStudio. Nat Commun 2020, 11: 4560.

    CAS  PubMed  PubMed Central  Google Scholar 

  4. Xia F, Kheirbek MA. Circuit-based biomarkers for mood and anxiety disorders. Trends Neurosci 2020, 43: 902–915.

    CAS  PubMed  PubMed Central  Google Scholar 

  5. Wiltschko AB, Tsukahara T, Zeine A, Anyoha R, Gillis WF, Markowitz JE. Revealing the structure of pharmacobehavioral space through motion sequencing. Nat Neurosci 2020, 23: 1433–1443.

    CAS  PubMed  PubMed Central  Google Scholar 

  6. Kaplan HS, Zimmer M. Brain-wide representations of ongoing behavior: A universal principle? Curr Opin Neurobiol 2020, 64: 60–69.

    CAS  PubMed  Google Scholar 

  7. Wei P, Liu N, Zhang Z, Liu X, Tang Y, He X, et al. Processing of visually evoked innate fear by a non-canonical thalamic pathway. Nat Commun 2015, 6: 6756.

    CAS  PubMed  Google Scholar 

  8. Zhou Z, Liu X, Chen S, Zhang Z, Liu Y, Montardy Q, et al. A VTA GABAergic neural circuit mediates visually evoked innate defensive responses. Neuron 2019, 103: 473-488.e6.

    CAS  PubMed  Google Scholar 

  9. Xiao Q, Zhou XY, Wei PF, Xie L, Han YN, Wang J, et al. A new GABAergic somatostatin projection from the BNST onto accumbal parvalbumin neurons controls anxiety. Mol Psychiatry 2020, https://doi.org/10.1038/s41380-020-0816-3.

    Article  PubMed  PubMed Central  Google Scholar 

  10. Yilmaz M, Meister M. Rapid innate defensive responses of mice to looming visual stimuli. Curr Biol 2013, 23: 2011–2015.

    CAS  PubMed  Google Scholar 

  11. Storchi R, Milosavljevic N, Allen AE, Zippo AG, Agnihotri A, Cootes TF, et al. A high-dimensional quantification of mouse defensive behaviors reveals enhanced diversity and stimulus specificity. Curr Biol 2020, 30: 4619-4630.e5.

    CAS  PubMed  PubMed Central  Google Scholar 

  12. De Franceschi G, Vivattanasarn T, Saleem AB, Solomon SG. Vision guides selection of freeze or flight defense strategies in mice. Curr Biol 2016, 26: 2150–2154.

    PubMed  Google Scholar 

  13. Vale R, Evans DA, Branco T. Rapid spatial learning controls instinctive defensive behavior in mice. Curr Biol 2017, 27: 1342–1349.

    CAS  PubMed  PubMed Central  Google Scholar 

  14. Huang Y, Li L, Dong K, Tang HS, Yang QN, Jia XL, et al. Topological shape changes weaken the innate defensive response to visual threat in mice. Neurosci Bull 2020, 36: 427–431.

    PubMed  Google Scholar 

  15. Sheshadri S, Dann B, Hueser T, Scherberger H. 3D reconstruction toolbox for behavior tracked with multiple cameras. J Open Source Softw 1849, 2020: 5.

    Google Scholar 

  16. Günel S, Rhodin H, Morales D, Campagnolo J, Ramdya P, Fua P. DeepFly3D, a deep learning-based approach for 3D limb and appendage tracking in tethered, adult Drosophila. Elife 2019, 8: e4851.

    Google Scholar 

  17. Nath T, Mathis A, Chen AC, Patel A, Bethge M, Mathis MW. Using DeepLabCut for 3D markerless pose estimation across species and behaviors. Nat Protoc 2019, 14: 2152–2176.

    CAS  PubMed  Google Scholar 

  18. Finkelstein A, Derdikman D, Rubin A, Foerster JN, Las L, Ulanovsky N. Three-dimensional head-direction coding in the bat brain. Nature 2015, 517: 159–164.

    CAS  PubMed  Google Scholar 

  19. Matsumoto J, Urakawa S, Takamura Y, Malcher-Lopes R, Hori E, Tomaz C, et al. A 3D-video-based computerized analysis of social and sexual interactions in rats. PLoS One 2013, 8: e78460. https://doi.org/10.1371/journal.pone.0078460.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  20. Mathis A, Schneider S, Lauer J, Mathis MW. A primer on motion capture with deep learning: Principles, pitfalls, and perspectives. Neuron 2020, 108: 44–65.

    CAS  PubMed  Google Scholar 

  21. Mathis A, Mamidanna P, Cury KM, Abe T, Murthy VN, Mathis MW, et al. DeepLabCut: markerless pose estimation of user-defined body parts with deep learning. Nat Neurosci 2018, 21: 1281–1289.

    CAS  PubMed  Google Scholar 

  22. Mei Y, Monteiro P, Zhou Y, Kim JA, Gao X, Fu ZY, et al. Adult restoration of Shank3 expression rescues selective autistic-like phenotypes. Nature 2016, 530: 481–484.

    CAS  PubMed  PubMed Central  Google Scholar 

  23. Hewitt BM, Yap MH, Hodson-Tole EF, Kennerley AJ, Sharp PS, Grant RA. A novel automated rodent tracker (ART), demonstrated in a mouse model of amyotrophic lateral sclerosis. J Neurosci Methods 2018, 300: 147–156.

    PubMed  Google Scholar 

  24. Karashchuk P, Rupp KL, Dickinson ES, Walling-Bell S, Sanders E, Azim E, Brunton BW, Tuthill JC. Anipose: a toolkit for robust markerless 3D pose estimation. BioRxiv 2021, https://doi.org/10.1101/2020.05.26.117325.

    Article  Google Scholar 

  25. von Ziegler L, Sturman O, Bohacek J. Big behavior: Challenges and opportunities in a new era of deep behavior profiling. Neuropsychopharmacology 2021, 46: 33–44.

    Google Scholar 

  26. Sturman O, von Ziegler L, Schläppi C, Akyol F, Privitera M, Slominski D, et al. Deep learning-based behavioral analysis reaches human accuracy and is capable of outperforming commercial solutions. Neuropsychopharmacology 2020, 45: 1942–1952.

    PubMed  PubMed Central  Google Scholar 

  27. Hollon NG, Jin X. Neural implementation of behavioral hierarchy. Neuron 2020, 105: 402–404.

    CAS  PubMed  Google Scholar 

  28. Markowitz JE, Datta SR. The striatum specifies the statistics of behavior. Neuropsychopharmacology 2020, 45: 222–223.

    PubMed  Google Scholar 

  29. Markowitz JE, Gillis WF, Beron CC, Neufeld SQ, Robertson K, Bhagat ND, et al. The striatum organizes 3D behavior via moment-to-moment action selection. Cell 2018, 174: 44-58.e17.

    CAS  PubMed  PubMed Central  Google Scholar 

  30. Marques JC, Lackner S, Félix R, Orger MB. Structure of the zebrafish locomotor repertoire revealed with unsupervised behavioral clustering. Curr Biol 2018, 28: 181-195.e5.

    CAS  PubMed  Google Scholar 

  31. Kaplan HS, Salazar Thula O, Khoss N, Zimmer M. Nested neuronal dynamics orchestrate a behavioral hierarchy across timescales. Neuron 2020, 105: 562-576.e9.

    CAS  PubMed  PubMed Central  Google Scholar 

  32. Johnson RE, Linderman S, Panier T, Wee CL, Song E, Herrera KJ, et al. Probabilistic models of larval zebrafish behavior reveal structure on many scales. Curr Biol 2020, 30: 70-82.e4.

    CAS  PubMed  Google Scholar 

  33. Vogelstein JT, Park Y, Ohyama T, Kerr RA, Truman JW, Priebe CE, et al. Discovery of brainwide neural-behavioral maps via multiscale unsupervised structure learning. Science 2014, 344: 386–392.

    CAS  PubMed  Google Scholar 

  34. Stringer C, Pachitariu M, Steinmetz N, Reddy CB, Carandini M, Harris KD. Spontaneous behaviors drive multidimensional, brainwide activity. Science 2019, 364: 255.

    PubMed  PubMed Central  Google Scholar 

  35. Zong WJ, Wu RL, Li ML, Hu YH, Li YJ, Li JH, et al. Fast high-resolution miniature two-photon microscopy for brain imaging in freely behaving mice. Nat Methods 2017, 14: 713–719.

    CAS  PubMed  Google Scholar 

  36. Meyer AF, Poort J, O’Keefe J, Sahani M, Linden JF. A head-mounted camera system integrates detailed behavioral monitoring with multichannel electrophysiology in freely moving mice. Neuron 2018, 100: 46-60.e7.

    CAS  PubMed  PubMed Central  Google Scholar 

  37. Sych Y, Chernysheva M, Sumanovski LT, Helmchen F. High-density multi-fiber photometry for studying large-scale brain circuit dynamics. Nat Methods 2019, 16: 553–560.

    CAS  PubMed  Google Scholar 

  38. Zong WJ, Wu RL, Chen SY, Wu JJ, Wang HB, Zhao Z, et al. Publisher Correction: Miniature two-photon microscopy for enlarged field-of-view, multi-plane and long-term brain imaging. Nat Methods 2021, 18: 220.

    CAS  PubMed  Google Scholar 

  39. Chen SY, Wang ZC, Zhang D, Wang AM, Chen LY, Cheng HP, et al. Miniature fluorescence microscopy for imaging brain activity in freely-behaving animals. Neurosci Bull 2020, 36: 1182–1190.

    PubMed  PubMed Central  Google Scholar 

  40. Huang K, Han YN, Chen K, Pan HL, Zhao GY, Yi WL, et al. A hierarchical 3D-motion learning framework for animal spontaneous behavior mapping. Nat Commun 2021, 12: 2784.

  41. Zhang Z. Flexible camera calibration by viewing a plane from unknown orientations. Proceedings of the Seventh IEEE International Conference on Computer Vision. 1999, 1: 666–673.

    Google Scholar 

  42. Xu GX, Meng YT, Qiu XY, Yu ZH, Wu X. Sentiment analysis of comment texts based on BiLSTM. IEEE Access 2019, 7: 51522–51532.

    Google Scholar 

  43. McInnes L, Healy J, Saul N, Großberger L. UMAP: uniform manifold approximation and projection. J Open Source Softw 2018, 3: 861.

    Google Scholar 

  44. Rani Y, Rohil H. A study of hierarchical clustering algorithm. Int J Info Comp Tech 2013, 3(11): 1225–1232.

  45. Coates A, Ng AY. Learning feature representations with K-means. In: Neural networks: Tricks of the trade, Springer, Berlin, 2012, pp 561–580.

    Google Scholar 

  46. Zhou HB, Gao JT. Automatic method for determining cluster number based on silhouette coefficient. Adv Mater Res 2014, 951: 227–230.

    Google Scholar 

  47. Luxem K, Fuhrmann F, Kürsch J, Remy S, Bauer P. Identifying behavioral structure from deep variational embeddings of animal motion. bioRxiv 2020, DOI:https://doi.org/10.1101/2020.05.14.095430.

  48. Jhuang H, Garrote E, Mutch J, Yu XL, Khilnani V, Poggio T, et al. Automated home-cage behavioural phenotyping of mice. Nat Commun 2010, 1: 68.

    PubMed  Google Scholar 

  49. Hagedorn J, Hailpern J, Karahalios KG. VCode and VData: illustrating a new framework for supporting the video annotation workflow. Proceedings of the working conference on Advanced visual interface. 2008: 317–321.

  50. Kipp M, von Hollen LF, Hrstka MC, Zamponi F. Single-Person and Multi-Party 3D Visualizations for Nonverbal Communication Analysis. LREC 2014: 3393–3397.

  51. LeCun Y, Bengio Y, Hinton G. Deep learning. Nature 2015, 521: 436–444.

    CAS  Google Scholar 

  52. Wang Z, Mirbozorgi SA, Ghovanloo M. An automated behavior analysis system for freely moving rodents using depth image. Med Biol Eng Comput 2018, 56: 1807–1821.

    PubMed  Google Scholar 

  53. Pachitariu M, Stringer C, Dipoppa M, Schröder S, Federico Rossi L, Dalgleish H, et al. Suite2p: beyond 10, 000 neurons with standard two-photon microscopy. bioRxiv 2017, DOI:https://doi.org/10.1101/061507.

  54. Hsu AI, Yttri EA. B-SOiD, an open-source unsupervised algorithm for identification and fast prediction of behaviors. Nat Commun 2021, 12: 5188.

  55. Zhou F, de la Torre F, Hodgins JK. Hierarchical aligned cluster analysis for temporal clustering of human motion. IEEE Trans Pattern Anal Mach Intell 2013, 35: 582–596.

    PubMed  Google Scholar 

  56. Zhou F, De la Torre F, Hodgins JK. Aligned cluster analysis for temporal segmentation of human motion. 2008 8th IEEE international conference on automatic face & gesture recognition 2008: 1–7.

  57. Gallego JA, Perich MG, Miller LE, Solla SA. Neural manifolds for the control of movement. Neuron 2017, 94: 978–984.

    CAS  PubMed  PubMed Central  Google Scholar 

  58. Gallego JA, Perich MG, Naufel SN, Ethier C, Solla SA, Miller LE. Cortical population activity within a preserved neural manifold underlies multiple motor behaviors. Nat Commun 2018, 9: 4233.

    PubMed  PubMed Central  Google Scholar 

  59. Salay LD, Ishiko N, Huberman AD. A midline thalamic circuit determines reactions to visual threat. Nature 2018, 557: 183–189.

    CAS  PubMed  PubMed Central  Google Scholar 

  60. Roelofs K. Freeze for action: Neurobiological mechanisms in animal and human freezing. Philos Trans R Soc Lond B Biol Sci 2017, 372: 20160206.

    PubMed  PubMed Central  Google Scholar 

  61. Shang CP, Liu AX, Li DP, Xie ZY, Chen ZJ, Huang MZ, et al. A subcortical excitatory circuit for sensory-triggered predatory hunting in mice. Nat Neurosci 2019, 22: 909–920.

    CAS  PubMed  Google Scholar 

  62. Barthas F, Kwan AC. Secondary motor cortex: Where ‘sensory’ meets ‘motor’ in the rodent frontal cortex. Trends Neurosci 2017, 40: 181–193.

    CAS  PubMed  Google Scholar 

  63. Hong WZ, Kennedy A, Burgos-Artizzu XP, Zelikowsky M, Navonne SG, Perona P, et al. Automated measurement of mouse social behaviors using depth sensing, video tracking, and machine learning. Proc Natl Acad Sci USA 2015, 112: E5351–E5360.

    CAS  PubMed  PubMed Central  Google Scholar 

  64. Sourioux M, Bestaven E, Guillaud E, Bertrand S, Cabanas M, Milan, et al. 3-D motion capture for long-term tracking of spontaneous locomotor behaviors and circadian sleep/wake rhythms in mouse. J Neurosci Methods 2018, 295: 51–57.

    PubMed  Google Scholar 

  65. Preisig DF, Kulic L, Krüger M, Wirth F, McAfoose J, Späni C, et al. High-speed video gait analysis reveals early and characteristic locomotor phenotypes in mouse models of neurodegenerative movement disorders. Behav Brain Res 2016, 311: 340–353.

    PubMed  Google Scholar 

  66. Okayama T, Goto T, Toyoda A. Assessing nest-building behavior of mice using a 3D depth camera. J Neurosci Methods 2015, 251: 151–157.

    PubMed  Google Scholar 

  67. Nakamura T, Matsumoto J, Nishimaru H, Bretas RV, Takamura Y, Hori E, et al. A markerless 3D computerized motion capture system incorporating a skeleton model for monkeys. PLoS One 2016, 11: e0166154. https://doi.org/10.1371/journal.pone.0166154.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  68. Datta SR. Q&A: Understanding the composition of behavior. BMC Biol 2019, 17: 44.

    PubMed  PubMed Central  Google Scholar 

  69. Wiltschko AB, Johnson MJ, Iurilli G, Peterson RE, Katon JM, Pashkovski SL, et al. Mapping sub-second structure in mouse behavior. Neuron 2015, 88: 1121–1135.

    CAS  PubMed  PubMed Central  Google Scholar 

  70. Hasan CZC, Jailani R, Tahir NM, Yassin IM, Rizman ZI. Automated classification of autism spectrum disorders gait patterns using discriminant analysis based on kinematic and kinetic gait features. Journal of Applied Environmental and Biological Sciences 2017, 7: 150–156.

    Google Scholar 

  71. Cho CW, Chao WH, Lin SH, Chen YY. A vision-based analysis system for gait recognition in patients with Parkinson’s disease. Expert Syst Appl 2009, 36: 7033–7039.

    Google Scholar 

  72. Gowen E, Miall RC. The cerebellum and motor dysfunction in neuropsychiatric disorders. Cerebellum 2007, 6: 268–279.

    CAS  PubMed  PubMed Central  Google Scholar 

  73. Michel D, Qammaz A, Argyros AA. Markerless 3d human pose estimation and tracking based on rgbd cameras: an experimental evaluation. Proceedings of the 10th International Conference on PErvasive Technologies Related to Assistive Environments 2017: 115–122.

  74. Gomez-Marin A, Paton JJ, Kampff AR, Costa RM, Mainen ZF. Big behavioral data: Psychology, ethology and the foundations of neuroscience. Nat Neurosci 2014, 17: 1455–1462.

    CAS  PubMed  Google Scholar 

  75. Arac A, Zhao PP, Dobkin BH, Carmichael ST, Golshani P. DeepBehavior: A deep learning toolbox for automated analysis of animal and human behavior imaging data. Front Syst Neurosci 2019, 13: 20.

    PubMed  PubMed Central  Google Scholar 

  76. Brodkin J, Frank D, Grippo R, Hausfater M, Gulinello M, Achterholt N, et al. Validation and implementation of a novel high-throughput behavioral phenotyping instrument for mice. J Neurosci Methods 2014, 224: 48–57.

    PubMed  Google Scholar 

  77. Uddin LQ. Bring the noise: Reconceptualizing spontaneous neural activity. Trends Cogn Sci 2020, 24: 734–746.

    PubMed  PubMed Central  Google Scholar 

  78. Altimus CM, Marlin BJ, Charalambakis NE, Colón-Rodriquez A, Glover EJ, Izbicki P, et al. The next 50 years of neuroscience. J Neurosci 2020, 40: 101–106.

    CAS  PubMed  PubMed Central  Google Scholar 

  79. Chen ZX, Zhang RH, Zhang YE, Zhou HW, Fang HS, Rock RR, et al. AlphaTracker: A multi-animal tracking and behavioral analysis tool. bioRxiv 2020, DOI:https://doi.org/10.1101/2020.12.04.405159.

  80. Pereira TD, Tabris N, Li JY, Ravindranath S, Papadoyannis ES, Yan Wang Z, et al. SLEAP: Multi-animal pose tracking. bioRxiv 2020, DOI:https://doi.org/10.1101/2020.08.31.276246.

  81. Ebbesen CL, Froemke RC. Automatic tracking of mouse social posture dynamics by 3D videography, deep learning and GPU-accelerated robust optimization. bioRxiv 2020, https://doi.org/10.1101/2020.05.21.109629.

  82. Cheng B, Collins MD, Zhu Y, Liu T, Huang TS, Adam H, et al. Panoptic-deeplab: A simple, strong, and fast baseline for bottom-up panoptic segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition 2020: 12475–12485.

  83. Zimmermann C, Schneider A, Alyahyay M, Brox T, Diester I. FreiPose: A deep learning framework for precise animal motion capture in 3D spaces. bioRxiv 2020, DOI:https://doi.org/10.1101/2020.02.27.967620.

  84. Marshall JD, Aldarondo DE, Dunn TW, Wang WL, Berman GJ, Ölveczky BP. Continuous whole-body 3D kinematic recordings across the rodent behavioral repertoire. Neuron 2021, 109: 420-437.e8.

    CAS  PubMed  Google Scholar 

Download references

Acknowledgements

This work was supported by the Key Area R&D Program of Guangdong Province, China (2018B030338001 and 2018B030331001), the National Key R&D Program of China (2018YFA0701403), the National Natural Science Foundation of China (31500861, 31630031, 91732304, and 31930047), Chang Jiang Scholars Program, the International Big Science Program Cultivating Project of the Chinese Academy of Science (CAS; 172644KYS820170004), the Strategic Priority Research Program of the CAS (XDB32030100), the Youth Innovation Promotion Association of the CAS (2017413), the CAS Key Laboratory of Brain Connectome and Manipulation (2019DP173024), Shenzhen Government Basic Research Grants (JCYJ20170411140807570, JCYJ20170413164535041), the Science, Technology and Innovation Commission of Shenzhen Municipality (JCYJ20160429185235132), a Helmholtz-CAS Joint Research grant (GJHZ1508), Guangdong Provincial Key Laboratory of Brain Connectome and Behavior (2017B030301017), the Ten Thousand Talent Program, the Guangdong Special Support Program, Key Laboratory of Shenzhen Institute of Advanced Technology (2019DP173024), and the Shenzhen Key Science and Technology Infrastructure Planning Project (ZDKJ20190204002).

Author information

Authors and Affiliations

Authors

Corresponding authors

Correspondence to Liping Wang or Pengfei Wei.

Ethics declarations

Conflict of Interest

All authors claim that there are no conflicts of interest.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Han, Y., Huang, K., Chen, K. et al. MouseVenue3D: A Markerless Three-Dimension Behavioral Tracking System for Matching Two-Photon Brain Imaging in Free-Moving Mice. Neurosci. Bull. 38, 303–317 (2022). https://doi.org/10.1007/s12264-021-00778-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12264-021-00778-6

Keywords

Navigation