Medical & Biological Engineering & Computing

, Volume 56, Issue 10, pp 1807–1821 | Cite as

An automated behavior analysis system for freely moving rodents using depth image

  • Zheyuan Wang
  • S. Abdollah Mirbozorgi
  • Maysam Ghovanloo
Original Article


A rodent behavior analysis system is presented, capable of automated tracking, pose estimation, and recognition of nine behaviors in freely moving animals. The system tracks three key points on the rodent body (nose, center of body, and base of tail) to estimate its pose and head rotation angle in real time. A support vector machine (SVM)-based model, including label optimization steps, is trained to classify on a frame-by-frame basis: resting, walking, bending, grooming, sniffing, rearing supported, rearing unsupported, micro-movements, and “other” behaviors. Compared to conventional red-green-blue (RGB) camera-based methods, the proposed system operates on 3D depth images provided by the Kinect infrared (IR) camera, enabling stable performance regardless of lighting conditions and animal color contrast with the background. This is particularly beneficial for monitoring nocturnal animals’ behavior. 3D features are designed to be extracted directly from the depth stream and combined with contour-based 2D features to further improve recognition accuracies. The system is validated on three freely behaving rats for 168 min in total. The behavior recognition model achieved a cross-validation accuracy of 86.8% on the rat used for training and accuracies of 82.1 and 83% on the other two “testing” rats. The automated head angle estimation aided by behavior recognition resulted in 0.76 correlation with human expert annotation.

Graphical abstract

Top view of a rat freely behaving in a standard homecage, captured by Kinect-v2 sensors. The depth image is used for constructing a 3D topography of the animal for pose estimation, behavior recognition, and head angle calculation. Results of the processed data are displayed on the user interface in various forms.


Kinect sensor Depth image Animal tracking Pose estimation Feature extraction Behavior recognition Support vector machine 



This work was supported in part by the National Institutes of Health award 1R21EB018561 and the National Science Foundation under award ECCS-1407880 and ECCS-1408318. The authors would like to thank Prof. Rainnie and members of his lab at the Emory University Department of Psychiatry for their assistance in conducting the in vivo experiments.

Compliance with ethical standards

The experiment, which was conducted as part of the evaluation of the EnerCage-HC2 system, was approved by the Institutional Animal Care and Use Committees (IACUC) at Emory University and Georgia Tech.

All procedures performed in studies involving animals were in accordance with the ethical standards of the institution or practice at which the studies were conducted.


  1. 1.
    Tecott LH, Nestler EJ (2004) Neurobehavioral assessment in the information age. Nat Neurosci 7:462–466CrossRefGoogle Scholar
  2. 2.
    van Meer PJK, Graham ML, Schuurman H (2015) The safety, efficacy and regulatory triangle in drug development: impact for animal models and the use of animals. Eur J Pharmacol 759:3–13CrossRefGoogle Scholar
  3. 3.
    Baker M (2011) Animal models: inside the minds of mice and men. Nature 475:123–128CrossRefGoogle Scholar
  4. 4.
    Rat Genome Sequencing Project Consortium (2004) Genome sequence of the Brown Norway rat yields insights into mammalian evolution. Nature 428:493–521CrossRefGoogle Scholar
  5. 5.
    Shi Q, Miyagishima S, Fumino S et al (2010) Development of a cognition system for analyzing rat’s behaviors. In: IEEE Int Conf Robotics and Biomimetics (ROBIO), pp 1399–1401Google Scholar
  6. 6.
    Shi Q, Ishii H, Konno S et al (2012) Image processing and behavior planning for robot-rat interaction. In: IEEE RAS & EMBS Int. Conf. Biomedical Robotics and Biomechatronics (BioRob), pp 967–973Google Scholar
  7. 7.
    Jhuang H, Garrote E, Yu X, Khilnani V, Poggio T, Steele AD, Serre T (2010) Automated home-cage behavioural phenotyping of mice. Nat Commun 1(68):1–9CrossRefGoogle Scholar
  8. 8.
    Burgos-Artizzu XP, Dollár P, Lin D et al (2012) Social behavior recognition in continuous videos. In: IEEE Conf. Computer vision and pattern recognition (CVPR 2012), pp 1322–1329Google Scholar
  9. 9.
    van Dam EA, van der Harst JE, ter Braak CJF et al (2013) An automated system for the recognition of various specific rat behaviours. J Neurosci Methods 218(2):214–224CrossRefGoogle Scholar
  10. 10.
    Patel TP, Gullotti DM, Hernandez P, O'Brien WT, Capehart BP, Morrison B, Bass C, Eberwine JE, Abel T, Meaney DF (2014) An open-source toolbox for automated phenotyping of mice in behavioral tasks. Front Behav Neurosci 8:349CrossRefGoogle Scholar
  11. 11.
    Brodkin J, Frank D, Grippo R, Hausfater M, Gulinello M, Achterholt N, Gutzen C (2014) Validation and implementation of a novel high-throughput behavioral phenotyping instrument for mice. J Neurosci Methods 224:48–57CrossRefGoogle Scholar
  12. 12.
    Lorbach M, Kyriakou EI, Poppe R et al (2017) Learning to recognize rat social behavior: Novel dataset and cross-dataset application. J Neurosci Methods. in press, available onlineGoogle Scholar
  13. 13.
    Ren Z, Noronha A, Ciernia AV et al (2017) Who moved my cheese? Automatic annotation of rodent behaviors with convolutional neural networks. In: IEEE Winter Conf. Applications computer vision (WACV), pp 1277–1286Google Scholar
  14. 14.
    Crispim-Junior CF, de Azevedo FM, Marino-Neto J (2017) What is my rat doing? Behavior understanding of laboratory animals. Pattern Recogn Lett 94:134–143CrossRefGoogle Scholar
  15. 15.
    Han J, Shao L, Xu D, Shotton J (2013) Enhanced computer vision with Microsoft Kinect sensor: a review. IEEE Trans Cybern 43(5):1318–1334CrossRefGoogle Scholar
  16. 16.
    Hu M, Chen C, Cheng W et al (2015) Real-time human movement retrieval and assessment with Kinect sensor. IEEE Trans Cybern 45(4):742–753CrossRefGoogle Scholar
  17. 17.
    Whitmire E, Latif T, Bozkurt A (2013) Kinect-based system for automated control of terrestrial insect biobots. In: Int. Conf. IEEE EMBC, pp 1470–1473Google Scholar
  18. 18.
    Wiltschko AB, Johnson MJ, Iurilli G, Peterson RE, Katon JM, Pashkovski SL, Abraira VE, Adams RP, Datta SR (2015) Mapping sub-second structure in mouse behavior. Neuron 88(6):1121–1135CrossRefGoogle Scholar
  19. 19.
    Lee B, Kiani M, Ghovanloo M (2014) A smart homecage system with 3D tracking for long-term behavioral experiments. In: Int. Conf. IEEE EMBC, pp 2016-2019Google Scholar
  20. 20.
    Lee B, Kiani M, Ghovanloo M (2015) A smart wirelessly powered homecage for long-term high-throughput behavioral experiments. IEEE Sensors J 14(9):4905–4916CrossRefGoogle Scholar
  21. 21.
    Ou-Yang TH, Tsai ML, Yen CT, Lin TT (2011) An infrared range camera-based approach for three-dimensional locomotion tracking and pose reconstruction in a rodent. J Neurosci Methods 201(1):116–123CrossRefGoogle Scholar
  22. 22.
    Matsumoto J, Urakawa S, Takamura Y, Malcher-Lopes R, Hori E, Tomaz C, Ono T, Nishijo H (2013) A 3D-video-based computerized analysis of social and sexual interactions in rats. PLoS One 8(10):e78460CrossRefGoogle Scholar
  23. 23.
    Matsumoto J, Nishimaru H, Ono T, Nishijo H (2017) 3D-video-based computerized behavioral analysis for in vivo neuropharmacology and neurophysiology in rodents. In: Philippu A (ed) In Vivo Neuropharmacology and Neurophysiology. Neuromethods, vol 121. Humana Press, New YorkGoogle Scholar
  24. 24.
    Nakamura A, Funaya H, Uezono N, Nakashima K, Ishida Y, Suzuki T, Wakana S, Shibata T (2015) Low-cost three-dimensional gait analysis system for mice with an infrared depth sensor. Neurosci Res 100:55–62CrossRefGoogle Scholar
  25. 25.
    Xu C, Govindarajan LN, Zhang Y, Cheng L (2017) Lie-X: depth image based articulated object pose estimation, tracking, and action recognition on lie groups. Int J Comput Vis (IJCV) 123(3):454–478CrossRefGoogle Scholar
  26. 26.
    Rezaei B, Lowe J, Yee JR et al (2016) Non-contact automatic respiration monitoring in restrained rodents. In: Int. Conf. IEEE EMBC, pp 4946–4950Google Scholar
  27. 27.
    Rezaei B, Huang X, Yee JR, Ostadabbas S (2017) Long-term non-contact tracking of caged rodents. In: 42nd IEEE Int. Conf. Acoustics, Speech and Signal Processing (ICASSP’17), pp 1952–1956Google Scholar
  28. 28.
    Monteiro JP, Oliveira HP, Aguirar P, Cardoso JS (2014) A depth-map approach for automatic mice behavior recognition. In: IEEE Int. Conf. Image Processing (ICIP), pp 2261–2265Google Scholar
  29. 29.
    Hong W, Kennedy A, Burgos-Artizzu XP, Zelikowsky M, Navonne SG, Perona P, Anderson DJ (2015) Automated measurement of mouse social behaviors using depth sensing, video tracking, and machine learning. PNAS 112(38):E5351–E5360CrossRefGoogle Scholar
  30. 30.
    Wang Z, Mirbozorgi SA, Ghovanloo M (2015) Towards a Kinect-based behavior recognition and analysis system for small animals. In: IEEE biomed. Circ. Sys. Conf. (BioCAS), pp 683–686Google Scholar
  31. 31.
    Jia Y, Wang Z, Canales D et al (2016) A wirelessly-powered homecage with animal behavior analysis and closed-loop power control. In: Int Conf IEEE EMBC, pp 6323–6326Google Scholar
  32. 32.
    Jia Y, Mirbozorgi SA, Wang Z, Ghovanloo M (2017) Position and orientation insensitive wireless power transmission for EnerCage-Homecage system. IEEE Trans Biomed Eng 64(10):2439–2449CrossRefGoogle Scholar
  33. 33.
    Gonzalez RC, Woods RE (2008) Digital image processing, 3rd edn. Prentice Hall, Upper Saddle RiverGoogle Scholar
  34. 34.
    Lorbach M, Poppe R, van Dam EA et al (2015) Automated recognition of social behavior in rats: the role of feature quality. In: Int. Conf. Image analysis processing (ICIAP 2015), pp 565–574CrossRefGoogle Scholar
  35. 35.
    Fuzesi T, Daviu N, Cusulin JIW et al (2016) Hypothalamic CRH neurons orchestrate complex behaviours after stress. Nat Commun 7:1–13CrossRefGoogle Scholar
  36. 36.
    Hsu CW, Lin CJ (2002) A comparison of methods for multiclass support vector machines. IEEE Trans Neural Netw 13(2):415–425CrossRefGoogle Scholar
  37. 37.
    Chang CC, Lin CJ (2011) LIBSVM: a library for support vector machines. ACM Trans Intell Syst Technol 2(2):1–27CrossRefGoogle Scholar
  38. 38.
    Kelland MD, Asdourian D, Kelland DZ (1988) Inhibition and excitation of neck and shoulder muscles during unilateral electrical stimulation of the rat neostriatum. Behav Brain Res 30:1–13CrossRefGoogle Scholar
  39. 39.
    Chen L, Chan SCY, Yung WH (2002) Rotational behavior and electrophysiological effects induced by GABAB receptor activation in rat globus pallidus. Neuroscience 144(2):417–425CrossRefGoogle Scholar
  40. 40.
    Goutte C, Gaussier E (2005) A probabilistic interpretation of precision, recall and F-score, with implication for evaluation. In: 27th European Conf. Advances inform. Retrieval research, pp 345–359Google Scholar

Copyright information

© International Federation for Medical and Biological Engineering 2018

Authors and Affiliations

  1. 1.GT-Bionics Lab, School of Electrical and Computer EngineeringGeorgia Institute of TechnologyAtlantaUSA

Personalised recommendations