Skip to main content

An Easily Compatible Eye-tracking System for Freely-moving Small Animals

Neuroscience Bulletin volume 38, pages 661–676 (2022)Cite this article

Abstract

Measuring eye movement is a fundamental approach in cognitive science as it provides a variety of insightful parameters that reflect brain states such as visual attention and emotions. Combining eye-tracking with multimodal neural recordings or manipulation techniques is beneficial for understanding the neural substrates of cognitive function. Many commercially-available and custom-built systems have been widely applied to awake, head-fixed small animals. However, the existing eye-tracking systems used in freely-moving animals are still limited in terms of their compatibility with other devices and of the algorithm used to detect eye movements. Here, we report a novel system that integrates a general-purpose, easily compatible eye-tracking hardware with a robust eye feature-detection algorithm. With ultra-light hardware and a detachable design, the system allows for more implants to be added to the animal’s exposed head and has a precise synchronization module to coordinate with other neural implants. Moreover, we systematically compared the performance of existing commonly-used pupil-detection approaches, and demonstrated that the proposed adaptive pupil feature-detection algorithm allows the analysis of more complex and dynamic eye-tracking data in free-moving animals. Synchronized eye-tracking and electroencephalogram recordings, as well as algorithm validation under five noise conditions, suggested that our system is flexibly adaptable and can be combined with a wide range of neural manipulation and recording technologies.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

USD 39.95

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

References

  1. Vázquez-Guardado A, Yang YY, Bandodkar AJ, Rogers JA. Recent advances in neurotechnologies with broad potential for neuroscience research. Nat Neurosci 2020, 23: 1522–1536.

    PubMed  Article  CAS  Google Scholar 

  2. Parker PRL, Brown MA, Smear MC, Niell CM. Movement-related signals in sensory areas: Roles in natural behavior. Trends Neurosci 2020, 43: 581–595.

    CAS  PubMed  PubMed Central  Article  Google Scholar 

  3. Byrom B, McCarthy M, Schueler P, Muehlhausen W. Brain monitoring devices in neuroscience clinical research: The potential of remote monitoring using sensors, wearables, and mobile devices. Clin Pharmacol Ther 2018, 104: 59–71.

    PubMed  PubMed Central  Article  Google Scholar 

  4. Mathôt S. Pupillometry: psychology, physiology, and function. J Cogn 2018, 1: 16.

    PubMed  PubMed Central  Article  Google Scholar 

  5. Lim JZ, Mountstephens J, Teo J. Emotion recognition using eye-tracking: Taxonomy, review and current challenges. Sensors (Basel) 2020, 20: E2384.

    Article  Google Scholar 

  6. Joshi S, Gold JI. Pupil size as a window on neural substrates of cognition. Trends Cogn Sci 2020, 24: 466–480.

    PubMed  PubMed Central  Article  Google Scholar 

  7. Kret ME, Sjak-Shie EE. Preprocessing pupil size data: Guidelines and code. Behav Res Methods 2019, 51: 1336–1342.

    PubMed  Article  Google Scholar 

  8. Dennis EJ, El Hady A, Michaiel A, Clemens A, Tervo DRG, Voigts J. Systems neuroscience of natural behaviors in rodents. J Neurosci 2021, 41: 911–919.

    CAS  PubMed  PubMed Central  Article  Google Scholar 

  9. Payne HL, Raymond JL. Magnetic eye tracking in mice. Elife 2017, 6: e29222.

    PubMed  PubMed Central  Article  Google Scholar 

  10. Tresanchez M, Pallejà T, Palacín J. Optical mouse sensor for eye blink detection and pupil tracking: Application in a low-cost eye-controlled pointing device. J Sensors 2019, 2019: 3931713.

    Article  Google Scholar 

  11. Fuhl W, Tonsen M, Bulling A, Kasneci E. Pupil detection for head-mounted eye tracking in the wild: An evaluation of the state of the art. Mach Vis Appl 2016, 27: 1275–1288.

    Article  Google Scholar 

  12. van der Wel P, van Steenbergen H. Pupil dilation as an index of effort in cognitive control tasks: A review. Psychon Bull Rev 2018, 25: 2005–2015.

    PubMed  PubMed Central  Article  Google Scholar 

  13. Maier SU, Grueschow M. Pupil dilation predicts individual self-regulation success across domains. Sci Rep 2021, 11: 1–18.

    Article  CAS  Google Scholar 

  14. Hu YZ, Jiang HH, Liu CR, Wang JH, Yu CY, Carlson S, et al. What interests them in the pictures? —Differences in eyetracking between rhesus monkeys and humans. Neurosci Bull 2013, 29: 553–564.

    PubMed  PubMed Central  Article  Google Scholar 

  15. Cheng YH, Liu WJ, Yuan XY, Jiang Y. The eyes have it: Perception of social interaction unfolds through pupil dilation. Neurosci Bull 2021, 37: 1595–1598.

    PubMed  Article  Google Scholar 

  16. Rusch T, Korn CW, Gläscher J. A two-way street between attention and learning. Neuron 2017, 93: 256–258.

    CAS  PubMed  Article  Google Scholar 

  17. Clewett D, Gasser C, Davachi L. Pupil-linked arousal signals track the temporal organization of events in memory. Nat Commun 2020, 11: 4007.

    CAS  PubMed  PubMed Central  Article  Google Scholar 

  18. Itti L. New eye-tracking techniques may revolutionize mental health screening. Neuron 2015, 88: 442–444.

    CAS  PubMed  Article  Google Scholar 

  19. Olpinska-Lischka M, Kujawa K, Wirth JA, Antosiak-Cyrak KZ, Maciaszek J. The influence of 24-hr sleep deprivation on psychomotor vigilance in young women and men. Nat Sci Sleep 2020, 12: 125–134.

    PubMed  PubMed Central  Article  Google Scholar 

  20. Stitt I, Zhou ZC, Radtke-Schuller S, Fröhlich F. Arousal dependent modulation of thalamo-cortical functional interaction. Nat Commun 2018, 9: 2455.

    PubMed  PubMed Central  Article  CAS  Google Scholar 

  21. Milton R, Shahidi N, Dragoi V. Dynamic states of population activity in prefrontal cortical networks of freely-moving macaque. Nat Commun 1948, 2020: 11.

    Google Scholar 

  22. Lawson RP, Mathys C, Rees G. Adults with autism overestimate the volatility of the sensory environment. Nat Neurosci 2017, 20: 1293–1299.

    CAS  PubMed  PubMed Central  Article  Google Scholar 

  23. Constantino JN, Kennon-Mcgill S, Weichselbaum C, Marrus N, Haider A, Glowinski AL, et al. Infant viewing of social scenes is under genetic control and is atypical in autism. Nature 2017, 547: 340–344.

    CAS  PubMed  PubMed Central  Article  Google Scholar 

  24. Lotankar S, Prabhavalkar KS, Bhatt LK. Biomarkers for Parkinson’s disease: Recent advancement. Neurosci Bull 2017, 33: 585–597.

    CAS  PubMed  PubMed Central  Article  Google Scholar 

  25. Katus L, Hayes NJ, Mason L, Blasi A, McCann S, Darboe MK, et al. Implementing neuroimaging and eye tracking methods to assess neurocognitive development of young infants in low- and middle-income countries. Gates Open Res 2019, 3: 1113.

    PubMed  PubMed Central  Article  Google Scholar 

  26. Lio G, Fadda R, Doneddu G, Duhamel JR, Sirigu A. Digit-tracking as a new tactile interface for visual perception analysis. Nat Commun 2019, 10: 5392.

    PubMed  PubMed Central  Article  CAS  Google Scholar 

  27. Simmonds L, Bellman S, Kennedy R, Nenycz-Thiel M, Bogomolova S. Moderating effects of prior brand usage on visual attention to video advertising and recall: An eye-tracking investigation. J Bus Res 2020, 111: 241–248.

    Article  Google Scholar 

  28. Guo J, Kurup U, Shah M. Is it safe to drive? An overview of factors, metrics, and datasets for driveability assessment in autonomous driving. IEEE Transactions on Intelligent Transportation Systems 2020, 21: 3135–3151.

    Article  Google Scholar 

  29. Privitera M, Ferrari KD, von Ziegler LM, Sturman O, Duss SN, Floriou-Servou A, et al. A complete pupillometry toolbox for real-time monitoring of locus coeruleus activity in rodents. Nat Protoc 2020, 15: 2301–2320.

    CAS  PubMed  Article  Google Scholar 

  30. Meng QS, Tan XR, Jiang CY, Xiong YY, Yan B, Zhang JY. Tracking eye movements during sleep in mice. Front Neurosci 2021, 15: 616760.

    PubMed  PubMed Central  Article  Google Scholar 

  31. Schwarz JS, Sridharan D, Knudsen EI. Magnetic tracking of eye position in freely behaving chickens. Front Syst Neurosci 2013, 7: 91.

    PubMed  PubMed Central  Article  Google Scholar 

  32. Yorzinski JL. A songbird inhibits blinking behaviour in flight. Biol Lett 2020, 16: 20200786.

    PubMed  PubMed Central  Article  Google Scholar 

  33. Wallace DJ, Greenberg DS, Sawinski J, Rulla S, Notaro G, Kerr JND. Rats maintain an overhead binocular field at the expense of constant fusion. Nature 2013, 498: 65–69.

    CAS  PubMed  Article  Google Scholar 

  34. Meyer AF, Poort J, O’Keefe J, Sahani M, Linden JF. A head-mounted camera system integrates detailed behavioral monitoring with multichannel electrophysiology in freely moving mice. Neuron 2018, 100: 46-60.e7.

    CAS  PubMed  PubMed Central  Article  Google Scholar 

  35. Michaiel AM, Abe ET, Niell CM. Dynamics of gaze control during prey capture in freely moving mice. Elife 2020, 9: e57458.

    CAS  PubMed  PubMed Central  Article  Google Scholar 

  36. Mathis A, Mamidanna P, Cury KM, Abe T, Murthy VN, Mathis MW, et al. DeepLabCut: markerless pose estimation of user-defined body parts with deep learning. Nat Neurosci 2018, 21: 1281–1289.

    CAS  PubMed  Article  Google Scholar 

  37. Chen LC, Zhu YK, Papandreou G, Schroff F, Adam H. Encoder-decoder with atrous separable convolution for semantic image segmentation. Comput Vis – ECCV 2018, 2018: 833–851.

    Google Scholar 

  38. Long J, Shelhamer E, Darrell T. Fully convolutional networks for semantic segmentation. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2015, 2015: 3431–3440.

    Google Scholar 

  39. Ronneberger O, Fischer P, Brox T. U-net: Convolutional networks for biomedical image segmentation. Med Image Comput Comput Assist Interv – MICCAI 2015, 2015: 234–241.

    Google Scholar 

  40. Lopes G, Bonacchi N, Frazão J, Neto JP, Atallah BV, Soares S, et al. Bonsai: an event-based framework for processing and controlling data streams. Front Neuroinform 2015, 9: 7.

    PubMed  PubMed Central  Article  Google Scholar 

  41. Naylor E, Aillon DV, Gabbert S, Harmon H, Johnson DA, Wilson GS, et al. Simultaneous real-time measurement of EEG/EMG and L-glutamate in mice: A biosensor study of neuronal activity during sleep. J Electroanal Chem (Lausanne) 2011, 656: 106–113.

    CAS  Article  Google Scholar 

  42. Yüzgeç Ö, Prsa M, Zimmermann R, Huber D. Pupil size coupling to cortical states protects the stability of deep sleep via parasympathetic modulation. Curr Biol 2018, 28: 392-400.e3.

    PubMed  PubMed Central  Article  CAS  Google Scholar 

  43. Bradley D, Roth G. Adaptive thresholding using the integral image. J Graph Tools 2007, 12: 13–21.

    Article  Google Scholar 

  44. Gao B, Hu G, Gao S, Zhong Y, Gu C. Multi-sensor optimal data fusion based on the adaptive fading unscented Kalman filter. Sensors (Basel) 2018, 18: 488.

    Article  Google Scholar 

  45. Akhlaghi S, Zhou N, Huang Z. Adaptive adjustment of noise covariance in Kalman filter for dynamic state estimation. IEEE Power & Energy Society General Meeting 2017, 2017: 1–5.

    Google Scholar 

  46. Mourikis AI, Roumeliotis SI. A multi-state constraint Kalman filter for vision-aided inertial navigation. Proceedings 2007 IEEE International Conference on Robotics and Automation, 2007: 3565–3572. DOI: https://doi.org/10.1109/robot.2007.364024.

  47. Lai ZL, Lei Y, Zhu SY, Xu YL, Zhang XH, Krishnaswamy S. Moving-window extended Kalman filter for structural damage detection with unknown process and measurement noises. Measurement 2016, 88: 428–440.

    Article  Google Scholar 

  48. Fuhl W, Santini T, Kübler TC, Kasneci E. ElSe: Ellipse selection for robust pupil detection in real-world environments. Proceedings of the Ninth Biennial ACM Symposium on Eye Tracking Research & Applications 2016: 123–130.

  49. Li DH, Winfield D, Parkhurst DJ. Starburst: A hybrid algorithm for video-based eye tracking combining feature-based and model-based approaches. 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) - Workshops 2005: 79.

  50. Fuhl W, Geisler D, Santini T, Appel T, Rosenstiel W, Kasneci E. CBF: Circular binary features for robust and real-time pupil center detection. Proceedings of the 2018 ACM Symposium on Eye Tracking Research & Applications 2018, Article 8: 1–6.

  51. Fuhl W, Kübler T, Sippel K, Rosenstiel W, Kasneci E. Excuse: Robust pupil detection in real-world scenarios. International Conference on Computer Analysis of Images and Patterns 2015: 39–51.

  52. Fuhl W. 1000 pupil segmentations in a second using haar like features and statistical learning. 2021: arXiv: 2102.01921[eess.IV]. https://arxiv.org/abs/2102.01921

  53. Fuhl W, Santini T, Kasneci G, Kasneci E. PupilNet: convolutional neural networks for robust pupil detection. 2016: arXiv: 1601.04902[cs.CV]. https://arxiv.org/abs/1601.04902

  54. Kitazumi K, Nakazawa A. Robust pupil segmentation and center detection from visible light images using convolutional neural network. 2018 IEEE International Conference on Systems, Man, and Cybernetics 2018: 862–868.

  55. Yiu YH, Aboulatta M, Raiser T, Ophey L, Flanagin VL, Zu Eulenburg P, et al. DeepVOG: Open-source pupil segmentation and gaze estimation in neuroscience using deep learning. J Neurosci Methods 2019, 324: 108307.

    PubMed  Article  Google Scholar 

  56. Fuhl W, Gao H, Kasneci E. Tiny convolution, decision tree, and binary neuronal networks for robust and real time pupil outline estimation. ACM Symposium on Eye Tracking Research and Applications. 2020, Article 5: 1–5.

  57. Bushnell M, Umino Y, Solessio E. A system to measure the pupil response to steady lights in freely behaving mice. J Neurosci Methods 2016, 273: 74–85.

    PubMed  PubMed Central  Article  Google Scholar 

  58. Zong WJ, Wu RL, Li ML, Hu YH, Li YJ, Li JH, et al. Fast high-resolution miniature two-photon microscopy for brain imaging in freely behaving mice. Nat Methods 2017, 14: 713–719.

    CAS  PubMed  Article  Google Scholar 

  59. Klioutchnikov A, Wallace DJ, Frosz MH, Zeltner R, Sawinski J, Pawlak V, et al. Three-photon head-mounted microscope for imaging deep cortical layers in freely moving rats. Nat Methods 2020, 17: 509–513.

    CAS  PubMed  Article  Google Scholar 

  60. Griffiths VA, Valera AM, Lau JY, Roš H, Younts TJ, Marin B, et al. Real-time 3D movement correction for two-photon imaging in behaving animals. Nat Methods 2020, 17: 741–748.

    CAS  PubMed  PubMed Central  Article  Google Scholar 

  61. Juavinett AL, Bekheet G, Churchland AK. Chronically implanted Neuropixels probes enable high-yield recordings in freely moving mice. Elife 2019, 8: e47188.

    CAS  PubMed  PubMed Central  Article  Google Scholar 

  62. Karl D. Optogenetics. Nat Methods 2011, 8: 26–29.

    Article  CAS  Google Scholar 

  63. Zhang F, Wang LP, Brauner M, Liewald JF, Kay K, Watzke N, et al. Multimodal fast optical interrogation of neural circuitry. Nature 2007, 446: 633–639.

    CAS  PubMed  Article  Google Scholar 

  64. Stowers JR, Hofbauer M, Bastien R, Griessner J, Higgins P, Farooqui S, et al. Virtual reality for freely moving animals. Nat Methods 2017, 14: 995–1002.

    CAS  PubMed  PubMed Central  Article  Google Scholar 

  65. Huang K, Han YN, Chen K, Pan HL, Zhao GY, Yi WL, et al. A hierarchical 3D-motion learning framework for animal spontaneous behavior mapping. Nat Commun 2021, 12: 2784.

    CAS  PubMed  PubMed Central  Article  Google Scholar 

  66. Lamirel C, Ajasse S, Moulignier A, Salomon L, Deschamps R, Gueguen A, et al. A novel method of inducing endogenous pupil oscillations to detect patients with unilateral optic neuritis. PLoS One 2018, 13: e0201730.

    PubMed  PubMed Central  Article  CAS  Google Scholar 

  67. Naber M, Alvarez GA, Nakayama K. Tracking the allocation of attention using human pupillary oscillations. Front Psychol 2013, 4: 919.

    PubMed  PubMed Central  Article  Google Scholar 

  68. Naber M, Roelofzen C, Fracasso A, Bergsma DP, van Genderen M, Porro GL, et al. Gaze-contingent flicker pupil perimetry detects scotomas in patients with cerebral visual impairments or glaucoma. Front Neurol 2018, 9: 558.

    PubMed  PubMed Central  Article  Google Scholar 

Download references

Acknowledgements

We thank Dr. Feng Wang for her comments on our manuscript and providing the animals. This work was supported in part by the National Key R&D Program of China (2021ZD0203902 and 2018YFA0701403), the Key Area R&D Program of Guangdong Province (2018B030338001 and 2018B030331001), the National Natural Science Foundation of China (31500861, 31630031, 91732304, and 31930047), the Chang Jiang Scholars Program and the Ten Thousand Talent Program, the International Big Science Program Cultivating Project of the Chinese Academy of Science (CAS) (172644KYS820170004), the Strategic Priority Research Program of the CAS (XDB32030100), the Youth Innovation Promotion Association of the CAS (2017413), Shenzhen Government Basic Research Grants (JCYJ20170411140807570, JCYJ20170413164535041), the Science, Technology and Innovation Commission of Shenzhen Municipality (JCYJ20160429185235132), a Helmholtz-CAS joint research grant (GJHZ1508), the Guangdong Provincial Key Laboratory of Brain Connectome and Behavior (2017B030301017), the Guangdong Special Support Program, the Key Laboratory of the CAS (2019DP173024), the Shenzhen Key Science and Technology Infrastructure Planning Project (ZDKJ20190204002).

Author information

Author notes

  1. Kang Huang, Qin Yang and Yaning Han have contributed equally to this work.

Authors and Affiliations

  1. Shenzhen Key Lab of Neuropsychiatric Modulation and Collaborative Innovation Center for Brain Science, Guangdong Provincial Key Laboratory of Brain Connectome and Behavior, CAS Center for Excellence in Brain Science and Intelligence Technology, Brain Cognition and Brain Disease Institute, Shenzhen-Hong Kong Institute of Brain Science-Shenzhen Fundamental Research Institutions, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen, 518055, China

    Kang Huang, Qin Yang, Yaning Han, Yulin Zhang, Liping Wang & Pengfei Wei

  2. University of Chinese Academy of Sciences, Beijing, 100049, China

    Kang Huang, Yaning Han, Liping Wang & Pengfei Wei

  3. Harbin Institute of Technology Shenzhen, Shenzhen, 518055, China

    Zhiyi Wang

Authors
  1. Kang Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Qin Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Yaning Han
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Yulin Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Zhiyi Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Liping Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Pengfei Wei
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Pengfei Wei.

Ethics declarations

Conflict of Interest

The authors declare no competing interests.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file2 (MP4 3993 kb)

Supplementary file3 (MP4 8064 kb)

Supplementary file4 (MP4 10553 kb)

Supplementary file1 (PDF 408 kb)

Rights and permissions

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Huang, K., Yang, Q., Han, Y. et al. An Easily Compatible Eye-tracking System for Freely-moving Small Animals. Neurosci. Bull. 38, 661–676 (2022). https://doi.org/10.1007/s12264-022-00834-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12264-022-00834-9

Keywords

  • Eye-tracking
  • Freely-moving
  • Head-mounted device
  • Pupil detection
  • Adaptive Kalman filter