When the Brain Meets the Eye: Tracking Object Motion

  • Guillaume S. Masson
  • Anna Montagnini
  • Uwe J. Ilg


To accurately track a moving object of interest with appropriate smooth eye movements, the brain needs to reconstruct a single velocity vector describing the global motion of this object. Because of the aperture problem (see Chap. 1), the visual system must integrate piecewise local information from either elongated edges and contours or particular features such as corners and texture elements. Here, we show that investigating smooth eye movements unveil several dynamical properties of this visual motion integration stage. Signals are weighted according to their uncertainties. The integration is highly dynamical - eye movements being always launched first in the simplest, linear (vector sum) prediction. Tracking trajectories are then progressively adjusted to match the object trajectory after 200 ms of pursuit. Such strategy is immune to higher factors such as prediction about incoming 2D target trajectory. On the contrary, mixing retinal and extra-retinal signals become important later during the pursuit to accommodate partial or total object motion occlusion for instance. We propose a framework computing and representing object motion through two recurrent loops (V1-MT and MST-FEF, respectively), with area MST playing the role of gear. Such architecture would accommodate two important constraints of motor behavior: quick reaction to a new visual event and utilization of extra-retinal information to smooth out transient changes in the image such as those occurring when an object moves in a crowded environment.


Smooth Pursuit Pursuit Initiation Motion Integration Extraretinal Signal Aperture Problem 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.



GM is supported by the CNRS, the Agence Nationale de la Recherche and the European Union (FACETS, FP6-2004-IST-FETPI-15879). AM is supported by a Marie Curie Intra-European Fellowship (GEMME, IEF-025213). We thank Lee Stone, Pascal Mamassian and Philippe Lefèvre for fruitful discussions about the ideas presented in the Chapter.


  1. Adelson EH, Movshon JA (1982) Phenomenal coherence of moving visual patterns. Nature 300:523–525PubMedCrossRefGoogle Scholar
  2. Albright TD (1984) Direction and orientation selectivity of neurons in area MT of the macaque. J Neurophysiol 52:1106–1130PubMedGoogle Scholar
  3. Albright TD, Desimone R (1987) Local precision of visuotopic organization in the middle temporal area (MT) of macaque. Exp Brain Res 65:582–592PubMedCrossRefGoogle Scholar
  4. Anderson B, Moore J (1979) Optimal filtering. Prentice Hall, Englewood Cliffs, NJGoogle Scholar
  5. Barnes G (1993) Visual-vestibular interactions in the control of head and eye movement: the role of visual feedback and predictive mechanism. Prog Neurobiol 41:435–472PubMedCrossRefGoogle Scholar
  6. Barnes G, Schmidt AM (2002) Sequence learning in human ocular ocular smooth pursuit. Exp Brain Res 144:322–335PubMedCrossRefGoogle Scholar
  7. Barthélemy FV, Perrinet LU, Castet E, Masson GS (2008) Dynamics of distributed 1D and 2D motion representations for short-latency ocular following. Vis Res 48:501–522PubMedCrossRefGoogle Scholar
  8. Bayerl P, Neumann H (2004) Disambiguating visual motion through contextual feedback modulation. Neural Comput 16:2041–2066PubMedCrossRefGoogle Scholar
  9. Bayerl P, Neumann H (2007) A fast biologically inspired algorithm for recurrent motion estimation. IEEE Trans Pattern Anal Mach Intell 29:246–260PubMedCrossRefGoogle Scholar
  10. Becker W, Fuchs AF (1985) Prediction in the oculomotor system: smooth pursuit during transient disappearance of a visual target. Exp Brain Res 57:562–575PubMedCrossRefGoogle Scholar
  11. Berzhanskaya J, Grossberg S, Mingolla E (2007) Laminar cortical dynamics of visual form and motion interactions during coherent object motion perception. Spatial Vis 20:337–395CrossRefGoogle Scholar
  12. Beutter BR, Stone LS (2000) Motion coherence affects human perception and pursuit similarly. Vis Neurosci 17:139–153PubMedCrossRefGoogle Scholar
  13. Born RT, Pack CC, Zhao R (2002) Integration of motion cues for the initiation of smooth pursuit eye movements. Prog Brain Res 140:225–237PubMedCrossRefGoogle Scholar
  14. Born RT, Pack CC, Ponce CR, Yi S (2006) Temporal evolution of 2-dimensional direction signals used to guide eye movements. J Neurophysiol 95:284–300PubMedGoogle Scholar
  15. Boussaoud D, Ungerleider LG, Desimone R (1990) Pathways for motion analysis: cortical connections of the medial superior temporal and fundus of the superior temporal visual areas in the macaque. J Comp Neurol 15:462–495CrossRefGoogle Scholar
  16. Braddick OJ (1993) Segmentation versus integration in visual motion processing. Trends Neurosci 16:263–268PubMedCrossRefGoogle Scholar
  17. Bullier J (2001) Integrated model of visual processing. Brain Res Rev 36:96–107PubMedCrossRefGoogle Scholar
  18. Busettini C, Masson GS, Miles FA (1996) A role for binocular stereo cues in the rapid visual stabilization of the eyes. Nature 380:342–345PubMedCrossRefGoogle Scholar
  19. Carl J, Gellman RS (1987) Human smooth pursuit: stimulus dependant responses. J Neurophysiol 57:1446–1463PubMedGoogle Scholar
  20. Castet E, Wuerger S (1997) Perception of moving lines: interactions between local perpendicular signals and 2D motion signals. Vis Res 37:705–720PubMedCrossRefGoogle Scholar
  21. Castet E, Lorenceau J, Shiffrar M, Bonnet M (1993) Perceived speed of moving lines depends on orientation, lenght, speed and luminance. Vis Res 33:1921–1936PubMedCrossRefGoogle Scholar
  22. Castet E, Charton V, Dufour A (1999) The extrinsic/intrinsic classification of two dimensional motion signals with barber-pole stimuli. Vis Res 39:915–932PubMedCrossRefGoogle Scholar
  23. Chang T, Grossberg S, Mingolla E (1998) Neural dynamics of motion integration and speed discrimination. Vis Res 38:2769–2786CrossRefGoogle Scholar
  24. Churchland MM, Chou I-H, Lisberger SG (2003) Evidence for object permanence in the smooth-pursuit eye movements of monkeys. J Neurophysiol 90:2205–2218PubMedCrossRefGoogle Scholar
  25. de Hemptinne C, Lefèvre P, Missal M (2006) Influence of cognitive expectation on the initiation of anticipatory and visual pursuit eye movements in the rhesus monkey. J Neurophysiol 95:3770–3782PubMedCrossRefGoogle Scholar
  26. Denève S (2004). Bayesian inference with recurrent spiking networks. In: Lawrence KS, Weiss Y, Bottou L (eds) Advances in neural information processing system. MIT Press, Cambridge, MA, pp 353–360Google Scholar
  27. Denève S, Duhamel J-R, Pouget A (2007) Optimal sensorimotor integration in recurrent cortical networks: A neural implementation of Kalman filters. J Neurosci 27:5744–5756PubMedCrossRefGoogle Scholar
  28. Dow BM, Snyder AZ, Vautin RG, Bauer R (1981) Magnification factor and receptive field size in foveal striate cortex. Exp Brain Res 44:213–228PubMedCrossRefGoogle Scholar
  29. Fennema CL, Thompson WB (1979) Velocity determination in scenes containing several moving images. Comput Graph Image Process 9:301–315CrossRefGoogle Scholar
  30. Freyberg S, Ilg U (2008) Anticipatory smooth-pursuit eye movements in man and monkey. Exp Brain Res 186(2):203–214Google Scholar
  31. Fukushima K, Yamatobe T, Shinmei Y, Fukushima J (2002) Predictive responses of periarcuate pursuit neurons to visual target motion. Exp Brain Res 145:104–120PubMedCrossRefGoogle Scholar
  32. Hafed ZM, Krauzlis RJ (2006) Ongoing eye movements contrain visual perception. Nat Neurosci 9:1449–1457PubMedCrossRefGoogle Scholar
  33. Hawken MJ, Gegenfurtner KR (2001) Pursuit eye movements to second-order motion targets. J Opt Soc Am A 18:2282–2296CrossRefGoogle Scholar
  34. Heinen SJ, Watamaniuk SN (1998) Spatial integration is human smooth pursuit. Vis Res 38:3785–3794PubMedCrossRefGoogle Scholar
  35. Heinen SJ, Badler TB, Ting W (2005) Timing and velocity randomization similarly affect anticipatory pursuit. J Vis 5:493–503PubMedCrossRefGoogle Scholar
  36. Hubel D, Wiesel TN (1965) Receptive fields and functional architecture in two striate visual areas (18 & 19) of the cat. J Neurophysiol 28:229–289PubMedGoogle Scholar
  37. Hubel DH, Wiesel TN (1974) Uniformity of monkey striate cortex: a parallel relationship between field size, scatter and magnification factors. J Comp Neurol 158:295–305PubMedCrossRefGoogle Scholar
  38. Hueng X, Albright TD, Stoner GR (2007) Adaptive surround modulation in cortical area MT. Neuron 53:761–770CrossRefGoogle Scholar
  39. Ilg UJ (2002) Smooth pursuit eye movements: from low-level to high-level vision. Prog Brain Res 140:279–298PubMedCrossRefGoogle Scholar
  40. Ilg UJ (2003) Visual-tracking neurons in area MST are activated during anticipatory pursuit eye movements. NeuroReport 14:2219–2223PubMedCrossRefGoogle Scholar
  41. Ilg UJ, Thier P (1999) Eye movements of rhesus monkeys directed towards imaginary targets. Vis Res 39:2143–2150PubMedCrossRefGoogle Scholar
  42. Ilg UJ, Thier P (2003) Visual tracking neurons in primate area MST are activated by smooth-pursuit eye movements of an “imaginary” target. J Neurophysiol 90:1489–1502PubMedCrossRefGoogle Scholar
  43. Ilg UJ, Schumann S, Thier P (2004) Posterior parietal cortex neurons encode target motion in world-centered coordinates. Neuron 43:145–151PubMedCrossRefGoogle Scholar
  44. Kalman RE (1960) A new approach to linear filtering and prediction theory. Trans. ASME J. Basic Eng. 82:35–35Google Scholar
  45. Kawano K (1999) Ocular tracking: behavior and neurophysiology. Curr. Opin. Neurobiol. 9(4): 467–473Google Scholar
  46. Kettner RE, Leung H-C, Petersen BW (1996) Predictive smooth pursuit and complex two-dimensional trajectories in monkey: component interactions. Exp Brain Res 108:221–235PubMedCrossRefGoogle Scholar
  47. Kodaka Y, Kawano K (2003) Preparatory modulation of the gain of visuo-motor transmission for smooth pursuit in monkeys. Exp Brain Res 149:391–394PubMedGoogle Scholar
  48. Koechlin E, Anton J-L, Burnod Y (1999) Bayesian inference in populations of cortical neurons: a model of motion integration and segmentation. Biol Cybern 80:25–44PubMedCrossRefGoogle Scholar
  49. Kowler E, Steinman R (1979a) The effect of expectations on slow oculomotor control II. Single target displacements. Vis Res 19:619–632PubMedCrossRefGoogle Scholar
  50. Kowler E, Steinman R (1979b) The effect of expectations on slow oculomotor control II. Single target displacements. Vis Res 19:633–646PubMedCrossRefGoogle Scholar
  51. Kowler E, Martins AJ, Pavel M (1985) The effect of expectations on slow oculomotor control IV. Anticipatory smooth eye movements depend on prior target motions. Vis Res 24:197–210CrossRefGoogle Scholar
  52. Krauzlis RJ (2004) Recasting the smooth pursuit eye movement system. J Neurophysiol 91:591–603PubMedCrossRefGoogle Scholar
  53. Krauzlis RJ, Adler SA (2001) Effects of directional expectations on motion perception and pursuit eye movements. Vis Neurosci 18:365–376PubMedCrossRefGoogle Scholar
  54. Krauzlis RJ, Stone LS (1999) Tracking with the mind’s eye. Trends Neurosci 22:544–550PubMedCrossRefGoogle Scholar
  55. Lennie P, Movshon JA (2005) Coding of color and form in the geniculostriate visual pathway. J Opt Soc of Am A 22:2013–2033CrossRefGoogle Scholar
  56. Lindner A, Ilg U (2000) Initiation of smooth –pursuit eye movements to first-order and second-order motion stimuli. Exp Brain Res 133:450–456PubMedCrossRefGoogle Scholar
  57. Lisberger SG, Movshon JA (1999) Visual motion analysis for pursuit eye movements in area MT of macaque monkeys. J Neurosci 19:2224–2246PubMedGoogle Scholar
  58. Lisberger SG, Morris EJ, Tyschen L (1987) Visual motion processing and sensory-motor integration for smooth pursuit eye movements. Annu Review Neurosci 10:97–129CrossRefGoogle Scholar
  59. Löffler G, Orbach HS (1999) Computing feature motion without feature detectors: a model for terminator motion without end-stopped cells. Vis Res 39:859–871PubMedCrossRefGoogle Scholar
  60. Lorenceau J, Alais D (2001) Form constraints in motion binding. Nat Neurosci 4:745–751PubMedCrossRefGoogle Scholar
  61. Lorenceau J, Shiffrar M, Wells N, Castet E (1993) Different motion sensitive units are involved in recovering the direction of moving lines. Vis Res 33:1207–1217PubMedCrossRefGoogle Scholar
  62. MacAvoy MG, Gottlieb JP, Bruce CJ (1991) Smooth-pursuit eye movement representation in the primate frontal eye field. Cereb Cortex 1:95–102PubMedCrossRefGoogle Scholar
  63. Madelain L, Krauzlis RJ (2003) Effect of learning on smooth pursuit during transient disappearance of a visual target. J Neurophysiol 90:972–982PubMedCrossRefGoogle Scholar
  64. Madelain L, Krauzlis RJ, Wallman J (2005) Spatial deployment of attention influences both saccadic and pursuit tracking. Vis Res 45:2685–2703PubMedCrossRefGoogle Scholar
  65. Majaj NJ, Carandini M, Movshon JA (2007) Motion integration by neurons in macaque MT is local, not global. J Neurosci 27:366–370PubMedCrossRefGoogle Scholar
  66. Masson GS (2004) From 1D to 2D via 3D: surface motion segmentation for gaze stabilisation in primates. J Physiol (Paris) 98:35–52CrossRefGoogle Scholar
  67. Masson GS, Castet E (2002) Parallel motion processing for the initiation of short-latency ocular following in humans. J Neurosci 22(12):5149–5163Google Scholar
  68. Masson GS, Stone LS (2002) From following edges to pursuing objects. J Neurophysiol 88:2869–2873PubMedCrossRefGoogle Scholar
  69. Masson GS, Rybarczyk Y, Castet E, Mestre DR (2000) Temporal dynamics of motion integration for the initiation of tracking eye movements at ultra-short latencies. Vis Neurosci 17:753–767PubMedCrossRefGoogle Scholar
  70. Masson GS, Busettini C, Yang D-S, Miles FA (2001) Short-latency ocular following in humans: sensitivity to binocular disparity. Vis Res 41:3371–3387PubMedCrossRefGoogle Scholar
  71. Masson GS, Yang D-S, Miles FA (2002) Reversed short-latency ocular following. Vis Res 42:2081–2087PubMedCrossRefGoogle Scholar
  72. Menz MD, Freeman RD (2003) Stereoscopic depth processing in the visual cortex: a coarse-to-fine mechanism. Nat Neurosci 6:59–65PubMedCrossRefGoogle Scholar
  73. Miles FA (1998) The neural processing of 3-D visual information: evidence from eye movements. Eur J Neurosci 10:811–822PubMedCrossRefGoogle Scholar
  74. Miles FA, Kawano K, Optican LM (1986) Short-latency ocular following responses of monkey I. Dependence on temporospatial properties of visual input. J Neurophysiol 56:1321–1354PubMedGoogle Scholar
  75. Mingolla E (2003) Neural models of motion integration and segmentation. Neural Network 16:939–945CrossRefGoogle Scholar
  76. Missal M, Heinen SJ (2004) Supplementary eye fields stimulation facilitates anticipatory pursuit. J Neurophysiol 92:1257–1262PubMedCrossRefGoogle Scholar
  77. Montagnini A, Spering M, Masson GS (2006) Predicting 2D target velocity cannot help 2D motion integration for smooth pursuit initiation. J Neurophysiol 96:3545–3550PubMedCrossRefGoogle Scholar
  78. Montagnini A, Mamassian P, Perrinet L, Castet E, Masson GS (2007) Bayesian modeling of dynamic motion integration. J Physiol (Paris) 101:64–77CrossRefGoogle Scholar
  79. Movshon JA, Newsome WT (1996) Visual response properties of striate cortical neurons projecting to area MT in macaque monkeys. J Neurosci 16(23):7733–7741Google Scholar
  80. Newsome WT, Wurtz RH, Komatsu H (1988) Relation of cortical areas MT and MST to pursuit eye movements II. Differentiation of retinal and extraretinal inputs. J Neurophysiol 60:604–620PubMedGoogle Scholar
  81. Nowlan SJ, Sejnowski TJ (1995) A selection model for motion processing in area MT of primates. Journal of Neuroscience 15:1195–1214PubMedGoogle Scholar
  82. Osborne LC, Bialek W, Lisberger SG (2004) Time course of information about motion direction in visual area MT. J Neurosci 24:3210–3222PubMedCrossRefGoogle Scholar
  83. Osborne LC, Lisberger SG, Bialek W (2005) A sensory source for motor variation. Nature 437:412–416PubMedCrossRefGoogle Scholar
  84. Pack CC, Born RT (2001) Temporal dynamics of a neural solution to the aperture problem in visual area MT. Nature 409:1040–1042PubMedCrossRefGoogle Scholar
  85. Pack CC, Livingstone MS, Duffy KR, Born RT (2003) End-stopping and the aperture problem: two-dimensional motion signals in macaque V1. Neuron 39:671–680PubMedCrossRefGoogle Scholar
  86. Pack CC, Gartland AJ, Born RT (2004) Integration of contour and terminator signals in visual area MT of alert macaque. J Neurosci 31:3268–3280CrossRefGoogle Scholar
  87. Pack CC, Hunter JN, Born RJ (2005) Contrast dependence of suppressive influence in cortical area MT of alert macaque. J Neurophysiol 93:1809–1815PubMedCrossRefGoogle Scholar
  88. Pola J, Wyatt HJ (1985) Active and passive smooth eye movements: effects of stimulus size and location. Vis Res 25:1063–1076PubMedCrossRefGoogle Scholar
  89. Priebe NJ, Lisberger SG (2004) Estimating target speed from the population response in visual area MT. J Neurosci 24:1907–1916PubMedCrossRefGoogle Scholar
  90. Priebe NJ, Churchland MM, Lisberger SG (2001) Reconstruction of target speed for the guidance of eye movements. J Neurosci 21:3196–3206PubMedGoogle Scholar
  91. Rao P (2004) Bayesian computation in recurrent neural circuits. Neural Comput 16:1–38PubMedCrossRefGoogle Scholar
  92. Rashbass C (1961) The relationship between saccadic and tracking eye movements. J Physiol (London) 159:326–338Google Scholar
  93. Ringach DL, Hawken MJ, Shapley R (1997) Dynamics of orientation tuning in macaque V1. Nature 387:281–284PubMedCrossRefGoogle Scholar
  94. Robinson DA (1986) The systems approach to the oculomotor system. Vis Res 26:91–99PubMedCrossRefGoogle Scholar
  95. Sceniak MP, Ringach DL, Hawken MJ, Shapley R (1999) Contrast’s effect on spatial summation in macaque V1 neurons. Nat Neurosci 2:733–739PubMedCrossRefGoogle Scholar
  96. Sheliga B, Chen KJ, Fitgibbon EJ, Miles FA (2005) Initial ocular following in humans: a response to first-order motion energy. Vis Res 45:3307–3321PubMedCrossRefGoogle Scholar
  97. Sillito AM, Cudeiro J, Jones HE (2006) Always returning: feedback and sensory processing in visual cortex and thalamus. Trends Neurosci 29:307–316PubMedCrossRefGoogle Scholar
  98. Simoncelli EP, Adelson EH, Heeger DJ (1991) Probability distributions of optical flow. In: Proceedings of the Conference on Computer Vision and Pattern Recognition, pp 310–315Google Scholar
  99. Smith MA, Majaj NJ, Movshon JA (2005) Dynamics of motion signaling by neurons in macaque monkeys. Nat Neurosci 8:220–228PubMedCrossRefGoogle Scholar
  100. Steinbach LM (1976) Pursuing the perceptual rather than the retinal stimulus. Science 16:1371–1376Google Scholar
  101. Steinman RM (1986) The need for an eclectic, rather than systems, approach to the study of the primate oculomotor system. Vis Res 26:101–112PubMedCrossRefGoogle Scholar
  102. Stocker AA, Simoncelli EP (2006) Noise characteristics and prior expectation in human visual speed perception. Nat Neurosci 9(4):578–585 Google Scholar
  103. Stone LS, Beutter BR, Lorenceau J (2000) Visual motion integration for perception and pursuit. Perception 29:771–787PubMedCrossRefGoogle Scholar
  104. Tabata H, Miura K, Kawano K (2005) Anticipatory gain modulation in preparation for smooth pursuit eye movements. J Cogn Neurosci 17:1962–1968PubMedCrossRefGoogle Scholar
  105. Tabata H, Miura K, Taki K, Matuura K, Kawano K (2006) Preparatory gain modulation of visuomotor transmission for smooth pursuit eye movements in moneys. J Neurophysiol 96:3051–3063PubMedCrossRefGoogle Scholar
  106. Thier P, Erickson RG (1992) Response to visual-tracking neurons from cortical area MST-l to visual, eye and head motion. Eur J Neurosci 4(6):539–553Google Scholar
  107. Tlapale E, Masson GS, Viéville T, Kornprobst P (2007). Model of motion field diffusion controlled by form cues. In: Perception. 26th ECVP Abstracts, 36; Supplement, 215Google Scholar
  108. Vercher JL, Quaccia D, Gauthier GM (1995) Oculo-manual coordination control: respective role of visual and non-visual information in ocular tracking of self-moved targets. Exp Brain Res 103:311–322PubMedCrossRefGoogle Scholar
  109. Verghese P, Stone LS (1995) Combining speed information across space. Vis Res 35:2811–2823PubMedCrossRefGoogle Scholar
  110. Wallace J, Stone LS, Masson GS (2005) Object motion computation for the initiation of smooth pursuit eye movements in humans. J Neurophysiol 93:2279–2293PubMedCrossRefGoogle Scholar
  111. Wallach H (1935). Uber Visuell Wahrgenommene Bewegungrichtung. Psychologishe Forschung 20:325–380. Translated by Wuerger S, Shapley B (1996) Perception 11:1317–1367Google Scholar
  112. Weiss Y, Fleet D (2002). Velocity likelihood in biological and machine vision. In: Rao RPN, Olshausen BA, Lewicki MS (eds) Probabilistic models of the brain: perception and neural function. MIT Press, Cambridge, MA, pp 77–96Google Scholar
  113. Weiss Y, Simoncelli EP, Adelson EH (2002) Motion illusions as optimal percepts. Nat Neurosci 5:598–604PubMedCrossRefGoogle Scholar
  114. Wilmer JB, Nakayama K (2007) Two distinct visual motion mechanisms for smooth pursuit: evidence from individual differences. Neuron 54:987–1000PubMedCrossRefGoogle Scholar
  115. Wilson HR, Ferrera VP, Yo C (1992) A psychophysically motivated model for two-dimensional motion perception. Vis Neurosci 9:79–97PubMedCrossRefGoogle Scholar
  116. Wyatt HJ, Pola J, Fortune B, Posner M (1994) Smooth pursuit eye movements with imaginary targets defined by extrafoveal cues. Vis Res 34:803–820PubMedCrossRefGoogle Scholar
  117. Yo C, Wilson HR (1992) Perceived direction of moving two-dimensional patterns that depends on duration, contrast and eccentricity. Vis Res 32:135–147PubMedCrossRefGoogle Scholar
  118. Zeki S, Shipp S (1988) The functional logic of cortical connections. Nature 335:311–317CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2009

Authors and Affiliations

  • Guillaume S. Masson
    • 1
  • Anna Montagnini
  • Uwe J. Ilg
  1. 1.Institut de Neurosciences Cogntiives de la MéditerranéeCNRS and Université de la MéditerranéeMarseilleFrance

Personalised recommendations