Abstract
We have previously tried to explain perceptual inference and learning under a free-energy principle that pursues Helmholtz’s agenda to understand the brain in terms of energy minimization. It is fairly easy to show that making inferences about the causes of sensory data can be cast as the minimization of a free-energy bound on the likelihood of sensory inputs, given an internal model of how they were caused. In this article, we consider what would happen if the data themselves were sampled to minimize this bound. It transpires that the ensuing active sampling or inference is mandated by ergodic arguments based on the very existence of adaptive agents. Furthermore, it accounts for many aspects of motor behavior; from retinal stabilization to goal-seeking. In particular, it suggests that motor control can be understood as fulfilling prior expectations about proprioceptive sensations. This formulation can explain why adaptive behavior emerges in biological agents and suggests a simple alternative to optimal control theory. We illustrate these points using simulations of oculomotor control and then apply to same principles to cued and goal-directed movements. In short, the free-energy formulation may provide an alternative perspective on the motor control that places it in an intimate relationship with perception.
Abbreviations
- \({{\bf \Psi}\supseteq \{ \tilde{\bf{x}},{\tilde {\bf v}},{\boldsymbol \theta},{\boldsymbol \gamma}\}}, {\Psi \supseteq \{\tilde {x},\tilde {v},\theta ,\gamma \}}\) :
-
Unknown causes of sensory input; variables in bold denote true values and those in italics denote variables assumed by the agent or model
- \({\tilde{x}(t) = [x, {x}', {x}'',\ldots]^T, \dot{\tilde{x}}(t) = f(\tilde{x}, \tilde{v}, \theta)+\tilde{w}}\) :
-
Generalised hidden-states that act on an agent. These are time-varying quantities that include all high-order temporal derivatives; they represent a point in generalised coordinates of motion that encodes a path or trajectory
- \({\tilde{v}(t) = [v, {v}', {v}'',\ldots]^T}\) :
-
Generalised forces or causal states that act on hidden states
- \({\tilde{s}(t) = g(\tilde{x}, \tilde{v}, \theta) + \tilde{z}}\) :
-
Generalised sensory states caused by hidden states
- \({\theta \supseteq \{\theta_1 ,\theta_2, \ldots\}}\) :
-
Parameters of the equations of motion and sensory mapping
- \({\gamma \supseteq \{\gamma^s, \gamma^x, \gamma^v\}}\) :
-
Parameters of the precision of random fluctuations \({\Pi(\gamma^i) : i \in s, x, v}\)
- \({\tilde{w}(t) = [w, {w}', {w}'', \ldots]^T}\) :
-
Generalised random fluctuations of the motion of hidden states
- \({\tilde{z}(t) = [z, {z}', {z}'', \ldots]^T}\) :
-
Generalised random fluctuations of sensory states
- \({\tilde{n}(t) = [n, {n}', {n}'', \ldots]^T}\) :
-
Generalised random fluctuations of causal states
- \({\Pi^i := \Pi(\gamma^i) = \Sigma(\gamma^i)^{-1}}\) :
-
Precisions or inverse covariances of generalised random fluctuations
- \({{\bf g}(\tilde{\bf{x}}, {\tilde{\bf v}}, {\bf \theta})}\) , \({{\bf f}(\tilde{\bf{x}}, {\tilde{\bf v}}, \tilde{a}, {\bf \theta})}\):
-
Sensory mapping and equations of motion generating sensory states
- \({g(\tilde{x}, \tilde{v}, \theta)}\) , \({f(\tilde{x}, \tilde{v}, \theta)}\):
-
Sensory mapping and equations of motion modeling sensory states
- a(t):
-
Policy: a scalar function of generalised sensory and internal states
- \({p(\tilde{\bf{x}}|m)}\) , \({p(\tilde{s}|m)}\):
-
Ensemble densities; the density of the hidden and sensory states of agents at equilibrium with their environment.
- \({D(q\vert \vert p) = \left\langle{{\rm ln}(q/p)}\right\rangle_q}\) :
-
Kullback-Leibler divergence or cross-entropy between two densities
- \({\langle \rangle_q }\) :
-
Expectation or mean of under the density q
- m :
-
Model or agent; entailing the form of a generative model
- \({H(X) = \left\langle {\ln p(\tilde{\bf{x}}\vert m)}\right\rangle_p H(S)=\left\langle {\ln p(\tilde {s}\vert m)}\right\rangle_p}\) :
-
Entropy of generalised hidden and sensory states
- \({}{-\ln p(\tilde{s}\vert m)}\) :
-
Surprise or self-information of generalised sensory states
- \({F(\tilde{s},\mu ) \ge -\ln p(\tilde{s}\vert m)}\) :
-
Free-energy bound on surprise
- q(Ψ|μ):
-
Recognition density on causes Ψ with sufficient statistics μ
- \({\mu =\{\tilde {\mu}(t),\mu _\theta ,\mu _\gamma \}} \tilde {\mu}=\{\tilde {\mu}_x ,\tilde {\mu}_v \}\) :
-
Conditional or posterior expectation of the causes Ψ; these are the sufficient statistics of the Gaussian recognition density
- \({\tilde{\eta}(t) = [\eta, {\eta}', {\eta}'', \ldots ]^T}\) :
-
Prior expectation of generalised causal states
- \({\xi_i = \Pi_i\tilde{\varepsilon}_i : i \in s, x, v}\) :
-
Precision-weighted generalised prediction errors
- \({{\tilde {\varepsilon}} = \left[ \begin{array}{l} \tilde {\varepsilon}_s =\tilde {s}-g(\mu)\\ \tilde {\varepsilon}_x =D\tilde {\mu}_x -f(\mu)\\ \tilde{\varepsilon}_v =\tilde {\mu}_v -\tilde {\eta}\end{array} \right]}\) :
-
Generalised prediction error on sensory states, the motion of hidden states and forces or causal states.
References
Abbott LF, Varela JA, Sen K, Nelson SB (1997) Synaptic depression and cortical gain control. Science 275(5297): 220–224
Andersen RA (1989) Visual and eye movement factions of the posterior parietal cortex. Annu Rev Neurosci 12: 377–405
Anosov DV (2001) Ergodic theory. In Hazewinkel M (ed) Encyclopaedia of mathematics. Kluwer Academic Publishers. ISBN 978-1556080104
Ballard DH, Hinton GE, Sejnowski TJ (1983) Parallel visual computation. Nature 306: 21–26
Barlow HB (1969) Pattern recognition and the responses of sensory neurons. Ann N Y Acad Sci 156: 872–881
Bays PM, Wolpert DM (2007) Computational principles of sensorimotor control that minimize uncertainty and variability. J Physiol 578(Pt 2): 387–396
Bell CC, Grant K (1989) Corollary discharge inhibition and preservation of temporal information in a sensory nucleus of mormyrid electric fish. J Neurosci 9(3): 1029–1044
Bellman R (1952) On the theory of dynamic programming. Proc Natl Acad Sci USA 38: 716–719
Bernard C (1974) Lectures on the phenomena common to animals and plants (trans: Hoff HE, Guillemin R, Guillemin L). Charles C Thomas, Springfield. ISBN 978-0398028572
Berret B, Darlot C, Jean F, Pozzo T, Papaxanthis C, Gauthier JP (2008) The inactivation principle: mathematical solutions minimizing the absolute work and biological implications for the planning of arm movements. PLoS Comput Biol 4(10): e1000194
Bruyn JL, Mason AH (2009) Temporal coordination during bimanual reach-to-grasp movements: the role of vision. Q J Exp Psychol (Colchester) 5: 1–15
Bütefisch CM, Davis BC, Sawaki L, Waldvogel D, Classen J, Kopylev L, Cohen LG (2002) Modulation of use-dependent plasticity by d-amphetamine. Ann Neurol 51(1): 59–68
Camerer CF (2003) Behavioural studies of strategic thinking in games. Trends Cogn Sci 7(5): 225–231
Crauel H, Flandoli F (1994) Attractor for random dynamical systems. Probab Theory Relat Fields 100: 365–393
Daw ND, Doya K (2006) The computational neurobiology of learning and reward. Curr Opin Neurobiol 16(2): 199–204
Dayan P, Hinton GE, Neal RM (1995) The Helmholtz machine. Neural Comput 7: 889–904
Deneve S (2008) Bayesian spiking neurons I: inference. Neural Comput 20(1): 91–117
Diedrichsen J, Dowling N (2009) Bimanual coordination as task-dependent linear control policies. Hum Mov Sci [Epub ahead of print]
Diedrichsen J, Verstynen T, Hon A, Zhang Y, Ivry RB (2007) Illusions of force perception: the role of sensori-motor predictions, visual information, and motor errors. J Neurophysiol 97(5): 3305– 3313
Disney AA, Aoki C, Hawken MJ (2007) Gain modulation by nicotine in macaque v1. Neuron 56(4): 701–713
Doya K (2002) Metalearning and neuromodulation. Neural Netw 15(4–6): 495–506
Evans DJ (2003) A non-equilibrium free-energy theorem for deterministic systems. Mol Phys 101: 1551–1554
Feldman AG (2009) New insights into action-perception coupling. Exp Brain Res 194(1): 39–58
Feldman AG, Levin MF (1995) The origin and use of positional frames of reference in motor control. Behav Brain Sci 18: 723–806
Feynman RP (1972) Statistical mechanics. Benjamin, Reading
Fourneret P, Jeannerod M (1998) Limited conscious monitoring of motor performance in normal subjects. Neuropsychologia 36(11): 1133–1140
Friston K (2005) A theory of cortical responses. Philos Trans R Soc Lond B Biol Sci 360(1456): 815–836
Friston K (2008) Hierarchical models in the brain. PLoS Comput Biol 4(11):e1000211. PMID: 18989391
Friston K (2010) The free-energy principle: a unified brain theory. Nat Rev Neurosci 11(2): 127–138
Friston K, Stephan KE (2007) Free-energy and the brain. Synthese 159: 417–458
Friston KJ, Tononi G, Reeke GN Jr, Sporns O, Edelman GM (1994) Value-dependent selection in the brain: simulation in a synthetic neural model. Neuroscience 59(2): 229–243
Friston K, Kilner J, Harrison L (2006) A free-energy principle for the brain. J Physiol (Paris) 100(1–3): 70–87
Friston KJ, Trujillo-Barreto N, Daunizeau J (2008) DEM: a variational treatment of dynamic systems. NeuroImage 41(3): 849–885
Friston KJ, Daunizeau J, Kiebel SJ (2009) Reinforcement learning or active inference?. PLoS One 4(7): e6421
Gontar V (2000) Entropy principle of extremality as a driving force in the discrete dynamics of complex and living systems. Chaos Solitons Fractals 11: 231–236
Gottlieb GL (1998) Rejecting the equilibrium-point hypothesis. Motor Control 2(1): 10–12
Grafton ST, Hamilton AF (2007) Evidence for a distributed hierarchy of action representation in the brain. Hum Mov Sci 26(4): 590–616
Grafton ST, Schmitt P, Van Horn J, Diedrichsen J (2008) Neural substrates of visuomotor learning based on improved feedback control and prediction. Neuroimage 39(3): 1383–1395
Helmholtz H (1860/1962). Handbuch der physiologischen optik, vol 3 (English trans: Southall JPC, ed). Dover, New York
Hinton GE, von Camp D (1993) Keeping neural networks simple by minimizing the description length of weights. In: Proceedings of COLT-93, pp 5–13
Huffman KJ, Krubitzer L (2001) Area 3a: topographic organization and cortical connections in marmoset monkeys. Cereb Cortex 11: 849–867
Jax SA, Rosenbaum DA (2007) Hand path priming in manual obstacle avoidance: evidence that the dorsal stream does not only control visually guided actions in real time. J Exp Psychol Hum Percept Perform 33(2): 425–441
Jones DS (1979) Elementary information theory. Clarendon Press, New York
Kawato M, Hayakawa H, Inui T (1993) A forward-inverse optics model of reciprocal connections between visual cortical areas. Network 4: 415–422
Kersten D, Mamassian P, Yuille A (2004) Object perception as Bayesian inference. Annu Rev Psychol 55: 271–304
Kiebel SJ, Daunizeau J, Friston KJ (2008) A hierarchy of time-scales and the brain. PLoS Comput Biol 4(11):e1000209. PMID: 19008936
Knill DC, Pouget A (2004) The Bayesian brain: the role of uncertainty in neural coding and computation. Trends Neurosci 27(12): 712–719
Körding KP, Wolpert DM (2004) Bayesian integration in sensorimotor learning. Nature 427(6971): 244–247
Körding KP, Beierholm U, Ma WJ, Quartz S, Tenenbaum JB, Shams L (2007) Causal inference in multisensory perception. PLoS ONE 2(9): e943
Kreisel SH, Hennerici MG, Bäzner H (2007) Pathophysiology of stroke rehabilitation: the natural course of clinical recovery, use-dependent plasticity and rehabilitative outcome. Cerebrovasc Dis 23(4): 243–255
Kulvicius T, Porr B, Wörgötter F (2007) Development of receptive fields in a closed-loop behavioral system. Neurocomputing 70: 2046–2049
Lee TS, Mumford D (2003) Hierarchical Bayesian inference in the visual cortex. J Opt Soc Am A Opt Image Sci Vis 20: 1434–1448
Linsker R (1990) Perceptual neural organization: some approaches based on network models and information theory. Annu Rev Neurosci 13: 257–281
Liu D, Todorov E (2007) Evidence for the flexible sensorimotor strategies predicted by optimal feedback control. J Neurosci 27(35): 9354–9368
Mackay DJC (1992) Information-based objective functions for active data selection. Neural Comput 4: 590–604
MacKay DJC (1995) Free-energy minimization algorithm for decoding and cryptoanalysis. Electron Lett 31: 445–447
Manoonpong P, Geng T, Kulvicius T, Porr B, Wörgötter F (2007) Adaptive, fast walking in a biped robot under neuronal control and learning. PLoS Comput Biol 3(7): e134
Maturana HR, Varela F (1972) De máquinas y seres vivos. Editorial Universitaria, Santiago. English version: “Autopoiesis: the organization of the living,” in Maturana HR, Varela FG (1980) Autopoiesis and cognition. Reidel, Dordrecht
Montague PR, Dayan P, Person C, Sejnowski TJ (1995) Bee foraging in uncertain environments using predictive Hebbian learning. Nature 377(6551): 725–728
Mumford D (1992) On the computational architecture of the neocortex. II. The role of cortico-cortical loops. Biol Cybern 66: 241–251
Mussa Ivaldi FA, Morasso P, Zaccaria R (1988) A distributed model for representing and regularizing motor redundancy. Biol Cybern 60(1): 1–16
Nakano E, Imamizu H, Osu R, Uno Y, Gomi H, Yoshioka T, Kawato M (1999) Quantitative examinations of internal representations for arm trajectory planning: minimum commanded torque change model. J Neurophysiol 81(5): 2140–2155
Neal RM, Hinton GE (1998) A view of the EM algorithm that justifies incremental, sparse, and other variants. In: Jordan MI (eds) Learning in graphical models. Kluwer Academic Publishers, Dordrecht, pp 355–368
Olshausen BA, Field DJ (1996) Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature 381: 607–609
O’Reilly RC, Noelle DC, Braver TS, Cohen JD (2002) Prefrontal cortex and dynamic categorization tasks: representational organization and neuromodulatory control. Cereb Cortex 12(3): 246–257
Paulignan Y, MacKenzie C, Marteniuk R, Jeannerod M (1991) Selective perturbation of visual input during prehension movements. 1. The effects of changing object position. Exp Brain Res 83(3): 502–512
Paulin MG (2005) Evolution of the cerebellum as a neuronal machine for Bayesian state estimation. J Neural Eng 2(3): S219–S234
Porr B, Wörgötter F (2003) Isotropic sequence order learning. Neural Comput 15(4): 831–864
Prinz AA (2006) Insights from models of rhythmic motor systems. Curr Opin Neurobiol 16(6): 615–620
Rao RP, Ballard DH (1998) Predictive coding in the visual cortex: a functional interpretation of some extra-classical receptive field effects. Nat Neurosci 2: 79–87
Redgrave P, Prescott TJ, Gurney K (1999) The basal ganglia: a vertebrate solution to the selection problem?. Neuroscience 89(4): 1009–1023
Rescorla RA, Wagner AR (1972) A theory of Pavlovian conditioning: variations in the effectiveness of reinforcement and nonreinforcement. In: Black AH, Prokasy WF (eds) Classical conditioning II: current research and theory. Appleton Century Crofts, New York, pp 64–99
Schaal S, Mohajerian P, Ijspeert A (2007) Dynamics systems vs. optimal control—a unifying view. Prog Brain Res 165: 425–445
Scheidt RA, Reinkensmeyer DJ, Conditt MA, Rymer WZ, Mussa-Ivaldi FA (2000) Persistence of motor adaptation during constrained, multi-joint, arm movements. J Neurophysiol 84: 853–862
Schultz W, Dayan P, Montague PR (1997) A neural substrate of prediction and reward. Science 275: 1593–1599
Schweitzer F (2003) Brownian agents and active particles: collective dynamics in the natural and social sciences. Springer Series in Synergetics, 1st ed 2003, 2nd printing 2007. ISBN: 978-3-540-73844-2
Shadmehr R, Krakauer JW (2008) A computational neuroanatomy for motor control. Exp Brain Res 185(3): 359–381
Sutton RS (1996) Generalization in reinforcement learning: successful examples using sparse coarse coding. Adv Neural Inf Process Syst 8: 1038–1044
Sutton RS, Barto AG (1981) Toward a modern theory of adaptive networks: expectation and prediction. Psychol Rev 88(2): 135– 170
Tatler BW, Wade NJ (2003) On nystagmus, saccades, and fixations. Perception 32(2): 167–184
Todorov E (2006) Linearly-solvable Markov decision problems. In: Scholkopf B et al (eds) Advances in neural information processing systems, vol 19. MIT Press, Cambridge, pp 1369–1376
Todorov E, Jordan MI (1998) Smoothness maximization along a predefined path accurately predicts the speed profiles of complex arm movements. J Neurophysiol 80(2): 696–714
Todorov E, Jordan MI (2002) Optimal feedback control as a theory of motor coordination. Nat Neurosci 5(11): 1226–1235
Toussaint M (2009) Probabilistic inference as a model of planned behavior. Künstliche Intelligenz Ger Artif Intell J (in press)
Tschacher W, Haken H (2007) Intentionality in non-equilibrium systems? The functional aspects of self-organized pattern formation. New Ideas Psychol 25: 1–15
Tseng YW, Diedrichsen J, Krakauer JW, Shadmehr R, Bastian AJ (2007) Sensory prediction errors drive cerebellum-dependent adaptation of reaching. J Neurophysiol 98(1): 54–62
Verschure PF, Voegtlin T (1998) A botom up approach towards the acquisition and expression of sequential representations applied to a behaving real-world device: Distributed Adaptive Control III. Neural Netw 11(7–8): 1531–1549
Verschure PF, Voegtlin T, Douglas RJ (2003) Environmentally mediated synergy between perception and behavior in mobile robots. Nature 425: 620–624
Voss M, Ingram JN, Wolpert DM, Haggard P (2008) Mere expectation to move causes attenuation of sensory signals. PLoS ONE 3(8): e2866
Watkins CJCH, Dayan P (1992) Q-learning. Mach Learn 8: 279–292
Wei K, Körding KP (2009) Relevance of error: what drives motor adaptation?. J Neurophysiol 101(2): 655–664
Wolpert DM, Miall RC (1996) Forward models for physiological motor control. Neural Netw 9(8): 1265–1279
Wolpert DM, Ghahramani Z, Jordan MI (1995) An internal model for sensorimotor integration. Science 269(5232): 1880–1882
Wörgötter F, Porr B (2005) Temporal sequence learning, prediction, and control: a review of different models and their relation to biological mechanisms. Neural Comput 17(2): 245–319
Yu AJ, Dayan P (2005) Uncertainty, neuromodulation and attention. Neuron 46: 681–692
Acknowledgements
We would like to thank Neil Burgess and Florentin Wörgötter for very helpful reviews of a previous version of this work.
Open Access
This article is distributed under the terms of the Creative Commons Attribution Noncommercial License which permits any noncommercial use, distribution, and reproduction in any medium, provided the original author(s) and source are credited.
Author information
Authors and Affiliations
Corresponding author
Additional information
The free-energy principle is an attempt to explain the structure and function of the brain, starting from the fact that we exist: This fact places constraints on our interactions with the world, which have been studied for years in evolutionary biology and systems theory. However, recent advances in statistical physics and machine learning point to a simple scheme that enables biological systems to comply with these constraints. If one looks at the brain as implementing this scheme (minimizing a free-energy bound on disorder), then many aspects of its anatomy and physiology start to make sense. In this article, we show that free-energy can be reduced by selectively sampling sensory inputs. This leads to adaptive responses and provides a new view of how movement control might work in the brain. The main conclusion is that we only need to have expectations about the sensory consequences of moving in order to elicit movement. This means we that can replace the notion of desired movements with expected movements and understand action in terms of perceptual expectations.
Rights and permissions
Open Access This is an open access article distributed under the terms of the Creative Commons Attribution Noncommercial License (https://creativecommons.org/licenses/by-nc/2.0), which permits any noncommercial use, distribution, and reproduction in any medium, provided the original author(s) and source are credited.
About this article
Cite this article
Friston, K.J., Daunizeau, J., Kilner, J. et al. Action and behavior: a free-energy formulation. Biol Cybern 102, 227–260 (2010). https://doi.org/10.1007/s00422-010-0364-z
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00422-010-0364-z
Keywords
- Computational
- Motor
- Control
- Bayesian
- Hierarchical
- Priors