Abstract
The present chapter discusses value-based habitual and goal-directed systems as studied in the animal and human learning literature. It focuses on the means by which these two systems might interact in knowledge transfer, particularly as it applies to social learning. Knowledge is conceived here in terms of types of logic computations as implemented by neural networks. A discussion of dual-process type structures in the brain is provided as well as neural-dynamic implementations thereof and considerations for how a perspective of the brain as carrying out logic computations might be useful for developing the general cognitive capacities of artificial agents.
Keywords
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsNotes
- 1.
Halford et al. [70] suggests that a transitive inference process that is not based on ‘omnidirectionality’, i.e. that ‘outputs’ of an otherwise feedforward process can be used to infer ‘inputs’ is only implicit and demonstrative of a lower order cognitive process.
- 2.
The first and second phase of the transfer of control paradigm can, in fact, be presented in any order though more standardly the initial instrumental phase is used first.
References
Seger, C.A.: How do the basal ganglia contribute to categorization? Their roles in generalization, response selection, and learning via feedback. Neurosci. Biobehav. Rev. 32, 265–278 (2008)
Phillips, S., Wilson, W.H., Halford, G.S.: What do transitive inference and class inclusion have in common? Categorical (co) products and cognitive development. PLoS Comput. Biol. 5(12) (2009)
Seger, C.A., Spiering, B.J.: A critical review of habit learning and the basal ganglia. Front. Syst. Neurosci. 5 (2011)
Davidson, R.J., Irwin, W.: The functional neuroanatomy of emotion and affective style. Trends Cognit. Sci. 3(1), 11–21 (1999)
Dickinson, A.: Actions and habits: the development of behavioural autonomy. Philos. Trans. R. Soc. Lond. B Biol. Sci. 308, 67–78 (1985)
Hirsh, J.: The hippocampus and contextual retrieval of information from memory: a theory. Behav. Biol. 12, 421–444 (1974)
Seger, C.A.: The involvement of corticostriatal loops in learning across tasks, species, and methodologies, In: Groenewegen, H.J., Voorn, P., Berendse, H.W., Mulder, A.B., Cools, A.R. (eds.) The Basal Ganglia IX, pp. 25–39. Springer, New York (2009)
Daw, N.D., O’Doherty, J.P.: Multiple systems for value learning. In: Neuroeconomics: Decision Making and the Brain, 2nd edn, pp. 393–410. Academic Press, London (2014)
Miller, K.J., Shenhav, A., Ludvig, E.A.: Habits without values. Psychol. Rev. (2019)
Lowe, R., Ziemke, T.: Exploring the relationship of reward and punishment in reinforcement learning. In: 2013 IEEE Symposium on Adaptive Dynamic Programming and Reinforcement Learning (ADPRL), pp. 140–147 (2013)
Lee, G., Lowe, R., Ziemke, T.: Modelling early infant walking: testing a generic CPG architecture on the NAO humanoid. In: 2011 IEEE International Conference on Development and Learning (ICDL), vol. 2, pp. 1–6 (2011)
Li, C., Lowe, R., Ziemke, T.: Crawling posture learning in humanoid robots using a natural-actor-critic CPG architecture. In: Artificial Life Conference Proceedings, 13, pp. 1182–1190. MIT Press, Cambridge, MA (2013)
Li, C., Lowe, R., Ziemke, T.: A novel approach to locomotion learning: Actor-Critic architecture using central pattern generators and dynamic motor primitives. Front. Neurorobotics 8, 23 (2014)
Rolls, E.T.: On the brain and emotion. Behav. Brain Sci. 23(2), 219–228 (2000)
Suri, R.E., Schultz, W.: Learning of sequential movements by neural network model with dopamine-like reinforcement signal. Exp. Brain Res. 121(3), 350–354 (1998)
Suri, R.E., Schultz, W.: Temporal difference model reproduces anticipatory neural activity. Neural Comput. 13(4), 841–862 (2001)
Peterson, G.B., Trapold, M.A.: Effects of altering outcome expectancies on pigeons’ delayed conditional discrimination performance. Learn. Motiv. 11, 267–288 (1980)
Daw, N.D., Niv, Y., Dayan, P.: Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nat. Neurosci. 8(12), 1704 (2005)
Sun, R.: Interpreting psychological notions: a dual-process computational theory. J. Appl. Res. Memory Cognit. 4(3), 191–196 (2015)
de Wit, S., Dickinson, A.: Associative theories of goal-directed behaviour: a case for animal–human translational models. Psychol. Res. PRPF 73(4), 463–476 (2009)
Mowrer, O.H.: On the dual nature of learning: A reinterpretation of ‘‘conditioning’’ and ‘‘problem-solving’’. Harv. Educ. Rev. 17, 102–148 (1947)
Trapold, M.A.: Are expectancies based upon different positive reinforcing events discriminably different? Learn. Motiv. 1, 129–140 (1970)
Cardinal, R.N., Parkinson, J.A., Hall, J., Everitt, B.J.: Emotion and motivation: the role of the amygdala, ventral striatum, and prefrontal cortex. Neurosci. Biobehav. Rev. 26(3), 321–352 (2002)
Urcuioli, P.: Behavioral and associative effects of differential outcomes in discriminating learning. Learn. Behav. 33(1), 1–21 (2005)
Lowe, R., Humphries, M., Ziemke, T.: The dual-route hypothesis: evaluating a neurocomputational model of fear conditioning in rats. Connect. Sci. 21(1), 15–37 (2009)
Lowe, R., Billing, E.: Affective-associative two-process theory: a neural network investigation of adaptive behaviour in differential outcomes training. Adapt. Behav. 25(1), 5–23 (2017)
Lowe, R., Almér, A., Billing, E., Sandamirskaya, Y., Balkenius, C.: Affective–associative two-process theory: a neurocomputational account of partial reinforcement extinction effects. Biol. Cybern. 111(5–6), 365–388 (2017)
Braver, T.S., et al.: Mechanisms of motivation–cognition interaction: challenges and opportunities. Cognit. Affect. Behav. Neurosci. 14(2), 443–472 (2014)
Urcuioli, P.: Some relationships between outcome expectancies and sample stimuli in pigeons’ delayed matching. Anim. Learn. Behav. 18(3), 302–314 (1990)
Urcuioli, P.: Stimulus control and stimulus class formation. In: Madden, G.J., Dube, W.V., Hackenberg, T.D., Hanley, G.P., Lattal, K.A. (eds.) APA Handbook of Behavior Analysis, vol. 1, pp. 361–386. American Psychological Association, Washington, DC (2013)
Amsel, A.: The role of frustrative nonreward in noncontinuous reward situations. Psychol. Bull. 55, 102–119 (1958)
Amsel, A.: Frustration Theory: An Analysis of Dispositional Learning and Memory. Cambridge University Press, Cambridge (1992)
Overmier, J.B., Lawry, J.A.: Pavlovian conditioning and the mediation of behavior. Psychol. Learn. Motiv. 13, 1–55 (1979)
McCulloch, W.S., Pitts, W.: A logical calculus of the ideas immanent in nervous activity. Bull. Math. Biophys. 5, 115–133 (1943)
Rosenblatt, F.: The perceptron: a probabilistic model for information storage and organization in the brain. Psychol. Rev. 65, 386 (1958)
Rumelhart, D.E., Hinton, G.E., McClelland, J.L.: A general framework for parallel distributed processing. In: Parallel Distributed Processing: Explorations in the Microstructure of Cognition, vol. 1, pp. 45–76 (1986)
Hornik, K., Stinchcombe, M., White, H.: Multilayer feedforward networks are universal approximators. Neural Netw. 2(5), 359–366 (1989)
Navarro-Guerrero, N., Lowe, R.J., Wermter, S.: The effects on adaptive behaviour of negatively valenced signals in reinforcement learning. In: 2017 Joint IEEE International Conference on Development and Learning and Epigenetic Robotics (ICDL-EpiRob), pp. 148–155 (2017)
Navarro-Guerrero, N., Lowe, R.J., Wermter, S.: Improving robot motor learning with negatively valenced reinforcement signals. Front. Neurorobotics 11, 10 (2017)
Wierman, M.J.: An Introduction to the Mathematics of Uncertainty, 149–150. Creighton University (2010)
Zadeh, L.A., et al.: Fuzzy Sets, Fuzzy Logic. World Scientific Press, Fuzzy Systems (1996)
Lange, S., Riedmiller, M., Voigtländer, A.: Autonomous reinforcement learning on raw visual input data in a real world application. In: The 2012 International Joint Conference on Neural Networks (IJCNN), pp. 1–8 (2012)
Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A.A., Veness, J., Bellemare, M.G., … Petersen, S.: Human-level control through deep reinforcement learning. Nature 518(7540), 529 (2015)
Yang, Z., Xie, Y., Wang, Z.: A theoretical analysis of deep Q-learning. arXiv preprint arXiv:1901.00137 (2019)
Lowe, R., Almer, A., Lindblad, G., Gander, P., Michael, J., Vesper, C.: Minimalist social-affective value for use in joint action: a neural-computational hypothesis. Front. Comput. Neurosci. 10 (2016)
Lowe, R., Almér, A., Gander, P., Balkenius, C.: Vicarious value learning and inference in human-human and human-robot interaction. In: 2019 8th International Conference on Affective Computing and Intelligent Interaction Workshops and Demos (ACIIW), pp. 395–400. IEEE (2019)
Bratman, M.E.: Shared cooperative activity. Philos. Rev. 101, 327–341 (1992)
Vesper, C., Butterfill, S., Knoblich, G., Sebanz, N.: A minimal architecture for joint action. Neural Netw. 23, 998–1003 (2010)
Kahneman, D., Tversky, A.: Rational choice and the framing of decisions. J. Bus. 59(4), 251–278 (1986)
Halford, G.S., Wilson, W.H., Phillips, S.: A conceptual complexity metric based on representational rank. Psychology Computer Science & Engineering, University of Queensland University of New South Wales, Queensland 4072, Australia; Sydney, NSW 2052 Australia (1999)
Halford, G.S., Wilson, W.H., Andrews, G., Phillips, S.: Categorizing Cognition: Toward Conceptual Coherence in the Foundations of Psychology. MIT Press (2014)
Halford, G.S., Wilson, W.H., Phillips, S.: Processing capacity defined by relational complexity: implications for comparative, developmental, and cognitive psychology. Behav. Brain Sci. 21(6), 803–831 (1998)
Halford, G.S., Wilson, W.H., Phillips, S.: Relational knowledge: the foundation of higher cognition. Trends Cognit. Sci. 14(11), 497–505 (2010)
Hummel, J.E., Holyoak, K.J.: Distributed representations of structure: a theory of analogical access and mapping. Psychol. Rev. 104(3), 427 (1997)
Hummel, J.E., Holyoak, K.J.: A process model of human transitive inference. In: Spatial Schemas in Abstract Thought, pp. 279–305 (2001)
Doumas, L.A., Morrison, R.G., Richland, L.E.: Individual differences in relational learning and analogical reasoning: a computational model of longitudinal change. Front. Psychol. 9 (2018)
Knowlton, B.J., Morrison, R.G., Hummel, J.E., Holyoak, K.J.: A neurocomputational system for relational reasoning. Trends Cognit. Sci. 16(7), 373–381 (2012)
Hummel, J. E., Holyoak, K. J., Green, C., Doumas, L. A., Devnich, D., Kittur, A., Kalar, D. J.: A solution to the binding problem for compositional connectionism. In: Levy, S.D., Gayler, R. (eds.) Compositional Connectionism in Cognitive Science: Papers from the AAAI Fall Symposium, pp. 31–34 (2004)
Lowe, R. Almér, A., Balkenius, C.: Bridging connectionism and relational cognition through bi-directional affective-associative processing. Open Inf. Sci. 3, 235–260 (2019)
Hinton, G.E., Salakhutdinov, R.R.: Reducing the dimensionality of data with neural networks. Science 313(5786), 504–507 (2006)
Frank, M.J., Rudy, J.W., Levy, W.B., O’Reilly, R.C.: When logic fails: implicit transitive inference in humans. Memory Cognit. 33(4), 742–750 (2005)
Amari, S.I.: Dynamics of pattern formation in lateral-inhibition type neural fields. Biol. Cybern. 27(2), 77–87 (1977)
Tran, S.N., Garcez, A.S.D.A.: Deep logic networks: inserting and extracting knowledge from deep belief networks. IEEE Trans. Neural Netw. Learn. Syst. (2016)
Rastegari, M., Ordonez, V., Redmon, J., Farhadi, A.: Xnor-net: Imagenet classification using binary convolutional neural networks. In: European Conference on Computer Vision, pp. 525–542. Springer International Publishing (2016)
Zhou, X., Li, S., Qin, K., Li, K., Tang, F., Hu, S., … & Lin, Z.: Deep adaptive network: an efficient deep neural network with sparse binary connections. IEEE Trans. Neural Netw. Learn. Syst. (forthcoming)
Goldental, A., Guberman, S., Vardi, R., Kanter, I.: A computational paradigm for dynamic logic-gates in neuronal activity. Front. Comput. Neurosci. 8 (2014)
Vardi, R., Guberman, S., Goldental, A., Kanter, I.: An experimental evidence-based computational paradigm for new logic-gates in neuronal activity. EPL 103, 66001 (2013)
Kiryazov, K., Lowe, R., Becker-Asano, C., Randazzo, M.: The role of arousal in two-resource problem tasks for humanoid service robots. In: 2013 IEEE ROMAN, pp. 62–69 (2013)
Lowe, R., Philippe, P., Montebelli, A., Morse, A., Ziemke, T.: Affective modulation of embodied dynamics. In: The Role of Emotion in Adaptive Behaviour and Cognitive Robotics, Electronic Proceedings of SAB Workshop (2008)
Halford, G.S., Andrews, G., Wilson, W.H., Phillips, S.: Computational models of relational processes in cognitive development. Cogn. Dev. 27(4), 481–499 (2012)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2020 Springer Nature Switzerland AG
About this chapter
Cite this chapter
Lowe, R. (2020). Habit-Based and Goal-Directed Systems: Knowledge Transfer in Individual and Social Learning. In: Giovagnoli, R., Lowe, R. (eds) The Logic of Social Practices. Studies in Applied Philosophy, Epistemology and Rational Ethics, vol 52. Springer, Cham. https://doi.org/10.1007/978-3-030-37305-4_10
Download citation
DOI: https://doi.org/10.1007/978-3-030-37305-4_10
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-37304-7
Online ISBN: 978-3-030-37305-4
eBook Packages: Religion and PhilosophyPhilosophy and Religion (R0)