Computational Neural Mechanisms of Goal-Directed Planning and Problem Solving

Abstract

The question of how animals and humans can solve arbitrary goal-driven problems remains open. Reinforcement learning (RL) methods have approached goal-directed control problems through model-based algorithms. However, RL focus on maximizing long-term reward is inconsistent with the psychological notion of planning to satisfy homeostatic drives, which involves setting goals first, then planning actions to achieve them. Optimal control theory suggests a solution: animals can learn a model of the world, learn where goals can be fulfilled, set a goal, and then act to minimize the difference between actual and desired world states. Here, we present a purely localist neural network model that can autonomously learn the structure of an environment and then achieve any arbitrary goal state in a changing environment without relearning reward values. The model, GOLSA, achieves this through a backwards spreading activation that propagates goal-values to an agent. The model elucidates how neural inhibitory mechanisms can support competition between goal representations, serving to push needs-based planning versus exploration. The model performs similar to humans in canonical revaluation tasks used to classify human and rodent behavior as goal-directed. The model revaluates optimal actions when goals, goal-values, world structure, and need to fulfill drive changes. The model also clarifies a number of issues inherent in other RL-based representations such as policy dependence in successor representations, while elucidating biological constraints such as the role of oscillations in gating information flow for learning versus action. Together, our proposed model suggests a biologically grounded framework for multi-step planning behaviors through consideration of how goal representations compete for behavioral expression in planning.