Reinforcement Learning Models Then-and-Now: From Single Cells to Modern Neuroimaging

Abstract

Although much ignored in some intellectual circles today, behaviorism and its models from the early to mid parts of the twentieth century provided the basis for some of the first computational accounts of reward learning. The best expression of this work emerged in the early 1970s with the Rescorla–Wagner model of Pavlovian conditioning. This model accounted for a range of behavioral data about how animals learn about cues that predict rewarding outcomes. The step forward in this account was that learning was depicted as being driven by failed predictions—that is, some system collected information, formed expectations about how much reward to expect (associated with “conditioned stimuli” or cs), and generated learning updates that were proportional to the size and sign of the error. While successful in describing a large body of data, the Rescorla–Wagner model failed at one critical aspect of simple learning—the capacity to “chain” important cues together into a trajectory of learned associations—a feature called secondary conditioning: “A predicts B predicts food,” for example.