The time for action is at hand

Abstract

The science of mental life and behavior has paid scant attention to the means by which mental life is translated into physical behavior. Why this is so was the topic of a 2005 American Psychologist article whose main title was “The Cinderella of Psychology.” In the present article, we briefly review some of the reasons why motor control was relegated to the sidelines of psychology. Then we point to work showing that experimental psychologists have much to contribute to research on action generation. We focus on studies showing that actions are generated in a way that, at least by default, minimize changes between successive actions. The method is computationally as well as physically economical but also requires consideration of costs, including costs of different kinds. How such costs are compared is discussed in the next section. The final section offers comments about the future of psychologically focused action research. Two additional themes of the review concern methods for studying action generation. First, much can be learned through naturalistic observation. Second, subsequent experiments, designed to check naturalistic observations, can use very simple equipment and procedures. This can make the study of action generation easy to pursue in the psychology laboratory.

In 1967, when Ulric Neisser published his book, Cognitive Psychology, he presented the outline of a new discipline focused on information intake. Neisser’s core questions were threefold: How do we perceive? What information gets stored? How does attention modulate storage? Neisser’s questions conveyed the emerging interests of investigators at the time, most of whom sought to continue the tradition of epistemology begun centuries earlier by the ancient Greeks—most notably by Plato and his disciples. Those ancient philosophers wondered how we can know the world given our limited perceptual experience of it.

The musings of the ancient Greek philosophers came to be supplanted by empirical inquiry by the likes of Ernst Heinrich Weber (1795–1878) and Gustav Theodor Fechner (1801–1887), whose psychophysical investigations helped make psychology a science. As more laboratory work was done, scholars saw fit to review it in a second wave of research. Here, writers like William James (1890), author of Principles of Psychology, Robert Woodworth (1938), author of Experimental Psychology, and S. S. Stevens (1951), editor of Handbook of Experimental Psychology, assembled a tide of new, exciting results. By the time Neisser offered his 1967 manifesto, he saw the makings of a new field—cognitive psychology, as he called it.

The study of perception continues to be hugely important in cognitive psychology (or experimental psychology, more generally), as is evident in journals like this one, whose name reflects a focus on information intake (perception) with regard to selectivity of storage (attention) studied quantitatively (psychophysics). These topics are immensely important, of course, but cognitive psychology (or cognitive neuroscience) cannot just be about how the brain absorbs information. It must also be concerned with how the brain expresses information. The brain is not just a perceptual assimilator. It is also an action generator.

Action for understanding perception

Including action research in the portfolio of psychological research need not imply a rejection of perception as a topic worth investigating. Understanding how action works can shed light on how perception functions. In this context, there is growing appreciation that the way perceivers perceive depends on how their bodies perform. This idea—the idea of embodied perception—can be traced back at least to James Gibson (1950, 1977) and philosophers like Maurice Merleau-Ponty (1945), Gilbert Ryle (1949), and Michael Polanyi (1958). The idea has revolutionized the study of perception, leading to dazzling, though sometimes controversial, findings, like the discovery that the perceived size or form of objects depends on how well they can be acted upon (Proffitt, Bhalla, Gossweiler, & Midgett, 1995; Witt, 2011). No less surprising, though less controversial, is the discovery that the detectability of visual stimuli is affected by where observer’s unseen hands are relative to those stimuli (Abrams, Davoli, Du, Knapp, & Paull, 2008; Reed, Grubb, & Steele, 2006; Thomas, 2017).

One would not have looked for such relations if one thought action could not contribute to perception. Linking action to perception has permitted other advances as well. These have built on the idea that if actions unfold in time, the way they do so may depend on ongoing perceptual processing. Taking this approach, several groups of investigators, beginning with Tipper, Lortie, and Baylis (1992) and continuing with Welsh, Elliott, and Weeks (1999), among others, have shown that people’s hand movements to visual targets are influenced by visual distractors. The veerings of the hand recorded with motion-tracking equipment betray inner dynamics that would have remained hidden otherwise. For a review, see Song and Nakayama (2009).

Others have pursued this general approach as well. Abrams and Balota (1991) and Balota and Abrams (1995) developed a functionally analogous method in connection with memory search and lexical decision. They replaced “yes” and “no” buttons (the standard lab devices) with a joystick whose motion from a central position terminated on a “yes” side or a “no” side. The kinematics of the lever revealed inner dynamics that the button-press times could only hint at.

The same general approach was also taken by Tanenhaus, Spivey-Knowlton, Eberhard, and Sedivy (1995), who recorded computer-mouse movements to targets with different spoken names. Moving the mouse to a picture of a candle when the alternative was a picture of a candy yielded a different motion path than moving the same computer mouse to the same candle when the alternative was a picture of a necklace, say (i.e., an object whose name started with a different sound). The fact that the hand moved more directly to the candy picture when the alternative was “necklace” than when the alternative was “candle” suggested that the process of word identification starts as soon as possible (Spivey, 2007).

Action in its own right

While it is useful to exploit action to study perception, it is just as important to appreciate that action can be studied on its own. Neisser was aware of this, as reflected in his inclusion of speech production in his 1967 book. Neisser probably felt that speech production was worth including because speech expresses thought. Ironically, however, Neisser omitted other physical acts like making beds or picking fruit. This is ironic, for it has turned out that robots can be made to speak more easily than they can be made to make beds or pick fruit. People who earn their livelihoods by making beds or picking fruit occupy the lowest rung of society. This need not imply that what they do is computationally trivial.

Others besides Neisser have recognized the importance of physical action per se. Among them were Robert Woodworth, who we referred to above in connection with his review of experimental psychology. Woodworth’s doctoral dissertation at Columbia University, later published as a monograph (Woodworth, 1899), focused on the control of visually guided aiming. Woodworth studied how people move their hands back and forth between targets that were visible all the time or were only visible early in the movements. Woodworth’s subjects performed these two types of movements in time with a metronome. Woodworth found that when the metronome went slowly, subjects did better with vision than without, as one would expect. But when the metronome went very quickly, performers did no better with vision than without vision. By finding the tempo at which vision no longer helped, Woodworth estimated the time it took to use visual feedback. His estimate was very close to estimates that were later arrived at using more technically advanced equipment.

Based on his work, Woodworth drew a distinction that is fundamental to psychology and neuroscience today—the distinction between automatic and controlled processes. Automatic processes occur without on-line intervention; they were equivalent in Woodworth’s study to performance without vision. Controlled processes do the opposite; they were equivalent in Woodworth’s study to performance affected by visual feedback.¹

Woodworth studied aiming, and so did another giant of experimental psychology, Paul Fitts (1954; Fitts & Peterson, 1964). Fitts is best known for his discovery of stimulus–response compatibility (Fitts & Seeger, 1953), an effect that has been exploited to shed light on a huge range of phenomena (Proctor & Vu, 2006), including implicit bias (Banaji & Greenwald, 2016). Fitts is also well known for his three-stage theory of skill learning, which is still influential today. Here, Fitts (1964) posited a deliberate, cognitive stage, followed by an associative stage where associations are formed between stimuli and responses, leading finally to an automatic stage where conscious attention and deliberate control are barely needed.

Another researcher who studied aiming as well as skill learning is David Meyer (Meyer, Abrams, Kornblum, Wright, & Smith, 1988a; Meyer, Smith, Kornblum, Abrams, & Wright, 1990). Meyer is best known in experimental psychology for his discovery of semantic priming in lexical decision, where, for example, the time to affirm that the letter string “doctor” is a word is shorter if it is preceded by a semantically related word such as nurse than if it is preceded by a semantically unrelated word such as table (Meyer & Schvaneveldt, 1971). Meyer also offered innovative approaches to the chronometric (time-based) analysis of human information processing (Meyer, Osman, Irwin, & Yantis, 1988b). His theory of skill learning, developed with David Kieras (Kieras & Meyer, 1997), shared elements with Fitts’ theory.

The involvement of distinguished researchers like Woodworth, Fitts, and Meyer in action-related research shows that experimental psychologists have recognized the importance of action. Their openness indicates that action research has a rightful place in the study of cognitive psychology and cognitive neuroscience. No one would deny this, of course, but it has been harder to get researchers to commit to this area of study than one might expect.

Action generation

What is the most fundamental question about action that can be pursued by experimental psychologists? The most fundamental question is, How are actions generated? That basic question can be extended to two more: What happens before actions are performed, and What distinguishes actions that are performed from actions that are not performed but are physically possible? The second question will resonate with readers attuned to linguistics, for just as there are many ways to verbalize a thought, there are many ways to achieve a physical objective. Just as there are grammars of language, there are grammars (or a grammar) of action (Goodnow & Levine, 1973).

Consider, for example, the trivial task of touching one’s nose. There are an infinite number of ways to do so. One can touch one’s nose with different parts of one’s hand or with one’s hand in different poses. For each of these hand positions, infinitely many spatial and temporal paths allow the hand to reach its terminal point.

What distinguishes the movements that are performed from the ones that are not? The performed movements are “natural” or “efficient” rather than “unnatural” or “inefficient.” The natural, efficient movements generally emerge with little cognitive effort, especially when they are well learned. Only in special circumstances—when one is at a very early stage of learning, when one has suffered bodily damage or disease, or when one is trying to perform in deliberately inefficient ways for theatric or didactic reasons (e.g., to teach about motor control)—does more cognitive effort come into play.

Conscious awareness may also occur when one is afraid of failing at a task, as having stage fright and other instances of psychological freezing. Ironically, paying more attention to the way a task is performed can make it harder to perform (Beilock, 2010; Wulf & Lewthwaite, 2016). On the other hand, conscious intervention can be used to alter the way habitual tasks are carried out, as in Monty Python funny walks or exaggerated stepping during military parades.

The first author often uses odd ways of moving in lectures to show that there are “soft constraints” in action generation. When he teaches about motor control, he often discusses the act of touching his nose and does some silly things, like snaking his hand around his head on the way to his nose to show that, through an act of will, he, like anyone, can deliberately act in inefficient ways. The fact that one can alter otherwise automatic motions for the sake of stylistic, theatrical, or pedagogic purposes is actually remarkable. Such a capability indicates that action generation relies on soft constraints, not just hard constraints governed by physical and biomechanics. Identifying those soft constraints and understanding how they are used can be a core challenge for experimental psychologists.

The neglect of action in psychology

Given this backdrop, one would think that action generation would be one of the most widely studied topics of psychology, Surprisingly, however, it has not been. In fact, action has received scant attention. This is astonishing when it is recalled that psychology is the science of mental life and behavior (Gleitman, 1981). Given such a definition, one would have thought that psychology would have paid a huge amount of attention to the means by which mental life is translated into behavior. Why it has not was the focus of an article titled “The Cinderella of Psychology: The Neglect of Motor Control in the Science of Mental Life and Behavior” (Rosenbaum, 2005). The point of referring to Cinderella was to suggest that action research (the study of motor control) has been swept aside, much as Cinderella was. Like Cinderella, motor control is entitled to more attention.

A core claim of the Cinderella article was that a major reason why action research received short shrift was that experimental psychologists were preoccupied with perception. That same view was expressed above. Another reason for the preoccupation with perception was also discussed in the Cinderella article, but it has not yet been mentioned here. That other reason was that a guiding principle in experimental psychology has been functionalism. By presenting stimuli and recording responses provoked by them, researchers can do controlled studies. They can control what stimuli they present and record what responses those stimuli engender. Accordingly, they can begin to infer functions relating stimuli to responses. Giving participants free reign to behave as they will and then trying to infer the basis for the actions is, or seems to be, much harder.

In the remainder of this article, we hope to show that experimental psychologists can in fact learn a great deal about action generation; they need not be daunted by subjects running rampant in the laboratory. With some gentle prodding or with open eyes, experimental psychologists can observe behavior and learn useful things about its basis.

The review that follows is meant to bear witness to this statement. We focus on work from our own lab, largely because the experiments that have been done in our laboratory have been very simple. In addition, most of the phenomena were based on everyday observations. We highlight these features of the work because the work may be taken to show that action research is not especially hard to do. On the contrary, it is actually very easy to do (once the main ideas have been worked out, which can always be challenging). As a practical pedagogic matter, student research assistants often find this kind of work fun.

The past meets the future

The first section of the work reviewed here is about the way actions are prepared with reference to actions that have just been performed. The name of this section, The Past Meets the Future, refers to this interface. A core claim is that the nervous system uses an efficient method to plan and prepare forthcoming actions: Whenever possible, forthcoming actions are planned to differ as little as possible from just-completed actions. This strategy is mechanically and computationally efficient.

The parameter remapping effect

The view just espoused—that forthcoming actions are planned to differ as little as possible from just-completed actions—came, rather surprisingly, from an experience at a rehearsal of a community orchestra. The first author, who happens to be an amateur violinist, was at a community-orchestra rehearsal where he, along with the other fiddlers, played a passage for the conductor, who wanted to know how the passage would sound if we bowed it one way or another. The passage consisted of a single quarter note followed by two eighth notes. We played the passage over and over again at a rapid pace. Critically, the way we spontaneously chose to play the passage was with a down bow for each quarter note followed by two successive up bows for the two eighth notes (see Fig. 1, top panel). The conductor wanted to hear how it sounded if we played it a different way, so he asked us to play it with strictly alternating up and down bow strokes (see Fig. 1, bottom panel). When we did, we fell apart. All of us, including the professional “ringers” in the orchestra, got tremendously confused and broke out in laughter. (The conductor was not amused, and soon thereafter the first author decided to just play chamber music.)

Fig. 1

The parameter remapping effect illustrated with bow bloopers. Down-bows have rectangular marks above them. Upstrokes have triangular marks above them.

Why did we collapse? The first author hypothesized that the strictly alternating bow-stroke passage was hard because the associations between bow-stroke directions and bow-stroke durations kept changing. The first time the quarter note was played, it was played with a down bow, but the next time the quarter note was played it was played with an up bow, and similarly for the eighth notes. By contrast, when the passage was played as we had played it at first, the mappings between bow-stroke directions and bow-stroke durations stayed constant.

If this interpretation was correct, then it meant that when an action is performed, a memory of it is maintained in working memory. Such a memory could be useful for the next action to be performed, for it could be prepared by just changing the features that distinguish it from the action that was just completed. The more changes that might be needed, the slower and more error-prone the performance might be, at least if there were some cost, in terms of time and accuracy, of remapping the relevant control parameters. By this account, the bow bloopers at the rehearsal reflected a more general parameter remapping effect.

To test this interpretation and to evaluate its generality, Rosenbaum, Weber, Hazelett, and Hindorff (1986) predicted that participants would have a very hard time performing functionally analogous tasks like reciting the first few letters of the alphabet over and over again with the requirement that the recitations must always alternate between stressed and unstressed pronunciations. The prediction was confirmed. Participants found it easy to say “AbAbAbAb . . .” … where the capital letter denotes a stressed letter and the lowercase letter denotes an unstressed letter, but the same subjects found it very hard to say “AbCaBcAbCa . . .” where again capital letters denote stressed pronunciations and lowercase letters denote unstressed pronunciations. As seen in Fig. 2, the odd-n cases were harder than the even-n cases. The smaller the value of n, the greater the difference.

Fig. 2

The parameter remapping effect illustrated with tongue twisters. Upper-case letter are stressed. Lower-case letters are unstressed. Adapted from Rosenbaum, Weber, Hazelett, & Hindorff (1986)

Why was it so hard to recite the beginning of the alphabet over and over again when the required number of letters, n, was odd, and why did the difference between odd n and even n grow as n got smaller?

The answer can be sketched as follows. When n was even, the mappings of stresses to letters stayed constant, but when n was odd, the mappings of stresses to letters kept changing. The rate at which the mappings changed grew as the value of odd n decreased. The shorter the time (or the smaller the number of letters) between successive productions of the same letter, the greater the strength of association between the letter and the stress. Said another way, the longer the time (or the larger the number of letters) between successive productions of the same letter, the smaller the strength of association between the letter and the stress due to decay. By hypothesis, then, the smaller the value of n, the greater the effect of the match or mismatch of successive letter-accent mappings. The more often the mappings of letters to accents had to change, the greater was the challenge of inhibiting the existing mapping and applying the new one. This could explain why the two curves diverged as n decreased and converged as n increased.

If the parameter remapping effect is a truly general effect, it should appear in other tasks as well. Rosenbaum, Weber, Hazelett, and Hindorff (1986) showed that it did in the production of memorized keyboard sequences. College students from the same population as those who did the speech task performed memorized finger-tapping sequences repeatedly and as quickly as possible (see Fig. 3). For one of the sequences, a given finger was pressed the same number of times each time the finger was pressed: RRMIIimmr (where “raised” letters are for the right hand—Ring, Middle, and Index—and lowercase letters are for the left—index, middle, and ring). For the other sequence, a given finger was pressed a different number of times each time that finger was used: MMIiiMIIi. As predicted, participants had a harder time in the latter case than the former. The latter sequence was harder because, by hypothesis, the mappings of number of finger taps to fingers changed. This forced participants to engage in parameter remapping (at least by hypothesis). When the mappings of number of finger taps to fingers stayed the same, they did not have to engage in this process, and their performance benefited from the lower demand.

Fig. 3

The parameter remapping effect illustrated with finger fumblers. Adapted from Rosenbaum, Weber, Hazelett, & Hindorff (1986)

These results are consistent with the view that future actions are prepared by changing just those features that distinguish what will be done from what has just been done before. Modifying what needs to be changed is more efficient than preparing each action from scratch. Reprogramming is more efficient than programming de novo.

This suggestion is reminiscent of, or arguably identical to, the principle of least action, a deep principle from physics (https://en.wikipedia.org/wiki/Principle_of_least_action). Within physics, the principle of least action explains (or describes) why the path taken by light is the path that minimizes time. As applied to biological motion, the principle of least action asserts that the path taken is the one that minimizes resistance or “effort.” To the best of our knowledge, the principle of least action has not been applied to the computational processes underlying action generation, at least explicitly. We propose here that it does or should.

Our belief that action planning might obey the principle of least action also encourages us to suggest that working memory is likely to be important for the retention and modification of motor commands. Working memory has classically been viewed as the site (or function) responsible for the temporary storage of new sensory data or the refreshment or reorganization of previously learned material (e.g., Cowan, 2001). The idea that working memory has a special role in action generation has been pursued by others, notably MacDonald (2016), in regard to speech. The idea also has roots in earlier work by Henry and Rogers (1960); Klapp (1977); Sternberg, Monsell, Knoll, and Wright (1978); and Rosenbaum (1987).

The hand-path priming effect

The parameter remapping effect can be defined as the tendency for action sequences to be slower and more error prone when action elements do not repeat than when they do. This is why, no doubt, babies engage in reduplicative babbling (e.g., saying “mama, mama”) before they engage in more complex speech.

Does repetition help the generation of actions that last longer than speech sounds? Speech sounds, like key presses and bow strokes, are generally short, at least if they are produced as words or proto-words rather than extended songs. If the benefit of repetition applies to the generation of more temporally extended actions, one should find evidence for it in other behavioral contexts.

Supporting evidence came from another personal experience. While at a dinner party, the first author found himself reaching repeatedly for olives in a dish on the table. To reach each olive, he had to reach around a wine bottle, take hold of one of the olives, bring it to his mouth, munch on it, then reach for the next olive, and so on. This happened over and over again as the first author listened intently to someone across from him. At some point, another guest at the dinner party removed the bottle, clearing a path to the olive dish. Nevertheless, the first author kept moving his hand back and forth in a curved path several times before realizing that his hand was moving in a path that was needlessly curved. The bottle’s removal was not subtle; it could not be missed. Nevertheless, its disappearance made little impression on the first author action wise.

Was this a reliable phenomenon? It turned out to be, as shown in laboratory tasks using virtual reality (Jax & Rosenbaum, 2007) and a task involving repeated touches on spatially arrayed targets on a table top when there were obstacles between some of the targets (van der Wel, Fleckenstein, Jax, & Rosenbaum, 2007). The phenomenon came to be called the hand-path priming effect. Like the hand paths shaped by attention, as mentioned above, the hand-path priming effect reflects a cognitive influence on the way hand movements are produced.

The grasp-height effect

The two phenomena reviewed so far, the parameter remapping effect and the hand-path priming effect, are retrospective effects, effects of the past on the present. The phenomena described in the next two sections are prospective effects, effects of the future, or anticipation of the future, on present performance. The phenomena to be described next, which are prospective effects, nevertheless reflect the same principle as those just reviewed: Actions tend to be selected in ways that minimize change.

Consider the grasp-height effect. This phenomenon came to light through yet another personal experience. While reaching for a standing bathroom plunger to be moved to a new location, the first author noticed that he took hold of the plunger high before bringing it to a low position (down to the floor). Upon noticing this, he purposely transported the same plunger from the same initial position to a high location. This time, he grasped it low. Apparently, he knew, at least implicitly, that it is sensible to grab objects in ways that permit avoidance of extreme body postures.

Laboratory tests confirmed this naturalistic observation (Cohen & Rosenbaum, 2004; Rosenbaum, Halloran, and Cohen, 2006). As shown in Figs. 4 and 5, when naïve subjects reached out to reposition a (fresh) standing plunger from a fixed height to various target heights, their grasp heights varied inversely with the height to which they would move the plunger. Either by dint of on-line computation or recall of previous experiences—Cohen and Rosenbaum (2004) did not try to distinguish between these alternatives—participants showed that they were aware, if only implicitly, that always grabbing a vertically oriented object at the same comfortable height would backfire, leading to extreme bends or extreme stretches, which would be uncomfortable or hard to control. By contrast, grabbing the plunger at a higher place for a low final placement or a lower place for a high final placement would avoid the need for deep bends if the object were brought low or the need for high stretches if the object were brought high.

Fig. 4

Three of the plunger transfer tasks used by Cohen and Rosenbaum (2004). Left panel: The plunger is to be brought to a high shelf. Middle panel: The plunger is to be brought to a mid-level shelf. Bottom panel: The plunger is to be brought to a low shelf. Both individuals shown here (D. A. Rosenbaum is the experimenter, standing in the background) and the participant agreed to have their pictures shown

Fig. 5

The grasp height effect. Adapted from Cohen and Rosenbaum (2004)

The end-state comfort effect

The phenomenon just described, the grasp-height effect, concerns where along the length of an object the object is grasped depending on the height to which it will be carried. The phenomenon described in this section concerns how the hand is positioned (which grasp orientation is adopted) depending on how an object will be moved. The phenomenon of interest was first observed at a restaurant. The first author saw a waiter picking up upside-down glasses to be filled with water. The waiter grasped each glass thumb down. This caught the first author’s eye because when one picks up a glass, one normally does so thumb up.

Why did the waiter pick up the inverted glasses with a thumb-down hand posture? The reason (or likely reason) was not hard to fathom. Grasping the upside-down glasses with a thumb-down grasp let the waiter end the flips with his thumb up, which was presumably more comfortable and easier to control than with his thumb down. Imagine holding a glass and trying to pour water into it with your hand in a thumb-down position. The awkwardness of the task is obvious.

The phenomenon was brought to the lab to check it. In the first experiment, participants were presented with a dowel lying on a cradle a few inches above the table on which the cradle stood (Rosenbaum et al., 1990). As Fig. 6 shows, one end of the dowel was white; the other end was black. University student participants were asked to pick up the dowel and place the white end down onto a nearby target or to pick up the dowel and place the black end down onto the same nearby target. The dowel was high enough above the table for participants to be able to pick up the dowel with an underhand grip or with an overhand grip, as they preferred. Participants were told to use the preferred hand and were also told in each trial which end of the dowel was to be placed on the target.

Fig. 6

The end-state comfort as shown by the eleven participants (all using the right hand) in the first experiment of Rosenbaum et al. (1990). Adapted from that source

The result was that all of the right-handed subjects picked up the dowel with an overhand grasp when the right end of the dowel was to be placed on the target, and these same subjects also picked up the dowel with an underhand grasp when the left end of the dowel was to be placed on the target. The one subject who used the left hand showed the mirror-image pattern (i.e., the same anatomically defined preferences).

Rosenbaum et al. (1990) collected comfort ratings to better understand the basis for the grasp choices. These were consistent with the hypothesis that the underhand grasp was less comfortable than the overhand grasp and that the thumb-up final posture was more comfortable than the thumb-down final posture. The least comfortable hand posture of all (the hand posture with the lowest comfort rating) was the thumb-down posture. Because the comfort ratings that best predicted the choices were the comfort ratings for the final position, Rosenbaum et al. (1990) concluded that participants chose the initial grasp in a way that avoided the least comfortable grasp at the end of the maneuver. Accordingly, the authors called this the end-state comfort effect.

Owing to the simplicity of this object transfer procedure as well as the robustness of the effect (it was seen in the vast majority of subjects in a wide range of tests), it became attractive for other researchers with varied interests. Among the scientists who picked up on the method were developmental psychologists who wondered whether children would show the effect as well. A surprise awaited them. They found that only by age 9 or so did children show the end-state comfort effect at frequencies approximating those seen in adults. This result was obtained in several laboratories (Comalli et al., 2016; Weigelt & Schack, 2010; Wunsch, Henning, Aschersleben, & Weigelt, 2013; Jovanovic and Schwarzer, 2011; Wunsch, Weiss, Schack, & Weigelt, 2014).

Why didn’t children show the effect as consistently as adults? One possibility is that the task was artificial, and children misunderstood what was expected of them. This hypothesis was essentially ruled out by Comalli et al. (2016), who asked children to pick up and use a hammer. The experimenters varied the position of the hammer and observed that the children picked it up with a comfortable hand posture even if they ended in an awkward hand posture at the time of hammering. It was not that the children had never hammered before or misunderstood what was expected of them.

How can such a surprising result be explained? It is doubtful that children cannot plan ahead in such a simple task, for children much younger than 9 can engage in sophisticated cognitive activities like forming complex linguistic constructions (McCarthy, 1933) or lying (Lewis, 1993). A more likely possibility is that the children simply experienced less discomfort in extreme arm positions than adults did, at least for the biomechanical dimension of importance here, pronation/supination. Pronation and supination are achieved by twisting the forearm about its long axis of rotation. Pronation involves turning the right thumb counterclockwise or the left thumb clockwise. Supination involves turning the right thumb clockwise or the left thumb counterclockwise.

Conceivably, for reasons that may be rooted in the nature of physical growth, the discomfort or loss of movement precision associated with extreme pronation or supination in adults may not hold for children. To check this possibility while preparing the present article, the authors e-mailed Dr. Kornelia Kulig, professor of biokinesiology and director of the Musculoskeletal Biomechanics Research Laboratory at the University of Southern California, whom we asked about this hypothesis. Dr. Kulig wrote back (personal communication, June 21, 2018), as follows: “The uncovertebral joints, which are absent at birth, appear between the age of 6 and 9, and that likely contributes to modified spinal (cervical) kinematics.” Through additional materials with Professor Kulig, we learned that the uncoverterbal joints (which may be read as “unco-vertebral” to better understand the term), are observed at the posterior-lateral rim of the superior vertebral body of the cervical spine and restrict primary and coupled motion there. These joints are also known as Luschka’s joints, as we learned from a Wikipedia article (https://en.wikipedia.org/wiki/Luschka%27s_joints). So according to Professor Kulig, it may be that reluctance to adopt extreme pronation/supination forearm angles in people ages 9 or older is indeed due to spinal changes that culminate around age 9—just when children show the end-state comfort effect to the same degree as adults.

If this interpretation is correct, children should show adult-like planning of forthcoming hand positions when the planning is with respect to a biomechanical variable that does not change reliably over the course of childhood. In other words, if there is a biomechanical variable that effectively takes on adult values early in childhood, young children should plan as well as adults when it comes to that variable.

Results that fit with this prediction were reported by Herbort, Büschelberger, and Janczyk (2018), who asked children to turn a knob horizontally to rotate a toy crane so it could be used to pick up another toy. The children placed their palms on the knob in a way that let them turn the crane such that the children would end the turns with their hands at midrange rather than extreme angles. In this respect, they acted like adults, as shown by the fact that, in an earlier study, Zhang and Rosenbaum (2008) demonstrated that adults placed their hands on an inverted kitchen pan prior to turning the pan to bring the handle to each of a number of targets in a way that let the hand come to rest at or near a midrange position on the adduction/abduction dimension (see Fig. 7). If one supposes that adduction/abduction of the hand is unaffected by spinal uncovertebral change, the ability of children to plan for abduction/adduction fits with the idea that biomechanics alone, not a limitation of planning, accounts for the failure of children to advance plan for pronation/supination.²

Fig. 7

Experimental setup (left panel) and hand placement angle on the inverted pan as a function of final hand angle (right panel) reported by and adapted from Zhang and Rosenbaum (2008)

The apples and oranges problem

The work described so far suggests that actions are typically selected to minimize costs. (Of course, we appreciate that when people deliberately exert themselves, as in athletic competition, they purposely violate this strategy.) If costs are minimized, an important question is how the costs are minimized when they are of different kinds. In planning actions, one often faces a kind of apples-and-oranges problem.

As a case in point, consider the simple act of walking to a table to pick up an object. If the object is close to one edge of the table but a long walk is required to get there, it might be better to take a long walk to allow for a short reach rather than a short walk requiring a long reach. Which choice is made in which circumstance?

To address this question, Rosenbaum, Brach, and Semenov (2011) conducted an experiment (see Fig. 8) in which university students stood several meters from the foot of a long table. They looked down the length of the table and saw two stools beyond it, one beyond the left end of the table and the other beyond the right end of the table. In different conditions, the distances to the left and right stools varied. On the table in all conditions stood a child’s beach bucket whose distance from the foot of the table was constant. The bucket occupied one of three horizontal positions—near the left edge of the table, in the middle, or near the right edge. The participant was invited to do whatever seemed easier: (1) walk along the left edge of the table, pick up the bucket with the right hand, and carry it to the left stool; or (2) walk along the right edge of the table, pick up the bucket with the left hand, and carry it to the right stool. The experimental design made it possible to present participants with choices of different reaching distances paired with short, medium, or long walks. The question was whether subjects would be willing to walk a long way to avoid a long reach. In general, what combinations of walking distances and reaching distances would subjects prefer?

Fig. 8

Representative setup used in the experiment of Rosenbaum, Brach, and Semenov (2011).

It turned out, as shown in Fig. 9 and as arrived at with some mathematical modeling, that 1 meter of left-hand reaching was about as costly as 12.5 meters of walking, and 1 meter of right-hand reaching was about as costly as 10.5 meters of walking. All of the participants were right-handed, so it made sense that the best estimates for the walking-compared-to-reaching costs were larger for the left hand than for the right. What is most important, however, is that, in general, reaching was estimated to be costlier than walking—about 11.5 costlier on average per meter. This outcome presumably reflects the fact that reaching over a large distance posed a challenge. Reaching very far could have caused participants to fall over. Walking that same distance would keep one on one’s feet.³

Fig. 9

Probability, p(L) of walking along the left side of the table, picking up the bucket with the right hand and carrying it to the left stool rather than walking along the right side of the table, picking up the bucket with the left hand and carrying it to the right stool as a function of Left Path Effort Minus Right Path Effort, where Left Path Effort was defined as Walking Distance (m) + 10.3 * Right-Hand Reaching Distance (m) and Right Path Functional Distance = Walking Distance (m) + 12.3 * Left-Hand Reaching Distance (m). Adapted from Rosenbaum, Brach, and Semenov (2011)

Moving and the serial position effect

In other work, we have pursued the question of the relations among costs of even more disparate activities, namely, moving and memorizing. Previous research, not summarized above, showed that there are surprising connections between moving and memorizing. Weigelt, Rosenbaum, Huelshorst, and Schack (2009) discovered an unexpected effect of physical action on the serial position curve for recall (see Fig. 10). Typically, one sees that most recent items are recalled better than earlier items, but earliest items are recalled better than items that come later, but not at the end of the presentation. So the serial position for recall is a U-shaped function relating probability of recall to the serial position of the item to be recalled. Weigelt et al. (2009) found something different. Rather than finding that most recent items are recalled better than earlier, noninitial items (the recency effect), these authors found that most recent items were recalled no better than earlier noninitial items. Meanwhile, initial items were recalled best, as in the classical serial position curve (the so-called primacy effect). For a review of work on the serial position curve, see Glanzer and Cunitz (1966).

Fig. 10

Apparatus (left panel), percent overhand and underhand grip in ascending (top middle panel) and descending sequences (bottom middle panel) and probability of correct recall in the ascending and descending conditions (top and bottom right panel, respectively, from Weigelt et al. (2009)

In the procedure that led to the surprising discovery by Weigelt et al. (2009), Weigelt et al. asked young adults to pull open vertically stacked drawers, one after the other (see Fig. 10, top panel). For any given drawer, the task was to open the drawer and reach inside, remove an inverted cup, turn the cup over, and look inside it to read a letter, then return the cup to its original inverted position, close the drawer, and go on to the next drawer. The same subjects were asked to make their way through the drawers in an ascending order (bottom to top) or in a descending order (top to bottom). The drawers had oval holes that afforded palm-up or palm-down hand insertions. These apertures were used to see whether subjects would use overhand (palm down) or underhand (palm up) grips, as in the other studies of the end-state comfort effect described above.

The main finding with respect to end-state comfort was that participants mainly used palm-up grips for low drawers and palm-down grips for high drawers (see Fig. 10, bottom panel). In addition, the transition from one kind of grip to the other depended on the order of drawer opening. If the drawers were opened in ascending order, the switch from palm-up to palm-down hand insertions occurred at a higher position than if the drawers were opened in descending order. Such a change in a transition point is known as a hysteresis effect. Hysteresis is familiar in heating systems, which are designed to switch on and off at different temperatures. When hysteresis is observed in nature, as it is in magnetism, it is usually taken to reflect conservation, much as the parameter remapping effect was taken to reflect conservation á la the principle of least action. The hand-path priming effect was interpreted this way as well. Observing hysteresis for hand postures in a task like opening successive drawers may be viewed as a further demonstration that, without a compelling reason to change the way an action is performed, no change is made in the way it is.⁴

Returning to the memorization result of Weigelt et al. (2009), what should one make of the finding that the recency portion of the serial position curve disappeared in the hands, as it were, of Weigelt et al.’s subjects? First, it is important to note that this result was replicated by others (Logan & Fischman, 2011, 2015). Second, a functional interpretation of the result can be pursued by considering the classical interpretation of the recency effect (Glanzer & Cunitz, 1966). According to this interpretation, initial items and terminal items are interfered with less than noninitial or nonterminal items. The noninitial and nonterminal items (the items in the middle) are interfered with from “both sides” whereas initial items are only interfered with by later items. Terminal items are only interfered with by earlier items. (This assumes, of course, that there is some sort of marking of the events as a distinct psychological episode; the initial and final items are not the first and last items of a participant’s life.) Moreover, first items have a good chance of being recalled because they have a high probability of getting into long-term memory, whereas recent items have a good chance of being recalled because they have a high probability of being retrieved from working memory. Items in the middle have the lowest chance of being recalled because they have the lowest chance of making it to either store.

The fact that the recency effect was knocked out in the contexts studied by Weigelt et al. (2009) and Logan & Fischman (2011, 2015)—also see Spiegel, Koester, Weigelt, and Schack (2012)—suggests that planning and performing physical actions uses working memory. Earlier studies in which the recency effect was observed had subjects just sit there, so to speak, and listen to or read items to be memorized; their physical action demands were trivial. The fact that the recency portion of the serial position curve was eliminated when significant activity was called for is consistent with the hypothesis, mentioned earlier in this article and supported by the hysteresis result just reported, that working memory plays a role in action. It seems reasonable to speculate, in fact, that a primary role of working memory is, indeed, to support action generation.

Metacognition, memory, and action

The final line of experimentation to be reviewed here continues the concern with the apples and oranges problem. The question is, How do people compare the difficulty of physical tasks and mental tasks? These two kinds of tasks are ostensibly different. Skills associated with physical tasks and mental tasks come on-line at different phylogenetic stages (planaria can swim but cannot read), and at different ontogenetic stages (children can walk before they can read). However, perceptual-motor abilities and mental abilities may have more in common than might be assumed at first. Analyses of the learning of perceptual-motor skills and more intellectual skills has turned up more similarities than differences between the two (Rosenbaum, Carlson, & Gilmore, 2001; Schmidt & Bjork, 1992). Also, as mentioned above, body-based representations affect perception. Such representations also support concept attainment (Beilock, 2015; Giles et al., 2018; Goldin-Meadow & Wagner, 2005).

One way to investigate the cross-talk between intellectual and perceptual-motor representations is to explore the way intellectual and perceptual-motor task difficulties are compared. The idea is to map out relations between cognitive task difficulty and perceptual-motor task difficulty, much as our lab has mapped out relations between different forms of perceptual-motor task difficulty (reaching distance and walking distance). If participants can systematically relate cognitive and perceptual-motor task difficulty, that would fit with the hypothesis that the two forms can be mapped to one other in predictable ways, perhaps because they share some common representation.

To pursue this possibility, Feghhi and Rosenbaum (2018) asked university students to choose between carrying a box through a wide (81 cm) gap or narrow (36 cm) gap after memorizing a list of six, seven, or eight digits. In each trial, one sic-digit, seven-digit, or eight-digit list was associated with the wide gap and a different six-digit, seven-digit, or eight-digit list was associated with the narrow gap (see Fig. 10, left panel). The question was which combination of gap width and memory length subjects would view as easiest. We thought choices of this kind would reflect people’s metacognitive beliefs about the resources needed to perform the tasks. We wondered whether the choices would minimize the error probabilities and whether independence or dependence between the two kinds of error probabilities would be associated with independence or dependence between the choices. The possible outcomes could be conceptualized as occupying a 2 × 2 matrix:

Choice probabilities	Error probabilities
	Independent	Dependent
Independent	a	b
Dependent	c	d

If the choices and error probabilities both reflected independence (Cell a) or both reflected dependence (Cell d), that outcome would suggest concordance between metacognition and performance. By contrast, if the choices reflected independence while the error probabilities reflected dependence (Cell b), that outcome or its complement (Cell c) would suggest discordance between metacognition and performance.

The result, as shown in the right panel of Fig. 11, was that the error probability data and choice probability data best supported Cell a. Participants made choices as if there was no interaction between the two demands, and the error probabilities also supported independence. Statistically, the likelihood of the data given each of the other three models (Cells b, c, or d) was much lower than the likelihood of the data given the independent–independent model (Cell a).

Fig. 11

Overhead schematic of the experimental setup (left panel) and best model of subjective difficulty for combined physical and mental task as a function of memory load (right panel). As seen in the left panel, the participant stood at a waist-high table with a box on it which the participant would pick up and carry. Two waist-high stools stood to either side, each with a card (not shown) on which appeared a list of 6, 7, or 8 random digits that were easily readable from the participant’s standing position. Beyond the box was a pointer that could turn if the participant touched it while attempting to pass through the gap on one side or the other. In this example, the left gap was narrow and the right gap was wide, but the narrow gap could be on the right with the wide gap on the left. A waist-high stool at the end of the alley was where the box would be deposited and where the participant would stand while attempting to recall the digits for the chosen side. As seen in the right panel, subjective difficulty was found to be an additive function of the number of digits in the list that was memorized and the challenge of passing through the gap, which was greater by the equivalent of .56 digits when the gap was narrow rather than wide. From Feghhi & Rosenbaum (submitted)

Because of space limitations, we have to refrain from reviewing the data and our method of analyzing it. Nevertheless, we want to mention an additional feature of the best model. That feature was that with this approach we could quantify the amount by which the extra challenge of passing through the narrow gap rather than the wide gap corresponded to the challenge of memorizing extra digits. As seen in the right panel of Fig. 11, the extra challenge of navigating the narrow gap turned out to be functionally equivalent to memorizing an extra .56 digits. That outcome is reminiscent of the outcome of the study described earlier, where it was found that reaching 1 meter was functionally equivalent to walking 11.5 meters. In both cases, psychophysical methods proved useful for providing quantitative estimates of psychological elements related to action generation.

Conclusions

Our aim in this article has been to show that investigators who read and contribute to journals like Attention, Perception, & Psychophysics can contribute to the study of action generation. This topic is not one that needs to be viewed as the sole province of some other field, such as neurophysiology. We refer to this other field because a remark we often encounter when we meet others and tell them what we work on is that we must be physiologists or neurologists dealing with neuromuscular diseases, such as Parkinson’s disease. This situation is gradually changing as more and more people are coming to appreciate the importance of action generation per se, thanks in part to the growing interest in embodiment.

Still, as we have tried to show here, action generation is not just a matter of movement disorders. Nor is it just a matter of reflexes (as in kicking one’s leg when a doctor taps one’s patellar tendon) or strength and agility (as in athletics). These are domains we also hear about when we tell others that we study action generation. The generation of actions occurs all the time in everyday life, usually in prosaic tasks like making beds, picking fruit, and so on. Given how ubiquitous action generation is and how fundamental it is to understanding the brain and behavior, it is incumbent on experimental psychologists to contribute what they can to its analysis. We have tried to show here that experimental psychologists can indeed contribute usefully to the elucidation of action generation.

Approaching the end of this article, we want to make some final observations. First, it is important to say that much more work has been done on action control than we could cover here. As we stated in the opening, this review mainly focused on our own lab’s projects. The rationale was that our lab has, for a long time and perhaps for a longer time than any other lab that primarily identifies itself as a cognitive lab, blended classical cognitive concerns with concerns of motor control. Meanwhile, broader reviews of the literature can be found in textbooks and monographs such as those by Graziano (2008); Hommel, Brown, and Nattkemper (2016); Jeannerod (2006); Rosenbaum (2010, 2017); and Schmidt, Lee, Winstein, Wulf, and Zelaznik (2018).

Among other topics covered in these works are the involvement of attention in motor performance, the role of intentions in movement production, learning of perceptual-motor skills, changes in action skills in old age, individual differences in perceptual-motor abilities, the neural control of movement, the support of action through perception, the importance of nonverbal communication, the expression of emotion, robotics, human factors, and clinically relevant changes in the capacity to move or hold steady. All of these fields have witnessed important advances.

A very important advance, which we have played up less than we would have liked to just because of the way our story has unfolded, is the growing understanding of the importance of a common mental representation for perception and action, the common coding theory of Prinz (1990) and Hommel, Müsseler, Aschersleben, and Prinz (2001). It is very likely that in choosing which actions to perform (whether consciously or not), a critically important source of information is the remembered perceptual consequences of those actions relative to the remembered perceptual consequences of other actions that are possible but are not chosen (either explicitly or only implicitly). For an example of how this process might play out, see Janczyk and Kunde (2014).

Hopefully, we have succeeded in communicating that experimental psychologists can bring their methods of observation, experimentation, and analysis to the study of action generation. In this connection, one of the most important lessons we hope we have conveyed is that no special equipment is needed to study action generation. Similarly, no special knowledge of physics or biomechanics is required to work in this area, though added knowledge is, of course, always helpful. Special equipment such as motion capture devices can, of course, also add immeasurably.

Virtually all the phenomena described here were found through observation of everyday behavior. Such naturalistic observation is often omitted in textbooks that cover methods in psychology. Having examples of naturalistic observations that yield useful results may help bring naturalistic observation to the fore. Naturalistic observation is important, in our view, not just because it heightens the chance of finding new phenomena but also because it helps combat the mind-set that the only questions one can ask are ones for which one has available equipment. Similarly, it is important not to treat any one task, such as a task for which one has the apparatus needed to record it, as emblematic of all the tasks that can possibly reflect a faculty of interest. For example, a great deal of clinically related research has treated the serial-reaction-time task as a kind of sine qua non of perceptual-motor ability. The serial-reaction-time task involves pushing buttons in response to series of stimuli. This is a task that draws on vision, action, and cognition, as shown by a popular application of this method where a repeated sequence of test stimuli is embedded in an otherwise random sequence. One can see reductions in reaction time (RT) accompanying repeats of the sequence once it has been learned, along with increases in RT when the implicitly expected pattern suddenly is changed (Nissen & Bullemer, 1987). This a good example of a perceptual-motor task, but so too are many others that may tap into other component abilities, such as walking and reaching. Until one has a carefully reasoned typology of tasks, one needs to be careful about equating particular tasks with an entire class of skill.

The last comment we wish to make is that the kind of work discussed here can have practical value. A better understanding of how skills are learned can help instructors teach skills more effectively, regardless of whether the skills are musical, athletic, surgical, or something else. Similarly, a better understanding of how people choose actions depending on the stresses they are under and the signals they are given can reduce the chance of accidents; think here of cyclists riding bicycles, drivers driving cars, or pilots flying planes. Finally, building on the last experiment, work spaces can be designed more intelligently by knowing how people choose actions within them. Imagine, for example, that you are asked to help design a medical examination or operating room. Doctors, it turns out, are reluctant to walk to a computer to get patient information; the farther away the computer, the greater the reluctance to go to it (or the lower the probability of doing so). Even if the computer is 5 meters away, doctors may hesitate to walk to it, in which case the chance of memory error increases (Yang, Wickens, Park, Fong, & Siah, 2015). A finding like this can be used to set up medical facilities so the chance of memory failure is reduced.

As this last example shows, lives can be saved by knowing about the trade-offs between memory and movement. In general, knowing more about how actions are generated can lead to safer environments and a more enlightened, inclusive body of research in the many fields where movement science and cognitive science come together. If the science of the mind is to advance, researchers will have to seize the opportunity to study how mental events are physically realized. Many useful strands are currently available to make this move possible. The time for action is at hand.

Author note

The work reviewed here benefited from generous support from the National Science Foundation, the National Institutes of Health, the German National Science Foundation, the Dutch National Science Foundation, the Air Force Office of Scientific Research, the Guggenheim Foundation, and the University of California, Riverside, Committee on Research. We thank Kornelia Kolig, Timothy Welsh, Howard Zelaznik, and an anonymous reviewer for helpful suggestions.