1 Representation of Motor Action

The representation and simulation of motor acts has a long and varied history in psychology and movement science. Johann Friedrich Herbart related movements to perceptual effects as early 1825 and proposed that the imagery of perceptual effects can elicit the related movements (p. 464). William James wrote some years later (1890, p. 526) “that every representation of a movement awakens in some degree the actual movement …” These and other approaches of an ideomotor understanding of human action fell out of fashion in the era of behaviorism (for a review, see Shin et al. 2010). However, around 100 years after James they became an important reference point for many experimental approaches, for example, ideomotor action (e.g., Knuf et al. 2001; Kunde 2001; Kunde et al. 2004; Koch et al. 2004), common coding (Prinz 1987, 1997), anticipative behavioral control (Hoffmann 1993; Hoffmann et al. 2004, 2007), theory of event coding (Hommel et al. 2001) and cognitive architecture of action approach (Schack and Hackfort 2007; Schack and Mechsner 2006; Schack and Ritter 2009, 2013). These action driven approaches emphasize the goal-directedness of actions, the importance of anticipated perceptual effects, and the crucial role of mental representations in action control, and thus share the idea that voluntary movements might be planned, performed, and stored in memory by way of representations of anticipated effects. Skillful coordination occurs if appropriate mental representations of the motor task and action goals are stored in memory, because cognitive representations govern the tuning of motor commands and muscular activity patterns. (for an overview of action science approaches, see Herwig et al. 2013).

Some of the dynamical systems approaches try to explain biological movements without alluding to cognitive levels (or internal models) while others explain cognition from a non-representational point of view (e.g. Seifert et al. 2017). Bernstein (1947) envisaged a complex architecture of human movement control ranging from “low” levels corresponding to involuntary movements, up to “high” cognitive levels that can be thought of as involving conceptual representations. The lowest level ist the level of muscle tone, responsible for background tension oft the skeletal muscle system. The second “lowest” level corresponds to synergistic processing, and this level has often been referred to in dynamical systems approaches (e.g., Ijspeert et al. 2002; Wolpert et al. 1995). We note that spinal (e.g., Poppele and Bosco 2003; D’Avella and Bizzi 1998) and some muscle synergies do not always require input from higher cognitive levels (e.g., Debicki and Gribble 2005). Such aspects of involuntary movements are often addressed in sensorimotor models of motor control (e.g., Todorov 2004; Kawato 1999). These processes run mostly automatically but can reach conscious levels if attention is directed towards them. In stark contrast to “low” levels, our understanding of cognitive movement control is far less advanced. Therefore, in this chapter, we focus on the cognitive dimensions and levels of human motor control (cognitive action architecture approach), and suggest that action representations should be differentiated from different types of action control (goal-oriented vs automated, cf. Schack 2004a, Schack and Ritter 2009, Schack et al. 2014). From that perspective involuntary motor control (e.g., reflexes and postural control) are also critically important as sensorimotor loops. There are many opportunities of functional cooperation within the whole action architecture (for an overview see: D'Avella et al. 2015, Table 1).

Table 1 Levels of motor action (modified from Schack and Ritter 2009)
Full size table

In such a context, Mylopoulus and Pacherie (Pacherie 2008; Mylopoulos and Pacherie 2019) discussed different levels and types of intentions within their (DPM) action model. From their perspective, action control is based on coordination between different levels of intentions and activity: distal, proximal and motor intentions, with distinctive roles in the guidance and control of actions. While distal intentions relate to reasoning and have a propositional format, proximal intentions integrate conceptual content with perceptual and memory information about the situation and one‘s motor repertoire. Motor representations, according to the DPM action model, with the functional role of „bringing about the actions they represent “(Mylopoulos and Pacherie 2019, p. 4). While originating from a philosophical background, this approach links well to the cognitive architecture of skilled action (Schack 2004a; Schack and Hackfort 2007), its idea of a hierarchical organization of motor action from mental to sensorimotor levels, and the central role of mental representations for skilled action.

2 Cognitive Architecture of Skilled Action

Prior to the current perspectives on anticipatory motor control, Bernstein had already pointed towards the large number of degrees of freedom in the human motor system, the need for continuous processing of sensory feedback to control this highly redundant system, and the importance of the anticipation of movement effects for movement organization. Bernstein (1947) proposed a model of the construction of movements according to which different organizational (and evolutionary) levels interact to generate and control different types of movement. These levels are thought to interact not simply in a fixed hierarchical manner, but their mode of interaction and hierarchical organization depends on the type of movement task and the level of expertise of the person.

As we know from actions in everyday life or from skilled and complex action in sports, certain aspects of our motor actions are automated and running without explicit attention, while others are goal directed and anticipative. Automatization (without explicit control) of actions occurs in the context of special stimuli (e.g., catching an object if it is thrown close to our body). The idea that movement control is hierarchically organized has been approached from different perspectives. One perspective focuses on a hierarchy of differing levels of representation (see, e.g. Jeannerod 2004; Rosenbaum 1987), while another perspective focused more strongly on the aspect of a hierarchical execution regulation (e.g. Keele et al. 1990; Marken 2002). The model proposed next, views the functional construction of actions (Schack and Hackfort 2007; Schack and Ritter 2009) on the basis of a reciprocal assignment of performance-oriented regulation and representational levels (see Table 1). These levels differ according to their central tasks on the regulation and representation levels. Each level is assumed to be functionally autonomous.

Both control levels, the level of sensorimotor control (I) and the level of mental control (IV), serve the main function of action control and regulation, whereas the level of sensorimotor representation (II) and the level of mental representation (III) are representational, and are closely connected to the two regulation levels. While the regulation level of mental control (IV) is induced intentionally and is relevant for the anticipation of action effects, the regulation level I is perceptually (externally) induced and is controlled by recent environmental stimuli. The level of mental control is responsible for volitional control while the level of sensorimotor control (I) is based on movement primitives and responsible for automatization. If we imagine a sport task like alpine skiing, it is easy to understand that particular elements of motor control are automatized (level I) like rhythmical swing moves. However, they can also be controlled voluntarily (level IV) in the case of difficult environmental circumstances or in the case of re-learning. Relevant modality-specific (e.g., kinesthetical, visual) information representing the effects of the particular movement is stored on the level of sensorimotor representation (II). Mental representations of sensorial effects and movement structures are located within the level of mental representation (III), and are based on the conceptual building blocks of action (Basic Action Concepts, BACs) that will be described in the following section.

3 Building Blocks of Action

It is a well-established idea in cognitive psychology and indeed it has received growing acceptance in the fields of motor control and sport psychology that actions are mentally represented in functional terms as a combination of the executed action and the intended or observed effects (Jeannerod 2006; Koch et al. 2004; Knuf et al. 2001; Hommel et al. 2001; Prinz 1997). The link between movements and perceptual movement effects is bi-directional and is thought to be stored hierarchically in long-term memory (LTM). Such movement representations are necessary because complex movements are unlikely to rely solely on online calculation due to human resource limitations. Rosenbaum and co-workers (Rosenbaum et al. 2007) demonstrated that movements can be understood as a serial and functional order of goal-related body postures, or goal postures, and their transitional states. That is, movements can be understood as the changes between body postures. Whereas body postures (keyframes) are represented in detail, the interframes (i.e., the movements between body postures) contain only differences between two successive keyframes. The better the order formation within cognitive movement representations, the more easily information can be accessed and retrieved (Schack and Ritter 2009). This leads to increased motor execution performance, which reduces the amount of attention required for successful performance (Beilock et al. 2002; Land et al. 2014; Raab and Johnson 2007). The nodes within such networks of movement representation contain functional subunits or building blocks that relate motor actions and associated perceptual effects.

Researchers from different fields, such as cognitive psychology, cognitive robotics and sport psychology (Maycock et al. 2010; Schack 2004a, b; Schack and Mechsner 2006; Schack and Ritter 2009; Tenenbaum et al. 2009), have provided evidence for so-called basic action concepts (BACs) in the control of human movements. Analogous to the well-established notion of basic object concepts (Rosch 1978), BACs are the mental counterparts of functional elementary components of complex movements. They can be thought of as the cognitive chunking of body postures and movement events concerning common functions in realizing action goals. BACs do not refer to behavior-related invariant properties of objects, as in the case in basic object concepts, but to perceptual invariant properties of movements. According to the cognitive action architecture (CAA) approach (Schack 2004a, b, 2010; Schack and Hackfort 2007; Schack and Ritter 2009), mental representations are thought to comprise such representational units (i.e., BACs) and their structural composition in relation to one another. Therefore, BACs relate to functional and perceptual dimensions of movements and body postures and are clearly linked to sensory movement effects. In this respect, BACs differ from perspectives that conceive motor programs as a prestructured set of muscle-oriented commands and which assume that movements are initiated without regard to sensory information (see Schmidt and Lee 1999).

4 Measurement of Mental Representation in Sport

A wide range of methods was used to study mental representations’ structures in long term memory (for an overview see Hodges et al. 2007; McPherson and Kernodle 2003). One of the first studies in sports was reported by French and Thomas (1987). They assessed various components of basketball performance (e.g., control of the basketball and cognitive decisions, dribbling and shooting skills), along with declarative knowledge in 8- to 12-year-old children. Declarative knowledge was measured using a paper-and-pencil test. Additionally, specific sorting techniques and interview methods have been used to confirm expertise-dependent differences in the classification and representation of context-specific problem states in springboard divers, judokas, triathletes, and weight lifters (Huber 1997; Russell 1990). Studies using categorization tasks have shown that experts classify problems according to underlying functional principles, whereas novices operate more strongly with superficial features (e.g. Allard and Burnett 1985; Russell and Salmela 1992). Furthermore, questionnaire methods and interviews have revealed the structure and organization of movement knowledge in, for example, tennis (McPherson and Thomas 1989, McPherson and Kernodle 2003), Volleyball (McPherson and Vickers 2004), and basketball (French and Thomas 1987). These methods facilitate the study of knowledge-based mental representations of movements in LTM (for an overview, see Hodges et al. 2007; Schack 2012). However, most of them focus on explicit knowledge and are non-experimental (e.g., interviews, questionnaires, paper-and-pencil tests). Therefore as an experimental method that avoids introspective statements, structural dimensional analysis of mental representation (SDA-M) method has been introduced (Schack 2004a, 2012). This method provides psychometric data on mental representations of complex movements and as such permits investigating the status and change of structures of mental movement representations.

In detail, the SDA-M (Schack 2012) maps mental representations as integrated networks of basic action concepts (BACs) across both individuals and social groups, by providing information on relational structures in a given set of concepts with respect to goal-oriented actions. The internal grouping of conceptual units (i.e., the clustering of BACs) delineates the structure of the knowledge representation of a certain movement. While mental representation structure refers to the relation and the grouping of BACs in LTM, learning can be considered as the modification of the mental representation structure over time.

The SDA-M consists of four steps (for details, see Schack 2012). First, a split procedure involving a multiple sorting task (pair-wise comparisons) delivers a distance scaling between BACs of a suitably predetermined set. Specifically, during this procedure, one concept of a given set of basic action concepts is permanently displayed on a computer screen (anchor concept) and all other concepts are compared to that anchor concept successively. Participants have to decide whether the two given concepts are related to each other during movement execution. The procedure continues, until all concepts have been compared to all other concepts. Second, a hierarchical cluster analysis is used to transform the set of BACs into a hierarchical structure. Third, a factor analysis reveals the dimensions in this structured set of BACs, and fourth, the cluster solutions are tested for invariance within or between groups.

As a result, one obtains the individual partitioning of the BACs in hierarchical tree-like structures, the so-called dendrograms (see Fig. 1). Cluster solutions are calculated for all individual participants and for groups. Each cluster solution is established by determining a critical Euclidean distance dcrit (marked by the solid horizontal line in Fig. 1). The critical value dcrit depends on the number of concepts. All junctures below the value dcrit are considered related, while the junctures above this value are considered unrelated. This results in a cluster solution. In an high level expert structure, the resulting cluster solution represents mostly the functional phases of the movement.

Fig. 1
figure 1

Dendrograms for the experts (a) and non-players (b) based on the hierarchical cluster analysis of BACs in the tennis serve. The horizontally aligned numbers denote the BACs (for the code, see text), the vertical numbers, the Euclidean distances. For every group, it holds n = 11; p = .05; dkrit = 3.46. A tennis serve consists of three distinct phases, each of which fulfils distinct functional and biomechanical demands. First, in the pre-activation phase, body and ball are brought into position, and tension energy is provided to prepare the strike. The following BACs were identified: (1) ball throw, (2) forward movement of the pelvis, (3) bending the knees, and (4) bending the elbow. Second, in the strike phase, energy is conveyed to the ball. The following BACs were identified: (5) frontal upper body rotation, (6) racket acceleration, (7) whole body stretch motion, and (8) hitting point. Third, in the final swing phase, the body is prevented from falling, and the racket movement is decelerated after the strike. The following BACs were identified: (9) wrist flap, (10) forward bending of the body, and (11) racket follow through. [Reprinted from Schack and Mechsner (2006), with permission]

Full size image

A good example to investigate the mental representation structures of a complex movement on different levels of expertise is the tennis serve (Schack and Mechsner 2006). For a tennis serve, many degrees of freedom have to be controlled in the musculoskeletal system, and the movement execution depends considerably on training and expertise. On the other hand it is a finite and recognizable action pattern of which the overall structure is well defined by biomechanical demands.

Experts in that study (n = 11 male tennis players; mean age, 24 ± 3.7 years) from the upper German leagues were ranked between 15th and 500th in the German men’s rankings, while the non-players (n = 11 males; mean age, 24 ± 6.7 years) had no tennis experience (maximum 5 h) and had never had any tennis lessons.

The single BACs and the adequate functional organization of the tennis serve were characterized in advance in collaboration with non-players, athletes with different levels of expertise, and coaches (as a combination of interviews and task analysis). Photographs of the evaluated tennis sub-movements were presented to experts and non-players together with linguistic markers of varying generality. The picture-word combination the appropriateness of which took the shortest time to judge was chosen, in analogy to classical methods (Rosch 1978).

Each BAC was characterized by a set of closely interconnected sensory and functional features. For example, BAC 7 (whole body stretch motion) is functionally related to providing energy to the ball, transforming tension into swing, stretching but remaining stable. Afferent sensory features of the corresponding sub-movement that allow monitoring of the initial conditions are bent knees, tilted shoulder axis, and body weight on the left foot. Re-afferent sensory features that allow monitoring of whether the functional demands of the sub-movements have been addressed successfully are muscles stretched and under tension, proprioceptive and, finally, visual perception of the swinging arm and ball in view.

Figure 1a depicts the group dendrogram for the experts. Their cognitive structure was very similar to an optimal cluster solution, matching the functional and biomechanical demands of the tennis serve. The three functional phases (i.e., pre-activation, strike, and final swing) form clearly separated clusters in the dendrogram. An invariance analysis (step four of the SDA-M) confirmed this interpretation. There was no significant difference between the cognitive BAC framework in experts and the biomechanical demand structure of the movement (presented as a hierarchical subtask structure). In contrast, the clustering of the BACs in the dendrogram of the non-players (Fig. 1b) did not mirror the functionally and biomechanically demanding phases so well. The BACs were less clearly grouped, with no close neighborhoods, and the partial clusters largely failed to reach significance. Moreover, the average novice structure differed significantly from the optimal cluster solution. In sum, in experts, these representational frameworks were organized in a distinctive hierarchical tree-like structure, were remarkably similar between individuals, and were well-matched with the functional and biomechanical demands of the task. In contrast, action representations in low-level players and novices were organized less hierarchically, were more variable among persons, and were less well-matched with functional and biomechanical demands.

More generally, if two BACs are frequently classified by participants as being “functionally related” during the split procedure, these BACs are characterized by a small Euclidean distance which is reflected in a low projection of the BACs on the vertical axis in the dendrogram (e.g., BACs 1 & 2 in Fig. 1a). If two BACs are not judged to be “functionally related,” the Euclidean distance is larger (a bigger number on the y-axis) and the projection of the two BACs is high in the tree diagram (e.g., BACs 9 & 10 in Fig. 1a).

In order to measure the inter-individual or inter-group differences between representation structures, a structural invariance measure λ is determined based on (1) the number of constructed clusters of the pair-wise cluster solutions, (2) the number of concepts within the constructed clusters and (3) the average quantities of the constructed clusters. The invariance measure λ ranges from 0 (no similarity at all) to 1 (tree diagrams are identical). Two cluster solutions (or representation structures) are considered to be invariant (i.e., the same) if λ > λcrit = .68 (which corresponds to a significance level of α = .05; for more detailed information, see Schack 2010, 2012).

Furthermore, as shown in a volleyball study (Schack 2004b), these mental representation structures are position- and task-dependent. Such representation structures are the outcome of an increasing, effort-reducing formation of order in LTM (see Figs. 1, 2 and 3). BACs are more and more embedded in a tree-like structure, with branches evolving from BACs which are functionally related to each other, representing functional movement problems such as „generating energy “or „transforming tension into swing”. This is the case in experts (see Fig. 1) and during motor learning in the context of increasing performance (Frank et al. 2013). In addition to the functional representation structure developing, a more appropriate temporal structure of the movement evolves in the representation based on the temporal order of relevant body postures and movement events. Altogether with increasing expertise, the representation of the movement corresponds more and more to the topological (spatiotemporal) structure of the movement itself. Within the course of learning and generating expertise there seems to be an increasing overlap between the representation structure (clusters) and the biomechanical structure of the movement (in terms of kinematics and functional phases). The more differentiated and functionally valid the structure of representation, the more appropriate are the anticipated and expected movement effects. Based on such memory structures, athletes are better able to read and understand propriceptive, auditive or visual information generated from the movement itself and indicating ongoing movement effects. Accordingly, movement control becomes possible by representing the anticipated perceptual (“simulated”) effects of the movement and comparing them with incoming action produced (“real”) perceptual effects. From that perspective, the balance between the expected and the incoming effects becomes the basis for ongoing movement corrections and functional adaptations within the whole action architecture.

Fig. 2
figure 2

Mean group dendrograms of the practice group (N = 12) for the golf putt at a pre-test and b retention-test. The control group is not shown. The numbers on the x-axis relate to the BAC number, the numbers on the y-axis display Euclidean distances. The lower the link between related BACs, the lower is the Euclidean distance. The horizontal dotted line marks dcrit for a given α-level (dcrit = 3.41; α = .05): links between BACs above this line are considered unrelated; horizontal grey lines on the bottom mark clusters. BACs: (1) shoulders parallel to target line, (2) align club face square to target line, (3) grip check, (4) look to the hole, (5) rotate shoulders away from the ball, (6) keep arms-shoulder triangle, (7) smooth transition, (8) rotate shoulders towards the ball, (9) accelerate club, (10) impact with the ball, (11) club face square to target line at impact, (12) follow-through, (13) rotate shoulders through the ball, (14) decelerate club, (15) direct clubhead to planned position, and (16) look to the outcome. [Reprinted from Frank et al. (2013), with permission]

Full size image
Fig. 3
figure 3

Top panel: End-state comfort satisfaction (proportion of children) in the bar transport task for the simple (white bars) and advanced planning condition (grey bars). Middle panel: The representational structures for comfortable and uncomfortable grips for the 7-year old children, the 8-year old children and the 9-year old children. Depicted are the averaged group data based on the SDA-M analysis. Bottom panel: Images of the stimuli used in the SDA-M task. Pictures 1 to 6 depict comfortable grasp postures, and pictures U1 to U3 depict uncomfortable grasp postures. [Reprinted from Stöckel et al. (2012), with permission

Full size image

5 Studies about Mental Representation in Skilled Action

Several studies in sport (e.g. Frank et al. 2013; Schack 2004a, b; Schack and Bar-Eli 2007; Schack and Hackfort 2007; Weigelt et al. 2009), dance (Bläsing et al. 2009; Bläsing 2010), movement rehabilitation after stroke (Braun et al. 2007), music (Schack 2004b), walking (Stöckel et al. 2015) and cognitive robotics (Schack and Ritter 2009; Schack and Ritter 2013) used the SDA-M-approach to measure the structure and dimension of mental representation in motor action. Most of the studies in sport are done to investigate the nature and role of mental representation in skilled athletic performance, and to derive new technologies in mental training and technical preparation (Schack and Hackfort 2007; Schack and Bar-Eli 2007; Schack 2004a). Findings from studies across sports, such as golf (Frank et al. 2013), soccer (Lex et al. 2015), tennis (Schack and Mechsner 2006), wind surfing (Schack and Hackfort 2007), volleyball (Schack 2004b), climbing (Bläsing et al. 2014) and judo (Weigelt et al. 2011) show that the mental representation structures relate to performance. These representation structures are the outcome of an increasing and effort-reducing order formation in LTM. In high-level experts, these representational frameworks were organized in a distinctive hierarchical tree-like structure, remarkably similar among individuals, and well matched onto the functional and biomechanical demands of the task. In comparison, action representations in low-level players and non-players were organized less hierarchically and more variably among individuals.

In addition to comparisons of representations across levels of expertise, a practical implication from these measurements of mental representations in skilled performance arises from the fact that these representation structures can also be analyzed on an individual level. On the one hand, this provides the opportunity to gain knowledge about an athlete’s specific movement problems (individualized diagnostics, e.g. Weigelt et al. 2011), and on the other hand, it gives coaches a tool to provide performance-related instructions and to optimize mental and technical training, which goes (as an alternative) beyond the use of traditional methods such as non-standardized questionnaires or observation methods from biomechanics (i.e. measurement of movement kinematics).

Accordingly, the structure of cognitive representations in LTM is also relevant for perception and visuomotor control in motor action. However, little is known about the relationship of cognitive representations and visuomotor control for complex movements. Therefore, in a recent study we investigated whether cognitive representations of complex movements influence (unconscious) visual perception (Güldenpenning et al. 2011). Novices and skilled high-jump athletes were shown to differ in that only skilled athletes seem to have a functionally structured, cognitive representation of the high jump movement (Fosbury flop). Both groups were asked to classify pictures of body postures of the high jump movement. In a so-called priming paradigm, each of these picture presentations were preceded by another picture of a high jump body posture that could not be perceived consciously. Participants had to classify whether the second pictures in each trial showed a body posture from the approach or from the flight phase. Importantly, the two pictures in each trial could differ with regard to the shown movement phase but also in temporal order. That is, both pictures could reflect the natural order within or between movement phases or, alternatively, they could be presented in a reversed order (e.g., flight before approach). We found a main effect of temporal order for skilled athletes, that is, faster reaction times for picture pairs that reflected the natural movement order as opposed to the reversed movement order. Novices showed a qualitatively different data pattern which was in line with superficial processing of visual features unrelated to the high jump movement. These results suggest that the structure of cognitive movement representations modulates the visual processing of body postures. Temporal information seems to be an important dimension of such representations (cf. also Güldenpenning et al. 2013) and can be processed automatically as the extraction of temporal order information requires unconscious processing of (one of) the pictures.

To learn more about different types and formats of such motor representation we studied the relationship between represented activities and the body schema (Bläsing et al. 2010b). The body schema has been described as the multimodal representation of the body that integrates somatosensory, proprioceptive, vestibular and visual information. Therefore, we investigated the mental representation of body parts and related activities in two individuals with congenitally absent limbs. One of the individuals was born without forearms and legs, and perceived phantoms of all four limbs, while the other individual was born without hands and arms and has two shortened legs with only one functionally intact foot, but has never experienced any phantom sensation. We measured cognitive action and body representations of these two individuals and compared their results to those of a group of paraplegic wheelchair athletes, and of a group of individuals with physically intact limbs. Results of the group of physically intact subjects and the group of paraplegic subjects reflected modularity of the body schema (Haggard and Wolpert 2005), with separate clusters for the lower body, upper body, fingers and head. The representation structure of the participant with congenital phantom limbs more closely resembled the structure of the matched control group with physically intact limbs, than the structure for the participant with congenitally absent limbs but without phantoms. The results provide evidence for a strong interaction of cognitive motor representation and cognitive body representations on a functional level. It seems that body representation and cognitive integration of perceptual body information are strongly influenced by cognitive motor representations (Bläsing et al. 2010b).

Based on these and other studies (e.g. Giummarra et al. 2007; Haggard and Wolpert 2005), we argue that major interfaces in the architecture of movement are cognitive in nature (without fully denying the relevance of automated processes such as reflexes or postural control of the whole body). Such a perspective does not view the motor system as being distinct from cognition. Instead, it considers both conscious and automatized processes of movement organization to be based functionally on mental representation structures. This does not ignore the significance of emotional or motivational processes; it simply puts them aside in order to focus on the cognitive architecture of movement (Schack and Ritter 2009; Tenenbaum et al. 2009).

6 Mental Representation and Learning

Differences in the mental representation structure between novices, intermediates, and experts (Schack and Mechsner 2006; Bläsing et al. 2009) suggest that the structure of mental representations of complex movements changes alongside of improvements in skill level. More specifically, the structure of the mental representation of a given complex movement might develop towards the functional structure of an expert over the course of practice. Therefore, a novice’s unstructured representation of a movement is thought to develop into a more structured representation during motor learning. Accordingly, we assume learning to be a product of modifying the mediating mental representation structures in long term memory (see Schack and Ritter 2013).

To the best of our knowledge, there are only a few studies examining how mental representation structures develop during practice (see for an overview Schack and Land 2016). As it seems crucial to learn more about whether and when changes in mental representations occur and how they develop during learning, we examined structural changes in mental representations of a complex movement during early skill acquisition (Frank et al. 2013). The acquisition of the golf putting movement was investigated in a group of novice golfers. After a three-day period of practice with the task, in the post-test, the mental representation of the practice group was compared to that of a control group. As expected, the mental representation structure showed functional changes (i.e., functional clusters in the group’s dendrogram) in the practice group along with performance improvement while no such changes were observed in the control group. Specifically, the mental representation structure of the practice group changed over the course of practice and became more similar to an expert’s structure. As shown in Fig. 2, the practice group’s mean dendrogram revealed an increased number of functional clusters, with BACs being clustered into three functional units relating to distinct movement phases (i.e., movement preparation, the forward swing, and the impact phase). In contrast, no changes were evident in the mental representation structure of the control group which did not practice at all between SDA-M measurements.

Similar to practicing by way of execution, mental representations have been found to change functionally with mental types of practice such as imagery (Frank et al. 2014, 2016, 2018b; Simonsmeier et al. 2018) and observation (Frank et al. 2018a). From these findings, mental practice seems to promote the cognitive adaptation process during motor learning, leading to more elaborate representations. This suggests that mental practice affects the perceptual-cognitive levels of action organization at an early stage of motor learning and that way induces learning from within (Frank 2016; Frank and Schack 2017).

These findings suggest that the functional development of mental representations plays a significant role during motor learning, presumably, for the development of movement expertise. Further investigations from a number of different activities (e.g., golf, soccer, wind surfing, volleyball, gymnastics, and dancing) also support the functional relation between mental representation structures and performance and expertise (Bläsing et al. 2009, 2010a; Land et al. 2013; Schack 2004a; Schack and Hackfort 2007; Schack and Bar-Eli 2007; Velentzas et al. 2011).

7 Skilled Manual Action

In manual actions we find a most impressive integration of capabilities to shape physical interaction, comprising all levels of action architecture ranging from sensorimotor control to mental control and integrating all types of operations and representations. Today, highly developed anthropoids, humans, and robots are able to perform manual actions. Manual actions allow for a particular object-related interaction and represent cognitive operations (specifically, classifications). Based on grasping movements, anthropoids can identify physically different objects as functionally equivalent tools for solving typical behavioral problems (such manipulating a screw). As current studies show, human grasping movements are cognitively represented on the basis of movement concepts (e.g., Basic Action Concepts) and build on effect-oriented target codes (in relation to space rather than to the body; Rosenbaum et al. 2007; Schack 2004a, b; Schack and Ritter 2009, 2013).

Hence, the perceptual-cognitive control of arm and hand movements has become a topic of study in psychology, biomechanics, cognitive motion science, cognitive robotics, and bio-cybernetics. To understand the biomechanical and cognitive background of manual action in more detail, we established lines of research on the movement primitives of manual action, with the help of biomechanical and cognitive analyses. Because the production of manual actions is affected by such factors as biomechanical constraints (Rosenbaum et al. 2001; Weigelt et al. 2006), we have carried out some developmental studies on the relationship between biomechanical constraints like the end-state-comfort effect and the cognitive representation of grasping postures in children (Stöckel et al. 2012). Other studies have explored the link between semantic and motor memory, especially in the case of grasping (Weigelt et al. 2009).

To strengthen links between experimental research and robotics, we have studied cognitive representations not in isolation but as parts of a cognitive action architecture (Maycock et al. 2010; Schack and Ritter 2009, 2013). Such a perspective has consequences for the design of experimental studies and other lines of reseach. For instance, until recently motor planning has been studied in situations where participants carry out physical actions without a particular purpose or overarching movement goal. Yet, when one examines everyday life, it is apparent that our physical interactions are normally planned and executed with higher-order goals in mind. Therefore, we examined whether two previously discovered motor planning phenomena – the end-state comfort effect and motor hysteresis – would hold up when the actions were carried out in the service of higher-order goals. As we claimed before, the end-state comfort effect is the tendency to avoid awkward postures at the end of a movement (Rosenbaum et al. 1990, 2001). Motor hysteresis is the tendency of the motor system to reuse the former motor plan and thus to persist in the former movement state. In binary tasks (e.g., Weigelt et al. 2009), this results in switching from one state to another at different values (e.g., drawer heights) depending on the movement history (e.g., on sequence direction).

The higher-order goal we chose to study was memorization. In focusing on memorization, we asked not only how motor planning is affected by the need to memorize, but also how memory performance might depend on the cognitive demands of motor planning. We asked students to retrieve cups from a column of drawers and memorize as many letters as possible from the insides of the cups. The end-state comfort effect and motor hysteresis were replicated in these conditions, indicating that the effects hold up when physical actions are carried out for the sake of a higher-order goal. However, one of the most reliable effects in memory research was eliminated, namely, recency effect. Recent items were not recalled better than items encountered earlier. This outcome was not an artifact of memory being uniformly poor, because the tendency of initial items to be recalled better than items in the middle of the sequence – the primacy effect – was obtained. The recency effect did not disappear because participants had to recall items in their correct order, for it was also eliminated when the items could be recalled in any order. These and other aspects of our results support recent claims for tighter links between basic mechanisms of perceptual-motor control and cognitive (symbolic) processing (representation of verbal information) than have been assumed in the past, with researchers mostly assuming seperate accounts for perceptual-motor and verbal performance (for more information, see Weigelt et al. 2009). These results indicate a link between the increasing cognitive demands for motor planning in the context of motor hysteresis (reuse of former motor plans) while opening the drawers and the memorization of verbal letters. This conflict leads to measurable effects at the end oft the task, where a switch of the grasp type is required. This cognitive replanning impairs memory capacity and the recency effect gets lost. The results support the idea that the interaction of cognitive and perceptual-motor components in human action works at different levels and is part of an overall architecture (e.g., Spiegel et al. 2012, 2013).

Linking mental representation to motor planning in manual action, Weigelt and Schack (2010) showed that the end-state comfort effect develops gradually with the sensory-motor maturation of children. The children performed a dowel placing task, reaching for a horizontal dowel and inserting one of its ends into a target disc. Although all children, regardless of age, reached for the dowel with an overhand grasp when this resulted in a comfortable end-state (i.e. thumb-up posture), a different pattern emerged when an underhand grip had to be selected. In this latter situation, only 18% of the 3-year-olds, 45% of the 4-year-olds, and 67% of the 5-year-olds used an underhand grip and finished the action in a comfortable end-state. These results show a distinct pattern of gradual improvement in children’s sensitivity to end-state comfort across the three age-groups. Such information about basic principles of end-state comfort planning in humans is of interest when designing valid robot architectures that are used to perform manual actions and interact with human beings in an appropriate, age-dependent manner (Gienger et al. 2008).

Stöckel et al. (2012) investigated anticipatory motor planning and the development of mental representations of grasping postures in children aged 7, 8, and 9 years. Overall, 9-year-old children were more likely than 7- or 8-year-olds to plan their movements to end in comfortable postures, and to have distinct representational structures for certain grasp postures. Additionally, sensitivity to comfortable end-states was related to the mental representation of certain grasp postures. Children with functionally well structured representations related to grasp comfort were more likely to have satisfied end-state comfort in both the simple and the advanced planning condition. In contrast, children whose cognitive representations were not structured by grasp comfort achieved far less satisfaction with grasp comfort in the advance planning condition. Using the SDA-M method, no distinct clusters were found for comfortable and uncomfortable grips in 7- and 8-year–olds (Fig. 3), indicating a limited cognitive capacity to distinguish between awkward and comfortable postures. In contrast, the 9–year-old children showed distinct clusters for comfortable and uncomfortable grips with uncomfortable postures represented very close together in long-term memory and representational structures near to an ideal categorization. The differences between the 7- to 8- and the 9-year-old children proved to be significant. Hence, the representational structures indicate that only the 9-year-old children represented the differences between comfortable and uncomfortable grasping postures efficiently in memory.

The results of this study support the notion that cognitive action representation plays an important role in the planning and control of grasp postures. Furthermore it shows that dexterity in manual action and task related object manipulation is accompanied by order formation in memory. Such order formation in action knowledge reduces the cognitive effort required to activate relevant information. From this perspective, we have to solve movement tasks purposefully and in a step by step fashion within the framework of voluntary organization of movement. Hence, we assume that advanced anticipatory motor planning requires a well-established cognitive representation of the respective task with its constraints and a variety of possible solutions (e.g., hand postures). Advanced anticipatory motor planning (as evidenced by the end-state comfort effect) seems to depend strongly on children’s hands-on experiences, which are necessary to establish efficient cognitive representation structures in long-term memory (Fig. 4).

Fig. 4
figure 4

The evolution of survival tools. From a 800,000-year-old biface (left) to a recent multilayer damast steel knife with a mammoth molar grip - handmade by the first author (right). Photograph: Gregor Panic

Full size image

From a broader perspective concerning skilled manual actions the development and the usage of technologies is associated with changes in the profile and spectrum of basic human action skills (Suelzenbrueck et al. 2011). Suelzenbrueck and colleagues (Suelzenbrueck et al. 2011) investigated such effects in the context of handwriting. To learn more about skilled action control in manual action we started to investigate the history, development and representation of former (recently lost) Basic Action Skills, like different types of skilled actions in forging, tool making and traditional survival skills (like bow drill fire making). Interestingly, early human beings were not only able to apply tool, they could also produce them for specific purposes. This led to new framing conditions for the further development of the species and for individual survival. The oldest known stone tools come from the Olduvai Gorge in Tanzania and are about 2 million years old. The production of such tools meant that not only the goal but also a chain of action steps and related operations had to be stored in memory (Klix 1993; Klix and Lanius 1999). It had to be possible to compare the goal with the sensory and functionally detectable effects of the tool produced. This led knowledge to acquire a new kind of functionality in skilled action.

In the context of evolutionary tool making it has been possible to confirm the formation of cognitive concepts in manual action (Rensch 1973). By reconstructing the tools and products of former cultures it is possible, from our perspective, to study and understand the involved cognitive, perceptual and motor related operations and the architecture and representation of skilled actions in more detail. This may help to understand and reflect the technology and culture dependent changes in the spectrum of Basic Motor Skills and the consequences for the development of perceptual-cognitive skills and skilled action.

The specific methods presented in this paper make it possible to take essential information regarding the underlying cognitive-perceptual action system into account, while still addressing the individual needs of an athlete, dancer or a scholar in different types of manual or sport action in a better way. Furthermore, the theoretical perspective on the cognitive architecture of action developed here, and the accompanying methods (technological steps) are not just relevant for optimizing the daily work of the sport psychologist or dance coach, but are also useful in opening up new perspectives to modify classical approaches of technical preparation and mental training in sport and rehabilitation.