Introduction

Research in Artificial Intelligence in Education (AIED) aims to develop flexible systems that will increase access to effective, personalized and engaging, anytime, anywhere learning throughout lifetimes across the full range of knowledge domains and skills and employing varied pedagogical approaches (Underwood and Luckin 2011).

Under this umbrella, this paper focuses on the psychomotor learning domain, which deals with physical movement, coordination and the use of the motor skills areas (Harrow 1972). Motor skills refer to bodily movements involving muscular activity (Gagné and Briggs 1979). More precisely, motor skills involve goal-oriented physical actions or tasks requiring voluntary body and/or limb movements to achieve the goal (Magill 1993). Thus, motor skills learning can be defined as achieving the ability to perform a function acquired with practice that requires body and/or limb movement to accomplish the goal of an action or task (Christensson 2005). It is about “doing” through imitation, practicing and habituating new skills (Penney 2011). Motor skills can be classified in three dimensions (Magill 1993):

  1. 1)

    precision of the movement, which considers i) gross motor skills that involve large musculatures and a goal where the precision of movement is not as important, and ii) fine motor skills that require control of small muscles of the body to achieve the goal of the skill and generally including hand-eye coordination;

  2. 2)

    defining the beginning and end points of the movement, which considers i) discrete motor skills that have clearly defined beginning and end points, ii) serial motor skills that put together several discrete motor skills, and iii) continuous motor skills in which the performer of the skill determines the beginning and end points; and

  3. 3)

    stability of the environment, which in a four category system combines i) the change or not from one execution to the next one, with ii) the execution either stationary or in motion.

Consolidating specific motor tasks into memory through repetition (thus, creating long-term muscle memory for a given task), is very relevant in diverse scenarios that support learning processes involving not only brain activity, but also physical activity, such as playing a musical instrument, handwriting, drawing, training for surgery, improving the technique in sports and martial arts, learning sign language, dancing. In these situations, learners have to train by repeating very specific movements till they learn the best way to carry them out effectively without conscious effort (Krakauer and Shadmehr 2006). The automatization of skills is very much shared between cognitive and motor skills, so in essence, all of the work being done in AIED over the last 30 years on cognitive skills acquisition, that can go back to (Anderson 1982), is worthy of reinvestigation with motor skills.

For that matter, learning motor skills goes beyond mere muscle memory, blending motor skills with cognitive (including meta-cognitive) and affective skills. Hence, research in AIED must build on existing cognitive (i.e., tell what to do when the learner answers something wrong), meta-cognitive (make learners aware of their learning process and thus, produce their own learning strategies) and affective research (turning negative emotional states into productive learning experiences). The cognitive, meta-cognitive and affective dimensions are popular focus areas in AIED. However, as discussed in this paper, there have not yet been corresponding AIED developments addressing the psychomotor learning domain, thus supporting the personalized learning of motor skills. Therefore, the focus of this paper is mainly on how the physical part related to muscle training can be supported from an AIED perspective, both in the modelling of the learner’s physical interaction and in the provision of personalized support during the training. This implies that the physical actions carried out while training are to be monitored in real-time, compared with experts’ performance and, when needed, corrected through appropriate feedback, in order to achieve successful motor skills learning (i.e., skills learning at a physical level).

Feedback is crucial in motor skills learning (Swinnen 1996). For example, learners do not often recognize motion errors, and experts need to provide objective advice to learners (Iwasako et al. 2014). Traditionally, a human instructor or coach is used to support motor skills learning (Wulf et al. 2010) following the apprenticeship model (i.e., learning by observation of an expert) (Christensson 2005), which can also be done using multimedia content (Kwon and Gross 2005). Methods include (van der Linden et al. 2011): i) verbal instructions, ii) physically guiding the learner’s limb movements, and iii) allowing learners to observe themselves in a mirror. This training approach requires a high degree of expertise in identifying how close the movement performed resembles the ideal one, especially when movements are complex and can be performed with different level. There are also several limitations when a human attempts to provide feedback (Rauter et al. 2011): i) a human cannot hold the same level of attention and concentration over a longer period of time, ii) a human is not able to observe all important physiological and biomechanical variables characterizing the movement, neither at the same time nor at high precision, and iii) a human can hardly provide augmented concurrent feedback (i.e., external stimulus during the movement in real-time). In addition, the process of learning by imitation can be improved with multimodal feedback systems (Portillo-Rodriguez et al. 2008), which are systems that provide feedback through different sensorial channels (i.e., visual, auditory and haptic).

Moreover, according to (Christensson 2005), personalized motor skills learning environments that follow a learner-centric and constructivist (learn-by-doing) approach are needed. In this way, the system can be challenging and supporting at the same time in order to properly engage and motivate the learner. Hence, it can adapt to each individual needs and progress, providing personalized feedback meanwhile the learner is performing the tasks, and changing the instructional strategy when needed (e.g., guide, teach, iterate, challenge).

Because there are multiple novel interactive technologies that can help make personalized motor skills learning possible, this paper argues that now is a good time to introduce motor skills learning as a main research direction in the AIED community. A variety of emerging resources are primed to support this new direction, including:

  • novel types of quantified-self Footnote 1 wearable devices from the so called “Internet of Me” that allow collecting data from the user in a non-intrusive way, such as with smart bracelets, watches, t-shirts, etc.,

  • techniques for big data analysis that deal with volume, variety, velocity, variability and veracity,

  • 3D modelling of living physical objects by developing a mathematical representation of their three-dimensional surface,

  • 3D printing of exoskeletons for motor control rehabilitation, and

  • augmented sensorial output with ambient intelligence (i.e., environments that are sensitive and responsive to the presence of people).

These technologies can enrich AIED research in order to develop procedural learning environments that can physically support -and when appropriate guide- the particular way each individual learner should move her body in order to achieve specific motor learning goals (i.e., learning goals that are related to improving motor skills acquisition). As these technologies are of much interest in the do-it-yourself (DIY) movement,Footnote 2 the AIED research community can take advantage of the resources and support already available online, thus reducing the learning curve for their integration in AIED systems. In addition, educators need to be involved in what is known as an elicitation process to help them identify the personalized support to be provided to learners training their motor skills. TORMES methodology (Santos and Boticario 2015) can facilitate this elicitation process.

The rationale for pursuing this new approach in the AIED field is that the learning of motor skills would benefit from the individualized instruction and support that it is the purpose of AIED to provide. The acquisition of motor skills poses familiar AIED problems, but in novel and challenging new forms, such as i) modelling correct (expert) motor movements, ii) diagnosing students’ movements, and iii) deciding the personalized intervention (i.e., what feedback to give and when and in what form, etc.). Emerging novel interaction technologies seem to fill the technological gap to make personalization possible in motor skills learning. In addition, service-oriented architecture approaches aimed to extend learning environments with adaptive navigation support (Santos and Boticario 2011) need to take into account recent trends in service-oriented and ubiquitous computing, moving to an ecosystem of standard-based interactive learning tools (Nye 2015). Figure 1 depicts aforementioned elements that are foreseen to frame the AIED research in the coming 25 years aimed to develop systems that can provide individualized instruction and support in motor skills training.

Fig. 1
figure 1

Elements for providing individualized instruction and support in motor skills training

Full size image

As preliminary work to this paper, a discussion on how the training of the Aikido martial art can benefit from an AIED procedural learning environment for personalized motor skills training (Santos 2015) was presented at the workshop “Les Contes du Marriage: Should AI stay married to ED?” that was organized during the AIED 2015 conference to examine the current and future identity of the AIED field.

The contents of this paper are structured as follows. First, a review of papers from the International Journal of AIED (IJAIED) is provided, which reveals a dearth of AIED research on personalized motor skills learning (despite a few examples that tackle related issues) inside the community. Next, a non-exhaustive list of motor skills learning systems from outside the AIED field are reviewed, revealing a lack of attention to personalization features. After that, challenges and opportunities for the AIED research community are identified, when considering emerging interactive technologies to build personalized systems for motor skills learning.

Motor Skills Learning Inside the AIED Community

Limited attention to physical and motor aspects of learning has been carried out in the AIED community so far. This is likely due to the lack of appropriate technological support available in the past for movement detection at low cost and with low intrusion. In order to provide some background of the AIED field regarding personalized support for motor skills learning (and since no specific IJAIED papers addressing motor skills personalized learning were found in the review), this section compiles articles published in IJAIED that tackle related issues from the procedural learning perspective. In particular, the acquisition of procedural skills has been addressed since the eighties and early nineties in the AIED field, primarily in the context of Intelligent Tutoring Systems (ITS) in closed domains (Andriessen and Sandberg 1999). Typically, these approaches involved comparisons between learner answers and experts’ by (Ohlsson 1993): 1) observing learners’ actions, 2) measuring progress in terms of correct problem solving steps, and 3) applying existing and well-founded theoretical notions on skill acquisition.

This review of IJAIED papers starts with some historical issues regarding procedural learning that arose during the discussion carried out among Schank and Edelson (1989/1990) and Eggert (1990) on their different viewpoints regarding the role for AIED, and how technology can be used to reshape education. In particular, Schank and Edelson proposed and discussed the benefits of simulation as a relevant instructional approach for computer-based education since it is realistic enough to facilitate learning-by-doing (through failure-driven learning) in domains that are too risky or expensive to be trained in, allowing also the learner to revise her hypothesis based on her observations. In addition to engaging the learner in the learning process thanks to the direct experience while acquiring knowledge, these authors highlighted that simulation environments encourage learners to formulate hypotheses in the form of actions in the simulation and then observe the effects of their hypothesis. In turn, the environment can respond immediately to the actions of the learners (with several levels of intervention) providing instant feedback. Eggert did not doubt on the educational value of the simulators and agreed on the benefits for learners from being able to make errors without the pressure of a human judging them. However, he considered that simulations were too expensive to build in a way that could allow true independent learning and avoid learners pushing system boundaries while wandering around materials that they do not fully understand.

Following this “learning-to-do-by-doing” approach, IJAIED papers in the same period discussed the usefulness of robots for procedural learning. The robot-based pedagogy (Nonnon and Theil 1990) was proposed to take advantage of computers to transform learning activities that are often very abstract into others that are more concrete and hands-on. The goal of this approach was to let learners discover by themselves the methods of experimental inquiry. For this, the learner can stock incrementally the procedural instructions pertaining to an experimental schema. When needed, the robot-aid can take over from the learner the step-by-step procedural details of the experiment under study, look after the collection of data and give back a (graphical) representation of the dynamics of the whole process. This pedagogical concept of managing via robot-aid the data collection and the graphical representation of the experimental process in real-time was named “bi-focal cognitive lenses” by the authors, as it provided the learner with the opportunity to simultaneously perform an action and receive a representation of it for consultation. In turn, Howell and Hay (1989) proposed robotics manipulators to support learners with motor impairments to transform their desired actions into purposeful manipulations through the robot movements. In this way, learners with functional diversity were also able to experience learning processes and educational concepts. In a different approach, Rabardel (1991) reported the effect on the cognitive activity of using a training robot for learning through discovery, integrating the role of physical action in the construction of knowledge. The goal here was to guide the conceptualization process in relation to the action to be used for the development of skills.

A decade later, AIED evolved to more directly address issues related to modelling (of natural phenomena) and to elevate dynamic aspects of learning as an essential feature (Andriessen and Sandberg 1999). In this context, the roles of models in AIED research were discussed (Baker 2000). Within the first role (i.e., models as a scientific tool for understanding and predicting some aspect of an educational situation), Baker proposed extending the notion of cognition to embrace action and perception, and suggested to consider for instance, the need for “understanding the coordination between hand, eye, brain, pencil and paper required when a child learns to write”. His concern was to avoid leaving out any significant phenomena (as purely cognitive models of teaching practice do). This author argued that models integrating cognition, perception and action could be elaborated. In this context, he posed the question of whether these integrated models are useful in solving important problems.

Recently, in 2015, a few papers that focus on procedural learning of motor skills have emerged. Alvarez et al. (2015) designed and evaluated a virtual training application built with an ITS to simulate complex scenarios and realistic behaviours that controls events using real-time task recognition. Another ITS combined with augmented reality (i.e., a live direct or indirect view of a physical, real-world environment whose elements are augmented -or supplemented- by computer-generated sensory input) has also been built and evaluated in a different system (Westerfield et al. 2015) with the purpose of training for manual assembly tasks. In particular, this research suggests that combining abstract concepts and 3D spatial information in the context of real-world objects makes augmented reality an approach for training tasks which require manipulation of objects. These tasks are inherently spatial in nature and can be difficult to teach without close instruction and supervision. In this way, hands-on training that provides more intuitive, interactive and efficient training is supported, and this can provide new possibilities for motor skills development. This work has also showed the benefit of the personalized support provided by the ITS (in terms of domain, student and pedagogical models) in order to learn and retain the skills under study in a more effective way (i.e., better learning outcome in the corresponding post-test).

Above selected and commented papers provide some insight on how procedural learning has been considered along IJAIED history. Although motor skills training has not been addressed explicitly in IJAIED literature, signs of its potential interest for the AIED field have been identified in this review, such as 1) diverse technology (e.g., robots, augmented reality) has been considered to improve learning through manipulations, 2) simulation environments have been used for training in realistic situations that preclude real-life rehearsals, and 3) awareness of activities, such as learning to write, which involve physical actions that need to be analysed in order to understand all the phenomena related to the learning process. Regarding the first point, a question that can be posed is which other emerging technologies might be leveraged to improve learning through movement and what kind of learning can be benefited from that. Another question that can be posed addressing the second point asks whether it is possible to make rehearsals in real-life environments under certain circumstances, supporting full immersion in the learning task and going beyond the limitations of simulated environments. Finally, in relation to the last point, the question to be posed is whether modelling the physical activities involved in the learning process is useful. With the aim to answer these issues, the next section reviews motor skills training systems reported in literature that has been published outside the AIED community.

Motor Skills Learning Ouside the AIED Community

With the aim to open the way for AIED researchers to develop procedural learning ecosystems that support physical practice, a non-exhaustive literature review has been carried out to identify systems outside the AIED community that support motor skills learning. As a result, 17 works have been analysed in terms of the following four aspects, as compiled in Table 1:

Table 1 Review of motor skills learning systems published outside AIED community
Full size table
  1. 1.

    Sensing the learner’s corporal movement as specific skills are acquired, and the context in which this movement takes place.

  2. 2.

    Modelling the interactions to allow comparing the learner movement against the accurate movement (e.g., how an expert would carry the movement out).

  3. 3.

    Designing the feedback to be provided (i.e., what kind of support, and when and how to provide it).

  4. 4.

    Delivering the feedback in an effective non-intrusive way to advice the learner on how the body and limbs should move to achieve the motor learning goal.

In addition, each system compiled in Table 1 includes in the first column its name (when available), the bibliographical reference that has been analysed, the learning activity involved and the number of participants who took part in the evaluation of the system (when reported). In the analysis carried out, personalization issues (when implemented) have been identified and highlighted in Table 1 with bold face to facilitate its identification, as those are the relevant parts where AIED research can make relevant contributions. Works have been ordered chronologically (from oldest to most recent).

The review compiled in Table 1 shows diverse systems aimed to support motor skills learning. Following Magill’s classification (1993) in terms of precision most of the activities imply gross motor skills, although there are also examples of fine motor skills (e.g., hand writing, playing a musical instrument), beginning and end points of the movement are arbitrary (continuous motor skills) except for (Oakes et al. 2015; Lieberman and Breazeal 2007) and the stability of the environment involves changes in the response of the participant from one execution to the next one (except for sign language or handwriting), some stationary (playing a musical instrument) and others in motion (practicing a sport or martial art technique).

Many of these systems are intended for a diverse range of activities (Lieberman and Breazeal 2007; Miaw and Raskar 2010; Dancu 2012), although studies are usually carried out in a specific one. When user studies are carried out (in 12 of the 17 systems analysed), they are usually done so with a limited number of participants, and with the goal to show that the system functions as expected. Some aspects of the users experience are also analysed, such as whether users have positive views of the system (Takahata et al. 2004; Iwasako et al. 2014; Kikukawaa et al. 2014), whether they improve their performance while using the system (Kwon and Gross 2005; van der Linden et al. 2011; Ghasemzadeh et al. 2009; Portillo-Rodriguez et al. 2008), or how users respond to different feedback alternatives (Dancu 2012; Lieberman and Breazeal 2007). In another vein, most of the learning environments reviewed here address novice users. However, some works suggests that motor skills acquisition is also relevant, and even more appropriate, for advanced users (Spelmezan and Borchers 2008). Advanced users have different needs (Van der Linden et al. 2011) and/or behaviours (e.g., expert users are more consistent with the movement and the action is more controlled) (Ahmadi et al. 2006). In fact, as users might evolve from novice to experts with training, the system should also personalized its support and provide it accordingly (Takahata et al. 2004; Miaw and Raskar 2010; Van der Linden et al. 2011).

Next, the 17 systems analysed are discussed in relation to the four aspects considered in the review.

Sensing Movement in Motor Skills Learning Systems

The sensing of the movement is done i) through wearable devices on the body or the instrument used -when it exists (i.e., golf club, musical instrument, oar, etc.)- such as in (Oakes et al. 2015; Amma et al. 2014; James et al. 2011; Kikukawaa et al. 2014; Ghasemzadeh et al. 2009; Iwasako et al. 2014; Baca and Kornfeind 2006; van der Linden et al. 2011; Takahata et al. 2004; Spelmezan and Borchers 2008), ii) though optical motion capture devices placed in the environment (which usually require markers on the participants body, except for the Kinect sensor) such as in (Bloomfield and Badler 2008; Lieberman and Breazeal 2007; Dancu 2012; Miaw and Raskar 2010), or iii) combining both, that is within the user and instruments and from outside such as in (Rauter et al. 2011; Kwon and Gross 2005; Portillo-Rodriguez et al. 2008).

It has been suggested to move sensing outside the laboratory because participants need to be supported in situ (Oakes et al. 2015), since laboratory settings cannot reproduce the usual training (James et al. 2011), and there might be differences in usage when they are used in long term real practice (Hoggan and Brewster 2010). In addition, sensing in the wild allows a variety of context factors to be taken into account (Van der Linden et al. 2011). Wearable devices (which usually include accelerometers) eliminate the need that experiments are carried out solely inside the laboratory to obtain real-time feedback, and can be run in the real-world (Ahmadi et al. 2006). They can measure activity and effort levels and are usually small and light enough to be placed on any part of the body without hindering performance (Baca and Kornfeind 2006). In addition, and in contrast to optical devices, accelerometers provide precise angles and speed changes which are not visible to the eye (Kwon and Gross 2005). In any case, it is very important that wearable devices do not inhibit the user, and are easy to put on and remove, and are lightweight and unobtrusive (Miaw and Raskar 2010).

Many of the works reported in Table 1 show that wearables can provide straightforward information and do not hamper users’ movement (Kwon and Gross 2005). In much work, the requirement of low cost is also stated (Kwon and Gross 2005; Baca and Kornfeind 2006). However, expensive equipment is sometimes considered as proof of concept for initial experiments, with the expectation that the technology considered in them will have lower cost in the future (Lieberman and Breazeal 2007).

Beside the need to move outside the laboratory, some work claims the relevance of virtual environments to first practise physical skills (Bloomfield and Badler 2008). These virtual reality systems have been used to check how a trainee follows an avatar (Yang 1999). They use visual sensors to reconstruct the user’s posture and check how the trainee imitates the trainer’s motion by matching to a skeleton model to measure how learner mimics avatar motions (Kwon and Gross 2005). Virtual reality environments can provide augmented feedback by overlaying learners’ performance with desired movement, emphasizing the difference between subject and reference movement, and thus, highlighting desired trajectories (Lieberman and Breazeal 2007).

Modelling Movement in Motor Skills Learning Systems

Regarding the type of modelling, approaches are diverse. Some approaches focus on gesture recognition (Amma et al. 2014; Kikukawaa et al. 2014; Portillo-Rodriguez et al. 2008), others on physical features (Oakes et al. 2015; James et al. 2011) which can be modelled with 3D techniques (Bloomfield and Badler 2008; Iwasako et al. 2014; Van der Linden et al. 2011; Lieberman and Breazeal 2007), still others construct the models with diverse machine learning algorithms (Amma et al. 2014; Rauter et al. 2011; Kwon and Gross 2005).

Comparison is usually done against the movement carried out by an expert (Rauter et al. 2011; Ghasemzadeh et al. 2009; Miaw and Raskar 2010; Kwon and Gross 2005). When comparing expert’s movements with learner’s movements, a threshold is usually taken into account, and only when the threshold is overcome, is the feedback activated (Miaw and Raskar 2010; Bloomfield and Badler 2008). When the movement is complex, it is usually suggested to break the movement into less complicated actions (Ghasemzadeh et al. 2009; Kwon and Gross 2005; Portillo-Rodriguez et al. 2008). Very few systems mention the existence of a user model that records participants’ performance (Rauter et al. 2011; Lieberman and Breazeal 2007).

Designing Feedback in Motor Skills Learning Systems

In most of the papers, the design of the support provided is not described, as it seems to be considered a straight forward approach either i) to report the actual performance, usually as compared with an accurate performance and reporting when an error is made (Rauter et al. 2011; Kikukawaa et al. 2014; Ghasemzadeh et al. 2009; Iwasako et al. 2014), or ii) to show when the correct movement has been made (Oakes et al. 2015). Nonetheless, there are some works that aim to guide the learners’ movements in the appropriate way (Bloomfield and Badler 2008). In some cases, user centred design methods such as interviews have been followed to understand the feedback requirements (Oakes et al. 2015; Spelmezan and Borchers 2008).

Delivering Feedback in Motor Skills Learning Systems

When providing feedback, the focus of attention should be taken into account. It seems that if the focus is external (that is, on the movement’s effects rather than on body movements themselves, which are internal) such as it has been suggested in (Dancu 2012), it enhances motor performance and learning (Wulf 2013). However, in most of the work, the focus considered is internal.

The modalities for delivering the support can consist of visual, auditory and/or tactile cues. Humans have a natural parallel multimodal communication and interaction and prefer to interact multimodally (Portillo-Rodriguez et al. 2008). When humans learn a new motor skill from a teacher, they learn using multiple channels (Lieberman and Breazeal 2007): i) high level information about the skill received orally (abstract behavioural instructions), ii) visual information about how others perform the skill (demonstrating the motion themselves and requiring the learner to replicate the teacher’s performance), and iii) tactile information through the teacher’s physical guidance. However, some systems still rely only on visual feedback (Kwon and Gross 2005; James et al. 2011; Kikukawa et al. 2014; Iwasako et al. 2014). Sound, which was used in (Takahata et al. 2004; Baca and Kornfeind 2006; Portillo-Rodriguez et al. 2008; Rauter et al. 2011; Dancu 2012) can extend human perception, and feedback information can be provided by changing shape, tone and volume (Hollander and Furness 1994). The third type of feedback (i.e., haptic) is unique in motor skills learning environments, as it can only be applied when physical movements are considered in the learning process.

Haptic or tactile feedback presents the most direct form of motion information and directly engages the motor learning system, but it is the most difficult for a teacher to give (Lieberman and Breazeal 2007). It is provided exclusively for the user, which has advantages (it does not disturb others), but also disadvantages (it cannot be used to provide external assistance) (Oakes et al. 2015). It was originally developed for sensory substitution (e.g., to transmit information to deaf and/or blind people) rather than sensory augmentation (Lieberman and Breazeal 2007). Tactile feedback not only physically engages learners in the learning process, but might be the appropriate sensorial channel when learners require the visual and hearing senses for the learning tasks, such as when playing a musical instrument (Van der Linden et al. 2011). This kind of feedback is more effective when it points out mistakes learners regularly make (Van der Linden et al. 2011). Anyway, when experiencing and testing the feasibility of physical access in fully immersive experiences that can support learning new motor skills that require fairly precise physical body posture, visual realism might not sufficient and haptic support aimed for sensory substitution needs to be provided (Bloomfield and Badler 2008).

Tactile feedback can be provided in three different ways (Miaw and Raskar 2010; Lieberman and Breazeal 2007): i) vibrations with lightweight vibrating motors or tactors (tactile actuators), ii) physical movement (torque) produced with motors, and iii) electrical stimulation of muscles. This third option is often dangerous and painful, and in fact, no work has been found using it. The other two options distinguish between haptic feedback on movement (first option) and haptic guidance (second option). As for the first option, vibrotactile feedback can be provided using a small buzz in key moments, thus consolidating (touch to reward) or correcting (push or pull a limb, joint or other part of the body) a movement (Van der Linden et al. 2011). However, vibrotactile feedback might be ignored or undetected when cognitive demands are high and tasks are complex (Van der Linden et al. 2011). Anyway, an acclimation period is required, but that might still not be sufficient and participants may still not know how to respond to vibration signals at times (Lieberman and Breazeal 2007). Regarding the second option, haptic guidance prompts when a movement should be made, giving the opportunity to experience the feel or correcting the current posture or movement (Van der Linden et al. 2011). This physical guidance can be provided either with exoskeletons (force feedback on large body areas) (Bloomfield and Badler 2008) or torques. Torques require higher power and usually imply lower portability (Lieberman and Breazeal 2007).

From Neuroscience, it is reported that patients that have lost their ability to form new long-term memories still can build new motor skills, which shows that the brain processes motor learning differently from other conscious types of learning (Scoville and Milner 1957). This suggests that it may be possible to eventually train users to accept tactile feedback subconsciously to learn motor skills, turning corrections into automatic muscle reflex instead of a conscious mediation (Lieberman and Breazeal 2007).

Challenges and Opportunities to Personalize Motor Skills Learning in AIED Systems

As reported in the previous sections, systems aimed to support learners in motor skills acquisition have been developed outside the AIED community. Thus, it is not surprising that their personalization support is very limited. From all the systems reviewed in Table 1, those that describe higher personalization capabilities are M3-trainer (Rauter et al. 2011) and the MusicJacket (Van der Linden et al. 2011). Anyway, from the analysis of those 17 systems, some personalization opportunities for AIED research have been identified in Table 1 (and have been bolded there to facilitate its identification). The main personalization issues identified are the following: 1) take into account the specific capabilities of each learner (Oakes et al. 2015; Bloomfield and Badler 2008; Rauter et al. 2011; Takahata et al. 2004; Kwon and Gross 2005) and/or physical features (Iwasako et al. 2014; Van der Linden et al. 2011; Lieberman and Breazeal 2007; Miaw and Raskar 2010; Spelmezan and Borchers 2008; Portillo-Rodriguez et al. 2008), and 2) select the delivery of feedback depending on the learner’s preferences and/or context (Bloomfield and Badler 2008; Rauter et al. 2011; Van der Linden et al. 2011), or level of ability (Takahata et al. 2004; Miaw and Raskar 2010).

Therefore, there are challenges and opportunities for AIED researchers to build procedural learning ecosystems that personalize motor skills learning. As compiled in Table 2, AIED directions can be classified into: 1) modelling and representing the movements of the learner by building a learner psychomotor interaction model as well as an accurate movement model, and 2) providing the appropriate personalized psychomotor support in the most effective way for each learner in each training context. Regarding the first direction (i.e., modelling psycho-motor interaction), several AIED issues have to be addressed: i) to detect the physical interaction carried out by the learners by collecting data of their movements; ii) to model the accurate movements to be trained, which are those performed by experts; and iii) to diagnose the mismatch between the movements carried out by the learner and the expert’s way. Regarding the second direction (i.e., providing appropriate personalized psychomotor support), the issues involved are: iv) to model the intervention to be applied; v) to decide the intervention strategy to provide (e.g., showing performance, guiding, etc.), and when to apply it; and vi) to deliver the personalized intervention.

Table 2 AIED directions, research issues and challenges and opportunities, as well as novel approaches to be considered (interactive technologies and design methodologies)
Full size table

Each of these issues poses several challenges and opportunities for AIED research in terms of representation and interaction issues, which might be addressed by applying novel interactive technologies such as wearable devices, big data processing, 3D modelling, 3D printing, ambient intelligence, as well as holistic design approaches such as TORMES methodology. These are compiled in Table 2 and discussed next.

Challenges and Opportunities in Sensing Movement in Motor Skills Learning Systems

Sensing the movement of a learner implies detecting and tracking the learner’s corporal movement as she learns specific skilled movements as well as the context in which this movement takes place.

Low cost and non-intrusive wearable sensors can collect, without interrupting the person’s movements, electronic inputs (rather than just collecting interaction data in the computer and observing problem solving steps as done in a pure cognitive domain) about pressure applied, gesture, speed, acceleration, angle, etc. from devices attached to the body (or the part(s) of the body that are involved in the movement) as well as on any object used by the learner. Novel interactive technologies, such as those provided by quantified-self wearable devices, can be used to gather dynamic indicators while making the movement. This can help to understand how the movements are performed and improve training. For instance, the movements carried out by a person can be monitored using diverse types of sensors (inertial, positional, physiological, etc.) (Schneider et al. 2015) for real-time motion studies outside the laboratory (Fong and Chan 2010).

In addition, as reported in Fleury et al. (2015) e-textile research already provides several solutions that allow remote and mobile measurement of functional movement and posture with low intrusiveness by using some sort of technology embedded directly into the fabric on clothes and objetcs worn by the learner and which allows tracking complex movement patterns (Bonato 2005). In particular, phenomena that can be sensed are strain (Gioberto and Dunne 2013; Giorgino et al. 2009; Preece et al. 2011; Tormene et al. 2012; Yamada et al. 2011), acceleration (Harms et al. 2010; Zysset et al. 2013), joint movements (Lee et al. 2011), bend angle (Lorussi et al. 2013), posture (Di Rienzo et al. 2009), and pressure (Shu et al. 2010).

The processing of the data collected needs to be carefully considered. It might be possible to do some processing within the tracking device, but at some point the device needs to wirelessly send its data streams to a cloud environment where multimodal data integration can take place in order to be able to obtain useful indicators that describe the physical interaction carried out. The interaction data streams continuously collected by these sensors become very difficult to make sense of on the fly. Due to its size, variability, and speed that are due to the continuous collection of information from diverse kinds of sensors, big data mining techniques need to be explored (Fan and Bifet 2013). The goal here is to extract movement indicators at the same rate that data are being processed, so the modelling can support just-in-time feedback to the learner.

Challenges and Opportunities in Modelling Movement in Motor Skills Learning Systems

Modelling a learner’s movements implies comparing the learner’s movements against accurate movements, such as the one that an expert with similar physical capabilities of the learner would perform. For a given activity, the movements for which personalized support needs to be provided have to be identified, in a way that they can be isolated from one another. Depending on the kind of activity, this can be already defined in the domain, or it might imply extracting this information from domain experts. For each of these movements, the accurate way to carry out the movement has to be modelled. This can be done by recording the physical interaction performed by experts in the field. Since it is not possible for any person to perform a given movement in the same exact way each time, the model of the accurate movement should be built from several recordings that should involve not only a series of movements by the same expert, but also by other experts. From this set of recordings, the “accurate” movement should be extrapolated in terms of the relevant parameters that characterize it.

In the field of virtual reality, there is work that builds virtual skeletal models for videogames from the information collected using wearable technology (e.g., biomechanical or inertial sensors), which both map the movement as well as recognize gestures with artificial intelligence techniques (Arsenault 2014). The movement controlled by sensors can also be represented in 3D models of the human body (Kifayat et al. 2010; Bae et al. 2012). The biometric function that represents the human body can parameterize the human posture (DellaGrotte et al. 2008) as well as gestures produced by the body limbs, such as the hands (Marcel 2002). The parameter should make possible to identify when a movement fits in the “accurate” model (i.e., it is equivalent to a movement that would be produced by an expert), and when not (i.e., it is produced by someone with less expertise who is trying to learn the movement), despite the inherent variability in the replication of a given movement commented upon above.

In addition, a learner model is needed to describe the learners’ body build as well as her psychomotor skills, level of ability and progress towards mastering the various movements. For this, the model should i) describe the learner’s physical features, such as hand length, maximum shoulder opening, etc.; ii) provide information about her interaction capabilities, such as force, speed, etc.; and iii) contain indicators regarding the way she can perform the different kinds of movements to be learnt (e.g., level of performance).

Challenges and Opportunities in Designing Feedback in Motor Skills Learning Systems

When the movement does not fit into the accurate movement model, an intervention might be necessary and needs to be designed. Thus, a key area where AIED has a big role to play is designing the feedback to be provided. In particular, whether it is appropriate to provide support at this moment and, if so, what kind of support to provide. In a personalized system, this implies deciding when to interact with the learner as well as what to ask her to do and how to advise her on how to do it. The focus in personalizing motor skills learning has to be put on identifying what is the most appropriate intervention in each case (considering cognitive, affective and psychomotor dimensions) and when and how it should be delivered in order to make a positive impact on the learning process. It should go beyond traditional intervention approaches which are based on giving instructions that describe what to do and how to do it. In fact, procedural learning in terms of motor skills is usually difficult to explain by the instructor and to understand by the learner. And it turns out that this procedural tutoring support is of major relevance in the case of novice learners, as they might get into a wrong habit if no timely feedback is provided to them while practicing on their own and, thus, they cannot understand why the movement is not accurate (Kikukawaa et al. 2014).

In acquiring motor skills there is a trade-off between learning (i.e., relatively permanent changes in behaviour that support long-term retention and transfer) and performance (i.e., temporary fluctuations in behaviour that are observed and measured during training or immediately thereafter) that should also be taken into account (Soderstrom and Bjork 2015). In particular, decisions that have a positive effect on long-term learning (such separating practice with time and other activities, varying conditions in practice, and making self-produced movements) introduce more performance errors, and thus, might frustrate the learner. Thus, when at some point the learner is frustrated with the practice, it might be better to support short-term performance by externally guiding the learner rather than forcing self-produced movements that would improve her long-term progress. Short-term performance (but not long-term learning) is also improved practicing the same thing over and over, and without varying the conditions, but that increases boredom. Such frustration and boredom mean that motivation needs to be strongly supported considering the state of the art in affective computing (Santos 2016a). Anyway, when a skill is acquired, feedback should be reduced to avoid a dependency on it (Sigrist et al. 2013).

Another issue of major importance is the selection of the feedback modality. Visual feedback can be used to understand the movement (Sigrist et al. 2013) and improve motor skills without the aid of a live instructor. In motion capture systems, feedback can be provided comparing the user movements with the body positions of an avatar performing expert movements. In addition to presenting sensor data to the learner, this allows supporting awareness and self-reflection of how movements carried out by the learner are aligned with the accurate version of the movements. 3D representations are of relevance here. Haptic feedback is a new modality that can take advantage of the physical interactions, but it might not always be the most appropriate approach. For instance (Marchal-Crespo et al. 2013) found that haptic guidance was especially suitable for less-skilled subjects and in especially difficult discrete tasks, while visual feedback seemed to benefit more skilled subjects. Additionally, haptic guidance seemed to promote learning in a time-critical tracking task.

Therefore, instructors need to be involved in an elicitation process in order to design the most effective feedback approaches that suit particular learners’ needs and their context. The knowledge to be elicited is tacit and implicit in the educators’ head, as it has been acquired through years of teaching, and typically never made explicit in a systematic way. In order to identify appropriate interventions that produce a positive impact in the learning process, TORMES elicitation methodology can be applied (Santos and Boticario 2015). TORMES combines user centred design methods with data mining techniques and extends the design cycle of interactive systems as defined by ISO 9241–210 (ISO 2010) with the life cycle of e-learning (Van Rosmalen et al. 2004) and the layered evaluation of adaptive systems (Paramythis et al. 2010). Resulting interventions are semantically described in terms of i) what is to be recommended, ii) how and where the recommendation is to be communicated, iii) when and to who the recommendation is to be offered, iv) why the recommendation is to be delivered, and v) which features characterize the recommendations themselves (Santos and Boticario 2011). In order to address the particularities required by the psychomotor learning domain and the requirements to model psychomotor interaction and provide appropriate support, an extension to TORMES might be needed. In any case, TORMES can be used by educators to apply appropriate user centred design methods to gather tacit knowledge through qualitative descriptions from psychomotor experts while analysing performance indicators of learners regarding movements carried out while training. These indicators can be computed with (big) data mining techniques in order to gather quantitative information from data collected in learning experiences with wearable devices. As a result, this elicitation process can facilitate the identification of the psychomotor support to be provided as the qualitative analysis on quantitative indicators can rise up unaware opportunities for motor skills personalization support that should improve training performance.

Challenges and Opportunities in Delivering Feedback in Motor Skills Learning Systems

Different types of feedback (either positive or negative) can promote motor skills acquisition (Ashby and O’Brien 2007). Feedback can be provided in several ways (e.g., visual, auditory, haptic), although a multimodal approach might produce better results (Sigrist et al. 2013; Portillo-Rodriguez et al. 2008). For instance, ambient intelligence can make the environment sensitive and responsive to the presence of people, hence it can be an interesting approach to provide multimodal sensorial feedback through different sensorial channels, as provided by the AICARP system that has been discussed elsewhere (Santos et al. 2015, 2016). In that approach, the learner configures her preferred sensorial channel(s) for feedback delivery, which could be stored in the learner model and even learnt by the system.

The main novelty in motor skills learning environments is the delivery of haptic support. This support can be provided with vibrotactile feedback or with haptic guidance. The former can be delivered through diverse actuators such as resistance, force, vibration, etc. to tell the learner when her movement does not reflect the accurate movement that is used as reference, or to reward a good movement to consolidate it. Actuators can be sewed on clothes to be worn by the learner, such as on the sleeve of a jacket. Sewable electronic modules controlled by Lilypad Arduino can be used for prototyping the system. An extended review on systems that deliver vibrotactile feedback is compiled elsewhere (Santos and Revilla 2016). In turn, haptic guidance can be delivered in terms of a tangible support that scaffolds the learner through an exoskeleton attached on the body on how the body and limbs should move to achieve the learning goal. As an example, the exoskeleton robotics developed by Datta (2014) can move the fingers of the learner to make the movement considered to be accurate by the system. The technology for 3D modelling combined with 3D printing make feasible the building of physical prototypes of tangible objects that can create such exoskeletons to physically scaffold the learner movement, as discussed in (Santos 2016b). In addition, the very emerging 4D technology (which produces physical objects that can change with time) can be an interesting option to explore in order to dynamically personalized the exoskeleton used for the guiding.

Conclusions

There exist many and diverse types of activities (e.g., playing a musical instrument, handwriting, drawing, training for surgery, improving the technique in sports and martial arts, learning sign language, dancing, etc.) that require long-term physical training to learn how to perform the movements in the most efficient way. The training process is based on the learning-by-doing approach and involves correcting the learner’s movements till her optimal movements (considering the own learner’s corporal features, specific physical abilities, performance to date as well as the particularities of the motor skills to be learnt) is achieved. Defining the accurate movement and when and how to provide the physical support is not trivial at all, but is indeed a modelling challenge for AIED.

No specific work on motor skills learning has been found in the review of IJAIED papers reported here (although some signs of their potential interest have been detected and commented upon in this paper). In turn, outside the AIED community, diverse learning environments for motor skills learning have been developed, but have almost no personalization features, which are clearly specific of AIED systems. This shows a gap in AIED research that can become an opportunity thanks to emerging novel interaction technologies and design methodologies.

In fact, the review of IJAIED papers has shown that AIED research has historically reflected the technology of the times (Schank and Edelson (1989/1990); Eggert (1990)). Following this, I concur that AIED systems should take advantage of those technological advances that can make the field progress, although there is a need to carefully analyse the best way to apply them. At this point in time, my vision for the future of the AIED field in the psychomotor learning domain is that the synergy of artificial intelligence techniques with novel interaction technologies such as quantified-self wearable devices from the “Internet of Me”, 3D modelling and 3D printing, big data streams processing on the fly and ambient intelligence sensorial support, among others, can breathe new life into the AIED research and open new opportunities for building procedural AIED ecosystem of standard-based interactive learning tools that can personalize motor skills learning by providing intelligent real-time feedback to scaffold learning in the psychomotor domain. Moreover, design methodologies such as TORMES can guide the elicitation of appropriate psychomotor interventions supported by models of expert and learner behaviour to underpin this personalization effort.

The selection and reporting of research and technologies in this paper does not aim to be exhaustive, but to suggest some technological directions to be explored by the AIED research community in the coming years. In particular, the work reported here suggests that there exists a technological context that can support the AIED research in developing systems that sense the learner’s corporal behaviour as she learns specific skilled movements and when needed, provide the appropriate support aimed to make the learner acquire new motor skills (or refine existing skills) aided by developing a muscle memory experience along the way. This requires novel forms of expert as well as learner modelling, novel forms of personalized support suitable for each learner in each training context, etc., so this is indeed a potential new research frontier in AIED. There are four main issues to be addressed: i) sensing the movement performed by the learner, ii) modelling and comparing learners’ and experts’ movements, iii) designing the required support to give, and iv) delivering the personalized support. Interestingly, by putting motor skills learning in the AIED research agenda for the next years, the field is somehow also going back to its roots due to the need to revisit many of our fundamental and driving questions, but in the context of motor skills. Substantial evidence exists to suggest that important similarities exist between motor and cognitive learning (Bjork and Bjork 1992).

Given the enormous amount of resources spent every year on motor skills training throughout society, this research area might turn out to be extremely important, and thus, it can turn into a good opportunity for the AIED field. The added value of AIED is to provide the personalization support required to build procedural learning ecosystems that can individualize the learning of motor skills.