Training the Body: The Potential of AIED to Support Personalized Motor Skills Learning

  • Olga C. Santos


This paper argues that the research field of Artificial Intelligence in Education (AIED) can benefit from integrating recent technological advances (e.g., wearable devices, big data processing, 3D modelling, 3D printing, ambient intelligence) and design methodologies, such as TORMES, when developing systems that address the psychomotor learning domain. In particular, the acquisition of motor skills could benefit from individualized instruction and support just as cognitive skills learning has over the last decades. To this point, procedural learning has been considered since the earliest days of AIED (dating back to the 1980’s). However, AIED developments in motor skills learning have lagged significantly behind. As technology has evolved, and supported by the do-it-yourself and quantified-self movements, it is now possible to integrate emerging interactive technologies in order to provide personal awareness and reflection for behavioural change at low cost and with low intrusion. Many activities exist that would benefit from personalizing motor skills learning, such as playing a musical instrument, handwriting, drawing, training for surgery, improving the technique in sports and martial arts, learning sign language, dancing, etc. In this context, my suggestions for AIED research in the coming 25 years focus on addressing challenges regarding 1) modelling the psychomotor interaction, and 2) providing appropriate personalized psychomotor support.


Procedural learning Motor skills learning Psychomotor learning domain Artificial intelligence Education Internet of me Quantified-self Wearable devices Big data 3D modelling 3D printing Ambient intelligence TORMES methodology 


Research in Artificial Intelligence in Education (AIED) aims to develop flexible systems that will increase access to effective, personalized and engaging, anytime, anywhere learning throughout lifetimes across the full range of knowledge domains and skills and employing varied pedagogical approaches (Underwood and Luckin 2011).

Under this umbrella, this paper focuses on the psychomotor learning domain, which deals with physical movement, coordination and the use of the motor skills areas (Harrow 1972). Motor skills refer to bodily movements involving muscular activity (Gagné and Briggs 1979). More precisely, motor skills involve goal-oriented physical actions or tasks requiring voluntary body and/or limb movements to achieve the goal (Magill 1993). Thus, motor skills learning can be defined as achieving the ability to perform a function acquired with practice that requires body and/or limb movement to accomplish the goal of an action or task (Christensson 2005). It is about “doing” through imitation, practicing and habituating new skills (Penney 2011). Motor skills can be classified in three dimensions (Magill 1993):
  1. 1)

    precision of the movement, which considers i) gross motor skills that involve large musculatures and a goal where the precision of movement is not as important, and ii) fine motor skills that require control of small muscles of the body to achieve the goal of the skill and generally including hand-eye coordination;

  2. 2)

    defining the beginning and end points of the movement, which considers i) discrete motor skills that have clearly defined beginning and end points, ii) serial motor skills that put together several discrete motor skills, and iii) continuous motor skills in which the performer of the skill determines the beginning and end points; and

  3. 3)

    stability of the environment, which in a four category system combines i) the change or not from one execution to the next one, with ii) the execution either stationary or in motion.


Consolidating specific motor tasks into memory through repetition (thus, creating long-term muscle memory for a given task), is very relevant in diverse scenarios that support learning processes involving not only brain activity, but also physical activity, such as playing a musical instrument, handwriting, drawing, training for surgery, improving the technique in sports and martial arts, learning sign language, dancing. In these situations, learners have to train by repeating very specific movements till they learn the best way to carry them out effectively without conscious effort (Krakauer and Shadmehr 2006). The automatization of skills is very much shared between cognitive and motor skills, so in essence, all of the work being done in AIED over the last 30 years on cognitive skills acquisition, that can go back to (Anderson 1982), is worthy of reinvestigation with motor skills.

For that matter, learning motor skills goes beyond mere muscle memory, blending motor skills with cognitive (including meta-cognitive) and affective skills. Hence, research in AIED must build on existing cognitive (i.e., tell what to do when the learner answers something wrong), meta-cognitive (make learners aware of their learning process and thus, produce their own learning strategies) and affective research (turning negative emotional states into productive learning experiences). The cognitive, meta-cognitive and affective dimensions are popular focus areas in AIED. However, as discussed in this paper, there have not yet been corresponding AIED developments addressing the psychomotor learning domain, thus supporting the personalized learning of motor skills. Therefore, the focus of this paper is mainly on how the physical part related to muscle training can be supported from an AIED perspective, both in the modelling of the learner’s physical interaction and in the provision of personalized support during the training. This implies that the physical actions carried out while training are to be monitored in real-time, compared with experts’ performance and, when needed, corrected through appropriate feedback, in order to achieve successful motor skills learning (i.e., skills learning at a physical level).

Feedback is crucial in motor skills learning (Swinnen 1996). For example, learners do not often recognize motion errors, and experts need to provide objective advice to learners (Iwasako et al. 2014). Traditionally, a human instructor or coach is used to support motor skills learning (Wulf et al. 2010) following the apprenticeship model (i.e., learning by observation of an expert) (Christensson 2005), which can also be done using multimedia content (Kwon and Gross 2005). Methods include (van der Linden et al. 2011): i) verbal instructions, ii) physically guiding the learner’s limb movements, and iii) allowing learners to observe themselves in a mirror. This training approach requires a high degree of expertise in identifying how close the movement performed resembles the ideal one, especially when movements are complex and can be performed with different level. There are also several limitations when a human attempts to provide feedback (Rauter et al. 2011): i) a human cannot hold the same level of attention and concentration over a longer period of time, ii) a human is not able to observe all important physiological and biomechanical variables characterizing the movement, neither at the same time nor at high precision, and iii) a human can hardly provide augmented concurrent feedback (i.e., external stimulus during the movement in real-time). In addition, the process of learning by imitation can be improved with multimodal feedback systems (Portillo-Rodriguez et al. 2008), which are systems that provide feedback through different sensorial channels (i.e., visual, auditory and haptic).

Moreover, according to (Christensson 2005), personalized motor skills learning environments that follow a learner-centric and constructivist (learn-by-doing) approach are needed. In this way, the system can be challenging and supporting at the same time in order to properly engage and motivate the learner. Hence, it can adapt to each individual needs and progress, providing personalized feedback meanwhile the learner is performing the tasks, and changing the instructional strategy when needed (e.g., guide, teach, iterate, challenge).

Because there are multiple novel interactive technologies that can help make personalized motor skills learning possible, this paper argues that now is a good time to introduce motor skills learning as a main research direction in the AIED community. A variety of emerging resources are primed to support this new direction, including:
  • novel types of quantified-self 1 wearable devices from the so called “Internet of Me” that allow collecting data from the user in a non-intrusive way, such as with smart bracelets, watches, t-shirts, etc.,

  • techniques for big data analysis that deal with volume, variety, velocity, variability and veracity,

  • 3D modelling of living physical objects by developing a mathematical representation of their three-dimensional surface,

  • 3D printing of exoskeletons for motor control rehabilitation, and

  • augmented sensorial output with ambient intelligence (i.e., environments that are sensitive and responsive to the presence of people).

These technologies can enrich AIED research in order to develop procedural learning environments that can physically support -and when appropriate guide- the particular way each individual learner should move her body in order to achieve specific motor learning goals (i.e., learning goals that are related to improving motor skills acquisition). As these technologies are of much interest in the do-it-yourself (DIY) movement,2 the AIED research community can take advantage of the resources and support already available online, thus reducing the learning curve for their integration in AIED systems. In addition, educators need to be involved in what is known as an elicitation process to help them identify the personalized support to be provided to learners training their motor skills. TORMES methodology (Santos and Boticario 2015) can facilitate this elicitation process.

The rationale for pursuing this new approach in the AIED field is that the learning of motor skills would benefit from the individualized instruction and support that it is the purpose of AIED to provide. The acquisition of motor skills poses familiar AIED problems, but in novel and challenging new forms, such as i) modelling correct (expert) motor movements, ii) diagnosing students’ movements, and iii) deciding the personalized intervention (i.e., what feedback to give and when and in what form, etc.). Emerging novel interaction technologies seem to fill the technological gap to make personalization possible in motor skills learning. In addition, service-oriented architecture approaches aimed to extend learning environments with adaptive navigation support (Santos and Boticario 2011) need to take into account recent trends in service-oriented and ubiquitous computing, moving to an ecosystem of standard-based interactive learning tools (Nye 2015). Figure 1 depicts aforementioned elements that are foreseen to frame the AIED research in the coming 25 years aimed to develop systems that can provide individualized instruction and support in motor skills training.
Fig. 1

Elements for providing individualized instruction and support in motor skills training

As preliminary work to this paper, a discussion on how the training of the Aikido martial art can benefit from an AIED procedural learning environment for personalized motor skills training (Santos 2015) was presented at the workshop “Les Contes du Marriage: Should AI stay married to ED?” that was organized during the AIED 2015 conference to examine the current and future identity of the AIED field.

The contents of this paper are structured as follows. First, a review of papers from the International Journal of AIED (IJAIED) is provided, which reveals a dearth of AIED research on personalized motor skills learning (despite a few examples that tackle related issues) inside the community. Next, a non-exhaustive list of motor skills learning systems from outside the AIED field are reviewed, revealing a lack of attention to personalization features. After that, challenges and opportunities for the AIED research community are identified, when considering emerging interactive technologies to build personalized systems for motor skills learning.

Motor Skills Learning Inside the AIED Community

Limited attention to physical and motor aspects of learning has been carried out in the AIED community so far. This is likely due to the lack of appropriate technological support available in the past for movement detection at low cost and with low intrusion. In order to provide some background of the AIED field regarding personalized support for motor skills learning (and since no specific IJAIED papers addressing motor skills personalized learning were found in the review), this section compiles articles published in IJAIED that tackle related issues from the procedural learning perspective. In particular, the acquisition of procedural skills has been addressed since the eighties and early nineties in the AIED field, primarily in the context of Intelligent Tutoring Systems (ITS) in closed domains (Andriessen and Sandberg 1999). Typically, these approaches involved comparisons between learner answers and experts’ by (Ohlsson 1993): 1) observing learners’ actions, 2) measuring progress in terms of correct problem solving steps, and 3) applying existing and well-founded theoretical notions on skill acquisition.

This review of IJAIED papers starts with some historical issues regarding procedural learning that arose during the discussion carried out among Schank and Edelson (1989/1990) and Eggert (1990) on their different viewpoints regarding the role for AIED, and how technology can be used to reshape education. In particular, Schank and Edelson proposed and discussed the benefits of simulation as a relevant instructional approach for computer-based education since it is realistic enough to facilitate learning-by-doing (through failure-driven learning) in domains that are too risky or expensive to be trained in, allowing also the learner to revise her hypothesis based on her observations. In addition to engaging the learner in the learning process thanks to the direct experience while acquiring knowledge, these authors highlighted that simulation environments encourage learners to formulate hypotheses in the form of actions in the simulation and then observe the effects of their hypothesis. In turn, the environment can respond immediately to the actions of the learners (with several levels of intervention) providing instant feedback. Eggert did not doubt on the educational value of the simulators and agreed on the benefits for learners from being able to make errors without the pressure of a human judging them. However, he considered that simulations were too expensive to build in a way that could allow true independent learning and avoid learners pushing system boundaries while wandering around materials that they do not fully understand.

Following this “learning-to-do-by-doing” approach, IJAIED papers in the same period discussed the usefulness of robots for procedural learning. The robot-based pedagogy (Nonnon and Theil 1990) was proposed to take advantage of computers to transform learning activities that are often very abstract into others that are more concrete and hands-on. The goal of this approach was to let learners discover by themselves the methods of experimental inquiry. For this, the learner can stock incrementally the procedural instructions pertaining to an experimental schema. When needed, the robot-aid can take over from the learner the step-by-step procedural details of the experiment under study, look after the collection of data and give back a (graphical) representation of the dynamics of the whole process. This pedagogical concept of managing via robot-aid the data collection and the graphical representation of the experimental process in real-time was named “bi-focal cognitive lenses” by the authors, as it provided the learner with the opportunity to simultaneously perform an action and receive a representation of it for consultation. In turn, Howell and Hay (1989) proposed robotics manipulators to support learners with motor impairments to transform their desired actions into purposeful manipulations through the robot movements. In this way, learners with functional diversity were also able to experience learning processes and educational concepts. In a different approach, Rabardel (1991) reported the effect on the cognitive activity of using a training robot for learning through discovery, integrating the role of physical action in the construction of knowledge. The goal here was to guide the conceptualization process in relation to the action to be used for the development of skills.

A decade later, AIED evolved to more directly address issues related to modelling (of natural phenomena) and to elevate dynamic aspects of learning as an essential feature (Andriessen and Sandberg 1999). In this context, the roles of models in AIED research were discussed (Baker 2000). Within the first role (i.e., models as a scientific tool for understanding and predicting some aspect of an educational situation), Baker proposed extending the notion of cognition to embrace action and perception, and suggested to consider for instance, the need for “understanding the coordination between hand, eye, brain, pencil and paper required when a child learns to write”. His concern was to avoid leaving out any significant phenomena (as purely cognitive models of teaching practice do). This author argued that models integrating cognition, perception and action could be elaborated. In this context, he posed the question of whether these integrated models are useful in solving important problems.

Recently, in 2015, a few papers that focus on procedural learning of motor skills have emerged. Alvarez et al. (2015) designed and evaluated a virtual training application built with an ITS to simulate complex scenarios and realistic behaviours that controls events using real-time task recognition. Another ITS combined with augmented reality (i.e., a live direct or indirect view of a physical, real-world environment whose elements are augmented -or supplemented- by computer-generated sensory input) has also been built and evaluated in a different system (Westerfield et al. 2015) with the purpose of training for manual assembly tasks. In particular, this research suggests that combining abstract concepts and 3D spatial information in the context of real-world objects makes augmented reality an approach for training tasks which require manipulation of objects. These tasks are inherently spatial in nature and can be difficult to teach without close instruction and supervision. In this way, hands-on training that provides more intuitive, interactive and efficient training is supported, and this can provide new possibilities for motor skills development. This work has also showed the benefit of the personalized support provided by the ITS (in terms of domain, student and pedagogical models) in order to learn and retain the skills under study in a more effective way (i.e., better learning outcome in the corresponding post-test).

Above selected and commented papers provide some insight on how procedural learning has been considered along IJAIED history. Although motor skills training has not been addressed explicitly in IJAIED literature, signs of its potential interest for the AIED field have been identified in this review, such as 1) diverse technology (e.g., robots, augmented reality) has been considered to improve learning through manipulations, 2) simulation environments have been used for training in realistic situations that preclude real-life rehearsals, and 3) awareness of activities, such as learning to write, which involve physical actions that need to be analysed in order to understand all the phenomena related to the learning process. Regarding the first point, a question that can be posed is which other emerging technologies might be leveraged to improve learning through movement and what kind of learning can be benefited from that. Another question that can be posed addressing the second point asks whether it is possible to make rehearsals in real-life environments under certain circumstances, supporting full immersion in the learning task and going beyond the limitations of simulated environments. Finally, in relation to the last point, the question to be posed is whether modelling the physical activities involved in the learning process is useful. With the aim to answer these issues, the next section reviews motor skills training systems reported in literature that has been published outside the AIED community.

Motor Skills Learning Ouside the AIED Community

With the aim to open the way for AIED researchers to develop procedural learning ecosystems that support physical practice, a non-exhaustive literature review has been carried out to identify systems outside the AIED community that support motor skills learning. As a result, 17 works have been analysed in terms of the following four aspects, as compiled in Table 1:
Table 1

Review of motor skills learning systems published outside AIED community

System / Reference/ Activity / Subjects







(Takahata et al., 2004)


10 learners (novice)

Wearable device (accelerometer)

Waist (rotation) and wrist

Model of karate (depends on each learner’s skilled level): not a unique karate training model

Goal: train learners’ movements to sounds

Feedback depends on learner level (beginners vs. advanced)

Sound feedback in waist and wrist (check the order of motions and practice in their rhythm) with correct actions (i.e., correct timing)

(Kwon and Gross, 2005)


6 learners (no experience) + 1 trainer

Body (accelerometer: tilt, movement and vibration + markers) and visual (camera) sensors

Motion data model with motion chunks (i.e., pieces of motion information)

Machine learning (HMM) to construct the motion model

Models posture + gesture

Different learners’ styles: focuses on improving gestures, static postures, both at the same time

Gender issues: males use more power, female focus on technique

Goal: explain and improve motion practice to help learners find the correct posture

Compares collected data with trainer’s so learner can adjust motion

Trainers create instructive material (training video), edited to select video frames and add explanatory information

Visual feedback: body sensor data on the images along the motion path (template that displays a moving circle along the path changing size with acceleration) + boolean feedback indicating if posture is correct

(Baca and Kornfeind, 2006)

Rowing / table tennis/biathlon


Sensors in the environment (not on person):

-rowing: force sensors

-table tennis: accelerometers + microphone

-biathlon: video system

Factors differ with activity:

-rowing: timing, rower’s centre of mass, force applied to oars

-table tennis: ball spin, position, time

-biathlon: motion of the rifle’s barrel

Goal: present performance data (feedback on the quality of the technique)

-Rowing: screen with graphics

-Table tennis: acoustic feedback on impact position and time + screen with impact position in the table

-Biathlon: screen with graphics + video


(Lieberman and Breazeal, 2007)

Copy movements from videos

40 learners

Vicon motion capture system (optical tracking) with infrared cameras + reflectors (markers)

3D models of markers can be inferred to build skeletal models

Compares learner’s and teacher’s performance

Subjects have different joint lengths and offsets, a calibration routines needs to be run

User’s performance is stored

Goal: correct the posture

Discrepancy wherever learner’s body differs from teacher’s (incorrect joint angles and rotations)

Feedback simultaneous to all joints. It can supplement visual and auditory feedback from teacher.

Vibrotactile feedback requires acclimation

Vibrotactile feedback: joints moving in error receive vibrations till position is corrected (magnitude proportional to error)

Sensory saltation phenomenona simulates rotating signal

(Spelmezan and Borchers, 2008)


3 advanced beginners / 11 instructors

Mobile device with Bluetooth & Arduino + inertial sensors, bend sensors, force-sensitive resistors + video

Back of each knee + inside boot + leg + upper body

Digital compass algorithms to measure rotation

Neutral position (weight central over the board and distributed equally between both feet + legs and ankles flexed + shoulders and hips in line + head up) is used as reference to find mistakes during ride → user dependent!

Correct posture also depends on slope gradient

Goal: alert incorrect movements and body positions

Practice in spatial and temporal challenging sports can only receive delayed feedback (after training) and there is no possibility of physical guiding by instructor

Instructors were interviewed to understand how they teach and the most common mistakes.

Even if immediate feedback is provided, beginners do not react because they are focused on keeping their balance better with some experience. Even experienced still can fine tune movements.

Audible and tactile (unimplemented)

(Portillo-Rodriguez et al., 2008)


5 learners (novel)

Vicon optical system with markers

Gesture recognition system (k-means clustering, PNN, FSM)

Gestures are subdivided in n-states (time independent)

Qualify learner’s movements with master’s

Calibration process is required to adjust the kinematics model to each user using a initial position (stand at attention)

Goal: understand and correct movements

Feedback stimuli to compensate errors during performance of gesture

Master performs the movements one step forward to learner and remains there till learner reaches actual movement

FW: develop a skill methodology by acquiring data from different experts and analyze their styles to select exercises to improve learners’ movements

-Audio: soft-repetitive sound that changes volume and pitch proportional to error and from the corresponding axis side (error location)

-Visual: master and second avatar + graphics (virtual energy line with users’ movements whose intensity varies with error distance)

-Vibrotactile: vibrations at different frequencies depending on error distance when arms reach bad position and need to be corrected

There is also delayed feedback when finished: replays movements performed + statistical information


(Bloomfield and Badler, 2008)


5 subjects (unexperienced)

Motion capture device

Real-time human model with body pose in 3D space

Wrist rotation + positions of elbow and wrist

Video frames analysis

Goal: guide the arm movement

Tactor (tactile actuators) activates when movement separates a predefined distance from accurate position

Predefined distances seemed not appropriate for all subjects

Visual (projection screen) vs. vibrotactile (sleeve of tactors with vibratory motors to the user’s right arm and hand)

Best modality may depend on situation

(Ghasemzadeh et al., 2009)

Golf swing

4 subjects (novice)

Wearable inertial motion sensors (accelerometer and gyroscope) on golf club + body (acceleration and angular velocity)

Quantitative model (LDA & PCA signal processing to quantify correctness of performed action)

Dataset of accurate movements obtained with swing trainer

Goal: Feedback on the quality of movement (FW)

Quality measure: linear projection of features space that changes while wrist rotation angle varies (deviation from target line)


Second Skin

(Miaw and Raskar, 2010)

Sports training/dance practice

No evaluation

Optical 3D tracking system (extension of Prakash to 3D): infrared projectors to track infrared photodetectors (no cameras) worn by the user

User motion tracking is analyzed against a reference (i.e., recorded movements of an expert)

Teacher performs the correct motions (reference angles)

Limbs angles to determine error (more tolerant for users with different builds and body structures)

Goal: aid in the correction of the movement and position errors

Tells how to adjust in order to correct motion

Threshold from correct movement to trigger feedback

FW: physically guiding the movement in activities with little time to consciously react to a skin stimuli (e.g., golf swing)

Vibrotactil feedback through tactile actuators (tactors)

2 modes: beginners (waits till the learner gets the correct position) and advanced (expect that are mostly performing correct motions at actual speed) learners

(James et al., 2011)


no evaluation

Wearable sensors (inertial sensors for swimming biomechanics: accelerometer and gyroscope) + video

Attached to wrist and ankle

Features: strokes, wall push-offs, turns and lap times

Algorithms developed using hand-timed data and underwater video as benchmarks

Goal: quantify performance and identify potential harm (multiple devices/ multiple users)

Poolside interventions and long term developmental planning

Visualization of raw and processed data (video and graphics)


(Rauter et al., 2011)


no evaluation

Sensors in the rowing simulator + optelectrical motion tracking system

“Perfect” reference data (measurements of physiological and biomechanical variables on professional rowers)

Quantifies errors: error classification in real-time by an algorithm for online data analysis (FFT, PCA, SVM).

Subject’s profile (errors & feedback given) + individual skills & perceptual abilities

Goal: challenge subjects within their abilities to maximize the overall performance

Select feedback to address the most important movement errors

Choose feedback modality and strategy based on the rowing error recognized

Augmented feedback: auditory (loud speakers), visual (screen projections) and haptic cues (tendon-based parallel robot) [note that details are not provided]

FW: feedback modality and strategy adapted to individual maximal capability and individual perception (actual performance)


(Van der Linden et al., 2011)


6 (beginners) and 4 (advanced) learners

Inertial motion capture sensors (accelerometers, gyroscope and magnetometer)

System can be configured to the body size of each user

3D position using a hierarchical skeleton model (movements and posture during bowing)

Goal: improve technique and awareness of execution

Defines a reference bow stroke (target path) to provide feedback appropriate to each player (build, way to hold the violin, teacher pedagogy). Obtained though calibration with teacher

Vibration motors in arms and torso (when away from target path)

Beginners required teachers’ indications on how to move to stay in path.

Feedback differs with ability levels

(Dancu, 2012)

Clinical applications /instructional games


Kinect sensor (body tracking based on images)

Representation of hand movements executed + next ones to be performed

2D avatar wih OpenNI and OGRE

Whole movement divided into submovements

Goal: inform if the correct movement is performed

External focus of attention: submovement goal is hitting the visual marker (i.e., a red sphere)

Shows when hand is off course

Visual representations of trainer and learner avatar + audio when hitting the target (volume/frequency proportional to distance from correct path) + score in the screen


(Amma et al., 2014)


9 users

Wearable input system (motion/inertial sensors applying accelerometers and gyroscopes)

Attached to the back of the hand.

3D handwriting recognition:

-SVM for segment identification

-HMM for segment recognition (text representation)

-statistical language model



(Kikukawa et al., 2014)

Erhu bows strokes

8 subjects (novice)

Motion tracking sensor (magnetic position sensor) on the instrument

Bow strokes (motions, speed, accelerations, angles)

Gesture recognition (accurate movements are described by an expert)

FW: finger positions

Goal: make learner aware of bad finger positions to learn accurate gestures

-Feedback errors (bow strokes in a virtual 3D space) + colouring

-Score window (erroneous notes in musical scale)

(Iwasako et al., 2014)

Sign language

20 subjects

Motion capture system with data gloves

FW: position sensors (arm movements)

3D finger models rendered with a 3D modeller software

Rotation angles (FW: joint angles)

Error is calculated based on similarity

FW: influence of differences among learners (e.g., size of hand)

Goal: allow learners to recognize errors

Overlaying skilled person’s 3D models with learner’s 3D models of hands and fingers

FW: only learner model emphasizing wrong points


(Oakes et al., 2015)

Gym training (squats)

14 (8+6) athletes + coachers

Wearable knee sleeve (bend sensor with Arduino + push button to calibrate depth target).

Sensor vertically placed on the posterior knee

FW: accelerometer

Squats movement according to guidelines

Physical cues (depth was selected because it cannot be checked without a mirror): knee angle

FW: form overall (other factors: knees out, chest up…)

Goal: maximize exercise and avoid injury (in group settings)

Interviews (contextual inquiry) + video recordings (while performing squats thinking aloud)

Green light when desired squats’ depth is achieved

The push buttons allows individualized training (define target where user is physically capable)

Wearable ambient display (visually displays squat depth with a set of RGB LEDs)

Alternatives not selected: buzzer (vibro tactile) as it is noisy and distracting (in a group); unaware for coach

FW: mix visual and auditory cues, not make user look down

Legend: FW, future works; SVM, Support Vector Machine; HMM, Hidden Markov Models; FFT, Fast Fourier Transform; PCA, Principal Component Analysis; LDA, Local Discriminant Analysis; PNN, Probabilistic Neural Networks; FSM, Finite State Machines

Remark: bolded parts involve personalization issues, where AIED research can provide solutions

aEffect also known as the “cutaneous rabbit illusion” consisting in a tactile illusion evoked by tapping two or more separate regions of the skin in rapid succession

  1. 1.

    Sensing the learner’s corporal movement as specific skills are acquired, and the context in which this movement takes place.

  2. 2.

    Modelling the interactions to allow comparing the learner movement against the accurate movement (e.g., how an expert would carry the movement out).

  3. 3.

    Designing the feedback to be provided (i.e., what kind of support, and when and how to provide it).

  4. 4.

    Delivering the feedback in an effective non-intrusive way to advice the learner on how the body and limbs should move to achieve the motor learning goal.


In addition, each system compiled in Table 1 includes in the first column its name (when available), the bibliographical reference that has been analysed, the learning activity involved and the number of participants who took part in the evaluation of the system (when reported). In the analysis carried out, personalization issues (when implemented) have been identified and highlighted in Table 1 with bold face to facilitate its identification, as those are the relevant parts where AIED research can make relevant contributions. Works have been ordered chronologically (from oldest to most recent).

The review compiled in Table 1 shows diverse systems aimed to support motor skills learning. Following Magill’s classification (1993) in terms of precision most of the activities imply gross motor skills, although there are also examples of fine motor skills (e.g., hand writing, playing a musical instrument), beginning and end points of the movement are arbitrary (continuous motor skills) except for (Oakes et al. 2015; Lieberman and Breazeal 2007) and the stability of the environment involves changes in the response of the participant from one execution to the next one (except for sign language or handwriting), some stationary (playing a musical instrument) and others in motion (practicing a sport or martial art technique).

Many of these systems are intended for a diverse range of activities (Lieberman and Breazeal 2007; Miaw and Raskar 2010; Dancu 2012), although studies are usually carried out in a specific one. When user studies are carried out (in 12 of the 17 systems analysed), they are usually done so with a limited number of participants, and with the goal to show that the system functions as expected. Some aspects of the users experience are also analysed, such as whether users have positive views of the system (Takahata et al. 2004; Iwasako et al. 2014; Kikukawaa et al. 2014), whether they improve their performance while using the system (Kwon and Gross 2005; van der Linden et al. 2011; Ghasemzadeh et al. 2009; Portillo-Rodriguez et al. 2008), or how users respond to different feedback alternatives (Dancu 2012; Lieberman and Breazeal 2007). In another vein, most of the learning environments reviewed here address novice users. However, some works suggests that motor skills acquisition is also relevant, and even more appropriate, for advanced users (Spelmezan and Borchers 2008). Advanced users have different needs (Van der Linden et al. 2011) and/or behaviours (e.g., expert users are more consistent with the movement and the action is more controlled) (Ahmadi et al. 2006). In fact, as users might evolve from novice to experts with training, the system should also personalized its support and provide it accordingly (Takahata et al. 2004; Miaw and Raskar 2010; Van der Linden et al. 2011).

Next, the 17 systems analysed are discussed in relation to the four aspects considered in the review.

Sensing Movement in Motor Skills Learning Systems

The sensing of the movement is done i) through wearable devices on the body or the instrument used -when it exists (i.e., golf club, musical instrument, oar, etc.)- such as in (Oakes et al. 2015; Amma et al. 2014; James et al. 2011; Kikukawaa et al. 2014; Ghasemzadeh et al. 2009; Iwasako et al. 2014; Baca and Kornfeind 2006; van der Linden et al. 2011; Takahata et al. 2004; Spelmezan and Borchers 2008), ii) though optical motion capture devices placed in the environment (which usually require markers on the participants body, except for the Kinect sensor) such as in (Bloomfield and Badler 2008; Lieberman and Breazeal 2007; Dancu 2012; Miaw and Raskar 2010), or iii) combining both, that is within the user and instruments and from outside such as in (Rauter et al. 2011; Kwon and Gross 2005; Portillo-Rodriguez et al. 2008).

It has been suggested to move sensing outside the laboratory because participants need to be supported in situ (Oakes et al. 2015), since laboratory settings cannot reproduce the usual training (James et al. 2011), and there might be differences in usage when they are used in long term real practice (Hoggan and Brewster 2010). In addition, sensing in the wild allows a variety of context factors to be taken into account (Van der Linden et al. 2011). Wearable devices (which usually include accelerometers) eliminate the need that experiments are carried out solely inside the laboratory to obtain real-time feedback, and can be run in the real-world (Ahmadi et al. 2006). They can measure activity and effort levels and are usually small and light enough to be placed on any part of the body without hindering performance (Baca and Kornfeind 2006). In addition, and in contrast to optical devices, accelerometers provide precise angles and speed changes which are not visible to the eye (Kwon and Gross 2005). In any case, it is very important that wearable devices do not inhibit the user, and are easy to put on and remove, and are lightweight and unobtrusive (Miaw and Raskar 2010).

Many of the works reported in Table 1 show that wearables can provide straightforward information and do not hamper users’ movement (Kwon and Gross 2005). In much work, the requirement of low cost is also stated (Kwon and Gross 2005; Baca and Kornfeind 2006). However, expensive equipment is sometimes considered as proof of concept for initial experiments, with the expectation that the technology considered in them will have lower cost in the future (Lieberman and Breazeal 2007).

Beside the need to move outside the laboratory, some work claims the relevance of virtual environments to first practise physical skills (Bloomfield and Badler 2008). These virtual reality systems have been used to check how a trainee follows an avatar (Yang 1999). They use visual sensors to reconstruct the user’s posture and check how the trainee imitates the trainer’s motion by matching to a skeleton model to measure how learner mimics avatar motions (Kwon and Gross 2005). Virtual reality environments can provide augmented feedback by overlaying learners’ performance with desired movement, emphasizing the difference between subject and reference movement, and thus, highlighting desired trajectories (Lieberman and Breazeal 2007).

Modelling Movement in Motor Skills Learning Systems

Regarding the type of modelling, approaches are diverse. Some approaches focus on gesture recognition (Amma et al. 2014; Kikukawaa et al. 2014; Portillo-Rodriguez et al. 2008), others on physical features (Oakes et al. 2015; James et al. 2011) which can be modelled with 3D techniques (Bloomfield and Badler 2008; Iwasako et al. 2014; Van der Linden et al. 2011; Lieberman and Breazeal 2007), still others construct the models with diverse machine learning algorithms (Amma et al. 2014; Rauter et al. 2011; Kwon and Gross 2005).

Comparison is usually done against the movement carried out by an expert (Rauter et al. 2011; Ghasemzadeh et al. 2009; Miaw and Raskar 2010; Kwon and Gross 2005). When comparing expert’s movements with learner’s movements, a threshold is usually taken into account, and only when the threshold is overcome, is the feedback activated (Miaw and Raskar 2010; Bloomfield and Badler 2008). When the movement is complex, it is usually suggested to break the movement into less complicated actions (Ghasemzadeh et al. 2009; Kwon and Gross 2005; Portillo-Rodriguez et al. 2008). Very few systems mention the existence of a user model that records participants’ performance (Rauter et al. 2011; Lieberman and Breazeal 2007).

Designing Feedback in Motor Skills Learning Systems

In most of the papers, the design of the support provided is not described, as it seems to be considered a straight forward approach either i) to report the actual performance, usually as compared with an accurate performance and reporting when an error is made (Rauter et al. 2011; Kikukawaa et al. 2014; Ghasemzadeh et al. 2009; Iwasako et al. 2014), or ii) to show when the correct movement has been made (Oakes et al. 2015). Nonetheless, there are some works that aim to guide the learners’ movements in the appropriate way (Bloomfield and Badler 2008). In some cases, user centred design methods such as interviews have been followed to understand the feedback requirements (Oakes et al. 2015; Spelmezan and Borchers 2008).

Delivering Feedback in Motor Skills Learning Systems

When providing feedback, the focus of attention should be taken into account. It seems that if the focus is external (that is, on the movement’s effects rather than on body movements themselves, which are internal) such as it has been suggested in (Dancu 2012), it enhances motor performance and learning (Wulf 2013). However, in most of the work, the focus considered is internal.

The modalities for delivering the support can consist of visual, auditory and/or tactile cues. Humans have a natural parallel multimodal communication and interaction and prefer to interact multimodally (Portillo-Rodriguez et al. 2008). When humans learn a new motor skill from a teacher, they learn using multiple channels (Lieberman and Breazeal 2007): i) high level information about the skill received orally (abstract behavioural instructions), ii) visual information about how others perform the skill (demonstrating the motion themselves and requiring the learner to replicate the teacher’s performance), and iii) tactile information through the teacher’s physical guidance. However, some systems still rely only on visual feedback (Kwon and Gross 2005; James et al. 2011; Kikukawa et al. 2014; Iwasako et al. 2014). Sound, which was used in (Takahata et al. 2004; Baca and Kornfeind 2006; Portillo-Rodriguez et al. 2008; Rauter et al. 2011; Dancu 2012) can extend human perception, and feedback information can be provided by changing shape, tone and volume (Hollander and Furness 1994). The third type of feedback (i.e., haptic) is unique in motor skills learning environments, as it can only be applied when physical movements are considered in the learning process.

Haptic or tactile feedback presents the most direct form of motion information and directly engages the motor learning system, but it is the most difficult for a teacher to give (Lieberman and Breazeal 2007). It is provided exclusively for the user, which has advantages (it does not disturb others), but also disadvantages (it cannot be used to provide external assistance) (Oakes et al. 2015). It was originally developed for sensory substitution (e.g., to transmit information to deaf and/or blind people) rather than sensory augmentation (Lieberman and Breazeal 2007). Tactile feedback not only physically engages learners in the learning process, but might be the appropriate sensorial channel when learners require the visual and hearing senses for the learning tasks, such as when playing a musical instrument (Van der Linden et al. 2011). This kind of feedback is more effective when it points out mistakes learners regularly make (Van der Linden et al. 2011). Anyway, when experiencing and testing the feasibility of physical access in fully immersive experiences that can support learning new motor skills that require fairly precise physical body posture, visual realism might not sufficient and haptic support aimed for sensory substitution needs to be provided (Bloomfield and Badler 2008).

Tactile feedback can be provided in three different ways (Miaw and Raskar 2010; Lieberman and Breazeal 2007): i) vibrations with lightweight vibrating motors or tactors (tactile actuators), ii) physical movement (torque) produced with motors, and iii) electrical stimulation of muscles. This third option is often dangerous and painful, and in fact, no work has been found using it. The other two options distinguish between haptic feedback on movement (first option) and haptic guidance (second option). As for the first option, vibrotactile feedback can be provided using a small buzz in key moments, thus consolidating (touch to reward) or correcting (push or pull a limb, joint or other part of the body) a movement (Van der Linden et al. 2011). However, vibrotactile feedback might be ignored or undetected when cognitive demands are high and tasks are complex (Van der Linden et al. 2011). Anyway, an acclimation period is required, but that might still not be sufficient and participants may still not know how to respond to vibration signals at times (Lieberman and Breazeal 2007). Regarding the second option, haptic guidance prompts when a movement should be made, giving the opportunity to experience the feel or correcting the current posture or movement (Van der Linden et al. 2011). This physical guidance can be provided either with exoskeletons (force feedback on large body areas) (Bloomfield and Badler 2008) or torques. Torques require higher power and usually imply lower portability (Lieberman and Breazeal 2007).

From Neuroscience, it is reported that patients that have lost their ability to form new long-term memories still can build new motor skills, which shows that the brain processes motor learning differently from other conscious types of learning (Scoville and Milner 1957). This suggests that it may be possible to eventually train users to accept tactile feedback subconsciously to learn motor skills, turning corrections into automatic muscle reflex instead of a conscious mediation (Lieberman and Breazeal 2007).

Challenges and Opportunities to Personalize Motor Skills Learning in AIED Systems

As reported in the previous sections, systems aimed to support learners in motor skills acquisition have been developed outside the AIED community. Thus, it is not surprising that their personalization support is very limited. From all the systems reviewed in Table 1, those that describe higher personalization capabilities are M3-trainer (Rauter et al. 2011) and the MusicJacket (Van der Linden et al. 2011). Anyway, from the analysis of those 17 systems, some personalization opportunities for AIED research have been identified in Table 1 (and have been bolded there to facilitate its identification). The main personalization issues identified are the following: 1) take into account the specific capabilities of each learner (Oakes et al. 2015; Bloomfield and Badler 2008; Rauter et al. 2011; Takahata et al. 2004; Kwon and Gross 2005) and/or physical features (Iwasako et al. 2014; Van der Linden et al. 2011; Lieberman and Breazeal 2007; Miaw and Raskar 2010; Spelmezan and Borchers 2008; Portillo-Rodriguez et al. 2008), and 2) select the delivery of feedback depending on the learner’s preferences and/or context (Bloomfield and Badler 2008; Rauter et al. 2011; Van der Linden et al. 2011), or level of ability (Takahata et al. 2004; Miaw and Raskar 2010).

Therefore, there are challenges and opportunities for AIED researchers to build procedural learning ecosystems that personalize motor skills learning. As compiled in Table 2, AIED directions can be classified into: 1) modelling and representing the movements of the learner by building a learner psychomotor interaction model as well as an accurate movement model, and 2) providing the appropriate personalized psychomotor support in the most effective way for each learner in each training context. Regarding the first direction (i.e., modelling psycho-motor interaction), several AIED issues have to be addressed: i) to detect the physical interaction carried out by the learners by collecting data of their movements; ii) to model the accurate movements to be trained, which are those performed by experts; and iii) to diagnose the mismatch between the movements carried out by the learner and the expert’s way. Regarding the second direction (i.e., providing appropriate personalized psychomotor support), the issues involved are: iv) to model the intervention to be applied; v) to decide the intervention strategy to provide (e.g., showing performance, guiding, etc.), and when to apply it; and vi) to deliver the personalized intervention.
Table 2

AIED directions, research issues and challenges and opportunities, as well as novel approaches to be considered (interactive technologies and design methodologies)

AIED directions

AIED research issues

Challenges & Opportunities for AIED

Novel Approaches

Modelling psychomotor interaction

Detect learner physical interaction

Sensing the movement performed by the learner

Wearable devices

Big data processing

Model accurate movements to be trained (expert)

Modelling to allow comparing learner and expert movements

3D modelling

Diagnose mismatch between movements (learner vs. expert)


Providing personalized psychomotor support

Model the intervention

Designing the required support to give (when, what, how)

TORMES methodology

Decide the intervention strategy, and when to apply it

Deliver the personalized intervention

Delivering the personalized support (e.g., augmented sensorial feedback including vibrotactile, haptic guidance)

Ambient intelligence

3D printing (and also 4D)

Each of these issues poses several challenges and opportunities for AIED research in terms of representation and interaction issues, which might be addressed by applying novel interactive technologies such as wearable devices, big data processing, 3D modelling, 3D printing, ambient intelligence, as well as holistic design approaches such as TORMES methodology. These are compiled in Table 2 and discussed next.

Challenges and Opportunities in Sensing Movement in Motor Skills Learning Systems

Sensing the movement of a learner implies detecting and tracking the learner’s corporal movement as she learns specific skilled movements as well as the context in which this movement takes place.

Low cost and non-intrusive wearable sensors can collect, without interrupting the person’s movements, electronic inputs (rather than just collecting interaction data in the computer and observing problem solving steps as done in a pure cognitive domain) about pressure applied, gesture, speed, acceleration, angle, etc. from devices attached to the body (or the part(s) of the body that are involved in the movement) as well as on any object used by the learner. Novel interactive technologies, such as those provided by quantified-self wearable devices, can be used to gather dynamic indicators while making the movement. This can help to understand how the movements are performed and improve training. For instance, the movements carried out by a person can be monitored using diverse types of sensors (inertial, positional, physiological, etc.) (Schneider et al. 2015) for real-time motion studies outside the laboratory (Fong and Chan 2010).

In addition, as reported in Fleury et al. (2015) e-textile research already provides several solutions that allow remote and mobile measurement of functional movement and posture with low intrusiveness by using some sort of technology embedded directly into the fabric on clothes and objetcs worn by the learner and which allows tracking complex movement patterns (Bonato 2005). In particular, phenomena that can be sensed are strain (Gioberto and Dunne 2013; Giorgino et al. 2009; Preece et al. 2011; Tormene et al. 2012; Yamada et al. 2011), acceleration (Harms et al. 2010; Zysset et al. 2013), joint movements (Lee et al. 2011), bend angle (Lorussi et al. 2013), posture (Di Rienzo et al. 2009), and pressure (Shu et al. 2010).

The processing of the data collected needs to be carefully considered. It might be possible to do some processing within the tracking device, but at some point the device needs to wirelessly send its data streams to a cloud environment where multimodal data integration can take place in order to be able to obtain useful indicators that describe the physical interaction carried out. The interaction data streams continuously collected by these sensors become very difficult to make sense of on the fly. Due to its size, variability, and speed that are due to the continuous collection of information from diverse kinds of sensors, big data mining techniques need to be explored (Fan and Bifet 2013). The goal here is to extract movement indicators at the same rate that data are being processed, so the modelling can support just-in-time feedback to the learner.

Challenges and Opportunities in Modelling Movement in Motor Skills Learning Systems

Modelling a learner’s movements implies comparing the learner’s movements against accurate movements, such as the one that an expert with similar physical capabilities of the learner would perform. For a given activity, the movements for which personalized support needs to be provided have to be identified, in a way that they can be isolated from one another. Depending on the kind of activity, this can be already defined in the domain, or it might imply extracting this information from domain experts. For each of these movements, the accurate way to carry out the movement has to be modelled. This can be done by recording the physical interaction performed by experts in the field. Since it is not possible for any person to perform a given movement in the same exact way each time, the model of the accurate movement should be built from several recordings that should involve not only a series of movements by the same expert, but also by other experts. From this set of recordings, the “accurate” movement should be extrapolated in terms of the relevant parameters that characterize it.

In the field of virtual reality, there is work that builds virtual skeletal models for videogames from the information collected using wearable technology (e.g., biomechanical or inertial sensors), which both map the movement as well as recognize gestures with artificial intelligence techniques (Arsenault 2014). The movement controlled by sensors can also be represented in 3D models of the human body (Kifayat et al. 2010; Bae et al. 2012). The biometric function that represents the human body can parameterize the human posture (DellaGrotte et al. 2008) as well as gestures produced by the body limbs, such as the hands (Marcel 2002). The parameter should make possible to identify when a movement fits in the “accurate” model (i.e., it is equivalent to a movement that would be produced by an expert), and when not (i.e., it is produced by someone with less expertise who is trying to learn the movement), despite the inherent variability in the replication of a given movement commented upon above.

In addition, a learner model is needed to describe the learners’ body build as well as her psychomotor skills, level of ability and progress towards mastering the various movements. For this, the model should i) describe the learner’s physical features, such as hand length, maximum shoulder opening, etc.; ii) provide information about her interaction capabilities, such as force, speed, etc.; and iii) contain indicators regarding the way she can perform the different kinds of movements to be learnt (e.g., level of performance).

Challenges and Opportunities in Designing Feedback in Motor Skills Learning Systems

When the movement does not fit into the accurate movement model, an intervention might be necessary and needs to be designed. Thus, a key area where AIED has a big role to play is designing the feedback to be provided. In particular, whether it is appropriate to provide support at this moment and, if so, what kind of support to provide. In a personalized system, this implies deciding when to interact with the learner as well as what to ask her to do and how to advise her on how to do it. The focus in personalizing motor skills learning has to be put on identifying what is the most appropriate intervention in each case (considering cognitive, affective and psychomotor dimensions) and when and how it should be delivered in order to make a positive impact on the learning process. It should go beyond traditional intervention approaches which are based on giving instructions that describe what to do and how to do it. In fact, procedural learning in terms of motor skills is usually difficult to explain by the instructor and to understand by the learner. And it turns out that this procedural tutoring support is of major relevance in the case of novice learners, as they might get into a wrong habit if no timely feedback is provided to them while practicing on their own and, thus, they cannot understand why the movement is not accurate (Kikukawaa et al. 2014).

In acquiring motor skills there is a trade-off between learning (i.e., relatively permanent changes in behaviour that support long-term retention and transfer) and performance (i.e., temporary fluctuations in behaviour that are observed and measured during training or immediately thereafter) that should also be taken into account (Soderstrom and Bjork 2015). In particular, decisions that have a positive effect on long-term learning (such separating practice with time and other activities, varying conditions in practice, and making self-produced movements) introduce more performance errors, and thus, might frustrate the learner. Thus, when at some point the learner is frustrated with the practice, it might be better to support short-term performance by externally guiding the learner rather than forcing self-produced movements that would improve her long-term progress. Short-term performance (but not long-term learning) is also improved practicing the same thing over and over, and without varying the conditions, but that increases boredom. Such frustration and boredom mean that motivation needs to be strongly supported considering the state of the art in affective computing (Santos 2016a). Anyway, when a skill is acquired, feedback should be reduced to avoid a dependency on it (Sigrist et al. 2013).

Another issue of major importance is the selection of the feedback modality. Visual feedback can be used to understand the movement (Sigrist et al. 2013) and improve motor skills without the aid of a live instructor. In motion capture systems, feedback can be provided comparing the user movements with the body positions of an avatar performing expert movements. In addition to presenting sensor data to the learner, this allows supporting awareness and self-reflection of how movements carried out by the learner are aligned with the accurate version of the movements. 3D representations are of relevance here. Haptic feedback is a new modality that can take advantage of the physical interactions, but it might not always be the most appropriate approach. For instance (Marchal-Crespo et al. 2013) found that haptic guidance was especially suitable for less-skilled subjects and in especially difficult discrete tasks, while visual feedback seemed to benefit more skilled subjects. Additionally, haptic guidance seemed to promote learning in a time-critical tracking task.

Therefore, instructors need to be involved in an elicitation process in order to design the most effective feedback approaches that suit particular learners’ needs and their context. The knowledge to be elicited is tacit and implicit in the educators’ head, as it has been acquired through years of teaching, and typically never made explicit in a systematic way. In order to identify appropriate interventions that produce a positive impact in the learning process, TORMES elicitation methodology can be applied (Santos and Boticario 2015). TORMES combines user centred design methods with data mining techniques and extends the design cycle of interactive systems as defined by ISO 9241–210 (ISO 2010) with the life cycle of e-learning (Van Rosmalen et al. 2004) and the layered evaluation of adaptive systems (Paramythis et al. 2010). Resulting interventions are semantically described in terms of i) what is to be recommended, ii) how and where the recommendation is to be communicated, iii) when and to who the recommendation is to be offered, iv) why the recommendation is to be delivered, and v) which features characterize the recommendations themselves (Santos and Boticario 2011). In order to address the particularities required by the psychomotor learning domain and the requirements to model psychomotor interaction and provide appropriate support, an extension to TORMES might be needed. In any case, TORMES can be used by educators to apply appropriate user centred design methods to gather tacit knowledge through qualitative descriptions from psychomotor experts while analysing performance indicators of learners regarding movements carried out while training. These indicators can be computed with (big) data mining techniques in order to gather quantitative information from data collected in learning experiences with wearable devices. As a result, this elicitation process can facilitate the identification of the psychomotor support to be provided as the qualitative analysis on quantitative indicators can rise up unaware opportunities for motor skills personalization support that should improve training performance.

Challenges and Opportunities in Delivering Feedback in Motor Skills Learning Systems

Different types of feedback (either positive or negative) can promote motor skills acquisition (Ashby and O’Brien 2007). Feedback can be provided in several ways (e.g., visual, auditory, haptic), although a multimodal approach might produce better results (Sigrist et al. 2013; Portillo-Rodriguez et al. 2008). For instance, ambient intelligence can make the environment sensitive and responsive to the presence of people, hence it can be an interesting approach to provide multimodal sensorial feedback through different sensorial channels, as provided by the AICARP system that has been discussed elsewhere (Santos et al. 2015, 2016). In that approach, the learner configures her preferred sensorial channel(s) for feedback delivery, which could be stored in the learner model and even learnt by the system.

The main novelty in motor skills learning environments is the delivery of haptic support. This support can be provided with vibrotactile feedback or with haptic guidance. The former can be delivered through diverse actuators such as resistance, force, vibration, etc. to tell the learner when her movement does not reflect the accurate movement that is used as reference, or to reward a good movement to consolidate it. Actuators can be sewed on clothes to be worn by the learner, such as on the sleeve of a jacket. Sewable electronic modules controlled by Lilypad Arduino can be used for prototyping the system. An extended review on systems that deliver vibrotactile feedback is compiled elsewhere (Santos and Revilla 2016). In turn, haptic guidance can be delivered in terms of a tangible support that scaffolds the learner through an exoskeleton attached on the body on how the body and limbs should move to achieve the learning goal. As an example, the exoskeleton robotics developed by Datta (2014) can move the fingers of the learner to make the movement considered to be accurate by the system. The technology for 3D modelling combined with 3D printing make feasible the building of physical prototypes of tangible objects that can create such exoskeletons to physically scaffold the learner movement, as discussed in (Santos 2016b). In addition, the very emerging 4D technology (which produces physical objects that can change with time) can be an interesting option to explore in order to dynamically personalized the exoskeleton used for the guiding.


There exist many and diverse types of activities (e.g., playing a musical instrument, handwriting, drawing, training for surgery, improving the technique in sports and martial arts, learning sign language, dancing, etc.) that require long-term physical training to learn how to perform the movements in the most efficient way. The training process is based on the learning-by-doing approach and involves correcting the learner’s movements till her optimal movements (considering the own learner’s corporal features, specific physical abilities, performance to date as well as the particularities of the motor skills to be learnt) is achieved. Defining the accurate movement and when and how to provide the physical support is not trivial at all, but is indeed a modelling challenge for AIED.

No specific work on motor skills learning has been found in the review of IJAIED papers reported here (although some signs of their potential interest have been detected and commented upon in this paper). In turn, outside the AIED community, diverse learning environments for motor skills learning have been developed, but have almost no personalization features, which are clearly specific of AIED systems. This shows a gap in AIED research that can become an opportunity thanks to emerging novel interaction technologies and design methodologies.

In fact, the review of IJAIED papers has shown that AIED research has historically reflected the technology of the times (Schank and Edelson (1989/1990); Eggert (1990)). Following this, I concur that AIED systems should take advantage of those technological advances that can make the field progress, although there is a need to carefully analyse the best way to apply them. At this point in time, my vision for the future of the AIED field in the psychomotor learning domain is that the synergy of artificial intelligence techniques with novel interaction technologies such as quantified-self wearable devices from the “Internet of Me”, 3D modelling and 3D printing, big data streams processing on the fly and ambient intelligence sensorial support, among others, can breathe new life into the AIED research and open new opportunities for building procedural AIED ecosystem of standard-based interactive learning tools that can personalize motor skills learning by providing intelligent real-time feedback to scaffold learning in the psychomotor domain. Moreover, design methodologies such as TORMES can guide the elicitation of appropriate psychomotor interventions supported by models of expert and learner behaviour to underpin this personalization effort.

The selection and reporting of research and technologies in this paper does not aim to be exhaustive, but to suggest some technological directions to be explored by the AIED research community in the coming years. In particular, the work reported here suggests that there exists a technological context that can support the AIED research in developing systems that sense the learner’s corporal behaviour as she learns specific skilled movements and when needed, provide the appropriate support aimed to make the learner acquire new motor skills (or refine existing skills) aided by developing a muscle memory experience along the way. This requires novel forms of expert as well as learner modelling, novel forms of personalized support suitable for each learner in each training context, etc., so this is indeed a potential new research frontier in AIED. There are four main issues to be addressed: i) sensing the movement performed by the learner, ii) modelling and comparing learners’ and experts’ movements, iii) designing the required support to give, and iv) delivering the personalized support. Interestingly, by putting motor skills learning in the AIED research agenda for the next years, the field is somehow also going back to its roots due to the need to revisit many of our fundamental and driving questions, but in the context of motor skills. Substantial evidence exists to suggest that important similarities exist between motor and cognitive learning (Bjork and Bjork 1992).

Given the enormous amount of resources spent every year on motor skills training throughout society, this research area might turn out to be extremely important, and thus, it can turn into a good opportunity for the AIED field. The added value of AIED is to provide the personalization support required to build procedural learning ecosystems that can individualize the learning of motor skills.


  1. 1.

    A movement to incorporate technology into data acquisition on aspects of a person’s daily life (i.e., self-knowledge through self-tracking with technology) (Wolf 2010).

  2. 2.

    The do-it-yourself movement encourages people in creating things by their own, such as those that involve physical objects connected to the Internet, the so called “Internet of Things” (De Roeck et al. 2012)



The author would like to thank her colleagues of the aDeNu research group at UNED as well as the participants of the workshop “Les Contes du Marriage: Should AI stay married to ED?” that took place during the AIED 2015 conference held in Madrid (Spain), for all the valuable comments provided when the ideas that have been elaborated in this paper were initially discussed with them. In addition, she would like to thank the suggestions provided by the reviewers of this paper which definitely hit the target, as well as the extraordinary support provided by the guest editors of the Special Issue “The next 25 Years: How advanced, interactive educational technologies will change the world” where this paper is published, support that has been channelled through Chad Lane and Gord McCalla. And also very special thanks to Gord McCalla and Ben du Boulay who were really effective in handling her request of searching missing early papers published by IJAIED in the late eighties end early nineties about procedural learning that were not available in the online archival of IJAIED, and which served to provide the historical background of the AIED research in this topic.


  1. Ahmadi, A., Rowlands, D.D., James, D.A., Ahmadi, A. (2006). Investigating the translational and rotational motion of the swing using accelerometers for athlete skill assessment. 5th IEEE Conference on Sensors, 980–983.Google Scholar
  2. Alvarez, N., Sanchez-Ruiz, A., Cavazza, M., Shigematsu, M., & Prendinger, H. (2015). Narrative balance management in an intelligent biosafety training application for improving user performance. Journal of Artificial Intelligence in Education, 25(1), 35–59.CrossRefGoogle Scholar
  3. Amma, C., Georgi, M., & Schultz, T. (2014). Airwriting: a wearable handwriting recognition system. Personal Ubiquitous Computing, 18(1), 191–203.CrossRefGoogle Scholar
  4. Anderson, J. R. (1982). Acquisition of cognitive skill. Psychological Review, 89, 369–403.CrossRefGoogle Scholar
  5. Andriessen, J., & Sandberg, J. (1999). Where is education heading and how about AI? Journal of Artificial Intelligence in Education, 10(2), 130–150.Google Scholar
  6. Arsenault, D. (2014). A quaternion-based motion tracking and gesture recognition system using wireless inertial sensors. Master of Applied Science in Human-Computer Interaction. Carleton University.Google Scholar
  7. Ashby, F. G., & O’Brien, J. B. (2007). The effects of positive versus negative feedback on information-integration category learning. Perception & Psychophysics, 69, 865–887.CrossRefGoogle Scholar
  8. Baca, A., & Kornfeind, P. (2006). Rapid feedback systems for elite sports training. IEEE Pervasive Computing, 5, 70–76.CrossRefGoogle Scholar
  9. Bae, J., Haninger, K., Wai, D., Garcia, X., Tomizuka, M. (2012). A network-based monitoring system for rehabilitation. The 2012 IEEE/ASME International Conference on Advanced Intelligent Mechatronics, 232–237.Google Scholar
  10. Baker, M. (2000). The roles of models in artificial intelligence in education research: a prospective view. Journal of Artificial Intelligence in Education, 11(2), 122–143.Google Scholar
  11. Bjork, R. A., & Bjork, E. L. (1992). A new theory of disuse and an old theory of stimulus fluctuation. From learning processes to cognitive processes: Essays in honor of William K. Estes, 2, 35–67.Google Scholar
  12. Bloomfield, A., Badler, N. (2008). Virtual training via vibrotactile arrays. Teleoperator and Virtual Environments 17.Google Scholar
  13. Bonato, P. (2005). Advances in wearable technology and applications in physical medicine and rehabilitation. Journal of NeuroEngineering and Rehabilitation, 2, doi: 10.1186/1743-0003-2-2.
  14. Brunner, J. (1964). The course of cognitive growth. American Pyschologist, 19, 1–15.CrossRefGoogle Scholar
  15. Christensson, J. (2005). Individualising surgical training in a virtual haptic environment. MSc Thesis in Interaction Design. IT University of Göteborg (Sweden).Google Scholar
  16. Dancu, A. (2012). Motor learning in a mixed really environment. NordiCHI, 2012, 811–812.CrossRefGoogle Scholar
  17. Datta, S. (2014). Forced fingers. Available from Github:
  18. De Roeck, D., Slegers, K., Criel, J., Godon, M., Claeys, L., Kilpi, K., & Jacobs, A. (2012). I would DiYSE for it!: A manifesto for do-it-yourself internet-of-things creation. In Proceedings of the 7th Nordic conference on human-computer interaction: Making sense through design (NordiCHI '12) (pp. 170–179). New York, NY, USA: ACM.CrossRefGoogle Scholar
  19. DellaGrotte, J., Ridi, R., Landi, M., & Stephens, J. (2008). Postural improvement using core integration to lengthen myofascia. Journal of Bodywork and Movement Therapies, 12, 231–245.CrossRefGoogle Scholar
  20. Di Rienzo, M., Rizzo, F., Meriggi, P., Castiglioni, P., Mazzoleni, P., Parati, G., Bordoni, B., Brambilla, G., & Ferratini, M. (2009). MagIC system. IEEE Engineering in Medicine and Biology Magazine, 28, 35–40.CrossRefGoogle Scholar
  21. Eggert, A. A. (1990). A rebuttal to “a role for AI in education: using technology to reshape education”. Journal of Artificial Intelligence in Education, 1(3), 3–9.Google Scholar
  22. Fan, W., & Bifet, A. (2013). Mining big data: current status, and forecast to the future. SIGKDD Explorations Newsletter, 14(2), 1–5.CrossRefGoogle Scholar
  23. Fleury, A., Sugar, M., & Chau, T. (2015). E-textiles in clinical rehabilitation: a scoping review. Electronics, 4, 173–203.CrossRefGoogle Scholar
  24. Fong, D. T.-P., & Chan, Y.-Y. (2010). The use of wearable inertial motion sensors in human lower limb biomechanics studies: a systematic review. Sensors, 10, 11556–11565.CrossRefGoogle Scholar
  25. Gagné, R. M., & Briggs, L. J. (1979). Principles of instructional design (2nd ed.). New York: Holt, Rinehart, and Winston, Inc.Google Scholar
  26. Ghasemzadeh, H., Loseu, V., & Jafari, R. (2009). Wearable coach for sport training: a quantitative model to evaluate wrist-rotation in golf. Environments, 1, 1–12.Google Scholar
  27. Gioberto, G., & Dunne, L. E. (2013). Overlock-stitched stretch sensors: characterization and effect of fabric property. Journal of Textile and Apparel Technology and Management, 8, 1–14.Google Scholar
  28. Giorgino, T., Tormene, P., Lorussi, F., de Rossi, D., & Quaglini, S. (2009). Sensor evaluation for wearable strain gauges in neurological rehabilitation. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 17, 409–415.CrossRefGoogle Scholar
  29. Harms, H., Amft, O., & Troester, G. (2010). Estimating posture-recognition performance in sensing garments using geometric wrinkle modeling. IEEE Transactions on Information Technology in Biomedicine, 14, 1436–1445.CrossRefGoogle Scholar
  30. Harrow, A. (1972). A taxonomy of psychomotor domain: A guide for developing behavioral objectives. New York, NY, USA: David McKay.Google Scholar
  31. Hoggan, E., & Brewster, S. A. (2010). CrossTrainer: Testing the use of multimodal interfaces in situ. In Proceedings of ACM CHI. Atlanta, USA: ACM Press.Google Scholar
  32. Hollander, A.J., & Furness III, T.A. (1994). Perception of virtual auditory shapes. In: Proceedings of the international conference on auditory displays.Google Scholar
  33. Howell, R. D., & Hay, K. E. (1989). Software-based access and control of robotics manipulators for severely physically disabled students. Journal of Artificial Intelligence in Education, 1(1), 53–72.Google Scholar
  34. ISO (2010). Ergonomics of human-system interaction - part 210: human-centred design for interactive systems. ISO 9241–210:2010.Google Scholar
  35. Iwasako, K., Soga, M., & Taki, H. (2014). Development of finger motion skill learning support system based on data gloves. 18th Int. Conf.on knowledge-based and intelligent information & engineering systems (KES2014). Procedia Computer Science, 35, 1307–1314.CrossRefGoogle Scholar
  36. James, D. A., Burkett, B., & Thiel, D. V. (2011). An unobtrusive swimming monitoring system for recreational and elite performance monitoring. Procedia Engineering, 13, 113–119.CrossRefGoogle Scholar
  37. Kagawa, T., Motomura, M. Nishino, H. (2014). A 3D editing method with hand gesture using sound information. Eighth International Conference on Complex, Intelligent and Software Intensive Systems, 637–642.Google Scholar
  38. Kifayat, K., Fergus, P., Cooper, S., Merabti, M. (2010). Body area networks for movement analysis in physiotherapy treatments. En IEEE 24th International Conference on Advanced Information Networking and Applications Workshops, 866–872.Google Scholar
  39. Kikukawaa, F., Sogab, M., & Takib, H. (2014). Development of a gesture learning environment for novices’ erhu bow strokes. 18th Int. Conf. on knowledge-based and intelligent information & engineering systems (KES2014). Procedia Computer Science, 35, 1323–1332.CrossRefGoogle Scholar
  40. Krakauer, J. W., & Shadmehr, R. (2006). Consolidation of motor memory. Trends in Neurosciences, 29, 58–64.CrossRefGoogle Scholar
  41. Kwon, D.Y., Gross, M. (2005). Combining body sensors and visual sensors for motion training. In Proceedings of advances in computer entertainment technology (pp. 94–101). Valencia, Spain.Google Scholar
  42. Lee, B. W., Lee, C., Kim, J., & Lee, M. (2011). Optimum conductive fabric sensor sites for evaluating the status of knee joint movements using bio-impedance. Biomedical Engineering Online, 10, 48.CrossRefGoogle Scholar
  43. Lieberman, J., & Breazeal, C. (2007). TIKL: development of a wearable vibrotactile feedback suit for improved human motor learning. IEEE Transactions on Robotics, 23(5), 919–926.CrossRefGoogle Scholar
  44. Lorussi, F., Galatolo, S., & Bartalesi, R. (2013). Modeling and characterization of extensible wearable textile-based electrogoniometers. IEEE Sensors Journal, 13, 217–228.CrossRefGoogle Scholar
  45. Magill, R. A. (1993). Motor learning: Concepts and applications (4th ed.). Madison: Brown & Benchmark.Google Scholar
  46. Marcel, S. (2002). Gestures for multi-modal interfaces: a review. IDIAP Research Report 02–34.Google Scholar
  47. Marchal-Crespo, L., van Raai, M., Rauter, G., Wolf, P., & Riener, R. (2013). The effect of haptic guidance and visual feedback on learning a complex task. Experimental Brain Research, 231(3), 277–291.CrossRefGoogle Scholar
  48. Miaw, D., & Raskar, R. (2010). “Second skin: motion capture with actuated feedback for motor learning.” In Virtual reality conference (VR), 2010 I.E. (pp. 289–290, 20–24).Google Scholar
  49. Nonnon, P., & Theil, J. P. (1990). Robot-based pedagogy. Journal of Artificial Intelligence in Education, 2(1), 15–20.Google Scholar
  50. Nye, B. D. (2015). AIED is splitting up (into services) and the next generation will be all right. In Les contes du marriage: Should AI stay married to ED? A workshop examining the current and future identity of the AIED field. CEUR Workshop Proceedings, AIED 2015 workshops 1432(4), 62–71.Google Scholar
  51. Oakes, K., Siek, K.A., MacLeod, H. (2015). Musclememory: identifying the scope of wearable technology in high intensity exercise communities. 9th International Conference on Pervasive Computing Technologies for Healthcare.Google Scholar
  52. Ohlsson, S. (1993). Learning to do and learning to understand: A lesson and a challenge for cognitive modeling. In P. Reimann & H. Spada (Eds.), Learning in humans and machines (pp. 37–62). Oxford: Pergamon Press.Google Scholar
  53. Paramythis, A., Weibelzahl, S., & Masthoff, J. (2010). Layered evaluation of interactive adaptive systems: framework and formative methods. User Modeling and User-Adapted Interaction, 20(5), 383–453.CrossRefGoogle Scholar
  54. Penney, S. (2011). Psychomotor domain. In B. Hoffman (Ed.), Encyclopedia of educational technology. San Diego, CA: SDSU Department of Educational Technology. Retrieved October 30, 2015, from Scholar
  55. Portillo-Rodriguez, O., Sandoval-Gonzalez, O., Ruffaldi, E., Leonardi, R., Avizzano, C., & Bergamasco, M. (2008). Real-time gesture recognition, evaluation and feed-forward correction of a multimodal Tai-Chi platform. En LNCS, 5270, 30–39.Google Scholar
  56. Preece, S. J., Kenney, L. P., Major, M. J., Dias, T., Lay, E., & Fernandes, B. T. (2011). Automatic identification of gait events using an instrumented sock. Journal of NeuroEngineering and Rehabilitation, 8, 32.CrossRefGoogle Scholar
  57. Rabardel, P. (1991). Activity with a training robot and the formation of knowledge. Journal of Artificial Intelligence in Education, 2(4), 3–14.Google Scholar
  58. Rauter, G., Sigrist, R., Baur, K., Baumgartner, L., Riener, R., & Wolf, P. (2011). A virtual trainer concept for robot-assisted human motor learning in rowing. BIO Web of Conferences, 1, 00072.CrossRefGoogle Scholar
  59. Santos, O. C. (2015). Education still needs artificial intelligence to support motor skill learning. a case study with Aikido. In Les contes du marriage: Should AI stay married to ED? A workshop examining the current and future identity of the AIED field. CEUR Workshop Proceedings, AIED 2015 workshops, 1432(4), 72–81.Google Scholar
  60. Santos, O. C. (2016a). Emotions and personality in e-learning systems: an affective computing perspective. In Tkalčič, DeCarolis, de Gemmis, Odić, Košir (Eds.), Emotions and personality in personalized services. Springer (accepted).Google Scholar
  61. Santos, O. C. (2016b) Beyond cognitive and affective issues. Tangible recommendations for psychomotor personalized learning. In Spector, J.M., Lockee, B.B. and Childress, M.D. (Eds.), Learning, Design, and Technology. An International Compendium of Theory, Research, Practice, and Policy. Springer (accepted).Google Scholar
  62. Santos, O. C., & Boticario, J. G. (2011). Requirements for semantic educational recommender systems in formal e-learning scenarios. Algorithms, 4, 131–154.CrossRefGoogle Scholar
  63. Santos, O. C., & Boticario, J. G. (2015). Practical guidelines for designing and evaluating educationally oriented recommendations. Computer and Education, 81, 354–374.CrossRefGoogle Scholar
  64. Santos, O. C. & Revilla, O. (2016) Designing Vibrotactile User Interfaces: from Wireframes to Wirebodies. Computer. Special issue on User Interface Design in the 21st Century (under review).Google Scholar
  65. Santos, O. C., Saneiro, M., Boticario, J.G., & Rodriguez-Sanchez, C. (2015). Towards interactive context-aware affective educational recommendations in computer assisted language learning. In New Review of Hypermedia and Multimedia, published on-line.Google Scholar
  66. Santos, O. C., Uria-Rivas, R., Rodriguez-Sanchez, M.C., & Boticario, J.G. (2016) An open sensing and acting platform for context-aware affective support in ambient intelligent educational settings. IEEE Sensors (accepted).Google Scholar
  67. Schank, R. C., Edelson, D. J. (1989/1990). A role for AI in education: using technology to reshape education. Journal of Artificial Intelligence in Education, 1(2), 3–20.Google Scholar
  68. Schneider, J., Börner, D., van Rosmalen, P., & Specht, M. (2015). Augmenting the senses: a review on sensor-based learning support. Sensors, 15, 4097–4133.CrossRefGoogle Scholar
  69. Scoville, W. B., & Milner, B. (1957). Loss of recent memory after bilateral hippocampal lesions. Journal of Neurology, Neurosurgery, and Psychiatry, 20(1), 11–21.CrossRefGoogle Scholar
  70. Shu, L., Hua, T., Wang, Y., Li, Q., Feng, D. D., & Tao, X. (2010). In-shoe plantar pressure measurement and analysis system based on fabric pressure sensing array. IEEE Transactions on Information Technology in Biomedicine, 14, 767–775.CrossRefGoogle Scholar
  71. Sigrist, R., Rauter, G., Riener, R., & Wolf, P. (2013). Augmented visual, auditory, haptic, and multimodal feedback in motor learning: a review. Psychonomic Bulletin & Review, 20, 21–53.CrossRefGoogle Scholar
  72. Soderstrom, N. C., & Bjork, R. A. (2015). Learning versus performance: an integrative review. Perspectives on Psychological Science, 10(2), 176–199.CrossRefGoogle Scholar
  73. Spelmezan, D., Borchers, J. (2008). Real-time snowboard training system. In Proceedings of the extended abstacts on human factors in computing systems (CHI’08), (pp. 3327–3332). Florence, Italy.Google Scholar
  74. Swinnen, S. P. (1996). Information feedback for motor skill learning: A review. In H. N. Zelaznik (Ed.), Advances in motor learning and control (pp. 37–66). Champaign: Human Kinetics.Google Scholar
  75. Takahata, M., Shiraki, K., Sakane, Y., Takebayashi, Y. (2004). Sound feedback for powerful karate training. In Proceedings of new interfaces for musical expression, (pp. 13–18). Hamamatsu, Japan.Google Scholar
  76. Tormene, P., Bartolo, M., De Nunzio, A. M., Fecchio, F., Quaglini, S., Tassorelli, C., & Sandrini, G. (2012). Estimation of human trunk movements by wearable strain sensors and improvement of sensor’s placement on intelligent biomedical clothes. Biomedical Engineering Online, 11, 95.CrossRefGoogle Scholar
  77. Underwood, J. & Luckin, R. (2011). What is AIED and why does education need it? A Report for the UK’s TLRP technology enhanced learning – artificial intelligence in education theme.
  78. Van der Linden, J., Johnson, R., Bird, J., Rogers, Y., Schoonderwaldt, E. (2011). Buzzing to play: Lessons learned from an in the wild study of real-time vibrotactile feedback. In Proceedings of the conference on human factors in computing systems (CHI’11), (pp. 533–543). Vancouver, BC, Canada.Google Scholar
  79. Van Rosmalen, P., Boticario, J. G., & Santos, O. C. (2004). The full life cycle of adaptation in aLFanet elearning environment. Learning Technology Newsletter, 4, 59–61.Google Scholar
  80. Westerfield, G., Mitrovic, A., & Billinghurst, M. (2015). Intelligent augmented reality training for motherboard assembly. Journal of Artificial Intelligence in Education, 25(1), 157–172.CrossRefGoogle Scholar
  81. Wolf, G. (2010). The quantified self. TED (conference). Available from
  82. Wulf, G. (2013). Attentional focus and motor learning: a review of 15 years. International Review of Sport and Exercise Psychology, 6(1), 77104.MathSciNetCrossRefGoogle Scholar
  83. Wulf, G., Shea, C., & Lewthwaite, R. (2010). Motor skill learning and performance: a review of influential factors. Medical Education, 44, 75–84.CrossRefGoogle Scholar
  84. Yamada, T., Hayamizu, Y., Yamamoto, Y., Yomogida, Y., Izadi-Najafabadi, A., Futaba, D. N., & Hata, K. (2011). A stretchable carbon nanotube strain sensor for human-motion detection. Nature Nanotechnology, 6, 296–301.CrossRefGoogle Scholar
  85. Yang, U. (1999). Just follow me: An immersive vr-based motion training system. In Proceedings of international conference on virtual systems and multimedia Google Scholar
  86. Zysset, C., Kinkeldei, T., Münzenrieder, N., Petti, L., Salvatore, G., & Tröster, G. (2013). Combining electronics on flexible plastic strips with textiles. Textile Research Journal, 83, 1130–1142.Google Scholar

Copyright information

© International Artificial Intelligence in Education Society 2016

Authors and Affiliations

  1. 1.aDeNu Research Group. Artificial Intelligence Dept. Computer Science SchoolUNEDMadridSpain

Personalised recommendations